Cortex Files

What's Changed

• [CHANGE] Deprecate -blocks-storage.tsdb.wal-compression-enabled flag (use blocks-storage.tsdb.wal-compression-type instead). [#6529]
• [CHANGE] OTLP: Change OTLP handler to be consistent with the Prometheus OTLP handler. [#6272]
• target_info metric is enabled by default and can be disabled via -distributor.otlp.disable-target-info=true flag
• Convert all attributes to labels is disabled by default and can be enabled via -distributor.otlp.convert-all-attributes=true flag
• You can specify the attributes converted to labels via -distributor.promote-resource-attributes flag. Supported only if -distributor.otlp.convert-all-attributes=false
• [CHANGE] Change all max async concurrency default values 50 to 3 [#6268]
• [CHANGE] Change default value of -blocks-storage.bucket-store.index-cache.multilevel.max-async-concurrency from 50 to 3 [#6265]
• [CHANGE] Enable Compactor and Alertmanager in target all. [#6204]
• [CHANGE] Update the cortex_ingester_inflight_push_requests metric to represent the maximum number of inflight requests recorded in the last minute. [#6437]
• [CHANGE] gRPC Client: Expose connection timeout and set default to value to 5s. [#6523]
• [CHANGE] Enable analysis on Thanos PromQL engine [#6472]
• [CHANGE] Log grafana headers along with query request before query execution [#6391]
• [CHANGE] Log query requests in QFE before query execution [#6390]
• [CHANGE] Log when a request starts running in querier [#6525]
• [CHANGE] Remove openstack swfit experimental in blocks storage [#6322]
• [CHANGE] Remove openstack swift experimental [#6316]
• [CHANGE] Replace cespare/xxhash with cespare/xxhash/v2 [#6467]
• [CHANGE] Stop using global instant query codec [#6328]
• [CHANGE] Unify query frontend instant and range protos [#6180]
• [FEATURE] Ruler: Add an experimental flag -ruler.query-response-format to retrieve query response as a proto format. [#6345]
• [FEATURE] Ruler: Pagination support for List Rules API. [#6299]
• [FEATURE] Query Frontend/Querier: Add protobuf codec -api.querier-default-codec and the option to choose response compression type -querier.response-compression. [#5527]
• [FEATURE] Ruler: Experimental: Add ruler.frontend-address to allow query to query frontends instead of ingesters. [#6151]
• [FEATURE] Ruler: Minimize chances of missed rule group evaluations that can occur due to OOM kills, bad underlying nodes, or due to an unhealthy ruler that appears in the ring as healthy. This feature is enabled via -ruler.enable-ha-evaluation flag. [#6129]
• [FEATURE] Store Gateway: Add an in-memory chunk cache. [#6245]
• [FEATURE] Chunk Cache: Support multi level cache and add metrics. [#6249]
• [FEATURE] Distributor: Accept multiple HA Tracker pairs in the same request. [#6256]
• [FEATURE] Ruler: Add support for per-user external labels [#6340]
• [FEATURE] Query Frontend: Support a metadata federated query when -tenant-federation.enabled=true. [#6461]
• [FEATURE] Query Frontend: Support an exemplar federated query when -tenant-federation.enabled=true. [#6455]
• [FEATURE] Ingester/StoreGateway: Add support for cache regex query matchers via -ingester.matchers-cache-max-items and -blocks-storage.bucket-store.matchers-cache-max-items. [#6477] [#6491]
• [FEATURE] Add reason why the key was evicted in the cortex_ingester_expanded_postings_cache_evicts metric [#6318]
• [FEATURE] Create feature flag to switch between current shuffle sharding group planner and partition compaction group planner [#6141]
• [FEATURE] Hook up partition compaction end to end implementation [#6510]
• [FEATURE] Implement partition compaction grouper [#6172]
• [FEATURE] Implement partition compaction planner [#6469]
• [FEATURE] Make LivenessCheck Timeout Configurable [#6227]
• [FEATURE] Querier: Add day range limit for LabelNames and LabelValues [#6233]
• [ENHANCEMENT] Query Frontend: Add more operation label values to the cortex_query_frontend_queries_total metric. [#6519]
• [ENHANCEMENT] Query Frontend: Add a source label to query stat metrics. [#6470]
• [ENHANCEMENT] Query Frontend: Add a flag -tenant-federation.max-tenant to limit the number of tenants for federated query. [#6493]
• [ENHANCEMENT] Querier: Add a -tenant-federation.max-concurrent flags to configure the number of worker processing federated query and add a cortex_querier_federated_tenants_per_query histogram to track the number of tenants per query. [#6449]
• [ENHANCEMENT] Query Frontend: Add a number of series in the query response to the query stat log. [#6423]
• [ENHANCEMENT] Store Gateway: Add a hedged request to reduce the tail latency. [#6388]
• [ENHANCEMENT] Ingester: Add metrics to track succeed/failed native histograms. [#6370]
• [ENHANCEMENT] Query Frontend/Querier: Add an experimental flag -querier.enable-promql-experimental-functions to enable experimental promQL functions. [#6355]
• [ENHANCEMENT] OTLP: Add -distributor.otlp-max-recv-msg-size flag to limit OTLP request size in bytes. [#6333]
• [ENHANCEMENT] S3 Bucket Client: Add a list objects version configs to configure list api object version. [#6280]
• [ENHANCEMENT] OpenStack Swift: Add application credential configs for Openstack swift object storage backend. [#6255]
• [ENHANCEMENT] Query Frontend: Add new query stats metrics cortex_query_samples_scanned_total and cortex_query_peak_samples to track scannedSamples and peakSample per user. [#6228]
• [ENHANCEMENT] Ingester: Add option ingester.disable-chunk-trimming to disable chunk trimming. [#6300]
• [ENHANCEMENT] Ingester: Add blocks-storage.tsdb.wal-compression-type to support zstd wal compression type. [#6232]
• [ENHANCEMENT] Query Frontend: Add info field to query response. [#6207]
• [ENHANCEMENT] Query Frontend: Add peakSample in query stats response. [#6188]
• [ENHANCEMENT] Ruler: Add new ruler metric cortex_ruler_rule_groups_in_store that is the total rule groups per tenant in store, which can be used to compare with cortex_prometheus_rule_group_rules to count the number of rule groups that are not loaded by a ruler. [#5869]
• [ENHANCEMENT] Ingester/Ring: New READONLY status on ring to be used by Ingester. New ingester API to change mode of ingester [#6163]
• [ENHANCEMENT] Ruler: Add query statistics metrics when --ruler.query-stats-enabled=true. [#6173]
• [ENHANCEMENT] Ingester: Add new API /ingester/all_user_stats which shows loaded blocks, active timeseries and ingestion rate for a specific ingester. [#6178]
• [ENHANCEMENT] Distributor: Add new cortex_reduced_resolution_histogram_samples_total metric to track the number of histogram samples which resolution was reduced. [#6182]
• [ENHANCEMENT] StoreGateway: Implement metadata API limit in queryable. [#6195]
• [ENHANCEMENT] Ingester: Add matchers to ingester LabelNames() and LabelNamesStream() RPC. [#6209]
• [ENHANCEMENT] KV: Add TLS configs to consul. [#6374]
• [ENHANCEMENT] Ingester/Store Gateway Clients: Introduce an experimental HealthCheck handler to quickly fail requests directed to unhealthy targets. [#6225] [#6257]
• [ENHANCEMENT] Upgrade build image and Go version to 1.23.2. [#6261] [#6262]
• [ENHANCEMENT] Ingester: Introduce a new experimental feature for caching expanded postings on the ingester. [#6296]
• [ENHANCEMENT] Querier/Ruler: Expose store_gateway_consistency_check_max_attempts for max retries when querying store gateway in consistency check. [#6276]
• [ENHANCEMENT] StoreGateway: Add new cortex_bucket_store_chunk_pool_inuse_bytes metric to track the usage in chunk pool. [#6310]
• [ENHANCEMENT] Distributor: Add new cortex_distributor_inflight_client_requests metric to track number of ingester client inflight requests. [#6358]
• [ENHANCEMENT] Distributor: Expose cortex_label_size_bytes native histogram metric. [#6372]
• [ENHANCEMENT] Add new option -server.grpc_server-num-stream-workers to configure the number of worker goroutines that should be used to process incoming streams. [#6386]
• [ENHANCEMENT] Distributor: Return HTTP 5XX instead of HTTP 4XX when instance limits are hit. [#6358]
• [ENHANCEMENT] Ingester: Make sure unregistered ingester joining the ring after WAL replay. [#6277]
• [ENHANCEMENT] Distributor: Add a new -distributor.num-push-workers flag to use a goroutine worker pool when sending data from distributor to ingesters. [#6406]
• [ENHANCEMENT] Ingester: If a limit per label set entry doesn't have any label, use it as the default partition to catch all series that doesn't match any other label sets entries. [#6435]
• [ENHANCEMENT] Querier: Add new cortex_querier_codec_response_size metric to track the size of the encoded query responses from queriers. [#6444]
• [ENHANCEMENT] Distributor: Added cortex_distributor_received_samples_per_labelset_total metric to calculate ingestion rate per label set. [#6443]
• [ENHANCEMENT] Added metric name in limiter per-metric exceeded errors. [#6416]
• [ENHANCEMENT] StoreGateway: Added cortex_bucket_store_indexheader_load_duration_seconds and cortex_bucket_store_indexheader_download_duration_seconds metrics for time of downloading and loading index header files. [#6445]
• [ENHANCEMENT] Add cleaner logic to clean partition compaction blocks and related files [#6507]
• [ENHANCEMENT] Add support for native histograms in querier protobuf codec [#6368]
• [ENHANCEMENT] Add timeout on lifecycler heartbeat [#6212]
• [ENHANCEMENT] Added UserReplicaGroupMetrics [#6463]
• [ENHANCEMENT] Blocks Storage: Allow use of non-dualstack endpoints for S3 blocks storage via -blocks-storage.s3.disable-dualstack. [#6522]
• [ENHANCEMENT] Discarded samples per labelset metrics for throttle by labelset [#6492]
• [ENHANCEMENT] Expanded Postings Cache can cache results without the nearly created series under high load. [#6417]
• [ENHANCEMENT] Ingester: Disable chunk trimming. [#6270]
• [ENHANCEMENT] Improve consistency check warn log [#6366]
• [ENHANCEMENT] Improve streaming on MetricsForLabelMatchersStream method [#6436]
• [ENHANCEMENT] Improve validation metrics for discarded samples and exemplars [#6218]
• [ENHANCEMENT] Query Frontend: add new field for dense native histogram format [#6199]
• [ENHANCEMENT] Return 503 on hitting distributor instance limits [#6387]
• [ENHANCEMENT] Reusing Batch Iterators [#6403]
• [ENHANCEMENT] Reusing the grpc client to peform healthcheck [#6260]
• [ENHANCEMENT] Store Gateway: Add pre add block ownership check [#6483]
• [ENHANCEMENT] Use slice pooling to populate the query stream response [#6466]
• [ENHANCEMENT] Using a single seed array for expanded postings cache on ingesters [#6365]
• [BUGFIX] Runtime-config: Handle absolute file paths when working directory is not / [#6224]
• [BUGFIX] Ruler: Allow rule evaluation to complete during shutdown. [#6326]
• [BUGFIX] Ring: update ring with new ip address when instance is lost, rejoins, but heartbeat is disabled. [#6271]
• [BUGFIX] Ingester: Fix regression on usage of cortex_ingester_queried_chunks. [#6398]
• [BUGFIX] Ingester: Fix possible race condition when active series per LabelSet is configured. [#6409]
• [BUGFIX] Query Frontend: Fix @ modifier not being applied correctly on sub queries. [#6450]
• [BUGFIX] Cortex Redis flags with multiple dots [#6476]
• [BUGFIX] Bug fix on JSON Tag [#6339]
• [BUGFIX] Calculate # of concurrency only once at the runner [#6506]
• [BUGFIX] Clean up ingester per labelset metrics [#6439]
• [BUGFIX] Cleanup dangling request queue metrics [#6433]
• [BUGFIX] Fix BenchmarkDistributor_Push benchmark [#6309]
• [BUGFIX] Fix data race on expanded postings Cache [#6369]
• [BUGFIX] Fix lazy postings merge bug [#6415]
• [BUGFIX] Fix race on chunks multilevel cache + Optimize to avoid refetching already found keys. [#6312]
• [BUGFIX] Fix race on the string interning [#6408]
• [BUGFIX] Fix race that can cause nil reference when using expanded postings [#6518]
• [BUGFIX] Fix regression of query range result cache unable to parse old cached results [#6196]
• [BUGFIX] Fix typo in usage message for querier.split-queries-by-interval flag [#6305]
• [BUGFIX] Fix: PostingCache promise should fetch data only once [#6314]
• [BUGFIX] Fix: fix slice init length [#6237]
• [BUGFIX] Fixed bug that blocks cannot be fully deleted from TSDB [#6231]
• [BUGFIX] Fixed ingester ReadOnly state related bugs [#6208]
• [BUGFIX] Preserve ingester state on restart [#6301]
• [BUGFIX] Purge expired postings cache items due inactivity [#6502]
• [BUGFIX] Util: Check context every N iterations [#6250]

New Contributors

• @afhassan made their first contribution in https://github.com/cortexproject/cortex/pull/5527
• @cuishuang made their first contribution in https://github.com/cortexproject/cortex/pull/6237
• @eduardscaueru made their first contribution in https://github.com/cortexproject/cortex/pull/6278
• @abdurrahman-osman made their first contribution in https://github.com/cortexproject/cortex/pull/6348
• @ecsumed made their first contribution in https://github.com/cortexproject/cortex/pull/6427
• @eeldaly made their first contribution in https://github.com/cortexproject/cortex/pull/6422
• @won-js made their first contribution in https://github.com/cortexproject/cortex/pull/6476
• @7h3-3mp7y-m4n made their first contribution in https://github.com/cortexproject/cortex/pull/6463
• @sam-mcbr made their first contribution in https://github.com/cortexproject/cortex/pull/6522

Full Changelog: https://github.com/cortexproject/cortex/compare/v1.18.1...v1.19.0-rc.1