Aliaksandr Valialkin 28a9e92b5e lib/streamaggr: huge pile of changes ... - Reduce memory usage by up to 5x when de-duplicating samples across big number of time series. - Reduce memory usage by up to 5x when aggregating across big number of output time series. - Add lib/promutils.LabelsCompressor, which is going to be used by other VictoriaMetrics components for reducing memory usage for marshaled []prompbmarshal.Label. - Add `dedup_interval` option at aggregation config, which allows setting individual deduplication intervals per each aggregation. - Add `keep_metric_names` option at aggregation config, which allows keeping the original metric names in the output samples. - Add `unique_samples` output, which counts the number of unique sample values. - Add `increase_prometheus` and `total_prometheus` outputs, which ignore the first sample per each newly encountered time series. - Use 64-bit hashes instead of marshaled labels as map keys when calculating `count_series` output. This makes obsolete https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5579 - Expose various metrics, which may help debugging stream aggregation: - vm_streamaggr_dedup_state_size_bytes - the size of data structures responsible for deduplication - vm_streamaggr_dedup_state_items_count - the number of items in the deduplication data structures - vm_streamaggr_labels_compressor_size_bytes - the size of labels compressor data structures - vm_streamaggr_labels_compressor_items_count - the number of entries in the labels compressor - vm_streamaggr_flush_duration_seconds - a histogram, which shows the duration of stream aggregation flushes - vm_streamaggr_dedup_flush_duration_seconds - a histogram, which shows the duration of deduplication flushes - vm_streamaggr_flush_timeouts_total - counter for timed out stream aggregation flushes, which took longer than the configured interval - vm_streamaggr_dedup_flush_timeouts_total - counter for timed out deduplication flushes, which took longer than the configured dedup_interval - Actualize docs/stream-aggregation.md The memory usage reduction increases CPU usage during stream aggregation by up to 30%. This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5850 Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5898		2024-03-02 02:42:50 +02:00
..
duration.go	lib/promutils, app/vmalert-tool/unittest: move promutils.Duration.ParseTime() to app/vmalert-tool/unittest.durationToTime()	2023-10-16 14:19:31 +02:00
duration_test.go	lib/promscrape: support prometheus-like duration in scrape configs (#2169)	2022-02-11 16:17:00 +02:00
labels.go	all: consistently clear prompbmarshal.Label by assigning an empty struct instead of zeroing Name and Value individually	2024-01-21 05:11:05 +02:00
labels_test.go	lib/promutils: properly return error when incorrect Prometheus label names are passed to NewLabelsFromString()	2023-05-12 16:52:29 -07:00
labels_timing_test.go	lib/promscrape: optimize service discovery speed	2022-11-29 21:26:00 -08:00
labelscompressor.go	lib/streamaggr: huge pile of changes	2024-03-02 02:42:50 +02:00
labelscompressor_test.go	lib/streamaggr: huge pile of changes	2024-03-02 02:42:50 +02:00
labelscompressor_timing_test.go	lib/streamaggr: huge pile of changes	2024-03-02 02:42:50 +02:00
time.go	lib/promutils: hide the math.Round() logic inside ParseTimeMsec() function	2024-02-23 00:55:32 +02:00
time_test.go	lib/promutils: hide the math.Round() logic inside ParseTimeMsec() function	2024-02-23 00:55:32 +02:00