VictoriaMetrics/app/vmselect/promql
Zakhar Bessarab 6a4bd5049b
app/vmselect/promql: propagate lower bucket values when fixing a histogram (#6547)
### Describe Your Changes

In most cases histograms are exposed in sorted manner with lower buckets
being first. This means that during scraping buckets with lower bounds
have higher chance of being updated earlier than upper ones.

Previously, values were propagated from upper to lower bounds, which
means that in most cases that would produce results higher than expected
once all buckets will become updated.
Propagating from upper bound effectively limits highest value of
histogram to the value of previous scrape. Once the data will become
consistent in the subsequent evaluation this causes spikes in the
result.

Changing propagation to be from lower to higher buckets reduces value
spikes in most cases due to nature of the original inconsistency.

 See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4580

An example histogram with previous(red) and updated(blue) versions:

![1719565540](https://github.com/VictoriaMetrics/VictoriaMetrics/assets/1367798/605c5e60-6abe-45b5-89b2-d470b60127b8)

This also makes logic of filling nan values with lower buckets values: [1 2 3 nan nan nan] => [1 2 3 3 3 3] obsolete.
Since buckets are now fixed from lower ones to upper this happens in the main loop, so there is no need in a second one.

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Andrii Chubatiuk <andrew.chubatiuk@gmail.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-07-10 15:15:29 +02:00
..
active_queries.go app: consistently use atomic.* types instead of atomic.* functions 2024-02-24 02:44:24 +02:00
aggr.go all: fix golangci-lint(revive) warnings after 0c0ed61ce7 2024-04-02 23:16:29 +03:00
aggr_incremental.go app/vmselect/promql/aggr_incremental.go: eliminate unnecessary memory allocation in incrementalAggrFuncContext.updateTimeseries 2024-01-23 02:28:30 +02:00
aggr_incremental_test.go app/vmselect/promql: fix TestIncrementalAggr test on systems less than 3 CPU cores 2023-03-20 20:37:18 -07:00
aggr_test.go app/vmselect/promql: simplify the code after 388d020b7c 2024-01-16 15:06:33 +02:00
binary_op.go app/vmselect: use strings.EqualFold instead of strings.ToLower where appropriate 2024-05-12 10:20:41 +02:00
eval.go all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:20:37 +02:00
eval_test.go app: consistently use atomic.* types instead of atomic.* functions 2024-02-24 02:44:24 +02:00
exec.go app/vmselect/promql: follow-up for dd0d2c77c8 and 6149adbe10 2024-07-03 00:47:10 +02:00
exec_test.go app/vmselect/promql: propagate lower bucket values when fixing a histogram (#6547) 2024-07-10 15:15:29 +02:00
memory_limiter.go app/vmselect: return back the logic for limits the amounts of memory occupied by concurrently executed queries if -search.maxMemoryPerQuery isn't set 2022-10-10 21:45:13 +03:00
memory_limiter_test.go app/vmselect: return back the logic for limits the amounts of memory occupied by concurrently executed queries if -search.maxMemoryPerQuery isn't set 2022-10-10 21:45:13 +03:00
parser.go all: add support for or filters in series selectors 2023-07-16 00:06:33 -07:00
rollup.go all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:20:37 +02:00
rollup_result_cache.go lib/slicesutil: add helper functions for setting slice length and extending its capacity 2024-05-12 11:32:17 +02:00
rollup_result_cache_test.go all: fix golangci-lint(revive) warnings after 0c0ed61ce7 2024-04-02 23:16:29 +03:00
rollup_test.go all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:20:37 +02:00
rollup_timing_test.go app/vmselect: use consistent randomizer in tests 2023-01-23 19:27:25 -08:00
timeseries.go lib/slicesutil: add helper functions for setting slice length and extending its capacity 2024-05-12 11:32:17 +02:00
timeseries_test.go app/vmselect/promql: use unsafe.Slice instead of deprecated reflect.SliceHeader 2024-02-29 17:50:07 +02:00
transform.go app/vmselect/promql: propagate lower bucket values when fixing a histogram (#6547) 2024-07-10 15:15:29 +02:00
transform_test.go app/vmselect/promql: propagate lower bucket values when fixing a histogram (#6547) 2024-07-10 15:15:29 +02:00
tzdata.go all: cleanup: remove // +build ... lines, since they are no longer needed after Go1.17, and the minimum supported Go version for VictoriaMetrics source code is Go1.20 2023-11-13 19:12:51 +01:00