Commit graph

1066 commits

Author SHA1 Message Date
Aliaksandr Valialkin
b33b620af6
app/vmselect/prometheus: do not drop match[] filters if -search.ignoreExtraFiltersAtLabelsAPI flag is set
The `match[]` filter is mandatory at /api/v1/series, so it mustn't be dropped here.

There is no sense in dropping `match[]` filter together with `extra_label` and `extra_filters[]`
at /api/v1/labels and /api/v1/label/.../values if -search.ignoreExtraFiltersAtLabelsAPI commnad-line flag is set,
since:
- the `match[]` filter triggers slow path at these APIs;
- the `extra_label` and `extra_filters[]` filters narrow down the number of matched time series,
  so they improve performance comparing to the case when only `match[]` filter is left,
  while `extra_label` and `extra_filters[]` filters are dropped.

This is a follow-up for 0b7a23a91d
2024-03-06 13:31:51 +02:00
Aliaksandr Valialkin
9ea69622a0
app/{vmselect,vlselect}/vmui: run make vmui-update vmui-logs-update after e130f29659
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5862
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5152
2024-03-01 00:50:42 +02:00
Aliaksandr Valialkin
0b7a23a91d
app/vmselect/prometheus: ignore match[] additionally to extra_filters[] and extra_label if -search.ignoreExtraFiltersAtLabelsAPI command-line flag is set
The match[] at /api/v1/labels and /api/v1/label/.../values also may lead to slow requests and
high resource usage if it matches big number of time series. So it must be igrnored if -search.ignoreExtraFiltersAtLabelsAPI
command-line flag is set.

This is a follow-up for fab02faa3f
2024-02-29 23:40:00 +02:00
Aliaksandr Valialkin
e5c69262e2
app/vmselect/promql: use unsafe.Slice instead of deprecated reflect.SliceHeader 2024-02-29 17:50:07 +02:00
Aliaksandr Valialkin
146fccc22d
app/vmselect/netstorage: usae unsafe.SliceData instead of deprecated reflect.SliceHeader 2024-02-29 17:36:28 +02:00
Hui Wang
dd7dd0b1db
metricsql: fix label_join() when dst_label is equal to one of the `… (#5886)
* metricsql: fix label_join() when `dst_label` is equal to one of the `src_label`

* Update app/vmselect/promql/transform.go

* Update docs/CHANGELOG.md

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-02-29 16:00:22 +02:00
Aliaksandr Valialkin
fab02faa3f
app/vmselect/prometheus: add -search.ignoreExtraFiltersAtLabelsAPI command-line flag for ignoring extra_filters and extra_label args at /api/v1/labels, /api/v1/label/.../values and /api/v1/series 2024-02-29 12:59:11 +02:00
Aliaksandr Valialkin
35f592a02c
app/vmselect/promql: properly handle args in count_values_over_time() function
Prevsiously they were swapped - the first arg should be the label name and the second arg should be label filters

This is a follow-up for e389b7b959e8144fdff5075bf7a5a39b2b0c6dd3

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5847
2024-02-25 01:48:18 +02:00
Aliaksandr Valialkin
6697da73e5
app: consistently use atomic.* types instead of atomic.* functions
See ea9e2b19a5
2024-02-24 02:44:24 +02:00
Aliaksandr Valialkin
f46eaf92eb
app/vmselect: add -search.maxLabelsAPIDuration and -search.maxLabelsAPISeries options for fine-tuning CPU and RAM usage for /api/v1/series , /api/v1/labels and /api/v1/label/.../values
This commit returns back limits for these endpoints, which have been removed at 5d66ee88bd ,
since it has been appeared that missing limits result in high CPU usage, while the introduced concurrency limiter
results in failed lightweight requests to these endpoints because of timeout when heavyweight requests are executed.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5055
2024-02-23 02:57:16 +02:00
Aliaksandr Valialkin
8995b04886
app/{vmselect,vlselect}/vmui: run make vmui-update vmui-logs-update after recent changes to app/vmui 2024-02-23 01:40:48 +02:00
Nikolay
07855de142
app/vmselect: change export/csv timestamp format for rfc3339 to respect milliseconds (#5853)
* app/vmselect: adds milliseconds to the csv export response for rfc3339
* milliseconds is a standard prescion for VictoriaMetrics query request responses
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5837

* app/victoria-metrics: adds tests for csv export/import
follow-up after 3541a8d0cf96dd4f8563624c4aab6816615d0756


---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-02-22 20:31:22 +01:00
Aliaksandr Valialkin
f7c3dee1c3
app/vmselect/promql: add count_values_over_time() MetricsQL function
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5847
2024-02-22 13:39:29 +02:00
Aliaksandr Valialkin
a6eacfdb11
app/vmselect/promql: move needSilenceIntervalForRollupFunc from eval.go to rollup.go
This should improve maintainability of the code related to rollup functions,
since it is located in rollup.go

While at it, properly return empty results from holt_winters(), rate_over_sum(),
sum2_over_time(), geomean_over_time() and distinct_over_time() when there are no real samples
on the selected lookbehind window. Previously the previous sample value was mistakenly
returned from these functions.
2024-02-22 13:39:28 +02:00
Dan Dascalescu
17cf031fa1
app/vmselect: simplify wording for too many samples error (#5827) 2024-02-20 16:26:38 +01:00
Aliaksandr Valialkin
53643b620a
app/vmselect/vmui: run make vmui-update after 1c9f13d6c7 2024-02-14 02:35:55 +02:00
Aliaksandr Valialkin
d6e22f2888
app/vmselect: add sum_eq_over_time, sum_gt_over_time and sum_le_over_time functions to MetricsQL
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4641
2024-02-13 23:40:07 +02:00
Aliaksandr Valialkin
aea8feee1a
docs: update -help output after 61d9df4c36
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/834
2024-02-08 14:50:49 +02:00
Aliaksandr Valialkin
61d9df4c36
app/vmselect: add ability to reset rollup result cache on startup by passing -search.resetRollupResultCacheOnStartup command-line flag
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/834
2024-02-08 14:40:40 +02:00
Aliaksandr Valialkin
0cf56c1ba5
app/vmselect/promql: properly handle precision errors in rollup functions
changes(), increases_over_time() and resets() shouldn't take into account
value changes, which may occur because of precision errors.

The maximum guaranteed precision for raw samples stored in VictoriaMetrics is 12 decimal digits.
So do not count relative changes for values if they are smaller than 1e-12 comparing to the value.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/767
2024-02-08 02:30:57 +02:00
Aliaksandr Valialkin
6a7c7ae391
app/{vmselect,vlselect}/vmui: run make vmui-update vmui-logs-update after the recent changes to app/vmui
This is a follow-up for the following commits:

- dcbdbc760e
- a81ccbd749
- 65b8002aeb
2024-02-07 01:48:46 +02:00
Aliaksandr Valialkin
0e3c532bf7
app/vmselect/netstorage: prevent from disk write IO when closing temporary files
Remove temporary file before closing it in order to signal the OS that it shouldn't
store the file contents from page cache to disk when the file is closed.

Gracefully handle the case when the file cannot be removed before being closed -
in this case remove the file after closing it. This allows working on Windows.

Also remove superflouos opening of temporary file for reading - re-use already opened file handle for writing.

This is a follow-up for 9b1e002287
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4020
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70
2024-02-01 19:12:44 +02:00
Aliaksandr Valialkin
88dc6cff70
app/vmselect: add missing whitespace into the description for -vmui.defaultTimezone command-line flag
This is a follow-up for eb6def0695
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5611
2024-02-01 14:49:51 +02:00
Aliaksandr Valialkin
db4623efc2
app/vmselect/netstorage: properly handle the case when an empty brsPool points to the end of brs.brs
This case is possible after a new brsPool is allocated. The fix is to verify whether len(brsPool) >= len(brs.brs)
before trying to append a new item to brsPool and sharing its contents with brs.brs.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5733
2024-01-31 10:27:50 +02:00
Aliaksandr Valialkin
ec0ca8e7eb
app/vmselect/promql: really keep metric names when keep_metric_names modifier is applied to binary operator
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5556
2024-01-31 02:32:55 +02:00
Roman Khavronenko
6939c53e48
app/vmselect: set proper timestamp for cached instant responses (#5723)
* app/vmselect: set proper timestamp for cached instant responses

The change updates `getSumInstantValues` to prefer timestamp
from the most recent results. Before, timestamp from cached series
was used.

The old behavior had negative impact on recording rules as they
were getting responses with shifted timestamps in past.
Subsequent recording or alerting rules fetching results of these
recording rules could get no result due to staleness interval.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5659
Signed-off-by: hagen1778 <roman@victoriametrics.com>

* wip

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-01-30 20:03:34 +00:00
Aliaksandr Valialkin
c12bdd6c28
app/vmselect/vmui: run make vmui-update after 81b5db04f6 2024-01-30 21:13:01 +02:00
Aliaksandr Valialkin
adf585f7ed
app/vmselect/vmui: run make vmui-update after 6e8995cfb92fb5a87fc6ad78609bf9ea5e0e712f 2024-01-30 18:45:57 +02:00
Aliaksandr Valialkin
5d66ee88bd
lib/storage: do not check the limit for -search.maxUniqueTimeseries when performing /api/v1/labels and /api/v1/label/.../values requests
This limit has little sense for these APIs, since:

- Thses APIs frequently result in scanning of all the time series on the given time range.
  For example, if extra_filters={datacenter="some_dc"} .

- Users expect these APIs shouldn't hit the -search.maxUniqueTimeseries limit,
  which is intended for limiting resource usage at /api/v1/query and /api/v1/query_range requests.

Also limit the concurrency for /api/v1/labels, /api/v1/label/.../values
and /api/v1/series requests in order to limit the maximum memory usage and CPU usage for these API.
This limit shouldn't affect typical use cases for these APIs:

- Grafana dashboard load when dashboard labels should be loaded
- Auto-suggestion list load when editing the query in Grafana or vmui

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5055
2024-01-29 16:45:12 +01:00
Aliaksandr Valialkin
b9dcaaa7f8
app/vmui: run make vmui-update after a7b11eff7c 2024-01-26 22:53:46 +01:00
Aliaksandr Valialkin
8e03bc6b53
app/vmselect/promql: do not spend CPU time on verifying whether the rollup cache needs to be reset for the given metric rows when it has been already instructed to reset 2024-01-26 21:13:38 +01:00
Alexander Marshalov
806c07ddd5
vmsingle/vmselect returns http status 429 (TooManyRequests) instead of 503 (ServiceUnavailable) when max concurrent requests limit is reached. (#5682) 2024-01-24 17:55:06 +01:00
Aliaksandr Valialkin
1c58c00618
app/vmselect/netstorage: limit the initial size for brsPoolCap with 32Kb
This should reduce the number of expensive memory allocations with sizes bigger than 32Kb
2024-01-23 22:29:39 +02:00
Aliaksandr Valialkin
43ecd5d258
app/vmselect/netstorage: pre-allocate memory for metricNamesBuf
This should reduce the number of metricNamesBuf re-allocations in append()
2024-01-23 21:34:16 +02:00
Aliaksandr Valialkin
41456d9569
app/vmselect/netstorage: limit the maximum brsPool size to 32Kb at ProcessSearchQuery()
This avoids slow path in Go runtime for allocating objects bigger than 32Kb -
see 704401ffa0/src/runtime/malloc.go (L11)

This also reduces memory usage a bit for vmselect and single-node VictoriaMetrics
after the commit 5dd37ad836 .

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5527
2024-01-23 14:04:49 +02:00
Aliaksandr Valialkin
1f1768d7af
app/vmselect/netstorage: limit the size of metricNamesBuf to 32Kb in order to avoid slow path at Go runtime for allocating a byte slice of bigger size
See 704401ffa0/src/runtime/malloc.go (L11)

This also reduces the average memory usage a bit for vmselect and single-node VictoriaMetrics
after the commit 508c608062

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5527
2024-01-23 13:46:37 +02:00
Aliaksandr Valialkin
15a15e5b99
app/vmselect/vmui: run make vmui-update in order to sync recent changes in app/vmui 2024-01-23 04:31:44 +02:00
Yury Molodov
eb6def0695
vmui: add flag for default timezone setting (#5611)
* vmui: add flag for default timezone setting #5375

* vmui: validate timezone before client return

* Update app/vmselect/vmui.go

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-01-23 04:11:19 +02:00
Aliaksandr Valialkin
bc7d19c8ca
app/vmselect/promql: remove superflouos memory allocations at aggrPrepareSeries()
While at it, also remove unneeded map lookup
2024-01-23 02:28:31 +02:00
Aliaksandr Valialkin
9240bc36a3
app/vmselect/promql/aggr_incremental.go: eliminate unnecessary memory allocation in incrementalAggrFuncContext.updateTimeseries 2024-01-23 02:28:30 +02:00
Aliaksandr Valialkin
e0399ec29a
app/vmselect/netstorage: remove tswPool, since it isnt efficient 2024-01-23 02:28:30 +02:00
Aliaksandr Valialkin
72a838a2a1
app/vmselect/netstorage: avoid metricName->blockRef lookup when processing multiple blocks for the same time series
This saves a few CPU cycles for common case
2024-01-23 02:28:29 +02:00
Aliaksandr Valialkin
5dd37ad836
app/vmselect/netstorage: use []blockRef from blockRefPool in order to reduce memory allocations 2024-01-23 02:28:29 +02:00
Aliaksandr Valialkin
7345567c29
app/vmselect/netstorage: substitute pointer to blockRefs by brssPool index at the metricName->blockRefs map
This should reduce the pressure on Go GC, since it will see lower number of pointers.

This change has been extracted from https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5527
2024-01-23 02:28:29 +02:00
Aliaksandr Valialkin
678234e9f0
app/vmselect/netstorage: reduce the number of allocations for blockRefs objects in ProcessSearchQuery()
This should reduce pressure on Go GC at vmselect

The change has been extracted from https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5527
2024-01-23 02:28:28 +02:00
Aliaksandr Valialkin
508c608062
app/vmselect/netstorage: reduce the number of memory allocations in ProcessSearchQuery() by storing all the metric names in a single byte slice
This reduces the number of memory allocations at the cost of possible memory usage increase,
since now different metric name strings may hold references to the previous byte slice.
This is good tradeoff, since ProcessSearchQuery is called in vmselect, and vmselect isn't usually limited by memory.

This change has been extracted from https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5527
2024-01-23 02:28:28 +02:00
Aliaksandr Valialkin
9c0863babc
docs: use persistent links to Grafana dashboards
These links do not depend on the dashboard name, so they do not break after the renaming of the dashboard.

This is a follow-up for ff33e60a3d
2024-01-22 00:17:17 +02:00
Aliaksandr Valialkin
1c7f990fad
app/vmselect: handle negative time range start in a generic manner inside NewSearchQuery()
This is a follow-up for cf03e11d89

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5553
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5630
2024-01-21 23:45:31 +02:00
Aliaksandr Valialkin
1f105dde98
all: allow dynamically reading *AuthKey flag values from files and urls
Examples:

1) -metricsAuthKey=file:///abs/path/to/file - reads flag value from the given absolute filepath
2) -metricsAuthKey=file://./relative/path/to/file - reads flag value from the given relative filepath
3) -metricsAuthKey=http://some-host/some/path?query_arg=abc - reads flag value from the given url

The flag value is automatically updated when the file contents changes.
2024-01-21 22:03:38 +02:00
Aliaksandr Valialkin
0b2ea1a7c7
all: call atomic.Load* in front of atomic.CompareAndSwap* at places where the atomic.CompareAndSwap* returns false most of the time
This allows avoiding slow inter-CPU synchornization induced by atomic.CompareAndSwap*
2024-01-21 14:04:54 +02:00