github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
2001 commits 150 branches 748 tags 347 MiB
Commit graph

425 commits

Author SHA1 Message Date
Aliaksandr Valialkin
285665e93b app/vmselect/promql: allow passing multiple args to aggregate functions such as avg(q1, q2, q3) 2020-08-15 01:15:16 +03:00
Aliaksandr Valialkin
6721e47ae9 app: respect CPU limits set via cgroups 
Update GOMAXPROCS to limits set via cgroups. This should reduce CPU trashing and reduce memory usage
for cases when VictoriaMetrics components run in containers with CPU limits.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685
 2020-08-11 23:01:03 +03:00
Aliaksandr Valialkin
62b6e54622 app/vmselect: reduce memory usage when exporting time series with big number of samples via /api/v1/export if max_rows_per_line is set to non-zero value 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685
 2020-08-10 20:57:43 +03:00
Aliaksandr Valialkin
c9f5c5623f app/vmselect/netstorage: vary batch size for data unpacking depending on the available CPU cores 
This should reduce contention on the channel with unpack work for systems with high number of CPU cores
 2020-08-10 15:16:48 +03:00
Aliaksandr Valialkin
95a8c492ef app/vmselect/promql: properly handle -n^m like Prometheus does 
`-n^m` must be handled as `-(n^m)` instead of `(-n)^m`.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/675
 2020-08-07 07:42:42 +03:00
Aliaksandr Valialkin
7f93d61a56 app/vmselect/promql: remove metric name after applying clamp_min and clamp_max functions in order to be consistent with Prometheus 
This improves VictoriaMetrics score at https://promlabs.com/promql-compliance-test-results-victoriametrics/
 2020-08-06 23:42:55 +03:00
Aliaksandr Valialkin
01000505a0 app/vmselect/promql: remove metric name after applying ceil, floor and round functions in order to be more consistent with Prometheus 
This improves VictoriaMetrics score at https://promlabs.com/promql-compliance-test-results-victoriametrics/
 2020-08-06 23:34:03 +03:00
Aliaksandr Valialkin
75bff1a567 app/vmselect/promql: remove metric name from results of certain rollup functions in order to be consistent with Prometheus 
Rollup functions:

  - avg_over_time
  - min_over_time
  - max_over_time
  - quantile_over_time

This improves VictoriaMetrics results at https://promlabs.com/promql-compliance-test-results-victoriametrics/
 2020-08-06 23:29:18 +03:00
Aliaksandr Valialkin
8835004a4c app/vmselect: properly handle PromQL queries like scalar1 < metric < scalar2 like Prometheus does 
This fixes some cases from https://promlabs.com/promql-compliance-test-results-victoriametrics/
 2020-08-06 23:21:14 +03:00
Aliaksandr Valialkin
14ddb8a34e app/vmselect/netstorage: reduce CPU contention when upacking time series blocks by unpacking batches of such blocks instead of a single block 
This should improve query performance on systems with big number of CPU cores (16 and more)
 2020-08-06 17:50:13 +03:00
Aliaksandr Valialkin
46c98cd97a app/vmselect/netstorage: reduce contention on unpackworkCh and timeseriesWorkCh for multi-CPU system by providing more capacity for these chans 2020-08-06 17:22:39 +03:00
Aliaksandr Valialkin
a04f4a3d9a app/vmselect: use warning level instead of info level for logging slow queries that take longer than -search.logSlowQueryDuration 2020-08-04 20:24:38 +03:00
Aliaksandr Valialkin
bdb881c43b app/vmselect/promql: add zscore-related functions: zscore_over_time(m[d]) and zscore(q) by (...) 2020-08-03 21:52:15 +03:00
Aliaksandr Valialkin
94471a1273 app: remove duplicate *-pure makefile rules 2020-07-31 20:01:30 +03:00
Aliaksandr Valialkin
a2aa3a60eb app/vmselect: show X-Forwarded-For contents on /api/v1/status/active_queries page 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/659
 2020-07-31 20:01:09 +03:00
Aliaksandr Valialkin
106e302d7a all: add mssing APP_NAME to vm*-GOARCH builds 2020-07-31 13:45:32 +03:00
Aliaksandr Valialkin
0c00fe70cf app/vmselect: do not adjust start and end query args passed to /api/v1/query_range when -search.disableCache command-line flag is set 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/563
 2020-07-30 23:14:56 +03:00
Aliaksandr Valialkin
338ee47d60 app/vmselect/promql: return non-empty value from rate_over_sum(m[d]) even if a single data point is located in the given [d] window 
Just divide the data point value by the window duration in this case.
 2020-07-29 12:37:34 +03:00
Aliaksandr Valialkin
717c554fb0 app/vmselect/promql: remove rollupFuncArg.realPrevValue handling, since the corner case in increase() is handled in another way now 
See e00cfc854d for the approach used now.
 2020-07-29 12:37:34 +03:00
Aliaksandr Valialkin
d9037b3970 app/vmselect/promql: fill gaps with 0 in rate_over_sum response when the last value before the selected time window isnt empty 2020-07-29 12:37:34 +03:00
Aliaksandr Valialkin
0f6f0d30d3 app/vmselect: show query origin (aka remote_addr or client address) on the /api/v1/status/active_queries page for every query 2020-07-28 15:14:40 +03:00
Aliaksandr Valialkin
9dccedc599 app/vmselect/promql: return empty values from group() if all the time series have no values at the given timestamp 
This aligns `group()` behaviour to Prometheus
 2020-07-28 13:41:04 +03:00
Aliaksandr Valialkin
b191e425b3 app/vmselect/promql: improve further the accuracy of buckets_limit() function 
The accuracy is increased by mergin the smallest bucket with the smallest adjacent bucket.
 2020-07-26 12:10:56 +03:00
Aliaksandr Valialkin
43871e79c6 app/vmselect/promql: avoid dropping inf bucket in buckets_limit 
The `le="inf"` bucket must be preserved in order to maintain the maximum level of accuracy.
 2020-07-25 17:00:25 +03:00
Aliaksandr Valialkin
978c1e930e app/vmselect/promql: optimize buckets_limit(k, buckets) for big number of buckets 2020-07-25 13:24:33 +03:00
Aliaksandr Valialkin
51cbf27077 app/vmselect/promql: improve the accuracy of buckets_limit(k, buckets) function 
Now it properly merges the bucket with the previous bucket after deletion.
 2020-07-24 17:07:30 +03:00
Aliaksandr Valialkin
cf69b1ea6f app/vmselect/promql: add buckets_limit(k, buckets) function, which limits the number of buckets per time series to k 
This function works with both Prometheus-style and VictoriaMetrics-style buckets.
The function removes buckets with the lowest values in order to reserve the highest precision.
The function is useful for building heatmaps in Grafana from too big number of buckets.
 2020-07-24 16:14:12 +03:00
Aliaksandr Valialkin
45334f61de app/vmselect: fix tests for rate_over_sum 2020-07-24 02:35:09 +03:00
Aliaksandr Valialkin
3526e8768a app/vmselect/promql: typo fix after 3e557c9861 2020-07-24 02:15:23 +03:00
Aliaksandr Valialkin
8d1721d128 app/vmselect/promql: add rate_over_sum(m[d]) function to MetricsQL, which returns rate over sum of m values over d duration 
Something like `sum_over_time(m[d]) / d`, but more accurate.
 2020-07-24 01:17:15 +03:00
Aliaksandr Valialkin
88e8bed0c9 app/vmselect/promql: allow setting [d] window smaller than the interval between raw points for avg_over_time 
This makes `avg_over_time` behavior consistent with `sum_over_time` and `count_over_time` behaviors.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/636
 2020-07-23 22:25:33 +03:00
Aliaksandr Valialkin
fb3d1380ac lib/storage: respect -search.maxQueryDuration when searching for time series in inverted index 
Previously the time spent on inverted index search could exceed the configured `-search.maxQueryDuration`.
This commit stops searching in inverted index on query timeout.
 2020-07-23 21:22:05 +03:00
Aliaksandr Valialkin
dbf3038637 lib/storage: add more fine-grained pace limiting for search 2020-07-23 19:21:49 +03:00
Aliaksandr Valialkin
16a4b1b20c app/vmselect/netstorage: protect from too smart compiler, which may break memory usage optimization in tmpBlocksFileWrapper.WriteBlocks 2020-07-23 17:57:24 +03:00
Aliaksandr Valialkin
55ed07add1 app/vmselect: typo fix after 0168e21fe32776e2f7f003f88e0e6e490eb2dcb0g 2020-07-23 14:11:15 +03:00
Aliaksandr Valialkin
7aa5b48508 app/vmselect: reduce memory usage when querying big number of time series with long labels 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646
 2020-07-23 13:48:58 +03:00
Aliaksandr Valialkin
20d0c41ac5 app/vmselect/prometheus: support d, w and y suffixes for durations passed to step in /api/v1/query_range like Prometheus does 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/641
 2020-07-22 16:27:27 +03:00
Aliaksandr Valialkin
bd4299fafe app/vmselect/netstorage: reduce memory allocations when unpacking time series data by using a pool for unpackWork entries 
This should slightly reduce load on GC when processing queries that touch big number of time series.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646 according to the provided memory profile
 2020-07-22 15:04:42 +03:00
Aliaksandr Valialkin
5bb4fe1ba4 app/vmselect: take into account the time spent in wait queue before query execution as time spent on the query 2020-07-21 19:00:00 +03:00
Aliaksandr Valialkin
0755cb3b50 app/vmselect/promql: skip the first value in time series passed to increase() if it exceeds by more than 10x the delta between the next value and the first value 
This should prvent from inflated `increase()` results for time series that start from big initial values.
Such cases may occur when a label value changes in a metric without counter reset.
 2020-07-21 17:24:28 +03:00
Aliaksandr Valialkin
71eba8dcf5 app/vmselect: log the total available memory for concurrent requests on not enough memory errors 
This should simplify root cause analysis
 2020-07-20 19:51:58 +03:00
Aliaksandr Valialkin
141e84b5a4 app/vmselect/prometheus: do not return time series with empty list of datapoints from /api/v1/query_range 
This matches Prometheus behaviour.

This should fix https://github.com/jacksontj/promxy/issues/329
 2020-07-20 15:30:13 +03:00
Aliaksandr Valialkin
4d2011a87d app/vmselect/promql: add mode() aggregate function 2020-07-20 15:30:11 +03:00
Aliaksandr Valialkin
31ef39e8da lib/httpserver: log remote address in error message from httpserver.Errorf 
This should improve detection of the root cause of errors.
Thanks to Anant for the idea.
 2020-07-20 14:06:29 +03:00
Aliaksandr Valialkin
427fa43ce2 app/vmselect/promql: add mode_over_time(m[d]) function 
See https://en.wikipedia.org/wiki/Mode_(statistics) and https://stackoverflow.com/questions/61134078/promql-query-to-return-the-value-from-a-range-vector-which-occurs-maximum-no-of
 2020-07-17 18:29:10 +03:00
Aliaksandr Valialkin
eb402a17bd app/vmselect/promql: optimize group(rollup(m)) calculations 2020-07-17 16:47:30 +03:00
Aliaksandr Valialkin
ea8dc85ba8 app/vmselect/promql: check that any() doesn't touch metric name 2020-07-17 16:23:11 +03:00
Aliaksandr Valialkin
fc8fe38a82 app/vmselect/promql: add group() aggregate function to MetricsQL 
This function has been added in Prometheus 2.20. See https://github.com/prometheus/prometheus/pull/7480
 2020-07-17 15:17:38 +03:00
Aliaksandr Valialkin
c64914a7e4 app/vmselect/promql: keep all labels for time series from any() call 2020-07-17 15:17:37 +03:00
Aliaksandr Valialkin
14dc426b45 app/vmselect: fix nil pointer dereference panic when unsuccessfully querying vmstorage 2020-07-16 19:15:18 +03:00
Aliaksandr Valialkin
0e7b2008b2 app/vmselect/prometheus: do not adjust last points in time series with timestamps exceeding the current time 
Such timestamps usually mean that the query contains `offset`.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/625
 2020-07-14 12:56:21 +03:00
Aliaksandr Valialkin
3898cc0285 app/vmselect/prometheus: minimize the diff for the change 1033dc7e2a over 619b0a25c9 2020-07-13 21:41:17 +03:00
faceair
bf39e67ade fix empty response template (#617) 2020-07-13 21:41:15 +03:00
Aliaksandr Valialkin
d449d0a0e1 app/vmselect/promql: add missing tests for ifnot binary operation 2020-07-09 13:24:12 +03:00
Aliaksandr Valialkin
7e706eea13 app/vmselect/promql: refactor implementations for and and unless binary operations, so they are closer to or implementation 2020-07-09 13:06:01 +03:00
Aliaksandr Valialkin
6c1a47b5e0 app/vmselect/promql/active_queries.go: simplify code a bit by inlining getNextActiveQueryID function 2020-07-09 11:18:53 +03:00
Aliaksandr Valialkin
fb86071552 app/vmselect: add /api/v1/status/active_queries page with the list of currently running queries 
This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/575

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/528
 2020-07-08 19:09:31 +03:00
DexterZhang
9930ce1fa9
Feat/query list vmselect (#575) 
* feat(vmselect): add support for listing current running queries and canceling specific query

* fix(vmselect): change current queries' pid from int64 counter to uuid

* feat(vmselect): add auth to internal operations like `/resetRollupResultCache`, `/query/list` and `/query/kill`. add flag `internalAuthKey` for these auth

* fix(vmselect): add more info to current queries

* review: delete some unnecessary code and use function instead of init

* review: returen *queriesMap in newQueriesMap

* review: delete unused var in struct queriesMap, add comments to exported functions

* review: add return if error occurs

* feat(vmselect): truncate query string in current running query list API since the size of query string might be large;
                use query string's pointer in struct `query` for the same reason;
		add query info API to get full access of query's info;
 2020-07-08 19:04:29 +03:00
Aliaksandr Valialkin
82871fb7a5 app/vmselect/prometheus: small fixes on top of 8bb762124a 2020-07-05 18:17:53 +03:00
faceair
17f175ff5a fix adjust last points avoid influence earlier value (#606) 2020-07-05 18:17:52 +03:00
Aliaksandr Valialkin
f10e8809c0 app/vmselect: add interpolate function for filling gaps with linearly interpolated values 
See https://stackoverflow.com/q/62565021/274937 for details
 2020-07-02 14:54:46 +03:00
Aliaksandr Valialkin
4cb3e7595c app/vmstorage: add -denyQueriesOutsideRetention command-line flag for denying queries outside the configured retention 2020-07-01 00:58:42 +03:00
Aliaksandr Valialkin
0c4e8aeb2b all: use errors.As for inspecting errors that implement httpserver.ErrorWithStatusCode 2020-07-01 00:03:11 +03:00
Aliaksandr Valialkin
d962568e93 all: use %w instead of %s for wrapping errors in fmt.Errorf 
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
 2020-06-30 23:33:46 +03:00
Aliaksandr Valialkin
12b87b2088 app/vmselect/netstorage: reset big result values every 10 seconds instead of after processing every time series 
This should reduce GC pressure when processing time series with big number of rows
 2020-06-24 19:37:35 +03:00
nicbaz
46c5c0772c vmselect: fix label_replace when mismatch (#579) 
As per documentation on `label_replace` function: "If the regular
expression doesn't match then the timeseries is returned unchanged".

Currently this behavior is not enforced, if a regexp on an existing
tag doesn't match then the tag value is copied as-is in the destination
tag. This fix first checks that the regular expression matches the
source tag before applying anything.

Given the current implementation, this fix also changes the behavior
of the **MetricsQL** `label_transform` function which does not
document this behavior at the moment.
 2020-06-23 23:54:29 +03:00
Aliaksandr Valialkin
0fdbe5de25 app/vmselect/netstorage: increase concurrency when processing small number of time series with big number of data points per each time series 
Previously VictoriaMetrics was processing up to 32 time series in a single goroutine.
This could be slow if each time series contains big number of data points (10M+ or more), since only a single CPU core could be loaded with work,
while other CPU cores were idle. Fix this by launching GOMAXPROCS workers for time series processing.

This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/572
 2020-06-23 22:45:57 +03:00
Aliaksandr Valialkin
70bf8218bb app/vmselect/promql: properly override label values from group_left and group_right lists like Prometheus does 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/577
 2020-06-21 16:32:27 +03:00
Aliaksandr Valialkin
6939e36fdd app/vmselect/promql: fill gaps on right side with values from left side of or operator in the same way as Prometheus does 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/552
 2020-06-18 23:05:23 +03:00
Aliaksandr Valialkin
cc2225cc49 app/vmselect: fix the error after 936f35920a 2020-06-12 22:00:45 +03:00
Aliaksandr Valialkin
936f35920a app/vmselect/prometheus: allow returning partial response from /api/v1/export if -search.denyPartialResponse=false 
This makes `/api/v1/export` behaviour consistent with other `/api/v1/*` handlers.
 2020-06-12 21:11:48 +03:00
Aliaksandr Valialkin
3d0a0b3785 lib/fs: optimize MustGetFreeSpace performance by caching the results for up to 2 seconds 2020-06-04 13:14:04 +03:00
DexterZhang
fa103875a0
feat(vmselect): add tmp block dir size metrics vm_tmp_blocks_files_size_total (#527) 
* feat(vmselect): add tmp block dir size metrics `vm_tmp_blocks_files_size_total`

* refactor(vmselect): use free space instead of used space in tmp block file metrics

* fix: add `bytes` suffix to tmp dir free space metric
 2020-06-04 13:05:50 +03:00
Aliaksandr Valialkin
d83c68ca03 app/vmselect/promql: add ascent_over_time(m[d]) and descent_over_time(m[d]) functions 
These functions could be useful in GPS tracking apps for calculating the summary for height gain/loss
over the given duration `d`.
 2020-05-21 12:06:34 +03:00
Aliaksandr Valialkin
8ff28f5b91 app/vmselect/promql: update numbers after the upgrade of github.com/VictoriaMetrics/metrics from v1.11.2 to v1.11.3 2020-05-20 03:07:07 +03:00
Aliaksandr Valialkin
7d46dd452a app/vmselect/promql: move common code from aggrFuncOutliersK and newAggrFuncRangeTopK into getRangeTopKTimeseries 2020-05-19 16:11:03 +03:00
Aliaksandr Valialkin
37068064dd app/vmselect/promql: fix outilersk calculations 2020-05-19 14:45:10 +03:00
Aliaksandr Valialkin
fc81ea38d4 app/vmselect/promql: add outliersk(N, m) aggregate function for anomaly detection across groups of similar time series 2020-05-19 13:52:44 +03:00
Aliaksandr Valialkin
2784015a4d all: print --help output to stdout instead of stderr 
This is easier to grep and pipe
 2020-05-16 12:03:06 +03:00
Aliaksandr Valialkin
3845420a8f lib: extract common code for returning fast unix timestamp into lib/fasttime 2020-05-14 23:06:50 +03:00
Aliaksandr Valialkin
f3d9a5b0ec app/vmselect/promql: suppress "SA4006: this value of dstValues is never used" error in golangci-lint 2020-05-13 11:46:05 +03:00
Aliaksandr Valialkin
18a0caee43 app/vmselect/promql: fix any(..) calculations - return all the data points instead of the first one 2020-05-12 20:36:49 +03:00
Aliaksandr Valialkin
81b8811cf4 app/vmselect/promql: remove -search.maxPointsPerTimeseries command-line flag 
Limit the estimated time series count after aggregation with grouping by the number of source time series.
 2020-05-12 19:54:44 +03:00
Aliaksandr Valialkin
408ade27a9 app/vmselect/promql: add any(x) by (y) aggregate function, which returns any time series from q for each group y 2020-05-12 19:50:29 +03:00
Aliaksandr Valialkin
21c2982ac8 app/vmselect/promql: support for sum(x) by (y) limit N syntax in order to limit the number of output time series after aggregation 2020-05-12 19:50:12 +03:00
Aliaksandr Valialkin
f341c6fcc4 Revert "app/vmselect: add -search.estimatedSeriesCountAfterAggregation command-line flag for tuning the probability of OOMs or false-positive not enough memory errors" 
This reverts commit fbb7986dd2380fce2fc8633b7eda8b67f419e74c.

Reason for revert: this commit has been removed from single-node version
 2020-05-12 19:50:08 +03:00
Aliaksandr Valialkin
9ed4951ec8 lib/metricsql: move it to a separate repository - github.com/VictoriaMetrics/metrics 2020-04-28 15:30:06 +03:00
Aliaksandr Valialkin
cd1145e5f4 app/vmselect: add -search.estimatedSeriesCountAfterAggregation command-line flag for tuning the probability of OOMs or false-positive not enough memory errors 2020-04-28 12:51:48 +03:00
Aliaksandr Valialkin
e933cbac16 lib/storage: postpone reading data from blocks during search 
This eliminates the need for storing block data into temporary files on a single-node VictoriaMetrics
during heavy queries, which touch big number of time series over long time ranges.

This improves single-node VM performance on heavy queries by up to 2x.
 2020-04-27 08:44:01 +03:00
Aliaksandr Valialkin
23a310cc68 app/vmselect/netstorage: substitute sorting packedTimeseries with the natural order of the fetched blocks 
This should minimize the number of disk seeks when reading data from temporary file.
 2020-04-26 16:46:17 +03:00
Aliaksandr Valialkin
32b3f959fc app/vmselect: fix description for -search.resetCacheAuthKey 2020-04-24 19:44:35 +03:00
Aliaksandr Valialkin
f9526809e5 app/vmselect: add /api/v1/status/tsdb page with useful stats for locating root cause for high cardinality issues 
See https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/425
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/268
 2020-04-22 22:03:23 +03:00
Aliaksandr Valialkin
b59f1f1504 app/vmselect: add -search.minStalenessInterval command-line flag for removing gaps on graphs built from time series with irregular duration between samples 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/426
 2020-04-20 19:42:41 +03:00
Aliaksandr Valialkin
603d4c9217 app/vmselect: merge -search.maxLookback and -search.maxStalenessInterval flags, since it has been appeared they have identical purpose :( 
Leave both flags for backwards compatibility reasons.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/426
 2020-04-20 19:28:28 +03:00
Aliaksandr Valialkin
b9b5641c2f app/vmselect: properly apply -search.maxLookback to queries sent to /api/v1/query 2020-04-17 12:31:18 +03:00
Aliaksandr Valialkin
d4bc60d63c lib/logger: add WARN level for logging expected errors such as invalid user queries 2020-04-15 20:50:45 +03:00
Aliaksandr Valialkin
a873b553cf app/vmselect: handle timestamp(metric offset X) the same way as Prometheus does 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/415
 2020-04-15 12:01:05 +03:00
Aliaksandr Valialkin
a012f6fe70 app/vmselect/promql: keep metric name after applying first_over_time and last_over_time functions 2020-04-04 14:54:02 +03:00
Aliaksandr Valialkin
a53e332a93 app/vmstorage: add missing shutdown for http server on graceful shutdown 
This could result in the following panic during graceful shutdown when `/metrics` page is requested:

http: panic serving 10.101.66.5:57366: runtime error: invalid memory address or nil pointer dereference
goroutine 2050 [running]:
net/http.(*conn).serve.func1(0xc00ef22000)
	net/http/server.go:1772 +0x139
panic(0xa0fc00, 0xe91d80)
	runtime/panic.go:973 +0x3e3
github.com/VictoriaMetrics/VictoriaMetrics/lib/workingsetcache.(*Cache).UpdateStats(0x0, 0xc0000516c8)
	github.com/VictoriaMetrics/VictoriaMetrics/lib/workingsetcache/cache.go:224 +0x37
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*indexDB).UpdateMetrics(0xc00b931d00, 0xc02c41acf8)
	github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/index_db.go:258 +0x9f
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*Storage).UpdateMetrics(0xc0000bc7e0, 0xc02c41ac00)
	github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/storage.go:413 +0x4c5
main.registerStorageMetrics.func1(0x0)
	github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage/main.go:186 +0xd9
main.registerStorageMetrics.func3(0xc00008c380)
	github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage/main.go:196 +0x26
main.registerStorageMetrics.func7(0xc)
	github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage/main.go:211 +0x26
github.com/VictoriaMetrics/metrics.(*Gauge).marshalTo(0xc000010148, 0xaa407d, 0x20, 0xb50d60, 0xc005319890)
	github.com/VictoriaMetrics/metrics@v1.11.2/gauge.go:38 +0x3f
github.com/VictoriaMetrics/metrics.(*Set).WritePrometheus(0xc000084300, 0x7fd56809c940, 0xc005319860)
	github.com/VictoriaMetrics/metrics@v1.11.2/set.go:51 +0x1e1
github.com/VictoriaMetrics/metrics.WritePrometheus(0x7fd56809c940, 0xc005319860, 0xa16f01)
	github.com/VictoriaMetrics/metrics@v1.11.2/metrics.go:42 +0x41
github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.writePrometheusMetrics(0x7fd56809c940, 0xc005319860)
	github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver/metrics.go:16 +0x44
github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.handlerWrapper(0xb5a120, 0xc005319860, 0xc005018f00, 0xc00002cc90)
	github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver/httpserver.go:154 +0x58d
github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.gzipHandler.func1(0xb5a120, 0xc005319860, 0xc005018f00)
	github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver/httpserver.go:119 +0x8e
net/http.HandlerFunc.ServeHTTP(0xc00002d110, 0xb5a660, 0xc0044141c0, 0xc005018f00)
	net/http/server.go:2012 +0x44
net/http.serverHandler.ServeHTTP(0xc004414000, 0xb5a660, 0xc0044141c0, 0xc005018f00)
	net/http/server.go:2807 +0xa3
net/http.(*conn).serve(0xc00ef22000, 0xb5bf60, 0xc010532080)
	net/http/server.go:1895 +0x86c
created by net/http.(*Server).Serve
	net/http/server.go:2933 +0x35c
 2020-04-02 21:09:55 +03:00
Aliaksandr Valialkin
c6cbc0bd19 app/vmselect/prometheus: allow passing relative time to start, end and time args of /api/v1/* queries 2020-03-29 21:56:52 +03:00
Aliaksandr Valialkin
cb8696699a app/vmselect/prometheus: code simplification: (d.Seconds()/1e3) -> d.Milliseconds() 2020-03-29 21:50:35 +03:00
Dmitry Naumov
b84071fc25
Rootless docker images by default (#358) 
* Rootless docker images by default

* Migrate to rootless base image

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2020-03-27 21:18:32 +02:00
Aliaksandr Valialkin
58cb7fc476 app/vmselect: adjust label_map() handling for corner cases 
The following corner cases now supported:
* label_map(q, "label", "", "foo") - adds `label="foo"` to series with missing `label`
* label_map(q, "label", "foo", "") - removes `label="foo"` from series

All the unmatched labels are kept unchanged.
 2020-03-13 18:41:52 +02:00
Aliaksandr Valialkin
0e7a71a245 app/vmselect: add label_map(q, label, srcValue1, dstValue1, ... srcValueN, dstValueN) function to MetricsQL 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/369
 2020-03-12 19:13:56 +02:00
Aliaksandr Valialkin
50555d89d3 app/vmselect: add -search.maxStalenessInterval for tuning Prometheus data model closer to Influx-style data model 2020-03-11 16:44:03 +02:00
Aliaksandr Valialkin
f6410ff2bf app/vmselect: add optional max_rows_per_line query arg to /api/v1/export 
This arg allows limiting the number of data points that may be exported on a single line.
 2020-03-10 21:47:43 +02:00
Aliaksandr Valialkin
0d893eff36 Makefile: add build and test rules with enabled race detector. These rules have -race suffix 
Fix also `unsafe pointer conversion` errors detected by Go1.14. See https://golang.org/doc/go1.14#compiler .
 2020-03-05 12:05:16 +02:00
Aliaksandr Valialkin
d23df53ba2 app/vmselect/prometheus: do not add __name__!= filter when searching for all the matching metric names via /api/v1/label/__name__/values with non-empty label filter 
This should reduce query time.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/343
 2020-02-28 23:36:38 +02:00
Aliaksandr Valialkin
8fa1cd24d8 app/vmselect/prometheus: properly pass filter for labelName=__name__ in labelValuesWithMatches 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/343
 2020-02-28 12:17:30 +02:00
Aliaksandr Valialkin
cf9aee4ec3 all: properly split vm_deduplicated_samples_total among cluster components 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/345
 2020-02-27 23:47:51 +02:00
Aliaksandr Valialkin
6ca1e58d98 app/vmselect/promql: properly take into account the first datapoint when calculating rollup_candlestick 2020-02-24 13:25:07 +02:00
Aliaksandr Valialkin
b58e3fc8a9 app/vmselect/promql: do not take into account values outside the current window in rollup_candlestick 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
 2020-02-23 18:06:26 +02:00
Yaroslav
c69d4b01f0 fix rollupOpen(), rollupHigh(), rollupLow() functions (#328) 2020-02-23 18:06:25 +02:00
Aliaksandr Valialkin
f22aefdb16 app/vmselect/promql: log when rollupResult cache is cleared 2020-02-21 20:06:53 +02:00
Aliaksandr Valialkin
d5c2a0ce64 app/vmselect: add -search.cacheTimestampOffset command-line flag 
This flag can be used for removing gaps on graphs if the difference between the current time
and the timestamps from the ingested data exceeds 5 minutes.

This is the case when the time between data sources and VictoriaMetrics is improperly synchronized.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/312
 2020-02-21 14:02:15 +02:00
Aliaksandr Valialkin
c70822db50 app/vmselect: add /internl/resetRollupResultCache handler for resetting response cache 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/312
 2020-02-21 14:02:12 +02:00
Aliaksandr Valialkin
846d7fa7e9 app/vmselect: add sort_by_label(q, label) and sort_by_label_desc(q, label) functions 
This is implementation of https://github.com/prometheus/prometheus/pull/1533 for VictoriaMetrics.
 2020-02-13 17:01:50 +02:00
Aliaksandr Valialkin
6e0013ca39
app/vmselect/prometheus: typo fix in -latencyOffset description 2020-02-12 14:00:38 +02:00
Edouard Hur
e8f92a4ee8
Cluster - prometheus metrics fix (#314) 
* add missing '/{}' in prom query range requests

* fix missing leading '/' on prom lavelValuesErrors path
 2020-02-10 22:15:21 +02:00
Aliaksandr Valialkin
1010a57882 all: allow setting flags via environment vars 
Now flags can be set via environment vars with the same names as flags.
Command-line flags override flags set via env vars.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/311
 2020-02-10 13:31:21 +02:00
Aliaksandr Valialkin
ea66212c93 lib/storage: move -dedup.minScrapeInterval flag outside lib/storage, so it doesnt show up in vminsert in cluster version 2020-02-10 13:07:25 +02:00
Aliaksandr Valialkin
e6d9ea3094 app/vmselect/promql: do not add step to range end, since this hack became obsolete since commit 9e1119dab8 2020-02-05 21:23:44 +02:00
Aliaksandr Valialkin
4a1de7fee9 app/vmselect/promql: properly adjust time range for data to select 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
 2020-02-05 21:23:43 +02:00
Aliaksandr Valialkin
8e77b54846 app/vmselect: unconditionally offset -step to rollup_candlestick. This makes results more consistent 2020-02-04 23:31:47 +02:00
Aliaksandr Valialkin
ce38b176bc app/vmselect/promql: automatically apply offset -step to rollup_candlestick function in order to obtain the expected OHLC results 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
 2020-02-04 23:24:04 +02:00
Aliaksandr Valialkin
4f7116d1ee app/vmselect/promql: adjust rollup_candlestick calculations to the exepcted results 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
 2020-02-04 22:43:28 +02:00
Aliaksandr Valialkin
ccd3aa4f15 app/vmselect: take into account the time the requests wait in the queue if -search.maxConcurrentRequests is exceeded 
This will prevent from excess CPU usage for timed out queries.
 2020-02-04 16:20:48 +02:00
Aliaksandr Valialkin
e6bf88a4d4 app/vmselect: add a placeholder for /api/v1/metadata, which could be requested by Grafana 
See https://prometheus.io/docs/prometheus/latest/querying/api/#querying-metric-metadata

VictoriaMetrics doesn't collect any metadata for metrics, so just return empty response.
 2020-02-04 15:56:01 +02:00
Aliaksandr Valialkin
7cde594696 all: do not clash flag description with back-quoted flag types 
See https://golang.org/pkg/flag/#PrintDefaults for more details.
 2020-02-04 15:56:01 +02:00
Aliaksandr Valialkin
45bc6c62f2 app/vmselect/promql: adjust and and unless binary operator handling to be consistent with Prometheus 2020-01-31 18:52:51 +02:00
Aliaksandr Valialkin
e3adc095bd all: add -dedup.minScrapeInterval command-line flag for data de-duplication 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/86
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/278
 2020-01-31 01:18:54 +02:00
Aliaksandr Valialkin
cb5c39ee70 lib/fs: optimize small reads for ReaderAt.MustReadAt by reading from memory-mapped space instead of reading from file descriptor 
This should improve performance when reading many small blocks.
 2020-01-30 15:16:16 +02:00
Aliaksandr Valialkin
170c1c3a4e app/vmselect/promql: add keep_next_value(q) for filling gaps with the next non-empty value 2020-01-29 00:48:14 +02:00
Aliaksandr Valialkin
b28c9a3944 app/vmselect/promql: return expected results from increase() over the beginning of time series, which start from big value 
Examples for such counters: OS-level counters for network or cpu stats.
 2020-01-28 16:31:05 +02:00
Aliaksandr Valialkin
3e304890a6 app/vmselect/promql: fix panic on a single zero vmrange bucket in prometheus_buckets() function 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/296
 2020-01-27 18:05:12 +02:00
Aliaksandr Valialkin
ea53a21b02 all: consistently log durations in seconds with millisecond precision 
This should improve logs readability
 2020-01-22 18:35:24 +02:00
Aliaksandr Valialkin
e1a264173a app/vmselect: mention the original query and time range in error messages 
This should simplify debugging invalid or heavy queries.
 2020-01-22 17:34:35 +02:00
Aliaksandr Valialkin
e127173984 app/vmselect: mention command-line flag, which could be used for adjusting query timeouts, in timeout errors 2020-01-22 15:53:42 +02:00
Aliaksandr Valialkin
f3b9f8b823 app/vmselect/prometheus: increase default value -maxExportDuration to 30 days, since 10 minutes beat users exporting bit amounts of data 2020-01-22 15:53:41 +02:00
Aliaksandr Valialkin
40e564eb9c app/vmselect/promql: add range_over_time(m[d]) function for calculating value range for m over d 2020-01-21 19:05:29 +02:00
Aliaksandr Valialkin
9eaa2ab871 app/vmselect/promql: add label_match(q, label, regexp) and label_mismatch(q, label, regexp) functions for filtering out time series with labels matching the given regexp 2020-01-21 15:00:35 +02:00
Aliaksandr Valialkin
54db08a60f app/vmselect/netstorage: make fmt 2020-01-17 17:46:20 +02:00
Aliaksandr Valialkin
d21cc2d16a app/vmselect/netstorage: limit the maximum size for in-memory buffer for temporary blocks file 
This should reduce memory usage on systems with more than 8GB RAM.
 2020-01-17 16:28:28 +02:00
Aliaksandr Valialkin
b05f6cf11c app/vmselect: limit the default value for -search.maxConcurrentRequests, so it plays well on systems with more than 16 vCPUs 
A single heavy request can saturate all the available CPUs, so let's limit the number of concurrent requests to lower value.
This will give more chances for executing insert path.
 2020-01-17 16:16:43 +02:00
Aliaksandr Valialkin
a9f683423c app/{vminsert,vmselect}: improve error messages when VictoriaMetrics cannot handle too high number of concurrent inserts / selects 2020-01-17 16:16:43 +02:00
Aliaksandr Valialkin
4b16b7fd11 all: mention command-line flags used for limiting the incoming request size in error messages 
This should improve error logs usability.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/287
 2020-01-16 13:06:43 +02:00
Aliaksandr Valialkin
ce0b602405 app/vmselect/promql: fix panic on sum(aggr_over_time(...)) with incorrect number of args 2020-01-15 16:26:16 +02:00
Aliaksandr Valialkin
bcd3f0c5bd app/vmselect/promql: add hoeffding_bound_upper(phi, m[d]) and hoeffding_bound_lower(phi, m[d]) functions 
These functions can be used for calculating Hoeffding bounds
for `m` over `d` time range and for the given `phi` in the range `[0..1]`.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/283
 2020-01-11 14:47:13 +02:00
Aliaksandr Valialkin
fc01b11ddc app/vmselect/promql: return continuous values for min_over_time and max_over_time when step is smaller than scrape_interval 2020-01-11 12:47:57 +02:00
Aliaksandr Valialkin
16fb128bbc app/vmselect/promql: do not take into account the previous point before time window in square brackets for min_over_time, max_over_time, rollup_first and rollup_last functions 
This makes the behaviour for these functions similar to Prometheus when processing broken time series with irregular data points
like `gitlab_runner_jobs`. See https://gitlab.com/gitlab-org/gitlab-exporter/issues/50 for details.
 2020-01-11 00:26:11 +02:00