github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
3132 commits 150 branches 748 tags 347 MiB
Commit graph

394 commits

Author SHA1 Message Date
Aliaksandr Valialkin
3389f1e474 docs/CHANGELOG.md: document d05cac6c98 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1486
 2021-07-30 12:26:34 +03:00
Aliaksandr Valialkin
99004a6a40 app/vmselect/netstorage: unpack time series data in mostly local big chunks 
This should improve performance on multi-CPU systems for queries selecting time series with big number of raw samples
 2021-07-30 12:26:33 +03:00
Aliaksandr Valialkin
b709fa387a app/vmselect: follow-up for ed95bc9531 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1493
 2021-07-29 09:48:47 +03:00
Aliaksandr Valialkin
49bf3abf67 app/vmselect: follow-up for 626073bca8 
* Rename -search.maxMetricsPointSearch to -search.maxSamplesPerQuery, so it is more consistent with the existing -search.maxSamplesPerSeries
* Move the -search.maxSamplesPerQuery from vmstorage to vmselect, so it could effectively limit the number of raw samples obtained from all the vmstorage nodes
* Document the -search.maxSamplesPerQuery in docs/CHANGELOG.md
 2021-07-28 18:00:04 +03:00
Aliaksandr Valialkin
857fa4e28a docs/CHANGELOG.md: document 9d45b46f4c 2021-07-27 12:39:10 +03:00
Aliaksandr Valialkin
3921d8afae app/vmselect: prevent from possible deadlock when f callback blocks inside RunParallel 2021-07-26 15:50:45 +03:00
Aliaksandr Valialkin
b047feeb8b app/vmselect/promql: properly handle (a op b) default N if (a op b) returns NaN series 
The result should be a series with `N` values and `a op b` labels. Previously such series has been removed from the result.
 2021-07-16 01:44:24 +03:00
Aliaksandr Valialkin
5830ce2706 app/vmselect/netstorage: add -search.maxSamplesPerSeries command-line option for limiting the number of samples a query can process per each series 
This should prevent from out of memory crashes like in https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1067
 2021-07-15 16:53:35 +03:00
Aliaksandr Valialkin
f847677513 docs/CHANGELOG.md: typo fix: suffxies->suffixes 2021-07-15 15:02:55 +03:00
Aliaksandr Valialkin
4ec57102e6 docs/CHANGELOG.md: cut v1.63.0 2021-07-15 14:02:45 +03:00
Aliaksandr Valialkin
e992754e79 lib/storage: remove cache directory if it contains reset_cache_on_startup file 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1447
 2021-07-13 17:59:51 +03:00
Aliaksandr Valialkin
e6edb85fa2 lib/httpserver: add is_set label to flag metrics 
This label allows determining the set flags with the query `flag{is_set="true"}`
 2021-07-13 15:10:18 +03:00
Aliaksandr Valialkin
9add9d86a6 app/vmselect/promql: duration handling improvements in MetricsQL queries 
- Support durations anywhere in MetricsQL queries. E.g. sum_over_time(m[1h])/1h is equivalent to sum_over_time(m[1h])/3600
- Support durations without suffix. E.g. rate(m[300]) is equivalent to rate(m[5m])
 2021-07-12 17:19:32 +03:00
Aliaksandr Valialkin
390a31ccfa docs/CHANGELOG.md: document the change from bfba4c28a4 2021-07-12 12:42:42 +03:00
Aliaksandr Valialkin
0b98f6c7ff app/vmselect: expose vmui at /vmselect/<accountID>/vmui/ instead of /vmselect/<accountID>/prometheus/vmui/ 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413
 2021-07-10 12:32:21 +03:00
Aliaksandr Valialkin
98e049ba6d app/vmui: move source code from https://github.com/VictoriaMetrics/vmui to app/vmui 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413
 2021-07-09 17:13:51 +03:00
Aliaksandr Valialkin
2c5e1cd893 app/vmselect: move web ui from /ui to /select/<accountID>/prometheus/ui 
This way the UI is available for every tenant (aka accountID) and the UI can automatically determine the needed per-tenant datasource path from page referer.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413
 2021-07-08 13:14:50 +03:00
Aliaksandr Valialkin
acb7a95c64 app/vmselect: follow-up after aa11ef6d3b 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4
 2021-07-07 17:45:09 +03:00
Aliaksandr Valialkin
41754e12f8 lib/mergeset: cache indexBlock items only on the second request 
This should reduce the indexdb/indexBlocks cache size, since it won't contain one-time-wonders items.
 2021-07-07 15:24:37 +03:00
Aliaksandr Valialkin
9c19719ad6 app/{vminsert,vmselect}: export vminsert_request_duration_seconds and vmselect_request_duration_seconds histograms 2021-07-07 13:27:23 +03:00
Aliaksandr Valialkin
ceda2b1df4 lib/httpserver: print full requestURI in httpserver.Errorf 
This should simplify debugging.
 2021-07-07 13:11:29 +03:00
Aliaksandr Valialkin
4b25e627f8 lib/workingsetcache: properly switch to whole mode 
Previously the switch from `split` to `whole` mode had been performed too early,
e.g. when the current cache size became bigger than 1/4 of the allowed cache size.

Now it is performed when the current cache size becomes bigger than 1/2 of the allowed cache size.

This change can reduce memory usage for data ingestion path when big number of active time series are ingested.
 2021-07-05 15:15:39 +03:00
Aliaksandr Valialkin
28a778dc9f docs/CHANGELOG.md: document the bugfix for vm_merge_need_free_disk_space metric at 9a83e9018d 2021-07-05 12:01:24 +03:00
Aliaksandr Valialkin
1c12c0f79c lib/promauth: reload TLS certificates from disk on every mTLS connection as Prometheus does 
This allows updating client certificates without the need to restart vmagent and/or single-node VictoriaMetrics.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1420
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/470
 2021-07-01 15:43:43 +03:00
Aliaksandr Valialkin
c1b15c7764 docs/CHANGELOG.md: document ae485c2bfd 2021-07-01 11:52:47 +03:00
Aliaksandr Valialkin
71c856beb8 lib/workingsetcache: reset the cache mode when the cache is reset 
This should reduce memory usage if the working set is reduced after the cache reset.
 2021-07-01 11:52:47 +03:00
Aliaksandr Valialkin
0c4c630839 lib/promscrape: typo fix in /targets output 
The typo has been introduced in fb72a2133f

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1408
 2021-06-28 21:27:22 +03:00
Aliaksandr Valialkin
db66ea9a5f docs/CHANGELOG.md: cut v1.62.0 2021-06-25 13:29:48 +03:00
Aliaksandr Valialkin
97d1ccfc8e lib/promscrape: split docker and dockerswarm service discovery code bases, since they have very little in common 
This is a follow up after c85a5b7fcb
 2021-06-25 13:22:16 +03:00
Aliaksandr Valialkin
1b6850bab3 docs/CHANGELOG.md: document the bugfix in increase_pure() function from the commit fb4f758715 2021-06-24 12:06:15 +03:00
Aliaksandr Valialkin
b84aea1e6e lib/protoparser/clusternative: do not pool unmarshalWork structs, since they can occupy big amounts of memory (more than 100MB per each struct) 
This should reduce memory usage for vmstorage under high ingestion rate when the vmstorage runs on a system with big number of CPU cores
 2021-06-23 15:45:08 +03:00
Aliaksandr Valialkin
856aecae05 app/vmselect/promql: return the last timestamp for the max / min value from tmax_over_time() and tmin_over_time() function as most users expect 2021-06-23 14:18:37 +03:00
Aliaksandr Valialkin
dffa2afefe docs/CHANGELOG.md: document the bugfix for incorrect stats collection for concurrently executed tag filter 
Follow up for c22114c6f0
 2021-06-23 14:06:33 +03:00
Aliaksandr Valialkin
c18017a9c3 app/vminsert/netstorage: sort the -storageNode list passed to vminsert nodes 
This should reduce resource usage (CPU, RAM, disk IO) at vmstorage nodes
if the addresses of vmstorage nodes are passed in random order to vminsert nodes.
 2021-06-23 14:00:08 +03:00
Aliaksandr Valialkin
4adf6c9766 lib/promscrape/discovery/http: follow up after e307bbb29a 2021-06-22 13:42:10 +03:00
Aliaksandr Valialkin
3ab3902f17 lib/promscrape/discovery: support generic auth configs in Consul service discovery in the same way as Prometheus 2.28 does 2021-06-22 13:18:51 +03:00
Aliaksandr Valialkin
61fc9b98e5 docs/CHANGELOG.md: document the support for Consul namepsace 
See 58a2989fe7
 2021-06-22 12:56:12 +03:00
Aliaksandr Valialkin
23fcafd437 docs/CHANGELOG.md: typo fixes 2021-06-18 19:15:29 +03:00
Aliaksandr Valialkin
b92d110cad app/vmselect: log slow requests to all the /api/v1/* handlers if their execution time exceeds -search.logSlowQueryDuration 2021-06-18 19:07:03 +03:00
Aliaksandr Valialkin
4acc4602b3 app/vmctl: limit JSON line size by 10K samples (#1394) 
This should reduce the maximum memory usage at VictoriaMetrics when importing time series with big number of samples.
 2021-06-18 15:41:34 +03:00
Aliaksandr Valialkin
cd697b88c5 docs/CHANGELOG.md: document the reduced disk write IO usage 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338
 2021-06-18 14:03:42 +03:00
Aliaksandr Valialkin
f9069ba32a lib/promscrape: show jobs with empty scrape targets on /targets page 2021-06-18 10:54:12 +03:00
Aliaksandr Valialkin
644102b03b docs/CHANGELOG.md: document the changed -remoteWrite.queues value 
This is a follow-up for 0a796f7c3a

See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1385
 2021-06-16 12:37:55 +03:00
Aliaksandr Valialkin
b133de1e37 lib/storage: move deletedMetricIDs set from indexDB to Storage 
This makes consitent the list of deleted metricIDs when it is used from both the current indexDB and the previous indexDB (aka extDB).
This should fix the issue, which could lead to storing new samples under deleted metricIDs after indexDB rotation.
See more details at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1347#issuecomment-861232136 .

Thanks to @tangqipengleoo for the initial analysis and the pull request - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1383 .

This commit resolves the issue in more generic way compared to https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1383 .

The downside of the commit is the deletedMetricIDs set isn't cleaned from the metricIDs outside the retention. It needs app restart.
This should be OK in most cases.
 2021-06-15 15:07:54 +03:00
Aliaksandr Valialkin
5f91a701fa lib/promscrape: show the number of samples collected during the last scrape at /targets and /api/v1/targets pages 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1377
 2021-06-14 14:04:35 +03:00
Aliaksandr Valialkin
5cd50d840f docs/CHANGELOG.md: document the addition of DigitalOcean service discovery 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1367
 2021-06-14 13:19:31 +03:00
Aliaksandr Valialkin
541429a9af docs/CHANGELOG.md: cut v1.61.1 2021-06-11 13:02:04 +03:00
Aliaksandr Valialkin
ce10bdc82a lib/storage: reset cache on disk during series deletion and during indexdb rotation 
This should prevent from inconsistent behavior (aka partially missing data for some time series) after unclean shutdown.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1347
 2021-06-11 12:54:36 +03:00
Aliaksandr Valialkin
d5973d3180 docs/CHANGELOG.md: document the bugfix from 7adfe878e1 2021-06-11 11:30:07 +03:00
Aliaksandr Valialkin
83adc2f3ac docs/CHANGELOG.md: cut v1.61.0 2021-06-09 19:04:59 +03:00
Aliaksandr Valialkin
1e13deaa2c docs/CHANGELOG.md: document the enterprise bugfix for the target property in Graphite Render API 2021-06-09 13:51:32 +03:00
Aliaksandr Valialkin
490783696a docs/CHANGELOG.md: document improvements in re-routing handling in vminsert 
See the following commits:

* 1c09e71f5b
* 0d067eb112
* 2c6b917749
 2021-06-09 13:42:12 +03:00
Aliaksandr Valialkin
f3749dedba docs: document rules replay feature for vmalert 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/836

This is a follow-up for 2a259ef5e7
 2021-06-09 12:30:54 +03:00
Aliaksandr Valialkin
8ad445474a docs/CHANGELOG.md: document the bugfix, which prevents panics for aborted http requests in vmauth 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1353

This is a follow-up for 6b29b955c0
 2021-06-09 12:12:47 +03:00
Aliaksandr Valialkin
269e35d676 app/{vmagent,vminsert}: follow-up after 2fe045e2a4 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1343
 2021-06-04 20:33:22 +03:00
Aliaksandr Valialkin
67cfc07004 docs/CHANGELOG.md: document the bugfix from 6f19bb23a1 2021-06-04 11:56:06 +03:00
Aliaksandr Valialkin
c53a90e5fc docs/CHANGELOG.md: document that it is possible to build VictoriaMetrics components for Solaris 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1322

This is a follow-up for ddc8022702
 2021-06-04 04:41:12 +03:00
Aliaksandr Valialkin
1c09e71f5b app/vminsert: add -disableRerouting command-line flag for disabling re-routing if some vmstorage nodes have lower performance than the others 
Refactor the rerouting mechanism and make it more resilient to cases when some of vmstorage nodes are temporarily unavailable.

Reduce the probability of rerouting storm.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/791
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1054
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1165
 2021-06-04 04:33:52 +03:00
Aliaksandr Valialkin
8cdecfc52c app/vmauth: allow balancing the load among multiple backend nodes by specifying multiple urls in url_prefix config 2021-05-29 01:04:22 +03:00
Aliaksandr Valialkin
97de72054e docs: document f0c21b6300 2021-05-27 15:04:13 +03:00
Aliaksandr Valialkin
820ac6cd0c docs/CHANGELOG.md: document changes from 2233d6ed8a and d210958fd0 2021-05-26 12:24:28 +03:00
Aliaksandr Valialkin
25ed1f0c4f docs/CHANGELOG.md: cut v1.60.0 2021-05-24 15:55:31 +03:00
Aliaksandr Valialkin
402a8ca710 lib/storage: do not populate MetricID->MetricName cache during data ingestion 
This cache isn't needed during data ingestion, so there is no need in spending RAM on it.

This reduces RAM usage on data ingestion path by 30%
 2021-05-24 03:06:40 +03:00
Aliaksandr Valialkin
68c3901ebd docs/CHANGELOG.md: small typo fix 2021-05-23 14:15:49 +03:00
Aliaksandr Valialkin
8ec3e876be docs/CHANGELOG.md: document the addition of extra_filter_labels at 84cc0513e1 2021-05-23 14:15:49 +03:00
Aliaksandr Valialkin
71ff7ee18d lib/promauth: follow-up after 5b8176c68e 2021-05-22 18:02:03 +03:00
Aliaksandr Valialkin
23355ca34c lib/fs: wait for a while before giving up on NFS file removal if the removal queue is full 
This should reduce the probability of the panic on a highly loaded VictoriaMetrics
accepting millions of samples per second.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1313
 2021-05-21 17:21:35 +03:00
Aliaksandr Valialkin
f664f7fb1d docs/CHANGELOG.md: mention the bugfix from d626c5c2a9 
Updates https://github.com/VictoriaMetrics/operator/issues/243
 2021-05-21 16:38:20 +03:00
Aliaksandr Valialkin
d77db9d813 all: do not skip SIGHUP signal during service initialization 
This can lead to stale or incomplete configs like in the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240
 2021-05-21 16:38:20 +03:00
Aliaksandr Valialkin
9aa22cccf0 docs/CHANGELOG.md: move tip to proper place 2021-05-20 17:58:24 +03:00
Aliaksandr Valialkin
165a9f9200 app/vmstorage: add ability to limit series cardinality via -storage.maxHourlySeries and -storage.maxDailySeries command-line flags 2021-05-20 15:31:57 +03:00
Aliaksandr Valialkin
7aad5c3f76 app/vmagent: add ability to limit series cardinality on a per-hour and per-day basis 2021-05-20 15:31:57 +03:00
Aliaksandr Valialkin
a613be1518 docs/CHANGELOG.md: document the bugfix in vmctl import for InfluxDB lines with identical names for field and tag 
See dcf8803bbd

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1299
 2021-05-20 12:07:15 +03:00
Aliaksandr Valialkin
2cc5567ab8 docs/CHANGELOG.md: refer to the issue related to timezone_offset() function 2021-05-20 12:02:32 +03:00
Aliaksandr Valialkin
180829b8c2 app/vmselect/promql: add timezone_offset(tz) function 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1306
 2021-05-20 11:54:06 +03:00
Aliaksandr Valialkin
1668280e67 docs/vmalert.md: document multitenant support 
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/740
 2021-05-18 16:25:21 +03:00
Aliaksandr Valialkin
7fe362deb1 app/vmauth: reload -auth.config on the request to /-/reload 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1194
 2021-05-18 02:24:37 +03:00
Aliaksandr Valialkin
bae4d61ef2 docs/CHANGELOG.md: document b38edec7ee 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1293
 2021-05-17 01:58:18 +03:00
Aliaksandr Valialkin
e08287f017 lib/promscrape: reload auth tokens from files every second 
Previously auth tokens were loaded at startup and couldn't be updated without vmagent restart.
Now there is no need in vmagent restart.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1297
 2021-05-14 20:03:35 +03:00
Aliaksandr Valialkin
a6cb4f10a7 app/{vmalert,vmauth}: explicitly set MaxIdleConnsPerHost in net/http.Client.Transport 
By default MaxIdleConnsPerHost is set to 2. This limits the possibility to re-use http keep-alive connections.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1300
 2021-05-14 18:13:34 +03:00
Aliaksandr Valialkin
e3f61d540b lib/promscrape: limit scrape_timeout by scrape_interval like Prometheus does 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1281
 2021-05-13 16:10:42 +03:00
Aliaksandr Valialkin
76e03b46df docs/CHANGELOG.md: document the bugfix from b4f5be8bd8 2021-05-13 11:19:31 +03:00
Aliaksandr Valialkin
f13585dc5d vendor: update github.com/VictoriaMetrics/fasthttp from v1.0.14 to v1.0.15 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1289
 2021-05-13 10:47:09 +03:00
Aliaksandr Valialkin
66c6976723 lib/cgroup: document the ability to detect cgroup v2 memory and cpu limits. This is follow-up for b50024812e 2021-05-13 09:27:35 +03:00
Nikolay
be87be34a4 Adds tsdb match filters (#1282) 
* init work on filters

* init propose for status filters

* fixes tsdb status
adds test

* fix bug

* removes checks from test
 2021-05-12 17:16:58 +03:00
Aliaksandr Valialkin
56b08390f6 app/vmselect/promql: allow to use 2x more memory for query processing in cluster mode compared to single-node mode 
`vmselect` has no `vmstorage`-related caches. So it can use more memory for query processing compared to single-node VictoriaMetrics.
 2021-05-12 14:43:49 +03:00
Aliaksandr Valialkin
f1317f7c6c lib/httpserver: return X-Server-Hostname http header in all the responses for better debuggability 2021-05-11 22:04:41 +03:00
Aliaksandr Valialkin
cca9670573 docs/CHANGELOG.md: document -datasource.roundDigits added at 5c448126dc 2021-05-10 11:18:58 +03:00
Aliaksandr Valialkin
f551b99082 docs/CHANGELOG.md: document vmalert fix for state restoration on startup 2021-05-10 11:10:34 +03:00
Aliaksandr Valialkin
4128c4db16 docs/CHANGELOG.md: mention the comment, which gives an example of multi-level vminsert setup 2021-05-08 22:50:40 +03:00
Aliaksandr Valialkin
9c505d27dd lib/ingestserver: properly close incoming connections during graceful shutdown 2021-05-08 19:53:45 +03:00
Aliaksandr Valialkin
4a5f45c77e app/vminsert: add support for data ingestion via other vminsert nodes 2021-05-08 19:53:45 +03:00
Aliaksandr Valialkin
3108cdb930 docs/CHANGELOG.md: document 904bbffc7f 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240
 2021-05-05 20:34:00 +03:00
Aliaksandr Valialkin
326c7995c1 docs/CHANGELOG.md: document 9cdd4696fe 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1252
 2021-05-05 20:33:59 +03:00
Aliaksandr Valialkin
ec6becd3f5 lib/httpserver: stop the process on panics in request handlers 
Panics may leave the process in inconsistent state. That's why it is better to stop the process after the panic
instead of recovering from the panic. Unfortunately, the standard net/http.Server recovers panics in request handlers.
See https://github.com/golang/go/issues/16542 . That's lib/httpserver must stop the process on itself after the panic.
 2021-05-03 12:00:44 +03:00
Aliaksandr Valialkin
a302f79e5e docs/CHANGELOG.md: document the bugfix for proper removal of stale parts (477369b62f) 2021-05-03 11:38:38 +03:00
Aliaksandr Valialkin
9aa44a2760 docs/CHANGELOG.md: cut v1.59.0 2021-05-01 09:43:35 +03:00
Aliaksandr Valialkin
b43ba6d85f lib/storage: log dropped labels if the number of labels in a metric exceeds -maxLabelsPerTimeseries command-line flag value 
This should improve debuggability for this case.
 2021-05-01 09:29:56 +03:00
Aliaksandr Valialkin
daf2778025 docs/CHANGELOG.md: document the change from f3a048288e 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1232
 2021-04-30 09:56:47 +03:00
Aliaksandr Valialkin
8be1cb297b app/vmagent: list user-visible endpoints at http://vmagent:8429/ 
While at it, use common WriteAPIHelp function for the listing in vmagent, vmalert and victoria-metrics
 2021-04-30 09:38:23 +03:00