github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
2906 commits 150 branches 748 tags 347 MiB
Commit graph

1127 commits

Author SHA1 Message Date
Aliaksandr Valialkin
8055439fe4 lib/storage: properly detect free disk space shortage during data merge 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1373
 2021-07-02 17:42:23 +03:00
Aliaksandr Valialkin
6fc3696260 lib/promscrape/discovery/consul: use case-insensitive comparison for service names 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1424
 2021-07-02 14:49:22 +03:00
Aliaksandr Valialkin
61e483a01c lib/protoparser/clusternative: remove unused field - unmarshalWork.lastResetTime 
This is a follow-up for b84aea1e6e
 2021-07-02 13:32:59 +03:00
Aliaksandr Valialkin
72de54f93e lib/promauth: cache the client TLS certificate for up to a second 
This should reduce CPU usage when TLS connections are established at a high rate.
 2021-07-02 13:20:18 +03:00
Aliaksandr Valialkin
1c12c0f79c lib/promauth: reload TLS certificates from disk on every mTLS connection as Prometheus does 
This allows updating client certificates without the need to restart vmagent and/or single-node VictoriaMetrics.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1420
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/470
 2021-07-01 15:43:43 +03:00
Nikolay
6bd2309449 fixes /targets button style (#1423) 
* fixes /targets button style
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1422

* updates boostrap version
 2021-07-01 11:52:47 +03:00
Aliaksandr Valialkin
71c856beb8 lib/workingsetcache: reset the cache mode when the cache is reset 
This should reduce memory usage if the working set is reduced after the cache reset.
 2021-07-01 11:52:47 +03:00
Aliaksandr Valialkin
bced9ee666 lib/{mergeset,storage}: reduce the maximum lifetime for cached indexdb and data blocks from 2 minutes to a minute 
This should reduce memory usage on a system with high number of active time series and a high churn rate.
One minute is enough for caching the blocks needed for repeated queries (e.g. alerting rules, recording rules and dashboard refreshes).
 2021-06-29 19:57:53 +03:00
Aliaksandr Valialkin
b7c0b3dde3 lib/mergeset: switch from sync.Pool to a channel for a pool for inmemoryBlock structs 
This should reduce memory usage for the pool on systems with big number of CPU cores.

The sync.Pool maintains per-CPU pools, so the total number of objects in the pool
is proportional to the number of available CPU cores. The channel limits the number
of pooled objects by its own capacity. This means smaller number of pooled objects on average.
 2021-06-29 19:57:52 +03:00
Aliaksandr Valialkin
2edfea8c36 lib/promscrape/discovery/docker: fix golint warning: struct field Id should be ID 2021-06-29 13:11:33 +03:00
Aliaksandr Valialkin
609ad6d9bf lib/storage: put indexDBName into the key for dateTagFilter cache and for uselessTagFilters cache 
This should prevent from stats overwriting when the previous indexdb is queried.
 2021-06-29 13:11:32 +03:00
Aliaksandr Valialkin
0c4c630839 lib/promscrape: typo fix in /targets output 
The typo has been introduced in fb72a2133f

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1408
 2021-06-28 21:27:22 +03:00
Aliaksandr Valialkin
97d1ccfc8e lib/promscrape: split docker and dockerswarm service discovery code bases, since they have very little in common 
This is a follow up after c85a5b7fcb
 2021-06-25 13:22:16 +03:00
Aliaksandr Valialkin
4461e20e7d lib/promscrape: consistently sort service discovery routines 
This should simplify further maintenance of the code
 2021-06-25 13:22:16 +03:00
Lu Jiajing
12b4cbb68f Support Docker ServiceDiscovery (#1402) 
* add docker discovery

* add test

* add labels test and add scrape work

* remove TODO

* refactor to merge apiConfig and sdConfig

* apply suggestion
 2021-06-25 13:22:16 +03:00
Nikolay
501429c3ff adds missing MustStop call to do and http sd (#1404) 2021-06-25 11:43:32 +03:00
Aliaksandr Valialkin
b84aea1e6e lib/protoparser/clusternative: do not pool unmarshalWork structs, since they can occupy big amounts of memory (more than 100MB per each struct) 
This should reduce memory usage for vmstorage under high ingestion rate when the vmstorage runs on a system with big number of CPU cores
 2021-06-23 15:45:08 +03:00
Aliaksandr Valialkin
a22f37599b lib/storage: tune tag filters search logic 
Tune the logic according to the logs provided at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338#issuecomment-864293624

The previous logic had a race when multiple concurrent queries execute the same tag filter without prior stats.
This could result in incorrectly stored stats for such tag filter, which then could result in non-optimal sorting of tag filters
for further queries.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338
 2021-06-23 13:30:36 +03:00
Aliaksandr Valialkin
f10fa0d1d7 lib/promscrape/discovery/consul: properly pass namespace to Consul watcher 
Follow-up for 58a2989fe7
 2021-06-22 17:43:20 +03:00
Aliaksandr Valialkin
4adf6c9766 lib/promscrape/discovery/http: follow up after e307bbb29a 2021-06-22 13:42:10 +03:00
Nikolay
e03a3d3a36 adds http_sd (#1399) 
* adds http_sd

* adds X-Prometheus-Refresh-Interval-Seconds header

* Update lib/promscrape/discovery/http/api.go

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2021-06-22 13:42:09 +03:00
Aliaksandr Valialkin
3ab3902f17 lib/promscrape/discovery: support generic auth configs in Consul service discovery in the same way as Prometheus 2.28 does 2021-06-22 13:18:51 +03:00
Nikolay
827a2396d2 adds consul enterprise namespace support (#1400) 
* adds consul enterprise namespace support

* Update lib/promscrape/discovery/consul/consul.go

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2021-06-22 12:56:11 +03:00
Aliaksandr Valialkin
f9069ba32a lib/promscrape: show jobs with empty scrape targets on /targets page 2021-06-18 10:54:12 +03:00
Nikolay
9ea1dca3dd fixes DO service discovery labels (#1389) 
adds test for digitalocean sd
 2021-06-17 17:21:10 +03:00
Aliaksandr Valialkin
a207be3ffb lib/storage: fix infinite loop introduced in aa9b56a046 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1244
 2021-06-17 14:27:30 +03:00
Aliaksandr Valialkin
0efd37cec1 lib/{mergeset,storage}: reduce the number of fsync calls on data ingestion path on systems with many cpu cores 
VictoriaMetrics maintains a buffer per CPU core for the ingested data. These buffers are flushed to disk every second.
These buffers are flushed to disk in parallel starting from the commit 56b6b893ce .
This resulted in increased write disk IO usage on systems with many cpu cores
as described at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338#issuecomment-863046999 .

This commit merges the per-CPU buffers into bigger in-memory buffers before flushing them to disk.
This should reduce the rate of fsync syscalls and, consequently, the write disk IO on systems with many CPU cores.

This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338
See also https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1244
 2021-06-17 13:51:42 +03:00
Aliaksandr Valialkin
b133de1e37 lib/storage: move deletedMetricIDs set from indexDB to Storage 
This makes consitent the list of deleted metricIDs when it is used from both the current indexDB and the previous indexDB (aka extDB).
This should fix the issue, which could lead to storing new samples under deleted metricIDs after indexDB rotation.
See more details at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1347#issuecomment-861232136 .

Thanks to @tangqipengleoo for the initial analysis and the pull request - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1383 .

This commit resolves the issue in more generic way compared to https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1383 .

The downside of the commit is the deletedMetricIDs set isn't cleaned from the metricIDs outside the retention. It needs app restart.
This should be OK in most cases.
 2021-06-15 15:07:54 +03:00
Aliaksandr Valialkin
ebaf68bcb0 lib/protoparser: stop reading the input stream as soon as the callback provided by the caller returns error 
This is a follow-up for af90c3c43b
 2021-06-14 15:20:38 +03:00
faceair
2ea187e801 lib/protoparser: stop read when callback error (#1380) 2021-06-14 15:20:37 +03:00
Aliaksandr Valialkin
5f91a701fa lib/promscrape: show the number of samples collected during the last scrape at /targets and /api/v1/targets pages 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1377
 2021-06-14 14:04:35 +03:00
Nikolay
e42da47608 adds digital ocean sd (#1376) 
* adds digital ocean sd config

* adds digital ocean sd
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1367

* typo fix
 2021-06-14 13:19:29 +03:00
Aliaksandr Valialkin
df057177a0 lib/promscrape: increase the duration for reading the full response in stream parsing mode 
Increase the duration from 10x to 30x of the configured `scrape_interval'.

This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1365
 2021-06-14 12:29:46 +03:00
Aliaksandr Valialkin
074b11fa69 lib/protoparser: measure the duration for reading the whole block of data instead of a single read operation 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1365
 2021-06-14 12:29:45 +03:00
Aliaksandr Valialkin
87d221f78a lib/protoparser/common: log the duration for reading a block of data in ReadLinesBlockExt on error 
This may help debugging issues like https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1365
 2021-06-14 12:21:21 +03:00
Aliaksandr Valialkin
0672cfffa2 app/vmauth: properly handle http.ErrAbortHandler panic 
This panic can be raised by the reverseProxy on aborted request to the backend.
So handle it (e.g. suppress) at reverseProxy.ServeHTTP call.

Do not suppress the panic at lib/httpserver generic HTTP handler,
since it may result in an inconsistent state left after the panicking handler.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1353
 2021-06-11 12:54:37 +03:00
Aliaksandr Valialkin
ce10bdc82a lib/storage: reset cache on disk during series deletion and during indexdb rotation 
This should prevent from inconsistent behavior (aka partially missing data for some time series) after unclean shutdown.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1347
 2021-06-11 12:54:36 +03:00
Aliaksandr Valialkin
eb335d2c29 lib/storage: consistency renaming: getMaxRawRowsPerPartition -> getMaxRawRowsPerShard 2021-06-11 10:52:31 +03:00
Aliaksandr Valialkin
d06c0e7a94 lib/storage: reduce the amounts of memory which can be occupied by rawRow items during data ingestion on a system with many CPU cores 2021-06-11 10:49:02 +03:00
Nikolay
2c1611d316 disables panic for net/httpAbortHandler (#1355) 2021-06-09 12:12:45 +03:00
Aliaksandr Valialkin
1e4a64844d lib/storage: properly account the number of loops spent when matching for or suffixes 
This may help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338
 2021-06-08 13:07:14 +03:00
Aliaksandr Valialkin
e7d353ee6a lib/promrelabel: add tests for labelsToString() function 2021-06-04 20:42:14 +03:00
Aliaksandr Valialkin
269e35d676 app/{vmagent,vminsert}: follow-up after 2fe045e2a4 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1343
 2021-06-04 20:33:22 +03:00
jelmd
d8b46908db new feature: debug relabeling (#1344) 
* new feature: relabel logging

Use scrape_configs[x].relabel_debug = true to log metric names inkl.
labels before and after relabeling. After relabeling related metrics
get dropped, i.e. not submitted to servers.

* vminsert wants relabel logging, too.
 2021-06-04 20:33:21 +03:00
Nikolay
3d89c01d07 fixes solaris build (#1345) 2021-06-04 11:56:06 +03:00
Hason Chan
439c2ed510 fix eureka_sd_configs HTTPClientConfig incorrect parsing (#1350) 2021-06-04 11:56:06 +03:00
Aliaksandr Valialkin
fc2565b4ee lib/storage: reduce memory allocations when syncing dateMetricIDCache 2021-06-03 16:20:02 +03:00
Aliaksandr Valialkin
0b9f0de0a1 lib/promscrape: fix tests after f0c21b6300 2021-05-28 01:33:28 +03:00
Aliaksandr Valialkin
6865f3b497 Revert "lib/mergeset: remove a pool for inmemoryBlock structs" 
This reverts commit 793fe39921.

Reason to revert: production testing revealed possible slowdown when registering big number of new time series
 2021-05-28 01:11:22 +03:00
Aliaksandr Valialkin
7b33bc67a1 lib/mergeset: remove a pool for inmemoryBlock structs 
The pool for inmemoryBlock struct doesn't give any performance gains in production workloads,
while it may result in excess memory usage for inmemoryBlock structs inside the pool during
background merge of indexdb.
 2021-05-27 22:00:50 +03:00