github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-01 14:47:38 +00:00
Code Issues Projects Releases Packages Wiki Activity
3300 commits 119 branches 748 tags 355 MiB
Commit graph

1224 commits

Author SHA1 Message Date
Aliaksandr Valialkin
327034b54f lib/promscrape/discovery/kubernetes: use v1 API instead of v1beta1 API for role: ingress and role: endpointslices 
This should fix service discovery for these roles in Kubernetes v1.22 and newer versions.
See https://kubernetes.io/docs/reference/using-api/deprecation-guide/#ingress-v122

The corresponding change in Prometheus - https://github.com/prometheus/prometheus/pull/9205
 2021-08-29 11:23:58 +03:00
Aliaksandr Valialkin
7fdb4db73d lib/promscrape: add ability to load scrape configs from multiple files 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1559
 2021-08-26 08:51:53 +03:00
Aliaksandr Valialkin
4a2d7aec7f lib/promscrape: expose promscrape_discovery_http_errors_total metric for tracking errors per each http_sd config 2021-08-25 13:05:29 +03:00
Aliaksandr Valialkin
b885bd9b7d lib/{mergeset,storage}: improve the detection of the needed free space for background merge 
This should prevent from possible out of disk space crashes during big merges.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1560
 2021-08-25 10:01:09 +03:00
Aliaksandr Valialkin
67bc407747 lib/promscrape: reduce memory and CPU usage when Prometheus staleness tracking is enabled for metrics from deleted / disappeared scrape targets 
Store the scraped response body instead of storing the parsed and relabeld metrics.
This should reduce memory usage, since the response body takes less memory than the parsed and relabeled metrics.
This is especially true for Kubernetes service discovery, which adds many long labels for all the scraped metrics.

This should also reduce CPU usage, since the marshaling of the parsed
and relabeld metrics has been substituted by response body copying.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526
 2021-08-21 21:24:07 +03:00
Aliaksandr Valialkin
c3b24882a7 lib/promscrape: use scrapeTimestamp when storing stale markers for failed scrape 
This will make timestamps for stale markers more consistent for timestamps for other samples
 2021-08-19 14:19:54 +03:00
Aliaksandr Valialkin
8ee575dee9 lib/promscrape: send stale markers for the previously scraped metrics on failed scrapes like Prometheus does 2021-08-18 22:00:46 +03:00
Aliaksandr Valialkin
5d92fafc40 app/vmselect: add -search.noStaleMarkers command-line flag for disabling stale markers handling in queries 
This option allows reducing CPU usage a bit when VictoriaMetrics is used
for collecting and processing non-Prometheus data. For example, InfluxDB line protocol, Graphite, OpenTSDB, CSV, etc.
 2021-08-18 13:58:06 +03:00
Aliaksandr Valialkin
f21fad53b4 lib/promscrape: add ability to disable sending Prometheus staleness markers with -promscrape.disableStaleMarkers command-line flag 
This option can be useful when vmagent consumes too much additional memory
for staleness markers functionality and when staleness markers aren't needed.
 2021-08-18 13:58:05 +03:00
Aliaksandr Valialkin
db34c40aec lib/promscrape: stop scrapers for the removed targets before starting scrapers for the added targets 
This should prevent from possible time series overlap when old target is substituted by new target (for example, during Kubernetes deployments).

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1530
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1509
 2021-08-17 01:00:40 +03:00
Aliaksandr Valialkin
5f13c519ee lib/promscrape: restore red highlighting for DOWN targets at /targets page 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1461
 2021-08-15 16:04:33 +03:00
Aliaksandr Valialkin
c1f81f08d4 all: add support for Prometheus staleness markers 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1509
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1530
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845
 2021-08-13 12:13:15 +03:00
Aliaksandr Valialkin
90efb5831b lib/envflag: add a link to docs for -envflag.enable 2021-08-11 10:32:40 +03:00
Aliaksandr Valialkin
b877538622 app/vmagent: follow-up after fe445f753b 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1491
 2021-08-05 09:51:00 +03:00
Omar Ghader
fe445f753b
feature: Add multitenant for vmagent (#1505) 
* feature: Add multitenant for vmagent

* Minor fix

* Fix rcs index out of range

* Minor fix

* Fix multi Init

* Fix multi Init

* Fix multi Init

* Add default multi

* Adjust naming

* Add TenantInserted metrics

* Add TenantInserted metrics

* fix: remove unused metrics for vmagent

* fix: remove unused metrics for vmagent

Co-authored-by: mghader <marc.ghader@ubisoft.com>
Co-authored-by: Sebastian YEPES <syepes@gmail.com>
 2021-08-05 09:44:29 +03:00
Aliaksandr Valialkin
77bb9e1656 lib/promscrape/discovery/gce: add __meta_gce_interface_ipv4_<name> labels as in Prometheus 2.29 
See https://github.com/prometheus/prometheus/pull/8978
 2021-08-03 15:51:45 +03:00
Aliaksandr Valialkin
336a2aa2e0 lib/promscrape/discovery/ec2: add __meta_ec2_availability_zone_id label as Prometheus 2.29 does 2021-08-03 13:28:13 +03:00
Aliaksandr Valialkin
c473d8ffe1 li/storage: re-use the per-day inverted index search code for searching in global index 
This allows removing a big pile of outdated code for global index search.

This may help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1486
 2021-07-30 10:28:20 +03:00
Nikolay
6d47e750be adds check for region with custom s3 endpoint (#1465) 2021-07-27 12:39:10 +03:00
Aliaksandr Valialkin
1950f57316 lib/storage: yet another attempt to properly determine disk space shortage, which prevents from optimal merges 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1373
 2021-07-27 12:03:31 +03:00
Aliaksandr Valialkin
92628f9f07 lib/promrelabel: add tests for verifying that regex works as expected in single quotes and double quotes 2021-07-27 10:53:03 +03:00
Aliaksandr Valialkin
5d255846ac all: add go:build lines for Go1.17 
See https://tip.golang.org/doc/go1.17#gofmt for more details
 2021-07-26 15:50:46 +03:00
Aliaksandr Valialkin
c857e05604 lib/promscrape: add missing whitespace at /targets page before up word 2021-07-26 12:23:06 +03:00
Aliaksandr Valialkin
376af3c956 lib/workingsetcache: switch from split cache to full cache after the cache size exceeds 95% of split capacity 
Previously the switch occurred when the cache size becomes 100% of its capacity. The cache size could never reach 100% capacity.
This could prevent from switching from the split cache to full cache, thus reducing the cache effectiveness.
 2021-07-15 16:53:35 +03:00
Aliaksandr Valialkin
9d3f9da5ad lib/storage: make sure the second call to DeduplicateSamples and deduplicateSamplesDuringMerge doesnt change samples 2021-07-15 12:18:38 +03:00
Aliaksandr Valialkin
e992754e79 lib/storage: remove cache directory if it contains reset_cache_on_startup file 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1447
 2021-07-13 17:59:51 +03:00
Aliaksandr Valialkin
e6edb85fa2 lib/httpserver: add is_set label to flag metrics 
This label allows determining the set flags with the query `flag{is_set="true"}`
 2021-07-13 15:10:18 +03:00
Aliaksandr Valialkin
51cd19d2e3 lib/storage: reset perKeyMisses stats less frequently 
This should reduce CPU usage for queries executed with intervals higher than 30 seconds
 2021-07-12 14:34:54 +03:00
Aliaksandr Valialkin
3f705fe8d7 lib/storage: properly limit the size of storage/date_metricID cache 2021-07-12 14:25:28 +03:00
Aliaksandr Valialkin
ef3c58d7a3 lib/storage: properly determine when the deduplication is needed in needsDedup 
Previously needsDedup() could return true if the de-duplication wasn't needed for the following case:

         d < interval
           /     \
   |        v | v        |
     interval   interval

Now it properly returns false for this case
 2021-07-12 10:54:51 +03:00
Aliaksandr Valialkin
41754e12f8 lib/mergeset: cache indexBlock items only on the second request 
This should reduce the indexdb/indexBlocks cache size, since it won't contain one-time-wonders items.
 2021-07-07 15:24:37 +03:00
Aliaksandr Valialkin
ceda2b1df4 lib/httpserver: print full requestURI in httpserver.Errorf 
This should simplify debugging.
 2021-07-07 13:11:29 +03:00
Aliaksandr Valialkin
9826f7c1be lib/storage: do not cache inmemoryBlock entries requested only once (aka one-time-wonder items) 
This should reduce the cache size and memory usage for the indexdb/dataBlocks cache
 2021-07-07 10:59:45 +03:00
Aliaksandr Valialkin
74ace9340d lib/storage: periodically reset prefetchedMetricIDs cache in order to limit its size under high churn rate 2021-07-07 10:59:39 +03:00
Aliaksandr Valialkin
a846febc89 Revert "lib/uint64set: allow reusing bucket16 structs inside uint64set.Set via uint64set.Release method" 
This reverts commit 7c6d3981bf.

Reason for revert: high contention at bucket16Pool on systems with big number of CPU cores.
This slows down query processing significantly.
 2021-07-06 18:26:56 +03:00
Aliaksandr Valialkin
b805a675f3 lib/{mergeset,storage}: switch from sync.Pool to chan-based pool for inmemoryPart objects 
This should reduce memory usage on systems with big number of CPU cores,
since every inmemoryPart object occupies at least 64KB of memory and sync.Pool maintains
a separate pool inmemoryPart objects per each CPU core.

Though the new scheme for the pool worsens per-cpu cache locality, this should be amortized
by big sizes of inmemoryPart objects.
 2021-07-06 16:33:25 +03:00
Aliaksandr Valialkin
d8e7c1ef27 lib/uint64set: allow reusing bucket16 structs inside uint64set.Set via uint64set.Release method 
This reduces the load on memory allocator in Go runtime in production workload.
 2021-07-06 16:33:24 +03:00
Aliaksandr Valialkin
db6bd69475 lib/mergeset: increase pool capacity for inmemoryBlock according to collected profiles from production workload 
CPU and memory profiles show that the pool capacity for inmemoryBlock objects is too small.
This results in the increased load on memory allocation code in Go runtime.
Increase the pool capacity in order to reduce the load on Go runtime.
 2021-07-06 13:44:27 +03:00
Aliaksandr Valialkin
fd32855a6c lib/mergeset: limit the frequency for flushCallback calls to once per 10 seconds 
This should improve hit ratio for tagFiltersCache when big number of new time series are constantly registered
(aka high churn rate). This, in turn, should reduce CPU usage for queries over such time series.
 2021-07-06 12:20:15 +03:00
Aliaksandr Valialkin
22c6e64bbc lib/storage: consistency renaming: tagCache -> tagFiltersCache 
This improves code readability
 2021-07-06 11:03:30 +03:00
Aliaksandr Valialkin
21abf487c3 lib/workingsetcache: properly update stats for requests and cache misses 
Previously the stats for cache misses could be improperly counted, because it had inflated cache misses
if the entry was missing in the curr cache, but was existing in the prev cache.

The same applies to cache requests - they were inflated if the entry was missing in the curr cache.
 2021-07-06 10:54:38 +03:00
Aliaksandr Valialkin
e5031d9aee lib/workingsetcache: fix cache capacity calculations after 4f0003f182 2021-07-05 17:16:35 +03:00
Aliaksandr Valialkin
bd71f102e8 lib/workingsetcache: typo fixes after d0c830039d 2021-07-05 15:35:51 +03:00
Aliaksandr Valialkin
4b25e627f8 lib/workingsetcache: properly switch to whole mode 
Previously the switch from `split` to `whole` mode had been performed too early,
e.g. when the current cache size became bigger than 1/4 of the allowed cache size.

Now it is performed when the current cache size becomes bigger than 1/2 of the allowed cache size.

This change can reduce memory usage for data ingestion path when big number of active time series are ingested.
 2021-07-05 15:15:39 +03:00
Aliaksandr Valialkin
51516b96e6 lib/storage: tune cache sizes according to production workload 2021-07-05 15:14:45 +03:00
Aliaksandr Valialkin
f12f97daa1 lib/{storage,mergeset}: increase cache timeout for data and index blocks from a minute to two minutes 
One minute cache timeout result in slower queries in some production workloads where the interval
between query execution is in the range 1 minute - 2 minutes.
 2021-07-05 14:25:59 +03:00
Aliaksandr Valialkin
377bb06b47 lib/cgroup: set GOGC to 50 by default if it isn't set 
This should reduce memory usage for typical VictoriaMetrics workloads by up to 50%
 2021-07-05 12:34:01 +03:00
Aliaksandr Valialkin
8055439fe4 lib/storage: properly detect free disk space shortage during data merge 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1373
 2021-07-02 17:42:23 +03:00
Aliaksandr Valialkin
6fc3696260 lib/promscrape/discovery/consul: use case-insensitive comparison for service names 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1424
 2021-07-02 14:49:22 +03:00
Aliaksandr Valialkin
61e483a01c lib/protoparser/clusternative: remove unused field - unmarshalWork.lastResetTime 
This is a follow-up for b84aea1e6e
 2021-07-02 13:32:59 +03:00
Aliaksandr Valialkin
72de54f93e lib/promauth: cache the client TLS certificate for up to a second 
This should reduce CPU usage when TLS connections are established at a high rate.
 2021-07-02 13:20:18 +03:00
Aliaksandr Valialkin
1c12c0f79c lib/promauth: reload TLS certificates from disk on every mTLS connection as Prometheus does 
This allows updating client certificates without the need to restart vmagent and/or single-node VictoriaMetrics.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1420
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/470
 2021-07-01 15:43:43 +03:00
Nikolay
6bd2309449 fixes /targets button style (#1423) 
* fixes /targets button style
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1422

* updates boostrap version
 2021-07-01 11:52:47 +03:00
Aliaksandr Valialkin
71c856beb8 lib/workingsetcache: reset the cache mode when the cache is reset 
This should reduce memory usage if the working set is reduced after the cache reset.
 2021-07-01 11:52:47 +03:00
Aliaksandr Valialkin
bced9ee666 lib/{mergeset,storage}: reduce the maximum lifetime for cached indexdb and data blocks from 2 minutes to a minute 
This should reduce memory usage on a system with high number of active time series and a high churn rate.
One minute is enough for caching the blocks needed for repeated queries (e.g. alerting rules, recording rules and dashboard refreshes).
 2021-06-29 19:57:53 +03:00
Aliaksandr Valialkin
b7c0b3dde3 lib/mergeset: switch from sync.Pool to a channel for a pool for inmemoryBlock structs 
This should reduce memory usage for the pool on systems with big number of CPU cores.

The sync.Pool maintains per-CPU pools, so the total number of objects in the pool
is proportional to the number of available CPU cores. The channel limits the number
of pooled objects by its own capacity. This means smaller number of pooled objects on average.
 2021-06-29 19:57:52 +03:00
Aliaksandr Valialkin
2edfea8c36 lib/promscrape/discovery/docker: fix golint warning: struct field Id should be ID 2021-06-29 13:11:33 +03:00
Aliaksandr Valialkin
609ad6d9bf lib/storage: put indexDBName into the key for dateTagFilter cache and for uselessTagFilters cache 
This should prevent from stats overwriting when the previous indexdb is queried.
 2021-06-29 13:11:32 +03:00
Aliaksandr Valialkin
0c4c630839 lib/promscrape: typo fix in /targets output 
The typo has been introduced in fb72a2133f

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1408
 2021-06-28 21:27:22 +03:00
Aliaksandr Valialkin
97d1ccfc8e lib/promscrape: split docker and dockerswarm service discovery code bases, since they have very little in common 
This is a follow up after c85a5b7fcb
 2021-06-25 13:22:16 +03:00
Aliaksandr Valialkin
4461e20e7d lib/promscrape: consistently sort service discovery routines 
This should simplify further maintenance of the code
 2021-06-25 13:22:16 +03:00
Lu Jiajing
12b4cbb68f Support Docker ServiceDiscovery (#1402) 
* add docker discovery

* add test

* add labels test and add scrape work

* remove TODO

* refactor to merge apiConfig and sdConfig

* apply suggestion
 2021-06-25 13:22:16 +03:00
Nikolay
501429c3ff adds missing MustStop call to do and http sd (#1404) 2021-06-25 11:43:32 +03:00
Aliaksandr Valialkin
b84aea1e6e lib/protoparser/clusternative: do not pool unmarshalWork structs, since they can occupy big amounts of memory (more than 100MB per each struct) 
This should reduce memory usage for vmstorage under high ingestion rate when the vmstorage runs on a system with big number of CPU cores
 2021-06-23 15:45:08 +03:00
Aliaksandr Valialkin
a22f37599b lib/storage: tune tag filters search logic 
Tune the logic according to the logs provided at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338#issuecomment-864293624

The previous logic had a race when multiple concurrent queries execute the same tag filter without prior stats.
This could result in incorrectly stored stats for such tag filter, which then could result in non-optimal sorting of tag filters
for further queries.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338
 2021-06-23 13:30:36 +03:00
Aliaksandr Valialkin
f10fa0d1d7 lib/promscrape/discovery/consul: properly pass namespace to Consul watcher 
Follow-up for 58a2989fe7
 2021-06-22 17:43:20 +03:00
Aliaksandr Valialkin
4adf6c9766 lib/promscrape/discovery/http: follow up after e307bbb29a 2021-06-22 13:42:10 +03:00
Nikolay
e03a3d3a36 adds http_sd (#1399) 
* adds http_sd

* adds X-Prometheus-Refresh-Interval-Seconds header

* Update lib/promscrape/discovery/http/api.go

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2021-06-22 13:42:09 +03:00
Aliaksandr Valialkin
3ab3902f17 lib/promscrape/discovery: support generic auth configs in Consul service discovery in the same way as Prometheus 2.28 does 2021-06-22 13:18:51 +03:00
Nikolay
827a2396d2 adds consul enterprise namespace support (#1400) 
* adds consul enterprise namespace support

* Update lib/promscrape/discovery/consul/consul.go

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2021-06-22 12:56:11 +03:00
Aliaksandr Valialkin
f9069ba32a lib/promscrape: show jobs with empty scrape targets on /targets page 2021-06-18 10:54:12 +03:00
Nikolay
9ea1dca3dd fixes DO service discovery labels (#1389) 
adds test for digitalocean sd
 2021-06-17 17:21:10 +03:00
Aliaksandr Valialkin
a207be3ffb lib/storage: fix infinite loop introduced in aa9b56a046 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1244
 2021-06-17 14:27:30 +03:00
Aliaksandr Valialkin
0efd37cec1 lib/{mergeset,storage}: reduce the number of fsync calls on data ingestion path on systems with many cpu cores 
VictoriaMetrics maintains a buffer per CPU core for the ingested data. These buffers are flushed to disk every second.
These buffers are flushed to disk in parallel starting from the commit 56b6b893ce .
This resulted in increased write disk IO usage on systems with many cpu cores
as described at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338#issuecomment-863046999 .

This commit merges the per-CPU buffers into bigger in-memory buffers before flushing them to disk.
This should reduce the rate of fsync syscalls and, consequently, the write disk IO on systems with many CPU cores.

This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338
See also https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1244
 2021-06-17 13:51:42 +03:00
Aliaksandr Valialkin
b133de1e37 lib/storage: move deletedMetricIDs set from indexDB to Storage 
This makes consitent the list of deleted metricIDs when it is used from both the current indexDB and the previous indexDB (aka extDB).
This should fix the issue, which could lead to storing new samples under deleted metricIDs after indexDB rotation.
See more details at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1347#issuecomment-861232136 .

Thanks to @tangqipengleoo for the initial analysis and the pull request - https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1383 .

This commit resolves the issue in more generic way compared to https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1383 .

The downside of the commit is the deletedMetricIDs set isn't cleaned from the metricIDs outside the retention. It needs app restart.
This should be OK in most cases.
 2021-06-15 15:07:54 +03:00
Aliaksandr Valialkin
ebaf68bcb0 lib/protoparser: stop reading the input stream as soon as the callback provided by the caller returns error 
This is a follow-up for af90c3c43b
 2021-06-14 15:20:38 +03:00
faceair
2ea187e801 lib/protoparser: stop read when callback error (#1380) 2021-06-14 15:20:37 +03:00
Aliaksandr Valialkin
5f91a701fa lib/promscrape: show the number of samples collected during the last scrape at /targets and /api/v1/targets pages 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1377
 2021-06-14 14:04:35 +03:00
Nikolay
e42da47608 adds digital ocean sd (#1376) 
* adds digital ocean sd config

* adds digital ocean sd
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1367

* typo fix
 2021-06-14 13:19:29 +03:00
Aliaksandr Valialkin
df057177a0 lib/promscrape: increase the duration for reading the full response in stream parsing mode 
Increase the duration from 10x to 30x of the configured `scrape_interval'.

This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1365
 2021-06-14 12:29:46 +03:00
Aliaksandr Valialkin
074b11fa69 lib/protoparser: measure the duration for reading the whole block of data instead of a single read operation 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1365
 2021-06-14 12:29:45 +03:00
Aliaksandr Valialkin
87d221f78a lib/protoparser/common: log the duration for reading a block of data in ReadLinesBlockExt on error 
This may help debugging issues like https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1365
 2021-06-14 12:21:21 +03:00
Aliaksandr Valialkin
0672cfffa2 app/vmauth: properly handle http.ErrAbortHandler panic 
This panic can be raised by the reverseProxy on aborted request to the backend.
So handle it (e.g. suppress) at reverseProxy.ServeHTTP call.

Do not suppress the panic at lib/httpserver generic HTTP handler,
since it may result in an inconsistent state left after the panicking handler.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1353
 2021-06-11 12:54:37 +03:00
Aliaksandr Valialkin
ce10bdc82a lib/storage: reset cache on disk during series deletion and during indexdb rotation 
This should prevent from inconsistent behavior (aka partially missing data for some time series) after unclean shutdown.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1347
 2021-06-11 12:54:36 +03:00
Aliaksandr Valialkin
eb335d2c29 lib/storage: consistency renaming: getMaxRawRowsPerPartition -> getMaxRawRowsPerShard 2021-06-11 10:52:31 +03:00
Aliaksandr Valialkin
d06c0e7a94 lib/storage: reduce the amounts of memory which can be occupied by rawRow items during data ingestion on a system with many CPU cores 2021-06-11 10:49:02 +03:00
Nikolay
2c1611d316 disables panic for net/httpAbortHandler (#1355) 2021-06-09 12:12:45 +03:00
Aliaksandr Valialkin
1e4a64844d lib/storage: properly account the number of loops spent when matching for or suffixes 
This may help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1338
 2021-06-08 13:07:14 +03:00
Aliaksandr Valialkin
e7d353ee6a lib/promrelabel: add tests for labelsToString() function 2021-06-04 20:42:14 +03:00
Aliaksandr Valialkin
269e35d676 app/{vmagent,vminsert}: follow-up after 2fe045e2a4 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1343
 2021-06-04 20:33:22 +03:00
jelmd
d8b46908db new feature: debug relabeling (#1344) 
* new feature: relabel logging

Use scrape_configs[x].relabel_debug = true to log metric names inkl.
labels before and after relabeling. After relabeling related metrics
get dropped, i.e. not submitted to servers.

* vminsert wants relabel logging, too.
 2021-06-04 20:33:21 +03:00
Nikolay
3d89c01d07 fixes solaris build (#1345) 2021-06-04 11:56:06 +03:00
Hason Chan
439c2ed510 fix eureka_sd_configs HTTPClientConfig incorrect parsing (#1350) 2021-06-04 11:56:06 +03:00
Aliaksandr Valialkin
fc2565b4ee lib/storage: reduce memory allocations when syncing dateMetricIDCache 2021-06-03 16:20:02 +03:00
Aliaksandr Valialkin
0b9f0de0a1 lib/promscrape: fix tests after f0c21b6300 2021-05-28 01:33:28 +03:00
Aliaksandr Valialkin
6865f3b497 Revert "lib/mergeset: remove a pool for inmemoryBlock structs" 
This reverts commit 793fe39921.

Reason to revert: production testing revealed possible slowdown when registering big number of new time series
 2021-05-28 01:11:22 +03:00
Aliaksandr Valialkin
7b33bc67a1 lib/mergeset: remove a pool for inmemoryBlock structs 
The pool for inmemoryBlock struct doesn't give any performance gains in production workloads,
while it may result in excess memory usage for inmemoryBlock structs inside the pool during
background merge of indexdb.
 2021-05-27 22:00:50 +03:00
Aliaksandr Valialkin
97de72054e docs: document f0c21b6300 2021-05-27 15:04:13 +03:00
faceair
b801b299f0 lib/promscrape: apply body size & sample limit to stream parse (#1331) 
* lib/promscrape: apply body size limit to stream parse

Signed-off-by: faceair <git@faceair.me>

* lib/promscrape: apply sample limit to stream parse

Signed-off-by: faceair <git@faceair.me>
 2021-05-27 15:04:11 +03:00
Aliaksandr Valialkin
49490ae5a7 lib/protoparser/clusternative: remove duplicate cannot read packet size phrase from the log message 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1336
 2021-05-27 12:09:17 +03:00