github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-11 14:53:49 +00:00
Code Issues Projects Releases Packages Wiki Activity
5649 commits 131 branches 758 tags 363 MiB
Commit graph

1876 commits

Author SHA1 Message Date
Aliaksandr Valialkin
22e48e6517
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files 2022-12-11 02:08:00 -08:00
Aliaksandr Valialkin
a30ae502ef
lib/promscrape: allow editing relabeling configs and labels at /target-relabel-debug page 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407
 2022-12-10 12:44:45 -08:00
Aliaksandr Valialkin
a8b8e23d68
lib/promscrape: implement target-level and metric-level relabel debugging 
Target-level debugging is performed by clicking the 'debug' link at the corresponding target
on either http://vmagent:8429/targets page or on http://vmagent:8428/service-discovery page.

Metric-level debugging is perfromed at http://vmagent:8429/metric-relabel-debug page.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407

See https://docs.victoriametrics.com/vmagent.html#relabel-debug
 2022-12-10 02:09:44 -08:00
Aliaksandr Valialkin
2406c0dcfd
docs/CHANGELOG.md: document the bugfix at 05b42601c3 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3247
 2022-12-08 18:35:28 -08:00
Zakhar Bessarab
05b42601c3
lib/promscrape/discovery/azure: remove API server from URL returned by azure (#3403) 
* lib/promscrape/discovery/azure: remove API server from URL returned by azure

* lib/promscrape/discovery/azure: validate nextLink contains same URL as apiServer
 2022-12-08 18:29:10 -08:00
Aliaksandr Valialkin
8434aa142d
lib/querytracer: fix remaining tests after 49ebc48809 2022-12-08 18:18:06 -08:00
Aliaksandr Valialkin
5b9e6b9d24
lib/storage: follow-up after 7c0ae3a86a 
- Update docs at https://docs.victoriametrics.com/#deduplication
- Optimize the deduplication loop a bit

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3333
 2022-12-08 18:16:57 -08:00
Roman Khavronenko
7c0ae3a86a
lib/storage: keep sample with the biggest value on timestamp conflict (#3421) 
The change leaves raw sample with the biggest value for identical
timestamps per each `-dedup.minScrapeInterval` discrete interval
when the deduplication is enabled.

```
benchstat old.txt new.txt
name                                         old time/op    new time/op    delta
DeduplicateSamples/minScrapeInterval=1s-10      817ns ± 2%     832ns ± 3%      ~     (p=0.052 n=10+10)
DeduplicateSamples/minScrapeInterval=2s-10     1.56µs ± 1%    2.12µs ± 0%   +35.19%  (p=0.000 n=9+7)
DeduplicateSamples/minScrapeInterval=5s-10     1.32µs ± 3%    1.65µs ± 2%   +25.57%  (p=0.000 n=10+10)
DeduplicateSamples/minScrapeInterval=10s-10    1.13µs ± 2%    1.50µs ± 1%   +32.85%  (p=0.000 n=10+10)

name                                         old speed      new speed      delta
DeduplicateSamples/minScrapeInterval=1s-10   10.0GB/s ± 2%   9.9GB/s ± 3%      ~     (p=0.052 n=10+10)
DeduplicateSamples/minScrapeInterval=2s-10   5.24GB/s ± 1%  3.87GB/s ± 0%   -26.03%  (p=0.000 n=9+7)
DeduplicateSamples/minScrapeInterval=5s-10   6.22GB/s ± 3%  4.96GB/s ± 2%   -20.37%  (p=0.000 n=10+10)
DeduplicateSamples/minScrapeInterval=10s-10  7.28GB/s ± 2%  5.48GB/s ± 1%   -24.74%  (p=0.000 n=10+10)
```

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3333
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2022-12-08 18:06:11 -08:00
Aliaksandr Valialkin
3019ec3da6
lib/querytracer: fix tests after 49ebc48809 2022-12-08 17:21:38 -08:00
Aliaksandr Valialkin
56b8980915
lib/promscrape: allow using sample_limit and series_limit options in stream parsing mode 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3458
 2022-12-08 16:33:38 -08:00
Aliaksandr Valialkin
49ebc48809
lib/querytracer: put the version of VictoriaMetrics in the first message of query trace 
This should simplify further debugging, since the first thing to start the debugging by query trace
is to know the version of VictoriaMetrics, which produced this trace.
 2022-12-07 09:46:39 -08:00
Aliaksandr Valialkin
1f0432b5c1
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files 2022-12-05 23:21:20 -08:00
Pedro Gonçalves
1e0666abb4
Datadog - Add device as a tag if it's present as a field in the series object (#3431) 
* Datadog - Add device as a tag if it's present as a field in the series object

* address PR comments
 2022-12-05 23:06:03 -08:00
Aliaksandr Valialkin
d99d222f0a
lib/{storage,mergeset}: log the duration for flushing in-memory parts on graceful shutdown 2022-12-05 21:30:48 -08:00
Aliaksandr Valialkin
8189770c50
all: add -inmemoryDataFlushInterval command-line flag for controlling the frequency of saving in-memory data to disk 
The main purpose of this command-line flag is to increase the lifetime of low-end flash storage
with the limited number of write operations it can perform. Such flash storage is usually
installed on Raspberry PI or similar appliances.

For example, `-inmemoryDataFlushInterval=1h` reduces the frequency of disk write operations
to up to once per hour if the ingested one-hour worth of data fits the limit for in-memory data.

The in-memory data is searchable in the same way as the data stored on disk.
VictoriaMetrics automatically flushes the in-memory data to disk on graceful shutdown via SIGINT signal.
The in-memory data is lost on unclean shutdown (hardware power loss, OOM crash, SIGKILL).

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337
 2022-12-05 15:16:14 -08:00
Aliaksandr Valialkin
544ea89f91
lib/{mergeset,storage}: add start background workers via startBackgroundWorkers() function 2022-12-04 00:01:04 -08:00
Aliaksandr Valialkin
33dda2809b
lib/mergeset: panic when too long item is passed to Table.AddItems() 2022-12-03 23:32:16 -08:00
Aliaksandr Valialkin
932c1f90ae
lib/storage: remove duplicate logging for filepath on errors 2022-12-03 23:15:22 -08:00
Aliaksandr Valialkin
044a304adb
lib/storage: pass a single arg - rowsPerBlock - to getCompressLevel() function instead of two args 2022-12-03 23:10:16 -08:00
Aliaksandr Valialkin
cb44976716
lib/{storage,mergeset}: use a single sync.WaitGroup for all background workers 
This simplifies the code
 2022-12-03 23:03:08 -08:00
Aliaksandr Valialkin
28e6d9e1ff
lib/storage: properly pass retentionMsecs to OpenStorage() at TestIndexDBRepopulateAfterRotation 2022-12-03 23:02:10 -08:00
Aliaksandr Valialkin
343c69fc15
lib/{mergeset,storage}: pass compressLevel to blockStreamWriter.InitFromInmemoryPart 
This allows packing in-memory blocks with different compression levels
depending on its contents. This may save memory usage.
 2022-12-03 22:46:48 -08:00
Aliaksandr Valialkin
6d87462f4b
lib/mergeset: use the given compressLevel for index and metaindex compression in in-memory part 
Previously only data was compressed with the given compressLevel
 2022-12-03 22:34:54 -08:00
Aliaksandr Valialkin
f3e3a3daeb
lib/{mergeset,storage}: take into account byte slice capacity when returning the size of in-memory part 
This results in more correct reporting of memory usage for in-memory parts
 2022-12-03 22:30:36 -08:00
Aliaksandr Valialkin
c4150995ad
lib/mergeset: reduce the time needed for the slowest tests 2022-12-03 22:26:33 -08:00
Aliaksandr Valialkin
45299efe22
lib/{storage,mergeset}: consistency rename: `flushRaw{Rows,Items} -> flushPending{Rows,Items} 2022-12-03 22:17:46 -08:00
Aliaksandr Valialkin
5ca58cc4fb
lib/storage: optimization: do not scan block for rows outside retention if it is covered by the retention 2022-12-03 22:14:12 -08:00
Aliaksandr Valialkin
152ac564ab
lib/storage: remove logging redundant path values in a single error message 2022-12-03 22:13:13 -08:00
Aliaksandr Valialkin
93764746c2
lib/filestream: remove logging redundant path values in a single error message 2022-12-03 22:01:51 -08:00
Aliaksandr Valialkin
4f28513b1a
lib/fs: remove logging redundant path values in a single error message 2022-12-03 22:00:20 -08:00
Aliaksandr Valialkin
7c3c08d102
lib/backup: remove logging duplicate path values in a single error message 2022-12-03 21:55:06 -08:00
Aliaksandr Valialkin
14660d4df5
all: typo fix: the the -> the 2022-12-03 21:53:01 -08:00
Aliaksandr Valialkin
ddc3d6b5c3
lib/mergeset: drop the crufty code responsible for direct upgrade from releases prior v1.28.0 
Upgrade to v1.84.0, wait until the "finished round 2 of background conversion" message
appears in the log and then upgrade to newer release.
 2022-12-03 21:17:31 -08:00
Aliaksandr Valialkin
05c65bd83f
lib/storage: speed up search for data block for the given tsids 
Use binary search instead of linear scan for looking up the needed
data block inside index block.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3425
 2022-12-03 20:58:32 -08:00
Aliaksandr Valialkin
079953b4ea
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files 2022-12-02 19:18:34 -08:00
Aliaksandr Valialkin
299285b147
lib/storage: fix TestUpdateCurrHourMetricIDs test when it runs on the first hour of the day by UTC 2022-12-02 18:52:37 -08:00
Aliaksandr Valialkin
e9636b4c69
lib/{mergeset,storage}: re-use the code for removing isInMerge flag at parts 
Move the common code into releasePartsToMerge() method and consistently use it throughout the code.
 2022-12-02 18:52:37 -08:00
Aliaksandr Valialkin
d92da32041
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files 2022-11-29 21:47:16 -08:00
Aliaksandr Valialkin
f325410c26
lib/promscrape: optimize service discovery speed 
- Return meta-labels for the discovered targets via promutils.Labels
  instead of map[string]string. This improves the speed of generating
  meta-labels for discovered targets by up to 5x.

- Remove memory allocations in hot paths during ScrapeWork generation.
  The ScrapeWork contains scrape settings for a single discovered target.
  This improves the service discovery speed by up to 2x.
 2022-11-29 21:26:00 -08:00
Aliaksandr Valialkin
295c84df66
lib/promscrape/discovery: add a benchmark for measuring the performance of creating pod meta-labels 2022-11-29 20:27:48 -08:00
Aliaksandr Valialkin
654e94f420
lib/promscrape: add exported_ prefix to metric names exported by scrape targets if they clash with automatically generated metrics 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3406
 2022-11-28 18:37:09 -08:00
匠心零度
fa0ce10275
lib/storage: remove extra error check (#3396) 2022-11-28 16:43:31 -08:00
Aliaksandr Valialkin
8548650c2d
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files 2022-11-25 20:13:43 -08:00
Aliaksandr Valialkin
58d459e8a8
app/{vminsert,vmagent}: follow-up after 53a63c6c4c 
Extend /api/v1/import/prometheus with the support for Pushgateway way of specifying additional labels.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1415
 2022-11-25 16:48:14 -08:00
Roman Khavronenko
03d88bc066
vmagent: expose metrics for tracking config state (#3375) 
Expose `vm_relabel_config_*` and `vm_promscrape_config_*` metrics
for tracking relabel and scrape configuration hot-reloads.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3345
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
 2022-11-22 00:38:43 +02:00
Aliaksandr Valialkin
95f0266558
lib/promscrape/discovery/gce: do not pass filter arg when discovering zones 
The filter arg isn't supported by zones API in GCE.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3202
 2022-11-21 22:32:05 +02:00
Aliaksandr Valialkin
353396aa23
lib/workingsetcache: expose -cacheExpireDuration command-line flag for fine-tuning of the cache expiration 
While at it, decrease -prevCacheRemovalPercent from 0.2 to 0.1 and increase -cacheExpireDuration from 20 minutes to 30 minutes.

This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343
 2022-11-17 19:59:13 +02:00
Aliaksandr Valialkin
2dd82e8355
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files 2022-11-17 01:33:16 +02:00
Aliaksandr Valialkin
5955d23232
lib/promscrape: add a benchmark for internLabelStrings() 2022-11-16 23:02:49 +02:00
Aliaksandr Valialkin
a75137c1c2
lib/mergeset: properly reset bsr.bhIdx after the call to blockStreamReader.readNextBHS() 
The issue has been introduced in 58b40f514c

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343
 2022-11-16 21:23:35 +02:00