Aliaksandr Valialkin
6e390f3b99
lib/querytracer: fix remaining tests after 49ebc48809
2022-12-08 18:18:50 -08:00
Aliaksandr Valialkin
e56d5e1918
lib/storage: follow-up after 7c0ae3a86a
...
- Update docs at https://docs.victoriametrics.com/#deduplication
- Optimize the deduplication loop a bit
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3333
2022-12-08 18:18:49 -08:00
Roman Khavronenko
909cd04c55
lib/storage: keep sample with the biggest value on timestamp conflict ( #3421 )
...
The change leaves raw sample with the biggest value for identical
timestamps per each `-dedup.minScrapeInterval` discrete interval
when the deduplication is enabled.
```
benchstat old.txt new.txt
name old time/op new time/op delta
DeduplicateSamples/minScrapeInterval=1s-10 817ns ± 2% 832ns ± 3% ~ (p=0.052 n=10+10)
DeduplicateSamples/minScrapeInterval=2s-10 1.56µs ± 1% 2.12µs ± 0% +35.19% (p=0.000 n=9+7)
DeduplicateSamples/minScrapeInterval=5s-10 1.32µs ± 3% 1.65µs ± 2% +25.57% (p=0.000 n=10+10)
DeduplicateSamples/minScrapeInterval=10s-10 1.13µs ± 2% 1.50µs ± 1% +32.85% (p=0.000 n=10+10)
name old speed new speed delta
DeduplicateSamples/minScrapeInterval=1s-10 10.0GB/s ± 2% 9.9GB/s ± 3% ~ (p=0.052 n=10+10)
DeduplicateSamples/minScrapeInterval=2s-10 5.24GB/s ± 1% 3.87GB/s ± 0% -26.03% (p=0.000 n=9+7)
DeduplicateSamples/minScrapeInterval=5s-10 6.22GB/s ± 3% 4.96GB/s ± 2% -20.37% (p=0.000 n=10+10)
DeduplicateSamples/minScrapeInterval=10s-10 7.28GB/s ± 2% 5.48GB/s ± 1% -24.74% (p=0.000 n=10+10)
```
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3333
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-08 18:18:36 -08:00
Aliaksandr Valialkin
323fc14e0a
lib/querytracer: fix tests after 49ebc48809
2022-12-08 17:21:47 -08:00
Aliaksandr Valialkin
a13f16e48a
lib/promscrape: allow using sample_limit and series_limit options in stream parsing mode
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3458
2022-12-08 17:04:12 -08:00
Aliaksandr Valialkin
255c04bc20
lib/querytracer: put the version of VictoriaMetrics in the first message of query trace
...
This should simplify further debugging, since the first thing to start the debugging by query trace
is to know the version of VictoriaMetrics, which produced this trace.
2022-12-07 09:49:51 -08:00
Pedro Gonçalves
84cfe4fcaf
Datadog - Add device as a tag if it's present as a field in the series object ( #3431 )
...
* Datadog - Add device as a tag if it's present as a field in the series object
* address PR comments
2022-12-05 23:10:42 -08:00
Aliaksandr Valialkin
0a9992a9c6
lib/{storage,mergeset}: log the duration for flushing in-memory parts on graceful shutdown
2022-12-05 21:55:21 -08:00
Aliaksandr Valialkin
7d5c64eb7a
all: add -inmemoryDataFlushInterval command-line flag for controlling the frequency of saving in-memory data to disk
...
The main purpose of this command-line flag is to increase the lifetime of low-end flash storage
with the limited number of write operations it can perform. Such flash storage is usually
installed on Raspberry PI or similar appliances.
For example, `-inmemoryDataFlushInterval=1h` reduces the frequency of disk write operations
to up to once per hour if the ingested one-hour worth of data fits the limit for in-memory data.
The in-memory data is searchable in the same way as the data stored on disk.
VictoriaMetrics automatically flushes the in-memory data to disk on graceful shutdown via SIGINT signal.
The in-memory data is lost on unclean shutdown (hardware power loss, OOM crash, SIGKILL).
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337
2022-12-05 15:28:09 -08:00
Aliaksandr Valialkin
9ac1174493
lib/{mergeset,storage}: add start background workers via startBackgroundWorkers() function
2022-12-04 00:01:14 -08:00
Aliaksandr Valialkin
a13d21513e
lib/mergeset: panic when too long item is passed to Table.AddItems()
2022-12-03 23:37:20 -08:00
Aliaksandr Valialkin
dccd70ce10
lib/storage: remove duplicate logging for filepath on errors
2022-12-03 23:15:28 -08:00
Aliaksandr Valialkin
813e8402f6
lib/storage: pass a single arg - rowsPerBlock - to getCompressLevel() function instead of two args
2022-12-03 23:10:26 -08:00
Aliaksandr Valialkin
bb93494eac
lib/{storage,mergeset}: use a single sync.WaitGroup for all background workers
...
This simplifies the code
2022-12-03 23:03:32 -08:00
Aliaksandr Valialkin
106332cd9f
lib/storage: properly pass retentionMsecs to OpenStorage() at TestIndexDBRepopulateAfterRotation
2022-12-03 23:03:30 -08:00
Aliaksandr Valialkin
ea55c16422
lib/{mergeset,storage}: pass compressLevel to blockStreamWriter.InitFromInmemoryPart
...
This allows packing in-memory blocks with different compression levels
depending on its contents. This may save memory usage.
2022-12-03 22:47:06 -08:00
Aliaksandr Valialkin
eca7f32151
lib/mergeset: use the given compressLevel for index and metaindex compression in in-memory part
...
Previously only data was compressed with the given compressLevel
2022-12-03 22:35:16 -08:00
Aliaksandr Valialkin
7ffa66d249
lib/{mergeset,storage}: take into account byte slice capacity when returning the size of in-memory part
...
This results in more correct reporting of memory usage for in-memory parts
2022-12-03 22:31:34 -08:00
Aliaksandr Valialkin
886ce94739
lib/mergeset: reduce the time needed for the slowest tests
2022-12-03 22:26:46 -08:00
Aliaksandr Valialkin
10a17bfa16
lib/{storage,mergeset}: consistency rename: `flushRaw{Rows,Items} -> flushPending{Rows,Items}
2022-12-03 22:18:05 -08:00
Aliaksandr Valialkin
233301a549
lib/storage: optimization: do not scan block for rows outside retention if it is covered by the retention
2022-12-03 22:14:20 -08:00
Aliaksandr Valialkin
fd9d0a550b
lib/storage: remove logging redundant path values in a single error message
2022-12-03 22:14:19 -08:00
Aliaksandr Valialkin
82f64072d2
lib/filestream: remove logging redundant path values in a single error message
2022-12-03 22:02:04 -08:00
Aliaksandr Valialkin
81400c80f0
lib/fs: remove logging redundant path values in a single error message
2022-12-03 22:00:43 -08:00
Aliaksandr Valialkin
dc890ab80c
lib/backup: remove logging duplicate path values in a single error message
2022-12-03 21:55:12 -08:00
Aliaksandr Valialkin
6910b1de2e
all: typo fix: the the -> the
2022-12-03 21:53:07 -08:00
Aliaksandr Valialkin
0b8e7deabd
lib/mergeset: drop the crufty code responsible for direct upgrade from releases prior v1.28.0
...
Upgrade to v1.84.0, wait until the "finished round 2 of background conversion" message
appears in the log and then upgrade to newer release.
2022-12-03 21:18:41 -08:00
Aliaksandr Valialkin
8e9822bc7f
lib/storage: speed up search for data block for the given tsids
...
Use binary search instead of linear scan for looking up the needed
data block inside index block.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3425
2022-12-03 20:59:59 -08:00
Aliaksandr Valialkin
d8303845ef
lib/storage: fix TestUpdateCurrHourMetricIDs test when it runs on the first hour of the day by UTC
2022-12-02 17:23:59 -08:00
Aliaksandr Valialkin
d8d4d21d7a
lib/{mergeset,storage}: re-use the code for removing isInMerge flag at parts
...
Move the common code into releasePartsToMerge() method and consistently use it throughout the code.
2022-12-02 17:07:52 -08:00
Aliaksandr Valialkin
be6da5053f
lib/promscrape: optimize service discovery speed
...
- Return meta-labels for the discovered targets via promutils.Labels
instead of map[string]string. This improves the speed of generating
meta-labels for discovered targets by up to 5x.
- Remove memory allocations in hot paths during ScrapeWork generation.
The ScrapeWork contains scrape settings for a single discovered target.
This improves the service discovery speed by up to 2x.
2022-11-29 21:26:23 -08:00
Aliaksandr Valialkin
2524d94fe1
lib/promscrape/discovery: add a benchmark for measuring the performance of creating pod meta-labels
2022-11-29 21:11:03 -08:00
Aliaksandr Valialkin
027ab74efb
lib/httpserver: link to url format docs in error message emitted on parse error for the provided url path
...
This should help users identifying and fixing improperly set up urls.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3402
2022-11-28 18:49:03 -08:00
Aliaksandr Valialkin
8ce5b095b7
lib/promscrape: add exported_ prefix to metric names exported by scrape targets if they clash with automatically generated metrics
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3406
2022-11-28 18:37:34 -08:00
匠心零度
d4808d5b84
lib/storage: remove extra error check ( #3396 )
2022-11-28 17:07:11 +01:00
Aliaksandr Valialkin
c5eebaffd8
app/{vminsert,vmagent}: follow-up after 53a63c6c4c
...
Extend /api/v1/import/prometheus with the support for Pushgateway way of specifying additional labels.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1415
2022-11-25 16:42:38 -08:00
Zakhar Bessarab
e407e7243a
{app/vmstorage,app/vmselect}: add API to get list of existing tenants ( #3348 )
...
* {app/vmstorage,app/vmselect}: add API to get list of existing tenants
* {app/vmstorage,app/vmselect}: add API to get list of existing tenants
* app/vmselect: fix error message
* {app/vmstorage,app/vmselect}: fix error messages
* app/vmselect: change log level for error handling
* wip
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-11-25 10:32:45 -08:00
Roman Khavronenko
d1169c1559
vmagent: expose metrics for tracking config state ( #3375 )
...
Expose `vm_relabel_config_*` and `vm_promscrape_config_*` metrics
for tracking relabel and scrape configuration hot-reloads.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3345
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-11-22 00:48:12 +02:00
Aliaksandr Valialkin
c33bcae457
lib/promscrape/discovery/gce: do not pass filter arg when discovering zones
...
The filter arg isn't supported by zones API in GCE.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3202
2022-11-21 22:32:45 +02:00
Aliaksandr Valialkin
48dda17e36
lib/workingsetcache: expose -cacheExpireDuration command-line flag for fine-tuning of the cache expiration
...
While at it, decrease -prevCacheRemovalPercent from 0.2 to 0.1 and increase -cacheExpireDuration from 20 minutes to 30 minutes.
This is needed for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343
2022-11-17 21:55:11 +02:00
Aliaksandr Valialkin
047fe3ee67
lib/promscrape: add a benchmark for internLabelStrings()
2022-11-16 23:02:41 +02:00
Aliaksandr Valialkin
b0b8f05fa4
lib/mergeset: properly reset bsr.bhIdx after the call to blockStreamReader.readNextBHS()
...
The issue has been introduced in 58b40f514chttps://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343
2022-11-16 21:22:51 +02:00
Aliaksandr Valialkin
c8d7e1312c
lib/workingsetcache: add -prevCacheRemovalPercent command-line flag for tuning memory usage vs CPU usage ratio
...
Reduce the default value of this flag from 1% to 0.2% after 71335e6024https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343
2022-11-16 12:41:37 +02:00
Aliaksandr Valialkin
06300fe9b8
lib/mergeset: retain the buffer with the data used by indexBlock.bhs, inside indexBlock.buf
...
Previously indexBlock.bhs pointed to the buffer, which could be changed over time.
This could result in incorrect time series search over time.
This is a follow-up for 58b40f514chttps://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343
2022-11-16 12:10:15 +02:00
Aliaksandr Valialkin
454060fd78
lib/mergeset: remove string allocation and copying when unmarshaling blockHeader
...
This should reduce CPU usage for the case from https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3343
2022-11-16 12:10:14 +02:00
Aliaksandr Valialkin
a5a0fa75bf
lib/workingsetcache: tune cache miss threshold for resetting the previous cache from 5% to 1%
...
It has been appeared that some production workloads could suffer for some time
after every reset of the previous cache when it gets less than 5% of requests
after the needed item isn't found in the current cache. This could result
in reduced cache hit rates, which, in turn, could increase CPU, disk IO and RAM
usage needed for reading, unpacking and caching the missed data from disk.
This commit reduces the cache miss threshold for resetting the previous cache from 5% to 1%.
This should reduce the possible negative impact after each cache reset by at least 5x,
while reducing the total memory used by caches.
This is a follow-up for d906d8573e
2022-11-10 12:38:53 +02:00
Aliaksandr Valialkin
24213eaeba
lib/promscrape: add more cases to TestAddRowToTimeseries
...
This is a follow-up for 16fdd2af8a
2022-11-09 16:15:32 +02:00
Jeremy PLANCKEEL
87375b004a
test(golang): add test to function addRowToTimeseries ( #3282 )
...
Co-authored-by: jplanckeel-externe <jplanckeel.externe@bedrockstreaming.com>
2022-11-09 16:15:30 +02:00
Aliaksandr Valialkin
2091693f16
lib/protoparser/opentsdb: follow-up after 04b0e4e7bf
...
- Simplify the parser code to be less error prone
- Document the change
- Add a test for OpenTSDB put line with trailing whitespace without tags
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3290
2022-11-09 15:36:15 +02:00
Roman Khavronenko
71dfe4d697
protoparser/opentsdb: allow lines without tags ( #3303 )
...
According to http://opentsdb.net/docs/build/html/api_telnet/put.html
"At least one tag pair must be present".
However, in VictoriaMetrics datamodel tags aren't required.
This could be confusing for users. Allowing accept lines without
tags seems to do no harm.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3290
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-11-09 15:36:13 +02:00
