github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-01 14:47:38 +00:00
Code Issues Projects Releases Packages Wiki Activity
4222 commits 119 branches 748 tags 355 MiB
Commit graph

1428 commits

Author SHA1 Message Date
Aliaksandr Valialkin
727b29d4a3
lib/promscrape/discovery/kubernetes: add __meta_kubernetes_endpointslice_{label,annotation}* labels to be consistent with other role values for Kubernetes service discovery 2022-02-11 14:56:10 +02:00
Nikolay
265938a385
fixes service discovery for kubernetes (#2173) 
* fixes service discovery for kubernetes
now it must take in account all pods that belong to the discovered endpoint and endpointslice
adds simple test for endpoints
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2134

* wip

* docs/CHANGELOG.md: document the change

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-02-11 13:35:34 +02:00
Aliaksandr Valialkin
895c9f4f11
lib/mergeset: tune indexdb/{indexBlocks,dataBlocks} cache sizes further according to production stats 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
 2022-02-10 19:11:13 +02:00
Aliaksandr Valialkin
102c9a4bf9
lib/blockcache: use higher number of shards for higher number of CPU cores 
This should reduce mutex contention and increase performance

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
 2022-02-10 19:11:11 +02:00
Aliaksandr Valialkin
ad19c9d302
lib/promscrape: fix errors in test config 
The errors were discovered after enabling strict parse mode by default.
See 9bb60ab00f
 2022-02-08 20:10:28 +02:00
Aliaksandr Valialkin
fae3040868
lib/blockcache: split the cache into multiple shards 
This should reduce contention on cache mutex on hosts with many CPU cores,
which, in turn, should increase overall throughput for the cache.

This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
 2022-02-08 19:48:32 +02:00
Aliaksandr Valialkin
a0a56d6c1c
lib/mergeset: tune sizes for indexdb/dataBlocks and indexdb/indexBlocks according to production workload 
This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007#issuecomment-1032308742
 2022-02-08 18:04:03 +02:00
Aliaksandr Valialkin
eed66b6640
lib/promscrape: set -promscrape.config.strictParse to true by default 
This allows detecting long-living silent errors in -promscrape.config
 2022-02-08 15:42:33 +02:00
Aliaksandr Valialkin
8863339b6b
lib/blockcache: make fmt 2022-02-08 15:42:31 +02:00
Aliaksandr Valialkin
1caee74235
lib/blockcache: eliminate possible race when Cache.Put is called for the same entry from multiple goroutines 
The race could result in incorrect cache size tracking, which, in turn, could result in too frequent cache cleaning
 2022-02-08 01:18:27 +02:00
Aliaksandr Valialkin
10476738a8
lib/blockcache: increase the lifetime for rarely accessed blocks from 2 minutes to 5 minutes 
This should improve data ingestion speed if time series samples are ingested with interval bigger than 2 minutes.
The actual interval could exceed 2 minutes if the original interval between samples doesn't exceed 2 minutes
in the case of slow inserts. Slow inserts may appear in the following cases:

* Big number of new time series are pushed to VictoriaMetrics, so they couldn't be registered in 2 minutes.
* MetricName->tsid cache reset on indexdb rotation or due to unclean shutdown.
  In this case VictoriaMetrics needs to load MetricName->tsid entries for all the incoming series from IndexDB.
  IndexDB uses the block cache for increasing lookup performance. If the cache has no the needed block,
  then IndexDB reads and unpacks the block from disk. This requires an extra disk read IO and CPU.
  See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007

This also should increase performance for periodically executed queries with intervals from 2 minutes to 5 minutes.
See the previous similar commit - 43103be011

It is possible that the timeout can be increased further. Let's collect production numbers for this change
so the timeout could be adjusted further.
 2022-02-08 01:18:27 +02:00
Aliaksandr Valialkin
b7cefff7b0
lib/workingsetcache: use the original cache size limits when rotating caches 
Previously limits for new caches were taken from cache stats.
These limits could mismatch the original limits. This could result in failed cache load
if the stored cache has been created with the limits obtained from cache stats.
 2022-02-08 01:18:27 +02:00
Aliaksandr Valialkin
87071640a7
lib/blockcache: return proper number of entries from the cache 
This has been broken in 0d7374ad2f
 2022-02-08 01:18:27 +02:00
Aliaksandr Valialkin
34d14c4940
all: substitute zeroTime with time.Time{}, since this generates more optimal binary code 2022-02-07 14:36:41 +02:00
Aliaksandr Valialkin
e2d12a25e0
lib/netutil: increase dial timeout from 1 second to 5 seconds 
There are real-world cases when TCP connection needs more than 1 second to be established.
 2022-02-07 12:33:40 +02:00
Aliaksandr Valialkin
d24e5d9efd
lib/promscrape: show the total number of scrapes and the total number of scrape errors per target at /targets page 
This information may be useful when debugging unreliable scrape targets
 2022-02-03 20:23:27 +02:00
Aliaksandr Valialkin
678b3e71db
lib/promscrape: provide the ability to fetch target responses on behalf of vmagent or single-node VictoriaMetrics 
This feature may be useful when debugging metrics for the given target located in isolated environment
 2022-02-03 19:02:12 +02:00
Aliaksandr Valialkin
5f266370c5
all: follow-up after 4bdd10ab90 
Properly use new bytesutil.Resize* functions
 2022-02-01 17:49:28 +02:00
Aliaksandr Valialkin
d8d59ff760
lib/mergeset: pre-allocate data and items for inmemoryBlock in order to reduce memory allocations under high churn rate 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
 2022-02-01 11:20:20 +02:00
Aliaksandr Valialkin
02b2bfcff3
lib/bytesutil: split Resize* funcs to MayOverallocate and NoOverallocate for more fine-grained control over memory allocations 
Follow-up for f4989edd96

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
 2022-02-01 11:20:20 +02:00
Aliaksandr Valialkin
084664d780
lib/encoding: substitute 64-bits.LeadingZeros64() with bits.Len64() 2022-02-01 11:20:20 +02:00
Aliaksandr Valialkin
0fbfa8c245
lib/storage: avoid allocations of tsidPrev on every blockStreamReader.NextBlock() call 
This is a follow-up for 00b7c97d2a

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2082
 2022-01-31 22:47:16 +02:00
Aliaksandr Valialkin
a02dde6cc7
lib/cgroup: fall back to runtime.NumCPU() when determining process_cpu_cores_available metric if it is impossible to determine cpu quota via cgroups 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2107
 2022-01-31 20:31:12 +02:00
Aliaksandr Valialkin
566e12874d
lib/cgroup: expose process_cpu_cores_available metric 
This metric shows the number of CPU cores available to the process.
This allows creating alerting rules on CPU saturation with the following query:

    rate(process_cpu_seconds_total[5m]) / process_cpu_cores_available > 0.9

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2107
 2022-01-31 20:25:15 +02:00
Aliaksandr Valialkin
776b7bc9f8
lib/storage/table.go: add missing tb.ptwsLock.Unlock() before the return 
This is a follow-up for a1083d0531

See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2103
 2022-01-28 12:12:58 +02:00
匠心零度
a1083d0531
optimized code (#2103) 
* optimized code ,because only the first error,so no need var errors []error

* optimized code ,because only the first error,so no need var errors []error

Co-authored-by: lirenzuo <lirenzuo@shein.com>
 2022-01-28 12:10:47 +02:00
Aliaksandr Valialkin
6232eaa938
lib/bytesutil: split Resize() into ResizeNoCopy() and ResizeWithCopy() functions 
Previously bytesutil.Resize() was copying the original byte slice contents to a newly allocated slice.
This wasted CPU cycles and memory bandwidth in some places, where the original slice contents wasn't needed
after slize resizing. Switch such places to bytesutil.ResizeNoCopy().

Rename the original bytesutil.Resize() function to bytesutil.ResizeWithCopy() for the sake of improved readability.

Additionally, allocate new slice with `make()` instead of `append()`. This guarantees that the capacity of the allocated slice
exactly matches the requested size. The `append()` could return a slice with bigger capacity as an optimization for further `append()` calls.
This could result in excess memory usage when the returned byte slice was cached (for instance, in lib/blockcache).

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
 2022-01-25 15:28:42 +02:00
Aliaksandr Valialkin
7ec0705b98
lib/mergeset: allocate the needed amounts of memory when unmarshaling inmemoryBlock 
This should reduce the memory required for indexdb/dataBlocks cache.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
 2022-01-24 18:52:22 +02:00
Aliaksandr Valialkin
65176726b3
lib/logger: removed broken test after 746ee191e8 2022-01-24 12:15:11 +02:00
Aliaksandr Valialkin
49650fe6aa
lib/logger/throttler.go: show the original location of the error and warning message 
Previously the location inside LogThrottler implementation was shown. This could complicate debugging.
 2022-01-23 13:55:48 +02:00
Aliaksandr Valialkin
233101137d
lib/blockcache: optimize blockcache a bit 
- Optimize Cache.RemoveBlocksFromPart(), so it doesn't need to iterate over all the cached blocks.
- Cache blocks if there were no cache misses during the last 2 minutes.
  This may be the case when new blocks are added simultaneously to the storage and to the cache.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
 2022-01-23 13:08:55 +02:00
Aliaksandr Valialkin
9edf407144
lib/mergeset: tune caches size limits for indexdb/dataBlocks and indexdb/indexBlocks 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
 2022-01-21 12:46:05 +02:00
Aliaksandr Valialkin
4e05298756
lib/storage: properly limit cardinality when ingesting multiple samples for the same time series in a single request 2022-01-21 12:38:22 +02:00
Aliaksandr Valialkin
e3277918e4
lib/storage: verify that blocks in a single part are sorted by TSID when reading sequential blocks from the part 
This may help narrowing down the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2082
 2022-01-20 20:37:28 +02:00
Aliaksandr Valialkin
54ee71e16d
lib/storage: set bsm.Block to nil on error, so the previous block couldn't be used. 
This may help nailing down the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2082
 2022-01-20 20:37:24 +02:00
Aliaksandr Valialkin
5159a9451f
lib/blockcache: add missing dependency after 145337792d 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
 2022-01-20 18:51:03 +02:00
Aliaksandr Valialkin
6ae584b9b3
lib/{mergeset,storage}: properly limit cache sizes for indexdb 
Previously these caches could exceed limits set via `-memory.allowedPercent` and/or `-memory.allowedBytes`,
since limits were set independently per each data part. If the number of data parts was big, then limits could be exceeded,
which could result to out of memory errors.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007
 2022-01-20 18:45:03 +02:00
Aliaksandr Valialkin
da95516a1f
lib/promscrape: expose promscrape_stale_samples_created_total metric for monitoring the number of created stale samples 2022-01-14 01:00:40 +02:00
Aliaksandr Valialkin
dd91759f1f
lib/promscrape/discovery/kubernetes: add __meta_kubernetes_node_provider_id label for discovered Kubernetes nodes in the same way as Prometheus does 
See https://github.com/prometheus/prometheus/pull/9603
 2022-01-13 23:17:24 +02:00
Aliaksandr Valialkin
bc18368c15
lib/promscrape/discovery/kubernetes: add the ability to limit service discovery to the current namespace 
See https://github.com/prometheus/prometheus/issues/9782 and https://github.com/prometheus/prometheus/pull/9881
 2022-01-13 22:44:59 +02:00
Aliaksandr Valialkin
de8299f465
lib/promscrape/discovery/dockerswarm: follow up after 68a117a25a 
- Document the bugfix at docs/CHANGELOG.md
- Set __address__ field after copying commonLabels to the resulting map of discovered labels.
  This makes sure that the correct __address__ label is used.
 2022-01-11 09:22:03 +02:00
Alexander Shtuchkin
45a92e6ce1
Fix for #2038: Make correct __address__ value for dockerswarm promscrape (#2041) 2022-01-11 09:22:02 +02:00
Aliaksandr Valialkin
fa89f3e5a5
lib/promscrape: do not send staleness markers on graceful shutdown 
This follows Prometheus behavior.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2013#issuecomment-1006994079
 2022-01-07 01:19:06 +02:00
Aliaksandr Valialkin
80fc3fda07
lib/storage: follow-up for 38bf5fc136 2022-01-05 16:02:17 +02:00
weng zhao
1e0fe615ad
vmstorage: fix query like {foo=~"bar|"} return extra timeseries cause by negative filter transformation malfunction (#2032) 
1. L2749 make kb.B remain the value of comonPrefix instead of tf.prefix
2. L2762 avoid change tf.value from "bar|" to ".+r|"
 2022-01-05 15:57:54 +02:00
Aliaksandr Valialkin
c1722003a2
lib/promscrape: scrape replicated targets at different offsets in vmagent replicated clustering mode 
This guarantees that the deduplication consistently leaves samples from the same vmagent replica.

See https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets
 2021-12-23 00:21:41 +02:00
Nikolay
6cdc934c3d
adds restore.lock (#1988) 
* adds restore.lock
it must prevent from running storage after incomplete restore process
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1958

* return back flock file deletion

* Apply suggestions from code review

* wip

* docs/CHANGELOG.md: document https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1958

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2021-12-22 13:10:56 +02:00
Aliaksandr Valialkin
727797a6fd
all: use logger.WithThrottler() where appropriate 2021-12-21 17:10:54 +02:00
Aliaksandr Valialkin
4dbf12254d
lib/promscrape: take into account the original job_name when creating an unique key per each scrape target 
This should handle the case when the original job_name has been changed in -promscrape.config ,
while the resulting job label remains the same because it is overriden via relabeling.
 2021-12-21 16:42:42 +02:00
Roman Khavronenko
23e1de06ee
vmagent: add error log for skipped data block when rejected by receiv… (#1956) 
* vmagent: add error log for skipped data block when rejected by receiving side

Previously, rejected data blocks were silently dropped - only metrics were update.
From operational perspective, having an additional logging for such cases is preferable.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1911

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmagent: throttle log messages about skipped blocks

The new type of logger was added to logger pacakge.
This new type supposed to control number of logged messages
by time.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* lib/logger: make LogThrottler public, so its methods can be inspected by external packages

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2021-12-21 16:42:38 +02:00
Aliaksandr Valialkin
053e85ff3d
all: typo fix: unexected -> unexpected 2021-12-20 17:40:13 +02:00
Aliaksandr Valialkin
406cb06f8c
lib/persistentqueue: check that readerOffset doesnt exceed writerOffset after each readerOffset increase 
This should help detecting the source of the panic from https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1981
 2021-12-20 17:26:07 +02:00
Aliaksandr Valialkin
f22aab411b
lib/storage: properly update per-part min_dedup_interval file contents after merge 
Previously 0s was always written even if -dedup.minScrapeInterval was set to non-zero value

This is a follow-up for 4ff647137a
 2021-12-17 20:12:18 +02:00
Aliaksandr Valialkin
5bd4e47a9e
lib/promscrape: allow up to 5 redirects when scraping a target by default 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1945
 2021-12-16 00:14:45 +02:00
Aliaksandr Valialkin
d36fdbe537
lib/storage: deduplicate samples more thoroughly 
Previously some duplicate samples may be left on disk for time series with high churn rate.
This may result in higher disk space usage.
 2021-12-15 16:00:30 +02:00
Aliaksandr Valialkin
bc3923111b
lib/storage: return dedup interval in milliseconds from GetDedupInterval() 
This removes duplicate .Milliseconds() calls after GetDedupInterval() calls.
 2021-12-15 13:27:27 +02:00
Aliaksandr Valialkin
cdfe854c9b
lib/storage: explicitly pass dedupInterval to DeduplicateSamples() and deduplicateSamplesDuringMerge() 
This improves the code readability and debuggability, since the output of these functions
stops depending on global state.
 2021-12-14 20:52:29 +02:00
Aliaksandr Valialkin
c922c7af9a
lib/storage: convert alternate regexps into Graphite wildcards inside __graphite__ pseudo-label 
For example, `{__graphite__=~"foo.(bar|baz)"}` is automatically converted to `{__graphite__=~"foo.{bar,baz}"}` before execution.
This allows using multi-value Grafana template variables such as `{__graphite__=~"foo.($app)"}`.
 2021-12-14 19:55:59 +02:00
Aliaksandr Valialkin
38f5bc7451
lib/httpserver: add missing 127.0.0.1 hostname to the logged address for http and pprof server if the address starts with ':' 
This allows copy-pasting the url to http server from logs.
 2021-12-08 16:15:12 +02:00
Aliaksandr Valialkin
9aa9b081a4
app/vminsert: add -maxLabelValueLen command-line flag 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1908
 2021-12-06 11:42:24 +02:00
Aliaksandr Valialkin
51f2eb3c46
lib/workingsetcache: fix unaligned 64-bit atomic operation panic on 32-bit architectures 
The panic has been introduced in 7275ebf91a
 2021-12-03 01:22:30 +02:00
Aliaksandr Valialkin
d40441947a
app: allow specifying http and https urls in the following command-line flags 
* -promscrape.config
* -relabelConfig
* -remoteWrite.relabelConfig
* -remoteWrite.urlRelabelConfig
 2021-12-03 00:11:47 +02:00
Aliaksandr Valialkin
daaea1eb2c
app/vmauth: follow-up for 13368bed18 
* Document the ability to specify http or https urls in `-auth.config` at docs/CHANGELOG.md
* Move the ReadFileOrHTTP to lib/fs, so it can be re-used in other places where a file
  should be read from the given path. For example, in `-promscrape.config` at `vmagent`.
 2021-12-02 23:34:15 +02:00
Aliaksandr Valialkin
b885a3b6e9
lib/httpserver: expose /-/healthy and /-/ready endpoints as Prometheus does 
This improves integration with third-party solutions, which rely on these endpoints.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1833
 2021-12-02 14:37:50 +02:00
Aliaksandr Valialkin
c540235470
app: use relative paths instead of absolute paths for the supported http handlers on the main page 
This allows hiding VictoriaMetrics components behind proxies, which serve pages at different path prefixes

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1858
 2021-12-02 13:54:15 +02:00
Aliaksandr Valialkin
d1289383eb
lib/protoparser/graphite: allow multiple separators between metric name, value and timestamp 2021-12-02 13:44:01 +02:00
Aliaksandr Valialkin
37a2bea072
lib/protoparser/graphite: properly parse Graphite line with whitespace after the timestamp 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1865
 2021-12-02 13:33:50 +02:00
Aliaksandr Valialkin
2b7dee15dd
app/{vmbackup,vmrestore}: export internal metrics at /metrics http handler 2021-12-02 11:56:34 +02:00
Aliaksandr Valialkin
ab4be24397
app/vmstorage: export vm_cache_size_max_bytes metrics for determining capacity of various caches 
The vm_cache_size_max_bytes metric can be used for determining caches which reach their capacity via the following query:

   vm_cache_size_bytes / vm_cache_size_max_bytes > 0.9
 2021-12-02 10:30:01 +02:00
Aliaksandr Valialkin
d4655beae8
lib/fs: add vm_filestream_read_duration_seconds_total and vm_filestream_write_duration_seconds_total metrics 
These metrics help determining persistent disk saturation with `rate(vm_filestream_read_duration_seconds_total) > 0.9`
 2021-12-02 09:13:20 +02:00
Aliaksandr Valialkin
2e43cd9d62
lib/storage: do not take into account -storage.minFreeDiskSpaceBytes during background merges 2021-12-01 12:30:03 +02:00
Nikolay
cf1d2f289b
removes FileSize from backup part key (#1872) 
* removes FileSize from backup part key
it should fix download restoration for backups

* Update lib/backup/common/part.go

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2021-12-01 12:30:03 +02:00
Aliaksandr Valialkin
71c0f7cce3
lib/storage: take into account -storage.minFreeDiskSpaceBytes when performing big merges 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/269
 2021-11-30 12:56:53 +02:00
guidao
6fa7ad69fc
fix #1830 (#1861) 
Co-authored-by: wangfeng <wangfeng@zhihu.com>
 2021-11-30 01:16:15 +02:00
Aliaksandr Valialkin
975498d402
lib/protoparser/prometheus: follow-up for 8e338632a3 
Do not spend CPU time on error message formatting if error logger is disabled
 2021-11-30 00:51:15 +02:00
Nikolay
40f0726147
Changes unmarshallRow logger to noop for getRowsDiff (#1835) 2021-11-30 00:51:14 +02:00
Aliaksandr Valialkin
4ad397188e
lib/protoparser: do not log connection reset by peer error when reading the data via InfluxDB, Graphite and OpenTSDB protocols over plain TCP connections 
This error is expected, so there is no need in spamming the log with this error.
 2021-11-29 21:58:11 +02:00
Aliaksandr Valialkin
e93f46187d
lib/persistentqueue: add vm_persistentqueue_read_duration_seconds_total and vm_persistentqueue_write_duration_seconds_total metrics for determining disk usage saturation at vmagent 2021-11-17 16:42:12 +02:00
Lan
6662714c6c
Add flag of S3ForcePathStyle (#1802) 2021-11-17 01:10:22 +02:00
Aliaksandr Valialkin
4fb19fe34b
all: consistently return application/json content-type without charset=utf-8 
The `application/json` content-type has utf-8 encoding by default.
See https://stackoverflow.com/questions/9254891/what-does-content-type-application-json-charset-utf-8-really-mean

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/897
 2021-11-09 18:07:22 +02:00
Aliaksandr Valialkin
f41c02e475
lib/promscrape: improve logging for scrape_config_files parse errors 
Log the actual file path, which led to the parse error.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1789
 2021-11-08 13:34:26 +02:00
Aliaksandr Valialkin
847004fa77
app/{vminsert,vmagent}: hide passwords and auth tokens by default at /config page 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1764
 2021-11-05 14:42:13 +02:00
Aliaksandr Valialkin
bc72b83102
lib/promauth: do not show empty values in oauth2 config section at /config page 2021-11-05 12:54:10 +02:00
Aliaksandr Valialkin
d445d22c0c
lib/promscrape: add -promscrape.maxResponseHeadersSize command-line flag for tuning the maximum http response headers size from Prometheus scrape targets 2021-11-03 22:27:55 +02:00
Aliaksandr Valialkin
6873d6d893
lib/protoparser/influx: automatically detect timestamp precision depending on the number of decimal digits in the timestamp 2021-10-28 12:48:34 +03:00
Aliaksandr Valialkin
105deb164c
lib/logger: show only explicitly set command-line flags in logs 
This reduces initial verbosity in logs
 2021-10-28 11:03:21 +03:00
Aliaksandr Valialkin
b626d6d606
lib/promscrape: add collapse and expand buttons per each group of targets from the same scrape job 2021-10-27 20:04:03 +03:00
Aliaksandr Valialkin
2ebee4e741
app/{vmalert,vmagent}: improve the distribution of scrape offsets among targets / rules 
Previously only the lower part of 64-bit hash was used for calculating the offset.
This may give uneven distribution in some cases. So let's use all the available 64 bits from the hash
for calculating the offset.
 2021-10-27 20:04:02 +03:00
Aliaksandr Valialkin
92d01db85a
lib/protoparser/prometheus: optimize GetRowsDiff() function 
This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1745 ,
since the provided profile shows that the majority of CPU and memory is spent in this function
during `streamParse` when `-promscrape.noStaleMarkers` wasn't set.
 2021-10-27 18:55:25 +03:00
Aliaksandr Valialkin
16f1aaf0b5
lib/protoparser/prometheus: add a benchmark for GetRowsDiff 2021-10-27 18:55:23 +03:00
Aliaksandr Valialkin
99784b21c1
all: fix build issues and tests for Apple M1 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1653
 2021-10-27 15:07:19 +03:00
Aliaksandr Valialkin
ad445a06cd
lib/promscrape: properly show proxy_url option value at /config page 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1755
 2021-10-26 21:24:22 +03:00
Aliaksandr Valialkin
b08f51f5d3
lib/promscrape: do not populate response body to memory in stream parsing mode if -promscrape.noStaleMarkers is set 
The response body isn't used if -promscrape.noStaleMarkers is set after the commit 2876137c92 ,
so there is no sense in pupulating it in memory. This should reduce memory usage when scraping big responses.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1728#issuecomment-949630694
 2021-10-22 16:49:21 +03:00
Aliaksandr Valialkin
6bc10f0623
lib/promscrape: do not sort original labels and do not intern label string for the original labels before the sharding code is executed 
This should reduce CPU and memory usage in shard mode when service discovery finds big number of scrape targets with many long labels.
See https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets

This is a follow-up after 9882cda8b9

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1728
 2021-10-22 13:55:39 +03:00
Aliaksandr Valialkin
a8bcc3c276
lib/promscrape: reduce memory usage if -promscrape.noStaleMarkers command-line flag is passed 
Do not store in memory the response from the last scrape per each target if -promscrape.noStaleMarkers option is enabled.
This should reduce memory usage when the scraped targets return large responses.
 2021-10-22 13:22:08 +03:00
Nikolay
83e1dfccba
adds tab as second separator for graphite text protocol (#1733) 
* adds tab as second separator for graphite text protocol

* changes indexFunc for indexAny

* Update lib/protoparser/graphite/parser_test.go

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2021-10-22 12:29:27 +03:00
Aliaksandr Valialkin
d56e676d71
lib/flagutil: do not expose sensitive info (passwords, keys and urls) at /flags page 2021-10-20 00:51:15 +03:00
Aliaksandr Valialkin
5705f4b6d1
lib/httpserver: expose command-line flags at /flags page 
This should simplify debugging.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1695
 2021-10-20 00:46:54 +03:00
Aliaksandr Valialkin
a105b71116
lib/envflag: use flag.Set for setting the flags from env vars 
This should make visible the set flags at flag.Visit(), which is used later for logging
and exporting the `is_set` label for these flags at /metrics page
 2021-10-20 00:46:53 +03:00
Aliaksandr Valialkin
93511b4be7
lib/storage: log a warning when the -storageDataPath has less than -storage.minFreeDiskSpaceBytes 
This should improve the debuggability of the readonly feature.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1727
 2021-10-19 23:58:09 +03:00
Aliaksandr Valialkin
ea69eef375
lib/promscrape/discovery/kubernetes: log a warning if role: endpoints discovers more than 1000 targets per a single endpoint 
In this case `role: endpointslice` must be used instead.

See the following references:

* https://kubernetes.io/docs/reference/labels-annotations-taints/#endpoints-kubernetes-io-over-capacity
* https://github.com/kubernetes/kubernetes/pull/99975
* https://github.com/prometheus/prometheus/issues/7572#issuecomment-934779398
 2021-10-19 13:22:28 +03:00
Nikolay
e84a063209
changes job source for /target api (#1723) 
use jobNameOriginal instead of relabeled as prometheus does

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1707
 2021-10-19 09:00:05 +03:00
Aliaksandr Valialkin
fbcc8b5c7d
lib/promscrape: set honor_timestamps: true by default if this option isnt set explicitly in scrape configs 
This aligns the behavior to Prometheus - see https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config
 2021-10-16 20:48:53 +03:00
Aliaksandr Valialkin
0a9be5ef9d
lib/promscrape: expose promscrape_series_limit_max_series and promscrape_series_limit_current_series metrics per each scrape target with the enabled unique series limiter 2021-10-16 19:14:13 +03:00
Aliaksandr Valialkin
99011c6b63
lib/promscrape: always initialize http client for stream parsing mode 
Stream parsing mode can be automatically enabled when scraping targets with big response bodies
exceeding the -promscrape.minResponseSizeForStreamParse , so it must be always initialized.
 2021-10-16 13:19:48 +03:00
Aliaksandr Valialkin
0f4fda1bda
lib/promscrape: store the last scraped response in compressed form if its size exceeds -promscrape.minResponseSizeForStreamParse 
This should reduce memory usage when scraping targets with big response bodies.
 2021-10-16 13:00:11 +03:00
Aliaksandr Valialkin
0452a8d4e8
lib/promscrape: store the full response in stream parsing mode in scrapeWork.lastScrape byte slice 
This allows sending staleness marks and properly calculate scrape_series_added metric in stream parsing mode
at the cost of the increased memory usage, since now the potentially big response is kept
in the lastScrape byte slice per each scrapeWork.

In practice the memory usage increase shouldn't be big, since the response size
is usually much smaller than the parsed metrics from this response after the relabeling,
which usually adds a big pile of target-specific labels per each metric.
 2021-10-15 15:26:24 +03:00
Aliaksandr Valialkin
3e9beb0f8d
lib/promscrape/discovery/kubernetes: rename endpointslices.go -> endpointslice.go in order to be consistent with EndpointSlice struct name 
This is a follow-up for 31b42b30b6
 2021-10-15 12:27:31 +03:00
Aliaksandr Valialkin
25421fa2ae
lib/promscrape: add -promscrape.minResponseSizeForStreamParse command-line option for automatic switching to stream parsing mode when scraping targets with big responses 
This should reduce memory usage when vmagent scrapes targets with non-uniform response sizes.
This is common case in Kubernetes monitoring.
 2021-10-14 12:30:55 +03:00
Aliaksandr Valialkin
bee130cc78
lib/promscrape: return error if sample_limit or series_limit options are set when stream parsing mode is enabled 2021-10-14 12:30:54 +03:00
Aliaksandr Valialkin
5b7d90d178
lib/promscrape: add ability to show the original labels for discovered targets at /targets page 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1698
 2021-10-13 16:44:34 +03:00
Roman Khavronenko
5dab25e8ad
lib/promscrape: make errcheck happy (#1703) 2021-10-13 15:11:45 +03:00
Aliaksandr Valialkin
c3a729d458
lib/promscrape: shard targets among cluster nodes after relabeling is applied 
This guarantees that targets with the same set of labels go to the same vmagent node.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1687#issuecomment-940629495
 2021-10-12 17:06:37 +03:00
Aliaksandr Valialkin
aeedfe2fe2
app/vmagent: expose -promscrape.config contents at /config page as Prometheus does 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1695
 2021-10-12 16:27:37 +03:00
Aliaksandr Valialkin
84aa08d93a
lib/promscrape: use Prometheus format for target labels at /targets page 
This should simplify copy-pasting the labels to/from PromQL / MetricsQL
 2021-10-11 12:42:18 +03:00
Aliaksandr Valialkin
a7a1305395
lib/storage: fix unaligned access on 32-bit architectures. 
The bug has been introduced at a171916ef5
 2021-10-08 19:38:20 +03:00
Aliaksandr Valialkin
a47754b689
lib/protoparser/clusternative: typo fix after 4fddcf4c83 2021-10-08 15:38:47 +03:00
Aliaksandr Valialkin
4fddcf4c83
app/{vminsert,vmstorage}: follow-up after a171916ef5 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/269
 2021-10-08 14:09:51 +03:00
Nikolay
a171916ef5
Adds read-only mode for vmstorage node (#1680) 
* adds read-only mode for vmstorage
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/269

* changes order a bit

* moves isFreeDiskLimitReached var to storage struct
renames functions to be consistent
change protoparser api - with optional storage limit check for given openned storage

* renames freeSpaceLimit to ReadOnly
 2021-10-08 12:52:56 +03:00
Ziqi Zhao
1db3aeab36
fix some typos (#1678) 
Co-authored-by: 柘远 <zzq237937@alibaba-inc.com>
 2021-10-06 14:43:56 +03:00
Aliaksandr Valialkin
522a404b79
lib/promscrape: reduce memory allocations in mergeLabels() after 48e3e6c8df 2021-09-30 16:56:43 +03:00
Aliaksandr Valialkin
7b69d478ec
lib/protoparser: go fmt 2021-09-29 21:17:49 +03:00
Aliaksandr Valialkin
6167890d0e
lib/protoparser/prometheus: compare invalid Prometheus lines in full 2021-09-29 19:41:23 +03:00
Aliaksandr Valialkin
8dcf814c48
app/{vmbackup,vmrestore}: switch from gcs://... to gs://... urls for backups to GCS 
The `gs://` urls are commonly used, so prefer them instead of `gcs://` urls,
while leaving support for `gcs://` urls for backwards compatibility.
 2021-09-29 12:12:37 +03:00
Nikolay
9be5689b3f
changes auth validation for openstack (#1663) 
* changes auth validation for openstack
must fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1655

* Apply suggestions from code review

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2021-09-29 00:33:38 +03:00
Aliaksandr Valialkin
4e65bfcc00
app/{vminsert,vmagent}: add ability to ingest data via DataDog "submit metrics" API 
See https://docs.datadoghq.com/api/latest/metrics/#submit-metrics

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/206
 2021-09-29 00:12:26 +03:00
Aliaksandr Valialkin
d15d036a5a
lib/storage: properly handle {__name__=~"prefix(suffix1|suffix2)",other_label="..."} queries 
They were broken in the commit 00cbb099b6

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1644
 2021-09-23 21:52:31 +03:00
Aliaksandr Valialkin
d8de26bbfd
lib/promscrape: add vm_promscrape_max_scrape_size_exceeded_errors_total metric for counting of the failed scrapes due to the exceeded response size 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1639
 2021-09-23 14:48:16 +03:00
Aliaksandr Valialkin
86bafe796c
lib/httpserver: add -enterprise and/or -cluster suffixes to short_version label of vm_app_version metric 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1635
 2021-09-21 23:12:50 +03:00
Aliaksandr Valialkin
c3e1f87048
lib/promrelabel: fix parsing regex: true in relabeling rules 2021-09-21 23:01:40 +03:00
Nikolay
dd53abf36d changes protoparser apis for accepting reading from io.Reader (#1624) 
adds InsertHandlerForReader apis to vmagent
 2021-09-20 14:54:20 +03:00
Nikolay
1ab2f844a2 makes filters optional for ec2 api requests (#1627) 
filters can be applied only for DescribeInstances requests, like prometheus does.
related issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1626
 2021-09-17 18:12:25 +03:00
Aliaksandr Valialkin
1493461244 lib/storage: follow up after 00cbb099b6 2021-09-14 14:23:02 +03:00
faceair
61a51f7c15 lib/storage: optimize convert multiple values regexp filter to composite tag filter (#1610) 
* lib/storage: optimize convert multiple values regexp filter to composite tag filter

* Apply suggestions from code review

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2021-09-14 14:23:01 +03:00
Aliaksandr Valialkin
184e145570 docs: consistency renaming: Influx -> InfluxDB 2021-09-13 17:14:45 +03:00
Aliaksandr Valialkin
b684624f67 lib/promscrape/discovery/docker: support host networking mode 
See https://github.com/prometheus/prometheus/issues/9116
 2021-09-13 13:30:55 +03:00
Aliaksandr Valialkin
6ed9f10da5 lib/promscrape/discovery/kubernetes: properly use https scheme for wildcard TLS certificates in ingress target discovery 
See https://github.com/prometheus/prometheus/issues/8902
 2021-09-13 13:04:43 +03:00
Aliaksandr Valialkin
d90834da70 lib/promscrape: generate scrape_timeout_seconds metric per each scrape target in the same way as Prometheus 2.30 does 
See https://github.com/prometheus/prometheus/pull/9247
 2021-09-12 15:21:26 +03:00
Aliaksandr Valialkin
279f37c9e7 lib/promscrape: make fmt 2021-09-12 13:35:21 +03:00
Aliaksandr Valialkin
6c97388dde lib/promscrape: add ability to configure scrape_timeout and scrape_interval via relabeling 
See https://github.com/prometheus/prometheus/pull/8911
 2021-09-12 13:35:20 +03:00
Aliaksandr Valialkin
09670479cd lib/promscrape: reduce CPU usage for common case when calculating scrape_series_added metric 
Also reduce CPU usage when applying `series_limit` to scrape targets with constant set of metrics.

The main idea is to perform the calculations on scrape_series_added and series_limit
only if the set of metrics exposed by the target has been changed.
Scrape targets rarely change the set of exposed metrics,
so this optimization should reduce CPU usage in general case.
 2021-09-12 12:53:45 +03:00
Aliaksandr Valialkin
c339642858 lib/promscrape: add the actual job name to the labels of promscrape_series_limit_rows_dropped_total metric 2021-09-11 11:03:38 +03:00
Aliaksandr Valialkin
6d6cf1b6e0 lib/storage: verify that the tsidsFound contain the needed tsids in tests added at f4dead529f 2021-09-11 11:02:56 +03:00
Aliaksandr Valialkin
5aaaa686a4 lib/promscrape: send stale markers for disappeared metrics like Prometheus does 2021-09-11 11:02:56 +03:00
Aliaksandr Valialkin
c2f37f049b lib/storage: properly search series by multiple tag filters matching empty labels such as foo{bar=~"baz|",x=~"y|"} 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1601
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/395
 2021-09-09 21:12:53 +03:00
Aliaksandr Valialkin
d613a018c8 lib/promscrape: add ability to set series_limit and stream_parse options via relabeling 
This allows managing these options on a per-target basis.

Typical use case: to manage these options for pods via Kubernetes annotations.
 2021-09-09 18:51:23 +03:00
Aliaksandr Valialkin
b64866e64c lib/promscrape: add scrape_ prefix to job and target labels exported by promscrape_series_limit_rows_dropped_total metric 
This is needed in order to prevent from possible clash with the corresponding (job, target) labels for the job, which scrapes this metric.
 2021-09-09 17:31:04 +03:00
Aliaksandr Valialkin
75c3514c5c lib/promrelabel: add keep_metrics and drop_metrics actions to relabeling rules 
These actions simlify metrics filtering. For example,

- action: keep_metrics
  regex: 'foo|bar|baz'

would leave only metrics with `foo`, `bar` and `baz` names, while the rest of metrics will be deleted.

The commit also makes possible to split long regexps into multiple lines. For example, the following config is equivalent to the config above:

- action: keep_metrics
  regex:
  - foo
  - bar
  - baz
 2021-09-09 16:25:09 +03:00
mxlxm
42e07cfaea
reset deadline, fix #1562. (#1597) 
* reset deadline, fix #1562.
reset deadline before we put it back to pool.

* make errcheck happy
 2021-09-07 20:54:17 +03:00
Aliaksandr Valialkin
c4df601f43 lib/promscrape: add the ability to limit the number of unique series per each scrape target 
The number of series per target can be limited with the following options:

* Global limit with `-promscrape.maxSeriesPerTarget` command-line option.
* Per-target limit with `max_series: N` option in `scrape_config` section.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1561
 2021-09-01 16:08:12 +03:00