github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-11 14:53:49 +00:00
Code Issues Projects Releases Packages Wiki Activity
1737 commits 131 branches 758 tags 363 MiB
Commit graph

645 commits

Author SHA1 Message Date
Aliaksandr Valialkin
9043a509a3 lib/storage: properly check timeouts and pace limits 
Previously they were checked on every iteration for small number of iterations
 2020-08-07 08:40:37 +03:00
Aliaksandr Valialkin
bc8381613d app/vmselect: reduce memory allocations by pre-allocatin memory for time series map and for a list of time series names 2020-08-06 19:17:58 +03:00
Aliaksandr Valialkin
8e44fba76d lib/storage: reduce the frequency (and overhead) for timeout and pace limiter checks by 4x 2020-08-06 18:45:55 +03:00
Aliaksandr Valialkin
7dbe335426 lib/pacelimiter: increase scalability for multi-CPU system 2020-08-06 18:32:59 +03:00
Aliaksandr Valialkin
ad730d8a17 lib/storage: optimize prefetching metric names for the given metricIDs 2020-08-06 16:53:10 +03:00
Aliaksandr Valialkin
639b26b40c lib/fs: export vm_nfs_pending_dirs_to_remove metric for monitoring the number of pending directories that couldn't be removed due to NFS lock 2020-08-06 15:31:34 +03:00
Aliaksandr Valialkin
8f16388428 lib/storage: limit the number of concurrent calls to storage.searchTSIDs to GOMAXPROCS*2 
This should limit the maximum memory usage and reduce CPU trashing on vmstorage
when multiple heavy queries are executed.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
 2020-08-05 18:30:07 +03:00
Aliaksandr Valialkin
aaa497ff0b Perform conversion from string to []byte according to rule #6 at https://golang.org/pkg/unsafe/#Pointer 2020-08-05 11:55:58 +03:00
Aliaksandr Valialkin
a034f02fb2 lib/backup: allow using ~/.aws/config without region 
Use us-west-2 for determining bucket region.
 2020-08-04 13:07:59 +03:00
Aliaksandr Valialkin
509d12643b app/vmselect: show X-Forwarded-For contents on /api/v1/status/active_queries page 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/659
 2020-07-31 20:05:18 +03:00
Aliaksandr Valialkin
5e71fab8a6 lib/storage: reduce the maximum number of concurrent merge workers to GOMAXPROCS/2 
Previously the limit has been raised to GOMAXPROCS, but it has been appeared that this
increases query latencies since more CPUs are busy with merges.

While at it, substitute `*MergeConcurrencyLimitCh` channels with simple integer limits.
 2020-07-31 17:46:56 +03:00
Aliaksandr Valialkin
e7959094f6 lib/storage: remove prioritizing of merging small parts over merging big parts, since it doesn't work as expected 
The prioritizing could lead to big merge starvation, which could end up in too big number of parts that must be merged into big parts.

Multiple big merges may be initiated after the migration from v1.39.0 or v1.39.1. It is OK - these merges should be finished soon,
which should return CPU and disk IO usage to normal levels.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/618
 2020-07-30 19:57:27 +03:00
Aliaksandr Valialkin
922d9aadf2 lib/storage: properly update vm_slow_row_inserts_total metric when importing multiple data points per time series at once 
Previously the `vm_slow_row_inserts_total` metric may be incremented multiple times for different data points per a single time series,
while only a single increment is needed when inserting the first data point for this time series.
 2020-07-30 16:17:24 +03:00
Aliaksandr Valialkin
67a64c142d lib/httpserver: emit X-Forwarded-For additionally to remoteAddr in error logs 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/659
 2020-07-29 13:12:42 +03:00
Sasasu
5b81bdde39
lib/storage: metaindexRow use memroy more efficiently (#655) 
due to memory align the metaindexRow structure use 64-byte pre object.
this commit changes the order of field, make metaindexRow use 56-byte pre
object.

Signed-off-by: Sasasu <su@sasasu.me>
 2020-07-27 19:02:53 +03:00
Aliaksandr Valialkin
865610a7c8 lib/protoparser/prometheus: add a test for cassandra-exporter 
Thanks to Seva
 2020-07-27 18:37:11 +03:00
Aliaksandr Valialkin
54ef2d8112 lib/storage: slightly reduce code difference between single-node and cluster versions 2020-07-24 00:31:16 +03:00
Aliaksandr Valialkin
039c9d2441 lib/storage: respect -search.maxQueryDuration when searching for time series in inverted index 
Previously the time spent on inverted index search could exceed the configured `-search.maxQueryDuration`.
This commit stops searching in inverted index on query timeout.
 2020-07-23 21:21:42 +03:00
Aliaksandr Valialkin
2a45871823 lib/storage: add more fine-grained pace limiting for search 2020-07-23 19:26:08 +03:00
Aliaksandr Valialkin
6f05c4d351 lib/storage: improve prioritizing of data ingestion over querying 
Prioritize also small merges over big merges.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
 2020-07-23 13:23:36 +03:00
Aliaksandr Valialkin
61c611f5ad lib/storage: properly calculate global metrics in UpdateStats() 2020-07-23 00:35:15 +03:00
Aliaksandr Valialkin
9224ede54f lib/mergeset: properly calculate global metrics in UpdateStats() 
Previously these metrics could be calculated multiple times for multiple mergeset.Table instances.
 2020-07-23 00:35:13 +03:00
Aliaksandr Valialkin
228d137936 lib/storage: reorder mergeBlockStreams() args in order to make them more consistent 2020-07-22 21:58:10 +03:00
Aliaksandr Valialkin
e4303d3d21 lib/storage: prevent possible race condition when all the goroutines exit Storage.AddRows, before goroutines other goroutines are blocked on searchTSIDsCond inside Storage.searchTSIDs 
This condition may occur after the following sequence of events:

1) A goroutine enters the loop body when len(addRowsConcurrencyCh) == cap(addRowsConcurrencyCh) inside Storage.searchTSIDs.
2) All the goroutines return from Storage.AddRows.
3) The goroutine from step 1 blocks on searchTSIDsCond.Wait() inside the loop body.

The goroutine remains blocked until the next call to Storage.AddRows, which calls searchTSIDsCond.Signal().
This may take indefinite time.
 2020-07-22 21:52:34 +03:00
Aliaksandr Valialkin
31ae5911a8 app/vmagent: add -remoteWrite.decimalPlaces command-line flag, which may be used for reducing disk space usage on the remote storage 2020-07-21 21:55:32 +03:00
Aliaksandr Valialkin
d3442b40b2 lib/uint64set: optimize adding items to the set via Set.AddMulti 2020-07-21 20:56:59 +03:00
Aliaksandr Valialkin
b35cb293f5 lib/httpserver: log remote address in error message from httpserver.Errorf 
This should improve detection of the root cause of errors.
Thanks to Anant for the idea.
 2020-07-20 14:11:22 +03:00
Aliaksandr Valialkin
e1107fec10 lib/storage: reset MetricName->TSID cache after marking metricIDs as deleted 
This is a follow-up commit after 12b16077c4 ,
which didn't reset the `tsidCache` in all the required places.
This could result in indefinite errors like:

    missing metricName by metricID ...; this could be the case after unclean shutdown; deleting the metricID, so it could be re-created next time

Fix this by resetting the cache inside deleteMetricIDs function.
 2020-07-14 14:06:32 +03:00
Aliaksandr Valialkin
cde18d1f43 lib/protoparser: properly update vm_protoparser_rows_read_total{type="promscrape"} metric 2020-07-14 12:16:35 +03:00
Seva Poliakov
457e61900d add vm_protoparser_rows_read_total metrics to promscrape (#624) 
* add vm_protoparser_rows_read_total metrics to promscrape

move vm_protoparser_rows_read_total for promscrape to better place

move vm_protoparser_rows_read_total for promscrape to better place

* remove possibility of infinity loop at prometheus parser
 2020-07-14 12:16:34 +03:00
Roman Khavronenko
7e347972c4
lib/flagutil: specify additional description for all Array type flags (#620) 
Array type flag is now defined as `value` type in flag description when printed.
This change adds additional description to every Array type flag so it would be
clear what exact type is used:
```
  -remoteWrite.urlRelabelConfig array
        Optional path to relabel config for the corresponding -remoteWrite.url
        Supports array of values separated by comma or specified via multiple flags.
```
 2020-07-13 21:56:37 +03:00
Roman Khavronenko
19dd121968
lib/persistentqueue: add vm_persistentqueue_bytes_pending metric (#619) 
Metric `vm_persistentqueue_bytes_pending` is a gauge that shows current amount
of bytes in persistentqueue flushed on disk as a difference between write and read
offsets. This metric is very similar to `vmagent_remotewrite_pending_data_bytes`
except of accounting for bytes in-memory.
 2020-07-13 21:54:09 +03:00
Roman Khavronenko
829ec4f9cf
Extend metric vm_promscrape_targets with status label (#615) 
The change to `vm_promscrape_targets` metric suppose to improve observability
for `vmagent` so it will be possible to track how many targets are up or down
for every specific scrape group:
```
vm_promscrape_targets{type="static_configs", status="down"} 1
vm_promscrape_targets{type="static_configs", status="up"} 2
```
 2020-07-13 21:52:03 +03:00
Aliaksandr Valialkin
cba820e390 app/{vminsert,vmagent}: add ability to import data in Prometheus exposition format via /api/v1/import/prometheus 2020-07-10 12:14:07 +03:00
Aliaksandr Valialkin
6fe3c48a6e properly calculate readCalls 2020-07-10 12:00:58 +03:00
Aliaksandr Valialkin
3e1d7d8489 lib/promscrape: send Accept header similar to Prometheus when scraping targets 
This should fix scraping Spring Boot servers, which return incorrect response
unless `Accept: text/plain` request header is set.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/608
 2020-07-08 19:48:22 +03:00
Aliaksandr Valialkin
cb92113632 lib/storage: limit the maximum concurrency for data ingestion to GOMAXPROCS 
Previously the concurrency has been limited to GOMAXPROCS*2. This had little sense,
since every call to Storage.AddRows is bound to CPU, so the maximum ingestion bandwidth
is achieved when the number of concurrent calls to Storage.AddRows is limited to the number of CPUs,
i.e. to GOMAXPROCS.
 2020-07-08 17:32:18 +03:00
Roman Khavronenko
e7557e0252
lib/protoparser: fix metric name of unmarshal errors in promremotewrite (#607) 
The change fixes the typo in metric name `vm_protoparser_unmarshal_errors` to
respect the naming standard.
 2020-07-08 14:18:41 +03:00
Aliaksandr Valialkin
e59b9916aa lib/protoparser/graphite: go fmt 2020-07-08 14:12:10 +03:00
Aliaksandr Valialkin
d0b694c5c8 lib/protoparser/graphite: add more tests after eb45185eef 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/610
 2020-07-08 14:10:35 +03:00
Seva Poliakov
eb45185eef
Fix graphite minus one timestamp (#609) 
* fix graphite -1 timestamp

* format the graphite fix -1 timestamp
 2020-07-08 13:59:19 +03:00
Aliaksandr Valialkin
32b9fb58b8 lib/storage: clarify out of retention period error message by mentioning -retentionPeriod command-line flag 2020-07-08 13:54:26 +03:00
Aliaksandr Valialkin
12b16077c4 lib/storage: reset MetricName->TSID cache after deleting time series 
This should prevent from adding new data points to deleted time series
without the need to check for the deleted time series.

This improves ingestion performance a bit when the `deleted time series ids` aka `dmis` set
contains big number of time series.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/596

Based on the idea from @n4mine at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/604
 2020-07-06 22:01:08 +03:00
Aliaksandr Valialkin
a23806f486 lib/fs: clarify description for -fs.disableMmap command-line flag 2020-07-06 14:28:34 +03:00
Aliaksandr Valialkin
6daa5f7500 lib/storage: prioritize data ingestion over heavy queries 
Heavy queries could result in the lack of CPU resources for processing the current data ingestion stream.
Prevent this by delaying queries' execution until free resources are available for data ingestion.

Expose `vm_search_delays_total` metric, which may be used in for alerting when there is no enough CPU resources
for data ingestion and/or for executing heavy queries.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291
 2020-07-05 19:42:05 +03:00
Aliaksandr Valialkin
ff6a0955eb lib/promscrape: use HostClient.DoDeadline instead of HostClient.Do in order to guarantee strict deadline across multiple scrape attempts 2020-07-03 21:33:22 +03:00
Aliaksandr Valialkin
8b133e40d5 lib/promscrape: prevent from too big deadline misses on scrape retries 
The maximum deadline miss duration is reduced to 2x scrape_interval in the worst case.
By default it is limited to scrape_interval configured for the given scrape target.
 2020-07-03 20:41:36 +03:00
Aliaksandr Valialkin
44a54b8b3d lib/promscrape: check for nil error before checking for the returned status code when scraping targets 2020-07-03 18:37:14 +03:00
Aliaksandr Valialkin
0b2086b7a5 app/vminsert: prevent from adding and/or selecting labels with empty values 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/600
 2020-07-02 23:14:11 +03:00
Aliaksandr Valialkin
91b3482894 app/vminsert: add ability to apply relabeling to all the incoming metrics if -relabelConfig command-line arg points to a file with a list of relabel_config entries 
See https://victoriametrics.github.io/#relabeling
 2020-07-02 20:39:28 +03:00
Aliaksandr Valialkin
4dd3de9286 lib/promscrape: add ability to set disable_compression and disable_keepalive options in scrape_config section of the config passed to -promscrape.config 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/580
 2020-07-02 14:19:14 +03:00
Aliaksandr Valialkin
8da3f773ae lib/promscrape: add -promscrape.disableKeepAlive command-line flag for disabling http keep-alive connections when scraping targets 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/580
 2020-07-01 02:20:20 +03:00
Aliaksandr Valialkin
b277ba8121 lib/httpserver: add Unwrap method to ErrorWithStatusCode, so As and Is functions in standard errors package may properly unwrap the error inside ErrorWithStatusCode 2020-07-01 00:54:01 +03:00
Aliaksandr Valialkin
56ccfa5218 all: use errors.As instead of type assertion for detecting net.Error 2020-07-01 00:15:34 +03:00
Aliaksandr Valialkin
7c2c8b2981 all: use errors.As for inspecting errors that implement httpserver.ErrorWithStatusCode 2020-07-01 00:04:34 +03:00
Aliaksandr Valialkin
d5dddb0953 all: use %w instead of %s for wrapping errors in fmt.Errorf 
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
 2020-06-30 23:05:11 +03:00
Aliaksandr Valialkin
586c5be404 lib/promscrape: add missing label sorting for autogenerated metrics 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/592
 2020-06-29 22:36:12 +03:00
Ween
1cd01b5359
Fix Auto metrics relabeled errors (#593) 
* Fix Auto metrics relabeled errors

* Finalize auto-genenated  Labels

* Fix Test Errors

Co-authored-by: xinyulong <xinyulong@kuaishou.com>
 2020-06-29 22:29:29 +03:00
Aliaksandr Valialkin
7eb171182b lib/promrelabel: properly apply ^ and $ anchors to regex value in Prometheus relabeling rules 2020-06-25 17:19:19 +03:00
Aliaksandr Valialkin
a560b4788e lib/fs: go fmt 2020-06-23 23:02:39 +03:00
Aliaksandr Valialkin
8141541e61 lib/fs: fall back to cgo copy for copying the last 4KB of mmaped data 
This probably should fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/581
 2020-06-23 22:55:22 +03:00
Aliaksandr Valialkin
2a39ba639d lib/promrelabel: add support for keep_if_equal and drop_if_equal actions to relabel configs 
These actions may be useful for filtering out unneeded targets and/or metrics if they contain equal label values.
For example, the following rule would leave the target only if __meta_kubernetes_annotation_prometheus_io_port
equals __meta_kubernetes_pod_container_port_number:

  - action: keep_if_equal
    source_labels: [__meta_kubernetes_annotation_prometheus_io_port, __meta_kubernetes_pod_container_port_number]
 2020-06-23 17:29:03 +03:00
Aliaksandr Valialkin
8f0bcec6cc lib/promscrape: preserve the previously discovered targets on discovery errors per each job_name 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/582
 2020-06-23 15:40:40 +03:00
Aliaksandr Valialkin
c970cb912c lib/fs: an attempt to fix SIGBUS error by rounding mmap`ed region to multiple of 4KB pages 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/581
 2020-06-23 13:39:49 +03:00
Aliaksandr Valialkin
b5206ce33f lib/logger: add -loggerErrorsPerSecondLimit for limiting the rate of ERROR messages 2020-06-23 12:41:36 +03:00
Aliaksandr Valialkin
4c7f216dfe lib/promscrape: retry performing the request to the server for up to 3 times before giving up when it closes keep-alive connections 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/580
 2020-06-23 12:33:54 +03:00
Aliaksandr Valialkin
7532dbcdf5 app/vmselect/promql: properly override label values from group_left and group_right lists like Prometheus does 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/577
 2020-06-21 16:33:01 +03:00
Aliaksandr Valialkin
e149019c00 lib/promscrape/discovery/consul: reduce load on Consul when discovering big number of targets by using background caching 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574
 2020-06-20 18:20:01 +03:00
Aliaksandr Valialkin
7bf2cbad32 lib/promscrape: reduce default value for -promscrape.discovery.concurrency from 500 to 100 
This should reduce load on Kubernetes API server and Consul when big number of targets are discovered

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574
 2020-06-20 17:53:42 +03:00
Aliaksandr Valialkin
6ff821c70d lib/promscrape/discovery/ec2: expose __meta_ec2_ami like the next Prometheus release will do 
See b5d61fb66c for details
 2020-06-20 17:45:23 +03:00
Aliaksandr Valialkin
5820c0ffb7 lib/promrelabel: allows regex capture groups in target_label like Prometheus does 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/569
 2020-06-19 02:21:24 +03:00
Tristan Su
ac3700ed1e
lib/storage: set big/small merge concurrency (#568) 
fixed #567

Co-authored-by: Tristan Su <suqing.sq@alibaba-inc.com>
 2020-06-19 01:25:48 +03:00
Aliaksandr Valialkin
b542e50680 app/vminsert: export metrics for determining ingested rows with dropped or truncated labels 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/565
 2020-06-19 01:10:37 +03:00
Aliaksandr Valialkin
08495360b0 lib/storage: add key!=".+" filter additionally to negative filter matching empty value such as key!~"|foo" 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/546
 2020-06-18 20:03:48 +03:00
Aliaksandr Valialkin
ae1cc0fc4b lib/storage: properly match {tag!="|foo"} filters 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/546
 2020-06-10 19:35:56 +03:00
Aliaksandr Valialkin
69a647b0d2 lib/httpserver: do not flush and do not close gzip writer if response compression is disabled 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/535
 2020-06-05 21:37:28 +03:00
Aliaksandr Valialkin
4e8d6b80e0 lib/backup: properly create missing parent directories in fs.CreateFile 2020-06-05 19:28:19 +03:00
Aliaksandr Valialkin
d120197676 lib/fs: optimize queries that read recent samples for big number of time series 
Use standard copy() func instead of mmap-aware copy func for reading recently touched mmap-ed data.
This improves read performance by up to 4x.
 2020-06-05 19:10:04 +03:00
Aliaksandr Valialkin
4cb3af1a36 lib/fs: add a benchmark for ReaderAt.MustReadAt 2020-06-05 19:10:03 +03:00
Aliaksandr Valialkin
4fd3f6f991 lib/backup/fsremote: create all the parent directories before creating file in CreateFile 2020-06-05 10:25:10 +03:00
Aliaksandr Valialkin
3d4008263f lib/fs: optimize MustGetFreeSpace performance by caching the results for up to 2 seconds 2020-06-04 13:15:47 +03:00
Vyacheslav Mitrofanov
72ff05255f
allow to use values lower than 10 with the flag -memory.allowedPercent (#531) 
Co-authored-by: Vyacheslav Mitrofanov <vmitrofanov@mfms.ru>
 2020-06-03 23:39:29 +03:00
Aliaksandr Valialkin
937338abdf lib/bytesutil: prevent from garbage collecting s before returning from ToUnsafeBytes 2020-06-03 00:23:40 +03:00
Aliaksandr Valialkin
a7797dae09 lib/storage: fix Graphite wildcard matching, which has been broken in v1.36.0 
v1.36.0 always returns empty responses for Graphite wildcards like the following

   {__name__=~"foo\\.[^.]*\\.bar\\.baz"}

Temporary workaround for v1.36.0 is to add `[^.]*` to the end of the regexp.
 2020-05-28 12:03:49 +03:00
Aliaksandr Valialkin
d186472081 lib/storage: improve search speed for time series matching Graphite whildcards such as foo.*.bar.baz 
Add index for reverse Graphite-like metric names with dots. Use this index during search for filters
like `__name__=~"foo\\.[^.]*\\.bar\\.baz"` which end with non-empty suffix with dots, i.e. `.bar.baz` in this case.

This change may "hide" historical time series during queries. The workaround is to add `[.]*` to the end of regexp label filter,
i.e. "foo\\.[^.]*\\.bar\\.baz" should be substituted with "foo\\.[^.]*\\.bar\\.baz[.]*".
 2020-05-27 21:45:52 +03:00
Aliaksandr Valialkin
f5a4731412 lib/httpserver: properly set status code for empty response 2020-05-24 23:55:28 +03:00
Aliaksandr Valialkin
947009f459 lib/httpserver: fix compression for static files 2020-05-24 22:17:21 +03:00
Aliaksandr Valialkin
c602284a99 lib/promscrape: mention about -promscrape.maxScrapeSize in the error message when target returns too big response 2020-05-24 14:41:14 +03:00
Aliaksandr Valialkin
b3fcd726e3 lib/httpserver: do not recompress already compressed response 
This shoud help with vmauth issue - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/514
 2020-05-22 16:45:04 +03:00
Aliaksandr Valialkin
f9847352b4 app/vmagent: add -dryRun option for checking all the configs mentioned in command-line flags without running vmagent 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/362
 2020-05-21 15:23:27 +03:00
Aliaksandr Valialkin
d1a9d8aa1c lib/promscrape: add -promscrape.config.dryRun flag for checking -promscrape.config for errors or unsupported options 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/508
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/362
 2020-05-21 14:55:11 +03:00
Aliaksandr Valialkin
cf87b810b7 lib/promscrape: add -promscrape.discovery.concurrency and -promscrape.discovery.concurrentWaitTime flags for tuning the number of concurrent requests to autodiscovery API servers at Consul or Kubernetes 2020-05-19 17:35:53 +03:00
Aliaksandr Valialkin
b19ca3eb5f lib/storage: do not increment vm_slow_metric_name_loads_total counter for metric_ids which shouldnt be prefetched, since this may mislead users 2020-05-16 10:21:17 +03:00
Aliaksandr Valialkin
4e850cd6a7 lib/persistentqueue: a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/484 2020-05-16 09:31:46 +03:00
肖贝贝
a0380a0a91
fix: fix vmagent multi queue may become one because sync bug (#484) 
Co-authored-by: xiaobeibei <xiaobeibei@bigo.sg>
 2020-05-16 09:19:52 +03:00
Aliaksandr Valialkin
3412d5d138 lib/backup: remove misleading -dst mention in error message 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/482
 2020-05-15 17:13:37 +03:00
Aliaksandr Valialkin
27f7cca7ff lib/backup: donload only the remaining parts for partially downloaded files after vmrestore restart 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/487
 2020-05-15 17:03:33 +03:00
Aliaksandr Valialkin
82ffbcb9a6 app/vmstorage: add vm_slow_metric_name_loads_total metric, which could be used as an indicator when more RAM is needed for improving query performance 2020-05-15 14:11:45 +03:00
Aliaksandr Valialkin
82ccdfaa91 app/vmstorage: add vm_slow_row_inserts_total and vm_slow_per_day_index_inserts_total metrics for determining whether VictoriaMetrics required more RAM for the current number of active time series 2020-05-15 13:44:32 +03:00
Aliaksandr Valialkin
0eacea1de1 lib/{storage,mergeset}: further tuning of compression levels depending on block size 
This should improve performance for querying newly added data, since it can be unpacked faster.
 2020-05-15 13:24:37 +03:00
Aliaksandr Valialkin
737d641920 lib/storage: wait for all the goroutines to finish in TestSearch in order to prevent racy behavior on test finish 2020-05-15 13:24:37 +03:00
Aliaksandr Valialkin
4fc33163c4 lib/storage: optimize ingestion pefrormance for new time series 2020-05-15 13:24:37 +03:00
Aliaksandr Valialkin
f9f3afb6af lib/mergeset: tune compression levels in order to improve ingestion performance a bit 2020-05-15 13:24:37 +03:00
Aliaksandr Valialkin
8b32e7c3a0 lib/storage: reduce indentation in Storage.add 2020-05-15 13:24:37 +03:00
Aliaksandr Valialkin
1573ececb2 lib/storage: return the first error instead of the last error, since the first error usually points to the root cause 2020-05-15 13:24:37 +03:00
Aliaksandr Valialkin
0afd48d2ee lib: extract common code for returning fast unix timestamp into lib/fasttime 2020-05-14 23:02:07 +03:00
Aliaksandr Valialkin
42866fa754 lib/{storage,mergeset}: return dst on error from unmarshalBlockHeaders, so it could be reused 2020-05-14 15:32:07 +03:00
Aliaksandr Valialkin
827a3a7866 lib/storage: document that getnerateUniqueMetricID should return dense ids 2020-05-14 14:08:45 +03:00
Aliaksandr Valialkin
606585f7be lib/{storage,mergeset}: cleanup: remove unused partSearch.indexBlockReuse 2020-05-14 14:03:03 +03:00
Aliaksandr Valialkin
4fe67504f9 lib/storage: optimize label matching for regexp ending with literal suffix 
For example, `{label=~"foo.*bar.+baz"}` contains literal suffix `baz`,
so it should work faster now.
 2020-05-13 11:47:07 +03:00
Aliaksandr Valialkin
96e001d254 app/vmagent: fix a bug with improper relabeling when multiple -remoteWrite.urlRelableConfig args are set 
This bug could result in incorrect relabeling and metrics' drop.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/467
 2020-05-12 22:02:58 +03:00
Aliaksandr Valialkin
a6f16dcc11 lib/fs: do not use mmap for 32-bit arches by default, since they cannot map files bigger than 4GB in RAM 2020-05-12 20:22:09 +03:00
Aliaksandr Valialkin
0a134ace63 app/vmagent: fix scraping mTLS targets, which has been broken in v1.35.1 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/470
 2020-05-12 17:23:03 +03:00
Aliaksandr Valialkin
8300cc17af app/vmagent,lib/promscrape: do not set HostClient.DialDualStack, since it isnt used if HostClient.Dial is set 2020-05-12 15:24:18 +03:00
Aliaksandr Valialkin
3232605524 lib/storage: properly initialize part struct before trying to close it on error 
This should prevent from nil pointer dereference bug at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/468 .
 2020-05-12 14:54:31 +03:00
Aliaksandr Valialkin
dbd0c552d5 lib/storage: gradually pre-populate per-day inverted index for the next day 
This should prevent from CPU usage spikes at 00:00 UTC every day when
inverted index for new day must be quickly created for all the active time series.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/430
 2020-05-12 12:13:05 +03:00
Aliaksandr Valialkin
cc00a2c453 lib/storage: typo fixes in error messages: or -> of 2020-05-12 12:12:42 +03:00
Aliaksandr Valialkin
ce2107bc52 lib/storage: speed up matching for common regexps in label filters 
The following regexps have been optimized:

* 'foo.+bar'
* 'foo.+bar.+baz'

This should improve performance for matching Graphite-like metrics.
 2020-05-11 22:40:55 +03:00
Aliaksandr Valialkin
12a1a71cc1 lib/storage: add a benchmark for Graphite-like regexps for metric names 2020-05-11 22:37:32 +03:00
Aliaksandr Valialkin
099e44005b lib/httpserver: add -http.shutdownDelay flag for a grace period before http server shutdown 
The http server returns 503 non-OK error at `/health` page during grace period,
so load balancers in front of the http server could re-route incoming requests
to other servers.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/463
 2020-05-07 15:30:35 +03:00
Aliaksandr Valialkin
787fcfba0c lib/httpserver: reduce typical duration for http server graceful shutdown 
Previously the duration for graceful shutdown for http server could take more than a minute
because of imporperly set timeouts in setNetworkTimeout.
Now typical duration for graceful shutdown should be reduced to less than 5 seconds.
 2020-05-07 14:12:39 +03:00
Aliaksandr Valialkin
91a49eecea lib/flagutil: make errcheck happy by explicitly ignoring Array.Set result in tests 2020-05-06 22:37:39 +03:00
Aliaksandr Valialkin
c4c447507d lib/flagutil: properly parse quoted flag values for flagutil.Array 2020-05-06 22:27:21 +03:00
Aliaksandr Valialkin
8a00807f60 app/vmagent: allow setting independent auth configs per each configured -remoteWrite.url 2020-05-06 16:51:41 +03:00
Aliaksandr Valialkin
68928bf3df lib/promscrape/discovery/gce: discover per-zone instances for gce_sd_config in parallel. This should reduce discovery latency 2020-05-06 15:00:09 +03:00
Aliaksandr Valialkin
3f52a97f9b lib/promscrape: add Prometheus-compatible DNS-based service discovery aka dns_sd_configs 2020-05-06 00:01:58 +03:00
Aliaksandr Valialkin
364789c24c lib/promscrape: properly connect to TCP6 addresses if -enableTCP6 is set 2020-05-06 00:01:57 +03:00
Aliaksandr Valialkin
7b5ef63384 lib/procutil: add NewSighupChan function, which returns a channel, which is triggered on every SIGHUP 2020-05-05 10:54:09 +03:00
Aliaksandr Valialkin
4fa817be10 lib/promscrape: allow explicitly setting empty token via token: "" in consul_sd_config 2020-05-05 07:50:15 +03:00
Aliaksandr Valialkin
40c3ffb359 lib/promscrape: add Prometheus-compatible service discovery for Consul aka consul_sd_configs 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/330
 2020-05-04 20:51:17 +03:00
Aliaksandr Valialkin
83f0e35b7b lib/promauth: properly set up client certificate in tls.Config 
Previously the client certificate has been mistakenly set up as a server certificate
 2020-05-04 20:51:08 +03:00
Aliaksandr Valialkin
218e566647 lib/promscrape: move common code for discovery api config map handling into discoveryutils 2020-05-04 20:51:01 +03:00
Aliaksandr Valialkin
6310b20e72 lib/promscrape/discovery/kubernetes/: unify apiConfig creation 2020-05-04 20:50:49 +03:00
Aliaksandr Valialkin
66b0ae79a5 lib/promscrape: remove debug line left after the commit e4aac6ea40 2020-05-03 17:15:32 +03:00
Aliaksandr Valialkin
69004a5f67 lib/promscrape: fix tests after the commit 658a8742ac 
The original commit copies `__address__` label to `instance` label when generating per-target labels as Prometheus does.

See https://www.robustperception.io/life-of-a-label for details.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/453
 2020-05-03 16:56:15 +03:00
DexterZhang
658a8742ac
fix(vmagent): different behavior as how prometheus deal with labels. [Issue#453] (#454) 2020-05-03 16:51:03 +03:00
Aliaksandr Valialkin
e4aac6ea40 lib/promscrape: make consistent scrape time offsets across reloads for the same ScrapeURL and Labels 
This should make consistent intervals between data points for scrape targets across reloads.
Previously these intervals were random.
 2020-05-03 14:30:21 +03:00
Aliaksandr Valialkin
d9f1b4d6a3 lib/promscrape: fix TestGetFileSDScrapeWorkSuccess after 3b234d82e5 2020-05-03 14:28:31 +03:00
Aliaksandr Valialkin
3b234d82e5 lib/promscrape: reload only modified scrapers on config changes 
This should improve scrape stability when big number of targets are scraped and these targets are frequently changed.

Thanks to @xbsura for the idea and initial implementation attempts at the following pull requests:

- https://github.com/VictoriaMetrics/VictoriaMetrics/pull/449
- https://github.com/VictoriaMetrics/VictoriaMetrics/pull/458
- https://github.com/VictoriaMetrics/VictoriaMetrics/pull/459
- https://github.com/VictoriaMetrics/VictoriaMetrics/pull/460
 2020-05-03 12:45:40 +03:00
Aliaksandr Valialkin
ee810e5f3a lib/httpserver: rename http.externalURL to http.pathPrefix and improve help message for this flag 
The `http.externalURL` flag name was slightly misleading, so it has been renamed to `http.pathPrefix`.
 2020-05-02 13:07:34 +03:00
DexterZhang
34743974d5
feat(httpserver): add http.externalUrl config to http server, it adds prefix to http path automatically (#452) 2020-05-02 12:42:53 +03:00
Aliaksandr Valialkin
432187ac3b app/vminsert: add /-/reload handler in the same way as for vmagent 2020-04-30 02:15:39 +03:00
Aliaksandr Valialkin
825a2dd554 lib/procutil: prevent from app termination on SIGHUP signal, since this signal is frequently used for config reload 2020-04-30 02:09:27 +03:00
Aliaksandr Valialkin
01c17092e1 lib/httpserver: mention that -http.maxGracefulShutdownDuration command-line flag value can be increased on shutdown timeout 2020-04-30 01:38:06 +03:00
Aliaksandr Valialkin
5ec036439d lib/promscrape: set 30 seconds timeout for discovery api requests 
Previously such requests could hang for long time. This could make debugging harder.
 2020-04-29 17:33:34 +03:00
Aliaksandr Valialkin
43c39dc36c vendor: use github.com/VictoriaMetrics/fasthttp instead of github.com/fasthttp/fasthttp 
The upstream fasthttp may contain issues like 996610f021 ,
plus a code that isn't used by VictoriaMetrics. So let's use a private copy under our control instead.
 2020-04-29 17:33:34 +03:00
Aliaksandr Valialkin
4e4f57b121 lib/metricsql: move it to a separate repository - github.com/VictoriaMetrics/metrics 2020-04-28 15:28:22 +03:00
Aliaksandr Valialkin
83aca79137 lib/storage: recover when metricID->metricName entry is missing in the inverted index after unclean shutdown 
Newly added index entries can be missing after unclean shutdown, since they didn't flush to persistent storage yet.
Log about this and delete the corresponding metricID, so it could be re-created next time.
 2020-04-28 12:00:33 +03:00
Aliaksandr Valialkin
521df0e2fc lib/promscrape: handle connection reset when targets responds with http redirect 2020-04-28 02:13:02 +03:00
肖贝贝
2b16c188e8
fix: vmagent not follow 301/302 redirect bug (#445) 
Co-authored-by: xiaobeibei <xiaobeibei@bigo.sg>
 2020-04-28 01:29:37 +03:00