Aliaksandr Valialkin
e4303d3d21
lib/storage: prevent possible race condition when all the goroutines exit Storage.AddRows, before goroutines other goroutines are blocked on searchTSIDsCond inside Storage.searchTSIDs
...
This condition may occur after the following sequence of events:
1) A goroutine enters the loop body when len(addRowsConcurrencyCh) == cap(addRowsConcurrencyCh) inside Storage.searchTSIDs.
2) All the goroutines return from Storage.AddRows.
3) The goroutine from step 1 blocks on searchTSIDsCond.Wait() inside the loop body.
The goroutine remains blocked until the next call to Storage.AddRows, which calls searchTSIDsCond.Signal().
This may take indefinite time.
2020-07-22 21:52:34 +03:00
Aliaksandr Valialkin
31ae5911a8
app/vmagent: add -remoteWrite.decimalPlaces
command-line flag, which may be used for reducing disk space usage on the remote storage
2020-07-21 21:55:32 +03:00
Aliaksandr Valialkin
d3442b40b2
lib/uint64set: optimize adding items to the set via Set.AddMulti
2020-07-21 20:56:59 +03:00
Aliaksandr Valialkin
b35cb293f5
lib/httpserver: log remote address in error message from httpserver.Errorf
...
This should improve detection of the root cause of errors.
Thanks to Anant for the idea.
2020-07-20 14:11:22 +03:00
Aliaksandr Valialkin
e1107fec10
lib/storage: reset MetricName->TSID
cache after marking metricIDs as deleted
...
This is a follow-up commit after 12b16077c4
,
which didn't reset the `tsidCache` in all the required places.
This could result in indefinite errors like:
missing metricName by metricID ...; this could be the case after unclean shutdown; deleting the metricID, so it could be re-created next time
Fix this by resetting the cache inside deleteMetricIDs function.
2020-07-14 14:06:32 +03:00
Aliaksandr Valialkin
cde18d1f43
lib/protoparser: properly update vm_protoparser_rows_read_total{type="promscrape"}
metric
2020-07-14 12:16:35 +03:00
Seva Poliakov
457e61900d
add vm_protoparser_rows_read_total metrics to promscrape ( #624 )
...
* add vm_protoparser_rows_read_total metrics to promscrape
move vm_protoparser_rows_read_total for promscrape to better place
move vm_protoparser_rows_read_total for promscrape to better place
* remove possibility of infinity loop at prometheus parser
2020-07-14 12:16:34 +03:00
Roman Khavronenko
7e347972c4
lib/flagutil: specify additional description for all Array type flags ( #620 )
...
Array type flag is now defined as `value` type in flag description when printed.
This change adds additional description to every Array type flag so it would be
clear what exact type is used:
```
-remoteWrite.urlRelabelConfig array
Optional path to relabel config for the corresponding -remoteWrite.url
Supports array of values separated by comma or specified via multiple flags.
```
2020-07-13 21:56:37 +03:00
Roman Khavronenko
19dd121968
lib/persistentqueue: add vm_persistentqueue_bytes_pending
metric ( #619 )
...
Metric `vm_persistentqueue_bytes_pending` is a gauge that shows current amount
of bytes in persistentqueue flushed on disk as a difference between write and read
offsets. This metric is very similar to `vmagent_remotewrite_pending_data_bytes`
except of accounting for bytes in-memory.
2020-07-13 21:54:09 +03:00
Roman Khavronenko
829ec4f9cf
Extend metric vm_promscrape_targets
with status
label ( #615 )
...
The change to `vm_promscrape_targets` metric suppose to improve observability
for `vmagent` so it will be possible to track how many targets are up or down
for every specific scrape group:
```
vm_promscrape_targets{type="static_configs", status="down"} 1
vm_promscrape_targets{type="static_configs", status="up"} 2
```
2020-07-13 21:52:03 +03:00
Aliaksandr Valialkin
cba820e390
app/{vminsert,vmagent}: add ability to import data in Prometheus exposition format via /api/v1/import/prometheus
2020-07-10 12:14:07 +03:00
Aliaksandr Valialkin
6fe3c48a6e
properly calculate readCalls
2020-07-10 12:00:58 +03:00
Aliaksandr Valialkin
3e1d7d8489
lib/promscrape: send Accept
header similar to Prometheus when scraping targets
...
This should fix scraping Spring Boot servers, which return incorrect response
unless `Accept: text/plain` request header is set.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/608
2020-07-08 19:48:22 +03:00
Aliaksandr Valialkin
cb92113632
lib/storage: limit the maximum concurrency for data ingestion to GOMAXPROCS
...
Previously the concurrency has been limited to GOMAXPROCS*2. This had little sense,
since every call to Storage.AddRows is bound to CPU, so the maximum ingestion bandwidth
is achieved when the number of concurrent calls to Storage.AddRows is limited to the number of CPUs,
i.e. to GOMAXPROCS.
2020-07-08 17:32:18 +03:00
Roman Khavronenko
e7557e0252
lib/protoparser: fix metric name of unmarshal errors in promremotewrite ( #607 )
...
The change fixes the typo in metric name `vm_protoparser_unmarshal_errors` to
respect the naming standard.
2020-07-08 14:18:41 +03:00
Aliaksandr Valialkin
e59b9916aa
lib/protoparser/graphite: go fmt
2020-07-08 14:12:10 +03:00
Aliaksandr Valialkin
d0b694c5c8
lib/protoparser/graphite: add more tests after eb45185eef
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/610
2020-07-08 14:10:35 +03:00
Seva Poliakov
eb45185eef
Fix graphite minus one timestamp ( #609 )
...
* fix graphite -1 timestamp
* format the graphite fix -1 timestamp
2020-07-08 13:59:19 +03:00
Aliaksandr Valialkin
32b9fb58b8
lib/storage: clarify out of retention period
error message by mentioning -retentionPeriod
command-line flag
2020-07-08 13:54:26 +03:00
Aliaksandr Valialkin
12b16077c4
lib/storage: reset MetricName->TSID cache after deleting time series
...
This should prevent from adding new data points to deleted time series
without the need to check for the deleted time series.
This improves ingestion performance a bit when the `deleted time series ids` aka `dmis` set
contains big number of time series.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/596
Based on the idea from @n4mine at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/604
2020-07-06 22:01:08 +03:00
Aliaksandr Valialkin
a23806f486
lib/fs: clarify description for -fs.disableMmap
command-line flag
2020-07-06 14:28:34 +03:00
Aliaksandr Valialkin
6daa5f7500
lib/storage: prioritize data ingestion over heavy queries
...
Heavy queries could result in the lack of CPU resources for processing the current data ingestion stream.
Prevent this by delaying queries' execution until free resources are available for data ingestion.
Expose `vm_search_delays_total` metric, which may be used in for alerting when there is no enough CPU resources
for data ingestion and/or for executing heavy queries.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291
2020-07-05 19:42:05 +03:00
Aliaksandr Valialkin
ff6a0955eb
lib/promscrape: use HostClient.DoDeadline instead of HostClient.Do in order to guarantee strict deadline across multiple scrape attempts
2020-07-03 21:33:22 +03:00
Aliaksandr Valialkin
8b133e40d5
lib/promscrape: prevent from too big deadline misses on scrape retries
...
The maximum deadline miss duration is reduced to 2x scrape_interval in the worst case.
By default it is limited to scrape_interval configured for the given scrape target.
2020-07-03 20:41:36 +03:00
Aliaksandr Valialkin
44a54b8b3d
lib/promscrape: check for nil error before checking for the returned status code when scraping targets
2020-07-03 18:37:14 +03:00
Aliaksandr Valialkin
0b2086b7a5
app/vminsert: prevent from adding and/or selecting labels with empty values
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/600
2020-07-02 23:14:11 +03:00
Aliaksandr Valialkin
91b3482894
app/vminsert: add ability to apply relabeling to all the incoming metrics if -relabelConfig
command-line arg points to a file with a list of relabel_config
entries
...
See https://victoriametrics.github.io/#relabeling
2020-07-02 20:39:28 +03:00
Aliaksandr Valialkin
4dd3de9286
lib/promscrape: add ability to set disable_compression
and disable_keepalive
options in scrape_config
section of the config passed to -promscrape.config
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/580
2020-07-02 14:19:14 +03:00
Aliaksandr Valialkin
8da3f773ae
lib/promscrape: add -promscrape.disableKeepAlive
command-line flag for disabling http keep-alive connections when scraping targets
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/580
2020-07-01 02:20:20 +03:00
Aliaksandr Valialkin
b277ba8121
lib/httpserver: add Unwrap method to ErrorWithStatusCode, so As
and Is
functions in standard errors
package may properly unwrap the error inside ErrorWithStatusCode
2020-07-01 00:54:01 +03:00
Aliaksandr Valialkin
56ccfa5218
all: use errors.As
instead of type assertion for detecting net.Error
2020-07-01 00:15:34 +03:00
Aliaksandr Valialkin
7c2c8b2981
all: use errors.As
for inspecting errors that implement httpserver.ErrorWithStatusCode
2020-07-01 00:04:34 +03:00
Aliaksandr Valialkin
d5dddb0953
all: use %w instead of %s for wrapping errors in fmt.Errorf
...
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
2020-06-30 23:05:11 +03:00
Aliaksandr Valialkin
586c5be404
lib/promscrape: add missing label sorting for autogenerated metrics
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/592
2020-06-29 22:36:12 +03:00
Ween
1cd01b5359
Fix Auto metrics relabeled errors ( #593 )
...
* Fix Auto metrics relabeled errors
* Finalize auto-genenated Labels
* Fix Test Errors
Co-authored-by: xinyulong <xinyulong@kuaishou.com>
2020-06-29 22:29:29 +03:00
Aliaksandr Valialkin
7eb171182b
lib/promrelabel: properly apply ^
and $
anchors to regex
value in Prometheus relabeling rules
2020-06-25 17:19:19 +03:00
Aliaksandr Valialkin
a560b4788e
lib/fs: go fmt
2020-06-23 23:02:39 +03:00
Aliaksandr Valialkin
8141541e61
lib/fs: fall back to cgo copy for copying the last 4KB of mmaped data
...
This probably should fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/581
2020-06-23 22:55:22 +03:00
Aliaksandr Valialkin
2a39ba639d
lib/promrelabel: add support for keep_if_equal
and drop_if_equal
actions to relabel configs
...
These actions may be useful for filtering out unneeded targets and/or metrics if they contain equal label values.
For example, the following rule would leave the target only if __meta_kubernetes_annotation_prometheus_io_port
equals __meta_kubernetes_pod_container_port_number:
- action: keep_if_equal
source_labels: [__meta_kubernetes_annotation_prometheus_io_port, __meta_kubernetes_pod_container_port_number]
2020-06-23 17:29:03 +03:00
Aliaksandr Valialkin
8f0bcec6cc
lib/promscrape: preserve the previously discovered targets on discovery errors per each job_name
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/582
2020-06-23 15:40:40 +03:00
Aliaksandr Valialkin
c970cb912c
lib/fs: an attempt to fix SIGBUS error by rounding mmap`ed region to multiple of 4KB pages
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/581
2020-06-23 13:39:49 +03:00
Aliaksandr Valialkin
b5206ce33f
lib/logger: add -loggerErrorsPerSecondLimit
for limiting the rate of ERROR messages
2020-06-23 12:41:36 +03:00
Aliaksandr Valialkin
4c7f216dfe
lib/promscrape: retry performing the request to the server for up to 3 times before giving up when it closes keep-alive connections
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/580
2020-06-23 12:33:54 +03:00
Aliaksandr Valialkin
7532dbcdf5
app/vmselect/promql: properly override label values from group_left
and group_right
lists like Prometheus does
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/577
2020-06-21 16:33:01 +03:00
Aliaksandr Valialkin
e149019c00
lib/promscrape/discovery/consul: reduce load on Consul when discovering big number of targets by using background caching
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574
2020-06-20 18:20:01 +03:00
Aliaksandr Valialkin
7bf2cbad32
lib/promscrape: reduce default value for -promscrape.discovery.concurrency
from 500 to 100
...
This should reduce load on Kubernetes API server and Consul when big number of targets are discovered
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/574
2020-06-20 17:53:42 +03:00
Aliaksandr Valialkin
6ff821c70d
lib/promscrape/discovery/ec2: expose __meta_ec2_ami
like the next Prometheus release will do
...
See b5d61fb66c
for details
2020-06-20 17:45:23 +03:00
Aliaksandr Valialkin
5820c0ffb7
lib/promrelabel: allows regex capture groups in target_label like Prometheus does
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/569
2020-06-19 02:21:24 +03:00
Tristan Su
ac3700ed1e
lib/storage: set big/small merge concurrency ( #568 )
...
fixed #567
Co-authored-by: Tristan Su <suqing.sq@alibaba-inc.com>
2020-06-19 01:25:48 +03:00
Aliaksandr Valialkin
b542e50680
app/vminsert: export metrics for determining ingested rows with dropped or truncated labels
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/565
2020-06-19 01:10:37 +03:00
Aliaksandr Valialkin
08495360b0
lib/storage: add key!=".+"
filter additionally to negative filter matching empty value such as key!~"|foo"
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/546
2020-06-18 20:03:48 +03:00
Aliaksandr Valialkin
ae1cc0fc4b
lib/storage: properly match {tag!="|foo"}
filters
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/546
2020-06-10 19:35:56 +03:00
Aliaksandr Valialkin
69a647b0d2
lib/httpserver: do not flush and do not close gzip writer if response compression is disabled
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/535
2020-06-05 21:37:28 +03:00
Aliaksandr Valialkin
4e8d6b80e0
lib/backup: properly create missing parent directories in fs.CreateFile
2020-06-05 19:28:19 +03:00
Aliaksandr Valialkin
d120197676
lib/fs: optimize queries that read recent samples for big number of time series
...
Use standard copy() func instead of mmap-aware copy func for reading recently touched mmap-ed data.
This improves read performance by up to 4x.
2020-06-05 19:10:04 +03:00
Aliaksandr Valialkin
4cb3af1a36
lib/fs: add a benchmark for ReaderAt.MustReadAt
2020-06-05 19:10:03 +03:00
Aliaksandr Valialkin
4fd3f6f991
lib/backup/fsremote: create all the parent directories before creating file in CreateFile
2020-06-05 10:25:10 +03:00
Aliaksandr Valialkin
3d4008263f
lib/fs: optimize MustGetFreeSpace performance by caching the results for up to 2 seconds
2020-06-04 13:15:47 +03:00
Vyacheslav Mitrofanov
72ff05255f
allow to use values lower than 10 with the flag -memory.allowedPercent ( #531 )
...
Co-authored-by: Vyacheslav Mitrofanov <vmitrofanov@mfms.ru>
2020-06-03 23:39:29 +03:00
Aliaksandr Valialkin
937338abdf
lib/bytesutil: prevent from garbage collecting s before returning from ToUnsafeBytes
2020-06-03 00:23:40 +03:00
Aliaksandr Valialkin
a7797dae09
lib/storage: fix Graphite wildcard matching, which has been broken in v1.36.0
...
v1.36.0 always returns empty responses for Graphite wildcards like the following
{__name__=~"foo\\.[^.]*\\.bar\\.baz"}
Temporary workaround for v1.36.0 is to add `[^.]*` to the end of the regexp.
2020-05-28 12:03:49 +03:00
Aliaksandr Valialkin
d186472081
lib/storage: improve search speed for time series matching Graphite whildcards such as foo.*.bar.baz
...
Add index for reverse Graphite-like metric names with dots. Use this index during search for filters
like `__name__=~"foo\\.[^.]*\\.bar\\.baz"` which end with non-empty suffix with dots, i.e. `.bar.baz` in this case.
This change may "hide" historical time series during queries. The workaround is to add `[.]*` to the end of regexp label filter,
i.e. "foo\\.[^.]*\\.bar\\.baz" should be substituted with "foo\\.[^.]*\\.bar\\.baz[.]*".
2020-05-27 21:45:52 +03:00
Aliaksandr Valialkin
f5a4731412
lib/httpserver: properly set status code for empty response
2020-05-24 23:55:28 +03:00
Aliaksandr Valialkin
947009f459
lib/httpserver: fix compression for static files
2020-05-24 22:17:21 +03:00
Aliaksandr Valialkin
c602284a99
lib/promscrape: mention about -promscrape.maxScrapeSize in the error message when target returns too big response
2020-05-24 14:41:14 +03:00
Aliaksandr Valialkin
b3fcd726e3
lib/httpserver: do not recompress already compressed response
...
This shoud help with vmauth issue - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/514
2020-05-22 16:45:04 +03:00
Aliaksandr Valialkin
f9847352b4
app/vmagent: add -dryRun
option for checking all the configs mentioned in command-line flags without running vmagent
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/362
2020-05-21 15:23:27 +03:00
Aliaksandr Valialkin
d1a9d8aa1c
lib/promscrape: add -promscrape.config.dryRun
flag for checking -promscrape.config
for errors or unsupported options
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/508
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/362
2020-05-21 14:55:11 +03:00
Aliaksandr Valialkin
cf87b810b7
lib/promscrape: add -promscrape.discovery.concurrency
and -promscrape.discovery.concurrentWaitTime
flags for tuning the number of concurrent requests to autodiscovery API servers at Consul or Kubernetes
2020-05-19 17:35:53 +03:00
Aliaksandr Valialkin
b19ca3eb5f
lib/storage: do not increment vm_slow_metric_name_loads_total
counter for metric_ids which shouldnt be prefetched, since this may mislead users
2020-05-16 10:21:17 +03:00
Aliaksandr Valialkin
4e850cd6a7
lib/persistentqueue: a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/484
2020-05-16 09:31:46 +03:00
肖贝贝
a0380a0a91
fix: fix vmagent multi queue may become one because sync bug ( #484 )
...
Co-authored-by: xiaobeibei <xiaobeibei@bigo.sg>
2020-05-16 09:19:52 +03:00
Aliaksandr Valialkin
3412d5d138
lib/backup: remove misleading -dst
mention in error message
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/482
2020-05-15 17:13:37 +03:00
Aliaksandr Valialkin
27f7cca7ff
lib/backup: donload only the remaining parts for partially downloaded files after vmrestore
restart
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/487
2020-05-15 17:03:33 +03:00
Aliaksandr Valialkin
82ffbcb9a6
app/vmstorage: add vm_slow_metric_name_loads_total
metric, which could be used as an indicator when more RAM is needed for improving query performance
2020-05-15 14:11:45 +03:00
Aliaksandr Valialkin
82ccdfaa91
app/vmstorage: add vm_slow_row_inserts_total
and vm_slow_per_day_index_inserts_total
metrics for determining whether VictoriaMetrics required more RAM for the current number of active time series
2020-05-15 13:44:32 +03:00
Aliaksandr Valialkin
0eacea1de1
lib/{storage,mergeset}: further tuning of compression levels depending on block size
...
This should improve performance for querying newly added data, since it can be unpacked faster.
2020-05-15 13:24:37 +03:00
Aliaksandr Valialkin
737d641920
lib/storage: wait for all the goroutines to finish in TestSearch in order to prevent racy behavior on test finish
2020-05-15 13:24:37 +03:00
Aliaksandr Valialkin
4fc33163c4
lib/storage: optimize ingestion pefrormance for new time series
2020-05-15 13:24:37 +03:00
Aliaksandr Valialkin
f9f3afb6af
lib/mergeset: tune compression levels in order to improve ingestion performance a bit
2020-05-15 13:24:37 +03:00
Aliaksandr Valialkin
8b32e7c3a0
lib/storage: reduce indentation in Storage.add
2020-05-15 13:24:37 +03:00
Aliaksandr Valialkin
1573ececb2
lib/storage: return the first error instead of the last error, since the first error usually points to the root cause
2020-05-15 13:24:37 +03:00
Aliaksandr Valialkin
0afd48d2ee
lib: extract common code for returning fast unix timestamp into lib/fasttime
2020-05-14 23:02:07 +03:00
Aliaksandr Valialkin
42866fa754
lib/{storage,mergeset}: return dst on error from unmarshalBlockHeaders, so it could be reused
2020-05-14 15:32:07 +03:00
Aliaksandr Valialkin
827a3a7866
lib/storage: document that getnerateUniqueMetricID should return dense ids
2020-05-14 14:08:45 +03:00
Aliaksandr Valialkin
606585f7be
lib/{storage,mergeset}: cleanup: remove unused partSearch.indexBlockReuse
2020-05-14 14:03:03 +03:00
Aliaksandr Valialkin
4fe67504f9
lib/storage: optimize label matching for regexp ending with literal suffix
...
For example, `{label=~"foo.*bar.+baz"}` contains literal suffix `baz`,
so it should work faster now.
2020-05-13 11:47:07 +03:00
Aliaksandr Valialkin
96e001d254
app/vmagent: fix a bug with improper relabeling when multiple -remoteWrite.urlRelableConfig
args are set
...
This bug could result in incorrect relabeling and metrics' drop.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/467
2020-05-12 22:02:58 +03:00
Aliaksandr Valialkin
a6f16dcc11
lib/fs: do not use mmap for 32-bit arches by default, since they cannot map files bigger than 4GB in RAM
2020-05-12 20:22:09 +03:00
Aliaksandr Valialkin
0a134ace63
app/vmagent: fix scraping mTLS targets, which has been broken in v1.35.1
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/470
2020-05-12 17:23:03 +03:00
Aliaksandr Valialkin
8300cc17af
app/vmagent,lib/promscrape: do not set HostClient.DialDualStack, since it isnt used if HostClient.Dial is set
2020-05-12 15:24:18 +03:00
Aliaksandr Valialkin
3232605524
lib/storage: properly initialize part struct before trying to close it on error
...
This should prevent from nil pointer dereference bug at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/468 .
2020-05-12 14:54:31 +03:00
Aliaksandr Valialkin
dbd0c552d5
lib/storage: gradually pre-populate per-day inverted index for the next day
...
This should prevent from CPU usage spikes at 00:00 UTC every day when
inverted index for new day must be quickly created for all the active time series.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/430
2020-05-12 12:13:05 +03:00
Aliaksandr Valialkin
cc00a2c453
lib/storage: typo fixes in error messages: or -> of
2020-05-12 12:12:42 +03:00
Aliaksandr Valialkin
ce2107bc52
lib/storage: speed up matching for common regexps in label filters
...
The following regexps have been optimized:
* 'foo.+bar'
* 'foo.+bar.+baz'
This should improve performance for matching Graphite-like metrics.
2020-05-11 22:40:55 +03:00
Aliaksandr Valialkin
12a1a71cc1
lib/storage: add a benchmark for Graphite-like regexps for metric names
2020-05-11 22:37:32 +03:00
Aliaksandr Valialkin
099e44005b
lib/httpserver: add -http.shutdownDelay
flag for a grace period before http server shutdown
...
The http server returns 503 non-OK error at `/health` page during grace period,
so load balancers in front of the http server could re-route incoming requests
to other servers.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/463
2020-05-07 15:30:35 +03:00
Aliaksandr Valialkin
787fcfba0c
lib/httpserver: reduce typical duration for http server graceful shutdown
...
Previously the duration for graceful shutdown for http server could take more than a minute
because of imporperly set timeouts in setNetworkTimeout.
Now typical duration for graceful shutdown should be reduced to less than 5 seconds.
2020-05-07 14:12:39 +03:00
Aliaksandr Valialkin
91a49eecea
lib/flagutil: make errcheck happy by explicitly ignoring Array.Set result in tests
2020-05-06 22:37:39 +03:00
Aliaksandr Valialkin
c4c447507d
lib/flagutil: properly parse quoted flag values for flagutil.Array
2020-05-06 22:27:21 +03:00
Aliaksandr Valialkin
8a00807f60
app/vmagent: allow setting independent auth configs per each configured -remoteWrite.url
2020-05-06 16:51:41 +03:00
Aliaksandr Valialkin
68928bf3df
lib/promscrape/discovery/gce: discover per-zone instances for gce_sd_config
in parallel. This should reduce discovery latency
2020-05-06 15:00:09 +03:00
Aliaksandr Valialkin
3f52a97f9b
lib/promscrape: add Prometheus-compatible DNS-based service discovery aka dns_sd_configs
2020-05-06 00:01:58 +03:00
Aliaksandr Valialkin
364789c24c
lib/promscrape: properly connect to TCP6 addresses if -enableTCP6
is set
2020-05-06 00:01:57 +03:00
Aliaksandr Valialkin
7b5ef63384
lib/procutil: add NewSighupChan function, which returns a channel, which is triggered on every SIGHUP
2020-05-05 10:54:09 +03:00
Aliaksandr Valialkin
4fa817be10
lib/promscrape: allow explicitly setting empty token via token: ""
in consul_sd_config
2020-05-05 07:50:15 +03:00
Aliaksandr Valialkin
40c3ffb359
lib/promscrape: add Prometheus-compatible service discovery for Consul aka consul_sd_configs
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/330
2020-05-04 20:51:17 +03:00
Aliaksandr Valialkin
83f0e35b7b
lib/promauth: properly set up client certificate in tls.Config
...
Previously the client certificate has been mistakenly set up as a server certificate
2020-05-04 20:51:08 +03:00
Aliaksandr Valialkin
218e566647
lib/promscrape: move common code for discovery api config map handling into discoveryutils
2020-05-04 20:51:01 +03:00
Aliaksandr Valialkin
6310b20e72
lib/promscrape/discovery/kubernetes/: unify apiConfig creation
2020-05-04 20:50:49 +03:00
Aliaksandr Valialkin
66b0ae79a5
lib/promscrape: remove debug line left after the commit e4aac6ea40
2020-05-03 17:15:32 +03:00
Aliaksandr Valialkin
69004a5f67
lib/promscrape: fix tests after the commit 658a8742ac
...
The original commit copies `__address__` label to `instance` label when generating per-target labels as Prometheus does.
See https://www.robustperception.io/life-of-a-label for details.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/453
2020-05-03 16:56:15 +03:00
DexterZhang
658a8742ac
fix(vmagent): different behavior as how prometheus deal with labels. [Issue#453] ( #454 )
2020-05-03 16:51:03 +03:00
Aliaksandr Valialkin
e4aac6ea40
lib/promscrape: make consistent scrape time offsets across reloads for the same ScrapeURL and Labels
...
This should make consistent intervals between data points for scrape targets across reloads.
Previously these intervals were random.
2020-05-03 14:30:21 +03:00
Aliaksandr Valialkin
d9f1b4d6a3
lib/promscrape: fix TestGetFileSDScrapeWorkSuccess after 3b234d82e5
2020-05-03 14:28:31 +03:00
Aliaksandr Valialkin
3b234d82e5
lib/promscrape: reload only modified scrapers on config changes
...
This should improve scrape stability when big number of targets are scraped and these targets are frequently changed.
Thanks to @xbsura for the idea and initial implementation attempts at the following pull requests:
- https://github.com/VictoriaMetrics/VictoriaMetrics/pull/449
- https://github.com/VictoriaMetrics/VictoriaMetrics/pull/458
- https://github.com/VictoriaMetrics/VictoriaMetrics/pull/459
- https://github.com/VictoriaMetrics/VictoriaMetrics/pull/460
2020-05-03 12:45:40 +03:00
Aliaksandr Valialkin
ee810e5f3a
lib/httpserver: rename http.externalURL
to http.pathPrefix
and improve help message for this flag
...
The `http.externalURL` flag name was slightly misleading, so it has been renamed to `http.pathPrefix`.
2020-05-02 13:07:34 +03:00
DexterZhang
34743974d5
feat(httpserver): add http.externalUrl
config to http server, it adds prefix to http path automatically ( #452 )
2020-05-02 12:42:53 +03:00
Aliaksandr Valialkin
432187ac3b
app/vminsert: add /-/reload
handler in the same way as for vmagent
2020-04-30 02:15:39 +03:00
Aliaksandr Valialkin
825a2dd554
lib/procutil: prevent from app termination on SIGHUP signal, since this signal is frequently used for config reload
2020-04-30 02:09:27 +03:00
Aliaksandr Valialkin
01c17092e1
lib/httpserver: mention that -http.maxGracefulShutdownDuration
command-line flag value can be increased on shutdown timeout
2020-04-30 01:38:06 +03:00
Aliaksandr Valialkin
5ec036439d
lib/promscrape: set 30 seconds timeout for discovery api requests
...
Previously such requests could hang for long time. This could make debugging harder.
2020-04-29 17:33:34 +03:00
Aliaksandr Valialkin
43c39dc36c
vendor: use github.com/VictoriaMetrics/fasthttp instead of github.com/fasthttp/fasthttp
...
The upstream fasthttp may contain issues like 996610f021
,
plus a code that isn't used by VictoriaMetrics. So let's use a private copy under our control instead.
2020-04-29 17:33:34 +03:00
Aliaksandr Valialkin
4e4f57b121
lib/metricsql: move it to a separate repository - github.com/VictoriaMetrics/metrics
2020-04-28 15:28:22 +03:00
Aliaksandr Valialkin
83aca79137
lib/storage: recover when metricID->metricName entry is missing in the inverted index after unclean shutdown
...
Newly added index entries can be missing after unclean shutdown, since they didn't flush to persistent storage yet.
Log about this and delete the corresponding metricID, so it could be re-created next time.
2020-04-28 12:00:33 +03:00
Aliaksandr Valialkin
521df0e2fc
lib/promscrape: handle connection reset when targets responds with http redirect
2020-04-28 02:13:02 +03:00
肖贝贝
2b16c188e8
fix: vmagent not follow 301/302 redirect bug ( #445 )
...
Co-authored-by: xiaobeibei <xiaobeibei@bigo.sg>
2020-04-28 01:29:37 +03:00
Aliaksandr Valialkin
303905cd84
lib/{encoding,decimal}: typo fixes in tests: epxecting->expecting
2020-04-28 00:01:55 +03:00
Aliaksandr Valialkin
36fa3078c2
lib/encoding: reduce possibility of failure in TestMarshalInt64ArraySize
2020-04-28 00:01:54 +03:00
Aliaksandr Valialkin
95942f1ac6
lib/promscrape/discovery/gce: make golangci-lint happy
2020-04-27 19:28:10 +03:00
Aliaksandr Valialkin
b768bc9a6a
lib/promscrape: add initial support for Prometheus-compatible service discovery for Amazon EC2 aka ec2_sd_configs
2020-04-27 19:25:53 +03:00
Aliaksandr Valialkin
de59703a16
lib/promscrape/discovery/gce: properly set filter
query arg in api url
2020-04-27 16:01:17 +03:00
Aliaksandr Valialkin
b4afe562c1
lib/storage: postpone reading data from blocks during search
...
This eliminates the need for storing block data into temporary files on a single-node VictoriaMetrics
during heavy queries, which touch big number of time series over long time ranges.
This improves single-node VM performance on heavy queries by up to 2x.
2020-04-27 11:45:24 +03:00
Aliaksandr Valialkin
0224071ebe
lib/promscrape/discovery/gce: allow empty project and zone for gce_sd_config
2020-04-27 11:45:02 +03:00
Aliaksandr Valialkin
6954d0edb7
lib/promscrape/discovery/gce: allow empty zone
arg in gce_sd_config
- in this case zones for the given project are automatically discovered
2020-04-26 14:34:11 +03:00
kreedom
fb967ae6c8
happy fmt
2020-04-26 14:16:32 +03:00
Aliaksandr Valialkin
d7c1ff8b0c
lib/storage: improve deduplication algorithm
...
Now it leaves only the first data point on each `-dedup.minScrapeInterval` interval.
Previously it may leave two data points on the interval. This could lead to unexpected results
for `histogram_quantile(phi, sum(rate(buckets)) by (le))` query.
2020-04-26 13:10:02 +03:00
Aliaksandr Valialkin
491b31b369
lib/storage: postpone label filters matching too many time series instead of giving up with error
...
This should reduce the frequency of the following errors:
cannot find tag filter matching less than N time series; either increase -search.maxUniqueTimeseries or use more specific tag filters
more than N time series found on the time range [...]; either increase -search.maxUniqueTimeseries or shrink the time range
2020-04-24 21:13:50 +03:00
Aliaksandr Valialkin
7b8008e0bd
lib/promscrape/discovery/gce: make golint happy by ignoring resp.Body.Close() result
2020-04-24 18:13:09 +03:00
Aliaksandr Valialkin
9ef5935552
lib/promscrape: initial implementation for gce_sd_configs
aga Prometheus-compatible service discovery for Google Compute Engine
2020-04-24 17:51:22 +03:00
Aliaksandr Valialkin
24461153bf
lib/promscrape: query /api/v1/namespaces/*
for the configured namespaces in kubernetes_sd_config
...
This should fix authroization issues described at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/432
2020-04-24 14:33:50 +03:00
Aliaksandr Valialkin
00e897119f
lib/promscrape: add -promscrape.configCheckInterval
command-line flag for automating config checking
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/431
2020-04-23 23:41:08 +03:00
Aliaksandr Valialkin
a9a7a7175e
lib/promscrape: access Config entries by reference, so they can be compared by addresses
2020-04-23 14:38:20 +03:00
Aliaksandr Valialkin
1c5d14a2eb
lib/promscrape: move KubernetesSDConfig to lib/promscrape/discovery/kubernetes
2020-04-23 11:34:22 +03:00
Aliaksandr Valialkin
a714568374
lib/promscrape/discovery/kubernetes: hide role switch logic behind GetLabels function
2020-04-22 22:16:11 +03:00
Aliaksandr Valialkin
364db13c9c
app/vmselect: add /api/v1/status/tsdb
page with useful stats for locating root cause for high cardinality issues
...
See https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/425
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/268
2020-04-22 22:03:43 +03:00
Aliaksandr Valialkin
2dc5593b75
lib/writeconcurrencylimiter: improve docs for -maxConcurrentInserts command-line flag
2020-04-20 21:03:00 +03:00
Aliaksandr Valialkin
5454b518a6
lib/promscrape/discovery/kubernetes: reuse a client for empty api_server
inside different jobs
2020-04-20 17:07:11 +03:00
Aliaksandr Valialkin
43375df923
lib/promscrape/discovery/kubernetes: update stale comments
2020-04-17 14:06:20 +03:00
Aliaksandr Valialkin
5d1537a395
lib/promscrape: suppress scrape errors if -promscrape.suppressScrapeErrors
flag is set
2020-04-16 23:41:30 +03:00
Aliaksandr Valialkin
600490131f
lib/promscrape: print all the labels for the target on error message for failed scrape
...
This should improve debuggability.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/420
2020-04-16 23:35:05 +03:00
Aliaksandr Valialkin
bd4c6d21dd
lib/promscrape: retry target scraping when the target closes previously established keep-alive connection to it
...
This should fix the following error:
the server closed connection before returning the first response byte. Make sure the server returns 'Connection: close' response header before closing the connection
2020-04-16 23:25:29 +03:00
Aliaksandr Valialkin
2fd2dec5eb
lib/logger: typo fix
2020-04-16 00:19:10 +03:00
Aliaksandr Valialkin
071fdf5518
lib/logger: add WARN level for logging expected errors such as invalid user queries
2020-04-15 20:50:26 +03:00
Aliaksandr Valialkin
6f7f64f757
app/vmselect: handle timestamp(metric offset X)
the same way as Prometheus does
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/415
2020-04-15 12:01:00 +03:00
Aliaksandr Valialkin
426a0567c4
lib/promscrape: code cleanup in runScraper func
2020-04-15 11:36:24 +03:00
Aliaksandr Valialkin
c1de3f67b4
lib/storage: skip metricID if the corresponding metricID->metricName is missing in inverted index during search
...
This case is possible when the corresponding metricID->metricName entry didn't propagate to inverted index yet.
This should fix the following error:
error when searching tsids for tfss [...]: cannot find metricName by metricID 1582417212213420669: EOF
2020-04-15 00:06:43 +03:00
Aliaksandr Valialkin
067c7afebc
lib/promscrape: show information on improperly configured scrape targets at the bottom of /targets
page
...
This is a common error whith improperly configured target autodiscovery and/or relabeling.
This error leads to duplicate scraping of the same targets with the same set of labels, which leads
to duplicate samples in time series.
2020-04-14 14:55:05 +03:00
Aliaksandr Valialkin
ac35635b71
lib/promscrape/discovery/kubernetes: remove only unused client for API server during cleaning
2020-04-14 14:19:21 +03:00
Aliaksandr Valialkin
78863d7066
lib/promscrape: add promrelabel.GetLabelValueByName helper function
2020-04-14 14:12:01 +03:00
Aliaksandr Valialkin
c64f003cfb
lib/promscrape: mention job name in error messages when target cannot be scraped
...
This should improve debuggability
2020-04-14 13:33:13 +03:00
Aliaksandr Valialkin
4718a5d951
lib/promscrape: reset ScrapeWork.ID in tests
2020-04-14 13:31:31 +03:00
Aliaksandr Valialkin
257521a634
lib/promscrape: properly expose statuses for targets with duplicate scrape urls at /targets
page
...
Previously targets with duplicate scrape urls were merged into a single line on the page.
Now each target with duplicate scrape url is displayed on a separate line.
2020-04-14 13:10:01 +03:00
Aliaksandr Valialkin
6a75c95194
lib/promscrape: remove labels starting with __meta_
after applying relabel_configs
as Prometheus does
...
This should reduce CPU load during scraping when target discovery generates
big number of `__meta_*` labels (for instance, k8s discovery).
See https://www.robustperception.io/life-of-a-label for details.
2020-04-14 12:23:22 +03:00
Aliaksandr Valialkin
01d7d799dc
lib/promscrape: rename 'scrape_config->scrape_limit' to 'scrape_config->sample_limit'
...
`scrape_config` block from Prometheus config contains `sample_limit` field,
while in `vmagent` this field was mistakenly named as `scrape_limit`.
2020-04-14 11:59:57 +03:00
Aliaksandr Valialkin
2e4e202c2b
lib/promscrape: add initial support for kubernetes_sd_config
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/334
2020-04-13 21:03:28 +03:00
Aliaksandr Valialkin
2814b1490f
lib/promscrape: add -promscrape.config.strictParse
flag for detecting errors in -promscrape.config
file
2020-04-13 13:15:44 +03:00
Aliaksandr Valialkin
90b4a6dd12
lib/promscrape: extract common auth code to lib/promauth
2020-04-13 12:59:10 +03:00
Aliaksandr Valialkin
4de6c6bbf0
lib/storage: disable deduplication after dedup tests are complete
...
The rest of tests expect that the de-duplication is disabled.
2020-04-10 17:28:31 +03:00
Aliaksandr Valialkin
ded0c0d3c7
lib/storage: correctly handle -dedup.minScrapeInterval
values smaller than 8ms
...
Such small values may be used for removing samples with duplicate timestamps.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/409 for details.
2020-04-10 16:36:41 +03:00
Aliaksandr Valialkin
7d73623c69
lib/{storage,mergeset}: make sure that requests
and misses
cache counters never go down
2020-04-10 14:45:01 +03:00
Aliaksandr Valialkin
e62afc7366
lib/protoparser: add -*TrimTimstamp
command-line flags for Influx, Graphite, OpenTSDB and CSV data
...
These flags can be used for reducing disk space usage for timestamps data ingested over the given protocols
2020-04-10 12:44:39 +03:00
Aliaksandr Valialkin
0681b4c27a
lib/workingsetcache: accumulate stat counters on cache rotation
...
This should prevent from cache stats counters going down after cache rotation,
which may corrupt `cache hit ratio` graph on the official Grafan dasbhoards
when using the following query:
1 - (sum(rate(vm_cache_misses_total[5m])) by (type) / sum(rate(vm_cache_requests_total[5m])) by (type))
2020-04-10 11:51:40 +03:00
Aliaksandr Valialkin
f86947d55c
lib/memory: add more details to -memory.allowedPercent
help message
2020-04-09 15:28:53 +03:00
kreedom
298eb0a0f8
[vmalert] improve external url handling
2020-04-01 22:29:11 +03:00
Aliaksandr Valialkin
cdf0a4cf8f
lib/httpserver: remove unnecessary http.HandlerFunc
wrapper in gzipHandler
2020-04-01 18:14:17 +03:00
Aliaksandr Valialkin
e0d0348f36
lib/storage: add missing reset for tagFilter.matchesEmptyValue on tagFilter.Init
2020-04-01 17:42:44 +03:00
Aliaksandr Valialkin
3e55c7e069
lib/promscrape: reduce timestamp jitter when scraping targets
...
This should improve compression for timestamps
2020-04-01 16:11:35 +03:00
Aliaksandr Valialkin
c4acd20d2a
lib/storage: remove duplicate data points on 7/8*minScrapeInterval interval instead of 1/2*minScrapeInterval
...
This should reduce storage usage and should improve deduplication accuracy
2020-04-01 15:48:48 +03:00
Aliaksandr Valialkin
b699c46046
lib/storage: handle errors returned from TagFilters.Add
when cloning TagFilters with negative filter
2020-03-31 16:18:02 +03:00
Aliaksandr Valialkin
972713bd79
lib/storage: add fast path for the previous indexdb search if it doesn't contain per-day inverted index yet
2020-03-31 12:51:21 +03:00
Aliaksandr Valialkin
5d99ca6cfc
lib/storage: optimize per-day inverted index search for tag filters matching big number of time series
...
- Sort tag filters in the ascending number of matching time series
in order to apply the most specific filters first.
- Fall back to metricName search for filters matching big number of time series
(usually this are negative filters or regexp filters).
2020-03-31 00:48:35 +03:00
Aliaksandr Valialkin
318326c309
lib/storage: properly handle {label=~"foo|"}
filters as Prometheus does
...
Such filters must match all the time series with `label="foo"` plus all the time series without `label`
Previously only time series with `label="foo"` were matched.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/395
2020-03-31 00:48:18 +03:00
Aliaksandr Valialkin
b98ca56d94
lib/envflag: add -envflag.prefix
for setting optional prefix for environment vars
2020-03-30 15:51:19 +03:00
kreedom
bf6c24d0f4
[vmalert] config parser ( #393 )
...
* [vmalert] config parser
* make linter be happy
* fix test
* fix sprintf add test for rule validation
2020-03-29 01:48:30 +02:00
Aliaksandr Valialkin
149f365f74
lib/httpserver: add -http.maxGracefulShutdownDuration
command-line flag for tuning the maximum duration required for graceful shutdown of http server
2020-03-27 21:23:30 +02:00
Aliaksandr Valialkin
047849e855
lib/uint64set: remove zero buckets after Set.Intersect
2020-03-27 01:15:58 +02:00
Aliaksandr Valialkin
f3ec424e7d
lib/uint64set: small code cleanup and perf tuning
...
* Remember the last accessed bucket on Has() call.
* Inline fast paths inside Add() and Has() calls.
* Remove fragile code with maxUnsortedBuckets inside bucket32.
2020-03-25 15:30:25 +02:00
Aliaksandr Valialkin
dde4a97534
lib/uint64set: go fmt
2020-03-24 22:28:43 +02:00
Aliaksandr Valialkin
f3e0c55ea1
lib/storage: serialize snapshot creation process with mutex
...
This guarantees that the snapshot contains all the recently added data
from inmemory buffers when multiple concurrent calls to Storage.CreateSnapshot are performed.
2020-03-24 22:27:05 +02:00
Aliaksandr Valialkin
97fb0edd07
lib/uint64set: added more tests
2020-03-24 22:27:04 +02:00
Aliaksandr Valialkin
df91d2d91f
lib/storage: remove obsolete code
2020-03-13 22:48:17 +02:00
Aliaksandr Valialkin
499594f421
lib/promscrape: allow overriding external_labels as Prometheus does
...
Prometheus docs at https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config say:
> In communication with external systems, they are always applied only
> when a time series does not have a given label yet and are ignored otherwise.
Though this may result in consistency chaos when scrape targets override `external_labels`,
let's stick with Prometheus behavior for the sake of backwards compatibility.
There is last resort in vmagent with `-remoteWrite.label`, which consistently
sets the configured labels to all the metrics before sending them to remote storage.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/366
2020-03-12 20:24:42 +02:00
Aliaksandr Valialkin
fdc2a9d1d7
app/vmselect: add label_map(q, label, srcValue1, dstValue1, ... srcValueN, dstValueN)
function to MetricsQL
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/369
2020-03-12 19:13:47 +02:00
Aliaksandr Valialkin
c8dc1cd218
lib/protoparser/csvimport: add missing metric vm_rows_invalid_total{type="csvimport"}
2020-03-12 15:27:45 +02:00
Aliaksandr Valialkin
d4beb17ebe
lib/promscrape: remove possible races when registering and de-registering scrape workers for /targets
page
2020-03-11 16:30:21 +02:00
Aliaksandr Valialkin
cdf70b7944
lib/promscrape: consistently update /targets
page after SIGHUP
2020-03-11 03:20:03 +02:00
Aliaksandr Valialkin
1fe66fb3cc
app/{vmagent,vminsert}: add support for importing csv data via /api/v1/import/csv
2020-03-10 21:15:35 +02:00
Aliaksandr Valialkin
49d7cb1a3f
all: fix golangci-lint
issues
2020-03-10 19:41:46 +02:00
Aliaksandr Valialkin
7c432da788
lib/promscrape: do not retry idempotent requests when scraping targets
...
This should prevent from the following unexpected side-effects of idempotent request retries:
- increased actual timeout when scraping the target comparing to the configured scrape_timeout
- increased load on the target
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/357
2020-03-09 13:31:52 +02:00
Aliaksandr Valialkin
986dba5ab3
app/vmagent: do not allow non-supported fields in -remoteWrite.relabelConfig
and file_sd_configs
...
This should reduce possible confusion like in the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/363
2020-03-06 20:19:13 +02:00
Aliaksandr Valialkin
c5f894b361
Makefile: add build and test rules with enabled race detector. These rules have -race
suffix
...
Fix also `unsafe pointer conversion` errors detected by Go1.14. See https://golang.org/doc/go1.14#compiler .
2020-03-05 12:03:38 +02:00
Aliaksandr Valialkin
9a944fd169
lib/promscrape: consistency renaming: stopCh -> globalStopCh
2020-03-03 20:08:08 +02:00
Aliaksandr Valialkin
76036c1897
app/vmagent: add -remoteWrite.maxDiskUsagePerURL
for limiting the maximum disk usage for each -remoteWrite.url
buffer
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/352
2020-03-03 19:49:07 +02:00
Aliaksandr Valialkin
1d7ab78b55
lib/protoparser/prometheus: allow trailing comma in tags list
...
The trailing comma is generated by cloudwatch exporter.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/350
2020-03-02 22:22:09 +02:00
Aliaksandr Valialkin
b785429ddb
lib/protoparser: metrics renaming: vm_protoparser_<type>_*
-> vm_protoparser_*{type="<type>"}
...
This should improve composability of these metrics in PromQL queries
2020-02-28 20:20:10 +02:00
Aliaksandr Valialkin
e22fdc1073
lib/persistentqueue: reset chunk file when the persistent queue is empty
2020-02-28 20:05:53 +02:00
Aliaksandr Valialkin
18af31a4c2
all: properly split vm_deduplicated_samples_total
among cluster components
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/345
2020-02-27 23:48:07 +02:00
Aliaksandr Valialkin
6819db5686
lib/envflag: typo fix in docs to -envflag.enable
: envoronment->environment
2020-02-27 21:47:58 +02:00
Aliaksandr Valialkin
b63e4464f4
lib/promscrape: properly reload new configs on SIGHUP
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/335
2020-02-26 13:54:00 +02:00
Aliaksandr Valialkin
6739c2749d
lib/promscrape: go fmt
2020-02-25 20:56:44 +02:00
Aliaksandr Valialkin
7a33da8fea
lib/promscrape: do not add missing port to __address__ label in order to be consistent with Prometheus behavior
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/331
2020-02-25 20:49:50 +02:00
Aliaksandr Valialkin
4e24839a2c
app/vmagent: do not allow sending unpacked requests with sizes exceeding -maxInsertRequestSize
2020-02-25 19:34:41 +02:00
Aliaksandr Valialkin
6386aeb1e0
app/vmagent: add ability to accept Influx line protocol data via TCP and UDP
...
Just set `-influxListenAddr` command-line flag
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/333
2020-02-25 19:12:49 +02:00
Aliaksandr Valialkin
7ef7c9368e
lib/fs: typo fix: read blocks bigger than 8KB via pread()
call instead of using mmap
2020-02-25 18:05:06 +02:00
Aliaksandr Valialkin
fed2959658
lib/envflag: substitute dots with underscores in env var names if -envflag.enable is set
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/311
2020-02-24 21:14:44 +02:00
Aliaksandr Valialkin
04762344c6
app/vmagent: initial implementation for vmagent
2020-02-23 13:36:03 +02:00
Aliaksandr Valialkin
d21cb43e48
lib/storage: add vm_ prefix to deduplicated_samples_total
metric to be conistent with other metrics
2020-02-21 19:33:59 +02:00
Aliaksandr Valialkin
71a52f5f90
lib/protoparser/prometheus: skip leading whitespace from tag names
2020-02-16 19:06:33 +02:00
Aliaksandr Valialkin
ce15cecae4
lib/storage: typo fix
2020-02-16 15:53:44 +02:00
Aliaksandr Valialkin
32e153e834
lib/storage: prevent from clobbering nin-nil lastError in Storage.add
2020-02-16 15:51:26 +02:00
Aliaksandr Valialkin
7b1c7051a3
app/vmselect: add sort_by_label(q, label)
and sort_by_label_desc(q, label)
functions
...
This is implementation of https://github.com/prometheus/prometheus/pull/1533 for VictoriaMetrics.
2020-02-13 17:01:37 +02:00
Aliaksandr Valialkin
7836ad8907
lib/mergeset: skip createing temporary part objects when merging source inmemory parts
...
This should reduce CPU usage when adding new entries to inverted index.
This should alos prevent from creating stalled cleaner goroutines for the created temporary parts,
since they were never closed.
This should fix the following issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/316 .
2020-02-13 14:09:32 +02:00
Aliaksandr Valialkin
eceaf13e5e
lib/{storage,mergeset}: use time.Ticker instead of time.Timer where appropriate
...
It has been appeared that time.Timer was used in places where time.Ticker must be used instead.
This could result in blocked goroutines as in the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/316 .
2020-02-13 13:10:07 +02:00
Aliaksandr Valialkin
4cc0163c7c
docs: migrate ExtendedPromQL->MetricsQL in order to be more consistent
2020-02-10 23:02:43 +02:00
Aliaksandr Valialkin
025297f15d
lib/envflag: check for incorrect flag values read from environment vars
2020-02-10 16:08:10 +02:00
Aliaksandr Valialkin
5d207b2025
lib/envflag: add -envflag.enable
command-line flag for enabling reading flags from environment vars
...
By default flags are read only from command line. They can be read from environment vars if `-envflag.enable` is set.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/311
2020-02-10 16:02:37 +02:00
Aliaksandr Valialkin
8466ab0034
all: allow setting flags via environment vars
...
Now flags can be set via environment vars with the same names as flags.
Command-line flags override flags set via env vars.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/311
2020-02-10 13:29:13 +02:00
Aliaksandr Valialkin
e210cd9da1
lib/storage: move -dedup.minScrapeInterval
flag outside lib/storage, so it doesnt show up in vminsert
in cluster version
2020-02-10 13:09:51 +02:00
Aliaksandr Valialkin
40c6ae2952
lib/logger: initialize output to os.Stderr by default
2020-02-04 22:40:44 +02:00
Aliaksandr Valialkin
e0a4c37fc1
lib/logger: add -loggerOutput
command-line flag
...
This flag allows changing log output from `stderr` to `stdout` if `-loggerOutput=stdout` is set.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/306
2020-02-04 21:47:56 +02:00
Aliaksandr Valialkin
7f3e3a6034
lib/logger: do not clutter -loggerFormat=json
output with stack trace
...
This should improve json parsing
2020-02-04 21:37:25 +02:00
Aliaksandr Valialkin
bd4698bb7a
lib/storage: do not deduplicate blocks with less than 32 samples during merge
...
This should improve deduplication accuracy for blocks with higher number of samples.
2020-02-04 18:41:54 +02:00
Aliaksandr Valialkin
42864bb52f
all: do not clash flag description with back-quoted flag types
...
See https://golang.org/pkg/flag/#PrintDefaults for more details.
2020-02-04 15:46:52 +02:00
Aliaksandr Valialkin
011a79da85
lib/fs: remove unused readerAt
interface
2020-01-31 15:12:43 +02:00
Aliaksandr Valialkin
c3d86eef96
all: add -dedup.minScrapeInterval
command-line flag for data de-duplication
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/86
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/278
2020-01-31 01:16:57 +02:00
Aliaksandr Valialkin
2152f6f0cd
lib/storage: re-use indexSearch inside Storage.prefetchMetricNames
2020-01-31 01:16:53 +02:00
Aliaksandr Valialkin
d70ba7eb37
lib/fs: optimize small reads for ReaderAt.MustReadAt
by reading from memory-mapped space instead of reading from file descriptor
...
This should improve performance when reading many small blocks.
2020-01-30 15:09:05 +02:00
Aliaksandr Valialkin
ad8af629bb
all: rename ReadAt* to MustReadAt* in order to dont clash with io.ReaderAt
2020-01-30 15:08:58 +02:00
Aliaksandr Valialkin
d68546aa4a
lib/storage: pre-fetch metricNames for the found metricIDs in Search.Init
...
This should speed up Search.NextMetricBlock loop for big number of found time series.
2020-01-30 15:08:51 +02:00
Aliaksandr Valialkin
5bb9ccb6bf
lib/mergeset: properly update lastAccesstime
in indexBlockCache entries
...
This is a follow-up for 6665f10e7b
2020-01-29 21:20:47 +02:00
Aliaksandr Valialkin
a462355b2f
app/vmselect/promql: add keep_next_value(q)
for filling gaps with the next non-empty value
2020-01-29 00:48:04 +02:00
Aliaksandr Valialkin
371e86194d
app/vminsert: moved -maxInsertRequestSize
command-line flag out of lib/prompb
in order to prevent its inclusion in vmselect
and vmstorage
apps
2020-01-28 23:02:08 +02:00
Aliaksandr Valialkin
6caa9bb51b
lib/logger: fix improperly set skipframes for all the logging functions
...
The bug has been introduced in the previous commit f6baee6efe
2020-01-26 18:34:27 +02:00
Aliaksandr Valialkin
f6baee6efe
lib/httpserver: log the caller of httpserver.Errorf
...
Previously log message contained `httpserver.Errorf`, not it contains the caller of `httpserver.Errorf`, which is more useful.
2020-01-25 20:17:59 +02:00
Aliaksandr Valialkin
9df5b2d1c3
app/victoria-metrics: add -selfScrapeInterval
flag for self-scraping /metrics
page
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/30
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/180
2020-01-25 19:19:59 +02:00
Aliaksandr Valialkin
2a0a0ed14d
lib/protoparser: add parser for Prometheus exposition text format
...
This parser will be used by vmagent
2020-01-24 20:11:02 +02:00
Aliaksandr Valialkin
6456c93dbb
app/vminsert: move ingestion protocol parsers to lib/protoparser, so they could be re-used in the upcoming vmagent
2020-01-24 16:53:00 +02:00
Aliaksandr Valialkin
680080887d
all: consistently log durations in seconds with millisecond precision
...
This should improve logs readability
2020-01-22 18:28:27 +02:00
Aliaksandr Valialkin
582dd01f42
app/vmselect/promql: add range_over_time(m[d])
function for calculating value range for m
over d
2020-01-21 19:05:17 +02:00
Aliaksandr Valialkin
36973ee975
app/vmselect/promql: add label_match(q, label, regexp)
and label_mismatch(q, label, regexp)
functions for filtering out time series with labels matching the given regexp
2020-01-21 15:00:20 +02:00
Aliaksandr Valialkin
6665f10e7b
lib/{mergeset,storage}: properly update lastAccessTime
in index and data block cache entries
2020-01-20 14:59:47 +02:00
Aliaksandr Valialkin
257e61195a
lib/uint64set: add missing bucket32.b16his values
2020-01-18 14:26:04 +02:00
Aliaksandr Valialkin
4cc0c44b9e
lib/uint64set: optimize Set.Union
...
This should improve performance for queries over big number of time series
2020-01-18 13:47:03 +02:00
Aliaksandr Valialkin
1b5f02e293
lib/uint64set: add benchmarks for Set.Union
2020-01-18 13:47:02 +02:00
Aliaksandr Valialkin
3748fb24b6
lib/storage: skip recovering timestamps order for lossless compression (PrecisionBits=64)
2020-01-18 00:09:33 +02:00
Aliaksandr Valialkin
c9472e4f3a
all: use github.com/klauspost/compress/gzip
instead of compress/gzip
...
`github.com/klauspost/compress/gzip` is more optimized than `compress/gzip`.
This gives better gzip compression and decompression speeds.
2020-01-17 23:58:46 +02:00
Aliaksandr Valialkin
bc0f897fcb
lib/uint64set: reduce memory allocations in Set.AppendTo
2020-01-17 22:33:09 +02:00
Aliaksandr Valialkin
f9289b804a
lib/storage: reduce memory allocations when merging metricID sets
2020-01-17 22:10:44 +02:00
Aliaksandr Valialkin
0c8ad08578
lib/uint64set: typo fix in Set.Intersect
2020-01-17 18:10:58 +02:00
Aliaksandr Valialkin
9f027ec176
lib/uint64set: optimize Intersect, Subtract and Union functions
...
This should improve performance for queries over big number of time series.
2020-01-17 16:11:49 +02:00
Aliaksandr Valialkin
cd53f7d177
lib/uint64set: improve benchmark for Set.Intersect
2020-01-17 16:08:17 +02:00
Aliaksandr Valialkin
8dbf430469
lib/uint64set: add benchmark for Set.Intersect
2020-01-17 00:31:07 +02:00
Aliaksandr Valialkin
0d7505b00e
all: mention command-line flags used for limiting the incoming request size in error messages
...
This should improve error logs usability.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/287
2020-01-16 13:03:30 +02:00
Aliaksandr Valialkin
605d588ba6
lib/uint64set: reduce memory usage in Union, Intersect and Subtract methods
...
Iterate items with newly added Set.ForEach method instead of allocating `[]uint64`
slice for all the items before the iteration.
2020-01-15 12:12:49 +02:00
Aliaksandr Valialkin
893b62c682
lib/{mergeset,storage}: fix uint64 counters alignment for 32-bit architectures (GOARCH=386, GOARCH=arm)
2020-01-14 22:47:04 +02:00
Aliaksandr Valialkin
7830c10eb2
lib/{storage,mergeset}: gradually remove stale entries from block cache and index caches
...
This should reduce memory usage in the long run when old blocks and indexes
aren't accessed anymore.
2020-01-14 21:38:44 +02:00
Aliaksandr Valialkin
8b14572f70
app/vmselect/promql: add hoeffding_bound_upper(phi, m[d])
and hoeffding_bound_lower(phi, m[d])
functions
...
These functions can be used for calculating Hoeffding bounds
for `m` over `d` time range and for the given `phi` in the range `[0..1]`.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/283
2020-01-11 14:46:23 +02:00
Aliaksandr Valialkin
164278151f
app/vmselect/promql: add aggr_over_time(("aggr_func1", "aggr_func2", ...), m[d])
function
...
This function can be used for simultaneous calculating of multiple `aggr_func*` functions
that accept range vector. For example, `aggr_over_time(("min_over_time", "max_over_time"), m[d])`
would calculate `min_over_time` and `max_over_time` for `m[d]`.
2020-01-10 21:18:06 +02:00
Aliaksandr Valialkin
c4632faa9d
app/vmselect/promql: add tmin_over_time(m[d])
and tmax_over_time(m[d])
functions
...
These functions return timestamp in seconds for the minimum and maximum value for `m` over time range `d`
2020-01-10 19:39:28 +02:00
Aliaksandr Valialkin
b8038a14e7
lib/backup/s3remote: check whether the file exists before deleting it
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/284
2020-01-09 23:20:31 +02:00
Aliaksandr Valialkin
f358fb72d1
app/{vmbackup,vmrestore}: add backup complete
file to backup when it is complete and check for this file before restoring from backup
...
This should prevent from restoring from incomplete backups.
Add `-skipBackupCompleteCheck` command-line flag to `vmrestore` in order to be able restoring from old backups without `backup complete` file.
2020-01-09 15:35:38 +02:00
Aliaksandr Valialkin
fc71602039
lib/storage: limit maxRaRowsPerPartition by 500K for any number of rawRowsShardsPerPartition
...
This should reduce write amplification for high ingestion rate on multi-CPU systems
2020-01-04 23:57:31 +02:00
Aliaksandr Valialkin
d9c4ac9978
lib/metricsql: export IsRollupFunc and IsTransformFunc, since they can be used by package users
2020-01-04 13:25:05 +02:00
Aliaksandr Valialkin
f409f2d050
app/vmselect/promql: add histogram_share(le, buckets)
function
2020-01-04 12:45:55 +02:00
Aliaksandr Valialkin
b1ded7cf9a
app/vmselect/promql: add absent_over_time(m[d])
func similar to the function in Prometheus 2.16
...
See https://github.com/prometheus/prometheus/issues/2882
2020-01-04 12:45:07 +02:00
Aliaksandr Valialkin
a8360d04c0
app/vmselect/promql: add histogram_over_time(m[d])
rollup function
2020-01-04 12:44:56 +02:00
Aliaksandr Valialkin
695682232f
lib/uint64set: reduce memory usage when storing big number of sparse metric_id values
2020-01-03 18:16:44 +02:00
Aliaksandr Valialkin
cb3a342882
app/vmselect/promql: add share_le_over_time
and share_gt_over_time
functions for SLI and SLO calculations
2020-01-03 00:41:16 +02:00
Aliaksandr Valialkin
0038365206
docs: refer to standalone MetricsQL package
2020-01-02 23:43:35 +02:00
Aliaksandr Valialkin
a21d786d3c
lib/metricsql: add example for ExpandWithExprs
2019-12-26 21:32:11 +02:00
Aliaksandr Valialkin
6f67e0b56b
lib/metricsq: add ExpandWithExprs
2019-12-25 22:20:30 +02:00
Aliaksandr Valialkin
1925ee038d
Rename lib/promql to lib/metricsql and apply small fixes
2019-12-25 22:03:59 +02:00
Mike Poindexter
bec62e4e43
Split Extended PromQL parsing to a separate library
2019-12-25 22:03:51 +02:00
Aliaksandr Valialkin
c18802af59
lib/fs: typo fix in fadvise_unix.go
2019-12-24 20:59:28 +02:00
Aliaksandr Valialkin
4ba4abe666
lib/encoding: log the compressed block contents if it cannot be decompressed or unmarshaled
...
This should help detecting the root cause of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281
2019-12-24 20:48:31 +02:00
Aliaksandr Valialkin
5bb39e757b
lib/encoding: mention src contents in error message returned from unmarshalInt64NearestDelta*
...
This should simplify detecting the root cause of the issue at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281
2019-12-24 20:41:52 +02:00
Aliaksandr Valialkin
d5c9841220
lib/encoding: mention unpacked block size in the error message if unparsed tail left
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281
2019-12-24 20:35:13 +02:00
Aliaksandr Valialkin
31d04fb5df
Revert "lib/logger: prevent from blocking when log output isn't consumed in timely manner"
...
This reverts commit e3c462f08a
.
Reason to revert: this leaves incomplete logs on app shutdown.
2019-12-24 12:21:39 +02:00
Aliaksandr Valialkin
5b75984aa9
app/vmselect/netstorage: move MustAdviseSequentialRead to lib/fs
2019-12-23 23:16:11 +02:00
Aliaksandr Valialkin
6a1499efa3
lib/encoding/zstd: prevent from possible encoder leak when concurrent goroutines create encoders for the same compressionLevel
...
Thanks to @klauspost for the pointer to this issue. See https://github.com/klauspost/compress/issues/195 for details.
2019-12-23 18:05:41 +02:00
Aliaksandr Valialkin
e3c462f08a
lib/logger: prevent from blocking when log output isn't consumed in timely manner
...
Drop log messages instead of blocking and increment `vm_log_messages_dropped_total` metric.
2019-12-20 11:49:34 +02:00
Aliaksandr Valialkin
1825893eef
lib/storage: scale ingestion performance by sharding rawRows on systems with more than 8 CPU cores
2019-12-19 18:18:29 +02:00
Aliaksandr Valialkin
97f70ccda7
lib/storage: optimize bulk import performance when multiple data points are inserted for the same time series
...
This should speed up `/api/v1/import` and make it more scalable on multi-core systems.
2019-12-19 18:18:29 +02:00
Aliaksandr Valialkin
bb530a0591
lib/httpserver: inline checkAuth code to make it more clear
2019-12-18 23:06:25 +02:00
koalaty-code
aea4c80dd7
Ignore /health endpoint when checking auth
2019-12-18 23:04:31 +02:00
Aliaksandr Valialkin
bc645152cb
app/vminsert: simultaneously accept telnet put
and HTTP /api/put
OpenTSDB metrics at -opentsdbListenAddr
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/266
2019-12-14 00:30:12 +02:00
Aliaksandr Valialkin
f5ac9b0721
lib/logger: add -loggerFormat
for choosing log message formats
...
Supported formats: default, json
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/265
2019-12-13 15:10:05 +02:00
Aliaksandr Valialkin
3c076544bf
app/vmselect/promql: allow negative offsets
...
Updates https://github.com/prometheus/prometheus/issues/6282
2019-12-11 01:00:23 +02:00
Aliaksandr Valialkin
0ed9258545
lib/{mergeset,storage}: log info message when both source and destination part paths from txn are missing during startup
...
This is expected condition after unclean shutdown (OOM, hard reset, `kill -9`) on NFS disk.
2019-12-09 15:44:53 +02:00
Aliaksandr Valialkin
72345eb5bd
lib/{mergeset,storage}: make sure pending transaction deletions are finished before and after runTransactions
call.
...
`runTransactions` call issues async deletions for transaction files. The previously issued transaction deletions
can race with the next call to `runTransactions`. Prevent this by waiting until all the pending transaction
deletions are funished in the beginning of `runTransactions`. Also make sure that all the pending transaction
deletions are finished before returning from `runTransactions`.
2019-12-04 21:40:30 +02:00
Aliaksandr Valialkin
1244ad810d
lib/httpserver: add /ping
handler for compatibility with Influx agents
...
Certain Influx agents check for `/ping` endpoint before starting
to send Influx line protocol data. See https://docs.influxdata.com/influxdb/v1.7/tools/api/#ping-http-endpoint
2019-12-04 19:15:52 +02:00
Aliaksandr Valialkin
a247236f61
lib/storage: fall back to global inverted index if a filter match too many time series in per-day index
...
Previously this resulted to error message. The query may succeed via search in global index.
2019-12-03 14:48:31 +02:00
Aliaksandr Valialkin
54741ee578
lib/storage: fix printing tag filters in TagFilters.String
2019-12-03 14:25:13 +02:00
Aliaksandr Valialkin
efbc83a13e
lib/storage: print __name__
instead of empty string in user-visible tag filters
2019-12-03 14:18:28 +02:00
Aliaksandr Valialkin
f52874dab4
lib/storage: optimize regexp filter search
2019-12-03 00:43:12 +02:00
Aliaksandr Valialkin
638a5cbb16
lib/{mergeset,storage}: remove transaction files only after the mentioned dirs are really removed
...
This should fix the issue on NFS when incompletely removed dirs may be left
after unclean shutdown (OOM, kill -9, hard reset, etc.), while the corresponding transaction
files are already removed.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162
2019-12-02 21:36:31 +02:00
Aliaksandr Valialkin
20812008a7
lib/storage: remove metricID with missing metricID->metricName entry
...
The metricID->metricName entry can be missing in the indexdb after unclean shutdown
when only a part of entries for new time series is written into indexdb.
Recover from such a situation by removing the broken metricID. New metricID
will be automatically created for time series with the given metricName
when new data point will arive to it.
2019-12-02 20:46:44 +02:00
Aliaksandr Valialkin
62a915f2b2
lib/storage: protect from time drift during indexdb rotation
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/248
2019-12-02 14:44:42 +02:00
Aliaksandr Valialkin
42da569bcd
lib/logger: merge file
and line
labels into location="file:line"
...
This should improve the usability for `vm_log_messages_total` metric during practical queries
2019-12-02 14:44:40 +02:00
Aliaksandr Valialkin
70b8191fab
lib/storage: generate more human-friendly result in TagFilters.String
2019-12-02 13:52:22 +02:00
Aliaksandr Valialkin
542b9c2043
lib/logger: consistency renaming from vm_log_messages_count
to vm_log_messages_total
, since this is a counter
2019-12-02 00:49:00 +02:00
Aliaksandr Valialkin
c567919f80
lib/logger: track the number of log messages by (level, file, line)
in the vm_log_messages_count
metric
2019-12-01 18:37:49 +02:00
Aliaksandr Valialkin
761645b20a
lib/netutil: use IPv6 for both listening and dialing if -enabledTCP6
is set
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/244
2019-12-01 02:57:13 +02:00
Aliaksandr Valialkin
f733cb2186
lib/backup: cosmetic fixes after #243
2019-11-29 18:07:04 +02:00
glebsam
15b7406f7b
Add option to provide custom endpoint for S3, add option to specify S3 config profile ( #243 )
...
* Add option to provide custom endpoint for S3 for use with s3-compatible storages, add option to specify S3 config profile
* make fmt
2019-11-29 17:59:56 +02:00
Aliaksandr Valialkin
9010c6a1d6
lib/netutil: add -enableTCP6
command-line flag for enabling listening for IPv6 additionally to IPv4 TCP ports
2019-11-29 17:32:47 +02:00
Aliaksandr Valialkin
a7125a5b7b
lib/backup: remove flock.lock
file in empty dirs
...
This fixes an issue when VictoriaMetrics doesn't see the restored data after the following operations:
1. Stop VictoriaMetrics.
2. Delete `<-storageDataPath>` dir.
3. Start VictoriaMetrics, then stop it.
4. Restore data from backup with `vmrestore`.
5. Start VictoriaMetrics.
`vmrestore` didn't delete properly empty dirs in `<-storageDataPath>/indexdb` because of the remaining `flock.lock` files in these dirs.
2019-11-28 13:38:58 +02:00
Aliaksandr Valialkin
14bcff6015
lib/httpserver: improve docs for -tls*
flags to be more clear
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/242
2019-11-26 18:08:35 +02:00
Aliaksandr Valialkin
da98703748
app/vmselect/promql: optimize binary search over big number of samples during rollup calculations
2019-11-25 14:01:46 +02:00
Aliaksandr Valialkin
5285acae3e
lib/decimal: calculate ln2/ln10 constant during compile time
2019-11-23 15:52:58 +02:00
Aliaksandr Valialkin
9d1fcb2be6
vendor: update github.com/VictoriaMetrics/metrics from v1.7.2 to v1.8.0. This version supports histograms
2019-11-23 00:20:27 +02:00
Aliaksandr Valialkin
7a4635f853
all: remove the remaining mentions of cluster version
2019-11-21 23:18:22 +02:00
Aliaksandr Valialkin
3e9b7addb1
lib/httpserver: typo fix in -httpAuth.password
command-line description
2019-11-21 21:54:26 +02:00
Aliaksandr Valialkin
f652c0f40f
lib/storage: move non-matching tag filters to the top at matchTagFilters
...
This should reduce the amount of useless work needed for matching the next metricNames.
2019-11-21 21:35:13 +02:00
Aliaksandr Valialkin
b8cde6cce1
lib/storage: speed up time series search for queries with multiple filters
...
Use optimized specialized binary search for uint64 metricIDs instead of generic sort.Search.
2019-11-21 18:43:17 +02:00
Aliaksandr Valialkin
5c1e4143e9
lib/storage: verify the number of returned metricIDs in BenchmarkHeadPostingForMatchers
2019-11-20 15:39:28 +02:00
Aliaksandr Valialkin
52d7ca6bf0
lib/decimal: increase decimal->float speed conversion for integer numbers
2019-11-20 13:04:34 +02:00
Aliaksandr Valialkin
75eeea21ee
lib/decimal: reduce rounding error when converting from decimal to float with negative exponent
...
While at it, slightly increase the conversion performance by moving fast path to the top of the loop.
2019-11-19 23:35:33 +02:00
Aliaksandr Valialkin
cfb9fa2100
lib/backup: retrieve only the required metadata when reading GCS objects
2019-11-19 21:06:34 +02:00
Aliaksandr Valialkin
443189fb0a
app/{vmbackup,vmrestore}: add -maxBytesPerSecond
command-line flag for limiting the used network bandwidth during backup / restore
2019-11-19 20:31:52 +02:00
Aliaksandr Valialkin
2db06f0ef8
lib/backup: prevent from restoring to directory which is in use by VictoriaMetrics during the restore
2019-11-19 18:36:23 +02:00
Aliaksandr Valialkin
b6f22a62cb
lib/storage: increase the number of created time series in BenchmarkHeadPostingForMatchers in order to be on par with Promethues
...
The previous commit was accidentally creating 10x smaller number of time series than Prometheus
and this led to invalid benchmark results.
The updated benchmark results:
benchmark old ns/op new ns/op delta
BenchmarkHeadPostingForMatchers/n="1" 272756688 6194893 -97.73%
BenchmarkHeadPostingForMatchers/n="1",j="foo" 138132923 10781372 -92.19%
BenchmarkHeadPostingForMatchers/j="foo",n="1" 134723762 10632834 -92.11%
BenchmarkHeadPostingForMatchers/n="1",j!="foo" 195823953 10679975 -94.55%
BenchmarkHeadPostingForMatchers/i=~".*" 7962582919 100118510 -98.74%
BenchmarkHeadPostingForMatchers/i=~".+" 7589543864 154955671 -97.96%
BenchmarkHeadPostingForMatchers/i=~"" 1142371741 258003769 -77.42%
BenchmarkHeadPostingForMatchers/i!="" 9964150263 159783895 -98.40%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",j="foo" 216995884 10937895 -94.96%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",i!="2",j="foo" 202541348 10990027 -94.57%
BenchmarkHeadPostingForMatchers/n="1",i!="" 486285711 87004349 -82.11%
BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo" 350776931 53342793 -84.79%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo" 380888565 54256156 -85.76%
BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo" 89500296 21823279 -75.62%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo" 379529654 46671359 -87.70%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.*",j="foo" 424563825 53915842 -87.30%
VictoriaMetrics uses 1GB of RAM during the benchmark (vs 3.5GB of RAM for Prometheus)
2019-11-18 19:50:58 +02:00
Aliaksandr Valialkin
8a0dfc6220
lib/storage: add BenchmarkHeadPostingForMatchers similar to the benchmark from Prometheus
...
See the corresponding benchmark in Prometheus - 23c0299d85/tsdb/head_bench_test.go (L52)
The benchmark allows performing apples-to-apples comparison of time series search
in Prometheus and VictoriaMetrics. The following article - https://www.robustperception.io/evaluating-performance-and-correctness -
contains incorrect numbers for VictoriaMetrics, since there wasn't this benchmark yet. Fix this.
Benchmarks can be repeated with the following commands from Prometheus and VictoriaMetrics source code roots:
- Prometheus: GOMAXPROCS=1 go test ./tsdb/ -run=111 -bench=BenchmarkHeadPostingForMatchers
- VictoriaMetrics: GOMAXPROCS=1 go test ./lib/storage/ -run=111 -bench=BenchmarkHeadPostingForMatchers
Benchmark results:
benchmark old ns/op new ns/op delta
BenchmarkHeadPostingForMatchers/n="1" 272756688 364977 -99.87%
BenchmarkHeadPostingForMatchers/n="1",j="foo" 138132923 1181636 -99.14%
BenchmarkHeadPostingForMatchers/j="foo",n="1" 134723762 1141578 -99.15%
BenchmarkHeadPostingForMatchers/n="1",j!="foo" 195823953 1148056 -99.41%
BenchmarkHeadPostingForMatchers/i=~".*" 7962582919 8716755 -99.89%
BenchmarkHeadPostingForMatchers/i=~".+" 7589543864 12096587 -99.84%
BenchmarkHeadPostingForMatchers/i=~"" 1142371741 16164560 -98.59%
BenchmarkHeadPostingForMatchers/i!="" 9964150263 12230021 -99.88%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",j="foo" 216995884 1173476 -99.46%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",i!="2",j="foo" 202541348 1299743 -99.36%
BenchmarkHeadPostingForMatchers/n="1",i!="" 486285711 11555193 -97.62%
BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo" 350776931 5607506 -98.40%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo" 380888565 6380335 -98.32%
BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo" 89500296 2078970 -97.68%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo" 379529654 6561368 -98.27%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.*",j="foo" 424563825 6757132 -98.41%
The first column (old) is for Prometheus, the second column (new) is for VictoriaMetrics.
As you can see, VictoriaMetrics outperforms Prometheus by more than 100x in almost all the test cases of this benchmark.
Prometheus was using 3.5GB of RAM during the benchmark, while VictoriaMetrics was using 400MB of RAM.
2019-11-18 18:45:06 +02:00
Aliaksandr Valialkin
2ab4cea5e5
lib/storage: always start using per-day inverted index on the next day after its creation
...
The current day could miss entries for already stopped time series before
enabling per-day index.
This fixes the issue when queries return empty results during the first hour after
upgrading to v1.29.*
2019-11-16 12:11:25 +02:00
Aliaksandr Valialkin
119dfd01bb
lib/storage: add vm_cache_size_bytes{type="storage/hour_metric_ids"}
metric
2019-11-13 20:24:21 +02:00
Aliaksandr Valialkin
86a1cd700b
lib/storage: remove inmemory index for recent hour, since it uses too much memory
...
Production workload shows that the index requires ~4Kb of RAM per active time series.
This is too much for high number of active time series, so let's delete this index.
Now the queries should fall back to the index for the current day instead of the index
for the recent hour. The query performance for the current day index should be good enough
given the 100M rows/sec scan speed per CPU core.
2019-11-13 17:58:07 +02:00
Aliaksandr Valialkin
33895d4a0f
lib/storage: add missing increment for recentHourInvertedIndexSearchCalls
2019-11-13 15:13:51 +02:00
Aliaksandr Valialkin
c57eb0ff83
lib/storage: add -disableRecentHourIndex
flag for disabling inmemory index for recent hour
...
This may be useful for saving RAM on high number of time series aka high cardinality
2019-11-13 15:02:51 +02:00
Aliaksandr Valialkin
e14ab14e54
lib/storage: verify marshaling for iidx.pendingMetricIDs in TestInmemoryInvertedIndexMarshalUnmarshal
2019-11-13 13:35:30 +02:00
Aliaksandr Valialkin
ca259864e2
lib/storage: return back inmemory inverted index for recent hour
...
Issues fixed:
- Slow startup times. Now the index is loaded from cache during start.
- High memory usage related to superflouos index copies every 10 seconds.
2019-11-13 13:11:04 +02:00
Aliaksandr Valialkin
01bb3c06c7
lib/storage: remove inmemory inverted index for recent hours
...
Production load with >10M active time series showed it could
slow down VictoriaMetrics startup times and could eat
all the memory leading to OOM.
Remove inmemory inverted index for recent hours until thorough
testing on production data shows it works OK.
2019-11-13 10:45:53 +02:00
Aliaksandr Valialkin
6c2303764e
Revert "lib/fs: do not postpone directory removal on NFS error"
...
This reverts commit 4c02e496f7
.
Reason for revert: the commit breaks on NFS - see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/234
2019-11-12 16:18:09 +02:00
Mike Poindexter
f3ad330635
Add test for invalid caching of tsids ( #232 )
...
* Add test for invalid caching of tsids
* Clean up error handling
2019-11-12 15:09:33 +02:00
Oleg Kovalov
b4f44befa3
fix misspelled words ( #229 )
2019-11-12 00:16:42 +02:00
Aliaksandr Valialkin
8e8f98f712
lib/storage: add tests for dateMetricIDCache
2019-11-11 13:21:57 +02:00
Aliaksandr Valialkin
c342f5e37e
lib/storage: eliminate data race when updating lastSyncTime in dateMetricIDCache.Has
2019-11-10 22:04:01 +02:00
Aliaksandr Valialkin
4c02e496f7
lib/fs: do not postpone directory removal on NFS error
...
Continue trying to remove NFS directory on temporary errors for up to a minute.
The previous async removal process breaks in the following case during VictoriaMetrics start
- VictoriaMetrics opens index, finds incomplete merge transactions and starts replaying them.
- The transaction instructs removing old directories for parts, which were already merged into bigger part.
- VictoriaMetrics removes these directories, but their removal is delayed due to NFS errors.
- VictoriaMetrics scans partition directory after all the incomplete merge transactions are finished
and finds directories, which should be removed, but weren't still removed due to NFS errors.
- VictoriaMetrics panics when it finds unexpected empty directory.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162
2019-11-10 13:24:51 +02:00
Aliaksandr Valialkin
3956003dd0
lib/storage: reorganize the code in getStartDateForPerDayInvertedIndex according to golangci-lint
2019-11-10 00:38:59 +02:00
Aliaksandr Valialkin
ee7765b10d
lib/storage: implement per-day inverted index
2019-11-10 00:02:46 +02:00
Aliaksandr Valialkin
5810ba57c2
lib/storage: use specialized cache for (date, metricID) entries
...
This improves ingestion performance.
2019-11-09 23:06:11 +02:00
Aliaksandr Valialkin
e573ef2126
lib/storage: remove unused code from getMetricIDsForTimeRange: it is expected that time range is always non-zero
2019-11-09 19:03:34 +02:00
Aliaksandr Valialkin
823fa085ef
lib/storage: properly set time range when deleting time series
2019-11-09 18:49:49 +02:00
Aliaksandr Valialkin
695c1dc5eb
lib/storage: obtain all the time series ids from (tag->metricIDs) rows instead of (metricID->TSID) rows, since this much faster
2019-11-09 18:04:33 +02:00
Aliaksandr Valialkin
cdbe848102
lib/storage: small code prettifying
2019-11-09 14:19:52 +02:00
Aliaksandr Valialkin
5c25070556
lib/uint64set: remove superflouos check for item existence before deleting it in Set.Subtract
2019-11-09 14:19:47 +02:00
Aliaksandr Valialkin
bb08bab263
lib/storage: inmemoryInvertedIndex prettifying
2019-11-09 14:19:41 +02:00
Aliaksandr Valialkin
6ad7fe8eeb
lib/storage: export vm_new_timeseries_created_total
metric for determining time series churn rate
2019-11-08 21:21:07 +02:00
Aliaksandr Valialkin
9ea549ed24
lib/storage: sync with cluster changes
2019-11-08 21:21:07 +02:00
Aliaksandr Valialkin
d888b21657
lib/storage: add inmemory inverted index for the last hour
...
It should improve performance for `last N hours` dashboards with update intervals smaller than 1 hour.
2019-11-08 21:21:07 +02:00
Aliaksandr Valialkin
1e46961d68
app/{vmbackup,vmrestore}: add vmbackup
and vmrestore
tools for creating backups on s3 or gcs from instant snapshots
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/203
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/38
2019-11-08 21:21:07 +02:00
Aliaksandr Valialkin
543dc8d337
lib/storage: populate partition names from both small
and big
directories
...
Certain partition directories may be missing after restoring from backups
if they had no data. Re-create such directories on start.
2019-11-06 19:49:34 +02:00
Aliaksandr Valialkin
e472f0b23b
lib/storage: substitute error message about unsorted items in the index block after metricIDs merge with counter
...
The origin of the error has been detected and documented in the code,
so it is enough to export a counter for such errors at `vm_index_blocks_with_metric_ids_incorrect_order_total`,
so it could be monitored and alerted on high error rates.
Export also the counter for processed index blocks with metricIDs - `vm_index_blocks_with_metric_ids_processed_total`,
so its' rate could be compared to `rate(vm_index_blocks_with_metric_ids_incorrect_order_total)`.
2019-11-06 14:28:11 +02:00
Aliaksandr Valialkin
c51ca04a43
lib/storage: take into account the requested time range when caching TSIDs for the given tag filters
2019-11-06 14:28:11 +02:00
Aliaksandr Valialkin
e37f06dc52
lib/storage: dump incorrectly sorted items on a single line; this should simplify error reporting
2019-11-05 18:44:22 +02:00
Aliaksandr Valialkin
5c2099ecfe
lib/storage: return back finalPartsToMerge from 2 to 3 in order to prevent from excessive merges in old partitions
2019-11-05 17:27:48 +02:00
Aliaksandr Valialkin
885ba17905
lib/storage: separate the max inverted index scan loops per metric into fast and slow loops
...
Slow loops could require seeks and expensive regexp matching, while fast loops just scans
all the metricIDs for the given `tag=value` prefix. So these operations must have separate
max loops multiplier.
2019-11-05 17:27:48 +02:00
Aliaksandr Valialkin
b9a06e8e74
lib/storage: skip repeated useless work when intersection of metricIDs with the given filter is too expensive
...
This should improve performance for query filters over big number of time series.
2019-11-05 14:19:13 +02:00
Aliaksandr Valialkin
30c8301b11
lib/storage: reduce the maximum inverted index scans before giving up to label filters matching by metric name
...
The new value reduces the amount of wasted work during index scans over big number of time series.
2019-11-05 14:19:06 +02:00
Aliaksandr Valialkin
e53f9e553d
lib/storage: try potentially faster tag filters at first, then apply slower tag filters
...
The fastest tag filters are non-negative non-regexp, since they are the most specific.
The slowest tag filters are negative regexp, since they require scanning
all the entries for the given label.
2019-11-05 14:19:01 +02:00
Aliaksandr Valialkin
3c90d77858
lib/storage: pass pointer to MetricName in Fatalf, so it is properly detected as an interface with String() method
...
This fixes lint errors
2019-11-04 01:07:19 +02:00
Artem Navoiev
478767d0ed
add unittests for bytesutil and storage ( #221 )
2019-11-04 00:54:46 +02:00
Aliaksandr Valialkin
02e0b19a62
lib/storage: tune the returned value from adjustMaxMetricsAdaptive
2019-11-04 00:44:37 +02:00
Aliaksandr Valialkin
6be4456d88
lib/{storage,uint64set}: add Set.Union() function and use it
2019-11-04 00:44:37 +02:00
Aliaksandr Valialkin
9becc26f4b
lib/storage: remove interface conversion in hot path during block merging
...
This should improve merge speed a bit for parts with big number of small blocks.
2019-11-03 12:33:34 +02:00
Aliaksandr Valialkin
c62399eb3e
lib/{storage,mergeset}: create missing partition directories after restoring from backups
...
Backup tools could skip empty directories. So re-create such directories on the first run.
2019-11-02 02:27:11 +02:00
Aliaksandr Valialkin
55d728c849
lib/{decimal,encoding}: optimize float64<->decimal conversion for arrays with zeros or ones
...
Time series with only zeros or ones frequently occur in monitoring, so it is worth optimizing their handling.
2019-11-01 16:48:12 +02:00
Aliaksandr Valialkin
808fc0971f
lib/{encoding,decimal}: add benchmarks for blocks containing zeros or ones
...
Time series with such values are quite common in monitoring space,
so it would be great to have benchmarks for them.
2019-11-01 16:48:12 +02:00
Aliaksandr Valialkin
370cfbb365
lib/uint64set: return an emptry set instead of nil set from Set.Clone
, since the caller may add data to the cloned set
...
This fixes the following panic in v1.28.1:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x783a7e]
goroutine 1155 [running]:
github.com/VictoriaMetrics/VictoriaMetrics/lib/uint64set.(*Set).Add(0x0, 0x15b3bfb41e8b71ec)
github.com/VictoriaMetrics/VictoriaMetrics@/lib/uint64set/uint64set.go:57 +0x2e
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*indexSearch).getMetricIDsForRecentHours(0xc5bdc0dd40, 0x16e273f6b50, 0x16e2745d3f0, 0x5b8d95, 0x10, 0x4a2f51, 0xaa01000000000000)
github.com/VictoriaMetrics/VictoriaMetrics@/lib/storage/index_db.go:1951 +0x260
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*indexSearch).getMetricIDsForTimeRange(0xc5bdc0dd40, 0x16e273f6b50, 0x16e2745d3f0, 0x5b8d95, 0x10, 0xb296c0, 0xc00009cd80, 0x9bc640)
2019-11-01 16:12:44 +02:00
Aliaksandr Valialkin
d18ea0c95b
app/vmstorage: add -bigMergeConcurrency
and -smallMergeConcurrency
flags for tuning the maximum number of CPU cores used during merges
2019-10-31 16:19:13 +02:00
Aliaksandr Valialkin
e0b292c6de
lib/storage: small cleanup in Storage.add
2019-10-31 14:30:34 +02:00
Aliaksandr Valialkin
e76e21e4c7
lib/decimal: speed up FromFloat for common case with integers
2019-10-31 13:24:59 +02:00
Aliaksandr Valialkin
cfa5e279c2
lib/decimal: increase float64->decimal conversion precision a bit
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/213
2019-10-30 02:04:56 +02:00
Aliaksandr Valialkin
26d570bb3a
lib/storage: get parts to merge after applying the limit on the number of concurrent merges
...
This should reduce write amplification under high ingestion rate.
2019-10-30 02:04:56 +02:00
Aliaksandr Valialkin
2e2eff90d5
lib/{mergeset,storage}: limit the maximum number of concurrent merges; leave smaller number of parts during final merge
2019-10-29 12:45:28 +02:00
Aliaksandr Valialkin
04e48ef064
lib/fs: typo fix in comment to WriteFileAtomically
2019-10-29 11:31:26 +02:00
Aliaksandr Valialkin
a42b5db39f
lib/decimal: increase float->decimal conversion precision for big numbers
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/213
2019-10-28 13:23:44 +02:00
hanzai
b3c946e35a
warns during rows addition ( #214 )
2019-10-20 23:41:07 +03:00
Aliaksandr Valialkin
e83fe938c8
all: make fmt
2019-10-17 20:04:34 +03:00
Aliaksandr Valialkin
97ce4e03a5
all: add support for GOARCH=386 and fix all the issues related to 32-bit architectures such as GOARCH=arm
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
2019-10-17 18:23:23 +03:00
Aliaksandr Valialkin
6ebf537153
lib/memory: properly handle int overflow in sysTotalMemory
...
This should fix builds on 32-bit architectures such as arm.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
2019-10-17 00:50:48 +03:00
Aliaksandr Valialkin
195cefd81a
lib/prompb: removed outdated README.md
2019-10-14 22:12:57 +03:00
Aliaksandr Valialkin
f6334bffa1
lib/storage: harden the check that the original items are sorted after mergeTagToMetricIDsRows fails to preserve sort order
2019-10-09 12:13:17 +03:00
Aliaksandr Valialkin
2abd5154e0
lib/storage: typo fix in comment to maxRowsPerSmallPart.
2019-10-08 18:51:20 +03:00
Aliaksandr Valialkin
c1cf7d9f93
lib/storage: add tests for mergeTagToMetricIDsRows and return the original items if the function breaks items` ordering.
...
This should save from data corruption issues revealed in the previous releases up to v1.28.0-beta5.
2019-10-08 16:27:35 +03:00
Aliaksandr Valialkin
fdc5cfd838
lib/mergeset: reduce the maximum number of cached blocks, since there are reports on OOMs due to too big caches
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/189
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/195
2019-09-30 12:25:40 +03:00
Aliaksandr Valialkin
0fa6df94a2
lib/storage: optimize TSID comparison
2019-09-26 14:16:02 +03:00
Aliaksandr Valialkin
c39355921e
lib/storage: verify whether items are sorted in the end of call to mergeTagToMetricIDsRows
...
This should prevent from inverted index corruption if bug in mergeTagToMetricIDsRows is discovered.
2019-09-26 13:13:41 +03:00
Aliaksandr Valialkin
0db9fcedd5
lib/storage: properly match labels against regexp with (?i)
flag
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/161
2019-09-26 11:03:10 +03:00
Aliaksandr Valialkin
2444433d83
lib/storage: add missing break in removeDuplicateMetricIDs
2019-09-25 18:23:43 +03:00
Aliaksandr Valialkin
ea4c828bae
lib/storage: remove duplicate MetricIDs in tag->metricIDs
items before writing them into inverted index
2019-09-25 17:55:13 +03:00
Aliaksandr Valialkin
aebc45ad26
lib/{mergeset,storage}: do not cache inverted index blocks containing tag->metricIDs
items
...
This should reduce the amounts of used RAM during queries with filters over big number of time series.
2019-09-25 14:02:15 +03:00
Aliaksandr Valialkin
2cb811b42f
lib/uint64set: optimize Set.AppendTo
2019-09-25 00:34:17 +03:00
Aliaksandr Valialkin
b986516fbe
lib/storage: create and use lib/uint64set
instead of map[uint64]struct{}
...
This should improve inverted index search performance for filters matching big number of time series,
since `lib/uint64set.Set` is faster than `map[uint64]struct{}` for both `Add` and `Has` calls.
See the corresponding benchmarks in `lib/uint64set`.
2019-09-24 21:17:55 +03:00
Aliaksandr Valialkin
ef2296e420
lib/storage: typo fix: return dstData instead of data from mergeTagToMetricIDsRows
2019-09-24 19:32:34 +03:00
Aliaksandr Valialkin
a6086cde78
lib/storage: limit the number of metricIDs in tag->metricIDs row
...
This reduces the overhead on index and metaindex in lib/mergeset
2019-09-24 00:49:51 +03:00
Aliaksandr Valialkin
c9063ece66
lib/storage: share tsids across all the partSearch instances
...
This should reduce memory usage when big number of time series matches the given query.
2019-09-23 22:35:15 +03:00
Aliaksandr Valialkin
4e26ad869b
lib/{storage,mergeset}: verify PrepareBlock callback results
...
Do not touch the first and the last item passed to PrepareBlock
in order to preserve sort order of mergeset blocks.
2019-09-23 20:43:13 +03:00
Aliaksandr Valialkin
0772191975
lib/mergeset: detect whether we are in test by executable suffix
2019-09-22 23:12:15 +03:00
Aliaksandr Valialkin
48999e5396
lib/workingsetcache: remove data race when resetting c.misses
2019-09-22 19:36:49 +03:00
Aliaksandr Valialkin
0adebae1f8
lib/storage: generate the first tag->metricIDs item in a mergeset block with a single metricID
...
The first item from each mergeset block goes into index (lib/mergeset.blockHeader),
so it must be short in order to reduce index size.
2019-09-22 19:21:33 +03:00
Aliaksandr Valialkin
0686ac52c3
lib/{storage,mergeset}: merge tag->metricID
rows into tag->metricIDs
rows for common tag
values
...
This should improve lookup performance if the same `label=value` pair exists
in big number of time series.
This should also reduce memory usage for mergeset data cache, since `tag->metricIDs` rows
occupy less space than the original `tag->metricID` rows.
2019-09-20 22:06:41 +03:00
Aliaksandr Valialkin
68722c3c74
lib/encoding: optimize UnmarshalUint* and UnmarshalInt*
2019-09-20 13:08:16 +03:00
Aliaksandr Valialkin
a544f49c2b
lib/storage: optimize selecting all the metricIDs by scanning MetricID->TSID entries instead of tag->MetricID entries
...
The number of MetricID->TSID entries is smaller than the number of tag->MetricID entries
and MetricID->TSID entries are usually shorter than tag->MetricID entries.
This should improve performance when selecting all the metricIDs.
2019-09-20 11:54:10 +03:00
Aliaksandr Valialkin
00cfb2d2b9
lib/mergeset: rename misleading mergeSmallParts to mergeExistingParts
2019-09-19 21:48:20 +03:00
Aliaksandr Valialkin
37dc223e25
lib/mergeset: use sort.IsSorted instead of sort.SliceIsSorted in inmemoryBlock.isSorted in order to reduce memory allocations
2019-09-19 20:13:08 +03:00
Aliaksandr Valialkin
a84fe76677
lib/storage: use sort.Sort instead of sort.slice in getSortedMetricIDs
2019-09-19 20:07:22 +03:00
Aliaksandr Valialkin
3a697a935a
lib/storage: skip duplicate call to intersectMetricIDsWithTagFilter on zero successful intersects
2019-09-19 17:49:56 +03:00
Aliaksandr Valialkin
51a21c7d4b
lib/mergeset: fill partHeader.firstItem on first block flush
2019-09-19 17:48:09 +03:00
Aliaksandr Valialkin
3d83f5d334
lib/storage: mark tag filter returning errFallbackToMetricNameMatch as useless
...
This will save CPU on subsequent calls for this filter
2019-09-18 19:10:32 +03:00
Aliaksandr Valialkin
8d35718dc6
lib/storage: properly construct keys for uselessTagFiltersCache and register useless negative tag filters there
2019-09-17 23:20:27 +03:00
Aliaksandr Valialkin
0c8057924f
lib/mergeset: properly check for sorted block headers
...
Fix a typo for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/181
2019-09-13 21:59:29 +03:00
hanzai
e2274714b1
lib/workingsetcache: adjust switching from mode=split
to mode=whole
smoothly and load cachefile successfully
2019-09-13 19:13:01 +03:00
Aliaksandr Valialkin
bad53e4207
lib/mergeset: dynamically calculate the maximum number of items per part, which can be cached in OS page cache
2019-09-11 14:53:45 +03:00
Aliaksandr Valialkin
88f8670ede
lib/fs: add MustStopDirRemover for waiting until pending directories are removed on graceful shutdown
...
This patch is mainly required for laggy NFS. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162
2019-09-05 11:13:17 +03:00
Aliaksandr Valialkin
9eb5de334f
lib/storage: typo fix
2019-09-04 19:58:01 +03:00
Aliaksandr Valialkin
16dd145586
lib/storage: remove duplicate tag keys on MetricName.Marshal
call
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/172
2019-09-04 18:13:45 +03:00
Aliaksandr Valialkin
4862e93024
lib/fs: try harder with directory removal on NFS in the event of temporary lock
...
Do not give up after 11 attempts of directory removal on laggy NFS.
Add `vm_nfs_dir_remove_failed_attempts_total` metric for counting the number of failed attempts
on directory removal.
Log failed attempts on directory removal after long sleep times.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162
2019-09-04 12:24:50 +03:00
Aliaksandr Valialkin
e1d76ec1f3
lib/storage: invalidate tagFilters -> TSIDS
cache when newly added index data becomes visible to search
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/163
2019-08-29 15:08:35 +03:00
Aliaksandr Valialkin
aeaa5de5fe
lib/prombp: apply ba06b47c16
...
The following commands used:
gofmt -r '(uint64(x)&0x7F)<<shift -> uint64(x&0x7F)<<shift' -w ./lib/prompb/
gofmt -r '(int64(x)&0x7F)<<shift -> int64(x&0x7F)<<shift' -w ./lib/prompb/
2019-08-29 13:35:27 +03:00
Aliaksandr Valialkin
9196c085a7
all: port to FreeBSD on GOARCH=amd64
2019-08-28 01:19:23 +03:00
Aliaksandr Valialkin
2655220c58
lib/storage: go fmt
2019-08-27 14:29:51 +03:00
Aliaksandr Valialkin
bf915fc0db
lib/storage: report proper maxMetrics limit when more than -search.maxUniqueTimeseries series match the given filters
2019-08-27 14:21:42 +03:00
Aliaksandr Valialkin
2fc157ff7a
lib/storage: properly handle (?i)
in the tag filter regexp
...
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/161
2019-08-26 00:44:45 +03:00
Aliaksandr Valialkin
0dc0006f34
lib/storage: calculate the maximum number of rows per small part from -memory.allowedPercent
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/159
This simplifies error detection additionally to the `vm_rows_ignored_total` counters.
2019-08-25 15:31:47 +03:00
Aliaksandr Valialkin
4b688fffee
lib/storage: calculate the maximum number of rows per small part from -memory.allowedPercent
...
This should improve query speed over recent data on machines with big amounts of RAM
2019-08-25 14:41:12 +03:00
Aliaksandr Valialkin
1402a6b981
lib/storage: properly limit the number of output rows in small and big parts storage
...
Previously small parts storage didn't take into account the available disk space for big parts.
2019-08-25 14:41:12 +03:00
Aliaksandr Valialkin
3308279c4e
lib/storage: remove outdated comment on maxRowsPerSmallPart
...
The commend became outdated after the commit ed6ac1a5df027f0dfc22448e3b27c26b6f77c67a,
which stops merging of small parts on graceful shutdown instead of waiting
for their completion.
2019-08-25 13:47:32 +03:00
Aliaksandr Valialkin
ac004665b5
all: return 503 http error if service is temporarily unavailable
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/156
2019-08-23 09:55:07 +03:00
Aliaksandr Valialkin
8c03a8c4b4
app/vminsert: allow setting the maximum number of labels per time series via -maxLabelsPerTimeseries
2019-08-23 08:45:26 +03:00
Aliaksandr Valialkin
380cae23a0
lib/storage: add benchmarks for regexp filter match / mismatch
...
These benchmarks allow estimate the performance of regexp filters in promql
2019-08-22 16:36:42 +03:00
Aliaksandr Valialkin
4f738c8a15
lib/storage: try slower path for searching the tag filter with the minimum number of matching time series before giving up with increase -search.maxUniqueTimeseries
error
2019-08-19 16:04:21 +03:00
Aliaksandr Valialkin
c23b66a1ad
lib/storage: pre-allocate memory for blockHeader slice in unmarshalBlockHeaders
...
This reduces memory usage and memory fragmentation when working with big number of time series
2019-08-19 12:46:33 +03:00
Aliaksandr Valialkin
483de1cc06
lib/workingsetcache: automatically detect when it is better to double cache capacity
2019-08-15 22:57:55 +03:00
Aliaksandr Valialkin
5b41122292
lib/storage: properly cache tagFilters -> TSIDs entries from historical index
2019-08-14 02:29:58 +03:00
Aliaksandr Valialkin
964c296f96
lib/storage: compress contents of cache for tagFilters -> TSIDs
...
This should increase cache capacity
2019-08-14 02:29:52 +03:00
Aliaksandr Valialkin
09fc6e22e5
all: use workingsetcache instead of fastcache
...
This should reduce the amount of RAM required for processing time series
with non-zero churn rate.
The previous cache behavior can be restored with `-cache.oldBehavior` command-line flag.
2019-08-13 21:39:34 +03:00
Aliaksandr Valialkin
99c37c2c96
lib/fs: add test for IsTemporaryFileName
2019-08-13 21:33:45 +03:00
Aliaksandr Valialkin
ec1b185991
lib/storage: remove broken BenchmarkIndexDBSearchTSIDs
2019-08-13 20:22:08 +03:00
Aliaksandr Valialkin
0967683ae9
lib: move common code for creating flock.lock file into fs.CreateFlockFile
2019-08-13 01:45:46 +03:00
Aliaksandr Valialkin
5d8d110010
lib/fs: atomically create file with the given contents on WriteFileAtomically
...
This should prevent from `transaction` and `metadata.json` files corruption
on unclean shutdown such as OOM, `kill -9`, power loss, etc.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/148
2019-08-12 15:02:55 +03:00
Aliaksandr Valialkin
0b488f1e37
lib/storage: do not change timestamps to constant rate if values are constant or have constant delta
...
This breaks the original timestamps, which results in issues like
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/120 and
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/141 .
2019-08-06 15:40:07 +03:00
Aliaksandr Valialkin
b8bb74ffc6
app/vmstorage: add vm_concurrent_addrows_*
metrics for tracking concurrency for Storage.AddRows calls
...
Track also the number of dropped rows due to the exceeded timeout
on concurrency limit for Storage.AddRows. This number is tracked in `vm_concurrent_addrows_dropped_rows_total`
2019-08-06 15:08:33 +03:00
Aliaksandr Valialkin
8822079b77
lib/storage: properly reset partSearch.fetchData
in partSearch.reset
2019-08-05 09:56:06 +03:00
Aliaksandr Valialkin
47e4b50112
app/vmselect: optimize /api/v1/series
by skipping storage data
...
Fetch and process only time series metainfo.
2019-08-04 23:01:28 +03:00
Aliaksandr Valialkin
10caad4728
lib/decimal: modernize tests a bit
2019-07-31 21:10:03 +03:00
Aliaksandr Valialkin
c14fd6c43f
lib/storage: typo fixes after a77e88db7d
2019-07-30 15:38:52 +03:00
Aliaksandr Valialkin
a77e88db7d
lib/storage: fix matching against tag filter with empty name
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/137
2019-07-30 15:15:09 +03:00
Aliaksandr Valialkin
f586e1f83c
lib/storage: add metrics for calculating skipped rows outside the retention
...
The metrics are:
- vm_too_big_timestamp_rows_total
- vm_too_small_timestamp_rows_total
2019-07-26 14:11:01 +03:00
Aliaksandr Valialkin
b339890dca
lib/encoding/zstd: go fmt
2019-07-25 01:37:16 +03:00
Aliaksandr Valialkin
6c4ca89d75
lib/encoding/zstd: disable CRC checks in pure Go
build
...
This should give slightly better compression and decompressions performance.
Additionally this shaves off 4 bytes per each compressed block.
2019-07-24 19:17:16 +03:00
Aliaksandr Valialkin
d07f616609
lib/encoding: small fixes in tests after the PR #114
2019-07-23 19:37:51 +03:00
Roman Khavronenko
5bf4e5ffb5
all: add Pure Go build (pull request #114 )
...
Updates #94
2019-07-23 19:26:39 +03:00
Aliaksandr Valialkin
0f2ecde0e6
lib/encoding: improve gauge series detection
...
- Series with negative values are always gauges
- Counters may only have increasing values with possible counter resets
This should improve compression ratio for gauge series which
were previously mistakenly detected as counters.
2019-07-20 14:05:09 +03:00
Aliaksandr Valialkin
daba0cdb05
lib/netutil: do not count timeouts as network errors
2019-07-15 23:05:35 +03:00
Aliaksandr Valialkin
e75d5f47c4
lib/storage: remove unused function isTooBigTimeRangeForDateMetricIDs
2019-07-12 02:28:23 +03:00
Aliaksandr Valialkin
fc90ebf43c
lib/storage: do not reduce maxMetrics
on time ranges exceeding maxDaysForDateMetricIDs
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/95
2019-07-12 02:20:34 +03:00
Aliaksandr Valialkin
2bd1a01d1a
lib/storage: do not pollute inverted index with data for samples outside the retention period
2019-07-11 17:04:56 +03:00
Aliaksandr Valialkin
d031e04023
lib/storage: use fast path for orSuffix when searching for metricIDs against plain tag value
2019-07-11 14:48:37 +03:00
Aliaksandr Valialkin
43ea4ce428
lib/storage: remember and skip individual tag filters matching too many metrics
...
This saves CPU time by skipping useless matching for individual tag filters.
2019-07-11 14:48:30 +03:00
Aliaksandr Valialkin
1fe6d784d8
all: consistency renaming: bytesSize -> sizeBytes
2019-07-10 00:47:36 +03:00
Aliaksandr Valialkin
56c154f45b
all: add vm_data_size_bytes
metrics for easy monitoring of on-disk data size and on-disk inverted index size
2019-07-04 19:42:30 +03:00
Aliaksandr Valialkin
c7034fc51b
lib/memory: attempt #3 to determine memory limit for LXC container
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/84
2019-07-01 14:01:13 +03:00
Aliaksandr Valialkin
a41955863a
lib/mergeset: make fmt
2019-06-29 14:25:26 +03:00
Aliaksandr Valialkin
2ecb117082
lib/storage: skip non-matching metricIDs in sortedFilter
...
This should improve performance for big sorteFilter lists.
2019-06-29 13:48:32 +03:00
Aliaksandr Valialkin
0c88afa386
lib/mergeset: speed up binarySearchKey by skipping the first item during binary search
2019-06-29 13:45:49 +03:00
Aliaksandr Valialkin
828078eb45
lib/memory: remove TestReadLXCMemoryLimit, since it doesnt work in Travis
2019-06-28 18:22:46 +03:00
Aliaksandr Valialkin
7b59466667
lib/memory: attempt #2 to determine memory limit inside LXC container
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/84
2019-06-28 18:08:15 +03:00
Aliaksandr Valialkin
593bd35aaa
lib/memory: an attempt to read proper memory limit inside LXC container
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/84
2019-06-28 15:34:30 +03:00
Aliaksandr Valialkin
c1be1e4342
lib/storage: optimize time series search by regexp filter
...
This should improve search speed on label filters like `{foo=~"bar.+baz"}`
2019-06-27 16:17:43 +03:00
Aliaksandr Valialkin
683bf2a11f
lib/storage: make sure non-nil args are passed to openIndexDB
2019-06-25 20:10:04 +03:00
Aliaksandr Valialkin
eb2283a029
lib/storage: reduce too big maxMetrics in getTagFilterWithMinMetricIDsCountAdaptive
...
This should improve performance on inverted index search for big amount of unique time series
when big -search.maxUniqueTimeseries is set.
2019-06-25 19:55:27 +03:00
Aliaksandr Valialkin
e8377011ab
lib/storage: free up memory from caches owned by indexDB when it is deleted
2019-06-25 14:42:44 +03:00
Aliaksandr Valialkin
33ea2120c3
lib/storage: use unversioned keys for tag cache in extDB
...
Data in ExtDB cannot be changed, so it is OK to use unversioned keys for tag cache.
This should improve performance for index lookups over big amount of time series.
2019-06-25 13:08:58 +03:00
Aliaksandr Valialkin
cf63669303
lib/storage: skip searching in extDB if it doesn't contain items for the given time range
...
This should improve inverted index search performance for big amount
of unique time series when the search is performed only on recent data.
2019-06-25 13:00:37 +03:00
Aliaksandr Valialkin
af2ceaaa0b
lib/storage: mention source parts on merge error
...
This should improve determining broken source part.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/76
2019-06-24 14:08:43 +03:00
Aliaksandr Valialkin
e88a03323a
all: initial stubs for Windows support; see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70
2019-06-20 20:07:10 +03:00
Aliaksandr Valialkin
80db24386e
lib/storage: typo fixes found by golangci-lint; updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/69
2019-06-20 14:37:55 +03:00
Aliaksandr Valialkin
296c14317f
lib/netutil: remove unused TCPListener.name; updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/69
2019-06-20 14:36:15 +03:00
Aliaksandr Valialkin
a78b3dba7f
app/vmstorage: add vm_cache_entries{type="storage/hour_metric_ids"}
metric for tracking active time series count
2019-06-19 18:36:47 +03:00
Aliaksandr Valialkin
710d6c33ea
lib/prompb: remove superflouos bytes copying in ReadSnappy
2019-06-18 20:37:51 +03:00
Aliaksandr Valialkin
6d6c9eb1f8
lib/flagutil: remove unused package
2019-06-18 10:43:55 +03:00
Aliaksandr Valialkin
d2c801029b
lib/storage: persist metric ids for the current and the previous hour on graceful shutdown
...
This should improve performance after restart when the db contains a lot of time series
with high time series churn (i.e. metrics from Kubernetes with many pods and frequent deployments)
2019-06-14 07:55:14 +03:00
Aliaksandr Valialkin
26f8d7ea1b
lib/fs: sync parent dir in MustRemoveAll only if it exists
...
The parent directory may be non-existing when the deleted directory
didn't exist before the MustRemoveAll call
2019-06-12 02:14:44 +03:00
Aliaksandr Valialkin
419197ba08
lib/fs: consolidate *RemoveAll* funcs into a single MustRemoveAll func
...
The func syncs parent dir in order to persist directory removal
in the event of power loss
2019-06-12 01:53:46 +03:00
Aliaksandr Valialkin
c1276edab5
lib/fs: panic with fatal error when directories cannot be removed
...
Unremoved directories may lead to inconsistent data directory,
so VictoriaMetrics will fail to start next time.
So panic on the first error when trying to remove directory in order
to simplify recover process.
2019-06-12 01:20:54 +03:00
Aliaksandr Valialkin
2322c9a45a
lib/fs: attempt #2 to work around NFS issue with directory removal
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/61
2019-06-12 01:07:05 +03:00
Aliaksandr Valialkin
935bfd7a18
lib/fs: consistency renaming SyncPath -> MustSyncPath, since it doesnt return error
2019-06-11 23:13:49 +03:00
Aliaksandr Valialkin
3dd36b8088
lib/fs: make sure the created directory remains visible in the fs in the event of power loss
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/63
2019-06-11 23:08:09 +03:00
Aliaksandr Valialkin
afb964670a
lib/fs: use filepath.Dir instead of filepath.Split, since the filename is unused
2019-06-11 22:54:26 +03:00
Aliaksandr Valialkin
20fc0e0e54
lib/{storage,mergeset}: sync filenames inside part when finalizing the part
...
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/63
2019-06-11 21:51:13 +03:00
Aliaksandr Valialkin
ac7b186f13
all: try hard removing directory with contents
...
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/61
2019-06-11 01:57:59 +03:00
Aliaksandr Valialkin
cbe692f0e2
app/vmselect: add /api/v1/labels/count
handler for quick detection of labels with the maximum number of distinct values
2019-06-10 19:55:38 +03:00
Aliaksandr Valialkin
7b6623558f
lib/storage: skip adaptive searching for tag filter matching the minimum number of metrics if the identical previous search didn't found such filter
...
This should improve speed for searching metrics among high number of time series
with high churn rate like in big Kubernetes clusters with frequent deployments.
2019-06-10 14:07:39 +03:00
Aliaksandr Valialkin
a1351bbaee
lib/storage: factor out getTagFilterWithMinMetricIDsCountAdaptive from updateMetricIDsForTagFilters
2019-06-10 13:26:44 +03:00
Aliaksandr Valialkin
b4d707d9bb
lib/storage: give clearer names to more functions
2019-06-10 13:01:23 +03:00
Aliaksandr Valialkin
bee7298f81
lib/storage: give more clear names to functions
2019-06-10 12:50:44 +03:00
Aliaksandr Valialkin
dbd217b8f0
lib/storage: test GetSeriesCount
2019-06-10 12:43:34 +03:00
Aliaksandr Valialkin
4d936b1524
lib/storage: make getSeriesCount func indexSearch method
2019-06-10 12:29:11 +03:00
Aliaksandr Valialkin
d37924900b
lib/storage: optimize time series lookup for recent hours when the db contains many millions of time series with high churn rate (aka frequent deployments in Kubernetes)
2019-06-09 19:13:56 +03:00
Aliaksandr Valialkin
a3e26de45e
lib/procutil: typo fix in comment to WaitForSigterm
2019-06-04 17:31:47 +03:00
Aliaksandr Valialkin
28f6c36ab4
lib/storage: tune updating a map with today`s metric ids
...
- Increase update iterval from 1s to 10s. This should reduce CPU usage
for large amounts of metric ids with constant churn.
- Reduce pendingTodayMetricIDsLock lock duration during the update.
2019-06-02 21:58:16 +03:00
Aliaksandr Valialkin
4794f894a4
lib/storage: speed up checking metricID existence in the list for the current date
2019-06-02 18:34:08 +03:00
Aliaksandr Valialkin
e307a4d92c
lib/timerpool: use timer pool in concurrency limiters
...
This should reduce the number of memory allocations in highly loaded system
2019-05-28 17:20:10 +03:00
Aliaksandr Valialkin
54fb8b21f9
all: fix misspellings
2019-05-25 21:51:11 +03:00
Aliaksandr Valialkin
d6523ffe90
Makefile: add -s flag to go fmt
in make fmt
command
2019-05-25 21:43:35 +03:00
Aliaksandr Valialkin
6b0ae332f8
lib/encoding: add vm_zstd_block_{compress|decompress}_calls_total
for determining the number CompressZSTD / DecompressZSTD calls
2019-05-24 13:01:02 +03:00
Aliaksandr Valialkin
6fb9dd09f5
lib/encoding: add vm_zstd_block_{original|compressed}_bytes_total
metrics for rough estimation of block compression ratio
2019-05-24 12:34:32 +03:00
Aliaksandr Valialkin
19b6643e5c
lib/encoding: substitute CompressZSTD with CompressZSTDLevel
2019-05-24 12:32:55 +03:00
Aliaksandr Valialkin
08b889ef09
lib/httpserver: add -http.disableResponseCompression
flag, which may help saving CPU resources at the cost of higher network bandwidth usage
2019-05-24 12:18:40 +03:00
Aliaksandr Valialkin
674888fdc9
lib/decimal: add a comment explaining weird code in maxUpExponent. Fixes #29
2019-05-23 17:18:35 +03:00
Aliaksandr Valialkin
1836c415e6
all: open-sourcing single-node version
2019-05-23 00:18:06 +03:00