github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
733 commits 150 branches 748 tags 347 MiB
Commit graph

733 commits

This branch
This branch
All branches
Author SHA1 Message Date
Aliaksandr Valialkin
46e67bb78c lib/storage: export vm_new_timeseries_created_total metric for determining time series churn rate 2019-11-08 19:58:21 +02:00
Aliaksandr Valialkin
0063c857f5 lib/storage: add inmemory inverted index for the last hour 
It should improve performance for `last N hours` dashboards with update intervals smaller than 1 hour.
 2019-11-08 19:37:46 +02:00
Aliaksandr Valialkin
33abbec6b4 app/vmselect/promql: adjust memory limits calculations for incremental aggregate functions 
Incremental aggregate functions don't keep all the selected time series in memory -
they keep only up to GOMAXPROCS time series for incremental aggregations.

Take into account that the number of time series in RAM can be higher if they are split
into many groups with `by (...)` or `without (...)` modifiers.

This should reduce the number of `not enough memory for processing ... data points` false
positive errors.
 2019-11-08 19:37:43 +02:00
Aliaksandr Valialkin
7d7fbf890e app/{vmbackup,vmrestore}: add vmbackup and vmrestore tools for creating backups on s3 or gcs from instant snapshots 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/203
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/38
 2019-11-07 21:26:43 +02:00
Roman Khavronenko
4e7a2a41a4 Cluster dashboard (#222) 
* add dashboard for cluster version

* fix queries and panels

* review fixes

* use resident memory for memory usage panel

* fix job selectors
 2019-11-07 12:09:27 +02:00
Aliaksandr Valialkin
89c03a5464 lib/storage: populate partition names from both small and big directories 
Certain partition directories may be missing after restoring from backups
if they had no data. Re-create such directories on start.
 2019-11-06 19:50:21 +02:00
Aliaksandr Valialkin
1c777e0245 lib/storage: substitute error message about unsorted items in the index block after metricIDs merge with counter 
The origin of the error has been detected and documented in the code,
so it is enough to export a counter for such errors at `vm_index_blocks_with_metric_ids_incorrect_order_total`,
so it could be monitored and alerted on high error rates.

Export also the counter for processed index blocks with metricIDs - `vm_index_blocks_with_metric_ids_processed_total`,
so its' rate could be compared to `rate(vm_index_blocks_with_metric_ids_incorrect_order_total)`.
 2019-11-06 14:32:41 +02:00
Aliaksandr Valialkin
c567a4353a lib/storage: take into account the requested time range when caching TSIDs for the given tag filters 2019-11-06 14:32:41 +02:00
Aliaksandr Valialkin
c6564c5d26 lib/storage: dump incorrectly sorted items on a single line; this should simplify error reporting 2019-11-05 18:41:50 +02:00
Aliaksandr Valialkin
2ef5082ead deployment/docker: update docker images from v1.28.2-cluster to v1.28.3-cluster 2019-11-05 18:08:50 +02:00
Aliaksandr Valialkin
a10c4cad85 lib/storage: return back finalPartsToMerge from 2 to 3 in order to prevent from excessive merges in old partitions 2019-11-05 17:28:57 +02:00
Aliaksandr Valialkin
e5b1fa0c38 lib/storage: separate the max inverted index scan loops per metric into fast and slow loops 
Slow loops could require seeks and expensive regexp matching, while fast loops just scans
all the metricIDs for the given `tag=value` prefix. So these operations must have separate
max loops multiplier.
 2019-11-05 17:28:57 +02:00
Aliaksandr Valialkin
f93c4f2493 lib/storage: skip repeated useless work when intersection of metricIDs with the given filter is too expensive 
This should improve performance for query filters over big number of time series.
 2019-11-05 14:35:55 +02:00
Aliaksandr Valialkin
f48e97263c lib/storage: reduce the maximum inverted index scans before giving up to label filters matching by metric name 
The new value reduces the amount of wasted work during index scans over big number of time series.
 2019-11-05 14:35:53 +02:00
Aliaksandr Valialkin
d2f688c550 lib/storage: try potentially faster tag filters at first, then apply slower tag filters 
The fastest tag filters are non-negative non-regexp, since they are the most specific.
The slowest tag filters are negative regexp, since they require scanning
all the entries for the given label.
 2019-11-05 14:35:48 +02:00
Aliaksandr Valialkin
a72b22a8b1 Makefile: add pprof-cpu rule for inspecting CPU profiles with PPROF_FILE=/path/to/cpu.pprof make pprof-cpu 2019-11-04 12:43:57 +02:00
Aliaksandr Valialkin
2a38d30f93 lib/storage: pass pointer to MetricName in Fatalf, so it is properly detected as an interface with String() method 
This fixes lint errors
 2019-11-04 01:06:45 +02:00
Artem Navoiev
e05500cbd4 add unittests for bytesutil and storage (#221) 2019-11-04 00:57:24 +02:00
Aliaksandr Valialkin
f5fbc3ffd7 lib/{storage,uint64set}: add Set.Union() function and use it 2019-11-04 00:48:32 +02:00
Aliaksandr Valialkin
23e078261e lib/storage: tune the returned value from adjustMaxMetricsAdaptive 2019-11-04 00:45:28 +02:00
Aliaksandr Valialkin
386c349c8c lib/storage: remove interface conversion in hot path during block merging 
This should improve merge speed a bit for parts with big number of small blocks.
 2019-11-03 12:33:48 +02:00
Aliaksandr Valialkin
26ffc77622 lib/{storage,mergeset}: create missing partition directories after restoring from backups 
Backup tools could skip empty directories. So re-create such directories on the first run.
 2019-11-02 02:27:19 +02:00
Aliaksandr Valialkin
5d439cc6f2 lib/{decimal,encoding}: optimize float64<->decimal conversion for arrays with zeros or ones 
Time series with only zeros or ones frequently occur in monitoring, so it is worth optimizing their handling.
 2019-11-01 16:46:08 +02:00
Aliaksandr Valialkin
1037053fed lib/{encoding,decimal}: add benchmarks for blocks containing zeros or ones 
Time series with such values are quite common in monitoring space,
so it would be great to have benchmarks for them.
 2019-11-01 16:46:08 +02:00
Aliaksandr Valialkin
46b8e13d8c deployment/docker: update image tag from v1.28.1-cluster to v1.28.2-cluster 2019-11-01 16:33:46 +02:00
Aliaksandr Valialkin
44fab198e2 lib/uint64set: return an emptry set instead of nil set from Set.Clone, since the caller may add data to the cloned set 
This fixes the following panic in v1.28.1:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x783a7e]

goroutine 1155 [running]:
github.com/VictoriaMetrics/VictoriaMetrics/lib/uint64set.(*Set).Add(0x0, 0x15b3bfb41e8b71ec)
  github.com/VictoriaMetrics/VictoriaMetrics@/lib/uint64set/uint64set.go:57 +0x2e
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*indexSearch).getMetricIDsForRecentHours(0xc5bdc0dd40, 0x16e273f6b50, 0x16e2745d3f0, 0x5b8d95, 0x10, 0x4a2f51, 0xaa01000000000000)
  github.com/VictoriaMetrics/VictoriaMetrics@/lib/storage/index_db.go:1951 +0x260
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*indexSearch).getMetricIDsForTimeRange(0xc5bdc0dd40, 0x16e273f6b50, 0x16e2745d3f0, 0x5b8d95, 0x10, 0xb296c0, 0xc00009cd80, 0x9bc640)
 2019-11-01 16:12:21 +02:00
Aliaksandr Valialkin
4a8251feff app/vmselect/promql: add lag(q[d]) function, which returns the lag between the current timestamp and the timstamp for the last data point in q 2019-11-01 12:21:43 +02:00
Aliaksandr Valialkin
bd065aad5e deployment/docker: update docker images from v1.28.0-cluster to v1.28.1-cluster 2019-10-31 17:03:27 +02:00
Aliaksandr Valialkin
6ab9c98a1e app/vmstorage: add -bigMergeConcurrency and -smallMergeConcurrency flags for tuning the maximum number of CPU cores used during merges 2019-10-31 16:17:29 +02:00
Aliaksandr Valialkin
6a22727676 lib/storage: optimize getMetricIDsForRecentHours for per-tenant lookups 2019-10-31 15:51:09 +02:00
Aliaksandr Valialkin
ca480915ca lib/storage: small cleanup in Storage.add 2019-10-31 14:30:22 +02:00
Aliaksandr Valialkin
22030b558f lib/decimal: speed up FromFloat for common case with integers 2019-10-31 13:25:09 +02:00
Aliaksandr Valialkin
6510258a80 lib/decimal: increase float64->decimal conversion precision a bit 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/213
 2019-10-30 02:04:28 +02:00
Aliaksandr Valialkin
a27e034a40 lib/storage: get parts to merge after applying the limit on the number of concurrent merges 
This should reduce write amplification under high ingestion rate.
 2019-10-30 00:09:44 +02:00
Aliaksandr Valialkin
5d2276dbf7 lib/{mergeset,storage}: limit the maximum number of concurrent merges; leave smaller number of parts during final merge 2019-10-29 12:45:37 +02:00
Aliaksandr Valialkin
78166cc478 vendor: update github.com/VictoriaMetrics/fastcache from v1.5.1 to v1.5.2 2019-10-29 11:31:36 +02:00
Aliaksandr Valialkin
f581b2736a lib/fs: typo fix in comment to WriteFileAtomically 2019-10-29 11:31:34 +02:00
Aliaksandr Valialkin
a638c6d4f8 vendor: make vendor-update 2019-10-28 13:41:13 +02:00
Roman Khavronenko
1750ee1575 * #215: update klauspost/compress lib 
* #215: bump klauspost/compress lib to 1.9.1
 2019-10-28 13:41:10 +02:00
Aliaksandr Valialkin
eb513e7ba3 lib/decimal: increase float->decimal conversion precision for big numbers 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/213
 2019-10-28 13:23:54 +02:00
Aliaksandr Valialkin
4e6bf6f538 app/vmselect: add -search.latencyOffset flag for tuning the time after data collection when data points become visible in query results 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/218
 2019-10-28 12:32:36 +02:00
Aliaksandr Valialkin
121be98325 deployment/docker: upgrade Go builder from go1.13.1 to go1.13.3 2019-10-20 23:49:18 +03:00
hanzai
52778da1f3 warns during rows addition (#214) 2019-10-20 23:38:51 +03:00
Aliaksandr Valialkin
6823aaaf08 README.md: add capacity planning chapter 2019-10-19 10:48:00 +03:00
Aliaksandr Valialkin
78fc35c9b1 all: make fmt 2019-10-17 20:05:12 +03:00
Aliaksandr Valialkin
88d793305d Makefile: disable structcheck in golangci-lint, since it gives false positive on embedded structs 2019-10-17 20:00:17 +03:00
Aliaksandr Valialkin
5b01b7fb01 all: add support for GOARCH=386 and fix all the issues related to 32-bit architectures such as GOARCH=arm 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
 2019-10-17 18:27:49 +03:00
Aliaksandr Valialkin
5d2af2cfa2 vendor: update github.com/valyala/quicktemplate from v1.2.0 to v1.3.1 2019-10-17 18:27:49 +03:00
Aliaksandr Valialkin
12c8afc3f2 lib/memory: properly handle int overflow in sysTotalMemory 
This should fix builds on 32-bit architectures such as arm.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
 2019-10-17 18:27:49 +03:00
Aliaksandr Valialkin
7d7d7a7d4e deployment/docker/docker-compose.yml: update Prometheus from v2.12.0 to v2.13.0 2019-10-16 12:39:17 +03:00