Commit graph

3430 commits

Author SHA1 Message Date
Aliaksandr Valialkin
02e0b19a62 lib/storage: tune the returned value from adjustMaxMetricsAdaptive 2019-11-04 00:44:37 +02:00
Aliaksandr Valialkin
6be4456d88 lib/{storage,uint64set}: add Set.Union() function and use it 2019-11-04 00:44:37 +02:00
Aliaksandr Valialkin
9becc26f4b lib/storage: remove interface conversion in hot path during block merging
This should improve merge speed a bit for parts with big number of small blocks.
2019-11-03 12:33:34 +02:00
Aliaksandr Valialkin
c62399eb3e lib/{storage,mergeset}: create missing partition directories after restoring from backups
Backup tools could skip empty directories. So re-create such directories on the first run.
2019-11-02 02:27:11 +02:00
Aliaksandr Valialkin
55d728c849 lib/{decimal,encoding}: optimize float64<->decimal conversion for arrays with zeros or ones
Time series with only zeros or ones frequently occur in monitoring, so it is worth optimizing their handling.
2019-11-01 16:48:12 +02:00
Aliaksandr Valialkin
808fc0971f lib/{encoding,decimal}: add benchmarks for blocks containing zeros or ones
Time series with such values are quite common in monitoring space,
so it would be great to have benchmarks for them.
2019-11-01 16:48:12 +02:00
Aliaksandr Valialkin
370cfbb365 lib/uint64set: return an emptry set instead of nil set from Set.Clone, since the caller may add data to the cloned set
This fixes the following panic in v1.28.1:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x783a7e]

goroutine 1155 [running]:
github.com/VictoriaMetrics/VictoriaMetrics/lib/uint64set.(*Set).Add(0x0, 0x15b3bfb41e8b71ec)
  github.com/VictoriaMetrics/VictoriaMetrics@/lib/uint64set/uint64set.go:57 +0x2e
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*indexSearch).getMetricIDsForRecentHours(0xc5bdc0dd40, 0x16e273f6b50, 0x16e2745d3f0, 0x5b8d95, 0x10, 0x4a2f51, 0xaa01000000000000)
  github.com/VictoriaMetrics/VictoriaMetrics@/lib/storage/index_db.go:1951 +0x260
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*indexSearch).getMetricIDsForTimeRange(0xc5bdc0dd40, 0x16e273f6b50, 0x16e2745d3f0, 0x5b8d95, 0x10, 0xb296c0, 0xc00009cd80, 0x9bc640)
2019-11-01 16:12:44 +02:00
Aliaksandr Valialkin
2f58f37f07 app/vmselect/promql: add lag(q[d]) function, which returns the lag between the current timestamp and the timstamp for the last data point in q 2019-11-01 12:21:33 +02:00
Aliaksandr Valialkin
d18ea0c95b app/vmstorage: add -bigMergeConcurrency and -smallMergeConcurrency flags for tuning the maximum number of CPU cores used during merges 2019-10-31 16:19:13 +02:00
Aliaksandr Valialkin
e0b292c6de lib/storage: small cleanup in Storage.add 2019-10-31 14:30:34 +02:00
Aliaksandr Valialkin
86f6be40db README.md: update information about vm_rows{type="indexdb"} metric
The previous information became outdated after v1.28.0, since now each row in the inverted index
can refer to multiple time series.
2019-10-31 13:30:29 +02:00
Aliaksandr Valialkin
e76e21e4c7 lib/decimal: speed up FromFloat for common case with integers 2019-10-31 13:24:59 +02:00
Aliaksandr Valialkin
cfa5e279c2 lib/decimal: increase float64->decimal conversion precision a bit
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/213
2019-10-30 02:04:56 +02:00
Aliaksandr Valialkin
fa7c3ab93a README.md: fix delimiter between {measurement} and {field_name} in the Influx line protocol example 2019-10-30 02:04:56 +02:00
Aliaksandr Valialkin
26d570bb3a lib/storage: get parts to merge after applying the limit on the number of concurrent merges
This should reduce write amplification under high ingestion rate.
2019-10-30 02:04:56 +02:00
Roman Khavronenko
62ed508546
Bump version requirements in description 2019-10-29 22:29:48 +00:00
Aliaksandr Valialkin
2e2eff90d5 lib/{mergeset,storage}: limit the maximum number of concurrent merges; leave smaller number of parts during final merge 2019-10-29 12:45:28 +02:00
Aliaksandr Valialkin
855e5c8963 vendor: update github.com/VictoriaMetrics/fastcache from v1.5.1 to v1.5.2 2019-10-29 11:31:29 +02:00
Aliaksandr Valialkin
04e48ef064 lib/fs: typo fix in comment to WriteFileAtomically 2019-10-29 11:31:26 +02:00
Roman Khavronenko
971206b514 update single-version dashboard with panels: (#219)
* concurrent inserts
* rows ignored
2019-10-28 13:54:10 +02:00
Aliaksandr Valialkin
d063bfaf83 vendor: make vendor-update 2019-10-28 13:39:05 +02:00
Roman Khavronenko
6ab48838bf #215: update klauspost/compress lib (#217)
* #215: update klauspost/compress lib

* #215: bump klauspost/compress lib to 1.9.1
2019-10-28 13:36:35 +02:00
Aliaksandr Valialkin
a42b5db39f lib/decimal: increase float->decimal conversion precision for big numbers
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/213
2019-10-28 13:23:44 +02:00
Aliaksandr Valialkin
b0295dbf2e app/vmselect: add -search.latencyOffset flag for tuning the time after data collection when data points become visible in query results
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/218
2019-10-28 12:31:07 +02:00
Petr Mikusek
3cea200309 Fix typo s/telergam/telegram/ in README.md 2019-10-23 19:30:36 +03:00
Aliaksandr Valialkin
32600ba4fc deployment/docker: upgrade Go builder from go1.13.1 to go1.13.3 2019-10-20 23:50:05 +03:00
hanzai
b3c946e35a warns during rows addition (#214) 2019-10-20 23:41:07 +03:00
Aliaksandr Valialkin
e83fe938c8 all: make fmt 2019-10-17 20:04:34 +03:00
Aliaksandr Valialkin
f708aa7003 Makefile: disable structcheck in golangci-lint, since it gives false positive on embedded structs 2019-10-17 19:59:10 +03:00
Aliaksandr Valialkin
97ce4e03a5 all: add support for GOARCH=386 and fix all the issues related to 32-bit architectures such as GOARCH=arm
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
2019-10-17 18:23:23 +03:00
Aliaksandr Valialkin
a398343bb6 vendor: update github.com/valyala/quicktemplate from v1.2.0 to v1.3.1 2019-10-17 18:23:19 +03:00
Aliaksandr Valialkin
6ebf537153 lib/memory: properly handle int overflow in sysTotalMemory
This should fix builds on 32-bit architectures such as arm.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
2019-10-17 00:50:48 +03:00
Aliaksandr Valialkin
f752479cb8 app/victoria-metrics/test: add missing docs to public funcs PopulateTimeTplString and PopulateTimeTpl 2019-10-17 00:50:46 +03:00
Aliaksandr Valialkin
61e956e175 app/victoria-metrics: add a test for max_lookback=<duration> query arg
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209
2019-10-15 21:31:48 +03:00
Aliaksandr Valialkin
c66a691593 app/vmselect/prometheus: add -search.maxLookback command-line flag for overriding dynamic calculations for max lookback interval
This flag is similar to `-search.lookback-delta` if set. The max lookback interval is determined dynamically
from interval between datapoints for each input time series if the flag isn't set.

The interval can be overriden on per-query basis by passing `max_lookback=<duration>` query arg to `/api/v1/query` and `/api/v1/query_range`.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209
2019-10-15 21:31:48 +03:00
Aliaksandr Valialkin
cc21b31502 app/victoria-metrics/test: add a test for PopulateTimeTplString 2019-10-15 21:31:48 +03:00
Aliaksandr Valialkin
195cefd81a lib/prompb: removed outdated README.md 2019-10-14 22:12:57 +03:00
Aliaksandr Valialkin
c1581c3810 vendor: make vendor-update 2019-10-13 23:17:47 +03:00
Aliaksandr Valialkin
16cae15c45 README.md: add integrations section 2019-10-11 19:14:28 +03:00
Aliaksandr Valialkin
f6334bffa1 lib/storage: harden the check that the original items are sorted after mergeTagToMetricIDsRows fails to preserve sort order 2019-10-09 12:13:17 +03:00
Aliaksandr Valialkin
2abd5154e0 lib/storage: typo fix in comment to maxRowsPerSmallPart. 2019-10-08 18:51:20 +03:00
Aliaksandr Valialkin
c1cf7d9f93 lib/storage: add tests for mergeTagToMetricIDsRows and return the original items if the function breaks items` ordering.
This should save from data corruption issues revealed in the previous releases up to v1.28.0-beta5.
2019-10-08 16:27:35 +03:00
Aliaksandr Valialkin
956fdd89d3 app/vmselect/promql: take into account the previous point when calculating max_over_time and min_over_time
This lines up with `first_over_time` function used in `rollup_candlestick`, so `rollup=low` always returns
the minimum value.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/204
2019-10-08 12:30:05 +03:00
Alexander Danilov
1bc6377863 Improve documentation a little bit 2019-10-07 22:18:40 +03:00
Artem Navoiev
1e2c511747 Add regression test for query apo
Part of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/187
cover:
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/153
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/150
2019-10-07 22:18:04 +03:00
Aliaksandr Valialkin
0eeffb910f vendor: make vendor-update 2019-10-06 15:47:23 +03:00
Aliaksandr Valialkin
4ba86f501a vendor: update github.com/VictoriaMetrics/metrics from v1.7.1 to v1.7.2 2019-10-06 11:20:45 +03:00
Aliaksandr Valialkin
fdc5cfd838 lib/mergeset: reduce the maximum number of cached blocks, since there are reports on OOMs due to too big caches
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/189
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/195
2019-09-30 12:25:40 +03:00
Artem Navoiev
a116f5e7c1 Add regression test for query apo (#194)
Part of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/187
cover:
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/184
2019-09-30 11:25:54 +03:00
Aliaksandr Valialkin
4e9e1ca0f7 app/vmselect/netstorage: hint the OS that tmpBlocksFile is read almost sequentially
This became the case after b7ee2e7af2 .
2019-09-30 00:11:14 +03:00