Commit graph

2325 commits

Author SHA1 Message Date
Aliaksandr Valialkin
0fa6df94a2 lib/storage: optimize TSID comparison 2019-09-26 14:16:02 +03:00
Aliaksandr Valialkin
c39355921e lib/storage: verify whether items are sorted in the end of call to mergeTagToMetricIDsRows
This should prevent from inverted index corruption if bug in mergeTagToMetricIDsRows is discovered.
2019-09-26 13:13:41 +03:00
Artem Navoiev
cf4786f34a add test for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/161 2019-09-26 12:45:19 +03:00
Aliaksandr Valialkin
3e67862676 README.md: typo fix 2019-09-26 11:03:14 +03:00
Aliaksandr Valialkin
0db9fcedd5 lib/storage: properly match labels against regexp with (?i) flag
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/161
2019-09-26 11:03:10 +03:00
Aliaksandr Valialkin
391530bb74 README.md: mention recommended ext4 options for mkfs.ext4 when creating multi-TB partition 2019-09-25 23:52:43 +03:00
Aliaksandr Valialkin
60c5b368bc README.md: tiny updates 2019-09-25 23:29:55 +03:00
Aliaksandr Valialkin
26dc21cf64 app/vmselect/promql: add increases_over_time and decreases_over_time functions
`increases_over_time(q[d])` returns the number of `q` increases during the given duration `d`.
`decreases_over_time(q[d])` returns the number of `q` decreases during the given duration `d`.
2019-09-25 20:38:44 +03:00
Aliaksandr Valialkin
2444433d83 lib/storage: add missing break in removeDuplicateMetricIDs 2019-09-25 18:23:43 +03:00
Aliaksandr Valialkin
ea4c828bae lib/storage: remove duplicate MetricIDs in tag->metricIDs items before writing them into inverted index 2019-09-25 17:55:13 +03:00
Aliaksandr Valialkin
aebc45ad26 lib/{mergeset,storage}: do not cache inverted index blocks containing tag->metricIDs items
This should reduce the amounts of used RAM during queries with filters over big number of time series.
2019-09-25 14:02:15 +03:00
Aliaksandr Valialkin
2cb811b42f lib/uint64set: optimize Set.AppendTo 2019-09-25 00:34:17 +03:00
Aliaksandr Valialkin
b986516fbe lib/storage: create and use lib/uint64set instead of map[uint64]struct{}
This should improve inverted index search performance for filters matching big number of time series,
since `lib/uint64set.Set` is faster than `map[uint64]struct{}` for both `Add` and `Has` calls.
See the corresponding benchmarks in `lib/uint64set`.
2019-09-24 21:17:55 +03:00
Aliaksandr Valialkin
ef2296e420 lib/storage: typo fix: return dstData instead of data from mergeTagToMetricIDsRows 2019-09-24 19:32:34 +03:00
Aliaksandr Valialkin
a6086cde78 lib/storage: limit the number of metricIDs in tag->metricIDs row
This reduces the overhead on index and metaindex in lib/mergeset
2019-09-24 00:49:51 +03:00
Aliaksandr Valialkin
c9063ece66 lib/storage: share tsids across all the partSearch instances
This should reduce memory usage when big number of time series matches the given query.
2019-09-23 22:35:15 +03:00
Aliaksandr Valialkin
4e26ad869b lib/{storage,mergeset}: verify PrepareBlock callback results
Do not touch the first and the last item passed to PrepareBlock
in order to preserve sort order of mergeset blocks.
2019-09-23 20:43:13 +03:00
Aliaksandr Valialkin
0772191975 lib/mergeset: detect whether we are in test by executable suffix 2019-09-22 23:12:15 +03:00
Aliaksandr Valialkin
48999e5396 lib/workingsetcache: remove data race when resetting c.misses 2019-09-22 19:36:49 +03:00
Aliaksandr Valialkin
0adebae1f8 lib/storage: generate the first tag->metricIDs item in a mergeset block with a single metricID
The first item from each mergeset block goes into index (lib/mergeset.blockHeader),
so it must be short in order to reduce index size.
2019-09-22 19:21:33 +03:00
Aliaksandr Valialkin
267efde5ae README.md: update troubleshooting and tuning sections according to recent questions from our users 2019-09-22 19:12:24 +03:00
Aliaksandr Valialkin
0686ac52c3 lib/{storage,mergeset}: merge tag->metricID rows into tag->metricIDs rows for common tag values
This should improve lookup performance if the same `label=value` pair exists
in big number of time series.
This should also reduce memory usage for mergeset data cache, since `tag->metricIDs` rows
occupy less space than the original `tag->metricID` rows.
2019-09-20 22:06:41 +03:00
Aliaksandr Valialkin
68722c3c74 lib/encoding: optimize UnmarshalUint* and UnmarshalInt* 2019-09-20 13:08:16 +03:00
Aliaksandr Valialkin
a544f49c2b lib/storage: optimize selecting all the metricIDs by scanning MetricID->TSID entries instead of tag->MetricID entries
The number of MetricID->TSID entries is smaller than the number of tag->MetricID entries
and MetricID->TSID entries are usually shorter than tag->MetricID entries.
This should improve performance when selecting all the metricIDs.
2019-09-20 11:54:10 +03:00
Aliaksandr Valialkin
d32f88c378 app/vminsert/opentsdbhttp: remove FATAL prefix from logger.Fatalf errors for the sake of consistency with other logger.Fatalf calls 2019-09-19 22:15:59 +03:00
Aliaksandr Valialkin
00cfb2d2b9 lib/mergeset: rename misleading mergeSmallParts to mergeExistingParts 2019-09-19 21:48:20 +03:00
Aliaksandr Valialkin
37dc223e25 lib/mergeset: use sort.IsSorted instead of sort.SliceIsSorted in inmemoryBlock.isSorted in order to reduce memory allocations 2019-09-19 20:13:08 +03:00
Aliaksandr Valialkin
a84fe76677 lib/storage: use sort.Sort instead of sort.slice in getSortedMetricIDs 2019-09-19 20:07:22 +03:00
Aliaksandr Valialkin
3a697a935a lib/storage: skip duplicate call to intersectMetricIDsWithTagFilter on zero successful intersects 2019-09-19 17:49:56 +03:00
Aliaksandr Valialkin
51a21c7d4b lib/mergeset: fill partHeader.firstItem on first block flush 2019-09-19 17:48:09 +03:00
Aliaksandr Valialkin
3d83f5d334 lib/storage: mark tag filter returning errFallbackToMetricNameMatch as useless
This will save CPU on subsequent calls for this filter
2019-09-18 19:10:32 +03:00
Aliaksandr Valialkin
6f3b2fd600 deployment/docker/docker-compose.yml: update Prometheus and Grafana image tags
Prometheus: from v2.10.0 to v2.12.0
Grafana: v6.2.1 from to v6.3.5
2019-09-18 18:29:09 +03:00
Aliaksandr Valialkin
8d35718dc6 lib/storage: properly construct keys for uselessTagFiltersCache and register useless negative tag filters there 2019-09-17 23:20:27 +03:00
Aliaksandr Valialkin
33975513d0 vendor: update github.com/valyala/gozstd from v1.6.1 to v1.6.2 2019-09-16 21:50:49 +03:00
Aliaksandr Valialkin
63f2b539df vendor: make vendor-update 2019-09-13 22:48:56 +03:00
Aliaksandr Valialkin
9428ec9c9f deployment/docker: remove file system paths from the compiled binary 2019-09-13 22:45:59 +03:00
Aliaksandr Valialkin
0c8057924f lib/mergeset: properly check for sorted block headers
Fix a typo for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/181
2019-09-13 21:59:29 +03:00
Aliaksandr Valialkin
d4218d27e6 app/vmselect/promql: properly handle subqueries like aggr_func(rollup_func(metric[window:step]))
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/184
2019-09-13 21:41:04 +03:00
hanzai
e2274714b1 lib/workingsetcache: adjust switching from mode=split to mode=whole smoothly and load cachefile successfully 2019-09-13 19:13:01 +03:00
Aliaksandr Valialkin
4d636c244d app/vmselect/promql: binary operation fixes according to Prometheus behaviour
The follosing issues were fixed:
- VictoriaMetrics could leave superflouos labels when using `on` or `ignoring` modifiers
- VictoriaMetrics could return `duplicate timeseries` error when using `group_left` or `group_right` with non-empty label list
2019-09-13 17:42:52 +03:00
Aliaksandr Valialkin
bad53e4207 lib/mergeset: dynamically calculate the maximum number of items per part, which can be cached in OS page cache 2019-09-11 14:53:45 +03:00
Artem Navoiev
3f581a9860 [ci] github actions - run pipeline on pull request. Fix running of test in external PR from forks 2019-09-11 09:30:11 +03:00
sundy-li
398e00aa54 README.md: fix ExtendedPromQL link url 2019-09-10 14:56:19 +03:00
Artem Navoiev
4fd741f40d [tests] check timestamp in tests (#177) 2019-09-08 19:48:38 +03:00
Artem Navoiev
4a2cd85b92 [ci] bump version of go to 1.13 in github actions config 2019-09-08 14:02:23 +03:00
Aliaksandr Valialkin
6c46afb087 vendor: update github.com/klauspost/compress from v1.7.6 to v1.8.2 2019-09-06 00:47:31 +03:00
Aliaksandr Valialkin
7343e8b408 vendor: update golang.org/x/sys 2019-09-06 00:47:31 +03:00
Artem Navoiev
22e3fabefd Add OpenTSDB and Prometheus integration tests (#168)
* [WIP] open tsdb and prometheus integration tests

* app/victoria-metrics: fix race condition on parallel tests
2019-09-05 17:55:38 +03:00
Aliaksandr Valialkin
88f8670ede lib/fs: add MustStopDirRemover for waiting until pending directories are removed on graceful shutdown
This patch is mainly required for laggy NFS. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162
2019-09-05 11:13:17 +03:00
Aliaksandr Valialkin
9eb5de334f lib/storage: typo fix 2019-09-04 19:58:01 +03:00