Aliaksandr Valialkin
9ea2bd822e
lib/storage: implement per-day inverted index
2019-11-10 00:20:32 +02:00
Aliaksandr Valialkin
dea2f3efed
lib/storage: use specialized cache for (date, metricID) entries
...
This improves ingestion performance.
2019-11-09 23:09:18 +02:00
Aliaksandr Valialkin
9a43902bd8
lib/storage: remove unused code from getMetricIDsForTimeRange: it is expected that time range is always non-zero
2019-11-09 19:03:51 +02:00
Aliaksandr Valialkin
c16e17dede
lib/storage: properly set time range when deleting time series
2019-11-09 18:50:02 +02:00
Aliaksandr Valialkin
8126007c15
lib/storage: obtain all the time series ids from (tag->metricIDs) rows instead of (metricID->TSID) rows, since this much faster
2019-11-09 18:04:26 +02:00
Aliaksandr Valialkin
50773348d3
lib/storage: small code prettifying
2019-11-09 14:01:24 +02:00
Aliaksandr Valialkin
0bc54c23ce
lib/storage: inmemoryInvertedIndex prettifying
2019-11-09 14:01:24 +02:00
Aliaksandr Valialkin
46e67bb78c
lib/storage: export vm_new_timeseries_created_total metric for determining time series churn rate
2019-11-08 19:58:21 +02:00
Aliaksandr Valialkin
0063c857f5
lib/storage: add inmemory inverted index for the last hour
...
It should improve performance for `last N hours` dashboards with update intervals smaller than 1 hour.
2019-11-08 19:37:46 +02:00
Aliaksandr Valialkin
89c03a5464
lib/storage: populate partition names from both small and big directories
...
Certain partition directories may be missing after restoring from backups
if they had no data. Re-create such directories on start.
2019-11-06 19:50:21 +02:00
Aliaksandr Valialkin
1c777e0245
lib/storage: substitute error message about unsorted items in the index block after metricIDs merge with counter
...
The origin of the error has been detected and documented in the code,
so it is enough to export a counter for such errors at `vm_index_blocks_with_metric_ids_incorrect_order_total`,
so it could be monitored and alerted on high error rates.
Export also the counter for processed index blocks with metricIDs - `vm_index_blocks_with_metric_ids_processed_total`,
so its' rate could be compared to `rate(vm_index_blocks_with_metric_ids_incorrect_order_total)`.
2019-11-06 14:32:41 +02:00
Aliaksandr Valialkin
c567a4353a
lib/storage: take into account the requested time range when caching TSIDs for the given tag filters
2019-11-06 14:32:41 +02:00
Aliaksandr Valialkin
c6564c5d26
lib/storage: dump incorrectly sorted items on a single line; this should simplify error reporting
2019-11-05 18:41:50 +02:00
Aliaksandr Valialkin
a10c4cad85
lib/storage: return back finalPartsToMerge from 2 to 3 in order to prevent from excessive merges in old partitions
2019-11-05 17:28:57 +02:00
Aliaksandr Valialkin
e5b1fa0c38
lib/storage: separate the max inverted index scan loops per metric into fast and slow loops
...
Slow loops could require seeks and expensive regexp matching, while fast loops just scans
all the metricIDs for the given `tag=value` prefix. So these operations must have separate
max loops multiplier.
2019-11-05 17:28:57 +02:00
Aliaksandr Valialkin
f93c4f2493
lib/storage: skip repeated useless work when intersection of metricIDs with the given filter is too expensive
...
This should improve performance for query filters over big number of time series.
2019-11-05 14:35:55 +02:00
Aliaksandr Valialkin
f48e97263c
lib/storage: reduce the maximum inverted index scans before giving up to label filters matching by metric name
...
The new value reduces the amount of wasted work during index scans over big number of time series.
2019-11-05 14:35:53 +02:00
Aliaksandr Valialkin
d2f688c550
lib/storage: try potentially faster tag filters at first, then apply slower tag filters
...
The fastest tag filters are non-negative non-regexp, since they are the most specific.
The slowest tag filters are negative regexp, since they require scanning
all the entries for the given label.
2019-11-05 14:35:48 +02:00
Aliaksandr Valialkin
2a38d30f93
lib/storage: pass pointer to MetricName in Fatalf, so it is properly detected as an interface with String() method
...
This fixes lint errors
2019-11-04 01:06:45 +02:00
Artem Navoiev
e05500cbd4
add unittests for bytesutil and storage ( #221 )
2019-11-04 00:57:24 +02:00
Aliaksandr Valialkin
f5fbc3ffd7
lib/{storage,uint64set}: add Set.Union() function and use it
2019-11-04 00:48:32 +02:00
Aliaksandr Valialkin
23e078261e
lib/storage: tune the returned value from adjustMaxMetricsAdaptive
2019-11-04 00:45:28 +02:00
Aliaksandr Valialkin
386c349c8c
lib/storage: remove interface conversion in hot path during block merging
...
This should improve merge speed a bit for parts with big number of small blocks.
2019-11-03 12:33:48 +02:00
Aliaksandr Valialkin
26ffc77622
lib/{storage,mergeset}: create missing partition directories after restoring from backups
...
Backup tools could skip empty directories. So re-create such directories on the first run.
2019-11-02 02:27:19 +02:00
Aliaksandr Valialkin
6ab9c98a1e
app/vmstorage: add -bigMergeConcurrency and -smallMergeConcurrency flags for tuning the maximum number of CPU cores used during merges
2019-10-31 16:17:29 +02:00
Aliaksandr Valialkin
6a22727676
lib/storage: optimize getMetricIDsForRecentHours for per-tenant lookups
2019-10-31 15:51:09 +02:00
Aliaksandr Valialkin
ca480915ca
lib/storage: small cleanup in Storage.add
2019-10-31 14:30:22 +02:00
Aliaksandr Valialkin
a27e034a40
lib/storage: get parts to merge after applying the limit on the number of concurrent merges
...
This should reduce write amplification under high ingestion rate.
2019-10-30 00:09:44 +02:00
Aliaksandr Valialkin
5d2276dbf7
lib/{mergeset,storage}: limit the maximum number of concurrent merges; leave smaller number of parts during final merge
2019-10-29 12:45:37 +02:00
hanzai
52778da1f3
warns during rows addition ( #214 )
2019-10-20 23:38:51 +03:00
Aliaksandr Valialkin
78fc35c9b1
all: make fmt
2019-10-17 20:05:12 +03:00
Aliaksandr Valialkin
5b01b7fb01
all: add support for GOARCH=386 and fix all the issues related to 32-bit architectures such as GOARCH=arm
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
2019-10-17 18:27:49 +03:00
Aliaksandr Valialkin
661b8ede5b
lib/storage: harden the check that the original items are sorted after mergeTagToMetricIDsRows fails to preserve sort order
2019-10-09 12:13:43 +03:00
Aliaksandr Valialkin
7f4a04ee6a
lib/storage: typo fix in comment to maxRowsPerSmallPart.
2019-10-08 18:51:56 +03:00
Aliaksandr Valialkin
7e410e1412
lib/storage: add tests for mergeTagToMetricIDsRows and return the original items if the function breaks items` ordering.
...
This should save from data corruption issues revealed in the previous releases up to v1.28.0-beta5.
2019-10-08 16:35:39 +03:00
Aliaksandr Valialkin
d0b4590099
lib/storage: optimize TSID comparison
2019-09-26 14:20:02 +03:00
Aliaksandr Valialkin
95e3d648cb
lib/storage: verify whether items are sorted in the end of call to mergeTagToMetricIDsRows
...
This should prevent from inverted index corruption if bug in mergeTagToMetricIDsRows is discovered.
2019-09-26 13:13:58 +03:00
Aliaksandr Valialkin
2b8358726f
lib/storage: properly match labels against regexp with (?i) flag
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/161
2019-09-26 11:03:26 +03:00
Aliaksandr Valialkin
4e3871ac1e
lib/storage: add missing break in removeDuplicateMetricIDs
2019-09-25 18:23:13 +03:00
Aliaksandr Valialkin
4468f9f966
lib/storage: remove duplicate MetricIDs in tag->metricIDs items before writing them into inverted index
2019-09-25 17:57:36 +03:00
Aliaksandr Valialkin
adc18c3ee6
lib/{mergeset,storage}: do not cache inverted index blocks containing tag->metricIDs items
...
This should reduce the amounts of used RAM during queries with filters over big number of time series.
2019-09-25 13:48:24 +03:00
Aliaksandr Valialkin
de0e4eee2c
lib/storage: create and use lib/uint64set instead of map[uint64]struct{}
...
This should improve inverted index search performance for filters matching big number of time series,
since `lib/uint64set.Set` is faster than `map[uint64]struct{}` for both `Add` and `Has` calls.
See the corresponding benchmarks in `lib/uint64set`.
2019-09-24 21:18:04 +03:00
Aliaksandr Valialkin
2212d0e421
lib/storage: typo fix: return dstData instead of data from mergeTagToMetricIDsRows
2019-09-24 19:32:58 +03:00
Aliaksandr Valialkin
9307de1b92
lib/storage: limit the number of metricIDs in tag->metricIDs row
...
This reduces the overhead on index and metaindex in lib/mergeset
2019-09-24 00:50:47 +03:00
Aliaksandr Valialkin
7734fc8012
lib/storage: share tsids across all the partSearch instances
...
This should reduce memory usage when big number of time series matches the given query.
2019-09-23 22:36:16 +03:00
Aliaksandr Valialkin
67a2bcb98a
lib/{storage,mergeset}: verify PrepareBlock callback results
...
Do not touch the first and the last item passed to PrepareBlock
in order to preserve sort order of mergeset blocks.
2019-09-23 20:46:33 +03:00
Aliaksandr Valialkin
d2ed8cb0b2
lib/storage: generate the first tag->metricIDs item in a mergeset block with a single metricID
...
The first item from each mergeset block goes into index (lib/mergeset.blockHeader),
so it must be short in order to reduce index size.
2019-09-22 19:37:50 +03:00
Aliaksandr Valialkin
7d13c31566
lib/{storage,mergeset}: merge tag->metricID rows into tag->metricIDs rows for common tag values
...
This should improve lookup performance if the same `label=value` pair exists
in big number of time series.
This should also reduce memory usage for mergeset data cache, since `tag->metricIDs` rows
occupy less space than the original `tag->metricID` rows.
2019-09-20 22:06:23 +03:00
Aliaksandr Valialkin
7e0c6d4ca6
lib/storage: optimize selecting all the metricIDs by scanning MetricID->TSID entries instead of tag->MetricID entries
...
The number of MetricID->TSID entries is smaller than the number of tag->MetricID entries
and MetricID->TSID entries are usually shorter than tag->MetricID entries.
This should improve performance when selecting all the metricIDs.
2019-09-20 11:57:57 +03:00
Aliaksandr Valialkin
89234f395d
lib/storage: use sort.Sort instead of sort.slice in getSortedMetricIDs
2019-09-19 20:08:13 +03:00
