github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
466 commits 150 branches 748 tags 347 MiB
Commit graph

102 commits

Author SHA1 Message Date
Artem Navoiev
e05500cbd4 add unittests for bytesutil and storage (#221) 2019-11-04 00:57:24 +02:00
Aliaksandr Valialkin
f5fbc3ffd7 lib/{storage,uint64set}: add Set.Union() function and use it 2019-11-04 00:48:32 +02:00
Aliaksandr Valialkin
23e078261e lib/storage: tune the returned value from adjustMaxMetricsAdaptive 2019-11-04 00:45:28 +02:00
Aliaksandr Valialkin
386c349c8c lib/storage: remove interface conversion in hot path during block merging 
This should improve merge speed a bit for parts with big number of small blocks.
 2019-11-03 12:33:48 +02:00
Aliaksandr Valialkin
26ffc77622 lib/{storage,mergeset}: create missing partition directories after restoring from backups 
Backup tools could skip empty directories. So re-create such directories on the first run.
 2019-11-02 02:27:19 +02:00
Aliaksandr Valialkin
6ab9c98a1e app/vmstorage: add -bigMergeConcurrency and -smallMergeConcurrency flags for tuning the maximum number of CPU cores used during merges 2019-10-31 16:17:29 +02:00
Aliaksandr Valialkin
6a22727676 lib/storage: optimize getMetricIDsForRecentHours for per-tenant lookups 2019-10-31 15:51:09 +02:00
Aliaksandr Valialkin
ca480915ca lib/storage: small cleanup in Storage.add 2019-10-31 14:30:22 +02:00
Aliaksandr Valialkin
a27e034a40 lib/storage: get parts to merge after applying the limit on the number of concurrent merges 
This should reduce write amplification under high ingestion rate.
 2019-10-30 00:09:44 +02:00
Aliaksandr Valialkin
5d2276dbf7 lib/{mergeset,storage}: limit the maximum number of concurrent merges; leave smaller number of parts during final merge 2019-10-29 12:45:37 +02:00
hanzai
52778da1f3 warns during rows addition (#214) 2019-10-20 23:38:51 +03:00
Aliaksandr Valialkin
78fc35c9b1 all: make fmt 2019-10-17 20:05:12 +03:00
Aliaksandr Valialkin
5b01b7fb01 all: add support for GOARCH=386 and fix all the issues related to 32-bit architectures such as GOARCH=arm 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
 2019-10-17 18:27:49 +03:00
Aliaksandr Valialkin
661b8ede5b lib/storage: harden the check that the original items are sorted after mergeTagToMetricIDsRows fails to preserve sort order 2019-10-09 12:13:43 +03:00
Aliaksandr Valialkin
7f4a04ee6a lib/storage: typo fix in comment to maxRowsPerSmallPart. 2019-10-08 18:51:56 +03:00
Aliaksandr Valialkin
7e410e1412 lib/storage: add tests for mergeTagToMetricIDsRows and return the original items if the function breaks items` ordering. 
This should save from data corruption issues revealed in the previous releases up to v1.28.0-beta5.
 2019-10-08 16:35:39 +03:00
Aliaksandr Valialkin
d0b4590099 lib/storage: optimize TSID comparison 2019-09-26 14:20:02 +03:00
Aliaksandr Valialkin
95e3d648cb lib/storage: verify whether items are sorted in the end of call to mergeTagToMetricIDsRows 
This should prevent from inverted index corruption if bug in mergeTagToMetricIDsRows is discovered.
 2019-09-26 13:13:58 +03:00
Aliaksandr Valialkin
2b8358726f lib/storage: properly match labels against regexp with (?i) flag 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/161
 2019-09-26 11:03:26 +03:00
Aliaksandr Valialkin
4e3871ac1e lib/storage: add missing break in removeDuplicateMetricIDs 2019-09-25 18:23:13 +03:00
Aliaksandr Valialkin
4468f9f966 lib/storage: remove duplicate MetricIDs in tag->metricIDs items before writing them into inverted index 2019-09-25 17:57:36 +03:00
Aliaksandr Valialkin
adc18c3ee6 lib/{mergeset,storage}: do not cache inverted index blocks containing tag->metricIDs items 
This should reduce the amounts of used RAM during queries with filters over big number of time series.
 2019-09-25 13:48:24 +03:00
Aliaksandr Valialkin
de0e4eee2c lib/storage: create and use lib/uint64set instead of map[uint64]struct{} 
This should improve inverted index search performance for filters matching big number of time series,
since `lib/uint64set.Set` is faster than `map[uint64]struct{}` for both `Add` and `Has` calls.
See the corresponding benchmarks in `lib/uint64set`.
 2019-09-24 21:18:04 +03:00
Aliaksandr Valialkin
2212d0e421 lib/storage: typo fix: return dstData instead of data from mergeTagToMetricIDsRows 2019-09-24 19:32:58 +03:00
Aliaksandr Valialkin
9307de1b92 lib/storage: limit the number of metricIDs in tag->metricIDs row 
This reduces the overhead on index and metaindex in lib/mergeset
 2019-09-24 00:50:47 +03:00
Aliaksandr Valialkin
7734fc8012 lib/storage: share tsids across all the partSearch instances 
This should reduce memory usage when big number of time series matches the given query.
 2019-09-23 22:36:16 +03:00
Aliaksandr Valialkin
67a2bcb98a lib/{storage,mergeset}: verify PrepareBlock callback results 
Do not touch the first and the last item passed to PrepareBlock
in order to preserve sort order of mergeset blocks.
 2019-09-23 20:46:33 +03:00
Aliaksandr Valialkin
d2ed8cb0b2 lib/storage: generate the first tag->metricIDs item in a mergeset block with a single metricID 
The first item from each mergeset block goes into index (lib/mergeset.blockHeader),
so it must be short in order to reduce index size.
 2019-09-22 19:37:50 +03:00
Aliaksandr Valialkin
7d13c31566 lib/{storage,mergeset}: merge tag->metricID rows into tag->metricIDs rows for common tag values 
This should improve lookup performance if the same `label=value` pair exists
in big number of time series.
This should also reduce memory usage for mergeset data cache, since `tag->metricIDs` rows
occupy less space than the original `tag->metricID` rows.
 2019-09-20 22:06:23 +03:00
Aliaksandr Valialkin
7e0c6d4ca6 lib/storage: optimize selecting all the metricIDs by scanning MetricID->TSID entries instead of tag->MetricID entries 
The number of MetricID->TSID entries is smaller than the number of tag->MetricID entries
and MetricID->TSID entries are usually shorter than tag->MetricID entries.
This should improve performance when selecting all the metricIDs.
 2019-09-20 11:57:57 +03:00
Aliaksandr Valialkin
89234f395d lib/storage: use sort.Sort instead of sort.slice in getSortedMetricIDs 2019-09-19 20:08:13 +03:00
Aliaksandr Valialkin
6e586fa09c lib/storage: skip duplicate call to intersectMetricIDsWithTagFilter on zero successful intersects 2019-09-19 17:51:10 +03:00
Aliaksandr Valialkin
c05885fb5f lib/storage: mark tag filter returning errFallbackToMetricNameMatch as useless 
This will save CPU on subsequent calls for this filter
 2019-09-18 19:11:44 +03:00
Aliaksandr Valialkin
db71c940ea lib/storage: properly construct keys for uselessTagFiltersCache and register useless negative tag filters there 2019-09-17 23:18:37 +03:00
Aliaksandr Valialkin
568ff61dcf lib/mergeset: dynamically calculate the maximum number of items per part, which can be cached in OS page cache 2019-09-09 11:42:45 +03:00
Aliaksandr Valialkin
2c2bd897dd lib/storage: remove duplicate tag keys on MetricName.Marshal call 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/172
 2019-09-04 18:13:51 +03:00
Aliaksandr Valialkin
0b0153ba3d lib/storage: invalidate tagFilters -> TSIDS cache when newly added index data becomes visible to search 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/163
 2019-08-29 15:08:44 +03:00
Aliaksandr Valialkin
604a4312f9 all: port to FreeBSD on GOARCH=amd64 2019-08-28 01:46:09 +03:00
Aliaksandr Valialkin
da07a6fb38 lib/storage: go fmt 2019-08-27 14:28:24 +03:00
Aliaksandr Valialkin
a63b69e9e2 lib/storage: report proper maxMetrics limit when more than -search.maxUniqueTimeseries series match the given filters 2019-08-27 14:21:31 +03:00
Aliaksandr Valialkin
82e813bad3 lib/storage: properly handle (?i) in the tag filter regexp 
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/161
 2019-08-26 00:44:56 +03:00
Aliaksandr Valialkin
e2eac858b5 lib/storage: calculate the maximum number of rows per small part from -memory.allowedPercent 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/159

This simplifies error detection additionally to the `vm_rows_ignored_total` counters.
 2019-08-25 15:29:09 +03:00
Aliaksandr Valialkin
0a8dd9cc9a lib/storage: calculate the maximum number of rows per small part from -memory.allowedPercent 
This should improve query speed over recent data on machines with big amounts of RAM
 2019-08-25 14:41:32 +03:00
Aliaksandr Valialkin
bc576fb386 lib/storage: properly limit the number of output rows in small and big parts storage 
Previously small parts storage didn't take into account the available disk space for big parts.
 2019-08-25 14:41:32 +03:00
Aliaksandr Valialkin
947decb3dd lib/storage: remove outdated comment on maxRowsPerSmallPart 
The commend became outdated after the commit ed6ac1a5df027f0dfc22448e3b27c26b6f77c67a,
which stops merging of small parts on graceful shutdown instead of waiting
for their completion.
 2019-08-25 13:46:10 +03:00
Aliaksandr Valialkin
e734076f0f app/vminsert: allow setting the maximum number of labels per time series via -maxLabelsPerTimeseries 2019-08-23 08:47:18 +03:00
Aliaksandr Valialkin
4ed63d033a lib/storage: add benchmarks for regexp filter match / mismatch 
These benchmarks allow estimate the performance of regexp filters in promql
 2019-08-22 16:37:19 +03:00
Aliaksandr Valialkin
6ec6a8d7c1 lib/storage: try slower path for searching the tag filter with the minimum number of matching time series before giving up with increase -search.maxUniqueTimeseries error 2019-08-19 16:07:05 +03:00
Aliaksandr Valialkin
c59f5c4865 lib/storage: pre-allocate memory for blockHeader slice in unmarshalBlockHeaders 
This reduces memory usage and memory fragmentation when working with big number of time series
 2019-08-19 12:46:45 +03:00
Aliaksandr Valialkin
99eed2ca14 lib/storage: properly cache tagFilters -> TSIDs entries from historical index 2019-08-14 02:32:25 +03:00