github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
412 commits 150 branches 748 tags 347 MiB
Commit graph

133 commits

Author SHA1 Message Date
Aliaksandr Valialkin
9fce611fbb lib/mergeset: reduce the maximum number of cached blocks, since there are reports on OOMs due to too big caches 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/189
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/195
 2019-09-30 12:27:30 +03:00
Aliaksandr Valialkin
d0b4590099 lib/storage: optimize TSID comparison 2019-09-26 14:20:02 +03:00
Aliaksandr Valialkin
95e3d648cb lib/storage: verify whether items are sorted in the end of call to mergeTagToMetricIDsRows 
This should prevent from inverted index corruption if bug in mergeTagToMetricIDsRows is discovered.
 2019-09-26 13:13:58 +03:00
Aliaksandr Valialkin
2b8358726f lib/storage: properly match labels against regexp with (?i) flag 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/161
 2019-09-26 11:03:26 +03:00
Aliaksandr Valialkin
4e3871ac1e lib/storage: add missing break in removeDuplicateMetricIDs 2019-09-25 18:23:13 +03:00
Aliaksandr Valialkin
4468f9f966 lib/storage: remove duplicate MetricIDs in tag->metricIDs items before writing them into inverted index 2019-09-25 17:57:36 +03:00
Aliaksandr Valialkin
adc18c3ee6 lib/{mergeset,storage}: do not cache inverted index blocks containing tag->metricIDs items 
This should reduce the amounts of used RAM during queries with filters over big number of time series.
 2019-09-25 13:48:24 +03:00
Aliaksandr Valialkin
c64fb91a43 lib/uint64set: optimize Set.AppendTo 2019-09-25 00:34:31 +03:00
Aliaksandr Valialkin
de0e4eee2c lib/storage: create and use lib/uint64set instead of map[uint64]struct{} 
This should improve inverted index search performance for filters matching big number of time series,
since `lib/uint64set.Set` is faster than `map[uint64]struct{}` for both `Add` and `Has` calls.
See the corresponding benchmarks in `lib/uint64set`.
 2019-09-24 21:18:04 +03:00
Aliaksandr Valialkin
2212d0e421 lib/storage: typo fix: return dstData instead of data from mergeTagToMetricIDsRows 2019-09-24 19:32:58 +03:00
Aliaksandr Valialkin
9307de1b92 lib/storage: limit the number of metricIDs in tag->metricIDs row 
This reduces the overhead on index and metaindex in lib/mergeset
 2019-09-24 00:50:47 +03:00
Aliaksandr Valialkin
7734fc8012 lib/storage: share tsids across all the partSearch instances 
This should reduce memory usage when big number of time series matches the given query.
 2019-09-23 22:36:16 +03:00
Aliaksandr Valialkin
67a2bcb98a lib/{storage,mergeset}: verify PrepareBlock callback results 
Do not touch the first and the last item passed to PrepareBlock
in order to preserve sort order of mergeset blocks.
 2019-09-23 20:46:33 +03:00
Aliaksandr Valialkin
3304dc1e85 lib/mergeset: detect whether we are in test by executable suffix 2019-09-22 23:12:35 +03:00
Aliaksandr Valialkin
d2ed8cb0b2 lib/storage: generate the first tag->metricIDs item in a mergeset block with a single metricID 
The first item from each mergeset block goes into index (lib/mergeset.blockHeader),
so it must be short in order to reduce index size.
 2019-09-22 19:37:50 +03:00
Aliaksandr Valialkin
0a9cb6368e lib/workingsetcache: remove data race when resetting c.misses 2019-09-22 19:37:09 +03:00
Aliaksandr Valialkin
7d13c31566 lib/{storage,mergeset}: merge tag->metricID rows into tag->metricIDs rows for common tag values 
This should improve lookup performance if the same `label=value` pair exists
in big number of time series.
This should also reduce memory usage for mergeset data cache, since `tag->metricIDs` rows
occupy less space than the original `tag->metricID` rows.
 2019-09-20 22:06:23 +03:00
Aliaksandr Valialkin
272e2f77c9 lib/encoding: optimize UnmarshalUint* and UnmarshalInt* 2019-09-20 13:08:24 +03:00
Aliaksandr Valialkin
7e0c6d4ca6 lib/storage: optimize selecting all the metricIDs by scanning MetricID->TSID entries instead of tag->MetricID entries 
The number of MetricID->TSID entries is smaller than the number of tag->MetricID entries
and MetricID->TSID entries are usually shorter than tag->MetricID entries.
This should improve performance when selecting all the metricIDs.
 2019-09-20 11:57:57 +03:00
Aliaksandr Valialkin
bf8505353a lib/mergeset: rename misleading mergeSmallParts to mergeExistingParts 2019-09-19 21:48:36 +03:00
Aliaksandr Valialkin
ebbef20535 lib/mergeset: use sort.IsSorted instead of sort.SliceIsSorted in inmemoryBlock.isSorted in order to reduce memory allocations 2019-09-19 20:13:54 +03:00
Aliaksandr Valialkin
89234f395d lib/storage: use sort.Sort instead of sort.slice in getSortedMetricIDs 2019-09-19 20:08:13 +03:00
Aliaksandr Valialkin
6e586fa09c lib/storage: skip duplicate call to intersectMetricIDsWithTagFilter on zero successful intersects 2019-09-19 17:51:10 +03:00
Aliaksandr Valialkin
410f993bf6 lib/mergeset: fill partHeader.firstItem on first block flush 2019-09-19 17:48:22 +03:00
Aliaksandr Valialkin
c05885fb5f lib/storage: mark tag filter returning errFallbackToMetricNameMatch as useless 
This will save CPU on subsequent calls for this filter
 2019-09-18 19:11:44 +03:00
Aliaksandr Valialkin
db71c940ea lib/storage: properly construct keys for uselessTagFiltersCache and register useless negative tag filters there 2019-09-17 23:18:37 +03:00
Aliaksandr Valialkin
41ef6b060e lib/mergeset: properly check for sorted block headers 
Fix a typo for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/181
 2019-09-13 21:59:38 +03:00
hanzai
08cde5e3f4 lib/workingsetcache: adjust switching from mode=split to mode=whole smoothly and load cachefile successfully 2019-09-13 19:13:16 +03:00
Aliaksandr Valialkin
b101064f8b all: report the number of bytes read on io.ReadFull error 
This should simplify error investigation similar to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/175
 2019-09-11 14:50:24 +03:00
Aliaksandr Valialkin
568ff61dcf lib/mergeset: dynamically calculate the maximum number of items per part, which can be cached in OS page cache 2019-09-09 11:42:45 +03:00
Aliaksandr Valialkin
2c654258ef lib/fs: add MustStopDirRemover for waiting until pending directories are removed on graceful shutdown 
This patch is mainly required for laggy NFS. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162
 2019-09-05 11:17:17 +03:00
Aliaksandr Valialkin
2c2bd897dd lib/storage: remove duplicate tag keys on MetricName.Marshal call 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/172
 2019-09-04 18:13:51 +03:00
Aliaksandr Valialkin
82bfe818d0 lib/fs: try harder with directory removal on NFS in the event of temporary lock 
Do not give up after 11 attempts of directory removal on laggy NFS.

Add `vm_nfs_dir_remove_failed_attempts_total` metric for counting the number of failed attempts
on directory removal.

Log failed attempts on directory removal after long sleep times.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162
 2019-09-04 12:24:41 +03:00
Aliaksandr Valialkin
0b0153ba3d lib/storage: invalidate tagFilters -> TSIDS cache when newly added index data becomes visible to search 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/163
 2019-08-29 15:08:44 +03:00
Aliaksandr Valialkin
8504a38214 lib/prombp: apply ba06b47c16 
The following commands used:

gofmt -r '(uint64(x)&0x7F)<<shift -> uint64(x&0x7F)<<shift' -w ./lib/prompb/
gofmt -r '(int64(x)&0x7F)<<shift -> int64(x&0x7F)<<shift' -w ./lib/prompb/
 2019-08-29 13:35:54 +03:00
Aliaksandr Valialkin
604a4312f9 all: port to FreeBSD on GOARCH=amd64 2019-08-28 01:46:09 +03:00
Aliaksandr Valialkin
da07a6fb38 lib/storage: go fmt 2019-08-27 14:28:24 +03:00
Aliaksandr Valialkin
a63b69e9e2 lib/storage: report proper maxMetrics limit when more than -search.maxUniqueTimeseries series match the given filters 2019-08-27 14:21:31 +03:00
Aliaksandr Valialkin
82e813bad3 lib/storage: properly handle (?i) in the tag filter regexp 
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/161
 2019-08-26 00:44:56 +03:00
Aliaksandr Valialkin
e2eac858b5 lib/storage: calculate the maximum number of rows per small part from -memory.allowedPercent 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/159

This simplifies error detection additionally to the `vm_rows_ignored_total` counters.
 2019-08-25 15:29:09 +03:00
Aliaksandr Valialkin
0a8dd9cc9a lib/storage: calculate the maximum number of rows per small part from -memory.allowedPercent 
This should improve query speed over recent data on machines with big amounts of RAM
 2019-08-25 14:41:32 +03:00
Aliaksandr Valialkin
bc576fb386 lib/storage: properly limit the number of output rows in small and big parts storage 
Previously small parts storage didn't take into account the available disk space for big parts.
 2019-08-25 14:41:32 +03:00
Aliaksandr Valialkin
947decb3dd lib/storage: remove outdated comment on maxRowsPerSmallPart 
The commend became outdated after the commit ed6ac1a5df027f0dfc22448e3b27c26b6f77c67a,
which stops merging of small parts on graceful shutdown instead of waiting
for their completion.
 2019-08-25 13:46:10 +03:00
Aliaksandr Valialkin
c197641978 all: return 503 http error if service is temporarily unavailable 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/156
 2019-08-23 09:49:50 +03:00
Aliaksandr Valialkin
e734076f0f app/vminsert: allow setting the maximum number of labels per time series via -maxLabelsPerTimeseries 2019-08-23 08:47:18 +03:00
Aliaksandr Valialkin
4ed63d033a lib/storage: add benchmarks for regexp filter match / mismatch 
These benchmarks allow estimate the performance of regexp filters in promql
 2019-08-22 16:37:19 +03:00
Aliaksandr Valialkin
6ec6a8d7c1 lib/storage: try slower path for searching the tag filter with the minimum number of matching time series before giving up with increase -search.maxUniqueTimeseries error 2019-08-19 16:07:05 +03:00
Aliaksandr Valialkin
c59f5c4865 lib/storage: pre-allocate memory for blockHeader slice in unmarshalBlockHeaders 
This reduces memory usage and memory fragmentation when working with big number of time series
 2019-08-19 12:46:45 +03:00
Aliaksandr Valialkin
f696cc503a lib/workingsetcache: automatically detect when it is better to double cache capacity 2019-08-15 22:58:04 +03:00
Aliaksandr Valialkin
99eed2ca14 lib/storage: properly cache tagFilters -> TSIDs entries from historical index 2019-08-14 02:32:25 +03:00