Aliaksandr Valialkin
c62399eb3e
lib/{storage,mergeset}: create missing partition directories after restoring from backups
...
Backup tools could skip empty directories. So re-create such directories on the first run.
2019-11-02 02:27:11 +02:00
Aliaksandr Valialkin
2e2eff90d5
lib/{mergeset,storage}: limit the maximum number of concurrent merges; leave smaller number of parts during final merge
2019-10-29 12:45:28 +02:00
Aliaksandr Valialkin
97ce4e03a5
all: add support for GOARCH=386 and fix all the issues related to 32-bit architectures such as GOARCH=arm
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
2019-10-17 18:23:23 +03:00
Aliaksandr Valialkin
fdc5cfd838
lib/mergeset: reduce the maximum number of cached blocks, since there are reports on OOMs due to too big caches
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/189
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/195
2019-09-30 12:25:40 +03:00
Aliaksandr Valialkin
aebc45ad26
lib/{mergeset,storage}: do not cache inverted index blocks containing tag->metricIDs items
...
This should reduce the amounts of used RAM during queries with filters over big number of time series.
2019-09-25 14:02:15 +03:00
Aliaksandr Valialkin
b986516fbe
lib/storage: create and use lib/uint64set instead of map[uint64]struct{}
...
This should improve inverted index search performance for filters matching big number of time series,
since `lib/uint64set.Set` is faster than `map[uint64]struct{}` for both `Add` and `Has` calls.
See the corresponding benchmarks in `lib/uint64set`.
2019-09-24 21:17:55 +03:00
Aliaksandr Valialkin
4e26ad869b
lib/{storage,mergeset}: verify PrepareBlock callback results
...
Do not touch the first and the last item passed to PrepareBlock
in order to preserve sort order of mergeset blocks.
2019-09-23 20:43:13 +03:00
Aliaksandr Valialkin
0772191975
lib/mergeset: detect whether we are in test by executable suffix
2019-09-22 23:12:15 +03:00
Aliaksandr Valialkin
0686ac52c3
lib/{storage,mergeset}: merge tag->metricID rows into tag->metricIDs rows for common tag values
...
This should improve lookup performance if the same `label=value` pair exists
in big number of time series.
This should also reduce memory usage for mergeset data cache, since `tag->metricIDs` rows
occupy less space than the original `tag->metricID` rows.
2019-09-20 22:06:41 +03:00
Aliaksandr Valialkin
00cfb2d2b9
lib/mergeset: rename misleading mergeSmallParts to mergeExistingParts
2019-09-19 21:48:20 +03:00
Aliaksandr Valialkin
37dc223e25
lib/mergeset: use sort.IsSorted instead of sort.SliceIsSorted in inmemoryBlock.isSorted in order to reduce memory allocations
2019-09-19 20:13:08 +03:00
Aliaksandr Valialkin
51a21c7d4b
lib/mergeset: fill partHeader.firstItem on first block flush
2019-09-19 17:48:09 +03:00
Aliaksandr Valialkin
0c8057924f
lib/mergeset: properly check for sorted block headers
...
Fix a typo for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/181
2019-09-13 21:59:29 +03:00
Aliaksandr Valialkin
bad53e4207
lib/mergeset: dynamically calculate the maximum number of items per part, which can be cached in OS page cache
2019-09-11 14:53:45 +03:00
Aliaksandr Valialkin
e1d76ec1f3
lib/storage: invalidate tagFilters -> TSIDS cache when newly added index data becomes visible to search
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/163
2019-08-29 15:08:35 +03:00
Aliaksandr Valialkin
9196c085a7
all: port to FreeBSD on GOARCH=amd64
2019-08-28 01:19:23 +03:00
Aliaksandr Valialkin
0967683ae9
lib: move common code for creating flock.lock file into fs.CreateFlockFile
2019-08-13 01:45:46 +03:00
Aliaksandr Valialkin
5d8d110010
lib/fs: atomically create file with the given contents on WriteFileAtomically
...
This should prevent from `transaction` and `metadata.json` files corruption
on unclean shutdown such as OOM, `kill -9`, power loss, etc.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/148
2019-08-12 15:02:55 +03:00
Aliaksandr Valialkin
1fe6d784d8
all: consistency renaming: bytesSize -> sizeBytes
2019-07-10 00:47:36 +03:00
Aliaksandr Valialkin
56c154f45b
all: add vm_data_size_bytes metrics for easy monitoring of on-disk data size and on-disk inverted index size
2019-07-04 19:42:30 +03:00
Aliaksandr Valialkin
a41955863a
lib/mergeset: make fmt
2019-06-29 14:25:26 +03:00
Aliaksandr Valialkin
0c88afa386
lib/mergeset: speed up binarySearchKey by skipping the first item during binary search
2019-06-29 13:45:49 +03:00
Aliaksandr Valialkin
af2ceaaa0b
lib/storage: mention source parts on merge error
...
This should improve determining broken source part.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/76
2019-06-24 14:08:43 +03:00
Aliaksandr Valialkin
419197ba08
lib/fs: consolidate *RemoveAll* funcs into a single MustRemoveAll func
...
The func syncs parent dir in order to persist directory removal
in the event of power loss
2019-06-12 01:53:46 +03:00
Aliaksandr Valialkin
935bfd7a18
lib/fs: consistency renaming SyncPath -> MustSyncPath, since it doesnt return error
2019-06-11 23:13:49 +03:00
Aliaksandr Valialkin
20fc0e0e54
lib/{storage,mergeset}: sync filenames inside part when finalizing the part
...
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/63
2019-06-11 21:51:13 +03:00
Aliaksandr Valialkin
ac7b186f13
all: try hard removing directory with contents
...
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/61
2019-06-11 01:57:59 +03:00
Aliaksandr Valialkin
54fb8b21f9
all: fix misspellings
2019-05-25 21:51:11 +03:00
Aliaksandr Valialkin
1836c415e6
all: open-sourcing single-node version
2019-05-23 00:18:06 +03:00
