github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
VictoriaMetrics/lib/storage
History
Artem Fetishev 6b9f57e5f7
lib/storage: Fix flaky test: TestStorageRotateIndexDB (#7267) 
This commit fixes the TestStorageRotateIndexDB flaky test reported at:
#6977. Sample test failure: https://pastebin.com/bTSs8HP1

The test fails because one goroutine adds items to the indexDB table
while another goroutine is closing that table. This may happen if
indexDB rotation happens twice during one Storage.add() operation:
-  Storage.add() takes the current indexDB and adds index recods to it
- First index db rotation makes the current index DB a previous one
(still ok at this point)
- Second index db rotation removes the indexDB that was current two
rotations earlier. It does this by setting the mustDrop flag to true and
decrementing the ref counter. The ref counter reaches zero which cases
the underlying indexdb table to release its resources gracefully.
Graceful release assumes that the table is not written anymore. But
Storage.add() still adds items to it.

The solution is to increment the indexDB ref counters while it is used
inside add().
The unit test has been changed a little so that the test fails reliably.
The idea is to make add() function invocation to last much longer,
therefore the test inserts not just one record at a time but thouthands
of them.

To see the test fail, just replace the idbsLocked() func with:

```go
unc (s *Storage) idbsLocked2() (*indexDB, *indexDB, func()) {
       return s.idbCurr.Load(), s.idbNext.Load(), func() {}
}
``` 


---------

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2024-10-23 11:48:21 +02:00
..
block.go lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit 2024-05-14 01:23:54 +02:00
block_header.go lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit 2024-05-14 01:23:54 +02:00
block_header_test.go lib/storage: typo fix: umarshal -> unmarshal 2021-03-02 20:47:59 +02:00
block_stream_merger.go all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:20:37 +02:00
block_stream_reader.go lib/storage: avoid superflouos copy of block header data 2024-02-22 20:21:14 +02:00
block_stream_reader_test.go Makefile: update golangci-lint from v1.51.2 to v1.54.2 2023-09-01 10:16:42 +02:00
block_stream_reader_timing_test.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:46:12 -07:00
block_stream_writer.go lib/storage: consistently use atomic.* types instead of atomic.* function calls on ordinary types 2024-02-24 00:15:26 +02:00
block_stream_writer_timing_test.go lib/storage: consistently use atomic.* types instead of atomic.* function calls on ordinary types 2024-02-24 00:15:26 +02:00
block_test.go all: consistently use %w instead of %s in when error is passed to fmt.Errorf() 2023-10-25 21:24:03 +02:00
dedup.go docs: mention staleNaN handling during deduplication 2024-01-11 11:53:58 +01:00
dedup_test.go lib/storage: follow-up after 7c0ae3a86a 2022-12-08 18:16:57 -08:00
dedup_timing_test.go lib/storage: follow-up after 7c0ae3a86a 2022-12-08 18:16:57 -08:00
filenames.go lib/storage: consistently use OS-independent separator in file paths 2023-03-25 14:33:58 -07:00
index_db.go vmstorage: auto calculate maxUniqueTimeseries based on resources (#6961) 2024-10-18 14:00:14 +02:00
index_db_test.go lib/storage: restore ability to put empty metric ID list into tagFiltersToMetricIDsCache (#7064) 2024-09-20 17:21:53 +02:00
index_db_timing_test.go lib/storage: restore ability to put empty metric ID list into tagFiltersToMetricIDsCache (#7064) 2024-09-20 17:21:53 +02:00
inmemory_part.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:46:12 -07:00
inmemory_part_test.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:46:12 -07:00
inmemory_part_timing_test.go lib/storage: use deterministic random generator in tests 2023-01-23 20:10:32 -08:00
merge.go all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:20:37 +02:00
merge_test.go lib/storage: consistently use atomic.* types instead of atomic.* function calls on ordinary types 2024-02-24 00:15:26 +02:00
merge_timing_test.go lib/storage: consistently use atomic.* types instead of atomic.* function calls on ordinary types 2024-02-24 00:15:26 +02:00
metaindex_row.go all: subsitute ioutil.ReadAll with io.ReadAll 2022-08-22 00:16:37 +03:00
metaindex_row_test.go lib/storage: correctly use maxBlockSize in various checks 2020-09-24 18:12:56 +03:00
metric_name.go lib/storage: change default value for maxLabelValueLen to 1024 (#6313) 2024-05-22 21:53:53 +02:00
metric_name_test.go app/vminsert: add support for data ingestion via other vminsert nodes 2021-05-08 19:52:57 +03:00
part.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:46:12 -07:00
part_header.go lib/storage: add ability to use downsampling for the given series filter (#733) 2024-03-30 04:12:23 +02:00
part_search.go lib/storage: optimize partSearch.searchBHS() for common case when the TSID for the current block header is bigger or equal to the current tsid 2022-12-19 10:28:03 -08:00
part_search_test.go lib/{storage,mergeset}: convert InitFromFilePart to MustInitFromFilePart 2023-04-14 15:46:12 -07:00
part_search_timing_test.go all: consistently use %w instead of %s in when error is passed to fmt.Errorf() 2023-10-25 21:24:03 +02:00
partition.go Revert "refactor(vmstorage): Refactor the code to reduce the time complexity of MustAddRows and improve readability (#6629)" 2024-07-25 14:32:09 +02:00
partition_search.go all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:20:37 +02:00
partition_search_test.go lib/storage: do not keep rawRows buffer across flush() calls 2024-02-22 17:22:26 +02:00
partition_test.go lib/storage: stop exposing vm_merge_need_free_disk_space metric 2023-09-25 16:52:39 +02:00
raw_block.go all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00
raw_row.go lib/storage: consistently use atomic.* types instead of atomic.* function calls on ordinary types 2024-02-24 00:15:26 +02:00
search.go lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit 2024-05-14 01:23:54 +02:00
search_test.go Fix inconsistent error handling in Storage.AddRows() (#6583) 2024-07-17 12:07:14 +02:00
storage.go (app|lib)/vmstorage: do not increment vm_rows_ignored_total on NaNs (#7166) 2024-10-02 12:37:27 +02:00
storage_test.go lib/storage: Fix flaky test: TestStorageRotateIndexDB (#7267) 2024-10-23 11:48:21 +02:00
storage_timing_test.go Fix inconsistent error handling in Storage.AddRows() (#6583) 2024-07-17 12:07:14 +02:00
table.go Revert "refactor(vmstorage): Refactor the code to reduce the time complexity of MustAddRows and improve readability (#6629)" 2024-07-25 14:32:09 +02:00
table_search.go all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:20:37 +02:00
table_search_test.go lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage() 2023-07-13 17:13:24 -07:00
table_search_timing_test.go lib/storage: replace the remaining atomic.* functions with atomic.* types for the sake of consistency 2024-02-24 00:53:30 +02:00
table_test.go flagutil: Make .Msecs private (#4906) 2023-09-03 10:33:37 +02:00
table_timing_test.go lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage() 2023-07-13 17:13:24 -07:00
tag_filters.go lib/logstorage: work-in-progress 2024-05-24 03:06:55 +02:00
tag_filters_test.go lib/logstorage: work-in-progress 2024-05-24 03:06:55 +02:00
tag_filters_timing_test.go all: fix golangci-lint(revive) warnings after 0c0ed61ce7 2024-04-02 23:16:29 +03:00
time.go app,lib: fix typos in comments (#3804) 2023-02-13 13:27:13 +01:00
time_test.go all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00
tsid.go all: remove the remaining mentions of cluster version 2019-11-21 23:18:22 +02:00
tsid_test.go all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00