Commit graph

6096 commits

Author SHA1 Message Date
Aliaksandr Valialkin
0bbb281c3d
lib/filestream: transform Open() -> MustOpen()
Callers of this function log the returned error and exit.
Let's log the error with the path to the filename and call stack
inside the function. This simplifies the code at callers' side
without reducing the level of debuggability.
2023-04-14 15:04:54 -07:00
Aliaksandr Valialkin
ee8be138b9
lib/fs: improve error logging at ReaderAt.MustReadAt()
- Add 'BUG:' prefix to error messages related to programming errors aka bugs.
- Consistently log the path to the file in all the messages in order to improve debuggability.
2023-04-14 14:52:14 -07:00
Aliaksandr Valialkin
b80d93d4b2
lib/fs: substitute ReadFullData with MustReadData
Callers of ReadFullData() log the error and then exit.
So let's log the error with the path to the filename and the call stack
inside MustReadData(). This simplifies the code at callers' side,
while leaving the debuggability at the same level.
2023-04-14 14:40:58 -07:00
Aliaksandr Valialkin
36559dfec2
lib/fs: improve error logging inside MustWriteData
Log the path to file on errors inside MustWriteData().
This improves debuggability of errors, which may occur inside MustWriteData().
2023-04-14 14:33:45 -07:00
Aliaksandr Valialkin
d544bfd73e
vendor: update github.com/VictoriaMetrics/metricsql from v0.56.1 to v0.56.2
This fixes panic when the duration in the query contains `M` suffix.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4120
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3589
2023-04-14 14:06:47 -07:00
Aliaksandr Valialkin
67df75484f
lib/{mergeset,storage}: remove isInMerge flag from parts only when they werent removed yet from the list of active parts
This prevents from possible panic during access to pw.p when it is set to nil at partWrapper.decRef() called inside swapSrcWithDstParts()
2023-04-14 00:16:18 -07:00
Aliaksandr Valialkin
2fd9fdacce
docs/vmctl.md: run make docs-sync after 2a5b9ff782 2023-04-13 23:54:50 -07:00
Aliaksandr Valialkin
e6d10106f7
docs/CHANGELOG.md: move the bugfix description into the correct place
This is a follow-up for 2a5b9ff782

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4092
2023-04-13 23:52:01 -07:00
Dmytro Kozlov
10351a9dbb
app/vmctl: fix performance degradation, add flag to disable backoff policy (#4097)
* app/vmctl: change api for getting metric names

* app/vmctl: fix tests

* app/vmctl: add flag to enable backoff policy, fix test, performance improvements

* app/vmctl: use one http client

* app/vmctl: made linter happy

* app/vmctl: updated documentation and CHANGELOG.md

* app/vmctl: cleanup

* app/vmctl: rename flag

* app/vmctl: cleanup

* app/vmctl: fix comments

* app/vmctl: fix metrics parser problem, improve tests
2023-04-13 23:51:11 -07:00
Aliaksandr Valialkin
7fb2b14ca0
docs/CHANGELOG.md: run at least 4 background mergers on systems with less than 4 CPU cores
This reduces the probability of sudden spike in the number of small parts when all the background mergers
are busy with big merges.
2023-04-13 23:37:05 -07:00
Aliaksandr Valialkin
8846ce5f1d
lib/{mergeset,storage}: make sure that getFlushToDiskDeadline() takes into account only in-memory parts 2023-04-13 23:17:24 -07:00
Aliaksandr Valialkin
f75b1b7a53
lib/fs: add Must prefix to CopyDirectory and CopyFile functions
Callers of these functions log the returned error and then exit.
Let's log the error with the call stack inside the function itself.
This simplifies the code at callers' side, while leaving the same
level of debuggability in case of errors.
2023-04-13 23:04:37 -07:00
Aliaksandr Valialkin
75b74aa837
lib/fs: rename SymlinkRelative to MustSymlinkRelative
Callers of this function log the returned error and then exit.
Let's log the error with the call stack inside the function itself.
This simplifies the code at callers' side, while leaving the same
level of debuggability in case of errors.
2023-04-13 22:53:11 -07:00
Aliaksandr Valialkin
624b86d065
lib/fs: rename HardLinkFiles to MustHardLinkFiles
Callers of this function log the returned error and then exit.
Let's log the error with the call stack inside the function itself.
This simplifies the code at callers' side, while leaving the same
level of debuggability in case of errors.
2023-04-13 22:49:38 -07:00
Aliaksandr Valialkin
c4638553a3
lib/fs: rename WriteFileAtomically to MustWriteAtomic
Callers of this function log the returned error and exit.
So let's just log the error with the given filepath and the call stack
inside the function itself and then exit. This simplifies the code
at callers' place while leaves the same level of debuggability in case of errors.
2023-04-13 22:43:30 -07:00
Aliaksandr Valialkin
aac3dccfd1
lib/fs: replace MkdirAllIfNotExist->MustMkdirIfNotExist and MkdirAllFailIfExist->MustMkdirFailIfExist
Callers of these functions log the returned error and then exit. The returned error already contains the path
to directory, which was failed to be created. So let's just log the error together with the call stack
inside these functions. This leaves the debuggability of the returned error at the same level
while allows simplifying the code at callers' side.

While at it, properly use MustMkdirFailIfExist instead of MustMkdirIfNotExist inside inmemoryPart.MustStoreToDisk().
It is expected that the inmemoryPart.MustStoreToDick() must fail if there is already a directory under the given path.
2023-04-13 22:22:08 -07:00
Aliaksandr Valialkin
b4c330ea2b
lib/fs: rename MustWriteFileAndSync to MustWriteSync in order to improve readability a bit
This is a follow-up for 2a8395be05
2023-04-13 22:20:31 -07:00
Aliaksandr Valialkin
cdee2cfc5c
lib/{mergeset,storage}: remove unused path field from blockStreamWriter
This is a follow-up after 42bba64aa7
2023-04-13 22:20:02 -07:00
Aliaksandr Valialkin
1cda542c48
lib/fs: replace WriteFileAndSync with MustWriteAndSync
When WriteFileAndSync fails, then the caller eventually logs the error message
and exits. The error message returned by WriteFileAndSync already contains the path
to the file, which couldn't be created. This information alongside the call stack
is enough for debugging the issue. So just use log.Panicf("FATAL: ...") inside MustWriteAndSync().
This simplifies error handling at caller side a bit.
2023-04-13 22:17:34 -07:00
Aliaksandr Valialkin
eb7df27e20
lib/{mergeset,storage}: properly fsync part directory listing after writing in-memory part to disk
This is a follow-up after 42bba64aa7

Previously the part directory listing was fsync'ed implicitly inside partHeader.WriteMetadata()
by calling fs.WriteFileAtomically(). Now it must be fsync'ed explicitly.

There is no need in fsync'ing the parent directory, since it is fsync'ed by the caller
when updating parts.json file.
2023-04-13 21:21:46 -07:00
Aliaksandr Valialkin
13d2350e6a
lib/{mergeset,storage}: explicitly fsync the created part directory listing
Previously the created part directory listing was fsynced implicitly
when storing metadata.json file in it.

Also remove superflouous fsync for part directory listing,
which was called at blockStreamWriter.MustClose().
After that the metadata.json file is created, so an additional fsync
for the directory contents is needed.
2023-04-13 21:07:33 -07:00
Aliaksandr Valialkin
cf53ce83a0
app/vmstorage: deprecate -bigMergeConcurrency command-line flag
Improperly configured -bigMergeConcurrency command-line flag usually leads to uncontrolled
growth of unmerged parts, which, in turn, increases CPU usage and query durations.

So it is better deprecating this flag. In rare cases -smallMergeConcurrency command-line flag
can be used instead for controlling the concurrency of background merges.
2023-04-13 20:42:22 -07:00
Aliaksandr Valialkin
e73dd1df2d
lib/{fs,persistentqueue}: use filepath.Join() instead of concatenating path parts with /
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4014
2023-04-13 20:14:07 -07:00
Aliaksandr Valialkin
e95e401e4d
app/vmbackupmanager: sync with enterprise-single-node branch after 41a54c775891c87e3d5ed59ff0769c869dd2fe71 2023-04-13 19:38:28 -07:00
Alexander Marshalov
e0a1b62569
Added extra information to docs about total output in stream aggregation. (#4130)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-04-13 19:37:05 -07:00
Zakhar Bessarab
217eea6e15
lib/backup/actions: store metadata(creation and completion time) in backup files (#4117)
This makes it easier to understand exact point in time which is included in this backup.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-04-13 19:20:34 -07:00
Aliaksandr Valialkin
61839d4807
vendor: update github.com/valyala/gozstd from v1.19.0 to v1.19.1 2023-04-10 11:30:26 -07:00
Aliaksandr Valialkin
6e8c4ee95c
app/vmbackupmanager/README.md: sync with docs/vmbackupmanager.md after 4b2cc1b32c 2023-04-10 10:52:34 -07:00
Aliaksandr Valialkin
b33dd2e577
docs/Single-server-VictoriaMetrics.md: document automatic switch from graph view to heatmap view for histogram buckets
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3384
2023-04-10 10:52:24 -07:00
Aliaksandr Valialkin
6c7087f65e
docs/Troubleshooting.md: mention that it is recommended to use default command-line flag values for VictoriaMetrics components 2023-04-10 10:52:10 -07:00
Aliaksandr Valialkin
4077b0fcb1
vendor: make vendor-update 2023-04-10 10:21:39 -07:00
Haleygo
7ee32ed06a
fix sort pendingDateMetricsIDs (#4102) 2023-04-10 10:16:36 -07:00
Aliaksandr Valialkin
7f55afd699
deployment: update VictoriaMetrics from v1.89.1 to v1.90.0
See https://docs.victoriametrics.com/CHANGELOG.html#v1900
2023-04-06 19:18:14 -07:00
Aliaksandr Valialkin
b21ac56894
docs/guides: update VictoriaMetrics from v1.89.1 to v1.90.0 2023-04-06 19:18:14 -07:00
Aliaksandr Valialkin
2bbd8754bb
docs/CHANGELOG.md: formatting fix 2023-04-06 19:18:14 -07:00
Aliaksandr Valialkin
64cf8fe492
app/vmctl/terminal: fix builds for GOOS=freebsd and GOOS=openbsd
This is a follow-up for 8da9502df6
2023-04-06 17:09:20 -07:00
Aliaksandr Valialkin
d15618f2ef
docs/CHANGELOG.md: cut v1.90.0 release 2023-04-06 16:16:52 -07:00
Aliaksandr Valialkin
26b361f4c3
app/vmselect/vmui: run make vmui-update after 01fc228fb0 2023-04-06 15:11:54 -07:00
Dmytro Kozlov
8ec5e7f53a
app/vmctl: add multiple filters defined in --vm-native-filter-match flag to discovered metric names (#4063)
* app/vmctl: add multiple filters defined in `--vm-native-filter-match` flag to discovered metric names

* app/vmctl: fix comments

* app/vmctl: move function buildMatchWithFilter to the correct place

* app/vmctl: update CHANGELOG.md

* app/vmctl: fix CI, remove error wrapping

* app/vmctl: fix CI, simplify `Set()`
2023-04-06 15:11:53 -07:00
Yury Molodov
d9876b36b5
vmui: heatmap fixes (#4086)
* fix: correct display of errors for query

* fix: change the logic of histogram detection

* feat: hide empty buckets from the graph

* fix: revert server url
2023-04-06 15:11:26 -07:00
Aliaksandr Valialkin
ff1891397a
docs/CHANGELOG.md: document the bugfix, which remove unneeded logger.Errorf() call during stream aggregation with the enabled deduplication
This is a follow-up for ff72ca14b9
2023-04-06 15:01:25 -07:00
Aliaksandr Valialkin
a241485262
app/vmselect/vmui: run make vmui-update after a1601929ec 2023-04-06 03:20:16 -07:00
Aliaksandr Valialkin
eca03d1f55
docs/CHANGELOG.md: document that VictoriaMetrics for Windows cannot delete snapshots
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70#issuecomment-1491529183
2023-04-06 03:16:20 -07:00
Aliaksandr Valialkin
c43dc072c4
docs/CHANGELOG.md: document v1.79.12 2023-04-06 03:10:12 -07:00
Aliaksandr Valialkin
c92731de93
docs/CHANGELOG.md: document v1.87.5 2023-04-06 00:44:30 -07:00
Timur Bakeyev
f21ea62ae1
Fix cut-n-paste error (#4079)
It seems that VMServiceScrape description was c-n-p from vmselect one into all other resources.
2023-04-06 00:44:29 -07:00
Yury Molodov
2263c412da
fix: correct the description of shortcut keys (#4057) 2023-04-05 22:20:12 -07:00
Zakhar Bessarab
0384b83736
docs: fix example operator spec for vmbackupmanager restore usage (#4074)
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-04-05 22:17:05 -07:00
Yury Molodov
7871ee0e43
vmui: implement heatmap improvements (#4078)
* fix: disabled limits for histogram

* fix: add sorted buckets by upper bound

* refactor: move line chart components to folder

* feat: implement heatmap improvements (https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3384#issuecomment-1484023162)

* app/vmselect/vmui: `make vmui-update`

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-04-05 22:15:23 -07:00
Aliaksandr Valialkin
bf545fcc14
lib/encoding: fix test after 4725549cb2 2023-04-05 21:38:48 -07:00