Commit graph

21 commits

Author SHA1 Message Date
Aliaksandr Valialkin
6d84b1beef
lib/filestream: do not measure read / write duration from / to in-memory buffers
Measuring read / write duration from / to in-memory buffers has little sense,
since it will be always fast. It is better to measure read / write duration from / to
real files at vm_filestream_write_duration_seconds_total and vm_filestream_read_duration_seconds_total metrics.
This also reduces overhead on time.Now() and Histogram.UpdateDuration() calls
per each filestream.Reader.Read() and filestream.Writer.Write() call when the data is read / written from / to in-memory buffers.

This is a follow-up for 2f63dec2e3
2024-01-23 14:52:22 +02:00
Aliaksandr Valialkin
388d020b7c
app/vmselect/promql: follow-up for ce4f26db02
- Document the bugfix at docs/CHANGELOG.md
- Filter out NaN values before sorting as suggested at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5509#discussion_r1447369218
- Revert unrelated changes in lib/filestream and lib/fs
- Use simpler test at app/vmselect/promql/exec_test.go

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5509
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5506
2024-01-16 02:55:08 +02:00
Zongyang
ce4f26db02
FIX bottomk doesn't return any data when there are no time range overlap between timeseries (#5509)
* FIX sort order in bottomk

* Add lessWithNaNsReversed for bottomk

* Add ut for TopK

* Move lt from loop

* FIX lint

* FIX lint

* FIX lint

* Mod log format

---------

Co-authored-by: xiaozongyang <xiaozngyang@kanyun.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-01-16 02:19:36 +02:00
Aliaksandr Valialkin
75dd7b30ba
lib/filestream: add -filestream.disableFadvise syscall for unconditional disabling of fadvise syscall
This may be needed in rare cases when performing backups on systems with big number of CPU cores
and big value passed to -concurrency command-line flag.

See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5120
2023-10-04 16:19:46 +02:00
Aliaksandr Valialkin
9183a439c7
lib/filestream: change Create() to MustCreate()
Callers of this function log the returned error and exit.
It is better logging the error together with the path to the filename
and call stack directly inside the function. This simplifies
the code at callers' side without reducing the level of debuggability
2023-04-14 15:12:48 -07:00
Aliaksandr Valialkin
5eb163a08a
lib/filestream: transform Open() -> MustOpen()
Callers of this function log the returned error and exit.
Let's log the error with the path to the filename and call stack
inside the function. This simplifies the code at callers' side
without reducing the level of debuggability.
2023-04-14 15:03:42 -07:00
Aliaksandr Valialkin
bd6de6406a
lib/fs: improve error logging inside MustWriteData
Log the path to file on errors inside MustWriteData().
This improves debuggability of errors, which may occur inside MustWriteData().
2023-04-14 14:32:45 -07:00
Aliaksandr Valialkin
b14d96618c
all: follow-up after 34634ec357
- Use windows.FlushFileBuffers() instead of windows.Fsync() at streamTracker.adviseDontNeed()
  for consistency with implementations for other architectures.
- Use filepath.Base() instead of filepath.Split(), since the dir part isn't used.
  This simplifies the code a bit.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70
2023-03-25 11:57:39 -07:00
Nikolay
34634ec357
lib/fs: adds memory map for windows (#3988)
This is a follow-up for 43b24164ef

* lib/fs: adds memory map for windows
it should improve performance for file reading

* lib/storage: replace '/' with os specific separator
it must fix an errors for windows

* lib/fs: mention windows fsync support

* lib/filestream: adds fdatasync for windows writes

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70
2023-03-25 11:43:19 -07:00
Aliaksandr Valialkin
93764746c2
lib/filestream: remove logging redundant path values in a single error message 2022-12-03 22:01:51 -08:00
Aliaksandr Valialkin
2f63dec2e3
lib/fs: add vm_filestream_read_duration_seconds_total and vm_filestream_write_duration_seconds_total metrics
These metrics help determining persistent disk saturation with `rate(vm_filestream_read_duration_seconds_total) > 0.9`
2021-12-02 10:30:42 +02:00
Nikolay
ddc8022702
fixes solaris build (#1345) 2021-05-31 09:21:23 +03:00
Aliaksandr Valialkin
7673839228 lib/{fs,filestream}: small consistency-related updates after cc90a548b1 2020-09-29 00:42:43 +03:00
Nikolay Khramchikhin
cc90a548b1
added openbsd implementations (#790)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/785

removed fadvise for openbsd, added freespace implemenation for openbsd
2020-09-29 00:29:04 +03:00
Aliaksandr Valialkin
f00e0e0103 lib/persistentqueue: sync data to file inside filestream.Writer.MustFlush 2020-09-19 12:51:41 +03:00
Aliaksandr Valialkin
d5dddb0953 all: use %w instead of %s for wrapping errors in fmt.Errorf
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
2020-06-30 23:05:11 +03:00
Aliaksandr Valialkin
04762344c6 app/vmagent: initial implementation for vmagent 2020-02-23 13:36:03 +02:00
Aliaksandr Valialkin
1e46961d68 app/{vmbackup,vmrestore}: add vmbackup and vmrestore tools for creating backups on s3 or gcs from instant snapshots
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/203
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/38
2019-11-08 21:21:07 +02:00
Aliaksandr Valialkin
9196c085a7 all: port to FreeBSD on GOARCH=amd64 2019-08-28 01:19:23 +03:00
Aliaksandr Valialkin
e88a03323a all: initial stubs for Windows support; see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70 2019-06-20 20:07:10 +03:00
Aliaksandr Valialkin
1836c415e6 all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00