github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-01 14:47:38 +00:00
Code Issues Projects Releases Packages Wiki Activity
5772 commits 119 branches 748 tags 355 MiB
Commit graph

72 commits

Author SHA1 Message Date
Zakhar Bessarab
c9547f4fba
lib/{fs,mergeset,storage}: skip .must-remove. dirs when creating snapshot (#3858) (#3867) 2023-02-24 12:44:35 -08:00
Scott Kevill
63653b53d6
lib/fs: use unix.Statfs() / unix.Statvfs() when using a path (#3663) 2023-01-17 21:22:02 -08:00
Aliaksandr Valialkin
81400c80f0
lib/fs: remove logging redundant path values in a single error message 2022-12-03 22:00:43 -08:00
Aliaksandr Valialkin
ecb71a7221
lib/fs: add canOverwrite arg to WriteFileAtomically when it is allowed to overwrite the file atomically if it already exists 2022-10-26 01:08:35 +03:00
Aliaksandr Valialkin
51f2e473f5
lib/storage: skip blocks outside the configured retention during search 
Blocks outside the configured retention are eventually deleted during background merge.
But such blocks may reside in the storage for long time until background merge.
Previously VictoriaMetrics could spend additional CPU time on processing such blocks
during search queries. Now these blocks are skipped.
 2022-10-24 02:56:13 +03:00
Aliaksandr Valialkin
5b488a339d
lib/mergeset: atomically remove part dirs 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038
 2022-09-13 15:56:33 +03:00
Aliaksandr Valialkin
fe52378f45
lib/storage: substitute remaining calls to fs.MustRemoveAll with fs.MustRemoveDirAtomic 2022-09-13 15:49:25 +03:00
Aliaksandr Valialkin
daa42e4f79
lib/storage: atomically remove partitions, which went outside the configured retention 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038
 2022-09-13 13:37:59 +03:00
Aliaksandr Valialkin
ff7188b6a5
lib/storage: atomically remove snapshot directories 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038
 2022-09-13 13:25:48 +03:00
Aliaksandr Valialkin
1905618d10
all: subsitute ioutil.ReadAll with io.ReadAll 
ioutil.ReadAll is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil
VictoriaMetrics requires at least Go1.18, so it is OK to switch from ioutil.ReadAll to io.ReadAll.

This is a follow-up for 02ca2342ab
 2022-08-22 00:16:04 +03:00
Aliaksandr Valialkin
88e0fe9469
all: use os.ReadDir instead of ioutil.ReadDir 
The ioutil.ReadDir is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil
VictoriaMetrics requires at least Go1.18, so it is time to switch from io.ReadDir to os.ReadDir

This is a follow-up for 02ca2342ab
 2022-08-22 00:04:09 +03:00
Aliaksandr Valialkin
06f6de6d47
all: use os.{Read|Write}File instead of ioutil.{Read|Write}File 
The ioutil.{Read|Write}File is deprecated since Go1.16 -
see https://tip.golang.org/doc/go1.16#ioutil

VictoriaMetrics needs at least Go1.18, so it is safe to remove ioutil usage
from source code.

This is a follow-up for 02ca2342ab
 2022-08-21 23:55:20 +03:00
Aliaksandr Valialkin
d40441947a
app: allow specifying http and https urls in the following command-line flags 
* -promscrape.config
* -relabelConfig
* -remoteWrite.relabelConfig
* -remoteWrite.urlRelabelConfig
 2021-12-03 00:11:47 +02:00
Aliaksandr Valialkin
daaea1eb2c
app/vmauth: follow-up for 13368bed18 
* Document the ability to specify http or https urls in `-auth.config` at docs/CHANGELOG.md
* Move the ReadFileOrHTTP to lib/fs, so it can be re-used in other places where a file
  should be read from the given path. For example, in `-promscrape.config` at `vmagent`.
 2021-12-02 23:34:15 +02:00
Aliaksandr Valialkin
5d255846ac all: add go:build lines for Go1.17 
See https://tip.golang.org/doc/go1.17#gofmt for more details
 2021-07-26 15:50:46 +03:00
Nikolay
3d89c01d07 fixes solaris build (#1345) 2021-06-04 11:56:06 +03:00
Aliaksandr Valialkin
745eda9e87 lib/fs: do not pass done callback to tryRemoveAll() func 
This improves code readability a bit.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1313
 2021-05-24 05:00:53 +03:00
Aliaksandr Valialkin
89e1a45cdb lib/fs: concurrently remove up to 1024 blocked NFS directories 
Previously the blocked directories were removed sequentially by a single goroutine.
This can be not enough for highly loaded VictoriaMetrics that accepts millions of sample per second,
when big number of LSM parts are created and removed at high rate.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1313
 2021-05-21 17:58:08 +03:00
Aliaksandr Valialkin
23355ca34c lib/fs: wait for a while before giving up on NFS file removal if the removal queue is full 
This should reduce the probability of the panic on a highly loaded VictoriaMetrics
accepting millions of samples per second.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1313
 2021-05-21 17:21:35 +03:00
Aliaksandr Valialkin
ab8008d6d7 lib/{storage,mergeset}: remove empty directories on startup. Such directories can be left after unclean shutdown on NFS storage 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1142
 2021-04-22 13:03:29 +03:00
Aliaksandr Valialkin
e03233f441 lib/fs: reduce the frequency of failed to remove directory ... due to NFS lock log warnings 
Log `failed to remove directory ... due to NFS lock` warning only if the directory cannot be removed in one second.
 2021-03-18 13:23:43 +02:00
Aliaksandr Valialkin
7f1302688f lib/fs: follow-up after f3a03c4164 2021-02-27 01:09:37 +02:00
Nikolay
d88fa5ebe4 Adds windows build (#1040) 
* fixes windows compilation,
adds signal impl for windows,
adds free space usage for windows,
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1036

NOTE victoria metrics database still CANNOT work under windows system,
only vmagent is supported.
To completly port victoria metrics, you have to fix issues with separators,
parsing and posix file removall

* rollback separator

* Adds windows setInformation api,
it must behave like unix, need to test it.
changes procutil

* check for invlaid param

* Fixes posix delete semantic

* refactored a bit

* fixes openbsd build

* removed windows api call

* Fixes code after windows add

* Update lib/procutil/signal_windows.go

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2021-02-27 01:06:22 +02:00
Aliaksandr Valialkin
c1b8729bd8 lib/fs: properly handle stale NFS file handle error during file deletion 
This error can appear when -storageDataPath points to NFS volume and the given file has been already removed.
 2021-02-26 23:24:46 +02:00
Aliaksandr Valialkin
148422bcba lib/storage: disable composite index usage when querying old data 2021-02-10 14:57:58 +02:00
Aliaksandr Valialkin
082eabf51e lib/fs: remove the code for tracking whether the given memory region is in page cache 
This code didn't give performance gains under production workload, so let's remove it in order to simplify the code.
 2021-02-09 16:51:11 +02:00
Aliaksandr Valialkin
588090765c lib/fs: properly initialize cleaner for pageCache bitmaps 
Previously it wasnt working because the timer was fired only once
 2021-01-27 00:40:39 +02:00
Aliaksandr Valialkin
ca8919e8e1 lib/storage: wait for pending transactions before closing and dropping the partition 
This deflakes `make test-full-386` test
 2020-12-25 11:46:47 +02:00
Aliaksandr Valialkin
f4fd917e4f lib/fs: replace fs.OpenReaderAt with fs.MustOpenReaderAt 
All the callers for fs.OpenReaderAt expect that the file will be opened.
So it is better to log fatal error inside fs.MustOpenReaderAt instead of leaving this to the caller.
 2020-11-23 09:57:30 +02:00
Aliaksandr Valialkin
97b836a6f4 lib/fs: fix GOOS=openbsd build by adding fadviseSequentialRead implementation. 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/785
 2020-10-05 23:32:28 +03:00
Aliaksandr Valialkin
81cdf2fa14 lib/{fs,filestream}: small consistency-related updates after cc90a548b1 2020-09-29 00:43:20 +03:00
Nikolay Khramchikhin
658a05ef0f added openbsd implementations (#790) 
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/785

removed fadvise for openbsd, added freespace implemenation for openbsd
 2020-09-29 00:43:19 +03:00
Aliaksandr Valialkin
fd9f1463df lib/fs: use WARN instead of ERROR log level for the message when NFS diretory removal temporarily fails 
this is expected condition, so it is better to use WARN log level for it
 2020-08-09 12:07:35 +03:00
Aliaksandr Valialkin
f789e0fa44 lib/fs: export vm_nfs_pending_dirs_to_remove metric for monitoring the number of pending directories that couldn't be removed due to NFS lock 2020-08-06 15:31:50 +03:00
Aliaksandr Valialkin
77bb0e6595 lib/fs: clarify description for -fs.disableMmap command-line flag 2020-07-06 14:28:57 +03:00
Aliaksandr Valialkin
d962568e93 all: use %w instead of %s for wrapping errors in fmt.Errorf 
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
 2020-06-30 23:33:46 +03:00
Aliaksandr Valialkin
fd7a3d880e lib/fs: go fmt 2020-06-23 23:03:08 +03:00
Aliaksandr Valialkin
08edb90814 lib/fs: fall back to cgo copy for copying the last 4KB of mmaped data 
This probably should fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/581
 2020-06-23 22:55:56 +03:00
Aliaksandr Valialkin
521c657f8d lib/fs: an attempt to fix SIGBUS error by rounding mmap`ed region to multiple of 4KB pages 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/581
 2020-06-23 13:40:20 +03:00
Aliaksandr Valialkin
ba1f764b29 lib/fs: optimize queries that read recent samples for big number of time series 
Use standard copy() func instead of mmap-aware copy func for reading recently touched mmap-ed data.
This improves read performance by up to 4x.
 2020-06-05 19:10:22 +03:00
Aliaksandr Valialkin
2358d9e41d lib/fs: add a benchmark for ReaderAt.MustReadAt 2020-06-05 19:10:21 +03:00
Aliaksandr Valialkin
3d0a0b3785 lib/fs: optimize MustGetFreeSpace performance by caching the results for up to 2 seconds 2020-06-04 13:14:04 +03:00
Aliaksandr Valialkin
c9ab6dc532 lib/fs: do not use mmap for 32-bit arches by default, since they cannot map files bigger than 4GB in RAM 2020-05-12 20:21:39 +03:00
Aliaksandr Valialkin
e933cbac16 lib/storage: postpone reading data from blocks during search 
This eliminates the need for storing block data into temporary files on a single-node VictoriaMetrics
during heavy queries, which touch big number of time series over long time ranges.

This improves single-node VM performance on heavy queries by up to 2x.
 2020-04-27 08:44:01 +03:00
Aliaksandr Valialkin
7a045125cc lib/fs: typo fix: read blocks bigger than 8KB via pread() call instead of using mmap 2020-02-25 18:04:06 +02:00
Aliaksandr Valialkin
7cde594696 all: do not clash flag description with back-quoted flag types 
See https://golang.org/pkg/flag/#PrintDefaults for more details.
 2020-02-04 15:56:01 +02:00
Aliaksandr Valialkin
9b25a2fb67 lib/fs: remove unused readerAt interface 2020-01-31 15:13:00 +02:00
Aliaksandr Valialkin
cb5c39ee70 lib/fs: optimize small reads for ReaderAt.MustReadAt by reading from memory-mapped space instead of reading from file descriptor 
This should improve performance when reading many small blocks.
 2020-01-30 15:16:16 +02:00
Aliaksandr Valialkin
da19fffa08 all: rename ReadAt* to MustReadAt* in order to dont clash with io.ReaderAt 2020-01-30 15:16:16 +02:00
Aliaksandr Valialkin
f22c9dbb0f lib/fs: typo fix in fadvise_unix.go 2019-12-24 21:00:04 +02:00