71 commits

Author	SHA1	Message	Date
Scott Kevill	46b3b76d6d	lib/fs: use unix.Statfs() / unix.Statvfs() when using a path (#3663)	2023-01-17 21:19:26 -08:00
Aliaksandr Valialkin	4f28513b1a	lib/fs: remove logging redundant path values in a single error message	2022-12-03 22:00:20 -08:00
Aliaksandr Valialkin	c4265322f4	lib/fs: add canOverwrite arg to WriteFileAtomically when it is allowed to overwrite the file atomically if it already exists	2022-10-26 01:07:34 +03:00
Aliaksandr Valialkin	dba218a8ce	lib/storage: skip blocks outside the configured retention during search ... Blocks outside the configured retention are eventually deleted during background merge. But such blocks may reside in the storage for long time until background merge. Previously VictoriaMetrics could spend additional CPU time on processing such blocks during search queries. Now these blocks are skipped.	2022-10-24 02:52:44 +03:00
Aliaksandr Valialkin	ce2c07c5a7	lib/mergeset: atomically remove part dirs ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 16:17:38 +03:00
Aliaksandr Valialkin	042a532f70	lib/storage: substitute remaining calls to fs.MustRemoveAll with fs.MustRemoveDirAtomic ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 16:17:38 +03:00
Aliaksandr Valialkin	340ada871d	lib/storage: atomically remove partitions, which went outside the configured retention ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 16:17:37 +03:00
Aliaksandr Valialkin	5f28ca1f42	lib/storage: atomically remove snapshot directories ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3038	2022-09-13 16:17:36 +03:00
Aliaksandr Valialkin	1f89278d88	all: subsitute ioutil.ReadAll with io.ReadAll ... ioutil.ReadAll is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil VictoriaMetrics requires at least Go1.18, so it is OK to switch from ioutil.ReadAll to io.ReadAll. This is a follow-up for 02ca2342ab	2022-08-22 00:16:37 +03:00
Aliaksandr Valialkin	2c3a89339d	all: use os.ReadDir instead of ioutil.ReadDir ... The ioutil.ReadDir is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil VictoriaMetrics requires at least Go1.18, so it is time to switch from io.ReadDir to os.ReadDir This is a follow-up for 02ca2342ab	2022-08-22 00:02:25 +03:00
Aliaksandr Valialkin	9f94c295ab	all: use os.{Read|Write}File instead of ioutil.{Read|Write}File ... The ioutil.{Read|Write}File is deprecated since Go1.16 - see https://tip.golang.org/doc/go1.16#ioutil VictoriaMetrics needs at least Go1.18, so it is safe to remove ioutil usage from source code. This is a follow-up for 02ca2342ab	2022-08-21 23:52:35 +03:00
Aliaksandr Valialkin	06642d97f5	app: allow specifying http and https urls in the following command-line flags ... * -promscrape.config * -relabelConfig * -remoteWrite.relabelConfig * -remoteWrite.urlRelabelConfig	2021-12-03 00:10:02 +02:00
Aliaksandr Valialkin	62b4efb3e7	app/vmauth: follow-up for 13368bed18 ... * Document the ability to specify http or https urls in `-auth.config` at docs/CHANGELOG.md * Move the ReadFileOrHTTP to lib/fs, so it can be re-used in other places where a file should be read from the given path. For example, in `-promscrape.config` at `vmagent`.	2021-12-02 23:32:05 +02:00
Aliaksandr Valialkin	8b7917cd81	all: add go:build lines for Go1.17 ... See https://tip.golang.org/doc/go1.17#gofmt for more details	2021-07-26 15:48:21 +03:00
Nikolay	ddc8022702	fixes solaris build (#1345)	2021-05-31 09:21:23 +03:00
Aliaksandr Valialkin	93d81b486d	lib/fs: do not pass done callback to tryRemoveAll() func ... This improves code readability a bit. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1313	2021-05-24 04:51:57 +03:00
Aliaksandr Valialkin	e05dd475f0	lib/fs: concurrently remove up to 1024 blocked NFS directories ... Previously the blocked directories were removed sequentially by a single goroutine. This can be not enough for highly loaded VictoriaMetrics that accepts millions of sample per second, when big number of LSM parts are created and removed at high rate. Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1313	2021-05-21 17:57:46 +03:00
Aliaksandr Valialkin	8e2985b53d	lib/fs: wait for a while before giving up on NFS file removal if the removal queue is full ... This should reduce the probability of the panic on a highly loaded VictoriaMetrics accepting millions of samples per second. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1313	2021-05-21 17:21:00 +03:00
Aliaksandr Valialkin	bbebdf9ba1	lib/{storage,mergeset}: remove empty directories on startup. Such directories can be left after unclean shutdown on NFS storage ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1142	2021-04-22 13:02:44 +03:00
Aliaksandr Valialkin	4fb049bcba	lib/fs: reduce the frequency of failed to remove directory ... due to NFS lock log warnings ... Log `failed to remove directory ... due to NFS lock` warning only if the directory cannot be removed in one second.	2021-03-18 13:24:46 +02:00
Aliaksandr Valialkin	7cc3d96a41	lib/fs: follow-up after f3a03c4164	2021-02-27 01:01:47 +02:00
Nikolay	f3a03c4164	Adds windows build (#1040) ... * fixes windows compilation, adds signal impl for windows, adds free space usage for windows, https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1036 NOTE victoria metrics database still CANNOT work under windows system, only vmagent is supported. To completly port victoria metrics, you have to fix issues with separators, parsing and posix file removall * rollback separator * Adds windows setInformation api, it must behave like unix, need to test it. changes procutil * check for invlaid param * Fixes posix delete semantic * refactored a bit * fixes openbsd build * removed windows api call * Fixes code after windows add * Update lib/procutil/signal_windows.go Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>	2021-02-27 00:37:07 +02:00
Aliaksandr Valialkin	8683ea85e6	lib/fs: properly handle stale NFS file handle error during file deletion ... This error can appear when -storageDataPath points to NFS volume and the given file has been already removed.	2021-02-26 23:25:14 +02:00
Aliaksandr Valialkin	553016ea99	lib/storage: disable composite index usage when querying old data	2021-02-10 14:57:50 +02:00
Aliaksandr Valialkin	31f6b9c977	lib/fs: remove the code for tracking whether the given memory region is in page cache ... This code didn't give performance gains under production workload, so let's remove it in order to simplify the code.	2021-02-09 16:49:03 +02:00
Aliaksandr Valialkin	3149ac7a7e	lib/fs: properly initialize cleaner for pageCache bitmaps ... Previously it wasnt working because the timer was fired only once	2021-01-27 00:39:26 +02:00
Aliaksandr Valialkin	490c69c64e	lib/storage: wait for pending transactions before closing and dropping the partition ... This deflakes `make test-full-386` test	2020-12-25 11:45:53 +02:00
Aliaksandr Valialkin	f765985947	lib/fs: replace fs.OpenReaderAt with fs.MustOpenReaderAt ... All the callers for fs.OpenReaderAt expect that the file will be opened. So it is better to log fatal error inside fs.MustOpenReaderAt instead of leaving this to the caller.	2020-11-23 09:57:21 +02:00
Aliaksandr Valialkin	caeb74f068	app/vmselect: reduce memory usage when query touches big number of time series	2020-11-04 17:04:04 +02:00
Aliaksandr Valialkin	7673839228	lib/{fs,filestream}: small consistency-related updates after cc90a548b1	2020-09-29 00:42:43 +03:00
Nikolay Khramchikhin	cc90a548b1	added openbsd implementations (#790) ... https://github.com/VictoriaMetrics/VictoriaMetrics/issues/785 removed fadvise for openbsd, added freespace implemenation for openbsd	2020-09-29 00:29:04 +03:00
Aliaksandr Valialkin	9d32fb1d9e	lib/fs: use WARN instead of ERROR log level for the message when NFS diretory removal temporarily fails ... this is expected condition, so it is better to use WARN log level for it	2020-08-09 12:07:30 +03:00
Aliaksandr Valialkin	639b26b40c	lib/fs: export vm_nfs_pending_dirs_to_remove metric for monitoring the number of pending directories that couldn't be removed due to NFS lock	2020-08-06 15:31:34 +03:00
Aliaksandr Valialkin	a23806f486	lib/fs: clarify description for -fs.disableMmap command-line flag	2020-07-06 14:28:34 +03:00
Aliaksandr Valialkin	d5dddb0953	all: use %w instead of %s for wrapping errors in fmt.Errorf ... This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode . See https://blog.golang.org/go1.13-errors for details.	2020-06-30 23:05:11 +03:00
Aliaksandr Valialkin	a560b4788e	lib/fs: go fmt	2020-06-23 23:02:39 +03:00
Aliaksandr Valialkin	8141541e61	lib/fs: fall back to cgo copy for copying the last 4KB of mmaped data ... This probably should fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/581	2020-06-23 22:55:22 +03:00
Aliaksandr Valialkin	c970cb912c	lib/fs: an attempt to fix SIGBUS error by rounding mmap`ed region to multiple of 4KB pages ... Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/581	2020-06-23 13:39:49 +03:00
Aliaksandr Valialkin	d120197676	lib/fs: optimize queries that read recent samples for big number of time series ... Use standard copy() func instead of mmap-aware copy func for reading recently touched mmap-ed data. This improves read performance by up to 4x.	2020-06-05 19:10:04 +03:00
Aliaksandr Valialkin	4cb3af1a36	lib/fs: add a benchmark for ReaderAt.MustReadAt	2020-06-05 19:10:03 +03:00
Aliaksandr Valialkin	3d4008263f	lib/fs: optimize MustGetFreeSpace performance by caching the results for up to 2 seconds	2020-06-04 13:15:47 +03:00
Aliaksandr Valialkin	a6f16dcc11	lib/fs: do not use mmap for 32-bit arches by default, since they cannot map files bigger than 4GB in RAM	2020-05-12 20:22:09 +03:00
Aliaksandr Valialkin	b4afe562c1	lib/storage: postpone reading data from blocks during search ... This eliminates the need for storing block data into temporary files on a single-node VictoriaMetrics during heavy queries, which touch big number of time series over long time ranges. This improves single-node VM performance on heavy queries by up to 2x.	2020-04-27 11:45:24 +03:00
Aliaksandr Valialkin	7ef7c9368e	lib/fs: typo fix: read blocks bigger than 8KB via pread() call instead of using mmap	2020-02-25 18:05:06 +02:00
Aliaksandr Valialkin	42864bb52f	all: do not clash flag description with back-quoted flag types ... See https://golang.org/pkg/flag/#PrintDefaults for more details.	2020-02-04 15:46:52 +02:00
Aliaksandr Valialkin	011a79da85	lib/fs: remove unused readerAt interface	2020-01-31 15:12:43 +02:00
Aliaksandr Valialkin	d70ba7eb37	lib/fs: optimize small reads for ReaderAt.MustReadAt by reading from memory-mapped space instead of reading from file descriptor ... This should improve performance when reading many small blocks.	2020-01-30 15:09:05 +02:00
Aliaksandr Valialkin	ad8af629bb	all: rename ReadAt* to MustReadAt* in order to dont clash with io.ReaderAt	2020-01-30 15:08:58 +02:00
Aliaksandr Valialkin	c18802af59	lib/fs: typo fix in fadvise_unix.go	2019-12-24 20:59:28 +02:00
Aliaksandr Valialkin	5b75984aa9	app/vmselect/netstorage: move MustAdviseSequentialRead to lib/fs	2019-12-23 23:16:11 +02:00