VictoriaMetrics/lib
rtm0 602bedf362
testing: allow disabling fsync to make tests run faster (#6871)
### Describe Your Changes

fsync() ensures that the data is written to disk. In production this is
needed for data durability. However, during the development, when the
unit tests are run, this level of durability is not needed. Therefore
fsync() can be disabled which will makes test runs two times faster.

The disabling is done by setting the `DISABLE_FSYNC_FOR_TESTING`
environment variable. The valid values for this variable are the same as
the values of the arg of `go doc strconv.ParseBool`:

```
1, t, T, TRUE, true, True, 0, f, F, FALSE, false, False.
```

Any other value means `false`.

The variable is set for all test build targets. Compare running times:

Build Target | DISABLE_FSYNC_FOR_TESTING=0 | DISABLE_FSYNC_FOR_TESTING=1
----------------- | ------------------------------------------------ |
-------------------------------------------------
make test | 1m5s  | 0m22s
make test-race | 3m1s | 1m42s
make test-pure | 1m7s | 0m20s
make test-full | 1m21s | 0m32s
make test-full-386 | 1m42s | 0m36s

When running tests for a given package, fsync can be disabled as
follows:

```shell
DISABLE_FSYNC_FOR_TESTING=1 go test ./lib/storage
```

Disabling fsync() is intended for testing purposes only and the name of
the variables reflects that.

What could also have been done but haven't:

- lib/filestream/filestream.go: `Writer.MustFlush()` also uses f.Sync()
but nothing has been done to it, because the Writer.MustFlush() is not
used anywhere in the VM codebase. A side question: what is the general
policy for the unused code?
- lib/filestream/filestream.go: Writer.Write() calls `adviceDontNeed()`
which calls unix.Fdatasync(). Disabling it could potentially improve
running time, but running tests with this code disabled has shown
otherwise.

### Checklist

The following checks are **mandatory**:

- [ x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Fetishev <wwctrsrx@gmail.com>

(cherry picked from commit 334cd92a6c)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-30 11:18:21 +02:00
..
appmetrics all: add -metrics.exposeMetadata command-line flag, which can be used for adding TYPE and HELP metadata for metrics exposed at /metrics page 2023-12-19 03:26:02 +02:00
auth lib/auth: add NewTokenPossibleMultitenant() for parsing auth token, which can be multitenant 2023-08-30 14:13:51 +02:00
awsapi lib/awsapi: properly assume role with webIdentity token (#5495) 2023-12-20 19:07:04 +02:00
backup lib/backup/s3remote: add retryer configuration (#6747) 2024-08-07 16:59:23 +02:00
blockcache all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
bloomfilter lib: consistently use atomic.* types instead of atomic.* functions 2024-02-24 02:10:04 +02:00
bufferedwriter app/vmselect: move common http functionality from app/vmselect/searchutils to lib/httputils 2023-07-06 17:22:23 -07:00
buildinfo all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00
bytesutil lib/bytesutil: smooth buffer growth rate (#6761) 2024-08-07 16:59:23 +02:00
cgroup lib/cgroup: remove SetGOGC() function 2024-02-05 12:13:08 +02:00
consts app/vminsert: reduce the max packet size, which vminsert can send to vmstorage 2022-04-05 15:39:58 +03:00
decimal lib/slicesutil: add helper functions for setting slice length and extending its capacity 2024-05-12 11:33:49 +02:00
encoding lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit 2024-05-14 01:30:25 +02:00
envflag lib/envflag: do not allow unsupported form for boolean command-line flags in the form -boolFlag value 2023-08-17 13:37:05 +02:00
envtemplate allowed using dashes and dots in environment variables names (#4009) 2023-03-24 17:57:19 -07:00
envutil testing: allow disabling fsync to make tests run faster (#6871) 2024-08-30 11:18:21 +02:00
fastnum lib/fastnum: use unsafe.Slice() instead of deprecated reflect.SliceHeader 2024-02-29 17:17:24 +02:00
fasttime lib: consistently use atomic.* types instead of atomic.* functions 2024-02-24 02:10:04 +02:00
filestream testing: allow disabling fsync to make tests run faster (#6871) 2024-08-30 11:18:21 +02:00
flagutil lib/httpserver: skip basic auth check for additional request paths, which should call httpserver.CheckAuthFlag() 2024-07-16 01:08:41 +02:00
formatutil app/vmbackupmanager: add metrics for better observability (#488) 2022-12-20 14:18:43 -08:00
fs testing: allow disabling fsync to make tests run faster (#6871) 2024-08-30 11:18:21 +02:00
handshake lib/handshake: substitute time.Now() with fastttime.UnixTimestamp(), since profiling shows time.Now() is slow 2024-01-23 18:39:28 +02:00
htmlcomponents lib/htmlcomponents: use relative links for the top page and for favicon.ico 2023-11-13 20:28:17 +01:00
httpserver all: consistently use stringsutil.JSONString() for formatting JSON strings with fmt.* functions instead of using "%q" formatter 2024-07-17 14:01:37 +02:00
httputils lib/httputils: parse URL before creating HTTP transport (#6820) 2024-08-16 11:34:49 +02:00
influxutils app/{vminsert,vmagent}: add healthcheck for influx ingestion endpoints (#6749) 2024-08-05 09:45:32 +02:00
ingestserver Revert c6c5a5a186 and b2765c45d0 2024-07-03 23:57:49 +02:00
leveledbytebufferpool lib/leveledbytebufferpool: do not pool byte slices bigger than 2^18 bytes 2024-06-13 17:02:05 +02:00
logger all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
logstorage app/vlinsert/loki: use easyproto instead for parsing Loki protobuf messages 2024-07-10 03:05:55 +02:00
lrucache all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
memory all: cleanup: remove // +build ... lines, since they are no longer needed after Go1.17, and the minimum supported Go version for VictoriaMetrics source code is Go1.20 2023-11-13 19:15:42 +01:00
mergeset lib/mergeset: fix typos in comments 2024-08-07 16:59:22 +02:00
metricsql all: make fmt via the upcoming Go1.19 2022-07-11 19:23:25 +03:00
netutil lib/promscrape: fixes proxy autorization (#6783) 2024-08-19 22:50:39 +02:00
persistentqueue app/vmagent/remotewrite: follow-up for 87fd400dfc 2024-07-13 02:30:10 +02:00
procutil all: cleanup: remove // +build ... lines, since they are no longer needed after Go1.17, and the minimum supported Go version for VictoriaMetrics source code is Go1.20 2023-11-13 19:15:42 +01:00
promauth lib/promscrape: fixes proxy autorization (#6783) 2024-08-19 22:50:39 +02:00
prompb Revert "Exemplar support (#5982)" 2024-07-03 16:09:18 +02:00
prompbmarshal Revert "Exemplar support (#5982)" 2024-07-03 16:09:18 +02:00
promrelabel lib/promrelabel: follow-up for 8958cecad6 2024-08-27 15:44:07 +02:00
promscrape lib/promrelabel: follow-up for 8958cecad6 2024-08-27 15:44:07 +02:00
promutils all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
protoparser all: consistently use stringsutil.JSONString() for formatting JSON strings with fmt.* functions instead of using "%q" formatter 2024-07-17 14:01:37 +02:00
proxy lib/promscrape: fixes proxy autorization (#6783) 2024-08-19 22:50:39 +02:00
pushmetrics lib/pushmetrics: wait until the background goroutines, which push metrics, are stopped at pushmetrics.Stop() 2024-01-16 21:18:22 +02:00
querytracer make go vet happy 2024-08-19 21:41:44 +02:00
ratelimiter app/vmagent: properly shutdown when -maxIngestionRate limit is reached 2024-04-03 02:41:11 +03:00
regexutil lib/logstorage: work-in-progress 2024-05-25 22:59:21 +02:00
slicesutil lib/slicesutil: add helper functions for setting slice length and extending its capacity 2024-05-12 11:33:49 +02:00
snapshot lib/httputils: parse URL before creating HTTP transport (#6820) 2024-08-16 11:34:49 +02:00
storage lib/storage: properly add previous indexDB metrics (#6890) 2024-08-28 11:17:23 +02:00
streamaggr stream aggregation: do not allow to enable -stream.keepInput and `k… (#6723) 2024-08-13 09:08:27 -04:00
stringsutil all: consistently use stringsutil.JSONString() for formatting JSON strings with fmt.* functions instead of using "%q" formatter 2024-07-17 14:01:37 +02:00
syncwg all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00
tenantmetrics lib/encoding/zstd: switch back from atomic.Pointer to atomic.Value for map[...]... 2023-07-20 21:54:51 -07:00
timerpool lib/timerpool: use timer pool in concurrency limiters 2019-05-28 17:30:10 +03:00
timeutil all: add up to 10% random jitter to the interval between periodic tasks performed by various components 2024-01-22 18:39:16 +02:00
uint64set lib/uint64set: optimize Set.Has() for nil Set - it should be inlined now 2024-07-16 00:00:46 +02:00
vmselectapi lib: consistently use atomic.* types instead of atomic.* functions 2024-02-24 02:10:04 +02:00
workingsetcache lib: consistently use atomic.* types instead of atomic.* functions 2024-02-24 02:10:04 +02:00
writeconcurrencylimiter app/vmagent/remotewrite: clarify the reason behind the default value for -remoteWrite.queues in the same way as the reason for -maxConcurrentInserts is defined at 73f5fb0f0c 2024-03-06 13:57:53 +02:00