Nikolay
1f91f22b5f
app/vmselect: reduce lock contention for heavy aggregation requests ( #5119 )
...
reduce lock contention for heavy aggregation requests
previously lock contetion may happen on machine with big number of CPU due to enabled string interning. sync.Map was a choke point for all aggregation requests.
Now instead of interning, new string is created. It may increase CPU and memory usage for some cases.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5087
2023-10-10 13:45:20 +02:00
Aliaksandr Valialkin
140e7b6b74
all: replace atomic.Value with atomic.Pointer[T]
...
This eliminates the need in .(*T) casting for results obtained from Load()
Leave atomic.Value for map, since atomic.Pointer[map[...]...] makes double pointer to map,
because map is already a pointer type.
2023-07-19 17:42:06 -07:00
Aliaksandr Valialkin
30425ca81a
lib/fs: rename WriteFileAtomically to MustWriteAtomic
...
Callers of this function log the returned error and exit.
So let's just log the error with the given filepath and the call stack
inside the function itself and then exit. This simplifies the code
at callers' place while leaves the same level of debuggability in case of errors.
2023-04-13 22:41:15 -07:00
Aliaksandr Valialkin
e8225d7d6b
app/vmselect/promql: prevent from cannot unmarshal timeseries from rollupResultCache panic after the upgrade to v1.89.0
...
The issue has been introduced in 0af9e2b693
2023-03-12 19:09:39 -07:00
Aliaksandr Valialkin
30ed33fae0
app/vmselect/promql: pre-allocate memory for values to be merged in mergeTimeseries()
...
This should reduce the number of memory re-allocations
2023-01-09 22:51:17 -08:00
Aliaksandr Valialkin
645c24dc5f
app/vmselect/promql: consistently intern series names obtained from marshalMetricNameSorted
...
This reduces memory allocations when the returned series names are used as map keys later
2023-01-09 22:45:40 -08:00
Aliaksandr Valialkin
2f3ddd4884
app/vmselect/promql: avoid memory allocations and copying from source timeseries to the returned result at timeseriesToResult()
2023-01-09 22:38:59 -08:00
Aliaksandr Valialkin
c4265322f4
lib/fs: add canOverwrite arg to WriteFileAtomically when it is allowed to overwrite the file atomically if it already exists
2022-10-26 01:07:34 +03:00
Aliaksandr Valialkin
a10647e0bf
app/vmselect/promql: expose missing metric vm_cache_size_max_bytes{type="promql/rollupResult"}
2022-10-23 12:13:47 +03:00
Aliaksandr Valialkin
9f94c295ab
all: use os.{Read|Write}File instead of ioutil.{Read|Write}File
...
The ioutil.{Read|Write}File is deprecated since Go1.16 -
see https://tip.golang.org/doc/go1.16#ioutil
VictoriaMetrics needs at least Go1.18, so it is safe to remove ioutil usage
from source code.
This is a follow-up for 02ca2342ab
2022-08-21 23:52:35 +03:00
Aliaksandr Valialkin
3e2dd85f7d
all: readability improvements for query traces
...
- show dates in human-readable format, e.g. 2022-05-07, instead of a numeric value
- limit the maximum length of queries and filters shown in trace messages
2022-06-30 18:20:33 +03:00
Aliaksandr Valialkin
a963b2a0aa
all: show timeRange in traces in human-readable format instead of timestamps in milliseconds
2022-06-27 13:45:51 +03:00
Aliaksandr Valialkin
12ac255dae
lib/querytracer: make it easier to use by passing trace context message to New and NewChild
...
The context message can be extended by calling Donef.
If there is no need to extend the message, then just call Done.
2022-06-08 21:06:52 +03:00
Aliaksandr Valialkin
41958ed5dd
all: add initial support for query tracing
...
See https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#query-tracing
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1403
2022-06-01 02:29:23 +03:00
Aliaksandr Valialkin
61c7f6beae
app/vmselect/promql: allow calling InitRollupResultCache+StopRollupResultCache multiple times during tests
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2406
2022-04-11 12:34:43 +03:00
Aliaksandr Valialkin
62b46007c5
lib/workingsetcache: reduce the default cache rotation period from hour to 20 minutes
...
This should reduce memory usage under high time series churn rate
2022-02-23 13:41:45 +02:00
Aliaksandr Valialkin
ff15a752c1
app/vmselect: accept optional extra_filters[] query args for all the supported Prometheus querying APIs
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1863
2021-12-06 17:07:09 +02:00
Aliaksandr Valialkin
bec18e4fe9
app/vmselect: add -search.disableAutoCacheReset command-line option for disabling automatic cache reset when a sample with old timestamp outside -search.cacheTimestampOffset is inserted
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1570
2021-08-27 17:15:31 +03:00
Aliaksandr Valialkin
aa81039b42
app/vmselect: log the metric which trigger rollup result cache reset
...
This should help finding the source of stale metrics
2021-03-25 21:31:39 +02:00
Aliaksandr Valialkin
cef010d5f7
app/vmselect/promql: increment key prefix for faster reset for rollup result cache
2021-03-22 11:59:07 +02:00
Nikolay
186c078fac
adds enforced tag filters into cache key ( #1095 )
2021-02-27 00:15:53 +02:00
Aliaksandr Valialkin
5ebfc275e6
app/victoria-metrics: automatically reset response cache when samples with too timestamps older than now - search.cacheTimestampOffset are ingested
2020-12-14 13:08:28 +02:00
Aliaksandr Valialkin
f0c678c41b
app/vmselect: do not adjust start and end query args passed to /api/v1/query_range when -search.disableCache command-line flag is set
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/563
2020-07-30 23:14:37 +03:00
Aliaksandr Valialkin
d5dddb0953
all: use %w instead of %s for wrapping errors in fmt.Errorf
...
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
2020-06-30 23:05:11 +03:00
Aliaksandr Valialkin
0afd48d2ee
lib: extract common code for returning fast unix timestamp into lib/fasttime
2020-05-14 23:02:07 +03:00
Aliaksandr Valialkin
4e4f57b121
lib/metricsql: move it to a separate repository - github.com/VictoriaMetrics/metrics
2020-04-28 15:28:22 +03:00
Aliaksandr Valialkin
2f55cabaa4
app/vmselect/promql: log when rollupResult cache is cleared
2020-02-21 20:07:01 +02:00
Aliaksandr Valialkin
ec9bf39b5b
app/vmselect: add -search.cacheTimestampOffset command-line flag
...
This flag can be used for removing gaps on graphs if the difference between the current time
and the timestamps from the ingested data exceeds 5 minutes.
This is the case when the time between data sources and VictoriaMetrics is improperly synchronized.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/312
2020-02-21 13:58:06 +02:00
Aliaksandr Valialkin
680080887d
all: consistently log durations in seconds with millisecond precision
...
This should improve logs readability
2020-01-22 18:28:27 +02:00
Aliaksandr Valialkin
3e09d38f29
app/vmselect/promql: fix results caching for multi-arg rollup functions such as quantile_over_time
...
Previosly only a single arg was taken into account, so caching didn't work properly for multi-arg rollup funcs.
2020-01-03 20:49:08 +02:00
Aliaksandr Valialkin
1925ee038d
Rename lib/promql to lib/metricsql and apply small fixes
2019-12-25 22:03:59 +02:00
Mike Poindexter
bec62e4e43
Split Extended PromQL parsing to a separate library
2019-12-25 22:03:51 +02:00
Aliaksandr Valialkin
f4a38f7fb1
app/vmselect/promql: fix panic on -search.disableCache
...
Reset the cache if it is disabled instead of stopping, since it is stopped on graceful shutdown.
2019-08-21 17:11:52 +03:00
Aliaksandr Valialkin
9ecb994671
app/vmselect/promql: store compressed results in the cache
...
This should increase rollup results cache capacity.
2019-08-14 02:29:45 +03:00
Aliaksandr Valialkin
09fc6e22e5
all: use workingsetcache instead of fastcache
...
This should reduce the amount of RAM required for processing time series
with non-zero churn rate.
The previous cache behavior can be restored with `-cache.oldBehavior` command-line flag.
2019-08-13 21:39:34 +03:00
Aliaksandr Valialkin
a336bb4e22
app/vmselect/promql: reduce RAM usage for aggregates over big number of time series
...
Calculate incremental aggregates for `aggr(metric_selector)` function instead of
keeping all the time series matching the given `metric_selector` in memory.
2019-07-10 13:04:39 +03:00
Aliaksandr Valialkin
1fe6d784d8
all: consistency renaming: bytesSize -> sizeBytes
2019-07-10 00:47:36 +03:00
Aliaksandr Valialkin
53ea90865d
app/vmselect/promql: add -search.disableCache flag for disabling response caching
...
This may be useful for data back-filling, when the response caching
could interfere badly with newly added data points with timestamps
in the past.
2019-06-04 17:30:45 +03:00
Aliaksandr Valialkin
1836c415e6
all: open-sourcing single-node version
2019-05-23 00:18:06 +03:00
