Aliaksandr Valialkin
bceb8082f6
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-04-30 10:11:24 +03:00
Aliaksandr Valialkin
87179c6839
lib/{storage,mergeset}: fix unaligned 64-bit atomic operation panic for 32-bit architectures
...
The panic has been introduced in 56b6b893ce
2021-04-27 16:41:32 +03:00
Aliaksandr Valialkin
56b6b893ce
lib/mergeset: split rows ingestion among multiple shards
...
This improves rows ingestion on systems with many CPU cores by reducing lock contention.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1244
Thanks to @waldoweng for the original idea and draft implementation at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1243
2021-04-27 15:36:34 +03:00
Aliaksandr Valialkin
f89c1f7f49
lib/storage: typo fix in info message when deleting the part outside the configured retention
...
Previously the message was displaying incorrect retention time
2021-04-27 13:32:46 +03:00
Aliaksandr Valialkin
276969500e
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-04-24 01:42:54 +03:00
Aliaksandr Valialkin
bbebdf9ba1
lib/{storage,mergeset}: remove empty directories on startup. Such directories can be left after unclean shutdown on NFS storage
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1142
2021-04-22 13:02:44 +03:00
Aliaksandr Valialkin
030e3a63f2
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-04-21 11:00:07 +03:00
Aliaksandr Valialkin
251747e253
lib/storage: code clarification: remove caching the found metricName in searchMetricName
2021-04-13 10:22:21 +03:00
Aliaksandr Valialkin
660c3c7251
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-04-08 00:54:19 +03:00
Aliaksandr Valialkin
df32d2836c
lib/storage: properly handle big time ranges passed to /api/v1/labels and /api/v1/label/<labelName>/values
...
It should be faster querying all the labels and/or all the values instead of querying per-day labels/values on time ranges exceeding maxDaysForPerDaySearch
2021-04-07 13:33:46 +03:00
Aliaksandr Valialkin
dc9eafcd02
app/{vminsert,vmagent}: add -sortLabels command-line option for sorting time series labels before ingesting them in the storage
...
This option can be useful when samples for the same time series are ingested with distinct order of labels.
For example, metric{k1="v1",k2="v2"} and metric{k2="v2",k1="v1"}.
2021-03-31 23:27:58 +03:00
Aliaksandr Valialkin
e1f699bb6c
lib/storage: reduce memory usage when ingesting samples for the same time series with distinct order of labels
2021-03-31 21:24:46 +03:00
Aliaksandr Valialkin
11ded82e60
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-03-29 19:15:52 +03:00
Aliaksandr Valialkin
2601cc0fb0
lib/storage: do not update b.nextIdx if no samples are removed because of retention
2021-03-29 12:00:21 +03:00
Aliaksandr Valialkin
aa81039b42
app/vmselect: log the metric which trigger rollup result cache reset
...
This should help finding the source of stale metrics
2021-03-25 21:31:39 +02:00
Aliaksandr Valialkin
343f444e87
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-03-25 19:15:08 +02:00
Aliaksandr Valialkin
6e855d4b82
lib/storage: tune loopsCountPerMetricNameMatch according to production workload
2021-03-25 13:27:47 +02:00
Aliaksandr Valialkin
3a3d2165f9
lib/storage: do not reload metricName for the same metricID in Search.NextMetricBlock
...
This should speed up Search.NextMetricBlock a bit
2021-03-23 17:56:49 +02:00
Aliaksandr Valialkin
3cfb3a3683
lib/storage: respect the deadline passed to Storage.SearchMetricNames
2021-03-22 23:03:17 +02:00
Aliaksandr Valialkin
8e2afdf568
lib/storage: improve Search.NextMetricBlock performance by using MetricID->MetricName cache
2021-03-22 22:49:18 +02:00
Aliaksandr Valialkin
910092ca4d
lib/storage: tune loopsCountPerMetricNameMatch
2021-03-22 12:53:17 +02:00
Aliaksandr Valialkin
726f6ad804
lib/storage: small code simplification after 6cee5338b2
2021-03-18 15:21:13 +02:00
Aliaksandr Valialkin
6cee5338b2
lib/storage: prevent from infinite loop if {__graphite__="..."} filter matches a metric name with *, [ or { chars
...
The idea has been borrowed from https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1137
2021-03-18 14:53:47 +02:00
Aliaksandr Valialkin
45dabfac1b
lib/storage: faster move heavy filters to the end of list
2021-03-17 15:12:13 +02:00
Aliaksandr Valialkin
16884c20c0
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-03-17 02:05:46 +02:00
Aliaksandr Valialkin
ccfb0ae2d3
lib/storage: limit loops count in order to reduce max CPU usage during filter search
2021-03-17 00:49:26 +02:00
Aliaksandr Valialkin
576a80b3d9
lib/storage: do not modify filterLoopsCount stats with loopsCount stats
...
Such a modification can result in incorrect filter sorting later
2021-03-17 00:49:26 +02:00
Aliaksandr Valialkin
fd86a7dc1d
lib/storage: time series search optimization according to production workload profiling
...
Do not pass filter metric ids to getMetricIDsForTagFilter, since it has been appeared that this slows down
the function by multiple times when it finds big number of metricIDs (tens of millions).
2021-03-16 20:01:43 +02:00
Aliaksandr Valialkin
e36fbfae5b
lib/storage: further tuning for time series search
2021-03-16 18:46:22 +02:00
Aliaksandr Valialkin
7d44cdd8ce
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-03-15 22:44:24 +02:00
Aliaksandr Valialkin
dd7e82c34f
app/vmstorage: add -logNewSeries command-line flag for determining the source of series churn rate
2021-03-15 22:38:50 +02:00
Aliaksandr Valialkin
9c77e34ef9
lib/storage: further tuning for time series selector code
2021-03-15 20:31:34 +02:00
Aliaksandr Valialkin
3ccf7ea20c
lib/storage: tune per-day index search
2021-03-15 13:31:55 +02:00
Aliaksandr Valialkin
f669531506
lib/storage: further tune filters sorting logic
2021-03-12 00:53:04 +02:00
Aliaksandr Valialkin
8582fba4b1
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-03-09 11:46:39 +02:00
John Belmonte
364fdf4a56
spelling fix: adjacent ( #1115 )
2021-03-09 09:18:19 +02:00
Aliaksandr Valialkin
345980f78f
lib/storage: go fmt
2021-03-08 12:03:31 +02:00
Aliaksandr Valialkin
18fe0ff14b
lib/storage: tune loopsCount estimations in getMetricIDsForTagFilterSlow
...
The adjusted estmations give up to 2x lower median response times on 200qps /api/v1/query_range workload
2021-03-07 21:12:35 +02:00
Aliaksandr Valialkin
6197440bb9
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-03-02 21:46:03 +02:00
Aliaksandr Valialkin
4de4da1e2a
lib/storage: typo fix: umarshal -> unmarshal
2021-03-02 20:47:59 +02:00
Aliaksandr Valialkin
0ad887fd4d
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-02-26 23:00:40 +02:00
Aliaksandr Valialkin
2c44178645
lib/storage: consistency renaming: durationsPerDateTagFilterCache -> loopsPerDateTagFilterCache
2021-02-23 15:47:19 +02:00
faceair
15d61c4879
lib/storage: correct tagfilter match cost ( #1079 )
2021-02-22 21:46:56 +02:00
Aliaksandr Valialkin
636c55b526
lib/mergeset: reduce memory usage for inmemoryBlock by using more compact items representation
...
This also should reduce CPU time spent by GC, since inmemoryBlock.items don't have pointers now,
so GC doesn't need visiting them.
2021-02-21 22:06:47 +02:00
Aliaksandr Valialkin
388cdb1980
lib/storage: do not re-calculate stats for heavy tag filters
...
This should reduce the number of slow queries when stats for heavy tag filters was recalculated.
2021-02-21 21:39:01 +02:00
Aliaksandr Valialkin
48656dcc38
lib/{mergeset,storage}: allow merging smaller number of small parts
...
While this may increase CPU and disk IO usage needed for background merge,
this also recudes CPU usage during queries in production. This is because
such queries tend to read recently added data and it is better to have lower number
of parts for such data in order to reduce CPU usage.
This partially reverts ebf8da3730
2021-02-21 21:28:36 +02:00
Aliaksandr Valialkin
cb311bb156
lib/{mergeset,storage}: do not use pools for indexBlock and inmemoryBlock during their caching, since this results in higher memory usage in production without any performance gains
2021-02-21 21:18:59 +02:00
Aliaksandr Valialkin
d5dde7f6b1
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-02-18 19:14:41 +02:00
Aliaksandr Valialkin
f9084611bd
lib/storage: use composite index for a query with a name filter and negative filters
2021-02-18 18:57:23 +02:00
Aliaksandr Valialkin
a537c4f602
lib/storage: properly handle queries containing a filter on metric name plus any number of negative filters and zero non-negative filters
...
Example: `node_cpu_seconds_total{mode!="idle"}`
2021-02-18 18:46:36 +02:00
