Aliaksandr Valialkin
e49bf9bc73
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-05-24 16:03:14 +03:00
Aliaksandr Valialkin
39ef1e7a51
lib/storage: do not stop data ingestion on the first error in Storage.AddRows
...
Continue data ingestion for the rest of blocks.
2021-05-24 15:32:47 +03:00
Aliaksandr Valialkin
4b01c9fb2e
lib/storage: limit the number of rows per each block in Storage.AddRows()
...
This should reduce memory usage when ingesting big blocks or rows.
2021-05-24 15:24:07 +03:00
Aliaksandr Valialkin
a4ff4b8e65
lib/storage: allow filling all the rows up to their capacity in rawRowsShard.addRows
...
This should reduce memory usage a bit on data ingestion path
2021-05-24 15:22:59 +03:00
Aliaksandr Valialkin
f54133b200
lib/storage: do not populate MetricID->MetricName cache during data ingestion
...
This cache isn't needed during data ingestion, so there is no need in spending RAM on it.
This reduces RAM usage on data ingestion path by 30%
2021-05-24 03:02:46 +03:00
Aliaksandr Valialkin
ec79abc382
lib/{mergeset,storage}: reduce the number of IFNO log messages like merged ... items across ... blocks in ... seconds
...
Log these messages if the merge takes more than 30 seconds instead of 10 seconds.
2021-05-23 14:03:21 +03:00
Aliaksandr Valialkin
ad73f226ff
app/vmstorage: add ability to limit series cardinality via -storage.maxHourlySeries
and -storage.maxDailySeries
command-line flags
2021-05-20 14:15:19 +03:00
Aliaksandr Valialkin
009e136d88
lib/storage: remove possible data race when logging dropped labels
2021-05-20 02:47:22 +03:00
Aliaksandr Valialkin
d7be2753c0
lib/storage: substitute GetTSDBStatusForDate with GetTSDBStatusWithFiltersForDate with nil tfss
2021-05-13 09:02:33 +03:00
Aliaksandr Valialkin
a22a17dc66
lib/storage: merge getTSDBStatusForDate with getTSDBStatusWithFiltersForDate
...
These functions are non-trivial, while their code has minimal differences.
It is better from maintainability PoV to merge these functions into a single function.
2021-05-12 17:56:53 +03:00
Aliaksandr Valialkin
832651c6c2
app/vmselect: follow up after 8a0678678b
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1168
2021-05-12 17:18:30 +03:00
Nikolay
8a0678678b
Adds tsdb match filters ( #1282 )
...
* init work on filters
* init propose for status filters
* fixes tsdb status
adds test
* fix bug
* removes checks from test
2021-05-12 15:18:45 +03:00
Aliaksandr Valialkin
33f7bacb01
lib/storage: properly apply time range when matching an empty filter
...
It must match all the time series on the given time range.
Previously it was matched to all the time series without the restriction on the given time range.
2021-05-11 01:12:05 +03:00
Aliaksandr Valialkin
75c2c813fc
lib/storage: remove dead code after the commit 3ccf7ea20c
2021-05-08 20:14:11 +03:00
Aliaksandr Valialkin
12d733dd5d
app/vminsert: add support for data ingestion via other vminsert nodes
2021-05-08 19:52:57 +03:00
Aliaksandr Valialkin
76027093ca
lib/storage: use WARNING instead of INFO level for logging dropped labels
2021-05-03 13:56:43 +03:00
Nikolay
477369b62f
adds stalePartsRemover ( #1261 )
...
for new created partitions
2021-05-03 11:34:00 +03:00
Aliaksandr Valialkin
a142390014
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-05-01 10:05:48 +03:00
Aliaksandr Valialkin
58d1e6eeea
lib/storage: log dropped labels if the number of labels in a metric exceeds -maxLabelsPerTimeseries
command-line flag value
...
This should improve debuggability for this case.
2021-05-01 09:27:55 +03:00
Aliaksandr Valialkin
bceb8082f6
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-04-30 10:11:24 +03:00
Aliaksandr Valialkin
87179c6839
lib/{storage,mergeset}: fix unaligned 64-bit atomic operation
panic for 32-bit architectures
...
The panic has been introduced in 56b6b893ce
2021-04-27 16:41:32 +03:00
Aliaksandr Valialkin
56b6b893ce
lib/mergeset: split rows ingestion among multiple shards
...
This improves rows ingestion on systems with many CPU cores by reducing lock contention.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1244
Thanks to @waldoweng for the original idea and draft implementation at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1243
2021-04-27 15:36:34 +03:00
Aliaksandr Valialkin
f89c1f7f49
lib/storage: typo fix in info message when deleting the part outside the configured retention
...
Previously the message was displaying incorrect retention time
2021-04-27 13:32:46 +03:00
Aliaksandr Valialkin
276969500e
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-04-24 01:42:54 +03:00
Aliaksandr Valialkin
bbebdf9ba1
lib/{storage,mergeset}: remove empty directories on startup. Such directories can be left after unclean shutdown on NFS storage
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1142
2021-04-22 13:02:44 +03:00
Aliaksandr Valialkin
030e3a63f2
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-04-21 11:00:07 +03:00
Aliaksandr Valialkin
251747e253
lib/storage: code clarification: remove caching the found metricName in searchMetricName
2021-04-13 10:22:21 +03:00
Aliaksandr Valialkin
660c3c7251
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-04-08 00:54:19 +03:00
Aliaksandr Valialkin
df32d2836c
lib/storage: properly handle big time ranges passed to /api/v1/labels
and /api/v1/label/<labelName>/values
...
It should be faster querying all the labels and/or all the values instead of querying per-day labels/values on time ranges exceeding maxDaysForPerDaySearch
2021-04-07 13:33:46 +03:00
Aliaksandr Valialkin
dc9eafcd02
app/{vminsert,vmagent}: add -sortLabels
command-line option for sorting time series labels before ingesting them in the storage
...
This option can be useful when samples for the same time series are ingested with distinct order of labels.
For example, metric{k1="v1",k2="v2"} and metric{k2="v2",k1="v1"}.
2021-03-31 23:27:58 +03:00
Aliaksandr Valialkin
e1f699bb6c
lib/storage: reduce memory usage when ingesting samples for the same time series with distinct order of labels
2021-03-31 21:24:46 +03:00
Aliaksandr Valialkin
11ded82e60
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-03-29 19:15:52 +03:00
Aliaksandr Valialkin
2601cc0fb0
lib/storage: do not update b.nextIdx if no samples are removed because of retention
2021-03-29 12:00:21 +03:00
Aliaksandr Valialkin
aa81039b42
app/vmselect: log the metric which trigger rollup result cache reset
...
This should help finding the source of stale metrics
2021-03-25 21:31:39 +02:00
Aliaksandr Valialkin
343f444e87
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-03-25 19:15:08 +02:00
Aliaksandr Valialkin
6e855d4b82
lib/storage: tune loopsCountPerMetricNameMatch according to production workload
2021-03-25 13:27:47 +02:00
Aliaksandr Valialkin
3a3d2165f9
lib/storage: do not reload metricName for the same metricID in Search.NextMetricBlock
...
This should speed up Search.NextMetricBlock a bit
2021-03-23 17:56:49 +02:00
Aliaksandr Valialkin
3cfb3a3683
lib/storage: respect the deadline passed to Storage.SearchMetricNames
2021-03-22 23:03:17 +02:00
Aliaksandr Valialkin
8e2afdf568
lib/storage: improve Search.NextMetricBlock performance by using MetricID->MetricName cache
2021-03-22 22:49:18 +02:00
Aliaksandr Valialkin
910092ca4d
lib/storage: tune loopsCountPerMetricNameMatch
2021-03-22 12:53:17 +02:00
Aliaksandr Valialkin
726f6ad804
lib/storage: small code simplification after 6cee5338b2
2021-03-18 15:21:13 +02:00
Aliaksandr Valialkin
6cee5338b2
lib/storage: prevent from infinite loop if {__graphite__="..."}
filter matches a metric name with *
, [
or {
chars
...
The idea has been borrowed from https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1137
2021-03-18 14:53:47 +02:00
Aliaksandr Valialkin
45dabfac1b
lib/storage: faster move heavy filters to the end of list
2021-03-17 15:12:13 +02:00
Aliaksandr Valialkin
16884c20c0
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-03-17 02:05:46 +02:00
Aliaksandr Valialkin
ccfb0ae2d3
lib/storage: limit loops count in order to reduce max CPU usage during filter search
2021-03-17 00:49:26 +02:00
Aliaksandr Valialkin
576a80b3d9
lib/storage: do not modify filterLoopsCount stats with loopsCount stats
...
Such a modification can result in incorrect filter sorting later
2021-03-17 00:49:26 +02:00
Aliaksandr Valialkin
fd86a7dc1d
lib/storage: time series search optimization according to production workload profiling
...
Do not pass filter metric ids to getMetricIDsForTagFilter, since it has been appeared that this slows down
the function by multiple times when it finds big number of metricIDs (tens of millions).
2021-03-16 20:01:43 +02:00
Aliaksandr Valialkin
e36fbfae5b
lib/storage: further tuning for time series search
2021-03-16 18:46:22 +02:00
Aliaksandr Valialkin
7d44cdd8ce
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2021-03-15 22:44:24 +02:00
Aliaksandr Valialkin
dd7e82c34f
app/vmstorage: add -logNewSeries
command-line flag for determining the source of series churn rate
2021-03-15 22:38:50 +02:00