Aliaksandr Valialkin
e1a6262302
lib/fs: replace fs.OpenReaderAt with fs.MustOpenReaderAt
...
All the callers for fs.OpenReaderAt expect that the file will be opened.
So it is better to log fatal error inside fs.MustOpenReaderAt instead of leaving this to the caller.
2020-11-23 09:55:41 +02:00
Aliaksandr Valialkin
36b748dfc7
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2020-11-16 21:06:21 +02:00
Aliaksandr Valialkin
266788be14
app/vmselect: use storage.NewSearchQuery() instead of constructing storage.SearchQuery in-place
...
This should prevent from bugs when AccountID and ProjectID aren't set in storage.SearchQuery.
2020-11-16 18:24:00 +02:00
Aliaksandr Valialkin
0208d8c103
lib/storage: add a test for Storage.SearchMetricNames
2020-11-16 13:15:16 +02:00
Aliaksandr Valialkin
465923b181
app/vmselect/graphite: add /tags/findSeries handler from Graphite Tags API
...
See https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags
2020-11-16 12:53:13 +02:00
Aliaksandr Valialkin
48d033a198
app/vminsert: add /tags/tagSeries
and /tags/tagMultiSeries
handlers from Graphite Tags API
...
See https://graphite.readthedocs.io/en/stable/tags.html#adding-series-to-the-tagdb
2020-11-16 02:39:58 +02:00
Aliaksandr Valialkin
4aaee33860
lib/storage: do not show artifically created label for reverse Graphite labels at /api/v1/labels page
2020-11-16 00:44:35 +02:00
immerrr again
51c529a2b6
app/vmstorage: add "/internal/force_flush" endpoint ( #893 )
2020-11-11 14:40:27 +02:00
Aliaksandr Valialkin
56303aee5b
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2020-11-05 02:51:08 +02:00
Aliaksandr Valialkin
368b69b4c4
app/vmselect: properly handle errors in GetLabelsOnTimeRange and GetLabelValuesOnTimeRange
2020-11-05 01:38:38 +02:00
Aliaksandr Valialkin
1cb78ba1a0
lib/storage: remove data race when updating rowsDeleted
2020-11-05 01:12:21 +02:00
Aliaksandr Valialkin
b378cd6ed8
app/vmselect: optimize querying for /api/v1/labels
and /api/v1/label/<name>/values
when start
and end
args are set
2020-11-05 01:01:33 +02:00
Aliaksandr Valialkin
8d8e2ccf5f
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2020-11-04 19:10:41 +02:00
Aliaksandr Valialkin
caeb74f068
app/vmselect: reduce memory usage when query touches big number of time series
2020-11-04 17:04:04 +02:00
Aliaksandr Valialkin
ae91a6883c
lib/{storage,mergeset}: clean cached index blocks and inmemory blocks more aggressively
...
Previously such blocks were cleaned after they weren't accessed during 10 minutes.
Now they are cleaned after one minute of missing access. This should reduce memory usage in general case.
2020-11-04 17:04:04 +02:00
Aliaksandr Valialkin
f3f62ab04e
lib/storage: do not report about the need of free disk space if parts cannot be merged due to too big write amplification
2020-11-03 15:32:02 +02:00
Aliaksandr Valialkin
c87fb9191e
lib/storage: remove unneeded fmt.Sprintf
2020-11-03 14:20:31 +02:00
Aliaksandr Valialkin
1b389674c0
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2020-11-02 21:58:57 +02:00
Aliaksandr Valialkin
fe289331dd
lib/storage: remove obsolete code
2020-11-02 19:11:59 +02:00
Aliaksandr Valialkin
1b112405a8
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2020-11-02 00:47:34 +02:00
Aliaksandr Valialkin
29e4e7f422
lib/storage: drop more samples outside the given retention during background merge
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/17
2020-11-01 23:35:06 +02:00
Aliaksandr Valialkin
d12c4914f0
lib/storage: properly handle the case when key="__name__" is passed to MetricName.AddTag*
2020-11-01 23:35:06 +02:00
Aliaksandr Valialkin
64e2d66014
lib/storage: code cleanup after 5bfd4e6218
2020-11-01 23:35:06 +02:00
Aliaksandr Valialkin
5bfd4e6218
app/vmstorage: support for -retentionPeriod
smaller than one month
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/173
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/17
2020-10-20 14:31:44 +03:00
Aliaksandr Valialkin
8bbc83e85e
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2020-10-17 12:13:56 +03:00
Aliaksandr Valialkin
01987f8c77
lib/storage: small code adjustements after d2960a20e0
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/781
2020-10-17 01:16:54 +03:00
faceair
d2960a20e0
evaluate the execution cost of all tag filters ( #824 )
...
* evaluate the execution cost of all tag filters
* fix suffixes typo
2020-10-17 00:46:55 +03:00
Aliaksandr Valialkin
4dc13754d8
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2020-10-13 18:37:48 +03:00
Aliaksandr Valialkin
68f0e00761
app/vmstorage: add vm_rows_added_to_storage_total
metric, which shows the total number of rows added to storage since app start
2020-10-09 13:35:48 +03:00
Aliaksandr Valialkin
e7f2907138
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2020-10-07 21:40:45 +03:00
Aliaksandr Valialkin
af90b3121c
app/vmstorage: add -finalMergeDelay
command-line flag for configuring the delay before final merge for per-month partitions after no new data is ingested to it
2020-10-07 17:35:44 +03:00
Aliaksandr Valialkin
bdaa9a91f3
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2020-10-01 19:31:26 +03:00
Aliaksandr Valialkin
764dc2499f
lib/storage: code cleanup after 10f2eedee0
...
Remove the code that uses metricIDs caches for the current and the previous hour during metricIDs search,
since this code became unused after implementing per-day inverted index almost a year ago.
While at it, fix a bug, which could prevent from finding time series with names containing dots (aka Graphite-like names
such as `foo.bar.baz`).
2020-10-01 19:06:23 +03:00
Aliaksandr Valialkin
10f2eedee0
lib/storage: imrpove cache effectiveness for time series ids matching the given filters
...
Previously the maximum cache lifetime has been limited by 10 seconds. Now it is extended up to a day.
This should reduce CPU usage in the following cases:
* when querying recently added data with small churn rate for time series
* when querying historical data
2020-10-01 14:38:25 +03:00
Aliaksandr Valialkin
8df5a3c5f6
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2020-09-30 08:55:57 +03:00
Aliaksandr Valialkin
bec9b31b81
lib/storage: allow set values higher than 1 for vm_merge_need_free_disk_space
if there are multiple partitions with deferred merges due to disk space shortage
2020-09-29 22:51:43 +03:00
Aliaksandr Valialkin
a9db81c4ab
app/vmstorage: add metrics for determining whether background merges need additional disk space to complete
...
These metrics are:
* vm_small_merge_need_free_disk_space
* vm_big_merge_need_free_disk_space
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686
2020-09-29 21:48:33 +03:00
Aliaksandr Valialkin
1659135752
lib/storage: fix tests for 32-bit arches such as GOARCH=386 and GOARCH=arm
2020-09-29 13:10:22 +03:00
Aliaksandr Valialkin
38789e4aa0
lib/storage: fix 32-bit builds for GOARH=386 or GOARCH=arm
2020-09-29 12:40:35 +03:00
Aliaksandr Valialkin
95688cbfc5
all: add native format for data export and import
...
The data can be exported via [/api/v1/export/native](https://victoriametrics.github.io/#how-to-export-data-in-native-format ) handler
and imported via [/api/v1/import/native](https://victoriametrics.github.io/#how-to-import-data-in-native-format ) handler.
2020-09-27 19:54:07 +03:00
Aliaksandr Valialkin
82973f8ae7
Revert "lib/storage: remove unused fetchData
arg from BlockRef.MustReadBlock"
...
This reverts commit bab6a15ae0
.
Reason for revert: the `fetchData` arg is used in cluster branch.
Leaving this arg in master branch makes smaller the diff with cluster branch.
2020-09-24 22:44:23 +03:00
Aliaksandr Valialkin
9d3eb3f4b8
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
2020-09-24 20:49:16 +03:00
Aliaksandr Valialkin
bab6a15ae0
lib/storage: remove unused fetchData
arg from BlockRef.MustReadBlock
...
This arg became unused after 23bdc1f107
2020-09-24 20:48:40 +03:00
Aliaksandr Valialkin
2cd48959d4
lib/storage: correctly use maxBlockSize in various checks
...
Previously `maxBlockSize` has been multiplied by 8 in certain checks. This is unnecessary.
2020-09-24 20:17:51 +03:00
Aliaksandr Valialkin
24ca30bf66
lib/storage: correctly use maxBlockSize in various checks
...
Previously `maxBlockSize` has been multiplied by 8 in certain checks. This is unnecessary.
2020-09-24 18:12:56 +03:00
Aliaksandr Valialkin
a70df4bd83
PMM-6401 Initial implementaton for reading data from Prometheus files
2020-09-23 14:26:39 +03:00
Aliaksandr Valialkin
a69234ed18
lib/storage: code prettifying after be5e1222f3
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/781
2020-09-22 00:36:45 +03:00
faceair
be5e1222f3
add filter to getMetricIDs ( #783 )
...
* add getMetricIDs filter
* check nil filter before use
2020-09-22 00:33:43 +03:00
Aliaksandr Valialkin
9739283dad
lib/storage: reduce CPU load for idle VictoriaMetrics by reducing the frequency for the need for background merges
2020-09-21 15:54:11 +03:00
Aliaksandr Valialkin
26115891db
lib/decimal: properly store Inf values
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/752
2020-09-18 19:07:07 +03:00