Nikolay
0b87f02602
fixes panic ( #979 )
...
* fixes panic
https://github.com/VictoriaMetrics/helm-charts/issues/89
* add fast-path
* Apply suggestions from code review
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2020-12-24 09:01:40 +02:00
Aliaksandr Valialkin
1a237c6903
all: properly handle CPU limits set on the host system/container
...
This can reduce memory usage on systems with enabled CPU limits.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/946
2020-12-08 21:07:03 +02:00
Aliaksandr Valialkin
7d76fdedcc
app/vmselect: use storage.NewSearchQuery() instead of constructing storage.SearchQuery in-place
...
This should prevent from bugs when AccountID and ProjectID aren't set in storage.SearchQuery.
2020-11-16 18:04:33 +02:00
Aliaksandr Valialkin
2f4421b86c
app/vmselect/prometheus: return __name__
label if match[]
query to /api/v1/labels
matches at least a single time series
2020-11-16 13:54:50 +02:00
Aliaksandr Valialkin
852aed62f7
app/vmselect/prometheus: improve performance for /api/v1/labels
and /api/v1/label/<labelName>/values
on time ranges exceeding one day when match[]
query arg is set
2020-11-16 13:46:51 +02:00
Aliaksandr Valialkin
e969346e3e
app/vmselect/prometheus: fix deadlock in /api/v1/series
on a time range exceeding one day
2020-11-16 13:30:57 +02:00
Aliaksandr Valialkin
eea1be0d5c
app/vmselect/graphite: add /tags/findSeries handler from Graphite Tags API
...
See https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags
2020-11-16 12:52:23 +02:00
Aliaksandr Valialkin
2ac5f00d98
app/vmselect: propagate errors from vmstorage
to response to the client if -search.denyPartialResponse
command-line flag is set
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/891
This commit also adds `"isPartial":{true|false}` field to `/api/v1/*` responses. `"isPartial":true` is set when the response
is based on a partial data because some of vmstorage nodes weren't available during query processing.
2020-11-14 13:20:10 +02:00
Aliaksandr Valialkin
7ceaf4ba8f
all: consistently return text-based HTTP responses with charset=utf-8
...
This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/897
2020-11-13 10:30:21 +02:00
Aliaksandr Valialkin
72011bcc45
app/vmselect: properly handle errors in GetLabelsOnTimeRange and GetLabelValuesOnTimeRange
2020-11-05 01:36:34 +02:00
Aliaksandr Valialkin
c5e6c5f5a6
app/vmselect: optimize querying for /api/v1/labels
and /api/v1/label/<name>/values
when start
and end
args are set
2020-11-05 01:19:29 +02:00
Aliaksandr Valialkin
d884ab13dc
app/vmselect/prometheus: fix golangci-lint warning
2020-10-13 09:36:18 +03:00
Aliaksandr Valialkin
0867dea5fc
app/vmselect: add ability to export data in CSV format via /api/v1/export/csv
2020-10-12 20:08:08 +03:00
Aliaksandr Valialkin
c47972d843
app/vmselect/prometheus: check for errors returned from bufferedwriter.Write
...
This makes `make errcheck` happy
2020-09-29 11:36:54 +03:00
Aliaksandr Valialkin
6d8c23fdbd
app/{vminsert,vmselect}: skip accountID and projectID when marshaling/unmarshaling MetricName in /api/v1/export/native and /api/v1/import/native
...
This is needed in order to be able to migrate native data from/to single-node VictoriaMetrics
2020-09-28 00:58:58 +03:00
Aliaksandr Valialkin
db14f22fc0
app/vmselect: stop /api/v1/export/*
execution if client disconnects
2020-09-27 23:52:42 +03:00
Aliaksandr Valialkin
aadbd014ff
all: add native format for data export and import
...
The data can be exported via [/api/v1/export/native](https://victoriametrics.github.io/#how-to-export-data-in-native-format ) handler
and imported via [/api/v1/import/native](https://victoriametrics.github.io/#how-to-import-data-in-native-format ) handler.
2020-09-27 17:36:38 +03:00
Aliaksandr Valialkin
8627365b48
app/vmselect/prometheus: code cleanup after 3ba507000c
2020-09-23 13:04:51 +03:00
Aliaksandr Valialkin
1fce79518a
app/vmselect/prometheus: return timestamps from /api/v1/query
, which match the time
query arg
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/720
2020-09-23 12:59:23 +03:00
Aliaksandr Valialkin
1f0c0b0f6b
app/vmselect: typo fix in -search.maxStalenessInterval description
2020-09-15 14:25:34 +03:00
Aliaksandr Valialkin
d3ad0d365e
app/vmselect: move Deadline from netstorage to searchutils
...
This removes dependency on netstorage from searchutils.
2020-09-11 13:39:13 +03:00
Aliaksandr Valialkin
f307e6f432
app/vmselect: initial implementation of Graphite Metrics API
...
See https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api
2020-09-11 00:30:20 +03:00
Aliaksandr Valialkin
804304c365
app/vmselect: add missing deletion for temporary files on partial responses when -search.denyPartialResponse=true
2020-09-04 02:23:12 +03:00
Aliaksandr Valialkin
6aab2f4989
all: allow using KB
, MB
, GB
, KiB
, MiB
and GiB
suffixes in command-line flag values related to byte sizes or byte rates
2020-08-16 17:08:28 +03:00
Aliaksandr Valialkin
62b6e54622
app/vmselect: reduce memory usage when exporting time series with big number of samples via /api/v1/export
if max_rows_per_line
is set to non-zero value
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685
2020-08-10 20:57:43 +03:00
Aliaksandr Valialkin
a2aa3a60eb
app/vmselect: show X-Forwarded-For
contents on /api/v1/status/active_queries
page
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/659
2020-07-31 20:01:09 +03:00
Aliaksandr Valialkin
0f6f0d30d3
app/vmselect: show query origin (aka remote_addr or client address) on the /api/v1/status/active_queries
page for every query
2020-07-28 15:14:40 +03:00
Aliaksandr Valialkin
20d0c41ac5
app/vmselect/prometheus: support d
, w
and y
suffixes for durations passed to step
in /api/v1/query_range
like Prometheus does
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/641
2020-07-22 16:27:27 +03:00
Aliaksandr Valialkin
5bb4fe1ba4
app/vmselect: take into account the time spent in wait queue before query execution as time spent on the query
2020-07-21 19:00:00 +03:00
Aliaksandr Valialkin
141e84b5a4
app/vmselect/prometheus: do not return time series with empty list of datapoints from /api/v1/query_range
...
This matches Prometheus behaviour.
This should fix https://github.com/jacksontj/promxy/issues/329
2020-07-20 15:30:13 +03:00
Aliaksandr Valialkin
0e7b2008b2
app/vmselect/prometheus: do not adjust last points in time series with timestamps exceeding the current time
...
Such timestamps usually mean that the query contains `offset`.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/625
2020-07-14 12:56:21 +03:00
Aliaksandr Valialkin
3898cc0285
app/vmselect/prometheus: minimize the diff for the change 1033dc7e2a
over 619b0a25c9
2020-07-13 21:41:17 +03:00
faceair
bf39e67ade
fix empty response template ( #617 )
2020-07-13 21:41:15 +03:00
Aliaksandr Valialkin
82871fb7a5
app/vmselect/prometheus: small fixes on top of 8bb762124a
2020-07-05 18:17:53 +03:00
faceair
17f175ff5a
fix adjust last points avoid influence earlier value ( #606 )
2020-07-05 18:17:52 +03:00
Aliaksandr Valialkin
d962568e93
all: use %w instead of %s for wrapping errors in fmt.Errorf
...
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
2020-06-30 23:33:46 +03:00
Aliaksandr Valialkin
cc2225cc49
app/vmselect: fix the error after 936f35920a
2020-06-12 22:00:45 +03:00
Aliaksandr Valialkin
936f35920a
app/vmselect/prometheus: allow returning partial response from /api/v1/export
if -search.denyPartialResponse=false
...
This makes `/api/v1/export` behaviour consistent with other `/api/v1/*` handlers.
2020-06-12 21:11:48 +03:00
Aliaksandr Valialkin
3845420a8f
lib: extract common code for returning fast unix timestamp into lib/fasttime
2020-05-14 23:06:50 +03:00
Aliaksandr Valialkin
9ed4951ec8
lib/metricsql: move it to a separate repository - github.com/VictoriaMetrics/metrics
2020-04-28 15:30:06 +03:00
Aliaksandr Valialkin
f9526809e5
app/vmselect: add /api/v1/status/tsdb
page with useful stats for locating root cause for high cardinality issues
...
See https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/425
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/268
2020-04-22 22:03:23 +03:00
Aliaksandr Valialkin
b59f1f1504
app/vmselect: add -search.minStalenessInterval
command-line flag for removing gaps on graphs built from time series with irregular duration between samples
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/426
2020-04-20 19:42:41 +03:00
Aliaksandr Valialkin
603d4c9217
app/vmselect: merge -search.maxLookback
and -search.maxStalenessInterval
flags, since it has been appeared they have identical purpose :(
...
Leave both flags for backwards compatibility reasons.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/426
2020-04-20 19:28:28 +03:00
Aliaksandr Valialkin
b9b5641c2f
app/vmselect: properly apply -search.maxLookback
to queries sent to /api/v1/query
2020-04-17 12:31:18 +03:00
Aliaksandr Valialkin
c6cbc0bd19
app/vmselect/prometheus: allow passing relative time to start
, end
and time
args of /api/v1/*
queries
2020-03-29 21:56:52 +03:00
Aliaksandr Valialkin
cb8696699a
app/vmselect/prometheus: code simplification: (d.Seconds()/1e3) -> d.Milliseconds()
2020-03-29 21:50:35 +03:00
Aliaksandr Valialkin
f6410ff2bf
app/vmselect: add optional max_rows_per_line
query arg to /api/v1/export
...
This arg allows limiting the number of data points that may be exported on a single line.
2020-03-10 21:47:43 +02:00
Aliaksandr Valialkin
d23df53ba2
app/vmselect/prometheus: do not add __name__!=
filter when searching for all the matching metric names via /api/v1/label/__name__/values
with non-empty label filter
...
This should reduce query time.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/343
2020-02-28 23:36:38 +02:00
Aliaksandr Valialkin
8fa1cd24d8
app/vmselect/prometheus: properly pass filter for labelName=__name__
in labelValuesWithMatches
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/343
2020-02-28 12:17:30 +02:00
Aliaksandr Valialkin
6e0013ca39
app/vmselect/prometheus: typo fix in -latencyOffset
description
2020-02-12 14:00:38 +02:00
Aliaksandr Valialkin
ccd3aa4f15
app/vmselect: take into account the time the requests wait in the queue if -search.maxConcurrentRequests
is exceeded
...
This will prevent from excess CPU usage for timed out queries.
2020-02-04 16:20:48 +02:00
Aliaksandr Valialkin
7cde594696
all: do not clash flag description with back-quoted flag types
...
See https://golang.org/pkg/flag/#PrintDefaults for more details.
2020-02-04 15:56:01 +02:00
Aliaksandr Valialkin
e1a264173a
app/vmselect: mention the original query and time range in error messages
...
This should simplify debugging invalid or heavy queries.
2020-01-22 17:34:35 +02:00
Aliaksandr Valialkin
e127173984
app/vmselect: mention command-line flag, which could be used for adjusting query timeouts, in timeout errors
2020-01-22 15:53:42 +02:00
Aliaksandr Valialkin
f3b9f8b823
app/vmselect/prometheus: increase default value -maxExportDuration
to 30 days, since 10 minutes beat users exporting bit amounts of data
2020-01-22 15:53:41 +02:00
Aliaksandr Valialkin
4b16b7fd11
all: mention command-line flags used for limiting the incoming request size in error messages
...
This should improve error logs usability.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/287
2020-01-16 13:06:43 +02:00
Aliaksandr Valialkin
453d71d082
Rename lib/promql to lib/metricsql and apply small fixes
2019-12-25 22:09:09 +02:00
Aliaksandr Valialkin
cc8a1bae0e
app/vmselect: add -search.maxExportDuration
command-line flag for limiting /api/v1/export
duration
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/275
2019-12-20 11:37:18 +02:00
Aliaksandr Valialkin
6a185b7809
app/vmselect: add ability to pass match[]
, start
and end
to /api/v1/labels
...
This makes the `/api/v1/labels` handler consistent with already existing functionality for `/api/v1/label/.../values`.
See https://github.com/prometheus/prometheus/issues/6178 for more details.
2019-12-15 00:20:43 +02:00
Aliaksandr Valialkin
f79b61e2a1
app/vmselect/promql: return matrix instead of vector on subqueries to /api/v1/query
like Prometheus does
2019-12-11 00:57:54 +02:00
Aliaksandr Valialkin
5d2ff573aa
app/vmselect/promql: allow negative offsets
...
Updates https://github.com/prometheus/prometheus/issues/6282
2019-12-11 00:57:51 +02:00
Aliaksandr Valialkin
def9ccd360
app/vmselect/prometheus: consistently apply nocache
arg to /api/v1/query
the same way ast to /api/v1/query_range
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/241
2019-11-26 22:55:50 +02:00
Aliaksandr Valialkin
e0ac068112
app/vmselect/prometheus: fix content-type for /api/v1/export
responses
...
The correct Content-Type should be `application/stream+json` instead of `application/json`
Thanks to Joshua Ryder for pointing to this.
2019-11-26 17:44:27 +02:00
Aliaksandr Valialkin
8811bec14e
app/vmselect/prometheus: reduce default value for -search.latencyOffset
from 60s to 30s
...
30 seconds should be enough for almost all the cases
2019-11-25 16:33:36 +02:00
Aliaksandr Valialkin
17d08c1fe0
app/vmselect: adjust end
arg instead of adjusting start
arg if start > end
...
`start` arg has higher chances to be set properly comparing to `end` arg,
so it is expected that the `end` arg could be adjusted if it was set incorrectly.
2019-11-22 16:12:53 +02:00
Aliaksandr Valialkin
5ae47e8940
app/vmselect/prometheus: properly adjust too big time time
on /api/v1/query
...
Too big `time` must be adjusted to `now()-queryOffset`.
2019-11-19 00:42:07 +02:00
Oleg Kovalov
74ba42d111
fix misspelled words ( #229 )
2019-11-12 00:18:24 +02:00
Aliaksandr Valialkin
4e6bf6f538
app/vmselect: add -search.latencyOffset
flag for tuning the time after data collection when data points become visible in query results
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/218
2019-10-28 12:32:36 +02:00
Aliaksandr Valialkin
5b01b7fb01
all: add support for GOARCH=386 and fix all the issues related to 32-bit architectures such as GOARCH=arm
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
2019-10-17 18:27:49 +03:00
Aliaksandr Valialkin
99786c2864
app/vmselect/prometheus: add -search.maxLookback
command-line flag for overriding dynamic calculations for max lookback interval
...
This flag is similar to `-search.lookback-delta` if set. The max lookback interval is determined dynamically
from interval between datapoints for each input time series if the flag isn't set.
The interval can be overriden on per-query basis by passing `max_lookback=<duration>` query arg to `/api/v1/query` and `/api/v1/query_range`.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209
2019-10-15 21:37:17 +03:00
Aliaksandr Valialkin
a7d5d611fe
app/vmselect/prometheus: do not adjust start
and end
args in /api/v1/query_range
if nocache=1
arg is set
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/171
2019-09-04 13:10:17 +03:00
Aliaksandr Valialkin
db1de4277c
app/vmselect/promql: remove NaNs from /api/v1/query_range
output like Prometheus does
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/153
2019-08-20 23:01:59 +03:00
Aliaksandr Valialkin
53c8f56436
app/vmselect: allow passing match[]
, start
and time
to /api/v1/label/<label_name>/values
...
`/api/v1/label/<label_name>/values?match[]=q` emulates emulates `label_values(q, <label_name>)`
call in Grafana templating.
2019-08-04 23:07:00 +03:00
Aliaksandr Valialkin
880b1d80b1
app/vmselect: optimize /api/v1/series
by skipping storage data
...
Fetch and process only time series metainfo.
2019-08-04 23:00:46 +03:00
Aliaksandr Valialkin
7f5afae1e3
app/vmselect/prometheus: prevent from fetching and scanning all the data on /api/v1/searies
call by default
2019-08-04 19:42:45 +03:00
Aliaksandr Valialkin
6875fb411a
app/vmselect/promql: parallelize incremental aggregation to multiple CPU cores
...
This may reduce response times for aggregation over big number of time series
with small step between output data points.
2019-07-12 15:53:12 +03:00
Aliaksandr Valialkin
4a8e6f47fe
app/vmselect/prometheus: set start
arg in /api/v1/series
to the minimum allowed time by default as Prometheus does
...
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/91
2019-07-11 17:11:37 +03:00
Aliaksandr Valialkin
3313cdf816
app/vmselect/prometheus: convert negative times to 0, since they arent supported by the storage
2019-07-11 17:11:35 +03:00
Aliaksandr Valialkin
d6ec95693d
app/vmselect/prometheus: typo fix
2019-07-07 23:34:04 +03:00
Aliaksandr Valialkin
36636c1f6f
app/vmselect/prometheus: handle minTime
and maxTime
values that may be set by Promxy or Prometheus client
...
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/88
2019-07-07 21:53:52 +03:00
Aliaksandr Valialkin
512a627855
app/vmselect/prometheus: update adjustLastPoints function
...
- Do not overwrite last points by the previous NaNs, since this may result in empty time series.
- Overwrite the last 2 points instead of 3. This should be enough in most cases.
2019-07-04 09:30:56 +03:00
Aliaksandr Valialkin
a3abed80ff
app/vmselect: do not return empty time series in /api/v1/query
result
2019-07-01 17:16:26 +03:00
Aliaksandr Valialkin
c3c60bee45
app/vmselect: add -search.denyPartialResponse
flag for disabling partial responses if some of vmstorage nodes are unavailable
...
Also accept `deny_partial_response` query arg in Prometheus API handlers. If it is set to true,
then return error if some of vmstorage nodes are unavailable.
2019-06-30 01:27:07 +03:00
Aliaksandr Valialkin
391bc8bf38
app/vmselect: fix 32bit arm build
...
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/83
2019-06-27 19:37:17 +03:00
Aliaksandr Valialkin
a3a53647ba
app/vmselect/prometheus: return better error messages on missing args to /api/v1/*
2019-06-20 14:07:44 +03:00
Aliaksandr Valialkin
e40224d5de
lib/flagutil: add NewArray helper func
2019-06-18 10:44:09 +03:00
Aliaksandr Valialkin
75a0acf72d
app/vmselect: add /api/v1/labels/count
handler for quick detection of labels with the maximum number of distinct values
2019-06-10 19:54:55 +03:00
Aliaksandr Valialkin
11979e4d85
app/vmselect/prometheus: report about incorrect time or duration instead of silently using the default value
...
This should prevent from incorrect usage of the querying API.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/52
2019-06-06 22:17:15 +03:00
Aliaksandr Valialkin
628708ad76
app/vmselect: consume resultsCh data in exportHandler if writeResponseFunc failed to consume it
2019-05-24 14:54:54 +03:00
Aliaksandr Valialkin
24578b4bb1
all: open-sourcing cluster version
2019-05-23 00:25:38 +03:00
Aliaksandr Valialkin
1836c415e6
all: open-sourcing single-node version
2019-05-23 00:18:06 +03:00