github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
4993 commits 150 branches 748 tags 347 MiB
Commit graph

1928 commits

Author SHA1 Message Date
Yury Molodov
236fc7d739
vmui: fix layout and add server url by default (#1519) 
* fix: change layout for correctly display big query

* fix: set default server from url

* fix: change get default server url
 2021-08-06 12:06:08 +03:00
Aliaksandr Valialkin
c1185363ca app/vmagent: typo fix in the description for -remoteWrite.queues 2021-08-05 10:01:35 +03:00
Roman Khavronenko
7416fdaa8b
vmalert: expose new metrics for tracking number of produced samples during last evaluation (#1518) 
* vmalert: expose new metrics for tracking number of produced samples during last evaluation

Two new metrics were added to track the number of samples produced during the last evaluation:
* vmalert_recording_rules_last_evaluation_samples
* vmalert_alerting_rules_last_evaluation_samples

The gauge type is used to remain consistent with Prometheus metric
`prometheus_rule_group_last_evaluation_samples` which is on the group level.
However, the counter type was considered as well.

Two metrics instead of one are used to make it easier to separate recording and
alerting rules. It is likely, number of samples produced by recording rules is
more important so people will refer to it more frequently.

The expected usage of the new metric is the following:
```
   - alert: RecordingRuleReturnsEmptyResults
        expr: sum(vmalert_recording_rules_last_evaluation_samples) by(recording) < 1
        annotations:
          summary: Recording rule {{$labels.recording}} returns empty results.
            Please verify expression correctness.
```

Addresses https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1494

* vmalert: rename `vmalert_alerts_error` to `vmalert_alerting_rules_error` to remain consistent with recording rules metrics
 2021-08-05 09:59:46 +03:00
Aliaksandr Valialkin
d826352688 app/vmagent: follow-up after fe445f753b 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1491
 2021-08-05 09:52:32 +03:00
Omar Ghader
46e27d60a6 feature: Add multitenant for vmagent (#1505) 
* feature: Add multitenant for vmagent

* Minor fix

* Fix rcs index out of range

* Minor fix

* Fix multi Init

* Fix multi Init

* Fix multi Init

* Add default multi

* Adjust naming

* Add TenantInserted metrics

* Add TenantInserted metrics

* fix: remove unused metrics for vmagent

* fix: remove unused metrics for vmagent

Co-authored-by: mghader <marc.ghader@ubisoft.com>
Co-authored-by: Sebastian YEPES <syepes@gmail.com>
 2021-08-05 09:52:31 +03:00
Aliaksandr Valialkin
e92fde7945 app/vmselect/promql: add present_over_time(m[d]) function, which will be available starting from Prometheus 2.29.0 
See https://github.com/prometheus/prometheus/releases/tag/v2.29.0-rc.0 and https://github.com/prometheus/prometheus/pull/9097
 2021-08-03 16:11:49 +03:00
Qifei Wan
fa9c5c5940
app/vmalert: update config state metrics if config parsed failed (#1507) 2021-08-03 12:55:29 +03:00
wusphinx
c1ed7b77aa
Update TimeSelector.tsx (#1515) 
delete garbled code
 2021-08-03 10:01:01 +03:00
Nikolay
7bbff7fb86
adds /rules and /alerts api for grafana (#1504) 
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2021-08-02 17:28:09 +03:00
Aliaksandr Valialkin
a1911e1330 app/vmselect/netstorage: unpack time series data in mostly local big chunks 
This should improve performance on multi-CPU systems for queries selecting time series with big number of raw samples
 2021-07-30 12:03:17 +03:00
Aliaksandr Valialkin
d05cac6c98 li/storage: re-use the per-day inverted index search code for searching in global index 
This allows removing a big pile of outdated code for global index search.

This may help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1486
 2021-07-30 10:31:37 +03:00
Aliaksandr Valialkin
74ffaa45d9 app/vmselect/netstorage: do not query Go maps with unsafe string keys, since this breaks in Go 1.17 2021-07-30 09:57:53 +03:00
Aliaksandr Valialkin
192dfbfd90 app/vmselect: follow-up for ed95bc9531 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1493
 2021-07-29 09:53:28 +03:00
arnoldyahad
00af4ff5a4 Add case prometheus/rules for grafana 8 (#1502) 2021-07-29 09:53:27 +03:00
assassins
a483044557
Performance optimization (#1481) 
There are redundant steps
 2021-07-28 19:26:20 +03:00
Aliaksandr Valialkin
8ee8660ac4 app/vmselect: follow-up for 626073bca8 
* Rename -search.maxMetricsPointSearch to -search.maxSamplesPerQuery, so it is more consistent with the existing -search.maxSamplesPerSeries
* Move the -search.maxSamplesPerQuery from vmstorage to vmselect, so it could effectively limit the number of raw samples obtained from all the vmstorage nodes
* Document the -search.maxSamplesPerQuery in docs/CHANGELOG.md
 2021-07-28 18:00:23 +03:00
Aliaksandr Valialkin
8b7917cd81 all: add go:build lines for Go1.17 
See https://tip.golang.org/doc/go1.17#gofmt for more details
 2021-07-26 15:48:21 +03:00
Aliaksandr Valialkin
95aff47330 app/vmselect: prevent from possible deadlock when f callback blocks inside RunParallel 2021-07-26 15:47:30 +03:00
Aliaksandr Valialkin
fcaf152480 app/vmselect: make vmui-update after a91d41f12a 2021-07-26 10:31:11 +03:00
Yury Molodov
a91d41f12a
Vmui/query editor (#1472) 
* fix: move request button to server input

* feat: add switch for query autocomplete

* refactor: rename state for popover open

* feat: add detect os by userAgent

* fix: change hotkey to run query for mac

* fix: change detect mac os

* fix: change div to span inside Typography

Co-authored-by: yury <yurymolodov@victoriametrics.com>
 2021-07-23 12:00:44 +03:00
Aliaksandr Valialkin
05672ffc32 app/vmselect/promql: properly handle (a op b) default N if (a op b) returns NaN series 
The result should be a series with `N` values and `a op b` labels. Previously such series has been removed from the result.
 2021-07-16 01:44:58 +03:00
Aliaksandr Valialkin
ed10141ff8 app/vmselect/netstorage: use more scalable algorithm for ditributing the work among among multiple channels on systems with big number of CPU cores 2021-07-16 00:35:23 +03:00
Aliaksandr Valialkin
ca75432e66 app/vmselect: do not track queries with less than 1ms execution time at /api/v1/status/top_queries 
This should improve the readability and usefullness of the /api/v1/status/top_queries when debugging slow queries
or queries that take too much cpu time.
 2021-07-15 16:44:28 +03:00
Aliaksandr Valialkin
f4e81aef7e app/vmselect/netstorage: add -search.maxSamplesPerSeries command-line option for limiting the number of samples a query can process per each series 
This should prevent from out of memory crashes like in https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1067
 2021-07-15 16:03:28 +03:00
Aliaksandr Valialkin
e6ef97a5ee app/vmselect/netstorage: improve scalability of series unpacking on multi-CPU systems 2021-07-15 15:41:58 +03:00
Aliaksandr Valialkin
f81d972581 app/vmui/README.md: typo fix: naviate->navigate 2021-07-15 15:02:04 +03:00
Aliaksandr Valialkin
a925d5a3e1 app/vmselect/promql: duration handling improvements in MetricsQL queries 
- Support durations anywhere in MetricsQL queries. E.g. sum_over_time(m[1h])/1h is equivalent to sum_over_time(m[1h])/3600
- Support durations without suffix. E.g. rate(m[300]) is equivalent to rate(m[5m])
 2021-07-12 17:16:41 +03:00
Aliaksandr Valialkin
bfba4c28a4 app/vmalert: accept Prometheus-like durations in interval config option inside group section 2021-07-12 12:35:17 +03:00
Aliaksandr Valialkin
8c764e88f0 app/vmui: move source code from https://github.com/VictoriaMetrics/vmui to app/vmui 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413
 2021-07-09 17:15:23 +03:00
Aliaksandr Valialkin
c5f0b454f0 app/vmselect: follow-up after aa11ef6d3b 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413
 2021-07-07 17:43:35 +03:00
tony
e9e35a7d6a add vmui for vmselect component (#1431) 
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2021-07-07 17:33:02 +03:00
Aliaksandr Valialkin
ed944313b0 app/{vminsert,vmselect}: export vminsert_request_duration_seconds and vmselect_request_duration_seconds histograms 2021-07-07 13:25:21 +03:00
Aliaksandr Valialkin
766edbc421 lib/httpserver: print full requestURI in httpserver.Errorf 
This should simplify debugging.
 2021-07-07 13:09:40 +03:00
Aliaksandr Valialkin
f3acf065c9 lib/storage: consistency renaming: tagCache -> tagFiltersCache 
This improves code readability
 2021-07-06 11:03:51 +03:00
Aliaksandr Valialkin
f6bb130898 app/{vmselect,vmstorage}: clarify the description for -dedup.minScrapeInterval command-line flag 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1426
 2021-07-02 15:05:57 +03:00
Aliaksandr Valialkin
3af2162085 docs/vmagent.md: mention about docker_sd_config support 2021-06-25 20:52:15 +03:00
Aliaksandr Valialkin
cdfae0117a app/vmselect/promql: return the last timestamp for the max / min value from tmax_over_time() and tmin_over_time() function as most users expect 2021-06-23 14:19:00 +03:00
Nikolay
e307bbb29a
adds http_sd (#1399) 
* adds http_sd

* adds X-Prometheus-Refresh-Interval-Seconds header

* Update lib/promscrape/discovery/http/api.go

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2021-06-22 13:33:37 +03:00
Roman Khavronenko
da8c901fab
vmctl: add more context to flags description in vm-native mode (#1395) 2021-06-18 19:20:01 +03:00
Aliaksandr Valialkin
83a4db813e app/vmselect: log slow requests to all the /api/v1/* handlers if their execution time exceeds -search.logSlowQueryDuration 2021-06-18 19:04:42 +03:00
Aliaksandr Valialkin
570f36b344
app/vmctl: limit JSON line size by 10K samples (#1394) 
This should reduce the maximum memory usage at VictoriaMetrics when importing time series with big number of samples.
 2021-06-18 15:26:47 +03:00
Aliaksandr Valialkin
d8ab409418 docs/{vmgateway,vmbackupmanager}: explicitly mention that these components are a part of an enterprise package 2021-06-17 17:19:49 +03:00
Aliaksandr Valialkin
12a83d25bf app/vmagent/remotewrite: go fmt after 0a796f7c3a 2021-06-17 13:52:06 +03:00
Zongyang
0a796f7c3a
Change default value of '-remoteWrite.queues' to cgroup.AvailableCPUs * 2 (#1385) 
* Change default value of '-remoteWrite.queues' to cgroup.AvailableCPUS() * 2 to reduce scrape interval

Default value of vmagent option '-remotewrite.queues' is 4 and default
size of vmagent ScheudleUnmarshalWorkers is number of CPUs, when available
CPUs is much greater than 4, e.g 32, worker are competing push queues
which will increase scrape interval and may cause scrape timeout.

* Update README and flag description

Co-authored-by: xiaozy <xiaozy01@fenbi.com>
 2021-06-16 12:16:44 +03:00
Roman Khavronenko
fb4f758715
promql: fix increase_pure calculation for cases with stale series (#1381) 
Due to staleness handling, increase_pure were using incorrect previous value
during calculation in cases where series disappears for period longer
than staleness period and then returns back. The fix suppose to account
for a real datapoint value before staleness takes place. The fix should
remove unexpected spikes while using `increase_pure` for staled series.
 2021-06-15 17:37:19 +03:00
Nikolay
729c4eeb9c
adds digital ocean sd (#1376) 
* adds digital ocean sd config

* adds digital ocean sd
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1367

* typo fix
 2021-06-14 13:15:04 +03:00
Roman Khavronenko
6d5a8c28cd
Vmalert docs (#1372) 
* vmalert: mention what happens if `for` is set to 0 or omitted

* vmalert: add more context to docs
 2021-06-11 13:25:53 +03:00
Aliaksandr Valialkin
ed83558646 app/vmauth: properly handle http.ErrAbortHandler panic 
This panic can be raised by the reverseProxy on aborted request to the backend.
So handle it (e.g. suppress) at reverseProxy.ServeHTTP call.

Do not suppress the panic at lib/httpserver generic HTTP handler,
since it may result in an inconsistent state left after the panicking handler.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1353
 2021-06-11 12:50:25 +03:00
Roman Khavronenko
7adfe878e1
vmalert: fix mistake with object reuse while parsing response (#1370) 
* vmalert: fix mistake with object reuse while parsing response

During the refactoring, the wrong optimisations was applied in
parse function which caused metric fields reset. The change removes
optimisation.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1369

* vmalert: add test to cover multiple metrics in one response
 2021-06-11 11:22:05 +03:00
John Belmonte
67b17cdd68
spelling fix: synonym (#1363) 2021-06-10 08:32:52 +03:00
Aliaksandr Valialkin
d0dca62026 app/vmselect/promql: typo fix in the comment 2021-06-09 18:34:31 +03:00
Aliaksandr Valialkin
2422b5091f app/vmauth: improve readability for a config with multiple src_paths 2021-06-09 15:39:32 +03:00
Aliaksandr Valialkin
f93a31b490 docs/vmagent.md: mention that vmagent supports scrape targets sharding 2021-06-09 12:29:34 +03:00
Aliaksandr Valialkin
ab15bf8c90 docs: document rules replay feature for vmalert 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/836

This is a follow-up for 2a259ef5e7
 2021-06-09 12:27:34 +03:00
Roman Khavronenko
2a259ef5e7
vmalert: support rules backfilling (aka replay) (#1358) 
* vmalert: support rules backfilling (aka `replay`)

vmalert can `replay` configured rules in the past
and backfill results via remote write protocol.
It supports MetricsQL/PromQL storage as data source,
and can backfill data to remote write compatible
storage.

Supports recording and alerting rules `replay`. See more
details in README.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/836

* vmalert: review fixes

* vmalert: readme fixes
 2021-06-09 12:20:38 +03:00
Aliaksandr Valialkin
78f83dc5ad app/{vmagent,vminsert}: follow-up after 2fe045e2a4 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1343
 2021-06-04 20:27:58 +03:00
jelmd
2fe045e2a4
new feature: debug relabeling (#1344) 
* new feature: relabel logging

Use scrape_configs[x].relabel_debug = true to log metric names inkl.
labels before and after relabeling. After relabeling related metrics
get dropped, i.e. not submitted to servers.

* vminsert wants relabel logging, too.
 2021-06-04 17:50:23 +03:00
Aliaksandr Valialkin
b22e380a34 app/vmauth: allow balancing the load among multiple backend nodes by specifying multiple urls in url_prefix config 2021-05-29 01:03:37 +03:00
Aliaksandr Valialkin
60341722d5 docs: document f0c21b6300 2021-05-27 15:03:30 +03:00
Roman Khavronenko
d210958fd0
vmalert: automatically reload configuration on file change (#1326) 
New flag `-rule.configCheckInterval` defines how often `vmalert` will re-read
config file. If it detects any changes, the config will be reloaded.
This behaviour is turned off by default.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/512
 2021-05-25 14:27:22 +01:00
Aliaksandr Valialkin
8e7d1f8824 app/vmagent/remotewrite: use WARN level instead of ERROR level for couldnt send a block with size ... bytes to ... log message 
This is really warning, since vmagent re-tries sending the data block until success.
 2021-05-24 15:43:59 +03:00
Roman Khavronenko
84cc0513e1
vmalert: support extra_filter_labels setting per-group (#1319) 
The new setting `extra_filter_labels` may be assigned to group.
If it is, then all rules within a group will automatically filter
for configured labels. The feature is well-described here
https://docs.victoriametrics.com#prometheus-querying-api-enhancements

New setting is compatible only with VM datasource.
 2021-05-23 00:26:01 +03:00
Aliaksandr Valialkin
78dddfb98f lib/promauth: follow-up after 5b8176c68e 2021-05-22 18:01:11 +03:00
Nikolay
5b8176c68e
basic OAuth2 support for remoteWrite and scrape targets (#1316) 
* adds OAuth2 support for remoteWrite and scrapping

* adds tests
changes init
 2021-05-22 16:20:18 +03:00
Aliaksandr Valialkin
c54bb73867 all: do not skip SIGHUP signal during service initialization 
This can lead to stale or incomplete configs like in the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240
 2021-05-21 16:34:06 +03:00
Nikolay
d626c5c2a9
changes vmalert query function (#1307) 
* changes vmalert query function
for prometheus rules compatibility its better to use labels as map.
it simplifies template evaluation and allow to ignore can't evaluate field error
because map will return default value.
fixes https://github.com/VictoriaMetrics/operator/issues/243
 2021-05-21 13:55:43 +03:00
Aliaksandr Valialkin
49ecbc765d app/vmauth: add ability to protect /-/reload endpoint with authKey 2021-05-20 18:47:01 +03:00
Aliaksandr Valialkin
4c7bb75fa2 Makefile: update golangci-lint from v1.29.0 to v1.40.1 2021-05-20 18:27:10 +03:00
Aliaksandr Valialkin
e394ff6466 app/vmagent/remotewrite: expose metrics with the current number of active series per day and per hour 
These numbers are exposed via the following metrics:

- vmagent_hourly_series_limit_current_series
- vmagent_daily_series_limit_current_series

Expose also the limits via the following metrics:

- vmagent_hourly_series_limit_max_series
- vmagent_daily_series_limit_max_series
 2021-05-20 15:28:09 +03:00
Aliaksandr Valialkin
ad73f226ff app/vmstorage: add ability to limit series cardinality via -storage.maxHourlySeries and -storage.maxDailySeries command-line flags 2021-05-20 14:15:19 +03:00
Aliaksandr Valialkin
7e526effaa app/vmagent: add ability to limit series cardinality on a per-hour and per-day basis 2021-05-20 13:13:40 +03:00
Roman Khavronenko
dcf8803bbd
vmctl: explicitly set ::tag type for labels selector in influx mode (#1310) 
The `::tag` type is needed in cases when field and tag names are equal, which
results into unexpected results in InfluxQL. Setting the type explicitly helps
InfluxDB to understand which exact column we apply filter to.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1299
 2021-05-20 12:03:16 +03:00
Aliaksandr Valialkin
0842bb9294 app/vmselect/promql: add timezone_offset(tz) function 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1306
 2021-05-20 11:53:09 +03:00
Aliaksandr Valialkin
f7e73fbe8b app/vmagent/remotewrite: sort labels before sending the series to per-remoteWrite.url queues 2021-05-20 02:12:36 +03:00
Neo He
e8a6c6927d
app/{vmbackup,vmrestore},docs/vmrestore.md: typo fix: vbackup -> vmbackup (#1305) 2021-05-18 16:37:28 +03:00
Aliaksandr Valialkin
f4719889da lib/httpserver: typo fix in -http.shutdownDelay command-line flag description: servier -> server 2021-05-18 16:26:16 +03:00
Aliaksandr Valialkin
b30925738b docs/vmalert.md: document multitenant support 
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/740
 2021-05-18 16:26:14 +03:00
Aliaksandr Valialkin
66aba00549 app/vmauth: reload -auth.config on the request to /-/reload 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1194
 2021-05-18 02:23:55 +03:00
Aliaksandr Valialkin
3339ea41e7 docs/vmbackup.md: typo fix: snaphosts -> snapshots 
Thanks to @jelmd - see 1ab27582a3 (r50884395)
 2021-05-18 01:12:36 +03:00
Aliaksandr Valialkin
6c944b86d8 docs: dealay -> delay 
Thanks to @jelmd . See 0b7e3510c8 (r50884991)
 2021-05-18 01:07:52 +03:00
Roman Khavronenko
b38edec7ee
vmalert: use stringified label keys for duplicates map in recroding rules (#1301) 
duplicates map helps to determine wheter extra labels has overriden
labels which make time series unique. It was using a sorted hashed
labels sequence as a key. But hashing algorithm could have collisions,
so it is more convenient to not use hashing at all.

Log message for recording rules duplicates was improved as well.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1293
 2021-05-15 13:25:57 +03:00
Aliaksandr Valialkin
10a47af631 app/{vmalert,vmauth}: explicitly set MaxIdleConnsPerHost in net/http.Client.Transport 
By default MaxIdleConnsPerHost is set to 2. This limits the possibility to re-use http keep-alive connections.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1300
 2021-05-14 18:12:24 +03:00
Aliaksandr Valialkin
a422165dc6 app/vmagent/remotewrite: clarify the comment explaining why vmagent drops blocks if remote storage returns 400 or 409 status code 2021-05-13 16:16:16 +03:00
Aliaksandr Valialkin
d7be2753c0 lib/storage: substitute GetTSDBStatusForDate with GetTSDBStatusWithFiltersForDate with nil tfss 2021-05-13 09:02:33 +03:00
Aliaksandr Valialkin
832651c6c2 app/vmselect: follow up after 8a0678678b 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1168
 2021-05-12 17:18:30 +03:00
Nikolay
8a0678678b
Adds tsdb match filters (#1282) 
* init work on filters

* init propose for status filters

* fixes tsdb status
adds test

* fix bug

* removes checks from test
 2021-05-12 15:18:45 +03:00
Aliaksandr Valialkin
229d9d6dd7 docs/CHANGELOG.md: document -datasource.roundDigits added at 5c448126dc 2021-05-10 11:18:26 +03:00
Roman Khavronenko
5c448126dc
vmalert: add support for round_digits param in datasource package (#1278) 
Starting from v1.56.0 VM supports `round_digits` which allows to limit
the number of digits after the decimal point in response value. The feature
can be used to reduce entropy of produced by recording rules values
and significantly improve the compression. See more details in link below.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/525
 2021-05-10 11:11:45 +03:00
Roman Khavronenko
4247168a2d
vmalert: fix error when rule didn't start if restore failed (#1279) 
Previously, `startGroup` could exit on restore errors despite the
`remoteRead.ignoreRestoreErrors` flag value. Now vmalert checks the
flag value before deciding whether to return error or just log it.
 2021-05-10 11:06:31 +03:00
Aliaksandr Valialkin
6ff19096be docs/vmagent.md: add stream parsing mode chapter 2021-05-08 23:14:07 +03:00
Aliaksandr Valialkin
12d733dd5d app/vminsert: add support for data ingestion via other vminsert nodes 2021-05-08 19:52:57 +03:00
Aliaksandr Valialkin
237e9f9fd7 app/vmalert: add missing comment for ErrStateRestore 2021-05-08 15:59:35 +03:00
Aliaksandr Valialkin
79ec35cef9 app/vmbackup: make sure that -snapshotName isnt set if -snapshot.createURL is set 2021-05-07 08:44:25 +03:00
Roman Khavronenko
9cdd4696fe
vmalert: add flag to control behaviour on startup for state restore errors (#1265) 
Alerting rules now can return specific error type ErrStateRestore to indicate
whether restore state procedure failed. Such errors were returned and logged
before as well. But now user can specify whether to just log these errors
(remoteRead.ignoreRestoreErrors=true) or to stop the process
(remoteRead.ignoreRestoreErrors=false). The latter is important when VM isn't
ready yet to serve queries from vmalert and it needs to wait.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1252
 2021-05-05 10:07:19 +03:00
Aliaksandr Valialkin
508f500477 docs/vmalert.md: update docs after afca7b430c 2021-04-30 11:49:01 +03:00
Roman Khavronenko
0f988e5a31
vmalert: fix the typo in ApplyParams func (#1259) 2021-04-30 10:31:45 +03:00
Roman Khavronenko
afca7b430c
vmalert: use rule's evaluationInterval as step param by default (#1258) 
User still can override param by specifying `datasource.queryStep` flag.
 2021-04-30 10:01:05 +03:00
Aliaksandr Valialkin
4394dc6cbb docs/CHANGELOG.md: document the change from f3a048288e 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1232
 2021-04-30 09:54:41 +03:00
Roman Khavronenko
f3a048288e
Vmalert: adjust time param for datasource queries according to evaluationInterval (#1257) 
* Simplify arguments list for fn `queryDataSource` to improve readbility

* vmalert: adjust `time` param according to rule evaluation interval

With this change, vmalert will start to use rule's evaluation interval
for truncating the `time` param. This is mostly needed to produce consistent
time series with timestamps unaffected by vmalert start time. Now, timestamp
becomes predictable.
Additionally, adjustment is similar to what Grafana does for plotting range graphs.
Hence, recording rule series and recording rule expression plotted in grafana
suppose to become similar in most of cases.
 2021-04-30 09:46:03 +03:00
Aliaksandr Valialkin
6fa5981e68 app/vmagent: list user-visible endpoints at http://vmagent:8429/ 
While at it, use common WriteAPIHelp function for the listing in vmagent, vmalert and victoria-metrics
 2021-04-30 09:36:43 +03:00
Nikolay
15609ee447
changes vmalert Querier with per rule querier (#1249) 
* changes vmalert Querier with per rule querier
it allows to changes some parametrs based on rule setting
for instance - alert type, tenant for cluster version or event endpoint url.
 2021-04-28 21:41:15 +01:00
Roman Khavronenko
87018650dd
vmalert: keep the returned timestamp when persisting recording rule (#1245) 
Previously, vmalert used `lastExecTime` timestamp when writing recording rules
to the remote storage. This may be incorrect, if vmalert uses `datasource.lookback` flag,
which means rule's expression will be executed at some moment in the past.
To avoid such situations, vmalert now will use returned timestamp instead of `lastExecTime`.
 2021-04-26 01:03:22 +03:00
Aliaksandr Valialkin
52b4b1605e app/vmagent/remotewrite: increase the maximum possible number of inmemory blocks for systems with high amounts of RAM 
This should reduce the probability of using much slower file-based persistent queue
when vmagent processes metrics at high rate (millions of metrics per second).

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1235
 2021-04-23 22:03:51 +03:00
Aliaksandr Valialkin
fb37b853e9 app/vmagent/remotewrite: count maxLabelsPerBlock as 10x of maxRowsPerBlock 
This should increase block sizes and subsequently increase the maximum possible bandwidth per each connection to remote storage.
This, in turn, should reduce the probability of storing the data in local buffers.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1235
 2021-04-23 21:55:47 +03:00
Aliaksandr Valialkin
1fbc04facd app/vmbackup: typo fix: snaphsot -> snapshot 
Follow-up for 9de0fa3649
 2021-04-22 11:16:56 +03:00
Aliaksandr Valialkin
7c4e460513 app/vmauth: parse url_prefix only once during config load 2021-04-21 10:55:29 +03:00
Aliaksandr Valialkin
6bc52fe41a all: rename https://victoriametrics.github.io to https://docs.victoriametrics.com 2021-04-20 20:16:17 +03:00
Aliaksandr Valialkin
5720b283fa all: consistency renaming Victoria Metrics -> VictoriaMetrics 
VMInsert -> vminsert
VMSelect -> vmselect
VMStorage -> vmstorage
 2021-04-20 11:45:48 +03:00
Aliaksandr Valialkin
187e3ec909 app/vmauth: follow-up for 6a81a89b3d 2021-04-20 10:58:29 +03:00
Nikolay
6a81a89b3d
adds query params support for vmauth urlPrefix (#1226) 
* adds query params support for vmauth urlPrefix

* Update app/vmauth/example_config.yml

* Update app/vmauth/example_config.yml

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2021-04-20 10:51:03 +03:00
Aliaksandr Valialkin
b3d88610fd app/vmctl: update README.md according to bfecd0fd55 2021-04-16 12:09:45 +03:00
Aliaksandr Valialkin
663a91bb82 docs: update -help output after the commit 77be3e3a82 2021-04-12 12:34:59 +03:00
Artem Navoiev
77be3e3a82
improve docs for cli flags (#1202) 
* improve docs for cli flags

* improve docs for cli flags.2
 2021-04-12 12:28:04 +03:00
Aliaksandr Valialkin
0b7e3510c8 docs: make docs-sync 2021-04-10 19:54:18 +03:00
Roman Khavronenko
7644efae01
Docs update (#1199) 
* docs: drop table of contents for `vmctl`

We already have it autogenerated on .github.io, so no need to keep it.

* docs: mention OpenTSDB migration feature for vmctl

* docs: sync docs for `vmalert`
 2021-04-10 15:46:08 +03:00
John Seekins
a9d76b06a7
Improve documentation on OpenTSDB migration tool and fix a bug with hard offsets (#1198) 
* add more documentation on OpenTSDB migration explaining what chunking means
* more clarification of OpenTSDB aggregations
* break out what a retention string becomes
* add more docs around retention strings
* add example of running program and fix mistake in how hard offsets are handled
* fix formatting
 2021-04-10 13:24:18 +01:00
John Seekins
cd9786c2a7
OpenTSDB migration to VictoriaMetrics (#1089) 2021-04-08 20:58:06 +01:00
Roman Khavronenko
4ed8de62ac
vmalert: document template functions and mention them in README (#1197) 2021-04-08 18:19:08 +03:00
Aliaksandr Valialkin
544821b719 app/vmselect/promql: fix tests after d3fa0ccabd 2021-04-08 00:18:01 +03:00
Aliaksandr Valialkin
d3fa0ccabd app/vmselect/promql: properly detect aggregate topk* and bottomk* aggregate functions in order to disable duplicate sorting 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1189
 2021-04-08 00:09:40 +03:00
Aliaksandr Valialkin
cb12a8f0a8 app/vmselect: return data:null instead of data:[] from /api/v1/query_exemplars, since Grafana throws an error otherwise 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1186
 2021-04-07 23:34:06 +03:00
Aliaksandr Valialkin
1177dca3da app/vmselect: do not sort series returned from topk* and bottomk* functions, since these series are already sorted in user-expected order 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1189
 2021-04-07 14:16:08 +03:00
Roman Khavronenko
c6a8ebb11f
docs: update docs ordering and formatting (#1192) 
The major change is adding `sort` directive to docs. For those docs which are copied
from internal packages `sort` is added via makefile command. For the rest it is added
manually since they're updated manually as well.

The rest of changes is connected with markdown formatting. For example, changing headers
in some files (`##` => `#`) makes navigation on .github.io to look better. This especially
useful for `changelog` docs.

Table of contents for `vmctl` is dropped, since we already have it autogenerated on .github.io.

No link changes expected. The corresponding PR to `cluster` branch will be made in follow-up PR.
 2021-04-07 13:39:16 +03:00
Aliaksandr Valialkin
7d23598b33 app/vmselect: return dumb response on /api/v1/query_exemplars request 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1186
 2021-04-05 23:25:08 +03:00
Aliaksandr Valialkin
9a97941e2a docs/vmagent.md: mention that vmagent supports scraping via socks5 proxy 2021-04-04 01:45:52 +03:00
Aliaksandr Valialkin
5153410ced lib/promscrape: support for simple HTTP proxies without CONNECT method support such as https://github.com/prometheus-community/PushProx 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1179
 2021-04-04 00:40:40 +03:00
Aliaksandr Valialkin
7a0b964e8d app/vmselect/promql: do not delete dst_label if src_label is empty in label_copy(q, src_label, dst_label) and label_move(q, src_label, dst_label) 2021-04-03 22:05:06 +03:00
Aliaksandr Valialkin
2256b79a89 docs/vmgateway.md: update docs 2021-04-03 00:29:19 +03:00
Aliaksandr Valialkin
3055ab0115 app/vmselect/promql: add ability to set label value additionally to label name for the remaining sum of time series returned from topk_* and bottomk_* functions in the form: topk_min(N, m, "label=value") 2021-04-02 23:55:54 +03:00
Aliaksandr Valialkin
25e19c75c7 docs/{vmauth,vmgateway}.md: small fixes 2021-04-02 23:15:00 +03:00
Aliaksandr Valialkin
3c1b39c978 app/vmgateway: publish docs 2021-04-02 23:08:41 +03:00
Aliaksandr Valialkin
0db901617d app: do not process non-GET requests on at / handler 2021-04-02 22:54:06 +03:00
Aliaksandr Valialkin
b1d0028e79 app/vmauth: add support for authorization via Authorization: Bearer <token> 2021-04-02 22:14:53 +03:00
Aliaksandr Valialkin
b88feb631e docs/vmagent.md: mention about proxy_authorization section 2021-04-02 21:24:11 +03:00
Aliaksandr Valialkin
df148f48b7 lib/promscrape: add support for authorization config in -promscrape.config as Prometheus 2.26 does 
See https://github.com/prometheus/prometheus/pull/8512
 2021-04-02 21:17:45 +03:00
Aliaksandr Valialkin
c79e4a2f90 app/vmselect/promql: remove the limit on the number of time series that can be sorted, since it may confuse users 
Always sort time series returned from `/api/v1/query` and `/api/v1/query_range` unless `sort_*` function is used at top level of the query.
 2021-04-02 15:02:08 +03:00
Aliaksandr Valialkin
dc9eafcd02 app/{vminsert,vmagent}: add -sortLabels command-line option for sorting time series labels before ingesting them in the storage 
This option can be useful when samples for the same time series are ingested with distinct order of labels.
For example, metric{k1="v1",k2="v2"} and metric{k2="v2",k1="v1"}.
 2021-03-31 23:27:58 +03:00
Aliaksandr Valialkin
e1f699bb6c lib/storage: reduce memory usage when ingesting samples for the same time series with distinct order of labels 2021-03-31 21:24:46 +03:00
Aliaksandr Valialkin
48275d8c12 app/vmagent/remotewrite: reduce memory usage when -remoteWrite.queues is set to a big value 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1167
 2021-03-31 16:16:33 +03:00
Aliaksandr Valialkin
33622c409c app/vmagent/remotewrite: reduce memory usage when samples with big number of labels are sent to remote storage 2021-03-31 00:44:54 +03:00
Aliaksandr Valialkin
e7fdea5953 app/vmselect: add -search.maxStatusRequestDuration command-line flag for limiting the duration of requests to /api/v1/status/* and /api/v1/series/count 2021-03-30 21:41:35 +03:00
Aliaksandr Valialkin
7962cf1af8 app/vmselect: prevent from possible incomplete query results after timed out query 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711
 2021-03-30 13:35:45 +03:00
Aliaksandr Valialkin
1db1a29ffa all: increase minimum supported Go version for building VictoriaMetrics components from v1.14 to v1.15 
This is needed after the commit c0ac740f93, which uses URL.Redacted() method,
which has been added in v1.15.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1147
 2021-03-29 23:04:53 +03:00
Aliaksandr Valialkin
78188decf9 docs: document that vmagent drops data blocks when remote storage replies with 400 and 409 http status codes 
This is a follow up for 1b7dc1e5a5.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1149
 2021-03-26 14:44:06 +02:00
Aliaksandr Valialkin
c54cb3e63c app/vmagent/remotewrite: remove superflouos code after 1b7dc1e5a5 2021-03-26 13:59:46 +02:00
Nikolay
1b7dc1e5a5
Adds blocks drop (#1151) 
* adds blocks drop at 400 BadRequest status code
recieved from remote storage,
not expected that remote storage will be able to handle it on retry

* removes error logging for dropped blocks,
its expected error
 2021-03-26 14:17:59 +03:00
Aliaksandr Valialkin
aa81039b42 app/vmselect: log the metric which trigger rollup result cache reset 
This should help finding the source of stale metrics
 2021-03-25 21:31:39 +02:00
Aliaksandr Valialkin
5ec9e49103 docs/vmagent.md: add an example for -remoteWrite.label 2021-03-25 17:54:55 +02:00
Aliaksandr Valialkin
f6529f932a docs: add a link to the repository from build instruction for all the VictoriaMetrics components 2021-03-25 17:14:42 +02:00
Aliaksandr Valialkin
2065d11300 docs/vmagent.md: cosmetic fixes 2021-03-25 17:11:19 +02:00
Aliaksandr Valialkin
35fb9bdee1 docs/vmagent.md: cosmetic fixes 2021-03-25 16:54:03 +02:00
Aliaksandr Valialkin
a647144616 docs/vmagent.md: typo fix: tupically -> typically 2021-03-25 16:48:45 +02:00
Aliaksandr Valialkin
c3c3e51f17 docs/vmalert.md: remove misleading -evaluationInterval=3s from example config args 
3s evaluation interval is too small for practical setups. It can result in increased load on datasource.
So it is better to remove it from example config args, which are usually copy-pasted by novice users.
 2021-03-25 15:29:06 +02:00
Aliaksandr Valialkin
0b2a66db30 app/vmselect/promql: do not merge time series during requests to /api/v1/query 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1141
 2021-03-25 13:56:07 +02:00
Aliaksandr Valialkin
d4aadba9fa app/vmagent: add -promscrape.consul.waitTime command-line flag for configuring Consul service discovery wait time 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1144
 2021-03-23 19:33:25 +02:00
Aliaksandr Valialkin
9c566f7db9 app/vmagent: mention -remoteWrite.maxDiskUsagePerURL in the descriptio of -remoteWrite.tmpDataPath flag 2021-03-23 16:38:48 +02:00
Aliaksandr Valialkin
331a6a2015 app/vmselect/graphite: accept and enforce extra_label in all the Graphite APIs 2021-03-23 15:29:16 +02:00
Aliaksandr Valialkin
b521d1d4f2 app/vmselect: move getEnforcedTagFiltersFromRequest to searchtuils, since it will be used in Graphite functions soon 2021-03-23 14:16:29 +02:00
Aliaksandr Valialkin
71b72304ae app/vmselect: improve description for -search.maxPointsPerTimeseries command-line flag 2021-03-22 16:45:34 +02:00
Aliaksandr Valialkin
44a6cc5eca app/{vminsert,vmagent}: use Influx field as metric name if measurement is empty and -influxSkipSingleField command-line is set 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1139
 2021-03-22 13:53:53 +02:00
Aliaksandr Valialkin
cef010d5f7 app/vmselect/promql: increment key prefix for faster reset for rollup result cache 2021-03-22 11:59:07 +02:00
Aliaksandr Valialkin
fb83e97170 app/victoria-metrics: use flag.Parse instead of envflag.Parse for avoiding possible side effects of envflag 2021-03-18 18:20:22 +02:00
Aliaksandr Valialkin
b0c956a178 app/vmselect/graphite: follow-up after 529d7be26b 2021-03-18 16:30:20 +02:00
Nikolay
529d7be26b
changes metricsFind api (#1137) 
it should be able mitigate crash if label value contains *,[ or { symbols
 2021-03-18 16:12:02 +02:00
Aliaksandr Valialkin
b1713e3fcd app/vmselect/promql: typo fix after 9666834045 2021-03-17 15:12:11 +02:00
Aliaksandr Valialkin
9666834045 app/vmselect/promql: merge adjancent buckets with the smallest summary number of hits in buckets_limit() function 
This should improve accuracy for the returned buckets
 2021-03-17 14:31:41 +02:00
Aliaksandr Valialkin
7bc728bf53 app/vmselect: add vm_index_search_duration_seconds histogram for monitoring the performance of index search 2021-03-17 01:17:41 +02:00
Aliaksandr Valialkin
828669e4e1 all: make golint happy 2021-03-17 00:49:28 +02:00
Aliaksandr Valialkin
f104f3eb2a all: make golangci-lint happy after the commit 6378205415 2021-03-17 00:24:40 +02:00
Aliaksandr Valialkin
6378205415 lib/netutil: enable IPv6 UDP listening if -enableTCP6 command-line flag is passed to VictoriaMetrics 
This is a follow-up for 18cfc4be7b

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1131
 2021-03-17 00:16:17 +02:00
Aliaksandr Valialkin
0ce557951f app/vmselect/netstorage: reduce mutex contention when unpacking data on a system with high number of CPU cores 2021-03-16 21:51:31 +02:00
Aliaksandr Valialkin
aba955fa16 Makefile: prepare vmutils-windows-*.zip archive on make release-vmutils command 
The archive contains the following executables for Windows:

* vmagent
* vmalert
* vmauth
* vmctl

Other components - vmbackup, vmrestore, victoria-metrics - aren't supported for Windows yet
 2021-03-16 20:52:41 +02:00
Aliaksandr Valialkin
f0a4157f89 app/vmselect/promql: do not crash if histogram_over_time() function name contains uppercase letters such as Histogram_over_time() 2021-03-16 12:24:21 +02:00
Aliaksandr Valialkin
dd7e82c34f app/vmstorage: add -logNewSeries command-line flag for determining the source of series churn rate 2021-03-15 22:38:50 +02:00
Aliaksandr Valialkin
85a95bf60c all: various fixes in command-line flag descriptions 2021-03-15 21:59:25 +02:00
Aliaksandr Valialkin
7c37e9aea9 app/{vminsert,vmagent}: a follow-up for b1aa8c3d8f 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1124
 2021-03-15 21:37:55 +02:00
Nikolay
b1aa8c3d8f
adds fake response for telegraph queries (#1130) 
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1124
 2021-03-15 21:10:47 +02:00
Aliaksandr Valialkin
923cdb0552 app/vmselect/promql: reduce overhead on scrape interval estimation 
It should be enough to use the first 20 datapoints instead of 100 datapoints for scrape interval estimation.
 2021-03-15 20:31:33 +02:00
Aliaksandr Valialkin
82aab87446 app/vmselect/promql: fix tests after 2dae0a2c47 2021-03-15 20:18:59 +02:00
Aliaksandr Valialkin
2dae0a2c47 app/vmselect: add round_digits query arg to /api/v1/query and /api/v1/query_range handlers for limiting the number of decimal digits after the point 2021-03-15 12:36:33 +02:00
Aliaksandr Valialkin
2096c6e464 app/vmselect/prometheus: typo fix after 7c002023d7 2021-03-12 12:19:36 +02:00
Aliaksandr Valialkin
7c002023d7 app/vmselect/prometheus: do not include datapoints with timestamps matching t-d when returning results from /api/v1/query?query=m[d]&time=t as Prometheus does 2021-03-12 12:16:50 +02:00
Aliaksandr Valialkin
a6a71ef861 lib/promscrape: add ability to configure proxy options via proxy_tls_config, proxy_basic_auth, proxy_bearer_token and proxy_bearer_token_file 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1116
 2021-03-12 03:36:19 +02:00
Ihor Borodin
d212f35ae8
Fixing examples of external.alert.source in documentation (#1120) 
* Fixing examples of external.alert.source in documentation
 2021-03-09 20:49:50 +00:00
Aliaksandr Valialkin
fe8b12fbad app/vmselect/promql: follow up for 433fff0006 2021-03-09 12:48:44 +02:00
Nikolay
433fff0006
duplicate timeseries fix for prometheus_buckets function (#1119) 
* try fix for prometheus_buckets

* merge possible end of the bucket collision
 2021-03-09 12:26:23 +02:00
Aliaksandr Valialkin
14a399dd06 lib/promscrape: add scrape_offset option to scrape_config 
This option can be used for specifying the particular offset per each scrape interval for target scraping
 2021-03-08 12:03:33 +02:00
Aliaksandr Valialkin
47ac2051bb app/vmauth: allow using regexps in url_map paths 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1112
 2021-03-05 18:21:36 +02:00
Aliaksandr Valialkin
a14053ffa0 app/vmselect/promql: add histogram_avg(), histogram_stddev() and histogram_stdvar() functions to MetricsQL 2021-03-04 14:12:07 +02:00
Aliaksandr Valialkin
423cd981fb lib/promscrape: add -promscrape.cluster.replicationFactor command-line flag for replicating scrape targets among vmagent instances in the cluster 2021-03-04 10:20:15 +02:00
Aliaksandr Valialkin
201b685b13 all: bump minimum supported Go version from 1.13 to 1.14 2021-03-03 15:57:13 +02:00
Aliaksandr Valialkin
f391e5a3a0 docs/vmagent.md: remove outdated suggestion for determining labels that lead to duplicate targets 
The original labels for duplicate targets is already printed in the error message starting from 71ea4935de
 2021-03-03 12:26:11 +02:00
Roman Khavronenko
621bf03745
Vmagent docs upd (#1104) 
* vmagent: port changes from https://github.com/VictoriaMetrics/VictoriaMetrics.github.io/pull/1

Thanks to @dereksfoster99 for this patch!

* vmagent: reword to make the meaning clear
 2021-03-03 10:51:51 +02:00
Aliaksandr Valialkin
e09a245b2b app/vmalert/README.md: sync with docs/vmalert.md 2021-03-03 10:43:59 +02:00
Aliaksandr Valialkin
baefe5a8ad docs: actualize -help output 2021-03-01 17:01:27 +02:00
Aliaksandr Valialkin
4e3cfe8461 app/vmagent/remotewrite: clarify -remoteWrite.flushInterval flag description 2021-03-01 11:50:54 +02:00
Aliaksandr Valialkin
369f01c738 app/vmagent/remotewrite: fix rate limiting logic for -remoteWrite.url 2021-03-01 00:58:34 +02:00
Aliaksandr Valialkin
cb943f35c7 app/vmagent: remove data race when applying rate limits to -remoteWrite.url with multiple queues 2021-03-01 00:29:07 +02:00
Aliaksandr Valialkin
8d021b73b5 docs/vmbackup.md: clarify docs on vmagent clustering 2021-02-28 22:00:15 +02:00
Aliaksandr Valialkin
2b53add6b2 app/vmselect/querystats: show the number of matching queries in the top by average duration and in the top by summary duration 
This should help debugging slow queries.
 2021-02-28 19:40:19 +02:00
Aliaksandr Valialkin
e5ca8ac0db lib/promscrape: add ability to spread scrape targets among multiple vmagent instances 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1084
 2021-02-28 18:41:08 +02:00
Nikolay
317b0cbed2
adds query params for vmalert (#1094) 
remoteWrite.url now accepts query params at provided url
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1087
 2021-02-27 10:04:58 +00:00
Aliaksandr Valialkin
4f8a72806a app/vmbackupmanager: add docs; the vmbackupmanager is available as a part of enterprise subscription 2021-02-27 02:17:55 +02:00
Aliaksandr Valialkin
7cc3d96a41 lib/fs: follow-up after f3a03c4164 2021-02-27 01:01:47 +02:00
Nikolay
f3a03c4164
Adds windows build (#1040) 
* fixes windows compilation,
adds signal impl for windows,
adds free space usage for windows,
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1036

NOTE victoria metrics database still CANNOT work under windows system,
only vmagent is supported.
To completly port victoria metrics, you have to fix issues with separators,
parsing and posix file removall

* rollback separator

* Adds windows setInformation api,
it must behave like unix, need to test it.
changes procutil

* check for invlaid param

* Fixes posix delete semantic

* refactored a bit

* fixes openbsd build

* removed windows api call

* Fixes code after windows add

* Update lib/procutil/signal_windows.go

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2021-02-27 00:37:07 +02:00
Nikolay
186c078fac
adds enforced tag filters into cache key (#1095) 2021-02-27 00:15:53 +02:00
Aliaksandr Valialkin
d86e9b49c4 app/vmselect/promql: increase accuracy for buckets_limit() function for small limits by skipping the first and the last buckets during merge 
The first and the last buckets are usually `[0 ... leMin]` and `(leMax ... +Inf)`. If they are merged with adjancent buckets,
then the resulting accuracy can suffer.
 2021-02-26 22:56:36 +02:00
Aliaksandr Valialkin
98854e5f2b app/vmselect: add sign(q) and clamp(q, min, max) functions, which will be added in the upcoming Prometheus release 
See https://twitter.com/roidelapluie/status/1363428376162295811

The `last_over_time(m[d])` function already exists in MetricsQL.
 2021-02-24 17:24:56 +02:00
Aliaksandr Valialkin
f4135b0d14 app/vmselect/promql: properly calculate histogram_quantile() over zero buckets and only a single non-zero le="+Inf"` bucket like Prometheus does 2021-02-24 00:42:22 +02:00
Aliaksandr Valialkin
fa03e0d210 app/vmselect/promql: add increase_pure() function to MetricsQL 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/962
 2021-02-22 19:14:15 +02:00
Aliaksandr Valialkin
d136081040 lib/promrelabel: add more optimizations for relabeling for common cases 2021-02-22 16:33:55 +02:00
Aliaksandr Valialkin
c2678754e4 app/vmagent: properly perform graceful shutdown, which was broken in the commit 1d1ba889fe 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1065
 2021-02-19 00:31:34 +02:00
Aliaksandr Valialkin
49e36e8d9d app/vmagent: fix scrape config example for scrape_align_interval option 2021-02-18 23:57:23 +02:00
Aliaksandr Valialkin
f26162ec99 lib/promscrape: add scrape_align_interval config option into scrape config 
This option allows aligning scrapes to a particular intervals.
 2021-02-18 23:53:44 +02:00
Aliaksandr Valialkin
9c70c1f21f app/vmselect/promql: reduce the probability of duplicate time series errors when querying Kubernetes metrics 2021-02-18 22:07:29 +02:00
Aliaksandr Valialkin
a537c4f602 lib/storage: properly handle queries containing a filter on metric name plus any number of negative filters and zero non-negative filters 
Example: `node_cpu_seconds_total{mode!="idle"}`
 2021-02-18 18:46:36 +02:00
Aliaksandr Valialkin
03ebc028f7 app/vmalert: add missing multiarch Dockerfile 2021-02-18 15:23:17 +02:00
Aliaksandr Valialkin
d7de4807e1 app/victoria-metrics/testdata: add a test for {__graphite__="foo.*.bar"} selector 2021-02-17 21:52:43 +02:00
Aliaksandr Valialkin
edcdc39eb3 app/vmagent/remotewrite: cleanup after 1d1ba889fe 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1065
 2021-02-17 21:42:55 +02:00
Nikolay
1d1ba889fe
adds pushback for fastqueue, (#1075) 
during shutdown currently sending block was lost,
now its pushed back to fast queue and will be flushed on disk,
it may lead to data duplication.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1065
 2021-02-17 21:23:38 +02:00
Aliaksandr Valialkin
d4849561ef app/vmstorage: export vm_composite_filter_success_conversions_total and vm_composite_filter_missing_conversions_total metrics 2021-02-17 19:13:38 +02:00
Aliaksandr Valialkin
7f2a6c7b54 docs: rename vmbackuper to vmbackupmanager 2021-02-16 22:00:39 +02:00
Aliaksandr Valialkin
c4756f94da app/vmselect/netstorage: reuse timeseriesWork objects in order to reduce memory allocations 2021-02-16 16:08:53 +02:00
Aliaksandr Valialkin
5a401225c7 app/vmselect/netstorage: use unsafe string as a key for a map when the map already contains the given key 
This should prevent from a memory allocation and a string copy.
 2021-02-16 15:43:10 +02:00
Aliaksandr Valialkin
4e39bf148c vendor: update github.com/VictoriaMetrics/metrics from v1.13.1 to v1.14.0 
The new version switches from log-linear histograms to log-based histograms,
which provide up to 3.6 times better accuracy.
 2021-02-15 15:12:29 +02:00
Aliaksandr Valialkin
802fabf0d7 deployment/docker: embed tzdata into prod Go app instead of installing it into base docker image 
While this increases app size by 700Kb, this allows using -loggerTimezone in a scratch base image
See https://github.com/golang/go/issues/38017
 2021-02-12 04:54:27 +02:00
Aliaksandr Valialkin
f9902b3372 deployment/docker: use docker buildx for creating multiarch builds 
See https://github.com/docker/buildx/
 2021-02-12 04:31:22 +02:00
Aliaksandr Valialkin
acdb401585 app/vmselect/prometheus: treat match query arg in the same way as match[] query arg 2021-02-11 15:02:21 +02:00
Aliaksandr Valialkin
1e38ad6d20 app/vmauth: add ability to route requests from a single users to multiple targets depending on the requested path 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1064
 2021-02-11 12:41:16 +02:00
Aliaksandr Valialkin
2d33230793 app/vmselect/promql: properly make copies of EvalConfig 2021-02-11 12:41:15 +02:00
Aliaksandr Valialkin
7a3a9421f3 app/vmselect/promql: make a copy of EvalConfig when executing q1 and q2 in parallel for q1 binary_op q2 
This should prevent from data races if the underlying functions modify EvalConfig contents.
 2021-02-10 23:05:55 +02:00
Aliaksandr Valialkin
04faea8b45 app/vmselect: parallelize q1 <binary_op> q2 queries by running q1 and q2 in parallel 
This should reduce query execution times.
 2021-02-10 22:59:39 +02:00
Aliaksandr Valialkin
cdecf83ce5 app/vmstorage: export vm_composite_index_min_timestamp metric 2021-02-10 17:14:08 +02:00
Aliaksandr Valialkin
553016ea99 lib/storage: disable composite index usage when querying old data 2021-02-10 14:57:50 +02:00
Aliaksandr Valialkin
c7dccebaef lib/storage: optimize search by label filters matching big number of time series 2021-02-10 00:44:54 +02:00
Aliaksandr Valialkin
cb96a1865b app/vmstorage: export missing vm_cache_size_bytes metrics for indexdb and data caches 2021-02-09 00:47:00 +02:00
Aliaksandr Valialkin
9b9cb04511 docs/vmctl.md: fix title, so it is properly displayed in the header of https://victoriametrics.github.io/ 2021-02-04 20:20:08 +02:00
Aliaksandr Valialkin
803a00102a app/{vminsert,vmselect}: accept requests to paths with /graphite and /prometheus prefixes 
This should improve compatibility with path prefixes from VictoriaMetrics cluster.
See https://victoriametrics.github.io/Cluster-VictoriaMetrics.html#url-format
 2021-02-04 20:01:18 +02:00
Aliaksandr Valialkin
4d43ab0875 app/vmselect: typo fix when stripping url path prefixes 2021-02-04 19:29:25 +02:00
Roman Khavronenko
2ff038d841
vmalert: mention -datasource.appendTypePrefix in README (#1052) 2021-02-03 23:44:37 +02:00
Dmitry Shevchuk
2baf98082b
Adds ability to query right vmselect endpoint based on the query type (#1050) 
* Adds ability to query right vmselect endpoint based on the query type

Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
 2021-02-03 21:26:30 +00:00
Aliaksandr Valialkin
a2344ef4b7 docs/vmalert.md: mention that type option can be set at group level additionally to rule level 2021-02-03 21:13:13 +02:00
Aliaksandr Valialkin
8dc6095749 app/vmagent: add Advanced usage section with the description for all the command-line flags 2021-02-03 21:02:59 +02:00
Aliaksandr Valialkin
8629fd8a72 app/vmselect: deprecate -search.treatDotsAsIsInRegexps in favor to {__graphite__="foo.*.bar"} syntax 2021-02-03 20:36:01 +02:00
Aliaksandr Valialkin
d16f22f3a1 app/vmselect,lib/storage: properly parse Graphite selectors with inner wildcards 
Example: foo{bar{x,yz},a[b-c],*de}
 2021-02-03 20:14:22 +02:00
Aliaksandr Valialkin
157c02622b app/vmselect: add ability to set Graphite-compatible filter via {__graphite__="foo.*.bar"} syntax 2021-02-03 01:21:54 +02:00
Aliaksandr Valialkin
b577cdd855 docs: increase heading sizes in vmagent, vmauth, vmbackup and vmrestore docs, so they match the heading sizes in VictoriaMetrics docs 2021-02-01 19:44:00 +02:00
Aliaksandr Valialkin
8164cd8932 docs/vmctl.md: update build instructions after the migration from github.com/VictoriaMetrics/vmctl to github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl 2021-02-01 19:39:08 +02:00
Aliaksandr Valialkin
b43b498fd8 app/vmselect: add ability to pass extra_label=<label>=<value> query arg to Prometheus Querying API 
This enforced `{label="value"}` label filter to the query.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1021
 2021-02-01 18:04:17 +02:00
Aliaksandr Valialkin
5d87dbfd65 docs: document ability to query Graphite datasource from vmalert 2021-02-01 15:26:33 +02:00
Nikolay
195341a7cf Graphite vmalert wip (#112) 
* init implementation for graphite alerts

* adds graphite support for vmalert

* small fix

* changes vmalert graphite api with type

* updates tests

* small fix

* fixes graphite parse

* Fixes graphite from time
 2021-02-01 15:05:32 +02:00
Aliaksandr Valialkin
a4ae945a79 app/victoria-metrics: fix tests after 8749c2dd92 2021-02-01 14:34:11 +02:00
Aliaksandr Valialkin
b2aa80e74b app/vmagent: add -remoteWrite.roundDigits command-line option for limiting the number of digits after the point for stored values 
This commit also adds --vm-round-digits command-line option to vmctl tool.
 2021-02-01 14:27:09 +02:00
Aliaksandr Valialkin
29a7067827 app/vmctl: fix make check-all warnings 2021-02-01 01:31:25 +02:00
Aliaksandr Valialkin
d5c180e680 app/vmctl: move vmctl code from github.com/VictoriaMetrics/vmctl 
It is better developing vmctl tool in VictoriaMetrics repository, so it could be released
together with the rest of vmutils tools such as vmalert, vmagent, vmbackup, vmrestore and vmauth.
 2021-02-01 01:10:20 +02:00
Aliaksandr Valialkin
de3c662e8a all: consistently use timers from timerpool 2021-01-27 00:39:26 +02:00
Aliaksandr Valialkin
419ad74269 app/vmagent: add -remoteWrite.rateLimit command-line flag for limiting data rate to remote storage 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1035
 2021-01-27 00:39:26 +02:00
weng zhao
cc3e69e963
vmalert: add option datasource.queryStep to allow user to address the inconsistency between grafana dashboards(query_range with step 15s usually) and ALERTS (#1027) 
Co-authored-by: zhao.weng <zhao.weng@shopee.com>
 2021-01-26 08:12:04 +00:00
Aliaksandr Valialkin
c164a8d231 app/vmselect/promql: improve documentation for -search.maxPointsPertimeseries command-line flag 
This should reduce incorrect usage and assumptions for this flag.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1020
 2021-01-22 13:00:10 +02:00
Aliaksandr Valialkin
8749c2dd92 app/vmselect: add -search.maxStepForPointsAdjustment command-line flag, which can be used for disabling adjustment for points returned from /api/v1/query_range handler if they have timestamps closer than -search.latencyOffset to the current time 2021-01-19 22:56:32 +02:00
Aliaksandr Valialkin
011c5da785 app/vmselect/graphite: extract getCanonicalPath() function from loop body inside getCanonicalPaths() 2021-01-18 17:30:26 +02:00
Aliaksandr Valialkin
75f7c51cab docs/vmagent.md: follow-up for 184a659c5f 2021-01-13 13:53:14 +02:00
Aliaksandr Valialkin
1051d8aa2d app/vmselect/promql: add ability to pass multiple labels to sort_by_label and sort_by_label_desc functions 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/992
 2021-01-13 12:44:51 +02:00
Aliaksandr Valialkin
acf1a2c72b app/vmselect/promql: properly parse escaped multibyte utf8 code sequences in metric names and labels names 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/990
 2021-01-13 10:59:42 +02:00
Nikolay
dc9d7aedd5
adds extra_label to all import apis (#1007) 
* adds extra_label to all import apis,
changes priority for extra_label - now it has priority over original labels

* Update README.md

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>

* Update README.md

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>

* adds extra labels to vmagent  import api
changes order for adding labels, now its added after user values

* adds tests for extra_label

* import fix

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2021-01-13 00:52:50 +02:00
Aliaksandr Valialkin
1837f2f7d3 app/vmselect/promql: add tfirst_over_time(m[d]) and tlast_over_time(m[d]) MetricsQL functions for returning timestamps for the first and the last samples in m over d 2021-01-12 16:12:12 +02:00
Nikolay
7976c22797
Fixes error handling for promscrape.streamParse (#1009) 
properly return error if client cannot read data,
properly suppress scraper errors
 2021-01-12 13:31:47 +02:00
Aliaksandr Valialkin
24ffad74c1 all: use net.Dial instead of fasthttp.Dial, because fasthttp.Dial limits the number of concurrent dials to 1000 2021-01-11 12:53:30 +02:00
Roman Khavronenko
2e2e4f7e21
vmalert-989: return non-empty result in template func query stub to pass validation (#1002) 
On templates validation stage vmalert does not acutally send queries, so for complex
chained expression validation may fail. To avoid this, we add a blank sample in response
so validation can pass successfully. Later, during the rule execution, stub will be replaced
with real `query` function.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/989
 2021-01-10 02:56:11 +03:00
Aliaksandr Valialkin
9dcb18e03d app/vmstorage: disable final merge by default, since it may result in high disk IO and CPU usage without measurable benefits such as increased query performance and reduced disk space usage 2021-01-08 00:16:05 +02:00
Aliaksandr Valialkin
c86286ec1d app/vmselect/promql: do not ajdust offset value provided in the query 
Previously it could be modified in order to improve response cache hit ratio.
This is unneeded, since cache hit ratio should remain good because the query time range
should be already aligned to multiple of `step` values.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/976
 2020-12-27 14:09:25 +02:00
Aliaksandr Valialkin
4b7105a65b app/vmselect: sync query stats handling with cluster version 2020-12-27 13:00:29 +02:00
Aliaksandr Valialkin
df0309eae0 app/vmselect/promql: simplify defer call for querystats.RegisterQuery 2020-12-27 12:06:04 +02:00
Aliaksandr Valialkin
ad4e6a9283 app/vmselect/querystats: reduce the default number of last queries to track from 100K to 20K 
This should reduce memory usage in constrained environments
 2020-12-25 17:40:47 +02:00
Aliaksandr Valialkin
59183f66d0 app/vmselect: refactor /api/v1/stats/top_queries 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/907
 2020-12-25 16:44:29 +02:00
Aliaksandr Valialkin
fb338c50a3 app/victoria-metrics: show usage info when incorrect command-line flag is passed to executable 2020-12-25 16:42:21 +02:00
Nikolay
86630350bf
Adds query stats handler (#945) 
* Adds query stat handler,
for query and query_range api, victoriametrics tracks query execution time,
stats are expored at /api/v1/status/queries endpoint with topN param
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/907

* fixed query stats bugs

* improves queryStats tracker

* improves query stat

* small fix

* fix tests

* added more tests

* fixes 386 tests

* naming fixes

* adds drop for outdated records
 2020-12-25 16:42:05 +02:00
Nikolay
1de15ad490
adds escape for CRLF (#984) 
at external.alert.source - \n and \r symbols was url encoded, instead of direct usage.
replace it from "\n" to `\n`  allows to skip url encoding.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/890
 2020-12-25 11:03:13 +02:00
Aliaksandr Valialkin
0326638c90 app/vmalert: typo fix in descriptions for notifier.basicAuth.username and notifier.basicAuth.password command-line flags 2020-12-24 12:48:59 +02:00
Aliaksandr Valialkin
9df60518bb docs: mention that it is possible to set multiple -notifier.tlsInsecureSkipVerify command-line flags for vmalert 
See c3a92968343c2b3619f1ab935702d0e9b3a46733
 2020-12-22 22:32:13 +02:00
Nikolay
c270f8f3e6
changes vmalert notifier flag, (#978) 
fixes issue with notifier insecure setting, now its possible to use multiple notifier.tlsInsecureSkipVerify multiple time.
 2020-12-22 23:23:04 +03:00
Roman Khavronenko
404cbd1522
vmalert-974: fix order for labels templating (#975) 
The change fixes bug caused by 3adf8c5a6f.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/974
 2020-12-19 14:10:59 +02:00
Aliaksandr Valialkin
88ac4dfc07 app/vmselect: properly parse negative combined offsets such as -1h2m3s 
Previously such offsets were parsed as `-1h + 2m + 3s`. Now they are parsed as `-(1h + 2m + 3s)`.
 2020-12-19 01:23:46 +02:00
Aliaksandr Valialkin
aa56b9217e app/vmagent: add vmagent_remotewrite_blocks_sent_total and vmagent_remotewrite_bytes_sent_total metrics per each -remoteWrite.url 2020-12-15 20:39:57 +02:00
Aliaksandr Valialkin
b10ad44692 docs/vmagent.md: typo fix: pearsed->parsed 2020-12-15 19:03:01 +02:00
Aliaksandr Valialkin
1eabbc0e27 docs/vmagent.md: mention that sample_limit option has no sense when stream parsing is enabled 2020-12-15 18:43:41 +02:00
Aliaksandr Valialkin
11fa458e39 app/vmselect/promql: return expected increase() result for the first point on the graph with value not exceeding 100 2020-12-15 13:40:46 +02:00
Nikolay
95c9b630cc
adds new Array Flags (#965) 
* adds ArrayDuration and ArrayBool flags,
makes sendTimeout and tlsInsecure configurable per remoteWrite url

* added backward compatibility testcases for ArrayDuration and ArrayBool

* fixes bool flag

* fixes test cases
 2020-12-15 12:51:12 +02:00
Roman Khavronenko
6247884057
vmalert: add function "query", "first" and "value" to alert templates functions (#960) 
The commit adds a support for template function `query`,
`first` and `value`. The function `query` executes
a MetricsQL query for active alerts. In vmalert we
update templates on every evaluation for active alerts
to keep them up to date. With `query` func it may become
a perf issue since it will fire a query on every execution.
We should keep it in mind for now.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/539
 2020-12-14 20:11:45 +02:00
Aliaksandr Valialkin
f8e7f433cf app/victoria-metrics: prettify / page output 2020-12-14 14:07:58 +02:00
Aliaksandr Valialkin
069c9ade52 app/{vmagent,vminsert}: follow-up for ce8c2dd1f1: return /targets page in HTML when requested via web browser 2020-12-14 14:06:00 +02:00
Nikolay
ce8c2dd1f1
Changes targets api (#961) 
* changes /targets api
adds html response if requester accepts text/html,
adds quick template for /targets api,
fixes pathPrefix for / requests

* changes namings

* renamed targets file

* Update app/victoria-metrics/main.go

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>

* adds trimspace to qtpl,
moves content-type for targets response closer to writer

* fixes bug with prefix

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2020-12-14 13:36:48 +02:00
Aliaksandr Valialkin
5ebfc275e6 app/victoria-metrics: automatically reset response cache when samples with too timestamps older than now - search.cacheTimestampOffset are ingested 2020-12-14 13:08:28 +02:00
Aliaksandr Valialkin
9d42546a27 docs: consistently use links to https://victoriametrics.github.io for documentation references 2020-12-11 21:08:18 +02:00
Aliaksandr Valialkin
ba8ac08739 app/vmselect/graphite: properly handle wildcards and charsets inside curly braces 
For example, `foo{bar*,[a-f]a*b}` should match `foobar`, `foobar123`, `foofab`, etc.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/952
 2020-12-11 17:24:12 +02:00
Aliaksandr Valialkin
4146fc4668 all: properly handle CPU limits set on the host system/container 
This can reduce memory usage on systems with enabled CPU limits.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/946
 2020-12-08 21:07:29 +02:00
Aliaksandr Valialkin
1906f841c9 app/vmselect/promql: do not reduce lookbehind window for any_rollup_func(m) to -search.maxStalenessInterval. It should equal to step value passed to /api/v1/query_range as most users expect 2020-12-08 15:16:17 +02:00
Aliaksandr Valialkin
b6b1b06d70 app/{vmalert,vmagent}: skip empty values in -remoteWrite.label and -label lists 2020-12-08 14:55:13 +02:00
Aliaksandr Valialkin
007dbf273d app/vmselect/graphite: remove duplicate name tag from /tags/autoComplete/tags handler 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/942
 2020-12-07 01:08:52 +02:00
Aliaksandr Valialkin
e2e8ef86d9 app/vmselect/promql: add count_eq_over_time(m[d], N) and count_ne_over_time(m[d], N) for calculating the number of samples in m over d that are equal / not equal to N 2020-12-05 12:30:46 +02:00
Aliaksandr Valialkin
9d787f9edd all: do not print usage info for all the flags when incorrect command-line flag is passed 
This should improve usability for VictoriaMetrics apps that have big number of command-line flags,
i.e. all the apps.
 2020-12-03 21:47:37 +02:00
Aliaksandr Valialkin
66379cc69f app/vmselect/promql: add label_uppercase(q, label1, ... labelN) and label_lowercase(q, label1, ... labelN) functions 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/936
 2020-12-03 21:47:36 +02:00
Aliaksandr Valialkin
44a34a0f5f app/vmselect/promql: make fmt 2020-12-02 21:33:35 +02:00
Aliaksandr Valialkin
1982505c2b app/vmselect/promql: return nan from minute(m) when m equals to nan 
This aligns VictoriaMetrics behaviour with Prometheus behaviour.

The issue has been spotted in https://promlabs.com/promql-compliance-test-results/2020-12-01/victoriametrics/
 2020-12-02 20:16:58 +02:00
Aliaksandr Valialkin
9d87496b50 app/vmselect/promql: do not return 0 value from sum_over_time(m[d]) when there are no samples on the given d window. 
This aligns the behaviour of `sum_over_time()` with other `_over_time()` functions and with Prometheus behavior.
 2020-12-02 13:12:50 +02:00
Aliaksandr Valialkin
91a4c279cc app/vmselect: return metric values from time() cmp_op metric query when cmp_op comparison is true 
This aligns MetricsQL behavior to Prometheus' one.

The issue has been identified at https://promlabs.com/promql-compliance-test-results/2020-12-01/victoriametrics/
 2020-12-02 12:09:34 +02:00
Aliaksandr Valialkin
700bda8e2e app/vmselect/promql: return nan from a >bool b if a is nan in the same way as Prometheus does 2020-12-02 00:28:26 +02:00
Aliaksandr Valialkin
efdc3c71af app/vmselect/searchutils: return elapsed time in Deadline.String() output 
This should improve debuggability for error messages containing Deadline.String() output
 2020-12-01 00:15:18 +02:00
Aliaksandr Valialkin
ca091bade3 app/vmbackup/snapshot: add missing status code check for the returned response when working with snapshot API 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/929
 2020-11-30 14:49:39 +02:00
Aliaksandr Valialkin
b35b3dc043 app/vmbackup/snapshot: log url and response body on failed JSON response parsing 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/929
 2020-11-29 12:16:16 +02:00
Nikolay
0463cb5550
fixes checksum calculation (#928) 
* fixes checksum calculation,
'for' rule param wasnt marshal properly during checksum calculation

* fixes error
 2020-11-29 09:48:42 +02:00
Aliaksandr Valialkin
b7fcdb528d app/{vmagent,victoria-metrics}: add -dryRun option and make more clear handling for -promscrape.config.dryRun 2020-11-25 22:59:13 +02:00
Aliaksandr Valialkin
dabbf930d8 app/vmagent: do not enable -promscrape.config.strictParse when -dryRun command-line flag is set 
Users can specify -promscrape.config.strictParse if -promscrape.config shouldn't contain unknown config entries
 2020-11-25 22:26:25 +02:00
BigFish
0acdab3ab9
Update main.go (#922) 
fix spelling mistake
 2020-11-23 17:33:17 +02:00
Aliaksandr Valialkin
7e8dcf9ddc app/vmbackup: cosmetic fixes 2020-11-23 17:10:04 +02:00
Aliaksandr Valialkin
a724dde90a app/vmselect: protect /tags/delSeries with -deleteAuthKey in the same way as /api/v1/admin/tsdb/delete_series 2020-11-23 15:35:59 +02:00
Aliaksandr Valialkin
f0c207fae2 app/vmselect: add /tags/delSeries handler from Graphite Tags API 
See https://graphite.readthedocs.io/en/stable/tags.html#removing-series-from-the-tagdb
 2020-11-23 15:27:21 +02:00
Aliaksandr Valialkin
d3794eb994 app/{vminsert,vmselect}: move /tags/tagSeries and /tags/tagMultiSeries api from vminsert to vmselect 
This is needed for consistency, since all the `/tags*` api handlers are located in vmselect.
 2020-11-23 12:33:19 +02:00
Aliaksandr Valialkin
f765985947 lib/fs: replace fs.OpenReaderAt with fs.MustOpenReaderAt 
All the callers for fs.OpenReaderAt expect that the file will be opened.
So it is better to log fatal error inside fs.MustOpenReaderAt instead of leaving this to the caller.
 2020-11-23 09:57:21 +02:00
Nikolay
eec76718e9
Adds eureka service discovery (#913) 
* Adds eureka service discovery
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/851
Netflix service discovery for AWS

* Apply suggestions from code review

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2020-11-20 13:38:12 +02:00
Aliaksandr Valialkin
2859a452d4 app/vmselect: add remoteAddr to slow query log in order to improve debuggability 
This will simplify identifying the client that sends slow queries to VictoriaMetrics.
 2020-11-18 20:38:32 +02:00
Nikolay
f818ab497b
Fixes snap script (#909) 2020-11-18 17:46:31 +03:00
Nikolay
c2afa3fdd7
adds snap package for victoria-metrics (#904) 
* adds snap package for victoria-metrics

* Update README.md

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2020-11-18 02:00:06 +02:00
Aliaksandr Valialkin
266788be14 app/vmselect: use storage.NewSearchQuery() instead of constructing storage.SearchQuery in-place 
This should prevent from bugs when AccountID and ProjectID aren't set in storage.SearchQuery.
 2020-11-16 18:24:00 +02:00
Aliaksandr Valialkin
cf18df367d app/vmselect/netstorage: apply Graphite filter after substituting __name__ with name 2020-11-16 15:52:16 +02:00
Aliaksandr Valialkin
2c67232565 app/vmselect/graphite: add /tags/autoComplete/values handler from Graphite Tags API 2020-11-16 15:29:35 +02:00
Aliaksandr Valialkin
86f99c6b55 app/vmselect/graphite: add /tags/autoComplete/tags handler from Graphite Tags API 
See https://graphite.readthedocs.io/en/stable/tags.html#auto-complete-support
 2020-11-16 14:50:05 +02:00
Aliaksandr Valialkin
3c1434118e app/vmselect/prometheus: return __name__ label if match[] query to /api/v1/labels matches at least a single time series 2020-11-16 13:54:34 +02:00
Aliaksandr Valialkin
27a417bcd3 app/vmselect/prometheus: improve performance for /api/v1/labels and /api/v1/label/<labelName>/values on time ranges exceeding one day when match[] query arg is set 2020-11-16 13:51:59 +02:00
Aliaksandr Valialkin
6fa806f1ca app/vmselect/prometheus: fix deadlock in /api/v1/series on a time range exceeding one day 2020-11-16 13:30:47 +02:00
Aliaksandr Valialkin
465923b181 app/vmselect/graphite: add /tags/findSeries handler from Graphite Tags API 
See https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags
 2020-11-16 12:53:13 +02:00
Aliaksandr Valialkin
414cd39659 app/vmselect/graphite: apply filter then limit 2020-11-16 04:09:14 +02:00
Aliaksandr Valialkin
d100341394 app/vmselect/graphite: add /tags/<tag_name> handler for Graphite Tags API 2020-11-16 03:42:25 +02:00
Aliaksandr Valialkin
6251762787 app/vmselect/graphite: add /tags handler from Graphite Tags API 
See https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags
 2020-11-16 03:00:01 +02:00
Aliaksandr Valialkin
48d033a198 app/vminsert: add /tags/tagSeries and /tags/tagMultiSeries handlers from Graphite Tags API 
See https://graphite.readthedocs.io/en/stable/tags.html#adding-series-to-the-tagdb
 2020-11-16 02:39:58 +02:00
Aliaksandr Valialkin
d9d01f976b app/vmselect/promql: remove spikes from increase() and delta() results on time series with spare irregular data points 
Do not take into account spare data point value if the next point will is located too far from the current point.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/894
 2020-11-13 15:23:44 +02:00
Aliaksandr Valialkin
1f19c167a4 app/vmselect/promql: assume that time series value doesnt change during gaps when calculating increase() and delta() 
This should remove unexpected spikes at the end of gaps.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/894
 2020-11-13 14:59:24 +02:00
Aliaksandr Valialkin
47a038401b all: consistently return text-based HTTP responses with charset=utf-8 
This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/897
 2020-11-13 10:35:41 +02:00
faceair
077f8cbe1c
add charset on targets response (#897) 2020-11-13 10:17:37 +02:00
Aliaksandr Valialkin
4057305148 docs/vmagent.md: added a link to https://valyala.medium.com/how-to-use-relabeling-in-prometheus-and-victoriametrics-8b90fc22c4b2 into Relabeling section 2020-11-12 12:27:06 +02:00
Aliaksandr Valialkin
bb06b98202 docs/vmagent.md: typo fix 2020-11-11 16:04:46 +02:00
Aliaksandr Valialkin
4adb96161a docs/vmagent.md: add Configuration update section 2020-11-11 16:01:15 +02:00
immerrr again
51c529a2b6
app/vmstorage: add "/internal/force_flush" endpoint (#893) 2020-11-11 14:40:27 +02:00
Aliaksandr Valialkin
348edd92fe app/vmselect: add -search.treatDotsAsIsInRegexps command-line flag for automatic escaping of dots in regexp label filters 2020-11-11 12:39:07 +02:00
Aliaksandr Valialkin
47390d8947 app/vmselect/promql: do not return data points in the end of the selected time range for time series ending in the middle of the selected time range 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/887
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845
 2020-11-10 14:51:44 +02:00
Aliaksandr Valialkin
ba4a2c8bca app/vmselect: typo fix in a description for -search.minStalenessInterval: mimimum->minimum 2020-11-10 01:18:08 +02:00
Roman Khavronenko
3adf8c5a6f
vmalert: explicitly set extra labels to alert entities (#886) 
The previous implementation treated extra labels (global and rule labels) as
separate label set to returned time series labels. Hence, time series always contained
only original labels and alert ID was generated from sorted labels key-values.
Extra labels didn't affect the generated ID and were applied on the following actions:
- templating for Summary and Annotations;
- persisting state via remote write;
- restoring state via remote read.

Such behaviour caused difficulties on restore procedure because extra labels had to be dropped
before checking the alert ID, but that not always worked. Consider the case when expression
returns the following time series `up{job="foo"}` and rule has extra label `job=bar`.
This would mean that restored alert ID will be always different to the real time series because
of collision.

To solve the situation extra labels are now always applied beforehand and `vmalert` doesn't
store original labels anymore. However, this could result into a new error situation.
Consider the case when expression returns two time series `up{job="foo"}` and `up{job="baz"}`,
while rule has extra label `job=bar`. In such case, applying extra labels will result into
two identical time series and `vmalert` will return error:
 `result contains metrics with the same labelset after applying rule labels`

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/870
 2020-11-10 00:27:32 +02:00
Aliaksandr Valialkin
bcd139362b lib/promscrape: add -promscrape.dropOriginalLabels command-line flag for reducing memory usage when discovering big number of scrape targets 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825
 2020-11-10 00:19:57 +02:00
Aliaksandr Valialkin
83df20b5b5 lib/promscrape: clean references to label name and label value strings after applying per-target relabeling 
This should reduce memory usage when per-target relabeling creates big number of temporary labels
with long names and/or values.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825
 2020-11-07 16:19:59 +02:00
Aliaksandr Valialkin
dd6bfa50e9 app/vmselect/promql: code cleanup after 43823addea 2020-11-06 01:30:50 +02:00
n4mine
43823addea
app/vmselect/promql: fix when the parameter of maxValue(), minValue() leading by NaN. it will cause {top,bottom}k_{max,min} return inappropriate result (#883) 2020-11-06 01:29:24 +02:00
Aliaksandr Valialkin
368b69b4c4 app/vmselect: properly handle errors in GetLabelsOnTimeRange and GetLabelValuesOnTimeRange 2020-11-05 01:38:38 +02:00
Aliaksandr Valialkin
b378cd6ed8 app/vmselect: optimize querying for /api/v1/labels and /api/v1/label/<name>/values when start and end args are set 2020-11-05 01:01:33 +02:00
Aliaksandr Valialkin
381ad564a2 docs/vmagent.md: update after 4c808d58bf 2020-11-04 20:31:43 +02:00
Nikolay
4c808d58bf
Adds ready probe (#874) 
* adds leading forward slash check for scrapeURL path
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/835

* adds ready probe for scrape config initialization,
it should prevent metrics loss during vmagent rolling update,
/ready api will return 425 http code, if some scrape config still waits for initialization.

* updates docs

* Update app/vmagent/README.md

* renames var

* Update app/vmagent/README.md

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2020-11-04 20:29:18 +02:00
Aliaksandr Valialkin
caeb74f068 app/vmselect: reduce memory usage when query touches big number of time series 2020-11-04 17:04:04 +02:00
Nikolay
e4182dd896
reduces memory usage for vmagent, (#880) 
* reduces memory usage for vmagent,
limits count of droppedTarget, that can be stored for /api/v1/targets page up to 999 items,
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878

* Update app/vmagent/README.md

* Update app/vmagent/README.md

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2020-11-04 17:03:43 +02:00
Aliaksandr Valialkin
600f225cff Revert "docs/vmagent.md: mention about -promscrape.dropOriginalLabels" 
This reverts commit bd81f926a4.
 2020-11-04 11:44:57 +02:00
Aliaksandr Valialkin
bd81f926a4 docs/vmagent.md: mention about -promscrape.dropOriginalLabels 2020-11-04 11:16:33 +02:00
Aliaksandr Valialkin
e0a91ef163 app/vmagent/remotewrite: drop packets only on 409 status code, since there are other valid 4xx status codes, which shouldnt result in packet drop 2020-11-03 14:25:10 +02:00
Aliaksandr Valialkin
5e75c389e6 app/vmselect/promql: allow dropping trailing sample only for default_rollup function 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/850
 2020-11-02 02:10:59 +02:00
Aliaksandr Valialkin
0b35da159c app/vmagent/remotewrite: drop packets if remote storage returns 4xx status code 
This makes consistent the behaviour with Prometheus.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/873
 2020-11-02 00:45:09 +02:00
Aliaksandr Valialkin
cb71af216a app/vmselect/promql: go fmt 2020-11-02 00:15:29 +02:00
Aliaksandr Valialkin
daacbc7e34 app/vmselect/promql: do not drop trailing datapoints for instant queries 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
 2020-11-02 00:12:37 +02:00
Aliaksandr Valialkin
e277c3d07b lib/promscrape: add stream parse mode for efficient scraping of targets that expose millions of metrics 2020-11-01 23:35:06 +02:00
Aliaksandr Valialkin
b7638f04a7 app/vmagent: expose /api/v1/targets page according to https://prometheus.io/docs/prometheus/latest/querying/api/#targets 
This page is exposed by vmagent and by a single-node VictoriaMetrics

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/643
 2020-11-01 23:35:06 +02:00
Aliaksandr Valialkin
c539494b36 app/vmselect/promql: allow passing optional third argument to topk_* and bottomk_* functions in order to obtain sum of time series outside top/bottom K 2020-11-01 23:35:06 +02:00
Roman Khavronenko
f0bdc5716e
vmalert: skip automatically added labels on alerts restore (#871) 
Label `alertgroup` was introduced in #611 and automatically added to generated
time series. By mistake, this new label wasn't correctly purged on restore event
and affected alert's ID uniqueness. This commit removes `alertgroup` label
in restore function.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/870
 2020-10-30 08:18:20 +00:00
kreedom
a51a7b2a20
vmbackup fix panic when no origin fs given (#859) 
* use fsnil when no origin fs
 2020-10-28 20:09:10 +02:00
Aliaksandr Valialkin
5bfd4e6218 app/vmstorage: support for -retentionPeriod smaller than one month 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/173
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/17
 2020-10-20 14:31:44 +03:00
kreedom
ef77120170
vmalert - add dryRun (#842) 
vmalert: add `dryRun` flag for rules validation without running the service
 2020-10-20 08:15:21 +01:00
Seva Poliakov
b3f3c078e5 Fix typo in vnrestore readme 2020-10-18 15:41:39 +03:00
Aliaksandr Valialkin
28353e48ca app/vmselect/promql: an attempt to improve heuristics for dropping trailing data points in time series 
Now trailing data points are additionally dropped for time series with a single raw sample

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
 2020-10-17 10:44:34 +03:00
Roman Khavronenko
bc42b5598f
vmalert: update docs to highlight the state restore requirements; (#833) 
Address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/830
 2020-10-13 18:32:43 +03:00
Aliaksandr Valialkin
a2b9476897 app/vmselect/promql: return a single time series at max from absent() function like Prometheus does 2020-10-13 15:56:04 +03:00
Aliaksandr Valialkin
9aa3b65766 app/vmselect/promql: improve time series staleness detection 
This should prevent from double counting for time series at the time when it changes label.
The most common case is in K8S, which changes pod uid label with each new deployment.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
 2020-10-13 12:19:57 +03:00
Aliaksandr Valialkin
d8af290947 app/vmselect/promql: fix mode_over_time calculations 
Previously `mode_over_time` could return garbage due to improper shuffling of input data points.
 2020-10-13 11:58:25 +03:00
Aliaksandr Valialkin
1e27420243 app/vmselect/prometheus: fix golangci-lint warning 2020-10-13 09:36:11 +03:00
Aliaksandr Valialkin
4f16a964e3 app/vmselect: add ability to export data in CSV format via /api/v1/export/csv 2020-10-12 20:08:17 +03:00
Aliaksandr Valialkin
63c4999e06 lib/promscrape: code prettifying after 9bd9f67718 2020-10-12 16:12:36 +03:00
Nikolay Khramchikhin
9bd9f67718
Adds dockerswarm sd (#818) 
* adds dockerswarm service discovery

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/656

 Following roles supported: services, tasks and nodes.
 Basic, token and tls auth supported.
 Added tests for labels generation.

* added unix socket support to discovery utils

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2020-10-12 13:38:21 +03:00
Aliaksandr Valialkin
762c967855 app/vmselect/promql: keep metric name after applying more functions, which dont change time series meaning 
Functions are:

* keep_last_value
* keep_next_value
* interpolate
* running_min
* running_max
* running_avg
* range_min
* range_max
* range_avg
* range_first
* range_last
* range_quantile
* smooth_exponential

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674
 2020-10-12 11:47:06 +03:00
Aliaksandr Valialkin
45f7cdc532 Revert "app/vmselect/promql: remove metric name after applying ceil, floor and round functions in order to be more consistent with Prometheus" 
This reverts commit ac45082216.

Reason for revert: the previous behavior for VictoriaMetrics is easier to understand and use by users -
functions, which don't change the meaning of the time series shouldn't drop metric name.

Now the following functions do not drop metric names:

* ceil
* floor
* round

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674
 2020-10-12 11:40:34 +03:00
Aliaksandr Valialkin
a94825b169 Revert "app/vmselect/promql: remove metric name after applying clamp_min and clamp_max functions in order to be consistent with Prometheus" 
This reverts commit bb61a4769b.

Reason for revert: the previous behavior for VictoriaMetrics is easier to understand and use by users -
functions, which don't change the meaning of the time series shouldn't drop metric name.

Now the following functions do not drop metric name:

* clamp_min
* clamp_max

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674
 2020-10-12 11:38:27 +03:00
Aliaksandr Valialkin
f7d28bddbf Revert "app/vmselect/promql: remove metric name from results of certain rollup functions in order to be consistent with Prometheus" 
This reverts commit e5202a4eae.

Reason for revert: the previous behavior for VictoriaMetrics is easier to understand and use by users -
functions, which don't change the meaning of the time series shouldn't drop metric name.

Now the following functions do not drop metric name:

* max_over_time
* min_over_time
* avg_over_time
* quantile_over_time
* geomean_over_time
* mode_over_time
* holt_winters
* predict_linear

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674
 2020-10-12 11:35:18 +03:00
Aliaksandr Valialkin
b449607181 lib/backup: add MustStop() method for all remote filesystems 2020-10-09 15:32:19 +03:00
Aliaksandr Valialkin
cf5f2874cd lib/backup/fslocal: add FS.MustStop() method for stopping bandwidth limiter 2020-10-09 15:12:03 +03:00
Aliaksandr Valialkin
68f0e00761 app/vmstorage: add vm_rows_added_to_storage_total metric, which shows the total number of rows added to storage since app start 2020-10-09 13:35:48 +03:00
Aliaksandr Valialkin
84227ea2fc app/{vminsert,vmagent}: take into account all the inserted rows before relabeling in vm_rows_inserted_total and vmagent_rows_inserted_total metrics 2020-10-09 13:29:51 +03:00
Aliaksandr Valialkin
f4e8687c88 app/vmalert: accept days, weeks and years in for: part of config like Prometheus does 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/817
 2020-10-08 20:13:15 +03:00
Aliaksandr Valialkin
6105d61d11 docs/vmagent.md: clarify -promscrape.suppressDuplicateScrapeTargetErrors command-line flag usage 2020-10-08 19:24:31 +03:00
Aliaksandr Valialkin
71ea4935de lib/promscrape: add -promscrape.suppressDuplicateScrapeTargetErrors command-line flag in order to suppress duplicate scrape target errors 
Show also original labels for duplicate targets in error message in order to simplify debugging the issue.

Now `/targets` endpoint accepts optional `show_original_labels=1` query arg, which shows original labels for each target.
This may simplify debugging for target relabeling.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/651
 2020-10-08 18:58:30 +03:00
Aliaksandr Valialkin
d423d73251 app/vmalert: do not pring description for all the flags on config errors 
The description is too big to consume by human and it just distracts humans.
 2020-10-08 13:35:57 +03:00
Aliaksandr Valialkin
e9f2e2cbc9 app/vmselect/promql: add missing label filters to binary operands before query execution 
This implements the optimization described at https://utcc.utoronto.ca/~cks/space/blog/sysadmin/PrometheusLabelNonOptimization

See also https://github.com/cortexproject/cortex/issues/3253
 2020-10-07 21:15:09 +03:00
Dmitry Shihovtsev
92e5d89fc9
Fix typos in the vmalert datasource (#814) 
* Fix typos in the vmalert datasource

* Fix typo in the vmalert datasource test
 2020-10-07 17:59:50 +03:00
Aliaksandr Valialkin
af90b3121c app/vmstorage: add -finalMergeDelay command-line flag for configuring the delay before final merge for per-month partitions after no new data is ingested to it 2020-10-07 17:35:44 +03:00
Aliaksandr Valialkin
d16dbfd639 app/vmagent: add a link to https://victoriametrics.github.io/vmagent.html from main page 2020-10-06 15:29:49 +03:00
Aliaksandr Valialkin
cfd720e772 app/victoria-metrics: add a link to https://victoriametrics.github.io/ from main page 2020-10-06 15:29:49 +03:00
Nikolay Khramchikhin
cbe3cf683b
Adds openstack sd (#811) 
* adds openstack service discovery

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/728

 implemented hypervisors and instance discovery with openstack v3 api.
 Added tests for labeling and data parsing.
 Added token refresh.

* Apply suggestions from code review

* Apply suggestions from code review

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2020-10-05 16:45:33 +03:00
Aliaksandr Valialkin
59c26feefa app/vmselect/promql: fill gaps on graphs for range_* and running_* functions 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/806
 2020-10-02 13:59:45 +03:00
Aliaksandr Valialkin
764dc2499f lib/storage: code cleanup after 10f2eedee0 
Remove the code that uses metricIDs caches for the current and the previous hour during metricIDs search,
since this code became unused after implementing per-day inverted index almost a year ago.

While at it, fix a bug, which could prevent from finding time series with names containing dots (aka Graphite-like names
such as `foo.bar.baz`).
 2020-10-01 19:06:23 +03:00
Roman Khavronenko
daa2d1c065
vmalert: make maxIdleConnections configurable for datasource HTTP client (#797) 
Address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/795
 2020-09-30 09:49:45 +03:00
Aliaksandr Valialkin
44bcda81ab app/vmstorage: rename vm_{big|small}_merge_need_free_disk_space to vm_merge_need_free_disk_space 
This simplifies alerting.
 2020-09-29 22:44:19 +03:00
Aliaksandr Valialkin
a9db81c4ab app/vmstorage: add metrics for determining whether background merges need additional disk space to complete 
These metrics are:

* vm_small_merge_need_free_disk_space
* vm_big_merge_need_free_disk_space

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686
 2020-09-29 21:48:33 +03:00
Aliaksandr Valialkin
127537d631 app/vmagent/remotewrite: do not show -remoteWrite.url in logs if -remoteWrite.showURL isn't set 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/773
 2020-09-29 19:49:12 +03:00
Aliaksandr Valialkin
f7636b0342 app/vmselect/graphite: do not substitute path and path. with path.. in /metrics/find/?format=completer output 2020-09-29 18:03:26 +03:00
Aliaksandr Valialkin
7dc67cd883 docs/{vmbackup,vmrestore}: formatting fixes 2020-09-29 13:19:07 +03:00
Aliaksandr Valialkin
efdefbc1cb docs/vmbackup.md: make docs about minio config more prominent 2020-09-29 13:16:04 +03:00
Aliaksandr Valialkin
9945b8c98d docs: improve readability a bit 2020-09-29 13:03:38 +03:00
Nikolay Khramchikhin
1e679f3e0d
update vmbackup/vmrestore README usage (#794) 
* update vmbackup/vmrestore README usage

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/381

with minio and configuration file examples.

* Apply suggestions from code review

Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>

* added backup/restore docs changes

* added example for relabelConfig flag

* Apply suggestions from code review

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
 2020-09-29 12:53:10 +03:00
Aliaksandr Valialkin
96ee276e6e app/vmselect/prometheus: check for errors returned from bufferedwriter.Write 
This makes `make errcheck` happy
 2020-09-29 11:37:01 +03:00
Aliaksandr Valialkin
6fdfc67620 app/vmselect/graphite: properly handle case when /metrics/find finds both leaf and node for the given query=prefix.* 
In this case only node must be returned with stripped dot in the end of id as carbonapi does
 2020-09-29 11:01:59 +03:00
Aliaksandr Valialkin
124f78857b app/{vminsert,vmagent}: improve data ingestion speed over a single connection 
Process data obtianed from a single connection on all the available CPU cores.
 2020-09-28 04:13:08 +03:00
Aliaksandr Valialkin
1b3efccb24 app/vmselect: stop /api/v1/export/* execution if client disconnects 2020-09-27 23:53:13 +03:00
Aliaksandr Valialkin
95688cbfc5 all: add native format for data export and import 
The data can be exported via [/api/v1/export/native](https://victoriametrics.github.io/#how-to-export-data-in-native-format) handler
and imported via [/api/v1/import/native](https://victoriametrics.github.io/#how-to-import-data-in-native-format) handler.
 2020-09-27 19:54:07 +03:00
Aliaksandr Valialkin
c00627c103 app/vminsert: code prettifying 2020-09-26 04:13:18 +03:00
Aliaksandr Valialkin
b6a976b98d app/vmagent: reduce memory usage when importing data via /api/v1/import 
Previously vmagent could use big amounts of RAM when each ingested JSON line
contained many samples.
 2020-09-26 04:10:24 +03:00
Aliaksandr Valialkin
82973f8ae7 Revert "lib/storage: remove unused fetchData arg from BlockRef.MustReadBlock" 
This reverts commit bab6a15ae0.

Reason for revert: the `fetchData` arg is used in cluster branch.
Leaving this arg in master branch makes smaller the diff with cluster branch.
 2020-09-24 22:44:23 +03:00
Aliaksandr Valialkin
bab6a15ae0 lib/storage: remove unused fetchData arg from BlockRef.MustReadBlock 
This arg became unused after 23bdc1f107
 2020-09-24 20:48:40 +03:00
Aliaksandr Valialkin
23bdc1f107 app/vmselect/netstorage: do not spend CPU time on unpacking empty blocks during /api/v1/series calls 2020-09-24 20:18:10 +03:00
Aliaksandr Valialkin
c584aece38 app/vmselect/promql: properly limit implicitly set rollup window to -search.maxStalenessInterval 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/784
 2020-09-23 23:23:59 +03:00
Aliaksandr Valialkin
2985077c35 all: consistently use "%w" formatting in fmt.Errorf for wrapped errors 2020-09-23 22:46:34 +03:00
Aliaksandr Valialkin
27500d7d4c app/vmselect/prometheus: code cleanup after 3ba507000c 2020-09-23 13:04:17 +03:00
Aliaksandr Valialkin
3ba507000c app/vmselect/prometheus: return timestamps from /api/v1/query, which match the time query arg 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/720
 2020-09-23 12:58:48 +03:00
Aliaksandr Valialkin
bed25e3c24 app/vmselect/netstorage: properly pre-allocate space for sbs 2020-09-22 23:49:55 +03:00
Aliaksandr Valialkin
09b0f7c202 app/vmselect/netstorage: release search resources on timeout errors 
Previously these resources weren't released, which could lead to resource leaks.
 2020-09-22 22:57:38 +03:00
Aliaksandr Valialkin
3b1e3a03e0 app/vmselect: make sure the request doesnt wait in pending queue more than the configured timeout 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711
 2020-09-22 01:23:19 +03:00
Aliaksandr Valialkin
94f7d00537 docs/vmagent.md: typo fix 2020-09-21 21:49:22 +03:00
Aliaksandr Valialkin
00b5145c69 app/vmselect/searchutils: fixed tests after 2eb72e09ab 2020-09-21 21:31:38 +03:00
Aliaksandr Valialkin
2eb72e09ab app/vmselect: use time value rounded to seconds if it isnt passed to /api/v1/query 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/720
 2020-09-21 21:24:40 +03:00
Nikolay Khramchikhin
312fead9a2
Add improvements to ec2_sd_discovery (#775) 
* Add improvements to ec2 discovery

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/771

 role_arn support with aws sts
 instance iam_role support
 refreshing temporary tokens

* Apply suggestions from code review

Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>

* changed implementation, removed tests, clean up code

* moved endpoint builder into getEC2APIResponse

Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
 2020-09-21 16:04:15 +03:00
Aliaksandr Valialkin
1e1a27d803 app/vmalert: remove unneeded UTC() call 
UTC() doesn't change the underlying timestamp, so the call isn't needed here
 2020-09-21 15:55:59 +03:00
Roman Khavronenko
5dffc7a553
vmalert: add support for datasource.lookback flag (#779) 
New datasource flag `datasource.lookback` defines how far to look into
past when evaluating queries.

Address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/668
 2020-09-21 15:53:49 +03:00
Roman Khavronenko
82c3bbce34
vmalert: fix the typo in error message (#782) 
The error will be always nil so no sense in printing it.
 2020-09-21 11:34:23 +03:00
Aliaksandr Valialkin
d50165ad59 app/vmagent: increase default value for -remoteWrite.queues from 1 to 4, since it has been appeared that many users hit this limit 2020-09-18 14:21:54 +03:00
Aliaksandr Valialkin
98d1cd0971 app/vmselect/graphite: return proper results /metrics/find?query=foo.*.bar according to Graphite Metrics API 2020-09-18 11:00:00 +03:00
Aliaksandr Valialkin
7a134b0fd7 app/vmstorage: added -forceMergeAuthKey command-line flag for protecting /internal/force_merge endpoint 2020-09-17 14:21:53 +03:00
Aliaksandr Valialkin
1f33dd717f lib/storage: add /internal/force_merge handler for running forced compactions on historical per-month partitions 
This may be useful for freeing up storage space after time series deletion.

See https://victoriametrics.github.io/#force-merge for more details.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686
 2020-09-17 12:20:40 +03:00
Aliaksandr Valialkin
ab53cb6f7b app/vmagent: substitute -remoteWrite.url with secret-url value in logs, since it may contain sensitive info such as passwords or auth tokens 
Pass `-remoteWrite.showURL` command-line flag in order to see real `-remoteWrite.url` values in logs and at `/metrics` page.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/773
 2020-09-16 22:36:25 +03:00
Aliaksandr Valialkin
9f79bcf64a app/vmselect: improve description for -search.maxQueryDuration 2020-09-16 21:15:41 +03:00
Aliaksandr Valialkin
1fec47a289 app/vmselect/netstorage: reduce memory usage when the time range from query touches big number of samples per each time series 2020-09-15 21:08:28 +03:00
Aliaksandr Valialkin
8c3d7c1a59 app/vmselect: typo fix in -search.maxStalenessInterval description 2020-09-15 14:24:27 +03:00
Aliaksandr Valialkin
0e533d1a9c app/vmselect/promql: support composite durations like Prometheus 2.21 does 
The following durations are supported now: `1h5m35s` or `1s543ms`

See https://github.com/prometheus/prometheus/releases/tag/v2.21.0
and https://github.com/prometheus/prometheus/pull/7713
 2020-09-11 23:39:13 +03:00
Roman Khavronenko
6ad6480400
vmalert: add Group name as label to generated alerts and timeseries (#761) 
Solves #611
 2020-09-11 20:52:56 +01:00
Roman Khavronenko
4cdffb04a4
vmalert: update groups on config reload only if changes detected (#759) 
On config reload event `vmalert` reloads configuration for every group. While
it works for simple configurations, the more complex and heavy installations may
suffer from frequent config reloads.
The change introduces the `checksum` field for every group and is set to md5 hash
of yaml configuration. The checksum will change if on any change to group
definition like rules order or annotation change. Comparing the `checksum` field
on config reload event helps to detect if group should be updated.
The groups update is now done concurrently, so reload duration will be limited by
the slowest group now.

Partially solves #691 by improving config reload speed.
 2020-09-11 20:14:30 +01:00
Aliaksandr Valialkin
ca856284e4 app/vmagent: allow setting multiple identical -remoteWrite.url values 
This may be useful when each url is authenticated via different `-remoteWrite.basicAuth.username`.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/755
 2020-09-11 15:17:22 +03:00
Aliaksandr Valialkin
a2f647d142 app/vmselect/prometheus: typo fix in the description for -search.latencyOffset command-line flag 2020-09-11 14:16:46 +03:00
Aliaksandr Valialkin
2380e9b017 app/{vminsert,vmagent}: allow passing timestamp via timestamp query arg when ingesting data to /api/v1/import/prometheus 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/750
 2020-09-11 13:27:14 +03:00
Aliaksandr Valialkin
f0005c3007 app/vmselect: move Deadline from netstorage to searchutils 
This removes dependency on netstorage from searchutils.
 2020-09-11 13:27:13 +03:00
Aliaksandr Valialkin
2114179e19 app/vmselect: substitute inf values at smooth_exponential with the previous values 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/757
 2020-09-11 12:24:14 +03:00
Aliaksandr Valialkin
204ec415b4 app/vmselect: skip infinite values when calculating smooth_exponential 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/757
 2020-09-11 11:29:58 +03:00
Aliaksandr Valialkin
8a8b5a73d3 app/vmselect/graphite: typo fix in label name for vm_request_duration_seconds metric 2020-09-11 01:58:28 +03:00
Aliaksandr Valialkin
f6bc608e86 app/vmselect: initial implementation of Graphite Metrics API 
See https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api
 2020-09-11 00:30:01 +03:00
Aliaksandr Valialkin
9d8fdff6c5 lib/storage: reuse timestamp blocks for adjancent metric blocks with identical timestamps 
This should reduce disk space usage when scraping targets containing metrics with identical names
such as `node_cpu_seconds_total`, histograms, quantiles, etc.

Expose `vm_timestamps_blocks_merged_total` and `vm_timestamps_bytes_saved_total` metrics for monitoring
the effectiveness of timestamp blocks merging.
 2020-09-09 23:59:32 +03:00
Aliaksandr Valialkin
d7c04db1fc docs: sync docs for vmalert, vmauth, vmbackup and vmrestore 2020-09-09 21:10:34 +03:00
Aliaksandr Valialkin
11eaa37111 docs/vmagent.md: clarified the case when -remoteWrite.queues must be tuned 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/745
 2020-09-08 20:15:27 +03:00
Aliaksandr Valialkin
62919eaf7e app/vmselect/promql: go fmt 2020-09-08 15:19:59 +03:00
Aliaksandr Valialkin
e6da63dffe app/vmselect/promql: adjust integrate() calculations to be more similar to calculations from InfluxDB: attempt #2 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/701
 2020-09-08 14:35:50 +03:00
Aliaksandr Valialkin
8e85b56737 app/vmselect/promql: adjust integrate() calculations to be more similar to calculations from InfluxDB 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/701
 2020-09-08 14:23:39 +03:00
Aliaksandr Valialkin
c0343a661b app/vmselect/promql: increase floating point calculations accuracy by dividing by 1e3 instead of multiplying by 1e-3 2020-09-08 14:00:47 +03:00
Aliaksandr Valialkin
f4e7e5fb90 app/vmselect/promql: add count_le_over_time(m[d], le) and count_gt_over_time(m[d], gt) functions 
These functions returns the number of raw samples that don't exceed `le` or are bigger than `gt`.
These functions are complement to already existing `share_le_over_time(m[d], le)` and `share_gt_over_time(m[d], gt)`.
 2020-09-03 15:29:10 +03:00
Aliaksandr Valialkin
e706e59d49 app/vmselect: unconditionally align time range boundaries to step for subqueries as Prometheus does 2020-09-03 13:29:50 +03:00
Aliaksandr Valialkin
ddabc13796 app/vmagent: properly flush big blocks of data 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/741

Thanks to @IceRain00 for the investigation and initial attempt to fix the issue
at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/742
 2020-09-03 12:12:39 +03:00
Aliaksandr Valialkin
7a839b461f app/vmagent: fix data race when accessing writeRequest.lastFlushTime 2020-09-03 12:12:37 +03:00
Nikolay Khramchikhin
764b3d4fda
changed vmalert behaviour (#738) 
* VMAlert start with empty rules dir

There are some applications (operator for instance), that generates alerts configuration at runtime
and vmalert must start correctly without rules to support this behaviour.
Later application will add rules files and send SIGHUP to vmalert,
which will trigger reading rules files and start rules exectuion.

Removing rules files with SIGHUP signal must stop rules execution and
vmalert will wait for new rules.

* imports sorted

* added test cases for empty rules, removed blank line

* fixed imports conflict

* updated tests
 2020-09-03 11:04:42 +03:00
Aliaksandr Valialkin
5f16ceb294 app/vmalert: imrovements over 3f932c2db1 2020-09-03 01:00:55 +03:00
DexterZhang
3f932c2db1
feat: spread load of rule evaluation by group when starting new groups (#724) 
* feat: spread load of rule evaluation by group when starting new groups

* review: reduce the resulting diff.

* Update app/vmalert/group.go

Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
 2020-09-03 00:58:54 +03:00
Aliaksandr Valialkin
f41b36bb9a app/{vminsert,vmagent}: allow adding extra labels when importing data via Prometheus, CSV and JSON line formats 
Extra labels may be added to the imported data by passing `extra_label=name=value` query args.
Multiple query args may be passed in order to add multiple extra labels.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/719
 2020-09-02 19:43:21 +03:00
Aliaksandr Valialkin
5af777469a app/vmagent: log unsuccessful attempt number when sending data to -remoteWrite.url 2020-08-30 21:40:22 +03:00
Aliaksandr Valialkin
2149733bd2 app/vmagent: apply sane limits to -remoteWrite.queues 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/707
 2020-08-30 21:25:37 +03:00
Roman Khavronenko
10601bc652
vmalert: update -rule flag description to enforce quotes using (#709) 
Description for `-rule` flag uses as example specific chars like asterisks
which could be interpreted wrong by different shells. To avoid this, description
now contains quoted flag values.

See also #708
 2020-08-20 22:36:38 +01:00
Roman Khavronenko
f5f59896ec
lib/decimal: rename significant decimal digits to significant figures (#698) 
The previous notion was inconsistent with what `decimal.Round` does.
According to [wiki](https://en.wikipedia.org/wiki/Significant_figures) rounding
applied to all significant figures, not just decimal ones.
 2020-08-16 17:21:35 +03:00
Aliaksandr Valialkin
147c35ebd4 all: allow using KB, MB, GB, KiB, MiB and GiB suffixes in command-line flag values related to byte sizes or byte rates 2020-08-16 17:05:52 +03:00
Aliaksandr Valialkin
7d89fafe1a app/vmselect/promql: allow passing multiple args to aggregate functions such as avg(q1, q2, q3) 2020-08-15 01:15:09 +03:00
Aliaksandr Valialkin
cd96248480 docs/vmagent.md: mention that gaps in remote storage may appear if vmagent cannot keep up with data ingestion 2020-08-14 20:47:57 +03:00
Aliaksandr Valialkin
184670fb9b docs: update docs 2020-08-14 19:13:42 +03:00
Aliaksandr Valialkin
576da0fe46 app/{vminsert,vmagent}: improve documentation for -influxListenAddr command-line flag 2020-08-14 18:04:44 +03:00
Aliaksandr Valialkin
c402265e88 all: support %{ENV_VAR} placeholders in yaml configs in all the vm* components 
Such placeholders are substituted by the corresponding environment variable values.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/583
 2020-08-13 17:15:25 +03:00
Aliaksandr Valialkin
ef7e2af8f5 app: respect CPU limits set via cgroups 
Update GOMAXPROCS to limits set via cgroups. This should reduce CPU trashing and reduce memory usage
for cases when VictoriaMetrics components run in containers with CPU limits.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685
 2020-08-11 22:59:19 +03:00
Aliaksandr Valialkin
e969ef2639 app/vmselect: reduce memory usage when exporting time series with big number of samples via /api/v1/export if max_rows_per_line is set to non-zero value 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685
 2020-08-10 20:57:36 +03:00
Aliaksandr Valialkin
8adba82c02 app/vmselect/netstorage: vary batch size for data unpacking depending on the available CPU cores 
This should reduce contention on the channel with unpack work for systems with high number of CPU cores
 2020-08-10 15:16:42 +03:00
Aliaksandr Valialkin
f3d33e23c9 app/vmstorage: improve error logging when the request times out 2020-08-10 13:23:26 +03:00
Roman Khavronenko
0be5b09fb4
app/vmalert: extend metrics set exported by vmalert #573 (#654) 
* app/vmalert: extend metrics set exported by `vmalert` #573

New metrics were added to improve observability:
+ vmalert_alerts_pending{alertname, group} - number of pending alerts per group
per alert;
+ vmalert_alerts_acitve{alertname, group} - number of active alerts per group
per alert;
+ vmalert_alerts_error{alertname, group} - is 1 if alertname ended up with error
during prev execution, is 0 if no errors happened;
+ vmalert_recording_rules_error{recording, group} - is 1 if recording rule
 ended up with error during prev execution, is 0 if no errors happened;
* vmalert_iteration_total{group, file} - now contains group and file name labels.
This should improve control over specific groups;
* vmalert_iteration_duration_seconds{group, file} - now contains group and file name labels. This should improve control over specific groups;

Some collisions for alerts and recording rules are possible, because neither
group name nor alert/recording rule name are unique for compatibility reasons.

Commit contains list of TODOs for Unregistering metrics since groups and rules
are ephemeral and could be removed without application restart. In order to
unlock Unregistering feature corresponding PR was filed - https://github.com/VictoriaMetrics/metrics/pull/13

* app/vmalert: extend metrics set exported by `vmalert` #573

The changes are following:
* add an ID label to rules metrics, since `name` collisions within one group is
a common case - see the k8s example alerts;
* supports metrics unregistering on rule updates. Consider the case when one rule
was added or removed from the group, or the whole group was added or removed.

The change depends on https://github.com/VictoriaMetrics/metrics/pull/16
where race condition for Unregister method was fixed.
 2020-08-09 09:41:29 +03:00
ofen
81746d14b9
401 Unauthorize HTTP error added (#681) 
401 Unauthorize HTTP error added to trigger browser credentials pop-up promt [RFC 7235 https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication]
 2020-08-09 09:38:41 +03:00
Aliaksandr Valialkin
716754fae6 app/vmselect/promql: properly handle -n^m like Prometheus does 
`-n^m` must be handled as `-(n^m)` instead of `(-n)^m`.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/675
 2020-08-07 07:42:18 +03:00
Aliaksandr Valialkin
bb61a4769b app/vmselect/promql: remove metric name after applying clamp_min and clamp_max functions in order to be consistent with Prometheus 
This improves VictoriaMetrics score at https://promlabs.com/promql-compliance-test-results-victoriametrics/
 2020-08-06 23:42:37 +03:00
Aliaksandr Valialkin
ac45082216 app/vmselect/promql: remove metric name after applying ceil, floor and round functions in order to be more consistent with Prometheus 
This improves VictoriaMetrics score at https://promlabs.com/promql-compliance-test-results-victoriametrics/
 2020-08-06 23:34:37 +03:00
Aliaksandr Valialkin
e5202a4eae app/vmselect/promql: remove metric name from results of certain rollup functions in order to be consistent with Prometheus 
Rollup functions:

  - avg_over_time
  - min_over_time
  - max_over_time
  - quantile_over_time

This improves VictoriaMetrics results at https://promlabs.com/promql-compliance-test-results-victoriametrics/
 2020-08-06 23:29:13 +03:00
Aliaksandr Valialkin
68e4f40a72 app/vmselect: properly handle PromQL queries like scalar1 < metric < scalar2 like Prometheus does 
This fixes some cases from https://promlabs.com/promql-compliance-test-results-victoriametrics/
 2020-08-06 23:21:03 +03:00
Aliaksandr Valialkin
bc8381613d app/vmselect: reduce memory allocations by pre-allocatin memory for time series map and for a list of time series names 2020-08-06 19:17:58 +03:00
Aliaksandr Valialkin
3f85c06b65 app/vmselect/netstorage: reduce CPU contention when upacking time series blocks by unpacking batches of such blocks instead of a single block 
This should improve query performance on systems with big number of CPU cores (16 and more)
 2020-08-06 17:50:17 +03:00
Aliaksandr Valialkin
d20c2156e4 app/vmselect/netstorage: reduce contention on unpackworkCh and timeseriesWorkCh for multi-CPU system by providing more capacity for these chans 2020-08-06 17:22:48 +03:00
Aliaksandr Valialkin
dbbdfbe7ee app/vmstorage: rename vm_cache_size_entries{type="storage/prefetchedMetricIDs"} to vm_cache_entries{type="storage/prefetchedMetricIDs"} to be consistent with other vm_cache_entries metrics 2020-08-06 16:34:24 +03:00
Aliaksandr Valialkin
8f16388428 lib/storage: limit the number of concurrent calls to storage.searchTSIDs to GOMAXPROCS*2 
This should limit the maximum memory usage and reduce CPU trashing on vmstorage
when multiple heavy queries are executed.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
 2020-08-05 18:30:07 +03:00
Aliaksandr Valialkin
c25b0c2cd5 app/vmagent: tune http client for sending data to remote storage in order to disable closing keep-alive connections 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/663
 2020-08-04 21:00:29 +03:00
Aliaksandr Valialkin
5d0c37bec0 app/vmselect: use warning level instead of info level for logging slow queries that take longer than -search.logSlowQueryDuration 2020-08-04 20:25:35 +03:00
Aliaksandr Valialkin
e6eee2bebf app/vmselect/promql: add zscore-related functions: zscore_over_time(m[d]) and zscore(q) by (...) 2020-08-03 21:52:18 +03:00
Aliaksandr Valialkin
509d12643b app/vmselect: show X-Forwarded-For contents on /api/v1/status/active_queries page 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/659
 2020-07-31 20:05:18 +03:00
Aliaksandr Valialkin
d01f3c1943 all: add mssing APP_NAME to vm*-GOARCH builds 2020-07-31 13:42:18 +03:00
Aliaksandr Valialkin
3f498cf2dc docs/{vmagent,vmalert}: add instruction on how to build for ARM 2020-07-31 09:25:22 +03:00
Aliaksandr Valialkin
f0c678c41b app/vmselect: do not adjust start and end query args passed to /api/v1/query_range when -search.disableCache command-line flag is set 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/563
 2020-07-30 23:14:37 +03:00
Aliaksandr Valialkin
e7959094f6 lib/storage: remove prioritizing of merging small parts over merging big parts, since it doesn't work as expected 
The prioritizing could lead to big merge starvation, which could end up in too big number of parts that must be merged into big parts.

Multiple big merges may be initiated after the migration from v1.39.0 or v1.39.1. It is OK - these merges should be finished soon,
which should return CPU and disk IO usage to normal levels.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/618
 2020-07-30 19:57:27 +03:00
Aliaksandr Valialkin
328b52e5ff app/vmselect/promql: return non-empty value from rate_over_sum(m[d]) even if a single data point is located in the given [d] window 
Just divide the data point value by the window duration in this case.
 2020-07-29 12:37:58 +03:00
Aliaksandr Valialkin
700737c181 app/vmselect/promql: remove rollupFuncArg.realPrevValue handling, since the corner case in increase() is handled in another way now 
See e00cfc854d for the approach used now.
 2020-07-29 12:37:58 +03:00
Aliaksandr Valialkin
2f735f112d app/vmselect/promql: fill gaps with 0 in rate_over_sum response when the last value before the selected time window isnt empty 2020-07-29 12:37:58 +03:00
Aliaksandr Valialkin
0f63da3698 app/{vmagent,vminsert}: properly preserve db tag from query string passed to Influx line protocol query 
Previously `db` tag from the query string wasn't added to metrics after encountering `db` tag in the Influx line

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/653
 2020-07-28 21:25:19 +03:00
Aliaksandr Valialkin
62ed38c6f0 app/vmagent/remotewrite: add missing resp.Body.Close() after pushing data to remote storage 
Missing body close could disable HTTP keep-alive connections.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/653
 2020-07-28 21:00:15 +03:00
Aliaksandr Valialkin
79c30cf4cb app/vmselect: show query origin (aka remote_addr or client address) on the /api/v1/status/active_queries page for every query 2020-07-28 15:13:08 +03:00
Roman Khavronenko
2f1e7298ce
app/vmalert: support external.label to specify global labelset for all rules #622 (#652) 
`external.label` flag supposed to help to distinguish alert or recording rules
source in situations when more than one `vmalert` runs for the same datasource
or AlertManager.
 2020-07-28 14:20:31 +03:00
Aliaksandr Valialkin
0da202023b app/vmselect/promql: return empty values from group() if all the time series have no values at the given timestamp 
This aligns `group()` behaviour to Prometheus
 2020-07-28 13:40:11 +03:00
Aliaksandr Valialkin
cb8c6908dc app/vmagent/remotewrite: create new request on failure to send a block of data to remote storage 
Previously the request body was already consumed before the retry, so this led to the following error:

    http: ContentLength=... with Body length 0
 2020-07-27 17:32:46 +03:00
Aliaksandr Valialkin
894dcb7c1c app/vmselect/promql: improve further the accuracy of buckets_limit() function 
The accuracy is increased by mergin the smallest bucket with the smallest adjacent bucket.
 2020-07-26 12:10:13 +03:00
Aliaksandr Valialkin
215eba0b82 app/vminsert: flush bufs if needed after the current row is added 
Previously the data for the added row could be overwritten by the flush
before the row addition is complete.
 2020-07-26 12:10:11 +03:00
Aliaksandr Valialkin
edb1eca6f1 app/vmselect/promql: avoid dropping inf bucket in buckets_limit 
The `le="inf"` bucket must be preserved in order to maintain the maximum level of accuracy.
 2020-07-25 17:00:36 +03:00
Aliaksandr Valialkin
97b6f5d223 app/vmselect/promql: optimize buckets_limit(k, buckets) for big number of buckets 2020-07-25 13:24:03 +03:00
Aliaksandr Valialkin
a090627059 app/vminsert: limit memory usage when ingesting data in big packets 2020-07-24 23:32:40 +03:00
Aliaksandr Valialkin
bb161497cf app/vmselect/promql: improve the accuracy of buckets_limit(k, buckets) function 
Now it properly merges the bucket with the previous bucket after deletion.
 2020-07-24 17:07:49 +03:00
Aliaksandr Valialkin
994fa2f3bf app/vmselect/promql: add buckets_limit(k, buckets) function, which limits the number of buckets per time series to k 
This function works with both Prometheus-style and VictoriaMetrics-style buckets.
The function removes buckets with the lowest values in order to reserve the highest precision.
The function is useful for building heatmaps in Grafana from too big number of buckets.
 2020-07-24 16:13:53 +03:00
Aliaksandr Valialkin
e151c5c644 app/vmselect: fix tests for rate_over_sum 2020-07-24 02:35:28 +03:00
Aliaksandr Valialkin
3107c633e3 app/vmselect/promql: typo fix after 3e557c9861 2020-07-24 02:15:58 +03:00
Aliaksandr Valialkin
3e557c9861 app/vmselect/promql: add rate_over_sum(m[d]) function to MetricsQL, which returns rate over sum of m values over d duration 
Something like `sum_over_time(m[d]) / d`, but more accurate.
 2020-07-24 01:17:42 +03:00
Aliaksandr Valialkin
b1f6843bd0 app/vmselect/promql: allow setting [d] window smaller than the interval between raw points for avg_over_time 
This makes `avg_over_time` behavior consistent with `sum_over_time` and `count_over_time` behaviors.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/636
 2020-07-23 22:25:43 +03:00
Aliaksandr Valialkin
039c9d2441 lib/storage: respect -search.maxQueryDuration when searching for time series in inverted index 
Previously the time spent on inverted index search could exceed the configured `-search.maxQueryDuration`.
This commit stops searching in inverted index on query timeout.
 2020-07-23 21:21:42 +03:00
Aliaksandr Valialkin
2a45871823 lib/storage: add more fine-grained pace limiting for search 2020-07-23 19:26:08 +03:00
Aliaksandr Valialkin
461481fbdf app/vmselect/netstorage: protect from too smart compiler, which may break memory usage optimization in ProcessSearchQuery 2020-07-23 17:54:01 +03:00
Aliaksandr Valialkin
4c8b49b193 app/vminsert: export vm_relabel_metrics_dropped_total metric that shows the number of metrics dropped due to relabeling 2020-07-23 14:57:53 +03:00
Aliaksandr Valialkin
e79de9774b app/vmselect: typo fix after 34563916f7 2020-07-23 14:12:28 +03:00
Aliaksandr Valialkin
34563916f7 app/vmselect: reduce memory usage when querying big number of time series with long labels 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646
 2020-07-23 13:53:52 +03:00
Aliaksandr Valialkin
9257eee982 app/vminsert: do not call ApplyRelabeling function if relabeling is disabled 
This should reduce CPU usage a bit when `-relabelConfig` isn't set
 2020-07-23 13:39:44 +03:00
Aliaksandr Valialkin
6f05c4d351 lib/storage: improve prioritizing of data ingestion over querying 
Prioritize also small merges over big merges.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
 2020-07-23 13:23:36 +03:00
Aliaksandr Valialkin
2f612e0c67 app/vminsert: fix relabeling for metrics ingested via Influx line protocol 
Previously the enabled relabeling with `-relabelConfig` command-line flag could result in missing labels
if a single Influx line protocol message contains multiple field values.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/638
 2020-07-23 13:23:14 +03:00
Aliaksandr Valialkin
4f526cc816 app/vmselect/prometheus: support d, w and y suffixes for durations passed to step in /api/v1/query_range like Prometheus does 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/641
 2020-07-22 16:26:18 +03:00
Aliaksandr Valialkin
dfb113f175 app/vmselect/netstorage: reduce memory allocations when unpacking time series data by using a pool for unpackWork entries 
This should slightly reduce load on GC when processing queries that touch big number of time series.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646 according to the provided memory profile there.
 2020-07-22 15:03:57 +03:00
Aliaksandr Valialkin
31ae5911a8 app/vmagent: add -remoteWrite.decimalPlaces command-line flag, which may be used for reducing disk space usage on the remote storage 2020-07-21 21:55:32 +03:00
Aliaksandr Valialkin
caa2952aa6 app/vmselect: take into account the time spent in wait queue before query execution as time spent on the query 2020-07-21 19:00:09 +03:00
Aliaksandr Valialkin
e00cfc854d app/vmselect/promql: skip the first value in time series passed to increase() if it exceeds by more than 10x the delta between the next value and the first value 
This should prvent from inflated `increase()` results for time series that start from big initial values.
Such cases may occur when a label value changes in a metric without counter reset.
 2020-07-21 17:24:10 +03:00
Aliaksandr Valialkin
b9c8f6bf34 app/vmselect: log the total available memory for concurrent requests on not enough memory errors 
This should simplify root cause analysis
 2020-07-20 19:51:40 +03:00
Aliaksandr Valialkin
ad6290953c app/vmagent: add -remoteWrite.proxyURL command-line option 
This option allows writing data to `-remoteWrite.url` via http, https or socks5 proxy.
This is similar to `proxy_url` option in `remote_write` section of Prometheus.
See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write
 2020-07-20 19:28:49 +03:00
Aliaksandr Valialkin
efcbb51968 docs/vmagent.md: sync with app/vmagent/README.md 2020-07-20 17:08:34 +03:00
Roman Khavronenko
ed0df37ee7
app/vmagent: mention grafana dashboard in README (#639) 2020-07-20 17:07:27 +03:00
Aliaksandr Valialkin
11be704109 app/vmagent/remotewrite: allow passing empty -remoteWrite.urlRelabelConfig entries 2020-07-20 15:49:27 +03:00
Aliaksandr Valialkin
5a4675c528 app/vmselect/prometheus: do not return time series with empty list of datapoints from /api/v1/query_range 
This matches Prometheus behaviour.

This should fix https://github.com/jacksontj/promxy/issues/329
 2020-07-20 15:31:21 +03:00
Aliaksandr Valialkin
ecb1b2564a app/vmselect/promql: add mode() aggregate function 2020-07-20 15:31:20 +03:00
Aliaksandr Valialkin
b35cb293f5 lib/httpserver: log remote address in error message from httpserver.Errorf 
This should improve detection of the root cause of errors.
Thanks to Anant for the idea.
 2020-07-20 14:11:22 +03:00
Aliaksandr Valialkin
1c641037e8 app/vmselect/promql: add mode_over_time(m[d]) function 
See https://en.wikipedia.org/wiki/Mode_(statistics) and https://stackoverflow.com/questions/61134078/promql-query-to-return-the-value-from-a-range-vector-which-occurs-maximum-no-of
 2020-07-17 18:28:45 +03:00
Aliaksandr Valialkin
6b5ad535ae app/vmselect/promql: optimize group(rollup(m)) calculations 2020-07-17 16:47:16 +03:00
Aliaksandr Valialkin
8949d65ad1 app/vmselect/promql: check that any() doesn't touch metric name 2020-07-17 16:23:21 +03:00
Aliaksandr Valialkin
aa5d88055d app/vmselect/promql: add group() aggregate function to MetricsQL 
This function has been added in Prometheus 2.20. See https://github.com/prometheus/prometheus/pull/7480
 2020-07-17 15:17:55 +03:00
Aliaksandr Valialkin
df01836818 app/vmselect/promql: keep all labels for time series from any() call 2020-07-17 15:17:54 +03:00
Aliaksandr Valialkin
8c14ca93fa app/vminsert/influx: properly handle the case when certain labels with empty values are removed by ApplyRelabeling() call 
Previously this could lead to `out of range` panic
 2020-07-17 00:07:06 +03:00
Aliaksandr Valialkin
e4e1cd1de2 app/vmselect: fix nil pointer dereference panic when unsuccessfully querying vmstorage 2020-07-16 19:15:43 +03:00
Aliaksandr Valialkin
ed7580ad22 app/vmalert: consistently use "%w" instead of "%s" in fmt.Errorf when wrapping errors 2020-07-15 13:56:47 +03:00
Aliaksandr Valialkin
328814ee60 docs/vmagent.md: make filtering rules for init container pods less confusing 2020-07-14 20:32:47 +03:00
Aliaksandr Valialkin
b442a42d8e app/vmagent/remotewrite: return proper value from tssRelabelPool.New 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/599
 2020-07-14 14:29:20 +03:00
Aliaksandr Valialkin
4081e2295e app/{vminsert,vmagent}: add -influxSkipMeasurement command-line flag for using field name as metric name 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/626
 2020-07-14 14:17:24 +03:00
Aliaksandr Valialkin
25f80d320b app/vmselect/prometheus: do not adjust last points in time series with timestamps exceeding the current time 
Such timestamps usually mean that the query contains `offset`.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/625
 2020-07-14 12:52:16 +03:00
Aliaksandr Valialkin
55d83e777d app/vmselect/prometheus: minimize the diff for the change 1033dc7e2a over 619b0a25c9 2020-07-13 21:40:38 +03:00
faceair
1033dc7e2a
fix empty response template (#617) 2020-07-13 21:31:19 +03:00
Aliaksandr Valialkin
619b0a25c9 docs/vmagent.md: sync with app/vmagent/README.md 2020-07-13 21:25:11 +03:00
ofen
666c795b98
Update README.md (#621) 
Troubleshooting section updated to help out with duplicate targets detection
 2020-07-13 21:18:54 +03:00
Aliaksandr Valialkin
a730b3f6a1 app/vmagent: fix data race when multiple -remoteWrite.urlRelabelConfig options are set 
Previously multiple goroutines could access remoteWriteCtx.tss concurrently, which could lead to data race
and improper relabeling. Now each goroutine has its own copy of tss during relabeling.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/599
 2020-07-10 15:16:59 +03:00
Aliaksandr Valialkin
508ad46e0e app/vmagent/remotewrite: typo fix in -remoteWrite.showURL help message 2020-07-10 14:07:08 +03:00
Aliaksandr Valialkin
cba820e390 app/{vminsert,vmagent}: add ability to import data in Prometheus exposition format via /api/v1/import/prometheus 2020-07-10 12:14:07 +03:00
Aliaksandr Valialkin
9c350bc20d app/vmselect/promql: add missing tests for ifnot binary operation 2020-07-09 13:24:06 +03:00
Aliaksandr Valialkin
256fd9a87e app/vmselect/promql: refactor implementations for and and unless binary operations, so they are closer to or implementation 2020-07-09 13:05:55 +03:00
Aliaksandr Valialkin
2d9b3ad5b3 app/vmselect/promql/active_queries.go: simplify code a bit by inlining getNextActiveQueryID function 2020-07-09 11:18:30 +03:00
Aliaksandr Valialkin
742da690f4 app/vmselect: add /api/v1/status/active_queries page with the list of currently running queries 
This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/598

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/575
 2020-07-08 18:55:38 +03:00
DexterZhang
99f54e44ff
feat(vmselect): add current running query list, add ability for getting the running query info and killing running query for master branch (#598) 2020-07-08 18:52:55 +03:00
Aliaksandr Valialkin
6daa5f7500 lib/storage: prioritize data ingestion over heavy queries 
Heavy queries could result in the lack of CPU resources for processing the current data ingestion stream.
Prevent this by delaying queries' execution until free resources are available for data ingestion.

Expose `vm_search_delays_total` metric, which may be used in for alerting when there is no enough CPU resources
for data ingestion and/or for executing heavy queries.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291
 2020-07-05 19:42:05 +03:00
Roman Khavronenko
703def4b2e
app/vmalert: add retries to remotewrite (#605) 
* app/vmalert: add retries to remotewrite

Remotewrite pkg now does limited number of retries if write request failed.
This suppose to make vmalert state persisting more reliable.

New metrics were added to remotewrite in order to track rows/bytes sent/dropped.

defaultFlushInterval was increased from 1s to 5s for sanity reasons.

* fix

* wip

* wip

* wip

* fix bits alignment bug for 32-bit systems

* fix mistakenly dropped field
 2020-07-05 18:46:52 +03:00
Aliaksandr Valialkin
de137aef98 app/victoria-metrics: fix tests after the commit acf828a759 2020-07-05 18:24:41 +03:00
Aliaksandr Valialkin
acf828a759 app/vmselect/prometheus: small fixes on top of 8bb762124a 2020-07-05 18:17:06 +03:00
faceair
8bb762124a
fix adjust last points avoid influence earlier value (#606) 2020-07-05 17:56:54 +03:00
Ween
d59cdbe90c
[VMAlert] Fix error log when remoteWrite queue size is full (#602) 
* Fix Auto metrics relabeled errors

* Finalize auto-genenated  Labels

* Fix Test Errors

* fix error logs when queue is full

Co-authored-by: xinyulong <xinyulong@kuaishou.com>
 2020-07-03 16:49:37 +03:00
Aliaksandr Valialkin
0b2086b7a5 app/vminsert: prevent from adding and/or selecting labels with empty values 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/600
 2020-07-02 23:14:11 +03:00
Aliaksandr Valialkin
8f628cd805 app/victoria-metrics: removed debug log message when -selfScrapeInterval is set 2020-07-02 20:39:41 +03:00
Aliaksandr Valialkin
91b3482894 app/vminsert: add ability to apply relabeling to all the incoming metrics if -relabelConfig command-line arg points to a file with a list of relabel_config entries 
See https://victoriametrics.github.io/#relabeling
 2020-07-02 20:39:28 +03:00
Aliaksandr Valialkin
e5500bfcf2 all: typo fix: exptected -> expected 2020-07-02 18:05:52 +03:00
Aliaksandr Valialkin
5d3db3ff7c app/vmselect: add interpolate function for filling gaps with linearly interpolated values 
See https://stackoverflow.com/q/62565021/274937 for details
 2020-07-02 14:54:21 +03:00
Aliaksandr Valialkin
4dd3de9286 lib/promscrape: add ability to set disable_compression and disable_keepalive options in scrape_config section of the config passed to -promscrape.config 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/580
 2020-07-02 14:19:14 +03:00
BigFish
9d5f5b6878
fix: spelling mistakes (#594) 
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2020-07-01 01:35:26 +03:00
Aliaksandr Valialkin
84a37098ed app/vmstorage: add -denyQueriesOutsideRetention command-line flag for denying queries outside the configured retention 
VictoriaMetrics returns `503 Service Unavailable` http error for requests with time ranges outside the configured retention
if `-denyQueriesOutsideRetention` command-line flag is set.
 2020-07-01 00:21:44 +03:00
Aliaksandr Valialkin
7c2c8b2981 all: use errors.As for inspecting errors that implement httpserver.ErrorWithStatusCode 2020-07-01 00:04:34 +03:00
Aliaksandr Valialkin
d5dddb0953 all: use %w instead of %s for wrapping errors in fmt.Errorf 
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
 2020-06-30 23:05:11 +03:00
Roman Khavronenko
88538df267
app/vmalert: support multiple notifier urls (#584) (#590) 
* app/vmalert: support multiple notifier urls (#584)

User now can set multiple notifier URLs in the same fashion
as for other vmutils (e.g. vmagent). The same is correct for
TLS setting for every configured URL. Alerts sending is done
in sequential way for respecting the specified URLs order.

* app/vmalert: add basicAuth support for notifier client (#585)

The change adds possibility to set basicAuth creds for notifier
client in the same fasion as for remote write/read and datasource.
 2020-06-29 22:21:03 +03:00
Roman Khavronenko
82ecfa3b32
app/vmalert: move flags description and initialization into subpackages 
The change adds no new functionality and aims to move flags definitions
to subpackages that are using them. This should improve readability
of the main function.
 2020-06-28 12:26:22 +01:00
kreedom
dc4e3f0e0b
app/vmalert: properly set transport for HTTP clients 
Fixes issue #586
 2020-06-27 08:31:54 +01:00
Aliaksandr Valialkin
8f2e88234f docs: update the info that docker images are built on top of alpine image now 
A follow-up after the commit ff624c9125
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/522
 2020-06-26 13:54:10 +03:00
Aliaksandr Valialkin
05d754d7bb app/vmselect/netstorage: reset big result values every 10 seconds instead of after processing every time series 
This should reduce GC pressure when processing time series with big number of rows
 2020-06-24 19:38:39 +03:00
nicbaz
774f7ca1c1
vmselect: fix label_replace when mismatch (#579) 
As per documentation on `label_replace` function: "If the regular
expression doesn't match then the timeseries is returned unchanged".

Currently this behavior is not enforced, if a regexp on an existing
tag doesn't match then the tag value is copied as-is in the destination
tag. This fix first checks that the regular expression matches the
source tag before applying anything.

Given the current implementation, this fix also changes the behavior
of the **MetricsQL** `label_transform` function which does not
document this behavior at the moment.
 2020-06-23 23:50:33 +03:00
Aliaksandr Valialkin
7209d58fbd app/vmselect/netstorage: increase concurrency when processing small number of time series with big number of data points per each time series 
Previously VictoriaMetrics was processing up to 32 time series in a single goroutine.
This could be slow if each time series contains big number of data points (10M+ or more), since only a single CPU core could be loaded with work,
while other CPU cores were idle. Fix this by launching GOMAXPROCS workers for time series processing.

This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/572
 2020-06-23 22:46:15 +03:00
nicbaz
72c90bfd8b
vmalert: add support for TLS configuration (#578) 
app/vmalert: add support for TLS configuration

Add support for TLS optional configuration in a similar fashion to what
is currently supported in other vmutils such as vmagent. TLS
configuration options are distinct for datasource, remoteRead,
remoteWrite as well as notifier.
 2020-06-23 20:45:45 +01:00
Aliaksandr Valialkin
2a39ba639d lib/promrelabel: add support for keep_if_equal and drop_if_equal actions to relabel configs 
These actions may be useful for filtering out unneeded targets and/or metrics if they contain equal label values.
For example, the following rule would leave the target only if __meta_kubernetes_annotation_prometheus_io_port
equals __meta_kubernetes_pod_container_port_number:

  - action: keep_if_equal
    source_labels: [__meta_kubernetes_annotation_prometheus_io_port, __meta_kubernetes_pod_container_port_number]
 2020-06-23 17:29:03 +03:00
Aliaksandr Valialkin
7532dbcdf5 app/vmselect/promql: properly override label values from group_left and group_right lists like Prometheus does 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/577
 2020-06-21 16:33:01 +03:00
kreedom
7ec6711f06
Support of custom URL path for alert (#560) 
app/vmalert: Support custom URL for alerts source

Add flag `external.alert.source` for configuring custom URL
for alert's source. This may be handy to re-point default source
URL to other systems like Grafana.
Updates #517
 2020-06-21 11:32:46 +01:00
Aliaksandr Valialkin
9f3e3a4d7a docs/vmauth.md: mention that we can provide custom integration with SAML 2020-06-19 13:14:07 +03:00
Aliaksandr Valialkin
b542e50680 app/vminsert: export metrics for determining ingested rows with dropped or truncated labels 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/565
 2020-06-19 01:10:37 +03:00
Aliaksandr Valialkin
88e1b7d144 app/vmselect/promql: fill gaps on right side with values from left side of or operator in the same way as Prometheus does 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/552
 2020-06-18 23:05:16 +03:00
Aliaksandr Valialkin
a12364ad37 docs/vmbackup.md: mention that backups from single-node and cluster versions are incompatible 2020-06-18 18:52:43 +03:00
Roman Khavronenko
e91d758831
vmalert-537: allow name duplication for rules within one group. (#559) 
Uniqueness of rule is now defined by combination of its name, expression and
labels. The hash of the combination is now used as rule ID and identifies rule within the group.

Set of rules from coreos/kube-prometheus was added for testing purposes to
verify compatibility. The check also showed that `vmalert` doesn't support
`query` template function that was mentioned as limitation in README.
 2020-06-15 20:15:47 +01:00
Clémence Saussez
e426434770
app/vmalert: fix link to testdata (#547) 
Fix broken link to vmalert test data
Signed-off-by: Clemence Saussez <clemence@zen.ly>
 2020-06-10 17:26:27 +01:00
Roman Khavronenko
3e277020a5
vmalert-491: allow to configure concurrent rules execution per group. (#542) 
The feature allows to speed up group rules execution by
executing them concurrently.

Change also contains README changes to reflect configuration
details.
 2020-06-09 15:21:20 +03:00
Roman Khavronenko
ffa75c423d
vmalert-521: allow to disable rules expression validation. (#536) 
This feature may be useful for using `vmalert` with PromQL
compatible datasources like Loki.
 2020-06-06 21:27:09 +01:00
Aliaksandr Valialkin
f5dd2a71a6 app/vmauth: disable automatic response compression/uncompression, since it may work improperly in some cases 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/535
 2020-06-05 20:13:56 +03:00
Aliaksandr Valialkin
4b98e436ef app/vmauth: emit fatal errors instead of panics when incorrect command-line flags are set 2020-06-05 20:13:55 +03:00
Aliaksandr Valialkin
0d92abfbf6 app/vmalert: print brief usage info for vmalert -help 2020-06-05 10:43:18 +03:00
Aliaksandr Valialkin
ff1a725a56 app/vmauth: print brief usage info for vmauth -help 2020-06-05 10:40:00 +03:00
Aliaksandr Valialkin
05ae1472e3 app/vmagent: print brief usage info for vmagent -help 2020-06-05 10:39:59 +03:00
Aliaksandr Valialkin
f8692a1d43 app/vmauth: log when -auth.config is reloaded in SIGHUP 2020-06-03 23:22:14 +03:00
Aliaksandr Valialkin
d2f30e8d79 app/vmalert: fix comment for UpdateWith exported methods 2020-06-01 14:35:32 +03:00
Roman Khavronenko
270552fde4
vmalert: Add recording rules support. (#519) 
* vmalert: Add recording rules support.

Recording rules support required additional service refactoring since
it wasn't planned to support them from the very beginning. The list
of changes is following:
* new entity RecordingRule was added for writing results of MetricsQL
expressions into remote storage;
* interface Rule now unites both recording and alerting rules;
* configuration parser was moved to separate package and now performs
more strict validation;
* new endpoint for listing all groups and rules in json format was added;
* evaluation interval may be set to every particular group;

* vmalert: uncomment tests

* vmalert: rm outdated TODO

* vmalert: fix typos in README
 2020-06-01 13:46:37 +03:00
Aliaksandr Valialkin
32652485e3 app/vmagent: reload -remoteWrite.relabelConfig and -remoteWrite.urlRelabelConfig on SIGHUP and on /-/reload 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/518
 2020-05-30 14:37:12 +03:00
Aliaksandr Valialkin
d988f89415 app/vmagent: log fatal errors instead of panics when improper command-line flags are passed to vmagent 2020-05-30 14:23:14 +03:00
Aliaksandr Valialkin
5b6a9675d8 app/vmauth: fix make run-vmauth command 2020-05-22 16:45:02 +03:00
Aliaksandr Valialkin
8905bc2a40 app/vmagent: check for error returned from flag.Set 2020-05-21 16:31:14 +03:00
Aliaksandr Valialkin
f9847352b4 app/vmagent: add -dryRun option for checking all the configs mentioned in command-line flags without running vmagent 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/362
 2020-05-21 15:23:27 +03:00
Aliaksandr Valialkin
dbd8beccfa app/vmselect/promql: add ascent_over_time(m[d]) and descent_over_time(m[d]) functions 
These functions could be useful in GPS tracking apps for calculating the summary for height gain/loss
over the given duration `d`.
 2020-05-21 12:07:48 +03:00
kreedom
6b23df2bec
vmalert add quotes escape function (#510) 
* vmalert add quotes escape function

Co-authored-by: kreedom
 2020-05-20 22:20:31 +03:00
Aaron France
619d4959c7 Update README.md 2020-05-20 09:04:31 +03:00
Aliaksandr Valialkin
70ea4e28a7 app/vmselect/promql: update numbers after the upgrade of github.com/VictoriaMetrics/metrics from v1.11.2 to v1.11.3 2020-05-20 03:06:23 +03:00
Aliaksandr Valialkin
80c18b7275 docs/vmagent.md: mention an alternative to refresh_interval option in scrape configs 2020-05-19 23:10:06 +03:00
Aliaksandr Valialkin
538fdfe133 app/vmselect/promql: move common code from aggrFuncOutliersK and newAggrFuncRangeTopK into getRangeTopKTimeseries 2020-05-19 16:11:14 +03:00
Aliaksandr Valialkin
f52769f6ee app/vmselect/promql: fix outilersk calculations 2020-05-19 14:44:53 +03:00
Aliaksandr Valialkin
a441cdd1d9 app/vmselect/promql: add outliersk(N, m) aggregate function for anomaly detection across groups of similar time series 2020-05-19 13:52:36 +03:00
Aliaksandr Valialkin
d0f08b4a58 app/vmalert/notifier: go fmt 2020-05-19 12:59:46 +03:00
kreedom
7e173655ba
vmalert - add expr to variables, add escape functions (#495) 
* vmalert - add expr to variables, add escape functions

Co-authored-by: kreedom
 2020-05-18 11:55:16 +03:00
Roman Khavronenko
92212f04da
vmalert: avoid sending resolves for pending alerts (#498) 
Before the change we were sending notifications to notifier
if following conditions are met:
* alert is in Fire state
* alert is in Inactive state

We were sending Inactive notifications to resolve alert ASAP. 
Unfortunately, we were sending resolves for Pending alerts that become
Inactive, which is wrong.

In this change we delete alert from the active list if
it was Pending and become Inactive. In this way we now
have Inactive alerts only if they were in state Fire before.
See test change for example.
 2020-05-17 15:13:22 +01:00
Roman Khavronenko
de60ad0cd6
vmalert: fix potential race during configuration reloads (#497) 
Configuration reload and rules evaluation can't be executed
in same time now. This may make reload time longer but
prevents from potential races.
 2020-05-17 15:12:09 +01:00
Aliaksandr Valialkin
eac3da478e app/vmalert: run make quicktemplate-gen from the root dir of the repository 2020-05-16 22:46:02 +03:00
Aliaksandr Valialkin
93c87d28f6 all: print --help output to stdout instead of stderr 
This is easier to grep and pipe
 2020-05-16 11:59:33 +03:00
Aliaksandr Valialkin
8cb35974af app/vmrestore: document better that vmrestore works like rsync --delete, i.e. it deletes files in -storageDataPath, which are missing in the backup 2020-05-16 09:22:17 +03:00
Aliaksandr Valialkin
3a68c47de0 app/vmagent/Makefile: fix make run-vmagent rule 2020-05-15 19:35:10 +03:00
Aliaksandr Valialkin
697b6af10f app/vmagent/remotewrite: remove unused import after the commit 93267f143f 2020-05-15 17:42:19 +03:00
Aliaksandr Valialkin
93267f143f app/vmagent/remotewrite: allow ingesting time series with multiple samples at once 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/481
 2020-05-15 17:36:36 +03:00
Aliaksandr Valialkin
82ffbcb9a6 app/vmstorage: add vm_slow_metric_name_loads_total metric, which could be used as an indicator when more RAM is needed for improving query performance 2020-05-15 14:11:45 +03:00
Aliaksandr Valialkin
82ccdfaa91 app/vmstorage: add vm_slow_row_inserts_total and vm_slow_per_day_index_inserts_total metrics for determining whether VictoriaMetrics required more RAM for the current number of active time series 2020-05-15 13:44:32 +03:00
Roman Khavronenko
a249cd9d22
vmalert: fix the access to rules slice element by wrong index (#486) 
During group's update rules deletion was causing slice
mutations while slice index was assumed to be unchanged.
This caused "slice bounds out of range" errors when multiple
rules were deleted sequentially.
 2020-05-15 07:55:22 +01:00
hagen1778
ef0e37cb9e vmalert: update README 2020-05-15 09:17:28 +03:00
Aliaksandr Valialkin
0afd48d2ee lib: extract common code for returning fast unix timestamp into lib/fasttime 2020-05-14 23:02:07 +03:00
Aliaksandr Valialkin
894e5d2b9b docs/vmbackup.md: add a link to vmbackuper tool 2020-05-13 22:54:22 +03:00
Roman Khavronenko
415b1ddfb5
vmalert: check if remoteRead object was initied before calling Restore (#473) 
The check for non-nil remoteRead was mistakenly dropped
during refactoring which caused panics when `vmalert`
wasn't configured with `remoteRead` flag.
 2020-05-13 19:32:58 +01:00
Roman Khavronenko
db7dd96346
vmalert: fix flag names and description in README (#475) 
Change also adds the recommendation for `remotewrite`
queue error.
 2020-05-13 19:32:21 +01:00
肖贝贝
ba48438b06
Feat/vmalert add max queue size (#472) 
* feat: add remoteWrite.maxQueueSize to reduce queue full
* rename remote(write|read) flags to remote(Write|Read) for the sake of consistency

Co-authored-by: xiaobeibei <xiaobeibei@bigo.sg>
 2020-05-13 18:58:56 +01:00
Aliaksandr Valialkin
7882a0dbbf app/vmselect/promql: suppress "SA4006: this value of dstValues is never used" error in golangci-lint 2020-05-13 11:47:08 +03:00
Aliaksandr Valialkin
96e001d254 app/vmagent: fix a bug with improper relabeling when multiple -remoteWrite.urlRelableConfig args are set 
This bug could result in incorrect relabeling and metrics' drop.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/467
 2020-05-12 22:02:58 +03:00
Aliaksandr Valialkin
faf92a0965 app/vmselect/promql: fix any(..) calculations - return all the data points instead of the first one 2020-05-12 20:36:42 +03:00
Aliaksandr Valialkin
cc311e20fe app/vmselect/promql: add any(x) by (y) aggregate function, which returns any time series from q for each group y 2020-05-12 19:45:56 +03:00
Aliaksandr Valialkin
574289c3fb app/vmselect/promql: support for sum(x) by (y) limit N syntax in order to limit the number of output time series after aggregation 2020-05-12 19:45:54 +03:00
Aliaksandr Valialkin
0a134ace63 app/vmagent: fix scraping mTLS targets, which has been broken in v1.35.1 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/470
 2020-05-12 17:23:03 +03:00
Aliaksandr Valialkin
8300cc17af app/vmagent,lib/promscrape: do not set HostClient.DialDualStack, since it isnt used if HostClient.Dial is set 2020-05-12 15:24:18 +03:00
Aliaksandr Valialkin
6273385618 app/vmagent/remotewrite: properly dial TCP6 addresses set via -remoteWrite.url 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/469
 2020-05-12 15:22:29 +03:00
Aliaksandr Valialkin
dbd0c552d5 lib/storage: gradually pre-populate per-day inverted index for the next day 
This should prevent from CPU usage spikes at 00:00 UTC every day when
inverted index for new day must be quickly created for all the active time series.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/430
 2020-05-12 12:13:05 +03:00
Roman Khavronenko
8c8ff5d0cb
vmalert: cleanup and restructure of code to improve maintainability (#471) 
The change introduces new entity `manager` which replaces
`watchdog`, decouples requestHandler and groups. Manager
supposed to control life cycle of groups, rules and
config reloads.

Groups export an ID method which returns a hash
from filename and group name. ID supposed to be unique
identifier across all loaded groups.

Some tests were added to improve coverage.

Bug with wrong annotation value if $value is used in
 templates after metrics being restored fixed.

Notifier interface was extended to accept context.

New set of metrics was introduced for config reload.
 2020-05-10 17:58:17 +01:00
Nikolay Khramchikhin
9e8733ff65
vmalert config reload 
added config hot reload for vmalert with sighup and api call
 2020-05-09 10:32:12 +01:00
Aliaksandr Valialkin
baedb25936 docs/vmauth.md: fix a link to docker images 2020-05-08 14:10:04 +03:00
Aliaksandr Valialkin
51291015a5 app/vmagent: return 200 from /-/reload endpoint as Prometheus does 2020-05-07 19:30:30 +03:00
Aliaksandr Valialkin
6afb25fd08 docs/{vmagent,vmauth}: small clarifications in the docs 2020-05-07 12:55:20 +03:00
Aliaksandr Valialkin
653d51694a app/vmauth: prevent from attacks with .. in path for accessing resources outside the configured url_prefix 2020-05-07 12:55:18 +03:00
Aliaksandr Valialkin
8a00807f60 app/vmagent: allow setting independent auth configs per each configured -remoteWrite.url 2020-05-06 16:51:41 +03:00
Aliaksandr Valialkin
b69eb7bf38 app/vmagent: properly set client-side TLS certificates for -remoteWrite.url. Previously they were mistakenly set as server-side 2020-05-06 16:50:30 +03:00
Aliaksandr Valialkin
e8936c9cb3 docs/vmagent.md: small fixes 2020-05-06 14:49:18 +03:00
Aliaksandr Valialkin
3f52a97f9b lib/promscrape: add Prometheus-compatible DNS-based service discovery aka dns_sd_configs 2020-05-06 00:01:58 +03:00
Aliaksandr Valialkin
08320cfcf4 docs/{vmauth,vmagent}: fix ports for profiling 2020-05-05 20:15:47 +03:00
Aliaksandr Valialkin
f65930b34d docs/vmauth.md: mention that we can help creating customized proxy 2020-05-05 12:34:42 +03:00
Aliaksandr Valialkin
266327642b docs/{vmagent,vmauth}: add Profiling section 2020-05-05 11:45:13 +03:00
Aliaksandr Valialkin
0c7cddfca6 docs: add vmauth.md 2020-05-05 11:17:23 +03:00
Aliaksandr Valialkin
e767aedd17 app/vmauth: add initial version of vmauth. See https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmauth/README.md for details 2020-05-05 10:54:17 +03:00
Aliaksandr Valialkin
b5a780930d docs/vmagent.md: /targets page doesnt expose infomration about imporperly configured scrape configs now. It is written in error log instead 2020-05-05 10:54:14 +03:00
Roman Khavronenko
0ba1b5c71b
app/vmalert: restore alerts state from datasource metrics (#461) 
* app/vmalert: restore alerts state from datasource metrics

Vmalert will restore alerts state for rules that have `rule.For` > 0 from previously written timeseries via `remotewrite.url` flag.

* app/vmalert: mention remotewerite and remoteread configuration in README
 2020-05-05 00:51:22 +03:00
Aliaksandr Valialkin
40c3ffb359 lib/promscrape: add Prometheus-compatible service discovery for Consul aka consul_sd_configs 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/330
 2020-05-04 20:51:17 +03:00
Aliaksandr Valialkin
432187ac3b app/vminsert: add /-/reload handler in the same way as for vmagent 2020-04-30 02:15:39 +03:00
DexterZhang
67511d4165
feat(vmagent): add promscrap config reload suppport via http (#450) 
* feat(vmagent): add promscrap config reload suppport via http endpoint `/-/reload`

* fix: typo fix
 2020-04-30 02:00:32 +03:00
Aliaksandr Valialkin
43c39dc36c vendor: use github.com/VictoriaMetrics/fasthttp instead of github.com/fasthttp/fasthttp 
The upstream fasthttp may contain issues like 996610f021 ,
plus a code that isn't used by VictoriaMetrics. So let's use a private copy under our control instead.
 2020-04-29 17:33:34 +03:00
Artem Navoiev
4487b454a8
Update README.md 2020-04-29 12:39:15 +03:00
Aliaksandr Valialkin
57407cca83 app/vmselect/promql: remove -search.maxPointsPerTimeseries command-line flag 
Limit the estimated time series count after aggregation with grouping by the number of source time series.
 2020-04-29 00:20:04 +03:00
Aliaksandr Valialkin
4e4f57b121 lib/metricsql: move it to a separate repository - github.com/VictoriaMetrics/metrics 2020-04-28 15:28:22 +03:00
Aliaksandr Valialkin
17d96e4503 app/vmselect: add -search.estimatedSeriesCountAfterAggregation command-line flag for tuning the probability of OOMs vs false-positive not enough memory errors 2020-04-28 12:52:37 +03:00
Aliaksandr Valialkin
1397612117 app/vmalert: added missing comments for public entities 2020-04-28 11:21:07 +03:00
Roman Khavronenko
3bfa41a95c
app/vmalert: initial remote-write support for alerts state persistence. (#442) 
* app/vmalert: initial remote-write support for alerts state persistence.

If `remotewrite.url` flag is set, vmalert will send alerts state  via remote-write protocol to remote storage. The sending is asynchronous to avoid blocking calls in rules evaluation loop.

* app/vmalert: merge with master

* app/vmalert: write both `instant` and `for` alerts timeseries states in remote storage.
 2020-04-28 00:18:02 +03:00
Aliaksandr Valialkin
90670cb55e app/vmalert: include it into the next release 2020-04-28 00:10:12 +03:00
Aliaksandr Valialkin
b768bc9a6a lib/promscrape: add initial support for Prometheus-compatible service discovery for Amazon EC2 aka ec2_sd_configs 2020-04-27 19:25:53 +03:00
Aliaksandr Valialkin
b4afe562c1 lib/storage: postpone reading data from blocks during search 
This eliminates the need for storing block data into temporary files on a single-node VictoriaMetrics
during heavy queries, which touch big number of time series over long time ranges.

This improves single-node VM performance on heavy queries by up to 2x.
 2020-04-27 11:45:24 +03:00
Aliaksandr Valialkin
0224071ebe lib/promscrape/discovery/gce: allow empty project and zone for gce_sd_config 2020-04-27 11:45:02 +03:00
Aliaksandr Valialkin
fcf57f9883 app/vmselect/netstorage: substitute sorting packedTimeseries with the natural order of the fetched blocks 
This should minimize the number of disk seeks when reading data from temporary file.
 2020-04-26 16:26:23 +03:00
Aliaksandr Valialkin
6954d0edb7 lib/promscrape/discovery/gce: allow empty zone arg in gce_sd_config - in this case zones for the given project are automatically discovered 2020-04-26 14:34:11 +03:00
kreedom
2c18548e08
alert - rename validate function and flags (#440) 
* alert - rename validate function and flags
 2020-04-26 14:15:04 +03:00
kreedom
5f61d43db9
vmalert - validate template in labels (#439) 2020-04-26 13:53:57 +03:00
肖贝贝
eeadfccdc5
fix: fix vmalert template label not complete bug (#435) 
Co-authored-by: xiaobeibei <xiaobeibei@bigo.sg>
 2020-04-26 13:30:10 +03:00
Aliaksandr Valialkin
1f3fd93b58 docs/{vmbackup,vmrestore}.md: update -help output 2020-04-24 22:44:21 +03:00
Jason Gardner
66af7e40f3
app/vmbackup: added ability to create and delete snapshots during backup (#428) 
* app/vmbackup: added ability to create and delete snapshots during backup

Resolves: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/422

* Add snapshot create and delete url flags

* Fixed errcheck warnings in build
 2020-04-24 22:35:03 +03:00
Aliaksandr Valialkin
a596aec82c app/vmselect: fix description for -search.resetCacheAuthKey 2020-04-24 19:45:50 +03:00
Aliaksandr Valialkin
9ef5935552 lib/promscrape: initial implementation for gce_sd_configs aga Prometheus-compatible service discovery for Google Compute Engine 2020-04-24 17:51:22 +03:00
Aliaksandr Valialkin
364db13c9c app/vmselect: add /api/v1/status/tsdb page with useful stats for locating root cause for high cardinality issues 
See https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/425
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/268
 2020-04-22 22:03:43 +03:00
Aliaksandr Valialkin
9ebc937685 app/vmselect: add -search.minStalenessInterval command-line flag for removing gaps on graphs built from time series with irregular duration between samples 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/426
 2020-04-20 19:42:15 +03:00
Aliaksandr Valialkin
fe57d46687 app/vmselect: merge -search.maxLookback and -search.maxStalenessInterval flags, since it has been appeared they have identical purpose :( 
Leave both flags for backwards compatibility reasons.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/209
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/426
 2020-04-20 19:26:31 +03:00
Aliaksandr Valialkin
851946af1e deployment/docker: allow building docker images on top of any base image set via ROOT_IMAGE environment var 
For example, the following command will build VictoriaMetrics docker image on top of alpine image:

    ROOT_IMAGE=alpine make package-victoria-metrics
 2020-04-20 01:16:57 +03:00
Aliaksandr Valialkin
936fb0eac3 app/vmagent/remotewrite: retry sending data if the server closes keep-alive connection 
This should fix the following error when sending data to remote storage:

couldn't send a block with size XX bytes to "YYY": the server closed connection before returning the first response byte. Make sure the server returns 'Connection: close' response header before closing the connection
 2020-04-17 15:52:42 +03:00
Aliaksandr Valialkin
79fb595732 docs/vmagent.md: typo fix: unvailable -> unavailable 2020-04-17 13:11:31 +03:00
Aliaksandr Valialkin
546d26523c app/vmagent/README.md: mention about prodmscrape.suppressScrapeErrors 2020-04-17 13:08:21 +03:00
Aliaksandr Valialkin
f41e6a7bd9 app/vmselect: properly apply -search.maxLookback to queries sent to /api/v1/query 2020-04-17 12:30:11 +03:00
Aliaksandr Valialkin
071fdf5518 lib/logger: add WARN level for logging expected errors such as invalid user queries 2020-04-15 20:50:26 +03:00
Aliaksandr Valialkin
6f7f64f757 app/vmselect: handle timestamp(metric offset X) the same way as Prometheus does 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/415
 2020-04-15 12:01:00 +03:00
Aliaksandr Valialkin
067c7afebc lib/promscrape: show information on improperly configured scrape targets at the bottom of /targets page 
This is a common error whith improperly configured target autodiscovery and/or relabeling.
This error leads to duplicate scraping of the same targets with the same set of labels, which leads
to duplicate samples in time series.
 2020-04-14 14:55:05 +03:00
Aliaksandr Valialkin
0b76c27fa1 docs/vmagent.md: mention that vmagent supports kubernetes_sd_configs now 2020-04-13 21:06:36 +03:00
hagen1778
2eed6c393f vmalert: prepare package for external usage 
* update README according to changes
* add Makefile with basic commands
 2020-04-12 15:32:42 +03:00
kreedom
948f8b6b5f [vmalert] fix linter issues 2020-04-12 15:08:11 +03:00
kreedom
8fca5f2819
[vmalert] add tests to webserver (#413) 2020-04-12 14:51:03 +03:00
Roman Khavronenko
7c9405f53d
Vmalert metrics (#412) 
vmalert: add basic list of metrics
 2020-04-11 20:42:01 +01:00
Roman Khavronenko
9f8cc8ae1b
Extend web responses for alerts: (#411) 
vmalert: Extend web responses for alerts

* populate apiAlert object with additional fields
* return all active alerts, not only firing
* sort list of API alerts for deterministic output
* add helper for available path list
 2020-04-11 16:49:23 +01:00
kreedom
90de3086b3
[vmalert] add webserver (#410) 
* [vmalert] add webserver
 2020-04-11 12:40:24 +03:00
Aliaksandr Valialkin
f94a090020 docs: update minimum supported Go version from 1.12 to 1.13 2020-04-07 13:38:37 +03:00
Roman Khavronenko
b099d84271
Vmalert/rules eval (#400) 
* Initial rules evaluation support.

Rules are now store alerts state in private field `alerts`. Every evaluation updates
the alerts and state. Every unique metric received from datastore represents a unique alert,
uniqueness is guaranteed by hashing ordered labelset.

* merge with master

* cleanup

* support endAt parameter as 3*evaluationInterval for active alerts

* make golint happy
 2020-04-06 14:44:03 +03:00
Aliaksandr Valialkin
895cadfae7 app/vmagent/remotewrite: add "X-Prometheus-Remote-Write-Version: 0.1.0" http header to remote_write request 
This header is required by Cortex (and, probably, other remote storage systems).
See 9c1f44d090/docs/apis.md (remote-api) .

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/399
 2020-04-04 16:24:56 +03:00
Aliaksandr Valialkin
57704aa584 app/victoria-metrics: add -selfScrapeInstance and -selfScrapeJob flags for tuning labels for self-scraped metrics 2020-04-04 14:57:22 +03:00
Aliaksandr Valialkin
f9b24d4899 app/vmselect/promql: keep metric name after applying first_over_time and last_over_time functions 2020-04-04 14:54:13 +03:00
Aliaksandr Valialkin
b38d048dd9 app/vmstorage: add vm_free_disk_space_bytes metric for monitoring the remaining disk space at -storageDataPath 2020-04-01 23:08:58 +03:00
kreedom
298eb0a0f8 [vmalert] improve external url handling 2020-04-01 22:29:11 +03:00
kreedom
12fe915b48
[vmalert] add prometheus template function (#396) 
* [vmalert] add prometheus template function

* make linter be happy

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2020-04-01 18:17:53 +03:00
Aliaksandr Valialkin
e71519b8b2 app/victoria-metrics/testdata: add a test for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/395 2020-03-31 12:51:25 +03:00
Aliaksandr Valialkin
318326c309 lib/storage: properly handle {label=~"foo|"} filters as Prometheus does 
Such filters must match all the time series with `label="foo"` plus all the time series without `label`

Previously only time series with `label="foo"` were matched.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/395
 2020-03-31 00:48:18 +03:00
Aliaksandr Valialkin
0c485f14d1 app/vmselect/prometheus: allow passing relative time to start, end and time args of /api/v1/* queries 2020-03-29 21:57:14 +03:00
Aliaksandr Valialkin
2ebf7d86ff app/vmselect/prometheus: code simplification: (d.Seconds()/1e3) -> d.Milliseconds() 2020-03-29 21:50:28 +03:00
kreedom
bf6c24d0f4
[vmalert] config parser (#393) 
* [vmalert] config parser

* make linter be happy

* fix test

* fix sprintf add test for rule validation
 2020-03-29 01:48:30 +02:00
Aliaksandr Valialkin
bd156cd088 docs/vmagent.md: add prometheus remote_write proxy use case 2020-03-28 23:16:38 +02:00
Dmitry Naumov
27373807c1 Rootless docker images by default (#358) 
* Rootless docker images by default

* Migrate to rootless base image

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2020-03-27 21:23:50 +02:00
kreedom
b22da547a2
[vmalert] - parse template annotaions (#387) 
* [vmalert] - parse template annotations
 2020-03-27 18:31:16 +02:00
Aliaksandr Valialkin
3c7c71a49c app/vmselect: adjust label_map() handling for corner cases 
The following corner cases now supported:
* label_map(q, "label", "", "foo") - adds `label="foo"` to series with missing `label`
* label_map(q, "label", "foo", "") - removes `label="foo"` from series

All the unmatched labels are kept unchanged.
 2020-03-13 18:45:03 +02:00
Aliaksandr Valialkin
4fc4912f0c app/vmalert/datasource: typo fix in docs: Labels -> Label 2020-03-13 12:22:33 +02:00
kreedom
a746cb62b6
vmalert add vm datasource, change alertmanager (#364) 
* vmalert add vm datasource, change alertmanager

* make linter be happy

* make linter be happy.2

* PR comments

* PR comments.1
 2020-03-13 12:19:31 +02:00
Aliaksandr Valialkin
fdc2a9d1d7 app/vmselect: add label_map(q, label, srcValue1, dstValue1, ... srcValueN, dstValueN) function to MetricsQL 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/369
 2020-03-12 19:13:47 +02:00
Aliaksandr Valialkin
8023ad7dbd app/vmselect: add -search.maxStalenessInterval for tuning Prometheus data model closer to Influx-style data model 2020-03-11 16:43:34 +02:00
Aliaksandr Valialkin
fcd91795d5 app/vmagent: mention that vmagent can filter data 2020-03-11 16:22:39 +02:00
Aliaksandr Valialkin
301c2acd61 app/vmstorage: return 500 status code instead of 200 status code on internal errors inside /snapshot/* handlers 2020-03-10 23:51:55 +02:00
Aliaksandr Valialkin
e17702fada app/vmselect: add optional max_rows_per_line query arg to /api/v1/export 
This arg allows limiting the number of data points that may be exported on a single line.
 2020-03-10 21:45:56 +02:00
Aliaksandr Valialkin
1fe66fb3cc app/{vmagent,vminsert}: add support for importing csv data via /api/v1/import/csv 2020-03-10 21:15:35 +02:00
Aliaksandr Valialkin
5fe38a84eb app/vmagent: properly apply -remoteWrite.sendTimeout to fasthttp.HostClient 2020-03-09 13:31:55 +02:00
Aliaksandr Valialkin
c386c5de57 app/vmagent: properly add labels set via -remoteWrite.label to metrics before sending them to -remoteWrite.url 2020-03-06 19:26:58 +02:00
Aliaksandr Valialkin
c5f894b361 Makefile: add build and test rules with enabled race detector. These rules have -race suffix 
Fix also `unsafe pointer conversion` errors detected by Go1.14. See https://golang.org/doc/go1.14#compiler .
 2020-03-05 12:03:38 +02:00
Aliaksandr Valialkin
754db0d22e app/vmagent/README.md: small fixes 2020-03-04 18:14:47 +02:00
Aliaksandr Valialkin
772312bf7b app/vmagent/README.md: typo fix 2020-03-04 18:05:09 +02:00
Aliaksandr Valialkin
871abfab7a app/vmagent/README.md: clarification 2020-03-04 18:03:48 +02:00
Aliaksandr Valialkin
007c591de8 app/vmagent/README.md: add iot and edge monitoring use case 2020-03-04 18:01:34 +02:00
Aliaksandr Valialkin
474a09c0f1 app/vmagent/README.md: add use cases section 2020-03-04 17:42:27 +02:00
Aliaksandr Valialkin
032c88561b app/vminsert/prompush: limit memory usage by pushing promscrape data in smaller blocks 2020-03-03 19:58:54 +02:00
Aliaksandr Valialkin
76036c1897 app/vmagent: add -remoteWrite.maxDiskUsagePerURL for limiting the maximum disk usage for each -remoteWrite.url buffer 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/352
 2020-03-03 19:49:07 +02:00
Aliaksandr Valialkin
c31d640eb9 app/vmagent/remotewrite: do not reset empty relabelCtx 2020-03-03 15:01:03 +02:00
Aliaksandr Valialkin
02b55c72dc app/vmagent: add -remoteWrite.urlRelabelConfig for applying individual relabeling for each -remoteWrite.url 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/320
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/308
 2020-03-03 13:12:16 +02:00
Aliaksandr Valialkin
7d178a40bd app/vmselect/prometheus: do not add __name__!= filter when searching for all the matching metric names via /api/v1/label/__name__/values with non-empty label filter 
This should reduce query time.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/343
 2020-02-28 23:35:55 +02:00
Aliaksandr Valialkin
f9a584b5c1 app/vmagent/remotewrite: yet another typo fix 2020-02-28 20:05:55 +02:00
Aliaksandr Valialkin
b9b46cb8dc app/vmagent/remotewrite: typo fix 2020-02-28 19:03:16 +02:00
Aliaksandr Valialkin
db6f4e4af1 app/vmagent/remotewrite: limit memory usage when big scrape blocks are pushed to remote storage 2020-02-28 18:58:01 +02:00
Aliaksandr Valialkin
5701b2f7bb app/vmselect/prometheus: properly pass filter for labelName=__name__ in labelValuesWithMatches 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/343
 2020-02-28 12:18:14 +02:00
Aliaksandr Valialkin
18af31a4c2 all: properly split vm_deduplicated_samples_total among cluster components 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/345
 2020-02-27 23:48:07 +02:00
Aliaksandr Valialkin
c458b521a2 app/vmagent: allow setting -httpListenAddr to empty string in order to disable listening for http requests 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/340
 2020-02-26 20:58:11 +02:00
Aliaksandr Valialkin
117c76311c app/vmagent/README.md: list service discovery mechanisms, which will be added soon 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/334
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/330
 2020-02-26 19:27:08 +02:00
Aliaksandr Valialkin
be37d762cd app/vmagent: add -remoteWrite.maxBlockSize command-line flag for limiting the maximum size of unpacked block to send to remote storage 2020-02-25 19:57:47 +02:00
Aliaksandr Valialkin
4e24839a2c app/vmagent: do not allow sending unpacked requests with sizes exceeding -maxInsertRequestSize 2020-02-25 19:34:41 +02:00
Aliaksandr Valialkin
6386aeb1e0 app/vmagent: add ability to accept Influx line protocol data via TCP and UDP 
Just set `-influxListenAddr` command-line flag

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/333
 2020-02-25 19:12:49 +02:00
Aliaksandr Valialkin
e453880084 app/vmagent/README.md: mention that vmagent exposes target statuses at /targets page 2020-02-25 18:15:58 +02:00
Aliaksandr Valialkin
4c4448b66e app/vminsert: add /targets handler, which exposes Prometheus targets defined in -promscrape.config file 2020-02-25 18:13:11 +02:00
Aliaksandr Valialkin
e1ef72af01 app/vmagent: logo fix 2020-02-25 00:09:19 +02:00
Aliaksandr Valialkin
56c70fe856 app/vmagent: update docs 2020-02-25 00:09:18 +02:00
Aliaksandr Valialkin
e7e4aa5243 app/vmagent/README.md: small fixes 2020-02-24 21:25:38 +02:00
Aliaksandr Valialkin
ae51300973 app/vmselect/promql: properly take into account the first datapoint when calculating rollup_candlestick 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
 2020-02-24 13:24:30 +02:00
Aliaksandr Valialkin
e65ec88779 app/vmselect/promql: do not take into account values outside the current window in rollup_candlestick 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
 2020-02-23 18:03:57 +02:00
Yaroslav
a6d0645539
fix rollupOpen(), rollupHigh(), rollupLow() functions (#328) 2020-02-23 18:01:53 +02:00
Aliaksandr Valialkin
04762344c6 app/vmagent: initial implementation for vmagent 2020-02-23 13:36:03 +02:00
kreedom
49390b8dbc
[vmalert] integration with AlertManager (#325) 2020-02-21 23:15:05 +02:00
Aliaksandr Valialkin
2f55cabaa4 app/vmselect/promql: log when rollupResult cache is cleared 2020-02-21 20:07:01 +02:00
Aliaksandr Valialkin
ec9bf39b5b app/vmselect: add -search.cacheTimestampOffset command-line flag 
This flag can be used for removing gaps on graphs if the difference between the current time
and the timestamps from the ingested data exceeds 5 minutes.

This is the case when the time between data sources and VictoriaMetrics is improperly synchronized.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/312
 2020-02-21 13:58:06 +02:00
Aliaksandr Valialkin
539139391c app/vmselect: add /internl/resetRollupResultCache handler for resetting response cache 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/312
 2020-02-21 13:58:05 +02:00
kreedom
3c06179184
basic vmalert backbone (#317) 
* basic vmalert backbone

* Resolve code review comments for vmalert backbone

* Second review fixes for vmalert backbone
 2020-02-16 20:59:02 +02:00
Aliaksandr Valialkin
7b1c7051a3 app/vmselect: add sort_by_label(q, label) and sort_by_label_desc(q, label) functions 
This is implementation of https://github.com/prometheus/prometheus/pull/1533 for VictoriaMetrics.
 2020-02-13 17:01:37 +02:00
Aliaksandr Valialkin
8466ab0034 all: allow setting flags via environment vars 
Now flags can be set via environment vars with the same names as flags.
Command-line flags override flags set via env vars.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/311
 2020-02-10 13:29:13 +02:00
Aliaksandr Valialkin
e210cd9da1 lib/storage: move -dedup.minScrapeInterval flag outside lib/storage, so it doesnt show up in vminsert in cluster version 2020-02-10 13:09:51 +02:00
Aliaksandr Valialkin
a6c6a2debc app/vmselect/promql: do not add step to range end, since this hack became obsolete since commit 9e1119dab8 2020-02-05 21:22:19 +02:00
Aliaksandr Valialkin
78b62dee87 app/vmselect/promql: properly adjust time range for data to select 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
 2020-02-05 21:22:18 +02:00
Aliaksandr Valialkin
366693b9f1 app/vmselect: unconditionally offset -step to rollup_candlestick. This makes results more consistent 2020-02-04 23:32:12 +02:00
Aliaksandr Valialkin
525101339e app/vmselect/promql: automatically apply offset -step to rollup_candlestick function in order to obtain the expected OHLC results 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
 2020-02-04 23:24:35 +02:00
Aliaksandr Valialkin
ada6a3da8d app/vmselect/promql: adjust rollup_candlestick calculations to the exepcted results 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/309
 2020-02-04 22:42:13 +02:00
Aliaksandr Valialkin
36a1ac8360 app/vmselect: take into account the time the requests wait in the queue if -search.maxConcurrentRequests is exceeded 
This will prevent from excess CPU usage for timed out queries.
 2020-02-04 16:15:08 +02:00
Aliaksandr Valialkin
834051e5b2 app/vmselect: add a placeholder for /api/v1/metadata, which could be requested by Grafana 
See https://prometheus.io/docs/prometheus/latest/querying/api/#querying-metric-metadata

VictoriaMetrics doesn't collect any metadata for metrics, so just return empty response.
 2020-02-04 15:53:47 +02:00
Aliaksandr Valialkin
42864bb52f all: do not clash flag description with back-quoted flag types 
See https://golang.org/pkg/flag/#PrintDefaults for more details.
 2020-02-04 15:46:52 +02:00
Artem Navoiev
a47f292295 [vmalert] add vmalert.png.2 2020-02-02 12:17:19 +02:00
Artem Navoiev
354232b62b [vmalert] add vmalert.png 2020-02-02 12:16:05 +02:00
Artem Navoiev
28778be0cc [vmalert] initial 2020-02-02 12:14:09 +02:00
Aliaksandr Valialkin
90cf356ea1 app/vmselect/promql: adjust and and unless binary operator handling to be consistent with Prometheus 2020-01-31 18:52:38 +02:00
Aliaksandr Valialkin
c3d86eef96 all: add -dedup.minScrapeInterval command-line flag for data de-duplication 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/86
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/278
 2020-01-31 01:16:57 +02:00
Aliaksandr Valialkin
d70ba7eb37 lib/fs: optimize small reads for ReaderAt.MustReadAt by reading from memory-mapped space instead of reading from file descriptor 
This should improve performance when reading many small blocks.
 2020-01-30 15:09:05 +02:00
Aliaksandr Valialkin
d68546aa4a lib/storage: pre-fetch metricNames for the found metricIDs in Search.Init 
This should speed up Search.NextMetricBlock loop for big number of found time series.
 2020-01-30 15:08:51 +02:00
Aliaksandr Valialkin
a462355b2f app/vmselect/promql: add keep_next_value(q) for filling gaps with the next non-empty value 2020-01-29 00:48:04 +02:00
Aliaksandr Valialkin
371e86194d app/vminsert: moved -maxInsertRequestSize command-line flag out of lib/prompb in order to prevent its inclusion in vmselect and vmstorage apps 2020-01-28 23:02:08 +02:00
Aliaksandr Valialkin
adbbc4fa1a app/vmselect/promql: return expected results from increase() over the beginning of time series, which start from big value 
Examples for such counters: OS-level counters for network or cpu stats.
 2020-01-28 16:30:11 +02:00
Aliaksandr Valialkin
75ad47a43c app/victoria-metrics: check for error arg passed to filepath.Walk callback 2020-01-27 20:56:45 +02:00
Aliaksandr Valialkin
6320a19a8c app/victoria-metrics: remove integration build tag from tests 
This simplifies testing with `go test ./app/victoria-metrics` without
the need to remember to pass `-tags=integration` to Go commands.
 2020-01-27 20:25:28 +02:00
Aliaksandr Valialkin
2392a359e1 app/vmselect/promql: fix panic on a single zero vmrange bucket in prometheus_buckets() function 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/296
 2020-01-27 18:04:55 +02:00
Aliaksandr Valialkin
9df5b2d1c3 app/victoria-metrics: add -selfScrapeInterval flag for self-scraping /metrics page 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/30
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/180
 2020-01-25 19:19:59 +02:00
Aliaksandr Valialkin
6456c93dbb app/vminsert: move ingestion protocol parsers to lib/protoparser, so they could be re-used in the upcoming vmagent 2020-01-24 16:53:00 +02:00
Aliaksandr Valialkin
680080887d all: consistently log durations in seconds with millisecond precision 
This should improve logs readability
 2020-01-22 18:28:27 +02:00
Aliaksandr Valialkin
9773022e50 app/vmselect: mention the original query and time range in error messages 
This should simplify debugging invalid or heavy queries.
 2020-01-22 17:36:36 +02:00
Aliaksandr Valialkin
efc7ad88ec app/vmselect: mention command-line flag, which could be used for adjusting query timeouts, in timeout errors 2020-01-22 15:50:48 +02:00
Aliaksandr Valialkin
ec9651e266 app/vmselect/prometheus: increase default value -maxExportDuration to 30 days, since 10 minutes beat users exporting bit amounts of data 2020-01-22 15:50:47 +02:00
Aliaksandr Valialkin
582dd01f42 app/vmselect/promql: add range_over_time(m[d]) function for calculating value range for m over d 2020-01-21 19:05:17 +02:00
Aliaksandr Valialkin
36973ee975 app/vmselect/promql: add label_match(q, label, regexp) and label_mismatch(q, label, regexp) functions for filtering out time series with labels matching the given regexp 2020-01-21 15:00:20 +02:00
Aliaksandr Valialkin
227cf53ef9 app/vminsert: increase default value for -insert.maxQueueDuration from 30s to 60s 
This should help catching up with high ingestion rate after VictoriaMetrics restart.
 2020-01-18 14:39:36 +02:00
Aliaksandr Valialkin
c9472e4f3a all: use github.com/klauspost/compress/gzip instead of compress/gzip 
`github.com/klauspost/compress/gzip` is more optimized than `compress/gzip`.
This gives better gzip compression and decompression speeds.
 2020-01-17 23:58:46 +02:00
Aliaksandr Valialkin
cdcacaea6d app/vmselect/netstorage: make fmt 2020-01-17 17:47:21 +02:00
Aliaksandr Valialkin
7327adbc86 app/vmselect/netstorage: limit the maximum size for in-memory buffer for temporary blocks file 
This should reduce memory usage on systems with more than 8GB RAM.
 2020-01-17 16:28:21 +02:00
Aliaksandr Valialkin
d0d258b314 app/vmselect: limit the default value for -search.maxConcurrentRequests, so it plays well on systems with more than 16 vCPUs 
A single heavy request can saturate all the available CPUs, so let's limit the number of concurrent requests to lower value.
This will give more chances for executing insert path.
 2020-01-17 15:43:54 +02:00
Aliaksandr Valialkin
d88725f133 app/{vminsert,vmselect}: improve error messages when VictoriaMetrics cannot handle too high number of concurrent inserts / selects 2020-01-17 13:24:37 +02:00
Aliaksandr Valialkin
0d7505b00e all: mention command-line flags used for limiting the incoming request size in error messages 
This should improve error logs usability.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/287
 2020-01-16 13:03:30 +02:00
Aliaksandr Valialkin
2839f4688a app/vmselect/promql: fix panic on sum(aggr_over_time(...)) with incorrect number of args 2020-01-15 16:26:09 +02:00
Aliaksandr Valialkin
8b14572f70 app/vmselect/promql: add hoeffding_bound_upper(phi, m[d]) and hoeffding_bound_lower(phi, m[d]) functions 
These functions can be used for calculating Hoeffding bounds
for `m` over `d` time range and for the given `phi` in the range `[0..1]`.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/283
 2020-01-11 14:46:23 +02:00
Aliaksandr Valialkin
8eaced8cae app/vmselect/promql: return continuous values for min_over_time and max_over_time when step is smaller than scrape_interval 2020-01-11 12:47:50 +02:00
Aliaksandr Valialkin
ea231f8167 app/victoria-metrics: adjust integration tests after the commit 99facd71cd6ac151d512cea1df73be91c10c7f83 2020-01-11 00:58:16 +02:00
Aliaksandr Valialkin
46bfdbe6cf app/vmselect/promql: do not take into account the previous point before time window in square brackets for min_over_time, max_over_time, rollup_first and rollup_last functions 
This makes the behaviour for these functions similar to Prometheus when processing broken time series with irregular data points
like `gitlab_runner_jobs`. See https://gitlab.com/gitlab-org/gitlab-exporter/issues/50 for details.
 2020-01-11 00:26:26 +02:00
Aliaksandr Valialkin
4f0a645f77 vendor: update github.com/valyala/fastjson from v1.4.2 to v1.4.5 
This should fix parsing Inf values in `/api/v1/import`. The previous attempt to fix this in VictoriaMetrics v1.32.1 was unsuccessful.
 2020-01-10 23:15:15 +02:00
Aliaksandr Valialkin
b829fe5e39 app/vmselect/promql: properly handle aggr(aggr_over_time(...)) 2020-01-10 21:57:18 +02:00
Aliaksandr Valialkin
164278151f app/vmselect/promql: add aggr_over_time(("aggr_func1", "aggr_func2", ...), m[d]) function 
This function can be used for simultaneous calculating of multiple `aggr_func*` functions
that accept range vector. For example, `aggr_over_time(("min_over_time", "max_over_time"), m[d])`
would calculate `min_over_time` and `max_over_time` for `m[d]`.
 2020-01-10 21:18:06 +02:00
Aliaksandr Valialkin
c4632faa9d app/vmselect/promql: add tmin_over_time(m[d]) and tmax_over_time(m[d]) functions 
These functions return timestamp in seconds for the minimum and maximum value for `m` over time range `d`
 2020-01-10 19:39:28 +02:00
Aliaksandr Valialkin
f358fb72d1 app/{vmbackup,vmrestore}: add backup complete file to backup when it is complete and check for this file before restoring from backup 
This should prevent from restoring from incomplete backups.

Add `-skipBackupCompleteCheck` command-line flag to `vmrestore` in order to be able restoring from old backups without `backup complete` file.
 2020-01-09 15:35:38 +02:00
Aliaksandr Valialkin
d5aeda0e1a app/vmselect/promql: skip rate calculation for the first point on time series 2020-01-08 14:42:53 +02:00
Aliaksandr Valialkin
d410c78c7e app/vmselect/promql: fix calculations for histogram_share 2020-01-04 14:44:48 +02:00
Aliaksandr Valialkin
d64699bb9f app/vmselect/promql: add missing MetricName into netstorage.Result in tests 2020-01-04 12:52:39 +02:00
Aliaksandr Valialkin
f409f2d050 app/vmselect/promql: add histogram_share(le, buckets) function 2020-01-04 12:45:55 +02:00
Aliaksandr Valialkin
b1ded7cf9a app/vmselect/promql: add absent_over_time(m[d]) func similar to the function in Prometheus 2.16 
See https://github.com/prometheus/prometheus/issues/2882
 2020-01-04 12:45:07 +02:00
Aliaksandr Valialkin
a8360d04c0 app/vmselect/promql: add histogram_over_time(m[d]) rollup function 2020-01-04 12:44:56 +02:00
Aliaksandr Valialkin
3e09d38f29 app/vmselect/promql: fix results caching for multi-arg rollup functions such as quantile_over_time 
Previosly only a single arg was taken into account, so caching didn't work properly for multi-arg rollup funcs.
 2020-01-03 20:49:08 +02:00
Aliaksandr Valialkin
a774120460 app/vmselect/promql: use scrapeInterval instead of window in denominator when calculating rate for the first point on the time series 
This should provide better estimation for `rate` in the beginning of time series.
 2020-01-03 19:01:50 +02:00
Aliaksandr Valialkin
b5645ccbdf app/vmselect/promql: increase the estimated number of time series returned by aggr() by (something) from 100 to 1K, since 100 may result in OOM for high number of time series 2020-01-03 01:02:21 +02:00
Aliaksandr Valialkin
cb3a342882 app/vmselect/promql: add share_le_over_time and share_gt_over_time functions for SLI and SLO calculations 2020-01-03 00:41:16 +02:00
Aliaksandr Valialkin
6f67e0b56b lib/metricsq: add ExpandWithExprs 2019-12-25 22:20:30 +02:00
Aliaksandr Valialkin
1925ee038d Rename lib/promql to lib/metricsql and apply small fixes 2019-12-25 22:03:59 +02:00
Mike Poindexter
bec62e4e43 Split Extended PromQL parsing to a separate library 2019-12-25 22:03:51 +02:00
Aliaksandr Valialkin
d880325cf6 app/vmselect/promql: make sure AdjustStartEnd returns time range covering the same number of points as the initial time range 
This should prevent from the following panic at app/vmselect/promql/binary_op.go:255:

    BUG: len(leftVaues) must match len(rightValues) and len(dstValues)
 2019-12-24 22:45:56 +02:00
Aliaksandr Valialkin
9e19949c6b app/vmselect/promql: adjust calculations for rate and increase for the first value 
These calculations should trigger alerts on `/api/v1/query` for counters starting from values greater than 0.
 2019-12-24 19:39:25 +02:00
Aliaksandr Valialkin
0455c03cb9 app/vmselect/promql: properly calculate rate on the first data point 
It is calculated as `value / scrape_interval`, since the value was missing on the previous scrape,
i.e. we can assume its value was 0 at this time.
 2019-12-24 15:55:52 +02:00
Aliaksandr Valialkin
5cb8d97743 all: use gozstd instead of pure Go zstd for GOARCH=amd64 2019-12-24 12:42:42 +02:00
Aliaksandr Valialkin
5b75984aa9 app/vmselect/netstorage: move MustAdviseSequentialRead to lib/fs 2019-12-23 23:16:11 +02:00
Aliaksandr Valialkin
bea5a8700a app/vmselect: add -search.maxExportDuration command-line flag for limiting /api/v1/export duration 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/275
 2019-12-20 11:35:22 +02:00
Andrii Dembitskyi
2fba7b6f35 Fix typo in log message 2019-12-19 14:33:20 +02:00
Aliaksandr Valialkin
d03827c57d app/vminsert: return StatusNoContent http response for /api/v1/import to be consistent with other insert handlers 2019-12-19 01:21:54 +02:00
Aliaksandr Valialkin
96ff8d9adb app/vmselect: add ability to pass match[], start and end to /api/v1/labels 
This makes the `/api/v1/labels` handler consistent with already existing functionality for `/api/v1/label/.../values`.

See https://github.com/prometheus/prometheus/issues/6178 for more details.
 2019-12-15 00:20:50 +02:00
Aliaksandr Valialkin
02f6566ce1 app/vmbackup: mention that backups are possible to Ceph and Swift 2019-12-14 01:08:49 +02:00
Aliaksandr Valialkin
bc645152cb app/vminsert: simultaneously accept telnet put and HTTP /api/put OpenTSDB metrics at -opentsdbListenAddr 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/266
 2019-12-14 00:30:12 +02:00
Aliaksandr Valialkin
cea5a14853 all: rename Extended PromQL to PromQL extensions 2019-12-12 19:25:58 +02:00
Aliaksandr Valialkin
6ddcd162ed all: publish Docker images for the following GOARCH: amd64, arm, arm64, ppc64le and 386 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/258
 2019-12-11 23:32:59 +02:00
Aliaksandr Valialkin
73b2a3d4b7 app/vmselect/promql: return lower and upper bounds for the estimated percentile from histogram_quantile if third arg is passed 
Updates https://github.com/prometheus/prometheus/issues/5706
 2019-12-11 13:57:26 +02:00
Aliaksandr Valialkin
caa4eb72d9 app/vmselect/promql: return matrix instead of vector on subqueries to /api/v1/query like Prometheus does 2019-12-11 01:00:26 +02:00
Aliaksandr Valialkin
3c076544bf app/vmselect/promql: allow negative offsets 
Updates https://github.com/prometheus/prometheus/issues/6282
 2019-12-11 01:00:23 +02:00
Aliaksandr Valialkin
68e1cf8942 app/vminsert: add /api/v1/import handler 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6
 2019-12-09 20:59:04 +02:00
Aliaksandr Valialkin
8501b4a48d app/vminsert: consistency renaming for counters 2019-12-09 16:43:10 +02:00
Aliaksandr Valialkin
293b541784 make vendor-update 2019-12-07 23:10:16 +02:00
Aliaksandr Valialkin
84b57e8974 app/vminsert/influx: add a test case from https://community.librenms.org/t/integration-with-victoriametrics/9689 2019-12-07 23:00:40 +02:00
Aliaksandr Valialkin
c09472dfd9 app/vmselect/promql: add {topk|bottomk}_{min|max|avg|median} aggregate functions for returning the exact k time series on the given time range 
The full list of functions added:
- `topk_min(k, q)` - returns top K time series with the max minimums on the given time range
- `topk_max(k, q)` - returns top K time series with the max maximums on the given time range
- `topk_avg(k, q)` - returns top K time series with the max averages on the given time range
- `topk_median(k, q)` - returns top K time series with the max medians on the given time range
- `bottomk_min(k, q)` - returns bottom K time series with the min minimums on the given time range
- `bottomk_max(k, q)` - returns bottom K time series with the min maximums on the given time range
- `bottomk_avg(k, q)` - returns bottom K time series with the min averages on the given time range
- `bottomk_median(k, q)` - returns bottom K time series with the min medians on the given time range
 2019-12-05 19:26:47 +02:00
Aliaksandr Valialkin
face3d57bf app/vmselect: add placeholders for /api/v1/rules and /api/v1/alerts 2019-12-03 19:36:33 +02:00
Aliaksandr Valialkin
20812008a7 lib/storage: remove metricID with missing metricID->metricName entry 
The metricID->metricName entry can be missing in the indexdb after unclean shutdown
when only a part of entries for new time series is written into indexdb.

Recover from such a situation by removing the broken metricID. New metricID
will be automatically created for time series with the given metricName
when new data point will arive to it.
 2019-12-02 20:46:44 +02:00
Aliaksandr Valialkin
9476b73527 app/vmselect/promql: estimate per-series scrape interval as 0.6 quantile for the first 100 intervals 
This should improve scrape interval estimation for tiem series with gaps.
 2019-12-02 13:42:33 +02:00
Aliaksandr Valialkin
811b7a8303 app/vminsert/influx: allow empty measurement in Influx line protocol 
In this case metric names are mapped directly from field names without any prefixes.
 2019-11-30 23:18:41 +02:00
Aliaksandr Valialkin
75ff524a4e app/vmselect/promql: fix corner case for increase over time series with gaps 
In this case `increase` could return invalid high value for the first point after the gap.
 2019-11-30 01:34:56 +02:00
Aliaksandr Valialkin
96492348cb deployment/docker/certs: update TLS certs source from alpine:3.9 to alpine:3.10 2019-11-29 19:57:29 +02:00
Aliaksandr Valialkin
f733cb2186 lib/backup: cosmetic fixes after #243 2019-11-29 18:07:04 +02:00
glebsam
15b7406f7b Add option to provide custom endpoint for S3, add option to specify S3 config profile (#243) 
* Add option to provide custom endpoint for S3 for use with s3-compatible storages, add option to specify S3 config profile

* make fmt
 2019-11-29 17:59:56 +02:00
Aliaksandr Valialkin
4c859d980c app/vmselect/prometheus: consistently apply nocache arg to /api/v1/query the same way ast to /api/v1/query_range 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/241
 2019-11-26 22:55:43 +02:00
Aliaksandr Valialkin
110235f789 app/vmselect/prometheus: fix content-type for /api/v1/export responses 
The correct Content-Type should be `application/stream+json` instead of `application/json`
Thanks to Joshua Ryder for pointing to this.
 2019-11-26 17:45:26 +02:00
Aliaksandr Valialkin
205233d9a7 app/vmselect/promql: remove zero timeseries from prometheus_buckets output 2019-11-25 19:10:23 +02:00
Aliaksandr Valialkin
3f99f39e9b app/vmselect/prometheus: reduce default value for -search.latencyOffset from 60s to 30s 
30 seconds should be enough for almost all the cases
 2019-11-25 16:33:42 +02:00
Aliaksandr Valialkin
e91cb34c0e app/vmselect/promql: allow nested parens 2019-11-25 16:13:41 +02:00
Aliaksandr Valialkin
826dfd63a5 vendor: update github.com/VictoriaMetrics/metrics from v1.9.0 to v1.9.1 2019-11-25 15:23:01 +02:00
Aliaksandr Valialkin
0401969d78 app/vmselect/promql: re-use metrics.Histogram when calculating histogram function for each point on the graph 
This should reduce the amounts memory allocations
 2019-11-25 14:24:21 +02:00
Aliaksandr Valialkin
da98703748 app/vmselect/promql: optimize binary search over big number of samples during rollup calculations 2019-11-25 14:01:46 +02:00
Aliaksandr Valialkin
c28876172f app/vmselect/promql: adjust tests after the upgrade of github.com/VictoriaMetrics/metrics from v1.8.3 to v1.9.0 2019-11-25 13:43:57 +02:00
Aliaksandr Valialkin
50ae1879c6 app/vmselect/promql: add histogram aggregate function, which is useful for building heatmaps from multiple time series 2019-11-24 00:04:25 +02:00
Aliaksandr Valialkin
8582b50360 app/vmselect/promql: do not take into account buckets with negative counters in prometheus_buckets 2019-11-23 14:19:25 +02:00
Aliaksandr Valialkin
19dfe52254 app/vmselect/promql: properly handle histogram_quantile(0, ...) with zero buckets 2019-11-23 14:02:35 +02:00
Aliaksandr Valialkin
4bb88843cf app/vmselect: add vm_per_query_{rows,series}_processed_count histograms 2019-11-23 13:23:26 +02:00
Aliaksandr Valialkin
7753c8c0a1 app/vmselect/promql: transparently apply prometheus_buckets in histogram_quantile 2019-11-23 11:48:51 +02:00
Aliaksandr Valialkin
c4287b3c86 app/vmselect/promql: add prometheus_buckets function for converting the upcoming histogram buckets from github.com/VictoriaMetrics/metrics to Prometheus-compatible buckets 2019-11-23 00:20:20 +02:00
Aliaksandr Valialkin
1f3fd2c910 app/vmselect: adjust end arg instead of adjusting start arg if start > end 
`start` arg has higher chances to be set properly comparing to `end` arg,
so it is expected that the `end` arg could be adjusted if it was set incorrectly.
 2019-11-22 16:12:19 +02:00
Aliaksandr Valialkin
7a4635f853 all: remove the remaining mentions of cluster version 2019-11-21 23:18:22 +02:00
Aliaksandr Valialkin
443189fb0a app/{vmbackup,vmrestore}: add -maxBytesPerSecond command-line flag for limiting the used network bandwidth during backup / restore 2019-11-19 20:31:52 +02:00
Aliaksandr Valialkin
0094bc4fc9 app/vmselect/prometheus: properly adjust too big time time on /api/v1/query 
Too big `time` must be adjusted to `now()-queryOffset`.
 2019-11-19 00:42:00 +02:00
Aliaksandr Valialkin
3f1637fae8 app/vmselect/promql: properly calculate integrate(q[d]) 2019-11-13 21:10:41 +02:00
Aliaksandr Valialkin
c56b9ed03b app/victoria-metrics: add build rules for GOARCH=ppc64le 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/235
 2019-11-13 20:24:33 +02:00
Aliaksandr Valialkin
3fd32e331a app/vmselect/promql: use universal approach for determining maxByteSliceLen on 32-bit and 64-bit archs 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/235
 2019-11-13 20:24:26 +02:00
Aliaksandr Valialkin
119dfd01bb lib/storage: add vm_cache_size_bytes{type="storage/hour_metric_ids"} metric 2019-11-13 20:24:21 +02:00
Aliaksandr Valialkin
86a1cd700b lib/storage: remove inmemory index for recent hour, since it uses too much memory 
Production workload shows that the index requires ~4Kb of RAM per active time series.
This is too much for high number of active time series, so let's delete this index.

Now the queries should fall back to the index for the current day instead of the index
for the recent hour. The query performance for the current day index should be good enough
given the 100M rows/sec scan speed per CPU core.
 2019-11-13 17:58:07 +02:00
Aliaksandr Valialkin
c57eb0ff83 lib/storage: add -disableRecentHourIndex flag for disabling inmemory index for recent hour 
This may be useful for saving RAM on high number of time series aka high cardinality
 2019-11-13 15:02:51 +02:00
Aliaksandr Valialkin
ca259864e2 lib/storage: return back inmemory inverted index for recent hour 
Issues fixed:
- Slow startup times. Now the index is loaded from cache during start.
- High memory usage related to superflouos index copies every 10 seconds.
 2019-11-13 13:11:04 +02:00
Aliaksandr Valialkin
01bb3c06c7 lib/storage: remove inmemory inverted index for recent hours 
Production load with >10M active time series showed it could
slow down VictoriaMetrics startup times and could eat
all the memory leading to OOM.

Remove inmemory inverted index for recent hours until thorough
testing on production data shows it works OK.
 2019-11-13 10:45:53 +02:00
Aliaksandr Valialkin
6c2303764e Revert "lib/fs: do not postpone directory removal on NFS error" 
This reverts commit 4c02e496f7.

Reason for revert: the commit breaks on NFS - see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/234
 2019-11-12 16:18:09 +02:00
Aliaksandr Valialkin
661dd190bb Refer to https://medium.com/@valyala/speeding-up-backups-for-big-time-series-databases-533c1a927883 from multiple places in README.md 2019-11-12 13:02:39 +02:00
Oleg Kovalov
b4f44befa3 fix misspelled words (#229) 2019-11-12 00:16:42 +02:00
Aliaksandr Valialkin
8e8f98f712 lib/storage: add tests for dateMetricIDCache 2019-11-11 13:21:57 +02:00
Aliaksandr Valialkin
56d7cc8a0d app/victoria-metrics: remove deprecated fs.MustStopDirRemover from main_test.go 2019-11-10 13:37:13 +02:00
Aliaksandr Valialkin
4c02e496f7 lib/fs: do not postpone directory removal on NFS error 
Continue trying to remove NFS directory on temporary errors for up to a minute.

The previous async removal process breaks in the following case during VictoriaMetrics start

- VictoriaMetrics opens index, finds incomplete merge transactions and starts replaying them.
- The transaction instructs removing old directories for parts, which were already merged into bigger part.
- VictoriaMetrics removes these directories, but their removal is delayed due to NFS errors.
- VictoriaMetrics scans partition directory after all the incomplete merge transactions are finished
  and finds directories, which should be removed, but weren't still removed due to NFS errors.
- VictoriaMetrics panics when it finds unexpected empty directory.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162
 2019-11-10 13:24:51 +02:00
Aliaksandr Valialkin
5c3fa59181 app/vmrestore: the upcoming release would be 1.29.0 2019-11-10 00:20:41 +02:00
Aliaksandr Valialkin
ee7765b10d lib/storage: implement per-day inverted index 2019-11-10 00:02:46 +02:00
Aliaksandr Valialkin
5810ba57c2 lib/storage: use specialized cache for (date, metricID) entries 
This improves ingestion performance.
 2019-11-09 23:06:11 +02:00
Aliaksandr Valialkin
6ad7fe8eeb lib/storage: export vm_new_timeseries_created_total metric for determining time series churn rate 2019-11-08 21:21:07 +02:00