Aliaksandr Valialkin
c18017a9c3
app/vminsert/netstorage: sort the -storageNode
list passed to vminsert
nodes
...
This should reduce resource usage (CPU, RAM, disk IO) at vmstorage nodes
if the addresses of vmstorage nodes are passed in random order to vminsert nodes.
2021-06-23 14:00:08 +03:00
Nikolay
e03a3d3a36
adds http_sd ( #1399 )
...
* adds http_sd
* adds X-Prometheus-Refresh-Interval-Seconds header
* Update lib/promscrape/discovery/http/api.go
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2021-06-22 13:42:09 +03:00
Roman Khavronenko
79474baf99
vmctl: add more context to flags description in vm-native mode ( #1395 )
2021-06-18 19:20:52 +03:00
Aliaksandr Valialkin
b92d110cad
app/vmselect: log slow requests to all the /api/v1/*
handlers if their execution time exceeds -search.logSlowQueryDuration
2021-06-18 19:07:03 +03:00
Aliaksandr Valialkin
4acc4602b3
app/vmctl: limit JSON line size by 10K samples ( #1394 )
...
This should reduce the maximum memory usage at VictoriaMetrics when importing time series with big number of samples.
2021-06-18 15:41:34 +03:00
Aliaksandr Valialkin
60bc35f550
docs/{vmgateway,vmbackupmanager}: explicitly mention that these components are a part of an enterprise package
2021-06-17 17:19:13 +03:00
Aliaksandr Valialkin
51fc469642
app/vmagent/remotewrite: go fmt
after 0a796f7c3a
2021-06-17 13:51:40 +03:00
Zongyang
cf506e300d
Change default value of '-remoteWrite.queues' to cgroup.AvailableCPUs * 2 ( #1385 )
...
* Change default value of '-remoteWrite.queues' to cgroup.AvailableCPUS() * 2 to reduce scrape interval
Default value of vmagent option '-remotewrite.queues' is 4 and default
size of vmagent ScheudleUnmarshalWorkers is number of CPUs, when available
CPUs is much greater than 4, e.g 32, worker are competing push queues
which will increase scrape interval and may cause scrape timeout.
* Update README and flag description
Co-authored-by: xiaozy <xiaozy01@fenbi.com>
2021-06-16 12:37:55 +03:00
Roman Khavronenko
a15c947045
promql: fix increase_pure
calculation for cases with stale series ( #1381 )
...
Due to staleness handling, increase_pure were using incorrect previous value
during calculation in cases where series disappears for period longer
than staleness period and then returns back. The fix suppose to account
for a real datapoint value before staleness takes place. The fix should
remove unexpected spikes while using `increase_pure` for staled series.
2021-06-15 17:37:51 +03:00
Nikolay
e42da47608
adds digital ocean sd ( #1376 )
...
* adds digital ocean sd config
* adds digital ocean sd
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1367
* typo fix
2021-06-14 13:19:29 +03:00
Roman Khavronenko
c5f493db8e
Vmalert docs ( #1372 )
...
* vmalert: mention what happens if `for` is set to 0 or omitted
* vmalert: add more context to docs
2021-06-14 11:43:01 +03:00
Aliaksandr Valialkin
0672cfffa2
app/vmauth: properly handle http.ErrAbortHandler panic
...
This panic can be raised by the reverseProxy on aborted request to the backend.
So handle it (e.g. suppress) at reverseProxy.ServeHTTP call.
Do not suppress the panic at lib/httpserver generic HTTP handler,
since it may result in an inconsistent state left after the panicking handler.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1353
2021-06-11 12:54:37 +03:00
Roman Khavronenko
f3cb2158a3
vmalert: fix mistake with object reuse while parsing response ( #1370 )
...
* vmalert: fix mistake with object reuse while parsing response
During the refactoring, the wrong optimisations was applied in
parse function which caused metric fields reset. The change removes
optimisation.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1369
* vmalert: add test to cover multiple metrics in one response
2021-06-11 11:30:07 +03:00
John Belmonte
3e79f3994e
spelling fix: synonym ( #1363 )
2021-06-11 10:58:48 +03:00
Aliaksandr Valialkin
e8e7f03394
app/vmselect/promql: typo fix in the comment
2021-06-09 18:34:57 +03:00
Aliaksandr Valialkin
247b2a5a08
app/vmauth: improve readability for a config with multiple src_paths
2021-06-09 15:38:09 +03:00
Aliaksandr Valialkin
520d62ade2
docs/vmagent.md: mention that vmagent supports scrape targets sharding
2021-06-09 12:30:54 +03:00
Aliaksandr Valialkin
f3749dedba
docs: document rules replay feature for vmalert
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/836
This is a follow-up for 2a259ef5e7
2021-06-09 12:30:54 +03:00
Roman Khavronenko
5aa7846900
vmalert: support rules backfilling (aka replay
) ( #1358 )
...
* vmalert: support rules backfilling (aka `replay`)
vmalert can `replay` configured rules in the past
and backfill results via remote write protocol.
It supports MetricsQL/PromQL storage as data source,
and can backfill data to remote write compatible
storage.
Supports recording and alerting rules `replay`. See more
details in README.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/836
* vmalert: review fixes
* vmalert: readme fixes
2021-06-09 12:30:54 +03:00
Aliaksandr Valialkin
2c6b917749
app/vminsert/netstorage: update storageNode.lastRerouteTime before the rerouting
...
This is needed for reliable detection of storage nodes with recent rerouting
2021-06-08 12:06:32 +03:00
Aliaksandr Valialkin
0d067eb112
app/vminsert/netstorage: tune re-routing algorithm
...
Do not re-route data to unavailable storage node. Send it to the remaining storage nodes instead
even if they cannot keep up with the load. This should spread the load more evenly among available
storage nodes.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/791
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1054
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1165
2021-06-05 16:23:44 +03:00
Aliaksandr Valialkin
269e35d676
app/{vmagent,vminsert}: follow-up after 2fe045e2a4
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1343
2021-06-04 20:33:22 +03:00
jelmd
d8b46908db
new feature: debug relabeling ( #1344 )
...
* new feature: relabel logging
Use scrape_configs[x].relabel_debug = true to log metric names inkl.
labels before and after relabeling. After relabeling related metrics
get dropped, i.e. not submitted to servers.
* vminsert wants relabel logging, too.
2021-06-04 20:33:21 +03:00
Aliaksandr Valialkin
1c09e71f5b
app/vminsert: add -disableRerouting
command-line flag for disabling re-routing if some vmstorage nodes have lower performance than the others
...
Refactor the rerouting mechanism and make it more resilient to cases when some of vmstorage nodes are temporarily unavailable.
Reduce the probability of rerouting storm.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/791
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1054
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1165
2021-06-04 04:33:52 +03:00
Aliaksandr Valialkin
8cdecfc52c
app/vmauth: allow balancing the load among multiple backend nodes by specifying multiple urls in url_prefix
config
2021-05-29 01:04:22 +03:00
Aliaksandr Valialkin
97de72054e
docs: document f0c21b6300
2021-05-27 15:04:13 +03:00
Roman Khavronenko
e183a5c532
vmalert: automatically reload configuration on file change ( #1326 )
...
New flag `-rule.configCheckInterval` defines how often `vmalert` will re-read
config file. If it detects any changes, the config will be reloaded.
This behaviour is turned off by default.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/512
2021-05-26 12:24:27 +03:00
Aliaksandr Valialkin
a0b001bfec
app/vmselect/netstorage: remove duplicate limiter on concurrent queries
...
It duplicates the `-search.maxConcurrentRequests` limiter.
2021-05-24 19:13:04 +03:00
Aliaksandr Valialkin
890e1bd826
app/vmagent/remotewrite: use WARN level instead of ERROR level for couldnt send a block with size ... bytes to ...
log message
...
This is really warning, since vmagent re-tries sending the data block until success.
2021-05-24 15:43:32 +03:00
Roman Khavronenko
beee24ecee
vmalert: support extra_filter_labels
setting per-group ( #1319 )
...
The new setting `extra_filter_labels` may be assigned to group.
If it is, then all rules within a group will automatically filter
for configured labels. The feature is well-described here
https://docs.victoriametrics.com#prometheus-querying-api-enhancements
New setting is compatible only with VM datasource.
2021-05-23 14:15:49 +03:00
Aliaksandr Valialkin
71ff7ee18d
lib/promauth: follow-up after 5b8176c68e
2021-05-22 18:02:03 +03:00
Nikolay
2780d6dbcd
basic OAuth2 support for remoteWrite and scrape targets ( #1316 )
...
* adds OAuth2 support for remoteWrite and scrapping
* adds tests
changes init
2021-05-22 18:02:01 +03:00
Nikolay
23a6c9c016
changes vmalert query function ( #1307 )
...
* changes vmalert query function
for prometheus rules compatibility its better to use labels as map.
it simplifies template evaluation and allow to ignore can't evaluate field error
because map will return default value.
fixes https://github.com/VictoriaMetrics/operator/issues/243
2021-05-21 16:38:20 +03:00
Aliaksandr Valialkin
d77db9d813
all: do not skip SIGHUP signal during service initialization
...
This can lead to stale or incomplete configs like in the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1240
2021-05-21 16:38:20 +03:00
Aliaksandr Valialkin
6139f6ed6d
app/vmauth: add ability to protect /-/reload
endpoint with authKey
2021-05-20 18:48:34 +03:00
Aliaksandr Valialkin
69e365cd48
Makefile: update golangci-lint from v1.29.0 to v1.40.1
2021-05-20 18:30:24 +03:00
Aliaksandr Valialkin
da0b32c31a
app/vmagent/remotewrite: expose metrics with the current number of active series per day and per hour
...
These numbers are exposed via the following metrics:
- vmagent_hourly_series_limit_current_series
- vmagent_daily_series_limit_current_series
Expose also the limits via the following metrics:
- vmagent_hourly_series_limit_max_series
- vmagent_daily_series_limit_max_series
2021-05-20 15:31:57 +03:00
Aliaksandr Valialkin
165a9f9200
app/vmstorage: add ability to limit series cardinality via -storage.maxHourlySeries
and -storage.maxDailySeries
command-line flags
2021-05-20 15:31:57 +03:00
Aliaksandr Valialkin
7aad5c3f76
app/vmagent: add ability to limit series cardinality on a per-hour and per-day basis
2021-05-20 15:31:57 +03:00
Roman Khavronenko
12d0c6b6e0
vmctl: explicitly set ::tag
type for labels selector in influx
mode ( #1310 )
...
The `::tag` type is needed in cases when field and tag names are equal, which
results into unexpected results in InfluxQL. Setting the type explicitly helps
InfluxDB to understand which exact column we apply filter to.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1299
2021-05-20 12:07:15 +03:00
Aliaksandr Valialkin
180829b8c2
app/vmselect/promql: add timezone_offset(tz)
function
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1306
2021-05-20 11:54:06 +03:00
Aliaksandr Valialkin
dcac849c1f
app/vmagent/remotewrite: sort labels before sending the series to per-remoteWrite.url queues
2021-05-20 11:54:06 +03:00
Neo He
c5ab00ebee
app/{vmbackup,vmrestore},docs/vmrestore.md: typo fix: vbackup -> vmbackup ( #1305 )
2021-05-18 16:38:15 +03:00
Aliaksandr Valialkin
74ef40034c
lib/httpserver: typo fix in -http.shutdownDelay
command-line flag description: servier -> server
2021-05-18 16:25:27 +03:00
Aliaksandr Valialkin
1668280e67
docs/vmalert.md: document multitenant support
...
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/740
2021-05-18 16:25:21 +03:00
Aliaksandr Valialkin
7fe362deb1
app/vmauth: reload -auth.config
on the request to /-/reload
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1194
2021-05-18 02:24:37 +03:00
Aliaksandr Valialkin
25ca108642
docs/vmbackup.md: typo fix: snaphosts -> snapshots
...
Thanks to @jelmd - see 1ab27582a3 (r50884395)
2021-05-18 01:14:01 +03:00
Aliaksandr Valialkin
6ea191d196
docs: dealay -> delay
2021-05-18 01:07:32 +03:00
Roman Khavronenko
3428df6f15
vmalert: use stringified label keys for duplicates map in recroding rules ( #1301 )
...
duplicates map helps to determine wheter extra labels has overriden
labels which make time series unique. It was using a sorted hashed
labels sequence as a key. But hashing algorithm could have collisions,
so it is more convenient to not use hashing at all.
Log message for recording rules duplicates was improved as well.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1293
2021-05-17 01:51:48 +03:00
Aliaksandr Valialkin
a6cb4f10a7
app/{vmalert,vmauth}: explicitly set MaxIdleConnsPerHost in net/http.Client.Transport
...
By default MaxIdleConnsPerHost is set to 2. This limits the possibility to re-use http keep-alive connections.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1300
2021-05-14 18:13:34 +03:00
Aliaksandr Valialkin
23afbd5094
app/vmagent/remotewrite: clarify the comment explaining why vmagent drops blocks if remote storage returns 400 or 409 status code
2021-05-13 16:17:09 +03:00
Aliaksandr Valialkin
2839055513
lib/storage: substitute GetTSDBStatusForDate with GetTSDBStatusWithFiltersForDate with nil tfss
2021-05-13 09:01:05 +03:00
Nikolay
be87be34a4
Adds tsdb match filters ( #1282 )
...
* init work on filters
* init propose for status filters
* fixes tsdb status
adds test
* fix bug
* removes checks from test
2021-05-12 17:16:58 +03:00
Aliaksandr Valialkin
56b08390f6
app/vmselect/promql: allow to use 2x more memory for query processing in cluster mode compared to single-node mode
...
`vmselect` has no `vmstorage`-related caches. So it can use more memory for query processing compared to single-node VictoriaMetrics.
2021-05-12 14:43:49 +03:00
Aliaksandr Valialkin
cca9670573
docs/CHANGELOG.md: document -datasource.roundDigits
added at 5c448126dc
2021-05-10 11:18:58 +03:00
Roman Khavronenko
a7f00101f5
vmalert: add support for round_digits
param in datasource package ( #1278 )
...
Starting from v1.56.0 VM supports `round_digits` which allows to limit
the number of digits after the decimal point in response value. The feature
can be used to reduce entropy of produced by recording rules values
and significantly improve the compression. See more details in link below.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/525
2021-05-10 11:18:56 +03:00
Roman Khavronenko
35237fe1f5
vmalert: fix error when rule didn't start if restore failed ( #1279 )
...
Previously, `startGroup` could exit on restore errors despite the
`remoteRead.ignoreRestoreErrors` flag value. Now vmalert checks the
flag value before deciding whether to return error or just log it.
2021-05-10 11:10:32 +03:00
Aliaksandr Valialkin
2dddd68feb
docs/vmagent.md: add stream parsing mode
chapter
2021-05-08 23:14:47 +03:00
Aliaksandr Valialkin
9c505d27dd
lib/ingestserver: properly close incoming connections during graceful shutdown
2021-05-08 19:53:45 +03:00
Aliaksandr Valialkin
4a5f45c77e
app/vminsert: add support for data ingestion via other vminsert nodes
2021-05-08 19:53:45 +03:00
Aliaksandr Valialkin
07bc021f58
app/vmalert: add missing comment for ErrStateRestore
2021-05-08 19:53:45 +03:00
Aliaksandr Valialkin
e8478e1e97
app/vmbackup: make sure that -snapshotName
isnt set if -snapshot.createURL
is set
2021-05-07 08:44:44 +03:00
Roman Khavronenko
bb7e113dd4
vmalert: add flag to control behaviour on startup for state restore errors ( #1265 )
...
Alerting rules now can return specific error type ErrStateRestore to indicate
whether restore state procedure failed. Such errors were returned and logged
before as well. But now user can specify whether to just log these errors
(remoteRead.ignoreRestoreErrors=true) or to stop the process
(remoteRead.ignoreRestoreErrors=false). The latter is important when VM isn't
ready yet to serve queries from vmalert and it needs to wait.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1252
2021-05-05 12:24:32 +03:00
Aliaksandr Valialkin
0a2e746175
docs/vmalert.md: update docs after afca7b430c
2021-04-30 11:49:40 +03:00
Roman Khavronenko
7394967841
vmalert: fix the typo in ApplyParams func ( #1259 )
2021-04-30 11:47:11 +03:00
Roman Khavronenko
6fbedd62b8
vmalert: use rule's evaluationInterval
as step
param by default ( #1258 )
...
User still can override param by specifying `datasource.queryStep` flag.
2021-04-30 10:03:50 +03:00
Aliaksandr Valialkin
daf2778025
docs/CHANGELOG.md: document the change from f3a048288e
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1232
2021-04-30 09:56:47 +03:00
Roman Khavronenko
b55677e93d
Vmalert: adjust time
param for datasource queries according to evaluationInterval
( #1257 )
...
* Simplify arguments list for fn `queryDataSource` to improve readbility
* vmalert: adjust `time` param according to rule evaluation interval
With this change, vmalert will start to use rule's evaluation interval
for truncating the `time` param. This is mostly needed to produce consistent
time series with timestamps unaffected by vmalert start time. Now, timestamp
becomes predictable.
Additionally, adjustment is similar to what Grafana does for plotting range graphs.
Hence, recording rule series and recording rule expression plotted in grafana
suppose to become similar in most of cases.
2021-04-30 09:56:46 +03:00
Aliaksandr Valialkin
8be1cb297b
app/vmagent: list user-visible endpoints at
http://vmagent:8429/
...
While at it, use common WriteAPIHelp function for the listing in vmagent, vmalert and victoria-metrics
2021-04-30 09:38:23 +03:00
Nikolay
2eb8ef7b2b
changes vmalert Querier with per rule querier ( #1249 )
...
* changes vmalert Querier with per rule querier
it allows to changes some parametrs based on rule setting
for instance - alert type, tenant for cluster version or event endpoint url.
2021-04-29 11:31:07 +03:00
Roman Khavronenko
0ceb4f7565
vmalert: keep the returned timestamp when persisting recording rule ( #1245 )
...
Previously, vmalert used `lastExecTime` timestamp when writing recording rules
to the remote storage. This may be incorrect, if vmalert uses `datasource.lookback` flag,
which means rule's expression will be executed at some moment in the past.
To avoid such situations, vmalert now will use returned timestamp instead of `lastExecTime`.
2021-04-27 00:16:45 +03:00
Aliaksandr Valialkin
e309b5a83b
app/vmagent/remotewrite: increase the maximum possible number of inmemory blocks for systems with high amounts of RAM
...
This should reduce the probability of using much slower file-based persistent queue
when vmagent processes metrics at high rate (millions of metrics per second).
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1235
2021-04-23 22:05:00 +03:00
Aliaksandr Valialkin
f92db26a93
app/vmagent/remotewrite: count maxLabelsPerBlock as 10x of maxRowsPerBlock
...
This should increase block sizes and subsequently increase the maximum possible bandwidth per each connection to remote storage.
This, in turn, should reduce the probability of storing the data in local buffers.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1235
2021-04-23 22:05:00 +03:00
Aliaksandr Valialkin
aaee80d158
app/vmbackup: typo fix: snaphsot -> snapshot
...
Follow-up for 9de0fa3649
2021-04-22 11:18:13 +03:00
Aliaksandr Valialkin
e7c4fde756
app/vmauth: parse url_prefix
only once during config load
2021-04-21 10:57:17 +03:00
Aliaksandr Valialkin
6dc5d3b357
all: rename https://victoriametrics.github.io to https://docs.victoriametrics.com
2021-04-20 20:20:01 +03:00
Aliaksandr Valialkin
64f1ddefe5
all: consistency renaming Victoria Metrics
-> VictoriaMetrics
...
VMInsert -> vminsert
VMSelect -> vmselect
VMStorage -> vmstorage
2021-04-20 11:45:02 +03:00
Aliaksandr Valialkin
8d869d112b
app/vmauth: follow-up for 6a81a89b3d
2021-04-20 10:59:22 +03:00
Nikolay
7d249d787d
adds query params support for vmauth urlPrefix ( #1226 )
...
* adds query params support for vmauth urlPrefix
* Update app/vmauth/example_config.yml
* Update app/vmauth/example_config.yml
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2021-04-20 10:59:20 +03:00
Aliaksandr Valialkin
bdcd978d56
app/vmctl: update README.md according to bfecd0fd55
2021-04-16 12:10:00 +03:00
Aliaksandr Valialkin
c872ba45b9
docs: update -help
output after the commit 77be3e3a82
2021-04-12 12:35:39 +03:00
Artem Navoiev
c3dcfdef8c
improve docs for cli flags ( #1202 )
...
* improve docs for cli flags
* improve docs for cli flags.2
2021-04-12 12:28:36 +03:00
Aliaksandr Valialkin
2190fd2148
docs: make docs-sync
2021-04-10 19:55:05 +03:00
Aliaksandr Valialkin
0f7ece84f3
app/vmstorage/transport: reduce memory allocations on data ingestion path
2021-04-10 17:36:00 +03:00
Roman Khavronenko
46e6fdb131
Docs update ( #1199 )
...
* docs: drop table of contents for `vmctl`
We already have it autogenerated on .github.io, so no need to keep it.
* docs: mention OpenTSDB migration feature for vmctl
* docs: sync docs for `vmalert`
2021-04-10 15:47:52 +03:00
John Seekins
9e7bf595a0
Improve documentation on OpenTSDB migration tool and fix a bug with hard offsets ( #1198 )
...
* add more documentation on OpenTSDB migration explaining what chunking means
* more clarification of OpenTSDB aggregations
* break out what a retention string becomes
* add more docs around retention strings
* add example of running program and fix mistake in how hard offsets are handled
* fix formatting
2021-04-10 15:28:56 +03:00
John Seekins
97fafce028
OpenTSDB migration to VictoriaMetrics ( #1089 )
2021-04-10 15:28:54 +03:00
Roman Khavronenko
712725b4a5
vmalert: document template functions and mention them in README ( #1197 )
2021-04-08 18:20:57 +03:00
Aliaksandr Valialkin
e48fd5776b
app/vmselect/promql: fix tests after d3fa0ccabd
2021-04-08 00:17:40 +03:00
Aliaksandr Valialkin
f90bf265f4
app/vmselect/promql: properly detect aggregate topk*
and bottomk*
aggregate functions in order to disable duplicate sorting
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1189
2021-04-08 00:10:33 +03:00
Aliaksandr Valialkin
9ce3b7e1dd
app/vmselect: return data:null
instead of data:[]
from /api/v1/query_exemplars
, since Grafana throws an error otherwise
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1186
2021-04-07 23:34:01 +03:00
Aliaksandr Valialkin
3d5f1f779f
app/vmselect: do not sort series returned from topk*
and bottomk*
functions, since these series are already sorted in user-expected order
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1189
2021-04-07 14:16:02 +03:00
Roman Khavronenko
ff3711eea2
docs: update docs ordering and formatting ( #1192 )
...
The major change is adding `sort` directive to docs. For those docs which are copied
from internal packages `sort` is added via makefile command. For the rest it is added
manually since they're updated manually as well.
The rest of changes is connected with markdown formatting. For example, changing headers
in some files (`##` => `#`) makes navigation on .github.io to look better. This especially
useful for `changelog` docs.
Table of contents for `vmctl` is dropped, since we already have it autogenerated on .github.io.
No link changes expected. The corresponding PR to `cluster` branch will be made in follow-up PR.
2021-04-07 13:43:01 +03:00
Aliaksandr Valialkin
7fce4e9fb4
app/vmselect: return dumb response on /api/v1/query_exemplars
request
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1186
2021-04-05 23:28:21 +03:00
Aliaksandr Valialkin
2585058a5f
Makefile: prepare arm64
and amd64
release archives for cluster version on make release
command
2021-04-05 23:01:45 +03:00
Aliaksandr Valialkin
92d4c80639
docs/vmagent.md: mention that vmagent supports scraping via socks5 proxy
2021-04-04 01:45:46 +03:00
Aliaksandr Valialkin
ab9e1eb41f
lib/promscrape: support for simple HTTP proxies without CONNECT
method support such as https://github.com/prometheus-community/PushProx
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1179
2021-04-04 00:40:58 +03:00
Aliaksandr Valialkin
4aa59cae7c
app/vmselect/promql: do not delete dst_label
if src_label
is empty in label_copy(q, src_label, dst_label)
and label_move(q, src_label, dst_label)
2021-04-03 22:06:50 +03:00
Aliaksandr Valialkin
dfec690548
docs/vmgateway.md: update docs
2021-04-03 00:29:57 +03:00
Aliaksandr Valialkin
b9469de410
app/vmselect/promql: add ability to set label value additionally to label name for the remaining sum of time series returned from topk_*
and bottomk_*
functions in the form: topk_min(N, m, "label=value")
2021-04-02 23:56:30 +03:00
Aliaksandr Valialkin
262da1c2e5
docs/{vmauth,vmgateway}.md: small fixes
2021-04-02 23:15:40 +03:00
Aliaksandr Valialkin
5edec0e57e
app/vmgateway: publish docs
2021-04-02 23:10:10 +03:00
Aliaksandr Valialkin
4028d692f5
app: do not process non-GET requests on at /
handler
2021-04-02 22:56:38 +03:00
Aliaksandr Valialkin
89771d082d
app/vmauth: add support for authorization via Authorization: Bearer <token>
2021-04-02 22:15:07 +03:00
Aliaksandr Valialkin
6c6174271a
docs/vmagent.md: mention about proxy_authorization section
2021-04-02 21:24:52 +03:00
Aliaksandr Valialkin
87700f1259
lib/promscrape: add support for authorization
config in -promscrape.config
as Prometheus 2.26 does
...
See https://github.com/prometheus/prometheus/pull/8512
2021-04-02 21:20:37 +03:00
Aliaksandr Valialkin
31d3d02d25
app/vmselect/promql: remove the limit on the number of time series that can be sorted, since it may confuse users
...
Always sort time series returned from `/api/v1/query` and `/api/v1/query_range` unless `sort_*` function is used at top level of the query.
2021-04-02 16:50:02 +03:00
Aliaksandr Valialkin
512addc608
app/{vminsert,vmagent}: add -sortLabels
command-line option for sorting time series labels before ingesting them in the storage
...
This option can be useful when samples for the same time series are ingested with distinct order of labels.
For example, metric{k1="v1",k2="v2"} and metric{k2="v2",k1="v1"}.
2021-03-31 23:27:21 +03:00
Aliaksandr Valialkin
ae1c653d55
lib/storage: reduce memory usage when ingesting samples for the same time series with distinct order of labels
2021-03-31 21:22:40 +03:00
Aliaksandr Valialkin
392ba94d1d
app/vmagent/remotewrite: reduce memory usage when -remoteWrite.queues
is set to a big value
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1167
2021-03-31 16:17:23 +03:00
Aliaksandr Valialkin
b873b965af
app/vmagent/remotewrite: reduce memory usage when samples with big number of labels are sent to remote storage
2021-03-31 00:45:42 +03:00
Aliaksandr Valialkin
d8c769e6af
app/vmselect: add -search.maxStatusRequestDuration
command-line flag for limiting the duration of requests to /api/v1/status/*
and /api/v1/series/count
2021-03-30 21:39:32 +03:00
Aliaksandr Valialkin
0e29d48628
app/vmselect: remove mentioning for -search.storageTimeout
from command-line flag descriptions
...
This is a follow-up for ecfd6fe78d
2021-03-30 15:07:40 +03:00
Aliaksandr Valialkin
ecfd6fe78d
app/vmselect: remove -search.storageTimeout
command-line flag, since it has the same meaning as -search.maxQueryDuration
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711
2021-03-30 14:54:36 +03:00
Aliaksandr Valialkin
7bafaad46d
app/vmselect: prevent from possible incomplete query results after timed out query
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711
2021-03-30 13:34:57 +03:00
Aliaksandr Valialkin
9b4e608199
app/vminsert: return back type
label to per-tenant metric `vm_tenant_inserted_rows_total
...
The `type` label has been removed in the commit 022fe4efd0
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/932
2021-03-30 00:06:59 +03:00
Aliaksandr Valialkin
0a8f0a4e2f
all: increase minimum supported Go version for building VictoriaMetrics components from v1.14 to v1.15
...
This is needed after the commit c0ac740f93
, which uses URL.Redacted() method,
which has been added in v1.15.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1147
2021-03-29 23:06:36 +03:00
Aliaksandr Valialkin
3f0534c7f8
app/vminsert/opentsdbhttp: remove rowsInsertedTenant metric, since it is substituted by tenantmetrics.RowsInsertedByTenant in the commit 022fe4efd0
2021-03-29 12:16:13 +03:00
Nikolay
022fe4efd0
makes tenant metrics names consistent with enterprise version ( #1152 )
2021-03-29 11:59:04 +03:00
Aliaksandr Valialkin
8a3939e93e
docs: document that vmagent drops data blocks when remote storage replies with 400 and 409 http status codes
...
This is a follow up for 1b7dc1e5a5
.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1149
2021-03-26 14:43:41 +02:00
Aliaksandr Valialkin
3fedf680f8
app/vmagent/remotewrite: remove superflouos code after 1b7dc1e5a5
2021-03-26 14:01:37 +02:00
Nikolay
15ff1f3a94
Adds blocks drop ( #1151 )
...
* adds blocks drop at 400 BadRequest status code
recieved from remote storage,
not expected that remote storage will be able to handle it on retry
* removes error logging for dropped blocks,
its expected error
2021-03-26 14:01:37 +02:00
Aliaksandr Valialkin
698ccca6ad
app/vmselect/prometheus: explain why partial responses are disallowed during data export
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1148
2021-03-25 18:39:07 +02:00
Aliaksandr Valialkin
b91e18af0e
docs/vmagent.md: add an example for -remoteWrite.label
2021-03-25 17:59:30 +02:00
Aliaksandr Valialkin
82047be90b
docs: add a link to the repository from build instruction for all the VictoriaMetrics components
2021-03-25 17:16:55 +02:00
Aliaksandr Valialkin
6de36b4e21
docs/vmagent.md: cosmetic fixes
2021-03-25 17:11:10 +02:00
Aliaksandr Valialkin
1b13d02728
docs/vmagent.md: cosmetic fixes
2021-03-25 16:54:37 +02:00
Aliaksandr Valialkin
155cd90fc8
docs/vmagent.md: typo fix: tupically -> typically
2021-03-25 16:48:27 +02:00
Aliaksandr Valialkin
450e23533d
docs/vmalert.md: remove misleading -evaluationInterval=3s
from example config args
...
3s evaluation interval is too small for practical setups. It can result in increased load on datasource.
So it is better to remove it from example config args, which are usually copy-pasted by novice users.
2021-03-25 15:31:10 +02:00
Aliaksandr Valialkin
b473c21915
app/vmselect/promql: do not merge time series during requests to /api/v1/query
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1141
2021-03-25 13:56:23 +02:00
Aliaksandr Valialkin
6b1f807418
app/vmagent: add -promscrape.consul.waitTime
command-line flag for configuring Consul service discovery wait time
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1144
2021-03-23 19:34:12 +02:00
Aliaksandr Valialkin
2041008d64
app/vmagent: mention -remoteWrite.maxDiskUsagePerURL in the descriptio of -remoteWrite.tmpDataPath
flag
2021-03-23 16:38:41 +02:00
Aliaksandr Valialkin
d1e773266f
app/vmselect/graphite: accept and enforce extra_label
in all the Graphite APIs
2021-03-23 15:30:15 +02:00
Aliaksandr Valialkin
27bb614016
app/vmselect: move getEnforcedTagFiltersFromRequest to searchtuils, since it will be used in Graphite functions soon
2021-03-23 14:57:57 +02:00
Aliaksandr Valialkin
7f9863254d
app/vmselect: improve description for -search.maxPointsPerTimeseries
command-line flag
2021-03-22 16:44:36 +02:00
Aliaksandr Valialkin
6ebf18ab97
app/{vminsert,vmagent}: use Influx field as metric name if measurement is empty and -influxSkipSingleField command-line is set
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1139
2021-03-22 13:55:11 +02:00
Aliaksandr Valialkin
92531a38c4
app/vmselect/promql: increment key prefix for faster reset for rollup result cache
2021-03-22 11:59:39 +02:00
Aliaksandr Valialkin
799461d8bf
app/vmselect/graphite: follow-up after 529d7be26b
2021-03-18 16:33:00 +02:00
Nikolay
8807410a00
changes metricsFind api ( #1137 )
...
it should be able mitigate crash if label value contains *,[ or { symbols
2021-03-18 16:32:59 +02:00
Aliaksandr Valialkin
667d129e1e
app/vmselect/promql: typo fix after 9666834045
2021-03-17 15:11:54 +02:00
Aliaksandr Valialkin
69201806f8
app/vmselect/promql: merge adjancent buckets with the smallest summary number of hits in buckets_limit()
function
...
This should improve accuracy for the returned buckets
2021-03-17 14:31:29 +02:00
Aliaksandr Valialkin
8ef1184adf
app/vmstorage: add vm_index_search_duration_seconds
histogram for monitoring the performance of index search
2021-03-17 01:13:15 +02:00
Aliaksandr Valialkin
5e77a939c2
all: make go vet
happy
2021-03-17 00:48:44 +02:00
Aliaksandr Valialkin
b997f4a418
all: make golangci-lint happy after the commit 6378205415
2021-03-17 00:24:31 +02:00
Aliaksandr Valialkin
8005ba26b9
lib/netutil: enable IPv6 UDP listening if -enableTCP6
command-line flag is passed to VictoriaMetrics
...
This is a follow-up for 18cfc4be7b
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1131
2021-03-17 00:19:30 +02:00
Aliaksandr Valialkin
05e301cfa0
app/vmselect/netstorage: reduce mutex contention when unpacking data on a system with high number of CPU cores
2021-03-16 21:52:05 +02:00
Aliaksandr Valialkin
3caac5edd4
Makefile: prepare vmutils-windows-*.zip archive on make release-vmutils
command
...
The archive contains the following executables for Windows:
* vmagent
* vmalert
* vmauth
* vmctl
Other components - vmbackup, vmrestore, victoria-metrics - aren't supported for Windows yet
2021-03-16 20:54:10 +02:00
Aliaksandr Valialkin
f54ece438d
app/vmselect/promql: do not crash if histogram_over_time()
function name contains uppercase letters such as Histogram_over_time()
2021-03-16 12:23:55 +02:00
Aliaksandr Valialkin
d074326970
app/vmstorage: add -logNewSeries
command-line flag for determining the source of series churn rate
2021-03-15 22:40:28 +02:00
Aliaksandr Valialkin
e2717d84c0
all: various fixes in command-line flag descriptions
2021-03-15 22:03:49 +02:00
Aliaksandr Valialkin
776b8b32ca
app/{vminsert,vmagent}: a follow-up for b1aa8c3d8f
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1124
2021-03-15 22:03:49 +02:00
Nikolay
a843dc0219
adds fake response for telegraph queries ( #1130 )
...
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1124
2021-03-15 22:03:49 +02:00
Aliaksandr Valialkin
30ab0eec27
app/vmselect/promql: reduce overhead on scrape interval estimation
...
It should be enough to use the first 20 datapoints instead of 100 datapoints for scrape interval estimation.
2021-03-15 20:32:35 +02:00
Aliaksandr Valialkin
8844b38745
app/vmselect/promql: fix tests after 2dae0a2c47
2021-03-15 20:20:27 +02:00
Aliaksandr Valialkin
6b9bba7448
app/vmselect: add round_digits
query arg to /api/v1/query
and /api/v1/query_range
handlers for limiting the number of decimal digits after the point
2021-03-15 12:38:59 +02:00
Aliaksandr Valialkin
afa5b58c2d
app/vminsert/opentsdbhttp: typo fix in vm_rows_inserted_total{type="opentsdbhttp"}
metric: opentsdb-http -> opentsdbhttp
2021-03-12 15:59:31 +02:00
Aliaksandr Valialkin
b6dcb37fca
app/vmselect/prometheus: do not include datapoints with timestamps matching t-d
when returning results from /api/v1/query?query=m[d]&time=t
as Prometheus does
2021-03-12 12:18:37 +02:00
Aliaksandr Valialkin
60e0280a94
lib/promscrape: add ability to configure proxy options via proxy_tls_config
, proxy_basic_auth
, proxy_bearer_token
and proxy_bearer_token_file
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1116
2021-03-12 03:36:11 +02:00
Ihor Borodin
933de6b9b1
Fixing examples of external.alert.source in documentation ( #1120 )
...
* Fixing examples of external.alert.source in documentation
2021-03-10 12:08:22 +02:00
Aliaksandr Valialkin
942890b1bb
app/vmselect/promql: follow up for 433fff0006
2021-03-09 12:54:23 +02:00
Nikolay
28e450cd7c
duplicate timeseries fix for prometheus_buckets function ( #1119 )
...
* try fix for prometheus_buckets
* merge possible end of the bucket collision
2021-03-09 12:54:21 +02:00
Aliaksandr Valialkin
502fab797a
lib/promscrape: add scrape_offset
option to scrape_config
...
This option can be used for specifying the particular offset per each scrape interval for target scraping
2021-03-08 11:59:32 +02:00
Aliaksandr Valialkin
26cb6f8861
app/vmauth: allow using regexps in url_map
paths
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1112
2021-03-05 18:53:40 +02:00
Aliaksandr Valialkin
5277507932
app/vmselect/promql: add histogram_avg()
, histogram_stddev()
and histogram_stdvar()
functions to MetricsQL
2021-03-04 14:12:52 +02:00
Aliaksandr Valialkin
133fb9fc00
lib/promscrape: add -promscrape.cluster.replicationFactor
command-line flag for replicating scrape targets among vmagent
instances in the cluster
2021-03-04 10:21:27 +02:00
Aliaksandr Valialkin
d109e17f46
all: bump minimum supported Go version from 1.13 to 1.14
2021-03-03 15:58:17 +02:00
Aliaksandr Valialkin
3c11ce9356
docs/vmagent.md: remove outdated suggestion for determining labels that lead to duplicate targets
...
The original labels for duplicate targets is already printed in the error message starting from 71ea4935de
2021-03-03 12:28:50 +02:00
Roman Khavronenko
219291e084
Vmagent docs upd ( #1104 )
...
* vmagent: port changes from https://github.com/VictoriaMetrics/VictoriaMetrics.github.io/pull/1
Thanks to @dereksfoster99 for this patch!
* vmagent: reword to make the meaning clear
2021-03-03 10:53:41 +02:00
Aliaksandr Valialkin
2ecee0515a
app/vmalert/README.md: sync with docs/vmalert.md
2021-03-03 10:42:54 +02:00
Aliaksandr Valialkin
c67a07b469
lib/handshake: log read/write operation duration on connection errors
...
This improve debuggability of network errors
2021-03-02 21:20:20 +02:00
Aliaksandr Valialkin
d9e8af0e8f
docs: actualize -help
output
2021-03-01 17:02:05 +02:00
Aliaksandr Valialkin
323af49234
app/vmagent/remotewrite: clarify -remoteWrite.flushInterval
flag description
2021-03-01 11:51:08 +02:00
Aliaksandr Valialkin
b8dd5e8292
app/vmagent/remotewrite: fix rate limiting logic for -remoteWrite.url
2021-03-01 00:59:57 +02:00
Aliaksandr Valialkin
d0be1f6f49
app/vmagent: remove data race when applying rate limits to -remoteWrite.url
with multiple queues
2021-03-01 00:28:20 +02:00
Aliaksandr Valialkin
17eb29206d
docs/vmbackup.md: clarify docs on vmagent
clustering
2021-02-28 21:59:55 +02:00
Aliaksandr Valialkin
3b1c2f03c3
app/vmselect/querystats: show the number of matching queries in the top by average duration and in the top by summary duration
...
This should help debugging slow queries.
2021-02-28 19:42:29 +02:00
Aliaksandr Valialkin
9a2bf65134
lib/promscrape: add ability to spread scrape targets among multiple vmagent instances
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1084
2021-02-28 18:40:42 +02:00
Nikolay
b52d1e4f19
adds query params for vmalert ( #1094 )
...
remoteWrite.url now accepts query params at provided url
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1087
2021-02-28 14:12:34 +02:00
Aliaksandr Valialkin
3721c5353a
app/vmbackupmanager: add docs; the vmbackupmanager is available as a part of enterprise subscription
2021-02-27 02:18:33 +02:00
Aliaksandr Valialkin
7f1302688f
lib/fs: follow-up after f3a03c4164
2021-02-27 01:09:37 +02:00
Nikolay
d88fa5ebe4
Adds windows build ( #1040 )
...
* fixes windows compilation,
adds signal impl for windows,
adds free space usage for windows,
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1036
NOTE victoria metrics database still CANNOT work under windows system,
only vmagent is supported.
To completly port victoria metrics, you have to fix issues with separators,
parsing and posix file removall
* rollback separator
* Adds windows setInformation api,
it must behave like unix, need to test it.
changes procutil
* check for invlaid param
* Fixes posix delete semantic
* refactored a bit
* fixes openbsd build
* removed windows api call
* Fixes code after windows add
* Update lib/procutil/signal_windows.go
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2021-02-27 01:06:22 +02:00
Nikolay
673b10dd7f
adds enforced tag filters into cache key ( #1095 )
2021-02-27 00:23:38 +02:00
Aliaksandr Valialkin
e38ca28d99
app/vmselect/promql: increase accuracy for buckets_limit()
function for small limits by skipping the first and the last buckets during merge
...
The first and the last buckets are usually `[0 ... leMin]` and `(leMax ... +Inf)`. If they are merged with adjancent buckets,
then the resulting accuracy can suffer.
2021-02-26 22:56:04 +02:00
Aliaksandr Valialkin
0144b164c7
app/vmselect: add sign(q)
and clamp(q, min, max)
functions, which will be added in the upcoming Prometheus release
...
See https://twitter.com/roidelapluie/status/1363428376162295811
The `last_over_time(m[d])` function already exists in MetricsQL.
2021-02-24 17:26:27 +02:00
Aliaksandr Valialkin
1b90a091cf
app/vmselect/netstorage: cleanup after 4805b80977
2021-02-24 11:43:09 +02:00
weng zhao
4805b80977
vmselect: fix vmselect cache partial result when timeout ( #1085 )
2021-02-24 11:39:56 +02:00
Aliaksandr Valialkin
d16effc29e
app/vmselect/promql: properly calculate histogram_quantile() over zero buckets and only a single non-zero
le="+Inf"` bucket like Prometheus does
2021-02-24 00:41:30 +02:00
Aliaksandr Valialkin
5dbe88a1c6
app/vmselect/promql: add increase_pure()
function to MetricsQL
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/962
2021-02-22 19:15:31 +02:00
Aliaksandr Valialkin
197ecca426
lib/promrelabel: add more optimizations for relabeling for common cases
2021-02-22 16:36:54 +02:00
Aliaksandr Valialkin
72eef964d9
app/vmagent: properly perform graceful shutdown, which was broken in the commit 1d1ba889fe
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1065
2021-02-19 00:34:17 +02:00
Aliaksandr Valialkin
41d3ff4f2b
app/vmagent: fix scrape config example for scrape_align_interval option
2021-02-18 23:57:45 +02:00
Aliaksandr Valialkin
502d0e2524
lib/promscrape: add scrape_align_interval config option into scrape config
...
This option allows aligning scrapes to a particular intervals.
2021-02-18 23:53:04 +02:00
Aliaksandr Valialkin
bd1d906eee
app/vmselect/promql: reduce the probability of duplicate time series
errors when querying Kubernetes metrics
2021-02-18 22:08:13 +02:00
Aliaksandr Valialkin
902a4f6486
app/victoria-metrics: remove accidentally added single-node tests
2021-02-18 18:33:03 +02:00
Aliaksandr Valialkin
901710b9e2
app/vmalert: add missing multiarch Dockerfile
2021-02-18 15:23:57 +02:00
Aliaksandr Valialkin
1dad7ecf54
app/victoria-metrics/testdata: add a test for {__graphite__="foo.*.bar"} selector
2021-02-17 21:53:24 +02:00
Aliaksandr Valialkin
9c81429299
app/vmagent/remotewrite: cleanup after 1d1ba889fe
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1065
2021-02-17 21:43:48 +02:00
Nikolay
40973eda1c
adds pushback for fastqueue, ( #1075 )
...
during shutdown currently sending block was lost,
now its pushed back to fast queue and will be flushed on disk,
it may lead to data duplication.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1065
2021-02-17 21:43:46 +02:00
Aliaksandr Valialkin
83da939947
app/vmstorage: export vm_composite_filter_success_conversions_total and vm_composite_filter_missing_conversions_total metrics
2021-02-17 19:13:49 +02:00
Aliaksandr Valialkin
3062ff0fdb
app/vmselect: export per-tenant stats on the number of requests and the cumulative request duration
...
The metrics are:
- vm_vmselect_http_requests_total{accountID="...",projectID="..."} - the total number of select requests per each tenant
- vm_vmselect_http_duration_ms_total{accountID="...",projectID="..."} - the total duration in milliseconds for per-tenant select requests
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/932
2021-02-16 23:30:29 +02:00
Aliaksandr Valialkin
667a7594b7
docs: rename vmbackuper to vmbackupmanager
2021-02-16 22:01:21 +02:00
Aliaksandr Valialkin
73c9da16b8
app/vmselect/netstorage: reuse timeseriesWork objects in order to reduce memory allocations
2021-02-16 16:09:00 +02:00
Aliaksandr Valialkin
a3a09a3c6e
app/vmselect/netstorage: use unsafe string as a key for a map when the map already contains the given key
...
This should prevent from a memory allocation and a string copy.
2021-02-16 15:46:31 +02:00
Aliaksandr Valialkin
46e98ed490
vendor: update github.com/VictoriaMetrics/metrics from v1.13.1 to v1.14.0
...
The new version switches from log-linear histograms to log-based histograms,
which provide up to 3.6 times better accuracy.
2021-02-15 15:11:15 +02:00
Aliaksandr Valialkin
c769f8321d
deployment/docker: embed tzdata into prod Go app instead of installing it into base docker image
...
While this increases app size by 700Kb, this allows using -loggerTimezone in a scratch base image
See https://github.com/golang/go/issues/38017
2021-02-12 04:56:27 +02:00
Aliaksandr Valialkin
ff7850aec0
deployment/docker: use docker buildx
for creating multiarch builds
...
See https://github.com/docker/buildx/
2021-02-12 04:35:35 +02:00
Aliaksandr Valialkin
ee4288987b
app/vmselect/prometheus: treat match
query arg in the same way as match[]
query arg
2021-02-11 15:01:55 +02:00
Aliaksandr Valialkin
9e88ff3075
app/vmauth: add ability to route requests from a single users to multiple targets depending on the requested path
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1064
2021-02-11 12:41:38 +02:00
Aliaksandr Valialkin
54d2f67924
app/vmselect/promql: properly make copies of EvalConfig
2021-02-11 12:41:37 +02:00
Aliaksandr Valialkin
5973fd4067
app/vmselect/promql: make a copy of EvalConfig when executing q1
and q2
in parallel for q1 binary_op q2
...
This should prevent from data races if the underlying functions modify EvalConfig contents.
2021-02-10 23:06:58 +02:00
Aliaksandr Valialkin
985c3e301d
app/vmselect: parallelize q1 <binary_op> q2
queries by running q1
and q2
in parallel
...
This should reduce query execution times.
2021-02-10 23:00:56 +02:00
Aliaksandr Valialkin
08f21d8761
app/vmstorage: export vm_composite_index_min_timestamp metric
2021-02-10 17:14:00 +02:00
Aliaksandr Valialkin
148422bcba
lib/storage: disable composite index usage when querying old data
2021-02-10 14:57:58 +02:00
Aliaksandr Valialkin
fa0ef143b1
lib/storage: optimize search by label filters matching big number of time series
2021-02-10 00:46:17 +02:00
Aliaksandr Valialkin
e8ee9fa7fe
app/vmstorage: export missing vm_cache_size_bytes
metrics for indexdb and data caches
2021-02-09 00:49:58 +02:00
Aliaksandr Valialkin
35eb04b7dd
docs/vmctl.md: fix title, so it is properly displayed in the header of https://victoriametrics.github.io/
2021-02-04 20:20:38 +02:00
Roman Khavronenko
2aa37b0450
vmalert: mention -datasource.appendTypePrefix
in README ( #1052 )
2021-02-03 23:48:45 +02:00
Dmitry Shevchuk
007fd6ce9c
Adds ability to query right vmselect endpoint based on the query type ( #1050 )
...
* Adds ability to query right vmselect endpoint based on the query type
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
2021-02-03 23:48:44 +02:00
Aliaksandr Valialkin
88ee836d0c
docs/vmalert.md: mention that type
option can be set at group level additionally to rule level
2021-02-03 21:12:39 +02:00
Aliaksandr Valialkin
a3aa9bdc9f
app/vmagent: add Advanced usage
section with the description for all the command-line flags
2021-02-03 21:03:52 +02:00
Aliaksandr Valialkin
8de3a329ff
app/vmselect: deprecate -search.treatDotsAsIsInRegexps
in favor to {__graphite__="foo.*.bar"}
syntax
2021-02-03 20:36:42 +02:00
Aliaksandr Valialkin
8249f13104
app/vmselect,lib/storage: properly parse Graphite selectors with inner wildcards
...
Example: foo{bar{x,yz},a[b-c],*de}
2021-02-03 20:16:28 +02:00
Aliaksandr Valialkin
4b930b9ffe
app/vmselect: add ability to set Graphite-compatible filter via {__graphite__="foo.*.bar"}
syntax
2021-02-03 01:17:19 +02:00
Aliaksandr Valialkin
03da0b728c
docs: increase heading sizes in vmagent, vmauth, vmbackup and vmrestore docs, so they match the heading sizes in VictoriaMetrics docs
2021-02-01 19:44:58 +02:00
Aliaksandr Valialkin
f4a4665857
docs/vmctl.md: update build instructions after the migration from github.com/VictoriaMetrics/vmctl to github.com/VictoriaMetrics/VictoriaMetrics/app/vmctl
2021-02-01 19:44:57 +02:00
Aliaksandr Valialkin
7d23f3ff3a
app/vmselect: add ability to set an additional label filters via extra_label
query arg
2021-02-01 17:42:35 +02:00
Aliaksandr Valialkin
6811445b64
docs: document ability to query Graphite datasource from vmalert
2021-02-01 15:28:31 +02:00
Nikolay
b8bc1c2e0f
Graphite vmalert wip ( #112 )
...
* init implementation for graphite alerts
* adds graphite support for vmalert
* small fix
* changes vmalert graphite api with type
* updates tests
* small fix
* fixes graphite parse
* Fixes graphite from time
2021-02-01 15:28:30 +02:00
Aliaksandr Valialkin
fdf9de98f8
app/vmagent: add -remoteWrite.roundDigits command-line option for limiting the number of digits after the point for stored values
...
This commit also adds --vm-round-digits command-line option to vmctl tool.
2021-02-01 14:42:15 +02:00
Aliaksandr Valialkin
05474aaa29
app/vmctl: fix make check-all
warnings
2021-02-01 01:31:45 +02:00
Aliaksandr Valialkin
7f4fb34182
app/vmctl: move vmctl code from github.com/VictoriaMetrics/vmctl
...
It is better developing vmctl tool in VictoriaMetrics repository, so it could be released
together with the rest of vmutils tools such as vmalert, vmagent, vmbackup, vmrestore and vmauth.
2021-02-01 01:18:39 +02:00
Aliaksandr Valialkin
5328a102e0
app/vmselect: unconditionally deny partial responses from /api/v1/export*
...
It is expected that `/api/v1/export*` returns full data, so there is no sense in partial responses there.
2021-01-27 14:39:53 +02:00
Aliaksandr Valialkin
4b324da947
all: consistently use timers from timerpool
2021-01-27 00:40:39 +02:00
Aliaksandr Valialkin
29bf531f7d
app/vmagent: add -remoteWrite.rateLimit command-line flag for limiting data rate to remote storage
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1035
2021-01-27 00:40:39 +02:00
weng zhao
2a8a34ea05
vmalert: add option datasource.queryStep to allow user to address the inconsistency between grafana dashboards(query_range with step 15s usually) and ALERTS ( #1027 )
...
Co-authored-by: zhao.weng <zhao.weng@shopee.com>
2021-01-26 16:38:20 +02:00
Aliaksandr Valialkin
44c74f1e79
app/vmselect/promql: improve documentation for -search.maxPointsPertimeseries
command-line flag
...
This should reduce incorrect usage and assumptions for this flag.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1020
2021-01-22 13:00:35 +02:00
Aliaksandr Valialkin
e55205220b
app/vmselect: add -search.maxStepForPointsAdjustment
command-line flag, which can be used for disabling adjustment for points returned from /api/v1/query_range
handler if they have timestamps closer than -search.latencyOffset
to the current time
2021-01-19 22:57:50 +02:00
Aliaksandr Valialkin
5856611291
app/vmselect/graphite: extract getCanonicalPath() function from loop body inside getCanonicalPaths()
2021-01-18 17:31:27 +02:00
Aliaksandr Valialkin
5640e6cbca
docs/vmagent.md: follow-up for 184a659c5f
2021-01-13 13:54:28 +02:00
Aliaksandr Valialkin
c5bdab5a4c
app/vmselect/promql: add ability to pass multiple labels to sort_by_label
and sort_by_label_desc
functions
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/992
2021-01-13 12:43:47 +02:00
Aliaksandr Valialkin
8cae98aa78
app/vmselect/promql: properly parse escaped multibyte utf8 code sequences in metric names and labels names
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/990
2021-01-13 10:59:32 +02:00
Nikolay
821492bc0b
adds extra_label to all import apis ( #1007 )
...
* adds extra_label to all import apis,
changes priority for extra_label - now it has priority over original labels
* Update README.md
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
* Update README.md
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
* adds extra labels to vmagent import api
changes order for adding labels, now its added after user values
* adds tests for extra_label
* import fix
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2021-01-13 01:07:24 +02:00
Aliaksandr Valialkin
df6e399f73
app/vmselect/promql: add tfirst_over_time(m[d])
and tlast_over_time(m[d])
MetricsQL functions for returning timestamps for the first and the last samples in m
over d
2021-01-12 16:12:47 +02:00
Nikolay
9f0a4fd00e
Fixes error handling for promscrape.streamParse ( #1009 )
...
properly return error if client cannot read data,
properly suppress scraper errors
2021-01-12 13:35:09 +02:00
Roman Khavronenko
304512b668
vmalert-989: return non-empty result in template func query
stub to pass validation ( #1002 )
...
On templates validation stage vmalert does not acutally send queries, so for complex
chained expression validation may fail. To avoid this, we add a blank sample in response
so validation can pass successfully. Later, during the rule execution, stub will be replaced
with real `query` function.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/989
2021-01-11 12:59:33 +02:00
Aliaksandr Valialkin
4ee53c3961
all: use net.Dial
instead of fasthttp.Dial
, because fasthttp.Dial
limits the number of concurrent dials to 1000
2021-01-11 12:52:51 +02:00
Aliaksandr Valialkin
d5a2b120e9
app/vmstorage: disable final merge by default, since it may result in high disk IO and CPU usage without measurable benefits such as increased query performance and reduced disk space usage
2021-01-08 00:12:12 +02:00
Aliaksandr Valialkin
47872ada7e
app/vmselect/promql: do not ajdust offset
value provided in the query
...
Previously it could be modified in order to improve response cache hit ratio.
This is unneeded, since cache hit ratio should remain good because the query time range
should be already aligned to multiple of `step` values.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/976
2020-12-27 14:10:15 +02:00
Aliaksandr Valialkin
5bbf200de2
app/vmselect: add per-tenant /api/v1/status/top_queries
handler
2020-12-27 12:53:50 +02:00
Aliaksandr Valialkin
0e739efc88
app/vmselect/promql: simplify defer call for querystats.RegisterQuery
2020-12-27 12:07:56 +02:00
Aliaksandr Valialkin
44932098b5
app/vmselect/querystats: reduce the default number of last queries to track from 100K to 20K
...
This should reduce memory usage in constrained environments
2020-12-25 17:40:32 +02:00
Aliaksandr Valialkin
e6deb39064
app/vmselect: refactor /api/v1/stats/top_queries
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/907
2020-12-25 17:24:25 +02:00
Nikolay
76d092c091
Adds query stats handler ( #945 )
...
* Adds query stat handler,
for query and query_range api, victoriametrics tracks query execution time,
stats are expored at /api/v1/status/queries endpoint with topN param
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/907
* fixed query stats bugs
* improves queryStats tracker
* improves query stat
* small fix
* fix tests
* added more tests
* fixes 386 tests
* naming fixes
* adds drop for outdated records
2020-12-25 17:24:24 +02:00
Nikolay
14915071d6
adds escape for CRLF ( #984 )
...
at external.alert.source - \n and \r symbols was url encoded, instead of direct usage.
replace it from "\n" to `\n` allows to skip url encoding.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/890
2020-12-25 11:06:47 +02:00
Aliaksandr Valialkin
b480585905
app/vmalert: typo fix in descriptions for notifier.basicAuth.username and notifier.basicAuth.password command-line flags
2020-12-24 12:49:40 +02:00
Nikolay
0b87f02602
fixes panic ( #979 )
...
* fixes panic
https://github.com/VictoriaMetrics/helm-charts/issues/89
* add fast-path
* Apply suggestions from code review
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2020-12-24 09:01:40 +02:00
Aliaksandr Valialkin
d8511b6651
docs: mention that it is possible to set multiple -notifier.tlsInsecureSkipVerify
command-line flags for vmalert
...
See c3a92968343c2b3619f1ab935702d0e9b3a46733
2020-12-22 22:32:56 +02:00
Nikolay
67e470e598
changes vmalert notifier flag, ( #978 )
...
fixes issue with notifier insecure setting, now its possible to use multiple notifier.tlsInsecureSkipVerify multiple time.
2020-12-22 22:27:03 +02:00
Roman Khavronenko
9ce8b36d2a
vmalert-974: fix order for labels templating ( #975 )
...
The change fixes bug caused by 3adf8c5a6f
.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/974
2020-12-19 14:21:27 +02:00
Aliaksandr Valialkin
262cf81757
app/vmselect: properly parse negative combined offsets such as -1h2m3s
...
Previously such offsets were parsed as `-1h + 2m + 3s`. Now they are parsed as `-(1h + 2m + 3s)`.
2020-12-19 01:25:03 +02:00
Aliaksandr Valialkin
49e800ba55
app/vmagent: add vmagent_remotewrite_blocks_sent_total
and vmagent_remotewrite_bytes_sent_total
metrics per each -remoteWrite.url
2020-12-15 20:41:08 +02:00
Aliaksandr Valialkin
9ab7ca1133
docs/vmagent.md: typo fix: pearsed->parsed
2020-12-15 19:03:35 +02:00
Aliaksandr Valialkin
11674a9b76
docs/vmagent.md: mention that sample_limit
option has no sense when stream parsing is enabled
2020-12-15 18:44:19 +02:00
Aliaksandr Valialkin
8d1031c29a
app/vmselect/promql: return expected increase()
result for the first point on the graph with value not exceeding 100
2020-12-15 14:10:50 +02:00
Nikolay
7064c4eb8e
adds new Array Flags ( #965 )
...
* adds ArrayDuration and ArrayBool flags,
makes sendTimeout and tlsInsecure configurable per remoteWrite url
* added backward compatibility testcases for ArrayDuration and ArrayBool
* fixes bool flag
* fixes test cases
2020-12-15 12:59:33 +02:00
Roman Khavronenko
9f578e389c
vmalert: add function "query", "first" and "value" to alert templates functions ( #960 )
...
The commit adds a support for template function `query`,
`first` and `value`. The function `query` executes
a MetricsQL query for active alerts. In vmalert we
update templates on every evaluation for active alerts
to keep them up to date. With `query` func it may become
a perf issue since it will fire a query on every execution.
We should keep it in mind for now.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/539
2020-12-14 20:12:16 +02:00
Aliaksandr Valialkin
a2eb451de4
app/{vmagent,vminsert}: follow-up for ce8c2dd1f1
: return /targets
page in HTML when requested via web browser
2020-12-14 14:13:01 +02:00
Nikolay
324e3aa1a5
Changes targets api ( #961 )
...
* changes /targets api
adds html response if requester accepts text/html,
adds quick template for /targets api,
fixes pathPrefix for / requests
* changes namings
* renamed targets file
* Update app/victoria-metrics/main.go
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
* adds trimspace to qtpl,
moves content-type for targets response closer to writer
* fixes bug with prefix
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2020-12-14 14:13:00 +02:00
Aliaksandr Valialkin
fc82c22e50
docs: consistently use links to https://victoriametrics.github.io for documentation references
2020-12-11 21:09:17 +02:00
Aliaksandr Valialkin
d6f9bf2d19
app/vmselect/graphite: properly handle wildcards and charsets inside curly braces
...
For example, `foo{bar*,[a-f]a*b}` should match `foobar`, `foobar123`, `foofab`, etc.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/952
2020-12-11 17:26:32 +02:00
Aliaksandr Valialkin
9e79fc27c8
app/vminsert/netstorage: properly update vm_rpc_rerouted_rows_processed_total
metric
...
Previously this metric wasn't updated because of improper defer call.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/955
Thanks to @xemxx for spotting the bug.
2020-12-11 13:07:05 +02:00
Aliaksandr Valialkin
1a237c6903
all: properly handle CPU limits set on the host system/container
...
This can reduce memory usage on systems with enabled CPU limits.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/946
2020-12-08 21:07:03 +02:00
Aliaksandr Valialkin
bd8eef2528
app/vmselect/promql: do not reduce lookbehind window for any_rollup_func(m)
to -search.maxStalenessInterval
. It should equal to step
value passed to /api/v1/query_range
as most users expect
2020-12-08 15:17:05 +02:00
Aliaksandr Valialkin
7bdf07883b
app/{vmalert,vmagent}: skip empty values in -remoteWrite.label
and -label
lists
2020-12-08 14:54:02 +02:00
Aliaksandr Valialkin
9660774fd1
app/vmselect/graphite: remove duplicate name
tag from /tags/autoComplete/tags
handler
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/942
2020-12-07 01:10:02 +02:00
Aliaksandr Valialkin
d242c2f2bd
app/vmselect/promql: add count_eq_over_time(m[d], N)
and count_ne_over_time(m[d], N)
for calculating the number of samples in m
over d
that are equal / not equal to N
2020-12-05 12:31:01 +02:00
Aliaksandr Valialkin
bdac2171f1
all: do not print usage info for all the flags when incorrect command-line flag is passed
...
This should improve usability for VictoriaMetrics apps that have big number of command-line flags,
i.e. all the apps.
2020-12-03 21:46:19 +02:00
Aliaksandr Valialkin
8cf76d8747
app/vmselect/promql: add label_uppercase(q, label1, ... labelN)
and label_lowercase(q, label1, ... labelN)
functions
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/936
2020-12-03 21:46:18 +02:00
Aliaksandr Valialkin
11bbb3552d
app/vmselect/promql: make fmt
2020-12-02 21:34:15 +02:00
Aliaksandr Valialkin
9e98a8f3d3
app/vmselect/promql: return nan
from minute(m)
when m
equals to nan
...
This aligns VictoriaMetrics behaviour with Prometheus behaviour.
The issue has been spotted in https://promlabs.com/promql-compliance-test-results/2020-12-01/victoriametrics/
2020-12-02 20:16:40 +02:00
Aliaksandr Valialkin
def513355e
app/vmselect/promql: do not return 0
value from sum_over_time(m[d])
when there are no samples on the given d
window.
...
This aligns the behaviour of `sum_over_time()` with other `_over_time()` functions and with Prometheus behavior.
2020-12-02 13:12:33 +02:00
Aliaksandr Valialkin
490c70a958
app/vmselect: return metric
values from time() cmp_op metric
query when cmp_op
comparison is true
...
This aligns MetricsQL behavior to Prometheus' one.
The issue has been identified at https://promlabs.com/promql-compliance-test-results/2020-12-01/victoriametrics/
2020-12-02 12:09:40 +02:00
Aliaksandr Valialkin
4ef7158e89
app/vmselect/promql: return nan
from a >bool b
if a
is nan
in the same way as Prometheus does
2020-12-02 00:28:56 +02:00
Aliaksandr Valialkin
adf45b730c
app/vmselect/searchutils: return elapsed time in Deadline.String() output
...
This should improve debuggability for error messages containing Deadline.String() output
2020-12-01 00:14:36 +02:00
Aliaksandr Valialkin
1dce37b2fa
app/vmbackup/snapshot: add missing status code check for the returned response when working with snapshot API
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/929
2020-11-30 14:49:29 +02:00
Aliaksandr Valialkin
8b5a38376d
app/vmbackup/snapshot: log url and response body on failed JSON response parsing
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/929
2020-11-29 12:16:08 +02:00
Nikolay
e4e33cb757
fixes checksum calculation ( #928 )
...
* fixes checksum calculation,
'for' rule param wasnt marshal properly during checksum calculation
* fixes error
2020-11-29 09:50:57 +02:00
Aliaksandr Valialkin
3f52e59efe
app/{vmagent,victoria-metrics}: add -dryRun
option and make more clear handling for -promscrape.config.dryRun
2020-11-25 23:01:39 +02:00
Aliaksandr Valialkin
ed06990609
app/vmagent: do not enable -promscrape.config.strictParse when -dryRun command-line flag is set
...
Users can specify -promscrape.config.strictParse if -promscrape.config shouldn't contain unknown config entries
2020-11-25 22:27:41 +02:00
BigFish
3159b41689
Update main.go ( #922 )
...
fix spelling mistake
2020-11-24 12:36:47 +02:00
Aliaksandr Valialkin
2cc288c023
app/vmbackup: cosmetic fixes
2020-11-23 17:10:13 +02:00
Aliaksandr Valialkin
e1297c0b78
app/vmselect: add /tags/delSeries
handler from Graphite Tags API
...
See https://graphite.readthedocs.io/en/stable/tags.html#removing-series-from-the-tagdb
2020-11-23 15:32:14 +02:00
Aliaksandr Valialkin
3d2ce31cad
app/vmselect/netstorage: code readability improvement: rename *RequestErrors to *Errors
2020-11-23 15:00:15 +02:00
Aliaksandr Valialkin
433ae806ac
app/vmselect: implement /tags/tagSeries
and /tags/tagMultiSeries` in order to be consistent with single-node VictoriaMetrics
2020-11-23 14:57:08 +02:00
Aliaksandr Valialkin
7987129baa
app/vmselect/netstorage: move common code for requests execution on all the storage nodes to startStorageNodesRequest func
2020-11-23 10:51:48 +02:00
Aliaksandr Valialkin
25a57ced6c
app/vmselect/netstorage: prevent from data races in ProcessSearchQuery and in Export funcs when -replicationFactor > 1
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711
2020-11-23 10:25:51 +02:00
Aliaksandr Valialkin
f4fd917e4f
lib/fs: replace fs.OpenReaderAt with fs.MustOpenReaderAt
...
All the callers for fs.OpenReaderAt expect that the file will be opened.
So it is better to log fatal error inside fs.MustOpenReaderAt instead of leaving this to the caller.
2020-11-23 09:57:30 +02:00
Aliaksandr Valialkin
1dcb438c3b
app/vmselect/netstorage: typo fix after 990eb29a9b
2020-11-23 01:09:43 +02:00
Aliaksandr Valialkin
85eecf5801
app/vmselect/netstorage: add -replicationFactor
command-line flag for reducing query duration when a part of vmstorage nodes are temporarily slow and/or temporarily unavailable
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711
2020-11-23 00:39:53 +02:00
Aliaksandr Valialkin
990eb29a9b
app/vmselect/netstorage: move common code for collecting query results from vmstorage nodes to collectResults function
2020-11-23 00:16:02 +02:00
Nikolay
bb2bcb9725
Adds eureka service discovery ( #913 )
...
* Adds eureka service discovery
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/851
Netflix service discovery for AWS
* Apply suggestions from code review
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2020-11-20 14:02:13 +02:00
Aliaksandr Valialkin
e72ccc9239
app/vmselect: add remoteAddr to slow query log in order to improve debuggability
...
This will simplify identifying the client that sends slow queries to VictoriaMetrics.
2020-11-18 20:40:02 +02:00
Aliaksandr Valialkin
ea4afb201b
app/vmselect/netstorage: typo fix in a comment inside SearchMetricNames func
2020-11-18 01:35:37 +02:00
Aliaksandr Valialkin
c6adcafedb
app/vminsert: export vm_rpc_vmstorage_is_reachable
metric, which can be used for monitoring reachability of vmstorage nodes from vminsert nodes
2020-11-17 22:13:26 +02:00
Aliaksandr Valialkin
7d76fdedcc
app/vmselect: use storage.NewSearchQuery() instead of constructing storage.SearchQuery in-place
...
This should prevent from bugs when AccountID and ProjectID aren't set in storage.SearchQuery.
2020-11-16 18:04:33 +02:00
Aliaksandr Valialkin
911c6d3bcd
app/vmselect: add missing graphite prefix to /tags/autoComplete/{tags,values}
2020-11-16 18:04:24 +02:00
Aliaksandr Valialkin
f7f866d83b
app/vmselect/netstorage: typo fix
2020-11-16 15:54:45 +02:00
Aliaksandr Valialkin
59fb75717e
app/vmselect/netstorage: apply Graphite filter after substituting __name__
with name
2020-11-16 15:50:53 +02:00
Aliaksandr Valialkin
eb763bcb9d
app/vmselect/graphite: add /tags/autoComplete/values
handler from Graphite Tags API
2020-11-16 15:29:29 +02:00
Aliaksandr Valialkin
f2f16d8e79
app/vmselect/graphite: add /tags/autoComplete/tags
handler from Graphite Tags API
...
See https://graphite.readthedocs.io/en/stable/tags.html#auto-complete-support
2020-11-16 14:58:10 +02:00
Aliaksandr Valialkin
2f4421b86c
app/vmselect/prometheus: return __name__
label if match[]
query to /api/v1/labels
matches at least a single time series
2020-11-16 13:54:50 +02:00
Aliaksandr Valialkin
852aed62f7
app/vmselect/prometheus: improve performance for /api/v1/labels
and /api/v1/label/<labelName>/values
on time ranges exceeding one day when match[]
query arg is set
2020-11-16 13:46:51 +02:00
Aliaksandr Valialkin
e969346e3e
app/vmselect/prometheus: fix deadlock in /api/v1/series
on a time range exceeding one day
2020-11-16 13:30:57 +02:00
Aliaksandr Valialkin
eea1be0d5c
app/vmselect/graphite: add /tags/findSeries handler from Graphite Tags API
...
See https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags
2020-11-16 12:52:23 +02:00
Aliaksandr Valialkin
97100b1d42
app/vmselect/graphite: apply filter then limit
2020-11-16 12:52:18 +02:00
Aliaksandr Valialkin
5889273920
app/vmselect/graphite: add /tags/<tag_name>
handler for Graphite Tags API
2020-11-16 03:41:41 +02:00
Aliaksandr Valialkin
99cb1a70cf
app/vmselect/graphite: add /tags
handler from Graphite Tags API
...
See https://graphite.readthedocs.io/en/stable/tags.html#exploring-tags
2020-11-16 02:57:20 +02:00
Aliaksandr Valialkin
2ac5f00d98
app/vmselect: propagate errors from vmstorage
to response to the client if -search.denyPartialResponse
command-line flag is set
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/891
This commit also adds `"isPartial":{true|false}` field to `/api/v1/*` responses. `"isPartial":true` is set when the response
is based on a partial data because some of vmstorage nodes weren't available during query processing.
2020-11-14 13:20:10 +02:00
Aliaksandr Valialkin
882e2e2099
app/vminsert/netstorage: return 503 status code to client when all the vmstorage nodes are unavailable
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/896
2020-11-14 00:44:41 +02:00
Aliaksandr Valialkin
8f42e59e05
app/vmselect/promql: remove spikes from increase()
and delta()
results on time series with spare irregular data points
...
Do not take into account spare data point value if the next point will is located too far from the current point.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/894
2020-11-13 15:23:37 +02:00
Aliaksandr Valialkin
da6d82a8dd
app/vmselect/promql: assume that time series value doesnt change during gaps when calculating increase() and delta()
...
This should remove unexpected spikes at the end of gaps.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/894
2020-11-13 14:59:32 +02:00
Aliaksandr Valialkin
7ceaf4ba8f
all: consistently return text-based HTTP responses with charset=utf-8
...
This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/897
2020-11-13 10:30:21 +02:00
faceair
64e99744f1
add charset on targets response ( #897 )
2020-11-13 10:18:13 +02:00
Aliaksandr Valialkin
f7a6ae3d11
docs/vmagent.md: added a link to https://valyala.medium.com/how-to-use-relabeling-in-prometheus-and-victoriametrics-8b90fc22c4b2 into Relabeling
section
2020-11-12 12:27:13 +02:00
Aliaksandr Valialkin
069979c367
docs/vmagent.md: typo fix
2020-11-11 16:05:04 +02:00
Aliaksandr Valialkin
7a0094adae
docs/vmagent.md: add Configuration update
section
2020-11-11 16:01:21 +02:00
immerrr again
1ec1a9f27f
app/vmstorage: add "/internal/force_flush" endpoint ( #893 )
2020-11-11 14:46:37 +02:00
Aliaksandr Valialkin
4f2c5877db
app/vmselect: add -search.treatDotsAsIsInRegexps
command-line flag for automatic escaping of dots in regexp label filters
2020-11-11 12:40:28 +02:00
Aliaksandr Valialkin
a78bf34ff3
app/vmselect: do not return isPartialResponse=true
when all the storageNodes return errors
2020-11-10 18:48:57 +02:00
Aliaksandr Valialkin
8f3339fa81
app/vmselect/promql: do not return data points in the end of the selected time range for time series ending in the middle of the selected time range
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/887
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845
2020-11-10 14:51:55 +02:00
Aliaksandr Valialkin
6385432611
app/vmselect: typo fix in a description for -search.minStalenessInterval
: mimimum->minimum
2020-11-10 01:18:59 +02:00
Roman Khavronenko
4fd2b6cd16
vmalert: explicitly set extra labels to alert entities ( #886 )
...
The previous implementation treated extra labels (global and rule labels) as
separate label set to returned time series labels. Hence, time series always contained
only original labels and alert ID was generated from sorted labels key-values.
Extra labels didn't affect the generated ID and were applied on the following actions:
- templating for Summary and Annotations;
- persisting state via remote write;
- restoring state via remote read.
Such behaviour caused difficulties on restore procedure because extra labels had to be dropped
before checking the alert ID, but that not always worked. Consider the case when expression
returns the following time series `up{job="foo"}` and rule has extra label `job=bar`.
This would mean that restored alert ID will be always different to the real time series because
of collision.
To solve the situation extra labels are now always applied beforehand and `vmalert` doesn't
store original labels anymore. However, this could result into a new error situation.
Consider the case when expression returns two time series `up{job="foo"}` and `up{job="baz"}`,
while rule has extra label `job=bar`. In such case, applying extra labels will result into
two identical time series and `vmalert` will return error:
`result contains metrics with the same labelset after applying rule labels`
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/870
2020-11-10 00:27:56 +02:00
Aliaksandr Valialkin
a8562d643b
lib/promscrape: add -promscrape.dropOriginalLabels
command-line flag for reducing memory usage when discovering big number of scrape targets
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825
2020-11-10 00:20:49 +02:00
Aliaksandr Valialkin
b8083b7659
lib/promscrape: clean references to label name and label value strings after applying per-target relabeling
...
This should reduce memory usage when per-target relabeling creates big number of temporary labels
with long names and/or values.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825
2020-11-07 16:19:52 +02:00
Aliaksandr Valialkin
efebc3b6fb
app/vmselect/promql: code cleanup after 43823addea
2020-11-06 01:31:33 +02:00
n4mine
3127aa92b5
app/vmselect/promql: fix when the parameter of maxValue()
, minValue()
leading by NaN
. it will cause {top,bottom}k_{max,min}
return inappropriate result ( #883 )
2020-11-06 01:31:31 +02:00
Aliaksandr Valialkin
767231f41f
app/vmstorage/transport: properly handle request to labelValuesOnTimeRange
2020-11-05 02:08:04 +02:00
Aliaksandr Valialkin
72011bcc45
app/vmselect: properly handle errors in GetLabelsOnTimeRange and GetLabelValuesOnTimeRange
2020-11-05 01:36:34 +02:00
Aliaksandr Valialkin
c5e6c5f5a6
app/vmselect: optimize querying for /api/v1/labels
and /api/v1/label/<name>/values
when start
and end
args are set
2020-11-05 01:19:29 +02:00
Aliaksandr Valialkin
1336e47c86
docs/vmagent.md: update after 4c808d58bf
2020-11-04 20:33:49 +02:00
Nikolay
5b235b902b
Adds ready probe ( #874 )
...
* adds leading forward slash check for scrapeURL path
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/835
* adds ready probe for scrape config initialization,
it should prevent metrics loss during vmagent rolling update,
/ready api will return 425 http code, if some scrape config still waits for initialization.
* updates docs
* Update app/vmagent/README.md
* renames var
* Update app/vmagent/README.md
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2020-11-04 20:33:48 +02:00
Nikolay
d0a9b24c5a
reduces memory usage for vmagent, ( #880 )
...
* reduces memory usage for vmagent,
limits count of droppedTarget, that can be stored for /api/v1/targets page up to 999 items,
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878
* Update app/vmagent/README.md
* Update app/vmagent/README.md
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2020-11-04 17:13:33 +02:00
Aliaksandr Valialkin
407a46c11e
Revert "docs/vmagent.md: mention about -promscrape.dropOriginalLabels
"
...
This reverts commit 1a80acc712
.
2020-11-04 11:45:35 +02:00
Aliaksandr Valialkin
1a80acc712
docs/vmagent.md: mention about -promscrape.dropOriginalLabels
2020-11-04 11:16:16 +02:00
Aliaksandr Valialkin
887a3c317f
app/vmagent/remotewrite: drop packets only on 409 status code, since there are other valid 4xx status codes, which shouldnt result in packet drop
2020-11-03 14:24:57 +02:00
Aliaksandr Valialkin
66de02fbb4
app/vmselect/promql: allow dropping trailing sample only for default_rollup
function
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/850
2020-11-02 02:11:06 +02:00
Aliaksandr Valialkin
ca2e0f1e04
app/vmagent/remotewrite: drop packets if remote storage returns 4xx status code
...
This makes consistent the behaviour with Prometheus.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/873
2020-11-02 00:45:01 +02:00
Aliaksandr Valialkin
6b623eba02
app/vmselect/promql: go fmt
2020-11-02 00:18:24 +02:00
Aliaksandr Valialkin
7c0b658865
app/vmselect/promql: do not drop trailing datapoints for instant queries
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
2020-11-02 00:12:53 +02:00
Roman Khavronenko
333675875f
vmalert: skip automatically added labels on alerts restore ( #871 )
...
Label `alertgroup` was introduced in #611 and automatically added to generated
time series. By mistake, this new label wasn't correctly purged on restore event
and affected alert's ID uniqueness. This commit removes `alertgroup` label
in restore function.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/870
2020-11-01 23:26:00 +02:00
kreedom
40172c0721
vmbackup fix panic when no origin fs given ( #859 )
...
* use fsnil when no origin fs
2020-11-01 23:17:01 +02:00
Aliaksandr Valialkin
ed724d25ba
lib/promscrape: add stream parse
mode for efficient scraping of targets that expose millions of metrics
2020-11-01 23:12:26 +02:00
Aliaksandr Valialkin
abdf22e0bb
app/vmagent: expose /api/v1/targets
page according to https://prometheus.io/docs/prometheus/latest/querying/api/#targets
...
This page is exposed by vmagent and by a single-node VictoriaMetrics
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/643
2020-10-20 21:55:14 +03:00
Aliaksandr Valialkin
c4464594b7
app/vmselect/promql: allow passing optional third argument to topk_*
and bottomk_*
functions in order to obtain sum of time series outside top/bottom K
2020-10-20 20:09:55 +03:00
Aliaksandr Valialkin
9c5cd5a6c5
lib/storage: code cleanup after 5bfd4e6218
2020-10-20 16:10:53 +03:00
Aliaksandr Valialkin
0db7c2b500
app/vmstorage: support for -retentionPeriod
smaller than one month
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/173
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/17
2020-10-20 14:42:46 +03:00
kreedom
4526cf92d3
vmalert - add dryRun ( #842 )
...
vmalert: add `dryRun` flag for rules validation without running the service
2020-10-20 10:49:22 +03:00
Seva Poliakov
e6bf9eaac7
Fix typo in vnrestore readme
2020-10-20 10:49:22 +03:00
Aliaksandr Valialkin
ee2902ddaf
app/vmselect/promql: an attempt to improve heuristics for dropping trailing data points in time series
...
Now trailing data points are additionally dropped for time series with a single raw sample
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
2020-10-17 10:44:26 +03:00
Roman Khavronenko
d6155a3f33
vmalert: update docs to highlight the state restore requirements; ( #833 )
...
Address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/830
2020-10-13 18:34:00 +03:00
Aliaksandr Valialkin
b9a4601c97
app/vmselect/promql: return a single time series at max from absent()
function like Prometheus does
2020-10-13 15:56:10 +03:00
Aliaksandr Valialkin
217c192c88
app/vmselect/promql: improve time series staleness detection
...
This should prevent from double counting for time series at the time when it changes label.
The most common case is in K8S, which changes pod uid label with each new deployment.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
2020-10-13 12:20:08 +03:00
Aliaksandr Valialkin
f877e703c8
app/vmselect/promql: fix mode_over_time
calculations
...
Previously `mode_over_time` could return garbage due to improper shuffling of input data points.
2020-10-13 11:58:30 +03:00
Aliaksandr Valialkin
d884ab13dc
app/vmselect/prometheus: fix golangci-lint warning
2020-10-13 09:36:18 +03:00
Aliaksandr Valialkin
0867dea5fc
app/vmselect: add ability to export data in CSV format via /api/v1/export/csv
2020-10-12 20:08:08 +03:00
Aliaksandr Valialkin
938b3b7ed1
lib/promscrape: code prettifying after 9bd9f67718
2020-10-12 16:13:59 +03:00
Nikolay Khramchikhin
7f96712b38
Adds dockerswarm sd ( #818 )
...
* adds dockerswarm service discovery
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/656
Following roles supported: services, tasks and nodes.
Basic, token and tls auth supported.
Added tests for labels generation.
* added unix socket support to discovery utils
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2020-10-12 16:13:58 +03:00
Aliaksandr Valialkin
2d03d0e2dd
app/vmselect/promql: keep metric name after applying more functions, which dont change time series meaning
...
Functions are:
* keep_last_value
* keep_next_value
* interpolate
* running_min
* running_max
* running_avg
* range_min
* range_max
* range_avg
* range_first
* range_last
* range_quantile
* smooth_exponential
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674
2020-10-12 11:48:38 +03:00
Aliaksandr Valialkin
3881c84afe
Revert "app/vmselect/promql: remove metric name after applying ceil
, floor
and round
functions in order to be more consistent with Prometheus"
...
This reverts commit ac45082216
.
Reason for revert: the previous behavior for VictoriaMetrics is easier to understand and use by users -
functions, which don't change the meaning of the time series shouldn't drop metric name.
Now the following functions do not drop metric names:
* ceil
* floor
* round
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674
2020-10-12 11:48:38 +03:00
Aliaksandr Valialkin
79d70480b7
Revert "app/vmselect/promql: remove metric name after applying clamp_min
and clamp_max
functions in order to be consistent with Prometheus"
...
This reverts commit bb61a4769b
.
Reason for revert: the previous behavior for VictoriaMetrics is easier to understand and use by users -
functions, which don't change the meaning of the time series shouldn't drop metric name.
Now the following functions do not drop metric name:
* clamp_min
* clamp_max
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674
2020-10-12 11:48:38 +03:00
Aliaksandr Valialkin
8c37b63ea9
Revert "app/vmselect/promql: remove metric name from results of certain rollup functions in order to be consistent with Prometheus"
...
This reverts commit e5202a4eae
.
Reason for revert: the previous behavior for VictoriaMetrics is easier to understand and use by users -
functions, which don't change the meaning of the time series shouldn't drop metric name.
Now the following functions do not drop metric name:
* max_over_time
* min_over_time
* avg_over_time
* quantile_over_time
* geomean_over_time
* mode_over_time
* holt_winters
* predict_linear
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/674
2020-10-12 11:48:38 +03:00
Aliaksandr Valialkin
de1c07b937
lib/backup: add MustStop() method for all remote filesystems
2020-10-09 15:32:13 +03:00
Aliaksandr Valialkin
bf6d523bef
lib/backup/fslocal: add FS.MustStop() method for stopping bandwidth limiter
2020-10-09 15:11:55 +03:00
Aliaksandr Valialkin
9b7ce5d004
app/{vminsert,vmagent}: take into account all the inserted rows before relabeling in vm_rows_inserted_total
and vmagent_rows_inserted_total
metrics
2020-10-09 13:38:49 +03:00
Aliaksandr Valialkin
d2e917d1cb
app/vmstorage: add vm_rows_added_to_storage_total
metric, which shows the total number of rows added to storage since app start
2020-10-09 13:36:17 +03:00
Aliaksandr Valialkin
4b1c401790
app/vmalert: accept days, weeks and years in for:
part of config like Prometheus does
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/817
2020-10-08 20:13:20 +03:00
Aliaksandr Valialkin
35b8ffaa17
docs/vmagent.md: clarify -promscrape.suppressDuplicateScrapeTargetErrors
command-line flag usage
2020-10-08 19:24:05 +03:00
Aliaksandr Valialkin
0d44e371f3
lib/promscrape: add -promscrape.suppressDuplicateScrapeTargetErrors
command-line flag in order to suppress duplicate scrape target
errors
...
Show also original labels for duplicate targets in error message in order to simplify debugging the issue.
Now `/targets` endpoint accepts optional `show_original_labels=1` query arg, which shows original labels for each target.
This may simplify debugging for target relabeling.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/651
2020-10-08 18:59:25 +03:00
Aliaksandr Valialkin
f9f8e4a39c
app/vmalert: do not pring description for all the flags on config errors
...
The description is too big to consume by human and it just distracts humans.
2020-10-08 13:35:46 +03:00
Aliaksandr Valialkin
f6ee6efc34
app/vmselect/promql: add missing label filters to binary operands before query execution
...
This implements the optimization described at https://utcc.utoronto.ca/~cks/space/blog/sysadmin/PrometheusLabelNonOptimization
See also https://github.com/cortexproject/cortex/issues/3253
2020-10-07 21:17:11 +03:00
Dmitry Shihovtsev
aec863e70b
Fix typos in the vmalert datasource ( #814 )
...
* Fix typos in the vmalert datasource
* Fix typo in the vmalert datasource test
2020-10-07 18:00:29 +03:00
Aliaksandr Valialkin
b51fa16177
app/vmstorage: add -finalMergeDelay
command-line flag for configuring the delay before final merge for per-month partitions after no new data is ingested to it
2020-10-07 17:42:31 +03:00
Aliaksandr Valialkin
c4d309aa41
app/vmagent: add a link to https://victoriametrics.github.io/vmagent.html from main page
2020-10-06 15:30:07 +03:00
Aliaksandr Valialkin
abfd3a8fab
app/{vminsert,vmselect,vmstorage}: add a link to https://victoriametrics.github.io/Cluster-VictoriaMetrics.html from main page of every cluster component
2020-10-06 15:30:07 +03:00
Nikolay Khramchikhin
b4c77fc6d2
Adds openstack sd ( #811 )
...
* adds openstack service discovery
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/728
implemented hypervisors and instance discovery with openstack v3 api.
Added tests for labeling and data parsing.
Added token refresh.
* Apply suggestions from code review
* Apply suggestions from code review
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2020-10-05 16:48:54 +03:00
Aliaksandr Valialkin
3e0c473cc9
app/vmselect/promql: fill gaps on graphs for range_*
and running_*
functions
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/806
2020-10-02 14:02:47 +03:00
Aliaksandr Valialkin
fd7dd5064a
lib/storage: code cleanup after 10f2eedee0
...
Remove the code that uses metricIDs caches for the current and the previous hour during metricIDs search,
since this code became unused after implementing per-day inverted index almost a year ago.
While at it, fix a bug, which could prevent from finding time series with names containing dots (aka Graphite-like names
such as `foo.bar.baz`).
2020-10-01 19:12:04 +03:00
Roman Khavronenko
368b890e11
vmalert: make maxIdleConnections configurable for datasource HTTP client ( #797 )
...
Address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/795
2020-09-30 09:51:14 +03:00
Aliaksandr Valialkin
536aa8779a
app/vmstorage: rename vm_{big|small}_merge_need_free_disk_space
to vm_merge_need_free_disk_space
...
This simplifies alerting.
2020-09-29 22:53:33 +03:00
Aliaksandr Valialkin
097a4c10dd
app/vmstorage: add metrics for determining whether background merges need additional disk space to complete
...
These metrics are:
* vm_small_merge_need_free_disk_space
* vm_big_merge_need_free_disk_space
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686
2020-09-29 21:47:47 +03:00
Aliaksandr Valialkin
5dca7bbe85
app/vmagent/remotewrite: do not show -remoteWrite.url
in logs if -remoteWrite.showURL
isn't set
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/773
2020-09-29 19:49:19 +03:00
Aliaksandr Valialkin
61f4801b93
app/vmselect/graphite: do not substitute path
and path.
with path..
in /metrics/find/?format=completer
output
2020-09-29 18:03:45 +03:00
Aliaksandr Valialkin
f4d74ccfec
docs/{vmbackup,vmrestore}: formatting fixes
2020-09-29 13:19:13 +03:00
Aliaksandr Valialkin
5179df7082
docs/vmbackup.md: make docs about minio config more prominent
2020-09-29 13:16:34 +03:00
Aliaksandr Valialkin
79b00f7b6b
docs: improve readability a bit
2020-09-29 13:05:05 +03:00
Nikolay Khramchikhin
d77f56fd97
update vmbackup/vmrestore README usage ( #794 )
...
* update vmbackup/vmrestore README usage
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/381
with minio and configuration file examples.
* Apply suggestions from code review
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
* added backup/restore docs changes
* added example for relabelConfig flag
* Apply suggestions from code review
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
2020-09-29 13:05:04 +03:00
Aliaksandr Valialkin
0d082cdf53
app/vmselect/netstorage: remove unused code
2020-09-29 11:55:08 +03:00
Aliaksandr Valialkin
c47972d843
app/vmselect/prometheus: check for errors returned from bufferedwriter.Write
...
This makes `make errcheck` happy
2020-09-29 11:36:54 +03:00
Aliaksandr Valialkin
e66f7edfc9
app/vmselect/graphite: properly handle case when /metrics/find
finds both leaf and node for the given query=prefix.*
...
In this case only node must be returned with stripped dot in the end of id as carbonapi does
2020-09-29 11:29:38 +03:00
Aliaksandr Valialkin
ffa6581c46
app/vminsert: refresh the list of healthy storage nodes only if the the row cannot be sent to destination storage node
...
Previously the list had been generated for each rerouted row. This could consume additional CPU time during rerouting,
which could lead to rerouting slowdown.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/791
2020-09-29 01:29:24 +03:00
Aliaksandr Valialkin
bc37f1cbec
app/vminsert: do not pollute logs with repated cannot dial storageNode
errors
...
Log only the first error per -storageNode
2020-09-29 00:20:32 +03:00
Aliaksandr Valialkin
2ee0dc27a6
app/vmstorage: parallelize data processing obtained from a single connection from vminsert
...
Previously vmstorage could use only a single CPU core for data processing from a single connection from vminsert.
Now all the CPU cores can be used for data processing from a single connection from vminsert.
This should improve the maximum data ingestion performance for a single vminsert->vmstorage connection.
2020-09-28 21:41:16 +03:00
Aliaksandr Valialkin
9d123eb22a
app/vminsert: remove useless delays when sending data to vmstorage
...
This improves the maximum data ingestion performance for cluster VictoriaMetrics
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/791
2020-09-28 21:41:15 +03:00
Aliaksandr Valialkin
8df33bd5c1
app/{vminsert,vmagent}: improve data ingestion speed over a single connection
...
Process data obtianed from a single connection on all the available CPU cores.
2020-09-28 04:14:51 +03:00
Aliaksandr Valialkin
6d8c23fdbd
app/{vminsert,vmselect}: skip accountID and projectID when marshaling/unmarshaling MetricName in /api/v1/export/native and /api/v1/import/native
...
This is needed in order to be able to migrate native data from/to single-node VictoriaMetrics
2020-09-28 00:58:58 +03:00
Aliaksandr Valialkin
db14f22fc0
app/vmselect: stop /api/v1/export/*
execution if client disconnects
2020-09-27 23:52:42 +03:00
Aliaksandr Valialkin
aadbd014ff
all: add native format for data export and import
...
The data can be exported via [/api/v1/export/native](https://victoriametrics.github.io/#how-to-export-data-in-native-format ) handler
and imported via [/api/v1/import/native](https://victoriametrics.github.io/#how-to-import-data-in-native-format ) handler.
2020-09-27 17:36:38 +03:00
Aliaksandr Valialkin
e83947a882
app/vminsert: code prettifying
2020-09-26 04:16:14 +03:00
Aliaksandr Valialkin
8fc9b77496
app/vmagent: reduce memory usage when importing data via /api/v1/import
...
Previously vmagent could use big amounts of RAM when each ingested JSON line
contained many samples.
2020-09-26 04:10:13 +03:00
Aliaksandr Valialkin
973df09686
app/vmselect/netstorage: do not spend CPU time on unpacking empty blocks during /api/v1/series
calls
2020-09-24 20:44:15 +03:00
Aliaksandr Valialkin
b8bce348c5
app/vmselect/promql: properly limit implicitly set rollup window to -search.maxStalenessInterval
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/784
2020-09-23 23:24:09 +03:00
Aliaksandr Valialkin
543f3aea97
all: consistently use "%w" formatting in fmt.Errorf for wrapped errors
2020-09-23 22:48:21 +03:00
Aliaksandr Valialkin
8627365b48
app/vmselect/prometheus: code cleanup after 3ba507000c
2020-09-23 13:04:51 +03:00
Aliaksandr Valialkin
1fce79518a
app/vmselect/prometheus: return timestamps from /api/v1/query
, which match the time
query arg
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/720
2020-09-23 12:59:23 +03:00
Aliaksandr Valialkin
0468cdf33e
app/vmselect/netstorage: properly pre-allocate space for sbs
2020-09-22 23:51:01 +03:00
Aliaksandr Valialkin
69eb9783e6
app/vmselect: make sure the request doesnt wait in pending queue more than the configured timeout
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711
2020-09-22 01:21:42 +03:00
Aliaksandr Valialkin
ed473c94ff
docs/vmagent.md: typo fix
2020-09-21 21:49:08 +03:00
Aliaksandr Valialkin
e564725641
app/vmselect/searchutils: fixed tests after 2eb72e09ab
2020-09-21 21:31:28 +03:00
Aliaksandr Valialkin
07c6226334
app/vmselect: use time
value rounded to seconds if it isnt passed to /api/v1/query
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/720
2020-09-21 21:24:46 +03:00
Nikolay Khramchikhin
0069353d5e
Add improvements to ec2_sd_discovery ( #775 )
...
* Add improvements to ec2 discovery
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/771
role_arn support with aws sts
instance iam_role support
refreshing temporary tokens
* Apply suggestions from code review
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
* changed implementation, removed tests, clean up code
* moved endpoint builder into getEC2APIResponse
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
2020-09-21 16:05:01 +03:00
Aliaksandr Valialkin
8cd89cb847
app/vmalert: remove unneeded UTC() call
...
UTC() doesn't change the underlying timestamp, so the call isn't needed here
2020-09-21 15:56:48 +03:00
Roman Khavronenko
d111969d39
vmalert: add support for datasource.lookback
flag ( #779 )
...
New datasource flag `datasource.lookback` defines how far to look into
past when evaluating queries.
Address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/668
2020-09-21 15:56:47 +03:00
Roman Khavronenko
0042b0f307
vmalert: fix the typo in error message ( #782 )
...
The error will be always nil so no sense in printing it.
2020-09-21 11:36:09 +03:00
Aliaksandr Valialkin
1c07d7bee7
app/vmagent: increase default value for -remoteWrite.queues
from 1 to 4, since it has been appeared that many users hit this limit
2020-09-18 14:22:02 +03:00
Aliaksandr Valialkin
a1bebb660c
app/vmselect/graphite: return proper results /metrics/find?query=foo.*.bar
according to Graphite Metrics API
2020-09-18 11:53:28 +03:00
Aliaksandr Valialkin
9b15b11f74
app/vmstorage: added -forceMergeAuthKey
command-line flag for protecting /internal/force_merge
endpoint
2020-09-17 14:24:20 +03:00
Aliaksandr Valialkin
d96858b921
lib/storage: add /internal/force_merge
handler for running forced compactions on historical per-month partitions
...
This may be useful for freeing up storage space after time series deletion.
See https://victoriametrics.github.io/#force-merge for more details.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686
2020-09-17 12:20:56 +03:00
Aliaksandr Valialkin
406f4fe445
app/vmagent: substitute -remoteWrite.url
with secret-url
value in logs, since it may contain sensitive info such as passwords or auth tokens
...
Pass `-remoteWrite.showURL` command-line flag in order to see real `-remoteWrite.url` values in logs and at `/metrics` page.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/773
2020-09-16 22:36:18 +03:00
Aliaksandr Valialkin
9793008734
app/vmselect: add -search.storageTimeout command-line flag for limiting the maximum duration of query execution per each -storageNode
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711
2020-09-16 21:33:47 +03:00
Aliaksandr Valialkin
a9205fe308
app/vmselect: prevent from closing connection to vmstorage on query timeout by setting +2 secs deadline on connection comparing to query deadline
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711
2020-09-16 21:14:00 +03:00
Aliaksandr Valialkin
1587f83fa0
app/vmselect/netstorage: typo fix after 03dfccfbed
2020-09-16 00:10:33 +03:00
Aliaksandr Valialkin
03dfccfbed
app/vmselect/netstorage: reduce memory usage when the time range from query touches big number of samples per each time series
2020-09-15 21:08:09 +03:00
Aliaksandr Valialkin
27cd5555e6
app/vmselect/netstorage: mention that RunParallel or Cancel must be called on the returned results from ProcessSearchQuery
2020-09-15 20:39:43 +03:00
Aliaksandr Valialkin
1f0c0b0f6b
app/vmselect: typo fix in -search.maxStalenessInterval description
2020-09-15 14:25:34 +03:00
Roman Khavronenko
e2b31590e6
vmalert: add Group name as label to generated alerts and timeseries ( #761 )
...
Solves #611
2020-09-11 23:41:12 +03:00
Roman Khavronenko
16e0bb496e
vmalert: update groups on config reload only if changes detected ( #759 )
...
On config reload event `vmalert` reloads configuration for every group. While
it works for simple configurations, the more complex and heavy installations may
suffer from frequent config reloads.
The change introduces the `checksum` field for every group and is set to md5 hash
of yaml configuration. The checksum will change if on any change to group
definition like rules order or annotation change. Comparing the `checksum` field
on config reload event helps to detect if group should be updated.
The groups update is now done concurrently, so reload duration will be limited by
the slowest group now.
Partially solves #691 by improving config reload speed.
2020-09-11 23:41:12 +03:00
Aliaksandr Valialkin
b776a93608
app/vmselect/promql: support composite durations like Prometheus 2.21 does
...
The following durations are supported now: `1h5m35s` or `1s543ms`
See https://github.com/prometheus/prometheus/releases/tag/v2.21.0
and https://github.com/prometheus/prometheus/pull/7713
2020-09-11 22:17:24 +03:00
Aliaksandr Valialkin
6382e8081a
app/vmagent: allow setting multiple identical -remoteWrite.url
values
...
This may be useful when each url is authenticated via different `-remoteWrite.basicAuth.username`.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/755
2020-09-11 15:17:29 +03:00
Aliaksandr Valialkin
d3ad0d365e
app/vmselect: move Deadline from netstorage to searchutils
...
This removes dependency on netstorage from searchutils.
2020-09-11 13:39:13 +03:00
Aliaksandr Valialkin
58d3b82ae5
app/{vminsert,vmagent}: allow passing timestamp via timestamp
query arg when ingesting data to /api/v1/import/prometheus
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/750
2020-09-11 13:28:31 +03:00
Aliaksandr Valialkin
579c20756a
app/vmselect: substitute inf values at smooth_exponential with the previous values
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/757
2020-09-11 12:23:56 +03:00
Aliaksandr Valialkin
d67e6d3d2e
app/vmselect: skip infinite values when calculating smooth_exponential
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/757
2020-09-11 11:57:53 +03:00
Aliaksandr Valialkin
06427a184f
app/vmselect/graphite: typo fix in label name for vm_request_duration_seconds metric
2020-09-11 01:59:52 +03:00
Aliaksandr Valialkin
f307e6f432
app/vmselect: initial implementation of Graphite Metrics API
...
See https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api
2020-09-11 00:30:20 +03:00
Aliaksandr Valialkin
f5cb213ef9
lib/storage: reuse timestamp blocks for adjancent metric blocks with identical timestamps
...
This should reduce disk space usage when scraping targets containing metrics with identical names
such as `node_cpu_seconds_total`, histograms, quantiles, etc.
Expose `vm_timestamps_blocks_merged_total` and `vm_timestamps_bytes_saved_total` metrics for monitoring
the effectiveness of timestamp blocks merging.
2020-09-09 23:59:21 +03:00
Aliaksandr Valialkin
475698d2ad
docs: sync docs for vmalert, vmauth, vmbackup and vmrestore
2020-09-09 21:10:48 +03:00
Aliaksandr Valialkin
5ab57f916b
docs/vmagent.md: clarified the case when -remoteWrite.queues
must be tuned
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/745
2020-09-08 20:15:49 +03:00
Aliaksandr Valialkin
f3a79abfb4
app/vmselect/promql: go fmt
2020-09-08 15:18:57 +03:00
Aliaksandr Valialkin
4f06eed1c1
app/vmselect/promql: adjust integrate()
calculations to be more similar to calculations from InfluxDB: attempt #2
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/701
2020-09-08 14:36:23 +03:00
Aliaksandr Valialkin
0d0b606455
app/vmselect/promql: adjust integrate()
calculations to be more similar to calculations from InfluxDB
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/701
2020-09-08 14:24:02 +03:00
Aliaksandr Valialkin
db91045348
app/vmselect/promql: increase floating point calculations accuracy by dividing by 1e3
instead of multiplying by 1e-3
2020-09-08 14:01:02 +03:00
Aliaksandr Valialkin
804304c365
app/vmselect: add missing deletion for temporary files on partial responses when -search.denyPartialResponse=true
2020-09-04 02:23:12 +03:00
Aliaksandr Valialkin
478d8f8393
app/vmselect/promql: add count_le_over_time(m[d], le)
and count_gt_over_time(m[d], gt)
functions
...
These functions returns the number of raw samples that don't exceed `le` or are bigger than `gt`.
These functions are complement to already existing `share_le_over_time(m[d], le)` and `share_gt_over_time(m[d], gt)`.
2020-09-03 15:28:58 +03:00
Aliaksandr Valialkin
3490160fd0
app/vmselect: unconditionally align time range boundaries to step for subqueries as Prometheus does
2020-09-03 13:22:06 +03:00
Aliaksandr Valialkin
a3cdef6b06
app/vmagent: properly flush big blocks of data
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/741
Thanks to @IceRain00 for the investigation and initial attempt to fix the issue
at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/742
2020-09-03 12:12:12 +03:00
Aliaksandr Valialkin
de216bab41
app/vmagent: fix data race when accessing writeRequest.lastFlushTime
2020-09-03 12:12:09 +03:00
Nikolay Khramchikhin
80a9dc79fe
changed vmalert behaviour ( #738 )
...
* VMAlert start with empty rules dir
There are some applications (operator for instance), that generates alerts configuration at runtime
and vmalert must start correctly without rules to support this behaviour.
Later application will add rules files and send SIGHUP to vmalert,
which will trigger reading rules files and start rules exectuion.
Removing rules files with SIGHUP signal must stop rules execution and
vmalert will wait for new rules.
* imports sorted
* added test cases for empty rules, removed blank line
* fixed imports conflict
* updated tests
2020-09-03 11:07:40 +03:00
Aliaksandr Valialkin
7ac10ee978
app/vmalert: imrovements over 3f932c2db1
2020-09-03 01:14:30 +03:00
DexterZhang
85f49ad439
feat: spread load of rule evaluation by group when starting new groups ( #724 )
...
* feat: spread load of rule evaluation by group when starting new groups
* review: reduce the resulting diff.
* Update app/vmalert/group.go
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
2020-09-03 01:14:26 +03:00
Aliaksandr Valialkin
4fa97430d7
app/{vminsert,vmagent}: allow adding extra labels when importing data via Prometheus, CSV and JSON line formats
...
Extra labels may be added to the imported data by passing `extra_label=name=value` query args.
Multiple query args may be passed in order to add multiple extra labels.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/719
2020-09-02 19:47:02 +03:00
Aliaksandr Valialkin
fe08b1eb26
app/vminsert: improve error message when the data cannot be sent to vmstorage - log reroutedBR buffer size
...
This should improve debuggability for improperly configured cluster
2020-08-31 17:51:44 +03:00
Aliaksandr Valialkin
6f9c1bc078
app/vmagent: log unsuccessful attempt number when sending data to -remoteWrite.url
2020-08-30 21:40:15 +03:00
Aliaksandr Valialkin
3b1ecac04b
app/vmagent: apply sane limits to -remoteWrite.queues
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/707
2020-08-30 21:25:51 +03:00
Roman Khavronenko
08b76cb26f
vmalert: update -rule
flag description to enforce quotes using ( #709 )
...
Description for `-rule` flag uses as example specific chars like asterisks
which could be interpreted wrong by different shells. To avoid this, description
now contains quoted flag values.
See also #708
2020-08-28 09:46:35 +03:00
Roman Khavronenko
4b89da9463
lib/decimal: rename significant decimal digits
to significant figures
( #698 )
...
The previous notion was inconsistent with what `decimal.Round` does.
According to [wiki](https://en.wikipedia.org/wiki/Significant_figures ) rounding
applied to all significant figures, not just decimal ones.
2020-08-16 17:22:40 +03:00
Aliaksandr Valialkin
6aab2f4989
all: allow using KB
, MB
, GB
, KiB
, MiB
and GiB
suffixes in command-line flag values related to byte sizes or byte rates
2020-08-16 17:08:28 +03:00
Aliaksandr Valialkin
285665e93b
app/vmselect/promql: allow passing multiple args to aggregate functions such as avg(q1, q2, q3)
2020-08-15 01:15:16 +03:00
Aliaksandr Valialkin
a2021d0dde
docs/vmagent.md: mention that gaps in remote storage may appear if vmagent cannot keep up with data ingestion
2020-08-14 20:48:17 +03:00
Aliaksandr Valialkin
e7c0b2ca56
docs: update docs
2020-08-14 19:14:46 +03:00
Aliaksandr Valialkin
b996280c65
app/{vminsert,vmagent}: improve documentation for -influxListenAddr
command-line flag
2020-08-14 18:03:08 +03:00
Aliaksandr Valialkin
60c7397be5
all: support %{ENV_VAR}
placeholders in yaml configs in all the vm* components
...
Such placeholders are substituted by the corresponding environment variable values.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/583
2020-08-13 17:17:06 +03:00
Aliaksandr Valialkin
6721e47ae9
app: respect CPU limits set via cgroups
...
Update GOMAXPROCS to limits set via cgroups. This should reduce CPU trashing and reduce memory usage
for cases when VictoriaMetrics components run in containers with CPU limits.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685
2020-08-11 23:01:03 +03:00
Aliaksandr Valialkin
62b6e54622
app/vmselect: reduce memory usage when exporting time series with big number of samples via /api/v1/export
if max_rows_per_line
is set to non-zero value
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685
2020-08-10 20:57:43 +03:00
Aliaksandr Valialkin
c9f5c5623f
app/vmselect/netstorage: vary batch size for data unpacking depending on the available CPU cores
...
This should reduce contention on the channel with unpack work for systems with high number of CPU cores
2020-08-10 15:16:48 +03:00
Aliaksandr Valialkin
b3d4ff7ee2
app/vmstorage: improve error logging when the request times out
2020-08-10 13:17:24 +03:00
Roman Khavronenko
78afc61896
app/vmalert: extend metrics set exported by vmalert
#573 ( #654 )
...
* app/vmalert: extend metrics set exported by `vmalert` #573
New metrics were added to improve observability:
+ vmalert_alerts_pending{alertname, group} - number of pending alerts per group
per alert;
+ vmalert_alerts_acitve{alertname, group} - number of active alerts per group
per alert;
+ vmalert_alerts_error{alertname, group} - is 1 if alertname ended up with error
during prev execution, is 0 if no errors happened;
+ vmalert_recording_rules_error{recording, group} - is 1 if recording rule
ended up with error during prev execution, is 0 if no errors happened;
* vmalert_iteration_total{group, file} - now contains group and file name labels.
This should improve control over specific groups;
* vmalert_iteration_duration_seconds{group, file} - now contains group and file name labels. This should improve control over specific groups;
Some collisions for alerts and recording rules are possible, because neither
group name nor alert/recording rule name are unique for compatibility reasons.
Commit contains list of TODOs for Unregistering metrics since groups and rules
are ephemeral and could be removed without application restart. In order to
unlock Unregistering feature corresponding PR was filed - https://github.com/VictoriaMetrics/metrics/pull/13
* app/vmalert: extend metrics set exported by `vmalert` #573
The changes are following:
* add an ID label to rules metrics, since `name` collisions within one group is
a common case - see the k8s example alerts;
* supports metrics unregistering on rule updates. Consider the case when one rule
was added or removed from the group, or the whole group was added or removed.
The change depends on https://github.com/VictoriaMetrics/metrics/pull/16
where race condition for Unregister method was fixed.
2020-08-09 09:42:05 +03:00
ofen
3fea7c39be
401 Unauthorize HTTP error added ( #681 )
...
401 Unauthorize HTTP error added to trigger browser credentials pop-up promt [RFC 7235 https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication ]
2020-08-09 09:39:37 +03:00
Aliaksandr Valialkin
67cacb22ac
lib/httpserver: add -tls
, -tlsCertFile
and -tlsKeyFile
command-line flags in every vm binary
...
This makes such binaries compatible with binaries from `master` branch (aka single-node version)
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/677
2020-08-07 10:57:32 +03:00
Aliaksandr Valialkin
95a8c492ef
app/vmselect/promql: properly handle -n^m
like Prometheus does
...
`-n^m` must be handled as `-(n^m)` instead of `(-n)^m`.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/675
2020-08-07 07:42:42 +03:00
Aliaksandr Valialkin
7f93d61a56
app/vmselect/promql: remove metric name after applying clamp_min
and clamp_max
functions in order to be consistent with Prometheus
...
This improves VictoriaMetrics score at https://promlabs.com/promql-compliance-test-results-victoriametrics/
2020-08-06 23:42:55 +03:00
Aliaksandr Valialkin
01000505a0
app/vmselect/promql: remove metric name after applying ceil
, floor
and round
functions in order to be more consistent with Prometheus
...
This improves VictoriaMetrics score at https://promlabs.com/promql-compliance-test-results-victoriametrics/
2020-08-06 23:34:03 +03:00
Aliaksandr Valialkin
75bff1a567
app/vmselect/promql: remove metric name from results of certain rollup functions in order to be consistent with Prometheus
...
Rollup functions:
- avg_over_time
- min_over_time
- max_over_time
- quantile_over_time
This improves VictoriaMetrics results at https://promlabs.com/promql-compliance-test-results-victoriametrics/
2020-08-06 23:29:18 +03:00
Aliaksandr Valialkin
8835004a4c
app/vmselect: properly handle PromQL queries like scalar1 < metric < scalar2
like Prometheus does
...
This fixes some cases from https://promlabs.com/promql-compliance-test-results-victoriametrics/
2020-08-06 23:21:14 +03:00
Aliaksandr Valialkin
14ddb8a34e
app/vmselect/netstorage: reduce CPU contention when upacking time series blocks by unpacking batches of such blocks instead of a single block
...
This should improve query performance on systems with big number of CPU cores (16 and more)
2020-08-06 17:50:13 +03:00
Aliaksandr Valialkin
46c98cd97a
app/vmselect/netstorage: reduce contention on unpackworkCh and timeseriesWorkCh for multi-CPU system by providing more capacity for these chans
2020-08-06 17:22:39 +03:00
Aliaksandr Valialkin
a455930ab4
app/vmstorage: rename vm_cache_size_entries{type="storage/prefetchedMetricIDs"}
to vm_cache_entries{type="storage/prefetchedMetricIDs"}
to be consistent with other vm_cache_entries
metrics
2020-08-06 16:34:18 +03:00
Aliaksandr Valialkin
a3e91c593b
lib/storage: limit the number of concurrent calls to storage.searchTSIDs to GOMAXPROCS*2
...
This should limit the maximum memory usage and reduce CPU trashing on vmstorage
when multiple heavy queries are executed.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
2020-08-05 18:27:21 +03:00
Aliaksandr Valialkin
a930460236
app/vmagent: tune http client for sending data to remote storage in order to disable closing keep-alive connections
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/663
2020-08-04 21:01:40 +03:00
Aliaksandr Valialkin
a04f4a3d9a
app/vmselect: use warning level instead of info level for logging slow queries that take longer than -search.logSlowQueryDuration
2020-08-04 20:24:38 +03:00
Aliaksandr Valialkin
bdb881c43b
app/vmselect/promql: add zscore-related functions: zscore_over_time(m[d])
and zscore(q) by (...)
2020-08-03 21:52:15 +03:00
Aliaksandr Valialkin
94471a1273
app: remove duplicate *-pure makefile rules
2020-07-31 20:01:30 +03:00
Aliaksandr Valialkin
a2aa3a60eb
app/vmselect: show X-Forwarded-For
contents on /api/v1/status/active_queries
page
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/659
2020-07-31 20:01:09 +03:00
Aliaksandr Valialkin
106e302d7a
all: add mssing APP_NAME to vm*-GOARCH builds
2020-07-31 13:45:32 +03:00
Aliaksandr Valialkin
945645f38f
docs/{vmagent,vmalert}: add instruction on how to build for ARM
2020-07-31 09:25:41 +03:00
Aliaksandr Valialkin
0c00fe70cf
app/vmselect: do not adjust start
and end
query args passed to /api/v1/query_range
when -search.disableCache
command-line flag is set
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/563
2020-07-30 23:14:56 +03:00
Aliaksandr Valialkin
29bbab0ec9
lib/storage: remove prioritizing of merging small parts over merging big parts, since it doesn't work as expected
...
The prioritizing could lead to big merge starvation, which could end up in too big number of parts that must be merged into big parts.
Multiple big merges may be initiated after the migration from v1.39.0 or v1.39.1. It is OK - these merges should be finished soon,
which should return CPU and disk IO usage to normal levels.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/618
2020-07-30 20:02:22 +03:00
Aliaksandr Valialkin
338ee47d60
app/vmselect/promql: return non-empty value from rate_over_sum(m[d])
even if a single data point is located in the given [d]
window
...
Just divide the data point value by the window duration in this case.
2020-07-29 12:37:34 +03:00
Aliaksandr Valialkin
717c554fb0
app/vmselect/promql: remove rollupFuncArg.realPrevValue handling, since the corner case in increase()
is handled in another way now
...
See e00cfc854d
for the approach used now.
2020-07-29 12:37:34 +03:00
Aliaksandr Valialkin
d9037b3970
app/vmselect/promql: fill gaps with 0 in rate_over_sum
response when the last value before the selected time window isnt empty
2020-07-29 12:37:34 +03:00
Aliaksandr Valialkin
f6d4275087
app/{vmagent,vminsert}: properly preserve db
tag from query string passed to Influx line protocol query
...
Previously `db` tag from the query string wasn't added to metrics after encountering `db` tag in the Influx line
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/653
2020-07-28 21:25:49 +03:00
Aliaksandr Valialkin
baebe86844
app/vmagent/remotewrite: add missing resp.Body.Close()
after pushing data to remote storage
...
Missing body close could disable HTTP keep-alive connections.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/653
2020-07-28 21:00:25 +03:00
Aliaksandr Valialkin
0f6f0d30d3
app/vmselect: show query origin (aka remote_addr or client address) on the /api/v1/status/active_queries
page for every query
2020-07-28 15:14:40 +03:00
Roman Khavronenko
ec6ed467c6
app/vmalert: support external.label
to specify global labelset for all rules #622 ( #652 )
...
`external.label` flag supposed to help to distinguish alert or recording rules
source in situations when more than one `vmalert` runs for the same datasource
or AlertManager.
2020-07-28 14:23:04 +03:00
Aliaksandr Valialkin
9dccedc599
app/vmselect/promql: return empty values from group()
if all the time series have no values at the given timestamp
...
This aligns `group()` behaviour to Prometheus
2020-07-28 13:41:04 +03:00
Aliaksandr Valialkin
d5057f6d04
app/vmagent/remotewrite: create new request on failure to send a block of data to remote storage
...
Previously the request body was already consumed before the retry, so this led to the following error:
http: ContentLength=... with Body length 0
2020-07-27 17:33:05 +03:00
Aliaksandr Valialkin
b191e425b3
app/vmselect/promql: improve further the accuracy of buckets_limit() function
...
The accuracy is increased by mergin the smallest bucket with the smallest adjacent bucket.
2020-07-26 12:10:56 +03:00
Aliaksandr Valialkin
43871e79c6
app/vmselect/promql: avoid dropping inf
bucket in buckets_limit
...
The `le="inf"` bucket must be preserved in order to maintain the maximum level of accuracy.
2020-07-25 17:00:25 +03:00
Aliaksandr Valialkin
978c1e930e
app/vmselect/promql: optimize buckets_limit(k, buckets)
for big number of buckets
2020-07-25 13:24:33 +03:00
Aliaksandr Valialkin
51cbf27077
app/vmselect/promql: improve the accuracy of buckets_limit(k, buckets)
function
...
Now it properly merges the bucket with the previous bucket after deletion.
2020-07-24 17:07:30 +03:00
Aliaksandr Valialkin
cf69b1ea6f
app/vmselect/promql: add buckets_limit(k, buckets)
function, which limits the number of buckets per time series to k
...
This function works with both Prometheus-style and VictoriaMetrics-style buckets.
The function removes buckets with the lowest values in order to reserve the highest precision.
The function is useful for building heatmaps in Grafana from too big number of buckets.
2020-07-24 16:14:12 +03:00
Aliaksandr Valialkin
45334f61de
app/vmselect: fix tests for rate_over_sum
2020-07-24 02:35:09 +03:00
Aliaksandr Valialkin
3526e8768a
app/vmselect/promql: typo fix after 3e557c9861
2020-07-24 02:15:23 +03:00
Aliaksandr Valialkin
8d1721d128
app/vmselect/promql: add rate_over_sum(m[d])
function to MetricsQL, which returns rate over sum of m
values over d
duration
...
Something like `sum_over_time(m[d]) / d`, but more accurate.
2020-07-24 01:17:15 +03:00
Aliaksandr Valialkin
88e8bed0c9
app/vmselect/promql: allow setting [d]
window smaller than the interval between raw points for avg_over_time
...
This makes `avg_over_time` behavior consistent with `sum_over_time` and `count_over_time` behaviors.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/636
2020-07-23 22:25:33 +03:00
Aliaksandr Valialkin
fb3d1380ac
lib/storage: respect -search.maxQueryDuration
when searching for time series in inverted index
...
Previously the time spent on inverted index search could exceed the configured `-search.maxQueryDuration`.
This commit stops searching in inverted index on query timeout.
2020-07-23 21:22:05 +03:00
Aliaksandr Valialkin
dbf3038637
lib/storage: add more fine-grained pace limiting for search
2020-07-23 19:21:49 +03:00
Aliaksandr Valialkin
16a4b1b20c
app/vmselect/netstorage: protect from too smart compiler, which may break memory usage optimization in tmpBlocksFileWrapper.WriteBlocks
2020-07-23 17:57:24 +03:00
Aliaksandr Valialkin
0750d2cec1
app/vminsert: export vm_relabel_metrics_dropped_total
metric that shows the number of metrics dropped due to relabeling
2020-07-23 14:58:02 +03:00
Aliaksandr Valialkin
55ed07add1
app/vmselect: typo fix after 0168e21fe32776e2f7f003f88e0e6e490eb2dcb0g
2020-07-23 14:11:15 +03:00
Aliaksandr Valialkin
7aa5b48508
app/vmselect: reduce memory usage when querying big number of time series with long labels
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646
2020-07-23 13:48:58 +03:00
Aliaksandr Valialkin
49a0011837
app/vminsert: do not call ApplyRelabeling function if relabeling is disabled
...
This should reduce CPU usage a bit when `-relabelConfig` isn't set
2020-07-23 13:35:36 +03:00
Aliaksandr Valialkin
c91ccce50c
app/vminsert: fix relabeling for metrics ingested via Influx line protocol
...
Previously the enabled relabeling with `-relabelConfig` command-line flag could result in missing labels
if a single Influx line protocol message contains multiple field values.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/638
2020-07-23 13:25:37 +03:00
Aliaksandr Valialkin
b8303afcd8
lib/storage: improve prioritizing of data ingestion over querying
...
Prioritize also small merges over big merges.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
2020-07-23 01:40:38 +03:00
Aliaksandr Valialkin
20d0c41ac5
app/vmselect/prometheus: support d
, w
and y
suffixes for durations passed to step
in /api/v1/query_range
like Prometheus does
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/641
2020-07-22 16:27:27 +03:00
Aliaksandr Valialkin
bd4299fafe
app/vmselect/netstorage: reduce memory allocations when unpacking time series data by using a pool for unpackWork entries
...
This should slightly reduce load on GC when processing queries that touch big number of time series.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646 according to the provided memory profile
2020-07-22 15:04:42 +03:00
Aliaksandr Valialkin
a3f48e395e
app/vmagent: add -remoteWrite.decimalPlaces
command-line flag, which may be used for reducing disk space usage on the remote storage
2020-07-21 21:55:42 +03:00
Aliaksandr Valialkin
5bb4fe1ba4
app/vmselect: take into account the time spent in wait queue before query execution as time spent on the query
2020-07-21 19:00:00 +03:00
Aliaksandr Valialkin
0755cb3b50
app/vmselect/promql: skip the first value in time series passed to increase()
if it exceeds by more than 10x the delta between the next value and the first value
...
This should prvent from inflated `increase()` results for time series that start from big initial values.
Such cases may occur when a label value changes in a metric without counter reset.
2020-07-21 17:24:28 +03:00
Aliaksandr Valialkin
71eba8dcf5
app/vmselect: log the total available memory for concurrent requests on not enough memory
errors
...
This should simplify root cause analysis
2020-07-20 19:51:58 +03:00
Aliaksandr Valialkin
3b246aa569
app/vmagent: add -remoteWrite.proxyURL
command-line option
...
This option allows writing data to `-remoteWrite.url` via http, https or socks5 proxy.
This is similar to `proxy_url` option in `remote_write` section of Prometheus.
See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write
2020-07-20 19:31:08 +03:00
Aliaksandr Valialkin
8bee3ef91b
docs/vmagent.md: sync with app/vmagent/README.md
2020-07-20 17:09:30 +03:00
Roman Khavronenko
8949ec961d
app/vmagent: mention grafana dashboard in README ( #639 )
2020-07-20 17:09:27 +03:00
Aliaksandr Valialkin
86b54f3768
app/vmagent/remotewrite: allow passing empty -remoteWrite.urlRelabelConfig
entries
2020-07-20 15:49:13 +03:00
Aliaksandr Valialkin
141e84b5a4
app/vmselect/prometheus: do not return time series with empty list of datapoints from /api/v1/query_range
...
This matches Prometheus behaviour.
This should fix https://github.com/jacksontj/promxy/issues/329
2020-07-20 15:30:13 +03:00
Aliaksandr Valialkin
4d2011a87d
app/vmselect/promql: add mode()
aggregate function
2020-07-20 15:30:11 +03:00
Aliaksandr Valialkin
31ef39e8da
lib/httpserver: log remote address in error message from httpserver.Errorf
...
This should improve detection of the root cause of errors.
Thanks to Anant for the idea.
2020-07-20 14:06:29 +03:00
Aliaksandr Valialkin
427fa43ce2
app/vmselect/promql: add mode_over_time(m[d])
function
...
See https://en.wikipedia.org/wiki/Mode_(statistics) and https://stackoverflow.com/questions/61134078/promql-query-to-return-the-value-from-a-range-vector-which-occurs-maximum-no-of
2020-07-17 18:29:10 +03:00
Aliaksandr Valialkin
eb402a17bd
app/vmselect/promql: optimize group(rollup(m))
calculations
2020-07-17 16:47:30 +03:00
Aliaksandr Valialkin
ea8dc85ba8
app/vmselect/promql: check that any()
doesn't touch metric name
2020-07-17 16:23:11 +03:00
Aliaksandr Valialkin
fc8fe38a82
app/vmselect/promql: add group()
aggregate function to MetricsQL
...
This function has been added in Prometheus 2.20. See https://github.com/prometheus/prometheus/pull/7480
2020-07-17 15:17:38 +03:00
Aliaksandr Valialkin
c64914a7e4
app/vmselect/promql: keep all labels for time series from any()
call
2020-07-17 15:17:37 +03:00
Aliaksandr Valialkin
f9b38f7f2d
app/vminsert/influx: properly handle the case when certain labels with empty values are removed by ApplyRelabeling() call
...
Previously this could lead to `out of range` panic
2020-07-17 00:05:24 +03:00
Aliaksandr Valialkin
14dc426b45
app/vmselect: fix nil pointer dereference
panic when unsuccessfully querying vmstorage
2020-07-16 19:15:18 +03:00
Aliaksandr Valialkin
ce381b3868
app/vmalert: consistently use "%w" instead of "%s" in fmt.Errorf
when wrapping errors
2020-07-15 13:55:13 +03:00
Aliaksandr Valialkin
e6d96bb0bd
docs/vmagent.md: make filtering rules for init container pods less confusing
2020-07-14 20:33:19 +03:00
Aliaksandr Valialkin
c2b4b9138d
app/vmagent/remotewrite: return proper value from tssRelabelPool.New
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/599
2020-07-14 14:28:14 +03:00
Aliaksandr Valialkin
86044f6561
app/{vminsert,vmagent}: add -influxSkipMeasurement
command-line flag for using field name as metric name
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/626
2020-07-14 14:18:40 +03:00
Aliaksandr Valialkin
0e7b2008b2
app/vmselect/prometheus: do not adjust last points in time series with timestamps exceeding the current time
...
Such timestamps usually mean that the query contains `offset`.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/625
2020-07-14 12:56:21 +03:00
Aliaksandr Valialkin
3898cc0285
app/vmselect/prometheus: minimize the diff for the change 1033dc7e2a
over 619b0a25c9
2020-07-13 21:41:17 +03:00
faceair
bf39e67ade
fix empty response template ( #617 )
2020-07-13 21:41:15 +03:00
Aliaksandr Valialkin
b6a5c29549
docs/vmagent.md: sync with app/vmagent/README.md
2020-07-13 21:26:00 +03:00
ofen
9ffa688846
Update README.md ( #621 )
...
Troubleshooting section updated to help out with duplicate targets detection
2020-07-13 21:25:59 +03:00
Aliaksandr Valialkin
4353ff7ef1
app/vmagent: fix data race when multiple -remoteWrite.urlRelabelConfig
options are set
...
Previously multiple goroutines could access remoteWriteCtx.tss concurrently, which could lead to data race
and improper relabeling. Now each goroutine has its own copy of tss during relabeling.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/599
2020-07-10 15:17:23 +03:00
Aliaksandr Valialkin
805a90f642
app/vmagent/remotewrite: typo fix in -remoteWrite.showURL
help message
2020-07-10 14:07:14 +03:00
Aliaksandr Valialkin
6373d377ef
app/{vminsert,vmagent}: add ability to import data in Prometheus exposition format via /api/v1/import/prometheus
2020-07-10 12:13:28 +03:00
Aliaksandr Valialkin
d449d0a0e1
app/vmselect/promql: add missing tests for ifnot
binary operation
2020-07-09 13:24:12 +03:00
Aliaksandr Valialkin
7e706eea13
app/vmselect/promql: refactor implementations for and
and unless
binary operations, so they are closer to or
implementation
2020-07-09 13:06:01 +03:00
Aliaksandr Valialkin
6c1a47b5e0
app/vmselect/promql/active_queries.go: simplify code a bit by inlining getNextActiveQueryID function
2020-07-09 11:18:53 +03:00
Aliaksandr Valialkin
fb86071552
app/vmselect: add /api/v1/status/active_queries
page with the list of currently running queries
...
This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/575
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/528
2020-07-08 19:09:31 +03:00
DexterZhang
9930ce1fa9
Feat/query list vmselect ( #575 )
...
* feat(vmselect): add support for listing current running queries and canceling specific query
* fix(vmselect): change current queries' pid from int64 counter to uuid
* feat(vmselect): add auth to internal operations like `/resetRollupResultCache`, `/query/list` and `/query/kill`. add flag `internalAuthKey` for these auth
* fix(vmselect): add more info to current queries
* review: delete some unnecessary code and use function instead of init
* review: returen *queriesMap in newQueriesMap
* review: delete unused var in struct queriesMap, add comments to exported functions
* review: add return if error occurs
* feat(vmselect): truncate query string in current running query list API since the size of query string might be large;
use query string's pointer in struct `query` for the same reason;
add query info API to get full access of query's info;
2020-07-08 19:04:29 +03:00
Aliaksandr Valialkin
0bff96fe4b
lib/storage: prioritize data ingestion over heavy queries
...
Heavy queries could result in the lack of CPU resources for processing the current data ingestion stream.
Prevent this by delaying queries' execution until free resources are available for data ingestion.
Expose `vm_search_delays_total` metric, which may be used in for alerting when there is no enough CPU resources
for data ingestion and/or for executing heavy queries.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291
2020-07-05 19:44:04 +03:00
Roman Khavronenko
9afd19d375
app/vmalert: add retries to remotewrite ( #605 )
...
* app/vmalert: add retries to remotewrite
Remotewrite pkg now does limited number of retries if write request failed.
This suppose to make vmalert state persisting more reliable.
New metrics were added to remotewrite in order to track rows/bytes sent/dropped.
defaultFlushInterval was increased from 1s to 5s for sanity reasons.
* fix
* wip
* wip
* wip
* fix bits alignment bug for 32-bit systems
* fix mistakenly dropped field
2020-07-05 18:47:38 +03:00
Aliaksandr Valialkin
82871fb7a5
app/vmselect/prometheus: small fixes on top of 8bb762124a
2020-07-05 18:17:53 +03:00
faceair
17f175ff5a
fix adjust last points avoid influence earlier value ( #606 )
2020-07-05 18:17:52 +03:00
Ween
d28fb0baf9
[VMAlert] Fix error log when remoteWrite queue size is full ( #602 )
...
* Fix Auto metrics relabeled errors
* Finalize auto-genenated Labels
* Fix Test Errors
* fix error logs when queue is full
Co-authored-by: xinyulong <xinyulong@kuaishou.com>
2020-07-03 16:50:43 +03:00
Aliaksandr Valialkin
8bb3622e9d
app/vminsert: prevent from adding and/or selecting labels with empty values
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/600
2020-07-02 23:17:12 +03:00
Aliaksandr Valialkin
6ebac3ab63
app/vminsert: add ability to apply relabeling to all the incoming metrics if -relabelConfig
command-line arg points to a file with a list of relabel_config
entries
...
See https://victoriametrics.github.io/#relabeling
2020-07-02 20:36:33 +03:00
Aliaksandr Valialkin
a45856570b
all: typo fix: exptected -> expected
2020-07-02 18:06:21 +03:00
Aliaksandr Valialkin
f10e8809c0
app/vmselect: add interpolate
function for filling gaps with linearly interpolated values
...
See https://stackoverflow.com/q/62565021/274937 for details
2020-07-02 14:54:46 +03:00
Aliaksandr Valialkin
2361ad8ab4
lib/promscrape: add ability to set disable_compression
and disable_keepalive
options in scrape_config
section of the config passed to -promscrape.config
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/580
2020-07-02 14:19:34 +03:00
BigFish
aa26b94f33
fix: spelling mistakes ( #594 )
...
Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
2020-07-01 01:36:40 +03:00
Aliaksandr Valialkin
4cb3e7595c
app/vmstorage: add -denyQueriesOutsideRetention
command-line flag for denying queries outside the configured retention
2020-07-01 00:58:42 +03:00
Aliaksandr Valialkin
0c4e8aeb2b
all: use errors.As
for inspecting errors that implement httpserver.ErrorWithStatusCode
2020-07-01 00:03:11 +03:00
Aliaksandr Valialkin
d962568e93
all: use %w instead of %s for wrapping errors in fmt.Errorf
...
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
2020-06-30 23:33:46 +03:00
Roman Khavronenko
156c83d112
app/vmalert: support multiple notifier urls ( #584 ) ( #590 )
...
* app/vmalert: support multiple notifier urls (#584 )
User now can set multiple notifier URLs in the same fashion
as for other vmutils (e.g. vmagent). The same is correct for
TLS setting for every configured URL. Alerts sending is done
in sequential way for respecting the specified URLs order.
* app/vmalert: add basicAuth support for notifier client (#585 )
The change adds possibility to set basicAuth creds for notifier
client in the same fasion as for remote write/read and datasource.
2020-06-29 22:21:56 +03:00
Roman Khavronenko
bbeab70de6
app/vmalert: move flags description and initialization into subpackages
...
The change adds no new functionality and aims to move flags definitions
to subpackages that are using them. This should improve readability
of the main function.
2020-06-29 22:18:29 +03:00
kreedom
63c36e2e69
app/vmalert: properly set transport for HTTP clients
...
Fixes issue #586
2020-06-29 22:18:25 +03:00
Aliaksandr Valialkin
2b504f17de
docs: update the info that docker images are built on top of alpine
image now
...
A follow-up after the commit ff624c9125
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/522
2020-06-26 13:52:25 +03:00
Aliaksandr Valialkin
a586b8b6d4
app/vminsert/netstorage: do not re-route every time series to more than two vmstorage nodes when certain vmstorage nodes are temporarily slower than the rest of them
...
Previously vminsert may spread data for a single time series across all the available vmstorage nodes
when vmstorage nodes couldn't handle the given ingestion rate. This could lead to increased usage
of CPU and memory on every vmstorage node, since every vmstorage node had to register all the time
series seen in the cluster. Now a time series may spread to maximum two vmstorage nodes under heavy load.
Every time series is routed to a single vmstorage node under normal load.
2020-06-25 16:42:37 +03:00
Aliaksandr Valialkin
12b87b2088
app/vmselect/netstorage: reset big result values every 10 seconds instead of after processing every time series
...
This should reduce GC pressure when processing time series with big number of rows
2020-06-24 19:37:35 +03:00
nicbaz
46c5c0772c
vmselect: fix label_replace when mismatch ( #579 )
...
As per documentation on `label_replace` function: "If the regular
expression doesn't match then the timeseries is returned unchanged".
Currently this behavior is not enforced, if a regexp on an existing
tag doesn't match then the tag value is copied as-is in the destination
tag. This fix first checks that the regular expression matches the
source tag before applying anything.
Given the current implementation, this fix also changes the behavior
of the **MetricsQL** `label_transform` function which does not
document this behavior at the moment.
2020-06-23 23:54:29 +03:00
nicbaz
ea2ed4b7e8
vmalert: add support for TLS configuration ( #578 )
...
app/vmalert: add support for TLS configuration
Add support for TLS optional configuration in a similar fashion to what
is currently supported in other vmutils such as vmagent. TLS
configuration options are distinct for datasource, remoteRead,
remoteWrite as well as notifier.
2020-06-23 22:47:23 +03:00
Aliaksandr Valialkin
0fdbe5de25
app/vmselect/netstorage: increase concurrency when processing small number of time series with big number of data points per each time series
...
Previously VictoriaMetrics was processing up to 32 time series in a single goroutine.
This could be slow if each time series contains big number of data points (10M+ or more), since only a single CPU core could be loaded with work,
while other CPU cores were idle. Fix this by launching GOMAXPROCS workers for time series processing.
This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/572
2020-06-23 22:45:57 +03:00
Aliaksandr Valialkin
3a444bb7bb
lib/promrelabel: add support for keep_if_equal
and drop_if_equal
actions to relabel configs
...
These actions may be useful for filtering out unneeded targets and/or metrics if they contain equal label values.
For example, the following rule would leave the target only if __meta_kubernetes_annotation_prometheus_io_port
equals __meta_kubernetes_pod_container_port_number:
- action: keep_if_equal
source_labels: [__meta_kubernetes_annotation_prometheus_io_port, __meta_kubernetes_pod_container_port_number]
2020-06-23 17:29:19 +03:00
kreedom
f227799c87
Support of custom URL path for alert ( #560 )
...
app/vmalert: Support custom URL for alerts source
Add flag `external.alert.source` for configuring custom URL
for alert's source. This may be handy to re-point default source
URL to other systems like Grafana.
Updates #517
2020-06-21 16:33:58 +03:00
Aliaksandr Valialkin
70bf8218bb
app/vmselect/promql: properly override label values from group_left
and group_right
lists like Prometheus does
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/577
2020-06-21 16:32:27 +03:00
Aliaksandr Valialkin
2fc2679a3f
app/vminsert/netstorage: remove possible race condition when broken connection may be recovered before acquiring storageNode.bcLock
2020-06-20 16:38:08 +03:00
Aliaksandr Valialkin
9409a31c07
docs/vmauth.md: mention that we can provide custom integration with SAML
2020-06-19 13:13:53 +03:00
Aliaksandr Valialkin
4400700832
app/vminsert: properly replicate data for the last RF-1
storage nodes for -replicationFactor=RF
...
Previously the data for the last `RF-1` storage noes has been incorrectly replicated to the first storage node.
2020-06-19 12:40:22 +03:00
Aliaksandr Valialkin
4f673a5201
app/vminsert: export metrics for determining ingested rows with dropped or truncated labels
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/565
2020-06-19 01:12:44 +03:00
Aliaksandr Valialkin
6939e36fdd
app/vmselect/promql: fill gaps on right side with values from left side of or
operator in the same way as Prometheus does
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/552
2020-06-18 23:05:23 +03:00
Aliaksandr Valialkin
85c1ccb8b8
app/vminsert/netstorage: add missing return
in storageNode.checkHealth on connection failure
2020-06-18 20:51:51 +03:00
Aliaksandr Valialkin
464682f380
app/vminsert/netstorage: periodically check for each -storageNode
health, so it could be marked as healthy when it is ready to accept data
...
This fixes uneven data routing in cluster version when `-replicationFactor` is set to 1 (default value),
i.e. when the replication is disabled.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/546
2020-06-18 20:42:43 +03:00
Roman Khavronenko
1a01fe2cf2
vmalert-537: allow name duplication for rules within one group. ( #559 )
...
Uniqueness of rule is now defined by combination of its name, expression and
labels. The hash of the combination is now used as rule ID and identifies rule within the group.
Set of rules from coreos/kube-prometheus was added for testing purposes to
verify compatibility. The check also showed that `vmalert` doesn't support
`query` template function that was mentioned as limitation in README.
2020-06-18 18:54:35 +03:00
Aliaksandr Valialkin
87151e825e
docs/vmbackup.md: mention that backups from single-node and cluster versions are incompatible
2020-06-18 18:54:34 +03:00
Aliaksandr Valialkin
cc2225cc49
app/vmselect: fix the error after 936f35920a
2020-06-12 22:00:45 +03:00
Aliaksandr Valialkin
936f35920a
app/vmselect/prometheus: allow returning partial response from /api/v1/export
if -search.denyPartialResponse=false
...
This makes `/api/v1/export` behaviour consistent with other `/api/v1/*` handlers.
2020-06-12 21:11:48 +03:00
Clémence Saussez
0b53e380cf
app/vmalert: fix link to testdata ( #547 )
...
Fix broken link to vmalert test data
Signed-off-by: Clemence Saussez <clemence@zen.ly>
2020-06-10 19:37:21 +03:00
Roman Khavronenko
d71b6e6584
vmalert-491: allow to configure concurrent rules execution per group. ( #542 )
...
The feature allows to speed up group rules execution by
executing them concurrently.
Change also contains README changes to reflect configuration
details.
2020-06-09 15:22:11 +03:00
Roman Khavronenko
5c049bf4dd
vmalert-521: allow to disable rules expression validation. ( #536 )
...
This feature may be useful for using `vmalert` with PromQL
compatible datasources like Loki.
2020-06-09 15:19:25 +03:00
Aliaksandr Valialkin
c1be462d42
app/vmauth: disable automatic response compression/uncompression, since it may work improperly in some cases
...
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/535
2020-06-05 20:14:07 +03:00
Aliaksandr Valialkin
7680b7155d
app/vmauth: emit fatal errors instead of panics when incorrect command-line flags are set
2020-06-05 20:14:05 +03:00
Aliaksandr Valialkin
01719f4949
app/vmstorage/transport: simplify setupTfss in order to prevent the possibility of nil tfs
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/534
2020-06-05 13:17:26 +03:00
Aliaksandr Valialkin
e4cef1b678
app/vmstorage: prevent from serving conns from vminsert and vmselect after the server is closed
...
Previously it was possible that the connection is served after the server is closed if the following
steps are performed:
1) Server accepts new connection.
2) Server.MustClose() is called and successfully finished.
3) Server starts processing the connection accepted at step 1. There could be various crashes
like in https://github.com/VictoriaMetrics/VictoriaMetrics/issues/534 since the storage may be already closed.
Now the server closes the connection at step 3 without processing it.
2020-06-05 11:55:48 +03:00
Aliaksandr Valialkin
58069f5a6a
app/vmalert: print brief usage info for vmalert -help
2020-06-05 10:43:24 +03:00
Aliaksandr Valialkin
3848ea3a4a
app/vmauth: print brief usage info for vmauth -help
2020-06-05 10:40:11 +03:00
Aliaksandr Valialkin
8ad8ca350a
app/vmagent: print brief usage info for vmagent -help
2020-06-05 10:40:10 +03:00
Aliaksandr Valialkin
3d0a0b3785
lib/fs: optimize MustGetFreeSpace performance by caching the results for up to 2 seconds
2020-06-04 13:14:04 +03:00
DexterZhang
fa103875a0
feat(vmselect): add tmp block dir size metrics vm_tmp_blocks_files_size_total
( #527 )
...
* feat(vmselect): add tmp block dir size metrics `vm_tmp_blocks_files_size_total`
* refactor(vmselect): use free space instead of used space in tmp block file metrics
* fix: add `bytes` suffix to tmp dir free space metric
2020-06-04 13:05:50 +03:00
Aliaksandr Valialkin
faea804b88
app/vmauth: log when -auth.config is reloaded in SIGHUP
2020-06-03 23:22:20 +03:00
Aliaksandr Valialkin
045b87c662
app/vmalert: fix comment for UpdateWith exported methods
2020-06-01 14:35:03 +03:00
Aliaksandr Valialkin
43b14b9569
app/vminsert/netstorage: free up unused memory in buffer after memory usage spikes
2020-06-01 14:33:35 +03:00
Roman Khavronenko
44c51c627f
vmalert: Add recording rules support. ( #519 )
...
* vmalert: Add recording rules support.
Recording rules support required additional service refactoring since
it wasn't planned to support them from the very beginning. The list
of changes is following:
* new entity RecordingRule was added for writing results of MetricsQL
expressions into remote storage;
* interface Rule now unites both recording and alerting rules;
* configuration parser was moved to separate package and now performs
more strict validation;
* new endpoint for listing all groups and rules in json format was added;
* evaluation interval may be set to every particular group;
* vmalert: uncomment tests
* vmalert: rm outdated TODO
* vmalert: fix typos in README
2020-06-01 13:53:46 +03:00
Aliaksandr Valialkin
37aa4fe282
app/vmagent: reload -remoteWrite.relabelConfig and -remoteWrite.urlRelabelConfig on SIGHUP and on /-/reload
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/518
2020-05-30 14:37:02 +03:00
Aliaksandr Valialkin
a646131a33
app/vmagent: log fatal errors instead of panics when improper command-line flags are passed to vmagent
2020-05-30 14:22:38 +03:00
Aliaksandr Valialkin
f41a01332a
app/vminsert/netstorage: evenly distribute rerouted rows among all the availalbe storage nodes
...
Previously such rows were distributed to the original storage node or to the next storage node.
This may result to uneven load among the remaining storage nodes.
2020-05-30 13:51:09 +03:00
Aliaksandr Valialkin
02b2064d8e
app/vminsert/netstorage: do not increment vm_rpc_rows_lost_total when all the vmstorage nodes are unavailable, since vminsert retries sending the data instead of dropping it
2020-05-28 22:36:56 +03:00
Aliaksandr Valialkin
7a61357b5d
app/vminsert/netstorage: make sure that the the data is always replicated among -replicationFactor vmstorage nodes
...
Previously vminsert could write multiple copies of the data to a single vmstorage node when the ingestion rate
exceeds the maximum throughput for connections to vmstorage nodes.
2020-05-28 19:59:07 +03:00
Aliaksandr Valialkin
77e5165e7b
app/vminsert: add -replicationFactor
command-line flag for enabling data replication among available -storageNode instances
2020-05-27 17:29:44 +03:00
Aliaksandr Valialkin
b4e3bffe4b
app/vminsert/netstorage: emit warnings instead of errors when re-routing data to healthy storage nodes
2020-05-27 16:31:41 +03:00
Aliaksandr Valialkin
75f2f3b09d
app/vminsert/netstorage: improve ingestion performance when a single vmstorage node is slower than other vmstorage nodes
...
Previously the ingestion performance has been limited by the slowest vmstorage node.
Now vminsert should re-route data from the slowest vmstorage node to the remaining nodes.
2020-05-27 15:08:22 +03:00
Aliaksandr Valialkin
9844845d79
app/vminsert: tune the maximum summary buffer size for pending data to 1/4 of available RAM, since 1/2 of RAM is too big considering GOGC overhead
2020-05-25 02:00:37 +03:00
Aliaksandr Valialkin
4a82631e44
app/vminsert: limit the summary buffer sizes for all the storage nodes to a half of the allowed memory
2020-05-25 01:39:33 +03:00
Aliaksandr Valialkin
4bd3d4b148
app/vminsert/netstorage: do not return error from storageNode.flushBufLocked when the buffer has been successfully re-routed to healthy nodes
...
This should reduce the number of false errors in the log and the number of falsely lost rows
2020-05-22 18:29:43 +03:00
Aliaksandr Valialkin
6edc33d9bb
app/vminsert/netstorage: capture the first error instead of the last error when sending data to vmstorage
...
The first error has more chances to point to the real root cause of the issue.
2020-05-22 17:49:33 +03:00
Aliaksandr Valialkin
bb4a2bf1aa
app/vmauth: fix make run-vmauth
command
2020-05-22 16:45:19 +03:00
Aliaksandr Valialkin
dcbdc009f5
app/vmagent: check for error returned from flag.Set
2020-05-21 16:30:48 +03:00
Aliaksandr Valialkin
b59e089ac7
app/vmagent: add -dryRun
option for checking all the configs mentioned in command-line flags without running vmagent
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/362
2020-05-21 15:23:18 +03:00
Aliaksandr Valialkin
901093279e
app/vmstorage/transport: update stale comment - vmstorage now sends small ack
packets to vminsert
2020-05-21 14:04:52 +03:00
kreedom
2752d6cb26
vmalert add quotes escape function ( #510 )
...
* vmalert add quotes escape function
Co-authored-by: kreedom
2020-05-21 12:10:35 +03:00
Aaron France
b26245c48b
Update README.md
2020-05-21 12:10:33 +03:00
Aliaksandr Valialkin
d83c68ca03
app/vmselect/promql: add ascent_over_time(m[d])
and descent_over_time(m[d])
functions
...
These functions could be useful in GPS tracking apps for calculating the summary for height gain/loss
over the given duration `d`.
2020-05-21 12:06:34 +03:00
Aliaksandr Valialkin
8ff28f5b91
app/vmselect/promql: update numbers after the upgrade of github.com/VictoriaMetrics/metrics from v1.11.2 to v1.11.3
2020-05-20 03:07:07 +03:00
Aliaksandr Valialkin
ddc9e69bd6
docs/vmagent.md: mention an alternative to refresh_interval
option in scrape configs
2020-05-19 23:10:16 +03:00
Aliaksandr Valialkin
7d46dd452a
app/vmselect/promql: move common code from aggrFuncOutliersK and newAggrFuncRangeTopK into getRangeTopKTimeseries
2020-05-19 16:11:03 +03:00
Aliaksandr Valialkin
37068064dd
app/vmselect/promql: fix outilersk
calculations
2020-05-19 14:45:10 +03:00
Aliaksandr Valialkin
fc81ea38d4
app/vmselect/promql: add outliersk(N, m)
aggregate function for anomaly detection across groups of similar time series
2020-05-19 13:52:44 +03:00
Aliaksandr Valialkin
9ca781b8f0
app/vmalert/notifier: go fmt
2020-05-19 13:00:18 +03:00
kreedom
27911ae179
vmalert - add expr to variables, add escape functions ( #495 )
...
* vmalert - add expr to variables, add escape functions
Co-authored-by: kreedom
2020-05-19 11:55:03 +03:00
Roman Khavronenko
c7f3e58032
vmalert: avoid sending resolves for pending alerts ( #498 )
...
Before the change we were sending notifications to notifier
if following conditions are met:
* alert is in Fire state
* alert is in Inactive state
We were sending Inactive notifications to resolve alert ASAP.
Unfortunately, we were sending resolves for Pending alerts that become
Inactive, which is wrong.
In this change we delete alert from the active list if
it was Pending and become Inactive. In this way we now
have Inactive alerts only if they were in state Fire before.
See test change for example.
2020-05-19 11:55:00 +03:00
Roman Khavronenko
e5f5342e18
vmalert: fix potential race during configuration reloads ( #497 )
...
Configuration reload and rules evaluation can't be executed
in same time now. This may make reload time longer but
prevents from potential races.
2020-05-19 11:54:55 +03:00
Aliaksandr Valialkin
b99d03a956
app/vmalert: run make quicktemplate-gen
from the root dir of the repository
2020-05-16 22:45:45 +03:00
Aliaksandr Valialkin
2784015a4d
all: print --help
output to stdout instead of stderr
...
This is easier to grep and pipe
2020-05-16 12:03:06 +03:00
Aliaksandr Valialkin
dbf8048134
app/vmrestore: document better that vmrestore
works like rsync --delete
, i.e. it deletes files in -storageDataPath
, which are missing in the backup
2020-05-16 09:02:46 +03:00
Aliaksandr Valialkin
e544155a82
app/vmagent/Makefile: fix make run-vmagent
rule
2020-05-15 19:35:16 +03:00
Aliaksandr Valialkin
6c43ba1cb1
app/vmagent/remotewrite: remove unused import after the commit 93267f143f
2020-05-15 17:42:31 +03:00
Aliaksandr Valialkin
1d71253653
app/vmagent/remotewrite: allow ingesting time series with multiple samples at once
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/481
2020-05-15 17:37:27 +03:00
Aliaksandr Valialkin
a853869e75
app/vmstorage/transport: prevent from uncontrolled memory usage growth when vminsert
sends big packets with too long labels
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/490
2020-05-15 15:42:54 +03:00
Aliaksandr Valialkin
1e5c1d7eaa
app/vmstorage: add vm_slow_metric_name_loads_total
metric, which could be used as an indicator when more RAM is needed for improving query performance
2020-05-15 14:12:24 +03:00
Aliaksandr Valialkin
d6b9a49481
app/vmstorage: add vm_slow_row_inserts_total
and vm_slow_per_day_index_inserts_total
metrics for determining whether VictoriaMetrics required more RAM for the current number of active time series
2020-05-15 13:46:57 +03:00