github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
1921 commits 150 branches 748 tags 347 MiB
Commit graph

781 commits

Author SHA1 Message Date
Aliaksandr Valialkin
bc37f1cbec app/vminsert: do not pollute logs with repated cannot dial storageNode errors 
Log only the first error per -storageNode
 2020-09-29 00:20:32 +03:00
Aliaksandr Valialkin
2ee0dc27a6 app/vmstorage: parallelize data processing obtained from a single connection from vminsert 
Previously vmstorage could use only a single CPU core for data processing from a single connection from vminsert.
Now all the CPU cores can be used for data processing from a single connection from vminsert.
This should improve the maximum data ingestion performance for a single vminsert->vmstorage connection.
 2020-09-28 21:41:16 +03:00
Aliaksandr Valialkin
9d123eb22a app/vminsert: remove useless delays when sending data to vmstorage 
This improves the maximum data ingestion performance for cluster VictoriaMetrics

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/791
 2020-09-28 21:41:15 +03:00
Aliaksandr Valialkin
8df33bd5c1 app/{vminsert,vmagent}: improve data ingestion speed over a single connection 
Process data obtianed from a single connection on all the available CPU cores.
 2020-09-28 04:14:51 +03:00
Aliaksandr Valialkin
6d8c23fdbd app/{vminsert,vmselect}: skip accountID and projectID when marshaling/unmarshaling MetricName in /api/v1/export/native and /api/v1/import/native 
This is needed in order to be able to migrate native data from/to single-node VictoriaMetrics
 2020-09-28 00:58:58 +03:00
Aliaksandr Valialkin
db14f22fc0 app/vmselect: stop /api/v1/export/* execution if client disconnects 2020-09-27 23:52:42 +03:00
Aliaksandr Valialkin
aadbd014ff all: add native format for data export and import 
The data can be exported via [/api/v1/export/native](https://victoriametrics.github.io/#how-to-export-data-in-native-format) handler
and imported via [/api/v1/import/native](https://victoriametrics.github.io/#how-to-import-data-in-native-format) handler.
 2020-09-27 17:36:38 +03:00
Aliaksandr Valialkin
e83947a882 app/vminsert: code prettifying 2020-09-26 04:16:14 +03:00
Aliaksandr Valialkin
8fc9b77496 app/vmagent: reduce memory usage when importing data via /api/v1/import 
Previously vmagent could use big amounts of RAM when each ingested JSON line
contained many samples.
 2020-09-26 04:10:13 +03:00
Aliaksandr Valialkin
973df09686 app/vmselect/netstorage: do not spend CPU time on unpacking empty blocks during /api/v1/series calls 2020-09-24 20:44:15 +03:00
Aliaksandr Valialkin
b8bce348c5 app/vmselect/promql: properly limit implicitly set rollup window to -search.maxStalenessInterval 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/784
 2020-09-23 23:24:09 +03:00
Aliaksandr Valialkin
543f3aea97 all: consistently use "%w" formatting in fmt.Errorf for wrapped errors 2020-09-23 22:48:21 +03:00
Aliaksandr Valialkin
8627365b48 app/vmselect/prometheus: code cleanup after 3ba507000c 2020-09-23 13:04:51 +03:00
Aliaksandr Valialkin
1fce79518a app/vmselect/prometheus: return timestamps from /api/v1/query, which match the time query arg 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/720
 2020-09-23 12:59:23 +03:00
Aliaksandr Valialkin
0468cdf33e app/vmselect/netstorage: properly pre-allocate space for sbs 2020-09-22 23:51:01 +03:00
Aliaksandr Valialkin
69eb9783e6 app/vmselect: make sure the request doesnt wait in pending queue more than the configured timeout 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711
 2020-09-22 01:21:42 +03:00
Aliaksandr Valialkin
ed473c94ff docs/vmagent.md: typo fix 2020-09-21 21:49:08 +03:00
Aliaksandr Valialkin
e564725641 app/vmselect/searchutils: fixed tests after 2eb72e09ab 2020-09-21 21:31:28 +03:00
Aliaksandr Valialkin
07c6226334 app/vmselect: use time value rounded to seconds if it isnt passed to /api/v1/query 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/720
 2020-09-21 21:24:46 +03:00
Nikolay Khramchikhin
0069353d5e Add improvements to ec2_sd_discovery (#775) 
* Add improvements to ec2 discovery

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/771

 role_arn support with aws sts
 instance iam_role support
 refreshing temporary tokens

* Apply suggestions from code review

Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>

* changed implementation, removed tests, clean up code

* moved endpoint builder into getEC2APIResponse

Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
 2020-09-21 16:05:01 +03:00
Aliaksandr Valialkin
8cd89cb847 app/vmalert: remove unneeded UTC() call 
UTC() doesn't change the underlying timestamp, so the call isn't needed here
 2020-09-21 15:56:48 +03:00
Roman Khavronenko
d111969d39 vmalert: add support for datasource.lookback flag (#779) 
New datasource flag `datasource.lookback` defines how far to look into
past when evaluating queries.

Address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/668
 2020-09-21 15:56:47 +03:00
Roman Khavronenko
0042b0f307 vmalert: fix the typo in error message (#782) 
The error will be always nil so no sense in printing it.
 2020-09-21 11:36:09 +03:00
Aliaksandr Valialkin
1c07d7bee7 app/vmagent: increase default value for -remoteWrite.queues from 1 to 4, since it has been appeared that many users hit this limit 2020-09-18 14:22:02 +03:00
Aliaksandr Valialkin
a1bebb660c app/vmselect/graphite: return proper results /metrics/find?query=foo.*.bar according to Graphite Metrics API 2020-09-18 11:53:28 +03:00
Aliaksandr Valialkin
9b15b11f74 app/vmstorage: added -forceMergeAuthKey command-line flag for protecting /internal/force_merge endpoint 2020-09-17 14:24:20 +03:00
Aliaksandr Valialkin
d96858b921 lib/storage: add /internal/force_merge handler for running forced compactions on historical per-month partitions 
This may be useful for freeing up storage space after time series deletion.

See https://victoriametrics.github.io/#force-merge for more details.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686
 2020-09-17 12:20:56 +03:00
Aliaksandr Valialkin
406f4fe445 app/vmagent: substitute -remoteWrite.url with secret-url value in logs, since it may contain sensitive info such as passwords or auth tokens 
Pass `-remoteWrite.showURL` command-line flag in order to see real `-remoteWrite.url` values in logs and at `/metrics` page.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/773
 2020-09-16 22:36:18 +03:00
Aliaksandr Valialkin
9793008734 app/vmselect: add -search.storageTimeout command-line flag for limiting the maximum duration of query execution per each -storageNode 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711
 2020-09-16 21:33:47 +03:00
Aliaksandr Valialkin
a9205fe308 app/vmselect: prevent from closing connection to vmstorage on query timeout by setting +2 secs deadline on connection comparing to query deadline 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711
 2020-09-16 21:14:00 +03:00
Aliaksandr Valialkin
1587f83fa0 app/vmselect/netstorage: typo fix after 03dfccfbed 2020-09-16 00:10:33 +03:00
Aliaksandr Valialkin
03dfccfbed app/vmselect/netstorage: reduce memory usage when the time range from query touches big number of samples per each time series 2020-09-15 21:08:09 +03:00
Aliaksandr Valialkin
27cd5555e6 app/vmselect/netstorage: mention that RunParallel or Cancel must be called on the returned results from ProcessSearchQuery 2020-09-15 20:39:43 +03:00
Aliaksandr Valialkin
1f0c0b0f6b app/vmselect: typo fix in -search.maxStalenessInterval description 2020-09-15 14:25:34 +03:00
Roman Khavronenko
e2b31590e6 vmalert: add Group name as label to generated alerts and timeseries (#761) 
Solves #611
 2020-09-11 23:41:12 +03:00
Roman Khavronenko
16e0bb496e vmalert: update groups on config reload only if changes detected (#759) 
On config reload event `vmalert` reloads configuration for every group. While
it works for simple configurations, the more complex and heavy installations may
suffer from frequent config reloads.
The change introduces the `checksum` field for every group and is set to md5 hash
of yaml configuration. The checksum will change if on any change to group
definition like rules order or annotation change. Comparing the `checksum` field
on config reload event helps to detect if group should be updated.
The groups update is now done concurrently, so reload duration will be limited by
the slowest group now.

Partially solves #691 by improving config reload speed.
 2020-09-11 23:41:12 +03:00
Aliaksandr Valialkin
b776a93608 app/vmselect/promql: support composite durations like Prometheus 2.21 does 
The following durations are supported now: `1h5m35s` or `1s543ms`

See https://github.com/prometheus/prometheus/releases/tag/v2.21.0
and https://github.com/prometheus/prometheus/pull/7713
 2020-09-11 22:17:24 +03:00
Aliaksandr Valialkin
6382e8081a app/vmagent: allow setting multiple identical -remoteWrite.url values 
This may be useful when each url is authenticated via different `-remoteWrite.basicAuth.username`.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/755
 2020-09-11 15:17:29 +03:00
Aliaksandr Valialkin
d3ad0d365e app/vmselect: move Deadline from netstorage to searchutils 
This removes dependency on netstorage from searchutils.
 2020-09-11 13:39:13 +03:00
Aliaksandr Valialkin
58d3b82ae5 app/{vminsert,vmagent}: allow passing timestamp via timestamp query arg when ingesting data to /api/v1/import/prometheus 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/750
 2020-09-11 13:28:31 +03:00
Aliaksandr Valialkin
579c20756a app/vmselect: substitute inf values at smooth_exponential with the previous values 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/757
 2020-09-11 12:23:56 +03:00
Aliaksandr Valialkin
d67e6d3d2e app/vmselect: skip infinite values when calculating smooth_exponential 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/757
 2020-09-11 11:57:53 +03:00
Aliaksandr Valialkin
06427a184f app/vmselect/graphite: typo fix in label name for vm_request_duration_seconds metric 2020-09-11 01:59:52 +03:00
Aliaksandr Valialkin
f307e6f432 app/vmselect: initial implementation of Graphite Metrics API 
See https://graphite-api.readthedocs.io/en/latest/api.html#the-metrics-api
 2020-09-11 00:30:20 +03:00
Aliaksandr Valialkin
f5cb213ef9 lib/storage: reuse timestamp blocks for adjancent metric blocks with identical timestamps 
This should reduce disk space usage when scraping targets containing metrics with identical names
such as `node_cpu_seconds_total`, histograms, quantiles, etc.

Expose `vm_timestamps_blocks_merged_total` and `vm_timestamps_bytes_saved_total` metrics for monitoring
the effectiveness of timestamp blocks merging.
 2020-09-09 23:59:21 +03:00
Aliaksandr Valialkin
475698d2ad docs: sync docs for vmalert, vmauth, vmbackup and vmrestore 2020-09-09 21:10:48 +03:00
Aliaksandr Valialkin
5ab57f916b docs/vmagent.md: clarified the case when -remoteWrite.queues must be tuned 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/745
 2020-09-08 20:15:49 +03:00
Aliaksandr Valialkin
f3a79abfb4 app/vmselect/promql: go fmt 2020-09-08 15:18:57 +03:00
Aliaksandr Valialkin
4f06eed1c1 app/vmselect/promql: adjust integrate() calculations to be more similar to calculations from InfluxDB: attempt #2 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/701
 2020-09-08 14:36:23 +03:00
Aliaksandr Valialkin
0d0b606455 app/vmselect/promql: adjust integrate() calculations to be more similar to calculations from InfluxDB 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/701
 2020-09-08 14:24:02 +03:00
Aliaksandr Valialkin
db91045348 app/vmselect/promql: increase floating point calculations accuracy by dividing by 1e3 instead of multiplying by 1e-3 2020-09-08 14:01:02 +03:00
Aliaksandr Valialkin
804304c365 app/vmselect: add missing deletion for temporary files on partial responses when -search.denyPartialResponse=true 2020-09-04 02:23:12 +03:00
Aliaksandr Valialkin
478d8f8393 app/vmselect/promql: add count_le_over_time(m[d], le) and count_gt_over_time(m[d], gt) functions 
These functions returns the number of raw samples that don't exceed `le` or are bigger than `gt`.
These functions are complement to already existing `share_le_over_time(m[d], le)` and `share_gt_over_time(m[d], gt)`.
 2020-09-03 15:28:58 +03:00
Aliaksandr Valialkin
3490160fd0 app/vmselect: unconditionally align time range boundaries to step for subqueries as Prometheus does 2020-09-03 13:22:06 +03:00
Aliaksandr Valialkin
a3cdef6b06 app/vmagent: properly flush big blocks of data 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/741

Thanks to @IceRain00 for the investigation and initial attempt to fix the issue
at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/742
 2020-09-03 12:12:12 +03:00
Aliaksandr Valialkin
de216bab41 app/vmagent: fix data race when accessing writeRequest.lastFlushTime 2020-09-03 12:12:09 +03:00
Nikolay Khramchikhin
80a9dc79fe changed vmalert behaviour (#738) 
* VMAlert start with empty rules dir

There are some applications (operator for instance), that generates alerts configuration at runtime
and vmalert must start correctly without rules to support this behaviour.
Later application will add rules files and send SIGHUP to vmalert,
which will trigger reading rules files and start rules exectuion.

Removing rules files with SIGHUP signal must stop rules execution and
vmalert will wait for new rules.

* imports sorted

* added test cases for empty rules, removed blank line

* fixed imports conflict

* updated tests
 2020-09-03 11:07:40 +03:00
Aliaksandr Valialkin
7ac10ee978 app/vmalert: imrovements over 3f932c2db1 2020-09-03 01:14:30 +03:00
DexterZhang
85f49ad439 feat: spread load of rule evaluation by group when starting new groups (#724) 
* feat: spread load of rule evaluation by group when starting new groups

* review: reduce the resulting diff.

* Update app/vmalert/group.go

Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
 2020-09-03 01:14:26 +03:00
Aliaksandr Valialkin
4fa97430d7 app/{vminsert,vmagent}: allow adding extra labels when importing data via Prometheus, CSV and JSON line formats 
Extra labels may be added to the imported data by passing `extra_label=name=value` query args.
Multiple query args may be passed in order to add multiple extra labels.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/719
 2020-09-02 19:47:02 +03:00
Aliaksandr Valialkin
fe08b1eb26 app/vminsert: improve error message when the data cannot be sent to vmstorage - log reroutedBR buffer size 
This should improve debuggability for improperly configured cluster
 2020-08-31 17:51:44 +03:00
Aliaksandr Valialkin
6f9c1bc078 app/vmagent: log unsuccessful attempt number when sending data to -remoteWrite.url 2020-08-30 21:40:15 +03:00
Aliaksandr Valialkin
3b1ecac04b app/vmagent: apply sane limits to -remoteWrite.queues 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/707
 2020-08-30 21:25:51 +03:00
Roman Khavronenko
08b76cb26f vmalert: update -rule flag description to enforce quotes using (#709) 
Description for `-rule` flag uses as example specific chars like asterisks
which could be interpreted wrong by different shells. To avoid this, description
now contains quoted flag values.

See also #708
 2020-08-28 09:46:35 +03:00
Roman Khavronenko
4b89da9463 lib/decimal: rename significant decimal digits to significant figures (#698) 
The previous notion was inconsistent with what `decimal.Round` does.
According to [wiki](https://en.wikipedia.org/wiki/Significant_figures) rounding
applied to all significant figures, not just decimal ones.
 2020-08-16 17:22:40 +03:00
Aliaksandr Valialkin
6aab2f4989 all: allow using KB, MB, GB, KiB, MiB and GiB suffixes in command-line flag values related to byte sizes or byte rates 2020-08-16 17:08:28 +03:00
Aliaksandr Valialkin
285665e93b app/vmselect/promql: allow passing multiple args to aggregate functions such as avg(q1, q2, q3) 2020-08-15 01:15:16 +03:00
Aliaksandr Valialkin
a2021d0dde docs/vmagent.md: mention that gaps in remote storage may appear if vmagent cannot keep up with data ingestion 2020-08-14 20:48:17 +03:00
Aliaksandr Valialkin
e7c0b2ca56 docs: update docs 2020-08-14 19:14:46 +03:00
Aliaksandr Valialkin
b996280c65 app/{vminsert,vmagent}: improve documentation for -influxListenAddr command-line flag 2020-08-14 18:03:08 +03:00
Aliaksandr Valialkin
60c7397be5 all: support %{ENV_VAR} placeholders in yaml configs in all the vm* components 
Such placeholders are substituted by the corresponding environment variable values.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/583
 2020-08-13 17:17:06 +03:00
Aliaksandr Valialkin
6721e47ae9 app: respect CPU limits set via cgroups 
Update GOMAXPROCS to limits set via cgroups. This should reduce CPU trashing and reduce memory usage
for cases when VictoriaMetrics components run in containers with CPU limits.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685
 2020-08-11 23:01:03 +03:00
Aliaksandr Valialkin
62b6e54622 app/vmselect: reduce memory usage when exporting time series with big number of samples via /api/v1/export if max_rows_per_line is set to non-zero value 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685
 2020-08-10 20:57:43 +03:00
Aliaksandr Valialkin
c9f5c5623f app/vmselect/netstorage: vary batch size for data unpacking depending on the available CPU cores 
This should reduce contention on the channel with unpack work for systems with high number of CPU cores
 2020-08-10 15:16:48 +03:00
Aliaksandr Valialkin
b3d4ff7ee2 app/vmstorage: improve error logging when the request times out 2020-08-10 13:17:24 +03:00
Roman Khavronenko
78afc61896 app/vmalert: extend metrics set exported by vmalert #573 (#654) 
* app/vmalert: extend metrics set exported by `vmalert` #573

New metrics were added to improve observability:
+ vmalert_alerts_pending{alertname, group} - number of pending alerts per group
per alert;
+ vmalert_alerts_acitve{alertname, group} - number of active alerts per group
per alert;
+ vmalert_alerts_error{alertname, group} - is 1 if alertname ended up with error
during prev execution, is 0 if no errors happened;
+ vmalert_recording_rules_error{recording, group} - is 1 if recording rule
 ended up with error during prev execution, is 0 if no errors happened;
* vmalert_iteration_total{group, file} - now contains group and file name labels.
This should improve control over specific groups;
* vmalert_iteration_duration_seconds{group, file} - now contains group and file name labels. This should improve control over specific groups;

Some collisions for alerts and recording rules are possible, because neither
group name nor alert/recording rule name are unique for compatibility reasons.

Commit contains list of TODOs for Unregistering metrics since groups and rules
are ephemeral and could be removed without application restart. In order to
unlock Unregistering feature corresponding PR was filed - https://github.com/VictoriaMetrics/metrics/pull/13

* app/vmalert: extend metrics set exported by `vmalert` #573

The changes are following:
* add an ID label to rules metrics, since `name` collisions within one group is
a common case - see the k8s example alerts;
* supports metrics unregistering on rule updates. Consider the case when one rule
was added or removed from the group, or the whole group was added or removed.

The change depends on https://github.com/VictoriaMetrics/metrics/pull/16
where race condition for Unregister method was fixed.
 2020-08-09 09:42:05 +03:00
ofen
3fea7c39be 401 Unauthorize HTTP error added (#681) 
401 Unauthorize HTTP error added to trigger browser credentials pop-up promt [RFC 7235 https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication]
 2020-08-09 09:39:37 +03:00
Aliaksandr Valialkin
67cacb22ac lib/httpserver: add -tls, -tlsCertFile and -tlsKeyFile command-line flags in every vm binary 
This makes such binaries compatible with binaries from `master` branch (aka single-node version)

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/677
 2020-08-07 10:57:32 +03:00
Aliaksandr Valialkin
95a8c492ef app/vmselect/promql: properly handle -n^m like Prometheus does 
`-n^m` must be handled as `-(n^m)` instead of `(-n)^m`.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/675
 2020-08-07 07:42:42 +03:00
Aliaksandr Valialkin
7f93d61a56 app/vmselect/promql: remove metric name after applying clamp_min and clamp_max functions in order to be consistent with Prometheus 
This improves VictoriaMetrics score at https://promlabs.com/promql-compliance-test-results-victoriametrics/
 2020-08-06 23:42:55 +03:00
Aliaksandr Valialkin
01000505a0 app/vmselect/promql: remove metric name after applying ceil, floor and round functions in order to be more consistent with Prometheus 
This improves VictoriaMetrics score at https://promlabs.com/promql-compliance-test-results-victoriametrics/
 2020-08-06 23:34:03 +03:00
Aliaksandr Valialkin
75bff1a567 app/vmselect/promql: remove metric name from results of certain rollup functions in order to be consistent with Prometheus 
Rollup functions:

  - avg_over_time
  - min_over_time
  - max_over_time
  - quantile_over_time

This improves VictoriaMetrics results at https://promlabs.com/promql-compliance-test-results-victoriametrics/
 2020-08-06 23:29:18 +03:00
Aliaksandr Valialkin
8835004a4c app/vmselect: properly handle PromQL queries like scalar1 < metric < scalar2 like Prometheus does 
This fixes some cases from https://promlabs.com/promql-compliance-test-results-victoriametrics/
 2020-08-06 23:21:14 +03:00
Aliaksandr Valialkin
14ddb8a34e app/vmselect/netstorage: reduce CPU contention when upacking time series blocks by unpacking batches of such blocks instead of a single block 
This should improve query performance on systems with big number of CPU cores (16 and more)
 2020-08-06 17:50:13 +03:00
Aliaksandr Valialkin
46c98cd97a app/vmselect/netstorage: reduce contention on unpackworkCh and timeseriesWorkCh for multi-CPU system by providing more capacity for these chans 2020-08-06 17:22:39 +03:00
Aliaksandr Valialkin
a455930ab4 app/vmstorage: rename vm_cache_size_entries{type="storage/prefetchedMetricIDs"} to vm_cache_entries{type="storage/prefetchedMetricIDs"} to be consistent with other vm_cache_entries metrics 2020-08-06 16:34:18 +03:00
Aliaksandr Valialkin
a3e91c593b lib/storage: limit the number of concurrent calls to storage.searchTSIDs to GOMAXPROCS*2 
This should limit the maximum memory usage and reduce CPU trashing on vmstorage
when multiple heavy queries are executed.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
 2020-08-05 18:27:21 +03:00
Aliaksandr Valialkin
a930460236 app/vmagent: tune http client for sending data to remote storage in order to disable closing keep-alive connections 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/663
 2020-08-04 21:01:40 +03:00
Aliaksandr Valialkin
a04f4a3d9a app/vmselect: use warning level instead of info level for logging slow queries that take longer than -search.logSlowQueryDuration 2020-08-04 20:24:38 +03:00
Aliaksandr Valialkin
bdb881c43b app/vmselect/promql: add zscore-related functions: zscore_over_time(m[d]) and zscore(q) by (...) 2020-08-03 21:52:15 +03:00
Aliaksandr Valialkin
94471a1273 app: remove duplicate *-pure makefile rules 2020-07-31 20:01:30 +03:00
Aliaksandr Valialkin
a2aa3a60eb app/vmselect: show X-Forwarded-For contents on /api/v1/status/active_queries page 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/659
 2020-07-31 20:01:09 +03:00
Aliaksandr Valialkin
106e302d7a all: add mssing APP_NAME to vm*-GOARCH builds 2020-07-31 13:45:32 +03:00
Aliaksandr Valialkin
945645f38f docs/{vmagent,vmalert}: add instruction on how to build for ARM 2020-07-31 09:25:41 +03:00
Aliaksandr Valialkin
0c00fe70cf app/vmselect: do not adjust start and end query args passed to /api/v1/query_range when -search.disableCache command-line flag is set 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/563
 2020-07-30 23:14:56 +03:00
Aliaksandr Valialkin
29bbab0ec9 lib/storage: remove prioritizing of merging small parts over merging big parts, since it doesn't work as expected 
The prioritizing could lead to big merge starvation, which could end up in too big number of parts that must be merged into big parts.

Multiple big merges may be initiated after the migration from v1.39.0 or v1.39.1. It is OK - these merges should be finished soon,
which should return CPU and disk IO usage to normal levels.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/618
 2020-07-30 20:02:22 +03:00
Aliaksandr Valialkin
338ee47d60 app/vmselect/promql: return non-empty value from rate_over_sum(m[d]) even if a single data point is located in the given [d] window 
Just divide the data point value by the window duration in this case.
 2020-07-29 12:37:34 +03:00
Aliaksandr Valialkin
717c554fb0 app/vmselect/promql: remove rollupFuncArg.realPrevValue handling, since the corner case in increase() is handled in another way now 
See e00cfc854d for the approach used now.
 2020-07-29 12:37:34 +03:00
Aliaksandr Valialkin
d9037b3970 app/vmselect/promql: fill gaps with 0 in rate_over_sum response when the last value before the selected time window isnt empty 2020-07-29 12:37:34 +03:00
Aliaksandr Valialkin
f6d4275087 app/{vmagent,vminsert}: properly preserve db tag from query string passed to Influx line protocol query 
Previously `db` tag from the query string wasn't added to metrics after encountering `db` tag in the Influx line

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/653
 2020-07-28 21:25:49 +03:00
Aliaksandr Valialkin
baebe86844 app/vmagent/remotewrite: add missing resp.Body.Close() after pushing data to remote storage 
Missing body close could disable HTTP keep-alive connections.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/653
 2020-07-28 21:00:25 +03:00
Aliaksandr Valialkin
0f6f0d30d3 app/vmselect: show query origin (aka remote_addr or client address) on the /api/v1/status/active_queries page for every query 2020-07-28 15:14:40 +03:00
Roman Khavronenko
ec6ed467c6 app/vmalert: support external.label to specify global labelset for all rules #622 (#652) 
`external.label` flag supposed to help to distinguish alert or recording rules
source in situations when more than one `vmalert` runs for the same datasource
or AlertManager.
 2020-07-28 14:23:04 +03:00
Aliaksandr Valialkin
9dccedc599 app/vmselect/promql: return empty values from group() if all the time series have no values at the given timestamp 
This aligns `group()` behaviour to Prometheus
 2020-07-28 13:41:04 +03:00
Aliaksandr Valialkin
d5057f6d04 app/vmagent/remotewrite: create new request on failure to send a block of data to remote storage 
Previously the request body was already consumed before the retry, so this led to the following error:

    http: ContentLength=... with Body length 0
 2020-07-27 17:33:05 +03:00
Aliaksandr Valialkin
b191e425b3 app/vmselect/promql: improve further the accuracy of buckets_limit() function 
The accuracy is increased by mergin the smallest bucket with the smallest adjacent bucket.
 2020-07-26 12:10:56 +03:00
Aliaksandr Valialkin
43871e79c6 app/vmselect/promql: avoid dropping inf bucket in buckets_limit 
The `le="inf"` bucket must be preserved in order to maintain the maximum level of accuracy.
 2020-07-25 17:00:25 +03:00
Aliaksandr Valialkin
978c1e930e app/vmselect/promql: optimize buckets_limit(k, buckets) for big number of buckets 2020-07-25 13:24:33 +03:00
Aliaksandr Valialkin
51cbf27077 app/vmselect/promql: improve the accuracy of buckets_limit(k, buckets) function 
Now it properly merges the bucket with the previous bucket after deletion.
 2020-07-24 17:07:30 +03:00
Aliaksandr Valialkin
cf69b1ea6f app/vmselect/promql: add buckets_limit(k, buckets) function, which limits the number of buckets per time series to k 
This function works with both Prometheus-style and VictoriaMetrics-style buckets.
The function removes buckets with the lowest values in order to reserve the highest precision.
The function is useful for building heatmaps in Grafana from too big number of buckets.
 2020-07-24 16:14:12 +03:00
Aliaksandr Valialkin
45334f61de app/vmselect: fix tests for rate_over_sum 2020-07-24 02:35:09 +03:00
Aliaksandr Valialkin
3526e8768a app/vmselect/promql: typo fix after 3e557c9861 2020-07-24 02:15:23 +03:00
Aliaksandr Valialkin
8d1721d128 app/vmselect/promql: add rate_over_sum(m[d]) function to MetricsQL, which returns rate over sum of m values over d duration 
Something like `sum_over_time(m[d]) / d`, but more accurate.
 2020-07-24 01:17:15 +03:00
Aliaksandr Valialkin
88e8bed0c9 app/vmselect/promql: allow setting [d] window smaller than the interval between raw points for avg_over_time 
This makes `avg_over_time` behavior consistent with `sum_over_time` and `count_over_time` behaviors.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/636
 2020-07-23 22:25:33 +03:00
Aliaksandr Valialkin
fb3d1380ac lib/storage: respect -search.maxQueryDuration when searching for time series in inverted index 
Previously the time spent on inverted index search could exceed the configured `-search.maxQueryDuration`.
This commit stops searching in inverted index on query timeout.
 2020-07-23 21:22:05 +03:00
Aliaksandr Valialkin
dbf3038637 lib/storage: add more fine-grained pace limiting for search 2020-07-23 19:21:49 +03:00
Aliaksandr Valialkin
16a4b1b20c app/vmselect/netstorage: protect from too smart compiler, which may break memory usage optimization in tmpBlocksFileWrapper.WriteBlocks 2020-07-23 17:57:24 +03:00
Aliaksandr Valialkin
0750d2cec1 app/vminsert: export vm_relabel_metrics_dropped_total metric that shows the number of metrics dropped due to relabeling 2020-07-23 14:58:02 +03:00
Aliaksandr Valialkin
55ed07add1 app/vmselect: typo fix after 0168e21fe32776e2f7f003f88e0e6e490eb2dcb0g 2020-07-23 14:11:15 +03:00
Aliaksandr Valialkin
7aa5b48508 app/vmselect: reduce memory usage when querying big number of time series with long labels 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646
 2020-07-23 13:48:58 +03:00
Aliaksandr Valialkin
49a0011837 app/vminsert: do not call ApplyRelabeling function if relabeling is disabled 
This should reduce CPU usage a bit when `-relabelConfig` isn't set
 2020-07-23 13:35:36 +03:00
Aliaksandr Valialkin
c91ccce50c app/vminsert: fix relabeling for metrics ingested via Influx line protocol 
Previously the enabled relabeling with `-relabelConfig` command-line flag could result in missing labels
if a single Influx line protocol message contains multiple field values.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/638
 2020-07-23 13:25:37 +03:00
Aliaksandr Valialkin
b8303afcd8 lib/storage: improve prioritizing of data ingestion over querying 
Prioritize also small merges over big merges.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
 2020-07-23 01:40:38 +03:00
Aliaksandr Valialkin
20d0c41ac5 app/vmselect/prometheus: support d, w and y suffixes for durations passed to step in /api/v1/query_range like Prometheus does 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/641
 2020-07-22 16:27:27 +03:00
Aliaksandr Valialkin
bd4299fafe app/vmselect/netstorage: reduce memory allocations when unpacking time series data by using a pool for unpackWork entries 
This should slightly reduce load on GC when processing queries that touch big number of time series.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/646 according to the provided memory profile
 2020-07-22 15:04:42 +03:00
Aliaksandr Valialkin
a3f48e395e app/vmagent: add -remoteWrite.decimalPlaces command-line flag, which may be used for reducing disk space usage on the remote storage 2020-07-21 21:55:42 +03:00
Aliaksandr Valialkin
5bb4fe1ba4 app/vmselect: take into account the time spent in wait queue before query execution as time spent on the query 2020-07-21 19:00:00 +03:00
Aliaksandr Valialkin
0755cb3b50 app/vmselect/promql: skip the first value in time series passed to increase() if it exceeds by more than 10x the delta between the next value and the first value 
This should prvent from inflated `increase()` results for time series that start from big initial values.
Such cases may occur when a label value changes in a metric without counter reset.
 2020-07-21 17:24:28 +03:00
Aliaksandr Valialkin
71eba8dcf5 app/vmselect: log the total available memory for concurrent requests on not enough memory errors 
This should simplify root cause analysis
 2020-07-20 19:51:58 +03:00
Aliaksandr Valialkin
3b246aa569 app/vmagent: add -remoteWrite.proxyURL command-line option 
This option allows writing data to `-remoteWrite.url` via http, https or socks5 proxy.
This is similar to `proxy_url` option in `remote_write` section of Prometheus.
See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write
 2020-07-20 19:31:08 +03:00
Aliaksandr Valialkin
8bee3ef91b docs/vmagent.md: sync with app/vmagent/README.md 2020-07-20 17:09:30 +03:00
Roman Khavronenko
8949ec961d app/vmagent: mention grafana dashboard in README (#639) 2020-07-20 17:09:27 +03:00
Aliaksandr Valialkin
86b54f3768 app/vmagent/remotewrite: allow passing empty -remoteWrite.urlRelabelConfig entries 2020-07-20 15:49:13 +03:00
Aliaksandr Valialkin
141e84b5a4 app/vmselect/prometheus: do not return time series with empty list of datapoints from /api/v1/query_range 
This matches Prometheus behaviour.

This should fix https://github.com/jacksontj/promxy/issues/329
 2020-07-20 15:30:13 +03:00
Aliaksandr Valialkin
4d2011a87d app/vmselect/promql: add mode() aggregate function 2020-07-20 15:30:11 +03:00
Aliaksandr Valialkin
31ef39e8da lib/httpserver: log remote address in error message from httpserver.Errorf 
This should improve detection of the root cause of errors.
Thanks to Anant for the idea.
 2020-07-20 14:06:29 +03:00
Aliaksandr Valialkin
427fa43ce2 app/vmselect/promql: add mode_over_time(m[d]) function 
See https://en.wikipedia.org/wiki/Mode_(statistics) and https://stackoverflow.com/questions/61134078/promql-query-to-return-the-value-from-a-range-vector-which-occurs-maximum-no-of
 2020-07-17 18:29:10 +03:00
Aliaksandr Valialkin
eb402a17bd app/vmselect/promql: optimize group(rollup(m)) calculations 2020-07-17 16:47:30 +03:00
Aliaksandr Valialkin
ea8dc85ba8 app/vmselect/promql: check that any() doesn't touch metric name 2020-07-17 16:23:11 +03:00
Aliaksandr Valialkin
fc8fe38a82 app/vmselect/promql: add group() aggregate function to MetricsQL 
This function has been added in Prometheus 2.20. See https://github.com/prometheus/prometheus/pull/7480
 2020-07-17 15:17:38 +03:00
Aliaksandr Valialkin
c64914a7e4 app/vmselect/promql: keep all labels for time series from any() call 2020-07-17 15:17:37 +03:00
Aliaksandr Valialkin
f9b38f7f2d app/vminsert/influx: properly handle the case when certain labels with empty values are removed by ApplyRelabeling() call 
Previously this could lead to `out of range` panic
 2020-07-17 00:05:24 +03:00
Aliaksandr Valialkin
14dc426b45 app/vmselect: fix nil pointer dereference panic when unsuccessfully querying vmstorage 2020-07-16 19:15:18 +03:00
Aliaksandr Valialkin
ce381b3868 app/vmalert: consistently use "%w" instead of "%s" in fmt.Errorf when wrapping errors 2020-07-15 13:55:13 +03:00
Aliaksandr Valialkin
e6d96bb0bd docs/vmagent.md: make filtering rules for init container pods less confusing 2020-07-14 20:33:19 +03:00
Aliaksandr Valialkin
c2b4b9138d app/vmagent/remotewrite: return proper value from tssRelabelPool.New 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/599
 2020-07-14 14:28:14 +03:00
Aliaksandr Valialkin
86044f6561 app/{vminsert,vmagent}: add -influxSkipMeasurement command-line flag for using field name as metric name 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/626
 2020-07-14 14:18:40 +03:00
Aliaksandr Valialkin
0e7b2008b2 app/vmselect/prometheus: do not adjust last points in time series with timestamps exceeding the current time 
Such timestamps usually mean that the query contains `offset`.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/625
 2020-07-14 12:56:21 +03:00
Aliaksandr Valialkin
3898cc0285 app/vmselect/prometheus: minimize the diff for the change 1033dc7e2a over 619b0a25c9 2020-07-13 21:41:17 +03:00
faceair
bf39e67ade fix empty response template (#617) 2020-07-13 21:41:15 +03:00