github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
3562 commits 150 branches 748 tags 347 MiB
Commit graph

1162 commits

Author SHA1 Message Date
Aliaksandr Valialkin
8dcf814c48
app/{vmbackup,vmrestore}: switch from gcs://... to gs://... urls for backups to GCS 
The `gs://` urls are commonly used, so prefer them instead of `gcs://` urls,
while leaving support for `gcs://` urls for backwards compatibility.
 2021-09-29 12:12:37 +03:00
Aliaksandr Valialkin
f35f9188bc
docs/vmagent.md: update Telegraf config in the section about Kafka 2021-09-29 12:01:46 +03:00
Aliaksandr Valialkin
16d7ee7b44
docs/vmagent.md: add docs about reading metrics from Kafka 2021-09-29 01:46:40 +03:00
Aliaksandr Valialkin
428c03c3b8
docs/CHANGELOG.md: document the bugfix from de810031bf 2021-09-29 00:42:21 +03:00
Aliaksandr Valialkin
2577e7c306
docs/CHANGELOG.md: document 3d17112a7e 2021-09-29 00:33:41 +03:00
Aliaksandr Valialkin
4e65bfcc00
app/{vminsert,vmagent}: add ability to ingest data via DataDog "submit metrics" API 
See https://docs.datadoghq.com/api/latest/metrics/#submit-metrics

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/206
 2021-09-29 00:12:26 +03:00
Yury Molodov
80df31b2ee
vmui: use Chart.js as default engine for graph (#1634) 
* feat: add Plotly as default engine for graph

* fix: remove unused components

* feat: use Chart.js as default engine graph

* fix: correct styles for loader

* feat: add zoom/pan for chart

* feat: add height for chart

* fix: remove unused code

* fix: remove empty units from duration

* fix: change debounce for pan to 500ms

* fix: add utility for plugins register globally

* fix: optimize render graph

* feat: add buffer data for zoom

* fix: add limits for zoom in/out

* fix: change update data while zooming

* app/vmselect: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2021-09-27 22:27:49 +03:00
Aliaksandr Valialkin
4b3951fd86
app/vmselect/promql: add rollup_scrape_interval(m[d]) function 
It calculates the min, max and avg scrape intervals for m over the given lookbehind window d
 2021-09-27 19:21:55 +03:00
Aliaksandr Valialkin
eff31c10ec
app/vmselect/promql: follow-up after 526dd93b32 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1625
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1612
 2021-09-27 18:57:03 +03:00
Aliaksandr Valialkin
a957d420f4
docs/BestPractices.md 
docs/BestPractices.md: update the doc
 2021-09-27 17:51:57 +03:00
Aliaksandr Valialkin
1115c2f235
app/vmselect/promql: align the behavior of or, and and unless operators with on (labels) modifier to Prometheus 
Previously VictoriaMetrics could return unexpected result of the right-hand side operand
had multiple time series with the given set of labels mentioned in `on(labels)`.

See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/1643
 2021-09-24 00:56:35 +03:00
Aliaksandr Valialkin
6fbfa026ca
docs/CHANGELOG.md: cut v1.66.2 2021-09-23 22:53:27 +03:00
Aliaksandr Valialkin
d15d036a5a
lib/storage: properly handle {__name__=~"prefix(suffix1|suffix2)",other_label="..."} queries 
They were broken in the commit 00cbb099b6

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1644
 2021-09-23 21:52:31 +03:00
Aliaksandr Valialkin
f93146c81e
docs: make docs-sync 2021-09-23 20:55:29 +03:00
Aliaksandr Valialkin
7edb65a9a3
docs/CHANGELOG.md: document 0e35fc9538 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1641
 2021-09-23 20:47:56 +03:00
Aaron France
e398d83829
fix: typo in vmagent.md (#1642) 2021-09-23 20:47:56 +03:00
Aliaksandr Valialkin
d8de26bbfd
lib/promscrape: add vm_promscrape_max_scrape_size_exceeded_errors_total metric for counting of the failed scrapes due to the exceeded response size 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1639
 2021-09-23 14:48:16 +03:00
Aliaksandr Valialkin
9b76515216
docs/CaseStudies.md: add a case study for Grammarly 2021-09-23 13:12:20 +03:00
Aliaksandr Valialkin
e128915c6e
docs/Articles.md: add https://cer6erus.medium.com/superset-bi-with-victoria-metrics-a109d3e91bc6 2021-09-23 10:44:14 +03:00
Aliaksandr Valialkin
5e5d5fa8fb
docs/CaseStudies.md: fix a link to third-party articles 2021-09-22 03:53:08 +03:00
Aliaksandr Valialkin
47422e9918
docs/CHANGELOG.md: cut v1.66.1 2021-09-22 01:47:26 +03:00
Aliaksandr Valialkin
25361d176d
docs/CHANGELOG.md: document 9ca1cbced1 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1635
 2021-09-21 23:17:26 +03:00
Aliaksandr Valialkin
c3e1f87048
lib/promrelabel: fix parsing regex: true in relabeling rules 2021-09-21 23:01:40 +03:00
Aliaksandr Valialkin
3f1104ea74
app/vmselect: fix accessing /graphite/* endpoints 2021-09-21 14:06:18 +03:00
Aliaksandr Valialkin
e8700b57b9 docs/vmagent.md: typo fix 2021-09-20 16:49:32 +03:00
Aliaksandr Valialkin
233c6a8a82 docs/vmagent.md: typo fixes in Prometheus staleness markers docs 2021-09-20 16:44:42 +03:00
Aliaksandr Valialkin
d43ebb8314 docs/CHANGELOG.md: cut v1.66.0 2021-09-20 15:20:36 +03:00
n4mine
a5fcc9e356
fix: typo, dddresses -> addresses (#1630) 2021-09-20 14:27:34 +03:00
Aliaksandr Valialkin
4150772ef8 docs/Articles.md: add a link to Open-source strategy at VictoriaMetrics 2021-09-20 11:46:38 +03:00
Aliaksandr Valialkin
76c650bbda app/vmselect/promql: add quantiles_over_time("phiLabel", phi1, ..., phiN, m[d]) function for calculating multiple quantiles at once 2021-09-17 23:35:04 +03:00
Aliaksandr Valialkin
727521d20a docs/CHANGELOG.md: document 0e09fdb8b0 2021-09-17 18:47:39 +03:00
Aliaksandr Valialkin
81d24045c2 app/vmselect/promql: add histogram_quantiles("phiLabel", phi1, ..., phiN, buckets) function 
This function calculates multiple quantiles over the given buckets at once

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1573
 2021-09-17 13:39:06 +03:00
Aliaksandr Valialkin
9333c4111e docs/CHANGELOG.md: document the change in enterprise apps, which allows passing -version without -eula flag 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1621
 2021-09-17 12:38:22 +03:00
Aliaksandr Valialkin
f9910113b5 docs/FAQ.md: extend VictoriaMetrics vs TimescaleDB section with real user experience 
See also https://github.com/timescale/promscale/issues/427 , which is mentioned in the https://abiosgaming.com/press/high-cardinality-aggregations/
 2021-09-16 20:47:15 +03:00
Roman Khavronenko
cec67a3129 docs: fix indentation for FAQ document (#1620) 2021-09-16 14:01:22 +03:00
Aliaksandr Valialkin
3f8de4f82f app/vmselect/promql: add mad(q) and outliers_mad(tolerance, q) functions to MetricsQL 2021-09-16 13:35:38 +03:00
Aliaksandr Valialkin
7ad54041fe app/{vminsert,vmselect}: automatically add missing port in -storageNode lists passed to vminsert and vmselect 
This should simplify manual setup of the cluster according to https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#cluster-setup
 2021-09-15 18:04:30 +03:00
Aliaksandr Valialkin
9ebccddfc2 docs/CHANGELOG.md: document 777ff75874 2021-09-15 17:45:18 +03:00
Aliaksandr Valialkin
e67b4fca11 vendor: update github.com/valyala/histogram from v1.1.2 to v1.2.0 
This fixes the non-repeatable quantile_over_time() results when the number of input samples exceeds 1000.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1612
 2021-09-15 09:35:40 +03:00
Roman Khavronenko
45cce85bf4 docs: add "Choosing a Time Series Database for High Cardinality Aggregations" article (#1617) 2021-09-15 09:35:38 +03:00
Aliaksandr Valialkin
63fb5a5ca5 docs/vmalert.md: follow-up after 68721f6e7d 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1608
 2021-09-14 14:48:45 +03:00
Dima Lazerka
5a256970d6 Remove port 3000 and add https to play-grafana link (#1616) 
* Remove port 3000 and add https to play-grafana link

* Fix typo

Co-authored-by: Dzmitry Lazerka <dlazerka@gmail.com>
 2021-09-14 14:27:16 +03:00
Aliaksandr Valialkin
1493461244 lib/storage: follow up after 00cbb099b6 2021-09-14 14:23:02 +03:00
Aliaksandr Valialkin
5e5ce27df7 app/vmauth: do not log invalid auth tokens by default for security reasons 
The logging can be enabled by passing `-logInvalidAuthTokens` command-line flag to vmauth
 2021-09-14 12:20:17 +03:00
Aliaksandr Valialkin
d3289bf276 docs/Single-server-VictoriaMetrics.md: link to cardinality limiter docs in vmagent 2021-09-13 21:27:25 +03:00
Aliaksandr Valialkin
a8f1b614e5 docs/vmagent.md: mention out of order sample errors, which are typically emitted by Thanos, Cortex or Prometheus 2021-09-13 19:36:09 +03:00
Aliaksandr Valialkin
a1b298a842 app/vminsert/netstorage: disable rerouting by default 
Production clusters work more stable with the disabled rerouting during rolling restarts and/or
during spikes in time series churn rate. So it would be better disabling the rerouting by default.

The re-routing can be enabled by passing `-disableRerouting=false` command-line flag to `vminsert` nodes.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/791
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1054
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1165
 2021-09-13 18:50:47 +03:00
Aliaksandr Valialkin
2932f495f3 docs/CHANGELOG.md: document 5494bc02a6 2021-09-13 17:14:45 +03:00
Aliaksandr Valialkin
184e145570 docs: consistency renaming: Influx -> InfluxDB 2021-09-13 17:14:45 +03:00
Aliaksandr Valialkin
b721110902 docs/vmbackup.md: update the outdated link to vmbackupmanager 2021-09-13 14:17:53 +03:00
Aliaksandr Valialkin
19516aedb5 app/vmselect/promql: use Prometheus-compatible label value formatting for count_values function 2021-09-13 13:48:18 +03:00
Aliaksandr Valialkin
b684624f67 lib/promscrape/discovery/docker: support host networking mode 
See https://github.com/prometheus/prometheus/issues/9116
 2021-09-13 13:30:55 +03:00
Aliaksandr Valialkin
6ed9f10da5 lib/promscrape/discovery/kubernetes: properly use https scheme for wildcard TLS certificates in ingress target discovery 
See https://github.com/prometheus/prometheus/issues/8902
 2021-09-13 13:04:43 +03:00
Aliaksandr Valialkin
d90834da70 lib/promscrape: generate scrape_timeout_seconds metric per each scrape target in the same way as Prometheus 2.30 does 
See https://github.com/prometheus/prometheus/pull/9247
 2021-09-12 15:21:26 +03:00
Aliaksandr Valialkin
6c97388dde lib/promscrape: add ability to configure scrape_timeout and scrape_interval via relabeling 
See https://github.com/prometheus/prometheus/pull/8911
 2021-09-12 13:35:20 +03:00
Aliaksandr Valialkin
09670479cd lib/promscrape: reduce CPU usage for common case when calculating scrape_series_added metric 
Also reduce CPU usage when applying `series_limit` to scrape targets with constant set of metrics.

The main idea is to perform the calculations on scrape_series_added and series_limit
only if the set of metrics exposed by the target has been changed.
Scrape targets rarely change the set of exposed metrics,
so this optimization should reduce CPU usage in general case.
 2021-09-12 12:53:45 +03:00
Aliaksandr Valialkin
2b64c1c95e docs/Single-server-VictoriaMetrics.md: refer to relabeling section for vmagent 
This removes duplicate docs about additional relabeling actions supported by VictoriaMetrics components
 2021-09-12 11:40:23 +03:00
Aliaksandr Valialkin
def0430318 docs/vmagent.md: sync with app/vmagent/README.md by running make docs-sync 2021-09-11 11:05:12 +03:00
Aliaksandr Valialkin
5aaaa686a4 lib/promscrape: send stale markers for disappeared metrics like Prometheus does 2021-09-11 11:02:56 +03:00
Aliaksandr Valialkin
e1632dc0df docs/vmalert.md: typo fix in Multitenancy chapter 2021-09-10 17:57:43 +03:00
Aliaksandr Valialkin
c2f37f049b lib/storage: properly search series by multiple tag filters matching empty labels such as foo{bar=~"baz|",x=~"y|"} 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1601
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/395
 2021-09-09 21:12:53 +03:00
Aliaksandr Valialkin
9b07cb2988 app/vmselect/promql: keep metric name in rollup_candlestick results, since they don't change the original series meaning 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1600
 2021-09-09 19:24:10 +03:00
Aliaksandr Valialkin
09edcfd21c docs/CHANGELOG.md: typo fix 2021-09-09 18:58:03 +03:00
Aliaksandr Valialkin
d613a018c8 lib/promscrape: add ability to set series_limit and stream_parse options via relabeling 
This allows managing these options on a per-target basis.

Typical use case: to manage these options for pods via Kubernetes annotations.
 2021-09-09 18:51:23 +03:00
Aliaksandr Valialkin
986807ebd9 docs/vmagent.md: typo fix in Relabeling chapter 2021-09-09 16:39:34 +03:00
Aliaksandr Valialkin
75c3514c5c lib/promrelabel: add keep_metrics and drop_metrics actions to relabeling rules 
These actions simlify metrics filtering. For example,

- action: keep_metrics
  regex: 'foo|bar|baz'

would leave only metrics with `foo`, `bar` and `baz` names, while the rest of metrics will be deleted.

The commit also makes possible to split long regexps into multiple lines. For example, the following config is equivalent to the config above:

- action: keep_metrics
  regex:
  - foo
  - bar
  - baz
 2021-09-09 16:25:09 +03:00
Aliaksandr Valialkin
708ebd59ad docs/vmalert.md: make docs-sync after 21f022e5f0 2021-09-09 16:25:08 +03:00
Aliaksandr Valialkin
2b19dd8601 docs/MetricsQL.md: add a link to VictoriaMetrics github 2021-09-08 00:15:34 +03:00
Aliaksandr Valialkin
608ed19655 app/vmalert: follow-up after 21f022e5f0 2021-09-07 22:45:47 +03:00
Aliaksandr Valialkin
9be7294331 docs/CHANGELOG.md: document 42e07cfaea 2021-09-07 22:32:01 +03:00
Aliaksandr Valialkin
304661c547 docs/vmagent.md: add API path for Prometheus text exposition format 2021-09-07 16:15:28 +03:00
Aliaksandr Valialkin
c5e030dfc4 docs/Articles.md: add a link to case studies 2021-09-06 10:58:48 +03:00
Aliaksandr Valialkin
52b44945a0 docs/Articles.md: add an url to https://www.vultr.com/docs/install-and-configure-victoriametrics-on-debian 2021-09-03 11:50:23 +03:00
Aliaksandr Valialkin
9f816dfee8 docs/Single-server-VictoriaMetrics.md: fix a link multitenancy docs for cluster version in VictoriaMetrics 2021-09-02 17:44:22 +03:00
Aliaksandr Valialkin
a88b02287d docs/CHANGELOG.md: cut v1.65.0 2021-09-01 17:12:44 +03:00
Aliaksandr Valialkin
41af6a73fe docs/FAQ.md: add questions on how to migrate data from various systems (Prometheus, InfluxDB, OpenTSDB, Graphite) to VictoriaMetrics 2021-09-01 16:48:05 +03:00
Aliaksandr Valialkin
5c58c8d55e docs: update -help output for victoria-metrics and vmagent after f77dde837a 2021-09-01 16:35:18 +03:00
Aliaksandr Valialkin
c4df601f43 lib/promscrape: add the ability to limit the number of unique series per each scrape target 
The number of series per target can be limited with the following options:

* Global limit with `-promscrape.maxSeriesPerTarget` command-line option.
* Per-target limit with `max_series: N` option in `scrape_config` section.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1561
 2021-09-01 16:08:12 +03:00
Aliaksandr Valialkin
835efb111e docs: make docs-sync 2021-09-01 12:25:17 +03:00
Aliaksandr Valialkin
816d4b6095 docs/CHANGELOG.md: document eff940aa76 2021-09-01 12:25:11 +03:00
Aliaksandr Valialkin
22bb18a449 docs/CHANGELOG.md: document 7c70dcbe3b 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1471
 2021-09-01 12:25:04 +03:00
Nikolay
210c76b51e adds external_labels per group for vmalert (#1485) 
* adds external_label per group for vmalert
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1471
 2021-09-01 12:19:49 +03:00
Aliaksandr Valialkin
9723ac30dd docs/Single-server-VictoriaMetrics.md: remove outdated link to VictoriaMetrics wiki 
VictoriaMetrics wiki became outdated after publishing all the docs at https://docs.victoriametrics.com
 2021-08-31 11:53:38 +03:00
Aliaksandr Valialkin
b3d240f7b4 docs/CHANGELOG.md: add a link to Prometheus staleness tracking 2021-08-31 11:53:38 +03:00
Aliaksandr Valialkin
146c14d879 lib/promscrape/discovery/kubernetes: return back support role: endpointslices, since it is used by VictoriaMetrics operator 
This is a follow up commit after 31b42b30b6
 2021-08-29 12:37:36 +03:00
Aliaksandr Valialkin
18d7adf731 lib/protoparser/opentsdb: follow-up after 8ee75ca45a 2021-08-29 11:50:01 +03:00
Aliaksandr Valialkin
ca61d7c82b lib/promscrape/discovery/kubernetes: rename role: endpointslices to role: endpointslice to be consistent with Prometheus 
See 2ec6c7dbb8/discovery/kubernetes/kubernetes.go (L99)
 2021-08-29 11:23:59 +03:00
Aliaksandr Valialkin
327034b54f lib/promscrape/discovery/kubernetes: use v1 API instead of v1beta1 API for role: ingress and role: endpointslices 
This should fix service discovery for these roles in Kubernetes v1.22 and newer versions.
See https://kubernetes.io/docs/reference/using-api/deprecation-guide/#ingress-v122

The corresponding change in Prometheus - https://github.com/prometheus/prometheus/pull/9205
 2021-08-29 11:23:58 +03:00
Aliaksandr Valialkin
0e809bd4b6 docs/Single-server-VictoriaMetrics.md: mention that downsampling doesnt improve query performance on high churn rate 2021-08-27 18:51:28 +03:00
Aliaksandr Valialkin
39bb6bdd79 app/vmselect/promql: add quantile("phiLabel", phi1, ..., phiN, q) aggregate function to MetricsQL 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1573
 2021-08-27 18:40:25 +03:00
Aliaksandr Valialkin
14d4b5aa83 app/vmselect: add -search.disableAutoCacheReset command-line option for disabling automatic cache reset when a sample with old timestamp outside -search.cacheTimestampOffset is inserted 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1570
 2021-08-27 17:17:43 +03:00
Aliaksandr Valialkin
6e5864b014 docs/{vmgateway,vmbackupmanager}: mention that enterprise binaries are free for download and evaluation 2021-08-27 14:53:48 +03:00
Aliaksandr Valialkin
adf71a415e docs: link to active time series, churn rate and high cardinality questions 2021-08-27 14:45:36 +03:00
Aliaksandr Valialkin
1a7646c142 docs/CHANGELOG.md: document the bugfix for possible timeout error in vmbackupmanager when making snapshots 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1571
 2021-08-27 13:04:33 +03:00
Aliaksandr Valialkin
125a94e6ca docs: mention that enterprise binaries can be downloaded and evaluated for free 2021-08-27 12:48:51 +03:00
Aliaksandr Valialkin
102ab795f8 docs/vmagent.md: document the ability to load scrape configs from multiple files 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1559
 2021-08-26 09:13:54 +03:00
Aliaksandr Valialkin
7fdb4db73d lib/promscrape: add ability to load scrape configs from multiple files 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1559
 2021-08-26 08:51:53 +03:00
Aliaksandr Valialkin
8d843d0754 docs/CHANGELOG.md: document 48f33d098b 2021-08-25 13:32:05 +03:00
Aliaksandr Valialkin
874660a4ae docs/Cluster-VictoriaMetrics.md: mention that the -replicationFactor at vmselect is an optional parameter 2021-08-25 13:10:31 +03:00
Aliaksandr Valialkin
4a2d7aec7f lib/promscrape: expose promscrape_discovery_http_errors_total metric for tracking errors per each http_sd config 2021-08-25 13:05:29 +03:00
Aliaksandr Valialkin
b885bd9b7d lib/{mergeset,storage}: improve the detection of the needed free space for background merge 
This should prevent from possible out of disk space crashes during big merges.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1560
 2021-08-25 10:01:09 +03:00
Aliaksandr Valialkin
ae8ec78c63 docs/FAQ.md: add more entries for frequently asked questions 
The following topics are covered:

* Active time series
* High cardinality
* High churn rate
* Slow inserts
 2021-08-24 11:34:31 +03:00
Aliaksandr Valialkin
9b39e078c0 docs/MetricsQL.md: typo fix: histogram_qunatile -> histogram_quantile 2021-08-23 23:08:28 +03:00
Aliaksandr Valialkin
3eca49c4a6 docs/MetricsQL.md: fix the indentation for median function 2021-08-23 12:04:43 +03:00
Aliaksandr Valialkin
a4948d92b5 docs/MetricsQL.md: typo fix: convesions->conversions 2021-08-23 12:01:34 +03:00
Aliaksandr Valialkin
8b9dc45c3c docs/MetricsQL.md: typo fixes 2021-08-23 12:00:17 +03:00
Aliaksandr Valialkin
5917c72ddd docs/MetricsQL.md: rehaul the documentation on MetricsQL 
* Document all the functions supported by MetricsQL, including PromQL functions
* Group functions by their type: rollup functions, transform functions, label manipulation functions and aggregate functions.
* Document implicit query transformations.
 2021-08-23 11:46:30 +03:00
Aliaksandr Valialkin
40b06e84f8 app/vmselect/promql: rename sign() function to sgn() in order to be consistent with Prometheus 
See https://github.com/prometheus/prometheus/pull/8457 for details.
 2021-08-23 11:46:29 +03:00
Aliaksandr Valialkin
8493159eed deployment/docker: update Go builder from Go1.16.7 to Go1.17.0 
This improves data ingestion and query performance by up to 5% according to benchmarks.

See https://go.dev/blog/go1.17
 2021-08-21 22:22:31 +03:00
Aliaksandr Valialkin
67bc407747 lib/promscrape: reduce memory and CPU usage when Prometheus staleness tracking is enabled for metrics from deleted / disappeared scrape targets 
Store the scraped response body instead of storing the parsed and relabeld metrics.
This should reduce memory usage, since the response body takes less memory than the parsed and relabeled metrics.
This is especially true for Kubernetes service discovery, which adds many long labels for all the scraped metrics.

This should also reduce CPU usage, since the marshaling of the parsed
and relabeld metrics has been substituted by response body copying.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526
 2021-08-21 21:24:07 +03:00
Aliaksandr Valialkin
ff4c7c1a3d docs/vmalert.md: run make docs-sync after 9ee3d0378f 2021-08-21 20:25:26 +03:00
Aliaksandr Valialkin
388e07b37f docs/CHANGELOG.md: document 9ee3d0378f 2021-08-21 20:23:22 +03:00
Aliaksandr Valialkin
3454f25e0f docs/CHANGELOG.md: document b5d6a0e499 2021-08-19 14:07:00 +03:00
Aliaksandr Valialkin
2e1eb33bfd docs/CHANGELOG.md: clarify the change, which adds -search.noStaleMarkers command-line flag 2021-08-19 13:55:43 +03:00
Aliaksandr Valialkin
ee1f3414d1 app/vmselect/promql: do not override [d] at last_over_time(m[d]) if [d] is smaller than scrape_interval 
Since most users do not expect the overriding of explicitly set `[d]`.
 2021-08-19 10:33:10 +03:00
Aliaksandr Valialkin
47a37b8cfc docs/CHANGELOG.md: cut v1.64.1 
This is mostly bugfix release, which includes fixes for staleness handling and a security update for Alpine base image
 2021-08-18 22:07:05 +03:00
Aliaksandr Valialkin
8ee575dee9 lib/promscrape: send stale markers for the previously scraped metrics on failed scrapes like Prometheus does 2021-08-18 22:00:46 +03:00
Aliaksandr Valialkin
355690a719 docs/CHANGELOG.md: document 06bf21c21b 2021-08-18 14:01:42 +03:00
Aliaksandr Valialkin
5d92fafc40 app/vmselect: add -search.noStaleMarkers command-line flag for disabling stale markers handling in queries 
This option allows reducing CPU usage a bit when VictoriaMetrics is used
for collecting and processing non-Prometheus data. For example, InfluxDB line protocol, Graphite, OpenTSDB, CSV, etc.
 2021-08-18 13:58:06 +03:00
Aliaksandr Valialkin
f21fad53b4 lib/promscrape: add ability to disable sending Prometheus staleness markers with -promscrape.disableStaleMarkers command-line flag 
This option can be useful when vmagent consumes too much additional memory
for staleness markers functionality and when staleness markers aren't needed.
 2021-08-18 13:58:05 +03:00
Aliaksandr Valialkin
49886ecbc8 app/vmselect/promql: add bitmap_and(), bitmap_or() and bitmap_xor() functions to MetricsQL 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1541
 2021-08-17 13:22:15 +03:00
Aliaksandr Valialkin
130a5cab7e docs/Single-server-VictoriaMetrics.md: mention that vmctl can migrate data from OpenTSDB to VictoriaMetrics 2021-08-17 11:17:38 +03:00
Aliaksandr Valialkin
db34c40aec lib/promscrape: stop scrapers for the removed targets before starting scrapers for the added targets 
This should prevent from possible time series overlap when old target is substituted by new target (for example, during Kubernetes deployments).

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1530
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1509
 2021-08-17 01:00:40 +03:00
Aliaksandr Valialkin
fe8c462044 app/vmalert: mention -remoteWrite.disablePathAppend in the description for -remoteWrite.url 2021-08-16 15:23:39 +03:00
Aliaksandr Valialkin
21974cb571 app/vmalert: follow-up for 2400f85761 2021-08-16 15:20:35 +03:00
Aliaksandr Valialkin
a47c1a734a docs/CHANGELOG.md: update urls to Prometheus 2.29 release 
Previously these urls were pointing to rc0 release
 2021-08-16 14:58:05 +03:00
Aliaksandr Valialkin
f05e827757 docs/CHANGELOG.md: typo fix: satureated -> saturated 2021-08-16 14:58:05 +03:00
Alexander Rickardsson
d27dc3721b vmalert: enable configuring explicit path (#1536) 
* vmalert: allow to disable automatically added path to remote write address via disablePathAppend flag
* docs: update docs to include remoteWrite.disablePathAppend
 2021-08-16 14:58:05 +03:00
Aliaksandr Valialkin
75a0345215 docs/CHANGELOG.md: cut v1.64.0 2021-08-15 23:52:24 +03:00
Aliaksandr Valialkin
5f13c519ee lib/promscrape: restore red highlighting for DOWN targets at /targets page 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1461
 2021-08-15 16:04:33 +03:00
Aliaksandr Valialkin
175727dace docs/CHANGELOG.md: mention the bugfix when more than 27 time series are selected at /vmui 2021-08-15 15:11:13 +03:00
Aliaksandr Valialkin
b09b035d3e docs/CHANGELOG.md: mention that VMUI automatically fills Server URL field 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1506
 2021-08-15 14:45:59 +03:00
Aliaksandr Valialkin
48920bdef8 app/vmagent/remotewrite: expose vmagent_remotewrite_send_duration_seconds_total metric 
This metric can be used for determining high saturation of every connection to remote storage with
an alerting query `rate(vmagent_remotewrite_send_duration_seconds_total) > 0.9s`.
This query triggers when a connection is satureated by more than 90%
 2021-08-15 13:34:07 +03:00
Aliaksandr Valialkin
c1f81f08d4 all: add support for Prometheus staleness markers 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1526
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/748
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1509
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1530
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/845
 2021-08-13 12:13:15 +03:00
Aliaksandr Valialkin
7feb62eea9 docs/Cluster-VictoriaMetrics.md: meniton that vmagent can be used for replicating the data among multiple clusters 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1491
 2021-08-12 12:47:08 +03:00
Denys Holius
1491767a9e added guide for HA monitoring setup in K8s via VM Cluster (#1523) 
* added guide for HA monitoring setup in K8s via VM Cluster

* fixed missed divs

* fixed different typos

* Update docs/guides/k8s-ha-monitoring-via-vm-cluster.md

Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>

* Update docs/guides/k8s-ha-monitoring-via-vm-cluster.md

Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>

* Update docs/guides/k8s-ha-monitoring-via-vm-cluster.md

Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>

* Update docs/guides/k8s-ha-monitoring-via-vm-cluster.md

Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>

* Update docs/guides/k8s-ha-monitoring-via-vm-cluster.md

Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>

* Update docs/guides/k8s-ha-monitoring-via-vm-cluster.md

Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>

* Update docs/guides/k8s-ha-monitoring-via-vm-cluster.md

Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>

* Update docs/guides/k8s-ha-monitoring-via-vm-cluster.md

Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>

* Update docs/guides/k8s-ha-monitoring-via-vm-cluster.md

Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>

* Update docs/guides/k8s-ha-monitoring-via-vm-cluster.md

Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>

* Update docs/guides/k8s-ha-monitoring-via-vm-cluster.md

Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>

Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
 2021-08-11 13:42:53 +03:00
Aliaksandr Valialkin
9eb828b2c2 app/vminsert: add vm_rpc_send_duration_seconds_total metric per each vminsert->vmstorage link 
This metric is useful for determining high link saturation with the following alerting rule:

rate(vm_rpc_send_duration_seconds_total) > 0.9s
 2021-08-11 11:42:33 +03:00
Aliaksandr Valialkin
3df6550153 docs/Cluster-VictoriaMetrics.md: update -help output for cluster components after the d375d9b878 2021-08-11 11:39:09 +03:00
Aliaksandr Valialkin
518d0aba18 docs: make docs-sync after e0ee69797d 2021-08-11 10:54:28 +03:00
Aliaksandr Valialkin
90efb5831b lib/envflag: add a link to docs for -envflag.enable 2021-08-11 10:32:40 +03:00
Aliaksandr Valialkin
d5ca07bd71 docs/CHANGELOG.md: document new metrics added to vmalert at 7416fdaa8b 2021-08-05 10:12:53 +03:00
Aliaksandr Valialkin
13d438d808 app/vmagent: typo fix in the description for -remoteWrite.queues 2021-08-05 10:00:58 +03:00
Aliaksandr Valialkin
b877538622 app/vmagent: follow-up after fe445f753b 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1491
 2021-08-05 09:51:00 +03:00
Aliaksandr Valialkin
44bf4cabea docs/Articles.md: actualize links and re-order some links 2021-08-03 16:11:30 +03:00
Aliaksandr Valialkin
77bb9e1656 lib/promscrape/discovery/gce: add __meta_gce_interface_ipv4_<name> labels as in Prometheus 2.29 
See https://github.com/prometheus/prometheus/pull/8978
 2021-08-03 15:51:45 +03:00
Aliaksandr Valialkin
336a2aa2e0 lib/promscrape/discovery/ec2: add __meta_ec2_availability_zone_id label as Prometheus 2.29 does 2021-08-03 13:28:13 +03:00
Aliaksandr Valialkin
60cfa5f100 app/vmselect/promql: add present_over_time(m[d]) function, which will be available starting from Prometheus 2.29.0 
See https://github.com/prometheus/prometheus/releases/tag/v2.29.0-rc.0 and https://github.com/prometheus/prometheus/pull/9097
 2021-08-03 12:21:53 +03:00
Roman Khavronenko
51329d9e1e docs: add "Scaling to trillions of metric data points" to articles (#1517) 2021-08-03 11:14:57 +03:00
Roman Khavronenko
e127dcf013 docs: mention "Push Prometheus metrics to VictoriaMetrics or other exporters" article (#1511) 2021-08-02 17:23:45 +03:00
Roman Khavronenko
8e15c96004 docs: fix indentation for guide articles (#1512) 2021-08-02 17:17:24 +03:00
Aliaksandr Valialkin
d9810a7403 docs/CaseStudies.md: typo fix: hed->had 2021-07-30 18:49:17 +03:00
Aliaksandr Valialkin
c2df339eb2 docs/CHANGELOG.md: typo fix 2021-07-30 12:35:28 +03:00
Aliaksandr Valialkin
3389f1e474 docs/CHANGELOG.md: document d05cac6c98 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1486
 2021-07-30 12:26:34 +03:00
Aliaksandr Valialkin
99004a6a40 app/vmselect/netstorage: unpack time series data in mostly local big chunks 
This should improve performance on multi-CPU systems for queries selecting time series with big number of raw samples
 2021-07-30 12:26:33 +03:00
Aliaksandr Valialkin
b709fa387a app/vmselect: follow-up for ed95bc9531 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1493
 2021-07-29 09:48:47 +03:00
Aliaksandr Valialkin
c582d95cf0 docs: remove SampleSizeCalculations.md, since it is outdated and no longer used 
There was a reference to this doc from the old victoriametrics.com site
 2021-07-28 19:25:45 +03:00
Denys Holius
113df2f3b7 Added new guide for monitoring k8s via VictoriaMetrics cluster (#1476) 
* renamed and moved screenshots

* fixed cluster guide, updated helm chart versions, added values.yaml for vm single

* renamed guide files

* fixed typo

* add some fixes

* fixed typos,added guide k8s-monitoring-via-vm-cluster

* added fixes for yamls
 2021-07-28 18:06:50 +03:00
Aliaksandr Valialkin
49bf3abf67 app/vmselect: follow-up for 626073bca8 
* Rename -search.maxMetricsPointSearch to -search.maxSamplesPerQuery, so it is more consistent with the existing -search.maxSamplesPerSeries
* Move the -search.maxSamplesPerQuery from vmstorage to vmselect, so it could effectively limit the number of raw samples obtained from all the vmstorage nodes
* Document the -search.maxSamplesPerQuery in docs/CHANGELOG.md
 2021-07-28 18:00:04 +03:00
Aliaksandr Valialkin
857fa4e28a docs/CHANGELOG.md: document 9d45b46f4c 2021-07-27 12:39:10 +03:00
Aaron France
3c22b5c41e fix: typo in metrics.md docs 2021-07-27 10:53:02 +03:00
Aliaksandr Valialkin
3921d8afae app/vmselect: prevent from possible deadlock when f callback blocks inside RunParallel 2021-07-26 15:50:45 +03:00
Aliaksandr Valialkin
349d45bbbe docs/Articles.md: add links to new articles 2021-07-23 21:09:00 +03:00
Aliaksandr Valialkin
b047feeb8b app/vmselect/promql: properly handle (a op b) default N if (a op b) returns NaN series 
The result should be a series with `N` values and `a op b` labels. Previously such series has been removed from the result.
 2021-07-16 01:44:24 +03:00
Aliaksandr Valialkin
df117f85bd app/vmselect: do not track queries with less than 1ms execution time at /api/v1/status/top_queries 
This should improve the readability and usefullness of the /api/v1/status/top_queries when debugging slow queries
or queries that take too much cpu time.
 2021-07-15 16:53:35 +03:00
Aliaksandr Valialkin
5830ce2706 app/vmselect/netstorage: add -search.maxSamplesPerSeries command-line option for limiting the number of samples a query can process per each series 
This should prevent from out of memory crashes like in https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1067
 2021-07-15 16:53:35 +03:00
Aliaksandr Valialkin
f847677513 docs/CHANGELOG.md: typo fix: suffxies->suffixes 2021-07-15 15:02:55 +03:00
Aliaksandr Valialkin
4ec57102e6 docs/CHANGELOG.md: cut v1.63.0 2021-07-15 14:02:45 +03:00
Aliaksandr Valialkin
e992754e79 lib/storage: remove cache directory if it contains reset_cache_on_startup file 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1447
 2021-07-13 17:59:51 +03:00
Aliaksandr Valialkin
ee7e9795ec docs: clarify why spare CPU and RAM resources are needed in capacity planning 2021-07-13 15:49:38 +03:00
Aliaksandr Valialkin
03f1c0b58a docs/Cluster-VictoriaMetrics.md: clarify the docs about the needed values for -dedup.minScrapeInterval at vmselect during replication when the data is pushed from HA pair 2021-07-13 15:29:43 +03:00
Aliaksandr Valialkin
e6edb85fa2 lib/httpserver: add is_set label to flag metrics 
This label allows determining the set flags with the query `flag{is_set="true"}`
 2021-07-13 15:10:18 +03:00
Aliaksandr Valialkin
9add9d86a6 app/vmselect/promql: duration handling improvements in MetricsQL queries 
- Support durations anywhere in MetricsQL queries. E.g. sum_over_time(m[1h])/1h is equivalent to sum_over_time(m[1h])/3600
- Support durations without suffix. E.g. rate(m[300]) is equivalent to rate(m[5m])
 2021-07-12 17:19:32 +03:00
Aliaksandr Valialkin
390a31ccfa docs/CHANGELOG.md: document the change from bfba4c28a4 2021-07-12 12:42:42 +03:00
Aliaksandr Valialkin
253fe7699c docs: update http://slack.victoriametrics.com to https://slack.victoriametrics.com 2021-07-12 10:58:19 +03:00
Aliaksandr Valialkin
0b98f6c7ff app/vmselect: expose vmui at /vmselect/<accountID>/vmui/ instead of /vmselect/<accountID>/prometheus/vmui/ 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413
 2021-07-10 12:32:21 +03:00
Aliaksandr Valialkin
98e049ba6d app/vmui: move source code from https://github.com/VictoriaMetrics/vmui to app/vmui 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413
 2021-07-09 17:13:51 +03:00
Aliaksandr Valialkin
9900a1f563 docs: clarify what does "workload" mean in capacity planning docs 2021-07-09 12:50:51 +03:00
Aliaksandr Valialkin
2c5e1cd893 app/vmselect: move web ui from /ui to /select/<accountID>/prometheus/ui 
This way the UI is available for every tenant (aka accountID) and the UI can automatically determine the needed per-tenant datasource path from page referer.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413
 2021-07-08 13:14:50 +03:00
Aliaksandr Valialkin
acb7a95c64 app/vmselect: follow-up after aa11ef6d3b 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1413
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4
 2021-07-07 17:45:09 +03:00
Aliaksandr Valialkin
6f8dd85deb docs/Cluster-VictoriaMetrics.md: improve capacity planning recommendations 2021-07-07 16:21:59 +03:00
Aliaksandr Valialkin
41754e12f8 lib/mergeset: cache indexBlock items only on the second request 
This should reduce the indexdb/indexBlocks cache size, since it won't contain one-time-wonders items.
 2021-07-07 15:24:37 +03:00
Aliaksandr Valialkin
9c19719ad6 app/{vminsert,vmselect}: export vminsert_request_duration_seconds and vmselect_request_duration_seconds histograms 2021-07-07 13:27:23 +03:00
Aliaksandr Valialkin
ceda2b1df4 lib/httpserver: print full requestURI in httpserver.Errorf 
This should simplify debugging.
 2021-07-07 13:11:29 +03:00
Aliaksandr Valialkin
4e87638877 docs: clarify capacity planning docs 2021-07-07 12:48:07 +03:00
Roman Khavronenko
0541e3108a add option to add Copy button for code snippets (#1433) 
To add a Copy button wrap code snippet with the following element:
```
<div class="with-copy" markdown="1">

<your-code-snippet>

</div>
```

See the changes to `Kubernetes monitoring with VictoriaMetrics Single` for details.
 2021-07-06 10:54:37 +03:00
Roman Khavronenko
e2b5f93170 move github-pages docs to the main repo (#1432) 
* move github-pages docs to the main repo

* rm github actions for copying docs to VictoriaMetrics/VictoriaMetrics.github.io
 2021-07-05 15:20:29 +03:00
Aliaksandr Valialkin
4b25e627f8 lib/workingsetcache: properly switch to whole mode 
Previously the switch from `split` to `whole` mode had been performed too early,
e.g. when the current cache size became bigger than 1/4 of the allowed cache size.

Now it is performed when the current cache size becomes bigger than 1/2 of the allowed cache size.

This change can reduce memory usage for data ingestion path when big number of active time series are ingested.
 2021-07-05 15:15:39 +03:00
Aliaksandr Valialkin
28a778dc9f docs/CHANGELOG.md: document the bugfix for vm_merge_need_free_disk_space metric at 9a83e9018d 2021-07-05 12:01:24 +03:00
Aliaksandr Valialkin
2386829ad6 docs/Articles.md: add an url to https://medium.com/ibm-garage/monitoring-of-multiple-openshift-clusters-with-victoriametrics-d4f0979e2544 2021-07-05 11:52:18 +03:00
Aliaksandr Valialkin
44855f0c9b app/{vmselect,vmstorage}: clarify the description for -dedup.minScrapeInterval command-line flag 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1426
 2021-07-02 15:06:41 +03:00
Aliaksandr Valialkin
4b4e69791f docs/Cluster-VictoriaMetrics.md: typo fix: siplify -> simplify 2021-07-02 10:50:26 +03:00
Aliaksandr Valialkin
9ec08213be docs/Cluster-VictoriaMetrics.md: add a chapter describing a toy cluster setup on a single host 
While at it, refer to available tools, which can simplify cluster setup
 2021-07-02 10:48:13 +03:00
Aliaksandr Valialkin
1c12c0f79c lib/promauth: reload TLS certificates from disk on every mTLS connection as Prometheus does 
This allows updating client certificates without the need to restart vmagent and/or single-node VictoriaMetrics.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1420
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/470
 2021-07-01 15:43:43 +03:00
Aliaksandr Valialkin
c1b15c7764 docs/CHANGELOG.md: document ae485c2bfd 2021-07-01 11:52:47 +03:00
Aliaksandr Valialkin
71c856beb8 lib/workingsetcache: reset the cache mode when the cache is reset 
This should reduce memory usage if the working set is reduced after the cache reset.
 2021-07-01 11:52:47 +03:00
Aliaksandr Valialkin
0c4c630839 lib/promscrape: typo fix in /targets output 
The typo has been introduced in fb72a2133f

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1408
 2021-06-28 21:27:22 +03:00
Aliaksandr Valialkin
4d8ab5d9fa docs/vmagent.md: mention about docker_sd_config support 2021-06-25 20:53:09 +03:00
Aliaksandr Valialkin
db66ea9a5f docs/CHANGELOG.md: cut v1.62.0 2021-06-25 13:29:48 +03:00
Aliaksandr Valialkin
97d1ccfc8e lib/promscrape: split docker and dockerswarm service discovery code bases, since they have very little in common 
This is a follow up after c85a5b7fcb
 2021-06-25 13:22:16 +03:00
Aliaksandr Valialkin
01af676436 docs: consistently put the link to articles and slides about VictoriaMetrics after the links to case studies 2021-06-24 15:38:49 +03:00