Commit graph

3387 commits

Author SHA1 Message Date
Roman Khavronenko
303d3616ec
vmalert: revert unittest feature (#4734)
* Revert "vmalert: unittest support stale datapoint (#4696)"

This reverts commit 0b44df7ec8.

* Revert "docs: specify min version and limitations for vmalert's unit tests"

This reverts commit a24541bd

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Revert "vmalert: init unit test (#4596)"

This reverts commit da60a68d

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* docs: mention unittest revert in changelog

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 9f1b9b86cc)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-28 11:09:52 +02:00
Aliaksandr Valialkin
63d9a92d3d
docs/vmalert.md: fix broken links to Web chapter 2023-07-27 18:05:25 -07:00
Aliaksandr Valialkin
174f6793f3
docs/CHANGELOG.md: delimit changes from update notes 2023-07-27 17:06:37 -07:00
Aliaksandr Valialkin
fd4b10330a
docs/CHANGELOG.md: cut v1.92.0 2023-07-27 14:55:52 -07:00
Aliaksandr Valialkin
1f30f53df2
lib/promscrape/discovery: close unused HTTP connections to service discovery servers
This should prevent from connection leaks

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4724
2023-07-27 14:47:55 -07:00
Nikolay
85de94e85c
lib/protoparser: adds opentelemetry parser (#2570)
* lib/protoparser: adds opentelemetry parser
app/{vmagent,vminsert}: adds opentelemetry ingestion path

Adds ability to ingest data with opentelemetry protocol
protobuf and json encoding is supported
data converted into prometheus protobuf timeseries
each data type has own converter and it may produce multiple timeseries
from single datapoint (for summary and histogram).
only cumulative aggregationFamily is supported for sum(prometheus
counter) and histogram.

Apply suggestions from code review

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

updates deps

fixes tests

wip

wip

wip

wip

lib/protoparser/opentelemetry: moves to vtprotobuf generator

go mod vendor

lib/protoparse/opentelemetry: reduce memory allocations

* wip

- Remove support for JSON parsing, since it is too fragile and is rarely used in practice.
  The most clients send OpenTelemetry metrics in protobuf.
  The JSON parser can be added in the future if needed.
- Remove unused code from lib/protoparser/opentelemetry/pb and lib/protoparser/opentelemetry/proto
- Do not re-use protobuf message between ParseStream() calls, since there is high chance
  of high fragmentation of the re-used message because of too complex nested structure of the message.

* wip

* wip

* wip

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-07-27 13:37:15 -07:00
Aliaksandr Valialkin
b3e93afd17
docs/CHANGELOG.md: add a link to Pushgateway protocol in the bugfix description for 74237ce5c0
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4692
2023-07-27 13:11:25 -07:00
Alexander Marshalov
74237ce5c0
fixed label values decoding for pushgateway compatibility (#4727)
Fixed decoding of label values with slash for pushgateway and prometheus golang client compatibility + added some tests. (#4962)
2023-07-27 13:03:48 -07:00
Haleygo
3c297e0253
vmalert: add keep_firing_for field for alerting rule (#4669)
vmalert: support `keep_firing_for` field for alerting rule

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4529

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2023-07-27 13:00:45 -07:00
hagen1778
1250ebc063
vmalert: clarify docs for state restore with additional details
The important change is to highlight that restore procedure happens
only once and only for already loaded rules. Config hot-reload
doesn't trigger the restore procedure.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-27 12:58:56 -07:00
Aliaksandr Valialkin
1b7d97787a
docs: use 1. instead of N. in numbered bullets, so they are automatically adjusted by Github Markdown engine
See https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#lists
2023-07-26 14:40:06 -07:00
Aliaksandr Valialkin
98676a5223
docs/Cluster-VictoriaMetrics.md: use 1. instead of N. in numbered bullets, so they are automatically adjusted by Github Markdown engine
See https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#lists
2023-07-26 14:13:24 -07:00
Aliaksandr Valialkin
da61f5dd34
docs/keyConcepts.md: use 1. instead of N. in numbered bullets, so they are automatically adjusted by Github Markdown engine
See https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#lists
2023-07-26 14:12:46 -07:00
Aliaksandr Valialkin
5466183fd9
docs/Single-server-VictoriaMetrics.md: use 1. instead of N. in numbered bullets, so they are automatically adjusted by Github Markdown engine
See https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#lists
2023-07-26 14:12:45 -07:00
Aliaksandr Valialkin
64ef01fccf
docs/Release-Guide.md: use 1. instead of N. in numbered bullets, so they are automatically adjusted by Github Markdown engine
See https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#lists
2023-07-26 14:12:32 -07:00
Aliaksandr Valialkin
3ef08d16fc
docs/Troubleshooting.md: use 1. instead of N. in numbered bullets, so they are automatically adjusted by Github Markdown engine
See https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#lists
2023-07-26 14:12:32 -07:00
hagen1778
290443f7d1
docs: add bullet for gaps in timeseries to Troubleshooting
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3878
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4697
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3849
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3451
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-26 14:12:32 -07:00
Aliaksandr Valialkin
fa295c7daa
app/vmagent: add ability to shard outgoing data among multiple remote storage systems
Add -remoteWrite.shardByURL command-line flag, which instructs vmagent to spread evenly
outgoing time series data among the configured remote storage systems specified via -remoteWrite.url .

Samples for the same time series go to the same -remoteWrite.url . This allows building horizontally
scalable stream aggregation when samples for counter and histogram series must be aggregated
by the same second-level vmagent instance.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4637
2023-07-24 18:18:04 -07:00
Aliaksandr Valialkin
6e43664e24
lib/promrelabel: add support for a list of series selectors at IfExpression
This makes possible specifying a list of series selectors at the following places:

- Inside `if` option at relabeling rules
- Inside `match` option at stream aggregation rules

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4635
2023-07-24 17:09:59 -07:00
Aliaksandr Valialkin
c049778ad1
lib/streamaggr: follow-up for 736197179e
- Use a byte slice instead of a map for tracking indexes for matching series.
  This improves performance, since access by slice index is faster than access by map key.
- Re-use the byte slice for tracking indexes for matching series.
  This removes unnecessary memory allocations and improves stream aggregation performance a bit.
- Add an ability to return to the previous behvaiour by specifying -remoteWrite.streamAggr.dropInput command-line flag.
  In this case all the input samples are dropped when stream aggregation is enabled.
- Backport the new stream aggregation behaviour from vmagent to single-node VictoriaMetrics when -streamAggr.config
  option is set.
- Improve docs regarding this change at docs/CHANGELOG.md
- Document the new behavior at docs/stream-aggregation.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4243
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4575
2023-07-24 17:06:09 -07:00
Zakhar Bessarab
470afac5ff
{lib/streamaggr,vmagent/remotewrite}: breaking change for keepInput flag (#4575)
* {lib/streamaggr,vmagent/remotewrite}: breaking change for keepInput flag

Changes default behaviour of keepInput flag to write series which did not match any aggregators to the remote write.
See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4243

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* Update app/vmagent/remotewrite/remotewrite.go

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-07-24 16:34:38 -07:00
hagen1778
42af241d59
docs: update datadog config recommendations
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-24 16:17:35 -07:00
Roman Khavronenko
08e66885ab
docs: mention lambdas and cloud functions for aggregation (#4705)
* docs: mention lambdas and cloud functions for aggregation

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-24 16:16:43 -07:00
Haleygo
ee1b3a48e9
vmalert: unittest support stale datapoint (#4696)
* vmalert: unittest support stale datapoint

* add stale ut case
2023-07-24 16:15:27 -07:00
Roman Khavronenko
f536f22dfb
docs: add case for histograms aggregation (#4702)
docs: add case for histograms aggregation

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-24 16:14:36 -07:00
Nikolay
30b32583f4
lib/storage: pre-create timeseries before indexDB rotation (#4652)
* lib/storage: pre-create timeseries before indexDB rotation
during an hour before indexDB rotation start creating records at the next indexDB
it must improve performance during switch for the next indexDB and remove ingestion issues.
Since there is no need for creation new index records for timeseries already ingested into current indexDB
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4563

* lib/storage: further work on indexdb rotation optimization

- Document the change at docs/CHAGNELOG.md
- Move back various caches from indexDB to Storage. This makes the change less intrusive.
  The dateMetricIDCache now takes into account indexDB generation, so it stores (date, metricID)
  entries for both the current and the next indexDB.
- Consolidate the code responsible for idbNext pre-filling into prefillNextIndexDB() function.
  This improves code readability and maintainability a bit.
- Rewrite and simplify the code responsible for calculating the next retention timestamp.
  Add various tests for corner cases of this code.
- Remove indexdb pre-filling from RegisterMetricNames() function, since this function is rarely called.
  It is OK to add indexdb entries on demand in this function. This simplifies the code.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401

* docs/CHANGELOG.md: refer to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4563

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-07-22 15:23:14 -07:00
yrka
7750c5a553
Update MetricsQL.md (#4689)
misspell
2023-07-22 14:19:30 -07:00
Zakhar Bessarab
2fe33b3d97
app/vmalert/datasource/graphite: allow overriding "from" parameter for datasource queries (#4687)
* app/vmalert/datasource/graphite: allow overriding "from" parameter for datasource queries

Fixes construction of URL parameters for graphite render to allow overriding "from" parameter.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4685
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vmalert/datasource/graphite: update flow for building URL parameters

Makes flow of building URL parameters same as Prometheus datasource has:
1) Setting all default values
2) Merging those values with provided `extraParams`

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* Update docs/CHANGELOG.md

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-07-22 14:18:52 -07:00
Aliaksandr Valialkin
cbc5367812
docs/VictoriaLogs/data-ingestion: mention that VictoriaLogs automatically extracts message and timestamp from data ingested via Loki JSON API 2023-07-20 21:55:46 -07:00
Aliaksandr Valialkin
9e6cc3c495
deployment: update VictoriaLogs Docker image tag from from v0.2.0-victorialogs to v0.3.0-victorialogs 2023-07-20 21:55:09 -07:00
Aliaksandr Valialkin
c94a5d95a2
docs/VictoriaLogs/CHANGELOG.md: cut v0.3.0-victorialogs 2023-07-20 21:54:28 -07:00
Aliaksandr Valialkin
324a3c5288
lib/promscrape: follow-up after 6aa50ca954
- Improve docs
- Hide `debug relabeling` column when -promscrape.dropOriginalLabels command-line flag is set
- Inline the code from the added template functions, since the code is harder to follow
  with the template functions, especially when these functions have misleading names.
  Also, these functions are used only in one place, e.g. they do not reduce the amounts of code.
- Hide `click to show original labels` title at `labels` column when original labels aren't available.
- Show the reason on whey original labels aren't available at /service-discovery page.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4597
2023-07-20 21:54:09 -07:00
Aliaksandr Valialkin
c921bc0833
app/{vmselect,vlselect}: run make vmui-update vmui-logs-update after recent changes to VMUI
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4604
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4676
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4294
2023-07-20 21:53:51 -07:00
Aliaksandr Valialkin
f086a5ba63
docs/VictoriaLogs/CHANGELOG.md: document the addition of Grafana Loki push protocol support (aka Promtail)
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4482
2023-07-20 21:52:33 -07:00
Aliaksandr Valialkin
30098ac8bd
app/vlinsert/loki: follow-up after 09df5b66fd
- Parse protobuf if Content-Type isn't set to `application/json` - this behavior is documented at https://grafana.com/docs/loki/latest/api/#push-log-entries-to-loki

- Properly handle gzip'ped JSON requests. The `gzip` header must be read from `Content-Encoding` instead of `Content-Type` header

- Properly flush all the parsed logs with the explicit call to vlstorage.MustAddRows() at the end of query handler

- Check JSON field types more strictly.

- Allow parsing Loki timestamp as floating-point number. Such a timestamp can be generated by some clients,
  which store timestamps in float64 instead of int64.

- Optimize parsing of Loki labels in Prometheus text exposition format.

- Simplify tests.

- Remove lib/slicesutil, since there are no more users for it.

- Update docs with missing info and fix various typos. For example, it should be enough to have `instance` and `job` labels
  as stream fields in most Loki setups.

- Allow empty of missing timestamps in the ingested logs.
  The current timestamp at VictoriaLogs side is then used for the ingested logs.
  This simplifies debugging and testing of the provided HTTP-based data ingestion APIs.

The remaining MAJOR issue, which needs to be addressed: victoria-logs binary size increased from 13MB to 22MB
after adding support for Loki data ingestion protocol at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4482 .
This is because of shitty protobuf dependencies. They must be replaced with another protobuf implementation
similar to the one used at lib/prompb or lib/prompbmarshal .
2023-07-20 21:52:11 -07:00
Dmytro Kozlov
41f0227af2
docs: update information about tsdb usage in cluster version (#4678)
update information about tsdb usage in cluster version
2023-07-20 21:50:58 -07:00
hagen1778
1d2a0e0e10
docs: fix the next release version for vmalert
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-20 21:49:34 -07:00
Zakhar Bessarab
5b3000f7e8
docs: fix broken link to managed VM (#4679)
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-07-20 21:49:06 -07:00
hagen1778
5b963e1971
docs: mention streaming aggregation as way to reduce cardinality
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-20 21:48:09 -07:00
Alexander Marshalov
9ba03b4838
allow configuring staleness interval in stream aggregation (#4667) (#4670)
---------

Signed-off-by: Alexander Marshalov <_@marshalov.org>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-07-20 21:47:29 -07:00
hagen1778
f7d60613a9
docs: specify min version and limitations for vmalert's unit tests
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-20 21:27:50 -07:00
Haleygo
939c8b8372
vmalert: init unit test (#4596)
vmalert: support unit tests

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2945
---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2023-07-20 21:19:45 -07:00
hagen1778
a0cc6eb7a1
docs: mention the simplest way to migrate data in vmctl docs
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-20 21:12:18 -07:00
Dmytro Kozlov
f0d8f77e6d
app/vmagent: fix creating target id if --promscrape.dropOriginalLabels flag was used (#4616)
* app/vmagent: fix creating target id if `--promscrape.dropOriginalLabels` flag was used

* app/vmagent: hide links if OriginalLabels was dropped

* app/vmagent: update CHANGELOG.md and added information to the docs

* app/vmagent: fix comments
2023-07-20 19:21:41 -07:00
Zakhar Bessarab
5b3cbd4db1
app/vlinsert: add support of loki push protocol (#4482)
* app/vlinsert: add support of loki push protocol

- implemented loki push protocol for both Protobuf and JSON formats
- added examples in documentation
- added example docker-compose

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vlinsert: move protobuf metric into its own file

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* deployment/docker/victorialogs/promtail: update reference to docker image

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* deployment/docker/victorialogs/promtail: make volume name unique

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vlinsert/loki: add license reference

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* deployment/docker/victorialogs/promtail: fix volume name

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs/VictoriaLogs/data-ingestion: add stream fields for loki JSON ingestion example

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vlinsert/loki: move entities to places where those are used

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vlinsert/loki: refactor to use common components

- use CommonParameters from insertutils
- stop ingestion after first error similar to elasticsearch and jsonline

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vlinsert/loki: address review feedback

- add missing logstorage.PutLogRows calls
- refactor tenant ID parsing to use common function
- reduce number of allocations for parsing by reusing  logfields slices
- add tests and benchmarks for requests processing funcs

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-07-20 16:49:43 -07:00
Aliaksandr Valialkin
5ec1ec39cd
docs/stream-aggregation.md: mention that sum_samples, stddev, stdvar, histogram_bucket and quantiles outputs must be applied only to gauge metrics 2023-07-20 11:31:38 -07:00
Roman Khavronenko
92cd98e366
docs: mention tags routing for new release (#4566)
* docs: mention tags routing for new release

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Update docs/Release-Guide.md

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-07-19 16:56:04 -07:00
Aliaksandr Valialkin
29c6bcbd5b
app/vmselect: follow-up after 6a96fd8ed5
- Add `Active queries` chapter to VMUI docs
- Set `Content-Type: json` header inside promql.WriteActiveQueries() handler,
  in order to be consistent with other request handlers called at app/vmselect/main.go
- Pass the request to promql.WriteActiveQueries() handler, so it can change its output
  depending on the provided request params. This also improves consistency of
  promql.WriteActiveQueries() args with other request hanlers at app/vmselect/main.go

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4653
2023-07-19 16:31:26 -07:00
Yury Molodov
3ad80e281f
vmui: add Active Queries page (#4653)
* feat: add page to display a list of active queries (#4598)

* app/vmagent: code formatting

* fix: remove console

---------

Co-authored-by: dmitryk-dk <kozlovdmitriyy@gmail.com>
2023-07-19 16:02:58 -07:00
Aliaksandr Valialkin
a3c8f902c1
vendor: update github.com/VictoriaMetrics/metricsql from v0.60.0 to v0.61.1
This adds support for passing durations via WITH template vars:

- `WITH (w = 5m) m[w]` is transformed to `m[5m]`
- `WITH (f(w, step, off) = m[w:step] offset off) f(5m, 10s, 1h)` is transformed to `m[5m:10s] offset 1h`

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4025
Updates https://github.com/VictoriaMetrics/metricsql/issues/12

See also the initial implementation by @lujiajing1126 at https://github.com/VictoriaMetrics/metricsql/pull/13
2023-07-19 15:04:38 -07:00
hagen1778
7703059746
docs: typo
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-19 14:49:31 -07:00
Artem Navoiev
a600cc3645
specify formating in vmalert doc
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-19 14:49:02 -07:00
Roman Khavronenko
80768d53dd
docs: follow-up after aec4b5db81 (#4638)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-19 14:48:17 -07:00
Roman Khavronenko
debe1793bb
vmalert: follow-up after d4ac4b7813 (#4659)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-18 16:03:28 -07:00
hagen1778
456a2e70fd
docs: mention change from 6f3fee197e
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-18 16:01:59 -07:00
Aliaksandr Valialkin
2042ea7bf3
docs/VictoriaLogs/CHANGELOG.md: remove newlines inside FEATURE descriptions, since they dont play nicely when copy-n-pasting to https://github.com/VictoriaMetrics/VictoriaMetrics/releases/ 2023-07-17 23:31:37 -07:00
Aliaksandr Valialkin
15ccde8dd3
all: use the new release for VictoriaLogs - v0.2.0-victorialogs 2023-07-17 23:20:07 -07:00
Aliaksandr Valialkin
8e42b2294c
docs/VictoriaLogs: add CHANGELOG.md 2023-07-17 23:14:23 -07:00
Aliaksandr Valialkin
da2ef397fa
lib/logstorage: add abilty to speficy offset for the selected _time filter
The following syntax is supported: _time:filter offset off
For example:

- _time:5m offset 1h - 5-minute duration one hour before the current time
- _time:2023 offset 2w - 2023 year with the 2 weeks offset in the past
2023-07-17 19:07:14 -07:00
Aliaksandr Valialkin
6751a08071
lib/logstorage: support for short form of _time:(now-duration, now] filter: _time:duration 2023-07-17 18:23:43 -07:00
Aliaksandr Valialkin
8fdfd13a29
lib/logstorage: LogsQL: replace exact_prefix("...") with exact("..."*)
This makes LogsQL queries more consistent with i("...") and i("..."*) syntax
2023-07-17 17:19:45 -07:00
Aliaksandr Valialkin
5ace0701d3
app/vmselect/promql: add the ability to copy all the labels from one side of group_left()/group_right() operation
This is performed by specifying `*` inside group_left()/group_right().
Also allow specifying prefix for the copied labels via `group_left(...) prefix "..."` and `group_right(...) prefix "..."` syntax.
For example, the following query adds all the namespace-related labels to pod info, and prefixes all the copied label names with "ns_" prefix:

  kube_pod_info * on(namespace) group_left(*) prefix "ns_" kube_namespace_labels

This resolves the following StackOverflow questions:

- https://stackoverflow.com/questions/76661818/how-to-add-namespace-labels-to-pod-labels-in-prometheus
- https://stackoverflow.com/questions/76653997/how-can-i-make-a-new-copy-of-kube-namespace-labels-metric-with-a-different-name
2023-07-17 16:58:30 -07:00
Aliaksandr Valialkin
cc54fa2a56
app/vmselect/promql: recommend to use (a op b) keep_metric_names instead of a op b keep_metric_names
The `a op b keep_metric_names` is ambigouos to `a op (b keep_metric_names)` when `b` is a transform or rollup function.
For example, `a + rate(b) keep_metric_names`. So it is better to use more clear syntax: `(a op b) keep_metric_names`

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3710
2023-07-16 23:47:15 -07:00
Aliaksandr Valialkin
ec375e2e15
docs/keyConcepts.md: clarify that the number of ,-delimited filters per each or group can be arbitrary
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3997
2023-07-16 23:47:15 -07:00
Aliaksandr Valialkin
8715e331c8
docs/VictoriaLogs/data-ingestion/README.md: document how to verify whether the data is successfully ingested into VictoriaLogs 2023-07-16 22:54:05 -07:00
Zakhar Bessarab
781947a7e2
metricsql: add support of using keep_metric_names for binary operations (#4109)
* metricsql: add support of using keep_metric_names for binary operations

This should help to avoid confusion with queries like one in the issue #3710.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* wip

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-07-16 03:01:27 -07:00
Artem Navoiev
30e84f4c66
docs: logs disable table of content for pages with no h2-h6, add aliases
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-16 00:08:06 -07:00
Artem Navoiev
6363a7bb80
docs: victorialogs add front-matter for readme pages
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-16 00:08:06 -07:00
Artem Navoiev
2963cbbb74
docs: victorialogs add front-matter for data-ingestion items
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-16 00:08:06 -07:00
Aliaksandr Valialkin
a7fdc3fcc7
all: add support for or filters in series selectors
This commit adds ability to select series matching distinct filters via a single series selector.
For example, the following selector selects series with either {env="prod",job="a"}
or {env="dev",job="b"} labels:

  {env="prod",job="a" or env="dev",job="b"}

The `or` filter is supported in all the VictoriaMetrics tools now.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3997
Uses https://github.com/VictoriaMetrics/metricsql/pull/14
2023-07-15 23:56:18 -07:00
Aliaksandr Valialkin
6a65af5112
all: replace ElasticSearch -> Elasticsearch for the sake of consistency
This is a follow-up for 7f6b5dc47b
2023-07-14 10:52:43 -07:00
Aliaksandr Valialkin
d721109961
docs/CHANGELOG.md: sync with master branch 2023-07-14 10:48:40 -07:00
Aliaksandr Valialkin
99ec7ef28f
docs/VictoriaLogs: use exact number when comparing efficiency of VictoriaLogs and Elasticsearch 2023-07-14 10:47:29 -07:00
Haleygo
5e5c805599
vmalert: fix evalTS after modify group interval (#4629) 2023-07-14 10:47:29 -07:00
subham sarkar
9d3cfc03cd
docs: Update README.md (#4613)
Fix grammatical mistakes and also s/ElasticSearch/Elasticsearch
2023-07-14 10:46:13 -07:00
Artem Navoiev
c245e38f8d
fix alias
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-14 10:44:04 -07:00
Artem Navoiev
6b98511ac9
add front-matter for some of victorialogs pages
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-14 10:44:04 -07:00
Roman Khavronenko
89b5a6a4d5
vmctl: mention replicationFactor during migration (#4633)
Addresses https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4624

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-14 10:36:46 -07:00
Aliaksandr Valialkin
e1cf962bad
lib/storage: switch from global to per-day index for MetricName -> TSID mapping
Previously all the newly ingested time series were registered in global `MetricName -> TSID` index.
This index was used during data ingestion for locating the TSID (internal series id)
for the given canonical metric name (the canonical metric name consists of metric name plus all its labels sorted by label names).

The `MetricName -> TSID` index is stored on disk in order to make sure that the data
isn't lost on VictoriaMetrics restart or unclean shutdown.

The lookup in this index is relatively slow, since VictoriaMetrics needs to read the corresponding
data block from disk, unpack it, put the unpacked block into `indexdb/dataBlocks` cache,
and then search for the given `MetricName -> TSID` entry there. So VictoriaMetrics
uses in-memory cache for speeding up the lookup for active time series.
This cache is named `storage/tsid`. If this cache capacity is enough for all the currently ingested
active time series, then VictoriaMetrics works fast, since it doesn't need to read the data from disk.

VictoriaMetrics starts reading data from `MetricName -> TSID` on-disk index in the following cases:

- If `storage/tsid` cache capacity isn't enough for active time series.
  Then just increase available memory for VictoriaMetrics or reduce the number of active time series
  ingested into VictoriaMetrics.

- If new time series is ingested into VictoriaMetrics. In this case it cannot find
  the needed entry in the `storage/tsid` cache, so it needs to consult on-disk `MetricName -> TSID` index,
  since it doesn't know that the index has no the corresponding entry too.
  This is a typical event under high churn rate, when old time series are constantly substituted
  with new time series.

Reading the data from `MetricName -> TSID` index is slow, so inserts, which lead to reading this index,
are counted as slow inserts, and they can be monitored via `vm_slow_row_inserts_total` metric exposed by VictoriaMetrics.

Prior to this commit the `MetricName -> TSID` index was global, e.g. it contained entries sorted by `MetricName`
for all the time series ever ingested into VictoriaMetrics during the configured -retentionPeriod.
This index can become very large under high churn rate and long retention. VictoriaMetrics
caches data from this index in `indexdb/dataBlocks` in-memory cache for speeding up index lookups.
The `indexdb/dataBlocks` cache may occupy significant share of available memory for storing
recently accessed blocks at `MetricName -> TSID` index when searching for newly ingested time series.

This commit switches from global `MetricName -> TSID` index to per-day index. This allows significantly
reducing the amounts of data, which needs to be cached in `indexdb/dataBlocks`, since now VictoriaMetrics
consults only the index for the current day when new time series is ingested into it.

The downside of this change is increased indexdb size on disk for workloads without high churn rate,
e.g. with static time series, which do no change over time, since now VictoriaMetrics needs to store
identical `MetricName -> TSID` entries for static time series for every day.

This change removes an optimization for reducing CPU and disk IO spikes at indexdb rotation,
since it didn't work correctly - see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401 .

At the same time the change fixes the issue, which could result in lost access to time series,
which stop receving new samples during the first hour after indexdb rotation - see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2698

The issue with the increased CPU and disk IO usage during indexdb rotation will be addressed
in a separate commit according to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401#issuecomment-1553488685

This is a follow-up for 1f28b46ae9
2023-07-13 17:03:50 -07:00
Aliaksandr Valialkin
df67b78f75
docs/CHANGELOG.md: clarify the description of the bugfix at 177a0c1ca9
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4555
2023-07-13 12:19:00 -07:00
Dmytro Kozlov
f31ac064f9
app/vmctl: fix panic --remote-read-filter-time-start flag not defined (#4605)
* app/vmctl: fix panic `--remote-read-filter-time-start` flag not defined

* app/vmctl: update CHANGELOG.md

---------

Co-authored-by: Nikolay <nik@victoriametrics.com>
2023-07-13 12:13:21 -07:00
Dmytro Kozlov
555a0a9d57
app/vmctl: fix issue with adding many seconds (#4617)
* app/vmctl: fix issue with adding many seconds

* app/vmagent: add CHANGELOG.md
2023-07-13 12:09:54 -07:00
Roman Khavronenko
fdccb56620
vmalert: check for negative offset for missed rounds (#4628)
It could happen for low evaluation intervals and irregular
delays during execution that evaluation time would get
a negative offset. This could result into cumulative
discrepancy between the actual time and evaluation time for rules.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-13 12:05:52 -07:00
Alexander Marshalov
09cc17a6b1
fixed typo (#4622)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-07-13 11:29:57 -07:00
Alexander Marshalov
7a5e5f6a89
add info about using stream aggregation as statsd alternative (#4600) (#4621)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-07-13 11:29:57 -07:00
Aliaksandr Valialkin
2636c35cec
docs/VictoriaLogs/FAQ.md: small fixes 2023-07-12 01:10:40 -07:00
Aliaksandr Valialkin
b07a1c85b9
all: update Go builder from 1.20.5 to 1.20.6
See https://github.com/golang/go/issues?q=milestone%3AGo1.20.6+label%3ACherryPickApproved
2023-07-12 01:00:24 -07:00
Aliaksandr Valialkin
bcea7a0a92
docs/keyConcepts.md: cosmetic fixes after b67bd156d5 2023-07-12 00:30:33 -07:00
Aliaksandr Valialkin
b99fcb7d7a
docs/VictoriaLogs: add FAQ 2023-07-12 00:30:33 -07:00
Aliaksandr Valialkin
f8299ee9d8
docs/VictoriaLogs/README.md: make it clear that VictoriaLogs is open source 2023-07-12 00:30:33 -07:00
Alexander Marshalov
8277b64ec3
follow up for #4612 and #4584 (#4614)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-07-12 00:30:06 -07:00
Zakhar Bessarab
a667bdaad5
doc: fix image src after b67bd156 (#4612)
Followup for b67bd156d5

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-07-12 00:29:36 -07:00
Alexander Marshalov
9e8c02520c
added info about search.latencyOffset to key concepts (#4567) (#4584)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-07-12 00:25:32 -07:00
Aliaksandr Valialkin
a5f55259f6
docs/VictoriaLogs: make more prominent the information about returned log fields in query responses
Thanks to @candlerb for suggestions on how to improve VictoriaLogs docs
at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4609#issuecomment-1629758426
2023-07-10 15:01:54 -07:00
Dmytro Kozlov
5c4ca4aea8
app/vmctl: remove undefined flag from the documentation. See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4552. (#4606) 2023-07-10 15:01:54 -07:00
Aliaksandr Valialkin
81fe089546
docs/Single-server-VictoriaMetrics.md: mention how to use Prometheus config file with unsupported options
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4274
2023-07-09 12:37:44 -07:00
Aliaksandr Valialkin
371036eb35
docs/VictoriaLogs: small clarifications 2023-07-09 12:36:38 -07:00
Zakhar Bessarab
ddd918b93c
docs: make httpAuth.* flags description less ambiguous (#4588)
* docs: make `httpAuth.*` flags description less ambiguous

Currently, it may confuse users whether `httpAuth.*` flags are used by HTTP client or server configuration(see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4586 for example).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs: fix a typo

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-07-09 12:36:14 -07:00
Haleygo
ef8e3eb9b3
vmselect: fix result in Prometheus query when time is small (#4578)
vmselect: fix result in Prometheus query when time is small

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-07-09 12:33:29 -07:00
Aliaksandr Valialkin
e1a2404db5
app/vmselect/netstorage: follow-up after 173ccf4333
- Clarify docs about -replicationFactor command-line flag at vmselect
- Clarify description for -replicationFactor and -search.skipSlowReplicas command-line flags
- Fix the logic for returning responses if -search.skipSlowReplicas command-line flag
  is enabled. The logic was broken in the 173ccf4333,
  so it could return responses only if some of vmstorage nodes return error,
  while it should return when query results are successfully collected from more than
  (len(storageNodes) - replicationFactor) vmstorage nodes.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1207
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711
2023-07-09 11:58:22 -07:00
Haleygo
3c2308fd52
vmalert:fix query request using rfc3339 format (#4577)
vmalert: consistently use time.RFC3339 format for time in queries

Co-authored-by: hagen1778 <roman@victoriametrics.com>
2023-07-09 11:03:10 -07:00
Roman Khavronenko
173ccf4333
vmselect: introduce search.skipSlowReplicas cmd-line flag (#4538)
* vmselect: introduce `search.skipSlowReplicas` cmd-line flag

vmselect has two logical conditions during request processing when
`-replicationFactor` cmd-line flag is set:
1. If at least `len(storageNodes) - replicationFactor` responded, it could skip
waiting for the rest of nodes to respond. This could lead to problems described
here https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1207.
2. Mark response as partial if less than `len(storageNodes) - replicationFactor` responded
without an error.

The P1 showed itself error-prone and became the main reason why
`-replicationFactor` wasn't recommended to use at vmselect level.
However, this optimization could be still very useful in situations
when there are slow and fast replicas in cluster.

But P2 remains viable and important conditionless.
Hiding P1 behind the feature-flag `search.skipSlowReplicas`
should make `-replicationFactor` flag usable again. And let users
choose whether they want P1 to be respected.

Related issues
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1207
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* docs: update changelog

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-07 11:50:26 +02:00
Artem Navoiev
bf49efc11a
update logo width in cluster doc to 300
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-06 23:16:42 -07:00
Roman Khavronenko
109e55f865
vmalert: allow disabling of step param attached to instant queries (#4574)
vmalert: allow disabling of `step` param attached to instant queries

This might be useful for using vmalert with datasources that to not support this param,
unlike VictoriaMetrics.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4573

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-06 23:13:56 -07:00
Aliaksandr Valialkin
0107d78639
docs/vmgateway.md: update -help output 2023-07-06 23:07:47 -07:00
Aliaksandr Valialkin
72dd0b9fac
docs/Cluster-VictoriaMetrics.md: update -help output 2023-07-06 23:06:11 -07:00
Aliaksandr Valialkin
ee4280d132
docs/vmbackupmanager.md: update -help output 2023-07-06 22:57:31 -07:00
Aliaksandr Valialkin
921d8b36b5
docs/vmrestore.md: update -help output 2023-07-06 22:55:26 -07:00
Aliaksandr Valialkin
e1993dadc2
docs/vmbackup.md: update -help output 2023-07-06 22:54:15 -07:00
Aliaksandr Valialkin
e35abdd2e4
docs/vmauth.md: update -help output 2023-07-06 22:52:48 -07:00
Aliaksandr Valialkin
316abe550d
docs/vmalert.md: update -help output 2023-07-06 22:50:47 -07:00
Aliaksandr Valialkin
b9790515e4
docs/vmagent.md: update -help output 2023-07-06 22:48:23 -07:00
Aliaksandr Valialkin
65d7194588
docs/Single-server-VictoriaMetrics.md: update -help output 2023-07-06 22:45:58 -07:00
Aliaksandr Valialkin
eea088d87f
docs/CHANGELOG.md: clarify description for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4336 bugfix
This is a follow-up for 5eb5df96e2
2023-07-06 22:42:02 -07:00
Aliaksandr Valialkin
eeb53660b8
docs/CHANGELOG.md: use the proper link to the issue related to the commit 7a92263459
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4402
2023-07-06 22:41:43 -07:00
Aliaksandr Valialkin
67a8992798
docs/CHANGELOG.md: remove redundant info from the url to consulagent_sd_configs docs
This is a follow-up for 40d12be607
2023-07-06 22:41:23 -07:00
Aliaksandr Valialkin
40f1ccba67
docs/CHANGELOG.md: clarify the description of the bugfix at ce7141383d 2023-07-06 22:41:03 -07:00
Aliaksandr Valialkin
dc89e1f644
app/vmselect/graphite: follow-up after c7884f8686
- Consistently use -search.maxGraphiteTagValues for limiting tag values from auto-complete API
- Use -search.maxGraphiteSeries for limiting paths (aka series), which can be returned from Graphite series API
- Clarify the change in docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4339
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2841
2023-07-06 22:33:30 -07:00
Alexander Marshalov
eb611c3dc3
fix removing storage data dir before restoring from backup (#598)
* fix removing storage data dir before restoring from backup

Signed-off-by: Alexander Marshalov <_@marshalov.org>

* fix review comment

Signed-off-by: Alexander Marshalov <_@marshalov.org>

* fix review comment

Signed-off-by: Alexander Marshalov <_@marshalov.org>

* fixes after merge with `enterprise-single-node` branch

Signed-off-by: Alexander Marshalov <_@marshalov.org>

---------

Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-07-06 22:32:12 -07:00
Aliaksandr Valialkin
2f19ba0f75
app/vmselect/netstorage: follow-up after 11ac551d52
- Clarify the scope of the fix at docs/CHANGELOG.md
- Handle the case when -search.maxSamplesPerSeries limit is exceeded
  in the same way as the -search.maxSamplesPerQuery limit.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4472
2023-07-06 22:26:47 -07:00
Roman Khavronenko
690f58c016
docs: explicitly mention errors processing for import APIs (#4583)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-06 22:26:04 -07:00
Denys Holius
97a5bdf4f0
docs: adds curl commands to clear the query cache (#4468)
adds curl commands to clear query cache on vmselect/VM Single
2023-07-06 22:25:21 -07:00
Aliaksandr Valialkin
85c134feec
docs/VictoriaLogs/LogsQL.md: various fixes according to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4579 2023-07-06 22:24:41 -07:00
Aliaksandr Valialkin
7cf5efc5b8
README.md: add a link to VictoriaLogs 2023-07-06 22:23:54 -07:00
Aliaksandr Valialkin
5b94246d92
docs: add Roblox case study 2023-07-06 22:23:13 -07:00
Aliaksandr Valialkin
ff96e9cfc7
docs/Single-server-VictoriaMetrics.md: fix link to Storage section after the ab2d184e42 2023-07-06 22:22:42 -07:00
Roman Khavronenko
bd5abb74fd
vmctl: interrupt explore procedure in influx mode if no numeric fields were found (#4576)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-06 22:21:18 -07:00
Roman Khavronenko
9cfd8d6b86
Docs retention (#4568)
* docs: mention parts and partitions in Retention section

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-06 22:19:43 -07:00
Roman Khavronenko
41f0ed48eb
docs: follow-up after 9da638aa66 (#4572)
9da638aa66

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-06 22:18:54 -07:00
Dmytro Kozlov
dd412a3757
app/vmalert: show on UI groups error after reload config (#4543)
show on UI groups error after reload config

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4076

Co-authored-by: hagen1778 <roman@victoriametrics.com>
2023-07-06 22:11:36 -07:00
Haleygo
b029286298
fix parse for invalid partial RFC3339 format (#4539)
The validation was needed for covering corner cases when storage is tested with data from 1970.
This resulted into unexpected search results, as year was parsed incorrectly from the given timestamp.


Co-authored-by: hagen1778 <roman@victoriametrics.com>
2023-07-06 22:09:35 -07:00
Artem Navoiev
7ff0ac1a33
docs: add command-line flags (#4550)
* add command-line flags

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-06 22:08:21 -07:00
Nikolay
68879061be
docs: adds v1.91.3 release docs (#4561) 2023-07-06 22:06:58 -07:00
Zakhar Bessarab
b801360075
docs: clarify downsampling periods requirements (#4542)
It is required for periods to be multiplies, but it was not stated clearly in documentation.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-07-06 21:58:40 -07:00
Yury Molodov
8c190ec8fb
vmui: fix app routing issues (#4408)
The change focuses on rectifying inconsistencies in the navigation behavior of the application
and eliminating issues encountered when manually altering the URL.

The key updates include:
- Refactoring of the routing mechanism to handle all possible routes and their states.
- Enhancement of the React Router usage to ensure a smoother navigation experience.
- Handling application state when the URL is manually changed.
2023-07-06 21:58:09 -07:00
Zakhar Bessarab
7eeb2d553f
docs/operator: add note about selectors for VMProbe (#4541)
Added a line for `probeSelector` and links to objects selectors section to make it easier to find more details about selectors.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-07-06 21:57:21 -07:00
Alexander Marshalov
677c8a5465
show backup progress percentage in vmbackup log during backup uploading and restoring progress percentage in vmrestore log during backup downloading (#4460) (#4530)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-07-06 21:56:54 -07:00
Roman Khavronenko
cf433c066a
vmauth: expose latency metrics per user (#4525)
expose `vmauth_user_request_duration_seconds`
and `vmauth_unauthorized_user_request_duration_seconds` summary metrics
for measuring requests latency per user.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-06 21:55:37 -07:00
Roman Khavronenko
8a15397b5c
vmauth: rm ip filters from non-ent config example (#4526)
It is impossible to run OS vmauth with the provided config.
The example of using ip filters should be only a part of docs.
All other examples should work seamlessly with OS version.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-06 21:52:51 -07:00
Haleygo
9e49a9e924
vmalert: add vmalert_remotewrite_sent_duration_seconds_total metric (#4517)
add `vmalert_remotewrite_sent_duration_seconds_total` metric
2023-07-06 21:51:31 -07:00
Dmytro Kozlov
316fe82dd6
docs: scroll to center of the view port (#4515) 2023-07-06 21:50:31 -07:00
Craig Rodrigues
4e0ca99da9
docs/VictoriaLogs: Fix curl command for fetching binary 2023-07-06 21:50:05 -07:00
Dmytro Kozlov
7498d139ca
docs: add scroll to the selected element (#4508)
* docs: add scroll to the selected element

* docs: scroll to root if element not found

* docs: simplify code

* docs: code cleanup

* docs: fix comments (fix code formatting, check element only inside sidebar container)
2023-07-06 21:48:53 -07:00
Artem Navoiev
28301d1b41
remove deleted repo from the docs
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-06 21:47:07 -07:00
Roman Khavronenko
d5e7ea5ef3
vmalert: update retry policy for pushing data to -remoteWrite.url (#4504)
By default, vmalert will make multiple retry attempts with exponential delay.
The total time spent during retry attempts shouldn't exceed `-remoteWrite.retryMaxTime` (default is 30s).
When retry time is exceeded vmalert drops the data dedicated for `-remoteWrite.url`.
Before, vmalert dropped data after 5 retry attempts with 1s delay between attempts (not configurable).

See `-remoteWrite.retryMinInterval` and `-remoteWrite.retryMaxTime` cmd-line flags.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2023-07-06 21:44:18 -07:00
Roman Khavronenko
311a81c7b0
vmalert: properly interrupt remotewrite retries on shutdown (#4505)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-06 21:43:04 -07:00
Zakhar Bessarab
7a000159d8
docs/changelog: followup for 830dac177f (#4499)
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-07-06 21:41:36 -07:00
Artem Navoiev
d48e697bbd
fix jsonline endpoint in docs
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-06 21:40:59 -07:00
Aliaksandr Valialkin
b215e6e044
docs/VictoriaLogs/QuickStart.md: add an option to run VictoriaLogs from pre-built binaries 2023-07-06 21:39:12 -07:00
Aliaksandr Valialkin
166456f340
docs/VictoriaLogs/README.md: add Upgrading section 2023-07-06 21:39:12 -07:00
Aliaksandr Valialkin
881dc29ef8
docs/VictoriaLogs/README.md: typo fix 2023-07-06 21:39:12 -07:00
Aliaksandr Valialkin
a3956b3d9c
all: update VictoriaLogs docker tag to v0.1.0-victorialogs 2023-07-06 21:39:12 -07:00
Aliaksandr Valialkin
4b10432435
app/vlselect: handle vmui at /select/vmui path instead of /vmui
This simplifies routing at auth proxies such as vmauth to vlselect component,
which serves VMUI - just route all the requests, which start with /select/, to vlselect.
2023-07-06 21:36:28 -07:00
Aliaksandr Valialkin
33e4d51636
docs/VictoriaLogs/data-ingestion/README.md: remove trailing spaces 2023-07-06 21:35:55 -07:00
Aliaksandr Valialkin
08634ae612
app/vlinsert/jsonline: code prettifying 2023-07-06 21:35:55 -07:00
Aliaksandr Valialkin
64b8aa108b
docs/VictoriaLogs/README.md: small fixes 2023-07-06 21:35:55 -07:00
Aliaksandr Valialkin
7206f91201
docs/VictoriaLogs/data-ingestion: small fixes 2023-07-06 21:35:22 -07:00
Aliaksandr Valialkin
ee720d1bbc
docs/VictoriaLogs/querying: rename VMUI -> 'Web UI' in order to reduce confusion with VictoriaMetrics UI 2023-07-06 21:35:03 -07:00
Aliaksandr Valialkin
2b43e5be80
docs/VictoriaLogs: small fixes 2023-07-06 21:34:41 -07:00
Alexander Marshalov
c6dba396db
added docs for vmui in victorialogs (#4494) 2023-07-06 21:33:49 -07:00
Alexander Marshalov
c12b5250c7
added more info and examples about data ingestion and collectors to VictoriaLogs docs (#4490) 2023-07-06 21:31:56 -07:00
Alexander Marshalov
d9d759bc90
jsonline support for data ingestion in vlinsert (#4487)
added json lines / json stream format for ingestion to vlinsert
2023-07-06 21:30:35 -07:00
Artem Navoiev
a972567faa
docs: change wording in victorialogs benchmarks section
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-06 21:29:55 -07:00
Artem Navoiev
b33f7a3181
docs: change wording in victorialogs benchmarks section
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-06 21:29:34 -07:00
Artem Navoiev
d9b625719f
fix typos
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-06 21:29:11 -07:00
Zakhar Bessarab
7ab9a4d5dc
docs/VictoriaLogs: add benchmarks section into readme
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-07-06 21:28:48 -07:00
Zakhar Bessarab
ed18c503ac
docs/VictoriaLogs: add reference to benchmark setup in readme
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-07-06 21:28:27 -07:00
hagen1778
b43d27275d
docs/metricsql: fix typo in expression
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-06 21:28:01 -07:00
Aliaksandr Valialkin
efee71986f
app/vlselect/logsql: sort query results by _time if their summary size doesnt exceed -select.maxSortBufferSize 2023-07-06 21:25:00 -07:00
Roman Khavronenko
4e99bf8c9e
docs/vmalert: specify version requirements for new features (#4480)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-06 21:24:38 -07:00
Aliaksandr Valialkin
c77f729680
docs/VictoriaLogs/README.md: typo fix 2023-07-06 21:23:45 -07:00
Aliaksandr Valialkin
74ac4722a9
docs/VictoriaLogs/README.md: add missing link to quick start docs 2023-07-06 21:23:21 -07:00
Aliaksandr Valialkin
fd6c2dd02e
docs/VictoriaLogs: change the structure of the docs in order to be more maintainable
The change is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4477
2023-07-06 21:22:59 -07:00
Aliaksandr Valialkin
ca41a164cc
docs/url-examples.md: add missing Content-Type: application/json header in curl examples for json data ingestion
If the `Content-Type: application/json` request header isn't set,
then the server can improperly consume the request body when parsing request parameters
2023-07-06 21:21:34 -07:00
Aliaksandr Valialkin
0edfa06daa
docs/Single-server-VictoriaMetrics.md: update link to JSON stream format, since the previous link became broken 2023-07-06 21:20:32 -07:00
Aliaksandr Valialkin
1c39af56ab
app/victoria-logs: add ability to debug data ingestion by passing debug query arg to data ingestion API 2023-07-06 21:19:58 -07:00
Aliaksandr Valialkin
5fdf82657f
docs/VictoriaLogs: mention that VictoriaLogs supports multitenancy and out of order logs ingestion 2023-07-06 21:19:33 -07:00
Aliaksandr Valialkin
61f9b25782
docs/Single-server-VictoriaMetrics.md: refer to Slack chat at the top of the docs 2023-07-06 21:18:59 -07:00
Artem Navoiev
28bef227c8
change title of the operator doc to match title frontmatter
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-06 21:18:21 -07:00
Roman Khavronenko
d4ee505f6f
vmalert: retry all errors except 4XX status codes (#4461)
vmalert: retry all errors except 4XX status codes

Retry all errors except 4XX status codes while pushing via remote-write
to the remote storage. Previously, errors like broken connection could
prevent vmalert from retrying the request.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-06 17:34:32 -07:00
Yury Molodov
0ad966a898
vmui: memory leak fix (#4455)
* fix: optimize the preparation of data for the graph

* fix: optimize tooltip rendering

* fix: optimize re-rendering of the chart

* vmui: memory leak fix
2023-07-06 17:33:54 -07:00
Aliaksandr Valialkin
8b479e9122
docs/VictoriaLogs/LogsQL.md: typo fixes 2023-07-06 17:33:18 -07:00
Aliaksandr Valialkin
f4137a28ff
docs/VictoriaLogs/LogsQL.md: typo fixes 2023-07-06 17:33:18 -07:00
Aliaksandr Valialkin
cbfbaf37f3
docs/VictoriaLogs/LogsQL.md: typo fix 2023-07-06 17:32:51 -07:00
Aliaksandr Valialkin
c96cdc75c2
docs/VictoriaLogs/LogsQL.md: typo fix 2023-07-06 17:32:27 -07:00
Aliaksandr Valialkin
4d1c2b1722
docs/VictoriaLogs/LogsQL.md: typo fixes 2023-07-06 17:32:06 -07:00
Aliaksandr Valialkin
b66fbf8509
docs/VictoriaLogs/README.md: typo fixes 2023-07-06 17:31:45 -07:00
Aliaksandr Valialkin
137cbc5529
docs/VictoriaLogs/README.md: document how to run docker image for VictoriaLogs 2023-07-06 17:31:14 -07:00
Aliaksandr Valialkin
374890294e
app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
Aliaksandr Valialkin
46210c4d5e
lib/promutils.ParseTime(): add support for timestamps in milliseconds
See https://stackoverflow.com/questions/76437098/how-to-handle-time-unit-and-step-while-ingesting-or-querying-in-victoriametrics/76438405

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4459
2023-07-06 17:11:54 -07:00
Nikolay
dd7ebd6779
lib/storage: creates parts.json on start-up if it not exists. (#4450)
* lib/storage: creates parts.json on start-up if it not exists.
It fixes migrations from versions below v1.90.0.
Previously parts.json was created only after successful merge.
But if merge was interruped for some reason (OOM or shutdown), parts.json wasn't created and partitions left after interruped merge weren't properly deleted.
Since VM cannot check if it must be removed or not.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4336

* Apply suggestions from code review

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* Update lib/storage/partition.go

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-07-06 17:10:26 -07:00
Roman Khavronenko
3937da289a
docs/ReleaseGuide: mention auth for Docker (#4453)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-06 17:02:07 -07:00
Roman Khavronenko
0e3e045f5b
docs: mention errors processing for /api/v1/import API (#4448)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4446

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-06 17:01:34 -07:00
Dmytro Kozlov
b32a270da7
vmctl: increase retry backoff policy delay (#4447)
vmctl: update backoff policy on retries to reduce probability of overloading for `source` or `destination` databases
2023-07-06 17:00:06 -07:00
Roman Khavronenko
b76c0d182c
docs/vmalert: mention same labelset error in docs (#4443)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-06 16:56:42 -07:00
Dmytro Kozlov
2e81c5f740
vmctl: finish retries if context canceled (#4442)
vmctl: interrupt backoff retries if import context is cancelled

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-07-06 16:56:00 -07:00
Roman Khavronenko
0e0b7bf87f
docs/release-guide: update instructions (#4391)
docs/release-guide: update instructions for MacOS users

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-06 16:54:46 -07:00
Alexander Marshalov
4084dba9e4
fixed service name detection for consulagent service discovery in case of a difference in service name and service id (#4390) (#4439)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-07-06 16:53:29 -07:00
Roman Khavronenko
ecd7ec4832
Dashboard upd (#4438)
dashboards: update dashboard for single-node version
* add anonymous mem usage panel;
* add syscall rate panel;
* add location to logs panel;
* update legend for panels to reflect instance name;
* update queries to aggregate per instance.

dashboards: update dashboard for cluster version
* add syscall rate panel;
* add drilldown to logs panel.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-06 16:49:42 -07:00
greynix
82a2d6aba5
docs/oprator/api.md: corrected broken links to Kubernetes documentation (#4433) 2023-07-06 16:47:44 -07:00
Roman Khavronenko
2e3c3cf7ea
docs: make docs-sync (#4430)
docs: fix typo

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-06 16:47:10 -07:00
Roman Khavronenko
91612b38cd
docs: mention stream aggregation as more efficient approach for aggregation (#4429)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-06 16:46:17 -07:00
Roman Khavronenko
5518cb2f9a
docs/keyConcepts: explicitly specify time unit for API args (#4428)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-06 16:45:27 -07:00
Aliaksandr Valialkin
ed868f47f9
docs/CHANGELOG.md: remove the change regarding http2 support at vmagent
This is a follow-up for 8a07621a0c

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4283
2023-07-06 16:06:44 -07:00
Aliaksandr Valialkin
dff199a745
app/vmselect/graphite: follow-up after c7884f8686
- Consistently use -search.maxGraphiteTagValues for limiting tag values from auto-complete API
- Use -search.maxGraphiteSeries for limiting paths (aka series), which can be returned from Graphite series API
- Clarify the change in docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4339
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2841
2023-07-06 15:19:07 -07:00
Aliaksandr Valialkin
6a3cee5c2c
lib/promscrape/discoveryutils: re-use checkRedirect function for both client and blockingClient
Also document follow_redirects option at https://docs.victoriametrics.com/sd_configs.html#http-api-client-options

This is a follow-up for b3d0ff463a

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4282
2023-07-06 10:52:13 -07:00
Aliaksandr Valialkin
ec75d9097d
app/vmselect/netstorage: follow-up after 11ac551d52
- Clarify the scope of the fix at docs/CHANGELOG.md
- Handle the case when -search.maxSamplesPerSeries limit is exceeded
  in the same way as the -search.maxSamplesPerQuery limit.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4472
2023-07-05 21:13:34 -07:00
Roman Khavronenko
11ac551d52
app/vmselect/netstorage: properly process -search.maxSamplesPerQuery limit (#4472)
Properly return the error to user when `-search.maxSamplesPerQuery` limit is exceeded.
Before, user could have received a partial response instead.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-06-23 13:17:34 +02:00
Dmytro Kozlov
88ac6116bd
docs: clarify -retentionPeriod flag usage (#4417)
app/vmstorage: clarify the min value for `-retentionPeriod` flag

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

(cherry picked from commit 24f34347f1)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-06-09 10:44:22 +02:00
Roman Khavronenko
4624fda00d
all: update Go builder from Go1.20.4 to Go1.20.5 (#4427)
See https://github.com/golang/go/issues?q=milestone%3AGo1.20.5+label%3ACherryPickApproved

Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 476c7bdd6f)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-06-09 10:42:15 +02:00
Roman Khavronenko
c42365dc31
docs/changelog: mention a6a7795b9e change (#4425)
docs/changelog: mention a6a7795b9e change

a6a7795b9e

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit d4c314d628)
2023-06-09 10:41:07 +02:00
Zakhar Bessarab
bcece4c5ce
doc: changelog followup for #4420 fix (#4421)
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit 9a490d0b5c)
2023-06-09 10:41:07 +02:00
Zakhar Bessarab
7925e9698f
app/vmagent/remotewrite: fix vmagent panic on shutdown (#4407)
app/vmagent/remotewrite: fix vmagent panic on shutdown

Currently, when vmagent is stopping it first flushes pending series in remote write context and proceeds to stop streaming aggregation. This leads to streaming aggregation being unable to write results into pending timeseries (since it is already nil) and panic.
This can lead to losing some aggregation results being lost almost silently.

The fix is reordering flow to first stop streaming aggregation and flush all pending time series after that.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit ce7141383d)
2023-06-09 10:40:52 +02:00
Roman Khavronenko
3544e66a95
docs: mention checksums and data corruption (#4404)
Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 96b40b044c)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-06-09 10:40:48 +02:00
Roman Khavronenko
71e9eaae0a
docs/keyConcepts: replace lookback window with more clear desc (#4405)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit bfa0fd3eb3)
2023-06-09 10:40:26 +02:00
Roman Khavronenko
fb9b8f6b1b
app/vmagent: mention enable_http2 in changelog (#4403)
Follow-up after
72c3cd47eb

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 3305a6901c)
2023-06-09 10:40:24 +02:00
Roman Khavronenko
d9131d71cd
docs/CHANGELOG.md: cut v1.91.2 (#4393)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit cc739e3f8d)
2023-06-09 10:40:13 +02:00
Dmytro Kozlov
dd89fb2e12
app/vmctl: add verbose output for docker installations or when TTY isn't available (#4333)
* app/vmctl: add verbose output for docker installations or when TTY isn't available

* app/vmctl: fix tests

* app/vmctl: make vmctl interactive if no tty

* app/vmctl: cleanup

* app/vmctl: add comment

---------

Co-authored-by: Nikolay <nik@victoriametrics.com>

(cherry picked from commit fc5292d8ed)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-06-09 10:39:38 +02:00
Dmytro Kozlov
c5debee3f4
app/{graphite,netstorage,prometheus}: fix graphite search tags api limits, remove redudant limit from SeriesHandler handler (#4352)
* app/{graphite,netstorage,prometheus}: fix graphite search tags api limits, remove unused limit from SeriesHandler handler,

* app/{graphite,netstorage,prometheus}: use search.maxTagValues for Graphite

* app/{graphite,netstorage,prometheus}: update CHANGELOG.md

* app/{graphite,netstorage,prometheus}: use own flags for Graphite API

* app/{graphite,netstorage,prometheus}: cleanup

* app/{graphite,netstorage,prometheus}: cleanup

* app/{graphite,netstorage,prometheus}: update docs

---------

Co-authored-by: Nikolay <nik@victoriametrics.com>

(cherry picked from commit c7884f8686)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-06-09 10:39:12 +02:00
Roman Khavronenko
a1b6a9317d
vmalert: fix nil map assignment (#4392)
* vmalert: fix nil map assignment

The storage instance with nil map params was created for remote-read purposes.
And before change 7a9ae9de0d this map was ignored in ApplyParams.
Now, it started to be used and vmalert panics in runtime.

The fix properly inits map for at `NewVMStorage` and verifies it is not nil
on assignment in `ApplyParams`.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: add to changelog

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: properly clone Storage params

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: properly clone Storage params

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: properly clone Storage params

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit de94812088)
2023-06-02 13:29:51 +02:00
Roman Khavronenko
e0d565bce1
docs/CHANGELOG.md: cut v1.91.1 (#4386)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit b771152039)
2023-06-02 13:29:49 +02:00
Nikolay
89c036f4f6
docs/changlelog: mention 6c84b61 (#4384)
(cherry picked from commit 2c876227e4)
2023-06-02 13:29:48 +02:00
Roman Khavronenko
598671c0e8
docs: mention fix for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4221 (#4382)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4221

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 4b5faf7efb)
2023-06-02 13:29:46 +02:00
Dmytro Kozlov
ce98e68fe1
app/vmui: fix behavior when changing url in global settings (#4332)
* app/vmui: fix behavior when changing url in global settings

* app/vmctl: minor fix

* app/vmui: fix behavior when changing url in global settings

(cherry picked from commit 9843ec0e1d)
2023-06-02 13:29:46 +02:00
Nikolay
5066907317
docs: mention recent changes at changelog (#4379)
(cherry picked from commit a0bf8f233f)
2023-06-02 13:29:45 +02:00
Roman Khavronenko
7da0c37d2b
docs: clarify deduplication docs (#4371)
The purpose of the change is too highlight what HA pair is
and how deduplication needs identical labels to be present
in raw samples.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4367

Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 8185c2466c)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-06-02 13:29:45 +02:00
Denys Holius
f78e34c0b0
docs/Quick-Start.md: adds missed command to 'Starting VM-Cluster via Docker' section (#4375)
(cherry picked from commit 0bca7d49b9)
2023-06-02 13:29:15 +02:00
Nikolay
043431093a
app/vmauth: properly handle LOCAL proxy protocol command (#4373)
app/vmauth: properly handle LOCAL proxy protocol command

It is required for handling health checks from load balancers

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335
(cherry picked from commit f263031fe9)
2023-06-02 13:29:15 +02:00
Roman Khavronenko
c862952cb0
docs: mention disk space requirements for downsampling (#4369)
Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit a0c040ea58)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-06-02 13:29:14 +02:00
Roman Khavronenko
288e8fec6b
vmalert: mention default value for external.url flag (#4365)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 1b24f4729b)
2023-06-02 13:28:08 +02:00
Artem Navoiev
c9c4c92e6e
docs: fix enterprise page title to match with official name
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit a8c58fc145)
2023-06-02 13:28:06 +02:00
Artem Navoiev
6dca7d245b
docs: fix markdown headers ordering
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit 5b0e8c797c)
2023-06-02 13:28:05 +02:00
Artem Navoiev
27960fd6ec
docs: fix markdown for title
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit 83c1944184)
2023-06-02 13:28:04 +02:00
Artem Navoiev
bbd1dd9d03
fix image path
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit 6967c3cb95)
2023-06-02 13:28:04 +02:00
Artem Navoiev
756f0e1545
docs: change images from markdown tag to html for migration
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit 8a4c89ea22)
2023-06-02 13:28:04 +02:00
Roman Khavronenko
74c4c1d909
vmalert: properly form assets address if httpPrefix set (#4351)
Properly form path to static assets in WEB UI
 if `http.pathPrefix` set.

 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4349

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 51cea6cad4)
2023-06-02 13:28:03 +02:00
Roman Khavronenko
a0825ca5cd
docs: mention multi-tenancy in docs (#4357)
The update should make understanding of multi-tenancy more clear
for influxdb users.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 1c3f50f791)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-06-02 13:28:02 +02:00
Artem Navoiev
3185b8048c
update docs-sync Makefile command, add hugo front-matter
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

(cherry picked from commit f791811b15)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-06-02 13:24:41 +02:00
Artem Navoiev
ec97b851bb
remove absolute links, fix aliase
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit 434358b551)
2023-06-02 13:19:33 +02:00
Artem Navoiev
db3d34ca9a
fix title in multi-regional-setup-dedicated-regions.md
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit c1ad48edf9)
2023-06-02 13:19:33 +02:00
Aliaksandr Valialkin
f67a2f1e0f
docs/CHANGELOG.md: document v1.79.13 LTS release 2023-05-18 19:56:43 -07:00
Aliaksandr Valialkin
3c4aeddfde
docs/CHANGELOG.md: document v1.87.6 LTS release 2023-05-18 17:50:22 -07:00
Aliaksandr Valialkin
b73a2336f0
deployment/docker: update VictoriaMetrics version from v1.90.0 to v1.91.0 in docker compose files 2023-05-18 15:34:54 -07:00
Aliaksandr Valialkin
63540f3ff1
docs/Release-Guide.md: fix a link to sandbox environment
The link became broken after 5f9d3f9cb5
2023-05-18 15:34:52 -07:00
Aliaksandr Valialkin
deab51e708
docs/CHANGELOG.md: cut v1.91.0 2023-05-18 12:37:20 -07:00
Aliaksandr Valialkin
0ebfb91aba
lib/storage: revert the migration from global to per-day index for (MetricName -> TSID)
This reverts the following commits:
- e0e16a2d36
- 2ce02a7fe6

The reason for revert: the updated logic breaks assumptions made
when fixing https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2698 .
For example, if a time series stop receiving new samples during the first
day after the indexdb rotation, there are chances that the time series
won't be registered in the new indexdb. This is OK until the next indexdb
rotation, since the time series is registered in the previous indexdb,
so it can be found during queries. But the time series will become invisible
for search after the next indexdb rotation, while its data is still there.

There is also incompletely solved issue with the increased CPU and disk IO resource
usage just after the indexdb rotation. There was an attempt to fix it, but it didn't fix
it in full, while introducing the issue mentioned above. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401

TODO: to find out the solution, which simultaneously solves the following issues:
- increased memory usage for setups high churn rate and long retention (e.g. what the reverted commit does)
- increased CPU and disk IO usage during indexdb rotation ( https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401 )
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2698
2023-05-18 11:28:54 -07:00
Dmytro Kozlov
6da5cfa3a8
app/vmctl: set default value for --vm-native-step-interval flag (#4327)
* app/vmctl: set default value for `--vm-native-step-interval` flag

* app/vmctl: update CHANGELOG.md

* app/vmctl: update CHANGELOG.md, fix docs

* app/vmctl: fix typo

* app/vmctl: fix typo
2023-05-18 11:03:26 -07:00
Denys Holius
211f48fed6
deployment/docker/Makefile: updated docker compose commands regarding migration from V1 to V2 (#4314)
deployment/docker/Makefile: updated docker compose commands regarding migration from V1 to V2
2023-05-18 10:48:35 -07:00
Aliaksandr Valialkin
0397b3f0f7
lib/handshake: do not pollute logs with cannot read hello messages on TCP health checks
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1762
2023-05-18 10:37:59 -07:00
Aliaksandr Valialkin
6aa8029f30
app/vmauth: simplify the code after 4a1d29126c
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4242
2023-05-17 00:38:04 -07:00
Nikolay
da115b5170
app/vmauth: retry common network dial errors (#4280)
with tracking request body read calls
it allows us to retry POST and PUT requests
2023-05-17 00:37:39 -07:00
Nikolay
953a5df43f
app/vmauth: do not return invalid credentials (#4288)
at http response by default
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4188

based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4190
Thanks @raj-kumar-j  for init implementation
2023-05-17 00:10:51 -07:00
Aliaksandr Valialkin
67beb8c856
lib/storage: follow-up after 2ce02a7fe6
- Document the change at docs/CHANGELOG.md
- Clarify comments for non-trivial code touched by the commit
- Improve the logic behind maybeCreateIndexes():
  - Correctly create per-day indexes if the indexdb rotation is performed during
    the first hour or the last hour of the day by UTC.
    Previously there was a possibility of missing index entries on that day.
  - Increase the duration for creating new indexes in the current indexdb for up to 22 hours
    after indexdb rotation. This should reduce the increased resource usage
    after indexdb rotation.
    It is safe to postpone index creation for the current day until the last hour
    of the current day after indexdb rotation by UTC, since the corresponding (date, ...)
    entries exist in the previous indexdb.
- Search for TSID by (date, MetricName) in both the current and the previous indexdb.
  Previously the search was performed only in the current indexdb. This could lead
  to excess creation of per-day indexes for the current day just after indexdb rotation.
- Search for (date, metricID) entries in both the current and the previous indexdb.
  Previously the search was performed only in the current indexdb. This could lead
  to excess creation of per-day indexes for the current day just after indexdb rotation.
2023-05-16 23:31:59 -07:00
Aliaksandr Valialkin
bc98ea9a8d
lib/storage: reduce the unimportant logging during Storage start / stop
This should improve the visibility of potentially important logs
2023-05-16 15:32:35 -07:00
Artem Navoiev
036ab185bf
update logo width in cluster doc to 300
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-05-16 23:05:30 +02:00
Aliaksandr Valialkin
9af04e4140
vendor: update github.com/VictoriaMetrics/metrics from v1.23.1 to v1.24.0
This change adds process_* metrics to VictoriaMetrics components under Windows OS

See https://github.com/VictoriaMetrics/metrics/pull/47
2023-05-16 11:37:18 -07:00
Aliaksandr Valialkin
4f6680f399
docs/vmbackupmanager.md: run make docs-sync after c7d8dda39225b716ea44df7223db5e4a125d407b 2023-05-16 11:27:39 -07:00
Alexander Marshalov
e895358939
added backup locking/unlocking against retention policy to vmbackupmanager (#558)
* added backup locking/unlocking against retention policy to vmbackupmanager

Signed-off-by: Alexander Marshalov <_@marshalov.org>

* added docs for new commands

Signed-off-by: Alexander Marshalov <_@marshalov.org>

* fix review comments

Signed-off-by: Alexander Marshalov <_@marshalov.org>

---------

Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-05-16 11:23:43 -07:00
Roman Khavronenko
42b90e5e9a
vmalert: follow-up after 669becd011 (#4318)
* vmalert: follow-up after 669becd011

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: follow-up after 669becd011

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: follow-up after 669becd011

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-05-16 10:14:15 -07:00
Roman Khavronenko
afd03f87fe
vmctl: update VictoriaMetrics migration section (#4310)
Remove unnecessary information to simplify the description and tips.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-05-16 10:07:47 -07:00
Artem Navoiev
b4aaeda72f
fix link in operator quick start docs.2
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-05-16 10:06:52 -07:00
Artem Navoiev
c771f4eccf
fix link in operator quick start docs
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-05-16 10:05:36 -07:00
Aliaksandr Valialkin
e6648d14b2
deployment/docker: update base docker image from 3.17.3 to 3.18.0
See https://www.alpinelinux.org/posts/Alpine-3.18.0-released.html
2023-05-12 17:31:35 -07:00
Aliaksandr Valialkin
3b1e40d73f
lib/promutils: properly return error when incorrect Prometheus label names are passed to NewLabelsFromString()
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4284
See also https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4304
2023-05-12 17:02:06 -07:00
Aliaksandr Valialkin
b24da0f901
Revert "lib/promrelabel: show error message if labels not in prometheus exposition format (#4304)"
This reverts commit 193a9c3328.

Reason for revert: the commit doesn't fix the real issue with promutils.NewLabelsFromString()
function, which must return error when improperly formatted Prometheus metric with labels is passed to it.
See https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md#text-format-example

E.g. the promutils.NewLabelsFromString() must return error when the following strings are passed to it:

- `{foo:"bar"}`, since `:` is disallowed in Prometheus text exposition format. The corect value is `{foo="bar"}`
- `{"foo":"bar"}`, since label name shouldn't be quoted. The correct value is `{foo="bar"}`.

The reverted commit introduces another set of bugs, which happily accept the following invalid input:

- `{foo=~"bar"}`
- `{foo!="bar"}`
- `{foo!~"bar"}`

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4284
See also https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4304
2023-05-12 17:01:23 -07:00
Aliaksandr Valialkin
4df7573858
lib/protoparser/csvimport: properly parse the last empty column in CSV line
Do not ignore the last empty column in CSV line.
While at it, properly parse CSV columns in single quotes, e.g. `'foo,bar',baz` is parsed as two columns - `foo,bar` and `baz`

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4048

See also https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4298
2023-05-12 16:59:50 -07:00
Aliaksandr Valialkin
6fd39e2000
Revert "lib/protoparser: fix skip csv line when metric can be collect from the line (#4298)"
This reverts commit 410ae99c2e.

Reason for revert: the commit masks the real issue instead of fixing it.
The real issue is that the scanner.NextColumn() skips the last column if it is empty.

The commit also introduces two bugs:

- a panic if all the metric values in CSV line are empty
- silent import of CSV lines with too small number of columns

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4048
See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4298
2023-05-12 16:59:11 -07:00
Yury Molodov
ff39df74d3
vmui: add notification for non-matching queries (#4301)
vmui: add notification for non-matching queries (#4211)

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4211
2023-05-12 16:58:48 -07:00
Roman Khavronenko
ddc3f0c5c5
docs: mention link to public relabeling playground (#4306)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-05-12 16:57:17 -07:00
Roman Khavronenko
f613d20466
docs: update docs about VMUI pages (#4305)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-05-12 16:56:00 -07:00
Dmytro Kozlov
24386f68db
lib/promrelabel: show error message if labels not in prometheus exposition format (#4304)
lib/promrelabel: show error message if labels not in prometheus exposition format

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4284
2023-05-12 16:53:59 -07:00
Dmytro Kozlov
714236c2d8
lib/protoparser: fix skip csv line when metric can be collect from the line (#4298)
* lib/protoparser: fix skip csv line when metric can be collect from the line

* lib/protoparser: fix comment
2023-05-12 15:53:52 -07:00
Yury Molodov
c530bc639c
vmui: enhancements to top queries page (#4299)
* feat: improvement of the top queries page

* vmui/docs: enhancements to top queries page

* Apply suggestions from code review

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-05-12 15:53:05 -07:00
Aliaksandr Valialkin
b4d919fbc7
docs/CHANGELOG.md: improve the description of the change at 7ea2531db0
Move the change description to the group of vmui changes.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4213
2023-05-11 13:36:34 -07:00
Aliaksandr Valialkin
d21a244641
lib/promutils: properly parse time strings with timezones at ParseTime() 2023-05-11 13:36:00 -07:00
Yury Molodov
55f77dd353
vmui: increase font-size and fix the text display (#4273)
vmui: change default font size to 14px for better readability
vmui: fix bug with missing text on buttons in safari

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-05-11 13:33:06 -07:00
Dmytro Kozlov
fc9e7b650b
app/vmui: added table where Labels with the highest number of unique values show (#4271)
* app/vmui: added Labels with the highest number of unique values

* app/vmui: cleanup

* app/vmui: cleanup

* app/vmui: add table description

* app/vmui: fix comment, updated CHANGELOG.md

* app/vmui: disable links

* app/vmui: added actions to the table, it will show values for selected label with the highest number of series

* app/vmui: fix comment
2023-05-11 13:31:56 -07:00
Aliaksandr Valialkin
e6e30667b1
docs/CHANGELOG.md: fix typo after 2caf0b05c6 2023-05-10 13:05:33 -07:00
Artem Navoiev
6108ef1758
fix typo in changelog
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-05-10 13:05:33 -07:00
Roman Khavronenko
fc623e2b84
Vmalert UI updates (#4276)
* vmalert: expand rule groups on anchor click

before, anchor click was only updating the URL.
To expand the group, user had to click on rule's block.
Now, group will toggle automatically.

* vmalert: allow filtering group in web UI

The new filter allows to filter groups and rules within
groups by: errors only or noMatch only.

The filtering supposed to help navigating big numbers of groups/rules.
Filtering is reflected in URL, so can be shared as a link.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-05-10 13:05:05 -07:00
Dmytro Kozlov
fd35f023dd
docs: Add information about datasource plugin (#4266)
docs: Add information about datasource plugin
2023-05-10 02:24:33 -07:00
Alexander Marshalov
d321ea91f2
fixed typos in documentation and commandline flags descriptions (#4275) 2023-05-10 02:22:06 -07:00
Aliaksandr Valialkin
15618c99af
docs/CHANGELOG.md: add a link to docs about never-firing alerts
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4039
2023-05-09 23:09:37 -07:00
Aliaksandr Valialkin
d2cbb34299
docs/CHANGELOG.md: document 8f4de6fa47 2023-05-09 23:07:22 -07:00
Roman Khavronenko
81c5739529
vminsert: properly reset labels object on aggregation (#4278)
Without reset, labels duplicates could have been added during stream aggregation.
Since `ctx.Labels` is reused during processing of many series, each series will
add its labels to the context. Even if the same labels were already addeded on prev
iteration. Now, we reset `ctx.Labels` on each iteration to contain so labels from
different series didn't interfere.

This could have cause exceeding of the limit on number of labels per pushed time series.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4277

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-05-09 22:59:07 -07:00
Aliaksandr Valialkin
a47b9e55ac
lib/promscrape/discovery/consulagent: substitute metaPrefix with the __meta_consulagent_ plaintext string
This simplifies future code navigation and search for the specific meta-label starting from __meta_consulagent_* prefix.
For example, `grep __meta_consulagent_namespace` finds the exact place where this label is defined.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3953
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4217
2023-05-09 22:58:08 -07:00
Aliaksandr Valialkin
a5a57a7c2a
docs/CHANGELOG.md: group changelog lines for tip release according to VictoriaMetrics apps 2023-05-09 22:55:17 -07:00
Aliaksandr Valialkin
b20bbb8f79
docs/CHANGELOG.md: document baf456978d
See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4223
2023-05-09 22:53:36 -07:00
Aliaksandr Valialkin
dbdf029f6b
docs/managed-victoriametrics/overview.md: typo fix after 51fbf58d89 2023-05-09 22:53:05 -07:00
Aliaksandr Valialkin
7adc1665ba
docs/CHANGELOG.md: refer to the author and the pull request of the notifier_headers feature at vmalert
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3260
2023-05-09 22:52:34 -07:00
Aliaksandr Valialkin
24aa295f43
docs/CHANGELOG.md: document 03150c8973
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4204
2023-05-09 22:50:10 -07:00
Aliaksandr Valialkin
493d8ec7e0
docs/vmalert.md: clarify docs regarding the support of recursive globs
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4041
2023-05-09 22:49:37 -07:00
Aliaksandr Valialkin
e2358d3bd5
docs: clarify docs after 5ee344824f
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4183
2023-05-09 22:49:13 -07:00
Aliaksandr Valialkin
079875a127
app/vmagent/remotewrite: make more user-friendly the warning message about too small -remoteWrite.maxdiskUsagePerURL value
This is a follow-up for bc17f4828c .
While at it, document the change at docs/CHANGELOG.md .

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4195
2023-05-09 22:48:40 -07:00
Aliaksandr Valialkin
f35181a4f8
docs/managed-victoriametrics: consistently use setup-notifications prefix for images used at docs/managed-victoriametrics/setup-notifications.md
This should simplify managment of the images belonging to the given docs.
See docs/assets/README.md for details

This is a follow-up for 4052c44ac1
2023-05-09 22:47:59 -07:00
Aliaksandr Valialkin
22aeeeef3e
app/vmauth: merge default_url example into multi-url example in order to reduce the amounts of text to read for the user
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4084

This is a follow-up for 041e188df8
2023-05-09 22:47:11 -07:00
Aliaksandr Valialkin
fbc28810b1
app/vmselect: small cleanup after 68e31a6000
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3811
2023-05-09 22:43:59 -07:00
Aliaksandr Valialkin
5dbaffe2c6
app/{vmselect,vmctl}: move ParseTime() to lib/promutils
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4091

This is a follow-up for e2053baf32
2023-05-09 22:42:35 -07:00
Aliaksandr Valialkin
3a23d90c3a
docs/CHANGELOG.md: typo fix after 45a551df9c: 'this doc' -> 'this feature request' 2023-05-09 22:42:07 -07:00
Aliaksandr Valialkin
143c5bc936
app/vmauth/README.md: mention about ip filters and concurrency limiter at Security chapter 2023-05-09 22:40:46 -07:00
Aliaksandr Valialkin
fe58dd7e4c
app/vmauth: refer ip_filters option in example auth config
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3491
2023-05-09 22:40:15 -07:00
Aliaksandr Valialkin
e55ddaa962
docs: document IP filters functionality in vmauth
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3491

This is a follow-up for 2f08ed3be2
2023-05-09 22:39:47 -07:00
Aliaksandr Valialkin
8cca9ed0d9
all: update Go builder from Go1.20.3 to Go1.20.4
See https://github.com/golang/go/issues?q=milestone%3AGo1.20.4+label%3ACherryPickApproved
2023-05-09 22:31:08 -07:00
Aliaksandr Valialkin
cb72449fc4
docs/CHANGELOG.md: document c77385e78f 2023-05-09 22:29:23 -07:00
Alexander Marshalov
de68e94c91
fixed vm_promscrape_config_last_reload_successful metric value recovery after successful reloading with unchanged content (#4260) (#4268)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-05-09 22:17:27 -07:00
Roman Khavronenko
54bfacdfde
vmalert: follow-up after cae87da (#4269)
* vmalert: follow-up after cae87da

cae87da4bb
Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: update struct comments

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: rm typo

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-05-09 22:07:26 -07:00
Alexander Marshalov
65f4590403
update generated docs for operator (#4267)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-05-09 22:06:28 -07:00
Roman Khavronenko
4edb97f4da
app/vmalert: detect alerting rules which don't match any series at all (#4198)
app/vmalert: detect alerting rules which don't match any series at all

vmalert starts to understand /query responses which contain object:
```
"stats":{"seriesFetched": "42"}
```
If object is present, vmalert parses it and populates a new field
`SeriesFetched`. This field is then used to populate the new metric
`vmalert_alerting_rules_last_evaluation_series_fetched` and to
display warnings in the vmalert's UI.

If response doesn't contain the new object (Prometheus or
VictoriaMetrics earlier than v1.90), then `SeriesFetched=nil`.
In this case, UI will contain no additional warnings.
And `vmalert_alerting_rules_last_evaluation_series_fetched` will
be set to `-1`. Negative value of the metric will help to compile
correct alerting rule in follow-up.

Thanks for the initial implementation to @Haleygo
See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4056

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4039

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-05-09 21:48:59 -07:00
Roman Khavronenko
3049754575
alerts: update TooHighMemoryUsage threshold (#4256)
It appears that 90% usage for anonymous mem usage
is already concerning. So we lowering the threshold to 80%.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-05-09 21:41:40 -07:00
Zakhar Bessarab
370a421ef4
lib/promscrape/discovery/kubernetes: follow-up for d5e94721db (#4255)
- add changelog reference to an author
- fix tests
- add metadata to match Prometheus behavior

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-05-09 21:29:27 -07:00
Zakhar Bessarab
348693ff84
lib/storage: fix indexdb rotation infinite loop (#4249)
When using `retentionTimezoneOffset` and having local timezone being more than 4 hours different from UTC indexdb retention calculation could return negative value. This caused indexdb rotation to get in loop.
Fix calculation of offset to use `retentionTimezoneOffset` value properly and add test to cover all legit timezone configs.
See:
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4207
- https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4206

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2023-05-09 21:23:01 -07:00
Alexander Marshalov
26fc4afff8
added new consulagent service discovery (#3953) (#4217) 2023-05-08 23:43:59 -07:00
Alexander Marshalov
f7dd084890
max value for memory.allowedPercent changed from 200 to 100 (#4171) (#4251)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-05-08 23:20:56 -07:00
Zakhar Bessarab
98e4265235
docs: changelog follow-up for 49b77ec01a (#4250)
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-05-08 23:20:09 -07:00
Nikolay
7bfa1d7d9e
lib/backup: fixes path generation for windows (#4133)
replaces custom fsync function with standard Fsync methods for files.
fixes pattern matching for parts and properly generate backup path for local fs.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70
2023-05-08 23:16:26 -07:00
Nikolay
5d0299ac19
lib/fs: do not panic at windows at dir deletion (#4132)
Windows doesn't allow to remove dir with opened files. Usually it's a case for snapshots, hard cannot be removed if file is openned.
With this change, dir will be renamed and properly deleted at the next process start.
It's recommended to restart vmstorage/vmsingle for snapshots deletion completion periodically.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70
2023-05-08 23:11:55 -07:00
Zakhar Bessarab
1b06af321f
lib/promscrape/discovery/kubernetes: add common labels to all ports discovered from endpoints (#4235)
* lib/promscrape/discovery/kubernetes: add common labels to all ports discovered from endpoints

Sets
`__meta_kubernetes_endpoints_name` and `__meta_kubernetes_namespace` labels to all ports of pod.
Prometheus sets those labels to all ports in pod (0ab9553611/discovery/kubernetes/endpoints.go (L267C15-L269)) even if port is not matching any service.

See: #4154

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/promscrape/discovery/kubernetes: fix test for updated discovery logic

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-05-08 22:15:37 -07:00
Artem Navoiev
f168da9524
remove information of releasing graphite render api from tip section as we released it in 1.90
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-05-08 22:01:13 -07:00
Artem Navoiev
46886260be
prepare operator docs to migration
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-05-08 21:59:48 -07:00
Artem Navoiev
1e0dcfaae6
update managed docs - prepare for migration
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-05-08 21:58:53 -07:00
Roman Khavronenko
f247160699
docs: note automatic conversion to ms for influx protocol (#4224)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-05-08 21:55:58 -07:00
Artem Navoiev
1e15eea76a
add wight do trobubleshooting docs
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-05-08 21:54:48 -07:00
Artem Navoiev
283a75afba
prepare static docs to migration
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-05-08 21:54:20 -07:00
Roman Khavronenko
8f1372bd43
vmalert: fix API to return non-nil values (#4222)
Properly return empty slices instead of nil for `/api/v1/rules` and `/api/v1/alerts` API handlers.
This improves compatibility with Grafana.

 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4221

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-05-08 21:47:51 -07:00
Roman Khavronenko
c6511bc2d0
Revert "http server: limit max concurrent requests (#4185)" (#4215)
This reverts commit 77f76371

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-05-08 17:22:27 -07:00
Roman Khavronenko
1c3bf0d0d8
app/vmalert: follow-up after 6c322b4a00 (#4214)
6c322b4a00

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-05-08 17:20:49 -07:00
Zakhar Bessarab
19eaf17e11
app/vmalert: add support of recursive path globs for rules and templates (#4148)
Supports using `**` for `-rule` and `-rule.templates`: `dir/**/*.tpl` loads contents of dir and all subdirectories recursively.

See: #4041

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Artem Navoiev <tenmozes@gmail.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2023-05-08 16:22:30 -07:00
Nikolay
cfa058dfec
lib/promscrape: adds filter for consul_sd_configs: (#4184)
* lib/promscrape: adds filter for consul_sd_configs:
it allows advanced filtering for consul service discovery requests
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4183

* typo fix

* removes deprecation mentions since it's not relevant

* Update docs/CHANGELOG.md

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-05-08 16:14:15 -07:00
Zakhar Bessarab
55d772ab39
app/vmalert: return an error when using query function in -external.alert.source flag (#4191)
Templating of `-external.alert.source` is not expected to have access to the query which was causing runtime error when query function was passed as nil.
See: #4181

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-05-08 15:48:16 -07:00
Dmytro Kozlov
f425123116
app/vmagent,lib/persistentqueue: show warning message if --remoteWrite.maxDiskUsagePerURL flag lower than 500MB (#4196)
* app/vmagent,lib/persistentqueue: show warning message if `--remoteWrite.maxDiskUsagePerURL` flag lower than 500MB

* app/vmagent,lib/persistentqueue: linter fix

* app/vmagent,lib/persistentqueue: fix comment
2023-05-08 15:45:21 -07:00
dmitryk-dk
f1717aa818
docs: fix type 2023-05-08 15:33:21 -07:00
dmitryk-dk
6168e7f95f
app/vmctl: fix comments 2023-05-08 15:32:24 -07:00
dmitryk-dk
aef925ea8a
app/vmctl: fix comments 2023-05-08 15:31:45 -07:00
dmitryk-dk
c2b58fd954
docs/managed-victoriametrics: change emails 2023-05-08 15:31:01 -07:00
dmitryk-dk
dbee230911
docs/managed-victoriametrics: add notifications setup 2023-05-08 15:30:21 -07:00
Alexander Marshalov
402d906d40
added default_url field in vmauth users config (#4084) (#4156)
* added default url field in vmauth users config (#4084)

---------

Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-05-08 15:13:46 -07:00
Yury Molodov
ddc5197bce
vmui: add metric relabel debug (#3889)
* feat: add metric relabel debug (#3807)

* fix: add link to relabeling cookbook

* lib/promrelabel: merge, fix conflicts

* lib/promrelabel: fix diff

* docs/vmui: add metric relabel playground

---------

Co-authored-by: dmitryk-dk <kozlovdmitriyy@gmail.com>
2023-05-08 14:59:35 -07:00
Alexander Marshalov
42b1e6443b
changelog for issue #4083 (#4197)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-05-08 14:41:39 -07:00
Yury Molodov
e0434f7b4e
vmui: display heatmap in the Explore Metrics (#4124)
* feat: display heatmap in the explore metrics (#4111)

* fix: correct calc step for heatmap

* fix: remove spaces in the result of getDurationFromMilliseconds
2023-05-08 14:40:38 -07:00
Yury Molodov
3c45256736
vmui: add a comparison of data to the Cardinality Explorer (#4123)
* feat: add button "show today" to date picker

* feat: add comparison with the prev day (#3967)

* vmui/docs: add comparison of data to cardinality page
2023-05-08 14:39:14 -07:00
Yury Molodov
382a2ff649
docs/vmui: fix CHANGELOG.md about WITH templates (#4194) 2023-05-08 14:37:08 -07:00
Yury Molodov
de35cbf251
vmui: Integrate WITH template playground (#3831)
* feat: add WithTemplate page

* app/vmselect/prometheus: enable json mode for expand with expr API

* app/vmselect/prometheus: enable CORS and add content type

* feat: add api for expand with templates

* fix: remove console from useExpandWithExprs

* app/vmselect/prometheus: fix escaping

* vmui:  integrate WITH template

* app/vmctl: check content type instead of form param

* fix: add content-type for fetch with-exprs

* fix: add a header to the server's response that allows the "Content-Type" header

* app/vmctl: added comment and cleanup

* app/vmctl: use format query param

---------

Co-authored-by: dmitryk-dk <kozlovdmitriyy@gmail.com>
2023-05-08 14:35:35 -07:00
Dmytro Kozlov
3da856af0c
app/vmctl: add support for the different time format in the native binary protocol (#4189)
* app/vmctl: add support for the different time format in the native binary protocol

* app/vmctl: update flag description, update CHANGELOG.md

* app/vmctl: add comment to exported function
2023-05-08 14:21:59 -07:00
Alexander Marshalov
574a0559d5
added unauthorized_user field in vmauth users config (#4083) (#4157)
added `unauthorized_user` field in vmauth users config (#4083)

---------

Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-05-08 13:36:54 -07:00
Roman Khavronenko
20b025dc88
http server: limit max concurrent requests (#4185)
* lib/httpserver: introduce `-http.maxConcurrentRequests` command-line flag

Introduce `-http.maxConcurrentRequests` command-line flag to protect
VM components from resource exhaustion during unexpected spikes of HTTP requests.
By default, the new flag's value is set to 0 which means no limits are applied.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* lib/httpserver: mention http.maxConcurrentRequests in docs

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-05-08 13:13:58 -07:00
Alexander Marshalov
7faf45d7f9
doc improvements (#4172) (#4186)
- added info about metric `vm_vminsert_metrics_read_total`,
- small doc refactoring
- and added make-command for running docs in docker.

Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-05-08 13:11:57 -07:00
Yury Molodov
23ed0467a2
vmui: fix freeze when query regular with heatmap query (#4093)
* fix: fix freeze when query regular with heatmap query

* vmui/docs: fix freeze when query regular with heatmap query
2023-05-08 13:09:49 -07:00
Aliaksandr Valialkin
45ec3ac158
docs: document IP filters functionality in vmauth
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3491

This is a follow-up for 2f08ed3be2
2023-05-08 12:14:05 -07:00
Yury Molodov
46765c677d
vmui: fix bug where tenant list was not displayed (#4162)
* fix: modify the condition for querying tenants

* fix: change getTenantIdFromUrl output to string
2023-05-08 11:00:19 -07:00
Artem Navoiev
624e13a1d6
move note about opensource of graphite in v1.90 release note
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-05-08 09:56:53 -07:00
Roman Khavronenko
e9ce67adb8
vmalert: retry datasource requests with EOF or unexpected EOF errors (#4146)
* vmalert: retry datasource requests with EOF or unexpected EOF errors

Retry failed read request on the closed connection one more time.
This may improve rules execution reliability when connection
between vmalert and datasource closes unexpectedly.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: fix old tests

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-05-08 09:49:49 -07:00
Zakhar Bessarab
79ee1749a1
lib/httpserver: add handler to serve /robots.txt and deny search indexing (#4143)
This handler will instruct search engines that indexing is not allowed for the content exposed to the internet. This should help to address issues like #4128 when instances are exposed to the internet without authentication.
2023-05-08 09:46:34 -07:00
Balamurugan Krishnamoorthy (Bala)
2a45a85c35
Removed duplicate third-party article reference (#4142)
"How do We Keep Metrics for a Long Time in VictoriaMetrics" article is referenced twice in "Third-party articles and slides about VictoriaMetrics" section
2023-05-08 09:44:19 -07:00
Artem Navoiev
dcac82638e
add graphite render api opensource to changelog
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-05-08 09:42:38 -07:00
Aliaksandr Valialkin
cd99f1fb3a
docs/CHANGELOG.md: document c77385e78f 2023-05-08 08:57:49 -07:00
Aliaksandr Valialkin
d544bfd73e
vendor: update github.com/VictoriaMetrics/metricsql from v0.56.1 to v0.56.2
This fixes panic when the duration in the query contains `M` suffix.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4120
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3589
2023-04-14 14:06:47 -07:00
Aliaksandr Valialkin
2fd9fdacce
docs/vmctl.md: run make docs-sync after 2a5b9ff782 2023-04-13 23:54:50 -07:00
Aliaksandr Valialkin
e6d10106f7
docs/CHANGELOG.md: move the bugfix description into the correct place
This is a follow-up for 2a5b9ff782

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4092
2023-04-13 23:52:01 -07:00
Dmytro Kozlov
10351a9dbb
app/vmctl: fix performance degradation, add flag to disable backoff policy (#4097)
* app/vmctl: change api for getting metric names

* app/vmctl: fix tests

* app/vmctl: add flag to enable backoff policy, fix test, performance improvements

* app/vmctl: use one http client

* app/vmctl: made linter happy

* app/vmctl: updated documentation and CHANGELOG.md

* app/vmctl: cleanup

* app/vmctl: rename flag

* app/vmctl: cleanup

* app/vmctl: fix comments

* app/vmctl: fix metrics parser problem, improve tests
2023-04-13 23:51:11 -07:00
Aliaksandr Valialkin
7fb2b14ca0
docs/CHANGELOG.md: run at least 4 background mergers on systems with less than 4 CPU cores
This reduces the probability of sudden spike in the number of small parts when all the background mergers
are busy with big merges.
2023-04-13 23:37:05 -07:00
Aliaksandr Valialkin
cf53ce83a0
app/vmstorage: deprecate -bigMergeConcurrency command-line flag
Improperly configured -bigMergeConcurrency command-line flag usually leads to uncontrolled
growth of unmerged parts, which, in turn, increases CPU usage and query durations.

So it is better deprecating this flag. In rare cases -smallMergeConcurrency command-line flag
can be used instead for controlling the concurrency of background merges.
2023-04-13 20:42:22 -07:00
Aliaksandr Valialkin
e95e401e4d
app/vmbackupmanager: sync with enterprise-single-node branch after 41a54c775891c87e3d5ed59ff0769c869dd2fe71 2023-04-13 19:38:28 -07:00
Alexander Marshalov
e0a1b62569
Added extra information to docs about total output in stream aggregation. (#4130)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-04-13 19:37:05 -07:00
Zakhar Bessarab
217eea6e15
lib/backup/actions: store metadata(creation and completion time) in backup files (#4117)
This makes it easier to understand exact point in time which is included in this backup.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-04-13 19:20:34 -07:00
Aliaksandr Valialkin
b33dd2e577
docs/Single-server-VictoriaMetrics.md: document automatic switch from graph view to heatmap view for histogram buckets
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3384
2023-04-10 10:52:24 -07:00
Aliaksandr Valialkin
6c7087f65e
docs/Troubleshooting.md: mention that it is recommended to use default command-line flag values for VictoriaMetrics components 2023-04-10 10:52:10 -07:00
Aliaksandr Valialkin
b21ac56894
docs/guides: update VictoriaMetrics from v1.89.1 to v1.90.0 2023-04-06 19:18:14 -07:00
Aliaksandr Valialkin
2bbd8754bb
docs/CHANGELOG.md: formatting fix 2023-04-06 19:18:14 -07:00
Aliaksandr Valialkin
d15618f2ef
docs/CHANGELOG.md: cut v1.90.0 release 2023-04-06 16:16:52 -07:00
Dmytro Kozlov
8ec5e7f53a
app/vmctl: add multiple filters defined in --vm-native-filter-match flag to discovered metric names (#4063)
* app/vmctl: add multiple filters defined in `--vm-native-filter-match` flag to discovered metric names

* app/vmctl: fix comments

* app/vmctl: move function buildMatchWithFilter to the correct place

* app/vmctl: update CHANGELOG.md

* app/vmctl: fix CI, remove error wrapping

* app/vmctl: fix CI, simplify `Set()`
2023-04-06 15:11:53 -07:00
Aliaksandr Valialkin
ff1891397a
docs/CHANGELOG.md: document the bugfix, which remove unneeded logger.Errorf() call during stream aggregation with the enabled deduplication
This is a follow-up for ff72ca14b9
2023-04-06 15:01:25 -07:00
Aliaksandr Valialkin
eca03d1f55
docs/CHANGELOG.md: document that VictoriaMetrics for Windows cannot delete snapshots
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70#issuecomment-1491529183
2023-04-06 03:16:20 -07:00
Aliaksandr Valialkin
c43dc072c4
docs/CHANGELOG.md: document v1.79.12 2023-04-06 03:10:12 -07:00
Aliaksandr Valialkin
c92731de93
docs/CHANGELOG.md: document v1.87.5 2023-04-06 00:44:30 -07:00
Timur Bakeyev
f21ea62ae1
Fix cut-n-paste error (#4079)
It seems that VMServiceScrape description was c-n-p from vmselect one into all other resources.
2023-04-06 00:44:29 -07:00
Zakhar Bessarab
0384b83736
docs: fix example operator spec for vmbackupmanager restore usage (#4074)
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-04-05 22:17:05 -07:00
Timur Bakeyev
210216a1e2
Fix reference to the imagepullsecrets description (#4080)
Looks like the original document has moved to the https://kubernetes.io/docs/concepts/containers/images/#referring-to-an-imagepullsecrets-on-a-pod.
Alternatively, it could be that https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ describes the meaning of the parameter in more detail.
2023-04-05 19:57:22 -07:00
Aliaksandr Valialkin
ac2fbf0a9c
docs/Troubleshooting.md: add missing help word after c7bcf750c2d031b1259cd8115d7464b67f40cb9eg 2023-04-05 14:28:19 -07:00
Aliaksandr Valialkin
e74c54aa86
docs/Troubleshooting.md: another typo fixes after c7bcf750c2d031b1259cd8115d7464b67f40cb9eg 2023-04-05 14:15:00 -07:00
Aliaksandr Valialkin
cd2ed803c2
docs/Troubleshooting.md: fix formatting after c7bcf750c2 2023-04-05 13:46:26 -07:00
Aliaksandr Valialkin
775eb1245d
docs/Troubleshooting.md: fix a typo in the link after c7bcf750c2 2023-04-05 13:40:32 -07:00
Aliaksandr Valialkin
52c46f49e1
all: update Go builder from Go1.20.2 to Go1.20.3
See https://github.com/golang/go/issues?q=milestone%3AGo1.20.3+label%3ACherryPickApproved
2023-04-05 13:38:44 -07:00
Artem Navoiev
c5397c0b77
update changelog
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-04-05 13:31:48 -07:00
Zakhar Bessarab
54edd6992a
app/vmalert: update Grafana URLs to match latest format (#4061)
See: #4019

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-04-05 13:31:06 -07:00
Artem Navoiev
1b05896ac5
fix closing divs in docs
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-04-05 13:30:14 -07:00
Aliaksandr Valialkin
ca4f94c704
docs/Troubleshooting.md: add General troubleshooting checklist
This checklist helps searching for the infromation related to some issue / question
about VictoriaMetrics
2023-04-05 13:29:12 -07:00
Aliaksandr Valialkin
286697b8b6
docs/CHANGELOG.md: document edb45d7fc1
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4013
2023-04-02 21:26:28 -07:00
Aliaksandr Valialkin
02ceebccc0
lib/promscrape: do not re-use previously loaded scrape targets on failed attempt to load updated scrape targets at file_sd_configs
The logic employed for re-using the previously loaded scrape target was broken initially.
The commit cc0427897c tried to fix it, but the new logic
became too complex and fragile. So it is better to just remove this logic,
since the targets from temporarily broken file should be eventually loaded on next
attempts every -promscrape.fileSDCheckInterval

This also allows removing fragile hacks around __vm_filepath label.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3989
2023-04-02 21:11:12 -07:00
Dmytro Kozlov
6f0512a81c
lib/promscrape: fix the problem with scrape work duplicates when file_sd_config can't be read (#4027)
* lib/promscrape: fix the problem with scrape work duplicates when file_sd_config can't be read

* lib/promscrape: clarified comment

* lib/promscrape: made better approach to handle a problem with growing []*ScrapeWork on each error when loading config

* lib/promscrape: added CHANGELOG.md

* Update docs/CHANGELOG.md

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-04-02 21:11:10 -07:00
Yury Molodov
17b4fc9470
vmui: tips for working with the graph and legend (#4045)
* feat: add tips for working with the graph and legend

* feat: add the ability to collapse the legend

* vmui/docs: add the ability to collapse the legend

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-04-01 00:41:01 -07:00
Roman Khavronenko
5f95f9d453
lib/storage: check for free disk space before opening tables (#4035)
* lib/storage: check for free disk space before opening tables

We check for free disk space before call to `openTable`,
so `Storage` can be set to ReadOnly before mergeWorkers start.

Before the change, there was a chance that merges will start
even if Storage has to start in ReadOnly mode because of
`-storage.minFreeDiskSpaceBytes` limit.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4023
Signed-off-by: hagen1778 <roman@victoriametrics.com>

* lib/storage: chore

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Update lib/storage/storage.go

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-03-31 23:50:56 -07:00
Aliaksandr Valialkin
db8fda4ec6
app/vmselect/graphite: open source Graphite Render API 2023-03-31 23:37:40 -07:00
Aliaksandr Valialkin
43b431d322
deployment/docker: update base Docker image from Alpine 3.17.2 to Alpine 3.17.3
This fixes security issues from https://alpinelinux.org/posts/Alpine-3.17.3-released.html

This is a follow-up for 59c350d0d2
2023-03-31 22:54:48 -07:00
Aliaksandr Valialkin
dad13c0a91
lib/streamaggr: follow-up for ff72ca14b9
- Make sure that the last successfully loaded config is used on hot-reload failure
- Properly cleanup resources occupied by already initialized aggregators
  when the current aggregator fails to be initialized
- Expose distinct vmagent_streamaggr_config_reload* metrics per each -remoteWrite.streamAggr.config
  This should simplify monitoring and debugging failed reloads
- Remove race condition at app/vminsert/common.MustStopStreamAggr when calling sa.MustStop() while sa
  could be in use at realoadSaConfig()
- Remove lib/streamaggr.aggregator.hasState global variable, since it may negatively impact scalability
  on system with big number of CPU cores at hasState.Store(true) call inside aggregator.Push().
- Remove fine-grained aggregator reload - reload all the aggregators on config change instead.
  This simplifies the code a bit. The fine-grained aggregator reload may be returned back
  if there will be demand from real users for it.
- Check -relabelConfig and -streamAggr.config files when single-node VictoriaMetrics runs with -dryRun flag
- Return back accidentally removed changelog for v1.87.4 at docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3639
2023-03-31 22:54:10 -07:00
Zakhar Bessarab
743d1e6536
docs/vmctl: add examples of URLs used for migration in different modes (#4042)
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-03-31 22:44:12 -07:00
Roman Khavronenko
0bde9722ed
vmalert: use missingkey=zero for templating (#4040)
Replace empty labels with "" instead of "<no value>"
during templating, as Prometheus does.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4012

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-03-31 22:43:39 -07:00
Zakhar Bessarab
46c8be4f98
lib/fs: verify response code when reading configuration over HTTP (#4036)
Verifying status code helps to avoid misleading errors caused by attempt to parse unsuccessful response.

Related issue: #4034

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-03-31 22:33:53 -07:00
Daria Karavaieva
83213e6786
Vmanomaly guide index fix (#4029)
* name and scrutture change

* fix indexing

* index fix

* name change

* line separator fix
2023-03-31 22:32:15 -07:00
Alexander Marshalov
8c14d17694
added hot reload support for stream aggregation configs (#3969) (#3970)
added hot reload support for stream aggregation configs (#3969)

Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-03-31 22:31:38 -07:00
Eliran Barnoy
7c70fb0fb9
Fix operator links to include VMPrometheusConverter for added visibility 2023-03-30 12:33:04 -07:00
Aliaksandr Valialkin
85ca077a88
lib/flagutil: ArrayString: support commas inside quoted strings and inside [], {} and () braces
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3915
2023-03-28 21:25:07 -07:00
Aliaksandr Valialkin
957d64e302
docs: mention that VictoriaMetrics rounds time range to UTC days at /api/v1/labels, /api/v1/label/.../values and /api/v1/series handlers
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3107
2023-03-28 17:01:09 -07:00
Aliaksandr Valialkin
6f5bbf096a
app/vmagent/remotewrite: cosmetic updates after f3a51e8b1d
- Compare directory names instead of paths to directory when determining which persistent queues must be deleted
  This is less error-prone solution, since paths to the same directory can differ, which could lead
  to accidental directory removal for the existing -remoteWrite.url

- Log the `removed %d dangling queues` message when at least a single queue has been removed

- Consistently use filepath.Join() for creating paths to persistent queues.
  This is needed for Windows support (see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70 )

- Clarify the description of the change at docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4014
2023-03-27 18:38:53 -07:00
Zakhar Bessarab
6ed6eb0c4c
app/vmagent: add -remoteWrite.removeDanglingQueues flag (#4017)
* app/vmagent: add `-remoteWrite.removeDanglingQueues` flag which allows to automatically remove dangling persistent queue contents

Related issue: #4014

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vmagent: address review feedback

- remove persistent queues files by default
- rename `remoteWrite.removeDanglingQueues` to `remoteWrite.keepDanglingQueues`
- update docs to reflect changed behaviour

Related issue: #4014

* Apply suggestions from code review

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-03-27 18:38:51 -07:00
Aliaksandr Valialkin
54b9537a76
app/vmselect/promql: follow-up for 79e1c6a6fc
- Document the fix at docs/CHANGELOG.md
- Add tests with multiple adjancent zero buckets
- Simplify the fix a bit

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/296
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4021
2023-03-27 18:04:30 -07:00
Yury Molodov
86a98fa131
vmui: heatmap (#3780)
* fix: add stroke and font for all axes

* feat: add util for generate gradient

* feat: add heatmap plugin

* feat: add heatmap legend

* feat: add heatmap graph (#3384)

* vmui: add heatmap graph (#3384)

* feat: add convert Prometheus to VictoriaMetrics histogram

* fix: prevent re-render graph

* feat: reset step for heatmap

* feat: normalize heatmap data

* fix: format heatmap legend

* wip

* app/vmselect/vmui: run `make vmui-update`

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-03-26 00:31:21 -07:00
Aliaksandr Valialkin
229b39ac7d
docs/MetricsQL.md: quote min, max and avg args for rollup_*() functions in order to reduce the level of confusion when users try to pass the second argument to these functions 2023-03-26 00:02:01 -07:00
Aliaksandr Valialkin
84b3b0fac2
docs/CHANGELOG.md: document v1.87.4 LTS release 2023-03-25 22:44:21 -07:00
Aliaksandr Valialkin
3698994953
app/{vmbackup,vmrestore}: publish vmbackup and vmrestore binaries for Windows
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70
2023-03-25 15:09:41 -07:00
Aliaksandr Valialkin
7aff6f872f
app/vmselect/promql: follow-up for 7205c79c5a
- Allocate and initialize seriesByWorkerID slice in a single go instead
  of initializing every item in the list separately.
  This should reduce CPU usage a bit.
- Properly set anti-false sharing padding at timeseriesWithPadding structure
- Document the change at docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3966
2023-03-24 23:39:43 -07:00
Zakhar Bessarab
3f38ed3171
app/vmbackup: delete created snapshot in case of error during backup (#4008)
Related issue: #2055

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-03-24 22:09:17 -07:00
Aliaksandr Valialkin
181e877092
docs/CHANGELOG.md: cosmetic fixes: remove trailing whitespace and consistently use -flag instead of --flag 2023-03-24 17:57:23 -07:00
Alexander Marshalov
0301b5018e
allowed using dashes and dots in environment variables names (#4009)
* allowed using dashes and dots in environment variables names for templating config files with envtemplate (#3999)

Signed-off-by: Alexander Marshalov <_@marshalov.org>

* Apply suggestions from code review

---------

Signed-off-by: Alexander Marshalov <_@marshalov.org>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-03-24 17:57:19 -07:00
Aliaksandr Valialkin
c54b8acba2
docs/vmauth.md: follow-up for 36edba9bfb
- Document `-configCheckInterval` command-line flag in `quick start` section
- Clarify the addition of `-configCheckInterval` at docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3990
2023-03-24 17:56:59 -07:00
Aliaksandr Valialkin
9d19c2d89b
docs/vmagent.md: clarify that there is no need to specify multiple -remoteWrite.url options when writing data to a single VictoriaMetrics cluster when data replication is needed
Also add a link to https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format from `getting started` section,
so users could quickly find how to write data to VictoriaMetrics cluster
2023-03-24 17:56:31 -07:00
Roman Khavronenko
ec6a20880c
vmalert: mention VMUI example for alert's source (#4005)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-03-24 17:55:30 -07:00
Roman Khavronenko
b866f334ed
docs: mention cluster URL for exporting series (#4002)
docs: mention cluster URL for exporting series

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-03-24 17:54:57 -07:00
Daria Karavaieva
53c3d6137f
Vmanomaly guide (#3834)
Setting up VMAnomaly on NodeExporter metrics with VictoriaMetrics and AlertManager.


* vmanomaly-guide-draft

* aletr graphs and description

* readme vmanomaly tutorial

* Added back fit_every param for performance

* vmanomaly guide fixes

* added spaces div

* spaces + resize image

* alert example grammar

* quotation marks

* docker link

* typo fixed

* more links

* reader section rephrased

* label change

* lower case for grafana service

* lower case for vm service

* yaml markdown

---------

Co-authored-by: Dima Lazerka <dima@victoriametrics.com>
2023-03-24 17:54:10 -07:00
Dmytro Kozlov
fe7acd5e8a
docs: follow up after dc2c712a29 (#4001) 2023-03-24 16:41:42 -07:00
Yury Molodov
5f77efa915
vmui: display errors for each query individually (#3987) (#3994) 2023-03-24 13:26:43 -07:00
Alexander Marshalov
b5027cff9c
added configCheckInterval flag for vmauth (#3990) (#3991)
* added configCheckInterval flag for vmauth (#3990)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-03-24 13:25:07 -07:00
Nikolay
9bb83cafa4
lib/netutil: log only parsing errors for proxy-protocol (#3985)
* lib/netutil: log only parsing errors for proxy-protocol

Previosly every error was logged. With configured TCP health checks at load-balancer or kubernetes, vmauth spams a lot of false positive error message into logs

* Update docs/CHANGELOG.md

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* Update lib/netutil/tcplistener.go

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-03-21 10:23:08 -07:00
Aliaksandr Valialkin
8ed9295109
docs/vmagent.md: mention in docs that the target relabel debug page shows target url now 2023-03-20 22:20:13 -07:00
Dmytro Kozlov
85b01c4aa7
lib/promrelabel: make target url from labels on target relabel page (#3882)
* lib/promrelabel: make target url from labels on target relabel page

* wip

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-03-20 22:08:39 -07:00
Aliaksandr Valialkin
1580d8bda7
docs/CHANGELOG.md: cosmetic fixes 2023-03-20 14:33:46 -07:00
Aliaksandr Valialkin
531b35b6c0
docs/Troubleshooting.md: document an additional case, which could result in slow inserts
If `-cacheExpireDuration` is lower than the interval between ingested samples for the same time series,
then vm_slow_row_inserts_total` metric is increased.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3976#issuecomment-1476883183
2023-03-20 14:33:27 -07:00
Dmytro Kozlov
aed59b9029
app/vmctl: automatically check tty (#3938)
app/vmctl: automatically detect if TTY is available
2023-03-20 14:14:43 -07:00
Yury Molodov
95b60c2777
vmui: support for drag'n'drop in the "Trace analyzer" page (#3971)
vmui: add drag-and-drop support for the trace analyzer page
2023-03-20 14:09:45 -07:00
Yury Molodov
b66953d8e1
vmui: improve usability of date/time picker (#3968)
* vmui: allow manually set input date and time
* vmui/docs: improve usability of date/time picker
2023-03-20 13:57:47 -07:00
Aliaksandr Valialkin
fc3d826d7f
all: add Windows build for VictoriaMetrics
This commit changes background merge algorithm, so it becomes compatible with Windows file semantics.

The previous algorithm for background merge:

1. Merge source parts into a destination part inside tmp directory.
2. Create a file in txn directory with instructions on how to atomically
   swap source parts with the destination part.
3. Perform instructions from the file.
4. Delete the file with instructions.

This algorithm guarantees that either source parts or destination part
is visible in the partition after unclean shutdown at any step above,
since the remaining files with instructions is replayed on the next restart,
after that the remaining contents of the tmp directory is deleted.

Unfortunately this algorithm doesn't work under Windows because
it disallows removing and moving files, which are in use.

So the new algorithm for background merge has been implemented:

1. Merge source parts into a destination part inside the partition directory itself.
   E.g. now the partition directory may contain both complete and incomplete parts.
2. Atomically update the parts.json file with the new list of parts after the merge,
   e.g. remove the source parts from the list and add the destination part to the list
   before storing it to parts.json file.
3. Remove the source parts from disk when they are no longer used.

This algorithm guarantees that either source parts or destination part
is visible in the partition after unclean shutdown at any step above,
since incomplete partitions from step 1 or old source parts from step 3 are removed
on the next startup by inspecting parts.json file.

This algorithm should work under Windows, since it doesn't remove or move files in use.
This algorithm has also the following benefits:

- It should work better for NFS.
- It fits object storage semantics.

The new algorithm changes data storage format, so it is impossible to downgrade
to the previous versions of VictoriaMetrics after upgrading to this algorithm.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3236
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3821
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70
2023-03-19 23:28:26 -07:00
Aliaksandr Valialkin
d2f85816ea
lib/{mergeset,storage}: prevent from long wait time when creating a snapshot under high data ingestion rate
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3551
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3873
2023-03-19 00:19:02 -07:00
Alexander Marshalov
31005f4980
updated api doc for operator (#3972)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-03-17 16:17:17 -07:00
Zakhar Bessarab
c397e56769
doc/vmgateway-grafana-openid-guide: fix formatting, add reproducible example and example results (#3964) 2023-03-17 16:15:45 -07:00
Aliaksandr Valialkin
c9e30dd892
all: typo fix in the same way as in e566d49e3a: 8248 -> 8428 2023-03-16 22:08:26 -07:00
Artem Navoiev
7e6be18c3d
managed quickstart fix typo
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-03-16 22:04:03 -07:00
Artem Navoiev
208aa85362
fix anchor after chaning manager quick start
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-03-16 22:03:46 -07:00
Nikolay
a61106d965
Vmagent kafka updates (#535)
* app/vmagent: allow vm proto for kafka consumer and producer
it should reduce network usage up to 50%.
According to benchmarks without any encoding at kafka topic, it reduces traffic up to 50%.
With enabled zstd at kafka topic, it shows no diffence in traffic. So it
doesn't make much sense to use it.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225

* mention eb61a7dd68b834b08d01727a918f207700348ada at changelog

* app/vmagent: bumps kafka lib version
it allows compiling vmagent for arm64 machines
fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2271

* mention d19b1a888248c96cfd7ccee00ba6f596d89be1d7 at change log

* app/vmagent: adds natural concurrency for kafka consumer
it should improve performance for data consumption
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1957

* mention change 0c143bb22ca2e7e0b7eec9bc84a94ee2b41626ca

* Update app/vmagent/kafka/consumer.go

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* Update app/vmagent/kafka/consumer_cgo.go

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-03-15 13:03:51 -07:00
Alexander Marshalov
3f10f19842
updated vars doc for operator (#3960)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-03-15 12:19:53 -07:00
dmitryk-dk
1ff82e23af
docs: cleanup 2023-03-15 12:19:16 -07:00
dmitryk-dk
e1c2c91bb6
app/vmctl: update managed quickstart guide 2023-03-15 12:18:48 -07:00
Zakhar Bessarab
a3f364869f
doc: add guide for vmgateway configuration with OpenID and Grafana (#3951)
docs: add guide for vmgateway configuration with OpenID and Grafana
2023-03-14 16:11:29 -07:00
Zakhar Bessarab
d1d108fe77
lib/storage: log original labels set when label value is truncated (#3952)
lib/storage: log original labels set when label value is truncated
2023-03-14 16:11:02 -07:00
Aliaksandr Valialkin
9aa6b63160
docs/Single-server-VictoriaMetrics.md: clarify that the cache directory can be removed manually when VictoriaMetrics is stopped 2023-03-13 00:24:01 -07:00
Aliaksandr Valialkin
d345619424
docs/CHANGELOG.md: document v1.87.3 2023-03-13 00:21:00 -07:00
Aliaksandr Valialkin
1ae73a5fe5
docs/CHANGELOG.md: document v1.79.11 LTS release 2023-03-12 23:23:05 -07:00
Aliaksandr Valialkin
47c5b77c8c
docs/CHANGELOG.md: cut v1.89.1 2023-03-12 19:14:29 -07:00
Aliaksandr Valialkin
3b4a3583bc
app/vmselect/promql: prevent from cannot unmarshal timeseries from rollupResultCache panic after the upgrade to v1.89.0 2023-03-12 19:09:11 -07:00
Aliaksandr Valialkin
a70b517d92
docs/CHANGELOG.md: cut v1.89.0 2023-03-12 17:29:50 -07:00
Aliaksandr Valialkin
a6a4beb89a
app/vmselect: remove data race on updating EvalConfig.IsPartialResponse from concurrently running goroutines
This properly returns `is_partial: true` for partial responses.
2023-03-12 16:53:03 -07:00
Artem Navoiev
0186c7fdfb
fix typos on image
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-03-12 16:32:37 -07:00
Aliaksandr Valialkin
5cd60c54d3
app/vmselect/promql: prevent from SIGBUS crash on architecures, which deny unaligned access to 8-byte words (e.g. ARM)
Thanks to @oliverpool for nailing down the root cause of the issue and for the initial attempt to fix it
at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3927
2023-03-12 16:29:18 -07:00
Yury Molodov
2a1bc14984
vmui: remove send step param for instant queries (#3931)
* fix: remove step param for instant queries (#3896)

* vmui: remove send step param for instant queries

* Update docs/CHANGELOG.md

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-03-12 03:14:05 -07:00
Aliaksandr Valialkin
6de1d034c8
docs/CHANGELOG.md: document 113a89904d 2023-03-12 01:58:02 -08:00
Aliaksandr Valialkin
97479bd9e3
docs/CHANGELOG.md: yet another typo fix 2023-03-12 01:52:17 -08:00
Aliaksandr Valialkin
22e8bd7222
docs/CHANGELOG.md: typo fix 2023-03-12 01:04:23 -08:00
Aliaksandr Valialkin
e491fee1f4
app/vmselect/netstorage: do not intern string representation of MetricName for time series received from vmstorage
It has been appeared that this interning may lead to increased memory usage and increased CPU usage
when vmselect performs queries, which select big number of time series.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3692
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3863
2023-03-12 00:44:08 -08:00
Aliaksandr Valialkin
d106ababb8
docs/CHANGELOG.md: document 927d9da270 2023-03-12 00:28:00 -08:00
Aliaksandr Valialkin
5ac8f89d2a
docs/CHANGELOG.md: typo fixes 2023-03-12 00:28:00 -08:00
Aliaksandr Valialkin
868970bd9e
docs/CHANGELOG.md: document c9f44daaee8f4282d9ed41e3ba799c7a33841313 2023-03-12 00:27:33 -08:00
Roman Khavronenko
8b907fffdb
vmalert: support concurrent reading from object storage (#532)
* vmalert: support concurrent reading from object storage

Config reading from GCS or S3 can be slow if object storage
contains a big number of files. Object storages are usually
fast for downloading and are slow for individual operations.
If there would be thousands of files to read, vmalert could
spend significant time for retrieving those because it is
done sequentially.

The change introduces ability to read configs from object
storage concurrently. By default, both GCS and S3 are now
read with 50 concurrent readers. This significantly reduces
the load time:
* loading 500 files with concurrency=1 takes 27s
* loading 500 files with concurrency=50 takes <1s

* vmalert: add note to Changelog

* vmalert: cleanup

* vmalert: use ticker properly

* app/vmalert: improve status reporting during config loading

* vmalert: support concurrent reading from object storage

Config reading from GCS or S3 can be slow if object storage
contains a big number of files. Object storages are usually
fast for downloading and are slow for individual operations.
If there would be thousands of files to read, vmalert could
spend significant time for retrieving those because it is
done sequentially.

The change introduces ability to read configs from object
storage concurrently. By default, both GCS and S3 are now
read with 50 concurrent readers. This significantly reduces
the load time:
* loading 500 files with concurrency=1 takes 27s
* loading 500 files with concurrency=50 takes <1s

* app/vmalert: make linter happy
2023-03-12 00:25:34 -08:00
Alexander Marshalov
78e9e4742d
Stream aggregation doc improvements based on users feedback (#3934)
docs: stream aggregation doc improvements based on users feedback
2023-03-12 00:19:22 -08:00
Zakhar Bessarab
3b7152b1d8
docs: add a note about cache reset for vmalert backfilling docs (#3940)
docs: add a note about cache reset for vmalert backfilling docs
2023-03-12 00:13:00 -08:00
Dmytro Kozlov
4029c539b9
app/vmctl: update importing tips when migrating data with overlapping time range (#3941)
app/vmctl: update importing tips when migrating data with overlapping time range
2023-03-12 00:10:08 -08:00
Dmytro Kozlov
f7d3efe3db
app/vmctl: add support of basic auth and barer token (#3921)
app/vmctl: add support of basic auth and bearer token
2023-03-12 00:08:04 -08:00
Roman Khavronenko
310b380a03
app/vmalert: log number of configration files found for each specified -rule (#3936)
The change also introduces `List` method to `FS` interface.
The `List` method can be used for wildcard support in object storage FS.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2023-03-11 23:40:40 -08:00
Dmytro Kozlov
afc6a6e218
app/vmctl: follow up after 09e3742a82 (#3937)
app/vmctl: follow up after 09e3742a82
2023-03-11 23:39:41 -08:00
Roman Khavronenko
b6ed977fa4
security: bump go version to 1.20.2 (#3935)
upgrade Go builder from Go1.20.1 to Go1.20.2
See the list of issues addressed in Go1.20.2 here (https://github.com/golang/go/issues?q=milestone%3AGo1.20.2+label%3ACherryPickApproved).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-03-11 23:38:14 -08:00
Denys Holius
e8a0d60ea5
Adds snap badge to README.md (#3930)
docs: adds snap badge to README.md
2023-03-11 23:31:38 -08:00
Aliaksandr Valialkin
54fe207cc0
all: follow-up for 7a3e16e774
- Sync the description for -httpListenAddr.useProxyProtocol command-line flag at vmagent and vmauth,
  so it is consistent with the description at vmauth and victoria-metrics
- Add a sample of panic text to docs/CHANGELOG.md, so it could be googled
- Mention the -httpListenAddr.useProxyProtocol command-line flag in the description for the bugfix

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335
2023-03-08 01:42:58 -08:00
Aliaksandr Valialkin
37981b276b
docs/CHANGELOG.md: improve description for 4b136abff8 2023-03-08 01:39:59 -08:00
Aliaksandr Valialkin
a911889817
docs/CHANGELOG.md: improve the description of the bugfix at 62beea23f7
- Make the description easier to read by humans :)
- Add a link to VictoriaMetrics datasource plugin for Grafana, so users could easily discover it
2023-03-08 01:39:39 -08:00
Aliaksandr Valialkin
bb5a39e5d1
docs/CHANGELOG.md: clarify the description for 6bfe9cc733
- Add the panic message to the description, so it is easier to google
- Add a link to the corresponding bugreport

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3897
2023-03-08 01:38:54 -08:00
Aliaksandr Valialkin
e91fe74618
all: typo fixes of the same type as in the d056be710b 2023-03-08 01:37:28 -08:00
Aliaksandr Valialkin
15f5639eb4
docs/CHANGELOG.md: clarify the description for the change at 8bab50dc29
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3600
2023-03-08 01:35:56 -08:00
Aliaksandr Valialkin
b60594a548
app/vmagent/remotewrite: follow-up for e3a756d82869f8c357b072f6e635ebfc7d65dd2c
- Document the fix
- Move the detection of VictoriaMetrics remoteWrite protocol from client.init() to newHTTPClient()
  This simplifies the fix to the following diff:

diff --git a/app/vmagent/remotewrite/client.go b/app/vmagent/remotewrite/client.go
index 099899c19..70b904af4 100644
--- a/app/vmagent/remotewrite/client.go
+++ b/app/vmagent/remotewrite/client.go
@@ -151,10 +151,6 @@ func newHTTPClient(argIdx int, remoteWriteURL, sanitizedURL string, fq *persiste
        }
        c.sendBlock = c.sendBlockHTTP

-       return c
-}
-
-func (c *client) init(argIdx, concurrency int, sanitizedURL string) {
        useVMProto := forceVMProto.GetOptionalArg(argIdx)
        usePromProto := forcePromProto.GetOptionalArg(argIdx)
        if useVMProto && usePromProto {
@@ -173,6 +169,10 @@ func (c *client) init(argIdx, concurrency int, sanitizedURL string) {
        }
        c.useVMProto = useVMProto

+       return c
+}
+
+func (c *client) init(argIdx, concurrency int, sanitizedURL string) {
2023-03-08 01:35:31 -08:00
Aliaksandr Valialkin
eb62b5548d
docs/CHANGELOG.md: document ec2abf9b69 2023-03-08 01:34:08 -08:00
Alexander Marshalov
0b7f83c846
added documentation about new templates field of vmalertmanager specification in operator (https://github.com/VictoriaMetrics/operator/issues/592) (#3924)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-03-08 01:33:42 -08:00
Nikolay
88f10d24a0
lib/netutil: fixes panic at proxy protocol (#3905)
it may occur if non proxy protocol message received by tcp server.
Listener Accept method must return only non-recoverable errors.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335
2023-03-08 01:33:01 -08:00
Yury Molodov
3e3e23a483
vmui: fix display of selected value in the selector (#3919)
vmui: fix selected value in dropdowns for Explore page
2023-03-08 01:13:54 -08:00
Nikolay
77e0e847a8
docs: updates operator api.md (#3922) 2023-03-08 01:12:16 -08:00
Craig Rodrigues
4c94fe0f0a
docs: move installation methods further up in README (#3904) 2023-03-08 01:09:35 -08:00
Dmytro Kozlov
d59a20c0a7
docs: follow up after 4b136abff8 (#3918)
docs: follow up after 4b136abff8
2023-03-08 01:08:07 -08:00
Alexander Marshalov
c682609fc4
added doc for placeholder support in vmagent specification for operator (https://github.com/VictoriaMetrics/operator/issues/592) (#3916) 2023-03-08 01:07:14 -08:00
Roman Khavronenko
e2527917a9
docs: follow-up after 62beea23f7 (#3907)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-03-08 01:01:16 -08:00
Artem Makhortov
f9723aca44
doc: vmctl vm-native-step-interval supported values (#3899) 2023-03-08 00:47:37 -08:00
Artem Navoiev
950abf7457
remove image width for vmalert managed vm guide
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-03-08 00:26:53 -08:00
Artem Navoiev
0ba6c42ea2
change title
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-03-08 00:26:22 -08:00
Artem Navoiev
470c5561d2
add vmalert managed vm integration guide
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-03-08 00:25:37 -08:00
Dmytro Kozlov
ff0e63ef0d
app/vmctl: add backoff retries to native protocol (#3859)
app/vmctl: vm-native - split migration on per-metric basis

`vm-native` mode now splits the migration process on per-metric basis. 
This allows to migrate metrics one-by-one according to the specified filter. 
This change allows to retry export/import requests for a specific metric and provides a better 
understanding of the migration progress.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2023-03-08 00:24:08 -08:00
Dmytro Kozlov
08df754903
app/vmctl: Increase http request timeout made by remote read client, add importing tips (#3879)
app/vmctl: Increase default http request timeout made by remote read client
2023-03-07 23:35:25 -08:00
Nikolay
361e1b1165
lib{mergset,storage}: prevent possible race condition with logging st… (#3900) (#3917)
lib{mergset,storage}: prevent possible race condition with logging stats for merges

Previously partwrapper could be release by background process and reference for part may be invalid 
during logging stats. It will lead to panic at vmstorage
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3897
2023-03-06 11:11:08 +01:00
Aliaksandr Valialkin
223bb6f367
docs/CHANGELOG.md: document v1.79.10 release 2023-02-27 17:36:13 -08:00
Aliaksandr Valialkin
ea1d7f28e5
docs/CHANGELOG.md: cut v1.88.1 2023-02-27 15:28:27 -08:00
Aliaksandr Valialkin
da419e7188
docs/CHANGELOG.md: link to the issue, which may benefit from -internStringDisableCache command-line flag
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3863
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3692
2023-02-27 14:56:22 -08:00
Aliaksandr Valialkin
086a4b4fca
lib/bytesutil: add -internStringDisableCache and -internStringCacheExpireDuration command-line flags
This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3872
2023-02-27 14:18:02 -08:00
Aliaksandr Valialkin
1ad0d22e80
lib/storage: follow-up for 39cdc546dd
- Use flag.Duration instead of flagutil.Duration for -snapshotCreateTimeout,
  since the flagutil.Duration is intended mostly for big durations, e.g. days, months and years,
  while the -snapshotCreateTimeout is usually smaller than one hour.
- Add links to https://docs.victoriametrics.com/#how-to-work-with-snapshots in docs/CHANGELOG.md,
  so readers could easily find the corresponding docs when reading the changelog.
- Properly remove all the created directories on unsuccessful attempt to create
  snapshot in Storage.CreateSnapshot().

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3551
2023-02-27 13:11:10 -08:00
Zakhar Bessarab
26682e369e
lib/storage: enhancements for snapshots process (#3873)
* lib/{fs,mergeset,storage}: skip `.must-remove.` dirs when creating snapshot (#3858)

* lib/{mergeset,storage}: add timeout configuration for snapshots creation, remove incomplete snapshots from storage

* docs: fix formatting

* app/vmstorage: add metrics to track status of snapshots

* app/vmstorage: use `vm_http_requests_total` metric for snapshot endpoints metrics, rename new flag to make name more clear

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vmstorage: update flag name in docs

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vmstorage: reflect new metrics names change in docs

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-27 13:11:06 -08:00
Zakhar Bessarab
1db010797e
lib/promscrape: correctly register vm_promscrape_config_* metrics (#3876)
* lib/promscrape: set `vm_promscrape_config_last_reload_successful` to 1 if there was no promscrape config provided

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/promscrape: register `vm_promscrape_config_*` metrics only in case promscrape config is used

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-27 12:06:49 -08:00
Zakhar Bessarab
9d04bb6004
doc: add changelog reference for vmgateway OpenID discovery (#3877)
* doc: add changelog reference for vmgateway OpenID discovery

* doc: add vmgateway docs

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-27 12:06:48 -08:00
Aliaksandr Valialkin
d3605ad072
app/vmselect/promql: fix panic when calculating aggr_func(rollup*())
The panic has been introduced in dac21d874b
2023-02-27 11:48:38 -08:00
Zakhar Bessarab
215206bded
app/vmgateway: add new flag doc
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-02-27 11:18:37 -08:00
Zakhar Bessarab
08a61d5c13
app/vmgateway: fix typo in docs
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-02-27 11:18:11 -08:00
Zakhar Bessarab
1b4871a9da
app/vmgateway: add OpenID discovery of JWKS endpoints
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-02-27 11:17:44 -08:00
Aliaksandr Valialkin
d9581cf3f3
app/vmagent: add -remoteWrite.vmProtoCompressLevel command-line flag for tuning the compression level for VictoriaMetrics remote write protocol 2023-02-27 11:04:11 -08:00
Dmytro Kozlov
65d9e281d6
app/vmctl: skip series if measurement not found (#3869)
app/vmctl: skip measurements with no fields for influxdb mode
2023-02-27 10:35:13 -08:00
Aliaksandr Valialkin
1e156ac3c3
app/vmagent: use the provided auth options when checking whether the remote storage supports VictoriaMetrics remote write protocol
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3847
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225
2023-02-26 12:19:53 -08:00
Aliaksandr Valialkin
a5e79b6f38
docs/CHANGELOG.md: cut v1.88.0 2023-02-24 17:54:35 -08:00
Aliaksandr Valialkin
90114759fa
docs: update -help output after e1c3267e34 2023-02-24 17:16:24 -08:00
Roman Khavronenko
66d0b45651
vmselect/promql: check for deadline in count_values fn (#3806)
* vmselect/promql: check for deadline in `count_values` fn

`count_values` could be very slow during the data processing.
Checking for deadline between iterations supposed to reduce
probability of exceeding `search.maxQueryDuration`.

The change also adds a new trace record, which captures the time
spent in aggregation function. Before that, the trace for aggr funcs
could be confusing since it doesn't account for all the places where
time was spent.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* wip

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-24 17:10:38 -08:00
Aliaksandr Valialkin
33df81cb4b
docs/CHANGELOG.md: document v1.87.2 release 2023-02-24 16:14:41 -08:00
Aliaksandr Valialkin
698aa1dd2a
docs/CHANGELOG.md: document v1.79.9 release 2023-02-24 15:10:58 -08:00
Roman Khavronenko
79eb33556e
metricsql: support optional 2nd argument for rollup functions (#3841)
* metricsql: support optional 2nd argument for rollup functions

Support optional 2nd argument `min`, `max` or `avg` for rollup functions:
 * rollup
 * rollup_delta
 * rollup_deriv
 * rollup_increase
 * rollup_rate
 * rollup_scrape_interval

 If second argument is passed, then rollup function will return only the selected aggregation type.
 This change can be useful for situations where only one type of rollup calculation is needed.
 For example, `rollup_rate(requests_total[5m], "max")`.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* wip

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-24 13:48:30 -08:00
Aliaksandr Valialkin
21fb56a498
docs/CHANGELOG.md: document d8eaa511b0 2023-02-24 12:43:45 -08:00
Aliaksandr Valialkin
0a9f5e178e
docs/CHANGELOG.md: typo fix: scrape scrape -> scrape 2023-02-24 12:33:37 -08:00
Aliaksandr Valialkin
aed2dbe45e
lib/promscrape: follow-up for 43e104a83f
- Return immediately on context cancel during the backoff sleep.
  This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3747

- Add a comment describing why the second attempt to obtain the response from remote side
  is perfromed immediately after the first attempt.

- Remove fasthttp dependency from lib/promscrape/discoveryutils

- Set context deadline before calling doRequestWithPossibleRetry().
  This simplifies the doRequestWithPossibleRetry() a bit.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3293
2023-02-24 12:25:36 -08:00
Zakhar Bessarab
5ea6d71cb3
fix: do not use exponential backoff for first retry of scrape request (#3824)
* fix: do not use exponential backoff for first retry of scrape request (#3293)

* lib/promscrape: refactor `doRequestWithPossibleRetry` backoff to simplify logic

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* Update lib/promscrape/client.go

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* lib/promscrape: refactor `doRequestWithPossibleRetry` to make it more straightforward

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-02-24 12:25:35 -08:00
Aliaksandr Valialkin
dce8afa4c4
app/vmselect/vmui: make vmui-update after d4fc0ed874 2023-02-23 19:26:08 -08:00
Aliaksandr Valialkin
3563c6882e
docs: update --help descriptions after recent changes 2023-02-23 19:02:32 -08:00
Aliaksandr Valialkin
6369c88a68
app/vmselect: add -search.logQueryMemoryUsage command-line flag for logging queries, which take big amounts of memory
Thanks to @michal-kralik for initial attempts for this feature:

- https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3651
- https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3715

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3553
2023-02-23 18:52:44 -08:00
Aliaksandr Valialkin
f579cac297
app/vmagent: automatically detect whether the remote storage supports VictoriaMetrics remote write protocol
Substitute -remoteWrite.useVMProto with -remoteWrite.forcePromProto command-line flag,
which can be used for forcing Prometheus remote write protocol in cases when the remote storage
supports VictoriaMetrics remote write protocol.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3847
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225
2023-02-23 17:38:47 -08:00
Aliaksandr Valialkin
e299854077
docs/CHANGELOG.md: document 6d019a3c37
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3830
2023-02-22 19:24:33 -08:00
Alexander Marshalov
9bb9bd266b
fix interpolate function for filling only intermediate gaps (#3816) (#3857)
* fix interpolate function for filling only intermediate gaps (#3816)

* Update docs/CHANGELOG.md

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-22 18:41:29 -08:00
Alexander Marshalov
66764ea4a0
fixed typo in dns+srv documentation (#3861) 2023-02-22 17:52:25 -08:00
Aliaksandr Valialkin
9609a44cc7
docs/CHANGELOG.md: document d2b92d3264
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3747
2023-02-22 17:50:55 -08:00
Aliaksandr Valialkin
1b70238dca
lib/promscrape/discovery/kuma: follow-up for 317fef95f9
- Do not generate __meta_server label, since it is unavailable in Prometheus.
- Add a link to https://docs.victoriametrics.com/sd_configs.html#kuma_sd_configs to docs/CHANGELOG.md,
  so users could click it and read the docs without the need to search the corresponding docs.
- Remove kumaTarget struct, since it is easier generating labels for discovered targets
  directly from the response returned by Kuma. This simplifies the code.
- Store the generated labels for discovered targets inside atomic.Value. This allows reading them
  from concurrent goroutines without the need to use mutex.
- Use synchronouse requests to Kuma instead of long polling, since there is a little sense
  in the long polling when the Kuma server may return 304 Not Modified response every -promscrape.kumaSDCheckInterval.
- Remove -promscrape.kuma.waitTime command-line flag, since it is no longer needed when long polling isn't used.
- Set default value for -promscrape.kumaSDCheckInterval to 30s in order to be consistent with Prometheus.
- Remove unnecessary indirections for string literals, which are used only once, in order to improve code readability.
- Remove unused fields from discoveryRequest and discoveryResponse.
- Update tests.
- Document why fetch_timeout and refresh_interval options are missing in kuma_sd_config.
- Add docs to discoveryutils.RequestCallback and discoveryutils.ResponseCallback,
  since these are public types.

Side notes: it is weird that Prometheus implementation for kuma_sd_configs sets `instance` label,
since usually this label is set by the Prometheus itself to __address__ after the relabeling phase.
See https://www.robustperception.io/life-of-a-label/

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3389

See https://github.com/prometheus/prometheus/issues/7919
and https://github.com/prometheus/prometheus/pull/8844
as a reference implementation in Prometheus
2023-02-22 17:50:54 -08:00
Aliaksandr Valialkin
55708fa474
docs/CHANGELOG.md: document 110c3896e7
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3600
2023-02-22 17:50:13 -08:00
Artem Navoiev
3abd6b367e
Add Dig Security case study
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-02-22 17:42:51 -08:00
Alexander Marshalov
173643a771
add kuma_sd_config for Kuma Control Plane targets discovery (#3389) (#3840) 2023-02-22 17:41:43 -08:00
Artem Navoiev
522179fbde
docs: fix typo OpentTSDB -> OpenTSDB (#3854)
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-02-22 12:11:48 -08:00
Aliaksandr Valialkin
ff8c57a964
app/vmselect: allow zero value for -search.latencyOffset command-line flag
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2061#issuecomment-1299109836

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/218
2023-02-21 18:07:27 -08:00
Aliaksandr Valialkin
57de6ad5dd
docs/Articles.md: mention rules backfilling via vmalert article
This is a follow-up for 5446ce0018
2023-02-21 17:44:45 -08:00
Roman Khavronenko
a29c1d3a02
docs: mention rules replay blogpost in vmalert docs (#3851)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-02-21 17:44:32 -08:00
panguicai
1b75d66bcd
docs: update operator release name to be consistent with the following (#3845)
Signed-off-by: panguicai008 <1121906548@qq.com>
2023-02-21 17:42:39 -08:00
Aliaksandr Valialkin
f783237866
docs/CHANGELOG.md: fix a link to VictoriaMetrics remote write protocol 2023-02-20 20:00:00 -08:00
Aliaksandr Valialkin
09b21fd800
docs/Single-server-VictoriaMetrics.md: remove + from start and end query args, since curl substitutes them with whitespace and breaks the query 2023-02-20 19:32:39 -08:00
Aliaksandr Valialkin
04ce4c4762
docs/guides/migrate-from-influx.md: remove misleading doublequotes from the query example 2023-02-20 19:25:07 -08:00
Zakhar Bessarab
8a35dc608f
vmgateway: add support of JWKS endpoint usage for JWT keys verification (#521) 2023-02-20 19:23:00 -08:00
Roman Khavronenko
fd139b463b
docs: update vmalert docs (#3843)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-02-20 19:13:00 -08:00
Corporte Gadfly
7d99b19478
docs: typo fix (#3839) 2023-02-20 19:11:59 -08:00
Aliaksandr Valialkin
fc5ea31a38
docs/vmagent.md: remove the claim that VictoriaMetrics remote write protocol reduces the network bandwidth usage by up to 10x comparing to Prometheus remote write protocol
The 10x savings are reproduced only on artificial data.
The savings on production data are usually in the range 2x-4x.
2023-02-20 19:11:11 -08:00
Aliaksandr Valialkin
bf359e8cc0
docs/vmagent.md: mention that Mimir doesnt support backfilling 2023-02-20 19:06:56 -08:00
Aliaksandr Valialkin
80c6d1e24c
app/vmagent: add support for VictoriaMetrics remote write protocol, which allows saving up to 10x on network bandwidth costs under high load
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225
2023-02-20 18:40:40 -08:00
Aliaksandr Valialkin
a15da5ff73
app/vmselect/promql: add share(q) aggregate function for normalizing results across multiple time series in [0..1] value range per each timestamp and aggregation group 2023-02-18 22:43:54 -08:00
Aliaksandr Valialkin
84b5532bc1
app/vmselect/promql: add range_zscore(q) and range_trim_zscore(z, q) functions
These functions may be useful for dropping outliers at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3759
2023-02-18 22:43:53 -08:00
Haleygo
9a274567f1
vmalert: fix maxResolveDuration flag note (#3827)
Signed-off-by: Haleygo <hui.wang@daocloud.io>
2023-02-18 15:20:30 -08:00
Aliaksandr Valialkin
450b6f6d39
app/vmselect/promql: add range_mad(q) and range_trim_outliers(k, q) functions
These functions may help trimming outliers during query time
for the use case described at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3759
2023-02-18 15:18:47 -08:00
Aliaksandr Valialkin
ecc84f7923
all: update Go builder from Go1.20.0 to Go1.20.1
See https://github.com/golang/go/issues?q=milestone%3AGo1.20.1+label%3ACherryPickApproved
2023-02-14 23:05:54 -08:00
Roman Khavronenko
c6251ec8aa
docs: improve troubleshooting docs for vmalert (#3812)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-02-13 09:42:18 -08:00
Aliaksandr Valialkin
caa2497aca
docs/CHANGELOG.md: improve the docs for 8ea02eaa8e 2023-02-13 09:41:22 -08:00
Roman Khavronenko
5941b5579d
follow-up after d1cbc35cf6 (#3813)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-02-13 09:37:09 -08:00
Artem Navoiev
253a78b9ab
change docs for VictoriaMetrics Managed (#3803)
* change docs for VictoriaMetrics Managedd

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

* Update docs/managed-victoriametrics/user-managment.md

Co-authored-by: Max Golionko <8kirk8@gmail.com>

---------

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
Co-authored-by: Max Golionko <8kirk8@gmail.com>
2023-02-13 09:33:32 -08:00
Aliaksandr Valialkin
8355e4a1ee
docs/sd_configs.md: properly escape __address__ string 2023-02-11 14:46:02 -08:00
Aliaksandr Valialkin
c8e41100e6
docs/sd_configs.md: document how the __address__ label is generated per each discovered target 2023-02-11 14:42:10 -08:00
Aliaksandr Valialkin
27b3209816
app/vmui: show median instead of avg on graph tooltip and line legend
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3706
2023-02-11 12:52:50 -08:00
Aliaksandr Valialkin
9053745a6f
lib/{mergeset,storage}: allow at least 3 concurrent flushes during background merges on systems with 1 or 2 CPU cores
This should prevent from data ingestion slowdown and query performance degradation
on systems with small number of CPU cores (1 or 2), when big merge is performed.

This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3790

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337
2023-02-11 12:09:13 -08:00
Aliaksandr Valialkin
1a6d266d1a
docs/Articles.md: add a link to https://www.techetio.com/2022/08/21/evaluating-backend-options-for-prometheus-metrics/ 2023-02-11 11:37:06 -08:00
Aliaksandr Valialkin
db7f237da9
app/vmselect/promql: add mad_over_time(m[d]) function
See https://github.com/prometheus/prometheus/issues/5514
2023-02-11 01:06:39 -08:00
Aliaksandr Valialkin
4d9aaff3e1
all: update alpine base docker image from 1.17.1 to 1.17.2
See https://alpinelinux.org/posts/Alpine-3.17.2-released.html
2023-02-11 00:37:28 -08:00
Aliaksandr Valialkin
9ed5b872df
app/vmauth: improve load balancing by sending incoming requests to backends with the lowest number of concurrent requests
While at it, stop sending requests to unavailable backend for 3 seconds
before the next attempt. This should reduce the amounts of useless work
and the number of useless network packets when the backend is temporarily unavailable.
2023-02-11 00:32:56 -08:00
Aliaksandr Valialkin
6c6b9c3d93
app/vmauth: add -maxConcurrentPerUserRequests command-line option for limiting the number of concurrent requests on a per-user basis
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3346
2023-02-10 21:58:30 -08:00
Aliaksandr Valialkin
769d7da25a
app/vmauth: automatically retry failing GET requests on the remaining backends 2023-02-09 21:06:09 -08:00
Dmytro Kozlov
c80fc8c77f
app/vmauth: add concurrent requests limit per auth record (#3749)
* app/vmauth: add concurent requests limit per auth record

* app/vmauth: added clarification comment

* app/vmauth: remove unused code

* app/vmauth: move read from limiter

* app/vmauth: fix text

* app/vmauth: fix comments

* - Clarify the docs for the max_concurrent_requests option at docs/vmauth.md
- Clarify the description of the change at docs/CHANGELOG.md
- Make sure that the -maxConcurrentRequests takes precedence over per-user max_concurrent_requests
- Update tests for verifying that the max_concurrent_requests option is parsed properly

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3346

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-09 20:03:47 -08:00
Aliaksandr Valialkin
ca61c276ca
app/vmalert: follow-up after d3c64aae8768d58781ee7e358bd7f3d8e0eb836d
- Document the change at docs/CHANGELOG.md
- Add `Reading rules from object storage` section to docs/vmalert.md
- Add `s3` prefix to command-line flags related to the configuration of s3 and gcs clients
- Explicitly mention that reading rules from object storage is supported only in enterprise version
2023-02-09 19:10:36 -08:00
Aliaksandr Valialkin
6ee15f94f8
docs/CHANGELOG.md: document d621d50d4fb3b43a0bcb4419bee979f0192d38fe 2023-02-09 14:40:15 -08:00
Aliaksandr Valialkin
da9d1fea10
docs/CHANGELOG.md: cut v1.87.1 and mark 1.87.x as LTS release 2023-02-09 11:21:04 -08:00
Zakhar Bessarab
bbf663bd04
lib/promscrape: fix cancelling in-flight scrape requests during configuration reload (#3791)
* lib/promscrape: fix cancelling in-flight scrape requests during configuration reload when using `streamParse` mode (see #3747)

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* Update docs/CHANGELOG.md

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-09 11:18:36 -08:00
Aliaksandr Valialkin
82b5fa2fd0
app/vmui: UX enhancements for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3706
- Display `min` value additionally to `avg`, `max` and `last`
- Allow copy-n-pasting metric name with its labels from both legend and tooltup
2023-02-09 11:05:57 -08:00
Aliaksandr Valialkin
dc7b92bc34
docs/vmagent.md: clarify that automatically generated metrics contain all the target-specific labels, including instance and job 2023-02-09 11:05:57 -08:00
Air
fe6e1015fe
Possibly spelling in the Quick start 2023-02-09 11:05:57 -08:00
Yury Molodov
5a7c63a324
vmui: improve tenant selector (#3794)
* fix: change styles tenant selector (#3792)

* docs/CHANGELOG.md: document the change

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3792

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-09 00:09:36 -08:00
Aliaksandr Valialkin
cd78396adb
docs/CHANGELOG.md: document changes at v1.79.8 LTS release 2023-02-08 23:39:01 -08:00
Yury Molodov
54bfd22ec5
vmui: add last/max/avg values (#3789)
* feat: add last/max/avg values (#3706)

* fix: change filter exclude values

* app/vmui: wip

- improve the visualization for avg/max/last values
- make getAvgFromArray() function resilient against inf/undefined/nil
- export getLastFromArray() function, which is resilient against inf/undefined/nil
- run `make vmui-update`

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-08 22:42:05 -08:00
Aliaksandr Valialkin
cfca7075c4
docs/CHANGELOG.md: document 75bcf86a31
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3740
2023-02-08 11:24:29 -08:00
Aliaksandr Valialkin
8d359b549e
docs/CHANGELOG.md: add more context to the bugfix description in Nomad service discovery
See 146fd2eca3
2023-02-08 09:24:51 -08:00
Aliaksandr Valialkin
b3bfca16de
docs/CHANGELOG.md: document the change at 67b01329a0
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3761
2023-02-08 08:46:21 -08:00
Roman Khavronenko
4e922eb93b
Vmalert fixes (#3788)
* vmalert: use group's ID in UI to avoid collisions

Identical group names are allowed. So we should used IDs
for various groupings and aggregations in UI.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: prevent disabling state updates tracking

The minimum number of update states to track is now set to 1.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: properly update `debug` and `update_entries_limit` params on hot-reload

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: display `debug` field for rule in UI

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: exclude `updates` field from json marhsaling

This field isn't correctly marshaled right now.
And implementing the correct marshaling for it doesn't
seem right, since json representation is mostly used
by systems like Grafana. And Grafana doesn't expect this
field to be present.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* fix test for disabled state

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* fix test for disabled state

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-02-08 08:45:25 -08:00
Roman Khavronenko
d260b90445
docs: follow-up after 2e4bfcce63 (#3785)
2e4bfcce63

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-02-08 08:37:48 -08:00
earthgecko
e8924ebfba
Clarifications between standalone/cluster ingestion endpoints (#3771)
docs: clarifications between standalone/cluster ingestion endpoints

This is an attempt to make it a bit clearer to the user that the cluster version ingestion URLs are different from the standalone ones.  I have also changed the order of the list items to make it a bit clearer and hopefully stop the user simply inferring that `/prometheus/api/v1` is only related to Prometheus data.
2023-02-07 09:34:02 -08:00
Max Golionko
eaa3716507
docs: update formatiing for k8s monitoring with Managed VictoriaMetrics (#3768)
* jekyll formatting madness
2023-02-07 09:29:34 -08:00
Roman Khavronenko
80bf0bcf8c
vmalert: update docs (#3770)
vmalert: update flags description

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-02-07 09:28:59 -08:00
Aliaksandr Valialkin
d8a54ab839
docs/CHANGELOG.md: fix formatting fro the change from 6fd10e8871 2023-02-07 09:26:38 -08:00
Roman Khavronenko
96db7ac52c
vmalert: speed up state restore procedure on start (#3758)
* vmalert: speed up state restore procedure on start

Alerts state restore procedure has been changed to become asynchronous.
It doesn't block groups start anymore which significantly improves vmalert's startup time.
Instead, state restore is called by each group in their goroutines after the first rules
evaluation.

While previously state restore attempt was made for all loaded alerting rules,
now it is called only for alerts which became active after the first evaluation.
This reduces the amount of API calls to the configured remote read URL.

This also means that `remoteRead.ignoreRestoreErrors` command-line flag becomes deprecated now
and will have no effect if configured.

See relevant issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2608

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* make lint happy

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Apply suggestions from code review

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-03 19:46:41 -08:00
Aliaksandr Valialkin
26614da548
docs/CHANGELOG.md: document f63f487787
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3707
2023-02-03 19:30:40 -08:00
Aliaksandr Valialkin
6910dd2a98
docs/MetricsQL.md: add links to "rollup results" explanation 2023-02-03 11:10:49 -08:00
Aliaksandr Valialkin
06c3e303ec
docs/CHANGELOG.md: remove duplicate line after b7cc0bfe3c 2023-02-03 10:48:54 -08:00
Aliaksandr Valialkin
ee800d6336
docs/managed-victoriametrics/how-to-monitor-k8s.md: rename image files according to docs/assets/README.md 2023-02-03 10:46:49 -08:00
Max Golionko
2d419021dc
docs: move managed victoria metics guide into right folder (#3750)
* move guide folder
* image width control
2023-02-03 10:46:26 -08:00
Max Golionko
1d528f3ed6
Update docs/managed_victoriametrics/how-to-monitor-k8s.md
Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-02-03 10:45:46 -08:00
Max Golionko
76d339b886
add guide to list of guides 2023-02-03 10:45:01 -08:00
Max Golionko
32b283d9c3
added k8s guide for managed VM 2023-02-03 10:43:45 -08:00
Aliaksandr Valialkin
f737674912
docs/CHANGELOG.md: typo fixes 2023-02-01 20:40:55 -08:00
Aliaksandr Valialkin
00a6ec85c6
docs/CHANGELOG.md: cut v1.87.0 2023-02-01 13:03:27 -08:00
Dmytro Kozlov
b7cc0bfe3c
app/vmctl: Add insecure skip verify flag for remote read protocol (#3611)
* app/vmctl: Add insecure skip verify flag for remote read protocol
2023-02-01 12:45:11 -08:00
Dmytro Kozlov
3c1e455805
lib/promscrape: fix honor_labels behavior (#3739)
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-01 12:02:04 -08:00
Aliaksandr Valialkin
03145842d7
docs/CHANGELOG.md: document 9254e494f9 2023-02-01 09:57:04 -08:00
Zakhar Bessarab
326a702559
app/vmbackup: fix deleting snapshot after backup completion (#3735) (#3736)
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-02-01 08:58:11 -08:00
Zakhar Bessarab
626bd22157
fix: vmselect multi-level setup panic (#3738)
* app/vmselect/netstorage: fix panic for multi-level cluster setup when `replicationFactor` was set and request contained `trace` parameter (#3734)

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vmselect/netstorage: use correct context for retry

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-02-01 08:56:36 -08:00
Aliaksandr Valialkin
e628289df4
docs/CHANGELOG.md: document 442a9f16b4
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3661
2023-01-31 13:04:20 -08:00
Roman Khavronenko
d93ac2b1ea
docs: mention -vmalert.proxyURL in vmalert docs (#3730)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-01-31 10:49:49 -08:00
Aliaksandr Valialkin
855d560789
lib/promscrape/discovery/azure: add __meta_azure_machine_size label in the same way as Prometheus does
See https://github.com/prometheus/prometheus/pull/11650
2023-01-27 17:07:57 -08:00