Commit graph

6823 commits

Author SHA1 Message Date
hagen1778
91eddeca52
vmalert: revert accidental changes to Makefile rule
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-27 12:58:26 -07:00
Aliaksandr Valialkin
2e39070f95
vendor: run make vendor-update 2023-07-26 15:09:26 -07:00
Aliaksandr Valialkin
1b7d97787a
docs: use 1. instead of N. in numbered bullets, so they are automatically adjusted by Github Markdown engine
See https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#lists
2023-07-26 14:40:06 -07:00
Aliaksandr Valialkin
98676a5223
docs/Cluster-VictoriaMetrics.md: use 1. instead of N. in numbered bullets, so they are automatically adjusted by Github Markdown engine
See https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#lists
2023-07-26 14:13:24 -07:00
Aliaksandr Valialkin
da61f5dd34
docs/keyConcepts.md: use 1. instead of N. in numbered bullets, so they are automatically adjusted by Github Markdown engine
See https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#lists
2023-07-26 14:12:46 -07:00
Aliaksandr Valialkin
5466183fd9
docs/Single-server-VictoriaMetrics.md: use 1. instead of N. in numbered bullets, so they are automatically adjusted by Github Markdown engine
See https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#lists
2023-07-26 14:12:45 -07:00
Aliaksandr Valialkin
64ef01fccf
docs/Release-Guide.md: use 1. instead of N. in numbered bullets, so they are automatically adjusted by Github Markdown engine
See https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#lists
2023-07-26 14:12:32 -07:00
Aliaksandr Valialkin
3ef08d16fc
docs/Troubleshooting.md: use 1. instead of N. in numbered bullets, so they are automatically adjusted by Github Markdown engine
See https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#lists
2023-07-26 14:12:32 -07:00
hagen1778
290443f7d1
docs: add bullet for gaps in timeseries to Troubleshooting
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3878
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4697
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3849
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3451
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-26 14:12:32 -07:00
Aliaksandr Valialkin
3179d0528b
lib/promrelabel: return correct string representation for IfExpression containing a single selector
This is a follow-up for 62651570bb

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4635
2023-07-24 19:33:04 -07:00
Aliaksandr Valialkin
2d88ebd7cb
app/vmalert/datasource: substitute golang.org/x/exp/slices.SortFunc with sort.Slice
This removes unnecessary third-party dependency on golang.org/x/exp.

This is a follow-up for da60a68d09
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2945
2023-07-24 19:17:19 -07:00
Aliaksandr Valialkin
fa295c7daa
app/vmagent: add ability to shard outgoing data among multiple remote storage systems
Add -remoteWrite.shardByURL command-line flag, which instructs vmagent to spread evenly
outgoing time series data among the configured remote storage systems specified via -remoteWrite.url .

Samples for the same time series go to the same -remoteWrite.url . This allows building horizontally
scalable stream aggregation when samples for counter and histogram series must be aggregated
by the same second-level vmagent instance.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4637
2023-07-24 18:18:04 -07:00
Aliaksandr Valialkin
16c343f882
app/{vmselect,vlselect}/vmui: run make vmui-update vmui-logs-update after b6ae325763 2023-07-24 17:15:26 -07:00
Aliaksandr Valialkin
6e43664e24
lib/promrelabel: add support for a list of series selectors at IfExpression
This makes possible specifying a list of series selectors at the following places:

- Inside `if` option at relabeling rules
- Inside `match` option at stream aggregation rules

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4635
2023-07-24 17:09:59 -07:00
Aliaksandr Valialkin
c049778ad1
lib/streamaggr: follow-up for 736197179e
- Use a byte slice instead of a map for tracking indexes for matching series.
  This improves performance, since access by slice index is faster than access by map key.
- Re-use the byte slice for tracking indexes for matching series.
  This removes unnecessary memory allocations and improves stream aggregation performance a bit.
- Add an ability to return to the previous behvaiour by specifying -remoteWrite.streamAggr.dropInput command-line flag.
  In this case all the input samples are dropped when stream aggregation is enabled.
- Backport the new stream aggregation behaviour from vmagent to single-node VictoriaMetrics when -streamAggr.config
  option is set.
- Improve docs regarding this change at docs/CHANGELOG.md
- Document the new behavior at docs/stream-aggregation.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4243
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4575
2023-07-24 17:06:09 -07:00
Zakhar Bessarab
470afac5ff
{lib/streamaggr,vmagent/remotewrite}: breaking change for keepInput flag (#4575)
* {lib/streamaggr,vmagent/remotewrite}: breaking change for keepInput flag

Changes default behaviour of keepInput flag to write series which did not match any aggregators to the remote write.
See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4243

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* Update app/vmagent/remotewrite/remotewrite.go

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-07-24 16:34:38 -07:00
Yury Molodov
9f440b1013
vmui: remove monaco-editor (#4698)
* fix: remove monaco editor due to bundle size

* vmui: update dependencies

* fix: disable source map generation
2023-07-24 16:27:57 -07:00
hagen1778
42af241d59
docs: update datadog config recommendations
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-24 16:17:35 -07:00
Roman Khavronenko
08e66885ab
docs: mention lambdas and cloud functions for aggregation (#4705)
* docs: mention lambdas and cloud functions for aggregation

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-24 16:16:43 -07:00
Haleygo
ee1b3a48e9
vmalert: unittest support stale datapoint (#4696)
* vmalert: unittest support stale datapoint

* add stale ut case
2023-07-24 16:15:27 -07:00
Roman Khavronenko
f536f22dfb
docs: add case for histograms aggregation (#4702)
docs: add case for histograms aggregation

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-24 16:14:36 -07:00
Aliaksandr Valialkin
9d14c29667
lib/streamaggr: skip de-duplication for series, which do not match the configured aggregation rules
Previously all the incoming samples were de-duplicated, even if their series doesn't
match aggregation rule filters. This could result in increased CPU usage.

Now the de-duplication isn't applied to samples for series, which do not match
aggregation rule filters. Such samples are just ignored.
2023-07-22 16:46:17 -07:00
Zakhar Bessarab
d800545055
app/vlinsert/loki: manually remove bloat dependecies for generate proto file (#4686)
Co-authored-by: f41gh7 <nik@victoriametrics.com>
2023-07-22 15:37:30 -07:00
Nikolay
30b32583f4
lib/storage: pre-create timeseries before indexDB rotation (#4652)
* lib/storage: pre-create timeseries before indexDB rotation
during an hour before indexDB rotation start creating records at the next indexDB
it must improve performance during switch for the next indexDB and remove ingestion issues.
Since there is no need for creation new index records for timeseries already ingested into current indexDB
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4563

* lib/storage: further work on indexdb rotation optimization

- Document the change at docs/CHAGNELOG.md
- Move back various caches from indexDB to Storage. This makes the change less intrusive.
  The dateMetricIDCache now takes into account indexDB generation, so it stores (date, metricID)
  entries for both the current and the next indexDB.
- Consolidate the code responsible for idbNext pre-filling into prefillNextIndexDB() function.
  This improves code readability and maintainability a bit.
- Rewrite and simplify the code responsible for calculating the next retention timestamp.
  Add various tests for corner cases of this code.
- Remove indexdb pre-filling from RegisterMetricNames() function, since this function is rarely called.
  It is OK to add indexdb entries on demand in this function. This simplifies the code.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401

* docs/CHANGELOG.md: refer to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4563

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-07-22 15:23:14 -07:00
yrka
7750c5a553
Update MetricsQL.md (#4689)
misspell
2023-07-22 14:19:30 -07:00
Zakhar Bessarab
2fe33b3d97
app/vmalert/datasource/graphite: allow overriding "from" parameter for datasource queries (#4687)
* app/vmalert/datasource/graphite: allow overriding "from" parameter for datasource queries

Fixes construction of URL parameters for graphite render to allow overriding "from" parameter.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4685
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vmalert/datasource/graphite: update flow for building URL parameters

Makes flow of building URL parameters same as Prometheus datasource has:
1) Setting all default values
2) Merging those values with provided `extraParams`

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* Update docs/CHANGELOG.md

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-07-22 14:18:52 -07:00
Roman Khavronenko
af56dff7ce
dependabot: disable version update for packages (#4684)
The change disables version updates for repo packages.
Please note, security updates should not be affected by the change
according to https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#open-pull-requests-limit:

```
open-pull-requests-limit
By default, Dependabot opens a maximum of five pull requests for version updates. Once there are five open pull requests from Dependabot, Dependabot will not open any new requests until some of those open requests are merged or closed.

This option has no impact on security updates, which have a separate, internal limit of ten open pull requests.
```

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-22 14:17:50 -07:00
Aliaksandr Valialkin
1ce82f874c
lib/streamaggr: follow up for 70773f53d7
- Round staleness_interval durations to the upper number of seconds.
  This should prevent from under-calculations for fractional staleness intervals.
- Rename stalenessInterval field at *AggrState structs into stalenessSecs, since it holds seconds.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4667
2023-07-20 21:56:36 -07:00
Aliaksandr Valialkin
cbc5367812
docs/VictoriaLogs/data-ingestion: mention that VictoriaLogs automatically extracts message and timestamp from data ingested via Loki JSON API 2023-07-20 21:55:46 -07:00
Aliaksandr Valialkin
8c59813c17
app/vlinsert/loki: fix build for architectures where int is 32-bit 2023-07-20 21:55:28 -07:00
Aliaksandr Valialkin
9e6cc3c495
deployment: update VictoriaLogs Docker image tag from from v0.2.0-victorialogs to v0.3.0-victorialogs 2023-07-20 21:55:09 -07:00
Aliaksandr Valialkin
dcf5b42670
lib/encoding/zstd: switch back from atomic.Pointer to atomic.Value for map[...]...
The map[...]... is already a pointer type, so atomic.Pointer[map[...]...] results in double pointer.

This is a follow-up for 140e7b6b74
2023-07-20 21:54:51 -07:00
Aliaksandr Valialkin
c94a5d95a2
docs/VictoriaLogs/CHANGELOG.md: cut v0.3.0-victorialogs 2023-07-20 21:54:28 -07:00
Aliaksandr Valialkin
324a3c5288
lib/promscrape: follow-up after 6aa50ca954
- Improve docs
- Hide `debug relabeling` column when -promscrape.dropOriginalLabels command-line flag is set
- Inline the code from the added template functions, since the code is harder to follow
  with the template functions, especially when these functions have misleading names.
  Also, these functions are used only in one place, e.g. they do not reduce the amounts of code.
- Hide `click to show original labels` title at `labels` column when original labels aren't available.
- Show the reason on whey original labels aren't available at /service-discovery page.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4597
2023-07-20 21:54:09 -07:00
Aliaksandr Valialkin
c921bc0833
app/{vmselect,vlselect}: run make vmui-update vmui-logs-update after recent changes to VMUI
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4604
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4676
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4294
2023-07-20 21:53:51 -07:00
Yury Molodov
7757a31a73
fix: change getting serverUrl for vmui-logs (#4604) (#4677) 2023-07-20 21:53:30 -07:00
Yury Molodov
02a182e2a1
feat: change columns for active queries (#4676) 2023-07-20 21:53:12 -07:00
Yury Molodov
15e1d16afc
vmui: enhancements multiline field editing (#4294)
* fix: change textarea for relabel page

* feat: add comment for monaco theme

* fix: change behavior of multiline fields

* vmui: merge master
2023-07-20 21:52:53 -07:00
Aliaksandr Valialkin
f086a5ba63
docs/VictoriaLogs/CHANGELOG.md: document the addition of Grafana Loki push protocol support (aka Promtail)
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4482
2023-07-20 21:52:33 -07:00
Aliaksandr Valialkin
30098ac8bd
app/vlinsert/loki: follow-up after 09df5b66fd
- Parse protobuf if Content-Type isn't set to `application/json` - this behavior is documented at https://grafana.com/docs/loki/latest/api/#push-log-entries-to-loki

- Properly handle gzip'ped JSON requests. The `gzip` header must be read from `Content-Encoding` instead of `Content-Type` header

- Properly flush all the parsed logs with the explicit call to vlstorage.MustAddRows() at the end of query handler

- Check JSON field types more strictly.

- Allow parsing Loki timestamp as floating-point number. Such a timestamp can be generated by some clients,
  which store timestamps in float64 instead of int64.

- Optimize parsing of Loki labels in Prometheus text exposition format.

- Simplify tests.

- Remove lib/slicesutil, since there are no more users for it.

- Update docs with missing info and fix various typos. For example, it should be enough to have `instance` and `job` labels
  as stream fields in most Loki setups.

- Allow empty of missing timestamps in the ingested logs.
  The current timestamp at VictoriaLogs side is then used for the ingested logs.
  This simplifies debugging and testing of the provided HTTP-based data ingestion APIs.

The remaining MAJOR issue, which needs to be addressed: victoria-logs binary size increased from 13MB to 22MB
after adding support for Loki data ingestion protocol at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4482 .
This is because of shitty protobuf dependencies. They must be replaced with another protobuf implementation
similar to the one used at lib/prompb or lib/prompbmarshal .
2023-07-20 21:52:11 -07:00
Dmytro Kozlov
41f0227af2
docs: update information about tsdb usage in cluster version (#4678)
update information about tsdb usage in cluster version
2023-07-20 21:50:58 -07:00
hagen1778
1d2a0e0e10
docs: fix the next release version for vmalert
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-20 21:49:34 -07:00
Zakhar Bessarab
5b3000f7e8
docs: fix broken link to managed VM (#4679)
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-07-20 21:49:06 -07:00
hagen1778
5b963e1971
docs: mention streaming aggregation as way to reduce cardinality
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-20 21:48:09 -07:00
Alexander Marshalov
9ba03b4838
allow configuring staleness interval in stream aggregation (#4667) (#4670)
---------

Signed-off-by: Alexander Marshalov <_@marshalov.org>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-07-20 21:47:29 -07:00
hagen1778
f7d60613a9
docs: specify min version and limitations for vmalert's unit tests
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-20 21:27:50 -07:00
Aliaksandr Valialkin
a0b7def89d
app/vmselect/promql: fix tests after 781947a7e2 2023-07-20 21:25:30 -07:00
Haleygo
939c8b8372
vmalert: init unit test (#4596)
vmalert: support unit tests

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2945
---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2023-07-20 21:19:45 -07:00
hagen1778
a0cc6eb7a1
docs: mention the simplest way to migrate data in vmctl docs
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-20 21:12:18 -07:00
Dmytro Kozlov
f0d8f77e6d
app/vmagent: fix creating target id if --promscrape.dropOriginalLabels flag was used (#4616)
* app/vmagent: fix creating target id if `--promscrape.dropOriginalLabels` flag was used

* app/vmagent: hide links if OriginalLabels was dropped

* app/vmagent: update CHANGELOG.md and added information to the docs

* app/vmagent: fix comments
2023-07-20 19:21:41 -07:00