Commit graph

6762 commits

Author SHA1 Message Date
Aliaksandr Valialkin
fa295c7daa
app/vmagent: add ability to shard outgoing data among multiple remote storage systems
Add -remoteWrite.shardByURL command-line flag, which instructs vmagent to spread evenly
outgoing time series data among the configured remote storage systems specified via -remoteWrite.url .

Samples for the same time series go to the same -remoteWrite.url . This allows building horizontally
scalable stream aggregation when samples for counter and histogram series must be aggregated
by the same second-level vmagent instance.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4637
2023-07-24 18:18:04 -07:00
Aliaksandr Valialkin
16c343f882
app/{vmselect,vlselect}/vmui: run make vmui-update vmui-logs-update after b6ae325763 2023-07-24 17:15:26 -07:00
Aliaksandr Valialkin
6e43664e24
lib/promrelabel: add support for a list of series selectors at IfExpression
This makes possible specifying a list of series selectors at the following places:

- Inside `if` option at relabeling rules
- Inside `match` option at stream aggregation rules

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4635
2023-07-24 17:09:59 -07:00
Aliaksandr Valialkin
c049778ad1
lib/streamaggr: follow-up for 736197179e
- Use a byte slice instead of a map for tracking indexes for matching series.
  This improves performance, since access by slice index is faster than access by map key.
- Re-use the byte slice for tracking indexes for matching series.
  This removes unnecessary memory allocations and improves stream aggregation performance a bit.
- Add an ability to return to the previous behvaiour by specifying -remoteWrite.streamAggr.dropInput command-line flag.
  In this case all the input samples are dropped when stream aggregation is enabled.
- Backport the new stream aggregation behaviour from vmagent to single-node VictoriaMetrics when -streamAggr.config
  option is set.
- Improve docs regarding this change at docs/CHANGELOG.md
- Document the new behavior at docs/stream-aggregation.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4243
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4575
2023-07-24 17:06:09 -07:00
Zakhar Bessarab
470afac5ff
{lib/streamaggr,vmagent/remotewrite}: breaking change for keepInput flag (#4575)
* {lib/streamaggr,vmagent/remotewrite}: breaking change for keepInput flag

Changes default behaviour of keepInput flag to write series which did not match any aggregators to the remote write.
See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4243

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* Update app/vmagent/remotewrite/remotewrite.go

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-07-24 16:34:38 -07:00
Yury Molodov
9f440b1013
vmui: remove monaco-editor (#4698)
* fix: remove monaco editor due to bundle size

* vmui: update dependencies

* fix: disable source map generation
2023-07-24 16:27:57 -07:00
hagen1778
42af241d59
docs: update datadog config recommendations
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-24 16:17:35 -07:00
Roman Khavronenko
08e66885ab
docs: mention lambdas and cloud functions for aggregation (#4705)
* docs: mention lambdas and cloud functions for aggregation

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-24 16:16:43 -07:00
Haleygo
ee1b3a48e9
vmalert: unittest support stale datapoint (#4696)
* vmalert: unittest support stale datapoint

* add stale ut case
2023-07-24 16:15:27 -07:00
Roman Khavronenko
f536f22dfb
docs: add case for histograms aggregation (#4702)
docs: add case for histograms aggregation

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-24 16:14:36 -07:00
Aliaksandr Valialkin
9d14c29667
lib/streamaggr: skip de-duplication for series, which do not match the configured aggregation rules
Previously all the incoming samples were de-duplicated, even if their series doesn't
match aggregation rule filters. This could result in increased CPU usage.

Now the de-duplication isn't applied to samples for series, which do not match
aggregation rule filters. Such samples are just ignored.
2023-07-22 16:46:17 -07:00
Zakhar Bessarab
d800545055
app/vlinsert/loki: manually remove bloat dependecies for generate proto file (#4686)
Co-authored-by: f41gh7 <nik@victoriametrics.com>
2023-07-22 15:37:30 -07:00
Nikolay
30b32583f4
lib/storage: pre-create timeseries before indexDB rotation (#4652)
* lib/storage: pre-create timeseries before indexDB rotation
during an hour before indexDB rotation start creating records at the next indexDB
it must improve performance during switch for the next indexDB and remove ingestion issues.
Since there is no need for creation new index records for timeseries already ingested into current indexDB
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4563

* lib/storage: further work on indexdb rotation optimization

- Document the change at docs/CHAGNELOG.md
- Move back various caches from indexDB to Storage. This makes the change less intrusive.
  The dateMetricIDCache now takes into account indexDB generation, so it stores (date, metricID)
  entries for both the current and the next indexDB.
- Consolidate the code responsible for idbNext pre-filling into prefillNextIndexDB() function.
  This improves code readability and maintainability a bit.
- Rewrite and simplify the code responsible for calculating the next retention timestamp.
  Add various tests for corner cases of this code.
- Remove indexdb pre-filling from RegisterMetricNames() function, since this function is rarely called.
  It is OK to add indexdb entries on demand in this function. This simplifies the code.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401

* docs/CHANGELOG.md: refer to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4563

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-07-22 15:23:14 -07:00
yrka
7750c5a553
Update MetricsQL.md (#4689)
misspell
2023-07-22 14:19:30 -07:00
Zakhar Bessarab
2fe33b3d97
app/vmalert/datasource/graphite: allow overriding "from" parameter for datasource queries (#4687)
* app/vmalert/datasource/graphite: allow overriding "from" parameter for datasource queries

Fixes construction of URL parameters for graphite render to allow overriding "from" parameter.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4685
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vmalert/datasource/graphite: update flow for building URL parameters

Makes flow of building URL parameters same as Prometheus datasource has:
1) Setting all default values
2) Merging those values with provided `extraParams`

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* Update docs/CHANGELOG.md

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-07-22 14:18:52 -07:00
Roman Khavronenko
af56dff7ce
dependabot: disable version update for packages (#4684)
The change disables version updates for repo packages.
Please note, security updates should not be affected by the change
according to https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#open-pull-requests-limit:

```
open-pull-requests-limit
By default, Dependabot opens a maximum of five pull requests for version updates. Once there are five open pull requests from Dependabot, Dependabot will not open any new requests until some of those open requests are merged or closed.

This option has no impact on security updates, which have a separate, internal limit of ten open pull requests.
```

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-22 14:17:50 -07:00
Aliaksandr Valialkin
1ce82f874c
lib/streamaggr: follow up for 70773f53d7
- Round staleness_interval durations to the upper number of seconds.
  This should prevent from under-calculations for fractional staleness intervals.
- Rename stalenessInterval field at *AggrState structs into stalenessSecs, since it holds seconds.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4667
2023-07-20 21:56:36 -07:00
Aliaksandr Valialkin
cbc5367812
docs/VictoriaLogs/data-ingestion: mention that VictoriaLogs automatically extracts message and timestamp from data ingested via Loki JSON API 2023-07-20 21:55:46 -07:00
Aliaksandr Valialkin
8c59813c17
app/vlinsert/loki: fix build for architectures where int is 32-bit 2023-07-20 21:55:28 -07:00
Aliaksandr Valialkin
9e6cc3c495
deployment: update VictoriaLogs Docker image tag from from v0.2.0-victorialogs to v0.3.0-victorialogs 2023-07-20 21:55:09 -07:00
Aliaksandr Valialkin
dcf5b42670
lib/encoding/zstd: switch back from atomic.Pointer to atomic.Value for map[...]...
The map[...]... is already a pointer type, so atomic.Pointer[map[...]...] results in double pointer.

This is a follow-up for 140e7b6b74
2023-07-20 21:54:51 -07:00
Aliaksandr Valialkin
c94a5d95a2
docs/VictoriaLogs/CHANGELOG.md: cut v0.3.0-victorialogs 2023-07-20 21:54:28 -07:00
Aliaksandr Valialkin
324a3c5288
lib/promscrape: follow-up after 6aa50ca954
- Improve docs
- Hide `debug relabeling` column when -promscrape.dropOriginalLabels command-line flag is set
- Inline the code from the added template functions, since the code is harder to follow
  with the template functions, especially when these functions have misleading names.
  Also, these functions are used only in one place, e.g. they do not reduce the amounts of code.
- Hide `click to show original labels` title at `labels` column when original labels aren't available.
- Show the reason on whey original labels aren't available at /service-discovery page.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4597
2023-07-20 21:54:09 -07:00
Aliaksandr Valialkin
c921bc0833
app/{vmselect,vlselect}: run make vmui-update vmui-logs-update after recent changes to VMUI
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4604
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4676
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4294
2023-07-20 21:53:51 -07:00
Yury Molodov
7757a31a73
fix: change getting serverUrl for vmui-logs (#4604) (#4677) 2023-07-20 21:53:30 -07:00
Yury Molodov
02a182e2a1
feat: change columns for active queries (#4676) 2023-07-20 21:53:12 -07:00
Yury Molodov
15e1d16afc
vmui: enhancements multiline field editing (#4294)
* fix: change textarea for relabel page

* feat: add comment for monaco theme

* fix: change behavior of multiline fields

* vmui: merge master
2023-07-20 21:52:53 -07:00
Aliaksandr Valialkin
f086a5ba63
docs/VictoriaLogs/CHANGELOG.md: document the addition of Grafana Loki push protocol support (aka Promtail)
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4482
2023-07-20 21:52:33 -07:00
Aliaksandr Valialkin
30098ac8bd
app/vlinsert/loki: follow-up after 09df5b66fd
- Parse protobuf if Content-Type isn't set to `application/json` - this behavior is documented at https://grafana.com/docs/loki/latest/api/#push-log-entries-to-loki

- Properly handle gzip'ped JSON requests. The `gzip` header must be read from `Content-Encoding` instead of `Content-Type` header

- Properly flush all the parsed logs with the explicit call to vlstorage.MustAddRows() at the end of query handler

- Check JSON field types more strictly.

- Allow parsing Loki timestamp as floating-point number. Such a timestamp can be generated by some clients,
  which store timestamps in float64 instead of int64.

- Optimize parsing of Loki labels in Prometheus text exposition format.

- Simplify tests.

- Remove lib/slicesutil, since there are no more users for it.

- Update docs with missing info and fix various typos. For example, it should be enough to have `instance` and `job` labels
  as stream fields in most Loki setups.

- Allow empty of missing timestamps in the ingested logs.
  The current timestamp at VictoriaLogs side is then used for the ingested logs.
  This simplifies debugging and testing of the provided HTTP-based data ingestion APIs.

The remaining MAJOR issue, which needs to be addressed: victoria-logs binary size increased from 13MB to 22MB
after adding support for Loki data ingestion protocol at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4482 .
This is because of shitty protobuf dependencies. They must be replaced with another protobuf implementation
similar to the one used at lib/prompb or lib/prompbmarshal .
2023-07-20 21:52:11 -07:00
Dmytro Kozlov
41f0227af2
docs: update information about tsdb usage in cluster version (#4678)
update information about tsdb usage in cluster version
2023-07-20 21:50:58 -07:00
hagen1778
1d2a0e0e10
docs: fix the next release version for vmalert
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-20 21:49:34 -07:00
Zakhar Bessarab
5b3000f7e8
docs: fix broken link to managed VM (#4679)
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-07-20 21:49:06 -07:00
hagen1778
5b963e1971
docs: mention streaming aggregation as way to reduce cardinality
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-20 21:48:09 -07:00
Alexander Marshalov
9ba03b4838
allow configuring staleness interval in stream aggregation (#4667) (#4670)
---------

Signed-off-by: Alexander Marshalov <_@marshalov.org>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-07-20 21:47:29 -07:00
hagen1778
f7d60613a9
docs: specify min version and limitations for vmalert's unit tests
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-20 21:27:50 -07:00
Aliaksandr Valialkin
a0b7def89d
app/vmselect/promql: fix tests after 781947a7e2 2023-07-20 21:25:30 -07:00
Haleygo
939c8b8372
vmalert: init unit test (#4596)
vmalert: support unit tests

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2945
---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2023-07-20 21:19:45 -07:00
hagen1778
a0cc6eb7a1
docs: mention the simplest way to migrate data in vmctl docs
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-20 21:12:18 -07:00
Dmytro Kozlov
f0d8f77e6d
app/vmagent: fix creating target id if --promscrape.dropOriginalLabels flag was used (#4616)
* app/vmagent: fix creating target id if `--promscrape.dropOriginalLabels` flag was used

* app/vmagent: hide links if OriginalLabels was dropped

* app/vmagent: update CHANGELOG.md and added information to the docs

* app/vmagent: fix comments
2023-07-20 19:21:41 -07:00
Zakhar Bessarab
7e24edd506
app/vlinsert/loki: fix compatibility with latest MetricsQL lib (#4675)
Loki uses default labels format without "or" operator. This format can't create a list of LabelFilters, so only first set of LabelFilters should be used.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-07-20 17:06:09 -07:00
Zakhar Bessarab
5b3cbd4db1
app/vlinsert: add support of loki push protocol (#4482)
* app/vlinsert: add support of loki push protocol

- implemented loki push protocol for both Protobuf and JSON formats
- added examples in documentation
- added example docker-compose

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vlinsert: move protobuf metric into its own file

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* deployment/docker/victorialogs/promtail: update reference to docker image

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* deployment/docker/victorialogs/promtail: make volume name unique

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vlinsert/loki: add license reference

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* deployment/docker/victorialogs/promtail: fix volume name

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs/VictoriaLogs/data-ingestion: add stream fields for loki JSON ingestion example

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vlinsert/loki: move entities to places where those are used

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vlinsert/loki: refactor to use common components

- use CommonParameters from insertutils
- stop ingestion after first error similar to elasticsearch and jsonline

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vlinsert/loki: address review feedback

- add missing logstorage.PutLogRows calls
- refactor tenant ID parsing to use common function
- reduce number of allocations for parsing by reusing  logfields slices
- add tests and benchmarks for requests processing funcs

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-07-20 16:49:43 -07:00
Aliaksandr Valialkin
5ec1ec39cd
docs/stream-aggregation.md: mention that sum_samples, stddev, stdvar, histogram_bucket and quantiles outputs must be applied only to gauge metrics 2023-07-20 11:31:38 -07:00
Aliaksandr Valialkin
0cbe5ccb4a
app/vmselect: rename promql.WriteActiveQueries() to promql.ActiveQueriesHandler()
This makes it more consistent with the rest of handlers inside app/vmselect/main.go

This is a follow-up for 6a96fd8ed5

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4598
2023-07-20 11:30:40 -07:00
Aliaksandr Valialkin
992c300ce9
all: replace atomic.Value with atomic.Pointer[T]
This eliminates the need in .(*T) casting for results obtained from Load()

Leave atomic.Value for map, since atomic.Pointer[map[...]...] makes double pointer to map,
because map is already a pointer type.
2023-07-19 17:48:26 -07:00
Roman Khavronenko
92cd98e366
docs: mention tags routing for new release (#4566)
* docs: mention tags routing for new release

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Update docs/Release-Guide.md

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-07-19 16:56:04 -07:00
Yury Molodov
44b4963ac6
feat: optimize vmui-log bundle size (#4602)
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-07-19 16:52:18 -07:00
Aliaksandr Valialkin
443c266406
app/vmalert/README.md: sync with docs/vmalert.md after 54b7bd4564 2023-07-19 16:31:30 -07:00
Aliaksandr Valialkin
29c6bcbd5b
app/vmselect: follow-up after 6a96fd8ed5
- Add `Active queries` chapter to VMUI docs
- Set `Content-Type: json` header inside promql.WriteActiveQueries() handler,
  in order to be consistent with other request handlers called at app/vmselect/main.go
- Pass the request to promql.WriteActiveQueries() handler, so it can change its output
  depending on the provided request params. This also improves consistency of
  promql.WriteActiveQueries() args with other request hanlers at app/vmselect/main.go

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4653
2023-07-19 16:31:26 -07:00
Yury Molodov
3ad80e281f
vmui: add Active Queries page (#4653)
* feat: add page to display a list of active queries (#4598)

* app/vmagent: code formatting

* fix: remove console

---------

Co-authored-by: dmitryk-dk <kozlovdmitriyy@gmail.com>
2023-07-19 16:02:58 -07:00
dependabot[bot]
9d55da5d26
build(deps-dev): bump word-wrap in /app/vmui/packages/vmui (#4664)
Bumps [word-wrap](https://github.com/jonschlinkert/word-wrap) from 1.2.3 to 1.2.4.
- [Release notes](https://github.com/jonschlinkert/word-wrap/releases)
- [Commits](https://github.com/jonschlinkert/word-wrap/compare/1.2.3...1.2.4)

---
updated-dependencies:
- dependency-name: word-wrap
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-19 15:43:59 -07:00