Commit graph

6838 commits

Author SHA1 Message Date
Aliaksandr Valialkin
ec375e2e15
docs/keyConcepts.md: clarify that the number of ,-delimited filters per each or group can be arbitrary
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3997
2023-07-16 23:47:15 -07:00
Aliaksandr Valialkin
8715e331c8
docs/VictoriaLogs/data-ingestion/README.md: document how to verify whether the data is successfully ingested into VictoriaLogs 2023-07-16 22:54:05 -07:00
Zakhar Bessarab
781947a7e2
metricsql: add support of using keep_metric_names for binary operations (#4109)
* metricsql: add support of using keep_metric_names for binary operations

This should help to avoid confusion with queries like one in the issue #3710.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* wip

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-07-16 03:01:27 -07:00
Artem Navoiev
30e84f4c66
docs: logs disable table of content for pages with no h2-h6, add aliases
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-16 00:08:06 -07:00
Artem Navoiev
6363a7bb80
docs: victorialogs add front-matter for readme pages
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-16 00:08:06 -07:00
Artem Navoiev
2963cbbb74
docs: victorialogs add front-matter for data-ingestion items
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-16 00:08:06 -07:00
Aliaksandr Valialkin
a7fdc3fcc7
all: add support for or filters in series selectors
This commit adds ability to select series matching distinct filters via a single series selector.
For example, the following selector selects series with either {env="prod",job="a"}
or {env="dev",job="b"} labels:

  {env="prod",job="a" or env="dev",job="b"}

The `or` filter is supported in all the VictoriaMetrics tools now.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3997
Uses https://github.com/VictoriaMetrics/metricsql/pull/14
2023-07-15 23:56:18 -07:00
Aliaksandr Valialkin
6a65af5112
all: replace ElasticSearch -> Elasticsearch for the sake of consistency
This is a follow-up for 7f6b5dc47b
2023-07-14 10:52:43 -07:00
Aliaksandr Valialkin
d721109961
docs/CHANGELOG.md: sync with master branch 2023-07-14 10:48:40 -07:00
Aliaksandr Valialkin
99ec7ef28f
docs/VictoriaLogs: use exact number when comparing efficiency of VictoriaLogs and Elasticsearch 2023-07-14 10:47:29 -07:00
Haleygo
5e5c805599
vmalert: fix evalTS after modify group interval (#4629) 2023-07-14 10:47:29 -07:00
subham sarkar
9d3cfc03cd
docs: Update README.md (#4613)
Fix grammatical mistakes and also s/ElasticSearch/Elasticsearch
2023-07-14 10:46:13 -07:00
Artem Navoiev
14d21cfc73
githubaciton: sync-docs change set output code as current one will be deprecated soon
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-14 10:44:04 -07:00
Artem Navoiev
c245e38f8d
fix alias
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-14 10:44:04 -07:00
Artem Navoiev
6b98511ac9
add front-matter for some of victorialogs pages
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-14 10:44:04 -07:00
Artem Navoiev
4c78a9cc90
Update .github/workflows/sync-docs.yml
Co-authored-by: Max Golionko <8kirk8@gmail.com>
2023-07-14 10:44:04 -07:00
Artem Navoiev
ec936262d1
enable only when changes in docs
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-14 10:44:04 -07:00
Artem Navoiev
bbc9e349bd
moar debug
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-14 10:44:04 -07:00
Artem Navoiev
f85611859c
fix path to main repo
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-14 10:44:04 -07:00
Artem Navoiev
abc737747e
more debug
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-14 10:44:04 -07:00
Artem Navoiev
64718a3a49
move debug
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-14 10:44:04 -07:00
Artem Navoiev
953f146091
remove env
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-14 10:44:04 -07:00
Artem Navoiev
8cad820526
debug workflow1
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-14 10:44:04 -07:00
Artem Navoiev
f01dea5728
add workdir to gpg step
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-14 10:44:04 -07:00
Artem Navoiev
c75202657c
add job to sync the docs
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-14 10:39:11 -07:00
Roman Khavronenko
89b5a6a4d5
vmctl: mention replicationFactor during migration (#4633)
Addresses https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4624

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-14 10:36:46 -07:00
Aliaksandr Valialkin
3d23fd9853
lib/storage: move series registration in caches from createAllIndexesForMetricName into a separate function - putSeriesToCache
This makes the code more clear and easier to read

This is a follow-up for 7094fa38bc
2023-07-13 23:17:14 -07:00
dependabot[bot]
a15a66ee89
build(deps): bump tough-cookie in /app/vmui/packages/vmui (#4603)
Bumps [tough-cookie](https://github.com/salesforce/tough-cookie) from 4.1.2 to 4.1.3.
- [Release notes](https://github.com/salesforce/tough-cookie/releases)
- [Changelog](https://github.com/salesforce/tough-cookie/blob/master/CHANGELOG.md)
- [Commits](https://github.com/salesforce/tough-cookie/compare/v4.1.2...v4.1.3)

---
updated-dependencies:
- dependency-name: tough-cookie
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-13 22:22:53 -07:00
Aliaksandr Valialkin
4b86522f4c
lib/mergeset: skip common prefix in binarySearchKey() function
This should improve performance a bit when the search if performed among items with long common prefix
2023-07-13 22:05:14 -07:00
Aliaksandr Valialkin
203a436066
lib/storage: optimize BenchmarkIndexDBGetTSIDs()
- Sort MetricName tags only once before the benchmark loop.
- Obtain indexSearch per each benchmark loop in order to give a chance for background merge
  for the recently created parts
2023-07-13 21:49:54 -07:00
Aliaksandr Valialkin
fbddb4ad32
lib/storage: typo fix after e1cf962bad
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2698
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401
2023-07-13 21:29:02 -07:00
Aliaksandr Valialkin
7d359d17d1
lib/storage: properly free up resources from newTestStorage() by calling stopTestStorage() 2023-07-13 17:13:34 -07:00
Aliaksandr Valialkin
e1cf962bad
lib/storage: switch from global to per-day index for MetricName -> TSID mapping
Previously all the newly ingested time series were registered in global `MetricName -> TSID` index.
This index was used during data ingestion for locating the TSID (internal series id)
for the given canonical metric name (the canonical metric name consists of metric name plus all its labels sorted by label names).

The `MetricName -> TSID` index is stored on disk in order to make sure that the data
isn't lost on VictoriaMetrics restart or unclean shutdown.

The lookup in this index is relatively slow, since VictoriaMetrics needs to read the corresponding
data block from disk, unpack it, put the unpacked block into `indexdb/dataBlocks` cache,
and then search for the given `MetricName -> TSID` entry there. So VictoriaMetrics
uses in-memory cache for speeding up the lookup for active time series.
This cache is named `storage/tsid`. If this cache capacity is enough for all the currently ingested
active time series, then VictoriaMetrics works fast, since it doesn't need to read the data from disk.

VictoriaMetrics starts reading data from `MetricName -> TSID` on-disk index in the following cases:

- If `storage/tsid` cache capacity isn't enough for active time series.
  Then just increase available memory for VictoriaMetrics or reduce the number of active time series
  ingested into VictoriaMetrics.

- If new time series is ingested into VictoriaMetrics. In this case it cannot find
  the needed entry in the `storage/tsid` cache, so it needs to consult on-disk `MetricName -> TSID` index,
  since it doesn't know that the index has no the corresponding entry too.
  This is a typical event under high churn rate, when old time series are constantly substituted
  with new time series.

Reading the data from `MetricName -> TSID` index is slow, so inserts, which lead to reading this index,
are counted as slow inserts, and they can be monitored via `vm_slow_row_inserts_total` metric exposed by VictoriaMetrics.

Prior to this commit the `MetricName -> TSID` index was global, e.g. it contained entries sorted by `MetricName`
for all the time series ever ingested into VictoriaMetrics during the configured -retentionPeriod.
This index can become very large under high churn rate and long retention. VictoriaMetrics
caches data from this index in `indexdb/dataBlocks` in-memory cache for speeding up index lookups.
The `indexdb/dataBlocks` cache may occupy significant share of available memory for storing
recently accessed blocks at `MetricName -> TSID` index when searching for newly ingested time series.

This commit switches from global `MetricName -> TSID` index to per-day index. This allows significantly
reducing the amounts of data, which needs to be cached in `indexdb/dataBlocks`, since now VictoriaMetrics
consults only the index for the current day when new time series is ingested into it.

The downside of this change is increased indexdb size on disk for workloads without high churn rate,
e.g. with static time series, which do no change over time, since now VictoriaMetrics needs to store
identical `MetricName -> TSID` entries for static time series for every day.

This change removes an optimization for reducing CPU and disk IO spikes at indexdb rotation,
since it didn't work correctly - see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401 .

At the same time the change fixes the issue, which could result in lost access to time series,
which stop receving new samples during the first hour after indexdb rotation - see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2698

The issue with the increased CPU and disk IO usage during indexdb rotation will be addressed
in a separate commit according to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1401#issuecomment-1553488685

This is a follow-up for 1f28b46ae9
2023-07-13 17:03:50 -07:00
Aliaksandr Valialkin
1bce67df06
lib/storage: fix possible test failure in TestStorageAddRowsConcurrent
The number of parts in the snapshot partition may be zero if concurrent goroutine just
started creating new partition, but didn't put data into it yet when the current
goroutine made a snapshot.
2023-07-13 15:03:51 -07:00
Aliaksandr Valialkin
733032e514
lib/mergeset: simplify fulsuhInmemoryParts() a bit 2023-07-13 12:33:43 -07:00
Aliaksandr Valialkin
df67b78f75
docs/CHANGELOG.md: clarify the description of the bugfix at 177a0c1ca9
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4555
2023-07-13 12:19:00 -07:00
Aliaksandr Valialkin
3e5e55d235
Revert "make: add goimports task (#4582)"
This reverts commit 20b18e9feb.

Reason for revert: running goimports on `make check-all` introduces the following issues:

- It runs only on modified files, which weren't commited yet into git repository.
  This means the formatting for the remaining files becomes different comparing to the formatting
  for the changed files. This also means that the goimports has no any effect
  at github actions and when the changed code is already commited to git repository.
- `gomiports` performs formatting in the same way as gofmt, so `make fmt` becomes unnecessary.
  But when `gofmt` is substituted with `goimports`, then it performs unnecessary formatting for *.qtpl.go files.
  It is possible to make a hack, which will prepare a list of all the *.go files at lib/ and app/
  without the *.qtpl.go files, and then feed this list to `goimports`, but this looks too fragile
  for the task of just fixing the ordering of Go imports.

So it is better to leave source code formatting as is with `gofmt`, while manually fixing improper ordering
of Go import from time to time in dedicated commits until better solution arises.
2023-07-13 12:18:02 -07:00
Zakhar Bessarab
64149782ac
make: add goimports task (#4582)
* make: add goimports task

Adds task to fix imports formatting implace.
Formats imports into:
- native library
- external libraries
- local packages based on github.com/VictoriaMetrics/VictoriaMetrics prefix

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* make: add goimports install task

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* make: run goimports only for changed files

Applying goimports to all existing files would create a lot of problems with cherry-picking changes between different branches used for development. To avoid this it was decided to only run goimports on changed files to fix formatting gradually.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* make: update goimports to run on all changed files

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-07-13 12:17:31 -07:00
Aliaksandr Valialkin
650af7c5ca
app/vmalert: silence golagci-lint at TestAlertingRule_Template
Add a break if gotAlert is nil

This removes the following golangci-lint warning:

app/vmalert/alerting_test.go:868:8: SA5011(related information): this check suggests that the pointer can be nil (staticcheck)
				if gotAlert == nil {
				   ^
2023-07-13 12:16:00 -07:00
Dmytro Kozlov
f31ac064f9
app/vmctl: fix panic --remote-read-filter-time-start flag not defined (#4605)
* app/vmctl: fix panic `--remote-read-filter-time-start` flag not defined

* app/vmctl: update CHANGELOG.md

---------

Co-authored-by: Nikolay <nik@victoriametrics.com>
2023-07-13 12:13:21 -07:00
Dmytro Kozlov
555a0a9d57
app/vmctl: fix issue with adding many seconds (#4617)
* app/vmctl: fix issue with adding many seconds

* app/vmagent: add CHANGELOG.md
2023-07-13 12:09:54 -07:00
Roman Khavronenko
fdccb56620
vmalert: check for negative offset for missed rounds (#4628)
It could happen for low evaluation intervals and irregular
delays during execution that evaluation time would get
a negative offset. This could result into cumulative
discrepancy between the actual time and evaluation time for rules.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-13 12:05:52 -07:00
Dmytro Kozlov
3d0f846a79
lib/logstorage: fix panic (#4620) 2023-07-13 12:04:59 -07:00
Alexander Marshalov
09cc17a6b1
fixed typo (#4622)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-07-13 11:29:57 -07:00
Alexander Marshalov
7a5e5f6a89
add info about using stream aggregation as statsd alternative (#4600) (#4621)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-07-13 11:29:57 -07:00
Aliaksandr Valialkin
d8b8fc0343
lib/logstorage: fix TestValuesEncoder() on 32-bit architectures 2023-07-13 11:28:04 -07:00
Aliaksandr Valialkin
2636c35cec
docs/VictoriaLogs/FAQ.md: small fixes 2023-07-12 01:10:40 -07:00
Aliaksandr Valialkin
b07a1c85b9
all: update Go builder from 1.20.5 to 1.20.6
See https://github.com/golang/go/issues?q=milestone%3AGo1.20.6+label%3ACherryPickApproved
2023-07-12 01:00:24 -07:00
Aliaksandr Valialkin
bcea7a0a92
docs/keyConcepts.md: cosmetic fixes after b67bd156d5 2023-07-12 00:30:33 -07:00
Aliaksandr Valialkin
b99fcb7d7a
docs/VictoriaLogs: add FAQ 2023-07-12 00:30:33 -07:00