Commit graph

9538 commits

Author SHA1 Message Date
hagen1778
73b073e298
docs/CHANGELOG.md: cut v1.97.10
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-18 14:30:26 +02:00
hagen1778
361afaec5b
docs/CHANGELOG.md: cut v1.105.0
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-18 14:28:14 +02:00
hagen1778
a710d43a20
app/{vmselect,vlselect}: run make vmui-update vmui-logs-update
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-18 14:26:47 +02:00
hagen1778
f9c79eba30
docs: re-order changes by priority in log
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-18 14:24:20 +02:00
Zhu Jiekun
8c50c38a80
vmstorage: auto calculate maxUniqueTimeseries based on resources (#6961)
### Describe Your Changes

Add support for
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6930

Calculate `-search.maxUniqueTimeseries` by
`-search.maxConcurrentRequests` and remaining memory if it's **not set**
or **less equal than 0**.

The remaining memory is affected by `-memory.allowedPercent`,
`-memory.allowedBytes` and cgroup memory limit.
### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

(cherry picked from commit 85f60237e2)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-18 14:00:14 +02:00
Andrii Chubatiuk
965a33c893
lib/promscrape: fixed reload on max_scrape_size change (#7282)
### Describe Your Changes

fixed reload on max_scrape_size change
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7260

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-10-18 11:35:23 +02:00
Hui Wang
c4fe23794a
vmalert: fix blocking hot-reload process if the old rule group hasn't started yet (#7258)
Group
[sleeps](daa7183749/app/vmalert/rule/group.go (L320))
random duration before start the evaluation, and during the sleep,
`g.updateCh <- new` will be blocked since there is no `<-g.updateCh`
waiting.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-10-18 11:18:24 +02:00
hagen1778
41e0bbb6d1
docs/vmctl: clarify the meaning of the comment
The comment was ambiguous and not clear to the readers.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-18 09:46:54 +02:00
Aliaksandr Valialkin
025eec2cb0
deployment: update VictoriaLogs from v0.36.0-victorialogs to v0.37.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.37.0-victorialogs
2024-10-18 02:40:13 +02:00
Aliaksandr Valialkin
14e33d93ef
app/vlselect/vmui: run make vmui-logs-update after 423df09d7d
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7206
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7117

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7167
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7133
2024-10-18 02:33:41 +02:00
Aliaksandr Valialkin
51cd3ba02b
docs/VictoriaLogs/CHANGELOG.md: cut v0.37.0-victorialogs release 2024-10-18 02:32:37 +02:00
Yury Molodov
423df09d7d
vmui/logs: add ability to hide hits chart (#7206)
### Describe Your Changes

**Added ability to hide the hits chart**

- Users can now hide or show the hits chart by clicking the "eye" icon
located in the upper-right corner of the chart.
- When the chart is hidden, it will stop sending requests to
`/select/logsql/hits`.
- Upon displaying the chart again, it will automatically refresh. If a
relative time range is set, the chart will update according to the time
period of the logs currently being displayed.

**Hits chart visible:**

![image](https://github.com/user-attachments/assets/577e877b-6417-4b83-8d84-c55e3d39864a)

**Hits chart hidden:**

![image](https://github.com/user-attachments/assets/068b1143-d140-4d72-8d65-663900124f32)

Related issue: #7117

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-10-18 02:30:56 +02:00
Yury Molodov
36a86c3aaf
vmui/logs: fix display of hits chart (#7167)
### Describe Your Changes

Fixed the display of hits chart in VictoriaLogs.
See #7133

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-10-18 02:28:23 +02:00
Aliaksandr Valialkin
064b9a6314
docs/VictoriaLogs/CHANGELOG.md: remove "index.html" trailer from the link to docs for the sake of consistency with other links to docs
This is a follow-up for 3538869942
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7252
2024-10-18 02:26:01 +02:00
Aliaksandr Valialkin
0f24078146
lib/logstorage: use simpler in-memory cache instead of workingsetcache for caching recently ingested _stream values and recently queried set of streams
These caches aren't expected to grow big, so it is OK to use the most simplest cache based on sync.Map.
The benefit of this cache compared to workingsetcache is better scalability on systems with many CPU cores,
since it doesn't use mutexes at fast path.
An additional benefit is lower memory usage on average, since the size of in-memory cache equals
working set for the last 3 minutes.

The downside is that there is no upper bound for the cache size, so it may grow big during workload spikes.
But this is very unlikely for typical workloads.
2024-10-18 02:22:43 +02:00
Aliaksandr Valialkin
8aa144fa74
lib/logstorage: do not persist streamIDCache, since it may go out of sync with partition directories, which can be changed manually between VictoriaLogs restarts
Partition directories can be manually deleted and copied from another sources such as backups or other VitoriaLogs instances.
In this case the persisted cache becomes out of sync with partitions. This can result in missing index entries
during data ingestion or in incorrect results during querying. So it is better to do not persist caches.
This shouldn't hurt VictoriaLogs performance just after the restart too much, since its caches usually contain
small amounts of data, which can be quickly re-populated from the persisted data.
2024-10-18 02:22:43 +02:00
Aliaksandr Valialkin
1892e357c3
lib/logstorage: consistently use "pHits := m[..]" pattern
Consistency improves maintainability of the code a bit.
2024-10-18 02:22:43 +02:00
Aliaksandr Valialkin
2023f017b1
lib/logstorage: optimize performance for queries, which select all the log fields for logs containing hundreds of log fields (aka "wide events")
Unpack the full columnsHeader block instead of unpacking meta-information per each individual column
when the query, which selects all the columns, is executed. This improves performance when scanning
logs with big number of fields.
2024-10-18 02:22:42 +02:00
Aliaksandr Valialkin
78c6fb0883
lib/logstorage: improve performance of top and field_values pipes on systems with many CPU cores
- Parallelize mering of per-CPU results.
- Parallelize writing the results to the next pipe.
2024-10-18 02:22:42 +02:00
Aliaksandr Valialkin
c4b2fdff70
lib/logstorage: optimize 'stats by(...)' calculations for by(...) fields with millions of unique values on multi-CPU systems
- Parallelize merging of per-CPU `stats by(...)` result shards.
- Parallelize writing `stats by(...)` results to the next pipe.
2024-10-18 02:22:41 +02:00
Aliaksandr Valialkin
192c07f76a
lib/logstorage: optimize performance for top pipe when it is applied to a field with millions of unique values
- Use parallel merge of per-CPU shard results. This improves merge performance on multi-CPU systems.
- Use topN heap sort of per-shard results. This improves performance when results contain millions of entries.
2024-10-18 02:21:56 +02:00
hagen1778
98fcd95438
docs/vmagent: distinguish between metrics, samples and series
Before, doc incorrectly used `metric` instead of `sample` or `series`.
This commit aligns description with https://docs.victoriametrics.com/keyconcepts/#structure-of-a-metric

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-17 19:21:52 +02:00
Artem Fetishev
d6bafe31d3
docs/troubleshooting: add reduce_mem_usage=1 to export query (#7286)
### Describe Your Changes

When debugging unexpected query results, add reduce_mem_usage=1 param to
export query to preserve duplicates.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2024-10-17 18:50:25 +02:00
Fred Navruzov
bc65c9f399
docs/vmanomaly: release v1.17.0 (#7285)
### Describe Your Changes

docs/vmanomaly: release v1.17.0

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-10-17 18:56:04 +03:00
Nikolay
635bdd130b
lib/storage: properly unmarshal SearchQuery (#7277)
After adding multitenant query feature at v1.104.0, searchQuery wasn't
properly unmarshalled at bottom vmselect in multi-level cluster setup.
It resulted into empty query responses.

This commit adds fallback to Unmarshal method of SearchQuery to fill
TenantTokens. It allows to properly execute search requests
at vmselect side.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7270

---------

Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-10-17 10:52:35 -03:00
Zakhar Bessarab
d036063c78
docs/vmbackup: add information about cluster backups (#7244)
### Describe Your Changes

Add more detailed information about performing backups for
VictoriaMetrics cluster setup.
More detailed explanation should help to address questions similar to
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7225

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-10-17 13:56:23 +02:00
hagen1778
aa6c237603
docs: follow-up after f0d1db81dc
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-17 13:49:51 +02:00
Roman Khavronenko
05ac508fbf
lib/flagutil: rename Duration to RetentionDuration (#7284)
The purpose of this change is to reduce confusion between using
`flag.Duration` and `flagutils.Duration`. The reason is that
`flagutils.Duration` was mistakenly used for cases that required `m`
support. See
ab0d31a7b0

The change in name should clearly indicate the purpose of this data
type.

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-17 13:47:48 +02:00
Alexander Frolov
f0d1db81dc
lib/flagutil: rm misleading minutes support from flagutil.Duration docs (#7066)
### Describe Your Changes

`flagutil.Duration` docs state that `m` suffix stands for `minute`, but
in fact this suffix is not supported due to ambiguity with `month`

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Alexander Frolov <winningpiece@gmail.com>
2024-10-17 13:29:51 +02:00
Hui Wang
ab0d31a7b0
vmagent: fix type of command-line flag -streamAggr.dedupInterval (#7081)
Previously unit `m` is not correctly supported.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-10-17 13:27:59 +02:00
Artem Fetishev
ca787c70d1
dashboards: fix vmagent monitoring chart descriptions (#7283)
### Describe Your Changes

Fix vmagent monitoring chart descriptions
### Checklist

The following checks are **mandatory**:

- [x ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Artem Fetishev <rtm@victoriametrics.com>
2024-10-17 12:12:47 +02:00
Zakhar Bessarab
65e9d19f3c
lib/flagutil/dict: properly update default value in case there is no key value set (#7211)
### Describe Your Changes

If a dict flag has only one value without a prefix it is supposed to
replace default value.

Previously, when flag was set to `-flag=2` and the default value in
`NewDictInt` was set to 1 the resulting value for any `flag.Get()` call
would be 1 which is not expected.

This commit updates default value for the flag in case there is only one
entry for flag and the entry is a number without a key.

This affects cluster version and specifically `replicationFactor` flag
usage with vmstorage [node
groups](https://docs.victoriametrics.com/cluster-victoriametrics/#vmstorage-groups-at-vmselect).
Previously, the following configuration would effectively be ignored:
```
/path/to/vmselect \
 -replicationFactor=2 \
 -storageNode=g1/host1,g1/host2,g1/host3 \
 -storageNode=g2/host4,g2/host5,g2/host6 \
 -storageNode=g3/host7,g3/host8,g3/host9
```

Changes from this PR will force default value for `replicationFactor`
flag to be set to `2` which is expected as the result of this
configuration.


---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2024-10-17 12:05:47 +02:00
hagen1778
23f8ab6f81
docs/contirubting: clarify the type of changelog line we expect
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-17 11:59:55 +02:00
Andrii Chubatiuk
3538869942
vlogs: added basic alerts (#7252)
### Describe Your Changes

Added basic VLogs alerts

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-10-17 11:33:06 +02:00
Hui Wang
4984e71da6
vmalert-tool: add more syntax checks for input_series and exp_samples (#7263)
address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7224,
allow using
```
        exp_samples:
          - labels: '{}'
```
for prometheus compatibility.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-10-17 11:00:34 +02:00
Hui Wang
c90adf566e
vmalert-tool: reduce victoriametrics health check interval (#7256)
address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6970.
This reduces the hard limit on duration for completing the test when
users run vmalert-tool on slow hosts.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-10-17 10:51:12 +02:00
Github Actions
c5fb281019
Automatic update helm docs from VictoriaMetrics/helm-charts@845bc1f (#7268)
Automated changes by
[create-pull-request](https://github.com/peter-evans/create-pull-request)
GitHub action

Signed-off-by: Github Actions <133988544+victoriametrics-bot@users.noreply.github.com>
Co-authored-by: f41gh7 <18450869+f41gh7@users.noreply.github.com>
2024-10-17 10:42:41 +02:00
Aliaksandr Valialkin
a72e1155b9
docs/VictoriaLogs/CHANGELOG.md: add missing part of the sentence 2024-10-16 20:22:19 +02:00
Aliaksandr Valialkin
677f1cd1be
docs/VictoriaLogs/CHANGELOG.md: typo fix: refer the correct endpoints for stats results 2024-10-16 20:19:22 +02:00
Aliaksandr Valialkin
9187ed0648
deployment: update VictoriaLogs Docker image from v0.35.0-victorialogs to v0.36.0-victorialogs 2024-10-16 20:10:05 +02:00
Aliaksandr Valialkin
6ca1b15134
docs/VictoriaLogs/README.md: fix copy-n-paste typo: partitions in VictoriaLogs are per-day, not per-month 2024-10-16 20:08:30 +02:00
Aliaksandr Valialkin
91987763d4
docs/VictoriaLogs/CHANGELOG.md: cut v0.36.0-victorialogs release 2024-10-16 20:00:35 +02:00
Aliaksandr Valialkin
a23aa87282
app/vlselect/vmui: run make vmui-logs-update after the commit 6c9772b101
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7204
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7097
2024-10-16 19:58:14 +02:00
Aliaksandr Valialkin
508e498ae3
lib/logstorage: follow-up for 72941eac36
- Allow dropping metrics if the query result contains at least a single metric.
- Allow copying by(...) fields.
- Disallow overriding by(...) fields via `math` pipe.
- Allow using `format` pipe in stats query. This is useful for constructing some labels from the existing by(...) fields.
- Add more tests.
- Remove the check for time range in the query filter according to https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7254/files#r1803405826

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7254
2024-10-16 19:43:52 +02:00
Hui Wang
72941eac36
victorialogs: add more checks for stats query APIs (#7254)
1. Verify if field in [fields
pipe](https://docs.victoriametrics.com/victorialogs/logsql/#fields-pipe)
exists. If not, it generates a metric with illegal float value "" for
prometheus metrics protocol.
2. check if multiple time range filters produce conflicted query time
range, for instance:
```
query: _time: 5m | stats count(), 
start:2024-10-08T10:00:00.806Z, 
end: 2024-10-08T12:00:00.806Z, 
time: 2024-10-10T10:02:59.806Z
```
must give no result due to invalid final time range.

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-10-16 19:25:43 +02:00
Aliaksandr Valialkin
202eb429a7
lib/logstorage: refactor storage format to be more efficient for querying wide events
It has been appeared that VictoriaLogs is frequently used for collecting logs with tens of fields.
For example, standard Kuberntes setup on top of Filebeat generates more than 20 fields per each log.
Such logs are also known as "wide events".

The previous storage format was optimized for logs with a few fields. When at least a single field
was referenced in the query, then the all the meta-information about all the log fields was unpacked
and parsed per each scanned block during the query. This could require a lot of additional disk IO
and CPU time when logs contain many fields. Resolve this issue by providing an (field -> metainfo_offset)
index per each field in every data block. This index allows reading and extracting only the needed
metainfo for fields used in the query. This index is stored in columnsHeaderIndexFilename ( columns_header_index.bin ).
This allows increasing performance for queries over wide events by 10x and more.

Another issue was that the data for bloom filters and field values across all the log fields except of _msg
was intermixed in two files - fieldBloomFilename ( field_bloom.bin ) and fieldValuesFilename ( field_values.bin ).
This could result in huge disk read IO overhead when some small field was referred in the query,
since the Operating System usually reads more data than requested. It reads the data from disk
in at least 4KiB blocks (usually the block size is much bigger in the range 64KiB - 512KiB).
So, if 512-byte bloom filter or values' block is read from the file, then the Operating System
reads up to 512KiB of data from disk, which results in 1000x disk read IO overhead. This overhead isn't visible
for recently accessed data, since this data is usually stored in RAM (aka Operating System page cache),
but this overhead may become very annoying when performing the query over large volumes of data
which isn't present in OS page cache.

The solution for this issue is to split bloom filters and field values across multiple shards.
This reduces the worst-case disk read IO overhead by at least Nx where N is the number of shards,
while the disk read IO overhead is completely removed in best case when the number of columns doesn't exceed N.
Currently the number of shards is 8 - see bloomValuesShardsCount . This solution increases
performance for queries over large volumes of newly ingested data by up to 1000x.

The new storage format is versioned as v1, while the old storage format is version as v0.
It is stored in the partHeader.FormatVersion.

Parts with the old storage format are converted into parts with the new storage format during background merge.
It is possible to force merge by querying /internal/force_merge HTTP endpoint - see https://docs.victoriametrics.com/victorialogs/#forced-merge .
2024-10-16 17:35:07 +02:00
Roman Khavronenko
1d637667a6
vmui: clarify the info for TotalSeries stat (#7271)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-16 15:15:28 +02:00
rusttech
87910e4fa8
app/vmctl: fixes opentsdb source metric tags
Previously it was incorrectly used append for pre-allocated slice of labels.

This commit fixes slice append by allocating zero length slice with needed capacity.

---------

Co-authored-by: Nikolay <nik@victoriametrics.com>
2024-10-16 10:35:17 +02:00
hagen1778
e347d90531
docs: update anchor level to fix menu rendering in changelog
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-15 20:40:58 +02:00
Yury Molodov
86029de0d4
vmui: fix alert display with long messages (#7228)
### Describe Your Changes

Fix `Alert` component to prevent it from overflowing the screen when
displaying long messages.

Related issue: #7207

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-10-15 16:35:57 +02:00