Commit graph

8595 commits

Author SHA1 Message Date
hagen1778
a41c2aeafb
app/(vminsert|vmagent): fix label names for newrelic metrics
Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 7cf615a73b)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-05 15:08:07 +02:00
Dmytro Kozlov
a89cf4ce12
app/(vminsert|vmagent): add support for new relic infrastructure agent (#4712)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit f60c08a7bd)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-05 15:06:05 +02:00
Zakhar Bessarab
99582f19d2
docs: add example service configuration for windows (#5129)
Use service configuration example from https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3781#issuecomment-1424671624

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

(cherry picked from commit cc7d5b7bab)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-05 15:00:50 +02:00
Denys Holius
f42c12e69a
docs: update social links
(cherry picked from commit 3d5d62e38a)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-05 15:00:12 +02:00
Github Actions
c488f225ec
Automatic update operator docs from VictoriaMetrics/operator@26d1b3b (#5130)
(cherry picked from commit b576acff5e)
2023-10-05 14:59:51 +02:00
Aliaksandr Valialkin
c3908350ea
docs/VictoriaLogs/Roadmap.md: add integration with Grafana 2023-10-05 11:41:01 +02:00
Zakhar Bessarab
07586c1d8f
docs/enterprise: add info about license flags (#5085)
* docs/enterprise: add info about license flags

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs/enterprise: add more examples on how to run enterprise components

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs/enterprise: address review feedback

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs/enterprise: specify release version

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs/enterprise: fix image tag in example

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs/enterprise: add docker-compose example, fix secret name

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-10-05 11:39:22 +02:00
Artem Navoiev
5c87066f8c
docs: victorialogs change the doc ordering.2
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-10-05 11:37:01 +02:00
Artem Navoiev
8f96709219
docs: victorialogs change the doc ordering
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-10-05 11:36:27 +02:00
Github Actions
380a5d2c9f
Automatic update operator docs from VictoriaMetrics/operator@5da4bf6 (#5127) 2023-10-05 11:35:16 +02:00
Aliaksandr Valialkin
7ccefffe27
docs/VictoriaLogs/CHANGELOG.md: add release date for v0.4.1-victorialogs 2023-10-04 17:57:25 +02:00
Github Actions
e20f969ed3
Automatic update operator docs from VictoriaMetrics/operator@9fb79f4 (#5124) 2023-10-04 17:57:25 +02:00
Aliaksandr Valialkin
e275be05b0
docs/vmbackup.md: mention that -filestream.disableFadvise command-line flag can be used for reducing CPU usage on systems with big number of CPU cores
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5120

This is a follow-up for 75dd7b30ba
2023-10-04 16:38:53 +02:00
Zakhar Bessarab
dbf0aa4b87
docs/vmbackupmanager: add missing step for restore flow in k8s (#5116)
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-10-04 16:38:53 +02:00
Aliaksandr Valialkin
4e5d3d38f8
lib/filestream: add -filestream.disableFadvise syscall for unconditional disabling of fadvise syscall
This may be needed in rare cases when performing backups on systems with big number of CPU cores
and big value passed to -concurrency command-line flag.

See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5120
2023-10-04 16:30:19 +02:00
Artem Navoiev
7a5a40e031
docs: fix wieghts for logs folders
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-10-04 16:29:31 +02:00
Artem Navoiev
1a2af79047
docs: remove id clashes for pages, properly cover victorialogs changelog page (#5123)
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-10-04 16:28:52 +02:00
Artem Navoiev
12b0e3e5cb
add hugo front matter for operator docs (#5122)
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-10-04 16:28:23 +02:00
Aliaksandr Valialkin
65e7a7cabb
deployment/docker: update VictoriaLogs from v0.4.0-victorialogs to v0.4.1-victorialogs
See https://docs.victoriametrics.com/VictoriaLogs/CHANGELOG.html#v041
2023-10-04 16:27:39 +02:00
f41gh7
10569b9d80
cut v0.4.1-victorialogs release 2023-10-04 14:30:22 +02:00
Aliaksandr Valialkin
776174f1cd
docs/Single-server-VictoriaMetrics.md: follow-up for 612ec2df6b
Move the description of JSON line format into a separate chapter

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5083
2023-10-03 17:53:15 +02:00
hagen1778
297f63a01e
alerting: account for vmauth component for alerts ServiceDown and TooManyRestarts
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-03 17:52:43 +02:00
Alexander Marshalov
44b51808ec
hide deprecated docs from menu (#5095) 2023-10-03 17:52:15 +02:00
Zakhar Bessarab
9f5e6fe0eb
docs/victorialogs: changelog followup for 650d72f3 (#5114)
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-10-03 17:50:49 +02:00
Zakhar Bessarab
85b604d414
lib/logstorage: fix free space check (#5113)
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-10-03 17:50:16 +02:00
hagen1778
c0b478d47f
docs: add more details explaining JSON line format
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5083

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-03 17:48:33 +02:00
Aliaksandr Valialkin
dc71db9470
deployment: update VictoriaLogs from v0.3.0-victorialogs to v0.4.0-victorialogs
See https://docs.victoriametrics.com/VictoriaLogs/CHANGELOG.html#v040
2023-10-03 02:34:28 +02:00
Aliaksandr Valialkin
ff471d9f2c
docs/VictoriaLogs/CHANGELOG.md: cut v0.4.0-victorialogs 2023-10-03 01:46:42 +02:00
Aliaksandr Valialkin
fcd43f11f5
deployment: update VictoriaMetrics from v1.93.5 to v1.94.0
See https://docs.victoriametrics.com/CHANGELOG.html#v1940
2023-10-03 01:40:21 +02:00
Aliaksandr Valialkin
cf93d27bca
docs/CHANGELOG.md: cut v1.94.0 2023-10-02 22:33:40 +02:00
Aliaksandr Valialkin
f0ff9c6f8c
docs/Articles.md: add a video from GopherCon 2023 for Writing a TSDB from scratch: performance optimization talk 2023-10-02 22:24:48 +02:00
Aliaksandr Valialkin
47741cd935
deployment/docker: update Alpine from 3.18.3 to 3.18.4
See https://alpinelinux.org/posts/Alpine-3.18.4-released.html
2023-10-02 21:54:57 +02:00
Aliaksandr Valialkin
cd1169618f
vendor: run make vendor-update 2023-10-02 21:53:35 +02:00
Aliaksandr Valialkin
b5812e2457
app/{vmselect,vlselect}: run make vmui-update vmui-logs-update 2023-10-02 21:44:21 +02:00
Yury Molodov
f5521ce4d4
vmui: add storage for query history (#5022)
* vmui: add storage for query history

* docs/vmui: add storage for query history
2023-10-02 21:41:32 +02:00
Roman Khavronenko
1f2cb594d9
lib/promscrape: make concurrency control optional (#5073)
* lib/promscrape: make concurrency control optional

Before, `-maxConcurrentInserts` was limiting all calls to `promscrape.Parse`
function: during ingestion and scraping. This behavior is incorrect.
Cmd-line flag `-maxConcurrentInserts` should have effect onl on ingestion.

Since both pipelines use the same `promscrape.Parse` function, we extend it
to make concurrency limiter optional. So caller can decide whether concurrency
should be limited or not.

This commit makes c53b5788b4
obsolete.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Revert "dashboards: move `Concurrent inserts` panel to Troubleshooting section"

This reverts commit c53b5788b4.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-02 21:34:41 +02:00
Yury Molodov
40973d37da
vmui: improve the appearance of the trace (#5091) 2023-10-02 21:24:51 +02:00
Dmytro Kozlov
d8d003af24
app/vmagent: fix check of the DataDog agent path requests when requests have trailing slashes (#5106)
* app/vmagent: fix check of the DataDog agent path requests when requests have trailing slashes

* app/vmagent: fix CHANGELOG.md description

* wip

* wip

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-10-02 21:18:47 +02:00
Aliaksandr Valialkin
d80ccf52a0
Revert "lib/promscrape: add metric vm_promscrape_scrapes_skipped_total (#5074)"
This reverts commit 74301cdbf5.

Reason for revert:

vmagent already provides better approach for detecting slow scrape targets via the following query:

    scrape_duration_seconds / scrape_timeout_seconds > 1

This query depends on automatically generated per-target metrics.
See https://docs.victoriametrics.com/vmagent.html#automatically-generated-metrics for more details.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5074
2023-10-02 21:08:13 +02:00
Aliaksandr Valialkin
5fd79f47f1
app/vmselect/promql: follow-up for 896c85a4a4
- Clarify the description of the change at docs/CHANGELOG.md
- Make sure that bitmap_*(X, NaN) returns NaN

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4996
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5021
2023-10-02 21:07:46 +02:00
Aliaksandr Valialkin
7bb5f75a2a
lib/logstorage: follow-up for 94627113db
- Move uniqueFields from rows to blockStreamMerger struct.
  This allows localizing all the references to uniqueFields inside blockStreamMerger.mustWriteBlock(),
  which should improve readability and maintainability of the code.

- Remove logging of the event when blocks cannot be merged because they contain more than maxColumnsPerBlock,
  since the provided logging didn't provide the solution for the issue with too many columns.
  I couldn't figure out the proper solution, which could be helpful for end user,
  so decided to remove the logging until we find the solution.

This commit also contains the following additional changes:

- It truncates field names longer than 128 chars during logs ingestion.
  This should prevent from ingesting bogus field names.
  This also should prevent from too big columnsHeader blocks,
  which could negatively affect search query performance,
  since columnsHeader is read on every scan of the corresponding data block.

- It limits the maximum length of const column value to 256.
  Longer values are stored in an ordinary columns.
  This helps limiting the size of columnsHeader blocks
  and improving search query performance by avoiding
  reading too long const columns on every scan of the corresponding data block.

- It deduplicates columns with identical names during data ingestion
  and background merging. Previously it was possible to pass columns with duplicate names
  to block.mustInitFromRows(), and they were stored as is in the block.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4762
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4969
2023-10-02 21:06:49 +02:00
Dmytro Kozlov
e7b35455bd
docs: add clarification of the retention filter usage (#5103)
docs: add clarification of the retention filter usage

Updated documentation regarding retention filter usage if duration is set lower than
`-retentionPeriod` flag value.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2023-10-02 21:05:54 +02:00
Roman Khavronenko
0df0b0f29e
lib/promscrape: add metric vm_promscrape_scrapes_skipped_total (#5074)
* lib/promscrape: add metric `vm_promscrape_scrapes_skipped_total`

add metric `vm_promscrape_scrapes_skipped_total`to show whether vmagent skips the scrapes.
This could happen if vmagent is overloaded or target is responding too slow for configured `scrape_interval`.

The follow-up commit should add a corresponding alerting rule and panel to vmagent dashboard.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* deployment/docker: add `TooManyScrapeSkips` alerting rule for vmagent

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* dashboards: add panels `Scrape duration 0.99 quantile` and `Skipped scrapes` to vmagent dashboard

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-02 20:38:23 +02:00
Aliaksandr Valialkin
120f3bc467
lib/logstorage: follow-up for 8a23d08c21
- Compare the actual free disk space to the value provided via -storage.minFreeDiskSpaceBytes
  directly inside the Storage.IsReadOnly(). This should work fast in most cases.
  This simplifies the logic at lib/storage.

- Do not take into account -storage.minFreeDiskSpaceBytes during background merges, since
  it results in uncontrolled growth of small parts when the free disk space approaches -storage.minFreeDiskSpaceBytes.
  The background merge logic uses another mechanism for determining whether there is enough
  disk space for the merge - it reserves the needed disk space before the merge
  and releases it after the merge. This prevents from out of disk space errors during background merge.

- Properly handle corner cases for flushing in-memory data to disk when the storage
  enters read-only mode. This is better than losing the in-memory data.

- Return back Storage.MustAddRows() instead of Storage.AddRows(),
  since the only case when AddRows() can return error is when the storage is in read-only mode.
  This case must be handled by the caller by calling Storage.IsReadOnly()
  before adding rows to the storage.
  This simplifies the code a bit, since the caller of Storage.MustAddRows() shouldn't handle
  errors returned by Storage.AddRows().

- Properly store parsed logs to Storage if parts of the request contain invalid log lines.
  Previously the parsed logs could be lost in this case.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4737
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4945
2023-10-02 20:38:00 +02:00
Aliaksandr Valialkin
cbbdf9cdf5
lib/logstorage: run up to GOMAXPROCS flushers of old in-memory parts to disk
One flusher isn't enough under high data ingestion rate.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4775
2023-10-02 20:36:53 +02:00
Github Actions
025a53dcb6
Automatic update operator docs from VictoriaMetrics/operator@44bdc27 (#5104) 2023-10-02 20:36:21 +02:00
Github Actions
835393a59c
Automatic update operator docs from VictoriaMetrics/operator@c7125bd (#5102) 2023-10-02 20:36:21 +02:00
hagen1778
25a006099d
app/vlinsert/loki: make fmt
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-02 20:35:45 +02:00
Aliaksandr Valialkin
78e9cda4b1
lib/logstorage: assist merging in-memory parts at data ingestion path if their number starts exceeding maxInmemoryPartsPerPartition
This is a follow-up for 9310e9f584 , which removed data ingestion pacing.
This can result in uncontrolled growth of in-memory parts under high data ingestion rate,
which, in turn, can result in unbounded RAM usage, OOM crashes and slow query performance.

While at it, consistently reset isInMerge field for parts passed to mergeParts() before returning from this function.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4775
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4828
2023-10-02 20:35:20 +02:00
Aliaksandr Valialkin
f55d114785
lib/{mergeset,storage}: consistently reset isInMerge field in parts passed to mergeParts() before returning from the function
While at it consistently check that the isInMerge field is set in all the parts passed to mergeParts()
2023-10-02 20:34:52 +02:00