Commit graph

6070 commits

Author SHA1 Message Date
Dmytro Kozlov
352dbd7e08
app/vmui: update cardinality page (#3986)
vmui: update cardinality page

---------

Co-authored-by: Yury Moladau <yurymolodov@gmail.com>
2023-03-24 13:34:01 -07:00
Yury Molodov
5f77efa915
vmui: display errors for each query individually (#3987) (#3994) 2023-03-24 13:26:43 -07:00
Alexander Marshalov
b5027cff9c
added configCheckInterval flag for vmauth (#3990) (#3991)
* added configCheckInterval flag for vmauth (#3990)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-03-24 13:25:07 -07:00
Nikolay
9bb83cafa4
lib/netutil: log only parsing errors for proxy-protocol (#3985)
* lib/netutil: log only parsing errors for proxy-protocol

Previosly every error was logged. With configured TCP health checks at load-balancer or kubernetes, vmauth spams a lot of false positive error message into logs

* Update docs/CHANGELOG.md

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* Update lib/netutil/tcplistener.go

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-03-21 10:23:08 -07:00
Dmytro Kozlov
4ba237ec14
app/vmctl: follow up after aed59b9029 (#3983) 2023-03-21 09:26:26 -07:00
Aliaksandr Valialkin
8ed9295109
docs/vmagent.md: mention in docs that the target relabel debug page shows target url now 2023-03-20 22:20:13 -07:00
Dmytro Kozlov
85b01c4aa7
lib/promrelabel: make target url from labels on target relabel page (#3882)
* lib/promrelabel: make target url from labels on target relabel page

* wip

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-03-20 22:08:39 -07:00
Aliaksandr Valialkin
79d8f0e7c6
app/vmselect/promql: pass workerID to the callback inside doParallel()
This opens the possibility to remove tssLock from evalRollupFuncWithSubquery()
in the follow-up commit from @zekker6 in order to speed up the code
for systems with many CPU cores.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3966
2023-03-20 20:57:34 -07:00
Aliaksandr Valialkin
e749a015a9
app/vmselect/promql: fix TestIncrementalAggr test on systems less than 3 CPU cores
This is a follow-up for 4856a4cf5a
2023-03-20 20:37:44 -07:00
Aliaksandr Valialkin
08da383eac
app/vmselect/netstorage: reduce the number of calls to runtime.Gosched() at timeseriesWorker() and unpackWorker()
Call runtime.Gosched() only when there is a work to steal from other workers.
Simplify the timeseriesWorker() and unpackWroker() code a bit by inlining stealTimeseriesWork() and stealUnpackWork().

This should reduce CPU usage when processing queries on systems with big number of CPU cores.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3966
2023-03-20 20:32:56 -07:00
Aliaksandr Valialkin
18af01c387
app/vmselect: optimize incremental aggregates a bit
Substitute sync.Map with an ordinary slice indexed by workerID.
This should reduce the overhead when updating the incremental aggregate state
2023-03-20 15:42:13 -07:00
Aliaksandr Valialkin
7a1e2f49cc
app/vmselect/vmui: make vmui-update after d4525bd2d0 2023-03-20 14:35:17 -07:00
Aliaksandr Valialkin
1580d8bda7
docs/CHANGELOG.md: cosmetic fixes 2023-03-20 14:33:46 -07:00
Aliaksandr Valialkin
531b35b6c0
docs/Troubleshooting.md: document an additional case, which could result in slow inserts
If `-cacheExpireDuration` is lower than the interval between ingested samples for the same time series,
then vm_slow_row_inserts_total` metric is increased.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3976#issuecomment-1476883183
2023-03-20 14:33:27 -07:00
Roman Khavronenko
6a7de761f4
vmalert: support logs suppressing during config reloads (#3973)
* vmalert: support logs suppressing during config reloads

The change is mostly required for ENT version of vmalert,
since it supports object-storage for config files.
Reading data from object storage could be time-consuming,
so vmalert emits logs to track the progress.

However, these logs are mostly needed on start or on
manual config reload. Printing these logs each time
`rule.configCheckInterval` is triggered would too verbose.
So the change allows to control logs emitting during
config reloads.

Now, logs are emitted during start up or when SIGHUP is receieved.
For periodicall config checks logs emitted by config pkg are suppressed.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: review fixes

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-03-20 14:25:26 -07:00
Dmytro Kozlov
aed59b9029
app/vmctl: automatically check tty (#3938)
app/vmctl: automatically detect if TTY is available
2023-03-20 14:14:43 -07:00
Yury Molodov
95b60c2777
vmui: support for drag'n'drop in the "Trace analyzer" page (#3971)
vmui: add drag-and-drop support for the trace analyzer page
2023-03-20 14:09:45 -07:00
dependabot[bot]
ba167df617
build(deps): bump actions/setup-go from 3 to 4 (#3962)
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 3 to 4.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](https://github.com/actions/setup-go/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-03-20 14:08:28 -07:00
Yury Molodov
b66953d8e1
vmui: improve usability of date/time picker (#3968)
* vmui: allow manually set input date and time
* vmui/docs: improve usability of date/time picker
2023-03-20 13:57:47 -07:00
Dmytro Kozlov
693a3de0a6
lib/storage: fix collect downsampling metrics (#489)
* lib/storage: fix downsampling

* lib/storage: update logic

* lib/storage: fix comments, removed unneeded check
2023-03-19 23:30:00 -07:00
Aliaksandr Valialkin
fc3d826d7f
all: add Windows build for VictoriaMetrics
This commit changes background merge algorithm, so it becomes compatible with Windows file semantics.

The previous algorithm for background merge:

1. Merge source parts into a destination part inside tmp directory.
2. Create a file in txn directory with instructions on how to atomically
   swap source parts with the destination part.
3. Perform instructions from the file.
4. Delete the file with instructions.

This algorithm guarantees that either source parts or destination part
is visible in the partition after unclean shutdown at any step above,
since the remaining files with instructions is replayed on the next restart,
after that the remaining contents of the tmp directory is deleted.

Unfortunately this algorithm doesn't work under Windows because
it disallows removing and moving files, which are in use.

So the new algorithm for background merge has been implemented:

1. Merge source parts into a destination part inside the partition directory itself.
   E.g. now the partition directory may contain both complete and incomplete parts.
2. Atomically update the parts.json file with the new list of parts after the merge,
   e.g. remove the source parts from the list and add the destination part to the list
   before storing it to parts.json file.
3. Remove the source parts from disk when they are no longer used.

This algorithm guarantees that either source parts or destination part
is visible in the partition after unclean shutdown at any step above,
since incomplete partitions from step 1 or old source parts from step 3 are removed
on the next startup by inspecting parts.json file.

This algorithm should work under Windows, since it doesn't remove or move files in use.
This algorithm has also the following benefits:

- It should work better for NFS.
- It fits object storage semantics.

The new algorithm changes data storage format, so it is impossible to downgrade
to the previous versions of VictoriaMetrics after upgrading to this algorithm.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3236
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3821
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70
2023-03-19 23:28:26 -07:00
Aliaksandr Valialkin
d2f85816ea
lib/{mergeset,storage}: prevent from long wait time when creating a snapshot under high data ingestion rate
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3551
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3873
2023-03-19 00:19:02 -07:00
Aliaksandr Valialkin
0627b845be
deployment/docker/Makefile: properly add amd64 suffix to windows binary names 2023-03-19 00:18:58 -07:00
Aliaksandr Valialkin
3473e83246
deployment/docker/Makefile: build CGO-enabled vmagent for GOARCH=arm64
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2271

This is a follow-up for 565497fb074321caedea38d5151044d98d92d759
2023-03-18 23:15:38 -07:00
Aliaksandr Valialkin
f54ac85133
SECURITY.md: update the list of VictoriaMetrics versions, which support security updates 2023-03-18 12:28:26 -07:00
Aliaksandr Valialkin
8aeee8bcca
lib/{fs,mergeset,storage}: substitute os.Open()+os.File.Readdir() with os.ReadDir()
This simplifies code a bit
2023-03-17 21:03:52 -07:00
Aliaksandr Valialkin
aeeab74388
app/vmctl: drop integration tests from cluster branch, since they expect single-node VictoriaMetrics
This is a follow-up for 235477628e
2023-03-17 16:43:58 -07:00
Alexander Marshalov
31005f4980
updated api doc for operator (#3972)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-03-17 16:17:17 -07:00
Roman Khavronenko
0ac57ef5b9
Vmalert tests (#3975)
* vmalert: add tests for notifier pkg

* vmalert: add tests for remotewrite pkg

* vmalert: add tests for template functions

* vmalert: add tests for web pages

* vmalert: fix int overflow in tests

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-03-17 16:16:13 -07:00
Zakhar Bessarab
c397e56769
doc/vmgateway-grafana-openid-guide: fix formatting, add reproducible example and example results (#3964) 2023-03-17 16:15:45 -07:00
Roman Khavronenko
66c5ddf2ad
alerts: add TooManyTSIDMisses alerting rule (#3959)
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3502#issuecomment-1358374954

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-03-17 16:14:50 -07:00
Aliaksandr Valialkin
c9e30dd892
all: typo fix in the same way as in e566d49e3a: 8248 -> 8428 2023-03-16 22:08:26 -07:00
oliverpool
8c708ca1e9
app/vmselect/promql: add test to ensure 8-byte alignment (#3948)
See 0af9e2b693
2023-03-16 22:07:13 -07:00
Artem Navoiev
7e6be18c3d
managed quickstart fix typo
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-03-16 22:04:03 -07:00
Artem Navoiev
208aa85362
fix anchor after chaning manager quick start
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-03-16 22:03:46 -07:00
Aliaksandr Valialkin
46521e20c8
vendor: make vendor-update 2023-03-15 13:24:22 -07:00
Nikolay
a61106d965
Vmagent kafka updates (#535)
* app/vmagent: allow vm proto for kafka consumer and producer
it should reduce network usage up to 50%.
According to benchmarks without any encoding at kafka topic, it reduces traffic up to 50%.
With enabled zstd at kafka topic, it shows no diffence in traffic. So it
doesn't make much sense to use it.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225

* mention eb61a7dd68b834b08d01727a918f207700348ada at changelog

* app/vmagent: bumps kafka lib version
it allows compiling vmagent for arm64 machines
fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2271

* mention d19b1a888248c96cfd7ccee00ba6f596d89be1d7 at change log

* app/vmagent: adds natural concurrency for kafka consumer
it should improve performance for data consumption
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1957

* mention change 0c143bb22ca2e7e0b7eec9bc84a94ee2b41626ca

* Update app/vmagent/kafka/consumer.go

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* Update app/vmagent/kafka/consumer_cgo.go

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-03-15 13:03:51 -07:00
Alexander Marshalov
3f10f19842
updated vars doc for operator (#3960)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-03-15 12:19:53 -07:00
dmitryk-dk
1ff82e23af
docs: cleanup 2023-03-15 12:19:16 -07:00
dmitryk-dk
e1c2c91bb6
app/vmctl: update managed quickstart guide 2023-03-15 12:18:48 -07:00
Aliaksandr Valialkin
5375e789fd
vendor: make vendor-update 2023-03-14 16:20:05 -07:00
Aliaksandr Valialkin
8be6c85eb3
vendor: update github.com/klauspost/compress from v1.16.0 to v1.16.3 2023-03-14 16:20:03 -07:00
Zakhar Bessarab
a3f364869f
doc: add guide for vmgateway configuration with OpenID and Grafana (#3951)
docs: add guide for vmgateway configuration with OpenID and Grafana
2023-03-14 16:11:29 -07:00
Zakhar Bessarab
d1d108fe77
lib/storage: log original labels set when label value is truncated (#3952)
lib/storage: log original labels set when label value is truncated
2023-03-14 16:11:02 -07:00
Dmytro Kozlov
235477628e
app/vmctl: integration test for native protocol (#3947)
* app/vmctl: integration test for native protocol

* app/vmctl: implemented two integration tests

* app/vmctl: cleanup

* app/vmctl: split storage init and filling data logic

* app/vmctl: cleanup

* app/vmctl: remove storage from server, used initialization process

* app/vmctl: prepare for parallel run, code cleanup

* app/vmctl: code cleanup

* app/vmctl: remove unused field
2023-03-14 16:08:40 -07:00
Aliaksandr Valialkin
9aa6b63160
docs/Single-server-VictoriaMetrics.md: clarify that the cache directory can be removed manually when VictoriaMetrics is stopped 2023-03-13 00:24:01 -07:00
Aliaksandr Valialkin
d345619424
docs/CHANGELOG.md: document v1.87.3 2023-03-13 00:21:00 -07:00
Aliaksandr Valialkin
1ae73a5fe5
docs/CHANGELOG.md: document v1.79.11 LTS release 2023-03-12 23:23:05 -07:00
Aliaksandr Valialkin
4d81f1c235
deployment: update VictoriaMetrics release from v1.88.0 to v1.89.1
See https://docs.victoriametrics.com/CHANGELOG.html#v1891
2023-03-12 20:05:32 -07:00
Aliaksandr Valialkin
47c5b77c8c
docs/CHANGELOG.md: cut v1.89.1 2023-03-12 19:14:29 -07:00