Commit graph

7642 commits

Author SHA1 Message Date
Dmytro Kozlov
38b2a5bc44
deployment/docker: add grafana datasource to the docker-compose files (#5363)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3920
https://github.com/VictoriaMetrics/grafana-datasource/issues/113
2024-01-22 15:45:31 +01:00
hagen1778
1075fcfc8c
app/vmctl/backoff: fix flaky test
The change removes artificial delay before returning error, which sometimes
caused less retry events than expected.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-22 12:21:14 +01:00
hagen1778
da556cc329
docs: fix Grafana link example for vmalert
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-22 09:35:18 +01:00
dependabot[bot]
df197723ae
build(deps): bump github/codeql-action from 2 to 3 (#5462)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2 to 3.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/v2...v3)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-01-22 01:49:17 +02:00
Aliaksandr Valialkin
d3ee3e0ef5
Revert "lib/promscrape: do not store last scrape response when stale markers … (#5577)"
This reverts commit cfec258803.

Reason for revert: the original code already doesn't store the last scrape response when stale markers are disabled.
The scrapeWork.areIdenticalSeries() function always returns true is stale markers are disabled.
This prevents from storing the last response at scrapeWork.processScrapedData().

It looks like the reverted commit could also return back the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3660

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5577
2024-01-22 00:43:48 +02:00
Aliaksandr Valialkin
9c0863babc
docs: use persistent links to Grafana dashboards
These links do not depend on the dashboard name, so they do not break after the renaming of the dashboard.

This is a follow-up for ff33e60a3d
2024-01-22 00:17:17 +02:00
Aliaksandr Valialkin
1c7f990fad
app/vmselect: handle negative time range start in a generic manner inside NewSearchQuery()
This is a follow-up for cf03e11d89

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5553
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5630
2024-01-21 23:45:31 +02:00
Artem Navoiev
3f7ed7e6b2
docs vmanomaly fix anchor
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-21 22:21:37 +01:00
Hui Wang
4e3242b02d
lib/promscrape/discovery/kubernetes: fix watcher start order for roles endpoints and endpointslice (#5557)
* lib/promscrape/discovery/kubernetes: fix watcher start order for roles endpoints and endpointslice

Previously the groupWatcher could be mistakenly stopped when requests for pod or services resources take too long.

* remove mislead comment

* docs/sd_configs.md: mention -promscrape.kubernetes.attachNodeMetadataAll flag in the description for attach_metadata section

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4640

* wip

* lib/promscrape/kubernetes: prevent from stopping groupWatcher when there are in-flight apiWatcher.mustStart() calls

groupWatcher is stopped if it has zero registered apiWatchers during 14 seconds.
But such a groupWatcher can be still in use if apiWatcher for `role: endpoints` or `role: endpointslice`
is being registered and the discovery of the associated `pod` and/or `service` objects takes longer
than 14 seconds - see the beginning of groupWatcher.startWatchersForRole() function for details.

Track the number of in-flight calls to apiWatcher.mustStart() and prevent from stopping the associated groupWatcher
if the number of in-flight calls is non-zero.

P.S. postponing the discovery of `pod` and/or `service` objects associated with `endpoints` or `endpointslice` roles
isn't the best solution, since it slows down initial discovery of `endpoints` and `endpointslice` targets.

* typo fix

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-01-21 23:13:15 +02:00
Aliaksandr Valialkin
1f105dde98
all: allow dynamically reading *AuthKey flag values from files and urls
Examples:

1) -metricsAuthKey=file:///abs/path/to/file - reads flag value from the given absolute filepath
2) -metricsAuthKey=file://./relative/path/to/file - reads flag value from the given relative filepath
3) -metricsAuthKey=http://some-host/some/path?query_arg=abc - reads flag value from the given url

The flag value is automatically updated when the file contents changes.
2024-01-21 22:03:38 +02:00
Fred Navruzov
7e68722686
- fix 404 links after renaming (#5653)
- improve wording on diagram
- swap enterprise/about chapters for page clarity
2024-01-21 21:24:29 +02:00
Fred Navruzov
0038102b98
docs: vmanomaly - add component interaction diagram (#5652)
* add interaction diagram for vmanomaly components
* small docs fixes
* resolve suggestions
2024-01-21 19:33:59 +02:00
Aliaksandr Valialkin
0b2ea1a7c7
all: call atomic.Load* in front of atomic.CompareAndSwap* at places where the atomic.CompareAndSwap* returns false most of the time
This allows avoiding slow inter-CPU synchornization induced by atomic.CompareAndSwap*
2024-01-21 14:04:54 +02:00
Aliaksandr Valialkin
3d83f3347d
docs/sd_configs.md: mention -promscrape.kubernetes.attachNodeMetadataAll flag in the description for attach_metadata section
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4640
2024-01-21 13:26:03 +02:00
Aliaksandr Valialkin
4eb9926125
lib/promscrape: code cleanup: send stale markers immediately after generating automatic metrics
This cleanup has been extracted from https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5557/files#diff-6b205cf6637d7b65a5c45d9417d08822d4efad94227268cb196f61aa2a0fc0f7
2024-01-21 05:18:22 +02:00
Aliaksandr Valialkin
12f2c5679b
all: consistently clear prompbmarshal.Label by assigning an empty struct instead of zeroing Name and Value individually 2024-01-21 05:11:05 +02:00
Aliaksandr Valialkin
90768aa418
lib/storage/partition.go: remove misleading comment, which falsely states that inmemoryParts isn't visible to search
Thanks to @satjd for raising attention to this comment at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5410
2024-01-21 04:49:35 +02:00
Nikolay
b3598ba2c1
app/vmauth: adds metric_labels and backend_errors counter (#5585)
* app/vmauth: adds metric_labels and backend_errors counter
it must improve observability for user requests with new metric - per user backend errors counter.
it's needed to calculate requests fail rate to the configured backends.
metric_labels configuration allows to perform additional aggregations on top of multiple users from configuration section.
It could be multiple clients or clients with separate read/write tokens
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5565

* wip

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-01-21 04:40:52 +02:00
Yury Molodov
3ea1294ad2
vmui: add autofocus to input for desktop version #5479 (#5592) 2024-01-21 03:24:16 +02:00
Aliaksandr Valialkin
7fba73ce11
lib/promscrape/discovery/kubernetes: add -promscrape.kubernetes.attachNodeMetadataAll command-line flag
This flag allows setting attach_metadata.node=true for all the kubernetes_sd_configs defined at -promscrape.config

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4640

Thanks to wasim-nihal for the initial implementation at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5593
2024-01-21 03:13:56 +02:00
Hui Wang
fad212c39c
app/vmselect/promql: properly handle possible negative results caused… (#5608)
* app/vmselect/promql: properly handle possible negative results caused by float operations precision error in rollup functions like rate() or increase()

* fix test
2024-01-21 02:53:29 +02:00
Nikolay
c9f39fd51f
app/vmselect/netstorage (#5649)
* app/vmselect/netstorage

correctly handle errGlobal set

* wip

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5649

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-01-21 02:47:29 +02:00
Nikolay
8ab0ce3ded
app/vmselect: abort streaming connections for vmselect (#5650)
* app/vmselect: abort streaming connections for vmselect
due to streaming nature of export APIs, curl and simmilr tools cannot
detect errors that happened after http.Header with status 200 was
written to it.

This PR tracks if body write was already started and closes connection.

It allows client to detect not expected chunk sequence and return error
to the caller.

Mostly it affects vmselect at cluster version

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5645

* wip

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5645
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5650

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-01-21 02:12:51 +02:00
Aliaksandr Valialkin
74448a7e57
lib/promscrape/discovery/hetzner: follow-up after 03a97dc678
- docs/sd_configs.md: moved hetzner_sd_configs docs to the correct place according to alphabetical order of SD names,
  document missing __meta_hetzner_role label.
- lib/promscrape/config.go: added missing MustStop() call for Hetzner SD,
  and moved the code to the correct place according to alphabetical order of SD names.
- lib/promscrape/discovery/hetzner: properly handle pagination for hloud API responses,
  populate missing __meta_hetzner_role label like Prometheus does.
- Properly populate __meta_hetzner_public_ipv6_network label like Prometheus does.
- Remove unused SDConfig.Token.
- Remove "omitempty" annotation from SDConfig.Role field, since this field is mandatory.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5550
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3154
2024-01-20 17:01:53 +02:00
Artem Navoiev
873483a782
vmanomly use proper title for overiview doc
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-19 19:59:17 +01:00
Hui Wang
cfec258803
lib/promscrape: do not store last scrape response when stale markers … (#5577)
* lib/promscrape: do not store last scrape response when stale markers are disabled

* update changelog
2024-01-20 00:53:41 +08:00
Artem Navoiev
6a2a8cd426
docs add docs titles
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-19 17:02:20 +01:00
Artem Navoiev
dfa43da1a2
vmanonamly docs fix sorting for jekill as far it doesnt support of sorting the folders
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-19 16:55:44 +01:00
Artem Navoiev
1af5faa4af
vmanomaly remove unused pages from menu
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-19 16:50:12 +01:00
Artem Navoiev
5e17636994
vmanomly specify the right menu parent for overview page
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-19 16:43:18 +01:00
Artem Navoiev
c425ec3088
docs vmanmoly fix sorting
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-19 16:15:40 +01:00
Artem Navoiev
ec85d32e21
Move vmanomaly page to its own section (#5646)
* docs: move vmanomaly overview page to its section

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

* add alias for backward compatibility

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

* fix links

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

* change title

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

---------

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-19 07:00:41 -08:00
Roman Khavronenko
7e374c227f
app/vmui: send step param for instant queries (#5639)
The change reverts https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3896
 due to reasons explained in https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3896#issuecomment-1896704401

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-19 08:48:16 +01:00
Fred Navruzov
69ae1d30bf
docs: vmanomaly slight improvements (#5637)
* - better messaging
- update links to dockerhub in guides
- added anomaly_score to FAQ
- improve model section (sort + use cases)
- slight refactor of a guide

* rename guide & change refs

* change wording in installation options

* - update remaining text reference
- add cross-link to component sections in guide

* add docs/.jekyll-metadata to .gitignore
2024-01-18 02:37:36 -08:00
hagen1778
0a5ffb3bc1
docs: remove slug from Grafana dashboard URLs
Each Grafana dashboard has unique ID which can be used to fetch the dashboard
from grafana.com: https://grafana.com/grafana/dashboards/11176
The same dashboard can be accessed via URL with slug: https://grafana.com/grafana/dashboards/11176-victoriametrics-cluster/
But using slug implies that any change to dashboard name will break the link.
So it is better to just use ID, so the dashboard URL will never break.

This is follow-up for ff33e60a3d

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-18 11:19:53 +01:00
Artem Navoiev
f89d16fc4c
docs: vmanomaly update vmanomaly + vmalert guide (#5636)
* docs: vmanomaly update vmanomaly + vmalert guide

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

* docs: vmanomaly update vmanomaly + vmalert guide. Update docker compose and monitoring section

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

* typos and fixes

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

---------

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-17 11:49:51 -08:00
Artem Navoiev
ff33e60a3d
fix link for grafana dashbaord for single node after its renaming
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-17 16:00:33 +01:00
Artem Navoiev
dab160cd74
docs: changelog fix the link to cluster
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-17 15:44:56 +01:00
Artem Navoiev
a3b3ea4d73
vmanomaly docs fix broken relative links
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-17 15:41:03 +01:00
Artem Navoiev
9a353ee695
docs/anomaly-detection/components/models.md add sort:1
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-17 15:26:38 +01:00
Artem Navoiev
0c06934a59
vmanonaly docs add .html for the section document models
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-17 15:25:30 +01:00
Artem Navoiev
5b419cfb2b
vmanomaly docs simplify the strcuture (#5634)
* vmanomaly docs simplify the strcuture

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

* fix links

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

---------

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-17 06:20:27 -08:00
hagen1778
71681fd1ca
docs/setup-size: rm tolerable churn rate %
It is likely this value was borrowed from panel `Slow inserts` panel
from Grafana dasbhoard for VM single/cluster installations. This is a mistake.
There is no such thing as "tolerable churn rate", as tolerancy depends on the amount
of allocated resources.

Although, it is unclear what is meant by 5%. If it refers to 5% of new time series per second,
then such churn rate is extremely high. It would mean that the avg life of a time series is 20s.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-17 14:44:28 +01:00
Roman Khavronenko
cf03e11d89
app/vmselect: properly calculate start param for queries with too big look-behind window (#5630)
Properly determine time range search for instant queries with too big look-behind window like `foo[100y]`.
 Previously, such queries could return empty responses even if `foo` is present in database.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5553

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-17 13:48:06 +01:00
Aliaksandr Valialkin
5bdf62de5b
lib/storage: do not prefetch metric names for small number of metricIDs
This eliminates prefetchedMetricIDsLock lock contention for queries, which return less than 500 time series.

This is a follow-up for 9d886a2eb0
2024-01-17 13:48:15 +02:00
Aliaksandr Valialkin
3c3450fc53
docs/keyConcepts.md: typo fixes after b7ffee2644
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5555
2024-01-17 13:27:47 +02:00
Aliaksandr Valialkin
befcd93305
docs/keyConcepts.md: document /internal/force_flush handler
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5555
2024-01-17 13:23:32 +02:00
Aliaksandr Valialkin
cc6819869a
docs/CHANGELOG*: move changes for 2023 year to docs/CHANGELOG_2023.md 2024-01-17 13:10:32 +02:00
hagen1778
8040bdc1d6
docs/troubleshooting: mention query latency in unexpected query results
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5555

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-17 11:59:47 +01:00
Roman Khavronenko
3a1ef3184d
app/victoriametrics: update the test suite (#5627)
app/victoriametrics: update the test suite

* simplify /query and /query_range test cases configuration and tests
* support instant queries with lookbehind window like `query=foo[5m]`
* support instant queries selecting scalar value like `query=42`
* add query_range test for prometheus

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-17 09:51:10 +01:00