Commit graph

468 commits

Author SHA1 Message Date
hagen1778
39e9a0848c
deployment/docker: bump grafana version to grafana/grafana:10.3.1
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-02-05 10:49:52 +02:00
Aliaksandr Valialkin
24437b6d70
all: update VictoriaMetrics Docker image from v1.97.0 to v1.97.1
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.1
2024-02-01 16:15:14 +02:00
Aliaksandr Valialkin
f1effc1133
all: update VictoriaMetrics Docker image from v1.96.0 to v1.97.0
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.97.0
2024-01-30 23:37:42 +02:00
Aliaksandr Valialkin
6d766abde7
deployment/docker: upgrade base Docker image from Alpine 3.19.0 to 3.19.1
See https://www.alpinelinux.org/posts/Alpine-3.19.1-released.html
2024-01-30 22:47:29 +02:00
Fred Navruzov
8043cf35d8
- update versions to 1.9.2 (#5714)
- update guide asset urls to flat
2024-01-29 18:20:38 +01:00
Fred Navruzov
790ee9af6d
update guide and changelog to 1.9.1 (#5706) 2024-01-29 17:23:06 +01:00
Daria Karavaieva
469bbbb2fa
version 1.9.1 update, dashboard viz flag (#5704) 2024-01-29 16:48:55 +01:00
Daria Karavaieva
b042982339
Vmanomaly Guide dashboard provisioning (#5679)
* dashboard provisioning

* delete dashboard filter, new query

* dashboard screens, guide fixes
2024-01-26 21:07:44 +01:00
hagen1778
d76e338505
docs: simplify instructions for spinning up docker env
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-26 20:50:31 +01:00
hagen1778
22a8664f51
deployment/docker: fix typo in commands example
Follow up after 38b2a5bc44

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-22 18:44:52 +02:00
hagen1778
644d43620f
deployment/docker: follow-up 38b2a5bc44
* Simplify folder structure
* mention datasource in README

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-22 18:43:54 +02:00
Dmytro Kozlov
69e59ac9b7
deployment/docker: add grafana datasource to the docker-compose files (#5363)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3920
https://github.com/VictoriaMetrics/grafana-datasource/issues/113
2024-01-22 18:43:16 +02:00
Fred Navruzov
3b8f4714c3
docs: vmanomaly slight improvements (#5637)
* - better messaging
- update links to dockerhub in guides
- added anomaly_score to FAQ
- improve model section (sort + use cases)
- slight refactor of a guide

* rename guide & change refs

* change wording in installation options

* - update remaining text reference
- add cross-link to component sections in guide

* add docs/.jekyll-metadata to .gitignore
2024-01-22 00:24:39 +02:00
hagen1778
a99d26633b
docs: remove slug from Grafana dashboard URLs
Each Grafana dashboard has unique ID which can be used to fetch the dashboard
from grafana.com: https://grafana.com/grafana/dashboards/11176
The same dashboard can be accessed via URL with slug: https://grafana.com/grafana/dashboards/11176-victoriametrics-cluster/
But using slug implies that any change to dashboard name will break the link.
So it is better to just use ID, so the dashboard URL will never break.

This is follow-up for ff33e60a3d

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-22 00:23:30 +02:00
Artem Navoiev
79b308fa54
docs: vmanomaly update vmanomaly + vmalert guide (#5636)
* docs: vmanomaly update vmanomaly + vmalert guide

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

* docs: vmanomaly update vmanomaly + vmalert guide. Update docker compose and monitoring section

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

* typos and fixes

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

---------

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-22 00:19:15 +02:00
Aliaksandr Valialkin
b7fcdb1985
deployment/docker: update Go builder from Go1.21.5 to Go1.21.6 2024-01-17 00:05:24 +02:00
hagen1778
343176438e
deployment/alerts: add job label to DiskRunsOutOfSpace alerting rule
So it is easier to understand to which installation the triggered instance belongs.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-16 22:21:34 +02:00
Aliaksandr Valialkin
b2dcb94322
deployment: update VictoriaMetrics images from v1.95.1 to v1.96.0
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.96.0
2023-12-13 01:21:12 +02:00
hagen1778
383dce7201
alerts: simplify aggregation of alerting rules
This is follow-up after
75196d7234

It updates some of the alerting rules to remove unnecessary aggregations.
It keeps aggregations for expressions which are using multiple time series
filters to make sure their label will match.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 8fb68152e6)
2023-12-11 15:38:16 +01:00
7840vz
231f6c7fb2
alerts: inverse grouping in vmagent alerts (#5429)
Aggregations with by() have one sideeffect, that any custom labels you add to hosts are dropped too which can be used for alerts routing.

Therefore, some good practice could be to use without() instead, with labels, like without(path) , or without(url) to get same aggregations but with any external labels left intact.

(cherry picked from commit 75196d7234)
2023-12-11 15:38:14 +01:00
Aliaksandr Valialkin
842aba3f46
deployment/docker: update base Docker image from alpine:3.18.5 to alpine:3.19.0
See https://www.alpinelinux.org/posts/Alpine-3.19.0-released.html
2023-12-10 02:28:31 +02:00
Alexander Marshalov
e9cf39f519
added field version to the response for /api/v1/status/buildinfo API for using more efficient API in Grafana for receiving label values, added additional info about setup Grafana datasource (#5370) (#5437) 2023-12-07 16:41:56 +02:00
Aliaksandr Valialkin
12e94f10cc
deployment/docker: update Go builder from Go1.21.4 to Go1.21.5
See https://github.com/golang/go/issues?q=milestone%3AGo1.21.5+label%3ACherryPickApproved
2023-12-06 22:33:27 +02:00
Dmytro Kozlov
6a41e1ec0c
app/vmalert: replace error metrics for gauges with counter metrics (#5217)
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5160

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 935bec447b)
2023-12-06 19:41:34 +01:00
Aliaksandr Valialkin
a3d0bbfcda
deployment/docker: update backe Docker image from alpine 3.18.4 to 3.18.5
See https://www.alpinelinux.org/posts/Alpine-3.15.11-3.16.8-3.17.6-3.18.5-released.html
2023-12-04 18:17:07 +02:00
Aliaksandr Valialkin
0ccc1aca0a
deployment/docker: remove built binaries at bin folder after creating docker image from them at make publish-via-docker 2023-11-21 14:33:50 +02:00
Aliaksandr Valialkin
6b75523468
Makefile: speedup release, publish and crossbuild rules by using parallel make 2023-11-20 23:07:11 +02:00
Aliaksandr Valialkin
1149a98873
deployment: update VictoriaMetrics docker image tag from v1.95.0 to v1.95.1
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.95.1
2023-11-17 15:43:43 +01:00
Aliaksandr Valialkin
656a6862d7
deployment: update reference to VictoriaLogs from v0.4.1-victorialogs to v0.4.2-victorialogs 2023-11-15 20:47:14 +01:00
Aliaksandr Valialkin
bafe62b487
deployment: update references to VictoriaMetrics components from v1.94.0 to v1.95.0 2023-11-15 20:38:51 +01:00
Aliaksandr Valialkin
7bbdecb79a
deployment: update Go builder from Go1.21.3 to Go1.21.4
See https://github.com/golang/go/issues?q=milestone%3AGo1.21.4+label%3ACherryPickApproved
2023-11-13 09:40:08 +01:00
PhracturedBlue
6675d08ebc
Support building images via podman (#4978) 2023-11-13 09:35:02 +01:00
Roman Khavronenko
4e8c762fd9
app/vmalert: add label file pointing to the group's filename to metrics (#5281)
The filename should help identifying alerting rules belonging to specific groups
with identical names but different filenames.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5267

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit b5254199c6)
2023-11-02 16:02:29 +01:00
hagen1778
659171686c
deployment/alerts: add TooManyMissedIterations alerting rule
The new rule for vmalert supposed to detect groups that miss their
evaulations due to slow queries.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 9866974a53)
2023-10-31 10:35:57 +01:00
hagen1778
d9118cdaab
deployment/alerts: update TooHighMemoryUsage annotation
The memory usage isn't measured on 5m interval anymore.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 4e0a779efe)
2023-10-25 14:39:48 +02:00
hagen1778
d349d6a9ce
deployment/alerts: make TooHighMemoryUsage more tolerable to spikes
Using `min_over_time` should reduce the amount of false positives when
component is running in near-the-threshold state. Now it should trigger
only if all collected samples were above the threshold on 10m interval.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 003ef3a518)
2023-10-25 14:39:48 +02:00
hagen1778
361a3dcc0c
deployment/alerts: make RemoteWriteConnectionIsSaturated expr readable
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 685f9c3c98)
2023-10-25 14:39:47 +02:00
Aliaksandr Valialkin
ca35e0d66a
deployment/docker: update Go builder from Go1.21.1 to Go1.21.3
See https://github.com/golang/go/issues?q=milestone%3AGo1.21.2+label%3ACherryPickApproved
and https://github.com/golang/go/issues?q=milestone%3AGo1.21.3+label%3ACherryPickApproved
2023-10-16 12:38:14 +02:00
Aliaksandr Valialkin
65e7a7cabb
deployment/docker: update VictoriaLogs from v0.4.0-victorialogs to v0.4.1-victorialogs
See https://docs.victoriametrics.com/VictoriaLogs/CHANGELOG.html#v041
2023-10-04 16:27:39 +02:00
hagen1778
297f63a01e
alerting: account for vmauth component for alerts ServiceDown and TooManyRestarts
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-03 17:52:43 +02:00
Aliaksandr Valialkin
dc71db9470
deployment: update VictoriaLogs from v0.3.0-victorialogs to v0.4.0-victorialogs
See https://docs.victoriametrics.com/VictoriaLogs/CHANGELOG.html#v040
2023-10-03 02:34:28 +02:00
Aliaksandr Valialkin
fcd43f11f5
deployment: update VictoriaMetrics from v1.93.5 to v1.94.0
See https://docs.victoriametrics.com/CHANGELOG.html#v1940
2023-10-03 01:40:21 +02:00
Aliaksandr Valialkin
47741cd935
deployment/docker: update Alpine from 3.18.3 to 3.18.4
See https://alpinelinux.org/posts/Alpine-3.18.4-released.html
2023-10-02 21:54:57 +02:00
Aliaksandr Valialkin
d80ccf52a0
Revert "lib/promscrape: add metric vm_promscrape_scrapes_skipped_total (#5074)"
This reverts commit 74301cdbf5.

Reason for revert:

vmagent already provides better approach for detecting slow scrape targets via the following query:

    scrape_duration_seconds / scrape_timeout_seconds > 1

This query depends on automatically generated per-target metrics.
See https://docs.victoriametrics.com/vmagent.html#automatically-generated-metrics for more details.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5074
2023-10-02 21:08:13 +02:00
Roman Khavronenko
0df0b0f29e
lib/promscrape: add metric vm_promscrape_scrapes_skipped_total (#5074)
* lib/promscrape: add metric `vm_promscrape_scrapes_skipped_total`

add metric `vm_promscrape_scrapes_skipped_total`to show whether vmagent skips the scrapes.
This could happen if vmagent is overloaded or target is responding too slow for configured `scrape_interval`.

The follow-up commit should add a corresponding alerting rule and panel to vmagent dashboard.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* deployment/docker: add `TooManyScrapeSkips` alerting rule for vmagent

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* dashboards: add panels `Scrape duration 0.99 quantile` and `Skipped scrapes` to vmagent dashboard

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-02 20:38:23 +02:00
Aliaksandr Valialkin
3db9db356d
deployment/docker/docker-compose-cluster.yml: follow-up for 4d1b572f46
Grafana and vmalert now depend on vmauth instead of individual vmselect nodes

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5057
2023-10-02 20:28:06 +02:00
Aliaksandr Valialkin
b366a22018
deployment: update VictoriaMetrics version from v1.93.4 to v1.93.5
See https://docs.victoriametrics.com/CHANGELOG.html#v1935
2023-10-02 20:16:54 +02:00
Roman Khavronenko
d39c8525e2
Docker add vmauth (#5057)
* docker-compose: add vmauth to cluster env

vmauth acts as a balancer and used as an example of how to interconnect
VM components via vmauth.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* docker-compose: add vmauth to cluster env

vmauth acts as a balancer and used as an example of how to interconnect
VM components via vmauth.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2023-10-01 21:24:01 +02:00
Aliaksandr Valialkin
1d9adc022d
deployment/docker: update VictoriaMetrics from v1.93.3 to v1.93.4
See https://docs.victoriametrics.com/CHANGELOG.html#v1934
2023-09-10 19:50:14 +02:00
Artem Navoiev
973c74df55
Update VL daashboard. Add Resource Section, add ds and job filters, a… (#4981)
* Update VL daashboard. Add Resource Section, add ds and job filters, add metric collection in docker compose from victorialogs, fix networkigs usage in docker compose

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

* add vl dashboard to docker compose

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

* add vl dashboard to docker compose

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

---------

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-09-10 15:05:19 +02:00
Aliaksandr Valialkin
677612c6cb
all: update Go builder from Go1.21.0 to Go1.21.1
See https://github.com/golang/go/issues?q=milestone%3AGo1.21.1+label%3ACherryPickApproved
2023-09-07 11:36:49 +02:00
Aliaksandr Valialkin
fe739e771b
deployment/docker: properly build armv5 production builds for GOARCH=arm
Pass GOARM=5 when building GOARCH=arm production builds, since the default value for this env var
has been changed to GOARM=6 since Go1.21.0.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4965
and https://github.com/golang/go/issues/62475
2023-09-07 11:19:40 +02:00
Aliaksandr Valialkin
02f25ecf75
deployment: update VictoriaMetrics tag from v1.93.2 to v1.93.3
See https://docs.victoriametrics.com/CHANGELOG.html#v1933
2023-09-02 11:16:36 +02:00
Aliaksandr Valialkin
ea254edb44
deployment: update VictoriaMetrics tag from v1.93.1 to v1.93.2
See https://docs.victoriametrics.com/CHANGELOG.html#v1932
2023-09-01 17:36:01 +02:00
Zakhar Bessarab
d8258be292
deployment/docker: add VictoriaLogs (#4929)
* deployment/docker: add VictoriaLogs configuration

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* deployment/docker/victorialogs: remove outdated comment

It was added in order to indicate that it is required to build VictoriaLogs manually before starting it at the time there was no public release available.
Currently, there is a public tag and it is not required to build it from sources.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* deployment/docker/victorialogs/fluentbit: include log path in stream configuration

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* deployment/docker: add reference to monitoring setup for VictoriaLogs

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-09-01 10:46:32 +02:00
Nikolay
e711c8877e
deployment/docker: disable provenance in buildx (#4911)
* deployment/docker: disable provenance in buildx
it must fix an issue with multi-platform manifest generation
at buildx >= 0.10 backward compatibility was broken and generated image cannot be used with docker systems that doesn't support oci.
disabling attestat temporary fixes it.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4907
https://docs.docker.com/build/attestations/slsa-provenance/

* Update docs/CHANGELOG.md

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-08-29 16:32:13 +02:00
Nikolay
50584cfeff
docs: release docs 1.93.1 (#4875)
* docs: mention v1.93.1 release

* deployment/docker: bumps image for v1.93.1 release

(cherry picked from commit 6abd575cbe)
2023-08-23 16:44:23 +02:00
Aliaksandr Valialkin
8fccdcfed5
deployment: update Docker image tags for VictoriaMetrics components from v1.92.1 to v1.93.0
See https://docs.victoriametrics.com/CHANGELOG.html#v1930
2023-08-12 12:26:12 -07:00
Aliaksandr Valialkin
181a6152ae
deployment/docker/Makefile: upgrade base Docker image from alpine:3.18.2 to alpine:3.18.3
See https://alpinelinux.org/posts/Alpine-3.15.10-3.16.7-3.17.5-3.18.3-released.html
2023-08-12 06:00:07 -07:00
Aliaksandr Valialkin
4470856cc4
all: update Go builder from Go1.20.7 to Go1.21.0
See https://tip.golang.org/doc/go1.21
and https://go.dev/blog/go1.21
2023-08-11 06:26:27 -07:00
Karan Sharma
e40231daad
fix: remove healthcheck in vector docker-compose (#4772) 2023-08-11 04:45:59 -07:00
Roman Khavronenko
7220f70506
security: bump go version from 1.20.6 to 1.20.7 (#4773)
The update includes a security fix to the crypto/tls package,
as well as bug fixes to the assembler and the compiler.

See the list of issues addressed in Go1.20.7 here:
https://github.com/golang/go/issues?q=milestone%3AGo1.20.7+label%3ACherryPickApproved

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-08-11 04:44:01 -07:00
hagen1778
15727b3d03
deployment: bump components version to 1.92.1
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-08-11 04:41:44 -07:00
hagen1778
f2b06484f2
alerts: move ConcurrentFlushesHitTheLimit alert to health alerts
The `ConcurrentFlushesHitTheLimit` could be related to components like
vminsert, vmstorage, vm-single-node and vmagent. Moving this alert
to the `health` section of alerts will be benefitial for all components
and will remove the duplicates from single/cluster alerts.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-08-11 04:39:28 -07:00
hagen1778
be7001351c
alerts: add docs section for the full list of alerting rules
The change also includes update of all references in other docs
to the alerting rules.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-08-11 04:38:30 -07:00
Anton Tykhyy
2107b3133e
Remove some repetitions in docker Makefile (#4764) 2023-08-11 04:00:26 -07:00
Aliaksandr Valialkin
feb879eb99
docs/VictoriaLogs: add missing information about Promtail support for data ingestion
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4728
2023-07-28 11:17:14 -07:00
Aliaksandr Valialkin
f850421103
deployment: update VictoriaMetrics release tag from v1.91.3 to v1.92.0 2023-07-27 14:58:57 -07:00
Aliaksandr Valialkin
9e6cc3c495
deployment: update VictoriaLogs Docker image tag from from v0.2.0-victorialogs to v0.3.0-victorialogs 2023-07-20 21:55:09 -07:00
Aliaksandr Valialkin
30098ac8bd
app/vlinsert/loki: follow-up after 09df5b66fd
- Parse protobuf if Content-Type isn't set to `application/json` - this behavior is documented at https://grafana.com/docs/loki/latest/api/#push-log-entries-to-loki

- Properly handle gzip'ped JSON requests. The `gzip` header must be read from `Content-Encoding` instead of `Content-Type` header

- Properly flush all the parsed logs with the explicit call to vlstorage.MustAddRows() at the end of query handler

- Check JSON field types more strictly.

- Allow parsing Loki timestamp as floating-point number. Such a timestamp can be generated by some clients,
  which store timestamps in float64 instead of int64.

- Optimize parsing of Loki labels in Prometheus text exposition format.

- Simplify tests.

- Remove lib/slicesutil, since there are no more users for it.

- Update docs with missing info and fix various typos. For example, it should be enough to have `instance` and `job` labels
  as stream fields in most Loki setups.

- Allow empty of missing timestamps in the ingested logs.
  The current timestamp at VictoriaLogs side is then used for the ingested logs.
  This simplifies debugging and testing of the provided HTTP-based data ingestion APIs.

The remaining MAJOR issue, which needs to be addressed: victoria-logs binary size increased from 13MB to 22MB
after adding support for Loki data ingestion protocol at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4482 .
This is because of shitty protobuf dependencies. They must be replaced with another protobuf implementation
similar to the one used at lib/prompb or lib/prompbmarshal .
2023-07-20 21:52:11 -07:00
Zakhar Bessarab
5b3cbd4db1
app/vlinsert: add support of loki push protocol (#4482)
* app/vlinsert: add support of loki push protocol

- implemented loki push protocol for both Protobuf and JSON formats
- added examples in documentation
- added example docker-compose

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vlinsert: move protobuf metric into its own file

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* deployment/docker/victorialogs/promtail: update reference to docker image

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* deployment/docker/victorialogs/promtail: make volume name unique

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vlinsert/loki: add license reference

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* deployment/docker/victorialogs/promtail: fix volume name

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs/VictoriaLogs/data-ingestion: add stream fields for loki JSON ingestion example

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vlinsert/loki: move entities to places where those are used

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vlinsert/loki: refactor to use common components

- use CommonParameters from insertutils
- stop ingestion after first error similar to elasticsearch and jsonline

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vlinsert/loki: address review feedback

- add missing logstorage.PutLogRows calls
- refactor tenant ID parsing to use common function
- reduce number of allocations for parsing by reusing  logfields slices
- add tests and benchmarks for requests processing funcs

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-07-20 16:49:43 -07:00
Haleygo
1794a97ebe
add vmalertmanager filter for health alerts (#4665) 2023-07-19 14:50:06 -07:00
Aliaksandr Valialkin
15ccde8dd3
all: use the new release for VictoriaLogs - v0.2.0-victorialogs 2023-07-17 23:20:07 -07:00
Aliaksandr Valialkin
b07a1c85b9
all: update Go builder from 1.20.5 to 1.20.6
See https://github.com/golang/go/issues?q=milestone%3AGo1.20.6+label%3ACherryPickApproved
2023-07-12 01:00:24 -07:00
Sergey
747c39d714
security: update base Alpine image to 3.18.2 to avoid security risks (#4571)
libcrypto3 and libssl3 in Alpine 3.18.0 have versions `3.1.0-r4`
which contains CVE-2023-2650:
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-2650
Use ALpine image 3.18.2 which contains fixed versions of libssl3
and libcrypto3: 3.1.1-r0
NB: In Openshift these containers are marked as vulnerabilities
because of these CVEs.
2023-07-06 22:12:20 -07:00
Roman Khavronenko
7d1e80e6e2
bump to 1.91.3 (#4565)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-06 22:10:54 -07:00
Max Golionko
75d9914916
CI: disable docker scan, enable auto release to sandbox (#4476)
* disable docker scan

* disable nightly, enable auto release to sandbox

* remove whitespace
2023-07-06 22:02:48 -07:00
Artem Navoiev
d48e697bbd
fix jsonline endpoint in docs
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-06 21:40:59 -07:00
Artem Navoiev
9e9506af31
fix link vmui links in docker examples after the path chnage for vmui
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-06 21:40:16 -07:00
Artem Navoiev
eaa74595dc
add information about how to access web UI
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-06 21:34:16 -07:00
Artem Navoiev
b16a5ee705
use latest image in example for VictoriaLogs and VictoriaMetrics to simplify the flow
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-06 21:30:15 -07:00
Alexander Marshalov
05ac9618c0
improved vector example for victoria logs
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-07-06 21:27:23 -07:00
Artem Navoiev
265dd66191
remove grafana notion from vector readme
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-06 21:26:55 -07:00
Artem Navoiev
d058f24650
examples add README
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-06 21:26:33 -07:00
Artem Navoiev
7c588af66e
remove docker from filebeat syslog
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-06 21:26:00 -07:00
Artem Navoiev
82b49d7194
Add docker compose examples: filebeat(docker, syslog), fluentbit(docker), logstash, vector(docker)
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-07-06 21:25:31 -07:00
Roman Khavronenko
4624fda00d
all: update Go builder from Go1.20.4 to Go1.20.5 (#4427)
See https://github.com/golang/go/issues?q=milestone%3AGo1.20.5+label%3ACherryPickApproved

Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 476c7bdd6f)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-06-09 10:42:15 +02:00
Roman Khavronenko
c9f6b34a82
Bump 1.91.2 (#4394)
* deployment/docker: update VictoriaMetrics version from v1.91.1 to v1.91.2 in docker compose files

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* deployment/marketplace: update VictoriaMetrics version from v1.91.1 to v1.91.2 in marketplace files

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 2d6a3bc61f)
2023-06-09 10:40:10 +02:00
Roman Khavronenko
b00509704f
deployment/docker: update VictoriaMetrics version from v1.91.0 to v1.91.1 in docker compose files (#4387)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 2c664a6d12)
2023-06-02 13:29:50 +02:00
Aliaksandr Valialkin
b73a2336f0
deployment/docker: update VictoriaMetrics version from v1.90.0 to v1.91.0 in docker compose files 2023-05-18 15:34:54 -07:00
Aliaksandr Valialkin
93f934eae3
deployment/docker/Makefile: use alpine 3.17.3 instead of alpine 3.18.0 for certs image, since alpine 3.18.0 doesnt work for cross-platform builds 2023-05-18 14:11:02 -07:00
Denys Holius
211f48fed6
deployment/docker/Makefile: updated docker compose commands regarding migration from V1 to V2 (#4314)
deployment/docker/Makefile: updated docker compose commands regarding migration from V1 to V2
2023-05-18 10:48:35 -07:00
Aliaksandr Valialkin
e6648d14b2
deployment/docker: update base docker image from 3.17.3 to 3.18.0
See https://www.alpinelinux.org/posts/Alpine-3.18.0-released.html
2023-05-12 17:31:35 -07:00
Aliaksandr Valialkin
8cca9ed0d9
all: update Go builder from Go1.20.3 to Go1.20.4
See https://github.com/golang/go/issues?q=milestone%3AGo1.20.4+label%3ACherryPickApproved
2023-05-09 22:31:08 -07:00
Zakhar Bessarab
2856e15a6e
deployment/docker: allow overriding docker namespace (#4265)
It makes it easier for users who build and self-host images to publish their images without changing tags manually.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-05-09 21:44:22 -07:00
Roman Khavronenko
3049754575
alerts: update TooHighMemoryUsage threshold (#4256)
It appears that 90% usage for anonymous mem usage
is already concerning. So we lowering the threshold to 80%.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-05-09 21:41:40 -07:00
Max Golionko
ef11e49d7b
add vmsingle filter for health alerts (#4238) 2023-05-08 22:00:43 -07:00
Max Golionko
0f14beff58
alerts: relax job filter to support job names created by VMOperator (#4203) 2023-05-08 15:52:31 -07:00
7840vz
d7157e582c
alerts: decrease severity to info for RecordingRulesNoData (#4089) 2023-05-08 15:46:15 -07:00
Aliaksandr Valialkin
7f55afd699
deployment: update VictoriaMetrics from v1.89.1 to v1.90.0
See https://docs.victoriametrics.com/CHANGELOG.html#v1900
2023-04-06 19:18:14 -07:00
Aliaksandr Valialkin
52c46f49e1
all: update Go builder from Go1.20.2 to Go1.20.3
See https://github.com/golang/go/issues?q=milestone%3AGo1.20.3+label%3ACherryPickApproved
2023-04-05 13:38:44 -07:00
Zakhar Bessarab
303d49a4c2
deployment/docker: update Grafana URLs to match latest format (#4060)
See: #4019

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-04-05 13:31:29 -07:00
Aliaksandr Valialkin
43b431d322
deployment/docker: update base Docker image from Alpine 3.17.2 to Alpine 3.17.3
This fixes security issues from https://alpinelinux.org/posts/Alpine-3.17.3-released.html

This is a follow-up for 59c350d0d2
2023-03-31 22:54:48 -07:00
Aliaksandr Valialkin
531b35b6c0
docs/Troubleshooting.md: document an additional case, which could result in slow inserts
If `-cacheExpireDuration` is lower than the interval between ingested samples for the same time series,
then vm_slow_row_inserts_total` metric is increased.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3976#issuecomment-1476883183
2023-03-20 14:33:27 -07:00
Aliaksandr Valialkin
0627b845be
deployment/docker/Makefile: properly add amd64 suffix to windows binary names 2023-03-19 00:18:58 -07:00
Aliaksandr Valialkin
3473e83246
deployment/docker/Makefile: build CGO-enabled vmagent for GOARCH=arm64
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2271

This is a follow-up for 565497fb074321caedea38d5151044d98d92d759
2023-03-18 23:15:38 -07:00
Roman Khavronenko
66c5ddf2ad
alerts: add TooManyTSIDMisses alerting rule (#3959)
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3502#issuecomment-1358374954

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-03-17 16:14:50 -07:00
Nikolay
a61106d965
Vmagent kafka updates (#535)
* app/vmagent: allow vm proto for kafka consumer and producer
it should reduce network usage up to 50%.
According to benchmarks without any encoding at kafka topic, it reduces traffic up to 50%.
With enabled zstd at kafka topic, it shows no diffence in traffic. So it
doesn't make much sense to use it.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225

* mention eb61a7dd68b834b08d01727a918f207700348ada at changelog

* app/vmagent: bumps kafka lib version
it allows compiling vmagent for arm64 machines
fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2271

* mention d19b1a888248c96cfd7ccee00ba6f596d89be1d7 at change log

* app/vmagent: adds natural concurrency for kafka consumer
it should improve performance for data consumption
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1957

* mention change 0c143bb22ca2e7e0b7eec9bc84a94ee2b41626ca

* Update app/vmagent/kafka/consumer.go

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* Update app/vmagent/kafka/consumer_cgo.go

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-03-15 13:03:51 -07:00
Aliaksandr Valialkin
4d81f1c235
deployment: update VictoriaMetrics release from v1.88.0 to v1.89.1
See https://docs.victoriametrics.com/CHANGELOG.html#v1891
2023-03-12 20:05:32 -07:00
Roman Khavronenko
b6ed977fa4
security: bump go version to 1.20.2 (#3935)
upgrade Go builder from Go1.20.1 to Go1.20.2
See the list of issues addressed in Go1.20.2 here (https://github.com/golang/go/issues?q=milestone%3AGo1.20.2+label%3ACherryPickApproved).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-03-11 23:38:14 -08:00
Roman Khavronenko
843c8cae44
deployment/docker: fix typo (#3883)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-03-07 23:35:58 -08:00
Aliaksandr Valialkin
1f24771779
deployment/docker: update VictoriaMetrics docker tag from v1.87.1 to v1.88.0 2023-02-24 18:59:01 -08:00
Duc Tran
7fe3cd3110
docs: fix links to alerts and alertmanager (#3829) 2023-02-18 15:19:48 -08:00
Aliaksandr Valialkin
ecc84f7923
all: update Go builder from Go1.20.0 to Go1.20.1
See https://github.com/golang/go/issues?q=milestone%3AGo1.20.1+label%3ACherryPickApproved
2023-02-14 23:05:54 -08:00
Aliaksandr Valialkin
4d9aaff3e1
all: update alpine base docker image from 1.17.1 to 1.17.2
See https://alpinelinux.org/posts/Alpine-3.17.2-released.html
2023-02-11 00:37:28 -08:00
Aliaksandr Valialkin
d63a244895
deployment/docker: update VictoriaMetrics Docker images from v1.87.0 to v1.87.1 2023-02-09 15:53:36 -08:00
Aliaksandr Valialkin
d02146663b
all: skip issues with low severity at docker scan 2023-02-09 14:25:33 -08:00
Aliaksandr Valialkin
34379d4cf1
all: run apk update && apk upgrade in base Alpine Docker image in order to get all the recent security fixes 2023-02-09 14:03:02 -08:00
Max Golionko
41bf9a481a
CI: speedup build by 2.4x. restore nightly build (#3772)
* setup docker buildx
* add snyk integration
* add go cache for docker build
* cancel redundant job if there is new commit into same PR or branch
2023-02-07 09:31:00 -08:00
Denys Holius
aa8411501c
fixed wrong vmstorage port number (#3769) 2023-02-07 09:27:28 -08:00
Aliaksandr Valialkin
1985b73cd0
deployment/docker: update Go builder from Go1.19.5 to Go1.20.0
See https://go.dev/blog/go1.20
2023-02-03 15:05:48 -08:00
Aliaksandr Valialkin
2cdc3fd500
deployment/docker: update VictoriaMetrics docker image tag from v1.86.2 to v1.87.0 2023-02-01 20:03:22 -08:00
Aliaksandr Valialkin
2ce69468cd
deployment/docker: sync with master branch 2023-01-23 22:59:42 -08:00
Aliaksandr Valialkin
0d91e0f035
deployment/docker: update VictoriaMetrics components in docker-compose from v1.86.0 to v1.86.2 2023-01-18 12:57:52 -08:00
Aliaksandr Valialkin
6d1bd761f8
deployment/docker: update Go builder from v1.19.4 to v1.19.5
See https://github.com/golang/go/issues?q=milestone%3AGo1.19.5+label%3ACherryPickApproved
2023-01-10 18:42:34 -08:00
Aliaksandr Valialkin
51b4f3c49f
deployment/docker: update Docker tag for VictoriaMetrics components from v1.85.3 to v1.86.0 2023-01-10 17:46:12 -08:00
Aliaksandr Valialkin
ee73c9a5e6
deployment/docker: update Alpine base image from v3.17.0 to v3.17.1
See https://alpinelinux.org/posts/Alpine-3.17.1-released.html
2023-01-10 16:15:06 -08:00
Denys Holius
e58fecb019
deployment/docker: update Alertmanager tag from v0.24.0 to v0.25.0 in docker-compose files (#3619)
deployment/docker: bump alertmanager to latest v0.25.0
2023-01-09 21:25:10 -08:00
Aliaksandr Valialkin
b275983403
lib/writeconcurrencylimiter: improve the logic behind -maxConcurrentInserts limit
Previously the -maxConcurrentInserts was limiting the number of established client connections,
which write data to VictoriaMetrics. Some of these connections could be idle.
Such connections do not consume big amounts of CPU and RAM, so there is a little sense in limiting
the number of such connections. So now the -maxConcurrentInserts command-line option
limits the number of concurrently executed insert requests, not including idle connections.

It is recommended removing -maxConcurrentInserts command-line option, since the default value
for this option should work good for most cases.
2023-01-06 22:07:16 -08:00
Artem Navoiev
393f4ab86f
update links to grafana dashboards (#3534)
docs: update links to grafana dashboards

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2022-12-28 11:22:02 -08:00
Aliaksandr Valialkin
ad010726fb
deployment/docker: update VictoriaMetrics tag from v1.85.2 to v1.85.3 in docker-compose files 2022-12-20 15:50:04 -08:00
Aliaksandr Valialkin
9d3c121feb
deployment/docker: use cluster-latest tag for docker images from cluster components
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3505
2022-12-20 10:38:51 -08:00
Aliaksandr Valialkin
371f9085c9
docs/CHANGELOG.md: cut v1.85.2 2022-12-19 13:09:48 -08:00
Aliaksandr Valialkin
676de127aa
all: update Go builder from v1.19.3 to v1.19.4
See https://github.com/golang/go/issues?q=milestone%3AGo1.19.4+label%3ACherryPickApproved
2022-12-08 17:04:41 -08:00
Roman Khavronenko
70c3664b40
alerts: remove show_at label for RequestErrorsToAPI alert (#3455)
Alert `RequestErrorsToAPI` could be permanently triggered due to
mistakes in clients configuration. However, such requests are unlikely
to cause VM health state change. So there is no need in displaying
this alert because there will be no correlation caused by it.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-07 09:48:59 -08:00
Denys Holius
5232775709
deployment/docker: bump Grafana version to v9.2.7 (#3454)
see https://grafana.com/blog/2022/11/29/grafana-security-release-new-versions-with-high-severity-security-fix-for-cve-2022-31097/
2022-12-07 09:48:15 -08:00
Roman Khavronenko
ff1c654006
deployment: update the README (#3447)
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-05 22:06:16 -08:00
Aliaksandr Valialkin
d2e34b8052
{dashboards,alerts}: subtitute {type="indexdb"} with {type=~"indexdb.*"} inside queries after 8189770c50
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337
2022-12-05 16:00:42 -08:00
Roman Khavronenko
73340dcb01
dashboards: add Disk space usage % and Disk space usage % by type panels (#3436)
The new panels have been added to the vmstorage and drilldown rows.

`Disk space usage %` is supposed to show disk space usage percentage.
This panel is now also referred by `DiskRunsOutOfSpace` alerting rule.
This panel has Drilldown option to show absolute values.

`Disk space usage % by type` shows the relation between datapoints
and indexdb size. It supposed to help identify cases when indexdb
starts to take too much disk space.
This panel has Drilldown option to show absolute values.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-05 00:19:36 -08:00
Roman Khavronenko
b2f45b4856
dashboards: update VM single dash (#3400)
The change list is the following:
* bump Grafana version to 9.2.6;
* replace old "Graph" panel with "TimeSeries" panel;
* show % usage of Mem and CPU additionally to of absolute values;
* `Caches` row was removed. All needed info for caches is now part of `Troubleshooting`;
* add Annotations for Alert triggers. Not all alerts are supposed to be displayed
on the dashboard, but only those with label `show_at: dashboard`.
See `alerts.yml` change.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-11-29 20:39:05 -08:00
Denys Holius
a90fbafa23
deployment/docker: bump grafana version to latest v9.2.6 (#3398) 2022-11-28 16:43:58 -08:00
Aliaksandr Valialkin
92a7f71ee1
deployment/docker: update VictoriaMetrics tag from v1.83.1 to v1.84.0 2022-11-25 22:29:20 -08:00
Aliaksandr Valialkin
d0461bec25
docs/CHANGELOG.md: cut v1.84.0 2022-11-25 19:59:30 -08:00
Roman Khavronenko
cae148d5c6
dashboards: cluster dashboard update (#3380)
The purpose of the update is to make the dash more usable
for large installations with many instances. Panels which showed
metrics per-instance (Mem, CPU) now are showing metrics per-job or min/max/avg
aggregations in % instead. This supposed to help immediately to identify
resource shortage and remain usable for small and big installations.

For cases when detailed info is needed, to the bottom of the dashboard
a new row `Drilldown` was added. Panels like Mem or CPU now contain
a `data-link` named `Drilldown` (cis shown on line click) which takes
user to more detailed panel.

The change list is the following:
* bump Grafana version to 9.1.0;
* replace old "Graph" panel with "TimeSeries" panel;
* improve Uptime panel to show number of instances per job;
* show % usage of Mem and CPU instead of absolute values;
* `Caches` row was removed. All needed info for caches is now part of `Troubleshooting`;
* add `Drilldown` section for detailed resource usage;
* add Annotations for Alert triggers. Not all alerts are supposed to be displayed
on the dashboard, but only those with label `show_at: dashboard`.
See `alerts-cluster.yml` change.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-11-24 13:20:10 -08:00
Aliaksandr Valialkin
afc35485c1
app/vminsert: add missing vm_relabel_config_* metrics after 03d88bc066
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3345
2022-11-22 00:48:13 +02:00
Roman Khavronenko
d1169c1559
vmagent: expose metrics for tracking config state (#3375)
Expose `vm_relabel_config_*` and `vm_promscrape_config_*` metrics
for tracking relabel and scrape configuration hot-reloads.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3345
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-11-22 00:48:12 +02:00
Roman Khavronenko
67c60dbf86
vmalert: bump alerting rules evaluation interval to reasonable 30s (#3374)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-11-21 16:48:09 +02:00
Roman Khavronenko
0475f8a38e
vmalert: add default list of alerting rules (#3373)
The default list of alerting rules contains the basic
rules for checking vmalert's health state and is recommended
to use for monitoring vmalert deployments.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-11-21 16:09:47 +02:00
Aliaksandr Valialkin
513257d536
deployment/docker: update VictoriaMetrics version from v1.83.0 to v1.83.1 2022-11-11 01:25:00 +02:00
Aliaksandr Valialkin
61bcbfd697
deployment/docker: update Go builder from v1.19.2 to v1.19.3
See https://github.com/golang/go/issues?q=milestone%3AGo1.19.3+label%3ACherryPickApproved
2022-11-05 10:19:24 +02:00