Commit graph

439 commits

Author SHA1 Message Date
Andrii Chubatiuk
dfcab4a47f
deployment/victorialogs: rename not executable compose files (#7124)
### Describe Your Changes

Renamed base compose files to prevent envs to be created from them

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-09-29 09:54:45 +02:00
Roman Khavronenko
fd890a2771
deployment: separate datasource between single and cluster versions (#7119)
Before, single and cluster deployments were provisioned with both
Grafana datasources: single and cluster. But this resulted into a
problem: single DS didn't work for cluster and vice versa. See
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7113

This PR splits datasource file into 2 files: single and cluster. Now,
these files are separately provisioned to single and cluster deployments
correspondingly.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-09-27 19:08:05 +02:00
Andrii Chubatiuk
00912bfa0f
docs/victorialogs: change fluentd input type in examples (#7123)
### Describe Your Changes

Use fluentd logging driver in examples to have enriched data in
VictoriaLogs

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-09-27 15:09:36 +02:00
Andrii Chubatiuk
05a64a8c14
victorialogs: marked fluentd support in roadmap, added syslog example (#7098)
### Describe Your Changes

Marked fluentd in victorialogs roadmap
Added fluentd syslog example setup

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-09-27 14:38:39 +02:00
Aliaksandr Valialkin
86c0eb816c
deployment: update VictoriaLogs docker image from v0.30.1-victorialogs to v0.31.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.31.0-victorialogs
2024-09-27 13:58:54 +02:00
Aliaksandr Valialkin
b670b0e9ff
deployment: update VictoriaLogs docker image tag from v0.30.0-victorialogs to v0.30.1-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.30.1-victorialogs
2024-09-27 11:26:15 +02:00
Aliaksandr Valialkin
6f1fde24dc
deployment: update VictoriaLogs docker image from v0.29.0-victorialogs to v0.30.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.30.0-victorialogs
2024-09-27 09:34:09 +02:00
Andrii Chubatiuk
e75ae1b274
deployment: restructure victorialogs examples (#6971)
### Describe Your Changes

- Use common compose.yaml file for all victorialogs setups to set
version in a single place and override it on demand for each agent and
protocol
- Replaced multiple victorialogs instances in HA setup with single setup
with `deploy.replica` parameter set
- Added fluentd setup

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-09-25 18:33:26 +02:00
Dmytro Kozlov
91b28d0527
deployment/docker: update grafana datasources to the latest version (#7083)
### Describe Your Changes

Updated grafana plugins to the latest releases 

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-09-24 11:51:18 +02:00
hagen1778
a5c002edef
deployment/alerts: fix copy&paste typo in TooHighGoroutineSchedulingLatency
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-09-24 11:48:19 +02:00
Roman Khavronenko
4d0b41e63b
deployment: add panel and alerts for displying go scheduler latency (#7078)
The panel and alerting rule should help to understand whether VM
component doesn't have enough CPU resources or gets throttled. The alert
is applicable for all VM components.
The panel was added to vmalert, vmagent, vmsingle, vm clusert and
victorialogs dashes.

-------------------

This alerting rule should have help us identify resource shortage for
sandbox vmagent - see [this
link](https://play.victoriametrics.com/select/accounting/1/6a716b0f-38bc-4856-90ce-448fd713e3fe/prometheus/graph/#/?g0.range_input=23d13h25m25s424ms&g0.end_input=2024-09-23T14%3A11%3A00&g0.relative_time=none&g0.tab=0&g0.expr=histogram_quantile%280.99%2C+sum%28rate%28go_sched_latencies_seconds_bucket%7Bjob%3D%22vmagent-monitoring-vmagent%22%7D%5B5m%5D%29%29+by+%28le%2C+job%2C+instance%29%29+%3E+0.1)
for example. We weren't aware of resource shortage, because VM metrics
assumed this vmagent had 1vCPU while in fact its limit was 0.2vCPU.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-09-23 16:54:42 +02:00
Thomas Danielsson
69b976f08e
docs: typo filebeat -> fluent-bit (#7000)
### Describe Your Changes

Fix a typo; `filebeat.yml` -> `fluent-bit.conf`

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Thomas Danielsson <thomas@elajt.se>
2024-09-19 14:33:12 +02:00
Thomas Danielsson
75cc7922c3
typo: added missing '{' (#7044)
### Describe Your Changes

Added missing `{` in vmalert rule.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-09-19 00:11:30 -07:00
Artem Navoiev
277fed9990
victorialogs: add HA example for logstash and fluentbit (#6968)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-09-09 14:33:05 -07:00
Artem Navoiev
47cd7bb4c1
docs: update roadmap, mark telegraf and otel as done and link the docs
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-09-09 17:35:14 +02:00
Aliaksandr Valialkin
0ed835026f
deployment: update VictoriaLogs from v0.28.0-victorialogs to v0.29.0-victorialogs
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.29.0-victorialogs
2024-09-08 21:23:16 +02:00
Aliaksandr Valialkin
4fbdde5852
deployment/docker: update base Alpine docker image from 3.20.2 to 3.20.3
See https://alpinelinux.org/posts/Alpine-3.17.10-3.18.9-3.19.4-3.20.3-released.html
2024-09-08 19:26:48 +02:00
Aliaksandr Valialkin
5261a84119
deployment: update Go builder from Go1.23.0 to Go1.23.1
See https://github.com/golang/go/issues?q=milestone%3AGo1.23.1+label%3ACherryPickApproved
2024-09-06 22:51:15 +02:00
Dmytro Kozlov
85acc8a7c4
deployment: upgrade datasource versions (#6935)
### Describe Your Changes

Upgraded victoriametrics and victorialogs data source versions.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-09-04 16:30:37 +02:00
Andrii Chubatiuk
01430a155c
vlinsert: added opentelemetry logs support
Commit adds the following changes:

* Adds support of OpenTelemetry logs for Victoria Logs with protobuf encoded messages

*  json encoding is not supported for the following reasons:
   - It brings a lot of fragile code, which works inefficiently.
   - json encoding is impossible to use with language SDK.

* splits metrics and logs structures at lib/protoparser/opentelemetry/pb package.

* adds docs with examples for opentelemetry logs.

---
Related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4839

Co-authored-by: AndrewChubatiuk <andrew.chubatiuk@gmail.com>
Co-authored-by: f41gh7 <nik@victoriametrics.com>
2024-09-03 20:12:05 +02:00
Andrii Chubatiuk
1731c0eabf
app/vlinsert: support getting _msg_field, _time_field, _stream_fields and _ignore_fields from headers
*  Many collectors don't support forwarding url query params to the remote system. It makes impossible to define stream fields for it. Workaround with proxy between VictoriaLogs and log shipper is too complicated solution.

* This commit adds the following changes:
 * Adds fallback to to headers params, if query param is empty for:
     _msg_field -> VL-Msg-Field
    _stream_fields -> VL-Stream-Fields
    _ignore_fields -> VL-Ignore-Fields
    _time_field -> VL-Time-Field
 * removes deprecations from victorialogs compose files, added more
output format examples for logstash, telegraf, fluent-bit

 related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5310
2024-09-03 17:43:26 +02:00
Aliaksandr Valialkin
91456ab5bb
all: suppress InvalidDefaultArgInFrom warning emitted by docker build when building Docker packages via make package-* command
Recent versions of `docker build` started generating the InvalidDefaultArgInFrom warning if Dockerfile contains
an ARG without default value. While this warning doesn't affect building Docker packages via `make package-*` commands,
it is better suppressing the warning, so it doesn't clutter `make package-*` output with the noise,
which can hide real issues in the future.
2024-09-03 14:00:28 +02:00
Marco Maurer (-Kilchhofer)
f17fca718d
deployment/alerts: add missing description for VMagent alerts (#6921)
The 3 alerts for VMagent:
- `RejectedRemoteWriteDataBlocksAreDropped`
- `TooManyScrapeErrors`
- `TooManyWriteErrors`

missed the description annotation.
I moved the summary to description and added a generic summary to these
alerts.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Marco Maurer <marco.kilchhofer@gmail.com>
2024-09-03 10:26:19 +02:00
hagen1778
b58a10c389
deployment: bump VM version to 1.103.0
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-29 15:13:19 +02:00
Roman Khavronenko
1b9f3b39b4
deployment/docker: update Go builder from Go1.22.5 to Go1.23.0 (#6861)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-08-22 23:55:50 +02:00
f41gh7
b85e3f8bbd
deployment: update compose images to v1.102.1 release
Signed-off-by: f41gh7 <nik@victoriametrics.com>
2024-08-02 11:29:12 +02:00
Hui Wang
b515a7b69b
security: upgrade base docker image (Alpine) from 3.20.1 to 3.20.2 (#6684)
See https://www.alpinelinux.org/posts/Alpine-3.20.1-released.html

>including security fix for:
OpenSSL CVE-2024-5535
2024-07-23 13:20:06 +02:00
Aliaksandr Valialkin
45794de0f9
deployment: update VictoriaMetrics Docker image from v1.101.0 to v1.102.0
See https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.102.0
2024-07-17 22:34:48 +02:00
Aliaksandr Valialkin
3224a12e25
Revert "deployment: build image for vmagent streamaggr benchmark (#6515)"
This reverts commit 6b128da811.

Reason for revert: this complicates and slows down CI/CD without giving significant benefits in return.

The idea of automatic building, publishing and deploying Docker images to our playground on every pull request
and commit isn't very bright because of the following reasons:

- It slows down CI/CD pipeline
- It increases costs on CPU time spent at CI/CD pipeline
- It contradicts goal #7 at https://docs.victoriametrics.com/goals/#goals and non-goal #8 at https://docs.victoriametrics.com/goals/#non-goals

The previous workflow was much better - if we need to deploy some new Docker image at playground or staging environment,
then just __manually__ build and deploy the needed Docker image there. If the manual process requires making too many
steps, then think on how to automate these steps into a single Makefile command.

Updates https://github.com/VictoriaMetrics/ops/pull/1297
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6515
2024-07-16 13:27:06 +02:00
Aliaksandr Valialkin
b173f24041
docs/VictoriaLogs/CHANGELOG.md: cut v0.28.0-victorialogs release 2024-07-10 03:08:16 +02:00
Hui Wang
8e9f98e725
security: upgrade base docker image (Alpine) from 3.20.0 to 3.20.1
See https://www.alpinelinux.org/posts/Alpine-3.20.1-released.html

>including security fixes for:
OPENSSL
[CVE-2024-4741](https://security.alpinelinux.org/vuln/CVE-2024-4741)
BUSYBOX
[CVE-2023-42364](https://security.alpinelinux.org/vuln/CVE-2023-42364)
[CVE-2023-42365](https://security.alpinelinux.org/vuln/CVE-2023-42365)
2024-07-09 11:38:05 +02:00
hagen1778
3d164d7e31
deployment: fix typo after cb19335a9f
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-07-05 11:02:01 +02:00
hagen1778
ee66fb4387
docs: update refereneces to victoriametrics-datasource plugin
The plugin was renamed in https://github.com/VictoriaMetrics/victoriametrics-datasource/pull/178

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-07-05 09:46:17 +02:00
Roman Khavronenko
c429bbf889
app/vmalert: add examples for source override (#6561)
The change adds a new docs section with examples on how source can be
overridden. It should address questions like
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6536

While there, fix the example in `external.alert.source` cmd-line flag
and docker-compose examples.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-07-05 08:47:59 +02:00
Aliaksandr Valialkin
bbf1f33818
docs/VictoriaLogs/CHANGELOG.md: cut v0.27.1-victorialogs 2024-07-05 01:22:24 +02:00
Aliaksandr Valialkin
f5518b2adc
deployment/docker: update Go builder from Go1.22.4 to Go1.22.5
See https://github.com/golang/go/issues?q=milestone%3AGo1.22.5+label%3ACherryPickApproved
2024-07-03 00:07:09 +02:00
Aliaksandr Valialkin
d506e47c5a
docs/VictoriaLogs/CHANGELOG.md: cut v0.27.0-victorialogs 2024-07-02 01:43:46 +02:00
Aliaksandr Valialkin
839596c00d
docs/VictoriaLogs/CHANGELOG.md: cut v0.26.1-victorialogs 2024-07-01 02:32:51 +02:00
Aliaksandr Valialkin
353e26cdfc
docs/VictoriaLogs/CHANGELOG.md: cut v0.26.0-victorialogs 2024-07-01 01:49:49 +02:00
Aliaksandr Valialkin
9877c086b9
docs/VictoriaLogs/CHANGELOG.md: cut v0.25.0-victorialogs 2024-06-28 19:17:49 +02:00
Aliaksandr Valialkin
87f1c8bd6c
lib/logstorage: work-in-progress 2024-06-27 14:20:43 +02:00
Aliaksandr Valialkin
0c90eeb388
docs/VictoriaLogs/CHANGELOG.md: cut v0.23.0-victorialogs release 2024-06-25 14:59:15 +02:00
hagen1778
bb0449d7ba
deployment: set consistent Grafana version for VM datasource env
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-25 13:57:52 +02:00
hagen1778
0dcb3acbf7
deployment: fix incorrect link to vmui from VictoriaMetrics plugin in Grafana
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-25 13:56:52 +02:00
hagen1778
5f218a772b
deployment: simplify Grafana plugins installation
rm custom scripts for downloading Grafana plugins for
VictoriaMetrics and VictoriaLogs. Use `GF_INSTALL_PLUGINS` instead.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-06-25 13:36:34 +02:00
Nikolay
14b9ef1e4d
dashboards: add dashboard and alerts for vmauth (#6491)
Signed-off-by: f41gh7 <nik@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2024-06-25 11:15:29 +02:00
Aliaksandr Valialkin
de7450b7e0
lib/logstorage: work-in-progress 2024-06-24 23:27:12 +02:00
Andrii Chubatiuk
6b128da811
deployment: build image for vmagent streamaggr benchmark (#6515)
### Describe Your Changes

optionally build vmagent image for benchmark
needed for https://github.com/VictoriaMetrics/ops/pull/1297

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-06-24 16:28:50 +02:00
Aliaksandr Valialkin
7229dd8c33
lib/logstorage: work-in-progress 2024-06-20 03:10:08 +02:00
Aliaksandr Valialkin
3eda4617c0
app/vlinsert: properly parse timestamps with nanosecond precision at /insert/jsonline HTTP endpoint
This has been broken in 2b6a634ec0
2024-06-18 00:23:25 +02:00