Commit graph

165 commits

Author SHA1 Message Date
Aliaksandr Valialkin
43cf221681
lib/logstorage: work-in-progress 2024-06-05 03:18:12 +02:00
Zakhar Bessarab
7dc9124ba7
deployment/docker: add scratch-based images (#6386)
### Describe Your Changes

Scratch based images will be using a separate tag: "(version)-scratch"
and will be built for the same architecture as regular images.
This is useful for environments with higher security standards. In this
case using alpine as base layer requires updating images more frequently
in order to get the latest updates for the base image, even in case the
user did not need to update VictoriaMetrics version.

Tested that scratch images work for:
- vmagent - enterprise with kafka and opensource
- cluster
- single-node

No issues observed so far.

cc: @tenmozes 

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-06-03 11:43:28 +02:00
Alexander Marshalov
a6cc7098fe
Update base Alpine image to 3.20.0 to avoid security risks (#6370)
fixes: CVE-2023-42366, CVE-2023-42363, CVE-2024-4603, CVE-2024-2511,
CVE-2024-24788, CVE-2024-24787
2024-05-28 19:36:15 +02:00
Aliaksandr Valialkin
bd75c0a898
deployment/docker/Makefile: group app-via-docker-* and package-via-docker-* rules with CGO_ENABLED=1 together for better maintainability 2024-05-12 23:09:55 +02:00
Aliaksandr Valialkin
cb19335a9f
deployment/docker/Makefile: rename EXTRA_ENVS to EXTRA_DOCKER_ENVS
The purpose of EXTRA_DOCKER_ENVS name is more clear than EXTRA_ENVS.

While at it, make the following small fixes:

- Pass GOARM=5 to Docker builder when building Docker packages for GOARCH=arm in the same way
  it is passed to the builder when building production binaries for GOARCH=arm.
  See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4965

- Set GCO_ENABLED=1 for package-via-docker-amd64, which has been accidentally removed in 07496d7d92

- Consistently use 'CGO_ENABLED=... GOARCH=...' order of env vars at package-via-docker-*,
  because this order is used in app-via-docker-*

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6158
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6172

This is a follow-up for 07496d7d92 and 7958f38864
2024-05-12 22:41:38 +02:00
Zhu Jiekun
02851d7800
chore: [deployment] upgrade from go 1.22.2 to 1.22.3 to include security fixes (#6238)
### Describe Your Changes

upgrade from go 1.22.2 to 1.22.3 to include security fixes. Also see:
- https://go.dev/doc/devel/release
-
https://github.com/golang/go/issues?q=milestone%3AGo1.22.3+label%3ACherryPickApproved

### Checklist

The following checks are **mandatory**:

- [X] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Jiekun <jiekun.dev@gmail.com>
2024-05-08 10:02:22 +02:00
Andrii Chubatiuk
07496d7d92
deployment: update makefile package-* targets (#6172)
Updated package targets in a same manner, how it's done for publish ones
in
7958f38864
2024-04-24 18:29:32 +02:00
Andrii Chubatiuk
7958f38864
deployment/docker: allow cross-platform building on arm64 platform (#6158)
Added x86_64 libraries to allow building cross-platform images on arm64
2024-04-23 12:12:10 +02:00
Aliaksandr Valialkin
d776c22592
deployment: update Go builder from 1.22.1 to 1.22.2
See https://github.com/golang/go/issues?q=milestone%3AGo1.22.2+label%3ACherryPickApproved
2024-04-04 01:40:28 +03:00
Aliaksandr Valialkin
97dd7e26ad
deployment/docker: update Go builder from Go1.21.7 to Go1.22.1
See https://github.com/golang/go/issues?q=milestone%3AGo1.22.1+label%3ACherryPickApproved
2024-03-06 21:04:11 +02:00
Aliaksandr Valialkin
62498a1e68
deployment/docker: downgrade Go builder from 1.22.0 to 1.21.7
Go1.22.0 contains the bug https://github.com/golang/go/issues/65705 ,
which prevents vmagent from normal operation.
2024-02-29 13:52:26 +02:00
Roman Khavronenko
f0b4dd7426
deployment: create a separate env for VictoriaLogs (#5857)
* deployment: create a separate env for VictoriaLogs

The new environment consists of the following components:
* VictoriaLogs
* fluentbit for collecting logs and sending to VictoriaLogs
* VictoriaMetrics for scraping and storing metrics from fluentbit and VictoriaLogs
* Grafana with VictoriaLogs datasource for monitoring

-----------------

The motivation for creating a separate environment is to simplify existing environments
and make it easier to update or modify them in future.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-02-26 10:33:04 +01:00
Aliaksandr Valialkin
95222b2079
all: upgrade Go builder from Go1.21.7 to Go1.22.0
See https://go.dev/doc/go1.22
2024-02-12 21:59:51 +02:00
Aliaksandr Valialkin
b431ccea5b
all: update Go builder from Go1.21.6 to Go1.21.7
See https://github.com/golang/go/issues?q=milestone%3AGo1.21.7+label%3ACherryPickApproved
2024-02-07 04:00:37 +02:00
Aliaksandr Valialkin
fcc8b14f86
deployment/docker: upgrade base Docker image from Alpine 3.19.0 to 3.19.1
See https://www.alpinelinux.org/posts/Alpine-3.19.1-released.html
2024-01-30 22:47:18 +02:00
hagen1778
0e5e502b3c
deployment/docker: follow-up 38b2a5bc44
* Simplify folder structure
* mention datasource in README

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-22 16:05:44 +01:00
Dmytro Kozlov
38b2a5bc44
deployment/docker: add grafana datasource to the docker-compose files (#5363)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3920
https://github.com/VictoriaMetrics/grafana-datasource/issues/113
2024-01-22 15:45:31 +01:00
Aliaksandr Valialkin
a74f6d63e0
deployment/docker: update Go builder from Go1.21.5 to Go1.21.6 2024-01-16 17:00:16 +02:00
Aliaksandr Valialkin
6d03779870
deployment/docker: update base Docker image from alpine:3.18.5 to alpine:3.19.0
See https://www.alpinelinux.org/posts/Alpine-3.19.0-released.html
2023-12-10 02:28:19 +02:00
Aliaksandr Valialkin
efbe25a678
deployment/docker: update Go builder from Go1.21.4 to Go1.21.5
See https://github.com/golang/go/issues?q=milestone%3AGo1.21.5+label%3ACherryPickApproved
2023-12-06 22:33:40 +02:00
Aliaksandr Valialkin
c7a2e4e90a
deployment/docker: update backe Docker image from alpine 3.18.4 to 3.18.5
See https://www.alpinelinux.org/posts/Alpine-3.15.11-3.16.8-3.17.6-3.18.5-released.html
2023-12-03 18:53:51 +02:00
Aliaksandr Valialkin
389f34cb57
deployment/docker: remove built binaries at bin folder after creating docker image from them at make publish-via-docker 2023-11-21 14:33:24 +02:00
Aliaksandr Valialkin
81ddee4f3a
Makefile: speedup release, publish and crossbuild rules by using parallel make 2023-11-20 22:53:23 +02:00
Aliaksandr Valialkin
2db1a664e1
deployment: update Go builder from Go1.21.3 to Go1.21.4
See https://github.com/golang/go/issues?q=milestone%3AGo1.21.4+label%3ACherryPickApproved
2023-11-10 22:28:44 +01:00
PhracturedBlue
2474281f1b
Support building images via podman (#4978) 2023-11-09 00:50:21 -08:00
Aliaksandr Valialkin
d984598e30
deployment/docker: update Go builder from Go1.21.1 to Go1.21.3
See https://github.com/golang/go/issues?q=milestone%3AGo1.21.2+label%3ACherryPickApproved
and https://github.com/golang/go/issues?q=milestone%3AGo1.21.3+label%3ACherryPickApproved
2023-10-12 09:41:41 +02:00
Aliaksandr Valialkin
5c28923c11
deployment/docker: update Alpine from 3.18.3 to 3.18.4
See https://alpinelinux.org/posts/Alpine-3.18.4-released.html
2023-10-02 21:52:38 +02:00
Aliaksandr Valialkin
2dc33e0ddc
all: update Go builder from Go1.21.0 to Go1.21.1
See https://github.com/golang/go/issues?q=milestone%3AGo1.21.1+label%3ACherryPickApproved
2023-09-07 11:36:16 +02:00
Aliaksandr Valialkin
448baf12a3
deployment/docker: properly build armv5 production builds for GOARCH=arm
Pass GOARM=5 when building GOARCH=arm production builds, since the default value for this env var
has been changed to GOARM=6 since Go1.21.0.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4965
and https://github.com/golang/go/issues/62475
2023-09-07 11:18:53 +02:00
Nikolay
c37d7dd567
deployment/docker: disable provenance in buildx (#4911)
* deployment/docker: disable provenance in buildx
it must fix an issue with multi-platform manifest generation
at buildx >= 0.10 backward compatibility was broken and generated image cannot be used with docker systems that doesn't support oci.
disabling attestat temporary fixes it.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4907
https://docs.docker.com/build/attestations/slsa-provenance/

* Update docs/CHANGELOG.md

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-08-29 16:29:14 +02:00
Aliaksandr Valialkin
e1235267a0
deployment/docker/Makefile: upgrade base Docker image from alpine:3.18.2 to alpine:3.18.3
See https://alpinelinux.org/posts/Alpine-3.15.10-3.16.7-3.17.5-3.18.3-released.html
2023-08-12 05:59:48 -07:00
Aliaksandr Valialkin
e0017b4d47
all: update Go builder from Go1.20.7 to Go1.21.0
See https://tip.golang.org/doc/go1.21
and https://go.dev/blog/go1.21
2023-08-11 06:25:54 -07:00
Roman Khavronenko
4c854c3ae2
security: bump go version from 1.20.6 to 1.20.7 (#4773)
The update includes a security fix to the crypto/tls package,
as well as bug fixes to the assembler and the compiler.

See the list of issues addressed in Go1.20.7 here:
https://github.com/golang/go/issues?q=milestone%3AGo1.20.7+label%3ACherryPickApproved

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-08-03 11:48:37 +02:00
Anton Tykhyy
f1a4c0b614
Remove some repetitions in docker Makefile (#4764) 2023-08-02 15:04:38 +04:00
Aliaksandr Valialkin
30cdcc751d
all: update Go builder from 1.20.5 to 1.20.6
See https://github.com/golang/go/issues?q=milestone%3AGo1.20.6+label%3ACherryPickApproved
2023-07-12 00:59:59 -07:00
Sergey
9bea7610f0
security: update base Alpine image to 3.18.2 to avoid security risks (#4571)
libcrypto3 and libssl3 in Alpine 3.18.0 have versions `3.1.0-r4`
which contains CVE-2023-2650:
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-2650
Use ALpine image 3.18.2 which contains fixed versions of libssl3
and libcrypto3: 3.1.1-r0
NB: In Openshift these containers are marked as vulnerabilities
because of these CVEs.
2023-07-04 09:23:53 +02:00
Max Golionko
d4099a75be
CI: disable docker scan, enable auto release to sandbox (#4476)
* disable docker scan

* disable nightly, enable auto release to sandbox

* remove whitespace
2023-06-30 13:45:45 +02:00
Roman Khavronenko
476c7bdd6f
all: update Go builder from Go1.20.4 to Go1.20.5 (#4427)
See https://github.com/golang/go/issues?q=milestone%3AGo1.20.5+label%3ACherryPickApproved

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-06-09 09:42:55 +02:00
Aliaksandr Valialkin
73dfb030dd
deployment/docker/Makefile: use alpine 3.17.3 instead of alpine 3.18.0 for certs image, since alpine 3.18.0 doesnt work for cross-platform builds 2023-05-18 14:10:29 -07:00
Denys Holius
c605d64a95
deployment/docker/Makefile: updated docker compose commands regarding migration from V1 to V2 (#4314)
deployment/docker/Makefile: updated docker compose commands regarding migration from V1 to V2
2023-05-17 13:14:24 +02:00
Aliaksandr Valialkin
2613110f75
deployment/docker: update base docker image from 3.17.3 to 3.18.0
See https://www.alpinelinux.org/posts/Alpine-3.18.0-released.html
2023-05-12 17:31:21 -07:00
Aliaksandr Valialkin
1b288e0a05
all: update Go builder from Go1.20.3 to Go1.20.4
See https://github.com/golang/go/issues?q=milestone%3AGo1.20.4+label%3ACherryPickApproved
2023-05-08 09:40:55 -07:00
Zakhar Bessarab
d71a2605d1
deployment/docker: allow overriding docker namespace (#4265)
It makes it easier for users who build and self-host images to publish their images without changing tags manually.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-05-08 10:34:05 +04:00
Aliaksandr Valialkin
5074cc672a
all: update Go builder from Go1.20.2 to Go1.20.3
See https://github.com/golang/go/issues?q=milestone%3AGo1.20.3+label%3ACherryPickApproved
2023-04-05 13:37:22 -07:00
Aliaksandr Valialkin
cddfc4d3f8
deployment/docker: update base Docker image from Alpine 3.17.2 to Alpine 3.17.3
This fixes security issues from https://alpinelinux.org/posts/Alpine-3.17.3-released.html

This is a follow-up for 59c350d0d2
2023-03-31 22:46:27 -07:00
Aliaksandr Valialkin
6d56149b9f
deployment/docker/Makefile: properly add amd64 suffix to windows binary names 2023-03-18 23:29:44 -07:00
Aliaksandr Valialkin
4ea27d6f6a
deployment/docker/Makefile: build CGO-enabled vmagent for GOARCH=arm64
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2271

This is a follow-up for 565497fb074321caedea38d5151044d98d92d759
2023-03-18 23:15:31 -07:00
Nikolay
91cbb9063d
Vmagent kafka updates (#535)
* app/vmagent: allow vm proto for kafka consumer and producer
it should reduce network usage up to 50%.
According to benchmarks without any encoding at kafka topic, it reduces traffic up to 50%.
With enabled zstd at kafka topic, it shows no diffence in traffic. So it
doesn't make much sense to use it.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225

* mention eb61a7dd68b834b08d01727a918f207700348ada at changelog

* app/vmagent: bumps kafka lib version
it allows compiling vmagent for arm64 machines
fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2271

* mention d19b1a888248c96cfd7ccee00ba6f596d89be1d7 at change log

* app/vmagent: adds natural concurrency for kafka consumer
it should improve performance for data consumption
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1957

* mention change 0c143bb22ca2e7e0b7eec9bc84a94ee2b41626ca

* Update app/vmagent/kafka/consumer.go

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* Update app/vmagent/kafka/consumer_cgo.go

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-03-15 13:03:44 -07:00
Roman Khavronenko
3de7fc5c71
security: bump go version to 1.20.2 (#3935)
upgrade Go builder from Go1.20.1 to Go1.20.2
See the list of issues addressed in Go1.20.2 here (https://github.com/golang/go/issues?q=milestone%3AGo1.20.2+label%3ACherryPickApproved).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-03-09 13:20:54 +01:00
Aliaksandr Valialkin
11ce30820b
all: update Go builder from Go1.20.0 to Go1.20.1
See https://github.com/golang/go/issues?q=milestone%3AGo1.20.1+label%3ACherryPickApproved
2023-02-14 23:05:16 -08:00