mirror of
https://github.com/VictoriaMetrics/VictoriaMetrics.git
synced 2024-11-21 14:44:00 +00:00
victorialogs: add HA example for logstash and fluentbit (#6968)
### Describe Your Changes Please provide a brief description of the changes you made. Be as specific as possible to help others understand the purpose and impact of your modifications. ### Checklist The following checks are **mandatory**: - [ ] My change adheres [VictoriaMetrics contributing guidelines](https://docs.victoriametrics.com/contributing/). --------- Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
This commit is contained in:
Artem Navoiev
GitHub
parent
153cceb124
commit
277fed9990
14 changed files with 541 additions and 1 deletions
|
|
@ -0,0 +1,88 @@
|
|||
# Docker compose Fluentbit integration with VictoriaLogs for docker. High-Availability example
|
||||
|
||||
The folder contains the example of integration of [fluentbit](https://docs.fluentbit.io/manual) with VictoriaLogs Single-Nodes(s) and [vmauth](https://docs.victoriametrics.com/vmauth/) for achieving High Availability.
|
||||
|
||||
Check [this documentation](https://docs.victoriametrics.com/victorialogs/#high-availability) with a description of the architecture and components.
|
||||
|
||||
To spin-up environment run the following command:
|
||||
|
||||
```shell
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
To shut down the docker-compose environment run the following command:
|
||||
|
||||
```shell
|
||||
docker compose down
|
||||
docker compose rm -f
|
||||
```
|
||||
|
||||
The docker compose file contains the following components:
|
||||
|
||||
* fluentbit - fluentbit is configured to collect logs from the `docker`, you can find configuration in the `fluent-bit.conf`. It writes data in VictoriaLogs
|
||||
* VictoriaLogs - the two instances of log database, they accept the data from `fluentbit` by json line protocol
|
||||
* vmauth - load balancer for proxying requests to one of VictoriaLogs
|
||||
|
||||
Querying the data
|
||||
|
||||
* [vmui](https://docs.victoriametrics.com/victorialogs/querying/#vmui) - a web UI is accessible by `http://localhost:8427/select/vmui/`
|
||||
* for querying the data via command-line please check [these docs](https://docs.victoriametrics.com/victorialogs/querying/#command-line)
|
||||
|
||||
|
||||
the example of fluentbit configuration(`filebeat.yml`)
|
||||
|
||||
```text
|
||||
[INPUT]
|
||||
name tail
|
||||
path /var/lib/docker/containers/**/*.log
|
||||
path_key path
|
||||
multiline.parser docker, cri
|
||||
Parser docker
|
||||
Docker_Mode On
|
||||
|
||||
[INPUT]
|
||||
Name syslog
|
||||
Listen 0.0.0.0
|
||||
Port 5140
|
||||
Parser syslog-rfc3164
|
||||
Mode tcp
|
||||
|
||||
[SERVICE]
|
||||
Flush 1
|
||||
Parsers_File parsers.conf
|
||||
|
||||
[OUTPUT]
|
||||
Name http
|
||||
Match *
|
||||
host victorialogs-2
|
||||
port 9428
|
||||
compress gzip
|
||||
uri /insert/jsonline?_stream_fields=stream,path&_msg_field=log&_time_field=date
|
||||
format json_lines
|
||||
json_date_format iso8601
|
||||
header AccountID 0
|
||||
header ProjectID 0
|
||||
|
||||
[OUTPUT]
|
||||
Name http
|
||||
Match *
|
||||
host victorialogs-1
|
||||
port 9428
|
||||
compress gzip
|
||||
uri /insert/jsonline?_stream_fields=stream,path&_msg_field=log&_time_field=date
|
||||
format json_lines
|
||||
json_date_format iso8601
|
||||
header AccountID 0
|
||||
header ProjectID 0
|
||||
```
|
||||
|
||||
Please, note that `_stream_fields` parameter must follow recommended [best practices](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) to achieve better performance.
|
||||
|
||||
The example of vmauth configuration (`auth.yml`)
|
||||
|
||||
```yaml
|
||||
unauthorized_user:
|
||||
url_prefix:
|
||||
- http://victorialogs-1:9428
|
||||
- http://victorialogs-2:9428
|
||||
```
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
# balance load among victorialogs instances
|
||||
# see https://docs.victoriametrics.com/vmauth/#load-balancing
|
||||
unauthorized_user:
|
||||
url_prefix:
|
||||
- http://victorialogs-1:9428
|
||||
- http://victorialogs-2:9428
|
||||
|
|
@ -0,0 +1,38 @@
|
|||
services:
|
||||
fluentbit:
|
||||
image: cr.fluentbit.io/fluent/fluent-bit:3.0.7
|
||||
volumes:
|
||||
- /var/lib/docker/containers:/var/lib/docker/containers:ro
|
||||
- ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
|
||||
depends_on: [victorialogs-1,victorialogs-2]
|
||||
ports:
|
||||
- "5140:5140"
|
||||
|
||||
victorialogs-1:
|
||||
image: docker.io/victoriametrics/victoria-logs:v0.29.0-victorialogs
|
||||
volumes:
|
||||
- victorialogs-fluentbit-vl-ha-single-1:/vlogs
|
||||
command:
|
||||
- -storageDataPath=/vlogs
|
||||
victorialogs-2:
|
||||
image: docker.io/victoriametrics/victoria-logs:v0.29.0-victorialogs
|
||||
volumes:
|
||||
- victorialogs-fluentbit-vl-ha-single-2:/vlogs
|
||||
command:
|
||||
- -storageDataPath=/vlogs
|
||||
vmauth:
|
||||
container_name: vmauth
|
||||
image: victoriametrics/vmauth:v1.103.0
|
||||
depends_on:
|
||||
- "victorialogs-1"
|
||||
- "victorialogs-2"
|
||||
volumes:
|
||||
- ./auth.yml:/etc/auth.yml
|
||||
command:
|
||||
- '--auth.config=/etc/auth.yml'
|
||||
ports:
|
||||
- 8427:8427
|
||||
restart: always
|
||||
volumes:
|
||||
victorialogs-fluentbit-vl-ha-single-1:
|
||||
victorialogs-fluentbit-vl-ha-single-2:
|
||||
|
|
@ -0,0 +1,42 @@
|
|||
[INPUT]
|
||||
name tail
|
||||
path /var/lib/docker/containers/**/*.log
|
||||
path_key path
|
||||
multiline.parser docker, cri
|
||||
Parser docker
|
||||
Docker_Mode On
|
||||
|
||||
[INPUT]
|
||||
Name syslog
|
||||
Listen 0.0.0.0
|
||||
Port 5140
|
||||
Parser syslog-rfc3164
|
||||
Mode tcp
|
||||
|
||||
[SERVICE]
|
||||
Flush 1
|
||||
Parsers_File parsers.conf
|
||||
|
||||
[OUTPUT]
|
||||
Name http
|
||||
Match *
|
||||
host victorialogs-2
|
||||
port 9428
|
||||
compress gzip
|
||||
uri /insert/jsonline?_stream_fields=stream,path&_msg_field=log&_time_field=date
|
||||
format json_lines
|
||||
json_date_format iso8601
|
||||
header AccountID 0
|
||||
header ProjectID 0
|
||||
|
||||
[OUTPUT]
|
||||
Name http
|
||||
Match *
|
||||
host victorialogs-1
|
||||
port 9428
|
||||
compress gzip
|
||||
uri /insert/jsonline?_stream_fields=stream,path&_msg_field=log&_time_field=date
|
||||
format json_lines
|
||||
json_date_format iso8601
|
||||
header AccountID 0
|
||||
header ProjectID 0
|
||||
|
|
@ -0,0 +1,71 @@
|
|||
# Docker compose Logstash integration with VictoriaLogs for docker. High-Availability example
|
||||
|
||||
The folder contains the example of integration of [logstash](https://www.elastic.co/logstash) with VictoriaLogs Single-Node(s) and [vmauth](https://docs.victoriametrics.com/vmauth/) for achieving High Availability.
|
||||
|
||||
Check [this documentation](https://docs.victoriametrics.com/victorialogs/#high-availability) with a description of the architecture and components.
|
||||
|
||||
To spin-up environment run the following command:
|
||||
|
||||
```shell
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
To shut down the docker-compose environment run the following command:
|
||||
|
||||
```shell
|
||||
docker compose down
|
||||
docker compose rm -f
|
||||
```
|
||||
|
||||
The docker compose file contains the following components:
|
||||
|
||||
* logstash - logstash is configured to read docker log files, you can find configuration in the `pipeline.conf`. It writes data in two instances of VictoriaLogs
|
||||
* VictoriaLogs - the two instances of log database, they accept the data from `fluentbit` by json line protocol
|
||||
* vmauth - load balancer for proxying requests to one of VictoriaLogs
|
||||
|
||||
Querying the data
|
||||
|
||||
* [vmui](https://docs.victoriametrics.com/victorialogs/querying/#vmui) - a web UI is accessible by `http://localhost:8427/select/vmui/`
|
||||
* for querying the data via command-line please check [these docs](https://docs.victoriametrics.com/victorialogs/querying/#command-line)
|
||||
|
||||
|
||||
Here is an example of logstash configuration(`pipeline.conf`):
|
||||
|
||||
```text
|
||||
input {
|
||||
file {
|
||||
path => "/var/lib/docker/containers/*/*.log"
|
||||
start_position => "beginning"
|
||||
type => "docker"
|
||||
sincedb_path => "/dev/null"
|
||||
codec => "json"
|
||||
add_field => {
|
||||
"path" => "%{[@metadata][path]}"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
output {
|
||||
http {
|
||||
url => "http://victorialogs-1:9428/insert/jsonline?_stream_fields=host.name,stream&_msg_field=log&_time_field=time"
|
||||
format => "json"
|
||||
http_method => "post"
|
||||
}
|
||||
http {
|
||||
url => "http://victorialogs-2:9428/insert/jsonline?_stream_fields=host.name,stream&_msg_field=log&_time_field=time"
|
||||
format => "json"
|
||||
http_method => "post"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Please, note that `_stream_fields` parameter must follow recommended [best practices](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) to achieve better performance.
|
||||
|
||||
The example of vmauth configuration (`auth.yml`)
|
||||
|
||||
```yaml
|
||||
unauthorized_user:
|
||||
url_prefix:
|
||||
- http://victorialogs-1:9428
|
||||
- http://victorialogs-2:9428
|
||||
```
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
# balance load among victorialogs instances
|
||||
# see https://docs.victoriametrics.com/vmauth/#load-balancing
|
||||
unauthorized_user:
|
||||
url_prefix:
|
||||
- http://victorialogs-1:9428
|
||||
- http://victorialogs-2:9428
|
||||
|
|
@ -0,0 +1,41 @@
|
|||
services:
|
||||
logstash:
|
||||
image: docker.elastic.co/logstash/logstash:8.8.1
|
||||
user: root
|
||||
volumes:
|
||||
- ./pipeline.conf:/usr/share/logstash/pipeline/logstash.conf:ro
|
||||
- ./logstash.yml:/usr/share/logstash/config/logstash.yml:ro
|
||||
- /var/lib/docker/containers:/var/lib/docker/containers:ro
|
||||
depends_on: [victorialogs-1,victorialogs-2]
|
||||
ports:
|
||||
- "5140:5140"
|
||||
|
||||
victorialogs-1:
|
||||
image: docker.io/victoriametrics/victoria-logs:v0.29.0-victorialogs
|
||||
volumes:
|
||||
- victorialogs-logstash-vl-ha-single-1:/vlogs
|
||||
command:
|
||||
- -storageDataPath=/vlogs
|
||||
victorialogs-2:
|
||||
image: docker.io/victoriametrics/victoria-logs:v0.29.0-victorialogs
|
||||
volumes:
|
||||
- victorialogs-logstash-vl-ha-single-2:/vlogs
|
||||
command:
|
||||
- -storageDataPath=/vlogs
|
||||
vmauth:
|
||||
container_name: vmauth
|
||||
image: victoriametrics/vmauth:v1.103.0
|
||||
depends_on:
|
||||
- "victorialogs-1"
|
||||
- "victorialogs-2"
|
||||
volumes:
|
||||
- ./auth.yml:/etc/auth.yml
|
||||
command:
|
||||
- '--auth.config=/etc/auth.yml'
|
||||
ports:
|
||||
- 8427:8427
|
||||
restart: always
|
||||
|
||||
volumes:
|
||||
victorialogs-logstash-vl-ha-single-1:
|
||||
victorialogs-logstash-vl-ha-single-2:
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
http.host: 0.0.0.0
|
||||
xpack.monitoring.enabled: false
|
||||
|
|
@ -0,0 +1,25 @@
|
|||
input {
|
||||
file {
|
||||
path => "/var/lib/docker/containers/*/*.log"
|
||||
start_position => "beginning"
|
||||
type => "docker"
|
||||
sincedb_path => "/dev/null"
|
||||
codec => "json"
|
||||
add_field => {
|
||||
"path" => "%{[@metadata][path]}"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
output {
|
||||
http {
|
||||
url => "http://victorialogs-1:9428/insert/jsonline?_stream_fields=host.name,stream&_msg_field=log&_time_field=time"
|
||||
format => "json"
|
||||
http_method => "post"
|
||||
}
|
||||
http {
|
||||
url => "http://victorialogs-2:9428/insert/jsonline?_stream_fields=host.name,stream&_msg_field=log&_time_field=time"
|
||||
format => "json"
|
||||
http_method => "post"
|
||||
}
|
||||
}
|
||||
104
deployment/docker/victorialogs/vector-ha-single-node/README.md
Normal file
104
deployment/docker/victorialogs/vector-ha-single-node/README.md
Normal file
|
|
@ -0,0 +1,104 @@
|
|||
# Docker compose Vector integration with VictoriaLogs for docker. High-Availability example
|
||||
|
||||
The folder contains the example of integration of [vector](https://vector.dev/docs/) with VictoriaLogs Single-Node(s) and [vmauth](https://docs.victoriametrics.com/vmauth/) for achieving High Availability.
|
||||
|
||||
Check [this documentation](https://docs.victoriametrics.com/victorialogs/#high-availability) with a description of the architecture and components.
|
||||
|
||||
|
||||
To spin-up environment run the following command:
|
||||
|
||||
```shell
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
To shut down the docker-compose environment run the following command:
|
||||
|
||||
```shell
|
||||
docker compose down
|
||||
docker compose rm -f
|
||||
```
|
||||
|
||||
The docker compose file contains the following components:
|
||||
|
||||
* vector - vector is configured to collect logs from the `docker`, you can find configuration in the `vector.yaml`. It writes data in two instances of VictoriaLogs
|
||||
* VictoriaLogs - the two instances of log database, they accept the data from `vector` by json line protocol
|
||||
* vmauth - load balancer for proxying requests to one of VictoriaLogs
|
||||
|
||||
Querying the data
|
||||
|
||||
* [vmui](https://docs.victoriametrics.com/victorialogs/querying/#vmui) - a web UI is accessible by `http://localhost:8427/select/vmui/`
|
||||
* for querying the data via command-line please check [these docs](https://docs.victoriametrics.com/victorialogs/querying/#command-line)
|
||||
|
||||
|
||||
the example of vector configuration(`vector.yaml`)
|
||||
|
||||
```yaml
|
||||
api:
|
||||
enabled: true
|
||||
address: 0.0.0.0:8686
|
||||
sources:
|
||||
docker:
|
||||
type: docker_logs
|
||||
transforms:
|
||||
msg_parser:
|
||||
type: remap
|
||||
inputs:
|
||||
- docker
|
||||
source: |
|
||||
if exists(.message) {
|
||||
.log, err = parse_json(.message)
|
||||
if err == null {
|
||||
del(.message)
|
||||
}
|
||||
}
|
||||
sinks:
|
||||
console_out:
|
||||
type: console
|
||||
inputs:
|
||||
- msg_parser
|
||||
encoding:
|
||||
codec: json
|
||||
vlogs_http_1:
|
||||
type: http
|
||||
inputs:
|
||||
- msg_parser
|
||||
uri: http://victorialogs-1:9428/insert/jsonline?_stream_fields=source_type,host,container_name&_msg_field=log.msg&_time_field=timestamp
|
||||
encoding:
|
||||
codec: json
|
||||
framing:
|
||||
method: newline_delimited
|
||||
compression: gzip
|
||||
healthcheck:
|
||||
enabled: false
|
||||
request:
|
||||
headers:
|
||||
AccountID: '0'
|
||||
ProjectID: '0'
|
||||
vlogs_http_2:
|
||||
type: http
|
||||
inputs:
|
||||
- msg_parser
|
||||
uri: http://victorialogs-2:9428/insert/jsonline?_stream_fields=source_type,host,container_name&_msg_field=log.msg&_time_field=timestamp
|
||||
encoding:
|
||||
codec: json
|
||||
framing:
|
||||
method: newline_delimited
|
||||
compression: gzip
|
||||
healthcheck:
|
||||
enabled: false
|
||||
request:
|
||||
headers:
|
||||
AccountID: '0'
|
||||
ProjectID: '0'
|
||||
```
|
||||
|
||||
Please, note that `_stream_fields` parameter must follow recommended [best practices](https://docs.victoriametrics.com/victorialogs/keyconcepts/#stream-fields) to achieve better performance.
|
||||
|
||||
The example of vmauth configuration (`auth.yml`)
|
||||
|
||||
```yaml
|
||||
unauthorized_user:
|
||||
url_prefix:
|
||||
- http://victorialogs-1:9428
|
||||
- http://victorialogs-2:9428
|
||||
```
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
# balance load among victorialogs instances
|
||||
# see https://docs.victoriametrics.com/vmauth/#load-balancing
|
||||
unauthorized_user:
|
||||
url_prefix:
|
||||
- http://victorialogs-1:9428
|
||||
- http://victorialogs-2:9428
|
||||
|
|
@ -0,0 +1,48 @@
|
|||
services:
|
||||
vector:
|
||||
image: docker.io/timberio/vector:0.40.1-distroless-static
|
||||
restart: on-failure
|
||||
volumes:
|
||||
- type: bind
|
||||
source: /var/run/docker.sock
|
||||
target: /var/run/docker.sock
|
||||
- type: bind
|
||||
source: /var/lib/docker
|
||||
target: /var/lib/docker
|
||||
- ./vector.yaml:/etc/vector/vector.yaml:ro
|
||||
user: root
|
||||
ports:
|
||||
- '8686:8686'
|
||||
depends_on: [victorialogs-1,victorialogs-2]
|
||||
|
||||
victorialogs-1:
|
||||
image: docker.io/victoriametrics/victoria-logs:v0.29.0-victorialogs
|
||||
volumes:
|
||||
- victorialogs-vector-docker-vl-ha-single-1:/vlogs
|
||||
command:
|
||||
- -storageDataPath=/vlogs
|
||||
- -loggerFormat=json
|
||||
victorialogs-2:
|
||||
image: docker.io/victoriametrics/victoria-logs:v0.29.0-victorialogs
|
||||
volumes:
|
||||
- victorialogs-vector-docker-vl-ha-single-2:/vlogs
|
||||
command:
|
||||
- -storageDataPath=/vlogs
|
||||
- -loggerFormat=json
|
||||
vmauth:
|
||||
container_name: vmauth
|
||||
image: victoriametrics/vmauth:v1.103.0
|
||||
depends_on:
|
||||
- "victorialogs-1"
|
||||
- "victorialogs-2"
|
||||
volumes:
|
||||
- ./auth.yml:/etc/auth.yml
|
||||
command:
|
||||
- '--auth.config=/etc/auth.yml'
|
||||
ports:
|
||||
- 8427:8427
|
||||
restart: always
|
||||
|
||||
volumes:
|
||||
victorialogs-vector-docker-vl-ha-single-1:
|
||||
victorialogs-vector-docker-vl-ha-single-2:
|
||||
|
|
@ -0,0 +1,58 @@
|
|||
api:
|
||||
enabled: true
|
||||
address: 0.0.0.0:8686
|
||||
sources:
|
||||
docker:
|
||||
type: docker_logs
|
||||
transforms:
|
||||
msg_parser:
|
||||
type: remap
|
||||
inputs:
|
||||
- docker
|
||||
source: |
|
||||
if exists(.message) {
|
||||
.log, err = parse_json(.message)
|
||||
if err == null {
|
||||
del(.message)
|
||||
}
|
||||
}
|
||||
sinks:
|
||||
console_out:
|
||||
type: console
|
||||
inputs:
|
||||
- msg_parser
|
||||
encoding:
|
||||
codec: json
|
||||
vlogs_http_1:
|
||||
type: http
|
||||
inputs:
|
||||
- msg_parser
|
||||
uri: http://victorialogs-1:9428/insert/jsonline?_stream_fields=source_type,host,container_name&_msg_field=log.msg&_time_field=timestamp
|
||||
encoding:
|
||||
codec: json
|
||||
framing:
|
||||
method: newline_delimited
|
||||
compression: gzip
|
||||
healthcheck:
|
||||
enabled: false
|
||||
request:
|
||||
headers:
|
||||
AccountID: '0'
|
||||
ProjectID: '0'
|
||||
vlogs_http_2:
|
||||
type: http
|
||||
inputs:
|
||||
- msg_parser
|
||||
uri: http://victorialogs-2:9428/insert/jsonline?_stream_fields=source_type,host,container_name&_msg_field=log.msg&_time_field=timestamp
|
||||
encoding:
|
||||
codec: json
|
||||
framing:
|
||||
method: newline_delimited
|
||||
compression: gzip
|
||||
healthcheck:
|
||||
enabled: false
|
||||
request:
|
||||
headers:
|
||||
AccountID: '0'
|
||||
ProjectID: '0'
|
||||
|
||||
|
|
@ -135,7 +135,7 @@ VictoriaLogs automatically creates the `-storageDataPath` directory on the first
|
|||
|
||||
This schema outlines how to configure a High Availability (HA) setup using VictoriaLogs Single-Node instances. The setup consists of the following components:
|
||||
|
||||
- **Log Collector**: The log collector should support multiplexing incoming data to multiple outputs (destinations). Popular log collectors like [Logstash](https://www.elastic.co/guide/en/logstash/current/output-plugins.html), [Fluentd](https://docs.fluentd.org/output/copy), and [Vector](https://vector.dev/docs/setup/configuration/sinks/) already offer this capability. Refer to their documentation for configuration details.
|
||||
- **Log Collector**: The log collector should support multiplexing incoming data to multiple outputs (destinations). Popular log collectors like [Fluent Bit](https://docs.fluentbit.io/manual/concepts/data-pipeline/router), [Logstash](https://www.elastic.co/guide/en/logstash/current/output-plugins.html), [Fluentd](https://docs.fluentd.org/output/copy), and [Vector](https://vector.dev/docs/setup/configuration/sinks/) already offer this capability. Refer to their documentation for configuration details.
|
||||
|
||||
- **VictoriaLogs Single-Node Instances**: Use two or more instances to achieve HA.
|
||||
|
||||
|
|
@ -143,6 +143,11 @@ This schema outlines how to configure a High Availability (HA) setup using Victo
|
|||
|
||||
|
||||
|
||||
Here are the working example of HA configuration for VictoriaLogs using Docker Compose:
|
||||
|
||||
- [Fluent Bit + VictoriaLogs Single-Node + vmauth](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/fluentbit-ha-single-node)
|
||||
- [Logstash + VictoriaLogs Single-Node + vmauth](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/logstash-ha-single-node)
|
||||
- [Vector + VictoriaLogs Single-Node + vmauth](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker/victorialogs/vector-ha-single-node)
|
||||
|
||||
## Backup and restore
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue