VictoriaMetrics/docs/VictoriaLogs/data-ingestion/opentelemetry.md at 92885f99dd109606c7a5fee0b3815940987c902f

mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00

docs: [VictoriaLogs] OTel Collector elasticsearchexporter header note (#7074 )

### Describe Your Changes

By default, the `elasticsearchexporter` in OTel Collector puts the log
message under a field other than `_msg` (e.g., `Body`). Without
specifying via an HTTP header, those logs may not be queried correctly.
See also:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6785.

This pull request updates the example configuration and notes for the
`elasticsearchexporter`.

### Checklist

The following checks are **mandatory**:

- [X] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit fea4433362)

2024-09-24 16:58:16 +02:00

4 KiB

Raw Blame History

weight

title

disableToc

aliases

OpenTelemetry setup

true

docs

parent	weight
victorialogs-data-ingestion	4

/VictoriaLogs/data-ingestion/OpenTelemetry.html

VictoriaLogs supports both client open-telemetry SDK and collector.

Client SDK

Specify EndpointURL for http-exporter builder.

Consider the following example for golang SDK:

 // Create the OTLP log exporter that sends logs to configured destination
 logExporter, err := otlploghttp.New(ctx,
  otlploghttp.WithEndpointURL("http://victorialogs:9428/insert/opentelemetry/v1/logs"),
 )

Optionally, stream fields could be defined via headers:

 // Create the OTLP log exporter that sends logs to configured destination
 logExporter, err := otlploghttp.New(ctx,
  otlploghttp.WithEndpointURL("http://victorialogs:9428/insert/opentelemetry/v1/logs"),
   otlploghttp.WithHeaders(map[string]string{"VL-Stream-Fields": "telemetry.sdk.language,severity"}),
 )

Given config defines 2 stream fields - severity and telemetry.sdk.language.

Collector configuration

VictoriaLogs supports given below OpenTelemetry collector exporters:

Elasticsearch
Loki
OpenTelemetry

Elasticsearch

exporters:
  elasticsearch:
    endpoints:
      - http://victorialogs:9428/insert/elasticsearch
    headers:
      VL-Msg-Field: "Body" # Optional.
receivers:
  filelog:
    include: [/tmp/logs/*.log]
    resource:
      region: us-east-1
service:
  pipelines:
    logs:
      receivers: [filelog]
      exporters: [elasticsearch]

Please note that every ingested log entry must contain at least a _msg field with the actual log message. By default, the Elasticsearch exporter may place the log message in the Body field. In this case, you can specify the field mapping via:

    headers:
      VL-Msg-Field: "Body"

VictoriaLogs also support specify AccountID, ProjectID, log timestamp and other fields via HTTP headers.

Loki

exporters:
  loki:
    endpoint: http://victorialogs:9428/insert/loki/api/v1/push
receivers:
  filelog:
    include: [/tmp/logs/*.log]
    resource:
      region: us-east-1
service:
  pipelines:
    logs:
      receivers: [filelog]
      exporters: [loki]

OpenTelemetry

Specify logs endpoint for OTLP/HTTP exporter in configuration file for sending the collected logs to VictoriaLogs:

exporters:
  otlphttp:
    logs_endpoint: http://localhost:9428/insert/opentelemetry/v1/logs