--- weight: 5 title: How to use OpenTelemetry metrics with VictoriaMetrics menu: docs: parent: "guides" weight: 5 --- VictoriaMetrics supports metrics ingestion with [OpenTelemetry metrics format](https://opentelemetry.io/docs/specs/otel/metrics/). This guide covers data ingestion via [opentelemetry-collector](https://opentelemetry.io/docs/collector/) and direct metrics push from application. ## Pre-Requirements * [kubernetes cluster](https://kubernetes.io/docs/tasks/tools/#kind) * [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) * [helm](https://helm.sh/docs/intro/install/) ### Install VictoriaMetrics single-server via helm chart Install single-server version: ```sh helm repo add vm https://victoriametrics.github.io/helm-charts/ helm repo update helm install victoria-metrics vm/victoria-metrics-single ``` Verify it's up and running: ```sh kubectl get pods # victoria-metrics-victoria-metrics-single-server-0 1/1 Running 0 3m1s ``` Helm chart provides the following urls for reading and writing data: ```text Write url inside the kubernetes cluster: http://victoria-metrics-victoria-metrics-single-server.default.svc.cluster.local:8428 Read Data: The following url can be used as the datasource url in Grafana: http://victoria-metrics-victoria-metrics-single-server.default.svc.cluster.local:8428 ``` ## Using opentelemetry-collector with VictoriaMetrics ### Deploy opentelemetry-collector and configure metrics forwarding ```sh helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts helm repo update # add values cat << EOF > values.yaml presets: clusterMetrics: enabled: true config: exporters: prometheusremotewrite: endpoint: "http://victoria-metrics-victoria-metrics-single-server.default.svc.cluster.local:8428/api/v1/write" service: pipelines: metrics: receivers: [otlp] processors: [] exporters: [prometheusremotewrite] EOF # install helm chart helm upgrade -i otl-collector open-telemetry/opentelemetry-collector --set mode=deployment -f values.yaml # check if pod is healthy kubectl get pod NAME READY STATUS RESTARTS AGE otl-collector-opentelemetry-collector-7467bbb559-2pq2n 1/1 Running 0 23m # forward port to local machine to verify metrics are ingested kubectl port-forward victoria-metrics-victoria-metrics-single-server-0 8428 # check metric `k8s_container_ready` via browser http://localhost:8428/vmui/#/?g0.expr=k8s_container_ready ``` The full version of possible configuration options could be found in [OpenTelemetry docs](https://opentelemetry.io/docs/collector/configuration/). ## Direct metrics push Metrics could be ingested into VictoriaMetrics directly with HTTP requests. You can use any compatible OpenTelemetry instrumentation [clients](https://opentelemetry.io/docs/languages/). In our example, we'll create a WEB server in [Golang](https://go.dev/) and instrument it with metrics. ### Building the Go application instrumented with metrics See the full source code of the example [here](/guides/getting-started-with-opentelemetry-app.go.example). The list of OpenTelemetry dependencies for `go.mod` is the following: ```go go 1.20 require ( go.opentelemetry.io/otel v1.7.0 go.opentelemetry.io/otel/exporters/otlp/otlpmetric v0.30.0 go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v0.30.0 go.opentelemetry.io/otel/metric v0.30.0 go.opentelemetry.io/otel/sdk v1.7.0 go.opentelemetry.io/otel/sdk/metric v0.30.0 ) ``` Let's create a new file `main.go` with basic implementation of the WEB server: ```go package main func main() { mux := http.NewServeMux() mux.HandleFunc("/api/fast", func(writer http.ResponseWriter, request *http.Request) { writer.WriteHeader(http.StatusOK) writer.Write([]byte(`fast ok`)) }) mux.HandleFunc("/api/slow", func(writer http.ResponseWriter, request *http.Request) { time.Sleep(time.Second * 2) writer.WriteHeader(http.StatusOK) writer.Write([]byte(`slow ok`)) }) mw, err := newMetricsMiddleware(mux) if err != nil { panic(fmt.Sprintf("cannot build metricMiddleWare: %q", err)) } go func() { http.ListenAndServe("localhost:8081", mw) }() } ``` In the code above, we used `newMetricsMiddleware` function to create a `handler` for our server. Let's define it below: ```go type metricMiddleWare struct { h http.Handler requestsCount syncint64.Counter requestsLatency syncfloat64.Histogram activeRequests int64 } func newMetricsMiddleware(h http.Handler) (*metricMiddleWare, error) { mw := &metricMiddleWare{h: h} mc, err := newMetricsController(ctx) if err != nil { return nil, fmt.Errorf("cannot build metrics collector: %w", err) } global.SetMeterProvider(mc) prov := mc.Meter("") mw.requestsLatency, err = prov.SyncFloat64().Histogram("http_request_latency_seconds") if err != nil { return nil, fmt.Errorf("cannot create histogram: %w", err) } mw.requestsCount, err = prov.SyncInt64().Counter("http_requests_total") if err != nil { return nil, fmt.Errorf("cannot create syncInt64 counter: %w", err) } ar, err := prov.AsyncInt64().Gauge("http_active_requests") if err != nil { return nil, fmt.Errorf("cannot create AsyncInt64 gauge: %w", err) } if err := prov.RegisterCallback([]instrument.Asynchronous{ar}, func(ctx context.Context) { ar.Observe(ctx, atomic.LoadInt64(&mw.activeRequests)) }); err != nil { return nil, fmt.Errorf("cannot Register int64 gauge: %w", err) } return mw, nil } ``` The new type `metricMiddleWare` is instrumented with 3 [metrics](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#timeseries-model) initialized in `newMetricsMiddleware` method: * counter `http_requests_total` * histogram `http_request_latency_seconds` * gauge `http_active_requests` Let's implement http.Handler interface for `metricMiddleWare` by adding `ServeHTTP` method: ```go func (m *metricMiddleWare) ServeHTTP(w http.ResponseWriter, r *http.Request) { t := time.Now() path := r.URL.Path m.requestsCount.Add(nil, 1, attribute.String("path", path)) atomic.AddInt64(&m.activeRequests, 1) defer func() { atomic.AddInt64(&m.activeRequests, -1) m.requestsLatency.Record(nil, time.Since(t).Seconds(), attribute.String("path", path)) }() m.h.ServeHTTP(w, r) } ``` In method above, our middleware processes received HTTP requests and updates metrics with each new request. But for these metrics to be shipped we need to add a new method `newMetricsController` to organize metrics collection: ```go func newMetricsController(ctx context.Context) (*controller.Controller, error) { options := []otlpmetrichttp.Option{ otlpmetrichttp.WithEndpoint(""), otlpmetrichttp.WithURLPath("/opentelemetry/api/v1/push"), } metricExporter, err := otlpmetrichttp.New(ctx, options...) if err != nil { return nil, fmt.Errorf("cannot create otlphttp exporter: %w", err) } resourceConfig, err := resource.New(ctx, resource.WithAttributes(attribute.String("job", "otlp"), attribute.String("instance", "localhost"))) if err != nil { return nil, fmt.Errorf("cannot create meter resource: %w", err) } meterController := controller.New( processor.NewFactory( selector.NewWithHistogramDistribution( histogram.WithExplicitBoundaries([]float64{0.01, 0.05, 0.1, 0.5, 0.9, 1.0, 5.0, 10.0, 100.0}), ), aggregation.CumulativeTemporalitySelector(), processor.WithMemory(true), ), controller.WithExporter(metricExporter), controller.WithCollectPeriod(time.Second * 10), controller.WithResource(resourceConfig), ) if err := meterController.Start(ctx); err != nil { return nil, fmt.Errorf("cannot start meter controller: %w", err) } return meterController, nil } ``` This controller will collect and push collected metrics to VictoriaMetrics address with interval of `10s`. See the full source code of the example [here](/guides/getting-started-with-opentelemetry-app.go.example). ### Test metrics ingestion In order to push metrics of our WEB server to VictoriaMetrics it is necessary to ensure that VictoriaMetrics ingestion endpoint is available locally. In previous steps we already deployed a single-server VictoriaMetrics, so let's make it available locally: ```sh # port-forward victoriametrics to ingest metrics kubectl port-forward victoria-metrics-victoria-metrics-single-server-0 8428 ``` Now let's run our WEB server and call its APIs: ```sh # build and run the app go run main.go 2024/03/25 19:27:41 Starting web server... 2024/03/25 19:27:41 web server started at localhost:8081. # execute few queries with curl curl http://localhost:8081/api/fast curl http://localhost:8081/api/slow ``` Open [vmui](https://docs.victoriametrics.com/#vmui) and query `http_requests_total` or `http_active_requests` with [metricsql](https://docs.victoriametrics.com/metricsql/). ## Limitations * VictoriaMetrics doesn't support experimental JSON encoding [format](https://github.com/open-telemetry/opentelemetry-proto/blob/main/examples/metrics.json). * VictoriaMetrics supports only `AggregationTemporalityCumulative` type for [histogram](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#histogram) and [summary](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#summary-legacy)