github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity

lib/promrelabel: support action: graphite relabeling

Browse source 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2737
This commit is contained in:
Aliaksandr Valialkin 2022-06-16 20:24:19 +03:00
parent 45c1e27937
commit c18f8cccfa
No known key found for this signature in database
GPG key ID: A72BEC6CD3D0DED1
16 changed files with 849 additions and 37 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

9
README.md
View file

@ -478,6 +478,8 @@ The `/api/v1/export` endpoint should return the following response:
{"metric":{"__name__":"foo.bar.baz","tag1":"value1","tag2":"value2"},"values":[123],"timestamps":[1560277406000]}
```
[Graphite relabeling](https://docs.victoriametrics.com/vmagent.html#graphite-relabeling) can be used if the imported Graphite data is going to be queried via [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html).
## Querying Graphite data
Data sent to VictoriaMetrics via `Graphite plaintext protocol` may be read via the following APIs:
@ -492,6 +494,9 @@ VictoriaMetrics supports `__graphite__` pseudo-label for selecting time series w
The `__graphite__` pseudo-label supports e.g. alternate regexp filters such as `(value1|...|valueN)`. They are transparently converted to `{value1,...,valueN}` syntax [used in Graphite](https://graphite.readthedocs.io/en/latest/render_api.html#paths-and-wildcards). This allows using [multi-value template variables in Grafana](https://grafana.com/docs/grafana/latest/variables/formatting-multi-value-variables/) inside `__graphite__` pseudo-label. For example, Grafana expands `{__graphite__=~"foo.($bar).baz"}` into `{__graphite__=~"foo.(x|y).baz"}` if `$bar` template variable contains `x` and `y` values. In this case the query is automatically converted into `{__graphite__=~"foo.{x,y}.baz"}` before execution.
VictoriaMetrics also supports Graphite query language - see [these docs](#graphite-render-api-usage).
## How to send data from OpenTSDB-compatible agents
VictoriaMetrics supports [telnet put protocol](http://opentsdb.net/docs/build/html/api_telnet/put.html)
@ -1132,7 +1137,9 @@ Example contents for `-relabelConfig` file:
regex: true
```
See [these docs](https://docs.victoriametrics.com/vmagent.html#relabeling) for more details about relabeling in VictoriaMetrics.
VictoriaMetrics components provide additional relabeling features such as Graphite-style relabeling.
See [these docs](https://docs.victoriametrics.com/vmagent.html#relabeling) for more details.
## Federation

45
app/vmagent/README.md
View file

@ -252,12 +252,13 @@ Labels can be added to metrics by the following mechanisms:
VictoriaMetrics components (including `vmagent`) support Prometheus-compatible relabeling.
They provide the following additional actions on top of actions from the [Prometheus relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config):
* `replace_all`: replaces all of the occurences of `regex` in the values of `source_labels` with the `replacement` and stores the results in the `target_label`.
* `labelmap_all`: replaces all of the occurences of `regex` in all the label names with the `replacement`.
* `keep_if_equal`: keeps the entry if all the label values from `source_labels` are equal.
* `drop_if_equal`: drops the entry if all the label values from `source_labels` are equal.
* `keep_metrics`: keeps all the metrics with names matching the given `regex`.
* `drop_metrics`: drops all the metrics with names matching the given `regex`.
* `replace_all`: replaces all of the occurences of `regex` in the values of `source_labels` with the `replacement` and stores the results in the `target_label`
* `labelmap_all`: replaces all of the occurences of `regex` in all the label names with the `replacement`
* `keep_if_equal`: keeps the entry if all the label values from `source_labels` are equal
* `drop_if_equal`: drops the entry if all the label values from `source_labels` are equal
* `keep_metrics`: keeps all the metrics with names matching the given `regex`
* `drop_metrics`: drops all the metrics with names matching the given `regex`
* `graphite`: applies Graphite-style relabeling to metric name. See [these docs](#graphite-relabeling)
The `regex` value can be split into multiple lines for improved readability and maintainability. These lines are automatically joined with `|` char when parsed. For example, the following configs are equivalent:
@ -305,6 +306,38 @@ You can read more about relabeling in the following articles:
* [Extracting labels from legacy metric names](https://www.robustperception.io/extracting-labels-from-legacy-metric-names)
* [relabel_configs vs metric_relabel_configs](https://www.robustperception.io/relabel_configs-vs-metric_relabel_configs)
## Graphite relabeling
VictoriaMetrics components support `action: graphite` relabeling rules, which allow extracting various parts from Graphite-style metrics
into the configued labels with the syntax similar to [Glob matching in statsd_exporter](https://github.com/prometheus/statsd_exporter#glob-matching).
Note that the `name` field must be substituted with explicit `__name__` option under `labels` section.
If `__name__` option is missing under `labels` section, then the original Graphite-style metric name is left unchanged.
For example, the following relabeling rule generates `requests_total{job="app42",instance="host124:8080"}` metric
from "app42.host123.requests.total" Graphite-style metric:
```yaml
- action: graphite
match: "*.*.*.total"
labels:
__name__: "${3}_total"
job: "$1"
instance: "${2}:8080"
```
Important notes about `action: graphite` relabeling rules:
- The relabeling rule is applied only to metrics, which match the given `match` expression. Other metrics remain unchanged.
- The `*` matches the maximum possible number of chars until the next dot or until the next part of the `match` expression whichever comes first.
It may match zero chars if the next char is `.`.
For example, `match: "app*foo.bar"` matches `app42foo.bar` and `42` becomes available to use at `labels` section via `$1` capture group.
- The `$0` capture group matches the original metric name.
- The relabeling rules are executed in order defined in the original config.
The `action: graphite` relabeling rules are easier to write and maintain than `action: replace` for labels extraction from Graphite-style metric names.
Additionally, the `action: graphite` relabeling rules usually work much faster than the equivalent `action: replace` rules.
## Prometheus staleness markers
`vmagent` sends [Prometheus staleness markers](https://www.robustperception.io/staleness-and-promql) to `-remoteWrite.url` in the following cases:

1
docs/CHANGELOG.md
View file

@ -34,6 +34,7 @@ The following tip changes can be tested by building VictoriaMetrics components f
* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): expose `/api/v1/status/config` endpoint in the same way as Prometheus does. See [these docs](https://prometheus.io/docs/prometheus/latest/querying/api/#config).
* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add `-promscrape.suppressScrapeErrorsDelay` command-line flag, which can be used for delaying and aggregating the logging of per-target scrape errors. This may reduce the amounts of logs when `vmagent` scrapes many unreliable targets. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2575). Thanks to @jelmd for [the initial implementation](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2576).
* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add `-promscrape.cluster.name` command-line flag, which allows proper data de-duplication when the same target is scraped from multiple [vmagent clusters](https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets). See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2679).
* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add `action: graphite` relabeling rules optimized for extracting labels from Graphite-style metric names. See [these docs](https://docs.victoriametrics.com/vmagent.html#graphite-relabeling) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2737).
* FEATURE: [VictoriaMetrics enterprise](https://victoriametrics.com/products/enterprise/): expose `vm_downsampling_partitions_scheduled` and `vm_downsampling_partitions_scheduled_size_bytes` metrics, which can be used for tracking the progress of initial [downsampling](https://docs.victoriametrics.com/#downsampling) for historical data. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2612).
* BUGFIX: support for data ingestion in [DataDog format](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-send-data-from-datadog-agent) from legacy clients / agents. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2670). Thanks to @elProxy for the fix.

13
docs/README.md
View file

@ -268,7 +268,8 @@ See the [example VMUI at VictoriaMetrics playground](https://play.victoriametric
VictoriaMetrics provides an ability to explore time series cardinality at `cardinality` tab in [vmui](#vmui) in the following ways:
- To identify metric names with the highest number of series.
- To idnetify labels with the highest number of series.
- To identify labels with the highest number of series.
- To identify values with the highest number of series for the selected label (aka `focusLabel`).
- To identify label=name pairs with the highest number of series.
- To identify labels with the highest number of unique values.
@ -477,6 +478,8 @@ The `/api/v1/export` endpoint should return the following response:
{"metric":{"__name__":"foo.bar.baz","tag1":"value1","tag2":"value2"},"values":[123],"timestamps":[1560277406000]}
```
[Graphite relabeling](https://docs.victoriametrics.com/vmagent.html#graphite-relabeling) can be used if the imported Graphite data is going to be queried via [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html).
## Querying Graphite data
Data sent to VictoriaMetrics via `Graphite plaintext protocol` may be read via the following APIs:
@ -491,6 +494,9 @@ VictoriaMetrics supports `__graphite__` pseudo-label for selecting time series w
The `__graphite__` pseudo-label supports e.g. alternate regexp filters such as `(value1|...|valueN)`. They are transparently converted to `{value1,...,valueN}` syntax [used in Graphite](https://graphite.readthedocs.io/en/latest/render_api.html#paths-and-wildcards). This allows using [multi-value template variables in Grafana](https://grafana.com/docs/grafana/latest/variables/formatting-multi-value-variables/) inside `__graphite__` pseudo-label. For example, Grafana expands `{__graphite__=~"foo.($bar).baz"}` into `{__graphite__=~"foo.(x|y).baz"}` if `$bar` template variable contains `x` and `y` values. In this case the query is automatically converted into `{__graphite__=~"foo.{x,y}.baz"}` before execution.
VictoriaMetrics also supports Graphite query language - see [these docs](#graphite-render-api-usage).
## How to send data from OpenTSDB-compatible agents
VictoriaMetrics supports [telnet put protocol](http://opentsdb.net/docs/build/html/api_telnet/put.html)
@ -1131,7 +1137,9 @@ Example contents for `-relabelConfig` file:
regex: true
```
See [these docs](https://docs.victoriametrics.com/vmagent.html#relabeling) for more details about relabeling in VictoriaMetrics.
VictoriaMetrics components provide additional relabeling features such as Graphite-style relabeling.
See [these docs](https://docs.victoriametrics.com/vmagent.html#relabeling) for more details.
## Federation
@ -1441,6 +1449,7 @@ VictoriaMetrics returns TSDB stats at `/api/v1/status/tsdb` page in the way simi
* `topN=N` where `N` is the number of top entries to return in the response. By default top 10 entries are returned.
* `date=YYYY-MM-DD` where `YYYY-MM-DD` is the date for collecting the stats. By default the stats is collected for the current day. Pass `date=1970-01-01` in order to collect global stats across all the days.
* `focusLabel=LABEL_NAME` returns label values with the highest number of time series for the given `LABEL_NAME` in the `seriesCountByFocusLabelValue` list.
* `match[]=SELECTOR` where `SELECTOR` is an arbitrary [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) for series to take into account during stats calculation. By default all the series are taken into account.
* `extra_label=LABEL=VALUE`. See [these docs](#prometheus-querying-api-enhancements) for more details.

9
docs/Single-server-VictoriaMetrics.md
View file

@ -482,6 +482,8 @@ The `/api/v1/export` endpoint should return the following response:
{"metric":{"__name__":"foo.bar.baz","tag1":"value1","tag2":"value2"},"values":[123],"timestamps":[1560277406000]}
```
[Graphite relabeling](https://docs.victoriametrics.com/vmagent.html#graphite-relabeling) can be used if the imported Graphite data is going to be queried via [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html).
## Querying Graphite data
Data sent to VictoriaMetrics via `Graphite plaintext protocol` may be read via the following APIs:
@ -496,6 +498,9 @@ VictoriaMetrics supports `__graphite__` pseudo-label for selecting time series w
The `__graphite__` pseudo-label supports e.g. alternate regexp filters such as `(value1|...|valueN)`. They are transparently converted to `{value1,...,valueN}` syntax [used in Graphite](https://graphite.readthedocs.io/en/latest/render_api.html#paths-and-wildcards). This allows using [multi-value template variables in Grafana](https://grafana.com/docs/grafana/latest/variables/formatting-multi-value-variables/) inside `__graphite__` pseudo-label. For example, Grafana expands `{__graphite__=~"foo.($bar).baz"}` into `{__graphite__=~"foo.(x|y).baz"}` if `$bar` template variable contains `x` and `y` values. In this case the query is automatically converted into `{__graphite__=~"foo.{x,y}.baz"}` before execution.
VictoriaMetrics also supports Graphite query language - see [these docs](#graphite-render-api-usage).
## How to send data from OpenTSDB-compatible agents
VictoriaMetrics supports [telnet put protocol](http://opentsdb.net/docs/build/html/api_telnet/put.html)
@ -1136,7 +1141,9 @@ Example contents for `-relabelConfig` file:
regex: true
```
See [these docs](https://docs.victoriametrics.com/vmagent.html#relabeling) for more details about relabeling in VictoriaMetrics.
VictoriaMetrics components provide additional relabeling features such as Graphite-style relabeling.
See [these docs](https://docs.victoriametrics.com/vmagent.html#relabeling) for more details.
## Federation

45
docs/vmagent.md
View file

@ -256,12 +256,13 @@ Labels can be added to metrics by the following mechanisms:
VictoriaMetrics components (including `vmagent`) support Prometheus-compatible relabeling.
They provide the following additional actions on top of actions from the [Prometheus relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config):
* `replace_all`: replaces all of the occurences of `regex` in the values of `source_labels` with the `replacement` and stores the results in the `target_label`.
* `labelmap_all`: replaces all of the occurences of `regex` in all the label names with the `replacement`.
* `keep_if_equal`: keeps the entry if all the label values from `source_labels` are equal.
* `drop_if_equal`: drops the entry if all the label values from `source_labels` are equal.
* `keep_metrics`: keeps all the metrics with names matching the given `regex`.
* `drop_metrics`: drops all the metrics with names matching the given `regex`.
* `replace_all`: replaces all of the occurences of `regex` in the values of `source_labels` with the `replacement` and stores the results in the `target_label`
* `labelmap_all`: replaces all of the occurences of `regex` in all the label names with the `replacement`
* `keep_if_equal`: keeps the entry if all the label values from `source_labels` are equal
* `drop_if_equal`: drops the entry if all the label values from `source_labels` are equal
* `keep_metrics`: keeps all the metrics with names matching the given `regex`
* `drop_metrics`: drops all the metrics with names matching the given `regex`
* `graphite`: applies Graphite-style relabeling to metric name. See [these docs](#graphite-relabeling)
The `regex` value can be split into multiple lines for improved readability and maintainability. These lines are automatically joined with `|` char when parsed. For example, the following configs are equivalent:
@ -309,6 +310,38 @@ You can read more about relabeling in the following articles:
* [Extracting labels from legacy metric names](https://www.robustperception.io/extracting-labels-from-legacy-metric-names)
* [relabel_configs vs metric_relabel_configs](https://www.robustperception.io/relabel_configs-vs-metric_relabel_configs)
## Graphite relabeling
VictoriaMetrics components support `action: graphite` relabeling rules, which allow extracting various parts from Graphite-style metrics
into the configued labels with the syntax similar to [Glob matching in statsd_exporter](https://github.com/prometheus/statsd_exporter#glob-matching).
Note that the `name` field must be substituted with explicit `__name__` option under `labels` section.
If `__name__` option is missing under `labels` section, then the original Graphite-style metric name is left unchanged.
For example, the following relabeling rule generates `requests_total{job="app42",instance="host124:8080"}` metric
from "app42.host123.requests.total" Graphite-style metric:
```yaml
- action: graphite
match: "*.*.*.total"
labels:
__name__: "${3}_total"
job: "$1"
instance: "${2}:8080"
```
Important notes about `action: graphite` relabeling rules:
- The relabeling rule is applied only to metrics, which match the given `match` expression. Other metrics remain unchanged.
- The `*` matches the maximum possible number of chars until the next dot or until the next part of the `match` expression whichever comes first.
It may match zero chars if the next char is `.`.
For example, `match: "app*foo.bar"` matches `app42foo.bar` and `42` becomes available to use at `labels` section via `$1` capture group.
- The `$0` capture group matches the original metric name.
- The relabeling rules are executed in order defined in the original config.
The `action: graphite` relabeling rules are easier to write and maintain than `action: replace` for labels extraction from Graphite-style metric names.
Additionally, the `action: graphite` relabeling rules usually work much faster than the equivalent `action: replace` rules.
## Prometheus staleness markers
`vmagent` sends [Prometheus staleness markers](https://www.robustperception.io/staleness-and-promql) to `-remoteWrite.url` in the following cases:

62
lib/promrelabel/config.go
View file

@ -23,6 +23,22 @@ type RelabelConfig struct {
Replacement *string `yaml:"replacement,omitempty"`
Action string `yaml:"action,omitempty"`
If *IfExpression `yaml:"if,omitempty"`
// Match is used together with Labels for `action: graphite`. For example:
// - action: graphite
// match: 'foo.*.*.bar'
// labels:
// job: '$1'
// instance: '${2}:8080'
Match string `yaml:"match,omitempty"`
// Labels is used together with Match for `action: graphite`. For example:
// - action: graphite
// match: 'foo.*.*.bar'
// labels:
// job: '$1'
// instance: '${2}:8080'
Labels map[string]string `yaml:"labels,omitempty"`
}
// MultiLineRegex contains a regex, which can be split into multiple lines.
@ -114,12 +130,12 @@ func (pcs *ParsedConfigs) String() string {
if pcs == nil {
return ""
}
var sb strings.Builder
var a []string
for _, prc := range pcs.prcs {
fmt.Fprintf(&sb, "%s,", prc.String())
s := "[" + prc.String() + "]"
a = append(a, s)
}
fmt.Fprintf(&sb, "relabelDebug=%v", pcs.relabelDebug)
return sb.String()
return fmt.Sprintf("%s, relabelDebug=%v", strings.Join(a, ","), pcs.relabelDebug)
}
// LoadRelabelConfigs loads relabel configs from the given path.
@ -200,11 +216,38 @@ func parseRelabelConfig(rc *RelabelConfig) (*parsedRelabelConfig, error) {
if rc.Replacement != nil {
replacement = *rc.Replacement
}
var graphiteMatchTemplate *graphiteMatchTemplate
if rc.Match != "" {
graphiteMatchTemplate = newGraphiteMatchTemplate(rc.Match)
}
var graphiteLabelRules []graphiteLabelRule
if rc.Labels != nil {
graphiteLabelRules = newGraphiteLabelRules(rc.Labels)
}
action := rc.Action
if action == "" {
action = "replace"
}
switch action {
case "graphite":
if graphiteMatchTemplate == nil {
return nil, fmt.Errorf("missing `match` for `action=graphite`; see https://docs.victoriametrics.com/vmagent.html#graphite-relabeling")
}
if len(graphiteLabelRules) == 0 {
return nil, fmt.Errorf("missing `labels` for `action=graphite`; see https://docs.victoriametrics.com/vmagent.html#graphite-relabeling")
}
if len(rc.SourceLabels) > 0 {
return nil, fmt.Errorf("`source_labels` cannot be used with `action=graphite`; see https://docs.victoriametrics.com/vmagent.html#graphite-relabeling")
}
if rc.TargetLabel != "" {
return nil, fmt.Errorf("`target_label` cannot be used with `action=graphite`; see https://docs.victoriametrics.com/vmagent.html#graphite-relabeling")
}
if rc.Replacement != nil {
return nil, fmt.Errorf("`replacement` cannot be used with `action=graphite`; see https://docs.victoriametrics.com/vmagent.html#graphite-relabeling")
}
if rc.Regex != nil {
return nil, fmt.Errorf("`regex` cannot be used with `action=graphite`; see https://docs.victoriametrics.com/vmagent.html#graphite-relabeling")
}
case "replace":
if targetLabel == "" {
return nil, fmt.Errorf("missing `target_label` for `action=replace`")
@ -274,6 +317,14 @@ func parseRelabelConfig(rc *RelabelConfig) (*parsedRelabelConfig, error) {
default:
return nil, fmt.Errorf("unknown `action` %q", action)
}
if action != "graphite" {
if graphiteMatchTemplate != nil {
return nil, fmt.Errorf("`match` config cannot be applied to `action=%s`; it is applied only to `action=graphite`", action)
}
if len(graphiteLabelRules) > 0 {
return nil, fmt.Errorf("`labels` config cannot be applied to `action=%s`; it is applied only to `action=graphite`", action)
}
}
return &parsedRelabelConfig{
SourceLabels: sourceLabels,
Separator: separator,
@ -284,6 +335,9 @@ func parseRelabelConfig(rc *RelabelConfig) (*parsedRelabelConfig, error) {
Action: action,
If: rc.If,
graphiteMatchTemplate: graphiteMatchTemplate,
graphiteLabelRules: graphiteLabelRules,
regexOriginal: regexOriginalCompiled,
hasCaptureGroupInTargetLabel: strings.Contains(targetLabel, "$"),
hasCaptureGroupInReplacement: strings.Contains(replacement, "$"),

163
lib/promrelabel/config_test.go
View file

@ -45,6 +45,13 @@ func TestRelabelConfigMarshalUnmarshal(t *testing.T) {
- null
- nan
`, "- regex:\n - \"-1.23\"\n - \"false\"\n - \"null\"\n - nan\n")
f(`
- action: graphite
match: 'foo.*.*.aaa'
labels:
instance: '$1-abc'
job: '${2}'
`, "- action: graphite\n match: foo.*.*.aaa\n labels:\n instance: $1-abc\n job: ${2}\n")
}
func TestLoadRelabelConfigsSuccess(t *testing.T) {
@ -53,8 +60,9 @@ func TestLoadRelabelConfigsSuccess(t *testing.T) {
if err != nil {
t.Fatalf("cannot load relabel configs from %q: %s", path, err)
}
if n := pcs.Len(); n != 14 {
t.Fatalf("unexpected number of relabel configs loaded from %q; got %d; want %d", path, n, 14)
nExpected := 16
if n := pcs.Len(); n != nExpected {
t.Fatalf("unexpected number of relabel configs loaded from %q; got %d; want %d", path, n, nExpected)
}
}
@ -77,6 +85,51 @@ func TestLoadRelabelConfigsFailure(t *testing.T) {
})
}
func TestParsedConfigsString(t *testing.T) {
f := func(rcs []RelabelConfig, sExpected string) {
t.Helper()
pcs, err := ParseRelabelConfigs(rcs, false)
if err != nil {
t.Fatalf("unexpected error: %s", err)
}
s := pcs.String()
if s != sExpected {
t.Fatalf("unexpected string representation for ParsedConfigs;\ngot\n%s\nwant\n%s", s, sExpected)
}
}
f([]RelabelConfig{
{
TargetLabel: "foo",
SourceLabels: []string{"aaa"},
},
}, "[SourceLabels=[aaa], Separator=;, TargetLabel=foo, Regex=^(.*)$, Modulus=0, Replacement=$1, Action=replace, If=, "+
"graphiteMatchTemplate=<nil>, graphiteLabelRules=[]], relabelDebug=false")
var ie IfExpression
if err := ie.Parse("{foo=~'bar'}"); err != nil {
t.Fatalf("unexpected error when parsing if expression: %s", err)
}
f([]RelabelConfig{
{
Action: "graphite",
Match: "foo.*.bar",
Labels: map[string]string{
"job": "$1-zz",
},
If: &ie,
},
}, "[SourceLabels=[], Separator=;, TargetLabel=, Regex=^(.*)$, Modulus=0, Replacement=$1, Action=graphite, If={foo=~'bar'}, "+
"graphiteMatchTemplate=foo.*.bar, graphiteLabelRules=[replaceTemplate=$1-zz, targetLabel=job]], relabelDebug=false")
f([]RelabelConfig{
{
Action: "replace",
SourceLabels: []string{"foo", "bar"},
TargetLabel: "x",
If: &ie,
},
}, "[SourceLabels=[foo bar], Separator=;, TargetLabel=x, Regex=^(.*)$, Modulus=0, Replacement=$1, Action=replace, If={foo=~'bar'}, "+
"graphiteMatchTemplate=<nil>, graphiteLabelRules=[]], relabelDebug=false")
}
func TestParseRelabelConfigsSuccess(t *testing.T) {
f := func(rcs []RelabelConfig, pcsExpected *ParsedConfigs) {
t.Helper()
@ -271,4 +324,110 @@ func TestParseRelabelConfigsFailure(t *testing.T) {
},
})
})
t.Run("uppercase-missing-sourceLabels", func(t *testing.T) {
f([]RelabelConfig{
{
Action: "uppercase",
TargetLabel: "foobar",
},
})
})
t.Run("lowercase-missing-targetLabel", func(t *testing.T) {
f([]RelabelConfig{
{
Action: "lowercase",
SourceLabels: []string{"foobar"},
},
})
})
t.Run("graphite-missing-match", func(t *testing.T) {
f([]RelabelConfig{
{
Action: "graphite",
Labels: map[string]string{
"foo": "bar",
},
},
})
})
t.Run("graphite-missing-labels", func(t *testing.T) {
f([]RelabelConfig{
{
Action: "graphite",
Match: "foo.*.bar",
},
})
})
t.Run("graphite-superflouous-sourceLabels", func(t *testing.T) {
f([]RelabelConfig{
{
Action: "graphite",
Match: "foo.*.bar",
Labels: map[string]string{
"foo": "bar",
},
SourceLabels: []string{"foo"},
},
})
})
t.Run("graphite-superflouous-targetLabel", func(t *testing.T) {
f([]RelabelConfig{
{
Action: "graphite",
Match: "foo.*.bar",
Labels: map[string]string{
"foo": "bar",
},
TargetLabel: "foo",
},
})
})
replacement := "foo"
t.Run("graphite-superflouous-replacement", func(t *testing.T) {
f([]RelabelConfig{
{
Action: "graphite",
Match: "foo.*.bar",
Labels: map[string]string{
"foo": "bar",
},
Replacement: &replacement,
},
})
})
var re MultiLineRegex
t.Run("graphite-superflouous-regex", func(t *testing.T) {
f([]RelabelConfig{
{
Action: "graphite",
Match: "foo.*.bar",
Labels: map[string]string{
"foo": "bar",
},
Regex: &re,
},
})
})
t.Run("non-graphite-superflouos-match", func(t *testing.T) {
f([]RelabelConfig{
{
Action: "uppercase",
SourceLabels: []string{"foo"},
TargetLabel: "foo",
Match: "aaa",
},
})
})
t.Run("non-graphite-superflouos-labels", func(t *testing.T) {
f([]RelabelConfig{
{
Action: "uppercase",
SourceLabels: []string{"foo"},
TargetLabel: "foo",
Labels: map[string]string{
"foo": "Bar",
},
},
})
})
}

212
lib/promrelabel/graphite.go Normal file
View file

@ -0,0 +1,212 @@
package promrelabel
import (
"fmt"
"strconv"
"strings"
"sync"
)
var graphiteMatchesPool = &sync.Pool{
New: func() interface{} {
return &graphiteMatches{}
},
}
type graphiteMatches struct {
a []string
}
type graphiteMatchTemplate struct {
sOrig string
parts []string
}
func (gmt *graphiteMatchTemplate) String() string {
return gmt.sOrig
}
type graphiteLabelRule struct {
grt *graphiteReplaceTemplate
targetLabel string
}
func (glr graphiteLabelRule) String() string {
return fmt.Sprintf("replaceTemplate=%s, targetLabel=%s", glr.grt, glr.targetLabel)
}
func newGraphiteLabelRules(m map[string]string) []graphiteLabelRule {
a := make([]graphiteLabelRule, 0, len(m))
for labelName, replaceTemplate := range m {
a = append(a, graphiteLabelRule{
grt: newGraphiteReplaceTemplate(replaceTemplate),
targetLabel: labelName,
})
}
return a
}
func newGraphiteMatchTemplate(s string) *graphiteMatchTemplate {
sOrig := s
var parts []string
for {
n := strings.IndexByte(s, '*')
if n < 0 {
parts = appendGraphiteMatchTemplateParts(parts, s)
break
}
parts = appendGraphiteMatchTemplateParts(parts, s[:n])
parts = appendGraphiteMatchTemplateParts(parts, "*")
s = s[n+1:]
}
return &graphiteMatchTemplate{
sOrig: sOrig,
parts: parts,
}
}
func appendGraphiteMatchTemplateParts(dst []string, s string) []string {
if len(s) == 0 {
// Skip empty part
return dst
}
return append(dst, s)
}
// Match matches s against gmt.
//
// On success it adds matched captures to dst and returns it with true.
// Of failre it returns false.
func (gmt *graphiteMatchTemplate) Match(dst []string, s string) ([]string, bool) {
dst = append(dst, s)
parts := gmt.parts
if len(parts) > 0 {
if p := parts[len(parts)-1]; p != "*" && !strings.HasSuffix(s, p) {
// fast path - suffix mismatch
return dst, false
}
}
for i := 0; i < len(parts); i++ {
p := parts[i]
if p != "*" {
if !strings.HasPrefix(s, p) {
// Cannot match the current part
return dst, false
}
s = s[len(p):]
continue
}
// Search for the matching substring for '*' part.
if i+1 >= len(parts) {
// Matching the last part.
if strings.IndexByte(s, '.') >= 0 {
// The '*' cannot match string with dots.
return dst, false
}
dst = append(dst, s)
return dst, true
}
// Search for the the start of the next part.
p = parts[i+1]
i++
n := strings.Index(s, p)
if n < 0 {
// Cannot match the next part
return dst, false
}
tmp := s[:n]
if strings.IndexByte(tmp, '.') >= 0 {
// The '*' cannot match string with dots.
return dst, false
}
dst = append(dst, tmp)
s = s[n+len(p):]
}
return dst, len(s) == 0
}
type graphiteReplaceTemplate struct {
sOrig string
parts []graphiteReplaceTemplatePart
}
func (grt *graphiteReplaceTemplate) String() string {
return grt.sOrig
}
type graphiteReplaceTemplatePart struct {
n int
s string
}
func newGraphiteReplaceTemplate(s string) *graphiteReplaceTemplate {
sOrig := s
var parts []graphiteReplaceTemplatePart
for {
n := strings.IndexByte(s, '$')
if n < 0 {
parts = appendGraphiteReplaceTemplateParts(parts, s, -1)
break
}
if n > 0 {
parts = appendGraphiteReplaceTemplateParts(parts, s[:n], -1)
}
s = s[n+1:]
if len(s) > 0 && s[0] == '{' {
// The index in the form ${123}
n = strings.IndexByte(s, '}')
if n < 0 {
parts = appendGraphiteReplaceTemplateParts(parts, "$"+s, -1)
break
}
idxStr := s[1:n]
s = s[n+1:]
idx, err := strconv.Atoi(idxStr)
if err != nil {
parts = appendGraphiteReplaceTemplateParts(parts, "${"+idxStr+"}", -1)
} else {
parts = appendGraphiteReplaceTemplateParts(parts, "${"+idxStr+"}", idx)
}
} else {
// The index in the form $123
n := 0
for n < len(s) && s[n] >= '0' && s[n] <= '9' {
n++
}
idxStr := s[:n]
s = s[n:]
idx, err := strconv.Atoi(idxStr)
if err != nil {
parts = appendGraphiteReplaceTemplateParts(parts, "$"+idxStr, -1)
} else {
parts = appendGraphiteReplaceTemplateParts(parts, "$"+idxStr, idx)
}
}
}
return &graphiteReplaceTemplate{
sOrig: sOrig,
parts: parts,
}
}
// Expand expands grt with the given matches into dst and returns it.
func (grt *graphiteReplaceTemplate) Expand(dst []byte, matches []string) []byte {
for _, part := range grt.parts {
if n := part.n; n >= 0 && n < len(matches) {
dst = append(dst, matches[n]...)
} else {
dst = append(dst, part.s...)
}
}
return dst
}
func appendGraphiteReplaceTemplateParts(dst []graphiteReplaceTemplatePart, s string, n int) []graphiteReplaceTemplatePart {
if len(s) > 0 {
dst = append(dst, graphiteReplaceTemplatePart{
s: s,
n: n,
})
}
return dst
}

93
lib/promrelabel/graphite_test.go Normal file
View file

@ -0,0 +1,93 @@
package promrelabel
import (
"reflect"
"testing"
)
func TestGraphiteTemplateMatchExpand(t *testing.T) {
f := func(matchTpl, s, replaceTpl, resultExpected string) {
t.Helper()
gmt := newGraphiteMatchTemplate(matchTpl)
matches, ok := gmt.Match(nil, s)
if !ok {
matches = nil
}
grt := newGraphiteReplaceTemplate(replaceTpl)
result := grt.Expand(nil, matches)
if string(result) != resultExpected {
t.Fatalf("unexpected result; got %q; want %q", result, resultExpected)
}
}
f("", "", "", "")
f("test.*.*.counter", "test.foo.bar.counter", "${2}_total", "bar_total")
f("test.*.*.counter", "test.foo.bar.counter", "$1_total", "foo_total")
f("test.*.*.counter", "test.foo.bar.counter", "total_$0", "total_test.foo.bar.counter")
f("test.dispatcher.*.*.*", "test.dispatcher.foo.bar.baz", "$3-$2-$1", "baz-bar-foo")
f("*.signup.*.*", "foo.signup.bar.baz", "$1-${3}_$2_total", "foo-baz_bar_total")
}
func TestGraphiteMatchTemplateMatch(t *testing.T) {
f := func(tpl, s string, matchesExpected []string, okExpected bool) {
t.Helper()
gmt := newGraphiteMatchTemplate(tpl)
tplGot := gmt.String()
if tplGot != tpl {
t.Fatalf("unexpected template; got %q; want %q", tplGot, tpl)
}
matches, ok := gmt.Match(nil, s)
if ok != okExpected {
t.Fatalf("unexpected ok result for tpl=%q, s=%q; got %v; want %v", tpl, s, ok, okExpected)
}
if okExpected {
if !reflect.DeepEqual(matches, matchesExpected) {
t.Fatalf("unexpected matches for tpl=%q, s=%q; got\n%q\nwant\n%q\ngraphiteMatchTemplate=%v", tpl, s, matches, matchesExpected, gmt)
}
}
}
f("", "", []string{""}, true)
f("", "foobar", nil, false)
f("foo", "foo", []string{"foo"}, true)
f("foo", "", nil, false)
f("foo.bar.baz", "foo.bar.baz", []string{"foo.bar.baz"}, true)
f("*", "foobar", []string{"foobar", "foobar"}, true)
f("**", "foobar", nil, false)
f("*", "foo.bar", nil, false)
f("*foo", "barfoo", []string{"barfoo", "bar"}, true)
f("*foo", "foo", []string{"foo", ""}, true)
f("*foo", "bar.foo", nil, false)
f("foo*", "foobar", []string{"foobar", "bar"}, true)
f("foo*", "foo", []string{"foo", ""}, true)
f("foo*", "foo.bar", nil, false)
f("foo.*", "foobar", nil, false)
f("foo.*", "foo.bar", []string{"foo.bar", "bar"}, true)
f("foo.*", "foo.bar.baz", nil, false)
f("*.*.baz", "foo.bar.baz", []string{"foo.bar.baz", "foo", "bar"}, true)
f("*.bar", "foo.bar.baz", nil, false)
f("*.bar", "foo.baz", nil, false)
}
func TestGraphiteReplaceTemplateExpand(t *testing.T) {
f := func(tpl string, matches []string, resultExpected string) {
t.Helper()
grt := newGraphiteReplaceTemplate(tpl)
tplGot := grt.String()
if tplGot != tpl {
t.Fatalf("unexpected template; got %q; want %q", tplGot, tpl)
}
result := grt.Expand(nil, matches)
if string(result) != resultExpected {
t.Fatalf("unexpected result for tpl=%q; got\n%q\nwant\n%q\ngraphiteReplaceTemplate=%v", tpl, result, resultExpected, grt)
}
}
f("", nil, "")
f("foo", nil, "foo")
f("$", nil, "$")
f("$1", nil, "$1")
f("${123", nil, "${123")
f("${123}", nil, "${123}")
f("${foo}45$sdf$3", nil, "${foo}45$sdf$3")
f("$1", []string{"foo", "bar"}, "bar")
f("$0-$1", []string{"foo", "bar"}, "foo-bar")
f("x-${0}-$1", []string{"foo", "bar"}, "x-foo-bar")
}

93
lib/promrelabel/graphite_timing_test.go Normal file
View file

@ -0,0 +1,93 @@
package promrelabel
import (
"fmt"
"testing"
)
func BenchmarkGraphiteMatchTemplateMatch(b *testing.B) {
b.Run("match-short", func(b *testing.B) {
tpl := "*.bar.baz"
s := "foo.bar.baz"
benchmarkGraphiteMatchTemplateMatch(b, tpl, s, true)
})
b.Run("mismtach-short", func(b *testing.B) {
tpl := "*.bar.baz"
s := "foo.aaa"
benchmarkGraphiteMatchTemplateMatch(b, tpl, s, false)
})
b.Run("match-long", func(b *testing.B) {
tpl := "*.*.*.bar.*.baz"
s := "foo.bar.baz.bar.aa.baz"
benchmarkGraphiteMatchTemplateMatch(b, tpl, s, true)
})
b.Run("mismatch-long", func(b *testing.B) {
tpl := "*.*.*.bar.*.baz"
s := "foo.bar.baz.bar.aa.bb"
benchmarkGraphiteMatchTemplateMatch(b, tpl, s, false)
})
}
func benchmarkGraphiteMatchTemplateMatch(b *testing.B, tpl, s string, okExpected bool) {
gmt := newGraphiteMatchTemplate(tpl)
b.ReportAllocs()
b.SetBytes(1)
b.RunParallel(func(pb *testing.PB) {
var matches []string
for pb.Next() {
var ok bool
matches, ok = gmt.Match(matches[:0], s)
if ok != okExpected {
panic(fmt.Errorf("unexpected ok=%v for tpl=%q, s=%q", ok, tpl, s))
}
}
})
}
func BenchmarkGraphiteReplaceTemplateExpand(b *testing.B) {
b.Run("one-replacement", func(b *testing.B) {
tpl := "$1"
matches := []string{"", "foo"}
resultExpected := "foo"
benchmarkGraphiteReplaceTemplateExpand(b, tpl, matches, resultExpected)
})
b.Run("one-replacement-with-prefix", func(b *testing.B) {
tpl := "x-$1"
matches := []string{"", "foo"}
resultExpected := "x-foo"
benchmarkGraphiteReplaceTemplateExpand(b, tpl, matches, resultExpected)
})
b.Run("one-replacement-with-prefix-suffix", func(b *testing.B) {
tpl := "x-$1-y"
matches := []string{"", "foo"}
resultExpected := "x-foo-y"
benchmarkGraphiteReplaceTemplateExpand(b, tpl, matches, resultExpected)
})
b.Run("two-replacements", func(b *testing.B) {
tpl := "$1$2"
matches := []string{"", "foo", "bar"}
resultExpected := "foobar"
benchmarkGraphiteReplaceTemplateExpand(b, tpl, matches, resultExpected)
})
b.Run("two-replacements-with-delimiter", func(b *testing.B) {
tpl := "$1-$2"
matches := []string{"", "foo", "bar"}
resultExpected := "foo-bar"
benchmarkGraphiteReplaceTemplateExpand(b, tpl, matches, resultExpected)
})
}
func benchmarkGraphiteReplaceTemplateExpand(b *testing.B, tpl string, matches []string, resultExpected string) {
grt := newGraphiteReplaceTemplate(tpl)
b.ReportAllocs()
b.SetBytes(1)
b.RunParallel(func(pb *testing.PB) {
var b []byte
for pb.Next() {
b = grt.Expand(b[:0], matches)
if string(b) != resultExpected {
panic(fmt.Errorf("unexpected result; got\n%q\nwant\n%q", b, resultExpected))
}
}
})
}

8
lib/promrelabel/if_expression.go
View file

@ -18,6 +18,14 @@ type IfExpression struct {
lfs []*labelFilter
}
// String returns string representation of ie.
func (ie *IfExpression) String() string {
if ie == nil {
return ""
}
return ie.s
}
// Parse parses `if` expression from s and stores it to ie.
func (ie *IfExpression) Parse(s string) error {
expr, err := metricsql.Parse(s)

31
lib/promrelabel/if_expression_test.go
View file

@ -2,6 +2,7 @@ package promrelabel
import (
"bytes"
"encoding/json"
"fmt"
"testing"
@ -36,6 +37,36 @@ func TestIfExpressionParseSuccess(t *testing.T) {
f(`foo{bar=~"baz", x!="y"}`)
}
func TestIfExpressionMarshalUnmarshalJSON(t *testing.T) {
f := func(s, jsonExpected string) {
t.Helper()
var ie IfExpression
if err := ie.Parse(s); err != nil {
t.Fatalf("cannot parse ifExpression %q: %s", s, err)
}
data, err := json.Marshal(&ie)
if err != nil {
t.Fatalf("cannot marshal ifExpression %q: %s", s, err)
}
if string(data) != jsonExpected {
t.Fatalf("unexpected value after json marshaling;\ngot\n%s\nwant\n%s", data, jsonExpected)
}
var ie2 IfExpression
if err := json.Unmarshal(data, &ie2); err != nil {
t.Fatalf("cannot unmarshal ifExpression from json %q: %s", data, err)
}
data2, err := json.Marshal(&ie2)
if err != nil {
t.Fatalf("cannot marshal ifExpression2: %s", err)
}
if string(data2) != jsonExpected {
t.Fatalf("unexpected data after unmarshal/marshal cycle;\ngot\n%s\nwant\n%s", data2, jsonExpected)
}
}
f("foo", `"foo"`)
f(`{foo="bar",baz=~"x.*"}`, `"{foo=\"bar\",baz=~\"x.*\"}"`)
}
func TestIfExpressionUnmarshalFailure(t *testing.T) {
f := func(s string) {
t.Helper()

27
lib/promrelabel/relabel.go
View file

@ -25,6 +25,9 @@ type parsedRelabelConfig struct {
Action string
If *IfExpression
graphiteMatchTemplate *graphiteMatchTemplate
graphiteLabelRules []graphiteLabelRule
regexOriginal *regexp.Regexp
hasCaptureGroupInTargetLabel bool
hasCaptureGroupInReplacement bool
@ -32,8 +35,8 @@ type parsedRelabelConfig struct {
// String returns human-readable representation for prc.
func (prc *parsedRelabelConfig) String() string {
return fmt.Sprintf("SourceLabels=%s, Separator=%s, TargetLabel=%s, Regex=%s, Modulus=%d, Replacement=%s, Action=%s",
prc.SourceLabels, prc.Separator, prc.TargetLabel, prc.Regex.String(), prc.Modulus, prc.Replacement, prc.Action)
return fmt.Sprintf("SourceLabels=%s, Separator=%s, TargetLabel=%s, Regex=%s, Modulus=%d, Replacement=%s, Action=%s, If=%s, graphiteMatchTemplate=%s, graphiteLabelRules=%s",
prc.SourceLabels, prc.Separator, prc.TargetLabel, prc.Regex, prc.Modulus, prc.Replacement, prc.Action, prc.If, prc.graphiteMatchTemplate, prc.graphiteLabelRules)
}
// Apply applies pcs to labels starting from the labelsOffset.
@ -147,6 +150,26 @@ func (prc *parsedRelabelConfig) apply(labels []prompbmarshal.Label, labelsOffset
return labels
}
switch prc.Action {
case "graphite":
metricName := GetLabelValueByName(src, "__name__")
gm := graphiteMatchesPool.Get().(*graphiteMatches)
var ok bool
gm.a, ok = prc.graphiteMatchTemplate.Match(gm.a[:0], metricName)
if !ok {
// Fast path - name mismatch
graphiteMatchesPool.Put(gm)
return labels
}
// Slow path - extract labels from graphite metric name
bb := relabelBufPool.Get()
for _, gl := range prc.graphiteLabelRules {
bb.B = gl.grt.Expand(bb.B[:0], gm.a)
valueStr := string(bb.B)
labels = setLabelValue(labels, labelsOffset, gl.targetLabel, valueStr)
}
relabelBufPool.Put(bb)
graphiteMatchesPool.Put(gm)
return labels
case "replace":
// Store `replacement` at `target_label` if the `regex` matches `source_labels` joined with `separator`
bb := relabelBufPool.Get()

65
lib/promrelabel/relabel_test.go
View file

@ -1580,7 +1580,6 @@ func TestApplyRelabelConfigs(t *testing.T) {
},
})
})
t.Run("upper-lower-case", func(t *testing.T) {
f(`
- action: uppercase
@ -1618,8 +1617,7 @@ func TestApplyRelabelConfigs(t *testing.T) {
Value: "bar;foo",
},
})
})
f(`
f(`
- action: lowercase
source_labels: ["foo"]
target_label: baz
@ -1627,15 +1625,58 @@ func TestApplyRelabelConfigs(t *testing.T) {
source_labels: ["bar"]
target_label: baz
`, []prompbmarshal.Label{
{
Name: "qux",
Value: "quux",
},
}, true, []prompbmarshal.Label{
{
Name: "qux",
Value: "quux",
},
{
Name: "qux",
Value: "quux",
},
}, true, []prompbmarshal.Label{
{
Name: "qux",
Value: "quux",
},
})
})
t.Run("graphite-match", func(t *testing.T) {
f(`
- action: graphite
match: foo.*.baz
labels:
__name__: aaa
job: ${1}-zz
`, []prompbmarshal.Label{
{
Name: "__name__",
Value: "foo.bar.baz",
},
}, true, []prompbmarshal.Label{
{
Name: "__name__",
Value: "aaa",
},
{
Name: "job",
Value: "bar-zz",
},
})
})
t.Run("graphite-mismatch", func(t *testing.T) {
f(`
- action: graphite
match: foo.*.baz
labels:
__name__: aaa
job: ${1}-zz
`, []prompbmarshal.Label{
{
Name: "__name__",
Value: "foo.bar.bazz",
},
}, true, []prompbmarshal.Label{
{
Name: "__name__",
Value: "foo.bar.bazz",
},
})
})
}

10
lib/promrelabel/testdata/relabel_configs_valid.yml vendored
View file

@ -38,4 +38,12 @@
action: uppercase
- source_labels: [__tmp_uppercase]
target_label: lower_aaa
action: lowercase
action: lowercase
- if: '{foo=~"bar.*",baz="aa"}'
target_label: aaa
replacement: foobar
- action: graphite
match: 'foo.*.bar'
labels:
instance: 'foo-$1'
job: '${1}-bar'