mirror of
https://github.com/VictoriaMetrics/VictoriaMetrics.git
synced 2024-12-01 14:47:38 +00:00
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
This commit is contained in:
Aliaksandr Valialkin
commit
90a1502335
57 changed files with 7444 additions and 3679 deletions
28
README.md
28
README.md
|
|
@ -102,7 +102,7 @@ Just download [VictoriaMetrics executable](https://github.com/VictoriaMetrics/Vi
|
|||
The following command-line flags are used the most:
|
||||
|
||||
* `-storageDataPath` - VictoriaMetrics stores all the data in this directory. Default path is `victoria-metrics-data` in the current working directory.
|
||||
* `-retentionPeriod` - retention for stored data. Older data is automatically deleted. Default retention is 1 month. See [these docs](#retention) for more details.
|
||||
* `-retentionPeriod` - retention for stored data. Older data is automatically deleted. Default retention is 1 month. See [the Retention section](#retention) for more details.
|
||||
|
||||
Other flags have good enough default values, so set them only if you really need this. Pass `-help` to see [all the available flags with description and default values](#list-of-command-line-flags).
|
||||
|
||||
|
|
@ -744,7 +744,7 @@ The delete API is intended mainly for the following cases:
|
|||
* One-off deleting of accidentally written invalid (or undesired) time series.
|
||||
* One-off deleting of user data due to [GDPR](https://en.wikipedia.org/wiki/General_Data_Protection_Regulation).
|
||||
|
||||
It isn't recommended using delete API for the following cases, since it brings non-zero overhead:
|
||||
Using the delete API is not recommended in the following cases, since it brings a non-zero overhead:
|
||||
|
||||
* Regular cleanups for unneeded data. Just prevent writing unneeded data into VictoriaMetrics.
|
||||
This can be done with [relabeling](#relabeling).
|
||||
|
|
@ -753,7 +753,7 @@ It isn't recommended using delete API for the following cases, since it brings n
|
|||
time series occupy disk space until the next merge operation, which can never occur when deleting too old data.
|
||||
[Forced merge](#forced-merge) may be used for freeing up disk space occupied by old data.
|
||||
|
||||
It is better using `-retentionPeriod` command-line flag for efficient pruning of old data.
|
||||
It's better to use the `-retentionPeriod` command-line flag for efficient pruning of old data.
|
||||
|
||||
|
||||
## Forced merge
|
||||
|
|
@ -1147,7 +1147,7 @@ write data to the same VictoriaMetrics instance. These vmagent or Prometheus ins
|
|||
VictoriaMetrics stores time series data in [MergeTree](https://en.wikipedia.org/wiki/Log-structured_merge-tree)-like
|
||||
data structures. On insert, VictoriaMetrics accumulates up to 1s of data and dumps it on disk to
|
||||
`<-storageDataPath>/data/small/YYYY_MM/` subdirectory forming a `part` with the following
|
||||
name pattern `rowsCount_blocksCount_minTimestamp_maxTimestamp`. Each part consists of two "columns":
|
||||
name pattern: `rowsCount_blocksCount_minTimestamp_maxTimestamp`. Each part consists of two "columns":
|
||||
values and timestamps. These are sorted and compressed raw time series values. Additionally, part contains
|
||||
index files for searching for specific series in the values and timestamps files.
|
||||
|
||||
|
|
@ -1177,24 +1177,24 @@ See also [how to work with snapshots](#how-to-work-with-snapshots).
|
|||
|
||||
## Retention
|
||||
|
||||
Retention is configured with `-retentionPeriod` command-line flag. For instance, `-retentionPeriod=3` means
|
||||
that the data will be stored for 3 months and then deleted.
|
||||
Data is split in per-month partitions inside `<-storageDataPath>/data/{small,big}` folders.
|
||||
Data partitions outside the configured retention are deleted on the first day of new month.
|
||||
Retention is configured with the `-retentionPeriod` command-line flag, which takes a number followed by a time unit character - `h(ours)`, `d(ays)`, `w(eeks)`, `y(ears)`. If the time unit is not specified, a month is assumed. For instance, `-retentionPeriod=3` means that the data will be stored for 3 months and then deleted. The default retention period is one month.
|
||||
|
||||
Data is split in per-month partitions inside `<-storageDataPath>/data/{small,big}` folders.
|
||||
Data partitions outside the configured retention are deleted on the first day of the new month.
|
||||
Each partition consists of one or more data parts with the following name pattern `rowsCount_blocksCount_minTimestamp_maxTimestamp`.
|
||||
Data parts outside of the configured retention are eventually deleted during
|
||||
[background merge](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282).
|
||||
|
||||
In order to keep data according to `-retentionPeriod` max disk space usage is going to be `-retentionPeriod` + 1 month.
|
||||
For example if `-retentionPeriod` is set to 1, data for January is deleted on March 1st.
|
||||
The maximum disk space usage for a given `-retentionPeriod` is going to be (`-retentionPeriod` + 1) months.
|
||||
For example, if `-retentionPeriod` is set to 1, data for January is deleted on March 1st.
|
||||
|
||||
VictoriaMetrics supports retention smaller than 1 month. For example, `-retentionPeriod=5d` would set data retention for 5 days.
|
||||
Please note, time range covered by data part is not limited by retention period unit. Hence, data part may contain data
|
||||
Please note, the time range covered by data part is not limited by retention period unit. Hence, data part may contain data
|
||||
for multiple days and will be deleted only when fully outside of the configured retention.
|
||||
|
||||
It is safe to extend `-retentionPeriod` on existing data. If `-retentionPeriod` is set to lower
|
||||
value than before then data outside the configured period will be eventually deleted.
|
||||
It is safe to extend `-retentionPeriod` on existing data. If `-retentionPeriod` is set to a lower
|
||||
value than before, then data outside the configured period will be eventually deleted.
|
||||
|
||||
VictoriaMetrics does not support indefinite retention, but you can specify an arbitrarily high duration, e.g. `-retentionPeriod=100y`.
|
||||
|
||||
## Multiple retentions
|
||||
|
||||
|
|
|
|||
|
|
@ -264,7 +264,7 @@ Labels can be added to metrics by the following mechanisms:
|
|||
|
||||
## Relabeling
|
||||
|
||||
`vmagent` and VictoriaMetrics support Prometheus-compatible relabeling.
|
||||
VictoriaMetrics components (including `vmagent`) support Prometheus-compatible relabeling.
|
||||
They provide the following additional actions on top of actions from the [Prometheus relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config):
|
||||
|
||||
* `replace_all`: replaces all of the occurences of `regex` in the values of `source_labels` with the `replacement` and stores the results in the `target_label`.
|
||||
|
|
@ -289,6 +289,21 @@ The `regex` value can be split into multiple lines for improved readability and
|
|||
- "foo_.+"
|
||||
```
|
||||
|
||||
VictoriaMetrics components support an optional `if` filter, which can be used for conditional relabeling. The `if` filter may contain arbitrary [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors). For example, the following relabeling rule drops targets, which don't match `foo{bar="baz"}` series selector:
|
||||
|
||||
```yaml
|
||||
- action: keep
|
||||
if: 'foo{bar="baz"}'
|
||||
```
|
||||
|
||||
This is equivalent to less clear traditional relabeling rule:
|
||||
|
||||
```yaml
|
||||
- action: keep
|
||||
source_labels: [__name__, bar]
|
||||
regex: 'foo;baz'
|
||||
```
|
||||
|
||||
The relabeling can be defined in the following places:
|
||||
|
||||
* At the `scrape_config -> relabel_configs` section in `-promscrape.config` file. This relabeling is applied to target labels. This relabeling can be debugged by passing `relabel_debug: true` option to the corresponding `scrape_config` section. In this case `vmagent` logs target labels before and after the relabeling and then drops the logged target.
|
||||
|
|
|
|||
|
|
@ -382,7 +382,7 @@ func getCommonLabelFilters(tss []*timeseries) []metricsql.LabelFilter {
|
|||
continue
|
||||
}
|
||||
values = getUniqueValues(values)
|
||||
if len(values) > 10000 {
|
||||
if len(values) > 1000 {
|
||||
// Skip the filter on the given tag, since it needs to enumerate too many unique values.
|
||||
// This may slow down the search for matching time series.
|
||||
continue
|
||||
|
|
|
|||
|
|
@ -76,7 +76,7 @@ var checkRollupResultCacheResetOnce sync.Once
|
|||
var rollupResultResetMetricRowSample atomic.Value
|
||||
|
||||
var rollupResultCacheV = &rollupResultCache{
|
||||
c: workingsetcache.New(1024*1024, time.Hour), // This is a cache for testing.
|
||||
c: workingsetcache.New(1024 * 1024), // This is a cache for testing.
|
||||
}
|
||||
var rollupResultCachePath string
|
||||
|
||||
|
|
@ -104,9 +104,9 @@ func InitRollupResultCache(cachePath string) {
|
|||
var c *workingsetcache.Cache
|
||||
if len(rollupResultCachePath) > 0 {
|
||||
logger.Infof("loading rollupResult cache from %q...", rollupResultCachePath)
|
||||
c = workingsetcache.Load(rollupResultCachePath, cacheSize, time.Hour)
|
||||
c = workingsetcache.Load(rollupResultCachePath, cacheSize)
|
||||
} else {
|
||||
c = workingsetcache.New(cacheSize, time.Hour)
|
||||
c = workingsetcache.New(cacheSize)
|
||||
}
|
||||
if *disableCache {
|
||||
c.Reset()
|
||||
|
|
|
|||
|
|
@ -1,12 +1,12 @@
|
|||
{
|
||||
"files": {
|
||||
"main.css": "./static/css/main.098d452b.css",
|
||||
"main.js": "./static/js/main.2a9382c2.js",
|
||||
"main.js": "./static/js/main.a292ed17.js",
|
||||
"static/js/27.939f971b.chunk.js": "./static/js/27.939f971b.chunk.js",
|
||||
"index.html": "./index.html"
|
||||
},
|
||||
"entrypoints": [
|
||||
"static/css/main.098d452b.css",
|
||||
"static/js/main.2a9382c2.js"
|
||||
"static/js/main.a292ed17.js"
|
||||
]
|
||||
}
|
||||
|
|
@ -1 +1 @@
|
|||
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="VM-UI is a metric explorer for Victoria Metrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,500,700&display=swap"/><script defer="defer" src="./static/js/main.2a9382c2.js"></script><link href="./static/css/main.098d452b.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>
|
||||
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="VM-UI is a metric explorer for Victoria Metrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,500,700&display=swap"/><script defer="defer" src="./static/js/main.a292ed17.js"></script><link href="./static/css/main.098d452b.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>
|
||||
File diff suppressed because one or more lines are too long
2
app/vmselect/vmui/static/js/main.a292ed17.js
Normal file
2
app/vmselect/vmui/static/js/main.a292ed17.js
Normal file
File diff suppressed because one or more lines are too long
|
|
@ -6,7 +6,7 @@
|
|||
* @license MIT
|
||||
*/
|
||||
|
||||
/** @license MUI v5.4.2
|
||||
/** @license MUI v5.4.4
|
||||
*
|
||||
* This source code is licensed under the MIT license found in the
|
||||
* LICENSE file in the root directory of this source tree.
|
||||
6034
app/vmui/packages/vmui/package-lock.json
generated
6034
app/vmui/packages/vmui/package-lock.json
generated
File diff suppressed because it is too large
Load diff
|
|
@ -13,11 +13,11 @@
|
|||
"@testing-library/jest-dom": "^5.16.2",
|
||||
"@testing-library/react": "^12.1.3",
|
||||
"@testing-library/user-event": "^13.5.0",
|
||||
"@types/jest": "^27.4.0",
|
||||
"@types/jest": "^27.4.1",
|
||||
"@types/lodash.debounce": "^4.0.6",
|
||||
"@types/lodash.get": "^4.4.6",
|
||||
"@types/lodash.throttle": "^4.1.6",
|
||||
"@types/node": "^17.0.19",
|
||||
"@types/node": "^17.0.21",
|
||||
"@types/qs": "^6.9.7",
|
||||
"@types/react": "^17.0.39",
|
||||
"@types/react-dom": "^17.0.11",
|
||||
|
|
@ -63,7 +63,7 @@
|
|||
"@typescript-eslint/eslint-plugin": "^5.12.1",
|
||||
"@typescript-eslint/parser": "^5.12.1",
|
||||
"customize-cra": "^1.0.0",
|
||||
"eslint-plugin-react": "^7.28.0",
|
||||
"eslint-plugin-react": "^7.29.2",
|
||||
"react-app-rewired": "^2.2.1"
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -15,6 +15,36 @@ The following tip changes can be tested by building VictoriaMetrics components f
|
|||
## tip
|
||||
|
||||
|
||||
## [v1.74.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.74.0)
|
||||
|
||||
Released at 03-03-2022
|
||||
|
||||
* FEATURE: add support for conditional relabeling via `if` filter. The `if` filter can contain arbitrary [series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors). For example, the following rule drops targets matching `foo{bar="baz"}` series selector:
|
||||
|
||||
```yml
|
||||
- action: drop
|
||||
if: 'foo{bar="baz"}'
|
||||
```
|
||||
|
||||
This rule is equivalent to less clear traditional one:
|
||||
|
||||
```yml
|
||||
- action: drop
|
||||
source_labels: [__name__, bar]
|
||||
regex: 'foo;baz'
|
||||
```
|
||||
|
||||
See [relabeling docs](https://docs.victoriametrics.com/vmagent.html#relabeling) and [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1998) for more details.
|
||||
|
||||
* FEATURE: reduce memory usage for various caches under [high churn rate](https://docs.victoriametrics.com/FAQ.html#what-is-high-churn-rate).
|
||||
* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): re-use Kafka client when pushing data from [many tenants](https://docs.victoriametrics.com/vmagent.html#multitenancy) to Kafka. Previously a separate Kafka client was created per each tenant. This could lead to increased load on Kafka. See [how to push data from vmagent to Kafka](https://docs.victoriametrics.com/vmagent.html#writing-metrics-to-kafka).
|
||||
* FEATURE: improve performance when registering new time series. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2247). Thanks to @ahfuzhang .
|
||||
|
||||
* BUGFIX: return the proper number of datapoints from `moving*()` functions such as `movingAverage()` in [Graphite Render API](https://docs.victoriametrics.com/#graphite-render-api-usage). Previously these functions could return too big number of samples if [maxDataPoints query arg](https://graphite.readthedocs.io/en/stable/render_api.html#maxdatapoints) is explicitly passed to `/render` API.
|
||||
* BUGFIX: properly handle [series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) containing a filter for multiple metric names plus a negative filter. For example, `{__name__=~"foo|bar",job!="baz"}` . Previously VictoriaMetrics could return series with `foo` or `bar` names and with `job="baz"`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2238).
|
||||
* BUGFIX: [vmgateway](https://docs.victoriametrics.com/vmgateway.html): properly parse JWT tokens if they are encoded with [URL-safe base64 encoding](https://datatracker.ietf.org/doc/html/rfc4648#section-5).
|
||||
|
||||
|
||||
## [v1.73.1](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.73.1)
|
||||
|
||||
Released at 22-02-2022
|
||||
|
|
@ -32,6 +62,7 @@ Released at 22-02-2022
|
|||
* BUGFIX: vmalert: add support for `$externalLabels` and `$externalURL` template vars in the same way as Prometheus does. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2193).
|
||||
* BUGFIX: vmalert: make sure notifiers are discovered during initialization if they are configured via `consul_sd_configs`. Previously they could be discovered in 30 seconds (the default value for `-promscrape.consulSDCheckInterval` command-line flag) after the initialization. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2202).
|
||||
* BUGFIX: update default value for `-promscrape.fileSDCheckInterval`, so it matches default duration used by Prometheus for checking for updates in `file_sd_configs`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2187). Thanks to @corporate-gadfly for the fix.
|
||||
* BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html): do not return partial responses from `vmselect` if at least a single `vmstorage` node was reachable and returned an app-level error. Such errors are usually related to cluster mis-configuration, so they must be returned to the caller instead of being masked by [partial responses](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#cluster-availability). Partial responses can be returned only if some of `vmstorage` nodes are unreachable during the query. This may help the following issues: [one](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1941), [two](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/678).
|
||||
|
||||
|
||||
## [v1.73.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.73.0)
|
||||
|
|
|
|||
|
|
@ -102,7 +102,7 @@ Just download [VictoriaMetrics executable](https://github.com/VictoriaMetrics/Vi
|
|||
The following command-line flags are used the most:
|
||||
|
||||
* `-storageDataPath` - VictoriaMetrics stores all the data in this directory. Default path is `victoria-metrics-data` in the current working directory.
|
||||
* `-retentionPeriod` - retention for stored data. Older data is automatically deleted. Default retention is 1 month. See [these docs](#retention) for more details.
|
||||
* `-retentionPeriod` - retention for stored data. Older data is automatically deleted. Default retention is 1 month. See [the Retention section](#retention) for more details.
|
||||
|
||||
Other flags have good enough default values, so set them only if you really need this. Pass `-help` to see [all the available flags with description and default values](#list-of-command-line-flags).
|
||||
|
||||
|
|
@ -744,7 +744,7 @@ The delete API is intended mainly for the following cases:
|
|||
* One-off deleting of accidentally written invalid (or undesired) time series.
|
||||
* One-off deleting of user data due to [GDPR](https://en.wikipedia.org/wiki/General_Data_Protection_Regulation).
|
||||
|
||||
It isn't recommended using delete API for the following cases, since it brings non-zero overhead:
|
||||
Using the delete API is not recommended in the following cases, since it brings a non-zero overhead:
|
||||
|
||||
* Regular cleanups for unneeded data. Just prevent writing unneeded data into VictoriaMetrics.
|
||||
This can be done with [relabeling](#relabeling).
|
||||
|
|
@ -753,7 +753,7 @@ It isn't recommended using delete API for the following cases, since it brings n
|
|||
time series occupy disk space until the next merge operation, which can never occur when deleting too old data.
|
||||
[Forced merge](#forced-merge) may be used for freeing up disk space occupied by old data.
|
||||
|
||||
It is better using `-retentionPeriod` command-line flag for efficient pruning of old data.
|
||||
It's better to use the `-retentionPeriod` command-line flag for efficient pruning of old data.
|
||||
|
||||
|
||||
## Forced merge
|
||||
|
|
@ -1147,7 +1147,7 @@ write data to the same VictoriaMetrics instance. These vmagent or Prometheus ins
|
|||
VictoriaMetrics stores time series data in [MergeTree](https://en.wikipedia.org/wiki/Log-structured_merge-tree)-like
|
||||
data structures. On insert, VictoriaMetrics accumulates up to 1s of data and dumps it on disk to
|
||||
`<-storageDataPath>/data/small/YYYY_MM/` subdirectory forming a `part` with the following
|
||||
name pattern `rowsCount_blocksCount_minTimestamp_maxTimestamp`. Each part consists of two "columns":
|
||||
name pattern: `rowsCount_blocksCount_minTimestamp_maxTimestamp`. Each part consists of two "columns":
|
||||
values and timestamps. These are sorted and compressed raw time series values. Additionally, part contains
|
||||
index files for searching for specific series in the values and timestamps files.
|
||||
|
||||
|
|
@ -1177,24 +1177,24 @@ See also [how to work with snapshots](#how-to-work-with-snapshots).
|
|||
|
||||
## Retention
|
||||
|
||||
Retention is configured with `-retentionPeriod` command-line flag. For instance, `-retentionPeriod=3` means
|
||||
that the data will be stored for 3 months and then deleted.
|
||||
Data is split in per-month partitions inside `<-storageDataPath>/data/{small,big}` folders.
|
||||
Data partitions outside the configured retention are deleted on the first day of new month.
|
||||
Retention is configured with the `-retentionPeriod` command-line flag, which takes a number followed by a time unit character - `h(ours)`, `d(ays)`, `w(eeks)`, `y(ears)`. If the time unit is not specified, a month is assumed. For instance, `-retentionPeriod=3` means that the data will be stored for 3 months and then deleted. The default retention period is one month.
|
||||
|
||||
Data is split in per-month partitions inside `<-storageDataPath>/data/{small,big}` folders.
|
||||
Data partitions outside the configured retention are deleted on the first day of the new month.
|
||||
Each partition consists of one or more data parts with the following name pattern `rowsCount_blocksCount_minTimestamp_maxTimestamp`.
|
||||
Data parts outside of the configured retention are eventually deleted during
|
||||
[background merge](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282).
|
||||
|
||||
In order to keep data according to `-retentionPeriod` max disk space usage is going to be `-retentionPeriod` + 1 month.
|
||||
For example if `-retentionPeriod` is set to 1, data for January is deleted on March 1st.
|
||||
The maximum disk space usage for a given `-retentionPeriod` is going to be (`-retentionPeriod` + 1) months.
|
||||
For example, if `-retentionPeriod` is set to 1, data for January is deleted on March 1st.
|
||||
|
||||
VictoriaMetrics supports retention smaller than 1 month. For example, `-retentionPeriod=5d` would set data retention for 5 days.
|
||||
Please note, time range covered by data part is not limited by retention period unit. Hence, data part may contain data
|
||||
Please note, the time range covered by data part is not limited by retention period unit. Hence, data part may contain data
|
||||
for multiple days and will be deleted only when fully outside of the configured retention.
|
||||
|
||||
It is safe to extend `-retentionPeriod` on existing data. If `-retentionPeriod` is set to lower
|
||||
value than before then data outside the configured period will be eventually deleted.
|
||||
It is safe to extend `-retentionPeriod` on existing data. If `-retentionPeriod` is set to a lower
|
||||
value than before, then data outside the configured period will be eventually deleted.
|
||||
|
||||
VictoriaMetrics does not support indefinite retention, but you can specify an arbitrarily high duration, e.g. `-retentionPeriod=100y`.
|
||||
|
||||
## Multiple retentions
|
||||
|
||||
|
|
|
|||
|
|
@ -106,7 +106,7 @@ Just download [VictoriaMetrics executable](https://github.com/VictoriaMetrics/Vi
|
|||
The following command-line flags are used the most:
|
||||
|
||||
* `-storageDataPath` - VictoriaMetrics stores all the data in this directory. Default path is `victoria-metrics-data` in the current working directory.
|
||||
* `-retentionPeriod` - retention for stored data. Older data is automatically deleted. Default retention is 1 month. See [these docs](#retention) for more details.
|
||||
* `-retentionPeriod` - retention for stored data. Older data is automatically deleted. Default retention is 1 month. See [the Retention section](#retention) for more details.
|
||||
|
||||
Other flags have good enough default values, so set them only if you really need this. Pass `-help` to see [all the available flags with description and default values](#list-of-command-line-flags).
|
||||
|
||||
|
|
@ -748,7 +748,7 @@ The delete API is intended mainly for the following cases:
|
|||
* One-off deleting of accidentally written invalid (or undesired) time series.
|
||||
* One-off deleting of user data due to [GDPR](https://en.wikipedia.org/wiki/General_Data_Protection_Regulation).
|
||||
|
||||
It isn't recommended using delete API for the following cases, since it brings non-zero overhead:
|
||||
Using the delete API is not recommended in the following cases, since it brings a non-zero overhead:
|
||||
|
||||
* Regular cleanups for unneeded data. Just prevent writing unneeded data into VictoriaMetrics.
|
||||
This can be done with [relabeling](#relabeling).
|
||||
|
|
@ -757,7 +757,7 @@ It isn't recommended using delete API for the following cases, since it brings n
|
|||
time series occupy disk space until the next merge operation, which can never occur when deleting too old data.
|
||||
[Forced merge](#forced-merge) may be used for freeing up disk space occupied by old data.
|
||||
|
||||
It is better using `-retentionPeriod` command-line flag for efficient pruning of old data.
|
||||
It's better to use the `-retentionPeriod` command-line flag for efficient pruning of old data.
|
||||
|
||||
|
||||
## Forced merge
|
||||
|
|
@ -1151,7 +1151,7 @@ write data to the same VictoriaMetrics instance. These vmagent or Prometheus ins
|
|||
VictoriaMetrics stores time series data in [MergeTree](https://en.wikipedia.org/wiki/Log-structured_merge-tree)-like
|
||||
data structures. On insert, VictoriaMetrics accumulates up to 1s of data and dumps it on disk to
|
||||
`<-storageDataPath>/data/small/YYYY_MM/` subdirectory forming a `part` with the following
|
||||
name pattern `rowsCount_blocksCount_minTimestamp_maxTimestamp`. Each part consists of two "columns":
|
||||
name pattern: `rowsCount_blocksCount_minTimestamp_maxTimestamp`. Each part consists of two "columns":
|
||||
values and timestamps. These are sorted and compressed raw time series values. Additionally, part contains
|
||||
index files for searching for specific series in the values and timestamps files.
|
||||
|
||||
|
|
@ -1181,24 +1181,24 @@ See also [how to work with snapshots](#how-to-work-with-snapshots).
|
|||
|
||||
## Retention
|
||||
|
||||
Retention is configured with `-retentionPeriod` command-line flag. For instance, `-retentionPeriod=3` means
|
||||
that the data will be stored for 3 months and then deleted.
|
||||
Data is split in per-month partitions inside `<-storageDataPath>/data/{small,big}` folders.
|
||||
Data partitions outside the configured retention are deleted on the first day of new month.
|
||||
Retention is configured with the `-retentionPeriod` command-line flag, which takes a number followed by a time unit character - `h(ours)`, `d(ays)`, `w(eeks)`, `y(ears)`. If the time unit is not specified, a month is assumed. For instance, `-retentionPeriod=3` means that the data will be stored for 3 months and then deleted. The default retention period is one month.
|
||||
|
||||
Data is split in per-month partitions inside `<-storageDataPath>/data/{small,big}` folders.
|
||||
Data partitions outside the configured retention are deleted on the first day of the new month.
|
||||
Each partition consists of one or more data parts with the following name pattern `rowsCount_blocksCount_minTimestamp_maxTimestamp`.
|
||||
Data parts outside of the configured retention are eventually deleted during
|
||||
[background merge](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282).
|
||||
|
||||
In order to keep data according to `-retentionPeriod` max disk space usage is going to be `-retentionPeriod` + 1 month.
|
||||
For example if `-retentionPeriod` is set to 1, data for January is deleted on March 1st.
|
||||
The maximum disk space usage for a given `-retentionPeriod` is going to be (`-retentionPeriod` + 1) months.
|
||||
For example, if `-retentionPeriod` is set to 1, data for January is deleted on March 1st.
|
||||
|
||||
VictoriaMetrics supports retention smaller than 1 month. For example, `-retentionPeriod=5d` would set data retention for 5 days.
|
||||
Please note, time range covered by data part is not limited by retention period unit. Hence, data part may contain data
|
||||
Please note, the time range covered by data part is not limited by retention period unit. Hence, data part may contain data
|
||||
for multiple days and will be deleted only when fully outside of the configured retention.
|
||||
|
||||
It is safe to extend `-retentionPeriod` on existing data. If `-retentionPeriod` is set to lower
|
||||
value than before then data outside the configured period will be eventually deleted.
|
||||
It is safe to extend `-retentionPeriod` on existing data. If `-retentionPeriod` is set to a lower
|
||||
value than before, then data outside the configured period will be eventually deleted.
|
||||
|
||||
VictoriaMetrics does not support indefinite retention, but you can specify an arbitrarily high duration, e.g. `-retentionPeriod=100y`.
|
||||
|
||||
## Multiple retentions
|
||||
|
||||
|
|
|
|||
|
|
@ -22,7 +22,7 @@ With CRD (Custom Resource Definition) you can define application configuration a
|
|||
|
||||
## Use cases
|
||||
|
||||
For kubernetes-cluster administrators, it simplifies installation, configuration, management for `VictoriaMetrics` application. And the main feature of operator - is ability to delegate applications monitoring configuration to the end-users.
|
||||
For kubernetes-cluster administrators, it simplifies installation, configuration and management for `VictoriaMetrics` application. The main feature of operator is its ability to delegate the configuration of applications monitoring to the end-users.
|
||||
|
||||
For applications developers, its great possibility for managing observability of applications. You can define metrics scraping and alerting configuration for your application and manage it with an application deployment process. Just define app_deployment.yaml, app_vmpodscrape.yaml and app_vmrule.yaml. That's it, you can apply it to a kubernetes cluster. Check [quick-start](/Operator/quick-start.html) for an example.
|
||||
|
||||
|
|
|
|||
|
|
@ -268,7 +268,7 @@ Labels can be added to metrics by the following mechanisms:
|
|||
|
||||
## Relabeling
|
||||
|
||||
`vmagent` and VictoriaMetrics support Prometheus-compatible relabeling.
|
||||
VictoriaMetrics components (including `vmagent`) support Prometheus-compatible relabeling.
|
||||
They provide the following additional actions on top of actions from the [Prometheus relabeling](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config):
|
||||
|
||||
* `replace_all`: replaces all of the occurences of `regex` in the values of `source_labels` with the `replacement` and stores the results in the `target_label`.
|
||||
|
|
@ -293,6 +293,21 @@ The `regex` value can be split into multiple lines for improved readability and
|
|||
- "foo_.+"
|
||||
```
|
||||
|
||||
VictoriaMetrics components support an optional `if` filter, which can be used for conditional relabeling. The `if` filter may contain arbitrary [time series selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors). For example, the following relabeling rule drops targets, which don't match `foo{bar="baz"}` series selector:
|
||||
|
||||
```yaml
|
||||
- action: keep
|
||||
if: 'foo{bar="baz"}'
|
||||
```
|
||||
|
||||
This is equivalent to less clear traditional relabeling rule:
|
||||
|
||||
```yaml
|
||||
- action: keep
|
||||
source_labels: [__name__, bar]
|
||||
regex: 'foo;baz'
|
||||
```
|
||||
|
||||
The relabeling can be defined in the following places:
|
||||
|
||||
* At the `scrape_config -> relabel_configs` section in `-promscrape.config` file. This relabeling is applied to target labels. This relabeling can be debugged by passing `relabel_debug: true` option to the corresponding `scrape_config` section. In this case `vmagent` logs target labels before and after the relabeling and then drops the logged target.
|
||||
|
|
|
|||
14
go.mod
14
go.mod
|
|
@ -11,7 +11,7 @@ require (
|
|||
github.com/VictoriaMetrics/fasthttp v1.1.0
|
||||
github.com/VictoriaMetrics/metrics v1.18.1
|
||||
github.com/VictoriaMetrics/metricsql v0.40.0
|
||||
github.com/aws/aws-sdk-go v1.43.3
|
||||
github.com/aws/aws-sdk-go v1.43.10
|
||||
github.com/cespare/xxhash/v2 v2.1.2
|
||||
github.com/cheggaaa/pb/v3 v3.0.8
|
||||
github.com/cpuguy83/go-md2man/v2 v2.0.1 // indirect
|
||||
|
|
@ -31,17 +31,17 @@ require (
|
|||
github.com/valyala/fasttemplate v1.2.1
|
||||
github.com/valyala/gozstd v1.16.0
|
||||
github.com/valyala/quicktemplate v1.7.0
|
||||
golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd
|
||||
golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8
|
||||
golang.org/x/sys v0.0.0-20220222172238-00053529121e
|
||||
golang.org/x/net v0.0.0-20220225172249-27dd8689420f
|
||||
golang.org/x/oauth2 v0.0.0-20220223155221-ee480838109b
|
||||
golang.org/x/sys v0.0.0-20220227234510-4e6760a101f9
|
||||
google.golang.org/api v0.70.0
|
||||
gopkg.in/yaml.v2 v2.4.0
|
||||
)
|
||||
|
||||
require (
|
||||
cloud.google.com/go v0.100.2 // indirect
|
||||
cloud.google.com/go/compute v1.4.0 // indirect
|
||||
cloud.google.com/go/iam v0.2.0 // indirect
|
||||
cloud.google.com/go/compute v1.5.0 // indirect
|
||||
cloud.google.com/go/iam v0.3.0 // indirect
|
||||
github.com/VividCortex/ewma v1.2.0 // indirect
|
||||
github.com/beorn7/perks v1.0.1 // indirect
|
||||
github.com/go-kit/log v0.2.0 // indirect
|
||||
|
|
@ -68,7 +68,7 @@ require (
|
|||
golang.org/x/text v0.3.7 // indirect
|
||||
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
|
||||
google.golang.org/appengine v1.6.7 // indirect
|
||||
google.golang.org/genproto v0.0.0-20220222154240-daf995802d7b // indirect
|
||||
google.golang.org/genproto v0.0.0-20220302033224-9aa15565e42a // indirect
|
||||
google.golang.org/grpc v1.44.0 // indirect
|
||||
google.golang.org/protobuf v1.27.1 // indirect
|
||||
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b // indirect
|
||||
|
|
|
|||
28
go.sum
28
go.sum
|
|
@ -43,13 +43,13 @@ cloud.google.com/go/bigtable v1.10.1/go.mod h1:cyHeKlx6dcZCO0oSQucYdauseD8kIENGu
|
|||
cloud.google.com/go/compute v0.1.0/go.mod h1:GAesmwr110a34z04OlxYkATPBEfVhkymfTBXtfbBFow=
|
||||
cloud.google.com/go/compute v1.2.0/go.mod h1:xlogom/6gr8RJGBe7nT2eGsQYAFUbbv8dbC29qE3Xmw=
|
||||
cloud.google.com/go/compute v1.3.0/go.mod h1:cCZiE1NHEtai4wiufUhW8I8S1JKkAnhnQJWM7YD99wM=
|
||||
cloud.google.com/go/compute v1.4.0 h1:tzSyCe254NKkL8zshJUSoVvI9mcgbFdSpCC44uUNjT0=
|
||||
cloud.google.com/go/compute v1.4.0/go.mod h1:TcrKl8VipL9ZM0wEjdooJ1eet/6YsEV/E/larxxkAdg=
|
||||
cloud.google.com/go/compute v1.5.0 h1:b1zWmYuuHz7gO9kDcM/EpHGr06UgsYNRpNJzI2kFiLM=
|
||||
cloud.google.com/go/compute v1.5.0/go.mod h1:9SMHyhJlzhlkJqrPAc839t2BZFTSk6Jdj6mkzQJeu0M=
|
||||
cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE=
|
||||
cloud.google.com/go/datastore v1.1.0/go.mod h1:umbIZjpQpHh4hmRpGhH4tLFup+FVzqBi1b3c64qFpCk=
|
||||
cloud.google.com/go/iam v0.1.1/go.mod h1:CKqrcnI/suGpybEHxZ7BMehL0oA4LpdyJdUlTl9jVMw=
|
||||
cloud.google.com/go/iam v0.2.0 h1:Ouq6qif4mZdXkb3SiFMpxvu0JQJB1Yid9TsZ23N6hg8=
|
||||
cloud.google.com/go/iam v0.2.0/go.mod h1:BCK88+tmjAwnZYfOSizmKCTSFjJHCa18t3DpdGEY13Y=
|
||||
cloud.google.com/go/iam v0.3.0 h1:exkAomrVUuzx9kWFI1wm3KI0uoDeUFPB4kKGzx6x+Gc=
|
||||
cloud.google.com/go/iam v0.3.0/go.mod h1:XzJPvDayI+9zsASAFO68Hk07u3z+f+JrT2xXNdp4bnY=
|
||||
cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I=
|
||||
cloud.google.com/go/pubsub v1.1.0/go.mod h1:EwwdRX2sKPjnvnqCa270oGRyludottCI76h+R3AArQw=
|
||||
cloud.google.com/go/pubsub v1.2.0/go.mod h1:jhfEVHT8odbXTkndysNHCcx0awwzvfOlguIAii9o8iA=
|
||||
|
|
@ -165,8 +165,8 @@ github.com/aws/aws-sdk-go v1.30.12/go.mod h1:5zCpMtNQVjRREroY7sYe8lOMRSxkhG6MZve
|
|||
github.com/aws/aws-sdk-go v1.34.28/go.mod h1:H7NKnBqNVzoTJpGfLrQkkD+ytBA93eiDYi/+8rV9s48=
|
||||
github.com/aws/aws-sdk-go v1.35.31/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2zKMmprdro=
|
||||
github.com/aws/aws-sdk-go v1.40.45/go.mod h1:585smgzpB/KqRA+K3y/NL/oYRqQvpNJYvLm+LY1U59Q=
|
||||
github.com/aws/aws-sdk-go v1.43.3 h1:qvCkC4FviA9rR4UvRk4ldr6f3mIJE0VaI3KrsDx1gTk=
|
||||
github.com/aws/aws-sdk-go v1.43.3/go.mod h1:OGr6lGMAKGlG9CVrYnWYDKIyb829c6EVBRjxqjmPepc=
|
||||
github.com/aws/aws-sdk-go v1.43.10 h1:lFX6gzTBltYBnlJBjd2DWRCmqn2CbTcs6PW99/Dme7k=
|
||||
github.com/aws/aws-sdk-go v1.43.10/go.mod h1:y4AeaBuwd2Lk+GepC1E9v0qOiTws0MIWAX4oIKwKHZo=
|
||||
github.com/aws/aws-sdk-go-v2 v0.18.0/go.mod h1:JWVYvqSMppoMJC0x5wdwiImzgXTI9FuZwxzkQq9wy+g=
|
||||
github.com/aws/aws-sdk-go-v2 v1.9.1/go.mod h1:cK/D0BBs0b/oWPIcX/Z/obahJK1TT7IPVjy53i/mX/4=
|
||||
github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.8.1/go.mod h1:CM+19rL1+4dFWnOQKwDc7H1KwXTz+h61oUSHyhV0b3o=
|
||||
|
|
@ -1176,9 +1176,9 @@ golang.org/x/net v0.0.0-20210510120150-4163338589ed/go.mod h1:9nx3DQGgdP8bBQD5qx
|
|||
golang.org/x/net v0.0.0-20210525063256-abc453219eb5/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
|
||||
golang.org/x/net v0.0.0-20210614182718-04defd469f4e/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
|
||||
golang.org/x/net v0.0.0-20210917221730-978cfadd31cf/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
|
||||
golang.org/x/net v0.0.0-20211216030914-fe4d6282115f/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
|
||||
golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd h1:O7DYs+zxREGLKzKoMQrtrEacpb0ZVXA5rIwylE2Xchk=
|
||||
golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
|
||||
golang.org/x/net v0.0.0-20220225172249-27dd8689420f h1:oA4XRj0qtSt8Yo1Zms0CUlsT3KG69V2UGQWPBxujDmc=
|
||||
golang.org/x/net v0.0.0-20220225172249-27dd8689420f/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
|
||||
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
|
||||
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
|
||||
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
|
||||
|
|
@ -1195,8 +1195,9 @@ golang.org/x/oauth2 v0.0.0-20210514164344-f6687ab2804c/go.mod h1:KelEdhl1UZF7XfJ
|
|||
golang.org/x/oauth2 v0.0.0-20210628180205-a41e5a781914/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
|
||||
golang.org/x/oauth2 v0.0.0-20210805134026-6f1e6394065a/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
|
||||
golang.org/x/oauth2 v0.0.0-20210819190943-2bc19b11175f/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
|
||||
golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8 h1:RerP+noqYHUQ8CMRcPlC2nvTa4dcBIjegkuWdcUDuqg=
|
||||
golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
|
||||
golang.org/x/oauth2 v0.0.0-20220223155221-ee480838109b h1:clP8eMhB30EHdc0bd2Twtq6kgU7yl5ub2cQLSdrv1Dg=
|
||||
golang.org/x/oauth2 v0.0.0-20220223155221-ee480838109b/go.mod h1:DAh4E804XQdzx2j+YRIaUnCqCV2RuMz24cGBJ5QYIrc=
|
||||
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
|
|
@ -1311,8 +1312,8 @@ golang.org/x/sys v0.0.0-20220114195835-da31bd327af9/go.mod h1:oPkhp1MJrh7nUepCBc
|
|||
golang.org/x/sys v0.0.0-20220128215802-99c3d69c2c27/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.0.0-20220204135822-1c1b9b1eba6a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.0.0-20220209214540-3681064d5158/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.0.0-20220222172238-00053529121e h1:AGLQ2aegkB2Y9RY8YdQk+7MDCW9da7YmizIwNIt8NtQ=
|
||||
golang.org/x/sys v0.0.0-20220222172238-00053529121e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.0.0-20220227234510-4e6760a101f9 h1:nhht2DYV/Sn3qOayu8lM+cU1ii9sTLUeBQwQQfUHtrs=
|
||||
golang.org/x/sys v0.0.0-20220227234510-4e6760a101f9/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
|
||||
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
|
||||
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
|
||||
|
|
@ -1556,8 +1557,9 @@ google.golang.org/genproto v0.0.0-20220207164111-0872dc986b00/go.mod h1:5CzLGKJ6
|
|||
google.golang.org/genproto v0.0.0-20220211171837-173942840c17/go.mod h1:kGP+zUP2Ddo0ayMi4YuN7C3WZyJvGLZRh8Z5wnAqvEI=
|
||||
google.golang.org/genproto v0.0.0-20220216160803-4663080d8bc8/go.mod h1:kGP+zUP2Ddo0ayMi4YuN7C3WZyJvGLZRh8Z5wnAqvEI=
|
||||
google.golang.org/genproto v0.0.0-20220218161850-94dd64e39d7c/go.mod h1:kGP+zUP2Ddo0ayMi4YuN7C3WZyJvGLZRh8Z5wnAqvEI=
|
||||
google.golang.org/genproto v0.0.0-20220222154240-daf995802d7b h1:wHqTlwZVR0x5EG2S6vKlCq63+Tl/vBoQELitHxqxDOo=
|
||||
google.golang.org/genproto v0.0.0-20220222154240-daf995802d7b/go.mod h1:kGP+zUP2Ddo0ayMi4YuN7C3WZyJvGLZRh8Z5wnAqvEI=
|
||||
google.golang.org/genproto v0.0.0-20220222213610-43724f9ea8cf/go.mod h1:kGP+zUP2Ddo0ayMi4YuN7C3WZyJvGLZRh8Z5wnAqvEI=
|
||||
google.golang.org/genproto v0.0.0-20220302033224-9aa15565e42a h1:uqouglH745GoGeZ1YFZbPBiu961tgi/9Qm5jaorajjQ=
|
||||
google.golang.org/genproto v0.0.0-20220302033224-9aa15565e42a/go.mod h1:kGP+zUP2Ddo0ayMi4YuN7C3WZyJvGLZRh8Z5wnAqvEI=
|
||||
google.golang.org/grpc v1.17.0/go.mod h1:6QZJwpn2B+Zp71q/5VxRsJ6NXXVCE5NRUHRo+f3cWCs=
|
||||
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
|
||||
google.golang.org/grpc v1.20.0/go.mod h1:chYK+tFQF0nDUGJgXMSgLCQk3phJEuONr2DCgLDdAQM=
|
||||
|
|
|
|||
|
|
@ -98,20 +98,20 @@ func (bsr *blockStreamReader) String() string {
|
|||
return bsr.ph.String()
|
||||
}
|
||||
|
||||
// InitFromInmemoryPart initializes bsr from the given ip.
|
||||
func (bsr *blockStreamReader) InitFromInmemoryPart(ip *inmemoryPart) {
|
||||
// InitFromInmemoryPart initializes bsr from the given mp.
|
||||
func (bsr *blockStreamReader) InitFromInmemoryPart(mp *inmemoryPart) {
|
||||
bsr.reset()
|
||||
|
||||
var err error
|
||||
bsr.mrs, err = unmarshalMetaindexRows(bsr.mrs[:0], ip.metaindexData.NewReader())
|
||||
bsr.mrs, err = unmarshalMetaindexRows(bsr.mrs[:0], mp.metaindexData.NewReader())
|
||||
if err != nil {
|
||||
logger.Panicf("BUG: cannot unmarshal metaindex rows from inmemory part: %s", err)
|
||||
}
|
||||
|
||||
bsr.ph.CopyFrom(&ip.ph)
|
||||
bsr.indexReader = ip.indexData.NewReader()
|
||||
bsr.itemsReader = ip.itemsData.NewReader()
|
||||
bsr.lensReader = ip.lensData.NewReader()
|
||||
bsr.ph.CopyFrom(&mp.ph)
|
||||
bsr.indexReader = mp.indexData.NewReader()
|
||||
bsr.itemsReader = mp.itemsData.NewReader()
|
||||
bsr.lensReader = mp.lensData.NewReader()
|
||||
|
||||
if bsr.ph.itemsCount <= 0 {
|
||||
logger.Panicf("BUG: source inmemoryPart must contain at least a single item")
|
||||
|
|
|
|||
|
|
@ -63,17 +63,17 @@ func (bsw *blockStreamWriter) reset() {
|
|||
bsw.mrFirstItemCaught = false
|
||||
}
|
||||
|
||||
func (bsw *blockStreamWriter) InitFromInmemoryPart(ip *inmemoryPart) {
|
||||
func (bsw *blockStreamWriter) InitFromInmemoryPart(mp *inmemoryPart) {
|
||||
bsw.reset()
|
||||
|
||||
// Use the minimum compression level for in-memory blocks,
|
||||
// since they are going to be re-compressed during the merge into file-based blocks.
|
||||
bsw.compressLevel = -5 // See https://github.com/facebook/zstd/releases/tag/v1.3.4
|
||||
|
||||
bsw.metaindexWriter = &ip.metaindexData
|
||||
bsw.indexWriter = &ip.indexData
|
||||
bsw.itemsWriter = &ip.itemsData
|
||||
bsw.lensWriter = &ip.lensData
|
||||
bsw.metaindexWriter = &mp.metaindexData
|
||||
bsw.indexWriter = &mp.indexData
|
||||
bsw.itemsWriter = &mp.itemsData
|
||||
bsw.lensWriter = &mp.lensData
|
||||
}
|
||||
|
||||
// InitFromFilePart initializes bsw from a file-based part on the given path.
|
||||
|
|
|
|||
|
|
@ -137,25 +137,6 @@ func (ib *inmemoryBlock) Add(x []byte) bool {
|
|||
// It must fit CPU cache size, i.e. 64KB for the current CPUs.
|
||||
const maxInmemoryBlockSize = 64 * 1024
|
||||
|
||||
func (ib *inmemoryBlock) sort() {
|
||||
sort.Sort(ib)
|
||||
data := ib.data
|
||||
items := ib.items
|
||||
bb := bbPool.Get()
|
||||
b := bytesutil.ResizeNoCopyMayOverallocate(bb.B, len(data))
|
||||
b = b[:0]
|
||||
for i, it := range items {
|
||||
bLen := len(b)
|
||||
b = append(b, it.String(data)...)
|
||||
items[i] = Item{
|
||||
Start: uint32(bLen),
|
||||
End: uint32(len(b)),
|
||||
}
|
||||
}
|
||||
bb.B, ib.data = data, b
|
||||
bbPool.Put(bb)
|
||||
}
|
||||
|
||||
// storageBlock represents a block of data on the storage.
|
||||
type storageBlock struct {
|
||||
itemsData []byte
|
||||
|
|
@ -195,7 +176,7 @@ func (ib *inmemoryBlock) isSorted() bool {
|
|||
// - returns the marshal type used for the encoding.
|
||||
func (ib *inmemoryBlock) MarshalUnsortedData(sb *storageBlock, firstItemDst, commonPrefixDst []byte, compressLevel int) ([]byte, []byte, uint32, marshalType) {
|
||||
if !ib.isSorted() {
|
||||
ib.sort()
|
||||
sort.Sort(ib)
|
||||
}
|
||||
ib.updateCommonPrefix()
|
||||
return ib.marshalData(sb, firstItemDst, commonPrefixDst, compressLevel)
|
||||
|
|
@ -251,7 +232,7 @@ func (ib *inmemoryBlock) marshalData(sb *storageBlock, firstItemDst, commonPrefi
|
|||
firstItemDst = append(firstItemDst, firstItem...)
|
||||
commonPrefixDst = append(commonPrefixDst, ib.commonPrefix...)
|
||||
|
||||
if len(ib.data)-len(ib.commonPrefix)*len(ib.items) < 64 || len(ib.items) < 2 {
|
||||
if len(data)-len(ib.commonPrefix)*len(ib.items) < 64 || len(ib.items) < 2 {
|
||||
// Use plain encoding form small block, since it is cheaper.
|
||||
ib.marshalDataPlain(sb)
|
||||
return firstItemDst, commonPrefixDst, uint32(len(ib.items)), marshalTypePlain
|
||||
|
|
@ -302,7 +283,7 @@ func (ib *inmemoryBlock) marshalData(sb *storageBlock, firstItemDst, commonPrefi
|
|||
bbLens.B = bLens
|
||||
bbPool.Put(bbLens)
|
||||
|
||||
if float64(len(sb.itemsData)) > 0.9*float64(len(ib.data)-len(ib.commonPrefix)*len(ib.items)) {
|
||||
if float64(len(sb.itemsData)) > 0.9*float64(len(data)-len(ib.commonPrefix)*len(ib.items)) {
|
||||
// Bad compression rate. It is cheaper to use plain encoding.
|
||||
ib.marshalDataPlain(sb)
|
||||
return firstItemDst, commonPrefixDst, uint32(len(ib.items)), marshalTypePlain
|
||||
|
|
|
|||
|
|
@ -67,7 +67,7 @@ func TestInmemoryBlockSort(t *testing.T) {
|
|||
}
|
||||
|
||||
// Sort ib.
|
||||
ib.sort()
|
||||
sort.Sort(&ib)
|
||||
sort.Strings(items)
|
||||
|
||||
// Verify items are sorted.
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@ package mergeset
|
|||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"testing"
|
||||
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
|
||||
|
|
@ -16,7 +17,7 @@ func BenchmarkInmemoryBlockMarshal(b *testing.B) {
|
|||
b.Fatalf("cannot add more than %d items", i)
|
||||
}
|
||||
}
|
||||
ibSrc.sort()
|
||||
sort.Sort(&ibSrc)
|
||||
|
||||
b.ResetTimer()
|
||||
b.SetBytes(itemsCount)
|
||||
|
|
|
|||
|
|
@ -1,25 +1,18 @@
|
|||
package mergeset
|
||||
|
||||
import (
|
||||
"sync"
|
||||
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/cgroup"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/encoding"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/fs"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
|
||||
)
|
||||
|
||||
type inmemoryPart struct {
|
||||
ph partHeader
|
||||
sb storageBlock
|
||||
bh blockHeader
|
||||
mr metaindexRow
|
||||
|
||||
unpackedIndexBlockBuf []byte
|
||||
packedIndexBlockBuf []byte
|
||||
|
||||
unpackedMetaindexBuf []byte
|
||||
packedMetaindexBuf []byte
|
||||
|
||||
metaindexData bytesutil.ByteBuffer
|
||||
indexData bytesutil.ByteBuffer
|
||||
itemsData bytesutil.ByteBuffer
|
||||
|
|
@ -28,16 +21,9 @@ type inmemoryPart struct {
|
|||
|
||||
func (mp *inmemoryPart) Reset() {
|
||||
mp.ph.Reset()
|
||||
mp.sb.Reset()
|
||||
mp.bh.Reset()
|
||||
mp.mr.Reset()
|
||||
|
||||
mp.unpackedIndexBlockBuf = mp.unpackedIndexBlockBuf[:0]
|
||||
mp.packedIndexBlockBuf = mp.packedIndexBlockBuf[:0]
|
||||
|
||||
mp.unpackedMetaindexBuf = mp.unpackedMetaindexBuf[:0]
|
||||
mp.packedMetaindexBuf = mp.packedMetaindexBuf[:0]
|
||||
|
||||
mp.metaindexData.Reset()
|
||||
mp.indexData.Reset()
|
||||
mp.itemsData.Reset()
|
||||
|
|
@ -48,37 +34,47 @@ func (mp *inmemoryPart) Reset() {
|
|||
func (mp *inmemoryPart) Init(ib *inmemoryBlock) {
|
||||
mp.Reset()
|
||||
|
||||
// Re-use mp.itemsData and mp.lensData in sb.
|
||||
// This eliminates copying itemsData and lensData from sb to mp later.
|
||||
// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2247
|
||||
sb := &storageBlock{}
|
||||
sb.itemsData = mp.itemsData.B[:0]
|
||||
sb.lensData = mp.lensData.B[:0]
|
||||
|
||||
// Use the minimum possible compressLevel for compressing inmemoryPart,
|
||||
// since it will be merged into file part soon.
|
||||
compressLevel := 0
|
||||
mp.bh.firstItem, mp.bh.commonPrefix, mp.bh.itemsCount, mp.bh.marshalType = ib.MarshalUnsortedData(&mp.sb, mp.bh.firstItem[:0], mp.bh.commonPrefix[:0], compressLevel)
|
||||
// See https://github.com/facebook/zstd/releases/tag/v1.3.4 for details about negative compression level
|
||||
compressLevel := -5
|
||||
mp.bh.firstItem, mp.bh.commonPrefix, mp.bh.itemsCount, mp.bh.marshalType = ib.MarshalUnsortedData(sb, mp.bh.firstItem[:0], mp.bh.commonPrefix[:0], compressLevel)
|
||||
|
||||
mp.ph.itemsCount = uint64(len(ib.items))
|
||||
mp.ph.blocksCount = 1
|
||||
mp.ph.firstItem = append(mp.ph.firstItem[:0], ib.items[0].String(ib.data)...)
|
||||
mp.ph.lastItem = append(mp.ph.lastItem[:0], ib.items[len(ib.items)-1].String(ib.data)...)
|
||||
|
||||
fs.MustWriteData(&mp.itemsData, mp.sb.itemsData)
|
||||
mp.itemsData.B = sb.itemsData
|
||||
mp.bh.itemsBlockOffset = 0
|
||||
mp.bh.itemsBlockSize = uint32(len(mp.sb.itemsData))
|
||||
mp.bh.itemsBlockSize = uint32(len(mp.itemsData.B))
|
||||
|
||||
fs.MustWriteData(&mp.lensData, mp.sb.lensData)
|
||||
mp.lensData.B = sb.lensData
|
||||
mp.bh.lensBlockOffset = 0
|
||||
mp.bh.lensBlockSize = uint32(len(mp.sb.lensData))
|
||||
mp.bh.lensBlockSize = uint32(len(mp.lensData.B))
|
||||
|
||||
mp.unpackedIndexBlockBuf = mp.bh.Marshal(mp.unpackedIndexBlockBuf[:0])
|
||||
mp.packedIndexBlockBuf = encoding.CompressZSTDLevel(mp.packedIndexBlockBuf[:0], mp.unpackedIndexBlockBuf, 0)
|
||||
fs.MustWriteData(&mp.indexData, mp.packedIndexBlockBuf)
|
||||
bb := inmemoryPartBytePool.Get()
|
||||
bb.B = mp.bh.Marshal(bb.B[:0])
|
||||
mp.indexData.B = encoding.CompressZSTDLevel(mp.indexData.B[:0], bb.B, 0)
|
||||
|
||||
mp.mr.firstItem = append(mp.mr.firstItem[:0], mp.bh.firstItem...)
|
||||
mp.mr.blockHeadersCount = 1
|
||||
mp.mr.indexBlockOffset = 0
|
||||
mp.mr.indexBlockSize = uint32(len(mp.packedIndexBlockBuf))
|
||||
mp.unpackedMetaindexBuf = mp.mr.Marshal(mp.unpackedMetaindexBuf[:0])
|
||||
mp.packedMetaindexBuf = encoding.CompressZSTDLevel(mp.packedMetaindexBuf[:0], mp.unpackedMetaindexBuf, 0)
|
||||
fs.MustWriteData(&mp.metaindexData, mp.packedMetaindexBuf)
|
||||
mp.mr.indexBlockSize = uint32(len(mp.indexData.B))
|
||||
bb.B = mp.mr.Marshal(bb.B[:0])
|
||||
mp.metaindexData.B = encoding.CompressZSTDLevel(mp.metaindexData.B[:0], bb.B, 0)
|
||||
inmemoryPartBytePool.Put(bb)
|
||||
}
|
||||
|
||||
var inmemoryPartBytePool bytesutil.ByteBufferPool
|
||||
|
||||
// It is safe calling NewPart multiple times.
|
||||
// It is unsafe re-using mp while the returned part is in use.
|
||||
func (mp *inmemoryPart) NewPart() *part {
|
||||
|
|
@ -96,25 +92,16 @@ func (mp *inmemoryPart) size() uint64 {
|
|||
}
|
||||
|
||||
func getInmemoryPart() *inmemoryPart {
|
||||
select {
|
||||
case mp := <-mpPool:
|
||||
return mp
|
||||
default:
|
||||
v := inmemoryPartPool.Get()
|
||||
if v == nil {
|
||||
return &inmemoryPart{}
|
||||
}
|
||||
return v.(*inmemoryPart)
|
||||
}
|
||||
|
||||
func putInmemoryPart(mp *inmemoryPart) {
|
||||
mp.Reset()
|
||||
select {
|
||||
case mpPool <- mp:
|
||||
default:
|
||||
// Drop mp in order to reduce memory usage.
|
||||
}
|
||||
inmemoryPartPool.Put(mp)
|
||||
}
|
||||
|
||||
// Use chan instead of sync.Pool in order to reduce memory usage on systems with big number of CPU cores,
|
||||
// since sync.Pool maintains per-CPU pool of inmemoryPart objects.
|
||||
//
|
||||
// The inmemoryPart object size can exceed 64KB, so it is better to use chan instead of sync.Pool for reducing memory usage.
|
||||
var mpPool = make(chan *inmemoryPart, cgroup.AvailableCPUs())
|
||||
var inmemoryPartPool sync.Pool
|
||||
|
|
|
|||
|
|
@ -227,7 +227,9 @@ func (pw *partWrapper) decRef() {
|
|||
}
|
||||
|
||||
if pw.mp != nil {
|
||||
putInmemoryPart(pw.mp)
|
||||
// Do not return pw.mp to pool via putInmemoryPart(),
|
||||
// since pw.mp size may be too big compared to other entries stored in the pool.
|
||||
// This may result in increased memory usage because of high fragmentation.
|
||||
pw.mp = nil
|
||||
}
|
||||
pw.p.MustClose()
|
||||
|
|
@ -740,7 +742,10 @@ func (tb *Table) mergeInmemoryBlocks(ibs []*inmemoryBlock) *partWrapper {
|
|||
|
||||
// Prepare blockStreamWriter for destination part.
|
||||
bsw := getBlockStreamWriter()
|
||||
mpDst := getInmemoryPart()
|
||||
// Do not obtain mpDst via getInmemoryPart(), since its size
|
||||
// may be too big comparing to other entries in the pool.
|
||||
// This may result in increased memory usage because of high fragmentation.
|
||||
mpDst := &inmemoryPart{}
|
||||
bsw.InitFromInmemoryPart(mpDst)
|
||||
|
||||
// Merge parts.
|
||||
|
|
|
|||
|
|
@ -22,6 +22,7 @@ type RelabelConfig struct {
|
|||
Modulus uint64 `yaml:"modulus,omitempty"`
|
||||
Replacement *string `yaml:"replacement,omitempty"`
|
||||
Action string `yaml:"action,omitempty"`
|
||||
If *IfExpression `yaml:"if,omitempty"`
|
||||
}
|
||||
|
||||
// MultiLineRegex contains a regex, which can be split into multiple lines.
|
||||
|
|
@ -44,7 +45,7 @@ type MultiLineRegex struct {
|
|||
func (mlr *MultiLineRegex) UnmarshalYAML(f func(interface{}) error) error {
|
||||
var v interface{}
|
||||
if err := f(&v); err != nil {
|
||||
return err
|
||||
return fmt.Errorf("cannot parse multiline regex: %w", err)
|
||||
}
|
||||
s, err := stringValue(v)
|
||||
if err != nil {
|
||||
|
|
@ -224,11 +225,11 @@ func parseRelabelConfig(rc *RelabelConfig) (*parsedRelabelConfig, error) {
|
|||
return nil, fmt.Errorf("`source_labels` must contain at least two entries for `action=drop_if_equal`; got %q", sourceLabels)
|
||||
}
|
||||
case "keep":
|
||||
if len(sourceLabels) == 0 {
|
||||
if len(sourceLabels) == 0 && rc.If == nil {
|
||||
return nil, fmt.Errorf("missing `source_labels` for `action=keep`")
|
||||
}
|
||||
case "drop":
|
||||
if len(sourceLabels) == 0 {
|
||||
if len(sourceLabels) == 0 && rc.If == nil {
|
||||
return nil, fmt.Errorf("missing `source_labels` for `action=drop`")
|
||||
}
|
||||
case "hashmod":
|
||||
|
|
@ -242,7 +243,7 @@ func parseRelabelConfig(rc *RelabelConfig) (*parsedRelabelConfig, error) {
|
|||
return nil, fmt.Errorf("unexpected `modulus` for `action=hashmod`: %d; must be greater than 0", modulus)
|
||||
}
|
||||
case "keep_metrics":
|
||||
if rc.Regex == nil || rc.Regex.s == "" {
|
||||
if (rc.Regex == nil || rc.Regex.s == "") && rc.If == nil {
|
||||
return nil, fmt.Errorf("`regex` must be non-empty for `action=keep_metrics`")
|
||||
}
|
||||
if len(sourceLabels) > 0 {
|
||||
|
|
@ -251,7 +252,7 @@ func parseRelabelConfig(rc *RelabelConfig) (*parsedRelabelConfig, error) {
|
|||
sourceLabels = []string{"__name__"}
|
||||
action = "keep"
|
||||
case "drop_metrics":
|
||||
if rc.Regex == nil || rc.Regex.s == "" {
|
||||
if (rc.Regex == nil || rc.Regex.s == "") && rc.If == nil {
|
||||
return nil, fmt.Errorf("`regex` must be non-empty for `action=drop_metrics`")
|
||||
}
|
||||
if len(sourceLabels) > 0 {
|
||||
|
|
@ -274,6 +275,7 @@ func parseRelabelConfig(rc *RelabelConfig) (*parsedRelabelConfig, error) {
|
|||
Modulus: modulus,
|
||||
Replacement: replacement,
|
||||
Action: action,
|
||||
If: rc.If,
|
||||
|
||||
regexOriginal: regexOriginalCompiled,
|
||||
hasCaptureGroupInTargetLabel: strings.Contains(targetLabel, "$"),
|
||||
|
|
|
|||
169
lib/promrelabel/if_expression.go
Normal file
169
lib/promrelabel/if_expression.go
Normal file
|
|
@ -0,0 +1,169 @@
|
|||
package promrelabel
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"regexp"
|
||||
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
|
||||
"github.com/VictoriaMetrics/metricsql"
|
||||
)
|
||||
|
||||
// IfExpression represents `if` expression at RelabelConfig.
|
||||
//
|
||||
// The `if` expression can contain arbitrary PromQL-like label filters such as `metric_name{filters...}`
|
||||
type IfExpression struct {
|
||||
s string
|
||||
lfs []*labelFilter
|
||||
}
|
||||
|
||||
// UnmarshalYAML unmarshals ie from YAML passed to f.
|
||||
func (ie *IfExpression) UnmarshalYAML(f func(interface{}) error) error {
|
||||
var s string
|
||||
if err := f(&s); err != nil {
|
||||
return fmt.Errorf("cannot unmarshal `if` option: %w", err)
|
||||
}
|
||||
expr, err := metricsql.Parse(s)
|
||||
if err != nil {
|
||||
return fmt.Errorf("cannot parse `if` series selector: %w", err)
|
||||
}
|
||||
me, ok := expr.(*metricsql.MetricExpr)
|
||||
if !ok {
|
||||
return fmt.Errorf("expecting `if` series selector; got %q", expr.AppendString(nil))
|
||||
}
|
||||
lfs, err := metricExprToLabelFilters(me)
|
||||
if err != nil {
|
||||
return fmt.Errorf("cannot parse `if` filters: %w", err)
|
||||
}
|
||||
ie.s = s
|
||||
ie.lfs = lfs
|
||||
return nil
|
||||
}
|
||||
|
||||
// MarshalYAML marshals ie to YAML.
|
||||
func (ie *IfExpression) MarshalYAML() (interface{}, error) {
|
||||
return ie.s, nil
|
||||
}
|
||||
|
||||
// Match returns true if ie matches the given labels.
|
||||
func (ie *IfExpression) Match(labels []prompbmarshal.Label) bool {
|
||||
for _, lf := range ie.lfs {
|
||||
if !lf.match(labels) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func metricExprToLabelFilters(me *metricsql.MetricExpr) ([]*labelFilter, error) {
|
||||
lfs := make([]*labelFilter, len(me.LabelFilters))
|
||||
for i := range me.LabelFilters {
|
||||
lf, err := newLabelFilter(&me.LabelFilters[i])
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("cannot parse %s: %w", me.AppendString(nil), err)
|
||||
}
|
||||
lfs[i] = lf
|
||||
}
|
||||
return lfs, nil
|
||||
}
|
||||
|
||||
// labelFilter contains PromQL filter for `{label op "value"}`
|
||||
type labelFilter struct {
|
||||
label string
|
||||
op string
|
||||
value string
|
||||
|
||||
// re contains compiled regexp for `=~` and `!~` op.
|
||||
re *regexp.Regexp
|
||||
}
|
||||
|
||||
func newLabelFilter(mlf *metricsql.LabelFilter) (*labelFilter, error) {
|
||||
lf := &labelFilter{
|
||||
label: toCanonicalLabelName(mlf.Label),
|
||||
op: getFilterOp(mlf),
|
||||
value: mlf.Value,
|
||||
}
|
||||
if lf.op == "=~" || lf.op == "!~" {
|
||||
// PromQL regexps are anchored by default.
|
||||
// See https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors
|
||||
reString := "^(?:" + lf.value + ")$"
|
||||
re, err := regexp.Compile(reString)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("cannot parse regexp for %s: %w", mlf.AppendString(nil), err)
|
||||
}
|
||||
lf.re = re
|
||||
}
|
||||
return lf, nil
|
||||
}
|
||||
|
||||
func (lf *labelFilter) match(labels []prompbmarshal.Label) bool {
|
||||
switch lf.op {
|
||||
case "=":
|
||||
return lf.equalValue(labels)
|
||||
case "!=":
|
||||
return !lf.equalValue(labels)
|
||||
case "=~":
|
||||
return lf.equalRegexp(labels)
|
||||
case "!~":
|
||||
return !lf.equalRegexp(labels)
|
||||
default:
|
||||
logger.Panicf("BUG: unexpected operation for label filter: %s", lf.op)
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (lf *labelFilter) equalValue(labels []prompbmarshal.Label) bool {
|
||||
labelNameMatches := 0
|
||||
for _, label := range labels {
|
||||
if toCanonicalLabelName(label.Name) != lf.label {
|
||||
continue
|
||||
}
|
||||
labelNameMatches++
|
||||
if label.Value == lf.value {
|
||||
return true
|
||||
}
|
||||
}
|
||||
if labelNameMatches == 0 {
|
||||
// Special case for {non_existing_label=""}, which matches anything except of non-empty non_existing_label
|
||||
return lf.value == ""
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (lf *labelFilter) equalRegexp(labels []prompbmarshal.Label) bool {
|
||||
labelNameMatches := 0
|
||||
for _, label := range labels {
|
||||
if toCanonicalLabelName(label.Name) != lf.label {
|
||||
continue
|
||||
}
|
||||
labelNameMatches++
|
||||
if lf.re.MatchString(label.Value) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
if labelNameMatches == 0 {
|
||||
// Special case for {non_existing_label=~"something|"}, which matches empty non_existing_label
|
||||
return lf.re.MatchString("")
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func toCanonicalLabelName(labelName string) string {
|
||||
if labelName == "__name__" {
|
||||
return ""
|
||||
}
|
||||
return labelName
|
||||
}
|
||||
|
||||
func getFilterOp(mlf *metricsql.LabelFilter) string {
|
||||
if mlf.IsNegative {
|
||||
if mlf.IsRegexp {
|
||||
return "!~"
|
||||
}
|
||||
return "!="
|
||||
}
|
||||
if mlf.IsRegexp {
|
||||
return "=~"
|
||||
}
|
||||
return "="
|
||||
}
|
||||
162
lib/promrelabel/if_expression_test.go
Normal file
162
lib/promrelabel/if_expression_test.go
Normal file
|
|
@ -0,0 +1,162 @@
|
|||
package promrelabel
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"fmt"
|
||||
"testing"
|
||||
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/protoparser/prometheus"
|
||||
"gopkg.in/yaml.v2"
|
||||
)
|
||||
|
||||
func TestIfExpressionUnmarshalFailure(t *testing.T) {
|
||||
f := func(s string) {
|
||||
t.Helper()
|
||||
var ie IfExpression
|
||||
err := yaml.UnmarshalStrict([]byte(s), &ie)
|
||||
if err == nil {
|
||||
t.Fatalf("expecting non-nil error")
|
||||
}
|
||||
}
|
||||
f(`{`)
|
||||
f(`{x:y}`)
|
||||
f(`[]`)
|
||||
f(`"{"`)
|
||||
f(`'{'`)
|
||||
f(`foo{bar`)
|
||||
f(`foo{bar}`)
|
||||
f(`foo{bar=`)
|
||||
f(`foo{bar="`)
|
||||
f(`foo{bar='`)
|
||||
f(`foo{bar=~"("}`)
|
||||
f(`foo{bar!~"("}`)
|
||||
f(`foo{bar==aaa}`)
|
||||
f(`foo{bar=="b"}`)
|
||||
f(`'foo+bar'`)
|
||||
f(`'foo{bar=~"a[b"}'`)
|
||||
}
|
||||
|
||||
func TestIfExpressionUnmarshalSuccess(t *testing.T) {
|
||||
f := func(s string) {
|
||||
t.Helper()
|
||||
var ie IfExpression
|
||||
if err := yaml.UnmarshalStrict([]byte(s), &ie); err != nil {
|
||||
t.Fatalf("unexpected error during unmarshal: %s", err)
|
||||
}
|
||||
b, err := yaml.Marshal(&ie)
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error during marshal: %s", err)
|
||||
}
|
||||
b = bytes.TrimSpace(b)
|
||||
if string(b) != s {
|
||||
t.Fatalf("unexpected marshaled data;\ngot\n%s\nwant\n%s", b, s)
|
||||
}
|
||||
}
|
||||
f(`'{}'`)
|
||||
f(`foo`)
|
||||
f(`foo{bar="baz"}`)
|
||||
f(`'{a="b", c!="d", e=~"g", h!~"d"}'`)
|
||||
f(`foo{bar="zs",a=~"b|c"}`)
|
||||
}
|
||||
|
||||
func TestIfExpressionMatch(t *testing.T) {
|
||||
f := func(ifExpr, metricWithLabels string) {
|
||||
t.Helper()
|
||||
var ie IfExpression
|
||||
if err := yaml.UnmarshalStrict([]byte(ifExpr), &ie); err != nil {
|
||||
t.Fatalf("unexpected error during unmarshal: %s", err)
|
||||
}
|
||||
labels, err := parseMetricWithLabels(metricWithLabels)
|
||||
if err != nil {
|
||||
t.Fatalf("cannot parse %s: %s", metricWithLabels, err)
|
||||
}
|
||||
if !ie.Match(labels) {
|
||||
t.Fatalf("unexpected mismatch of ifExpr=%s for %s", ifExpr, metricWithLabels)
|
||||
}
|
||||
}
|
||||
f(`foo`, `foo`)
|
||||
f(`foo`, `foo{bar="baz",a="b"}`)
|
||||
f(`foo{bar="a"}`, `foo{bar="a"}`)
|
||||
f(`foo{bar="a"}`, `foo{x="y",bar="a",baz="b"}`)
|
||||
f(`'{a=~"x|abc",y!="z"}'`, `m{x="aa",a="abc"}`)
|
||||
f(`'{a=~"x|abc",y!="z"}'`, `m{x="aa",a="abc",y="qwe"}`)
|
||||
f(`'{__name__="foo"}'`, `foo{bar="baz"}`)
|
||||
f(`'{__name__=~"foo|bar"}'`, `bar`)
|
||||
f(`'{__name__!=""}'`, `foo`)
|
||||
f(`'{__name__!=""}'`, `bar{baz="aa",b="c"}`)
|
||||
f(`'{__name__!~"a.+"}'`, `bar{baz="aa",b="c"}`)
|
||||
f(`foo{a!~"a.+"}`, `foo{a="baa"}`)
|
||||
f(`'{foo=""}'`, `bar`)
|
||||
f(`'{foo!=""}'`, `aa{foo="b"}`)
|
||||
f(`'{foo=~".*"}'`, `abc`)
|
||||
f(`'{foo=~".*"}'`, `abc{foo="bar"}`)
|
||||
f(`'{foo!~".+"}'`, `abc`)
|
||||
f(`'{foo=~"bar|"}'`, `abc`)
|
||||
f(`'{foo=~"bar|"}'`, `abc{foo="bar"}`)
|
||||
f(`'{foo!~"bar|"}'`, `abc{foo="baz"}`)
|
||||
}
|
||||
|
||||
func TestIfExpressionMismatch(t *testing.T) {
|
||||
f := func(ifExpr, metricWithLabels string) {
|
||||
t.Helper()
|
||||
var ie IfExpression
|
||||
if err := yaml.UnmarshalStrict([]byte(ifExpr), &ie); err != nil {
|
||||
t.Fatalf("unexpected error during unmarshal: %s", err)
|
||||
}
|
||||
labels, err := parseMetricWithLabels(metricWithLabels)
|
||||
if err != nil {
|
||||
t.Fatalf("cannot parse %s: %s", metricWithLabels, err)
|
||||
}
|
||||
if ie.Match(labels) {
|
||||
t.Fatalf("unexpected match of ifExpr=%s for %s", ifExpr, metricWithLabels)
|
||||
}
|
||||
}
|
||||
f(`foo`, `bar`)
|
||||
f(`foo`, `a{foo="bar"}`)
|
||||
f(`foo{bar="a"}`, `foo`)
|
||||
f(`foo{bar="a"}`, `foo{bar="b"}`)
|
||||
f(`foo{bar="a"}`, `foo{baz="b",a="b"}`)
|
||||
f(`'{a=~"x|abc",y!="z"}'`, `m{x="aa",a="xabc"}`)
|
||||
f(`'{a=~"x|abc",y!="z"}'`, `m{x="aa",a="abc",y="z"}`)
|
||||
f(`'{__name__!~".+"}'`, `foo`)
|
||||
f(`'{a!~"a.+"}'`, `foo{a="abc"}`)
|
||||
f(`'{foo=""}'`, `bar{foo="aa"}`)
|
||||
f(`'{foo!=""}'`, `aa`)
|
||||
f(`'{foo=~".+"}'`, `abc`)
|
||||
f(`'{foo!~".+"}'`, `abc{foo="x"}`)
|
||||
f(`'{foo=~"bar|"}'`, `abc{foo="baz"}`)
|
||||
f(`'{foo!~"bar|"}'`, `abc`)
|
||||
f(`'{foo!~"bar|"}'`, `abc{foo="bar"}`)
|
||||
}
|
||||
|
||||
func parseMetricWithLabels(metricWithLabels string) ([]prompbmarshal.Label, error) {
|
||||
// add a value to metricWithLabels, so it could be parsed by prometheus protocol parser.
|
||||
s := metricWithLabels + " 123"
|
||||
var rows prometheus.Rows
|
||||
var err error
|
||||
rows.UnmarshalWithErrLogger(s, func(s string) {
|
||||
err = fmt.Errorf("error during metric parse: %s", s)
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(rows.Rows) != 1 {
|
||||
return nil, fmt.Errorf("unexpected number of rows parsed; got %d; want 1", len(rows.Rows))
|
||||
}
|
||||
r := rows.Rows[0]
|
||||
var lfs []prompbmarshal.Label
|
||||
if r.Metric != "" {
|
||||
lfs = append(lfs, prompbmarshal.Label{
|
||||
Name: "__name__",
|
||||
Value: r.Metric,
|
||||
})
|
||||
}
|
||||
for _, tag := range r.Tags {
|
||||
lfs = append(lfs, prompbmarshal.Label{
|
||||
Name: tag.Key,
|
||||
Value: tag.Value,
|
||||
})
|
||||
}
|
||||
return lfs, nil
|
||||
}
|
||||
|
|
@ -23,6 +23,7 @@ type parsedRelabelConfig struct {
|
|||
Modulus uint64
|
||||
Replacement string
|
||||
Action string
|
||||
If *IfExpression
|
||||
|
||||
regexOriginal *regexp.Regexp
|
||||
hasCaptureGroupInTargetLabel bool
|
||||
|
|
@ -137,8 +138,17 @@ func FinalizeLabels(dst, src []prompbmarshal.Label) []prompbmarshal.Label {
|
|||
// See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
|
||||
func (prc *parsedRelabelConfig) apply(labels []prompbmarshal.Label, labelsOffset int) []prompbmarshal.Label {
|
||||
src := labels[labelsOffset:]
|
||||
if prc.If != nil && !prc.If.Match(labels) {
|
||||
if prc.Action == "keep" {
|
||||
// Drop the target on `if` mismatch for `action: keep`
|
||||
return labels[:labelsOffset]
|
||||
}
|
||||
// Do not apply prc actions on `if` mismatch.
|
||||
return labels
|
||||
}
|
||||
switch prc.Action {
|
||||
case "replace":
|
||||
// Store `replacement` at `target_label` if the `regex` matches `source_labels` joined with `separator`
|
||||
bb := relabelBufPool.Get()
|
||||
bb.B = concatLabelValues(bb.B[:0], src, prc.SourceLabels, prc.Separator)
|
||||
if prc.Regex == defaultRegexForRelabelConfig && !prc.hasCaptureGroupInTargetLabel {
|
||||
|
|
@ -174,6 +184,8 @@ func (prc *parsedRelabelConfig) apply(labels []prompbmarshal.Label, labelsOffset
|
|||
relabelBufPool.Put(bb)
|
||||
return setLabelValue(labels, labelsOffset, nameStr, valueStr)
|
||||
case "replace_all":
|
||||
// Replace all the occurences of `regex` at `source_labels` joined with `separator` with the `replacement`
|
||||
// and store the result at `target_label`
|
||||
bb := relabelBufPool.Get()
|
||||
bb.B = concatLabelValues(bb.B[:0], src, prc.SourceLabels, prc.Separator)
|
||||
sourceStr := string(bb.B)
|
||||
|
|
@ -208,6 +220,15 @@ func (prc *parsedRelabelConfig) apply(labels []prompbmarshal.Label, labelsOffset
|
|||
}
|
||||
return labels
|
||||
case "keep":
|
||||
// Keep the target if `source_labels` joined with `separator` match the `regex`.
|
||||
if prc.Regex == defaultRegexForRelabelConfig {
|
||||
// Fast path for the case with `if` and without explicitly set `regex`:
|
||||
//
|
||||
// - action: keep
|
||||
// if: 'some{label=~"filters"}'
|
||||
//
|
||||
return labels
|
||||
}
|
||||
bb := relabelBufPool.Get()
|
||||
bb.B = concatLabelValues(bb.B[:0], src, prc.SourceLabels, prc.Separator)
|
||||
keep := prc.matchString(bytesutil.ToUnsafeString(bb.B))
|
||||
|
|
@ -217,6 +238,15 @@ func (prc *parsedRelabelConfig) apply(labels []prompbmarshal.Label, labelsOffset
|
|||
}
|
||||
return labels
|
||||
case "drop":
|
||||
// Drop the target if `source_labels` joined with `separator` don't match the `regex`.
|
||||
if prc.Regex == defaultRegexForRelabelConfig {
|
||||
// Fast path for the case with `if` and without explicitly set `regex`:
|
||||
//
|
||||
// - action: drop
|
||||
// if: 'some{label=~"filters"}'
|
||||
//
|
||||
return labels[:labelsOffset]
|
||||
}
|
||||
bb := relabelBufPool.Get()
|
||||
bb.B = concatLabelValues(bb.B[:0], src, prc.SourceLabels, prc.Separator)
|
||||
drop := prc.matchString(bytesutil.ToUnsafeString(bb.B))
|
||||
|
|
@ -226,6 +256,7 @@ func (prc *parsedRelabelConfig) apply(labels []prompbmarshal.Label, labelsOffset
|
|||
}
|
||||
return labels
|
||||
case "hashmod":
|
||||
// Calculate the `modulus` from the hash of `source_labels` joined with `separator` and store it at `target_label`
|
||||
bb := relabelBufPool.Get()
|
||||
bb.B = concatLabelValues(bb.B[:0], src, prc.SourceLabels, prc.Separator)
|
||||
h := xxhash.Sum64(bb.B) % prc.Modulus
|
||||
|
|
@ -233,6 +264,7 @@ func (prc *parsedRelabelConfig) apply(labels []prompbmarshal.Label, labelsOffset
|
|||
relabelBufPool.Put(bb)
|
||||
return setLabelValue(labels, labelsOffset, prc.TargetLabel, value)
|
||||
case "labelmap":
|
||||
// Replace label names with the `replacement` if they match `regex`
|
||||
for i := range src {
|
||||
label := &src[i]
|
||||
labelName, ok := prc.replaceFullString(label.Name, prc.Replacement, prc.hasCaptureGroupInReplacement)
|
||||
|
|
@ -242,12 +274,14 @@ func (prc *parsedRelabelConfig) apply(labels []prompbmarshal.Label, labelsOffset
|
|||
}
|
||||
return labels
|
||||
case "labelmap_all":
|
||||
// Replace all the occurences of `regex` at label names with `replacement`
|
||||
for i := range src {
|
||||
label := &src[i]
|
||||
label.Name, _ = prc.replaceStringSubmatches(label.Name, prc.Replacement, prc.hasCaptureGroupInReplacement)
|
||||
}
|
||||
return labels
|
||||
case "labeldrop":
|
||||
// Drop labels with names matching the `regex`
|
||||
dst := labels[:labelsOffset]
|
||||
for i := range src {
|
||||
label := &src[i]
|
||||
|
|
@ -257,6 +291,7 @@ func (prc *parsedRelabelConfig) apply(labels []prompbmarshal.Label, labelsOffset
|
|||
}
|
||||
return dst
|
||||
case "labelkeep":
|
||||
// Keep labels with names matching the `regex`
|
||||
dst := labels[:labelsOffset]
|
||||
for i := range src {
|
||||
label := &src[i]
|
||||
|
|
|
|||
|
|
@ -134,6 +134,25 @@ func TestApplyRelabelConfigs(t *testing.T) {
|
|||
source_labels: ["foo"]
|
||||
target_label: "bar"
|
||||
regex: ".+"
|
||||
`, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "xxx",
|
||||
Value: "yyy",
|
||||
},
|
||||
}, false, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "xxx",
|
||||
Value: "yyy",
|
||||
},
|
||||
})
|
||||
})
|
||||
t.Run("replace-if-miss", func(t *testing.T) {
|
||||
f(`
|
||||
- action: replace
|
||||
if: '{foo="bar"}'
|
||||
source_labels: ["xxx", "foo"]
|
||||
target_label: "bar"
|
||||
replacement: "a-$1-b"
|
||||
`, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "xxx",
|
||||
|
|
@ -152,6 +171,29 @@ func TestApplyRelabelConfigs(t *testing.T) {
|
|||
source_labels: ["xxx", "foo"]
|
||||
target_label: "bar"
|
||||
replacement: "a-$1-b"
|
||||
`, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "xxx",
|
||||
Value: "yyy",
|
||||
},
|
||||
}, false, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "bar",
|
||||
Value: "a-yyy;-b",
|
||||
},
|
||||
{
|
||||
Name: "xxx",
|
||||
Value: "yyy",
|
||||
},
|
||||
})
|
||||
})
|
||||
t.Run("replace-if-hit", func(t *testing.T) {
|
||||
f(`
|
||||
- action: replace
|
||||
if: '{xxx=~".y."}'
|
||||
source_labels: ["xxx", "foo"]
|
||||
target_label: "bar"
|
||||
replacement: "a-$1-b"
|
||||
`, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "xxx",
|
||||
|
|
@ -333,6 +375,26 @@ func TestApplyRelabelConfigs(t *testing.T) {
|
|||
},
|
||||
})
|
||||
})
|
||||
t.Run("replace_all-if-miss", func(t *testing.T) {
|
||||
f(`
|
||||
- action: replace_all
|
||||
if: 'foo'
|
||||
source_labels: ["xxx"]
|
||||
target_label: "xxx"
|
||||
regex: "-"
|
||||
replacement: "."
|
||||
`, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "xxx",
|
||||
Value: "a-b-c",
|
||||
},
|
||||
}, false, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "xxx",
|
||||
Value: "a-b-c",
|
||||
},
|
||||
})
|
||||
})
|
||||
t.Run("replace_all-hit", func(t *testing.T) {
|
||||
f(`
|
||||
- action: replace_all
|
||||
|
|
@ -340,6 +402,26 @@ func TestApplyRelabelConfigs(t *testing.T) {
|
|||
target_label: "xxx"
|
||||
regex: "-"
|
||||
replacement: "."
|
||||
`, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "xxx",
|
||||
Value: "a-b-c",
|
||||
},
|
||||
}, false, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "xxx",
|
||||
Value: "a.b.c",
|
||||
},
|
||||
})
|
||||
})
|
||||
t.Run("replace_all-if-hit", func(t *testing.T) {
|
||||
f(`
|
||||
- action: replace_all
|
||||
if: '{non_existing_label=~".*"}'
|
||||
source_labels: ["xxx"]
|
||||
target_label: "xxx"
|
||||
regex: "-"
|
||||
replacement: "."
|
||||
`, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "xxx",
|
||||
|
|
@ -530,6 +612,33 @@ func TestApplyRelabelConfigs(t *testing.T) {
|
|||
},
|
||||
}, true, []prompbmarshal.Label{})
|
||||
})
|
||||
t.Run("keep-if-miss", func(t *testing.T) {
|
||||
f(`
|
||||
- action: keep
|
||||
if: '{foo="bar"}'
|
||||
`, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "foo",
|
||||
Value: "yyy",
|
||||
},
|
||||
}, false, []prompbmarshal.Label{})
|
||||
})
|
||||
t.Run("keep-if-hit", func(t *testing.T) {
|
||||
f(`
|
||||
- action: keep
|
||||
if: '{foo="yyy"}'
|
||||
`, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "foo",
|
||||
Value: "yyy",
|
||||
},
|
||||
}, false, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "foo",
|
||||
Value: "yyy",
|
||||
},
|
||||
})
|
||||
})
|
||||
t.Run("keep-hit", func(t *testing.T) {
|
||||
f(`
|
||||
- action: keep
|
||||
|
|
@ -577,6 +686,33 @@ func TestApplyRelabelConfigs(t *testing.T) {
|
|||
},
|
||||
}, true, []prompbmarshal.Label{})
|
||||
})
|
||||
t.Run("keep_metrics-if-miss", func(t *testing.T) {
|
||||
f(`
|
||||
- action: keep_metrics
|
||||
if: 'bar'
|
||||
`, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "__name__",
|
||||
Value: "foo",
|
||||
},
|
||||
}, true, []prompbmarshal.Label{})
|
||||
})
|
||||
t.Run("keep_metrics-if-hit", func(t *testing.T) {
|
||||
f(`
|
||||
- action: keep_metrics
|
||||
if: 'foo'
|
||||
`, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "__name__",
|
||||
Value: "foo",
|
||||
},
|
||||
}, true, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "__name__",
|
||||
Value: "foo",
|
||||
},
|
||||
})
|
||||
})
|
||||
t.Run("keep_metrics-hit", func(t *testing.T) {
|
||||
f(`
|
||||
- action: keep_metrics
|
||||
|
|
@ -617,6 +753,33 @@ func TestApplyRelabelConfigs(t *testing.T) {
|
|||
},
|
||||
})
|
||||
})
|
||||
t.Run("drop-if-miss", func(t *testing.T) {
|
||||
f(`
|
||||
- action: drop
|
||||
if: '{foo="bar"}'
|
||||
`, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "foo",
|
||||
Value: "yyy",
|
||||
},
|
||||
}, true, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "foo",
|
||||
Value: "yyy",
|
||||
},
|
||||
})
|
||||
})
|
||||
t.Run("drop-if-hit", func(t *testing.T) {
|
||||
f(`
|
||||
- action: drop
|
||||
if: '{foo="yyy"}'
|
||||
`, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "foo",
|
||||
Value: "yyy",
|
||||
},
|
||||
}, true, []prompbmarshal.Label{})
|
||||
})
|
||||
t.Run("drop-hit", func(t *testing.T) {
|
||||
f(`
|
||||
- action: drop
|
||||
|
|
@ -659,6 +822,33 @@ func TestApplyRelabelConfigs(t *testing.T) {
|
|||
},
|
||||
})
|
||||
})
|
||||
t.Run("drop_metrics-if-miss", func(t *testing.T) {
|
||||
f(`
|
||||
- action: drop_metrics
|
||||
if: bar
|
||||
`, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "__name__",
|
||||
Value: "foo",
|
||||
},
|
||||
}, true, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "__name__",
|
||||
Value: "foo",
|
||||
},
|
||||
})
|
||||
})
|
||||
t.Run("drop_metrics-if-hit", func(t *testing.T) {
|
||||
f(`
|
||||
- action: drop_metrics
|
||||
if: foo
|
||||
`, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "__name__",
|
||||
Value: "foo",
|
||||
},
|
||||
}, true, []prompbmarshal.Label{})
|
||||
})
|
||||
t.Run("drop_metrics-hit", func(t *testing.T) {
|
||||
f(`
|
||||
- action: drop_metrics
|
||||
|
|
@ -694,6 +884,48 @@ func TestApplyRelabelConfigs(t *testing.T) {
|
|||
},
|
||||
})
|
||||
})
|
||||
t.Run("hashmod-if-miss", func(t *testing.T) {
|
||||
f(`
|
||||
- action: hashmod
|
||||
if: '{foo="bar"}'
|
||||
source_labels: [foo]
|
||||
target_label: aaa
|
||||
modulus: 123
|
||||
`, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "foo",
|
||||
Value: "yyy",
|
||||
},
|
||||
}, true, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "foo",
|
||||
Value: "yyy",
|
||||
},
|
||||
})
|
||||
})
|
||||
t.Run("hashmod-if-hit", func(t *testing.T) {
|
||||
f(`
|
||||
- action: hashmod
|
||||
if: '{foo="yyy"}'
|
||||
source_labels: [foo]
|
||||
target_label: aaa
|
||||
modulus: 123
|
||||
`, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "foo",
|
||||
Value: "yyy",
|
||||
},
|
||||
}, true, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "aaa",
|
||||
Value: "73",
|
||||
},
|
||||
{
|
||||
Name: "foo",
|
||||
Value: "yyy",
|
||||
},
|
||||
})
|
||||
})
|
||||
t.Run("hashmod-hit", func(t *testing.T) {
|
||||
f(`
|
||||
- action: hashmod
|
||||
|
|
@ -716,6 +948,62 @@ func TestApplyRelabelConfigs(t *testing.T) {
|
|||
},
|
||||
})
|
||||
})
|
||||
t.Run("labelmap-copy-label-if-miss", func(t *testing.T) {
|
||||
f(`
|
||||
- action: labelmap
|
||||
if: '{foo="yyy",foobar="aab"}'
|
||||
regex: "foo"
|
||||
replacement: "bar"
|
||||
`, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "foo",
|
||||
Value: "yyy",
|
||||
},
|
||||
{
|
||||
Name: "foobar",
|
||||
Value: "aaa",
|
||||
},
|
||||
}, true, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "foo",
|
||||
Value: "yyy",
|
||||
},
|
||||
{
|
||||
Name: "foobar",
|
||||
Value: "aaa",
|
||||
},
|
||||
})
|
||||
})
|
||||
t.Run("labelmap-copy-label-if-hit", func(t *testing.T) {
|
||||
f(`
|
||||
- action: labelmap
|
||||
if: '{foo="yyy",foobar="aaa"}'
|
||||
regex: "foo"
|
||||
replacement: "bar"
|
||||
`, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "foo",
|
||||
Value: "yyy",
|
||||
},
|
||||
{
|
||||
Name: "foobar",
|
||||
Value: "aaa",
|
||||
},
|
||||
}, true, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "bar",
|
||||
Value: "yyy",
|
||||
},
|
||||
{
|
||||
Name: "foo",
|
||||
Value: "yyy",
|
||||
},
|
||||
{
|
||||
Name: "foobar",
|
||||
Value: "aaa",
|
||||
},
|
||||
})
|
||||
})
|
||||
t.Run("labelmap-copy-label", func(t *testing.T) {
|
||||
f(`
|
||||
- action: labelmap
|
||||
|
|
@ -830,6 +1118,58 @@ func TestApplyRelabelConfigs(t *testing.T) {
|
|||
},
|
||||
})
|
||||
})
|
||||
t.Run("labelmap_all-if-miss", func(t *testing.T) {
|
||||
f(`
|
||||
- action: labelmap_all
|
||||
if: foobar
|
||||
regex: "\\."
|
||||
replacement: "-"
|
||||
`, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "foo.bar.baz",
|
||||
Value: "yyy",
|
||||
},
|
||||
{
|
||||
Name: "foobar",
|
||||
Value: "aaa",
|
||||
},
|
||||
}, true, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "foo.bar.baz",
|
||||
Value: "yyy",
|
||||
},
|
||||
{
|
||||
Name: "foobar",
|
||||
Value: "aaa",
|
||||
},
|
||||
})
|
||||
})
|
||||
t.Run("labelmap_all-if-hit", func(t *testing.T) {
|
||||
f(`
|
||||
- action: labelmap_all
|
||||
if: '{foo.bar.baz="yyy"}'
|
||||
regex: "\\."
|
||||
replacement: "-"
|
||||
`, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "foo.bar.baz",
|
||||
Value: "yyy",
|
||||
},
|
||||
{
|
||||
Name: "foobar",
|
||||
Value: "aaa",
|
||||
},
|
||||
}, true, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "foo-bar-baz",
|
||||
Value: "yyy",
|
||||
},
|
||||
{
|
||||
Name: "foobar",
|
||||
Value: "aaa",
|
||||
},
|
||||
})
|
||||
})
|
||||
t.Run("labelmap_all", func(t *testing.T) {
|
||||
f(`
|
||||
- action: labelmap_all
|
||||
|
|
@ -895,6 +1235,66 @@ func TestApplyRelabelConfigs(t *testing.T) {
|
|||
Value: "bbb",
|
||||
},
|
||||
})
|
||||
// if-miss
|
||||
f(`
|
||||
- action: labeldrop
|
||||
if: foo
|
||||
regex: dropme
|
||||
`, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "xxx",
|
||||
Value: "yyy",
|
||||
},
|
||||
{
|
||||
Name: "dropme",
|
||||
Value: "aaa",
|
||||
},
|
||||
{
|
||||
Name: "foo",
|
||||
Value: "bar",
|
||||
},
|
||||
}, false, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "dropme",
|
||||
Value: "aaa",
|
||||
},
|
||||
{
|
||||
Name: "foo",
|
||||
Value: "bar",
|
||||
},
|
||||
{
|
||||
Name: "xxx",
|
||||
Value: "yyy",
|
||||
},
|
||||
})
|
||||
// if-hit
|
||||
f(`
|
||||
- action: labeldrop
|
||||
if: '{xxx="yyy"}'
|
||||
regex: dropme
|
||||
`, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "xxx",
|
||||
Value: "yyy",
|
||||
},
|
||||
{
|
||||
Name: "dropme",
|
||||
Value: "aaa",
|
||||
},
|
||||
{
|
||||
Name: "foo",
|
||||
Value: "bar",
|
||||
},
|
||||
}, false, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "foo",
|
||||
Value: "bar",
|
||||
},
|
||||
{
|
||||
Name: "xxx",
|
||||
Value: "yyy",
|
||||
},
|
||||
})
|
||||
f(`
|
||||
- action: labeldrop
|
||||
regex: dropme
|
||||
|
|
@ -1059,6 +1459,62 @@ func TestApplyRelabelConfigs(t *testing.T) {
|
|||
Value: "aaa",
|
||||
},
|
||||
})
|
||||
// if-miss
|
||||
f(`
|
||||
- action: labelkeep
|
||||
if: '{aaaa="awefx"}'
|
||||
regex: keepme
|
||||
`, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "keepme",
|
||||
Value: "aaa",
|
||||
},
|
||||
{
|
||||
Name: "aaaa",
|
||||
Value: "awef",
|
||||
},
|
||||
{
|
||||
Name: "keepme-aaa",
|
||||
Value: "234",
|
||||
},
|
||||
}, false, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "aaaa",
|
||||
Value: "awef",
|
||||
},
|
||||
{
|
||||
Name: "keepme",
|
||||
Value: "aaa",
|
||||
},
|
||||
{
|
||||
Name: "keepme-aaa",
|
||||
Value: "234",
|
||||
},
|
||||
})
|
||||
// if-hit
|
||||
f(`
|
||||
- action: labelkeep
|
||||
if: '{aaaa="awef"}'
|
||||
regex: keepme
|
||||
`, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "keepme",
|
||||
Value: "aaa",
|
||||
},
|
||||
{
|
||||
Name: "aaaa",
|
||||
Value: "awef",
|
||||
},
|
||||
{
|
||||
Name: "keepme-aaa",
|
||||
Value: "234",
|
||||
},
|
||||
}, false, []prompbmarshal.Label{
|
||||
{
|
||||
Name: "keepme",
|
||||
Value: "aaa",
|
||||
},
|
||||
})
|
||||
f(`
|
||||
- action: labelkeep
|
||||
regex: keepme
|
||||
|
|
|
|||
|
|
@ -142,9 +142,9 @@ func openIndexDB(path string, s *Storage, rotationTimestamp uint64) (*indexDB, e
|
|||
tb: tb,
|
||||
name: name,
|
||||
|
||||
tagFiltersCache: workingsetcache.New(mem/32, time.Hour),
|
||||
tagFiltersCache: workingsetcache.New(mem / 32),
|
||||
s: s,
|
||||
loopsPerDateTagFilterCache: workingsetcache.New(mem/128, time.Hour),
|
||||
loopsPerDateTagFilterCache: workingsetcache.New(mem / 128),
|
||||
}
|
||||
return db, nil
|
||||
}
|
||||
|
|
@ -608,6 +608,11 @@ func (db *indexDB) getOrCreateTSID(dst *TSID, metricName []byte, mn *MetricName)
|
|||
|
||||
func generateTSID(dst *TSID, mn *MetricName) {
|
||||
dst.MetricGroupID = xxhash.Sum64(mn.MetricGroup)
|
||||
// Assume that the job-like metric is put at mn.Tags[0], while instance-like metric is put at mn.Tags[1]
|
||||
// This assumption is true because mn.Tags must be sorted with mn.sortTags() before calling generateTSID() function.
|
||||
// This allows grouping data blocks for the same (job, instance) close to each other on disk.
|
||||
// This reduces disk seeks and disk read IO when data blocks are read from disk for the same job and/or instance.
|
||||
// For example, data blocks for time series matching `process_resident_memory_bytes{job="vmstorage"}` are physically adjancent on disk.
|
||||
if len(mn.Tags) > 0 {
|
||||
dst.JobID = uint32(xxhash.Sum64(mn.Tags[0].Value))
|
||||
}
|
||||
|
|
@ -1868,7 +1873,12 @@ func (is *indexSearch) containsTimeRange(tr TimeRange) (bool, error) {
|
|||
ts := &is.ts
|
||||
kb := &is.kb
|
||||
|
||||
// Verify whether the maximum date in `ts` covers tr.MinTimestamp.
|
||||
// Verify whether the tr.MinTimestamp is included into `ts` or is smaller than the minimum date stored in `ts`.
|
||||
// Do not check whether tr.MaxTimestamp is included into `ts` or is bigger than the max date stored in `ts` for performance reasons.
|
||||
// This means that containsTimeRange() can return true if `tr` is located below the min date stored in `ts`.
|
||||
// This is OK, since this case isn't encountered too much in practice.
|
||||
// The main practical case allows skipping searching in prev indexdb (`ts`) when `tr`
|
||||
// is located above the max date stored there.
|
||||
minDate := uint64(tr.MinTimestamp) / msecPerDay
|
||||
kb.B = is.marshalCommonPrefix(kb.B[:0], nsPrefixDateToMetricID)
|
||||
prefix := kb.B
|
||||
|
|
|
|||
|
|
@ -1851,9 +1851,9 @@ func newTestStorage() *Storage {
|
|||
s := &Storage{
|
||||
cachePath: "test-storage-cache",
|
||||
|
||||
metricIDCache: workingsetcache.New(1234, time.Hour),
|
||||
metricNameCache: workingsetcache.New(1234, time.Hour),
|
||||
tsidCache: workingsetcache.New(1234, time.Hour),
|
||||
metricIDCache: workingsetcache.New(1234),
|
||||
metricNameCache: workingsetcache.New(1234),
|
||||
tsidCache: workingsetcache.New(1234),
|
||||
}
|
||||
s.setDeletedMetricIDs(&uint64set.Set{})
|
||||
return s
|
||||
|
|
|
|||
|
|
@ -614,6 +614,12 @@ func unmarshalBytesFast(src []byte) ([]byte, []byte, error) {
|
|||
|
||||
// sortTags sorts tags in mn to canonical form needed for storing in the index.
|
||||
//
|
||||
// The sortTags tries moving job-like tag to mn.Tags[0], while instance-like tag to mn.Tags[1].
|
||||
// See commonTagKeys list for job-like and instance-like tags.
|
||||
// This guarantees that indexdb entries for the same (job, instance) are located
|
||||
// close to each other on disk. This reduces disk seeks and disk read IO when metrics
|
||||
// for a particular job and/or instance are read from the disk.
|
||||
//
|
||||
// The function also de-duplicates tags with identical keys in mn. The last tag value
|
||||
// for duplicate tags wins.
|
||||
//
|
||||
|
|
|
|||
|
|
@ -1255,6 +1255,13 @@ func (pt *partition) mergeParts(pws []*partWrapper, stopCh <-chan struct{}) erro
|
|||
|
||||
func getCompressLevelForRowsCount(rowsCount, blocksCount uint64) int {
|
||||
avgRowsPerBlock := rowsCount / blocksCount
|
||||
// See https://github.com/facebook/zstd/releases/tag/v1.3.4 about negative compression levels.
|
||||
if avgRowsPerBlock <= 10 {
|
||||
return -5
|
||||
}
|
||||
if avgRowsPerBlock <= 50 {
|
||||
return -2
|
||||
}
|
||||
if avgRowsPerBlock <= 200 {
|
||||
return -1
|
||||
}
|
||||
|
|
|
|||
|
|
@ -993,7 +993,7 @@ func (s *Storage) mustLoadCache(info, name string, sizeBytes int) *workingsetcac
|
|||
path := s.cachePath + "/" + name
|
||||
logger.Infof("loading %s cache from %q...", info, path)
|
||||
startTime := time.Now()
|
||||
c := workingsetcache.Load(path, sizeBytes, time.Hour)
|
||||
c := workingsetcache.Load(path, sizeBytes)
|
||||
var cs fastcache.Stats
|
||||
c.UpdateStats(&cs)
|
||||
logger.Infof("loaded %s cache from %q in %.3f seconds; entriesCount: %d; sizeBytes: %d",
|
||||
|
|
|
|||
|
|
@ -49,7 +49,10 @@ func convertToCompositeTagFilters(tfs *TagFilters) []*TagFilters {
|
|||
hasPositiveFilter = true
|
||||
}
|
||||
}
|
||||
if len(names) == 0 {
|
||||
// If tfs have no filters on __name__ or have no non-negative filters,
|
||||
// then it is impossible to construct composite tag filter.
|
||||
// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2238
|
||||
if len(names) == 0 || !hasPositiveFilter {
|
||||
atomic.AddUint64(&compositeFilterMissingConversions, 1)
|
||||
return []*TagFilters{tfs}
|
||||
}
|
||||
|
|
@ -61,7 +64,7 @@ func convertToCompositeTagFilters(tfs *TagFilters) []*TagFilters {
|
|||
tfsNew := make([]tagFilter, 0, len(tfs.tfs))
|
||||
for _, tf := range tfs.tfs {
|
||||
if len(tf.key) == 0 {
|
||||
if !hasPositiveFilter || tf.isNegative {
|
||||
if tf.isNegative {
|
||||
// Negative filters on metric name cannot be used for building composite filter, so leave them as is.
|
||||
tfsNew = append(tfsNew, tf)
|
||||
continue
|
||||
|
|
|
|||
|
|
@ -185,7 +185,7 @@ func TestConvertToCompositeTagFilters(t *testing.T) {
|
|||
IsRegexp: false,
|
||||
},
|
||||
{
|
||||
Key: []byte("\xfe\x03barfoo"),
|
||||
Key: []byte("foo"),
|
||||
Value: []byte("abc"),
|
||||
IsNegative: true,
|
||||
IsRegexp: false,
|
||||
|
|
@ -588,21 +588,7 @@ func TestConvertToCompositeTagFilters(t *testing.T) {
|
|||
IsRegexp: true,
|
||||
},
|
||||
{
|
||||
Key: []byte("\xfe\x03barfoo"),
|
||||
Value: []byte("abc"),
|
||||
IsNegative: true,
|
||||
IsRegexp: false,
|
||||
},
|
||||
},
|
||||
{
|
||||
{
|
||||
Key: nil,
|
||||
Value: []byte("bar|foo"),
|
||||
IsNegative: false,
|
||||
IsRegexp: true,
|
||||
},
|
||||
{
|
||||
Key: []byte("\xfe\x03foofoo"),
|
||||
Key: []byte("foo"),
|
||||
Value: []byte("abc"),
|
||||
IsNegative: true,
|
||||
IsRegexp: false,
|
||||
|
|
|
|||
|
|
@ -17,6 +17,8 @@ const (
|
|||
whole = 2
|
||||
)
|
||||
|
||||
const defaultExpireDuration = 20 * time.Minute
|
||||
|
||||
// Cache is a cache for working set entries.
|
||||
//
|
||||
// The cache evicts inactive entries after the given expireDuration.
|
||||
|
|
@ -48,10 +50,18 @@ type Cache struct {
|
|||
}
|
||||
|
||||
// Load loads the cache from filePath and limits its size to maxBytes
|
||||
// and evicts inactive entries in 20 minutes.
|
||||
//
|
||||
// Stop must be called on the returned cache when it is no longer needed.
|
||||
func Load(filePath string, maxBytes int) *Cache {
|
||||
return LoadWithExpire(filePath, maxBytes, defaultExpireDuration)
|
||||
}
|
||||
|
||||
// LoadWithExpire loads the cache from filePath and limits its size to maxBytes
|
||||
// and evicts inactive entires after expireDuration.
|
||||
//
|
||||
// Stop must be called on the returned cache when it is no longer needed.
|
||||
func Load(filePath string, maxBytes int, expireDuration time.Duration) *Cache {
|
||||
func LoadWithExpire(filePath string, maxBytes int, expireDuration time.Duration) *Cache {
|
||||
curr := fastcache.LoadFromFileOrNew(filePath, maxBytes)
|
||||
var cs fastcache.Stats
|
||||
curr.UpdateStats(&cs)
|
||||
|
|
@ -60,8 +70,10 @@ func Load(filePath string, maxBytes int, expireDuration time.Duration) *Cache {
|
|||
// The cache couldn't be loaded with maxBytes size.
|
||||
// This may mean that the cache is split into curr and prev caches.
|
||||
// Try loading it again with maxBytes / 2 size.
|
||||
curr := fastcache.LoadFromFileOrNew(filePath, maxBytes/2)
|
||||
prev := fastcache.New(maxBytes / 2)
|
||||
// Put the loaded cache into `prev` instead of `curr`
|
||||
// in order to limit the growth of the cache for the current period of time.
|
||||
prev := fastcache.LoadFromFileOrNew(filePath, maxBytes/2)
|
||||
curr := fastcache.New(maxBytes / 2)
|
||||
c := newCacheInternal(curr, prev, split, maxBytes)
|
||||
c.runWatchers(expireDuration)
|
||||
return c
|
||||
|
|
@ -74,11 +86,18 @@ func Load(filePath string, maxBytes int, expireDuration time.Duration) *Cache {
|
|||
return newCacheInternal(curr, prev, whole, maxBytes)
|
||||
}
|
||||
|
||||
// New creates new cache with the given maxBytes capacity and the given expireDuration
|
||||
// New creates new cache with the given maxBytes capacity.
|
||||
//
|
||||
// Stop must be called on the returned cache when it is no longer needed.
|
||||
func New(maxBytes int) *Cache {
|
||||
return NewWithExpire(maxBytes, defaultExpireDuration)
|
||||
}
|
||||
|
||||
// NewWithExpire creates new cache with the given maxBytes capacity and the given expireDuration
|
||||
// for inactive entries.
|
||||
//
|
||||
// Stop must be called on the returned cache when it is no longer needed.
|
||||
func New(maxBytes int, expireDuration time.Duration) *Cache {
|
||||
func NewWithExpire(maxBytes int, expireDuration time.Duration) *Cache {
|
||||
curr := fastcache.New(maxBytes / 2)
|
||||
prev := fastcache.New(1024)
|
||||
c := newCacheInternal(curr, prev, split, maxBytes)
|
||||
|
|
@ -153,6 +172,9 @@ func (c *Cache) cacheSizeWatcher() {
|
|||
return
|
||||
case <-t.C:
|
||||
}
|
||||
if c.loadMode() != split {
|
||||
continue
|
||||
}
|
||||
var cs fastcache.Stats
|
||||
curr := c.curr.Load().(*fastcache.Cache)
|
||||
curr.UpdateStats(&cs)
|
||||
|
|
@ -169,10 +191,10 @@ func (c *Cache) cacheSizeWatcher() {
|
|||
// Do this in the following steps:
|
||||
// 1) switch to mode=switching
|
||||
// 2) move curr cache to prev
|
||||
// 3) create curr with the double size
|
||||
// 4) wait until curr size exceeds maxBytesSize, i.e. it is populated with new data
|
||||
// 3) create curr cache with doubled size
|
||||
// 4) wait until curr cache size exceeds maxBytesSize, i.e. it is populated with new data
|
||||
// 5) switch to mode=whole
|
||||
// 6) drop prev
|
||||
// 6) drop prev cache
|
||||
|
||||
c.mu.Lock()
|
||||
c.setMode(switching)
|
||||
|
|
|
|||
7
vendor/cloud.google.com/go/iam/CHANGES.md
generated
vendored
7
vendor/cloud.google.com/go/iam/CHANGES.md
generated
vendored
|
|
@ -1,5 +1,12 @@
|
|||
# Changes
|
||||
|
||||
## [0.3.0](https://github.com/googleapis/google-cloud-go/compare/iam/v0.2.0...iam/v0.3.0) (2022-02-23)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* **iam:** set versionClient to module version ([55f0d92](https://github.com/googleapis/google-cloud-go/commit/55f0d92bf112f14b024b4ab0076c9875a17423c9))
|
||||
|
||||
## [0.2.0](https://github.com/googleapis/google-cloud-go/compare/iam/v0.1.1...iam/v0.2.0) (2022-02-14)
|
||||
|
||||
|
||||
|
|
|
|||
3
vendor/github.com/aws/aws-sdk-go/aws/config.go
generated
vendored
3
vendor/github.com/aws/aws-sdk-go/aws/config.go
generated
vendored
|
|
@ -170,6 +170,9 @@ type Config struct {
|
|||
//
|
||||
// For example S3's X-Amz-Meta prefixed header will be unmarshaled to lower case
|
||||
// Metadata member's map keys. The value of the header in the map is unaffected.
|
||||
//
|
||||
// The AWS SDK for Go v2, uses lower case header maps by default. The v1
|
||||
// SDK provides this opt-in for this option, for backwards compatibility.
|
||||
LowerCaseHeaderMaps *bool
|
||||
|
||||
// Set this to `true` to disable the EC2Metadata client from overriding the
|
||||
|
|
|
|||
276
vendor/github.com/aws/aws-sdk-go/aws/endpoints/defaults.go
generated
vendored
276
vendor/github.com/aws/aws-sdk-go/aws/endpoints/defaults.go
generated
vendored
|
|
@ -814,6 +814,61 @@ var awsPartition = partition{
|
|||
}: endpoint{},
|
||||
},
|
||||
},
|
||||
"amplifyuibuilder": service{
|
||||
Endpoints: serviceEndpoints{
|
||||
endpointKey{
|
||||
Region: "ap-northeast-1",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "ap-northeast-2",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "ap-south-1",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "ap-southeast-1",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "ap-southeast-2",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "ca-central-1",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "eu-central-1",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "eu-north-1",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "eu-west-1",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "eu-west-2",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "eu-west-3",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "me-south-1",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "sa-east-1",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "us-east-1",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "us-east-2",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "us-west-1",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "us-west-2",
|
||||
}: endpoint{},
|
||||
},
|
||||
},
|
||||
"api.detective": service{
|
||||
Defaults: endpointDefaults{
|
||||
defaultKey{}: endpoint{
|
||||
|
|
@ -1669,6 +1724,147 @@ var awsPartition = partition{
|
|||
},
|
||||
},
|
||||
},
|
||||
"api.tunneling.iot": service{
|
||||
Defaults: endpointDefaults{
|
||||
defaultKey{}: endpoint{},
|
||||
defaultKey{
|
||||
Variant: fipsVariant,
|
||||
}: endpoint{
|
||||
Hostname: "api.tunneling.iot-fips.{region}.{dnsSuffix}",
|
||||
},
|
||||
},
|
||||
Endpoints: serviceEndpoints{
|
||||
endpointKey{
|
||||
Region: "ap-east-1",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "ap-northeast-1",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "ap-northeast-2",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "ap-south-1",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "ap-southeast-1",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "ap-southeast-2",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "ca-central-1",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "ca-central-1",
|
||||
Variant: fipsVariant,
|
||||
}: endpoint{
|
||||
Hostname: "api.tunneling.iot-fips.ca-central-1.amazonaws.com",
|
||||
},
|
||||
endpointKey{
|
||||
Region: "eu-central-1",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "eu-north-1",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "eu-west-1",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "eu-west-2",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "eu-west-3",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "fips-ca-central-1",
|
||||
}: endpoint{
|
||||
Hostname: "api.tunneling.iot-fips.ca-central-1.amazonaws.com",
|
||||
CredentialScope: credentialScope{
|
||||
Region: "ca-central-1",
|
||||
},
|
||||
Deprecated: boxedTrue,
|
||||
},
|
||||
endpointKey{
|
||||
Region: "fips-us-east-1",
|
||||
}: endpoint{
|
||||
Hostname: "api.tunneling.iot-fips.us-east-1.amazonaws.com",
|
||||
CredentialScope: credentialScope{
|
||||
Region: "us-east-1",
|
||||
},
|
||||
Deprecated: boxedTrue,
|
||||
},
|
||||
endpointKey{
|
||||
Region: "fips-us-east-2",
|
||||
}: endpoint{
|
||||
Hostname: "api.tunneling.iot-fips.us-east-2.amazonaws.com",
|
||||
CredentialScope: credentialScope{
|
||||
Region: "us-east-2",
|
||||
},
|
||||
Deprecated: boxedTrue,
|
||||
},
|
||||
endpointKey{
|
||||
Region: "fips-us-west-1",
|
||||
}: endpoint{
|
||||
Hostname: "api.tunneling.iot-fips.us-west-1.amazonaws.com",
|
||||
CredentialScope: credentialScope{
|
||||
Region: "us-west-1",
|
||||
},
|
||||
Deprecated: boxedTrue,
|
||||
},
|
||||
endpointKey{
|
||||
Region: "fips-us-west-2",
|
||||
}: endpoint{
|
||||
Hostname: "api.tunneling.iot-fips.us-west-2.amazonaws.com",
|
||||
CredentialScope: credentialScope{
|
||||
Region: "us-west-2",
|
||||
},
|
||||
Deprecated: boxedTrue,
|
||||
},
|
||||
endpointKey{
|
||||
Region: "me-south-1",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "sa-east-1",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "us-east-1",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "us-east-1",
|
||||
Variant: fipsVariant,
|
||||
}: endpoint{
|
||||
Hostname: "api.tunneling.iot-fips.us-east-1.amazonaws.com",
|
||||
},
|
||||
endpointKey{
|
||||
Region: "us-east-2",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "us-east-2",
|
||||
Variant: fipsVariant,
|
||||
}: endpoint{
|
||||
Hostname: "api.tunneling.iot-fips.us-east-2.amazonaws.com",
|
||||
},
|
||||
endpointKey{
|
||||
Region: "us-west-1",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "us-west-1",
|
||||
Variant: fipsVariant,
|
||||
}: endpoint{
|
||||
Hostname: "api.tunneling.iot-fips.us-west-1.amazonaws.com",
|
||||
},
|
||||
endpointKey{
|
||||
Region: "us-west-2",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "us-west-2",
|
||||
Variant: fipsVariant,
|
||||
}: endpoint{
|
||||
Hostname: "api.tunneling.iot-fips.us-west-2.amazonaws.com",
|
||||
},
|
||||
},
|
||||
},
|
||||
"apigateway": service{
|
||||
Endpoints: serviceEndpoints{
|
||||
endpointKey{
|
||||
|
|
@ -2810,6 +3006,9 @@ var awsPartition = partition{
|
|||
},
|
||||
"braket": service{
|
||||
Endpoints: serviceEndpoints{
|
||||
endpointKey{
|
||||
Region: "eu-west-2",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "us-east-1",
|
||||
}: endpoint{},
|
||||
|
|
@ -13574,6 +13773,9 @@ var awsPartition = partition{
|
|||
endpointKey{
|
||||
Region: "ap-southeast-2",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "ap-southeast-3",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "ca-central-1",
|
||||
}: endpoint{},
|
||||
|
|
@ -21963,6 +22165,16 @@ var awscnPartition = partition{
|
|||
}: endpoint{},
|
||||
},
|
||||
},
|
||||
"api.tunneling.iot": service{
|
||||
Endpoints: serviceEndpoints{
|
||||
endpointKey{
|
||||
Region: "cn-north-1",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "cn-northwest-1",
|
||||
}: endpoint{},
|
||||
},
|
||||
},
|
||||
"apigateway": service{
|
||||
Endpoints: serviceEndpoints{
|
||||
endpointKey{
|
||||
|
|
@ -23471,6 +23683,14 @@ var awsusgovPartition = partition{
|
|||
},
|
||||
},
|
||||
"acm": service{
|
||||
Defaults: endpointDefaults{
|
||||
defaultKey{}: endpoint{},
|
||||
defaultKey{
|
||||
Variant: fipsVariant,
|
||||
}: endpoint{
|
||||
Hostname: "acm.{region}.{dnsSuffix}",
|
||||
},
|
||||
},
|
||||
Endpoints: serviceEndpoints{
|
||||
endpointKey{
|
||||
Region: "us-gov-east-1",
|
||||
|
|
@ -23761,6 +23981,54 @@ var awsusgovPartition = partition{
|
|||
},
|
||||
},
|
||||
},
|
||||
"api.tunneling.iot": service{
|
||||
Defaults: endpointDefaults{
|
||||
defaultKey{}: endpoint{},
|
||||
defaultKey{
|
||||
Variant: fipsVariant,
|
||||
}: endpoint{
|
||||
Hostname: "api.tunneling.iot-fips.{region}.{dnsSuffix}",
|
||||
},
|
||||
},
|
||||
Endpoints: serviceEndpoints{
|
||||
endpointKey{
|
||||
Region: "fips-us-gov-east-1",
|
||||
}: endpoint{
|
||||
Hostname: "api.tunneling.iot-fips.us-gov-east-1.amazonaws.com",
|
||||
CredentialScope: credentialScope{
|
||||
Region: "us-gov-east-1",
|
||||
},
|
||||
Deprecated: boxedTrue,
|
||||
},
|
||||
endpointKey{
|
||||
Region: "fips-us-gov-west-1",
|
||||
}: endpoint{
|
||||
Hostname: "api.tunneling.iot-fips.us-gov-west-1.amazonaws.com",
|
||||
CredentialScope: credentialScope{
|
||||
Region: "us-gov-west-1",
|
||||
},
|
||||
Deprecated: boxedTrue,
|
||||
},
|
||||
endpointKey{
|
||||
Region: "us-gov-east-1",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "us-gov-east-1",
|
||||
Variant: fipsVariant,
|
||||
}: endpoint{
|
||||
Hostname: "api.tunneling.iot-fips.us-gov-east-1.amazonaws.com",
|
||||
},
|
||||
endpointKey{
|
||||
Region: "us-gov-west-1",
|
||||
}: endpoint{},
|
||||
endpointKey{
|
||||
Region: "us-gov-west-1",
|
||||
Variant: fipsVariant,
|
||||
}: endpoint{
|
||||
Hostname: "api.tunneling.iot-fips.us-gov-west-1.amazonaws.com",
|
||||
},
|
||||
},
|
||||
},
|
||||
"apigateway": service{
|
||||
Endpoints: serviceEndpoints{
|
||||
endpointKey{
|
||||
|
|
@ -24697,6 +24965,14 @@ var awsusgovPartition = partition{
|
|||
},
|
||||
},
|
||||
"ec2": service{
|
||||
Defaults: endpointDefaults{
|
||||
defaultKey{}: endpoint{},
|
||||
defaultKey{
|
||||
Variant: fipsVariant,
|
||||
}: endpoint{
|
||||
Hostname: "ec2.{region}.{dnsSuffix}",
|
||||
},
|
||||
},
|
||||
Endpoints: serviceEndpoints{
|
||||
endpointKey{
|
||||
Region: "us-gov-east-1",
|
||||
|
|
|
|||
2
vendor/github.com/aws/aws-sdk-go/aws/version.go
generated
vendored
2
vendor/github.com/aws/aws-sdk-go/aws/version.go
generated
vendored
|
|
@ -5,4 +5,4 @@ package aws
|
|||
const SDKName = "aws-sdk-go"
|
||||
|
||||
// SDKVersion is the version of this SDK
|
||||
const SDKVersion = "1.43.3"
|
||||
const SDKVersion = "1.43.10"
|
||||
|
|
|
|||
10
vendor/github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamapi/transport.go
generated
vendored
Normal file
10
vendor/github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamapi/transport.go
generated
vendored
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
//go:build go1.18
|
||||
// +build go1.18
|
||||
|
||||
package eventstreamapi
|
||||
|
||||
import "github.com/aws/aws-sdk-go/aws/request"
|
||||
|
||||
// This is a no-op for Go 1.18 and above.
|
||||
func ApplyHTTPTransportFixes(r *request.Request) {
|
||||
}
|
||||
19
vendor/github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamapi/transport_go1.17.go
generated
vendored
Normal file
19
vendor/github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamapi/transport_go1.17.go
generated
vendored
Normal file
|
|
@ -0,0 +1,19 @@
|
|||
//go:build !go1.18
|
||||
// +build !go1.18
|
||||
|
||||
package eventstreamapi
|
||||
|
||||
import "github.com/aws/aws-sdk-go/aws/request"
|
||||
|
||||
// ApplyHTTPTransportFixes applies fixes to the HTTP request for proper event
|
||||
// stream functionality. Go 1.15 through 1.17 HTTP client could hang forever
|
||||
// when an HTTP/2 connection failed with an non-200 status code and err. Using
|
||||
// Expect 100-Continue, allows the HTTP client to gracefully handle the non-200
|
||||
// status code, and close the connection.
|
||||
//
|
||||
// This is a no-op for Go 1.18 and above.
|
||||
func ApplyHTTPTransportFixes(r *request.Request) {
|
||||
r.Handlers.Sign.PushBack(func(r *request.Request) {
|
||||
r.HTTPRequest.Header.Set("Expect", "100-Continue")
|
||||
})
|
||||
}
|
||||
3177
vendor/github.com/aws/aws-sdk-go/service/s3/api.go
generated
vendored
3177
vendor/github.com/aws/aws-sdk-go/service/s3/api.go
generated
vendored
File diff suppressed because it is too large
Load diff
4
vendor/github.com/aws/aws-sdk-go/service/s3/s3iface/interface.go
generated
vendored
4
vendor/github.com/aws/aws-sdk-go/service/s3/s3iface/interface.go
generated
vendored
|
|
@ -244,6 +244,10 @@ type S3API interface {
|
|||
GetObjectAclWithContext(aws.Context, *s3.GetObjectAclInput, ...request.Option) (*s3.GetObjectAclOutput, error)
|
||||
GetObjectAclRequest(*s3.GetObjectAclInput) (*request.Request, *s3.GetObjectAclOutput)
|
||||
|
||||
GetObjectAttributes(*s3.GetObjectAttributesInput) (*s3.GetObjectAttributesOutput, error)
|
||||
GetObjectAttributesWithContext(aws.Context, *s3.GetObjectAttributesInput, ...request.Option) (*s3.GetObjectAttributesOutput, error)
|
||||
GetObjectAttributesRequest(*s3.GetObjectAttributesInput) (*request.Request, *s3.GetObjectAttributesOutput)
|
||||
|
||||
GetObjectLegalHold(*s3.GetObjectLegalHoldInput) (*s3.GetObjectLegalHoldOutput, error)
|
||||
GetObjectLegalHoldWithContext(aws.Context, *s3.GetObjectLegalHoldInput, ...request.Option) (*s3.GetObjectLegalHoldOutput, error)
|
||||
GetObjectLegalHoldRequest(*s3.GetObjectLegalHoldInput) (*request.Request, *s3.GetObjectLegalHoldOutput)
|
||||
|
|
|
|||
8
vendor/github.com/aws/aws-sdk-go/service/s3/s3manager/upload.go
generated
vendored
8
vendor/github.com/aws/aws-sdk-go/service/s3/s3manager/upload.go
generated
vendored
|
|
@ -124,6 +124,14 @@ func WithUploaderRequestOptions(opts ...request.Option) func(*Uploader) {
|
|||
// The Uploader structure that calls Upload(). It is safe to call Upload()
|
||||
// on this structure for multiple objects and across concurrent goroutines.
|
||||
// Mutating the Uploader's properties is not safe to be done concurrently.
|
||||
//
|
||||
// The ContentMD5 member for pre-computed MD5 checksums will be ignored for
|
||||
// multipart uploads. Objects that will be uploaded in a single part, the
|
||||
// ContentMD5 will be used.
|
||||
//
|
||||
// The Checksum members for pre-computed checksums will be ignored for
|
||||
// multipart uploads. Objects that will be uploaded in a single part, will
|
||||
// include the checksum member in the request.
|
||||
type Uploader struct {
|
||||
// The buffer size (in bytes) to use when buffering data into chunks and
|
||||
// sending them as parts to S3. The minimum allowed part size is 5MB, and
|
||||
|
|
|
|||
68
vendor/github.com/aws/aws-sdk-go/service/s3/s3manager/upload_input.go
generated
vendored
68
vendor/github.com/aws/aws-sdk-go/service/s3/s3manager/upload_input.go
generated
vendored
|
|
@ -11,6 +11,14 @@ import (
|
|||
// to an object in an Amazon S3 bucket. This type is similar to the s3
|
||||
// package's PutObjectInput with the exception that the Body member is an
|
||||
// io.Reader instead of an io.ReadSeeker.
|
||||
//
|
||||
// The ContentMD5 member for pre-computed MD5 checksums will be ignored for
|
||||
// multipart uploads. Objects that will be uploaded in a single part, the
|
||||
// ContentMD5 will be used.
|
||||
//
|
||||
// The Checksum members for pre-computed checksums will be ignored for
|
||||
// multipart uploads. Objects that will be uploaded in a single part, will
|
||||
// include the checksum member in the request.
|
||||
type UploadInput struct {
|
||||
_ struct{} `locationName:"PutObjectRequest" type:"structure" payload:"Body"`
|
||||
|
||||
|
|
@ -35,9 +43,9 @@ type UploadInput struct {
|
|||
// When using this action with Amazon S3 on Outposts, you must direct requests
|
||||
// to the S3 on Outposts hostname. The S3 on Outposts hostname takes the form
|
||||
// AccessPointName-AccountId.outpostID.s3-outposts.Region.amazonaws.com. When
|
||||
// using this action using S3 on Outposts through the Amazon Web Services SDKs,
|
||||
// using this action with S3 on Outposts through the Amazon Web Services SDKs,
|
||||
// you provide the Outposts bucket ARN in place of the bucket name. For more
|
||||
// information about S3 on Outposts ARNs, see Using S3 on Outposts (https://docs.aws.amazon.com/AmazonS3/latest/userguide/S3onOutposts.html)
|
||||
// information about S3 on Outposts ARNs, see Using Amazon S3 on Outposts (https://docs.aws.amazon.com/AmazonS3/latest/userguide/S3onOutposts.html)
|
||||
// in the Amazon S3 User Guide.
|
||||
//
|
||||
// Bucket is a required field
|
||||
|
|
@ -57,6 +65,51 @@ type UploadInput struct {
|
|||
// (http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9).
|
||||
CacheControl *string `location:"header" locationName:"Cache-Control" type:"string"`
|
||||
|
||||
// Indicates the algorithm used to create the checksum for the object when using
|
||||
// the SDK. This header will not provide any additional functionality if not
|
||||
// using the SDK. When sending this header, there must be a corresponding x-amz-checksum
|
||||
// or x-amz-trailer header sent. Otherwise, Amazon S3 fails the request with
|
||||
// the HTTP status code 400 Bad Request. For more information, see Checking
|
||||
// object integrity (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html)
|
||||
// in the Amazon S3 User Guide.
|
||||
//
|
||||
// If you provide an individual checksum, Amazon S3 ignores any provided ChecksumAlgorithm
|
||||
// parameter.
|
||||
//
|
||||
// The AWS SDK for Go v1 does not support automatic computing request payload
|
||||
// checksum. This feature is available in the AWS SDK for Go v2. If a value
|
||||
// is specified for this parameter, the matching algorithm's checksum member
|
||||
// must be populated with the algorithm's checksum of the request payload.
|
||||
ChecksumAlgorithm *string `location:"header" locationName:"x-amz-sdk-checksum-algorithm" type:"string" enum:"ChecksumAlgorithm"`
|
||||
|
||||
// This header can be used as a data integrity check to verify that the data
|
||||
// received is the same data that was originally sent. This header specifies
|
||||
// the base64-encoded, 32-bit CRC32 checksum of the object. For more information,
|
||||
// see Checking object integrity (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html)
|
||||
// in the Amazon S3 User Guide.
|
||||
ChecksumCRC32 *string `location:"header" locationName:"x-amz-checksum-crc32" type:"string"`
|
||||
|
||||
// This header can be used as a data integrity check to verify that the data
|
||||
// received is the same data that was originally sent. This header specifies
|
||||
// the base64-encoded, 32-bit CRC32C checksum of the object. For more information,
|
||||
// see Checking object integrity (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html)
|
||||
// in the Amazon S3 User Guide.
|
||||
ChecksumCRC32C *string `location:"header" locationName:"x-amz-checksum-crc32c" type:"string"`
|
||||
|
||||
// This header can be used as a data integrity check to verify that the data
|
||||
// received is the same data that was originally sent. This header specifies
|
||||
// the base64-encoded, 160-bit SHA-1 digest of the object. For more information,
|
||||
// see Checking object integrity (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html)
|
||||
// in the Amazon S3 User Guide.
|
||||
ChecksumSHA1 *string `location:"header" locationName:"x-amz-checksum-sha1" type:"string"`
|
||||
|
||||
// This header can be used as a data integrity check to verify that the data
|
||||
// received is the same data that was originally sent. This header specifies
|
||||
// the base64-encoded, 256-bit SHA-256 digest of the object. For more information,
|
||||
// see Checking object integrity (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html)
|
||||
// in the Amazon S3 User Guide.
|
||||
ChecksumSHA256 *string `location:"header" locationName:"x-amz-checksum-sha256" type:"string"`
|
||||
|
||||
// Specifies presentational information for the object. For more information,
|
||||
// see http://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html#sec19.5.1 (http://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html#sec19.5.1).
|
||||
ContentDisposition *string `location:"header" locationName:"Content-Disposition" type:"string"`
|
||||
|
|
@ -76,6 +129,9 @@ type UploadInput struct {
|
|||
// it is optional, we recommend using the Content-MD5 mechanism as an end-to-end
|
||||
// integrity check. For more information about REST request authentication,
|
||||
// see REST Authentication (https://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html).
|
||||
//
|
||||
// If the ContentMD5 is provided for a multipart upload, it will be ignored.
|
||||
// Objects that will be uploaded in a single part, the ContentMD5 will be used.
|
||||
ContentMD5 *string `location:"header" locationName:"Content-MD5" type:"string"`
|
||||
|
||||
// A standard MIME type describing the format of the contents. For more information,
|
||||
|
|
@ -83,8 +139,8 @@ type UploadInput struct {
|
|||
ContentType *string `location:"header" locationName:"Content-Type" type:"string"`
|
||||
|
||||
// The account ID of the expected bucket owner. If the bucket is owned by a
|
||||
// different account, the request will fail with an HTTP 403 (Access Denied)
|
||||
// error.
|
||||
// different account, the request fails with the HTTP status code 403 Forbidden
|
||||
// (access denied).
|
||||
ExpectedBucketOwner *string `location:"header" locationName:"x-amz-expected-bucket-owner" type:"string"`
|
||||
|
||||
// The date and time at which the object is no longer cacheable. For more information,
|
||||
|
|
@ -132,8 +188,8 @@ type UploadInput struct {
|
|||
|
||||
// Confirms that the requester knows that they will be charged for the request.
|
||||
// Bucket owners need not specify this parameter in their requests. For information
|
||||
// about downloading objects from requester pays buckets, see Downloading Objects
|
||||
// in Requestor Pays Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html)
|
||||
// about downloading objects from Requester Pays buckets, see Downloading Objects
|
||||
// in Requester Pays Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html)
|
||||
// in the Amazon S3 User Guide.
|
||||
RequestPayer *string `location:"header" locationName:"x-amz-request-payer" type:"string" enum:"RequestPayer"`
|
||||
|
||||
|
|
|
|||
17
vendor/golang.org/x/net/http2/go118.go
generated
vendored
Normal file
17
vendor/golang.org/x/net/http2/go118.go
generated
vendored
Normal file
|
|
@ -0,0 +1,17 @@
|
|||
// Copyright 2021 The Go Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
//go:build go1.18
|
||||
// +build go1.18
|
||||
|
||||
package http2
|
||||
|
||||
import (
|
||||
"crypto/tls"
|
||||
"net"
|
||||
)
|
||||
|
||||
func tlsUnderlyingConn(tc *tls.Conn) net.Conn {
|
||||
return tc.NetConn()
|
||||
}
|
||||
17
vendor/golang.org/x/net/http2/not_go118.go
generated
vendored
Normal file
17
vendor/golang.org/x/net/http2/not_go118.go
generated
vendored
Normal file
|
|
@ -0,0 +1,17 @@
|
|||
// Copyright 2021 The Go Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
//go:build !go1.18
|
||||
// +build !go1.18
|
||||
|
||||
package http2
|
||||
|
||||
import (
|
||||
"crypto/tls"
|
||||
"net"
|
||||
)
|
||||
|
||||
func tlsUnderlyingConn(tc *tls.Conn) net.Conn {
|
||||
return nil
|
||||
}
|
||||
41
vendor/golang.org/x/net/http2/transport.go
generated
vendored
41
vendor/golang.org/x/net/http2/transport.go
generated
vendored
|
|
@ -735,7 +735,6 @@ func (cc *ClientConn) healthCheck() {
|
|||
err := cc.Ping(ctx)
|
||||
if err != nil {
|
||||
cc.closeForLostPing()
|
||||
cc.t.connPool().MarkDead(cc)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
|
@ -907,6 +906,24 @@ func (cc *ClientConn) onIdleTimeout() {
|
|||
cc.closeIfIdle()
|
||||
}
|
||||
|
||||
func (cc *ClientConn) closeConn() error {
|
||||
t := time.AfterFunc(250*time.Millisecond, cc.forceCloseConn)
|
||||
defer t.Stop()
|
||||
return cc.tconn.Close()
|
||||
}
|
||||
|
||||
// A tls.Conn.Close can hang for a long time if the peer is unresponsive.
|
||||
// Try to shut it down more aggressively.
|
||||
func (cc *ClientConn) forceCloseConn() {
|
||||
tc, ok := cc.tconn.(*tls.Conn)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if nc := tlsUnderlyingConn(tc); nc != nil {
|
||||
nc.Close()
|
||||
}
|
||||
}
|
||||
|
||||
func (cc *ClientConn) closeIfIdle() {
|
||||
cc.mu.Lock()
|
||||
if len(cc.streams) > 0 || cc.streamsReserved > 0 {
|
||||
|
|
@ -921,7 +938,7 @@ func (cc *ClientConn) closeIfIdle() {
|
|||
if VerboseLogs {
|
||||
cc.vlogf("http2: Transport closing idle conn %p (forSingleUse=%v, maxStream=%v)", cc, cc.singleUse, nextID-2)
|
||||
}
|
||||
cc.tconn.Close()
|
||||
cc.closeConn()
|
||||
}
|
||||
|
||||
func (cc *ClientConn) isDoNotReuseAndIdle() bool {
|
||||
|
|
@ -938,7 +955,7 @@ func (cc *ClientConn) Shutdown(ctx context.Context) error {
|
|||
return err
|
||||
}
|
||||
// Wait for all in-flight streams to complete or connection to close
|
||||
done := make(chan error, 1)
|
||||
done := make(chan struct{})
|
||||
cancelled := false // guarded by cc.mu
|
||||
go func() {
|
||||
cc.mu.Lock()
|
||||
|
|
@ -946,7 +963,7 @@ func (cc *ClientConn) Shutdown(ctx context.Context) error {
|
|||
for {
|
||||
if len(cc.streams) == 0 || cc.closed {
|
||||
cc.closed = true
|
||||
done <- cc.tconn.Close()
|
||||
close(done)
|
||||
break
|
||||
}
|
||||
if cancelled {
|
||||
|
|
@ -957,8 +974,8 @@ func (cc *ClientConn) Shutdown(ctx context.Context) error {
|
|||
}()
|
||||
shutdownEnterWaitStateHook()
|
||||
select {
|
||||
case err := <-done:
|
||||
return err
|
||||
case <-done:
|
||||
return cc.closeConn()
|
||||
case <-ctx.Done():
|
||||
cc.mu.Lock()
|
||||
// Free the goroutine above
|
||||
|
|
@ -1001,9 +1018,9 @@ func (cc *ClientConn) closeForError(err error) error {
|
|||
for _, cs := range cc.streams {
|
||||
cs.abortStreamLocked(err)
|
||||
}
|
||||
defer cc.cond.Broadcast()
|
||||
defer cc.mu.Unlock()
|
||||
return cc.tconn.Close()
|
||||
cc.cond.Broadcast()
|
||||
cc.mu.Unlock()
|
||||
return cc.closeConn()
|
||||
}
|
||||
|
||||
// Close closes the client connection immediately.
|
||||
|
|
@ -1978,7 +1995,7 @@ func (cc *ClientConn) forgetStreamID(id uint32) {
|
|||
cc.vlogf("http2: Transport closing idle conn %p (forSingleUse=%v, maxStream=%v)", cc, cc.singleUse, cc.nextStreamID-2)
|
||||
}
|
||||
cc.closed = true
|
||||
defer cc.tconn.Close()
|
||||
defer cc.closeConn()
|
||||
}
|
||||
|
||||
cc.mu.Unlock()
|
||||
|
|
@ -2025,8 +2042,8 @@ func isEOFOrNetReadError(err error) bool {
|
|||
|
||||
func (rl *clientConnReadLoop) cleanup() {
|
||||
cc := rl.cc
|
||||
defer cc.tconn.Close()
|
||||
defer cc.t.connPool().MarkDead(cc)
|
||||
cc.t.connPool().MarkDead(cc)
|
||||
defer cc.closeConn()
|
||||
defer close(cc.readerDone)
|
||||
|
||||
if cc.idleTimer != nil {
|
||||
|
|
|
|||
23
vendor/golang.org/x/sys/unix/ioctl_linux.go
generated
vendored
23
vendor/golang.org/x/sys/unix/ioctl_linux.go
generated
vendored
|
|
@ -194,3 +194,26 @@ func ioctlIfreqData(fd int, req uint, value *ifreqData) error {
|
|||
// identical so pass *IfreqData directly.
|
||||
return ioctlPtr(fd, req, unsafe.Pointer(value))
|
||||
}
|
||||
|
||||
// IoctlKCMClone attaches a new file descriptor to a multiplexor by cloning an
|
||||
// existing KCM socket, returning a structure containing the file descriptor of
|
||||
// the new socket.
|
||||
func IoctlKCMClone(fd int) (*KCMClone, error) {
|
||||
var info KCMClone
|
||||
if err := ioctlPtr(fd, SIOCKCMCLONE, unsafe.Pointer(&info)); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &info, nil
|
||||
}
|
||||
|
||||
// IoctlKCMAttach attaches a TCP socket and associated BPF program file
|
||||
// descriptor to a multiplexor.
|
||||
func IoctlKCMAttach(fd int, info KCMAttach) error {
|
||||
return ioctlPtr(fd, SIOCKCMATTACH, unsafe.Pointer(&info))
|
||||
}
|
||||
|
||||
// IoctlKCMUnattach unattaches a TCP socket file descriptor from a multiplexor.
|
||||
func IoctlKCMUnattach(fd int, info KCMUnattach) error {
|
||||
return ioctlPtr(fd, SIOCKCMUNATTACH, unsafe.Pointer(&info))
|
||||
}
|
||||
|
|
|
|||
3
vendor/golang.org/x/sys/unix/mkerrors.sh
generated
vendored
3
vendor/golang.org/x/sys/unix/mkerrors.sh
generated
vendored
|
|
@ -205,6 +205,7 @@ struct ltchars {
|
|||
#include <linux/bpf.h>
|
||||
#include <linux/can.h>
|
||||
#include <linux/can/error.h>
|
||||
#include <linux/can/netlink.h>
|
||||
#include <linux/can/raw.h>
|
||||
#include <linux/capability.h>
|
||||
#include <linux/cryptouser.h>
|
||||
|
|
@ -231,6 +232,7 @@ struct ltchars {
|
|||
#include <linux/if_packet.h>
|
||||
#include <linux/if_xdp.h>
|
||||
#include <linux/input.h>
|
||||
#include <linux/kcm.h>
|
||||
#include <linux/kexec.h>
|
||||
#include <linux/keyctl.h>
|
||||
#include <linux/landlock.h>
|
||||
|
|
@ -503,6 +505,7 @@ ccflags="$@"
|
|||
$2 ~ /^O?XTABS$/ ||
|
||||
$2 ~ /^TC[IO](ON|OFF)$/ ||
|
||||
$2 ~ /^IN_/ ||
|
||||
$2 ~ /^KCM/ ||
|
||||
$2 ~ /^LANDLOCK_/ ||
|
||||
$2 ~ /^LOCK_(SH|EX|NB|UN)$/ ||
|
||||
$2 ~ /^LO_(KEY|NAME)_SIZE$/ ||
|
||||
|
|
|
|||
17
vendor/golang.org/x/sys/unix/zerrors_linux.go
generated
vendored
17
vendor/golang.org/x/sys/unix/zerrors_linux.go
generated
vendored
|
|
@ -260,6 +260,17 @@ const (
|
|||
BUS_USB = 0x3
|
||||
BUS_VIRTUAL = 0x6
|
||||
CAN_BCM = 0x2
|
||||
CAN_CTRLMODE_3_SAMPLES = 0x4
|
||||
CAN_CTRLMODE_BERR_REPORTING = 0x10
|
||||
CAN_CTRLMODE_CC_LEN8_DLC = 0x100
|
||||
CAN_CTRLMODE_FD = 0x20
|
||||
CAN_CTRLMODE_FD_NON_ISO = 0x80
|
||||
CAN_CTRLMODE_LISTENONLY = 0x2
|
||||
CAN_CTRLMODE_LOOPBACK = 0x1
|
||||
CAN_CTRLMODE_ONE_SHOT = 0x8
|
||||
CAN_CTRLMODE_PRESUME_ACK = 0x40
|
||||
CAN_CTRLMODE_TDC_AUTO = 0x200
|
||||
CAN_CTRLMODE_TDC_MANUAL = 0x400
|
||||
CAN_EFF_FLAG = 0x80000000
|
||||
CAN_EFF_ID_BITS = 0x1d
|
||||
CAN_EFF_MASK = 0x1fffffff
|
||||
|
|
@ -337,6 +348,7 @@ const (
|
|||
CAN_RTR_FLAG = 0x40000000
|
||||
CAN_SFF_ID_BITS = 0xb
|
||||
CAN_SFF_MASK = 0x7ff
|
||||
CAN_TERMINATION_DISABLED = 0x0
|
||||
CAN_TP16 = 0x3
|
||||
CAN_TP20 = 0x4
|
||||
CAP_AUDIT_CONTROL = 0x1e
|
||||
|
|
@ -1274,6 +1286,8 @@ const (
|
|||
IUTF8 = 0x4000
|
||||
IXANY = 0x800
|
||||
JFFS2_SUPER_MAGIC = 0x72b6
|
||||
KCMPROTO_CONNECTED = 0x0
|
||||
KCM_RECV_DISABLE = 0x1
|
||||
KEXEC_ARCH_386 = 0x30000
|
||||
KEXEC_ARCH_68K = 0x40000
|
||||
KEXEC_ARCH_AARCH64 = 0xb70000
|
||||
|
|
@ -2446,6 +2460,9 @@ const (
|
|||
SIOCGSTAMPNS = 0x8907
|
||||
SIOCGSTAMPNS_OLD = 0x8907
|
||||
SIOCGSTAMP_OLD = 0x8906
|
||||
SIOCKCMATTACH = 0x89e0
|
||||
SIOCKCMCLONE = 0x89e2
|
||||
SIOCKCMUNATTACH = 0x89e1
|
||||
SIOCOUTQNSD = 0x894b
|
||||
SIOCPROTOPRIVATE = 0x89e0
|
||||
SIOCRTMSG = 0x890d
|
||||
|
|
|
|||
90
vendor/golang.org/x/sys/unix/ztypes_linux.go
generated
vendored
90
vendor/golang.org/x/sys/unix/ztypes_linux.go
generated
vendored
|
|
@ -3771,6 +3771,8 @@ const (
|
|||
ETHTOOL_A_TUNNEL_INFO_MAX = 0x2
|
||||
)
|
||||
|
||||
const SPEED_UNKNOWN = -0x1
|
||||
|
||||
type EthtoolDrvinfo struct {
|
||||
Cmd uint32
|
||||
Driver [32]byte
|
||||
|
|
@ -4070,3 +4072,91 @@ const (
|
|||
NL_POLICY_TYPE_ATTR_MASK = 0xc
|
||||
NL_POLICY_TYPE_ATTR_MAX = 0xc
|
||||
)
|
||||
|
||||
type CANBitTiming struct {
|
||||
Bitrate uint32
|
||||
Sample_point uint32
|
||||
Tq uint32
|
||||
Prop_seg uint32
|
||||
Phase_seg1 uint32
|
||||
Phase_seg2 uint32
|
||||
Sjw uint32
|
||||
Brp uint32
|
||||
}
|
||||
|
||||
type CANBitTimingConst struct {
|
||||
Name [16]uint8
|
||||
Tseg1_min uint32
|
||||
Tseg1_max uint32
|
||||
Tseg2_min uint32
|
||||
Tseg2_max uint32
|
||||
Sjw_max uint32
|
||||
Brp_min uint32
|
||||
Brp_max uint32
|
||||
Brp_inc uint32
|
||||
}
|
||||
|
||||
type CANClock struct {
|
||||
Freq uint32
|
||||
}
|
||||
|
||||
type CANBusErrorCounters struct {
|
||||
Txerr uint16
|
||||
Rxerr uint16
|
||||
}
|
||||
|
||||
type CANCtrlMode struct {
|
||||
Mask uint32
|
||||
Flags uint32
|
||||
}
|
||||
|
||||
type CANDeviceStats struct {
|
||||
Bus_error uint32
|
||||
Error_warning uint32
|
||||
Error_passive uint32
|
||||
Bus_off uint32
|
||||
Arbitration_lost uint32
|
||||
Restarts uint32
|
||||
}
|
||||
|
||||
const (
|
||||
CAN_STATE_ERROR_ACTIVE = 0x0
|
||||
CAN_STATE_ERROR_WARNING = 0x1
|
||||
CAN_STATE_ERROR_PASSIVE = 0x2
|
||||
CAN_STATE_BUS_OFF = 0x3
|
||||
CAN_STATE_STOPPED = 0x4
|
||||
CAN_STATE_SLEEPING = 0x5
|
||||
CAN_STATE_MAX = 0x6
|
||||
)
|
||||
|
||||
const (
|
||||
IFLA_CAN_UNSPEC = 0x0
|
||||
IFLA_CAN_BITTIMING = 0x1
|
||||
IFLA_CAN_BITTIMING_CONST = 0x2
|
||||
IFLA_CAN_CLOCK = 0x3
|
||||
IFLA_CAN_STATE = 0x4
|
||||
IFLA_CAN_CTRLMODE = 0x5
|
||||
IFLA_CAN_RESTART_MS = 0x6
|
||||
IFLA_CAN_RESTART = 0x7
|
||||
IFLA_CAN_BERR_COUNTER = 0x8
|
||||
IFLA_CAN_DATA_BITTIMING = 0x9
|
||||
IFLA_CAN_DATA_BITTIMING_CONST = 0xa
|
||||
IFLA_CAN_TERMINATION = 0xb
|
||||
IFLA_CAN_TERMINATION_CONST = 0xc
|
||||
IFLA_CAN_BITRATE_CONST = 0xd
|
||||
IFLA_CAN_DATA_BITRATE_CONST = 0xe
|
||||
IFLA_CAN_BITRATE_MAX = 0xf
|
||||
)
|
||||
|
||||
type KCMAttach struct {
|
||||
Fd int32
|
||||
Bpf_fd int32
|
||||
}
|
||||
|
||||
type KCMUnattach struct {
|
||||
Fd int32
|
||||
}
|
||||
|
||||
type KCMClone struct {
|
||||
Fd int32
|
||||
}
|
||||
|
|
|
|||
4
vendor/golang.org/x/sys/unix/ztypes_linux_s390x.go
generated
vendored
4
vendor/golang.org/x/sys/unix/ztypes_linux_s390x.go
generated
vendored
|
|
@ -210,8 +210,8 @@ type PtraceFpregs struct {
|
|||
}
|
||||
|
||||
type PtracePer struct {
|
||||
_ [0]uint64
|
||||
_ [32]byte
|
||||
Control_regs [3]uint64
|
||||
_ [8]byte
|
||||
Starting_addr uint64
|
||||
Ending_addr uint64
|
||||
Perc_atmid uint16
|
||||
|
|
|
|||
14
vendor/modules.txt
vendored
14
vendor/modules.txt
vendored
|
|
@ -5,10 +5,10 @@ cloud.google.com/go/internal
|
|||
cloud.google.com/go/internal/optional
|
||||
cloud.google.com/go/internal/trace
|
||||
cloud.google.com/go/internal/version
|
||||
# cloud.google.com/go/compute v1.4.0
|
||||
# cloud.google.com/go/compute v1.5.0
|
||||
## explicit; go 1.15
|
||||
cloud.google.com/go/compute/metadata
|
||||
# cloud.google.com/go/iam v0.2.0
|
||||
# cloud.google.com/go/iam v0.3.0
|
||||
## explicit; go 1.15
|
||||
cloud.google.com/go/iam
|
||||
# cloud.google.com/go/storage v1.21.0
|
||||
|
|
@ -34,7 +34,7 @@ github.com/VictoriaMetrics/metricsql/binaryop
|
|||
# github.com/VividCortex/ewma v1.2.0
|
||||
## explicit; go 1.12
|
||||
github.com/VividCortex/ewma
|
||||
# github.com/aws/aws-sdk-go v1.43.3
|
||||
# github.com/aws/aws-sdk-go v1.43.10
|
||||
## explicit; go 1.11
|
||||
github.com/aws/aws-sdk-go/aws
|
||||
github.com/aws/aws-sdk-go/aws/arn
|
||||
|
|
@ -265,7 +265,7 @@ go.opencensus.io/trace/tracestate
|
|||
go.uber.org/atomic
|
||||
# go.uber.org/goleak v1.1.11-0.20210813005559-691160354723
|
||||
## explicit; go 1.13
|
||||
# golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd
|
||||
# golang.org/x/net v0.0.0-20220225172249-27dd8689420f
|
||||
## explicit; go 1.17
|
||||
golang.org/x/net/context
|
||||
golang.org/x/net/context/ctxhttp
|
||||
|
|
@ -277,7 +277,7 @@ golang.org/x/net/internal/socks
|
|||
golang.org/x/net/internal/timeseries
|
||||
golang.org/x/net/proxy
|
||||
golang.org/x/net/trace
|
||||
# golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8
|
||||
# golang.org/x/oauth2 v0.0.0-20220223155221-ee480838109b
|
||||
## explicit; go 1.11
|
||||
golang.org/x/oauth2
|
||||
golang.org/x/oauth2/authhandler
|
||||
|
|
@ -290,7 +290,7 @@ golang.org/x/oauth2/jwt
|
|||
# golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
|
||||
## explicit
|
||||
golang.org/x/sync/errgroup
|
||||
# golang.org/x/sys v0.0.0-20220222172238-00053529121e
|
||||
# golang.org/x/sys v0.0.0-20220227234510-4e6760a101f9
|
||||
## explicit; go 1.17
|
||||
golang.org/x/sys/internal/unsafeheader
|
||||
golang.org/x/sys/unix
|
||||
|
|
@ -338,7 +338,7 @@ google.golang.org/appengine/internal/socket
|
|||
google.golang.org/appengine/internal/urlfetch
|
||||
google.golang.org/appengine/socket
|
||||
google.golang.org/appengine/urlfetch
|
||||
# google.golang.org/genproto v0.0.0-20220222154240-daf995802d7b
|
||||
# google.golang.org/genproto v0.0.0-20220302033224-9aa15565e42a
|
||||
## explicit; go 1.15
|
||||
google.golang.org/genproto/googleapis/api/annotations
|
||||
google.golang.org/genproto/googleapis/iam/v1
|
||||
|
|
|
|||
Loading…
Reference in a new issue