mirror of
https://github.com/VictoriaMetrics/VictoriaMetrics.git
synced 2025-03-11 15:34:56 +00:00
Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files
This commit is contained in:
commit
e8ff658b2e
78 changed files with 6449 additions and 1091 deletions
|
@ -429,7 +429,7 @@ The `__graphite__` pseudo-label supports e.g. alternate regexp filters such as `
|
|||
|
||||
VictoriaMetrics supports [telnet put protocol](http://opentsdb.net/docs/build/html/api_telnet/put.html)
|
||||
and [HTTP /api/put requests](http://opentsdb.net/docs/build/html/api_http/put.html) for ingesting OpenTSDB data.
|
||||
The same protocol is used for [ingesting data in KairosDB](https://kairosdb.github.io/docs/build/html/PushingData.html).
|
||||
The same protocol is used for [ingesting data in KairosDB](https://kairosdb.github.io/docs/PushingData.html).
|
||||
|
||||
### Sending data via `telnet put` protocol
|
||||
|
||||
|
@ -1604,7 +1604,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
|
|||
-envflag.prefix string
|
||||
Prefix for environment variables if -envflag.enable is set
|
||||
-eula
|
||||
By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf
|
||||
By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/legal/eula/
|
||||
-finalMergeDelay duration
|
||||
The delay before starting final merge for per-month partition after no new data is ingested into it. Final merge may require additional disk IO and CPU resources. Final merge may increase query speed and reduce disk space usage in some cases. Zero value disables final merge
|
||||
-forceFlushAuthKey string
|
||||
|
|
|
@ -6,7 +6,7 @@ Supported storage systems for backups:
|
|||
|
||||
* [GCS](https://cloud.google.com/storage/). Example: `gs://<bucket>/<path/to/backup>`
|
||||
* [S3](https://aws.amazon.com/s3/). Example: `s3://<bucket>/<path/to/backup>`
|
||||
* Any S3-compatible storage such as [MinIO](https://github.com/minio/minio), [Ceph](https://docs.ceph.com/en/pacific/radosgw/s3/) or [Swift](https://www.swiftstack.com/docs/admin/middleware/s3_middleware.html). See [these docs](#advanced-usage) for details.
|
||||
* Any S3-compatible storage such as [MinIO](https://github.com/minio/minio), [Ceph](https://docs.ceph.com/en/pacific/radosgw/s3/) or [Swift](https://platform.swiftstack.com/docs/admin/middleware/s3_middleware.html). See [these docs](#advanced-usage) for details.
|
||||
* Local filesystem. Example: `fs://</absolute/path/to/backup>`
|
||||
|
||||
`vmbackup` supports incremental and full backups. Incremental backups are created automatically if the destination path already contains data from the previous backup.
|
||||
|
|
|
@ -207,7 +207,7 @@ The shortlist of configuration flags include the following:
|
|||
-envflag.prefix string
|
||||
Prefix for environment variables if -envflag.enable is set
|
||||
-eula
|
||||
By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf
|
||||
By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/legal/eula/
|
||||
-fs.disableMmap
|
||||
Whether to use pread() instead of mmap() for reading data files. By default, mmap() is used for 64-bit arches and pread() is used for 32-bit arches as they cannot read data files larger than 2^32 bytes in memory. mmap() is usually faster for reading small data chunks than pread()
|
||||
-http.connTimeout duration
|
||||
|
|
|
@ -1,12 +1,12 @@
|
|||
{
|
||||
"files": {
|
||||
"main.css": "./static/css/main.79ff1ad2.css",
|
||||
"main.js": "./static/js/main.cc7d894f.js",
|
||||
"main.css": "./static/css/main.098d452b.css",
|
||||
"main.js": "./static/js/main.eef1a9bb.js",
|
||||
"static/js/27.cc1b69f7.chunk.js": "./static/js/27.cc1b69f7.chunk.js",
|
||||
"index.html": "./index.html"
|
||||
},
|
||||
"entrypoints": [
|
||||
"static/css/main.79ff1ad2.css",
|
||||
"static/js/main.cc7d894f.js"
|
||||
"static/css/main.098d452b.css",
|
||||
"static/js/main.eef1a9bb.js"
|
||||
]
|
||||
}
|
|
@ -1 +1 @@
|
|||
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="VM-UI is a metric explorer for Victoria Metrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,500,700&display=swap"/><script defer="defer" src="./static/js/main.cc7d894f.js"></script><link href="./static/css/main.79ff1ad2.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>
|
||||
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="VM-UI is a metric explorer for Victoria Metrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,500,700&display=swap"/><script defer="defer" src="./static/js/main.eef1a9bb.js"></script><link href="./static/css/main.098d452b.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>
|
|
@ -1 +1 @@
|
|||
body{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;font-family:-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Oxygen,Ubuntu,Cantarell,Fira Sans,Droid Sans,Helvetica Neue,sans-serif}code{font-family:source-code-pro,Menlo,Monaco,Consolas,Courier New,monospace}.MuiAccordionSummary-content{margin:0!important}.uplot,.uplot *,.uplot :after,.uplot :before{box-sizing:border-box}.uplot{font-family:system-ui,-apple-system,Segoe UI,Roboto,Helvetica Neue,Arial,Noto Sans,sans-serif,Apple Color Emoji,Segoe UI Emoji,Segoe UI Symbol,Noto Color Emoji;line-height:1.5;width:-webkit-min-content;width:min-content}.u-title{font-size:18px;font-weight:700;text-align:center}.u-wrap{position:relative;-webkit-user-select:none;-ms-user-select:none;user-select:none}.u-over,.u-under{position:absolute}.u-under{overflow:hidden}.uplot canvas{display:block;height:100%;position:relative;width:100%}.u-axis{position:absolute}.u-legend{font-size:14px;margin:auto;text-align:center}.u-inline{display:block}.u-inline *{display:inline-block}.u-inline tr{margin-right:16px}.u-legend th{font-weight:600}.u-legend th>*{display:inline-block;vertical-align:middle}.u-legend .u-marker{background-clip:padding-box!important;height:1em;margin-right:4px;width:1em}.u-inline.u-live th:after{content:":";vertical-align:middle}.u-inline:not(.u-live) .u-value{display:none}.u-series>*{padding:4px}.u-series th{cursor:pointer}.u-legend .u-off>*{opacity:.3}.u-select{background:rgba(0,0,0,.07)}.u-cursor-x,.u-cursor-y,.u-select{pointer-events:none;position:absolute}.u-cursor-x,.u-cursor-y{left:0;top:0;will-change:transform;z-index:100}.u-hz .u-cursor-x,.u-vt .u-cursor-y{border-right:1px dashed #607d8b;height:100%}.u-hz .u-cursor-y,.u-vt .u-cursor-x{border-bottom:1px dashed #607d8b;width:100%}.u-cursor-pt{background-clip:padding-box!important;border:0 solid;border-radius:50%;left:0;pointer-events:none;position:absolute;top:0;will-change:transform;z-index:100}.u-axis.u-off,.u-cursor-pt.u-off,.u-cursor-x.u-off,.u-cursor-y.u-off,.u-select.u-off,.u-tooltip{display:none}.u-tooltip{grid-gap:12px;word-wrap:break-word;background:rgba(57,57,57,.9);border-radius:4px;color:#fff;font-family:monospace;font-size:10px;font-weight:500;line-height:1.4em;max-width:300px;padding:8px;pointer-events:none;position:absolute;z-index:100}.u-tooltip-data{align-items:center;display:flex;flex-wrap:wrap;font-size:11px;line-height:150%}.u-tooltip-data__value{font-weight:700;padding:4px}.u-tooltip__info{grid-gap:4px;display:grid}.u-tooltip__marker{height:12px;margin-right:4px;width:12px}.legendWrapper{grid-gap:20px;cursor:default;display:grid;grid-template-columns:repeat(auto-fit,minmax(400px,1fr));margin-top:20px;position:relative}.legendGroup{margin-bottom:24px}.legendGroupTitle{align-items:center;display:grid;font-size:11px;grid-template-columns:43px auto;padding:10px}.legendGroupQuery{grid-column:1/3;opacity:.6}.legendGroupLine{margin-right:10px}.legendItem{grid-gap:6px;align-items:start;background-color:#fff;cursor:pointer;display:inline-grid;grid-template-columns:auto auto;justify-content:start;padding:5px 10px;transition:.2s ease}.legendItemHide{opacity:.5;text-decoration:line-through}.legendItem:hover{background-color:rgba(0,0,0,.1)}.legendMarker{border-style:solid;border-width:2px;box-sizing:border-box;height:12px;margin:3px 0;transition:.2s ease;width:12px}.legendLabel{font-size:11px;font-weight:400}.legendWrapperHotkey{align-items:center;display:flex;font-size:11px}.legendWrapperHotkey p{margin-right:20px}.legendWrapperHotkey code{word-wrap:break-word;background-color:#f2f2f2;border:1px solid #dedede;border-radius:2px;color:#0a0a0a;display:inline;font-size:10px;font-weight:400;max-width:100%;padding:4px 6px}
|
||||
body{-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;font-family:-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Oxygen,Ubuntu,Cantarell,Fira Sans,Droid Sans,Helvetica Neue,sans-serif}code{font-family:source-code-pro,Menlo,Monaco,Consolas,Courier New,monospace}.MuiAccordionSummary-content{margin:0!important}.uplot,.uplot *,.uplot :after,.uplot :before{box-sizing:border-box}.uplot{font-family:system-ui,-apple-system,Segoe UI,Roboto,Helvetica Neue,Arial,Noto Sans,sans-serif,Apple Color Emoji,Segoe UI Emoji,Segoe UI Symbol,Noto Color Emoji;line-height:1.5;width:-webkit-min-content;width:min-content}.u-title{font-size:18px;font-weight:700;text-align:center}.u-wrap{position:relative;-webkit-user-select:none;-ms-user-select:none;user-select:none}.u-over,.u-under{position:absolute}.u-under{overflow:hidden}.uplot canvas{display:block;height:100%;position:relative;width:100%}.u-axis{position:absolute}.u-legend{font-size:14px;margin:auto;text-align:center}.u-inline{display:block}.u-inline *{display:inline-block}.u-inline tr{margin-right:16px}.u-legend th{font-weight:600}.u-legend th>*{display:inline-block;vertical-align:middle}.u-legend .u-marker{background-clip:padding-box!important;height:1em;margin-right:4px;width:1em}.u-inline.u-live th:after{content:":";vertical-align:middle}.u-inline:not(.u-live) .u-value{display:none}.u-series>*{padding:4px}.u-series th{cursor:pointer}.u-legend .u-off>*{opacity:.3}.u-select{background:rgba(0,0,0,.07)}.u-cursor-x,.u-cursor-y,.u-select{pointer-events:none;position:absolute}.u-cursor-x,.u-cursor-y{left:0;top:0;will-change:transform;z-index:100}.u-hz .u-cursor-x,.u-vt .u-cursor-y{border-right:1px dashed #607d8b;height:100%}.u-hz .u-cursor-y,.u-vt .u-cursor-x{border-bottom:1px dashed #607d8b;width:100%}.u-cursor-pt{background-clip:padding-box!important;border:0 solid;border-radius:50%;left:0;pointer-events:none;position:absolute;top:0;will-change:transform;z-index:100}.u-axis.u-off,.u-cursor-pt.u-off,.u-cursor-x.u-off,.u-cursor-y.u-off,.u-select.u-off,.u-tooltip{display:none}.u-tooltip{grid-gap:12px;word-wrap:break-word;background:rgba(57,57,57,.9);border-radius:4px;color:#fff;font-family:monospace;font-size:10px;font-weight:500;line-height:1.4em;max-width:300px;padding:8px;pointer-events:none;position:absolute;z-index:100}.u-tooltip-data{align-items:center;display:flex;flex-wrap:wrap;font-size:11px;line-height:150%}.u-tooltip-data__value{font-weight:700;padding:4px}.u-tooltip__info{grid-gap:4px;display:grid}.u-tooltip__marker{height:12px;margin-right:4px;width:12px}.legendWrapper{grid-gap:20px;cursor:default;display:grid;grid-template-columns:repeat(auto-fit,minmax(400px,1fr));margin-top:20px;position:relative}.legendGroup{margin-bottom:24px}.legendGroupTitle{align-items:center;display:grid;font-size:11px;grid-template-columns:43px auto;padding:10px}.legendGroupQuery{grid-column:1/3;opacity:.6}.legendGroupLine{margin-right:10px}.legendItem{grid-gap:6px;align-items:start;background-color:#fff;cursor:pointer;display:inline-grid;grid-template-columns:auto auto;justify-content:start;padding:7px 50px 7px 10px;transition:.2s ease}.legendItemHide{opacity:.5;text-decoration:line-through}.legendItem:hover{background-color:rgba(0,0,0,.1)}.legendMarker{border-style:solid;border-width:2px;box-sizing:border-box;height:12px;transition:.2s ease;width:12px}.legendLabel{font-size:11px;font-weight:400;line-height:12px}.legendFreeFields{cursor:pointer;padding:3px}.legendFreeFields:hover{text-decoration:underline}.legendFreeFields:not(:last-child):after{content:","}.legendWrapperHotkey{align-items:center;display:flex;font-size:11px}.legendWrapperHotkey p{margin-right:20px}.legendWrapperHotkey code{word-wrap:break-word;background-color:#f2f2f2;border:1px solid #dedede;border-radius:2px;color:#0a0a0a;display:inline;font-size:10px;font-weight:400;max-width:100%;padding:4px 6px}
|
File diff suppressed because one or more lines are too long
2
app/vmselect/vmui/static/js/main.eef1a9bb.js
Normal file
2
app/vmselect/vmui/static/js/main.eef1a9bb.js
Normal file
File diff suppressed because one or more lines are too long
|
@ -658,11 +658,8 @@ func registerStorageMetrics() {
|
|||
metrics.NewGauge(`vm_cache_entries{type="storage/next_day_metric_ids"}`, func() float64 {
|
||||
return float64(m().NextDayMetricIDCacheSize)
|
||||
})
|
||||
metrics.NewGauge(`vm_cache_entries{type="storage/bigIndexBlocks"}`, func() float64 {
|
||||
return float64(tm().BigIndexBlocksCacheSize)
|
||||
})
|
||||
metrics.NewGauge(`vm_cache_entries{type="storage/smallIndexBlocks"}`, func() float64 {
|
||||
return float64(tm().SmallIndexBlocksCacheSize)
|
||||
metrics.NewGauge(`vm_cache_entries{type="storage/indexBlocks"}`, func() float64 {
|
||||
return float64(tm().IndexBlocksCacheSize)
|
||||
})
|
||||
metrics.NewGauge(`vm_cache_entries{type="indexdb/dataBlocks"}`, func() float64 {
|
||||
return float64(idbm().DataBlocksCacheSize)
|
||||
|
@ -689,11 +686,8 @@ func registerStorageMetrics() {
|
|||
metrics.NewGauge(`vm_cache_size_bytes{type="storage/metricName"}`, func() float64 {
|
||||
return float64(m().MetricNameCacheSizeBytes)
|
||||
})
|
||||
metrics.NewGauge(`vm_cache_size_bytes{type="storage/bigIndexBlocks"}`, func() float64 {
|
||||
return float64(tm().BigIndexBlocksCacheSizeBytes)
|
||||
})
|
||||
metrics.NewGauge(`vm_cache_size_bytes{type="storage/smallIndexBlocks"}`, func() float64 {
|
||||
return float64(tm().SmallIndexBlocksCacheSizeBytes)
|
||||
metrics.NewGauge(`vm_cache_size_bytes{type="storage/indexBlocks"}`, func() float64 {
|
||||
return float64(tm().IndexBlocksCacheSizeBytes)
|
||||
})
|
||||
metrics.NewGauge(`vm_cache_size_bytes{type="indexdb/dataBlocks"}`, func() float64 {
|
||||
return float64(idbm().DataBlocksCacheSizeBytes)
|
||||
|
@ -726,11 +720,8 @@ func registerStorageMetrics() {
|
|||
metrics.NewGauge(`vm_cache_size_max_bytes{type="storage/metricName"}`, func() float64 {
|
||||
return float64(m().MetricNameCacheSizeMaxBytes)
|
||||
})
|
||||
metrics.NewGauge(`vm_cache_size_max_bytes{type="storage/bigIndexBlocks"}`, func() float64 {
|
||||
return float64(tm().BigIndexBlocksCacheSizeMaxBytes)
|
||||
})
|
||||
metrics.NewGauge(`vm_cache_size_max_bytes{type="storage/smallIndexBlocks"}`, func() float64 {
|
||||
return float64(tm().SmallIndexBlocksCacheSizeMaxBytes)
|
||||
metrics.NewGauge(`vm_cache_size_max_bytes{type="storage/indexBlocks"}`, func() float64 {
|
||||
return float64(tm().IndexBlocksCacheSizeMaxBytes)
|
||||
})
|
||||
metrics.NewGauge(`vm_cache_size_max_bytes{type="indexdb/dataBlocks"}`, func() float64 {
|
||||
return float64(idbm().DataBlocksCacheSizeMaxBytes)
|
||||
|
@ -751,11 +742,8 @@ func registerStorageMetrics() {
|
|||
metrics.NewGauge(`vm_cache_requests_total{type="storage/metricName"}`, func() float64 {
|
||||
return float64(m().MetricNameCacheRequests)
|
||||
})
|
||||
metrics.NewGauge(`vm_cache_requests_total{type="storage/bigIndexBlocks"}`, func() float64 {
|
||||
return float64(tm().BigIndexBlocksCacheRequests)
|
||||
})
|
||||
metrics.NewGauge(`vm_cache_requests_total{type="storage/smallIndexBlocks"}`, func() float64 {
|
||||
return float64(tm().SmallIndexBlocksCacheRequests)
|
||||
metrics.NewGauge(`vm_cache_requests_total{type="storage/indexBlocks"}`, func() float64 {
|
||||
return float64(tm().IndexBlocksCacheRequests)
|
||||
})
|
||||
metrics.NewGauge(`vm_cache_requests_total{type="indexdb/dataBlocks"}`, func() float64 {
|
||||
return float64(idbm().DataBlocksCacheRequests)
|
||||
|
@ -779,11 +767,8 @@ func registerStorageMetrics() {
|
|||
metrics.NewGauge(`vm_cache_misses_total{type="storage/metricName"}`, func() float64 {
|
||||
return float64(m().MetricNameCacheMisses)
|
||||
})
|
||||
metrics.NewGauge(`vm_cache_misses_total{type="storage/bigIndexBlocks"}`, func() float64 {
|
||||
return float64(tm().BigIndexBlocksCacheMisses)
|
||||
})
|
||||
metrics.NewGauge(`vm_cache_misses_total{type="storage/smallIndexBlocks"}`, func() float64 {
|
||||
return float64(tm().SmallIndexBlocksCacheMisses)
|
||||
metrics.NewGauge(`vm_cache_misses_total{type="storage/indexBlocks"}`, func() float64 {
|
||||
return float64(tm().IndexBlocksCacheMisses)
|
||||
})
|
||||
metrics.NewGauge(`vm_cache_misses_total{type="indexdb/dataBlocks"}`, func() float64 {
|
||||
return float64(idbm().DataBlocksCacheMisses)
|
||||
|
|
|
@ -3,11 +3,11 @@ import {getQueryOptions, getQueryRangeUrl, getQueryUrl} from "../../../../api/qu
|
|||
import {useAppState} from "../../../../state/common/StateContext";
|
||||
import {InstantMetricResult, MetricBase, MetricResult} from "../../../../api/types";
|
||||
import {isValidHttpUrl} from "../../../../utils/url";
|
||||
import {useAuthState} from "../../../../state/auth/AuthStateContext";
|
||||
import {ErrorTypes} from "../../../../types";
|
||||
import {useGraphState} from "../../../../state/graph/GraphStateContext";
|
||||
import {getAppModeEnable, getAppModeParams} from "../../../../utils/app-mode";
|
||||
import throttle from "lodash.throttle";
|
||||
import {DisplayType} from "../DisplayTypeSwitch";
|
||||
|
||||
const appModeEnable = getAppModeEnable();
|
||||
const {serverURL: appServerUrl} = getAppModeParams();
|
||||
|
@ -22,7 +22,6 @@ export const useFetchQuery = (): {
|
|||
} => {
|
||||
const {query, displayType, serverUrl, time: {period}, queryControls: {nocache}} = useAppState();
|
||||
|
||||
const {basicData, bearerData, authMethod} = useAuthState();
|
||||
const {customStep} = useGraphState();
|
||||
|
||||
const [queryOptions, setQueryOptions] = useState([]);
|
||||
|
@ -39,22 +38,14 @@ export const useFetchQuery = (): {
|
|||
}
|
||||
}, [error]);
|
||||
|
||||
const fetchData = async (fetchUrl: string[] | undefined) => {
|
||||
const fetchData = async (fetchUrl: string[] | undefined, fetchQueue: AbortController[], displayType: DisplayType) => {
|
||||
if (!fetchUrl?.length) return;
|
||||
const controller = new AbortController();
|
||||
setFetchQueue([...fetchQueue, controller]);
|
||||
setIsLoading(true);
|
||||
|
||||
const headers = new Headers();
|
||||
if (authMethod === "BASIC_AUTH") {
|
||||
headers.set("Authorization", "Basic " + btoa(`${basicData?.login || ""}:${basicData?.password || ""}`));
|
||||
}
|
||||
if (authMethod === "BEARER_AUTH") {
|
||||
headers.set("Authorization", bearerData?.token || "");
|
||||
}
|
||||
|
||||
try {
|
||||
const responses = await Promise.all(fetchUrl.map(url => fetch(url, {headers, signal: controller.signal})));
|
||||
const responses = await Promise.all(fetchUrl.map(url => fetch(url, {signal: controller.signal})));
|
||||
const tempData = [];
|
||||
let counter = 1;
|
||||
for await (const response of responses) {
|
||||
|
@ -83,8 +74,9 @@ export const useFetchQuery = (): {
|
|||
const throttledFetchData = useCallback(throttle(fetchData, 300), []);
|
||||
|
||||
const fetchOptions = async () => {
|
||||
if (!serverUrl) return;
|
||||
const url = getQueryOptions(serverUrl);
|
||||
const server = appModeEnable ? appServerUrl : serverUrl;
|
||||
if (!server) return;
|
||||
const url = getQueryOptions(server);
|
||||
|
||||
try {
|
||||
const response = await fetch(url);
|
||||
|
@ -121,7 +113,7 @@ export const useFetchQuery = (): {
|
|||
|
||||
// TODO: this should depend on query as well, but need to decide when to do the request. Doing it on each query change - looks to be a bad idea. Probably can be done on blur
|
||||
useEffect(() => {
|
||||
throttledFetchData(fetchUrl);
|
||||
throttledFetchData(fetchUrl, fetchQueue, displayType);
|
||||
}, [fetchUrl]);
|
||||
|
||||
useEffect(() => {
|
||||
|
|
|
@ -97,7 +97,7 @@ export const TimeSelector: FC = () => {
|
|||
label="From"
|
||||
ampm={false}
|
||||
value={from}
|
||||
onChange={date => dispatch({type: "SET_FROM", payload: date as unknown as Date})}
|
||||
onChange={date => date && dispatch({type: "SET_FROM", payload: date as unknown as Date})}
|
||||
onError={console.log}
|
||||
inputFormat={formatDate}
|
||||
mask="____-__-__ __:__:__"
|
||||
|
@ -110,7 +110,7 @@ export const TimeSelector: FC = () => {
|
|||
label="To"
|
||||
ampm={false}
|
||||
value={until}
|
||||
onChange={date => dispatch({type: "SET_UNTIL", payload: date as unknown as Date})}
|
||||
onChange={date => date && dispatch({type: "SET_UNTIL", payload: date as unknown as Date})}
|
||||
onError={console.log}
|
||||
inputFormat={formatDate}
|
||||
mask="____-__-__ __:__:__"
|
||||
|
|
|
@ -1,9 +1,10 @@
|
|||
import React, {FC, useMemo} from "preact/compat";
|
||||
import React, {FC, useMemo, useState} from "preact/compat";
|
||||
import {hexToRGB} from "../../utils/color";
|
||||
import {useAppState} from "../../state/common/StateContext";
|
||||
import {LegendItem} from "../../utils/uplot/types";
|
||||
import "./legend.css";
|
||||
import {getDashLine} from "../../utils/uplot/helpers";
|
||||
import Tooltip from "@mui/material/Tooltip";
|
||||
|
||||
export interface LegendProps {
|
||||
labels: LegendItem[];
|
||||
|
@ -13,10 +14,18 @@ export interface LegendProps {
|
|||
const Legend: FC<LegendProps> = ({labels, onChange}) => {
|
||||
const {query} = useAppState();
|
||||
|
||||
const [copiedValue, setCopiedValue] = useState("");
|
||||
|
||||
const groups = useMemo(() => {
|
||||
return Array.from(new Set(labels.map(l => l.group)));
|
||||
}, [labels]);
|
||||
|
||||
const handleClickFreeField = async (val: string, id: string) => {
|
||||
await navigator.clipboard.writeText(val);
|
||||
setCopiedValue(id);
|
||||
setTimeout(() => setCopiedValue(""), 2000);
|
||||
};
|
||||
|
||||
return <>
|
||||
<div className="legendWrapper">
|
||||
{groups.map((group) => <div className="legendGroup" key={group}>
|
||||
|
@ -39,7 +48,25 @@ const Legend: FC<LegendProps> = ({labels, onChange}) => {
|
|||
borderColor: legendItem.color,
|
||||
backgroundColor: `rgba(${hexToRGB(legendItem.color)}, 0.1)`
|
||||
}}/>
|
||||
<div className="legendLabel">{legendItem.label}</div>
|
||||
<div className="legendLabel">
|
||||
{legendItem.freeFormFields.__name__ || `Query ${legendItem.group} result`}
|
||||
{!!Object.keys(legendItem.freeFormFields).length && <>
|
||||
 {
|
||||
{Object.keys(legendItem.freeFormFields).filter(f => f !== "__name__").map((f) => {
|
||||
const freeField = `${f}="${legendItem.freeFormFields[f]}"`;
|
||||
const fieldId = `${legendItem.group}.${legendItem.label}.${freeField}`;
|
||||
return <Tooltip arrow key={f} open={copiedValue === fieldId} title={"Copied!"}>
|
||||
<span className="legendFreeFields" onClick={(e) => {
|
||||
e.stopPropagation();
|
||||
handleClickFreeField(freeField, fieldId);
|
||||
}}>
|
||||
{f}: {legendItem.freeFormFields[f]}
|
||||
</span>
|
||||
</Tooltip>;
|
||||
})}
|
||||
}
|
||||
</>}
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
|
|
|
@ -34,7 +34,7 @@
|
|||
grid-gap: 6px;
|
||||
align-items: start;
|
||||
justify-content: start;
|
||||
padding: 5px 10px;
|
||||
padding: 7px 50px 7px 10px;
|
||||
background-color: #FFF;
|
||||
cursor: pointer;
|
||||
transition: 0.2s ease;
|
||||
|
@ -56,14 +56,27 @@
|
|||
border-style: solid;
|
||||
box-sizing: border-box;
|
||||
transition: 0.2s ease;
|
||||
margin: 3px 0;
|
||||
}
|
||||
|
||||
.legendLabel {
|
||||
font-size: 11px;
|
||||
line-height: 12px;
|
||||
font-weight: normal;
|
||||
}
|
||||
|
||||
.legendFreeFields {
|
||||
padding: 3px;
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.legendFreeFields:hover {
|
||||
text-decoration: underline;
|
||||
}
|
||||
|
||||
.legendFreeFields:not(:last-child):after {
|
||||
content: ",";
|
||||
}
|
||||
|
||||
.legendWrapperHotkey {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
|
|
|
@ -10,6 +10,7 @@ export const getSeriesItem = (d: MetricResult, hideSeries: string[]): Series =>
|
|||
return {
|
||||
label,
|
||||
dash: getDashLine(d.group),
|
||||
class: JSON.stringify(d.metric),
|
||||
width: 1.4,
|
||||
stroke: getColorLine(d.group, label),
|
||||
show: !includesHideSeries(label, d.group, hideSeries),
|
||||
|
@ -25,7 +26,8 @@ export const getLegendItem = (s: Series, group: number): LegendItem => ({
|
|||
group,
|
||||
label: s.label || "",
|
||||
color: s.stroke as string,
|
||||
checked: s.show || false
|
||||
checked: s.show || false,
|
||||
freeFormFields: JSON.parse(s.class || "{}"),
|
||||
});
|
||||
|
||||
export const getHideSeries = ({hideSeries, legend, metaKey, series}: HideSeriesArgs): string[] => {
|
||||
|
|
|
@ -36,4 +36,5 @@ export interface LegendItem {
|
|||
label: string;
|
||||
color: string;
|
||||
checked: boolean;
|
||||
freeFormFields: {[key: string]: string};
|
||||
}
|
|
@ -6,13 +6,18 @@ sort: 15
|
|||
|
||||
## tip
|
||||
|
||||
* BUGFIX: return proper results from `highestMax()` function at [Graphite render API](https://docs.victoriametrics.com/#graphite-render-api-usage). Previously it was incorrectly returning timeseries with min peaks instead of max peaks.
|
||||
* BUGFIX: properly limit indexdb cache sizes. Previously they could exceed values set via `-memory.allowedPercent` and/or `-memory.allowedBytes` when `indexdb` contained many data parts. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2007).
|
||||
* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix a bug, which could break time range picker when editing `From` or `To` input fields. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2080).
|
||||
* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix a bug, which could break switching between `graph`, `json` and `table` views. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2084).
|
||||
|
||||
|
||||
## [v1.72.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.72.0)
|
||||
|
||||
Released at 18-01-2022
|
||||
|
||||
* FEATURE: [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html): add support for `@` modifier, which is enabled by default in Prometheus starting from [Prometheus v2.33.0](https://github.com/prometheus/prometheus/pull/10121). See [these docs](https://prometheus.io/docs/prometheus/latest/querying/basics/#modifier) and [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1348). VictoriaMetrics extends `@` modifier with the following additional features:
|
||||
* It can contain arbitrary expression. For example, `foo @ (end() - 1h)` would return `foo` value at `end - 1 hour` timestamp on the selected time range `[start ... end]`. Another example: `foo @ now() - 10m` would return `foo` value 10 minutes ago from the current time.
|
||||
* It can contain arbitrary expression. For example, `foo @ (end() - 1h)` would return `foo` value at `end - 1 hour` timestamp on the selected time range `[start ... end]`. Another example: `foo @ (now() - 10m)` would return `foo` value 10 minutes ago from the current time.
|
||||
* It can be put everywhere in the query. For example, `sum(foo) @ start()` would calculate `sum(foo)` at `start` timestamp on the selected time range `[start ... end]`.
|
||||
* FEATURE: [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html): add support for optional `keep_metric_names` modifier, which can be applied to all the [rollup functions](https://docs.victoriametrics.com/MetricsQL.html#rollup-functions) and [transform functions](https://docs.victoriametrics.com/MetricsQL.html#transform-functions). This modifier prevents from deleting metric names from function results. For example, `rate({__name__=~"foo|bar"}[5m]) keep_metric_names` leaves `foo` and `bar` metric names in `rate()` results. This feature provides an additional workaround for [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/949).
|
||||
* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add support for Kubernetes service discovery in the current namespace in the same way as [Prometheus does](https://github.com/prometheus/prometheus/pull/9881). For example, the following config limits pod discovery to the namespace where vmagent runs:
|
||||
|
@ -27,7 +32,7 @@ Released at 18-01-2022
|
|||
```
|
||||
|
||||
* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add `__meta_kubernetes_node_provider_id` label for discovered Kubernetes nodes in the same way as [Prometheus does](https://github.com/prometheus/prometheus/pull/9603).
|
||||
* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): log error message when remote storage returns 400 or 409 http errors. This should simplify detection and debugging of this case. See [this issue](vmagent_remotewrite_packets_dropped_total).
|
||||
* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): log error message when remote storage returns 400 or 409 http errors. This should simplify detection and debugging of this case. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1911).
|
||||
* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): expose `promscrape_stale_samples_created_total` metric for monitoring the total number of created stale samples when scraping Prometheus targets. See [these docs](https://docs.victoriametrics.com/vmagent.html#prometheus-staleness-markers) for the information on when stale samples (aka staleness markers) can be created.
|
||||
* FEATURE: [vmrestore](https://docs.victoriametrics.com/vmrestore.html): store `restore-in-progress` file in `-dst` directory while `vmrestore` is running. This file is automatically deleted when `vmrestore` is successfully finished. This helps detecting incompletely restored data on VictoriaMetrics start. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1958).
|
||||
* FEATURE: [vmctl](https://docs.victoriametrics.com/vmctl.html): print the last sample timestamp when the data migration is interrupted either by user or by error. This helps continuing the data migration from the interruption moment. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1236).
|
||||
|
|
|
@ -491,7 +491,7 @@ Below is the output for `/path/to/vminsert -help`:
|
|||
-envflag.prefix string
|
||||
Prefix for environment variables if -envflag.enable is set
|
||||
-eula
|
||||
By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf
|
||||
By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/legal/eula/
|
||||
-fs.disableMmap
|
||||
Whether to use pread() instead of mmap() for reading data files. By default mmap() is used for 64-bit arches and pread() is used for 32-bit arches, since they cannot read data files bigger than 2^32 bytes in memory. mmap() is usually faster for reading small data chunks than pread()
|
||||
-graphiteListenAddr string
|
||||
|
@ -614,7 +614,7 @@ Below is the output for `/path/to/vmselect -help`:
|
|||
-envflag.prefix string
|
||||
Prefix for environment variables if -envflag.enable is set
|
||||
-eula
|
||||
By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf
|
||||
By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/legal/eula/
|
||||
-fs.disableMmap
|
||||
Whether to use pread() instead of mmap() for reading data files. By default mmap() is used for 64-bit arches and pread() is used for 32-bit arches, since they cannot read data files bigger than 2^32 bytes in memory. mmap() is usually faster for reading small data chunks than pread()
|
||||
-graphiteTrimTimestamp duration
|
||||
|
@ -742,7 +742,7 @@ Below is the output for `/path/to/vmstorage -help`:
|
|||
-envflag.prefix string
|
||||
Prefix for environment variables if -envflag.enable is set
|
||||
-eula
|
||||
By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf
|
||||
By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/legal/eula/
|
||||
-finalMergeDelay duration
|
||||
The delay before starting final merge for per-month partition after no new data is ingested into it. Final merge may require additional disk IO and CPU resources. Final merge may increase query speed and reduce disk space usage in some cases. Zero value disables final merge
|
||||
-forceFlushAuthKey string
|
||||
|
|
|
@ -153,7 +153,7 @@ The main differences between Cortex and VictoriaMetrics:
|
|||
- Thanos re-uses Prometheus source code, while VictoriaMetrics is written from scratch.
|
||||
- VictoriaMetrics accepts data via the [standard remote_write API for Prometheus](https://prometheus.io/docs/practices/remote_write/),
|
||||
while Thanos uses a non-standard [sidecar](https://github.com/thanos-io/thanos/blob/master/docs/components/sidecar.md) which must run alongside each Prometheus instance.
|
||||
- The Thanos sidecar requires disabling data compaction in Prometheus, which may hurt Prometheus performance and increase RAM usage. See [these docs](https://thanos.io/components/sidecar.md/) for more details.
|
||||
- The Thanos sidecar requires disabling data compaction in Prometheus, which may hurt Prometheus performance and increase RAM usage. See [these docs](https://thanos.io/tip/components/sidecar.md/) for more details.
|
||||
- Thanos stores data in object storage (Amazon S3 or Google GCS), while VictoriaMetrics stores data in block storage
|
||||
([GCP persistent disks](https://cloud.google.com/compute/docs/disks#pdspecs), Amazon EBS or bare metal HDD).
|
||||
While object storage is usually less expensive, block storage provides much lower latencies and higher throughput.
|
||||
|
|
|
@ -429,7 +429,7 @@ The `__graphite__` pseudo-label supports e.g. alternate regexp filters such as `
|
|||
|
||||
VictoriaMetrics supports [telnet put protocol](http://opentsdb.net/docs/build/html/api_telnet/put.html)
|
||||
and [HTTP /api/put requests](http://opentsdb.net/docs/build/html/api_http/put.html) for ingesting OpenTSDB data.
|
||||
The same protocol is used for [ingesting data in KairosDB](https://kairosdb.github.io/docs/build/html/PushingData.html).
|
||||
The same protocol is used for [ingesting data in KairosDB](https://kairosdb.github.io/docs/PushingData.html).
|
||||
|
||||
### Sending data via `telnet put` protocol
|
||||
|
||||
|
@ -1604,7 +1604,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
|
|||
-envflag.prefix string
|
||||
Prefix for environment variables if -envflag.enable is set
|
||||
-eula
|
||||
By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf
|
||||
By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/legal/eula/
|
||||
-finalMergeDelay duration
|
||||
The delay before starting final merge for per-month partition after no new data is ingested into it. Final merge may require additional disk IO and CPU resources. Final merge may increase query speed and reduce disk space usage in some cases. Zero value disables final merge
|
||||
-forceFlushAuthKey string
|
||||
|
|
|
@ -22,7 +22,7 @@ sort: 17
|
|||
- snapcraft binary, can be installed with commands:
|
||||
for MacOS `brew install snapcraft` and [install mutipass](https://discourse.ubuntu.com/t/installing-multipass-on-macos/8329),
|
||||
for Ubuntu - `sudo snap install snapcraft --classic`
|
||||
- login with `snapcraft login`
|
||||
- exported snapcraft login to `~/.snap/login.json` with `snapcraft export-login login.json && mkdir -p ~/.snap && mv login.json ~/.snap/`
|
||||
- already created release at github (it operates `git describe` version, so git tag must be annotated).
|
||||
|
||||
0. checkout to the latest git tag for single-node version.
|
||||
|
|
|
@ -433,7 +433,7 @@ The `__graphite__` pseudo-label supports e.g. alternate regexp filters such as `
|
|||
|
||||
VictoriaMetrics supports [telnet put protocol](http://opentsdb.net/docs/build/html/api_telnet/put.html)
|
||||
and [HTTP /api/put requests](http://opentsdb.net/docs/build/html/api_http/put.html) for ingesting OpenTSDB data.
|
||||
The same protocol is used for [ingesting data in KairosDB](https://kairosdb.github.io/docs/build/html/PushingData.html).
|
||||
The same protocol is used for [ingesting data in KairosDB](https://kairosdb.github.io/docs/PushingData.html).
|
||||
|
||||
### Sending data via `telnet put` protocol
|
||||
|
||||
|
@ -1608,7 +1608,7 @@ Pass `-help` to VictoriaMetrics in order to see the list of supported command-li
|
|||
-envflag.prefix string
|
||||
Prefix for environment variables if -envflag.enable is set
|
||||
-eula
|
||||
By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf
|
||||
By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/legal/eula/
|
||||
-finalMergeDelay duration
|
||||
The delay before starting final merge for per-month partition after no new data is ingested into it. Final merge may require additional disk IO and CPU resources. Final merge may increase query speed and reduce disk space usage in some cases. Zero value disables final merge
|
||||
-forceFlushAuthKey string
|
||||
|
|
19
docs/operator/README.md
Normal file
19
docs/operator/README.md
Normal file
|
@ -0,0 +1,19 @@
|
|||
---
|
||||
sort: 22
|
||||
---
|
||||
|
||||
# VictoriaMetrics Operator
|
||||
|
||||
1. [VictoriaMetrics Operator](VictoriaMetrics-Operator.html)
|
||||
2. [Additional Scrape Configuration](additional-scrape.html)
|
||||
3. [API Docs](api.html)
|
||||
4. [Authorization and exposing components](auth.html)
|
||||
5. [vmbackupmanager](backups.html)
|
||||
6. [Design](design.html)
|
||||
7. [High Availability](high-availability.html)
|
||||
8. [VMAlert, VMAgent, VMAlertmanager, VMSingle version](managing-versions.html)
|
||||
9. [Victoria Metrics Operator Quick Start](quick-start.html)
|
||||
10. [VMAgent relabel](relabeling.html)
|
||||
11. [CRD Validation](resources-validation.html)
|
||||
12. [Security](security.html)
|
||||
13. [Auto Generated vars for package config](vars.html)
|
79
docs/operator/VictoriaMetrics-Operator.md
Normal file
79
docs/operator/VictoriaMetrics-Operator.md
Normal file
|
@ -0,0 +1,79 @@
|
|||
---
|
||||
sort: 1
|
||||
---
|
||||
|
||||
# VictoriaMetrics operator
|
||||
|
||||
## Overview
|
||||
|
||||
Design and implementation inspired by [prometheus-operator](https://github.com/prometheus-operator/prometheus-operator). It's great a tool for managing monitoring configuration of your applications. VictoriaMetrics operator has api capability with it.
|
||||
So you can use familiar CRD objects: `ServiceMonitor`, `PodMonitor`, `PrometheusRule` and `Probe`. Or you can use VictoriaMetrics CRDs:
|
||||
- `VMServiceScrape` - defines scraping metrics configuration from pods backed by services.
|
||||
- `VMPodScrape` - defines scraping metrics configuration from pods.
|
||||
- `VMRule` - defines alerting or recording rules.
|
||||
- `VMProbe` - defines a probing configuration for targets with blackbox exporter.
|
||||
|
||||
Besides, operator allows you to manage VictoriaMetrics applications inside kubernetes cluster and simplifies this process [quick-start](/Operator/quick-start.html)
|
||||
With CRD (Custom Resource Definition) you can define application configuration and apply it to your cluster [crd-objects](/Operator/api.html).
|
||||
|
||||
Operator simplifies VictoriaMetrics cluster installation, upgrading and managing.
|
||||
|
||||
It has integration with VictoriaMetrics `vmbackupmanager` - advanced tools for making backups. Check backup [docs](/Operator/backups.html)
|
||||
|
||||
## Use cases
|
||||
|
||||
For kubernetes-cluster administrators, it simplifies installation, configuration, management for `VictoriaMetrics` application. And the main feature of operator - is ability to delegate applications monitoring configuration to the end-users.
|
||||
|
||||
For applications developers, its great possibility for managing observability of applications. You can define metrics scraping and alerting configuration for your application and manage it with an application deployment process. Just define app_deployment.yaml, app_vmpodscrape.yaml and app_vmrule.yaml. That's it, you can apply it to a kubernetes cluster. Check [quick-start](/Operator/quick-start.html) for an example.
|
||||
|
||||
## Operator vs helm-chart
|
||||
|
||||
VictoriaMetrics provides [helm charts](https://github.com/VictoriaMetrics/helm-charts). Operator makes the same, simplifies it and provides advanced features.
|
||||
|
||||
|
||||
## Configuration
|
||||
|
||||
Operator configured by env variables, list of it can be found at [link](/vars.html)
|
||||
|
||||
It defines default configuration options, like images for components, timeouts, features.
|
||||
|
||||
|
||||
## Kubernetes' compatibility versions
|
||||
|
||||
operator tested at kubernetes versions
|
||||
from 1.16 to 1.22
|
||||
|
||||
For clusters version below 1.16 you must use legacy CRDs from [path](config/crd/legacy)
|
||||
and disable CRD controller with flag: `--controller.disableCRDOwnership=true`
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- cannot apply crd at kubernetes 1.18 + version and kubectl reports error:
|
||||
```bash
|
||||
Error from server (Invalid): error when creating "release/crds/crd.yaml": CustomResourceDefinition.apiextensions.k8s.io "vmalertmanagers.operator.victoriametrics.com" is invalid: [spec.validation.openAPIV3Schema.properties[spec].properties[initContainers].items.properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property, spec.validation.openAPIV3Schema.properties[spec].properties[containers].items.properties[ports].items.properties[protocol].default: Required value: this property is in x-kubernetes-list-map-keys, so it must have a default or be a required property]
|
||||
Error from server (Invalid): error when creating "release/crds/crd.yaml": CustomResourceDefinition.apiextensions.k8s.io "vmalerts.operator.victoriametrics.com" is invalid: [
|
||||
```
|
||||
upgrade to the latest release version. There is a bug with kubernetes objects at the early releases.
|
||||
|
||||
|
||||
## Development
|
||||
|
||||
- operator-sdk verson v1.0.0 + [https://github.com/operator-framework/operator-sdk]
|
||||
- golang 1.15 +
|
||||
- minikube or kind
|
||||
|
||||
start:
|
||||
```bash
|
||||
make run
|
||||
```
|
||||
|
||||
for test execution run:
|
||||
```bash
|
||||
#unit tests
|
||||
|
||||
make test
|
||||
|
||||
# you need minikube or kind for e2e, do not run it on live cluster
|
||||
#e2e tests with local binary
|
||||
make e2e-local
|
||||
```
|
85
docs/operator/additional-scrape.MD
Normal file
85
docs/operator/additional-scrape.MD
Normal file
|
@ -0,0 +1,85 @@
|
|||
---
|
||||
sort: 2
|
||||
---
|
||||
|
||||
# Additional Scrape Configuration
|
||||
|
||||
AdditionalScrapeConfigs allows specifying a key of a Secret containing
|
||||
additional Prometheus scrape configurations or define scrape configuration at CRD spec.
|
||||
Scrape configurations specified
|
||||
are appended to the configurations generated by the operator.
|
||||
|
||||
Job configurations specified must have the form as specified in the official
|
||||
[Prometheus documentation](
|
||||
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config).
|
||||
As scrape configs are appended, the user is responsible to make sure it is
|
||||
valid.
|
||||
|
||||
## Creating an additional configuration inline at CRD
|
||||
|
||||
Add needed scrape configuration directly to the vmagent spec.inlineScrapeConfig
|
||||
|
||||
```yaml
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMAgent
|
||||
metadata:
|
||||
name: example-vmagent
|
||||
spec:
|
||||
serviceScrapeSelector: {}
|
||||
replicas: 1
|
||||
serviceAccountName: vmagent
|
||||
inlineScrapeConfig: |
|
||||
- job_name: "prometheus"
|
||||
static_configs:
|
||||
- targets: ["localhost:9090"]
|
||||
remoteWrite:
|
||||
- url: "http://vmagent-example-vmsingle.default.svc:8429/api/v1/write"
|
||||
EOF
|
||||
```
|
||||
|
||||
NOTE: Do not use password and tokens with inlineScrapeConfig.
|
||||
|
||||
|
||||
## Creating an additional configuration with secret
|
||||
|
||||
First, you will need to create the additional configuration.
|
||||
Below we are making a simple "prometheus" config. Name this
|
||||
`prometheus-additional.yaml` or something similar.
|
||||
|
||||
```yaml
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: additional-scrape-configs
|
||||
stringData:
|
||||
prometheus-additional.yaml: |
|
||||
- job_name: "prometheus"
|
||||
static_configs:
|
||||
- targets: ["localhost:9090"]
|
||||
EOF
|
||||
```
|
||||
|
||||
Finally, reference this additional configuration in your `vmagent.yaml` CRD.
|
||||
|
||||
```yaml
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMAgent
|
||||
metadata:
|
||||
name: example-vmagent
|
||||
spec:
|
||||
serviceScrapeSelector: {}
|
||||
replicas: 1
|
||||
serviceAccountName: vmagent
|
||||
additionalScrapeConfigs:
|
||||
name: additional-scrape-configs
|
||||
key: prometheus-additional.yaml
|
||||
remoteWrite:
|
||||
- url: "http://vmagent-example-vmsingle.default.svc:8429/api/v1/write"
|
||||
EOF
|
||||
```
|
||||
|
||||
NOTE: Use only one secret for ALL additional scrape configurations.
|
||||
|
2004
docs/operator/api.MD
Normal file
2004
docs/operator/api.MD
Normal file
File diff suppressed because it is too large
Load diff
182
docs/operator/auth.MD
Normal file
182
docs/operator/auth.MD
Normal file
|
@ -0,0 +1,182 @@
|
|||
---
|
||||
sort: 4
|
||||
---
|
||||
|
||||
# Authorization and exposing components
|
||||
|
||||
## Exposing components
|
||||
|
||||
|
||||
CRD objects doesn't have `ingress` configuration. Instead, you can use `VMAuth` as proxy between ingress-controller and VM app components.
|
||||
It adds missing authorization and access control features and enforces it.
|
||||
|
||||
Access can be given with `VMUser` definition. It supports basic auth and bearer token authentication.
|
||||
|
||||
```yaml
|
||||
cat << EOF | kubectl apply -f -
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMAuth
|
||||
metadata:
|
||||
name: main-router
|
||||
spec:
|
||||
userNamespaceSelector: {}
|
||||
userSelector: {}
|
||||
ingress: {}
|
||||
EOF
|
||||
```
|
||||
|
||||
Advanced configuration with cert-manager annotations:
|
||||
```yaml
|
||||
cat << EOF | kubectl apply -f -
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMAuth
|
||||
metadata:
|
||||
name: router-main
|
||||
spec:
|
||||
podMetadata:
|
||||
labels:
|
||||
component: vmauth
|
||||
userSelector: {}
|
||||
userNamespaceSelector: {}
|
||||
replicaCount: 2
|
||||
resources:
|
||||
requests:
|
||||
cpu: "250m"
|
||||
memory: "350Mi"
|
||||
limits:
|
||||
cpu: "500m"
|
||||
memory: "850Mi"
|
||||
ingress:
|
||||
tlsSecretName: vmauth-tls
|
||||
annotations:
|
||||
cert-manager.io/cluster-issuer: base
|
||||
class_name: nginx
|
||||
tlsHosts:
|
||||
- vm-access.example.com
|
||||
EOF
|
||||
```
|
||||
|
||||
|
||||
simple static routing with read-only access to vmagent for username - `user-1` with password `Asafs124142`
|
||||
```yaml
|
||||
# curl vmauth:8427/metrics -u 'user-1:Asafs124142'
|
||||
cat << EOF | kubectl apply -f
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMUser
|
||||
metadata:
|
||||
name: user-1
|
||||
spec:
|
||||
password: Asafs124142
|
||||
targetRefs:
|
||||
- static:
|
||||
url: http://vmagent-base.default.svc:8429
|
||||
paths: ["/targets/api/v1","/targets","/metrics"]
|
||||
EOF
|
||||
```
|
||||
|
||||
With bearer token access:
|
||||
|
||||
```yaml
|
||||
# curl vmauth:8427/metrics -H 'Authorization: Bearer Asafs124142'
|
||||
cat << EOF | kubectl apply -f
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMUser
|
||||
metadata:
|
||||
name: user-2
|
||||
spec:
|
||||
bearerToken: Asafs124142
|
||||
targetRefs:
|
||||
- static:
|
||||
url: http://vmagent-base.default.svc:8429
|
||||
paths: ["/targets/api/v1","/targets","/metrics"]
|
||||
EOF
|
||||
```
|
||||
|
||||
It's also possible to use service discovery for objects:
|
||||
```yaml
|
||||
# curl vmauth:8427/metrics -H 'Authorization: Bearer Asafs124142'
|
||||
cat << EOF | kubectl apply -f
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMUser
|
||||
metadata:
|
||||
name: user-3
|
||||
spec:
|
||||
bearerToken: Asafs124142
|
||||
targetRefs:
|
||||
- crd:
|
||||
kind: VMAgent
|
||||
name: base
|
||||
namespace: default
|
||||
paths: ["/targets/api/v1","/targets","/metrics"]
|
||||
EOF
|
||||
```
|
||||
|
||||
Cluster components supports auto path generation for single tenant view:
|
||||
```yaml
|
||||
cat << EOF | kubectl apply -f -
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMUser
|
||||
metadata:
|
||||
name: vmuser-tenant-1
|
||||
spec:
|
||||
bearerToken: some-token
|
||||
targetRefs:
|
||||
- crd:
|
||||
kind: VMCluster/vminsert
|
||||
name: test-persistent
|
||||
namespace: default
|
||||
target_path_suffix: "/insert/1"
|
||||
- crd:
|
||||
kind: VMCluster/vmselect
|
||||
name: test-persistent
|
||||
namespace: default
|
||||
target_path_suffix: "/select/1"
|
||||
- static:
|
||||
url: http://vmselect-test-persistent.default.svc:8481/
|
||||
paths:
|
||||
- /internal/resetRollupResultCache
|
||||
EOF
|
||||
```
|
||||
|
||||
For each `VMUser` operator generates corresponding secret with username/password or bearer token at the same namespace as `VMUser`.
|
||||
|
||||
## Basic auth for targets
|
||||
|
||||
To authenticate a `VMServiceScrape`s over a metrics endpoint use [`basicAuth`](../api.html#basicauth)
|
||||
|
||||
```yaml
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMServiceScrape
|
||||
metadata:
|
||||
labels:
|
||||
k8s-apps: basic-auth-example
|
||||
name: basic-auth-example
|
||||
spec:
|
||||
endpoints:
|
||||
- basicAuth:
|
||||
password:
|
||||
name: basic-auth
|
||||
key: password
|
||||
username:
|
||||
name: basic-auth
|
||||
key: user
|
||||
port: metrics
|
||||
selector:
|
||||
matchLabels:
|
||||
app: myapp
|
||||
EOF
|
||||
```
|
||||
|
||||
```yaml
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: basic-auth
|
||||
data:
|
||||
password: dG9vcg== # toor
|
||||
user: YWRtaW4= # admin
|
||||
type: Opaque
|
||||
EOF
|
||||
```
|
136
docs/operator/backups.MD
Normal file
136
docs/operator/backups.MD
Normal file
|
@ -0,0 +1,136 @@
|
|||
---
|
||||
sort: 5
|
||||
---
|
||||
|
||||
# Backups
|
||||
|
||||
## vmbackupmanager
|
||||
|
||||
## vmbackupmanager is proprietary software.
|
||||
|
||||
Before using it, you must have signed contract and accept EULA https://victoriametrics.com/assets/VM_EULA.pdf
|
||||
|
||||
## Usage examples
|
||||
|
||||
`VMSingle` and `VMCluster` has built-in backup configuration, it uses `vmbackupmanager` - proprietary tool for backups.
|
||||
It supports incremental backups (hours, daily, etc) with popular object storages (aws s3, google cloud storage).
|
||||
|
||||
You can enable it with the simple configuration, define secret
|
||||
|
||||
```yaml
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: remote-storage-keys
|
||||
type: Opaque
|
||||
stringData:
|
||||
credentials: |-
|
||||
[default]
|
||||
aws_access_key_id = your_access_key_id
|
||||
aws_secret_access_key = your_secret_access_key
|
||||
---
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMSingle
|
||||
metadata:
|
||||
name: example-vmsingle
|
||||
spec:
|
||||
# Add fields here
|
||||
retentionPeriod: "1"
|
||||
vmBackup:
|
||||
# This is Enterprise Package feature you need to have signed contract to use it
|
||||
# and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf
|
||||
acceptEULA: true
|
||||
destination: "s3://your_bucket/folder"
|
||||
credentialsSecret:
|
||||
name: remote-storage-keys
|
||||
key: credentials
|
||||
---
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMCluster
|
||||
metadata:
|
||||
name: example-vmcluster-persistent
|
||||
spec:
|
||||
retentionPeriod: "4"
|
||||
replicationFactor: 2
|
||||
vmstorage:
|
||||
replicaCount: 2
|
||||
vmBackup:
|
||||
# This is Enterprise Package feature you need to have signed contract to use it
|
||||
# and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf
|
||||
acceptEULA: true
|
||||
destination: "s3://your_bucket/folder"
|
||||
credentialsSecret:
|
||||
name: remote-storage-keys
|
||||
key: credentials
|
||||
|
||||
```
|
||||
|
||||
NOTE: for cluster version operator adds suffix for `destination: "s3://your_bucket/folder"`, it becomes `"s3://your_bucket/folder/$(POD_NAME)"`.
|
||||
It's needed to make consistent backups for each storage node.
|
||||
|
||||
You can read more about backup configuration options and mechanics [here](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmbackup)
|
||||
|
||||
Possible configuration options for backup crd can be found at [link](/docs/api.html#vmbackup)
|
||||
|
||||
|
||||
## Restoring backups
|
||||
|
||||
|
||||
It can be done with [vmrestore](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmrestore)
|
||||
|
||||
There two ways:
|
||||
|
||||
First:
|
||||
You have to stop `VMSingle` by scaling it replicas to zero and manually restore data to the database directory.
|
||||
|
||||
Steps:
|
||||
1) edit `VMSingle` CRD, set replicaCount: 0
|
||||
2) wait until database stops
|
||||
3) ssh to some server, where you can mount `VMSingle` disk and mount it manually
|
||||
4) restore files with `vmrestore`
|
||||
5) umount disk
|
||||
6) edit `VMSingle` CRD, set replicaCount: 1
|
||||
7) wait database start
|
||||
|
||||
Second:
|
||||
|
||||
1) add init container with vmrestore command to `VMSingle` CRD, example:
|
||||
```yaml
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMSingle
|
||||
metadata:
|
||||
name: vmsingle-restored
|
||||
namespace: monitoring-system
|
||||
spec:
|
||||
initContainers:
|
||||
- name: vmrestore
|
||||
image: victoriametrics/vmrestore:latest
|
||||
volumeMounts:
|
||||
- mountPath: /victoria-metrics-data
|
||||
name: data
|
||||
- mountPath: /etc/vm/creds
|
||||
name: secret-remote-storage-keys
|
||||
readOnly: true
|
||||
args:
|
||||
- -storageDataPath=/victoria-metrics-data
|
||||
- -src=s3://your_bucket/folder/latest
|
||||
- -credsFilePath=/etc/vm/creds/credentials
|
||||
vmBackup:
|
||||
# This is Enterprise Package feature you need to have signed contract to use it
|
||||
# and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf
|
||||
acceptEULA: true
|
||||
destination: "s3://your_bucket/folder"
|
||||
extraArgs:
|
||||
runOnStart: "true"
|
||||
image:
|
||||
repository: victoriametrics/vmbackupmanager
|
||||
tag: v1.67.0-enterprise
|
||||
credentialsSecret:
|
||||
name: remote-storage-keys
|
||||
key: credentials
|
||||
|
||||
```
|
||||
2) apply it, and db will be restored from s3
|
||||
|
||||
3) remove initContainers and apply crd.
|
229
docs/operator/design.MD
Normal file
229
docs/operator/design.MD
Normal file
|
@ -0,0 +1,229 @@
|
|||
---
|
||||
sort: 6
|
||||
---
|
||||
|
||||
# Design
|
||||
|
||||
This document describes the design and interaction between the custom resource definitions (CRD) that the Victoria
|
||||
Metrics Operator introduces.
|
||||
|
||||
Operator introduces the following custom resources:
|
||||
|
||||
* [VMSingle](#vmsingle)
|
||||
* [VMCluster](#vmcluster)
|
||||
* [VMAgent](#vmagent)
|
||||
* [VMAlert](#vmalert)
|
||||
* [VMServiceScrape](#vmservicescrape)
|
||||
* [VMPodScrape](#vmpodscrape)
|
||||
* [VMAlertmanager](#vmalertmanager)
|
||||
* [VMAlertmanagerConfig](#vmalertmanagerconfig)
|
||||
* [VMRule](#vmrule)
|
||||
* [VMProbe](#vmprobe)
|
||||
* [VMNodeScrape](#vmodescrape)
|
||||
* [VMStaticScrape](#vmstaticscrape)
|
||||
* [VMAuth](#vmauth)
|
||||
* [VMUser](#vmuser)
|
||||
|
||||
## VMSingle
|
||||
|
||||
The `VMSingle` CRD declaratively defines a [single-node VM](https://github.com/VictoriaMetrics/VictoriaMetrics)
|
||||
installation to run in a Kubernetes cluster.
|
||||
|
||||
For each `VMSingle` resource, the Operator deploys a properly configured `Deployment` in the same namespace.
|
||||
The VMSingle `Pod`s are configured to mount an empty dir or `PersistentVolumeClaimSpec` for storing data.
|
||||
Deployment update strategy set to [recreate](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#recreate-deployment).
|
||||
No more than one replica allowed.
|
||||
|
||||
For each `VMSingle` resource, the Operator adds `Service` and `VMServiceScrape` in the same namespace prefixed with
|
||||
name `<VMSingle-name>`.
|
||||
|
||||
## VMCluster
|
||||
|
||||
The `VMCluster` CRD defines a [cluster version VM](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/cluster).
|
||||
|
||||
For each `VMCluster` resource, the Operator creates `VMStorage` as `StatefulSet`, `VMSelect` as `StatefulSet` and `VMInsert`
|
||||
as deployment. For `VMStorage` and `VMSelect` headless services are created. `VMInsert` is created as service with clusterIP.
|
||||
|
||||
There is a strict order for these objects creation and reconciliation:
|
||||
1. `VMStorage` is synced - the Operator waits until all its pods are ready;
|
||||
2. Then it syncs `VMSelect` with the same manner;
|
||||
3. `VMInsert` is the last object to sync.
|
||||
|
||||
All statefulsets are created with [OnDelete](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#on-delete)
|
||||
update type. It allows to manually manage the rolling update process for Operator by deleting pods one by one and waiting
|
||||
for the ready status.
|
||||
|
||||
Rolling update process may be configured by the operator env variables.
|
||||
The most important is `VM_PODWAITREADYTIMEOUT=80s` - it controls how long to wait for pod's ready status.
|
||||
|
||||
## VMAgent
|
||||
|
||||
The `VMAgent` CRD declaratively defines a desired [VMAgent](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmagent)
|
||||
setup to run in a Kubernetes cluster.
|
||||
|
||||
For each `VMAgent` resource Operator deploys a properly configured `Deployment` in the same namespace.
|
||||
The VMAgent `Pod`s are configured to mount a `Secret` prefixed with `<VMAgent-name>` containing the configuration
|
||||
for VMAgent.
|
||||
|
||||
For each `VMAgent` resource, the Operator adds `Service` and `VMServiceScrape` in the same namespace prefixed with
|
||||
name `<VMAgent-name>`.
|
||||
|
||||
The CRD specifies which `VMServiceScrape` should be covered by the deployed VMAgent instances based on label selection.
|
||||
The Operator then generates a configuration based on the included `VMServiceScrape`s and updates the `Secret` which
|
||||
contains the configuration. It continuously does so for all changes that are made to the `VMServiceScrape`s or the
|
||||
`VMAgent` resource itself.
|
||||
|
||||
If no selection of `VMServiceScrape`s is provided - Operator leaves management of the `Secret` to the user,
|
||||
so user can set custom configuration while still benefiting from the Operator's capabilities of managing VMAgent setups.
|
||||
|
||||
## VMAlert
|
||||
|
||||
The `VMAlert` CRD declaratively defines a desired [VMAlert](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/app/vmalert)
|
||||
setup to run in a Kubernetes cluster.
|
||||
|
||||
For each `VMAlert` resource, the Operator deploys a properly configured `Deployment` in the same namespace.
|
||||
The VMAlert `Pod`s are configured to mount a list of `Configmaps` prefixed with `<VMAlert-name>-number` containing
|
||||
the configuration for alerting rules.
|
||||
|
||||
For each `VMAlert` resource, the Operator adds `Service` and `VMServiceScrape` in the same namespace prefixed with
|
||||
name `<VMAlert-name>`.
|
||||
|
||||
The CRD specifies which `VMRule`s should be covered by the deployed VMAlert instances based on label selection.
|
||||
The Operator then generates a configuration based on the included `VMRule`s and updates the `Configmaps` containing
|
||||
the configuration. It continuously does so for all changes that are made to `VMRule`s or to the `VMAlert` resource itself.
|
||||
|
||||
Alerting rules are filtered by selector `ruleNamespaceSelector` in `VMAlert` CRD definition. For selecting rules from all
|
||||
namespaces you must specify it to empty value:
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
ruleNamespaceSelector: {}
|
||||
```
|
||||
|
||||
|
||||
## VMServiceScrape
|
||||
|
||||
The `VMServiceScrape` CRD allows to define a dynamic set of services for monitoring. Services
|
||||
and scraping configurations can be matched via label selections. This allows an organization to introduce conventions
|
||||
for how metrics should be exposed. Following these conventions new services will be discovered automatically without
|
||||
need to reconfigure.
|
||||
|
||||
Monitoring configuration based on `discoveryRole` setting. By default, `endpoints` is used to get objects from kubernetes api.
|
||||
Its also possible to use `discoveryRole: service` or `discoveryRole: endpointslices`
|
||||
|
||||
`Endpoints` objects are essentially lists of IP addresses.
|
||||
Typically, `Endpoints` objects are populated by `Service` object. `Service` object discovers `Pod`s by a label
|
||||
selector and adds those to the `Endpoints` object.
|
||||
|
||||
A `Service` may expose one or more service ports backed by a list of one or multiple endpoints pointing to
|
||||
specific `Pod`s. The same reflected in the respective `Endpoints` object as well.
|
||||
|
||||
The `VMServiceScrape` object discovers `Endpoints` objects and configures VMAgent to monitor `Pod`s.
|
||||
|
||||
The `Endpoints` section of the `VMServiceScrapeSpec` is used to configure which `Endpoints` ports should be scraped.
|
||||
For advanced use cases, one may want to monitor ports of backing `Pod`s, which are not a part of the service endpoints.
|
||||
Therefore, when specifying an endpoint in the `endpoints` section, they are strictly used.
|
||||
|
||||
> Note: `endpoints` (lowercase) is the field in the `VMServiceScrape` CRD, while `Endpoints` (capitalized) is the Kubernetes object kind.
|
||||
|
||||
Both `VMServiceScrape` and discovered targets may belong to any namespace. It is important for cross-namespace monitoring
|
||||
use cases, e.g. for meta-monitoring. Using the `serviceScrapeSelector` of the `VMAgentSpec`
|
||||
one can restrict the namespaces from which `VMServiceScrape`s are selected from by the respective VMAgent server.
|
||||
Using the `namespaceSelector` of the `VMServiceScrape` one can restrict the namespaces from which `Endpoints` can be
|
||||
discovered from. To discover targets in all namespaces the `namespaceSelector` has to be empty:
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
namespaceSelector: {}
|
||||
```
|
||||
|
||||
## VMPodScrape
|
||||
|
||||
The `VMPodScrape` CRD allows to declaratively define how a dynamic set of pods should be monitored.
|
||||
Use label selections to match pods for scraping. This allows an organization to introduce conventions
|
||||
for how metrics should be exposed. Following these conventions new services will be discovered automatically without
|
||||
need to reconfigure.
|
||||
|
||||
A `Pod` is a collection of one or more containers which can expose Prometheus metrics on a number of ports.
|
||||
|
||||
The `VMPodScrape` object discovers pods and generates the relevant scraping configuration.
|
||||
|
||||
The `PodMetricsEndpoints` section of the `VMPodScrapeSpec` is used to configure which ports of a pod are going to be
|
||||
scraped for metrics and with which parameters.
|
||||
|
||||
Both `VMPodScrapes` and discovered targets may belong to any namespace. It is important for cross-namespace monitoring
|
||||
use cases, e.g. for meta-monitoring. Using the `namespaceSelector` of the `VMPodScrapeSpec` one can restrict the
|
||||
namespaces from which `Pods` are discovered from. To discover targets in all namespaces the `namespaceSelector` has to
|
||||
be empty:
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
namespaceSelector:
|
||||
any: true
|
||||
```
|
||||
|
||||
## VMAlertmanager
|
||||
|
||||
The `VMAlertmanager` CRD declaratively defines a desired Alertmanager setup to run in a Kubernetes cluster.
|
||||
It provides options to configure replication and persistent storage.
|
||||
|
||||
For each `Alertmanager` resource, the Operator deploys a properly configured `StatefulSet` in the same namespace.
|
||||
The Alertmanager pods are configured to include a `Secret` called `<alertmanager-name>` which holds the used
|
||||
configuration file in the key `alertmanager.yaml`.
|
||||
|
||||
When there are two or more configured replicas the Operator runs the Alertmanager instances in high availability mode.
|
||||
|
||||
## VMAlertmanagerConfig
|
||||
|
||||
The `VMAlertmanagerConfig` provides way to configure `VMAlertmanager` configuration with CRD. It allows to define different configuration parts,
|
||||
which will be merged by operator into config. It behaves like other config parts - `VMServiceScrape` and etc.
|
||||
|
||||
## VMRule
|
||||
|
||||
The `VMRule` CRD declaratively defines a desired Prometheus rule to be consumed by one or more VMAlert instances.
|
||||
|
||||
Alerts and recording rules can be saved and applied as YAML files, and dynamically loaded without requiring any restart.
|
||||
|
||||
|
||||
## VMPrometheusConverter
|
||||
|
||||
By default, the Operator converts and updates existing prometheus-operator API objects:
|
||||
|
||||
`ServiceMonitor` into `VMServiceScrape`
|
||||
`PodMonitor` into `VMPodScrape`
|
||||
`PrometheusRule` into `VMRule`
|
||||
`Probe` into `VMProbe`
|
||||
Removing prometheus-operator API objects wouldn't delete any converted objects. So you can safely migrate or run
|
||||
two operators at the same time.
|
||||
|
||||
|
||||
## VMProbe
|
||||
|
||||
The `VMProbe` CRD provides probing target ability with a prober. The most common prober is [blackbox exporter](https://github.com/prometheus/blackbox_exporter).
|
||||
By specifying configuration at CRD, operator generates config for `VMAgent` and syncs it. Its possible to use static targets
|
||||
or use standard k8s discovery mechanism with `Ingress`.
|
||||
You have to configure blackbox exporter before you can use this feature. The second requirement is `VMAgent` selectors,
|
||||
it must match your `VMProbe` by label or namespace selector.
|
||||
|
||||
## VMNodeScrape
|
||||
|
||||
The `VMNodeScrape` CRD provides discovery mechanism for scraping metrics kubernetes nodes.
|
||||
By specifying configuration at CRD, operator generates config for `VMAgent` and syncs it. Its useful for cadvisor scraping,
|
||||
node-exporter or other node-based exporters. `VMAgent` nodeScrapeSelector must match `VMNodeScrape` labels.
|
||||
|
||||
## VMStaticScrape
|
||||
|
||||
The `VMStaticScrape` CRD provides mechanism for scraping metrics from static targets, configured by CRD targets.
|
||||
By specifying configuration at CRD, operator generates config for `VMAgent` and syncs it. It's useful for external targets management,
|
||||
when service-discovery is not available. `VMAgent` staticScrapeSelector must match `VMStaticScrape` labels.
|
||||
|
||||
## VMAuth
|
||||
|
||||
The `VMAuth` CRD provides mechanism for exposing application with authorization to outside world or to other applications inside kubernetes cluster.
|
||||
For first case, user can configure `ingress` setting at `VMAuth` CRD. For second one, operator will create secret with `username` and `password` at `VMUser` CRD name.
|
||||
So it will be possible to access this credentials from any application by targeting corresponding kubernetes secret.
|
||||
|
||||
## VMUser
|
||||
|
||||
The `VMUser` CRD describes user configuration, its authentication methods `basic auth` or `Authorization` header. User access permissions, with possible routing information.
|
||||
User can define routing target with `static` config, by entering target `url`, or with `CRDRef`, in this case, operator queries kubernetes API, retrieves information about CRD and builds proper url.
|
322
docs/operator/high-availability.MD
Normal file
322
docs/operator/high-availability.MD
Normal file
|
@ -0,0 +1,322 @@
|
|||
---
|
||||
sort: 7
|
||||
---
|
||||
|
||||
# High Availability
|
||||
|
||||
High availability is not only important for customer-facing software but if the monitoring infrastructure is not highly available, then there is a risk that operations people are not notified for alerts. Therefore high availability must be just as thought through for the monitoring stack, as for anything else.
|
||||
|
||||
## VMAgent
|
||||
|
||||
To run VMAgent in a highly available manner you have to configure deduplication at Victoria Metrics first [doc](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/Single-server-VictoriaMetrics.md#deduplication)
|
||||
|
||||
Then increase replicas for VMAgent.
|
||||
|
||||
create `VMSingle` with dedup flag
|
||||
```yaml
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMSingle
|
||||
metadata:
|
||||
name: example-vmsingle-persisted
|
||||
spec:
|
||||
retentionPeriod: "1"
|
||||
extraArgs:
|
||||
dedup.minScrapeInterval: 60s
|
||||
EOF
|
||||
```
|
||||
create `VMAgent` with 2 replicas
|
||||
```yaml
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMAgent
|
||||
metadata:
|
||||
name: example-vmagent
|
||||
spec:
|
||||
serviceScrapeNamespaceSelector: {}
|
||||
podScrapeNamespaceSelector: {}
|
||||
podScrapeSelector: {}
|
||||
serviceScrapeSelector: {}
|
||||
scrapeInterval: 60s
|
||||
vmAgentExternalLabelName: vmagent-ha
|
||||
replicaCount: 2
|
||||
remoteWrite:
|
||||
- url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429/api/v1/write"
|
||||
EOF
|
||||
|
||||
```
|
||||
|
||||
Sharding for `VMAgent` distributes scraping between multiple deployments of `VMAgent`.
|
||||
more info https://victoriametrics.github.io/vmagent.html#scraping-big-number-of-targets
|
||||
|
||||
Example usage:
|
||||
```yaml
|
||||
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMAgent
|
||||
metadata:
|
||||
name: example-vmagent
|
||||
spec:
|
||||
serviceScrapeNamespaceSelector: {}
|
||||
podScrapeNamespaceSelector: {}
|
||||
podScrapeSelector: {}
|
||||
serviceScrapeSelector: {}
|
||||
scrapeInterval: 60s
|
||||
vmAgentExternalLabelName: vmagent-ha
|
||||
shardCount: 5
|
||||
replicaCount: 2
|
||||
remoteWrite:
|
||||
- url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429/api/v1/write"
|
||||
EOF
|
||||
```
|
||||
|
||||
This configuration produces 5 deployments with 2 replicas at each. Each deployment has own shard num
|
||||
and scrapes only 1/5 of all targets.
|
||||
|
||||
|
||||
## VMAlert
|
||||
|
||||
It can be launched with multiple replicas without an additional configuration, alertmanager is responsible for alert deduplication.
|
||||
Note, if you want to use `VMAlert` with high-available `VMAlertmanager`, which has more then 1 replica. You have to specify all pod fqdns
|
||||
at `VMAlert.spec.notifiers.[url]`. Or you can use service discovery for notifier, examples:
|
||||
|
||||
alertmanager:
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: vmalertmanager-example-alertmanager
|
||||
labels:
|
||||
app: vm-operator
|
||||
type: Opaque
|
||||
stringData:
|
||||
alertmanager.yaml: |
|
||||
global:
|
||||
resolve_timeout: 5m
|
||||
route:
|
||||
group_by: ['job']
|
||||
group_wait: 30s
|
||||
group_interval: 5m
|
||||
repeat_interval: 12h
|
||||
receiver: 'webhook'
|
||||
receivers:
|
||||
- name: 'webhook'
|
||||
webhook_configs:
|
||||
- url: 'http://alertmanagerwh:30500/'
|
||||
|
||||
---
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMAlertmanager
|
||||
metadata:
|
||||
name: example
|
||||
namespace: default
|
||||
labels:
|
||||
usage: dedicated
|
||||
spec:
|
||||
replicaCount: 2
|
||||
configSecret: vmalertmanager-example-alertmanager
|
||||
configSelector: {}
|
||||
configNamespaceSelector: {}
|
||||
```
|
||||
vmalert with fqdns:
|
||||
```yaml
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMAlert
|
||||
metadata:
|
||||
name: example-ha
|
||||
namespace: default
|
||||
spec:
|
||||
datasource:
|
||||
url: http://vmsingle-example.default.svc:8429
|
||||
notifiers:
|
||||
- url: http://vmalertmanager-example-0.vmalertmanager-example.default.svc:9093
|
||||
- url: http://vmalertmanager-example-1.vmalertmanager-example.default.svc:9093
|
||||
```
|
||||
|
||||
vmalert with service discovery:
|
||||
```yaml
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMAlert
|
||||
metadata:
|
||||
name: example-ha
|
||||
namespace: default
|
||||
spec:
|
||||
datasource:
|
||||
url: http://vmsingle-example.default.svc:8429
|
||||
notifiers:
|
||||
- selector:
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- default
|
||||
labelSelector:
|
||||
matchLabels:
|
||||
usage: dedicated
|
||||
```
|
||||
|
||||
|
||||
## VMSingle
|
||||
|
||||
It doesn't support high availability by default, for such purpose use VMCluster or duplicate the setup.
|
||||
|
||||
|
||||
## VMCluster
|
||||
|
||||
Cluster version provides a full set of high availability features - metrics replication, node failover, horizontal scaling.
|
||||
|
||||
For using cluster version you have to create corresponding CRD object:
|
||||
|
||||
```yaml
|
||||
cat << EOF | kubectl apply -f -
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMCluster
|
||||
metadata:
|
||||
name: example-vmcluster-persistent
|
||||
spec:
|
||||
retentionPeriod: "4"
|
||||
replicationFactor: 2
|
||||
vmstorage:
|
||||
replicaCount: 2
|
||||
storageDataPath: "/vm-data"
|
||||
podMetadata:
|
||||
labels:
|
||||
owner: infra
|
||||
affinity:
|
||||
podAntiAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
- labelSelector:
|
||||
matchExpressions:
|
||||
- key: "app.kubernetes.io/name"
|
||||
operator: In
|
||||
values:
|
||||
- "vmstorage"
|
||||
topologyKey: "kubernetes.io/hostname"
|
||||
storage:
|
||||
volumeClaimTemplate:
|
||||
spec:
|
||||
resources:
|
||||
requests:
|
||||
storage: 10Gi
|
||||
resources:
|
||||
limits:
|
||||
cpu: "2"
|
||||
memory: 2048Mi
|
||||
vmselect:
|
||||
replicaCount: 2
|
||||
cacheMountPath: "/select-cache"
|
||||
podMetadata:
|
||||
labels:
|
||||
owner: infra
|
||||
affinity:
|
||||
podAntiAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
- labelSelector:
|
||||
matchExpressions:
|
||||
- key: "app.kubernetes.io/name"
|
||||
operator: In
|
||||
values:
|
||||
- "vmselect"
|
||||
topologyKey: "kubernetes.io/hostname"
|
||||
|
||||
storage:
|
||||
volumeClaimTemplate:
|
||||
spec:
|
||||
resources:
|
||||
requests:
|
||||
storage: 2Gi
|
||||
resources:
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: "500Mi"
|
||||
vminsert:
|
||||
replicaCount: 2
|
||||
podMetadata:
|
||||
labels:
|
||||
owner: infra
|
||||
affinity:
|
||||
podAntiAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
- labelSelector:
|
||||
matchExpressions:
|
||||
- key: "app.kubernetes.io/name"
|
||||
operator: In
|
||||
values:
|
||||
- "vminsert"
|
||||
topologyKey: "kubernetes.io/hostname"
|
||||
resources:
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: "500Mi"
|
||||
|
||||
EOF
|
||||
```
|
||||
|
||||
Then wait for the cluster becomes ready
|
||||
|
||||
```bash
|
||||
kubectl get vmclusters -w
|
||||
NAME INSERT COUNT STORAGE COUNT SELECT COUNT AGE STATUS
|
||||
example-vmcluster-persistent 2 2 2 2s expanding
|
||||
example-vmcluster-persistent 2 2 2 30s operational
|
||||
```
|
||||
|
||||
Get links for connection by executing command:
|
||||
|
||||
```bash
|
||||
kubectl get svc -l app.kubernetes.io/instance=example-vmcluster-persistent
|
||||
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
||||
vminsert-example-vmcluster-persistent ClusterIP 10.96.34.94 <none> 8480/TCP 69s
|
||||
vmselect-example-vmcluster-persistent ClusterIP None <none> 8481/TCP 79s
|
||||
vmstorage-example-vmcluster-persistent ClusterIP None <none> 8482/TCP,8400/TCP,8401/TCP 85s
|
||||
```
|
||||
|
||||
Now you can connect vmagent to vminsert and vmalert to vmselect
|
||||
|
||||
>NOTE do not forget to create rbac for vmagent
|
||||
|
||||
```yaml
|
||||
cat << EOF | kubectl apply -f -
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMAgent
|
||||
metadata:
|
||||
name: example-vmagent
|
||||
spec:
|
||||
serviceScrapeNamespaceSelector: {}
|
||||
serviceScrapeSelector: {}
|
||||
podScrapeNamespaceSelector: {}
|
||||
podScrapeSelector: {}
|
||||
# Add fields here
|
||||
replicaCount: 1
|
||||
remoteWrite:
|
||||
- url: "http://vminsert-example-vmcluster-persistent.default.svc.cluster.local:8480/insert/0/prometheus/api/v1/write"
|
||||
EOF
|
||||
```
|
||||
|
||||
Config for vmalert
|
||||
|
||||
```yaml
|
||||
cat << EOF | kubectl apply -f -
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMAlert
|
||||
metadata:
|
||||
name: example-vmalert
|
||||
spec:
|
||||
# Add fields here
|
||||
replicas: 1
|
||||
datasource:
|
||||
url: "http://vmselect-example-vmcluster-persistent.default.svc.cluster.local:8481/select/0/prometheus"
|
||||
notifier:
|
||||
url: "http://alertmanager-operated.default.svc:9093"
|
||||
evaluationInterval: "10s"
|
||||
ruleSelector: {}
|
||||
EOF
|
||||
```
|
||||
|
||||
|
||||
## Alertmanager
|
||||
|
||||
The final step of the high availability scheme is Alertmanager, when an alert triggers, actually fires alerts against *all* instances of an Alertmanager cluster.
|
||||
|
||||
The Alertmanager, starting with the `v0.5.0` release, ships with a high availability mode. It implements a gossip protocol to synchronize instances of an Alertmanager cluster regarding notifications that have been sent out, to prevent duplicate notifications. It is an AP (available and partition tolerant) system. Being an AP system means that notifications are guaranteed to be sent at least once.
|
||||
|
||||
The Victoria Metrics Operator ensures that Alertmanager clusters are properly configured to run highly available on Kubernetes.
|
BIN
docs/operator/logo.png
Normal file
BIN
docs/operator/logo.png
Normal file
Binary file not shown.
After Width: | Height: | Size: 15 KiB |
85
docs/operator/managing-versions.MD
Normal file
85
docs/operator/managing-versions.MD
Normal file
|
@ -0,0 +1,85 @@
|
|||
---
|
||||
sort: 8
|
||||
---
|
||||
|
||||
|
||||
# Managing application versions
|
||||
|
||||
## VMAlert, VMAgent, VMAlertmanager, VMSingle version
|
||||
|
||||
|
||||
for those objects you can specify following settings at `spec.Image`
|
||||
|
||||
for instance, to set `VMSingle` version add `spec.image.tag` name from [releases](https://github.com/VictoriaMetrics/VictoriaMetrics/releases)
|
||||
|
||||
```yaml
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMSingle
|
||||
metadata:
|
||||
name: example-vmsingle
|
||||
spec:
|
||||
image:
|
||||
repository: victoriametrics/victoria-metrics
|
||||
tag: v1.39.2
|
||||
pullPolicy: Always
|
||||
retentionPeriod: "1"
|
||||
EOF
|
||||
```
|
||||
|
||||
Also, you can specify `imagePullSecrets` if you are pulling images from private repo:
|
||||
```yaml
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMSingle
|
||||
metadata:
|
||||
name: example-vmsingle
|
||||
spec:
|
||||
imagePullSecrets:
|
||||
- name: my-repo-secret
|
||||
image:
|
||||
repository: my-repo-url/victoria-metrics
|
||||
tag: v1.39.2
|
||||
retentionPeriod: "1"
|
||||
EOF
|
||||
```
|
||||
|
||||
|
||||
# VMCluster
|
||||
|
||||
for `VMCluster` you can specify tag and repository setting per cluster object.
|
||||
But `imagePullSecrets` is global setting for all `VMCluster` specification.
|
||||
```yaml
|
||||
cat << EOF | kubectl apply -f -
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMCluster
|
||||
metadata:
|
||||
name: example-vmcluster
|
||||
spec:
|
||||
imagePullSecrets:
|
||||
- name: my-repo-secret
|
||||
# Add fields here
|
||||
retentionPeriod: "1"
|
||||
vmstorage:
|
||||
replicaCount: 2
|
||||
image:
|
||||
repository: victoriametrics/vmstorage
|
||||
tag: v1.39.2-cluster
|
||||
pullPolicy: Always
|
||||
vmselect:
|
||||
replicaCount: 2
|
||||
image:
|
||||
repository: victoriametrics/vmselect
|
||||
tag: v1.39.2-cluster
|
||||
pullPolicy: Always
|
||||
vminsert:
|
||||
replicaCount: 2
|
||||
image:
|
||||
repository: victoriametrics/vminsert
|
||||
tag: v1.39.2-cluster
|
||||
pullPolicy: Always
|
||||
EOF
|
||||
```
|
||||
|
||||
|
||||
|
1519
docs/operator/quick-start.MD
Normal file
1519
docs/operator/quick-start.MD
Normal file
File diff suppressed because it is too large
Load diff
242
docs/operator/relabeling.MD
Normal file
242
docs/operator/relabeling.MD
Normal file
|
@ -0,0 +1,242 @@
|
|||
---
|
||||
sort: 10
|
||||
---
|
||||
|
||||
# Relabeling
|
||||
|
||||
## VMAgent relabel
|
||||
|
||||
|
||||
`VMAgent` supports global relabeling for all metrics and per remoteWrite target relabel config.
|
||||
|
||||
> Note in some cases, you don't need relabeling,
|
||||
> key=value label pairs can be added to the all scrapped metrics with `spec.externalLabels` for `VMAgent`.
|
||||
>
|
||||
```yaml
|
||||
# simple label add config
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMAgent
|
||||
metadata:
|
||||
name: stack
|
||||
spec:
|
||||
externalLabels:
|
||||
clusterid: some_cluster
|
||||
```
|
||||
|
||||
|
||||
|
||||
It supports relabeling with custom configMap or inline defined at CRD
|
||||
|
||||
## Configmap example
|
||||
|
||||
Quick tour how to to create `Confimap` with relabeling configuration
|
||||
|
||||
```yaml
|
||||
cat << EOF | kubectl apply -f -
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: vmagent-relabel
|
||||
data:
|
||||
global-relabel.yaml: |
|
||||
- target_label: bar
|
||||
- source_labels: [aa]
|
||||
separator: "foobar"
|
||||
regex: "foo.+bar"
|
||||
target_label: aaa
|
||||
replacement: "xxx"
|
||||
- action: keep
|
||||
source_labels: [aaa]
|
||||
- action: drop
|
||||
source_labels: [aaa]
|
||||
target-1-relabel.yaml: |
|
||||
- action: keep_if_equal
|
||||
source_labels: [foo, bar]
|
||||
- action: drop_if_equal
|
||||
source_labels: [foo, bar]
|
||||
EOF
|
||||
```
|
||||
|
||||
Second, add `relabelConfig` to `VMagent` spec for global relabeling with name of `Configmap` - `vmagent-relabel` and key `global-relabel.yaml`.
|
||||
For relabeling per remoteWrite target, add `urlRelabelConfig` name of `Configmap` - `vmagent-relabel` and key `target-1-relabel.yaml` to one of remoteWrite target for relabeling only
|
||||
for those target.
|
||||
|
||||
```yaml
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMAgent
|
||||
metadata:
|
||||
name: example-vmagent
|
||||
spec:
|
||||
serviceScrapeNamespaceSelector: {}
|
||||
podScrapeNamespaceSelector: {}
|
||||
podScrapeSelector: {}
|
||||
serviceScrapeSelector: {}
|
||||
replicaCount: 1
|
||||
serviceAccountName: vmagent
|
||||
relabelConfig:
|
||||
name: "vmagent-relabel"
|
||||
key: "global-relabel.yaml"
|
||||
remoteWrite:
|
||||
- url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429/api/v1/write"
|
||||
- url: "http://vmsingle-example-vmsingle.default.svc:8429/api/v1/write"
|
||||
urlRelabelConfig:
|
||||
name: "vmagent-relabel"
|
||||
key: "target-1-relabel.yaml"
|
||||
EOF
|
||||
```
|
||||
|
||||
|
||||
## Inline example
|
||||
|
||||
```yaml
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMAgent
|
||||
metadata:
|
||||
name: example-vmagent
|
||||
spec:
|
||||
serviceScrapeNamespaceSelector: {}
|
||||
podScrapeNamespaceSelector: {}
|
||||
podScrapeSelector: {}
|
||||
serviceScrapeSelector: {}
|
||||
replicaCount: 1
|
||||
serviceAccountName: vmagent
|
||||
inlineRelabelConfig:
|
||||
- target_label: bar
|
||||
- source_labels: [aa]
|
||||
separator: "foobar"
|
||||
regex: "foo.+bar"
|
||||
target_label: aaa
|
||||
replacement: "xxx"
|
||||
- action: keep
|
||||
source_labels: [aaa]
|
||||
- action: drop
|
||||
source_labels: [aaa]
|
||||
remoteWrite:
|
||||
- url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429/api/v1/write"
|
||||
- url: "http://vmsingle-example-vmsingle.default.svc:8429/api/v1/write"
|
||||
inlineUrlRelabelConfig:
|
||||
- action: keep_if_equal
|
||||
source_labels: [foo, bar]
|
||||
- action: drop_if_equal
|
||||
source_labels: [foo, bar]
|
||||
EOF
|
||||
```
|
||||
|
||||
|
||||
## Combined example
|
||||
|
||||
Its also possible to use both features in combination.
|
||||
|
||||
First will be added relabeling configs from `inlineRelabelConfig`, then `relabelConfig` from configmap.
|
||||
|
||||
```yaml
|
||||
cat << EOF | kubectl apply -f -
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: vmagent-relabel
|
||||
data:
|
||||
global-relabel.yaml: |
|
||||
- target_label: bar
|
||||
- source_labels: [aa]
|
||||
separator: "foobar"
|
||||
regex: "foo.+bar"
|
||||
target_label: aaa
|
||||
replacement: "xxx"
|
||||
- action: keep
|
||||
source_labels: [aaa]
|
||||
- action: drop
|
||||
source_labels: [aaa]
|
||||
target-1-relabel.yaml: |
|
||||
- action: keep_if_equal
|
||||
source_labels: [foo, bar]
|
||||
- action: drop_if_equal
|
||||
source_labels: [foo, bar]
|
||||
EOF
|
||||
```
|
||||
|
||||
```yaml
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMAgent
|
||||
metadata:
|
||||
name: example-vmagent
|
||||
spec:
|
||||
serviceScrapeNamespaceSelector: {}
|
||||
podScrapeNamespaceSelector: {}
|
||||
podScrapeSelector: {}
|
||||
serviceScrapeSelector: {}
|
||||
replicaCount: 1
|
||||
serviceAccountName: vmagent
|
||||
inlineRelabelConfig:
|
||||
- target_label: bar1
|
||||
- source_labels: [aa]
|
||||
relabelConfig:
|
||||
name: "vmagent-relabel"
|
||||
key: "global-relabel.yaml"
|
||||
remoteWrite:
|
||||
- url: "http://vmsingle-example-vmsingle-persisted.default.svc:8429/api/v1/write"
|
||||
- url: "http://vmsingle-example-vmsingle.default.svc:8429/api/v1/write"
|
||||
urlRelabelConfig:
|
||||
name: "vmagent-relabel"
|
||||
key: "target-1-relabel.yaml"
|
||||
inlineUrlRelabelConfig:
|
||||
- action: keep_if_equal
|
||||
source_labels: [foo1, bar2]
|
||||
EOF
|
||||
```
|
||||
|
||||
|
||||
Resulted configmap:
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
data:
|
||||
global_relabeling.yaml: |
|
||||
- target_label: bar1
|
||||
- source_labels:
|
||||
- aa
|
||||
- target_label: bar
|
||||
- source_labels: [aa]
|
||||
separator: "foobar"
|
||||
regex: "foo.+bar"
|
||||
target_label: aaa
|
||||
replacement: "xxx"
|
||||
- action: keep
|
||||
source_labels: [aaa]
|
||||
- action: drop
|
||||
source_labels: [aaa]
|
||||
url_rebaling-1.yaml: |
|
||||
- source_labels:
|
||||
- foo1
|
||||
- bar2
|
||||
action: keep_if_equal
|
||||
- action: keep_if_equal
|
||||
source_labels: [foo, bar]
|
||||
- action: drop_if_equal
|
||||
source_labels: [foo, bar]
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
finalizers:
|
||||
- apps.victoriametrics.com/finalizer
|
||||
labels:
|
||||
app.kubernetes.io/component: monitoring
|
||||
app.kubernetes.io/instance: example-vmagent
|
||||
app.kubernetes.io/name: vmagent
|
||||
managed-by: vm-operator
|
||||
name: relabelings-assets-vmagent-example-vmagent
|
||||
namespace: default
|
||||
ownerReferences:
|
||||
- apiVersion: operator.victoriametrics.com/v1beta1
|
||||
blockOwnerDeletion: true
|
||||
controller: true
|
||||
kind: VMAgent
|
||||
name: example-vmagent
|
||||
uid: 7e9fb838-65da-4443-a43b-c00cd6c4db5b
|
||||
```
|
||||
|
||||
|
||||
## Additional information
|
||||
|
||||
`VMAgent` also has some extra options for relabeling actions, you can check it [docs](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/app/vmagent/README.md#relabeling)
|
36
docs/operator/resources-validation.MD
Normal file
36
docs/operator/resources-validation.MD
Normal file
|
@ -0,0 +1,36 @@
|
|||
---
|
||||
sort: 11
|
||||
---
|
||||
|
||||
# CRD Validation
|
||||
|
||||
## Description
|
||||
Operator supports validation admission webhook [docs](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/)
|
||||
|
||||
It checks resources configuration and returns errors to caller before resource will be created at kubernetes api.
|
||||
This should reduce errors and simplify debugging.
|
||||
|
||||
## Configuration
|
||||
|
||||
Validation hooks at operator side must be enabled with flags:
|
||||
```
|
||||
--webhook.enable
|
||||
# optional configuration for certDir and tls names.
|
||||
--webhook.certDir=/tmp/k8s-webhook-server/serving-certs/
|
||||
--webhook.keyName=tls.key
|
||||
--webhook.certName=tls.crt
|
||||
```
|
||||
|
||||
You have to mount correct certificates at give directory.
|
||||
It can be simplified with cert-manager and kustomize command: `kustomize build config/deployments/webhook/ `
|
||||
|
||||
|
||||
## Requirements
|
||||
|
||||
- Valid certificate with key must be provided to operator
|
||||
- Valid CABundle must be added to the `ValidatingWebhookConfiguration`
|
||||
|
||||
|
||||
## Useful links
|
||||
- [k8s admission webhooks](https://banzaicloud.com/blog/k8s-admission-webhooks/)
|
||||
- [olm webhooks](https://docs.openshift.com/container-platform/4.5/operators/user/olm-webhooks.html)
|
85
docs/operator/security.MD
Normal file
85
docs/operator/security.MD
Normal file
|
@ -0,0 +1,85 @@
|
|||
---
|
||||
sort: 12
|
||||
---
|
||||
|
||||
# Security
|
||||
|
||||
VictoriaMetrics operator provides several security features, such as [PodSecurityPolicies](https://kubernetes.io/docs/concepts/policy/pod-security-policy/), [PodSecurityContext](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/).
|
||||
|
||||
|
||||
## PodSecurityPolicy.
|
||||
|
||||
By default, operator creates serviceAccount for each cluster resource and binds default `PodSecurityPolicy` to it.
|
||||
|
||||
Default psp:
|
||||
```yaml
|
||||
apiVersion: policy/v1beta1
|
||||
kind: PodSecurityPolicy
|
||||
metadata:
|
||||
name: vmagent-example-vmagent
|
||||
spec:
|
||||
allowPrivilegeEscalation: false
|
||||
fsGroup:
|
||||
rule: RunAsAny
|
||||
hostNetwork: true
|
||||
requiredDropCapabilities:
|
||||
- ALL
|
||||
runAsUser:
|
||||
rule: RunAsAny
|
||||
seLinux:
|
||||
rule: RunAsAny
|
||||
supplementalGroups:
|
||||
rule: RunAsAny
|
||||
volumes:
|
||||
- persistentVolumeClaim
|
||||
- secret
|
||||
- emptyDir
|
||||
- configMap
|
||||
- projected
|
||||
- downwardAPI
|
||||
- nfs
|
||||
```
|
||||
|
||||
This behaviour may be disabled with env variable passed to operator:
|
||||
```yaml
|
||||
- name: VM_PSPAUTOCREATEENABLED
|
||||
value: "false"
|
||||
```
|
||||
|
||||
User may also override default pod security policy with setting: `spec.podSecurityPolicyName: "psp-name"`.
|
||||
|
||||
|
||||
## PodSecurityContext
|
||||
|
||||
`PodSecurityContext` can be configured with spec setting. It may be useful for mounted volumes, with `VMSingle` for example:
|
||||
|
||||
```yaml
|
||||
apiVersion: operator.victoriametrics.com/v1beta1
|
||||
kind: VMSingle
|
||||
metadata:
|
||||
name: vmsingle-f
|
||||
namespace: monitoring-system
|
||||
spec:
|
||||
retentionPeriod: "2"
|
||||
removePvcAfterDelete: true
|
||||
securityContext:
|
||||
runAsUser: 1000
|
||||
fsGroup: 1000
|
||||
runAsGroup: 1000
|
||||
extraArgs:
|
||||
dedup.minScrapeInterval: 10s
|
||||
storage:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 25Gi
|
||||
resources:
|
||||
requests:
|
||||
cpu: "0.5"
|
||||
memory: "512Mi"
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: "1512Mi"
|
||||
|
||||
```
|
110
docs/operator/vars.MD
Normal file
110
docs/operator/vars.MD
Normal file
|
@ -0,0 +1,110 @@
|
|||
# Auto Generated vars for package config
|
||||
updated at Fri Jan 21 15:57:41 UTC 2022
|
||||
|
||||
|
||||
| varible name | variable default value | variable required | variable description |
|
||||
| --- | --- | --- | --- |
|
||||
| VM_USECUSTOMCONFIGRELOADER | false | false | enables custom config reloader for vmauth and vmagent,it should speed-up config reloading process. |
|
||||
| VM_CUSTOMCONFIGRELOADERIMAGE | victoriametrics/operator:config-reloader-0.1.0 | false | - |
|
||||
| VM_PSPAUTOCREATEENABLED | true | false | - |
|
||||
| VM_VMALERTDEFAULT_IMAGE | victoriametrics/vmalert | false | - |
|
||||
| VM_VMALERTDEFAULT_VERSION | v1.72.0 | false | - |
|
||||
| VM_VMALERTDEFAULT_PORT | 8080 | false | - |
|
||||
| VM_VMALERTDEFAULT_USEDEFAULTRESOURCES | true | false | - |
|
||||
| VM_VMALERTDEFAULT_RESOURCE_LIMIT_MEM | 500Mi | false | - |
|
||||
| VM_VMALERTDEFAULT_RESOURCE_LIMIT_CPU | 200m | false | - |
|
||||
| VM_VMALERTDEFAULT_RESOURCE_REQUEST_MEM | 200Mi | false | - |
|
||||
| VM_VMALERTDEFAULT_RESOURCE_REQUEST_CPU | 50m | false | - |
|
||||
| VM_VMALERTDEFAULT_CONFIGRELOADERCPU | 100m | false | - |
|
||||
| VM_VMALERTDEFAULT_CONFIGRELOADERMEMORY | 25Mi | false | - |
|
||||
| VM_VMALERTDEFAULT_CONFIGRELOADIMAGE | jimmidyson/configmap-reload:v0.3.0 | false | - |
|
||||
| VM_VMAGENTDEFAULT_IMAGE | victoriametrics/vmagent | false | - |
|
||||
| VM_VMAGENTDEFAULT_VERSION | v1.72.0 | false | - |
|
||||
| VM_VMAGENTDEFAULT_CONFIGRELOADIMAGE | quay.io/prometheus-operator/prometheus-config-reloader:v0.48.1 | false | - |
|
||||
| VM_VMAGENTDEFAULT_PORT | 8429 | false | - |
|
||||
| VM_VMAGENTDEFAULT_USEDEFAULTRESOURCES | true | false | - |
|
||||
| VM_VMAGENTDEFAULT_RESOURCE_LIMIT_MEM | 500Mi | false | - |
|
||||
| VM_VMAGENTDEFAULT_RESOURCE_LIMIT_CPU | 200m | false | - |
|
||||
| VM_VMAGENTDEFAULT_RESOURCE_REQUEST_MEM | 200Mi | false | - |
|
||||
| VM_VMAGENTDEFAULT_RESOURCE_REQUEST_CPU | 50m | false | - |
|
||||
| VM_VMAGENTDEFAULT_CONFIGRELOADERCPU | 100m | false | - |
|
||||
| VM_VMAGENTDEFAULT_CONFIGRELOADERMEMORY | 25Mi | false | - |
|
||||
| VM_VMSINGLEDEFAULT_IMAGE | victoriametrics/victoria-metrics | false | - |
|
||||
| VM_VMSINGLEDEFAULT_VERSION | v1.72.0 | false | - |
|
||||
| VM_VMSINGLEDEFAULT_PORT | 8429 | false | - |
|
||||
| VM_VMSINGLEDEFAULT_USEDEFAULTRESOURCES | true | false | - |
|
||||
| VM_VMSINGLEDEFAULT_RESOURCE_LIMIT_MEM | 1500Mi | false | - |
|
||||
| VM_VMSINGLEDEFAULT_RESOURCE_LIMIT_CPU | 1200m | false | - |
|
||||
| VM_VMSINGLEDEFAULT_RESOURCE_REQUEST_MEM | 500Mi | false | - |
|
||||
| VM_VMSINGLEDEFAULT_RESOURCE_REQUEST_CPU | 150m | false | - |
|
||||
| VM_VMSINGLEDEFAULT_CONFIGRELOADERCPU | 100m | false | - |
|
||||
| VM_VMSINGLEDEFAULT_CONFIGRELOADERMEMORY | 25Mi | false | - |
|
||||
| VM_VMCLUSTERDEFAULT_USEDEFAULTRESOURCES | true | false | - |
|
||||
| VM_VMCLUSTERDEFAULT_VMSELECTDEFAULT_IMAGE | victoriametrics/vmselect | false | - |
|
||||
| VM_VMCLUSTERDEFAULT_VMSELECTDEFAULT_VERSION | v1.72.0-cluster | false | - |
|
||||
| VM_VMCLUSTERDEFAULT_VMSELECTDEFAULT_PORT | 8481 | false | - |
|
||||
| VM_VMCLUSTERDEFAULT_VMSELECTDEFAULT_RESOURCE_LIMIT_MEM | 1000Mi | false | - |
|
||||
| VM_VMCLUSTERDEFAULT_VMSELECTDEFAULT_RESOURCE_LIMIT_CPU | 500m | false | - |
|
||||
| VM_VMCLUSTERDEFAULT_VMSELECTDEFAULT_RESOURCE_REQUEST_MEM | 500Mi | false | - |
|
||||
| VM_VMCLUSTERDEFAULT_VMSELECTDEFAULT_RESOURCE_REQUEST_CPU | 100m | false | - |
|
||||
| VM_VMCLUSTERDEFAULT_VMSTORAGEDEFAULT_IMAGE | victoriametrics/vmstorage | false | - |
|
||||
| VM_VMCLUSTERDEFAULT_VMSTORAGEDEFAULT_VERSION | v1.72.0-cluster | false | - |
|
||||
| VM_VMCLUSTERDEFAULT_VMSTORAGEDEFAULT_VMINSERTPORT | 8400 | false | - |
|
||||
| VM_VMCLUSTERDEFAULT_VMSTORAGEDEFAULT_VMSELECTPORT | 8401 | false | - |
|
||||
| VM_VMCLUSTERDEFAULT_VMSTORAGEDEFAULT_PORT | 8482 | false | - |
|
||||
| VM_VMCLUSTERDEFAULT_VMSTORAGEDEFAULT_RESOURCE_LIMIT_MEM | 1500Mi | false | - |
|
||||
| VM_VMCLUSTERDEFAULT_VMSTORAGEDEFAULT_RESOURCE_LIMIT_CPU | 1000m | false | - |
|
||||
| VM_VMCLUSTERDEFAULT_VMSTORAGEDEFAULT_RESOURCE_REQUEST_MEM | 500Mi | false | - |
|
||||
| VM_VMCLUSTERDEFAULT_VMSTORAGEDEFAULT_RESOURCE_REQUEST_CPU | 250m | false | - |
|
||||
| VM_VMCLUSTERDEFAULT_VMINSERTDEFAULT_IMAGE | victoriametrics/vminsert | false | - |
|
||||
| VM_VMCLUSTERDEFAULT_VMINSERTDEFAULT_VERSION | v1.72.0-cluster | false | - |
|
||||
| VM_VMCLUSTERDEFAULT_VMINSERTDEFAULT_PORT | 8480 | false | - |
|
||||
| VM_VMCLUSTERDEFAULT_VMINSERTDEFAULT_RESOURCE_LIMIT_MEM | 500Mi | false | - |
|
||||
| VM_VMCLUSTERDEFAULT_VMINSERTDEFAULT_RESOURCE_LIMIT_CPU | 500m | false | - |
|
||||
| VM_VMCLUSTERDEFAULT_VMINSERTDEFAULT_RESOURCE_REQUEST_MEM | 200Mi | false | - |
|
||||
| VM_VMCLUSTERDEFAULT_VMINSERTDEFAULT_RESOURCE_REQUEST_CPU | 150m | false | - |
|
||||
| VM_VMALERTMANAGER_CONFIGRELOADERIMAGE | jimmidyson/configmap-reload:v0.3.0 | false | - |
|
||||
| VM_VMALERTMANAGER_CONFIGRELOADERCPU | 100m | false | - |
|
||||
| VM_VMALERTMANAGER_CONFIGRELOADERMEMORY | 25Mi | false | - |
|
||||
| VM_VMALERTMANAGER_ALERTMANAGERDEFAULTBASEIMAGE | prom/alertmanager | false | - |
|
||||
| VM_VMALERTMANAGER_ALERTMANAGERVERSION | v0.22.2 | false | - |
|
||||
| VM_VMALERTMANAGER_LOCALHOST | 127.0.0.1 | false | - |
|
||||
| VM_VMALERTMANAGER_USEDEFAULTRESOURCES | true | false | - |
|
||||
| VM_VMALERTMANAGER_RESOURCE_LIMIT_MEM | 256Mi | false | - |
|
||||
| VM_VMALERTMANAGER_RESOURCE_LIMIT_CPU | 100m | false | - |
|
||||
| VM_VMALERTMANAGER_RESOURCE_REQUEST_MEM | 56Mi | false | - |
|
||||
| VM_VMALERTMANAGER_RESOURCE_REQUEST_CPU | 30m | false | - |
|
||||
| VM_DISABLESELFSERVICESCRAPECREATION | false | false | - |
|
||||
| VM_VMBACKUP_IMAGE | victoriametrics/vmbackupmanager | false | - |
|
||||
| VM_VMBACKUP_VERSION | v1.72.0-enterprise | false | - |
|
||||
| VM_VMBACKUP_PORT | 8300 | false | - |
|
||||
| VM_VMBACKUP_USEDEFAULTRESOURCES | true | false | - |
|
||||
| VM_VMBACKUP_RESOURCE_LIMIT_MEM | 500Mi | false | - |
|
||||
| VM_VMBACKUP_RESOURCE_LIMIT_CPU | 500m | false | - |
|
||||
| VM_VMBACKUP_RESOURCE_REQUEST_MEM | 200Mi | false | - |
|
||||
| VM_VMBACKUP_RESOURCE_REQUEST_CPU | 150m | false | - |
|
||||
| VM_VMBACKUP_LOGLEVEL | INFO | false | - |
|
||||
| VM_VMAUTHDEFAULT_IMAGE | victoriametrics/vmauth | false | - |
|
||||
| VM_VMAUTHDEFAULT_VERSION | v1.72.0 | false | - |
|
||||
| VM_VMAUTHDEFAULT_CONFIGRELOADIMAGE | quay.io/prometheus-operator/prometheus-config-reloader:v0.48.1 | false | - |
|
||||
| VM_VMAUTHDEFAULT_PORT | 8427 | false | - |
|
||||
| VM_VMAUTHDEFAULT_USEDEFAULTRESOURCES | true | false | - |
|
||||
| VM_VMAUTHDEFAULT_RESOURCE_LIMIT_MEM | 300Mi | false | - |
|
||||
| VM_VMAUTHDEFAULT_RESOURCE_LIMIT_CPU | 200m | false | - |
|
||||
| VM_VMAUTHDEFAULT_RESOURCE_REQUEST_MEM | 100Mi | false | - |
|
||||
| VM_VMAUTHDEFAULT_RESOURCE_REQUEST_CPU | 50m | false | - |
|
||||
| VM_VMAUTHDEFAULT_CONFIGRELOADERCPU | 100m | false | - |
|
||||
| VM_VMAUTHDEFAULT_CONFIGRELOADERMEMORY | 25Mi | false | - |
|
||||
| VM_ENABLEDPROMETHEUSCONVERTER_PODMONITOR | true | false | - |
|
||||
| VM_ENABLEDPROMETHEUSCONVERTER_SERVICESCRAPE | true | false | - |
|
||||
| VM_ENABLEDPROMETHEUSCONVERTER_PROMETHEUSRULE | true | false | - |
|
||||
| VM_ENABLEDPROMETHEUSCONVERTER_PROBE | true | false | - |
|
||||
| VM_ENABLEDPROMETHEUSCONVERTEROWNERREFERENCES | false | false | - |
|
||||
| VM_HOST | 0.0.0.0 | false | - |
|
||||
| VM_LISTENADDRESS | 0.0.0.0 | false | - |
|
||||
| VM_DEFAULTLABELS | managed-by=vm-operator | false | - |
|
||||
| VM_LABELS | - | false | - |
|
||||
| VM_CLUSTERDOMAINNAME | - | false | - |
|
||||
| VM_PODWAITREADYTIMEOUT | 80s | false | - |
|
||||
| VM_PODWAITREADYINTERVALCHECK | 5s | false | - |
|
||||
| VM_PODWAITREADYINITDELAY | 10s | false | - |
|
|
@ -10,7 +10,7 @@ Supported storage systems for backups:
|
|||
|
||||
* [GCS](https://cloud.google.com/storage/). Example: `gs://<bucket>/<path/to/backup>`
|
||||
* [S3](https://aws.amazon.com/s3/). Example: `s3://<bucket>/<path/to/backup>`
|
||||
* Any S3-compatible storage such as [MinIO](https://github.com/minio/minio), [Ceph](https://docs.ceph.com/en/pacific/radosgw/s3/) or [Swift](https://www.swiftstack.com/docs/admin/middleware/s3_middleware.html). See [these docs](#advanced-usage) for details.
|
||||
* Any S3-compatible storage such as [MinIO](https://github.com/minio/minio), [Ceph](https://docs.ceph.com/en/pacific/radosgw/s3/) or [Swift](https://platform.swiftstack.com/docs/admin/middleware/s3_middleware.html). See [these docs](#advanced-usage) for details.
|
||||
* Local filesystem. Example: `fs://</absolute/path/to/backup>`
|
||||
|
||||
`vmbackup` supports incremental and full backups. Incremental backups are created automatically if the destination path already contains data from the previous backup.
|
||||
|
|
|
@ -211,7 +211,7 @@ The shortlist of configuration flags include the following:
|
|||
-envflag.prefix string
|
||||
Prefix for environment variables if -envflag.enable is set
|
||||
-eula
|
||||
By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/assets/VM_EULA.pdf
|
||||
By specifying this flag, you confirm that you have an enterprise license and accept the EULA https://victoriametrics.com/legal/eula/
|
||||
-fs.disableMmap
|
||||
Whether to use pread() instead of mmap() for reading data files. By default, mmap() is used for 64-bit arches and pread() is used for 32-bit arches as they cannot read data files larger than 2^32 bytes in memory. mmap() is usually faster for reading small data chunks than pread()
|
||||
-http.connTimeout duration
|
||||
|
|
14
go.mod
14
go.mod
|
@ -11,7 +11,7 @@ require (
|
|||
github.com/VictoriaMetrics/fasthttp v1.1.0
|
||||
github.com/VictoriaMetrics/metrics v1.18.1
|
||||
github.com/VictoriaMetrics/metricsql v0.37.0
|
||||
github.com/aws/aws-sdk-go v1.42.35
|
||||
github.com/aws/aws-sdk-go v1.42.39
|
||||
github.com/cespare/xxhash/v2 v2.1.2
|
||||
github.com/cheggaaa/pb/v3 v3.0.8
|
||||
github.com/cpuguy83/go-md2man/v2 v2.0.1 // indirect
|
||||
|
@ -31,7 +31,7 @@ require (
|
|||
github.com/valyala/fasttemplate v1.2.1
|
||||
github.com/valyala/gozstd v1.15.1
|
||||
github.com/valyala/quicktemplate v1.7.0
|
||||
golang.org/x/net v0.0.0-20220114011407-0dd24b26b47d
|
||||
golang.org/x/net v0.0.0-20220121210141-e204ce36a2ba
|
||||
golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8
|
||||
golang.org/x/sys v0.0.0-20220114195835-da31bd327af9
|
||||
google.golang.org/api v0.65.0
|
||||
|
@ -40,21 +40,21 @@ require (
|
|||
|
||||
require (
|
||||
cloud.google.com/go v0.100.2 // indirect
|
||||
cloud.google.com/go/compute v1.0.0 // indirect
|
||||
cloud.google.com/go/iam v0.1.0 // indirect
|
||||
cloud.google.com/go/compute v1.1.0 // indirect
|
||||
cloud.google.com/go/iam v0.1.1 // indirect
|
||||
github.com/VividCortex/ewma v1.2.0 // indirect
|
||||
github.com/beorn7/perks v1.0.1 // indirect
|
||||
github.com/go-kit/log v0.2.0 // indirect
|
||||
github.com/go-logfmt/logfmt v0.5.1 // indirect
|
||||
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
|
||||
github.com/golang/protobuf v1.5.2 // indirect
|
||||
github.com/google/go-cmp v0.5.6 // indirect
|
||||
github.com/google/go-cmp v0.5.7 // indirect
|
||||
github.com/googleapis/gax-go/v2 v2.1.1 // indirect
|
||||
github.com/jmespath/go-jmespath v0.4.0 // indirect
|
||||
github.com/mattn/go-isatty v0.0.14 // indirect
|
||||
github.com/matttproud/golang_protobuf_extensions v1.0.1 // indirect
|
||||
github.com/pkg/errors v0.9.1 // indirect
|
||||
github.com/prometheus/client_golang v1.11.0 // indirect
|
||||
github.com/prometheus/client_golang v1.12.0 // indirect
|
||||
github.com/prometheus/client_model v0.2.0 // indirect
|
||||
github.com/prometheus/procfs v0.7.3 // indirect
|
||||
github.com/rivo/uniseg v0.2.0 // indirect
|
||||
|
@ -68,7 +68,7 @@ require (
|
|||
golang.org/x/text v0.3.7 // indirect
|
||||
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
|
||||
google.golang.org/appengine v1.6.7 // indirect
|
||||
google.golang.org/genproto v0.0.0-20220114231437-d2e6a121cae0 // indirect
|
||||
google.golang.org/genproto v0.0.0-20220118154757-00ab72f36ad5 // indirect
|
||||
google.golang.org/grpc v1.43.0 // indirect
|
||||
google.golang.org/protobuf v1.27.1 // indirect
|
||||
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b // indirect
|
||||
|
|
30
go.sum
30
go.sum
|
@ -40,12 +40,12 @@ cloud.google.com/go/bigquery v1.8.0/go.mod h1:J5hqkt3O0uAFnINi6JXValWIb1v0goeZM7
|
|||
cloud.google.com/go/bigtable v1.2.0/go.mod h1:JcVAOl45lrTmQfLj7T6TxyMzIN/3FGGcFm+2xVAli2o=
|
||||
cloud.google.com/go/bigtable v1.3.0/go.mod h1:z5EyKrPE8OQmeg4h5MNdKvuSnI9CCT49Ki3f23aBzio=
|
||||
cloud.google.com/go/compute v0.1.0/go.mod h1:GAesmwr110a34z04OlxYkATPBEfVhkymfTBXtfbBFow=
|
||||
cloud.google.com/go/compute v1.0.0 h1:SJYBzih8Jj9EUm6IDirxKG0I0AGWduhtb6BmdqWarw4=
|
||||
cloud.google.com/go/compute v1.0.0/go.mod h1:GAesmwr110a34z04OlxYkATPBEfVhkymfTBXtfbBFow=
|
||||
cloud.google.com/go/compute v1.1.0 h1:pyPhehLfZ6pVzRgJmXGYvCY4K7WSWRhVw0AwhgVvS84=
|
||||
cloud.google.com/go/compute v1.1.0/go.mod h1:2NIffxgWfORSI7EOYMFatGTfjMLnqrOKBEyYb6NoRgA=
|
||||
cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE=
|
||||
cloud.google.com/go/datastore v1.1.0/go.mod h1:umbIZjpQpHh4hmRpGhH4tLFup+FVzqBi1b3c64qFpCk=
|
||||
cloud.google.com/go/iam v0.1.0 h1:W2vbGCrE3Z7J/x3WXLxxGl9LMSB2uhsAA7Ss/6u/qRY=
|
||||
cloud.google.com/go/iam v0.1.0/go.mod h1:vcUNEa0pEm0qRVpmWepWaFMIAI8/hjB9mO8rNCJtF6c=
|
||||
cloud.google.com/go/iam v0.1.1 h1:4CapQyNFjiksks1/x7jsvsygFPhihslYk5GptIrlX68=
|
||||
cloud.google.com/go/iam v0.1.1/go.mod h1:CKqrcnI/suGpybEHxZ7BMehL0oA4LpdyJdUlTl9jVMw=
|
||||
cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I=
|
||||
cloud.google.com/go/pubsub v1.1.0/go.mod h1:EwwdRX2sKPjnvnqCa270oGRyludottCI76h+R3AArQw=
|
||||
cloud.google.com/go/pubsub v1.2.0/go.mod h1:jhfEVHT8odbXTkndysNHCcx0awwzvfOlguIAii9o8iA=
|
||||
|
@ -161,8 +161,8 @@ github.com/aws/aws-sdk-go v1.30.12/go.mod h1:5zCpMtNQVjRREroY7sYe8lOMRSxkhG6MZve
|
|||
github.com/aws/aws-sdk-go v1.34.28/go.mod h1:H7NKnBqNVzoTJpGfLrQkkD+ytBA93eiDYi/+8rV9s48=
|
||||
github.com/aws/aws-sdk-go v1.35.31/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2zKMmprdro=
|
||||
github.com/aws/aws-sdk-go v1.40.45/go.mod h1:585smgzpB/KqRA+K3y/NL/oYRqQvpNJYvLm+LY1U59Q=
|
||||
github.com/aws/aws-sdk-go v1.42.35 h1:N4N9buNs4YlosI9N0+WYrq8cIZwdgv34yRbxzZlTvFs=
|
||||
github.com/aws/aws-sdk-go v1.42.35/go.mod h1:OGr6lGMAKGlG9CVrYnWYDKIyb829c6EVBRjxqjmPepc=
|
||||
github.com/aws/aws-sdk-go v1.42.39 h1:6Lso73VoCI8Zmv3zAMv4BNg2gHAKNOlbLv1s/ew90SI=
|
||||
github.com/aws/aws-sdk-go v1.42.39/go.mod h1:OGr6lGMAKGlG9CVrYnWYDKIyb829c6EVBRjxqjmPepc=
|
||||
github.com/aws/aws-sdk-go-v2 v0.18.0/go.mod h1:JWVYvqSMppoMJC0x5wdwiImzgXTI9FuZwxzkQq9wy+g=
|
||||
github.com/aws/aws-sdk-go-v2 v1.9.1/go.mod h1:cK/D0BBs0b/oWPIcX/Z/obahJK1TT7IPVjy53i/mX/4=
|
||||
github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.8.1/go.mod h1:CM+19rL1+4dFWnOQKwDc7H1KwXTz+h61oUSHyhV0b3o=
|
||||
|
@ -484,8 +484,9 @@ github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/
|
|||
github.com/google/go-cmp v0.5.3/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
|
||||
github.com/google/go-cmp v0.5.4/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
|
||||
github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
|
||||
github.com/google/go-cmp v0.5.6 h1:BKbKCqvP6I+rmFHt06ZmyQtvB8xAkWdhFyr0ZUNZcxQ=
|
||||
github.com/google/go-cmp v0.5.6/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
|
||||
github.com/google/go-cmp v0.5.7 h1:81/ik6ipDQS2aGcBfIN5dHDB36BwrStyeAQquSYCV4o=
|
||||
github.com/google/go-cmp v0.5.7/go.mod h1:n+brtR0CgQNWTVd5ZUFpTBC8YFBDLK/h/bpaJ8/DtOE=
|
||||
github.com/google/go-querystring v1.0.0/go.mod h1:odCYkC5MyYFN7vkCjXpyrEuKhc/BUO6wN/zVPAxq5ck=
|
||||
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
|
||||
github.com/google/gofuzz v1.1.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
|
||||
|
@ -840,8 +841,9 @@ github.com/prometheus/client_golang v1.5.1/go.mod h1:e9GMxYsXl05ICDXkRhurwBS4Q3O
|
|||
github.com/prometheus/client_golang v1.6.0/go.mod h1:ZLOG9ck3JLRdB5MgO8f+lLTe83AXG6ro35rLTxvnIl4=
|
||||
github.com/prometheus/client_golang v1.7.1/go.mod h1:PY5Wy2awLA44sXw4AOSfFBetzPP4j5+D6mVACh+pe2M=
|
||||
github.com/prometheus/client_golang v1.8.0/go.mod h1:O9VU6huf47PktckDQfMTX0Y8tY0/7TSWwj+ITvv0TnM=
|
||||
github.com/prometheus/client_golang v1.11.0 h1:HNkLOAEQMIDv/K+04rukrLx6ch7msSRwf3/SASFAGtQ=
|
||||
github.com/prometheus/client_golang v1.11.0/go.mod h1:Z6t4BnS23TR94PD6BsDNk8yVqroYurpAkEiz0P2BEV0=
|
||||
github.com/prometheus/client_golang v1.12.0 h1:C+UIj/QWtmqY13Arb8kwMt5j34/0Z2iKamrJ+ryC0Gg=
|
||||
github.com/prometheus/client_golang v1.12.0/go.mod h1:3Z9XVyYiZYEO+YQWt3RD2R3jrbd179Rt297l4aS6nDY=
|
||||
github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
|
||||
github.com/prometheus/client_model v0.0.0-20190115171406-56726106282f/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
|
||||
github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
|
||||
|
@ -1166,8 +1168,8 @@ golang.org/x/net v0.0.0-20210525063256-abc453219eb5/go.mod h1:9nx3DQGgdP8bBQD5qx
|
|||
golang.org/x/net v0.0.0-20210614182718-04defd469f4e/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
|
||||
golang.org/x/net v0.0.0-20210917221730-978cfadd31cf/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
|
||||
golang.org/x/net v0.0.0-20211216030914-fe4d6282115f/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
|
||||
golang.org/x/net v0.0.0-20220114011407-0dd24b26b47d h1:1n1fc535VhN8SYtD4cDUyNlfpAF2ROMM9+11equK3hs=
|
||||
golang.org/x/net v0.0.0-20220114011407-0dd24b26b47d/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
|
||||
golang.org/x/net v0.0.0-20220121210141-e204ce36a2ba h1:6u6sik+bn/y7vILcYkK3iwTBWN7WtBvB0+SZswQnbf8=
|
||||
golang.org/x/net v0.0.0-20220121210141-e204ce36a2ba/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
|
||||
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
|
||||
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
|
||||
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
|
||||
|
@ -1299,6 +1301,7 @@ golang.org/x/sys v0.0.0-20220114195835-da31bd327af9 h1:XfKQ4OlFl8okEOr5UvAqFRVj8
|
|||
golang.org/x/sys v0.0.0-20220114195835-da31bd327af9/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
|
||||
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
|
||||
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
|
||||
golang.org/x/text v0.0.0-20160726164857-2910a502d2bf/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
|
||||
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
|
||||
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
|
||||
|
@ -1446,6 +1449,7 @@ google.golang.org/api v0.57.0/go.mod h1:dVPlbZyBo2/OjBpmvNdpn2GRm6rPy75jyU7bmhdr
|
|||
google.golang.org/api v0.58.0/go.mod h1:cAbP2FsxoGVNwtgNAmmn3y5G1TWAiVYRmg4yku3lv+E=
|
||||
google.golang.org/api v0.61.0/go.mod h1:xQRti5UdCmoCEqFxcz93fTl338AVqDgyaDRuOZ3hg9I=
|
||||
google.golang.org/api v0.63.0/go.mod h1:gs4ij2ffTRXwuzzgJl/56BdwJaA194ijkfn++9tDuPo=
|
||||
google.golang.org/api v0.64.0/go.mod h1:931CdxA8Rm4t6zqTFGSsgwbAEZ2+GMYurbndwSimebM=
|
||||
google.golang.org/api v0.65.0 h1:MTW9c+LIBAbwoS1Gb+YV7NjFBt2f7GtAS5hIzh2NjgQ=
|
||||
google.golang.org/api v0.65.0/go.mod h1:ArYhxgGadlWmqO1IqVujw6Cs8IdD33bTmzKo2Sh+cbg=
|
||||
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
|
||||
|
@ -1524,9 +1528,11 @@ google.golang.org/genproto v0.0.0-20211118181313-81c1377c94b1/go.mod h1:5CzLGKJ6
|
|||
google.golang.org/genproto v0.0.0-20211206160659-862468c7d6e0/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc=
|
||||
google.golang.org/genproto v0.0.0-20211208223120-3a66f561d7aa/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc=
|
||||
google.golang.org/genproto v0.0.0-20211221195035-429b39de9b1c/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc=
|
||||
google.golang.org/genproto v0.0.0-20211223182754-3ac035c7e7cb/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc=
|
||||
google.golang.org/genproto v0.0.0-20220107163113-42d7afdf6368/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc=
|
||||
google.golang.org/genproto v0.0.0-20220114231437-d2e6a121cae0 h1:aCsSLXylHWFno0r4S3joLpiaWayvqd2Mn4iSvx4WZZc=
|
||||
google.golang.org/genproto v0.0.0-20220114231437-d2e6a121cae0/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc=
|
||||
google.golang.org/genproto v0.0.0-20220111164026-67b88f271998/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc=
|
||||
google.golang.org/genproto v0.0.0-20220118154757-00ab72f36ad5 h1:zzNejm+EgrbLfDZ6lu9Uud2IVvHySPl8vQzf04laR5Q=
|
||||
google.golang.org/genproto v0.0.0-20220118154757-00ab72f36ad5/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc=
|
||||
google.golang.org/grpc v1.17.0/go.mod h1:6QZJwpn2B+Zp71q/5VxRsJ6NXXVCE5NRUHRo+f3cWCs=
|
||||
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
|
||||
google.golang.org/grpc v1.20.0/go.mod h1:chYK+tFQF0nDUGJgXMSgLCQk3phJEuONr2DCgLDdAQM=
|
||||
|
|
225
lib/blockcache/blockcache.go
Normal file
225
lib/blockcache/blockcache.go
Normal file
|
@ -0,0 +1,225 @@
|
|||
package blockcache
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/fasttime"
|
||||
)
|
||||
|
||||
// Cache caches Block entries.
|
||||
//
|
||||
// Call NewCache() for creating new Cache.
|
||||
type Cache struct {
|
||||
// Atomically updated fields must go first in the struct, so they are properly
|
||||
// aligned to 8 bytes on 32-bit architectures.
|
||||
// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
|
||||
requests uint64
|
||||
misses uint64
|
||||
|
||||
// sizeBytes contains an approximate size for all the blocks stored in the cache.
|
||||
sizeBytes int64
|
||||
|
||||
// getMaxSizeBytes() is a callback, which returns the maximum allowed cache size in bytes.
|
||||
getMaxSizeBytes func() int
|
||||
|
||||
// mu protects all the fields below.
|
||||
mu sync.RWMutex
|
||||
|
||||
// m contains cached blocks keyed by Key.Part and then by Key.Offset
|
||||
m map[interface{}]map[uint64]*cacheEntry
|
||||
|
||||
// perKeyMisses contains per-block cache misses.
|
||||
//
|
||||
// Blocks with less than 2 cache misses aren't stored in the cache in order to prevent from eviction for frequently accessed items.
|
||||
perKeyMisses map[Key]int
|
||||
}
|
||||
|
||||
// Key represents a key, which uniquely identifies the Block.
|
||||
type Key struct {
|
||||
// Part must contain a pointer to part structure where the block belongs to.
|
||||
Part interface{}
|
||||
|
||||
// Offset is the offset of the block in the part.
|
||||
Offset uint64
|
||||
}
|
||||
|
||||
// Block is an item, which may be cached in the Cache.
|
||||
type Block interface {
|
||||
// SizeBytes must return the approximate size of the given block in bytes
|
||||
SizeBytes() int
|
||||
}
|
||||
|
||||
type cacheEntry struct {
|
||||
// Atomically updated fields must go first in the struct, so they are properly
|
||||
// aligned to 8 bytes on 32-bit architectures.
|
||||
// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
|
||||
lastAccessTime uint64
|
||||
|
||||
// block contains the cached block.
|
||||
block Block
|
||||
}
|
||||
|
||||
// NewCache creates new cache.
|
||||
//
|
||||
// Cache size in bytes is limited by the value returned by getMaxSizeBytes() callback.
|
||||
func NewCache(getMaxSizeBytes func() int) *Cache {
|
||||
var c Cache
|
||||
c.getMaxSizeBytes = getMaxSizeBytes
|
||||
c.m = make(map[interface{}]map[uint64]*cacheEntry)
|
||||
c.perKeyMisses = make(map[Key]int)
|
||||
go c.cleaner()
|
||||
return &c
|
||||
}
|
||||
|
||||
// RemoveBlocksForPart removes all the blocks for the given part from the cache.
|
||||
func (c *Cache) RemoveBlocksForPart(p interface{}) {
|
||||
c.mu.Lock()
|
||||
sizeBytes := 0
|
||||
for _, e := range c.m[p] {
|
||||
sizeBytes += e.block.SizeBytes()
|
||||
// do not delete the entry from c.perKeyMisses, since it is removed by Cache.cleaner later.
|
||||
}
|
||||
c.updateSizeBytes(-sizeBytes)
|
||||
delete(c.m, p)
|
||||
c.mu.Unlock()
|
||||
}
|
||||
|
||||
func (c *Cache) updateSizeBytes(n int) {
|
||||
atomic.AddInt64(&c.sizeBytes, int64(n))
|
||||
}
|
||||
|
||||
// cleaner periodically cleans least recently used entries in c.
|
||||
func (c *Cache) cleaner() {
|
||||
ticker := time.NewTicker(30 * time.Second)
|
||||
defer ticker.Stop()
|
||||
perKeyMissesTicker := time.NewTicker(2 * time.Minute)
|
||||
defer perKeyMissesTicker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ticker.C:
|
||||
c.cleanByTimeout()
|
||||
case <-perKeyMissesTicker.C:
|
||||
c.mu.Lock()
|
||||
c.perKeyMisses = make(map[Key]int, len(c.perKeyMisses))
|
||||
c.mu.Unlock()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Cache) cleanByTimeout() {
|
||||
currentTime := fasttime.UnixTimestamp()
|
||||
c.mu.Lock()
|
||||
for _, pes := range c.m {
|
||||
for offset, e := range pes {
|
||||
// Delete items accessed more than two minutes ago.
|
||||
// This time should be enough for repeated queries.
|
||||
if currentTime-atomic.LoadUint64(&e.lastAccessTime) > 2*60 {
|
||||
c.updateSizeBytes(-e.block.SizeBytes())
|
||||
delete(pes, offset)
|
||||
// do not delete the entry from c.perKeyMisses, since it is removed by Cache.cleaner later.
|
||||
}
|
||||
}
|
||||
}
|
||||
c.mu.Unlock()
|
||||
}
|
||||
|
||||
// GetBlock returns a block for the given key k from c.
|
||||
func (c *Cache) GetBlock(k Key) Block {
|
||||
atomic.AddUint64(&c.requests, 1)
|
||||
var e *cacheEntry
|
||||
c.mu.RLock()
|
||||
pes := c.m[k.Part]
|
||||
if pes != nil {
|
||||
e = pes[k.Offset]
|
||||
}
|
||||
c.mu.RUnlock()
|
||||
if e != nil {
|
||||
// Fast path - the block already exists in the cache, so return it to the caller.
|
||||
currentTime := fasttime.UnixTimestamp()
|
||||
if atomic.LoadUint64(&e.lastAccessTime) != currentTime {
|
||||
atomic.StoreUint64(&e.lastAccessTime, currentTime)
|
||||
}
|
||||
return e.block
|
||||
}
|
||||
// Slow path - the entry is missing in the cache.
|
||||
c.mu.Lock()
|
||||
c.perKeyMisses[k]++
|
||||
c.mu.Unlock()
|
||||
atomic.AddUint64(&c.misses, 1)
|
||||
return nil
|
||||
}
|
||||
|
||||
// PutBlock puts the given block b under the given key k into c.
|
||||
func (c *Cache) PutBlock(k Key, b Block) {
|
||||
c.mu.RLock()
|
||||
// If the entry wasn't accessed yet (e.g. c.perKeyMisses[k] == 0), then cache it, since it is likely it will be accessed soon.
|
||||
// Do not cache the entry only if there was only a single unsuccessful attempt to access it.
|
||||
// This may be one-time-wonders entry, which won't be accessed more, so there is no need in caching it.
|
||||
doNotCache := c.perKeyMisses[k] == 1
|
||||
c.mu.RUnlock()
|
||||
if doNotCache {
|
||||
// Do not cache b if it has been requested only once (aka one-time-wonders items).
|
||||
// This should reduce memory usage for the cache.
|
||||
return
|
||||
}
|
||||
|
||||
// Store b in the cache.
|
||||
c.mu.Lock()
|
||||
e := &cacheEntry{
|
||||
lastAccessTime: fasttime.UnixTimestamp(),
|
||||
block: b,
|
||||
}
|
||||
pes := c.m[k.Part]
|
||||
if pes == nil {
|
||||
pes = make(map[uint64]*cacheEntry)
|
||||
c.m[k.Part] = pes
|
||||
}
|
||||
pes[k.Offset] = e
|
||||
c.updateSizeBytes(e.block.SizeBytes())
|
||||
maxSizeBytes := c.getMaxSizeBytes()
|
||||
if c.SizeBytes() > maxSizeBytes {
|
||||
// Entries in the cache occupy too much space. Free up space by deleting some entries.
|
||||
for _, pes := range c.m {
|
||||
for offset, e := range pes {
|
||||
c.updateSizeBytes(-e.block.SizeBytes())
|
||||
delete(pes, offset)
|
||||
// do not delete the entry from c.perKeyMisses, since it is removed by Cache.cleaner later.
|
||||
if c.SizeBytes() < maxSizeBytes {
|
||||
goto end
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
end:
|
||||
c.mu.Unlock()
|
||||
}
|
||||
|
||||
// Len returns the number of blocks in the cache c.
|
||||
func (c *Cache) Len() int {
|
||||
c.mu.RLock()
|
||||
n := len(c.m)
|
||||
c.mu.RUnlock()
|
||||
return n
|
||||
}
|
||||
|
||||
// SizeBytes returns an approximate size in bytes of all the blocks stored in the cache c.
|
||||
func (c *Cache) SizeBytes() int {
|
||||
return int(atomic.LoadInt64(&c.sizeBytes))
|
||||
}
|
||||
|
||||
// SizeMaxBytes returns the max allowed size in bytes for c.
|
||||
func (c *Cache) SizeMaxBytes() int {
|
||||
return c.getMaxSizeBytes()
|
||||
}
|
||||
|
||||
// Requests returns the number of requests served by c.
|
||||
func (c *Cache) Requests() uint64 {
|
||||
return atomic.LoadUint64(&c.requests)
|
||||
}
|
||||
|
||||
// Misses returns the number of cache misses for c.
|
||||
func (c *Cache) Misses() uint64 {
|
||||
return atomic.LoadUint64(&c.misses)
|
||||
}
|
|
@ -4,46 +4,39 @@ import (
|
|||
"fmt"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
"unsafe"
|
||||
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/fasttime"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/blockcache"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/filestream"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/fs"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/memory"
|
||||
)
|
||||
|
||||
func getMaxCachedIndexBlocksPerPart() int {
|
||||
maxCachedIndexBlocksPerPartOnce.Do(func() {
|
||||
n := memory.Allowed() / 1024 / 1024 / 4
|
||||
if n == 0 {
|
||||
n = 10
|
||||
}
|
||||
maxCachedIndexBlocksPerPart = n
|
||||
var idxbCache = blockcache.NewCache(getMaxIndexBlocksCacheSize)
|
||||
var ibCache = blockcache.NewCache(getMaxInmemoryBlocksCacheSize)
|
||||
|
||||
func getMaxIndexBlocksCacheSize() int {
|
||||
maxIndexBlockCacheSizeOnce.Do(func() {
|
||||
maxIndexBlockCacheSize = int(0.2 * float64(memory.Allowed()))
|
||||
})
|
||||
return maxCachedIndexBlocksPerPart
|
||||
return maxIndexBlockCacheSize
|
||||
}
|
||||
|
||||
var (
|
||||
maxCachedIndexBlocksPerPart int
|
||||
maxCachedIndexBlocksPerPartOnce sync.Once
|
||||
maxIndexBlockCacheSize int
|
||||
maxIndexBlockCacheSizeOnce sync.Once
|
||||
)
|
||||
|
||||
func getMaxCachedInmemoryBlocksPerPart() int {
|
||||
maxCachedInmemoryBlocksPerPartOnce.Do(func() {
|
||||
n := memory.Allowed() / 1024 / 1024 / 4
|
||||
if n == 0 {
|
||||
n = 10
|
||||
}
|
||||
maxCachedInmemoryBlocksPerPart = n
|
||||
func getMaxInmemoryBlocksCacheSize() int {
|
||||
maxInmemoryBlockCacheSizeOnce.Do(func() {
|
||||
maxInmemoryBlockCacheSize = int(0.3 * float64(memory.Allowed()))
|
||||
})
|
||||
return maxCachedInmemoryBlocksPerPart
|
||||
return maxInmemoryBlockCacheSize
|
||||
}
|
||||
|
||||
var (
|
||||
maxCachedInmemoryBlocksPerPart int
|
||||
maxCachedInmemoryBlocksPerPartOnce sync.Once
|
||||
maxInmemoryBlockCacheSize int
|
||||
maxInmemoryBlockCacheSizeOnce sync.Once
|
||||
)
|
||||
|
||||
type part struct {
|
||||
|
@ -58,9 +51,6 @@ type part struct {
|
|||
indexFile fs.MustReadAtCloser
|
||||
itemsFile fs.MustReadAtCloser
|
||||
lensFile fs.MustReadAtCloser
|
||||
|
||||
idxbCache *indexBlockCache
|
||||
ibCache *inmemoryBlockCache
|
||||
}
|
||||
|
||||
func openFilePart(path string) (*part, error) {
|
||||
|
@ -112,9 +102,6 @@ func newPart(ph *partHeader, path string, size uint64, metaindexReader filestrea
|
|||
p.lensFile = lensFile
|
||||
|
||||
p.ph.CopyFrom(ph)
|
||||
p.idxbCache = newIndexBlockCache()
|
||||
p.ibCache = newInmemoryBlockCache()
|
||||
|
||||
if len(errors) > 0 {
|
||||
// Return only the first error, since it has no sense in returning all errors.
|
||||
err := fmt.Errorf("error opening part %s: %w", p.path, errors[0])
|
||||
|
@ -129,8 +116,8 @@ func (p *part) MustClose() {
|
|||
p.itemsFile.MustClose()
|
||||
p.lensFile.MustClose()
|
||||
|
||||
p.idxbCache.MustClose()
|
||||
p.ibCache.MustClose()
|
||||
idxbCache.RemoveBlocksForPart(p)
|
||||
ibCache.RemoveBlocksForPart(p)
|
||||
}
|
||||
|
||||
type indexBlock struct {
|
||||
|
@ -145,346 +132,3 @@ func (idxb *indexBlock) SizeBytes() int {
|
|||
}
|
||||
return n
|
||||
}
|
||||
|
||||
type indexBlockCache struct {
|
||||
// Atomically updated counters must go first in the struct, so they are properly
|
||||
// aligned to 8 bytes on 32-bit architectures.
|
||||
// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
|
||||
requests uint64
|
||||
misses uint64
|
||||
|
||||
m map[uint64]*indexBlockCacheEntry
|
||||
mu sync.RWMutex
|
||||
|
||||
perKeyMisses map[uint64]int
|
||||
perKeyMissesLock sync.Mutex
|
||||
|
||||
cleanerStopCh chan struct{}
|
||||
cleanerWG sync.WaitGroup
|
||||
}
|
||||
|
||||
type indexBlockCacheEntry struct {
|
||||
// Atomically updated counters must go first in the struct, so they are properly
|
||||
// aligned to 8 bytes on 32-bit architectures.
|
||||
// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
|
||||
lastAccessTime uint64
|
||||
|
||||
idxb *indexBlock
|
||||
}
|
||||
|
||||
func newIndexBlockCache() *indexBlockCache {
|
||||
var idxbc indexBlockCache
|
||||
idxbc.m = make(map[uint64]*indexBlockCacheEntry)
|
||||
idxbc.perKeyMisses = make(map[uint64]int)
|
||||
idxbc.cleanerStopCh = make(chan struct{})
|
||||
idxbc.cleanerWG.Add(1)
|
||||
go func() {
|
||||
defer idxbc.cleanerWG.Done()
|
||||
idxbc.cleaner()
|
||||
}()
|
||||
return &idxbc
|
||||
}
|
||||
|
||||
func (idxbc *indexBlockCache) MustClose() {
|
||||
close(idxbc.cleanerStopCh)
|
||||
idxbc.cleanerWG.Wait()
|
||||
idxbc.m = nil
|
||||
idxbc.perKeyMisses = nil
|
||||
}
|
||||
|
||||
// cleaner periodically cleans least recently used items.
|
||||
func (idxbc *indexBlockCache) cleaner() {
|
||||
ticker := time.NewTicker(30 * time.Second)
|
||||
defer ticker.Stop()
|
||||
perKeyMissesTicker := time.NewTicker(2 * time.Minute)
|
||||
defer perKeyMissesTicker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ticker.C:
|
||||
idxbc.cleanByTimeout()
|
||||
case <-perKeyMissesTicker.C:
|
||||
idxbc.perKeyMissesLock.Lock()
|
||||
idxbc.perKeyMisses = make(map[uint64]int, len(idxbc.perKeyMisses))
|
||||
idxbc.perKeyMissesLock.Unlock()
|
||||
case <-idxbc.cleanerStopCh:
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (idxbc *indexBlockCache) cleanByTimeout() {
|
||||
currentTime := fasttime.UnixTimestamp()
|
||||
idxbc.mu.Lock()
|
||||
for k, idxbe := range idxbc.m {
|
||||
// Delete items accessed more than two minutes ago.
|
||||
// This time should be enough for repeated queries.
|
||||
if currentTime-atomic.LoadUint64(&idxbe.lastAccessTime) > 2*60 {
|
||||
delete(idxbc.m, k)
|
||||
}
|
||||
}
|
||||
idxbc.mu.Unlock()
|
||||
}
|
||||
|
||||
func (idxbc *indexBlockCache) Get(k uint64) *indexBlock {
|
||||
atomic.AddUint64(&idxbc.requests, 1)
|
||||
idxbc.mu.RLock()
|
||||
idxbe := idxbc.m[k]
|
||||
idxbc.mu.RUnlock()
|
||||
|
||||
if idxbe != nil {
|
||||
currentTime := fasttime.UnixTimestamp()
|
||||
if atomic.LoadUint64(&idxbe.lastAccessTime) != currentTime {
|
||||
atomic.StoreUint64(&idxbe.lastAccessTime, currentTime)
|
||||
}
|
||||
return idxbe.idxb
|
||||
}
|
||||
idxbc.perKeyMissesLock.Lock()
|
||||
idxbc.perKeyMisses[k]++
|
||||
idxbc.perKeyMissesLock.Unlock()
|
||||
atomic.AddUint64(&idxbc.misses, 1)
|
||||
return nil
|
||||
}
|
||||
|
||||
// Put puts idxb under the key k into idxbc.
|
||||
func (idxbc *indexBlockCache) Put(k uint64, idxb *indexBlock) {
|
||||
idxbc.perKeyMissesLock.Lock()
|
||||
doNotCache := idxbc.perKeyMisses[k] == 1
|
||||
idxbc.perKeyMissesLock.Unlock()
|
||||
if doNotCache {
|
||||
// Do not cache ib if it has been requested only once (aka one-time-wonders items).
|
||||
// This should reduce memory usage for the ibc cache.
|
||||
return
|
||||
}
|
||||
|
||||
idxbc.mu.Lock()
|
||||
// Remove superfluous entries.
|
||||
if overflow := len(idxbc.m) - getMaxCachedIndexBlocksPerPart(); overflow > 0 {
|
||||
// Remove 10% of items from the cache.
|
||||
overflow = int(float64(len(idxbc.m)) * 0.1)
|
||||
for k := range idxbc.m {
|
||||
delete(idxbc.m, k)
|
||||
overflow--
|
||||
if overflow == 0 {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Store idxb in the cache.
|
||||
idxbe := &indexBlockCacheEntry{
|
||||
lastAccessTime: fasttime.UnixTimestamp(),
|
||||
idxb: idxb,
|
||||
}
|
||||
idxbc.m[k] = idxbe
|
||||
idxbc.mu.Unlock()
|
||||
}
|
||||
|
||||
func (idxbc *indexBlockCache) Len() uint64 {
|
||||
idxbc.mu.RLock()
|
||||
n := len(idxbc.m)
|
||||
idxbc.mu.RUnlock()
|
||||
return uint64(n)
|
||||
}
|
||||
|
||||
func (idxbc *indexBlockCache) SizeBytes() uint64 {
|
||||
n := 0
|
||||
idxbc.mu.RLock()
|
||||
for _, e := range idxbc.m {
|
||||
n += e.idxb.SizeBytes()
|
||||
}
|
||||
idxbc.mu.RUnlock()
|
||||
return uint64(n)
|
||||
}
|
||||
|
||||
func (idxbc *indexBlockCache) SizeMaxBytes() uint64 {
|
||||
avgBlockSize := float64(64 * 1024)
|
||||
blocksCount := idxbc.Len()
|
||||
if blocksCount > 0 {
|
||||
avgBlockSize = float64(idxbc.SizeBytes()) / float64(blocksCount)
|
||||
}
|
||||
return uint64(avgBlockSize * float64(getMaxCachedIndexBlocksPerPart()))
|
||||
}
|
||||
|
||||
func (idxbc *indexBlockCache) Requests() uint64 {
|
||||
return atomic.LoadUint64(&idxbc.requests)
|
||||
}
|
||||
|
||||
func (idxbc *indexBlockCache) Misses() uint64 {
|
||||
return atomic.LoadUint64(&idxbc.misses)
|
||||
}
|
||||
|
||||
type inmemoryBlockCache struct {
|
||||
// Atomically updated counters must go first in the struct, so they are properly
|
||||
// aligned to 8 bytes on 32-bit architectures.
|
||||
// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
|
||||
requests uint64
|
||||
misses uint64
|
||||
|
||||
m map[inmemoryBlockCacheKey]*inmemoryBlockCacheEntry
|
||||
mu sync.RWMutex
|
||||
|
||||
perKeyMisses map[inmemoryBlockCacheKey]int
|
||||
perKeyMissesLock sync.Mutex
|
||||
|
||||
cleanerStopCh chan struct{}
|
||||
cleanerWG sync.WaitGroup
|
||||
}
|
||||
|
||||
type inmemoryBlockCacheKey struct {
|
||||
itemsBlockOffset uint64
|
||||
}
|
||||
|
||||
func (ibck *inmemoryBlockCacheKey) Init(bh *blockHeader) {
|
||||
ibck.itemsBlockOffset = bh.itemsBlockOffset
|
||||
}
|
||||
|
||||
type inmemoryBlockCacheEntry struct {
|
||||
// Atomically updated counters must go first in the struct, so they are properly
|
||||
// aligned to 8 bytes on 32-bit architectures.
|
||||
// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
|
||||
lastAccessTime uint64
|
||||
|
||||
ib *inmemoryBlock
|
||||
}
|
||||
|
||||
func newInmemoryBlockCache() *inmemoryBlockCache {
|
||||
var ibc inmemoryBlockCache
|
||||
ibc.m = make(map[inmemoryBlockCacheKey]*inmemoryBlockCacheEntry)
|
||||
ibc.perKeyMisses = make(map[inmemoryBlockCacheKey]int)
|
||||
ibc.cleanerStopCh = make(chan struct{})
|
||||
ibc.cleanerWG.Add(1)
|
||||
go func() {
|
||||
defer ibc.cleanerWG.Done()
|
||||
ibc.cleaner()
|
||||
}()
|
||||
return &ibc
|
||||
}
|
||||
|
||||
func (ibc *inmemoryBlockCache) MustClose() {
|
||||
close(ibc.cleanerStopCh)
|
||||
ibc.cleanerWG.Wait()
|
||||
ibc.m = nil
|
||||
ibc.perKeyMisses = nil
|
||||
}
|
||||
|
||||
// cleaner periodically cleans least recently used items.
|
||||
func (ibc *inmemoryBlockCache) cleaner() {
|
||||
ticker := time.NewTicker(30 * time.Second)
|
||||
defer ticker.Stop()
|
||||
perKeyMissesTicker := time.NewTicker(2 * time.Minute)
|
||||
defer perKeyMissesTicker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ticker.C:
|
||||
ibc.cleanByTimeout()
|
||||
case <-perKeyMissesTicker.C:
|
||||
ibc.perKeyMissesLock.Lock()
|
||||
ibc.perKeyMisses = make(map[inmemoryBlockCacheKey]int, len(ibc.perKeyMisses))
|
||||
ibc.perKeyMissesLock.Unlock()
|
||||
case <-ibc.cleanerStopCh:
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (ibc *inmemoryBlockCache) cleanByTimeout() {
|
||||
currentTime := fasttime.UnixTimestamp()
|
||||
ibc.mu.Lock()
|
||||
for k, ibe := range ibc.m {
|
||||
// Delete items accessed more than two minutes ago.
|
||||
// This time should be enough for repeated queries.
|
||||
if currentTime-atomic.LoadUint64(&ibe.lastAccessTime) > 2*60 {
|
||||
delete(ibc.m, k)
|
||||
}
|
||||
}
|
||||
ibc.mu.Unlock()
|
||||
}
|
||||
|
||||
func (ibc *inmemoryBlockCache) Get(k inmemoryBlockCacheKey) *inmemoryBlock {
|
||||
atomic.AddUint64(&ibc.requests, 1)
|
||||
|
||||
ibc.mu.RLock()
|
||||
ibe := ibc.m[k]
|
||||
ibc.mu.RUnlock()
|
||||
|
||||
if ibe != nil {
|
||||
currentTime := fasttime.UnixTimestamp()
|
||||
if atomic.LoadUint64(&ibe.lastAccessTime) != currentTime {
|
||||
atomic.StoreUint64(&ibe.lastAccessTime, currentTime)
|
||||
}
|
||||
return ibe.ib
|
||||
}
|
||||
ibc.perKeyMissesLock.Lock()
|
||||
ibc.perKeyMisses[k]++
|
||||
ibc.perKeyMissesLock.Unlock()
|
||||
atomic.AddUint64(&ibc.misses, 1)
|
||||
return nil
|
||||
}
|
||||
|
||||
// Put puts ib under key k into ibc.
|
||||
func (ibc *inmemoryBlockCache) Put(k inmemoryBlockCacheKey, ib *inmemoryBlock) {
|
||||
ibc.perKeyMissesLock.Lock()
|
||||
doNotCache := ibc.perKeyMisses[k] == 1
|
||||
ibc.perKeyMissesLock.Unlock()
|
||||
if doNotCache {
|
||||
// Do not cache ib if it has been requested only once (aka one-time-wonders items).
|
||||
// This should reduce memory usage for the ibc cache.
|
||||
return
|
||||
}
|
||||
|
||||
ibc.mu.Lock()
|
||||
// Clean superfluous entries in cache.
|
||||
if overflow := len(ibc.m) - getMaxCachedInmemoryBlocksPerPart(); overflow > 0 {
|
||||
// Remove 10% of items from the cache.
|
||||
overflow = int(float64(len(ibc.m)) * 0.1)
|
||||
for k := range ibc.m {
|
||||
delete(ibc.m, k)
|
||||
overflow--
|
||||
if overflow == 0 {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Store ib in the cache.
|
||||
ibe := &inmemoryBlockCacheEntry{
|
||||
lastAccessTime: fasttime.UnixTimestamp(),
|
||||
ib: ib,
|
||||
}
|
||||
ibc.m[k] = ibe
|
||||
ibc.mu.Unlock()
|
||||
}
|
||||
|
||||
func (ibc *inmemoryBlockCache) Len() uint64 {
|
||||
ibc.mu.RLock()
|
||||
n := len(ibc.m)
|
||||
ibc.mu.RUnlock()
|
||||
return uint64(n)
|
||||
}
|
||||
|
||||
func (ibc *inmemoryBlockCache) SizeBytes() uint64 {
|
||||
n := 0
|
||||
ibc.mu.RLock()
|
||||
for _, e := range ibc.m {
|
||||
n += e.ib.SizeBytes()
|
||||
}
|
||||
ibc.mu.RUnlock()
|
||||
return uint64(n)
|
||||
}
|
||||
|
||||
func (ibc *inmemoryBlockCache) SizeMaxBytes() uint64 {
|
||||
avgBlockSize := float64(128 * 1024)
|
||||
blocksCount := ibc.Len()
|
||||
if blocksCount > 0 {
|
||||
avgBlockSize = float64(ibc.SizeBytes()) / float64(blocksCount)
|
||||
}
|
||||
return uint64(avgBlockSize * float64(getMaxCachedInmemoryBlocksPerPart()))
|
||||
}
|
||||
|
||||
func (ibc *inmemoryBlockCache) Requests() uint64 {
|
||||
return atomic.LoadUint64(&ibc.requests)
|
||||
}
|
||||
|
||||
func (ibc *inmemoryBlockCache) Misses() uint64 {
|
||||
return atomic.LoadUint64(&ibc.misses)
|
||||
}
|
||||
|
|
|
@ -5,6 +5,7 @@ import (
|
|||
"io"
|
||||
"sort"
|
||||
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/blockcache"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/encoding"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
|
||||
|
@ -25,9 +26,6 @@ type partSearch struct {
|
|||
// The remaining block headers to scan in the current metaindexRow.
|
||||
bhs []blockHeader
|
||||
|
||||
idxbCache *indexBlockCache
|
||||
ibCache *inmemoryBlockCache
|
||||
|
||||
// err contains the last error.
|
||||
err error
|
||||
|
||||
|
@ -45,8 +43,6 @@ func (ps *partSearch) reset() {
|
|||
ps.p = nil
|
||||
ps.mrs = nil
|
||||
ps.bhs = nil
|
||||
ps.idxbCache = nil
|
||||
ps.ibCache = nil
|
||||
ps.err = nil
|
||||
|
||||
ps.indexBuf = ps.indexBuf[:0]
|
||||
|
@ -65,8 +61,6 @@ func (ps *partSearch) Init(p *part) {
|
|||
ps.reset()
|
||||
|
||||
ps.p = p
|
||||
ps.idxbCache = p.idxbCache
|
||||
ps.ibCache = p.ibCache
|
||||
}
|
||||
|
||||
// Seek seeks for the first item greater or equal to k in ps.
|
||||
|
@ -261,16 +255,20 @@ func (ps *partSearch) nextBHS() error {
|
|||
}
|
||||
mr := &ps.mrs[0]
|
||||
ps.mrs = ps.mrs[1:]
|
||||
idxbKey := mr.indexBlockOffset
|
||||
idxb := ps.idxbCache.Get(idxbKey)
|
||||
if idxb == nil {
|
||||
var err error
|
||||
idxb, err = ps.readIndexBlock(mr)
|
||||
idxbKey := blockcache.Key{
|
||||
Part: ps.p,
|
||||
Offset: mr.indexBlockOffset,
|
||||
}
|
||||
b := idxbCache.GetBlock(idxbKey)
|
||||
if b == nil {
|
||||
idxb, err := ps.readIndexBlock(mr)
|
||||
if err != nil {
|
||||
return fmt.Errorf("cannot read index block: %w", err)
|
||||
}
|
||||
ps.idxbCache.Put(idxbKey, idxb)
|
||||
b = idxb
|
||||
idxbCache.PutBlock(idxbKey, b)
|
||||
}
|
||||
idxb := b.(*indexBlock)
|
||||
ps.bhs = idxb.bhs
|
||||
return nil
|
||||
}
|
||||
|
@ -293,17 +291,20 @@ func (ps *partSearch) readIndexBlock(mr *metaindexRow) (*indexBlock, error) {
|
|||
}
|
||||
|
||||
func (ps *partSearch) getInmemoryBlock(bh *blockHeader) (*inmemoryBlock, error) {
|
||||
var ibKey inmemoryBlockCacheKey
|
||||
ibKey.Init(bh)
|
||||
ib := ps.ibCache.Get(ibKey)
|
||||
if ib != nil {
|
||||
return ib, nil
|
||||
ibKey := blockcache.Key{
|
||||
Part: ps.p,
|
||||
Offset: bh.itemsBlockOffset,
|
||||
}
|
||||
ib, err := ps.readInmemoryBlock(bh)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
b := ibCache.GetBlock(ibKey)
|
||||
if b == nil {
|
||||
ib, err := ps.readInmemoryBlock(bh)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
b = ib
|
||||
ibCache.PutBlock(ibKey, b)
|
||||
}
|
||||
ps.ibCache.Put(ibKey, ib)
|
||||
ib := b.(*inmemoryBlock)
|
||||
return ib, nil
|
||||
}
|
||||
|
||||
|
|
|
@ -22,16 +22,6 @@ import (
|
|||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/syncwg"
|
||||
)
|
||||
|
||||
// These are global counters for cache requests and misses for parts
|
||||
// which were already merged into another parts.
|
||||
var (
|
||||
historicalDataBlockCacheRequests uint64
|
||||
historicalDataBlockCacheMisses uint64
|
||||
|
||||
historicalIndexBlockCacheRequests uint64
|
||||
historicalIndexBlockCacheMisses uint64
|
||||
)
|
||||
|
||||
// maxParts is the maximum number of parts in the table.
|
||||
//
|
||||
// This number may be reached when the insertion pace outreaches merger pace.
|
||||
|
@ -443,27 +433,21 @@ func (tb *Table) UpdateMetrics(m *TableMetrics) {
|
|||
m.ItemsCount += p.ph.itemsCount
|
||||
m.SizeBytes += p.size
|
||||
|
||||
m.DataBlocksCacheSize += p.ibCache.Len()
|
||||
m.DataBlocksCacheSizeBytes += p.ibCache.SizeBytes()
|
||||
m.DataBlocksCacheSizeMaxBytes += p.ibCache.SizeMaxBytes()
|
||||
m.DataBlocksCacheRequests += p.ibCache.Requests()
|
||||
m.DataBlocksCacheMisses += p.ibCache.Misses()
|
||||
|
||||
m.IndexBlocksCacheSize += p.idxbCache.Len()
|
||||
m.IndexBlocksCacheSizeBytes += p.idxbCache.SizeBytes()
|
||||
m.IndexBlocksCacheSizeMaxBytes += p.idxbCache.SizeMaxBytes()
|
||||
m.IndexBlocksCacheRequests += p.idxbCache.Requests()
|
||||
m.IndexBlocksCacheMisses += p.idxbCache.Misses()
|
||||
|
||||
m.PartsRefCount += atomic.LoadUint64(&pw.refCount)
|
||||
}
|
||||
tb.partsLock.Unlock()
|
||||
|
||||
m.DataBlocksCacheRequests = atomic.LoadUint64(&historicalDataBlockCacheRequests)
|
||||
m.DataBlocksCacheMisses = atomic.LoadUint64(&historicalDataBlockCacheMisses)
|
||||
m.DataBlocksCacheSize = uint64(ibCache.Len())
|
||||
m.DataBlocksCacheSizeBytes = uint64(ibCache.SizeBytes())
|
||||
m.DataBlocksCacheSizeMaxBytes = uint64(ibCache.SizeMaxBytes())
|
||||
m.DataBlocksCacheRequests = ibCache.Requests()
|
||||
m.DataBlocksCacheMisses = ibCache.Misses()
|
||||
|
||||
m.IndexBlocksCacheRequests = atomic.LoadUint64(&historicalIndexBlockCacheRequests)
|
||||
m.IndexBlocksCacheMisses = atomic.LoadUint64(&historicalIndexBlockCacheMisses)
|
||||
m.IndexBlocksCacheSize = uint64(idxbCache.Len())
|
||||
m.IndexBlocksCacheSizeBytes = uint64(idxbCache.SizeBytes())
|
||||
m.IndexBlocksCacheSizeMaxBytes = uint64(idxbCache.SizeMaxBytes())
|
||||
m.IndexBlocksCacheRequests = idxbCache.Requests()
|
||||
m.IndexBlocksCacheMisses = idxbCache.Misses()
|
||||
}
|
||||
|
||||
// AddItems adds the given items to the tb.
|
||||
|
@ -1466,10 +1450,6 @@ func removeParts(pws []*partWrapper, partsToRemove map[*partWrapper]bool) ([]*pa
|
|||
dst = append(dst, pw)
|
||||
continue
|
||||
}
|
||||
atomic.AddUint64(&historicalDataBlockCacheRequests, pw.p.ibCache.Requests())
|
||||
atomic.AddUint64(&historicalDataBlockCacheMisses, pw.p.ibCache.Misses())
|
||||
atomic.AddUint64(&historicalIndexBlockCacheRequests, pw.p.idxbCache.Requests())
|
||||
atomic.AddUint64(&historicalIndexBlockCacheMisses, pw.p.idxbCache.Misses())
|
||||
removedParts++
|
||||
}
|
||||
return dst, removedParts
|
||||
|
|
|
@ -88,12 +88,14 @@ func (bsm *blockStreamMerger) nextBlock() error {
|
|||
}
|
||||
|
||||
if err := bsrMin.Error(); err != nil {
|
||||
bsm.Block = nil
|
||||
return err
|
||||
}
|
||||
|
||||
heap.Pop(&bsm.bsrHeap)
|
||||
|
||||
if len(bsm.bsrHeap) == 0 {
|
||||
bsm.Block = nil
|
||||
return io.EOF
|
||||
}
|
||||
|
||||
|
|
|
@ -175,7 +175,7 @@ func (bsr *blockStreamReader) InitFromFilePart(path string) error {
|
|||
timestampsFile.MustClose()
|
||||
valuesFile.MustClose()
|
||||
indexFile.MustClose()
|
||||
return fmt.Errorf("cannot unmarshal metaindex rows from inmemoryPart: %w", err)
|
||||
return fmt.Errorf("cannot unmarshal metaindex rows from file part %q: %w", metaindexPath, err)
|
||||
}
|
||||
|
||||
bsr.path = path
|
||||
|
@ -213,16 +213,19 @@ func (bsr *blockStreamReader) NextBlock() bool {
|
|||
if bsr.err != nil {
|
||||
return false
|
||||
}
|
||||
|
||||
tsidPrev := bsr.Block.bh.TSID
|
||||
bsr.Block.Reset()
|
||||
|
||||
err := bsr.readBlock()
|
||||
if err == nil {
|
||||
if bsr.Block.bh.RowsCount > 0 {
|
||||
return true
|
||||
if bsr.Block.bh.TSID.Less(&tsidPrev) {
|
||||
bsr.err = fmt.Errorf("possible data corruption: the next TSID=%v is smaller than the previous TSID=%v", &bsr.Block.bh.TSID, &tsidPrev)
|
||||
return false
|
||||
}
|
||||
bsr.err = fmt.Errorf("invalid block read with zero rows; block=%+v", &bsr.Block)
|
||||
return false
|
||||
if bsr.Block.bh.RowsCount == 0 {
|
||||
bsr.err = fmt.Errorf("invalid block read with zero rows; block=%+v", &bsr.Block)
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
if err == io.EOF {
|
||||
bsr.err = io.EOF
|
||||
|
|
|
@ -4,30 +4,26 @@ import (
|
|||
"fmt"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
"unsafe"
|
||||
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/fasttime"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/blockcache"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/filestream"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/fs"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/memory"
|
||||
)
|
||||
|
||||
func getMaxCachedIndexBlocksPerPart() int {
|
||||
maxCachedIndexBlocksPerPartOnce.Do(func() {
|
||||
n := memory.Allowed() / 1024 / 1024 / 8
|
||||
if n < 16 {
|
||||
n = 16
|
||||
}
|
||||
maxCachedIndexBlocksPerPart = n
|
||||
var ibCache = blockcache.NewCache(getMaxIndexBlocksCacheSize)
|
||||
|
||||
func getMaxIndexBlocksCacheSize() int {
|
||||
maxIndexBlockCacheSizeOnce.Do(func() {
|
||||
maxIndexBlockCacheSize = int(0.1 * float64(memory.Allowed()))
|
||||
})
|
||||
return maxCachedIndexBlocksPerPart
|
||||
return maxIndexBlockCacheSize
|
||||
}
|
||||
|
||||
var (
|
||||
maxCachedIndexBlocksPerPart int
|
||||
maxCachedIndexBlocksPerPartOnce sync.Once
|
||||
maxIndexBlockCacheSize int
|
||||
maxIndexBlockCacheSizeOnce sync.Once
|
||||
)
|
||||
|
||||
// part represents a searchable part containing time series data.
|
||||
|
@ -47,8 +43,6 @@ type part struct {
|
|||
indexFile fs.MustReadAtCloser
|
||||
|
||||
metaindex []metaindexRow
|
||||
|
||||
ibCache *indexBlockCache
|
||||
}
|
||||
|
||||
// openFilePart opens file-based part from the given path.
|
||||
|
@ -105,9 +99,7 @@ func newPart(ph *partHeader, path string, size uint64, metaindexReader filestrea
|
|||
p.timestampsFile = timestampsFile
|
||||
p.valuesFile = valuesFile
|
||||
p.indexFile = indexFile
|
||||
|
||||
p.metaindex = metaindex
|
||||
p.ibCache = newIndexBlockCache()
|
||||
|
||||
if len(errors) > 0 {
|
||||
// Return only the first error, since it has no sense in returning all errors.
|
||||
|
@ -133,8 +125,7 @@ func (p *part) MustClose() {
|
|||
p.valuesFile.MustClose()
|
||||
p.indexFile.MustClose()
|
||||
|
||||
isBig := p.size > maxSmallPartSize()
|
||||
p.ibCache.MustClose(isBig)
|
||||
ibCache.RemoveBlocksForPart(p)
|
||||
}
|
||||
|
||||
type indexBlock struct {
|
||||
|
@ -144,148 +135,3 @@ type indexBlock struct {
|
|||
func (idxb *indexBlock) SizeBytes() int {
|
||||
return cap(idxb.bhs) * int(unsafe.Sizeof(blockHeader{}))
|
||||
}
|
||||
|
||||
type indexBlockCache struct {
|
||||
// Put atomic counters to the top of struct in order to align them to 8 bytes on 32-bit architectures.
|
||||
// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
|
||||
requests uint64
|
||||
misses uint64
|
||||
|
||||
m map[uint64]*indexBlockCacheEntry
|
||||
mu sync.RWMutex
|
||||
|
||||
cleanerStopCh chan struct{}
|
||||
cleanerWG sync.WaitGroup
|
||||
}
|
||||
|
||||
type indexBlockCacheEntry struct {
|
||||
// Atomically updated counters must go first in the struct, so they are properly
|
||||
// aligned to 8 bytes on 32-bit architectures.
|
||||
// See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
|
||||
lastAccessTime uint64
|
||||
|
||||
ib *indexBlock
|
||||
}
|
||||
|
||||
func newIndexBlockCache() *indexBlockCache {
|
||||
var ibc indexBlockCache
|
||||
ibc.m = make(map[uint64]*indexBlockCacheEntry)
|
||||
|
||||
ibc.cleanerStopCh = make(chan struct{})
|
||||
ibc.cleanerWG.Add(1)
|
||||
go func() {
|
||||
defer ibc.cleanerWG.Done()
|
||||
ibc.cleaner()
|
||||
}()
|
||||
return &ibc
|
||||
}
|
||||
|
||||
func (ibc *indexBlockCache) MustClose(isBig bool) {
|
||||
close(ibc.cleanerStopCh)
|
||||
ibc.cleanerWG.Wait()
|
||||
ibc.m = nil
|
||||
}
|
||||
|
||||
// cleaner periodically cleans least recently used items.
|
||||
func (ibc *indexBlockCache) cleaner() {
|
||||
ticker := time.NewTicker(30 * time.Second)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ticker.C:
|
||||
ibc.cleanByTimeout()
|
||||
case <-ibc.cleanerStopCh:
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (ibc *indexBlockCache) cleanByTimeout() {
|
||||
currentTime := fasttime.UnixTimestamp()
|
||||
ibc.mu.Lock()
|
||||
for k, ibe := range ibc.m {
|
||||
// Delete items accessed more than two minutes ago.
|
||||
// This time should be enough for repeated queries.
|
||||
if currentTime-atomic.LoadUint64(&ibe.lastAccessTime) > 2*60 {
|
||||
delete(ibc.m, k)
|
||||
}
|
||||
}
|
||||
ibc.mu.Unlock()
|
||||
}
|
||||
|
||||
func (ibc *indexBlockCache) Get(k uint64) *indexBlock {
|
||||
atomic.AddUint64(&ibc.requests, 1)
|
||||
|
||||
ibc.mu.RLock()
|
||||
ibe := ibc.m[k]
|
||||
ibc.mu.RUnlock()
|
||||
|
||||
if ibe != nil {
|
||||
currentTime := fasttime.UnixTimestamp()
|
||||
if atomic.LoadUint64(&ibe.lastAccessTime) != currentTime {
|
||||
atomic.StoreUint64(&ibe.lastAccessTime, currentTime)
|
||||
}
|
||||
return ibe.ib
|
||||
}
|
||||
atomic.AddUint64(&ibc.misses, 1)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (ibc *indexBlockCache) Put(k uint64, ib *indexBlock) {
|
||||
ibc.mu.Lock()
|
||||
|
||||
// Clean superfluous cache entries.
|
||||
if overflow := len(ibc.m) - getMaxCachedIndexBlocksPerPart(); overflow > 0 {
|
||||
// Remove 10% of items from the cache.
|
||||
overflow = int(float64(len(ibc.m)) * 0.1)
|
||||
for k := range ibc.m {
|
||||
delete(ibc.m, k)
|
||||
overflow--
|
||||
if overflow == 0 {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Store frequently requested ib in the cache.
|
||||
ibe := &indexBlockCacheEntry{
|
||||
lastAccessTime: fasttime.UnixTimestamp(),
|
||||
ib: ib,
|
||||
}
|
||||
ibc.m[k] = ibe
|
||||
ibc.mu.Unlock()
|
||||
}
|
||||
|
||||
func (ibc *indexBlockCache) Requests() uint64 {
|
||||
return atomic.LoadUint64(&ibc.requests)
|
||||
}
|
||||
|
||||
func (ibc *indexBlockCache) Misses() uint64 {
|
||||
return atomic.LoadUint64(&ibc.misses)
|
||||
}
|
||||
|
||||
func (ibc *indexBlockCache) Len() uint64 {
|
||||
ibc.mu.Lock()
|
||||
n := uint64(len(ibc.m))
|
||||
ibc.mu.Unlock()
|
||||
return n
|
||||
}
|
||||
|
||||
func (ibc *indexBlockCache) SizeBytes() uint64 {
|
||||
n := 0
|
||||
ibc.mu.Lock()
|
||||
for _, e := range ibc.m {
|
||||
n += e.ib.SizeBytes()
|
||||
}
|
||||
ibc.mu.Unlock()
|
||||
return uint64(n)
|
||||
}
|
||||
|
||||
func (ibc *indexBlockCache) SizeMaxBytes() uint64 {
|
||||
avgBlockSize := float64(64 * 1024)
|
||||
blocksCount := ibc.Len()
|
||||
if blocksCount > 0 {
|
||||
avgBlockSize = float64(ibc.SizeBytes()) / float64(blocksCount)
|
||||
}
|
||||
return uint64(avgBlockSize * float64(getMaxCachedIndexBlocksPerPart()))
|
||||
}
|
||||
|
|
|
@ -7,6 +7,7 @@ import (
|
|||
"sort"
|
||||
"strings"
|
||||
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/blockcache"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/bytesutil"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/encoding"
|
||||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/logger"
|
||||
|
@ -32,8 +33,6 @@ type partSearch struct {
|
|||
|
||||
metaindex []metaindexRow
|
||||
|
||||
ibCache *indexBlockCache
|
||||
|
||||
bhs []blockHeader
|
||||
|
||||
compressedIndexBuf []byte
|
||||
|
@ -48,7 +47,6 @@ func (ps *partSearch) reset() {
|
|||
ps.tsids = nil
|
||||
ps.tsidIdx = 0
|
||||
ps.metaindex = nil
|
||||
ps.ibCache = nil
|
||||
ps.bhs = nil
|
||||
ps.compressedIndexBuf = ps.compressedIndexBuf[:0]
|
||||
ps.indexBuf = ps.indexBuf[:0]
|
||||
|
@ -76,7 +74,6 @@ func (ps *partSearch) Init(p *part, tsids []TSID, tr TimeRange) {
|
|||
}
|
||||
ps.tr = tr
|
||||
ps.metaindex = p.metaindex
|
||||
ps.ibCache = p.ibCache
|
||||
|
||||
// Advance to the first tsid. There is no need in checking
|
||||
// the returned result, since it will be checked in NextBlock.
|
||||
|
@ -154,19 +151,23 @@ func (ps *partSearch) nextBHS() bool {
|
|||
|
||||
// Found the index block which may contain the required data
|
||||
// for the ps.BlockRef.bh.TSID and the given timestamp range.
|
||||
indexBlockKey := mr.IndexBlockOffset
|
||||
ib := ps.ibCache.Get(indexBlockKey)
|
||||
if ib == nil {
|
||||
indexBlockKey := blockcache.Key{
|
||||
Part: ps.p,
|
||||
Offset: mr.IndexBlockOffset,
|
||||
}
|
||||
b := ibCache.GetBlock(indexBlockKey)
|
||||
if b == nil {
|
||||
// Slow path - actually read and unpack the index block.
|
||||
var err error
|
||||
ib, err = ps.readIndexBlock(mr)
|
||||
ib, err := ps.readIndexBlock(mr)
|
||||
if err != nil {
|
||||
ps.err = fmt.Errorf("cannot read index block for part %q at offset %d with size %d: %w",
|
||||
&ps.p.ph, mr.IndexBlockOffset, mr.IndexBlockSize, err)
|
||||
return false
|
||||
}
|
||||
ps.ibCache.Put(indexBlockKey, ib)
|
||||
b = ib
|
||||
ibCache.PutBlock(indexBlockKey, b)
|
||||
}
|
||||
ib := b.(*indexBlock)
|
||||
ps.bhs = ib.bhs
|
||||
return true
|
||||
}
|
||||
|
|
|
@ -25,16 +25,6 @@ import (
|
|||
"github.com/VictoriaMetrics/VictoriaMetrics/lib/uint64set"
|
||||
)
|
||||
|
||||
// These are global counters for cache requests and misses for parts
|
||||
// which were already merged into another parts.
|
||||
var (
|
||||
historicalBigIndexBlocksCacheRequests uint64
|
||||
historicalBigIndexBlocksCacheMisses uint64
|
||||
|
||||
historicalSmallIndexBlocksCacheRequests uint64
|
||||
historicalSmallIndexBlocksCacheMisses uint64
|
||||
)
|
||||
|
||||
func maxSmallPartSize() uint64 {
|
||||
// Small parts are cached in the OS page cache,
|
||||
// so limit their size by the remaining free RAM.
|
||||
|
@ -306,17 +296,11 @@ func newPartition(name, smallPartsPath, bigPartsPath string, getDeletedMetricIDs
|
|||
type partitionMetrics struct {
|
||||
PendingRows uint64
|
||||
|
||||
BigIndexBlocksCacheSize uint64
|
||||
BigIndexBlocksCacheSizeBytes uint64
|
||||
BigIndexBlocksCacheSizeMaxBytes uint64
|
||||
BigIndexBlocksCacheRequests uint64
|
||||
BigIndexBlocksCacheMisses uint64
|
||||
|
||||
SmallIndexBlocksCacheSize uint64
|
||||
SmallIndexBlocksCacheSizeBytes uint64
|
||||
SmallIndexBlocksCacheSizeMaxBytes uint64
|
||||
SmallIndexBlocksCacheRequests uint64
|
||||
SmallIndexBlocksCacheMisses uint64
|
||||
IndexBlocksCacheSize uint64
|
||||
IndexBlocksCacheSizeBytes uint64
|
||||
IndexBlocksCacheSizeMaxBytes uint64
|
||||
IndexBlocksCacheRequests uint64
|
||||
IndexBlocksCacheMisses uint64
|
||||
|
||||
BigSizeBytes uint64
|
||||
SmallSizeBytes uint64
|
||||
|
@ -362,11 +346,6 @@ func (pt *partition) UpdateMetrics(m *partitionMetrics) {
|
|||
for _, pw := range pt.bigParts {
|
||||
p := pw.p
|
||||
|
||||
m.BigIndexBlocksCacheSize += p.ibCache.Len()
|
||||
m.BigIndexBlocksCacheSizeBytes += p.ibCache.SizeBytes()
|
||||
m.BigIndexBlocksCacheSizeMaxBytes += p.ibCache.SizeMaxBytes()
|
||||
m.BigIndexBlocksCacheRequests += p.ibCache.Requests()
|
||||
m.BigIndexBlocksCacheMisses += p.ibCache.Misses()
|
||||
m.BigRowsCount += p.ph.RowsCount
|
||||
m.BigBlocksCount += p.ph.BlocksCount
|
||||
m.BigSizeBytes += p.size
|
||||
|
@ -376,11 +355,6 @@ func (pt *partition) UpdateMetrics(m *partitionMetrics) {
|
|||
for _, pw := range pt.smallParts {
|
||||
p := pw.p
|
||||
|
||||
m.SmallIndexBlocksCacheSize += p.ibCache.Len()
|
||||
m.SmallIndexBlocksCacheSizeBytes += p.ibCache.SizeBytes()
|
||||
m.SmallIndexBlocksCacheSizeMaxBytes += p.ibCache.SizeMaxBytes()
|
||||
m.SmallIndexBlocksCacheRequests += p.ibCache.Requests()
|
||||
m.SmallIndexBlocksCacheMisses += p.ibCache.Misses()
|
||||
m.SmallRowsCount += p.ph.RowsCount
|
||||
m.SmallBlocksCount += p.ph.BlocksCount
|
||||
m.SmallSizeBytes += p.size
|
||||
|
@ -392,11 +366,11 @@ func (pt *partition) UpdateMetrics(m *partitionMetrics) {
|
|||
|
||||
pt.partsLock.Unlock()
|
||||
|
||||
m.BigIndexBlocksCacheRequests = atomic.LoadUint64(&historicalBigIndexBlocksCacheRequests)
|
||||
m.BigIndexBlocksCacheMisses = atomic.LoadUint64(&historicalBigIndexBlocksCacheMisses)
|
||||
|
||||
m.SmallIndexBlocksCacheRequests = atomic.LoadUint64(&historicalSmallIndexBlocksCacheRequests)
|
||||
m.SmallIndexBlocksCacheMisses = atomic.LoadUint64(&historicalSmallIndexBlocksCacheMisses)
|
||||
m.IndexBlocksCacheSize = uint64(ibCache.Len())
|
||||
m.IndexBlocksCacheSizeBytes = uint64(ibCache.SizeBytes())
|
||||
m.IndexBlocksCacheSizeMaxBytes = uint64(ibCache.SizeMaxBytes())
|
||||
m.IndexBlocksCacheRequests = ibCache.Requests()
|
||||
m.IndexBlocksCacheMisses = ibCache.Misses()
|
||||
|
||||
m.ActiveBigMerges += atomic.LoadUint64(&pt.activeBigMerges)
|
||||
m.ActiveSmallMerges += atomic.LoadUint64(&pt.activeSmallMerges)
|
||||
|
@ -1311,15 +1285,6 @@ func removeParts(pws []*partWrapper, partsToRemove map[*partWrapper]bool, isBig
|
|||
dst = append(dst, pw)
|
||||
continue
|
||||
}
|
||||
requests := pw.p.ibCache.Requests()
|
||||
misses := pw.p.ibCache.Misses()
|
||||
if isBig {
|
||||
atomic.AddUint64(&historicalBigIndexBlocksCacheRequests, requests)
|
||||
atomic.AddUint64(&historicalBigIndexBlocksCacheMisses, misses)
|
||||
} else {
|
||||
atomic.AddUint64(&historicalSmallIndexBlocksCacheRequests, requests)
|
||||
atomic.AddUint64(&historicalSmallIndexBlocksCacheMisses, misses)
|
||||
}
|
||||
removedParts++
|
||||
}
|
||||
return dst, removedParts
|
||||
|
|
|
@ -1787,6 +1787,11 @@ func (s *Storage) add(rows []rawRow, dstMrs []*MetricRow, mrs []MetricRow, preci
|
|||
// Fast path - the current mr contains the same metric name as the previous mr, so it contains the same TSID.
|
||||
// This path should trigger on bulk imports when many rows contain the same MetricNameRaw.
|
||||
r.TSID = prevTSID
|
||||
if s.isSeriesCardinalityExceeded(r.TSID.MetricID, mr.MetricNameRaw) {
|
||||
// Skip the row, since the limit on the number of unique series has been exceeded.
|
||||
j--
|
||||
continue
|
||||
}
|
||||
continue
|
||||
}
|
||||
slowInsertsCount++
|
||||
|
@ -1800,14 +1805,14 @@ func (s *Storage) add(rows []rawRow, dstMrs []*MetricRow, mrs []MetricRow, preci
|
|||
j--
|
||||
continue
|
||||
}
|
||||
s.putTSIDToCache(&r.TSID, mr.MetricNameRaw)
|
||||
prevTSID = r.TSID
|
||||
prevMetricNameRaw = mr.MetricNameRaw
|
||||
if s.isSeriesCardinalityExceeded(r.TSID.MetricID, mr.MetricNameRaw) {
|
||||
// Skip the row, since the limit on the number of unique series has been exceeded.
|
||||
j--
|
||||
continue
|
||||
}
|
||||
s.putTSIDToCache(&r.TSID, mr.MetricNameRaw)
|
||||
prevTSID = r.TSID
|
||||
prevMetricNameRaw = mr.MetricNameRaw
|
||||
}
|
||||
idb.putIndexSearch(is)
|
||||
putPendingMetricRows(pmrs)
|
||||
|
|
7
vendor/cloud.google.com/go/iam/CHANGES.md
generated
vendored
7
vendor/cloud.google.com/go/iam/CHANGES.md
generated
vendored
|
@ -1,5 +1,12 @@
|
|||
# Changes
|
||||
|
||||
### [0.1.1](https://www.github.com/googleapis/google-cloud-go/compare/iam/v0.1.0...iam/v0.1.1) (2022-01-14)
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* **iam:** run formatter ([#5277](https://www.github.com/googleapis/google-cloud-go/issues/5277)) ([8682e4e](https://www.github.com/googleapis/google-cloud-go/commit/8682e4ed57a4428a659fbc225f56c91767e2a4a9))
|
||||
|
||||
## v0.1.0
|
||||
|
||||
This is the first tag to carve out iam as its own module. See
|
||||
|
|
1
vendor/cloud.google.com/go/iam/go_mod_tidy_hack.go
generated
vendored
1
vendor/cloud.google.com/go/iam/go_mod_tidy_hack.go
generated
vendored
|
@ -14,6 +14,7 @@
|
|||
|
||||
// This file, and the cloud.google.com/go import, won't actually become part of
|
||||
// the resultant binary.
|
||||
//go:build modhack
|
||||
// +build modhack
|
||||
|
||||
package iam
|
||||
|
|
2
vendor/github.com/aws/aws-sdk-go/aws/version.go
generated
vendored
2
vendor/github.com/aws/aws-sdk-go/aws/version.go
generated
vendored
|
@ -5,4 +5,4 @@ package aws
|
|||
const SDKName = "aws-sdk-go"
|
||||
|
||||
// SDKVersion is the version of this SDK
|
||||
const SDKVersion = "1.42.35"
|
||||
const SDKVersion = "1.42.39"
|
||||
|
|
17
vendor/github.com/google/go-cmp/cmp/compare.go
generated
vendored
17
vendor/github.com/google/go-cmp/cmp/compare.go
generated
vendored
|
@ -36,7 +36,6 @@ import (
|
|||
"strings"
|
||||
|
||||
"github.com/google/go-cmp/cmp/internal/diff"
|
||||
"github.com/google/go-cmp/cmp/internal/flags"
|
||||
"github.com/google/go-cmp/cmp/internal/function"
|
||||
"github.com/google/go-cmp/cmp/internal/value"
|
||||
)
|
||||
|
@ -319,7 +318,6 @@ func (s *state) tryMethod(t reflect.Type, vx, vy reflect.Value) bool {
|
|||
}
|
||||
|
||||
func (s *state) callTRFunc(f, v reflect.Value, step Transform) reflect.Value {
|
||||
v = sanitizeValue(v, f.Type().In(0))
|
||||
if !s.dynChecker.Next() {
|
||||
return f.Call([]reflect.Value{v})[0]
|
||||
}
|
||||
|
@ -343,8 +341,6 @@ func (s *state) callTRFunc(f, v reflect.Value, step Transform) reflect.Value {
|
|||
}
|
||||
|
||||
func (s *state) callTTBFunc(f, x, y reflect.Value) bool {
|
||||
x = sanitizeValue(x, f.Type().In(0))
|
||||
y = sanitizeValue(y, f.Type().In(1))
|
||||
if !s.dynChecker.Next() {
|
||||
return f.Call([]reflect.Value{x, y})[0].Bool()
|
||||
}
|
||||
|
@ -372,19 +368,6 @@ func detectRaces(c chan<- reflect.Value, f reflect.Value, vs ...reflect.Value) {
|
|||
ret = f.Call(vs)[0]
|
||||
}
|
||||
|
||||
// sanitizeValue converts nil interfaces of type T to those of type R,
|
||||
// assuming that T is assignable to R.
|
||||
// Otherwise, it returns the input value as is.
|
||||
func sanitizeValue(v reflect.Value, t reflect.Type) reflect.Value {
|
||||
// TODO(≥go1.10): Workaround for reflect bug (https://golang.org/issue/22143).
|
||||
if !flags.AtLeastGo110 {
|
||||
if v.Kind() == reflect.Interface && v.IsNil() && v.Type() != t {
|
||||
return reflect.New(t).Elem()
|
||||
}
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
func (s *state) compareStruct(t reflect.Type, vx, vy reflect.Value) {
|
||||
var addr bool
|
||||
var vax, vay reflect.Value // Addressable versions of vx and vy
|
||||
|
|
1
vendor/github.com/google/go-cmp/cmp/export_panic.go
generated
vendored
1
vendor/github.com/google/go-cmp/cmp/export_panic.go
generated
vendored
|
@ -2,6 +2,7 @@
|
|||
// Use of this source code is governed by a BSD-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
//go:build purego
|
||||
// +build purego
|
||||
|
||||
package cmp
|
||||
|
|
1
vendor/github.com/google/go-cmp/cmp/export_unsafe.go
generated
vendored
1
vendor/github.com/google/go-cmp/cmp/export_unsafe.go
generated
vendored
|
@ -2,6 +2,7 @@
|
|||
// Use of this source code is governed by a BSD-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
//go:build !purego
|
||||
// +build !purego
|
||||
|
||||
package cmp
|
||||
|
|
1
vendor/github.com/google/go-cmp/cmp/internal/diff/debug_disable.go
generated
vendored
1
vendor/github.com/google/go-cmp/cmp/internal/diff/debug_disable.go
generated
vendored
|
@ -2,6 +2,7 @@
|
|||
// Use of this source code is governed by a BSD-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
//go:build !cmp_debug
|
||||
// +build !cmp_debug
|
||||
|
||||
package diff
|
||||
|
|
1
vendor/github.com/google/go-cmp/cmp/internal/diff/debug_enable.go
generated
vendored
1
vendor/github.com/google/go-cmp/cmp/internal/diff/debug_enable.go
generated
vendored
|
@ -2,6 +2,7 @@
|
|||
// Use of this source code is governed by a BSD-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
//go:build cmp_debug
|
||||
// +build cmp_debug
|
||||
|
||||
package diff
|
||||
|
|
10
vendor/github.com/google/go-cmp/cmp/internal/flags/toolchain_legacy.go
generated
vendored
10
vendor/github.com/google/go-cmp/cmp/internal/flags/toolchain_legacy.go
generated
vendored
|
@ -1,10 +0,0 @@
|
|||
// Copyright 2019, The Go Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
// +build !go1.10
|
||||
|
||||
package flags
|
||||
|
||||
// AtLeastGo110 reports whether the Go toolchain is at least Go 1.10.
|
||||
const AtLeastGo110 = false
|
10
vendor/github.com/google/go-cmp/cmp/internal/flags/toolchain_recent.go
generated
vendored
10
vendor/github.com/google/go-cmp/cmp/internal/flags/toolchain_recent.go
generated
vendored
|
@ -1,10 +0,0 @@
|
|||
// Copyright 2019, The Go Authors. All rights reserved.
|
||||
// Use of this source code is governed by a BSD-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
// +build go1.10
|
||||
|
||||
package flags
|
||||
|
||||
// AtLeastGo110 reports whether the Go toolchain is at least Go 1.10.
|
||||
const AtLeastGo110 = true
|
7
vendor/github.com/google/go-cmp/cmp/internal/value/name.go
generated
vendored
7
vendor/github.com/google/go-cmp/cmp/internal/value/name.go
generated
vendored
|
@ -9,6 +9,8 @@ import (
|
|||
"strconv"
|
||||
)
|
||||
|
||||
var anyType = reflect.TypeOf((*interface{})(nil)).Elem()
|
||||
|
||||
// TypeString is nearly identical to reflect.Type.String,
|
||||
// but has an additional option to specify that full type names be used.
|
||||
func TypeString(t reflect.Type, qualified bool) string {
|
||||
|
@ -20,6 +22,11 @@ func appendTypeName(b []byte, t reflect.Type, qualified, elideFunc bool) []byte
|
|||
// of the same name and within the same package,
|
||||
// but declared within the namespace of different functions.
|
||||
|
||||
// Use the "any" alias instead of "interface{}" for better readability.
|
||||
if t == anyType {
|
||||
return append(b, "any"...)
|
||||
}
|
||||
|
||||
// Named type.
|
||||
if t.Name() != "" {
|
||||
if qualified && t.PkgPath() != "" {
|
||||
|
|
1
vendor/github.com/google/go-cmp/cmp/internal/value/pointer_purego.go
generated
vendored
1
vendor/github.com/google/go-cmp/cmp/internal/value/pointer_purego.go
generated
vendored
|
@ -2,6 +2,7 @@
|
|||
// Use of this source code is governed by a BSD-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
//go:build purego
|
||||
// +build purego
|
||||
|
||||
package value
|
||||
|
|
1
vendor/github.com/google/go-cmp/cmp/internal/value/pointer_unsafe.go
generated
vendored
1
vendor/github.com/google/go-cmp/cmp/internal/value/pointer_unsafe.go
generated
vendored
|
@ -2,6 +2,7 @@
|
|||
// Use of this source code is governed by a BSD-style
|
||||
// license that can be found in the LICENSE file.
|
||||
|
||||
//go:build !purego
|
||||
// +build !purego
|
||||
|
||||
package value
|
||||
|
|
2
vendor/github.com/google/go-cmp/cmp/path.go
generated
vendored
2
vendor/github.com/google/go-cmp/cmp/path.go
generated
vendored
|
@ -178,7 +178,7 @@ type structField struct {
|
|||
unexported bool
|
||||
mayForce bool // Forcibly allow visibility
|
||||
paddr bool // Was parent addressable?
|
||||
pvx, pvy reflect.Value // Parent values (always addressible)
|
||||
pvx, pvy reflect.Value // Parent values (always addressable)
|
||||
field reflect.StructField // Field information
|
||||
}
|
||||
|
||||
|
|
3
vendor/github.com/google/go-cmp/cmp/report_reflect.go
generated
vendored
3
vendor/github.com/google/go-cmp/cmp/report_reflect.go
generated
vendored
|
@ -207,9 +207,10 @@ func (opts formatOptions) FormatValue(v reflect.Value, parentKind reflect.Kind,
|
|||
// Check whether this is a []byte of text data.
|
||||
if t.Elem() == reflect.TypeOf(byte(0)) {
|
||||
b := v.Bytes()
|
||||
isPrintSpace := func(r rune) bool { return unicode.IsPrint(r) && unicode.IsSpace(r) }
|
||||
isPrintSpace := func(r rune) bool { return unicode.IsPrint(r) || unicode.IsSpace(r) }
|
||||
if len(b) > 0 && utf8.Valid(b) && len(bytes.TrimFunc(b, isPrintSpace)) == 0 {
|
||||
out = opts.formatString("", string(b))
|
||||
skipType = true
|
||||
return opts.WithTypeMode(emitType).FormatType(t, out)
|
||||
}
|
||||
}
|
||||
|
|
6
vendor/github.com/google/go-cmp/cmp/report_slices.go
generated
vendored
6
vendor/github.com/google/go-cmp/cmp/report_slices.go
generated
vendored
|
@ -80,7 +80,7 @@ func (opts formatOptions) CanFormatDiffSlice(v *valueNode) bool {
|
|||
}
|
||||
|
||||
// Use specialized string diffing for longer slices or strings.
|
||||
const minLength = 64
|
||||
const minLength = 32
|
||||
return vx.Len() >= minLength && vy.Len() >= minLength
|
||||
}
|
||||
|
||||
|
@ -563,10 +563,10 @@ func cleanupSurroundingIdentical(groups []diffStats, eq func(i, j int) bool) []d
|
|||
nx := ds.NumIdentical + ds.NumRemoved + ds.NumModified
|
||||
ny := ds.NumIdentical + ds.NumInserted + ds.NumModified
|
||||
var numLeadingIdentical, numTrailingIdentical int
|
||||
for i := 0; i < nx && i < ny && eq(ix+i, iy+i); i++ {
|
||||
for j := 0; j < nx && j < ny && eq(ix+j, iy+j); j++ {
|
||||
numLeadingIdentical++
|
||||
}
|
||||
for i := 0; i < nx && i < ny && eq(ix+nx-1-i, iy+ny-1-i); i++ {
|
||||
for j := 0; j < nx && j < ny && eq(ix+nx-1-j, iy+ny-1-j); j++ {
|
||||
numTrailingIdentical++
|
||||
}
|
||||
if numIdentical := numLeadingIdentical + numTrailingIdentical; numIdentical > 0 {
|
||||
|
|
2
vendor/github.com/prometheus/client_golang/prometheus/README.md
generated
vendored
2
vendor/github.com/prometheus/client_golang/prometheus/README.md
generated
vendored
|
@ -1 +1 @@
|
|||
See [](https://godoc.org/github.com/prometheus/client_golang/prometheus).
|
||||
See [](https://pkg.go.dev/github.com/prometheus/client_golang/prometheus).
|
||||
|
|
38
vendor/github.com/prometheus/client_golang/prometheus/build_info_collector.go
generated
vendored
Normal file
38
vendor/github.com/prometheus/client_golang/prometheus/build_info_collector.go
generated
vendored
Normal file
|
@ -0,0 +1,38 @@
|
|||
// Copyright 2021 The Prometheus Authors
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
package prometheus
|
||||
|
||||
import "runtime/debug"
|
||||
|
||||
// NewBuildInfoCollector is the obsolete version of collectors.NewBuildInfoCollector.
|
||||
// See there for documentation.
|
||||
//
|
||||
// Deprecated: Use collectors.NewBuildInfoCollector instead.
|
||||
func NewBuildInfoCollector() Collector {
|
||||
path, version, sum := "unknown", "unknown", "unknown"
|
||||
if bi, ok := debug.ReadBuildInfo(); ok {
|
||||
path = bi.Main.Path
|
||||
version = bi.Main.Version
|
||||
sum = bi.Main.Sum
|
||||
}
|
||||
c := &selfCollector{MustNewConstMetric(
|
||||
NewDesc(
|
||||
"go_build_info",
|
||||
"Build information about the main Go module.",
|
||||
nil, Labels{"path": path, "version": version, "checksum": sum},
|
||||
),
|
||||
GaugeValue, 1)}
|
||||
c.init(c.self)
|
||||
return c
|
||||
}
|
8
vendor/github.com/prometheus/client_golang/prometheus/counter.go
generated
vendored
8
vendor/github.com/prometheus/client_golang/prometheus/counter.go
generated
vendored
|
@ -133,10 +133,14 @@ func (c *counter) Inc() {
|
|||
atomic.AddUint64(&c.valInt, 1)
|
||||
}
|
||||
|
||||
func (c *counter) Write(out *dto.Metric) error {
|
||||
func (c *counter) get() float64 {
|
||||
fval := math.Float64frombits(atomic.LoadUint64(&c.valBits))
|
||||
ival := atomic.LoadUint64(&c.valInt)
|
||||
val := fval + float64(ival)
|
||||
return fval + float64(ival)
|
||||
}
|
||||
|
||||
func (c *counter) Write(out *dto.Metric) error {
|
||||
val := c.get()
|
||||
|
||||
var exemplar *dto.Exemplar
|
||||
if e := c.exemplar.Load(); e != nil {
|
||||
|
|
494
vendor/github.com/prometheus/client_golang/prometheus/go_collector.go
generated
vendored
494
vendor/github.com/prometheus/client_golang/prometheus/go_collector.go
generated
vendored
|
@ -16,32 +16,209 @@ package prometheus
|
|||
import (
|
||||
"runtime"
|
||||
"runtime/debug"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
type goCollector struct {
|
||||
func goRuntimeMemStats() memStatsMetrics {
|
||||
return memStatsMetrics{
|
||||
{
|
||||
desc: NewDesc(
|
||||
memstatNamespace("alloc_bytes"),
|
||||
"Number of bytes allocated and still in use.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.Alloc) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("alloc_bytes_total"),
|
||||
"Total number of bytes allocated, even if freed.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.TotalAlloc) },
|
||||
valType: CounterValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("sys_bytes"),
|
||||
"Number of bytes obtained from system.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.Sys) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("lookups_total"),
|
||||
"Total number of pointer lookups.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.Lookups) },
|
||||
valType: CounterValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("mallocs_total"),
|
||||
"Total number of mallocs.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.Mallocs) },
|
||||
valType: CounterValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("frees_total"),
|
||||
"Total number of frees.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.Frees) },
|
||||
valType: CounterValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("heap_alloc_bytes"),
|
||||
"Number of heap bytes allocated and still in use.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.HeapAlloc) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("heap_sys_bytes"),
|
||||
"Number of heap bytes obtained from system.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.HeapSys) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("heap_idle_bytes"),
|
||||
"Number of heap bytes waiting to be used.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.HeapIdle) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("heap_inuse_bytes"),
|
||||
"Number of heap bytes that are in use.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.HeapInuse) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("heap_released_bytes"),
|
||||
"Number of heap bytes released to OS.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.HeapReleased) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("heap_objects"),
|
||||
"Number of allocated objects.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.HeapObjects) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("stack_inuse_bytes"),
|
||||
"Number of bytes in use by the stack allocator.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.StackInuse) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("stack_sys_bytes"),
|
||||
"Number of bytes obtained from system for stack allocator.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.StackSys) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("mspan_inuse_bytes"),
|
||||
"Number of bytes in use by mspan structures.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.MSpanInuse) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("mspan_sys_bytes"),
|
||||
"Number of bytes used for mspan structures obtained from system.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.MSpanSys) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("mcache_inuse_bytes"),
|
||||
"Number of bytes in use by mcache structures.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.MCacheInuse) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("mcache_sys_bytes"),
|
||||
"Number of bytes used for mcache structures obtained from system.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.MCacheSys) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("buck_hash_sys_bytes"),
|
||||
"Number of bytes used by the profiling bucket hash table.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.BuckHashSys) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("gc_sys_bytes"),
|
||||
"Number of bytes used for garbage collection system metadata.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.GCSys) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("other_sys_bytes"),
|
||||
"Number of bytes used for other system allocations.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.OtherSys) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("next_gc_bytes"),
|
||||
"Number of heap bytes when next garbage collection will take place.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.NextGC) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("gc_cpu_fraction"),
|
||||
"The fraction of this program's available CPU time used by the GC since the program started.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return ms.GCCPUFraction },
|
||||
valType: GaugeValue,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
type baseGoCollector struct {
|
||||
goroutinesDesc *Desc
|
||||
threadsDesc *Desc
|
||||
gcDesc *Desc
|
||||
gcLastTimeDesc *Desc
|
||||
goInfoDesc *Desc
|
||||
|
||||
// ms... are memstats related.
|
||||
msLast *runtime.MemStats // Previously collected memstats.
|
||||
msLastTimestamp time.Time
|
||||
msMtx sync.Mutex // Protects msLast and msLastTimestamp.
|
||||
msMetrics memStatsMetrics
|
||||
msRead func(*runtime.MemStats) // For mocking in tests.
|
||||
msMaxWait time.Duration // Wait time for fresh memstats.
|
||||
msMaxAge time.Duration // Maximum allowed age of old memstats.
|
||||
}
|
||||
|
||||
// NewGoCollector is the obsolete version of collectors.NewGoCollector.
|
||||
// See there for documentation.
|
||||
//
|
||||
// Deprecated: Use collectors.NewGoCollector instead.
|
||||
func NewGoCollector() Collector {
|
||||
return &goCollector{
|
||||
func newBaseGoCollector() baseGoCollector {
|
||||
return baseGoCollector{
|
||||
goroutinesDesc: NewDesc(
|
||||
"go_goroutines",
|
||||
"Number of goroutines that currently exist.",
|
||||
|
@ -54,243 +231,28 @@ func NewGoCollector() Collector {
|
|||
"go_gc_duration_seconds",
|
||||
"A summary of the pause duration of garbage collection cycles.",
|
||||
nil, nil),
|
||||
gcLastTimeDesc: NewDesc(
|
||||
memstatNamespace("last_gc_time_seconds"),
|
||||
"Number of seconds since 1970 of last garbage collection.",
|
||||
nil, nil),
|
||||
goInfoDesc: NewDesc(
|
||||
"go_info",
|
||||
"Information about the Go environment.",
|
||||
nil, Labels{"version": runtime.Version()}),
|
||||
msLast: &runtime.MemStats{},
|
||||
msRead: runtime.ReadMemStats,
|
||||
msMaxWait: time.Second,
|
||||
msMaxAge: 5 * time.Minute,
|
||||
msMetrics: memStatsMetrics{
|
||||
{
|
||||
desc: NewDesc(
|
||||
memstatNamespace("alloc_bytes"),
|
||||
"Number of bytes allocated and still in use.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.Alloc) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("alloc_bytes_total"),
|
||||
"Total number of bytes allocated, even if freed.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.TotalAlloc) },
|
||||
valType: CounterValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("sys_bytes"),
|
||||
"Number of bytes obtained from system.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.Sys) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("lookups_total"),
|
||||
"Total number of pointer lookups.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.Lookups) },
|
||||
valType: CounterValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("mallocs_total"),
|
||||
"Total number of mallocs.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.Mallocs) },
|
||||
valType: CounterValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("frees_total"),
|
||||
"Total number of frees.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.Frees) },
|
||||
valType: CounterValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("heap_alloc_bytes"),
|
||||
"Number of heap bytes allocated and still in use.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.HeapAlloc) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("heap_sys_bytes"),
|
||||
"Number of heap bytes obtained from system.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.HeapSys) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("heap_idle_bytes"),
|
||||
"Number of heap bytes waiting to be used.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.HeapIdle) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("heap_inuse_bytes"),
|
||||
"Number of heap bytes that are in use.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.HeapInuse) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("heap_released_bytes"),
|
||||
"Number of heap bytes released to OS.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.HeapReleased) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("heap_objects"),
|
||||
"Number of allocated objects.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.HeapObjects) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("stack_inuse_bytes"),
|
||||
"Number of bytes in use by the stack allocator.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.StackInuse) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("stack_sys_bytes"),
|
||||
"Number of bytes obtained from system for stack allocator.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.StackSys) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("mspan_inuse_bytes"),
|
||||
"Number of bytes in use by mspan structures.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.MSpanInuse) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("mspan_sys_bytes"),
|
||||
"Number of bytes used for mspan structures obtained from system.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.MSpanSys) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("mcache_inuse_bytes"),
|
||||
"Number of bytes in use by mcache structures.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.MCacheInuse) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("mcache_sys_bytes"),
|
||||
"Number of bytes used for mcache structures obtained from system.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.MCacheSys) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("buck_hash_sys_bytes"),
|
||||
"Number of bytes used by the profiling bucket hash table.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.BuckHashSys) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("gc_sys_bytes"),
|
||||
"Number of bytes used for garbage collection system metadata.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.GCSys) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("other_sys_bytes"),
|
||||
"Number of bytes used for other system allocations.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.OtherSys) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("next_gc_bytes"),
|
||||
"Number of heap bytes when next garbage collection will take place.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.NextGC) },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("last_gc_time_seconds"),
|
||||
"Number of seconds since 1970 of last garbage collection.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return float64(ms.LastGC) / 1e9 },
|
||||
valType: GaugeValue,
|
||||
}, {
|
||||
desc: NewDesc(
|
||||
memstatNamespace("gc_cpu_fraction"),
|
||||
"The fraction of this program's available CPU time used by the GC since the program started.",
|
||||
nil, nil,
|
||||
),
|
||||
eval: func(ms *runtime.MemStats) float64 { return ms.GCCPUFraction },
|
||||
valType: GaugeValue,
|
||||
},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func memstatNamespace(s string) string {
|
||||
return "go_memstats_" + s
|
||||
}
|
||||
|
||||
// Describe returns all descriptions of the collector.
|
||||
func (c *goCollector) Describe(ch chan<- *Desc) {
|
||||
func (c *baseGoCollector) Describe(ch chan<- *Desc) {
|
||||
ch <- c.goroutinesDesc
|
||||
ch <- c.threadsDesc
|
||||
ch <- c.gcDesc
|
||||
ch <- c.gcLastTimeDesc
|
||||
ch <- c.goInfoDesc
|
||||
for _, i := range c.msMetrics {
|
||||
ch <- i.desc
|
||||
}
|
||||
}
|
||||
|
||||
// Collect returns the current state of all metrics of the collector.
|
||||
func (c *goCollector) Collect(ch chan<- Metric) {
|
||||
var (
|
||||
ms = &runtime.MemStats{}
|
||||
done = make(chan struct{})
|
||||
)
|
||||
// Start reading memstats first as it might take a while.
|
||||
go func() {
|
||||
c.msRead(ms)
|
||||
c.msMtx.Lock()
|
||||
c.msLast = ms
|
||||
c.msLastTimestamp = time.Now()
|
||||
c.msMtx.Unlock()
|
||||
close(done)
|
||||
}()
|
||||
|
||||
func (c *baseGoCollector) Collect(ch chan<- Metric) {
|
||||
ch <- MustNewConstMetric(c.goroutinesDesc, GaugeValue, float64(runtime.NumGoroutine()))
|
||||
n, _ := runtime.ThreadCreateProfile(nil)
|
||||
ch <- MustNewConstMetric(c.threadsDesc, GaugeValue, float64(n))
|
||||
|
@ -305,63 +267,19 @@ func (c *goCollector) Collect(ch chan<- Metric) {
|
|||
}
|
||||
quantiles[0.0] = stats.PauseQuantiles[0].Seconds()
|
||||
ch <- MustNewConstSummary(c.gcDesc, uint64(stats.NumGC), stats.PauseTotal.Seconds(), quantiles)
|
||||
ch <- MustNewConstMetric(c.gcLastTimeDesc, GaugeValue, float64(stats.LastGC.UnixNano())/1e9)
|
||||
|
||||
ch <- MustNewConstMetric(c.goInfoDesc, GaugeValue, 1)
|
||||
|
||||
timer := time.NewTimer(c.msMaxWait)
|
||||
select {
|
||||
case <-done: // Our own ReadMemStats succeeded in time. Use it.
|
||||
timer.Stop() // Important for high collection frequencies to not pile up timers.
|
||||
c.msCollect(ch, ms)
|
||||
return
|
||||
case <-timer.C: // Time out, use last memstats if possible. Continue below.
|
||||
}
|
||||
c.msMtx.Lock()
|
||||
if time.Since(c.msLastTimestamp) < c.msMaxAge {
|
||||
// Last memstats are recent enough. Collect from them under the lock.
|
||||
c.msCollect(ch, c.msLast)
|
||||
c.msMtx.Unlock()
|
||||
return
|
||||
}
|
||||
// If we are here, the last memstats are too old or don't exist. We have
|
||||
// to wait until our own ReadMemStats finally completes. For that to
|
||||
// happen, we have to release the lock.
|
||||
c.msMtx.Unlock()
|
||||
<-done
|
||||
c.msCollect(ch, ms)
|
||||
}
|
||||
|
||||
func (c *goCollector) msCollect(ch chan<- Metric, ms *runtime.MemStats) {
|
||||
for _, i := range c.msMetrics {
|
||||
ch <- MustNewConstMetric(i.desc, i.valType, i.eval(ms))
|
||||
}
|
||||
func memstatNamespace(s string) string {
|
||||
return "go_memstats_" + s
|
||||
}
|
||||
|
||||
// memStatsMetrics provide description, value, and value type for memstat metrics.
|
||||
// memStatsMetrics provide description, evaluator, runtime/metrics name, and
|
||||
// value type for memstat metrics.
|
||||
type memStatsMetrics []struct {
|
||||
desc *Desc
|
||||
eval func(*runtime.MemStats) float64
|
||||
valType ValueType
|
||||
}
|
||||
|
||||
// NewBuildInfoCollector is the obsolete version of collectors.NewBuildInfoCollector.
|
||||
// See there for documentation.
|
||||
//
|
||||
// Deprecated: Use collectors.NewBuildInfoCollector instead.
|
||||
func NewBuildInfoCollector() Collector {
|
||||
path, version, sum := "unknown", "unknown", "unknown"
|
||||
if bi, ok := debug.ReadBuildInfo(); ok {
|
||||
path = bi.Main.Path
|
||||
version = bi.Main.Version
|
||||
sum = bi.Main.Sum
|
||||
}
|
||||
c := &selfCollector{MustNewConstMetric(
|
||||
NewDesc(
|
||||
"go_build_info",
|
||||
"Build information about the main Go module.",
|
||||
nil, Labels{"path": path, "version": version, "checksum": sum},
|
||||
),
|
||||
GaugeValue, 1)}
|
||||
c.init(c.self)
|
||||
return c
|
||||
}
|
||||
|
|
107
vendor/github.com/prometheus/client_golang/prometheus/go_collector_go116.go
generated
vendored
Normal file
107
vendor/github.com/prometheus/client_golang/prometheus/go_collector_go116.go
generated
vendored
Normal file
|
@ -0,0 +1,107 @@
|
|||
// Copyright 2021 The Prometheus Authors
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//go:build !go1.17
|
||||
// +build !go1.17
|
||||
|
||||
package prometheus
|
||||
|
||||
import (
|
||||
"runtime"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
type goCollector struct {
|
||||
base baseGoCollector
|
||||
|
||||
// ms... are memstats related.
|
||||
msLast *runtime.MemStats // Previously collected memstats.
|
||||
msLastTimestamp time.Time
|
||||
msMtx sync.Mutex // Protects msLast and msLastTimestamp.
|
||||
msMetrics memStatsMetrics
|
||||
msRead func(*runtime.MemStats) // For mocking in tests.
|
||||
msMaxWait time.Duration // Wait time for fresh memstats.
|
||||
msMaxAge time.Duration // Maximum allowed age of old memstats.
|
||||
}
|
||||
|
||||
// NewGoCollector is the obsolete version of collectors.NewGoCollector.
|
||||
// See there for documentation.
|
||||
//
|
||||
// Deprecated: Use collectors.NewGoCollector instead.
|
||||
func NewGoCollector() Collector {
|
||||
return &goCollector{
|
||||
base: newBaseGoCollector(),
|
||||
msLast: &runtime.MemStats{},
|
||||
msRead: runtime.ReadMemStats,
|
||||
msMaxWait: time.Second,
|
||||
msMaxAge: 5 * time.Minute,
|
||||
msMetrics: goRuntimeMemStats(),
|
||||
}
|
||||
}
|
||||
|
||||
// Describe returns all descriptions of the collector.
|
||||
func (c *goCollector) Describe(ch chan<- *Desc) {
|
||||
c.base.Describe(ch)
|
||||
for _, i := range c.msMetrics {
|
||||
ch <- i.desc
|
||||
}
|
||||
}
|
||||
|
||||
// Collect returns the current state of all metrics of the collector.
|
||||
func (c *goCollector) Collect(ch chan<- Metric) {
|
||||
var (
|
||||
ms = &runtime.MemStats{}
|
||||
done = make(chan struct{})
|
||||
)
|
||||
// Start reading memstats first as it might take a while.
|
||||
go func() {
|
||||
c.msRead(ms)
|
||||
c.msMtx.Lock()
|
||||
c.msLast = ms
|
||||
c.msLastTimestamp = time.Now()
|
||||
c.msMtx.Unlock()
|
||||
close(done)
|
||||
}()
|
||||
|
||||
// Collect base non-memory metrics.
|
||||
c.base.Collect(ch)
|
||||
|
||||
timer := time.NewTimer(c.msMaxWait)
|
||||
select {
|
||||
case <-done: // Our own ReadMemStats succeeded in time. Use it.
|
||||
timer.Stop() // Important for high collection frequencies to not pile up timers.
|
||||
c.msCollect(ch, ms)
|
||||
return
|
||||
case <-timer.C: // Time out, use last memstats if possible. Continue below.
|
||||
}
|
||||
c.msMtx.Lock()
|
||||
if time.Since(c.msLastTimestamp) < c.msMaxAge {
|
||||
// Last memstats are recent enough. Collect from them under the lock.
|
||||
c.msCollect(ch, c.msLast)
|
||||
c.msMtx.Unlock()
|
||||
return
|
||||
}
|
||||
// If we are here, the last memstats are too old or don't exist. We have
|
||||
// to wait until our own ReadMemStats finally completes. For that to
|
||||
// happen, we have to release the lock.
|
||||
c.msMtx.Unlock()
|
||||
<-done
|
||||
c.msCollect(ch, ms)
|
||||
}
|
||||
|
||||
func (c *goCollector) msCollect(ch chan<- Metric, ms *runtime.MemStats) {
|
||||
for _, i := range c.msMetrics {
|
||||
ch <- MustNewConstMetric(i.desc, i.valType, i.eval(ms))
|
||||
}
|
||||
}
|
364
vendor/github.com/prometheus/client_golang/prometheus/go_collector_go117.go
generated
vendored
Normal file
364
vendor/github.com/prometheus/client_golang/prometheus/go_collector_go117.go
generated
vendored
Normal file
|
@ -0,0 +1,364 @@
|
|||
// Copyright 2021 The Prometheus Authors
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//go:build go1.17
|
||||
// +build go1.17
|
||||
|
||||
package prometheus
|
||||
|
||||
import (
|
||||
"math"
|
||||
"runtime"
|
||||
"runtime/metrics"
|
||||
"sync"
|
||||
|
||||
//nolint:staticcheck // Ignore SA1019. Need to keep deprecated package for compatibility.
|
||||
"github.com/golang/protobuf/proto"
|
||||
"github.com/prometheus/client_golang/prometheus/internal"
|
||||
dto "github.com/prometheus/client_model/go"
|
||||
)
|
||||
|
||||
type goCollector struct {
|
||||
base baseGoCollector
|
||||
|
||||
// rm... fields all pertain to the runtime/metrics package.
|
||||
rmSampleBuf []metrics.Sample
|
||||
rmSampleMap map[string]*metrics.Sample
|
||||
rmMetrics []Metric
|
||||
|
||||
// With Go 1.17, the runtime/metrics package was introduced.
|
||||
// From that point on, metric names produced by the runtime/metrics
|
||||
// package could be generated from runtime/metrics names. However,
|
||||
// these differ from the old names for the same values.
|
||||
//
|
||||
// This field exist to export the same values under the old names
|
||||
// as well.
|
||||
msMetrics memStatsMetrics
|
||||
}
|
||||
|
||||
// NewGoCollector is the obsolete version of collectors.NewGoCollector.
|
||||
// See there for documentation.
|
||||
//
|
||||
// Deprecated: Use collectors.NewGoCollector instead.
|
||||
func NewGoCollector() Collector {
|
||||
descriptions := metrics.All()
|
||||
descMap := make(map[string]*metrics.Description)
|
||||
for i := range descriptions {
|
||||
descMap[descriptions[i].Name] = &descriptions[i]
|
||||
}
|
||||
|
||||
// Generate a Desc and ValueType for each runtime/metrics metric.
|
||||
metricSet := make([]Metric, 0, len(descriptions))
|
||||
sampleBuf := make([]metrics.Sample, 0, len(descriptions))
|
||||
sampleMap := make(map[string]*metrics.Sample, len(descriptions))
|
||||
for i := range descriptions {
|
||||
d := &descriptions[i]
|
||||
namespace, subsystem, name, ok := internal.RuntimeMetricsToProm(d)
|
||||
if !ok {
|
||||
// Just ignore this metric; we can't do anything with it here.
|
||||
// If a user decides to use the latest version of Go, we don't want
|
||||
// to fail here. This condition is tested elsewhere.
|
||||
continue
|
||||
}
|
||||
|
||||
// Set up sample buffer for reading, and a map
|
||||
// for quick lookup of sample values.
|
||||
sampleBuf = append(sampleBuf, metrics.Sample{Name: d.Name})
|
||||
sampleMap[d.Name] = &sampleBuf[len(sampleBuf)-1]
|
||||
|
||||
var m Metric
|
||||
if d.Kind == metrics.KindFloat64Histogram {
|
||||
_, hasSum := rmExactSumMap[d.Name]
|
||||
m = newBatchHistogram(
|
||||
NewDesc(
|
||||
BuildFQName(namespace, subsystem, name),
|
||||
d.Description,
|
||||
nil,
|
||||
nil,
|
||||
),
|
||||
hasSum,
|
||||
)
|
||||
} else if d.Cumulative {
|
||||
m = NewCounter(CounterOpts{
|
||||
Namespace: namespace,
|
||||
Subsystem: subsystem,
|
||||
Name: name,
|
||||
Help: d.Description,
|
||||
})
|
||||
} else {
|
||||
m = NewGauge(GaugeOpts{
|
||||
Namespace: namespace,
|
||||
Subsystem: subsystem,
|
||||
Name: name,
|
||||
Help: d.Description,
|
||||
})
|
||||
}
|
||||
metricSet = append(metricSet, m)
|
||||
}
|
||||
return &goCollector{
|
||||
base: newBaseGoCollector(),
|
||||
rmSampleBuf: sampleBuf,
|
||||
rmSampleMap: sampleMap,
|
||||
rmMetrics: metricSet,
|
||||
msMetrics: goRuntimeMemStats(),
|
||||
}
|
||||
}
|
||||
|
||||
// Describe returns all descriptions of the collector.
|
||||
func (c *goCollector) Describe(ch chan<- *Desc) {
|
||||
c.base.Describe(ch)
|
||||
for _, i := range c.msMetrics {
|
||||
ch <- i.desc
|
||||
}
|
||||
for _, m := range c.rmMetrics {
|
||||
ch <- m.Desc()
|
||||
}
|
||||
}
|
||||
|
||||
// Collect returns the current state of all metrics of the collector.
|
||||
func (c *goCollector) Collect(ch chan<- Metric) {
|
||||
// Collect base non-memory metrics.
|
||||
c.base.Collect(ch)
|
||||
|
||||
// Populate runtime/metrics sample buffer.
|
||||
metrics.Read(c.rmSampleBuf)
|
||||
|
||||
for i, sample := range c.rmSampleBuf {
|
||||
// N.B. switch on concrete type because it's significantly more efficient
|
||||
// than checking for the Counter and Gauge interface implementations. In
|
||||
// this case, we control all the types here.
|
||||
switch m := c.rmMetrics[i].(type) {
|
||||
case *counter:
|
||||
// Guard against decreases. This should never happen, but a failure
|
||||
// to do so will result in a panic, which is a harsh consequence for
|
||||
// a metrics collection bug.
|
||||
v0, v1 := m.get(), unwrapScalarRMValue(sample.Value)
|
||||
if v1 > v0 {
|
||||
m.Add(unwrapScalarRMValue(sample.Value) - m.get())
|
||||
}
|
||||
m.Collect(ch)
|
||||
case *gauge:
|
||||
m.Set(unwrapScalarRMValue(sample.Value))
|
||||
m.Collect(ch)
|
||||
case *batchHistogram:
|
||||
m.update(sample.Value.Float64Histogram(), c.exactSumFor(sample.Name))
|
||||
m.Collect(ch)
|
||||
default:
|
||||
panic("unexpected metric type")
|
||||
}
|
||||
}
|
||||
|
||||
// ms is a dummy MemStats that we populate ourselves so that we can
|
||||
// populate the old metrics from it.
|
||||
var ms runtime.MemStats
|
||||
memStatsFromRM(&ms, c.rmSampleMap)
|
||||
for _, i := range c.msMetrics {
|
||||
ch <- MustNewConstMetric(i.desc, i.valType, i.eval(&ms))
|
||||
}
|
||||
}
|
||||
|
||||
// unwrapScalarRMValue unwraps a runtime/metrics value that is assumed
|
||||
// to be scalar and returns the equivalent float64 value. Panics if the
|
||||
// value is not scalar.
|
||||
func unwrapScalarRMValue(v metrics.Value) float64 {
|
||||
switch v.Kind() {
|
||||
case metrics.KindUint64:
|
||||
return float64(v.Uint64())
|
||||
case metrics.KindFloat64:
|
||||
return v.Float64()
|
||||
case metrics.KindBad:
|
||||
// Unsupported metric.
|
||||
//
|
||||
// This should never happen because we always populate our metric
|
||||
// set from the runtime/metrics package.
|
||||
panic("unexpected unsupported metric")
|
||||
default:
|
||||
// Unsupported metric kind.
|
||||
//
|
||||
// This should never happen because we check for this during initialization
|
||||
// and flag and filter metrics whose kinds we don't understand.
|
||||
panic("unexpected unsupported metric kind")
|
||||
}
|
||||
}
|
||||
|
||||
var rmExactSumMap = map[string]string{
|
||||
"/gc/heap/allocs-by-size:bytes": "/gc/heap/allocs:bytes",
|
||||
"/gc/heap/frees-by-size:bytes": "/gc/heap/frees:bytes",
|
||||
}
|
||||
|
||||
// exactSumFor takes a runtime/metrics metric name (that is assumed to
|
||||
// be of kind KindFloat64Histogram) and returns its exact sum and whether
|
||||
// its exact sum exists.
|
||||
//
|
||||
// The runtime/metrics API for histograms doesn't currently expose exact
|
||||
// sums, but some of the other metrics are in fact exact sums of histograms.
|
||||
func (c *goCollector) exactSumFor(rmName string) float64 {
|
||||
sumName, ok := rmExactSumMap[rmName]
|
||||
if !ok {
|
||||
return 0
|
||||
}
|
||||
s, ok := c.rmSampleMap[sumName]
|
||||
if !ok {
|
||||
return 0
|
||||
}
|
||||
return unwrapScalarRMValue(s.Value)
|
||||
}
|
||||
|
||||
func memStatsFromRM(ms *runtime.MemStats, rm map[string]*metrics.Sample) {
|
||||
lookupOrZero := func(name string) uint64 {
|
||||
if s, ok := rm[name]; ok {
|
||||
return s.Value.Uint64()
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// Currently, MemStats adds tiny alloc count to both Mallocs AND Frees.
|
||||
// The reason for this is because MemStats couldn't be extended at the time
|
||||
// but there was a desire to have Mallocs at least be a little more representative,
|
||||
// while having Mallocs - Frees still represent a live object count.
|
||||
// Unfortunately, MemStats doesn't actually export a large allocation count,
|
||||
// so it's impossible to pull this number out directly.
|
||||
tinyAllocs := lookupOrZero("/gc/heap/tiny/allocs:objects")
|
||||
ms.Mallocs = lookupOrZero("/gc/heap/allocs:objects") + tinyAllocs
|
||||
ms.Frees = lookupOrZero("/gc/heap/frees:objects") + tinyAllocs
|
||||
|
||||
ms.TotalAlloc = lookupOrZero("/gc/heap/allocs:bytes")
|
||||
ms.Sys = lookupOrZero("/memory/classes/total:bytes")
|
||||
ms.Lookups = 0 // Already always zero.
|
||||
ms.HeapAlloc = lookupOrZero("/memory/classes/heap/objects:bytes")
|
||||
ms.Alloc = ms.HeapAlloc
|
||||
ms.HeapInuse = ms.HeapAlloc + lookupOrZero("/memory/classes/heap/unused:bytes")
|
||||
ms.HeapReleased = lookupOrZero("/memory/classes/heap/released:bytes")
|
||||
ms.HeapIdle = ms.HeapReleased + lookupOrZero("/memory/classes/heap/free:bytes")
|
||||
ms.HeapSys = ms.HeapInuse + ms.HeapIdle
|
||||
ms.HeapObjects = lookupOrZero("/gc/heap/objects:objects")
|
||||
ms.StackInuse = lookupOrZero("/memory/classes/heap/stacks:bytes")
|
||||
ms.StackSys = ms.StackInuse + lookupOrZero("/memory/classes/os-stacks:bytes")
|
||||
ms.MSpanInuse = lookupOrZero("/memory/classes/metadata/mspan/inuse:bytes")
|
||||
ms.MSpanSys = ms.MSpanInuse + lookupOrZero("/memory/classes/metadata/mspan/free:bytes")
|
||||
ms.MCacheInuse = lookupOrZero("/memory/classes/metadata/mcache/inuse:bytes")
|
||||
ms.MCacheSys = ms.MCacheInuse + lookupOrZero("/memory/classes/metadata/mcache/free:bytes")
|
||||
ms.BuckHashSys = lookupOrZero("/memory/classes/profiling/buckets:bytes")
|
||||
ms.GCSys = lookupOrZero("/memory/classes/metadata/other:bytes")
|
||||
ms.OtherSys = lookupOrZero("/memory/classes/other:bytes")
|
||||
ms.NextGC = lookupOrZero("/gc/heap/goal:bytes")
|
||||
|
||||
// N.B. LastGC is omitted because runtime.GCStats already has this.
|
||||
// See https://github.com/prometheus/client_golang/issues/842#issuecomment-861812034
|
||||
// for more details.
|
||||
ms.LastGC = 0
|
||||
|
||||
// N.B. GCCPUFraction is intentionally omitted. This metric is not useful,
|
||||
// and often misleading due to the fact that it's an average over the lifetime
|
||||
// of the process.
|
||||
// See https://github.com/prometheus/client_golang/issues/842#issuecomment-861812034
|
||||
// for more details.
|
||||
ms.GCCPUFraction = 0
|
||||
}
|
||||
|
||||
// batchHistogram is a mutable histogram that is updated
|
||||
// in batches.
|
||||
type batchHistogram struct {
|
||||
selfCollector
|
||||
|
||||
// Static fields updated only once.
|
||||
desc *Desc
|
||||
hasSum bool
|
||||
|
||||
// Because this histogram operates in batches, it just uses a
|
||||
// single mutex for everything. updates are always serialized
|
||||
// but Write calls may operate concurrently with updates.
|
||||
// Contention between these two sources should be rare.
|
||||
mu sync.Mutex
|
||||
buckets []float64 // Inclusive lower bounds.
|
||||
counts []uint64
|
||||
sum float64 // Used if hasSum is true.
|
||||
}
|
||||
|
||||
func newBatchHistogram(desc *Desc, hasSum bool) *batchHistogram {
|
||||
h := &batchHistogram{desc: desc, hasSum: hasSum}
|
||||
h.init(h)
|
||||
return h
|
||||
}
|
||||
|
||||
// update updates the batchHistogram from a runtime/metrics histogram.
|
||||
//
|
||||
// sum must be provided if the batchHistogram was created to have an exact sum.
|
||||
func (h *batchHistogram) update(his *metrics.Float64Histogram, sum float64) {
|
||||
counts, buckets := his.Counts, his.Buckets
|
||||
// Skip a -Inf bucket altogether. It's not clear how to represent that.
|
||||
if math.IsInf(buckets[0], -1) {
|
||||
buckets = buckets[1:]
|
||||
counts = counts[1:]
|
||||
}
|
||||
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
|
||||
// Check if we're initialized.
|
||||
if h.buckets == nil {
|
||||
// Make copies of counts and buckets. It's really important
|
||||
// that we don't retain his.Counts or his.Buckets anywhere since
|
||||
// it's going to get reused.
|
||||
h.buckets = make([]float64, len(buckets))
|
||||
copy(h.buckets, buckets)
|
||||
|
||||
h.counts = make([]uint64, len(counts))
|
||||
}
|
||||
copy(h.counts, counts)
|
||||
if h.hasSum {
|
||||
h.sum = sum
|
||||
}
|
||||
}
|
||||
|
||||
func (h *batchHistogram) Desc() *Desc {
|
||||
return h.desc
|
||||
}
|
||||
|
||||
func (h *batchHistogram) Write(out *dto.Metric) error {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
|
||||
sum := float64(0)
|
||||
if h.hasSum {
|
||||
sum = h.sum
|
||||
}
|
||||
dtoBuckets := make([]*dto.Bucket, 0, len(h.counts))
|
||||
totalCount := uint64(0)
|
||||
for i, count := range h.counts {
|
||||
totalCount += count
|
||||
if !h.hasSum {
|
||||
// N.B. This computed sum is an underestimate.
|
||||
sum += h.buckets[i] * float64(count)
|
||||
}
|
||||
|
||||
// Skip the +Inf bucket, but only for the bucket list.
|
||||
// It must still count for sum and totalCount.
|
||||
if math.IsInf(h.buckets[i+1], 1) {
|
||||
break
|
||||
}
|
||||
// Float64Histogram's upper bound is exclusive, so make it inclusive
|
||||
// by obtaining the next float64 value down, in order.
|
||||
upperBound := math.Nextafter(h.buckets[i+1], h.buckets[i])
|
||||
dtoBuckets = append(dtoBuckets, &dto.Bucket{
|
||||
CumulativeCount: proto.Uint64(totalCount),
|
||||
UpperBound: proto.Float64(upperBound),
|
||||
})
|
||||
}
|
||||
out.Histogram = &dto.Histogram{
|
||||
Bucket: dtoBuckets,
|
||||
SampleCount: proto.Uint64(totalCount),
|
||||
SampleSum: proto.Float64(sum),
|
||||
}
|
||||
return nil
|
||||
}
|
28
vendor/github.com/prometheus/client_golang/prometheus/histogram.go
generated
vendored
28
vendor/github.com/prometheus/client_golang/prometheus/histogram.go
generated
vendored
|
@ -116,6 +116,34 @@ func ExponentialBuckets(start, factor float64, count int) []float64 {
|
|||
return buckets
|
||||
}
|
||||
|
||||
// ExponentialBucketsRange creates 'count' buckets, where the lowest bucket is
|
||||
// 'min' and the highest bucket is 'max'. The final +Inf bucket is not counted
|
||||
// and not included in the returned slice. The returned slice is meant to be
|
||||
// used for the Buckets field of HistogramOpts.
|
||||
//
|
||||
// The function panics if 'count' is 0 or negative, if 'min' is 0 or negative.
|
||||
func ExponentialBucketsRange(min, max float64, count int) []float64 {
|
||||
if count < 1 {
|
||||
panic("ExponentialBucketsRange count needs a positive count")
|
||||
}
|
||||
if min <= 0 {
|
||||
panic("ExponentialBucketsRange min needs to be greater than 0")
|
||||
}
|
||||
|
||||
// Formula for exponential buckets.
|
||||
// max = min*growthFactor^(bucketCount-1)
|
||||
|
||||
// We know max/min and highest bucket. Solve for growthFactor.
|
||||
growthFactor := math.Pow(max/min, 1.0/float64(count-1))
|
||||
|
||||
// Now that we know growthFactor, solve for each bucket.
|
||||
buckets := make([]float64, count)
|
||||
for i := 1; i <= count; i++ {
|
||||
buckets[i-1] = min * math.Pow(growthFactor, float64(i-1))
|
||||
}
|
||||
return buckets
|
||||
}
|
||||
|
||||
// HistogramOpts bundles the options for creating a Histogram metric. It is
|
||||
// mandatory to set Name to a non-empty string. All other fields are optional
|
||||
// and can safely be left at their zero value, although it is strongly
|
||||
|
|
77
vendor/github.com/prometheus/client_golang/prometheus/internal/go_runtime_metrics.go
generated
vendored
Normal file
77
vendor/github.com/prometheus/client_golang/prometheus/internal/go_runtime_metrics.go
generated
vendored
Normal file
|
@ -0,0 +1,77 @@
|
|||
// Copyright 2021 The Prometheus Authors
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//go:build go1.17
|
||||
// +build go1.17
|
||||
|
||||
package internal
|
||||
|
||||
import (
|
||||
"path"
|
||||
"runtime/metrics"
|
||||
"strings"
|
||||
|
||||
"github.com/prometheus/common/model"
|
||||
)
|
||||
|
||||
// RuntimeMetricsToProm produces a Prometheus metric name from a runtime/metrics
|
||||
// metric description and validates whether the metric is suitable for integration
|
||||
// with Prometheus.
|
||||
//
|
||||
// Returns false if a name could not be produced, or if Prometheus does not understand
|
||||
// the runtime/metrics Kind.
|
||||
//
|
||||
// Note that the main reason a name couldn't be produced is if the runtime/metrics
|
||||
// package exports a name with characters outside the valid Prometheus metric name
|
||||
// character set. This is theoretically possible, but should never happen in practice.
|
||||
// Still, don't rely on it.
|
||||
func RuntimeMetricsToProm(d *metrics.Description) (string, string, string, bool) {
|
||||
namespace := "go"
|
||||
|
||||
comp := strings.SplitN(d.Name, ":", 2)
|
||||
key := comp[0]
|
||||
unit := comp[1]
|
||||
|
||||
// The last path element in the key is the name,
|
||||
// the rest is the subsystem.
|
||||
subsystem := path.Dir(key[1:] /* remove leading / */)
|
||||
name := path.Base(key)
|
||||
|
||||
// subsystem is translated by replacing all / and - with _.
|
||||
subsystem = strings.ReplaceAll(subsystem, "/", "_")
|
||||
subsystem = strings.ReplaceAll(subsystem, "-", "_")
|
||||
|
||||
// unit is translated assuming that the unit contains no
|
||||
// non-ASCII characters.
|
||||
unit = strings.ReplaceAll(unit, "-", "_")
|
||||
unit = strings.ReplaceAll(unit, "*", "_")
|
||||
unit = strings.ReplaceAll(unit, "/", "_per_")
|
||||
|
||||
// name has - replaced with _ and is concatenated with the unit and
|
||||
// other data.
|
||||
name = strings.ReplaceAll(name, "-", "_")
|
||||
name = name + "_" + unit
|
||||
if d.Cumulative {
|
||||
name = name + "_total"
|
||||
}
|
||||
|
||||
valid := model.IsValidMetricName(model.LabelValue(namespace + "_" + subsystem + "_" + name))
|
||||
switch d.Kind {
|
||||
case metrics.KindUint64:
|
||||
case metrics.KindFloat64:
|
||||
case metrics.KindFloat64Histogram:
|
||||
default:
|
||||
valid = false
|
||||
}
|
||||
return namespace, subsystem, name, valid
|
||||
}
|
1
vendor/github.com/prometheus/client_golang/prometheus/process_collector_other.go
generated
vendored
1
vendor/github.com/prometheus/client_golang/prometheus/process_collector_other.go
generated
vendored
|
@ -11,6 +11,7 @@
|
|||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//go:build !windows
|
||||
// +build !windows
|
||||
|
||||
package prometheus
|
||||
|
|
6
vendor/github.com/prometheus/client_golang/prometheus/value.go
generated
vendored
6
vendor/github.com/prometheus/client_golang/prometheus/value.go
generated
vendored
|
@ -21,7 +21,7 @@ import (
|
|||
|
||||
//nolint:staticcheck // Ignore SA1019. Need to keep deprecated package for compatibility.
|
||||
"github.com/golang/protobuf/proto"
|
||||
"github.com/golang/protobuf/ptypes"
|
||||
"google.golang.org/protobuf/types/known/timestamppb"
|
||||
|
||||
dto "github.com/prometheus/client_model/go"
|
||||
)
|
||||
|
@ -183,8 +183,8 @@ const ExemplarMaxRunes = 64
|
|||
func newExemplar(value float64, ts time.Time, l Labels) (*dto.Exemplar, error) {
|
||||
e := &dto.Exemplar{}
|
||||
e.Value = proto.Float64(value)
|
||||
tsProto, err := ptypes.TimestampProto(ts)
|
||||
if err != nil {
|
||||
tsProto := timestamppb.New(ts)
|
||||
if err := tsProto.CheckValid(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
e.Timestamp = tsProto
|
||||
|
|
16
vendor/modules.txt
vendored
16
vendor/modules.txt
vendored
|
@ -5,10 +5,10 @@ cloud.google.com/go/internal
|
|||
cloud.google.com/go/internal/optional
|
||||
cloud.google.com/go/internal/trace
|
||||
cloud.google.com/go/internal/version
|
||||
# cloud.google.com/go/compute v1.0.0
|
||||
# cloud.google.com/go/compute v1.1.0
|
||||
## explicit; go 1.11
|
||||
cloud.google.com/go/compute/metadata
|
||||
# cloud.google.com/go/iam v0.1.0
|
||||
# cloud.google.com/go/iam v0.1.1
|
||||
## explicit; go 1.11
|
||||
cloud.google.com/go/iam
|
||||
# cloud.google.com/go/storage v1.18.2
|
||||
|
@ -33,7 +33,7 @@ github.com/VictoriaMetrics/metricsql/binaryop
|
|||
# github.com/VividCortex/ewma v1.2.0
|
||||
## explicit; go 1.12
|
||||
github.com/VividCortex/ewma
|
||||
# github.com/aws/aws-sdk-go v1.42.35
|
||||
# github.com/aws/aws-sdk-go v1.42.39
|
||||
## explicit; go 1.11
|
||||
github.com/aws/aws-sdk-go/aws
|
||||
github.com/aws/aws-sdk-go/aws/arn
|
||||
|
@ -126,8 +126,8 @@ github.com/golang/protobuf/ptypes/timestamp
|
|||
# github.com/golang/snappy v0.0.4
|
||||
## explicit
|
||||
github.com/golang/snappy
|
||||
# github.com/google/go-cmp v0.5.6
|
||||
## explicit; go 1.8
|
||||
# github.com/google/go-cmp v0.5.7
|
||||
## explicit; go 1.11
|
||||
github.com/google/go-cmp/cmp
|
||||
github.com/google/go-cmp/cmp/internal/diff
|
||||
github.com/google/go-cmp/cmp/internal/flags
|
||||
|
@ -175,7 +175,7 @@ github.com/oklog/ulid
|
|||
# github.com/pkg/errors v0.9.1
|
||||
## explicit
|
||||
github.com/pkg/errors
|
||||
# github.com/prometheus/client_golang v1.11.0
|
||||
# github.com/prometheus/client_golang v1.12.0
|
||||
## explicit; go 1.13
|
||||
github.com/prometheus/client_golang/prometheus
|
||||
github.com/prometheus/client_golang/prometheus/internal
|
||||
|
@ -264,7 +264,7 @@ go.opencensus.io/trace/tracestate
|
|||
go.uber.org/atomic
|
||||
# go.uber.org/goleak v1.1.11-0.20210813005559-691160354723
|
||||
## explicit; go 1.13
|
||||
# golang.org/x/net v0.0.0-20220114011407-0dd24b26b47d
|
||||
# golang.org/x/net v0.0.0-20220121210141-e204ce36a2ba
|
||||
## explicit; go 1.17
|
||||
golang.org/x/net/context
|
||||
golang.org/x/net/context/ctxhttp
|
||||
|
@ -337,7 +337,7 @@ google.golang.org/appengine/internal/socket
|
|||
google.golang.org/appengine/internal/urlfetch
|
||||
google.golang.org/appengine/socket
|
||||
google.golang.org/appengine/urlfetch
|
||||
# google.golang.org/genproto v0.0.0-20220114231437-d2e6a121cae0
|
||||
# google.golang.org/genproto v0.0.0-20220118154757-00ab72f36ad5
|
||||
## explicit; go 1.11
|
||||
google.golang.org/genproto/googleapis/api/annotations
|
||||
google.golang.org/genproto/googleapis/iam/v1
|
||||
|
|
Loading…
Reference in a new issue