github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2025-03-11 15:34:56 +00:00
Code Issues Projects Releases Packages Wiki Activity

Merge branch 'public-single-node' into pmm-6401-read-prometheus-data-files

Browse source
This commit is contained in:
Aliaksandr Valialkin 2022-01-18 22:38:35 +02:00
commit 8d6d4e8033
No known key found for this signature in database
GPG key ID: A72BEC6CD3D0DED1
41 changed files with 747 additions and 701 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

15
README.md
View file

@ -1144,16 +1144,21 @@ write data to the same VictoriaMetrics instance. These vmagent or Prometheus ins
Retention is configured with `-retentionPeriod` command-line flag. For instance, `-retentionPeriod=3` means
that the data will be stored for 3 months and then deleted.
Data is split in per-month subdirectories inside `<-storageDataPath>/data/small` and `<-storageDataPath>/data/big` folders.
Directories for months outside the configured retention are deleted on the first day of new month.
Data is split in per-month partitions inside `<-storageDataPath>/data/small` and `<-storageDataPath>/data/big` folders.
Data partitions outside the configured retention are deleted on the first day of new month.
Each partition consists of one or more data parts with the following name pattern `rowsCount_blocksCount_minTimestamp_maxTimestamp`.
Data parts outside of the configured retention are eventually deleted during [background merge](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282).
In order to keep data according to `-retentionPeriod` max disk space usage is going to be `-retentionPeriod` + 1 month.
For example if `-retentionPeriod` is set to 1, data for January is deleted on March 1st.
It is safe to extend `-retentionPeriod` on existing data. If `-retentionPeriod` is set to lower
value than before then data outside the configured period will be eventually deleted.
VictoriaMetrics supports retention smaller than 1 month. For example, `-retentionPeriod=5d` would set data retention for 5 days.
Older data is eventually deleted during [background merge](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282).
Please note, time range covered by data part is not limited by retention period unit. Hence, data part may contain data
for multiple days and will be deleted only when fully outside of the configured retention.
It is safe to extend `-retentionPeriod` on existing data. If `-retentionPeriod` is set to lower
value than before then data outside the configured period will be eventually deleted.
## Multiple retentions

2
app/vmbackup/README.md
View file

@ -6,7 +6,7 @@ Supported storage systems for backups:
* [GCS](https://cloud.google.com/storage/). Example: `gs://<bucket>/<path/to/backup>`
* [S3](https://aws.amazon.com/s3/). Example: `s3://<bucket>/<path/to/backup>`
* Any S3-compatible storage such as [MinIO](https://github.com/minio/minio), [Ceph](https://docs.ceph.com/docs/mimic/radosgw/s3/) or [Swift](https://www.swiftstack.com/docs/admin/middleware/s3_middleware.html). See [these docs](#advanced-usage) for details.
* Any S3-compatible storage such as [MinIO](https://github.com/minio/minio), [Ceph](https://docs.ceph.com/en/pacific/radosgw/s3/) or [Swift](https://www.swiftstack.com/docs/admin/middleware/s3_middleware.html). See [these docs](#advanced-usage) for details.
* Local filesystem. Example: `fs://</absolute/path/to/backup>`
`vmbackup` supports incremental and full backups. Incremental backups are created automatically if the destination path already contains data from the previous backup.

4
app/vmgateway/README.md
View file

@ -36,6 +36,7 @@ jwt token must be in following format:
"team": "dev",
"project": "mobile"
},
"extra_filters": ["{env~=\"prod|dev\",team!=\"test\"}"],
"mode": 1
}
}
@ -44,7 +45,8 @@ Where:
- `exp` - required, expire time in unix_timestamp. If the token expires then `vmgateway` rejects the request.
- `vm_access` - required, dict with claim info, minimum form: `{"vm_access": {"tenand_id": {}}`
- `tenant_id` - optional, for cluster mode, routes requests to the corresponding tenant.
- `extra_labels` - optional, key-value pairs for label filters added to the ingested or selected metrics.
- `extra_labels` - optional, key-value pairs for label filters added to the ingested or selected metrics. Multiple filters are added with `and` operation. If defined, `extra_label` from original request removed.
- `extra_filters` - optional, [series selectors](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) added to the select query requests. Multiple selectors are added with `or` operation. If defined, `extra_filter` from original request removed.
- `mode` - optional, access mode for api - read, write, or full. Supported values: 0 - full (default value), 1 - read, 2 - write.
## QuickStart

4
app/vmselect/vmui/asset-manifest.json
View file

@ -1,12 +1,12 @@
{
"files": {
"main.css": "./static/css/main.79ff1ad2.css",
"main.js": "./static/js/main.31aed9a0.js",
"main.js": "./static/js/main.c94cd6ac.js",
"static/js/27.cc1b69f7.chunk.js": "./static/js/27.cc1b69f7.chunk.js",
"index.html": "./index.html"
},
"entrypoints": [
"static/css/main.79ff1ad2.css",
"static/js/main.31aed9a0.js"
"static/js/main.c94cd6ac.js"
]
}

2
app/vmselect/vmui/index.html
View file

@ -1 +1 @@
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="VM-UI is a metric explorer for Victoria Metrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,500,700&display=swap"/><script defer="defer" src="./static/js/main.31aed9a0.js"></script><link href="./static/css/main.79ff1ad2.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="VM-UI is a metric explorer for Victoria Metrics"/><link rel="apple-touch-icon" href="./apple-touch-icon.png"/><link rel="icon" type="image/png" sizes="32x32" href="./favicon-32x32.png"><link rel="manifest" href="./manifest.json"/><title>VM UI</title><link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,500,700&display=swap"/><script defer="defer" src="./static/js/main.c94cd6ac.js"></script><link href="./static/css/main.79ff1ad2.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>

2
app/vmselect/vmui/static/js/main.31aed9a0.js
View file

File diff suppressed because one or more lines are too long

2
app/vmselect/vmui/static/js/main.c94cd6ac.js Normal file
View file

File diff suppressed because one or more lines are too long

0
app/vmselect/vmui/static/js/main.31aed9a0.js.LICENSE.txt → app/vmselect/vmui/static/js/main.c94cd6ac.js.LICENSE.txt
View file

12
app/vmui/packages/vmui/src/components/Header/Header.tsx
View file

@ -5,10 +5,11 @@ import Link from "@mui/material/Link";
import Toolbar from "@mui/material/Toolbar";
import Typography from "@mui/material/Typography";
import {ExecutionControls} from "../Home/Configurator/Time/ExecutionControls";
import {DisplayTypeSwitch} from "../Home/Configurator/DisplayTypeSwitch";
import Logo from "../common/Logo";
import makeStyles from "@mui/styles/makeStyles";
import {setQueryStringWithoutPageReload} from "../../utils/query-string";
import {TimeSelector} from "../Home/Configurator/Time/TimeSelector";
import GlobalSettings from "../Home/Configurator/Settings/GlobalSettings";
const useStyles = makeStyles({
logo: {
@ -22,8 +23,6 @@ const useStyles = makeStyles({
}
},
issueLink: {
position: "absolute",
bottom: "6px",
textAlign: "center",
fontSize: "10px",
opacity: ".4",
@ -45,7 +44,7 @@ const Header: FC = () => {
window.location.reload();
};
return <AppBar position="static">
return <AppBar position="static" sx={{px: 1, boxShadow: "none"}}>
<Toolbar>
<Box display="grid" alignItems="center" justifyContent="center">
<Box onClick={onClickLogo} className={classes.logo}>
@ -60,10 +59,11 @@ const Header: FC = () => {
create an issue
</Link>
</Box>
<Box ml={4} flexGrow={1}>
<Box display="grid" gridTemplateColumns="repeat(3, auto)" gap={1} alignItems="center" ml="auto" mr={0}>
<TimeSelector/>
<ExecutionControls/>
<GlobalSettings/>
</Box>
<DisplayTypeSwitch/>
</Toolbar>
</AppBar>;
};

57
app/vmui/packages/vmui/src/components/Home/Configurator/DisplayTypeSwitch.tsx
View file

@ -2,48 +2,39 @@ import React, {FC} from "preact/compat";
import TableChartIcon from "@mui/icons-material/TableChart";
import ShowChartIcon from "@mui/icons-material/ShowChart";
import CodeIcon from "@mui/icons-material/Code";
import ToggleButton from "@mui/material/ToggleButton";
import ToggleButtonGroup from "@mui/material/ToggleButtonGroup";
import Tabs from "@mui/material/Tabs";
import Tab from "@mui/material/Tab";
import {useAppDispatch, useAppState} from "../../../state/common/StateContext";
import withStyles from "@mui/styles/withStyles";
import {SyntheticEvent} from "react";
export type DisplayType = "table" | "chart" | "code";
const StylizedToggleButton = withStyles({
root: {
display: "grid",
gridTemplateColumns: "18px auto",
gridGap: 6,
padding: "8px 12px",
color: "white",
lineHeight: "19px",
"&.Mui-selected": {
color: "white"
}
}
})(ToggleButton);
const tabs = [
{value: "chart", icon: <ShowChartIcon/>, label: "Graph"},
{value: "code", icon: <CodeIcon/>, label: "JSON"},
{value: "table", icon: <TableChartIcon/>, label: "Table"}
];
export const DisplayTypeSwitch: FC = () => {
const {displayType} = useAppState();
const dispatch = useAppDispatch();
return <ToggleButtonGroup
const handleChange = (event: SyntheticEvent, newValue: DisplayType) => {
dispatch({type: "SET_DISPLAY_TYPE", payload: newValue ?? displayType});
};
return <Tabs
value={displayType}
exclusive
onChange={
(e, val) =>
// Toggle Button Group returns null in case of click on selected element, avoiding it
dispatch({type: "SET_DISPLAY_TYPE", payload: val ?? displayType})
}>
<StylizedToggleButton value="chart" aria-label="display as chart">
<ShowChartIcon/><span>Query Range as Chart</span>
</StylizedToggleButton>
<StylizedToggleButton value="code" aria-label="display as code">
<CodeIcon/><span>Instant Query as JSON</span>
</StylizedToggleButton>
<StylizedToggleButton value="table" aria-label="display as table">
<TableChartIcon/><span>Instant Query as Table</span>
</StylizedToggleButton>
</ToggleButtonGroup>;
onChange={handleChange}
sx={{minHeight: "0", marginBottom: "-1px"}}
>
{tabs.map(t =>
<Tab key={t.value}
icon={t.icon}
iconPosition="start"
label={t.label} value={t.value}
sx={{minHeight: "41px"}}
/>)}
</Tabs>;
};

56
app/vmui/packages/vmui/src/components/Home/Configurator/Graph/GraphSettings.tsx
View file

@ -2,13 +2,14 @@ import SettingsIcon from "@mui/icons-material/Settings";
import React, {FC, useState} from "preact/compat";
import AxesLimitsConfigurator from "./AxesLimitsConfigurator";
import Box from "@mui/material/Box";
import Button from "@mui/material/Button";
import IconButton from "@mui/material/IconButton";
import Paper from "@mui/material/Paper";
import Popover from "@mui/material/Popover";
import Popper from "@mui/material/Popper";
import Tooltip from "@mui/material/Tooltip";
import Typography from "@mui/material/Typography";
import makeStyles from "@mui/styles/makeStyles";
import CloseIcon from "@mui/icons-material/Close";
import ClickAwayListener from "@mui/material/ClickAwayListener";
const useStyles = makeStyles({
popover: {
@ -32,40 +33,39 @@ const useStyles = makeStyles({
}
});
const title = "Axes Settings";
const GraphSettings: FC = () => {
const [anchorEl, setAnchorEl] = useState<HTMLButtonElement | null>(null);
const open = Boolean(anchorEl);
const classes = useStyles();
return <Box display="flex" px={2}>
<Button variant="outlined" aria-describedby="settings-popover"
onClick={(e) => setAnchorEl(e.currentTarget)} >
<SettingsIcon sx={{fontSize: 16, marginRight: "4px"}}/>
<span style={{lineHeight: 1, paddingTop: "1px"}}>{open ? "Hide" : "Show"} graph settings</span>
</Button>
<Popover
id="settings-popover"
return <Box>
<Tooltip title={title}>
<IconButton onClick={(e) => setAnchorEl(e.currentTarget)}>
<SettingsIcon/>
</IconButton>
</Tooltip>
<Popper
open={open}
anchorEl={anchorEl}
onClose={() => setAnchorEl(null)}
anchorOrigin={{
vertical: "top",
horizontal: anchorEl ? anchorEl.offsetWidth + 15 : 200
}}
>
<Paper elevation={3} className={classes.popover}>
<div id="handle" className={classes.popoverHeader}>
<Typography variant="body1"><b>Graph Settings</b></Typography>
<IconButton size="small" onClick={() => setAnchorEl(null)}>
<CloseIcon style={{color: "white"}}/>
</IconButton>
</div>
<Box className={classes.popoverBody}>
<AxesLimitsConfigurator/>
</Box>
</Paper>
</Popover>
placement="left-start"
modifiers={[{name: "offset", options: {offset: [0, 6]}}]}>
<ClickAwayListener onClickAway={() => setAnchorEl(null)}>
<Paper elevation={3} className={classes.popover}>
<div id="handle" className={classes.popoverHeader}>
<Typography variant="body1"><b>{title}</b></Typography>
<IconButton size="small" onClick={() => setAnchorEl(null)}>
<CloseIcon style={{color: "white"}}/>
</IconButton>
</div>
<Box className={classes.popoverBody}>
<AxesLimitsConfigurator/>
</Box>
</Paper>
</ClickAwayListener>
</Popper>
</Box>;
};

101
app/vmui/packages/vmui/src/components/Home/Configurator/Query/QueryConfigurator.tsx
View file

@ -1,21 +1,12 @@
import React, {FC, useEffect, useRef, useState} from "preact/compat";
import Accordion from "@mui/material/Accordion";
import AccordionDetails from "@mui/material/AccordionDetails";
import AccordionSummary from "@mui/material/AccordionSummary";
import React, {FC, useEffect, useRef} from "preact/compat";
import Box from "@mui/material/Box";
import IconButton from "@mui/material/IconButton";
import Typography from "@mui/material/Typography";
import Tooltip from "@mui/material/Tooltip";
import Button from "@mui/material/Button";
import Portal from "@mui/material/Portal";
import QueryEditor from "./QueryEditor";
import {TimeSelector} from "../Time/TimeSelector";
import {useAppDispatch, useAppState} from "../../../../state/common/StateContext";
import ExpandMoreIcon from "@mui/icons-material/ExpandMore";
import HighlightOffIcon from "@mui/icons-material/HighlightOff";
import AddIcon from "@mui/icons-material/Add";
import AddCircleOutlineIcon from "@mui/icons-material/AddCircleOutline";
import PlayCircleOutlineIcon from "@mui/icons-material/PlayCircleOutline";
import ServerConfigurator from "./ServerConfigurator";
import AdditionalSettings from "./AdditionalSettings";
import {ErrorTypes} from "../../../../types";
@ -28,15 +19,11 @@ const QueryConfigurator: FC<QueryConfiguratorProps> = ({error, queryOptions}) =>
const {query, queryHistory, queryControls: {autocomplete}} = useAppState();
const dispatch = useAppDispatch();
const [expanded, setExpanded] = useState(true);
const queryContainer = useRef<HTMLDivElement>(null);
const queryRef = useRef(query);
useEffect(() => {
queryRef.current = query;
}, [query]);
const onSetDuration = (dur: string) => dispatch({type: "SET_DURATION", payload: dur});
const updateHistory = () => {
dispatch({
type: "SET_QUERY_HISTORY", payload: query.map((q, i) => {
@ -80,62 +67,34 @@ const QueryConfigurator: FC<QueryConfiguratorProps> = ({error, queryOptions}) =>
payload: {value: {values, index: newIndexHistory}, queryNumber: indexQuery}
});
};
return <>
<Accordion expanded={expanded} onChange={() => setExpanded(prev => !prev)}>
<AccordionSummary
expandIcon={<IconButton><ExpandMoreIcon/></IconButton>}
aria-controls="panel1a-content"
id="panel1a-header"
sx={{alignItems: "flex-start", padding: "15px"}}
>
<Box mr={2}>
<Typography variant="h6" component="h2">Query Configuration</Typography>
</Box>
<Box flexGrow={1} onClick={e => e.stopPropagation()} onFocusCapture={e => e.stopPropagation()}>
<Portal disablePortal={!expanded} container={queryContainer.current}>
{query.map((q, i) =>
<Box key={i} display="grid" gridTemplateColumns="1fr auto" gap="4px" width="100%"
mb={i === query.length - 1 ? 0 : 2}>
<QueryEditor query={query[i]} index={i} autocomplete={autocomplete} queryOptions={queryOptions}
error={error} setHistoryIndex={setHistoryIndex} runQuery={onRunQuery} setQuery={onSetQuery}/>
{i === 0 && <Tooltip title="Execute Query">
<IconButton onClick={onRunQuery}>
<PlayCircleOutlineIcon/>
</IconButton>
</Tooltip>}
{i > 0 && <Tooltip title="Remove Query">
<IconButton onClick={() => onRemoveQuery(i)}>
<HighlightOffIcon/>
</IconButton>
</Tooltip>}
</Box>)}
</Portal>
</Box>
</AccordionSummary>
<AccordionDetails>
<Box display="flex" flexWrap="wrap" gap={2}>
<Box flexGrow="2" minWidth="50%">
<ServerConfigurator error={error}/>
{/* for portal QueryEditor */}
<div ref={queryContainer}/>
{query.length < 2 && <Box display="inline-block" minHeight="40px" mt={2}>
<Button onClick={onAddQuery} variant="outlined">
<AddIcon sx={{fontSize: 16, marginRight: "4px"}}/>
<span style={{lineHeight: 1, paddingTop: "1px"}}>Query</span>
</Button>
</Box>}
</Box>
<Box flexGrow="1">
<TimeSelector setDuration={onSetDuration}/>
</Box>
<Box flexBasis="100%" pt={1}>
<AdditionalSettings/>
</Box>
</Box>
</AccordionDetails>
</Accordion>
</>;
return <Box boxShadow="rgba(99, 99, 99, 0.2) 0px 2px 8px 0px;" p={4} pb={2} m={-4} mb={2}>
<Box>
{query.map((q, i) =>
<Box key={i} display="grid" gridTemplateColumns="1fr auto auto" gap="4px" width="100%"
mb={i === query.length - 1 ? 0 : 2.5}>
<QueryEditor query={query[i]} index={i} autocomplete={autocomplete} queryOptions={queryOptions}
error={error} setHistoryIndex={setHistoryIndex} runQuery={onRunQuery} setQuery={onSetQuery}/>
{i === 0 && <Tooltip title="Execute Query">
<IconButton onClick={onRunQuery} sx={{height: "49px", width: "49px"}}>
<PlayCircleOutlineIcon/>
</IconButton>
</Tooltip>}
{query.length < 2 && <Tooltip title="Add Query">
<IconButton onClick={onAddQuery} sx={{height: "49px", width: "49px"}}>
<AddCircleOutlineIcon/>
</IconButton>
</Tooltip>}
{i > 0 && <Tooltip title="Remove Query">
<IconButton onClick={() => onRemoveQuery(i)} sx={{height: "49px", width: "49px"}}>
<HighlightOffIcon/>
</IconButton>
</Tooltip>}
</Box>)}
</Box>
<Box mt={3}>
<AdditionalSettings/>
</Box>
</Box>;
};
export default QueryConfigurator;

97
app/vmui/packages/vmui/src/components/Home/Configurator/Query/QueryEditor.tsx
View file

@ -1,9 +1,12 @@
import React, {FC, useEffect, useState} from "preact/compat";
import React, {FC, useEffect, useMemo, useRef, useState} from "preact/compat";
import {KeyboardEvent} from "react";
import {ErrorTypes} from "../../../../types";
import Autocomplete from "@mui/material/Autocomplete";
import Popper from "@mui/material/Popper";
import TextField from "@mui/material/TextField";
import {queryToBreakLine} from "../../../../utils/query-string";
import Box from "@mui/material/Box";
import Paper from "@mui/material/Paper";
import MenuItem from "@mui/material/MenuItem";
import MenuList from "@mui/material/MenuList";
export interface QueryEditorProps {
setHistoryIndex: (step: number, index: number) => void;
@ -28,18 +31,43 @@ const QueryEditor: FC<QueryEditorProps> = ({
queryOptions
}) => {
const [value, setValue] = useState(query);
const [downMetaKeys, setDownMetaKeys] = useState<string[]>([]);
const [focusField, setFocusField] = useState(false);
const [focusOption, setFocusOption] = useState(-1);
const autocompleteAnchorEl = useRef<HTMLDivElement>(null);
const wrapperEl = useRef<HTMLUListElement>(null);
useEffect(() => {
setValue(queryToBreakLine(query));
}, [query]);
const openAutocomplete = useMemo(() => {
return !(!autocomplete || downMetaKeys.length || query.length < 2 || !focusField);
}, [query, downMetaKeys, autocomplete, focusField]);
const handleKeyDown = (e: KeyboardEvent<HTMLDivElement>): void => {
if (e.ctrlKey || e.metaKey) setDownMetaKeys([...downMetaKeys, e.key]);
const actualOptions = useMemo(() => {
if (!openAutocomplete) return [];
try {
const regexp = new RegExp(String(query), "i");
return queryOptions.filter((item) => regexp.test(item) && item !== query);
} catch (e) {
return [];
}
}, [autocomplete, query, queryOptions]);
const handleKeyDown = (e: KeyboardEvent<HTMLDivElement>) => {
const {key, ctrlKey, metaKey, shiftKey} = e;
if (ctrlKey || metaKey) setDownMetaKeys([...downMetaKeys, e.key]);
if (key === "ArrowUp" && openAutocomplete && actualOptions.length) {
e.preventDefault();
setFocusOption((prev) => prev === 0 ? 0 : prev - 1);
} else if (key === "ArrowDown" && openAutocomplete && actualOptions.length) {
e.preventDefault();
setFocusOption((prev) => prev >= actualOptions.length - 1 ? actualOptions.length - 1 : prev + 1);
} else if (key === "Enter" && openAutocomplete && actualOptions.length && !shiftKey) {
e.preventDefault();
setQuery(actualOptions[focusOption], index);
}
return true;
};
const handleKeyUp = (e: KeyboardEvent<HTMLDivElement>): void => {
const handleKeyUp = (e: KeyboardEvent<HTMLDivElement>) => {
const {key, ctrlKey, metaKey} = e;
if (downMetaKeys.includes(key)) setDownMetaKeys(downMetaKeys.filter(k => k !== key));
const ctrlMetaKey = ctrlKey || metaKey;
@ -52,25 +80,36 @@ const QueryEditor: FC<QueryEditorProps> = ({
}
};
return <Autocomplete
freeSolo
fullWidth
disableClearable
options={autocomplete && !downMetaKeys.length ? queryOptions : []}
onChange={(event, value) => setQuery(value, index)}
onKeyDown={handleKeyDown}
onKeyUp={handleKeyUp}
value={value}
renderInput={(params) =>
<TextField
{...params}
label={`Query ${index + 1}`}
multiline
error={!!error}
onChange={(e) => setQuery(e.target.value, index)}
/>
}
/>;
useEffect(() => {
if (!wrapperEl.current) return;
const target = wrapperEl.current.childNodes[focusOption] as HTMLElement;
if (target?.scrollIntoView) target.scrollIntoView({block: "center"});
}, [focusOption]);
return <Box ref={autocompleteAnchorEl}>
<TextField
defaultValue={query}
fullWidth
label={`Query ${index + 1}`}
multiline
error={!!error}
onFocus={() => setFocusField(true)}
onBlur={() => setFocusField(false)}
onKeyUp={handleKeyUp}
onKeyDown={handleKeyDown}
onChange={(e) => setQuery(e.target.value, index)}
/>
<Popper open={openAutocomplete} anchorEl={autocompleteAnchorEl.current} placement="bottom-start">
<Paper elevation={3} sx={{ maxHeight: 300, overflow: "auto" }}>
<MenuList ref={wrapperEl} dense>
{actualOptions.map((item, i) =>
<MenuItem key={item} sx={{bgcolor: `rgba(0, 0, 0, ${i === focusOption ? 0.12 : 0})`}}>
{item}
</MenuItem>)}
</MenuList>
</Paper>
</Popper>
</Box>;
};
export default QueryEditor;

51
app/vmui/packages/vmui/src/components/Home/Configurator/Query/ServerConfigurator.tsx
View file

@ -1,51 +0,0 @@
import React, {FC, useEffect, useState} from "preact/compat";
import Box from "@mui/material/Box";
import TextField from "@mui/material/TextField";
import Tooltip from "@mui/material/Tooltip";
import IconButton from "@mui/material/IconButton";
import SecurityIcon from "@mui/icons-material/Security";
import {useAppDispatch, useAppState} from "../../../../state/common/StateContext";
import {AuthDialog} from "../Auth/AuthDialog";
import {ErrorTypes} from "../../../../types";
import {getAppModeEnable, getAppModeParams} from "../../../../utils/app-mode";
export interface ServerConfiguratorProps {
error?: ErrorTypes | string;
}
const ServerConfigurator: FC<ServerConfiguratorProps> = ({error}) => {
const appModeEnable = getAppModeEnable();
const {serverURL: appServerUrl} = getAppModeParams();
const {serverUrl} = useAppState();
const dispatch = useAppDispatch();
const onSetServer = ({target: {value}}: {target: {value: string}}) => {
dispatch({type: "SET_SERVER", payload: value});
};
const [dialogOpen, setDialogOpen] = useState(false);
useEffect(() => {
if (appModeEnable) dispatch({type: "SET_SERVER", payload: appServerUrl});
}, [appServerUrl]);
return <>
<Box display="grid" gridTemplateColumns="1fr auto" gap="4px" alignItems="center" width="100%" mb={2} minHeight={50}>
<TextField variant="outlined" fullWidth label="Server URL" value={serverUrl || ""} disabled={appModeEnable}
error={error === ErrorTypes.validServer || error === ErrorTypes.emptyServer}
inputProps={{style: {fontFamily: "Monospace"}}}
onChange={onSetServer}/>
<Box>
<Tooltip title="Request Auth Settings">
<IconButton onClick={() => setDialogOpen(true)}>
<SecurityIcon/>
</IconButton>
</Tooltip>
</Box>
</Box>
<AuthDialog open={dialogOpen} onClose={() => setDialogOpen(false)}/>
</>;
};
export default ServerConfigurator;

52
app/vmui/packages/vmui/src/components/Home/Configurator/Query/useFetchQuery.ts
View file

@ -1,12 +1,13 @@
import {useEffect, useMemo, useState} from "preact/compat";
import {useEffect, useMemo, useCallback, useState} from "preact/compat";
import {getQueryOptions, getQueryRangeUrl, getQueryUrl} from "../../../../api/query-range";
import {useAppState} from "../../../../state/common/StateContext";
import {InstantMetricResult, MetricBase, MetricResult} from "../../../../api/types";
import {isValidHttpUrl} from "../../../../utils/url";
import {useAuthState} from "../../../../state/auth/AuthStateContext";
import {ErrorTypes, TimeParams} from "../../../../types";
import {ErrorTypes} from "../../../../types";
import {useGraphState} from "../../../../state/graph/GraphStateContext";
import {getAppModeEnable, getAppModeParams} from "../../../../utils/app-mode";
import throttle from "lodash.throttle";
const appModeEnable = getAppModeEnable();
const {serverURL: appServerUrl} = getAppModeParams();
@ -19,7 +20,7 @@ export const useFetchQuery = (): {
error?: ErrorTypes | string,
queryOptions: string[],
} => {
const {query, displayType, serverUrl, time: {period}, queryControls: {nocache, autoRefresh}} = useAppState();
const {query, displayType, serverUrl, time: {period}, queryControls: {nocache}} = useAppState();
const {basicData, bearerData, authMethod} = useAuthState();
const {customStep} = useGraphState();
@ -29,7 +30,7 @@ export const useFetchQuery = (): {
const [graphData, setGraphData] = useState<MetricResult[]>();
const [liveData, setLiveData] = useState<InstantMetricResult[]>();
const [error, setError] = useState<ErrorTypes | string>();
const [prevPeriod, setPrevPeriod] = useState<TimeParams>();
const [fetchQueue, setFetchQueue] = useState<AbortController[]>([]);
useEffect(() => {
if (error) {
@ -38,19 +39,11 @@ export const useFetchQuery = (): {
}
}, [error]);
const needUpdateData = useMemo(() => {
if (!prevPeriod || autoRefresh) return true;
const duration = (prevPeriod.end - prevPeriod.start) / 3;
const factorLimit = duration / (period.end - period.start) >= 0.7;
const maxLimit = period.end > (prevPeriod.end + duration);
const minLimit = period.start < (prevPeriod.start - duration);
return factorLimit || maxLimit || minLimit;
}, [period]);
const fetchData = async () => {
const fetchData = async (fetchUrl: string[] | undefined) => {
if (!fetchUrl?.length) return;
const controller = new AbortController();
setFetchQueue([...fetchQueue, controller]);
setIsLoading(true);
setPrevPeriod(period);
const headers = new Headers();
if (authMethod === "BASIC_AUTH") {
@ -61,7 +54,7 @@ export const useFetchQuery = (): {
}
try {
const responses = await Promise.all(fetchUrl.map(url => fetch(url, {headers})));
const responses = await Promise.all(fetchUrl.map(url => fetch(url, {headers, signal: controller.signal})));
const tempData = [];
let counter = 1;
for await (const response of responses) {
@ -79,12 +72,16 @@ export const useFetchQuery = (): {
}
displayType === "chart" ? setGraphData(tempData) : setLiveData(tempData);
} catch (e) {
if (e instanceof Error) setError(`${e.name}: ${e.message}`);
if (e instanceof Error && e.name !== "AbortError") {
setError(`${e.name}: ${e.message}`);
}
}
setIsLoading(false);
};
const throttledFetchData = useCallback(throttle(fetchData, 300), []);
const fetchOptions = async () => {
if (!serverUrl) return;
const url = getQueryOptions(serverUrl);
@ -108,11 +105,9 @@ export const useFetchQuery = (): {
} else if (query.every(q => !q.trim())) {
setError(ErrorTypes.validQuery);
} else if (isValidHttpUrl(server)) {
const duration = (period.end - period.start) / 2;
const bufferPeriod = {...period, start: period.start - duration, end: period.end + duration};
if (customStep.enable) bufferPeriod.step = customStep.value;
if (customStep.enable) period.step = customStep.value;
return query.filter(q => q.trim()).map(q => displayType === "chart"
? getQueryRangeUrl(server, q, bufferPeriod, nocache)
? getQueryRangeUrl(server, q, period, nocache)
: getQueryUrl(server, q, period));
} else {
setError(ErrorTypes.validServer);
@ -124,18 +119,17 @@ export const useFetchQuery = (): {
fetchOptions();
}, [serverUrl]);
useEffect(() => {
setPrevPeriod(undefined);
}, [query]);
// TODO: this should depend on query as well, but need to decide when to do the request. Doing it on each query change - looks to be a bad idea. Probably can be done on blur
useEffect(() => {
fetchData();
}, [serverUrl, displayType, customStep]);
throttledFetchData(fetchUrl);
}, [fetchUrl]);
useEffect(() => {
if (needUpdateData) fetchData();
}, [period]);
const fetchPast = fetchQueue.slice(0, -1);
if (!fetchPast.length) return;
fetchPast.map(f => f.abort());
setFetchQueue(fetchQueue.filter(f => !f.signal.aborted));
}, [fetchQueue]);
return { fetchUrl, isLoading, graphData, liveData, error, queryOptions: queryOptions };
};

82
app/vmui/packages/vmui/src/components/Home/Configurator/Settings/GlobalSettings.tsx Normal file
View file

@ -0,0 +1,82 @@
import React, {FC, useState} from "preact/compat";
import Tooltip from "@mui/material/Tooltip";
import SettingsIcon from "@mui/icons-material/Settings";
import Button from "@mui/material/Button";
import Box from "@mui/material/Box";
import Modal from "@mui/material/Modal";
import ServerConfigurator from "./ServerConfigurator";
import Typography from "@mui/material/Typography";
import CloseIcon from "@mui/icons-material/Close";
import IconButton from "@mui/material/IconButton";
import {useAppDispatch, useAppState} from "../../../../state/common/StateContext";
import {getAppModeEnable} from "../../../../utils/app-mode";
const modalStyle = {
position: "absolute" as const,
top: "50%",
left: "50%",
transform: "translate(-50%, -50%)",
bgcolor: "background.paper",
p: 3,
borderRadius: "4px",
width: "80%",
maxWidth: "800px"
};
const title = "Setting Server URL";
const GlobalSettings: FC = () => {
const appModeEnable = getAppModeEnable();
const {serverUrl} = useAppState();
const dispatch = useAppDispatch();
const [changedServerUrl, setChangedServerUrl] = useState(serverUrl);
const setServer = () => {
if (!appModeEnable) dispatch({type: "SET_SERVER", payload: changedServerUrl});
handleClose();
};
const [open, setOpen] = useState(false);
const handleOpen = () => setOpen(true);
const handleClose = () => setOpen(false);
return <>
<Tooltip title={title}>
<Button variant="contained" color="primary"
sx={{
color: "white",
border: "1px solid rgba(0, 0, 0, 0.2)",
minWidth: "34px",
padding: "6px 8px",
boxShadow: "none",
}}
startIcon={<SettingsIcon style={{marginRight: "-8px", marginLeft: "4px"}}/>}
onClick={handleOpen}>
</Button>
</Tooltip>
<Modal open={open} onClose={handleClose}>
<Box sx={modalStyle}>
<Box display="grid" gridTemplateColumns="1fr auto" alignItems="center" mb={4}>
<Typography id="modal-modal-title" variant="h6" component="h2">
{title}
</Typography>
<IconButton size="small" onClick={handleClose}>
<CloseIcon/>
</IconButton>
</Box>
<ServerConfigurator setServer={setChangedServerUrl}/>
<Box display="grid" gridTemplateColumns="auto auto" gap={1} justifyContent="end" mt={4}>
<Button variant="outlined" onClick={handleClose}>
Cancel
</Button>
<Button variant="contained" onClick={setServer}>
apply
</Button>
</Box>
</Box>
</Modal>
</>;
};
export default GlobalSettings;

41
app/vmui/packages/vmui/src/components/Home/Configurator/Settings/ServerConfigurator.tsx Normal file
View file

@ -0,0 +1,41 @@
import React, {FC, useEffect, useState} from "preact/compat";
import TextField from "@mui/material/TextField";
import {useAppDispatch, useAppState} from "../../../../state/common/StateContext";
import {ErrorTypes} from "../../../../types";
import {getAppModeEnable, getAppModeParams} from "../../../../utils/app-mode";
import {ChangeEvent} from "react";
export interface ServerConfiguratorProps {
error?: ErrorTypes | string;
setServer: (url: string) => void
}
const ServerConfigurator: FC<ServerConfiguratorProps> = ({error, setServer}) => {
const appModeEnable = getAppModeEnable();
const {serverURL: appServerUrl} = getAppModeParams();
const {serverUrl} = useAppState();
const dispatch = useAppDispatch();
const [changedServerUrl, setChangedServerUrl] = useState(serverUrl);
useEffect(() => {
if (appModeEnable) {
dispatch({type: "SET_SERVER", payload: appServerUrl});
setChangedServerUrl(appServerUrl);
}
}, [appServerUrl]);
const onChangeServer = (e: ChangeEvent<HTMLTextAreaElement | HTMLInputElement>) => {
const value = e.target.value || "";
setChangedServerUrl(value);
setServer(value);
};
return <TextField variant="outlined" fullWidth label="Server URL" value={changedServerUrl || ""} disabled={appModeEnable}
error={error === ErrorTypes.validServer || error === ErrorTypes.emptyServer}
inputProps={{style: {fontFamily: "Monospace"}}}
onChange={onChangeServer}/>;
};
export default ServerConfigurator;

140
app/vmui/packages/vmui/src/components/Home/Configurator/Time/ExecutionControls.tsx
View file

@ -1,92 +1,100 @@
import React, {FC, useEffect, useState} from "preact/compat";
import Box from "@mui/material/Box";
import FormControlLabel from "@mui/material/FormControlLabel";
import IconButton from "@mui/material/IconButton";
import Tooltip from "@mui/material/Tooltip";
import EqualizerIcon from "@mui/icons-material/Equalizer";
import {useAppDispatch, useAppState} from "../../../../state/common/StateContext";
import CircularProgressWithLabel from "../../../common/CircularProgressWithLabel";
import makeStyles from "@mui/styles/makeStyles";
import BasicSwitch from "../../../../theme/switch";
import Button from "@mui/material/Button";
import Popper from "@mui/material/Popper";
import Paper from "@mui/material/Paper";
import ClickAwayListener from "@mui/material/ClickAwayListener";
import AutorenewIcon from "@mui/icons-material/Autorenew";
import KeyboardArrowDownIcon from "@mui/icons-material/KeyboardArrowDown";
import List from "@mui/material/List";
import ListItem from "@mui/material/ListItem";
import ListItemText from "@mui/material/ListItemText";
const useStyles = makeStyles({
colorizing: {
color: "white"
}
});
interface AutoRefreshOption {
seconds: number
title: string
}
const delayOptions: AutoRefreshOption[] = [
{seconds: 0, title: "Off"},
{seconds: 1, title: "1s"},
{seconds: 2, title: "2s"},
{seconds: 5, title: "5s"},
{seconds: 10, title: "10s"},
{seconds: 30, title: "30s"},
{seconds: 60, title: "1m"},
{seconds: 300, title: "5m"},
{seconds: 900, title: "15m"},
{seconds: 1800, title: "30m"},
{seconds: 3600, title: "1h"},
{seconds: 7200, title: "2h"}
];
export const ExecutionControls: FC = () => {
const classes = useStyles();
const dispatch = useAppDispatch();
const {queryControls: {autoRefresh}} = useAppState();
const [delay, setDelay] = useState<(1|2|5)>(5);
const [lastUpdate, setLastUpdate] = useState<number|undefined>();
const [progress, setProgress] = React.useState(100);
const [selectedDelay, setSelectedDelay] = useState<AutoRefreshOption>(delayOptions[0]);
const handleChange = () => {
dispatch({type: "TOGGLE_AUTOREFRESH"});
const handleChange = (d: AutoRefreshOption) => {
if ((autoRefresh && !d.seconds) || (!autoRefresh && d.seconds)) {
dispatch({type: "TOGGLE_AUTOREFRESH"});
}
setSelectedDelay(d);
setAnchorEl(null);
};
useEffect(() => {
const delay = selectedDelay.seconds;
let timer: number;
if (autoRefresh) {
setLastUpdate(new Date().valueOf());
timer = setInterval(() => {
setLastUpdate(new Date().valueOf());
dispatch({type: "RUN_QUERY_TO_NOW"});
}, delay * 1000) as unknown as number;
} else {
setSelectedDelay(delayOptions[0]);
}
return () => {
timer && clearInterval(timer);
};
}, [delay, autoRefresh]);
}, [selectedDelay, autoRefresh]);
useEffect(() => {
const timer = setInterval(() => {
if (autoRefresh && lastUpdate) {
const delta = (new Date().valueOf() - lastUpdate) / 1000; //s
const nextValue = Math.floor(delta / delay * 100);
setProgress(nextValue);
}
}, 16);
return () => {
clearInterval(timer);
};
}, [autoRefresh, lastUpdate, delay]);
const [anchorEl, setAnchorEl] = useState<HTMLButtonElement | null>(null);
const open = Boolean(anchorEl);
const iterateDelays = () => {
setDelay(prev => {
switch (prev) {
case 1:
return 2;
case 2:
return 5;
case 5:
return 1;
default:
return 5;
}
});
};
return <Box display="flex" alignItems="center">
{<FormControlLabel
control={<BasicSwitch className={classes.colorizing} checked={autoRefresh} onChange={handleChange} />}
label="Auto-refresh"
/>}
{autoRefresh && <>
<CircularProgressWithLabel className={classes.colorizing} label={delay} value={progress}
onClick={() => {iterateDelays();}} />
<Tooltip title="Change delay refresh">
<Box ml={1}>
<IconButton onClick={() => {iterateDelays();}}>
<EqualizerIcon style={{color: "white"}} />
</IconButton>
</Box>
</Tooltip>
</>}
</Box>;
return <>
<Tooltip title="Auto-refresh control">
<Button variant="contained" color="primary"
sx={{
minWidth: "110px",
color: "white",
border: "1px solid rgba(0, 0, 0, 0.2)",
justifyContent: "space-between",
boxShadow: "none",
}}
startIcon={<AutorenewIcon/>}
endIcon={<KeyboardArrowDownIcon sx={{transform: open ? "rotate(180deg)" : "none"}}/>}
onClick={(e) => setAnchorEl(e.currentTarget)}
>
{selectedDelay.title}
</Button>
</Tooltip>
<Popper
open={open}
anchorEl={anchorEl}
placement="bottom-end"
modifiers={[{name: "offset", options: {offset: [0, 6]}}]}>
<ClickAwayListener onClickAway={() => setAnchorEl(null)}>
<Paper elevation={3}>
<List style={{minWidth: "110px",maxHeight: "208px", overflow: "auto", padding: "20px 0"}}>
{delayOptions.map(d =>
<ListItem key={d.seconds} button onClick={() => handleChange(d)}>
<ListItemText primary={d.title}/>
</ListItem>)}
</List>
</Paper>
</ClickAwayListener></Popper>
</>;
};

31
app/vmui/packages/vmui/src/components/Home/Configurator/Time/TimeDurationPopover.tsx
View file

@ -1,31 +0,0 @@
import React, {FC} from "preact/compat";
import Paper from "@mui/material/Paper";
import Table from "@mui/material/Table";
import TableBody from "@mui/material/TableBody";
import TableCell from "@mui/material/TableCell";
import TableContainer from "@mui/material/TableContainer";
import TableHead from "@mui/material/TableHead";
import TableRow from "@mui/material/TableRow";
import {supportedDurations} from "../../../../utils/time";
export const TimeDurationPopover: FC = () => {
return <TableContainer component={Paper}>
<Table aria-label="simple table" size="small">
<TableHead>
<TableRow>
<TableCell>Long</TableCell>
<TableCell>Short</TableCell>
</TableRow>
</TableHead>
<TableBody>
{supportedDurations.map((row, index) => (
<TableRow key={index}>
<TableCell component="th" scope="row">{row.long}</TableCell>
<TableCell>{row.short}</TableCell>
</TableRow>
))}
</TableBody>
</Table>
</TableContainer>;
};

132
app/vmui/packages/vmui/src/components/Home/Configurator/Time/TimeDurationSelector.tsx
View file

@ -1,103 +1,47 @@
import React, {FC, useEffect, useState} from "preact/compat";
import {ChangeEvent, MouseEvent, KeyboardEvent} from "react";
import Box from "@mui/material/Box";
import Popover from "@mui/material/Popover";
import TextField from "@mui/material/TextField";
import Typography from "@mui/material/Typography";
import {checkDurationLimit} from "../../../../utils/time";
import {TimeDurationPopover} from "./TimeDurationPopover";
import {InlineBtn} from "../../../common/InlineBtn";
import {useAppState} from "../../../../state/common/StateContext";
import React, {FC} from "preact/compat";
import List from "@mui/material/List";
import ListItem from "@mui/material/ListItem";
import ListItemText from "@mui/material/ListItemText";
import dayjs from "dayjs";
interface TimeDurationSelector {
setDuration: (str: string) => void;
setDuration: (str: string, from: Date) => void;
}
interface DurationOption {
duration: string,
title?: string,
from?: () => Date,
}
const durationOptions: DurationOption[] = [
{duration: "5m", title: "Last 5 minutes"},
{duration: "15m", title: "Last 15 minutes"},
{duration: "30m", title: "Last 30 minutes"},
{duration: "1h", title: "Last 1 hour"},
{duration: "3h", title: "Last 3 hours"},
{duration: "6h", title: "Last 6 hours"},
{duration: "12h", title: "Last 12 hours"},
{duration: "24h", title: "Last 24 hours"},
{duration: "2d", title: "Last 2 days"},
{duration: "7d", title: "Last 7 days"},
{duration: "30d", title: "Last 30 days"},
{duration: "90d", title: "Last 90 days"},
{duration: "6m", title: "Last 6 months"},
{duration: "1y", title: "Last 1 year"},
{duration: "1d", from: () => dayjs().subtract(1, "day").endOf("day").toDate(), title: "Yesterday"},
{duration: "1d", from: () => dayjs().endOf("day").toDate(), title: "Today"},
];
const TimeDurationSelector: FC<TimeDurationSelector> = ({setDuration}) => {
const {time: {duration}} = useAppState();
// setDurationString("5m"))
const [anchorEl, setAnchorEl] = React.useState<Element | null>(null);
const [durationString, setDurationString] = useState<string>(duration);
const [durationStringFocused, setFocused] = useState(false);
const open = Boolean(anchorEl);
const handleDurationChange = (event: ChangeEvent<HTMLInputElement>) => {
setDurationString(event.target.value);
};
const handlePopoverOpen = (event: MouseEvent<HTMLSpanElement>) => {
setAnchorEl(event.currentTarget);
};
const handlePopoverClose = () => {
setAnchorEl(null);
};
const onKeyUp = (event: KeyboardEvent<HTMLInputElement>) => {
if (event.key !== "Enter") return;
const target = event.target as HTMLInputElement;
target.blur();
setDurationString(target.value);
};
useEffect(() => {
setDurationString(duration);
}, [duration]);
useEffect(() => {
if (!durationStringFocused) {
const value = checkDurationLimit(durationString);
setDurationString(value);
setDuration(value);
}
}, [durationString, durationStringFocused]);
return <>
<Box>
<TextField label="Duration" value={durationString} onChange={handleDurationChange}
variant="standard"
fullWidth={true}
onKeyUp={onKeyUp}
onBlur={() => {
setFocused(false);
}}
onFocus={() => {
setFocused(true);
}}
/>
</Box>
<Box mt={2}>
<Typography variant="body2">
<span aria-owns={open ? "mouse-over-popover" : undefined}
aria-haspopup="true"
style={{cursor: "pointer"}}
onMouseEnter={handlePopoverOpen}
onMouseLeave={handlePopoverClose}>
Possible options:&nbsp;
</span>
<Popover
open={open}
anchorEl={anchorEl}
anchorOrigin={{
vertical: "bottom",
horizontal: "left",
}}
transformOrigin={{
vertical: "top",
horizontal: "left",
}}
style={{pointerEvents: "none"}} // important
onClose={handlePopoverClose}
disableRestoreFocus
>
<TimeDurationPopover/>
</Popover>
<InlineBtn handler={() => setDurationString("5m")} text="5m"/>,&nbsp;
<InlineBtn handler={() => setDurationString("1h")} text="1h"/>,&nbsp;
<InlineBtn handler={() => setDurationString("1h 30m")} text="1h 30m"/>
</Typography>
</Box>
</>;
return <List style={{maxHeight: "168px", overflow: "auto", paddingRight: "15px"}}>
{durationOptions.map(d =>
<ListItem key={d.duration} button onClick={() => setDuration(d.duration, d.from ? d.from() : new Date())}>
<ListItemText primary={d.title || d.duration}/>
</ListItem>)}
</List>;
};
export default TimeDurationSelector;

166
app/vmui/packages/vmui/src/components/Home/Configurator/Time/TimeSelector.tsx
View file

@ -1,40 +1,32 @@
import React, {FC, useEffect, useState} from "preact/compat";
import Box from "@mui/material/Box";
import TextField from "@mui/material/TextField";
import Typography from "@mui/material/Typography";
import DateTimePicker from "@mui/lab/DateTimePicker";
import React, {FC, useEffect, useState, useMemo} from "preact/compat";
import {useAppDispatch, useAppState} from "../../../../state/common/StateContext";
import {dateFromSeconds, formatDateForNativeInput} from "../../../../utils/time";
import {InlineBtn} from "../../../common/InlineBtn";
import makeStyles from "@mui/styles/makeStyles";
import TimeDurationSelector from "./TimeDurationSelector";
import dayjs from "dayjs";
import QueryBuilderIcon from "@mui/icons-material/QueryBuilder";
import Box from "@mui/material/Box";
import TextField from "@mui/material/TextField";
import DateTimePicker from "@mui/lab/DateTimePicker";
import Button from "@mui/material/Button";
import Popper from "@mui/material/Popper";
import Paper from "@mui/material/Paper";
import Divider from "@mui/material/Divider";
import ClickAwayListener from "@mui/material/ClickAwayListener";
import Tooltip from "@mui/material/Tooltip";
interface TimeSelectorProps {
setDuration: (str: string) => void;
}
const formatDate = "YYYY-MM-DD HH:mm:ss";
const useStyles = makeStyles({
container: {
display: "grid",
gridTemplateColumns: "200px 1fr",
gridGap: "20px",
height: "100%",
gridTemplateColumns: "200px auto 200px",
gridGap: "10px",
padding: "20px",
borderRadius: "4px",
borderColor: "#b9b9b9",
borderStyle: "solid",
borderWidth: "1px"
},
timeControls: {
display: "grid",
gridTemplateColumns: "1fr",
gridTemplateRows: "auto 1fr",
gridGap: "16px 0",
},
datePickers: {
display: "grid",
gridTemplateColumns: "repeat(auto-fit, 200px)",
gridTemplateRows: "auto 1fr auto",
gridGap: "16px 0",
},
datePickerItem: {
@ -42,7 +34,7 @@ const useStyles = makeStyles({
},
});
export const TimeSelector: FC<TimeSelectorProps> = ({setDuration}) => {
export const TimeSelector: FC = () => {
const classes = useStyles();
@ -60,46 +52,88 @@ export const TimeSelector: FC<TimeSelectorProps> = ({setDuration}) => {
setFrom(formatDateForNativeInput(dateFromSeconds(start)));
}, [start]);
return <Box className={classes.container}>
{/*setup duration*/}
<Box>
<TimeDurationSelector setDuration={setDuration}/>
</Box>
{/*setup end time*/}
<Box className={classes.timeControls}>
<Box className={classes.datePickers}>
<Box className={classes.datePickerItem}>
<DateTimePicker
label="From"
ampm={false}
value={from}
onChange={date => dispatch({type: "SET_FROM", payload: date as unknown as Date})}
onError={console.log}
inputFormat="DD/MM/YYYY HH:mm:ss"
mask="__/__/____ __:__:__"
renderInput={(params) => <TextField {...params} variant="standard"/>}
maxDate={dayjs(until)}
/>
</Box>
<Box className={classes.datePickerItem}>
<DateTimePicker
label="Until"
ampm={false}
value={until}
onChange={date => dispatch({type: "SET_UNTIL", payload: date as unknown as Date})}
onError={console.log}
inputFormat="DD/MM/YYYY HH:mm:ss"
mask="__/__/____ __:__:__"
renderInput={(params) => <TextField {...params} variant="standard"/>}
/>
</Box>
</Box>
<Box>
<Typography variant="body2">
Will be changed to current time for auto-refresh mode.&nbsp;
<InlineBtn handler={() => dispatch({type: "RUN_QUERY_TO_NOW"})} text="Switch to now"/>
</Typography>
</Box>
</Box>
</Box>;
const setDuration = (dur: string, from: Date) => {
dispatch({type: "SET_UNTIL", payload: from});
setAnchorEl(null);
dispatch({type: "SET_DURATION", payload: dur});
};
const formatRange = useMemo(() => {
const startFormat = dayjs(dateFromSeconds(start)).format(formatDate);
const endFormat = dayjs(dateFromSeconds(end)).format(formatDate);
return {
start: startFormat,
end: endFormat
};
}, [start, end]);
const [anchorEl, setAnchorEl] = useState<HTMLButtonElement | null>(null);
const open = Boolean(anchorEl);
return <>
<Tooltip title="Time range controls">
<Button variant="contained" color="primary"
sx={{
color: "white",
border: "1px solid rgba(0, 0, 0, 0.2)",
boxShadow: "none"
}}
startIcon={<QueryBuilderIcon/>}
onClick={(e) => setAnchorEl(e.currentTarget)}>
{formatRange.start} - {formatRange.end}
</Button>
</Tooltip>
<Popper
open={open}
anchorEl={anchorEl}
placement="bottom-end"
modifiers={[{name: "offset", options: {offset: [0, 6]}}]}>
<ClickAwayListener onClickAway={() => setAnchorEl(null)}>
<Paper elevation={3}>
<Box className={classes.container}>
<Box className={classes.timeControls}>
<Box className={classes.datePickerItem}>
<DateTimePicker
label="From"
ampm={false}
value={from}
onChange={date => dispatch({type: "SET_FROM", payload: date as unknown as Date})}
onError={console.log}
inputFormat={formatDate}
mask="____-__-__ __:__:__"
renderInput={(params) => <TextField {...params} variant="standard"/>}
maxDate={dayjs(until)}
PopperProps={{disablePortal: true}}/>
</Box>
<Box className={classes.datePickerItem}>
<DateTimePicker
label="To"
ampm={false}
value={until}
onChange={date => dispatch({type: "SET_UNTIL", payload: date as unknown as Date})}
onError={console.log}
inputFormat={formatDate}
mask="____-__-__ __:__:__"
renderInput={(params) => <TextField {...params} variant="standard"/>}
PopperProps={{disablePortal: true}}/>
</Box>
<Box display="grid" gridTemplateColumns="auto 1fr" gap={1}>
<Button variant="outlined" onClick={() => setAnchorEl(null)}>
Cancel
</Button>
<Button variant="contained" onClick={() => dispatch({type: "RUN_QUERY_TO_NOW"})}>
switch to now
</Button>
</Box>
</Box>
{/*setup duration*/}
<Divider orientation="vertical" flexItem />
<Box>
<TimeDurationSelector setDuration={setDuration}/>
</Box>
</Box>
</Paper>
</ClickAwayListener>
</Popper>
</>;
};

28
app/vmui/packages/vmui/src/components/Home/HomeLayout.tsx
View file

@ -10,6 +10,8 @@ import QueryConfigurator from "./Configurator/Query/QueryConfigurator";
import {useFetchQuery} from "./Configurator/Query/useFetchQuery";
import JsonView from "./Views/JsonView";
import Header from "../Header/Header";
import {DisplayTypeSwitch} from "./Configurator/DisplayTypeSwitch";
import GraphSettings from "./Configurator/Graph/GraphSettings";
const HomeLayout: FC = () => {
@ -20,18 +22,16 @@ const HomeLayout: FC = () => {
return (
<Box id="homeLayout">
<Header/>
<Box p={4} display="grid" gridTemplateRows="auto 1fr" gap={"20px"} style={{minHeight: "calc(100vh - 64px)"}}>
<Box>
<QueryConfigurator error={error} queryOptions={queryOptions}/>
</Box>
<Box height={"100%"}>
<Box p={4} display="grid" gridTemplateRows="auto 1fr" style={{minHeight: "calc(100vh - 64px)"}}>
<QueryConfigurator error={error} queryOptions={queryOptions}/>
<Box height="100%">
{isLoading && <Fade in={isLoading} style={{
transitionDelay: isLoading ? "300ms" : "0ms",
}}>
<Box alignItems="center" justifyContent="center" flexDirection="column" display="flex"
style={{
width: "100%",
maxWidth: "calc(100vw - 32px)",
maxWidth: "calc(100vw - 64px)",
position: "absolute",
height: "50%",
background: "linear-gradient(rgba(255,255,255,.7), rgba(255,255,255,.7), rgba(255,255,255,0))"
@ -40,12 +40,16 @@ const HomeLayout: FC = () => {
</Box>
</Fade>}
{<Box height={"100%"} bgcolor={"#fff"}>
{error &&
<Alert color="error" severity="error" style={{fontSize: "14px", whiteSpace: "pre-wrap"}}>
{error}
</Alert>}
{graphData && period && (displayType === "chart") &&
<GraphView data={graphData}/>}
<Box display="grid" gridTemplateColumns="1fr auto" alignItems="center" mx={-4} px={4} mb={2}
borderBottom={1} borderColor="divider">
<DisplayTypeSwitch/>
{displayType === "chart" && <GraphSettings/>}
</Box>
{error && <Alert color="error" severity="error"
style={{fontSize: "14px", whiteSpace: "pre-wrap", marginTop: "20px"}}>
{error}
</Alert>}
{graphData && period && (displayType === "chart") && <GraphView data={graphData}/>}
{liveData && (displayType === "code") && <JsonView data={liveData}/>}
{liveData && (displayType === "table") && <TableView data={liveData}/>}
</Box>}

2
app/vmui/packages/vmui/src/components/Home/Views/GraphView.tsx
View file

@ -7,7 +7,6 @@ import {useGraphDispatch, useGraphState} from "../../../state/graph/GraphStateCo
import {getHideSeries, getLegendItem, getSeriesItem} from "../../../utils/uplot/series";
import {getLimitsYAxis, getTimeSeries} from "../../../utils/uplot/axes";
import {LegendItem} from "../../../utils/uplot/types";
import GraphSettings from "../Configurator/Graph/GraphSettings";
import {useAppState} from "../../../state/common/StateContext";
export interface GraphViewProps {
@ -82,7 +81,6 @@ const GraphView: FC<GraphViewProps> = ({data = []}) => {
return <>
{(data.length > 0)
? <div>
<GraphSettings/>
<LineChart data={dataChart} series={series} metrics={data}/>
<Legend labels={legend} onChange={onChangeLegend}/>
</div>

24
app/vmui/packages/vmui/src/components/Home/Views/JsonView.tsx
View file

@ -15,18 +15,24 @@ const JsonView: FC<JsonViewProps> = ({data}) => {
return (
<Box position="relative">
<Box flexDirection="column" justifyContent="flex-end" display="flex"
<Box
style={{
position: "fixed",
right: "16px"
position: "sticky",
top: "16px",
display: "flex",
justifyContent: "flex-end",
}}>
<Button variant="outlined" onClick={(e) => {
navigator.clipboard.writeText(formattedJson);
showInfoMessage("Formatted JSON has been copied");
e.preventDefault(); // needed to avoid snackbar immediate disappearing
}}>Copy JSON</Button>
<Button variant="outlined"
fullWidth={false}
onClick={(e) => {
navigator.clipboard.writeText(formattedJson);
showInfoMessage("Formatted JSON has been copied");
e.preventDefault(); // needed to avoid snackbar immediate disappearing
}}>
Copy JSON
</Button>
</Box>
<pre>{formattedJson}</pre>
<pre style={{margin: 0}}>{formattedJson}</pre>
</Box>
);
};

5
app/vmui/packages/vmui/src/components/Home/Views/TableView.tsx
View file

@ -1,7 +1,6 @@
import React, {FC, useMemo} from "preact/compat";
import {InstantMetricResult} from "../../../api/types";
import {InstantDataSeries} from "../../../types";
import Paper from "@mui/material/Paper";
import Table from "@mui/material/Table";
import TableBody from "@mui/material/TableBody";
import TableCell from "@mui/material/TableCell";
@ -37,7 +36,7 @@ const TableView: FC<GraphViewProps> = ({data}) => {
return (
<>
{(rows.length > 0)
? <TableContainer component={Paper}>
? <TableContainer>
<Table aria-label="simple table">
<TableHead>
<TableRow>
@ -48,7 +47,7 @@ const TableView: FC<GraphViewProps> = ({data}) => {
</TableHead>
<TableBody>
{rows.map((row, index) => (
<TableRow key={index}>
<TableRow key={index} hover>
{row.metadata.map((rowMeta, index2) => {
const prevRowValue = rows[index - 1] && rows[index - 1].metadata[index2];
return (

4
app/vmui/packages/vmui/src/components/LineChart/LineChart.tsx
View file

@ -4,7 +4,7 @@ import uPlot, {AlignedData as uPlotData, Options as uPlotOptions, Series as uPlo
import {useGraphState} from "../../state/graph/GraphStateContext";
import {defaultOptions} from "../../utils/uplot/helpers";
import {dragChart} from "../../utils/uplot/events";
import {getAxes} from "../../utils/uplot/axes";
import {getAxes, getMinMaxBuffer} from "../../utils/uplot/axes";
import {setTooltip} from "../../utils/uplot/tooltip";
import {MetricResult} from "../../api/types";
import {limitsDurations} from "../../utils/time";
@ -87,7 +87,7 @@ const LineChart: FC<LineChartProps> = ({data, series, metrics = []}) => {
const getRangeX = (): Range.MinMax => [xRange.min, xRange.max];
const getRangeY = (u: uPlot, min = 0, max = 1, axis: string): Range.MinMax => {
if (yaxis.limits.enable) return yaxis.limits.range[axis];
return min && max ? [min - (min * 0.25), max + (max * 0.25)] : [-1, 1];
return getMinMaxBuffer(min, max);
};
const getScales = (): Scales => {

26
app/vmui/packages/vmui/src/components/common/CircularProgressWithLabel.tsx
View file

@ -1,26 +0,0 @@
import Box from "@mui/material/Box";
import CircularProgress, {CircularProgressProps} from "@mui/material/CircularProgress";
import Typography from "@mui/material/Typography";
import React, {FC} from "preact/compat";
const CircularProgressWithLabel: FC<CircularProgressProps & { label: number }> = (props) => {
return (
<Box position="relative" display="inline-flex">
<CircularProgress variant="determinate" {...props} />
<Box
top={0}
left={0}
bottom={0}
right={0}
position="absolute"
display="flex"
alignItems="center"
justifyContent="center"
>
<Typography variant="caption" component="div">{`${props.label}s`}</Typography>
</Box>
</Box>
);
};
export default CircularProgressWithLabel;

19
app/vmui/packages/vmui/src/components/common/InlineBtn.tsx
View file

@ -1,19 +0,0 @@
import makeStyles from "@mui/styles/makeStyles";
import React from "preact/compat";
import Link from "@mui/material/Link";
const useStyles = makeStyles({
inlineBtn: {
"&:hover": {
cursor: "pointer"
},
}
});
export const InlineBtn: React.FC<{handler: () => void; text: string}> = ({handler, text}) => {
const classes = useStyles();
return <Link component="span" className={classes.inlineBtn}
onClick={handler}>
{text}
</Link>;
};

23
app/vmui/packages/vmui/src/hooks/useResize.ts
View file

@ -1,5 +1,26 @@
import { useState, useEffect } from "preact/compat";
const getScrollbarWidth = () => {
// Creating invisible container
const outer = document.createElement("div");
outer.style.visibility = "hidden";
outer.style.overflow = "scroll"; // forcing scrollbar to appear
document.body.appendChild(outer);
// Creating inner element and placing it in the container
const inner = document.createElement("div");
outer.appendChild(inner);
// Calculating difference between container's full width and the child width
const scrollbarWidth = (outer.offsetWidth - inner.offsetWidth);
// Removing temporary elements from the DOM
inner.remove();
outer.remove();
return scrollbarWidth;
};
const useResize = (node: HTMLElement | null): {width: number, height: number} => {
const [windowSize, setWindowSize] = useState({
width: 0,
@ -9,7 +30,7 @@ const useResize = (node: HTMLElement | null): {width: number, height: number} =>
if (!node) return;
const handleResize = () => {
setWindowSize({
width: node.offsetWidth,
width: node.offsetWidth - getScrollbarWidth(),
height: node.offsetHeight,
});
};

6
app/vmui/packages/vmui/src/state/common/reducer.ts
View file

@ -11,7 +11,7 @@ import {
} from "../../utils/time";
import {getFromStorage} from "../../utils/storage";
import {getDefaultServer} from "../../utils/default-server-url";
import {breakLineToQuery, getQueryArray, getQueryStringValue} from "../../utils/query-string";
import {getQueryArray, getQueryStringValue} from "../../utils/query-string";
import dayjs from "dayjs";
export interface TimeState {
@ -59,7 +59,7 @@ const query = getQueryArray();
export const initialState: AppState = {
serverUrl: getDefaultServer(),
displayType: getQueryStringValue("tab", "chart") as DisplayType,
displayType: getQueryStringValue("g0.tab", "chart") as DisplayType,
query: query, // demo_memory_usage_bytes
queryHistory: query.map(q => ({index: 0, values: [q]})),
time: {
@ -88,7 +88,7 @@ export function reducer(state: AppState, action: Action): AppState {
case "SET_QUERY":
return {
...state,
query: action.payload.map(q => breakLineToQuery(q))
query: action.payload.map(q => q)
};
case "SET_QUERY_HISTORY":
return {

30
app/vmui/packages/vmui/src/theme/theme.ts
View file

@ -49,17 +49,27 @@ const THEME = createTheme({
MuiAccordion: {
styleOverrides: {
root: {
boxShadow: "rgba(0, 0, 0, 0.16) 0px 1px 4px;"
boxShadow: "rgba(0, 0, 0, 0.16) 0px 1px 4px"
},
},
},
MuiPaper: {
styleOverrides: {
elevation3: {
boxShadow: "rgba(0, 0, 0, 0.2) 0px 3px 8px;"
root: {
boxShadow: "rgba(0, 0, 0, 0.2) 0px 3px 8px"
},
},
},
MuiButton: {
styleOverrides: {
contained: {
boxShadow: "rgba(17, 17, 26, 0.1) 0px 0px 16px",
"&:hover": {
boxShadow: "rgba(0, 0, 0, 0.1) 0px 4px 12px",
},
}
}
},
MuiIconButton: {
defaultProps: {
size: "large",
@ -77,6 +87,20 @@ const THEME = createTheme({
borderRadius: "20%",
}
}
},
MuiTooltip: {
styleOverrides: {
tooltip: {
fontSize: "10px"
}
}
},
MuiAlert: {
styleOverrides: {
root: {
boxShadow: "rgba(0, 0, 0, 0.08) 0px 4px 12px"
}
}
}
},
typography: {

6
app/vmui/packages/vmui/src/utils/query-string.ts
View file

@ -48,7 +48,7 @@ export const setQueryStringValue = (newValue: Record<string, unknown>): void =>
newQsValue.push(`g${i}.${queryKey}=${valueEncoded}`);
}
});
newQsValue.push(`g${i}.expr=${breakLineToQuery(q)}`);
newQsValue.push(`g${i}.expr=${encodeURIComponent(q)}`);
});
setQueryStringWithoutPageReload(newQsValue.join("&"));
@ -69,7 +69,3 @@ export const getQueryArray = (): string[] => {
return getQueryStringValue(`g${i}.expr`, "") as string;
});
};
export const breakLineToQuery = (q: string): string => q.replace(/\n/g, "%20");
export const queryToBreakLine = (q: string): string => q.replace(/%20/g, "\n");

12
app/vmui/packages/vmui/src/utils/uplot/axes.ts
View file

@ -19,13 +19,21 @@ export const getTimeSeries = (times: number[], defaultStep: number, period: Time
return new Array(length*2).fill(startTime).map((d, i) => roundTimeSeconds(d + (defaultStep * i)));
};
export const getMinMaxBuffer = (min: number, max: number): [number, number] => {
const minCorrect = isNaN(min) ? -1 : min;
const maxCorrect = isNaN(max) ? 1 : max;
const valueRange = Math.abs(maxCorrect - minCorrect) || Math.abs(minCorrect) || 1;
const padding = 0.02*valueRange;
return [minCorrect - padding, maxCorrect + padding];
};
export const getLimitsYAxis = (values: { [key: string]: number[] }): AxisRange => {
const result: AxisRange = {};
for (const key in values) {
const numbers = values[key];
const min = getMinFromArray(numbers);
const max = getMaxFromArray(numbers);
result[key] = min && max ? [min - (min * 0.25), max + (max * 0.25)] : [-1, 1];
result[key] = getMinMaxBuffer(min, max);
}
return result;
};
};

5
docs/CHANGELOG.md
View file

@ -30,6 +30,9 @@ sort: 15
* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): add `stripPort` template function in the same way as [Prometheus does](https://github.com/prometheus/prometheus/pull/10002).
* FEATURE: [vmalert](https://docs.victoriametrics.com/vmalert.html): add `parseDuration` template function in the same way as [Prometheus does](https://github.com/prometheus/prometheus/pull/8817).
* FEATURE: [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html): add `stale_samples_over_time(m[d])` function for calculating the number of [staleness marks](https://docs.victoriametrics.com/vmagent.html#prometheus-staleness-markers) for time series `m` over the duration `d`. This function may be useful for detecting flapping metrics at scrape targets, which periodically disappear and then appear again.
* FEATURE: [vmgateway](https://docs.victoriametrics.com/vmgateway.html): add support for `extra_filters` option. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1863).
* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): improve UX according to [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1960). Thanks to @Loori-R .
* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): limit the number of requests sent to VictoriaMetrics during zooming / scrolling. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2064).
* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): make sure that `vmagent` replicas scrape the same targets at different time offsets when [replication is enabled in vmagent clustering mode](https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets). This guarantees that the [deduplication](https://docs.victoriametrics.com/#deduplication) consistently leaves samples from the same `vmagent` replica.
* BUGFIX: return the proper response stub from `/api/v1/query_exemplars` handler, which is needed for Grafana v8+. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1999).
@ -37,6 +40,8 @@ sort: 15
* BUGFIX: fix possible data race when searching for time series matching `{key=~"value|"}` filter over time range covering multipe days. See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2032). Thanks to @waldoweng for the provided fix.
* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): do not send staleness markers on graceful shutdown. This follows Prometheus behavior. See [this comment](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2013#issuecomment-1006994079).
* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): properly set `__address__` label in `dockerswarm_sd_config`. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2038). Thanks to @ashtuchkin for the fix.
* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix incorrect calculations for graph limits on y axis. This could result in incorrect graph rendering in some cases. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2037).
* BUGFIX: [vmui](https://docs.victoriametrics.com/#vmui): fix handling for multi-line queries. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2039).
## [v1.71.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.71.0)

161
docs/FAQ.md
View file

@ -38,7 +38,7 @@ Yes. See [these docs](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.h
Yes in most cases. VictoriaMetrics can substitute Prometheus in the following aspects:
* Prometheus-compatible service discovery and target scraping can be done with [vmagent](https://docs.victoriametrics.com/vmagent.html) and with single-node VictoriaMetrics - see [these docs](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter).
* Prometheus-compatible service discovery and target scraping can be done with [vmagent](https://docs.victoriametrics.com/vmagent.html) and with single-node VictoriaMetrics. See [these docs](https://docs.victoriametrics.com/#how-to-scrape-prometheus-exporters-such-as-node-exporter).
* Prometheus-compatible alerting rules and recording rules can be processed with [vmalert](https://docs.victoriametrics.com/vmalert.html).
* Prometheus-compatible querying in Grafana is supported by VictoriaMetrics. See [these docs](https://docs.victoriametrics.com/#grafana-setup).
@ -49,13 +49,13 @@ While both [vmagent](https://docs.victoriametrics.com/vmagent.html) and Promethe
according to the provided Prometheus-compatible [scrape configs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config)
and send data to multiple remote storage systems, vmagent has the following additional features:
- vmagent usually requires lower amounts of CPU, RAM and disk IO comparing to Prometheus when scraping big number of targets (more than 1000)
or targets with big number of exposed metrics.
- vmagent provides independent disk-backed buffers per each configured remote storage (aka `-remoteWrite.url`). This means that slow or temporarily unavailable storage
doesn't prevent from sending data to healthy storage in parallel. Prometheus uses a single shared buffer for all the configured remote storage systems (aka `remote_write->url`)
with the hardcoded retention of 2 hours.
- vmagent may accept, relabel and filter data obtained via multiple data ingestion protocols additionally to data scraped from Prometheus targets.
I.e. it supports both `pull` and `push` protocols for data ingestion.
- vmagent usually requires lower amounts of CPU, RAM and disk IO compared to Prometheus when scraping an enormous number of targets (more than 1000)
or targets with a great number of exposed metrics.
- vmagent provides independent disk-backed buffers for each configured remote storage (see `-remoteWrite.url`). This means that slow or temporarily unavailable storage
doesn't prevent it from sending data to healthy storage in parallel. Prometheus uses a single shared buffer for all the configured remote storage systems (see `remote_write->url`)
with a hardcoded retention of 2 hours.
- vmagent may accept, relabel and filter data obtained via multiple data ingestion protocols in addition to data scraped from Prometheus targets.
That means it supports both `pull` and `push` protocols for data ingestion.
See [these docs](https://docs.victoriametrics.com/vmagent.html#features) for details.
- vmagent may be used in different use cases:
- [IoT and edge monitoring](https://docs.victoriametrics.com/vmagent.html#iot-and-edge-monitoring)
@ -68,16 +68,16 @@ and send data to multiple remote storage systems, vmagent has the following addi
## What is the difference between vmagent and Prometheus agent?
Both [vmagent](https://docs.victoriametrics.com/vmagent.html) and [Prometheus agent](https://prometheus.io/blog/2021/11/16/agent/) serve the same purpose - to efficently scrape Prometheus-compatible targets at the edge. The have the following differences:
Both [vmagent](https://docs.victoriametrics.com/vmagent.html) and [Prometheus agent](https://prometheus.io/blog/2021/11/16/agent/) serve the same purpose – to efficently scrape Prometheus-compatible targets at the edge. They have the following differences:
- vmagent usually requires lower amounts of CPU, RAM and disk IO comparing to Prometheus agent.
- Prometheus agent supports only pull-based data collection (e.g. it can scrape Prometheus-compatible targets), while vmagent supports both pull and push data collection - it can accept data via many popular data ingestion protocols such as InfluxDB line protocol, Graphite protocol, OpenTSDB protocol, DataDog protocol, Prometheus protocol, CSV and JSON - see [these docs](https://docs.victoriametrics.com/vmagent.html#features).
- vmagent can easily scale horizontally to multiple instances for scraping big number of targets - see [these docs](https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets).
- vmagent usually requires lower amounts of CPU, RAM and disk IO compared to the Prometheus agent.
- Prometheus agent supports only pull-based data collection (e.g. it can scrape Prometheus-compatible targets), while vmagent supports both pull and push data collection – it can accept data via many popular data ingestion protocols such as InfluxDB line protocol, Graphite protocol, OpenTSDB protocol, DataDog protocol, Prometheus protocol, CSV and JSON – see [these docs](https://docs.victoriametrics.com/vmagent.html#features).
- vmagent can easily scale horizontally to multiple instances for scraping a big number of targets – see [these docs](https://docs.victoriametrics.com/vmagent.html#scraping-big-number-of-targets).
- vmagent supports [improved relabeling](https://docs.victoriametrics.com/vmagent.html#relabeling).
- vmagent can limit the number of scraped metrics per target. See [these docs](https://docs.victoriametrics.com/vmagent.html#cardinality-limiter).
- vmagent supports loading scrape configs from multiple files - see [these docs](https://docs.victoriametrics.com/vmagent.html#loading-scrape-configs-from-multiple-files).
- vmagent supports data reading and data writing from/to Kafka - see [these docs](https://docs.victoriametrics.com/vmagent.html#kafka-integration).
- vmagent can read and update scrape configs from http and https urls, while Prometheus agent can read scrape configs only from local filesystem.
- vmagent can limit the number of scraped metrics per target – see [these docs](https://docs.victoriametrics.com/vmagent.html#cardinality-limiter).
- vmagent supports loading scrape configs from multiple files – see [these docs](https://docs.victoriametrics.com/vmagent.html#loading-scrape-configs-from-multiple-files).
- vmagent supports data reading and data writing from/to Kafka – see [these docs](https://docs.victoriametrics.com/vmagent.html#kafka-integration).
- vmagent can read and update scrape configs from http and https URLs, while the Prometheus agent can read them only from the local file system.
## Is it safe to enable [remote write](https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage) in Prometheus?
@ -92,14 +92,14 @@ and writing data to VictoriaMetrics.
## How does VictoriaMetrics compare to other remote storage solutions for Prometheus such as [M3 from Uber](https://eng.uber.com/m3/), [Thanos](https://github.com/thanos-io/thanos), [Cortex](https://github.com/cortexproject/cortex), etc.?
VictoriaMetrics is simpler, faster, more cost-effective and it provides [MetricsQL query language](MetricsQL) based on PromQL. The simplicity is twofold:
- It is simpler to configure and operate. There is no need in configuring [sidecars](https://github.com/thanos-io/thanos/blob/master/docs/components/sidecar.md),
fighting [gossip protocol](https://github.com/improbable-eng/thanos/blob/030bc345c12c446962225221795f4973848caab5/docs/proposals/completed/201809_gossip-removal.md)
- It is simpler to configure and operate. There is no need for configuring [sidecars](https://github.com/thanos-io/thanos/blob/master/docs/components/sidecar.md),
fighting the [gossip protocol](https://github.com/improbable-eng/thanos/blob/030bc345c12c446962225221795f4973848caab5/docs/proposals/completed/201809_gossip-removal.md)
or setting up third-party systems such as [Consul](https://github.com/cortexproject/cortex/issues/157), [Cassandra](https://cortexmetrics.io/docs/production/cassandra/),
[DynamoDB](https://cortexmetrics.io/docs/production/aws/) or [Memcached](https://cortexmetrics.io/docs/production/caching/).
- VictoriaMetrics has simpler architecture. This means less bugs and more useful features in the long run comparing to competing TSDBs.
- VictoriaMetrics has a simpler architecture. This means fewer bugs and more useful features in the long run compared to competing TSDBs.
See [comparing Thanos to VictoriaMetrics cluster](https://medium.com/@valyala/comparing-thanos-to-victoriametrics-cluster-b193bea1683)
and [Remote Write Storage Wars](https://promcon.io/2019-munich/talks/remote-write-storage-wars/) talk from [PromCon 2019](https://promcon.io/2019-munich/talks/remote-write-storage-wars/).
and the [Remote Write Storage Wars](https://promcon.io/2019-munich/talks/remote-write-storage-wars/) talk from [PromCon 2019](https://promcon.io/2019-munich/talks/remote-write-storage-wars/).
VictoriaMetrics also [uses less RAM than Thanos components](https://github.com/thanos-io/thanos/issues/448).
@ -107,12 +107,12 @@ VictoriaMetrics also [uses less RAM than Thanos components](https://github.com/t
## What is the difference between VictoriaMetrics and [QuestDB](https://questdb.io/)?
- QuestDB needs more than 20x storage space than VictoriaMetrics. This translates to higher storage costs and slower queries over historical data, which must be read from the disk.
- QuestDB is much harder to setup and operate than VictoriaMetrics. Compare [setup instructions for QuestDB](https://questdb.io/docs/get-started/binaries) to [setup instructions for VictoriaMetrics](https://docs.victoriametrics.com/#how-to-start-victoriametrics).
- VictoriaMetrics provides [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html) query language, which is better suited for typical queries over time series data than SQL-like query language provided by QuestDB. See [this article](https://valyala.medium.com/promql-tutorial-for-beginners-9ab455142085) for details.
- VictoriaMetrics can be queried via [Prometheus querying API](https://docs.victoriametrics.com/#prometheus-querying-api-usage) and via [Graphite API](https://docs.victoriametrics.com/#graphite-api-usage).
- Thanks to PromQL support, VictoriaMetrics [can be used as a drop-in replacement for Prometheus in Grafana](https://docs.victoriametrics.com/#grafana-setup), while QuestDB needs full rewrite of existing dashboards in Grafana.
- Thanks to Prometheus remote_write API support, VictoriaMetrics can be used as a long-term storage for Prometheus or for [vmagent](https://docs.victoriametrics.com/vmagent.html), while QuestDB has no integration with Prometheus.
- QuestDB [supports smaller range of popular data ingestion protocols](https://questdb.io/docs/develop/insert-data) compared to VictoriaMetrics (compare to [the list of supported data ingestion protocols for VictoriaMetrics](https://docs.victoriametrics.com/#how-to-import-time-series-data)).
- QuestDB is much harder to set up and operate than VictoriaMetrics. Compare [setup instructions for QuestDB](https://questdb.io/docs/get-started/binaries) to [setup instructions for VictoriaMetrics](https://docs.victoriametrics.com/#how-to-start-victoriametrics).
- VictoriaMetrics provides the [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html) query language, which is better suited for typical queries over time series data than the SQL-like query language provided by QuestDB. See [this article](https://valyala.medium.com/promql-tutorial-for-beginners-9ab455142085) for details.
- VictoriaMetrics can be queried via the [Prometheus querying API](https://docs.victoriametrics.com/#prometheus-querying-api-usage) and via [Graphite's API](https://docs.victoriametrics.com/#graphite-api-usage).
- Thanks to PromQL support, VictoriaMetrics [can be used as a drop-in replacement for Prometheus in Grafana](https://docs.victoriametrics.com/#grafana-setup), while QuestDB needs a full rewrite of existing dashboards in Grafana.
- Thanks to Prometheus' remote_write API support, VictoriaMetrics can be used as a long-term storage for Prometheus or for [vmagent](https://docs.victoriametrics.com/vmagent.html), while QuestDB has no integration with Prometheus.
- QuestDB [supports a smaller range of popular data ingestion protocols](https://questdb.io/docs/develop/insert-data) compared to VictoriaMetrics (compare to [the list of supported data ingestion protocols for VictoriaMetrics](https://docs.victoriametrics.com/#how-to-import-time-series-data)).
- [VictoriaMetrics supports backfilling (e.g. storing historical data) out of the box](https://docs.victoriametrics.com/#backfilling), while QuestDB provides [very limited support for backfilling](https://questdb.io/blog/2021/05/10/questdb-release-6-0-tsbs-benchmark#the-problem-with-out-of-order-data).
@ -121,79 +121,79 @@ VictoriaMetrics also [uses less RAM than Thanos components](https://github.com/t
VictoriaMetrics is similar to Cortex in the following aspects:
- Both systems accept data from [vmagent](https://docs.victoriametrics.com/vmagent.html) or Prometheus
via standard [remote_write API](https://prometheus.io/docs/practices/remote_write/), i.e. there is no need in running sidecars
unlike in [Thanos](https://github.com/thanos-io/thanos) case.
via the standard [remote_write API](https://prometheus.io/docs/practices/remote_write/), so there is no need for running sidecars
unlike in [Thanos](https://github.com/thanos-io/thanos)' case.
- Both systems support multi-tenancy out of the box. See [the corresponding docs for VictoriaMetrics](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multitenancy).
- Both systems support data replication. See [replication in Cortex](https://github.com/cortexproject/cortex/blob/fe56f1420099aa1bf1ce09316c186e05bddee879/docs/architecture.md#hashing) and [replication in VictoriaMetrics](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#replication-and-data-safety).
- Both systems scale horizontally to multiple nodes. See [these docs](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#cluster-resizing-and-scalability) for details.
- Both systems support alerting and recording rules via the corresponding tools such as [vmalert](https://docs.victoriametrics.com/vmalert.html).
- Both systems can be queried via [Prometheus querying API](https://prometheus.io/docs/prometheus/latest/querying/api/) and integrate perfectly with Grafana.
- Both systems can be queried via the [Prometheus querying API](https://prometheus.io/docs/prometheus/latest/querying/api/) and integrate perfectly with Grafana.
The main differences between Cortex and VictoriaMetrics:
- Cortex re-uses Prometheus source code, while VictoriaMetrics is written from scratch.
- Cortex heavily relies on third-party services such as Consul, Memcache, DynamoDB, BigTable, Cassandra, etc.
This may increase operational complexity and reduce system reliability comparing to VictoriaMetrics' case,
which doesn't use any external services. Compare [Cortex Architecture](https://github.com/cortexproject/cortex/blob/master/docs/architecture.md)
to [VictoriaMetrics architecture](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#architecture-overview).
This may increase operational complexity and reduce system reliability compared to VictoriaMetrics' case,
which doesn't use any external services. Compare [Cortex' Architecture](https://github.com/cortexproject/cortex/blob/master/docs/architecture.md)
to [VictoriaMetrics' architecture](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#architecture-overview).
- VictoriaMetrics provides [production-ready single-node solution](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html),
which is much easier to setup and operate than Cortex cluster.
- Cortex may lose up to 12 hours of recent data on Ingestor failure - see [the corresponding docs](https://github.com/cortexproject/cortex/blob/fe56f1420099aa1bf1ce09316c186e05bddee879/docs/architecture.md#ingesters-failure-and-data-loss).
which is much easier to set up and operate than a Cortex cluster.
- Cortex may lose up to 12 hours of recent data on Ingestor failure – see [the corresponding docs](https://github.com/cortexproject/cortex/blob/fe56f1420099aa1bf1ce09316c186e05bddee879/docs/architecture.md#ingesters-failure-and-data-loss).
VictoriaMetrics may lose only a few seconds of recent data, which isn't synced to persistent storage yet.
See [this article for details](https://medium.com/@valyala/wal-usage-looks-broken-in-modern-time-series-databases-b62a627ab704).
- Cortex is usually slower and requires more CPU and RAM than VictoriaMetrics. See [this talk from adidas at PromCon 2019](https://promcon.io/2019-munich/talks/remote-write-storage-wars/) and [other case studies](https://docs.victoriametrics.com/CaseStudies.html).
- VictoriaMetrics accepts data in multiple popular data ingestion protocols additionally to Prometheus remote_write protocol - InfluxDB, OpenTSDB, Graphite, CSV, JSON, native binary.
- VictoriaMetrics accepts data in multiple popular data ingestion protocols additionally to Prometheus remote_write protocol – InfluxDB, OpenTSDB, Graphite, CSV, JSON, native binary.
See [these docs](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-import-time-series-data) for details.
- VictoriaMetrics provides [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html) query language, while Cortex provides [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) query language.
- VictoriaMetrics can be queried via [Graphite API](https://docs.victoriametrics.com/#graphite-api-usage).
- VictoriaMetrics provides the [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html) query language, while Cortex provides the [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) query language.
- VictoriaMetrics can be queried via [Graphite's API](https://docs.victoriametrics.com/#graphite-api-usage).
## What is the difference between VictoriaMetrics and [Thanos](https://github.com/thanos-io/thanos)?
- Thanos re-uses Prometheus source code, while VictoriaMetrics is written from scratch.
- VictoriaMetrics accepts data via [standard remote_write API for Prometheus](https://prometheus.io/docs/practices/remote_write/),
while Thanos uses non-standard [Sidecar](https://github.com/thanos-io/thanos/blob/master/docs/components/sidecar.md), which must run alongside each Prometheus instance.
- Thanos Sidecar requires disabling data compaction in Prometheus, which may hurt Prometheus performance and increase RAM usage. See [these docs](https://thanos.io/components/sidecar.md/) for more details.
- VictoriaMetrics accepts data via the [standard remote_write API for Prometheus](https://prometheus.io/docs/practices/remote_write/),
while Thanos uses a non-standard [sidecar](https://github.com/thanos-io/thanos/blob/master/docs/components/sidecar.md) which must run alongside each Prometheus instance.
- The Thanos sidecar requires disabling data compaction in Prometheus, which may hurt Prometheus performance and increase RAM usage. See [these docs](https://thanos.io/components/sidecar.md/) for more details.
- Thanos stores data in object storage (Amazon S3 or Google GCS), while VictoriaMetrics stores data in block storage
([GCP persistent disks](https://cloud.google.com/compute/docs/disks#pdspecs), Amazon EBS or bare metal HDD).
While object storage is usually less expensive, block storage provides much lower latencies and higher throughput.
VictoriaMetrics works perfectly with HDD-based block storage - there is no need in using more expensive SSD or NVMe disks in most cases.
VictoriaMetrics works perfectly with HDD-based block storage – there is no need for using more expensive SSD or NVMe disks in most cases.
- Thanos may lose up to 2 hours of recent data, which wasn't uploaded yet to object storage. VictoriaMetrics may lose only a few seconds of recent data,
which isn't synced to persistent storage yet. See [this article for details](https://medium.com/@valyala/wal-usage-looks-broken-in-modern-time-series-databases-b62a627ab704).
- VictoriaMetrics provides [production-ready single-node solution](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html),
which is much easier to setup and operate than Thanos components.
- Thanos may be harder to setup and operate comparing to VictoriaMetrics, since it has more moving parts, which can be connected with less reliable networks.
which hasn't been synced to persistent storage yet. See [this article for details](https://medium.com/@valyala/wal-usage-looks-broken-in-modern-time-series-databases-b62a627ab704).
- VictoriaMetrics provides a [production-ready single-node solution](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html),
which is much easier to set up and operate than Thanos components.
- Thanos may be harder to set up and operate compared to VictoriaMetrics, since it has more moving parts, which can be connected with fewer reliable networks.
See [this article for details](https://medium.com/faun/comparing-thanos-to-victoriametrics-cluster-b193bea1683).
- Thanos is usually slower and requires more CPU and RAM than VictoriaMetrics. See [this talk from adidas at PromCon 2019](https://promcon.io/2019-munich/talks/remote-write-storage-wars/).
- VictoriaMetrics accepts data in multiple popular data ingestion protocols additionally to Prometheus remote_write protocol - InfluxDB, OpenTSDB, Graphite, CSV, JSON, native binary.
- VictoriaMetrics accepts data via multiple popular data ingestion protocols in addition to the Prometheus remote_write protocol – InfluxDB, OpenTSDB, Graphite, CSV, JSON, native binary.
See [these docs](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-import-time-series-data) for details.
- VictoriaMetrics provides [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html) query language, while Thanos provides [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) query language.
- VictoriaMetrics can be queried via [Graphite API](https://docs.victoriametrics.com/#graphite-api-usage).
- VictoriaMetrics provides the [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html) query language, while Thanos provides the [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) query language.
- VictoriaMetrics can be queried via [Graphite's API](https://docs.victoriametrics.com/#graphite-api-usage).
## How does VictoriaMetrics compare to [InfluxDB](https://www.influxdata.com/time-series-platform/influxdb/)?
- VictoriaMetrics requires [10x less RAM](https://medium.com/@valyala/insert-benchmarks-with-inch-influxdb-vs-victoriametrics-e31a41ae2893) and it [works faster](https://medium.com/@valyala/measuring-vertical-scalability-for-time-series-databases-in-google-cloud-92550d78d8ae).
- VictoriaMetrics needs lower amounts of storage space than InfluxDB on production data.
- VictoriaMetrics provides better query language - [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html) - than InfluxQL or Flux. See [this tutorial](https://medium.com/@valyala/promql-tutorial-for-beginners-9ab455142085) for details.
- VictoriaMetrics accepts data in multiple popular data ingestion protocols additionally to InfluxDB - Prometheus remote_write, OpenTSDB, Graphite, CSV, JSON, native binary.
- VictoriaMetrics needs lower amounts of storage space than InfluxDB for production data.
- VictoriaMetrics provides a better query language – [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html) – than InfluxQL or Flux. See [this tutorial](https://medium.com/@valyala/promql-tutorial-for-beginners-9ab455142085) for details.
- VictoriaMetrics accepts data in multiple popular data ingestion protocols in addition to InfluxDB – Prometheus remote_write, OpenTSDB, Graphite, CSV, JSON, native binary.
See [these docs](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-import-time-series-data) for details.
- VictoriaMetrics can be queried via [Graphite API](https://docs.victoriametrics.com/#graphite-api-usage).
- VictoriaMetrics can be queried via [Graphite's API](https://docs.victoriametrics.com/#graphite-api-usage).
## How does VictoriaMetrics compare to [TimescaleDB](https://www.timescale.com/)?
- TimescaleDB insists on using SQL as a query language. While SQL is more powerful than PromQL, this power is rarely required during typical TSDB usage. Real-world queries usually [look clearer and simpler when written in PromQL than in SQL](https://medium.com/@valyala/promql-tutorial-for-beginners-9ab455142085).
- VictoriaMetrics requires [up to 70x less storage space comparing to TimescaleDB](https://medium.com/@valyala/when-size-matters-benchmarking-victoriametrics-vs-timescale-and-influxdb-6035811952d4) for storing the same amount of time series data. The gap in storage space usage can be lowered from 70x to 3x if [compression in TimescaleDB is properly configured](https://docs.timescale.com/latest/using-timescaledb/compression) (it isn't an easy task in general case :)).
- TimescaleDB insists on using SQL as a query language. While SQL is more powerful than PromQL, this power is rarely required during typical usages of a TSDB. Real-world queries usually [look clearer and simpler when written in PromQL than in SQL](https://medium.com/@valyala/promql-tutorial-for-beginners-9ab455142085).
- VictoriaMetrics requires [up to 70x less storage space compared to TimescaleDB](https://medium.com/@valyala/when-size-matters-benchmarking-victoriametrics-vs-timescale-and-influxdb-6035811952d4) for storing the same amount of time series data. The gap in storage space usage can be lowered from 70x to 3x if [compression in TimescaleDB is properly configured](https://docs.timescale.com/latest/using-timescaledb/compression) (it isn't an easy task in general :)).
- VictoriaMetrics requires up to 10x less CPU and RAM resources than TimescaleDB for processing production data. See [this article](https://abiosgaming.com/press/high-cardinality-aggregations/) for details.
- TimescaleDB is [harder to setup, configure and operate](https://docs.timescale.com/timescaledb/latest/how-to-guides/install-timescaledb/self-hosted/ubuntu/installation-apt-ubuntu/) than VictoriaMetrics (see [how to run VictoriaMetrics](https://docs.victoriametrics.com/#how-to-start-victoriametrics)).
- VictoriaMetrics accepts data in multiple popular data ingestion protocols - InfluxDB, OpenTSDB, Graphite, CSV, while TimescaleDB supports only SQL inserts.
- VictoriaMetrics can be queried via [Graphite API](https://docs.victoriametrics.com/#graphite-api-usage).
- TimescaleDB is [harder to set up, configure and operate](https://docs.timescale.com/timescaledb/latest/how-to-guides/install-timescaledb/self-hosted/ubuntu/installation-apt-ubuntu/) than VictoriaMetrics (see [how to run VictoriaMetrics](https://docs.victoriametrics.com/#how-to-start-victoriametrics)).
- VictoriaMetrics accepts data in multiple popular data ingestion protocols – InfluxDB, OpenTSDB, Graphite, CSV – while TimescaleDB supports only SQL inserts.
- VictoriaMetrics can be queried via [Graphite's API](https://docs.victoriametrics.com/#graphite-api-usage).
## Does VictoriaMetrics use Prometheus technologies like other clustered TSDBs built on top of Prometheus such as [Thanos](https://github.com/thanos-io/thanos) or [Cortex](https://github.com/cortexproject/cortex)?
No. VictoriaMetrics core is written in Go from scratch by [fasthttp](https://github.com/valyala/fasthttp) [author](https://github.com/valyala).
No. VictoriaMetrics core is written in Go from scratch by [fasthttp](https://github.com/valyala/fasthttp)'s [author](https://github.com/valyala).
The architecture is [optimized for storing and querying large amounts of time series data with high cardinality](https://medium.com/devopslinks/victoriametrics-creating-the-best-remote-storage-for-prometheus-5d92d66787ac). VictoriaMetrics storage uses [certain ideas from ClickHouse](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282). Special thanks to [Alexey Milovidov](https://github.com/alexey-milovidov).
@ -215,14 +215,14 @@ The following commercial versions of VictoriaMetrics are planned:
[Contact us](mailto:info@victoriametrics.com) for more information on our plans.
## Why VictoriaMetrics doesn't support [Prometheus remote read API](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cremote_read%3E)?
## Why doesn't VictoriaMetrics support the [Prometheus remote read API](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cremote_read%3E)?
Remote read API requires transferring all the raw data for all the requested metrics over the given time range. For instance,
if a query covers 1000 metrics with 10K values each, then the remote read API had to return `1000*10K`=10M metric values to Prometheus.
The remote read API requires transferring all the raw data for all the requested metrics over the given time range. For instance,
if a query covers 1000 metrics with 10K values each, then the remote read API has to return `1000*10K`=10M metric values to Prometheus.
This is slow and expensive.
Prometheus remote read API isn't intended for querying foreign data aka `global query view`. See [this issue](https://github.com/prometheus/prometheus/issues/4456) for details.
Prometheus' remote read API isn't intended for querying foreign data – aka `global query view`. See [this issue](https://github.com/prometheus/prometheus/issues/4456) for details.
So just query VictoriaMetrics directly via [vmui](https://docs.victoriametrics.com/#vmui), [Prometheus Querying API](https://docs.victoriametrics.com/#prometheus-querying-api-usage)
So just query VictoriaMetrics directly via [vmui](https://docs.victoriametrics.com/#vmui), the [Prometheus Querying API](https://docs.victoriametrics.com/#prometheus-querying-api-usage)
or via [Prometheus datasource in Grafana](https://docs.victoriametrics.com/#grafana-setup).
@ -238,7 +238,7 @@ Source code for the following versions is available in the following places:
* [Cluster version](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/cluster)
## Does VictoriaMetrics fit for data from IoT sensors and industrial sensors?
## Is VictoriaMetrics a good fit for data from IoT sensors and industrial sensors?
VictoriaMetrics is able to handle data from hundreds of millions of IoT sensors and industrial sensors.
It supports [high cardinality data](https://medium.com/@valyala/high-cardinality-tsdb-benchmarks-victoriametrics-vs-timescaledb-vs-influxdb-13e6ee64dd6b),
@ -260,14 +260,14 @@ Questions about VictoriaMetrics can be asked via the following channels:
File bugs and feature requests [here](https://github.com/VictoriaMetrics/VictoriaMetrics/issues).
## Where I can find information about multi-tenancy?
## Where can I find information about multi-tenancy?
See [these docs](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multitenancy). Multitenancy is supported only by [cluster version](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html) of VictoriaMetrics.
See [these docs](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#multitenancy). Multitenancy is supported only by the [cluster version](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html) of VictoriaMetrics.
## How to set a memory limit for VictoriaMetrics components?
All the VictoriaMetrics component provide command-line flags to control the size of internal buffers and caches: `-memory.allowedPercent` and `-memory.allowedBytes` (pass `-help` to any VictoriaMetrics component in order to see the description for these flags). These limits don't take into account additional memory, which may be needed for processing incoming queries. Hard limits may be enforced only by the OS via [cgroups](https://en.wikipedia.org/wiki/Cgroups), Docker (see [these docs](https://docs.docker.com/config/containers/resource_constraints)) or Kubernetes (see [these docs](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers)).
All the VictoriaMetrics components provide command-line flags to control the size of internal buffers and caches: `-memory.allowedPercent` and `-memory.allowedBytes` (pass `-help` to any VictoriaMetrics component in order to see the description for these flags). These limits don't take into account additional memory, which may be needed for processing incoming queries. Hard limits may be enforced only by the OS via [cgroups](https://en.wikipedia.org/wiki/Cgroups), Docker (see [these docs](https://docs.docker.com/config/containers/resource_constraints)) or Kubernetes (see [these docs](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers)).
Memory usage for VictoriaMetrics components can be tuned according to the following docs:
@ -282,22 +282,23 @@ Memory usage for VictoriaMetrics components can be tuned according to the follow
VictoriaMetrics is included in FreeBSD ports, so just install it from there. See [this link](https://www.freebsd.org/cgi/ports.cgi?query=victoria&stype=all).
## Does VictoriaMetrics support Graphite query language?
## Does VictoriaMetrics support the Graphite query language?
Yes. See [these docs](https://docs.victoriametrics.com/#graphite-api-usage).
## What is active time series?
## What is an active time series?
A time series is uniquely identified by its name plus a set of its labels. For example, `temperature{city="NY",country="US"}` and `temperature{city="SF",country="US"}` are two distinct series, since they differ by `city` label. A time series is considered active if it receives at least a single new sample during the last hour.
A time series is uniquely identified by its name plus a set of its labels. For example, `temperature{city="NY",country="US"}` and `temperature{city="SF",country="US"}` are two distinct series, since they differ by the `city` label. A time series is considered active if it receives at least a single new sample during the last hour.
## What is high churn rate?
If old time series are constantly substituted by new time series at a high rate, then such a state is called `high churn rate`. High churn rate has the following negative consequences:
* Increased total number of time series stored in the database.
* Increased size of inverted index, which is stored at `<-storageDataPath>/indexdb`, since the inverted index contains entries for every label of every time series with at least a single ingested sample
* Slow down of queries over multiple days.
* Increased size of inverted index, which is stored at `<-storageDataPath>/indexdb`, since the inverted index contains entries for every label of every time series with at least a single ingested sample.
* Slow-down of queries over multiple days.
The main reason for high churn rate is a metric label with frequently changed value. Examples of such labels:
@ -311,12 +312,12 @@ The solution against high churn rate is to identify and eliminate labels with fr
## What is high cardinality?
High cardinality usually means high number of [active time series](#what-is-active-time-series). High cardinality may lead to high memory usage and/or to high percentage of [slow inserts](#what-is-slow-insert). The source of high cardinality is usually a label with big number of unique values, which presents in big share of the ingested time series. The solution is to identify and remove the source of high cardinality with the help of [/api/v1/status/tsdb](https://docs.victoriametrics.com/#tsdb-stats).
High cardinality usually means a high number of [active time series](#what-is-active-time-series). High cardinality may lead to high memory usage and/or to a high percentage of [slow inserts](#what-is-slow-insert). The source of high cardinality is usually a label with a large number of unique values, which presents a big share of the ingested time series. The solution is to identify and remove the source of high cardinality with the help of [/api/v1/status/tsdb](https://docs.victoriametrics.com/#tsdb-stats).
## What is slow insert?
## What is a slow insert?
VictoriaMetrics maintains in-memory cache for mapping of [active time series](#what-is-active-time-series) into internal series ids. The cache size depends on the available memory for VictoriaMetrics in the host system. If the information about all the active time series doesn't fit the cache, then VictoriaMetrics needs to read and unpack the information from disk on every incoming sample for time series missing in the cache. This operation is much slower than the cache lookup, so such insert is named `slow insert`. High percentage of slow inserts on the [official dashboard for VictoriaMetrics](https://docs.victoriametrics.com/#monitoring) indicates on memory shortage for the current number of [active time series](#what-is-active-time-series). Such a condition usually leads to significant slowdown for data ingestion, to significantly increased disk IO and CPU usage. The solution is to add more memory or to reduce the number of [active time series](#what-is-active-time-series). The `/api/v1/status/tsdb` page can be helpful for locating the source of high number of active time seriess - see [these docs](https://docs.victoriametrics.com/#tsdb-stats).
VictoriaMetrics maintains in-memory cache for mapping of [active time series](#what-is-active-time-series) into internal series ids. The cache size depends on the available memory for VictoriaMetrics in the host system. If the information about all the active time series doesn't fit the cache, then VictoriaMetrics needs to read and unpack the information from disk on every incoming sample for time series missing in the cache. This operation is much slower than the cache lookup, so such an insert is named a `slow insert`. A high percentage of slow inserts on the [official dashboard for VictoriaMetrics](https://docs.victoriametrics.com/#monitoring) indicates a memory shortage for the current number of [active time series](#what-is-active-time-series). Such a condition usually leads to a significant slowdown for data ingestion and to significantly increased disk IO and CPU usage. The solution is to add more memory or to reduce the number of [active time series](#what-is-active-time-series). The `/api/v1/status/tsdb` page can be helpful for locating the source of high number of active time seriess – see [these docs](https://docs.victoriametrics.com/#tsdb-stats).
## How to optimize MetricsQL query?
@ -324,7 +325,7 @@ VictoriaMetrics maintains in-memory cache for mapping of [active time series](#w
See [this article](https://valyala.medium.com/how-to-optimize-promql-and-metricsql-queries-85a1b75bf986).
## Why MetricsQL isn't 100% compatible with PromQL?
## Why isn't MetricsQL 100% compatible with PromQL?
[MetricsQL](https://docs.victoriametrics.com/MetricsQL.html) provides better user experience than PromQL. It fixes a few annoying issues in PromQL. This prevents MetricsQL to be 100% compatible with PromQL. See [this article](https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e) for details.
@ -346,12 +347,12 @@ Please see [these docs](https://docs.victoriametrics.com/vmctl.html#migrating-da
## How to migrate data from Graphite to VictoriaMetrics?
Please use [whisper-to-graphite](https://github.com/bzed/whisper-to-graphite) tool for reading the data from Graphite and pushing it to VictoriaMetrics via [Graphite import API](https://docs.victoriametrics.com/#how-to-send-data-from-graphite-compatible-agents-such-as-statsd).
Please use the [whisper-to-graphite](https://github.com/bzed/whisper-to-graphite) tool for reading data from Graphite and pushing them to VictoriaMetrics via [Graphite's import API](https://docs.victoriametrics.com/#how-to-send-data-from-graphite-compatible-agents-such-as-statsd).
## Why do same metrics have differences in VictoriaMetrics and Prometheus dashboards?
## Why do the same metrics have differences in VictoriaMetrics' and Prometheus' dashboards?
There could be a slight difference in stored values for time series. Due to different compression algorithms, VM may reduce precision for float values with more than 12 significant decimal digits. Please see [this article](https://valyala.medium.com/evaluating-performance-and-correctness-victoriametrics-response-e27315627e87)
There could be a slight difference in stored values for time series. Due to different compression algorithms, VM may reduce the precision for float values with more than 12 significant decimal digits. Please see [this article](https://valyala.medium.com/evaluating-performance-and-correctness-victoriametrics-response-e27315627e87).
The query engine may behave differently for some functions. Please see [this article](https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e)
The query engine may behave differently for some functions. Please see [this article](https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e).

15
docs/README.md
View file

@ -1144,16 +1144,21 @@ write data to the same VictoriaMetrics instance. These vmagent or Prometheus ins
Retention is configured with `-retentionPeriod` command-line flag. For instance, `-retentionPeriod=3` means
that the data will be stored for 3 months and then deleted.
Data is split in per-month subdirectories inside `<-storageDataPath>/data/small` and `<-storageDataPath>/data/big` folders.
Directories for months outside the configured retention are deleted on the first day of new month.
Data is split in per-month partitions inside `<-storageDataPath>/data/small` and `<-storageDataPath>/data/big` folders.
Data partitions outside the configured retention are deleted on the first day of new month.
Each partition consists of one or more data parts with the following name pattern `rowsCount_blocksCount_minTimestamp_maxTimestamp`.
Data parts outside of the configured retention are eventually deleted during [background merge](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282).
In order to keep data according to `-retentionPeriod` max disk space usage is going to be `-retentionPeriod` + 1 month.
For example if `-retentionPeriod` is set to 1, data for January is deleted on March 1st.
It is safe to extend `-retentionPeriod` on existing data. If `-retentionPeriod` is set to lower
value than before then data outside the configured period will be eventually deleted.
VictoriaMetrics supports retention smaller than 1 month. For example, `-retentionPeriod=5d` would set data retention for 5 days.
Older data is eventually deleted during [background merge](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282).
Please note, time range covered by data part is not limited by retention period unit. Hence, data part may contain data
for multiple days and will be deleted only when fully outside of the configured retention.
It is safe to extend `-retentionPeriod` on existing data. If `-retentionPeriod` is set to lower
value than before then data outside the configured period will be eventually deleted.
## Multiple retentions

15
docs/Single-server-VictoriaMetrics.md
View file

@ -1148,16 +1148,21 @@ write data to the same VictoriaMetrics instance. These vmagent or Prometheus ins
Retention is configured with `-retentionPeriod` command-line flag. For instance, `-retentionPeriod=3` means
that the data will be stored for 3 months and then deleted.
Data is split in per-month subdirectories inside `<-storageDataPath>/data/small` and `<-storageDataPath>/data/big` folders.
Directories for months outside the configured retention are deleted on the first day of new month.
Data is split in per-month partitions inside `<-storageDataPath>/data/small` and `<-storageDataPath>/data/big` folders.
Data partitions outside the configured retention are deleted on the first day of new month.
Each partition consists of one or more data parts with the following name pattern `rowsCount_blocksCount_minTimestamp_maxTimestamp`.
Data parts outside of the configured retention are eventually deleted during [background merge](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282).
In order to keep data according to `-retentionPeriod` max disk space usage is going to be `-retentionPeriod` + 1 month.
For example if `-retentionPeriod` is set to 1, data for January is deleted on March 1st.
It is safe to extend `-retentionPeriod` on existing data. If `-retentionPeriod` is set to lower
value than before then data outside the configured period will be eventually deleted.
VictoriaMetrics supports retention smaller than 1 month. For example, `-retentionPeriod=5d` would set data retention for 5 days.
Older data is eventually deleted during [background merge](https://medium.com/@valyala/how-victoriametrics-makes-instant-snapshots-for-multi-terabyte-time-series-data-e1f3fb0e0282).
Please note, time range covered by data part is not limited by retention period unit. Hence, data part may contain data
for multiple days and will be deleted only when fully outside of the configured retention.
It is safe to extend `-retentionPeriod` on existing data. If `-retentionPeriod` is set to lower
value than before then data outside the configured period will be eventually deleted.
## Multiple retentions

10
docs/guides/README.md
View file

@ -4,8 +4,8 @@ sort: 21
# Guides
1. [K8s monitoring via VM Single](k8s-monitoring-via-vm-single.html)
2. [K8s monitoring via VM Cluster](k8s-monitoring-via-vm-cluster.html)
3. [HA monitoring setup in K8s via VM Cluster](k8s-ha-monitoring-via-vm-cluster.html)
4. [Getting started with VM Operator](getting-started-with-vm-operator.html)
5. [Multi Retention Setup within VictoriaMetrics Cluster](guide-vmcluster-multiple-retention-setup.html)
1. [K8s monitoring via VM Single](https://docs.victoriametrics.com/guides/k8s-monitoring-via-vm-single.html)
2. [K8s monitoring via VM Cluster](https://docs.victoriametrics.com/guides/k8s-monitoring-via-vm-cluster.html)
3. [HA monitoring setup in K8s via VM Cluster](https://docs.victoriametrics.com/guides/k8s-ha-monitoring-via-vm-cluster.html)
4. [Getting started with VM Operator](https://docs.victoriametrics.com/guides/getting-started-with-vm-operator.html)
5. [Multi Retention Setup within VictoriaMetrics Cluster](https://docs.victoriametrics.com/guides/guide-vmcluster-multiple-retention-setup.html)

2
docs/guides/k8s-monitoring-via-vm-cluster.md
View file

@ -24,7 +24,7 @@ We will use:
> For this guide we will use Helm 3 but if you already use Helm 2 please see this [https://github.com/VictoriaMetrics/helm-charts#for-helm-v2](https://github.com/VictoriaMetrics/helm-charts#for-helm-v2)
You need to add the VictoriaMetrics Helm repository to install VictoriaMetrics components. We’re going to use [VictoriaMetrics Cluster](https://monitoring.monster/Cluster-VictoriaMetrics.html). You can do this by running the following command:
You need to add the VictoriaMetrics Helm repository to install VictoriaMetrics components. We’re going to use [VictoriaMetrics Cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html). You can do this by running the following command:
<div class="with-copy" markdown="1">

2
docs/vmbackup.md
View file

@ -10,7 +10,7 @@ Supported storage systems for backups:
* [GCS](https://cloud.google.com/storage/). Example: `gs://<bucket>/<path/to/backup>`
* [S3](https://aws.amazon.com/s3/). Example: `s3://<bucket>/<path/to/backup>`
* Any S3-compatible storage such as [MinIO](https://github.com/minio/minio), [Ceph](https://docs.ceph.com/docs/mimic/radosgw/s3/) or [Swift](https://www.swiftstack.com/docs/admin/middleware/s3_middleware.html). See [these docs](#advanced-usage) for details.
* Any S3-compatible storage such as [MinIO](https://github.com/minio/minio), [Ceph](https://docs.ceph.com/en/pacific/radosgw/s3/) or [Swift](https://www.swiftstack.com/docs/admin/middleware/s3_middleware.html). See [these docs](#advanced-usage) for details.
* Local filesystem. Example: `fs://</absolute/path/to/backup>`
`vmbackup` supports incremental and full backups. Incremental backups are created automatically if the destination path already contains data from the previous backup.

4
docs/vmgateway.md
View file

@ -40,6 +40,7 @@ jwt token must be in following format:
"team": "dev",
"project": "mobile"
},
"extra_filters": ["{env~=\"prod|dev\",team!=\"test\"}"],
"mode": 1
}
}
@ -48,7 +49,8 @@ Where:
- `exp` - required, expire time in unix_timestamp. If the token expires then `vmgateway` rejects the request.
- `vm_access` - required, dict with claim info, minimum form: `{"vm_access": {"tenand_id": {}}`
- `tenant_id` - optional, for cluster mode, routes requests to the corresponding tenant.
- `extra_labels` - optional, key-value pairs for label filters added to the ingested or selected metrics.
- `extra_labels` - optional, key-value pairs for label filters added to the ingested or selected metrics. Multiple filters are added with `and` operation. If defined, `extra_label` from original request removed.
- `extra_filters` - optional, [series selectors](https://prometheus.io/docs/prometheus/latest/querying/basics/#time-series-selectors) added to the select query requests. Multiple selectors are added with `or` operation. If defined, `extra_filter` from original request removed.
- `mode` - optional, access mode for api - read, write, or full. Supported values: 0 - full (default value), 1 - read, 2 - write.
## QuickStart