Commit graph

1108 commits

Author SHA1 Message Date
Noah Labrecque
fbb572a180
fix: apply correct bounds to sf and tf (#5274) 2023-11-14 01:19:47 +01:00
Aliaksandr Valialkin
356deada8c
lib/htmlcomponents: use relative links for the top page and for favicon.ico
This allows hiding VictoriaMetrics components behind proxies with arbitrary path prefixes.
For example, vmagent HTTP handlers can be served via /vmagent/ path prefix:

- http://proxy/vmagent/targets
- http://proxy/vmagent/service-discovery

The path prefix can be arbitrary. For example, below are vmagent urls
for /tenantID/vmagent/ path prefix:

- http://proxy/tenantID/vmagent/targets
- http://proxy/tenantID/vmagent/service-discovery

While at it, consistently serve favicon.ico from any path directory.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5306
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5307
2023-11-13 20:28:17 +01:00
Aliaksandr Valialkin
a45cbc101f
all: cleanup: remove // +build ... lines, since they are no longer needed after Go1.17, and the minimum supported Go version for VictoriaMetrics source code is Go1.20 2023-11-13 19:15:42 +01:00
Aliaksandr Valialkin
d9ecc3f6d7
lib/logger: add -loggerMaxArgLen command-line flag for fine-tuning the maximum length of logged args 2023-11-13 09:43:49 +01:00
Aliaksandr Valialkin
c916294b61
app/vmselect/promql: optimize instant queries with min_over_time() and max_over_time() rollup functions
This is a follow-up for 41a0fdaf39
2023-11-13 09:43:18 +01:00
Aliaksandr Valialkin
bf01a97f17
docs/CHANGELOG.md: update the description of the optimization for SLO/SLI-like queries according to latest changes
See commits 4497a08e3d and 92826b0b4a
2023-11-02 20:09:22 +01:00
Aliaksandr Valialkin
7fc5178a4b
app/vmselect/promql: add missing trace message in rollupResultCache.GetSeries() 2023-11-02 09:17:13 +01:00
Aliaksandr Valialkin
ece7024f11
app/vmselect/promql: reduce the minimum lookbehind window for enabling SLO/SLI optimizations from 24 hours to 6 hours
This reduction is based on production testing.

Also expose -search.minWindowForInstantRollupOptimization command-line flag, so users could fine-tune this arg for their needs
2023-11-01 20:19:19 +01:00
Aliaksandr Valialkin
e4365dbe3e
app/vmselect: run make quicktemplate-gen after b8739bc00b 2023-11-01 17:53:30 +01:00
Aliaksandr Valialkin
ae9b4c94bc
app/vmselect: return stats.seriesFetched as string instead of number
vmalert expects string value for stats.seriesFetched, so it is impossible
switching to number without breaking compatibility with old vmalert releases :(

It is still unclear why stats.seriesFetched has string type in the first place...
2023-11-01 17:49:28 +01:00
Aliaksandr Valialkin
6a98f9df54
app/vmui: show query execution duration in the header of query input field
This should simplify the process of query optimization
2023-11-01 16:46:42 +01:00
Aliaksandr Valialkin
c5e3b11762
app/vmselect/promql: apply SLO-like optimization to all the count_*_over_time() functions
This is a follow-up for 41a0fdaf39
2023-11-01 09:58:50 +01:00
Aliaksandr Valialkin
b96d55e1e4
app/vmselect/promql: typo fix, which could lead to panic during range query execution
The panic is:

  BUG: unexpected values after merging new values

This is a follow-up for 41a0fdaf39
2023-11-01 09:58:50 +01:00
Aliaksandr Valialkin
28f0610e14
app/vmui: fix non-working Disable cache checkbox at JSON and Table views 2023-10-31 22:58:15 +01:00
Aliaksandr Valialkin
7b7ad44e84
app/vmselect/promql: properly calculate rollup result if lookbehind window isn't set
This is a follow-up for 41a0fdaf39
2023-10-31 22:23:04 +01:00
Aliaksandr Valialkin
744f8c3fe7
app/vmselect/promql: add outliers_iqr(q) and outlier_iqr_over_time(m[d]) functions
These functions allow detecting anomalies in series and samples using Interquartile range method.
See Outliers section at https://en.wikipedia.org/wiki/Interquartile_range for more details.
2023-10-31 22:14:14 +01:00
Aliaksandr Valialkin
9661918bb4
app/vmselect/promql: optimize repeated SLI-like instant queries with lookbehind windows >= 1d
Repeated instant queries with long lookbehind windows, which contain one of the following rollup functions,
are optimized via partial result caching:

- sum_over_time()
- count_over_time()
- avg_over_time()
- increase()
- rate()

The basic idea of optimization is to calculate

  rf(m[d] @ t)

as

  rf(m[offset] @ t) + rf(m[d] @ (t-offset)) - rf(m[offset] @ (t-d))

where rf(m[d] @ (t-offset)) is cached query result, which was calculated previously

The offset may be in the range of up to 1 hour.
2023-10-31 20:08:38 +01:00
Aliaksandr Valialkin
9ba007a636
app/vmselect/promql: wrap too long line after a950873fff 2023-10-31 19:11:05 +01:00
Roman Khavronenko
9d8f93050c
app/vmselect: expose vm_memory_intensive_queries_total counter metric (#5208)
The new metric gets increased each time `-search.logQueryMemoryUsage` memory limit
is exceeded by a query. This metric should help to identify expensive and heavy queries
without inspecting the logs.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-31 19:02:22 +01:00
Aliaksandr Valialkin
a66c261b55
app/vmui: change the order of tables at Top queries tab
Move the most interesting table - queries with the most summary time to execute - to the top
2023-10-28 11:57:08 +02:00
Aliaksandr Valialkin
36a1fdca6c
all: consistently use %w instead of %s in when error is passed to fmt.Errorf()
This allows consistently using errors.Is() for verifying whether the given error wraps some other known error.
2023-10-26 09:44:40 +02:00
Roman Khavronenko
cd2247b24a
app/vmselect: limit the number of parallel workers by 32 (#5195)
* app/vmselect: limit the number of parallel workers by 32

The change should improve performance and memory usage during query processing
on machines with big number of CPU cores. The number of parallel workers for
query processing is controlled via `-search.maxWorkersPerQuery` command-line flag.
By default, the number of workers is limited by the number of available CPU cores,
but not more than 32. The limit can be increased via `-search.maxWorkersPerQuery`.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* wip

- The `-search.maxWorkersPerQuery` command-line flag doesn't limit resource usage,
  so move it from the `resource usage limits` to `troubleshooting` chapter at docs/Single-server-VictoriaMetrics.md

- Make more clear the description for the `-search.maxWorkersPerQuery` command-line flag

- Add the description of `-search.maxWorkersPerQuery` to docs/Cluster-VictoriaMetrics.md

- Limit the maximum value, which can be passed to `-search.maxWorkersPerQuery`, to GOMAXPROCS,
  because bigger values may worsen query performance and increase CPU usage

- Improve the the description of the change at docs/CHANGELOG.md. Mark it as FEATURE instead of BUGFIX,
  since it is closer to a feature than to a bugfix.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5087

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-10-26 09:15:27 +02:00
Aliaksandr Valialkin
8642418e5a
app/vmselect: follow-up for 348c1bcec0: cache static contents served from /select/tenantID/prometheus/vmui/static/... 2023-10-16 23:27:06 +02:00
Aliaksandr Valialkin
627a4e9330
app/vmselect/promql: add labels_equal(q, "label1", "label2", ...) function
This function returns q series, which have identical values for the listed labels
"label1", "label2", ...

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5148
2023-10-16 21:51:13 +02:00
Aliaksandr Valialkin
b2f9b9d634
app/vmselect/promql: add drop_empty_series() function for dropping empty series before performing additional calculations
This can be useful in the following queries:

   drop_empty_series(temperature <= 30) default 40

This query drops temperature series with all the values bigger than 30 on the selected time range,
while replacing gaps in the remaining series with 40.

The query without drop_empty_series:

  (temperature <= 30) default 40

would leave all the temperature series with all the values bigger than 30 on the selected time range,
and replace all their values with 40. This is not what could be epxected in some cases
like here - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5071
2023-10-16 20:59:21 +02:00
Aliaksandr Valialkin
4278b00a66
app/vmselect/promql: do not use unsafe conversion from bytes slice to string when storing a value by map key
The assigned map key shouldn't change over time, otherwise the map won't work properly.

This is a follow-up for 1f91f22b5f
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5087
2023-10-16 13:55:40 +02:00
Aliaksandr Valialkin
b86bec8109
app/vmui: small UX enhancements
- Reduce vertical space usage, so more information is available on the screen without the need to scroll.
- Show information for lines with higher values at the top of the legend under the graph.
  This should simplify graph analysis when it contains many lines.
2023-10-16 12:39:32 +02:00
Aliaksandr Valialkin
348c1bcec0
app/{vmselect,vlselect}: enable caching of static contents from /vmui/static/ folder at client side
This should improve repated VMUI page load times on slow networks

See https://developer.chrome.com/docs/lighthouse/performance/uses-long-cache-ttl/
2023-10-16 12:36:34 +02:00
Nikolay
4a50e9400c
app/vmselect: reduce lock contention for heavy aggregation requests (#5119)
reduce lock contention for heavy aggregation requests
previously lock contetion may happen on machine with big number of CPU due to enabled string interning. sync.Map was a choke point for all aggregation requests.
Now instead of interning, new string is created. It may increase CPU and memory usage for some cases.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5087
2023-10-10 13:44:02 +02:00
Aliaksandr Valialkin
b5812e2457
app/{vmselect,vlselect}: run make vmui-update vmui-logs-update 2023-10-02 21:44:21 +02:00
Aliaksandr Valialkin
5fd79f47f1
app/vmselect/promql: follow-up for 896c85a4a4
- Clarify the description of the change at docs/CHANGELOG.md
- Make sure that bitmap_*(X, NaN) returns NaN

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4996
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/5021
2023-10-02 21:07:46 +02:00
Dmytro Kozlov
90b189dab8
app/vmselect: fix bitmap_*() functions behavior (#5021)
Related issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4996

Signed-off-by: dmitryk-dk d.kozlov@victoriametrics.com

Signed-off-by: dmitryk-dk d.kozlov@victoriametrics.com
Co-authored-by: Nikolay <nik@victoriametrics.com>
2023-10-02 20:13:27 +02:00
Aliaksandr Valialkin
538dc6058d
app/vmselect/promql: run make fmt after 3b9605dba5 2023-09-25 16:15:58 +02:00
Aliaksandr Valialkin
b43ff80d21
app/vmselect/promql: do not sort q1 or q2 results
This makes sure that `q2` series are returned after `q1` series in the same way as Prometheus does

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4763
2023-09-25 16:15:02 +02:00
Aliaksandr Valialkin
c954019e43
app/vmselect/promql: completely substitute median_over_time() WITH template with regular median_over_time() rollup function
This is a follow-up for 34d7a670d0

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5034
2023-09-25 15:31:25 +02:00
Zakhar Bessarab
fd6ca57c14
app/vmselect/promql: add implementation of median_over_time for rollup functions list (#5042)
`median_over_time` is handled by predefined WITH template in MetricsQL library which translates it to `quantile_over_time(0.5)`
This makes it impossble to use `median_over_time` as a usual rollup function for `aggr_over_time`.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5034

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-09-25 15:31:25 +02:00
Konstantin
c1a8a2d54c
app/vmselect: return +Inf as null in graphite render api (#5009)
Signed-off-by: Konstantin Kulikov <k.kulikov2@gmail.com>
2023-09-18 16:41:39 +02:00
Dmytro Kozlov
5477b52991
vmagent: add validation of MetricsQL functions (#4991)
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-09-15 13:16:22 +02:00
Aliaksandr Valialkin
9c3a37597c
app/vmselect/netstorage: run make fmt after 58326dbf25 2023-09-10 15:18:15 +02:00
Aliaksandr Valialkin
58326dbf25
app/vmselect: return 503 status code when partial responses are denied and some of vmstorage nodes are temporarily unavailable
This should help detecting this case and automatic retrying the query at healthy cluster replica
in another availability zone.

This commit is needed as a preparation for automatic query retry at another backend at vmauth on 5xx errors
as described at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4792#issuecomment-1674338561
2023-09-07 16:07:06 +02:00
Aliaksandr Valialkin
4d3c24492c
app/vmselect: run make vmui-update 2023-09-06 10:29:59 +02:00
Aliaksandr Valialkin
b9b2fbc7cd
app/vmselect: run make vmui-update after c112dd7367 2023-09-01 10:54:22 +02:00
Aliaksandr Valialkin
d8afd7fe98
Makefile: update golangci-lint from v1.51.2 to v1.54.2
See https://github.com/golangci/golangci-lint/releases/tag/v1.54.2
2023-09-01 10:25:49 +02:00
Aliaksandr Valialkin
1ca3b660f0
app/vmselect/promql: add support for _ delimiters in numeric values
For example, 1_234_567_890 is equivalent to 1234567890,
while 1.234_567_890 is equivalent to 1.234567890
2023-08-30 14:35:58 +02:00
Aliaksandr Valialkin
3a2d035283
lib/auth: add NewTokenPossibleMultitenant() for parsing auth token, which can be multitenant
Disallow parsing multitenant token at auth.NewToken().

Use auth.NewTokenPossibleMultitenant() at vminsert only. All the other callers should call auth.NewToken(),
since they do not support multitenant token.

This is a follow-up for f0c06b428e

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4910
2023-08-30 14:13:51 +02:00
hagen1778
bda9699657
app/vmselect: follow-up after f0c06b428e
Remove extra error message when auth token is nil. The default message
about unsupported path should be more clear to the user who mistakenly
requested /multitenant path.

f0c06b428e
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-08-29 21:52:50 +02:00
Zakhar Bessarab
f0c06b428e
app/vmselect: fix panic when using /select/multitenant endpoint (#4912)
app/vmselect: fix panic when using `/select/multitenant` endpoint

Such requests must be rejected as not found since vmselect does not support multitenant endpoint.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4910

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-08-29 21:48:12 +02:00
Aliaksandr Valialkin
5e8dfcf65e
app/{vmselect,vlselect}: run make vmui-update vmui-logs-update after recent changes to app/vmui 2023-08-29 12:58:58 +02:00
Aliaksandr Valialkin
19d61737c1
app/{vminsert,vmselect}: follow-up after 2b7b3293c1
- Document the change at docs/CHANGELOG.md
- Set the default value for -vmstorageUserTimeout to 3 seconds. This is much better
  than the 0 value, which means that TCP connection to unreachable vmstorage could block
  for up to 16 minutes.
- Document -vmstorageUserTimeout at docs/Cluster-VictoriaMetrics.md
2023-08-29 12:17:39 +02:00
Will Jordan
2b7b3293c1
Add vmstorageUserTimeout flags to configure TCP user timeout (Linux) (#4423)
`TCP_USER_TIMEOUT` (since Linux 2.6.37) specifies the maximum amount of
time that transmitted data may remain unacknowledged before TCP will
forcibly close the connection and return `ETIMEDOUT` to the application.

Setting a low TCP user timeout allows RPC connections quickly reroute
around unavailable storage nodes during network interruptions.
2023-08-29 11:46:39 +02:00