github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity

app/vmselect: parallelize q1 <binary_op> q2 queries by running q1 and q2 in parallel

Browse source 
This should reduce query execution times.
This commit is contained in:
Aliaksandr Valialkin 2021-02-10 22:59:35 +02:00
parent 4e645a5fd3
commit 985c3e301d
2 changed files with 40 additions and 8 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

45
app/vmselect/promql/eval.go
View file

@ -284,14 +284,45 @@ func evalExpr(ec *EvalConfig, e metricsql.Expr) ([]*timeseries, error) {
return rv, nil
}
if be, ok := e.(*metricsql.BinaryOpExpr); ok {
left, err := evalExpr(ec, be.Left)
if err != nil {
return nil, err
}
right, err := evalExpr(ec, be.Right)
if err != nil {
return nil, err
// Execute left and right sides of the binary operation in parallel.
// This should reduce execution times for heavy queries.
// On the other side this can increase CPU and RAM usage when executing heavy queries.
// TODO: think on how to limit CPU and RAM usage while leaving short execution times.
var left, right []*timeseries
var mu sync.Mutex
var wg sync.WaitGroup
var errGlobal error
wg.Add(1)
go func() {
defer wg.Done()
tss, err := evalExpr(ec, be.Left)
mu.Lock()
if err != nil {
if errGlobal == nil {
errGlobal = err
}
}
left = tss
mu.Unlock()
}()
wg.Add(1)
go func() {
defer wg.Done()
tss, err := evalExpr(ec, be.Right)
mu.Lock()
if err != nil {
if errGlobal == nil {
errGlobal = err
}
}
right = tss
mu.Unlock()
}()
wg.Wait()
if errGlobal != nil {
return nil, errGlobal
}
bf := getBinaryOpFunc(be.Op)
if bf == nil {
return nil, fmt.Errorf(`unknown binary op %q`, be.Op)

3
docs/CHANGELOG.md
View file

@ -2,7 +2,8 @@
# tip
* FEATURE: optimize searching for time series by label filters where individual filters match big number of time series (more than a million). For example, the query `up{job="foobar"}` should work faster if `{job="foobar"}` matches a million of time series, while `up{job="foobar"}` matches much lower number of time series.
* FEATURE: optimize searching for matching metrics for `metric{<label_filters>}` queries if `<label_filters>` contains at least a single filter. For example, the query `up{job="foobar"}` should find the matching time series much faster than previously.
* FEATURE: reduce execution times for `q1 <binary_op> q2` queries by executing `q1` and `q2` in parallel.
* FEATURE: single-node VictoriaMetrics now accepts requests to handlers with `/prometheus` and `/graphite` prefixes such as `/prometheus/api/v1/query`. This improves compatibility with [handlers from VictoriaMetrics cluster](https://victoriametrics.github.io/Cluster-VictoriaMetrics.html#url-format).
* FEATURE: expose `process_open_fds` and `process_max_fds` metrics. These metrics can be used for alerting when `process_open_fds` reaches `process_max_fds`. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/402 and https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1037
* FEATURE: vmalert: add `-datasource.appendTypePrefix` command-line option for querying both Prometheus and Graphite datasource in cluster version of VictoriaMetrics. See [these docs](https://victoriametrics.github.io/vmalert.html#graphite) for details.