Commit graph

7748 commits

Author SHA1 Message Date
Aliaksandr Valialkin
8253790157 app/vmstorage: consistency renaming for ignored rows metrics
vm_too_big_timestamp_rows_total -> vm_rows_ignored_total{reason="big_timestamp"}
  vm_too_small_timestamp_rows_total -> vm_rows_ignored_total{reason="small_timestamp"}
2019-07-26 20:02:24 +03:00
Aliaksandr Valialkin
c6bec48927 lib/storage: add metrics for calculating skipped rows outside the retention
The metrics are:

    - vm_too_big_timestamp_rows_total
    - vm_too_small_timestamp_rows_total
2019-07-26 14:11:56 +03:00
Aliaksandr Valialkin
aac482517f app/vmselect/promql: return NaN from count() over zero time series
This aligns `count` behavior with Prometheus.
2019-07-25 22:02:34 +03:00
Aliaksandr Valialkin
0e52357f35 app/vmselect/promql: properly calculate incremental aggregations grouped by __name__
Previously the following query may fail on multiple distinct metric names match:

    sum(count_over_time{__name__!=''}) by (__name__)
2019-07-25 21:53:26 +03:00
Aliaksandr Valialkin
f2e8d54fb0 lib/encoding/zstd: go fmt 2019-07-25 01:37:57 +03:00
Aliaksandr Valialkin
97b5dc7122 lib/encoding/zstd: disable CRC checks in pure Go build
This should give slightly better compression and decompressions performance.
Additionally this shaves off 4 bytes per each compressed block.
2019-07-24 19:17:32 +03:00
Aliaksandr Valialkin
54f035d4ce all: small updates after PR #114 2019-07-24 17:43:43 +03:00
Aliaksandr Valialkin
7a133567fb lib/encoding: small fixes in tests after the PR #114 2019-07-24 17:43:39 +03:00
Roman Khavronenko
fcf09aaa3c all: add Pure Go build (pull request #114)
Updates #94
2019-07-24 17:43:32 +03:00
Aliaksandr Valialkin
dd7bba94a3 dashboards: use rate instead of irate, because irate doesn't capture spikes
See https://medium.com/@valyala/why-irate-from-prometheus-doesnt-capture-spikes-45f9896d7832 for details
2019-07-20 15:55:48 +03:00
Aliaksandr Valialkin
3fae34eeb4 lib/encoding: improve gauge series detection
- Series with negative values are always gauges
- Counters may only have increasing values with possible counter resets

This should improve compression ratio for gauge series which
were previously mistakenly detected as counters.
2019-07-20 14:05:25 +03:00
Aliaksandr Valialkin
b335a811c3 deployment: switch builder from go1.12.6 to go1.12.7 2019-07-20 12:14:05 +03:00
Jiri Tyr
0aed0e0b5d Adding Grafana dashboards for VM cluster (#105) 2019-07-20 10:25:09 +03:00
Aliaksandr Valialkin
cb8104cf77 app: clarify error messages when -storageNode arg is missing in vminsert and vmselect 2019-07-20 10:21:59 +03:00
Aliaksandr Valialkin
fab1962e02 deployment/k8s/helm: use correct default ports for -storageNode
Previously these ports were swapped. Correct ports are:

- vminsert: -storageNode=*:8400
- vmselect: -storageNode=*:8401
2019-07-20 01:24:32 +03:00
Aliaksandr Valialkin
e3dcfe5851 deployment/docker/docker-compose.yml: use default ports for vminsert and vmselect services
These ports were swapped. Correct default ports are:

- vminsert: -httpListenAddr=:8480, -storageNode=*:8400
- vmselect: -httpListenAddr=:8481, -storageNode=*:8401
2019-07-20 01:20:08 +03:00
Thor Anker Kvisgård Lange
f576b267eb Fixed small bug in vmstorage name template
Signed-off-by: Thor Anker Kvisgård Lange <thanl@mhivestasoffshore.com>
2019-07-17 13:30:23 +03:00
Aliaksandr Valialkin
76b947dcb4 deployment: update Docker images 2019-07-15 23:56:24 +03:00
Aliaksandr Valialkin
7abb96b454 lib/netutil: do not count timeouts as network errors 2019-07-15 23:06:13 +03:00
Aliaksandr Valialkin
2b4254d01f app/vminsert: use netutil.TCPListener for collecting network-related metrics for Graphite and OpenTSDB TCP traffic 2019-07-15 22:58:35 +03:00
Aliaksandr Valialkin
092c9b39a8 app/vmselect/promql: remove empty time series after applying filters like q > 0
This should reduce CPU and RAM usage for queries over high number of time series.
2019-07-12 19:59:49 +03:00
Aliaksandr Valialkin
3bc9d3a14c vendor: update github.com/VictoriaMetrics/metrics to v1.7.0
This version adds support for `process_*` metrics similar
to metrics exposed by https://github.com/prometheus/client_golang .

Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/92
2019-07-12 17:24:58 +03:00
Aliaksandr Valialkin
6875fb411a app/vmselect/promql: parallelize incremental aggregation to multiple CPU cores
This may reduce response times for aggregation over big number of time series
with small step between output data points.
2019-07-12 15:53:12 +03:00
Aliaksandr Valialkin
be0ce54010 deployment: update docker images 2019-07-12 02:35:09 +03:00
Aliaksandr Valialkin
73a47d2a53 lib/storage: remove unused function isTooBigTimeRangeForDateMetricIDs 2019-07-12 02:28:40 +03:00
Aliaksandr Valialkin
97f9397687 lib/storage: do not reduce maxMetrics on time ranges exceeding maxDaysForDateMetricIDs
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/95
2019-07-12 02:21:52 +03:00
Aliaksandr Valialkin
1de6ef5f51 deployment: update Docker images 2019-07-11 19:10:35 +03:00
Aliaksandr Valialkin
4a8e6f47fe app/vmselect/prometheus: set start arg in /api/v1/series to the minimum allowed time by default as Prometheus does
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/91
2019-07-11 17:11:37 +03:00
Aliaksandr Valialkin
3313cdf816 app/vmselect/prometheus: convert negative times to 0, since they arent supported by the storage 2019-07-11 17:11:35 +03:00
Aliaksandr Valialkin
4ca66344ee lib/storage: do not pollute inverted index with data for samples outside the retention period 2019-07-11 17:11:33 +03:00
Aliaksandr Valialkin
0522efb2d6 lib/storage: add missing tagFilter.Marshal func 2019-07-11 15:01:01 +03:00
Aliaksandr Valialkin
12b1d67b41 lib/storage: use fast path for orSuffix when searching for metricIDs against plain tag value 2019-07-11 14:48:51 +03:00
Aliaksandr Valialkin
bf2e1b0ac1 lib/storage: remember and skip individual tag filters matching too many metrics
This saves CPU time by skipping useless matching for individual tag filters.
2019-07-11 14:48:47 +03:00
Aliaksandr Valialkin
cbab86fd9d app/vmselect/promql: reduce RAM usage for aggregates over big number of time series
Calculate incremental aggregates for `aggr(metric_selector)` function instead of
keeping all the time series matching the given `metric_selector` in memory.
2019-07-10 13:03:36 +03:00
Aliaksandr Valialkin
ba8195c58e all: consistency renaming: bytesSize -> sizeBytes 2019-07-10 00:47:42 +03:00
Aliaksandr Valialkin
df6f17b82c app/vmselect/promql: mention -search.logSlowQueryDuration flag value in the slow query log message 2019-07-10 00:43:01 +03:00
Aliaksandr Valialkin
73ae889244 app/vmselect/promql: extract rmoeveGroupTags function for removing unneeded tags from MetricName according to the given modifierExpr 2019-07-09 23:20:58 +03:00
Aliaksandr Valialkin
603b34edbd app/vmselect/promql: properly preserve metric name after applying functions in any case from transformFuncsKeepMetricGroup 2019-07-09 23:10:49 +03:00
Aliaksandr Valialkin
d6ec95693d app/vmselect/prometheus: typo fix 2019-07-07 23:34:04 +03:00
Aliaksandr Valialkin
61f6f63964 README.md: mention that vminsert spreads data amount vmstorage nodes using consistent hashing 2019-07-07 22:00:45 +03:00
Aliaksandr Valialkin
36636c1f6f app/vmselect/prometheus: handle minTime and maxTime values that may be set by Promxy or Prometheus client
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/88
2019-07-07 21:53:52 +03:00
Aliaksandr Valialkin
50c5894dc0 deployment: update docker images 2019-07-04 19:54:27 +03:00
Aliaksandr Valialkin
bba07d05fe app/vmselect/promql: remove empty timeseries left after topk call 2019-07-04 19:43:07 +03:00
Aliaksandr Valialkin
41f512af1c all: add vm_data_size_bytes metrics for easy monitoring of on-disk data size and on-disk inverted index size 2019-07-04 19:43:04 +03:00
Aliaksandr Valialkin
512a627855 app/vmselect/prometheus: update adjustLastPoints function
- Do not overwrite last points by the previous NaNs, since this may result in empty time series.
- Overwrite the last 2 points instead of 3. This should be enough in most cases.
2019-07-04 09:30:56 +03:00
Aliaksandr Valialkin
858746fa6c app/vmselect/promql: gracefully handle duplicate timestamps in irate and rollup_rate funcs
Previously such timestamps result in `+Inf` results. Now the previous timestamp is used
for the calculations.
2019-07-03 12:41:30 +03:00
Aliaksandr Valialkin
81da1c7b47 README.md: stylistic updates 2019-07-02 15:57:45 +03:00
Aliaksandr Valialkin
a3abed80ff app/vmselect: do not return empty time series in /api/v1/query result 2019-07-01 17:16:26 +03:00
Aliaksandr Valialkin
6682a35731 lib/memory: attempt #3 to determine memory limit for LXC container
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/84
2019-07-01 14:01:57 +03:00
Aliaksandr Valialkin
c3c60bee45 app/vmselect: add -search.denyPartialResponse flag for disabling partial responses if some of vmstorage nodes are unavailable
Also accept `deny_partial_response` query arg in Prometheus API handlers. If it is set to true,
then return error if some of vmstorage nodes are unavailable.
2019-06-30 01:27:07 +03:00