Commit graph

8634 commits

Author SHA1 Message Date
Aliaksandr Valialkin
e2eac858b5 lib/storage: calculate the maximum number of rows per small part from -memory.allowedPercent
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/159

This simplifies error detection additionally to the `vm_rows_ignored_total` counters.
2019-08-25 15:29:09 +03:00
Aliaksandr Valialkin
0a8dd9cc9a lib/storage: calculate the maximum number of rows per small part from -memory.allowedPercent
This should improve query speed over recent data on machines with big amounts of RAM
2019-08-25 14:41:32 +03:00
Aliaksandr Valialkin
bc576fb386 lib/storage: properly limit the number of output rows in small and big parts storage
Previously small parts storage didn't take into account the available disk space for big parts.
2019-08-25 14:41:32 +03:00
Aliaksandr Valialkin
947decb3dd lib/storage: remove outdated comment on maxRowsPerSmallPart
The commend became outdated after the commit ed6ac1a5df027f0dfc22448e3b27c26b6f77c67a,
which stops merging of small parts on graceful shutdown instead of waiting
for their completion.
2019-08-25 13:46:10 +03:00
Artem Navoiev
ce7798a6a2 [deployment] add ingresses and service monitors 2019-08-25 01:04:56 +03:00
Aliaksandr Valialkin
38711526d3 app/vminsert/influx: set db label only if Influx line doesnt have db tag 2019-08-24 13:55:01 +03:00
Aliaksandr Valialkin
023675c33e vendor: update github.com/valyala/quicktemplate from v1.1.1 to v1.2.0 2019-08-24 13:39:46 +03:00
Aliaksandr Valialkin
1ee536f9fd app/vminsert: skip empty tags 2019-08-24 13:36:41 +03:00
Aliaksandr Valialkin
a283023d16 app/vminsert/opentsdbhttp: skip invalid rows and continue parsing the remaining rows
Invalid rows are logged and counted in `vm_rows_invalid_total{type="opentsdb-http"}` metric
2019-08-24 13:36:41 +03:00
Aliaksandr Valialkin
38b9615c53 app/vminsert/opentsdb: skip invalid rows and continue parsing the remaining rows
Invalid rows are logged and counted in `vm_rows_invalid_total{type="opentsdb"}` metric
2019-08-24 13:36:41 +03:00
Aliaksandr Valialkin
2a8fc41bab app/vminsert/graphite: skip invalid rows and continue parsing the remaining rows
Invalid rows are logged and counted in `vm_rows_invalid_total{type="graphite"}` metric
2019-08-24 13:36:41 +03:00
Aliaksandr Valialkin
22685ef94d app/vminsert/influx: skip invalid rows and continue parsing the remaining rows
Invalid influx lines are logged and counted in `vm_rows_invalid_total{type="influx"}` metric.
2019-08-24 13:36:41 +03:00
Aliaksandr Valialkin
425a81a6c7 app/vminsert/influx: do not allow escaping newline char, since they dont occur in real life
The prefious report with escaped newline chars in influx line protocol was false alarm.
2019-08-23 18:43:00 +03:00
Aliaksandr Valialkin
8da8dd0876 app/vminsert/opentsdbhttp: allow timestamp as float64 and as string, since it occurs in real life 2019-08-23 18:35:52 +03:00
Aliaksandr Valialkin
0ea21eb9dc app/vminsert/influx: handle \r\n aka crlf influx line endings from windows world
Such lines exist in real life.
2019-08-23 18:28:54 +03:00
Aliaksandr Valialkin
b3502b2b39 app/vminsert/influx: allow escaping newline char
Though newline char isn't mentioned in escape rules at https://docs.influxdata.com/influxdb/v1.7/write_protocols/line_protocol_tutorial/ ,
there are reports that such chars occur in real life
2019-08-23 15:14:58 +03:00
Aliaksandr Valialkin
f1f8fce4f7 app/vminsert/influx: skip comments starting with # in influx line protocol 2019-08-23 14:43:24 +03:00
Aliaksandr Valialkin
697de90893 app/vminsert: do not drop data in reroutedBuf if all the storage nodes are unhealthy 2019-08-23 10:38:19 +03:00
Aliaksandr Valialkin
a5dc54efc3 app/vminsert: properly limit the size of reroutedBuf 2019-08-23 10:29:51 +03:00
Aliaksandr Valialkin
c50975e12d vendor: make vendor-update 2019-08-23 10:03:42 +03:00
Aliaksandr Valialkin
c197641978 all: return 503 http error if service is temporarily unavailable
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/156
2019-08-23 09:49:50 +03:00
Aliaksandr Valialkin
e734076f0f app/vminsert: allow setting the maximum number of labels per time series via -maxLabelsPerTimeseries 2019-08-23 08:47:18 +03:00
Aliaksandr Valialkin
4ed63d033a lib/storage: add benchmarks for regexp filter match / mismatch
These benchmarks allow estimate the performance of regexp filters in promql
2019-08-22 16:37:19 +03:00
Aliaksandr Valialkin
559dd03181 deployment: update docker image tags from v1.25.2-cluster to v1.26.0-cluster 2019-08-22 14:58:58 +03:00
Aliaksandr Valialkin
e9db22a551 app/vmselect/promql: attempt to repair invalid bucket counts passed to histogram_quantile
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/136
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/154
2019-08-22 14:39:24 +03:00
Aliaksandr Valialkin
0697164b4f app/vminsert: add ability to ingest data via HTTP OpenTSDB /api/put requests
This is manual merge of the https://github.com/VictoriaMetrics/VictoriaMetrics/pull/152
Thanks to nustinov@gmail.com for the initial pull request.
2019-08-22 12:46:54 +03:00
Aliaksandr Valialkin
4d555c7c87 app/vminsert/opentsdb: fix BenchmarkRowsUnmarshal by adding missing put prefixes to each line 2019-08-21 19:15:04 +03:00
Aliaksandr Valialkin
90a4b00b10 app/vmselect/promql: fix panic on -search.disableCache
Reset the cache if it is disabled instead of stopping, since it is stopped on graceful shutdown.
2019-08-21 17:12:01 +03:00
Aliaksandr Valialkin
491b1762c8 app/vmselect/promql: explain why empty timeseries arent removed in transformLabelValue 2019-08-21 11:29:41 +03:00
Aliaksandr Valialkin
db1de4277c app/vmselect/promql: remove NaNs from /api/v1/query_range output like Prometheus does
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/153
2019-08-20 23:01:59 +03:00
Aliaksandr Valialkin
99331606e1 app/vmselect/promql: pre-allocate memory for map for checking for duplicate timeseries
This should reduce memory allocations for big number of timeseries
2019-08-20 23:01:57 +03:00
Aliaksandr Valialkin
1101765adb app/vmselect/promql: add label_value(q, label_name) func, which returns numeric value labels with name label_name in q 2019-08-20 00:28:44 +03:00
Aliaksandr Valialkin
6ec6a8d7c1 lib/storage: try slower path for searching the tag filter with the minimum number of matching time series before giving up with increase -search.maxUniqueTimeseries error 2019-08-19 16:07:05 +03:00
Aliaksandr Valialkin
940349ccb9 app/vmselect/promql: independently track offset hints for tStart and tEnd
This should improve performance if timeseries starts or ends on the selected time range
2019-08-19 13:40:24 +03:00
Aliaksandr Valialkin
6ae4b4190f app/vmselect/promql: optimize search for timestamp boundaries in rollupConfig.Do
This should improve the performance of queries over big number of time series
with big number of output points.
2019-08-19 13:03:38 +03:00
Aliaksandr Valialkin
c59f5c4865 lib/storage: pre-allocate memory for blockHeader slice in unmarshalBlockHeaders
This reduces memory usage and memory fragmentation when working with big number of time series
2019-08-19 12:46:45 +03:00
Aliaksandr Valialkin
45e57be590 deployment: update docker image tags from v1.25.1-cluster to v1.25.2-cluster 2019-08-18 22:56:11 +03:00
Aliaksandr Valialkin
0f45273e20 deployment/docker: switch Go builder from go1.12.8 to go1.12.9 2019-08-18 22:09:21 +03:00
Aliaksandr Valialkin
005aabd305 app/vmselect/promql: add scrape_interval(q[d]) function, which would return scrape interval for q over d 2019-08-18 21:08:15 +03:00
Aliaksandr Valialkin
218cb4623a app/vmselect/promql: hande comparisons with NaN similar to Prometheus
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/150
2019-08-18 00:25:58 +03:00
Aliaksandr Valialkin
dcce92c63c app/vmselect/promql: add lifetime(q[d]) function, which returns the lifetime of q over d in seconds.
This function is useful for determining time series lifetime.
`d` must exceed the expected lifetime of the time series, otherwise
the function would return values close to `d`.
2019-08-16 11:59:51 +03:00
Aliaksandr Valialkin
0cb66a8f95 app/vmselect/promql: fix corner-case calculation for ideriv 2019-08-16 11:59:50 +03:00
Aliaksandr Valialkin
1b5b9ced27 app/vmselect/promql: properly handle corner cases for rollup functions 2019-08-15 23:31:28 +03:00
Aliaksandr Valialkin
f696cc503a lib/workingsetcache: automatically detect when it is better to double cache capacity 2019-08-15 22:58:04 +03:00
Aliaksandr Valialkin
97634d7101 deployment/docker: switch Go builder from go1.12.7 to go1.12.8 2019-08-15 20:43:23 +03:00
Aliaksandr Valialkin
e6541a7676 deployment: update docker images 2019-08-15 14:18:43 +03:00
Aliaksandr Valialkin
e399b948de Makefile: remove duplicate -cluster suffix from tar.gz file generated by make release, since this suffix must be already present in PKG_TAG 2019-08-15 14:07:43 +03:00
Aliaksandr Valialkin
1dd736a75c Makefile: add make release rule for building release tar.gz file with cluster binaries
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/146
2019-08-15 14:05:56 +03:00
Aliaksandr Valialkin
c15dfc6cea vendor: update github.com/valyala/gozstd from v1.5.1 to v1.6.0 2019-08-15 12:57:59 +03:00
Aliaksandr Valialkin
83ed5d3109 deployment: update docker images 2019-08-14 03:12:49 +03:00