Aliaksandr Valialkin
946ca438a6
app/vmselect/netstorage: marshal block outside tmpBlocksFile.WriteBlock
...
This also allows marshaling outside lock, thus reducing the amount of work under the lock.
2019-09-28 20:57:20 +03:00
Aliaksandr Valialkin
e92e39eddf
app/vmselect/netstorage: reduce the number of disk seeks when the query processes big number of time series
2019-09-28 20:57:20 +03:00
Aliaksandr Valialkin
56dff57f77
app/vmselect/netstorage: reduce memory usage when fetching big number of data blocks from vmstorage
...
Dump data blocks directly to temporary file instead of buffering them in RAM
2019-09-28 12:21:57 +03:00
Aliaksandr Valialkin
ba460f62e6
app/vmselect/promql: do not generate timestamps for NaN values in timestamp
function according to Prometheus logic
2019-09-27 18:55:16 +03:00
Stian Øvrevåge
a9dac3829e
Update README.md - Fix helm command typos
...
`victoria-mertrics` -> `victoria-metrics` in helm commands.
2019-09-27 12:33:48 +03:00
Aliaksandr Valialkin
de919574a5
deployment/docker: switch Go builder image from v1.13.0 to v1.13.1
2019-09-26 17:12:52 +03:00
Aliaksandr Valialkin
d0b4590099
lib/storage: optimize TSID comparison
2019-09-26 14:20:02 +03:00
Aliaksandr Valialkin
95e3d648cb
lib/storage: verify whether items are sorted in the end of call to mergeTagToMetricIDsRows
...
This should prevent from inverted index corruption if bug in mergeTagToMetricIDsRows is discovered.
2019-09-26 13:13:58 +03:00
Aliaksandr Valialkin
2b8358726f
lib/storage: properly match labels against regexp with (?i)
flag
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/161
2019-09-26 11:03:26 +03:00
Aliaksandr Valialkin
bd1cf053f6
app/vmselect/promql: add increases_over_time
and decreases_over_time
functions
...
`increases_over_time(q[d])` returns the number of `q` increases during the given duration `d`.
`decreases_over_time(q[d])` returns the number of `q` decreases during the given duration `d`.
2019-09-25 20:38:51 +03:00
Aliaksandr Valialkin
4e3871ac1e
lib/storage: add missing break in removeDuplicateMetricIDs
2019-09-25 18:23:13 +03:00
Aliaksandr Valialkin
4468f9f966
lib/storage: remove duplicate MetricIDs in tag->metricIDs
items before writing them into inverted index
2019-09-25 17:57:36 +03:00
Aliaksandr Valialkin
adc18c3ee6
lib/{mergeset,storage}: do not cache inverted index blocks containing tag->metricIDs
items
...
This should reduce the amounts of used RAM during queries with filters over big number of time series.
2019-09-25 13:48:24 +03:00
Aliaksandr Valialkin
8d398af92f
app/vminsert/netstorage: mention the data size that cannot be sent to vmstorage
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/175
2019-09-25 12:53:41 +03:00
Aliaksandr Valialkin
73ac7b8dd6
app/vminsert/netstorage: make sure the conn exists before closing it in storageNode.closeBrokenConn
...
The conn can be missing or already closed during the call to storageNode.closeBrokenConn.
Prevent `nil pointer dereference` panic by verifying whether the conn is already closed.
Thanks to @CH-anhngo for reporting the issue.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/189
2019-09-25 10:36:50 +03:00
Aliaksandr Valialkin
c64fb91a43
lib/uint64set: optimize Set.AppendTo
2019-09-25 00:34:31 +03:00
Aliaksandr Valialkin
de0e4eee2c
lib/storage: create and use lib/uint64set
instead of map[uint64]struct{}
...
This should improve inverted index search performance for filters matching big number of time series,
since `lib/uint64set.Set` is faster than `map[uint64]struct{}` for both `Add` and `Has` calls.
See the corresponding benchmarks in `lib/uint64set`.
2019-09-24 21:18:04 +03:00
Aliaksandr Valialkin
2212d0e421
lib/storage: typo fix: return dstData instead of data from mergeTagToMetricIDsRows
2019-09-24 19:32:58 +03:00
Aliaksandr Valialkin
9307de1b92
lib/storage: limit the number of metricIDs in tag->metricIDs row
...
This reduces the overhead on index and metaindex in lib/mergeset
2019-09-24 00:50:47 +03:00
Aliaksandr Valialkin
7734fc8012
lib/storage: share tsids across all the partSearch instances
...
This should reduce memory usage when big number of time series matches the given query.
2019-09-23 22:36:16 +03:00
Aliaksandr Valialkin
67a2bcb98a
lib/{storage,mergeset}: verify PrepareBlock callback results
...
Do not touch the first and the last item passed to PrepareBlock
in order to preserve sort order of mergeset blocks.
2019-09-23 20:46:33 +03:00
Aliaksandr Valialkin
3304dc1e85
lib/mergeset: detect whether we are in test by executable suffix
2019-09-22 23:12:35 +03:00
Aliaksandr Valialkin
d2ed8cb0b2
lib/storage: generate the first tag->metricIDs item in a mergeset block with a single metricID
...
The first item from each mergeset block goes into index (lib/mergeset.blockHeader),
so it must be short in order to reduce index size.
2019-09-22 19:37:50 +03:00
Aliaksandr Valialkin
0a9cb6368e
lib/workingsetcache: remove data race when resetting c.misses
2019-09-22 19:37:09 +03:00
Aliaksandr Valialkin
7d13c31566
lib/{storage,mergeset}: merge tag->metricID
rows into tag->metricIDs
rows for common tag
values
...
This should improve lookup performance if the same `label=value` pair exists
in big number of time series.
This should also reduce memory usage for mergeset data cache, since `tag->metricIDs` rows
occupy less space than the original `tag->metricID` rows.
2019-09-20 22:06:23 +03:00
Aliaksandr Valialkin
272e2f77c9
lib/encoding: optimize UnmarshalUint* and UnmarshalInt*
2019-09-20 13:08:24 +03:00
Aliaksandr Valialkin
7e0c6d4ca6
lib/storage: optimize selecting all the metricIDs by scanning MetricID->TSID entries instead of tag->MetricID entries
...
The number of MetricID->TSID entries is smaller than the number of tag->MetricID entries
and MetricID->TSID entries are usually shorter than tag->MetricID entries.
This should improve performance when selecting all the metricIDs.
2019-09-20 11:57:57 +03:00
Aliaksandr Valialkin
b0c738ae8b
app/vminsert/opentsdbhttp: remove FATAL prefix from logger.Fatalf errors for the sake of consistency with other logger.Fatalf calls
2019-09-19 22:16:11 +03:00
Aliaksandr Valialkin
bf8505353a
lib/mergeset: rename misleading mergeSmallParts to mergeExistingParts
2019-09-19 21:48:36 +03:00
Aliaksandr Valialkin
ebbef20535
lib/mergeset: use sort.IsSorted instead of sort.SliceIsSorted in inmemoryBlock.isSorted in order to reduce memory allocations
2019-09-19 20:13:54 +03:00
Aliaksandr Valialkin
89234f395d
lib/storage: use sort.Sort instead of sort.slice in getSortedMetricIDs
2019-09-19 20:08:13 +03:00
Aliaksandr Valialkin
6e586fa09c
lib/storage: skip duplicate call to intersectMetricIDsWithTagFilter on zero successful intersects
2019-09-19 17:51:10 +03:00
Aliaksandr Valialkin
410f993bf6
lib/mergeset: fill partHeader.firstItem on first block flush
2019-09-19 17:48:22 +03:00
Aliaksandr Valialkin
c05885fb5f
lib/storage: mark tag filter returning errFallbackToMetricNameMatch as useless
...
This will save CPU on subsequent calls for this filter
2019-09-18 19:11:44 +03:00
Aliaksandr Valialkin
e041a196a7
deployment/docker/docker-compose.yml: update Prometheus from v2.3.2 to v2.12.0
2019-09-18 18:30:02 +03:00
Aliaksandr Valialkin
db71c940ea
lib/storage: properly construct keys for uselessTagFiltersCache and register useless negative tag filters there
2019-09-17 23:18:37 +03:00
Artem Navoiev
ccb6dc6925
[deployment] clean up helm ( #185 )
2019-09-16 21:58:16 +03:00
Aliaksandr Valialkin
491b1317f4
vendor: update github.com/valyala/gozstd from v1.6.1 to v1.6.2
2019-09-16 21:50:02 +03:00
Aliaksandr Valialkin
5666112de2
deployment: switch docker image tag from v1.27.2-cluster to v1.27.3-cluster
2019-09-14 11:33:18 +03:00
Aliaksandr Valialkin
ba21622b78
vendor: make vendor-update
2019-09-13 22:49:34 +03:00
Aliaksandr Valialkin
020341d13a
deployment/docker: remove file system paths from the compiled binary
2019-09-13 22:46:07 +03:00
Aliaksandr Valialkin
550a12415a
app/vminsert/netstorage: log network errors when sending data to vmstorage nodes
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/175
2019-09-13 22:26:24 +03:00
Aliaksandr Valialkin
41ef6b060e
lib/mergeset: properly check for sorted block headers
...
Fix a typo for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/181
2019-09-13 21:59:38 +03:00
Aliaksandr Valialkin
ee4585db33
app/vmselect/promql: properly handle subqueries like aggr_func(rollup_func(metric[window:step]))
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/184
2019-09-13 21:42:11 +03:00
hanzai
08cde5e3f4
lib/workingsetcache: adjust switching from mode=split
to mode=whole
smoothly and load cachefile successfully
2019-09-13 19:13:16 +03:00
Aliaksandr Valialkin
828e5f6d26
app/vmselect/promql: binary operation fixes according to Prometheus behaviour
...
The follosing issues were fixed:
- VictoriaMetrics could leave superflouos labels when using `on` or `ignoring` modifiers
- VictoriaMetrics could return `duplicate timeseries` error when using `group_left` or `group_right` with non-empty label list
2019-09-13 17:43:09 +03:00
Artem Navoiev
62b424bc4c
[ci] github actions - run pipeline on pull request. Fix running of test in external PR from forks
2019-09-11 14:54:45 +03:00
Aliaksandr Valialkin
ed50b8792b
app/vminsert/netstorage: reduce the maximum buffer size for rerouted rows, so it occupies less RAM
2019-09-11 14:50:30 +03:00
Aliaksandr Valialkin
b101064f8b
all: report the number of bytes read on io.ReadFull error
...
This should simplify error investigation similar to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/175
2019-09-11 14:50:24 +03:00
Aliaksandr Valialkin
2f4c950fe9
app/vminsert/netstorage: send per-storageNode bufs to vmstorage nodes in parallel
...
This should improve the maximum ingestion throughput.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/175
2019-09-11 14:50:19 +03:00