Aliaksandr Valialkin
110cce24d9
lib/storage: add vm_ prefix to deduplicated_samples_total
metric
2020-02-21 19:33:36 +02:00
Aliaksandr Valialkin
9d279e26a7
lib/protoparser/prometheus: skip leading whitespace from tag names
2020-02-16 19:06:23 +02:00
Aliaksandr Valialkin
a2b81b71b9
lib/storage: typo fix
2020-02-16 15:53:48 +02:00
Aliaksandr Valialkin
ad4cb9f3ca
lib/storage: prevent from clobbering nin-nil lastError in Storage.add
2020-02-16 15:51:35 +02:00
Aliaksandr Valialkin
846d7fa7e9
app/vmselect: add sort_by_label(q, label)
and sort_by_label_desc(q, label)
functions
...
This is implementation of https://github.com/prometheus/prometheus/pull/1533 for VictoriaMetrics.
2020-02-13 17:01:50 +02:00
Aliaksandr Valialkin
e3b18ca1ab
lib/mergeset: skip createing temporary part objects when merging source inmemory parts
...
This should reduce CPU usage when adding new entries to inverted index.
This should alos prevent from creating stalled cleaner goroutines for the created temporary parts,
since they were never closed.
This should fix the following issue: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/316 .
2020-02-13 14:09:13 +02:00
Aliaksandr Valialkin
347aaba79d
lib/{storage,mergeset}: use time.Ticker instead of time.Timer where appropriate
...
It has been appeared that time.Timer was used in places where time.Ticker must be used instead.
This could result in blocked goroutines as in the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/316 .
2020-02-13 13:21:48 +02:00
Aliaksandr Valialkin
e7d1037210
docs: migrate ExtendedPromQL->MetricsQL in order to be more consistent
2020-02-10 23:03:31 +02:00
Aliaksandr Valialkin
fcdd95a6ef
lib/envflag: check for incorrect flag values read from environment vars
2020-02-10 16:09:03 +02:00
Aliaksandr Valialkin
9c5db9400c
lib/envflag: add -envflag.enable
command-line flag for enabling reading flags from environment vars
...
By default flags are read only from command line. They can be read from environment vars if `-envflag.enable` is set.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/311
2020-02-10 16:09:01 +02:00
Aliaksandr Valialkin
1010a57882
all: allow setting flags via environment vars
...
Now flags can be set via environment vars with the same names as flags.
Command-line flags override flags set via env vars.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/311
2020-02-10 13:31:21 +02:00
Aliaksandr Valialkin
ea66212c93
lib/storage: move -dedup.minScrapeInterval
flag outside lib/storage, so it doesnt show up in vminsert
in cluster version
2020-02-10 13:07:25 +02:00
Aliaksandr Valialkin
8b360a25e9
lib/logger: initialize output to os.Stderr by default
2020-02-04 22:43:26 +02:00
Aliaksandr Valialkin
1f271a9815
lib/logger: add -loggerOutput
command-line flag
...
This flag allows changing log output from `stderr` to `stdout` if `-loggerOutput=stdout` is set.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/306
2020-02-04 21:48:24 +02:00
Aliaksandr Valialkin
49ab3fa076
lib/logger: do not clutter -loggerFormat=json
output with stack trace
...
This should improve json parsing
2020-02-04 21:40:20 +02:00
Aliaksandr Valialkin
56d6b8ed0a
lib/storage: do not deduplicate blocks with less than 32 samples during merge
...
This should improve deduplication accuracy for blocks with higher number of samples.
2020-02-04 18:41:37 +02:00
Aliaksandr Valialkin
7cde594696
all: do not clash flag description with back-quoted flag types
...
See https://golang.org/pkg/flag/#PrintDefaults for more details.
2020-02-04 15:56:01 +02:00
Aliaksandr Valialkin
9b25a2fb67
lib/fs: remove unused readerAt
interface
2020-01-31 15:13:00 +02:00
Aliaksandr Valialkin
e3adc095bd
all: add -dedup.minScrapeInterval
command-line flag for data de-duplication
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/86
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/278
2020-01-31 01:18:54 +02:00
Aliaksandr Valialkin
a45f25699c
lib/storage: re-use indexSearch inside Storage.prefetchMetricNames
2020-01-31 01:18:53 +02:00
Aliaksandr Valialkin
cb5c39ee70
lib/fs: optimize small reads for ReaderAt.MustReadAt
by reading from memory-mapped space instead of reading from file descriptor
...
This should improve performance when reading many small blocks.
2020-01-30 15:16:16 +02:00
Aliaksandr Valialkin
da19fffa08
all: rename ReadAt* to MustReadAt* in order to dont clash with io.ReaderAt
2020-01-30 15:16:16 +02:00
Aliaksandr Valialkin
1332ddc15e
lib/storage: pass missing AccountID and ProjectID to searchMetricName
2020-01-30 15:16:16 +02:00
Aliaksandr Valialkin
4ed5e9a7ce
lib/storage: pre-fetch metricNames for the found metricIDs in Search.Init
...
This should speed up Search.NextMetricBlock loop for big number of found time series.
2020-01-30 15:16:16 +02:00
Aliaksandr Valialkin
cb2a2f281f
lib/mergeset: properly update lastAccesstime
in indexBlockCache entries
...
This is a follow-up for 6665f10e7b
2020-01-29 21:21:01 +02:00
Aliaksandr Valialkin
170c1c3a4e
app/vmselect/promql: add keep_next_value(q)
for filling gaps with the next non-empty value
2020-01-29 00:48:14 +02:00
Aliaksandr Valialkin
a9c1d5b351
app/vminsert: moved -maxInsertRequestSize
command-line flag out of lib/prompb
in order to prevent its inclusion in vmselect
and vmstorage
apps
2020-01-28 22:53:50 +02:00
Aliaksandr Valialkin
81ba371eaf
lib/logger: fix improperly set skipframes for all the logging functions
2020-01-26 18:34:58 +02:00
Aliaksandr Valialkin
9f595cb2b1
lib/httpserver: log the caller of httpserver.Errorf
...
Previously log message contained `httpserver.Errorf`, not it contains the caller of `httpserver.Errorf`, which is more useful.
2020-01-25 20:18:06 +02:00
Aliaksandr Valialkin
36a1a21d6e
lib/protoparser: add parser for Prometheus exposition text format
...
This parser will be used by vmagent
2020-01-24 20:11:19 +02:00
Aliaksandr Valialkin
0cda6afa8e
app/vminsert: move ingestion protocol parsers to lib/protoparser, so they could be re-used in the upcoming vmagent
2020-01-24 16:55:18 +02:00
Aliaksandr Valialkin
ea53a21b02
all: consistently log durations in seconds with millisecond precision
...
This should improve logs readability
2020-01-22 18:35:24 +02:00
Aliaksandr Valialkin
40e564eb9c
app/vmselect/promql: add range_over_time(m[d])
function for calculating value range for m
over d
2020-01-21 19:05:29 +02:00
Aliaksandr Valialkin
9eaa2ab871
app/vmselect/promql: add label_match(q, label, regexp)
and label_mismatch(q, label, regexp)
functions for filtering out time series with labels matching the given regexp
2020-01-21 15:00:35 +02:00
Aliaksandr Valialkin
62b041e90a
lib/{mergeset,storage}: properly update lastAccessTime
in index and data block cache entries
2020-01-20 15:00:10 +02:00
Aliaksandr Valialkin
607d4418b8
lib/uint64set: add missing bucket32.b16his values
2020-01-18 14:26:23 +02:00
Aliaksandr Valialkin
e3379537cd
lib/uint64set: optimize Set.Union
...
This should improve performance for queries over big number of time series
2020-01-18 13:47:34 +02:00
Aliaksandr Valialkin
5077efd3f7
lib/uint64set: add benchmarks for Set.Union
2020-01-18 13:47:33 +02:00
Aliaksandr Valialkin
a851c75703
lib/storage: skip recovering timestamps order for lossless compression (PrecisionBits=64)
2020-01-17 23:59:19 +02:00
Aliaksandr Valialkin
2084921e64
all: use github.com/klauspost/compress/gzip
instead of compress/gzip
...
`github.com/klauspost/compress/gzip` is more optimized than `compress/gzip`.
This gives better gzip compression and decompression speeds.
2020-01-17 23:59:17 +02:00
Aliaksandr Valialkin
ab4d5d72eb
lib/uint64set: reduce memory allocations in Set.AppendTo
2020-01-17 22:33:00 +02:00
Aliaksandr Valialkin
476c7fb109
lib/storage: reduce memory allocations when merging metricID sets
2020-01-17 22:10:56 +02:00
Aliaksandr Valialkin
29d21259f0
lib/uint64set: typo fix in Set.Intersect
2020-01-17 18:11:46 +02:00
Aliaksandr Valialkin
ed1d259b10
lib/uint64set: optimize Intersect, Subtract and Union functions
...
This should improve performance for queries over big number of time series.
2020-01-17 16:16:43 +02:00
Aliaksandr Valialkin
68d35357b1
lib/uint64set: improve benchmark for Set.Intersect
2020-01-17 16:16:43 +02:00
Aliaksandr Valialkin
ffe352ad31
lib/uint64set: add benchmark for Set.Intersect
2020-01-17 16:16:43 +02:00
Aliaksandr Valialkin
4b16b7fd11
all: mention command-line flags used for limiting the incoming request size in error messages
...
This should improve error logs usability.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/287
2020-01-16 13:06:43 +02:00
Aliaksandr Valialkin
7d429e2806
lib/uint64set: reduce memory usage in Union, Intersect and Subtract methods
...
Iterate items with newly added Set.ForEach method instead of allocating `[]uint64`
slice for all the items before the iteration.
2020-01-15 12:15:48 +02:00
Aliaksandr Valialkin
caffb0cd01
lib/{mergeset,storage}: fix uint64 counters alignment for 32-bit architectures (GOARCH=386, GOARCH=arm)
2020-01-14 22:47:42 +02:00
Aliaksandr Valialkin
b03ccbf6f7
lib/{storage,mergeset}: gradually remove stale entries from block cache and index caches
...
This should reduce memory usage in the long run when old blocks and indexes
aren't accessed anymore.
2020-01-14 21:38:29 +02:00
Aliaksandr Valialkin
bcd3f0c5bd
app/vmselect/promql: add hoeffding_bound_upper(phi, m[d])
and hoeffding_bound_lower(phi, m[d])
functions
...
These functions can be used for calculating Hoeffding bounds
for `m` over `d` time range and for the given `phi` in the range `[0..1]`.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/283
2020-01-11 14:47:13 +02:00
Aliaksandr Valialkin
87a106702b
app/vmselect/promql: add aggr_over_time(("aggr_func1", "aggr_func2", ...), m[d])
function
...
This function can be used for simultaneous calculating of multiple `aggr_func*` functions
that accept range vector. For example, `aggr_over_time(("min_over_time", "max_over_time"), m[d])`
would calculate `min_over_time` and `max_over_time` for `m[d]`.
2020-01-10 21:18:12 +02:00
Aliaksandr Valialkin
c314d9a219
app/vmselect/promql: add tmin_over_time(m[d])
and tmax_over_time(m[d])
functions
...
These functions return timestamp in seconds for the minimum and maximum value for `m` over time range `d`
2020-01-10 19:39:34 +02:00
Aliaksandr Valialkin
1029b6ab34
lib/backup/s3remote: check whether the file exists before deleting it
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/284
2020-01-09 23:20:51 +02:00
Aliaksandr Valialkin
705af61587
app/{vmbackup,vmrestore}: add backup complete
file to backup when it is complete and check for this file before restoring from backup
...
This should prevent from restoring from incomplete backups.
Add `-skipBackupCompleteCheck` command-line flag to `vmrestore` in order to be able restoring from old backups without `backup complete` file.
2020-01-09 15:35:45 +02:00
Aliaksandr Valialkin
53e176ed67
lib/storage: limit maxRaRowsPerPartition by 500K for any number of rawRowsShardsPerPartition
...
This should reduce write amplification for high ingestion rate on multi-CPU systems
2020-01-04 23:58:23 +02:00
Aliaksandr Valialkin
89b551201c
lib/metricsql: export IsRollupFunc and IsTransformFunc, since they can be used by package users
2020-01-04 13:25:13 +02:00
Aliaksandr Valialkin
6f29d37cb5
app/vmselect/promql: add histogram_share(le, buckets)
function
2020-01-04 12:53:08 +02:00
Aliaksandr Valialkin
2290503140
app/vmselect/promql: add absent_over_time(m[d])
func similar to the function in Prometheus 2.16
...
See https://github.com/prometheus/prometheus/issues/2882
2020-01-04 12:53:01 +02:00
Aliaksandr Valialkin
67f94bbe12
app/vmselect/promql: add histogram_over_time(m[d])
rollup function
2020-01-04 12:52:56 +02:00
Aliaksandr Valialkin
588531dd76
lib/uint64set: reduce memory usage when storing big number of sparse metric_id values
2020-01-03 18:17:17 +02:00
Aliaksandr Valialkin
e0abf45d45
app/vmselect/promql: add share_le_over_time
and share_gt_over_time
functions for SLI and SLO calculations
2020-01-03 00:41:36 +02:00
Aliaksandr Valialkin
19962e2732
docs: refer to standalone MetricsQL package
2020-01-02 23:43:43 +02:00
Aliaksandr Valialkin
0d2e83e9d7
lib/metricsql: add example for ExpandWithExprs
2019-12-26 21:31:15 +02:00
Aliaksandr Valialkin
eb1a66c577
lib/metricsq: add ExpandWithExprs
2019-12-25 22:20:21 +02:00
Aliaksandr Valialkin
453d71d082
Rename lib/promql to lib/metricsql and apply small fixes
2019-12-25 22:09:09 +02:00
Mike Poindexter
009d1559db
Split Extended PromQL parsing to a separate library
2019-12-25 22:09:07 +02:00
Aliaksandr Valialkin
f22c9dbb0f
lib/fs: typo fix in fadvise_unix.go
2019-12-24 21:00:04 +02:00
Aliaksandr Valialkin
d3c185f0ca
lib/encoding: log the compressed block contents if it cannot be decompressed or unmarshaled
...
This should help detecting the root cause of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281
2019-12-24 20:48:25 +02:00
Aliaksandr Valialkin
091e35cf0c
lib/encoding: mention src contents in error message returned from unmarshalInt64NearestDelta*
...
This should simplify detecting the root cause of the issue at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281
2019-12-24 20:41:38 +02:00
Aliaksandr Valialkin
0e51058a0d
lib/encoding: mention unpacked block size in the error message if unparsed tail left
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281
2019-12-24 20:35:20 +02:00
Aliaksandr Valialkin
afa8b34d27
Revert "lib/logger: prevent from blocking when log output isn't consumed in timely manner"
...
This reverts commit 9f50232e70
.
Reason to revert: this leaves incomplete logs on app shutdown.
2019-12-24 12:20:45 +02:00
Aliaksandr Valialkin
6358cf3d47
app/vmselect/netstorage: move MustAdviseSequentialRead to lib/fs
2019-12-23 23:16:26 +02:00
Aliaksandr Valialkin
44f886cc9c
lib/encoding/zstd: typo fix
2019-12-23 18:37:20 +02:00
Aliaksandr Valialkin
108a60d69e
lib/encoding/zstd: call zstd.Decoder.Close instead of zstd.Decoder.Reset in order to free up occupied goroutines
...
This should fix goroutine leak for https://github.com/klauspost/compress/issues/195
2019-12-23 18:32:28 +02:00
Aliaksandr Valialkin
335bd0ac0a
lib/encoding/zstd: prevent from possible encoder leak when concurrent goroutines create encoders for the same compressionLevel
...
Thanks to @klauspost for the pointer to this issue. See https://github.com/klauspost/compress/issues/195 for details.
2019-12-23 18:06:02 +02:00
Aliaksandr Valialkin
9f50232e70
lib/logger: prevent from blocking when log output isn't consumed in timely manner
...
Drop log messages instead of blocking and increment `vm_log_messages_dropped_total` metric.
2019-12-20 11:49:42 +02:00
Aliaksandr Valialkin
a37a006f11
lib/storage: scale ingestion performance by sharding rawRows on systems with more than 8 CPU cores
2019-12-19 18:17:05 +02:00
Aliaksandr Valialkin
8d79412b26
lib/storage: optimize bulk import performance when multiple data points are inserted for the same time series
...
This should speed up `/api/v1/import` and make it more scalable on multi-core systems.
2019-12-19 15:13:36 +02:00
Aliaksandr Valialkin
05ec8afb3a
lib/httpserver: sync the code with master branch
2019-12-18 23:08:32 +02:00
Aliaksandr Valialkin
a7bf8e77af
app/vminsert: simultaneously accept telnet put
and HTTP /api/put
OpenTSDB metrics at -opentsdbListenAddr
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/266
2019-12-14 00:42:18 +02:00
Aliaksandr Valialkin
bc3984a5b3
lib/logger: add -loggerFormat
for choosing log message formats
...
Supported formats: default, json
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/265
2019-12-13 15:09:18 +02:00
Aliaksandr Valialkin
5d2ff573aa
app/vmselect/promql: allow negative offsets
...
Updates https://github.com/prometheus/prometheus/issues/6282
2019-12-11 00:57:51 +02:00
Aliaksandr Valialkin
3694efd005
lib/{mergeset,storage}: log info message when both source and destination part paths from txn are missing during startup
...
This is expected condition after unclean shutdown (OOM, hard reset, `kill -9`) on NFS disk.
2019-12-09 15:45:23 +02:00
Aliaksandr Valialkin
639967db59
lib/{mergeset,storage}: make sure pending transaction deletions are finished before and after runTransactions
call.
...
`runTransactions` call issues async deletions for transaction files. The previously issued transaction deletions
can race with the next call to `runTransactions`. Prevent this by waiting until all the pending transaction
deletions are funished in the beginning of `runTransactions`. Also make sure that all the pending transaction
deletions are finished before returning from `runTransactions`.
2019-12-04 21:40:52 +02:00
Aliaksandr Valialkin
7c0dd85a7c
lib/httpserver: add /ping
handler for compatibility with Influx agents
...
Certain Influx agents check for `/ping` endpoint before starting
to send Influx line protocol data. See https://docs.influxdata.com/influxdb/v1.7/tools/api/#ping-http-endpoint
2019-12-04 19:18:18 +02:00
Aliaksandr Valialkin
534da0a8c3
lib/storage: fall back to global inverted index if a filter match too many time series in per-day index
...
Previously this resulted to error message. The query may succeed via search in global index.
2019-12-03 14:48:08 +02:00
Aliaksandr Valialkin
6eb698d1cc
lib/storage: fix printing tag filters in TagFilters.String
2019-12-03 14:25:20 +02:00
Aliaksandr Valialkin
c04f60db35
lib/storage: print __name__
instead of empty string in user-visible tag filters
2019-12-03 14:18:18 +02:00
Aliaksandr Valialkin
625f6ca761
lib/storage: optimize regexp filter search
2019-12-03 00:33:53 +02:00
Aliaksandr Valialkin
b9616c017f
lib/{mergeset,storage}: remove transaction files only after the mentioned dirs are really removed
...
This should fix the issue on NFS when incompletely removed dirs may be left
after unclean shutdown (OOM, kill -9, hard reset, etc.), while the corresponding transaction
files are already removed.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162
2019-12-02 21:34:37 +02:00
Aliaksandr Valialkin
4e22b521c2
lib/storage: remove metricID with missing metricID->metricName entry
...
The metricID->metricName entry can be missing in the indexdb after unclean shutdown
when only a part of entries for new time series is written into indexdb.
Recover from such a situation by removing the broken metricID. New metricID
will be automatically created for time series with the given metricName
when new data point will arive to it.
2019-12-02 20:52:13 +02:00
Aliaksandr Valialkin
5a62415bec
lib/storage: protect from time drift during indexdb rotation
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/248
2019-12-02 14:43:11 +02:00
Aliaksandr Valialkin
cf85c567d1
lib/logger: merge file
and line
labels into location="file:line"
...
This should improve the usability for `vm_log_messages_total` metric during practical queries
2019-12-02 14:43:09 +02:00
Aliaksandr Valialkin
f055dbefda
lib/storage: generate more human-friendly result in TagFilters.String
2019-12-02 13:56:40 +02:00
Aliaksandr Valialkin
29f39f866e
lib/logger: consistency renaming from vm_log_messages_count
to vm_log_messages_total
, since this is a counter
2019-12-02 00:47:12 +02:00
Aliaksandr Valialkin
15eaff1745
lib/logger: track the number of log messages by (level, file, line)
in the vm_log_messages_count
metric
2019-12-01 18:38:30 +02:00
Aliaksandr Valialkin
d456ec7589
lib/netutil: use IPv6 for both listening and dialing if -enabledTCP6
is set
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/244
2019-12-01 02:52:53 +02:00
Aliaksandr Valialkin
7e734433a3
lib/backup: cosmetic fixes after #243
2019-11-29 18:07:41 +02:00
glebsam
4a192cb832
Add option to provide custom endpoint for S3, add option to specify S3 config profile ( #243 )
...
* Add option to provide custom endpoint for S3 for use with s3-compatible storages, add option to specify S3 config profile
* make fmt
2019-11-29 18:07:39 +02:00
Aliaksandr Valialkin
4810f1dde6
lib/netutil: add -enableTCP6
command-line flag for enabling listening for IPv6 additionally to IPv4 TCP ports
2019-11-29 17:33:07 +02:00
Aliaksandr Valialkin
409c939621
lib/backup: remove flock.lock
file in empty dirs
...
This fixes an issue when VictoriaMetrics doesn't see the restored data after the following operations:
1. Stop VictoriaMetrics.
2. Delete `<-storageDataPath>` dir.
3. Start VictoriaMetrics, then stop it.
4. Restore data from backup with `vmrestore`.
5. Start VictoriaMetrics.
`vmrestore` didn't delete properly empty dirs in `<-storageDataPath>/indexdb` because of the remaining `flock.lock` files in these dirs.
2019-11-28 13:39:28 +02:00
Aliaksandr Valialkin
0f184affa7
app/vmselect/promql: optimize binary search over big number of samples during rollup calculations
2019-11-25 14:01:54 +02:00
Aliaksandr Valialkin
d24fc87a6f
lib/decimal: calculate ln2/ln10 constant during compile time
2019-11-23 15:52:39 +02:00
Aliaksandr Valialkin
2af7ca1122
vendor: update github.com/VictoriaMetrics/metrics from v1.7.2 to v1.8.0. This version supports histograms
2019-11-23 00:21:57 +02:00
Aliaksandr Valialkin
b9e53490b9
lib/storage: move non-matching tag filters to the top at matchTagFilters
...
This should reduce the amount of useless work needed for matching the next metricNames.
2019-11-21 21:40:36 +02:00
Aliaksandr Valialkin
33d9d63393
lib/storage: speed up time series search for queries with multiple filters
...
Use optimized specialized binary search for uint64 metricIDs instead of generic sort.Search.
2019-11-21 18:43:40 +02:00
Aliaksandr Valialkin
a02a57fbe9
lib/storage: verify the number of returned metricIDs in BenchmarkHeadPostingForMatchers
2019-11-20 15:40:03 +02:00
Aliaksandr Valialkin
3d1f4408cf
lib/decimal: increase decimal->float speed conversion for integer numbers
2019-11-20 14:09:10 +02:00
Aliaksandr Valialkin
f1f2eff08f
lib/decimal: reduce rounding error when converting from decimal to float with negative exponent
...
While at it, slightly increase the conversion performance by moving fast path to the top of the loop.
2019-11-19 23:34:41 +02:00
Aliaksandr Valialkin
17eca31989
lib/backup: retrieve only the required metadata when reading GCS objects
2019-11-19 21:30:51 +02:00
Aliaksandr Valialkin
216a260ced
app/{vmbackup,vmrestore}: add -maxBytesPerSecond
command-line flag for limiting the used network bandwidth during backup / restore
2019-11-19 20:32:43 +02:00
Aliaksandr Valialkin
9d1ee1e2ae
lib/backup: prevent from restoring to directory which is in use by VictoriaMetrics during the restore
2019-11-19 18:35:59 +02:00
Aliaksandr Valialkin
6ca4b94511
lib/storage: increase the number of created time series in BenchmarkHeadPostingForMatchers in order to be on par with Promethues
...
The previous commit was accidentally creating 10x smaller number of time series than Prometheus
and this led to invalid benchmark results.
The updated benchmark results:
benchmark old ns/op new ns/op delta
BenchmarkHeadPostingForMatchers/n="1" 272756688 6194893 -97.73%
BenchmarkHeadPostingForMatchers/n="1",j="foo" 138132923 10781372 -92.19%
BenchmarkHeadPostingForMatchers/j="foo",n="1" 134723762 10632834 -92.11%
BenchmarkHeadPostingForMatchers/n="1",j!="foo" 195823953 10679975 -94.55%
BenchmarkHeadPostingForMatchers/i=~".*" 7962582919 100118510 -98.74%
BenchmarkHeadPostingForMatchers/i=~".+" 7589543864 154955671 -97.96%
BenchmarkHeadPostingForMatchers/i=~"" 1142371741 258003769 -77.42%
BenchmarkHeadPostingForMatchers/i!="" 9964150263 159783895 -98.40%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",j="foo" 216995884 10937895 -94.96%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",i!="2",j="foo" 202541348 10990027 -94.57%
BenchmarkHeadPostingForMatchers/n="1",i!="" 486285711 87004349 -82.11%
BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo" 350776931 53342793 -84.79%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo" 380888565 54256156 -85.76%
BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo" 89500296 21823279 -75.62%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo" 379529654 46671359 -87.70%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.*",j="foo" 424563825 53915842 -87.30%
VictoriaMetrics uses 1GB of RAM during the benchmark (vs 3.5GB of RAM for Prometheus)
2019-11-18 19:48:27 +02:00
Aliaksandr Valialkin
6f61fd367a
lib/storage: add BenchmarkHeadPostingForMatchers similar to the benchmark from Prometheus
...
See the corresponding benchmark in Prometheus - 23c0299d85/tsdb/head_bench_test.go (L52)
The benchmark allows performing apples-to-apples comparison of time series search
in Prometheus and VictoriaMetrics. The following article - https://www.robustperception.io/evaluating-performance-and-correctness -
contains incorrect numbers for VictoriaMetrics, since there wasn't this benchmark yet. Fix it.
Benchmarks can be repeated with the following commands from Prometheus and VictoriaMetrics source code roots:
- Prometheus: GOMAXPROCS=1 go test ./tsdb/ -run=111 -bench=BenchmarkHeadPostingForMatchers
- VictoriaMetrics: GOMAXPROCS=1 go test ./lib/storage/ -run=111 -bench=BenchmarkHeadPostingForMatchers
Benchmark results:
benchmark old ns/op new ns/op delta
BenchmarkHeadPostingForMatchers/n="1" 272756688 364977 -99.87%
BenchmarkHeadPostingForMatchers/n="1",j="foo" 138132923 1181636 -99.14%
BenchmarkHeadPostingForMatchers/j="foo",n="1" 134723762 1141578 -99.15%
BenchmarkHeadPostingForMatchers/n="1",j!="foo" 195823953 1148056 -99.41%
BenchmarkHeadPostingForMatchers/i=~".*" 7962582919 8716755 -99.89%
BenchmarkHeadPostingForMatchers/i=~".+" 7589543864 12096587 -99.84%
BenchmarkHeadPostingForMatchers/i=~"" 1142371741 16164560 -98.59%
BenchmarkHeadPostingForMatchers/i!="" 9964150263 12230021 -99.88%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",j="foo" 216995884 1173476 -99.46%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",i!="2",j="foo" 202541348 1299743 -99.36%
BenchmarkHeadPostingForMatchers/n="1",i!="" 486285711 11555193 -97.62%
BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo" 350776931 5607506 -98.40%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo" 380888565 6380335 -98.32%
BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo" 89500296 2078970 -97.68%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo" 379529654 6561368 -98.27%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.*",j="foo" 424563825 6757132 -98.41%
The first column (old) is for Prometheus, the second column (new) is for VictoriaMetrics.
Prometheus was using 3.5GB of RAM during the benchmark, while VictoriaMetrics was using 400MB of RAM.
2019-11-18 18:47:02 +02:00
Aliaksandr Valialkin
d297b65089
lib/storage: add vm_cache_size_bytes{type="storage/hour_metric_ids"}
metric
2019-11-13 20:26:05 +02:00
Aliaksandr Valialkin
494ad0fdb3
lib/storage: remove inmemory index for recent hour, since it uses too much memory
...
Production workload shows that the index requires ~4Kb of RAM per active time series.
This is too much for high number of active time series, so let's delete this index.
Now the queries should fall back to the index for the current day instead of the index
for the recent hour. The query performance for the current day index should be good enough
given the 100M rows/sec scan speed per CPU core.
2019-11-13 18:08:58 +02:00
Aliaksandr Valialkin
633dd81bb5
lib/storage: add -disableRecentHourIndex
flag for disabling inmemory index for recent hour
...
This may be useful for saving RAM on high number of time series aka high cardinality
2019-11-13 15:10:12 +02:00
Aliaksandr Valialkin
f1620ba7c0
lib/storage: fix inmemory inverted index issues found in v1.29
...
Issues fixed:
- Slow startup times. Now the index is loaded from cache during start.
- High memory usage related to superflouos index copies every 10 seconds.
2019-11-13 13:35:38 +02:00
Aliaksandr Valialkin
87b39222be
Revert "lib/fs: do not postpone directory removal on NFS error"
...
This reverts commit 21aeb02b46649ac9906cb37733f7b155a77a0db9.
2019-11-12 16:29:50 +02:00
Mike Poindexter
955a592106
Add test for invalid caching of tsids ( #232 )
...
* Add test for invalid caching of tsids
* Clean up error handling
2019-11-12 15:52:46 +02:00
Oleg Kovalov
74ba42d111
fix misspelled words ( #229 )
2019-11-12 00:18:24 +02:00
Aliaksandr Valialkin
c48e39eea9
lib/storage: add tests for dateMetricIDCache
2019-11-11 13:21:05 +02:00
Aliaksandr Valialkin
6bdde0d6d4
lib/storage: eliminate data race when updating lastSyncTime in dateMetricIDCache.Has
2019-11-10 22:04:23 +02:00
Aliaksandr Valialkin
5f52eb7653
lib/fs: do not postpone directory removal on NFS error
...
Continue trying to remove NFS directory on temporary errors for up to a minute.
The previous async removal process breaks in the following case during VictoriaMetrics start
- VictoriaMetrics opens index, finds incomplete merge transactions and starts replaying them.
- The transaction instructs removing old directories for parts, which were already merged into bigger part.
- VictoriaMetrics removes these directories, but their removal is delayed due to NFS errors.
- VictoriaMetrics scans partition directory after all the incomplete merge transactions are finished
and finds directories, which should be removed, but weren't still removed due to NFS errors.
- VictoriaMetrics panics when it finds unexpected empty directory.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162
2019-11-10 13:27:16 +02:00
Aliaksandr Valialkin
9ea2bd822e
lib/storage: implement per-day inverted index
2019-11-10 00:20:32 +02:00
Aliaksandr Valialkin
dea2f3efed
lib/storage: use specialized cache for (date, metricID) entries
...
This improves ingestion performance.
2019-11-09 23:09:18 +02:00
Aliaksandr Valialkin
9a43902bd8
lib/storage: remove unused code from getMetricIDsForTimeRange: it is expected that time range is always non-zero
2019-11-09 19:03:51 +02:00
Aliaksandr Valialkin
c16e17dede
lib/storage: properly set time range when deleting time series
2019-11-09 18:50:02 +02:00
Aliaksandr Valialkin
8126007c15
lib/storage: obtain all the time series ids from (tag->metricIDs) rows instead of (metricID->TSID) rows, since this much faster
2019-11-09 18:04:26 +02:00
Aliaksandr Valialkin
50773348d3
lib/storage: small code prettifying
2019-11-09 14:01:24 +02:00
Aliaksandr Valialkin
44fa8226df
lib/uint64set: remove superflouos check for item existence before deleting it in Set.Subtract
2019-11-09 14:01:24 +02:00
Aliaksandr Valialkin
0bc54c23ce
lib/storage: inmemoryInvertedIndex prettifying
2019-11-09 14:01:24 +02:00
Aliaksandr Valialkin
46e67bb78c
lib/storage: export vm_new_timeseries_created_total
metric for determining time series churn rate
2019-11-08 19:58:21 +02:00
Aliaksandr Valialkin
0063c857f5
lib/storage: add inmemory inverted index for the last hour
...
It should improve performance for `last N hours` dashboards with update intervals smaller than 1 hour.
2019-11-08 19:37:46 +02:00
Aliaksandr Valialkin
7d7fbf890e
app/{vmbackup,vmrestore}: add vmbackup
and vmrestore
tools for creating backups on s3 or gcs from instant snapshots
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/203
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/38
2019-11-07 21:26:43 +02:00
Aliaksandr Valialkin
89c03a5464
lib/storage: populate partition names from both small
and big
directories
...
Certain partition directories may be missing after restoring from backups
if they had no data. Re-create such directories on start.
2019-11-06 19:50:21 +02:00
Aliaksandr Valialkin
1c777e0245
lib/storage: substitute error message about unsorted items in the index block after metricIDs merge with counter
...
The origin of the error has been detected and documented in the code,
so it is enough to export a counter for such errors at `vm_index_blocks_with_metric_ids_incorrect_order_total`,
so it could be monitored and alerted on high error rates.
Export also the counter for processed index blocks with metricIDs - `vm_index_blocks_with_metric_ids_processed_total`,
so its' rate could be compared to `rate(vm_index_blocks_with_metric_ids_incorrect_order_total)`.
2019-11-06 14:32:41 +02:00
Aliaksandr Valialkin
c567a4353a
lib/storage: take into account the requested time range when caching TSIDs for the given tag filters
2019-11-06 14:32:41 +02:00
Aliaksandr Valialkin
c6564c5d26
lib/storage: dump incorrectly sorted items on a single line; this should simplify error reporting
2019-11-05 18:41:50 +02:00
Aliaksandr Valialkin
a10c4cad85
lib/storage: return back finalPartsToMerge from 2 to 3 in order to prevent from excessive merges in old partitions
2019-11-05 17:28:57 +02:00
Aliaksandr Valialkin
e5b1fa0c38
lib/storage: separate the max inverted index scan loops per metric into fast and slow loops
...
Slow loops could require seeks and expensive regexp matching, while fast loops just scans
all the metricIDs for the given `tag=value` prefix. So these operations must have separate
max loops multiplier.
2019-11-05 17:28:57 +02:00
Aliaksandr Valialkin
f93c4f2493
lib/storage: skip repeated useless work when intersection of metricIDs with the given filter is too expensive
...
This should improve performance for query filters over big number of time series.
2019-11-05 14:35:55 +02:00
Aliaksandr Valialkin
f48e97263c
lib/storage: reduce the maximum inverted index scans before giving up to label filters matching by metric name
...
The new value reduces the amount of wasted work during index scans over big number of time series.
2019-11-05 14:35:53 +02:00
Aliaksandr Valialkin
d2f688c550
lib/storage: try potentially faster tag filters at first, then apply slower tag filters
...
The fastest tag filters are non-negative non-regexp, since they are the most specific.
The slowest tag filters are negative regexp, since they require scanning
all the entries for the given label.
2019-11-05 14:35:48 +02:00
Aliaksandr Valialkin
2a38d30f93
lib/storage: pass pointer to MetricName in Fatalf, so it is properly detected as an interface with String() method
...
This fixes lint errors
2019-11-04 01:06:45 +02:00
Artem Navoiev
e05500cbd4
add unittests for bytesutil and storage ( #221 )
2019-11-04 00:57:24 +02:00
Aliaksandr Valialkin
f5fbc3ffd7
lib/{storage,uint64set}: add Set.Union() function and use it
2019-11-04 00:48:32 +02:00
Aliaksandr Valialkin
23e078261e
lib/storage: tune the returned value from adjustMaxMetricsAdaptive
2019-11-04 00:45:28 +02:00
Aliaksandr Valialkin
386c349c8c
lib/storage: remove interface conversion in hot path during block merging
...
This should improve merge speed a bit for parts with big number of small blocks.
2019-11-03 12:33:48 +02:00
Aliaksandr Valialkin
26ffc77622
lib/{storage,mergeset}: create missing partition directories after restoring from backups
...
Backup tools could skip empty directories. So re-create such directories on the first run.
2019-11-02 02:27:19 +02:00
Aliaksandr Valialkin
5d439cc6f2
lib/{decimal,encoding}: optimize float64<->decimal conversion for arrays with zeros or ones
...
Time series with only zeros or ones frequently occur in monitoring, so it is worth optimizing their handling.
2019-11-01 16:46:08 +02:00
Aliaksandr Valialkin
1037053fed
lib/{encoding,decimal}: add benchmarks for blocks containing zeros or ones
...
Time series with such values are quite common in monitoring space,
so it would be great to have benchmarks for them.
2019-11-01 16:46:08 +02:00
Aliaksandr Valialkin
44fab198e2
lib/uint64set: return an emptry set instead of nil set from Set.Clone
, since the caller may add data to the cloned set
...
This fixes the following panic in v1.28.1:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x783a7e]
goroutine 1155 [running]:
github.com/VictoriaMetrics/VictoriaMetrics/lib/uint64set.(*Set).Add(0x0, 0x15b3bfb41e8b71ec)
github.com/VictoriaMetrics/VictoriaMetrics@/lib/uint64set/uint64set.go:57 +0x2e
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*indexSearch).getMetricIDsForRecentHours(0xc5bdc0dd40, 0x16e273f6b50, 0x16e2745d3f0, 0x5b8d95, 0x10, 0x4a2f51, 0xaa01000000000000)
github.com/VictoriaMetrics/VictoriaMetrics@/lib/storage/index_db.go:1951 +0x260
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*indexSearch).getMetricIDsForTimeRange(0xc5bdc0dd40, 0x16e273f6b50, 0x16e2745d3f0, 0x5b8d95, 0x10, 0xb296c0, 0xc00009cd80, 0x9bc640)
2019-11-01 16:12:21 +02:00
Aliaksandr Valialkin
6ab9c98a1e
app/vmstorage: add -bigMergeConcurrency
and -smallMergeConcurrency
flags for tuning the maximum number of CPU cores used during merges
2019-10-31 16:17:29 +02:00
Aliaksandr Valialkin
6a22727676
lib/storage: optimize getMetricIDsForRecentHours for per-tenant lookups
2019-10-31 15:51:09 +02:00
Aliaksandr Valialkin
ca480915ca
lib/storage: small cleanup in Storage.add
2019-10-31 14:30:22 +02:00
Aliaksandr Valialkin
22030b558f
lib/decimal: speed up FromFloat for common case with integers
2019-10-31 13:25:09 +02:00
Aliaksandr Valialkin
6510258a80
lib/decimal: increase float64->decimal conversion precision a bit
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/213
2019-10-30 02:04:28 +02:00
Aliaksandr Valialkin
a27e034a40
lib/storage: get parts to merge after applying the limit on the number of concurrent merges
...
This should reduce write amplification under high ingestion rate.
2019-10-30 00:09:44 +02:00
Aliaksandr Valialkin
5d2276dbf7
lib/{mergeset,storage}: limit the maximum number of concurrent merges; leave smaller number of parts during final merge
2019-10-29 12:45:37 +02:00
Aliaksandr Valialkin
f581b2736a
lib/fs: typo fix in comment to WriteFileAtomically
2019-10-29 11:31:34 +02:00
Aliaksandr Valialkin
eb513e7ba3
lib/decimal: increase float->decimal conversion precision for big numbers
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/213
2019-10-28 13:23:54 +02:00
hanzai
52778da1f3
warns during rows addition ( #214 )
2019-10-20 23:38:51 +03:00
Aliaksandr Valialkin
78fc35c9b1
all: make fmt
2019-10-17 20:05:12 +03:00
Aliaksandr Valialkin
5b01b7fb01
all: add support for GOARCH=386 and fix all the issues related to 32-bit architectures such as GOARCH=arm
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
2019-10-17 18:27:49 +03:00
Aliaksandr Valialkin
12c8afc3f2
lib/memory: properly handle int overflow in sysTotalMemory
...
This should fix builds on 32-bit architectures such as arm.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
2019-10-17 18:27:49 +03:00
Aliaksandr Valialkin
dc2f822577
lib/prompb: removed outdated README.md
2019-10-14 22:16:36 +03:00
Aliaksandr Valialkin
661b8ede5b
lib/storage: harden the check that the original items are sorted after mergeTagToMetricIDsRows fails to preserve sort order
2019-10-09 12:13:43 +03:00
Aliaksandr Valialkin
7f4a04ee6a
lib/storage: typo fix in comment to maxRowsPerSmallPart.
2019-10-08 18:51:56 +03:00
Aliaksandr Valialkin
7e410e1412
lib/storage: add tests for mergeTagToMetricIDsRows and return the original items if the function breaks items` ordering.
...
This should save from data corruption issues revealed in the previous releases up to v1.28.0-beta5.
2019-10-08 16:35:39 +03:00
Aliaksandr Valialkin
9fce611fbb
lib/mergeset: reduce the maximum number of cached blocks, since there are reports on OOMs due to too big caches
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/189
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/195
2019-09-30 12:27:30 +03:00
Aliaksandr Valialkin
d0b4590099
lib/storage: optimize TSID comparison
2019-09-26 14:20:02 +03:00
Aliaksandr Valialkin
95e3d648cb
lib/storage: verify whether items are sorted in the end of call to mergeTagToMetricIDsRows
...
This should prevent from inverted index corruption if bug in mergeTagToMetricIDsRows is discovered.
2019-09-26 13:13:58 +03:00
Aliaksandr Valialkin
2b8358726f
lib/storage: properly match labels against regexp with (?i)
flag
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/161
2019-09-26 11:03:26 +03:00
Aliaksandr Valialkin
4e3871ac1e
lib/storage: add missing break in removeDuplicateMetricIDs
2019-09-25 18:23:13 +03:00
Aliaksandr Valialkin
4468f9f966
lib/storage: remove duplicate MetricIDs in tag->metricIDs
items before writing them into inverted index
2019-09-25 17:57:36 +03:00
Aliaksandr Valialkin
adc18c3ee6
lib/{mergeset,storage}: do not cache inverted index blocks containing tag->metricIDs
items
...
This should reduce the amounts of used RAM during queries with filters over big number of time series.
2019-09-25 13:48:24 +03:00
Aliaksandr Valialkin
c64fb91a43
lib/uint64set: optimize Set.AppendTo
2019-09-25 00:34:31 +03:00
Aliaksandr Valialkin
de0e4eee2c
lib/storage: create and use lib/uint64set
instead of map[uint64]struct{}
...
This should improve inverted index search performance for filters matching big number of time series,
since `lib/uint64set.Set` is faster than `map[uint64]struct{}` for both `Add` and `Has` calls.
See the corresponding benchmarks in `lib/uint64set`.
2019-09-24 21:18:04 +03:00
Aliaksandr Valialkin
2212d0e421
lib/storage: typo fix: return dstData instead of data from mergeTagToMetricIDsRows
2019-09-24 19:32:58 +03:00
Aliaksandr Valialkin
9307de1b92
lib/storage: limit the number of metricIDs in tag->metricIDs row
...
This reduces the overhead on index and metaindex in lib/mergeset
2019-09-24 00:50:47 +03:00
Aliaksandr Valialkin
7734fc8012
lib/storage: share tsids across all the partSearch instances
...
This should reduce memory usage when big number of time series matches the given query.
2019-09-23 22:36:16 +03:00
Aliaksandr Valialkin
67a2bcb98a
lib/{storage,mergeset}: verify PrepareBlock callback results
...
Do not touch the first and the last item passed to PrepareBlock
in order to preserve sort order of mergeset blocks.
2019-09-23 20:46:33 +03:00
Aliaksandr Valialkin
3304dc1e85
lib/mergeset: detect whether we are in test by executable suffix
2019-09-22 23:12:35 +03:00
Aliaksandr Valialkin
d2ed8cb0b2
lib/storage: generate the first tag->metricIDs item in a mergeset block with a single metricID
...
The first item from each mergeset block goes into index (lib/mergeset.blockHeader),
so it must be short in order to reduce index size.
2019-09-22 19:37:50 +03:00
Aliaksandr Valialkin
0a9cb6368e
lib/workingsetcache: remove data race when resetting c.misses
2019-09-22 19:37:09 +03:00
Aliaksandr Valialkin
7d13c31566
lib/{storage,mergeset}: merge tag->metricID
rows into tag->metricIDs
rows for common tag
values
...
This should improve lookup performance if the same `label=value` pair exists
in big number of time series.
This should also reduce memory usage for mergeset data cache, since `tag->metricIDs` rows
occupy less space than the original `tag->metricID` rows.
2019-09-20 22:06:23 +03:00
Aliaksandr Valialkin
272e2f77c9
lib/encoding: optimize UnmarshalUint* and UnmarshalInt*
2019-09-20 13:08:24 +03:00
Aliaksandr Valialkin
7e0c6d4ca6
lib/storage: optimize selecting all the metricIDs by scanning MetricID->TSID entries instead of tag->MetricID entries
...
The number of MetricID->TSID entries is smaller than the number of tag->MetricID entries
and MetricID->TSID entries are usually shorter than tag->MetricID entries.
This should improve performance when selecting all the metricIDs.
2019-09-20 11:57:57 +03:00
Aliaksandr Valialkin
bf8505353a
lib/mergeset: rename misleading mergeSmallParts to mergeExistingParts
2019-09-19 21:48:36 +03:00
Aliaksandr Valialkin
ebbef20535
lib/mergeset: use sort.IsSorted instead of sort.SliceIsSorted in inmemoryBlock.isSorted in order to reduce memory allocations
2019-09-19 20:13:54 +03:00
Aliaksandr Valialkin
89234f395d
lib/storage: use sort.Sort instead of sort.slice in getSortedMetricIDs
2019-09-19 20:08:13 +03:00
Aliaksandr Valialkin
6e586fa09c
lib/storage: skip duplicate call to intersectMetricIDsWithTagFilter on zero successful intersects
2019-09-19 17:51:10 +03:00
Aliaksandr Valialkin
410f993bf6
lib/mergeset: fill partHeader.firstItem on first block flush
2019-09-19 17:48:22 +03:00
Aliaksandr Valialkin
c05885fb5f
lib/storage: mark tag filter returning errFallbackToMetricNameMatch as useless
...
This will save CPU on subsequent calls for this filter
2019-09-18 19:11:44 +03:00
Aliaksandr Valialkin
db71c940ea
lib/storage: properly construct keys for uselessTagFiltersCache and register useless negative tag filters there
2019-09-17 23:18:37 +03:00
Aliaksandr Valialkin
41ef6b060e
lib/mergeset: properly check for sorted block headers
...
Fix a typo for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/181
2019-09-13 21:59:38 +03:00
hanzai
08cde5e3f4
lib/workingsetcache: adjust switching from mode=split
to mode=whole
smoothly and load cachefile successfully
2019-09-13 19:13:16 +03:00
Aliaksandr Valialkin
b101064f8b
all: report the number of bytes read on io.ReadFull error
...
This should simplify error investigation similar to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/175
2019-09-11 14:50:24 +03:00
Aliaksandr Valialkin
568ff61dcf
lib/mergeset: dynamically calculate the maximum number of items per part, which can be cached in OS page cache
2019-09-09 11:42:45 +03:00
Aliaksandr Valialkin
2c654258ef
lib/fs: add MustStopDirRemover for waiting until pending directories are removed on graceful shutdown
...
This patch is mainly required for laggy NFS. See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162
2019-09-05 11:17:17 +03:00
Aliaksandr Valialkin
2c2bd897dd
lib/storage: remove duplicate tag keys on MetricName.Marshal
call
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/172
2019-09-04 18:13:51 +03:00
Aliaksandr Valialkin
82bfe818d0
lib/fs: try harder with directory removal on NFS in the event of temporary lock
...
Do not give up after 11 attempts of directory removal on laggy NFS.
Add `vm_nfs_dir_remove_failed_attempts_total` metric for counting the number of failed attempts
on directory removal.
Log failed attempts on directory removal after long sleep times.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162
2019-09-04 12:24:41 +03:00
Aliaksandr Valialkin
0b0153ba3d
lib/storage: invalidate tagFilters -> TSIDS
cache when newly added index data becomes visible to search
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/163
2019-08-29 15:08:44 +03:00
Aliaksandr Valialkin
8504a38214
lib/prombp: apply ba06b47c16
...
The following commands used:
gofmt -r '(uint64(x)&0x7F)<<shift -> uint64(x&0x7F)<<shift' -w ./lib/prompb/
gofmt -r '(int64(x)&0x7F)<<shift -> int64(x&0x7F)<<shift' -w ./lib/prompb/
2019-08-29 13:35:54 +03:00
Aliaksandr Valialkin
604a4312f9
all: port to FreeBSD on GOARCH=amd64
2019-08-28 01:46:09 +03:00
Aliaksandr Valialkin
da07a6fb38
lib/storage: go fmt
2019-08-27 14:28:24 +03:00
Aliaksandr Valialkin
a63b69e9e2
lib/storage: report proper maxMetrics limit when more than -search.maxUniqueTimeseries series match the given filters
2019-08-27 14:21:31 +03:00
Aliaksandr Valialkin
82e813bad3
lib/storage: properly handle (?i)
in the tag filter regexp
...
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/161
2019-08-26 00:44:56 +03:00
Aliaksandr Valialkin
e2eac858b5
lib/storage: calculate the maximum number of rows per small part from -memory.allowedPercent
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/159
This simplifies error detection additionally to the `vm_rows_ignored_total` counters.
2019-08-25 15:29:09 +03:00
Aliaksandr Valialkin
0a8dd9cc9a
lib/storage: calculate the maximum number of rows per small part from -memory.allowedPercent
...
This should improve query speed over recent data on machines with big amounts of RAM
2019-08-25 14:41:32 +03:00
Aliaksandr Valialkin
bc576fb386
lib/storage: properly limit the number of output rows in small and big parts storage
...
Previously small parts storage didn't take into account the available disk space for big parts.
2019-08-25 14:41:32 +03:00
Aliaksandr Valialkin
947decb3dd
lib/storage: remove outdated comment on maxRowsPerSmallPart
...
The commend became outdated after the commit ed6ac1a5df027f0dfc22448e3b27c26b6f77c67a,
which stops merging of small parts on graceful shutdown instead of waiting
for their completion.
2019-08-25 13:46:10 +03:00
Aliaksandr Valialkin
c197641978
all: return 503 http error if service is temporarily unavailable
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/156
2019-08-23 09:49:50 +03:00
Aliaksandr Valialkin
e734076f0f
app/vminsert: allow setting the maximum number of labels per time series via -maxLabelsPerTimeseries
2019-08-23 08:47:18 +03:00
Aliaksandr Valialkin
4ed63d033a
lib/storage: add benchmarks for regexp filter match / mismatch
...
These benchmarks allow estimate the performance of regexp filters in promql
2019-08-22 16:37:19 +03:00
Aliaksandr Valialkin
6ec6a8d7c1
lib/storage: try slower path for searching the tag filter with the minimum number of matching time series before giving up with increase -search.maxUniqueTimeseries
error
2019-08-19 16:07:05 +03:00
Aliaksandr Valialkin
c59f5c4865
lib/storage: pre-allocate memory for blockHeader slice in unmarshalBlockHeaders
...
This reduces memory usage and memory fragmentation when working with big number of time series
2019-08-19 12:46:45 +03:00
Aliaksandr Valialkin
f696cc503a
lib/workingsetcache: automatically detect when it is better to double cache capacity
2019-08-15 22:58:04 +03:00
Aliaksandr Valialkin
99eed2ca14
lib/storage: properly cache tagFilters -> TSIDs entries from historical index
2019-08-14 02:32:25 +03:00
Aliaksandr Valialkin
f1d81b9405
lib/storage: compress contents of cache for tagFilters -> TSIDs
...
This should increase cache capacity
2019-08-14 02:32:22 +03:00
Aliaksandr Valialkin
8c2158af24
all: use workingsetcache instead of fastcache
...
This should reduce the amount of RAM required for processing time series
with non-zero churn rate.
The previous cache behavior can be restored with `-cache.oldBehavior` command-line flag.
2019-08-13 21:40:28 +03:00
Aliaksandr Valialkin
51263b1a45
lib/fs: add test for IsTemporaryFileName
2019-08-13 21:33:54 +03:00
Aliaksandr Valialkin
5a7ab0d90b
lib/storage: remove broken BenchmarkIndexDBSearchTSIDs
2019-08-13 20:21:23 +03:00
Aliaksandr Valialkin
39f3f3a517
lib: move common code for creating flock.lock file into fs.CreateFlockFile
2019-08-13 01:46:20 +03:00
Aliaksandr Valialkin
73f866d874
lib/fs: atomically create file with the given contents on WriteFileAtomically
...
This should prevent from `transaction` and `metadata.json` files corruption
on unclean shutdown such as OOM, `kill -9`, power loss, etc.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/148
2019-08-12 15:02:04 +03:00
Aliaksandr Valialkin
4fb635b0c9
lib/storage: do not change timestamps to constant rate if values are constant or have constant delta
...
This breaks the original timestamps, which results in issues like
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/120 and
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/141 .
2019-08-06 15:40:17 +03:00
Aliaksandr Valialkin
f56c1298ad
app/vmstorage: add vm_concurrent_addrows_*
metrics for tracking concurrency for Storage.AddRows calls
...
Track also the number of dropped rows due to the exceeded timeout
on concurrency limit for Storage.AddRows. This number is tracked in `vm_concurrent_addrows_dropped_rows_total`
2019-08-06 15:08:43 +03:00
Aliaksandr Valialkin
a3ecf3c1f7
lib/storage: properly reset partSearch.fetchData
in partSearch.reset
2019-08-05 09:55:50 +03:00
Aliaksandr Valialkin
880b1d80b1
app/vmselect: optimize /api/v1/series
by skipping storage data
...
Fetch and process only time series metainfo.
2019-08-04 23:00:46 +03:00
Aliaksandr Valialkin
a99e89945e
lib/decimal: modernize tests a bit
2019-07-31 21:09:54 +03:00
Aliaksandr Valialkin
b7c4b0c6d2
lib/storage: fix matching against tag filter with empty name
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/137
2019-07-30 15:15:21 +03:00
Aliaksandr Valialkin
c6bec48927
lib/storage: add metrics for calculating skipped rows outside the retention
...
The metrics are:
- vm_too_big_timestamp_rows_total
- vm_too_small_timestamp_rows_total
2019-07-26 14:11:56 +03:00
Aliaksandr Valialkin
f2e8d54fb0
lib/encoding/zstd: go fmt
2019-07-25 01:37:57 +03:00
Aliaksandr Valialkin
97b5dc7122
lib/encoding/zstd: disable CRC checks in pure Go
build
...
This should give slightly better compression and decompressions performance.
Additionally this shaves off 4 bytes per each compressed block.
2019-07-24 19:17:32 +03:00
Aliaksandr Valialkin
54f035d4ce
all: small updates after PR #114
2019-07-24 17:43:43 +03:00
Aliaksandr Valialkin
7a133567fb
lib/encoding: small fixes in tests after the PR #114
2019-07-24 17:43:39 +03:00
Roman Khavronenko
fcf09aaa3c
all: add Pure Go build (pull request #114 )
...
Updates #94
2019-07-24 17:43:32 +03:00
Aliaksandr Valialkin
3fae34eeb4
lib/encoding: improve gauge series detection
...
- Series with negative values are always gauges
- Counters may only have increasing values with possible counter resets
This should improve compression ratio for gauge series which
were previously mistakenly detected as counters.
2019-07-20 14:05:25 +03:00
Aliaksandr Valialkin
7abb96b454
lib/netutil: do not count timeouts as network errors
2019-07-15 23:06:13 +03:00
Aliaksandr Valialkin
73a47d2a53
lib/storage: remove unused function isTooBigTimeRangeForDateMetricIDs
2019-07-12 02:28:40 +03:00
Aliaksandr Valialkin
97f9397687
lib/storage: do not reduce maxMetrics
on time ranges exceeding maxDaysForDateMetricIDs
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/95
2019-07-12 02:21:52 +03:00
Aliaksandr Valialkin
4ca66344ee
lib/storage: do not pollute inverted index with data for samples outside the retention period
2019-07-11 17:11:33 +03:00
Aliaksandr Valialkin
0522efb2d6
lib/storage: add missing tagFilter.Marshal func
2019-07-11 15:01:01 +03:00
Aliaksandr Valialkin
12b1d67b41
lib/storage: use fast path for orSuffix when searching for metricIDs against plain tag value
2019-07-11 14:48:51 +03:00
Aliaksandr Valialkin
bf2e1b0ac1
lib/storage: remember and skip individual tag filters matching too many metrics
...
This saves CPU time by skipping useless matching for individual tag filters.
2019-07-11 14:48:47 +03:00
Aliaksandr Valialkin
ba8195c58e
all: consistency renaming: bytesSize -> sizeBytes
2019-07-10 00:47:42 +03:00
Aliaksandr Valialkin
41f512af1c
all: add vm_data_size_bytes
metrics for easy monitoring of on-disk data size and on-disk inverted index size
2019-07-04 19:43:04 +03:00
Aliaksandr Valialkin
6682a35731
lib/memory: attempt #3 to determine memory limit for LXC container
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/84
2019-07-01 14:01:57 +03:00
Aliaksandr Valialkin
b6ea1a7d5e
lib/mergeset: make fmt
2019-06-29 14:25:46 +03:00
Aliaksandr Valialkin
ffc1bb00f6
lib/storage: skip non-matching metricIDs in sortedFilter
...
This should improve performance for big sorteFilter lists.
2019-06-29 13:49:40 +03:00
Aliaksandr Valialkin
2257dcd278
lib/mergeset: speed up binarySearchKey by skipping the first item during binary search
2019-06-29 13:49:32 +03:00
Aliaksandr Valialkin
6ea12a079e
lib/memory: remove TestReadLXCMemoryLimit, since it doesnt work in Travis
2019-06-28 18:23:06 +03:00
Aliaksandr Valialkin
d0732d3137
lib/memory: attempt #2 to determine memory limit inside LXC container
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/84
2019-06-28 18:08:57 +03:00
Aliaksandr Valialkin
ad436757c3
lib/memory: an attempt to read proper memory limit inside LXC container
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/84
2019-06-28 15:35:02 +03:00
Aliaksandr Valialkin
416d27ef11
lib/storage: optimize time series search by regexp filter
...
This should improve search speed on label filters like `{foo=~"bar.+baz"}`
2019-06-27 16:18:00 +03:00
Aliaksandr Valialkin
ee23a143b9
lib/storage: make sure non-nil args are passed to openIndexDB
2019-06-25 20:10:08 +03:00
Aliaksandr Valialkin
8b0a63722f
lib/storage: reduce too big maxMetrics in getTagFilterWithMinMetricIDsCountAdaptive
...
This should improve performance on inverted index search for big amount of unique time series
when big -search.maxUniqueTimeseries is set.
2019-06-25 19:57:31 +03:00
Aliaksandr Valialkin
0263cb0adc
lib/storage: free up memory from caches owned by indexDB when it is deleted
2019-06-25 14:41:16 +03:00
Aliaksandr Valialkin
362e187011
lib/storage: use unversioned keys for tag cache in extDB
...
Data in ExtDB cannot be changed, so it is OK to use unversioned keys for tag cache.
This should improve performance for index lookups over big amount of time series.
2019-06-25 13:15:42 +03:00
Aliaksandr Valialkin
51e2f3b48f
lib/storage: skip searching in extDB if it doesn't contain items for the given time range
...
This should improve inverted index search performance for big amount
of unique time series when the search is performed only on recent data.
2019-06-25 12:57:56 +03:00
Aliaksandr Valialkin
fb9358635d
lib/storage: mention source parts on merge error
...
This should improve determining broken source part.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/76
2019-06-24 14:09:46 +03:00
Aliaksandr Valialkin
9164c223ec
all: initial stubs for Windows support; see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70
2019-06-20 20:07:41 +03:00
Aliaksandr Valialkin
786beb8fc8
Makefile: enable golangci-lint in make check_all
2019-06-20 15:00:58 +03:00
Aliaksandr Valialkin
9cac11db64
lib/storage: typo fixes found by golangci-lint; updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/69
2019-06-20 14:38:45 +03:00
Aliaksandr Valialkin
7778030f9f
lib/netutil: remove unused TCPListener.name; updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/69
2019-06-20 14:36:19 +03:00
Aliaksandr Valialkin
a0c22a6830
app/vmstorage: add vm_cache_entries{type="storage/hour_metric_ids"}
metric for tracking active time series count
2019-06-19 18:37:38 +03:00
Aliaksandr Valialkin
24ae3ef532
lib/prompb: remove superflouos bytes copying in ReadSnappy
2019-06-18 21:02:02 +03:00
Aliaksandr Valialkin
e40224d5de
lib/flagutil: add NewArray helper func
2019-06-18 10:44:09 +03:00
Aliaksandr Valialkin
f9e1d32168
lib/storage: persist metric ids for the current and the previous hour on graceful shutdown
...
This should improve performance after restart when the db contains a lot of time series
with high time series churn (i.e. metrics from Kubernetes with many pods and frequent deployments)
2019-06-14 07:55:09 +03:00
Aliaksandr Valialkin
b0b93e3d50
lib/fs: sync parent dir in MustRemoveAll only if it exists
...
The parent directory may be non-existing when the deleted directory
didn't exist before the MustRemoveAll call
2019-06-12 02:16:15 +03:00
Aliaksandr Valialkin
18d6f293f7
lib/fs: consolidate *RemoveAll* funcs into a single MustRemoveAll func
...
The func syncs parent dir in order to persist directory removal
in the event of power loss
2019-06-12 01:55:18 +03:00
Aliaksandr Valialkin
28d9904efc
lib/fs: panic with fatal error when directories cannot be removed
...
Unremoved directories may lead to inconsistent data directory,
so VictoriaMetrics will fail to start next time.
So panic on the first error when trying to remove directory in order
to simplify recover process.
2019-06-12 01:20:10 +03:00
Aliaksandr Valialkin
d897bc3f08
lib/fs: attempt #2 to work around NFS issue with directory removal
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/61
2019-06-12 01:07:29 +03:00
Aliaksandr Valialkin
51e2e255a6
lib/fs: consistency renaming SyncPath -> MustSyncPath, since it doesnt return error
2019-06-11 23:13:45 +03:00
Aliaksandr Valialkin
3fa4c28f6b
lib/fs: make sure the created directory remains visible in the fs in the event of power loss
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/63
2019-06-11 23:08:17 +03:00
Aliaksandr Valialkin
0b7f751f60
lib/fs: use filepath.Dir instead of filepath.Split, since the filename is unused
2019-06-11 22:54:23 +03:00
Aliaksandr Valialkin
b491045a4b
lib/{storage,mergeset}: sync filenames inside part when finalizing the part
...
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/63
2019-06-11 21:51:19 +03:00
Aliaksandr Valialkin
3437c30180
all: try hard removing directory with contents
...
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/61
2019-06-11 01:58:08 +03:00
Aliaksandr Valialkin
75a0acf72d
app/vmselect: add /api/v1/labels/count
handler for quick detection of labels with the maximum number of distinct values
2019-06-10 19:54:55 +03:00
Aliaksandr Valialkin
0ccedbdfd2
lib/storage: mention the accountID and projectID in error message when filtering out other (accountID, projectID) entries
2019-06-10 14:43:53 +03:00
Aliaksandr Valialkin
d54f5fec0b
lib/storage: skip adaptive searching for tag filter matching the minimum number of metrics if the identical previous search didn't found such filter
...
This should improve speed for searching metrics among high number of time series
with high churn rate like in big Kubernetes clusters with frequent deployments.
2019-06-10 14:07:47 +03:00
Aliaksandr Valialkin
27e50e86f4
lib/storage: factor out getTagFilterWithMinMetricIDsCountAdaptive from updateMetricIDsForTagFilters
2019-06-10 13:26:00 +03:00
Aliaksandr Valialkin
b69d3dbd0c
lib/storage: filter out metricIDs from another (AccountID, ProjectID) in getMetricIDsForRecentHours
2019-06-10 13:05:16 +03:00
Aliaksandr Valialkin
3059ae7be0
lib/storage: give clearer names to more functions
2019-06-10 12:59:33 +03:00
Aliaksandr Valialkin
d3a024d2d6
lib/storage: give more clear names to functions
2019-06-10 12:50:22 +03:00
Aliaksandr Valialkin
00e0760608
lib/storage: test GetSeriesCount
2019-06-10 12:40:33 +03:00
Aliaksandr Valialkin
e4cba5a7ed
lib/storage: make getSeriesCount func indexSearch method
2019-06-10 12:29:24 +03:00
Aliaksandr Valialkin
d882afa905
lib/storage: optimize time series lookup for recent hours when the db contains many millions of time series with high churn rate (aka frequent deployments in Kubernetes)
2019-06-09 19:14:04 +03:00
Aliaksandr Valialkin
8cf0a0e59c
app/vminsert: split vm_rows_inserted_total into per-(accountID, projectID) metrics
...
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/59
2019-06-07 22:11:20 +03:00
Aliaksandr Valialkin
1d87abc8eb
lib/procutil: typo fix in comment to WaitForSigterm
2019-06-03 16:54:37 +03:00
Aliaksandr Valialkin
a2986cde70
lib/storage: tune updating a map with today`s metric ids
...
- Increase update iterval from 1s to 10s. This should reduce CPU usage
for large amounts of metric ids with constant churn.
- Reduce pendingTodayMetricIDsLock lock duration during the update.
2019-06-02 22:00:13 +03:00
Aliaksandr Valialkin
e27fd5148a
lib/storage: speed up checking metricID existence in the list for the current date
2019-06-02 18:34:20 +03:00
Aliaksandr Valialkin
a6d02ff275
lib/timerpool: use timer pool in concurrency limiters
...
This should reduce the number of memory allocations in highly loaded system
2019-05-28 17:30:10 +03:00
Aliaksandr Valialkin
bdf696ef18
all: fix misspellings
2019-05-25 21:51:24 +03:00
Aliaksandr Valialkin
121a920a18
Makefile: add -s flag to go fmt
in make fmt
command
2019-05-25 21:44:36 +03:00
Aliaksandr Valialkin
9b64dfee4b
lib/encoding: add vm_zstd_block_{compress|decompress}_calls_total
for determining the number CompressZSTD / DecompressZSTD calls
2019-05-24 13:01:15 +03:00
Aliaksandr Valialkin
b23352dc9e
lib/encoding: add vm_zstd_block_{original|compressed}_bytes_total
metrics for rough estimation of block compression ratio
2019-05-24 12:34:51 +03:00
Aliaksandr Valialkin
f67f40d63a
lib/encoding: substitute CompressZSTD with CompressZSTDLevel
2019-05-24 12:32:49 +03:00
Aliaksandr Valialkin
a26e774eca
lib/httpserver: add -http.disableResponseCompression
flag, which may help saving CPU resources at the cost of higher network bandwidth usage
2019-05-24 12:20:36 +03:00
Aliaksandr Valialkin
820cdae88d
lib/decimal: add a comment explaining weird code in maxUpExponent. Fixes #29
2019-05-23 17:18:50 +03:00
Aliaksandr Valialkin
24578b4bb1
all: open-sourcing cluster version
2019-05-23 00:25:38 +03:00
Aliaksandr Valialkin
1836c415e6
all: open-sourcing single-node version
2019-05-23 00:18:06 +03:00