github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
647 commits 150 branches 748 tags 347 MiB
Commit graph

239 commits

Author SHA1 Message Date
Aliaksandr Valialkin
453d71d082 Rename lib/promql to lib/metricsql and apply small fixes 2019-12-25 22:09:09 +02:00
Mike Poindexter
009d1559db Split Extended PromQL parsing to a separate library 2019-12-25 22:09:07 +02:00
Aliaksandr Valialkin
f22c9dbb0f lib/fs: typo fix in fadvise_unix.go 2019-12-24 21:00:04 +02:00
Aliaksandr Valialkin
d3c185f0ca lib/encoding: log the compressed block contents if it cannot be decompressed or unmarshaled 
This should help detecting the root cause of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281
 2019-12-24 20:48:25 +02:00
Aliaksandr Valialkin
091e35cf0c lib/encoding: mention src contents in error message returned from unmarshalInt64NearestDelta* 
This should simplify detecting the root cause of the issue at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281
 2019-12-24 20:41:38 +02:00
Aliaksandr Valialkin
0e51058a0d lib/encoding: mention unpacked block size in the error message if unparsed tail left 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/281
 2019-12-24 20:35:20 +02:00
Aliaksandr Valialkin
afa8b34d27 Revert "lib/logger: prevent from blocking when log output isn't consumed in timely manner" 
This reverts commit 9f50232e70.

Reason to revert: this leaves incomplete logs on app shutdown.
 2019-12-24 12:20:45 +02:00
Aliaksandr Valialkin
6358cf3d47 app/vmselect/netstorage: move MustAdviseSequentialRead to lib/fs 2019-12-23 23:16:26 +02:00
Aliaksandr Valialkin
44f886cc9c lib/encoding/zstd: typo fix 2019-12-23 18:37:20 +02:00
Aliaksandr Valialkin
108a60d69e lib/encoding/zstd: call zstd.Decoder.Close instead of zstd.Decoder.Reset in order to free up occupied goroutines 
This should fix goroutine leak for https://github.com/klauspost/compress/issues/195
 2019-12-23 18:32:28 +02:00
Aliaksandr Valialkin
335bd0ac0a lib/encoding/zstd: prevent from possible encoder leak when concurrent goroutines create encoders for the same compressionLevel 
Thanks to @klauspost for the pointer to this issue. See https://github.com/klauspost/compress/issues/195 for details.
 2019-12-23 18:06:02 +02:00
Aliaksandr Valialkin
9f50232e70 lib/logger: prevent from blocking when log output isn't consumed in timely manner 
Drop log messages instead of blocking and increment `vm_log_messages_dropped_total` metric.
 2019-12-20 11:49:42 +02:00
Aliaksandr Valialkin
a37a006f11 lib/storage: scale ingestion performance by sharding rawRows on systems with more than 8 CPU cores 2019-12-19 18:17:05 +02:00
Aliaksandr Valialkin
8d79412b26 lib/storage: optimize bulk import performance when multiple data points are inserted for the same time series 
This should speed up `/api/v1/import` and make it more scalable on multi-core systems.
 2019-12-19 15:13:36 +02:00
Aliaksandr Valialkin
05ec8afb3a lib/httpserver: sync the code with master branch 2019-12-18 23:08:32 +02:00
Aliaksandr Valialkin
a7bf8e77af app/vminsert: simultaneously accept telnet put and HTTP /api/put OpenTSDB metrics at -opentsdbListenAddr 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/266
 2019-12-14 00:42:18 +02:00
Aliaksandr Valialkin
bc3984a5b3 lib/logger: add -loggerFormat for choosing log message formats 
Supported formats: default, json

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/265
 2019-12-13 15:09:18 +02:00
Aliaksandr Valialkin
5d2ff573aa app/vmselect/promql: allow negative offsets 
Updates https://github.com/prometheus/prometheus/issues/6282
 2019-12-11 00:57:51 +02:00
Aliaksandr Valialkin
3694efd005 lib/{mergeset,storage}: log info message when both source and destination part paths from txn are missing during startup 
This is expected condition after unclean shutdown (OOM, hard reset, `kill -9`) on NFS disk.
 2019-12-09 15:45:23 +02:00
Aliaksandr Valialkin
639967db59 lib/{mergeset,storage}: make sure pending transaction deletions are finished before and after runTransactions call. 
`runTransactions` call issues async deletions for transaction files. The previously issued transaction deletions
can race with the next call to `runTransactions`. Prevent this by waiting until all the pending transaction
deletions are funished in the beginning of `runTransactions`. Also make sure that all the pending transaction
deletions are finished before returning from `runTransactions`.
 2019-12-04 21:40:52 +02:00
Aliaksandr Valialkin
7c0dd85a7c lib/httpserver: add /ping handler for compatibility with Influx agents 
Certain Influx agents check for `/ping` endpoint before starting
to send Influx line protocol data. See https://docs.influxdata.com/influxdb/v1.7/tools/api/#ping-http-endpoint
 2019-12-04 19:18:18 +02:00
Aliaksandr Valialkin
534da0a8c3 lib/storage: fall back to global inverted index if a filter match too many time series in per-day index 
Previously this resulted to error message. The query may succeed via search in global index.
 2019-12-03 14:48:08 +02:00
Aliaksandr Valialkin
6eb698d1cc lib/storage: fix printing tag filters in TagFilters.String 2019-12-03 14:25:20 +02:00
Aliaksandr Valialkin
c04f60db35 lib/storage: print __name__ instead of empty string in user-visible tag filters 2019-12-03 14:18:18 +02:00
Aliaksandr Valialkin
625f6ca761 lib/storage: optimize regexp filter search 2019-12-03 00:33:53 +02:00
Aliaksandr Valialkin
b9616c017f lib/{mergeset,storage}: remove transaction files only after the mentioned dirs are really removed 
This should fix the issue on NFS when incompletely removed dirs may be left
after unclean shutdown (OOM, kill -9, hard reset, etc.), while the corresponding transaction
files are already removed.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/162
 2019-12-02 21:34:37 +02:00
Aliaksandr Valialkin
4e22b521c2 lib/storage: remove metricID with missing metricID->metricName entry 
The metricID->metricName entry can be missing in the indexdb after unclean shutdown
when only a part of entries for new time series is written into indexdb.

Recover from such a situation by removing the broken metricID. New metricID
will be automatically created for time series with the given metricName
when new data point will arive to it.
 2019-12-02 20:52:13 +02:00
Aliaksandr Valialkin
5a62415bec lib/storage: protect from time drift during indexdb rotation 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/248
 2019-12-02 14:43:11 +02:00
Aliaksandr Valialkin
cf85c567d1 lib/logger: merge file and line labels into location="file:line" 
This should improve the usability for `vm_log_messages_total` metric during practical queries
 2019-12-02 14:43:09 +02:00
Aliaksandr Valialkin
f055dbefda lib/storage: generate more human-friendly result in TagFilters.String 2019-12-02 13:56:40 +02:00
Aliaksandr Valialkin
29f39f866e lib/logger: consistency renaming from vm_log_messages_count to vm_log_messages_total, since this is a counter 2019-12-02 00:47:12 +02:00
Aliaksandr Valialkin
15eaff1745 lib/logger: track the number of log messages by (level, file, line) in the vm_log_messages_count metric 2019-12-01 18:38:30 +02:00
Aliaksandr Valialkin
d456ec7589 lib/netutil: use IPv6 for both listening and dialing if -enabledTCP6 is set 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/244
 2019-12-01 02:52:53 +02:00
Aliaksandr Valialkin
7e734433a3 lib/backup: cosmetic fixes after #243 2019-11-29 18:07:41 +02:00
glebsam
4a192cb832 Add option to provide custom endpoint for S3, add option to specify S3 config profile (#243) 
* Add option to provide custom endpoint for S3 for use with s3-compatible storages, add option to specify S3 config profile

* make fmt
 2019-11-29 18:07:39 +02:00
Aliaksandr Valialkin
4810f1dde6 lib/netutil: add -enableTCP6 command-line flag for enabling listening for IPv6 additionally to IPv4 TCP ports 2019-11-29 17:33:07 +02:00
Aliaksandr Valialkin
409c939621 lib/backup: remove flock.lock file in empty dirs 
This fixes an issue when VictoriaMetrics doesn't see the restored data after the following operations:

1. Stop VictoriaMetrics.
2. Delete `<-storageDataPath>` dir.
3. Start VictoriaMetrics, then stop it.
4. Restore data from backup with `vmrestore`.
5. Start VictoriaMetrics.

`vmrestore` didn't delete properly empty dirs in `<-storageDataPath>/indexdb` because of the remaining `flock.lock` files in these dirs.
 2019-11-28 13:39:28 +02:00
Aliaksandr Valialkin
0f184affa7 app/vmselect/promql: optimize binary search over big number of samples during rollup calculations 2019-11-25 14:01:54 +02:00
Aliaksandr Valialkin
d24fc87a6f lib/decimal: calculate ln2/ln10 constant during compile time 2019-11-23 15:52:39 +02:00
Aliaksandr Valialkin
2af7ca1122 vendor: update github.com/VictoriaMetrics/metrics from v1.7.2 to v1.8.0. This version supports histograms 2019-11-23 00:21:57 +02:00
Aliaksandr Valialkin
b9e53490b9 lib/storage: move non-matching tag filters to the top at matchTagFilters 
This should reduce the amount of useless work needed for matching the next metricNames.
 2019-11-21 21:40:36 +02:00
Aliaksandr Valialkin
33d9d63393 lib/storage: speed up time series search for queries with multiple filters 
Use optimized specialized binary search for uint64 metricIDs instead of generic sort.Search.
 2019-11-21 18:43:40 +02:00
Aliaksandr Valialkin
a02a57fbe9 lib/storage: verify the number of returned metricIDs in BenchmarkHeadPostingForMatchers 2019-11-20 15:40:03 +02:00
Aliaksandr Valialkin
3d1f4408cf lib/decimal: increase decimal->float speed conversion for integer numbers 2019-11-20 14:09:10 +02:00
Aliaksandr Valialkin
f1f2eff08f lib/decimal: reduce rounding error when converting from decimal to float with negative exponent 
While at it, slightly increase the conversion performance by moving fast path to the top of the loop.
 2019-11-19 23:34:41 +02:00
Aliaksandr Valialkin
17eca31989 lib/backup: retrieve only the required metadata when reading GCS objects 2019-11-19 21:30:51 +02:00
Aliaksandr Valialkin
216a260ced app/{vmbackup,vmrestore}: add -maxBytesPerSecond command-line flag for limiting the used network bandwidth during backup / restore 2019-11-19 20:32:43 +02:00
Aliaksandr Valialkin
9d1ee1e2ae lib/backup: prevent from restoring to directory which is in use by VictoriaMetrics during the restore 2019-11-19 18:35:59 +02:00
Aliaksandr Valialkin
6ca4b94511 lib/storage: increase the number of created time series in BenchmarkHeadPostingForMatchers in order to be on par with Promethues 
The previous commit was accidentally creating 10x smaller number of time series than Prometheus
and this led to invalid benchmark results.

The updated benchmark results:

benchmark                                                          old ns/op      new ns/op     delta
BenchmarkHeadPostingForMatchers/n="1"                              272756688      6194893       -97.73%
BenchmarkHeadPostingForMatchers/n="1",j="foo"                      138132923      10781372      -92.19%
BenchmarkHeadPostingForMatchers/j="foo",n="1"                      134723762      10632834      -92.11%
BenchmarkHeadPostingForMatchers/n="1",j!="foo"                     195823953      10679975      -94.55%
BenchmarkHeadPostingForMatchers/i=~".*"                            7962582919     100118510     -98.74%
BenchmarkHeadPostingForMatchers/i=~".+"                            7589543864     154955671     -97.96%
BenchmarkHeadPostingForMatchers/i=~""                              1142371741     258003769     -77.42%
BenchmarkHeadPostingForMatchers/i!=""                              9964150263     159783895     -98.40%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",j="foo"              216995884      10937895      -94.96%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",i!="2",j="foo"       202541348      10990027      -94.57%
BenchmarkHeadPostingForMatchers/n="1",i!=""                        486285711      87004349      -82.11%
BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo"                350776931      53342793      -84.79%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo"              380888565      54256156      -85.76%
BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo"             89500296       21823279      -75.62%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo"       379529654      46671359      -87.70%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.*",j="foo"     424563825      53915842      -87.30%

VictoriaMetrics uses 1GB of RAM during the benchmark (vs 3.5GB of RAM for Prometheus)
 2019-11-18 19:48:27 +02:00
Aliaksandr Valialkin
6f61fd367a lib/storage: add BenchmarkHeadPostingForMatchers similar to the benchmark from Prometheus 
See the corresponding benchmark in Prometheus - 23c0299d85/tsdb/head_bench_test.go (L52)

The benchmark allows performing apples-to-apples comparison of time series search
in Prometheus and VictoriaMetrics. The following article - https://www.robustperception.io/evaluating-performance-and-correctness -
contains incorrect numbers for VictoriaMetrics, since there wasn't this benchmark yet. Fix it.

Benchmarks can be repeated with the following commands from Prometheus and VictoriaMetrics source code roots:

- Prometheus: GOMAXPROCS=1 go test ./tsdb/ -run=111 -bench=BenchmarkHeadPostingForMatchers
- VictoriaMetrics: GOMAXPROCS=1 go test ./lib/storage/ -run=111 -bench=BenchmarkHeadPostingForMatchers

Benchmark results:
benchmark                                                          old ns/op      new ns/op     delta
BenchmarkHeadPostingForMatchers/n="1"                              272756688      364977        -99.87%
BenchmarkHeadPostingForMatchers/n="1",j="foo"                      138132923      1181636       -99.14%
BenchmarkHeadPostingForMatchers/j="foo",n="1"                      134723762      1141578       -99.15%
BenchmarkHeadPostingForMatchers/n="1",j!="foo"                     195823953      1148056       -99.41%
BenchmarkHeadPostingForMatchers/i=~".*"                            7962582919     8716755       -99.89%
BenchmarkHeadPostingForMatchers/i=~".+"                            7589543864     12096587      -99.84%
BenchmarkHeadPostingForMatchers/i=~""                              1142371741     16164560      -98.59%
BenchmarkHeadPostingForMatchers/i!=""                              9964150263     12230021      -99.88%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",j="foo"              216995884      1173476       -99.46%
BenchmarkHeadPostingForMatchers/n="1",i=~".*",i!="2",j="foo"       202541348      1299743       -99.36%
BenchmarkHeadPostingForMatchers/n="1",i!=""                        486285711      11555193      -97.62%
BenchmarkHeadPostingForMatchers/n="1",i!="",j="foo"                350776931      5607506       -98.40%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",j="foo"              380888565      6380335       -98.32%
BenchmarkHeadPostingForMatchers/n="1",i=~"1.+",j="foo"             89500296       2078970       -97.68%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!="2",j="foo"       379529654      6561368       -98.27%
BenchmarkHeadPostingForMatchers/n="1",i=~".+",i!~"2.*",j="foo"     424563825      6757132       -98.41%

The first column (old) is for Prometheus, the second column (new) is for VictoriaMetrics.

Prometheus was using 3.5GB of RAM during the benchmark, while VictoriaMetrics was using 400MB of RAM.
 2019-11-18 18:47:02 +02:00