github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
2873 commits 150 branches 748 tags 347 MiB
Commit graph

80 commits

Author SHA1 Message Date
Aliaksandr Valialkin
b84aea1e6e lib/protoparser/clusternative: do not pool unmarshalWork structs, since they can occupy big amounts of memory (more than 100MB per each struct) 
This should reduce memory usage for vmstorage under high ingestion rate when the vmstorage runs on a system with big number of CPU cores
 2021-06-23 15:45:08 +03:00
Aliaksandr Valialkin
ebaf68bcb0 lib/protoparser: stop reading the input stream as soon as the callback provided by the caller returns error 
This is a follow-up for af90c3c43b
 2021-06-14 15:20:38 +03:00
faceair
2ea187e801 lib/protoparser: stop read when callback error (#1380) 2021-06-14 15:20:37 +03:00
Aliaksandr Valialkin
074b11fa69 lib/protoparser: measure the duration for reading the whole block of data instead of a single read operation 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1365
 2021-06-14 12:29:45 +03:00
Aliaksandr Valialkin
87d221f78a lib/protoparser/common: log the duration for reading a block of data in ReadLinesBlockExt on error 
This may help debugging issues like https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1365
 2021-06-14 12:21:21 +03:00
Aliaksandr Valialkin
49490ae5a7 lib/protoparser/clusternative: remove duplicate cannot read packet size phrase from the log message 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1336
 2021-05-27 12:09:17 +03:00
Aliaksandr Valialkin
9c505d27dd lib/ingestserver: properly close incoming connections during graceful shutdown 2021-05-08 19:53:45 +03:00
Aliaksandr Valialkin
4a5f45c77e app/vminsert: add support for data ingestion via other vminsert nodes 2021-05-08 19:53:45 +03:00
Aliaksandr Valialkin
e2717d84c0 all: various fixes in command-line flag descriptions 2021-03-15 22:03:49 +02:00
Aliaksandr Valialkin
f686174329 lib/promscrape/discovery/ec2: follow-up after f6114345de 2021-03-02 13:47:35 +02:00
Aliaksandr Valialkin
e45c399467 lib/protoparser/prometheus: properly unescape label values in Prometheus exposition format 
Unescape only `\n`, `\"` and `\\` sequences as Prometheus does. Other escape sequences shouldn't be unescaped.
 2021-03-02 13:22:10 +02:00
Aliaksandr Valialkin
f4969a624d lib/protoparser/graphite: fix parsing of a Graphite line with empty tags such as foo; 1 2 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1100
 2021-03-01 17:17:01 +02:00
Aliaksandr Valialkin
7869d38043 lib/protoparser/influx: make sure that escaped whitespace can be put in measurement, tag names and field names 2021-02-16 13:59:11 +02:00
Nikolay
9f0a4fd00e Fixes error handling for promscrape.streamParse (#1009) 
properly return error if client cannot read data,
properly suppress scraper errors
 2021-01-12 13:35:09 +02:00
Aliaksandr Valialkin
4fd2973e7c lib/protoparser/prometheus: follow-up commit after 7d38627b9f6f212ae602aea6a72f469fe3c70ba2 
Document the bugfix in docs/CHANGELOG.md and add a test for the bugfix.
 2020-12-16 23:42:17 +02:00
BigFish
60dd48c9eb lib/protoparser/prometheus/parser.go (#970) 
fix parse timestamp error if there are some whitespaces after timestamp
 2020-12-16 23:42:16 +02:00
Aliaksandr Valialkin
1a237c6903 all: properly handle CPU limits set on the host system/container 
This can reduce memory usage on systems with enabled CPU limits.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/946
 2020-12-08 21:07:03 +02:00
Aliaksandr Valialkin
3d00613076 lib/protoparser/influx: allow multiple whitespace chars between measurement, fields and timestamp in Influx line protocol 2020-12-06 12:00:28 +02:00
Aliaksandr Valialkin
2ed721e457 lib/protoparser/prometheus: properly parse OpenMetrics timestamps 
OpenMetrics timestamps are floating-point numbers, that represent Unix timestamp in seconds.
This differs from Prometheus exposition format, where timestamps are integer numbers representing Unix timestamp in milliseconds.
 2020-11-27 14:54:36 +02:00
Aliaksandr Valialkin
26e699c440 lib/protoparser/prometheus: properly parse "infinity" values in OpenMetrics format 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/924
 2020-11-24 19:02:50 +02:00
Aliaksandr Valialkin
7f3e884a31 all: spelling fix: superflouos->superfluous. This is a follow-up for 0acdab3ab9 2020-11-24 12:42:04 +02:00
Aliaksandr Valialkin
dad8b76a0e lib/protoparser/prometheus: properly parse metrics with exemplars 
Examplars have been introduced in OpenMetrics - see https://github.com/OpenObservability/OpenMetrics/blob/master/OpenMetrics.md#exemplars-1
Previously VictoriaMetrics couldn't parse the following metric

    foo{bar="baz"} 123 # exemplar here

This commit fixes this. Note that VictoriaMetrics ignores the exemplar as for now.
 2020-11-24 12:30:34 +02:00
Aliaksandr Valialkin
4be5b5733a app/vminsert: add /tags/tagSeries and /tags/tagMultiSeries handlers from Graphite Tags API 
See https://graphite.readthedocs.io/en/stable/tags.html#adding-series-to-the-tagdb
 2020-11-16 02:40:04 +02:00
Aliaksandr Valialkin
f80d6473e1 lib/protoparser/promremotewrite: log the time spent on unsuccessful data read from the network 
This should help with debugging `connection timed out` errors.
 2020-11-13 17:49:21 +02:00
Aliaksandr Valialkin
d21b1606a1 lib/protoparser/opentsdbhttp: increment errors counter on unmarshal errors 
This is a follow-up for 149c0c4a6d
 2020-11-13 13:23:11 +02:00
Aliaksandr Valialkin
22c1e29284 lib/protoparser: propagate callback error to the caller of ParseStream for every supported data ingestion protocols 
The caller of ParseStream then can generate HTTP 503 responses for non-nil errors occured in callbacks when processing incoming requests.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/896
 2020-11-13 13:05:34 +02:00
Aliaksandr Valialkin
9dfe00c962 lib/protoparser/promremotewrite: synchronously process Prometheus remote_write requests 
There is no reason in processing these requests asynchronously in the face of https://github.com/VictoriaMetrics/VictoriaMetrics/issues/896
Synchronous processing code is easier to read and understand than the previous async code
 2020-11-13 12:17:32 +02:00
Aliaksandr Valialkin
739b88c1e4 lib/protoparser/promremotewrite: forward errors, which can occur during data ingestion, to the caller of ParseStream, so it could properly return HTTP 503 status code on non-nil error 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/896
 2020-11-13 11:00:41 +02:00
Aliaksandr Valialkin
ed724d25ba lib/promscrape: add stream parse mode for efficient scraping of targets that expose millions of metrics 2020-11-01 23:12:26 +02:00
Aliaksandr Valialkin
54ff78c6c9 lib/protoparser/graphite: support parsing floating-point timestamp like Graphite does 
Such timestamps are rounded to seconds like Carbon does.
See b0ba62a62d/lib/carbon/protocols.py (L197)
 2020-10-06 11:38:35 +03:00
Aliaksandr Valialkin
9a7c863bd8 lib/protoparser/influx: add -influx.maxLineSize command-line flag for configuring the maximum size for a single Influx line during parsing 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/807
 2020-10-05 15:19:11 +03:00
Aliaksandr Valialkin
0b0259c42c lib/protoparser/prometheus: sort rows before comparing them in TestParseStream, since the order for callback calls is non-deterministic 2020-09-29 12:29:50 +03:00
Aliaksandr Valialkin
8f25206d8c lib/protoparser/prometheus: fix TestParseStream after 124f78857b 2020-09-29 12:12:33 +03:00
Aliaksandr Valialkin
1481d6d8ff lib/protoparser: avoid copying of buffer read from the network to unmarshal buffer 2020-09-28 17:19:04 +03:00
Aliaksandr Valialkin
8df33bd5c1 app/{vminsert,vmagent}: improve data ingestion speed over a single connection 
Process data obtianed from a single connection on all the available CPU cores.
 2020-09-28 04:14:51 +03:00
Aliaksandr Valialkin
7072db75cb lib/protoparser: use 64KB read buffer instead of default 4KB buffer provided by net/http.Server 
This should reduce syscall overhead when reading big amounts of data
 2020-09-28 02:07:19 +03:00
Aliaksandr Valialkin
6d8c23fdbd app/{vminsert,vmselect}: skip accountID and projectID when marshaling/unmarshaling MetricName in /api/v1/export/native and /api/v1/import/native 
This is needed in order to be able to migrate native data from/to single-node VictoriaMetrics
 2020-09-28 00:58:58 +03:00
Aliaksandr Valialkin
aadbd014ff all: add native format for data export and import 
The data can be exported via [/api/v1/export/native](https://victoriametrics.github.io/#how-to-export-data-in-native-format) handler
and imported via [/api/v1/import/native](https://victoriametrics.github.io/#how-to-import-data-in-native-format) handler.
 2020-09-27 17:36:38 +03:00
Aliaksandr Valialkin
00ec2b7189 lib/protoparser: use all the available CPU cores for processing ingested data from a single /api/v1/import stream 
Previously a single data ingestion stream to /api/v1/import could load only a single CPU core.
 2020-09-26 04:22:06 +03:00
Aliaksandr Valialkin
6ce52e3702 lib/protoparser/vmimport: add more testcases for invalid timestamps and values 
Updates https://github.com/VictoriaMetrics/vmctl/issues/25
 2020-09-16 02:21:53 +03:00
Aliaksandr Valialkin
cd87ca303f lib/protoparser: report more errors for incorrect timestamps and/or values 
Previously certain errors in timestamps and/or values could be silently skipped,
which could lead to samples with zero values stored in the database.

Updates https://github.com/VictoriaMetrics/vmctl/issues/25
 2020-09-16 02:16:15 +03:00
Aliaksandr Valialkin
5c4e111b43 lib/protoparser/graphite: return error when value or timestamp cannot be properly parsed 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/99
 2020-09-16 02:16:13 +03:00
Aliaksandr Valialkin
5cf5a0e8c4 lib/protoparser/common: do not read request body when parsing timestamp query arg 
This was preventing from reading data via /api/v1/prometheus/import .

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/750
 2020-09-11 14:45:21 +03:00
Aliaksandr Valialkin
8bd5aa3516 lib/protoparser: accept timestamp in milliseconds instead of seconds at /api/v1/import/prometheus 
This improves consistency with timestamps in Prometheus text exposition format
 2020-09-11 14:05:24 +03:00
Aliaksandr Valialkin
58d3b82ae5 app/{vminsert,vmagent}: allow passing timestamp via timestamp query arg when ingesting data to /api/v1/import/prometheus 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/750
 2020-09-11 13:28:31 +03:00
Aliaksandr Valialkin
4fa97430d7 app/{vminsert,vmagent}: allow adding extra labels when importing data via Prometheus, CSV and JSON line formats 
Extra labels may be added to the imported data by passing `extra_label=name=value` query args.
Multiple query args may be passed in order to add multiple extra labels.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/719
 2020-09-02 19:47:02 +03:00
Aliaksandr Valialkin
6aab2f4989 all: allow using KB, MB, GB, KiB, MiB and GiB suffixes in command-line flag values related to byte sizes or byte rates 2020-08-16 17:08:28 +03:00
Aliaksandr Valialkin
d9f7ea1c6e lib/protoparser: removed unnecessary call to SetReadDeadline when reading a stream of data 
The OS should return any buffered data in the stream without the need to set the read timeout.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/696
 2020-08-15 15:38:35 +03:00
Aliaksandr Valialkin
3efa4e4e1c lib/protoparser: move common code for detecting timeouts to ReadLinesBlockExt 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/696
 2020-08-14 20:39:51 +03:00
Aliaksandr Valialkin
c6b0547847 lib/protoparser: prevent from busy loop on repeated timeout errors when reading streams of ingested data 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/696
 2020-08-14 20:13:37 +03:00