Aliaksandr Valialkin
e028ad241a
lib/protoparser: stop reading the input stream as soon as the callback provided by the caller returns error
...
This is a follow-up for af90c3c43b
2021-06-14 15:18:49 +03:00
Aliaksandr Valialkin
4146fc4668
all: properly handle CPU limits set on the host system/container
...
This can reduce memory usage on systems with enabled CPU limits.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/946
2020-12-08 21:07:29 +02:00
Aliaksandr Valialkin
149c0c4a6d
lib/protoparser: propagate callback error to the caller of ParseStream for every supported data ingestion protocols
...
The caller of ParseStream then can generate HTTP 503 responses for non-nil errors occured in callbacks when processing incoming requests.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/896
2020-11-13 13:05:24 +02:00
Aliaksandr Valialkin
e277c3d07b
lib/promscrape: add stream parse mode for efficient scraping of targets that expose millions of metrics
2020-11-01 23:35:06 +02:00
Aliaksandr Valialkin
7500146321
lib/protoparser: avoid copying of buffer read from the network to unmarshal buffer
2020-09-28 17:19:16 +03:00
Aliaksandr Valialkin
124f78857b
app/{vminsert,vmagent}: improve data ingestion speed over a single connection
...
Process data obtianed from a single connection on all the available CPU cores.
2020-09-28 04:13:08 +03:00
Aliaksandr Valialkin
5cdad60a6f
lib/protoparser: use 64KB read buffer instead of default 4KB buffer provided by net/http.Server
...
This should reduce syscall overhead when reading big amounts of data
2020-09-28 02:07:10 +03:00
Aliaksandr Valialkin
d8183c3124
lib/protoparser: report more errors for incorrect timestamps and/or values
...
Previously certain errors in timestamps and/or values could be silently skipped,
which could lead to samples with zero values stored in the database.
Updates https://github.com/VictoriaMetrics/vmctl/issues/25
2020-09-16 02:14:18 +03:00
Aliaksandr Valialkin
ed00eb3f33
lib/protoparser: removed unnecessary call to SetReadDeadline when reading a stream of data
...
The OS should return any buffered data in the stream without the need to set the read timeout.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/696
2020-08-15 15:38:08 +03:00
Aliaksandr Valialkin
7554be172d
lib/protoparser: move common code for detecting timeouts to ReadLinesBlockExt
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/696
2020-08-14 20:40:15 +03:00
Aliaksandr Valialkin
15aa6142ef
lib/protoparser: clarify that the string passed to Unmarshal() function must remain available when the parsed rows are in use
2020-08-11 17:04:39 +03:00
Aliaksandr Valialkin
56ccfa5218
all: use errors.As instead of type assertion for detecting net.Error
2020-07-01 00:15:34 +03:00
Aliaksandr Valialkin
d5dddb0953
all: use %w instead of %s for wrapping errors in fmt.Errorf
...
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
2020-06-30 23:05:11 +03:00
Aliaksandr Valialkin
0afd48d2ee
lib: extract common code for returning fast unix timestamp into lib/fasttime
2020-05-14 23:02:07 +03:00
Aliaksandr Valialkin
e62afc7366
lib/protoparser: add -*TrimTimstamp command-line flags for Influx, Graphite, OpenTSDB and CSV data
...
These flags can be used for reducing disk space usage for timestamps data ingested over the given protocols
2020-04-10 12:44:39 +03:00
Aliaksandr Valialkin
b785429ddb
lib/protoparser: metrics renaming: vm_protoparser_<type>_* -> vm_protoparser_*{type="<type>"}
...
This should improve composability of these metrics in PromQL queries
2020-02-28 20:20:10 +02:00
Aliaksandr Valialkin
04762344c6
app/vmagent: initial implementation for vmagent
2020-02-23 13:36:03 +02:00
Aliaksandr Valialkin
6456c93dbb
app/vminsert: move ingestion protocol parsers to lib/protoparser, so they could be re-used in the upcoming vmagent
2020-01-24 16:53:00 +02:00
