Aliaksandr Valialkin
fe196e0b7a
lib/protoparser/influx: extract stream parsing code into a separate stream package
...
This is a follow-up for 057698f7fb
2023-02-13 09:58:52 -08:00
Aliaksandr Valialkin
c63755c316
lib/writeconcurrencylimiter: improve the logic behind -maxConcurrentInserts limit
...
Previously the -maxConcurrentInserts was limiting the number of established client connections,
which write data to VictoriaMetrics. Some of these connections could be idle.
Such connections do not consume big amounts of CPU and RAM, so there is a little sense in limiting
the number of such connections. So now the -maxConcurrentInserts command-line option
limits the number of concurrently executed insert requests, not including idle connections.
It is recommended removing -maxConcurrentInserts command-line option, since the default value
for this option should work good for most cases.
2023-01-06 22:20:19 -08:00
Aliaksandr Valialkin
4de9d35458
lib/flagutil/bytes.go: properly handle values bigger than 2GiB on 32-bit architectures
...
This fixes handling of values bigger than 2GiB for the following command-line flags:
- -storage.minFreeDiskSpaceBytes
- -remoteWrite.maxDiskUsagePerURL
2022-12-14 19:26:31 -08:00
Aliaksandr Valialkin
5acd70109b
lib/protoparser: remove superflowous memory allocations during protocol parsing
2022-04-06 14:00:08 +03:00
Aliaksandr Valialkin
bb87949d5c
lib/protoparser/influx: automatically detect timestamp precision depending on the number of decimal digits in the timestamp
2021-10-28 12:47:22 +03:00
Aliaksandr Valialkin
7f0a8d4bdb
docs: consistency renaming: Influx -> InfluxDB
2021-09-13 17:05:16 +03:00
Aliaksandr Valialkin
e028ad241a
lib/protoparser: stop reading the input stream as soon as the callback provided by the caller returns error
...
This is a follow-up for af90c3c43b
2021-06-14 15:18:49 +03:00
Aliaksandr Valialkin
b861a64510
lib/protoparser/influx: make sure that escaped whitespace can be put in measurement, tag names and field names
2021-02-16 13:59:18 +02:00
Aliaksandr Valialkin
4146fc4668
all: properly handle CPU limits set on the host system/container
...
This can reduce memory usage on systems with enabled CPU limits.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/946
2020-12-08 21:07:29 +02:00
Aliaksandr Valialkin
b0e4b234cb
lib/protoparser/influx: allow multiple whitespace chars between measurement, fields and timestamp in Influx line protocol
2020-12-06 12:01:27 +02:00
Aliaksandr Valialkin
149c0c4a6d
lib/protoparser: propagate callback error to the caller of ParseStream for every supported data ingestion protocols
...
The caller of ParseStream then can generate HTTP 503 responses for non-nil errors occured in callbacks when processing incoming requests.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/896
2020-11-13 13:05:24 +02:00
Aliaksandr Valialkin
e277c3d07b
lib/promscrape: add stream parse mode for efficient scraping of targets that expose millions of metrics
2020-11-01 23:35:06 +02:00
Aliaksandr Valialkin
bbeac0ba46
lib/protoparser/influx: add -influx.maxLineSize command-line flag for configuring the maximum size for a single Influx line during parsing
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/807
2020-10-05 15:19:05 +03:00
Aliaksandr Valialkin
7500146321
lib/protoparser: avoid copying of buffer read from the network to unmarshal buffer
2020-09-28 17:19:16 +03:00
Aliaksandr Valialkin
124f78857b
app/{vminsert,vmagent}: improve data ingestion speed over a single connection
...
Process data obtianed from a single connection on all the available CPU cores.
2020-09-28 04:13:08 +03:00
Aliaksandr Valialkin
5cdad60a6f
lib/protoparser: use 64KB read buffer instead of default 4KB buffer provided by net/http.Server
...
This should reduce syscall overhead when reading big amounts of data
2020-09-28 02:07:10 +03:00
Aliaksandr Valialkin
d8183c3124
lib/protoparser: report more errors for incorrect timestamps and/or values
...
Previously certain errors in timestamps and/or values could be silently skipped,
which could lead to samples with zero values stored in the database.
Updates https://github.com/VictoriaMetrics/vmctl/issues/25
2020-09-16 02:14:18 +03:00
Aliaksandr Valialkin
576da0fe46
app/{vminsert,vmagent}: improve documentation for -influxListenAddr command-line flag
2020-08-14 18:04:44 +03:00
Aliaksandr Valialkin
15aa6142ef
lib/protoparser: clarify that the string passed to Unmarshal() function must remain available when the parsed rows are in use
2020-08-11 17:04:39 +03:00
Aliaksandr Valialkin
c098988a18
lib/protoparser/influx: accept precision=us and precision=µ according to https://docs.influxdata.com/influxdb/v1.8/tools/api/#write-http-endpoint
2020-08-10 20:23:26 +03:00
Aliaksandr Valialkin
6fe3c48a6e
properly calculate readCalls
2020-07-10 12:00:58 +03:00
Aliaksandr Valialkin
d5dddb0953
all: use %w instead of %s for wrapping errors in fmt.Errorf
...
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
2020-06-30 23:05:11 +03:00
Aliaksandr Valialkin
e62afc7366
lib/protoparser: add -*TrimTimstamp command-line flags for Influx, Graphite, OpenTSDB and CSV data
...
These flags can be used for reducing disk space usage for timestamps data ingested over the given protocols
2020-04-10 12:44:39 +03:00
Aliaksandr Valialkin
b785429ddb
lib/protoparser: metrics renaming: vm_protoparser_<type>_* -> vm_protoparser_*{type="<type>"}
...
This should improve composability of these metrics in PromQL queries
2020-02-28 20:20:10 +02:00
Aliaksandr Valialkin
6386aeb1e0
app/vmagent: add ability to accept Influx line protocol data via TCP and UDP
...
Just set `-influxListenAddr` command-line flag
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/333
2020-02-25 19:12:49 +02:00
Aliaksandr Valialkin
04762344c6
app/vmagent: initial implementation for vmagent
2020-02-23 13:36:03 +02:00
Aliaksandr Valialkin
6456c93dbb
app/vminsert: move ingestion protocol parsers to lib/protoparser, so they could be re-used in the upcoming vmagent
2020-01-24 16:53:00 +02:00
