github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-11 14:53:49 +00:00
Code Issues Projects Releases Packages Wiki Activity
9548 commits 131 branches 758 tags 363 MiB
Commit graph

49 commits

Author SHA1 Message Date
Andrii Chubatiuk
9cfdbc582f
refactoring: changed prompb to prompbmarshal everythere where internal series transformations are happening (#7409) 
### Describe Your Changes

doing similar changes for both vmagent and vminsert (like one in
https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7399) ends up
with almost same implementations for each of packages instead of having
this shared code in one place. one of the reasons is the same Timeseries
and Labels structure from different prompb and prompbmarshal packages.
My proposal is to use structures from prompb package only to
marshal/unmarshal sent/received data, but for internal transformations
use only structures from prompbmarshal package

Another example, where it already can help to simplify code is streaming
aggregation pipeline for vmsingle (now it first marshals
prompb.Timeseries to storage.MetricRow and then if streaming aggregation
or deduplication is enabled it unmarshals all the series back but to
prompbmarshal.Timeseries)

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
 2024-11-26 12:45:17 +01:00
Andrii Chubatiuk
daa7183749
lib/protoparser/influx: enable batch processing by default (#7165) 
### Describe Your Changes

Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7090

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
 2024-10-15 11:48:40 +02:00
Aliaksandr Valialkin
8942f290eb
app/vminsert: replace hybrid sync.Pool+channel-based pool scheme for poolCtx with plain sync.Pool 
This simplifies the code, while doesn't increase memory usage under low and high data ingestion rate.

This is a follow-up for 1decbcf6eb
 2024-04-20 21:44:53 +02:00
Aliaksandr Valialkin
1decbcf6eb
app/vminsert/influx: replace hybrid channel-based pool+sync.Pool with plain sync.Pool for pushCtx 
The memory usage for plain sync.Pool doesn't increase comparing to the memory usage for the hybrid scheme,
so it is better to use plain sync.Pool in order to simplify the code and make it more readable and maintainable.

This is a follow-up for c22da2f917
 2024-04-20 21:41:06 +02:00
Aliaksandr Valialkin
f2229c2e42
lib/prompb: change type of Label.Name and Label.Value from []byte to string 
This makes it more consistent with lib/prompbmarshal.Label
 2024-01-14 22:33:21 +02:00
Alexander Marshalov
2e494e2375
fixed typos in documentation and commandline flags descriptions (#4275) 2023-05-10 09:50:41 +02:00
Aliaksandr Valialkin
fe196e0b7a
lib/protoparser/influx: extract stream parsing code into a separate stream package 
This is a follow-up for 057698f7fb
 2023-02-13 09:58:52 -08:00
Aliaksandr Valialkin
c63755c316
lib/writeconcurrencylimiter: improve the logic behind -maxConcurrentInserts limit 
Previously the -maxConcurrentInserts was limiting the number of established client connections,
which write data to VictoriaMetrics. Some of these connections could be idle.
Such connections do not consume big amounts of CPU and RAM, so there is a little sense in limiting
the number of such connections. So now the -maxConcurrentInserts command-line option
limits the number of concurrently executed insert requests, not including idle connections.

It is recommended removing -maxConcurrentInserts command-line option, since the default value
for this option should work good for most cases.
 2023-01-06 22:20:19 -08:00
Eugene
3d19fa6932
Add -influxDBLabel parameter (#2203) 2022-02-16 23:27:06 +02:00
Aliaksandr Valialkin
7f0a8d4bdb docs: consistency renaming: Influx -> InfluxDB 2021-09-13 17:05:16 +03:00
Aliaksandr Valialkin
dc9eafcd02 app/{vminsert,vmagent}: add -sortLabels command-line option for sorting time series labels before ingesting them in the storage 
This option can be useful when samples for the same time series are ingested with distinct order of labels.
For example, metric{k1="v1",k2="v2"} and metric{k2="v2",k1="v1"}.
 2021-03-31 23:27:58 +03:00
Aliaksandr Valialkin
44a6cc5eca app/{vminsert,vmagent}: use Influx field as metric name if measurement is empty and -influxSkipSingleField command-line is set 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1139
 2021-03-22 13:53:53 +02:00
Nikolay
dc9d7aedd5
adds extra_label to all import apis (#1007) 
* adds extra_label to all import apis,
changes priority for extra_label - now it has priority over original labels

* Update README.md

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>

* Update README.md

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>

* adds extra labels to vmagent  import api
changes order for adding labels, now its added after user values

* adds tests for extra_label

* import fix

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2021-01-13 00:52:50 +02:00
Aliaksandr Valialkin
4146fc4668 all: properly handle CPU limits set on the host system/container 
This can reduce memory usage on systems with enabled CPU limits.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/946
 2020-12-08 21:07:29 +02:00
Aliaksandr Valialkin
84227ea2fc app/{vminsert,vmagent}: take into account all the inserted rows before relabeling in vm_rows_inserted_total and vmagent_rows_inserted_total metrics 2020-10-09 13:29:51 +03:00
Aliaksandr Valialkin
0f63da3698 app/{vmagent,vminsert}: properly preserve db tag from query string passed to Influx line protocol query 
Previously `db` tag from the query string wasn't added to metrics after encountering `db` tag in the Influx line

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/653
 2020-07-28 21:25:19 +03:00
Aliaksandr Valialkin
a090627059 app/vminsert: limit memory usage when ingesting data in big packets 2020-07-24 23:32:40 +03:00
Aliaksandr Valialkin
2f612e0c67 app/vminsert: fix relabeling for metrics ingested via Influx line protocol 
Previously the enabled relabeling with `-relabelConfig` command-line flag could result in missing labels
if a single Influx line protocol message contains multiple field values.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/638
 2020-07-23 13:23:14 +03:00
Aliaksandr Valialkin
8c14ca93fa app/vminsert/influx: properly handle the case when certain labels with empty values are removed by ApplyRelabeling() call 
Previously this could lead to `out of range` panic
 2020-07-17 00:07:06 +03:00
Aliaksandr Valialkin
4081e2295e app/{vminsert,vmagent}: add -influxSkipMeasurement command-line flag for using field name as metric name 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/626
 2020-07-14 14:17:24 +03:00
Aliaksandr Valialkin
0b2086b7a5 app/vminsert: prevent from adding and/or selecting labels with empty values 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/600
 2020-07-02 23:14:11 +03:00
Aliaksandr Valialkin
91b3482894 app/vminsert: add ability to apply relabeling to all the incoming metrics if -relabelConfig command-line arg points to a file with a list of relabel_config entries 
See https://victoriametrics.github.io/#relabeling
 2020-07-02 20:39:28 +03:00
Aliaksandr Valialkin
6386aeb1e0 app/vmagent: add ability to accept Influx line protocol data via TCP and UDP 
Just set `-influxListenAddr` command-line flag

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/333
 2020-02-25 19:12:49 +02:00
Aliaksandr Valialkin
04762344c6 app/vmagent: initial implementation for vmagent 2020-02-23 13:36:03 +02:00
Aliaksandr Valialkin
42864bb52f all: do not clash flag description with back-quoted flag types 
See https://golang.org/pkg/flag/#PrintDefaults for more details.
 2020-02-04 15:46:52 +02:00
Aliaksandr Valialkin
6456c93dbb app/vminsert: move ingestion protocol parsers to lib/protoparser, so they could be re-used in the upcoming vmagent 2020-01-24 16:53:00 +02:00
Aliaksandr Valialkin
68e1cf8942 app/vminsert: add /api/v1/import handler 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6
 2019-12-09 20:59:04 +02:00
Aliaksandr Valialkin
8501b4a48d app/vminsert: consistency renaming for counters 2019-12-09 16:43:10 +02:00
Aliaksandr Valialkin
293b541784 make vendor-update 2019-12-07 23:10:16 +02:00
Aliaksandr Valialkin
84b57e8974 app/vminsert/influx: add a test case from https://community.librenms.org/t/integration-with-victoriametrics/9689 2019-12-07 23:00:40 +02:00
Aliaksandr Valialkin
811b7a8303 app/vminsert/influx: allow empty measurement in Influx line protocol 
In this case metric names are mapped directly from field names without any prefixes.
 2019-11-30 23:18:41 +02:00
Aliaksandr Valialkin
fb909cf710 app/vminsert/influx: set db label only if Influx line doesnt have db tag 2019-08-24 13:52:48 +03:00
Aliaksandr Valialkin
9c9221d1b2 app/vminsert: skip empty tags 2019-08-24 13:36:29 +03:00
Aliaksandr Valialkin
9d3f1fcbb9 app/vminsert/influx: skip invalid rows and continue parsing the remaining rows 
Invalid influx lines are logged and counted in `vm_rows_invalid_total{type="influx"}` metric.
 2019-08-24 13:36:29 +03:00
Aliaksandr Valialkin
ba7b3806be app/vminsert/influx: do not allow escaping newline char, since they dont occur in real life 
The prefious report with escaped newline chars in influx line protocol was false alarm.
 2019-08-23 18:42:05 +03:00
Aliaksandr Valialkin
4da34b11f8 app/vminsert/influx: handle \r\n aka crlf influx line endings from windows world 
Such lines exist in real life.
 2019-08-23 18:28:49 +03:00
Aliaksandr Valialkin
a18317adbc app/vminsert/influx: allow escaping newline char 
Though newline char isn't mentioned in escape rules at https://docs.influxdata.com/influxdb/v1.7/write_protocols/line_protocol_tutorial/ ,
there are reports that such chars occur in real life
 2019-08-23 15:14:46 +03:00
Aliaksandr Valialkin
44d7fc599d app/vminsert/influx: skip comments starting with # in influx line protocol 2019-08-23 14:43:09 +03:00
Aliaksandr Valialkin
5f33fc8e46 app/vminsert: add ability to ingest data via HTTP OpenTSDB /api/put requests 
This is manual merge of the https://github.com/VictoriaMetrics/VictoriaMetrics/pull/152
Thanks to nustinov@gmail.com for the initial pull request.
 2019-08-22 12:28:32 +03:00
Aliaksandr Valialkin
a9f39168d2 app/vminsert/influx: round automatically generated timestamp according to the given precision arg 2019-08-02 00:24:06 +03:00
Aliaksandr Valialkin
1eade9b358 app/vminsert: add vm_rows_per_insert summary metric 
This metric should help tuning batch sizes on clients writing data to VictoriaMetrics
 2019-07-27 13:21:46 +03:00
Aliaksandr Valialkin
bf6cbb762c app/vminsert: improve error messages for Influx, OpenTSDB and Graphite parsing 
Include in the error message the line which failed to parse.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/127
 2019-07-26 22:08:52 +03:00
Jiri Tyr
e0fccc6c60 Change the default influxMeasurementFieldSeparator 2019-06-26 13:22:03 +03:00
Aliaksandr Valialkin
1f7d9a213a app/vminsert: fix inifinite loop when reading two lines without newline in the end 
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/82
 2019-06-26 02:51:56 +03:00
Aliaksandr Valialkin
86d3d907a5 app/vminsert/influx: add -influxSkipSingleField flag for using {measurement} instead of {measurement}{separator}{field_name} for Influx lines with a single field 
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/66
 2019-06-17 19:05:57 +03:00
Aliaksandr Valialkin
269285848f app/vminsert/influx: add -influxMeasurementFieldSeparator flag for the ability to change separator for {measurement}{separator}{field_name} metric name 2019-06-14 10:00:12 +03:00
Aliaksandr Valialkin
968d094524 app/vminsert: reduce memory usage for Influx, Graphite and OpenTSDB protocols 
Do not buffer per-connection data and just store it as it arrives
 2019-05-28 18:47:23 +03:00
Aliaksandr Valialkin
feb6b203a4 app/vminsert/influx: try converting string values to numeric values, since Influx agents may send numeric values as strings 
Fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/34
 2019-05-26 22:11:19 +03:00
Aliaksandr Valialkin
1836c415e6 all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00