github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00
Code Issues Projects Releases Packages Wiki Activity
5512 commits 150 branches 748 tags 347 MiB
Commit graph

68 commits

Author SHA1 Message Date
Aliaksandr Valialkin
b275983403
lib/writeconcurrencylimiter: improve the logic behind -maxConcurrentInserts limit 
Previously the -maxConcurrentInserts was limiting the number of established client connections,
which write data to VictoriaMetrics. Some of these connections could be idle.
Such connections do not consume big amounts of CPU and RAM, so there is a little sense in limiting
the number of such connections. So now the -maxConcurrentInserts command-line option
limits the number of concurrently executed insert requests, not including idle connections.

It is recommended removing -maxConcurrentInserts command-line option, since the default value
for this option should work good for most cases.
 2023-01-06 22:07:16 -08:00
Aliaksandr Valialkin
c5eebaffd8
app/{vminsert,vmagent}: follow-up after 53a63c6c4c 
Extend /api/v1/import/prometheus with the support for Pushgateway way of specifying additional labels.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1415
 2022-11-25 16:42:38 -08:00
Zakhar Bessarab
fd36c9b3b1
docs/{vminsert,vmselect}: clarify flag description for clusternativeListenAddr (#3314) 
docs/{vminsert,vmselect}: clarify flag description for `clusternativeListenAddr`
 2022-11-04 09:49:53 +01:00
Zakhar Bessarab
7afa67dba0
{app/vmselect,app/vminsert}: ensure -storageNodes flag has no empty values (#3291) 2022-11-01 14:54:55 +02:00
Aliaksandr Valialkin
09cf136154
app/{vminsert,vmselect}: add support for automatic discovery and update of vmstorage nodes 
Thanks to @dmitryk-dk for the initial implemenation at https://github.com/VictoriaMetrics/VictoriaMetrics-enterprise/pull/446
 2022-10-28 13:13:03 +03:00
Aliaksandr Valialkin
4f53147ed4
app/{vminsert,vmselect}/netstorage: allow calling Init()+MustStop() in a loop 
Previously netstorage.MustStop() call didn't free up all the resources,
so the subsequent call to nestorage.Init() would panic.

This allows writing tests, which call nestorage.Init() + nestorage.MustStop() in a loop.
 2022-10-25 14:43:05 +03:00
Aliaksandr Valialkin
6f9ce3f6d6
lib/flagutil: rename Array to ArrayString 
This makes the ArrayString more consistent with other Array* types.

While at it, add ArrayBytes type, which will be used for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3071
 2022-10-01 18:28:19 +03:00
Nikolay
505d359b39
app/vminsert: allows parsing tenant id from labels (#3009) 
* app/vminsert: allows parsing tenant id from labels
it should help mitigate issues with vmagent's multiTenant mode, which works incorrectly at heavy load
and it cannot handle more then 100 different tenants.
This functional hidden with flag and do not change vminsert default behaviour
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2970

* Update docs/Cluster-VictoriaMetrics.md

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* wip

* app/vminsert/netstorage: clean remaining labels in order to free up GC

* docs/Cluster-VictoriaMetrics.md: typo fix

* wip

* wip

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-09-30 17:28:35 +03:00
Dmytro Kozlov
ec273eafef
app/{vmselect, vminsert}: fails with error when user defines equal addresses in the -storageNodes flag (#3082) 2022-09-08 21:17:58 +03:00
Aliaksandr Valialkin
c0c9f30870
lib/pushmetrics: properly handle errors when initializing pushmetrics 2022-07-22 13:38:25 +03:00
Aliaksandr Valialkin
f00a6bf837
all: add ability to push internal metrics to remote storage system specified via -pushmetrics.url 2022-07-21 20:15:29 +03:00
Dmytro Kozlov
f0cb3c4999
Added a stub for datadog endpoint (#2710) 
* Added a stub for datadog endpoint

* Update app/vmagent/main.go

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-06-13 09:56:26 +03:00
Aliaksandr Valialkin
d830f43608
app/vminsert/main.go: add missing strings import after 0c5e199469 2022-06-07 15:42:23 +03:00
Aliaksandr Valialkin
2b72d21840
docs/CHANGELOG.md: document e755d0ec3f 2022-06-07 15:21:10 +03:00
elProxy
0c5e199469
Support legacy datadog agent (#2670) 
dd-agent v5 can issue some requests with trailing slashes.
(e.g.
526559be73/ddagent.py (L303))
Trim trailing slashes for request on /datadog/ paths to accomodate for
that.

Co-authored-by: Pierre Rossi <pierre.rossi@schibsted.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
 2022-06-07 15:21:09 +03:00
Aliaksandr Valialkin
de143323d0
app/{vmagent,vminsert}: mention port 8089 instead of 8189 in the description for -influxListenAddr flag 
InfluxDB uses 8089 port for sending plain Influx line protocol data over TCP and UDP.
See https://docs.influxdata.com/influxdb/v1.8/administration/ports/

This is a follow-up for 20cef877a1
 2022-05-13 16:59:16 +03:00
Aliaksandr Valialkin
ce91456982
app/vminsert: add X-Influxdb-Version response header for InfluxDB API requests 
This is needed for some clients, which expect this header.
See https://github.com/ntop/ntopng/issues/5449#issuecomment-1005347597
 2022-02-17 12:49:24 +02:00
Aliaksandr Valialkin
021ee53ba8
app/vminsert: improve re-routing logic in order to spread rows more evenly among the available storage nodes 2022-02-06 20:20:02 +02:00
Aliaksandr Valialkin
9aa9b081a4
app/vminsert: add -maxLabelValueLen command-line flag 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1908
 2021-12-06 11:42:24 +02:00
Aliaksandr Valialkin
4fb19fe34b
all: consistently return application/json content-type without charset=utf-8 
The `application/json` content-type has utf-8 encoding by default.
See https://stackoverflow.com/questions/9254891/what-does-content-type-application-json-charset-utf-8-really-mean

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/897
 2021-11-09 18:07:22 +02:00
Aliaksandr Valialkin
64b6f3f1c8
app/vminsert: fix uneven distribution of time series among storage nodes 
Use distinct seed for distribution hash calculations on the second level of vminsert nodes.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1672
 2021-10-07 12:22:39 +03:00
Aliaksandr Valialkin
4e65bfcc00
app/{vminsert,vmagent}: add ability to ingest data via DataDog "submit metrics" API 
See https://docs.datadoghq.com/api/latest/metrics/#submit-metrics

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/206
 2021-09-29 00:12:26 +03:00
Aliaksandr Valialkin
7ad54041fe app/{vminsert,vmselect}: automatically add missing port in -storageNode lists passed to vminsert and vmselect 
This should simplify manual setup of the cluster according to https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#cluster-setup
 2021-09-15 18:04:30 +03:00
Aliaksandr Valialkin
184e145570 docs: consistency renaming: Influx -> InfluxDB 2021-09-13 17:14:45 +03:00
Aliaksandr Valialkin
9c19719ad6 app/{vminsert,vmselect}: export vminsert_request_duration_seconds and vmselect_request_duration_seconds histograms 2021-07-07 13:27:23 +03:00
Aliaksandr Valialkin
ceda2b1df4 lib/httpserver: print full requestURI in httpserver.Errorf 
This should simplify debugging.
 2021-07-07 13:11:29 +03:00
Aliaksandr Valialkin
4a5f45c77e app/vminsert: add support for data ingestion via other vminsert nodes 2021-05-08 19:53:45 +03:00
Aliaksandr Valialkin
6dc5d3b357 all: rename https://victoriametrics.github.io to https://docs.victoriametrics.com 2021-04-20 20:20:01 +03:00
Aliaksandr Valialkin
4028d692f5 app: do not process non-GET requests on at / handler 2021-04-02 22:56:38 +03:00
Aliaksandr Valialkin
e2717d84c0 all: various fixes in command-line flag descriptions 2021-03-15 22:03:49 +02:00
Aliaksandr Valialkin
776b8b32ca app/{vminsert,vmagent}: a follow-up for b1aa8c3d8f 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1124
 2021-03-15 22:03:49 +02:00
Nikolay
a843dc0219 adds fake response for telegraph queries (#1130) 
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1124
 2021-03-15 22:03:49 +02:00
Aliaksandr Valialkin
a2eb451de4 app/{vmagent,vminsert}: follow-up for ce8c2dd1f1: return /targets page in HTML when requested via web browser 2020-12-14 14:13:01 +02:00
Aliaksandr Valialkin
1a237c6903 all: properly handle CPU limits set on the host system/container 
This can reduce memory usage on systems with enabled CPU limits.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/946
 2020-12-08 21:07:03 +02:00
Aliaksandr Valialkin
bdac2171f1 all: do not print usage info for all the flags when incorrect command-line flag is passed 
This should improve usability for VictoriaMetrics apps that have big number of command-line flags,
i.e. all the apps.
 2020-12-03 21:46:19 +02:00
BigFish
3159b41689 Update main.go (#922) 
fix spelling mistake
 2020-11-24 12:36:47 +02:00
Aliaksandr Valialkin
abfd3a8fab app/{vminsert,vmselect,vmstorage}: add a link to https://victoriametrics.github.io/Cluster-VictoriaMetrics.html from main page of every cluster component 2020-10-06 15:30:07 +03:00
Aliaksandr Valialkin
8df33bd5c1 app/{vminsert,vmagent}: improve data ingestion speed over a single connection 
Process data obtianed from a single connection on all the available CPU cores.
 2020-09-28 04:14:51 +03:00
Aliaksandr Valialkin
aadbd014ff all: add native format for data export and import 
The data can be exported via [/api/v1/export/native](https://victoriametrics.github.io/#how-to-export-data-in-native-format) handler
and imported via [/api/v1/import/native](https://victoriametrics.github.io/#how-to-import-data-in-native-format) handler.
 2020-09-27 17:36:38 +03:00
Aliaksandr Valialkin
b996280c65 app/{vminsert,vmagent}: improve documentation for -influxListenAddr command-line flag 2020-08-14 18:03:08 +03:00
Aliaksandr Valialkin
6721e47ae9 app: respect CPU limits set via cgroups 
Update GOMAXPROCS to limits set via cgroups. This should reduce CPU trashing and reduce memory usage
for cases when VictoriaMetrics components run in containers with CPU limits.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685
 2020-08-11 23:01:03 +03:00
Aliaksandr Valialkin
31ef39e8da lib/httpserver: log remote address in error message from httpserver.Errorf 
This should improve detection of the root cause of errors.
Thanks to Anant for the idea.
 2020-07-20 14:06:29 +03:00
Aliaksandr Valialkin
6373d377ef app/{vminsert,vmagent}: add ability to import data in Prometheus exposition format via /api/v1/import/prometheus 2020-07-10 12:13:28 +03:00
Aliaksandr Valialkin
6ebac3ab63 app/vminsert: add ability to apply relabeling to all the incoming metrics if -relabelConfig command-line arg points to a file with a list of relabel_config entries 
See https://victoriametrics.github.io/#relabeling
 2020-07-02 20:36:33 +03:00
Aliaksandr Valialkin
4f673a5201 app/vminsert: export metrics for determining ingested rows with dropped or truncated labels 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/565
 2020-06-19 01:12:44 +03:00
Aliaksandr Valialkin
2784015a4d all: print --help output to stdout instead of stderr 
This is easier to grep and pipe
 2020-05-16 12:03:06 +03:00
Aliaksandr Valialkin
a53e332a93 app/vmstorage: add missing shutdown for http server on graceful shutdown 
This could result in the following panic during graceful shutdown when `/metrics` page is requested:

http: panic serving 10.101.66.5:57366: runtime error: invalid memory address or nil pointer dereference
goroutine 2050 [running]:
net/http.(*conn).serve.func1(0xc00ef22000)
	net/http/server.go:1772 +0x139
panic(0xa0fc00, 0xe91d80)
	runtime/panic.go:973 +0x3e3
github.com/VictoriaMetrics/VictoriaMetrics/lib/workingsetcache.(*Cache).UpdateStats(0x0, 0xc0000516c8)
	github.com/VictoriaMetrics/VictoriaMetrics/lib/workingsetcache/cache.go:224 +0x37
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*indexDB).UpdateMetrics(0xc00b931d00, 0xc02c41acf8)
	github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/index_db.go:258 +0x9f
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*Storage).UpdateMetrics(0xc0000bc7e0, 0xc02c41ac00)
	github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/storage.go:413 +0x4c5
main.registerStorageMetrics.func1(0x0)
	github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage/main.go:186 +0xd9
main.registerStorageMetrics.func3(0xc00008c380)
	github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage/main.go:196 +0x26
main.registerStorageMetrics.func7(0xc)
	github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage/main.go:211 +0x26
github.com/VictoriaMetrics/metrics.(*Gauge).marshalTo(0xc000010148, 0xaa407d, 0x20, 0xb50d60, 0xc005319890)
	github.com/VictoriaMetrics/metrics@v1.11.2/gauge.go:38 +0x3f
github.com/VictoriaMetrics/metrics.(*Set).WritePrometheus(0xc000084300, 0x7fd56809c940, 0xc005319860)
	github.com/VictoriaMetrics/metrics@v1.11.2/set.go:51 +0x1e1
github.com/VictoriaMetrics/metrics.WritePrometheus(0x7fd56809c940, 0xc005319860, 0xa16f01)
	github.com/VictoriaMetrics/metrics@v1.11.2/metrics.go:42 +0x41
github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.writePrometheusMetrics(0x7fd56809c940, 0xc005319860)
	github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver/metrics.go:16 +0x44
github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.handlerWrapper(0xb5a120, 0xc005319860, 0xc005018f00, 0xc00002cc90)
	github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver/httpserver.go:154 +0x58d
github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver.gzipHandler.func1(0xb5a120, 0xc005319860, 0xc005018f00)
	github.com/VictoriaMetrics/VictoriaMetrics/lib/httpserver/httpserver.go:119 +0x8e
net/http.HandlerFunc.ServeHTTP(0xc00002d110, 0xb5a660, 0xc0044141c0, 0xc005018f00)
	net/http/server.go:2012 +0x44
net/http.serverHandler.ServeHTTP(0xc004414000, 0xb5a660, 0xc0044141c0, 0xc005018f00)
	net/http/server.go:2807 +0xa3
net/http.(*conn).serve(0xc00ef22000, 0xb5bf60, 0xc010532080)
	net/http/server.go:1895 +0x86c
created by net/http.(*Server).Serve
	net/http/server.go:2933 +0x35c
 2020-04-02 21:09:55 +03:00
Aliaksandr Valialkin
2f0a36044c app/{vmagent,vminsert}: add support for importing csv data via /api/v1/import/csv 2020-03-10 21:17:40 +02:00
Aliaksandr Valialkin
2471340e0d app/vmagent: add ability to accept Influx line protocol data via TCP and UDP 
Just set `-influxListenAddr` command-line flag

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/333
 2020-02-25 19:18:01 +02:00
Aliaksandr Valialkin
7ee7614e90 app/vmagent: initial implementation for vmagent 2020-02-23 17:31:54 +02:00