Commit graph

47 commits

Author SHA1 Message Date
Aliaksandr Valialkin
e4bb2808f1
app/vmselect: add support for vmstorage groups with independent -replicationFactor per group
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5197

See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#vmstorage-groups-at-vmselect

Thanks to @zekker6 for the initial pull request at https://github.com/VictoriaMetrics/VictoriaMetrics-enterprise/pull/718
2023-12-13 00:14:34 +02:00
Aliaksandr Valialkin
a45cbc101f
all: cleanup: remove // +build ... lines, since they are no longer needed after Go1.17, and the minimum supported Go version for VictoriaMetrics source code is Go1.20 2023-11-13 19:15:42 +01:00
Aliaksandr Valialkin
36a1fdca6c
all: consistently use %w instead of %s in when error is passed to fmt.Errorf()
This allows consistently using errors.Is() for verifying whether the given error wraps some other known error.
2023-10-26 09:44:40 +02:00
Aliaksandr Valialkin
d8afd7fe98
Makefile: update golangci-lint from v1.51.2 to v1.54.2
See https://github.com/golangci/golangci-lint/releases/tag/v1.54.2
2023-09-01 10:25:49 +02:00
Aliaksandr Valialkin
339879edd0
lib/netutil/tcpdialer.go: reduce the code difference with enterprise branch 2023-08-29 12:45:57 +02:00
Aliaksandr Valialkin
19d61737c1
app/{vminsert,vmselect}: follow-up after 2b7b3293c1
- Document the change at docs/CHANGELOG.md
- Set the default value for -vmstorageUserTimeout to 3 seconds. This is much better
  than the 0 value, which means that TCP connection to unreachable vmstorage could block
  for up to 16 minutes.
- Document -vmstorageUserTimeout at docs/Cluster-VictoriaMetrics.md
2023-08-29 12:17:39 +02:00
Will Jordan
2b7b3293c1
Add vmstorageUserTimeout flags to configure TCP user timeout (Linux) (#4423)
`TCP_USER_TIMEOUT` (since Linux 2.6.37) specifies the maximum amount of
time that transmitted data may remain unacknowledged before TCP will
forcibly close the connection and return `ETIMEDOUT` to the application.

Setting a low TCP user timeout allows RPC connections quickly reroute
around unavailable storage nodes during network interruptions.
2023-08-29 11:46:39 +02:00
Roman Khavronenko
80768d53dd
docs: follow-up after aec4b5db81 (#4638)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-19 14:48:17 -07:00
Aliaksandr Valialkin
833a0e25a7
lib/netutil: ignore arificial timeout generated by net/http.Server
This prevents from the inflated vm_tcplistener_read_timeouts_total counter
2023-07-06 17:26:15 -07:00
Nikolay
043431093a
app/vmauth: properly handle LOCAL proxy protocol command (#4373)
app/vmauth: properly handle LOCAL proxy protocol command

It is required for handling health checks from load balancers

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335
(cherry picked from commit f263031fe9)
2023-06-02 13:29:15 +02:00
Alexander Marshalov
d321ea91f2
fixed typos in documentation and commandline flags descriptions (#4275) 2023-05-10 02:22:06 -07:00
Nikolay
9bb83cafa4
lib/netutil: log only parsing errors for proxy-protocol (#3985)
* lib/netutil: log only parsing errors for proxy-protocol

Previosly every error was logged. With configured TCP health checks at load-balancer or kubernetes, vmauth spams a lot of false positive error message into logs

* Update docs/CHANGELOG.md

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* Update lib/netutil/tcplistener.go

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-03-21 10:23:08 -07:00
Nikolay
88f10d24a0
lib/netutil: fixes panic at proxy protocol (#3905)
it may occur if non proxy protocol message received by tcp server.
Listener Accept method must return only non-recoverable errors.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335
2023-03-08 01:33:01 -08:00
Oleksandr Redko
0e1c395609
app,lib: fix typos in comments (#3804) 2023-02-13 09:32:35 -08:00
Aliaksandr Valialkin
bccbe07c33
lib/netutil: move IsTrivialNetworkError() function there, since it is used in multiple places across the code 2023-01-27 13:24:44 -08:00
Aliaksandr Valialkin
eb10102521
lib/netutil: typo fix in the error message 2023-01-27 11:31:05 -08:00
Aliaksandr Valialkin
b17857c7a4
lib/netutil: limit the time needed for reading proxy protocol headers
This should prevent from misconfigured proxies and from possible Slowloris-type DoS attacks
(see https://en.wikipedia.org/wiki/Slowloris_(computer_security) )

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335
2023-01-26 23:47:06 -08:00
Nikolay
ebebaecd94
lib/netutil: init implimentation of proxy protocol (#3687)
* lib/netutil: init implimentation of proxy protocol
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-01-26 23:25:22 -08:00
Aliaksandr Valialkin
4f53147ed4
app/{vminsert,vmselect}/netstorage: allow calling Init()+MustStop() in a loop
Previously netstorage.MustStop() call didn't free up all the resources,
so the subsequent call to nestorage.Init() would panic.

This allows writing tests, which call nestorage.Init() + nestorage.MustStop() in a loop.
2022-10-25 14:43:05 +03:00
Aliaksandr Valialkin
41882222d3
lib/netutil/tls.go: consistently use tlsMinVersion name across source code
This should simplify further code maintenance and refactoring

This is a follow-up after 6ab1cede62
2022-09-26 17:59:07 +03:00
Dmytro Kozlov
28dcff5791
lib/{httpserver,netutil}: allow to define min and max TLS version of the http server (#3109)
* lib/{httpserver,netutil}: allow to define min and max TLS version of the http server

* lib/httpserver: added descriptions about tls supported versions

* lib/netutil: check minimal tls version, added supported tls versions to error

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-09-26 17:38:43 +03:00
Aliaksandr Valialkin
288d13af8d
lib/netutil: parallelize background pings for remote addresses
This should improve the time needed for determining unavailale remote addresses
across big numer of ConnPool's.

This is a follow-up for a1629bd3be

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711
2022-06-21 13:32:27 +03:00
Aliaksandr Valialkin
a1629bd3be
lib/netutil.ConnPool: skip dialing remote address if the previous dial attempt was unsuccessful
If the previous dial attempt was unsuccessful, then all the new dial attempts are skipped
until the background goroutine determines that the given address can be successfully dialed.

This reduces query latency when some of vmstorage nodes are unavailable and dialing them is slow.

This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711

This commit is based on ideas from the https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2756

The main differences are:

- The check for healthy/unhealthy storage nodes is moved one level lower from app/vmselect/netstorage to lib/netutil.ConnPool.
  This makes possible re-using this feature everywhere lib/netutil.ConnPool is used.
- The check doesn't take into account handshake errors for already established connections.
  Handshake errors usually mean improperly configured VictoriaMetrics cluster, so they shouldn't be ignored.
2022-06-20 17:33:54 +03:00
Aliaksandr Valialkin
b28c6febf9
app/{vminsert,vmselect}: add -vmstorageDialTimeout command-line flag for tuning the maximum time needed for establishing connections to vmstorage
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711
2022-06-20 15:17:34 +03:00
Aliaksandr Valialkin
d8a276fbe4
lib/netutil: limit the number of concurrently established connections when calling ConnPool.Get()
This should reduce potential spikes in the number of established connections in the following cases:
- when the connection establishing procedure becomes temporarily slow
- after a temporary spike in the rate of ConnPool.Get() calls

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2552
2022-05-11 14:11:06 +03:00
Aliaksandr Valialkin
190c8b463c
lib/netutil: close connections in ConnPool if they are idle for more than 30 seconds
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2508
2022-05-02 15:01:52 +03:00
Aliaksandr Valialkin
a7689e1b0c
app/vmstorage: add support for mTLS cipher suites via -cluster.tlsCipherSuites command-line flag
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2404
2022-04-16 16:36:38 +03:00
Aliaksandr Valialkin
27e74f25d6
lib/httpserver: follow up after def0032c7d 2022-04-16 15:52:44 +03:00
Dmytro Kozlov
26ae50ec26
lib/httpserver: added tlsCipherSuites flag (#2468)
* lib/httpserver: added tlsCipherSuites flag

* lib/httpserver: compare lower case strings

* lib/httpserver: use EqualFold

* lib/httpserver: used flagutil.NewArray, supported only strings cipher suites

* lib/httpserver: updated flag description, added flag to documentation

* Update lib/httpserver/httpserver.go

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-04-16 15:52:42 +03:00
Aliaksandr Valialkin
698458b742
lib/httpserver: extract the code responsible for initializing server-side TLS config into netutil.GetServerTLSConfig 2022-03-17 19:46:20 +02:00
Aliaksandr Valialkin
e2d12a25e0
lib/netutil: increase dial timeout from 1 second to 5 seconds
There are real-world cases when TCP connection needs more than 1 second to be established.
2022-02-07 12:33:40 +02:00
mxlxm
42e07cfaea
reset deadline, fix #1562. (#1597)
* reset deadline, fix #1562.
reset deadline before we put it back to pool.

* make errcheck happy
2021-09-07 20:54:17 +03:00
Aliaksandr Valialkin
8005ba26b9 lib/netutil: enable IPv6 UDP listening if -enableTCP6 command-line flag is passed to VictoriaMetrics
This is a follow-up for 18cfc4be7b

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1131
2021-03-17 00:19:30 +02:00
Aliaksandr Valialkin
367fc17933 lib/promscrape: code prettifying for 8dd03ecf19 2020-12-24 10:57:20 +02:00
Nikolay
b00f7816e2 adds proxy_url support, (#980)
* adds proxy_url support,
adds proxy_url to the dockerswarm, eureka, kubernetes and consul service discovery,
adds proxy_url to the scrape_config for targets scrapping,
http based proxy is supported atm,
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/503

* fixes imports
2020-12-24 10:57:19 +02:00
Aliaksandr Valialkin
fe77d661b3 all: use errors.As instead of type assertion for detecting net.Error 2020-07-01 00:16:13 +03:00
Aliaksandr Valialkin
d962568e93 all: use %w instead of %s for wrapping errors in fmt.Errorf
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
2020-06-30 23:33:46 +03:00
Aliaksandr Valialkin
3052b479b7 lib/httpserver: reduce typical duration for http server graceful shutdown
Previously the duration for graceful shutdown for http server could take more than a minute
because of imporperly set timeouts in setNetworkTimeout.
Now typical duration for graceful shutdown should be reduced to less than 5 seconds.
2020-05-07 14:16:38 +03:00
Aliaksandr Valialkin
7ee7614e90 app/vmagent: initial implementation for vmagent 2020-02-23 17:31:54 +02:00
Aliaksandr Valialkin
a7bf8e77af app/vminsert: simultaneously accept telnet put and HTTP /api/put OpenTSDB metrics at -opentsdbListenAddr
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/266
2019-12-14 00:42:18 +02:00
Aliaksandr Valialkin
d456ec7589 lib/netutil: use IPv6 for both listening and dialing if -enabledTCP6 is set
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/244
2019-12-01 02:52:53 +02:00
Aliaksandr Valialkin
4810f1dde6 lib/netutil: add -enableTCP6 command-line flag for enabling listening for IPv6 additionally to IPv4 TCP ports 2019-11-29 17:33:07 +02:00
Aliaksandr Valialkin
5b01b7fb01 all: add support for GOARCH=386 and fix all the issues related to 32-bit architectures such as GOARCH=arm
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/212
2019-10-17 18:27:49 +03:00
Aliaksandr Valialkin
7abb96b454 lib/netutil: do not count timeouts as network errors 2019-07-15 23:06:13 +03:00
Aliaksandr Valialkin
7778030f9f lib/netutil: remove unused TCPListener.name; updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/69 2019-06-20 14:36:19 +03:00
Aliaksandr Valialkin
24578b4bb1 all: open-sourcing cluster version 2019-05-23 00:25:38 +03:00
Aliaksandr Valialkin
1836c415e6 all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00