Commit graph

167 commits

Author SHA1 Message Date
Nikolay
d44058bcd6
app/vmauth: adds idleConnTimeout flag, retry trivial errors (#6388)
* adds idleConnTimeout flag, which must reduce probability of `broken
pipe` and `connection reset` errors.
* one-time retry trivial network requests for the same backend

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-06-10 12:36:37 +02:00
Hui Wang
61dce6f2a1
lib/httpserver: allow reloadAuthKey and configAuthKey to override htt… (#6338)
…pAuth.*

address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6329, 
makes `reloadAuthKey`, `configAuthKey`, `flagsAuthKey`, `pprofAuthKey`
behavior the same way,
but keys like `-snapshotAuthKey`, `-forceMergeAuthKey` are still
protected by httpAuth.*. All the available key are listed in
https://docs.victoriametrics.com/single-server-victoriametrics/#security.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-06-10 12:09:47 +02:00
Arkadii Yakovets
c740a8042e
docs: fix docs/ and README.md spelling errors (#6362)
Fixes `docs/` and `README.md` typos and errors.

Signed-off-by: Arkadii Yakovets <ark@victoriametrics.com>
2024-06-03 10:04:13 +02:00
Aliaksandr Valialkin
bac28f2e4d
docs/vmauth.md: small fixes after proofreading 2024-05-12 12:35:23 +02:00
hagen1778
6aaf1768f4
Revert "docs: removed code-style highlighting for commanad-line flags of VM components (#6147)"
This reverts commit 9bedbcfa2f.
2024-04-25 11:23:02 +02:00
Denys Holius
9bedbcfa2f
docs: removed code-style highlighting for commanad-line flags of VM components (#6147)
Using `sh` or `console` formatting doesn't do word-breaking on render. This makes flags description
harder to read, as users need to scroll the web page horizontally.
Removing the formatting renders the description with normal word-breaking.
2024-04-22 14:53:26 +02:00
Aliaksandr Valialkin
f4b1cbfef0
all: replace old https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html url with the new one - https://docs.victoriametrics.com/cluster-victoriametrics/ 2024-04-18 02:54:20 +02:00
Aliaksandr Valialkin
e128ef7ace
all: replace old https://docs.victoriametrics.com/keyConcepts.html url with the new one - https://docs.victoriametrics.com/keyconcepts/ 2024-04-18 02:33:29 +02:00
Aliaksandr Valialkin
6c14d08cb3
all: replace old https://docs.victoriametrics.com/vmgateway.html url with the new one - https://docs.victoriametrics.com/vmgateway/ 2024-04-18 02:08:18 +02:00
Aliaksandr Valialkin
1e24b334f1
all: replace old https://docs.victoriametrics.com/vmauth.html url with the new one - https://docs.victoriametrics.com/vmauth/ 2024-04-18 01:49:41 +02:00
Aliaksandr Valialkin
b4fac26360
all: replace old https://docs.victoriametrics.com/vmalert.html url with the new one - https://docs.victoriametrics.com/vmalert/ 2024-04-18 01:44:12 +02:00
Aliaksandr Valialkin
c81a633b02
all: replace the outdated url https://docs.victoriametrics.com/vmagent.html with the new one - https://docs.victoriametrics.com/vmagent/ 2024-04-18 01:31:37 +02:00
Aliaksandr Valialkin
50ac22df78
lib/httpserver: add support for automatic issuing of TLS certificates via Lets Encrypt service
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5949
2024-04-17 23:50:57 +02:00
Aliaksandr Valialkin
b426d10847
app/vmauth: add support for configuring backends via DNS SRV urls 2024-04-17 20:46:22 +02:00
Aliaksandr Valialkin
07ed958b82
app/vmauth: add support for client TLS sertificates for backend requests over https
While at it, also add support for TLS ServerName for backend requests over https
2024-04-17 17:12:22 +02:00
Aliaksandr Valialkin
21a9b1f920
docs/vmauth.md: add Authorization and Routing chapters 2024-04-17 16:17:14 +02:00
Aliaksandr Valialkin
db85744e04
app/vmauth: follow-up for b155b20de4
- Use exact matching by default for the query arg value provided via arg=value syntax at src_query_args.
  Regex matching can be enabled by using =~ instead of = . For example, arg=~regex.
  This ensures that the exact matching works as expected without the need to escape special regex chars.

- Add helper functions for creating QueryArg, Header and Regex structs in tests.
  This improves maintainability of the tests.

- Remove url.QueryUnescape() call on the url in TestCreateTargetURLSuccess(), since this is bogus approach.
  The url.QueryUnescape() must be applied to individual query args, and it mustn't be applied to the whole url,
  since in this case it may perform invalid unescaping in the context of the url, or make the resulting url invalid.

While at it, properly marshal all the fields inside UserInfo config to yaml in tests.
Previously Header and QueryArg structs were improperly marshaled because the custom MarshalYAML
is called only on pointers to Header and QueryArg structs. This improves test coverage.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6070
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6115
2024-04-17 14:27:52 +02:00
Roman Khavronenko
b155b20de4
app/vmauth: support regex matching in src_query_args (#6115)
Support regex matching when routing incoming requests based on HTTP query args
via `src_query_args` option at `url_map`.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6070

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-04-17 09:54:43 +02:00
Aliaksandr Valialkin
00e5e00a5a
docs: change old url from https://docs.victoriametrics.com/enterprise.html to new url https://docs.victoriametrics.com/enterprise/ 2024-04-04 15:21:59 +03:00
Aliaksandr Valialkin
be36ceb1cf
app/vmauth: add ability to authorize via any opaque HTTP request header value
This can be done via `auth_token` option at -auth.config - see https://docs.victoriametrics.com/vmauth/#auth-config
2024-04-02 21:16:11 +03:00
Aliaksandr Valialkin
21bfb66650
app/vmauth: add ability to read auth tokens from multiple http request headers
This is needed for VictoriaMetrics Cloud, where the same token could be passed either
via Authorization or via X-Amz-Firehose-Access-Key header - see 4487dac30b (r140500722)

This is a follow-up for 4487dac30b

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6009
2024-04-02 19:29:00 +03:00
Aliaksandr Valialkin
4487dac30b
app/vmauth: follow-up for bc90f4aae6
- Allow specifying only a single HTTP header for reading auth tokens via -httpAuthHeader command-line flag.
  This is better from security PoV, since this prevents from accidental reading of auth token from undesired
  HTTP header. By default the -httpAuthHeader equals to Authorization. When it is overridden, then
  auth token isn't read from Authorization header - it is read only from the specified header.

- Document the -httpAuthHeader command-line flag at https://docs.victoriametrics.com/vmauth/#reading-auth-tokens-from-other-http-headers

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6009
2024-04-02 18:35:21 +03:00
Andrii Chubatiuk
bc90f4aae6
vmauth: support other auth header names besides Authorization (#6009) 2024-03-26 13:21:07 +01:00
Aliaksandr Valialkin
98b31b7f7c
docs/vmauth.md: update -help output after e08b91baafc95da090f75e9c29a27d8f62a2b76e 2024-03-07 01:37:39 +02:00
Aliaksandr Valialkin
c0a93cf183
docs/vmauth.md: typo fixes after 7b2b980181 2024-03-07 01:08:33 +02:00
Aliaksandr Valialkin
7b2b980181
app/vmauth: allow discovering backend ips behind shared hostname and spreading load among the discovered ips
This is done with the `discover_backend_ips` option at `user` and `url_map` level.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5707
2024-03-07 01:02:16 +02:00
Aliaksandr Valialkin
76ef84fcae
app/vmauth: add src_headers option at url_map, which allows routing incoming requests to different backends depending on request headers 2024-03-06 21:56:32 +02:00
Aliaksandr Valialkin
8efe12d66e
app/vmauth: simplify configuration for src_query_args
Use the shorter form:

src_query_args:
- arg1=value1
- arg2=value2

instead of

src_query_args:
- name: arg1
  value: value2
- name: arg2
  value: value2

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5878
2024-03-06 21:19:45 +02:00
Aliaksandr Valialkin
b2efacb624
docs/vmauth.md: mention that request query args can used for routing decisions
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5878
2024-03-06 20:58:29 +02:00
Aliaksandr Valialkin
61d1af8050
app/vmauth: add ability to route requests based on HTTP query args via src_query_args option
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5878
2024-03-06 20:52:25 +02:00
Aliaksandr Valialkin
5b33da5e19
docs: update -help output after recent changes in VictoriaMetrics components 2024-03-01 05:31:05 +02:00
Aliaksandr Valialkin
cf94522389
lib/httpserver: return back the default value for -http.connTimeout to 2 minutes
It has been appeared that there are VictoriaMetrics users, who rely on the fact that
VictoriaMetrics components were closing incoming connections to -httpListenAddr every 2 minutes
by default. So let's return back this value by default in order to fix the breaking change
made at d8c1db7953 .

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1304#issuecomment-1961891450 .
2024-02-23 22:03:37 +02:00
Aliaksandr Valialkin
f120477231
docs/vmauth.md: add Config reload chapter, which explains how to reload -auth.config at vmauth
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1194
2024-02-13 20:29:35 +02:00
Aliaksandr Valialkin
cafdc7e21a
docs/vmauth.md: add missing dot 2024-02-13 01:09:04 +02:00
Aliaksandr Valialkin
062cbb1130
app/vmauth: add support for mTLS-based routing of incoming requests to different backends depending on the subject field in the TLS certificate provided by the user
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1547
2024-02-13 01:03:20 +02:00
Aliaksandr Valialkin
95222b2079
all: upgrade Go builder from Go1.21.7 to Go1.22.0
See https://go.dev/doc/go1.22
2024-02-12 21:59:51 +02:00
Aliaksandr Valialkin
64723e591e
docs: update docs after ae8a867924
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1470
2024-02-09 04:18:53 +02:00
Aliaksandr Valialkin
7bc3af1224
lib/httpserver: add support for mTLS for requests to -httpListenAddr 2024-02-06 17:46:19 +02:00
Aliaksandr Valialkin
6136c19fbc
docs: mention -metrics.exposeMetadata command-line flag in Monitoring sections
This is a follow-up for 326a77c697
2024-02-05 10:49:16 +02:00
Aliaksandr Valialkin
8f9eddb1e4
docs: sync -help output after recent changes 2024-02-01 15:06:25 +02:00
Aliaksandr Valialkin
9922a486a6
docs/vmauth.md: typo fix after 68be182075 2024-01-31 00:13:50 +02:00
hagen1778
6c6c2c185f
docs: follow-up after 491287ed15
* port un-synced changed from docs/readme to readme
* consistently use `sh` instead of `console` highlight, as it looks like
a more appropriate syntax highlight
* consistently use `sh` instead of `bash`, as it is shorter
* consistently use `yaml` instead of `yml`

See syntax codes here https://gohugo.io/content-management/syntax-highlighting/

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-27 19:29:11 +01:00
Artem Navoiev
3e0a117ddf
remove all <div> as far they obsolete and can break markdown (#5701)
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2024-01-26 12:52:21 -08:00
Aliaksandr Valialkin
1f105dde98
all: allow dynamically reading *AuthKey flag values from files and urls
Examples:

1) -metricsAuthKey=file:///abs/path/to/file - reads flag value from the given absolute filepath
2) -metricsAuthKey=file://./relative/path/to/file - reads flag value from the given relative filepath
3) -metricsAuthKey=http://some-host/some/path?query_arg=abc - reads flag value from the given url

The flag value is automatically updated when the file contents changes.
2024-01-21 22:03:38 +02:00
Nikolay
b3598ba2c1
app/vmauth: adds metric_labels and backend_errors counter (#5585)
* app/vmauth: adds metric_labels and backend_errors counter
it must improve observability for user requests with new metric - per user backend errors counter.
it's needed to calculate requests fail rate to the configured backends.
metric_labels configuration allows to perform additional aggregations on top of multiple users from configuration section.
It could be multiple clients or clients with separate read/write tokens
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5565

* wip

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-01-21 04:40:52 +02:00
Aliaksandr Valialkin
326a77c697
all: add -metrics.exposeMetadata command-line flag, which can be used for adding TYPE and HELP metadata for metrics exposed at /metrics page
This may be needed for systems, which require this metadata such as Google Cloud Managed Prometheus.
See https://cloud.google.com/stackdriver/docs/managed-prometheus/troubleshooting#missing-metric-type
2023-12-19 03:20:40 +02:00
Aliaksandr Valialkin
4b529562ce
lib/pushmetrics: add -pushmetrics.header and -pushmetrics.disableCompression command-line flags 2023-12-17 19:56:46 +02:00
Aliaksandr Valialkin
68be182075
app/vmauth: add ability to route requests to different backends depending on the request host 2023-12-14 00:46:36 +02:00
Aliaksandr Valialkin
a5bc9d93cc
docs: sync -help output for VictoriaMetrics components after recent changes 2023-12-10 01:13:41 +02:00
Aliaksandr Valialkin
b1fed78e0b
app: make more clear that -tls enables https at -httpListenAddr 2023-12-10 00:25:01 +02:00