github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2025-01-10 15:14:09 +00:00
Code Issues Projects Releases Packages Wiki Activity

app/vmalert: hide sensetive info in the vmalert (#5059)

Browse source 
Strip sensitive information such as auth headers or passwords from datasource, remote-read,
remote-write or notifier URLs in log messages or UI. This behavior is by default and is controlled via
`-datasource.showURL`, `-remoteRead.showURL`, `remoteWrite.showURL` or `-notifier.showURL` cmd-line flags.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5044
(cherry picked from commit 244c887825)
This commit is contained in:
Dmytro Kozlov 2023-10-10 11:40:27 +02:00 committed by hagen1778
parent a906837403
commit 1cc6cd3d4f
No known key found for this signature in database
GPG key ID: 3BF75F3741CA9640
10 changed files with 164 additions and 36 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

27
app/vmalert/README.md
View file

@ -820,7 +820,8 @@ and check the `Last updates` section:
Rows in the section represent ordered rule evaluations and their results. The column `curl` contains an example of
HTTP request sent by vmalert to the `-datasource.url` during evaluation. If specific state shows that there were
no samples returned and curl command returns data - then it is very likely there was no data in datasource on the
moment when rule was evaluated.
moment when rule was evaluated. Sensitive info is stripped from the `curl` examples - see [security](#security) section
for more details.
### Debug mode
@ -836,6 +837,8 @@ Just set `debug: true` in rule's configuration and vmalert will start printing a
2022-09-15T13:36:56.153Z DEBUG rule "TestGroup":"Conns" (2601299393013563564) at 2022-09-15T15:36:56+02:00: alert 10705778000901301787 {alertgroup="TestGroup",alertname="Conns",cluster="east-1",instance="localhost:8429",replica="a"} PENDING => FIRING: 1m0s since becoming active at 2022-09-15 15:35:56.126006 +0200 CEST m=+39.384575417
```
Sensitive info is stripped from the `curl` examples - see [security](#security) section for more details.
### Never-firing alerts
vmalert can detect if alert's expression doesn't match any time series in runtime
@ -880,6 +883,20 @@ The same issue can be caused by collision of configured `labels` on [Group](#gro
To fix it one should avoid collisions by carefully picking label overrides in configuration.
## Security
See general recommendations regarding security [here](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#security).
vmalert [web UI](#web) exposes configuration details such as list of [Groups](#groups), active alerts,
[alerts state](#alerts-state), [notifiers](#notifier-configuration-file). Notifier addresses (sanitized) are attached
as labels to metrics `vmalert_alerts_sent_.*` on `http://<vmalert>/metrics` page. Consider limiting user's access
to the web UI or `/metrics` page if this information is sensitive.
[Alerts state](#alerts-state) page or [debug mode](#debug-mode) could emit additional information about configured
datasource URL, GET params and headers. Sensitive information such as passwords or auth tokens is stripped by default.
To disable stripping of such info pass `-datasource.showURL` cmd-line flag to vmalert.
## Profiling
`vmalert` provides handlers for collecting the following [Go profiles](https://blog.golang.org/profiling-go-programs):
@ -962,7 +979,8 @@ The shortlist of configuration flags is the following:
-datasource.roundDigits int
Adds "round_digits" GET param to datasource requests. In VM "round_digits" limits the number of digits after the decimal point in response values.
-datasource.showURL
Whether to show -datasource.url in the exported metrics. It is hidden by default, since it can contain sensitive info such as auth key
Whether to avoid stripping sensitive information such as auth headers or passwords from URLs in log messages or UI and exported metrics.
It is hidden by default, since it can contain sensitive info such as auth key.
-datasource.tlsCAFile string
Optional path to TLS CA file to use for verifying connections to -datasource.url. By default, system CA is used
-datasource.tlsCertFile string
@ -982,7 +1000,7 @@ The shortlist of configuration flags is the following:
-disableAlertgroupLabel
Whether to disable adding group's Name as label to generated alerts and time series.
-dryRun
Whether to check only config files without running vmalert. The rules file are validated. The -rule flag must be specified.
Whether to check only config files without running vmalert. The rules file are validated. The -rule flag must be specified.
-enableTCP6
Whether to enable IPv6 for listening and dialing. By default, only IPv4 TCP and UDP are used
-envflag.enable
@ -1111,6 +1129,9 @@ The shortlist of configuration flags is the following:
-notifier.url array
Prometheus Alertmanager URL, e.g. http://127.0.0.1:9093. List all Alertmanager URLs if it runs in the cluster mode to ensure high availability.
Supports an array of values separated by comma or specified via multiple flags.
-notifier.showURL bool
Whether to avoid stripping sensitive information such as passwords from URLs in log messages or UI for -notifier.url.
It is hidden by default, since it can contain sensitive info such as auth key.
-notifier.blackhole bool
Whether to blackhole alerting notifications. Enable this flag if you want vmalert to evaluate alerting rules without sending any notifications to external receivers (eg. alertmanager). `-notifier.url`, `-notifier.config` and `-notifier.blackhole` are mutually exclusive.
-pprofAuthKey string

7
app/vmalert/datasource/init.go
View file

@ -16,7 +16,7 @@ var (
addr = flag.String("datasource.url", "", "Datasource compatible with Prometheus HTTP API. It can be single node VictoriaMetrics or vmselect URL. Required parameter. "+
"E.g. http://127.0.0.1:8428 . See also -remoteRead.disablePathAppend and -datasource.showURL")
appendTypePrefix = flag.Bool("datasource.appendTypePrefix", false, "Whether to add type prefix to -datasource.url based on the query type. Set to true if sending different query types to the vmselect URL.")
showDatasourceURL = flag.Bool("datasource.showURL", false, "Whether to show -datasource.url in the exported metrics. "+
showDatasourceURL = flag.Bool("datasource.showURL", false, "Whether to avoid stripping sensitive information such as auth headers or passwords from URLs in log messages or UI and exported metrics. "+
"It is hidden by default, since it can contain sensitive info such as auth key")
headers = flag.String("datasource.headers", "", "Optional HTTP extraHeaders to send with each request to the corresponding -datasource.url. "+
@ -62,6 +62,11 @@ func InitSecretFlags() {
}
}
// ShowDatasourceURL whether to show -datasource.url with sensitive information
func ShowDatasourceURL() bool {
return *showDatasourceURL
}
// Param represents an HTTP GET param
type Param struct {
Key, Value string

10
app/vmalert/datasource/vm.go
View file

@ -198,17 +198,21 @@ func (s *VMStorage) QueryRange(ctx context.Context, query string, start, end tim
}
func (s *VMStorage) do(ctx context.Context, req *http.Request) (*http.Response, error) {
ru := req.URL.Redacted()
if *showDatasourceURL {
ru = req.URL.String()
}
if s.debug {
logger.Infof("DEBUG datasource request: executing %s request with params %q", req.Method, req.URL.RawQuery)
logger.Infof("DEBUG datasource request: executing %s request with params %q", req.Method, ru)
}
resp, err := s.c.Do(req.WithContext(ctx))
if err != nil {
return nil, fmt.Errorf("error getting response from %s: %w", req.URL.Redacted(), err)
return nil, fmt.Errorf("error getting response from %s: %w", ru, err)
}
if resp.StatusCode != http.StatusOK {
body, _ := io.ReadAll(resp.Body)
_ = resp.Body.Close()
return nil, fmt.Errorf("unexpected response code %d for %s. Response body %s", resp.StatusCode, req.URL.Redacted(), body)
return nil, fmt.Errorf("unexpected response code %d for %s. Response body %s", resp.StatusCode, ru, body)
}
return resp, nil
}

1
app/vmalert/main.go
View file

@ -101,6 +101,7 @@ func main() {
remoteread.InitSecretFlags()
remotewrite.InitSecretFlags()
datasource.InitSecretFlags()
notifier.InitSecretFlags()
buildinfo.Init()
logger.Init()
pushmetrics.Init()

29
app/vmalert/notifier/alertmanager.go
View file

@ -6,6 +6,7 @@ import (
"fmt"
"io"
"net/http"
"net/url"
"strings"
"time"
@ -17,7 +18,7 @@ import (
// AlertManager represents integration provider with Prometheus alert manager
// https://github.com/prometheus/alertmanager
type AlertManager struct {
addr string
addr *url.URL
argFunc AlertURLGenerator
client *http.Client
timeout time.Duration
@ -48,7 +49,12 @@ func (am *AlertManager) Close() {
}
// Addr returns address where alerts are sent.
func (am AlertManager) Addr() string { return am.addr }
func (am AlertManager) Addr() string {
if *showNotifierURL {
return am.addr.String()
}
return am.addr.Redacted()
}
// Send an alert or resolve message
func (am *AlertManager) Send(ctx context.Context, alerts []Alert, headers map[string]string) error {
@ -64,7 +70,7 @@ func (am *AlertManager) send(ctx context.Context, alerts []Alert, headers map[st
b := &bytes.Buffer{}
writeamRequest(b, alerts, am.argFunc, am.relabelConfigs)
req, err := http.NewRequest(http.MethodPost, am.addr, b)
req, err := http.NewRequest(http.MethodPost, am.addr.String(), b)
if err != nil {
return err
}
@ -91,12 +97,16 @@ func (am *AlertManager) send(ctx context.Context, alerts []Alert, headers map[st
defer func() { _ = resp.Body.Close() }()
amURL := am.addr.Redacted()
if *showNotifierURL {
amURL = am.addr.String()
}
if resp.StatusCode != http.StatusOK {
body, err := io.ReadAll(resp.Body)
if err != nil {
return fmt.Errorf("failed to read response from %q: %w", am.addr, err)
return fmt.Errorf("failed to read response from %q: %w", amURL, err)
}
return fmt.Errorf("invalid SC %d from %q; response body: %s", resp.StatusCode, am.addr, string(body))
return fmt.Errorf("invalid SC %d from %q; response body: %s", resp.StatusCode, amURL, string(body))
}
return nil
}
@ -136,8 +146,15 @@ func NewAlertManager(alertManagerURL string, fn AlertURLGenerator, authCfg proma
return nil, fmt.Errorf("failed to configure auth: %w", err)
}
amURL, err := url.Parse(alertManagerURL)
if err != nil {
return nil, fmt.Errorf("provided incorrect notifier url: %w", err)
}
if !*showNotifierURL {
alertManagerURL = amURL.Redacted()
}
return &AlertManager{
addr: alertManagerURL,
addr: amURL,
argFunc: fn,
authCfg: aCfg,
relabelConfigs: relabelCfg,

9
app/vmalert/notifier/init.go
View file

@ -19,6 +19,8 @@ var (
addrs = flagutil.NewArrayString("notifier.url", "Prometheus Alertmanager URL, e.g. http://127.0.0.1:9093. "+
"List all Alertmanager URLs if it runs in the cluster mode to ensure high availability.")
showNotifierURL = flag.Bool("notifier.showURL", false, "Whether to avoid stripping sensitive information such as passwords from URL in log messages or UI for -notifier.url. "+
"It is hidden by default, since it can contain sensitive info such as auth key")
blackHole = flag.Bool("notifier.blackhole", false, "Whether to blackhole alerting notifications. "+
"Enable this flag if you want vmalert to evaluate alerting rules without sending any notifications to external receivers (eg. alertmanager). "+
"`-notifier.url`, `-notifier.config` and `-notifier.blackhole` are mutually exclusive.")
@ -129,6 +131,13 @@ func Init(gen AlertURLGenerator, extLabels map[string]string, extURL string) (fu
return cw.notifiers, nil
}
// InitSecretFlags must be called after flag.Parse and before any logging
func InitSecretFlags() {
if !*showNotifierURL {
flagutil.RegisterSecretFlag("notifier.url")
}
}
func notifiersFromFlags(gen AlertURLGenerator) ([]Notifier, error) {
var notifiers []Notifier
for i, addr := range *addrs {

20
app/vmalert/utils.go
View file

@ -7,6 +7,7 @@ import (
"strings"
"time"
"github.com/VictoriaMetrics/VictoriaMetrics/app/vmalert/datasource"
"github.com/VictoriaMetrics/VictoriaMetrics/lib/prompbmarshal"
)
@ -80,6 +81,9 @@ func requestToCurl(req *http.Request) string {
schema := req.URL.Scheme
requestURL := req.URL.String()
if !datasource.ShowDatasourceURL() {
requestURL = req.URL.Redacted()
}
if schema == "" {
schema = "http"
if req.TLS != nil {
@ -103,9 +107,25 @@ func requestToCurl(req *http.Request) string {
for _, k := range keys {
cw.add("-H")
if !datasource.ShowDatasourceURL() && isSecreteHeader(k) {
cw.addWithEsc(fmt.Sprintf("%s: <secret>", k))
continue
}
cw.addWithEsc(fmt.Sprintf("%s: %s", k, strings.Join(req.Header[k], " ")))
}
cw.addWithEsc(requestURL)
return cw.string()
}
var secretWords = []string{"auth", "pass", "key", "secret", "token"}
func isSecreteHeader(str string) bool {
s := strings.ToLower(str)
for _, secret := range secretWords {
if strings.Contains(s, secret) {
return true
}
}
return false
}

68
app/vmalert/utils_test.go
View file

@ -7,41 +7,69 @@ import (
func TestRequestToCurl(t *testing.T) {
f := func(req *http.Request, exp string) {
t.Helper()
got := requestToCurl(req)
if got != exp {
t.Fatalf("expected to have %q; got %q instead", exp, got)
}
}
req, _ := http.NewRequest(http.MethodPost, "foo.com", nil)
newReq := func(url string, queryParams ...string) *http.Request {
t.Helper()
r, err := http.NewRequest(http.MethodPost, url, nil)
if err != nil {
t.Fatal(err)
}
params := r.URL.Query()
for i := 0; i < len(queryParams); i += 2 {
params.Add(queryParams[i], queryParams[i+1])
}
r.URL.RawQuery = params.Encode()
return r
}
req := newReq("foo.com")
f(req, "curl -X POST 'http://foo.com'")
req, _ = http.NewRequest(http.MethodGet, "https://foo.com", nil)
f(req, "curl -k -X GET 'https://foo.com'")
req, _ = http.NewRequest(http.MethodPost, "foo.com", nil)
req = newReq("foo.com")
req.Header.Set("foo", "bar")
req.Header.Set("baz", "qux")
f(req, "curl -X POST -H 'Baz: qux' -H 'Foo: bar' 'http://foo.com'")
req, _ = http.NewRequest(http.MethodPost, "foo.com", nil)
params := req.URL.Query()
params.Add("query", "up")
params.Add("step", "10")
req.URL.RawQuery = params.Encode()
req = newReq("foo.com", "query", "up", "step", "10")
f(req, "curl -X POST 'http://foo.com?query=up&step=10'")
req, _ = http.NewRequest(http.MethodPost, "http://foo.com", nil)
params = req.URL.Query()
params.Add("query", "up")
params.Add("step", "10")
req.URL.RawQuery = params.Encode()
req = newReq("http://foo.com", "query", "up", "step", "10")
f(req, "curl -X POST 'http://foo.com?query=up&step=10'")
req, _ = http.NewRequest(http.MethodPost, "https://foo.com", nil)
params = req.URL.Query()
params.Add("query", "up")
params.Add("step", "10")
req.URL.RawQuery = params.Encode()
req = newReq("https://foo.com", "query", "up", "step", "10")
f(req, "curl -k -X POST 'https://foo.com?query=up&step=10'")
req = newReq("https://user:pass@foo.com", "query", "up", "step", "10")
f(req, "curl -k -X POST 'https://user:xxxxx@foo.com?query=up&step=10'")
req = newReq("https://user:pass@foo.com")
req.Header.Set("Authorisation", "Bearer 123456")
f(req, "curl -k -X POST -H 'Authorisation: <secret>' 'https://user:xxxxx@foo.com'")
req = newReq("https://user:pass@foo.com")
req.Header.Set("Authorisation", "Basic 123456")
f(req, "curl -k -X POST -H 'Authorisation: <secret>' 'https://user:xxxxx@foo.com'")
req = newReq("https://foo.com")
req.Header.Set("My-Password", "mypassword")
f(req, "curl -k -X POST -H 'My-Password: <secret>' 'https://foo.com'")
req = newReq("https://foo.com")
req.Header.Set("key-for", "my-new-key")
f(req, "curl -k -X POST -H 'Key-For: <secret>' 'https://foo.com'")
req = newReq("https://foo.com")
req.Header.Set("My-Secret-Org", "secret-organisation")
f(req, "curl -k -X POST -H 'My-Secret-Org: <secret>' 'https://foo.com'")
req = newReq("https://foo.com")
req.Header.Set("Token", "secret-token")
f(req, "curl -k -X POST -H 'Token: <secret>' 'https://foo.com'")
}

2
docs/CHANGELOG.md
View file

@ -33,6 +33,8 @@ The sandbox cluster installation is running under the constant load generated by
* FEATURE: [Alerting rules for VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker#alerts): account for `vmauth` component for alerts `ServiceDown` and `TooManyRestarts`.
* FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add support for functions, labels, values in autocomplete. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3006).
* BUGFIX: [vmalert](https://docs.victoriametrics.com/vmalert.html): strip sensitive information such as auth headers or passwords from datasource, remote-read, remote-write or notifier URLs in log messages or UI. This behavior is by default and is controlled via `-datasource.showURL`, `-remoteRead.showURL`, `remoteWrite.showURL` or `-notifier.showURL` cmd-line flags. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5044).
## [v1.94.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.94.0)
Released at 2023-10-02

27
docs/vmalert.md
View file

@ -831,7 +831,8 @@ and check the `Last updates` section:
Rows in the section represent ordered rule evaluations and their results. The column `curl` contains an example of
HTTP request sent by vmalert to the `-datasource.url` during evaluation. If specific state shows that there were
no samples returned and curl command returns data - then it is very likely there was no data in datasource on the
moment when rule was evaluated.
moment when rule was evaluated. Sensitive info is stripped from the `curl` examples - see [security](#security) section
for more details.
### Debug mode
@ -847,6 +848,8 @@ Just set `debug: true` in rule's configuration and vmalert will start printing a
2022-09-15T13:36:56.153Z DEBUG rule "TestGroup":"Conns" (2601299393013563564) at 2022-09-15T15:36:56+02:00: alert 10705778000901301787 {alertgroup="TestGroup",alertname="Conns",cluster="east-1",instance="localhost:8429",replica="a"} PENDING => FIRING: 1m0s since becoming active at 2022-09-15 15:35:56.126006 +0200 CEST m=+39.384575417
```
Sensitive info is stripped from the `curl` examples - see [security](#security) section for more details.
### Never-firing alerts
vmalert can detect if alert's expression doesn't match any time series in runtime
@ -891,6 +894,20 @@ The same issue can be caused by collision of configured `labels` on [Group](#gro
To fix it one should avoid collisions by carefully picking label overrides in configuration.
## Security
See general recommendations regarding security [here](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#security).
vmalert [web UI](#web) exposes configuration details such as list of [Groups](#groups), active alerts,
[alerts state](#alerts-state), [notifiers](#notifier-configuration-file). Notifier addresses (sanitized) are attached
as labels to metrics `vmalert_alerts_sent_.*` on `http://<vmalert>/metrics` page. Consider limiting user's access
to the web UI or `/metrics` page if this information is sensitive.
[Alerts state](#alerts-state) page or [debug mode](#debug-mode) could emit additional information about configured
datasource URL, GET params and headers. Sensitive information such as passwords or auth tokens is stripped by default.
To disable stripping of such info pass `-datasource.showURL` cmd-line flag to vmalert.
## Profiling
`vmalert` provides handlers for collecting the following [Go profiles](https://blog.golang.org/profiling-go-programs):
@ -973,7 +990,8 @@ The shortlist of configuration flags is the following:
-datasource.roundDigits int
Adds "round_digits" GET param to datasource requests. In VM "round_digits" limits the number of digits after the decimal point in response values.
-datasource.showURL
Whether to show -datasource.url in the exported metrics. It is hidden by default, since it can contain sensitive info such as auth key
Whether to avoid stripping sensitive information such as auth headers or passwords from URLs in log messages or UI and exported metrics.
It is hidden by default, since it can contain sensitive info such as auth key.
-datasource.tlsCAFile string
Optional path to TLS CA file to use for verifying connections to -datasource.url. By default, system CA is used
-datasource.tlsCertFile string
@ -993,7 +1011,7 @@ The shortlist of configuration flags is the following:
-disableAlertgroupLabel
Whether to disable adding group's Name as label to generated alerts and time series.
-dryRun
Whether to check only config files without running vmalert. The rules file are validated. The -rule flag must be specified.
Whether to check only config files without running vmalert. The rules file are validated. The -rule flag must be specified.
-enableTCP6
Whether to enable IPv6 for listening and dialing. By default, only IPv4 TCP and UDP are used
-envflag.enable
@ -1122,6 +1140,9 @@ The shortlist of configuration flags is the following:
-notifier.url array
Prometheus Alertmanager URL, e.g. http://127.0.0.1:9093. List all Alertmanager URLs if it runs in the cluster mode to ensure high availability.
Supports an array of values separated by comma or specified via multiple flags.
-notifier.showURL bool
Whether to avoid stripping sensitive information such as passwords from URLs in log messages or UI for -notifier.url.
It is hidden by default, since it can contain sensitive info such as auth key.
-notifier.blackhole bool
Whether to blackhole alerting notifications. Enable this flag if you want vmalert to evaluate alerting rules without sending any notifications to external receivers (eg. alertmanager). `-notifier.url`, `-notifier.config` and `-notifier.blackhole` are mutually exclusive.
-pprofAuthKey string