From 79ee1749a19d64da3e42fb4bd611adafcb4c1fc8 Mon Sep 17 00:00:00 2001 From: Zakhar Bessarab Date: Tue, 18 Apr 2023 16:47:26 +0400 Subject: [PATCH] lib/httpserver: add handler to serve `/robots.txt` and deny search indexing (#4143) This handler will instruct search engines that indexing is not allowed for the content exposed to the internet. This should help to address issues like #4128 when instances are exposed to the internet without authentication. --- docs/CHANGELOG.md | 2 ++ lib/httpserver/httpserver.go | 5 +++++ 2 files changed, 7 insertions(+) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index c05e4d1c5..156af5d32 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -16,6 +16,8 @@ The following tip changes can be tested by building VictoriaMetrics components f ## tip +* SECURITY: serve `/robots.txt` content to disallow indexing of the exposed instances by search engines. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4128) for details. + * FEATURE: open source [Graphite Render API](https://docs.victoriametrics.com/#graphite-render-api-usage). This API allows using VictoriaMetrics as a drop-in replacement for Graphite at both data ingestion and querying sides and reducing infrastructure costs by up to 10x comparing to Graphite. See [this case study](https://docs.victoriametrics.com/CaseStudies.html#grammarly) as an example. * FEATURE: [vmbackup](https://docs.victoriametrics.com/vmbackup.html): store backup creation and completion time in `backup_complete.ignore` file of backup contents. This is useful to determine point in time when backup was created and completed. * FEATURE: [vmbackupmanager](https://docs.victoriametrics.com/vmbackupmanager.html): add `created_at` field to the output of `/api/v1/backups` API and `vmbackupmanager backup list` command. See this [doc](https://docs.victoriametrics.com/vmbackupmanager.html#api-methods) for data format details. diff --git a/lib/httpserver/httpserver.go b/lib/httpserver/httpserver.go index 9171c7ddc..a8335c3b8 100644 --- a/lib/httpserver/httpserver.go +++ b/lib/httpserver/httpserver.go @@ -323,6 +323,11 @@ func handlerWrapper(s *server, w http.ResponseWriter, r *http.Request, rh Reques // See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1833 fmt.Fprintf(w, "VictoriaMetrics is Ready.\n") return + case "/robots.txt": + // This prevents search engines from indexing contents + // See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4128 + fmt.Fprintf(w, "User-agent: *\nDisallow: /\n") + return default: if strings.HasPrefix(r.URL.Path, "/debug/pprof/") { pprofRequests.Inc()