mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-11-21 14:44:00 +00:00

lib/promscrape/discovery/kubernetes: fix watcher start order for roles endpoints and endpointslice (#5557 )

* lib/promscrape/discovery/kubernetes: fix watcher start order for roles endpoints and endpointslice

Previously the groupWatcher could be mistakenly stopped when requests for pod or services resources take too long.

* remove mislead comment

* docs/sd_configs.md: mention -promscrape.kubernetes.attachNodeMetadataAll flag in the description for attach_metadata section

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4640

* wip

* lib/promscrape/kubernetes: prevent from stopping groupWatcher when there are in-flight apiWatcher.mustStart() calls

groupWatcher is stopped if it has zero registered apiWatchers during 14 seconds.
But such a groupWatcher can be still in use if apiWatcher for `role: endpoints` or `role: endpointslice`
is being registered and the discovery of the associated `pod` and/or `service` objects takes longer
than 14 seconds - see the beginning of groupWatcher.startWatchersForRole() function for details.

Track the number of in-flight calls to apiWatcher.mustStart() and prevent from stopping the associated groupWatcher
if the number of in-flight calls is non-zero.

P.S. postponing the discovery of `pod` and/or `service` objects associated with `endpoints` or `endpointslice` roles
isn't the best solution, since it slows down initial discovery of `endpoints` and `endpointslice` targets.

* typo fix

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>

2024-01-21 23:13:15 +02:00

41 KiB

Raw Blame History

sort

weight

title

aliases

100

CHANGELOG

docs

parent	weight
victoriametrics	100

/CHANGELOG.html

CHANGELOG

The following tip changes can be tested by building VictoriaMetrics components from the latest commits according to the following docs:

Metrics of the latest version of VictoriaMetrics cluster are available for viewing at our sandbox. The sandbox cluster installation is running under the constant load generated by prometheus-benchmark and used for testing latest releases.

tip

SECURITY: upgrade Go builder from Go1.21.5 to Go1.21.6. See the list of issues addressed in Go1.21.6.
FEATURE: vmagent: add support for discovering Hetzner Cloud and Hetzner Robot scrape targets. See this feature request and these docs.
FEATURE: graphite: add support for negative index in groupByNode and aliasByNode functions. Thanks to @rbizos for the pull request.
FEATURE: vmagent: add support for DataDog v2 data ingestion protocol. See these docs and this feature request.
FEATURE: vmagent: expose ability to set OAuth2 endpoint parameters per each -remoteWrite.url via the command-line flag -remoteWrite.oauth2.endpointParams. See these docs. Thanks to @mhill-holoplot for the pull request.
FEATURE: vmagent: add ability to set attach_metadata.node=true option for all the kubernetes_sd_configs defined at -promscrape.config via -promscrape.kubernetes.attachNodeMetadataAll command-line flag. See this feature request. Thanks to @wasim-nihal for the initial implementation.
FEATURE: vmalert: expose ability to set OAuth2 endpoint parameters via the following command-line flags:
- -datasource.oauth2.endpointParams for -datasource.url
- -notifier.oauth2.endpointParams for -notifier.url
- -remoteRead.oauth2.endpointParams for -remoteRead.url
- -remoteWrite.oauth2.endpointParams for -remoteWrite.url
FEATURE: vmauth: add ability to proxy incoming requests to different backends based on the requested host via src_hosts option at url_map. See these docs.
FEATURE: vmauth: expose vmauth_user_request_backend_errors_total and vmauth_unauthorized_user_request_backend_errors_total metrics, which track the number of failed requests because of backend errors. See this feature request.
FEATURE: vmauth: add an ability to specify additional labels for per-user metrics via metric_labels section. See this feature request.
FEATURE: all VictoriaMetrics components: break HTTP client connection if an error occurs after the server at -httpListenAddr already sent response status code. Previously such an error couldn't be detected at client side. Now the client will get an error about invalid chunked response. The error message is simultaneously written to the server log and in the last line of the response. This should help detecting errors when migrating data between VictoriaMetrics instances by vmctl. See this issue.
FEATURE: all VictoriaMetrics components: add ability to specify arbitrary HTTP headers to send with every request to -pushmetrics.url. See push metrics docs.
FEATURE: all VictoriaMetrics components: add -metrics.exposeMetadata command-line flag, which allows displaying TYPE and HELP metadata at /metrics page exposed at -httpListenAddr. This may be needed when the /metrics page is scraped by collector, which requires the TYPE and HELP metadata such as Google Cloud Managed Prometheus.
FEATURE: all VictoriaMetrics components: add ability to dynamically re-read auth keys and passwords from files and urls when using file:///path/to/file or http://host/path syntax for the following command-line flags: -configAuthKey, -deleteAuthKey, -flagsAuthKey, -forceMergeAuthKey, -forceFlushAuthKey, -httpAuth.password, -metricsAuthKey, -pprofAuthKey, -reloadAuthKey, -search.resetCacheAuthKey, -snapshotAuthKey. For example, -httpAuth.password=file:///path/to/password. See these docs for details.
FEATURE: dashboards/cluster: add panels for detailed visualization of traffic usage between vmstorage, vminsert, vmselect components and their clients. New panels are available in the rows dedicated to specific components.
FEATURE: dashboards/cluster: update "Slow Queries" panel to show percentage of the slow queries to the total number of read queries served by vmselect. The percentage value should make it more clear for users whether there is a service degradation.
FEATURE: dashboards/single: change dashboard title from VictoriaMetrics to VictoriaMetrics - single-node. The new title should provide better understanding of this dashboard purpose.
FEATURE: vmctl: rename cmd-line flag vm-native-disable-retries to vm-native-disable-per-metric-migration to better reflect its meaning.
FEATURE: vmctl: add -vm-native-src-insecure-skip-verify and -vm-native-dst-insecure-skip-verify command-line flags for native protocol. It can be used for skipping TLS certificate verification when connecting to the source or destination addresses.
FEATURE: Alerting rules for VictoriaMetrics: add job label to DiskRunsOutOfSpace alerting rule, so it is easier to understand to which installation the triggered instance belongs.
BUGFIX: properly return errors from export APIs. Previously these errors were silently suppressed. See this pull request.
BUGFIX: VictoriaMetrics cluster: properly return full results when -search.skipSlowReplicas command-line flag is passed to vmselect and when vmstorage groups are in use. Previously partial results could be returned in this case.
BUGFIX: vminsert: properly accept samples via OpenTelemetry data ingestion protocol when these samples have no resource attributes. Previously such samples were silently skipped.
BUGFIX: vmstorage: added missing -inmemoryDataFlushInterval command-line flag, which was missing in VictoriaMetrics cluster after implementing this feature in v1.85.0.
BUGFIX: vmstorage: properly expire storage/prefetchedMetricIDs cache. Previously this cache was never expired, so it could grow big under high churn rate. This could result in increasing CPU load over time.
BUGFIX: vmalert: check -external.url schema when starting vmalert, must be http or https. Before, alertmanager could reject alert notifications if -external.url contained no or wrong schema.
BUGFIX: vmalert: automatically add exported_ prefix for original evaluation result label if it's conflicted with external or reserved one, previously it was overridden. See this issue.
BUGFIX: MetricsQL: consistently sort results for q1 or q2 query, so they do not change colors with each refresh in Grafana. See this issue.
BUGFIX: MetricsQL: properly return results from bottomk and bottomk_*() functions when some of these results contain NaN values. See this issue. Thanks to @xiaozongyang for the fix.
BUGFIX: MetricsQL: properly handle queries, which wrap rollup functions with multiple arguments without explicitly specified lookbehind window in square brackets into aggregate functions. For example, sum(quantile_over_time(0.5, process_resident_memory_bytes)) was resulting to expecting at least 2 args to ...; got 1 args error. Thanks to @atykhyy for the pull request.
BUGFIX: vmctl: retry on import errors in vm-native mode. Before, retries happened only on writes into a network connection between source and destination. But errors returned by server after all the data was transmitted were logged, but not retried.
BUGFIX: vmagent: properly assume role with AWS IRSA authorization. Previously role chaining was not supported. See this issue for details.
BUGFIX: vmagent: exit if there is config syntax error in scrape_config_files when -promscrape.config.strictParse=true. See this issue.
BUGFIX: vmagent: properly discover targets for role: endpoints and role: endpointslice in kubernetes_sd_configs. Previously some endpoints and endpointslice targets could be left undiscovered or some targets could have missing __meta_* labels when performing service discovery in busy Kubernetes clusters with large number of pods. See this pull request.
BUGFIX: vmagent: do not store scrape response for target in memory when staleness markers are disabled. See this pull request for details.
BUGFIX: vmui: fix a link for the statistic inaccuracy explanation in the cardinality explorer tool. See this issue.
BUGFIX: vmui: send step param for instant queries. The change reverts this issue due to reasons explained in this comment.
BUGFIX: all: fix potential panic during components shutdown when metrics push is configured. See this issue. Thanks to @zhdd99 for the pull request.
BUGFIX: vmselect: properly determine time range search for instant queries with too big look-behind window like foo[100y]. Previously, such queries could return empty responses even if foo is present in database.
BUGFIX: MetricsQL: properly handle possible negative results caused by float operations precision error in rollup functions like rate() or increase(). See this issue.

v1.96.0

See changes here

v1.95.1

See changes here

v1.95.0

See changes here

v1.94.0

See changes here

v1.93.10

Released at 2024-01-17

v1.93.x is a line of LTS releases (e.g. long-time support). It contains important up-to-date bugfixes. The v1.93.x line will be supported for at least 12 months since v1.93.0 release

SECURITY: upgrade Go builder from Go1.21.5 to Go1.21.6. See the list of issues addressed in Go1.21.6.
BUGFIX: vminsert: properly accept samples via OpenTelemetry data ingestion protocol when these samples have no resource attributes. Previously such samples were silently skipped.
BUGFIX: vmstorage: added missing -inmemoryDataFlushInterval command-line flag, which was missing in VictoriaMetrics cluster after implementing this feature in v1.85.0.
BUGFIX: vmstorage: properly expire storage/prefetchedMetricIDs cache. Previously this cache was never expired, so it could grow big under high churn rate. This could result in increasing CPU load over time.
BUGFIX: vmalert: check -external.url schema when starting vmalert, must be http or https. Before, alertmanager could reject alert notifications if -external.url contained no or wrong schema.
BUGFIX: MetricsQL: properly return results from bottomk and bottomk_*() functions when some of these results contain NaN values. See this issue. Thanks to @xiaozongyang for the fix.
BUGFIX: MetricsQL: properly handle queries, which wrap rollup functions with multiple arguments without explicitly specified lookbehind window in square brackets into aggregate functions. For example, sum(quantile_over_time(0.5, process_resident_memory_bytes)) was resulting to expecting at least 2 args to ...; got 1 args error. Thanks to @atykhyy for the pull request.
BUGFIX: vmagent: properly assume role with AWS IRSA authorization. Previously role chaining was not supported. See this issue for details.
BUGFIX: all: fix potential panic during components shutdown when metrics push is configured. See this issue. Thanks to @zhdd99 for the pull request.
BUGFIX: vmctl: check for Error field in response from influx client during migration. Before, only network errors were checked. Thanks to @wozz for the pull request.
BUGFIX: vmctl: retry on import errors in vm-native mode. Before, retries happened only on writes into a network connection between source and destination. But errors returned by server after all the data was transmitted were logged, but not retried.

SECURITY: upgrade Go builder from Go1.21.5 to Go1.21.6. See the list of issues addressed in Go1.21.6.
BUGFIX: vmstorage: added missing -inmemoryDataFlushInterval command-line flag, which was missing in VictoriaMetrics cluster after implementing this feature in v1.85.0.
BUGFIX: MetricsQL: properly handle queries, which wrap rollup functions with multiple arguments without explicitly specified lookbehind window in square brackets into aggregate functions. For example, sum(quantile_over_time(0.5, process_resident_memory_bytes)) was resulting to expecting at least 2 args to ...; got 1 args error. Thanks to @atykhyy for the pull request.
BUGFIX: vmstorage: properly expire storage/prefetchedMetricIDs cache. Previously this cache was never expired, so it could grow big under high churn rate. This could result in increasing CPU load over time.
BUGFIX: MetricsQL: properly return results from bottomk and bottomk_... functions when some of these results contain NaN values. See this issue. Thanks to @xiaozongyang for the fix.
BUGFIX: all: fix potential panic during components shutdown when metrics push is configured. See this issue. Thanks to @zhdd99 for the pull request.