Commit graph

2951 commits

Author SHA1 Message Date
Aliaksandr Valialkin
a5a57a7c2a
docs/CHANGELOG.md: group changelog lines for tip release according to VictoriaMetrics apps 2023-05-09 22:55:17 -07:00
Aliaksandr Valialkin
b20bbb8f79
docs/CHANGELOG.md: document baf456978d
See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4223
2023-05-09 22:53:36 -07:00
Aliaksandr Valialkin
dbdf029f6b
docs/managed-victoriametrics/overview.md: typo fix after 51fbf58d89 2023-05-09 22:53:05 -07:00
Aliaksandr Valialkin
7adc1665ba
docs/CHANGELOG.md: refer to the author and the pull request of the notifier_headers feature at vmalert
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3260
2023-05-09 22:52:34 -07:00
Aliaksandr Valialkin
24aa295f43
docs/CHANGELOG.md: document 03150c8973
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4204
2023-05-09 22:50:10 -07:00
Aliaksandr Valialkin
493d8ec7e0
docs/vmalert.md: clarify docs regarding the support of recursive globs
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4041
2023-05-09 22:49:37 -07:00
Aliaksandr Valialkin
e2358d3bd5
docs: clarify docs after 5ee344824f
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4183
2023-05-09 22:49:13 -07:00
Aliaksandr Valialkin
079875a127
app/vmagent/remotewrite: make more user-friendly the warning message about too small -remoteWrite.maxdiskUsagePerURL value
This is a follow-up for bc17f4828c .
While at it, document the change at docs/CHANGELOG.md .

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4195
2023-05-09 22:48:40 -07:00
Aliaksandr Valialkin
f35181a4f8
docs/managed-victoriametrics: consistently use setup-notifications prefix for images used at docs/managed-victoriametrics/setup-notifications.md
This should simplify managment of the images belonging to the given docs.
See docs/assets/README.md for details

This is a follow-up for 4052c44ac1
2023-05-09 22:47:59 -07:00
Aliaksandr Valialkin
22aeeeef3e
app/vmauth: merge default_url example into multi-url example in order to reduce the amounts of text to read for the user
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4084

This is a follow-up for 041e188df8
2023-05-09 22:47:11 -07:00
Aliaksandr Valialkin
fbc28810b1
app/vmselect: small cleanup after 68e31a6000
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3811
2023-05-09 22:43:59 -07:00
Aliaksandr Valialkin
5dbaffe2c6
app/{vmselect,vmctl}: move ParseTime() to lib/promutils
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4091

This is a follow-up for e2053baf32
2023-05-09 22:42:35 -07:00
Aliaksandr Valialkin
3a23d90c3a
docs/CHANGELOG.md: typo fix after 45a551df9c: 'this doc' -> 'this feature request' 2023-05-09 22:42:07 -07:00
Aliaksandr Valialkin
143c5bc936
app/vmauth/README.md: mention about ip filters and concurrency limiter at Security chapter 2023-05-09 22:40:46 -07:00
Aliaksandr Valialkin
fe58dd7e4c
app/vmauth: refer ip_filters option in example auth config
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3491
2023-05-09 22:40:15 -07:00
Aliaksandr Valialkin
e55ddaa962
docs: document IP filters functionality in vmauth
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3491

This is a follow-up for 2f08ed3be2
2023-05-09 22:39:47 -07:00
Aliaksandr Valialkin
8cca9ed0d9
all: update Go builder from Go1.20.3 to Go1.20.4
See https://github.com/golang/go/issues?q=milestone%3AGo1.20.4+label%3ACherryPickApproved
2023-05-09 22:31:08 -07:00
Aliaksandr Valialkin
cb72449fc4
docs/CHANGELOG.md: document c77385e78f 2023-05-09 22:29:23 -07:00
Alexander Marshalov
de68e94c91
fixed vm_promscrape_config_last_reload_successful metric value recovery after successful reloading with unchanged content (#4260) (#4268)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-05-09 22:17:27 -07:00
Roman Khavronenko
54bfacdfde
vmalert: follow-up after cae87da (#4269)
* vmalert: follow-up after cae87da

cae87da4bb
Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: update struct comments

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: rm typo

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-05-09 22:07:26 -07:00
Alexander Marshalov
65f4590403
update generated docs for operator (#4267)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-05-09 22:06:28 -07:00
Roman Khavronenko
4edb97f4da
app/vmalert: detect alerting rules which don't match any series at all (#4198)
app/vmalert: detect alerting rules which don't match any series at all

vmalert starts to understand /query responses which contain object:
```
"stats":{"seriesFetched": "42"}
```
If object is present, vmalert parses it and populates a new field
`SeriesFetched`. This field is then used to populate the new metric
`vmalert_alerting_rules_last_evaluation_series_fetched` and to
display warnings in the vmalert's UI.

If response doesn't contain the new object (Prometheus or
VictoriaMetrics earlier than v1.90), then `SeriesFetched=nil`.
In this case, UI will contain no additional warnings.
And `vmalert_alerting_rules_last_evaluation_series_fetched` will
be set to `-1`. Negative value of the metric will help to compile
correct alerting rule in follow-up.

Thanks for the initial implementation to @Haleygo
See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4056

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4039

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-05-09 21:48:59 -07:00
Roman Khavronenko
3049754575
alerts: update TooHighMemoryUsage threshold (#4256)
It appears that 90% usage for anonymous mem usage
is already concerning. So we lowering the threshold to 80%.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-05-09 21:41:40 -07:00
Zakhar Bessarab
370a421ef4
lib/promscrape/discovery/kubernetes: follow-up for d5e94721db (#4255)
- add changelog reference to an author
- fix tests
- add metadata to match Prometheus behavior

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-05-09 21:29:27 -07:00
Zakhar Bessarab
348693ff84
lib/storage: fix indexdb rotation infinite loop (#4249)
When using `retentionTimezoneOffset` and having local timezone being more than 4 hours different from UTC indexdb retention calculation could return negative value. This caused indexdb rotation to get in loop.
Fix calculation of offset to use `retentionTimezoneOffset` value properly and add test to cover all legit timezone configs.
See:
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4207
- https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4206

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2023-05-09 21:23:01 -07:00
Alexander Marshalov
26fc4afff8
added new consulagent service discovery (#3953) (#4217) 2023-05-08 23:43:59 -07:00
Alexander Marshalov
f7dd084890
max value for memory.allowedPercent changed from 200 to 100 (#4171) (#4251)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-05-08 23:20:56 -07:00
Zakhar Bessarab
98e4265235
docs: changelog follow-up for 49b77ec01a (#4250)
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-05-08 23:20:09 -07:00
Nikolay
7bfa1d7d9e
lib/backup: fixes path generation for windows (#4133)
replaces custom fsync function with standard Fsync methods for files.
fixes pattern matching for parts and properly generate backup path for local fs.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70
2023-05-08 23:16:26 -07:00
Nikolay
5d0299ac19
lib/fs: do not panic at windows at dir deletion (#4132)
Windows doesn't allow to remove dir with opened files. Usually it's a case for snapshots, hard cannot be removed if file is openned.
With this change, dir will be renamed and properly deleted at the next process start.
It's recommended to restart vmstorage/vmsingle for snapshots deletion completion periodically.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70
2023-05-08 23:11:55 -07:00
Zakhar Bessarab
1b06af321f
lib/promscrape/discovery/kubernetes: add common labels to all ports discovered from endpoints (#4235)
* lib/promscrape/discovery/kubernetes: add common labels to all ports discovered from endpoints

Sets
`__meta_kubernetes_endpoints_name` and `__meta_kubernetes_namespace` labels to all ports of pod.
Prometheus sets those labels to all ports in pod (0ab9553611/discovery/kubernetes/endpoints.go (L267C15-L269)) even if port is not matching any service.

See: #4154

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/promscrape/discovery/kubernetes: fix test for updated discovery logic

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-05-08 22:15:37 -07:00
Artem Navoiev
f168da9524
remove information of releasing graphite render api from tip section as we released it in 1.90
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-05-08 22:01:13 -07:00
Artem Navoiev
46886260be
prepare operator docs to migration
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-05-08 21:59:48 -07:00
Artem Navoiev
1e0dcfaae6
update managed docs - prepare for migration
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-05-08 21:58:53 -07:00
Roman Khavronenko
f247160699
docs: note automatic conversion to ms for influx protocol (#4224)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-05-08 21:55:58 -07:00
Artem Navoiev
1e15eea76a
add wight do trobubleshooting docs
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-05-08 21:54:48 -07:00
Artem Navoiev
283a75afba
prepare static docs to migration
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-05-08 21:54:20 -07:00
Roman Khavronenko
8f1372bd43
vmalert: fix API to return non-nil values (#4222)
Properly return empty slices instead of nil for `/api/v1/rules` and `/api/v1/alerts` API handlers.
This improves compatibility with Grafana.

 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4221

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-05-08 21:47:51 -07:00
Roman Khavronenko
c6511bc2d0
Revert "http server: limit max concurrent requests (#4185)" (#4215)
This reverts commit 77f76371

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-05-08 17:22:27 -07:00
Roman Khavronenko
1c3bf0d0d8
app/vmalert: follow-up after 6c322b4a00 (#4214)
6c322b4a00

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-05-08 17:20:49 -07:00
Zakhar Bessarab
19eaf17e11
app/vmalert: add support of recursive path globs for rules and templates (#4148)
Supports using `**` for `-rule` and `-rule.templates`: `dir/**/*.tpl` loads contents of dir and all subdirectories recursively.

See: #4041

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Artem Navoiev <tenmozes@gmail.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2023-05-08 16:22:30 -07:00
Nikolay
cfa058dfec
lib/promscrape: adds filter for consul_sd_configs: (#4184)
* lib/promscrape: adds filter for consul_sd_configs:
it allows advanced filtering for consul service discovery requests
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4183

* typo fix

* removes deprecation mentions since it's not relevant

* Update docs/CHANGELOG.md

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

---------

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-05-08 16:14:15 -07:00
Zakhar Bessarab
55d772ab39
app/vmalert: return an error when using query function in -external.alert.source flag (#4191)
Templating of `-external.alert.source` is not expected to have access to the query which was causing runtime error when query function was passed as nil.
See: #4181

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-05-08 15:48:16 -07:00
Dmytro Kozlov
f425123116
app/vmagent,lib/persistentqueue: show warning message if --remoteWrite.maxDiskUsagePerURL flag lower than 500MB (#4196)
* app/vmagent,lib/persistentqueue: show warning message if `--remoteWrite.maxDiskUsagePerURL` flag lower than 500MB

* app/vmagent,lib/persistentqueue: linter fix

* app/vmagent,lib/persistentqueue: fix comment
2023-05-08 15:45:21 -07:00
dmitryk-dk
f1717aa818
docs: fix type 2023-05-08 15:33:21 -07:00
dmitryk-dk
6168e7f95f
app/vmctl: fix comments 2023-05-08 15:32:24 -07:00
dmitryk-dk
aef925ea8a
app/vmctl: fix comments 2023-05-08 15:31:45 -07:00
dmitryk-dk
c2b58fd954
docs/managed-victoriametrics: change emails 2023-05-08 15:31:01 -07:00
dmitryk-dk
dbee230911
docs/managed-victoriametrics: add notifications setup 2023-05-08 15:30:21 -07:00
Alexander Marshalov
402d906d40
added default_url field in vmauth users config (#4084) (#4156)
* added default url field in vmauth users config (#4084)

---------

Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-05-08 15:13:46 -07:00
Yury Molodov
ddc5197bce
vmui: add metric relabel debug (#3889)
* feat: add metric relabel debug (#3807)

* fix: add link to relabeling cookbook

* lib/promrelabel: merge, fix conflicts

* lib/promrelabel: fix diff

* docs/vmui: add metric relabel playground

---------

Co-authored-by: dmitryk-dk <kozlovdmitriyy@gmail.com>
2023-05-08 14:59:35 -07:00
Alexander Marshalov
42b1e6443b
changelog for issue #4083 (#4197)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-05-08 14:41:39 -07:00
Yury Molodov
e0434f7b4e
vmui: display heatmap in the Explore Metrics (#4124)
* feat: display heatmap in the explore metrics (#4111)

* fix: correct calc step for heatmap

* fix: remove spaces in the result of getDurationFromMilliseconds
2023-05-08 14:40:38 -07:00
Yury Molodov
3c45256736
vmui: add a comparison of data to the Cardinality Explorer (#4123)
* feat: add button "show today" to date picker

* feat: add comparison with the prev day (#3967)

* vmui/docs: add comparison of data to cardinality page
2023-05-08 14:39:14 -07:00
Yury Molodov
382a2ff649
docs/vmui: fix CHANGELOG.md about WITH templates (#4194) 2023-05-08 14:37:08 -07:00
Yury Molodov
de35cbf251
vmui: Integrate WITH template playground (#3831)
* feat: add WithTemplate page

* app/vmselect/prometheus: enable json mode for expand with expr API

* app/vmselect/prometheus: enable CORS and add content type

* feat: add api for expand with templates

* fix: remove console from useExpandWithExprs

* app/vmselect/prometheus: fix escaping

* vmui:  integrate WITH template

* app/vmctl: check content type instead of form param

* fix: add content-type for fetch with-exprs

* fix: add a header to the server's response that allows the "Content-Type" header

* app/vmctl: added comment and cleanup

* app/vmctl: use format query param

---------

Co-authored-by: dmitryk-dk <kozlovdmitriyy@gmail.com>
2023-05-08 14:35:35 -07:00
Dmytro Kozlov
3da856af0c
app/vmctl: add support for the different time format in the native binary protocol (#4189)
* app/vmctl: add support for the different time format in the native binary protocol

* app/vmctl: update flag description, update CHANGELOG.md

* app/vmctl: add comment to exported function
2023-05-08 14:21:59 -07:00
Alexander Marshalov
574a0559d5
added unauthorized_user field in vmauth users config (#4083) (#4157)
added `unauthorized_user` field in vmauth users config (#4083)

---------

Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-05-08 13:36:54 -07:00
Roman Khavronenko
20b025dc88
http server: limit max concurrent requests (#4185)
* lib/httpserver: introduce `-http.maxConcurrentRequests` command-line flag

Introduce `-http.maxConcurrentRequests` command-line flag to protect
VM components from resource exhaustion during unexpected spikes of HTTP requests.
By default, the new flag's value is set to 0 which means no limits are applied.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* lib/httpserver: mention http.maxConcurrentRequests in docs

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-05-08 13:13:58 -07:00
Alexander Marshalov
7faf45d7f9
doc improvements (#4172) (#4186)
- added info about metric `vm_vminsert_metrics_read_total`,
- small doc refactoring
- and added make-command for running docs in docker.

Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-05-08 13:11:57 -07:00
Yury Molodov
23ed0467a2
vmui: fix freeze when query regular with heatmap query (#4093)
* fix: fix freeze when query regular with heatmap query

* vmui/docs: fix freeze when query regular with heatmap query
2023-05-08 13:09:49 -07:00
Aliaksandr Valialkin
45ec3ac158
docs: document IP filters functionality in vmauth
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3491

This is a follow-up for 2f08ed3be2
2023-05-08 12:14:05 -07:00
Yury Molodov
46765c677d
vmui: fix bug where tenant list was not displayed (#4162)
* fix: modify the condition for querying tenants

* fix: change getTenantIdFromUrl output to string
2023-05-08 11:00:19 -07:00
Artem Navoiev
624e13a1d6
move note about opensource of graphite in v1.90 release note
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-05-08 09:56:53 -07:00
Roman Khavronenko
e9ce67adb8
vmalert: retry datasource requests with EOF or unexpected EOF errors (#4146)
* vmalert: retry datasource requests with EOF or unexpected EOF errors

Retry failed read request on the closed connection one more time.
This may improve rules execution reliability when connection
between vmalert and datasource closes unexpectedly.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: fix old tests

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-05-08 09:49:49 -07:00
Zakhar Bessarab
79ee1749a1
lib/httpserver: add handler to serve /robots.txt and deny search indexing (#4143)
This handler will instruct search engines that indexing is not allowed for the content exposed to the internet. This should help to address issues like #4128 when instances are exposed to the internet without authentication.
2023-05-08 09:46:34 -07:00
Balamurugan Krishnamoorthy (Bala)
2a45a85c35
Removed duplicate third-party article reference (#4142)
"How do We Keep Metrics for a Long Time in VictoriaMetrics" article is referenced twice in "Third-party articles and slides about VictoriaMetrics" section
2023-05-08 09:44:19 -07:00
Artem Navoiev
dcac82638e
add graphite render api opensource to changelog
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-05-08 09:42:38 -07:00
Aliaksandr Valialkin
cd99f1fb3a
docs/CHANGELOG.md: document c77385e78f 2023-05-08 08:57:49 -07:00
Aliaksandr Valialkin
d544bfd73e
vendor: update github.com/VictoriaMetrics/metricsql from v0.56.1 to v0.56.2
This fixes panic when the duration in the query contains `M` suffix.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4120
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3589
2023-04-14 14:06:47 -07:00
Aliaksandr Valialkin
2fd9fdacce
docs/vmctl.md: run make docs-sync after 2a5b9ff782 2023-04-13 23:54:50 -07:00
Aliaksandr Valialkin
e6d10106f7
docs/CHANGELOG.md: move the bugfix description into the correct place
This is a follow-up for 2a5b9ff782

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4092
2023-04-13 23:52:01 -07:00
Dmytro Kozlov
10351a9dbb
app/vmctl: fix performance degradation, add flag to disable backoff policy (#4097)
* app/vmctl: change api for getting metric names

* app/vmctl: fix tests

* app/vmctl: add flag to enable backoff policy, fix test, performance improvements

* app/vmctl: use one http client

* app/vmctl: made linter happy

* app/vmctl: updated documentation and CHANGELOG.md

* app/vmctl: cleanup

* app/vmctl: rename flag

* app/vmctl: cleanup

* app/vmctl: fix comments

* app/vmctl: fix metrics parser problem, improve tests
2023-04-13 23:51:11 -07:00
Aliaksandr Valialkin
7fb2b14ca0
docs/CHANGELOG.md: run at least 4 background mergers on systems with less than 4 CPU cores
This reduces the probability of sudden spike in the number of small parts when all the background mergers
are busy with big merges.
2023-04-13 23:37:05 -07:00
Aliaksandr Valialkin
cf53ce83a0
app/vmstorage: deprecate -bigMergeConcurrency command-line flag
Improperly configured -bigMergeConcurrency command-line flag usually leads to uncontrolled
growth of unmerged parts, which, in turn, increases CPU usage and query durations.

So it is better deprecating this flag. In rare cases -smallMergeConcurrency command-line flag
can be used instead for controlling the concurrency of background merges.
2023-04-13 20:42:22 -07:00
Aliaksandr Valialkin
e95e401e4d
app/vmbackupmanager: sync with enterprise-single-node branch after 41a54c775891c87e3d5ed59ff0769c869dd2fe71 2023-04-13 19:38:28 -07:00
Alexander Marshalov
e0a1b62569
Added extra information to docs about total output in stream aggregation. (#4130)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-04-13 19:37:05 -07:00
Zakhar Bessarab
217eea6e15
lib/backup/actions: store metadata(creation and completion time) in backup files (#4117)
This makes it easier to understand exact point in time which is included in this backup.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-04-13 19:20:34 -07:00
Aliaksandr Valialkin
b33dd2e577
docs/Single-server-VictoriaMetrics.md: document automatic switch from graph view to heatmap view for histogram buckets
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3384
2023-04-10 10:52:24 -07:00
Aliaksandr Valialkin
6c7087f65e
docs/Troubleshooting.md: mention that it is recommended to use default command-line flag values for VictoriaMetrics components 2023-04-10 10:52:10 -07:00
Aliaksandr Valialkin
b21ac56894
docs/guides: update VictoriaMetrics from v1.89.1 to v1.90.0 2023-04-06 19:18:14 -07:00
Aliaksandr Valialkin
2bbd8754bb
docs/CHANGELOG.md: formatting fix 2023-04-06 19:18:14 -07:00
Aliaksandr Valialkin
d15618f2ef
docs/CHANGELOG.md: cut v1.90.0 release 2023-04-06 16:16:52 -07:00
Dmytro Kozlov
8ec5e7f53a
app/vmctl: add multiple filters defined in --vm-native-filter-match flag to discovered metric names (#4063)
* app/vmctl: add multiple filters defined in `--vm-native-filter-match` flag to discovered metric names

* app/vmctl: fix comments

* app/vmctl: move function buildMatchWithFilter to the correct place

* app/vmctl: update CHANGELOG.md

* app/vmctl: fix CI, remove error wrapping

* app/vmctl: fix CI, simplify `Set()`
2023-04-06 15:11:53 -07:00
Aliaksandr Valialkin
ff1891397a
docs/CHANGELOG.md: document the bugfix, which remove unneeded logger.Errorf() call during stream aggregation with the enabled deduplication
This is a follow-up for ff72ca14b9
2023-04-06 15:01:25 -07:00
Aliaksandr Valialkin
eca03d1f55
docs/CHANGELOG.md: document that VictoriaMetrics for Windows cannot delete snapshots
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70#issuecomment-1491529183
2023-04-06 03:16:20 -07:00
Aliaksandr Valialkin
c43dc072c4
docs/CHANGELOG.md: document v1.79.12 2023-04-06 03:10:12 -07:00
Aliaksandr Valialkin
c92731de93
docs/CHANGELOG.md: document v1.87.5 2023-04-06 00:44:30 -07:00
Timur Bakeyev
f21ea62ae1
Fix cut-n-paste error (#4079)
It seems that VMServiceScrape description was c-n-p from vmselect one into all other resources.
2023-04-06 00:44:29 -07:00
Zakhar Bessarab
0384b83736
docs: fix example operator spec for vmbackupmanager restore usage (#4074)
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-04-05 22:17:05 -07:00
Timur Bakeyev
210216a1e2
Fix reference to the imagepullsecrets description (#4080)
Looks like the original document has moved to the https://kubernetes.io/docs/concepts/containers/images/#referring-to-an-imagepullsecrets-on-a-pod.
Alternatively, it could be that https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ describes the meaning of the parameter in more detail.
2023-04-05 19:57:22 -07:00
Aliaksandr Valialkin
ac2fbf0a9c
docs/Troubleshooting.md: add missing help word after c7bcf750c2d031b1259cd8115d7464b67f40cb9eg 2023-04-05 14:28:19 -07:00
Aliaksandr Valialkin
e74c54aa86
docs/Troubleshooting.md: another typo fixes after c7bcf750c2d031b1259cd8115d7464b67f40cb9eg 2023-04-05 14:15:00 -07:00
Aliaksandr Valialkin
cd2ed803c2
docs/Troubleshooting.md: fix formatting after c7bcf750c2 2023-04-05 13:46:26 -07:00
Aliaksandr Valialkin
775eb1245d
docs/Troubleshooting.md: fix a typo in the link after c7bcf750c2 2023-04-05 13:40:32 -07:00
Aliaksandr Valialkin
52c46f49e1
all: update Go builder from Go1.20.2 to Go1.20.3
See https://github.com/golang/go/issues?q=milestone%3AGo1.20.3+label%3ACherryPickApproved
2023-04-05 13:38:44 -07:00
Artem Navoiev
c5397c0b77
update changelog
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-04-05 13:31:48 -07:00
Zakhar Bessarab
54edd6992a
app/vmalert: update Grafana URLs to match latest format (#4061)
See: #4019

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-04-05 13:31:06 -07:00
Artem Navoiev
1b05896ac5
fix closing divs in docs
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-04-05 13:30:14 -07:00
Aliaksandr Valialkin
ca4f94c704
docs/Troubleshooting.md: add General troubleshooting checklist
This checklist helps searching for the infromation related to some issue / question
about VictoriaMetrics
2023-04-05 13:29:12 -07:00
Aliaksandr Valialkin
286697b8b6
docs/CHANGELOG.md: document edb45d7fc1
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4013
2023-04-02 21:26:28 -07:00
Aliaksandr Valialkin
02ceebccc0
lib/promscrape: do not re-use previously loaded scrape targets on failed attempt to load updated scrape targets at file_sd_configs
The logic employed for re-using the previously loaded scrape target was broken initially.
The commit cc0427897c tried to fix it, but the new logic
became too complex and fragile. So it is better to just remove this logic,
since the targets from temporarily broken file should be eventually loaded on next
attempts every -promscrape.fileSDCheckInterval

This also allows removing fragile hacks around __vm_filepath label.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3989
2023-04-02 21:11:12 -07:00
Dmytro Kozlov
6f0512a81c
lib/promscrape: fix the problem with scrape work duplicates when file_sd_config can't be read (#4027)
* lib/promscrape: fix the problem with scrape work duplicates when file_sd_config can't be read

* lib/promscrape: clarified comment

* lib/promscrape: made better approach to handle a problem with growing []*ScrapeWork on each error when loading config

* lib/promscrape: added CHANGELOG.md

* Update docs/CHANGELOG.md

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-04-02 21:11:10 -07:00
Yury Molodov
17b4fc9470
vmui: tips for working with the graph and legend (#4045)
* feat: add tips for working with the graph and legend

* feat: add the ability to collapse the legend

* vmui/docs: add the ability to collapse the legend

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-04-01 00:41:01 -07:00
Roman Khavronenko
5f95f9d453
lib/storage: check for free disk space before opening tables (#4035)
* lib/storage: check for free disk space before opening tables

We check for free disk space before call to `openTable`,
so `Storage` can be set to ReadOnly before mergeWorkers start.

Before the change, there was a chance that merges will start
even if Storage has to start in ReadOnly mode because of
`-storage.minFreeDiskSpaceBytes` limit.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4023
Signed-off-by: hagen1778 <roman@victoriametrics.com>

* lib/storage: chore

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Update lib/storage/storage.go

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-03-31 23:50:56 -07:00
Aliaksandr Valialkin
db8fda4ec6
app/vmselect/graphite: open source Graphite Render API 2023-03-31 23:37:40 -07:00
Aliaksandr Valialkin
43b431d322
deployment/docker: update base Docker image from Alpine 3.17.2 to Alpine 3.17.3
This fixes security issues from https://alpinelinux.org/posts/Alpine-3.17.3-released.html

This is a follow-up for 59c350d0d2
2023-03-31 22:54:48 -07:00
Aliaksandr Valialkin
dad13c0a91
lib/streamaggr: follow-up for ff72ca14b9
- Make sure that the last successfully loaded config is used on hot-reload failure
- Properly cleanup resources occupied by already initialized aggregators
  when the current aggregator fails to be initialized
- Expose distinct vmagent_streamaggr_config_reload* metrics per each -remoteWrite.streamAggr.config
  This should simplify monitoring and debugging failed reloads
- Remove race condition at app/vminsert/common.MustStopStreamAggr when calling sa.MustStop() while sa
  could be in use at realoadSaConfig()
- Remove lib/streamaggr.aggregator.hasState global variable, since it may negatively impact scalability
  on system with big number of CPU cores at hasState.Store(true) call inside aggregator.Push().
- Remove fine-grained aggregator reload - reload all the aggregators on config change instead.
  This simplifies the code a bit. The fine-grained aggregator reload may be returned back
  if there will be demand from real users for it.
- Check -relabelConfig and -streamAggr.config files when single-node VictoriaMetrics runs with -dryRun flag
- Return back accidentally removed changelog for v1.87.4 at docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3639
2023-03-31 22:54:10 -07:00
Zakhar Bessarab
743d1e6536
docs/vmctl: add examples of URLs used for migration in different modes (#4042)
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-03-31 22:44:12 -07:00
Roman Khavronenko
0bde9722ed
vmalert: use missingkey=zero for templating (#4040)
Replace empty labels with "" instead of "<no value>"
during templating, as Prometheus does.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4012

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-03-31 22:43:39 -07:00
Zakhar Bessarab
46c8be4f98
lib/fs: verify response code when reading configuration over HTTP (#4036)
Verifying status code helps to avoid misleading errors caused by attempt to parse unsuccessful response.

Related issue: #4034

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-03-31 22:33:53 -07:00
Daria Karavaieva
83213e6786
Vmanomaly guide index fix (#4029)
* name and scrutture change

* fix indexing

* index fix

* name change

* line separator fix
2023-03-31 22:32:15 -07:00
Alexander Marshalov
8c14d17694
added hot reload support for stream aggregation configs (#3969) (#3970)
added hot reload support for stream aggregation configs (#3969)

Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-03-31 22:31:38 -07:00
Eliran Barnoy
7c70fb0fb9
Fix operator links to include VMPrometheusConverter for added visibility 2023-03-30 12:33:04 -07:00
Aliaksandr Valialkin
85ca077a88
lib/flagutil: ArrayString: support commas inside quoted strings and inside [], {} and () braces
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3915
2023-03-28 21:25:07 -07:00
Aliaksandr Valialkin
957d64e302
docs: mention that VictoriaMetrics rounds time range to UTC days at /api/v1/labels, /api/v1/label/.../values and /api/v1/series handlers
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3107
2023-03-28 17:01:09 -07:00
Aliaksandr Valialkin
6f5bbf096a
app/vmagent/remotewrite: cosmetic updates after f3a51e8b1d
- Compare directory names instead of paths to directory when determining which persistent queues must be deleted
  This is less error-prone solution, since paths to the same directory can differ, which could lead
  to accidental directory removal for the existing -remoteWrite.url

- Log the `removed %d dangling queues` message when at least a single queue has been removed

- Consistently use filepath.Join() for creating paths to persistent queues.
  This is needed for Windows support (see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70 )

- Clarify the description of the change at docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4014
2023-03-27 18:38:53 -07:00
Zakhar Bessarab
6ed6eb0c4c
app/vmagent: add -remoteWrite.removeDanglingQueues flag (#4017)
* app/vmagent: add `-remoteWrite.removeDanglingQueues` flag which allows to automatically remove dangling persistent queue contents

Related issue: #4014

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vmagent: address review feedback

- remove persistent queues files by default
- rename `remoteWrite.removeDanglingQueues` to `remoteWrite.keepDanglingQueues`
- update docs to reflect changed behaviour

Related issue: #4014

* Apply suggestions from code review

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-03-27 18:38:51 -07:00
Aliaksandr Valialkin
54b9537a76
app/vmselect/promql: follow-up for 79e1c6a6fc
- Document the fix at docs/CHANGELOG.md
- Add tests with multiple adjancent zero buckets
- Simplify the fix a bit

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/296
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4021
2023-03-27 18:04:30 -07:00
Yury Molodov
86a98fa131
vmui: heatmap (#3780)
* fix: add stroke and font for all axes

* feat: add util for generate gradient

* feat: add heatmap plugin

* feat: add heatmap legend

* feat: add heatmap graph (#3384)

* vmui: add heatmap graph (#3384)

* feat: add convert Prometheus to VictoriaMetrics histogram

* fix: prevent re-render graph

* feat: reset step for heatmap

* feat: normalize heatmap data

* fix: format heatmap legend

* wip

* app/vmselect/vmui: run `make vmui-update`

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-03-26 00:31:21 -07:00
Aliaksandr Valialkin
229b39ac7d
docs/MetricsQL.md: quote min, max and avg args for rollup_*() functions in order to reduce the level of confusion when users try to pass the second argument to these functions 2023-03-26 00:02:01 -07:00
Aliaksandr Valialkin
84b3b0fac2
docs/CHANGELOG.md: document v1.87.4 LTS release 2023-03-25 22:44:21 -07:00
Aliaksandr Valialkin
3698994953
app/{vmbackup,vmrestore}: publish vmbackup and vmrestore binaries for Windows
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70
2023-03-25 15:09:41 -07:00
Aliaksandr Valialkin
7aff6f872f
app/vmselect/promql: follow-up for 7205c79c5a
- Allocate and initialize seriesByWorkerID slice in a single go instead
  of initializing every item in the list separately.
  This should reduce CPU usage a bit.
- Properly set anti-false sharing padding at timeseriesWithPadding structure
- Document the change at docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3966
2023-03-24 23:39:43 -07:00
Zakhar Bessarab
3f38ed3171
app/vmbackup: delete created snapshot in case of error during backup (#4008)
Related issue: #2055

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-03-24 22:09:17 -07:00
Aliaksandr Valialkin
181e877092
docs/CHANGELOG.md: cosmetic fixes: remove trailing whitespace and consistently use -flag instead of --flag 2023-03-24 17:57:23 -07:00
Alexander Marshalov
0301b5018e
allowed using dashes and dots in environment variables names (#4009)
* allowed using dashes and dots in environment variables names for templating config files with envtemplate (#3999)

Signed-off-by: Alexander Marshalov <_@marshalov.org>

* Apply suggestions from code review

---------

Signed-off-by: Alexander Marshalov <_@marshalov.org>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-03-24 17:57:19 -07:00
Aliaksandr Valialkin
c54b8acba2
docs/vmauth.md: follow-up for 36edba9bfb
- Document `-configCheckInterval` command-line flag in `quick start` section
- Clarify the addition of `-configCheckInterval` at docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3990
2023-03-24 17:56:59 -07:00
Aliaksandr Valialkin
9d19c2d89b
docs/vmagent.md: clarify that there is no need to specify multiple -remoteWrite.url options when writing data to a single VictoriaMetrics cluster when data replication is needed
Also add a link to https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format from `getting started` section,
so users could quickly find how to write data to VictoriaMetrics cluster
2023-03-24 17:56:31 -07:00
Roman Khavronenko
ec6a20880c
vmalert: mention VMUI example for alert's source (#4005)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-03-24 17:55:30 -07:00
Roman Khavronenko
b866f334ed
docs: mention cluster URL for exporting series (#4002)
docs: mention cluster URL for exporting series

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-03-24 17:54:57 -07:00
Daria Karavaieva
53c3d6137f
Vmanomaly guide (#3834)
Setting up VMAnomaly on NodeExporter metrics with VictoriaMetrics and AlertManager.


* vmanomaly-guide-draft

* aletr graphs and description

* readme vmanomaly tutorial

* Added back fit_every param for performance

* vmanomaly guide fixes

* added spaces div

* spaces + resize image

* alert example grammar

* quotation marks

* docker link

* typo fixed

* more links

* reader section rephrased

* label change

* lower case for grafana service

* lower case for vm service

* yaml markdown

---------

Co-authored-by: Dima Lazerka <dima@victoriametrics.com>
2023-03-24 17:54:10 -07:00
Dmytro Kozlov
fe7acd5e8a
docs: follow up after dc2c712a29 (#4001) 2023-03-24 16:41:42 -07:00
Yury Molodov
5f77efa915
vmui: display errors for each query individually (#3987) (#3994) 2023-03-24 13:26:43 -07:00
Alexander Marshalov
b5027cff9c
added configCheckInterval flag for vmauth (#3990) (#3991)
* added configCheckInterval flag for vmauth (#3990)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-03-24 13:25:07 -07:00
Nikolay
9bb83cafa4
lib/netutil: log only parsing errors for proxy-protocol (#3985)
* lib/netutil: log only parsing errors for proxy-protocol

Previosly every error was logged. With configured TCP health checks at load-balancer or kubernetes, vmauth spams a lot of false positive error message into logs

* Update docs/CHANGELOG.md

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* Update lib/netutil/tcplistener.go

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-03-21 10:23:08 -07:00
Aliaksandr Valialkin
8ed9295109
docs/vmagent.md: mention in docs that the target relabel debug page shows target url now 2023-03-20 22:20:13 -07:00
Dmytro Kozlov
85b01c4aa7
lib/promrelabel: make target url from labels on target relabel page (#3882)
* lib/promrelabel: make target url from labels on target relabel page

* wip

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-03-20 22:08:39 -07:00
Aliaksandr Valialkin
1580d8bda7
docs/CHANGELOG.md: cosmetic fixes 2023-03-20 14:33:46 -07:00
Aliaksandr Valialkin
531b35b6c0
docs/Troubleshooting.md: document an additional case, which could result in slow inserts
If `-cacheExpireDuration` is lower than the interval between ingested samples for the same time series,
then vm_slow_row_inserts_total` metric is increased.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3976#issuecomment-1476883183
2023-03-20 14:33:27 -07:00
Dmytro Kozlov
aed59b9029
app/vmctl: automatically check tty (#3938)
app/vmctl: automatically detect if TTY is available
2023-03-20 14:14:43 -07:00
Yury Molodov
95b60c2777
vmui: support for drag'n'drop in the "Trace analyzer" page (#3971)
vmui: add drag-and-drop support for the trace analyzer page
2023-03-20 14:09:45 -07:00
Yury Molodov
b66953d8e1
vmui: improve usability of date/time picker (#3968)
* vmui: allow manually set input date and time
* vmui/docs: improve usability of date/time picker
2023-03-20 13:57:47 -07:00
Aliaksandr Valialkin
fc3d826d7f
all: add Windows build for VictoriaMetrics
This commit changes background merge algorithm, so it becomes compatible with Windows file semantics.

The previous algorithm for background merge:

1. Merge source parts into a destination part inside tmp directory.
2. Create a file in txn directory with instructions on how to atomically
   swap source parts with the destination part.
3. Perform instructions from the file.
4. Delete the file with instructions.

This algorithm guarantees that either source parts or destination part
is visible in the partition after unclean shutdown at any step above,
since the remaining files with instructions is replayed on the next restart,
after that the remaining contents of the tmp directory is deleted.

Unfortunately this algorithm doesn't work under Windows because
it disallows removing and moving files, which are in use.

So the new algorithm for background merge has been implemented:

1. Merge source parts into a destination part inside the partition directory itself.
   E.g. now the partition directory may contain both complete and incomplete parts.
2. Atomically update the parts.json file with the new list of parts after the merge,
   e.g. remove the source parts from the list and add the destination part to the list
   before storing it to parts.json file.
3. Remove the source parts from disk when they are no longer used.

This algorithm guarantees that either source parts or destination part
is visible in the partition after unclean shutdown at any step above,
since incomplete partitions from step 1 or old source parts from step 3 are removed
on the next startup by inspecting parts.json file.

This algorithm should work under Windows, since it doesn't remove or move files in use.
This algorithm has also the following benefits:

- It should work better for NFS.
- It fits object storage semantics.

The new algorithm changes data storage format, so it is impossible to downgrade
to the previous versions of VictoriaMetrics after upgrading to this algorithm.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3236
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3821
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70
2023-03-19 23:28:26 -07:00
Aliaksandr Valialkin
d2f85816ea
lib/{mergeset,storage}: prevent from long wait time when creating a snapshot under high data ingestion rate
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3551
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3873
2023-03-19 00:19:02 -07:00
Alexander Marshalov
31005f4980
updated api doc for operator (#3972)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-03-17 16:17:17 -07:00
Zakhar Bessarab
c397e56769
doc/vmgateway-grafana-openid-guide: fix formatting, add reproducible example and example results (#3964) 2023-03-17 16:15:45 -07:00
Aliaksandr Valialkin
c9e30dd892
all: typo fix in the same way as in e566d49e3a: 8248 -> 8428 2023-03-16 22:08:26 -07:00
Artem Navoiev
7e6be18c3d
managed quickstart fix typo
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-03-16 22:04:03 -07:00
Artem Navoiev
208aa85362
fix anchor after chaning manager quick start
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-03-16 22:03:46 -07:00
Nikolay
a61106d965
Vmagent kafka updates (#535)
* app/vmagent: allow vm proto for kafka consumer and producer
it should reduce network usage up to 50%.
According to benchmarks without any encoding at kafka topic, it reduces traffic up to 50%.
With enabled zstd at kafka topic, it shows no diffence in traffic. So it
doesn't make much sense to use it.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225

* mention eb61a7dd68b834b08d01727a918f207700348ada at changelog

* app/vmagent: bumps kafka lib version
it allows compiling vmagent for arm64 machines
fixes https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2271

* mention d19b1a888248c96cfd7ccee00ba6f596d89be1d7 at change log

* app/vmagent: adds natural concurrency for kafka consumer
it should improve performance for data consumption
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1957

* mention change 0c143bb22ca2e7e0b7eec9bc84a94ee2b41626ca

* Update app/vmagent/kafka/consumer.go

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* Update app/vmagent/kafka/consumer_cgo.go

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-03-15 13:03:51 -07:00
Alexander Marshalov
3f10f19842
updated vars doc for operator (#3960)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-03-15 12:19:53 -07:00
dmitryk-dk
1ff82e23af
docs: cleanup 2023-03-15 12:19:16 -07:00
dmitryk-dk
e1c2c91bb6
app/vmctl: update managed quickstart guide 2023-03-15 12:18:48 -07:00
Zakhar Bessarab
a3f364869f
doc: add guide for vmgateway configuration with OpenID and Grafana (#3951)
docs: add guide for vmgateway configuration with OpenID and Grafana
2023-03-14 16:11:29 -07:00
Zakhar Bessarab
d1d108fe77
lib/storage: log original labels set when label value is truncated (#3952)
lib/storage: log original labels set when label value is truncated
2023-03-14 16:11:02 -07:00
Aliaksandr Valialkin
9aa6b63160
docs/Single-server-VictoriaMetrics.md: clarify that the cache directory can be removed manually when VictoriaMetrics is stopped 2023-03-13 00:24:01 -07:00
Aliaksandr Valialkin
d345619424
docs/CHANGELOG.md: document v1.87.3 2023-03-13 00:21:00 -07:00
Aliaksandr Valialkin
1ae73a5fe5
docs/CHANGELOG.md: document v1.79.11 LTS release 2023-03-12 23:23:05 -07:00
Aliaksandr Valialkin
47c5b77c8c
docs/CHANGELOG.md: cut v1.89.1 2023-03-12 19:14:29 -07:00
Aliaksandr Valialkin
3b4a3583bc
app/vmselect/promql: prevent from cannot unmarshal timeseries from rollupResultCache panic after the upgrade to v1.89.0 2023-03-12 19:09:11 -07:00
Aliaksandr Valialkin
a70b517d92
docs/CHANGELOG.md: cut v1.89.0 2023-03-12 17:29:50 -07:00
Aliaksandr Valialkin
a6a4beb89a
app/vmselect: remove data race on updating EvalConfig.IsPartialResponse from concurrently running goroutines
This properly returns `is_partial: true` for partial responses.
2023-03-12 16:53:03 -07:00
Artem Navoiev
0186c7fdfb
fix typos on image
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-03-12 16:32:37 -07:00
Aliaksandr Valialkin
5cd60c54d3
app/vmselect/promql: prevent from SIGBUS crash on architecures, which deny unaligned access to 8-byte words (e.g. ARM)
Thanks to @oliverpool for nailing down the root cause of the issue and for the initial attempt to fix it
at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3927
2023-03-12 16:29:18 -07:00
Yury Molodov
2a1bc14984
vmui: remove send step param for instant queries (#3931)
* fix: remove step param for instant queries (#3896)

* vmui: remove send step param for instant queries

* Update docs/CHANGELOG.md

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-03-12 03:14:05 -07:00
Aliaksandr Valialkin
6de1d034c8
docs/CHANGELOG.md: document 113a89904d 2023-03-12 01:58:02 -08:00
Aliaksandr Valialkin
97479bd9e3
docs/CHANGELOG.md: yet another typo fix 2023-03-12 01:52:17 -08:00
Aliaksandr Valialkin
22e8bd7222
docs/CHANGELOG.md: typo fix 2023-03-12 01:04:23 -08:00
Aliaksandr Valialkin
e491fee1f4
app/vmselect/netstorage: do not intern string representation of MetricName for time series received from vmstorage
It has been appeared that this interning may lead to increased memory usage and increased CPU usage
when vmselect performs queries, which select big number of time series.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3692
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3863
2023-03-12 00:44:08 -08:00
Aliaksandr Valialkin
d106ababb8
docs/CHANGELOG.md: document 927d9da270 2023-03-12 00:28:00 -08:00
Aliaksandr Valialkin
5ac8f89d2a
docs/CHANGELOG.md: typo fixes 2023-03-12 00:28:00 -08:00
Aliaksandr Valialkin
868970bd9e
docs/CHANGELOG.md: document c9f44daaee8f4282d9ed41e3ba799c7a33841313 2023-03-12 00:27:33 -08:00
Roman Khavronenko
8b907fffdb
vmalert: support concurrent reading from object storage (#532)
* vmalert: support concurrent reading from object storage

Config reading from GCS or S3 can be slow if object storage
contains a big number of files. Object storages are usually
fast for downloading and are slow for individual operations.
If there would be thousands of files to read, vmalert could
spend significant time for retrieving those because it is
done sequentially.

The change introduces ability to read configs from object
storage concurrently. By default, both GCS and S3 are now
read with 50 concurrent readers. This significantly reduces
the load time:
* loading 500 files with concurrency=1 takes 27s
* loading 500 files with concurrency=50 takes <1s

* vmalert: add note to Changelog

* vmalert: cleanup

* vmalert: use ticker properly

* app/vmalert: improve status reporting during config loading

* vmalert: support concurrent reading from object storage

Config reading from GCS or S3 can be slow if object storage
contains a big number of files. Object storages are usually
fast for downloading and are slow for individual operations.
If there would be thousands of files to read, vmalert could
spend significant time for retrieving those because it is
done sequentially.

The change introduces ability to read configs from object
storage concurrently. By default, both GCS and S3 are now
read with 50 concurrent readers. This significantly reduces
the load time:
* loading 500 files with concurrency=1 takes 27s
* loading 500 files with concurrency=50 takes <1s

* app/vmalert: make linter happy
2023-03-12 00:25:34 -08:00
Alexander Marshalov
78e9e4742d
Stream aggregation doc improvements based on users feedback (#3934)
docs: stream aggregation doc improvements based on users feedback
2023-03-12 00:19:22 -08:00
Zakhar Bessarab
3b7152b1d8
docs: add a note about cache reset for vmalert backfilling docs (#3940)
docs: add a note about cache reset for vmalert backfilling docs
2023-03-12 00:13:00 -08:00
Dmytro Kozlov
4029c539b9
app/vmctl: update importing tips when migrating data with overlapping time range (#3941)
app/vmctl: update importing tips when migrating data with overlapping time range
2023-03-12 00:10:08 -08:00
Dmytro Kozlov
f7d3efe3db
app/vmctl: add support of basic auth and barer token (#3921)
app/vmctl: add support of basic auth and bearer token
2023-03-12 00:08:04 -08:00
Roman Khavronenko
310b380a03
app/vmalert: log number of configration files found for each specified -rule (#3936)
The change also introduces `List` method to `FS` interface.
The `List` method can be used for wildcard support in object storage FS.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2023-03-11 23:40:40 -08:00
Dmytro Kozlov
afc6a6e218
app/vmctl: follow up after 09e3742a82 (#3937)
app/vmctl: follow up after 09e3742a82
2023-03-11 23:39:41 -08:00
Roman Khavronenko
b6ed977fa4
security: bump go version to 1.20.2 (#3935)
upgrade Go builder from Go1.20.1 to Go1.20.2
See the list of issues addressed in Go1.20.2 here (https://github.com/golang/go/issues?q=milestone%3AGo1.20.2+label%3ACherryPickApproved).

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-03-11 23:38:14 -08:00
Denys Holius
e8a0d60ea5
Adds snap badge to README.md (#3930)
docs: adds snap badge to README.md
2023-03-11 23:31:38 -08:00
Aliaksandr Valialkin
54fe207cc0
all: follow-up for 7a3e16e774
- Sync the description for -httpListenAddr.useProxyProtocol command-line flag at vmagent and vmauth,
  so it is consistent with the description at vmauth and victoria-metrics
- Add a sample of panic text to docs/CHANGELOG.md, so it could be googled
- Mention the -httpListenAddr.useProxyProtocol command-line flag in the description for the bugfix

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335
2023-03-08 01:42:58 -08:00
Aliaksandr Valialkin
37981b276b
docs/CHANGELOG.md: improve description for 4b136abff8 2023-03-08 01:39:59 -08:00
Aliaksandr Valialkin
a911889817
docs/CHANGELOG.md: improve the description of the bugfix at 62beea23f7
- Make the description easier to read by humans :)
- Add a link to VictoriaMetrics datasource plugin for Grafana, so users could easily discover it
2023-03-08 01:39:39 -08:00
Aliaksandr Valialkin
bb5a39e5d1
docs/CHANGELOG.md: clarify the description for 6bfe9cc733
- Add the panic message to the description, so it is easier to google
- Add a link to the corresponding bugreport

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3897
2023-03-08 01:38:54 -08:00
Aliaksandr Valialkin
e91fe74618
all: typo fixes of the same type as in the d056be710b 2023-03-08 01:37:28 -08:00
Aliaksandr Valialkin
15f5639eb4
docs/CHANGELOG.md: clarify the description for the change at 8bab50dc29
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3600
2023-03-08 01:35:56 -08:00
Aliaksandr Valialkin
b60594a548
app/vmagent/remotewrite: follow-up for e3a756d82869f8c357b072f6e635ebfc7d65dd2c
- Document the fix
- Move the detection of VictoriaMetrics remoteWrite protocol from client.init() to newHTTPClient()
  This simplifies the fix to the following diff:

diff --git a/app/vmagent/remotewrite/client.go b/app/vmagent/remotewrite/client.go
index 099899c19..70b904af4 100644
--- a/app/vmagent/remotewrite/client.go
+++ b/app/vmagent/remotewrite/client.go
@@ -151,10 +151,6 @@ func newHTTPClient(argIdx int, remoteWriteURL, sanitizedURL string, fq *persiste
        }
        c.sendBlock = c.sendBlockHTTP

-       return c
-}
-
-func (c *client) init(argIdx, concurrency int, sanitizedURL string) {
        useVMProto := forceVMProto.GetOptionalArg(argIdx)
        usePromProto := forcePromProto.GetOptionalArg(argIdx)
        if useVMProto && usePromProto {
@@ -173,6 +169,10 @@ func (c *client) init(argIdx, concurrency int, sanitizedURL string) {
        }
        c.useVMProto = useVMProto

+       return c
+}
+
+func (c *client) init(argIdx, concurrency int, sanitizedURL string) {
2023-03-08 01:35:31 -08:00
Aliaksandr Valialkin
eb62b5548d
docs/CHANGELOG.md: document ec2abf9b69 2023-03-08 01:34:08 -08:00
Alexander Marshalov
0b7f83c846
added documentation about new templates field of vmalertmanager specification in operator (https://github.com/VictoriaMetrics/operator/issues/592) (#3924)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-03-08 01:33:42 -08:00
Nikolay
88f10d24a0
lib/netutil: fixes panic at proxy protocol (#3905)
it may occur if non proxy protocol message received by tcp server.
Listener Accept method must return only non-recoverable errors.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335
2023-03-08 01:33:01 -08:00
Yury Molodov
3e3e23a483
vmui: fix display of selected value in the selector (#3919)
vmui: fix selected value in dropdowns for Explore page
2023-03-08 01:13:54 -08:00
Nikolay
77e0e847a8
docs: updates operator api.md (#3922) 2023-03-08 01:12:16 -08:00
Craig Rodrigues
4c94fe0f0a
docs: move installation methods further up in README (#3904) 2023-03-08 01:09:35 -08:00
Dmytro Kozlov
d59a20c0a7
docs: follow up after 4b136abff8 (#3918)
docs: follow up after 4b136abff8
2023-03-08 01:08:07 -08:00
Alexander Marshalov
c682609fc4
added doc for placeholder support in vmagent specification for operator (https://github.com/VictoriaMetrics/operator/issues/592) (#3916) 2023-03-08 01:07:14 -08:00
Roman Khavronenko
e2527917a9
docs: follow-up after 62beea23f7 (#3907)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-03-08 01:01:16 -08:00
Artem Makhortov
f9723aca44
doc: vmctl vm-native-step-interval supported values (#3899) 2023-03-08 00:47:37 -08:00
Artem Navoiev
950abf7457
remove image width for vmalert managed vm guide
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-03-08 00:26:53 -08:00
Artem Navoiev
0ba6c42ea2
change title
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-03-08 00:26:22 -08:00
Artem Navoiev
470c5561d2
add vmalert managed vm integration guide
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-03-08 00:25:37 -08:00
Dmytro Kozlov
ff0e63ef0d
app/vmctl: add backoff retries to native protocol (#3859)
app/vmctl: vm-native - split migration on per-metric basis

`vm-native` mode now splits the migration process on per-metric basis. 
This allows to migrate metrics one-by-one according to the specified filter. 
This change allows to retry export/import requests for a specific metric and provides a better 
understanding of the migration progress.

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2023-03-08 00:24:08 -08:00
Dmytro Kozlov
08df754903
app/vmctl: Increase http request timeout made by remote read client, add importing tips (#3879)
app/vmctl: Increase default http request timeout made by remote read client
2023-03-07 23:35:25 -08:00
Nikolay
361e1b1165
lib{mergset,storage}: prevent possible race condition with logging st… (#3900) (#3917)
lib{mergset,storage}: prevent possible race condition with logging stats for merges

Previously partwrapper could be release by background process and reference for part may be invalid 
during logging stats. It will lead to panic at vmstorage
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3897
2023-03-06 11:11:08 +01:00
Aliaksandr Valialkin
223bb6f367
docs/CHANGELOG.md: document v1.79.10 release 2023-02-27 17:36:13 -08:00
Aliaksandr Valialkin
ea1d7f28e5
docs/CHANGELOG.md: cut v1.88.1 2023-02-27 15:28:27 -08:00
Aliaksandr Valialkin
da419e7188
docs/CHANGELOG.md: link to the issue, which may benefit from -internStringDisableCache command-line flag
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3863
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3692
2023-02-27 14:56:22 -08:00
Aliaksandr Valialkin
086a4b4fca
lib/bytesutil: add -internStringDisableCache and -internStringCacheExpireDuration command-line flags
This commit is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3872
2023-02-27 14:18:02 -08:00
Aliaksandr Valialkin
1ad0d22e80
lib/storage: follow-up for 39cdc546dd
- Use flag.Duration instead of flagutil.Duration for -snapshotCreateTimeout,
  since the flagutil.Duration is intended mostly for big durations, e.g. days, months and years,
  while the -snapshotCreateTimeout is usually smaller than one hour.
- Add links to https://docs.victoriametrics.com/#how-to-work-with-snapshots in docs/CHANGELOG.md,
  so readers could easily find the corresponding docs when reading the changelog.
- Properly remove all the created directories on unsuccessful attempt to create
  snapshot in Storage.CreateSnapshot().

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3551
2023-02-27 13:11:10 -08:00
Zakhar Bessarab
26682e369e
lib/storage: enhancements for snapshots process (#3873)
* lib/{fs,mergeset,storage}: skip `.must-remove.` dirs when creating snapshot (#3858)

* lib/{mergeset,storage}: add timeout configuration for snapshots creation, remove incomplete snapshots from storage

* docs: fix formatting

* app/vmstorage: add metrics to track status of snapshots

* app/vmstorage: use `vm_http_requests_total` metric for snapshot endpoints metrics, rename new flag to make name more clear

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vmstorage: update flag name in docs

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vmstorage: reflect new metrics names change in docs

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-27 13:11:06 -08:00
Zakhar Bessarab
1db010797e
lib/promscrape: correctly register vm_promscrape_config_* metrics (#3876)
* lib/promscrape: set `vm_promscrape_config_last_reload_successful` to 1 if there was no promscrape config provided

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/promscrape: register `vm_promscrape_config_*` metrics only in case promscrape config is used

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-27 12:06:49 -08:00
Zakhar Bessarab
9d04bb6004
doc: add changelog reference for vmgateway OpenID discovery (#3877)
* doc: add changelog reference for vmgateway OpenID discovery

* doc: add vmgateway docs

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-27 12:06:48 -08:00
Aliaksandr Valialkin
d3605ad072
app/vmselect/promql: fix panic when calculating aggr_func(rollup*())
The panic has been introduced in dac21d874b
2023-02-27 11:48:38 -08:00
Zakhar Bessarab
215206bded
app/vmgateway: add new flag doc
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-02-27 11:18:37 -08:00
Zakhar Bessarab
08a61d5c13
app/vmgateway: fix typo in docs
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-02-27 11:18:11 -08:00
Zakhar Bessarab
1b4871a9da
app/vmgateway: add OpenID discovery of JWKS endpoints
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-02-27 11:17:44 -08:00
Aliaksandr Valialkin
d9581cf3f3
app/vmagent: add -remoteWrite.vmProtoCompressLevel command-line flag for tuning the compression level for VictoriaMetrics remote write protocol 2023-02-27 11:04:11 -08:00
Dmytro Kozlov
65d9e281d6
app/vmctl: skip series if measurement not found (#3869)
app/vmctl: skip measurements with no fields for influxdb mode
2023-02-27 10:35:13 -08:00
Aliaksandr Valialkin
1e156ac3c3
app/vmagent: use the provided auth options when checking whether the remote storage supports VictoriaMetrics remote write protocol
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3847
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225
2023-02-26 12:19:53 -08:00
Aliaksandr Valialkin
a5e79b6f38
docs/CHANGELOG.md: cut v1.88.0 2023-02-24 17:54:35 -08:00
Aliaksandr Valialkin
90114759fa
docs: update -help output after e1c3267e34 2023-02-24 17:16:24 -08:00
Roman Khavronenko
66d0b45651
vmselect/promql: check for deadline in count_values fn (#3806)
* vmselect/promql: check for deadline in `count_values` fn

`count_values` could be very slow during the data processing.
Checking for deadline between iterations supposed to reduce
probability of exceeding `search.maxQueryDuration`.

The change also adds a new trace record, which captures the time
spent in aggregation function. Before that, the trace for aggr funcs
could be confusing since it doesn't account for all the places where
time was spent.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* wip

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-24 17:10:38 -08:00
Aliaksandr Valialkin
33df81cb4b
docs/CHANGELOG.md: document v1.87.2 release 2023-02-24 16:14:41 -08:00
Aliaksandr Valialkin
698aa1dd2a
docs/CHANGELOG.md: document v1.79.9 release 2023-02-24 15:10:58 -08:00
Roman Khavronenko
79eb33556e
metricsql: support optional 2nd argument for rollup functions (#3841)
* metricsql: support optional 2nd argument for rollup functions

Support optional 2nd argument `min`, `max` or `avg` for rollup functions:
 * rollup
 * rollup_delta
 * rollup_deriv
 * rollup_increase
 * rollup_rate
 * rollup_scrape_interval

 If second argument is passed, then rollup function will return only the selected aggregation type.
 This change can be useful for situations where only one type of rollup calculation is needed.
 For example, `rollup_rate(requests_total[5m], "max")`.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* wip

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-24 13:48:30 -08:00
Aliaksandr Valialkin
21fb56a498
docs/CHANGELOG.md: document d8eaa511b0 2023-02-24 12:43:45 -08:00
Aliaksandr Valialkin
0a9f5e178e
docs/CHANGELOG.md: typo fix: scrape scrape -> scrape 2023-02-24 12:33:37 -08:00
Aliaksandr Valialkin
aed2dbe45e
lib/promscrape: follow-up for 43e104a83f
- Return immediately on context cancel during the backoff sleep.
  This should help with https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3747

- Add a comment describing why the second attempt to obtain the response from remote side
  is perfromed immediately after the first attempt.

- Remove fasthttp dependency from lib/promscrape/discoveryutils

- Set context deadline before calling doRequestWithPossibleRetry().
  This simplifies the doRequestWithPossibleRetry() a bit.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3293
2023-02-24 12:25:36 -08:00
Zakhar Bessarab
5ea6d71cb3
fix: do not use exponential backoff for first retry of scrape request (#3824)
* fix: do not use exponential backoff for first retry of scrape request (#3293)

* lib/promscrape: refactor `doRequestWithPossibleRetry` backoff to simplify logic

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* Update lib/promscrape/client.go

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>

* lib/promscrape: refactor `doRequestWithPossibleRetry` to make it more straightforward

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-02-24 12:25:35 -08:00
Aliaksandr Valialkin
dce8afa4c4
app/vmselect/vmui: make vmui-update after d4fc0ed874 2023-02-23 19:26:08 -08:00
Aliaksandr Valialkin
3563c6882e
docs: update --help descriptions after recent changes 2023-02-23 19:02:32 -08:00
Aliaksandr Valialkin
6369c88a68
app/vmselect: add -search.logQueryMemoryUsage command-line flag for logging queries, which take big amounts of memory
Thanks to @michal-kralik for initial attempts for this feature:

- https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3651
- https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3715

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3553
2023-02-23 18:52:44 -08:00
Aliaksandr Valialkin
f579cac297
app/vmagent: automatically detect whether the remote storage supports VictoriaMetrics remote write protocol
Substitute -remoteWrite.useVMProto with -remoteWrite.forcePromProto command-line flag,
which can be used for forcing Prometheus remote write protocol in cases when the remote storage
supports VictoriaMetrics remote write protocol.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3847
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225
2023-02-23 17:38:47 -08:00
Aliaksandr Valialkin
e299854077
docs/CHANGELOG.md: document 6d019a3c37
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3830
2023-02-22 19:24:33 -08:00
Alexander Marshalov
9bb9bd266b
fix interpolate function for filling only intermediate gaps (#3816) (#3857)
* fix interpolate function for filling only intermediate gaps (#3816)

* Update docs/CHANGELOG.md

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-22 18:41:29 -08:00
Alexander Marshalov
66764ea4a0
fixed typo in dns+srv documentation (#3861) 2023-02-22 17:52:25 -08:00
Aliaksandr Valialkin
9609a44cc7
docs/CHANGELOG.md: document d2b92d3264
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3747
2023-02-22 17:50:55 -08:00
Aliaksandr Valialkin
1b70238dca
lib/promscrape/discovery/kuma: follow-up for 317fef95f9
- Do not generate __meta_server label, since it is unavailable in Prometheus.
- Add a link to https://docs.victoriametrics.com/sd_configs.html#kuma_sd_configs to docs/CHANGELOG.md,
  so users could click it and read the docs without the need to search the corresponding docs.
- Remove kumaTarget struct, since it is easier generating labels for discovered targets
  directly from the response returned by Kuma. This simplifies the code.
- Store the generated labels for discovered targets inside atomic.Value. This allows reading them
  from concurrent goroutines without the need to use mutex.
- Use synchronouse requests to Kuma instead of long polling, since there is a little sense
  in the long polling when the Kuma server may return 304 Not Modified response every -promscrape.kumaSDCheckInterval.
- Remove -promscrape.kuma.waitTime command-line flag, since it is no longer needed when long polling isn't used.
- Set default value for -promscrape.kumaSDCheckInterval to 30s in order to be consistent with Prometheus.
- Remove unnecessary indirections for string literals, which are used only once, in order to improve code readability.
- Remove unused fields from discoveryRequest and discoveryResponse.
- Update tests.
- Document why fetch_timeout and refresh_interval options are missing in kuma_sd_config.
- Add docs to discoveryutils.RequestCallback and discoveryutils.ResponseCallback,
  since these are public types.

Side notes: it is weird that Prometheus implementation for kuma_sd_configs sets `instance` label,
since usually this label is set by the Prometheus itself to __address__ after the relabeling phase.
See https://www.robustperception.io/life-of-a-label/

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3389

See https://github.com/prometheus/prometheus/issues/7919
and https://github.com/prometheus/prometheus/pull/8844
as a reference implementation in Prometheus
2023-02-22 17:50:54 -08:00
Aliaksandr Valialkin
55708fa474
docs/CHANGELOG.md: document 110c3896e7
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3600
2023-02-22 17:50:13 -08:00
Artem Navoiev
3abd6b367e
Add Dig Security case study
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-02-22 17:42:51 -08:00
Alexander Marshalov
173643a771
add kuma_sd_config for Kuma Control Plane targets discovery (#3389) (#3840) 2023-02-22 17:41:43 -08:00
Artem Navoiev
522179fbde
docs: fix typo OpentTSDB -> OpenTSDB (#3854)
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-02-22 12:11:48 -08:00
Aliaksandr Valialkin
ff8c57a964
app/vmselect: allow zero value for -search.latencyOffset command-line flag
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2061#issuecomment-1299109836

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/218
2023-02-21 18:07:27 -08:00
Aliaksandr Valialkin
57de6ad5dd
docs/Articles.md: mention rules backfilling via vmalert article
This is a follow-up for 5446ce0018
2023-02-21 17:44:45 -08:00
Roman Khavronenko
a29c1d3a02
docs: mention rules replay blogpost in vmalert docs (#3851)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-02-21 17:44:32 -08:00
panguicai
1b75d66bcd
docs: update operator release name to be consistent with the following (#3845)
Signed-off-by: panguicai008 <1121906548@qq.com>
2023-02-21 17:42:39 -08:00
Aliaksandr Valialkin
f783237866
docs/CHANGELOG.md: fix a link to VictoriaMetrics remote write protocol 2023-02-20 20:00:00 -08:00
Aliaksandr Valialkin
09b21fd800
docs/Single-server-VictoriaMetrics.md: remove + from start and end query args, since curl substitutes them with whitespace and breaks the query 2023-02-20 19:32:39 -08:00
Aliaksandr Valialkin
04ce4c4762
docs/guides/migrate-from-influx.md: remove misleading doublequotes from the query example 2023-02-20 19:25:07 -08:00
Zakhar Bessarab
8a35dc608f
vmgateway: add support of JWKS endpoint usage for JWT keys verification (#521) 2023-02-20 19:23:00 -08:00
Roman Khavronenko
fd139b463b
docs: update vmalert docs (#3843)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-02-20 19:13:00 -08:00
Corporte Gadfly
7d99b19478
docs: typo fix (#3839) 2023-02-20 19:11:59 -08:00
Aliaksandr Valialkin
fc5ea31a38
docs/vmagent.md: remove the claim that VictoriaMetrics remote write protocol reduces the network bandwidth usage by up to 10x comparing to Prometheus remote write protocol
The 10x savings are reproduced only on artificial data.
The savings on production data are usually in the range 2x-4x.
2023-02-20 19:11:11 -08:00
Aliaksandr Valialkin
bf359e8cc0
docs/vmagent.md: mention that Mimir doesnt support backfilling 2023-02-20 19:06:56 -08:00
Aliaksandr Valialkin
80c6d1e24c
app/vmagent: add support for VictoriaMetrics remote write protocol, which allows saving up to 10x on network bandwidth costs under high load
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1225
2023-02-20 18:40:40 -08:00
Aliaksandr Valialkin
a15da5ff73
app/vmselect/promql: add share(q) aggregate function for normalizing results across multiple time series in [0..1] value range per each timestamp and aggregation group 2023-02-18 22:43:54 -08:00
Aliaksandr Valialkin
84b5532bc1
app/vmselect/promql: add range_zscore(q) and range_trim_zscore(z, q) functions
These functions may be useful for dropping outliers at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3759
2023-02-18 22:43:53 -08:00
Haleygo
9a274567f1
vmalert: fix maxResolveDuration flag note (#3827)
Signed-off-by: Haleygo <hui.wang@daocloud.io>
2023-02-18 15:20:30 -08:00
Aliaksandr Valialkin
450b6f6d39
app/vmselect/promql: add range_mad(q) and range_trim_outliers(k, q) functions
These functions may help trimming outliers during query time
for the use case described at https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3759
2023-02-18 15:18:47 -08:00
Aliaksandr Valialkin
ecc84f7923
all: update Go builder from Go1.20.0 to Go1.20.1
See https://github.com/golang/go/issues?q=milestone%3AGo1.20.1+label%3ACherryPickApproved
2023-02-14 23:05:54 -08:00
Roman Khavronenko
c6251ec8aa
docs: improve troubleshooting docs for vmalert (#3812)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-02-13 09:42:18 -08:00
Aliaksandr Valialkin
caa2497aca
docs/CHANGELOG.md: improve the docs for 8ea02eaa8e 2023-02-13 09:41:22 -08:00
Roman Khavronenko
5941b5579d
follow-up after d1cbc35cf6 (#3813)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-02-13 09:37:09 -08:00
Artem Navoiev
253a78b9ab
change docs for VictoriaMetrics Managed (#3803)
* change docs for VictoriaMetrics Managedd

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

* Update docs/managed-victoriametrics/user-managment.md

Co-authored-by: Max Golionko <8kirk8@gmail.com>

---------

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
Co-authored-by: Max Golionko <8kirk8@gmail.com>
2023-02-13 09:33:32 -08:00
Aliaksandr Valialkin
8355e4a1ee
docs/sd_configs.md: properly escape __address__ string 2023-02-11 14:46:02 -08:00
Aliaksandr Valialkin
c8e41100e6
docs/sd_configs.md: document how the __address__ label is generated per each discovered target 2023-02-11 14:42:10 -08:00
Aliaksandr Valialkin
27b3209816
app/vmui: show median instead of avg on graph tooltip and line legend
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3706
2023-02-11 12:52:50 -08:00
Aliaksandr Valialkin
9053745a6f
lib/{mergeset,storage}: allow at least 3 concurrent flushes during background merges on systems with 1 or 2 CPU cores
This should prevent from data ingestion slowdown and query performance degradation
on systems with small number of CPU cores (1 or 2), when big merge is performed.

This should help https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3790

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337
2023-02-11 12:09:13 -08:00
Aliaksandr Valialkin
1a6d266d1a
docs/Articles.md: add a link to https://www.techetio.com/2022/08/21/evaluating-backend-options-for-prometheus-metrics/ 2023-02-11 11:37:06 -08:00
Aliaksandr Valialkin
db7f237da9
app/vmselect/promql: add mad_over_time(m[d]) function
See https://github.com/prometheus/prometheus/issues/5514
2023-02-11 01:06:39 -08:00
Aliaksandr Valialkin
4d9aaff3e1
all: update alpine base docker image from 1.17.1 to 1.17.2
See https://alpinelinux.org/posts/Alpine-3.17.2-released.html
2023-02-11 00:37:28 -08:00
Aliaksandr Valialkin
9ed5b872df
app/vmauth: improve load balancing by sending incoming requests to backends with the lowest number of concurrent requests
While at it, stop sending requests to unavailable backend for 3 seconds
before the next attempt. This should reduce the amounts of useless work
and the number of useless network packets when the backend is temporarily unavailable.
2023-02-11 00:32:56 -08:00
Aliaksandr Valialkin
6c6b9c3d93
app/vmauth: add -maxConcurrentPerUserRequests command-line option for limiting the number of concurrent requests on a per-user basis
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3346
2023-02-10 21:58:30 -08:00
Aliaksandr Valialkin
769d7da25a
app/vmauth: automatically retry failing GET requests on the remaining backends 2023-02-09 21:06:09 -08:00
Dmytro Kozlov
c80fc8c77f
app/vmauth: add concurrent requests limit per auth record (#3749)
* app/vmauth: add concurent requests limit per auth record

* app/vmauth: added clarification comment

* app/vmauth: remove unused code

* app/vmauth: move read from limiter

* app/vmauth: fix text

* app/vmauth: fix comments

* - Clarify the docs for the max_concurrent_requests option at docs/vmauth.md
- Clarify the description of the change at docs/CHANGELOG.md
- Make sure that the -maxConcurrentRequests takes precedence over per-user max_concurrent_requests
- Update tests for verifying that the max_concurrent_requests option is parsed properly

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3346

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-09 20:03:47 -08:00
Aliaksandr Valialkin
ca61c276ca
app/vmalert: follow-up after d3c64aae8768d58781ee7e358bd7f3d8e0eb836d
- Document the change at docs/CHANGELOG.md
- Add `Reading rules from object storage` section to docs/vmalert.md
- Add `s3` prefix to command-line flags related to the configuration of s3 and gcs clients
- Explicitly mention that reading rules from object storage is supported only in enterprise version
2023-02-09 19:10:36 -08:00
Aliaksandr Valialkin
6ee15f94f8
docs/CHANGELOG.md: document d621d50d4fb3b43a0bcb4419bee979f0192d38fe 2023-02-09 14:40:15 -08:00
Aliaksandr Valialkin
da9d1fea10
docs/CHANGELOG.md: cut v1.87.1 and mark 1.87.x as LTS release 2023-02-09 11:21:04 -08:00
Zakhar Bessarab
bbf663bd04
lib/promscrape: fix cancelling in-flight scrape requests during configuration reload (#3791)
* lib/promscrape: fix cancelling in-flight scrape requests during configuration reload when using `streamParse` mode (see #3747)

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* Update docs/CHANGELOG.md

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-09 11:18:36 -08:00
Aliaksandr Valialkin
82b5fa2fd0
app/vmui: UX enhancements for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3706
- Display `min` value additionally to `avg`, `max` and `last`
- Allow copy-n-pasting metric name with its labels from both legend and tooltup
2023-02-09 11:05:57 -08:00
Aliaksandr Valialkin
dc7b92bc34
docs/vmagent.md: clarify that automatically generated metrics contain all the target-specific labels, including instance and job 2023-02-09 11:05:57 -08:00
Air
fe6e1015fe
Possibly spelling in the Quick start 2023-02-09 11:05:57 -08:00
Yury Molodov
5a7c63a324
vmui: improve tenant selector (#3794)
* fix: change styles tenant selector (#3792)

* docs/CHANGELOG.md: document the change

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3792

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-09 00:09:36 -08:00
Aliaksandr Valialkin
cd78396adb
docs/CHANGELOG.md: document changes at v1.79.8 LTS release 2023-02-08 23:39:01 -08:00
Yury Molodov
54bfd22ec5
vmui: add last/max/avg values (#3789)
* feat: add last/max/avg values (#3706)

* fix: change filter exclude values

* app/vmui: wip

- improve the visualization for avg/max/last values
- make getAvgFromArray() function resilient against inf/undefined/nil
- export getLastFromArray() function, which is resilient against inf/undefined/nil
- run `make vmui-update`

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-08 22:42:05 -08:00
Aliaksandr Valialkin
cfca7075c4
docs/CHANGELOG.md: document 75bcf86a31
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3740
2023-02-08 11:24:29 -08:00
Aliaksandr Valialkin
8d359b549e
docs/CHANGELOG.md: add more context to the bugfix description in Nomad service discovery
See 146fd2eca3
2023-02-08 09:24:51 -08:00
Aliaksandr Valialkin
b3bfca16de
docs/CHANGELOG.md: document the change at 67b01329a0
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3761
2023-02-08 08:46:21 -08:00
Roman Khavronenko
4e922eb93b
Vmalert fixes (#3788)
* vmalert: use group's ID in UI to avoid collisions

Identical group names are allowed. So we should used IDs
for various groupings and aggregations in UI.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: prevent disabling state updates tracking

The minimum number of update states to track is now set to 1.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: properly update `debug` and `update_entries_limit` params on hot-reload

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: display `debug` field for rule in UI

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: exclude `updates` field from json marhsaling

This field isn't correctly marshaled right now.
And implementing the correct marshaling for it doesn't
seem right, since json representation is mostly used
by systems like Grafana. And Grafana doesn't expect this
field to be present.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* fix test for disabled state

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* fix test for disabled state

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-02-08 08:45:25 -08:00
Roman Khavronenko
d260b90445
docs: follow-up after 2e4bfcce63 (#3785)
2e4bfcce63

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-02-08 08:37:48 -08:00
earthgecko
e8924ebfba
Clarifications between standalone/cluster ingestion endpoints (#3771)
docs: clarifications between standalone/cluster ingestion endpoints

This is an attempt to make it a bit clearer to the user that the cluster version ingestion URLs are different from the standalone ones.  I have also changed the order of the list items to make it a bit clearer and hopefully stop the user simply inferring that `/prometheus/api/v1` is only related to Prometheus data.
2023-02-07 09:34:02 -08:00
Max Golionko
eaa3716507
docs: update formatiing for k8s monitoring with Managed VictoriaMetrics (#3768)
* jekyll formatting madness
2023-02-07 09:29:34 -08:00
Roman Khavronenko
80bf0bcf8c
vmalert: update docs (#3770)
vmalert: update flags description

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-02-07 09:28:59 -08:00
Aliaksandr Valialkin
d8a54ab839
docs/CHANGELOG.md: fix formatting fro the change from 6fd10e8871 2023-02-07 09:26:38 -08:00
Roman Khavronenko
96db7ac52c
vmalert: speed up state restore procedure on start (#3758)
* vmalert: speed up state restore procedure on start

Alerts state restore procedure has been changed to become asynchronous.
It doesn't block groups start anymore which significantly improves vmalert's startup time.
Instead, state restore is called by each group in their goroutines after the first rules
evaluation.

While previously state restore attempt was made for all loaded alerting rules,
now it is called only for alerts which became active after the first evaluation.
This reduces the amount of API calls to the configured remote read URL.

This also means that `remoteRead.ignoreRestoreErrors` command-line flag becomes deprecated now
and will have no effect if configured.

See relevant issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2608

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* make lint happy

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* Apply suggestions from code review

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-03 19:46:41 -08:00
Aliaksandr Valialkin
26614da548
docs/CHANGELOG.md: document f63f487787
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3707
2023-02-03 19:30:40 -08:00
Aliaksandr Valialkin
6910dd2a98
docs/MetricsQL.md: add links to "rollup results" explanation 2023-02-03 11:10:49 -08:00
Aliaksandr Valialkin
06c3e303ec
docs/CHANGELOG.md: remove duplicate line after b7cc0bfe3c 2023-02-03 10:48:54 -08:00
Aliaksandr Valialkin
ee800d6336
docs/managed-victoriametrics/how-to-monitor-k8s.md: rename image files according to docs/assets/README.md 2023-02-03 10:46:49 -08:00
Max Golionko
2d419021dc
docs: move managed victoria metics guide into right folder (#3750)
* move guide folder
* image width control
2023-02-03 10:46:26 -08:00
Max Golionko
1d528f3ed6
Update docs/managed_victoriametrics/how-to-monitor-k8s.md
Co-authored-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-02-03 10:45:46 -08:00
Max Golionko
76d339b886
add guide to list of guides 2023-02-03 10:45:01 -08:00
Max Golionko
32b283d9c3
added k8s guide for managed VM 2023-02-03 10:43:45 -08:00
Aliaksandr Valialkin
f737674912
docs/CHANGELOG.md: typo fixes 2023-02-01 20:40:55 -08:00
Aliaksandr Valialkin
00a6ec85c6
docs/CHANGELOG.md: cut v1.87.0 2023-02-01 13:03:27 -08:00
Dmytro Kozlov
b7cc0bfe3c
app/vmctl: Add insecure skip verify flag for remote read protocol (#3611)
* app/vmctl: Add insecure skip verify flag for remote read protocol
2023-02-01 12:45:11 -08:00
Dmytro Kozlov
3c1e455805
lib/promscrape: fix honor_labels behavior (#3739)
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-02-01 12:02:04 -08:00
Aliaksandr Valialkin
03145842d7
docs/CHANGELOG.md: document 9254e494f9 2023-02-01 09:57:04 -08:00
Zakhar Bessarab
326a702559
app/vmbackup: fix deleting snapshot after backup completion (#3735) (#3736)
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-02-01 08:58:11 -08:00
Zakhar Bessarab
626bd22157
fix: vmselect multi-level setup panic (#3738)
* app/vmselect/netstorage: fix panic for multi-level cluster setup when `replicationFactor` was set and request contained `trace` parameter (#3734)

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vmselect/netstorage: use correct context for retry

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-02-01 08:56:36 -08:00
Aliaksandr Valialkin
e628289df4
docs/CHANGELOG.md: document 442a9f16b4
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3661
2023-01-31 13:04:20 -08:00
Roman Khavronenko
d93ac2b1ea
docs: mention -vmalert.proxyURL in vmalert docs (#3730)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-01-31 10:49:49 -08:00
Aliaksandr Valialkin
855d560789
lib/promscrape/discovery/azure: add __meta_azure_machine_size label in the same way as Prometheus does
See https://github.com/prometheus/prometheus/pull/11650
2023-01-27 17:07:57 -08:00
Aliaksandr Valialkin
134f7622d6
lib/promscrape/discovery/kubernetes: add support for __meta_kubernetes_pod_container_id
See https://github.com/prometheus/prometheus/issues/11843
and https://github.com/prometheus/prometheus/pull/11844
2023-01-27 16:33:57 -08:00
Yury Molodov
730025d1dc
vmui: add select of Tenant ID (#3673)
* feat: add select of tenantID

* feat: replace tenantID to default url

* fix: move the tenantID selector to the top header

* fix: hide tenantID selector by condition

* fix: correct z-index

* app/vmselect/vmui: `make vmui-update`

---------

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-01-27 15:54:09 -08:00
Aliaksandr Valialkin
1df4f63bc8
docs: update security chapters after bd716d1b0c 2023-01-27 15:44:49 -08:00
Denys Holius
1259a931c8
Improving docs by adding additional security sections (#3713)
* docs/Cluster-VictoriaMetrics.md: adds security section

* docs/Quick-Start.md: adds Security recommendation section
2023-01-27 15:41:16 -08:00
Aliaksandr Valialkin
cd25945ff6
app/vmagent: properly return 200 response code when importing data via Prometheus PushGateway protocol
This is the same fix as has been already applied to app/vminsert at cdb6d651e9

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3636
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1415
2023-01-27 14:40:26 -08:00
Aliaksandr Valialkin
8838c07360
docs/vmauth.md: update docs after ff39a91147
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3346
2023-01-27 14:10:29 -08:00
Aliaksandr Valialkin
18bf18b8b9
app/vmauth: limit the number of concurrent requests served by vmauth with the -maxConcurrentRequests command-line flag
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3346

This commit is based on the https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3486
2023-01-27 14:07:53 -08:00
Aliaksandr Valialkin
27ec56ffb1
app/vmauth: do not use net/http/httputil.ReverseProxy
This allows better controlling requests to backends and providing better error logging.
For example, if the backend was unavailable, then the ReverseProxy was logging the error
message without client ip and the initial request uri. This could harden debugging.

This is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3486
2023-01-27 13:40:53 -08:00
Aliaksandr Valialkin
ae2b9f9204
docs/managed-victoriametrics: typo fix in links to images 2023-01-27 11:36:29 -08:00
Aliaksandr Valialkin
c13e341d16
docs/assets/README.md: mention that locally placed doc-specific images simplify referring them from various views without the need to deal with folder prefixes 2023-01-27 11:31:49 -08:00
Aliaksandr Valialkin
8bde11e1a3
docs/managed-victoriametrics/user-management.md: move the associated images to docs/managed-victoriametrics/ folder with user-management_ prefix according to docs/assets/README.md 2023-01-27 11:31:49 -08:00
Aliaksandr Valialkin
955e1c1729
docs/managed-victoriametrics/quickstart.md: move the associated images to docs/managed-victoriametrics/ folder with quickstart_ prefix according to docs/assets/README.md 2023-01-27 11:31:49 -08:00
Aliaksandr Valialkin
b024981bed
docs/assets: add README.md with the explanation on which files can be put into the docs/assets folder 2023-01-27 11:31:49 -08:00
Aliaksandr Valialkin
c06ec450ee
docs/Cluster-VictoriaMetrics.md: move Naive_cluster_scheme.png from the docs/assets/images/ folder into docs/ folder and add Cluster-VictoriaMetrics_ prefix to the image name
The docs/assets folder should be used only for assets specific to docs generation at https://docs.victoriametrics.com, e.g. css, js and images.

All the other assets related to specific docs should be placed in the same folder as the corresponding *.md file.
These assets should have the same name prefix as the corresponding doc file name. This simplifies tracking the lifetime of these assets.
For example, if the doc is removed, it is very easy to remove all assets associated with it with a simple `rm -rf docs/doc-name*` command.

This also simplifies generating correct urls for doc-specific assets from both https://docs.victoriametrics.com
and from https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/ - just refer to the asset name without any directory prefixes.
2023-01-27 11:31:21 -08:00
dmitryk-dk
7cf9752a7f
docs: move how to register from dbaas to docs 2023-01-27 11:30:45 -08:00
dmitryk-dk
0d81aff91d
docs: use absolute path 2023-01-27 11:29:54 -08:00
dmitryk-dk
6ed4e70a00
docs: add documentation of user management on managed-vm 2023-01-27 11:26:36 -08:00
Aliaksandr Valialkin
9e8cfd23ad
docs/Cluster-VictoriaMetrics.md: update command-line descriptions after ebebaecd94 2023-01-27 00:04:54 -08:00
Aliaksandr Valialkin
4cf4c307ea
docs: update command-line descriptions after 73256fe438 2023-01-27 00:01:14 -08:00
Nikolay
ebebaecd94
lib/netutil: init implimentation of proxy protocol (#3687)
* lib/netutil: init implimentation of proxy protocol
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-01-26 23:25:22 -08:00
Aliaksandr Valialkin
e8ee219fba
docs/CHANGELOG.md: make the description for the bugfix from 465a285324 more reader-friendly 2023-01-26 10:08:19 -08:00
Nikolay
4af05065d1
lib/storage: properly release parts inMerge lock (#3711)
if storage doesn't have enough disk space, finalDedupWatcher holds inMerge lock for all parts and never release it until storage restart
2023-01-26 08:57:36 -08:00
Roman Khavronenko
29ea196bf0
docs: specify the time window for series_limit (#3708)
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-01-25 09:31:09 -08:00
Aliaksandr Valialkin
bd809db4d9
docs: update the list of command-line flags according to the latest changes 2023-01-25 09:22:23 -08:00
Aliaksandr Valialkin
5defa99a2e
lib/streamaggr: add ability to de-duplicate input samples before aggregation 2023-01-25 09:22:03 -08:00
Yury Molodov
29fd95d426
vmui: include fonts in its bundle (#3705)
* feat: include fonts in the build

* fix: reduce size fonts

* wip

- Document the change at docs/CHANGELOG.md
- Run `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-01-24 09:31:37 -08:00
Yury Molodov
97aa9b03de
vmui: improvements to the UI styles (#3704)
* feat: add dark theme

* update packages

* feat: add multilevel menu (#3678)

* fix: correct styles

* fix: update link to cardinality-explorer

* fix: remove unused scss variables

* docs/CHANGELOG.md: document the changes

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-01-24 09:21:20 -08:00
Roman Khavronenko
c24c2d20d4
discover/ec2: follow-up after e2b4ab8384 (#3703)
Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-01-24 09:10:46 -08:00
Aliaksandr Valialkin
0698467ae5
lib/bytesutil: do not intern long strings, since they may need big amounts of additional memory for the cache
Allow users fine-tuning the maximum string length for interning via -internStringMaxLen command-line flag.
This may be used for fine-tuning RAM vs CPU usage for certain workloads.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3692
2023-01-23 23:37:08 -08:00
Aliaksandr Valialkin
71a170d404
lib/promscrape: follow-up for 393876e52a
- Document the change in docs/CHANGELOG.md
- Reduce memory usage when sending stale markers even more by parsing the response in stream parsing mode
- Update the TestSendStaleSeries

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3668
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3675
2023-01-23 21:56:18 -08:00
Denys Holius
0f25a77acd
Fix/remove vmanomaly from release guide (#3699)
* docs/Release-Guide.md: remove vmanomaly from release guide because it has own release cycle

* fixed a typo
2023-01-23 08:08:02 -08:00
Aliaksandr Valialkin
f6d4fde45b
docs/Articles.md: added missing third-party articles about VictoriaMetrics 2023-01-22 14:23:55 -08:00
Aliaksandr Valialkin
7d3d664eb8
docs/Single-server-VictoriaMetrics.md: make it clear that VictoriaMetrics supports both pull and push protocols at how to import time series data chapter 2023-01-22 14:05:15 -08:00
Aliaksandr Valialkin
7f35d81cf4
docs/Articles.md: add https://dev.to/aws-builders/ultra-monitoring-with-victoria-metrics-1p2 2023-01-22 13:52:13 -08:00
Aliaksandr Valialkin
d8329e47cf
lib/vmselectapi: propagate timeout errors from vmselect to vmstorage instead of closing the connection established from vmselect to vmstorage
This is a follow-up for 20e9598254
2023-01-20 19:30:22 -08:00
Aliaksandr Valialkin
b046af8a4d
app/vmselect: make vmui-update after df7b81b44d 2023-01-20 12:07:29 -08:00
Yury Molodov
57f7ffca5b
vmui: add support for time zone selection for older versions of browsers (#3680)
* fix: add check for support of getting time zones

* vmui: add support for time zone selection for older versions of browsers
2023-01-20 12:07:28 -08:00
Denys Holius
4429e7e53e
Adds some improvements to release guide docs (#3679)
* docs/Release-Guide.md: fixed a typo

* Release-Guide.md: adds missed steps for updating vmanomaly and vmgateway helm charts
2023-01-19 10:01:54 -08:00
Aliaksandr Valialkin
069042653c
docs/CHANGELOG.md: cut v1.86.2 2023-01-18 12:01:14 -08:00
Aliaksandr Valialkin
cd55e82208
docs/CHANGELOG.md: document 777038fe44
See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3672
2023-01-18 11:40:53 -08:00
Aliaksandr Valialkin
a844b97942
lib/promscrape: follow-up for d79f1b106c
- Document the fix at docs/CHANGELOG.md
- Limit the concurrency for sendStaleMarkers() function in order to limit its memory usage
  when big number of targets disappear and staleness markers are sent
  for all the metrics exposed by these targets.
- Make sure that the writeRequestCtx is returned to the pool
  when there is no need to send staleness markers.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3668
2023-01-17 23:13:08 -08:00
Zakhar Bessarab
40d524edb8
discovery/{consul,nomad}: fix cancelling serviceWatcher in-flight requests (#3658)
* lib/promscrape/discovery/{consul,nomad}: fix background service update watches not canceling requests on serviceWatcher stop

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/promscrape/discovery/{consul,nomad}: fix closing serviseWatcher during scrape job restart

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* wip

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-01-17 21:47:51 -08:00
Yury Molodov
0561ba3557
vmui: correctly display range results in Table view (#3657)
* fix: properly display range results

* fix: set range values to empty array

* wip

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-01-17 21:04:50 -08:00
Aliaksandr Valialkin
d241a71baa
docs/CHANGELOG.md: fix the link to feature request implemented at e58921aa8f
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3571
2023-01-17 20:28:04 -08:00
Yury Molodov
6e033b93b0
vmui: give more visually different colors to graph lines (#3656)
* feat: make more different colors of graph lines

* docs/CHANGELOG.md: give more visually different colors to graph lines
2023-01-17 20:26:23 -08:00
Aliaksandr Valialkin
42fa6cee19
vendor: update github.com/VictoriaMetrics/metricsql from v0.51.1 to v0.51.2
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3664
2023-01-17 11:26:11 -08:00
Aliaksandr Valialkin
c33728befb
lib/promscrape: properly apply series limit
Fix the following issues:

- Series limit wasn't applied when staleness tracking was disabled.
- Series limit didn't prevent from sending staleness markers for new series exceeding the limit.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3660

Thanks to @hagen1778 for the initial attempt to fix the issue
at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3665
2023-01-17 10:30:16 -08:00
Aliaksandr Valialkin
e370daecda
docs/CHANGELOG.md: document the 74f196c37813a18c2c8f28831696ed7d9f535604 2023-01-17 09:10:51 -08:00
Aliaksandr Valialkin
103dfd0525
lib/{mergeset,storage}: do not slow down concurrently executed queries during assisted merges
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3647
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3641
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/291
2023-01-16 14:45:40 -08:00
Yury Molodov
060780af69
vmui: make the step input field global across all the tabs and views (#3644)
* feat: make the step input field global

* fix: correct get step from url

* fix: set minimumSignificantDigits to 1

* app/vmselect/vmui: `make vmui-update`

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-01-15 13:47:48 -08:00
Aliaksandr Valialkin
556484a52f
app/vminsert: return 200 OK status code when importing data in pushgateway format
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1415
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3636
2023-01-15 13:26:09 -08:00
Denys Holius
41ea90e6a2
guides/guide-delete-or-replace-metrics.md: fixed wrong curl command (#3652) 2023-01-15 13:03:32 -08:00
Corporte Gadfly
5815a98957
docs: fix typo (#3648) 2023-01-15 13:01:23 -08:00
Aliaksandr Valialkin
26f6cfd3b2
app/vmselect/netstorage: tune the number of blocks per series which should be unpacked by a single goroutine instead of spinning up multiple goroutines
This reduces overhead on time series data unpacking for typical cases,
this reducing CPU usage at vmselect

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3641
2023-01-12 09:35:15 -08:00
Aliaksandr Valialkin
b735d49625
docs/CHANGELOG.md: update the description of the change at 20f28eb9d6 2023-01-12 09:00:07 -08:00
Nikolay
43d1f2d0c4
/lib/promscrape: use correct err logger for scrape unmarshalling (#3645)
/lib/promscrape: use correct err logger for scrape unmarshalling
It correctly suppresses scrape errors and adds correct context for err msg
2023-01-12 09:00:06 -08:00
Aliaksandr Valialkin
a6988eb8c3
docs/Cluster-VictoriaMetrics.md: mention the -vmui.customDashboardsPath command-line flag at vmselect
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3322
2023-01-11 23:39:18 -08:00
Aliaksandr Valialkin
820357f434
app/vmselect: follow-up after 820312a2b1
- Move the feature description at the correct place at docs/CHANGELOG.md
- Run `make vmui-update`
- Various cosmetic fixes

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3322
2023-01-11 23:37:51 -08:00
Dmytro Kozlov
f2862f088b
app/vmui: define custom path for dashboards json file (#3545)
* app/vmui: define custom path for dashboards json file

* app/vmui: remove unneeded code

* app/vmui: move handler to own file, fix show dashboards,

* app/vmui: move flag to handler, add flag description

* app/vmauth: fix part of the comments

* feat: add store for dashboards

* fix: prevent fetch dashboards for app mode

* app/vmauth: use simple cache for predefined dashboards

* app/vmauth: update dashboards doc

* app/vmauth: fix ci

* app/vmui: decrease timeout

* app/vmselect: removed cache, fix comments

* app/vmselect: remove unused const

* app/vmselect: fix error log, use slice byte instead of struct

Co-authored-by: Yury Moladau <yurymolodov@gmail.com>
2023-01-11 23:33:30 -08:00
Zakhar Bessarab
ae5b85966a
lib/promscrape/discovery/dockerswarm: fix discovery filters being applied to all objects (#3632)
* lib/promscrape/discovery/dockerswarm: fix discovery filters being applied to all objects

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* Update docs/CHANGELOG.md

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-01-11 22:56:40 -08:00
Artem Navoiev
efec0f150f
fix operator docs hugo schema
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-01-11 17:00:14 -08:00
Artem Navoiev
8d238f1bc2
docs: prepare operator docs to migration
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-01-11 16:59:52 -08:00
Artem Navoiev
3991c977af
docs: operator *.MD -> *.md
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-01-11 16:59:24 -08:00
Aliaksandr Valialkin
23ed835043
docs/CHANGELOG.md: cut v1.86.1 2023-01-11 01:27:01 -08:00
Artem Navoiev
e91814e332
vmagent: add minimal scrape file exampe for vmagent quick start (#3627)
* vmagent: add minimal scrape file exampe for vmagent quick start

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

* replace example with link to your prometheus.yml in docker

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-01-11 01:23:07 -08:00
Aliaksandr Valialkin
af58ac25f6
lib/vmselectapi: properly calculate query timeout
vmselect passes query timeout to vmstorage in seconds.
The commit 20e9598254 treated it as timeout in nanoseconds.
Fix this in order to prevent from the following errors under vmstorage load:

cannot process vmselect request: cannot execute "search_v7": couldn't start executing the request in 0.000 seconds,
since -search.maxConcurrentRequests=... concurrent requests are already executed.
2023-01-11 01:21:55 -08:00
Aliaksandr Valialkin
3d22532bb8
app/vmselect/promql: make a copy of per-series timestamps before their modification
The per-series timestamps are usually shared among series, so it is unsafe modifying them.

The issue has been appeared after the optimization at 2f3ddd4884
2023-01-11 00:59:02 -08:00
Aliaksandr Valialkin
d1a62af3bd
docs/CHANGELOG.md: document v1.79.7 LTS release 2023-01-10 21:24:44 -08:00
Aliaksandr Valialkin
e51d28aae2
docs/CHANGELOG.md: add release date for v1.86.0 2023-01-10 17:46:13 -08:00
Aliaksandr Valialkin
a2488f9e24
docs/CHANGELOG.md: cut v1.86.0 2023-01-10 16:22:53 -08:00
Zakhar Bessarab
10f314cdbd
Use httpAuth.* flags as a fallback for endpoints protected by *AuthKey flags (#3582)
* {lib/server, app/}: use `httpAuth.*` flag as fallback for `*AuthKey` if it is not set

* lib/ingestserver/opentsdbhttp: fix opentdb HTTP handler not respecting `httpAuth.*` flags

* Apply suggestions from code review

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-01-10 15:57:55 -08:00
Aliaksandr Valialkin
ab318660cd
lib/promscrape/discovery/gce: follow-up for b2ccdaaa2f
- Use promutils.Labels.GetLabels() instead of comparing promutils.Labels.Labels to nil.
  This make the code more consistent with other places.

- Mention the release where the issue has been introduced at docs/CHANGELOG.md.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3624
2023-01-10 13:51:57 -08:00
Zakhar Bessarab
02f5c16433
lib/promscrape/discovery/gce: fix crash in case instance does not have any labels set (#3625)
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-01-10 13:51:35 -08:00
Denys Holius
ceea223caa
docs/Release-Guide.md: added missed link to rpm repository (#3623) 2023-01-10 13:43:45 -08:00
Aliaksandr Valialkin
43a4dcdaf8
lib/promscrape/discovery/nomad: sync nomad_sd_configs fields with the Prometheus implementation
See the list of configs supported by Prometheus at f88a0a7d83/discovery/nomad/nomad.go (L76-L84)

- Removed "token" option. In can be set either via NOMAD_TOKEN env var or via `bearer_token` config option.
- Removed "scheme" option. It is automatically detected depending on whether the `tls_config` is set.
- Removed "services" and "tags" options, since they aren't supported by Prometheus.
- Added "region" option. If it is missing, then the region is read from NOMAD_REGION env var.
  If this var is empty, then it is set to "global" in the same way as Nomad client does.
  See 865ee8d37c/api/api.go (L297)
  and 865ee8d37c/api/api.go (L555-L556)
- If the "server" option is missing, then it is read from NOMAD_ADDR in the same way
  as Nomad client does - see 865ee8d37c/api/api.go (L294-L296)

This is a follow-up for 8aee209c53

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367
2023-01-09 21:30:19 -08:00
Roman Khavronenko
ca5136a0ee
lib/promscrape: remove datacenter field from nomad_sd_config (#3612)
Looks like `datacenter` field isn't part of `/v1/services` API.
See https://developer.hashicorp.com/nomad/api-docs/services#list-services
and https://developer.hashicorp.com/nomad/api-docs/services#read-service

Related issues:
https://github.com/traefik/traefik/issues/9109
https://github.com/prometheus/prometheus/issues/11776

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-01-09 21:24:46 -08:00
Aliaksandr Valialkin
96f04c9863
app/vmselect/netstorage: consistently select the sample with the biggest value out of samples with identical timestamps
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3333

This fix is based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3620 ,
but doesn't slow down the common case with merging replicated data blocks so significantly.

Benchmark results:

Before the change:

BenchmarkMergeSortBlocks/replicationFactor-1-4         	   13968	     85643 ns/op	 956.53 MB/s	    1700 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-2-4         	   10806	    109171 ns/op	1500.77 MB/s	    2191 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-3-4         	    8887	    130623 ns/op	1881.45 MB/s	    2660 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-4-4         	    7440	    157348 ns/op	2082.52 MB/s	    3174 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-5-4         	    6534	    184473 ns/op	2220.38 MB/s	    3612 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/overlapped-blocks-bestcase-4  	   13419	     85205 ns/op	 961.44 MB/s	    2213 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/overlapped-blocks-worstcase-4 	     579	   1894900 ns/op	  43.23 MB/s	   46760 B/op	       1 allocs/op

After the change:

BenchmarkMergeSortBlocks/replicationFactor-1-4         	   13832	     85298 ns/op	 960.40 MB/s	    1716 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-2-4         	    8833	    134222 ns/op	1220.66 MB/s	    2675 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-3-4         	    6487	    184830 ns/op	1329.65 MB/s	    3636 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-4-4         	    4977	    236318 ns/op	1386.61 MB/s	    4733 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/replicationFactor-5-4         	    4088	    296734 ns/op	1380.36 MB/s	    5761 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/overlapped-blocks-bestcase-4  	   14083	     84067 ns/op	 974.47 MB/s	    2110 B/op	       1 allocs/op
BenchmarkMergeSortBlocks/overlapped-blocks-worstcase-4 	     536	   2043534 ns/op	  40.09 MB/s	   50511 B/op	       1 allocs/op
2023-01-09 12:58:18 -08:00
Dmytro Kozlov
f739e44802
lib/storage: fix returning camelcase label names (#3608)
* lib/storage: fix returning camelcase label names

* doc: add change log

* Update docs/CHANGELOG.md

* Update docs/CHANGELOG.md

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-01-07 01:11:10 -08:00
Aliaksandr Valialkin
5876821a16
all: small improvements in error messages and command-line flag descriptions related to concurrency limiters 2023-01-07 00:12:24 -08:00
Aliaksandr Valialkin
7fb02f536a
lib/promscrape: limit the concurrency during parsing and relabeling the scraped samples
This should reduce memory usage when scraping big number of targets,
since this limits the summary memory usage during concurrent parsing and relabeling
by the number of available CPU cores.
2023-01-06 23:01:18 -08:00
Aliaksandr Valialkin
b275983403
lib/writeconcurrencylimiter: improve the logic behind -maxConcurrentInserts limit
Previously the -maxConcurrentInserts was limiting the number of established client connections,
which write data to VictoriaMetrics. Some of these connections could be idle.
Such connections do not consume big amounts of CPU and RAM, so there is a little sense in limiting
the number of such connections. So now the -maxConcurrentInserts command-line option
limits the number of concurrently executed insert requests, not including idle connections.

It is recommended removing -maxConcurrentInserts command-line option, since the default value
for this option should work good for most cases.
2023-01-06 22:07:16 -08:00
Aliaksandr Valialkin
20e9598254
lib/vmselectapi: limit the number of concurrently executed requests
This should prevent from out of memory errors when big number of vmselect
nodes send many concurrent requests to vmstorage

The limit can be controlled at vmstorage via the following command-line flags:
- search.maxConcurrentRequests
- search.maxQueueDuration

See https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#resource-usage-limits
2023-01-06 18:39:46 -08:00
Aliaksandr Valialkin
54410bf51b
lib/promscrape: follow-up after bced9fb978
- Document the bugfix at docs/CHANGELOG.md
- Wait until all the worker goroutines are done in consulWatcher.mustStop()
- Do not log `context canceled` errors when discovering consul serviceNames
- Removed explicit handling of gzipped responses at lib/promscrape/discoveryutils.Client,
  since this handling is automatically performed by net/http.Transport.
  See DisableCompression option at https://pkg.go.dev/net/http#Transport .
- Remove explicit handling of the proxyURL, since it is automatically handled
  by net/http.Transport. See Proxy option at https://pkg.go.dev/net/http#Transport .
- Expliticly set MaxIdleConnsPerHost, since its default value equals to 2.
  Such a small value may result in excess tcp connection churn
  when more than 2 concurrent requests are processed by lib/promscrape/discoveryutils.Client.
- Do not set explicitly the `Host` request header, since it is automatically set by net/http.Client.
- Backport the bugfix to the recently added nomad_sd_configs - see https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468
2023-01-05 21:23:21 -08:00
Aliaksandr Valialkin
750d309f63
lib/promscrape/discovery/nomad: follow-up after 48f371a46c
- Remove undocumented `username` and `password` config options from `nomad_sd_config`.
  TODO: probably, remove these options from `consul_sd_config` too?
  These options exist there for backwards compatibility purposes.

- Add __meta_nomad_service_alloc_id and __meta_nomad_service_job_id meta-labels
  These labels contain AllocID and JobID fields for the discovered Nomad services.

- Various typo fixes.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3367
2023-01-05 18:09:23 -08:00
Karan Sharma
8f42c5a024
lib/promscrape: add Prometheus-compatible service discovery for Nomad (#3549)
Add nomad_sd_config support for service discovery
2023-01-05 18:07:02 -08:00
Zakhar Bessarab
eee7e75278
doc: add vmbackupmanager monitoring section (#3605)
* doc: add vmbackupmanager monitoring section

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-01-05 16:49:59 -08:00
Artem Navoiev
9d85c62ce2
Add Understand Your Setup Size Guide (#3572)
docs: add Understand Your Setup Size Guide

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-01-05 16:48:28 -08:00
Aliaksandr Valialkin
21be927afa
docs/CHANGELOG.md: add missing dot 2023-01-05 03:35:20 -08:00
Zakhar Bessarab
99f9b02283
lib/promscrape/discovery/dockerswarm: fix query encoding of filters (#3586)
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-01-05 03:35:18 -08:00
Aliaksandr Valialkin
1d16cc9349
lib/promscrape: pre-fetch metric_relabel_configs rules when debugging metric relabeling for a particular target
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3407
2023-01-05 03:28:14 -08:00
Aliaksandr Valialkin
634e24e685
lib/promscrape: follow-up for a7e29c38bc
- Document the bugfix at docs/CHANGELOG.md
- Make the fix more durable against future changes when droppedTargetsMap.Register may be called from other places.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3580
2023-01-05 02:51:45 -08:00
Aliaksandr Valialkin
c97d6ed6a4
lib/streamaggr: sort by and without labels in the aggregate output metric name
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3460
2023-01-05 02:08:59 -08:00
Aliaksandr Valialkin
78114e85d6
vendor: update github.com/VictoriaMetrics/metricsql from v0.50.0 to v0.51.0
Updates https://github.com/VictoriaMetrics/metricsql/pull/7
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3589
2023-01-05 01:50:26 -08:00
Denys Holius
28649f0773
docs/Release-Guide.md: adds release scenario for RPM LTS packages (#3588) 2023-01-04 08:39:10 -08:00
Aliaksandr Valialkin
ac890b3081
docs: update -help outputs for vm* tools 2023-01-03 23:27:31 -08:00
Aliaksandr Valialkin
3369371636
app/{vmagent,vminsert}: add support for streaming aggregation
See https://docs.victoriametrics.com/stream-aggregation.html

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3460
2023-01-03 22:22:07 -08:00
Aliaksandr Valialkin
b49927bcb1
docs/Single-server-VictoriaMetrics.md: impromve formatting for prominent features chapter 2023-01-03 21:58:38 -08:00
Aliaksandr Valialkin
ff3e8d096d
docs/vmbackupmanager.md: run make docs-sync after fa842d6534 2023-01-03 21:53:57 -08:00
Roman Khavronenko
dde750e7c1
vmalert: mention specifics of Alertmanager HA mode (#3573)
Stress the importance of specifying of all Alertmanager
URLs in vmalert's `-notifier.url` or `notifier.config`
if it runs in cluster mode.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3547

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-01-03 21:46:00 -08:00
Aliaksandr Valialkin
c03adc2bce
docs/CHANGELOG.md: document 1720bddb4f
Updats https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3530
2022-12-29 12:32:05 -08:00
Roman Khavronenko
c22122212d
csvimport: support empty values (#3565)
Before, if the imported line contained multiple metrics and one
or more of them had an empty values - the whole line was ignored.

Now, only metrics with empty values are ignored, and the rest
of the metrics are accepted successfully.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3540

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-12-29 11:56:02 -08:00
Aliaksandr Valialkin
a6d4596d84
docs/CHANGELOG.md: add a link to the docs and the pull request for update_entries_limit option in vmalert
This is a follow up for 6588fcbfca
2022-12-29 10:41:51 -08:00
Aliaksandr Valialkin
5e87ebabc7
vendor: update github.com/valyala/fastjson/fastfloat from v1.6.3 to v1.6.4
This should properly parse floating-point numbers with missing integer or fractional parts.
For example, 123. or .123

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3544
2022-12-29 10:41:51 -08:00
Roman Khavronenko
5cf2998af8
vmalert: allow configuring the default number of stored rule's update states (#3556)
Allow configuring the default number of stored rule's update states in memory
 via global `-rule.updateEntriesLimit` command-line flag or per-rule via rule's
 `update_entries_limit` configuration param.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-29 10:41:51 -08:00
Aliaksandr Valialkin
f27bb19213
app/vmselect/searchutils: accept partial RFC3339 values at time, start and end query args
This simplifies manual usage of the APIs. For example, the following query
would return the results over the 2022 year.

  /api/v1/query_range?start=2022&end=2023&step=1d&query=...

This is equivalent to:

  /api/v1/query_range?start=2022-01-01T00:00:00Z&end=2023-01-01T00:00:00Z&step=1d&query=...
2022-12-28 19:46:21 -08:00
Aliaksandr Valialkin
824fd70ec2
docs: remove a case study for Dreamteam.gg, since it looks like the company no longer exists 2022-12-28 11:45:38 -08:00
Artem Navoiev
1ae98867a5
fix broken links
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2022-12-28 11:30:53 -08:00
Artem Navoiev
393f4ab86f
update links to grafana dashboards (#3534)
docs: update links to grafana dashboards

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2022-12-28 11:22:02 -08:00
Aliaksandr Valialkin
182e62acde
app/vmui: small improvements for header panel
- Rename `Custom panel` tab to more clear `Query` tab
- Rename `Cardinality` tab to `Explore cardinality`, so it becomes consistent with `Explore metrics` tab
- Move `Dashboards` tab to the end, since it isn't used too much
2022-12-23 22:26:22 -08:00
Aliaksandr Valialkin
8a1775f3a6
app/vmui: follow-up after f6d31f5216
- Document the feature at docs/CHANGELOG.md.
- Document the metrics explorer at https://docs.victoriametrics.com/#metrics-explorer .
- Properly set `start` and `end` args for the selected time range
  when performing the request, which returns metric names.
- Improve queries, so they return lower number of lines and labels.
  This should improve metrics' exploration.
- Properly encode label filters and query args before passing them to VictoriaMetrics.
- Various cosmetic fixes.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3386
2022-12-22 17:17:58 -08:00
Yury Molodov
ec2b24f3f3
vmui: add explore tab for exploration of metrics, which belong to a particular job/instance (#3470)
* feat: add "Explore" page

* feat: add graphs for explore page

* vmui: add explore tab for exploration of metrics, which belong to a particular job/instance

* refactor: rename variables

* refactor: extract graph to ExploreMetricItemGraph.tsx

* feat: add searchable for Select.tsx

* feat: improve metrics explorer

* feat: set document title by page

* feat: add page to view icons

* fix: improve styles

* fix: add encodeURIComponent to query
2022-12-22 17:17:47 -08:00
Aliaksandr Valialkin
3834602c9d
docs/CHANGELOG.md: document 1df87807fd
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3513
2022-12-22 15:22:21 -08:00
Aliaksandr Valialkin
cc482b89a3
lib/promrelabel: add support for keepequal and dropequal relabeling actions
These actions are supported by Prometheus starting from v2.41.0

See https://github.com/prometheus/prometheus/pull/11564 ,
https://github.com/prometheus/prometheus/issues/11556
and https://github.com/prometheus/prometheus/issues/3756

Side note:

It's a pity that Prometheus developers decided inventing `keepequal` and `dropequal`
relabeling actions instead of adding support for `keep_if_equal` and `drop_if_equal` relabeling
actions supported by VictoriaMetrics since June 2020 - see 2a39ba639d .
2022-12-21 20:06:09 -08:00
Yury Molodov
6f17c1e866
vmui: fix change timezone (#3519)
vmui: fix time picker with changed time zone
2022-12-21 13:08:42 -08:00
Aliaksandr Valialkin
30b15111aa
docs/CHANGELOG.md: cut v1.85.3 2022-12-20 14:51:42 -08:00
Roman Khavronenko
998d89685f
vmagent: respect -usePromCompatibleNaming if no relabeling is set (#3511)
* vmagent: respect `-usePromCompatibleNaming` if no relabeling is set

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3493

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmagent: upd test

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2022-12-20 14:34:29 -08:00
Yury Molodov
88f408941c
fix: change the logic for hide query (#3514) 2022-12-20 14:30:21 -08:00
Zakhar Bessarab
decf46d72b
app/vmbackupmanager: add metrics for better observability (#488)
* app/vmbackupmanager: add metrics for better observability, include more information to `/api/v1/backups` API call response

* app/vmbackupmanager: drop old metrics before creating new ones

* app/vmbackupmanager: use `_total` postfix for counter metrics

* app/vmbackupmanager: remove `_total` postfix for gauge-like metrics

* app/vmbackupmanager: add `_last_run_failed` metrics for backups and retention

* app/vmbackupmanager: address review feedback

* app/vmbackupmanager: fix metric name

* app/vmbackupmanager: address review feedback, remove background updates of metrics, add restoring state of `_last_run_failed` metric from remote storage

* app/vmbackupmanager: improve performance for backup size calculation

* app/vmbackupmanager: refactor backup and retention runs to deduplicate each run logic

* {app/vmbackupmanager,lib/formatutil}: move HumanizeBytes into lib package

* app/vmbackupmanager: fix creating new metrics instead of reusing existing ones

* lit/formatutil: add comment to make linter happy

* app/vmbackupmanager: address review feedback
2022-12-20 14:18:43 -08:00
Aliaksandr Valialkin
1ff62629f4
lib/storage: clear the err if it is set to io.EOF when searching for the TSID by metricID
This is expected error after when recently added indexdb data isn't available for search yet
or wasn't flushed to disk after unclean shutdown of VictoriaMetrics.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3515
2022-12-20 14:05:53 -08:00
Roman Khavronenko
f2e7e11382
docs: fix link typo in operator docs (#3508) 2022-12-20 10:22:40 -08:00
Aliaksandr Valialkin
874aea98b3
docs/CHANGELOG.md: document the change at 547f07463b29c09c62c9af35eac9cee6764b3286
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2612
2022-12-20 10:21:22 -08:00
Aliaksandr Valialkin
276e867119
docs/CHANGELOG.md: formatting fix 2022-12-20 01:06:42 -08:00
Aliaksandr Valialkin
7fb314bb59
app/vmselect/promql: do not extend too short lookbehind window for rate() function if it is set explicitly
Previously too short lookbehind window d for rate(m[d]) could be automatically extended
if it didn't cover at least two raw samples. This was needed in order to guarantee
non-empty results from rate(m[d]) on short time ranges.

Now the lookbehind window isn't extended if it is set explicitly,
since it is expected that the user knows what he is doing.

The lookbehind window continues to be extended when needed if it isn't set explicitly.
For example, in the case of rate(m).

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3483
2022-12-20 00:18:32 -08:00
Aliaksandr Valialkin
4c6c8c50b5
docs/Articles.md: add a link to https://www.youtube.com/watch?v=Mesc6JBFNhQ 2022-12-19 21:41:02 -08:00
Aliaksandr Valialkin
1ed0d07aa7
docs/CHANGELOG.md: add a warning for releases between v1.83.0 and v1.85.1 that it is recommended upgrading to v1.85.2 because of the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3502
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3502
2022-12-19 13:37:20 -08:00
Aliaksandr Valialkin
cac56859eb
docs/CHANGELOG.md: consistently use YYYY-MM-DD format for release dates
The previously used DD-MM-YYYY format could be confused with the MM-DD-YYYY format.
The YYYY-MM-DD format reduces this confusion.
2022-12-19 13:33:09 -08:00
Aliaksandr Valialkin
371f9085c9
docs/CHANGELOG.md: cut v1.85.2 2022-12-19 13:09:48 -08:00
Aliaksandr Valialkin
9330da3195
lib/promscrape/discovery/consul: expose service tags in individual labels __meta_consul_tag_<tagname>
This simplifies copying service tags to target labels with the following relabeling rule:

- action: labelmap
  regex: __meta_consul_tag_(.+)

See https://stackoverflow.com/questions/44339461/relabeling-in-prometheus
2022-12-19 13:02:56 -08:00
Aliaksandr Valialkin
11bd290201
lib/storage: search for TSIDs for the given metricIDs in the previous indexdb if they aren't found in the current indexdb
The issue triggers after the indexdb rotation for time series, which stop receiving new samples.
This results in missing data for such time series in query responses.

This commit should address the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3502

The issue has been introduced in 2dd93449d8
2022-12-19 11:56:49 -08:00
Aliaksandr Valialkin
70c720d640
lib/logger: follow-up for 72f8fce107
- Document the change at docs/CHANELOG.md
- Log fatal errors if the -loggerJSONFields contains unexpected values
- Rename -loggerJsonFields to -loggerJSONFields for the sake of consistency naming commonly used in Go

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2348
2022-12-16 17:42:31 -08:00
Aliaksandr Valialkin
2a229a319e
docs/vmalert.md: mention latency_offset query arg, which has been added in 86dae56bd0
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3481
2022-12-16 17:20:51 -08:00
Aliaksandr Valialkin
8d7d8f0125
app/vmselect/prometheus: follow-up after 86dae56bd0
Return error if the provided latency_offset query arg cannot be parsed.
This should simplify debugging in production.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3481
2022-12-16 17:16:26 -08:00
Roman Khavronenko
f8a3514e99
vmselect: support overriding of -search.latencyOffset (#3489)
support overriding of `-search.latencyOffset` value via
URL param `latency_offset`.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3481

Signed-off-by: hagen1778 <roman@victoriametrics.com>

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2022-12-16 17:14:23 -08:00
Aliaksandr Valialkin
2e597555ec
lib/promscrape: stop dropping metric name if relabeling rules do not instruct to do this on the /metric-relabel-debug page 2022-12-16 16:44:12 -08:00
Aliaksandr Valialkin
c8942f819f
docs/keyConcepts.md: update the list of supported data ingestion protocols
- Add DataDog protocol
- Remove native protocol, since it isn't intended for general-purpose usage by external clients
2022-12-15 12:02:51 -08:00
Aliaksandr Valialkin
1a88fe5b1f
lib/flagutil/bytes.go: properly handle values bigger than 2GiB on 32-bit architectures
This fixes handling of values bigger than 2GiB for the following command-line flags:

- -storage.minFreeDiskSpaceBytes
- -remoteWrite.maxDiskUsagePerURL
2022-12-14 19:29:57 -08:00