Commit graph

2624 commits

Author SHA1 Message Date
Zakhar Bessarab
7385766805
app/vmselect/promql: add implementation of median_over_time for rollup functions list (#5042)
`median_over_time` is handled by predefined WITH template in MetricsQL library which translates it to `quantile_over_time(0.5)`
This makes it impossble to use `median_over_time` as a usual rollup function for `aggr_over_time`.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/5034

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-09-25 15:44:09 +02:00
Aliaksandr Valialkin
a14198ba97
app/vmselect/promql: fix tests after the upgrade of github.com/VictoriaMetrics/metricsql from v0.64.0 to v0.65.0 in bfbd0b478e 2023-09-19 01:03:57 +02:00
Konstantin
db57fbc5d6
app/vmselect: return +Inf as null in graphite render api (#5009)
Signed-off-by: Konstantin Kulikov <k.kulikov2@gmail.com>
2023-09-18 16:42:07 +02:00
Aliaksandr Valialkin
63ca7b62e5
app/vmagent/remotewrite: fix data race when extra labels are added to samples before sending them to multiple remote storage systems
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4972
2023-09-09 06:15:23 +02:00
Roman Khavronenko
cad2c28c6d
vmalert: correctly add duplicated params to the query (#4955)
Fix the bug when Group's `params` fields with multiple values were
overriding each other instead of adding up.
The bug was introduced in this commit eccecdf177
 starting from v1.91.1 https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.91.1

 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4908

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-09-08 23:34:56 +02:00
Aliaksandr Valialkin
ed2484ecfa
all: update Go builder from Go1.21.0 to Go1.21.1
See https://github.com/golang/go/issues?q=milestone%3AGo1.21.1+label%3ACherryPickApproved
2023-09-07 11:46:00 +02:00
Aliaksandr Valialkin
761510c054
Makefile: update golangci-lint from v1.51.2 to v1.54.2
See https://github.com/golangci/golangci-lint/releases/tag/v1.54.2
2023-09-01 11:12:12 +02:00
Aliaksandr Valialkin
9fa5a07887
app/vmagent/remotewrite: do not retry request immediately on io.ErrUnexpectedEOF, since this error isn't returned on stale connection
Also, mention the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4139 in comments to the code
in order to simplify further maintenance of this code.

This is a follow-up for 992a1c0a3a
2023-08-29 09:47:21 +02:00
hagen1778
f9604bcf3a
app/vmagent: fix comment typo after 992a1c0a3a
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-08-29 09:47:12 +02:00
Roman Khavronenko
3d8dec3e4c
vmagent: retry failed write request on the closed connection (#4857)
* vmagent: retry failed write request on the closed connection

 Retry failed write request on the closed connection immediately,
 without waiting for backoff. This should improve data delivery speed
 and reduce amount of error logs emitted by vmagent when using idle connections.

 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4139

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmagent: retry failed write request on the closed connection

Re-instantinate request before retry as body could have been already spoiled.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2023-08-29 09:46:42 +02:00
Roman Khavronenko
0459aaf8f9
vmalert: correctly re-instantinate HTTP req on retries (#4864)
* vmalert: correctly re-instantinate HTTP req on retries

Previosly, request retry to datasource re-used existing HTTP request.
But if request object was already partially processed (body was read),
then retry will be unsuccessful.

The change re-instantinates HTTP request object before retry.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: review fix

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-08-28 09:56:59 +02:00
Aliaksandr Valialkin
d688f9a744
app/vmagent/remotewrite: follow-up after a27c2f3773
- Fix Prometheus-compatible naming after applying the relabeling if -usePromCompatibleNaming command-line flag is set.
  This should prevent from possible Prometheus-incompatible metric names and label names generated by the relabeling.
- Do not return anything from relabelCtx.appendExtraLabels() function, since it cannot change the number of time series
  passed to it. Append labels for the passed time series in-place.
- Remove promrelabel.FinalizeLabels() call after adding extra labels to time series, since this call has been already
  made at relabelCtx.applyRelabeling(). It is user's responsibility if he passes labels with double underscore prefixes
  to -remoteWrite.label.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4247
2023-08-17 14:47:11 +02:00
Alexander Marshalov
c060c6d839
vmagent: fixed premature release of the context (after #4247 / #4824) (#4849)
Follow-up after a27c2f3773

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4247

Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-08-17 14:46:54 +02:00
Alexander Marshalov
927ded6c3b
fixed applying remoteWrite.label for pushed metrics (#4247) (#4824)
vmagent: properly add extra labels before sending data to remote storage

labels from `remoteWrite.label` are now added to sent metrics just before they
 are pushed to `remoteWrite.url` after all relabelings, including stream aggregation relabelings (#4247)

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4247

Signed-off-by: Alexander Marshalov <_@marshalov.org>
Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-08-17 14:46:41 +02:00
Aliaksandr Valialkin
b4c79fc606
lib/promrelabel: properly replace : char with _ in metric names when -usePromCompatibleNaming command-line flag is set
This addresses https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3113#issuecomment-1275077071 comment from @johnseekins
2023-08-17 13:52:45 +02:00
Roman Khavronenko
b4529df08d
vmbackup: correctly check if specified -dst belongs to specified -storageDataPath (#4841)
See this issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4837

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-08-17 13:50:42 +02:00
Dmytro Kozlov
a63fb21ab2
app/vmctl: fix migration process if tenant have no data (#4799)
app/vmctl: don't interrupt migration process if tenant has no data

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: Alexander Marshalov <_@marshalov.org>
2023-08-17 13:48:55 +02:00
Aliaksandr Valialkin
f96804658b
app/vmctl: document that -vm-native-step-interval command-line option now supports week value
This is a follow-up for d322ee4b35

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4738
2023-08-12 07:32:58 -07:00
Nikolay
8faa17493b
opentelemetry: return human readable error for json encoding. (#4822)
Opentelemetry parser supports only protobuf atm.

Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-08-12 05:05:16 -07:00
Aliaksandr Valialkin
072d891ed9
app/vmselect: prevent from panic when lookbehind window inside rollup function is parsed into negative value
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4795
2023-08-12 04:47:53 -07:00
Aliaksandr Valialkin
d7067c46d0
lib/flagutil: add defaultValue arg to NewArray{Int,Bytes,Duration} functions
The defaultValue is printed in the flag description when passing -help to the app.

This is a follow-up for aef31f201a

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4776
2023-08-12 04:19:05 -07:00
Aliaksandr Valialkin
acbe327fdf
app/{vmselect,vlselect}: run make vmui-update vmui-logs-update after 86f1459ca6 2023-08-11 07:00:39 -07:00
Zakhar Bessarab
b007f78a2e
app/vlinsert/elasticsearch: add a command-line flag to provide ES version (#4778)
* app/vlinsert/elasticsearch: add a command-line flag to provide ES version

Adds a flag which will allow to change version which will be reported by ES endpoint for compatibility checks performed by external logs shippers(such as filebeat).
See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4777

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* Document the -elasticsearch.version command-line flag

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4777

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-08-11 06:52:57 -07:00
Aliaksandr Valialkin
e0017b4d47
all: update Go builder from Go1.20.7 to Go1.21.0
See https://tip.golang.org/doc/go1.21
and https://go.dev/blog/go1.21
2023-08-11 06:25:54 -07:00
Aliaksandr Valialkin
a19a65b3a5
app/vmagent/remotewrite: go fmt 2023-08-11 06:23:00 -07:00
Aliaksandr Valialkin
4c4bcdf0b1
docs/CHANGELOG.md: add a link to stream aggregation for the description of the bugfix at a4a1884237
This makes the description more clear.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4804
2023-08-11 05:38:30 -07:00
Aliaksandr Valialkin
ac0b7e0421
Revert "vmui: change the response for active queries (#4782)"
This reverts commit 252643d100.

Reason for revert: the commit incorrectly fixes the the issue.
The `remoteAddr` must be properly quoted inside lib/httpserver.GetQuotedRemoteAddr().
It isn't quoted properly if the request contains X-Forwarded-For header.

The proper fix will be included in the follow-up commit.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4676
2023-08-11 05:06:40 -07:00
Aliaksandr Valialkin
2328e4cabc
app/vmagent/remotewrite: keep in sync the default value for -remoteWrite.sendTimeout option in the description with the actually used timeout
This is a follow-up for aef31f201a

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4776
2023-08-11 04:52:00 -07:00
Aliaksandr Valialkin
b50ed5ddd1
app/vmctl: follow-up after 5aed369132
- Fix default value for --remote-read-disable-path-append
- Clarify description for the change at docs/CHANGELOG.md

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4655

TODO: address the comment at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4744
2023-08-11 03:46:10 -07:00
Zakhar Bessarab
a4a1884237
{vmagent/remotewrite,vminsert/common}: fix dropInput and keepInput flags inconsistency (#4809)
{vmagent/remotewrite,vminsert/common}: fix dropInput and keepInput flags inconsistency

Sync behavior for dropInput and keepInput flags between single-node and vmagent.
Fix vmagent not respecting dropInput flag and reverse logic for keepInput.
2023-08-10 14:27:21 +02:00
Yury Molodov
252643d100
vmui: change the response for active queries (#4782)
* fix: change the response to a valid json (#4676)

* vmui/docs: fix response of active queries

https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4676
2023-08-10 12:27:28 +02:00
Yury Molodov
86f1459ca6
vmui: hide "Logs Explorer" for the base build (#4761)
LogsExplorer should be a part of VictoriaLogs binaries, as well as VMUI is now part of VictoriaMetrics binaries.
2023-08-10 11:42:13 +02:00
Yury Molodov
cc7bfaca6c
vmui: allow displaying the full error message on click (#4760)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4719
2023-08-10 11:34:25 +02:00
Abirdcfly
835c03fb47
vmalert: fix vmalert_remotewrite_send_duration_seconds_total metric value (#4801)
The deferred call's arguments are evaluated immediately, but the function call is not executed until the surrounding function returns.

Signed-off-by: Abirdcfly <fp544037857@gmail.com>
2023-08-10 10:51:44 +08:00
hagen1778
bc5065fd14
vmalert: mention vmalert_iteration_duration_seconds metric in README
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-08-09 22:03:22 +02:00
Haleygo
262c888d4b
vmalert: fix redundant clean up move (#4803)
Follow-up after 55ae2c2d57
2023-08-09 21:16:00 +02:00
Roman Khavronenko
1d4a0796f4
vmalert: cleanup config reload metrics handling (#4790)
* rename `configErr` to `lastConfigErr` to reduce confusion
* add tests to verify metrics and msg are set properly
* fix mistake when config success metric wasn't restored after an error

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-08-07 21:58:40 +02:00
Haleygo
527cfa2f74
vmalert: fix uncleaned tmp files in tests (#4788) 2023-08-07 14:24:56 +02:00
Zakhar Bessarab
2290252119
docs: make phrase about dedup and evaluation interval relation less obscure (#4781)
Value of `-dedup.minScrapeInterval` comand-line flag must be higher than `evaluation_interval` in order to make sure that only one sample on each evaluation will be left after deduplication.
Moreover, value of `-dedup.minScrapeInterval` must be a multiple of vmalert's `evaluation_interval` in order to make sure that samples will be aligned between deduplication window periods.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4774#issuecomment-1663940811

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-08-03 17:04:19 +02:00
Alexander Marshalov
aef31f201a
add info about remoteWrite.sendTimeout default value (#4776)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-08-03 16:25:11 +04:00
Roman Khavronenko
4c854c3ae2
security: bump go version from 1.20.6 to 1.20.7 (#4773)
The update includes a security fix to the crypto/tls package,
as well as bug fixes to the assembler and the compiler.

See the list of issues addressed in Go1.20.7 here:
https://github.com/golang/go/issues?q=milestone%3AGo1.20.7+label%3ACherryPickApproved

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-08-03 11:48:37 +02:00
SunKyu Lee
df37a47d4b
vmauth: add broken backend backoff flag (#4416)
vmauth: allow configuring deadline for a backend to be excluded from the rotation

The new flag `-failTimeout` allows overriding default time for a bad backend
to be excluded from rotation. The override option could be useful for systems
where it is expected for backends to be off for significant periods of time. 

Co-authored-by: Zakhar Bessarab <zekker6@gmail.com>
2023-08-02 14:30:21 +02:00
Yury Molodov
8f4961fbbd
vmui: display partial response warning (#4742)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4721
2023-08-02 14:21:52 +02:00
Dmytro Kozlov
5aed369132
app/vmctl: add flag where use can define path to the source remote read protocol (#4744)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4655
2023-08-01 16:43:00 +02:00
Roman Khavronenko
833ab331b1
vmctl: allow disabling binary export protocol (#4716)
Binary export API protocol can be disabled via `-vm-native-disable-binary-protocol` cmd-line flag when migrating data from VictoriaMetrics. Disabling binary protocol
can be useful for deduplication of the exported data before ingestion.
For this, deduplication need to be configured at `-vm-native-src-addr` side
and `-vm-native-disable-binary-protocol` should be set on vmctl side.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-08-01 09:45:50 +02:00
Aliaksandr Valialkin
f35d27aa2b
app/vlstorage: expose vl_data_size_bytes metric at /metrics page for tracking the on-disk data size (both indexdb and the data itself) 2023-07-31 07:56:53 -07:00
Dmytro Kozlov
d322ee4b35
app/vmctl: add support the week step for time-based chunks (#4743)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4738
2023-07-31 16:55:59 +02:00
Dmytro Kozlov
525c44e916
app/vmctl: remove ping from remote read protocol (#4749)
Ping method was never used, so we remove it.
2023-07-31 16:53:32 +02:00
Roman Khavronenko
216d4091f7
vmalert: remove deprecated in v1.79.0 web links with */status suffix (#4747)
Links of form `/api/v1/<groupID>/<alertID>/status` were deprecated
in favour of `/api/v1/alerts?group_id=<>&alert_id=<>` links in
v1.79.0. See more details here https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2825

This change removes code responsible for deprecated functionality.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-31 16:51:41 +02:00
hagen1778
3b524f671b
docs: rm typo of naming vmalert as a stateless service
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-07-31 16:47:50 +02:00