Commit graph

5916 commits

Author SHA1 Message Date
Haleygo
a0752b20c7
vmselect: fix result in Prometheus query when time is small (#4578)
vmselect: fix result in Prometheus query when time is small

Co-authored-by: Roman Khavronenko <roman@victoriametrics.com>
2023-08-12 13:32:05 -07:00
Aliaksandr Valialkin
b351f73984
lib/promscrape/discovery: close unused HTTP connections to service discovery servers
This should prevent from connection leaks

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4724
2023-08-12 13:30:25 -07:00
Aliaksandr Valialkin
4c211028af
docs/CHANGELOG.md: add a link to Pushgateway protocol in the bugfix description for 74237ce5c0
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4692
2023-08-12 13:24:43 -07:00
Alexander Marshalov
5dba522eaa
fixed label values decoding for pushgateway compatibility (#4727)
Fixed decoding of label values with slash for pushgateway and prometheus golang client compatibility + added some tests. (#4962)
2023-08-12 13:23:14 -07:00
Aliaksandr Valialkin
63a793b506
docs/CHANGELOG.md: clarify the description of the bugfix at ce7141383d 2023-08-12 13:21:09 -07:00
Zakhar Bessarab
0ee30dc00e
app/vmagent/remotewrite: fix vmagent panic on shutdown (#4407)
app/vmagent/remotewrite: fix vmagent panic on shutdown

Currently, when vmagent is stopping it first flushes pending series in remote write context and proceeds to stop streaming aggregation. This leads to streaming aggregation being unable to write results into pending timeseries (since it is already nil) and panic.
This can lead to losing some aggregation results being lost almost silently.

The fix is reordering flow to first stop streaming aggregation and flush all pending time series after that.

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-08-12 13:20:22 -07:00
Aliaksandr Valialkin
8234741c51
app/vmselect/netstorage: follow-up after 11ac551d52
- Clarify the scope of the fix at docs/CHANGELOG.md
- Handle the case when -search.maxSamplesPerSeries limit is exceeded
  in the same way as the -search.maxSamplesPerQuery limit.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4472
2023-08-12 13:11:24 -07:00
Roman Khavronenko
26e5cf4160
app/vmselect/netstorage: properly process -search.maxSamplesPerQuery limit (#4472)
Properly return the error to user when `-search.maxSamplesPerQuery` limit is exceeded.
Before, user could have received a partial response instead.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-08-12 13:09:11 -07:00
Aliaksandr Valialkin
2711770b10
vendor: run make vendor-update 2023-08-12 12:43:23 -07:00
Aliaksandr Valialkin
041a6369e7
Upgrade Go builder from 1.20.4 to 1.21.0 2023-08-12 12:35:12 -07:00
Aliaksandr Valialkin
e7b3a338fe
deployment/docker/Makefile: upgrade base Docker image from alpine:3.18.2 to alpine:3.18.3
See https://alpinelinux.org/posts/Alpine-3.15.10-3.16.7-3.17.5-3.18.3-released.html
2023-08-12 12:33:03 -07:00
Zakhar Bessarab
7487a8de1c
lib/promrelabel: fix relabeling if clause (#4816)
* lib/promrelabel: fix relabeling if clause being applied to labels outside of current context

Relabeling is applied to each metric row separately, but in order to lower amount of memory allocations it is reusing labels.

Functions which are working on current metric row labels are supposed to use only current metric labels by using provided offset, but if clause matcher was using the whole labels set instead of local metrics.

This leaded to invalid relabeling results such as one described here: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4806

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs/CHANGELOG.md: document the bugfix

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1998
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4806

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2023-08-12 12:31:36 -07:00
Roman Khavronenko
7dc31ab080
docs: mention fix for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4221 (#4382)
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4221

Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 4b5faf7efb)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-06-05 11:43:53 +02:00
Roman Khavronenko
74a92110bd
vmalert: do not return nil rules for /api/v1/rules (#4344)
The fix addresses a case when vmalert is configured with a group
which has `name`, but doesn't have `rules` configured. In this
case it still returns a `nil` instead of `[]` slice.

Fixing this via current commit.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4221

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 66ed6fe62f)
2023-06-05 11:43:09 +02:00
Roman Khavronenko
6e02885431
vmalert: properly form assets address if httpPrefix set (#4351)
Properly form path to static assets in WEB UI
 if `http.pathPrefix` set.

 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4349

Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit 51cea6cad4)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-06-05 11:41:48 +02:00
hagen1778
e8448fe8fd
docs/changelog: mention 20dc3db71e
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-06-05 11:40:49 +02:00
Roman Khavronenko
a8835b443f
vmalert: fix nil map assignment (#4392)
* vmalert: fix nil map assignment

The storage instance with nil map params was created for remote-read purposes.
And before change 7a9ae9de0d this map was ignored in ApplyParams.
Now, it started to be used and vmalert panics in runtime.

The fix properly inits map for at `NewVMStorage` and verifies it is not nil
on assignment in `ApplyParams`.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: add to changelog

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: properly clone Storage params

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: properly clone Storage params

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* vmalert: properly clone Storage params

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit de94812088)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-06-05 11:38:41 +02:00
Roman Khavronenko
e082c3dd27
app/vmalert: follow-up after 7a9ae9de0d (#4381)
7a9ae9de0d

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit eccecdf177)
2023-06-05 11:37:43 +02:00
gsakun
bab0808a7f
app/vmalert: fix datasource.roundDigits Parameter (#4341)
app/vmalert: fix querybuild clone and extraParams merge logic

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4340

(cherry picked from commit 20dc3db71e)
2023-06-05 11:36:18 +02:00
Nikolay
278c580055
app/vmauth: properly handle LOCAL proxy protocol command (#4373)
It is required for handling health checks from load balancers

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3335

(cherry picked from commit f263031fe9)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-06-05 11:34:41 +02:00
Aliaksandr Valialkin
4c1241d336
vendor: run make vendor-update 2023-05-18 15:48:55 -07:00
Aliaksandr Valialkin
b3a3260b1d
docs/CHANGELOG.md: cut v1.87.6 2023-05-18 15:42:40 -07:00
Aliaksandr Valialkin
2783b99457
deployment/docker/Makefile: use alpine 3.17.3 instead of alpine 3.18.0 for certs image, since alpine 3.18.0 doesnt work for cross-platform builds 2023-05-18 14:12:55 -07:00
Nikolay
a0ece56649
app/vmauth: do not return invalid credentials (#4288)
at http response by default
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4188

based on https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4190
Thanks @raj-kumar-j  for init implementation
2023-05-17 00:11:40 -07:00
Aliaksandr Valialkin
1ae8912f0f
lib/backup/actions: add missing import after 25b527f1e4 2023-05-16 15:42:46 -07:00
Aliaksandr Valialkin
26b9f41e08
lib/mergeset: close and open the table before making snapshots at TestTableCreateSnapshotAt()
This gives guarantees that all the in-memory data is written to disk at the snapshot time.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4272
See https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4316
2023-05-16 15:42:13 -07:00
Aliaksandr Valialkin
a0d84278ce
deployment/docker: update base docker image from 3.17.3 to 3.18.0
See https://www.alpinelinux.org/posts/Alpine-3.18.0-released.html
2023-05-12 17:33:30 -07:00
Aliaksandr Valialkin
90ee6f174c
lib/promutils: properly return error when incorrect Prometheus label names are passed to NewLabelsFromString()
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4284
See also https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4304
2023-05-12 17:04:07 -07:00
Aliaksandr Valialkin
78cd76d96a
lib/protoparser/csvimport: properly parse the last empty column in CSV line
Do not ignore the last empty column in CSV line.
While at it, properly parse CSV columns in single quotes, e.g. `'foo,bar',baz` is parsed as two columns - `foo,bar` and `baz`

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4048

See also https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4298
2023-05-12 17:00:48 -07:00
Alexander Marshalov
25b527f1e4
fixed error with double slash in vmbackupmanager (#557)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-05-11 13:38:59 -07:00
Aliaksandr Valialkin
8febf3c5e6
docs/CHANGELOG.md: document 8f4de6fa47 2023-05-09 23:09:06 -07:00
Roman Khavronenko
470c82919e
vminsert: properly reset labels object on aggregation (#4278)
Without reset, labels duplicates could have been added during stream aggregation.
Since `ctx.Labels` is reused during processing of many series, each series will
add its labels to the context. Even if the same labels were already addeded on prev
iteration. Now, we reset `ctx.Labels` on each iteration to contain so labels from
different series didn't interfere.

This could have cause exceeding of the limit on number of labels per pushed time series.

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4277

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-05-09 23:05:53 -07:00
Aliaksandr Valialkin
f4f6bb359e
all: update Go builder from Go1.20.3 to Go1.20.4
See https://github.com/golang/go/issues?q=milestone%3AGo1.20.4+label%3ACherryPickApproved
2023-05-09 22:34:21 -07:00
Alexander Marshalov
591fb53b58
fixed vm_promscrape_config_last_reload_successful metric value recovery after successful reloading with unchanged content (#4260) (#4268)
Signed-off-by: Alexander Marshalov <_@marshalov.org>
2023-05-09 22:26:35 -07:00
Roman Khavronenko
fff051b889
alerts: update TooHighMemoryUsage threshold (#4256)
It appears that 90% usage for anonymous mem usage
is already concerning. So we lowering the threshold to 80%.

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-05-09 21:43:35 -07:00
Nikolay
e5e10988ed
lib/storage: properly update link for entry at dateMetricID cache (#4258)
previously during sync for mutable and immutable cache parts, link for hotEntry with current date may be not properly updated
it corrupts cache for backfilling metrics and increased cpu load
2023-05-09 21:40:11 -07:00
Zakhar Bessarab
4c3e0d4411
lib/promscrape/discovery/kubernetes: follow-up for d5e94721db (#4255)
- add changelog reference to an author
- fix tests
- add metadata to match Prometheus behavior

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-05-09 21:32:02 -07:00
Vasilchenko Anton
e6d24e68f7
Add endpoint labels for pod targets discovered form endpoint but has different ports (#4253)
Signed-off-by: Vasilchenko Anton <vasilchenko-as@yandex.ru>
2023-05-09 21:30:09 -07:00
Zakhar Bessarab
a50d63c376
lib/storage: fix indexdb rotation infinite loop (#4249)
When using `retentionTimezoneOffset` and having local timezone being more than 4 hours different from UTC indexdb retention calculation could return negative value. This caused indexdb rotation to get in loop.
Fix calculation of offset to use `retentionTimezoneOffset` value properly and add test to cover all legit timezone configs.
See:
- https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4207
- https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4206

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2023-05-09 21:28:20 -07:00
Roman Khavronenko
89ec85c5db
vmselect: exit early from queue on context cancel (#4223)
* vmselect: exit early from queue on context cancel

When `-search.maxConcurrentRequests` is reached, vmselect puts
request in the queue. It is expected, that requests in the queue
will be processed as soon as it would be enough capacity to do so.

However, it could happen that while request was waiting its turn,
the client could have already cancel it (close the connection,
or just close the tab with UI). In this case, we should de-queue
such requests to avoid spending extra resources on them.

Signed-off-by: hagen1778 <roman@victoriametrics.com>

* app/vmselect: address review comments

Signed-off-by: hagen1778 <roman@victoriametrics.com>

---------

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-05-08 22:58:36 -07:00
Zakhar Bessarab
7cc7b4436f
lib/promscrape/discovery/kubernetes: add common labels to all ports discovered from endpoints (#4235)
* lib/promscrape/discovery/kubernetes: add common labels to all ports discovered from endpoints

Sets
`__meta_kubernetes_endpoints_name` and `__meta_kubernetes_namespace` labels to all ports of pod.
Prometheus sets those labels to all ports in pod (0ab9553611/discovery/kubernetes/endpoints.go (L267C15-L269)) even if port is not matching any service.

See: #4154

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/promscrape/discovery/kubernetes: fix test for updated discovery logic

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-05-08 22:16:36 -07:00
Roman Khavronenko
d09ba9f504
vmalert: fix API to return non-nil values (#4222)
Properly return empty slices instead of nil for `/api/v1/rules` and `/api/v1/alerts` API handlers.
This improves compatibility with Grafana.

 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4221

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-05-08 21:49:13 -07:00
Aliaksandr Valialkin
ab1d226485
docs/CHANGELOG.md: document 03150c8973
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4204
2023-05-08 16:32:07 -07:00
Haleygo
c7a407368a
lib/opentsdbhttp: fix a typo preventing from using writeconcurrencylimiter (#4208) 2023-05-08 16:31:05 -07:00
Zakhar Bessarab
56350eb1ce
lib/httpserver: add handler to serve /robots.txt and deny search indexing (#4143)
This handler will instruct search engines that indexing is not allowed for the content exposed to the internet. This should help to address issues like #4128 when instances are exposed to the internet without authentication.
2023-05-08 09:52:04 -07:00
Aliaksandr Valialkin
70d9a2e679
docs/CHANGELOG.md: document c77385e78f 2023-05-08 08:56:32 -07:00
Aliaksandr Valialkin
d324241707
vendor: update github.com/VictoriaMetrics/metricsql from v0.56.1 to v0.56.2
This fixes panic when the duration in the query contains `M` suffix.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4120
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3589
2023-04-14 14:09:21 -07:00
Aliaksandr Valialkin
112c9a8118
app/vmstorage: deprecate -bigMergeConcurrency command-line flag
Improperly configured -bigMergeConcurrency command-line flag usually leads to uncontrolled
growth of unmerged parts, which, in turn, increases CPU usage and query durations.

So it is better deprecating this flag. In rare cases -smallMergeConcurrency command-line flag
can be used instead for controlling the concurrency of background merges.
2023-04-13 23:40:40 -07:00
Aliaksandr Valialkin
03ad293ba2
docs/CHANGELOG.md: run at least 4 background mergers on systems with less than 4 CPU cores
This reduces the probability of sudden spike in the number of small parts when all the background mergers
are busy with big merges.
2023-04-13 23:38:40 -07:00
Haleygo
74c7c49335
fix sort pendingDateMetricsIDs (#4102) 2023-04-10 12:23:25 -07:00