Commit graph

1686 commits

Author SHA1 Message Date
Aliaksandr Valialkin
a70df4bd83 PMM-6401 Initial implementaton for reading data from Prometheus files 2020-09-23 14:26:39 +03:00
Aliaksandr Valialkin
30c7269814 vendor: make vendor-update 2020-09-23 14:23:39 +03:00
Aliaksandr Valialkin
27500d7d4c app/vmselect/prometheus: code cleanup after 3ba507000c 2020-09-23 13:04:17 +03:00
Aliaksandr Valialkin
3ba507000c app/vmselect/prometheus: return timestamps from /api/v1/query, which match the time query arg
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/720
2020-09-23 12:58:48 +03:00
Aliaksandr Valialkin
c5ef0e6327 lib/persistentqueue: protect from multiple concurrent opening for the same persistent queue 2020-09-23 02:17:47 +03:00
Aliaksandr Valialkin
bed25e3c24 app/vmselect/netstorage: properly pre-allocate space for sbs 2020-09-22 23:49:55 +03:00
Aliaksandr Valialkin
5c42965853 lib/cgroup: attempt to obtain available CPU cores via /sys/devices/system/cpu/online
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685#issuecomment-674423728
2020-09-22 23:27:19 +03:00
Aliaksandr Valialkin
09b0f7c202 app/vmselect/netstorage: release search resources on timeout errors
Previously these resources weren't released, which could lead to resource leaks.
2020-09-22 22:57:38 +03:00
Aliaksandr Valialkin
36eb5427eb vendor: make vendor-update 2020-09-22 17:07:37 +03:00
Aliaksandr Valialkin
31ce0e29cd docs/Single-server-VictoriaMetrics.md: VictoriaMetrics properly stores Inf values after 26115891db 2020-09-22 17:02:39 +03:00
Aliaksandr Valialkin
3b1e3a03e0 app/vmselect: make sure the request doesnt wait in pending queue more than the configured timeout
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/711
2020-09-22 01:23:19 +03:00
Aliaksandr Valialkin
a69234ed18 lib/storage: code prettifying after be5e1222f3
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/781
2020-09-22 00:36:45 +03:00
faceair
be5e1222f3
add filter to getMetricIDs (#783)
* add getMetricIDs filter

* check nil filter before use
2020-09-22 00:33:43 +03:00
Aliaksandr Valialkin
94f7d00537 docs/vmagent.md: typo fix 2020-09-21 21:49:22 +03:00
Aliaksandr Valialkin
f6f5c4118c docs: make docs-sync 2020-09-21 21:47:47 +03:00
Aliaksandr Valialkin
00b5145c69 app/vmselect/searchutils: fixed tests after 2eb72e09ab 2020-09-21 21:31:38 +03:00
Aliaksandr Valialkin
2eb72e09ab app/vmselect: use time value rounded to seconds if it isnt passed to /api/v1/query
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/720
2020-09-21 21:24:40 +03:00
Aliaksandr Valialkin
29108cc53e lib/logger: add -loggerDisableTimestamps command-line flag for disabling timestamps in logs
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/778
2020-09-21 19:28:04 +03:00
Aliaksandr Valialkin
964bc7595c lib/promscrape/discovery/ec2: code prettifying after 312fead9a2
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/771
2020-09-21 18:43:34 +03:00
Nikolay Khramchikhin
312fead9a2
Add improvements to ec2_sd_discovery (#775)
* Add improvements to ec2 discovery

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/771

 role_arn support with aws sts
 instance iam_role support
 refreshing temporary tokens

* Apply suggestions from code review

Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>

* changed implementation, removed tests, clean up code

* moved endpoint builder into getEC2APIResponse

Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
2020-09-21 16:04:15 +03:00
Aliaksandr Valialkin
1e1a27d803 app/vmalert: remove unneeded UTC() call
UTC() doesn't change the underlying timestamp, so the call isn't needed here
2020-09-21 15:55:59 +03:00
Aliaksandr Valialkin
9739283dad lib/storage: reduce CPU load for idle VictoriaMetrics by reducing the frequency for the need for background merges 2020-09-21 15:54:11 +03:00
Roman Khavronenko
5dffc7a553
vmalert: add support for datasource.lookback flag (#779)
New datasource flag `datasource.lookback` defines how far to look into
past when evaluating queries.

Address https://github.com/VictoriaMetrics/VictoriaMetrics/issues/668
2020-09-21 15:53:49 +03:00
Roman Khavronenko
82c3bbce34
vmalert: fix the typo in error message (#782)
The error will be always nil so no sense in printing it.
2020-09-21 11:34:23 +03:00
Aliaksandr Valialkin
3e8569f456 lib/decimal: optimize maxUpExponent() by eliminating division from hot path 2020-09-19 13:50:09 +03:00
Aliaksandr Valialkin
f00e0e0103 lib/persistentqueue: sync data to file inside filestream.Writer.MustFlush 2020-09-19 12:51:41 +03:00
Aliaksandr Valialkin
26115891db lib/decimal: properly store Inf values
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/752
2020-09-18 19:07:07 +03:00
Aliaksandr Valialkin
d50165ad59 app/vmagent: increase default value for -remoteWrite.queues from 1 to 4, since it has been appeared that many users hit this limit 2020-09-18 14:21:54 +03:00
Aliaksandr Valialkin
63d3c88c3b vendor: update github.com/valyala/quicktemplate from v1.6.2 to v1.6.3 2020-09-18 13:10:48 +03:00
Aliaksandr Valialkin
1a9ee39b0e lib/promscrape: avoid copying response body when scraping targets.
This should reduce memory usage when scraping targets with millions of metrics.
2020-09-18 13:05:43 +03:00
Aliaksandr Valialkin
70c721c01b lib/persistentqueue: flush data to disk every second
Previously small amounts of data may be left unflushed for extended periods of time if vmagent collects small amounts of data.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/687
2020-09-18 13:05:40 +03:00
Aliaksandr Valialkin
74e3198281 vendor: udpate github.com/VictoriaMetrics/fasthttp from v1.0.5 to v1.0.7 2020-09-18 12:20:29 +03:00
Aliaksandr Valialkin
98d1cd0971 app/vmselect/graphite: return proper results /metrics/find?query=foo.*.bar according to Graphite Metrics API 2020-09-18 11:00:00 +03:00
Aliaksandr Valialkin
7a134b0fd7 app/vmstorage: added -forceMergeAuthKey command-line flag for protecting /internal/force_merge endpoint 2020-09-17 14:21:53 +03:00
Aliaksandr Valialkin
1f33dd717f lib/storage: add /internal/force_merge handler for running forced compactions on historical per-month partitions
This may be useful for freeing up storage space after time series deletion.

See https://victoriametrics.github.io/#force-merge for more details.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/686
2020-09-17 12:20:40 +03:00
Aliaksandr Valialkin
8beb0da6ad lib/{mergeset,storage}: compare errors with errors.Is() 2020-09-17 03:03:02 +03:00
Aliaksandr Valialkin
067d7c1ea1 lib/{mergeset,storage}: code prettifying 2020-09-17 02:06:31 +03:00
Aliaksandr Valialkin
020bd8685e lib/storage: removed duplicate checks for empty parts during merge - another check is in the beginning of mergeParts functions 2020-09-17 01:49:03 +03:00
Aliaksandr Valialkin
f2a449983d vendor: make vendor-update 2020-09-17 01:43:19 +03:00
Aliaksandr Valialkin
8674963f6a docs/Single-server-VictoriaMetrics.md: document that /api/v1/series/count may count delete time series
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/770
2020-09-17 01:38:17 +03:00
Aliaksandr Valialkin
ab53cb6f7b app/vmagent: substitute -remoteWrite.url with secret-url value in logs, since it may contain sensitive info such as passwords or auth tokens
Pass `-remoteWrite.showURL` command-line flag in order to see real `-remoteWrite.url` values in logs and at `/metrics` page.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/773
2020-09-16 22:36:25 +03:00
Aliaksandr Valialkin
9f79bcf64a app/vmselect: improve description for -search.maxQueryDuration 2020-09-16 21:15:41 +03:00
Aliaksandr Valialkin
39dee12ed7 lib/persistentqueue: code simplification after d455764a6f 2020-09-16 21:14:19 +03:00
Aliaksandr Valialkin
d455764a6f lib/persistentqueue: make the persistent queue more durable against unclean shutdown (kill -9, OOM, hard reset)
The strategy is:

- Periodical flushing of inmemory blocks to files, so they aren't lost on unclean shutdown.
- Periodical syncing of metadata for persisted queues, so the metadata remains in sync with the persisted data.
- Automatic adjusting of too big chunk size when opening the queue. The chunk size may be bigger than the writer offset after unclean shutdown.
- Skipping of broken chunk file if it cannot be read.
- Fsyncing finalized chunk files.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/687
2020-09-16 18:13:44 +03:00
Aliaksandr Valialkin
ffadf035fa lib/protoparser/vmimport: add more testcases for invalid timestamps and values
Updates https://github.com/VictoriaMetrics/vmctl/issues/25
2020-09-16 02:22:06 +03:00
Aliaksandr Valialkin
d8183c3124 lib/protoparser: report more errors for incorrect timestamps and/or values
Previously certain errors in timestamps and/or values could be silently skipped,
which could lead to samples with zero values stored in the database.

Updates https://github.com/VictoriaMetrics/vmctl/issues/25
2020-09-16 02:14:18 +03:00
Aliaksandr Valialkin
9bc8484ab6 lib/protoparser/graphite: return error when value or timestamp cannot be properly parsed
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/99
2020-09-16 01:35:12 +03:00
Aliaksandr Valialkin
26fa94ba8d vendor: update github.com/valyala/fastjson from v1.5.4 to v1.5.5
This should properly parse `+Inf` values when importing JSON lines via `/api/v1/import`

Updates https://github.com/VictoriaMetrics/vmctl/issues/25
2020-09-16 00:07:56 +03:00
Aliaksandr Valialkin
0bccb58e80 docs/Single-server-VictoriaMetrics.md: mention that VictoriaMetrics ignores NaN and Inf values during data ingestion
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/752
2020-09-15 23:40:28 +03:00
Aliaksandr Valialkin
1fec47a289 app/vmselect/netstorage: reduce memory usage when the time range from query touches big number of samples per each time series 2020-09-15 21:08:28 +03:00