github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2024-12-01 14:47:38 +00:00
Code Issues Projects Releases Packages Wiki Activity
5890 commits 119 branches 748 tags 355 MiB
Commit graph

450 commits

Author SHA1 Message Date
Aliaksandr Valialkin
b91e18af0e docs/vmagent.md: add an example for -remoteWrite.label 2021-03-25 17:59:30 +02:00
Aliaksandr Valialkin
6de36b4e21 docs/vmagent.md: cosmetic fixes 2021-03-25 17:11:10 +02:00
Aliaksandr Valialkin
1b13d02728 docs/vmagent.md: cosmetic fixes 2021-03-25 16:54:37 +02:00
Aliaksandr Valialkin
155cd90fc8 docs/vmagent.md: typo fix: tupically -> typically 2021-03-25 16:48:27 +02:00
Aliaksandr Valialkin
6b1f807418 app/vmagent: add -promscrape.consul.waitTime command-line flag for configuring Consul service discovery wait time 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1144
 2021-03-23 19:34:12 +02:00
Aliaksandr Valialkin
2041008d64 app/vmagent: mention -remoteWrite.maxDiskUsagePerURL in the descriptio of -remoteWrite.tmpDataPath flag 2021-03-23 16:38:41 +02:00
Aliaksandr Valialkin
6ebf18ab97 app/{vminsert,vmagent}: use Influx field as metric name if measurement is empty and -influxSkipSingleField command-line is set 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1139
 2021-03-22 13:55:11 +02:00
Aliaksandr Valialkin
5e77a939c2 all: make go vet happy 2021-03-17 00:48:44 +02:00
Aliaksandr Valialkin
b997f4a418 all: make golangci-lint happy after the commit 6378205415 2021-03-17 00:24:31 +02:00
Aliaksandr Valialkin
8005ba26b9 lib/netutil: enable IPv6 UDP listening if -enableTCP6 command-line flag is passed to VictoriaMetrics 
This is a follow-up for 18cfc4be7b

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1131
 2021-03-17 00:19:30 +02:00
Aliaksandr Valialkin
3caac5edd4 Makefile: prepare vmutils-windows-*.zip archive on make release-vmutils command 
The archive contains the following executables for Windows:

* vmagent
* vmalert
* vmauth
* vmctl

Other components - vmbackup, vmrestore, victoria-metrics - aren't supported for Windows yet
 2021-03-16 20:54:10 +02:00
Aliaksandr Valialkin
e2717d84c0 all: various fixes in command-line flag descriptions 2021-03-15 22:03:49 +02:00
Aliaksandr Valialkin
776b8b32ca app/{vminsert,vmagent}: a follow-up for b1aa8c3d8f 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1124
 2021-03-15 22:03:49 +02:00
Nikolay
a843dc0219 adds fake response for telegraph queries (#1130) 
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1124
 2021-03-15 22:03:49 +02:00
Aliaksandr Valialkin
60e0280a94 lib/promscrape: add ability to configure proxy options via proxy_tls_config, proxy_basic_auth, proxy_bearer_token and proxy_bearer_token_file 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1116
 2021-03-12 03:36:11 +02:00
Aliaksandr Valialkin
502fab797a lib/promscrape: add scrape_offset option to scrape_config 
This option can be used for specifying the particular offset per each scrape interval for target scraping
 2021-03-08 11:59:32 +02:00
Aliaksandr Valialkin
133fb9fc00 lib/promscrape: add -promscrape.cluster.replicationFactor command-line flag for replicating scrape targets among vmagent instances in the cluster 2021-03-04 10:21:27 +02:00
Aliaksandr Valialkin
d109e17f46 all: bump minimum supported Go version from 1.13 to 1.14 2021-03-03 15:58:17 +02:00
Aliaksandr Valialkin
3c11ce9356 docs/vmagent.md: remove outdated suggestion for determining labels that lead to duplicate targets 
The original labels for duplicate targets is already printed in the error message starting from 71ea4935de
 2021-03-03 12:28:50 +02:00
Roman Khavronenko
219291e084 Vmagent docs upd (#1104) 
* vmagent: port changes from https://github.com/VictoriaMetrics/VictoriaMetrics.github.io/pull/1

Thanks to @dereksfoster99 for this patch!

* vmagent: reword to make the meaning clear
 2021-03-03 10:53:41 +02:00
Aliaksandr Valialkin
d9e8af0e8f docs: actualize -help output 2021-03-01 17:02:05 +02:00
Aliaksandr Valialkin
323af49234 app/vmagent/remotewrite: clarify -remoteWrite.flushInterval flag description 2021-03-01 11:51:08 +02:00
Aliaksandr Valialkin
b8dd5e8292 app/vmagent/remotewrite: fix rate limiting logic for -remoteWrite.url 2021-03-01 00:59:57 +02:00
Aliaksandr Valialkin
d0be1f6f49 app/vmagent: remove data race when applying rate limits to -remoteWrite.url with multiple queues 2021-03-01 00:28:20 +02:00
Aliaksandr Valialkin
17eb29206d docs/vmbackup.md: clarify docs on vmagent clustering 2021-02-28 21:59:55 +02:00
Aliaksandr Valialkin
9a2bf65134 lib/promscrape: add ability to spread scrape targets among multiple vmagent instances 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1084
 2021-02-28 18:40:42 +02:00
Aliaksandr Valialkin
7f1302688f lib/fs: follow-up after f3a03c4164 2021-02-27 01:09:37 +02:00
Nikolay
d88fa5ebe4 Adds windows build (#1040) 
* fixes windows compilation,
adds signal impl for windows,
adds free space usage for windows,
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/70
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1036

NOTE victoria metrics database still CANNOT work under windows system,
only vmagent is supported.
To completly port victoria metrics, you have to fix issues with separators,
parsing and posix file removall

* rollback separator

* Adds windows setInformation api,
it must behave like unix, need to test it.
changes procutil

* check for invlaid param

* Fixes posix delete semantic

* refactored a bit

* fixes openbsd build

* removed windows api call

* Fixes code after windows add

* Update lib/procutil/signal_windows.go

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2021-02-27 01:06:22 +02:00
Aliaksandr Valialkin
197ecca426 lib/promrelabel: add more optimizations for relabeling for common cases 2021-02-22 16:36:54 +02:00
Aliaksandr Valialkin
72eef964d9 app/vmagent: properly perform graceful shutdown, which was broken in the commit 1d1ba889fe 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1065
 2021-02-19 00:34:17 +02:00
Aliaksandr Valialkin
41d3ff4f2b app/vmagent: fix scrape config example for scrape_align_interval option 2021-02-18 23:57:45 +02:00
Aliaksandr Valialkin
502d0e2524 lib/promscrape: add scrape_align_interval config option into scrape config 
This option allows aligning scrapes to a particular intervals.
 2021-02-18 23:53:04 +02:00
Aliaksandr Valialkin
9c81429299 app/vmagent/remotewrite: cleanup after 1d1ba889fe 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1065
 2021-02-17 21:43:48 +02:00
Nikolay
40973eda1c adds pushback for fastqueue, (#1075) 
during shutdown currently sending block was lost,
now its pushed back to fast queue and will be flushed on disk,
it may lead to data duplication.
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1065
 2021-02-17 21:43:46 +02:00
Aliaksandr Valialkin
c769f8321d deployment/docker: embed tzdata into prod Go app instead of installing it into base docker image 
While this increases app size by 700Kb, this allows using -loggerTimezone in a scratch base image
See https://github.com/golang/go/issues/38017
 2021-02-12 04:56:27 +02:00
Aliaksandr Valialkin
ff7850aec0 deployment/docker: use docker buildx for creating multiarch builds 
See https://github.com/docker/buildx/
 2021-02-12 04:35:35 +02:00
Aliaksandr Valialkin
a3aa9bdc9f app/vmagent: add Advanced usage section with the description for all the command-line flags 2021-02-03 21:03:52 +02:00
Aliaksandr Valialkin
03da0b728c docs: increase heading sizes in vmagent, vmauth, vmbackup and vmrestore docs, so they match the heading sizes in VictoriaMetrics docs 2021-02-01 19:44:58 +02:00
Aliaksandr Valialkin
fdf9de98f8 app/vmagent: add -remoteWrite.roundDigits command-line option for limiting the number of digits after the point for stored values 
This commit also adds --vm-round-digits command-line option to vmctl tool.
 2021-02-01 14:42:15 +02:00
Aliaksandr Valialkin
4b324da947 all: consistently use timers from timerpool 2021-01-27 00:40:39 +02:00
Aliaksandr Valialkin
29bf531f7d app/vmagent: add -remoteWrite.rateLimit command-line flag for limiting data rate to remote storage 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1035
 2021-01-27 00:40:39 +02:00
Aliaksandr Valialkin
5640e6cbca docs/vmagent.md: follow-up for 184a659c5f 2021-01-13 13:54:28 +02:00
Nikolay
821492bc0b adds extra_label to all import apis (#1007) 
* adds extra_label to all import apis,
changes priority for extra_label - now it has priority over original labels

* Update README.md

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>

* Update README.md

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>

* adds extra labels to vmagent  import api
changes order for adding labels, now its added after user values

* adds tests for extra_label

* import fix

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2021-01-13 01:07:24 +02:00
Nikolay
9f0a4fd00e Fixes error handling for promscrape.streamParse (#1009) 
properly return error if client cannot read data,
properly suppress scraper errors
 2021-01-12 13:35:09 +02:00
Aliaksandr Valialkin
4ee53c3961 all: use net.Dial instead of fasthttp.Dial, because fasthttp.Dial limits the number of concurrent dials to 1000 2021-01-11 12:52:51 +02:00
Aliaksandr Valialkin
49e800ba55 app/vmagent: add vmagent_remotewrite_blocks_sent_total and vmagent_remotewrite_bytes_sent_total metrics per each -remoteWrite.url 2020-12-15 20:41:08 +02:00
Aliaksandr Valialkin
9ab7ca1133 docs/vmagent.md: typo fix: pearsed->parsed 2020-12-15 19:03:35 +02:00
Aliaksandr Valialkin
11674a9b76 docs/vmagent.md: mention that sample_limit option has no sense when stream parsing is enabled 2020-12-15 18:44:19 +02:00
Nikolay
7064c4eb8e adds new Array Flags (#965) 
* adds ArrayDuration and ArrayBool flags,
makes sendTimeout and tlsInsecure configurable per remoteWrite url

* added backward compatibility testcases for ArrayDuration and ArrayBool

* fixes bool flag

* fixes test cases
 2020-12-15 12:59:33 +02:00
Aliaksandr Valialkin
a2eb451de4 app/{vmagent,vminsert}: follow-up for ce8c2dd1f1: return /targets page in HTML when requested via web browser 2020-12-14 14:13:01 +02:00
Nikolay
324e3aa1a5 Changes targets api (#961) 
* changes /targets api
adds html response if requester accepts text/html,
adds quick template for /targets api,
fixes pathPrefix for / requests

* changes namings

* renamed targets file

* Update app/victoria-metrics/main.go

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>

* adds trimspace to qtpl,
moves content-type for targets response closer to writer

* fixes bug with prefix

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2020-12-14 14:13:00 +02:00
Aliaksandr Valialkin
fc82c22e50 docs: consistently use links to https://victoriametrics.github.io for documentation references 2020-12-11 21:09:17 +02:00
Aliaksandr Valialkin
1a237c6903 all: properly handle CPU limits set on the host system/container 
This can reduce memory usage on systems with enabled CPU limits.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/946
 2020-12-08 21:07:03 +02:00
Aliaksandr Valialkin
7bdf07883b app/{vmalert,vmagent}: skip empty values in -remoteWrite.label and -label lists 2020-12-08 14:54:02 +02:00
Aliaksandr Valialkin
bdac2171f1 all: do not print usage info for all the flags when incorrect command-line flag is passed 
This should improve usability for VictoriaMetrics apps that have big number of command-line flags,
i.e. all the apps.
 2020-12-03 21:46:19 +02:00
Aliaksandr Valialkin
3f52e59efe app/{vmagent,victoria-metrics}: add -dryRun option and make more clear handling for -promscrape.config.dryRun 2020-11-25 23:01:39 +02:00
Aliaksandr Valialkin
ed06990609 app/vmagent: do not enable -promscrape.config.strictParse when -dryRun command-line flag is set 
Users can specify -promscrape.config.strictParse if -promscrape.config shouldn't contain unknown config entries
 2020-11-25 22:27:41 +02:00
Nikolay
bb2bcb9725 Adds eureka service discovery (#913) 
* Adds eureka service discovery
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/851
Netflix service discovery for AWS

* Apply suggestions from code review

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2020-11-20 14:02:13 +02:00
Aliaksandr Valialkin
7ceaf4ba8f all: consistently return text-based HTTP responses with charset=utf-8 
This is a follow-up for https://github.com/VictoriaMetrics/VictoriaMetrics/pull/897
 2020-11-13 10:30:21 +02:00
faceair
64e99744f1 add charset on targets response (#897) 2020-11-13 10:18:13 +02:00
Aliaksandr Valialkin
f7a6ae3d11 docs/vmagent.md: added a link to https://valyala.medium.com/how-to-use-relabeling-in-prometheus-and-victoriametrics-8b90fc22c4b2 into Relabeling section 2020-11-12 12:27:13 +02:00
Aliaksandr Valialkin
069979c367 docs/vmagent.md: typo fix 2020-11-11 16:05:04 +02:00
Aliaksandr Valialkin
7a0094adae docs/vmagent.md: add Configuration update section 2020-11-11 16:01:21 +02:00
Aliaksandr Valialkin
a8562d643b lib/promscrape: add -promscrape.dropOriginalLabels command-line flag for reducing memory usage when discovering big number of scrape targets 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825
 2020-11-10 00:20:49 +02:00
Aliaksandr Valialkin
b8083b7659 lib/promscrape: clean references to label name and label value strings after applying per-target relabeling 
This should reduce memory usage when per-target relabeling creates big number of temporary labels
with long names and/or values.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/825
 2020-11-07 16:19:52 +02:00
Aliaksandr Valialkin
1336e47c86 docs/vmagent.md: update after 4c808d58bf 2020-11-04 20:33:49 +02:00
Nikolay
5b235b902b Adds ready probe (#874) 
* adds leading forward slash check for scrapeURL path
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/835

* adds ready probe for scrape config initialization,
it should prevent metrics loss during vmagent rolling update,
/ready api will return 425 http code, if some scrape config still waits for initialization.

* updates docs

* Update app/vmagent/README.md

* renames var

* Update app/vmagent/README.md

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2020-11-04 20:33:48 +02:00
Nikolay
d0a9b24c5a reduces memory usage for vmagent, (#880) 
* reduces memory usage for vmagent,
limits count of droppedTarget, that can be stored for /api/v1/targets page up to 999 items,
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/878

* Update app/vmagent/README.md

* Update app/vmagent/README.md

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2020-11-04 17:13:33 +02:00
Aliaksandr Valialkin
407a46c11e Revert "docs/vmagent.md: mention about -promscrape.dropOriginalLabels" 
This reverts commit 1a80acc712.
 2020-11-04 11:45:35 +02:00
Aliaksandr Valialkin
1a80acc712 docs/vmagent.md: mention about -promscrape.dropOriginalLabels 2020-11-04 11:16:16 +02:00
Aliaksandr Valialkin
887a3c317f app/vmagent/remotewrite: drop packets only on 409 status code, since there are other valid 4xx status codes, which shouldnt result in packet drop 2020-11-03 14:24:57 +02:00
Aliaksandr Valialkin
ca2e0f1e04 app/vmagent/remotewrite: drop packets if remote storage returns 4xx status code 
This makes consistent the behaviour with Prometheus.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/873
 2020-11-02 00:45:01 +02:00
Aliaksandr Valialkin
ed724d25ba lib/promscrape: add stream parse mode for efficient scraping of targets that expose millions of metrics 2020-11-01 23:12:26 +02:00
Aliaksandr Valialkin
abdf22e0bb app/vmagent: expose /api/v1/targets page according to https://prometheus.io/docs/prometheus/latest/querying/api/#targets 
This page is exposed by vmagent and by a single-node VictoriaMetrics

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/643
 2020-10-20 21:55:14 +03:00
Aliaksandr Valialkin
938b3b7ed1 lib/promscrape: code prettifying after 9bd9f67718 2020-10-12 16:13:59 +03:00
Nikolay Khramchikhin
7f96712b38 Adds dockerswarm sd (#818) 
* adds dockerswarm service discovery

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/656

 Following roles supported: services, tasks and nodes.
 Basic, token and tls auth supported.
 Added tests for labels generation.

* added unix socket support to discovery utils

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2020-10-12 16:13:58 +03:00
Aliaksandr Valialkin
9b7ce5d004 app/{vminsert,vmagent}: take into account all the inserted rows before relabeling in vm_rows_inserted_total and vmagent_rows_inserted_total metrics 2020-10-09 13:38:49 +03:00
Aliaksandr Valialkin
35b8ffaa17 docs/vmagent.md: clarify -promscrape.suppressDuplicateScrapeTargetErrors command-line flag usage 2020-10-08 19:24:05 +03:00
Aliaksandr Valialkin
0d44e371f3 lib/promscrape: add -promscrape.suppressDuplicateScrapeTargetErrors command-line flag in order to suppress duplicate scrape target errors 
Show also original labels for duplicate targets in error message in order to simplify debugging the issue.

Now `/targets` endpoint accepts optional `show_original_labels=1` query arg, which shows original labels for each target.
This may simplify debugging for target relabeling.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/651
 2020-10-08 18:59:25 +03:00
Aliaksandr Valialkin
c4d309aa41 app/vmagent: add a link to https://victoriametrics.github.io/vmagent.html from main page 2020-10-06 15:30:07 +03:00
Nikolay Khramchikhin
b4c77fc6d2 Adds openstack sd (#811) 
* adds openstack service discovery

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/728

 implemented hypervisors and instance discovery with openstack v3 api.
 Added tests for labeling and data parsing.
 Added token refresh.

* Apply suggestions from code review

* Apply suggestions from code review

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2020-10-05 16:48:54 +03:00
Aliaksandr Valialkin
5dca7bbe85 app/vmagent/remotewrite: do not show -remoteWrite.url in logs if -remoteWrite.showURL isn't set 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/773
 2020-09-29 19:49:19 +03:00
Aliaksandr Valialkin
79b00f7b6b docs: improve readability a bit 2020-09-29 13:05:05 +03:00
Aliaksandr Valialkin
8df33bd5c1 app/{vminsert,vmagent}: improve data ingestion speed over a single connection 
Process data obtianed from a single connection on all the available CPU cores.
 2020-09-28 04:14:51 +03:00
Aliaksandr Valialkin
aadbd014ff all: add native format for data export and import 
The data can be exported via [/api/v1/export/native](https://victoriametrics.github.io/#how-to-export-data-in-native-format) handler
and imported via [/api/v1/import/native](https://victoriametrics.github.io/#how-to-import-data-in-native-format) handler.
 2020-09-27 17:36:38 +03:00
Aliaksandr Valialkin
8fc9b77496 app/vmagent: reduce memory usage when importing data via /api/v1/import 
Previously vmagent could use big amounts of RAM when each ingested JSON line
contained many samples.
 2020-09-26 04:10:13 +03:00
Aliaksandr Valialkin
ed473c94ff docs/vmagent.md: typo fix 2020-09-21 21:49:08 +03:00
Nikolay Khramchikhin
0069353d5e Add improvements to ec2_sd_discovery (#775) 
* Add improvements to ec2 discovery

https://github.com/VictoriaMetrics/VictoriaMetrics/issues/771

 role_arn support with aws sts
 instance iam_role support
 refreshing temporary tokens

* Apply suggestions from code review

Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>

* changed implementation, removed tests, clean up code

* moved endpoint builder into getEC2APIResponse

Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
 2020-09-21 16:05:01 +03:00
Aliaksandr Valialkin
1c07d7bee7 app/vmagent: increase default value for -remoteWrite.queues from 1 to 4, since it has been appeared that many users hit this limit 2020-09-18 14:22:02 +03:00
Aliaksandr Valialkin
406f4fe445 app/vmagent: substitute -remoteWrite.url with secret-url value in logs, since it may contain sensitive info such as passwords or auth tokens 
Pass `-remoteWrite.showURL` command-line flag in order to see real `-remoteWrite.url` values in logs and at `/metrics` page.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/773
 2020-09-16 22:36:18 +03:00
Aliaksandr Valialkin
6382e8081a app/vmagent: allow setting multiple identical -remoteWrite.url values 
This may be useful when each url is authenticated via different `-remoteWrite.basicAuth.username`.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/755
 2020-09-11 15:17:29 +03:00
Aliaksandr Valialkin
58d3b82ae5 app/{vminsert,vmagent}: allow passing timestamp via timestamp query arg when ingesting data to /api/v1/import/prometheus 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/750
 2020-09-11 13:28:31 +03:00
Aliaksandr Valialkin
5ab57f916b docs/vmagent.md: clarified the case when -remoteWrite.queues must be tuned 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/745
 2020-09-08 20:15:49 +03:00
Aliaksandr Valialkin
a3cdef6b06 app/vmagent: properly flush big blocks of data 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/741

Thanks to @IceRain00 for the investigation and initial attempt to fix the issue
at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/742
 2020-09-03 12:12:12 +03:00
Aliaksandr Valialkin
de216bab41 app/vmagent: fix data race when accessing writeRequest.lastFlushTime 2020-09-03 12:12:09 +03:00
Aliaksandr Valialkin
4fa97430d7 app/{vminsert,vmagent}: allow adding extra labels when importing data via Prometheus, CSV and JSON line formats 
Extra labels may be added to the imported data by passing `extra_label=name=value` query args.
Multiple query args may be passed in order to add multiple extra labels.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/719
 2020-09-02 19:47:02 +03:00
Aliaksandr Valialkin
6f9c1bc078 app/vmagent: log unsuccessful attempt number when sending data to -remoteWrite.url 2020-08-30 21:40:15 +03:00
Aliaksandr Valialkin
3b1ecac04b app/vmagent: apply sane limits to -remoteWrite.queues 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/707
 2020-08-30 21:25:51 +03:00
Roman Khavronenko
4b89da9463 lib/decimal: rename significant decimal digits to significant figures (#698) 
The previous notion was inconsistent with what `decimal.Round` does.
According to [wiki](https://en.wikipedia.org/wiki/Significant_figures) rounding
applied to all significant figures, not just decimal ones.
 2020-08-16 17:22:40 +03:00
Aliaksandr Valialkin
6aab2f4989 all: allow using KB, MB, GB, KiB, MiB and GiB suffixes in command-line flag values related to byte sizes or byte rates 2020-08-16 17:08:28 +03:00
Aliaksandr Valialkin
a2021d0dde docs/vmagent.md: mention that gaps in remote storage may appear if vmagent cannot keep up with data ingestion 2020-08-14 20:48:17 +03:00
Aliaksandr Valialkin
b996280c65 app/{vminsert,vmagent}: improve documentation for -influxListenAddr command-line flag 2020-08-14 18:03:08 +03:00
Aliaksandr Valialkin
60c7397be5 all: support %{ENV_VAR} placeholders in yaml configs in all the vm* components 
Such placeholders are substituted by the corresponding environment variable values.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/583
 2020-08-13 17:17:06 +03:00
Aliaksandr Valialkin
6721e47ae9 app: respect CPU limits set via cgroups 
Update GOMAXPROCS to limits set via cgroups. This should reduce CPU trashing and reduce memory usage
for cases when VictoriaMetrics components run in containers with CPU limits.

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685
 2020-08-11 23:01:03 +03:00
Aliaksandr Valialkin
a930460236 app/vmagent: tune http client for sending data to remote storage in order to disable closing keep-alive connections 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/663
 2020-08-04 21:01:40 +03:00
Aliaksandr Valialkin
106e302d7a all: add mssing APP_NAME to vm*-GOARCH builds 2020-07-31 13:45:32 +03:00
Aliaksandr Valialkin
945645f38f docs/{vmagent,vmalert}: add instruction on how to build for ARM 2020-07-31 09:25:41 +03:00
Aliaksandr Valialkin
f6d4275087 app/{vmagent,vminsert}: properly preserve db tag from query string passed to Influx line protocol query 
Previously `db` tag from the query string wasn't added to metrics after encountering `db` tag in the Influx line

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/653
 2020-07-28 21:25:49 +03:00
Aliaksandr Valialkin
baebe86844 app/vmagent/remotewrite: add missing resp.Body.Close() after pushing data to remote storage 
Missing body close could disable HTTP keep-alive connections.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/653
 2020-07-28 21:00:25 +03:00
Aliaksandr Valialkin
d5057f6d04 app/vmagent/remotewrite: create new request on failure to send a block of data to remote storage 
Previously the request body was already consumed before the retry, so this led to the following error:

    http: ContentLength=... with Body length 0
 2020-07-27 17:33:05 +03:00
Aliaksandr Valialkin
c91ccce50c app/vminsert: fix relabeling for metrics ingested via Influx line protocol 
Previously the enabled relabeling with `-relabelConfig` command-line flag could result in missing labels
if a single Influx line protocol message contains multiple field values.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/638
 2020-07-23 13:25:37 +03:00
Aliaksandr Valialkin
a3f48e395e app/vmagent: add -remoteWrite.decimalPlaces command-line flag, which may be used for reducing disk space usage on the remote storage 2020-07-21 21:55:42 +03:00
Aliaksandr Valialkin
3b246aa569 app/vmagent: add -remoteWrite.proxyURL command-line option 
This option allows writing data to `-remoteWrite.url` via http, https or socks5 proxy.
This is similar to `proxy_url` option in `remote_write` section of Prometheus.
See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write
 2020-07-20 19:31:08 +03:00
Aliaksandr Valialkin
8bee3ef91b docs/vmagent.md: sync with app/vmagent/README.md 2020-07-20 17:09:30 +03:00
Roman Khavronenko
8949ec961d app/vmagent: mention grafana dashboard in README (#639) 2020-07-20 17:09:27 +03:00
Aliaksandr Valialkin
86b54f3768 app/vmagent/remotewrite: allow passing empty -remoteWrite.urlRelabelConfig entries 2020-07-20 15:49:13 +03:00
Aliaksandr Valialkin
31ef39e8da lib/httpserver: log remote address in error message from httpserver.Errorf 
This should improve detection of the root cause of errors.
Thanks to Anant for the idea.
 2020-07-20 14:06:29 +03:00
Aliaksandr Valialkin
e6d96bb0bd docs/vmagent.md: make filtering rules for init container pods less confusing 2020-07-14 20:33:19 +03:00
Aliaksandr Valialkin
c2b4b9138d app/vmagent/remotewrite: return proper value from tssRelabelPool.New 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/599
 2020-07-14 14:28:14 +03:00
Aliaksandr Valialkin
86044f6561 app/{vminsert,vmagent}: add -influxSkipMeasurement command-line flag for using field name as metric name 
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/626
 2020-07-14 14:18:40 +03:00
Aliaksandr Valialkin
b6a5c29549 docs/vmagent.md: sync with app/vmagent/README.md 2020-07-13 21:26:00 +03:00
ofen
9ffa688846 Update README.md (#621) 
Troubleshooting section updated to help out with duplicate targets detection
 2020-07-13 21:25:59 +03:00
Aliaksandr Valialkin
4353ff7ef1 app/vmagent: fix data race when multiple -remoteWrite.urlRelabelConfig options are set 
Previously multiple goroutines could access remoteWriteCtx.tss concurrently, which could lead to data race
and improper relabeling. Now each goroutine has its own copy of tss during relabeling.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/599
 2020-07-10 15:17:23 +03:00
Aliaksandr Valialkin
805a90f642 app/vmagent/remotewrite: typo fix in -remoteWrite.showURL help message 2020-07-10 14:07:14 +03:00
Aliaksandr Valialkin
6373d377ef app/{vminsert,vmagent}: add ability to import data in Prometheus exposition format via /api/v1/import/prometheus 2020-07-10 12:13:28 +03:00
Aliaksandr Valialkin
6ebac3ab63 app/vminsert: add ability to apply relabeling to all the incoming metrics if -relabelConfig command-line arg points to a file with a list of relabel_config entries 
See https://victoriametrics.github.io/#relabeling
 2020-07-02 20:36:33 +03:00
Aliaksandr Valialkin
2361ad8ab4 lib/promscrape: add ability to set disable_compression and disable_keepalive options in scrape_config section of the config passed to -promscrape.config 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/580
 2020-07-02 14:19:34 +03:00
Aliaksandr Valialkin
d962568e93 all: use %w instead of %s for wrapping errors in fmt.Errorf 
This will simplify examining the returned errors such as httpserver.ErrorWithStatusCode .
See https://blog.golang.org/go1.13-errors for details.
 2020-06-30 23:33:46 +03:00
Aliaksandr Valialkin
2b504f17de docs: update the info that docker images are built on top of alpine image now 
A follow-up after the commit ff624c9125
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/522
 2020-06-26 13:52:25 +03:00
Aliaksandr Valialkin
3a444bb7bb lib/promrelabel: add support for keep_if_equal and drop_if_equal actions to relabel configs 
These actions may be useful for filtering out unneeded targets and/or metrics if they contain equal label values.
For example, the following rule would leave the target only if __meta_kubernetes_annotation_prometheus_io_port
equals __meta_kubernetes_pod_container_port_number:

  - action: keep_if_equal
    source_labels: [__meta_kubernetes_annotation_prometheus_io_port, __meta_kubernetes_pod_container_port_number]
 2020-06-23 17:29:19 +03:00
Aliaksandr Valialkin
8ad8ca350a app/vmagent: print brief usage info for vmagent -help 2020-06-05 10:40:10 +03:00
Aliaksandr Valialkin
37aa4fe282 app/vmagent: reload -remoteWrite.relabelConfig and -remoteWrite.urlRelabelConfig on SIGHUP and on /-/reload 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/518
 2020-05-30 14:37:02 +03:00
Aliaksandr Valialkin
a646131a33 app/vmagent: log fatal errors instead of panics when improper command-line flags are passed to vmagent 2020-05-30 14:22:38 +03:00
Aliaksandr Valialkin
dcbdc009f5 app/vmagent: check for error returned from flag.Set 2020-05-21 16:30:48 +03:00
Aliaksandr Valialkin
b59e089ac7 app/vmagent: add -dryRun option for checking all the configs mentioned in command-line flags without running vmagent 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/362
 2020-05-21 15:23:18 +03:00
Aaron France
b26245c48b Update README.md 2020-05-21 12:10:33 +03:00
Aliaksandr Valialkin
ddc9e69bd6 docs/vmagent.md: mention an alternative to refresh_interval option in scrape configs 2020-05-19 23:10:16 +03:00
Aliaksandr Valialkin
2784015a4d all: print --help output to stdout instead of stderr 
This is easier to grep and pipe
 2020-05-16 12:03:06 +03:00
Aliaksandr Valialkin
e544155a82 app/vmagent/Makefile: fix make run-vmagent rule 2020-05-15 19:35:16 +03:00
Aliaksandr Valialkin
6c43ba1cb1 app/vmagent/remotewrite: remove unused import after the commit 93267f143f 2020-05-15 17:42:31 +03:00
Aliaksandr Valialkin
1d71253653 app/vmagent/remotewrite: allow ingesting time series with multiple samples at once 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/481
 2020-05-15 17:37:27 +03:00
Aliaksandr Valialkin
3845420a8f lib: extract common code for returning fast unix timestamp into lib/fasttime 2020-05-14 23:06:50 +03:00
Aliaksandr Valialkin
3b0f66a227 app/vmagent: fix a bug with improper relabeling when multiple -remoteWrite.urlRelableConfig args are set 
This bug could result in incorrect relabeling and metrics' drop.

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/467
 2020-05-12 22:03:45 +03:00
Aliaksandr Valialkin
d54a93fc81 app/vmagent: fix scraping mTLS targets, which has been broken in v1.35.1 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/470
 2020-05-12 17:23:43 +03:00
Aliaksandr Valialkin
405cf44aed app/vmagent,lib/promscrape: do not set HostClient.DialDualStack, since it isnt used if HostClient.Dial is set 2020-05-12 15:24:53 +03:00
Aliaksandr Valialkin
da6a84e147 app/vmagent/remotewrite: properly dial TCP6 addresses set via -remoteWrite.url 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/469
 2020-05-12 15:24:50 +03:00
Aliaksandr Valialkin
28ad350a31 app/vmagent: return 200 from /-/reload endpoint as Prometheus does 2020-05-07 19:29:48 +03:00
Aliaksandr Valialkin
dc04040781 docs/{vmagent,vmauth}: small clarifications in the docs 2020-05-07 12:55:06 +03:00
Aliaksandr Valialkin
20538a2a5d app/vmagent: allow setting independent auth configs per each configured -remoteWrite.url 2020-05-06 16:52:32 +03:00
Aliaksandr Valialkin
12dbb9e22c app/vmagent: properly set client-side TLS certificates for -remoteWrite.url. Previously they were mistakenly set as server-side 2020-05-06 16:50:37 +03:00
Aliaksandr Valialkin
8665c2edb1 docs/vmagent.md: small fixes 2020-05-06 14:49:25 +03:00
Aliaksandr Valialkin
8ab5e47b5c lib/promscrape: add Prometheus-compatible DNS-based service discovery aka dns_sd_configs 2020-05-06 00:02:41 +03:00
Aliaksandr Valialkin
21b91599c2 docs/{vmauth,vmagent}: fix ports for profiling 2020-05-05 20:16:09 +03:00
Aliaksandr Valialkin
20e958789a docs/{vmagent,vmauth}: add Profiling section 2020-05-05 11:45:29 +03:00
Aliaksandr Valialkin
61df59b9ea docs/vmagent.md: /targets page doesnt expose infomration about imporperly configured scrape configs now. It is written in error log instead 2020-05-05 10:56:18 +03:00
Aliaksandr Valialkin
89aa6dbf56 lib/promscrape: add Prometheus-compatible service discovery for Consul aka consul_sd_configs 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/330
 2020-05-04 20:53:06 +03:00
Aliaksandr Valialkin
b21b73115a app/vminsert: add /-/reload handler in the same way as for vmagent 2020-04-30 02:18:08 +03:00
DexterZhang
ae215e5538 feat(vmagent): add promscrap config reload suppport via http (#450) 
* feat(vmagent): add promscrap config reload suppport via http endpoint `/-/reload`

* fix: typo fix
 2020-04-30 02:18:01 +03:00
Aliaksandr Valialkin
b6d88bac04 vendor: use github.com/VictoriaMetrics/fasthttp instead of github.com/fasthttp/fasthttp 
The upstream fasthttp may contain issues like 996610f021 ,
plus a code that isn't used by VictoriaMetrics. So let's use a private copy under our control instead.
 2020-04-29 16:43:09 +03:00
Aliaksandr Valialkin
86a1d9cb0c lib/promscrape: add initial support for Prometheus-compatible service discovery for Amazon EC2 aka ec2_sd_configs 2020-04-27 19:29:22 +03:00
Aliaksandr Valialkin
0daa37fa02 lib/promscrape/discovery/gce: allow empty project and zone for gce_sd_config 2020-04-27 11:45:45 +03:00
Aliaksandr Valialkin
31861c5b8e lib/promscrape/discovery/gce: allow empty zone arg in gce_sd_config - in this case zones for the given project are automatically discovered 2020-04-26 14:37:38 +03:00
Aliaksandr Valialkin
069690e3bd lib/promscrape: initial implementation for gce_sd_configs aga Prometheus-compatible service discovery for Google Compute Engine 2020-04-24 17:53:43 +03:00
Aliaksandr Valialkin
db5fe03170 deployment/docker: allow building docker images on top of any base image set via ROOT_IMAGE environment var 
For example, the following command will build VictoriaMetrics docker image on top of alpine image:

    ROOT_IMAGE=alpine make package-victoria-metrics
 2020-04-20 01:16:21 +03:00
Aliaksandr Valialkin
1b911f6965 app/vmagent/remotewrite: retry sending data if the server closes keep-alive connection 
This should fix the following error when sending data to remote storage:

couldn't send a block with size XX bytes to "YYY": the server closed connection before returning the first response byte. Make sure the server returns 'Connection: close' response header before closing the connection
 2020-04-17 15:53:17 +03:00
Aliaksandr Valialkin
9105f72f17 docs/vmagent.md: typo fix: unvailable -> unavailable 2020-04-17 13:12:13 +03:00
Aliaksandr Valialkin
d46311fd93 app/vmagent/README.md: mention about prodmscrape.suppressScrapeErrors 2020-04-17 13:09:08 +03:00
Aliaksandr Valialkin
6ec582acb9 lib/promscrape: show information on improperly configured scrape targets at the bottom of /targets page 
This is a common error whith improperly configured target autodiscovery and/or relabeling.
This error leads to duplicate scraping of the same targets with the same set of labels, which leads
to duplicate samples in time series.
 2020-04-14 14:55:13 +03:00
Aliaksandr Valialkin
755f649c72 docs/vmagent.md: mention that vmagent supports kubernetes_sd_configs now 2020-04-13 21:07:00 +03:00
Aliaksandr Valialkin
38256bd66d docs: update minimum supported Go version from 1.12 to 1.13 2020-04-07 13:39:15 +03:00
Aliaksandr Valialkin
2b4d3effad app/vmagent/remotewrite: add "X-Prometheus-Remote-Write-Version: 0.1.0" http header to remote_write request 
This header is required by Cortex (and, probably, other remote storage systems).
See 9c1f44d090/docs/apis.md (remote-api) .

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/399
 2020-04-04 16:24:47 +03:00
Aliaksandr Valialkin
ceb6d1459f docs/vmagent.md: add prometheus remote_write proxy use case 2020-03-28 23:17:41 +02:00
Dmitry Naumov
b84071fc25
Rootless docker images by default (#358) 
* Rootless docker images by default

* Migrate to rootless base image

Co-authored-by: Aliaksandr Valialkin <valyala@gmail.com>
 2020-03-27 21:18:32 +02:00
Aliaksandr Valialkin
b46af9678e app/vmagent: mention that vmagent can filter data 2020-03-11 16:23:10 +02:00
Aliaksandr Valialkin
2f0a36044c app/{vmagent,vminsert}: add support for importing csv data via /api/v1/import/csv 2020-03-10 21:17:40 +02:00
Aliaksandr Valialkin
3fc6599aa2 app/vmagent: properly apply -remoteWrite.sendTimeout to fasthttp.HostClient 2020-03-09 13:31:22 +02:00
Aliaksandr Valialkin
47e986c26f app/vmagent: properly add labels set via -remoteWrite.label to metrics before sending them to -remoteWrite.url 2020-03-06 19:28:14 +02:00
Aliaksandr Valialkin
0d893eff36 Makefile: add build and test rules with enabled race detector. These rules have -race suffix 
Fix also `unsafe pointer conversion` errors detected by Go1.14. See https://golang.org/doc/go1.14#compiler .
 2020-03-05 12:05:16 +02:00
Aliaksandr Valialkin
0176fc4206 app/vmagent/README.md: small fixes 2020-03-04 18:15:24 +02:00
Aliaksandr Valialkin
ac03be5a2c app/vmagent/README.md: typo fix 2020-03-04 18:05:43 +02:00
Aliaksandr Valialkin
9354b9177a app/vmagent/README.md: clarification 2020-03-04 18:04:06 +02:00
Aliaksandr Valialkin
4302555228 app/vmagent/README.md: add iot and edge monitoring use case 2020-03-04 18:01:40 +02:00
Aliaksandr Valialkin
ea5904fd76 app/vmagent/README.md: add use cases section 2020-03-04 17:42:57 +02:00
Aliaksandr Valialkin
f01d1bf4a8 app/vmagent: add -remoteWrite.maxDiskUsagePerURL for limiting the maximum disk usage for each -remoteWrite.url buffer 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/352
 2020-03-03 19:49:20 +02:00
Aliaksandr Valialkin
808c17e250 app/vmagent/remotewrite: do not reset empty relabelCtx 2020-03-03 15:01:21 +02:00
Aliaksandr Valialkin
af19ca2483 app/vmagent: add -remoteWrite.urlRelabelConfig for applying individual relabeling for each -remoteWrite.url 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/320
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/308
 2020-03-03 13:13:06 +02:00
Aliaksandr Valialkin
0eed71c7f4 app/vmagent/remotewrite: yet another typo fix 2020-02-28 20:07:00 +02:00
Aliaksandr Valialkin
6cdc97a53f app/vmagent/remotewrite: typo fix 2020-02-28 19:05:11 +02:00
Aliaksandr Valialkin
cc39c9d74b app/vmagent/remotewrite: limit memory usage when big scrape blocks are pushed to remote storage 2020-02-28 18:58:13 +02:00
Aliaksandr Valialkin
0597f1e39a app/vmagent: allow setting -httpListenAddr to empty string in order to disable listening for http requests 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/340
 2020-02-26 20:58:26 +02:00
Aliaksandr Valialkin
266101feb4 app/vmagent/README.md: list service discovery mechanisms, which will be added soon 
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/334
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/330
 2020-02-26 19:28:19 +02:00
Aliaksandr Valialkin
c6c7843e93 app/vmagent: add -remoteWrite.maxBlockSize command-line flag for limiting the maximum size of unpacked block to send to remote storage 2020-02-25 19:58:11 +02:00
Aliaksandr Valialkin
c4194020ef app/vmagent: do not allow sending unpacked requests with sizes exceeding -maxInsertRequestSize 2020-02-25 19:35:43 +02:00
Aliaksandr Valialkin
2471340e0d app/vmagent: add ability to accept Influx line protocol data via TCP and UDP 
Just set `-influxListenAddr` command-line flag

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/333
 2020-02-25 19:18:01 +02:00
Aliaksandr Valialkin
f96fb93ca5 app/vmagent: add a counter for /targets handler calls 2020-02-25 18:17:25 +02:00
Aliaksandr Valialkin
25c570dae7 app/vmagent/README.md: mention that vmagent exposes target statuses at /targets page 2020-02-25 18:16:08 +02:00
Aliaksandr Valialkin
ca28a3e805 app/vmagent: logo fix 2020-02-25 00:09:55 +02:00
Aliaksandr Valialkin
777a39f7a1 app/vmagent: update docs 2020-02-25 00:09:53 +02:00
Aliaksandr Valialkin
61e67b8922 app/vmagent/README.md: small fixes 2020-02-24 21:26:12 +02:00
Aliaksandr Valialkin
7ee7614e90 app/vmagent: initial implementation for vmagent 2020-02-23 17:31:54 +02:00