Nikolay Khramchikhin
fb356c434b
Changed s3 configProfile flag default, ( #749 )
...
aws sdk has complicated logic for chosing profile name and we shouldn't set
it to `default` value. It leads to bugs and improper configuration.
Set it to empty value by default is safe. It will be automatically set to `default` by sdk.
2020-09-07 21:55:45 +03:00
Aliaksandr Valialkin
d2f6f96e4a
lib/memory: fall back to reading hierarchical memory limit in cgroups when the default limit isn't set
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/699
2020-09-04 00:04:57 +03:00
Aliaksandr Valialkin
281d715060
lib/httpserver: add -http.connTimeout command-line flag for limiting the lifetime for incoming http connections
...
This can be useful for balancing incoming connections among multiple services.
2020-09-03 22:23:55 +03:00
Aliaksandr Valialkin
4fa97430d7
app/{vminsert,vmagent}: allow adding extra labels when importing data via Prometheus, CSV and JSON line formats
...
Extra labels may be added to the imported data by passing `extra_label=name=value` query args.
Multiple query args may be passed in order to add multiple extra labels.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/719
2020-09-02 19:47:02 +03:00
Aliaksandr Valialkin
95ce89e7d7
lib/promscrape: use the number of parsed rows as a basis for writeRequestCtxPool leveling
...
The previous basis on `cap(sw.labels)` doesn't work anymore after 7785869ccc
2020-09-02 18:46:55 +03:00
Aliaksandr Valialkin
bc1ca4b20b
lib/httpserver: add -http.idleConnTimeout command-line flag for tuning the timeout for incoming idle http connections
2020-09-01 15:33:31 +03:00
Aliaksandr Valialkin
a01c56104a
lib/promscrape: fix applying sample_limit when scraping targets with big number of metrics
...
This has been broken at 7785869ccc
2020-09-01 11:09:25 +03:00
Aliaksandr Valialkin
deff8d419a
lib/promscrape: reduce memory usage when scraping targets with millions of metrics
...
This should help when scraping /federate endpoints from Prometheus instances,
which scrape millions of metrics. See https://prometheus.io/docs/prometheus/latest/federation/
2020-09-01 10:55:24 +03:00
Aliaksandr Valialkin
5f2277624a
lib/{promscrape,leveledbytebufferpool}: rename getPoolIdAndCapacity to getPoolIDAndCapacity in order to make golint happy
2020-08-28 09:49:22 +03:00
Aliaksandr Valialkin
45e770ed20
lib/cgroup: limit the maximum GOMAXPROCS value to the number of available CPU cores
...
There is no sense in setting GOMAXPROCS to value higher than the number of available CPU cores.
2020-08-28 09:49:22 +03:00
Roman Khavronenko
34ef10fbcc
lib/flagutil: avoid int overflow for arch 386 ( #710 )
...
Arch 386 is a 32-bit architecture and interprets int type for numbers as an explicit int32,
whereas on most modern CPUs int is implicitly an int64. This makes tests to fail with
`int overflow` error.
2020-08-28 09:46:35 +03:00
Aliaksandr Valialkin
9a77ae9d1c
lib/promscrape: reduce memory usage when scraping targets with big number of metrics alongside targets with small number of labels
...
Previously targets with big number of metrics and/or labels could generated too big buffers,
which then could be re-used when scraping targets with small number of metrics.
This resulted in memory waste.
Now big buffers are used only for targets with big number of metrics / labels,
while small buffers are used for targets with small number of metrics / labels.
2020-08-16 22:30:34 +03:00
Aliaksandr Valialkin
3ea6444219
lib/leveledbytebufferpool: allocate byte buffers with capacity rounded to the upper boundary for the given bucket
...
This should reduce the number of resizings for the returned byte buffers.
2020-08-16 22:13:38 +03:00
Roman Khavronenko
4b89da9463
lib/decimal: rename significant decimal digits to significant figures ( #698 )
...
The previous notion was inconsistent with what `decimal.Round` does.
According to [wiki](https://en.wikipedia.org/wiki/Significant_figures ) rounding
applied to all significant figures, not just decimal ones.
2020-08-16 17:22:40 +03:00
Aliaksandr Valialkin
6aab2f4989
all: allow using KB, MB, GB, KiB, MiB and GiB suffixes in command-line flag values related to byte sizes or byte rates
2020-08-16 17:08:28 +03:00
Aliaksandr Valialkin
1b5467f7fd
lib/memory: improve log message about the memory allowed to use by VictoriaMetrics
2020-08-16 17:08:27 +03:00
Aliaksandr Valialkin
d9f7ea1c6e
lib/protoparser: removed unnecessary call to SetReadDeadline when reading a stream of data
...
The OS should return any buffered data in the stream without the need to set the read timeout.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/696
2020-08-15 15:38:35 +03:00
Aliaksandr Valialkin
edf3ee2a9b
lib: dump compressed block contents on error during decompression
...
This should improve detecting root cause for https://github.com/facebook/zstd/issues/2222
2020-08-15 14:51:14 +03:00
Aliaksandr Valialkin
6e863376f7
lib/leveledbytebufferpool: pre-allocate byte slice with the given capacity if the pool is empty
...
This should reduce memory allocations and copying when the byte slice is growing.
2020-08-15 01:41:59 +03:00
Aliaksandr Valialkin
3efa4e4e1c
lib/protoparser: move common code for detecting timeouts to ReadLinesBlockExt
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/696
2020-08-14 20:39:51 +03:00
Aliaksandr Valialkin
c6b0547847
lib/protoparser: prevent from busy loop on repeated timeout errors when reading streams of ingested data
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/696
2020-08-14 20:13:37 +03:00
Aliaksandr Valialkin
9d79a3a99d
lib/memory: add -memory.allowedBytes command-line flag for setting absolute memory limit for VictoriaMetrics caches
2020-08-14 19:19:10 +03:00
Aliaksandr Valialkin
b996280c65
app/{vminsert,vmagent}: improve documentation for -influxListenAddr command-line flag
2020-08-14 18:03:08 +03:00
Aliaksandr Valialkin
c82a485cf6
lib/protoparser/prometheus: typo fix in error message
2020-08-14 11:04:15 +03:00
Aliaksandr Valialkin
9e67343756
lib/promscrape: use a hint on body length instead of body capacity
...
This should reduce memory usage for https://github.com/VictoriaMetrics/VictoriaMetrics/issues/689
2020-08-14 01:17:46 +03:00
Aliaksandr Valialkin
b4119bb51e
lib/promscrape: reduce memory usage when scraping big number of targets
...
Thanks to @dxtrzhang for the original idea at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/688
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/689
2020-08-14 01:05:04 +03:00
Aliaksandr Valialkin
1724cc241e
lib/promscrape: properly retry requests on the server closed connection before returning the first response byte error during service discover API calls and target scrapes
2020-08-13 22:32:29 +03:00
Aliaksandr Valialkin
60c7397be5
all: support %{ENV_VAR} placeholders in yaml configs in all the vm* components
...
Such placeholders are substituted by the corresponding environment variable values.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/583
2020-08-13 17:17:06 +03:00
Aliaksandr Valialkin
6721e47ae9
app: respect CPU limits set via cgroups
...
Update GOMAXPROCS to limits set via cgroups. This should reduce CPU trashing and reduce memory usage
for cases when VictoriaMetrics components run in containers with CPU limits.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/685
2020-08-11 23:01:03 +03:00
Aliaksandr Valialkin
59d95961b8
lib/protoparser: clarify that the string passed to Unmarshal() function must remain available when the parsed rows are in use
2020-08-11 17:05:21 +03:00
Aliaksandr Valialkin
890cfe5b61
lib/protoparser/influx: accept precision=us and precision=µ according to https://docs.influxdata.com/influxdb/v1.8/tools/api/#write-http-endpoint
2020-08-10 20:23:20 +03:00
Aliaksandr Valialkin
e3439a6cd0
lib/promscrape: optimize per-metric hash calculations
...
This increases vmagent performance by up to 10% when scraping big number of metrics
2020-08-10 19:47:50 +03:00
Aliaksandr Valialkin
4ce1368e4b
lib/storage: mention time range used in the query that led to error message
...
This should improve detecting slow queries with too big time ranges
2020-08-10 13:46:29 +03:00
Aliaksandr Valialkin
f92255e803
lib/storage: mention tag filters used in the query that led to error message
...
This should improve detecting invalid or heavy queries that lead to errors.
2020-08-10 13:36:54 +03:00
Aliaksandr Valialkin
b3d4ff7ee2
app/vmstorage: improve error logging when the request times out
2020-08-10 13:17:24 +03:00
Aliaksandr Valialkin
e3999ac010
lib/promscrape: show real timestamp and real duration for the scape on /targets page
...
Previously the scrape duration may be negative when calculated scrape timestamp drifts away from the real scrape timestamp
2020-08-10 12:40:49 +03:00
Aliaksandr Valialkin
c09c881264
lib/promscrape: make errcheck happy
2020-08-09 13:17:30 +03:00
Aliaksandr Valialkin
2dfb42a8b4
lib/promscrape: export scrape_samples_added per-target metric like Prometheus does
...
This metric may be useful for detecting targets with high churn rate for the exported metrics.
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/683
2020-08-09 12:45:30 +03:00
Aliaksandr Valialkin
fd9f1463df
lib/fs: use WARN instead of ERROR log level for the message when NFS diretory removal temporarily fails
...
this is expected condition, so it is better to use WARN log level for it
2020-08-09 12:07:35 +03:00
Aliaksandr Valialkin
d4be3efc60
lib/promscrape: add a test for scrape config for blackbox exporter
...
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/684
2020-08-09 12:03:51 +03:00
Aliaksandr Valialkin
67cacb22ac
lib/httpserver: add -tls, -tlsCertFile and -tlsKeyFile command-line flags in every vm binary
...
This makes such binaries compatible with binaries from `master` branch (aka single-node version)
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/677
2020-08-07 10:57:32 +03:00
Aliaksandr Valialkin
307281e922
lib/storage: slow down concurrent searches when the number of concurrent inserts reaches the limit
...
This should improve data ingestion performance when heavy searches are executed
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/618
2020-08-07 08:49:13 +03:00
Aliaksandr Valialkin
dd1d59f57a
lib/storage: properly check timeouts and pace limits
...
Previously they were checked on every iteration for small number of iterations
2020-08-07 08:40:56 +03:00
Aliaksandr Valialkin
a2039b3bbc
app/vmselect: return the upper bound on the number of found time series from storage.Search.Init
...
This is used by a single-node version in order to reduce memory allocations during search.
See bc8381613d
2020-08-06 19:20:31 +03:00
Aliaksandr Valialkin
b690eeff53
lib/storage: reduce the frequency (and overhead) for timeout and pace limiter checks by 4x
2020-08-06 18:45:47 +03:00
Aliaksandr Valialkin
6c0a92a1ee
lib/pacelimiter: increase scalability for multi-CPU system
2020-08-06 18:33:07 +03:00
Aliaksandr Valialkin
13f8644f8e
lib/storage: optimize prefetching metric names for the given metricIDs
2020-08-06 16:52:58 +03:00
Aliaksandr Valialkin
f789e0fa44
lib/fs: export vm_nfs_pending_dirs_to_remove metric for monitoring the number of pending directories that couldn't be removed due to NFS lock
2020-08-06 15:31:50 +03:00
Aliaksandr Valialkin
a3e91c593b
lib/storage: limit the number of concurrent calls to storage.searchTSIDs to GOMAXPROCS*2
...
This should limit the maximum memory usage and reduce CPU trashing on vmstorage
when multiple heavy queries are executed.
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/648
2020-08-05 18:27:21 +03:00
Aliaksandr Valialkin
76064ba9e7
Perform conversion from string to []byte according to rule #6 at https://golang.org/pkg/unsafe/#Pointer
2020-08-05 11:55:12 +03:00
