github-mirrors/VictoriaMetrics
1
0
Fork 0
mirror of https://github.com/VictoriaMetrics/VictoriaMetrics.git synced 2025-03-21 15:45:01 +00:00
Code Issues Projects Releases Packages Wiki Activity
VictoriaMetrics/lib
History
Max Kotliar c05ffa906d
lib/promscrape: improve streamParse performance 
Previously, performance of stream.Parse could be limited by mutex.Lock on callback function. It used shared writeContext. With complicated relabeling rules and any slowness at pushData function, it could significantly decrease parsed rows processing performance.

 This commit removes locks and makes parsed rows processing lock-free in the same manner as `stream.Parse` processing implemented at push ingestion processing.

 Implementation details:
- Removing global lock around stream.Parse callback.
- Using atomic operations for counters
- Creating write contexts per callback instead of sharing
- Improving series limit checking with sync.Once
- Optimizing labels hash calculation with buffer pooling
- Adding comprehensive tests for concurrency correctness

 Benchmark performance:
```
# before
BenchmarkScrapeWorkScrapeInternalStreamBigData-10             13          81973945 ns/op          37.68 MB/s    18947868 B/op        197 allocs/op

# after
goos: darwin
goarch: arm64
pkg: github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape
cpu: Apple M1 Pro
BenchmarkScrapeWorkScrapeInternalStreamBigData-10             74          15761331 ns/op         195.98 MB/s    15487399 B/op        148 allocs/op
PASS
ok      github.com/VictoriaMetrics/VictoriaMetrics/lib/promscrape       1.806s
```

Related issue:
 https://github.com/VictoriaMetrics/VictoriaMetrics/issues/8159
---------
Signed-off-by: Maksim Kotlyar <kotlyar.maksim@gmail.com>
Co-authored-by: Roman Khavronenko <hagen1778@gmail.com>
2025-03-20 16:49:24 +01:00
..
appmetrics all: add -metrics.exposeMetadata command-line flag, which can be used for adding TYPE and HELP metadata for metrics exposed at /metrics page 2023-12-19 03:20:40 +02:00
auth lib/auth: add NewTokenPossibleMultitenant() for parsing auth token, which can be multitenant 2023-08-30 14:17:55 +02:00
awsapi spelling and grammar fixes via codespell () 2025-03-17 16:32:10 +01:00
backup lib/backup/s3remote: add retries for "IncompleteBody" errors 2025-03-20 12:43:18 +01:00
blockcache all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:20:37 +02:00
bloomfilter lib: consistently use atomic.* types instead of atomic.* functions 2024-02-24 02:07:53 +02:00
bufferedwriter app/vmselect: properly cancel long running requests on client connection close 2025-01-29 16:09:32 +01:00
buildinfo all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00
bytesutil spelling and grammar fixes via codespell () 2025-03-17 16:32:10 +01:00
cgroup lib/cgroup: warn users about using fractional CPU quotas () 2025-01-29 13:19:08 +01:00
chunkedbuffer lib/chunkedbuffer: add Buffer.Len() method, which returns the byte length of the data stored in the buffer 2025-03-19 13:24:39 +01:00
contextutil lib/contextutil: make golanci-lint happy by substituing unused function arg name with _ 2024-09-26 17:06:48 +02:00
decimal spelling and grammar fixes via codespell () 2025-03-17 16:32:10 +01:00
encoding spelling and grammar fixes via codespell () 2025-03-17 16:32:10 +01:00
envflag lib/envflag: do not allow unsupported form for boolean command-line flags in the form -boolFlag value 2023-08-17 13:26:53 +02:00
envtemplate allowed using dashes and dots in environment variables names () 2023-03-24 15:43:05 -07:00
fastnum lib/fastnum: use unsafe.Slice() instead of deprecated reflect.SliceHeader 2024-02-29 17:17:13 +02:00
fasttime lib: consistently use atomic.* types instead of atomic.* functions 2024-02-24 02:07:53 +02:00
filestream lib/filestream: use smaller sizes for read buffers than for write buffers 2025-02-19 12:35:05 +01:00
flagutil app/vlinsert: follow-up for 37ed1842ab 2025-03-15 00:03:03 +01:00
formatutil app/vmbackupmanager: add metrics for better observability () 2022-12-20 14:18:06 -08:00
fs spelling and grammar fixes via codespell () 2025-03-17 16:32:10 +01:00
htmlcomponents lib/htmlcomponents: use relative links for the top page and for favicon.ico 2023-11-13 20:29:05 +01:00
httpserver spelling and grammar fixes via codespell () 2025-03-17 16:32:10 +01:00
httputils lib/httputils: always set up TLS config 2025-03-13 23:27:49 +01:00
influxutils app/{vminsert,vmagent}: add healthcheck for influx ingestion endpoints () 2024-08-05 09:34:54 +02:00
ingestserver Revert c6c5a5a186 and b2765c45d0 2024-07-03 23:51:56 +02:00
leveledbytebufferpool lib/leveledbytebufferpool: do not pool byte slices bigger than 2^18 bytes 2024-06-13 16:56:25 +02:00
logger app/vlogscli: add interactive command-line tool for querying VictoriaLogs 2024-10-01 12:23:07 +02:00
logstorage lib/logstorage: typo fix in the comment to Storage.GetStreamFieldValues() function 2025-03-19 13:22:16 +01:00
lrucache all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:20:37 +02:00
memory all: cleanup: remove // +build ... lines, since they are no longer needed after Go1.17, and the minimum supported Go version for VictoriaMetrics source code is Go1.20 2023-11-13 19:12:51 +01:00
mergeset spelling and grammar fixes via codespell () 2025-03-17 16:32:10 +01:00
metricsql all: make fmt via the upcoming Go1.19 2022-07-11 19:22:15 +03:00
netutil spelling and grammar fixes via codespell () 2025-03-17 16:32:10 +01:00
persistentqueue spelling and grammar fixes via codespell () 2025-03-17 16:32:10 +01:00
procutil spelling and grammar fixes via codespell () 2025-03-17 16:32:10 +01:00
promauth spelling and grammar fixes via codespell () 2025-03-17 16:32:10 +01:00
prompb spelling and grammar fixes via codespell () 2025-03-17 16:32:10 +01:00
prompbmarshal lib/prompbmarshal: move MustParsePromMetrics to protoparser/prometheus () 2025-02-27 22:50:27 +01:00
promrelabel lib/promrelabel: comment typo () 2025-03-17 16:40:03 +01:00
promscrape lib/promscrape: improve streamParse performance 2025-03-20 16:49:24 +01:00
promutils lib/promutils: move time-related funcs from promutils to timeutil () 2025-03-03 10:25:42 +01:00
protoparser lib/promscrape: improve streamParse performance 2025-03-20 16:49:24 +01:00
proxy lib/promscrape: fixes proxy autorization () 2024-08-19 22:31:18 +02:00
pushmetrics lib/pushmetrics: wait until the background goroutines, which push metrics, are stopped at pushmetrics.Stop() 2024-01-15 13:50:36 +02:00
querytracer app/vmselect: fixes panic data race at query tracing 2025-01-24 13:56:09 +01:00
ratelimiter app/vmagent: properly shutdown when -maxIngestionRate limit is reached 2024-03-30 06:43:48 +02:00
regexutil spelling and grammar fixes via codespell () 2025-03-17 16:32:10 +01:00
slicesutil lib/slicesutil: add helper functions for setting slice length and extending its capacity 2024-05-12 11:32:17 +02:00
snapshot spelling and grammar fixes via codespell () 2025-03-17 16:32:10 +01:00
storage spelling and grammar fixes via codespell () 2025-03-17 16:32:10 +01:00
streamaggr lib/streamaggr: fix threshold update, when deduplication and windows are enabled () 2025-03-20 09:54:35 +01:00
stringsutil spelling and grammar fixes via codespell () 2025-03-17 16:32:10 +01:00
syncwg all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00
tenantmetrics lib/tenantmetrics: improves CounterMap performance with large numbers of tenants 2024-11-20 18:42:47 +01:00
timerpool lib/timerpool: use timer pool in concurrency limiters 2019-05-28 17:20:10 +03:00
timeserieslimits lib/timeserieslimits: follow-up for 564e6ea024 2024-12-11 11:00:27 +01:00
timeutil lib/timeutil: add test for ParseDuration 2025-03-03 10:46:01 +01:00
uint64set spelling and grammar fixes via codespell () 2025-03-17 16:32:10 +01:00
workingsetcache lib: consistently use atomic.* types instead of atomic.* functions 2024-02-24 02:07:53 +02:00
writeconcurrencylimiter app/vmagent/remotewrite: clarify the reason behind the default value for -remoteWrite.queues in the same way as the reason for -maxConcurrentInserts is defined at 73f5fb0f0c 2024-03-06 13:43:08 +02:00