VictoriaMetrics/lib
Aliaksandr Valialkin 30098ac8bd
app/vlinsert/loki: follow-up after 09df5b66fd
- Parse protobuf if Content-Type isn't set to `application/json` - this behavior is documented at https://grafana.com/docs/loki/latest/api/#push-log-entries-to-loki

- Properly handle gzip'ped JSON requests. The `gzip` header must be read from `Content-Encoding` instead of `Content-Type` header

- Properly flush all the parsed logs with the explicit call to vlstorage.MustAddRows() at the end of query handler

- Check JSON field types more strictly.

- Allow parsing Loki timestamp as floating-point number. Such a timestamp can be generated by some clients,
  which store timestamps in float64 instead of int64.

- Optimize parsing of Loki labels in Prometheus text exposition format.

- Simplify tests.

- Remove lib/slicesutil, since there are no more users for it.

- Update docs with missing info and fix various typos. For example, it should be enough to have `instance` and `job` labels
  as stream fields in most Loki setups.

- Allow empty of missing timestamps in the ingested logs.
  The current timestamp at VictoriaLogs side is then used for the ingested logs.
  This simplifies debugging and testing of the provided HTTP-based data ingestion APIs.

The remaining MAJOR issue, which needs to be addressed: victoria-logs binary size increased from 13MB to 22MB
after adding support for Loki data ingestion protocol at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/4482 .
This is because of shitty protobuf dependencies. They must be replaced with another protobuf implementation
similar to the one used at lib/prompb or lib/prompbmarshal .
2023-07-20 21:52:11 -07:00
..
appmetrics all: replace atomic.Value with atomic.Pointer[T] 2023-07-19 17:48:26 -07:00
auth app/vminsert: allows parsing tenant id from labels (#3009) 2022-09-30 17:28:35 +03:00
awsapi .golangci.yml: properly enable revive linter and fix all the warnings it detects 2023-02-26 12:19:58 -08:00
backup fix removing storage data dir before restoring from backup (#598) 2023-07-06 22:32:12 -07:00
blockcache all: remove explicit "xxhash" name when importing github.com/cespare/xxhash/v2 package 2022-06-21 20:24:28 +03:00
bloomfilter all: replace atomic.Value with atomic.Pointer[T] 2023-07-19 17:48:26 -07:00
bufferedwriter app/vmselect: move common http functionality from app/vmselect/searchutils to lib/httputils 2023-07-06 17:22:23 -07:00
buildinfo all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00
bytesutil docs: follow-up after aec4b5db81 (#4638) 2023-07-19 14:48:17 -07:00
cgroup lib/cgroup: add SetGOGC() function 2023-07-06 17:24:31 -07:00
consts app/vminsert: reduce the max packet size, which vminsert can send to vmstorage 2022-04-05 15:39:58 +03:00
decimal lib/decimal: use consistent randomizer in tests 2023-01-23 19:24:05 -08:00
encoding all: replace atomic.Value with atomic.Pointer[T] 2023-07-19 17:48:26 -07:00
envflag docs: follow-up after aec4b5db81 (#4638) 2023-07-19 14:48:17 -07:00
envtemplate allowed using dashes and dots in environment variables names (#4009) 2023-03-24 17:57:19 -07:00
fastnum Makefile: add build and test rules with enabled race detector. These rules have -race suffix 2020-03-05 12:05:16 +02:00
fasttime lib: extract common code for returning fast unix timestamp into lib/fasttime 2020-05-14 23:06:50 +03:00
filestream lib/filestream: change Create() to MustCreate() 2023-04-14 15:14:24 -07:00
flagutil lib/flagutil: ArrayString: support commas inside quoted strings and inside [], {} and () braces 2023-03-28 21:25:07 -07:00
formatutil app/vmbackupmanager: add metrics for better observability (#488) 2022-12-20 14:18:43 -08:00
fs lib/fs: add ReaderAt.Path() function 2023-07-06 17:25:19 -07:00
handshake lib/handshake: do not pollute logs with cannot read hello messages on TCP health checks 2023-05-18 10:37:59 -07:00
htmlcomponents app/vmselect: remove dependency on lib/promscrape from app/vmselect 2023-01-03 23:27:36 -08:00
httpserver docs: make httpAuth.* flags description less ambiguous (#4588) 2023-07-09 12:36:14 -07:00
httputils lib/httputils: fix test after b49d04b3dc 2023-07-06 22:21:43 -07:00
influxutils lib/flagutil: rename Array to ArrayString 2022-10-01 18:28:19 +03:00
ingestserver lib/netutil: init implimentation of proxy protocol (#3687) 2023-01-26 23:25:22 -08:00
leveledbytebufferpool all: make fmt via the upcoming Go1.19 2022-07-11 19:23:25 +03:00
logger add error handler for parsing prometheus text format to vmagent and v… (#3693) 2023-01-23 22:36:23 -08:00
logjson app/vlinsert/jsonline: code prettifying 2023-07-06 21:35:55 -07:00
logstorage app/vlinsert/loki: follow-up after 09df5b66fd 2023-07-20 21:52:11 -07:00
lrucache all: remove explicit "xxhash" name when importing github.com/cespare/xxhash/v2 package 2022-06-21 20:24:28 +03:00
memory docs: follow-up after aec4b5db81 (#4638) 2023-07-19 14:48:17 -07:00
mergeset lib/mergeset: skip common prefix in binarySearchKey() function 2023-07-13 22:05:14 -07:00
metricsql all: make fmt via the upcoming Go1.19 2022-07-11 19:23:25 +03:00
netutil docs: follow-up after aec4b5db81 (#4638) 2023-07-19 14:48:17 -07:00
persistentqueue app/vmagent,lib/persistentqueue: show warning message if --remoteWrite.maxDiskUsagePerURL flag lower than 500MB (#4196) 2023-05-08 15:45:21 -07:00
procutil lib/procutil: stop immediately after receiving the second SIGINT or SIGTERM signal 2022-10-20 21:58:49 +03:00
promauth lib/promscrape: disable support for service discovery and metrics scrape via http2 2023-07-06 16:04:31 -07:00
prompb app/vminsert: moved -maxInsertRequestSize command-line flag out of lib/prompb in order to prevent its inclusion in vmselect and vmstorage apps 2020-01-28 22:53:50 +02:00
prompbmarshal all: use %w instead of %s for wrapping errors in fmt.Errorf 2020-06-30 23:33:46 +03:00
promrelabel all: add support for or filters in series selectors 2023-07-15 23:56:18 -07:00
promscrape app/vmagent: fix creating target id if --promscrape.dropOriginalLabels flag was used (#4616) 2023-07-20 19:21:41 -07:00
promutils vmalert: init unit test (#4596) 2023-07-20 21:19:45 -07:00
protoparser lib/promutils: properly return error when incorrect Prometheus label names are passed to NewLabelsFromString() 2023-05-12 17:02:06 -07:00
proxy lib/promauth: add ability to send additional http headers in requests to scrape targets 2022-06-22 20:40:50 +03:00
pushmetrics fixed typos in documentation and commandline flags descriptions (#4275) 2023-05-10 02:22:06 -07:00
querytracer lib/querytracer: fix remaining tests after 49ebc48809 2022-12-08 18:18:50 -08:00
regexutil app,lib: fix typos in comments (#3804) 2023-02-13 09:32:35 -08:00
snapshot app/vmbackup: prevent password leaks (#3672) 2023-01-18 11:40:52 -08:00
storage all: replace atomic.Value with atomic.Pointer[T] 2023-07-19 17:48:26 -07:00
streamaggr allow configuring staleness interval in stream aggregation (#4667) (#4670) 2023-07-20 21:47:29 -07:00
syncwg all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00
tenantmetrics app/vminsert: allows parsing tenant id from labels (#3009) 2022-09-30 17:28:35 +03:00
timerpool lib/timerpool: use timer pool in concurrency limiters 2019-05-28 17:30:10 +03:00
uint64set lib/uint64set: use repeatable randomizer in tests 2023-01-23 19:24:05 -08:00
vmselectapi lib/vmselectapi: move the code for checking the expected client errors into a isExpectedError() function 2023-07-06 16:37:59 -07:00
workingsetcache all: replace atomic.Value with atomic.Pointer[T] 2023-07-19 17:48:26 -07:00
writeconcurrencylimiter lib/writeconcurrencylimiter: initialize concurrencyLimitCh before exporting vm_concurrent_insert_capacity and vm_concurrent_insert_current metrics 2023-02-07 11:08:39 -08:00