VictoriaMetrics/lib/logstorage
Zakhar Bessarab 5b3cbd4db1
app/vlinsert: add support of loki push protocol (#4482)
* app/vlinsert: add support of loki push protocol

- implemented loki push protocol for both Protobuf and JSON formats
- added examples in documentation
- added example docker-compose

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vlinsert: move protobuf metric into its own file

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* deployment/docker/victorialogs/promtail: update reference to docker image

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* deployment/docker/victorialogs/promtail: make volume name unique

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vlinsert/loki: add license reference

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* deployment/docker/victorialogs/promtail: fix volume name

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* docs/VictoriaLogs/data-ingestion: add stream fields for loki JSON ingestion example

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vlinsert/loki: move entities to places where those are used

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vlinsert/loki: refactor to use common components

- use CommonParameters from insertutils
- stop ingestion after first error similar to elasticsearch and jsonline

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* app/vlinsert/loki: address review feedback

- add missing logstorage.PutLogRows calls
- refactor tenant ID parsing to use common function
- reduce number of allocations for parsing by reusing  logfields slices
- add tests and benchmarks for requests processing funcs

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
2023-07-20 16:49:43 -07:00
..
arena.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
block.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
block_data.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
block_data_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
block_header.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
block_header_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
block_search.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
block_stream_merger.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
block_stream_reader.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
block_stream_writer.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
block_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
block_timing_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
bloomfilter.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
bloomfilter_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
consts.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
datadb.go lib/logstorage: fix panic (#4620) 2023-07-13 12:04:59 -07:00
datadb_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
encoding.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
encoding_test.go lib/logstorage: fix make test-pure tests 2023-07-06 22:22:08 -07:00
encoding_timing_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
filenames.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
filters.go lib/logstorage: LogsQL: replace exact_prefix("...") with exact("..."*) 2023-07-17 17:19:45 -07:00
filters_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
hash128.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
hash128_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
hash128_timing_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
index_block_header.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
index_block_header_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
indexdb.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
indexdb_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
inmemory_part.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
inmemory_part_test.go lib/logstorage: go fmt 2023-07-06 22:24:18 -07:00
inmemory_part_timing_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
log_rows.go app/victoria-logs: add ability to debug data ingestion by passing debug query arg to data ingestion API 2023-07-06 21:19:58 -07:00
log_rows_timing_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
parser.go lib/logstorage: properly encode "offset" search word just after _time filter 2023-07-18 16:03:57 -07:00
parser_test.go vmui: add Active Queries page (#4653) 2023-07-19 16:02:58 -07:00
part.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
part_header.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
part_header_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
partition.go app/victoria-logs: add ability to debug data ingestion by passing debug query arg to data ingestion API 2023-07-06 21:19:58 -07:00
partition_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
rows.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
rows_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
storage.go lib/logstorage: log the -retentionPeriod and -futureRetention values when the ingested log entry has timestamp outside the configured retention 2023-07-17 18:23:45 -07:00
storage_search.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
storage_search_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
storage_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
stream_filter.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
stream_id.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
stream_id_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
stream_tags.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
tenant_id.go app/vlinsert: add support of loki push protocol (#4482) 2023-07-20 16:49:43 -07:00
tenant_id_test.go app/vlinsert: add support of loki push protocol (#4482) 2023-07-20 16:49:43 -07:00
tokenizer.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
tokenizer_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
tokenizer_timing_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
u128.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
u128_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
values_encoder.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00
values_encoder_test.go lib/logstorage: fix TestValuesEncoder() on 32-bit architectures 2023-07-13 11:28:04 -07:00
values_encoder_timing_test.go app/victoria-logs: initial code release 2023-07-06 17:30:05 -07:00