VictoriaMetrics/lib
Nikolay e45556fc05
lib/logstorage: fixes panic at Block.MustInitFromRows (#7695)
Previously Block columns wasn't properly limited by maxColumnsPerBlock.
And it was possible a case, when more columns per block added than
expected.
 For example, if ingested log stream has many unuqie fields
and it's sum exceed maxColumnsPerBlock.
 We only enforce fieldsPerBlock limit during row parsing, which limits
isn't enough to mitigate this issue. Also it
would be very expensive to apply maxColumnsPerBlock limit during
ingestion, since it requires to track all possible field tags
combinations.

 This commit adds check for maxColumnsPerBlock limit during
MustInitFromRows function call. And it returns offset of the rows and
timestamps added to the block.
 Function caller must create another block and ingest remaining rows
into it.

Related issue:
https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7568

### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: f41gh7 <nik@victoriametrics.com>
Co-authored-by: Aliaksandr Valialkin <valyala@victoriametrics.com>
2024-11-30 18:06:58 +01:00
..
appmetrics all: add -metrics.exposeMetadata command-line flag, which can be used for adding TYPE and HELP metadata for metrics exposed at /metrics page 2023-12-19 03:26:02 +02:00
auth vmselect: add support of multi-tenant queries (#6346) 2024-10-01 16:37:18 +02:00
awsapi lib/awsapi: properly assume role with webIdentity token (#5495) 2023-12-20 19:07:04 +02:00
backup lib/backup/s3remote: add retryer configuration (#6747) 2024-08-07 16:59:23 +02:00
blockcache all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
bloomfilter lib: consistently use atomic.* types instead of atomic.* functions 2024-02-24 02:10:04 +02:00
bufferedwriter app/vmselect: move common http functionality from app/vmselect/searchutils to lib/httputils 2023-07-06 17:22:23 -07:00
buildinfo all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00
bytesutil lib/bytesutil: smooth buffer growth rate (#6761) 2024-08-07 16:59:23 +02:00
cgroup lib/cgroup: round GOMAXPROCS to the lower integer value of cpuQuota 2024-09-23 16:11:59 +02:00
consts app/vminsert: reduce the max packet size, which vminsert can send to vmstorage 2022-04-05 15:39:58 +03:00
contextutil lib/contextutil: make golanci-lint happy by substituing unused function arg name with _ 2024-09-26 17:07:25 +02:00
decimal lib/slicesutil: add helper functions for setting slice length and extending its capacity 2024-05-12 11:33:49 +02:00
encoding lib/encoding: optimize UnmarshalVarUint64, UnmarshalVarInt64 and UnmarshalBytes a bit 2024-05-14 01:30:25 +02:00
envflag lib/envflag: do not allow unsupported form for boolean command-line flags in the form -boolFlag value 2023-08-17 13:37:05 +02:00
envtemplate allowed using dashes and dots in environment variables names (#4009) 2023-03-24 17:57:19 -07:00
envutil testing: allow disabling fsync to make tests run faster (#6871) 2024-08-30 11:18:21 +02:00
fastnum lib/fastnum: use unsafe.Slice() instead of deprecated reflect.SliceHeader 2024-02-29 17:17:24 +02:00
fasttime lib: consistently use atomic.* types instead of atomic.* functions 2024-02-24 02:10:04 +02:00
filestream vlinsert: added opentelemetry logs support 2024-09-03 20:24:01 +02:00
flagutil docs: follow-up after f0d1db81dc 2024-10-17 11:19:03 -03:00
formatutil app/vmbackupmanager: add metrics for better observability (#488) 2022-12-20 14:18:43 -08:00
fs fscore: rollback trailing space trim (#7106) 2024-09-29 14:48:36 +02:00
handshake lib/handshake: substitute time.Now() with fastttime.UnixTimestamp(), since profiling shows time.Now() is slow 2024-01-23 18:39:28 +02:00
htmlcomponents lib/htmlcomponents: use relative links for the top page and for favicon.ico 2023-11-13 20:28:17 +01:00
httpserver tests: Initial version of integration tests (#7253) 2024-10-30 15:22:06 +01:00
httputils app/vlinsert: implement the ability to add extra fields to the ingested logs 2024-11-04 10:23:16 -03:00
influxutils app/{vminsert,vmagent}: add healthcheck for influx ingestion endpoints (#6749) 2024-08-05 09:45:32 +02:00
ingestserver Revert c6c5a5a186 and b2765c45d0 2024-07-03 23:57:49 +02:00
leveledbytebufferpool lib/leveledbytebufferpool: do not pool byte slices bigger than 2^18 bytes 2024-06-13 17:02:05 +02:00
logger app/vlogscli: add interactive command-line tool for querying VictoriaLogs 2024-10-01 12:24:53 +02:00
logstorage lib/logstorage: fixes panic at Block.MustInitFromRows (#7695) 2024-11-30 18:06:58 +01:00
lrucache all: consistently use 'any' instead of 'interface{}' 2024-07-10 00:23:26 +02:00
memory all: cleanup: remove // +build ... lines, since they are no longer needed after Go1.17, and the minimum supported Go version for VictoriaMetrics source code is Go1.20 2023-11-13 19:15:42 +01:00
mergeset Revert "lib/mergeset: add sparse indexdb cache (#7269)" 2024-11-04 10:33:22 -03:00
metricsql all: make fmt via the upcoming Go1.19 2022-07-11 19:23:25 +03:00
netutil lib/promscrape: fixes proxy autorization (#6783) 2024-08-19 22:50:39 +02:00
persistentqueue app/vmagent/remotewrite: follow-up for 87fd400dfc 2024-07-13 02:30:10 +02:00
procutil all: cleanup: remove // +build ... lines, since they are no longer needed after Go1.17, and the minimum supported Go version for VictoriaMetrics source code is Go1.20 2023-11-13 19:15:42 +01:00
promauth vmagent: add support of HTTP2 client for Kubernetes SD (#7114) 2024-10-08 10:37:48 +02:00
prompb Revert "Exemplar support (#5982)" 2024-07-03 16:09:18 +02:00
prompbmarshal Revert "Exemplar support (#5982)" 2024-07-03 16:09:18 +02:00
promrelabel lib/promrelabel: follow-up for 8958cecad6 2024-08-27 15:44:07 +02:00
promscrape lib/promscrape/discovery: properly apply the resource_group filter for Azure service discovery 2024-11-26 19:08:31 +01:00
promutils stream aggregation: fix possible duplicated aggregation results (#7118) 2024-09-30 14:30:34 +02:00
protoparser lib/protoparser/influx: enable batch processing by default (#7165) 2024-10-15 11:51:48 +02:00
proxy lib/promscrape: fixes proxy autorization (#6783) 2024-08-19 22:50:39 +02:00
pushmetrics lib/pushmetrics: wait until the background goroutines, which push metrics, are stopped at pushmetrics.Stop() 2024-01-16 21:18:22 +02:00
querytracer make go vet happy 2024-08-19 21:41:44 +02:00
ratelimiter app/vmagent: properly shutdown when -maxIngestionRate limit is reached 2024-04-03 02:41:11 +03:00
regexutil lib/logstorage: work-in-progress 2024-05-25 22:59:21 +02:00
slicesutil lib/slicesutil: add helper functions for setting slice length and extending its capacity 2024-05-12 11:33:49 +02:00
snapshot lib/httputils: parse URL before creating HTTP transport (#6820) 2024-08-16 11:34:49 +02:00
storage refactoring: changed prompb to prompbmarshal everythere where internal series transformations are happening (#7409) 2024-11-26 19:02:06 +01:00
streamaggr lib/streamaggr: add ignore_first_sample_interval param for streamaggr cfg (#7313) 2024-11-22 17:38:13 +01:00
stringsutil all: consistently use stringsutil.JSONString() for formatting JSON strings with fmt.* functions instead of using "%q" formatter 2024-07-17 14:01:37 +02:00
syncwg all: open-sourcing single-node version 2019-05-23 00:18:06 +03:00
tenantmetrics lib/tenantmetrics: improves CounterMap performance with large numbers of tenants 2024-11-20 18:36:06 +01:00
timerpool lib/timerpool: use timer pool in concurrency limiters 2019-05-28 17:30:10 +03:00
timeutil vmselect: add support of multi-tenant queries (#6346) 2024-10-01 16:37:18 +02:00
uint64set lib/uint64set: optimize Set.Has() for nil Set - it should be inlined now 2024-07-16 00:00:46 +02:00
vmselectapi lib: consistently use atomic.* types instead of atomic.* functions 2024-02-24 02:10:04 +02:00
workingsetcache lib: consistently use atomic.* types instead of atomic.* functions 2024-02-24 02:10:04 +02:00
writeconcurrencylimiter app/vmagent/remotewrite: clarify the reason behind the default value for -remoteWrite.queues in the same way as the reason for -maxConcurrentInserts is defined at 73f5fb0f0c 2024-03-06 13:57:53 +02:00