Commit graph

76 commits

Author SHA1 Message Date
Aliaksandr Valialkin
c73a9cbdaa
docs: refer to the https://itnext.io/how-do-open-source-solutions-for-logs-work-elasticsearch-loki-and-victorialogs-9f7097ecbc2f in the appropriate places 2024-11-15 16:08:18 +01:00
Aliaksandr Valialkin
ad55eb29dd
Add --delete option to rsync command, so it removes non-existing files from the destination
- The --delete option is needed to be passed to rsync during backups,
Since otherwise the backup may contain superfluous files after the second run of rsync,
because these files can be already removed at the source because of background merge

- the --delete option is needed when restoring from backup in order to remove superfluous files
from the destination directory. Otherwise these files may lead to inconsistent data at VictoriaLogs.
2024-11-14 17:21:29 +01:00
Aliaksandr Valialkin
50d9804df6
docs/VictoriaLogs: remove support for alerting from the roadmap, since it has been already implemented in the commit 68bad22fd2
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7255
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6706
2024-11-14 17:21:29 +01:00
Aliaksandr Valialkin
ab2ce3728b
app/vlinsert/syslog: allow changing the default set of log fields to use as stream fields during syslog data ingestion
Thanks to @AndrewChubatiuk for the initial implementation at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7488
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7480

See https://docs.victoriametrics.com/victorialogs/data-ingestion/syslog/#stream-fields
2024-11-14 17:21:27 +01:00
Aliaksandr Valialkin
0e9767cd77
docs/VictoriaLogs/README.md: add Security chapter
It is inspired by https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7428#discussion_r1831555716

(cherry picked from commit 62e6c9bd6f)
2024-11-08 17:07:58 +01:00
Aliaksandr Valialkin
2d1bc39fe5
docs/VictoriaLogs/README.md: follow-up for 0f6b9e9490: consistently use canonical url for Grafana dashboards
See also 0a5ffb3bc1

(cherry picked from commit f88e2ae9fb)
2024-10-31 14:11:09 +01:00
Aliaksandr Valialkin
cce047c27a
docs/VictoriaLogs/README.md: recommend installing Grafana dashboard for VictoriaLogs at Monitoring chapter
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6886

(cherry picked from commit 0f6b9e9490)
2024-10-31 14:11:08 +01:00
hagen1778
0e6ed4171b
deployment/alerts: consistently update path to alerting rules
Follow-up after 68bad22fd2

Signed-off-by: hagen1778 <roman@victoriametrics.com>
(cherry picked from commit 6494606924)
2024-10-30 16:44:51 +01:00
Aliaksandr Valialkin
b4f5e5c1b8
app/vlinsert: accept logs with empty _msg field
In this case the _msg field is set to the value specified in the -defaultMsgValue command-line flag.

This should simplify first-time migration to VictoriaLogs from other systems.

(cherry picked from commit 16ee470da6)
2024-10-30 15:19:52 +01:00
Aliaksandr Valialkin
40be393f5a
docs/VictoriaLogs/README.md: add tuning chapter
(cherry picked from commit 67b4059aa4)
2024-10-30 09:52:50 +01:00
Hui Wang
451bd164f0
docs: clarify flags -search.maxxxDuration can only be overridden to… (#7227)
… a smaller value with `timeout` arg
2024-10-27 20:25:17 +01:00
hagen1778
d14cf4b679
docs: follow-up after f0d1db81dc
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-10-17 11:19:03 -03:00
Andrii Chubatiuk
7b49d4f5dc
vlogs: added basic alerts (#7252)
### Describe Your Changes

Added basic VLogs alerts

Signed-off-by: hagen1778 <roman@victoriametrics.com>
Co-authored-by: hagen1778 <roman@victoriametrics.com>
2024-10-17 11:11:45 -03:00
Aliaksandr Valialkin
303ada493a
docs/VictoriaLogs/README.md: fix copy-n-paste typo: partitions in VictoriaLogs are per-day, not per-month 2024-10-17 11:09:17 -03:00
Andrii Chubatiuk
822d6bbdbc
docs/victorialogs: fixed HA examples links (#7249)
### Describe Your Changes

Fixed VictoriaLogs HA examples references in docs

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

(cherry picked from commit 41e0b62099)
2024-10-15 11:49:31 +02:00
Aliaksandr Valialkin
f627d7f686
app/vlstorage: add support for forced merge via /internal/force_merge HTTP endpoint
(cherry picked from commit 3c73dbbacc)
2024-10-15 11:49:31 +02:00
Zakhar Bessarab
4b1fe0d656
docs/victoria-logs: fix typo (#7245)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
(cherry picked from commit 2239f5829f)
2024-10-15 11:49:29 +02:00
Aliaksandr Valialkin
8c55b699f4
app/vlogscli: add interactive command-line tool for querying VictoriaLogs 2024-10-01 12:24:53 +02:00
Artem Navoiev
bcab5861ae
victorialogs: add HA example for logstash and fluentbit (#6968)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

---------

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit 277fed9990)
2024-09-19 15:46:52 +02:00
Artem Navoiev
d5cfce693a
docs: Add VictoriaLogs HA example for single nodes (#6945)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [x] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).

Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit f9a8c09fe8)
2024-09-09 16:06:43 +02:00
Zhu Jiekun
51988ad45d
docs: Add backup and restore doc for VictoriaLogs
This commit adds doc for VictoriaLogs backup and restore based on `rsync`.
2024-08-27 23:08:29 +02:00
Andrii Chubatiuk
4a01709b90
added _index.md for hugo, which point to README.md (#6686)
### Describe Your Changes

Please provide a brief description of the changes you made. Be as
specific as possible to help others understand the purpose and impact of
your modifications.

### Checklist

The following checks are **mandatory**:

- [ ] My change adheres [VictoriaMetrics contributing
guidelines](https://docs.victoriametrics.com/contributing/).
2024-07-25 12:20:40 +02:00
Aliaksandr Valialkin
8b76a40715
lib/httpserver: skip basic auth check for additional request paths, which should call httpserver.CheckAuthFlag()
This is a follow-up for 61dce6f2a1

Updates https://github.com/VictoriaMetrics/VictoriaMetrics/pull/6338
Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6329
2024-07-16 01:08:41 +02:00
Aliaksandr Valialkin
ab28a1f93e
app/vlinsert/syslog: add an ability to use log ingestion time as the _time field 2024-07-02 01:39:45 +02:00
Aliaksandr Valialkin
516c7b2ca0
docs/VictoriaLogs: remove "preview" warning - VictoriaLogs is ready for prod 2024-07-01 01:52:40 +02:00
Aliaksandr Valialkin
4b3477e62b
lib/logstorage: add stream_context pipe, which allows selecting surrounding logs for the matching logs 2024-06-28 19:15:19 +02:00
Aliaksandr Valialkin
dd62a2b9d6
lib/logstorage: work-in-progress 2024-06-27 14:21:03 +02:00
Aliaksandr Valialkin
75d45068a5
docs/VictoriaLogs/README.md: document the relation between -retention.maxDiskSpaceUsageBytes and -retentionPeriod command-line flags 2024-06-26 12:03:29 +02:00
Aliaksandr Valialkin
d5cbda3424
app/vlstorage: add -retention.maxDiskSpaceUsageBytes command-line flag for limiting the retention at VictoriaLogs by disk space usage 2024-06-25 17:30:46 +02:00
Aliaksandr Valialkin
c10a646d19
app/vlinsert/syslog: allow accepting syslog messages with different configs at different ports 2024-06-17 23:16:58 +02:00
Aliaksandr Valialkin
b7b3a9e9a3
lib/logstorage: work-in-progress 2024-06-04 01:50:55 +02:00
Aliaksandr Valialkin
3152df2bce
lib/logstorage: work-in-progress 2024-05-25 00:31:55 +02:00
Artem Navoiev
f4743dcef2
docs: victorialogs mention that our bench suite can be run against Loki as well
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit cee4bfebd7)
2024-05-21 16:37:21 +02:00
Aliaksandr Valialkin
b1ee7bca1a
lib/logstorage: work-in-progress 2024-05-14 03:06:02 +02:00
Aliaksandr Valialkin
728aceebfe
all: replace old https://docs.victoriametrics.com/vmauth.html url with the new one - https://docs.victoriametrics.com/vmauth/ 2024-04-18 01:50:16 +02:00
Aliaksandr Valialkin
a99005eff6
all: replace old https://docs.victoriametrics.com/vmalert.html url with the new one - https://docs.victoriametrics.com/vmalert/ 2024-04-18 01:44:54 +02:00
Aliaksandr Valialkin
0211a04a52
all: replace the outdated url https://docs.victoriametrics.com/vmagent.html with the new one - https://docs.victoriametrics.com/vmagent/ 2024-04-18 01:32:57 +02:00
Aliaksandr Valialkin
dc5b1e4dc1
lib/httpserver: return back the default value for -http.connTimeout to 2 minutes
It has been appeared that there are VictoriaMetrics users, who rely on the fact that
VictoriaMetrics components were closing incoming connections to -httpListenAddr every 2 minutes
by default. So let's return back this value by default in order to fix the breaking change
made at d8c1db7953 .

See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1304#issuecomment-1961891450 .
2024-02-24 00:20:11 +02:00
Artem Navoiev
e0569a355b
docs: mention slack inviter and slack channel (#5817)
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
(cherry picked from commit 5b652bccad)
2024-02-20 13:35:01 +01:00
hagen1778
216f268c1a
docs: follow-up after 491287ed15
* port un-synced changed from docs/readme to readme
* consistently use `sh` instead of `console` highlight, as it looks like
a more appropriate syntax highlight
* consistently use `sh` instead of `bash`, as it is shorter
* consistently use `yaml` instead of `yml`

See syntax codes here https://gohugo.io/content-management/syntax-highlighting/

Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-29 17:06:26 +01:00
Aliaksandr Valialkin
885ee160c2
all: allow dynamically reading *AuthKey flag values from files and urls
Examples:

1) -metricsAuthKey=file:///abs/path/to/file - reads flag value from the given absolute filepath
2) -metricsAuthKey=file://./relative/path/to/file - reads flag value from the given relative filepath
3) -metricsAuthKey=http://some-host/some/path?query_arg=abc - reads flag value from the given url

The flag value is automatically updated when the file contents changes.
2024-01-22 01:23:23 +02:00
hagen1778
bcc3c9c12e
docs: fix typo in VictoriaLogs upgrading procedure
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2024-01-16 17:43:59 +02:00
Dmytro Kozlov
7afec3d813
docs: remove default value from the maxConcurrentInserts flag (#5494) 2023-12-20 14:15:20 +02:00
Dima Lazerka
ed8fc04898
lib/httpserver: add flags to specify HSTS / Frame-Options / CSP headers for httpserver (#5111)
support `Strict-Transport-Security`, `Content-Security-Policy` and `X-Frame-Options`
HTTP headers in all VictoriaMetrics components.
The values for headers can be specified by users via the following flags:
`-http.header.hsts`, `-http.header.csp` and `-http.header.frameOptions`.

Co-authored-by: hagen1778 <roman@victoriametrics.com>

(cherry picked from commit ad839aa492)
Signed-off-by: hagen1778 <roman@victoriametrics.com>
2023-10-30 11:41:38 +01:00
Aliaksandr Valialkin
94e061087f
docs: use https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest instead of https://github.com/VictoriaMetrics/VictoriaMetrics/releases link where needed
The https://github.com/VictoriaMetrics/VictoriaMetrics/releases link may show non-latest
releases at the top, such as LTS releases or VictoriaLogs releases.
So it is better to use https://github.com/VictoriaMetrics/VictoriaMetrics/releases/latest link,
which always redirect to the latest available release of VictoriaMetrics.
2023-10-26 09:23:17 +02:00
Aliaksandr Valialkin
3b79f98459
Revert "docs victorialogs use relative links"
This reverts commit 3d7a77bf82.

Reason for revert: relative links do not work properly at GitHub code
and at GitHub wiki. For example, the following page contains broken links
before reverting this commit:

https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/docs/VictoriaLogs/CHANGELOG.md

It is always better to use absolute links thank relative links, since the page contents
can be copy-n-pasted to other pages, which are located in vastly different directories,
and all the links will remain working.
2023-10-16 14:03:21 +02:00
Artem Navoiev
13a3daf700
docs victorialogs use relative links
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-10-16 13:42:26 +02:00
Artem Navoiev
c9a1fc73a6
docs/vmlogs fix aliases
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-10-16 13:26:11 +02:00
Artem Navoiev
8f96709219
docs: victorialogs change the doc ordering
Signed-off-by: Artem Navoiev <tenmozes@gmail.com>
2023-10-05 11:36:27 +02:00
Zakhar Bessarab
dfdada055c
lib/logstorage: switch to read-only mode when running out of disk space (#4945)
* lib/logstorage: switch to read-only mode when running out of disk space

Added support of `--storage.minFreeDiskSpaceBytes` command-line flag to allow graceful handling of running out of disk space at `--storageDataPath`.

See: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/4737
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/logstorage: fix error handling logic during merge

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

* lib/logstorage: fix log level

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>

---------

Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
2023-10-02 17:09:57 +02:00