mirror of
https://github.com/VictoriaMetrics/VictoriaMetrics.git
synced 2024-11-21 14:44:00 +00:00
No description
9fcfba3927
`TL;DR` This PR improves the metric IDs search in IndexDB:
- Avoid seaching for metric IDs twice when `maxMetrics` limit is
exceeded
- Use correct error type for indicating that the `maxMetrics` limit is
exceded
- Simplify the logic of deciding between per-day and global index search
A unit test has been added to ensure that this refactoring does not
break anything.
---
Function calls before the fix:
```
idb.searchMetricIDs
|__ is.searchMetricIDs
|__ is.searchMetricIDsInternal
|__ is.updateMetricIDsForTagFilters
|__ is.tryUpdatingMetricIDsForDateRange
| |
|__ is.getMetricIDsForDateAndFilters
```
- `searchMetricIDsInternal` searches metric IDs for each filter set. It
maintains a metric ID set variable which is updated every time the
`updateMetricIDsForTagFilters` function is called. After each successful
call, the function checks the length of the updated metric ID set and if
it is greater than `maxMetrics`, the function returns `too many
timeseries` error.
- `updateMetricIDsForTagFilters` uses either per-day or global index to
search metric IDs for the given filter set. The decision of which index
to use is made is made within the `tryUpdatingMetricIDsForDateRange`
function and if it returns `fallback to global search` error then the
function uses global index by calling `getMetricIDsForDateAndFilters`
with zero date.
- `tryUpdatingMetricIDsForDateRange` first checks if the given time
range is larger than 40 days and if so returns `fallback to global
search` error. Otherwise it proceeds to searching for metric IDs within
that time range by calling `getMetricIDsForDateAndFilters` for each
date.
- `getMetricIDsForDateAndFilters` searches for metric IDs for the given
date and returns `fallback to global search` error if the number of
found metric IDs is greater than `maxMetrics`.
Problems with this solution:
1. The `fallback to global search` error returned by
`getMetricIDsForDateAndFilters` in case when maxMetrics is exceeded is
misleading.
2. If `tryUpdatingMetricIDsForDateRange` proceeds to date range search
and returns `fallback to global search` error (because
`getMetricIDsForDateAndFilters` returns it) then this will trigger
global search in `updateMetricIDsForTagFilters`. However the global
search uses the same maxMetrics value which means this search is
destined to fail too. I.e. the same search is performed twice and fails
twice.
3. `too many timeseries` error is already handled in
`searchMetricIDsInternal` and therefore handing this error in
`updateMetricIDsForTagFilters` is redundant
4. updateMetricIDsForTagFilters is a better place to make a decision on
whether to use per-day or global index.
Solution:
1. Use a dedicated error for `too many timeseries` case
2. Handle `too many timeseries` error in `searchMetricIDsInternal` only
3. Move the per-day or global search decision from
`tryUpdatingMetricIDsForDateRange` to `updateMetricIDsForTagFilters` and
remove `fallback to global search` error.
---------
Signed-off-by: Artem Fetishev <wwctrsrx@gmail.com>
Co-authored-by: Nikolay <nik@victoriametrics.com>
|
||
|---|---|---|
| .github | ||
| app | ||
| cspell | ||
| dashboards | ||
| deployment | ||
| docs | ||
| lib | ||
| package | ||
| ports/OpenBSD | ||
| vendor | ||
| .dockerignore | ||
| .gitignore | ||
| .golangci.yml | ||
| .wwhrd.yml | ||
| CODE_OF_CONDUCT.md | ||
| CONTRIBUTING.md | ||
| go.mod | ||
| go.sum | ||
| LICENSE | ||
| logo.png | ||
| Makefile | ||
| README.md | ||
| SECURITY.md | ||
| VM_logo.zip | ||
VictoriaMetrics
VictoriaMetrics is a fast, cost-saving, and scalable solution for monitoring and managing time series data. It delivers high performance and reliability, making it an ideal choice for businesses of all sizes.
Here are some resources and information about VictoriaMetrics:
- Documentation: docs.victoriametrics.com
- Case studies: Grammarly, Roblox, Wix,....
- Available: Binary releases, Docker images, Source code
- Deployment types: Single-node version, Cluster version, and Enterprise version
- Changelog: CHANGELOG, and How to upgrade
- Community: Slack, Twitter, LinkedIn, YouTube
Yes, we open-source both the single-node VictoriaMetrics and the cluster version.
Prominent features
VictoriaMetrics is optimized for timeseries data, even when old time series are constantly replaced by new ones at a high rate, it offers a lot of features:
- Long-term storage for Prometheus or as a drop-in replacement for Prometheus and Graphite in Grafana.
- Powerful stream aggregation: Can be used as a StatsD alternative.
- Ideal for big data: Works well with large amounts of time series data from APM, Kubernetes, IoT sensors, connected cars, industrial telemetry, financial data and various Enterprise workloads.
- Query language: Supports both PromQL and the more performant MetricsQL.
- Easy to setup: No dependencies, single small binary, configuration through command-line flags, but the default is also fine-tuned; backup and restore with instant snapshots.
- Global query view: Multiple Prometheus instances or any other data sources may ingest data into VictoriaMetrics and queried via a single query.
- Various Protocols: Support metric scraping, ingestion and backfilling in various protocol.
- Prometheus exporters, Prometheus remote write API, Prometheus exposition format.
- InfluxDB line protocol over HTTP, TCP and UDP.
- Graphite plaintext protocol with tags.
- OpenTSDB put message.
- HTTP OpenTSDB /api/put requests.
- JSON line format.
- Arbitrary CSV data.
- Native binary format.
- DataDog agent or DogStatsD.
- NewRelic infrastructure agent.
- OpenTelemetry metrics format.
- NFS-based storages: Supports storing data on NFS-based storages such as Amazon EFS, Google Filestore.
- And many other features such as metrics relabeling, cardinality limiter, etc.
Enterprise version
In addition, the Enterprise version includes extra features:
- Anomaly detection: Automation and simplification of your alerting rules, covering complex anomalies found in metrics data.
- Backup automation: Automates regular backup procedures.
- Multiple retentions: Reducing storage costs by specifying different retentions for different datasets.
- Downsampling: Reducing storage costs and increasing performance for queries over historical data.
- Stable releases with long-term support lines (LTS).
- Comprehensive support: First-class consulting, feature requests and technical support provided by the core VictoriaMetrics dev team.
- Many other features, which you can read about on the Enterprise page.
Contact us if you need enterprise support for VictoriaMetrics. Or you can request a free trial license here, downloaded Enterprise binaries are available at Github Releases.
We strictly apply security measures in everything we do. VictoriaMetrics has achieved security certifications for Database Software Development and Software-Based Monitoring Services. See Security page for more details.
Benchmarks
Some good benchmarks VictoriaMetrics achieved:
- Minimal memory footprint: handling millions of unique timeseries with 10x less RAM than InfluxDB, up to 7x less RAM than Prometheus, Thanos or Cortex.
- Highly scalable and performance for data ingestion and querying, 20x outperforms InfluxDB and TimescaleDB.
- High data compression: 70x more data points may be stored into limited storage than TimescaleDB, 7x less storage space is required than Prometheus, Thanos or Cortex.
- Reducing storage costs: 10x more effective than Graphite according to the Grammarly case study.
- A single-node VictoriaMetrics can replace medium-sized clusters built with competing solutions such as Thanos, M3DB, Cortex, InfluxDB or TimescaleDB. See VictoriaMetrics vs Thanos, Measuring vertical scalability, Remote write storage wars - PromCon 2019.
- Optimized for storage: Works well with high-latency IO and low IOPS (HDD and network storage in AWS, Google Cloud, Microsoft Azure, etc.).
Community and contributions
Feel free asking any questions regarding VictoriaMetrics:
If you like VictoriaMetrics and want to contribute, then please read these docs.
VictoriaMetrics Logo
Zip contains three folders with different image orientations (main color and inverted version).
Files included in each folder:
- 2 JPEG Preview files
- 2 PNG Preview files with transparent background
- 2 EPS Adobe Illustrator EPS10 files
Logo Usage Guidelines
Font used
- Lato Black
- Lato Regular
Color Palette
We kindly ask
- Please don't use any other font instead of suggested.
- To keep enough clear space around the logo.
- Do not change spacing, alignment, or relative locations of the design elements.
- Do not change the proportions for any of the design elements or the design itself. You may resize as needed but must retain all proportions.