Modern Log Aggregation With Grafana Loki and Vector: Save Storage Without a Giant Index

Your Elasticsearch cluster is greedy for RAM and wasteful with storage? Not every word in a log needs indexing. Meet the duo of Grafana Loki and Vector, far kinder to your wallet.

Read in Indonesian

For anyone who has run an Elasticsearch cluster or an ELK Stack, one classic complaint will be familiar. It is greedy for RAM and wasteful with storage. 😅

Elasticsearch is a brilliant full-text search technology. But the moment you make it the home for millions of log lines a day, it builds an inverted index on almost every word. The index size then often balloons. Sometimes it ends up bigger than the raw log files themselves!

In the era of microservices and Kubernetes containers, logs are produced in massive volume and fast. Do you really need to index every random word in a log? 🤔

The answer: not always. This is where the combination of Grafana Loki and Vector comes in. Two modern tools that are efficient and kind to your wallet.

Key takeaways

  • Modern log aggregation has shifted from full-text indexing (Elasticsearch) to label-based indexing (Grafana Loki), which is far lighter on RAM and storage.
  • Vector is a high-performance log shipper built on Rust, able to transform data, filter sensitive fields (PII), and forward logs efficiently.
  • Basic LogQL gives you a way to filter log streams by label, plus on-the-fly regex parsing right inside the Grafana dashboard.
  • A zero-cost log storage architecture keeps compressed log chunks directly in object storage like S3 or MinIO.

The Loki Paradigm: Like Prometheus, but for Logs

The philosophy behind Loki is very simple. It indexes metadata only (labels), not the text content of the log.

Instead of dissecting every log sentence into search tokens, Loki indexes only the stream labels. For example app="payment-service", environment="production", level="error". The actual log text is compressed into chunks. Then it is stored in cheap object storage like Amazon S3, Google Cloud Storage, or a local MinIO.

This approach brings several big advantages:

  • A very low memory footprint. Loki keeps no giant inverted index in RAM. Its resource needs are far smaller than Elasticsearch's.
  • Cheap storage cost. Logs are stored as chunks compressed with gzip/snappy. Even months of retention will not balloon the cloud bill.
  • Smooth integration with Grafana. You can jump from a Prometheus metric graph to the related logs with one click. All in the same dashboard.

Why Choose Vector as the Log Shipper?

The Grafana ecosystem used to rely on Promtail to ship logs into Loki. But in modern production architectures, many teams have moved to Vector from Datadog.

Vector is an open source log collector and pipeline processor. It is written in Rust. Its main advantages look like this:

  1. Extreme speed and efficiency. Its throughput is high, with minimal CPU and RAM consumption. All thanks to Rust's memory safety.
  2. Data transformation without the load. It has a built-in expression language called Vector Remap Language (VRL). Its job is parsing JSON and removing sensitive fields. For example a JWT token or a credit card number.
  3. Connectors for everything. Vector can pull logs from Docker, Kubernetes, syslog, local files, or Kafka. Then it sends them in parallel to Loki, S3, or ClickHouse.

The Vector to Grafana Loki Flow

The log process runs in three stages, in order.

First, ingestion in Vector. Vector acts as a DaemonSet in Kubernetes or an agent on each VM. It reads container stdout/stderr and parses the JSON format. After that it attaches labels like the pod name, namespace, and service.

Second, storage and chunking in Loki. Loki receives log batches from Vector and groups messages into streams by label. Only then does it write them to object storage periodically.

Third, querying with LogQL in Grafana. Developers query logs with LogQL, a query language similar to PromQL. A query first filters streams by label. That part is very fast. Only then does it filter the log text itself with regex, in parallel.

A Simple Pipeline Configuration Example

Here is an example vector.yaml pipeline configuration. It reads Docker container logs and parses the JSON format. It also filters sensitive data, then sends it to Loki:

YAML
sources:
  docker_logs:
    type: docker_logs

transforms:
  parse_and_sanitize:
    type: remap
    inputs:
      - docker_logs
    source: |
      # Parse the log message as JSON when possible
      parsed, err = parse_json(.message)
      if err == null {
        . = merge(., parsed)
      }

      # Mask sensitive data such as passwords or tokens
      if exists(.password) {
        .password = "[REDACTED]"
      }
      if exists(.authorization) {
        .authorization = "[REDACTED]"
      }

sinks:
  loki_sink:
    type: loki
    inputs:
      - parse_and_sanitize
    endpoint: "http://loki:3100"
    encoding:
      codec: json
    labels:
      app: "{{ container_name }}"
      env: "production"

Once the logs reach Loki, you can explore them directly in Grafana. Use this LogQL expression:

LOGQL
{app="payment-service", env="production"} |= "error" | json | latency > 500ms

The query above filters the log stream from the payment-service service in production. It looks for lines containing the word "error". Then it parses the JSON fields on-the-fly. The result is only requests with latency above 500 ms.

When to Move, When to Stay on ELK?

Hmm, moving from full-text search to Loki and Vector does not mean your observability shrinks. The opposite. You get cost efficiency, easier operations, and stronger data correlation. Especially alongside metrics and tracing.

If your server keeps running out of memory just because of the log aggregator, try the Loki and Vector duo on your next project.

Happy log aggregation, and may your cloud bill stay kind to your wallet. 👋

Inva

Writer

Get better at shipping code.

Join the other developers getting tech updates, tips, and relaxed tutorials in their inbox every week.

Stay in the loop

Tech updates, coding tips, and relaxed insights straight to your inbox every week.

No spam. Unsubscribe anytime.

Read next

See all

For developers who learn by struggling

© 2026 Inva.dev.