How to tier OpenTelemetry Collector with different pipelines per data stream

By design, charmed OpenTelemetry Collector (otelcol) forwards all receivers to all exporters. For this reason, in order to mimic the wide range of architectures that the pipeline config supports, multiple OpenTelemetry Collector charms may need to be deployed in a tiered topology. One such use case is for processing data differently per receiver or exporter.

Tiering outgoing data streams

One imaginable scenario is splitting a log stream into hot and cold data based on log levels. For compliance reasons we may also want to implement a redaction processor for removing sensitive data. Additionally, the batch processor improves efficiency of both log streams via compression. Low-severity levels like TRACE, DEBUG and INFO often have a greater frequency in log streams and indicate normal workload operation. This can be filtered out in a log stream which is sent to long-term (cold) storage to minimize cost while maintaining compliance. Conversely, the hot storage could include INFO logs, since storage is short-term, while still filtering out TRACE and DEBUG logs.

To understand how to filter telemetry with otelcol, refer to the selectively drop telemetry documentation or see the examples for log-level filtering.

        flowchart TB

flog[flog] --> fan-out
fan-out["opentelemetry-collector<br>(redact & batch)"]
fan-out --send-otlp--> warn
fan-out --send-otlp--> info
warn["opentelemetry-collector<br>(cold filter)"] --> loki-cold
info["opentelemetry-collector<br>(hot filter)"] --> loki-hot
loki-hot["loki<br>(hot storage)"]
loki-cold["loki<br>(cold storage)"]

class fan-out,warn,info thickStroke;
classDef thickStroke stroke-width:2px, stroke:#FFA500;
    

With Juju config we use the otelcol processor config to:

1. Set the minimum severity level to WARNING

  cold-filter:
          logs:
            log_record:
              - ContainsValue(Keys(ParseJSON(body)), "level") and
                (ParseJSON(body)["level"] == "INFO" or
                ParseJSON(body)["level"] == "DEBUG" or
                ParseJSON(body)["level"] == "TRACE")
    constraints: arch=amd64
    storage:
      persisted: kubernetes,1,1024M

2. Set the minimum severity level to INFO

    scale: 1
                (ParseJSON(body)["level"] == "DEBUG" or
                ParseJSON(body)["level"] == "TRACE")
    constraints: arch=amd64
    storage:
      persisted: kubernetes,1,1024M
    trust: true
  loki-cold:
    charm: loki-k8s

3. Redact sensitive log messages and batch

        batch:
    trust: true
relations:
  - - flog:log-proxy
    - redact-and-batch:receive-loki-logs
  - - redact-and-batch:send-loki-logs
    - hot-filter:receive-loki-logs

Tiering incoming data streams

Another imaginable scenario is classifying log streams prior to ingestion into a common storage destination. Each flog log source has unique downstream data processing, useful for environment classification and identification. Both data streams benefit from the redact & batch otelcol using the redaction processor for compliance reasons and the batch processor for efficiency. Additionally, they have an attributes processor, uniquely configured, to classify the logging source environment.

        flowchart TB

flog-dev["flog<br>(dev)"] --> dev
dev["opentelemetry-collector<br>(dev attributes)"] --send-otlp--> fan-in

flog-prod["flog<br>(prod)"] --> prod
prod["opentelemetry-collector<br>(prod attributes)"] --send-otlp--> fan-in

fan-in["opentelemetry-collector<br>(redact & batch)"] --> loki[loki]

class fan-in,dev,prod thickStroke;
classDef thickStroke stroke-width:2px, stroke:#FFA500;
    

With Juju config we use the otelcol processor config to:

1. Label the log stream as development and originating from region-a

  dev-attributes:
          actions:
            - key: "region-a.environment"
              value: "dev"
              action: upsert
    constraints: arch=amd64
    storage:
      persisted: kubernetes,1,1024M

2. Label the log stream as production and originating from region-a

    scale: 1
              action: upsert
    constraints: arch=amd64
    storage:
      persisted: kubernetes,1,1024M
    trust: true
  redact-and-batch:
    charm: opentelemetry-collector-k8s

3. Redact sensitive log messages and batch

        batch:
      persisted: kubernetes,1,1024M
    trust: true
relations:
- - flog-dev:log-proxy
  - dev-attributes:receive-loki-logs
- - flog-prod:log-proxy
  - prod-attributes:receive-loki-logs