Apache Kafka logs and metrics with OpenTelemetry

Collect Apache Kafka broker logs and cluster metrics — controller and request logs, partitions, and consumer lag — with the OpenTelemetry Collector.

Collect Apache Kafka broker logs — the Log4j application logs such as server.log — with the OpenTelemetry Collector. The same Collector can also forward traces and metrics from your apps — see Connect OpenTelemetry Collector to Bronto.

This covers the broker’s operational logs, not the commit-log data Kafka stores for topics.

Prerequisites

• A Bronto account and API key (how to create one)
• Kafka brokers writing Log4j logs to disk. The default location is $KAFKA_HOME/logs/ (often /var/log/kafka/), with server.log as the main broker log
• An OpenTelemetry Collector (recommended) or Fluent Bit installed on the same host

Configure the Collector

Kafka stack traces span multiple lines. Use multiline keyed on the Log4j [timestamp] prefix:

/etc/otel/config.yaml

receivers:
  filelog/kafka:
    include:
      - /var/log/kafka/server.log
      - /var/log/kafka/controller.log
    multiline:
      line_start_pattern: '^\[\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}\]'
    resource:
      service.name: kafka
      service.namespace: <YOUR_COLLECTION_NAME>

processors:
  batch:

exporters:
  otlphttp/brontologs:
    logs_endpoint: "https://ingestion.<REGION>.bronto.io/v1/logs"
    compression: gzip
    headers:
      x-bronto-api-key: <YOUR_API_KEY>

service:
  pipelines:
    logs:
      receivers: [filelog/kafka]
      processors: [batch]
      exporters: [otlphttp/brontologs]

Set <REGION> to eu or us.

Metrics

The Collector Contrib kafkametrics receiver reads broker, topic, partition, and consumer-group state from the Kafka protocol:

receivers:
  kafkametrics:
    brokers: [localhost:9092]
    protocol_version: <YOUR_KAFKA_PROTOCOL_VERSION>
    scrapers: [brokers, topics, consumers]
    collection_interval: 30s

Set protocol_version to a value supported by your cluster and Collector version, then add the receiver to the Bronto metrics pipeline. See the Collector Contrib Kafka Metrics receiver documentation for authentication and scraper options. Use a JMX-based receiver when you also need JVM and broker-process measurements.

What you will see in Bronto

Open Search and filter by service.name = kafka. Entries follow the Log4j pattern — timestamp, level (INFO, WARN, ERROR), logger, and message — covering broker lifecycle, controller and partition events, ISR changes, and exceptions with stack traces (kept intact by the multiline setting). Bronto’s Custom Parser extracts these fields server-side.

Distributed tracing

The Kafka broker does not emit traces — Apache Kafka has no native tracing, so the entries above are the broker’s operational logs only. To trace messages end to end, instrument your producer and consumer applications with the OpenTelemetry SDK: it creates send/receive spans and propagates trace context through Kafka message headers. Point those apps at the same Collector and traces flow to Bronto alongside these logs — see Send Traces to Bronto and the OpenTelemetry SDK guides. On Kubernetes, Strimzi can additionally enable OpenTelemetry tracing for Kafka Connect, MirrorMaker, and the Kafka Bridge.

Troubleshooting

• Where are the logs? Check the kafka.logs.dir / LOG_DIR setting or the appender file path in your log4j.properties, and adjust include.
• Want structured fields at the source? Configure a Log4j JSON layout for direct field extraction.
• For general issues, see OTel Collector troubleshooting.

Alternative: Fluent Bit

If you already run Fluent Bit, tail the broker log files and forward them over HTTP. The built-in java multiline parser keeps stack traces intact:

fluent-bit.conf

[INPUT]
    name              tail
    path              /var/log/kafka/server.log,/var/log/kafka/controller.log
    multiline.parser  java
    tag               kafka

[OUTPUT]
    name        http
    match       kafka
    host        ingestion.<REGION>.bronto.io
    port        443
    tls         on
    format      json_lines
    compress    gzip
    header      x-bronto-api-key    <YOUR_API_KEY>
    header      x-bronto-dataset    kafka
    header      x-bronto-collection <YOUR_COLLECTION_NAME>

Set Format to json_lines, not json. See Connect Fluent Bit to Bronto for installation.