Skip to content
Netilion Extension Platform

Kafka Bridge Guide

What is a Kafka Bridge?

A Kafka Bridge connects your MQTT data streams directly to a Kafka topic. This means you can use a Kafka topic to receive data instead of connecting to the MQTT broker. It’s like having a direct line from your MQTT data to your Kafka-based applications.

How Does it Work?

  • MQTT to Kafka: Data flows from MQTT to Kafka, which we call ‘inbound data.’
  • Kafka to MQTT: The bridge also works the other way, sending data from Kafka back to MQTT, reaching your edge devices that communicate via MQTT.

Setting Up Inbound Data (MQTT to Kafka)

You can set up different bridges to different Kafka topics for inbound data.
For each bridge you can define a name an individual partition strategy and the MQTT topics which should be transferred.

For each bridge for inbound data, all messages from defined MQTT topics (except those coming from the Kafka to MQTT outbound bridge) get sent to a Kafka topic named [p].bridge.inbound.[name]. Here, [p] is a placeholder for your company and project names and [p]is a placeholder for the name of the bridge.

What Happens to Your Data?

  • The MQTT topic becomes a part of the kafka message header with the key topic and the MQTT topic as corresponding value
  • The MQTT message content is sent as the body of your Kafka message.

Customizing Your Kafka Topic

You can adjust your Kafka topic settings with options like:

  • Number of Partitions: More partitions mean more clients can process messages simultaneously.
  • Cleanup Policy: Choose between ‘delete’ (standard) or ‘compact’ (only keeps the last message per key).
  • Retention Time (ms): How long messages are kept with the ‘delete’ policy.
  • Retention Size (bytes): The maximum size of a partition before old messages are deleted.
  • Partition Strategy: Either ‘key_dispatch’ for having all messages with the same key in the same partition or ‘random’ to distribute the data over all partitions.

Example

An MQTT message like this:

{
  "firstName": "John",
  "lastName": "Smith",
  "age": 25
}

becomes a Kafka message in [p].bridge.inbound with the header {“topic”:“my_device/incoming_data”} and the same content.

Setting Up Outbound Data (Kafka to MQTT)

For outbound data, messages from the Kafka topic [p].bridge.outbound are sent to MQTT topics defined in the Kafka message key.

What Happens to Your Data?

  • The Kafka message key becomes the topic of your MQTT message.
  • The Kafka message content is sent as the body of your MQTT message.

Quality of Service (QoS) for Outgoing Messages

The QoS of all messages can be configured globally for your Kafka bridge.

  • QoS 0: Messages aren’t retained if no client is connected to the MQTT topic.
  • QoS 1: Messages are retained until a client reconnects, but this requires unique client IDs and at least one prior connection.

Example

A Kafka message like this

{
  "action": "restartDevice",
  "timeout": 25
}

with the key
my_device/outgoind_data

is sent to an MQTT topic my_device/outgoing_data with the same content.

Accessing Your Kafka Instance

You can access your Kafka instance using credentials provided by the NEXT team, stored in an Azure Key Vault and a Kubernetes secret named keyvault-project.

Connection Details:

  • URL: Stored in the variable confluent-bootstrap-url.
  • Key: Use confluent-sa-project-key for the connection.
  • Secret: Use confluent-sa-project-secret for the connection.