Fleet Telemetry Streaming Architecture: Kafka & Kinesis

Jul 6, 2026 Resolute Dynamics

A real-time streaming architecture ingests high-volume vehicle telemetry as a continuous, ordered stream of events and delivers it to processing and storage within milliseconds, built on a durable partitioned log rather than periodic batch uploads. For a fleet, that architecture is what turns a flood of GPS coordinates, CAN bus readings, and sensor events from thousands of vehicles into data a safety system can act on while the vehicle is still moving.

This guide explains the four stages of a fleet telemetry pipeline, the role of MQTT at the vehicle, how Apache Kafka and Amazon Kinesis each carry the stream, the alternatives beyond them, and how to assemble the pieces into a working reference architecture.

This article states the technical position as of mid-2026. Streaming platforms version frequently, so version-specific facts — Kafka 4.0, Kinesis shard limits — carry a date or a source so the page ages honestly.

What Is a Real-Time Streaming Architecture for Fleet Telemetry?

A real-time streaming architecture is a pipeline that moves fleet data as it is generated, treating each telemetry reading as an event appended to a log that many systems can read independently and at their own pace.

It replaces the older pattern of collecting data on the vehicle and uploading it in periodic batches, which is unsuitable for anything that needs a fast response. The design principle is a durable, append-only, partitioned log: events are written once, kept in order, and made available to multiple consumers simultaneously.

The Four Stages — Ingestion, Streaming, Processing, and Serving

A fleet telemetry pipeline runs across four stages, and each solves a distinct problem. Ingestion collects data from vehicles over a lightweight protocol built for unreliable networks. Streaming buffers and orders that data in a durable log that decouples producers from consumers. Processing transforms, aggregates, and analyzes the stream, either as it flows or in windows.

Serving delivers the result to a control system, a dashboard, or long-term storage. Separating the stages lets each scale independently, so a spike in vehicle data does not overwhelm the analytics layer.

Why Fleets Stream Instead of Batch

Fleets stream rather than batch because a safety or control decision loses its value the moment it is late. A harsh-braking event, an overspeed condition, or a geofence breach has to reach the platform in near real time to trigger a response, and a batch upload every few minutes makes that impossible.

Streaming also decouples the vehicle from the backend, so a consumer that falls behind or fails does not block the producer. The related question of how much data to send from the vehicle and how often is examined in the comparison of event-driven versus continuous data capture for fleets.

The Ingestion Layer — MQTT from the Vehicle

Message Queuing Telemetry Transport (MQTT) is the de facto protocol for getting telemetry off the vehicle, because it was built for exactly the constrained, unreliable conditions a moving fleet operates in.

Created by IBM in 1999 and standardized by OASIS and ISO, MQTT uses a publish/subscribe model with a central broker and a binary header as small as 2 bytes, which minimizes bandwidth and battery use over cellular and satellite links.

A vehicle publishes to topics such as vehicle/123/gps or vehicle/123/can, and backend systems subscribe to the topics they need.

How MQTT Works and Its Three QoS Levels

MQTT offers three Quality of Service (QoS) levels that let a fleet trade reliability against speed per message type. The three levels are precise:

• QoS 0 — at most once. Fire and forget, with no acknowledgement or retry. Suitable for high-frequency, non-critical readings such as routine location pings where an occasional loss is acceptable.
• QoS 1 — at least once. The broker acknowledges receipt and the sender retries until acknowledged, which can produce duplicates. Suitable for events that must arrive but can tolerate a duplicate.
• QoS 2 — exactly once. A four-part handshake guarantees a single delivery with no loss or duplication. Suitable for critical events such as a crash alert, at the cost of the most overhead.

MQTT adds features built for intermittent connectivity, including persistent sessions, retained messages, and a Last Will and Testament message the broker sends if a vehicle disconnects unexpectedly.

Why MQTT and a Streaming Platform Are Complementary

MQTT and a streaming platform such as Kafka solve different halves of the problem, so a connected fleet uses both rather than choosing between them. MQTT connects large numbers of constrained, intermittently connected vehicles at the edge and handles thousands to tens of thousands of messages per second per broker; a streaming platform provides the high-throughput, fault-tolerant, replayable backbone that carries hundreds of thousands to millions of messages per second to backend systems.

A bridge maps MQTT topics onto streaming topics, so telemetry flows from vehicle to broker to stream in one path. The interface design that governs how this data is structured and exposed is covered in the guide to telematics API design best practices, and the network path the data travels before it reaches the cloud is described in the analysis of vehicle-to-cloud connectivity architecture.

Apache Kafka for Fleet Telemetry

Apache Kafka is a distributed, append-only commit log that has become the default open-source backbone for high-volume event streaming. It stores every event durably on disk according to a retention policy, keeps events strictly ordered within a partition, and lets many independent consumer groups read the same data at their own pace.

For a fleet, Kafka is the layer that ingests millions of telemetry events, holds them, and feeds them to safety processing, analytics, and archival at once without any consumer blocking another.

Topics, Partitions, Brokers, and Offsets

Kafka organizes data into topics, splits each topic into partitions, spreads partitions across brokers, and tracks each consumer’s position with an offset. A topic such as fleet-gps is divided into partitions that are each an ordered, immutable sequence of records; the partitions are distributed across broker servers and replicated to others according to a replication factor for fault tolerance.

Producers append records to partitions, and consumers read forward through each partition, committing an offset that marks how far they have read. This structure is what gives Kafka its parallelism — more partitions means more consumers can process the fleet’s data at once.

Ordering, Delivery Guarantees, and Exactly-Once Semantics

Kafka guarantees strict ordering within a single partition and provides exactly-once semantics through idempotent producers and transactions. By default Kafka delivers at least once, which can produce duplicates; enabling idempotent producers and transactional writes raises that to exactly once, so a telemetry event is neither lost nor double-counted.

Ordering is per partition, not per topic, so a fleet that needs every event for one vehicle processed in order routes that vehicle’s data to a single partition using the vehicle identifier as the key. Requiring total ordering across an entire topic means using a single partition, which removes parallelism, so most fleets order per vehicle instead.

KRaft and the End of ZooKeeper

As of Apache Kafka 4.0, released on 18 March 2025, ZooKeeper has been removed entirely, and KRaft is the only supported metadata mode. KRaft — Kafka Raft — is a consensus protocol built into Kafka that manages cluster metadata internally, replacing the external ZooKeeper ensemble that Kafka depended on for more than a decade.

The change matters for a large fleet platform for two reasons: KRaft raises the practical partition ceiling from roughly 200,000 under ZooKeeper to around 1.9 million, and it removes a second system to configure, secure, and operate.

A cluster still running an older ZooKeeper-based version must migrate to KRaft on a 3.x release before upgrading to 4.0, because a direct upgrade from ZooKeeper mode is not supported.

Tiered Storage and Long-Term Telemetry Retention

Kafka’s tiered storage moves older telemetry off broker disks to object storage while keeping recent data local, which decouples retention from broker capacity. With tiered storage, a fleet retains months or years of telemetry in Amazon S3, Google Cloud Storage, or Azure Blob Storage without sizing expensive broker disks to match, while recent events stay on the broker for fast access.

This lets the same platform serve real-time safety processing and long-horizon analytics or compliance replay from one system.

Amazon Kinesis for Fleet Telemetry

Amazon Kinesis Data Streams is a fully managed, serverless streaming service that removes the operational work of running a cluster. Where Kafka is software a team operates, Kinesis is a service AWS operates, so a fleet platform provisions capacity and writes data without managing brokers, patching, or scaling infrastructure.

It suits teams already committed to AWS that want streaming without the operational burden of self-managed Kafka.

Shards, Throughput Limits, and Partition Keys

The shard is the base throughput unit of Kinesis, and each shard supports 1 MB/s and 1,000 records per second for writes and 2 MB/s for reads. A stream’s capacity is the sum of its shards, and a partition key on each record determines which shard the record lands on — the same role the key plays in Kafka partitioning.

A fleet that uses the vehicle identifier as the partition key keeps each vehicle’s events on one shard and therefore in order, but a key that concentrates traffic on one shard creates a hot shard that throttles, so the key scheme has to distribute load evenly.

On-Demand vs. Provisioned Capacity

Kinesis offers two capacity modes, and the choice trades control against convenience. In on-demand mode the stream scales automatically, starting at a default of 4 MB/s and 4,000 records per second for writes and scaling up to 200 MB/s and 200,000 records per second, with no capacity planning.

In provisioned mode the operator specifies the shard count and is charged per shard per hour, which gives predictable cost and capacity for a known, steady telemetry volume. A fleet with volatile or growing traffic favours on-demand; a fleet with a well-understood baseline favours provisioned.

Consumers, Enhanced Fan-Out, and Retention

Kinesis supports multiple independent consumers and retains data for a configurable window, with Enhanced Fan-Out giving each consumer dedicated throughput. Standard consumers share a shard’s 2 MB/s read capacity, while Enhanced Fan-Out supports up to 20 consumer applications on a stream, each with its own dedicated read throughput, which matters when safety processing, analytics, and archival all read the same telemetry.

Retention defaults to 24 hours and extends up to 365 days, so a fleet can replay recent telemetry for reprocessing or investigation.

Kafka vs. Kinesis for a Fleet Platform

The choice between Kafka and Kinesis turns on operational model, ecosystem, and cloud commitment rather than raw capability, because both handle fleet-scale telemetry. Kafka gives maximum control, portability across clouds, and the largest ecosystem, at the cost of operating a cluster.

Kinesis gives a managed, serverless service with deep AWS integration, at the cost of vendor lock-in and per-unit pricing.

When Each Fits a Fleet Telemetry Workload

Kafka fits a fleet platform that runs across clouds, needs deep control, or wants to avoid lock-in, while Kinesis fits a platform built entirely on AWS that prioritizes low operational overhead. A team without the expertise to operate a Kafka cluster can still choose Kafka through a managed offering such as Amazon MSK or Confluent Cloud, which narrows the operational gap between the two.

The decision is rarely about whether either can handle the telemetry volume — both can — and usually about who operates the system and where it runs.

Beyond Kafka and Kinesis

Several platforms beyond Kafka and Kinesis address specific fleet-telemetry constraints, and each changes one thing about the trade-off. The four below are the most relevant alternatives for a fleet platform.

Apache Pulsar — Separated Compute and Storage

Apache Pulsar separates its serving layer from its storage layer, using Apache BookKeeper for durable storage, which lets compute and storage scale independently. This architecture supports scaling to over a million topics and native geo-replication across clusters, which suits a fleet operating across multiple regions that wants data replicated between them.

Pulsar also offers tiered storage and protocol handlers that speak the Kafka and MQTT protocols, easing migration.

Redpanda — Kafka-Compatible, No JVM or ZooKeeper

Redpanda is a Kafka API-compatible platform written in C++ that runs as a single binary with no Java Virtual Machine and no ZooKeeper. It uses an embedded Raft-based consensus protocol and works with existing Kafka clients and tools, which appeals to teams that find Kafka’s JVM tuning burdensome and want lower latency and simpler operations.

Known gaps against Apache Kafka around transactions, exactly-once semantics, and some Kafka Streams or Connect usage should be evaluated against a fleet’s specific requirements before committing.

Google Cloud Pub/Sub and Azure Event Hubs

Google Cloud Pub/Sub and Azure Event Hubs are the managed streaming services of their respective clouds, each suited to teams already on that platform. Azure Event Hubs is a partitioned, time-ordered, durable log that exposes a Kafka-compatible endpoint, so most Kafka clients work by changing only the bootstrap server, and it can capture streams directly to Azure Blob Storage or Data Lake in Avro or Parquet.

Google Cloud Pub/Sub provides fully managed messaging that integrates with the wider Google Cloud analytics stack.

Managed Kafka — Confluent Cloud and Amazon MSK

Managed Kafka services deliver the Kafka API without the operational burden of running the cluster. Confluent Cloud and Amazon Managed Streaming for Apache Kafka (MSK) both run Kafka on the operator’s behalf, so a fleet platform keeps Kafka’s ecosystem and portability of skills while offloading brokers, upgrades, and scaling. This is often the pragmatic middle path for a team that wants Kafka but not the operations.

Processing the Stream — Turning Telemetry into Action

Streaming platforms transport and store telemetry, but a separate processing layer turns that stream into decisions. Kafka, Kinesis, and their peers move and retain events; a stream processing engine filters, aggregates, joins, and evaluates them against rules to produce an alert, a control instruction, or an enriched record.

Stream Processing with Apache Flink and Kafka Streams

Apache Flink and Kafka Streams are the two common engines for processing fleet telemetry in motion, and both support stateful, windowed computation. Flink is a standalone distributed processing engine that handles large-scale stateful stream processing with exactly-once guarantees; Kafka Streams is a library embedded in an application that processes Kafka topics directly.

A fleet uses either to compute rolling averages of speed, detect a geofence crossing, join a vehicle’s live position against a route, or trigger a safety rule the instant its condition is met.

Edge Processing and Deciding What to Stream

Deciding what to process at the edge and what to stream to the backend shapes the entire pipeline’s cost and latency. Filtering and aggregating on the vehicle or a local gateway reduces the volume that crosses the network and reaches the streaming platform, which lowers cost and speeds the most time-critical decisions.

The trade-off between processing locally and streaming everything continuously is examined in detail in the comparison of event-driven versus continuous data capture for fleets, and the way raw data is structured across a mixed fleet before it is streamed is covered in the guide to vehicle data capture architecture for heterogeneous fleets.

Designing a Fleet Telemetry Streaming Architecture

Designing a fleet telemetry streaming architecture comes down to four decisions: how data enters, how it is partitioned and ordered, where it lives, and how it is secured. Getting these right early determines whether the platform scales cleanly or hits a wall, because each decision constrains the ones above it.

A Reference Pipeline — Vehicle to Control and Storage

A working fleet telemetry pipeline flows through five stages in sequence. The vehicle publishes telemetry over MQTT to a broker; a bridge maps MQTT topics onto a streaming platform such as Kafka or Kinesis; the streaming platform holds the ordered, durable event log; a processing engine such as Flink or Kafka Streams evaluates the stream for safety and analytics rules; and the results flow to control systems, dashboards, and long-term storage.

Each stage is independently scalable, so the pipeline absorbs a surge at the vehicle without breaking the analytics behind it.

Partitioning, Ordering, Residency, and Security Decisions

The partition key, the ordering requirement, the data’s location, and its protection are the four decisions that define a compliant, reliable fleet stream. Keying by vehicle identifier preserves per-vehicle ordering while distributing load; the streaming platform’s region and retention determine where telemetry physically lives, which is a legal question addressed in the guide to fleet data sovereignty and cross-border vehicle data flows; the network fabric that carries the stream in an industrial or port site is examined in the analysis of private LTE and 5G networks for fleet operations; and the encryption and access controls that protect the stream are detailed in the guide to securing fleet data with AI and telematics.

How Capture, Connect, and Control Map to the Stream

The three stages of a fleet safety platform map directly onto a streaming architecture. Sensors and telematics units capture the telemetry that becomes the event stream; the streaming platform is how that data is connected reliably from vehicle to backend; and processing feeds control systems that return an instruction inside the pipeline’s latency budget.

A team building or scaling a fleet telemetry platform can talk to Resolute Dynamics about the telematics and control layer that runs over a streaming backbone.

Frequently Asked Questions

Is Kafka or Kinesis better for fleet telemetry? Neither is universally better; the choice depends on operations and cloud strategy. Kafka gives control, portability across clouds, and the largest ecosystem but requires operating a cluster, while Amazon Kinesis is a fully managed AWS service with low operational overhead but is tied to AWS.

Both handle fleet-scale telemetry, and a team wanting Kafka without the operations can use Amazon MSK or Confluent Cloud.

Do you need both MQTT and Kafka? Yes, most connected-fleet architectures use both, because they solve different problems. MQTT connects constrained, intermittently connected vehicles at the edge with a lightweight protocol, while Kafka provides the high-throughput, durable, replayable backbone for backend processing.

A bridge maps MQTT topics onto Kafka topics so data flows through both in one path.

What throughput can a single Kinesis shard handle? A single Kinesis shard supports 1 MB/s and 1,000 records per second for writes and 2 MB/s for reads. A stream’s total capacity is the sum of its shards, so scaling means adding shards in provisioned mode or letting on-demand mode scale automatically up to 200 MB/s and 200,000 records per second by default.

Does Kafka still need ZooKeeper? No. As of Apache Kafka 4.0, released in March 2025, ZooKeeper has been removed entirely, and KRaft is the only supported metadata mode. Clusters on older versions must migrate to KRaft on a 3.x release before upgrading to 4.0.

How is streaming different from batch for fleet data? Streaming processes each telemetry event as it arrives, within milliseconds, while batch collects data and processes it in periodic chunks. Streaming is required for real-time safety and control decisions that lose value if delayed, whereas batch suits reporting and analytics that tolerate latency.