Apache Kafka In a Nutshell

Overview

Apache Kafka is a distributed, append-only commit log designed for high-throughput, low-latency, fault-tolerant event streaming and messaging in large-scale systems. It underpins modern data platforms as a central backbone for event-driven architectures, stream processing, and real-time integration.

Kafka was built at LinkedIn to address the need for a unified, scalable pipeline for activity data, metrics, and application events, and later open-sourced and donated to the Apache Software Foundation. It has since become the de facto standard for durable event streaming in microservices and data engineering stacks.

1. What and Why

1.1 Problem Space

Traditional messaging systems and ETL pipelines struggle to handle modern requirements for:

High message volume (millions of events per second) across many producers and consumers.
Long-term retention of ordered event streams, not just transient messages.
Multiple independent consumers with different processing speeds and replay needs.
Strong durability and fault tolerance without sacrificing throughput.

Legacy message queues typically delete messages once consumed, making it hard to introduce new consumers or reprocess history, and often do not scale horizontally to cluster-wide throughput demands. ETL systems are usually batch-oriented, not designed for low-latency event propagation.

1.2 Kafka’s Value Proposition

Kafka exists to provide a distributed, scalable, durable event log that decouples producers and consumers while supporting both message queue and publish-subscribe semantics.

Key capabilities include:

Persistent, ordered, partitioned logs with configurable retention.
Horizontal scalability via partitioning across brokers.
High availability via leader–follower replication.
Flexible consumption: at-least-once, at-most-once, and exactly-once processing semantics.
Support for stream processing through Kafka Streams, ksqlDB, and integrations with frameworks like Apache Flink and Apache Spark.

1.3 Where Kafka Fits

Typical roles in an enterprise architecture:

Central event backbone for microservices, replacing point-to-point messaging.
Ingestion layer for analytics, data lakes, and warehousing (e.g., to object storage or warehouses).
Integration bus between heterogeneous systems (legacy, SaaS, mainframe).
Real-time analytics and monitoring pipelines.
Event sourcing and CQRS backplanes for domain-driven systems.

Kafka is usually positioned between application services and downstream data systems, acting as the durable buffer that smooths rate mismatches, isolates failures, and enables independent evolution of producers and consumers.

2. Core Architecture

2.1 High-Level Components

Core Kafka components:

Broker: A Kafka server responsible for storing partitions, handling client requests, and participating in replication.
Topic: A named category of records, partitioned for parallelism and scalability.
Partition: An ordered, append-only log within a topic, replicated across brokers.
Producer: A client that writes records to topics.
Consumer: A client that reads records from topics.
Consumer Group: A set of consumers that coordinate to consume partitions of a topic for horizontal scaling and fault tolerance.
Controller: A special broker role responsible for partition leadership assignments and cluster metadata management.
Storage: Log segments on disk with index files, leveraging sequential I/O and OS page cache.

2.2 Cluster Architecture

A Kafka cluster comprises multiple brokers forming a single logical service. Topics are divided into partitions which are distributed across brokers, each with one leader and zero or more follower replicas.

Leaders handle all reads and writes for their partitions.
Followers replicate the leader’s log by fetching data, persisting it locally, and reporting their progress.
The controller broker manages leader elections and tracks which replicas are in the in-sync replica set (ISR).

Prior to KRaft, Zookeeper was used for metadata and controller election; newer Kafka versions use an internal Raft-based metadata quorum (KRaft), eliminating the external dependency while providing stronger metadata consistency and fault tolerance.

2.3 Log Storage Model

Each partition is a time-ordered, immutable sequence of records identified by monotonically increasing offsets. Records are written sequentially to disk in log segments, with periodic index files for efficient offset-based lookup.

Important characteristics:

Sequential disk writes maximize throughput and minimize random I/O.
The log is append-only; deletions occur via log compaction or segment deletion.
Offsets are logical positions, not message IDs managed by the broker per consumer.

2.4 Replication and High Watermark

Kafka uses leader–follower replication with a well-defined in-sync replica set to provide durability and availability.

Each partition has exactly one leader and multiple followers.
Followers pull data from the leader and acknowledge once persisted.
The high watermark is the offset of the last record replicated to all in-sync replicas.
Consumers can only read up to the high watermark, ensuring they never see records that could be lost if the leader fails before replication completes.

The ISR is the set of replicas that are fully caught up within a configured lag window (for example, controlled by a replica lag time parameter). Only replicas in the ISR are eligible for leader election, which protects acknowledged records from being lost during failover.

2.5 Controller and Metadata

The controller broker manages cluster metadata and critical control-plane operations:

Partition leader elections and reassignments.
Tracking ISR membership and handling out-of-sync replicas.
Managing topic creation, deletion, and configuration changes.

In KRaft mode, a Raft quorum of controller nodes stores and replicates metadata, and one controller is elected as the active leader handling administrative duties, while others remain hot standbys.

3. Key Concepts and Terminology

3.1 Topics and Partitions

Topic: A named feed of records. Topics are logical categories to which producers write and from which consumers read.
Partition: A shard of a topic, represented by an ordered, append-only log. Each partition is replicated and has a leader and followers.
Replication Factor: Number of replicas (leader plus followers) per partition for fault tolerance.

Partitioning enables horizontal scaling:

Parallelism across consumers in a consumer group.
Distribution of load across brokers.
Fault isolation at partition granularity.

3.2 Offsets and Consumer Position

Offset: A monotonically increasing logical index within a partition that uniquely identifies a record.
Consumer Position: The offset from which a consumer will read next; maintained by the consumer, not by the broker per-consumer.
Commit: Operation by which a consumer records its processed offset, typically in an internal Kafka topic or external store.

Offsets enable replay and time-travel by allowing consumers to seek to an earlier offset and reprocess history.

3.3 Consumer Groups and Rebalancing

Consumer Group: A group of consumers that share a group identifier and coordinate to consume partitions of a topic cooperatively.
Group Coordinator: Broker component that manages membership and partition assignments for a group.
Rebalance: Process triggered when group membership or topic partitioning changes, causing partitions to be reassigned across group members.

Kafka guarantees that each partition is consumed by at most one consumer instance within a group, providing queue-like semantics for that group, while multiple groups can consume the same topic independently for pub-sub style fan-out.

3.4 Delivery Semantics

Kafka supports distinct message delivery semantics, depending on producer, broker, and consumer configuration:

At-most-once: Messages may be lost but never redelivered. Typical when producers or consumers do not retry or do not commit after processing.
At-least-once: Messages are never lost but may be redelivered. Achieved via retries and committing offsets after processing.
Exactly-once: Messages are processed once and only once, even in failure and retry scenarios, using idempotent producers and transactional APIs.

3.5 Idempotent Producer and Transactions

Idempotent Producer: Producer that includes a producer ID and sequence numbers on records, allowing the broker to detect and discard duplicates, achieving idempotent writes per partition.
Transactional Producer: Extends idempotence to multiple partitions and topics, enabling atomic writes across them and atomic commit of consumer offsets along with produced results, supporting end-to-end exactly-once processing.

3.6 Log Compaction and Retention

Log Retention: Policy controlling how long data is kept, either time-based (for example, retain for a configured number of hours) or size-based (for example, retain up to a configured number of bytes per partition).
Log Compaction: Feature that retains only the latest record per key in a compacted topic, discarding older versions while keeping a complete set of current keys.

Compacted topics are well-suited for changelog-style data such as entity state snapshots or configuration tables.

3.7 Acknowledgment Modes and acks

Producer acknowledgment configuration strongly influences durability and latency:

acks=0: Producer does not wait for broker acknowledgment; highest throughput, lowest durability.
acks=1: Producer waits for leader acknowledgment; moderate durability, good throughput.
acks=all (or -1): Producer waits until all in-sync replicas have acknowledged; strongest durability but higher latency.

The min.insync.replicas configuration defines the minimum number of in-sync replicas required for acking writes when using acks=all, preventing the degenerate case where the ISR shrinks to only the leader.

3.8 In-Sync Replicas (ISR)

ISR: The set of replicas that are caught up to the leader within a configured lag bound.
Only ISR replicas are eligible for leader election.
Producers using acks=all rely on the ISR size and min.insync.replicas to define a meaningful durability guarantee.

4. Data Structures and Models

4.1 Log Segments and Indexes

Each partition is physically represented by multiple log segments:

Log Segment: A file containing a contiguous range of records.
Index File: Companion file mapping relative offsets to physical file positions for efficient seek operations.

Segments allow Kafka to delete old data by dropping entire segment files instead of deleting individual records, retaining high throughput.

4.2 Record Format

A Kafka record contains:

Topic and partition (implicit from target).
Offset (assigned by broker on append).
Key (optional, used for partitioning and log compaction).
Value (payload bytes; schema is external, for example via Avro, Protobuf, or JSON).
Headers (optional metadata key–value pairs).
Timestamp (creation or log append time).

Kafka itself is schema-agnostic; schema evolution and validation are typically handled through schema registries and client-side tooling.

4.3 Partition Assignment and Keys

Partitioners map records to partitions based on:

Explicit partition ID provided by producer.
Hash of record key for key-based affinity.
Round-robin or other strategies for load balancing when no key is provided.

Key-based partitioning ensures that all records for the same key are routed to the same partition, preserving per-key order and enabling stateful processing keyed by entity.

4.4 Compacted vs Non-Compacted Topics

Two main retention models:

Non-compacted (delete-based): Segments are deleted when records exceed retention time or size limits.
Compacted: Older records with the same key are removed during background compaction, leaving the latest value for each key and optionally tombstones for deletes.

Compacted topics are used for:

State changelogs for stream processors.
Configuration, routing tables, or metadata distributions.
Recoverable materialized views.

5. Core Operations and Capabilities

5.1 Messaging and Event Streaming

Kafka can act as:

Durable message queue: Using consumer groups for work distribution, each message processed by one consumer instance per group.
Publish–subscribe system: Multiple independent consumer groups subscribe to the same topic and receive all messages.

It supports high-throughput writes and reads with millisecond-level end-to-end latency in well-tuned clusters.

5.2 Stream Processing

Kafka provides first-class support for stream processing via:

Kafka Streams: A client-side library for building stateful stream processing applications with features like windowing, joins, state stores, and exactly-once semantics.
ksqlDB: A SQL-based streaming engine layered on Kafka Streams.
Integrations with external frameworks such as Apache Flink and Apache Spark Streaming.

These enable continuous computation over unbounded event streams for use cases like anomaly detection, metrics aggregation, and enrichment.

5.3 Connectors and Integration (Kafka Connect)

Kafka Connect is a framework for scalable, fault-tolerant data integration:

Source Connectors: Pull data from external systems into Kafka (for example, databases, SaaS APIs, logs).
Sink Connectors: Push data from Kafka into target systems (for example, data warehouses, search indexes, object storage).

Connect workers handle offset management, scaling, configuration, and fault recovery for connectors.

5.4 Administrative Operations

Key administrative capabilities include:

Topic lifecycle management: creating, updating, and deleting topics with specified partitions and replication factors.
Rebalancing and reassignment: moving partitions across brokers for capacity or failure remediation.
Quotas: limiting producer and consumer throughput per client or per user.
Access control and security configuration at topic and cluster level.

5.5 Multi-Cluster and Geo-Replication

Kafka supports cross-cluster replication via tools such as MirrorMaker, MirrorMaker 2, or vendor-specific replication solutions.

Common patterns:

Active–active replication across data centers for locality and resilience.
Active–passive DR clusters with lagged replication.
Aggregation clusters fed by regional Kafka clusters.

6. Configuration and Deployment

6.1 Deployment Models

Common deployment models include:

Self-managed clusters on bare metal or virtual machines.
Managed services (for example, cloud providers and Kafka-as-a-service offerings).
Kubernetes-based deployments with operators for automated cluster lifecycle management.

Factors influencing model choice include operational maturity, control requirements, regulatory constraints, and total cost of ownership.

6.2 Broker Configuration Fundamentals

Critical broker-level settings typically include:

log.dirs: Paths where partition data is stored.
num.partitions: Default number of partitions per new topic.
default.replication.factor: Default replication factor for new topics.
min.insync.replicas: Minimum ISR size required for acks=all acknowledgments.
replica.lag.time.max.ms (or equivalent): Time threshold before a slow follower is removed from ISR.

These parameters directly shape durability, availability, throughput, and failure behavior.

6.3 Topic-Level Configuration

Per-topic configuration allows fine-grained control:

partitions: Degree of parallelism and throughput.
replication.factor: Fault tolerance level.
retention.ms and retention.bytes: Retention policies.
cleanup.policy: delete or compact (or both).
min.insync.replicas: Overrides broker default for specific topics.

Designing topic configuration requires understanding consumer patterns, SLAs, and storage constraints.

6.4 Cluster Sizing and Capacity Planning

Key considerations:

Expected throughput (messages per second and bytes per second) per topic and per partition.
Storage footprint given retention policy, replication factor, and data volume.
Network bandwidth for replication and client traffic.
Headroom for broker failures and rebalancing.

Replication factor 3 with min.insync.replicas 2 is a common baseline for high availability, though stricter requirements may demand higher replication, at the cost of more storage, CPU, and network utilization.

6.5 KRaft vs Zookeeper

Modern Kafka clusters increasingly use KRaft mode, which replaces Zookeeper with Kafka’s own metadata quorum:

Reduces operational complexity and failure modes associated with a separate Zookeeper ensemble.
Uses Raft consensus for metadata updates, providing strong consistency.
Separates controller and broker roles logically, though they may share nodes in smaller deployments.

Migration from Zookeeper to KRaft requires careful planning and version compatibility checks.

7. Integration Patterns

7.1 Microservices and Event-Driven Architectures

Kafka is a backbone for event-driven microservices:

Services publish domain events to topics after state changes.
Other services subscribe to these events and react asynchronously.
Consumer groups scale processing horizontally per service.

This reduces temporal coupling and allows independent evolution of services and their data models.

7.2 Request–Reply over Kafka

While Kafka is optimized for streaming, it can support request–reply patterns by:

Including correlation identifiers in messages.
Having reply topics for responses.
Using timeouts and transient state in services.

However, this usage must be carefully evaluated against dedicated RPC systems; Kafka’s strengths lie in asynchronous, event-driven communication.

7.3 Integration with Spring and Java Ecosystem

Spring Kafka and Spring Cloud Stream offer integration layers:

Declarative configuration of producers and consumers as Spring beans.
Message-driven POJOs, bindings, and abstractions for topic mappings.
Integration with Spring Boot configuration and actuator for monitoring.

These enable idiomatic integration of Kafka into Java and Spring-based microservices.

7.4 Integration with Data Systems

Common integration patterns via Kafka Connect and custom consumers:

Change data capture from relational databases into Kafka topics, then into data lakes or analytics stores.
Streaming logs from web servers or applications into search systems.
Feeding real-time analytics engines and dashboards from Kafka topics.

Kafka often acts as the convergence point between OLTP systems, analytics platforms, and machine learning pipelines.

7.5 Cloud and Hybrid Deployments

In cloud environments, Kafka integrates with:

Cloud object storage as sink targets for long-term archiving.
Managed relational and NoSQL databases.
Authentication and authorization systems such as IAM roles, OIDC, and secrets managers.

Hybrid patterns include on-premise Kafka clusters replicating to cloud clusters for analytics or disaster recovery.

8. Performance and Scalability

8.1 Scaling via Partitions and Brokers

Throughput is primarily scaled by increasing the number of partitions and distributing them across brokers.

Key principles:

Each partition is single-writer and single-reader per consumer group, so more partitions enable more parallelism.
Partitions should be balanced across brokers to avoid hot spots.
Consumer group size is effectively bounded by the number of partitions for that topic (idle consumers if group size exceeds partition count).

8.2 Producer Performance Tuning

Performance levers on the producer side include:

Batching: Aggregating records before send to amortize overhead.
Compression: Using compression codecs to reduce bandwidth at the cost of CPU.
acks: Choosing appropriate acknowledgment mode for durability–latency trade-offs.

Idempotent producers add minimal overhead while providing much better safety against duplicates in retry scenarios.

8.3 Consumer Performance Tuning

Consumer throughput depends on:

Max in-flight requests and fetch sizes.
Parallelism via multiple consumer instances.
Efficient message processing logic.
Avoiding long-lived synchronous calls inside processing loops that block partition progress.

Rebalancing costs can be reduced with cooperative partition assignment strategies, which aim to minimize partition movement during rebalances.

8.4 Broker and Storage Performance

Broker throughput is bounded by:

Disk throughput for sequential writes and reads.
Network bandwidth for serving clients and replication.
CPU, especially for compression and TLS encryption.

Kafka relies on the operating system page cache to minimize disk seeks and flushes, so system-level tuning of disk, filesystem, and OS parameters is critical.

8.5 Typical Bottlenecks and Trade-Offs

Common bottlenecks:

Insufficient partitions limiting consumer parallelism.
Storage saturation due to high replication factor and long retention.
Network saturation in replication-heavy clusters or geo-replication setups.

Every increase in durability (higher replication, acks=all, higher min.insync.replicas) generally increases end-to-end latency and resource usage, so SLAs must be balanced against cost and throughput requirements.

9. Security

9.1 Security Layers

Kafka security is typically implemented in three layers:

Encryption: TLS secures data in transit between clients and brokers and between brokers.
Authentication: SASL mechanisms or TLS client authentication verify client identity.
Authorization: Access control lists (ACLs) define which principals can perform which operations on which resources.

All three layers are required for a robust security posture in production clusters.

9.2 Encryption with TLS

TLS provides confidentiality and integrity for data in transit:

Brokers are configured with server certificates and private keys.
Clients use truststores to validate broker certificates.
Optional mutual TLS can authenticate clients based on certificates.

TLS should be mandatory for all production deployments to protect credentials and payloads from eavesdropping or tampering.

9.3 Authentication with SASL

Kafka supports several SASL mechanisms:

SASL/PLAIN: Simple username/password mechanism; must be used only over TLS.
SASL/SCRAM: Credential hashing and challenge–response for stronger protection.
SASL/GSSAPI (Kerberos): Integration with enterprise identity systems.

Best practices include preferring SCRAM or Kerberos over PLAIN, rotating credentials regularly, and keeping authentication separate from authorization configuration.

9.4 Authorization with ACLs

Kafka ACLs define permissions on resources:

Resources include topics, consumer groups, and cluster-wide operations.
Operations include Read, Write, Create, Delete, Describe, and administrative actions.
ACLs can be scoped to specific principals or groups and can use prefix-based resource patterns.

Security best practice is to implement least-privilege ACLs and disable permissive defaults.

9.5 Network and Operational Security

Additional security considerations:

Isolating Kafka brokers in private networks and restricting access via firewalls.
Using bastions or VPNs for administrative access.
Monitoring for suspicious authentication failures and access patterns.
Planning for certificate and credential rotation from the start to avoid outages.

10. Persistence and Reliability

10.1 Durability Guarantees

Kafka provides strong durability guarantees when configured appropriately:

Records are written to the leader’s log, optionally fsynced depending on configuration.
Replication to followers and ISR mechanics ensure redundancy.
Producer acks and min.insync.replicas determine when a write is considered committed.

With replication factor 3, acks=all, and min.insync.replicas set to 2 or higher, Kafka can tolerate multiple broker failures without losing committed data, at the cost of increased latency.

10.2 Failure Modes and Recovery

Key failure scenarios:

Leader failure: A new leader is elected from the ISR set; consumers reconnect and continue from high watermark.
Follower failure: Remaining replicas stay in ISR; the failed replica re-syncs from leader upon recovery.
Disk failure: Data on failed disk is lost for partitions whose replicas were on that disk; replication ensures other replicas survive.

Design must account for correlated failures, such as rack or availability-zone outages, by spreading replicas across failure domains.

10.3 Exactly-Once Processing

Kafka’s exactly-once semantics depend on:

Idempotent producer to avoid duplicate writes on retries.
Transactions to group multiple writes and offset commits in an atomic unit.
Stream processing frameworks that correctly integrate with transactional APIs.

Properly configured, this enables end-to-end exactly-once processing even across multi-stage streaming topologies.

10.4 Backup and Disaster Recovery

Kafka’s durability relies primarily on replication rather than external backups, but DR strategies may include:

Cross-cluster replication to a remote cluster for disaster recovery.
Periodic offloading of topic data to object storage.
Snapshotting of critical compacted topics representing state.

Restoring from such backups is more operationally complex than typical database restores, so design must consider RPO and RTO explicitly.

11. Monitoring and Observability

11.1 Core Metrics

Key broker metrics include:

Request rate and latency for produce and fetch requests.
Under-replicated partitions and partitions without a leader.
ISR size and fluctuation frequency.
Disk usage per broker and per log directory.

Producer and consumer metrics:

Record send rate, batch sizes, and error rates.
Consumer lag (difference between log end offset and committed offset) per partition.
Rebalance frequency and duration.

11.2 Logging and Auditing

Logs are critical for:

Troubleshooting broker and client errors.
Investigating authentication and authorization failures.
Tracking administrative operations such as topic changes and ACL updates.

Audit trails may need to be retained to satisfy regulatory or compliance requirements.

11.3 Health Checks and Alerting

Operational health indicators:

All partitions have an active leader.
No or very few under-replicated partitions under normal load.
Stable ISR membership without frequent flapping.
Acceptable producer and consumer latencies against SLAs.

Alerting should fire on sustained deviations rather than transient spikes to avoid alert fatigue.

11.4 Tooling Ecosystem

Common observability tools include:

Metrics exporters integrated with Prometheus and dashboards visualized in Grafana.
Vendor-specific monitoring stacks with Kafka support.
Command-line tools and administrative APIs for ad hoc inspection.

Kafka’s extensive metrics surface and JMX integration make it amenable to deep operational visibility in mature environments.

12. Common Patterns and Best Practices

12.1 Topic and Partition Design

Recommended practices:

Design topics around business domains and data contracts, not low-level implementation details.
Choose partition counts based on expected parallelism and growth, with room for future scaling.
Avoid excessively many small partitions, which increase overhead, and excessively large partitions that become hotspots.

12.2 Schema Management and Evolution

Since Kafka is schema-agnostic, teams typically:

Use Avro, Protobuf, or similar formats with schema registries.
Version schemas carefully and use backward/forward compatibility strategies.
Validate schemas at producer side and enforce contracts via CI and governance.

Strong schema discipline is essential to prevent downstream breakage and data quality issues.

12.3 Handling Failures and Retries

Patterns for robustness:

Configure producers with sensible retry policies and idempotence enabled.
Implement dead-letter queues or topics for messages that repeatedly fail processing.
Use idempotent or transactional consumers where side effects must not be duplicated.

12.4 Multi-Tenancy and Governance

In multi-team clusters:

Establish naming conventions and ownership per topic.
Use quotas and ACLs to control resource usage and access.
Provide onboarding documentation and testing environments.

Governance processes help avoid cluster sprawl, unbounded retention, and schema chaos.

12.5 Anti-Patterns

Notable anti-patterns:

Using Kafka as a key–value store or general-purpose database instead of an event log.
Overloading a single cluster with unrelated workloads without resource isolation.
Treating Kafka as a conventional message queue with strict per-message ordering guarantees across all data; ordering is only per partition.

Avoiding these anti-patterns keeps Kafka aligned with its design strengths.

13. Comparison with Alternatives

13.1 Kafka vs Traditional Message Queues

Traditional queues (for example, JMS, RabbitMQ) typically:

Delete messages upon consumption, making replay and multi-consumer patterns harder.
Emphasize complex routing, transactional queues, and per-message acknowledgments.

Kafka:

Retains messages for a configured period, enabling replay and multiple consumer groups.
Emphasizes partitioned logs, high throughput, and horizontal scalability.

Kafka is preferable for high-volume event streaming and long-term log retention, while traditional queues may be preferable for low-latency transactional messaging with complex routing semantics.

13.2 Kafka vs Cloud-Native Streaming Services

Cloud providers offer managed streaming services with Kafka-like or proprietary APIs. Differences often include:

Operational model: fully managed vs self-managed.
Throughput and partitioning semantics.
Integration with other cloud-native services.

Kafka provides portability and ecosystem depth but may require more operational investment compared to tightly integrated managed offerings.

13.3 Kafka vs Stream Processing Frameworks

Kafka is an event log and basic stream processing platform (via Kafka Streams); frameworks like Flink and Spark Streaming are more focused on complex stateful computations, large-scale windowing, and batch–stream unification.

In practice, Kafka often feeds or underpins these frameworks rather than replacing them.

13.4 When to Choose Kafka

Kafka is appropriate when:

There is a need for high-throughput, durable, ordered event streams.
Multiple independent consumers must share and replay the same data.
A central event backbone is needed to decouple producers and consumers.

It may be less appropriate for:

Low-volume, low-latency RPC where request–response is dominant.
Workflows requiring strong global ordering across all messages.
Use cases better suited to OLTP databases or key–value stores.

14. Interview Essentials

14.1 Core Concept Areas

Senior-level interviews on Kafka typically probe:

Architectural understanding: brokers, topics, partitions, replication, controller, ISR, high watermark.
Partitioning and consumer groups: how they map to scalability and fault tolerance, how rebalancing works.
Delivery semantics: at-most-once, at-least-once, exactly-once, idempotence, and transactions.
Durability and availability trade-offs: replication factor, acks, min.insync.replicas, and failure scenarios.
Log compaction and retention: when and why to use compacted topics.
Security: TLS, SASL, ACLs, and best practices for securing a cluster.
Operations and monitoring: key metrics, under-replicated partitions, consumer lag, rebalances.

14.2 Typical Interview Questions

Examples of questions and areas of depth:

Explain Kafka’s storage model and how it achieves high throughput on disk.
How do topics, partitions, and consumer groups interact to provide both queue and pub-sub semantics?
What happens when a leader broker for a partition fails? How does Kafka avoid data loss in that scenario?
Compare at-most-once, at-least-once, and exactly-once semantics in Kafka, including configuration requirements.
How does the ISR work, and how do acks and min.insync.replicas interact to provide durability guarantees?
Describe log compaction and give examples of when it is appropriate.
How would you design topics and partitions for a high-traffic application with strict SLAs?
What are the main security mechanisms in Kafka, and how would you secure a production cluster end to end?
How do you monitor Kafka health, and which metrics and alerts are critical in production?
How does KRaft differ from the older Zookeeper-based architecture, and what are the implications for operations?

Candidates are usually expected to connect configuration knobs to concrete failure modes and SLA requirements.

15. Quick Reference Summary

15.1 Critical Facts and Defaults

Core role: Distributed, durable, partitioned commit log for high-throughput event streaming and messaging.
Scaling unit: Partition; each partition is single-writer and single-reader per consumer group instance.
Availability model: Leader–follower replication with in-sync replica set and high watermark; only ISR replicas are eligible leaders.
Typical replication factor: 3 for production, with min.insync.replicas 2 and producer acks=all for strong durability.
Ordering guarantees: Per partition, not global across topic.
Delivery semantics:
- At-most-once: no retries or commit-before-processing.
- At-least-once: retries enabled and commit-after-processing.
- Exactly-once: idempotent, transactional producer and transactional-aware processing.
Retention:
- delete-based: time or size based.
- compacted: retain latest record per key.

15.2 Key Design and Ops Decisions

Topic design: align with business domains and contracts; choose partition counts based on throughput and consumer parallelism.
Durability vs latency: choose replication factor, acks, and min.insync.replicas based on acceptable data loss and latency budgets.
Security baseline: enforce TLS everywhere, use SASL/SCRAM or Kerberos for authentication, and ACLs with least privilege.
Observability: monitor under-replicated partitions, leaderless partitions, ISR churn, consumer lag, and request latencies.
Failure planning: distribute replicas across racks or zones; model broker and rack failures in capacity planning; define DR strategy via cross-cluster replication.

A senior engineer mastering these concepts can design, review, and operate Kafka-based systems with confidence and can reason about trade-offs between performance, durability, and complexity in enterprise-scale deployments.