Setting Up Apache Kafka in KRaft Mode on WSL: A Step-by-Step Guide

Apache Kafka 3.x introduced KRaft (Kafka Raft) mode, which eliminates the dependency on ZooKeeper for metadata management. This guide will walk you through setting up Kafka in KRaft mode on Windows Subsystem for Linux (WSL).

Prerequisites

• WSL2 installed on Windows
• Ubuntu distribution in WSL
• Java 11 or later installed
• Basic familiarity with terminal commands

Step 1: Install Java (if not already installed)

sudo apt update
sudo apt install default-jre
java -version

Step 2: Download and Extract Kafka

First, create a directory for Kafka and download the latest version:

mkdir ~/kafka
cd ~/kafka
wget https://downloads.apache.org/kafka/3.5.1/kafka_2.13-3.5.1.tgz
tar -xzf kafka_2.13-3.5.1.tgz
cd kafka_2.13-3.5.1

Step 3: Generate Cluster UUID

Generate a unique cluster ID that will be used in the configuration:

KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"

Step 4: Configure Kafka for KRaft Mode

Create a new configuration file named kraft-config.properties:

cat > config/kraft-config.properties << EOF
node.id=1
process.roles=broker,controller
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
controller.listener.names=CONTROLLER
advertised.listeners=PLAINTEXT://localhost:9092
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kraft-combined-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
EOF

Step 5: Format Storage Directory

Format the storage directory with the cluster ID:

bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft-config.properties

Step 6: Start Kafka Server

Start the Kafka server using the KRaft configuration:

bin/kafka-server-start.sh config/kraft-config.properties

Step 7: Testing the Setup

Open a new terminal and create a test topic:

bin/kafka-topics.sh --create --topic test-topic --partitions 1 --replication-factor 1 --bootstrap-server localhost:9092

List all topics to verify creation:

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Step 8: Test Producer and Consumer

In one terminal, start a console producer:

bin/kafka-console-producer.sh --topic test-topic --bootstrap-server localhost:9092

In another terminal, start a console consumer:

bin/kafka-console-consumer.sh --topic test-topic --from-beginning --bootstrap-server localhost:9092

Troubleshooting Tips

• If Kafka fails to start, check the logs in logs/server.log
• Ensure ports 9092 and 9093 are not being used by other services
• Check that the log directory has proper write permissions
• Verify that WSL can access the necessary network ports

Creating a Startup Script (Optional)

Create a startup script to simplify the Kafka server startup process:

cat > ~/kafka/start-kafka.sh << EOF
#!/bin/bash
cd ~/kafka/kafka_2.13-3.5.1
bin/kafka-server-start.sh config/kraft-config.properties
EOF

chmod +x ~/kafka/start-kafka.sh

Conclusion

You now have a working Kafka setup in KRaft mode on WSL! This configuration is suitable for development and testing purposes. For production environments, you'll need to adjust various parameters like replication factors, partition counts, and security settings.

Remember that KRaft mode is the future of Kafka, offering improved scalability and simpler architecture by eliminating the ZooKeeper dependency. As you develop applications, you can now leverage this modern Kafka setup for your event streaming needs.