setup storm kafka testing infrastructure

Because of the version confusion of Storm, Kafka, Storm-Kafka, many people still cannot make the kafka and storm work together, even though there is a starter, but it uses the embedded zookeeper and kafka server instances. We need to get our hand dirty to go step by step to build the infrastructure, then writing analyze engine bolt will be easy.

Following are 10 steps that I have done to make the data flow from producer to Printer bolt in storm. Hopefully these detail steps will help other developers. And thanks to the owner of storm-kafka.

Under the directory:
yhuangMac:kafka_2.10-0.8.1 yhuang$ pwd
yhuangMac:kafka_2.10-0.8.1 yhuang$

1. on terminal 1, start zookeeper server
./bin/ config/

2. on terminal 2, start kafka server
./bin/ config/

3. on termina 3, create topic
./bin/ –create –zookeeper localhost:2181 –replication-factor 1 –partition 1 –topic storm-sentence

4. on terminal 3, run producer
./bin/ –broker-list localhost:9092 –topic storm-sentence

5. on terminal 4, run consumer
./bin/ –zookeeper localhost:2181 –topic storm-sentence –from-beginning

Under directory:
yhuangMac:storm-kafka-0.8-plus-test yhuang$ pwd

6. on Eclipse, import storm-kafka-0.8-plus-test maven project, in pom.xml, it has:


7. on Eclipse, run Java application of, set breakpoint at printlin of the execute method:
public void execute(Tuple tuple, BasicOutputCollector collector) {

8. on terminal 3, input: fdsafadsfadsffffffffffffffff

9. on Eclipse, we can get the breakpoint of println(), and the eclipse console will show:
44572 [Thread-18-words] INFO storm.kafka.PartitionManager – Committed offset 7 for Partition{host=localhost:9092, partition=0} for topology: 8983d811-d8b1-4cd2-9906-dfe349c9e3b3
source: words:3, stream: default, id: {3911753592003778923=-6203653086213542255}, [fdsafadsfadsfffffffffffffffff]
46573 [Thread-18-words] INFO storm.kafka.PartitionManager – Committing offset for Partition{host=localhost:9092, partition=0}

10. On terminal 4, we can also see kafka consumer print out:

This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s