Getting only a single record out of many the first time kafka consumer gets a record?

I'm using spring-kafka and spring-kafka-test versions 1.0.2.RELEASE. In one of my tests, my application sends 100 records in a row to a single TopicPartion on an EmbeddedKafka instance using the KafkaTemplate and mostly default config settings. I use the KafkaTestUtils.getRecords(consumer) method to try to get the records from the Kafka instance and verify that they have all been sent. The first time I call getRecords, I only receive a single record. If I call it again, I get the other 99. If I explicitly set the consumer's position to the beginning of the TopicPartition and then call getRecords, I get all 100. Why would getRecords only get a single record the first time? Is there some better way to get all 100 at once then by explicitly calling seekToBeginning on the consumer?

asked Aug 11, 2016 at 17:10 Joseph Downing Joseph Downing 1,119 3 3 gold badges 13 13 silver badges 27 27 bronze badges

2 Answers 2

This sounds like a timing issue. It's quite possible that only one message was available the first time you called poll() - that method makes no guarantees as to how many messages will be fetched. When you write code you shouldn't assume that you will receive X records at a go. There is a consumer property from Kafka 0.10 max.poll.records that for test purposes you may want to set to 1, and then perform a receive loop until you have polled all 100.

answered Aug 13, 2016 at 11:30 Jakub Korab Jakub Korab 5,024 2 2 gold badges 25 25 silver badges 34 34 bronze badges

Most likely just a race condition - the consumer is sitting in poll() and the broker sends the first message as soon as it arrives.

See properties fetch.min.bytes and fetch.max.wait.ms in the kafka docs.

fetch.min.bytes is 1 by default.

EDIT

You could also try flush() ing the KafkaTemplate before calling getRecords() .

However, your test should not really rely on getting all the messages in one fetch - too brittle.