Member-only story
Surprises from Polling Kinesis
AWS Kinesis is probably a great tool for certain jobs, but it’s frequently used inappropriately. Its main intended use case seems to be real-time stream processing, but it’s frequently used as a pub-sub channel or message queue. A stream can function as a channel or queue, for sure. So can a database. So can a flat-file. So can a chalkboard. But that doesn’t mean those are the best solutions.
Our team was recently implementing a consumer to process events that a third party application was publishing to a Kinesis stream. We wanted a pub-sub channel, what we had was a Kinesis stream. C’est la vie. This article describes some of the surprising things we discovered about Kinesis along the way.
A quick overview of Kinesis
In case you’re not familiar with Kinesis, here’s a quick summary of how you get records out of a stream.
First of all, Kinesis streams are sharded, so you’ll need to know which shard you want to read from. How you do that depends on what you’re doing, but for the most part, you’ll have a consumer for each shard in the stream. To simplify things, we’ll mostly consider a shard to be a stream in its own right because practically speaking, it is.
To read from a shard, you need to ask Kinesis for a shard iterator. Conceptually, a shard iterator is a pointer into…
