Surprises from Polling Kinesis

AWS Kinesis is probably a great tool for certain jobs, but it’s frequently used inappropriately. Its main intended use case seems to be real-time stream processing, but it’s frequently used as a pub-sub channel or message queue. A stream can function as a channel or queue, for sure. So can a database. So can a flat-file. So can a chalkboard. But that doesn’t mean those are the best solutions.

Our team was recently implementing a consumer to process events that a third party application was publishing to a Kinesis stream. We wanted a pub-sub channel, what we had was a Kinesis…