Implementing Kafka with Gunicorn[Gevent Workers]
The above image demonstrates how a real-time Python microservices architecture looks like. Being a web application each microservice uses Gunicorn as the application Service.
The motive of writing this blog is to help users solve the Gunicorn & Kafka integration issue. The Major issue while using Kafka with Gunicorn is the Kafka consumer goes idle after reading a few messages.
Gunicorn uses the pre-fork mechanism using different types of workers.
Below blog will help in selecting suitable workers
For my scenario, I used Gevent as the worker for better performance and concurrency. The issue I faced using Gunicorn[Gevent] and Kafka consumer is the Consumer moved to hang state.
After multiple iterations, I found that Gevent blocks consumers from reading messages continuously. So Monkey patching Gevent with disabling socket & thread.
Below is the snippet of the solution.
Happy Coding!!!
References
https://hackernoon.com/thorough-introduction-to-apache-kafka-6fbf2989bbc1
