Celery: part 1

Is simply a task queue that receives it’s tasks from a message broker( will talk about it later and provide examples for several message brokers) and simply execute this task and return the response back to the message broker that gets back to front end part of your application

This is the hierarchy of the whole process of a simple web application using celery task handler so the first question that comes to one’s mind is when to use smth like celery.

Use cases

• I have found a common use case which is trying to send an email to all your users that are saved in your database so you will use the email server smtp, so will send requests for each user to this server. Imagine if there’s a delayed response from the server or any problem so this email couldn’t reach the user.
• If you need to perform a long running task in the background, that couldn’t be handled through http Rest) like for example fitting a machine learning model and return it’s results.

Now let’s go into much details about the processes after there’s a request to a simple views function in django framework.

Message broker

There are too much examples of message brokers like Rabbit MQ, Active MQ and even SQL DB and No SQL DB like Redis could be used as message broker for task handlers like celery. The function of the message broker from a top view is that it take tasks from the web app and pass it to the right workers which are the base blocks of celery, let’s get back to the message broker eco-system (Rabbit MQ using AMQP potocol).

There are 3 base blocks for the message broker which are Exchange, Binding and Queue.

Exchange

Considered the bottle neck for this broker as it takes the messages and transmit it to their right queues

Types of routing

• Direct , map the message to a certain queue when routing_key == binding_key, routing key is set as a unique key per message and the binding key is set by you when you need to specify a certain queue for a certain message.
• Fanout, is just to simply distribute message to all the queues

This is a great blog from which I understood these techniques, it has lots of visuals that will help you grasp the concepts I explained above.

“An introduction to RabbitMQ, a broker that deals in messages” by Chandrabhan https://link.medium.com/jNms2eInv1

Bindings

Are the rules defining which message to be passed to which queue as you have seen we can set a unique binding key to certain messages to be directed to a certain queue.

Queue

Similar to the data structure queue follows FIFO used to handle and several messages in a row.

This is the last contact of the message broker then if you have seen the visuals in the blog I have mentioned you will notice that after that the next node is the actual worker node of celery, they called it the consumers these guys are the actual workers that takes a good care of the task we have handed.

This was an introduction to asyc tasks, there will be a practical blog with code in Python describes the two cases I have mentioned above.

THANKS