Job Queues with Flask, Redis and Docker

First, let us set up a bare-bone Flask project. Make sure that python 3 is installed in your system as it is the only dependency needed to create a flask project.

In case you need to install python, here are the official tutorials for almost every OS:

Official Python 3 Installation

After making sure that python is installed and working correctly let’s create our directory structure.

$ mkdir flask-redis-docker
$ cd flask-redis-docker

Next, we need a virtual environment for our Flask project. I will be running the commands for a MacOS environment in this article. If you are using another OS, go to the Flask documentation link in the sources of this article and look at the commands from there.

$ mkdir sample-flask
$ cd sample-flask
$ python3 -m venv venv

Let’s activate the environment and install Flask.

$ . venv/bin/activate
$ pip install Flask
$ pip freeze > requirements.txt

We should now have a basic Flask project with the minimal requirements. At the time of writing this post, Flask 1.1.1 is the latest stable version and this is the requirements.txt we have:

Click==7.0
Flask==1.1.1
itsdangerous==1.1.0
Jinja2==2.10.3
MarkupSafe==1.1.1
Werkzeug==0.16.0

Let try to create and run the most minimal Flask application. Create the file hello.py with the following python code:

from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello, World!'

Now, let’s run the flask application in debug mode:

$ export FLASK_APP=hello.py
$ export FLASK_ENV=development
$ flask run

You should see something like:

 * Serving Flask app "hello.py" (lazy loading)
 * Environment: development
 * Debug mode: on
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

Opening http://localhost:5000 on a browser should show up the “Hello, World” page.

Now, let’s dockerize this flask app. Before proceeding, make sure that docker is installed in your system. In case you are having trouble, check-out the official docs for installing docker:

Docker installation tutorial

You should have the following directory structure so far:

- flask-redis-docker
    - sample-flask
        - hello.py
        - requirements.txt
        - venv

Here is a sample Dockerfile (development mode) for Flask:

FROM python

RUN mkdir app

COPY ./requirements.txt /app/requirements.txt

RUN pip install -r /app/requirements.txt

COPY . /app

EXPOSE 5000

WORKDIR /app

CMD ["flask", "run", "--host=0.0.0.0"]

Create this file in the “sample-flask” folder. Let’s build and run this.

$ cd sample-flask
$ docker build -t sample-flask .
$ docker run -p 5000:5000 -e FLASK_APP=hello.py -e FLASK_ENV=development sample-flask

If you try localhost:5000 again, you should get the Hello, World page. This is basically the same application as before but running inside a container with an exposed port.

Let’s add the missing pieces and try to connect them step by step. We will need to:

Create a docker-compose file.
Use the docker-compose file to run our application (with hot-reloading).
Add Redis to the docker-compose file.
Queue jobs.
Visualize jobs/failed jobs.

Let’s start with the initial docker-compose file:

version: '3'
services:
  sample-flask:
    build:
        context: ./sample-flask
    ports:
      - "5000:5000"
    environment:
      - FLASK_APP=hello.py
      - FLASK_ENV=development
    volumes:
        - ./sample-flask:/app
  redis:
    image: "redis:alpine" # Lightweight redis image

Create the docker profile in the root folder of this tutorial. You should have the following structure now:

- flask-redis-docker
    - sample-flask
        - hello.py
        - requirements.txt
        - venv
    - docker-compose.yml

Running ‘docker-compose up’ should yield the same result as the 2 previous cases when you open localhost on port 5000 from a browser.

$ docker-compose up

Since we configured a volume in the docker-compose file and we are running flask in development mode, changing the code will automatically re-run the project even though it is inside the container.

We are going to use the Flask-RQ2 package to handle the integration of Flask with RQ.

$ cd sample-flask
$ pip install Flask-RQ2
$ pip freeze > requirements.txt

To make use of our new dependencies, we need to build again using docker-compose:

$ docker-compose build
$ docker-compose up

Now, let’s refactor our flask application code and use the application factory pattern. This will be the final project structure:

- flask-redis-docker
    - sample-flask
        - app
           - __init__.py
        - jobs
           - __init__.py
        - routes
           - __init__.py
           - default_blueprint.py
        - Dockerfile
        - worker.Dockerfile
        - requirements.txt
    - .gitignore
    - .dockerignore # Necessary to ignore the venv files for the volume
    - docker-compose.yml

Contents of the new files:

app/__init__.py

from flask import Flask

def create_app():
    app = Flask(__name__)

    # RQ
    from jobs import rq
    rq.init_app(app)

    # Blueprints
    from routes import default_blueprint
    app.register_blueprint(default_blueprint)

    return app

jobs/__init__.py

from flask_rq2 import RQ
rq = RQ()

routes/__init__.py

from .default_blueprint import default_blueprint

routes/default_blueprint.py

from flask import Blueprint
default_blueprint = Blueprint('default_blueprint', __name__)

@default_blueprint.route('/')
def index():
    return "Hello, World!"

worker.Dockerfile: We need to add the dockerfile for the worker. The worker will be listening for jobs and run the job methods.

FROM python

RUN mkdir app

COPY ./requirements.txt /app/requirements.txt

RUN pip install -r /app/requirements.txt

COPY . /app

WORKDIR /app

CMD ["flask", "rq", "worker"]

Building and running again, should yield the same result. The Flask application with the new structure should be running and printing hello world when opened on the browser.

Let’s create 2 sample jobs in the “jobs” folder. Create “sample_jobs.py” in this folder with the following code:

from jobs import rq

from random import randint
from time import sleep

@rq.job
def sample_job1():
    sleep(randint(1, 10))

    # Chance of failing
    if randint(1, 100) < 10:
        raise Exception("Fail!")

@rq.job
def sample_job2():
    sleep(randint(10,100))

Add the following 2 endpoints to the default_blueprint.py file:

# Add import at the top of the file
from jobs.sample_jobs import sample_job1, sample_job2

# ....
# ....

@default_blueprint.route('/job1')
def start_job1():
    sample_job1.queue()
    return "Job 1 queued"

@default_blueprint.route('/job2')
def start_job2():
    sample_job2.queue()
    return "Job 2 queued"

We need to update the docker-compose file and add environment variable in the Flask app (for Redis url) and add RQ-Dashboard to view the current/failed jobs in nice web interface.

version: '3'
services:
  sample-flask:
    build:
        context: ./sample-flask
    ports:
      - "5000:5000"
    environment:
      - FLASK_ENV=development
      - REDIS_URL=redis://redis:6379
    volumes:
        - ./sample-flask:/app

  sample-flask-worker:
    build:
        context: ./sample-flask
        dockerfile: ./worker.Dockerfile
    environment:
      - REDIS_URL=redis://redis:6379
    volumes:
        - ./sample-flask:/app
    depends_on: 
        - redis

  rq-dashboard:
    image: eoranged/rq-dashboard
    ports:
      - "9181:9181"
    environment: 
     - RQ_DASHBOARD_REDIS_URL=redis://redis:6379
    depends_on: 
      - redis
  
  redis:
    image: "redis:alpine" # Lightweight redis image

Also, we need to add the Redis url in our flask app config. Update the “app/__init__.py” file to:

from flask import Flask
import os

def create_app():
    app = Flask(__name__)
    
    # --> NEW LINES START
    # Redis URL
    app.config['RQ_REDIS_URL'] = os.getenv('REDIS_URL')
    # <-- NEW LINES END

    # RQ
    from jobs import rq
    rq.init_app(app)

    # Blueprints
    from routes import default_blueprint
    app.register_blueprint(default_blueprint)

    return app

It’s time to build and run with docker-compose again.

To see the whole thing in action, open “http://localhost:9181/” in a web browser. This is the RQ Dashboard which visualizes the jobs in a very nice way. A GET request in “http://localhost:5000/job1” will queue the first job. The first job has a chance of failing. In the case, the job status will be Failed and you can re-queue from the RQ Dashboard interface. Sending a get request at “http://localhost:5000/job2” will queue a job that will never fail and will last 10-100 seconds. Try queuing many jobs you will see the rq worker going through them one by one.

You should have enough information to start playing around in development with Flask, Docker and Redis. DO NOT USE THIS IN PRODUCTION! This is meant for tutorial purposes in a development environment. However, it shouldn’t be hard to move this into production. That might be discussed in a future article.

You can find the whole code for this tutorial on my github!

Job Queues with Flask, Redis and Docker

Related posts

Python & Golang comparison

Top Python Web Frameworks

Job queues with Django, Celery & Redis

Platform Engineering & Developer Experience

Distribute a Python package with public PyPi

Leave a Reply Cancel reply