Using sorl-thumbnail with Redis on Heroku

I recently added sorl-thumbnail to a project for creating smaller image files. I chose to configure with a Redis Key-Value Store, as I’ve heard Redis is super hip, but ran into trouble during deployment.

In this post, I’ll document how I set up Redis locally with Docker, then in production on Heroku.

Local Docker Setup

The logical place to start is with the sorl-thumbnail documentation.

Install the package

python3 -m pip install sorl-thumbnail

Add package to settings.py

# settings.py
INSTALLED_APPS += ['sorl.thumbnail']

Migrate database

python manage.py migrate

This should create a model called KVStore.

Add redis server

If we navigate to the documentation’s requirements page, we’ll see we need some sort of key-value store and image processing library.

As the title suggests, we’re going to use Redis. Let’s add that in our docker-compose.yml file.

# docker-compose.yml
version: "3.7"

services:
  web:
    build: ./django
    command: python /code/manage.py runserver 0.0.0.0:8000
    env_file:
      - ./.env.dev
    volumes:
      - ./django:/code
    ports:
      - 8000:8000
      - 443:443
    depends_on:
      - db
      - redis

  db:
    image: postgres
    volumes:
      - postgres_data:/var/lib/postgresql
    environment:
      - POSTGRES_HOST_AUTH_METHOD=trust
      - POSTGRES_USER=username
      - POSTGRES_PASS=supercomplexpasswordthatishardtocrack
      - POSTGRES_DBNAME=pg
    ports:
      - "5432:5432"

  redis:
    restart: always
    image: redis:latest
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/var/lib/redis

volumes:
  postgres_data:
  redis_data:

networks:
  default:

I’m assuming here that your Django web and db services and volumes are already configured. Use this tutorial to get started with Docker, Django, and Postgres.

First, we’re adding a redis service (redis:) and volume (redis_data). Then, we’re telling the web service that we need redis for things to work properly.

Add Redis client

python3 -m pip install redis

Here’s a link to this redis-py project on Github.

Add a Python image library

If you’ve gotten to this page, you probably already have Pillow installed, but you can use a few different image libraries.

Connect Redis server and client

Lastly, sorl-thumbnail has some settings to configure to link it up to your Redis key-value store.

# settings.py
THUMBNAIL_KVSTORE = 'sorl.thumbnail.kvstores.redis_kvstore.KVStore'

But if we try to docker-compose up with just this settings, we receive an error from Django:

Connection error. Error 99 connecting to localhost:6379. Cannot assign requested address.

The sorl-thumbnail package tries to use some sensible default values for connecting to the Redis server:

THUMBNAIL_REDIS_DB = 0
THUMBNAIL_REDIS_PASSWORD = ''
THUMBNAIL_REDIS_HOST = 'localhost'
THUMBNAIL_REDIS_PORT = 6379

The problem is that we called our Redis host redis, so sorl-thumbnail cannot connect.

I like to be explicit about variables at the expense of verbosity. Here’s all my sorl-thumbnail settings:

# settings.py

# sorl-thumbnail
THUMBNAIL_KVSTORE = 'sorl.thumbnail.kvstores.redis_kvstore.KVStore'
THUMBNAIL_REDIS_DB = 0  # the default
THUMBNAIL_REDIS_HOST = 'redis'
THUMBNAIL_REDIS_PORT = 6379  # the default

Then you should be all set locally!

Heroku Production Setup

Production is a little different, but the principles still apply:

Redis server
Redis client
Connect sorl-thumbnail to server

Sorl-thumbnail uses the redis-py package (the Redis client) to connect to the Redis server. At the time of writing, sorl-thumbnail’s reference documentation only showed our above settings as configurable.

But I went and looked at the code and found a nice little secret! We’ll use it later.

Provision heroku-redis add-on

The Heroku documentation shows you how to create a new heroku-redis add-on from the command line or Heroku dashboard.

heroku addons:create heroku-redis:hobby-dev -a your-app-name

The hobby-dev tier is free.

This will add a REDIS_URL configuration or environment variable to your Heroku app. It should look like “redis://…”.

But we didn’t use this variable to connect our project to Redis in our local environment. How will we use REDIS_URL?

That’s the secret I just mentioned at the end of the previous section!

Configure Django settings to use `REDIS_URL`

Here’s a link to the sorl-thumbnail code used to connect to Redis. It first searches for a THUMBNAIL_REDIS_URL environment variable, which contains the Redis username, password, host, and port.

We want to use the REDIS_URL variable set by heroku-redis because that URL might change in the future without us knowing, but will always be stored as REDIS_URL.

We need to replace our previous Django settings for sorl-thumbnail and Redis connections.

# sorl-thumbnail
THUMBNAIL_KVSTORE = 'sorl.thumbnail.kvstores.redis_kvstore.KVStore'
THUMBNAIL_REDIS_URL = os.environ.get('REDIS_URL')

Conclusion

And that should be all! Hopefully everything works for you on your first try and you feel like a genius.

I’ve opened an issue for including the THUMBNAIL_REDIS_URL in the Reference documentation.

Overall lesson today is to keep your eye out for strange anomalies like this. Sometimes there is a way to do what you need to do, but someone hasn’t gotten around to documenting it yet.

Maybe that someone could be you!