Saturday, February 10, 2018

Docker: Run as non root user

It's good practice to run processes within a container as a non-root user with restricted permissions.  Even though containers are isolated from the host operating system, they do share the same kernel as the host. Also, processes within a container should be prevented from writing to where they shouldn't be allowed to as extra protection against exploitation.

Running a Docker process as a non-root user has been a Docker feature as of version 1.10. To run a Docker process as a non-root user, permissions need to be accounted for meticulously.  This permission adjustment needs to be done when building a Dockerfile. You need to be aware of where in the filesystem your app might write to, and adjust the permissions accordingly.  Since everything in a container is considered disposable, the container process really shouldn't be writing to too many locations once build.

Here is an annotated example of how you might create a Dockerfile where the process that runs within runs as a user with restricted permissions.  This is just one example of how you can configure this.

Set your image an install a minimal set of OS dependencies as normal.

FROM ubuntu:16.04

RUN apt-get update && apt-get install -y build-essential git \
    python3-dev python3-venv \
    libpq-dev postgresql-client && rm -rf /var/lib/apt/lists/*

Create a user and a group that the application will run as. In this example, we create an "app" user and group.

RUN groupadd -r app && \
    useradd -r -g app -d /home/app -s /sbin/nologin -c "Docker image user" app

Next, create a base directory (SITE_DIR in this case) owned by that app user and group, and set the WORKDIR to that base directory.  This is where everything in the app will live, so everything will be owned by the app user. 


RUN install -g app -o app -d ${SITE_DIR}


Create sub directories needed for the application, also owned by the app user and group.

RUN install -g app -o app -d proj/ var/log/ htdocs/

The group sticky bit needs to be added to every directory in the base directory, so new files and directories will inherit the user and group of their parent directory.

RUN find ${SITE_DIR} -type d -exec chmod g+s {} \;

The group write permission should be set on every file and directory in the base directory.

RUN chmod -R g+w ${SITE_DIR}

Next, switch to the app user created, and subsequent commands will run as that user. We can create a virtualenv and install a packages into that virtualenv.  These installed packages will be owned by that app user.

USER app

RUN python3 -m venv env/
COPY requirements/base.txt requirements.txt

RUN env/bin/pip install -r requirements.txt

Then proceed with the rest of the Dockerfile install instructions as needed.

CMD ["./docker-utils/"]
ENTRYPOINT ["./docker-utils/"]

When running the container with Docker or docker-compose, the process executed as ENTRYPOINT and CMD will run as the user that you specified.

However, if you'd like to override the "USER" Dockerfile directive when running the container, you can pass in the --user parameter and provide a different username.


    To override the default user when using docker-compose:
    docker-compose run --user root app  /bin/bash

    To override the default user when using Docker:

    docker run --user root -it 47b03692214d /bin/bash

Tuesday, October 17, 2017

Docker: Develop with Private Git Repositories in requirements.txt file

In trying to define my ideal Docker and Django development setup, I've been looking for a solution to the following needs:
  1. I've often found the need to install my project's private Git repositories into Docker container via a standard requirements.txt file. 
  2. I also want to easily develop on those private Git repositories, in addition to the Git repo main project repository that contains my Django project.
As you've probably encountered, a standard pip requirements file for installing both pypi packages and private Git repos might look something like the following.

# requirements.txt

If you are not using Docker and you run pip install -r requirements.txt inside of a virtualenv, pip will download pypi packages (in this case Django) into the lib/python3.x/site-packages/ directory of your virtualenv and will create Git checkouts of the Git urls into the "src/" directory of your virtualenv.  If the Git urls are private Git repositories, pip will use your ssh private key to gain access to the git remote (Github in this case). By default, pip will look for your private key in ~/.ssh/id_rsa (but that location can be changed through configuring ssh).

While this might work fine for checking out private Git code from a requirements.txt on your local desktop or onto a secure webserver, installing this code into a Docker container produces other challenges.

Specifically, your Docker container needs access to your ssh private key in order to download the code from the private Git repos. However, if you were to copy your private key into a Docker container using the Dockerfile "COPY" command, a new Docker layer would be created and that key would be stored in the Docker "layer-stack" permanently, even if you were to delete it in a subsequent layer. This is not ideal for security reasons; if you were to push your Docker image to Dockerhub or some other Docker remote, your ssh private key would be baked into it for all of your colleagues to see (and hopefully not steal).

I spent a lot of time looking for a good solution to this problem, but every "solution" seemed hack-ish.

I first thought that maybe you could pass the key into the Docker container via a Docker volume mount. However, the volume mount happens at container runtime (when you execute the "docker run" command), but the container needs access to that key prior to that, at container "build" time (when you execute the "docker build" command). So that really didn't seem like a workable solution.

Second, some blog posts on this topic suggested running an http server on your local machine or elsewhere that would serve up the ssh private key to the container. The Dockerfile could use a single RUN command to download the key into the container, use the key to install the private requirements, and then delete the key. This would prevent the layer from being saved with the key in it. However, this solution seemed over the top, as it requires running a server just for this step.

Another approach that I considered was to pre-download the code from the Git repos outside of the container, with the "pip download" command, prior to running a Docker build (i.e. pip download --no-deps -r requirements.txt -d ./vendor/). The downloaded packages directory could then be copied into the container with the Dockerfile COPY command, and then could be installed inside of the container with "pip install". However, I simply couldn't work out how to make this solution work without defining two requirements files: one with the Git urls and one with filesystem urls to the downloaded packages. I tried a number of things with this approach, but none seemed to be very satisfactory.

I also considered using the "docker secrets" feature to pass in the key to the container. However, this feature seemed to be designed only for use with Docker Swarm, and not plain old Docker containers.  I wanted this solution to work for options other than just Docker Swarm, so didn't end up going this route.

Finally, I found a solution in Docker Mulit-stage builds. Multi-stage builds are a new feature of Docker (as of Docker 17.05) that lets you use one image to build your code and then use a second image to run your code. Multi-stage builds are often used so that you can build your Docker codebase - with all of your compilers and development tools - in the first image, but then only copy over the compiled distribution code to the "runtime image", thus allowing you to fashion more lightweight images.

One key thing about multi-stage builds is that the layer history of the first image is not carried over to the second image. This is perfect for the case of our ssh private key. The first image is where we copy in our private key, create a virtualenv, and install the private repos and other pypi packages into our virtualenv. We can then create a second image, copy the already built virtualenv into that second image, then run that image in a container, and push the image to a Docker remote. At no point does the private ssh key make it into the second image.

Let's get into specifics.

We first need to temporary set your private key to an environment variable so it can be passed into the container at build time.

export ssh_prv_key="$(cat ~/.ssh/id_rsa)"

Next, we need to define a docker-compose file that reads in the private key environment variable and passes that in as a Docker build argument.


version: '2.2'
      context: ./
      dockerfile: Dockerfile
        ssh_prv_key: "$ssh_prv_key"
      #- ./vendor/:/site/vendor/
      - .:/site/proj/
      - DEBUG=True


Next, we define a Docker file. We start with a Docker FROM statement; however, notice the "as builder". This is part of the new magic of multi-stage builds. It lets us temporarily name the image so we may reference this image in a future build step.

FROM python:3.6 as builder

Next, we define and create a working directory where we can store our code. This is mostly Docker housekeeping.

ARG site_dir=/site/
RUN mkdir -p $site_dir
WORKDIR $site_dir

Next, we install whatever Apt packages needed for the build. Notice how we are following the Docker best-practice of removing the temporary Apt apt files.

RUN apt-get update; apt-get install -y \
    python3-dev python3-venv \
    && rm -rf /var/lib/apt/lists/*

Here we read the ssh private key in from an environment variable, save it to a private location, /root/.ssh/id_rsa, and adjust permissions.

RUN chmod 700 /root/.ssh; \
    echo "$ssh_prv_key" > /root/.ssh/id_rsa; \
    chmod 600 /root/.ssh/id_rsa

Next, we tell the ssh config about the key. These settings here are needed to allow the git+ssh pip install to succeed within the container.
RUN echo " IdentityFile /root/.ssh/id_rsa" >> /etc/ssh/ssh_config; \
    echo " StrictHostKeyChecking=no" >> /etc/ssh/ssh_config; \
    echo " UserKnownHostsFile=/dev/null" >> /etc/ssh/ssh_config; \
    echo " GlobalKnownHostsFile=/dev/null" >> /etc/ssh/ssh_config

Then, we create a virtual environment. This not only used to isolate the project packages from the OS image Python packages, but it will be needed to copy all of the Python dependencies from our "build image" to our "runtime image". More on this later.
RUN python3 -m venv env; \
    env/bin/pip install --upgrade pip

Docker copies the requirements file into the container, then pip installs them. I chose to specify the --src flag, which will install all of the Git-downloaded python requirements into the "vendor" directory.

COPY requirements.txt $site_dir/requirements.txt
RUN env/bin/pip install -r $site_dir/requirements.txt --src vendor/

This what makes it all possible! We can define a second FROM statement in a single Dockerfile. Again, this defines a completely new image with no shared layer history from the first.

FROM python:3.6
ARG site_dir=/site/

We install whatever Apt packages are needed to run the app. 

RUN apt-get update; apt-get install -y \
    python3-dev python3-venv \
    && rm -rf /var/lib/apt/lists/*

RUN mkdir -p $site_dir
WORKDIR $site_dir

Finally, we copy the entire site directory (virtualenv included) from the first container to the second. The --from flag lets us specify that the first argument is a directory in the first container.

COPY --from=builder $site_dir $site_dir

CMD ["echo", "done"]

That's basically it! You can build the image as normal using Docker Compose:

docker-compose up --build

And if you have a look at the image that is built, using docker history command, you'll notice that the final image does not contain the ssh private key as a build argument, nor do any of the layers contain the ssh key.

$ docker history testdockermulti_app
0391cd115e89 4 hours ago /bin/sh -c #(nop) CMD ["echo" "done"] 0B
93525a31703c 4 hours ago /bin/sh -c #(nop) COPY dir:49bbdd1f1b1ee2b... 55.1MB
931472817f48 26 hours ago |1 site_dir=/site/ /bin/sh -c apt-get upda... 90MB
2086dc5bbc53 3 days ago /bin/sh -c #(nop) WORKDIR /site/ 0B
5c0aa0069dae 3 days ago |1 site_dir=/site/ /bin/sh -c mkdir -p $si... 0B
969057afd396 3 days ago /bin/sh -c #(nop) ARG site_dir=/site/ 0B
01fd71a97c19 8 days ago /bin/sh -c #(nop) CMD ["python3"] 0B
<missing> 8 days ago /bin/sh -c set -ex; wget -O '... 5.23MB
<missing> 8 days ago /bin/sh -c #(nop) ENV PYTHON_PIP_VERSION=... 0B
<missing> 8 days ago /bin/sh -c cd /usr/local/bin && ln -s idl... 32B

Finally, if you want to work on those private repositories (on your Docker host) as part of your development workflow, you can create a "vendor" directory within your project and clone the codebases into the exact same location that pip would install them.  Then, using a Docker volume mount, you can place the Docker host's copy of the "vendor" directory into the images "vendor" directory.

# on Docker build host
mkdir vendor; cd vendor
git clone  my_project1
git clone  my_project2

# then mount the "vendor" directory 

Anyway, hopefully this might give you some ideas as to how to build Docker images with private resources in requirements.txt.  I'm still refining this process and am working it into my workflow, but I'm glad to hear your thoughts on the topic.

Thursday, June 15, 2017

Django Docker and Celery

I've finally had the time to create a Django+Celery project that can be completely run using Docker and Docker Compose. I'd like to share some of the steps that helped me achieve this. I've created an example project that I've used to demo this process. The example project can be viewed here on Github.

To run this example, you will need to have a recent version of Docker and Docker Compose installed. I'm using Docker 17.03.1 and Docker Compose 1.11.2.

Let's take a look at one of the core files, the docker-compose.yml.   You'll notice that I have defined the following services:
  •  db - the service running the Postgres database container, needed for the Django app
  • rabbitmq - service running the RabbitMQ container, needed for queuing jobs submitted by Celery
  • app - the service containing Django app container
  • worker - the service that runs the Celery worker container
  • web - the service that runs the Nginx container, which proxies web requests to the Django service.  
Some of these services depend on others, and are specified as such by using the depends_on Docker Compose parameter. Some of the services set environment variables to configure the behavior of each container.

       - DATABASE_URL=postgres://postgres@db/postgres
       - CELERY_BROKER_URL=amqp://guest:guest@rabbitmq:5672//
       - SITE_DIR=/site/
       - PROJECT_NAME=dddemo
       - DJANGO_DEBUG=False

Each of these services belong to the network named jaz, and can communicate with each other. For services that only need to communicate internally with other services, I've used the expose configuration parameter to expose given ports to the other containers. Only for the Nginx container, I've used the ports parameter to expose tcp ports 80 and 443 to the Docker host and make the app available for browsing with a web browser.

     image: nginx:1.11
       - "80:80"
       - "443:443"
       - jaz


I've created a Docker volume called static-volume to hold static files generated by the app. These are where static files - copied by Django's collectstatic management command - get copied to. This volume is shared between the app service and the Nginx service, and Nginx serves up the staticfiles. 

         image: nginx:1.11
           - static-volume:/static


The Postgres, RabbitMQ, and Nginx services all are built off of official Docker images for those services.  However, the app and worker services run the Django codebase, and they require some environmental customization.  The build docker-compose keyword lets us specify the Dockerfile to use and define build context (the subdirectory tree the Dockerfile build process has access to).

       context: .
       dockerfile: Dockerfile

Therefore, I've created a Dockerfile to build the app service properly. The Dockerfile uses the ubuntu:16.04 base image; however, a more lightweight image could be used instead. One Docker best practice is to run all the apt-get installs plus a cleanup step in a single Dockerfile RUN command. This ensures that when the Docker layer is created, it is not created with extra temporary files needed by Apt. (Specifically, the layer is created only after doing some space cleanup.)

    RUN apt-get update && apt-get install -y \
        build-essential \
        zlib1g-dev \
        && rm -rf /var/lib/apt/lists/*

Next, in the Dockerfile, we create a virtualenv and some other supporting directories. There is some debate about the need for creating a virtualenv inside of a Docker container. I personally think it is a good idea because it isolates the application Python modules from the OS-level Python modules (just as it does in a non-Docker setup). It offers one more layer of isolation.

    RUN mkdir -p $SITE_DIR
    RUN mkdir -p proj/ var/log/ htdocs/
    RUN python3 -mvenv env/

After creating the virtualenv, I force an upgrade of pip to ensure I'm using the most recent pip version.

    RUN env/bin/pip install pip --upgrade

One important step that may seem out of place is that I copy in the Python requirements.txt file and install the Python requirements early on in the Dockerfile build process. Installing the Python requirements is a time-consuming process, and I can leverage Docker's build-in caching feature to ensure that Docker only needs to install the requirements if a change is specifically made to the requirements file. If I were push that step further down in the Dockerfile, I'd risk unnecessarily re-installing the Python requirements every time I make an arbitrary change to the code.

    COPY requirements.txt requirements.txt
    RUN env/bin/pip install -r requirements.txt

I like to explicitly install uwsgi to ensure it's present, since it's required to even attempt to start the app. I also set some environment variables, such as the database connection string. Note: these environment variables can be overridden by defining them in the environment section of the docker-compose.yml file.

    RUN env/bin/pip install uwsgi


    ENV DJANGO_DATABASE_URL=postgres://postgres@db/postgres

Next, the Dockefile copies a docker-utils folder into the container. This folder contains most of the Docker-specific scripts and configuration files needed to run Django and other services, and this copy makes these files available to the container.

    COPY docker-utils/ docker-utils/

After that, the entire codebase is copied from the current directory to a proj/ directory inside the container. The proj directory will be where the container runs the codebase from.

    COPY . proj/

Finally, the Dockerfile ENTRYPOINT is set to a script called Normally, the entrypoint defaults to a shell executable, such as /bin/bash. However, overriding it allows us to do some interesting things - more on that later.  The Dockerfile CMD is set to another shell script, This specifies the default command (or script) that should be run when the container starts. In this case, the actually runs the Django process by executing the uwsgi command.

    ENTRYPOINT ["./docker-utils/"]
    CMD ["./docker-utils/"]

Let's take a closer look at the This was one of the files in the docker-utils folder copied to the container. The is called every time the container starts, regardless of arguments passed to the container. It is just a shell script that lets us add some additional optional logic. In this example, if someone passes "init" as an argument to the docker-compose run command, we run the Django migrate and collectstatic management commands.


    set -eoux pipefail

    if [ "$1" == 'init' ]; then
        echo "Run Migrations"
        ${SITE_DIR}/env/bin/python ${SITE_DIR}/proj/ migrate
        ${SITE_DIR}/env/bin/python ${SITE_DIR}/proj/ collectstatic --no-input
    elif [ "$1" == 'manage' ]; then
        echo " $@"
        ${SITE_DIR}/env/bin/python ${SITE_DIR}/proj/ $@
        exec "$@"

The command is pretty straightforward as it is a simple wrapper to call uwsgi and run the Django app. There are a few things worth mentioning about it. First, several configuration options are passed into it via environment variables. We use the --chdir flag to change directory to the proj/ directory (containing the codebase) before running. As required by Django, we set the DJANGO_SETTINGS_MODULE environment variable to make uwsgi aware of the Django settings. Another useful setting is the --python-autoreload=1 parameter, which tells uwsgi to reload when it detects a change to the codebase. This operates very similarly to how runserver reloads and is very useful for development. Many of the options are fairly standard uwsgi command options.


echo "Starting uWSGI for ${PROJECT_NAME}"

$SITE_DIR/env/bin/uwsgi --chdir ${SITE_DIR}proj/ \
    --module=${PROJECT_NAME}.wsgi:application \
    --master \
    --vacuum \
    --max-requests=5000 \
    --virtualenv ${SITE_DIR}env/ \
    --socket \
    --processes $NUM_PROCS \
    --threads $NUM_THREADS \

Now that we've talked a bit about the Dockerfile and docker build process, let's take a closer look at the Nginx service in the docker-compose.yml file. The official Nginx Docker image has a very specific way that you are supposed to customize the Nginx configuration.  Specifically, an Nginx configuration template (default.template.conf) is passed into the container via a Docker volume. When the container is executed, the envsubst command combines the configuration template with environment variables from the container and generates the actual Nginx configuration. This allows you to dynamically craft an Nginx configuration file by passing in different environment variables via the docker-compose.yml file.

     image: nginx:1.11

       - ./docker-utils/nginx/default.template.conf:/root/default.template.conf

     command: /bin/bash -c "envsubst '$$NGINX_HTTP_PORT $$NGINX_HTTPS_PORT' < /root/default.template.conf > /etc/nginx/conf.d/default.conf && nginx -g 'daemon off;'"

       - NGINX_HTTP_PORT=80
       - NGINX_HTTPS_PORT=443

Lets take a look at the Celery worker service in the docker-compose.yml file. This service uses the same Dockerfile that was used for the build of the app service, but a different command executes when the container runs. There is nothing magic going on with this command; this simply executes Celery inside of the virtualenv.

       context: .
       dockerfile: Dockerfile
     container_name: dddemo-worker
     command: /site/env/bin/celery worker -A dddemo --workdir /site/proj/ -l info

Finally, we can move away from the Docker-related configuration and take a look at the Celery configuration in the Django project. Much of the following configuration is boilerplate from the Celery 4.0 docs, so I won't go into too much detail.

First, there exists a file inside of the Django project. This integrates Celery into the Django project, and reads in the Celery configuration from Next, there exists an at the root of the project, which initializes the aforementioned The Django contains some Celery configuration, including how to connect to the RabbitMQ service. A very simple Celery add task is defined in; this task will add two numbers passed to it. A very simple Django view hosts a page at the root url and will execute the add task in the background. The results of the add task can be viewed in the taskresult module in the Django Admin.

Though this is a very basic prototype example, it demonstrates a complete Django project and backing services needed to execute Celery jobs. The entire distributed system needed for this application can be executed with only two Docker Compose commands.

To get started, the README describes how to run the project and provides some other common commands that can be used to administer the project. Hope this is useful, and let me know your feedback.

Monday, July 25, 2016

uWSGI Basic Django Setup

Here are two basic examples of almost the same uWSGI configuration to run a Django project; one is configured via an ini configuration file and the other is configured via a command line argument.

This does not represent a production-ready example, but can be used as a starting point for the configuration.

Setup for this example:
# create dir for virtualenv and activate
mkdir envs/
virtualenv envs/runrun/
. ./envs/runrun/bin/activate

# create dir for project codebase
mkdir proj/

# install some django deps
pip install django uwsgi whitenose

# create a new django project
cd proj/
django-admin startproject runrun
cd runrun/

# Add to or modify django to setup static file serving with Whitenoise.
# Note: for prod environments, staticfiles can be served via Nginx.

STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')

Create the config file
$ cat run.ini 
module = runrun.wsgi:application
virtualenv = ../../envs/runrun/
env = DJANGO_SETTINGS_MODULE=runrun.settings 
env = PYTHONPATH="$PYTHONPATH:./runrun/" 
master = true
processes = 5
enable-threads = true
max-requests = 5000
harakiri = 20
vacuum = true
http = :7777
stats =

Execute config
uwsgi --ini run.ini

Or create a shell script
$ cat
 uwsgi --module=runrun.wsgi:application \
      -H ../../envs/runrun/ \
      --env DJANGO_SETTINGS_MODULE=runrun.settings \
      --env PYTHONPATH="$PYTHONPATH:./runrun/" \
      --master \
      --processes=5 \
      --max-requests=5000 \
      --harakiri=20 \
      --vacuum \
      --http= \

Execute the shell script:

Command line reference
--env = set environment variable
--max-requests = reload workers after the specified amount of managed requests
--https2 = http2.0
--http = run in http mode
--socket = use in place of http if using an Nginx reverse proxy
--vacuum = clear environment on exit; try to remove all of the generated file/sockets
--uid = setuid to the specified user/uid
--gid = setgid to the specified group/gid
--stats = run a stats server
--H, --virtualenv = virtualenv dir
--processes = num of worker processes
--module = python module entry point

Wednesday, December 9, 2015

Automatic Maintenance Page for Nginx+Django app

If you've used Django with Nginx, you are probably familiar with how to configure the Nginx process group to reverse proxy to a second Gunicorn or uWSGI Django process group.  (The proxy_pass Nginx parameter passes traffic through Nginx to Django.)

One benefit of this approach is that if your Django process crashes or if you are preforming an upgrade and take Django offline, Nginx can still be available to serve static content and offer some sort of "the system is down" message to users.  With just a few lines of configuration, Nginx can be set to automatically display a splash page in the above scenarios.

If the Django process running behind the reverse proxy becomes unavailable, a 502 error will be emitted by Nginx.  By default, that 502 will be returned to the browser as an ugly error message.  However, Nginx can be configured to catch that 502 error and respond with custom behavior.  Specifically, if a 502 is raised by Nginx, Nginx can check for a custom html error page document and serve up that document if it exists.  If that document does not exist, Nginx can just return the default ugly 502 error message.  The Nginx configuration below executes this described behavior.

This Nginx example is stripped down to just demonstrate the concept described.  It only shows the server section portion of the Nginx config.  This example assumes that you have created your custom error page at the following location: /sites/joejasinski/htdocs/50x.html.


# reverse proxy boilerplate
upstream app_server {
    server unix:/sites/joejasinski/var/run/django.socket;


server {
    listen   443;

    # if Django is unreachable, a 502 is raised...
    error_page 502 @502;
    location @502 {
      root   /sites/joejasinski/htdocs/;
      # try to load a file called 50x.html at the document root
      # or re-raise a generic 502 if no such file is present.
      try_files /50x.html =502;

    # reverse proxy boilerplate for hosting the Django app 
    location / {
        try_files $uri @proxy_to_app;

    # reverse proxy boilerplate for hosting the Django app
    location @proxy_to_app {
       proxy_pass http://app_server;

It's worth mentioning that, by default, if your Django app raises a 502 error behind the reverse proxy, Nginx will not catch it and thus will not respond with the custom Nginx 502 page.  The Nginx 502 error page is only displayed for 502 errors that originate from Nginx; an unavailable reverse proxy application qualifies as such.

This behavior is desirable since Django has it's own error handler for handling 50x errors within the application; we want Nginx to display an error page when Django is down, but Django should handle all errors generated by itself.

Note: you can change this behavior to have Nginx handle any error generated by the reverse proxied application.  This can be done with the proxy_intercept_errors Nginx parameter.

Note: using a similar approach to the above, you can also create error Nginx-based error handlers for other error codes generated by Nginx.

In summary, this approach allows you to automatically show users a friendly error page if your Django process crashes or is down for maintenance.

Wednesday, July 20, 2011

Django: Using Caching to Track Online Users

Recently I wanted a simple solution to track whether a user is online on a given Django site.  The definition of "online" on a site is kind of ambiguous, so I'll define that a user is considered to be online if they have made any request to the site in the last five minutes.

I found that one approach is to use Django's caching framework to track when a user last accessed the site.  For example, upon each request, I can have a middleware set the current time as a cache value associated with a given user.  This allows us to store some basic information about logged-in user's online state without having to hit the database on each request and easily retrieve it by accessing the cache.

My approach below.  Comments welcome.

# add the middleware that you are about to create to settings

# Setup caching per Django docs. In actuality, you'd probably use memcached instead of local memory.
    'default': {
        'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
        'LOCATION': 'default-cache'

# Number of seconds of inactivity before a user is marked offline

# Number of seconds that we will keep track of inactive users for before 
# their last seen is removed from the cache
USER_LASTSEEN_TIMEOUT = 60 * 60 * 24 * 7

import datetime
from django.core.cache import cache
from django.conf import settings

class ActiveUserMiddleware:

    def process_request(self, request):
        current_user = request.user
        if request.user.is_authenticated():
            now =
            cache.set('seen_%s' % (current_user.username), now, 

In your UserProfile module or some other model associated with the user:
class UserProfile(models.Model):

    def last_seen(self):
        return cache.get('seen_%s' % self.user.username)

    def online(self):
        if self.last_seen():
            now =
            if now > self.last_seen() + datetime.timedelta(
                return False
                return True
            return False

Then in the template where you want to display whether the user is online or not:
{% with request.user.get_profile as profile %}

   <tr><th>Last Seen</th><td>{% if profile.last_seen %}{{ profile.last_seen|timesince }}{% else %}awhile{% endif %} ago</td></tr>
   <tr><th>Online</th><td>{{ }}</td></tr>

{% endwith %}

 - Simple solution
 - Doesn't need to hit the database for saving the timestamp each request

  - Last user access times are cleared if the server is rebooted or cache is reset
  - Last user access times are accessible only as long as they exist in the cache.

Friday, July 8, 2011

Django Permission TemplateTag

In a previous post, I wrote about a way to keep track of user permissions on a model instance.  For example, I suggested that each model have a permissions subclass that could be instantiated with a user instance passed as a constructor argument.  Methods on that permissions class could then be called to determine if that user has permission to perform various actions.

I also suggested that the threadlocals module could then be used to pass in the user instance to the permissions object in the Django template.  However, from various readings, I get the impression that threadlocals may not be the best thing for passing arguments in a template function.   Therefore, I decided to use a more traditional route of creating a template tag to do something similar.

I created a template tag that lets you surround a block of HTML code to hide or show the contents based on the return value of the permission function.  The tag below basically says, "if the logged in user has 'can_edit_group' permission on the given 'group' object instance, then display the Edit link".

Reference the original post for details.

In the Django template
{% load permission_tags %}
{% permission request.user can_edit_group on group %}
<a href="">Edit</a>
{% endpermission %}

Here is the templatetag definition that fits the example above.

In templatetags/
from django import template
register = template.Library()

def permission(parser, token):
        # get the arguments passed to the template tag; 
        # first argument is the tag name
        tag_name, username, permission, onkeyword, object = token.split_contents()
    except ValueError:
        raise template.TemplateSyntaxError("%r tag requires exactly 4 arguments" % token.contents.split()[0])
    # look for the 'endpermission' terminator tag
    nodelist = parser.parse(('endpermission',))
    return PermissionNode(nodelist, username, permission, object)

class PermissionNode(template.Node):
    def __init__(self, nodelist, user, permission, object):
        self.nodelist = nodelist
        # evaluate the user instance as a variable and store
        self.user = template.Variable(user)
        # store the permission string
        self.permission = permission
        # evaluate the object instance as a variable and store
        self.object = template.Variable(object)

    def render(self, context):
        user_inst = self.user.resolve(context)
        object_inst = self.object.resolve(context)
        # create a new permissions object by calling a permissions 
        # factory method of the model class
        permissions_obj = object_inst.permissions(user_inst)
        content = self.nodelist.render(context)
        if hasattr(permissions_obj, self.permission):
            # check to see if the permissions object has the permissions method
            # provided in the template tag
            perm_func = getattr(permissions_obj, self.permission)
            # execute that permissions method
            if perm_func():
                return content 
        return ""

register.tag('permission', permission)

This tag currently works like an 'if' template tag and shows/hides anything wrapped between the permission and endpermission tags.  A future goal may be to make this work like an if/else tag so I can specify an else condition.