Monday, July 25, 2016

uWSGI Basic Django Setup

Here are two basic examples of almost the same uWSGI configuration to run a Django project; one is configured via an ini configuration file and the other is configured via a command line argument.

This does not represent a production-ready example, but can be used as a starting point for the configuration.

Setup for this example:
# create dir for virtualenv and activate
mkdir envs/
virtualenv envs/runrun/
. ./envs/runrun/bin/activate

# create dir for project codebase
mkdir proj/

# install some django deps
pip install django uwsgi whitenose

# create a new django project
cd proj/
django-admin startproject runrun
cd runrun/

# Add to or modify django settings.py to setup static file serving with Whitenoise.
# Note: for prod environments, staticfiles can be served via Nginx.
# settings.py 
MIDDLEWARE_CLASSES = [
    ....
    'whitenoise.middleware.WhiteNoiseMiddleware',

]
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'

Create the config file
$ cat run.ini 
[uwsgi]
module = runrun.wsgi:application
virtualenv = ../../envs/runrun/
env = DJANGO_SETTINGS_MODULE=runrun.settings 
env = PYTHONPATH="$PYTHONPATH:./runrun/" 
master = true
processes = 5
enable-threads = true
max-requests = 5000
harakiri = 20
vacuum = true
http = :7777
stats = 127.0.0.1:9191

Execute config
uwsgi --ini run.ini

Or create a shell script
$ cat run.sh
 uwsgi --module=runrun.wsgi:application \
      -H ../../envs/runrun/ \
      --env DJANGO_SETTINGS_MODULE=runrun.settings \
      --env PYTHONPATH="$PYTHONPATH:./runrun/" \
      --master \
      --processes=5 \
      --max-requests=5000 \
      --harakiri=20 \
      --vacuum \
      --http=127.0.0.1:7777 \
      --stats 127.0.0.1:9191 

Execute the shell script:
./run.sh

Command line reference
--env = set environment variable
--max-requests = reload workers after the specified amount of managed requests
--https2 = http2.0
--http = run in http mode
--socket = use in place of http if using an Nginx reverse proxy
--vacuum = clear environment on exit; try to remove all of the generated file/sockets
--uid = setuid to the specified user/uid
--gid = setgid to the specified group/gid
--stats = run a stats server
--H, --virtualenv = virtualenv dir
--processes = num of worker processes
--module = python module entry point

Wednesday, December 9, 2015

Automatic Maintenance Page for Nginx+Django app

If you've used Django with Nginx, you are probably familiar with how to configure the Nginx process group to reverse proxy to a second Gunicorn or uWSGI Django process group.  (The proxy_pass Nginx parameter passes traffic through Nginx to Django.)

One benefit of this approach is that if your Django process crashes or if you are preforming an upgrade and take Django offline, Nginx can still be available to serve static content and offer some sort of "the system is down" message to users.  With just a few lines of configuration, Nginx can be set to automatically display a splash page in the above scenarios.

If the Django process running behind the reverse proxy becomes unavailable, a 502 error will be emitted by Nginx.  By default, that 502 will be returned to the browser as an ugly error message.  However, Nginx can be configured to catch that 502 error and respond with custom behavior.  Specifically, if a 502 is raised by Nginx, Nginx can check for a custom html error page document and serve up that document if it exists.  If that document does not exist, Nginx can just return the default ugly 502 error message.  The Nginx configuration below executes this described behavior.

This Nginx example is stripped down to just demonstrate the concept described.  It only shows the server section portion of the Nginx config.  This example assumes that you have created your custom error page at the following location: /sites/joejasinski/htdocs/50x.html.

server.conf
...

# reverse proxy boilerplate
upstream app_server {
    server unix:/sites/joejasinski/var/run/django.socket;

}

server {
    listen   443;
    server_name  joejasinski.com www.joejasinski.com;
    ...

    # if Django is unreachable, a 502 is raised...
    error_page 502 @502;
    location @502 {
      root   /sites/joejasinski/htdocs/;
      # try to load a file called 50x.html at the document root
      # or re-raise a generic 502 if no such file is present.
      try_files /50x.html =502;
   }

    # reverse proxy boilerplate for hosting the Django app 
    location / {
        try_files $uri @proxy_to_app;
    }

    # reverse proxy boilerplate for hosting the Django app
    location @proxy_to_app {
        ....
       proxy_pass http://app_server;
    }
}


It's worth mentioning that, by default, if your Django app raises a 502 error behind the reverse proxy, Nginx will not catch it and thus will not respond with the custom Nginx 502 page.  The Nginx 502 error page is only displayed for 502 errors that originate from Nginx; an unavailable reverse proxy application qualifies as such.

This behavior is desirable since Django has it's own error handler for handling 50x errors within the application; we want Nginx to display an error page when Django is down, but Django should handle all errors generated by itself.

Note: you can change this behavior to have Nginx handle any error generated by the reverse proxied application.  This can be done with the proxy_intercept_errors Nginx parameter.

Note: using a similar approach to the above, you can also create error Nginx-based error handlers for other error codes generated by Nginx.

In summary, this approach allows you to automatically show users a friendly error page if your Django process crashes or is down for maintenance.

Wednesday, July 20, 2011

Django: Using Caching to Track Online Users

Recently I wanted a simple solution to track whether a user is online on a given Django site.  The definition of "online" on a site is kind of ambiguous, so I'll define that a user is considered to be online if they have made any request to the site in the last five minutes.

I found that one approach is to use Django's caching framework to track when a user last accessed the site.  For example, upon each request, I can have a middleware set the current time as a cache value associated with a given user.  This allows us to store some basic information about logged-in user's online state without having to hit the database on each request and easily retrieve it by accessing the cache.

My approach below.  Comments welcome.

In settings.py:
# add the middleware that you are about to create to settings
MIDDLEWARE_CLASSES = (
    ....
    'middleware.activeuser_middleware.ActiveUserMiddleware',
    ....
)

# Setup caching per Django docs. In actuality, you'd probably use memcached instead of local memory.
CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
        'LOCATION': 'default-cache'
    }
}

# Number of seconds of inactivity before a user is marked offline
USER_ONLINE_TIMEOUT = 300

# Number of seconds that we will keep track of inactive users for before 
# their last seen is removed from the cache
USER_LASTSEEN_TIMEOUT = 60 * 60 * 24 * 7

In activeuser_middleware.py:
import datetime
from django.core.cache import cache
from django.conf import settings

class ActiveUserMiddleware:

    def process_request(self, request):
        current_user = request.user
        if request.user.is_authenticated():
            now = datetime.datetime.now()
            cache.set('seen_%s' % (current_user.username), now, 
                           settings.USER_LASTSEEN_TIMEOUT)

In your UserProfile module or some other model associated with the user:
class UserProfile(models.Model):
    ....
    ....

    def last_seen(self):
        return cache.get('seen_%s' % self.user.username)

    def online(self):
        if self.last_seen():
            now = datetime.datetime.now()
            if now > self.last_seen() + datetime.timedelta(
                         seconds=settings.USER_ONLINE_TIMEOUT):
                return False
            else:
                return True
        else:
            return False

Then in the template where you want to display whether the user is online or not:
{% with request.user.get_profile as profile %}

 <table>
   <tr><th>Last Seen</th><td>{% if profile.last_seen %}{{ profile.last_seen|timesince }}{% else %}awhile{% endif %} ago</td></tr>
   <tr><th>Online</th><td>{{ profile.online }}</td></tr>
 </table>

{% endwith %}

Pros:
 - Simple solution
 - Doesn't need to hit the database for saving the timestamp each request

Cons:
  - Last user access times are cleared if the server is rebooted or cache is reset
  - Last user access times are accessible only as long as they exist in the cache.

Friday, July 8, 2011

Django Permission TemplateTag

In a previous post, I wrote about a way to keep track of user permissions on a model instance.  For example, I suggested that each model have a permissions subclass that could be instantiated with a user instance passed as a constructor argument.  Methods on that permissions class could then be called to determine if that user has permission to perform various actions.

I also suggested that the threadlocals module could then be used to pass in the user instance to the permissions object in the Django template.  However, from various readings, I get the impression that threadlocals may not be the best thing for passing arguments in a template function.   Therefore, I decided to use a more traditional route of creating a template tag to do something similar.

I created a template tag that lets you surround a block of HTML code to hide or show the contents based on the return value of the permission function.  The tag below basically says, "if the logged in user has 'can_edit_group' permission on the given 'group' object instance, then display the Edit link".

Reference the original post for details.

In the Django template
{% load permission_tags %}
{% permission request.user can_edit_group on group %}
<a href="">Edit</a>
{% endpermission %}

Here is the templatetag definition that fits the example above.

In templatetags/permissions_tags.py
from django import template
register = template.Library()

def permission(parser, token):
    try:
        # get the arguments passed to the template tag; 
        # first argument is the tag name
        tag_name, username, permission, onkeyword, object = token.split_contents()
    except ValueError:
        raise template.TemplateSyntaxError("%r tag requires exactly 4 arguments" % token.contents.split()[0])
    # look for the 'endpermission' terminator tag
    nodelist = parser.parse(('endpermission',))
    parser.delete_first_token()
    return PermissionNode(nodelist, username, permission, object)


class PermissionNode(template.Node):
    def __init__(self, nodelist, user, permission, object):
        self.nodelist = nodelist
        # evaluate the user instance as a variable and store
        self.user = template.Variable(user)
        # store the permission string
        self.permission = permission
        # evaluate the object instance as a variable and store
        self.object = template.Variable(object)

    def render(self, context):
        user_inst = self.user.resolve(context)
        object_inst = self.object.resolve(context)
        
        # create a new permissions object by calling a permissions 
        # factory method of the model class
        permissions_obj = object_inst.permissions(user_inst)
        
        content = self.nodelist.render(context)
        
        if hasattr(permissions_obj, self.permission):
            # check to see if the permissions object has the permissions method
            # provided in the template tag
            perm_func = getattr(permissions_obj, self.permission)
            # execute that permissions method
            if perm_func():
                return content 
        return ""

register.tag('permission', permission)

This tag currently works like an 'if' template tag and shows/hides anything wrapped between the permission and endpermission tags.  A future goal may be to make this work like an if/else tag so I can specify an else condition.

Wednesday, June 22, 2011

Django: Logging Quickstart in Django 1.3

Logging is always one of those things that seems like there is a lot of boilerplate code that I tend to always have to lookup oneline each time.  Though logging is very customizable, most of the time I need simply to only need log to a file or files.   Therefore, I'm writing this in hopes of compling a simple configuration that can be dropped into a project.  My understanding is still rudimentary so feel free to drop a comment.

A logging framework is built into Django 1.3 which makes it easier to integrate into a given project.  The logging framework contains three main components: formatters, handlers, and loggers.  Formatters determine how the output will be displayed when it is logged.  Handlers handle where output is logged, whether that be a file, the counsole, or an email.  Loggers are associated with handlers and are the objects that one interacts with when logging something.  As in the example below, more than one formatter, handler or logger can be defined in a project.

In settings.py:
import os

# choose a path that is your virtual environment root
VENV_ROOT = os.path.join('/','web','myvenv')

LOGGING = {
    'version': 1,
    'disable_existing_loggers': True,
    'formatters': {
        'verbose': {
            'format': '%(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s'
        },
        'simple': {
            'format': '%(levelname)s %(message)s'
        },
    },
    'handlers': {
        'file_userlogins': {                # define and name a handler
            'level': 'DEBUG',
            'class': 'logging.FileHandler', # set the logging class to log to a file
            'formatter': 'verbose',         # define the formatter to associate
            'filename': os.path.join(VENV_ROOT, 'log', 'userlogins.log') # log file
        },

        'file_usersaves': {                 # define and name a second handler
            'level': 'DEBUG',
            'class': 'logging.FileHandler', # set the logging class to log to a file
            'formatter': 'verbose',         # define the formatter to associate
            'filename': os.path.join(VENV_ROOT, 'log', 'usersaves.log')  # log file
        },
    },
    'loggers': {
        'logview.userlogins': {              # define a logger - give it a name
            'handlers': ['file_userlogins'], # specify what handler to associate
            'level': 'INFO',                 # specify the logging level
            'propagate': True,
        },     

        'logview.usersaves': {               # define another logger
            'handlers': ['file_usersaves'],  # associate a different handler
            'level': 'INFO',                 # specify the logging level
            'propagate': True,
        },        
    }       
}

In application code such as in views.py:
import logging   # import the required logging module
logger_logins = logging.getLogger('logview.userlogins')  # logger from settings.py
logger_logins.info('Log info')
logger_logins.debug('Log Debug information %s' % ("can pass in variables"))

# we can log to a different file using the other logger
logger_saves = logging.getLogger('logview.usersaves')
logger_saves.error('log an error')

Tuesday, June 7, 2011

Django: Executing manage.py from Anywhere

One nice convenience would be to have a way to call manage.py from anywhere within a virtual environment.  For example, if I want to execute 'manage.py shell', it would be nice to be able to execute that from anywhere rather than having to change directory to where manage.py lives.

We can already run management commands from anywhere using django-admin by utilizing the --settings flag, but doing this requires a lot of typing. And wrapping the django-admin in a shell script seems inelegant somehow.

Instead, I wrote a small python script based off of manage.py.  This can be placed in the ./bin directory in the virtualenv.  The DjangoManageSettings object is meant to be customized on a per project basis, and I intend to make it so it can be passed in from a file configuration somewhere.

Anyway, thought I would share.  I'm curious to see other approaches to this.

# ---------- ${VIRTUAL_ENV}/bin/django_manage.py -------------------------
#!/usr/bin/env python

import os, sys
from django.core.management import execute_manager


class DjangoManageSettings(object):

    def __init__(self):
        self.VENV_ROOT = os.getenv("VIRTUAL_ENV")
        # The name of the module that settings.py resides in
        self.PROJECT_NAME="myproject"
        # The path to the directory where the Django project lives. 
        # I made it relative to the virtualenv root. 
        self.PROJECT_PARENT_PATH = os.path.join(self.VENV_ROOT, "app")
        # The name of the django settings model in the project
        self.SETTINGS_MODULE_NAME='settings'


dm_settings = DjangoManageSettings()

if not dm_settings.VENV_ROOT:
    print("A virtual environment must be activated before using this script.")
    exit(1)


sys.path.insert(0, os.path.join(dm_settings.PROJECT_PARENT_PATH))
mod = __import__(dm_settings.PROJECT_NAME, globals(), locals(), 
                 [dm_settings.SETTINGS_MODULE_NAME])
settings = getattr(mod, dm_settings.SETTINGS_MODULE_NAME)

if __name__ == "__main__":
    execute_manager(settings)

Wednesday, May 25, 2011

Django Models Mixins

One thing I've been experimenting with is model Mixins.  For example, the aim is to create small abstract classes that are each focused around a particular function.  These abstract classes can then be added to arbitrary models to apply those functions to models as desired.

For example, say I define a RatingsFields abstract class and a TrackingFields abstract class.  These abstract classes can be mixed into any other model that we wish to add rating or tracking functionality to.

core/mixins.py
from djangoratings.fields import RatingField # 3rd party module

class RatingFields(models.Model):
    
    rating = RatingField(range=5) # 5 possible rating values, 1-5

    class Meta:
        abstract = True
        

class TrackingFields(models.Model):

    deleted_on = models.DateTimeField(blank=True, null=True)
    created = models.DateTimeField(auto_now_add=True)
    modified = models.DateTimeField(auto_now=True)

    class Meta:
        abstract = True
        

Since we applied the abstract classes to the Post model below, the model now has rating and tracking capabilities.  This is useful to help simplify code where a lot of models share fields or methods with the same function.

myapp/models.py
from core import mixins as core_mixins
class Post(core_mixins.TrackingFields, core_mixins.RatingFields):
    name = models.CharField(max_length=128)
    slug = models.SlugField(max_length=128)
    ...

Comments welcome.
Joe