Wednesday, July 20, 2011

Django: Using Caching to Track Online Users

Recently I wanted a simple solution to track whether a user is online on a given Django site.  The definition of "online" on a site is kind of ambiguous, so I'll define that a user is considered to be online if they have made any request to the site in the last five minutes.

I found that one approach is to use Django's caching framework to track when a user last accessed the site.  For example, upon each request, I can have a middleware set the current time as a cache value associated with a given user.  This allows us to store some basic information about logged-in user's online state without having to hit the database on each request and easily retrieve it by accessing the cache.

My approach below.  Comments welcome.

In settings.py:
# add the middleware that you are about to create to settings
MIDDLEWARE_CLASSES = (
    ....
    'middleware.activeuser_middleware.ActiveUserMiddleware',
    ....
)

# Setup caching per Django docs. In actuality, you'd probably use memcached instead of local memory.
CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
        'LOCATION': 'default-cache'
    }
}

# Number of seconds of inactivity before a user is marked offline
USER_ONLINE_TIMEOUT = 300

# Number of seconds that we will keep track of inactive users for before 
# their last seen is removed from the cache
USER_LASTSEEN_TIMEOUT = 60 * 60 * 24 * 7

In activeuser_middleware.py:
import datetime
from django.core.cache import cache
from django.conf import settings

class ActiveUserMiddleware:

    def process_request(self, request):
        current_user = request.user
        if request.user.is_authenticated():
            now = datetime.datetime.now()
            cache.set('seen_%s' % (current_user.username), now, 
                           settings.USER_LASTSEEN_TIMEOUT)

In your UserProfile module or some other model associated with the user:
class UserProfile(models.Model):
    ....
    ....

    def last_seen(self):
        return cache.get('seen_%s' % self.user.username)

    def online(self):
        if self.last_seen():
            now = datetime.datetime.now()
            if now > self.last_seen() + datetime.timedelta(
                         seconds=settings.USER_ONLINE_TIMEOUT):
                return False
            else:
                return True
        else:
            return False

Then in the template where you want to display whether the user is online or not:
{% with request.user.get_profile as profile %}

 <table>
   <tr><th>Last Seen</th><td>{% if profile.last_seen %}{{ profile.last_seen|timesince }}{% else %}awhile{% endif %} ago</td></tr>
   <tr><th>Online</th><td>{{ profile.online }}</td></tr>
 </table>

{% endwith %}

Pros:
 - Simple solution
 - Doesn't need to hit the database for saving the timestamp each request

Cons:
  - Last user access times are cleared if the server is rebooted or cache is reset
  - Last user access times are accessible only as long as they exist in the cache.

Friday, July 8, 2011

Django Permission TemplateTag

In a previous post, I wrote about a way to keep track of user permissions on a model instance.  For example, I suggested that each model have a permissions subclass that could be instantiated with a user instance passed as a constructor argument.  Methods on that permissions class could then be called to determine if that user has permission to perform various actions.

I also suggested that the threadlocals module could then be used to pass in the user instance to the permissions object in the Django template.  However, from various readings, I get the impression that threadlocals may not be the best thing for passing arguments in a template function.   Therefore, I decided to use a more traditional route of creating a template tag to do something similar.

I created a template tag that lets you surround a block of HTML code to hide or show the contents based on the return value of the permission function.  The tag below basically says, "if the logged in user has 'can_edit_group' permission on the given 'group' object instance, then display the Edit link".

Reference the original post for details.

In the Django template
{% load permission_tags %}
{% permission request.user can_edit_group on group %}
<a href="">Edit</a>
{% endpermission %}

Here is the templatetag definition that fits the example above.

In templatetags/permissions_tags.py
from django import template
register = template.Library()

def permission(parser, token):
    try:
        # get the arguments passed to the template tag; 
        # first argument is the tag name
        tag_name, username, permission, onkeyword, object = token.split_contents()
    except ValueError:
        raise template.TemplateSyntaxError("%r tag requires exactly 4 arguments" % token.contents.split()[0])
    # look for the 'endpermission' terminator tag
    nodelist = parser.parse(('endpermission',))
    parser.delete_first_token()
    return PermissionNode(nodelist, username, permission, object)


class PermissionNode(template.Node):
    def __init__(self, nodelist, user, permission, object):
        self.nodelist = nodelist
        # evaluate the user instance as a variable and store
        self.user = template.Variable(user)
        # store the permission string
        self.permission = permission
        # evaluate the object instance as a variable and store
        self.object = template.Variable(object)

    def render(self, context):
        user_inst = self.user.resolve(context)
        object_inst = self.object.resolve(context)
        
        # create a new permissions object by calling a permissions 
        # factory method of the model class
        permissions_obj = object_inst.permissions(user_inst)
        
        content = self.nodelist.render(context)
        
        if hasattr(permissions_obj, self.permission):
            # check to see if the permissions object has the permissions method
            # provided in the template tag
            perm_func = getattr(permissions_obj, self.permission)
            # execute that permissions method
            if perm_func():
                return content 
        return ""

register.tag('permission', permission)

This tag currently works like an 'if' template tag and shows/hides anything wrapped between the permission and endpermission tags.  A future goal may be to make this work like an if/else tag so I can specify an else condition.

Wednesday, June 22, 2011

Django: Logging Quickstart in Django 1.3

Logging is always one of those things that seems like there is a lot of boilerplate code that I tend to always have to lookup oneline each time.  Though logging is very customizable, most of the time I need simply to only need log to a file or files.   Therefore, I'm writing this in hopes of compling a simple configuration that can be dropped into a project.  My understanding is still rudimentary so feel free to drop a comment.

A logging framework is built into Django 1.3 which makes it easier to integrate into a given project.  The logging framework contains three main components: formatters, handlers, and loggers.  Formatters determine how the output will be displayed when it is logged.  Handlers handle where output is logged, whether that be a file, the counsole, or an email.  Loggers are associated with handlers and are the objects that one interacts with when logging something.  As in the example below, more than one formatter, handler or logger can be defined in a project.

In settings.py:
import os

# choose a path that is your virtual environment root
VENV_ROOT = os.path.join('/','web','myvenv')

LOGGING = {
    'version': 1,
    'disable_existing_loggers': True,
    'formatters': {
        'verbose': {
            'format': '%(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s'
        },
        'simple': {
            'format': '%(levelname)s %(message)s'
        },
    },
    'handlers': {
        'file_userlogins': {                # define and name a handler
            'level': 'DEBUG',
            'class': 'logging.FileHandler', # set the logging class to log to a file
            'formatter': 'verbose',         # define the formatter to associate
            'filename': os.path.join(VENV_ROOT, 'log', 'userlogins.log') # log file
        },

        'file_usersaves': {                 # define and name a second handler
            'level': 'DEBUG',
            'class': 'logging.FileHandler', # set the logging class to log to a file
            'formatter': 'verbose',         # define the formatter to associate
            'filename': os.path.join(VENV_ROOT, 'log', 'usersaves.log')  # log file
        },
    },
    'loggers': {
        'logview.userlogins': {              # define a logger - give it a name
            'handlers': ['file_userlogins'], # specify what handler to associate
            'level': 'INFO',                 # specify the logging level
            'propagate': True,
        },     

        'logview.usersaves': {               # define another logger
            'handlers': ['file_usersaves'],  # associate a different handler
            'level': 'INFO',                 # specify the logging level
            'propagate': True,
        },        
    }       
}

In application code such as in views.py:
import logging   # import the required logging module
logger_logins = logging.getLogger('logview.userlogins')  # logger from settings.py
logger_logins.info('Log info')
logger_logins.debug('Log Debug information %s' % ("can pass in variables"))

# we can log to a different file using the other logger
logger_saves = logging.getLogger('logview.usersaves')
logger_saves.error('log an error')

Tuesday, June 7, 2011

Django: Executing manage.py from Anywhere

One nice convenience would be to have a way to call manage.py from anywhere within a virtual environment.  For example, if I want to execute 'manage.py shell', it would be nice to be able to execute that from anywhere rather than having to change directory to where manage.py lives.

We can already run management commands from anywhere using django-admin by utilizing the --settings flag, but doing this requires a lot of typing. And wrapping the django-admin in a shell script seems inelegant somehow.

Instead, I wrote a small python script based off of manage.py.  This can be placed in the ./bin directory in the virtualenv.  The DjangoManageSettings object is meant to be customized on a per project basis, and I intend to make it so it can be passed in from a file configuration somewhere.

Anyway, thought I would share.  I'm curious to see other approaches to this.

# ---------- ${VIRTUAL_ENV}/bin/django_manage.py -------------------------
#!/usr/bin/env python

import os, sys
from django.core.management import execute_manager


class DjangoManageSettings(object):

    def __init__(self):
        self.VENV_ROOT = os.getenv("VIRTUAL_ENV")
        # The name of the module that settings.py resides in
        self.PROJECT_NAME="myproject"
        # The path to the directory where the Django project lives. 
        # I made it relative to the virtualenv root. 
        self.PROJECT_PARENT_PATH = os.path.join(self.VENV_ROOT, "app")
        # The name of the django settings model in the project
        self.SETTINGS_MODULE_NAME='settings'


dm_settings = DjangoManageSettings()

if not dm_settings.VENV_ROOT:
    print("A virtual environment must be activated before using this script.")
    exit(1)


sys.path.insert(0, os.path.join(dm_settings.PROJECT_PARENT_PATH))
mod = __import__(dm_settings.PROJECT_NAME, globals(), locals(), 
                 [dm_settings.SETTINGS_MODULE_NAME])
settings = getattr(mod, dm_settings.SETTINGS_MODULE_NAME)

if __name__ == "__main__":
    execute_manager(settings)

Wednesday, May 25, 2011

Django Models Mixins

One thing I've been experimenting with is model Mixins.  For example, the aim is to create small abstract classes that are each focused around a particular function.  These abstract classes can then be added to arbitrary models to apply those functions to models as desired.

For example, say I define a RatingsFields abstract class and a TrackingFields abstract class.  These abstract classes can be mixed into any other model that we wish to add rating or tracking functionality to.

core/mixins.py
from djangoratings.fields import RatingField # 3rd party module

class RatingFields(models.Model):
    
    rating = RatingField(range=5) # 5 possible rating values, 1-5

    class Meta:
        abstract = True
        

class TrackingFields(models.Model):

    deleted_on = models.DateTimeField(blank=True, null=True)
    created = models.DateTimeField(auto_now_add=True)
    modified = models.DateTimeField(auto_now=True)

    class Meta:
        abstract = True
        

Since we applied the abstract classes to the Post model below, the model now has rating and tracking capabilities.  This is useful to help simplify code where a lot of models share fields or methods with the same function.

myapp/models.py
from core import mixins as core_mixins
class Post(core_mixins.TrackingFields, core_mixins.RatingFields):
    name = models.CharField(max_length=128)
    slug = models.SlugField(max_length=128)
    ...

Comments welcome.
Joe

Tuesday, April 26, 2011

Django Admin Override Save for Model

Sometimes it's nice to be able to add custom code to the save method of objects in the Django Admin.  So, when editing an object on the Admin object detail page (change form), adding the following method override to your ModelAdmin in admin.py will allow you to add custom code to the save function.

In admin.py:  
class MyModelAdmin(admin.ModelAdmin):

    def save_model(self, request, obj, form, change):
        # custom stuff here
        obj.save()

This is documented in the Django Docs, but I found it particularly useful.

Wednesday, April 13, 2011

Django Pattern for Reporting Errors/Messages in Views Part 2: Managers

In my last post, I discussed how you can use a simple UserMessage class in your views to report simple error messages.   This way, you can keep track of errors and report them to the users in a clean fashion.

One thing I didn't expand upon much is how to create some manager methods to cleanly lookup model instances and report any errors to the UserMessage class.  I think this approach is a little nicer than catching exceptions all over the place, because you don't have to clutter your views with try/catch blocks.

In the example below, we create a custom manager that looks up a model instance of "MyModel" with the given hash.  If the selected instance of "MyModel" does not exist or if some other error occurs, it writes an error message to "message" instance passed into it.  You can customize this as you see fit.  For example, in the code below, I pass in a "user" instance that can be used to check permissions in this method. 

class MyModelManager(models.Manager):
    # In this case, we pass in a user, a unique hash for a 
    # "MyModel" object instance, and an optional message. 
    def get_mymodel(self, user, mymodel_hash, message=None):
        mymodel = []
        # If the message previously set an error, return immediately  
        if message:
            return None, message
        try:
            # if the user is an admin, search all MyModels for the matching hash 
            if user.is_staff:
                mymodel = MyModel.objects.get(hash=mymodel_hash, enabled=True)
            # if the user is NOT an admin, search user-related MyModels for the matching hash 
            else:
                mymodel = user.mymodel_set.get(hash=mymodel_hash, enabled=True)
        except MyModel.DoesNotExist:
            message = UserMessage()
            message.title = _("Not found")
            message.text = _("MyModel does not exist")
        return mymodel, message

Saturday, April 2, 2011

Django Pattern for Reporting Errors/Messages in Views

I've tried tried to find a decent way to design my Django views so that I can smoothly report errors to users.  The errors that I am concerned about are errors that users encounter if they are preforming actions that are typically NOT normally encountered through normal use of the user interface.  For example, if users try to directly access a URL of an object that doesn't exist, if they post incorrect values to a URL, or a required value doesn't exist in the user's session.

In a utility module, I create a message class that I to store message information.  While view processing is taking place, if an error takes place, this class will be used to store the error title, text, and "back" url link.

core/utils.py
class UserMessage():
    def __init__(self, title="", text=[], url=None):
        self.title = title
        self.text = text if hasattr(text, '__iter__') else [text]
        self.url = url 

The view continually checks for the existance of a UserMessage instance.  (In the example below, "message" variable is an instance of UserMessage)  If an error condition occurs, part of the handing will be to create a new UserMessage, and the view renders an error page using the template defined by message_template_name and the variables set in the UserMessage instance.

myapp/views.py
def  detail_view(request, myobject_slug, template_name="", 
                 message_template_name="core/message.html"):
    context = {}
    message = None
    if not message: 
        # call custom manager method; pass in message for error handling
        myobject, message = MyObject.objects.get_myobject(myobject_slug, message)
    if not message:
        if not myobject.is_authroized(...):   # do come check here 
            message = utils.UserMessage("Permission Denied",  
                     "You are not authorized to view this.") 
    if not message:
        if myobject.is_wrong_type(...):   # do some check here
            message = utils.UserMessage("Invalid Type",   
                    "This is the wrong type")
            message.url = reverse('index')
    if not message:
        # normal view stuff here  
        form = forms.MyForm(....)
        if request.method == "POST":
            form = forms.MyForm(data=request.POST, ....)
            if form.is_valid():
                form.save() 
                # do other stuff here as needed
                return HttpResponseRedirect(reverse('success_page'))
    else:
        template_name=message_template_name
        context.update({'message':message})
    return render_to_response(template_name, context,
                              context_instance=RequestContext(request))

Create a message template to display the errors.

templates/message.html
{% extends "base.html" %}

{% block title %}{{ block.super }}{{ message.title }}{% endblock title %} 

{% block content %} 
<div>
  <div>{{ message.title }}</div>
  {% for text in message.text %}
  <div>{{ text }}</div>
  {% endfor %}
  <div><a href="{{ message.url }}">Back</a></div>
</div>
{% endblock content %} 

With some modifications, this method can be used with ajax requests.  Note that I do not recommend using the system for catching Django form errors.  Django handles that nicely as it is.

There are probably better approaches for this sort of thing, but this approach seems to work for me in many cases.  Hope it's useful.  Comments welcome.
Joe

Wednesday, March 30, 2011

Django Pattern For Model Permissions

I think I've found an interesting way to set up a permissions system in Django.  This system allows you to specify granular permissions on specific object instances and is template friendly.

Setup The Permissions Infrastructure

First, you create an "abstract" Permissions class that models will use as a permissions base class.  This base permissions class sets up simple caching for the permissions and creates some common methods to be used by all permissions objects.  The current_user represents the logged in that you want to check permissions for a particular object.

Notice the use of threadlocals here.  If current_user is not supplied in the constructor, the class will try to lookup the current_user from a variable by the threadlocals middleware (see below).  Be warned, however, some people don't seem to like the threadlocals approach.  To be honest, I'm still trying to evaluate the pluses and minuses, but it sure makes certain things in the templates easier.  Comments are welcome.

core/permissions.py

from core.middleware import threadlocals                        
                                                                       
class Permissions(object):

    def __init__(self, obj, current_user=False):
        if not current_user:
            current_user=threadlocals.get_current_user()
        if not current_user:
            current_user = None
        self.current_user = current_user
        self.obj = obj

        self.cache = {}
            
    def clear_cache(self, key=None):
        if key:
            try:
               del self.cache[key]
            except KeyError:
               pass
        else:
            self.cache.clear()

    def get_current_user(self):
        return self.current_user

    def __unicode__(self):
        return "Permissions for: %s" % (self.obj)

The thredlocals middleware stores the current user in the local thread.   You need to include this middleware in my settings.py middleware settings.

core/middleware/threadlocals.py - Reference

import threading

_thread_locals = threading.local()

def get_current_user():
    return getattr(_thread_locals, 'user', None)

class ThreadLocals(object):
    """Middleware that gets various objects from the
    request object and saves them in thread local storage."""
    def process_request(self, request):
        _thread_locals.user = getattr(request, 'user', None)

Now you can create a Permission subclass within my model class.  This Permissions model inherits the abstract Permissions class that we defined earlier.   Now, you can create simple permissions methods within my Permissions child class.  For example, I've created is_creator() and can_edit() methods in this class.

Finally, inside of MyModel class, you create a permissions() method that instantiates a permissions instance and returns that instance.

myapp/models.py

from core import permissions 

class MyModel(models.Model):

    creator = models.ForeignKey('auth.User')
    field1 = models...
    field2 = models...  # define Django Model fields here 

    class Permissions(permissions.Permissions):

        def can_edit(self):
            KEY='cache_can_edit'
            if self.cache.has_key(KEY):
                return self.cache[KEY]

            if self.is_creator():
                return_value = True
            else:
                return_value = False

            self.cache.update({KEY:return_value})
            return return_value


        def is_creator(self):
            KEY="cache_is_creator"
            if self.cache.has_key(KEY):
                return self.cache[KEY]
            if self.obj.creator == self.current_user:
                return_value = True
            else:
                return_value = False
            self.cache.update({KEY:return_value})
            return return_value


    def permissions(self, current_user=None):
         return self.Permissions(self, current_user)


Using The Permissions

You can use the permissions system in a view as exemplified by the following code.  Because the abstract Permissions class contains some basic caching, multiple calls to the same permissions function will not hit the database multiple times.

myapp/views.py

from myapp import models.py

@login_required
def index(request, .... ):
    current_user = request.user 
    
    mymodel = MyModel.objects.get_mymodel_instance(.....)  

    mymodel_permissions = mymodel.permissions(current_user=current_user)

    if mymodel_permissions.can_edit():
        do_something_special()

    ....


Now what's really cool is how you can make use of this in templates.  Because of the threadlocals, you simply can make calls the my Permission instance methods directly in the template without using a templatetag to pass in the current user -- because of the framework that we setup above, the current user will be assumed automatically.   Using the "with" statement, you can instantiate the permissions object only once so you can take advantage of the caching in the template.

templates/myapp/index.html
{% with mymodel.permissions as permissions %}
   <div>{% if permissions.can_edit %}<a href="">edit</a>{% endif %}</div>
{% endwith %}

You can still unittest this pattern easily by creating your tests similar to the way you create your views; you can simply pass in the current_user to the mymodel.permissions() function.

I hope this pattern is useful to someone.  I'm sure there are better approaches.  I'm glad to hear your comments.
Thanks!
Joe

Tuesday, March 22, 2011

Django Environment Quick Setup

aptitude install python-virtualenv  # for debian-based systems
virtualenv --no-site-packages myproject
cd myproject/
source ./bin/activate
pip install django  south  django-extensions
pip install flup  psycopg2
mkdir -p ./etc/ ./var/log/ ./app/django/
cd ./app/django/
django-admin.py startproject mymain

Django Project Setup Conveniences

Whenever I start a new Django project, there are several common steps that I to help standardize my setups.

In settings.py...

I define a PROJECT_ROOT variable which contains the path to the project directory. 

This is useful for various settings in the setup.py:
import sys, os
PROJECT_ROOT = os.path.dirname(__file__)

I set my MEDIA_ROOT to be a path relative to the PROJECT_ROOT.  
Of course, I will need to create this directory on the filesystem. 
MEDIA_ROOT = os.path.join(PROJECT_ROOT, '..','htdocs','media').replace('\\','/') + '/'

And I set a relative TEMPLATE_DIRS path.  I will need to make this directory as well.
TEMPLATE_DIRS = (
    os.path.join(os.path.dirname(__file__), 'templates').replace('\\','/'),
)

Sometimes I add extra directories to the path so I can store specific packages within the project directory. (I might use this if I want to keep all related modules together).
sys.path.insert(0, os.path.join(PROJECT_DIR, "contrib"))
sys.path.insert(0, os.path.join(PROJECT_DIR, "src"))

I generally like to put admin media in the following location:
ADMIN_MEDIA_PREFIX = '/media/admin/'

Almost all the time, I install the following modules in INSTALLED_APPS:
'south', # django-south
'django_extensions', # django-command-extensions

In urls.py...


If I want to use the Django debug server, I'll set this near the top:
from django.conf import settings
urlpatterns = patterns('',)
if settings.DEBUG:
    urlpatterns = patterns('',
        (r'^media/(?P<path>.*)$', 'django.views.static.serve',
                   {'document_root': settings.MEDIA_ROOT}),
    )


Introduction

This blog will document my adventures in Django, the Python web framework.  It is a direct fork of my Linux Info blog, but will be specifically utilized for documenting my Django activity.  Though my aim is to post Django code and examples that are helpful, there will most certainly be better/different ways to approach various challenges.  Any comments are much appreciated.
Joe