Monday, March 29, 2010

Roll Your Own Template Tags

Django URLs and Regular Expressions

So I'm at this point where I need to start figuring out how admins and users interact with activity and commitment objects. These two objects are things that users can participate in for points in the competition. The main difference between the two is that a user needs to request the points for an activity for an admin, while points are awarded for commitments after some period of time (like a week). Since these two objects (actually 3, but I haven't gotten around to the third item yet) are fairly similar, I use the item type as a parameter to some of my Django URLs.

from django.conf.urls.defaults import *

urlpatterns = patterns('',
'activities.views.add_participation', name='add_participation'),
'activities.views.remove_participation', name='remove_participation'),
'activities.views.request_points', name='request_points'),

So my plan has been to have one method that takes the item type as a parameter and then decide what to do after that. The use of regular expressions in the URLs is something I've found to be very powerful in Django and I think it helps keep my code DRY.

Creating Template Tags

I found the need to do similar things with my templates. Commitments and activities have add/remove links, but only activities have the request points link (In the url patterns above, commitments will work. That will probably be removed for the next release). So what I would like is a simple template that creates the appropriate links. However, I didn't see a way to include a parameter to an include tag. After doing a bit of digging, I figured that the only way to accomplish what I want was to create my own template tags.

from django import template
from django.core.exceptions import ObjectDoesNotExist
from django.contrib.auth.models import User

from activities.models import Activity, Commitment, ActivityMember, CommitmentMember

register = template.Library()

def render_user_tools(user, item):
"""Renders the form used to add/remove activities and to request points."""

if not isinstance(user, User):
user = User.objects.get(username=user)
except User.DoesNotExist():
return ""

if isinstance(item, Commitment):
return __generate_commitment_form(user, item)
elif isinstance(item, Activity):
return __generate_activity_form(user, item)
return "";


# Private methods for constructing the form.
def __generate_commitment_form(user, item):
# Check that the user is involved with this item.
return_string = ""
# Exception thrown if user cannot be found.
item_join = CommitmentMember.objects.get(user=user, commitment=item)

return_string += '<form action="/activities/remove_{0}/{}'
return_string += '/" method="post" style="display:inline"><a href="#"'
return_string += 'onclick="parentNode.submit()">Remove</a></form>'

except ObjectDoesNotExist:
return_string += '<form action="/activities/add_{0}/{}'
return_string += '/" method="post" style="display:inline">'
return_string += '<a href="#" onclick="parentNode.submit()">Add</a></form>'

# return_string is a format string with places to insert the item type and item.
return return_string.format("commitment", item)

def __generate_activity_form(user, item):
# Check that the user is involved with this item.
return_string = ""
# Exception thrown if user cannot be found.
item_join = ActivityMember.objects.get(user=user, activity=item)
if item_join.approval_status == u"unapproved":
return_string += '<form action="/activities/request_{0}_points/{}'
return_string += '/" method="post" style="display:inline"><a href="#"'
return_string += 'onclick="parentNode.submit()">Request Points</a></form>&nbsp'
elif item_join.approval_status == u"pending":
return_string += "<span class=\"pending_activity\">Pending approval</span>&nbsp"

return_string += '<form action="/activities/remove_{0}/{}'
return_string += '/" method="post" style="display:inline"><a href="#"'
return_string += 'onclick="parentNode.submit()">Remove</a></form>'

except ObjectDoesNotExist:
return_string += '<form action="/activities/add_{0}/{}'
return_string += '/" method="post" style="display:inline">'
return_string += '<a href="#" onclick="parentNode.submit()">Add</a></form>'

# return_string is a format string with places to insert the item type and item.
return return_string.format("activity", item)

It's yet another interesting feature of Django that will also prevent me from duplicating code (although the strings in the two private methods are fairly similar).

Monday, March 22, 2010


So one of the goals for my thesis is to get a working prototype of the Kukui Cup system ready in time for June. In order to do this, it was suggested that I switch to milestones every two weeks instead of every month (4 weeks). So, this week would mark a "half-milestone", which is fine except for that it's spring break! Fortunately, as a grad student, I don't really have much of a life anyway. Which means that milestone 2.5 is ready and up on GitHub.

Extending Models and Django

The specification I've been following recommends that there be two types of activities that users can commit to: events and regular activities. Events are basically activities with an additional "event date" field, which is required and indicates the date and time the event occurs. So, my initial idea was to have a base Activity model and another Event model that extends Activity and adds the additional required field.

class Activity(CommonActivity):
    ('text', 'Text'),
    ('image', 'Image Upload'),
  confirm_code = models.CharField(blank=True, max_length=20)
  pub_date = models.DateField(
  expire_date = models.DateField()
  users = models.ManyToManyField(User, through="ActivityMember")
  confirm_type = models.CharField(max_length=20, choices=CONFIRM_CHOICES)

class Event(Activity):
  event_date = models.DateTimeField(null=True, blank=True)
However, adding this to the admin interface is a little tricky. Ideally, there would be one form to create activities or events by providing a drop-down with the options. Which is possible in the admin interface, but would require a little extra work to hack (especially if I want some fancy Javascript to hide/unhide the event_date field). Which is fine, cause I'm up for a little admin template hacking and I'll probably need to do it eventually. The other issue is that, in Django 1.0.4, events will show up in activities (which we want), but the template doesn't know if an activity is an event or not.

The simple solution then was to just create one activity with a boolean field "is_event" and the "event_date" field. Then, I created a custom admin form validator that checks if "is_event" is true. If it is true, then check if the "event_date" field is filled in. If it isn't, throw an error.

class ActivityAdminForm(ModelForm):
  class Meta:
    model = Activity
  def clean(self):
    """Checks that an event has an event date."""
    cleaned_data = self.cleaned_data
    is_event = cleaned_data.get("is_event")
    event_date = cleaned_data.get("event_date")
    has_date = cleaned_data.has_key("event_date") #Check if this is in the data dict.
    if is_event and has_date and not event_date:
      self._errors["event_date"] = ErrorList([u"Events require an event date."])
      del cleaned_data["is_event"]
      del cleaned_data["event_date"]
    return cleaned_data
And the template code is easy. We just check if the activity has the "is_event" field set and then print out the event date. The small catch is that regular activities can now set an event date, which isn't important to fix right now. Eventually, I'd like to do some admin hacking so that non-events have the "event_date" field hidden.

I'd also like to add that I found this awesome syntax highlighter at easy syntax highlighting for blogger.

Monday, March 15, 2010

Git-ing it Done

A lot of things got done since the last time I blogged. I probably should've blogged about them as they happened, but instead I think I'll roll them all up into this one.

Using Git and GitHub
I've been using Subversion for a long time. When I took ICS 413 in fall of 07, we used Subversion and Google Code to host our assignments and projects. When I had my research assistantship in LILT, we used Subversion to host all of our Ruby code. So I have more than a passing familiarity with Subversion, although I never really got into the ins and outs of branching and merging.

But it's hard to get away from the Git hype. I spoke to a colleague (the former LILT release engineer) and he was very hyped about Git and how Heroku uses it for deploying Rails projects. We briefly covered Git in our software engineering class last fall, but we still used Subversion. So I jumped at the opportunity to get familiar with it more, and I'm enjoying it. I created a branch called "release" that will only hold stable versions of my code while I work off of master. I did this in part because I want other developers to start working on a stable version rather than my working branch because many things could change. Since I'm on a 2 week milestone schedule, I'll be updating the release branch fairly regularly.

The network graph of my GitHub repository.

In addition to the release branch, I'm tagging the release branch with milestone markers. It's neat how GitHub resolves these tags into downloadable packages of the source code. The very act of tagging and pushing the tag onto GitHub made a downloadable distribution available. I wish I had known that sooner so that I could use a different commit message.

Downloads for the project.

So I'm pretty psyched about using Git now too. It'll be interesting to see what happens when the other collaborators come on. I'm encouraging them to work on a separate branch so that they don't get all of the small changes that I push onto GitHub. The database models can change and that would require the other collaborators to blow away their database and rebuild it (since Django doesn't have migrations without the south app). This way, I can merge in my stable versions and have collaborators work off of those.

Selenium was one of the most awesome testing tools I've used. We use it on a Rails project that I work on. However, integrating it with Django seems to be tricky. There are people working on it, but there doesn't appear to be a perfect solution. However, people have also mentioned another testing framework called Windmill that has better Django integration. So I thought I'd check it out and see if I can get it working with our project.

Running a Windmill test.

The results are mostly satisfying. I created a basic test using the IDE that browses through the tabs and checks a few items. I copied the resulting Python code and added it to my project. After writing a short script to check if Windmill is installed and to prepare it for Django, I have it running as part of my unit tests. The only downer is that it seems to throw errors on our continuous integration server. It's something that I should probably fix because integrating Windmill with Hudson would be an important step as far as making sure everything's working properly.
# Generated by the windmill services transformer
from windmill.authoring import WindmillTestClient

def test_check_tabs():
client = WindmillTestClient(__name__)'Resources')
client.waits.forElement(link=u'Energy Hub', timeout=u'8000')'Energy Hub')
client.waits.forElement(link=u'Billboard', timeout=u'8000')'Billboard')
client.waits.forElement(link=u'About', timeout=u'8000')'About')
client.asserts.assertText(xpath=u"//div[@id='content']/div[2]/div/h3", validator=u'About the Kukui Cup Competition')
The actual Python script that Windmill is running.

Feeling Pythonic?
Most of the work I've done thus far is creating content. I haven't done any real Python programming except for my testing scripts. But as I implement the features, I'm learning more and more about Django and Python. I still think of myself as a beginner, but I'm learning a lot about the framework and the language. Over the past few days, I implemented activities and commitments on our profile page. During the process, I learned about generator expressions, properties, and how the model filter/exclude methods work. I used abstract models since many properties of activities, commitments, and goals are similar. I even brushed up on regular expressions by creating urls that dynamically call functions. Instead of a "add_activity" and "add_commitment" url entry, I have a regular expression that's basically "add_(commitment|activity)" that calls a view method called "add_membership". "add_membership" takes the object type (commitment or activity) as a parameter and calls the necessary method.

Current user profile with commitments and activities.

I'm enjoying working on the project more and more. I'm excited to see what I'll learn next.

Monday, March 8, 2010

Making It Easy For Developers To Extend My Code

Last semester, our ICS 613 professor (and my advisor) Philip Johnson started off with what he calls the three prime directives of open source software engineering. We want our dorm energy competition implementation to be open source and available for anyone to install themselves. Unfortunately, it hasn't been well documented and the source code was unavailable.

Over the past week, I made some efforts to satisfy the 3 prime directives. First, the source code is now available as a public project on GitHub. This means that anyone can download the source code and either develop or install the application. There's little in the way of functionality right now, but it is available when other developers start working on it (which may happen this week). This is the first time I've really used Git (other than to clone other projects), so I'm interested in seeing how it works once the other developers set it up.

The kukui-cup-pinax project has also been added to Hudson, our continuous integration server. To do this required a few adjustments. First, I needed to set up Pinax on our server, which required installing the Apple Developer Tools (for gcc) and the Python Imaging Library. Once that was set up, I installed Pinax on the server. Next, since Hudson was set up for Subversion, I needed to install Git on the server and then install the Hudson Git plugin. After all of that, I was able to have Hudson check out the project.

Now I just needed to run the tests. I wrote a shell script to initialize the Pinax environment, copy the file, and run the tests. However, there were several tests that failed out of the box. I looked around online and it seems that in this version of Pinax, there are a few tests that are known to fail. Also, since I was using a different authentication backend (django_cas) and an earlier version of Django, some of Django's tests were failing too. I decided to set up a Python script that ran only the tests that I created in my apps directory. That way, the continuous integration kind of works and I don't get hammered by build failed emails.

The upside of this entire process is that it helped me write the README file. Since new users are also going to have to install prerequisites, walking through the process again reminded me of all the little things that I had to do. That piece of documentation is the other important step I needed to do to hopefully the 3 prime directives. When the new users come on, I'll have to see how this process goes.

Programming-wise, I haven't made a whole lot of progress. But I think the application is now in a presentable state and ready for other developers to start working.

Monday, March 1, 2010

Data Loss Is Bad

Admittedly, I did not get much done this week. Was shooting for getting a lot of work done this past weekend, but I caught a pretty bad cold and slept for the most part. Too bad, because it seems like more people might be switching over to my Pinax implementation. Guess that means I better get started on that documentation as well.

That's not to say I did nothing. Before my meeting with Philip last weekend, I had done something pretty stupid. I deleted my database and resynced it. Of course, it's only a development database, so nothing important was lost. However, since most of the content on the site is backed by the database, I lost all of it and had to retype it in before the meeting. To prevent this from happening again, I thought I'd create a sqlite dump so that I can rebuild it. However, I realized that I didn't need the create table statements (since they'll be made when I resync the database). Also, some of the Django tables already have data in them, which caused sqlite to complain about having duplicate entries in some tables. Not the way to go.

In Rails, we filled our development database with fixtures, which are some sample content that is stored in a few files. So I looked for a way to dump my data into fixture files and then have them load up in Django. Using " dumpdata", I dumped only the tables that contained the content in the site. Then, I removed the database and resynced it. Then, using " loaddata", I restored the content. Unfortunately, I had to change a few foreign keys, but it seems to be okay now. So my little dumb mistake lead me to this discovery, which will be important when other programmers start working on it.

My meeting with Philip was fairly quick. It was then that it was decided that this will probably be the main implementation of the dorm energy competition. I started doing some research on integrating Python applications with Hudson, our continuous integration server. I found a blog post entitled Django and Hudson and Bears, Oh My! (and twill, and figleaf), which is a great walkthrough of how to integrate Django and Hudson. I also need to investigate using Django/Pinax, Hudson, and Selenium together to do automated browser testing. A lot to do in a week, so we'll see how it goes.