Monday, May 3, 2010

Deployment Newbie

I've been trying to get our Pinax-based web application on our web server for the past few days. As a Master's student, I have had experience working with web frameworks but I've never had to deploy them. So dealing with the httpd.conf, .htaccess, and various permissions errors tripped me up big time. Thankfully, I've had the benefit of both Google and colleagues. Hopefully, other newbie deployers will find my foray into Apache and mod_wsgi useful.

Permission Denied: /Users/username/.htaccess pcfg_openfile?

So I got this error. A lot. And I couldn't quite figure out why. Everything I read from Google lead me to think that I had permissions errors somewhere in directory. But I changed everything to 755 and it still didn't seem to work. So I asked Robert, a colleague, and he asked me to check my home directory. The permissions on my home directory were 700. Robert mentioned that since Apache needs to go in, it needs access to the files. So once I 'chmodded' those, things started to work.

I had a few errors after that, but they were mostly related to permissions on the project file (they should all be 755).

Setting Up Virtual Hosts:

This was a real newbie mistake. Most of the tutorials I've seen online for Pinax deployment don't mention virtual hosts. And if they do, the code sample is basically about what to put in the VirtualHost definition. Of course, being a newbie, I thought "Hey, I can change this to be any port that I want and it'll just work!" After looking through Apache's examples, I noticed that I was missing Listen and NameVirtualHost statements. So, the actual configuration would look something like (assuming you're using mod_wsgi):



Just put this in a configuration file, edit the paths, and point to it from the httpd.conf with an Include statement and things should work! I've got to admit that it could've been a lot more painful if Pinax did not provide the WSGI script.

But after all this, I got it to work, at least on my Mac through Apache. Now to put it on our web server.

Monday, April 26, 2010

Javascript Hell

A Crash Course in jQuery

When I first got in to web development with Rails, Prototype and Scriptaculous were the most popular Javascript libraries. Since I was relatively new to Javascript at the time, they were a little tricky. After working with Prototype and Scriptaculous for some time, I think I have a pretty good handle on it. But, this new fancy Javascript library called jQuery came up and now everyone's using it. By default, version 1.3.2 comes with Pinax 0.7.1. So, in order to get with the times (and debug Pinax's Javascript), I went through a crash course in jQuery.

Following some tutorials, I implemented a toggle for my news articles. Even in this short script, I can see how jQuery differs from Prototype and Scriptaculous. I like how easy it is to implement the toggle without having to assign unique div ids for each article.


//News toggle
$(document).ready(function(){
//Switch the "Open" and "Close" state per click
$("h4.trigger").toggle(function(){
$(this).addClass("active");
}, function () {
$(this).removeClass("active");
});

//Hide and unhide on click.
$("h4.trigger").click(function(){
$(this).next(".article_container").toggle();
});
});
"Hacking" Google Gadgets

The WattDepot Google Gadgets group have done some great work in implementing visualizations for gadgets to be added in iGoogle. I wanted to add the Gauge visualization and the BioHeatMap visualization to my web application. However, I know very little about how visualizations work. Plus, the group created gadgets, so they are not embeddable on web pages right away. But they're only HTML and Javascript, and I have the source code. So why don't I just extract out the Javascript and add it to my web application?

For the most part, it worked out quite well. I added the gauge visualization connected to a WattDepot data source fairly easily. I hard coded some values since they are user preferences (eventually, I'd like to change that) and changed some variables around. Is it just me or are there automagically created Javascript variables based on the ids in the HTML section of the gadget?

However, the BioHeatMap visualization was a little tricky. When I added it, I got some weird errors. It was telling me things like "$(document).ready is not a function". It broke my previously working jQuery code somehow! After a little bit of debugging, it turns out that the BioHeatMap visualization actually requires my old friend Prototype. And Prototype also uses the "$" shortcut. Fortunately, someone at jQuery thought of this and created a function called "noConflict()" which relinquishes the usage of the $ symbol. The jQuery code above would then have to be rewritten (replace the "$" with "jQuery"), but it was a minor issue.

But after those things were resolved, everything ran smoothly! I been to a little Javascript hell and survived to blog about it.

Monday, April 19, 2010

Working Git Out

I have been using Subversion for over 3 years now. Ever since I learned it, I've been with organizations that use it as their primary version control system. When I went to RailsConf 2008 in Portland, Oregon, people were just starting to use this thing called Git. I didn't understand it at the time and the hype didn't really catch up to me until the beginning of the year. It was then I decided to put my project on GitHub instead of Google Code; my default choice.

For the most part, I used Git like I used Subversion. I checked out the master branch, made my changes on that branch, and committed them periodically. I thought Git was okay, except for that I needed that extra "git push" to commit to the remote repository.

Then, I started to branch out (excuse the pun). I created a branch to hold my stable releases and made that the default branch. It was easy for me to merge changes between the two branches (sometimes I changed things in release like documentation changes). I even did a cherry-pick to get something I fixed in master to release. And when other users came on, it seemed only natural that I have them work off of another branch that is separate from my changes. Merging between them has never been an issue.

Recently, I found an article by Andy Croll that outlined his typically daily workflow using Git. Since it is so easy to create local branches, why not create a branch for each individual feature? I had never thought of using Git that way and it made total sense, especially if you have a few features going on at the same time. When using Subversion, I found myself frequently picking files to not commit since I was still working on them. By using this Git workflow, I can easily merge completed features into the master branch and push that when its ready.

I have to say that this article opened my eyes to the possibilities of using a Distributed Version Control System like Mercurial or Git. While the workflow does not work as well for the small features I've been implementing, I can see it being great for when I'm working on one huge feature while submitting bug fixes.

Monday, April 12, 2010

It's the Small Things

I guess at this point, you'd assume that I'm an expert at hacking Django forms. After all, I already wrote two blog posts about my custom validation and saving. But this post is all about the small things I've tried to do. Some with success, and some without.

Fields that are Not Part of the ModelForm

One of the newer requirements in our use case was to create a place in the form for generating activity confirmation codes if the activity's confirmation type is "code". The field would take the number of codes the admin wants to generate. The easy way to do this is to simply have the number of codes be a property of the activity model. Then, the admin form can generate the field and have it on the Django admin form. However, there's no reason for the number of codes to be a part of the model. So there must be a way to present it as a non-model field.

Number of codes field in the Activity admin interface

The solution is to simply add the form as an additional IntegerField in the definition of the ModelForm. Then, when the form is being validated or saved, it is one of the parameters. Then I have a static method in my confirmation code model that generates a number of codes for a given activity. Great, so now I can generate a number of codes for an activity. Where should I put a link to view the codes?

I also started inserting help text into the model fields so that they're presented on the form. However, for an activity with confirmation codes, the text should change if the activity has already been created. So this presents an interesting opportunity. If I can change the help text so that it says "Number of additional codes to generate" instead of "Number of confirmation codes to generate", then I can insert a link to view codes.

Well, then we just override the init method (the method called to initialize an object) to edit the num_codes field. There, if the activity has been created (i.e. has a created_at field), then we can change the help text and insert a link to our view codes view.


class ActivityAdminForm(forms.ModelForm):
num_codes = forms.IntegerField(required=False,
label="Number of codes",
help_text="Number of confirmation codes to generate",
initial=0
)

def __init__(self, *args, **kwargs):
"""Override to change number of codes help text if we are editing an activity."""

super(ActivityAdminForm, self).__init__(*args, **kwargs)
# Instance points to an instance of the model.
if self.instance and self.instance.created_at and self.instance.confirm_type == "code":
self.fields["num_codes"].help_text = "Number of additional codes to generate <a href=\""
self.fields["num_codes"].help_text += reverse("activities.views.view_codes", args=(self.instance.pk,))
self.fields["num_codes"].help_text += "\" target=\"_blank\">View codes</a>"
Read-Only Fields

I like that the Django admin presents an interface for editing the different fields of a Django model. However, there are a few fields that need to be displayed but not edited. A good example is the ActivityMember model, which is a model that represents a user's participation in an activity. These are the models that are approved/rejected when a user requests points for their participation. There is no reason for admins to be able to change the activity or comments from the user. However, they need to be displayed so that admins have some context.

As it turns out, Django has a field for ModelForms that can specify fields that should be read-only. However, this feature is only enabled for the development version of Django (version 1.2). The version of Pinax I am using is 0.7.1 and it only has Django 1.0.4. A friend of mine is trying to convince me to update to the dev version of Pinax (0.9), which uses Django 1.2. I think I'll hold off updating until the summer rolls around. I'm hoping that both are stable enough by then so that I can use it in production in October. I also found this interesting post on StackOverflow that deals with this issue. Perhaps if I have time, I can add it in.

Check out the current implementation at GitHub.

Monday, April 5, 2010

More Django Form Hacking

Inline Formsets:

The activities in the Kukui Cup competition are fairly complex objects. They're complicated enough that the basic Django form validations will not work without a fair bit of tweaking. Some of the fields in an activity are either optional or required depending on the values of other fields. Here's a few things we need outside of the basic form processing:
  1. If an activity is an event (is_event = True), then it must have an event date.
  2. If the confirmation type is either "confirmation code" or "image upload", then a prompt is required. Examples would be "Enter the confirmation code you received at the event" or "Upload a photo of yourself holding a CFL and an incandescent light bulb".
  3. If the confirmation type is "question and answer", then at least one question and answer is required.
  4. Publication date must be before the expiration date.
1, 2, and 4 are pretty straightforward, especially since I had already taken care of 1. Here's the new activity admin form.


class ActivityAdminForm(ModelForm):
class Meta:
model = Activity

def clean(self):
# Data that has passed validation.
cleaned_data = self.cleaned_data

#1 Check that an event has an event date.
is_event = cleaned_data.get("is_event")
event_date = cleaned_data.get("event_date")
has_date = cleaned_data.has_key("event_date") #Check if this is in the data dict.

if is_event and has_date and not event_date:
self._errors["event_date"] = ErrorList([u"Events require an event date."])
del cleaned_data["is_event"]
del cleaned_data["event_date"]

#2 Check the verification type.
confirm_type = cleaned_data.get("confirm_type")
prompt = cleaned_data.get("confirm_prompt")
if confirm_type != "text" and len(prompt) == 0:
self._errors["confirm_prompt"] = ErrorList([u"This confirmation type requires a confirmation prompt."])
del cleaned_data["confirm_type"]
del cleaned_data["confirm_prompt"]

#4 Publication date must be before the expiration date.
if cleaned_data.has_key("pub_date") and cleaned_data.has_key("expire_date"):
pub_date = cleaned_data.get("pub_date")
expire_date = cleaned_data.get("expire_date")

if pub_date >= expire_date:
self._errors["expire_date"] = ErrorList([u"The expiration date must be after the pub date."])
del cleaned_data["expire_date"]

return cleaned_data
Number 3 is a little tricky, because there can be one or many question and answer pairs. I was already aware of inline forms from the Django tutorial. So the first step was to add in inline forms for the questions and answers.

Question and answer fields in the admin form.

Easy enough, but now I have to implement the validation. These questions and answers should only be provided if the confirm type is "text". I did some research and figured out that I needed to extend the BaseInlineFormSet class to provide my custom validation behavior.

class TextQuestionInlineFormSet(BaseInlineFormSet):
"""Custom formset model to override validation."""

def clean(self):
"""Validates the form data and checks if the activity confirmation type is text."""

# Form that represents the activity.
activity_form = self.instance

# Count the number of questions.
count = 0
for form in self.forms:
try:
if form.cleaned_data:
count += 1
except AttributeError:
pass

if activity_form.confirm_type == "text" and count == 0:
raise ValidationError("At least one question is required if the activity's confirmation type is text.")

elif activity_form.confirm_type != "text" and count > 0:
raise ValidationError("Questions are not required for this confirmation type.")

class TextQuestionInline(admin.StackedInline):
model = TextPromptQuestion
extra = 3
formset = TextQuestionInlineFormSet
I chose to raise the validation error if the confirm type is not text because I don't want any question and answers saved for activities that do not need it. This should take care of most of the activity admin interface, but the requirements can always change.

Themes:

There are other students working on my Kukui Cup implementation. Their goal is to redesign the interface through the use of HTML and CSS. We don't want just one redesign; we want them to attempt many redesigns so that we can evaluate each one. So, instead of having them hack the settings.py file to change themes, I added a drop down form at the top so that they can change the CSS files easily. This also required moving the original files into folders to create a "default" theme. I also created custom template tags so that any file that ends with ".css" is imported in the header.


def render_css_import():
"""Renders the CSS import header statements for a template."""

return_string = ""
css_dir = os.path.join(settings.PROJECT_ROOT, "media", settings.KUKUI_CSS_THEME, "css")
if os.path.isdir(css_dir):
items = (item for item in os.listdir(css_dir) if string.find(item, "css") >= 0)
for item in items:
return_string += "<link rel=\"stylesheet\" href=\"/site_media/static/" + settings.KUKUI_CSS_THEME
return_string += "/css/" + item + "\" />\n"

return return_string

register.simple_tag(render_css_import)
This should make it easy for the other students to see how the interface changes as they develop their own CSS.

Monday, March 29, 2010

Roll Your Own Template Tags

Django URLs and Regular Expressions

So I'm at this point where I need to start figuring out how admins and users interact with activity and commitment objects. These two objects are things that users can participate in for points in the competition. The main difference between the two is that a user needs to request the points for an activity for an admin, while points are awarded for commitments after some period of time (like a week). Since these two objects (actually 3, but I haven't gotten around to the third item yet) are fairly similar, I use the item type as a parameter to some of my Django URLs.

from django.conf.urls.defaults import *

urlpatterns = patterns('',
url(r'^add_(?P<item_type>activity|commitment)/(?P<item_id>\d+)/$',
'activities.views.add_participation', name='add_participation'),
url(r'^remove_(?P<item_type>activity|commitment)/(?P<item_id>\d+)/$',
'activities.views.remove_participation', name='remove_participation'),
url(r'^request_(?P<item_type>activity|commitment)_points/(?P<item_id>\d+)/$',
'activities.views.request_points', name='request_points'),
)

So my plan has been to have one method that takes the item type as a parameter and then decide what to do after that. The use of regular expressions in the URLs is something I've found to be very powerful in Django and I think it helps keep my code DRY.

Creating Template Tags

I found the need to do similar things with my templates. Commitments and activities have add/remove links, but only activities have the request points link (In the url patterns above, commitments will work. That will probably be removed for the next release). So what I would like is a simple template that creates the appropriate links. However, I didn't see a way to include a parameter to an include tag. After doing a bit of digging, I figured that the only way to accomplish what I want was to create my own template tags.

from django import template
from django.core.exceptions import ObjectDoesNotExist
from django.contrib.auth.models import User

from activities.models import Activity, Commitment, ActivityMember, CommitmentMember

register = template.Library()

def render_user_tools(user, item):
"""Renders the form used to add/remove activities and to request points."""

if not isinstance(user, User):
try:
user = User.objects.get(username=user)
except User.DoesNotExist():
return ""

if isinstance(item, Commitment):
return __generate_commitment_form(user, item)
elif isinstance(item, Activity):
return __generate_activity_form(user, item)
else:
return "";

register.simple_tag(render_user_tools)

# Private methods for constructing the form.
def __generate_commitment_form(user, item):
# Check that the user is involved with this item.
return_string = ""
try:
# Exception thrown if user cannot be found.
item_join = CommitmentMember.objects.get(user=user, commitment=item)

return_string += '<form action="/activities/remove_{0}/{1.id}'
return_string += '/" method="post" style="display:inline"><a href="#"'
return_string += 'onclick="parentNode.submit()">Remove</a></form>'

except ObjectDoesNotExist:
return_string += '<form action="/activities/add_{0}/{1.id}'
return_string += '/" method="post" style="display:inline">'
return_string += '<a href="#" onclick="parentNode.submit()">Add</a></form>'

# return_string is a format string with places to insert the item type and item.
return return_string.format("commitment", item)

def __generate_activity_form(user, item):
# Check that the user is involved with this item.
return_string = ""
try:
# Exception thrown if user cannot be found.
item_join = ActivityMember.objects.get(user=user, activity=item)
if item_join.approval_status == u"unapproved":
return_string += '<form action="/activities/request_{0}_points/{1.id}'
return_string += '/" method="post" style="display:inline"><a href="#"'
return_string += 'onclick="parentNode.submit()">Request Points</a></form>&nbsp'
elif item_join.approval_status == u"pending":
return_string += "<span class=\"pending_activity\">Pending approval</span>&nbsp"

return_string += '<form action="/activities/remove_{0}/{1.id}'
return_string += '/" method="post" style="display:inline"><a href="#"'
return_string += 'onclick="parentNode.submit()">Remove</a></form>'

except ObjectDoesNotExist:
return_string += '<form action="/activities/add_{0}/{1.id}'
return_string += '/" method="post" style="display:inline">'
return_string += '<a href="#" onclick="parentNode.submit()">Add</a></form>'

# return_string is a format string with places to insert the item type and item.
return return_string.format("activity", item)

It's yet another interesting feature of Django that will also prevent me from duplicating code (although the strings in the two private methods are fairly similar).

Monday, March 22, 2010

Half-Milestones

So one of the goals for my thesis is to get a working prototype of the Kukui Cup system ready in time for June. In order to do this, it was suggested that I switch to milestones every two weeks instead of every month (4 weeks). So, this week would mark a "half-milestone", which is fine except for that it's spring break! Fortunately, as a grad student, I don't really have much of a life anyway. Which means that milestone 2.5 is ready and up on GitHub.

Extending Models and Django

The specification I've been following recommends that there be two types of activities that users can commit to: events and regular activities. Events are basically activities with an additional "event date" field, which is required and indicates the date and time the event occurs. So, my initial idea was to have a base Activity model and another Event model that extends Activity and adds the additional required field.


class Activity(CommonActivity):
  CONFIRM_CHOICES = (
    ('text', 'Text'),
    ('image', 'Image Upload'),
  )
  
  confirm_code = models.CharField(blank=True, max_length=20)
  pub_date = models.DateField(default=datetime.date.today())
  expire_date = models.DateField()
  users = models.ManyToManyField(User, through="ActivityMember")
  confirm_type = models.CharField(max_length=20, choices=CONFIRM_CHOICES)

class Event(Activity):
  event_date = models.DateTimeField(null=True, blank=True)
However, adding this to the admin interface is a little tricky. Ideally, there would be one form to create activities or events by providing a drop-down with the options. Which is possible in the admin interface, but would require a little extra work to hack (especially if I want some fancy Javascript to hide/unhide the event_date field). Which is fine, cause I'm up for a little admin template hacking and I'll probably need to do it eventually. The other issue is that, in Django 1.0.4, events will show up in activities (which we want), but the template doesn't know if an activity is an event or not.

The simple solution then was to just create one activity with a boolean field "is_event" and the "event_date" field. Then, I created a custom admin form validator that checks if "is_event" is true. If it is true, then check if the "event_date" field is filled in. If it isn't, throw an error.

class ActivityAdminForm(ModelForm):
  class Meta:
    model = Activity
    
  def clean(self):
    """Checks that an event has an event date."""
    cleaned_data = self.cleaned_data
    is_event = cleaned_data.get("is_event")
    event_date = cleaned_data.get("event_date")
    has_date = cleaned_data.has_key("event_date") #Check if this is in the data dict.
    
    if is_event and has_date and not event_date:
      self._errors["event_date"] = ErrorList([u"Events require an event date."])
      del cleaned_data["is_event"]
      del cleaned_data["event_date"]
      
    return cleaned_data
And the template code is easy. We just check if the activity has the "is_event" field set and then print out the event date. The small catch is that regular activities can now set an event date, which isn't important to fix right now. Eventually, I'd like to do some admin hacking so that non-events have the "event_date" field hidden.

I'd also like to add that I found this awesome syntax highlighter at easy syntax highlighting for blogger.

Monday, March 15, 2010

Git-ing it Done

A lot of things got done since the last time I blogged. I probably should've blogged about them as they happened, but instead I think I'll roll them all up into this one.

Using Git and GitHub
I've been using Subversion for a long time. When I took ICS 413 in fall of 07, we used Subversion and Google Code to host our assignments and projects. When I had my research assistantship in LILT, we used Subversion to host all of our Ruby code. So I have more than a passing familiarity with Subversion, although I never really got into the ins and outs of branching and merging.

But it's hard to get away from the Git hype. I spoke to a colleague (the former LILT release engineer) and he was very hyped about Git and how Heroku uses it for deploying Rails projects. We briefly covered Git in our software engineering class last fall, but we still used Subversion. So I jumped at the opportunity to get familiar with it more, and I'm enjoying it. I created a branch called "release" that will only hold stable versions of my code while I work off of master. I did this in part because I want other developers to start working on a stable version rather than my working branch because many things could change. Since I'm on a 2 week milestone schedule, I'll be updating the release branch fairly regularly.

The network graph of my GitHub repository.

In addition to the release branch, I'm tagging the release branch with milestone markers. It's neat how GitHub resolves these tags into downloadable packages of the source code. The very act of tagging and pushing the tag onto GitHub made a downloadable distribution available. I wish I had known that sooner so that I could use a different commit message.

Downloads for the project.

So I'm pretty psyched about using Git now too. It'll be interesting to see what happens when the other collaborators come on. I'm encouraging them to work on a separate branch so that they don't get all of the small changes that I push onto GitHub. The database models can change and that would require the other collaborators to blow away their database and rebuild it (since Django doesn't have migrations without the south app). This way, I can merge in my stable versions and have collaborators work off of those.

Windmill
Selenium was one of the most awesome testing tools I've used. We use it on a Rails project that I work on. However, integrating it with Django seems to be tricky. There are people working on it, but there doesn't appear to be a perfect solution. However, people have also mentioned another testing framework called Windmill that has better Django integration. So I thought I'd check it out and see if I can get it working with our project.

Running a Windmill test.

The results are mostly satisfying. I created a basic test using the IDE that browses through the tabs and checks a few items. I copied the resulting Python code and added it to my project. After writing a short script to check if Windmill is installed and to prepare it for Django, I have it running as part of my unit tests. The only downer is that it seems to throw errors on our continuous integration server. It's something that I should probably fix because integrating Windmill with Hudson would be an important step as far as making sure everything's working properly.
# Generated by the windmill services transformer
from windmill.authoring import WindmillTestClient

def test_check_tabs():
client = WindmillTestClient(__name__)

client.click(link=u'Resources')
client.waits.forPageLoad(timeout=u'20000')
client.waits.forElement(link=u'Energy Hub', timeout=u'8000')
client.click(link=u'Energy Hub')
client.waits.forPageLoad(timeout=u'20000')
client.waits.forElement(link=u'Billboard', timeout=u'8000')
client.click(link=u'Billboard')
client.waits.forPageLoad(timeout=u'20000')
client.waits.forElement(link=u'About', timeout=u'8000')
client.click(link=u'About')
client.waits.forPageLoad(timeout=u'20000')
client.asserts.assertText(xpath=u"//div[@id='content']/div[2]/div/h3", validator=u'About the Kukui Cup Competition')
The actual Python script that Windmill is running.

Feeling Pythonic?
Most of the work I've done thus far is creating content. I haven't done any real Python programming except for my testing scripts. But as I implement the features, I'm learning more and more about Django and Python. I still think of myself as a beginner, but I'm learning a lot about the framework and the language. Over the past few days, I implemented activities and commitments on our profile page. During the process, I learned about generator expressions, properties, and how the model filter/exclude methods work. I used abstract models since many properties of activities, commitments, and goals are similar. I even brushed up on regular expressions by creating urls that dynamically call functions. Instead of a "add_activity" and "add_commitment" url entry, I have a regular expression that's basically "add_(commitment|activity)" that calls a view method called "add_membership". "add_membership" takes the object type (commitment or activity) as a parameter and calls the necessary method.


Current user profile with commitments and activities.

I'm enjoying working on the project more and more. I'm excited to see what I'll learn next.

Monday, March 8, 2010

Making It Easy For Developers To Extend My Code

Last semester, our ICS 613 professor (and my advisor) Philip Johnson started off with what he calls the three prime directives of open source software engineering. We want our dorm energy competition implementation to be open source and available for anyone to install themselves. Unfortunately, it hasn't been well documented and the source code was unavailable.

Over the past week, I made some efforts to satisfy the 3 prime directives. First, the source code is now available as a public project on GitHub. This means that anyone can download the source code and either develop or install the application. There's little in the way of functionality right now, but it is available when other developers start working on it (which may happen this week). This is the first time I've really used Git (other than to clone other projects), so I'm interested in seeing how it works once the other developers set it up.

The kukui-cup-pinax project has also been added to Hudson, our continuous integration server. To do this required a few adjustments. First, I needed to set up Pinax on our server, which required installing the Apple Developer Tools (for gcc) and the Python Imaging Library. Once that was set up, I installed Pinax on the server. Next, since Hudson was set up for Subversion, I needed to install Git on the server and then install the Hudson Git plugin. After all of that, I was able to have Hudson check out the project.

Now I just needed to run the tests. I wrote a shell script to initialize the Pinax environment, copy the settings.py file, and run the tests. However, there were several tests that failed out of the box. I looked around online and it seems that in this version of Pinax, there are a few tests that are known to fail. Also, since I was using a different authentication backend (django_cas) and an earlier version of Django, some of Django's tests were failing too. I decided to set up a Python script that ran only the tests that I created in my apps directory. That way, the continuous integration kind of works and I don't get hammered by build failed emails.

The upside of this entire process is that it helped me write the README file. Since new users are also going to have to install prerequisites, walking through the process again reminded me of all the little things that I had to do. That piece of documentation is the other important step I needed to do to hopefully the 3 prime directives. When the new users come on, I'll have to see how this process goes.

Programming-wise, I haven't made a whole lot of progress. But I think the application is now in a presentable state and ready for other developers to start working.

Monday, March 1, 2010

Data Loss Is Bad

Admittedly, I did not get much done this week. Was shooting for getting a lot of work done this past weekend, but I caught a pretty bad cold and slept for the most part. Too bad, because it seems like more people might be switching over to my Pinax implementation. Guess that means I better get started on that documentation as well.

That's not to say I did nothing. Before my meeting with Philip last weekend, I had done something pretty stupid. I deleted my database and resynced it. Of course, it's only a development database, so nothing important was lost. However, since most of the content on the site is backed by the database, I lost all of it and had to retype it in before the meeting. To prevent this from happening again, I thought I'd create a sqlite dump so that I can rebuild it. However, I realized that I didn't need the create table statements (since they'll be made when I resync the database). Also, some of the Django tables already have data in them, which caused sqlite to complain about having duplicate entries in some tables. Not the way to go.

In Rails, we filled our development database with fixtures, which are some sample content that is stored in a few files. So I looked for a way to dump my data into fixture files and then have them load up in Django. Using "manage.py dumpdata", I dumped only the tables that contained the content in the site. Then, I removed the database and resynced it. Then, using "manage.py loaddata", I restored the content. Unfortunately, I had to change a few foreign keys, but it seems to be okay now. So my little dumb mistake lead me to this discovery, which will be important when other programmers start working on it.

My meeting with Philip was fairly quick. It was then that it was decided that this will probably be the main implementation of the dorm energy competition. I started doing some research on integrating Python applications with Hudson, our continuous integration server. I found a blog post entitled Django and Hudson and Bears, Oh My! (and twill, and figleaf), which is a great walkthrough of how to integrate Django and Hudson. I also need to investigate using Django/Pinax, Hudson, and Selenium together to do automated browser testing. A lot to do in a week, so we'll see how it goes.

Monday, February 22, 2010

UH Dorm Energy Website: Pinax Style

I decided to go with Pinax for creating the UH Dorm Energy website. Since removing Pinax apps can be kind of a pain, I started by cloning the CMS project and built up from there. And in my short experience with it, I'm beginning to think that it is the better way to go.

Pinax uses a wide collection of plugins from the Django community along with some extra tweaks. For example, the blocks of text in the CMS are managed by a Django app called django-generic-flatblocks. With this plugin, one can lay out the blocks on the page and then use the admin panel to change the content (which can be images, text, or whatever else you want). However, you don't really want to go back and forth between the admin panel and the page to see if the content is just right. So Pinax added some additional jQuery Javascript to allow an admin to edit the content within the page. Once the content is changed, the page is refreshed to show the admin the results of their change.

Editing some text within the Kukui Cup page.

I created individual pages for the different tabs and started laying out the blocks. After about an hour or two, I reproduced most of the pages in the set of mockups. There's still some work to do as some styles may need to be changed. However, for the most part, the layout is there.

Mockup of the front page in Balsamiq.

Reproduction of the front page using Pinax

Of course, we don't need everything that Pinax has. I replaced the login functionality in Pinax with django-cas, a plugin for CAS authentication. I used that to login and logout with our UH usernames. It works fairly well out of the box and didn't require any additional setup. I still kept the Pinax login, which could be useful when I'm testing the different user roles.

I have something that looks okay, but it still needs a little bit more work. My next tasks are to continue mocking up the user page and to start creating some Django apps for the functionality that we need.

Monday, February 15, 2010

Diving Into Django

After our previous milestone, I decided on a little change of pace. I was feeling okay with Elgg, but I wanted to see if I could get something done using a different framework. I've been meaning to learn a little Python and I know people who have used the Django framework. To start, I began researching Django tutorials and plugins.

First, I went through a few tutorials on Django, including the one in the Django documentation. Coming from a Rails background, I noticed quite a few differences. First, there aren't that many generated files. Rails creates quite a few folders and files when you start a new project. However, the django-admin.py script only creates a few Python files. Which is okay as long as I don't need to dig through the Django install to change things. If all of the things I need are in my newly created app, that's fine.

I guess the real difference is simply that there's more configuration in Django. A templates folder is neither created nor required unless you change the settings.py. Even after an app (as in the plugin within your Django web app) is created, it is not used until it is added to the INSTALLED_APPS field of the settings.py.

After going through these short tutorials, I know I've only scratched the surface. I borrowed a book from a friend and we have another book on order from Amazon. Going through those books should help me get more comfortable with Django.

I did some tinkering around with two Django based applications: Pinax and Django-CMS. I was interested in Pinax because they tout themselves as being an easy way to get started quickly. I found that when you start a Pinax project, you do get a pretty functional interface. However, I was a little soured on it because adding and removing the included apps was kind of a pain. There is some template file in the Pinax code that needs to be changed when I remove an installed app. There's probably some work around

Django CMS was a little tricky to set up especially with some of the recommended software (like reversion and south). Once I got it working, I found that at least the admin site works like any other Django site. Django CMS does not come with any out of the box templates though, so I need to create my own template files. Not ideal for getting started right away.

Pinax seems like the obvious choice for getting started right away. But if I can create some decent templates in Django-CMS, then that could be a viable alternative.

Time to flex them Pythons!

Monday, February 8, 2010

Drive By Elgg-ing



We are approaching the first milestone and the Elgg implementation of the Kukui Cup is in a pretty good state. Some of the things we set out to do for this milestone was to get commitments and actions going, get a Google Gadget inserted, and figure out a way to do CAS authentication. I'm happy to say that most if not all have been implemented.

Commitments and actions are supposed to be things a user can do to earn points in the dorm energy competition. They can be things that promote energy literacy like watching videos or making sure the lights in the common room are off if no one is there. Users should be able to commit to an commitment or action and have it posted on their profile. However, only admins can create them.

Creating a commitment in the admin interface.

List of actions

Getting a Google Gadget inserted was a little tricky. Something in the Elgg engine kept redirecting when I inserted the code into the HTML. Fortunately, there's an Elgg plugin called Xgadget that creates a widget. Right now, a user adds the widget to their profile and inserts the Google Gadget code. What we want is some widgets that contain pre-defined Google Gadget code. In a future revision, this can hopefully be fixed.


Xgadget widget with a Google Calendar embedded.

CAS (Central Authentication System) login for UH was another difficult feature. Ideally, we'd have the UH login system handle the username and password stuff so that we don't need to track it. Fortunately, there's a plugin for this as well! The cas_auth and ldap_auth plugins work together to log the user in. I'm able to successfully log in with my UH username and password. However, there was one little catch. Student directory listings are private and are unaccessible via LDAP. Thus, a student would currently be unable to use this system. We'll need to look into that in the future.

Login page with tiny CAS connection button

Finished user profile page.

About the next milestone, I'm considering making a switch to using a different framework. I've been wanting to look into Django for a while. There's a module called Pinax that'll add a lot of social networking functionality to the site. As soon as tomorrow's presentation is over, I'm planning on checking it out. We'll see if it's any better than Elgg.

Monday, February 1, 2010

Elgg Me On

This week was continuing work on Elgg. After my last blog, I got it to a state where it looked like the mockups provided to us at the beginning of the semester. It was also somewhat functional. Admins can create commitments and actions for the users to take and messages could be posted to each person's profile. I even put in a simple Javascript function to count down from 10 to 0 to simulate the real time use.

"Complete" profile view as of last week.

What has helped me get this far are the plugins that come with Elgg. It's not the functionality of these plugins. Rather, it's the source code for the plugins themselves. For example, I wasn't sure how to make sure only admins were allowed to create commitments and actions. I took a look at the source of the default widgets plugin, which allows admins to define what widgets are automatically added to a user's profile or dashboard. By looking at the default widgets initialization script, I figured out how to add links to create commitments and actions in the admin interface.

Admin interface for creating a new commitment

Of course, there's a good amount of fakery in "version 0.01 pre-alpha". The created commitments and actions automatically belong to the user who created them. While it looks good as a fake interface, that's not how the actual application is going to be. I had tried to avoid going deep into Elgg, but since I'm working on this again this week, I might just have to.

Even at this point, I'm still wrapping my head around how the data in Elgg is arranged. I wasn't sure how to create an object like a commitment and have multiple users be associated with it. While poking around the source code, I did stumble on a function that creates relationships between two Elgg objects. So I think I can put together some kind of interface for next time. Source code to the rescue!

The lack of detailed documentation is kind of a bummer, but having the source code helps a lot (the API can be useful too if you know what you're looking for). And from what I can tell, the community is fairly active. I haven't had to ask them any questions yet, but Philip did suggest that I ask a question to gauge their response time. Just haven't thought of a good question to ask yet.

So, in short, my tasks for this week and next are to implement the commitment/action workflows, create a widget for an arbitrary Google Gadget, implement the default index page, and remove some extraneous features that we probably don't need. And we'll see how it goes from there.

Monday, January 25, 2010

What the Elgg?

This past week, I've been researching content management systems and social network plugins to base our dorm energy competition on. I eventually got to the point where I felt using a large CMS like Joomla! or Drupal was probably more than what we really needed. There is a lot going for Joomla! and Drupal. The developer base is immense and there's no shortage of documentation. However, I think we would need to do a lot of customization for the competition and we'd need to learn how the code is structured. I'm guessing there's a lot of code in Drupal and Joomla! that we'd need to figure out.

I've been leaning toward social networking engines and plugins like CommunityEngine and Elgg because they are designed to be customized to our liking. These plugins may not be nearly as popular as Joomla! or Drupal, but they have a fair amount of documentation and activity. Since they aren't as complicated as CMSs, I felt that I'd be able to produce results quicker with a plugin rather than a full fledged system.

The plugin I chose was Elgg. There were several contenders, but Elgg has an active community and development is still ongoing. Elgg turned out to be a really choice because it's easy to implement some of the mockups we have. Elgg allows users to attach widgets to their profile and dashboard (the dashboard is what they see when they log in), which is very similar to the mockups we have.



Mockup and Elgg Implementation (in progress)

That's not to say that there weren't a few headaches. Even though I had Apache and PHP set up on my Mac, I still had to do some debugging with the Elgg install to make sure the appropriate ModRewrite rules were in place. Then was learning the framework, which is a little tricky. Elgg is designed so that the core functionality never needs to be modified. Rather, we write plugins that override the core's views and objects. It's taking me a little time to get up to speed, but I'm starting to feel more comfortable with it. As you can see in the mockups, I'm able to add in widgets and change some profile fields. I also overrode the header to add additional text (links to be added later).

Right now, I'm trying to wrap my head around the data model used in Elgg so that I can add actions and commitments to the widgets. But I'm hopeful that I'll get something kind of working by the end of the week. I'm actually more interested in what other people come up with since we're attempting to create prototypes using different CMSs/engines/plugins.

Monday, January 18, 2010

Starting From Scratch

The reason I switched projects from my iPhone application to Philip's work is that I'm interested in this dorm energy competition that he wants to have. This idea isn't exactly brand new. Other universities have had dorm energy competitions before. I'm more interested in the social networking aspect of the dorm energy competition. Because we are providing energy data for each floor in the dormitory, members of a floor can influence each other in reducing their overall usage.

Right now, we are still deciding on a technology to use in creating this social networking site. We could build it from scratch using one of the many web frameworks out there. However, our development time is somewhat limited. This application needs to be ready by the fall semester (August 2010). While I am comfortable using Ruby on Rails, getting other programmers up to speed would take some time. Our current approach is to investigate CMSs like Joomla! and Drupal and use one of those as our starting point. We still need to get up to speed on a new framework, but we'll have time due to the fact that most of the functionality we need will be already implemented.

The first CMS I investigated was Joomla! While I have heard of Joomla and CMSs, I never really investigated them until now. There's a wealth of information on the Joomla site and numerous other sites devoted to extensions. I found some of the things we were looking for, like an LDAP extension (so that users can use their UH login to authenticate) and an extension that allows subcategories for content (we hope to arrange the content based on a hierarchy that starts from the user's dorm floor and goes up to the entire state).

Of course, there's no better way to familiarize yourself with a piece of software than to download it and try it out yourself. It took a little bit of command line configuration with Apache and PHP, but I eventually got it to work. I plan on doing some digging around in the code to see if I can make changes.

Another interesting thing I found was the Community Builder plugin. This seems like it will provide most of the social networking functionality that we need. I haven't tried installing the extension yet, but it's something to look into.

I also started to look at Plone. Two websites in the department (the ICS website and the CSDL website) both use Plone. I'm a little hesitant to use Plone mostly because these two websites look almost identical. However, there are a lot of themes available for download. I also noticed that they have an LDAP plugin. I'm still looking into this CMS, but it seems like it has potential. Also, I've always wanted an excuse to learn Python (which is the language used in Plone). I'll have to ask Philip and see what he thinks since his lab is one of the sites that uses Plone.