Feeding the Cloud

Debugging OpenWRT routers by shipping logs to a remote syslog server

2012-01-14T21:45:00.000+13:00

Trying to debug problems with consumer-grade routers is notoriously difficult due to a lack of decent debugging information. It's quite hard to know what's going on without at least a few good error messages.

Here is how I made my OpenWRT-based Gargoyle router send its log messages to a network server running rsyslog.

Server Configuration

Given that the router (192.168.1.1) will be sending its log messages on UDP port 514, I started by opening that port in my firewall:

iptables -A INPUT -s 192.168.1.1 -p udp --dport 514 -j ACCEPT

Then I enabled the UDP module for rsyslog and redirected messages to a separate log file (so that it doesn't fill up /var/log/syslog) by putting the following (a modified version of these instructions) in /etc/rsyslog.d/10-gargoyle-router.conf:

$ModLoad imudp
$UDPServerRun 514
:fromhost-ip, isequal, "192.168.1.1" /var/log/gargoyle-router.log
& ~

The name of the file is important because this configuration snipet needs to be loaded before the directive which writes to /var/log/syslog for the discard statement (the "& ~" line) to work correctly.

Router Configuration

Finally, I followed the instructions on the Gargoyle wiki to get the router to forward its log messages to my server (192.168.1.2).

After logging into the router via ssh, I ran the following commands:

uci set system.@system[0].log_ip=192.168.1.2
uci set system.@system[0].conloglevel=7
uci commit

before rebooting the router.

Now whenever I have to troubleshoot network problems, I can keep a terminal open on my server and get some visibility on what the router is doing:

tail -f /var/log/gargoyle-router.log

Installing Etherpad on Debian/Ubuntu

2011-12-16T08:00:00.000+13:00

Etherpad is an excellent Open Source web application for collaborative text editing. Like Google Docs, it allows you to share documents with others through a secret URL or to set up private documents for which people need a login.

It's a little tricky to install so here's how I did it.

Build a Debian package

Because the official repository is not kept up to date, you must build the package yourself:

Grab the master branch from the official git repository:
```
git clone git://github.com/ether/pad.git etherpad
```
Build the package:
```
dpkg-buildpackage -us -uc
```

Now, install some of its dependencies:

apt-get install --no-install-recommends dbconfig-common python-uno mysql-server

before installing the .deb you built:

dpkg -i etherpad_1.1.deb
apt-get install --no-install-recommends -f

Application configuration

You will likely need to change a few minor things in the default configuration at /etc/etherpad/etherpad.local.properties:

useHttpsUrls = true
customBrandingName = ExamplePad
customEmailAddress = etherpad@example.com
topdomains = etherpad.example.com,your.external.ip.address,127.0.0.1,localhost,localhost.localdomain

Nginx configuration

If you use Nginx as your web server of choice, create a vhost file in /etc/nginx/sites-available/etherpad:

server {
  listen 443;
  server_name etherpad.example.com *.etherpad.example.com;
  add_header Strict-Transport-Security max-age=15768000;

  ssl on;
  ssl_certificate  /etc/ssl/certs/etherpad.example.com.crt;
  ssl_certificate_key  /etc/ssl/certs/etherpad.example.com.pem;

  ssl_session_timeout  5m;
  ssl_session_cache shared:SSL:1m;

  ssl_protocols  TLSv1;
  ssl_ciphers  RC4-SHA:HIGH:!kEDH;
  ssl_prefer_server_ciphers   on;

  access_log /var/log/nginx/etherpad.access.log;
  error_log /var/log/nginx/etherpad.error.log;

  location / {
    proxy_pass http://localhost:9000/;
    proxy_set_header Host $host;
  }
}

and then enable it and restart Nginx:

/etc/init.d/nginx restart

Apache configuration

If you prefer to use Apache instead, make sure that the required modules are enabled:

a2enmod proxy
a2enmod proxy_http

and then create a vhost file in /etc/apache2/sites-available/etherpad:

<VirtualHost *:443>
   ServerName etherpad.example.com
   ServerAlias *.etherpad.example.com

   SSLEngine on
   SSLCertificateFile /etc/apache2/ssl/etherpad.example.com.crt
   SSLCertificateKeyFile /etc/apache2/ssl/etherpad.example.com.pem
   SSLCertificateChainFile /etc/apache2/ssl/etherpad.example.com-chain.pem

   SSLProtocol TLSv1
   SSLHonorCipherOrder On
   SSLCipherSuite RC4-SHA:HIGH:!kEDH
   Header add Strict-Transport-Security: "max-age=15768000"

   <Proxy>
       Order deny,allow
       Allow from all
   </Proxy>

   Alias /sitemap.xml /ep/tag/\?format=sitemap
   Alias /static /usr/share/etherpad/etherpad/src/static

   ProxyPreserveHost On
   SetEnv proxy-sendchunked 1
   ProxyRequests Off
   ProxyPass / http://localhost:9000/
   ProxyPassReverse / http://localhost:9000/
</VirtualHost>

before enabling that new vhost and restarting Apache:

a2ensite etherpad
apache2ctl configtest
apache2ctl graceful

DNS setup

The final step is to create these two DNS entries to point to your web server:

*.etherpad.example.com
etherpad.example.com

Also, as a precaution against an OpenOffice/LibreOffice-related bug, I suggest that you add the following entry to your web server's /etc/hosts file to avoid flooding your DNS resolver with bogus queries:

127.0.0.1 localhost.(none) localhost.(none).fulldomain.example.com

where fulldomain.example.com is the search base defined in /etc/resolv.conf.

Other useful instructions

Here are the most useful pages I used while setting this up:

Optimising PNG files

2011-12-04T18:30:00.001+13:00

I have written about using lossless optimisations techniques to reduce the size of images before, but I recently learned of a few other tools to further reduce the size of PNG images.

Basic optimisation

While you could use Smush.it to manually optimise your images, if you want a single Open Source tool you can use in your scripts, optipng is the most effective one:

optipng -o9 image.png

Removing unnecessary chunks

While not as effective as optipng in its basic optimisation mode, pngcrush can be used remove unnecessary chunks from PNG files:

pngcrush -q -rem gAMA -rem alla -rem text image.png image.crushed.png

Depending on the software used to produce the original PNG file, this can yield significant savings so I usually start with this.

Reducing the colour palette

When optimising images uploaded by users, it's not possible to know whether or not the palette size can be reduced without too much quality degradation. On the other hand, if you are optimising your own images, it might be worth trying this lossy optimisation technique.

For example, this image went from 7.2 kB to 5.2 kB after running it through pngnq:

pngnq -f -n 32 -s 3 image.png

Re-compressing final image

Most PNG writers use zlib to compress the final output but it turns out that there are better algorithms to do this.

Using AdvanceCOMP I was able to bring the same image as above from 5.1kB to 4.6kB:

advpng -z -4 image.png

When the source image is an SVG

Another thing I noticed while optimising PNG files is that rendering a PNG of the right size straight from an SVG file produces a smaller result than exporting a large PNG from that same SVG and then resizing the PNG to smaller sizes.

Here's how you can use Inkscape to generate an 80x80 PNG:

inkscape --without-gui --export-width=80 --export-height=80 --export-png=80.png image.svg

Ideal OpenSSL configuration for Apache and nginx

2011-11-15T04:00:00.001+13:00

After recently reading a number of SSL/TLS-related articles, I decided to experiment and look for the ideal OpenSSL configuration for Apache (using mod_ssl since I haven't tried mod_gnutls yet) and nginx.

By "ideal" I mean that this configuration needs to be compatible with most user agents likely to interact with my website as well as being fast and secure.

Here is what I came up with for Apache:

SSLProtocol TLSv1
SSLHonorCipherOrder On
SSLCipherSuite RC4-SHA:HIGH:!kEDH

and for nginx:

ssl_protocols  TLSv1;
ssl_ciphers  RC4-SHA:HIGH:!kEDH;
ssl_prefer_server_ciphers   on;

Cipher and protocol selection

In terms of choosing a cipher to use, this configuration does three things:

disables all weak ciphers and protocols
disables very slow ciphers that use ephemeral Diffie-Hellman exchanges
gives priority to the RC4 cipher to minimize CPU usage and defend against the BEAST attack

Testing tools

The main tool I used while testing various configurations was the SSL labs online tool. The CipherFox extension for Firefox was also quite useful to quickly identify the selected cipher.

Of course, you'll want to make sure that your configuration works in common browsers, but you should also test with tools like wget, curl and httping. Many of the online monitoring services are based on these.

Other considerations

To increase the performance and security of your connections, you should ensure that the following features are enabled:

SSL session caching with a session store shared between all of your web servers
HSTS headers to let browsers know that they should always visit your site over HTTPS

Note: If you have different SSL-enabled name-based vhosts on the same IP address (using SNI), make sure that their SSL cipher and protocol settings are identical.

Using BrowserID and Content Security Policy together

2011-11-01T22:40:00.002+13:00

While looking into why BrowserID logins on Libravatar didn't work on Firefox, I remembered that I had recently added Content Security Policy headers. Here's what I had to do to make BrowserID work on a CSP-enabled site.

Create a hidden form and a login link

This is what the login button looked like before CSP:

<form id="browserid-form" action="/account/login_browserid" method="post">
<input id="browserid-assertion" type="hidden" name="assertion" value="">
<input style="display: none" type="submit">
</form>

<a href="javascript:try_browserid()">Login with BrowserID</a>

<script type="text/javascript">
function try_browserid() {
navigator.id.getVerifiedEmail(function(assertion) {
    if (assertion) {
        document.getElementById('browserid-assertion').setAttribute('value', assertion);
        document.getElementById('browserid-form').submit();
    }
});
}
</script>

The hidden form is there because the assertion needs to be sent to the application via POST to avoid leaking it out, but otherwise the code is pretty straightforward.

Now of course, with CSP turned ON, there are two problems:

links cannot use javascript: URIs
inline Javascript is forbidden

So we can start by converting the login link to:

<a id="browserid-link" href="#">Login with BrowserID</a>

<script src="browserid_stuff.js" type="text/javascript">

then moving the try_browserid() function to a separate file to be served from the same domain and finally hooking it up to the try_browserid() function using Javascript (in that same browserid_stuff.js file):

var link = document.getElementById('browserid-link');
link.onclick = try_browserid;
link.addEventListener('click', try_browserid, false);

Exposing the right X-Content-Security-Policy header

In order to load the Javascript code from browserid.org, we need the following as part of the policy:

script-src https://browserid.org

but that's not enough since the BrowserID login form seems to use some sort of <iframe> trick and so we need to add this extra permission as well:

frame-src https://browserid.org

Here is the final policy I ended up setting (using Apache mod_headers) for the Libravatar login page:

<Location /account/login>
Header set X-Content-Security-Policy: "default-src 'self'; frame-src 'self' https://browserid.org ; script-src 'self' https://browserid.org"
</Location>

Reducing the size of Apache 301 and 302 responses

2011-10-23T12:50:00.000+13:00

Looking through the Libravatar access logs, I found that most of the traffic we currently serve consists of 302 redirects to Gravatar. Optimising that path is therefore very important.

While Apache allows admins to provide custom error pages for things like 404 or 500, it's not quite that straightforward for 30x return codes.

Standard 301 / 302 responses

By default, Apache (and most web servers out there) returns a fairly large HTML page along with a 30x redirection. Try it for yourself by disabling automatic redirections in Firefox (Preferences | Advanced | General | Accessibility) or by installing the Request Policy add-on.

The 302 responses sent by Libravatar looked like this:

$ curl -i http://cdn.libravatar.org/avatar/12345678901234567890123456789012
HTTP/1.1 302 Found
Date: Wed, 21 Sep 2011 01:51:52 GMT
Server: Apache
Cache-Control: max-age=86400
Location: http://www.gravatar.com/avatar/12345678901234567890123456789012.jpg?r=g&s=80&d=http://cdn.libravatar.org/nobody/80.png
Vary: Accept-Encoding
Content-Length: 310
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="http://www.gravatar.com/avatar/12345678901234567890123456789012.jpg?r=g&s=80&d=http://cdn.libravatar.org/nobody/80.png">here</a>.</p>
</body></html>

As you can see, the body of the response is just as large as the headers and isn't really necessary.

Body-less 301 responses

After reading about the ErrorDocument directive, I created an empty file called 302 in the root of the web server and included this directive in my vhost configuration file:

ErrorDocument 302 /302

which made the responses look like this:

$ curl -i http://example.com/redir
HTTP/1.1 302 Found
Date: Wed, 21 Sep 2011 03:39:26 GMT
Server: Apache
Last-Modified: Wed, 21 Sep 2011 03:39:17 GMT
ETag: "8024d-0-4ad6b52201036"
Accept-Ranges: bytes
Content-Length: 0
Content-Type: text/plain

This one does have a completely empty body, however, there's an important problem with this solution: the Location header is missing! Not much point in reducing the size of the redirect if it's no longer working.

Custom 302 response page

The next thing I tried (and ended up settling on) is this:

ErrorDocument 302 " "

which results in a 1-byte response (a single space) in the body:

$ curl -i http://example.com/redir
HTTP/1.1 302 Found
Date: Wed, 21 Sep 2011 03:37:50 GMT
Server: Apache
Location: http://www.example.com
Vary: Accept-Encoding
Content-Length: 1
Content-Type: text/html; charset=iso-8859-1

There is still a little bit of unnecessary information in this response (character set, Vary and Server headers), but it's a major improvement over the original.

If you know of any other ways to reduce this further, please leave a comment!

Three Firefox extensions to enhance SSL security

2011-10-03T21:57:00.007+13:00

There has been a lot of talk recently questioning the trust authorities that underpin the SSL/TLS world. After a few high-profile incidents, it is clear that there is something wrong with this structure.

While some people have suggested that DNSSEC might solve this problem, here are three Firefox add-ons that can be used today to enhance the security of HTTPS:

Certificate Patrol applies the "trust on first use" principle familiar to most ssh users. It keeps track of the certificates you get for the sites you visit and displays a warning if important elements (e.g. the certificate authority) change.

Perspectives uses a network of certificate notaries to issue a warning when you encounter a certificate that is different from what other Perspectives users see.

Monkeysphere takes advantage of the PGP/GPG web-of-trust to verify the authenticity of certificates.

Unlike the Convergence approach which completely takes over certificate handling, all three of the above add-ons can be used together.

Adding X-Content-Security-Policy headers in a Django application

2011-09-18T20:30:00.004+12:00

Content Security Policy is a proposed HTTP extension which allows websites to restrict the external content that can be displayed by visiting web browsers. By expressing a set of rules to be enforced by the browser, a website is able to prevent the injection of outside resources by malicious users.

While adding support for the March 2011 draft in Libravatar, I looked at three different approaches.

Controlling the headers in the application

The first approach I considered was to have the Django application output all of the headers, which is what the django-csp module does. Unfortunately, I need to be able to vary the policy between pages (the views in Libravatar have different requirements) and that's one of the things that hasn't been implemented yet in that module.

Producing the same headers by hand is fairly simple:

response = render_to_response('app/view.html')
response['X-Content-Security-Policy'] = "allow 'self'"
return response

but it would mean adding a bit of code to every view and/or writing a custom wrapper for render_to_response().

Setting a default header in Apache

Ideally, I'd like to be able to set a default header in Apache using mod_headers and then override it as needed inside the application.

The first problem with this solution is that it's not possible (as far I can tell) for a Django application to override a header set by Apache:

mod_headers adds its response header after mod_wsgi has returned (unless early processing is used).
Django's response objects cannot see the early headers set by Apache.

The second problem is that mod_headers doesn't have an action that adds/sets a header only if it didn't already exist. It does have append and merge actions which could in theory be used to add extra terms to the policy but it unfortunately uses a different separator (the comma) from the CSP spec (which uses semi-colons).

Always set headers in Apache

While I would have liked to get the second approach working, in the end, I included all of the CSP directives within the main Apache config file:

Header set X-Content-Security-Policy: "allow 'self'; options inline-script; img-src 'self' data:"

<Location /account/confirm_email>
  Header set X-Content-Security-Policy: "allow 'self'; options inline-script; img-src *"
</Location>

<Location /tools/check>
  Header set X-Content-Security-Policy: "allow 'self'; options inline-script; img-src *"
</Location>

The first Header call sets a default policy which is later overriden based on the path to the Django view that's being used.

Related technologies

If you are interested in Content Security Policy, you may also want to look into Application Boundaries Enforcer (part of the NoScript Firefox extension) for more security rules that can be supplied by the server and enforced client-side.

It's also worth mentioning the excellent Request Policy extension which solves the same problem by letting users whitelist the cross-site requests they want to allow.

RequestTracker workflow with only one person using the web UI to allocate tickets to external people

2011-09-07T18:30:00.000+12:00

Here are some notes on the way I set an instance of RequestTracker up to support the following workflow:

Customers use a web form which sends an email to RT.
A contact person receives all new tickets by email and uses the RT web UI to allocate it to a another person.
The person handling the ticket gets a copy of the ticket (and doesn't see any of the other tickets) via email and never has to use RT.

Groups and Queues

For each product, I created the following RT groups:

productname-contact: Users receiving all emails sent to the ProductName queues
productname-support: People who can be assigned to ProductName queues

as well as a ProductName queue:

productname-contact is a watcher
productname-contact has all of the permissions under "Group Rights"
productname-support has all of the permissions except for AdminCC under "Group Rights"

User accounts

The contact person (can be more than one person), responsible for allocating tickets to support people gets a full RT account with a password and is a member of the productname-contact group.

The support people, who reply to tickets, get a limited RT account without a password (so they cannot login) and are members of the productname-support group.

Forwarding the initial email

In a stock RT install, this workflow works well, except for one important point: when a ticket is allocated to someone else, all they get is a "this ticket has been assigned to you" email. They have no idea what the ticket is about and since they can't login into RT and are not CCed on the emails going into that queue, that's pretty useless to them.

The solution I found is to add a new ForwardFirstMessage global template:

{
 my $Transactions = $Ticket->Transactions;
 $Transactions->Limit( FIELD => 'Type', VALUE => 'Create' );
 my $first_message;
 my $CreateObj = $Transactions->First;
 if( $CreateObj && $CreateObj->id ) {
   $first_message = $CreateObj->Content;
 }

 $first_message;
}

and then edit the On Owner Change Notify Owner global scrip to make use of that template.

With this, the person that is allocated to a ticket will receive the original email from the requestor as if it had been sent directly to their email address in the first place.

Translating Django applications using Launchpad

2011-08-02T11:58:00.004+12:00

One of my Django-based projects, Libravatar, makes use of Launchpad's interface to keep its translations current. Unfortunately, the PO file layout that Django insists on using isn't directly compatible with the one that Launchpad needs in order to setup automatic import of translations on a branch.

The solution I found is to use the mandatory Launchpad layout and then create symlinks in the right places for Django.

Here is where the Libravatar PO files are located in the bzr repository:

po/libravatar/de.po
po/libravatar/fr.po
po/libravatar/libravatar.pot

and here are the (relative) symbolic links:

libravatar/locale/de/LC_MESSAGES/django.po
  -> ../../../../po/libravatar/de.po
libravatar/locale/en/LC_MESSAGES/django.po
  -> ../../../../po/libravatar/libravatar.pot
libravatar/locale/fr/LC_MESSAGES/django.po
  -> ../../../../po/libravatar/fr.po

Note that the "en" localization is left untranslated and used as the translation template (POT file).

Reducing the number of HTTP connections used by a webapp

2011-06-28T20:30:00.000+12:00

One of the factors affecting the speed (latency) of a web application is the number of HTTP connections needed to download a given page. These connections are often reused and some of the resources that a page needs will be downloaded in parallel, but that's not always the case (browsers will limit the number of connections they make to each web server for example) so it pays to take that into consideration while optimizing a page.

Client-side image maps can often help, but here are two other ways to reduce the number of external images a page needs.

Embed images as Base64 data

Thanks to the Data:URI scheme, it is possible to embed images directly into a CSS file, as Base64 data.

Of course this trick only makes sense for very small images (typically a small PNG icons) since the Base64 version encoding of a binary image will be larger than the original image (though some of that will be reclaimed when the CSS file is gzipped).

First of all, make sure your PNG image is as small as possible using optipng:

optipng -o9 image.png

then convert it to text:

base64 image.png

and paste it into your CSS file:

#object-id {
    background: url('data:image/png;base64,iVBORw0KGgoAAAANSU...K5CYII=');
}

Make sure that the Base64 data is all on one line otherwise it apparently won't work on old versions of Firefox.

Note that the downside of this technique is that it doesn't work on IE7 so you might want to limit it to non-essential icons.

Use CSS sprites

This second technique does work on Internet Explorer and it can be particularly good when dealing with lots of small images. After all, a small image that weighs only a few bytes could be quite large once you factor in the HTTP response headers that will come with it when it is served to browsers.

I suggest you get started by reading this excellent introduction, but the idea is to create a single image which contains all of the smaller ones. Then each element displays a different region of the same image (which is only transferred once).

Here's what the stylesheet looks like for two side-by-side clickable images:

#sponsors {
  position: relative;
  height: 96px;
}

#sponsors li {
  background-image: url("sponsors.png");
  height: 96px;
  width: 96px;
  list-style: none;
  position: absolute;
}

#sponsors li a {
  height: 96px;
  width: 96px;
  display: block;
}

#sponsor1-logo {
  background-position: 96px 0px;
  left: 0px;
}

#sponsor2-logo {
  background-position: 0px 0px;
  left: 96px;
}

and the matching HTML fragment:


<ul id="sponsors">
  <li id="logo1"><a href="http://example.org" title="Sponsor1"></a></li>
  <li id="logo2"><a href="http://example.com" title="Sponsor2"></a></li>
</ul>

Integrating Launchpad and Gerrit Code Review

2011-05-30T18:30:00.002+12:00

While talking about how Mahara uses Gerrit and Gitorious for code reviews, I forgot to include details on how to link bugs in the commit logs to our bug tracker (Launchpad).

Links to Launchpad in the Gerrit review UI

Turning the bug numbers in git commit messages to links was just a matter of adding this to our etc/gerrit.config file.

[commentlink "launchpad"]
  match = "([Bb]ug\\s+#?)(\\d+)"
  link = https://bugs.launchpad.net/mahara/+bug/$2

All mentions of a bug number like "(bug #1234)" or "bug 5678" in a commit message or Gerrit review will be converted into links to the appropriate page on Launchpad.

Updating Launchpad bugs after Gerrit merges

Linking the original bug reports in Launchpad to committed code (and its review in Gerrit) was a little more complicated. It makes use of the hooks available in Gerrit.

The first step was to create a new Launchpad account for these automated updates and to confirm the email address it will be sending emails from. In our case, the account is MaharaBot.

Then, we wrote a custom hook script (inspired by one for JIRA) which we placed in ~/mahara_reviews/etc/gerrit.config.

For an example of what it sends to the bug tracker, see bug 788457.

Code reviews with Gerrit and Gitorious

2011-04-17T17:00:00.009+12:00

The Mahara project has just moved to mandatory code reviews for every commit that gets applied to core code.

Here is a description of how Gerrit Code Review, the peer-review system used by Android, was retrofitted into our existing git repository on Gitorious.

(If you want to know more about Gerrit, listen to this FLOSS Weekly interview.)

Replacing existing Gitorious committers with a robot

The first thing to do was to log into Gitorious and remove commit rights from everyone in the main repository. Then I created a new maharabot account with a password-less SSH key (stored under /home/gerrit/.ssh/) and made that new account the sole committer.

This is to ensure that nobody pushes to the repository by mistake since all of these changes would be overwritten by Gerrit.

Basic Gerrit installation

After going through the installation instructions, I logged into the Gerrit admin interface and created a new "mahara" project.

I picked the "merge if necessary" submit action because "cherry-pick" would disable dependency tracking which is quite a handy feature.

Reverse proxy using Nginx

Since we wanted to offer Gerrit over HTTPS, I decided to run it behind an Nginx proxy. This is the Nginx configuration I ended up with:

server {
     listen 443;
     server_name reviews.mahara.org;
     add_header Strict-Transport-Security max-age=15768000;

     ssl on;
     ssl_certificate  /etc/ssl/certs/reviews.mahara.org.crt;
     ssl_certificate_key  /etc/ssl/certs/reviews.mahara.org.pem;

     ssl_session_timeout  5m;
     ssl_session_cache shared:SSL:1m;

     ssl_protocols  TLSv1;
     ssl_ciphers  HIGH:!ADH;
     ssl_prefer_server_ciphers   on;

     location / {
             proxy_pass   http://127.0.0.1:8081;
             proxy_set_header X-Forwarded-For $remote_addr;
             proxy_set_header Host $host;
     }
}

Things to note:

An HTTP to HTTPS redirection is not provided.

The HSTS headers indicates to modern browsers that this URL should only be accessed via HTTPS.

Only strong SSL ciphers are enabled.

Before proxying requests to Gerrit, Nginx adds a few headers to identify the origin of the request. The Host header in particular must be present otherwise built-in Gerrit redirections will fail.

Mail setup

To enable Gerrit to email reviewers and committers, I installed Postfix and used "reviews.mahara.org" as the "System mail name".

Then I added the following to /home/gerrit/mahara_reviews/etc/gerrit.config:

[user]
 email = "gerrit@reviews.mahara.org"

to fix the From address in outgoing emails.

Init script and cron

Following the installation instructions, I created these symlinks:

ln -s /home/gerrit/mahara_reviews/bin/gerrit.sh /etc/init.d/gerrit
cd /etc/rc2.d && ln -s ../init.d/gerrit S19gerrit
cd /etc/rc3.d && ln -s ../init.d/gerrit S19gerrit
cd /etc/rc4.d && ln -s ../init.d/gerrit S19gerrit
cd /etc/rc5.d && ln -s ../init.d/gerrit S19gerrit
cd /etc/rc0.d && ln -s ../init.d/gerrit K21gerrit
cd /etc/rc1.d && ln -s ../init.d/gerrit K21gerrit
cd /etc/rc6.d && ln -s ../init.d/gerrit K21gerrit

and put the following settings into /etc/default/gerritcodereview:

GERRIT_SITE=/home/gerrit/mahara_reviews
GERRIT_USER=gerrit
GERRIT_WAR=/home/gerrit/gerrit.war

to automatically start and stop Gerrit.

I also added a cron job in /etc/cron.d/gitcleanup to ensure that the built-in git repository doesn't get bloated:

MAILTO=admin@example.com
20 4 * * * gerrit GIT_DIR=/home/gerrit/mahara_reviews/git/mahara.git git gc --quiet

Configuration enhancements

To allow images in change requests to be displayed inside the browser, I marked them as safe in /home/gerrit/mahara_reviews/etc/gerrit.config:

[mimetype "image/*"]
 safe = true

Another thing I did to enhance the review experience was to enable the gitweb repository browser:

apt-get install gitweb

and to make checkouts faster by enabling anonymous Git access:

[gerrit]
  canonicalGitUrl = git://reviews.mahara.org/git/
[download]
  scheme = ssh
  scheme = anon_http
  scheme = anon_git

which requires that you have a git daemon running and listening on port 9418:

apt-get install git-daemon-run
ln -s /home/gerrit/mahara_reviews/git/mahara.git /var/cache/git/
touch /home/gerrit/mahara_reviews/git/mahara.git/git-daemon-export-ok

Finally, I included the Mahara branding in the header and footer of each page by providing valid XHTML fragments in /home/gerrit/mahara_reviews/etc/GerritSiteHeader.html and GerritSiteFooter.html.

Initial import and replication

Once Gerrit was fully working, I performed the initial code import by using my administrator account to push the exiting Gitorious branches to the internal git repository:

git remote add gerrit ssh://username@reviews.mahara.org:29418/mahara
git push gerrit 1.2_STABLE
git push gerrit 1.3_STABLE
git push gerrit master

Note that I had to temporarily disable "Require Change IDs" in the project settings in order to import the old commits which didn't have these.

To replicate the internal Gerrit repository back to Gitorious, I created a new /home/gerrit/mahara_reviews/etc/replication.config file:

[remote "gitorious"]
 url = gitorious.org:mahara/${name}.git
 push = +refs/heads/*:refs/heads/*
 push = +refs/tags/*:refs/tags/*

(The ${name} variable is required even when you have a single project.)

Contributor instructions

This is how developers can get a working checkout of our code now:

git clone git://gitorious.org/mahara/mahara.git
cd mahara
git remote add gerrit ssh://username@reviews.mahara.org:29418/mahara
git fetch gerrit
scp -p -P 29418 reviews.mahara.org:hooks/commit-msg .git/hooks/

and this is how they can submit local changes to Gerrit:

git push gerrit HEAD:refs/for/master

Anybody can submit change requests or comment on them but make sure you do not have the Cookie Pie Firefox extension installed or you will be unable to log into Gerrit.

Using plupload inside a Django application

2011-04-08T08:00:00.000+12:00

Plupload is a reusable component which fully takes advantage of the file upload capabilities of your browser and its plugins. It will use HTML5, Flash and Google Gears if they are available, but otherwise, it can gracefully degrade down to HTML4 it it needs to. Here's how it can be integrated within a Django web application.

(I have posted the sample application I will refer to and you may use it anyway you like.)

Creating a basic upload form

The first step is to create a simple one-file upload form that will be used in the case where Javascript is disabled:

class UploadForm(forms.Form):
   file = forms.FileField()

   def save(self, uploaded_file):
       print 'File "%s" would presumably be saved to disk now.' % uploaded_file
       pass

Then you can add this form to one of your templates:

<form enctype="multipart/form-data" action="{% url plupload_sample.upload.views.upload_file %}" method="post">
{% csrf_token %}

<div id="uploader">
 {{form.file.errors}}{{form.file}}
 <input type="submit" value="Upload" />
</div>

</form>

And create a new method to receive the form data:

@csrf_protect
def upload_file(request):
   if request.method == 'POST':
       form = UploadForm(request.POST, request.FILES)
       if form.is_valid():
           uploaded_file = request.FILES['file']
           form.save(uploaded_file)

           return HttpResponseRedirect(reverse('plupload_sample.upload.views.upload_file'))
   else:
       form = UploadForm()

   return render_to_response('upload_file.html', {'form': form}, context_instance=RequestContext(request))

Adding Plupload to the template

In order to display the right Javascript-based upload form, add the following code, based on the official example, to the head of your template:

<link rel="stylesheet" href="/css/plupload.queue.css" type="text/css">
<script type="text/javascript" src="/js/jquery.min.js"></script>
<script type="text/javascript" src="/js/plupload.full.min.js"></script>
<script type="text/javascript" src="/js/jquery.plupload.queue.min.js"></script>

<script type="text/javascript">
$(function() {
   $("#uploader").pluploadQueue({
       runtimes : 'html5,html4',
       url : '{% url plupload_sample.upload.views.upload_file %}',
       max_file_size : '1mb',
       chunk_size: '1mb',
       unique_names : false,
       multipart: true,

       headers : {'X-Requested-With' : 'XMLHttpRequest', 'X-CSRFToken' : '{{csrf_token}}'},
   });

   $('form').submit(function(e) {
       var uploader = $('#uploader').pluploadQueue();

       // Validate number of uploaded files
       if (uploader.total.uploaded == 0) {
           // Files in queue upload them first
           if (uploader.files.length > 0) {
               // When all files are uploaded submit form
               uploader.bind('UploadProgress', function() {
                   if (uploader.total.uploaded == uploader.files.length)
                       $('form').submit();
               });

               uploader.start();
           } else {
               alert('You must at least upload one file.');
           }

           e.preventDefault();
       }
   });
});
</script>

Pay close attention to the extra headers that need to be added to ensure that the AJAX requests will pass the Django CSRF checks:

X-Requested-With: XMLHttpRequest
X-CSRFToken: {{csrf_token}}

Adding Plupload to the view method

Now in order to properly receive the files uploaded by Plupload via AJAX calls, we need to revise our upload_file() method:

@csrf_protect
def upload_file(request):
   if request.method == 'POST':
       form = UploadForm(request.POST, request.FILES)
       if form.is_valid():
           uploaded_file = request.FILES['file']
           form.save(uploaded_file)

           if request.is_ajax():
               response = HttpResponse('{"jsonrpc" : "2.0", "result" : null, "id" : "id"}', mimetype='text/plain; charset=UTF-8')
               response['Expires'] = 'Mon, 1 Jan 2000 01:00:00 GMT'
               response['Cache-Control'] = 'no-store, no-cache, must-revalidate, post-check=0, pre-check=0'
               response['Pragma'] = 'no-cache'
               return response
           else:
               return HttpResponseRedirect(reverse('plupload_sample.upload.views.upload_file'))
   else:
       form = UploadForm()

   return render_to_response('upload_file.html', {'form': form}, context_instance=RequestContext(request))

The above includes the response (which is not really documented as far as I can tell) that needs to be sent back to Plupload to make sure it knows that the file has been received successfully:

{"jsonrpc" : "2.0", "result" : null, "id" : "id"}

Adding support for multipart files

Our solution so far works fine except when uploading large files that need to be sent in multiple chunks.

This involves writing to a temporary file until all parts have been received:

@csrf_protect
def upload_file(request):
   if request.method == 'POST':
       form = UploadForm(request.POST, request.FILES)
       if form.is_valid():
           uploaded_file = request.FILES['file']
           chunk = request.REQUEST.get('chunk', '0')
           chunks = request.REQUEST.get('chunks', '0')
           name = request.REQUEST.get('name', '')

           if not name:
               name = uploaded_file.name

           temp_file = '/tmp/insecure.tmp'
           with open(temp_file, ('wb' if chunk == '0' else 'ab')) as f:
               for content in uploaded_file.chunks():
                   f.write(content)

           if int(chunk) + 1 >= int(chunks):
               form.save(temp_file, name)

           if request.is_ajax():
               response = HttpResponse('{"jsonrpc" : "2.0", "result" : null, "id" : "id"}', mimetype='text/plain; charset=UTF-8')
               response['Expires'] = 'Mon, 1 Jan 2000 01:00:00 GMT'
               response['Cache-Control'] = 'no-store, no-cache, must-revalidate, post-check=0, pre-check=0'
               response['Pragma'] = 'no-cache'
               return response
           else:
               return HttpResponseRedirect(reverse('plupload_sample.upload.views.upload_file'))
   else:
       form = UploadForm()

   return render_to_response('upload_file.html', {'form': form}, context_instance=RequestContext(request))

Note that I have used /tmp/insecure.tmp for brevity. In a real application, you do need to use a secure mechanism to create the temporary file or you would expose yourself to a tempfile vulnerability.

Encrypted system backup to DVD

2011-04-03T11:30:00.006+12:00

Inspired by World Backup Day, I decided to take a backup of my laptop. Thanks to using a free operating system I don't have to backup any of my software, just configuration and data files, which fit on a single DVD.

In order to avoid worrying too much about secure storage and disposal of these backups, I have decided to encrypt them using a standard encrypted loopback filesystem.

(Feel free to leave a comment if you can suggest an easier way of doing this.)

Cryptmount setup

Install cryptmount:

apt-get install cryptmount

and setup two encrypted mount points in /etc/cryptmount/cmtab:

backup {
  dev=/backup.dat
  dir=/backup
  fstype=ext2    fsoptions=defaults     cipher=aes

  keyfile=/backup.key
  keyhash=sha1    keycipher=des3
}

testbackup {
  dev=/cdrom/backup.dat
  dir=/backup
  fstype=ext2    fsoptions=defaults    cipher=aes

  keyfile=/cdrom/backup.key
  keyhash=sha1    keycipher=des3
}

Initialize the encrypted filesystem

Make sure you have at least 4.3 GB of free disk space on / and then run:

mkdir /backup
dd if=/dev/zero of=/backup.dat bs=1M count=4096
cryptmount --generate-key 32 backup
cryptmount --prepare backup
mkfs.ext2 -m 0 /dev/mapper/backup
cryptmount --release backup

Burn the data to a DVD

Mount the newly created partition:

cryptmount backup

and then copy the files you want to /backup/ before unmounting that partition:

cryptmount -u backup

Finally, use your favourite DVD-burning program to burn these two files:

/backup.dat
/backup.key

Test your backup

Before deleting these two files, test the DVD you've just burned by mounting it:

mount /cdrom
cryptmount testbackup

and looking at a random sampling of the files contained in /backup.

Once you are satisfied that your backup is fine, umount the DVD:

cryptmount -u testbackup
umount /cdrom

and remove the temporary files:

rm /backup.dat /backup.key

Installing Fallout 3 on Ubuntu Lucid

2011-03-31T22:00:00.002+13:00

Here are some notes on what I did to install Fallout 3 (Game of the year edition) on Ubuntu Lucid using Wine.

Prerequisites

use the Lucid NVIDIA drivers, the ones in the X Updates PPA currently do not work

install the latest version (actually 1.3.6 is broken, so use 1.3.5 in the meantime) of Wine using the Ubuntu Wine Team PPA

Basic Installation

follow the WineHQ instructions

after installing both DVDs, apply the patch

set the appropriate performance settings for your hardware (make sure you use the same resolution as the one you use in Xorg)

click on "Data Files" and enable all of the Fallout 3 add-ons you have installed

Performance Tweaks

Change these settings in your ~/My Games/Fallout 3/FALLOUT.INI:

bUseThreadedParticleSystem=1
bUseThreadedBlood=1
bUseThreadedMorpher=1
bUseThreadedAI=1
iNumHWThreads=2
iNumHavokThreads=5
bDoSpecularPass=0
bUseRefractionShader=0
bInvalidateOlderFiles=1

Then create a direct3d.reg file (replacing 512 with the correct amount of video memory you have, in MB):

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\SOFTWARE\Wine\Direct3D]
"OffscreenRenderingMode"="backbuffer"
"VideoMemorySize"="512"

[HKEY_CURRENT_USER\Control Panel\Desktop]
"FontSmoothing"="2"
"FontSmoothingType"=dword:00000002
"FontSmoothingGamma"=dword:00000578
"FontSmoothingOrientation"=dword:00000001

and import these settings (which come from the Wine wiki and its appdb) into the registry:

regedit file.reg

Savegame corruption

At some point, I did run into savegame corruption (mostly due to my hard disk filling up) and I created this cronjob in /etc/cron.hourly/fallout3-savegames to detect corrupt savegame:

#!/bin/sh
find /path/to/Fallout3/Savegames/ -type f -name "*.tmp" -print

Installing mods

If you want to install user-contributed mods (for example, if you can't be bothered with the lockpicking and terminal hacking mini-games), you will most likely want to install FOMM.

To do that, follow these instructions to install dotnet20 and gdiplus using winetricks otherwise you may run into this bug.

Finally, pay attention to the order in which your selected mods are loaded since some of the mods will overwrite files provided by another.

Hints and tips

There is an incredibly dedicated community maintaining a wiki for this game as well as all of the other "chapters" of the Fallout series. Use it wisely!

Setting up RAID on an existing Debian/Ubuntu installation

2011-03-13T23:25:00.001+13:00

I run RAID1 on all of the machines I support. While such hard disk mirroring is not a replacement for having good working backups, it means that a single drive failure is not going to force me to have to spend lots of time rebuilding a machine.

The best possible time to set this up is of course when you first install the operating system. The Debian installer will set everything up for you if you choose that option and Ubuntu has alternate installation CDs which allow you to do the same.

This post documents the steps I followed to retrofit RAID1 into an existing Debian squeeze installation. Getting a mirrored setup after the fact.

Overview

Before you start, make sure the following packages are installed:

apt-get install mdadm rsync initramfs-tools

Then go through these steps:

Partition the new drive.
Create new degraded RAID arrays.
Install GRUB2 on both drives.
Copy existing data onto the new drive.
Reboot using the RAIDed drive and test system.
Wipe the original drive by adding it to the RAID array.
Test booting off of the original drive.
Resync drives.
Test booting off of the new drive.
Reboot with the two drives and resync the array.

(My instructions are mostly based on this old tutorial but also on this more recent one.)

1- Partition the new drive

Once you have connected the new drive (/dev/sdb), boot into your system and use one of cfdisk or fdisk to display the partition information for the existing drive (/dev/sda on my system).

The idea is to create partitions of the same size on the new drive. (If the new drive is bigger, leave the rest of the drive unpartitioned.)

Partition types should all be: fd (or "linux raid autodetect").

2- Create new degraded RAID arrays

The newly partioned drive, consisting of a root and a swap partition, can be added to new RAID1 arrays using mdadm:

mdadm --create /dev/md0 --level=1 --raid-devices=2 missing /dev/sdb1
mdadm --create /dev/md1 --level=1 --raid-devices=2 missing /dev/sdb2

and formatted like this:

mkswap /dev/md1
mkfs.ext4 /dev/md0

Specify these devices explicitly in /etc/mdadm/mdadm.conf:

DEVICE /dev/sda* /dev/sdb*

and append the RAID arrays to the end of that file:

mdadm --detail --scan >> /etc/mdadm/mdadm.conf
dpkg-reconfigure mdadm

You can check the status of your RAID arrays at any time by running this command:

cat /proc/mdstat

3- Install GRUB2 on both drives

The best way to ensure that GRUB2, the default bootloader in Debian and Ubuntu, is installed on both drives is to reconfigure its package:

dpkg-reconfigure grub-pc

and select both /dev/sda and /dev/sdb (but not /dev/md0) as installation targets.

This should cause the init ramdisk (/boot/initrd.img-2.6.32-5-amd64) and the grub menu (/boot/grub/grub.cfg) to be rebuilt with RAID support.

4- Copy existing data onto the new drive

Copy everything that's on the existing drive onto the new one using rsync:

mkdir /tmp/mntroot
mount /dev/md0 /tmp/mntroot
rsync -auHxv --exclude=/proc/* --exclude=/sys/* --exclude=/tmp/* /* /tmp/mntroot/

5- Reboot using the RAIDed drive and test system

Before rebooting, open /tmp/mntroot/etc/fstab, and change /dev/sda1 and /dev/sda2 to /dev/md0 and /dev/md1 respectively.

Then reboot and from within the GRUB menu, hit "e" to enter edit mode and make sure that you will be booting off of the new disk:

set root='(md/0)'
linux /boot/vmlinuz-2.6.32-5-amd64 root=/dev/md0 ro quiet

Once the system is up, you can check that the root partition is indeed using the RAID array by running mount and looking for something like:

/dev/md0 on / type ext4 (rw,noatime,errors=remount-ro)

6- Wipe the original drive by adding it to the RAID array

Once you have verified that everything is working on /dev/sdb, it's time to change the partition types on /dev/sda to fd and to add the original drive to the degraded RAID array:

mdadm /dev/md0 -a /dev/sda1
mdadm /dev/md1 -a /dev/sda2

You'll have to wait until the two partitions are fully synchronized but you can check the sync status using:

watch -n1 cat /proc/mdstat

7- Test booting off of the original drive

Once the sync is finished, update the boot loader menu:

update-grub

and shut the system down:

shutdown -h now

before physically disconnecting /dev/sdb and turning the machine back on to test booting with only /dev/sda present.

After a successful boot, shut the machine down and plug the second drive back in before powering it up again.

8- Resync drives

If everything works, you should see the following after running cat /proc/mdstat:

md0 : active raid1 sda1[1]
280567040 blocks [2/1] [_U]

indicating that the RAID array is incomplete and that the second drive is not part of it.

To add the second drive back in and start the sync again:

mdadm /dev/md0 -a /dev/sdb1

9- Test booting off of the new drive

To complete the testing, shut the machine down, pull /dev/sda out and try booting with /dev/sdb only.

10- Reboot with the two drives and resync the array

Once you are satisfied that it works, reboot with both drives plugged in and re-add the first drive to the array:

mdadm /dev/md0 -a /dev/sda1

Your setup is now complete and fully tested.

Ongoing maintenance

I recommend making sure the two RAIDed drives stay in sync by enabling periodic RAID checks. The easiest way is to enable the checks that are built into the Debian package:

dpkg-reconfigure mdadm

but you can also create a weekly or monthly cronjob which does the following:

echo "check" > /sys/block/md0/md/sync_action

Something else you should seriously consider is to install the smartmontools package and run weekly SMART checks by putting something like this in your /etc/smartd.conf:

/dev/sda -a -d ata -o on -S on -s (S/../.././02|L/../../6/03)
/dev/sdb -a -d ata -o on -S on -s (S/../.././02|L/../../6/03)

These checks, performed by the hard disk controllers directly, could warn you of imminent failures ahead of time. Personally, when I start seeing errors in the SMART log (smartctl -a /dev/sda), I order a new drive straight away.

Keeping a log of branch updates on a git server

2011-02-01T08:00:00.001+13:00

Using a combination of bad luck and some of the more advanced git options, it is possible to mess up a centralised repository by accidentally pushing a branch and overwriting the existing branch pointer (or "head") on the server.

If you know where the head was pointing prior to that push, recovering it is a simple matter of running this on the server:

git update-ref /refs/heads/branchname commit_id

However, if you don't know the previous commit ID, then you pretty much have to dig through the history using git log.

Enabling a server-side reflog

One option to prevent this from happening is to simply enable the reflog, which is disabled by default in bare repositories, on the server.

Simply add this to your git config file on the server:

[core]
   logallrefupdates = true

and then whenever a head is updated, an entry will be added to the reflog.

Serving pre-compressed files using Apache

2011-01-25T17:00:00.007+13:00

The easiest way to compress the data that is being served to the visitors of your web application is to make use of mod_deflate. Once you have enabled that module and provided it with a suitable configuration file, it will compress all releant files on the fly as it is serving them.

Given that I was already going to minify my Javascript and CSS files ahead of time (i.e. not using mod_pagespeed), I figured that there must be a way for me to serve gzipped files directly.

"Compiling" Static Files

I decided to treat my web application like a c program. After all, it starts as readable source code and ends up as an unreadable binary file.

So I created a Makefile to minify and compress all CSS and Javascript files using YUI Compressor and gzip:

all: build

build:
     find static/css -type f -name "[^.]*.css" -execdir yui-compressor -o {}.css {} \;
     find static/js -type f -name "[^.]*.js"  -execdir yui-compressor -o {}.js {} \;
     cd static/css && for f in *.css.css ; do gzip -c $$f > `basename $$f .css`.gz ; done
     cd static/js && for f in *.js.js ; do gzip -c $$f > `basename $$f .js`.gz ; done

clean:
     find static/css -name "*.css.css" -delete
     find static/js -name "*.js.js" -delete
     find static/css -name "*.css.gz" -delete
     find static/js -name "*.js.gz" -delete
     find -name "*.pyc" -delete

This leaves the original files intact and adds minified .css.css and .js.js files as well as minified and compressed .css.gz and .js.gz files.

How browsers advertise gzip support

The nice thing about serving compressed content to browsers is that browsers that support receiving gzipped content (almost all of them nowadays) include the following HTTP header in their requests:

Accept-Encoding = gzip,deflate

(Incidently, if you want to test what non-gzipped enable browsers see, just browse to about:config and remove what's in the network.http.accept-encoding variable.)

Serving compressed files to clients

To serve different files to different browsers, all that's needed is to enable Multiviews in our Apache configuration (as suggested on the Apache mailing list):

<Directory /var/www/static/css>
 AddEncoding gzip gz
 ForceType text/css
 Options +Multiviews
 SetEnv force-no-vary
 Header set Cache-Control "private"
</Directory>

<Directory /var/www/static/js>
 AddEncoding gzip gz
 ForceType text/javascript
 Options +Multiviews
 SetEnv force-no-vary
 Header set Cache-Control "private"
</Directory>

The ForceType directive is there to force the mimetype (as described in this solution) and to make sure that browsers (including Firefox) don't download the files to disk.

As for the SetEnv directive, it turns out that on Internet Explorer, most files with a Vary header (added by Apache) are not cached and so we must make sure it gets stripped out before the response goes out.

Finally, the Cache-Control headers are set to private to prevent intermediate/transparent proxies from caching our CSS and Javascript files, while allowing browsers to do so. If intermediate proxies start caching compressed content, they may incorrectly serve it to clients without gzip support.

Sample Python application using Libgearman

2011-01-12T20:20:00.003+13:00

Gearman is a distributed queue with several language bindings.

While Gearman has a nice Python implementation (python-gearman) of the client and worker, I chose to use the libgearman bindings (python-libgearman) directly since they are already packaged for Debian (as python-gearman.libgearman).

Unfortunately, these bindings are not very well documented, so here's the sample application I wished I had seen before I started.

Using the command-line tools

Before diving into the Python bindings, you should make sure that you can get a quick application working on the command line (using the gearman-tools package).

Here's a very simple worker which returns verbatim the input it receives:

gearman -w -f myfunction cat

and here is the matching client:

gearman -f myfunction 'test'

You can have have a look at the status of the queues in the server by connecting to gearmand via telnet (port 4730) and issuing the status command.

Using the Python libgearman bindings

Once your gearman setup is working (debugging is easier with the command-line tools), you can roll the gearman connection code into your application.

Here's a simple Python worker which returns what it receives:

#!/usr/bin/python

from gearman import libgearman

def work(job):
   workload = job.get_workload()
   return workload

gm_worker = libgearman.Worker()
gm_worker.add_server('localhost')
gm_worker.add_function('myfunction', work)

while True:
   gm_worker.work()

and a matching client:

#!/usr/bin/python

from gearman import libgearman

gm_client = libgearman.Client()
gm_client.add_server('localhost')

result = gm_client.do('myfunction', 'test')
print result

This should behave in exactly the same way as the command-line examples above.

Returning job errors

If you want to expose to the client errors in the processing done by the worker, modify the worker like this:

#!/usr/bin/python

from gearman import libgearman

def work(job):
   workload = job.get_workload()
   if workload == 'fail':
       job.send_fail()
   return workload

gm_worker = libgearman.Worker()
gm_worker.add_server('localhost')
gm_worker.add_function('myfunction', work)

while True:
   gm_worker.work()

and the client this way:

#!/usr/bin/python

from gearman import libgearman

gm_client = libgearman.Client()
gm_client.add_server('localhost')

result = gm_client.do('myfunction', 'fail')
print result

License

The above source code is released under the following terms:

To the extent possible under law, Francois Marier has waived all copyright and related or neighboring rights to this sample libgearman Python application. This work is published from: New Zealand.

Peer-to-peer video-conferencing using free software

2010-12-19T19:55:00.008+13:00

I was looking for a simple free software solution which would allow me to have a video call with someone else (I don't care about sound since I've already got that working through Asterisk) and I ended up writing a Gstreamer-based poor man's videoconf solution because I wasn't satisfied with the other options I considered.

Empathy

Empathy was my first choice since it seems to be the preferred GNOME communication software nowadays.

While the quality of the video was excellent, the latency between New Zealand and Canada was unbearable: a full 6 seconds. I suspect that this is due to the fact that it runs everything through the Google Talk STUN server and I couldn't find how to force it to go directly from one host to the other.

Ekiga

Ekiga was my second choice since I had used it succesfully in the past.

It was not too bad latency-wise, but the quality of the video was not as good as Empathy (it was smaller and choppier). Also, given that it was running over SIP, it was interfering with my VoIP phone.

Direct peer-to-peer streaming

Given that I wasn't gonna use the voice features of these video-conference tools, I figured that there must be an easy way to just stream video from one peer to the other. That's when I thought of looking into Gstreamer (apt-get install gstreamer0.10-tools on Debian/Ubuntu).

To stream video from my webcam onto port 5000, I ran:

gst-launch v4l2src device=/dev/video0 ! videorate ! video/x-raw-yuv,width=640,height=480,framerate=6/1 ! jpegenc quality=30 ! multipartmux ! tcpserversink port=5000

which is the best I could do within 85 kbps (100-120 kbps is about the maximum reliable synchronous bandwidth I get between New Zealand and Canada):

resolution of 640x480
6 frames per second
jpeg quality of 30%

On the other computer, I simply ran this to connect and display the remote stream:

gst-launch tcpclientsrc host=stream.example.com port=5000 ! multipartdemux ! jpegdec ! autovideosink

Then I swapped the roles around to also stream video the other way around. That's it: two-way peer-to-peer video link!

Small tweaks to the Gstreamer pipeline

There are quite a few plugins that can be used within Gstreamer pipelines.

If you have problems with autovideosink refusing to load (I did on one of the two computers), you can also install the gstreamer0.10-sdl package and replace autovideosink with sdlvideosink:

gst-launch tcpclientsrc host=example.com port=5000 ! multipartdemux ! jpegdec ! sdlvideosink

Another change I had to make on one of the machines was to flip the image coming out of the webcam (which insists on giving me a mirror image instead of acting like a real camera):

gst-launch v4l2src device=/dev/video0 ! videorate ! video/x-raw-yuv,width=640,height=480,framerate=6/1 ! videoflip method=horizontal-flip ! jpegenc quality=30 ! multipartmux ! tcpserversink port=5000

Possible improvements

I got down to about 1-2 seconds of latency, which isn't bad considering the processing to be done and the distance bits have to travel, but I would love to further reduce this.

Using jpegenc was a lot better than theoraenc which added an extra 3-4 seconds of latency. Is there a better codec I should be using?

Another thing I thought of trying was to switch from TCP to UDP. I'm currently using tcpserversink and tcpclientsrc but since I don't care about having a few dropped frames, maybe I should look into the udp and rtp plugins. It seems like it might help but it also seems to be quite a bit more complicated and I have yet to find an easy way to make use of the RTP stack in Gstreamer.

Please feel free leave a comment if you can suggest ways of improving my quick 'n dirty solution.

RAID1 alternative for SSD drives

2010-11-02T20:10:00.000+13:00

I recently added a solid-state drive to my desktop computer to take advantage of the performance boost rumored to come with these drives. For reliability reasons, I've always tried to use software RAID1 to avoid having to reinstall my machine from backups should a hard drive fail. While this strategy is fairly cheap with regular hard drives, it's not really workable with SSD drives which are still an order of magnitude more expensive.

The strategy I settled on is this one:

continue to have all partitions (/, /home and /data) on my RAID1 hard drives,
put another copy of the root partition (/) on the SSD drive, and
leave my /tmp and swap partitions in RAID0 arrays on my rotational hard drives to reduce the number of writes on the SSD.

This setup has the benefit of using a very small SSD to speed up the main partition while keeping all important data on the larger mirrored drives.

Resetting the SSD

The first thing I did, given that I purchased a second-hand drive, was to completely erase the drive and mark all sectors as empty using an ATA secure erase. Because SSDs have a tendency to get slower as data is added to them, it is necessary to clear the drive in a way that will let the controller know that every byte is now free to be used again.

There is a lot of advice on the web on how to do this and many tutorials refer to an old piece of software called Secure Erase. There is a much better solution on Linux: issuing the commands directly using hdparm.

Partitioning the SSD

Once the drive is empty, it's time to create partitions on it. I'm not sure how important it is to align the partitions to the SSD erase block size on newer drives, but I decided to follow Ted Ts'o's instructions anyways.

Another thing I did is leave 20% of the drive unpartitioned. I've often read that SSDs are faster the more free space they have so I figured that limiting myself to 80% of the drive should help the drive maintain its peak performance over time. In fact, I've heard that extra unused unpartitionable space is one of the main differences between the value and extreme series of Intel SSDs. I'd love to see an official confirmation of this from Intel of course!

Keeping the RAID1 array in sync with the SSD

Once I added the solid-state drive to my computer and copied my root partition on it, I adjusted my fstab and grub settings to boot from that drive. I also setup the following cron job (running twice daily) to keep a copy of my root partition on the old RAID1 drives (mounted on /mnt):

nice ionice -c3 rsync -aHx --delete --exclude=/proc/* --exclude=/sys/* --exclude=/tmp/* --exclude=/home/* --exclude=/mnt/* --exclude=/lost+found/* --exclude=/data/* /* /mnt/

Tuning the SSD

Finally, after reading this excellent LWN article, I decided to tune the SSD drive (/dev/sda) by adjusting three things:

Add the discard mount option (also know as ATA TRIM and introduced in the 2.6.33 Linux kernel) to the root partition in /etc/fstab:

/dev/sda1  /  ext4  discard,errors=remount-ro,noatime  0  1

Use the noop IO scheduler by adding these lines to /etc/rc.local:

echo noop > /sys/block/sda/queue/scheduler
echo 1 > /sys/block/sda/queue/iosched/fifo_batch

Turn off entropy gathering (for kernels 2.6.36 or later) by adding this line to /etc/rc.local:

echo 0 > /sys/block/sda/queue/add_random

Is there anything else I should be doing to make sure I get the most out of my SSD?

Manipulating debconf settings on the command line

2010-10-19T07:30:00.000+13:00

It's not very easy to find information on how to adjust debconf settings after a package has been installed and configured. Most of the information out there is for Debian developers wanting to add support for debconf in their maintainer scripts.

I ran into the problem of being unable to change a package's configuration options through dpkg-reconfigure and I found the following commands to do it manually:

debconf-show packagename

to show the list of debconf values that a package has stored,

echo "get packagename/pgsql/app-pass" | debconf-communicate

to query the current value of an option in the debconf database, and

echo "set packagename/pgsql/app-pass password1" | debconf-communicate

to change that value.

I'm not convinced that this is the easiest way for system administrators to manually lookup and modify debconf options, but that's the best I could find at the time.

Taking the max/min of two columns in PostgreSQL

2010-10-11T14:00:00.002+13:00

As part of a database view, I found myself wanting to get Postgres to display values from one of two columns, whichever was the largest.

My first attempt was:

SELECT id, MAX(column1, column2) FROM table1;

which of course didn't work because MAX() is an aggregate function which only takes a single parameter.

What I was looking for instead, is something that was introduced in 8.1:

SELECT id, GREATEST(column1, column2) FROM table1;

where GREATEST() (and its opposite: LEAST()) is a conditional expression.

Freedom in the Cloud Miniconf at linux.conf.au 2011

2010-09-25T21:02:00.003+12:00

Jon Phillips and I are putting together a Freedom in the Cloud Miniconf for linux.conf.au 2011.

If you are interested in submitting a proposal (5, 25 or 45 minutes), we want to hear from you!

The deadline is 22 October 2010.

Whether or not 2011 will be known as the year of the Federated Social Web remains to be seen, but the cloud is certainly not going away and we'd like to propose a Miniconf to cater to all cloud enthusiasts.

Specifically, we'd like to solicit proposals from two main groups. The first one is the hard-core technical cloud engineers who build their tech on Open Source software. The kind of people who know so much about tweaking their kernels and scaling their infrastructures that they could out-Google Google.

Possible topics could include: cloud storage, virtualization, high availability, security, cloud performance and infrastructure management.

The second group of people we'd like to attract are the free network services and federated social web people. These folks know that the growth of netbooks and smartphones has meant a great dependence on centralized network services. But as more and more computing moves into "the cloud", users and companies are losing direct control over their data and processes. They see a very high risk to businesses and individuals.

This group would cover ways that individuals and organizations can ensure that their cloud computing is as dependable and freedom-respecting as running Open Source software on computers they own and control. They will concentrate on using Open Source, Open Data, and Open Standards to keep control in the hands of cloud computing users.

Speakers will vary from theoretical discussions about Free network services; developers of Free and federated alternatives to popular centralized cloud services; and tutorials on how to build an Open Source cloud network that you and your business can depend on.