Django admin select count(*) from …

Shut up and just give me the code? Ok, look at the bottom of this post.

What is the slowest query on road.lk? It’s the select count(*) on the traffic alerts table that Django admin insists on executing each time someone visit the admin page for the alerts. Though we are fully operational only in Sri Lanka and Bangladesh at the moment, we do have twitter stream parsers running for all countries since a few months back. As a result, at the time of writing this blog post the traffic alerts table contains 762,779 entries. That’s a number that Postgres can handle easily without breaking a sweat; as long as you don’t do a select count(*) on it. In fact Postgres is not the only database that has trouble with these unqualified count queries. Mysql does even worse.

This cannot exactly be described as a bug but it certainly is an issue that has been plaguing django for five years. On the bug tracker it’s perhaps rightly categorized as ‘wont fix’, because to fix it would be to break the paginator which is used in most places. But this can be solved easily by caching the page nav tabs as a template fragment right? Wrong! If you overide the default change_list template, you will see that the paging section does appear to be cached, yet surpsingly, the debug tool bar and the pgsql slow query log reveals that the query is still being executed.

Looks like creating a custom paginator is the only way forward. In the end it turned out to be very easy. All that’s needed is to grab the code for the default paginator and change just a few lines of code.

Oct 6th, 2013 | Filed under Uncategorized

Jekyll as a replacement for SSI

This site does not use server side includes.  Banish the thought! However this site does use PHP as if it was SSI. The site with the LAMP was first created in 2002, at that time CMS weren’t as advanced as they are now. In spite of the millions of hours that have been invested in CMS by so many developers it’s still a load of crap, so you can imagine what it was like at that time (or you might have had the misfortune of using a CMS at the turn of the century). You can read my rant from 2005 here.

To cut a long story short, the part of the site that’s not under wordpress is created by using using PHP to each out the headers and footers and what not. Which means there is are a few lines of PHP code at the top of each page, a few lines a the bottom and in between it’s all HTML. Initially I toyed with the idea of using a regex to convert this mess to markdown, but regex sucks. It it dawned on me that modifying the PHP code that acts as the template to generate markdown instead of HTML would be far easier, and that’s exactly what I did.

Jekyll supports both pages and posts so the non blog part of the site can easily be converted to jekyll pages. As to how to deal with the WordPress powered blog; well dozens of experiments were done and it’s all covered here in the archives.

Apr 18th, 2013 | Filed under PHP

The lady finally sings

It aint over until the lady sings and she refused to do so because she wasn’t so happy with the commenting system. Initially I had tried to use Disqus but it had an unspecified error while importing from wordpress. Then I tried to use intense debate. Using intense debate is kind of like going back to wordpress but it didn’t happen cause their admin area is just pathetic. I simply could’t find the javascript to embed into the template for use with Jekyll.  So it was back to disqus and the unspecified error this time wasn’t there.  They probably had some site issues when I tried it out the first time.

Earlier I mentioned running across a plugin that did both categories and rss. Turns out that the RSS generated is for the  categories which I am not using a the moment and I am not sure if I want to add it either.  Besides Disqus comes with it’s own feed for each comment thread so it’s going to be very confusing to the visitor to figure out which RSS feed to subscribe to. The simple solution is to  just place a rss template file at the root level. Being lazy; I copied and pasted from this template.

Having done all this it was time to flip the switch – but on the photo blog. Yes, folks, all this time I have been concentrating on it because it’s so much simpler and doesn’t even have half the posts in this blog. The photo above is what it used to look like. Click through to the site, to see what it looks like now.

Mar 5th, 2013 | Filed under Uncategorized

Using a CF card as a disk buffer

A post that was originally written on Dec 28, 2010, but wasn’t published, but being released now to clear up the draft folder before the switch to Jekyll. Since the post was written dm-cache has in fact made it’s way into the main branch of the kernel.

Windows has something called ReadyBoost – us linux users have been having half of what Readyboost provides for as long as flash drives have been in existence. That is we have been able to use a flash drive as a swap file for a long long time which is essentially what ReadyBoost does. But there is something more.

Part of ReadyBoost is the ability to store some of the frequently used data on disk on a flash drive. In other words, the flash drive acts as a disk buffer. Again this is nothing new, the linux kernel automatically uses all available RAM as a disk buffer. When applications ask for more memory the buffers are reduced. MS DOS also used to have disk buffering systems but they fell by the way side when Windows came into being before making a come back.

What the mainstream linux kernel doesn’t have is the ability to use a flash drive as a buffer. Fortunately there is a third party kernel module called dm-cache that does provide this facility. What’s sad is that beaurocracy has kept it out of the mainstream kernel for close upon two years.

The fact that the module hasn’t been included in the main source tree means it hasn’t been maintained. So the version that’s downloadable from github is compatible only with kernel versions upto 2.6.29. Let’s see if there is a patch somewhere.

References: http://en.wikipedia.org/wiki/ReadyBoost

Mar 3rd, 2013 | Filed under Linux

SPA 3102 Getting Started

Still stuck here on wordpress, need to clean out the drafts folder before flipping the switch for Jekyll. There were 23 drafts in all Some more than six years old! Only one or two are worth preserving. Here is one, that was originally written on Nov 22, 2008 but never published. This was part of my Quadruple play at home scenario. This device in question was taken out by lightening some time later.

You can get started with the SPA 3102 telephone adapter by plugging in your phone and dialing ‘****’. Then you are connected to an IVR and you can make many changes such as define PPPoE connections etc. But one thing that you can’t do is to change the LAN IP address. It’s set by default to 192.168.0.1 .

My home network the IP address range is 192.168.1.0/255.255.255.0 so I had to create a virtual interface (eth0:1) and assign and IP in the 192.168.0.0 network before I could connect to it. Another option would have been to use the IVR to set the WAN port to obtain it’s IP address via DHCP but then I wasn’t sure if the device would run the web based admin panel through the external interface so I didn’t even bother to try. Later on I found that you can use the ’7932′ code in the IVR to enable/disable this feature. Much easier to create eth0:0 , login, change the LAN IP on the device to my network and then to delete the eth0:0 again.

Since my router is actualy my desktop (with to NICs connected to it). I asigned another IP in the 192.168.1.0 network to the external interace on the telephone adapter. The moment I did that my entire network stopped working and the switch started blinking like crazy. It took a me a few minutes even to figure this out. I eventually reached the conclusion that whatever packets that were coming in through the internal interface it was sending out again through the external interface int he mistaken belief that those packets should be NATed this leads to a flood. I suppose this is akin to the ‘howling’ you hear when a mic is placed in front of a speaker.

I was really struggling to get the voice adapter to connect to my Asterisk. It was only after a lot of frustration that I figured out that the Asterisk server’s IP has to be entered as in the proxy field (in the line 1 tab

Mar 2nd, 2013 | Filed under Linux

Batch minimize js with python and Closure

There maybe much better ways of  batch minimizing Javascript files. But this is my way and sharing it just in case someone else finds it useful.

 

import os, os.path
import glob
import re
import shutil

root = "static/js/";

fs = glob.glob(root + "*dev.js")
for dev in fs:
    if os.path.getsize(dev) == 0:
        print dev
        minimized = re.sub('.dev','', dev)
        # dev is currently empty and minimized has the non minimized version!
        shutil.copy(minimized, dev)

        sed = "sed -E 's/console.(log|debug|info|...|count)\((.*)\);?//g'"
        os.system('java -jar compiler.jar --js %s | %s > %s' % (dev, sed , minimized))
    else :
        print dev + ' - skipped'

The sed regex was lifted from a Stackoverflow answer. And the Closure Compiler can be downloaded from the google dev site.  As you can see from the code, during development the unminimized code is being used. Then prior to deployment the script will move that unminimized code to a different file, use that as input for closure, remove any console messages and put the minimized code back into the original file

Mar 1st, 2013 | Filed under Uncategorized

Finishing up Jekyll migration – this time for real

Yesterday’s goal was to finish off the migration from WordPress to Jekyll (a destination that was arrived at after a long journey), however there still was a couple of sticking points including paginated categories and tags. Thankfully I found a really well documented plugin which someone with only very elementary Ruby knowledge such as myself could understand at realjenius.com

The Realjuenius plugin is a Swiss army knife. It does categories, tags , RSS and includes pagination to boot. By the way do people still use RSS and atom? I see precious few requests for the feed in my log files but the saying is that you shouldn’t look a gift horse in the mouth. The rss feed comes at no extra cost, so keep it. Only a few teeny weeny mods were needed to make this plugin behave the way I wanted it to. One is that category names need to be downcaseD and special characters in category and tag names need to be escaped, but these are things that can be managed with my elementary ruby knowledge.

Now comes the most boring bit, working with the design. I am not too keen on preserving the existing design and I am not bothered about changing it either. So I am going to put myself firmly on the fence; adapt Twitter Bootstrap with a UI that looks pretty much like the existing design (which is based on the now unmaintained WordPress Magallen theme)

Feb 23rd, 2013 | Filed under Uncategorized

Jekyll finishing up

So it’s been an eventful journey; after deciding to do away with wordpress (this time for real), I started looking at static blog generators with Jekyll being the very first. Thereafter I investigated Hyde, Mynt, Punch, Pelican and a few other as well. All of them had their pros and cons but in the end, I settled on Jekyll.

Jekyll and wordpress do have something in common;  they are both completely useless without plugins. So far I have installed plugins to paginate the home page and create category and archives (yes, even making category pages needs a plugin). Now I need to paginate the category pages and also create pages for tags. So let’s look at the options available.

The first thing I tried was to modify Recursive Design’s category plugin based on Ben Curtiss archive generator plugin, after all the latter is based on category plugin. But it didn’t quite work out,  for some strange reason categories seem to get mixed up and doesn’t seem to be displaying in the reverse chronological order. Then I looked at couple of other category plugins including one on marran.com which didn’t produce anything at all for categories!. This lead me to the conclusion that these plugins expect the directory structure to be organized along category lines – ie subfolder for each category and blog posts placed inside. Simply having the category name in markdown may not be enough.

At this point, I must add that though I have learned and forgotten quite a few languages Ruby doesn’t happen to be in that list. I do have the ability to read and understand ruby code, thanks to a ‘learn ruby in 24 hours‘ adventure I undertook last year. At this point time, I don’t have the energy or the proficiency to wade through Ruby code and figure out what needs to be changed. But then, laziness is a virtue. If I install a tags plugin, this will be quite easy to figure out.

 

Feb 22nd, 2013 | Filed under Uncategorized