DoctorMo's Blog

Hello Community,

I’ve finished writing a new addon for xbmc (the tv media center for Ubuntu) called Librarian. She will take a look at your impressive video library and check for various inconsistencies and potential problems which you might want to look into. This 1.0 release includes the following features:

• List Movie Files not Included in the Database
• List Movies which have incorrect length, i.e. misidentified or corrupt (requires ffmpeg installed)
• Lists TV Shop Episodes separately with both above features
• Tells you which seasons and which episodes of each show you’re missing
• Shows you which TV Show Paths are being ignored completely.

More checking can be added as good ideas come in, I’ve also written an addon module called xbmcdata which wraps sqlite3 the xbmc httpapi to give a consistant inside xbmc and outside testing/scripting interface. This makes addon development _much_ easier. For instance listing movies is now just a case of:

from xbmcdata import Movies

for movie in Movies():
    print "%s (%s)" % (unicode(movie), movie['Year'])

Please download the code here: http://divajutta.com/doctormo/doctormo-xbmc-addons.tar.gz and install in ~/.xbmc/addons/ then test, entry is available under ‘Programs’. Please report issues back to me with full logs from ~/.xbmc/temp/xbmc.log included.

Have fun.

One of the fun things to do with python is to use the language to bend the rules of programming. One neat way of using your resources wisely with objects is to create objects in the right places, but not generate or call costly data gathering operations until you absolutely need to.

So called ‘late data initialisation’ is useful in certain kinds of programs. I document here the best pattern I’ve found to turn a python list into a late data class:

class newList(list):
    @property
    def data(self):
        if self.populate:
            self.populate()
        return self

    def populate(self):
        print "Generating Now"
        for x in range(42):
            self.append( x )
        self.populate = None

    def __getitem__(self, key):
        return super(newList, self.data).__getitem__(key)

    def __len__(self):
        return super(newList, self.data).__len__()

    def __iter__(self):
        return super(newList, self.data).__iter__()

Basically populate can be any piece of code which calls what ever it needs in order to gather the data. It’ll only be called once and to prevent having to use an __init__ variable we just clobber the method attribute. Sneaky!

When the list is used for an iteration (for loop), or if you delve into it directly for a specific item, or need to know it’s length, then we can get the data behind our object and have it contained within the list object we’re inheriting. No need for secondary list object variables dangling off of self. Ugly! Although this pattern does require that every use you want to put the object to (i.e. string, representation, slicing, dicing, mincing or conditionalising) you’ll have to make a new super method to wrap around to make sure that the data is generated if that’s the first way the list will be used.

What are your thoughts about this pattern? Do you know how to fix the init issue with a better pattern?

The Genetic Wallpapers project is supposed to be able to help any artist create a wallpaper capable of shifting and moving over time to produce interesting and unique results over time.

The Idea

The problem is that artists have all sorts of crazy ideas about what and how the wallpaper should progress and currently the logic for how the wallpaper will mutate is contained within the python code of the main project.

So I had this crazy idea, what if I could move that logic to javascript, a language which far more people know and each logic mechanism would be self contained within the svg wallpaper it’s supposed to effect.

I thought this was such a cool idea I rushed to google to do a search to find out how I could execute self contained javascript from python without a browser (because it’s svg, not html) and it only needs to be able to modify the Document Object Model to move and resize the objects in the image.

The Problem

All of the projects I encountered searching for a solution couldn’t really do the job. Either they were hacks, tied to web browsers, didn’t really work or were mostly converters to make javascript out of python. Which isn’t what we need for this to work. I looked at python-jswebkit, python-spidermonkey and pyv8, none of which could really do the job of running javascript to act on a DOM.

Possible Solution

Instead of trying to use python, I could farm it out to perl. I hear perl has better javascript execution support and best of all, documentation to show you how to do it.

The alternative is to ask the community. But this is quite a highly specialised bit of functionality, very rarely do we need to run one bit of JIT code on another. But any ideas would be very welcome.

Comments and thoughts on the idea and the direction, please to post below.

The GtkMe python lib provides a form window which allows me to construct a multi-page form with data validation. I’m pleased with the results I’ve committed today that allow a new user to register themselves within the login screen.

class RegisterWindow(TranslatableFormWindow):
"""Display a create user window"""
name = 'register'
primary = False
fields = [ 'postcode', 'age', 'gender', 'passphrase', 'computeruse' ]
def __init__(self, *args, **kwargs):
self.realname = kwargs.pop('real_name')
TranslatableFormWindow.__init__(self, *args, **kwargs)
intro = self.widget('label').get_label()
self.widget('label').set_label(intro % self.realname)
def cancel(self, widget, event=None):
if isinstance(widget, Gtk.Button) or (event and \
event.key.keyval == Gdk.KEY_Escape):
self.destroy()
def get_data(self):
"""Return all the user entered data"""
result = self.field_data()
result['name'] = self.realname
return result
def pre_exit(self):
register_service.create_user(self.get_data())


While I’m not finished debugging it all, the idea is to be able to make Gtk apps which have a wizard style layout with minimum effort. Each page has validation checks (if you want) and standard signals which you can stick into glade and have next, back, cancel and destroy handled for you.

This code controls a 5 page registration window as so:

Thoughts?

The scene is set, you have made a wonderfully useful dbus based service in python and you’re ready to serve interesting data in a desktop agnostic and thread safe fashion with fancy asyncronous calls and lovely event signalling. But woe! you have to write some tests¹ and you find quite quickly that this would require you to have a running test service and even that won’t make your tests work.

Before I move into my blog entry showing the world how I solved this really tricky problem, I want to say that I did do due diligence and did extensive searches for existing solutions, modules, examples and so on. The little I found and the mass of questions and difficulties people have had testing their dbus services in python is a tribute to how different the python testing framework is to the dbus server/client requirements.²

In a shared base module that is used by both daemon and client modules: example code

This gives you session bus, root address and service address and all will be correct in context. The client module is very simple for me, just some helper functions: example code

The daemon module is as you would expect, a bunch of dbus.service.Object classes with an extra helper: example code

Then you’ll need a few bits for the test suite, a test base module that all test cases inherit example, a daemon service script which is run when tests are run example and finally a test suite with some tests example³.

As you can see you need a running service with a test address and a way to start and stop that service if it’s not already running. You also need to manage the gobject loops for testing signals from your dbus service and this is done using the much loved python threading module.

I hope these examples will prove useful to others.

¹ Or had a requirement to write them prior to writing any code, depending on your method.
² Do post a comment below if you know of other people’s examples and successes.
³ I’ve hacked this together from my project code to give example, it may not run.

I managed to port one of my libraries that I made way back from perl to python, the experience was quite interesting, although very tiring. I don’t think I can recommend perl again since the python just looks so much cleaner and there were whole sections I could remove because they were handled by python.

If your interested in data validation check out lp:~doctormo/doctormo-random/xsdvalidate and have a look at the tests, it’s basically a module similar to the existing xml.xsd but with a focus on python structures.

Because it’s a library few of you are going to know how to run the tests and have a dig around at what it can do. For those in the know, it’s got a structural error reporting mechanism which allows the code to know what failed to validate. In perl I was using this module hooked up to a django like framework which would be able to check if the form information was valid and which fields failed and how, this information was actually fed directly back to the template for human warnings so the entire process was automated to a frightening degree, you really didn’t need any code for most transactions.

I’ve already heard that I should have used DTD instead of XSD, I never really liked DTD, they seem really ugly to me. Also a nod was given to using this kind of thing for ajax validation, but I somehow think that javascript must have an existing client side validator in the browser.

I have a test suite but I could always do with more tests, if you find something it’s not covering let me know.

I’ve making a generic gtk class which will allow me to set up threading for a nautilus python extension, the idea is to set off a new gtk main loop with a window (using gtk builder) and set off a thread. To bind them together I’m using gobject timers with stacks of calls which cross the boundaries.

So far it works and has improved the load speed and usability of the ground control project. The generic classes mean I can expand it and reduce code complexity.

What are your thoughts on this problem? Is this the right pattern to use?

I wrote this nifty script in python to take the output from the cal command and parse it, using an svg template it outputs an entire calendar in your own style, with your own pictures and everything.

It was a bit rushed because I was making a personal calendar for my wife with birthdays, anniversaries and our family pictures on it. And it came out really well too, she’s very happy with it! Here is a page from the calendar:

A big shout out to Inkscape, which again was flexible enough to allow me to create my calendar template without complaining about missing images or custom svg xml. If you want to have a go yourself at making a calendar then just download the following package:

calendar-creator.tar.bz2

Populate the flips directory with your own png files 01.png – 12.png and a title.png file for the front page, add any extra dates you want to the dates.lst, then run `./create-cal.py 2010` on the command line this will make a whole set of svg files for each page. You can then run `./make-book.sh` which will use inkscape (make sure it’s installed) to generate png files of each page.

Once you have your images, you can print them out in order or create a pdf of them using imagemagik’s convert command: `convert pngs/*.png full-calendar.pdf` but be aware this file might get big and generating these things takes time.

I will post a complete calendar tomorrow.

This is what python threading does to people:
My work on groundcontrol is trialling thanks to nautilus being based on gtk and the Thread class which is supposed to provide easy threading. And bzrlib, which may or may not be thread safe, but never actually functions in a thread even when self contained in one thread.

It’s turning into a real dogs breakfast and advice from other devels says to avoid threading in python… that may not be acceptable for this project. I’ve managed to solve now most problems via not having threads but instead loading at opportunistic times. but it’s ridiculous to me how badly Thread, gtk and bzrlib mix and how much time I’ve wasted trying different solutions. So desperate was I that I even tried using subprocess to fork out a separated script.

Now I know a lot of this is due to not knowing the precise operating functionality of threads in python… but the subject seems to operate on some kind of uncertainty principle where the more a person knows the less likely they are to explain how it works, conversely the less someone knows the more likely they are to try and explain it wrong.

So anyone know any _really_ good guides on python threading that explain it and demonstrate it? So far the guides I’ve found have been wrong.

Update: The community does not disappoint… a great simple guide in the form of a pygtk faq was most helpful, provided by M.K. Erlandsen below in the comments. Also useful is the multiprocessing explanation provided by Jess S.

Working with a nautilus extension and with bazaar is interesting, you can’t just open up the process of a bzr branch or pull and expect everything to be rosy. It’ll lock out the GUI and then you’ll be stuck without progress reports.

So I’m happy to report that I figured out how to make it work by using examples set forth in the qbzr project (QT-Bzr) so a lil bit of rewriting and a quick re-tooling using threading.Thread and we’re ready to start making sure all the dialogues and displays don’t lock up and deliver useful progress reports of the work they’re doing.

You can get the code branch I’m working on from lp:nautilus-lp

In unrelated news, I just got the results back from an x-ray, turns out I chipped one of the bones in my foot. So I’ve been walking around UDS for the last four days with a serious injury. You won’t believe the ear full I got off the doctor for assuming all that was rong was a simple sprain.