GPG with Confirmation

I got tired of Evolution email client giving me those horrid error messages when ever I try to email someone who’s key isn’t in my current list of keys.

The design of this is appallingly bad. It discourages the use of GPG rather than encouraging the importing of keys and it makes no mention of helping you acquire keys if possible. It also allows for no additional or optional footer to explain to the recipient that their message couldn’t be encrypted because they don’t use GPG.

While I couldn’t do much about the later, without hacking on the evolution codebase directly. I did do a bit of hacking on the former with a gpg middlware. Yes, when I say hack, I mean HACK. A dangerous and potentially devastating way of wrapping the gpg binary with my own python script that could intercept the evolution call and do work to search, display and add keys to encourage the use of encryption overall.

The design was simple. When we are asked to encrypt for a person who we don’t have the keys for, we do a search. The results are shown in a GUI to the user and they can select a key to use. This then is added to the key ring and used to encrypt the email.

This setup allows for experimentation with user prompting and workflow. It’s not something I would recommend be installed on user’s computers. But for designers and developers, this sort of match-stick making is a valuable platform to build, try, test and rebuild quickly.

I use zenity for the user interface. This is a Gtk command line tool that lets you launch a window from the command line and the interface is good enough to support photos in lists and returning which item was selected. Very cool.

Bellow you will find the script I created for this hack, this is saved to /usr/bin/gpg and gpg is moved to gpg.orig:

#!/usr/bin/python
#
# Wrap the gpg command to provide evolution with a bit of extra functionality
# This is certainly a hack and you should feel very bad about using it.
#
# Public Domain, Authored by Martin Owens  2016
#
import os
import sys
import atexit

from collections import defaultdict
from subprocess import Popen, PIPE, call
from tempfile import mkdtemp, mktemp
from datetime import date
from shutil import rmtree

to_date = lambda d: date(*[int(p) for p in d.split('-')])


class GPG(object):
    keyserver = 'hkp://pgp.mit.edu'
    remote_commands = ['--search-keys', '--recv-keys']

    def __init__(self, cmd='/usr/bin/gpg', local=False):
        self.command = cmd
        self.photos = []
        self.local = local
        self.homedir = mkdtemp() if local else None
        atexit.register(self.at_exit)

    def at_exit(self):
        """Remove any temporary files and cleanup"""
        # Clean up any used local home directory (only if it's local)
        if self.local and self.homedir and os.path.isdir(self.homedir):
            rmtree(self.homedir)

        # Clean up any downloaded photo-ids
        for photo in self.photos:
            if os.path.isfile(photo):
                os.unlink(photo)
            try:
                os.rmdir(os.path.dirname(photo))
            except OSError:
                pass

    def __call__(self, *args):
        """Call gpg command for result"""
        # Add key server if required
        if any([cmd in args for cmd in self.remote_commands]):
            args = ('--keyserver', self.keyserver) + args
        if self.homedir:
            args = ('--homedir', self.homedir) + args

        command = Popen([self.command, '--batch'] + list(args), stdout=PIPE)
        (out, err) = command.communicate()
        self.status = command.returncode
        return out

    def list_keys(self, *keys, **options):
        """Returns a list of keys (with photos if needed)"""
        with_photos = options.get('photos', False)
        args = ()
        if with_photos:
            args += ('--list-options', 'show-photos',
                     '--photo-viewer', 'echo PHOTO:%I')
        out = self(*(args + ('--list-keys',) + keys))

        # Processing the output with this parser
        units = []
        current = defaultdict(list)
        for line in out.split('\n'):
            if not line.strip():
                # We should always output entries if they have a uid and key
                if current and 'uid' in current and 'key' in current:
                    # But ignore revoked keys if revoked option is True
                    if not (current.get('revoked', False) and options.get('revoked', False)):
                        units.append(dict(current))

                current = defaultdict(list)

            elif line.startswith('PHOTO:'):
                current['photo'] = line.split(':', 1)[-1]
                self.photos.append(current['photo'])
            elif ' of size ' in line:
                continue
            elif '   ' in line:
                (kind, line) = line.split('   ', 1)
                if kind == 'pub':
                    current['expires'] = False
                    current['revoked'] = False

                    if '[' in line:
                        (line, mod) = line.strip().split('[', 1)
                        (mod, _) = mod.split(']', 1)
                        if ': ' in mod:
                            (mod, edited) = mod.split(': ', 1)
                            current[mod] = to_date(edited)

                    (key, created) = line.split(' ', 1)
                    current['created'] = to_date(created)
                    (current['bits'], current['key']) = key.split('/', 1)
                elif kind in ('uid', 'sub'):
                    current[kind].append(line.strip())
                else:
                    current[kind] = line.strip()

        return units

    @property
    def default_photo(self):
        if not hasattr(self, '_photo'):
            self._photo = mktemp('.svg')
            with open(self._photo, 'w') as fhl:
                fhl.write("""
  
""")
            self.photos.append(self._photo)
        return self._photo

    def recieve_keys(self, *keys, **options):
        """Present the opotunity to add the key to the user:
         
        Returns
          - True if the key was already or is now imported.
          - False if keys were available but the user canceled.
          - None if no keys were found within the search.

        """
        keys = self.search_keys(*keys)
        if not keys:
            return None # User doesn't have GPG

        # Always use a temporary gpg home to review keys
        gpg = GPG(cmd=self.command, local=True) if not self.local else self

        # B. Import each of the keys
        gpg('--recv-keys', *zip(*keys)[0])

        # C. List keys (with photo options)
        choices = []
        for key in gpg.list_keys(photos=True):
            choices.append(key.get('photo', self.default_photo))
            choices.append('\n'.join(key['uid']))
            choices.append(key['key'])
            choices.append(str(key['expires']))

        if len(choices) / 4 == 1:
            title = "Can I use this GPG key to encrypt for this user?"
        else:
            title = "Please select the GPG key to use for encryption"

        # Show using gtk zenity (easier than gtk3 directly)
        p = Popen(['zenity',
            '--width', '900', '--height', '700', '--title', title,
            '--list', '--imagelist', '--print-column', '3',
              '--column', 'Photo ID',
              '--column', 'ID',
              '--column', 'Key',
              '--column', 'Expires',
            ] + choices, stdout=PIPE, stderr=PIPE)

        # Returncode is generated after communicate!
        key = p.communicate()[0].strip()

        # Select the default first key if one choice.
        # (person pressed ok without looking)
        if not key and len(choices) == 4:
            key = choices[2]

        if p.returncode != 0:
            # Cancel was pressed
            return False

        # E. Import the selected key
        self('--recv-keys', key)
        return self.status == 0

    def is_key_available(self, search):
        """Return False if the email is not found in the local key list"""
        self('--list-keys', search)
        if self.status == 2: # Keys not found
            return False
        # We return true, even if gpg returned some other kind of error
        # Because this prevents us running more commands to a broken gpg
        return True

    def search_keys(self, *keys):
        """Returns a list of (key_id, info) tuples from a search"""
        out = self('--search-keys', *keys)
        found = []
        prev = []
        for line in out.split("\n"):
            if line.startswith('gpg:'):
                continue
            if 'created:' in line:
                key_id = line.split('key ')[-1].split(',')[0]
                if '(revoked)' not in line:
                    found.append((key_id, prev))
                prev = []
            else:
                prev.append(line)
        return found


if __name__ == '__main__':
    cmd = sys.argv[0] + '.orig'
    if not os.path.isfile(cmd):
        sys.stderr.write("Can't find pass-through command '%s'\n" % args[0])
        sys.exit(-13)

    args = [cmd] + sys.argv[1:]
    # Check to see if call is from an application
    if 'GIO_LAUNCHED_DESKTOP_FILE' in os.environ:
        # We use our moved gpg command file
        gpg = GPG(cmd=cmd)
        # Check if we've got a missing key during an encryption, we get the
        # very next argument after a -r or -R argument (which should be
        # the email address)
        for recipient in [args[i+1] for (i, v) in enumerate(args) if v in ('-r', '-R')]:
            # Only check email addresses
            if '@' in recipient:
                if not gpg.is_key_available(recipient):
                    if gpg.recieve_keys(recipient) is None:
                        pass
                        # We can add a footer to the message here explaining GPG
                        # We can't do this, evolution will wrap it all up in a
                        # message structure.
                        #msg = sys.stdin.read()
                        #if msg:
                        #    msg += GPG_TRIED_FOOTER
                        #sys.stdout.write(msg)
                        #sys.exit(0)

    # We call and do not PIPE anything (pass-through)
    try:
        sys.exit(call(args))
    except KeyboardInterrupt:
        sys.exit(-14)

Short Text Alert Notification (STAN)

I recently got a new phone, and a new place to keep it. My previous phone was damaged when I sat on it and realized that keeping it in a back pocket was a bad idea. I’m keeping my new phone in a belt pouch with a Velcro seal and it takes me slightly longer to get at the phone than it would if it was in my pocket.

And so I was, a few days ago, messaging people using SMS and then putting my phone away. Several times my interlocutors would send me short affirmative messages which I would have to read by pulling out my phone from it’s pouch, turn it on with it’s (not always first try) security and then read a message that says “OK” or “Yes”.

So I thought; why not have the phone change it’s notification sound for simple messages? A table of possible short messages which would then translate into a different notification sound and allow the user to understand a simple message without having to pick up their phone, unlock it and read the message.

This would certainly make sending short messages slightly more polite as it can seem aggravating sometimes to have to read short messages. Even though it’s not the fault of the sender really.

  1. The idea would require a customized default messaging app for android, or possibly system configuration, I’m unsure how notification sounds are stored.
  2. A research stage where short messages on multiple phones was collected into a corpus of English messages. Each classified into what it probably means.
  3. An additional corpus of emoji messages that can be easily converted into sound. Example a kissing emoji into a kissing sound.
  4. A restriction on the time between the previous message to/from the sender and the notification. Say half an hour. This is to prevent an affirmative messages which arrive a week after the last correspondence from confusing the recipient.
  5. A secondary research phase where a group of people’s would have the app installed (replacing their messaging app) and would self report if the new notification sounds made any difference over a couple of months of use.

This is a sort of imaginary plan, but I could see this being useful. But the only way it would be come mainstream is if it was adopted by Google directly into it’s own messaging app. A stepping stone towards that would be adoption of the modified message app into after market android such as the Cyanogenmod project. The stepping stone would certainly allow more data to be collected about it’s functional use out int he wild.

Another alternative stepping stone would be to approach a manufacturer or telco network. But quite often apps these layers install are seen as bloat-ware and it might be better to avoid that route if possible.

What do you think? Good idea? bad idea?

What is Art? is code Art?

The musings of today’s Thought for Today on BBc Radio Four are often interesting perspectives that drive at something both personal and social. Today’s subject was the concept of modern art, it’s valuation and the way in which artists invest in the art while knowing little about it.

This got me thinking about code. You see code is something that requires an imense amount of creative thinking. Not just problem solving and puzzle mastery; but down right honest to god design and humble craftsmanship to boot. A piece of code must be more than just functional for the user, it must be maintainable in an ever changing world.

This requires that the code be readable and possibly even attractive to potential maintainers as a learning exercise. The best code is obvious where is can be and smartly presented where it needs to be clever. It must deliver it’s cleverness carefully and in reasonable chunks, much like a classical lesson in latin or a course of antibiotics. The code needs to cozy up to the reader and be as familiar with it’s patterns, syntax choices and variable naming conventions as a well worn pair of slippers.

Start using single letter names, odd abbreviations or inventing undocumented artifice and you’ll lose the audience. You’ll alienate the future from your comfortable seat in the past with a smug sense of converse hindsight. The arrogant developer assumes all things are known in the future and all maintainers are themselves or someone very much like themselves. And the trouble with people is, no matter how many you know, there’s always one strange outliers you’ve never met and one day they’ll be looking at your code thinking to themselves that you must have been enjoying your legal high quite a lot on the day you wrote /this/.

So what is art and how does it fit into this whole “understandable code” thought?

I’m not going to pretend there’s not seven billion ways to define art. But I believe art to be “the intentional communication through emotional language”. This means I consider stand up comics to be artists, I consider Fox news to be an arts show and music like rap to be one of the most powerful forms of art around today. But art can be bad like Fox, art can be good like Banksy and that doesn’t detract from it’s medium.

Art can be a failure when it fails to deliver the intended emotions like most modern visual art (to the general population anyway). We can feel disappointed in politicians for failing to be concise and factual, while at the same time marveling at their artistry for using their home spun bull shit to evoke the emotions they want in their audience. It’s wonderfully successful art, and a terrible education for the public. Not that art needs to be true, or that it needs to not be true of course.

Code in this narrowed definition of art, can be art. Sure as above we really want code to be artful as in crafted well; but we also could have code that intends to and successfully delivers an emotion. It has two ways. The usual way is that the code runs a game or some other intended visual art say. It’s the mechanism by which art is delivered and the code in there is part of the whole art.

I remember the radio head “Big Ideas” video that uses a specrum and hard disk array. That delivers art through it’s code is some interesting ways.

But I think most interesting to developers is how their emotions can be engaged by just reading code and repositories. I think source code poetry is a pretty well established way of making art out of code and I really enjoy reading some of it and running it. There are code flowers and other clever mechanisms that evoke wonder and joy as they are compiled and run.

But what of every day code. I think all our code evokes some emotion in those that have to read it and fix it. Mostly this is frustration and annoyance that you didn’t write it in a way more comforting to the reader. But there’s got to be scope here for making functional code that’s beautiful, interesting, passionate, lovely, hateful or just plain fun.

And not just for the user.

What do you think? Can your code be art?

Using PhantomJS with django for d3 testing

I’ve been working with phantomjs recently. It’s a javascript based system wrapper around a webkit backend. This allows one to make requests against a website and run the client side javascript, and make screenshots of the way the webpage would be generated including all css.

I’m using it to generate d3 graphs for testings whereby images of the webpage as saved and compared to expected images.

Of course I’m using python since my websites are all in python and usually django these days. So calling a javascript command and pointing it at a running webserver sounds like a really hard task. But actually django provides a live TestCase which runs a test web server which phantomjs can use as it’s source. It even includes fixtures and other test features which makes it fairly easy to put together the basics of what’s needed.

import logging
from subprocess import Popen, PIPE
from django.test import LiveServerTestCase

class GraphTestCase(LiveServerTestCase):
# Tests and fixtures go here as well as login via self.client (as usual)

def phantom(self, filename, **kwargs):
"""Run a phantomjs script and return True if it was successful"""
pre_args = []

# These cookies require the target js to support them.
cn = settings.SESSION_COOKIE_NAME
if cn in self.client.cookies:
kwargs['cookie-' + cn] = self.client.cookies[cn].value

args = ['--%s=%s' % item for item in kwargs.items()]
cmd = [self.script] + pre_args + [self.get_js_file(filename)] + args

process = Popen(cmd, stdout=PIPE, stderr=PIPE, universal_newlines=True)
out, err = process.communicate()
if process.returncode != 0:
logging.info(out)
logging.error(err)
raise IOError("Process did not complete: %d" % process.returncode)
return out

The above code should be well formatted, but wordpress (or this version of it) doesn’t want to use the right tags for code. :-/

Django Projects

I haven’t posted in a two years. I don’t know if anyone is still following my blog or not, but if you are. Hello and thanks for your patience.

These last few years I’ve been doing a lot of django contracts. It seems like every man and he’s dog business wants a django website to do complicated things involving user data. And django has matured in the last few years to become more sensible and consistent.

I remember back in 2009 talking with a developer about migrations support and how django had really let us all down by not really having it’s own migration framework built in. Now though, we have what is a pretty good (but not perfect) migration system.

These are some of the things that I have to do for many of the django projects I work on.

  • django-autotest – A way to run the testing framework over and over again without re-creating the database schema, auto rerun on file changes, shell title bar status updates, some nice base test extras like getting objects, making requests and setting up sessions which often crop up in test code. (include useful haystack base tests too)
  • django-fast-fixtures – A wrapper for dumpdata and loaddata to have media in fixtures. Useful for those statuses with icons and those tests with media files.
  • cache middleware – Caching middleware. When using django-cms, it’s VERY important to use the full page cache. Without it, your site is dead (or at least inkscape’s website would be) I’m still working on the expiration logic for all object types.
  • django-cats – I’ve been trying to convert my haystack/django category and search view into it’s own module. So far, it’s in a collection module I’ve used with multiple projects. The basic idea is that you can make a view that can slice up a queryset into multiple categories and haystack searches. Categories provide counts and easy to display and link iterators for templates. As well as the handy automatic haystack attribute creation and url key word support.
  • settings and admin tweaks – Most sites need a local settings template. This is because django doesn’t yet seperate out developer configuration from sysadmin configuration. So we still have to do it manually. It also doesn’t allow admin sites to be designed/changed, these two are taken care of in these standard patterns.
  • urls.py – And lastly is url_tree, a two line addition that ends up in almost all of my urls.py files as it’s just so useful. Take a look at the complexity of this urls.py, it’s made a lot more understandable by reducing the three calls into one simple call. Sometimes I tweak it to allow the view module name in too, but it’s mostly never needed.

What have you been up to in the last few years?

Python Crontab 1.4 Released

Today I’ve put the finishing touches on python-crontab 1.4. This release updates the tests and fixes a couple of bugs as well as two really interesting features.

The first is the ability to pull a log from a crontab or cronjob. When creating a crontab in the usual way you can specify a optional log file, if not specified the value ‘/var/log/syslog’ is substituted. The log attribute then provides you a filtered log view which limits the lines returned by the iterator to those attributed to the user your crontab controls. This is of course by default the ‘root’ user, but can be other users too.

Conversely each job in the crontab can return a log iterator which limits the retuned entries to those for this job only. This allows the programmer to tell when cron jobs were last run.

The second large feature is the scheduler. Using the croniter python module, this feature returns date/times when cron jobs would run from any given datetime offset. This allows the programmer to compile a list of datetimes when the job would run in the future and in the past. Use the schedule attribute from the cron job object.

You can download the new release here: python-crontab-1.4 on PyPi

XBMC Frodo JsonRPC Snippet

This is an example script for python’s jsonrpclib and XBMC Frodo which has an updated rpc library which is finkey about the headers and testy about usernames and passwords. Recorded here to help others in search of a solution.

If you get http errors 415 Media type not found, json decode error, or 401 Not authorised, you need this:

#!/usr/bin/python

from jsonrpclib.jsonrpc import TransportMixIn, XMLTransport
from jsonrpclib import Server

class Transport(TransportMixIn, XMLTransport):
    """Replaces the json-rpc mixin so we can specify the http headers."""
    def send_content(self, connection, request_body):
        connection.putheader("Content-Type", "application/json")
        connection.putheader("Content-Length", str(len(request_body)))
        connection.endheaders()
        if request_body:
            connection.send(request_body)

server = Server("http://username:password@localhost:8080/jsonrpc", transport=Transport())

print server.VideoLibrary.Scan()

On Ubuntu you can install jsonrpclib via pip as it’s not packaged as a deb.

Developer Kits

With the new Ubuntu Phone on the horizon I though I might pen something about developer kits. I think we’ve come to the group-conclusion that development on Ubuntu sucks, and it sucks in a few ways I’m going to ramble on through:

Coherence

The idea of Quickly was to make a few opinionated decisions about how programs should be made. But it seemed to very quickly delve into the empire of choices on libraries, languages project management and deployment strategies. And that’s a reflection on how hard it is to please everyone in the open source universe.

If we want to have a good developer kit, we need to have strong, very strong opinionated decisions which nail all the technical functionality that will be required for all sorts of apps. For the Ubuntu phone, this looks like Qt and OpenGL. For the desktop it looks like C/Vala/python and Gtk.

What we don’t have is a good story for the project management and deployment. It looks like deployment is going to be patched up with automated package testing and an automated security system. I’m not sure about project management on the developer’s work station yet, it’s still very ad-hoc with too much of the best workflow in the minds of the developers and not in the tools being used.

Investment

The developer desktop, tools, apis and workflows require an enormous amount of investment. Look at how much money and time Google has spent just on Android developer tools.

It’s practically another Ubuntu sized project. While traditionally we’ve had very little investment and have used whatever tools people have cobbled together as they went along with their own developer needs. It’s produced a very disjointed experience for new developers, where so much needed to be known in order to be a good developer.

That’s turning round with the new developer site and api documentation. but I’m not sure the size of investment is the right sort of scale for where we want to go with app development. Bringing together all the API documentation from our various libraries would go a long way towards centralising and authenticating the previous opinionated decisions.

The Klein Toolbox

It’s hard to know where a toolbox starts for app developers and ends for desktop/core developers. If you think about tools like bzr and Make; these tools are designed for core developers and not really intended for app developers. They require the developer to understand where to get the best workflows and be pre-trained. And this really means that as soon as a developer is capable of using these tools, they’re no longer capable of making good developer tools for app developers… how can someone be sympathetic to the needs of more casual developers when they were required to learn so very much?

I’ve never liked Makefiles, I don’t think they’re good enough and I’m damn petty about all the tools that generate makefiles too. I don’t see good design in that space and I know paultag disagrees with me quite strongly about how unawesome makefiles are.

Conclusion

We have a lot of work to do.

What do you conclude?