flexo: The Tilley Hemp Hat (Default)
I know I'm a bit behind with my monthly updates (August was a big month and I have been busy throughout September too) but I like noting down code help things as-and-when.

So, I have a PyQt4 application (and presumably this is an issue for a regular Qt program too) that doesn't like being X-forwarded - I just get a grey window that never renders, and the following errors on the console. xclock and xeyes work fine.

X Error: BadAccess (attempt to access private resource denied) 10
  Extension:    130 (MIT-SHM)
  Minor opcode: 1 (X_ShmAttach)
  Resource id:  0x1e0000b
X Error: BadShmSeg (invalid shared segment parameter) 128
  Extension:    130 (MIT-SHM)
  Minor opcode: 3 (X_ShmPutImage)
  Resource id:  0x1e0000a

Turns out I had to tell Qt to use the "native" (X11) graphics system:

QT_GRAPHICSSYSTEM=native DISPLAY=:10 python curses_smels.py

Apparently this needs Qt 4.7 minimum.
flexo: The Tilley Hemp Hat (Default)
At work at the moment I'm rewriting a parsing routine for a C codebase, which currently uses Lemon, a LALR(1) parser generator. Lemon's great for converting a grammar into a nice routine for your code to call, but I've found the grammar has quickly expanded to a near-unmaintainable state to cover all our use cases.

As part of an abandoned rewrite of the parser, I replaced the hand-coded tokenisation with the Flex lexical analyser, which I found I rather liked, even if its lack of Unicode support might leave me having to rewrite that part by hand at some point.

With Lemon out of the way, I realised that it might not be a bad idea to move prototyping my self-devised algorithms (later superceded by the rather nifty Shunting Yard algorithm) and processing code into Python rather than C to speed up development. However, Flex doesn't provide any Python bindings.

Enter ctypes )
flexo: Good ol' footface (eleventh doctor)
At work I'm learning Django to make a quick demo site, something I've been meaning to do for some time.

For context, I was brought up with Turbogears 1.0, and my native environment generally centres around CherryPy, Genshi, and running as far away from SQLAlchemy as possible (fortunately working with the company's own search engine negates the need to interact with SQL too much).

My first impression was how easy Django was to install. I opted for a virtualenv automatically, being used to TurboGears and Pylon's massive dependency list dragging in most thing under the sun. However, while a virtualenv is always a good thing while developing, I was delighted that Django is just a single package. Having spent an afternoon recently trying to track down moved, deprecated, and obsolete packages to try to get an old but maintained TurboGears 1.0 project installed on a new server, it's reassuring to know that in a few years time I'll only have to hunt down one package should the latest Django at that time be incompatible.

Django has an equivalent to TurboGears' tg-admin script in the form of django-admin.py, which does the job. I don't have a problem with not using Paste to start a project, although I prefer CherryPy's five-line write-it-yourself philosophy.

While their templating language is apparently swappable (thank goodness; their own one is massively inferior to genshi in my preference) it looks like they've gone for the Routes approach of regex-style dispatching rather than attribute-based lookup which makes CherryPy a joy to use. And unlike Pylons, they make you write actual regexes in Python's clunky syntax, which makes it even less fun.

The different terminology to everything else is a bit weird. They have a FAQ regarding their terminology, which I assume is cover for "we didn't know the terms when we started the project and we're damned if we're changing them now". With regards to adherence to the Python style guide, they're not bad, but HttpResponse really bugs me; "HTTP" is even a specific example in there!

Finally, I was rather annoyed to find that I was unable to pull up online help very easily to find out what SQL field types were available. I wasn't able to even import the module without using django-admin.py shell and then help(django.db.models) doesn't actually show any field types. They must be hidden elsewhere...

Anyway, overall I think we're going to get on just fine, but I think I'll continue to do with CherryPy and Genshi in the future for new projects. Ultimately, I suppose, it's a matter of what you're familiar with and used to.
flexo: The Tilley Hemp Hat (Default)
Ah, SQLAlchemy, my old nemesis. I'm still stuck using SQLAlchemy 0.4.x/0.5.x at work, so I don't know if this has changed in later releases, but I hope so. Whereas things in Python like file objects will automatically close themselves when they go out of scope, something like this will leave hanging connections open:

code )

Now, this is annoying to debug because rather than getting an error immediately, you find out about it sometime in the future when the QueuePool queue exhausts all of its available connections and raises an error on some future unfortunate self.engine.execute().

Fortunately I can edit SQLAlchemy's code pretty easily, but throwing an error (my usual way of figuring out where code is misbehaving) when connections are established doesn't work due to the queue semantics above. After that, traceback.print_stack() is pretty useful to hack in, but the stacks end up huge through SQLAlchemy's internals, and I had a lot of database reconnections going on with all those engine.execute()s around.

So I decided to write something that would tell me where in my code the connections were established and closed to see where my leak was. The following code shows a self-contained file that demonstrates what's going on:

code )

Essentially, I'm creating a stack trace that only shows the lines of the stack that I'm interested in.

The relevant part of SQLAlchemy to add these into are in pool.py's QueuePool methods do_get() and do_return_conn().

With this extra output it was relatively easy to spot which lines of my code were checking out more connections than they were putting back, and I found the bug quickly after that.

Anyway, I thought it was a neat hack and I'll probably end up using it again sometime, so in the blog it goes.

Edit: It seems that engine.execute(...).fetchall() does actually close the connection, whereas fetchone() doesn't (SQLA 0.5.8). Consistent!
flexo: The Tilley Hemp Hat (Default)
What with Google Code Search having been closed down, and the Pylons documentation largely broken since that project merged into Pyramid, it took a few leaps of faith on my part to figure out how to set up profiling on a project using the repoze.profile middleware.

from repoze.profile.profiler import AccumulatingProfileMiddleware

# ...

def make_app(global_conf, full_stack=True, static_files=True, **app_conf):
    # Configure the Pylons environment
    load_environment(global_conf, app_conf)

    # The Pylons WSGI app
    app = PylonsApp()

    #Profile the app
    app = AccumulatingProfileMiddleware(

    # Routing/Session/Cache Middleware
    app = RoutesMiddleware(app, config['routes.map'])
    app = SessionMiddleware(app, config)
    app = CacheMiddleware(app, config)

    # CUSTOM MIDDLEWARE HERE (filtered by error handling middlewares)
    # ...

Then go to http://localhost:5000/__profile__ for the stats. It looks like this might be rather helpful as of writing, but we shall see now I have it set up.

Edit: You'll need python2.6+ for the pstats module, which caused me some trouble (yes, I know it's 2012), but seems to have paid off. I've already got some nice juicy stats that have exposed at least one function that's taking longer than ideal.
flexo: The Tilley Hemp Hat (Default)
Another code post, but don't worry, I have things to talk about for my regular update too.

I've been trying to set up a captcha field - specifically Google's ReCaptcha - on a not-as-old-as-you'd-think Pylons project which uses ToscaWidgets for rendering forms. I'd link you to Pylon's page as well but it merged with Repoze and apparently maintenance is hard. I'm pretty frowny about this but that's for another day.

On top of that, the 0.8 release of tw.recaptcha, which is supposed to make it easy to add recaptcha into ToscaWidget forms, didn't work, due to two things:

* You have to give it the remote IP in the Validator's constructor, while the Validator needs to be created before you know that.
* Formencode (apparently) validates every form twice for some reason, and of course this doesn't work on things like captcha where the correct answer changes per request.

Anyway, someone kindly left a comment on the wiki page for tw.recaptcha and although Google managed to mangle it by forcing WikiMarkup on people, I managed to reconstruct it into something that works. Below is a diff that can be applied to the 0.8 release in Pypi and makes it approximately 20% cooler.

Diff and code below )
flexo: The Tilley Hemp Hat (Default)
We interrupt this journal to bring you an important productivity-related update!

I've been vaguely trying to figure out how to display text/calendar ical meeting invite attachments in mutt, which I receive from colleagues sending requests from their fancy-pants graphical email clients. If you've looked at a vCalendar/iCalendar file you'll know it's not a very pleasant thing to read, especially since Outlook seems to like sending invites relative to Eastern Time from and to people located in the UK.

Outlook used to send a text summary along with the attachment so this hasn't been particularly high on my priority lists until lately, now that both the iPhone and the latest versions of Outlook simply send a blank message (in both text and HTML parts!) with the invite attached.

I found a collection of Ruby scripts after a few attempts to get my head around a couple of Python vCalendar-parsing libraries without much luck. The viewical.rb file included in that package worked fairly well, but had a rounding error causing HH:30 to turn into HH:29 and didn't compensate for Outlook insanity, so I've fixed that (at least, during daylight savings, we'll have to see what happens after that). The package from the page above also provides some utilities to Accept/Decline meeting invites, but I don't have any personal use for them at the moment so I haven't set them up. My copy of viewical.rb now looks like this:

#!/usr/bin/env ruby

require "rubygems" # apt-get install rubygems
require "icalendar" # gem install icalendar
require "date"

class DateTime
  def myformat
    (self.offset == 0 ? (DateTime.parse(self.strftime("%a %b %d %Y, %H:%M ") + self.icalendar_tzid)) : self).
        new_offset(Rational(Time.now.utc_offset - 60*60, 24*60*60)).strftime("%a %b %d %Y, %H:%M")
        # - 60*60 to compensate for icalendar gem/Outlook mismatch

cals = Icalendar.parse($<)
cals.each do |cal|
  cal.events.each do |event|
    puts "Organizer: #{event.organizer}"
    puts "Event:     #{event.summary}"
    puts "Starts:    #{event.dtstart.myformat} local time"
    puts "Ends:      #{event.dtend.myformat}"
    puts "Location:  #{event.location}"
    puts "Contact:   #{event.contacts}"
    puts "Description:\n#{event.description}"
    puts ""

Put that in your PATH somewhere, then put this in your .mailcap:

text/calendar; icalview.rb; copiousoutput

And this in your .muttrc:

auto_view text/calendar

Now whenever you view an email with a text/calendar attachment, mutt will display it in-line where the message usually appears (above the message, if there is one).

As an aside, it was frustrating to find out that although Google Calendar has an "Import Calendar" function on their frontend, there's no equivalent call in their API, so my idea to write an application to just pump the attachment directly to my Google Calendar account fell rather flat on its face.

Edit (2016-02-18): A more recent version of the Ruby icalendar package breaks the above script, and the time zone stuff is still broken in it anyway. In the meantime it looks like the offerings from the Python side of things have improved. The ics package seems to parse the files really well, including timezone information using the arrow library. I'm impressed. Here's the Python equivalent of the ruby script:

#!/usr/bin/env python
import sys

import ics # pip install ics
import arrow # pulled in with ics, else pip install arrow

cal = ics.Calendar(sys.stdin.read().decode('utf-8'))

tz = arrow.now().tzinfo
tzformat = 'ddd DD MMM HH:mm'

for event in cal.events:
    extra = {}
    for item in event._unused:
        if not isinstance(item, ics.parse.ContentLine):
            extra[item.name.lower()] = item.value
            print item
    print "Event:    ", event.name
    print "Status:   ", extra['status'].title()
    print "Organiser:", extra['organizer']
    print "Starts:   ", event.begin.to(tz).format(tzformat), "(local time)"
    print "Ends:     ", event.end.to(tz).format(tzformat)
    print "Location: ", event.location
    print "Description:"
    print event.description


flexo: The Tilley Hemp Hat (Default)

September 2017

101112 13141516


RSS Atom

Most Popular Tags

Expand Cut Tags

No cut tags