How to make a Python script run as if it was started in interactive mode

If you make little utility scripts in Python that prepares some kind of a work environment, and you then want it to run as if you had started the script with ‘python -i’, do the following:

import os
os.environ['PYTHONINSPECT'] = '1'

After exiting the script, you will be dropped into an interactive Python session.

I made a small script that lives in my /usr/local/bin named aws that imports the boto libraries and sets up a connection. When I would like to interact with some files in S3, I run that script to get the job done.

Making better auto-complete INPUT text fields

The first time you see an AJAXy drop-down auto-complete text input field as a web developer, you just have to make one yourself. It is a compulsion. But then you make it, and it seems clunky. Here is a tip to improve the usability: use the current word your cursor is positioned on to retrieve the list of suggestions from your database, and not the entire text field.

I did this yesterday and am delighted with the results. You can see how it works on the Arkyves welcome page, just type some words in the search box in the top left-hand corner of the page. You will notice how the suggestions are based on what word your cursor is positioned at. Nifty, eh?

Here is the crucial snippet of Javascript to make this happen:

(pasting this code stripped the angle brackets, I really need to figure out a better way of posting code snippets. Grrrr. I will leave this snippet below for now, but please note that you can’t just cut-and-paste it and expect it to work. )

getSpaceChunk = function(textElem) {
var v = textElem.value;
if (v.length s1) {
return v.substr(s1, (s2-s1))
}
var startSpaceFound = 0;
for(var i=0; i= s1) break;
if(v.charAt(i) == ' ') {
startSpaceFound = i+1;
}
}
var lastSpaceFound = v.length;
for(var i=startSpaceFound+1; i<v.length; i++) {
if(v.charAt(i) == ' ') {
lastSpaceFound = i;
break;
}
}
return v.substr(startSpaceFound, (lastSpaceFound-startSpaceFound));
}

If someone has suggestions how this function can be improved, ping me. I don’t claim to be a Javascript guru. And credit where credit is due, I first saw this kind of niftyness on del.icio.us, so they inspired me to fix mine.

Machine Massage

The past few weeks I have actually been working way too much. The start of the year was great, I was on a roll, and things were dandy. And then I did some stupid things and created way too much extra work for myself. Net result? Very stiff shoulders and neck from sitting behind a desk excessively.

Normally I go and lie in the sauna and get a massage to help with the muscle pains. While getting a breath of fresh air today I walked by a hairdresser with a sign in the window advertising ‘chair massages’ for a very cheap price. Well, seeing that it was just my shoulder, I figured: “Why not?”

Turns out that it is really a chair massage, not a person doing it but a mechanical device. You sit down the buttons are pushed and whir-whir-whir you are pummeled. The very firm grasping of the legs was alarming at first. I had visions of science fiction movies where the machine runs amok. But it was enjoyable.

Better than a proper massage by a real person? Not sure. I will have to go back and try again. But the mere fact that there is doubt is a good sign.

Better than a poke in the eye with a sharp stick

I first heard that phrase from an illustrator that I was collaborating with many moons ago. We had some plans to make a computer game, he was also into motorbikes and told me tall tales. Not really knowing what it meant, I did like the turn of it, and have used the phrase ever since.

But now I know where it comes from, thanks to Arkyves. It is a scene from one of the Greek heroic legends (Homer, Odyssey IX) where Ulysses blinds Polyphemus the Cyclops.

Kind of a bummer if you only have one eye, and it gets poked out.

Image from ‘Emblematum liber’ by Alciatus. For more information have a look at the entry on the excellent emblems site of Glasgow University. (for which I made the thematic browsing 😉

Iterating through a PyLucene index to find problems with sorting

Doing some sorting of results using PyLucene produced the following error:

JavaError: java.lang.RuntimeException: there are more terms than documents in field "UID", but it's impossible to sort on tokenized fields


Oh bugger. And I had upgraded my server especially to the latest version of PyLucene to be able to do sorting (which segfaulted in the version which I had compiled myself for some reason).

This error is apparently triggered when you have multiple fields in a document with the same name. For something like a UID, I could not recall how my documents could have multiples, this needed some investigation.

So it took me a while to figure out that I needed a snippet that looked something like this:

i = 0
while i < READERS.maxDoc():
dflds = READERS.document(i).getFields(UID)
if len(dflds) > 1: print i, dflds
i += 1

And yes, many of my document did indeed have multiple fields added for the UID. This is obviously an artefact of some old code used during indexing. The only real solution is now to re-generate the index for the entire database.

This is very time-consuming as there is some extensive ICONCLASS mojo needed per document. As I need to add some other features to the indexing code anyway, I might as well do those and then re-do the index is one go.

This will take a while.

PS. the code snippet above has the indentation stripped out by WordPress. Grrrr. I need to figure out how to fix that, and while we’re at it, some syntax highlighting would be nice too. Can anyone suggest a fix, Lazyweb?

Hiding ICONCLASS codes from users

To users of websites who are not experienced with ICONCLASS, the notations (or ‘codes’) are just confusing.

While it is the best way to do very precise searches in large databases like the Bildindex how do you find the correct code to use?

This post on Livejournal reminded me to say something about the use of ICONCLASS codes in searching.

One way is to do a search on www.iconclass.org/browse and then copy and paste the chosen code into the Bildindex Expert search
web form. This is however unwieldy and error-prone, and usually doesn’t give satisfactory results. For a start, it can be a pain to find the code in the first place. Since late 2006 it has been possible to do full-text searches in English, German, French and Italian on the iconclass.org to find the required code, but this is still not the ideal way to do searches. A further problem is that for example at Bildindex they use/store the codes in ‘German’ format, which means a code like

34B114 walking the dog

needs to be entered in the format

34 b 11 4

to yield useful results.

The better way

It is far better to completely hide the codes from the user, and do the code selection and searching ‘in one go’. This is the approach that we use in Arkyves. If you were looking for dogs in the bible, that is what you type in the query box: ‘dog bible’, or ‘dog mythology’ as a different example. The Arkyves system does some advanced indexing using the ICONCLASS codes to give you the requested results, making full use of the hierarchical nature of the system. So if you had searched for ‘domestic animal bible’ you would have also found the biblical canines.

And if you really wanted to drill-down to do super exact matches, a visual tree browser for the ICONCLASS codes with integrated search results is shown.

Here is a video illustrating these points.

Web Frameworks Hosting

I have to take the opportunity to plug WebFaction as a satisfied customer. They do hosting for all manner of dynamic languages, (including Python/PHP/Rails) Subversion, etc. Their latest service modifications to allow unlimited domains, websites, processes, applications, databases all within the purchased plan limits is very attractive. One of the biggest selling points is their easy way to set up ‘applications’ via a web based control panel interface. There are videos on their site showing how to do it.

The engineering and setup on their servers are impeccable. Highly recommended.

(this post in response to an entry by Jens Alfke commenting on the current situation where cheap commodity hosting operations have difficulty in providing good Rails/Python service)

La Linea

Via Frits’s blog I am delighted to find that La Linea is on Youtube. Hooray!

I loved these as a child on TV. Would watch it with my brother and we would imitate the talking. The spitting maniacal laughter always stayed in my mind.

Hand-made vacuum tubes

A marvelous video (scroll to bottom of page) of the process of hand-made vacuum tubes. I shouldn’t be spending time watching this, but I couldn’t stop. It loosened so many ideas in my head about craft, technology, the tools & knowledge needed to do a job well, and how pioneering things that are so complex becomes so commonplace.

Makes me want a workshop with space to make things.

UPDATE: And in the comments to the Slashdot article, this crops up. A 1:3 scale Ferrari 312PB built in 15 years, every single part hand-made. 100cc flat-12 engine. Everything works, including the 5-speed gearbox. And it makes the Ferrari music.

“It would go, if you could find someone small enough to drive it”

Mind boggling. Absolutely, unbelievably mind boggling. I need to lie down a bit. My head is spinning.

Upgrading old (non-supported) Ubuntu versions

The past weekend was the first working day of the new year for me. What better way to start than to do all those chores which you have been putting off for months, and in some cases years. Number one on my list was upgrading one of my servers form an old version of Ubuntu 5.04 to a more recent version.

The first hurdle was that 5.04 aka Hoary Hedgehog is not officially supported any more. All upgrading instructions that are found online thus do not work any more. Bummer! I should have done the upgrade sooner. Bit like wishing I had brushed my teeth more often as a child to prevent all those fillings later on.

At first I thought that this was a show-stopper and that I would need to do a fresh re-install from CD, but a bit of nosing around revealed the solution: http://old-releases.ubuntu.com/

Hooray! Using this link I could first upgrade the server from Hoary, to Breezy and then on to Dapper. As Dapper is listed as the ‘Long Term Support’ version, I am going to stick with that on my server for now, and use a spare desktop machine (or Parallels) to play with the more recent versions.

The actual update process was so slick, I was gobsmacked. What a fantastic job the Ubuntu people have done to make bumping the version for the entire operating system.

Problems? Yeah sure, but they were my own fault. During the first upgrade to Breezy there was some error messages, but it seemed to me that the whole procedure had completed, as the process had continued after the error. I then forged ahead and rebooted the server, but after that it never came back. Called the hosting company for a manual reboot, but that still didn’t help. So I drove over there and did some manual troubleshooting with a screen connected to the actual machine.

At first glance the eth0 network interface was not up, and some error message showed up during the boot for name resolution, and looking at dmesg a problem loading deferred execution. When I did a manual ifup eth0, things seemed OK, but then I could not start any ‘screen’ sessions. Trying to run a screen i would get ‘No More PTYs, Sorry could not find a PTY’. Trying to remove and add the screen package made me realize that the culprit was actually a dangling ‘ant’ package with some unresolved dependencies. Once I removed that package with an ‘apt-get remove’, and restarted the update procedure, it all ran smoothly to completion. The config files were fixed, and I could reboot and continue upgrading with a smile.

Now my auto complete searches for Arkyves finally works as they should, and I can do sorting in PyLucene as intended. Hopefully this can be a good omen for the start of a great year.