I Laughed, I Cried

Godzilla and Beaker Investigate the Credit Crunch

Finance has never been the natural home of philosopher kings. It is instead the home of the rapacious vultures, dead eyed killers and Brooks Bros clad Nazgul – financiers, neither living nor dead. Two things have happened recently which have made the whole things worse. One, the old rather pissed boys who wore pinstripes and did business over five hour lunches with old, equally pissed, chums from Eton or Yale have slowly left. These guys were no good (see The Great Depression for example) but they were pissed which meant they weren’t that quick. Now they have been replaced with guys who are sober (at least during the day – at night they like to take coke and abuse strippers or the homeless), sharp as blades and twice as morality free. The second bad thing is that we have given them really powerful computers and bunch of maths PHDs who should be doing physics to run them.

Thoughts on .Astronomy

Having spent the afternoon at my desk sorting out my to-do list for tomorrow, the buzz from attending dotastronomy has almost worn off. The odd thing is, it felt more like LugRadioLive year one, rather than a scientific conference.

You might wonder what a biologist turned sysadmin was doing at an event themed around “Networked Astronomy and New Media”. I ended up there purely to network on behalf of my department. Expecting potential researchers to come to you is great, but nothing beats going out and talking to them on their home territory. However, it turned out to be a much more useful experience than that.

The conference broke down into several topics:

  • The basics of blogging and web2.0 tools
  • Using the internet for eduction and outreach
  • International Year of Astronomy
  • Robotic telescope networks and the Virtual Observatory

What really surprised my was how much of this was directly relevant to out work at ARCCA. Because we have a mission to expand the user-base of HPC at Cardiff we obviously have to be reaching out to non-specialist. I have every intention of trying to apply some of the ideas from this conference in my day job.

After Iain Steele’s talk briefly mentioned market-based assignment of telescope time the idea of a commodities market in telescope time, perhaps unfairly, became the event’s running joke. With the following lunch being spent working out all the best ways to game the market. I was particularly tickled by the thought of a consortium of astronomy bloggers disparaging the service of a particular telescope in order to artificially depress the price of it’s observing time.

To bring things back round to a more serious note Andy Lawrence’s talk on the Virtual Observatory contained much food for though. Everyone is facing the prospect of dealing with larger and larger datasets. Obviously the particle physicists are out in front, but even biological datasets start getting unwieldy when you start dealing with things like population-wide microarray surveys. The basic point being that manipulating and searching the data at the site it was captured is easier than trying to ship the entire dataset to the researchers home institution. Eventually the norm is probably going to be for compute facilities like ourselves to be hooked into systems like the VO so that computational analysis of the data in a distributed fashion becomes as easy as distributed search and filter.

The highlight of the show for me was to get to see Cardiff University’s new half-metre telescope. If any of the astronomers want a tour of the new supercomputer I’d be glad to return the favour.

In short .Astronomy was fantastic amounts of fun. I’m sure I will return to this topic in more detail when the talks start to appear on youtube and the conference proceedings come out.

Learning Programming Languages

I’m a sucker for a good meme. So here is a list of programming languages in the order that I learned them.

  1. Basic
  2. Python
  3. Shell
  4. Perl
  5. C
  6. Tcl
  7. Java

Of those I’ve written production code in all of them apart from Basic (which I learned at school).

I learnt Python after I got into Linux (during my ill-fated stint as a PhD student). Shell, Perl, C and Tcl were languages I had to come to grips with while I worked at MPC. Shell, Perl and C won’t surprise anyone since I’m a sysadmin. Tcl might seem odd until you realise that quite a lot of the Pixar tools communicate by throwing Tcl files around. Funnily enough we’ve just implemented Modules at work so I’m writing Tcl again.

Learning Java was driven by necessity to write some simple extensions to Globus during my brief sojourn in the Grid community.

Of all these languages the only ones I will use by choice are Python and Shell (in my case usually Bash). Shell for quick scripts that are primarily manipulating other programs and Python for anything more complicated.

Jobs and Prizes

Apparently we’ve made the shortlist of the British Computer Society 2008 IT Industry Awards. This is in the environmental category for our new compute cluster install.

In slightly less esoteric news work are advertising for two new posts at the Advanced Research Computing Division of Cardiff University. We are looking for:

Both these positions are co-funded by Bull Information Systems and will involve a significant amount of collaboration with them. On a personal note, I’ve found the Bull R&D team in France and the support team in the UK to be an absolute pleasure to work with. I would happily apply for one of these jobs except that: a) I hate people and b) I despise Fortran.

Dealing With Stupid Programs That Think They Need X

The new compute cluster is beginning to feel like a production system. I’m currently run off my feet installing software for the stream of new users. Mostly this is fine, but occasionally I run into software that makes me want to band my head repeatedly on my desk until the pain goes away; or more accurately makes me want to bang the programmer’s head on the desk.

Just today we received a linux port of a code that has been running on the Windows Condor pool for a while now. Everything seemed fine except for it’s stubborn refusal to run if it couldn’t find a windowing system. Bear in mind that it doesn’t actually produce any graphical output it just dies if it can’t connect to X. After a bit of futzing around we discover that the people that normally run this code do something like:

Xvfb :1 -server 1 1024x1024x8 &
export DISPLAY=:1
./stupid_code_that_wants_X

Xvfb is the X virtual framebuffer. It creates a running X client without actually needing any graphics to be running.

Which works just great locally but if you want to launch that as a script in the job scheduling system (we use PBSpro) then you need to be a bit more careful. What happens if two of these jobs try to launch on the same machine? Obviously one of them will fail because display 1 is already allocated. What I really needed was a script that will try to launch Xvfb and increment DISPLAY on failure until it finds a display that is free. For your edification here it is:

get_xvfb_pid () {
	XVFB_PID=`ps -efww | grep -v grep | grep Xvfb |\
       grep $USERNAME | tail -n 1 | awk '{print $2}'`
	}

create_xvfb () {
	USERNAME=`whoami`
	DISPLAYNO=1
	while [ -z $xvfb_success ]
		do
		get_xvfb_pid
		old_XVFB_PID=$XVFB_PID
		XVFB_PID=""
		Xvfb :${DISPLAYNO} -screen 0 1024x1024x8 >& /dev/null &
		sleep 1
		get_xvfb_pid
		if ! [ -z $old_XVFB_PID ]
			then
			if [ -z $XFVB_PID ] && ! [ $XVFB_PID == $old_XVFB_PID ]
				then
				echo "Started XVFB on display $DISPLAYNO process $XVFB_PID"
				xvfb_success=1
			else
				DISPLAYNO=$(($DISPLAYNO + 1))
				XVFB_PID=""
			fi
		else
			if [ -z $XFVB_PID ]
                                then
                                echo "Started XVFB on display $DISPLAYNO process $XVFB_PID"
                                xvfb_success=1
                        else
                                DISPLAYNO=$(($DISPLAYNO + 1))
                                echo "FAIL!" $XVFB_PID
                                XVFB_PID=""
                        fi
		fi
 		done
	export XVFB_PID
	export DISPLAY=:${DISPLAYNO}
	}

kill_xvfb () {
	kill $XVFB_PID
	}

Which you can call from a script like thus:

[arccacluster8]$. ./xvfb_helper
[arccacluster8]$ create_xvfb
Started XVFB on display 1 process 9563
[arccacluster8 ~]$ echo $DISPLAY
:1
[arccacluster8 ~]$ echo $XVFB_PID
9563
[arccacluster8 ~]$ ps -efw | grep Xvfb
username    9563  9498  0 19:31 pts/8    00:00:00 Xvfb :1 -screen 0 1024x1024x8
[arccacluster8 ~]$ kill_xvfb
[arccacluster8 ~]$ ps -efw | grep Xvfb
[arccacluster8 ~]$

I submit that this is a disgraceful hack, but it might come in handy to someone else.