All posts by Huw Lynes

Weekend Stuff

After spending Friday eveneing skating further and faster that I ever have before my knees now hate me. But I have the warm fuzzy feeling that in a couple of weeks I may no longer look like a complete gimp while I’m skating.

swskate are lovely people and are entirely to blame for my currently aching joints. And before anyone sends me any emails about the javascript on that website I’ve already talked to their web chap about the proper way to do things. In his defence he’s only just learning and appears amenable to clue.

Spent much of Sunday getting my head around generating and parsing XML in python. The upshot of this is that I can now import OPML feed lists straight into peapod. OPML export is next on the list, after which adding pcast support should be fairly straight-forward. Juri Pakaste’s OPML library made this all very easy. A little bit of XML knowledge should come in handy now that I am beginning to wade hip-deep into the new generation of grid toolkits.

Jonothan Coulton has done it again. Codemonkey should warm the cockles of all programmers’ hearts everywhere. Enjoy.

Grid Computing Talk For Gllug and LRL2006

Coincidentally I’m attending an “Improve Your Presentations” course on Wednesday so I though I’d better put together a first attempt at my Grid Computing talk.

It’s fairly hand-wavy at the moment but contains the basic outline of my “Grid Computing as seen from the Stratosphere” talk. If you don’t mind it not being a surprise when I present at Gllug later in the year then you can read it here.

SWIG and C++ on Solaris

Here at WeSC we have a potentially very useful piece of kit from Cybula which is essentially a set of neural-net based pattern matching algorithms blown into some FPGAs.

Now I have to advise some researchers about how easy it will be to use this kit for a particular project. My ability to advise is limited by the fact that the API for this kit is written in C++. This makes it hard for a mere sysadmin like myself to play with it.

I have been vaguely aware of SWIG for a while as a mechanism for building language-bindings to C and C++ so I decided to give it a whirl.

I installed swig from sunfreeware. After browsing the tutorial for a while I decided to take the simple approach to building an interface file.

%module aura
%{
/* Includes the header in the wrapper code */
#include "aura.h"
%}

/* Parse the header file to generate wrappers */
%include "aura.h"

Then I used swig to generate the python and C++ code.

swig -python -c++ -I/opt/Aura/2.4.2/include aura.i

And built the libraries from that

g++ -shared -I/usr/local/include/python2.4 -I/opt/Aura/2.4.2/include -c aura_wrap.cxx

This left me with aura.py and _aura.so which look like a python module to me.

Python 2.4 (#1, Dec  1 2004, 05:20:08)
[GCC 3.3.2] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>>import aura

Now I just need to work out how to use it. Assuming this module actually works and isn’t just an empty shell I may have to try building Java and Perl modules later.

Edited: 16:27 to include some bug-fixes

After playing around with this some more I realized that I had indeed created a completely empty binding. So I went through aura.h and manually included all the header files that it pulled in.

%module aura
%{
/* Includes the header in the wrapper code */
#include "aura.h"
#include "auraErrorTable.h"
#include "auraTypes.h"
#include "auraStructures.h"
#include "auraException.h"
#include "auraSearchEngine.h"
%}

/* Parse the header file to generate wrappers */
%include "aura.h"
%include "auraErrorTable.h"
%include "auraTypes.h"
%include "auraStructures.h"
%include "auraException.h"
%include "auraSearchEngine.h"

This produced much larger python and c++ files so I guessed I was on the right track. I also realized that the g++ command wasn’t linking in the aura library and that I would have to do it by hand

g++ -shared -c aura_wrap.cxx -o aura.o -I/usr/local/include/python2.4 -I/opt/Aura/library/include

ld -G -o _aura.so aura.o /opt/Aura/library/lib/SunOS/libaura.a

The path changes were introduced when I realised I was trying to build a 64bit version of libaura into a 32bit version of python.

Python 2.4 (#1, Dec  1 2004, 05:20:08)
[GCC 3.3.2] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>> import aura
>>> dir(aura)
['ERROR_CANNOT_READ_STREAM', 'ERROR_CANNOT_WRITE_STREAM', 'ERROR_COLUMN_INVALID',
'ERROR_CONSTRAINTS', 'ERROR_FORMAT_INVALID', 'ERROR_INITIALISED', 'ERROR_MASK_INVALID',
'ERROR_MAX_SCORE', 'ERROR_MEMORY', 'ERROR_MISMATCH', 'ERROR_OPTIMISATION_SETTING_INVALID',
'ERROR_PARTITION_INVALID', 'ERROR_PATTERN_INVALID', 'ERROR_RANGES_INVALID',
'ERROR_RETAINED_ID_INVALID', 'ERROR_ROW_INVALID', 'ERROR_SAME_OBJECT', 'ERROR_SIZES_INVALID',
'ERROR_UNIMPLEMENTED', 'ERROR_UNINITIALISED', 'Exception', 'Exception_swigregister', 'NULL', 'OK',
'Pattern', 'Pattern_swigregister', 'Preferences', 'Preferences_swigregister',
'RetainedInfo', 'RetainedInfo_swigregister', 'SearchCharacteristics', 'SearchCharacteristics_swigregister', 'SearchConstraints',
'SearchConstraints_swigregister', 'SearchControl', 'SearchControl_swigregister', 'SearchEngine', 'SearchEngine_swigregister',
'SerialiseControl', 'SerialiseControl_swigregister', 'Sizes', 'Sizes_swigregister', 'StorageType', 'StorageType_swigregister',
'WARN_COLUMN_IGNORED', 'WARN_NO_BITS_TO_STORE', 'WARN_RETAINED_ID_INVALID',
'WARN_ROW_IGNORED', 'WARN_SHRINKAGE_IGNORED', 'WARN_STORE_CHANGED', 'WARN_TRUNCATED',
'__builtins__', '__doc__', '__file__', '__name__', '_aura', '_newclass', '_object', '_swig_getattr', '_swig_repr',
'_swig_setattr', '_swig_setattr_nondynamic', 'cvar', 'new', 'new_instancemethod']
>>> x = aura.SearchEngine
>>> dir(x)
['ClearRetained', 'Copy', 'GetColumn', 'GetRetainedIds', 'GetRow', 'GetSizes', 'IncreaseSize', 'Initialise',
'Merge', 'Optimise', 'Remove', 'Search', 'Serialise', 'Store', '__class__', '__del__', '__delattr__', '__dict__',
'__doc__', '__getattr__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__',
'__reduce_ex__', '__repr__', '__setattr__', '__str__', '__swig_destroy__', '__swig_getmethods__',
'__swig_setmethods__', '__weakref__']

Result!!

Aftermath

Happily I can report that my two Sun V880 servers are now running again. I discovered early this morning that the reason I couldn’t boot from DVD is that the Toshiba DVD drives on these servers have a firmware bug which means they cannot read 2k sectors. So I had to spend some time downloading the Solaris 10 CD image. So if anyone is interested in the Solaris equivalent of booting from a rescue disk here it is.

At the OpenBoot prompt (you may have to send BREAK to the boot process to get the prompt which on minicom is CtrlA-f) type:

boot cdrom -s

and wait for what seems like an eternity while it boots up. You are then free to mount filesystems and fsck things to your hearts content.

Of course once I had the Sun boxes up and running we discovered that the Origin300 cluster was completely dead. The PROM has lost all it’s hardware info.

Current Mood: fatalistic

Mr Murphy, I see you’ve brought your SledgeHammer

Came into work to find that about half my machine were down. Mostly just switched off, but some in indeterminate stages of “not working right”. Powered-on all the machines that were obviously down and re-booted the others (I wouldn’t normally do that but with so many machines down I had to remove the easy targets, not to mention the fact that console management in the machine room is woefully inadequate for anything more than brute-force diagnostics.)

So at this point it was becoming fairly clear that we had had a little incident with the power supply. While waiting to see what would come back up I tried to log into my mail and found that down also. Clearly whatever interrupted our supply hit the whole building. A look through the logs of those machines that were up shows that everything re-booted on Saturday morning. Definitely a power problem.

Further prodding of those services that refuse to come up reveal two culprits: bouscat (the server that world+dog get dumped on) has let the magic smoke out, it can probably be frankensteined. More seriously the two Sun V880 servers that run one of our main data-stores are refusing to boot. This elicits much running about in search of the correct serial cable (see previous post in which I clearly caught the attention of the fuckup fairy).

After finding a cable that would give me console access I discover that the root filesystem is failing fsck and mounting ro. This apparently is enough to completely hose a Solaris8 box. Oh Joy. After consulting with #solaris I can poke OpenBoot in the requisite manner to boot in single-user mode. This gets me precisely no-where. Even in single-user mode it refuse to get me to a shell where I could do anything useful. Tonight I shall mostly be reading the Solaris documentation to try to work out how to get the box to boot from DVD so that I have some hope of fixing it from the rescue system.

Current Mood: Maybe London Wasn’t So Bad

Random Updates

After some gentle ribbing from friends about the lack of updates to this blog I’ve realised that it is indeed time to get back in the saddle. I apologise in advance for the disconnected nature of this post.

So work is almost a green-field site in that while they have quite a few systems in production they are all being managed in a piece-meal fashion. I’ve been free to specify a standard linux build (CentOS4) and whatever management systems I want. Now that my new server is in place I can start setting up cfengine.

Given that these systems represent a fresh start I decided to leave SElinux turned on (at $ORK-1 we turned it off because it objected strongly to the way we had things set up). This has lead to a fairly arduous learning process. The upside is that I now mostly understand it but I probably hate it. The fact that the SElinux policy is monolithic makes things hard to manage. I’m not sure how I’ll deal with this as the systems get more complex. I’ll probably end up building a selinux-policy-wesc package and try and ignore the ugliness.

In other security related news I’ve finally gotten around to learning about PAM. It’s strange but mesmerizingly powerful. Together with mod_auth_pam you can get apache logins to do very strange things. Now if only someone would write a PAM module for Shibboleth.

It’s the little things that are annoying me. Like that fact that someone has walked off with the case keys for my two Sun V880 servers.

We have no backups. Nothing to see here, move along.

My shiny iAudio X5 is now running rockbox. It is fantastic. Browse by ID3 tags. List files by date. Revolutionary!

I shall be giving a lightning talk on Grid Computing at LugRadio Live 2006. If you happen to be attending, prepare to be be informed and entertained.

The ramble endeth here..

Settling In

Since lots of people have been badgering me for photos of my new flat I’ve finally managed to summon up the organasational energy necessary to remember to bring my photos into work so that I can upload them.

So for your edification. I present:

A picture of the block I’m living in.
Anson Court

The view from my living room.
View of the East Bute Dock from my Living Room

My ADSL should finally be connected at the end of the week so normal blog service will be resumed shortly thereafter.

Chasing the (Red) Dragon

So, as some of you already know, the time has come for me to leave the film industry. I’ve been working at MPC for over four years now, the single longest period of time that I’ve ever been in one job. My feet have been getting increasingly itchy over the last year or so and I’ve been keeping my eye out for interesting jobs. When a position opened for a “Grid Technology Specialist” at Cardiff (my alma mater) I jumped at the chance. Cardiff is a lovely city and I thorougly look forward to moving there. In case you are interested I shall be working for these guys.

Four years in post production/VFX have pretty much taught me all that I know about adminning big *nix systems. It’ll be interesting to try and apply all that knowledge in a more academic environment. I’m not sure how I’m going to adapt to the comparative lack of pressure. I keep thinking ’35 hour week’ and grinning.

The advert for my replacement is already up. Anyone interested in talking my place should apply PDQ as there have already been responses.

So long and thanks for all the fish.