I recently had to produce a number of diagrams for some talks and papers. In the end, I found that the best tool to put them together was Keynote.

Which made me think: Apple has all the components for a really good drawing program. They could market it as "Graphics for the rest of us".

I do hope that Apple are thinking this way. It could be the next great app for iWork 09. 

iWork's Pages and Keynote are less that completely usable for scientists for one simple reason; unlike Microsoft Office, there is no equation editor. My documents and talks always contain equations in one form or other.

I had always assumed that sometime like an equation editor would be too expensive to be included within iWork. iWork is priced at a remarkably low cost (well done Apple!), and Microsoft license their equation editor from a third party. So an equation editor for iWork seemed out of the question.

But then I discovered that Apple's Grapher application has more than the basis of a decent equation editor. It lacks the h-bar symbol, and one or two other things, but in most respects it is almost suitable for inclusion within the iWorks apps.

So will an equation editor feature in iWork 09? Pity about not making it into 08; it could have been in there.

I have been with OS X 10.x since x was zero (I am afraid that I didn't ever try the public beta because it seemed to be impossible to actually work with it). It was fun being an early adopter. Things were rough around the edges, but we were getting an idea of a new way of working. I still remember when I first saw the dock with its magnification, and it was striking just how different it was. 10.1 came as a free upgrade (what else could Apple do – 10.0 really wasn't really ready for production use except for the adventurous few). 10.2 saw what I thought was a proper professionalisation of OS X, and 10.3 saw what I thought was more-or-less the completion of the transition (with new features such as Exposé and fast user switching). During these versions we saw the introduction of tools such as ichat, and public beta versions of ical, isync, safari and X11. It was actually quite exciting.

Then we got to 10.4, which seemed to be to be somewhat underwhelming. I fear I have hardly made any use of widgets at all, mostly because they didn't seem to give me anything of value that a bookmarked browser would give. Spotlight seemed like a decent idea, but over time it seemed to me to be not quite as useful as I had been hoping. 10.4 stuck me, in conclusion, as being a set of small upgrades that certainly improved things in many ways (including new tools such as Dictionary), but without any obvious vision.

So we come to 10.5, which seemed to me to be prefaced by quite a lot of hype. Having now had it running on my three systems for a few days, I am no less underwhelmed than I was with 10.4. In fact it is worse, because some of the things I was hoping for seem not to work. In fact, I wonder whether in the rush to get 10.5 released (remember it was slowed down by the iPhone – it is almost as if 10.5 took too low a priority within Apple) there are too many loose ends still remaining to be tied. Take for example the translucent menu bar. Whilst this appears to be reviled by many, there appears to be no way to change how transparent it is. But there are a number of systems for which there is no transparency at all, and a strange note to this effect in Apple's knowledge base. I don't care whether I have a transparent menu bar or not, but the fact that I am supposed to have it and don't suggests to me that we will be seeing some bug fixes in the coming months.

The feature I was really looking forward to was "Back to my mac". I have a .mac account and I do need to access my home computer from other places. But it just doesn't work, and again, there is a page on this in Apple's knowledge base.

Spaces is a nice feature and appears to be better implemented than VirtueDesktop which I had before (I had to give up on that because palettes became separated from main windows in applications such as Pages – Spaces appears to do a better job in this regard). However, it is not near to being perfect. I don't always end up in the right place when going to a different application, and too often I watch it going around various windows before it finds what I want (making one almost sea sick!). It doesn't always go to an open Finder window correctly.

I will never use Bootcamp nor Webclip. Dictionary having access to wikipedia is neat, but hardly essential. When I buy a new disk Time Machine looks worth a look (I don't want to overwrite my current backup disks). I hope that Spotlight proves more useful that its previous incarnation. I am hoping that Stacks will work, but I suspect it will need a bit more work than I was hoping. Finder coverflow seems to me to be too slow to be useful. On a negative, something in the security model (probably with TSL) has killed my use of my department's email system.

In short, 10.5 looks different round the edges, and it seems to have some small useful tweaks (like Dictionary), but I see no big vision. In fact I would say that the number of small tweaks is actually quite impressive, and I can imagine it being fun coming across them one by one. For example, I have just come across the link between Address Book and Google Maps. It is nice. Screen sharing could be interesting, but when I tried it out my daughter was using my desktop and it came down to a tool for spying! 'Creepy' was the reaction. Icon preview could be useful, but there will be times when it might be a nuisance. Quicklook might be better.

What seems to me to be a pity is that some nice tweaks could be given away for free in the incremental updates. For example, adding tabs to terminal, or obtaining address information from emails, are small tweaks that would be nice to suddenly fund in an incremental update. The fact that Apple stores these up for one of the 300 new features in a large paid-for upgrade suggests to me that Apple is running out of a big vision for OS X. Of course, it would be churlish to complain, because effectively we now have a very robust and usably operating system that is apparently very hard to improve upon. 

I had been told that g95 is not as efficient as other Fortran95 compilers, but I have been amazed at today's experience.

Job times for my simulation with identical parameters on the National Grid service:

g95 compiler: 2762 s

ifort compiler: 468.3 s

This is a factor of 6 difference!

I wasn't doing anything clever with either compiler. Presumably both compilers can be tweaked in their performance, and the gap in performance narrowed. 

I have spent a couple of days helping someone to start running jobs on some of the large UK grid resources using grid methods. It is worth documenting the experiences, both what we achieved and what we didn't.

Objectives

Our plan was the following:

1. Learn to use the Storage Resource Broker for data management;
2. Learn to use the eMinerals metadata management tools;
3. Learn to use the escience digital certificate for grid access;
4. Learn to submit jobs to grid resources (NW-grid, National Grid Service, eMinerals minigrid) using the RMCS tool.

To add to this we had two benchmark targets:

1. A simple test run using my ossia code; this is quick enough to be a good test case;
2. A production job using the CASTEP electronic structure code.

Starting point

The requirement was to be able to run grid jobs from a windows laptop.

The grid tools we were planning to use are

1. SRB client tools, including the Scommands, InQ and TobysSRB. InQ is an Explorer-like Windows interface to the SRB that has a long-standing tradition of usage. We have never looked at the windows Scommands client tools.
2. Metadata tools, including the STFC Metadata Manager (a web interface) and a web interface to the RCommands. These require the use of an escience digital certificate. This was the safest part of the task.
3. RMCS client command-line tools and the RMCS java GUI. RMCS requires that the user has an escience digital certificate. Here we were really exploring; the client command-line tools have been been compiled for windows but not tested in detail, and the RMCS GUI had not been used at all for production work as far as I could tell.

At the outset, we had a digital certificate exported from the browser that was used to request it, but it had not yet been prepared for use by Globus (the p12 certificate had to be split into the userkey.pem and usercert.pem key files).

We had a working test case for ossia. However, we didn't have a compilation for CASTEP.

Day 1: a dream start

Day 1 started with coffee at 11 am, followed by a walk through the various tools. The harder work started around lunch time. What we achieved in more-or-less the order I can remember was:

1. We established that we had an account on the SRB. We walked through TobysSRB interface. We then downloaded the InQ explorer interface, and used this to upload some files (including setting up all the configuration details). Finally we installed the Scommand line-command binaries and tested them from the DOS shell. This included setting up the MdasEnv and MdasAuth files.
2. We then sorted out the escience certificate. We installed OpenSSL on the windows laptop, but in the end we cut short the time by preparing the certificate for Globus on my mac laptop.
3. We downloaded the java myproxy upload tool for the digital certificate. This required us to download and install java on the windows laptop. We then created a proxy and uploaded it. At the same time, we downloaded and installed the certificate authority certificates.
4. We had accounts created for the RCommands metadata tool and the RMCS grid submission tool. We then looked at the Metadata Manager and the web interface to the RCommands, and created some metadata for a data object, having created a test study and dataset levels. This part was relatively easy.
5. Our target grid resource was the NW-Grid, and we compiled a serial version of the XML-ised version of CASTEP. The parallel version was left to Day 2.
6. We then installed and launched the RMCS java GUI, although at this stage we were not ready to use it. I attempted to show it in use on my mac laptop; although it worked okay for tasks I had used it for (namely to check on jobs), we came unstuck using it for job submission.

Day 2: not quite as good as Day 1!

On the second day some things worked and some didn't. It got off to a bad start when the resource on which we were compiling CASTEP started on a slow death, preventing us from carrying out the compilation of the parallel version of CASTEP and also preventing us from downloading the previous day's compilation of the serial version. Thus we could only aim at getting the ossia job ran.

Our first task was to install the RMCS command line tools on the windows laptop. We had a couple of instances of successful installation, but the particular laptop offered some resistance. It turned out that the problems arise from a prior installation of cygwin, but after coffee the problem was fixed.

We then set up for submitting jobs to the grid. We got the files into the right place on the SRB to start with, and edited the MCS file. We had a created metadata dataset prepared for the data object produced. We needed to log into each resource to create the srb and rcommands user configuration files. This raised an interesting issue. We aim to create tools that prevent users from logging in to grid resources, and from needing to install globus, but users need these tools in order to log in. In the event, one of our team installed these files on one of the NW-Grid resources, and we used the GSI-SSH java gui to log into the resource from which we were not debarred by the firewall. The GSI-SSH gui is quite nice for this sort of thing. It isn't really up to long-term production use because it is noticeably slower than a real terminal, but it is easier than having to install Globus or have someone create an account for you on a computer than does have Globus installed. There is a serious issue in this.

Then we had a couple of set-backs in using NW-grid. We first found that one grid resource was so busy that we were only queuing. Then the other resource had our account set up without being part of a user group that was needed. Thus in the end we reverted to running our ossia test case on the eminerals minigrid. All the setting up required was a bit of a faff, emphasising that the start-up time is not insignificant (although not necessarily any more so than faced by people running on any cluster). 

Finally we were able to run our ossia job using rmcs on the eMinerals minigrid. This job created output files on the SRB and a complete set of metadata. Success - by mid-afternoon on Day 2 we had a complete grid job instance from the laptop with everything working. What was pleasing was that this is perhaps our first instance of doing this all from a windows computer and some of the components had not been tested in production mode.

The creator of the RMCS java gui fixed the problems with job submission for us. However, we ran into a problem getting this working on the windows laptop, a problem we have left for tomorrow morning.

Conclusions

Barring the fact that events conspired to prevent us compiling CASTEP, 2 days seems to be okay to go from no grid experience to running proper grid jobs – by which we mean jobs with proper data management and metadata collection, not just submitting a job to a cluster. There is a steep learning curve, but with preparation this can be flattened a bit. The biggest issue in my mind was the set-up required. It only took a couple of hours I think, but it was a bit of a chore. More to the point, it reveals some usability issues. Surely it should not be left to users to have to do all this if our aim is to have tools that do not encourage users to log in to the compute resources.

I have noted that this was the first time we have enabled someone to run grid jobs from a windows laptop. This was quite an achievement in my view, because on one hand most grid users actually are users of a unix/linux environment (including via mac OS X). Well, I know that people will submit jobs via portals, which are supposed to be OS-agnostic, but that isn't the point here. What we want are tools that users can run using command line tools.

No-one will have noticed this (if SciSpace.net security worked okay), but I wrote two grant proposals last week using SciSpace.net in collaboration with two sets of people based in different parts of the country.

I have done this before often. Typically we used email and Microsoft Word. This approach works okay until you get to the last few days. Then everyone starts to send you their edits, and then you are managing the task of merging several documents. And all this at a time of maximum stress. It is a nightmare, and things always get lost. I have done this before often enough to have experience of suddenly discovering you are editing an old version.

So Hello to SciSpace.net, and the use of the wiki and blog tools. We set up the main case to edit in the wiki, and people took turns to edit in their stuff. Comments were added either in the blog tool or in other wiki pages.

And no-one knew this was happening, because we used access control, which is exactly what we want.

In time I think that we will see better collaborative document tools, but for now the SciSpace tools worked well.

Then we reached the point where we had to copy to a Word document, and at that point we shifted from SciSpace.net to the traditional way of handing around a Word document via email. And it just reminded me how bad email is as a collaborative tool, because the nightmare of multiple copies came back!

Things I learned from this process:

1. The text edit window isn't really big enough for editing a document, but for now there isn't much we can do about it (this is an issue for Elgg);
2. You have the danger of competing edits; we are discussing this with the creator of the Folio wiki tool (and he is very receptive to ideas, which is great);
3. The Elgg text edit box is interesting in that it will pick up pasted in styles, which is not what you really want – I resorted to editing in html on occasions;
4. In fact I kept open a general text edit for my main text, and pasted in the whole text rather than editing in place.

At a recent workshop, I found myself using the title of this blog in both of the talks I gave, and, as happens, I found myself feeling quite passionate about the point!

But let me start with a bit of personal history. When I started in science, the primary tool for collaboration at a distance (apart from public transport) was the postal mail service (telephone was not allowed for PhD students or post docs in some of the institutes I worked in). One could send papers to people who you thought should know about your work, or send data and draft documents to collaborators. The big breakthrough for collaboration was the introduction of the Fax machine, because you could send graphs to your distant collaborators quickly and easily. eMail was a revolution, but actually the revolution was rather more recent than we imagine because it required your collaborators to also have access to email (and uptake was not uniformly quick), and interface development was quite late in coming.

For a while, email transformed collaboration. One could send ideas and engage with collaborators, and attachments allowed us to send data, figures and documents. For a few years it played a significant role in supporting collaborations. But then it all started to go wrong!

Why it went wrong is very simple: it was killed by its own success. The ability to spray out instant communications meant that is exactly what people did, until we started to drown in emails. I stress this is nothing to do with junk or spam email; my email client (Apple's email application) has an excellent junk/spam filter. It is just that we get too much email to be able to cope with, and now collaboration messages are lost in the noise of business emails, many of which are urgent.

In short, email is a communication tool that has served as a tool for collaboration, but its value as a collaboration tool is now much diminished. The longer version of this point is as follows.

1. With a huge number of emails in my inbox, it is now easy to lose track of emails and thus not deal with them. They quickly disappear down the inbox. Some might say that more organisation would help, but to be honest, I don't want to organise my emails; it just wouldn't work! I actually like to use the search tool provided by my email client to locate messages.
2. Collaborators don't always help with organisation of emails. eMails sent with poor titles, or worse, no titles, do not make browsing easy. For example, I might ping someone with an email with the title "Hello" (admittedly not very good for long-term management, but on the other hand, ping emails are not really supposed to be archived), and they might then use this as a starting point for a more serious discussion, each titled "Re. Hello". Poor use of titles is the equivalent of the metadata problem.
3. On a point of principle, I do not want to mix collaboration (which is enjoyable) with business (which often is not). I groan when I launch email and see all the demands coming in. This is not the environment I want to use for collaboration!
4. eMail is really designed for one person sending a message to someone else, and not really for one-to-many. Sure, it has the CC field, but often I have seen this not used properly (e.g. one person gets left off by mistake). Moreover, unless I include myself in the CC field (and thereby duplicate my sent-mail box into my inbox) I don't have a complete record in one place of the various stages of communication.
5. Whilst I can organise my view of any collaborative exchange of emails, I cannot affect how my collaborators view their version of the exchange.
6. And a new collaborator is not privy to what has already been exchanged, unless we send them an archive, but such archives are not designed to be read as such.
   
7. Once I have sent an email, it is out of my control. If I want to edit the content after reflection, or add to the content, I can't. If I have been ambiguous or unclear, I am at the mercy of the person reading the email (and email is notoriously easy to misread).

eMail is not all bad. It is actually excellent at providing a central point for my communications, and can be alerted by incoming messages.

eMail is okay as a tool for collaboration, but only "just okay". It is not good, merely okay. 

Now we have the potential for something better. This blog entry is not supposed to be an advert for SciSpace.net, but the sort of infrastructure we are trying to develop here (and which perhaps other people could develop) has the potential, I believe, to transform how we collaborate.

SciSpace has been a journey. It started at a workshop held at the National Instutute for Environmental eScience at the end of January 2007 on the topic of organic pollutants, organised by Kat Austen (see here for the link to the event web page). The idea was to link simulation scientists with experimentalist and field scientists. In breakout groups it was clear that everyone was saying, even though they didn't know it, that what the community needs is a MySpace for scientists. Hence SciSpace, which has developed from discussions that began at the event and were developed further in Cambridge between members of the NIEeS and the eMinerals and MaterialsGrid projects.

We tried a number of approaches, including a start-from-scratch approach and this approach using the Elgg software. Shortly after we noted the creation of Nature Network, which is aiming to be a social networking site for scientists. We experimented with Nature Network for a while, but concluded that in the short term at the least it is not developing in the way that will be useful to scientists. If anyone is interested, I was encouraged by the Nature Network team to deposit my comments, which you can read from this forum link.

Thus we decided that there is a market space for SciSpace, and we have now brought it to this state. As I write, we have not finished development work from our end, but it is now good enough to go live with.