As any fule no, QNames are how XML does namespaces. Where a namespace has been declared:

<c:cml xmlns:c="http://www.xml-cml.org/schema/>

and the "c" prefix on the element name is associated, via the xmlns attribute, with the namespace URI. This is trivially manipulable with any namespace-aware tool.

So far so good. However, when QNames are used in content (typically, as an attribute value) then the situation is more complex. The two nodes below are equivalent under QName-in-content processing.

<c:cml xmlns:c="http://www.xml-cml.org/schema"
  att="c:comp"/>
<d:cml xmlns:d="http://www.xml-cml.org/schema"
  att="d:comp"/>

This usage is blessed by the W3C, http://www.w3.org/2001/tag/doc/qnameids.html, and XSLT depends on it working.

But it's significantly harder to work with using most XML toolkits.

node()[@att='string']

The above XPath returns all nodes which have att="string". However, it turns out that matching on a namespace-resolved QName needs the following:

node()[substring-after(@att, ':')='comp'
       and @att[../namespace::*
                 [name()=substring-before(../@att,':')]
                ='http://www.xml-cml.org/schema']
      ]

if you only allow for prefixed QNames (eg c:comp above). If you want to be able to match unprefixed QNames as well, that is, QNames in the default namespace:

<cml xmlns="http://www.xml-cml.org/schema"
  att="comp"/>

then you need to extend the expression to the following:

node()[(substring-after(@att, ':')='comp'
        and @att[../namespace::*
                  [name()=substring-before(../@att,':')]
                 ='http://www.xml-cml.org/schema'])
    or (@att='comp' and
         and namespace::*[name()='']
              ='http://www.xml-cml.org/schema')
       ]

which is hardly transparent!

Much as I think XPath 2 is a bad idea in general, this is one area where it is a significant step forward; it offers node functions:

• fn:local-name-from-QName

• fn:namespace-uri-from-QName

which will do what they suggest. Of course XPath 2 then buggers things up again by saying:

Who needs backwards compatibility anyway?

But since libxml2 doesn't support XPath2, I don't propose to worry very much about it.

In any case, unwieldy though the above solutions are, they work correctly.

Just to get the goods up front:

define(`m4Y', `dnl
pushdef(`m4Y_recur',dnl
`pushdef(`m4Y_LL',dnl
`$1''changequote([,])(['changequote([,])`changequote`]`$[]1'(``$[]1'')dnl
['changequote([,])'changequote`])changequote`dnl
(changequote([,])`$[]1'changequote)dnl
`popdef(`m4Y_LL')')'dnl
`m4Y_LL')dnl
pushdef(`m4Y_LL',`dnl
m4Y_recur(`m4Y_recur')'changequote([,])(`$[]1')changequote`dnl
popdef(`m4Y_recur')`'popdef(`m4Y_LL')')`'dnl
'`m4Y_LL')`'dnl

So as seems to be popular, I've been working my way through The Little Schemer over the last few weeks.

And, as is equally common, I ground to a halt at the derivation at the end of Chapter IX, where they spring the Y Combinator on the unsuspecting audience. The best way to understand it is to work through it by yourself, so I thought I would see if you could do one in m4. And it turns out you can, though it's not very pretty!

Clearly what the world needs is to know about it, so I wrote it up, and you can follow the derivation in two essays:

1. Higher-Order Programming in m4, which shows you how to do proper quoting to get macro Currying to work.

2. The Y Combinator in m4, which uses those quoting techniques to do the full derivation of m4Y above.

As I've mentioned before, all the entries in this blog are written in asciidoc, which is very nice for a lightweight markup language, particularly in terms of embedding code fragments and having them marked up nicely.

Asciidoc uses GNU Source-highlight as its backend for generating pretty code fragments, which does a reasonable job, and its author is very responsive to feedback - he's fixed a couple of bugs in the Fortran and XML modules for me.

However, I've been growing dissatisfied with its use, for two reasons.

1. it has a very heavyweight dependency on boost, for its regex library. Compiling boost takes several hours, and this seems to me like massive overkill for a bit of code highlighting.

2. It is purely regex-driven. Furthermore, all language front-ends are defined in terms of a mini-regex language. This means that its markup capabilities are fundamentally limited to a very simple regex subset.

In any case, it can't approach the expressiveness of Emacs font-lock highlighting, which is what I'm used to.

So, I thought it ought to be possible to abuse one of several available emacs-lisp packages to do the job, and indeed it was. The script available here is a wrapper around a modified version of htmlfontify, and works like so:

htmlfontify -mode $MODENAME $FILENAME

or if $FILENAME is -, it takes input on stdin. It will print out a properly marked-up fragment of HTML on stdout, marked up according to emacs, in $MODENAME-mode fontification.

This was easy to write an asciidoc filter for, so code in this blog will henceforth be marked up by emacs.

As a follow-up to my last post on this, some good news and some bad.

Good news: Apple Engineering got back rapidly, with a good understanding of the authentication issue, and a good suggestion for how they might fix it. Of course the fix won't emerge until at least 10.5.3.

Bad news: there is another bug lurking in Finder's webdav implementation that I keep coming across. I haven't characterized it well enough to report, but I'm noting it down here so Google has some record of it at least.

The symptom is that after the state of the webdav server changes in some way (Certainly not every time it changes; I think this occurs when a directory that was previously readable has become unreadable because permissions have changed) then when you try and eject the mounted disk, Finder refuses with one of two error messages.

1. It complains that the disk is in use, even when it's not - and this can be confirmed by

   sh-3.2# lsof /Volumes/webdav_mount
   lsof: WARNING: can't stat() webdav file system /Volumes/webdav_mount
         Output information may be incomplete.
         assuming "dev=2d000009" from mount table
   

   The fix for this is a simple

   umount -f /Volumes/webdav_mount
   

2. It gives the unhelpful message: "error code -8072"

   In this case, the fix is first to unmount the disk with umount as above, and then to restart Finder.

   Note	Make sure to unmount the disk first!!! If you try and restart Finder (or logout, or reboot) without doing so, then the OS is liable to hang in an unretrieveable state, so that only pulling the power cable fixes it - which has happened to me more than once.

Borrowed from plasmasturm.

sloth:~ tow$ history|awk '{a[$2]++} END
  {for(i in a){printf "%5dt%sn",a[i],i}}'|sort -rn|head
   99   ls
   85   cd
   66   git
   53   xsltproc
   44   vi
   38   ssh
   22   python
   15   grep
    8   wget
    8   rm

People seem to like visual (or graphical) programming languages, (VPLs), but I don't think they should.

Reports of success

This was prompted by a talk on Monday, at the Royal Society meeting on environmental e-Science. One of the speakers (I forget who) was demonstrating a workflow system, run through a VPL environment. He talked about having given a workshop, showing scientists how to use the system, and quoted one of them as saying (paraphrased from memory):

and used this as evidence for how wonderful such "friendly" VPL environments are.

I've been to a number of talks where such things are shown off, and it's undoubtedly true that placing these tools into the hands of some scientists does result in such reactions (although I suspect less often that their proponents like to think; and I'm not sure how long-lived such reactions are).

However, contrary to the conclusions usually drawn, I don't think that the praise should be given to the VPL environment - indeed I think such things are actively harmful, and actually, you could get the same reactions via different means.

Cleaner interfaces

I suspect that actually, such positive reactions aren't actually caused by the visual nature of the environment, so much as the fact that (compared with the typical workflow system of bodged-together Perl scripts)

1. interfaces between components are much simpler,

2. they've been pre-written by someone else

3. they've been designed to plug together.

But because humans are very visual creatures, it is the obvious differences in the visual aspect of the interface that is noticed, and it's to that that positive effects are ascribed. It may well be that there is a small advantage there, but I think it is fairly small, and very easily overstated; see for example "Why looking isn't always seeing: readership skills and graphical programming", which I don't think enough relevant people have read.

When pulling things together using bodged Perl scripts, then you are reliant on whatever interfaces to the script, and to whatever other programs are being called, that someone else has written.

These interfaces are probably not quite what you want for your purposes, so you'll need to munge them a bit.

They're almost certainly not well-designed - indeed probably little thought has gone into interface design at all, so much as ensuring that the necessary scientific job gets done.

They may not be very functionally oriented; i.e. not well-suited to being called as part of a larger workflow. For example, there may be lots of out-of-band set-up required in the way of global environment variables and so forth.

All of these problems will be ameliorated in building components for any workflow system, certainly those of the type likely to underlie typical VPLs. Interface design will be an integral part of making components that fit together to make workflows of the type envisaged; components will have been co-designed, so that interfaces between them match well their intended use; and they will have to be built such that they don't rely on global state.

The end result is that the components of such a system can be pulled together and made to interact far far easier than a set of programs designed in isolation with little thought for reuse. (Though if badly done, it may result in components which can't be easily re-used outside the original workflow domain.) And of course this is true regardless of what programming interface is used to edit the resulting workflows.

However, I also believe that beyond this, VPLs are actively harmful.

Text-munging tools

My objection boils down to the fact that we have an enormous range of text-munging tools, but we have far fewer tools for munging whatever graphical representations are fed to us on the screen.

The issue that most obviously shows itself to me is version control, or revision tracking.

As any halfway-sensible programmer does, I keep all my projects under version control. The reasons are well-rehearsed, but I firmly believe that just as any program longer than 10 lines probably has a bug, any program longer than 10 lines should be kept under version control. The same applies to programs in VPLs. If you've got more than 10 or so components strung together, each of which probably has 5 or 6 tunable parameters, then you want to be able to preserve the state of the system, and record changes between them.

And this is not just in order to be able to roll back to previous versions; but to be able to usefully compare source trees:

• so that you can see the difference between last week's and this week's version

• so that you can see the difference between your version and your colleagues version.

• so that you can usefully merge in adaptations from variant source trees.

The tools for doing this are well-established for text source, as are standard practices for improving source code. For example, reformatting - whitespace-only - updates to source code are often deliberately isolated from semantic changes. Similarly, groups of related changes are often made together in changesets.

Such practices facilitate the use of text-based tools, and as a result, management of multiple similar source trees is well understood and documented.

By comparison; if code were stored, for example, in a Word document (or to make the same point more extravagantly, as a bitmap image showing the text), this would be impossible. It would be just as easy for the programmer to read - but you'd lose all the power of automatic comparisons; source trees diverge wildly with every minor change. This wouldn't stop you using version control, and backtracking to previous versions, but your diffs would carry no useful information.

And so with visual programming. Clearly there is some serialization underlying whatever interface you're given, and more than likely that serialization is textual. So in principle, you could keep your programs under version control (though your IDE might not make it very easy.

However, the visual interface is at liberty to rewrite the serialization without consulting you; and what might be a minor, or indeed insignificant change to you (moving a box without changing its connectors; adding one additional connector) might well result in the serialization being completely restructured.

As a result, you end up in the same situation as with your Word-encoded source. You can't compare your workflows with a colleagues. You can't trivially diff your current version with that of 6 months ago, and identify key differences. You can't check it against yesterday's version, and work out what you changed which caused the testsuite breakage that you noticed this morning.

Although version control is the most obvious to me as a useful application of text-munging. There's any number of others that are useful; automatic bug-finding tools; refactoring tools, etc, which rely on the well-understood machinery of text analysis.

Even if some of these may exist in a given visual IDE (and I wouldn't be surprised if a few did), they certainly don't all, nor would it be easy to transfer them between IDEs, since there is no agreed upon standard way to do visual programming.

Source control

The open source movement relies on source to programs being available. What that means in precise terms begins to break down when we move away from the realm of traditional compiled languages; but in any case, what it means to me is the ability to usefully manipulate and inspect the algorithms behind a program.

You can't do this if you're given only the compiled program - all the usefully-human-readable information is in the source code, and is thrown away when compiling; minor changes in source code can result in very different machine code, and decompilers are very imperfect instruments.

In fact, that's not quite true; there's three layers of information. There's the human-readable description of what the code should do. That's probably in someone's head. There's the partly human-, partly machine-readable form in the source code, which can be programmatically manipulated (or "compiled" as we like to say) into alternative forms, one of which is machine code for execution.

When writing in VPLs, you radically lower the utility of the intermediate stage. The human-accessible code is in people's heads, and is transferred to a graphical form that isn't actually very human-readable - at least, not in the sense that I can write tools to do anything with it.

So until people devote as much time to research into software engineering and code management tasks for non-textual code representations, I don't think VPLs are ever going to be properly successful.

In fact, any code that is written in such environments is information that is being essentially thrown away - expertise that is being just as much wasted as if you threw away the source code to your compiled programs.

Finishing this political digression:

The motion mentioned above passed under the rules of the local branch. A motion is now going forward to the UCU national Congress requesting that UCU formally affiliate itself to the NO2ID campaign.

Thanks to everyone who took the time to email a vote. The next stage is Congress itself, which is at the end of May.

Recording this for the benefit of anyone else who comes across this problem:

The Mac OS X Finder can act as a webdav client, allowing data exposed through a webdav interface to be mounted as an apparently native filesystem. This is how, for example, iDisk works.

WebDAV authentication is taken care of the same way as HTTP; for any given operation, the server can demand authentication, by returning a 401 error and requiring an authentication token from the client. This token can be of various forms, but the simplest is Basic authentication - a username/password combination.

And indeed, when you ask to mount a webdav filesystem protected by Basic authentication, Finder will pop up an appropriate dialogue box, and authenticate you.

However - HTTP authentication works per-operation, not per-server. That is, different resources on the same server can have different authentication requirements; indeed different operations on the same resource may have different requirements (for example, user A may be allowed to read and write, user B may only be allowed to read).

Finder doesn't completely understand this; or rather, the UI it presents to deal with this is broken. If you try and mount a directory, only some of whose sub-directories you have access to, then Finder will try and read all of the subdirectories. Given that you don't have access to everything it is trying to read, it will then conclude you don't have access to anything at all, and will only let you read any resources which are completely unauthenticated.

A temporary work-around is on the server to move any such subdirectories elsewhere, and put temporary HTTP redirects in place. This seems to quieten Finder down, and give the desired result. Unfortunately, it doesn't usefully scale where there are many such cases; nor where these permissions are liable to relatively frequent change.

As a result, it's essentially impossible to use Finder as a webdav client for a server with any complex ACL policy.

I've reported this to Apple as bug 5837223.