Christophe Weblog Wiki Code Publications Music
R academia addiction advertising advogato araucaria arduino arm art audiodb azed backquote backward compatibility being human biebl boat cambridge ccl chalicea charpentier charybdis children clisp clos clweb code-walking coffee conference creative crequillon crossword crowdfunding data data science de profundis de ribera debian earl early music ecda ecologue eightsome reels electronics els emacs employability enterprise european lisp symposium excoll eye crossword fairpayinhe ferret financial services flac funding games generalizers ggplot git-annex gnome gnome-shell gnupg gnus goldsmiths gombert google gridsvg gsoc guardian hamelin hea hearing wagner hyperion ikiwiki inquisitor international programmes introversion ircam iterate jago javascript josquin lattice lazyweb leisure leitmotif let-over-lambda lisp london londonr macroexpand-dammit magrittr mango solutions markdown mcclim mercurial meta method-combination middlesex misanthropy misogyny mobile world congress moodle mop multiple viewpoints music-ir musicbrainz musicology mysociety networking nginx nutmeg open access openmcl openml org-mode otterden overlaps pandoc paul personal pgcert phi phones physical computing plane portfolio postgraduate study preprint pretty-printer pretty-printing printers devilry project protocol prototypes quasiquote quote racket rameau raspberry pi readtable release management reproducible research research sales sayit sbcl set-difference shellshape slate slime small world smile machine software specializers spem spoiler startup strike summer of code sustainability svg svgannotation swankr tallis teaching teclo tibco time tmus tours train training ucea unicode union wagner wiki workshop worm wrong number xml

This portion of the ikiwiki installation is for notes organized as a blog, with more permanent observations recorded statically.

hearing wagner data preparations

Last week’s activity – in between the paperwork, the teaching, the paperwork, the paperwork, the teaching and the paperwork – was mostly taken up in preparations for the Hearing Wagner event, part of the AHRC’s Being Human festival.

Being a part of the Being Human festival gave us the opportunity to work to collect data that we wouldn’t otherwise have had access to: because of the fortuitous timing of the Mariinsky Theatre’s production of the Ring at the Birmingham Hippodrome between 5th and 9th November, we were able to convince funders to allow us to offer free tickets to Birmingham Conservatoire students, in exchange for being wired up to equipment measuring their electrodermal activity, blood flow, and hand motion.

Why collect these data? Well, on of the themes of the Transforming Musicology project as a whole is to examine the perception of leitmotive, particularly Wagner’s use of them in the Ring, and the idea behind gathering these data is to have ecologically-valid (in as much as that is possible when there’s a device strapped to you) measurements of participants’ physical responses to the performance, where those physical responses are believed to correlate with emotional arousal. Using those measurements, we can then go looking for signals of responses to leitmotives, or to other musical or production cues: as well as the students attending the performance, some of the research team were present backstage, noting down the times of events in the staging of (subjective) particular significance – lighting changes, for example.

And then all of these data come back to base, and we have to go through the process of looking for signal. And before we can do anything else, we have to make sure that all of our data are aligned to a reference timeline. For each of the operas, we ended up with around 2GB of data files: up to 10 sets of data from the individual participants, sampled at 120Hz or so; times of page turns in the vocal score, noted by a musicologist member of the research team (a coarse approximation to the sound experienced by the participants); timestamped performance annotations, generated by a second musicologist and dramaturge. How to get all of this onto a common timeline?

Well, in the best of all possible worlds, all of the clocks in the system would have been synchronized by ntp, and that synchronization would have been stable and constant throughout the process. In this case, the Panglossians would have been disappointed: in fact none of the various devices was sufficiently stably synchronized with any of the others to be able to get away with no alignment.

Fortunately, the experimental design was carried out by people with a healthy amount of paranoia: the participants were twice asked to clap in unison: once in the backstage area, where there was only speed-of-sound latency to the listeners (effectively negligible), and once when seated in the auditorium, where there was additional latency from the audio feed from the auditorium to backstage. Those claps gave us enough information, on the rather strong assumption that they were actually simultaneous, to tie everything together: the first clap could be found on each individual measuring device by looking at the accelerometer data for the signature, which establishes a common timeline for the measurement data and the musicologists; the second clap gives a measure for the additional latency introduced by the audio feed. Since the participants’ claps weren’t actually simultaneous – despite the participants being music students, and the clap being conducted – we have a small error, but it’s likely to be no more than about one second.

And this week? This week we’ll actually be looking for interesting signal; there’s reason to believe that electrodermal activity (basically, the change in skin conductance due to sweat) is indicative of emotional arousal, and quite a sensitive measure of music-induced emotion. This is by its nature an exploratory study: at least to start with, we’re looking at particular points of interest (specified by musicologists, in advance) for any correlation with biosignal response – and we’ll be presenting initial results about anything we find at the Hearing Wagner event in Birmingham this weekend. The clock is ticking...

edit: see also a similar post on the project blog

reproducible builds - a month ahead of schedule

I think this might be my last blog entry on the subject of building SBCL for a while.

One of the premises behind SBCL as a separate entity from CMUCL, its parent, was to make the result of its build be independent of the compiler used to build it. To a world where separate compilation is the norm, the very idea that building some software should persistently modify the state of the compiler probably seems bizarre, but the Lisp world evolved in that way and Lisp environments (at least those written in themselves) developed build recipes where the steps to construct a new Lisp system from an old one and the source code would depend critically on internal details of both the old and the new one: substantial amounts of introspection on the build host were used to bootstrap the target, so if the details revealed by introspection were no longer valid for the new system, there would need to be some patching in the middle of the build process. (How would you know whether that was necessary? Typically, because the build would fail with a more-or-less – usually more – cryptic error.)

Enter SBCL, whose strategy is essentially to use the source files first to build an SBCL!Compiler running in a host Common Lisp implementation, and then to use that SBCL!Compiler to compile the source files again to produce the target system. This requires some contortions in the source files: we must write enough of the system in portable Common Lisp so that an arbitrary host can execute SBCL!Compiler to compile SBCL-flavoured sources (including the standard headache-inducing (defun car (list) (car list)) and similar, which works because SBCL!Compiler knows how to compile calls to car).

How much is “enough” of the system? Well, one answer might be when the build output actually works, at least to the point of running and executing some Lisp code. We got there about twelve years ago, when OpenMCL (as it was then called) compiled SBCL. And yet... how do we know there aren't odd differences that depend on the host compiler lurking, which will not obviously affect normal operation but will cause hard-to-debug trouble later? (In fact there were plenty of those, popping up at inopportune moments).

I’ve been working intermittently on dealing with this, by attempting to make the Common Lisp code that SBCL!Compiler is written in sufficiently portable that executing it on different implementations generates bitwise-identical output. Because then, and only then, can we be confident that we are not depending in some unforseen way on a particular implementation-specific detail; if output files are different, it might be a harmless divergence, for example a difference in ordering of steps where neither depends on the other, or it might in fact indicate a leak from the host environment into the target. Before this latest attack at the problem, I last worked on it seriously in 2009, getting most of the way there but with some problems remaining, as measured by the number of output files (out of some 330 or so) whose contents differed depending on which host Common Lisp implementation SBCL!Compiler was running on.

Over the last month, then, I have been slowly solving these problems, one by one. This has involved refining what is probably my second most useless skill, working out what SBCL fasl files are doing by looking at their contents in a text editor, and from that intuiting the differences in the implementations that give rise to the differences in the output files. The final pieces of the puzzle fell into place earlier this week, and the triumphant commit announces that as of Wednesday all 335 target source files get compiled identically by SBCL!Compiler, whether that is running under Clozure Common Lisp (32- or 64-bit versions), CLISP, or a different version of SBCL itself.

Oh but wait. There is another component to the build: as well as SBCL!Compiler, we have SBCL!Loader, which is responsible for taking those 335 output files and constructing from them a Lisp image file which the platform executable can use to start a Lisp session. (SBCL!Loader is maybe better known as “genesis”; but it is to load what SBCL!Compiler is to compile-file). And it was slightly disheartening to find that despite having 335 identical output files, the resulting cold-sbcl.core file differed between builds on different host compilers, even after I had remembered to discount the build fingerprint constructed to be different for every build.

Fortunately, the actual problem that needed fixing was relatively small: a call to maphash, which (understandably) makes no guarantees about ordering, was used to affect the Lisp image data directly. I then spent a certain amount of time being thoroughly confused, having managed to construct for myself a Lisp image where the following forms executed with ... odd results:

(loop for x being the external-symbols of "CL" count 1)
; => 1032
(length (delete-duplicates (loop for x being the external-symbols of "CL" collect x)))
; => 978

(shades of times gone by). Eventually I realised that

(unless (member (package-name package) '("COMMON-LISP" "KEYWORD" :test #'string=))

was not the same as

(unless (member (package-name package) '("COMMON-LISP" "KEYWORD") :test #'string=)

and all was well again, and as of this commit the cold-sbcl.core output file is identical no matter the build host.

It might be interesting to survey the various implementation-specific behaviours that we have run into during the process of making this build completely repeatable. The following is a probably non-exhaustive list – it has been twelve years, after all – but maybe is some food for thought, or (if you have a particularly demonic turn of mind) an ingredients list for a maximally-irritating CL implementation...

  • Perhaps most obviously, various constants are implementation-defined. The ones which caused the most trouble were undoubtably most-positive-fixnum and most-negative-fixnum – particularly since they could end up being used in ways where their presence wasn’t obvious. For example, (deftype fd () `(integer 0 ,most-positive-fixnum)) has, in the SBCL build process, a subtly different meaning from (deftype fd () (and fixnum unsigned-byte)) – in the second case, the fd type will have the intended meaning in the target system, using the target’s fixnum range, while in the first case we have no way of intercepting or translating the host’s value of most-positive-fixnum. Special mentions go to array-dimension-limit, which caused Bill Newman to be cross on the Internet, and to internal-time-units-per-second; I ended up tracking down one difference in output machine code from a leak of the host’s value of that constant into target code.
  • Similarly, random and sxhash quite justifiably differ between implementations. The practical upshot of that is that these functions can’t be used to implement a cache in SBCL!Compiler, because the access patterns and hence the patterns of cache hits and misses will be different depending on the host implementation.
  • As I’ve already mentioned, maphash does not iterate over hash-table contents in a specified order, and in fact that order need not be deterministic; similarly, with-package-iterator can generate symbols in arbitrary orders, and set operations (intersection, set-difference and friends) will return the set as a list whose elements are in an arbitrary order. Incautious use of these functions tended to give rise to harmless but sometimes hard-to-diagnose differences in output; the solution was typically to sort the iteration output before operating on any of it, to introduce determinism...
  • ... but it was possible to get that wrong in a harder-to-detect way, because sort isnot specified to be stable. In some implementations, it actually is a stable sort in some conditions, but for cases where it’s important to preserve an already-existing partial order, stable-sort is the tool for the job.
  • The language specification explicitly says that the initial contents of uninitialized arrays are undefined. In most implementations, at most times, executing (make-array 8 :element-type (unsigned-byte 8)) will give a zero-filled array, but there are circumstances in some implementations where the returned array will have arbitrary data.
  • Not only are some constants implementation-defined, but so also are the effects of normal operation on some variables. *gensym-counter* is affected by macroexpansion if the macro function calls gensym, and implementations are permitted to macroexpand macros an arbitrary number of times. That means that our use of gensym needs to be immune to whatever the host implementation’s macroexpansion and evaluation strategy is.
  • The object returned by byte to represent a bitfield with size and position is implementation-defined. Implementations (variously) return bitmasks, conses, structures, vectors; host return values of byte must not be used during the execution of SBCL!Compiler. More subtly, the various boole-related constants (boole-and and friends) also need special treatment; at one point, their host values were used when SBCL!Compiler compiled the boole function itself, and it so happens that CLISP and SBCL both represent the constants as integers between 0 and 15... but with a different mapping between operation and integer.
  • my last blog entry talked about constant coalescing, and about printing of (quote foo). In fact printing in general has been a pain, and there are still significant differences in interpretation or at least in implementation of pretty-printing: to the extent that at one point we had to minimize printing at all in order for the build to complete under some implementations.
  • there are a number of things which are implementation-defined but have caused a certain amount of difficulty. Floating point in general is a problem, not completely solved (SBCL will not build correctly if its host doesn’t have distinct single- and double-float types that are at least approximately IEEE754-compliant). Some implementations lack denormalized numbers; some do not expose signed zeros to the user; and some implementations compute (log 2d0 10d0) more accurately than others, including SBCL itself, do. The behaviour of the host implementation on legal but dubious code is also potentially tricky: SBCL’s build treats full warnings as worthy of stopping, but some hosts emit full warnings for constructs that are tricky to write in other ways: for example to write portable code to handle multiple kinds of string, one might write (typecase string (simple-base-string ...) ((simple-array character (*)) ...)) (string ...)) but some implementations emit full warnings if a clause in a typecase is completely shadowed by other clauses, and if base-char and character are identical in that implementation the typecase above will signal.

There were probably other, more minor differences between implementations, but the above list gives a flavour of the things that needed doing in order to get to this point, where we have some assurance that our code is behaving as intended. And all of this is a month ahead of my self-imposed deadline of SBCL’s 15th birthday: SBCL was announced to the world on December 14th, 1999. (I’m hoping to be able to put on an sbcl15 workshop in conjunction with the European Lisp Symposium around April 20th/21st/22nd – if that sounds interesting, please pencil the dates in the diary and let me know...)

still working on reproducible builds

It’s been nearly fifteen years, and SBCL still can’t be reliably built by other Lisp compilers.

Of course, other peoples’ definition of “reliably” might differ. We did achieve successful building under unrelated Lisp compilers twelve years ago; there were a couple of nasty bugs along the way, found both before and after that triumphant announcement, but at least with a set of compilers whose interpretation of the standard was sufficiently similar to SBCL’s own, and with certain non-mandated but expected features (such as the type (array (unsigned-byte 8) (*)) being distinct from simple-vector, and single-float being distinct from double-float), SBCL achieved its aim of being buildable on a system without an SBCL binary installed (indeed, using CLISP or XCL as a build host, SBCL could in theory be bootstrapped starting with only gcc).

For true “reliability”, though, we should not be depending on any particular implementation-defined features other than ones we actually require – or if we are, then the presence or absence of them should not cause a visible difference in the resulting SBCL. The most common kind of leak from the host lisp to the SBCL binary was the host’s value of most-positive-fixnum influencing the target, causing problems from documentation errors all the way up to type errors in the assembler. Those leaks were mostly plugged a while ago, though they do recur every so often; there are other problems, and over the last week I spent some time tracking down three of them.

The first: if you’ve ever done (apropos "PRINT") or something similar at the SBCL prompt, you might wonder at the existence of functions named something like SB-VM::|CACHED-FUN--PINSRB[(EXT-2BYTE-XMM-REG/MEM ((PREFIX (QUOTE (102))) (OP1 (QUOTE (58))) (OP2 (QUOTE (32))) (IMM NIL TYPE (QUOTE IMM-BYTE))) (QUOTE (NAME TAB REG , REG/MEM ...)))]-EXT-2BYTE-XMM-REG/MEM-PRINTER|.

What is going on there? Well, these functions are a part of the disassembler machinery; they are responsible for taking a certain amount of the machine code stream and generating a printed representation of the corresponding assembly: in this case, for the PINSRB instruction. Ah, but (in most instruction sets) related instructions share a fair amount of structure, and decoding and printing a PINSRD instruction is basically the same as for PINSRB, with just one #x20 changed to a #x22 – in both cases we want the name of the instruction, then a tab, then the destination register, a comma, the source, another comma, and the offset in the destination register. So SBCL arranges to reuse the PINSRB instruction printer for PINSRD; it maintains a cache of printer functions, looked up by printer specification, and reuses them when appropriate. So far, so normal; the ugly name above is the generated name for such a function, constructed by interning a printed, string representation of some useful information.

Hm, but wait. See those (QUOTE (58)) fragments inside the name? They result from printing the list (quote (58)). Is there a consensus on how to print that list? Note that *print-pretty* is bound to nil for this printing; prior experience has shown that there are strong divergences between implementations, as well as long-standing individual bugs, in pretty-printer support. So, what happens if I do (write-to-string '(quote foo) :pretty nil)?

  • SBCL: "(QUOTE FOO)", unconditionally
  • CCL: "'FOO" by default; "(QUOTE FOO)" if ccl:*print-abbreviate-quote* is set to nil
  • CLISP: "'FOO", unconditionally (I read the .d code with comments in half-German to establish this)

So, if SBCL was compiled using CLISP, the name of the same function in the final image would be SB-VM::|CACHED-FUN--PINSRB[(EXT-2BYTE-XMM-REG/MEM ((PREFIX '(102)) (OP1 '(58)) (OP2 '(32)) (IMM NIL TYPE 'IMM-BYTE)) '(NAME TAB REG , REG/MEM ...))]-EXT-2BYTE-XMM-REG/MEM-PRINTER|. Which is shorter, and maybe marginally easier to read, but importantly for my purposes is not bitwise-identical.

Thus, here we have a difference between host Common Lisp compilers which leaks over into the final image, and it must be eliminated. Fortunately, this was fairly straightforward to eliminate; those names are never in fact used to find the function object, so generating a unique name for functions based on a counter makes the generated object file bitwise identical, no matter how the implementation prints two-element lists beginning with quote.

The second host leak is also related to quote, and to our old friend backquote – though not related in any way to the new implementation. Consider this apparently innocuous fragment, which is a simplified version of some code to implement the :type option to defstruct:

(macrolet ((def (name type n)
                (declaim (inline ,name (setf ,name)))
                (defun ,name (thing)
                  (declare (type simple-vector thing))
                  (the ,type (elt thing ,n)))
                (defun (setf ,name) (value thing)
                  (declare (type simple-vector thing))
                  (declare (type ,type value))
                  (setf (elt thing ,n) value)))))
  (def foo fixnum 0)
  (def bar string 1))

What’s the problem here? Well, the functions are declaimed to be inline, so SBCL records their source code. Their source code is generated by a macroexpander, and so is made up of conses that are generated programmatically (as opposed to freshly consed by the reader). That source code is then stored as a literal object in an object file, which means in practice that instructions for reconstructing a similar object are dumped, to be executed when the object file is processed by load.

Backquote is a reader macro that expands into code that, when evaluated, generates list structure with appropriate evaluation and splicing of unquoted fragments. What does this mean in practice? Well, one reasonable implementation of reading `(type ,type value) might be:

(cons 'type (cons type '(value)))

and indeed you might (no guarantees) see something like that if you do

(macroexpand '`(type ,type value))

in the implementation of your choice. Similarly, reading `(setf (elt thing ,n) value) will eventually generate code like

(cons 'setf (cons (cons 'elt (list 'thing n)) '(value)))

Now, what is “similar”? In this context, it has a technical definition: it relates two objects in possibly-unrelated Lisp images, such that they can be considered to be equivalent despite the fact that they can’t be compared:

similar adj. (of two objects) defined to be equivalent under the similarity relationship.

similarity n. a two-place conceptual equivalence predicate, which is independent of the Lisp image so that two objects in different Lisp images can be understood to be equivalent under this predicate. See Section 3.2.4 (Literal Objects in Compiled Files).

Following that link, we discover that similarity for conses is defined in the obvious way:

Two conses, S and C, are similar if the car of S is similar to the car of C, and the cdr of S is similar to the cdr of C.

and also that implementations have some obligations:

Objects containing circular references can be externalizable objects. The file compiler is required to preserve eqlness of substructures within a file.

and some freedom:

With the exception of symbols and packages, any two literal objects in code being processed by the file compiler may be coalesced if and only if they are similar [...]

Put this all together, and what do we have? That def macro above generates code with similar literal objects: there are two instances of '(value) in it. A host compiler may, or may not, choose to coalesce those two literal '(value)s into a single literal object; if it does, the inline expansion of foo (and bar) will have a circular reference, which must be preserved, showing up as a difference in the object files produced during the SBCL build. The fix? It’s ugly, but portable: since we can’t stop an aggressive compiler from coalescing constants which are similar but not identical, we must make sure that any similar substructure is in fact identical:

(macrolet ((def (name type n)
             (let ((value '(value)))
                  (declaim (inline ,name (setf ,name)))
                  (defun ,name (thing)
                    (declare (type simple-vector thing))
                    (the ,type (elt thing ,n)))
                  (defun (setf ,name) (value thing)
                    (declare (type simple-vector thing))
                    (declare (type ,type . ,value))
                    (setf (elt thing ,n) . ,value)))))
  (def foo fixnum 0)
  (def bar string 1))

Having dealt with a problem with quote, and a problem with backquote, what might the Universe serve up for my third problem? Naturally, it would be a problem with a code walker. This code walker is somewhat naïve, assuming as it does that its body is made up of forms or tags; it is the assemble macro, which is used implicitly in the definition of VOPs (reusable assembly units); for example, like

(assemble ()
  (move ptr object)
  (zeroize count)
  (inst cmp ptr nil-value)
  (inst jmp :e DONE)
  (loadw ptr ptr cons-cdr-slot list-pointer-lowtag)
  (inst add count (fixnumize 1))
  (inst cmp ptr nil-value)
  (inst jmp :e DONE)
  (%test-lowtag ptr LOOP nil list-pointer-lowtag)
  (error-call vop 'object-not-list-error ptr)

which generates code to compute the length of a list. The expander for assemble scans its body for any atoms, and generates binding forms for those atoms to labels:

(let ((new-labels (append labels
                          (set-difference visible-labels inherited-labels))))
  `(let (,@(mapcar (lambda (name) `(,name (gen-label))) new-labels))

The problem with this, from a reproducibility point of view, is that set-difference (and the other set-related functions: union, intersection, set-exclusive-or and their n-destructive variants) do not return the sets with a specified order – which is fine when the objects are truly treated as sets, but in this case the LOOP and DONE label objects ended up in different stack locations depending on the order of their binding. Consequently the machine code for the function emitting code for computing a list’s length – though not the machine code emitted by that function – would vary depending on the host’s implementation of set-difference. The fix here was to sort the result of the set operations, knowing that all the labels would be symbols and that they could be treated as string designators.

And after all this is? We’re still not quite there: there are three to four files (out of 330 or so) which are not bitwise-identical for differing host compilers. I hope to be able to rectify this situation in time for SBCL’s 15th birthday...

interesting pretty-printer bug

One of SBCL’s Google Summer of Code students, Krzysztof Drewniak (no relation) just got to merge in his development efforts, giving SBCL a far more complete set of Unicode operations.

Given that this was the merge of three months’ out-of-tree work, it’s not entirely surprising that there were some hiccups, and indeed we spent some time diagnosing and fixing a 1000-fold slowdown in char-downcase. Touch wood, all seems mostly well, except that Jan Moringen reported that, when building without the :sb-unicode feature (and hence having a Lisp with 8-bit characters) one of the printer consistency tests was resulting in an error.

Tracking this down was fun; it in fact had nothing in particular to do with the commit that first showed the symptom, but had been lying latent for a while and had simply never shown up in automated testing. I’ve expressed my admiration for the Common Lisp standard before, and I’ll do it again: both as a user of the language and as an implementor, I think the Common Lisp standard is a well-executed document. But that doesn’t stop it from having problems, and this is a neat one:

When a line break is inserted by any type of conditional newline, any blanks that immediately precede the conditional newline are omitted from the output and indentation is introduced at the beginning of the next line.

(from pprint-newline)

For the graphic standard characters, the character itself is always used for printing in #\ notation---even if the character also has a name[5].

(from CLHS

Space is defined to be graphic.

(from CLHS glossary entry for ‘graphic’)

What do these three requirements together imply? Imagine printing the list (#\a #\b #\c #\Space #\d #\e #\f) with a right-margin of 17:

(write-to-string '(#\a #\b #\c #\Space #\d #\e #\f) :pretty t :right-margin 17)
; => "(#\\a #\\b #\\c #\\
; #\\d #\\e #\\f)"

The #\Space character is defined to be graphic; therefore, it must print as #\ rather than #\Space; if it happens to be printed just before a conditional newline (such as, for example, generated by using pprint-fill to print a list), the pretty-printer will helpfully remove the space character that has just been printed before inserting the newline. This means that a #\Space character, printed at or near the right margin, will be read back as a #\Newline character.

It’s interesting to see what other implementations do. CLISP 2.49 in its default mode always prints #\Space; in -ansi mode it prints #\ but preserves the space even before a conditional newline. CCL 1.10 similarly preserves the space; there’s an explicit check in output-line-and-setup-for-next for an “escaped” space (and a comment that acknowledges that this is a heuristic that can be wrong in the other direction). I’m not sure what the best fix for this is; it’s fairly clear that the requirements on the printer aren’t totally consistent. For SBCL, I have merged a one-line change that makes the printer print using character names even for graphic characters, if the *print-readably* printer control variable is true; it may not be ideal that print/read round-tripping was broken in the normal case, but in the case where it’s explicitly been asked for it is clearly wrong.

settings for gnome-shell extension

A long, long time ago, I configured my window manager. What can I say? I was a student, with too much free time; obviously hoursdays spent learning some configuration file format and tweaking some aspect of behaviour would be repaid many times over the course of a working life. I suppose one thing it led to was my current career, so it’s probably not all a loss.

However, the direct productivity benefits almost certainly were a chimera; unfortunately, systems (hardware and software) changed too often for the productivity benefit (if any) to amortize the fixed set-up time, and so as a reaction I stopped configuring my window manager. In fact, I went the other way, becoming extremely conservative about upgrades of anything at all; I had made my peace with GNOME 2, accepting that there was maybe enough configurability and robustness in the 2.8 era or so for me not to have to change too much, and then not changing anything.

Along comes GNOME 3, and there are howls all over the Internet about the lack of friendly behaviour – “friendly” to people like me, who like lots of terminals and lots of editor buffers; there wasn’t much of an outcry from other people with more “normal” computing needs; who knows why? In any case, I stuck with GNOME 2 for a long time, eventually succumbing at the point of an inadvisable apt-get upgrade and not quite hitting the big red ABORT button in time.

So, GNOME 3. I found that I shared a certain amount of frustration with the vocal crowd: dynamic, vertically-arranged workspaces didn’t fit my model; I felt that clicking icons should generate new instances of applications rather than switch to existing instances, and so on. But, in the same timeframe, I adopted a more emacs-centric workflow, and the improvements of the emacs daemon meant that I was less dependent on particular behaviours of window manager and shell, so I gave it another try, and, with the right extensions, it stuck.

The right extensions? “What are those?” I hear you cry. Well, in common with illustrious Debian Project Leaders past, I found that a tiling extension made many of the old focus issues less pressing. My laptop is big enough, and I have enough (fixed) workspaces, that dividing up each screen between applications mostly works. I also have a bottom panel, customized to a height of 0 pixels, purely to give me the fixed number of workspaces; the overview shows them in a vertical arrangement, but the actual workspace arrangement is the 2x4 that I’m used to.

One issue with a tiled window arrangement, though, is an obvious cue to which window has focus. I have also removed all window decorations, so the titlebar or border don’t help with this; instead, a further extension to shade inactive windows helps to minimize visual distraction. And that’s where the technical part of this blog entry starts...

One of the things I do for my employer is deliver a module on Perception and Multimedia Computing. In the course of developing that module, I learnt a lot about how we see what we see, and also how digital displays work. And one of the things I learnt to be more conscious about was attention: in particular, how my attention can be drawn (for example, I can not concentrate on anything where there are animated displays, such as are often present in semi-public spaces, such as bars or London City airport.)

The shade inactive windows extension adds a brightness-reducing effect to windows without focus. So, that was definitely useful, but after a while I noticed that emacs windows with some text in error-face (bold, bright red) in them were still diverting my attention, even when they were unfocussed and therefore substantially dimmed.

So I worked on a patch to the extension to add a saturation-reducing effect in addition to reducing the brightness. And that was all very well – a classic example of taking code that almost does what you want it to do, and then maintenance-programming it into what you really want it to do – until I also found that the hard-wired time over which the effect took hold (300ms) was a bit too long for my taste, and I started investigating what it would take to make these things configurable.

Some time later, after exploring the world with my most finely-crafted google queries, I came to the conclusion that there was in fact no documentation for this at all. The tutorials that I found were clearly out-dated, and there were answers to questions on various forums whose applicability was unclear. This is an attempt to document the approach that worked for me; I make no claims that this is ‘good’ or even acceptable, but maybe there’s some chance that it will amortize the cost of the time I spent flailing about over other people wanting to customize their GNOME shell.

The first thing that something, anything with a preference needs, is a schema for that preference. In this instance, we’re starting with the shade-inactive-windows in the namespace, so our schema will have a path that begins “/fi/iki/hepaajan/shade-inactive-windows”, and we're defining preferences, so let’s add “/preferences” to that.

<?xml version="1.0" encoding="UTF-8"?>
  <schema path="/fi/iki/hepaajan/shade-inactive-windows/preferences/"

a schema also needs an id, which should probably resemble the path


except that there are different conventions for hierarchy (. vs /).


and then here’s a cargo-culted gettext thing, which is probably relevant if the rest of the schema will ever be translated into any non-English language.

In this instance, I am interested in a preference that can be used to change the time over which the shading of inactie windows happens. It’s probably easiest to define this as an integer (the "i" here; other GVariant types are possible):

    <key type="i" name="shade-time">

which corresponds to the number of milliseconds

      <summary>Time in milliseconds over which shading occurs</summary>
        The time over which the shading effect is applied, in milliseconds.

which we will constrain to be between 0 and 1000, so that the actual time is between 0s and 1s, with a default of 0.3s:

      <range min="0" max="1000"/>

and then there's some XML noise


and that completes our schema. For reasons that will become obvious later, we need to store that schema in a directory data/glib-2.0/schemas relative to our base extension directory; giving it a name that corresponds to the schema id (so fi.iki.hepaajan.shade-inactive-windows.preferences.gschema.xml in this case) is wise, but probably not essential. In order for the schema to be useful later, we also need to compile it: that’s as simple as executing glib-compile-schemas . within the schemas directory, which should produce a gschemas.compiled file in the same directory.

Then, we also need to adapt the extension in question to lookup a preference value when needed, rather than hard-coding the default value. I have no mental model of the namespacing, or other aspects of the environment, applied to GNOME shell extensions’ javascript code, so being simultaneously conservative and a javascript novice I added a top-level variable unlikely to collide with anything:

var ShadeInactiveWindowsSettings = {};
function init() {

The extension previously didn’t need to do anything on init(); now, however, we need to initialize the settings object, including looking up our schema to discover what settings there are. But where is our schema? Well, if we’re running this extension in-place, or as part of a user installation, we want to look in data/glib-2.0/schemas/ relative to our own path; if we have performed a global installation, the schema will presumably be in a path that is already searched for by the default schema finding methods. So...

    var schemaDir = ExtensionUtils.getCurrentExtension().dir.get_child('data').get_child('glib-2.0').get_child('schemas');
    var schemaSource = Gio.SettingsSchemaSource.get_default();

    if(schemaDir.query_exists(null)) {
        schemaSource = Gio.SettingsSchemaSource.new_from_directory(schemaDir.get_path(), schemaSource, false);

... we distinguish between those two cases by checking to see if we can find a data/glib-2.0/schemas/ directory relative to the extension’s own directory; if we can, we prepend that directory to the schema source search path. Then, we lookup our schema using the id we gave it, and initialize a new object with that schema.

    var schemaObj = schemaSource.lookup('fi.iki.hepaajan.shade-inactive-windows.preferences', true);
    if(!schemaObj) {
        throw new Error('failure to look up schema');
    ShadeInactiveWindowsSettings = new Gio.Settings({ settings_schema: schemaObj });

Then, whenever we use the shade time in the extension, we must make sure to look it up afresh:

var shade_time = ShadeInactiveWindowsSettings.get_int('shade-time') / 1000;

in order that any changes made by the user take effect immediately. And that’s it. There’s an additional minor wrinkle, in that altering that configuration variable is not totally straightforward; dconf and gettings also need to be told where to look for their schema; that’s done using the XDG_DATA_DIRS configuration variable. For example, once the extension is installed locally, you should be able to run

XDG_DATA_DIRS=$HOME/.local/gnome-shell/extensions/$XDG_DATA_DIRS dconf

and then navigate to the fi/iki/hepaajan/shade-inactive-windows/preferences schema and alter the shade-time preference entry.

Hooray! After doing all of that, we have wrestled things into being configurable: we can use the normal user preferences user interface to change the time over which the shading animation happens. I’m not going to confess how many times I had to restart my gnome session, insert logging code, look at log files that are only readable by root, and otherwise leave my environment; I will merely note that we are quite a long way away from the “scriptable user interface” – and that if you want to do something similar (not identical, but similar) in an all-emacs world, it might be as simple as evaluating these forms in your *scratch* buffer...

(set-face-attribute 'default nil :background "#eeeeee")
(defvar my/current-buffer-background-overlay nil)

(defun my/lighten-current-buffer-background ()
  (unless (window-minibuffer-p (selected-window))
    (unless my/current-buffer-background-overlay
      (setq my/current-buffer-background-overlay (make-overlay 1 1))
      (overlay-put my/current-buffer-background-overlay
       'face '(:background "white")))
    (overlay-put my/current-buffer-background-overlay 'window
    (move-overlay my/current-buffer-background-overlay
                  (point-min) (point-max))))
(defun my/unlighten-current-buffer-background ()
  (when my/current-buffer-background-overlay
    (delete-overlay my/current-buffer-background-overlay)))

(add-hook 'pre-command-hook #'my/unlighten-current-buffer-background) 
(add-hook 'post-command-hook #'my/lighten-current-buffer-background)
inquisitor 1352

I think I’ve got this one, but it’s a little hard to tell. Starting off with a completely empty grid is rarely straightforward (I suppose that’s the point...) and it wasn’t until late in the process, after getting 9dn and most of the across clues, that I started being able to make real deductions about the layout; on the plus side, maybe to make this crossword tractable at all, the clues themselves were noticeably more straightforward than usual. I feel as if I might be missing a little something about the end of the rubric; unless it really means exactly what it says (you don’t strictly need the numbers in the grid), I don’t know what it actually means.

Identity Crisis

The grid should be appropriately filled by correctly locating the solutions to the normal, correctly numbered clues. Numbers are not essential in the completed grid, which displays 180° symmetry, but their inclusion may assist solvers in arriving at the solution.

  • 8ac: TAMED: A little bit about me being under control: TAD (a little bit) around ME
  • 10ac: SEVERE: Very bad cut to top of ear...: SEVER (cut) + E(ar)
  • 11ac: SLASH: ... and another small one on edge of eyelid: S(mall) + LASH (one on edge of eyelid)
  • 13ac: GREASED: Did this to palms, maybe – as in avarice: AS in GREED (avarice)
  • 14ac: INKED: Shut one eye with hit, made black: WINKED (shut one eye) - W(ith)
  • 15ac: IDAHO: State out loud: “I owned a sort of garden tool”: I’D A HOE
  • 16ac: KNEE: King born to genuflect? Once: K(ing) + NEE (born)
  • 17ac: TUNER: It picks up radio signals: the sounds of fish?: TUNA
  • 19ac: BATES: Wild beast rages: *(BEAST)
  • 22ac: TANA: Shrew in a cop shop?: Double Definition
  • 23ac: PEERS: Looks short-sightedly at Members of the Upper House: Double Definition
  • 24ac: TILTH: Tent put on hospital land: TILT (tent) + H(ospital)
  • 25ac: ORLEANS: Fabric – or thin bit of stuff: OR + LEAN (thin) + S(tuff)
  • 26ac: DIET: Finish with last of burnt food?: DIE (finish) + (burn)T
  • 27ac: TEARY: Crying: “Rent boy’s rear end!”: TEAR (rent) + (bo)Y
  • 28ac: KICKER: Player showing an unpaired card: Double Definition
  • 29ac: BUYER: First of emeralds set with ruby for customer: E(meralds) in *(RUBY)
  • 30ac: LOPED: Took a running jump – and cut off half of cheese!: LOP (cut) + ED(am) (half of cheese)
  • 1dn: SAVANNA: Rescue nearly ready for old Asian in African landscape: SAV(e) (rescue) + ANNA (old Indian “ready”, i.e. money)
  • 2dn: LESSEE: The French notice one who rents: LES (fr. “the”) + SEE (notice)
  • 3dn: IDLED: Didn’t do very much, as I’d been out in front: I’Ð LED
  • 4dn: TRAGIC: Time to give up Havana, perhaps? It’s very sad: T(ime) + CIGAR (Havana, perhaps) reversed
  • 5dn: REHEAT: Theatre not started? Organise warm-up: *((t)HEATRE)
  • 7dn: DEODAR: Fix up cutting very expensive tree: DO (fix) reversed in DEAR (very expensive)
  • 9dn: MARKETPLACE: World of commerce in which metal-packer’s involved: *(METALPACKER)
  • 18dn: ENTERED: Went in and perhaps uprooted 7, for example, in the finale: *(TREE) (7, for example) in END (finale)
  • 20dn: SEATED: Teased terribly, not standing for it!: *(TEASED)
  • 21dn: ASSAIL: Attack like a ship: AS SAIL
  • 22dn: TIDY UP: Put everything back three times a day? Yes! (2 words): TID (three times a day) + YUP (yes)
  • 24dn: TURBO: Jet-powered mollusc?: Double Definition

I had alternative solutions for 28ac (JOKER) and 24dn (SQUID) which misled me for a while, but once I had most of the pieces, putting them together wasn’t too bad; once you’ve deduced that 9dn must be on the second row, it’s easy to put together plausible crossings for it, and the rest follows. That gives the notion that the crossword is undecided between BARS (12ac) and BLOCKS (19dn), and hence has a SPLIT PERSONALITY (1ac/6dn).

All told, then:


I wonder whether it’s wise for a new setter (“IQ debutant Jago”, according to Nimrod) to set a crossword whose theme is, basically, crosswords? To me, it doesn’t feel quite special enough – unless, of course, I have missed something in the rubric to suggest an extra step?

code walking for pipe sequencing

Since it seems still topical to talk about Lisp and code-transformation macros, here’s another worked example – this time inspired by the enthusiasm for the R magrittr package.

The basic idea behind the magrittr package is, as Hadley said at EARL2014, to convert from a form of code where arguments to the same function are far apart to one where they’re essentially close together; the example he presented was converting

      filter(babynames, name == "Hadley"),
    total = sum(n)


b0 <- babynames
b1 <- filter(b0, name == "Hadley")
b2 <- group_by(b1, year)
b3 <- summarise(b2, total = sum(n))
b4 <- arrange(b3, desc(year))

only without the danger of mistyping one of the variable names along the way and failing to perform the computation that was intended.

R, as I have said before, is a Lisp-1 with weird syntax and wacky evaluation semantics. One of the things that ordinary user code can do is inspect the syntactic form of its arguments, before evaluating them. This means that when looking at a fragment of code such as

foo(bar(2,3), 4)

where a call-by-value language would first evaluate bar(2,3), then call foo with two arguments (the value resulting from the evaluation, and 4), R instead uses a form of call-by-need evaluation, and also provides operators for inspecting the promise directly. This means R users can do such horrible things as

foo <- function(x) {
    tmp <- substitute(x)
    sgn <- 1
    while(class(tmp) == "(") {
        tmp <- tmp[[2]]
        sgn <- sgn * -1
    sgn * eval.parent(tmp)
foo(3) # 3
foo((3)) # -3
foo(((3))) # 3
foo((((3)))) # -3 (isn’t this awesome?  I did say “wacky”)

In the case of magrittr, the package authors have taken advantage of this to invent some new syntax; the pipe operator %>% is charged with inserting its first argument (its left-hand side, in normal operation) as the first argument to the call of its second argument (right-hand side). Hadley’s example is

babynames %>%
  filter(name == "Hadley") %>%
  group_by(year) %>%
  summarise(total = sum(n)) %>%

and this is effective because the data flow in this case really is a pipeline: there's a dataset, which needs filtering, then grouping, then summarization, then sorting, and each operation works on the result of the previous. This already needs to inspect the syntactic form of the argument; an additional feature is recognizing the presence of .s in the call, and placing the left-hand side value in that argument position instead of as the first argument if it is present.

In Common Lisp, there are some piping or chaining operators out there (e.g. one two three (search for ablock) four and probably many others), and they do well enough. However! They mostly suffer from similar problems that we’ve seen before: doing code transformations with not quite enough understanding of the semantics of the code that they’re transforming; again, that’s fine for normal use, but for didactic purposes let’s pretend that we really care about this.

The -> macro from is basically the same as the magrittr %>% operator: it converts symbols in the pipeline to function calls, and places the result of the previous evaluation as the first argument of the current operator, except if a $ is present in the arguments, in which case it replaces that. (This version doesn’t support more than one $ in the argument list; it would be a little bit of a pain to support that, needing a temporary name, but it’s straightforward in principle).

Since the -> macro does its job, a code-walker implementation isn’t strictly necessary: pure syntactic manipulation is good enough, and if it’s used with just the code it expects, it will do it well. It is of course possible to express what it does using a code-walker; we’ll fix the multiple-$ ‘bug’ along the way, by explicitly introducing bindings rather than replacements of symbols:

(defmacro -> (form &body body)
  (labels ((find-$ (form env)
             (sb-walker:walk-form form env
              (lambda (f c e)
                  ((eql f '$) (return-from find-$ t))
                  ((eql f form) f)
                  (t (values f t)))))
           (walker (form context env)
               ((symbolp form) (list form))
               ((atom form) form)
               (t (if (find-$ form env)
                      (values `(setq $ ,form) t)
                      (values `(setq $ ,(list* (car form) '$ (cdr form))) t))))))
    `(let (($ ,form))
       ,@(mapcar (lambda (f) (sb-walker:walk-form f nil #'walker)) body))))

How to understand this implementation? Well, clearly, we need to understand what sb-walker:walk does. Broadly, it calls the walker function (its third argument) on successive evaluated subforms of the original form (and on variable names set by setq); the primary return value is used as the interim result of the walk, subject to further walking (macroexpansion and walking of its subforms) except if the second return value from the walker function is t.

Now, let’s start with the find-$ local function: its job is to walk a form, and returns t if it finds a $ variable to be evaluated at toplevel and nil otherwise. It does that by returning t if the form it’s given is $; otherwise, if the form it’s given is the original form, we need to walk its subforms, so return f; otherwise, return its form argument f with a secondary value of t to inhibit further walking. This operation is slightly at odds with the use of a code walker: we are explicitly not taking advantage of the fact that it understands the semantics of the code it’s walking. This might explain why the find-$ function itself looks a bit weird.

The walker local function is responsible for most of the code transformation. It binds $ to the value of the first form, then repeatedly sets $ to the value of successive forms, rewritten to interpolate a $ in the first argument position if there isn’t one in the form already (as reported by find-$). If any of the forms is a symbol, it gets listified and subsequently re-walked. Thus

(macroexpand-1 '(-> "THREE" string-downcase (char 0)))
; => (LET (($ "THREE"))
;      (SETQ $ (CHAR $ 0))),
;    T

So far, so good. Now, what could we do with a code-walker that we can’t without? Well, the above implementation of -> supports chaining simple function calls, so one answer is “chaining things that aren’t just function calls”. Another refinement is to support eliding the insertion of $ when there are any uses of $ in the form, not just as a bare argument. Looking at the second one first, since it’s less controversial:

(defmacro -> (form &body body)
  (labels ((find-$ (form env)
             (sb-walker:walk-form form env
              (lambda (f c e)
                  ((and (eql f '$) (eql c :eval))
                   (return-from find-$ t))
                  (t f))))
           (walker (form context env)
               ((symbolp form) (list form))
               ((atom form) form)
               (t (if (find-$ form env)
                      (values `(setq $ ,form) t)
                      (values `(setq $ ,(list* (car form) '$ (cdr form))) t))))))
    `(let (($ ,form))
       ,@(mapcar (lambda (f) (sb-walker:walk-form f nil #'walker)) body))))

The only thing that’s changed here is the definition of find-$, and in fact it’s a little simpler: the task is now to walk the entire form and find uses of $ in an evaluated position, no matter how deep in the evaluation. Because this is a code-walker, this will correctly handle macros, backquotes, quoted symbols, and so on, and this allows code of the form

(macroexpand-1 '(-> "THREE" string-downcase (char 0) char-code (complex (1+ $) (1- $))))
; => (LET (($ "THREE"))
;      (SETQ $ (CHAR-CODE $))
;      (SETQ $ (COMPLEX (1+ $) (1- $)))),
;    T

which, as far as I can tell, is not supported in magrittr: doing 3 %>% complex(.+1,.-1) is met with the error that “object '.' not found”. Supporting this might, of course, not be a good idea, but at least the code walker shows that it’s possible.

What if we wanted to augment -> to handle binding forms, or special forms in general? This is probably beyond the call of duty, but let’s just briefly imagine that we wanted to be able to support binding special variables around the individual calls in the chain; for example, we want

(-> 3 (let ((*random-state* (make-random-state))) rnorm) mean)

to expand to

(let (($ 3))
  (setq $ (let ((*random-state* (make-random-state))) (rnorm $)))
  (setq $ (mean $)))

and let us also say, to make it interesting, that uses of $ in the bindings clauses of the let should not count against inhibiting the insertion of $ in the first argument position of the first form in the body of the let, so

(-> 3 (let ((y (1+ $))) (atan y)))

should expand to

(let (($ 3)) (setq $ (let ((y (1+ $))) (atan $ y))))

So our code walker needs to walk the bindings of the let, merely collecting information into the walker’s lexical environment, then walk the body performing the same rewrite as before. CHALLENGE ACCEPTED:

(defmacro -> (&body forms)
  (let ((rewrite t))
    (declare (special rewrite))
    (labels ((find-$ (form env)
               (sb-walker:walk-form form env
                (lambda (f c e)
                    ((and (eql f '$) (eql c :eval))
                     (return-from find-$ t))
                    (t f))))
             (walker (form context env)
               (declare (ignore context))
               (typecase form
                 (symbol (if rewrite (list form) form))
                 (atom form)
                 ((cons (member with-rewriting without-rewriting))
                  (let ((rewrite (eql (car form) 'with-rewriting)))
                    (declare (special rewrite))
                    (values (sb-walker:walk-form (cadr form) env #'walker) t)))
                 ((cons (member let let*))
                  (unless rewrite
                    (return-from walker form))
                  (let* ((body (member 'declare (cddr form)
                                       :key (lambda (x) (when (consp x) (car x))) :test-not #'eql))
                         (declares (ldiff (cddr form) body))
                         (rewritten (sb-walker:walk-form
                                          (,(car form) ,(cadr form)
                                     env #'walker)))
                    (values rewritten t)))
                  (unless rewrite
                    (return-from walker form))
                  (if (find-$ form env)
                      (values `(setq $ ,form) t)
                      (values `(setq $ ,(list* (car form) '$ (cdr form))) t))))))
      `(let (($ ,(car forms)))
         ,@(mapcar (lambda (f) (sb-walker:walk-form f nil #'walker)) (cdr forms))))))

Here, find-$ is unchanged from the previous version; all the new functionality is in walker. How does it work? The default branch of the walker function is also unchanged; what has changed is handling of let and let* forms. The main trick is to communicate information between successive calls to the walker function, and turn the rewriting on and off appropriately: we wrap parts of the form in new pseudo-special operators with-rewriting and without-rewriting, which is basically a tacky and restricted implementation of compiler-let – if we needed to, we could do a proper one with macrolet. Within the scope of a without-rewriting, walker doesn’t do anything special, but merely return the form it was given, except if the form it’s given is a with-rewriting form. This is a nice illustration, incidentally, of the idea that lexical scope in the code translates nicely to dynamic scope in the compiler; I can’t remember where I read that first (but it’s certainly not a new idea).

And now

(macroexpand '(-> 3 (let ((*random-state* (make-random-state))) rnorm) mean))
; => (LET (($ 3))
;        (SETQ $ (RNORM $)))
;      (SETQ $ (MEAN $))),
;    T
(macroexpand '(-> 3 (let ((y (1+ $))) (atan y))))
; => (LET (($ 3))
;      (LET ((Y (1+ $)))
;        (SETQ $ (ATAN $ Y)))),
;    T

Just to be clear: this post isn’t advocating a smarter pipe operator; I don’t have a clear enough view, but I doubt that the benefits of the smartness outweigh the complexity. It is demonstrating what can be done, in a reasonably controlled way, using a code-walker: ascribing semantics to fragments of Common Lisp code, and combining those fragments in a particular way, and of course it’s another example of sb-walker:walk in use.

Finally, if something like this does in fact get used, people sometimes get tripped up by the package system: the special bits of syntax are symbols, and importing or package-qualifying -> without doing the corresponding thing to $ would lead to cryptic errors, wrong results and/or confusion. One possibility to handle that is to invent a bit more reader syntax:

(set-macro-character #\¦
 (defun pipe-reader (stream char)
   (let ((*readtable* (copy-readtable)))
     (set-macro-character #\·
      (lambda (stream char)
        (declare (ignore stream char))
        '$) t)
   (cons '-> (read-delimited-list char stream t)))) nil)
¦"THREE" string-downcase (find-if #'alpha-char-p ·) char-code¦

If this is the exported syntax, it has the advantage that the interface can only be misused intentionally: the actual macro and its anaphoric symbol are both hidden from the programmer; and the syntax is reasonably easy to type – on my keyboard ¦ is AltGr+| and · is AltGr+. – and moderately mnemonic from shell pipes and function notation respectively. It also has all the usual disadvantages of reader-based interfaces, such as composability, somewhat mitigated if pipe-reader is part of the macro’s exported interface.

inquisitor 1351

Another De Profundis weekend, another pair of train journeys, another crossword – and, surprisingly, another success. Unlike last week, no external motivation was necessary, and it all fell into place quite nicely.

How to Spell CAT

Six words, occupying three rows of the grid in total, are clued without their common definition. Using that common definition, with particular reference to one of the six and an answer elsewhere in the grid, each of the three rows can be converted into a letter, spelling the word to be entered below the puzzle. Failed attempts at spelling will need to be removed from clues before solving, adjusting punctuation and spacing where necessary.

  • *3ac: CONFOUND: Study institute (8): CON (study) + FOUND (institute)
  • 11ac: ONUS: Australian crook shirking college responsibility (4):
  • 12ac: DEER: Put off dismissing fellow in moot case, say (4): DEFER (put off) - F(ellow)
  • 13ac: CREMASTERS: Male muscles concerning male among film scouts? (10): RE (concerning) + M(ale) in CASTERS (film scouts)
  • 14ac: AUSTER: American author? American outcast, nothing less, about to backtrack (6): A(merican) + OUST - O (nothing) + reversed RE (about); ref. Paul Auster again!
  • 15ac: OORIAL: See one creature from Oz coming back – a sheep (6): LA (see) + I (one) + ROO (creature from Oz) all reversed
  • 16ac: NEIF: Direct action for Newcastle’s provided by elderly duke (4): NE (Newcastle’s region is ‘the North East’) + IF (provided)
  • 17ac: MENU: Choctaw soldiers heading for Utah (4): MEN (soldiers) + U(tah)
  • 19ac: ESTH: Religious book, one of religious origin (4): Double Definition
  • 20ac: OUTDO: Perform better than party that’s not elected? (5): OUT (not elected) DO (party)
  • 22ac: SPIKY: Spot punctured by one tack with sharp point (5): SPY (spot) around I (one) K
  • *24ac: BLOW: Bass, definitely not soprano... (4): B(ass) + LOW (not soprano)
  • *25ac: RUSH: ... consequence of time off hard for tenor... (4): RUST (consequence of time off) - T(enor) + H(ard)
  • *26ac: LONG: ... soprano abandoning hard work to secure note (4): SLOG (hard work) - S(oprano) around N(ote)
  • 27ac: ESTEEM: Regard half of bees in front of swarm (6): (be)ES + TEEM (swarm)
  • 28ac: I SAY: Look at this approach to saving yen (4, two words): ISA (approach to saving; ref. ISA) + Y(en)
  • 32ac: FARE: Funk’s heading live do (4): F(unk) + ARE (live)
  • 34ac: TEEN: No participant in society recalls this grief of old (4): NEET (no participant in society; ref. NEET) reversed
  • 36ac: COOS: Old relative ignoring fashionable signs of affection (4): COOSIN (obs. “cousin”) - IN (fashionable)
  • 37ac: YUAN: Chinese plant not accepting pounds – but accepting these? (4): YULAN (Chinese plant) - L (pounds) &lit.
  • 39ac: SCOUTHER: Spy that woman’s toast in Edinburgh (8): SCOUT (spy) + HER (that woman)
  • 40ac: SLOB: Awkward, lacking dash, recalled as a boor (4):
  • 41ac: DELI: Composer avoiding us in foodstore (6): DELIUS (ref. Frederick Delius) - US
  • 42ac: FEES: Payment fact judge rejected (4): F + SEE (judge) reversed
  • *43ac: HYPHEN: Offend old woman disrepectfully (6): HYP (obs. “offend”) + HEN (slang “woman”)
  • *44ac: SPRINT: Journey time includes end of tour (6): SPIN (journey) + T(ime) around (tou)R
  • 1dn: ROMANO-BRITISH: Harm bistro with action arranged as description of historic style? (13): *(HARM BISTRO ION)
  • 2dn: ON CUE: Lactate around third quarter of hour, when required (5, two words): ONCE (late) around (ho)U(r)
  • 3dn: CURSITOR: Ancient Court official ruined our tactics before King (8): *(OUR TICS) + R (King)
  • 4dn: NAME: A million resident in Nebraska state (4): A M(illion) in NE (Nebraska)
  • 5dn: FLARE-OUT: Area not about to be placed in mock component of airstrip design: AREA - A(bout) in FLOUT (mock)
  • 6dn: OPSONISE: Improve edibility of pies, soon, with cooking (8): *(PIES SOON)
  • 7dn: UNTO: To point, as before, shows no hint of politeness (5): PUNTO (obs. “point”) - P(oliteness)
  • 8dn: DERISION: Tacit gibes rumoured to circle around King and I (8): NOISED (rumoured) reversed around R (King) + I
  • 9dn: RESAT: Stun gun, kicking upward, was tested again (5): TASER (stun gun) reversed
  • 10dn: ORAL HYGIENIST: This legionary in a flat cap could do for health worker (13): *(THIS LEGIONARY)
  • 17dn: MORSE CODE: Symbolic language, increasingly imitating English, used around Sweden (9, two words): MORE (increasingly) + COD (imitating) + E(nglish) around S(weden)
  • 18dn: USHERSHIP: Escort’s function, taking our group plundering up north (9): US (our group) + HERSHIP (Scot. “plundering”, again)
  • 21dn: ULES: Top’s worn off red rubbers (4): (g)ULES (red)
  • 23dn: KNAR: Tack North on a river, making one knot (4): K + N(orth) + A + R(iver)
  • 29dn: AESOP: Story-teller’s promoted main work (5): SEA (main) reversed + OP (work)
  • 30dn: ROUEN: Libertine taking on new hectare in France? (5): ROUE (libertine) + N(ew)
  • 31dn: ROTLS: Talks nonsense, including pound in variable quantities (5): ROTS (talks nonsense) around L (pound)
  • 33dn: AUREI: Former operation’s brought in excellent antique gold coins (5): URE (obs. “operation”) in AI (excellent)
  • 35dn: EELY: No longer happy to relinquish first of fish (4): (s)EELY (obs. “happy”)
  • 38dn: AMEN: Agree broth’s recipe’s forgotten (4): RAMEN (broth) - R(ecipe)

Anagrams of CAT (not quite all of them; they’re left alone in 37ac) have to be removed from some of the clues before solving; I particularly liked “in a flat cap” → “in a flap”. The unclued entries’ common definition is dash; writing them out gives —/— — —/— —, or with particular reference to MORSE CODE, ‘TOM’. Which is how to spell CAT.

earl conference

This week, I went to the Effective Applications of the R Language conference. I’d been alerted to its existence from my visit to a londonR meeting in June. Again, I went for at least two reasons: one as an R enthusiast, though admittedly one (as usual) more interested in tooling than in applications; and one as postgraduate coordinator in Goldsmiths Computing, where for one of our modules in the new Data Science programme (starting todayyesterday! Hooray for “Welcome Week”!) involves exposing students to live data science briefs from academia and industry, with the aim of fostering a relevant and interesting final project.

A third reason? Ben Goldacre as invited speaker. A fantastic choice of keynote, even if he did call us ‘R dorks’ a lot and confess that he was a Stata user. The material, as one might expect, was derived from his books and related experience, but the delivery was excellent, and the drive clear to see. There were some lovely quotes that it’s tempting to include out of context; in the light of my as-yet unposed ‘research question’ for the final module of the PG Certificate in Higher Education – that I am still engaged on – it is tempting to bait the “infestation of qualitative researchers in the educational research establishment”, and attempt a Randomised Controlled Trial, or failing that a statistical analysis of assessments to try to uncover suitable hypotheses for future testing.

Ben’s unintended warm-up act – they were the other way around in the programme, but clearly travelling across London is complicated – was Hadley Wickham, of ggplot2 fame. His talk, about RStudio, his current view on the data analysis workflow, and new packages to support it, was a nice counterpoint: mostly tools, not applications, but clearly focussed to help make sense of complicated (and initially untidy) datasets. I liked the shiny-based in-browser living documents in R-markdown, which is not a technology that I’ve investigated yet; at this rate I will have more options for reproducible research than reports written. He, and others at the conference, were advocating a pipe-based code sequencing structure – the R implementation of this is called magrittr (ha, ha) and has properties that are made possible to user code through R’s nature of a Lisp-1 with crazy evaluation semantics, on which more in another post.

The rest of the event was made up of shorter, usually more domain-specific talks: around 20 minutes for each speaker. I think it suffered a little bit from many of the participants not being able to speak freely – a natural consequence of a mostly-industrial event, but frustrating. I think it was also probably a mistake to schedule in the first one of the regular (parallel) sessions of the event a reflective slot for three presentations about comparing R with other languages (Python, Julia, and a more positive one about R’s niche): there hadn’t really been time for a positive tone to be established, and it just felt like a bit of a downer. (Judging by the room, most of the delegates – perhaps wisely – had opted for the other track, on “Business Applications of R”).

Highlights of the shorter talks, for me:

  • YPlan’s John Sandall talking about “agile” data analytics, leading to agile business practices relentlessly focussed on one KPI. At the time, I wondered whether the focus on the first time a user gives them money would act against building something of lasting value – analytics give the power to make decisions, but the underlying strategy still has to be thought about. On the other hand, I’m oh-too familiar with the notion that startups must survive first and building something “awesome” is a side-effect, and focussing on money in is pretty sensible.
  • Richard Pugh’s (from Mango Solutions) talk about modelling and simulating the behaviour of a sales team did suffer from the confidentiality problem (“I can’t talk about the project this comes from, or the data”) but was at least entertaining: the behaviours he talked about (optimistic opportunity value, interactions of CRM closing dates with quarter boundaries) were highly plausible, and the question of whether he was applying the method to his own sales team quite pointed. (no)
  • the team from Simpson Carpenter Ltd, as well as saying that “London has almost as many market research agencies as pubs” (which rings true) had what I think is a fair insight: R is perhaps less of a black-box than certain commercial tools; there’s a certain retrocomputing feel to starting R, being at the prompt, and thinking “now what?” That implies that to actually do something with R, you need to know a bit more about what you’re doing. (That didn’t stop a few egregiously bad graphs being used in other presentations, including my personal favourite of a graph of workflow expressed as business value against time, with the inevitable backwards-arrows).
  • some other R-related tools to look into:

And then there was of course the hallwaybreak room track; Tower Hotel catered admirably for us, with free-flowing coffee, nibbles and lunch. I had some good conversations with a number of people, and am optimistic that students with the right attitude could both benefit and gain hugely from data science internships. I’m sure I was among the most tool-oriented of the attendees (most of the delegates were actually using R), but I did get to have a conversation with Hadley about “Advanced R”, and we discussed object systems, and conditions and restarts. More free-form notes about the event on my wiki.

Meanwhile, in related news, parts of the swank backend implementation of SLIME changed, mostly moving symbols to new packages. I've updated swankr to take account of the changes, and (I believe, untested) preserved compatibility with older (pre 2014-09-13) SLIMEs.

inquisitor 1350

I’ve been on holiday abroad, so away from crosswords. What little reflection I did on holiday (see also) gave rise to an ambition to graduate from Azed to something one step harder – if only so that I leave time and space on this blog for non-crossword entries. So, coming home to Inquisitor 1349, I obviously proceeded to solve a total of zero clues.

However! The following weekend was a De Profundis weekend (too late to blog encouraging you all to come! Lovely Josquin, Brumel, and other Franco-Flemish superstars; next gig 25th April), and De Profundis weekends offer a certain amount of train travel, which itself affords almost-uninterrupted crossword contemplation. So, I took along Inquisitor 1350, and made some decent progress on the way up to Cambridge, and a bit more in my rehearsal breaks. And then I met up with Uri, my old friend and partner in crossword solving, who quite unfairly had most of Sunday free while I had more intense rehearsal and travel away from reference books. Cheating! So when he e-mailed to say that he had identified the theme and solved most of the grid, that was enough motivation to stay up past bedtime (mine as well as the rest of the household’s) banging my head against the crossword, then the wall as the pieces finally started fitting together.

3 Down

The unclued grey grid entries include three thematic film titles (chronologically, 3dn/10ac/40ac, 9dn and 35ac. The name of the protagonist of the first film (32dn) is also associated with 35ac. 5dn’s 9dn to become 3dn/10ac/40ac was a success: contents of pink squares can be arranged to form a word that hints at a failure who must be highlighted in the completed grid (6 cells)

  • 1ac: KAT: Stimulant to swallow doing the rounds when there’s no ecstasy (3): TAKE (swallow) reversed - E(cstasy)
  • 4ac: RIFLE: Firearm sack (5): Double Definition
  • 7ac: OWS: Pained expressions and arguments start off (3): (r)OWS (arguments)
  • 11ac: REEST: Birds with little time go off for bacon (5): REES (birds) + T(ime)
  • 12ac: LOUDLY: Look duly chastened – how people shout! (6): LO (look) + *(DULY)
  • 13ac: DINE OFF: Make a meal of cooking fine food half-heartedly (7, two words): *(FINE FO(o)D)
  • 15ac: MOATS: Water surrounds a thousand cereal crops (5): M (thousand) + OATS (cereal crops)
  • 16ac: ANURA: Two adults hurry back inside for amphibians (5): RUN (hurry) reversed inside A A (two adults)
  • 17ac: ABETTAL: Suspect help’s coming from retro coffee bar shortly (7): LATTE (coffee) + BA(r) reversed
  • 19ac: ZEREBA: Authorize rebate that includes animal protection measure (6): authoriZE REBAte
  • 20ac: OOSE: Tartan nap more likely to fall off without covers (4): (l)OOSE(r) (more likely to fall off)
  • 22ac: GUEREZA: For colobus Montezuma’s revenge struck after faltering movements noth taken (7): *(MONTEZUMAS REVENGE - MOVEMENTS N(orth))
  • 24ac: VACUUMS: Cleaners regularly dug up mess after holiday (7): VAC (holiday) + dUg Up MeSs
  • 25ac: STAR: Celebrity’s rush for Scotch (4): Double Definition(?)
  • 29ac: GAINST: A poet’s not in favour of financial profit – good man! (6): GAIN (financial profit) + ST (saint)
  • 31ac: TATOUAY: Armadillo always following French film star, not once united (7): TATOU (ref. Audrey Tatou) - U(nited) + AY (always)
  • 33ac: APING: Taking off recording that’s not starting (5): (t)APING (recording)
  • 34ac: CTENE: Cardinal gripped by church organ that’s moving (5): TEN (cardinal number) in CE (Church of England)
  • 36ac: EILEEN: In hearing I have a preference for Irish girl (6): sounds like “I lean” (I have a preference for)
  • 37ac: JETES: Leaps narrow streams consuming energy (5): JETS (narrow streams) around E(nergy)
  • 38ac: CONVERTOR: Cover torn, with end torn off and put inside – it makes a change currently? (9): COVER TORN - N around N
  • 39ac: EEK: Decapitated nerd? Gulp! (3): (g)EEK
  • 41ac: NEY: French marshal longing to make a comeback (3): YEN (longing); ref. Michel Ney again
  • 1dn: KARAMAZOV: Bangle of 50% vanadium worn by mum and unknown literary brothers (9): KARA (bangle) + O(f) + V (50% of VA(nadium)) around MA (mum) + Z (unknown); ref. The Brothers Karamazov
  • 2dn: AWESOME: Dreadful pitiful scotch after knocking first two back (7): WAESOME (Scot. “pitiful”) with first two letters reversed
  • 4dn: ROSE TREE: Reassembling stereo in middle of area is a standard bloomer (8, two words): *(STEREO) in (a)RE(a)
  • 6dn: ELUENTS: Washing liquids free-flowing when top is off going in different directions (7): (f)LUENT (free-flowing) in E S (different directions)
  • 8dn: WOLFRAM: Elemental example of US beasts (7): WOLF + RAM
  • 14dn: OUTLET: Retail market of stateside egg dish switching from Maine to Utah (6): OMELET (Am. egg dish) - ME (Maine) + UT(ah)
  • 18dn: EWES: Sheep’s milk, say (4): sounds like “use” (milk)
  • 21dn: SCOTIA: Company invested in islands up in Burns’ home? (6): CO(mpany) in AITS (islands) reversed
  • 22dn: GURU: Returned wig to university teacher (4): RUG (wig) reversed + U(niversity)
  • 23dn: ARTLESSLY: He goes off in a callous way like some innocent (9): (he)ARTLESSLY (in a callous way)
  • 25dn: SIXTIETH: An unknown knot seen in this tangled diamond? (8): X (unknown) TIE (knot) in *(THIS)
  • 26dn: ETAGERE: A stand made by shelving retired menial film actor (7): ETA (menial) + GERE (film actor; ref. Richard Gere)
  • 27dn: CAPRICE: Fancy hat with woody bits for locals? (7): CAP (hat) + RICE (dial. woody bits)
  • 28dn: CONVENE: Very little space within 3-D structure for rally (7): V(ery) EN (little space) in CONE (3-D structure)
  • 30dn: SINCERE: Genuine increase without a high (7): *(INCREASE - A)

I think key for me in this puzzle was getting 9dn (SKYFALL), realising that that was currently the latest of the James Bond films (hence not followed by anything chronologically), casting around for other things to be the theme, and then having a night-time epiphany of THE MAN WHO FELL TO EARTH (3dn/10ac/40ac). After that it was plain sailing to include FELIX BAUMGARTNER (5dn), GRAVITY (35ac), NEWTON (32dn); with a couple of Ws and an X in the pink squares, it was fairly easy to guess that ICARUS (6 cells, falling head-first down and to the right) who has a WAX WING was the failure.

pgcert essays

One of the (perhaps) surprising things about academia, at least in the UK, is that people with job titles such as “Lecturer” or “Professor” need not have any qualification, or indeed training, in teaching.

In fact, the academic career structure positively seems to incentivise against teaching: the typical route to a permanent academic job (there is no such thing as tenure in the UK any more – see e.g. this article – but we do have employment rights) is through one or more postdoctoral research assistantships, where the job consists of 90% research, guided by a “Principal Investigator”, and 90% hustling for the next postdoctoral research assistanship. In some cases, typically RCUK-funded positions, these research positions are allowed to incorporate a small amount of teaching for career development (six hours a week or so); EU-funded positions explicitly exclude anything that’s not in the work programme, and enforce that through rigid project management.

It’s not just the route that leads to a permanent academic position that incentivises against teaching, though: I think it’s fair to say that while lip-service might be paid to putting research quality and teaching quality on an equal footing in universities in general, the overwhelming perception is that research income, through grants or consultancy, trumps most other things: perhaps because research income is an immediate, unarguable number that appears to sustain the university’s activities (even if in principle each RCUK-funded “full Economic Costing” grant in fact is a loss to the institution – in practice the utterly untransparent “overheads” make that point moot). To be fair to academic managers, they too are victims in the academic career structure, just a couple of steps ahead: it’s not like the academic career structure incentivises learning to manage large, bureaucratic institutions either.

A new academic year is just starting, again, and the feeling of impending doom before the insanity that is the Autumn Term hits does prompt a certain amount of reflection. This year, though, I got my reflection in early: having enrolled in a postgraduate certificate in teaching and learning last year, and just about scraped the attendance requirements, I found myself on my holidays with a deadline for three essays. Yes of course I had known about the deadline all year, and of course I had had the opportunity in principle to have a tutorial on my topics – the irony was palpable as I e-mailed to check, wistfully, that the deadline was immovable, immutable and non-negotiable. (Of course it was.)

In the end, though, I did manage to write the essays: one on plagiarism, one on blogging, and a longer one (though still only about 2000 words) on structured discussion. While I’m sufficiently self-confident to feel that they’re potentially of interest, if HE pedagogy is your thing, I’m a lot less confident that they deserve a passing grade: if I were playing this game with an eye to winning I would have cross-checked the documented learning outcomes for each assessment with identifiable, obvious points to make the marking easy, and instead I wrote them in somewhat of a hurry (not quite on the day of the deadline, but...).

A couple of minor technical points might also be of interest to posterity. I wrote the essays in markdown, rather than my normal flow of starting in org-mode – in part because they started off as notes in my wiki anyway, and it was natural to expand them into essays in situ. In this case, they ended up as pandoc-flavoured markdown (since I had to generate PDFs for submission), rather than ikiwiki-flavoured, so the wiki versions will probably look a little bit odd (there is pandoc support for ikiwiki, but that’s a bit beyond the call of duty for now. In any case, it was not too much effort to convert the wiki pages into content suitable for submission, so that was nice; it’s probably just about topical to mention the new standard common markdown (obligatory obligatory), which would not have helped at all, lacking as it does several of the features I actually used (footnotes, title metadata), though I suppose effectively the rest of the text was effectively written in CommonMark.

Having written an essay where I conceded that the use of Moodle, the particular Virtual Learning Environment used at Goldsmiths, would have been beneficial and appropriate, I then had to use Moodle for the submission, and was instantly reminded why I avoid using it in the first place: it may have certain affordances for group cohesion and discussion, but that doesn’t stop it from being painful to use. Oh well.

I get feedback and module results in November. I wait with bated breath...

naive vs proper code-walking

I said in my discussion about backquote representations that some utilities had defects made manifest by SBCL 1.2.2’s new internal representation for backquote and related operators, and that those defects could have been avoided by using a code-walker. I’m going to look at let-over-lambda code here, to try to demonstrate what I meant by that, and show how a proper code-walker can quite straightforwardly be used for the code transformations that have been implemented using a naïve walker (typically walking over a tree of conses), removing whole classes of defects in the process.

The let-over-lambda code I’m discussing is from, specifically this version. This isn’t intended to be a hatchet job on the utility – clearly, it is of use to its users – but to show up potential problems and offer solutions for how to fix them. I should also state up front that I haven’t read the Let over Lambda book, but it’s entirely possible that discussing and using a full code-walker would have been out of scope (as it explicitly was for On Lisp).

Firstly, let’s deal with how the maintainer of the let-over-lambda code is dealing with the change in backquote representations, since it’s still topical:

;; package definition here just in case someone decides to paste
;; things into a Lisp session, and for private namespacing
(defpackage "LOL" (:use "CL"))
(in-package "LOL")
;; actual excerpts from let-over-lambda code from
;; <>
;; begins here:
(if (string-lessp (lisp-implementation-version) "1.2.2")
    (pushnew :safe-sbcl *features*)
    (setq *features* (remove :safe-sbcl *features*)))
(defun flatten (x)
  (labels ((rec (x acc)
             (cond ((null x) acc)
                   #+(and sbcl (not safe-sbcl))
                   ((typep x 'sb-impl::comma) (rec (sb-impl::comma-expr x) acc))
                   ((atom x) (cons x acc))
                   (t (rec (car x) (rec (cdr x) acc))))))
    (rec x nil)))

The issues around the (*features*) handling here have been reported at github; for the purpose of this blog entry, I will just say that I wrote about them in Maintaining Portable Lisp Programs, a long time ago, and that a better version might look a bit like this:

(eval-when (:compile-toplevel :execute)
  (defun comma-implementation ()
    (typecase '`,x
      (symbol 'old)
      ((cons symbol (cons structure-object)) 'new)))
  (if (eql (comma-implementation) 'old)
      (pushnew 'cons-walkable-backquote *features*)
      (setq *features* (remove 'cons-walkable-backquote *features*))))
(defun flatten (x)
  (labels ((rec (x acc)
             (cond ((null x) acc)
                   ((typep x 'sb-impl::comma) (rec (sb-impl::comma-expr x) acc))
                   ((atom x) (cons x acc))
                   (t (rec (car x) (rec (cdr x) acc))))))
    (rec x nil)))

With these changes, the code is (relatively) robustly testing for the particular feature it needs to know about at the time that it needs to know, and recording it in a way that doesn’t risk confusion or contention with any other body of code. What is the let-over-lambda library using flatten for?

(defun g!-symbol-p (thing)
  (and (symbolp thing)
       (eql (mismatch (symbol-name thing) "G!") 2)))
(defmacro defmacro/g! (name args &rest body)
  (let ((syms (remove-duplicates
               (remove-if-not #'g!-symbol-p (flatten body)))))
    `(defmacro ,name ,args
       (let ,(mapcar
              (lambda (s)
                `(,s (gensym ,(subseq (symbol-name s) 2))))

The intent behind this macro-defining macro, defmacro/g!, appears to be automatic gensym generation: being able to write

(defmacro/g! with-foo ((foo) &body body)
  `(let ((,g!foo (activate-foo ,foo)))
         (progn ,@body)
       (deactivate-foo ,g!foo))))

without any explicit calls to gensym but retaining the protection that gensyms give against name capture:

(macroexpand-1 '(with-foo (3) 4))
; => (let ((#1=#:FOO1 (activate-foo 3)))
;      (unwind-protect
;          (progn 4)
;        (deactivate-foo #1#)))

That's fine; it’s reasonable to want something like this. Are there any issues with this, apart from the one exposed by SBCL’s new backquote implementation? In its conventional use, probably not – essentially, all uses of g! symbols are unquoted (i.e. behind commas) – but there are a couple of more theoretical points. One issue is that flatten as it currently stands will look for all symbols beginning with g! in the macroexpander function source, whether or not they are actually variable evaluations:

(defmacro/g! with-bar ((bar) &body body)
  `(block g!block
     (let ((,g!bar ,bar)) ,@body)))
; unused variable G!BLOCK
(macroexpand-1 '(with-bar (3) 4))
; => (block g!block (let ((#:BAR1 3)) 4))

In this example, that’s fair enough: it’s probably user error to have those g! symbols not be unquoted; this probably only becomes a real problem if there are macro-defining macros, with both the definer and the definition using g! symbols. It's not totally straightforward to demonstrate other problems with this simple approach to Lisp code transformation using just this macro; the transformation is sufficiently minimal, and the symptoms of problems relatively innocuous, that existing programming conventions are strong enough to prevent anything seriously untoward going wrong.

Before getting on to another example where the problems with this approach become more apparent, how could this transformation be done properly? By “properly” here I mean that the defmacro/g! should arrange to bind gensyms only for those g! symbols which are to be evaluated by the macroexpander, and not for those which are used for any other purpose. This is a task for a code-walker: a piece of code which exploits the fact that Lisp code is made up of Lisp data structures, all of which are introspectable, and the semantics of which in terms of effect on environment and execution are known. It is tedious, though possible, to write a mostly-portable code-walker (there needs to be some hook into the implementation’s representation of environments); I’m not going to do that here, but instead will use SBCL’s built-in code-walker.

The sb-walker:walk-form function takes three arguments: a form to walk, an initial environment to walk it in, and a walker function to perform whatever action is necessary on the walk. That walker function itself takes three arguments, a form, context and environment, and the walker arranges for it to be called on every macroexpanded or evaluated subform in the original form. The walker function should return a replacement form for the subform it is given (or the subform itself if it doesn’t want to take any action), and a secondary value of t if no further walking of that form should take place.

To do g! symbol detection and binding is fairly straightforward. If a symbol is in a context for evaluation, we collect it, and here we can take the first benefit from a proper code walk: we only collect g! symbols if the code-walker deems that they will be evaluated and there isn't an already-existing lexical binding for it:

(defmacro defmacro/g!-walked (name args &body body)
  (let* (g!symbols)
    (flet ((g!-walker (subform context env)
             (declare (ignore context))
             (typecase subform
                (when (and (g!-symbol-p subform)
                           (not (sb-walker:var-lexical-p subform env)))
                  (pushnew subform g!symbols))
               (t subform))))
      (sb-walker:walk-form `(progn ,@body) nil #'g!-walker)
      `(defmacro ,name ,args
         (let ,(mapcar (lambda (s) (list s `(gensym ,(subseq (symbol-name s) 2))))

The fact that we only collect symbols which will be evaluated deals with the problem exhibited by with-bar, above:

(defmacro/g!-walked with-bar/walked ((bar) &body body)
  `(block g!block
     (let ((,g!bar ,bar)) ,@body)))
(macroexpand-1 '(with-bar/walked (3) 4))
; => (block g!block (let ((#:BAR1 3)) 4))

Only gathering symbols which don’t have lexical bindings (testing sb-walker:var-lexical-p) deals with another minor problem:

(defmacro/g!-walked with-baz ((baz) &body body)
  (let ((g!sym 'sym))
    `(let ((,g!sym ,baz)) ,@body)))
(macroexpand-1 '(with-baz (3) 4))
; => (let ((sym 3)) 4)

(the cons-walker – flatten – would not be able to detect that there is already a binding for g!sym, and would introduce another one, again leading to an unused variable warning.)

OK, time to recap. So far, we’ve corrected the code that tests for particular backquote implementations, which was used in flatten, which itself was used to perform a code-walk; we’ve also seen some low-impact or theoretical problems with that simple code-walking technique, and have used a proper code-walker instead of flatten to deal with those problems. If the odd extra unused variable binding were the worst thing that could happen, there wouldn’t be much benefit from using a code-walker (other than the assurance that the walker is dealing with forms for execution); however, let us now turn our attention to the other macro in let-over-lambda’s code which does significant codewalking:

(defun dollar-symbol-p (thing)
  (and (symbolp thing)
       (char= (char (symbol-name thing) 0) #\$)
       (ignore-errors (parse-integer (subseq (symbol-name thing) 1)))))
(defun prune-if-match-bodies-from-sub-lexical-scope (tree)
  (if (consp tree)
      (if (or (eq (car tree) 'if-match)
              (eq (car tree) 'when-match))
          (cddr tree)
          (cons (prune-if-match-bodies-from-sub-lexical-scope (car tree))
                (prune-if-match-bodies-from-sub-lexical-scope (cdr tree))))
;; WARNING: Not %100 correct. Removes forms like (... if-match ...) from the
;; sub-lexical scope even though this isn't an invocation of the macro.
(defmacro! if-match ((test str) conseq &optional altern)
  (let ((dollars (remove-duplicates
                  (remove-if-not #'dollar-symbol-p
                                 (flatten (prune-if-match-bodies-from-sub-lexical-scope conseq))))))
    (let ((top (or (car (sort (mapcar #'dollar-symbol-p dollars) #'>)) 0)))
      `(let ((,g!str ,str))
         (multiple-value-bind (,g!s ,g!e ,g!ms ,g!me) (,test ,g!str)
           (declare (ignorable ,g!e ,g!me))
           (if ,g!s
               (if (< (length ,g!ms) ,top)
                   (error "ifmatch: too few matches")
                   ;; lightly edited here to remove irrelevant use of #`
                   (let ,(mapcar (lambda (a1) `(,(symb "$" a1)
                                                (subseq ,g!str (aref ,g!ms ,(1- a1))
                                                               (aref ,g!me ,(1- a1)))))
                                 (loop for i from 1 to top collect i))
(defmacro when-match ((test str) conseq &rest more-conseq)
  `(if-match (,test ,str)
     (progn ,conseq ,@more-conseq)))

What’s going on here? We have a prune-if-match-bodies-from-sub-lexical-scope function which, again, performs some kind of cons-based tree walk, removing some conses whose car is if-match or when-match. We have a trivial macro when-match which transforms into an if-match; the if-match macro is more involved. Any symbols named as a $ sign followed by an integer (in base 10) are treated specially; the intent is that they will be bound to capture groups of the cl-ppcre match. So it would be used in something like something like

(defun key-value (line)
  (if-match ((lambda (s) (scan "^\\(.*\\): \\(.*\\)$" s)) line)
      (list $1 $2)
      (error "not actually a key-value line: ~S" line)))

and that would macroexpand to, roughly,

(defun key-value (line)
  (multiple-value-bind (s e ms me)
      ((lambda (s) (scan "^\\(.*\\): \\(.*\\)$" s)) line)
    (if s
        (if (< (length ms) 2)
            (error "if-match: not enough matches)
            (let (($1 (subseq line (aref ms 0) (aref me 0)))
                  ($2 (subseq line (aref ms 1) (aref me 1))))
              (list $1 $2)))
        (error "not actually a key-value line: ~S" line))))

(there's additional reader macrology in let-over-lambda to make that lambda form unnecessary, but we can ignore that for our purposes).

Now, if-match has a similar problem that defmacro/g! had: since the tree walker doesn’t make a distinction between symbols present for evaluation and symbols for any other purpose, it is possible to confuse the walker. For example:

(if-match (scanner string)
    (if (> (length $1) 6)

This form, if macroexpanded, will attempt to bind one million variables to matched groups; even if the compiler doesn’t choke on that, evaluation will go wrong, as the matcher is unlikely to match one million groups (so the “not enough matches” error branch will be taken) – whereas of course the quoted one million dollar symbol is not intended for evaluation.

But the nesting problems are more obvious in this case than for defmacro/g!. Firstly, take the simple case:

(if-match (scanner string)
    (list $1
          (if-match (scanner2 string)

Here, the $2 is in the scope of the inner if-match, and so mustn’t be included for the macroexpansion of the outer if-match. This case is handled in let-over-lambda’s implementation by the prune-if-match-bodies-from-sub-lexical-scope: the consequent of the inner if-match is pruned from the dollar-symbol accumulator. However, there are several issues with this; the first is that the test is pruned:

(if-match (scanner string)
    (if-match (scanner2 $2)

In this example, the $2 is ‘invisible’ to the outer if-match, and so won’t get a binding. That’s straightforwardly fixable, along with the mishandling of when-let’s syntax (the entire body of when-let should be pruned, not just the first form), and what I think is an error in the pruning of if-match (it should recurse on the cdddr, not the cddr; github issue).

Not fixable at all while still using naïve code-walking are two other problems, one of which is noted in the comment present in the let-over-lambda code: the pruner doesn’t distinguish between if-match forms for evaluation and other conses whose car is if-match. Triggering this problem does involve some contortions – in order for it to matter, we need an if-match not for evaluation followed by a dollar symbol which is to be evaluated; but, for example:

(defmacro list$/q (&rest args)
  `(list ,@(mapcar (lambda (x) (if (dollar-symbol-p x) x `',x)) args)))
(if-match (scanner string)
    (list$/q foo if-match $2)

Here, although the $2 is in a position for evaluation (after macroexpansion), it will have no binding because it will have been pruned when naïvely walking the outer if-match macro. The if-match symbol argument to `list$/q ends up quoted, and should not be treated as a macro call.

Also, the pruner function must have special knowledge not just about the semantics of if-match, but also of any macro which can expand to if-match – see the attempt to handle when-match in the pruner. If a user were to have the temerity to define case-match

(defmacro case-match (string &rest clauses)
  (if (null clauses)
      `(if-match (,(caar clauses) ,string)
           (progn ,@(cdar clauses))
           (case-match string ,@(cdr clauses)))))

any attempt to nest a case-match inside an outer if-match is liable to fail, as the pruner has no knowledge of how to handle the case-match form.

All of these problems are solvable by using a proper code-walker. The code-walker should collect up all dollar symbols to be evaluated in the consequent of an if-match form, so that bindings for them can be generated, except for those with already existing lexical bindings within the if-match (not those from outside, otherwise nesting won’t work). For testing purposes, we’ll also signal a diagnostic condition within the macroexpander to indicate which dollar symbols we’ve found.

(define-condition if-match/walked-diagnostic (condition)
  ((symbols :initarg :symbols :reader if-match-symbols)))
(defmacro if-match/walked ((test string) consequent &optional alternative)
  (let* (dollar-symbols)
    (flet ((dollar-walker (subform context env)
             (declare (ignore context))
             (typecase subform
                (when (and (dollar-symbol-p subform)
                           (not (sb-walker:var-lexical-p subform env)))
                  (pushnew subform dollar-symbols))
               (t subform))))
      (handler-bind ((if-match/walked-diagnostic #'continue))
        (sb-walker:walk-form consequent nil #'dollar-walker))
      (let* ((dollar-symbols (sort dollar-symbols #'> :key #'dollar-symbol-p))
             (top (dollar-symbol-p (car dollar-symbols))))
        (with-simple-restart (continue "Ignore diagnostic condition")
          (signal 'if-match/walked-diagnostic :symbols dollar-symbols))
        (sb-int:with-unique-names (start end match-start match-end)
          (sb-int:once-only ((string string))
            `(multiple-value-bind (,start ,end ,match-start ,match-end)
                 (,test ,string)
               (declare (ignore ,end) (ignorable ,match-end))
               (if ,start
                   (if (< (length ,match-start) ,top)
                       (error "~S: too few matches: needed ~D, got ~D." 'if-match
                              ,top (length ,match-start))
                       (let ,(mapcar (lambda (s)
                                       (let ((i (1- (dollar-symbol-p s))))
                                         `(,s (subseq ,string (aref ,match-start ,i) (aref ,match-end ,i)))))
                                     (reverse dollar-symbols))

(I'm using sb-int:once-only and sb-int:with-unique-names to avoid having to include their definitions in this post, which is getting a bit lengthy). Testing this looks like

(defmacro test-if-match (form expected-symbols)
  `(handler-case (macroexpand-1 ',form)
     (if-match/walked-diagnostic (c)
       (assert (equal (if-match-symbols c) ',expected-symbols)))
     (:no-error (&rest values) (declare (ignore values)) (error "no diagnostic"))))
(test-if-match (if-match/walked (test string) (list $1 $2) 'foo) ($2 $1))
(test-if-match (if-match/walked (test string) (if (> (length $1) 6) '$10 '$8) nil) ($1))
(test-if-match (if-match/walked (scanner string)
                   (list $1
                         (if-match/walked (scanner2 string)
(test-if-match (if-match/walked (scanner string) (list$/q foo if-match/walked $3) nil) ($3))
(defmacro case-match/walked (string &rest clauses)
  (if (null clauses)
      `(if-match/walked (,(caar clauses) ,string)
           (progn ,@(cdar clauses))
           (case-match/walked string ,@(cdr clauses)))))
(test-if-match (if-match/walked (scanner string)
                   (case-match/walked $1
                     (foo $2)
                     (bar $3)))

To summarize: I’ve shown here how to make use of a full code-walker to make a couple of code transforming macros more robust. Full code-walkers can do more than just what I've shown here: the sb-walker:walk-form interface can also inhibit macroexpansion, transform function calls into calls to other functions, while respecting the semantics of the Lisp operators in the code that is being walked and allowing some introspection of the lexical environment. Here, we have called sb-walker:walk-form for side effects from the walker function we’ve provided; it is also possible to use its value (that’s how sb-cltl2:macroexpand-all is implemented, for example). I hope that this can help users affected by the change in internal representation of backquote, as well as others who want to write advanced code-transforming macros. If the thought of using an SBCL-internal code-walker makes you a bit queasy (as well it might), you could instead start by looking at one or two other more explicitly-portable code-walkers out there, for example John Fremlin’s macroexpand-dammit, the walker in Alex Plotnick's CLWEB literate programming system (github link), or the code walker in iterate.

backquote and pretty printing

There was a bit of a kerfuffle following the 1.2.2 release of SBCL, regarding the incompatible change in the internals of the backquote reader macro.

Formally, implementations can choose how to implement the backquote reader macro (and its comma-based helpers): the semantics of backquote are defined only after evaluation:

An implementation is free to interpret a backquoted form F1 as any form F2 that, when evaluated, will produce a result that is the same under equal as the result implied by the above definition, provided that the side-effect behavior of the substitute form F2 is also consistent with the description given above.

(CLHS 2.4.6; emphasis mine)

There are also two advisory notes about the representation:

Often an implementation will choose a representation that facilitates pretty printing of the expression, so that (pprint '`(a ,b)) will display `(a ,b) and not, for example, (list 'a b). However, this is not a requirement.

(CLHS; added quote in example mine), and:

Implementors who have no particular reason to make one choice or another might wish to refer to IEEE Standard for the Scheme Programming Language, which identifies a popular choice of representation for such expressions that might provide useful to be useful compatibility for some user communities.

(CLHS; the Scheme representation reads `(foo ,bar) as (quasiquote (foo (unquote bar))))

The problem the new implementation of backquote is attempting to address is the first one: pretty printing. To understand what the problem is, an example might help: imagine that we as Common Lisp programmers (i.e. not implementors, and aiming for portability) have written a macro bind which is exactly equivalent to let:

(defmacro bind (bindings &body body)
  `(let ,bindings ,@body))

and we want to implement a pretty printer for it, so that (pprint '(progn (bind ((x 2) (z 3)) (if *print-pretty* (1+ x) (1- y))))) produces

  (bind ((x 2)
         (z 3))
    (if *print-pretty*
        (1+ x)
        (1- y))))

What does that look like? Writing pretty-printers is a little bit of a black art; a first stab is something like:

(defun pprint-bind (stream object)
  (pprint-logical-block (stream object :prefix "(" :suffix ")")
    (write (pprint-pop) :stream stream)
    (write-char #\Space stream)
    (pprint-logical-block (stream (pprint-pop) :prefix "(" :suffix ")")
        (write (pprint-pop) :stream stream)
        (pprint-newline :mandatory stream)))
    (pprint-indent :block 1 stream)
    (pprint-newline :mandatory stream)
      (write (pprint-pop) :stream stream)
      (pprint-newline :mandatory stream))))
(set-pprint-dispatch '(cons (eql bind)) 'pprint-bind)

The loop noise is necessary because we're using :mandatory newlines; a different newline style, such as :linear, might have let us use a standard utility function such as pprint-linear. But otherwise, this is straightforward pretty-printing code, doing roughly the equivalent of SBCL's internal pprint-let implementation, which is:

(formatter "~:<~^~W~^ ~@_~:<~@{~:<~^~W~@{ ~_~W~}~:>~^ ~_~}~:>~1I~:@_~@{~W~^ ~_~}~:>")

A few tests at the repl should show that this works with nasty, malformed inputs (“malformed” in the sense of not respecting the semantics of bind) as well as expected ones:

(pprint '(bind))
(pprint '(bind x))
(pprint '(bind x y))
(pprint '(bind (x y) z))
(pprint '(bind ((x 1) (y 2)) z))
(pprint '(bind ((x 1) (y 2)) z w))
(pprint '(bind . 3))
(pprint '(bind x . 4))
(pprint '(bind (x . y) z))
(pprint '(bind ((x . 0) (y . 1)) z))
(pprint '(bind ((x) (y)) . z))
(pprint '(bind ((x) y) z . w))

Meanwhile, imagine a world where the backquote reader macro simply wraps (quasiquote ...) around its argument, and comma likewise wraps (unquote ...):

(set-macro-character #\` (defun read-backquote (stream char)
                           (list 'quasiquote (read stream t nil t))))
(set-macro-character #\, (defun read-comma (stream char)
                           (list 'unquote (read stream t nil t))))

Writing pretty-printer support for that is easy, right?

(defun pprint-quasiquote (stream object)
  (write-char #\` stream)
  (write (cadr object) :stream stream))
(defun pprint-unquote (stream object)
  (write-char #\, stream)
  (write (cadr object) :stream stream))
(set-pprint-dispatch '(cons (eql quasiquote) (cons t null)) 'pprint-quasiquote)
(set-pprint-dispatch '(cons (eql unquote) (cons t null)) 'pprint-unquote)

(ignoring for the moment what happens if the printed representation of object happens to start with a @ or .)

(pprint '(quasiquote (x (unquote y))))

The problem arises when we try to combine these two things. In particular, what happens when we attempt to print backquoted forms:

(pprint '`(bind ,y z))

What we would hope to see is something like

`(bind ,y

but what we actually get is

`(bind (unquote

because each of the bindings in bind is printed individually, rather than the bindings being printed as a whole. And, lest there be hopes that this can be dealt with by a slightly different way of handling the pretty printing in pprint-bind, note that it's important that (pprint '(bind (function y) z)) print as

(bind (function

and not as

(bind #'y

so the only way to handle this is to know the magical symbols involved in backquote and comma reader macros – but that is not portably possible. So, we've come to the point where the conclusion is inevitable: it is not possible for an implementation to support list-structured quasiquote and unquote reader macros and general pretty printing for user-defined operators. (This isn’t the only failure mode for the combination of unquote-as-list-structure and pretty-printing; it’s surprisingly easy to write pretty-printing functions that fail to print accurately, not just cosmetically as above but catastrophically, producing output that cannot be read back in, or reads as a structurally unequal object to the original.)

The new implementation, by Douglas Katzman, preserves the implementation of the backquote reader macro as a simple list, but comma (and related reader macros) read as internal, literal structures. Since these internal structures are atoms, not lists, they are handled specially by pprint-logical-block and friends, and so their own particular pretty-printing routines always fire. The internal quasiquote macro ends up extracting and arranging for appropriate evaluation and splicing of unquoted material, and everything ends up working.

Everything? Well, not quite: one or two programmer libraries out there implemented some utility functionality – typically variable renaming, automatic lambda generation, or similar – without performing a full macroexpansion and proper codewalk. That code was in general already broken, but it is true that in the past generating an example to demonstrate the breakage would have to violate the general expectation of what “normal” Lisp code would look like, whereas as a result of the new implementation of backquote in SBCL the symptoms of breakage were much easier to generate. Several of these places were fixed before the new implementation was activated, such as iterate's #l macro; among the things to be dealt with after the new implementation was released was the utility code from let-over-lambda (a workaround has been installed in the version distributed from github), and there is still a little bit of fallout being dealt with (e.g. a regression in the accuracy of source-location tracking). But overall, I think the new implementation of backquote has substantial advantages in maintainability and correctness, and while it’s always possible to convince maintainers that they’ve made a mistake, I hope this post explains some of why the change was made.

Meanwhile, I've released SBCL version 1.2.3 – hopefully a much less “exciting” release...

azed 2199

Relatively simple, I feel, though I did start it early in the week and only finished it on Saturday, so “simple” doesn’t necessarily mean “quick to solve”. I’ve learnt some things this week, for example that baseball teams can be described as Enneads.

  • 1ac: ALPHABETIFORM: English habit with pal from abroad – like letters (13): *(E(nglish) HABIT PAL FORM)
  • 11ac: CARIB: Native American piranha, tailless (5): CARIB(e) (piranha)
  • 13ac: NAPOO: Down on ducks, dead (5): NAP (down) + OO (ducks, as in cricket
  • 14ac: TWEEDIEST: Chirrup about withers, like most hearty outdoor types (9): TWEET (chirrup) around DIES (withers)
  • 15ac: DODDARD: Like decayed old trees, not even protected by Pakistani possibly (7)
  • 17ac: ENNEADS: Baseball teams? BBC rarely shows such, we hear (7): the BBC is funded through the licence fee, and hence rarely shows “any ads”.
  • 19ac: THENCE: Article about cheer our age lost – and after that? (6): THE (article) around ENCOURAGE (cheer) - OUR - AGE
  • 20ac: MOORVA: Fibre requiring one to travel speedily from east to west (6): A (one) + VROOM (travel speedily) reversed
  • 22ac: OUSTER: Being dispossessed one has to change routes (6): *(ROUTES)
  • 24ac: INSTEP: Capital cast by Epstein remoulded for arch (6): *((e)PSTEIN)
  • 26ac: INFALLS: More than one confluence is including northern and southern river length (7): IS around N(orthern) + FAL (southern river) + L(ength)
  • 28ac: RHENISH: It’s damaged her shin, or possibly hock (7): *(HER SHIN)
  • 31ac: APPOINTEE: Primate holding tiptop position? He’s got the job (9): APE (primate) around POINTE (tiptop position)
  • 32ac: LEARN: Master Edward by name? (5): LEAR (Edward; ref. Edward Lear) + N(ame)
  • 33ac: TITUP: Canter nag, mounted (5): TIT (nag) + UP (mounted)
  • 34ac: SCANDAL SHEETS: Yellow press, unbridled and scatheless, scents lead and has broadcast (13, 2 words): *(AND SCATHELESS) and *(SCENTS LEAD HAS). With more Azed exposure I’ve become used to triple clues, but I think this is the first time I’ve seen a double anagram.
  • 1dn: ACADEMICALS: E.g. Hamilton, with special gown, cap, etc (11): ACADEMICAL (ref. Hamilton Academical) + S(pecial)
  • 2dn: LAZO: Llama’s head has me round? Well, possibly (4): L(lama) + AZ (“me” = Azed) + O (round) &lit.
  • 3dn: PRODNOSE: Morse, say, writing about crooked don (8): PROSE (writing) around *(DON)
  • 4dn: HINDER: Back block (6): Double Definition
  • 5dn: BOWR: Stewart’s motor maybe requiring some elbow-room (4): elBOW Room
  • 6dn: TRENTON: River beside state capital (7): TRENT (river) + ON (beside); ref. Trenton, NJ
  • 7dn: INDY: Motor race creating row, quiet banished from the start (4): SHINDY (row) - SH (quiet)
  • 8dn: FAINE: The old pretend being sensitive about acting (5): FAE (sensitive) around IN (acting)
  • 9dn: ROSACE: Prize badge like this is held up in contest (6): SO (like this) reversed in RACE (contest)
  • 10dn: MOTHERSHIPS: Supply craft most deployed when bagging plunder in Scotland (11): *(MOST) around HERSHIP (Scot. plunder)
  • 12dn: REDSHIRTS: College sportsmen (selection deferred) managed thirds in residence (9): *(THIRDS) in RES(idence)
  • 16dn: ANTLIATE: Insect item (not head), turned up proboscis-like (8): ANT (insect) + (d)ETAIL reversed
  • 18dn: DAPHNIA: Had a nip after fiddling with water fleas (7): *(HAD A NIP)
  • 21dn: ON SPEC: Somebody seizing odds on colt, as a gamble (6, 2 words): ONE (somebody) around S(tarting) P(rice) (odds) + C(olt)
  • 23dn: SANNIE: Famous orphan after special bit of beachwear in Largs? (6): S(pecial) + ANNIE (famous orphan; ref. Little Orphan Annie
  • 25dn: THORN: Letter penned by author (Norse) (5): auTHOR Norse
  • 27dn: LIND: Nightingale (so-called) in Tilia tree, first to fly off in dark (4): Double Definition + (b)LIND (dark); ref. Jenny Lind
  • 27dn: HEWS: Hacks: ten staggered away from what dailies serve up? (4): THE NEWS (what dailies serve up) - TEN
  • 30dn: SMUT: Black spot? One of two in smash belly up (4): S (one of two in SmaSh) + TUM (belly) reversed
london employment visualization part 2

Previously, I did all the hard work to obtain and transform some data related to London, including borough and MSOA shapes, population counts, and employment figures, and used them to generate some subjectively pretty pictures. I promised a followup on the gridSVG approach to generating visualizations with more potential for interactivity than a simple picture; this is the beginning of that.

Having done all the heavy lifting in the last post, including being able to generate ggplot objects (whose printing results in the pictures), it is relatively simple to wrap output to SVG instead of output to PNG around it all. In fact it is extremely simple to output to SVG; simply use an SVG output device

svg("/tmp/london.svg", width=16, height=10)

rather than a PNG one

png("/tmp/london.png", width=1536, height=960)

(which brings back for me memories of McCLIM, and my implementation of an SVG backend, about a decade ago). So what does that look like? Well, if you’ve entered those forms at the R repl, close the png device

and then (the currently active device being the SVG one)


default (cairo) SVG device

That produces an SVG file, and if SVG in and of itself is the goal, that’s great. But I would expect that the main reason for producing SVG isn’t so much for the format itself (though it is nice that it is a vector image format rather than rasterized, so that zooming in principle doesn’t cause artifacts) but for the ability to add scripting to it: and since the output SVG doesn’t retain any information about the underlying data that was used to generate it, it is very difficult to do anything meaningful with it.

I write “very difficult” rather than “impossible”, because in fact the SVGAnnotation package aimed to do just that: specifically, read the SVG output produced by the R SVG output device, and (with a bit of user assistance and a liberal sprinkling of heuristics) attempt to identify the regions of the plot corresponding to particular slices of datasets. Then, using a standard XML library, the user could decorate the SVG with extra information, add links or scripts, and essentially do whatever they needed to do; this was all wrapped up in an svgPlot function. The problem with this approach is that it is fragile: for example, one heuristic used to identify a lattice plot area was that there should be no text in it, which fails for custom panel functions with labelled guidlines. It is possible to override the default heuristic, but it’s difficult to build a robust system this way (and in fact when I tried to run some two-year old analysis routines recently, the custom SVG annotation that I wrote broke into multiple pieces given new data).

gridSVG’s approach is a little bit different. Instead of writing SVG out and reading it back in, it relies on the grid graphics engine (so does not work with so-called base graphics, the default graphics system in R), and on manipulating the grid object which represents the current scene. The gridsvg pseudo-graphics-device does the behind-the-scenes rendering for us, with some cost related to yet more wacky interactions with R’s argument evaluation semantics which we will pay later.

gridsvg("/tmp/gridsvg-london.svg", width=16, height=10)

Because ggplot uses grid graphics, this just works, and generates a much more structured svg file, which should render identically to the previous one:

SVG from gridSVG device

If it renders identically, why bother? Well, because now we have something that writes out the current grid scene, we can alter that scene before writing out the document (at time). For example, we might want to add tooltips to the MSOAs so that their name and the quantity value can be read off by a human. Wrapping it all up into a function, we get <- function(expr, subsetexpr=TRUE, filename="/tmp/london.svg") {

We need to compute the subset in this function, even though we’re going to be using the full dataset in when we call it, in order to get the values and zone labels. <- droplevels(, list(london$msoa.fortified, substitute(subsetexpr))))

Then we need to map (pun mostly intended) the values in the fortified data frame to the polygons drawn; without delving into the format, my intuition is that the fortified data frame contains vertex information, whereas the grid (and hence SVG) data is organized by polygons, and there may be more than one polygon for a region (for example if there are islands in the Thames). Here we simply generate an index from a group identifier to the first row in the dataframe in that group, and use it to pull out the appropriate value and label.

    is <- match(levels($group),$group)
    vals <- eval(substitute(expr),[is]
    labels <- levels($zonelabel)[$zonelabel[is]]

Then we pay the cost of the argument evaluation semantics. My first try at this line was gridsvg(filename, width=16, height=10), which I would have (perhaps naïvely) expected to work, but which in fact gave me an odd error suggesting that the environment filename was being evaluated in was the wrong one. Calling gridsvg like this forces evaluation of filename before the call, so there should be less that can go wrong., list(filename, width=16, height=10))

And, as before, we have to do substitutions rather than evaluations to get the argument expressions evaluated in the right place:

    print(, list(substitute(expr), substitute(subsetexpr))))

Now comes the payoff. At this point, we have a grid scene, which we can investigate using Doing so suggests that the map data is in a grid object named like GRID.polygon followed by an integer, presumably in an attempt to make names unique. We can “garnish” that object with attributes that we want: some javascript callbacks, and the values and labels that we previously calculated.

                 onmouseover=rep("showTooltip(evt)", length(is)),
                 onmouseout=rep("hideTooltip()", length(is)),
                 zonelabel=labels, value=vals,
                 group=FALSE, grep=TRUE)

We need also to provide implementations of those callbacks. It is possible to do that inline, but for simplicity here we simply link to an external resource.


Then close the gridsvg device, and we’re done!

Then produces:

proportion employed full-time

which is some kind of improvement over a static image for data of this complexity.

And yet... the perfectionist in me is not quite satisfied. At issue is a minor graphical glitch, but it’s enough to make me not quite content; the border of each MSOA is stroked in a slightly lighter colour than the fill colour, but that stroke extends beyond the border of the MSOA region (the stroke’s centre is along the polygon edge). This means that the strokes from adjacent MSOAs overlie each other, so that the most recently drawn obliterates any drawn previously. This also causes some odd artifacts around the edges of London (and into the Thames, and pretty much obscures the river Lea).

This can be fixed by clipping; I think the trick to clip a path to itself counts as well-known. But clipping in SVG is slightly hard, and the gridSVG facilities for doing it work on a grob-by-grob basis, while the map is all one big polygon grid object. So to get the output I want, I am going to have to perform surgery on the SVG document itself after all; we are still in a better position than before, because we will start with a sensible hierarchical arrangement of graphical objects in the SVG XML structure, and gridSVG furthermore provides some introspective capabilities to give XML ids or XPath query strings for particular grobs.

grid.export exports the current grid scene to SVG, returning a list with the SVG XML itself along with this mapping information. We have in the SVG output an arbitrary number of polygon objects; our task is to arrange such that each of those polygons has a clip mask which is itself. In order to do that, we need for each polygon a clipPath entry with a unique id in a defs section somewhere, where each clipPath contains a use pointing to the original polygon’s ID; then each polygon needs to have a clip-path style property pointing to the corresponding clipPath object. Clear?

addClipPaths <- function(gridsvg, id) {

given the return value of grid.export and the identifier of the map grob, we want to get the set of XML nodes corresponding to the polygons within that grob.

    ns <- getNodeSet(gridsvg$svg, sprintf("%s/*", gridsvg$mappings$grobs[[id]]$xpath))

Then for each of those nodes, we want to set a clip path.

    for (i in 1:length(ns)) {
        addAttributes(ns[[i]], style=sprintf("clip-path: url(#clipPath%s)", i))

For each of those nodes, we also need to define a clip path

    clippaths <- list()
    for (i in 1:length(ns)) {
        clippaths[[i]] <- newXMLNode("clipPath", attrs=c(id=sprintf("clipPath%s", i)))
        use <- newXMLNode("use", attrs = c("xlink:href"=sprintf("#%s", xmlAttrs(ns[[i]])[["id"]])))
        addChildren(clippaths[[i]], kids=list(use))

And hook it into the existing XML

    defs <- newXMLNode("defs")
    addChildren(defs, kids=clippaths)
    top <- getNodeSet(gridsvg$svg, "//*[@id='gridSVG']")[[1]]
    addChildren(top, kids=list(defs))

Then our driver function needs some slight modifications:

gridsvg.london2 <- function(expr, subsetexpr=TRUE, filename="/tmp/london.svg") { <- droplevels(, list(london$msoa.fortified, substitute(subsetexpr))))
    is <- match(levels($group),$group)
    vals <- eval(substitute(expr),[is]
    labels <- levels($zonelabel)[$zonelabel[is]]

Until here, everything is the same, but we can’t use the gridsvg pseudo-graphics device any more, so we need to do graphics device handling ourselves:

    pdf(width=16, height=10)
    print(, list(substitute(expr), substitute(subsetexpr))))
                 onmouseover=rep("showTooltip(evt)", length(is)),
                 onmouseout=rep("hideTooltip()", length(is)),
                 zonelabel=labels, value=vals,
                 group=FALSE, grep=TRUE)

Now we export the scene to SVG,

    gridsvg <- grid.export()

find the grob containing all the map polygons,

    grobnames <-, print=FALSE)$name
    grobid <- grobnames[[grep("GRID.polygon", grobnames)[1]]]

add the clip paths,

    addClipPaths(gridsvg, grobid)
    saveXML(gridsvg$svg, file=filename)

and we’re done!

Then gridsvg.london2(fulltime/(allages-younger-older)) produces:

proportion employed full-time (with polygon clipping)

and I leave whether the graphical output is worth the effort to the beholder’s judgment.

As before, these images contain National Statistics and Ordnance Survey data © Crown copyright and database right 2012.

azed 2198

Even though I was on holiday, I still only managed to complete one crossword. Holidaying with two small children turns out to be fairly similar in that respect to staying at home with two small children. Away from dictionaries, hence rather more speculative clue parses than usual.

  • 1ac: GUIMBARD: Twangy instrument I am, bass, kept in box? (8): IM (I am) + B(ass) in GUARD (box)
  • 7ac: REAM: Widen barrel in making Scotch overflow (4)
  • 10ac: MORTADELLA: Spiced sausage: a little bit’s wrapped in mushroom, look (10): TAD (a little bit) in MOREL (mushroom) + LA (look)
  • 11ac: GIBE: Scoff to present to Scots will include wee bit o’ bannock (4): GIE (scot. “present”) around B(annock)
  • 12ac: EMBAY: Millions involved in online sales forum, no longer steep (5): M(illions) in EBAY (online sales forum; ref. eBay
  • 14ac: EQUINIA: ‘Spangle’ not first? Reverse of ideal – bad news for stables (7): (s)EQUIN (spangle) + AI (ideal) reversed
  • 17ac: BOX GIRDER: Lass almost dead bitten by dog? Strong support needed (9, 2 words): GIR(l) (lass) + D(ead) in BOXER (dog)
  • 18ac: CADENT: Falling for the Bard, study cast in production of act (6): DEN (study) in *(ACT)
  • 20ac: ENSURE: Guarantee supplied by nurse, second joining old practice (5): N(urse) + S(econd) in EURE (obs. practice?)
  • 21ac: TRIXIE: Girl familiarly trapping former farceur in marriage (6): RIX (former farceur, ref. Brian Rix
  • 23ac: SCENDS: Cloudy and close on board, pitches into trough sometimes (6): C(loudy) + END (close) in SS (on board)
  • 24ac: OIL TANKER: Sheik’s ultimate relation is rolling? Herein lies his wealth maybe (9, 2 words): *((shei)K + RELATION) &lit.
  • 27ac: GEMMING: Once budding twins I’ll have left for good, pocketing millions (7): GEMINI (twins) - I + G(ood) outside M(illions)
  • 29ac: INSET: We’ll get this for delightful gusset? (5)
  • 30ac: BRIO: Life? Account of one involving queen (4): BIO (account of one) around R(egina)
  • 31ac: SPEAR GRASS: What’s the point in rolling gasper’s sharp herbage? (10, 2 words)
  • 32ac: HURT: Front of railcar goes into e.g. caboose, causing damage (4): R(ailcar) in HUT (e.g. caboose)
  • 33ac: VESTMENT: State robe – in this is something you hope’ll appreciate (8): IN + VESTMENT = “something you hope’ll appreciate”
  • 1dn: GAGE: Stake old fruit tree (4): Double Definitiona
  • 2dn: UBIQUARIAN: As in Rome ruins of Urban II are seen around all over the place (10): QUA (lat. “as”) in *(URBAN II)
  • 3dn: IMBUE: Contents of a cloud landing on earth, wet (5): (n)IMBU(s) (cloud) + E(ARTH)
  • 4dn: BRUNONIAN: Typical of Edinburgh doctor to keep talking, one breaking tabu (9): RUN ON (keep talking) + I (one) in BAN (tabu)
  • 5dn: ATWIXT: A fool’s wrong thus? (6): A TWIT atwixt (thus) X (wrong) &lit.
  • 6dn: RAVAGE: Plunder silver, crazy about it (6) AG (silver) in RAVE (crazy)
  • 7dn: REMORSE: Some err if misled – this may result (7): *(SOME ERR) &lit.
  • 8dn: ELBE: River bass swallows wriggling eel? The opposite (4): *(EEL) around B(ass)
  • 9dn: MAYORESS: Magistrate’s woman long ago in US state (briefly) (8): YORE (long ago) in MASS (US state briefly)
  • 13dn: ABERDONIAN: English denied a drink on fellow, a Scot maybe, miserly (10) A BEER (drink) - E(nglish) + DON (fellow) + IAN (a Scot maybe)
  • 15dn: MINCEMEAT: Pie filling? Do this for team, say (9): TEAM = *(MEAT)
  • 16dn: ACTORISH: Hit ‘gobsmacked’ with Oscar, luvvie-style (8): *(HIT + OSCAR)
  • 19dn: EXTREAT: Making copy of old transactions leave rings (7)
  • 22dn: ENGAGE: English part of history book (6): EN(glish) + GAGE (obs. “part”)
  • 23dn: SKEERS: Scats smell awful when hauled up on board (6): REEK (smell awful) reversed in SS (on board)
  • 25dn: BIRSE: Early part of Jock’s beard beginning to rustle in the wind (5): R(ustle) in BISE (wind)
  • 26dn: USER: Legal right, established over time, because realistic in part (4): becaUSE Realistic
  • 28dn: GOAT: Butter to attack vigorously (4): GO AT (attack vigorously)
azed 2197
  • 1ac: OUTFANGTHIEF: Fit to hang, feu being exercised? It could clobber distant felon (12): *(FIT TO HANG FEU)
  • 10ac: FERETORY: Con, after running free, in shrine (8): *(FREE) + TORY (Con)
  • 11ac: ARNA: Source of ghee? You’ll find this is used in Asian recipe (4): (ARNA + IS) = ASIAN R(ecipe)
  • 12ac: SLOT: Soak imbibing litre in bar ...; (4): SOT (soak) + L(itre)
  • 13ac: STOPLOSS: ... Soak similarly drunk with slops meant to minimize wastage? (8): *(SOT) (soak similarly) + *(SLOPS)
  • 14ac: TAPIOCA: Pudding, alternative to pie in a covering, rolled round (7): PI (alternative of “pie”) in A COAT (covering) reversed
  • 16ac: ERICA: Bushy plant amateur removed from Uncle Sam (5): AMERICA (Uncle Sam) - AM(ateur)
  • 18ac: CUPEL: Goldsmith’s vessel made of copper I can cut from alembic (5): CU (copper) + PELICAN (alembic) - I CAN
  • 19ac: PETITORY: Troubled type makes case for it before submitting formal request (8): IT + OR (before) with *(TYPE) around
  • 22ac: INTERLAY: Place between Tay and liner at sea (8): *(TAY LINER)
  • 24ac: NAIAS: Wild Asian water plants (5): *(ASIAN)
  • 27ac: FETES: Honours school half-heartedly? (5): FET(t)ES; ref. Fettes
  • 29ac: HIGHSET: Cracking eight’s about No. 1 in Henley, pre-eminently placed (7): *(EIGHTS) around H(enley)
  • 30ac: ROTENONE: Saw inside their eggs what may be fatal for fish (8): TENON (saw) in ROE (fish eggs)
  • 31ac: ACOL: Amateur given saddle in system leading to contracts? (4): A(mature) + COL (saddle); ref. Acol
  • 32ac: MUMM: Masquerade displaying fizz (4): Double Definition; ref. Mumm
  • 33ac: STILLAGE: Stand for casks badly stored in shelf (8): ILL (badly) in STAGE (shelf)
  • 34ac: STREET SMARTS: Strong teamsters organized in wily Chicago-style know-how (12, 2 words): STR(ong) + *(TEAMSTERS)
  • 1dn: OFSTED: Watchdog ’ad position in chair, fine inside (6): ’OSTED (’ad position in chair) around F(ine); ref. Ofsted
  • 2dn: TROPISTIC: Responding to stimulus, I mostly remain after drink’s served up (9): PORT (drink) reversed + I + STIC(k) (remain)
  • 3dn: FETICHE: Obsession one’s trapped by was effective endlessly (7): I in FETCHE(d) (was effective)
  • 4dn: NOTCHEL: Denial of liability in brief letter left involving children (7): NOTE (brief letter) around CH(ildren) + L(eft)
  • 5dn: GROAT: Old-style fourpenny one’s attack catching Bunter’s rear (5): GO AT (attack) + (bunte)R
  • 6dn: TYPIC: Representative showing selection of pretty pictures (5): pretTY PICtures
  • 7dn: HALM: Holy place I do out in straw (4): HALIDOM (holy place) - I DO
  • 8dn: ENS PER SE: Pen’s translated Irish Gaelic? It exists purely in the mind (8, 3 words): *(PENS) + ERSE (Irish Gaelic)
  • 9dn: FASTLY: A stone inset by fashionable American firmly (old) (6): A ST(one) in FLY (?)
  • 15dn: SPORTS CAR: Morgan maybe having to exhibit evidence of injury (9, 2 words): SPORT (exhibit) SCAR (evidence of injury)
  • 17dn: RUNABOUT: Light car leaked fluid around autobahn (8): RUN OUT (leaked fluid) around A(uto)B(ahn)
  • 20dn: TAHINIS: Oily pastes from sea fish containing measure of fluid (7): TAIS (sea fish) around HIN (Hebr. fluid measure)
  • 21dn: TREHALA: ‘Turkish delight’? Watchful when it comes up to grab centre of that! (7): ALERT (watchful) reversed around (t)HA(t)
  • 22dn: INARMS: Like wee one belonging to a besotted Mrs (6): IN (belonging to) + A + *(MRS)
  • 23dn: ISTLES: Plant fibres nursed by botanist Leschenault (6): botanIST LESchenault
  • 25dn: SENSE: Since old, presiding over English faculty (5): SENS (obs. “since”) + E(nglish)
  • 26dn: SHOTT: Hot and very dry around watercourse, shallow (5): H(ot) in SO (very) TT (dry)
  • 28dn: DEME: Classical township is wiped out in the end (4): DEMISE (the end) - IS
london employment visualization

I went to londonR a month or so ago, and the talk whose content stayed with me most was the one given by Simon Hailstone on map-making and data visualisation with ggplot2. The take-home message for me from that talk was that it was likely to be fairly straightforward to come up with interesting visualisations and data mashups, if you already know what you are doing.

Not unrelatedly, in the Goldsmiths Data Science MSc programme which we are running from October, it is quite likely that I will be teaching a module in Open Data, where the students will be expected to produce visualisations and mashups of publically-accessible open data sets.

So, it is then important for me to know what I am doing, in order that I can help the students to learn to know what they are doing. My first, gentle steps are to look at life in London; motivated at least partly by the expectation that I encounter when dropping off or picking up children from school that I need more things to do with my time (invitations to come and be trained in IT at a childrens' centre at 11:00 are particularly misdirected, I feel) I was interested to try to find out how local employment varies. This blog serves at least partly as a stream-of-exploration of not just the data but also ggplot and gridsvg; as a (mostly satisfied) user of lattice and SVGAnnotation in a previous life, I'm interested to see how the R graphing tools have evolved.

Onwards! Using R 3.1 or later: first, we load in the libraries we will need.


Then, let's define a couple of placeholder lists to accumulate related data structures.

london <- list()
tmp <- list()

Firstly, let's get hold of some data. I want to look at employment; that means getting hold of counts of employed people, broken down by area, and also counts of people living in that area in the first place. I found some Excel spreadsheets published by London datastore and the Office for National Statistics, with data from the 2011 census, and extracted from them some useful data in CSV format. (See the bottom of this post for more detail about the data sources).

london$labour <- read.csv("2011-labour.csv", header=TRUE, skip=1)
london$population <- read.csv("mid-2011-lsoa-quinary-estimates.csv", header=TRUE, skip=3)

The labour information here does include working population counts by area – specifically by Middle Layer Super Output Area (MSOA), a unit of area holding roughly eight thousand people. Without going into too much detail, these units of area are formed by aggregating Lower Layer Super Output Areas (LSOAs), which themselves are formed by aggregating census Output Areas. In order to get the level of granularity we want, we have to do some of that aggregation ourselves.

The labour data is published in one sheet by borough (administrative area), ward (local electoral area) and LSOA. The zone IDs for these areas begin E09, E05 and E01; we need to manipulate the data frame we have to include just the LSOA entries

london$labour.lsoa <- droplevels(subset(london$labour, grepl("^E01", ZONEID)))

and then aggregate LSOAs into MSOAs “by hand”, by using the fact that LSOAs are named such that the name of their corresponding MSOA is all but the last letter of the LSOA name (e.g. “Newham 001A” and “Newham 001B” LSOAs both have “Newham 001” as their parent MSOA). In doing the aggregation, all the numeric columns are count data, so the correct aggregation is to sum corresponding entries of numerical data (and discard non-numeric ones)

tmp$nonNumericNames <- c("DISTLABEL", "X", "X.1", "ZONEID", "ZONELABEL")
tmp$msoaNames <- sub(".$", "", london$labour.lsoa$ZONELABEL)
tmp$data <- london$labour.lsoa[,!names(london$labour.lsoa) %in% tmp$nonNumericNames]

london$labour.msoa <- aggregate(tmp$data, list(tmp$msoaNames), sum)
tmp <- list()

The population data is also at LSOA granularity; it is in fact for all LSOAs in England and Wales, rather than just London, and includes some aggregate data at a higher level. We can restrict our attention to London LSOAs by reusing the zone IDs from the labour data, and then performing the same aggregation of LSOAs into MSOAs.

london$population.lsoa <- droplevels(subset(london$population, Area.Codes %in% levels(london$labour.lsoa$ZONEID)))
tmp$nonNumericNames <- c("Area.Codes", "Area.Names", "X")
tmp$msoaNames <- sub(".$", "", london$population.lsoa$X)
tmp$data <- london$population.lsoa[,!(names(london$population.lsoa) %in% tmp$nonNumericNames)]
london$population.msoa <- aggregate(tmp$data, list(tmp$msoaNames), sum)
tmp <- list()

This is enough data to be getting on with; it only remains to join the data frames up. The MSOA name column is common between the two tables; we can therefore reorder one of them so that it lines up with the other before simply binding columns together:

london$employment.msoa <- cbind(london$labour.msoa, london$population.msoa[match(london$labour.msoa$Group.1, london$population.msoa$Group.1)])

(In practice in this case, the order of the two datasets is the same, so the careful reordering is in fact the identity transformation)

tmp$notequals <- london$population.msoa$Group.1 != london$labour.msoa$Group.1
tmp <- list()

Now, finally, we are in a position to look at some of the data. The population data contains a count of all people in the MSOA, along with counts broken down into 5-year age bands (so 0-5 years old, 5-10, and so on, up to 90+). We can take a look at the proportion of each MSOA's population in each age band:

ggplot(reshape(london$population.msoa, varying=list(3:21),
               direction="long", idvar="Group.1")) +
    geom_line(aes(x=5*time-2.5, y=X0.4/All.Ages, group=Group.1),
              alpha=0.03, show_guide=FALSE) +
    scale_x_continuous(name="age / years") +
    scale_y_continuous(name="fraction") +

age distributions in London MSOAs

This isn't a perfect visualisation; it does show the broad overall shape of the London population, along with some interesting deviations (some MSOAs having a substantial peak in population in the 20-25 year band, while rather more peak in the 25-30 band; a concentration around 7% of 0-5 year olds, with some significant outliers; and a tail of older people with some interesting high outliers in the 45-50 and 60-65 year bands, as well as one outlier in the 75-80 and 80-85).

That was just population; what about the employment statistics? Here's a visualisation of the fraction of people in full-time employment, treated as a distribution

ggplot(london$employment.msoa, aes(x=Full.time/All.Ages)) +
    geom_density() + theme_bw()

fraction in full-time employment in London MSOAs

showing an overal concentration of full-time employment level around 35%. But this of course includes in the population estimate people of non-working ages; it may be more relevant to view the distribution of full-time employment as a fraction of those of “working age”, which here we will define as those between 15 and 65 (not quite right but close enough):

london$employment.msoa$younger = rowSums(london$employment.msoa[,62:64])
london$employment.msoa$older = rowSums(london$employment.msoa[,75:80])
ggplot(london$employment.msoa, aes(x=Full.time/(All.Ages-younger-older))) +
    geom_density() + theme_bw()

fraction of those of working age in full-time employment in London MSOAs

That's quite a different picture: we've lost the bump of outliers of high employment around 60%, and instead there's substantial structure at lower employment rates (plateaus in the density function around 50% and 35% of the working age population). What's going on? The proportions of people not of working age must not be evenly distributed between MSOAs; what does it look like?

ggplot(london$employment.msoa, aes(x=older/All.Ages, y=younger/All.Ages)) +
    geom_point(alpha=0.1) + geom_density2d() + coord_equal() + theme_bw()

distribution of younger vs older people in London MSOAs

So here we see that a high proportion of older people in an MSOA (around 20%) is a strong predictor of the proportion of younger people (around 17%); similarly, if the younger population is 25% or so of the total population, there are likely to be less than 10% of people of older non-working ages. But at intermediate values of either older or younger proportions, 10% of older or 15% of younger, the observed variation of the other proportion is very high: what this says is that the proportion of people of working age in an MSOA varies quite strongly, and for a given proportion of people of working age, some MSOAs have more older people and some more younger people. Not a world-shattering discovery, but important to note.

So, going back to employment: what are the interesting cases? One way of trying to characterize them is to fit the central portion of the observations to some parameterized distribution, and then examining cases where the observation is of a sufficiently low probability. The extremevalues R package does some of this for us; it offers the option to fit a variety of distributions to a set of observations with associated quantiles. Unfortunately, the natural distribution to use for proportions, the Beta distribution, is not one of the options (fitting involves messing around with derivatives of the inverse of the incomplete Beta function, which is a bit of a pain), but we can cheat: instead of doing the reasonable thing and fitting a beta distribution, we can transform the proportion using the logit transform and fit a Normal distribution instead, because everything is secretly a Normal distribution. Right? Right? Anyway.

tmp$outliers <- with(london$employment.msoa, getOutliers(log(Full.time/All.Ages/(1-Full.time/All.Ages))))
tmp$high <- london$employment.msoa$Group.1[tmp$outliers$iRight]
tmp$low <-london$employment.msoa$Group.1[tmp$outliers$iLeft]
print(tmp[c("high", "low")])
tmp <- list()

This shows up 9 MSOAs as having a surprisingly high full employment rate, and one (poor old Hackney 001) as having a surprisingly low rate (where “surprising” is defined relative to our not-well-motivated-at-all Normal distribution of logit-transformed full employment rates, and so shouldn't be taken too seriously). Where are these things? Clearly the next step is to put these MSOAs on a map. Firstly, let's parse some shape files (again, see the bottom of this post for details of where to get them from).

london$borough <- readShapePoly("London_Borough_Excluding_MHW")
london$msoa <- readShapePoly("MSOA_2011_London_gen_MHW")

In order to plot these shape files using ggplot, we need to convert the shape file into a “fortified” form:

london$borough.fortified <- fortify(london$borough)
london$msoa.fortified <- fortify(london$msoa)

and we will also need to make sure that we have the MSOA (and borough) information as well as the shapes:

london$msoa.fortified$borough <- london$msoa@data[as.character(london$msoa.fortified$id), "LAD11NM"]
london$msoa.fortified$zonelabel <- london$msoa@data[as.character(london$msoa.fortified$id), "MSOA11NM"]

For convenience, let's add to the “fortified” MSOA data frame the columns that we've been investigating so far:

tmp$is <- match(london$msoa.fortified$zonelabel, london$employment.msoa$Group.1)
london$msoa.fortified$fulltime <- london$employment.msoa$Full.time[tmp$is]
london$msoa.fortified$allages <- london$employment.msoa$All.Ages[tmp$is]
london$msoa.fortified$older <- london$employment.msoa$older[tmp$is]
london$msoa.fortified$younger <- london$employment.msoa$younger[tmp$is]

And now, a side note about R and its evaluation semantics. R has... odd semantics. The basic evaluation model is that function arguments are evaluated lazily, when the callee needs them, but in the caller's environment – except if the argument is a defaulted one (when no argument was actually passed), in which ase it's in the callee's environment active at the time of evaluation, except that these rules can be subverted using various mechanisms, such as simply looking at the call object that is currently being evaluated. There's significant complexity lurking, and libraries like ggplot (and lattice, and pretty much anything you could mention) make use of that complexity to give a convenient surface interface to their functions. Which is great, until the time comes for someone to write a function that calls these libraries, such as the one I'm about to present which draws pictures of London. I have to know about the evaluation semantics of the functions that I'm calling – which I suppose would be reasonable if there weren't quite so many possible options – and necessary workarounds that I need to install so that my function has the right, convenient evaluation semantics while still being able to call the library functions that I need to. I can cope with this because I have substantial experience with programming languages with customizeable evaluation semantics; I wonder how to present this to Data Science students.

With all that said, I present a fairly flexible function for drawing bits of London. This presentation of the finished object of course obscures all my exploration of things that didn't work, some for obvious reasons and some which remain a mystery to me. <- function(expr, subsetexpr=TRUE) {

We will call this function with an (unevaluated) expression to be displayed on a choropleth, to be evaluated in the data frame london$msoa.fortified, and an optional subsetting expression to restrict the plotting. We compute the range of the expression, independent of any data subsetting, so that we have a consistent scale between calls

    expr.limits <- range(eval(substitute(expr), london$msoa.fortified))

and compute a suitable default aesthetic structure for ggplot

    expr.aes <-, list(fill=substitute(expr), colour=substitute(expr)))

where substitute is used to propagate the convenience of an expression interface in our function to the same expression interface in the ggplot functions, and is necessary because aes does not evaluate its arguments, while we do need to perform the substitution before the call. <- droplevels(, list(london$msoa.fortified, substitute(subsetexpr))))

This is another somewhat complicated set of calls to make sure that the subset expression is evaluated when we need it, in the environment of the data frame. at this point contains the subset of the MSOAs that we will be drawing.

    (ggplot(london$borough.fortified) + coord_equal() +

Then, we set up the plot, specify that we want a 1:1 aspect ratio,

     (theme_minimal() %+replace%
      theme(panel.background = element_rect(fill="black"),
            panel.grid.major = element_blank(),
            panel.grid.minor = element_blank())) +

a black background with no grid lines,

     geom_map(,, modifyList(aes(map_id=id, x=long, y=lat), expr.aes), label="map") +

and plot the map.

     scale_fill_gradient(limits=expr.limits, low=mnsl2hex(darker(rgb2mnsl(coords(hex2RGB("#13432B", TRUE))))), high=mnsl2hex(darker(rgb2mnsl(coords(hex2RGB("#56F7B1", TRUE)))))) +
     scale_colour_gradient(limits=expr.limits, low="#13432B", high="#56F7B1", guide=FALSE) +

We specify a greenish colour scale, with the fill colour being slightly darker than the outline colour for prettiness, and

     geom_map(map=london$borough.fortified, aes(map_id=id, x=long, y=lat), colour="white", fill=NA, size=0.1)

also draw a superposition of borough boundaries to help guide the eye.


So, now, we can draw a map of full time employment:

proportion employed full-time

or the same, but just the outlier MSOAs:, zonelabel %in% london$outliers)

proportion employed full-time (outlier MSOAs)

or the same again, plotting only those MSOAs which are within 0.1 of the extremal values:, pmin(fulltime/(allages-younger-older) - min(fulltime/(allages-younger-older)), max(fulltime/(allages-younger-older)) - fulltime/(allages-younger-older)) < 0.1)

proportion employed full-time (near-extremal MSOAs)

Are there take-home messages from this? On the data analysis itself, probably not; it's not really a surprise to discover that there's substantial underemployment in the North and East of London, and relatively full employment in areas not too far from the City of London; it is not unreasonable for nursery staff to ask whether parents dropping children off might want training; it might be that they have the balance of probabilities on their side.

From the tools side, it seems that they can be made to work; I've only really been kicking the tyres rather than pushing them hard, and the number of times I had to rewrite a call to ggplot, aes or one of the helper functions as a response to an error message, presumably because of some evaluation not happening at quite the right time, is high – but the flexibility and interactivity they offer for exploration feels to me as if the benefits outweigh the costs.

The R commands in this post are here; the shape files are extracted from Statistical GIS Boundary Files for London and redistributed here under the terms of the Open Government Licence; they contain National Statistics and Ordnance Survey data © Crown copyright and database right 2012. The CSV data file for population counts is derived from an ONS publication, distributed here under the terms of the Open Government Licence, and contains National Statistics data © Crown copyright and database right 2013; the CSV data file for labour statistics is derived from a spreadsheet published by the London datastore, distributed here under the terms of the Open Government Licence, and contains National Statistics data © Crown copyright and database right 2013.

Still here after all that licence information? I know I promised some gridSVG evaluation too, but it's getting late and I've been drafting this post for long enough already. I will come back to it, though, because it was the more fun bit, and it makes some nifty interactive visualizations fairly straightforward. Watch this space.

azed 2196

The addition of the overlaps made for an amusing diversion; on the other hand, it strongly dictated the order of solving, which was to attack down clues as much as possible before working on the accrosses.

Down clues are normal. Each across clue, except 30, leads to two words which overlap, the last three letters of the first being the first three of the second. The overlaps occupy the central section of the diagram, bordered left and right by broken bars. Each across clue consists of clues to the two overlapping words, in either order; one of these is normal, but in the other the cryptic indication refers only to the full word minus the overlapping portion. Either of the two overlapping words may be clued first, and the two types of clue that make up each double clue may be in either order.

Let's see if I can represent the solving process clearly enough, with the power of colour!

  • 1ac: OVERSHOTCHPOT: Again, host subtly exceeded mutton broth, cold, with warm round bit of polenta (8, 8): OVER (again) + *(HOST) C(old) + HOT (warm) round P(olenta)
  • 11ac: PERICOPERABLE: Fit to function after visit to theatre, a rebel reread church extract, fine following page (8, 8): P(age) + ERIC (fine) OP (visit to theatre) + *(A REBEL)
  • 14ac: TRASHCANICULA: Destination for junk star guided by height the moon turned in about star (8,8): *(STAR) + H(eight) LUNICA (moon) reversed in CA (about)
  • 16ac: MODULATELIERS: Earliest stirring in workshops to adjust subject of painting for all to see shifting base? (8, 8): MODEL - E (base) + U (for all to see) *(EARLIEST)
  • 18ac: YESTEREFA: Forbidden meal interrupted by whistle-blowerlast, for sure (6, 6): YES (for sure) TEA (meal) outside REF (whistle-blower)
  • 23ac: TEAPLANTHEMIA: Commercial bush, petal trained by worker, reverse of ideal alongside border plant-forms in art (8, 8): *(PETAL) + ANT (worker) AI (A1, ideal) reversed after HEM (border)
  • 29ac: IVORIESTONIAN: Like e.g. Cameron once welcoming southern European national, perhaps warn one going after bit of ice creamier than the rest? (8, 8): I(ce) + VOR (perh. warn, Shakesp.) + I (one) ETONIAN (like David Cameron in school days) around S(outhern)
  • 30ac: SEMANTRABEATE: Bell equivalents; built of horizontal beams (8, 8)
  • 31ac: TRUDGINGULFED: Overwhelmed, nursing damaged lung, I took food, making slow progress with date in fruit basket (8, 8): D(ate) in TRUG (fruit basket) I FED (took food) around *(LUNG)
  • 1dn: OPTOMETRIST: Work on volume once causing grief for eye expert (11): OP (work) + TOME (volume) + TRIST (obs. grief)
  • 2dn: VERBOSE: Wordy book’s half in poetry (7): BO(ok) in VERSE (poetry)
  • 3dn: RISQUE: Queen captured in revolt? Danger past (6): QU(een) in RISE (revolt)
  • 4dn: SCHI: First of snow covering hills, I ski in Oberstdorf (4): S(now) C(overing) H(ills) + I
  • 5dn: HOCK: Wine jar, not the first (4): (s)HOCK (jar)
  • 6dn: TENDER: Painful? It’s often de rigueur in part (6) ofTEN DE Rigeur
  • 7dn: CRIBLE: Dotty translation (too literal?) given with the French, ignoring accent (6): CRIB (literal translation) + LE (fr. “the”); “ignoring accent” to deal with the acute in “criblé”
  • 8dn: HACK: Pick mercenary, but lacking old gun (4): HACKBUT (old gun) - BUT
  • 9dn: OLLA: 50% of TV comedy served up in miscellany? (4): the TV comedy is 'Allo 'Allo!
  • 10dn: TEARSTAINED: Wretchedly triste, Danaë may appear so? (11): *(TRISTE DANAE) &lit.
  • 12dn: RANDY: Tomboy exhibiting extremes of roguery (5): extremes of “roguery” are R AND Y
  • 13dn: BUTE: Evidence of horse-doping? Objection leading to ending of licence (4): BUT (objection) + (licenc)E
  • 15dn: ATTENDS: Marks & Spencer initially stocking alcohol-free wine, without success (7) AND (&) + S(penser) around TT (alcohol-free) + WINE - WIN (success)
  • 17dn: RUINATE: By implication, a true Shakespearean smash? (7): A TRUE = RU IN ATE
  • 19dn: SLUING: At home, in grip of strong drink, spinning uncontrollably (6): IN (at home) in SLUG (strong drink)
  • 20dn: TAPETI: Rabbit from part of SE Asia that’s indulged in (6): TAI (part of SE Asia) with PET (indulged) in
  • 21dn: FENNEL: Herb names one has instinct about (6): NN (names) with FEEL (instinct) around
  • 22dn: AMNIA: Fluid containers? It’s madness switching first couple (5): MANIA (madness) with first two letters swapped
  • 24dn: ATOM: Anything tiny, of matter principally (4): A(nything) T(iny) O(f) M(atter) &lit.
  • 25dn: AVER: Old Scots nag, mean, ageless (4): AVERAGE (mean) - AGE
  • 26dn: GRAD: Successful finalist, task over (4): DRAG (task) reversed
  • 27dn: STAG: Grass? It’s cut short among spinach (4): T (it, shortened) in SAG (spinach)
  • 28dn: ZOBU: Cross male, Unionist sketch writer sent up (4): U(nionist) + BOZ (sketch writer; ref. Boz) reversed
azed 2195

Delayed because of my time away at ECDA 2014; I worked on it on Saturday morning, doing most of it in one go. phew.

  • 1ac: GAMESTER: Old lecher getting me in the stomach (8): ME in GASTER (stomach)
  • 7ac: SOPH: Soggy bit of bread, hot – one was fresher earlier (4): SOP (soggy bit of bread) + H(ot); ref. Sophomore
  • 10ac: ARISTOLOCHIA: Tropical climbers: one is ariot growing wild round lake (12): A (one) + *(IS ARIOT) around LOCH (lake)
  • 11ac: MANG: Archaic slang talk confined to German glossaries (4): gerMAN Glossaries
  • 13ac: DIABLE: Cooking pot – one put in what’s fit to eat, removing lid (6): A (one) in (e)DIBLE
  • 14ac: PROFANE: Secular piece of glass on account of being inserted wrong way round (7): PANE (piece of glass) around FOR (on account of) reversed
  • 16ac: QUESTRIST: One who earnestly sought rest disturbed with pigeon around (9): *(REST) inside QUIST (pigeon)
  • 17ac: SHIKSE: Gentile girl he excited with kiss (6): *(HE KISS)
  • 19ac: IOTA: Tiny amount, or nothing, held in palm (4): O (nothing) in ITA (palm)
  • 21ac: HIDE: Leather shroud (4): Double definition
  • 23ac: GASSED: Was chattering fool catching fish? The opposite (6): ASS (fool) in GED (fish)
  • 24ac: STIGMARIA: Part of fossilized tree – it’s vaguely discernible on grand lunar levels (9): *(ITS) + G(rand) + MARIA (lunar plains)
  • 28ac: MANTEEL: Soldier with sword, its head concealed in his cloak (7): MAN (soldier) + (s)TEEL (sword)
  • 29ac: FAVELL: Crafty old horse a length behind favourite (only third) (6): FAVE + L(ength) + (on)L(y)
  • 30ac: ANDA: Classical pianist: you’ll find Verdi — may be varied (4): VERDI and A = *(VARIED); ref. Géza Anda
  • 31ac: OVARIOTOMIST: Gynaecological surgeon having nothing against out of control MO’s messing with it (12): O (nothing) + V (against) + ARIOT (out of control) + *(MOS IT); odd that “ariot” is used here and in 10ac
  • 32ac: SEWN: Overcast? Band of rain we see heading westwards (4): raiN WE See (reversed); nice surface
  • 33ac: CREPANCE: Training cord round front of pastern – does it make horse lame? (8): CREANCE (training cord) round P(astern)
  • 1dn: GAMPISH: Bulging crone turned up clutching wayward imps (7): HAG (crone) reversed around *(IMPS)
  • 2dn: MINOXIDIL: I cover up after love in hussy’s embrace – it offers new hope for baldies! (9): MINX (hussy) around O (love) + I + LID (cover) reversed; ref. Minoxidil
  • 3dn: STRAUSS: Former skipper trained tars on American ship (7): *(TARS) + USS (American ship); ref. Andrew Strauss
  • 4dn: TOWNEE: No country-dweller, wet one when it’s stormy (6): *(WET ONE)
  • 5dn: ELDEST: First-born’s share of Nuffield estate (6): nuffiELD ESTate
  • 6dn: ROIST: Carouse is getting stuck into Franconian red (5): IS in ROT (Ger. “red”)
  • 7dn: SCAW: Was looking up after getting round constant low point? (4): WAS reversed around C(onstant)
  • 8dn: PILASTERED: Showing certain columns I penned, after long lunch in El Vino’s? (10): I in PLASTERED (drunk)
  • 9dn: HAEM: Meat, round end of T-bone, the colour of blood (4): HAM (meat) around (t-bon)E
  • 12dn: ARCHITRAVE: Part of entablature with e.g. pictures put up in repository (10): ART (e.g. pictures) reversed in ARCHIVE (repository)
  • 15dn: DIOSGENIN: Extract of yams? Stirred dose in gin (9): *(DOSE IN GIN)
  • 18dn: MAINTOP: Keen on opening chart in crow’s-nest? (7): INTO (keen) in MAP (chart)
  • 20dn: ADULATE: Butter potato beginning to dry up, upset about it (7): ALU (potato; var. “aloo”) + D(ry) reveresed in ATE (upset)
  • 22dn: SAMLOR: Jack I exchanged for money for autorickshaw? (6): SAILOR (Jack) + M(oney) - I
  • 23dn: Adorned old jar with square inside (6): GRATE (jar) around S(quare)
  • 25dn: MALIC: What serpent shows cutting ultimate bit of sour apple? (5): MALIC(e)
  • 26dn: UFOS: What only the gullible believe in? Objectively we’ll admit owning up (4): US (objective case of “we”) around OF (owning) reversed
  • 27dn: KERN: Old peasant, one remembered for his tunes (4): Double Definition; ref. Jerome Kern
ecda days 2 and 3

Yeah, technical sessions starting at 08:30 in the morning... even if the majority of the delegates are staying onsite, that’s a little optimistic. It certainly was in my case; recovering from Wednesday’s early start was somewhat necessary. So, yes, I missed Thursday’s first plenary session on incremental optimization of performance measures, in favour of chocolate-festooned muesli (not convinced that I got the best of the deal, though) and joined the conference sessions at Alfred Inselberg’s presentation of parallel coordinates for exploration and visualization. As far as I can tell from the presentation, the principal benefit of parallel coordinates is providing a two-dimensional visualisation allowing for fairly straightforward interactive exploration of a dataset; although one particular parallel coordinate system imposes an ordering on the data dimensions, it doesn’t take too many permutations of the ordering (about half the overall dimensionality, by my count) to be able to visualise correlations between all pairs of dimensions. I’m not sure that Alfred’s exhortation to trust our eyes for pattern-finding is such a good one; maybe to an audience of this sophistication it doesn’t matter, but the balance between ease of finding patterns and ease of finding spurious patterns isn’t obvious. That said, the demo of the parallAX software was mildly nifty; not quite my cup of tea, but fine.

I then went to the session on data analysis in social sciences, largely because of the title of the first talk: effects of parenthood on well-being. The summary of the results of an investigation by Evgenia Samoilova with Colin Vance, trying to decouple various effects, might be: women show a rise on average in well-being when their adult children leave home; men, by contrast, show a fall at the same point (unless they had a particularly low well-being score in the first place); also on average, the strongest positive effect of children on well-being is exhibited by those who were already reporting a high well-being before becoming parents. The results should be taken with the usual pinches of salt, of course... the other talks in that session covered interventions to increase takeup of physical activity; a questionnaire evaluation of pilates treatment for urinary incontinence; student life-style; and, as a bonus talk, an introduction to using Latent Dirichlet Allocation for topic modelling, with an extension to include markov modelling between words (i.e. somewhat removing the “bag-of-words” document model).

I didn’t get very much out of Andreas Geyer-Schulz’s presentation on decision-makers without preferences; whether his approach really explains why the independence of irrelevant alternatives axiom is typically violated by human behaviour, I don’t know. (Really, this is a strike against the ECDA requiring and distributing only abstracts; the topic of the talk was interesting enough that I’d like to read the conference paper... but I can’t, because it doesn’t exist.) I next went to the “big data” session, and for me the best of the talks was definitely a small data one: Eyke Hüllermeier presenting Ammar Shaker’s work (minus his data, because of a cycling accident) on the distinction between epistemic and aleatoric uncertainty, and distinguishing between them in uncertainty sampling. I think my inner Bayesian would have represented this distinction using beta distributions rather than plausibilities (epistemic uncertainty being Beta(0,0) and aleatoric uncertainty for a coin flip being Beta(50,100) or so) but it was a nice presentation. Claus Weihs gave a talk on the relationships between big data analytics and “classical” data science, with some observations about the curse of dimensionality, and Dieter Joenssen talked about the dangers of naïve missing data imputation in the big data context.

And then it was time for the social activity: a brief tour of Bremen old town, a pleasant boat ride back to Bremen-Vegesack (topics of discussion on the boat: academic careers and publication metrics; the future of academic publishing; wind, solar and coal power; and other cheerful stuff. Over dinner we talked about Hong Kong, image analysis in marketing and branding, and I also talked a bit about my work, going on the basis that there might be only few people in the session where I was due to present formally on Friday, and so I wouldn’t be spoiling things for anyway.

And so it eventually proved. The data analysis in musicology session was not the most popular; who knows what everyone was doing instead? (Not tweeting) but I got a nice laugh from superposing SIFT features on a picture of a cute kitten, and the surprise payoffs in the image/music similarity analysis (spoiler: parody motif reuse, movable type reuse, title pages) got a reaction. The questions, justifiably, were about scaling; I don’t know if the audience bought the idea that it was mostly OK to do this book-by-book (and so not pay too many penalties for super-linear algorithms); honestly, I don’t know if I buy that myself, though it’s an easy thing to try...

The other talks in the session covered a comparison of metadata and audio features for genre classification (from Igor Vatolkin), where the result was, as usual, that even dodgy metadata from playlist co-occurrence appears to beat anything that we can do with audio. Tillman Weyde presented the state of the Digital Music Lab project, at this point describing the aims and infrastructure design, rather than any results (they hope to have some results as well as infrastructure before the end of the project). The last talk was about characterising musical scales; apparently there are thousands of scales in the Scala (no relation to the programming language) scale dataset, few of which have any metadata beyond a name. They tried looking at the relationship between the scale notes (expressed as a bitvector of 1200 entries and the name, finding pretty much nothing; I suggested that they model the scale as a (multi)set of intervals between notes in the scale rather than a set of intervals between notes and the root; I don’t expect that there will be any improved association between name and content, but maybe the cluster profiles will be more rich in the end.

For the final semi-plenary (an odd choice, to divide the audience in two at the last) I went to listen to Joaquin Vanschoren talk about his not-quite-launched OpenML project, which I think is an attempt to be myexperiment for data analysts (but, why not use myexperiment?). Certainly, the aims are good – to get people out of their silos; to encourage code and data reuse, and experiment reproducibility; to increase academic credit given for collecting data and writing programmes – but will it catch on? Who knows. It didn’t seem totally compelling to me, and I think the audience had reservations for other reasons – there were worries expressed about loss of control, and about the Open Source requirement for software. I think the question at the end (how many users does it have? 45? And not all of those are actually active?) was perhaps the most problematic, and just reflects that bootstrapping is hard.

Then it was really all over (my full, telegraphic notes here), and after a sandwich lunch I went back to Bremen city to see multiple instantiations of the town musicians, and to hear the glockenspiel. Then I did my usual thing of walking to the airport (not a particularly exciting or revealing walk, this time, though I did go through quite a nice park), and it’ll soon be time to submit to the multiple indignities involved in getting home. Will I go to ECDA next year? Well, on the plus side it’s fairly local (in Colchester), so it could be more practical; on the minus side, it does seem that music informatics is a minority sport, bolted on because someone on the committee is interested in it as a sideline. On the plus side, I found quite a lot of the vanilla data analysis pretty interesting anyway, even if I’m not actively working in the direct area, so I think it’s on the “possibly” list.

ecda day 1

I made it to Jacobs University Bremen, somewhat bleary-eyed thanks to the brutal 04:1504:30 start (Stansted airport at 06:00 reminded me why I hate air travel). I was travelling with my Transforming Musicology colleague-in-arms David Baker, both of us due to present small pieces of our work at the European Conference on Data Analysis. Mind you, I had the lucky draw; my presentation is scheduled for Friday, whereas David had to give his that afternoon.

We got here Wikimedia user Muck123 CC-BY-SA, and we’re definitely not in New Cross any more David Howard CC-BY. I haven’t actually been to a campus University for ages, and I don’t remember being on one as charming as this one: it’s small, there are lawns everywhere, the pond has a duck house... after some confusion in attempting to learn how to operate doors, I got to the first keynote presentation, by Themis Palpanas of Université Paris Decartes – which happened to be on indexing structures for exact and approximate data series query and retrieval, so the whole trip was probably already worthwhile. I have a long list of references to chase.

The first “semi-plenary” was also directly relevant to me, this time to the work that I am due to present; Rebecca Nugent talked about random forest approaches to aggregating the same individuals in human-maintained records, for example in patent databases. The main point I took away from her talk was that the internal states of the trees in the random forests (and in the forests themselves if using forests of random forests) effectively induce a distribution over probabilities, and those distributions can be used to distinguish between certain and uncertain probability predictions. I found it odd that she and her colleagues at the NSF Census Research “node” had chosen skew as the distributional measure to evaluate, rather than some measure of spread – I probably missed something there. The twist at the two-thirds point, to consider casualty lists in Syria using the same methodology, was very effective: though the topic is of course gloomy, attempting to validate the various claims and counterclaims of the various sources is closer to my heart than identifying unique patent authors or making the US census a bit cheaper.

The afternoon brought on the parallel sessions – lots of them: seven sessions ranging from machine learning and knowledge discovery to data analysis in marketing, though where data analysis in musicology fits on that implied continuum I don't know. In any case, I went to support David, regretfully therefore not being able to attend the session on “high-dimensional data: using all dimensions”. I thought David’s talk went very well; the musical examples (and audience participation) kept everyone awake, and the questions from the small but engaged audience showed that they had definitely understood what we were up to. The other talks in the session were interesting too: Nadja Bauer presented a method for model-selection in the parameter space of onset detection methods. Denis Amelynck showed work on classifying people’s movement to Brahms’ Piano Concerto No. 1 – the clustering method based on Dirichlet Process Models was fine, but the surprising observation was that even something as crude as summing the “directogram”s of 35 participants’ movements (a similarity matrix where the feature was which of 32 directions the dominant hand moved in each frame) reproduced, by eye at least, the coarse similarity matrix of the music.

Finally in the session on Musicology, Tillman Weyde presented the somewhat surprising result that in attempting to separate solo piano music from other classes in a 600-strong database of recording, the audio was no help given even extremely poor metadata. This result is similar to our result about chroma distance being only marginally associated with leitmotif recognition difficulty, and I think the conclusion is the same in each case: the audio feature just isn’t good enough. (In our case, there’s no particular surprise: a 12-dimensional chroma barely captures any melodic information, while leitmotifs are substantially melodic objects; in Tillman’s case, I suspect the issue is that the instrument identification, done using Emmanouil Benetos’ transcription method, depends on instrument sound templates, which might match poorly to the actual historical recordings – as well as the fact that even the very poor metadata is good enough to separate out most of the piano solo music in this database).

The last event of the first day was a slightly chaotic panel discussion on the future of publications in classification and data sciences, which was definitely oriented from the point of view of the German Classification Society – and in particular how the conference (post-)proceedings should be published. The current state is as a Springer volume; the discussion centered around whether there would be benefit to the society, and its members, to switching to an Open Access journal working with a “less prestigious” publisher.

This is a difficult question, in the general context of academic publishing which at the moment I would characterize as “a total mess”. Some of the contradictions were explicitly stated: early career researchers are often made to “publish [in ‘high-quality’ journals] or perish”; UK funding bodies now insist that publications should be Open Access (“green” or “gold”); Open Access currently requires either funding to pay journal fees, or an embargo period and an institutional repository. (I am simplifying; it’s more complicated than this). The feeling in the room, it seemed to me, was that there was broad acceptance of an Open Access venue for the conference proceedings, provided that there were no author charges (other than, I guess, the need to attend the conference in the first place). There was also a tension between the needs of the Classification Society (a plausible venue for its proceedings) and the needs of the research community as a whole (do we really need another Open Access journal in Data Science?). But I left unsatisfied because of more than that; I was a bit depressed that there seemed to be a tacit acceptance that good work goes in “high-quality” journals, or even worse that the impersonal academic system judges that a work published in a “high-quality” journal is good – a mistake that no Data Scientist would make in their domain of expertise.

After that, it was time to explore the local area. After a comedy period of striking out in completely the wrong direction, caused largely by the cunning trick of the University distributing a campus map with South pointing up the page; I noticed our directional confusion eventually by the lack of river, and corrected it by navigating by the sun and my memory of the map of the area. Eventually, we got to Bremen-Vegesack and had a fine beer and fish supper, and got back to the campus past an improbable number of impeccably-kept front gardens without getting substantially more lost.

azed 2194
  • 1ac: COBBY: Local’s lively, and evasive about books (5): COY around BB (books)
  • 5ac: MAMMOCK: Mother has to ridicule old-fashioned mangle (7): MAM (mother) + MOCK (ridicule)
  • 10ac: OLEAGINOUS: Like a toady, using love deviously behind e.g. privets (10): OLEA (privet genus) + *(USING O (love))
  • 12ac: CLEGS: Clubs introducing entertainment, not how they’ll sting you! (5): C(lubs) + LEG-SHOW (entertainment) - HOW
  • 13ac: DRAIN: What’ll fill it after start of downpour? (5): D(ownpour) + RAIN (what’ll fill drain) &lit.
  • 14ac: PROCINCT: Nothing replaces energy in US copshop, being prepared as of old (8): PRECINCT - E(nergy) + O (nothing)
  • 15ac: SEELY: Good old foxy gobbling bits of eviscerated eirack (5): SLY (foxy) around E(viscerated) E(irack)
  • 16ac: HAKAM: A king embraced by patriarch’s son, a Hebrew sage (5): HAM (patriarch’s son; ref. [Ham]) around A K(ing)
  • 18ac: HARD AT IT: Art I’d worked on in e.g. Leghorn, very busy (8, 3 words): *(ART ID) in HAT (e.g. Leghorn)
  • 22ac: COLLEGER: Pass Latin, English and German as assisted pupil at Eton (8): COL (pass) + L(atin) + E(nglish) + GER(man)
  • 24ac: TRUSS: Selection of citrus slices to skewer for cooking, say (5): ciTRUS Slices
  • 25ac: KVASS: What’ll get Slovaks drunk – this, with none left? (5): comp. anag. KVASS + O (none) + L(eft) = *(SLOVAKS) &lit.
  • 28ac: STICKIES: Adhesive labels to mark off, i.e. when put in steamer (8): TICK (mark off) + IE in SS (steamship)
  • 29ac: LARUS: Seabirds gone astray in Urals (5): *(URALS)
  • 30ac: RIGEL: Shackle that’s released on backing for big star (5): LEG-IRON (shackle) - ON reversed; ref. Rigel
  • 31ac: TRIATHLETE: Versatile competitor, worn-out, ’ealth broken internally (10): TRITE (worn-out) around *(EALTH)
  • 32ac: MICELLE: Group of chains one found in front of dungeon in Middle East (7): I (one) + CELL (dungeon) in M(iddle) E(ast)
  • 33ac: KASHA: Porridge as served in nameless eastern inn (5): AS in KHAN (Eastern inn) - N

  • 1dn: COCKSHOT: Aunt Sally with a high temperature after piles (8): COCKS (piles) + HOT (high temperature)

  • 2dn: OLLIE: Winter sport manoeuvre? Flipped badly in what involved gambling (5): ILL (badly) in EO (old gambling game) reversed
  • 3dn: BEEZER: Piece of cake, second to last? A cracker for Jock (6): BREEZE (piece of cake) with R (second letter) moved to last place
  • 4dn: BAGEL: Roll keeping a long time in barrel (5): AGE (long time) in BL (barrel)
  • 5dn: MICROTOMICAL: What’s funny about joke after I’m upended preparing for a slide? (12): MI (I’m upended) + COMICAL (what’s funny) around ROT (joke)
  • 6dn: ANDOUILLETTE: Sausage, ‘netted’, all in bits? Certainly French will tuck into that (12): *(NETTED ALL) around OUI (fr. “certainly”)
  • 7dn: MURIATE: Without speaking about it, look up ‘archaic chlorine compound’ (7): MUTE (without speaking) around AIR (look) reversed
  • 8dn: CHICANE: Church accepts this Roman article as a bit of trickery (7): CE (Church) around HIC (lat. “this”) + AN (article)
  • 9dn: KANT: Turn on edge one knight captured by another (4): A (one) N (knight) in KT (another [knight])
  • 11dn: SANKO: Guitar went low – nothing lower (5): SANK (went low) + O (nothing)
  • 17dn: MARSILEA: Water ferns, strain that SA rodent devours (8): SILE (strain) in MARA (SA rodent)
  • 19dn: AIR TAXI: Nine main apartments put up offering chopper for hire? (7, 2 words): IX (nine) ATRIA (main apartments) reversed
  • 20dn: DISCURE: Cruised at sea to learn old-style? (7): *(CRUISED)
  • 21dn: QUIRT: Whip Bunter’s rear, cutting leave (5): (bunte)R in QUIT (leave)
  • 23dn: GANGES: River crews rounding end of course (6): GANGS (crews) around (cours)E
  • 26dn: VOILA: Palm-fringed island, look (5): VOLA (palm) around I(sland)
  • 27dn: SIETH: In time it’s the end of the noose for this old bushwhacker? (5)
  • 28dn: SLIM: Rarely crafty general reduced line in feigned situation (4): L(ine) in SIM (feigned situation); ref. William “Bill” Slim
azed 2193
  • 1ac: ZING: Nodding off, with party lacking vitality (4): DOZING (nodding off) - DO (party)
  • 7ac: SPALD: Shoulder of e.g. lamb, left in, well done? One must be left out (5): SPA (well) + DONE around L(eft) - ONE
  • 12ac: ABORAL: Wind covered by centre of palm away from the mouth (6) BORA (wind) in (p)AL(m)
  • 13ac: ATOCIA: A coati operated on for sterility (6): *(A COAT I)
  • 14ac: KITEMARK: Team working in church, guarantee of quality? (8): *(TEAM) in KIRK (church); ref. Kitemark
  • 15ac: ABCEE: Rudimentary knowledge about taken in by a self-help group? (5): C (circa, about) in A BEE (self-help group)
  • 16ac: MARID: Very strong spirit I included in tot knocked back (5): I in DRAM (tot) reversed
  • 18ac: TICKING: Reprimand when one’s cut off strong cloth (7): TICKING-OFF (reprimand) - OFF
  • 19ac: PORTERS’ LODGE: Bagging degree, poor Gretel’s delirious - students often pass this (12, 2 words, apostrophe): D(egree) in *(POOR GRETELS)
  • 21ac: CONGRATULATE: Salute by means of dance bidding farewell to a local chief, deceased (12): CONGA (dance) - A + RATU (Fijian chief) + LATE (deceased)
  • 24ac: IGARAPE: One sort of pike to take off waterway for canoeists (7): I (one) + GAR (pike) + APE (take off)
  • 28ac: RIMED: I am clad in scarlet, coated as in winter? (5): I’M (I am) in RED (scarlet)
  • 29ac: AORTA: Major vessel is removed from tense area (5): AORIST (Greek tense) - IS + A(rea)
  • 30ac: HEIRLOOM: Ruined home spanning Ireland (old), part of estate? (8): *(HOME) around IRL (Ireland) + O(old)
  • 31ac: ZANIER: Cuckoo circled by duck? Not duck – more like a cuckoo (6): ANI (cuckoo) inside ZERO (duck) - O (duck)
  • 32ac: ANANKE: A nude almost clutching her head in need (6): A + NAKE(d) (nude) around N (head of “nude”)
  • 33ac: ALTAR: Brig wrecked off rock table? (5): GIBRALTAR (rock) - *(BRIG)
  • 34ac: GAIN: Old boot one caught in trap (4): A (one) in GIN (trap)
  • 1dn: ZAKAT: Charity tax, zero on American tea leaves (5): Z(ero) A(merican) KAT (tea leaves)
  • 2dn: IBIBIO: Some Nigerians – in the same place others will accept one (6): IB (abbr. ibidem, in the same place) + IBO (other Nigerian people; ref. Ibo) around I (one); ref. Ibibio
  • 3dn: GREEK: Take in odd misfit, a cunning old rogue (5): R (take) in GEEK (odd misfit)
  • 4dn: LAMETER: Person disabled formerly, one learning a measure (7): L(earner) + A + METER (measure)
  • 5dn: CLAPTRAPPERY: Applaud attempt to restrict paper that’s broadcast bunkum (12): CLAP (applaud) + TRY (attempt) around *(PAPER)
  • 6dn: BARRISTERIAL: Air Sir Albert flourished, showing skill in court? (12): *(AIR SIR ALBERT)
  • 8dn: POBAK: Worshipper in e.g. Sofia mosque, climbing old crooked work (5): KAB (obs. crooked) + OP (work) reversed
  • 9dn: ACARIDAN: It’s maybe a mite dry inside a chimney pot (8): ARID (dry) in A CAN (chimney pot)
  • 10dn: LINING: Piece of muslin in green furnishing cushion? (6): musLIN IN Green
  • 11dn: DAUD: Large piece of Scotland, area in grip of failure (4): A in DUD (failure)
  • 17dn: ORNAMENT: King given title, not upset about that bit of added knick-knackery (8): R (king) + NAME (title) in *(NOT)
  • 20dn: LUCARNE: Church window unclear, being refurbished (7): *(UNCLEAR)
  • 22dn: OGIVAL: Can I proceed heavenward, like a rocket’s cone? (6): LAV (can) + I + GO (proceed) reversed
  • 23dn: TITOKI: Tree with reddish flowers it’s all right for monkey to devour (6): OK (all right) in TITI (monkey)
  • 25dn: REDIA: Some explore diagrams showing form in life cycle of parasites (5): exploRE DIAgrams
  • 26dn: ROLAG: Wool for spinning from cloth scrap? Lubricant I dropped in (5): OIL (lubricant) - I in RAG (cloth scrap)
  • 27dn: YAMEN: Whence mandarin rules, forever elevated above people (5): AY (forever) reversed on MEN (people)
  • 28dn: RIZA: American rose atop a ‘window’ allowing view of icon (4): RIZ (US dial. “rose”) + A
thame chamber choir

I’m on a train! Not the one that I wanted; that one was cancelled. But a train, nevertheless, and one which is going (slowly) in approximately the right direction. “Change at Reading”, they say. I was in Oxford for the afternoon – just, after my outbound train was also cancelled – to sing with the Thame Chamber Choir. They broke the metaphorical glass behind which the emergency alto was kept, and possibly the effect of singing in Spem a fortnight ago was to raise my singer’s social media profile; hmm, Spem in alium as the equivalent of LinkedIn...

I was made welcome by the choir, and as a bonus I learnt some new music: Crux fidelis by Jean Roger-Ducasse; A prayer of King Henry VI by Gabriel Jackson; and Salve Regina H.24 by Marc-Antoine Charpentier. The Charpentier is interesting. It's for two equal choirs and a smaller group, and the section in the text about “moaning and weeping in this valley of tears” (gementes et flentes in hac lacrimarum valle) is set for the solo group to an astonishing sequence of diminished and augmented chords: word painting, for sure, but more extreme than I have grown used to expect in this repertoire. It put me in mind of the second trio des Parques from Rameau’s Hippolyte et Aricie, but as well as being an odd sequence, each chord is individually strange. Really worth a listen.

And now I have to work out how to get back home on this antiquated transport system of ours. Assuming this goes better than the last substantially delayed journey, it will have been worth it.

[ update: just. ]

londonr june

I went to the londonR meetup, at Balls Brothers near Fenchurch Street. Partly this was to get out and about at geeky meetups, and partly this was to do a little bit of background activity for the Goldsmiths Data Science MSc. My first observation is that, compared to my more usual Lisp or Emacs meetups, this was pretty well-populated: there were maybe 100 (one hundred!) people attending the 3 (three!) presentations. Maybe some of that was the free bar effect, but since it wasn’t advertised as such, maybe not. The event in general was organized by Mango Solutions, but the particular instantiation was sponsored by Tibco. So, thanks!

Perhaps poetic justice for my slightly impure motives for attending was being pushed, fairly gently, the EARL event in September, which in fact looks interesting: I feel like I should make an effort to attend, even given the fairly steep registration fee. I’m less interested in attending the R in insurance conference, because it’s a bit too specific (though also substantially less expensive...)

I didn’t make the ggplot workshop – too much to do during the day – so I arrived just after the doors opened for the main event. There was some time for chatting before the presentations started; I got to talk to some of the Mango Solutions representatives, including a project manager hoping to apply data analytics to his own job. Who knows how that will go. But soon enough it was time to sit down (to make sure that I had a seat, the latest newspaper scare story notwithstanding) and try to follow and take notes on the presentations.

The first presentation, by Chris Campbell of Mango Solutions, talked about the work they were doing on validating R. The point here was to be able to satisfy regulatory requirements on software, in order to be able to use R in industries with heavy regulation (pharmaceuticals being perhaps the most obvious). The presentation didn’t go into detail – 30 minutes is not long, which is perhaps encouraging for my forthcoming 20-minute presentation in Bremen – but gave an overview of the basic problem, and flagged up some packages scheduled to be released at EARL in September: functionMap, for visualizing call graphs; testCoverage for code coverage reports (like sb-cover for SBCL fans); and visualTest for testing for image similarity, given that the same R code on different operating systems produces not-quite-identical output. There wasn't really time to go into the detail of the image similarity algorithm, but the slide saying “Fourier Transform” made me worry a bit; it was picked up in the question section, and Chris motivated their current approach as being a sufficiently-sized hammer for their purposes – they’d experimented with wavelets to localize differences, but found it was unnecessary. Still, I’d wonder whether a SIFT-based approach would be potentially useful, and it’ll be interesting to see whether the similarity threshold in arbitrary units actually works...

The second presentation was given by Ana Costa e Silva of Tibco. She introduced the Spotfire suite of products, addressing all elements of the “tower of data” (from KPIs to real-time analytics), and demonstrated one in particular, the “Tibco Enterprise Runtime for R”, being a reimplementation of R with a different (might one say “saner”?) runtime engine; they offer a “community edition”, but it’s behind a sign-up form so I have not looked at it. The demonstrations focussed on the ability of developers to build user interfaces so that users (business decision-makers) could build their own models to answer their own questions; this makes data scientists happy. The question session, short but focussed, probed the real level of compatibility between TERR and R, and the company representatives did own up to the imperfections of package support: mostly residing in Rcpp’s use of undocumented internals. I suppose that’s reasonable; while R’s semantics do in theory allow for the redefinition of the { function (yes, really), I doubt any published package of importance actually uses that.

The third presentation was probably the most entertaining; Simon Hailstone from the Royal Free NHS Foundation talked about ggplot and map-making, using as the case study building interesting maps relating the locations of hospital A&E departments, deprivation indexes, and various categories of ambulance service incidents. For this audience, the point was not the inferences that one could draw, but rather the tips for making good-quality output with a reasonable amount of time and effort, but Simon also covered a variety of data sources, how to manage shape files at different scales, tips for better-quality graphical output and geocoding.

And then there was the networking; I did manage to speak to a few people, offering up in advance the enthusiastic but untrained labour of project students; we'll see what that brings. I had an enjoyable time, and I’ll try to make the next event in November.

azed 2192

Apart from the clue numbering being off in the printed version of this, it was not too bad; the two triple definitions stumped me for a while – just as I have more or less got the hang of the comp. anag. clues...

  • 2ac: CACKLE: Laugh raucously, gold having been lost from primitive boat carrying king (6): CORACLE (primitive boat) - OR (gold) + K(ing)
  • 8ac: KHAYA: Jackdaw with dry grass nested in African tree (5): HAY (dry grass) inside KA (African tree)
  • 13ac: SIROC: Wind becoming moister and giving ten baths is from the east (5): COR (ten baths) + IS reversed
  • 14ac: HAYMOW: Rick demonstrating way Hom cooked (6): *(WAY HOM)
  • 15ac: T-CARTS: Pros clubs installed? They’ll take you for a ride (6): TARTS (pros) around C(lubs)
  • 16ac: PREP: Homework to bring forward when one’s knocked off (4): PREPONE (bring forward) - ONE
  • 17ac: ATHETOSIS: Advanced, this involves twisting of toes (9): A(dvanced) + THIS around *(TOES) &lit.
  • 18ac: TRANSIENTNESS: Time progressed – copper turned in imps – it shows nothing lasts (13): T(ime) + RAN (progressed) + SEN (copper) reversed in SIENTS (imps)
  • 21ac: KOBAN: Waterbuck on an old gold coin (5): KOB (waterbuck) + AN
  • 23ac: SIDHA: A dish, shimmering, one that’s achieved perfection (5): *(A DISH)
  • 25ac: ITALICISATION: Romans exemplify this? Wrong – it is Catilina Cicero ultimately trounced (13): *(IT IS CATILINA (cicer)O)
  • 26ac: VEDUTISTA: One paints broad townscapes, representing vastitude (9): *(VASTITUDE)
  • 30ac: CAST: Seek rejected players (4): Triple definition
  • 31ac: EROTIC: Steamy boiled rice, ’eated within (6): *(RICE) around (h)OT (’eated)
  • 32ac: BESIDE: Au reste, composition of Delibes abandoned by learner (6): *(DELIBES) - L(earner)
  • 33ac: QUELL: Slaying in Shakespeare, as that in Gallic lines (5): QUE (fr. “that”) + LL (lines)
  • 34ac: SASSE: Old lock, rusty at its centre, sea’s ruined (5): (ru)S(ty) + *(SEAS)
  • 35ac: CHESTY: Being well-endowed, i.e steeped in riches typically? (6): riCHES TYpically
  • 1dn: ASTATKI: Petroleum by-product, shoddy stuff fed to a runner (7): TAT (shoddy stuff) inside A SKI (runner)
  • 2dn: CICERO: English ‘croci’ (faulty form), type with 13 points? (6): *(E(nglish) CROCI); ref., I think, to Cicero the unit of type, rather than to Marcus Tullius, even though he is just the type to have “and thirteenthly” in one of his speeches...
  • 3dn: ARAB: Good-looking mount – what’s unclear about framing? (4): uncleAR ABout
  • 4dn: CORONA LUCIS: Colours I can get twinkling in church chandelier (11, 2 words): *(COLOURS I CAN)
  • 5dn: LIST: Choice lean fillet (4): Triple Definition
  • 6dn: EUPHEMISTIC: Knocked chemist up, i.e. as ‘tired and emotional’? (11): nice ref. to one of Private Eye’s stock phrases
  • 7dn: THRENOS: Old-style lamentation boozer raised about short time with measure (7): SOT (boozed) reversed around HR (abbr. for “hour”) + EN (measure)
  • 9dn: HYPONITROUS: This acid can oxidize – about time your siphon is replaced (11): *(YOUR SIPHON) around T(ime)
  • 10dn: AMUSED: Employment limited by a CEO was diverting (6): USE (employment) in A MD (CEO)
  • 11dn: YOGI: One practising physical discipline twice likewise curtailed that drink (4): YO(n) GI(n) (that drink), both words curtailed the same way
  • 12dn: AWLS: Prods, maybe? How all working long shifts get started (4): A(ll) W(orking) L(ong) S(hifts)
  • 19dn: INCISED: Groovy disc i.e. incorporating new hit? (7): *(DISC IE) around N(ew)
  • 20dn: SANICLE: Woodland plant cut endlessly when in demand (7) NIC(k) (cut, endlessly) in SALE (demand)
  • 22dn: BADASS: Aggressive Yank, knife cutting deep (6): DA (knife) in BASS (deep)
  • 24dn: HOMILY: Around first year of millennium old saint lies prone? (6): HOLY (obs. “saint”) around M1 (first year of millennium)
  • 26dn: VIBS: Hard-wearing shoes, black, strength all round (4): B(lack) inside VIS (strength)
  • 27dn: ELEA: Eastern field where philosophers gathered (4): E(astern) + LEA (field)
  • 28dn: TECH: Wherein you may see some inculcate chemistry? (4): inculcaTE CHemistry &lit.
  • 29dn: STET: Son’s taken off hat – don’t remove (4): STETSON (hat) - SON
azed 2191

Gah! A busy weekend with Excoll, singing Spem in alium among other things (including Biebl's Ave Maria, totally new to me and very effective) means that I forgot to blog the crossword – and, indeed, an incautious software upgrade before that meant that I lost my place in the online Java applet provided by the Observer. (Thanks, Chrome, for assuring my security with your new PPAPI plugin architecture, but why does that mean I cannot have the shiny?)

Ahem. In any case, I did do Azed 2191; here it is. (Did I have time to think about a clue for the competition word? Did I heck.)

  • 1ac: SPLATTERPUNK: Violent novels portraying charger in spirit (12): PLATTER (charger) in SPUNK (spirit)
  • 10ac: CLARAIN: Streaks in coal? Could be such sets, OK, when mined (7): *(STREAKS IN COAL) = CLARAIN SETS OK &lit.
  • 11ac: GREY: Pale, like a gannet that’s deprived of half feed (4): GREEDY (like a gannet) - (fe)ED
  • 12ac: RURU: Aussie bird? A hearty sport, twice (4): R(ugby) U(nion) twice
  • 13ac: CHIDER: Edrich batting, a combative fellow of old (6): *(EDRICH)
  • 15ac: NAIL GUN: Less hard work than a hammer? See chippy align one (7, 2 words): *(ALIGN) + UN (one)
  • 17ac: PLEB: Commoner section of people believe (giving his word contentiously!) (4): peoPLE Believe; ref. Plebgate
  • 18ac: SENDABLE: Object in fur that can be shipped (8): END (object) in SABLE (fur)
  • 21ac: STANDEE: Not seated? That may be disturbing to one (7): *(NOT SEATED) = TO STANDEE &lit.
  • 23ac: DECEASE: Die, peace replacing speech as part of honouring (7): DECORATION (honouring) - ORATION (speech) + EASE (peace)
  • 25ac: ORGANONS: Investigation methods soon entering tangled gorse mostly (8): ANON (soon) inside *(GORS(e))
  • 26ac: INCE: Footballer from number kept in reserve (4): N(umber) in ICE (reserve)
  • 28ac: SALIENT: Hush when foreigner comes in, striking (7): ALIEN (foreigner) in ST (hush)
  • 31ac: TURTLE: Soup? Half of what it’s in, left simmering, female ignored (6): TUR(een) (half of what soup is in) + *(LE(f)T)
  • 32ac: GRIS: Steps formerly displaying charm if half cut? Exactly (4): GRIS(gris) (charm)
  • 33ac: OTIC: Like some drums? See one beside backing (4) C (see) I (one) TO (beside) all reversed
  • 34ac: MISTICO: Small ship, obscure in a way before one joins company (7): MIST (obscure) + I (one) + CO(mpany)
  • 35ac: REMAINDER-MAN: I’ll acquire interest in estate ditch – general fell in? (12): REAN (ditch) around MAIN (general) + DERM (fell)
  • 1dn: SCRIPT DOCTOR: Film editor cited poor crits hammered, i.e. rejected (12, 2 words): *(CITED POOR CRITS) - IE
  • *2dn: PLUG: Stopper (4)
  • 3dn: LARNEY: Wealthy Springbok (not Habana) nearly ruined (6): *(NEARLY)
  • 4dn: TALISMAN: Amulet in metal is man-made (8): meTAL IS MAN-made
  • 5dn: TIPLESS: Lacking bit at the end, like a disgruntled waiter? (7): Double Definition
  • 6dn: RAHU: What gets Hindu trembling with fear – this fiend? (4): *(HINDU FEAR) = RAHU FIEND
  • 7dn: URDY: Pointed bean appearing before end of July (4): URD (bean) + (jul)Y
  • 8dn: NEEDLES: Are they pointless? Not quite, or not at all (7): NEEDLESS - S &lit.
  • 9dn: KYRIE ELEISON: Crowning garlands worn irk eye badly in part of mass (12, 2 words): LEIS (garlands) ON (worn) under (crowned by) *(IRK EYE)
  • 14dn: IN-AND-IN: Endlessly senseless hullabaloo ensuring good breeding? (7) INAN(e) (endlessly senseless) + DIN (hullabaloo)
  • 16dn: ABREAST: Rule circumvented by a copper keeping in line (7): R(ule) inside A BEAST (copper)
  • 19dn: DANSEUSE: Ballerina? That’s her in English habit (8): DANS (her in, assuming her [= DANSEUSE] is French) + E(nglish) + USE (habit)
  • 20dn: VERSUTE: Crafty poems about ‘do’ (as it was called) (7): VERSE (poems) around UT (do, as in Solmization)
  • 22dn: TENIOID: Like a US ferret I catch, brought up on fish (7): I NET (catch) reversed + O (on) + ID (fish)
  • 24dn: UNTRIM: I turn out with maiden free of ornamental extras (6): *(I TURN) + M(aiden)
  • 27dn: GRIM: Make a face when losing top card, depressing (4): GRIMACE (make a face) - ACE (top card)
  • 29dn: ALDI: Face having to exchange portions in supermarket (4): DIAL (face) with halves swapped
  • 30dn: MICA: Rock-forming mineral, not for surfacing material (4): FORMICA (surfacing material) - FOR
azed 2190
  • 1ac: BEATUP: A ram must —, much the worse for wear (6): A ram must “be a tup”
  • 6ac: SISALS: Rope-makers having lives restricted by endless cut back (6): IS (lives) in SLAS(h) (endless cut) reversed
  • 11ac: ENDURO: Long-distance race – messed up Round One, on dropping out (6): *(ROUND ONE - ON)
  • 12ac: OUTLIE: Dark blue tile, cracked, to remain in the open? Rarely (6): OU (for Oxford University whose sporting colour is dark blue) + *(TILE)
  • 13ac: PARRYING: Railway enters cutting, turning aside (8): RY (railway) in PARING (cutting)
  • 14ac: ASTRAGAL: Moulding of a star with investment of a grand (8): ASTRAL (of a star) outside A G(rand)
  • 15ac: SMILE: I’ll cut trees back to make a beam (5): I in ELMS (trees) reversed
  • 16ac: FINCH: E.g. Peter, one of many demonstrating charm? (5): Double Definition; ref. Peter Finch
  • 18ac: TUTELAGE: Charge made by leader of Trinovantes as Roman eagle wavering (8): T(rinovantes) + UT (Latin for “as”) + *(EAGLE)
  • 21ac: TETCHIER: Bite inside layer that’s increasingly peppery (8): ETCH (bite) in TIER (layer)
  • 24ac: RAKEE: Inclination around start of evening for something like ouzo (5): RAKE (inclination) around E(vening)
  • 26ac: STERN: Hard old gang leader causing a fair bit of consternation (5): conSTERNation + Double Definition
  • 28ac: KNACKERY: King’s English penned by cranky, doddery old hack ends up here? (8): K(ing’s) E(nglish) inside *(CRANKY)
  • 29ac: SNUB NOSE: After backward rolls one’s damaged part of pug’s face? (8, 2 words): BUNS (rolls) reversed + *(ONES)
  • 30ac: SAPEGO: Former skin disease to drain self-confidence (6): SAP (drain) + EGO (self-confidence)
  • 31ac: SHELTA: Travellers use this – such tips are given by spiel that’s rendered deviously (6): SHELTA TIPS = *(SPIEL THATS)
  • 32ac: SHADOW: Trouble experienced with slut around? (6): HAD (experienced) inside SOW (slut)
  • 33ac: HAREEM: You’ll find wives congregating here are absorbed in a bit of sewing (6): ARE in HEM (bit of sewing)
  • 1dn: BEPESTER: Annoy greatly, as tiresome fellow interrupting pub drink? (8): PEST (tiresome fellow) inside BEER (pub drink)
  • 2dn: A DROITE: Keep this on the autoroutes? Roadie’s worried about it (7, 2 words): *(ROADIE) around ’T (it)
  • 3dn: TURTLENECKED: Tenter lucked out, reaching top of crag? (12): *(TENTER LUCKED)
  • 4dn: POISHA: Some Bangladeshi cash, I, in old money one quarter of an anna (6): I in POSH (old money) + A(nna) (one quarter of an anna)
  • 5dn: SONTAG: Knitted cape (from Sweden on label) (6): S(weden) + ON + TAG (label)
  • 6dn: STRAIGHTENER: Bung the last of veg, cooked, in sieve (12): *(THE + (ve)G) inside STRAINER (sieve)
  • 7dn: ALIGN: Ditzy gal with ‘in’ dress (5): *(GAL + IN)
  • 8dn: LINACRE: College: learner having entered one must accept rule (7): L(earner) + IN (having entered) + ACE (one) around R(ule)
  • 9dn: SEEL: Colander remains upside down (4): LEES (remains) reversed
  • 11dn: NAAM: Seizure of a kind starts on nerves and activates muscles (4): N(erves) A(nd) A(ctivates) M(uscles)
  • 17dn: HORNBEAM: Hardwood tree, note, coated with old frostiness in the morning (8): NB (Nota Bene, note) inside HORE (obs. frostiness) + AM (in the morning)
  • 19dn: UM AND AH: Speak hesitantly, as genial pop, first to last (7, 3 words): HUMAN (genial) + DA (pop) with H (first) moved to the end
  • 20dn: FIESOLE: English well in line for Tuscan location (7): E(nglish) + SO (well) in FILE (line)
  • 22dn: ERENOW: Partners at the table holding dab up earlier (6): E + W (bridge partners) around ONER (dab) reversed
  • 23dn: THRUSH: Mavis, displaying fungal disease, hurts badly – hospital follows (6): *(HURTS) + H(ospital)
  • 25dn: KALPA: A pal’s shattered after end of work – a very long day! (5): *(A PAL) after (wor)K
  • 27dn: RAST: Cur is cut with this, once scratched (4): cur is cut with R as T
  • 28dn: KISS: Bites of — etc melting could show softest bickies (4): *(BITES OF KISS ETC) = SOFTEST BICKIES
off to tours

I’m on a train. And, more than usually, I don’t know what I’m doing.

Because of my involvement with Transforming Musicology, I’ve been invited to participate in a seminar on Renaissance musicat the Centre d'Études Supérieurs de la Renaissance at Tours. This is a real privilege, and the guest list is a bit intimidating – I went through a French Baroque phase 20 years ago, and it’s therefore nice to be able to bring the book on Marc-Antoine Charpentier to get it autographed by the author.

But if the guest list is intimidating, that’s nothing compared to the format. It has been made very clear that this is not a conference – we don’t have to present research we've already done, but rather our plans for future research that might be of interest to the assembled community. And it will be really interesting for me to hear that, of course, but I will of course be called on to justify my own presence there.

I think I might have two stories to tell. The first is about tools for directly improving the musicologist’s life, with the exemplar I would use being the Gombert/Josquin case study that, depressingly, remains aspirational rather than actual. Here, although more of the pieces are in place than previously, and I do have a complete workflow for examining individual fragments, the thing that is holding up the investigation is that aggregation of search results for those individual fragments into a useful summary suitable for digestion is more challenging and involves more time for concentrated thought than I currently have available.

The second story involves Bernardino de Ribera, and would aim to tell the musicological story from as near as possible to the start – Bruno Turner’s discovery of manuscripts at Toledo and Valencia – to finish, a professional-quality recording. The hope here is that by making the intermediate “data” available (electronic versions of manuscripts, critical editions, recording sessions) as well as links to the commercial performing editions and recording release, we enable a wider community of researchers and enthusiasts to engage with the material, enriching the commons. I should make clear that this story, too, is aspirational; I do not yet have clearance for releasing /any/ of this, and there are many hurdles to overcome before that clearance is given – but here this is intended as a case study of how musicologists and associated parties can publish their work and make them accessible to allow more people to engage with their work. This might address the fears of some that archive-based musicology is a dying art, by making the fruits of the labour much more visible.

The common theme underlying these two stories is, for me, the validation of new knowledge. I am not one who has completely bought in to the “impact” agenda currently driving a fair amount of the evaluation of academic utility in the UK, but I do perhaps take a harder line about the creation of knowledge than some of my colleagues at Goldsmiths, where I strongly feel that new knowledge itself should be communicable to others and not just used in the context of artistic practice. (I should say that my own line has softened quite substantially in the last decade; it is entirely possible that given a few more I will be as arty as the next Goldsmiths academic). The validation of the new knowledge does not have to be fully quantitative or “scientific” – I have certainly mellowed enough to allow that – but there has to be some kind of proof of the pudding, and I strongly think that that proof should be able to be entered into the permanent record and made as accessible as possible, and that this is orthogonal to making a value judgment on individuals or fields of research.

[ update: I’ve arrived. The weather is horrible, but the natives are friendly ]

azed 2189

Oo, a special one: “Wrong Number”. Saith the rubric:

Each clue contains a one-word definition of the word required at the number where it stands but belongs as a whole to a word of the same length elsewhere. Method recommended is to find, after solving a clue, a definition of the solution in one of the other clues to words of its length: this will show where the word is to go.

This was fun; in many respects this was harder, because of having fewer checking letters for a long time, but one way in which I found this made it easier was in helping to parse the clues, because there would be another word in another clue of the same length which would also work as the definition: for example it wasn't too hard to guess that “neighbourhood” and “vicinity” were such a pair, and so that was likely to be the definition for one or other of 24ac and 29ac – though of course it didn't help much for answers like “viver”, neither of whose definitions I knew. In-clude definitions are underlined as usual; destination definitions are italic.

  • 1ac: SPUR → 31ac: Strut: what cat does, tail to the fore (4): PURS (obs for “purrs”, what cat does) with S moved to front
  • 4ac: ETRENNES → 30ac: Wept if denied odds on races once these are given at start of year (8): wEpT + RENNES (obs. races)
  • 10ac: THEN → 33ac: Put name against article immediately (4): THE (article) + N(ame)
  • 11ac: DIAGRAPH → 17dn: Device for e.g. copying this poet? I had got paper sorted (8): DIAGRAPH + POET = *(I HAD GOT PAPER)
  • 12ac: ROUNDTRIP → 5dn: Rake dressing rotund dissolute for circuitous journey (9): RIP (rake) around *(ROTUND)
  • 13ac: RECANT → 23ac: Disclaim wandering in France without fine temperature (6): *(FRANCE) - F(ine) + T(emperature)
  • 15ac: GIGUE → 25dn: One under command of US NCO, not one fit for dance (5): GI (under command of US NCO) + AGUE (fit) - A (one)
  • 16ac: HECTORING → 14dn: Run in chino and get prepared for bluster (9): R(un) in *(CHINO GET)
  • 18ac: OVERDO → 6dn: Ducks forming limit for little bird dropping in fatigue (6): VERDIN (little bird) - IN in OO (ducks)
  • 23ac: PTERIN → 28ac: Soak tucking into drink has to withdraw, colouring (6): RET (soak) in NIP (drink) all reversed
  • 24ac: VOISINAGE → 29ac: Position love lives amid wine, mature vicinity (9): O (love) + IS (lives) in VIN (wine) + AGE (mature)
  • 26ac: DEARE → 9dn: Costly end of marriage once badly affecting one’s conscience? (5): DEAR (costly) + (marriag)E
  • 28ac: TROPIC → 18ac: Bit of pigment acceptable in such as Priam’s circle (6)
  • 29ac: VIEWPOINT → 24ac: Scenic spot to contend with e.g. beer round centre of neighbourhood (9): VIE (contend) + W(ith) + PINT (e.g. beer) around (neighb)O(urhood)
  • 30ac: TROUVERE → 11ac: One like Blondel who presents reworked overture (8): *(OVERTURE)
  • 31ac: WORD → 27dn: Promise ends in vow to eager stud (4): voW tO eageR stuD
  • 32ac: STRIGOSE → 3dn: Stop replacing work with wild orgies, bristling (8): STOP - OP (work) + *(ORGIES)
  • 33ac: WEED → 7dn: Grass: water next day (4): WEE (water) + D(ay)
  • 1dn: STONERN → 21dn: It was very hard getting powdered colour into tin (7): TONER (powdered colour) in SN (tin)
  • 2dn: INWIT → 26ac: Single twin re-examined self-knowledge? (5)
  • 3dn: SORROWED → 4ac: Lamented one who famously streaked in age-old grassy patch (8): ROWE (thinking of Erica Roe, I think) in SORD (grassy patch)
  • 4dn: ERMELIN → 8dn: Capital? No capital – see me stuck in this old fur! (7): (b)ERLIN (Capital without its first letter) around ME
  • 5dn: GARNITURE → 12ac: Regina distressed about return of routine apparel (9): *(REGINA) around RUT (routine) reversed
  • 6dn: SWIVEL → 22dn: Ring women in veils one’s seen to harass (6): W(omen) in *(VEILS)
  • 7dn: MUSO → 1ac: Missouri’s garment for American player (4): MO (Missouri) around US (American)
  • 8dn: MAGENTA → 1dn: Mum’s making coat for sales rep, purplish (7): MA around AGENT (sales rep)
  • 9dn: ORDER → 15ac: Regulate centre of sore showing inflammation reversed (5): sORe + RED reversed.
  • 14dn: SKEDADDLE → 16ac: Flight burden that restricts annoying pest (9): SADDLE (burden) round KED (annoying pest)
  • 17dn: REDLIGHT → 32ac: Sleazy illumination for dark room enlarger? (8): Double Definition, I think
  • 19dn: DENTIST → 20dn: Stinted? Spoiled rather – if it’s filling you want, come to me (7): *(STINTED)
  • 20dn: STICKIT → 4dn: Needing doctor? About time it’s inserted for Jock (7): SICK (needing doctor) around T(ime) + IT
  • 21dn: RUINATE → 19dn: After end of supper auntie, rocky, collapsed (old) (7): (suppe)R + *(AUNTIE)
  • 22dn: ERIACH → 13ac: Hearts tend to turn about one getting serious penalty (6): H(earts) + CARE (tend) reversed around I (one)
  • 24dn: UNARY → 2dn: Posh form of yarn involving one component (5): U (posh) + *(YARN)
  • 25dn: VIVER → 24dn: With endless verve I’ll jig in pond (5): *(VIVE(r) + I)
  • 27dn: ANTI → 10ac: Opposed to a hint, convoluted, husband dropped (4): *(A HINT) - H(usband)

I could feel my brain being stretched as I did this one.

azed 2188

I found this one particularly straightforward, and (unusually for me) finished it in one session, more or less top-left to bottom-right. Maybe the months of practice have finally taken.

  • 1ac: PHENGITES: Filler for old windows (eight panes cracked, one missing): *(EIGHT PANES) - A (one)
  • 8ac: GADGET: Get away from doings, wandering: GADGET (doings) - GET
  • 11ac: LOPER: Wooden part giving support in classical opera: classicaL OPERa
  • 12ac: CULTURE: Growing bacteria left in wound, former practice: L(eft) in CUT (wound) + URE (obs. practice)
  • 13ac: UPHOER: Part of paraphernalia for awning erected by versatile hero?: UP (erected) + *(HERO)
  • 14ac: DIRAC: Physicist, reverse of barren, given constant: ARID (barren) reversed + C(onstant)
  • 16ac: PAIRED: Like couple fiddling with diaper: *(DIAPER)
  • 17ac: BISTRO: No home of haute cuisine, drowning recipe in gravy?: R(ecipe) in BISTO (gravy; ref. Bisto)
  • 19ac: SATURNALIA: A performance during endless Frankish orgy: A TURN (performance) in SALIA(n) (Frankish)
  • 21ac: REALISABLE: Ariel skippingly before dark: ‘That may be accomplished’: *(ARIEL) + SABLE (dark)
  • 26ac: BRASIL: Herb enveloping centre of burrito – a foreign country for Escoffier?: burRito in BASIL (herb)
  • 27ac: LALDIE: Women not quite swallowing their first whacking Scotch: LADIE(s) (women) with L (first of “ladies”) inside
  • 29ac: RAGGA: Type of rap music from Asia, including a touch of grunge: RAGA (music from Asia) around G (touch of “grunge”)
  • 30ac: TAPETI: Rabbit? What might that be, tucking into tree with tasty roots?: A PET (what a rabbit might be) inside TI (tree with tasty roots)
  • 31ac: CANE RAT: Brisk northerly around a long time, bane of plantation owners?: CANT (brisk) around ERA (a long time)
  • 32ac: RICIN: Toxic stuff, dose of arsenic irrationally knocked back: arseNIC IRrationally reversed
  • 33ac: KET: Road kill in Scotland, south-east area, not north: KENT (south-east area) - N(orth)
  • 34ac: REVERENCE: Homage shown as famous rider goes round enclosure: REVERE (famous rider; ref. Paul Revere) around ENC(losure)
  • 1dn: PLUMBER-BLOCK: Case for revolving shaft made of US wood, sheathed in lead, secure: LUMBER (US wood) in PB (lead) + LOCK (secure)
  • 2dn: HOP-VINE: Twining stem: there’s five in tree below house: V (five) in PINE (tree) under HO(use)
  • 3dn: NERITA: Marine gasteropod caught by mariner it attacked: mariNER IT Attacked
  • 4dn: GROK: Rumble starts in gastric region all right: G(astric) R(egion) + OK (all right)
  • 5dn: ICE HOUSE: Freezer (sort of) antiquated, so juice, scrapping starter, he mixed: *(SO (j)UICE HE)
  • 6dn: ELIAN: Fan of writer, one in grip of stylish vigour: I (one) in ELAN (stylish vigour); ref. Charles Lamb
  • 7dn: STRIGA: Reverse of dressed, clothed in sexiness, reveals a fine streak! GIRT (dressed) reversed in SA (“sex appeal”)
  • 8dn: GUARDLESS: Free of custody, lad’s urges will roam free: *(LADS URGES)
  • 9dn: ARCED: Making a bow, parts of wood are transposed: I suppose this is CEDAR with AR moved to the front, but I don’t really see how that could be deduced ab initio.
  • 10dn: DEAD-BALL LINE: Dull dance, note, limit of play: DEAD (dull) + BALL (dance) + LINE (note)
  • 15dn: ASSAILANT: Attacker who’s very little latitude, soldier possible: ASSAI (very) + L(atitude) + ANT (soldier)
  • 18dn: DRABETTE: Coarse linen put on in messy trade: BET (put) in *(TRADE)
  • 20dn: IDIOTIC: Foolish one doubled holding diamonds, reverse of quickly: II (one doubled) around D(iamons) + CITO (quickly) reversed
  • 22dn: LIDGER: Record of old accounts concealed in girdle: *(GIRDLE)
  • 23dn: LAPPIE: Cape Town rag to wrap part of picnic lunch?: LAP (wrap) + PIE (part of picnic)
  • 24dn: MARAE: One must have mum with attention rising in meeting place: MA (mum) + EAR (attention) reversed
  • 25dn: TIGRE: The inimitable Georges’s regular eponym? (Think leaders): first letters of phrase &lit.; ref Georges Clemenceau
  • 28dn: YARR: B-bit of sun brought up weed in cornfield: R-RAY (B-bit of sun) reversed
languages for teaching programming

There’s seems to be something about rainy weekends in May that stimulates academics in Computing departments to have e-mail discussions about programming languages and teaching. The key ingredients probably include houseboundness, the lull between the end of formal teaching and the start of exams, the beginning of contemplation of next year’s workload, and enthusiasm; of course, the academic administrative timescale is such that any changes that we contemplate now, in May 2014, can only be put in place for the intake in September 2015... if you’ve ever wondered why University programmes in fashionable subjects seem to lag about two years behind the fashion (e.g. the high growth of Masters programmes in “Financial Engineering” or similar around 2007 – I suspect that demand for paying surprisingly high tuition fees for a degree in Synthetic Derivative Construction weakened shortly after those programmes came on stream, but I don’t actually have the data to be certain).

Meanwhile, I was at the European Lisp Symposium in Paris last week, where there was a presentation of a very apposite nature: the Computing Department at Middlesex university has implemented an integrated first-year of undergraduate teaching, covering a broad range of the computing curriculum (possibly not as broad as at Goldsmiths, though) through an Arduino and Raspberry Pi robot with a Racket-based programmer interface. Students’ progress is evaluated not through formal tests, courseworks or exams, but through around 100 binary judgments in the natural context of “student observable behaviours” at three levels (‘threshold’, which students must exhibit to progress to the second year; ‘typical’, and ‘excellent’).

This approach has a number of advantages, I think, over a more traditional division of the year into four thirty-credit modules (e.g. Maths, Java, Systems, and Profession): for one, it pretty much guarantees a coherent approach to the year, where in the divided modules case it is surprisingly easy for one module to be updated in syllabus or course content without adjustments to the others, leaving for example some programming material insufficiently supported by the maths (and some maths taught without motivation). The assessment method is in principle transparent to the students, who know what they have to do to progress (and to get better marks); I'm not convinced that this is always a good thing, but for an introductory and core course I think the benefits substantially outweigh the disadvantages. The use of Racket as the teaching language has an equalising effect – it’s unlikely that students will have prior experience with it, so everyone starts off at the same point at least with respect to the language – and the use of a robot provides visceral feedback and a sense of achievement when it is made to do something in a way that text and even pixels on a screen might not. (This feedback and tangible sense of achievement is perhaps why the third-year option of Physical Computing at Goldsmiths is so popular: often oversubscribed by a huge margin).

With these thoughts bubbling around in my head, then, when the annual discussion kicked off at the weekend I decided to try to articulate my thoughts in a less ephemeral way than in the middle of a hydra-like discussion: so I wrote a wiki page, and circulated that. One of the points of having a personal wiki is that the content on it can evolve, but before I eradicate the evidence of what was there before, and since it got at least one response (beyond “why don’t you allow comments on your wiki?”) it's worth trying to continue the dialogue.

Firstly, Chris Cannam pulls me up on not including an Understand goal, or one like it: teaching students to understand and act on their understanding of computing artifacts, hardware and software. I could make the argument that this lives at the intersection of my Think and Experiment goals, but I think that would be retrospective justification and that there is a distinct aim there. I’m not sure why I left it out; possibly, I am slightly hamstrung in this discussion about pedagogy by a total absence of formal computing education; one course in fundamentals of computing as a 17-year-old, and one short course on Fortran and numerical methods as an undergraduate, and that’s it. It's in some ways ironic that I left out Understand, given that in my use of computers as a hobby it’s largely what I do: Lisp software maintenance is often a cross between debugger-oriented programming and software archaeology. But maybe that irony is not as strong as it might seem; I learnt software and computing as a craft, apprenticed to a master; I learnt the shibboleths (“OAOO! OAOO! OAOO!”) and read the training manuals, but I learnt by doing, and I’m sure I’m far from alone, even in academic Computing let alone among programmers or computing professionals more generally.

Maybe the Middlesex approach gets closer, in the University setting, to the apprenticeship (or pre-apprenticeship) period; certainly, there are clear echoes in their approach of the switch by MIT from the SICP- and Scheme-based 6.001 to the Robotics- and Python-based introduction to programming – and Sussman’s argument from the time of the switch points to a qualitative difference in the role of programmers, which happens to dovetail with current research on active learning. In my conversation with the lecturers involved after their presentation, they said that the students take a more usual set of languages in their second-years (Java, C++); obviously, since this is the first year of their approach, they don’t yet know how the transition will go.

And then there was the slightly cynical Job vs Career distinction that I drew, being the difference between a graduate-level job six months after graduation as distinct from a fulfilling career. One can of course lead to the other, but it’s by no means guaranteed, and I would guess that if asked most people would idealistically say we as teachers in Universities should be attempting to optimize for the latter. Unfortunately, we are measured in our performance in the former; one of the “Key Information Sets” collected by higher-education agencies and presented to prospective undergraduates is the set of student ‘destinations’. Aiming to optimize this statistic is somewhat akin to schools optimizing their GCSE results with respect to the proportion of pupils gaining at least 5 passes: ideally, the measurement should be a reflection of the pedagogical practice, but the foreknowledge of the measurement has the potential to distort the allocation of effort and resources. In the case of the school statistic, there’s evidence that extra effort is concentrated on borderline pupils, at the expense of both the less able and potential high-fliers; the distortion isn’t so stark in the case of programming languages, because the students have significant agency between teaching and measurement, but there is certainly pressure to ensure that we teach the most common language used in assessment centres.

In Chris’ case, C++ might have had a positive effect on both; the programming language snob in me, though, wants to believe that there are hordes of dissatisfied programmers out there, having got a job using their competence in some “industry-standard” language, desperately wanting to know about a better way of doing things. I might be overprojecting the relationship of a professional programmer with their tools, of course: for many a career in programming and development is just that, rather than a love-hate relationship with their compiler and linker. (Also, there’s the danger of showing the students that there is a better way, but that the market doesn’t let them use it...)

Is it possible to conclude anything from all of this? Well, as I said in my initial thoughts, this is an underspecified problem; I think that a sensible decision can only be taken in practice once the priorities for teaching programming at all have been established, in combination with the resources available for delivering teaching. I’m also not enough of a polyglot in practice to offer a mature judgment on many fashionable languages; I know enough of plenty of languages to modify and maintain code, but am comfortable writing from scratch in far fewer.

So with all those caveats, an ideal programming curriculum (reflecting my personal preferences and priorities) might include in its first two years: something in the Lisp family, for Think and Experiment; ARM Assembler, for Think and Understand; C++, for Job and Career (and all three together for Society). Study should probably be handled in individual third-year electives, and I would probably also include a course hybrid between programming, database and system administration to cover a Web programming stack for more Job purposes. Flame on (but not here, because system administration is hard).

european lisp symposium 2014

My train ride to Paris was uneventful, and I arrived at my accommodation only several hours after bedtime. I did manage to write my talk, and it was good to discover the number of obsessive planet.lisp readers when I showed up to register – “good to see you again. How’s the talk?” was the median greeting. For the record, I had not only written my talk on the train but also had a chance to relax a little. Go trains.

The conference was fun; gatherings of like-minded people usually are, but of course it takes substantial effort from determined people to create the conditions for that to happen. Kent’s work as programme chair, both before and during the conference, came with a feeling of apparent serenity while never for a moment looking out of control, and the groundwork that the local organizing team (Didier Verna, Gérard Assayag and Sylvie Benoit) had done meant that even the problem of the registrations exceeding the maximum capacity of the room – nice problem to have! – could be dealt with.

I liked the variety in the keynotes. It was very interesting to hear Richard Gabriel’s talk on his research in mood in Natural Language processing and generation; in many ways, those research directions are similar to those in the Transforming Musicology world. The distinction he drew between programming for a purpose and programming for exploration was very clear, too: he made it clear that he considered them two completely different things, and with my hat as a researcher on I have to agree: usually when I'm programming for research I don’t know what I'm doing, and the domain of investigation is so obviously more unknown than the program structure that I had better have a malleable environment, so that I can minimize the cost of going down the wrong path. Pascal Costanza gave a clear and detailed view of the problems in parallel programming, drawing a distinction between parallelism and concurrency, and happened to use examples from several of my previous lives (Smooth-Particle Hydrodynamics, Sequence Alignment) to illustrate his points. Gábor Melis talked about his own learning and practice in the context of machine learning, with a particular focus on his enviable competition record; his call to aim for the right-hand side of the curves (representing clear understanding and maximum use-case coverage) was accompanied by announcements of two libraries, mgl-pax and mgl-mat.

My own presentation was, I suppose, competent enough (slides). Afterwards, I had a good talk with my previous co-author in the generalizes research line, Jim Newton, about the details, and Gábor told me he’d like to try it “on by default”. But the perils of trying to get across a highly-technical topic struck, and I got a number of comments of the form that the talk had been enjoyable but I’d “lost them at compute-applicable-methods-using-classes”. I suppose I could still win if the talk was enjoyable enough for them to read and work through the paper; maybe next time I might risk the demo effect rather more than I did and actually do some programming live on stage, to help ground the ideas in people’s minds. I did get a specific request: to write a blog post about eval-when in the context of metaobject programming, and hopefully I'll find time for the in the next few train journeys...

Meanwhile, highlights (for me) among the contributed papers: Nick Levine driving Lispworks’ CAPI graphical user interface library from SBCL using his Common Lisp AUdience Expansion toolkit (preaching to the choir, though: his real target is Python developers); Faré Rideau’s description of a decade-long exploration of defsystem design space; François-Xavier Bois’ demonstration of web-mode.el, an Emacs mode capable of handling CSS, Javascript and PHP simultaneously; and two talks motivated by pedagogy: Pedro Ramos’ discussion of the design tradeoffs involved in an implementation of Python in Racket, and the team presentation of the approach taken for a new robotics- and Scheme-oriented undergraduate first-year at Middlesex University, on which more in a subsequent post.

Lightning talks of particular note to me: Martin Simmons talking about Lispworks for mobile; Didier Verna and Marco Antoniotti talking about their respective documentation generation systems (my response); Mikhail Raskin’s argument about the opportunity to push Julia in a lispy direction; and probably others which will come back to mind later.

I was also pleased to be able to contribute to the last full session of the symposium, a workshop/panel about Lisp in the area of music applications: an area which is serendipitously close to the day job. I worked on getting a version of audioDB, our feature-vector search engine for similarity matching, built and working on my laptop, and finding a sufficiently interesting search among my Gombert/Josquin collection to demo – and I also had the chance to talk about Raymond Whorley’s work on using multiple viewpoint systems for hymn harmonizations, and what that teaches us about how people do it (slides, for what they're worth). Other systems of interest in the session included OpenMusic (of course, given where we were), PWGL, OMax, modalys, and overtone; there was an interesting conversation about whether the choice of implementation language was restricting the userbase, particularly for tools such as OpenMusic where the normal interface is a graphical one but a large fraction of users end up wanting to customize behaviour or implement their own custom patches.

And then it was all over bar the dinner! On a boat, sadly immobile, but with good conversation and good company. The walk home in company was fun, though in retrospect it was probably a mistake to stop off at a bar for a nightcap... the train journey back to the UK the following morning was definitely less productive than it could have been; closing eyes and letting the world go past was much more attractive.

But now I'm on another train, going off to record the complete works of Bernadino de Ribera. Productivity yay.

azed 2187

More references to people than recently, I think. No time at all to work on a clue for WEST, sadly; as it is I’m typing up this in between rehearsal and concert. Oh well. Maybe next month...

  • 1ac: PSYCHASTHENIA: Pay is adjusted with mostly modest woman involved in mental affliction: *(PAY IS) around CHAST(e) (modest) + HEN (woman)
  • 10ac: BULLA: Game played on deck with a seal
  • 11ac: JEHU: Driver best given a wide berth, for whom it’s a game injecting heroin: JEU (game) around H(eroin)
  • 12ac: SIDA: A Queensland native, it could be said: *(SAID)
  • 14ac: UNPAYS: A country using French currency is making good financially: UN PAYS (“a country” in French)
  • 16ac: ROGER: Bacon, say, OK for goose?: Triple Definition (ref. Roger Bacon)
  • 17ac: SAUTERNE: Tea-urn’s misused with English wine: *(TEA URNS) + E(nglish)
  • 18ac: DISTENT: What’s bloated in poetry captured by parodist entertainingly: paroDIST ENTertainingly
  • 20ac: OSTEOPETROSIS: Bone disease ties stooper so in knots: *(TIES STOOPER SO)
  • 24ac: HORNIST: Musician trapping nurse in lift: RN (nurse) in HOIST (lift)
  • 26ac: URUSHIOL: Ancient ox, last to munch oil secreted in juice of poison ivy: URUS (ox) + (munc)H + *(OIL)
  • 28ac: SADZA: Porridge from South Africa, doughy at first: ZA (South Africa) after SAD (doughy)
  • 29ac: OSTEAL: Of bone: O (of) + STEAL (bone) &lit.
  • 30ac: TWEE: Former interval Ben’s shrugged off, sickly: BETWEEN (interval) - BEN
  • *31ac: WEST: Cardinal point
  • 32ac: ETTLE: Scots aim to annoy, ignoring north at the outset: NETTLE (to annoy) - N(orth)
  • 33ac: UNSYMMETRISED: Direness, with tummy spreading, not kept in proportion: *(DIRENESS TUMMY)
  • 2dn: SUNI: Antelope, one found in part of NZ: UN (one) in SI (South Island)
  • 3dn: CLATTER: Yak cart let out: *(CART LET)
  • 4dn: HAYLE: Mills maybe mostly producing sound for poet of yore: HAYLE(y) (ref. Hayley Mills)
  • 5dn: SENATE HOUSE The USA’s one reverberates with ultimate in eloquence?: *(THE USAS ONE) + eloquencE &lit.
  • 6dn: THRUST: Force? It briefly worsened ...: T (“’t”, it briefly) + *(HURTS) (...)
  • 7dn: HURTS: ... Such wounds – unit required by the sound of it: homonym with Hertz (SI unit)
  • 8dn: NIGROSINE: Drink up wine that’s bottled in very dark colour: GIN (drink) reversed + ROSE (wine) with IN inside
  • 9dn: IDENT: Tell-tale sound sequence – it’s absorbing study: IT around DEN (study)
  • 10dn: BUNDOBUST: Indian settlement British dismantle, insolvent: B(ritish) + UNDO (dismantle) + BUST (insolvent)
  • 13dn: AREOSTYLE: Colonnade having ruined storey guarded by porter?: *(STOREY) inside ALE (porter)
  • 15dn: POSTLUDES: Posset drunk, lord bedded – we’ll finish off works: *(POSSET) around LUD (lord)
  • 19dn: NONETTI: Chamber pieces? Count me out – that’s about conclusive: NET (conclusive) in NOT I (“count me out”)
  • 21dn: PHILEM: Brief letter, one from Greece that’s yellow on fading: PHI (Greek letter) + MEL (yellow) reversed; ref. Epistle to Philemon
  • 22dn: DRAWN: Rule in e.g. French described: R(ule) in DAWN (ref. Dawn French)
  • 23dn: SHAWM: Gently (G) – who’s playing that period instrument?
  • 25dn: RISER: End of gutter is on inverted vertical pipe: (gutte)R + IS + RE (on) reversed
  • 27dn: BALE: E.g. Christian wretchedness, as of old: Double Definition; ref. Christian Bale
on my way to els2014

After a packed weekend of music-making with De Profundis (go buy tickets!), I’m now at St Pancras, waiting for a train to Paris to go to the European Lisp Symposium, where I’ll be presenting my work (with Jan Moringen and David Lichteblau) on generalizer metaobjects. I’m looking forward to the event: it’s being held at IRCAM, which as an institution neatly combines two of my interests (computing and music) – there’s going to be a special session on Tuesday on Lisp and composition, though I might try to participate and broaden it a bit to talk about some of our current problems and approaches in Transforming Musicology.

I’m not (yet) looking forward to my talk. Over the next few hours, I’m going to be putting to the test my current belief that I am most productive on trains, because I haven’t yet written my talk, and it turns out that I am in the first session tomorow morning. (I know, I know, this is the sound of the world’s tiniest violin). 2h20 should be enough fo anyone.

The flip side of the stress of being first is the ability to enjoy the rest of the symposium without the stress of an upcoming presentation hanging over me – as well as the ability to ask nasty difficult and probing questions without too much fear of instant reprisal. (I suppose that by putting this out there before the talk I open myself up to pre-reprisals; all I can say is that I welcome a constructive and robust exchange of views: there is definitely more work to be done in the world of extended dispatch, and suggestions will be welcome.)

And of course it will be good to meet up with a community of like-minded people; I may not be highly active in the Lisp world these days (though I have been making an effort to be more engaged with at least the SBCL team for more than the one day a month of release activity, and it’s been fun to watch the ARM port progress on #sbcl) but I retain a good helping of interest in what’s going on, and it will be great to meet up with old friends and maybe even to make new ones. (Registration got to the room capacity very quickly, and hopefully the overflow room will have a good atmosphere too). Hopefully the conversations I have and ideas generated will mean that I have a productive train ride back, too!

azed 2186

Typing this up in Sidney Sussex chapel, in between rehearsing for my contributions to De Profundis' concert on 10th May featuring the surviving works of Bernadino de Ribera. Readers, first buy a ticket for the concert, and then come back for the clue analysis of Azed 2186...

  • 1ac: SKUG: Mooching around, left family out in shelter: SLULKING (“mooching around”) - L(eft) - KIN (family)
  • 4ac: DOORMATS: Cook leftovers – mum included such as may regularly be exploited: DO (cook) + ORTS (leftovers) around MA (mum)
  • 11ac: WANY: Tending to decline method that includes small number: WAY (method) around N(umber)
  • 12ac: SOARE: Young hawk once flew aloft endlessly: SOARE(d) (flew aloft)
  • 13ac: ATAP: Roofing material that a palm provides (see within): thaT A PAlm
  • 14ac: REQUITAL: Reward to discharge in bit of Brazilian money: QUIT (discharge) inside REAL (Brazilian money)
  • 15ac: MAISTERING: Scots gaining control of drove after month abroad: MAI (fr. May) + STERING (drove)
  • 16ac: MISANDROUS: Detesting sex? Sounds not quite right for Julie!: sounds like “Miss Andrews”; ref. Julie Andrews
  • 17ac: CHEKIST: Guardian of national security maltreating the sick: *(THE SICK); ref. Cheka
  • 22ac: FALSISM: As films will misguidedly present it, it’s manifestly untrue: *(AS FILMS)
  • 24ac: PRIESTSHIP: Ministry persist with change, following latest political ideas?: *(PERSIST) + HIP (? latest political ideas)
  • 28ac: SHAKEDOWN: Temporary straw bed appeared to contain a wingless pest: SHOWN (appeared) around A KED (wingless louse)
  • 29ac: EGOMANIA: Abnormal conceit, reverse of mature, gripping ME Arab?: AGE (mature) reversed around OMANI (Middle-Eastern Arab)
  • 30ac: GLUE: Leaders in good long underwear – every size: G(ood) L(ong) U(nderwear) E(very)
  • 31ac: OKAPI: African creature, fine sloth, with pressure in grip: OK (fine) + AI (sloth) around P(ressure)
  • 32ac: PINS: Cut back nails: SNIP (cut) reversed
  • 33ac: STONE-AGE: What’s extremely retro one replaces even in Home Counties location: STEVENAGE (Home Counties location) - EVEN + ONE
  • 34ac: INGS: Sign showing diversion for riverside meadows: *(SIGN)
  • 1dn: SWAMP CYPRESS: Inhabitant of everglades was immersed, c-crowd hugging country’s borders: SWAM (was immersed) P-PRESS (c-crowd) around C(ountr)Y
  • 2dn: KATAL: SI unit tailpiece in a discussion brought to the fore: A TALK (a discussion) with K brought to front; ref. Katal
  • 3dn: UNAIMED: Random spraying of medium round piece of art: as best as I can see, a mistake: *(MEDIUM) around A(rt), but one of the ‘M’s needs to be an ‘N’
  • 4dn: DARTS: Board game is starting of a sudden: Double Definition
  • 5dn: OPERAS: ‘Soaper’ represents such (at least nominally, in US): *(SOAPER) &lit.
  • 6dn: ROUNDEL: Disc with undulations in part (end lost): UNDE (undulations) in ROL(e) (part)
  • 7dn: MAIGRE: Fish from the Med, with bit of work I am hauled up: ERG (bit of work) + I AM all reversed
  • 8dn: ARTY: Would-be aesthete some lowly tradesman raised: lowlY TRAdesman reversed
  • 9dn: TEACUPS: Kitchen measures in paste stirred round copper: *PASTE) around CU (copper)
  • 10dn: SELF-SAMENESS: Identity crookedly false chaps put in ‘tax’ form: *(FALSE) + MEN (chaps) in SESS (tax)
  • 18dn: HARD-GOT: Excited dog brought in deer attained with difficulty: *(DOG) in HART (deer)
  • 19dn: IN SHAPE: Trendy one exercises, holding belt up, to get fit: IN (trendy) A (one) PE (exercises; ref. Physical Education) around SH (belt up)
  • 20dn: CIPOLIN: Heading for Carrara, Italy, I cut up new marble: C(arrara) I(taly) + I LOP (cut) reversed + N(ew)
  • 21dn: DESMAN: Insectivore inhabiting everglades (mangroves): everglaDES MANgroves
  • 23dn: ASKING: This price precedes haggling – fell with silver around: SKIN (fell) inside AG (silver)
  • 25dn: HEAVE: Reach harbour, circling east, not north: HAVEN (harbour) around E(ast) - N(orth); “reach” as in “retch”
  • 26dn: SWUNG: Women in oriental dynasty were lively: W(omen) in SUNG (ref. Song dynasty)
  • 27dn: TOKO: Is it over the top when involving king’s head? TOO (over the top) around K(ing)
azed 2185

I'm a bit late blogging this because of my travels. All done and dusted now, though, phew. Not all that much of particular interest in this one, I think; the usual budget of obscurities in vocabulary but the cultural references were solidly in my ambit.

  • 1ac: STOCK-PUNISHT: Liable to have been pelted with rotten veg, thus conk-tip’s battered?: *(THUS CONK TIPS)
  • 10ac: ORPHEANS: Amateur music society, English, going into wards, often?: E(nglish) inside ORPHANS (often wards)
  • 12ac: LOFI: Extract from Annal of India not particularly well reproduced: annaL OF India
  • 13ac: CRIBLÉ: Bone left in Church, riddled?: RIB (bone) + L(eft) in CE (church)
  • 14ac: DUMPLE: Shape for cooking the French put on counter: DUMP (put) on LE (“the” in French)
  • 15ac: NEELE: Old sewer? Not old when fish is found in it: EEL (fish) inside NE (obs. “not”)
  • 17ac: PARADROP: Feature of e.g. Arnhem road damaged in blast: *(ROAD) inside PARP (blast); ref. Battle of Arnhem
  • 18ac: CERES: Minor planet, dry and cold when viewed from behind: SERE (dry) + C(old) reversed
  • 20ac: SCRAW: Spy dropping out acquires unprepared bit of thin turf: SCOUT (spy) - OUT + RAW (unprepared)
  • 22ac: HOTCH: Central heating working properly when on, making Scots fidget: HOT + C(entral) H(eating)
  • 23ac: PETRE: Meat preservative? Portion of tripe treated: triPE TREated
  • 25ac: AESTHETE: After the tide turns, former drudge becomes ...: SEA reversed + THETE (a poor freeman in ancient Athens) &lit. with next clue; I took forever to parse this, though not to enter the word, because I could see THE and SEA and ETA (another kind of “former drudge”) but not how to glue them together.
  • 28ac: TRIAS: ... One absorbed by arts, adrift in ‘rock period’?: I (one) in *(ARTS)
  • 29ac: AVAUNT: Acting against gossip from the past, the old move on: A(cting) V(ersus) AUNT (obs. “gossip”)
  • 31ac: COMPOT: Count no longer fed with duck or fruit pudding: COMPT (obs. “count”) around O (duck)
  • 32ac: STIE: Old mount is obliging, moving rear to the front: TIES are obliging, I guess;
  • 33ac: FRONT-END: X inscribed in leaf – it involves interaction with the public: TEN (“X”) inside FROND (leaf)
  • 34ac: DISINTERESTED: En suite trend transformed, rendering one apathetic?: *(EN SUITE TREND)
  • 2dn: TROULE: The old wag that could cause annoyance if penning book: TROUBLE (annoyance) - B(ook)
  • 3dn: CHIPSET: Computer group with particular functions maybe approach apparatus: CHIP (maybe approach, as in e.g. Rugby) + SET (apparatus)
  • 4dn: KELL: Film star princess appearing endlessly in film?: KELL(y); ref. Grace Kelly
  • 5dn: PALEA: The old man has bottled beer in this receptacle: PA (the old man) around ALE (beer)
  • 6dn: UNCURSE: Lift ban on exercise involving neuter dog: USE (exercise) around N(euter) CUR (dog)
  • 7dn: IVIED: Lake neglected, furious about how it ends, overgrown: LIVID (furious) - L(ake) around (lak)E
  • 8dn: SUBERATE: Salt from America coming up before reprimand: US reversed + BERATE (reprimand).
  • 9dn: HALLOW: Who’s bemused about the whole bang shoot? Bless!: *(WHO) around ALL
  • 10dn: Nick Compton’s opener featuring in varied Lord’s chat: C(ompton) in *(LORDS CHAT); ref. Names for the Devil
  • 11dn: DEEP-SEATED: Edward aboard his bathyscape? It’s in the blood: DEEP-SEA TED
  • 16dn: PROSIMII: Some primates, old-maidish about bits of obscene speech I repeated: PRIM (old-maidish) around O(bscene) S(peech) + II
  • 19dn: THEATRE: Gaiety e.g. as mad Hatter grabs dormouse’s tail: *(HATTER) + (dormous)E
  • 21dn: RECASTS: Switches part of current account put into reserve funds: C(urrent) A(ccount) inside RESTS (reserve funds)
  • 22dn: HEROIN: E.g. dynamite blasted iron junk?: H(igh) E(xplosive) (e.g. dynamite) + *(IRON)
  • 24dn: RANINE: Might it be synonymous with ‘sublingual’, artist muses?: RA (artist) + NINE (muses)
  • 26dn: TAPIN: Gettin’ the measure of a simple putt: TAPIN(g) (measuring)
  • 27dn: TABOR: What pipe cleaner clears round half of bowl – something you get with a pipe: TAR (what pipe cleaner clears) around BO(wl)
  • 30dn: VINE: Climber from Argentina getting lost at head of gorge: RAVINE (gorge) - RA (Republic of Argentina)
just like old times

Sometimes, it feels like old times, rolling back the years, and so on. One of the things I did while avoiding work on my PhD (on observational tests of exotic cosmologies inspired by then-fashionable high-energy physics theories, since you ask) was to assist in resurrecting old and new CMUCL backends for SBCL, and to do some additional backend hacking. I learnt a lot! (To be fair, I learnt a lot about General Relativity too). And I even wrote about some of the funny bits when porting, such as getting to a working REPL with a completely broken integer multiply on SPARC, or highly broken bignums when working on a proper 64-bit backend for alpha.

On the #sbcl IRC channel over the last few days, the ARM porting crew (not me! I've just been kibitzing) has made substantial progress, to the point that it's now possible to boot a mostly-working REPL. Some of the highlights of the non-workingness:

(truncate (expt 2 32) 10)

(fib 50)

(expt 2.0 3)

(sb-ext:run-program "uname" '() :search t :output *standard-output*)
Memory fault at 3c

Meanwhile, given that most ARM cores don't have an integer division instruction, it's a nice bit of synchronicity that one of this year's Summer of Code projects for SBCL is to improve division by constant integers; I'm very much looking forward to getting involved with that, and hopefully also bringing some of the next generation of SBCL hackers into the community. Maybe my writing a while back about a self-sustaining system wasn't totally specious.

azed 2184

So, a slightly different solve for me this week: away from Chambers, so with rather more reliance on the Internet for filling in gaps and for word meaning confirmation. One proud moment was looking up TRON using the power of grep through my previous solves, to find that it did indeed mean “market”.

  • 2ac: FLAMBOYANTE: Following former whim, lad gets stake for tropical tree: F + LAM + BOY + ANTE
  • 10ac: SLUMP: Depression about closing of theatrical flop: SUMP around (theatrica)L
  • 11ac: ZABIAN: Semi-Christian sectarian from South Africa, one curbed by taboo: ZA + BAN (taboo) around I
  • 13ac: SUMPIT: Tot given mine delivers devastating darts: SUM + PIT
  • 14ac: OPPO: One’s mate, very low before and after love: PP (pianissimo) before and after O (love)
  • 16ac: KIPPAS: Snooze before action requiring special headgear: KIP + PAS
  • 17ac: STYLOLITE: Sort of stone boundary to till? Yes, laboriously: *(TO TILL YES)
  • 18ac: BEZEL: Elizabeth may make a hit with this oblique stone’s face: *(ELIZABETH - A HIT)
  • 19ac: ELYTRON: Plate of worms appearing in Cambs. city market: ELY + TRON
  • 21ac: REPULSE: Check up showing relapse – reels shakily about that: *(REELS) around UP reversed(?)
  • 23ac: YEZDI: One must give the Devil his due – even features of Azed likely to appear contorted!: *(aZeDlIkElY)
  • 27ac: CASSAREEP: Ingredient of WI cuisine like it is used to stuff toad: AS (like) + SA (?) in CREEP (toad)
  • 30ac: DREDGE: Sprinkle a bit of demerara, a form of greed: D(emerara) + *(GREED)
  • 31ac: GEDS: River fish caught in ringed seines: rinGED Seines
  • 32ac: ITALIE: Palm tree situation? It has its place in Paris: not sure about the parsing, other than Place d'Italie being in Paris
  • 33ac: EGOIST: I’m not interested in what others think or say on ’eight of sail: EG (say) + OIST (HOIST = “height of sail” - H)
  • 34ac: STEER: Guide more obdurate when avoiding lie: STEELIER (more obdurate) - LIE
  • 35ac: BOG STANDARD: Dirk e.g. (not English) edging suit of armour, basic: BOGARDE (ref. Dirk Bogarde - E + STAND (?)
  • 1dn: ASSES’ BRIDGE: Rigs shifting with sea-beds? It blocks progress completely: *(RIGS SEA BEDS)
  • 2dn: FLUATE: Former compound one put into wineglass: FLUTE (wineglass) around A (one)
  • 3dn: LUMMY: Excellent Dickensian upper class, deprived of power initially: PLUMMY (upper class) - P(ower)
  • 4dn: AMPULE: Drug container from Panama secreted by a smuggler thereof?: A MULE (drug-smuggler) around P(anama)
  • 5dn: BOTTLES: Earns by busking what’ll often end in banks: Double Definition
  • 6dn: OZEKI: Heavyweight champ? Little weight put on as turning up with loss of pounds: OZ (little weight) + (LIKE - L) reversed; ref. Ōzeki
  • 7dn: YAKITY-YAK: Gossip? It’s variable with pair tended by Tibetan milkmaids around?: IT + Y (variable) inside YAK twice
  • 8dn: NIPPER: Cracked pipe splitting near wire?: NR (near) outside *(PIPE)
  • 9dn: TAPAS: Pieces of Italian staple exchanged as appetizers: PASTA (Italian staple) rearranged
  • 12dn: NOSENIPPERS: No drinkers will imbibe small measure in these glasses: NO + SIPPERS (drinkers) around EN (small measure)
  • 15dn: COLLAGIST: ‘Scrap merchant’? I irritate when going up in price: I GALL (irritate) reversed in COST (price)
  • 20dn: LESBIAN: Selina, mad about baron? Probably not: *(SELINA) around B(aron) &-sort-of-lit.
  • 22dn: PIE-DOG: Pedigree? No – wild one ne’er displays that: *(PEDIGREE NO) = NE’ER + PIE-DOG
  • 24dn: ERRATA: Slips revealing solver ratability to some extent: solvER RATAbility
  • 26dn: GREGO: More than half of calendar calls for such an overcoat? GREGO(rian) (ref. Gregorian Calendar)
  • 25dn: DEFIED: Was daring exploit interrupted by rising uncertainty?: DEED (exploit) around IF reversed
  • 28dn: SEPTA: Dividers, a pest when broken: *(A PEST)
  • 29dn: EULER: Mathematician requiring rubber in middle of numerals: ULE (rubber) in numERals; ref. Leonhard Euler
azed 2183

Oh, fun. Something a bit different: an Eightsome Reels puzzle, where each answer is eight letters, and the answers must be written around the clue number starting anywhere and going in either direction. I didn't find this too hard; once I'd got three clues which intersected in one square (in an ‘L’ shape) the rest fell fairly quickly. The clues themselves were pretty straightforward, so here's the grid, with the clue numbers indicated in the most excellent base-36 encoding with digits 1-9a-z.


And now I have to clue ‘SANGRAIL’, and all I can think of is something to do with Skimbleshanks. How about “Reva Rice maybe did for cup final?” – probably too obscure, but hey, it's not like I'm going to actually enter...

azed 2182

I did complete Azed 2182, but cunningly knocked over a full coffee cup in my study and soggily obliterated my solution (and, more importantly for blogging, the clues). fifteensquared has the analysis: last to go in for me was either SCAFF-RAFF or FOIL; both needed confirmation from Chambers.

Meanwhile, my coffee accident was astonishingly non-damaging; despite there being the bits and pieces of four computers on my desk at the time, the only component which got even slightly soggy was my sadly unused Raspberry Pi (still no SBCL port) and the time-honoured strategy of turning it off and on again seemed to make it as functional as it ever has been.

student-run software houses

A while ago, I attended an HEA event with the title of “Student-Run Software Houses”. This is a topic which is both personally interesting and relevant to one of my departmental employability brief; it fits nicely both with the narrative that a University should be empowering its students to learn and live well, and with the positive noises about the Digital Economy that a Computing department should be supporting.

The day opened with the customary introduction by a senior academic, in this case Professor Palmer-Brown, Dean of the Faculty of Life Sciences and Computing. Typically, these introductions are hilarious because the senior academic has no idea what the event is, who the speakers are and why they should care; they might have had the introduction in the diary for ages, or it might have been sprung on them, but in either case they're almost certainly unprepared. Usually they'll therefore depend on the speaker schedule and titles to give them a sense for what they should say; in this case, the schedule was laconic, and it was mildly embarrassing to see the resulting flailing. Maybe the moral is for the local organizer of the event to write a script for their senior colleague? (Or else, skip the pointless welcome).

The morning sessions included a number of short(ish) talks from a number of practitioners: people helping to run their own take on student software development. The talks were interesting, not least because of the breadth of approaches, and how those approaches were often dictated by particular facilities or lack of them.

For example, Amit Chopra's Studios in Lancaster's Software Engineering programme was particularly distinguished, in my notes at least, by having their own dedicated studio space for their students. With 24-hour access. For the students. Space. My mind was blown, particularly when I also heard about the low enrolment numbers for the course. Liz Coulter-Smith's experience at Worcester was a litany of institutional dysfunction, fighting not to fall between the cracks (is student software development teaching? research? industrial engagement?) At the opposite extreme was Graham Roberts' experience from UCL, where the description of institutional support had some of the group drooling: having “2 to 3” full-time members of staff just to manage relationships between the CS courses and potential collaborators was my favourite data point. With all that, maybe the most painfully hilarious unintended consequence was the result of Apple's policy that there should be only one App Store developer account per insitution: bad luck to the student software team if the University marketing department has got there first...

One common thread among the presentations: they weren't about student-run software houses. In all cases, the organizational aspects where handled by academic staff, whether within the context of a credit-bearing module, running a digital agency on a consultancy basis (as our host Yanguo Jing from London Met does) or coordinating volunteer effort (as Elaine Major does at Greenwich). I can see why this happens: there's a natural assumption that continuity is essential, and managing reputational risk is important; on the other hand, it ought to be possible to expose students to more of the reality of running an organization as well as participating in it, to give them an appreciation for all the “other stuff” – and also to allow the non-developers among the students a chance to be substantially useful.

There was an interesting point about competition. One problem identified by academics in smaller towns is that setting up a consultancy or digital agency for coordinating student work runs the risk of competing with local business, and undercutting both through cross-subsidy and through the use of under-market-rate student labour. This is obviously not ideal; it's even worse if the competition is primarily made up of companies formed by the University's own graduates. The obvious answer is to aim to do things that are substantially different from local businesses; on the other hand, doing things that no-one else is doing introduces risks, because possible reasons for no-one doing things include “they're too hard” and “no-one wants them”. Still, there might be valuable lessons there for students too.

inquisitor 1326

Another fun one; I somewhat prefer the inquisitors with slightly more obscure vocabulary and slightly less messing with clues – perhaps because at least I stand a chance.

In the completed grid, one member of a family is leaving a location. One member of another family is entering the same location. Each uses the same technique to pass an obstacle, different in each case. Solvers must highlight both members. Unchecked letters of the unclued families, location, technique and obstacles could form LOCAL ACT, VIVA RACE.

  • 8ac: URAO: Beginning of unholy row over onshore deposit: U(nholy) + OAR (row) reversed
  • 9ac: GATS: Guns to follow in retreat: STAG (follow) reversed
  • 12ac: TENNOS: Recall verse titles: SONNET reversed
  • 13ac: SOREE: Rail about joining members of secret service: RE (about) inside SOE (ref. Special Operations Executive)
  • 15ac: DROMOI: Gypsy involved in swindle; the first to occur in Irish racecourses: ROM (Gypsy) in DO (swindle) + I(rish)
  • 18ac: MOA: Bird no longer starts to moult outside aviary: M(oult) O(utside) A(viary); ref. Moa
  • 21ac: RENAME: Marlene left out mobile to call again: *(MARLENE) - L
  • 26ac: FOEHN: Wind flipping over fluttering hen: F(lipping) O(ver) + *(HEN)
  • 27ac: AROUSE: Nothing in a trick generates alarm: O in A RUSE
  • 29ac: OGEE: Moulding's highest point apparently missing: APOGEE - AP(parently)
  • 30ac: DRIP: Weak person finally shed tear: (she)D + RIP (tear)
  • 1dn: AUTO: Mini drama?: Double definition
  • 2dn: RANEES: Princesses fled uprising to await further: RAN (fled) + SEE (await further) reversed
  • 3dn: ION: Play by Euripides – one currently being performed: I (one) + ON (currently being performed); ref. Ion
  • 4dn: EASE: Manoeuvre bit by bit to stop Corps leaving: CEASE (stop) - C(orps)
  • 5dn: ISOPOD: Woodlouse say, in case belonging to audio player: SO (in case) inside IPOD (audio player)
  • 6dn: DARI: Organized raid to find grass: *(RAID)
  • 7dn: ELEGIES: Support that is shown in El Salvador for songs: LEG (support) + IE (that is) in E(l) S(alvador)
  • 9dn: GOLEM: Robot to work with moon vehicle: GO (work) + LEM (Lunar Excursion Module)
  • 10dn: TSAR: Despot so far heading north in Turkey: AS (so far) reversed + TR (Turkey)
  • 11dn: VENOM: Divers move to contain nuclear poison...: *(MOVE) around N(uclear)
  • 14dn: CARAFES: ...carriage made safe in flasks: CAR (carriage) + *(SAFE)
  • 15dn: DOE: Drover regularly encountered deer: DrOvEr
  • 17dn: FATHOM: Discover money in short: FAT (money) + HOM(e) (in, shortened)
  • 19dn: AFORE: Going about Faroes almost getting lost once: *(FAROE(s))
  • 20dn: PLEURA: Dispute besieging ancient city's side-walls: PLEA (dispute) around UR (ancient city)
  • 22dn: MING: Aircraft designers engulf navy in stink: MIG (aircraft designers) around N(avy)
  • 23ac: CELL: Mobile phone's a disappointment by the sound of it: homophone with SELL (disappointment)
  • 24ac: CAEN: John goes round Eastern French city: CAN (John) around E(astern)
  • 25ac: NEPS: Dispensaries stocking up catmints in some places: diSPENsaries reversed
  • 28ac: ODD: Out-of-the-way force deserted: OD (force) + D(eserted)

It then turns out that a LEMMING, a member of the CRICETIDAE family, deals with a CLEVE by OVERLEAPING it, while a SALMON, a member of the SALMONIDAIE family, also passes FALLS by OVERLEAPING. (LEMMING appears vertically downwards through CLEVE, while SALMON goes upwards through FALLS. Nice touch.)

azed 2181
  • 1ac: SPADILLO: Solid player's half miscued ace, unbeatable?: *(SOLID PLA(yer))
  • 7ac: PIPA: Hump on a toad? It's pear-shaped: Double Definition; ref. Pipa Pipa pipa (?)
  • 10ac: AULARIAN: Member of Oxford hall – university lecturer leads song in one: U(niversity) L(ecturer) + ARIA (song) in AN (one)
  • 11ac: IN ON: Party to love, held in local: O (love) in INN (local)
  • 12ac: VENDITION: Sale: don't start it in Nov., I scribbled: END IT (don't start it) in *(NOV I)
  • 13ac: CABIN: Area for passengers (cases cooler): CA(ses) + BIN (cooler)
  • 14ac: MOWRA: It has edible flowers a worm chewed up: *(A WORM)
  • 16ac: TRADE GAP: Sort of balance that's summoned skill when going backeards: PAGED (summoned) ART (skill) all reversed
  • 18ac: MEGILLAH: Lengthy US screed badly penned by very good husband: ILL (badly) inside MEGA (very good) H(usband)
  • 21ac: MAD-APPLE: Aubergine mum marked with spots: MA (mum) + DAPPLE (mark with spots)
  • 26ac: COUNTESS: Grand lady Luxembourg picked out from very many: COUNTLESS (very many) - L(uxembourg)
  • 27ac: CONIN: Not in favour of being involved with poisonous liquid: CON (not in favour of) + IN (involved with)
  • 28ac: VERST: Silverstone laps... such a distance?: silVERSTone
  • 30ac: UNSTOPPER: Open, and spurt may result when you do it: *(OPEN + SPURT) &lit.
  • 31ac: FENT: Modern poet, not on crack: FENTON (ref. James Fenton) - ON
  • 32ac: ELEGANCE: Refinement, as in ‘cast’: EG (e.g. = “as”) in ELANCE (throw = “cast”)
  • 33ac: FRAY: Extremes of fervour always apparent in brawl: F(ervou)R + AY (always)
  • 34ac: DERANGED: Crazy, like old kitchen after modern makeover: an old kitchen would be DE-RANGED (have its range removed) in modernization
  • 1dn: SANCTUM: If old, court goes in to chief points in private room: AN (obs. if) + CT (court) in SUM (summary = chief points)
  • 2dn: PUJA: Hindu festival (NB not found in one Indian state): PUNJAB - NB
  • 3dn: A L'ABANDON: A research place soon going round Director recklessly: A LAB ANON around D(irector)
  • 4dn: DAVIDIA: Help offered up inveigles greedy Chinese native: AID (help) reversed around AVID (greedy)
  • 5dn: LINAGE: String of tinsel in a genipap tree (pretty old): tinseL IN A GEnipap
  • 6dn: LAD MAG: Mad when banged up inside, convict displays place for pin-ups?: *(MAD) inside LAG (convict)
  • 7dn: PITON: Mountaineers' peg? This gear men use, possibly: *(MOUNTAINEERS PEG) = *(GEAR MEN USE + PITON)
  • 8dn: POOR LAWS: Hands around contents of pool, right and left? They assist the needy: PAWS (hands) around pOOl + R(ight) + L(eft)
  • 9dn: ANNAM: Delicious food is served up in this former French protectorate: MANNA (delicious food) reversed
  • 15dn: WELTERING: Rolling about in the sea, very hot when topless: (s)WELTERING (very hot)
  • 17dn: REASONER: Participant in debate? it's rare one's trounced: *(RARE ONES)
  • 19dn: LATERAN: Relating to church of Rome, I'll yield to age in what's spoken therein: LATIN (spoken in Roman churches) - I + ERA (age)
  • 20dn: HUSTLED: Lutes broadcast in sharp focus, crowded together roughly: *(LUTES) in HD (high definition = “sharp focus”)
  • 22dn: POPPLE: Oar, very soft inside, getting to heave in rough water: PP (pianissimo = “very soft”) inside POLE (oar)
  • 23dn: LUMPER: Casual worker eading tax maybe, member breaking rule shockingly: MP (member) in *(RULE)
  • 24dn: SCUFF: Graze sheep initially on place for links? S(heep) + CUFF (place for (cuff)-links)
  • 25dn: BITTY: Disjointed, near to ringing communications giant: homophone of BT (ref. BT)
  • 29dn: SYCE: Chauffeur locating carbon in filter: C(arbon) in SYE (filter); interesting because SICE is the same word meaning chauffeur, and the I/Y was unchecked – it's just SYE meaning filted that disambiguates
azed 2180

Perhaps fewer obscurities and references this week? One fun thing to do with copious free time would be to examine the crossword data for patterns. Some day.

  • 1ac: CHLORACNE: Skin condition left over after hospital in dispersal of cancer: L(eft) O(ver) after H(ospital) in *(CANCER)
  • 10ac: ROUX: Right page in book wife follows for thickening in recipe: RO (recto) + UX (wife)
  • 11ac: GOA BEAN: Tropical African plant Gabon developed at edges of river: *(GABON) around EA (river)
  • 12ac: OREADES: Nymphs are dancing in e.g. Horatian verses: *(ARE) inside ODES (ref. Horace's Odes)
  • 13ac: MARSH: It may suggest Romney is disfiguring horse's head: MARS (is disfiguring) + H(orse)
  • 14ac: WINTERISE: Astute about earth, prepare for colder times: WISE (astute) around INTER (earth)
  • 15ac: BEIN: Scots well-to-do, not the last in essence: BEIN(g) (essense)
  • 16ac: PEAR: E.g. conference held in deep earnest: deeP EARnest; ref.
  • 17ac: HUCKSTERESS: Suckers she's bamboozled about a bit of tat?: *(SUCKERS SHES) about T(at) &lit.
  • 19ac: MONTICOLOUS: Sign of Japanese family, lacking heart (does one hear?), living the high life: MON (sign) + TI (?) + COLOUS (sounds like “callous” = heartless)
  • 23ac: ISMS: Lissomest will regularly display such practices: lIsSoMeSt
  • 26ac: SPRY: One disembarks from jet with a spring in one's step: SPR(a)Y (= jet)
  • 28ac: RHEOTROPE: It switches current or the flow in e.g. cable: *(OR THE) in ROPE
  • 29ac: GARNI: Trimmed turns in college entertainment: IN RAG (college ent.) reversed
  • 30ac: UNITION: Making one fortification metre short at front: (m)UNITION (fortification)
  • 31ac: ENTROPY: Field of medicine not good? This may measure system disorder: ENT (Ear Nose and Throat) + ROPY (not good)
  • 32ac: IMAC: Range of modern computers (mica chips): *(MICA)
  • 33ac: INTEGRATE: Fashionable neck/breast of lamb turning on end of rotisserie, whole: IN + TARGET (neck/breast of lamb) reversed + (rotisseri)E
  • 1dn: CROMB: Crook with card concealing queen: COMB (card) around R (Regina = queen)
  • 2dn: HORDEUM: Grain crop giving many people starters for usual meals: HORDE (many people) + U(sual) M(eals)
  • 3dn: LUETIC: Clue – it's contrived, pestilential (and unjustifiable): *(CLUE IT)
  • 4dn: RADIO STATION: Broadcaster delivering farewell, on time, within quota: ADIOS + T(ime) inside RATION (quota)
  • 5dn: COSTAE: Company grub served up: bones: CO + EATS reversed
  • 6dn: NAMEDROPPING: Attempt to impress in garden pomp laid out: *(IN GARDEN POMP)
  • 7dn: SERIES: Sequence I'll appear in is drying up: I inside SERES (is drying up)
  • 8dn: LASSA: Source of fever, most serious drugs, first to be avoided: (c)LASS A (most serious drugs)
  • 9dn: INHERES: Section of cabin he restored is natural: cabIN HE REStored
  • 11dn: GENS: Clan left out of highland ghylls: GLENS (ghylls) - L(eft)
  • 17dn: HAIRGEL: A girl he's taken out displaying superior dressing: *(A GIRL HE)
  • 18dn: SURCOAT: Jock's undershirt made by company in coarse cotton: CO(mpany) in SURAT (cotton)
  • 20dn: OMERTA: It stops one informing on crooked mate that's run in: O (on) + *(MATE) around R(un)
  • 21dn: IRRUPT: Italian holding roller aloft to force entry: IT(alian) around PURR (roller) reversed
  • 22dn: OPTIMA: Work period cut short, accepted most favourable conditions: OP (work) + TIM(e) + A(ccepted)
  • 24dn: SHANK: Bad shot was degrading (about fifth from rough): SANK (was degrading) around (roug)H
  • 25dn: TONY: Wally getting award from fine play?: Double definition (ref. Tonys)
  • 27dn: YINCE: St Mirren's formerly given run around by playin' Celtic: plaYIN CEltic
finally a use for my wiki

I've been wondering what to do with the wiki associated with this blog; being able to draft blog entries on the train using ikiwiki is a clear win, but I have too many choices for tools to organize information: my exploding todo list is kept in org-mode, for example.

This week, though, an opportunity presented itself: I was reminded that I'd agreed to present some undergraduate project ideas to our students on the Science without Borders scheme, and of course had totally failed to prepare anything (which, come to think of it, is perhaps not the best advertisement for org-mode – but I think it's my current use of it that's wrong). Luckily, because of SBCL's participation in Google Summer of Code, I already had a useful template, so it was a matter of a few tens of minutes of work to construct a page with a list of student projects that I'd be interested in supervising. (I didn't actually have tens of minutes to spare before I had to give the talk, unfortunately; on the other hand it's quite nice to present without being bound by a fixed agenda of slideware and bullet-points).

I'm resisting (fairly easily, for now) the temptation to use advanced ikiwiki features, having one page per project, each individually tagged and then inlining them in some complicated way – but this might come, as and when students (or anyone else on the lazyweb!) take up these projects and make progress. On the more general point of which tool contains which information, I think it's fairly clear that project ideas which don't have to be kept confidential for whatever reason could usefully be public by default; there's probably no immediate benefit – it's not as if I expect my readership to obsessively check for idea updates – but it might make for a slightly less panicky project pitch presentation.


Many people have their own ways of managing their digital music collection. This is my way.

I am a relatively late convert to medium-less music. I think I might have been a relatively early adopter of digital music – or at least I was actively making purchasing decisions at a time when CDs were generally available and not too much more expensive than cassette tapes. But I've written before about my reluctance to embrace the shiny; I've been resisting digital audio files for longer than mobile phones have been able to play music (not counting Annoying Things).

And yet, there are obvious and clear advantages to having a digital and easily portable collection of music – brought home to me when I was regularly commuting 90 minutes on trains in each direction for work, and my “mp3 player” was a 1TB external hard drive weighing upwards of 1kg. Over a few years I used grip, and then sound-juicer, to convert my physical music collection of CDs to audio files; at the moment, because of something (I don't know what) in the sound-juicer chain causing the generation of broken FLACs, I use gstreamer directly:

for track in $(seq 1 $(cd-discid | cut -d\  -f 2))
  gst-launch-0.10 cdparanoiasrc track=$track paranoia-mode=255 ! \
    flacenc ! filesink location=track$track.flac

One of the nice things that sound-juicer did automatically was to import curated metadata from MusicBrainz; gstreamer doesn't do that itself, so I have had to learn how to use the MusicBrainz tagger picard to add decent-quality metadata to the audio files automatically – at least when it's available at MusicBrainz; perhaps more commonly, given my long-tail predilections, I use picard to use the CD Table of Contents as an initial seed for providing that metadata for the community. (There's a reason why most of my CD recordings of opera are unshifted as yet: the metadata standards for opera on MusicBrainz are fiddly, tedious and error-prone).

Now I have losslessly-compressed CD-quality audio files with high-quality metadata, and all is well. But if there's one thing that working with computers teaches us, it's that if the data is not backed up, it's already lost: you just don't know it yet. Not only that, but it would be good to have access to the music collection wherever I am, ideally without having to carry a “portable” hard drive along with everything else. The good news is that there's a tool for this: git-annex. (I suspect it is not coincidental that I use multiple tools by the same author: git-annex is written by Joey Hess, who is also responsible for ikiwiki). I have git annex repositories on computers an external hard drives both at work and at home, and any additions – or modifications, for example from retagging – can be synchronised across the checkouts. As long as I remember to get the content as well as git-annex's symbolic links, distribution and offsite backup requirements are automatically satisfied, and git annex even stores old copies in case of human error, which is definitely liberating: I have a safety net, so I'm free to try to fly. (Maybe that's over the top for a digital audio tagging workflow...).

Once the audio files are updated everywhere and checked out, the only thing remaining for digital audio utopia is to ask the various media servers to reindex the content. On my GNOME-based desktop, I'm a little bit out of luck at the moment; the bright new hope for music playing is gnome-music, which uses tracker for its indexing, and tracker doesn't currently follow symlinks – and the workaround of checking out an annex in direct mode is like cutting holes in my safety net. For my home music system, I use Logitech Media Server, and there it's as simple as M-x squeeze RET resc TAB RET (and I'd like it to be simpler! Hacking welcome).

To summarize:

  • Workflow:
    1. format-shift from CD audio to FLAC using gstreamer
    2. retag using picard
    3. import into git-annex
    4. sync git-annex repositories
    5. perform git-annex get on
      • the external hard drive (the same one!) which acts as the music source for my Slim Squeezebox Logitech Media Server;
      • the checkout in ~/Music/ on my workstation in the office.
    6. cause various servers to rescan or reindex their Music databases.
  • Current bugs:

All this was brought to mind because I recently made my first purchase of recordings in digital audio file form: I was listening idly to Radio 3, and heard this after the end of the evening concert. My ears tuned in gradually, and then my brain started sending contradictory messages: “it's by Pergolesi” “it's not by Pergolesi” “it's by Stravinksy” “wait what?”. I missed the announcer's statement of what the piece was, and spent a frustrating evening trying to google things like “piano arrangement "se tu m'ami"” and totally failing to find it.

The next day, I belatedly realised why they had played what they did: the following evening's concert was Marc-André Hamelin playing Katchaturian, and again I was listening idly and not really paying attention. But I paid attention to the encore, announced as: “the Minute Waltz... played in seconds!” and again, I thought “wait, what?”. Have a listen....

And, convinced that I wanted to have easy access to more of Hamelin's recorded material, I was pleased to discover that he has recorded for hyperion – which company offers digital downloads in losslessly-encoded DRM-free CD-quality FLAC format. Hooray! I feel like I have joined the 21st century.

inquisitor 1324

An enjoyable puzzle. I didn't know the Gone With the Wind quote, but I had enough letters to guess “childbirth”, and it happened to be the top Google hit for me for “childbirth quote”. As for the other... more on that a bit later.

The three unclued rows contain two part-quotations. A superfluous letter is yielded by the wordplay in each clue. Read in order, these letters complete the first quotation. The surname of the author of the second one should be highlighted in the grid.

  • T 10ac: ACCESSORY: Additional trial involving difficult company cases: not entirely sure about the wordplay here; maybe TRY around *(CO CASES)?
  • H 13ac: ORCA: Tolkein's ogre finally got away from old enclosed fruit garden: ORCHA(t); I think the definition is wrong, sadly.
  • E 14ac: SPOOM: US elks retreating gathering power, scud before the wind: MOOSE (US elks) reversed around P(ower)
  • R 15ac: SENNA: Nurse slipped back laxative drug: SEN (ref. State Enrolled Nurse) + RAN (= slipped)
  • E 17ac: LAPSANG: Variety of tea, unlimited range after falling away: LAPSE (= falling away) + (r)ANG(e); ref. Lapsang souchong
  • S 19ac: LIER: More secretive one who is incumbent: Double Definition (SLIER = more secretive; LIER = one who lies down)
  • N 20ac: LOOK YOU: Welsh observe liberal sex with African fellow: L(iberal) + NOOKY + OU (South African slang for man)
  • E 21ac: EATING: Consumption surprisingly negative, lacking value initially: *(NEGATIVE - V(alue))
  • V 27ac: CACOON: Decrepit van with rotten coco seed: *(VAN COCO)
  • E 29ac: PLAYERS: Actors' appeal not accepted in very long time: PLEA + YEARS - A(ccepted)
  • R 32ac: TRON: Bishop among people who are in Lanark market place: RR (bishop) in TON (dial. people)
  • A 35ac: BUFFOON: Wrongly making fun of Abo comic: *(FUN OF ABO)
  • N 36ac: ALIST: Stalin perversely belonging to top star group: *(STALIN)
  • Y 37ac: ROTOR: Revolving constituent operations research returned Conservative: O(perations) R(esearch) reversed + TORY
  • C 38ac: GAEA: Fixed cage about divinity: *(CAGE) + A(bout)
  • O 39ac: SEPTUPLET: Group of notes curiously spelt out en passant: *(SPELT OUT E(n) P(assant))
  • N 1dn: DAMSEL: Old unmarried woman to censure Scots individual: DAMN (censure) + SEL (Scot. self)
  • V 2dn: ECHELON: Mostly authentic half-speed (loosely) pursued by new ranked arrangement of troops: ECH(t) (authentic) + VELO(city) (speed, loosely) + N(ew)
  • E 3dn: TERNE: Unchangeable old alloy: Double Definition (ETERNE = unchangeable)
  • N 4dn: AS ALSO: Concerning nose bone turned round too: NASAL (concerning nose) + OS (bone) reversed
  • I 5dn: NOGAKU: Drama organised in August briefly OK: *(IN AUG OK)
  • E 6dn: DRIPFED: Supplied food through tube, irregularly predefined, flustered nurse away: *(PREDEFINED - E(nrolled) N(urse)); isn't it great that there are so many kinds of nurse?
  • N 7dn: APPALTI: Italian turns up supporting mate chasing cloth contracts in Rome: NAP (cloth) + PAL (mate) on IT (Italian) reversed
  • T 8dn: EXOGEN: Old plant export not originally convincing: EX(port) + (c)OGENT (convincing)
  • T 9dn: SIMURGH: Rising fog by old city with bodiless ghoulish monstrous bird: MIST reversed + UR (old city) + G(houlis)H; ref. Simurgh. Has everyone read Worm yet? Budget some spare time, then go read it.
  • I 11dn: CON-ROD: Short linking device working within raised type of Greek pillar: ON (working) in DORIC (Greek pillar) reversed
  • M 12dn: NONI: Plant found in overblown omnipresence: overblowN OMNIpresence
  • E 16dn: ANYHOW: Maltreated husband one way at least: *(H ONE WAY)
  • F 18dn: KABAYA: Indeed getting into spirit securing excellent short tunic: AY (indeed) inside KA (spirit) outside FAB (excellent)
  • O 22dn: ACT DROP: Advanced mode of cutting including one's own curtain: A(dvanced) + CROP (cutting) around TOD (own)
  • R 23dn: CONFORM: Adapt grain species: CORN (grain) + FORM (species)
  • A 24dn: INWOVEN: Complicated hotel, hosting without formal greeting: INN (hotel) around WO (without) + AVE (formal greeting)
  • N 25dn: RETIAL: Of a network of nerves connected with light sensitive tissue: Double Definition (RETINAL = regarding the retina)
  • Y 26dn: TRESSEL: Very French cunning accepting English support: TRES (Fr. very) + SLY (cunning) around E(nglish)
  • O 28dn: ARBOUR: Furiously roar about cropped bower of trees: *(ROA(r) ABOU(t))
  • F 29dn: PROMPT: Don associated with politician to primarily help with words: PROF (university don) + MP (politician) + T(o)
  • T 30dn: LENGTH: Stretch temporarily provided great height: LENT (temporarily provided) + GT (great) + H(eight)
  • H 31dn: SHTETL: Jewish village hotel regularly supporting troubled Esth: H(o)T(e)L under *(ESTH)
  • E 33dn: OUTS: Excels, abandoning party, reveals sexual orientation: OUTDOES (excels) - DO (party)
  • M 34dn: ALEPH: Heartless paragraph at end of masculine letter: P(aragrap)H after MALE (masculine)

So the extra words complete the quotation in 1ac and 22ac, which read (after filling in the blanks) “Death and taxes and childbirth”. The bottom line read PR?M?NTH?H?LL, and after a little bit of meandering – “prom on the hill”? – the obvious “pram in the hall” came to mind, and indeed Connolly is visible down the main diagonal.

That gives me an opportunity to mention (relatively safely, since surely no-one reads these crossword blogs) another aspect of my productivity. Like some, I don't think that a pram in the hall is necessarily the ultimate “sombre enemy of good art”, but I think it is true that having children can change priorities: in BC days, it was perfectly possible and reasonably common for me to stay at work late, or work in the evenings or weekends in addition to a more-than-full working week. This is not to say that this doesn't happen at all any more, but it's certainly much less routine, and to an extent the evening and weekend work is now necessary to get my working time up to a full working week, given the need to do school runs or hospital appointments. And one thing that successful work in academia might share with art is its vulnerability to distraction: in academia, prestige and progression come from high-profile research, which both needs focus and tends to be what is achieved after all the other things have been dealt with: teaching, certainly, and any substantial administrative responsibilities. It's perfectly possible to do useful amounts of administration in the 20-minute block of time when a child is entertaining themselves; deep thought, as with programming “flow”, takes time to achieve.

And maybe the moral is that it comes back: in retrospect, it might not have been sensible to move house, start a company and have a child within the space of one year. I suspect there's no good time to do any of those things, let alone all three, and some of the lack of motivation and energy was down to, well, tiredness: but it feels like I am substantially more productive than I was two years ago, and that is a good feeling.

smile and mysociety

It was a pleasure to welcome Matt Green back to Goldsmiths; I had the pleasure of teaching him in his Foundation year way back in 2007, and also for Creative Computing 2 (as was). Matt went off to work in the digital agency world, first with Euan Millar at disturb media, and now at Smile Machine. In retrospect it was perhaps wrong to spend the 5 minutes or so of warm-up act ranting about mobile world congress and mobile phones in general, given how much of a digital agency's time is taken up by making sure that the latest and greatest brand campaign is directly beamed to everyone's brain; this in no way resembles satire from the 1950s, no, not at all.

Still, Matt managed to shrug off the awkwardness, and talked through some of his recent projects – the spirit level was cute – and perhaps more importantly some of the working issues. It was good to hear him talk about the hours: I paraphrase, but something like “sure, some people are there when I get in, and are still there when I leave... but I have a kid at home” is an important message. I think it's possible that since he is a recent(ish) graduate, the current students felt that they had enough of a shared background with Matt to ask him more detailed and personal questions than they have tended to in this series.

The second speaker for the session was Mark Longair of mySociety – in one of those “small world” or possibly “shared background” events, I wrote to mySociety's generic contact address asking whether anyone would be interested in talking to our undergraduates about development and technology work in the third sector, and Mark replied. I don't think we'd actually met before, but we certainly had mutual friends from our undergraduate days (not least Chris Lightfoot), and just as personally, my final undergraduate project was done under the supervision of Malcolm Longair. Small world – how many people with a surname of “Longair” have a CRSid for their Twitter handle?

I liked the slightly subversive slant Mark gave to mySociety's work – or at least the emphasis on speaking truth to power. The Freedom of Information platform WhatDoTheyKnow was perhaps the most obvious of those: by making it really easy to make FoI requests through a platform which stores the requests and responses, and makes them public, it provides an audit trail for later inspection. Also, deploying the WDTK platform in countries with no FoI legislation is a pretty cool hack.

And of course it's not just one site; FixMyStreet is a highly useful way of logging issues, and Mark described the fact that it's effectively a public bugtracker for local councils as having for a side effect increasing the visibility of mundane problems to those in power. I'm not sure that that is the best possible way of relating citizens to local authorities, but I suppose the status quo is already a long way away from the best of all possible worlds, and the FixMyStreet site at least doesn't go out of its way to give Outraged of Anywhere a direct way of venting noisily.

And then over lunch, my student Justin Gagen got to talk to Mark about repurposing sayIt for displaying the librettos for Wagner operas, to help visualise and link results from our research into leitmotif. In an ideal world, this blog entry would announce the successful import and display of the libretto as Akoma Ntoso, but sometimes the real world doesn't work out quite as neatly as it should. Another time.

azed 2179

Pff. Quite a lot of obscurities (to me, at least), and I have to confess to outside assistance for 28dn. Finished in the nick of time to preserve my 100% record (since mid-January, so not really much of a 100% record, but it's important to me).

  • 1ac: CLUMP: Book lost by club member – clot: CLUB - B(ook) + MP (member of parliament)
  • 8ac: SUBER: Cork it's hell getting back, leaving aside opening: (e)REBUS (hell) reversed
  • 12ac: HORA: Dance music, the limit for harmonica: H OR A (limits of harmonic); ref. Hora, which is not in my Chambers (1993)
  • 13ac: INAURATE: A ruin, battered and corroded, rarely gilded: *(A RUIN) + ATE (corroded)
  • 14ac: URNED: Like tea or coffee made for an audience?: sounds like “earned” (made)
  • 16ac: PANTAGRUELION: Magic herb adding puff to a thin liquor, one gracing menu?: PANT (puff) + A GRUEL (thin liquor) + I (one) + ON (on the menu)
  • 17ac: REIF: Pillage of Scots creating touching condition: RE (creating?) + IF (condition)
  • 19ac: MARSALA: Strong sweet wine sadly beak sent back: ALAS (sadly) + RAM (beak)
  • 20ac: CROSSFISH: Rather angry about following asteroid: CROSSISH (rather angry) around F(ollowing); ref. Sea star
  • 23ac: STEALTH: Whisked lattes, hot – secret process: *(LATTES H(ot))
  • 26ac: JANN: You'll find Trojan never imbibing such weakish spirits: troJAN Never
  • 27ac: SELF-ASSURANCE: Fuss as cleaner is sacked? Confidence required: *(FUSS AS CLEANER)
  • 31ac: ALLEY: It's certainly bordered by lines? Yes, or the other way round: YEA (yes) around LL (lines) reversed; double use of “the other way round”
  • 32ac: EPIDURAL: Dial up re being feverish, wanting relief from labour: *(EPIDURAL); this was for a long time this week the only entry I had in the grid, bringing amusement to the veterans of childbirth in the house.
  • 33ac: ZATI: Monkey from South Africa with reverse of appeal: ZA (South Africa) + IT (appeal) reversed
  • 34ac: LENOS: Thin fabrics – love coming into contact?: LENS (as in “contact lens”) around O (love)
  • 35ac: SAROS: Rails over very long time cycle: SORAS (rails) reversed.
  • 1dn: CHAPRASSI: Fellow accompanying prince is brought up as an orderly: CHAP (fellow) + RAS (prince) + IS reversed
  • 2dn: LOMA: Sort of fringe an old pro cut from below: A MOL(l) (old “pro” as in prostitute) reversed
  • 3dn: URANIC: Once celestial queen is found in the heart of such: RANI (queen) in (s)UC(h)
  • 4dn: PARABOLANUS: Church layman tending the sick of old (as S. Paul on being upset about urchin): *(S PAUL ON) around ARAB (urchin)
  • 5dn: DINGUS: Beat with American doodad: DING (beat) + US (American)
  • 6dn: ONER: Bouncer from Jonson, erratic: jonsON ERratic; though ONER is “lie” and bouncer is a “liar”, so, hm.
  • 7dn: PADUA: Where Anthony lived and died, going into shell: PAUA (shell) around D(ied); ref. Anthony of Padua
  • 8dn: SUPERIORESS: Is she unsettling to pure sisters if not TT? *(TO PURE SISTERS - TT) &lit.
  • 9dn: URAL: River suggesting the country separated from its source: (r)URAL (country)
  • 10dn: BANIA: Climbing a hilltop, I planted a fig tree: A NAB (hilltop) reversed around I
  • 11dn: ETIOLIN: Plants denied light display it: see one riverain's holding up: NILOTE (riverain) around I (one) reversed
  • 15dn: ANAMNESIS: One muddled names and lives in recollection of Proust: AN (one) + *(NAMES) + IS (lives)
  • 18dn: EUTERPE: Energy required in scrambling up tree, palm genus: *(UP TREE) around E(nergy)
  • 21dn: FOULLY: Disgustingly posh, boused in e.g. mock ruin: U (posh) in FOLLY (mock ruin)
  • 22dn: HANGAR: Surrogate mother keeping son, so abandoned, in large shed: HAGAR (surrogate mother; ref. Hagar) around SON - SO
  • 24dn: ELSIN: Sort of Scottish punch – single drunk, not good: *(SINGLE - G)
  • 25dn: TSARS: Official adjudicators giving ‘special’ in ratings?: S(pecial) in TARS (ratings, as in “sailors”)
  • 28dn: FADO: Following difficulty striking new driver, golfer displays dejected air: F(ollowing) + ADO (difficulty) / FALDO (ref. Nick Faldo) - L; ref. Fado
  • 29dn: SLAB: Substantial piece of cake, second, shortened party: S(econd) + LAB (ref. Labour Party)
  • 30dn: CITO: Heads for courier in treating order with despatch: C(ourier) I(n) T(reating) O(rder)

I promised a non-trivial example of a use for generalized specializers a while ago. Here it is: automatic handling of HTTP (RFC 2616) Content-Type negotiation in computed responses.

In RESTful services and things of that ilk, a client indicates that it wants to apply a verb (GET, for example) to a particular resource (named by a URN, or possibly identified by a URI). This resource is a conceptual object; it will have zero or more concrete manifestations, and content negotiation provides a way for the client to indicate which of those manifestations it would prefer to receive.

That's all a bit abstract. To take a concrete example, consider the woefully incomplete list of books in my living room at openlibrary. A user operating a graphical web browser to access that resource is presumed to want to retrieve HTML and associated resources, in order to view a shiny representation of the information associated with that resource (a “web page”, if you will). But the human-oriented representation of the information is not the only possible one, and it is common practice in some circles to provide machine-readable representations as well as human-oriented ones, at the same URL; for example, try:

curl -H 'Accept: application/json'

and observe the difference between that and visiting the same URL in a graphical browser.

How does the web server know which representation to send? Well, the example has given away the punchline (if the links above to RFC sections haven't already). The graphical web browser will send an Accept header indicating that it prefers to receive objects with presentational content types – text/html, image/* and so on; the browser I have to hand sends text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 as its Accept header, meaning “please give me text/html or application/xhtml+xml, or failing that application/xml, or failing that anything else”. If the server has more than one representation for a given resource, it can use this client-supplied information to provide the best possible content; if the client has particular requirements – for example, attempting to find machine-readable content for further processing – it can declare this by specifying particular acceptable content-types in its Accept header.

For a resource for which more than one representation exists, then, the server must dispatch between them based on the client Accept header. And this is exactly a non-standard dispatch of the kind I've been discussing. Consider a resource http://foo.example/ which is implemented by sending the return value of the generic function foo back to the client:

(defgeneric foo (request)
  (:generic-function-class accept-generic-function))

The default behaviour is somewhat a matter of taste, but one reasonable choice is that if no content-type matches we should use the defined HTTP status code to indicate that the responses we could generate are not acceptable to the client:

(defmethod foo ((request t))
  (http:406 request))

Maybe we have a couple of presentational representations for the resource:

(defmethod foo ((request (accept "text/plain")))

(defmethod foo ((request (accept "text/html")))
  "<!DOCTYPE html>

And we might have some machine-readable representations:

(defmethod foo ((request (accept "text/turtle")))
  "@prefix foo: <> .
@prefix : <> .
foo:bar foo: : .")

(defmethod foo ((request (accept "application/rdf+xml")))
  "<?xml version=\"1.0\"?>
<rdf:RDF xmlns:rdf=\"\"
  <rdf:Description about=\"\">
      <rdf:Description about=\"\"/>

(I apologize to any fans of XML/RDF if I have mangled that).

Now a graphical web browser sending an accept header of text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 as above will cause the server to send the HTML version, as that is the most specific applicable method to that accept string. Given this, it is perfectly possible to construct specialized clients with alternative preferences expressed in the accept header. A terminal-based client might prioritize text/plain over text/html (though in fact neither w3m nor lynx does that, at least in the versions I have installed). A client for the Semantic Web might instead accept data in serialized RDF, preferring more modern serializations, by sending an accept string of text/turtle,application/rdf+xml;q=0.9. All these clients could each be served the resource in their preferred format.

The case of serving one of a set of alternative files hosted on the filesystem in response to a request with an arbitrary accept string is different; in this case, it doesn't make sense to do the dispatch through methods. If we were to try to do so, it would look something like

(defmethod filesystem-handler (url (content-type (accept "text/html")))
  (let ((prospect (pathname-for-url url "html")))
    (if (probe-file prospect)
        (serve-file prospect "text/html")

but we would need to define one such method per possible mime-type we might want to serve: doable, but unnecessary compared with the alternative strategy of finding all content-types servable for a given url, then choosing the one with the highest score:

(defun filesystem-handler (url accept)
  (do* ((prospects (files-for url) (cdr prospects))
        (mime-types (mapcar #'mime-type prospects) (cdr mime-types))
        result mime-type (q 0))
       ((null prospects) (serve-file result mime-type))
     (when (> (q (car mime-types) accept) q)
       (setf result (car prospects) 
             mime-type (car mime-types) 
             q (q (car mime-types))))))

(the set of files on the filesystem effectively already define a set of methods for a given url; it doesn't make sense to try to mirror that as a set of reified methods on a generic function. Also, I've written this out using do* largely to keep the do*-is-not-that-bad society alive.)

Anyway. There's an interesting detail I've elided so far; not only do response-generating functions have to generate the content they wish to send in the response; they also have to indicate what content-type they are actually sending. Our accept-generic-function already handles dispatching on content-type; can it also take responsibility for setting the content-type of the response?

Why yes! The way to do this is using a method combination; it might look something like this:

(defvar *actual-content-type*)

(defgeneric handle-content-type (request))

(define-method-combination content-negotiation/or ()
  ((around (:around))
   (primary () :required t))
  (:arguments request)
  (labels ((transform/1 (method)
                 (let ((s (car (sb-mop:method-specializers ,method))))
                   (when (typep s 'accept-specializer)
                     (setf *actual-content-type* (media-type s))))
                 (call-method ,method))))
           (transform (primaries)
             (mapcar #'(lambda (x) `(call-method ,(transform/1 x)))
           (wrap (form)
             `(let ((*actual-content-type*))
                  (handle-content-type ,request)))))
    (let ((form (if (rest primary)
                    `(or ,@(transform primary))
                    `(call-method ,(transform/1 (car primary))))))
      (if around
          (wrap `(call-method ,(first around)
                              (,@(rest around) (make-method ,form))))
          (wrap form)))))

This behaves just like the or built-in method-combination, except that when calling a primary method whose specializer for the first argument is one of our accept-specializers, the content-type of the specializer is stored in a special variable; the last thing the effective method does is to call the new handle-content-type generic function, passing it the original generic function's first argument.

Now let's redefine our foo generic function to have the new method combination, and a method on handle-content-type:

(defgeneric foo (request)
  (:generic-function-class accept-generic-function)
  (:method-combination content-negotiation/or))

(defmethod handle-content-type ((x string))
  (format t "Content-Type: ~A~%" *actual-content-type*))

and now, finally, we can try it all out:

SPECIALIZABLE> (foo "text/plain,text/html;q=0.9,*/*;q=0.8")
Content-Type: text/plain

SPECIALIZABLE> (foo "text/turtle,application/rdf+xml;q=0.9")
Content-Type: text/turtle
"@prefix foo: <> .
@prefix : <> .
foo:bar foo: : ."

SPECIALIZABLE> (foo "audio/mp3")

OK, but by what magic do these accept-specializer objects exist and function? I wrote a paper about that, with Jan Moringen and David Lichteblau: as part of my ongoing open access experimentation, the version we submitted to the European Lisp Symposium is viewable at Goldsmiths' e-prints repository or on arXiv. The ELS Chairs have just announced a deadline extension, so there's still time (until March 23) for anyone to submit technical papers or abstracts for tutorials and demonstration sessions: please do, and I hope to see many of my readers there.

postgraduate study and digital art

For the first session after half-term reading week, I invited our postgraduate course leaders to present their postgraduate programmes, in the context of a general discussion about further study as an option. As it happens, most of them were unable to make it; Andy Thomason, from the MSc in Computer Games and Entertainment, was able to come, and he brought along some of the current students. The message that he was delivering was clear (to me at least): if you want to be hired as a games developer, it's the C++ and maths skills that are in greatest demand.

The students have had plenty of chances to hear the message that they should have a portfolio of work to be able to demonstrate to employers what they can do. Andy's comments might have made it a bit clearer that the portfolio isn't just a collection of work: it's a product in itself, and it can be optimized for particular purposes just as any other product. As a showcase to gain employment, it should put the product of the in-demand skills front and centre: in particular, if the skills in demand are technical, such as C++ and maths, then construct examples that demonstrate those skills. Procedurally-generated terrain and realistic fog trump pretty terrain and artistic fog.

And of course I was there, wearing my MSc programme leader hat for our MSc in Computing, and very neatly suffering from a conflict of interest. In the sessions so far, I have been the voice of cynicism, trying to make the students think critically about what they were hearing; now, I was selling further study – worse, “more of the same” study – to the same audience. The good news is that I could claim any outcome as a victory; for the record, the number admitting to considering further study decreased after the talks, though I hesitate to draw a causal link from anything in particular anyone said.

To counterbalance the cynicism, and the sales pitch, I asked Becky Stewart from Codasign and Anti-Alias Labs to talk about some of the things she's been up to recently. She started by adding one possible motivation for postgraduate study: as she put it, it is way easier to go to a foreign country on a student visa than to be allowed to stay for any length of time for work – and she spoke with the voice of bitter experience. Although we in the UK have (at least for the moment) the privilege of being allowed to move about freely within the EU, it's not something that we can necessarily take for granted for very long. Following its recent referendum on immigration, Switzerland is finding that it is no longer benefiting from all the privileges previously afforded to it, and the possibility that either the anti-immigration crowd or the broader anti-EU campaigners get their way in the UK and take us out of associations is unfortunately non-negligible – and travel elsewhere is tricky and can involve significant paperwork.

Becky then showed some of her portfolio: projects she's worked on, complete with well-produced videos; for example, the GPS Shoe. This was particularly apposite, given Andy's talk about portfolios: the video is high-quality, clearly documents the project – and some distance from the “truth” of the matter, because the video was made before the electronics were complete, adding extra meaning to “fixing it in post-production”.

And, analogously, after presenting her other work, participating in art projects and leading workshops in creative electronics (including sewing workshops at technology conferences) Becky calmly told the students that although the art projects are what she talks about, and they're how she markets herself at least some of the time, they're not the projects that pay the bills: she earns money from freelancing in the creative undustries, so that she can work on high-profile art. And so we come full circle: portfolios as showcases. I don't know if Becky will thank me or not to draw attention to the hyperbolic trousers she made for my daughter some years back; maybe the next version needs more LEDs?

azed 2178

A prize puzzle this week, with PARISON the word to clue, meaning a lump of glass, ready to be moulded. Inspiration did not strike. There were a number of errors in the printed version of the crossword, which made some of it more interesting until twitter pointed me at the discussion about it. Clues below are the corrected versions.

  • 1ac: ASTROCOMPASS: Pilot's direction-finder or mascot adjusted before transfer: *(OR MASCOT) + PASS (transfer); I didn't find astrocompass under either astro- or compass in my Chambers, but Wikipedia has it.
  • 10ac: PARISON: A lump of glass for moulding
  • 11ac: OATHS: Curses pipes around start of Hogmanay: OATS (= pipes) around H(ogmanay)
  • 13ac: QUAPAWS: Their womenfolk will appear leaderless, burying father: (s)QUAWS around PA; &lit (ref. Quapaw)
  • 14ac: COOT: Bill's other half cut it – silly: COO (ref. bill and coo) + (i)T
  • 16ac: APOPHYGE: Work extremes of hilarity into edited page – part at top or foot of column: OP (work) + H(ilarit)Y inside *(PAGE)
  • 17ac: GIRNS: Ugly expressions from sailors captured by foreign troops: RN (sailors; ref. Royal Navy) inside GIS (foreign troops; ref. G.I.)
  • 18ac: PILUM: Part of old soldier's kit – this set gets miles put out: *(MILES PUT - SET) = PILUM
  • 20ac: GEOPHAGY: Wildly eggy about a hop going wrong – sign of mania?: *(EGGY) around *(AHOP); ref. Geophagy
  • 23ac: CHARISMA: Personality one follows is weathed in allure: CHARM (= allure) around IS + A
  • 27ac: LAKIN: Little old lady is havin' a work break up north: LAKIN(g) (= dial. play)
  • 28ac: ACTOR: Player in Milan team, rubbish back: AC (ref. A.C. Milan) + ROT reversed
  • 30ac: ANTABUSE: One pill to take as treatment for alcoholism: AN (one) + TAB (pill) + USE (take)
  • 32ac: STAP: Portion of ripest apples and stuff that's gone: ripeST APples
  • 33ac: STINGOS: Tossing off strong drinks: *(TOSSING)
  • 34ac: SITUS: Positions befit me (as editor might put it): SIT (befit) + US (editorial me)
  • 35ac: EERIEST: Extremely weird lake in east of France: ERIE (ref. Lake Erie in EST (= fr. East); this clue in the printed had “Germany” instead of “France”, which was mildly confusing
  • 36ac: CROSSDRESSER: Perry maybe balancing on sideboard: CROSS (= balancing) + DRESSER (= sideboard); ref. Grayson Perry
  • 2dn: SAULIE: What'll make Scots keener? That is following old king? SAUL (ref. Saul) + IE (that is)
  • 3dn: TRAPROCK: E.g. basalt for filling course: PRO (for) inside TRACK (= course)
  • 4dn: RIPEN: Port wine's bottled? Stow to mature thus: ?
  • 5dn: COWPEA: Poet giving away recipe for a legume: COWPER (ref. William Cowper - R(ecipe) + A
  • 6dn: ON SONG: Performing well, Ben maybe loses his first grand: (j)ONSON (ref. Ben Jonson) + G(rand)
  • 7dn: MOOP: The Scots speak indistinctly, low and soft: MOO (lowing sound) + P (piano)
  • 8dn: PACHISI: Brewed cha coming in I drink up – while playing this? *(CHA) in I SIP (= drink) reversed; ref. Pachisi
  • 9dn: SHOGUN: Military governor has shortened broad cannon?: S (has shortened) + HO (slang for sexually loose woman) + GUN (cannon).
  • 12dn: STEMWARE: Mats we're given in arrangement for wine glasses, etc: *(MATS WERE)
  • 15dn: EGG GLASS: Timer with which girl follows say gee-gee: EG (say, as e.g.) + GG (gee-gee) + LASS (= girl)
  • 19dn: LASTAGES: Some duties go on for a long time: Double Definition
  • 21dn: PRIAPUS: Guardian of gardens? Matter holding partner up: SUP (matter) around PAIR (partner) reversed
  • 22dn: SANTIR: Its music is hammered out, a lilting strain: *(STRAIN)
  • 24dn: HOUSED: Put up sort of trough, to employ inside: HOD (= trough) around USE (= employ)
  • 25dn: AUSTER: Visitor to Italy blowing in from the south, e.g. Paul: Double definition; ref. Paul Auster
  • 26dn: MOROSE: Like the Carpenter, or his mate, swallowing first of oysters? MORSE (= Walrus) around O(ysters); ref. The Walrus and the Carpenter
  • 29dn: CANIS: It includes Obaman's ‘Bo’, among Americanisms: ameriCANISms; ref. Bo
  • 31dn: BUSS: Fishing vessel, or smack: Double Definition (smack = kiss)
inquisitor 1322

I like Inquisitors; solving them usually teaches me something. This one was no exception; I did not know that the Brownies were known as the Rosebuds until 1915. In this one, the rubric did not look too scary:

Clashes in a small number of cells must be resolved, and the central square filled, to display a prominent 25 associated a century ago with the name at 6ac. The revised name, adopted the following year, must be entered in the blank space at 40, forming new words downwards. One answer is an acronym.

Not only did it not look too scary, but 25dn was clued, and CITIZEN came out fairly quickly, and when I saw WELLES down the main diagonal, and the final D in 6 across from DOTS, that gave ROSEBUD, and the clashes in the grid could be resolved to give ORSON to precede Welles, and all looks well. But! Citizen Kane was from 1941: by no stretch of the imagination was that a century ago. So, what were the other letters from the clashes? DENP. Hm. Oh, look, the first two letters on the main diagonal are BA – and BADEN-POWELL would indeed fit the preamble. And 6ac clues for ROSEBUD in both contexts. Very nice indeed. (See fifteensquared for more analysis.)

  • 6ac: ROSEBUD: Potential peace maker?: Cryptic Definition; see above

  • 40ac: BROWNIE (unclued)

azed 2177

Not too hard this week; in contrast to last week, this didn't keep me up at night. I enjoyed discovering that “beeswing” was an upper crust (on port, what else?), and that “earlierise” is a verb meaning to make earlier.

  • 1ac: BEESWING: Upper crust gathering accompanied by big-band jazz: BEE (gathering) + SWING (big-band jazz); Beeswing
  • 7ac: OSSA: Bones when in this condition must be retrorse: AS (when) SO (in this condition) reversed
  • 10ac: OPHIOLATER: E.g. moccasins I adore – I loathe going out in old pair: O(ld) P(ai)R around *(I LOATHE)
  • 11ac: FREE-ARM: Weapon without charge, with no support for member: sort-of Double Definition I suppose
  • 13ac: FLOR Pro bottling left in what adds to sherry's flavour: FOR around L(eft); Flor
  • 14ac: MISHNAH: I'm backing king about new legal code: I'M reversed + SHAH (king) around N(ew)
  • 15ac: CISTERN: Rains etc will be wasted without a water storage device: *(RAINS ETC) - A; nice surface reading
  • 16ac: SCENA: Part of opera: you'll see I hum its — haltingly for the musicians: *(SCENA + I HUM ITS) = *(THE MUSICIANS)
  • 17ac: LECHWE: Antelope, female ruminant devouring bits of coarse hay beside lake: L(ake) + EWE (female ruminant) around bits of C(oarse) H(ay); Lechwe
  • 22ac: FRANCS: Foreign money from Norway wrapped in folded sash?: N(orway) inside SCARF reversed
  • 23ac: DIPSO: Compulsive boozer ducks round: DIPS (ducks) + O (round)
  • 24ac: HUMMAUM: Sweaty place, creating pong, one gone round by mother: HUM (creating pong, smelly) + MUM (mother) around A (one)
  • 26ac: PSYLLID: More than half tipsy, Aussie fool, backward sucker?: (ti)PSY + DILL (Aus. fool) reversed; Psyllid
  • 28ac: BARP: Scottish cairn you'll get to from top thus: to = top BAR P
  • 30ac: GONIDIA: Lichen cells active after application of indigo: *(INDIGO) + A(ctive); Gonidium (at time of writing, "This Alga-related article is a stub. You can help Wikipedia by expanding it". So if you're an expert on algae, or lichen, go ahead!)
  • 31ac: LEADARMING: Queer male in drag? It helps to ascertain bed's contours!: *(MALE IN DRAG)
  • 32ac: LITE: Advertising's not so bad fo you, one's learned rate free: LITERATE - RATE
  • 33ac: TEOSINTE: Casse Noisette? It's devoured by herds in e.g. Mexico: *(NOISETTE); Teosinte
  • 1dn: BAFF: Mishit once, reverse of t-triff: F-FAB reversed
  • 2dn: EARLIERISE: Bring forward noble flowers mostly round end of June: EARL (noble) + IRISE(s) (flowers) around (Jun)E
  • 3dn: SPERTHE: Old battle-axe? The woman's saucy inside: SHE around PERT (saucy)
  • 4dn: WHARE: House for Maoris won over race: W(on) over HARE (= race); Whare
  • 5dn: NOMINA: Clan names? One I'm assigned to is coming up: AN (one) + IM + ON (assigned to) reversed
  • 6dn: GLOSSER: Commentator covering bad result in Germany: LOSS (bad result) in GER(many)
  • 7dn: OATH: Part of credo, a theological appeal to heaven: credO A THeological
  • 8dn: STONERN: Rocky's skin lotion kept in tin: TONER (skin lotion) in SN (tin)
  • 9dn: ARCHAISM: Tea's limited in power, what you'd expect from a real oldie: CHAI'S (or CHA IS) in ARM (power)
  • 12dn: MANICURIST: One caring for e.g. nails, crazy over more than half of rustiness dealt with: MANIC (crazy) + *(RUSTI)ness
  • 15dn: CLODPOLL: Dolt? Class rejecting one's messy dollop: CLASS - ASS (dolt) + *(DOLLOP)
  • 18dn: COPYCAT: Pacy wriggling in bed – purr maybe imitated?: *(PACY) inside COT (bed)
  • 19dn: WOOLFAT: Virginia perhaps aiming for something in the ointment: WOOLF + AT (aiming for something); ref. Virginia Woolf
  • 20dn: BAMBINI: Children admitted being bitten by favourite deer? IN inside BAMBI; ref. Bambi
  • 21dn: CHIGRE: Hosted by such, I greet parasite getting under one's skin: suCH I GREet
  • 25dn: MANIS: Sort of cuckoo found in Mississippi: it eats ants: ANI (cuckoo) in MS (Mississippi)
  • 27dn: LODE: Bottom of thick overcoat disappearing in open ditch: LODE(n); ref. Loden cape
  • 29dn: PANE: Quarry group of workmen approach from below: workmEN APproach reversed; ref. Leadlight
eye crossword 515

More in-jokes, smut and topical references than in the last issue; 1ac is particularly self-referential, and should we worry about “try in vain” and “all is lost”? (This issue of Private Eye is also notable for publishing correspondence from a friend of mine – at least, how many Julian Thomases can possibly live in Tonbridge, Kent?)

  • 1ac: UGANDA: Roger somebody or other to discuss this country?: Cryptic Definition; ref. Ugandan relations
  • 5ac: CATCALL: To cry after pisspoor act is a bad thing for luvvies: *(ACT) + CALL (cry)
  • 9ac: OUTCOME: Having revealed your sexual inclination, climax – result!: OUT (revealed sexual inclination) + COME (climax)
  • 10ac: POLITE: Refined sex needs protective support: IT (sex) inside (protected by) POLE (support)
  • 11ac/25ac: ONLY JUST: “Barely lust!” Joy snapped grabbing penis at its centre: *(LUST JOY) around (pe)N(is)
  • 12ac: DOPE PUSHER: One who puts pressure on idiot dealer: Double definition, sort of
  • 13ac: WHITE FLAG: Light a few twists as sign that you've had enough: *(LIGHT A FEW)
  • 16ac/18ac: GOOD GUYS: Slush fund finally takes the piss out of the police: GOO (slush) + (fun)D + GUYS (mocks)
  • 19ac: EDUCATION: Miliband abandoned caution in Gove's unreliable hands?: ED (ref. Ed Miliband) + *(CAUTION); ref. Michael Gove, Secretary of State for Education
  • 21ac: FRONT BENCH: Support beneath ministerial arses: Cryptic Definition
  • 26ac: SPRAWL: Clasp raw legs, centrally spread: claSP RAW Legs
  • 27ac: SUBSIDY: Lunatic busy roping in Tony's old sidekick for public financial support: *(BUSY) around SID (ref. Sid James, or possibly Sid Jenkins)
  • 28ac: DYNASTY: Maybe House of Windsor's extremely dreary, repulsive...: D(rear)Y + NASTY (repulsive)
  • 29ac: TATERS: ...crap Queen Elizabeths and King Edwards?: TAT (crap) + ERS; ref. King Edward.
  • 2dn: GROUNDHOG: Rodent Gordon – ugh! – going ballistic: *(GROUNDHOG)
  • 3dn: NUTTY: Balls-like crackpot: Double Definition
  • 4dn/23dn: A LOAD OF BALLS: Answer: Ed's commitment is claptrap?: A(nswer) + LOAD (commitment) OF BALLS (Ed's; ref. Ed Balls)
  • 5dn/15dn/24dn: CHEAP AND NASTY: Offensively worthless Andy's penis is out – a chat is out: *(ANDYS PEN(is) A CHAT)
  • 6dn: TIP UP: “Turn over” hint on having a stiffy: TIP (hint) + UP (erect)
  • 7dn: ALL IS LOST: Sun is back in one line agenda – prepare for the worst: SOL reversed in A (one) + L(ine) + LIST (agenda)
  • 8dn: LATTE: Recoil at terrorists stocking drink: recoiL AT TErrorists
  • 14dn: TRY IN VAIN: Scoring is futile: fail to reach satisfactory climax: TRY (scoring) IN VAIN (futile)
  • 17dn: DOOMSAYER: Mayor does broadcast: his forecasts are dire: *(MAYOR DOES)
  • 20dn: COHABIT: Live with your partner cook's 50% drug dependency: CO(ok) + HABIT (drug addiction)
  • 22dn: RAPID: End of banker – paid off, pdq: (banke)R + *(PAID)
  • 25dn: JOINT: Common carve-up?: Double Definition
guardian 26190

A fun diversion from Paul, with some nice surfaces (e.g. 12ac, 27ac, 3dn). Not too hard.

  • 9ac: LAGER LOUT: Fancy rogue and tall yob: *(ROGUE TALL)
  • 10ac: HATER: One despising another mad character half-heartedly: HAT(t)ER
  • 11ac: CHICORY: Herb that's trendy with duck wings on rosemary: CHIC (trendy) + O (duck) + R(osemar)Y
  • 12ac: GOOD-BYE: Word used in separation – “a snip”, say?: sounds like “good buy”
  • 13ac: ROOK: Con man: Double Definition
  • 14ac: DETACHMENT: Unit chatted about outstanding soldiers: *(CHATTED) around MEN (soldiers)
  • 15ac: COARSEN: Make roughly a hundred rowers minus a thousand: C + OARSMEN - M
  • 17ac: SARDINE: Fish to eat by river while lying back: R(iver) AS reversed + DINE (eat)
  • 19ac: LIP-READING: University on the border studying speech: LIP (border) + READING (University?)
  • 22ac: SEMI: Half of house missed: houSE MIssed
  • 23ac: NEW YORK: A city doubled in number?: ?
  • 24ac: TEASING: Carol, after a drink, flirting: TEA (drink) + SING (carol)
  • 26ac: E COLI: Company crushed by priest, a sickener: CO(mpany) inside ELI (priest)
  • 27ac: IDRIS ELBA: A risible dickhead floundering as Mandela interpreter: *(A RISIBLE D(ickhead))
  • 1dn: ELECTRIC BLANKET: Exciting total one's charged for heating: ELECTRIC (exciting) BLANKET (total)
  • 2dn: AGRICOLA: Rowling character shaving head and bottom, sparkling stuff in general: hAGRId (ref. Hagrid) + COLA (sparkling stuff); ref. Agricola
  • 3dn: ARVO: A propotion of popular vote for Australian PM: populAR VOte; meaning “afternoon” (p.m.)
  • 4dn: TONY BENN: Popular political veteran an insubstantial figure, according to Spooner: BONY TEN, spoonerized
  • 5dn: STIGMA: Mark, Athenian character shackling Turkish leader: SIGMA around T(urkish)
  • 6dn: THE OTHER: Sex neither this nor that?: Double Definition
  • 7dn: STABLE: Secure horses: Double Definition
  • 8dn: ERNEST HEMINGWAY: Author writing about men – his weren't gay: *(MEN HIS WERENT GAY)
  • 16dn: STEP ON IT: Crush something underfoot – hurry up!: Double Definition
  • 17dn: SANITARY: Hygienic topless bar – madness?: this is tricky: we have (b)AR (topless bar) (in)SANITY (madness)
  • 18dn: INEDIBLE: Poisonous jerks lie in bed: *(LIE IN BED)
  • 20dn: POWWOW: Work having arisen, duck into internet meeting: OP (work) reversed and O (duck) inside WWW; life is probably too short to complain about the conflation of the Web and the Internet in crosswords...
  • 21dn: DIK-DIK: Antelope bringing up a couple of children: KID twice reversed
  • 25dn: ALSO: A capital's musicians on top: A + LSO (ref. LSO)
sbcl release management in the air

Just because I'm attending mobile world congress doesn't mean that everything else stops. It's the end of the month, so it must be time to release sbcl. This month's is a little unsatisfying, because we've only achieved one of the two things I'd hoped for: we've been cautious and conservative after last month's landing of the new register allocator, but we haven't sorted out what's going on to cause the less active architectures to fail to build. There are some workarounds that have been mooted; for one reason and another no-one has had the time to check whether they actually work, and while there's partial progress on identifying the root cause of the build failure on sparc it is only partial.

Nevertheless, minor architectures have been broken before, and no-one particularly benefits from not releasing, so 1.1.16 it is. Actually making the release is a little more challenging than usual: I aim to release by the end of the month, and in practice that means it must be done today, 28th February. However, this is the day that I am returning from Barcelona, so I am less in control of laptop power and connectivity than usual for a release day. And to add to the challenge, I am trying this time to address the valid complaints that the binaries built on my laptop don't actually run on released versions of Linux, thanks to the change in the semantics of memcpy (glibc changed the implementation in its version 2.14 to exploit the freedom given to return undefined results if the source and destination overlap) and the behaviour of the linker and versioned symbols.

So over breakfast I dusted off my squeeze chroot (that doesn't sound dodgy at all), and tried to work out how to get a runnable version of SBCL in there at all (answer: build using clisp and link to the chroot's libc). At lunchtime, I used the café's wireless to check for any last-minute changes, and in the airport I found a power socket, so I started the release build. Of course it didn't finish before the gate opened, and in any case I wasn't going to chance uploading sbcl binaries over the free airport wifi (15 minutes, woo!)

I've performed some release stunts before. SBCL 0.9 was released by William Harold Newman and simultaneously announced by me at the first European Common Lisp Meeting in Amsterdam in 2005. Last year, I released SBCL 1.1.8 “live” as part of a lightning talk at the European Lisp Symposium in Madrid. But I think this is the first time that an SBCL release was even partially effected from the air. Next stop, space?

mobile world congress

I'm in a restaurant! I would obviously prefer blogging from the train, but I was crossing London during the rush hour, so the idea that there would be a seat, or even floor space, was obviously fanciful.

So here I am, in the lovely surroundings of Gatwick North Terminal. Actually, at this time in the evening it's not so bad; there are only nine more flights on the departure board for this evening (I am choosing not to say “nine more flights leaving this evening”, as I have counted chickens before), so my natural misanthropy is not exercised into overdrive: the music is only borderline intolerably loud, and my fellow travellers are not overwhelming.

I'm on my way to Barcelona, for the tail-end of Mobile World Congress. From previous experience, and news reports, the sharp end of the event is the first couple of days; certainly, when I've been before with my company, the majority of the contacts made and business-cards collected have been from meetings on the first couple of days. On the other hand, when I've been before with my company, I have emerged from the week with a skin tone the colour of my suit, a hacking cough that took months to subside, and a deep desire to hide away from everything in a darkened room.

There are a number of things that make MWC unsuited to the introvert, particularly the introvert exhibitor (not that there are any punters at all, I'd expect). The noise, the harsh lighting, the constant need to intercept and interrupt people in order to deliver the 5-second pitch. (The experience for me improved vastly when we hit upon the trick, for a change finding an advantage in our Swiss corporate identity, of offering chocolate to our interlocutors: then if they were totally uninterested in our product, they were up at least one chocolate in the transaction.)

MWC was described in the weekend newspapers as “less glamorous”. Mind you, in the freesheet I read on the way to Gatwick Mark Zuckerberg was described as a “baby-faced bearer of evil”, so clearly you can believe some things you read in the papers. I haven't attended any of the presumably more “glamorous” trade shows (I suppose they mean the likes of CeBIT and CES; certainly mass-media coverage of MWC concentrates on handset announcements and other consumer electronics rather than infrastructure). I wonder though if the “glamour” is one of those dog-whistle words; the first two MWCs I attended certainly had their share of blatant objectification of women: and while I haven't personally experienced it as much in the last two years I suspect that's because I only attend for the last day, and mostly stay within the confines of our stand, and not because the ethos has miraculously changed.

And, of course, I feel odd going to MWC, as I remain stubbornly resistant to the mobile revolution. I don't own a smartphone; I routinely forget to charge my dumbphone; I recognize that if I have an addictive personality, my productivity will not improve if I have an instant dopamine dispenser in my pocket. So along with the the general misanthropy and introversion that goes with exhibiting at a trade show with 80,000 attendees, there's the overall antipathy to the whole show in the first place: why are all these people wasting so much time and effort on shiny? I have made my peace with my segment of the market: communications infrastructure may be “unglamorous” but it is on balance probably a good thing; the rest of the industry, including practically everything consumer-facing, leaves me not just cold, but actively wondering how as a community we have got to this position: prioritizing the matching of coloured jewels over anything meaningful. (I recognize the irony, but at least my addictions generally don't involve the upload of my contacts database to the NSA's command-and-control centre.)

The experience of the show itself was tolerable; my colleagues were jealous of my freshness at the start of the day, and by the end of the day I'd more or less turned the same shade of pale grey as everyone else. I don't do well in loud, fluorescent-lit sensory-overload environments. But I participated in some potential partner and customer meetings, and of course helped in the post-show debriefing session, and of course there's plenty of time at Barcelona airport to finish this blog entry.

workload and industrial relations

On Monday a fortnight ago, I was more than usually embarrassed by the current state of industrial relations in academia: as I was wondering out loud why the lecture room was not in use beforehand, the students present reminded me that there was a two-hour stoppage as part of the UCU and Unison Fair Pay in Education campaign. Given the current lull in the dispute – at present, no further action is scheduled, though a date has been set for a marking boycott – it's perhaps time to take stock.

I am not a union member, and I have not participated in the strikes in this campaign so far; I have had strong (but polite) discussions at picket lines about this. But since it has become an issue, and since I would like to believe that I support the principle of collective action, I think it's important to explain why I am not currently participating, and to identify the things which would need to change in either direction for me to participate.

The first point I'd like to make regards a fundamental disagreement about the premise of the campaign: although the industrial action is ostensibly about both pay and working conditions, in practice all the attention, and the campaign literature's focus, is on money. That would be no bad thing if pay were the strongest part of the unions' (workers') case, but I'm not sure it is: in general, I would say that pay for academics is good, and while it is not so good for support staff in general the related benefits (pension scheme, annual leave) are decent.

The headline case made by the unions is that pay in Higher Education has gone down by 13% in real terms (i.e. after adjusting for inflation) over the last five years. For what it's worth, I believe this figure to be approximately accurate, within the terms that are used to justify it: however, I also think that it is a misleading figure given existing practice in Higher Education institutions. The other main point made by the unions is that the HE sector as a whole made a surplus of £1bn in the 2011-12 financial year; this figure, on which more below, is also accurate as far as I know.

Why do I say that the figure of 13% decrease in pay in real terms is misleading? Well, what has actually happened is that the national pay scales have suffered a 13% decrease over four years, relative to purchasing power. This means that a staff member on spine point 37, say, would be earning 13% less in today's currency today than they would have been in 2009-10. However, that does not mean that a typical staff member who was on spine point 37 in 2009-10 is earning 13% less today than they were in 2009-10, because there are mechanisms for spine point progression, and it's fairly likely that a typical staff member on spine point 37 in 2009-10 is at spine point 40 or 41 now, based on incrementing the spine point yearly, and their pay has therefore stayed approximately constant in inflation-adjusted terms.

In some institutions, including my own, spine point progression is pretty much automatic within a particular grade; the jobs we do (“roles”) are assessed for responsibility and after turning a handle on the magic role-o-tron they are placed within a certain band of spine points, which are then those available to those who do that particular job. A newcomer to the job, absent particular experience, is likely to be offered near the lower end of the spine points for that grade, and will therefore receive automatic spine point increases for a period of five or six years. At institutions where incremental pay rises are not automatic, they are typically dependent on an annual appraisal or review process; in practice, I wonder whether the primary effect of this is to distort personal development, as there are then potential direct financial consequences to a frank discussion about personal and career progression.

Under the current system, there is a problem related to those who have reached the top of their pay grade. The typical hard-nosed capitalist might say that if people have grown as far as they can in a particular role, they should move on, but that is the kind of attitude that leads to engineers being made to manage people because that's the only way their career can “progress”. Universities, like many organizations, can benefit from people who are expert at their job, whether that's as a research assistant or a quality assurance officer, and building pay scale obsolescence into the system for these roles seems counterproductive.

(There are other problems related to pay in the sector: the overuse of hourly-paid teaching from graduate students, and other casual labour; the shameful lack of consideration to maintenance and cleaning staff; the spectacular political own-goal of pay awards to upper management that vastly outweigh personal risk taken. While these are not the focus of the current campaign, the unions deserve credit for drawing attention to them.)

The second plank in the #fairpayinHE campaign is sector surplus; viewed collectively, it does appear to be true that Higher Education as a whole currently makes a £1bn yearly surplus, as presented in HESA reports. £1bn per year sounds like a lot of money. It is a lot of money. But if it were entirely and evenly divided among all the staff in the Higher Education sector, it would allow a one-off pay rise of £2600 (total employer cost: before taxes, NI contributions, pension contributions etc.) – and then that's it; if that £1bn surplus was entirely structural, the pay rise would turn it into £0bn surplus, and there'd be no surplus for next year's pay rise. In practice, an evenly-distributed pay rise (and such a progressive distribution is hilariously unlikely) would cause a number of institutions to start shedding staff (or make other significant decreases in cost or increases in income) – since the surplus is not uniformly distributed over the sector, but national pay scales are, the effect of a uniform pay rise is to turn marginally-viable institutions into unviable ones.

With all that said, the union position does have its merits, particularly in its opposition to the continual erosion of working conditions. The transfer of inflation risk from pension scheme to individuals; the growth of zero-hour contracts in the sector; a particular bugbear for academic staff is the constant expansion of administrative responsibilities. Collective action here could be an obvious win: if career progression depends on indicators associated with unproductive work (what is “unproductive”? related, related), non-cooperation is a reasonable signal (if management's ears are not receptive to merely being informed) – but non-cooperation only works if there's widespread agreement.

This isn't straightforward; the fight against administrative oversight would be a long one, and while unions can contribute to the debate there is legitimate cause for disagreement: one academic's intrusive paperwork burden is one administrator's due process, which is one politician's accountability. But the unions could aim for immediate positive impact on the typical working conditions by campaigning for the use of institutional surpluses to hire people to share the burden. Of course, this is almost diametrically opposed to the use of surpluses for pay rises, but as far as immediate return on investment goes, I suspect it would do much more to improve the typical university worker's quality of working life.

So, a pledge. I will support (legal) industrial action if either: the campaign focuses on using some of the higher-education sector surplus to hire more staff rather than pay rises; or the campaign trades off increments (automatic or PDR-related) for automatic cost-of-living pay increases, along with a lightweight process for increments.

(At present, my own institution's Human Resources pages state that non-unionized staff members are not allowed to take strike action alongside their unionized colleagues. This seems in direct contradiction to the Government's own information on industrial action; I have asked for clarification in this respect from HR.)

sbcl summer of code 2014 participation

I received notification yesterday that SBCL was accepted as a “mentoring organization” to Google's Summer of Code 2014 programme. What this means is that eligible students can think about applying to be mentored by members of the SBCL team to work on a development project – and to be paid a stipend for that work.

The next steps for students are:

  • get involved with the community: join #lisp and #sbcl on the freenode IRC network; read the sbcl-devel mailing list (e.g. at gmane.lisp.steel-bank.devel; have a go at some of the bugs marked as ‘easy’
  • look at the existing list of project ideas, and see if any of those ideas would form a good basis for a project that would sustain your interest for three months or so.
  • read Paul Khuong's essay on getting started with SBCL from last year: it is still absolutely applicable to this year's programme, or indeed getting into SBCL hacking independently.

The period for student applications to be mentored as part of SBCL's participation is from 10th March until 21st March 2014; for best results, get engaged with the community and potential mentors well before the cutoff point.

Last year two projects were funded, and both completed successfully, even if merging them into the mainline is/was slower than one might like. I'm looking forward to seeing what this year brings!

azed 2176

This one took me somewhat longer than last week's – and there's clearly a danger of starting late in the evening when the variability of solving time is still somewhat large; the morning after is substantially worse. Also, this time there were a couple of clues that must be what they are, and yet I really can't see why: 3dn and 5dn were both highly resistant to my analysis. On the other hand I was glad that this time I used the inclusive anagram (in 21ac) to solve the clue, rather than the usual “OK, it must be this, now why?”. (On which note, I'm enjoying @anagramatron – finally a good use of social media. (Related: haiku robot)

Also, I'm out of time to do a full writeup. Interesting clues only, then.

  • 11ac: AMIENS: Shakespearean character (when getting look in): AS (= when) around MIEN (look in); ref. Amiens
  • 20ac: SHIA: Ismailian, he: ‘I am involved with — line’ maybe: *(I AM LINE + SHIA) = *(ISMAILIAN HE)
  • 31ac: TEIL TREE: Support required with litre drunk inside? Better go for lime!: TEE (support) around *(LITRE); not sure about the definition part of this – it's clearly a reference to Linden, but what's the “better go” about?
  • 3dn: SCABIES: Grub dropping of fir trees – it causes severe itching?: ?
  • 5dn: TINGLY: Run off for a moment? It's thrilling: ?
  • 10dn: JACOBETHAN: A round object in black stone, Chinese, showing blend of styles: A + COB (round object) in JET (black stone) + HAN; this one made me laugh when I got it.

[ update: ah, the two I couldn't get were similar in construction! FLEETINGLY - FLEE and SCOFF - OFF + ABIES. I can feel the learning happening right now. ]

reading week and taking stock

Reading week – a chance to breathe? For me, a chance to visit Reading, or rather Winnersh Triangle. But before that: five undergraduate project supervision meetings, two research student supervision meetings, one postgraduate programmes meeting (including imminent deadlines for Annual Programme Reviews and new programme submissions to Programme Scrutiny Subcommittee); one journal launch event; three research group meetings and one visit to a major outsourcing company. Not actually bad – I don't have to run an employability seminar session this week, and some other things are in abeyance too – but not really the breathing space that would be ideal.

What would I use the space for? Well, I'm preparing a submission for the European Lisp Symposium; I would like to write a paper for the Joint Conference on Digital Libraries; I intend to write an abstract for the European Conference on Data Analysis, all with deadlines in the next few weeks. The ECDL submission is in the most danger: most of the work is done for the ELS submission (though there's still plenty of polishing to go), and while it's not ideal, the ECDA abstract can be written in the academic “past-future” tense: by the time the conference comes round, the work will have been done.

But more than these short-term projects, there's also the issue of longer-term projects that are effectively on hold. Some of the point of this blog was to try to find out more about myself and my interests, and that's slightly gone by the wayside: this has been a valuable exercise in documenting day to day activities (and crosswords), but the potential for longer, deeper reflection is as-yet untapped – as evidence, compare the quantity of material on the blog to the equivalent on the wiki. Partly, I think that this is a question of workflow: I haven't got the workflow right yet (and now I have N+1 things to check as part of my routine), but of course sorting out the workflow is also something that needs time to reflect, take stock, and act.

Another longer-term project is developing a comfortable workflow for a more open research process. Some of that is successful; teaser blogs on work around generalized specializers and on audio search got conversations of sorts going (and collaborators to work with!) – I'm not sure whether I would have done better to enable comments, as Athene Donald suggests; the fear there is that all my time would become occupied with comment moderation. In this area, what definitely hasn't worked (yet) is opening up the paper-writing process itself; the link to the gitweb above notwithstanding, I definitely feel uncomfortable sharing the work-in-progress, even with potential co-authors. It might have been easier if I had better tooling around the paper writing process; as it is, while I like org-mode for drafting, I haven't yet got it all connected and working with slime, let alone imagining being able to share that setup with co-authors, and that's a brake on the process – as is a submission style which mandates a different (I can't in fairness say “non-standard”) set of LaTeX headers.

One of the clear wins of this blogging exercise, on the other hand, has been the repurposing of commuting time: the 10-minute slots on trains, if I can get a seat, tend to get used to write, or sometimes minor hacks, rather than wasted reading the advertising- or oligarch-supported freesheets. Maybe the next step is to deal with my current belief that more substantial blocks of time are necessary to make progress on the bigger or trickier questions? At the moment, I feel that it's pointless to try doing something fundamentally novel in those 10 minute slots, because remembering where my thoughts had got to would tend to take up most of that time, leaving little for productive work – it could be that this belief is just wrong, and a period of bootstrapping would get over that.

So, I'll be carrying on, and trying to improve particularly the quality of interaction. At the moment that is still going to be centred on e-mail and physical meetings, but hopefully the future will benefit from the affordances of more immediate interaction styles.

eye crossword 514

A couple of mildly amusing topical clues in this one, but nothing that really made me laugh. Best (though a bit too obvious) was perhaps the reference to the Russian persecution of homosexuals in conjunction with the start of the Sochi Winter Olympic Games in 15ac/24ac.

  • 8ac: ON THE HOUSE: As regards a big part of Westminster, it's much to freebie-seekers’ liking: Double Definition
  • 9ac: EVA: Moon-walking mate of the Führer?: sort-of Double Definition; ref. Eva Braun
  • 10ac: STEP ON IT: Potential nepotist? Don't crawl: *(NEPOTIST)
  • 11ac: IF ONLY: Ffion, abandoned, foot out, extremely lusty? I wish!: *(FFION) - F(oot) + L(ust)Y
  • 12ac: CYPRUS: Miley claims power for the land: CYRUS around P(ower); ref. Miley Cyrus
  • 14ac: RUN SHORT: Come to the end of Trots, Roth's novel: RUNS + *(ROTH)
  • 15ac/24ac: WINTER OLYMPICS: At which village Putin will tolerate no minor encampment (you might say): Cryptic Definition; ref. 2014 Winter Olympics
  • 16ac: FLABBY: Limp old pop group dropping its last member in hurry: ABB(a) in FLY; ref. ABBA
  • 19ac: IN CLOVER: Nick endlessly rattled by sex partner having an easy life: (p)INC(h) + LOVER
  • 21ac: BICEPS: Arm-raisers wanting head from Baby Spice maybe?: B(aby) + *(SPICE)
  • 23ac: STRAIN: Tax only 10% of Stagecoach's rolling stock?: S(tagecoach) + TRAIN (= rolling stock)
  • 27ac: CLEAN SWEEP: Is a scrubber possibly putting on blubber for a complete change: CLEANS (is a scrubber) + WEEP (blubber)
  • 1dn: SNOTTY: Arrogantly stand-offish and rather drippy: Double Definition
  • 3dn: THANKS BE: The banks reforming? Glory to God!: *(THE BANKS)
  • 4dn/26ac: CUSTARD PIE: Silent in-yer-face style of humour?: Cryptic Definition
  • 5dn: RESIGN: Brenda's taken up hint to make way for Brian?: ER (Brenda) reversed + SIGN (= hint)
  • 6dn: XENOPHOBIC: UKIP-like phone box's dodgy electronic component?: *(PHONE BOX) + I(ntegrated) C(ircuit)
  • 7dn: EARL GREY: Peer who liked his drink, eagerly disposed to accommodate Right: *(EAGERLY) around R(ight)
  • 13dn: RINGLEADER: Call Cameron the head of an unprincipled bunch: RING (call) + LEADER (Cameron); ref. David Cameron
  • 15dn: WINGTIPS: Robin's Left and Right extremes?: Cryptic Definition
  • 17dn: LOBBYING: Producing a whopper, holding Bob roughly and applying pressure to member: LYING around *(BOB)
  • 18dn: Bent copper's chief taken to the cleaners: C(opper) + ROOKED (swindled)
  • 20dn: VENICE: City corruption: hoarding English money centrally: VICE around E(nglish) (mo)N(ey)
  • 22dn: PICKET: Take action against a scab? Sounds like it, too!: Double Definition
  • 25dn/2dn: PAWN SHOP: Board members bound for establishment with balls?: PAWNS (chess pieces) + HOP (bound); ref. Medici

Not quite solved at boiled-egg speed, but definitely a breakfast puzzle.

google zoo

For the last session in the employability series in this half term, we welcomed Leon Bayliss and Richard Hale from Google's Zoo, one of two ‘creative agency’ (my words, not theirs) groups within the organization. It was nice to see that the pre-session publicity had worked, and that some of the students from the MA in Computational Arts and the MSc in Computing had chosen to come. This might have slightly discombobulated Richard, whose first question about whether there were “creatives in the audience” met with a surprisingly positive response.

Framing the career narratives as journeys (Richard's and Leon's, and Google's) can give a powerful message; I particularly liked that they included each time where the journey started, in real space as well as with technology. It made for a well-rounded presentation, finishing off with the “moonshot thinking” video video from solveforx, which uses as an example the Polynesian canoeist just setting off into the wide ocean: the students' own view of themselves being about to step out into a new world might not be so different. Of course the students have plenty of examples to learn from – that's the whole point of this workshop series, after all – but that doesn't negate the view that they are about to undergo a transition, a change in their state from pre-career to career. And one of the messages related to that was in big numbers: Richard freely admitted that he'd had a restless career, moving from one place to the other: he said that he seemed settled now at Google, albeit only six months in, but it had taken him 6209 days of his working life to get there.

Some of the projects that were presented were new to me: I hadn't come across the Cadbury's giant party popper, nor #grannysamsung, nor videos displayed, synchronized, across four adjacent phones (“so that you can only see it when you've got your friends round”). The grandmother avatar responsive to tweets reminded me about the #lookup campaign for BA, also displayed on the Picadilly Circus screens, where the digital screen points at BA aeroplanes; I suppose if even I can be affected by campaigns like that, they must work, though it's a bit depressing that this clever advertising is aimed towards brand awareness, as opposed to a more socially useful kind of advertising with the aim of market development – making people aware of products or opportunities that they wouldn't otherwise have heard of.

Nevertheless, the main message from the presentation was clear and worth reiterating: aim high, and as well as the immediate and medium term, it's worth having the idea of a 10- or 20-year project that would achieve something enduring. I'm beginning to try to absorb this lesson myself; if the students get a chance to hear it early, that can only be good.

I was pleased that in the Q&A session, students took the time to ask some questions about other ethical issues. In particular, it was interesting to hear about the philosophy behind Reader, Fiber, and GMail: that actually it was reasonable for Google to shut Reader down, because it was so entrenched that it was inhibiting innovation in that space – and that this was consistent with starting GMail and Google Fibre, where the point was and is to disrupt stagnant, entrenched ecosystems. I'm not sure I buy the story that Glass is merely an innocent attempt to force legislatures to confront technological reality; it seems too easily targeted at a particular demographic, which could be caricatured as one that would have few compunctions over using Glass-like artifacts in exactly the ways which many would find objectionable. ObSF: The Dead Past; I also found Alex Kozinski's perspective on privacy fascinating, and not just because we share an enduring memory of the last line in the Asimov story.

After lunch, Leon and Richard were kind enough to visit the Creative Project labs, where the second-year undergraduates were working on their projects. A good number of the students volunteered to do demos on their works-in-various-progress, and I think the feedback they got was valuable – it definitely sounded enthusiastic. At one point in the conversation, I defended the Americans against accusations of parochialism, saying that the World Series was named because of a newspaper sponsorship deal. Snopes suggests that my generosity of spirit was misplaced; that'll teach me.

azed 2175

Another weekend, another Azed. It's nice that I seem to have got the hang of them; it's a bit of a shame that it's not an augmentation to my weekend's solving: I made absolutely no progress on the inquisitor. Still, there's hope: although solving this one was absolutely not in the boiled-egg ballpark, it was a two-hour wall-clock-time exercise, so as long as I can start my solving earlier than 9:30 on Sunday evening I should be able to do more than one crossword a week.

Analysis below is reconstructed while on an unexpectedly long train journey: away from books, and increasingly tired. I don't understand the cryptic half of 12ac at all. [ update: fifteensquared on this one suggests UR inside AMOS. (ref Amos Oz) ]

  • 2ac: PRIMP: Tidy pages grasping skirt: PP (pages) outside RIM (skirt)
  • 6ac: APANAGE: Dependent territory, hence absorbing one: APAGE around AN
  • 11ac: AIOLI: Dressing; odd bits rejected by Jamie Oliver, half cut: jAmIe + OLI(ver)
  • 12ac: AMOURS: Old-style sex yielding hesitant utterance in Oz?: ?
  • 14ac: CROSSLET: Pendant perhaps left trapped in tight corsets: *(CORSETS) around L(eft)
  • 15ac: KAPELLMEISTER: E.g. Bach, confusedly losing lines, buried in task, i.e. right frantic: *(IN TASK IE R) around PELL-MELL - LL (lines)
  • 17ac: ACTUATE: Tiptop, reverse of tense inside? So get going: ACE around TAUT reversed
  • 19ac: JUBA: Jacket on the short side? You may see me sell this in jumble sale: *(JUMBLE SALE) - ME SELL
  • 20ac: DAMAR: Artist crazy about a type of resin: RA (artist) + MAD all reversed
  • 22ac: COXA: Bone from farm animal, roughly round: OX (farm animal) inside CA (circa)
  • 24ac: OKIMONO: Oriental figurine, fine one, early reproduction: OK (fine) + I + MONO (early reproduction)
  • 28ac: NUMEROLOGICAL: Silly me, uncool with a girl, making prediction from figures: *(ME UNCOOL A GIRL)
  • 31ac: BIRIYANI: Spicy dish from Iran brought in by exotic bird: IRI (Islamic Republic of Iran) in BY + ANI (exotic bird)
  • 32ac: STOLID: Dull time in firm: T(ime) in SOLID
  • 33ac: MELON: Fruit pieces placed round centre of charlottes: MEN around charLOttes
  • 34ac: VANESSA: Virginia bags a Bill perhaps in this novel: VA (Virginia) around A + NESS (bill)
  • 35ac: LEMAN: Celebrity left back, a darling of yore: NAME + L(eft) all reversed
  • 1dn: RACKABONES: Barack O. upset rising senator, scraggy American?: *(BARACK O) + SEN(ator) reversed
  • 2dn: PYRANA: Fish tucking into tropical plant, voracious one
  • 3dn: ROUPY: A bit off colour, consuming nothing, like Scot with a bad throat: ROPY (a bit off colour) around O (nothing)
  • 4dn: MISLED: Major road vehicle with runners taken for a ride: MI (M1 motorway) + SLED (vehicle with runners)
  • 5dn: PULLMAN: Plush saloon diverted up Mall, last in procession: *(UPMALL) + (processio)N
  • 6dn: AHEM: Attention-seeker, one on fringe: A + HEM
  • 7dn: AMRIT: Baptismal ceremony, part in Malayam ritually: malayAM RITually
  • 8dn: NONSUCH: Wayward nun's admitting love – church reveals this once unique thing: *(NUNS) around O (love) + CH(urch)
  • 9dn: AUNT: Decapitating cow, one provides practical advice: dAUNT (cow)
  • 10dn: GREET: Hail ends in spring weather we like best: sprinG weatheR wE likE besT
  • 13dn: STREAMLING: See activated mangle stir a mere trickle: *(MANGLE STIR)
  • 16dn: JUMELLE: Sweet drink (not soft) with honey in – might it to make you see double?: JULEP (sweet drink) - P(iano) outside MEL (honey)
  • 18dn: CAPORAL: Dad looking up and in the pink – strong stuff drawn on?: PA reversed inside CORAL
  • 21dn: REGIME: Administration that's put settler up exchangine final pair: EMIGRÉ with last two letters swapped and all reversed
  • 23dn: XOANON: Primitive statue? Rising pollutant envelops a number: NOX reversed around A NO (number)
  • 25dn: KURTA: Dressed in clean one? Keralan tunic possibly: *(KERALAN TUNIC) = *(IN CLEAN + KURTA)
  • 26dn: ORRIS: Upholsterer's trim – it smells of violets : Double Definition
  • 27dn: SCALA: Ear part: coiled canals may reveal one, note: *(CANALS) - N(ote)
  • 29dn: MOON: Satellite? Computer device shows use with this: MOUSE (computer device) ON (shows?) - USE
  • 30dn: LIPA: Some Croatian money a twit laid up but without lock: A PILLOCK - LOCK reversed.
tmus mini-projects workshop

I'm on a train!

Not the logical train that I was booked on, thanks to the excitement caused by high winds and speed restrictions. But ticket restrictions having been lifted, it seemed prudent to get on to the first arriving train out of Lancaster, coincidentally delayed by just the right amount of time for it to leave at my train's scheduled time, rather than waiting for the West Coast Main Line to close due to the “extremely severe” weather.

I did in the end finish my talk (previously previously), and delivered it if not to rapturous applause, then at least applause. I think I pitched it about right, in that even the music informaticians will have most likely seen something new in there, but it was a talk with sufficient illustrations (less charitably, short on detail) that the non-technologists in the audience will have absorbed a certain amount of the flavour of possibility.

Slides are here for posterity; they'll also go up on the transforming musicology website, as will a video of my burblings to accompany the visuals. In case it's not clear from the slides: the survey is inevitably partial (in both senses); I was perhaps most disappointed in not being able to give a vast list of resources for subsequent investigation, so if any reader feels slighted by the lack of publicity given to their (available) music informatics tool, let me know. It was slightly disconcerting to be viewed as an authority in this context; what with my various startup and child-rearing activities, it's been a while since I've been fully engaged in the the music informatics world. I suppose that is something that having a part-share in a £200,000 funding budget brings – I must be an expert, I'm tangentially involved in the distribution of money!

Meanwhile, having agonized over the talk so comprehensively (I think the version I ended up giving was my third attempt at a coherent story, the others having foundered on the rocks of unavailability or sunk into the whirlpool of incomprehensibility) I had forgotten that I was also chairing a discussion session on tools for audio analysis. Fortunately, there were some interesting and knowledgable parties in the room, and we had a lively discussion, covering source separation, features for emotion classification (we discussed the problem of interpretability, and the Netflix prize), and the failure of music informatics tools to deal with singing in almost all contexts.

I hope that there is some work done on improving the situation for singing in general, and for non-autotuned music in general – I'm not sure that Transforming Musicology is the place for that work, given that it's not a project for technology development, though one of the things I hope to do myself is to discover how far a straightforward application of the state-of-the-art can get you. It was a pleasure to be able to discuss Polina Proutskova's work on establishing a foundation for inference of aspects of vocal production from audio recordings, and I may have committed myself to listening to an unbounded number of Chinese opera fragments...

[ update: the train manager has just announced “overhead line damage at Crewe, a tree on the line at Stafford – we'll do our best to get you to London... today...” ]

[ update update: the train is stationary. Quite stationary. In Warrington Bank Quay station. (The clue is in the name.) ]

[ update³: the train will be moving towards Crewe, as the tree is on the other line. “What happens at Crewe, I have no idea” says the train manager. Answers on a postcard. ]

[ update⁴: currently stopped not quite at Crewe. Still have no idea what happens if we get to Crewe. (Nor what happens if we don't.) ]

[ update⁵: we still don't know what happens once we get to Crewe, beyond that we are moving down the track. It's quite exciting; a voyage into the unknown! ]

[ update⁶: we have just stopped outside one of our extra bonus stops (5 between Crewe and London). Apparently we are delayed because of a fire caused by a lightning strike. ]

[ update⁷: apparently the fire and the lightning strike were separate events. Lightning caused signal failures outside Stafford; Rugby station was closed due to fire. Exciting. ]

[ update⁸: we are now delayed just outside Watford Junction, because the fast train lines have closed for engineering works. Engineering works. Yes. Sorely needed. ]

[ update⁹: it would be tempting fate to claim that all is well, but we are now within the tube network. The roundel has never seemed so welcoming. ]

[ update¹⁰: awesome final stop announcement.
“In a few moments, we will probably be arriving at London Euston, where this train terminates. We apologize for the late arrival of this service, which is due to... where do we start? There were speed restrictions on the line from Glasgow; two fires, one at Crewe, one at Rugby; several trees on the line, we hit one of them; signal failures caused by lightning in the Stafford area; and engineering works between Milton Keynes and Euston. We will shortly arrive at Euston, with just over five hours' delay. Thank you for travelling on Virgin Trains” ]

still writing mini-projects workshop talk

I'm on a train!

Specifically, I'm on a train, still writing my talk on Tools for Music Informatics (previously). It's to be delivered to a mixed audience, including technologists already embedded in the Music Informatics area, and musicologists who have no idea what it is but have heard that there might be some money attached if they can spin an idea the right way. This mixed audience has led to the talk having something of an identity crisis, anthropomorphically speaking: should it be focussed on tools that have been used to perform musicological tasks? Should it be focussed on workflows that are simple to relate to? In the end, constrained by the fact that my slot is 25 minutes long at most, I have gone for surveying tools that are commonly used in the Music Informatics world, and biasing my survey towards those which provide visuals that can be used in a talk.

It turns out that the constraint on having example pictures is quite a strong one. I appreciate that academic researchers are not necessarily oriented towards powerpoint-style marketing; for many, and generally I include myself in this category, the ideas and results stand on their own merits. But in this area of software as output, where there's the possibility of scaling one's impact, moving beyond the set of people one can talk to, to anyone who can discover and use the software, why is the user-facing documentation so poor.

Well, one reason is that creating good documentation is hard, and not necessarily a skill that goes hand in hand with academic research or software development, so in the lone wolf model of academia the chances of finding someone who can do all three are low. But the fact that there are several software packages in Music Informatics, all doing approximately the same thing and all lamentably documented, suggests that there might other reasons, and one of the benefits of being on a train and needing to procrastinate away from finishing the talk is that I get to discuss reasons for this with my captive colleagues.

So, here's a cynical view. There are two sets of competing incentives that, together, might explain why there are a fair number of underdocumented PhD-level software frameworks, and why there aren't substantial communities formed around the further development of such software. The first relates to the principal developer of the software, assuming that they are still in academia – if they're not, then there's probably no incentive to do anything more with the software at all. In the academic case, the career ladder and prestige indicators are constructed such that the credit for new things is vastly greater than the credit for improving already-existing things. In particular, once a permanent academic position has come along, a potentially demanding userbase is actively detrimental, taking time away from the real business of academics, which is to write funding applications to be able to hire researchers to do research and write papers on which you gain author credit. At that point, the software might still be a help to generate the research results, but the most valuable users are likely to be in close physical proximity with the lead developer, and so they can learn directly from the horse's mouth.

The second incentive applies to the principal source of uncommitted labour in the system: the graduate student. A graduate student, in theory, could be doing anything with their time; in practice, they are guided in their activities by a supervisor, who suggests ideas and projects and has as a goal the successful construction of a dissertation showing substantial work at PhD level. As with the previous case, the student gains more credit from new things than improving existing things, and the contribution of the student to independent work will be clearer for the supervisors and the doctoral examiners than the contribution to a collaboration, particularly a dynamic collaboration such as in software development and maintenance. So an otherwise uncommitted supervisor will likely recommend to a student interested in software development for music informatics to develop their own software rather than invest time and effort in one of the existing ones: the outcome for the student's dissertation is likely to be better, as an independent implementation can make up a chapter or more, while participating in a development community is most likely not to be usable in a dissertation at all.

After all the cynicism, what's the solution? Changing the academic prestige system is clearly a long-term prospect, and likely to run into game theory buffers; while it's nice to imagine a world where community contribution is as valued as independent “discovery”, I wouldn't hold my breath. But this, it seems to me, is where agitating for more reproducible research is important; not only does it lower the barrier to participating in the scientific process, it also provides built-in tutorial material in a real research context for any and all software that is used in the research itself. Initiatives such as conferences awarding prizes for reproducible research, or journals mandating that reproducibility materials be deposited alongside the manuscripts, has the potential to lead to an improvement in reuse of software as well as in proving the research results themselves.

200 unread music-ir messages

I have to give a talk on Wednesday about tools for music informatics, with a particular slant towards things that might be useful for and usable by musicologists.

Conveniently, I've been burying my head in the sand allowing a backlog of unread messages on the music-ir (Information Retrieval) mailing list to accumulate, naggingly, in my *Group* buffer. So this evening I went through 200 or so of them to see whether there are any announcements of software, publications of data sets, discussion of tools, or similar.

Firstly, a high-level categorization of messages by topic: some of course fit in multiple categories. I have made no attempt to be rigorous about this; it is the fruit of a Sunday evening's scan.

  • research discussion (including surveys): 51
  • research meetings (calls for papers / participation): 52
  • open positions: 52
    • MSc: 4
    • PhD: 24
    • Post-doctoral Researcher: 21
    • Fellowship: 2
    • Intern: 3
    • real job: 2
  • community housekeeping: 20
  • datasets: 36
  • software announcements / discussion: 14

Much of the discussion about datasets was about the relative lack of them, or issues about their accessibility. However, four datasets were announced (or re-announced); they were:

Meanwhile, a number of feature extractors or feature extractor frameworks were mentioned; I picked up on big beasts such as Marsyas, jMIR and aubio, along with announcements for individual feature extractors such as segmenter-vamp-plugin, beatroot-vamp and pyin. In the discussion of the lack of data, Sonic Annotator Web Application was mentioned as a way of bringing analysis tools to audio data rather than the reverse; there was also a genuinely new announcement, about the open-sourcing of Essentia, a multi-extractor toolbox developed at the Music Technology Group at UPF Barcelona.

The other software in the period included two audio source separation projects: FASST, a toolbox, and ISSE, an interactive editor; possibly related (I haven't really tried it). There was also: a melody annotator and editor, tony, which looks interesting (and the project is doing open development of code and publications); a score follower, antescofo, which is entirely behind an IRCAM paywall, so I can't say anything more about it; a library for computing melodic similarity, and three mobile apps: a game, a game with a purpose, and a smart microphone widget.

All this is worth knowing, and it was a good exercise: if nothing else, my *Group* buffer looks less intimidating now. Whether it gets me much closer to a talk about music informatics tools for musicologists is another matter..

winkball and ig

This was the first of the proper sessions in the employability workshop series. I can pontificate about careers; the Careers service can inform about careers; but there's nothing quite like having people who actually have careers talk about their careers, or ventures, or just relate their experience of the Outside World.

It was a pleasure to welcome James Ohene-Djan back, to talk about the history of and latest news from Winkball. I think I could take or leave the Winkball story itself, though several students commented to me afterwards that they were particularly glad that they got to hear a story of failure – or maybe success at the wrong thing? After all, Winkball did scale; it got big, at least for the right metric – it just never got to the point where investors believed that it would be big enough to justify funding that growth.

James did offer some important insights from his experience of growing Winkball, which might have been lost on the students: one point was on finding out when and how to pass on things for someone else (or some other company) to do; another was the mistake of not taking people's names for the later “Google yourself” automatic marketing factor. But I think there were some really inspiring messages, too: firstly, the fact that they're pivoting to a business-to-business product with a subscription model, providing a targetted video advertising and marketing platform; not that I am particularly fascinated by video advertising, but the real drive to build something that works is something to behold.

But secondly, and this was something that I really didn't know beforehand: when asked by a student what he really thought Winkball had achieved, James revealed that the technology, which had originally been motivated to provide a platform for the deaf and hard-of-hearing, did survive in that realm: he pointed to, a Winkball analogue tailored for the deaf, and the commercial arm aiming to be a business producing marketing materials. Good luck to them.

The second presentation of the session, given by Sion Elias and Ed Worthy of IG, was perhaps a more conventional undergraduate career presentation. The serendipitous advertising of IG's graduate scheme, with a particular focus on front-end developer skills, gave me the opportunity to invite them over – which neatly ticked the “larger employer” and “financial services” boxes. Sion's career trajectory was interesting, and superposing his timeline on a graph showing the FTSE 100 share index between 1997 and the present couldn't help but be entertaining.

Among Sion's points was that not all graduate training schemes are the same: I think he was at pains to contrast IG with more hidebound competitors for talent, in particular the merchant banks, where the divison of work areas into front-/middle-/back-office silos is very established. The emphasis in the presentation on development methodology (agile, XP and pair-programming) was interesting; are the details of the corporate development methodology still so distinctive that they differentiate in the competition for talent? I can believe, as Sion said, that they save at least some money by using these processes to have fewer bugs in production (and hence less downtime, lower churn, and so on); as a signal in a graduate recruitment pitch, though, I'm less convinced. But maybe I'm underestimating student discrimination of development practices?

I was a little surprised at the relative lack of questions from the students; maybe the two-hour session is a little intense? There were some good ones, and the rump of students who stayed at the end to discuss pivoting and lean startup methodology had a good conversation, but to me it felt a bit like a missed opportunity – or maybe the financial services industry still has too many negative associations for the students among the hotbed of idealism that is Goldsmiths?

The two presentations, along with their respective Q&A sessions, ran exactly to time: that was good, but that meant that I didn't really have any time to wrap up or go more deeply into some of the issues raised – unlike last year, where I had an explicit recap session built-in after lunch. Given the chance, what I might have pointed out was that both talks, from substantially different perspectives, really highlighted how little of a viable business, even a viable technology business, was the technology itself; yes, it's important that the technology is there, and works, and is cutting-edge, but that alone does not a viable business make. The other theme in common between the talks was the constant need for learning, about technology, about business and about everything else.

nnir vs gnus-bonus-el

I read my mail through Gnus, an emacs-based news (Usenet, remember that?) and mail reader. (The main reason I do so is documented at Dan Barlow's diary, from an era when I routinely sent out patches by e-mail, and frequently sent out messages saying "please see the attached" while failing to attach anything to the message. Thmutt.)

In the intervening years, I've been accumulating mail accounts, responsibilities, and I've often been frustrated by my inability to search through non-local mail archives. It turns out that the reason I've been unable to do so is largely my own fault: Kai Großjohann wrote nnir.el as an extension to Gnus, and Debian packagers helpfully included that in the emacs-goodies / gnus-bonus-el packages. In the meantime, the Gnus developers quite rightly integrated nnir.el into their distribution, augmenting and extending it over the years, and in particular giving it functionality to swap out search backends depending on the mail server's capabilities.

Unfortunately, the 2003-era version of nnir.el installed by gnus-bonus-el was loaded in preference to that provided by Gnus itself, in an otherwise fairly standard Debian installation of emacs. It was only when I was looking at the source to see how much work it would be to make multiple search backends possible (not too much but more than an afternoon; it's lucky I don't charge by the hour) that I noticed the date in the copyright header, and then that I had three copies of nnir.el installed... come to think of it, this is possibly a sound argument for keeping the dates in the copyright headers of source files up to date.

So, now, I can actually search (separately) my various personal and professional mail accounts. I don't think I can do terribly intelligent combined searches, and of course I still have to contend with Office365's dubious implementation of IMAP. But still, not a bad result.

Ironically, when I went to report all this upstream, I discovered that just last week gnus-bonus-el had been removed from Debian, with this particular issue having been reported three years ago.

azed 2174

On looking at this on Sunday, I almost gave up without trying. “Printer's Devilry”?! Where do you even start?

Well, after thinking about the basic approach to clues (find the bit in them that looks even weirder than the rest), and doing my level best not to think of any of the clue as a definition... I started with

As soon as the teacher's goon is finished (9)

The weirdest bit in that is definitely “goon”. As soon as the teacher's go, as soon as the teacher's goNE aha. As soon as the teacher's goNE, THE LESSon is finished. Well that makes sense. And Chambers (1993) is happy that NETHELESS is a word, so off we go. Last to fall was 24dn (DALILA), which was the only word in the dictionary; only after its entry did I find ameDAL I LAnded.

  • 1ac: Anyone who's awASTER IS Making too little of his life
  • 6ac: Some early Huguenot insurgents were known as cAMISards; ref. Camisard
  • 10ac: In such, a new developMENT A LISTed building cannot be modified
  • 11ac: Did guilt-ridden Paris cry, ‘Pleas forgive, heLEN I ENTreat’?
  • 13ac: I always enjoy my mEAL Eating heartily
  • 14ac: CHARET: Archaic horse-drawn vehicle
  • 15ac: As soon as the teacher's goNE THE LESSon is finished
  • 16ac: Devoted to the cap he wears? His beRET IN All weathers
  • 18ac: A well-known liAR TAking? Few for a ride
  • 20ac: He likes to think he is a good saxophonist but he isnO BECHE't
  • 22ac: After a rouch voyage there's many a boSUN DRInking in the quayside inn!
  • 25ac: There are many for whoM A NEAT INGle fireplace – really sets off a cosy room
  • 28ac: A skilful drover's soon coaxed into rUN A BLEating flock
  • 29ac: Increased taxation, waS LIMiting for businesses eager to grow
  • 30ac: ‘I really fancy a soldier,’ she said, ‘It's NO TAR I NEed!’
  • 31ac: TriBAL LAD IS Teste don? His ability to survive alone in the jungle!
  • 32ac: Single scroungers are perhaps forced to bEG ALone
  • 33ac: Venues like this: PLACE RAN Top shows for months on end
  • 1dn: The new mini model is too small to give my lanky PAL LEGROom
  • 2dn: We should make lots in renT ON LETting our second property
  • 3dn: Bare hillsides are not tREED ENough to prevent soil erosion
  • 4dn: Mums will take drastic action if their kids gaIN NITs at school
  • 5dn: The main character in the filM AS HERe portrayed differs a lot from the book
  • 6dn: Cruise passengers often like towALK A LINer's deck
  • 7dn: Was your entry perMIT REceived in time to ensure access
  • 8dn: Not many points were scored – juST A Try unconverted
  • 9dn: Police announcing new anti-crime drive – won't pLEASE BANDits?
  • 12dn: In the foREST RAIN Tre, espied wagtails (rarely spotted); “trees” used as verb
  • 17dn: Pearl fishers may be fine for some but I prefer a nICE BALLet; ref. Pearl Fishers
  • 19dn: Will the efforts of ACAS prevent the rALIMEN Taking action?; ref. ACAS
  • 21dn: He's the man, she swears – she sHALL ALways love
  • 23dn: Over AdUNE A SEa, Holly grows wild; ref. sea holly
  • 24dn: Despite commendation for my race performance, it wasn't ameDAL I LAnded
  • 26dn: Noble bearing ensures AmaN A BLAzon
  • 27dn: BasIC TICket prices won't get you into the favoured spots
  • 28dn: Park rangers tagging wildlife only aim to stUN BEars

I think I'd previously been reluctant to engage with Printer's Devilry crosswords before, because they felt qualitatively different from regular cryptic crosswords, and in particular the lack of a definition put me off. But on mature reflection I can see the analogy, and it seems to me that going about solving a PD puzzle depends on the same kinds of ability to handle words as sequences of letters and words as meaningful entities. Let's say I'm open-minded towards them.

My attempt at a clue for 14ac: I come top – I turn to greatness / I come to pitCH A RETurn to greatness. Which is not great, but it at least follows the rules.

Three in a row!

eye crossword 513

More straightforward than usual, perhaps? Certainly I started it on the plane from Zürich (after finishing my reading of a draft International Programmes subject guide on Advanced Graphics and Animation), and finished it on the bus between City and Plaistow, which speaks either to the level of the crossword or to the relative lack of distractions in those environments.

  • 7ac: WASHED UP: Took part in after-dinner activity – knackered: Double Definition
  • 9ac/10ac: PARTY ANIMAL: Good-time Johnny, Labour's ‘Beast of Bolsover’?: Double Definition
  • 11ac: NARCOSIS: In a cross-dressed sort of stupor: *(INACROSS)
  • 12ac: GREENFLY: Rose's downfall, inexpert trouser opening: GREEN (= inexpert) + FLY (trouser opening)
  • 14ac: LIB-LAB: “Past alliance – Balls at the centre!” Cyclops squeals: (ba)L(ls) + I BLAB (Cyclops squeals)
  • 16ac/27ac: POLITICAL CLOUT: What Clegg hopes the coalition will give him? Iodine-impregnated fancy lilac top and sock combination: *(I LILAC TOP) + CLOUT (= sock)
  • 20ac: SPOT ON: As Page 3 girls would have to make a comeback? Exactly: NO TOPS reversed
  • 22ac: SCRUBBER: Boris's ultimate carbon johnny cleaner: (bori)S + C(arbon) + RUBBER (= johnny)
  • 23ac: IN A PADDY: Rice to be seen here throwing a wobbly: Double Definition
  • 26ac: ON SALE: Noel's pissed about uprising of his band's nos. 2 and 3 – yours for a fee: *(NOEL) around (o)AS(is) reversed (ref. Noel Gallagher, Oasis))
  • 28ac: RUNNER UP: Not the top choice, ‘Bean Out of Bed’: RUNNER (= bean) + UP (= out of bed)
  • 1dn: LAUNDRY: Steamy place, Los Angeles, not banning alcohol?: L(os) A(ngeles) + UN-DRY
  • 2dn: CHAMBER POT: Where parliamentarians may be seen with drug dumping receptacle: CHAMBER (ref. Chambers of parliament) + POT (marijuana)
  • 3dn: SPUN: Proprietor primarily involved in tabloid cover-up is put in a good light: P(roprietor) in SUN
  • 4dn: SPIRAL: Is lap dancing about right to screw?: *(ISLAP) around R(ight)
  • 5dn: BRIO: Eye rather grandly on the up, absorbs Independent – dash!: ORB reversed around I(ndependent)
  • 6dn: CYNICAL: Disenchanted Clegg not quite squeezing into lacy creation: NIC(k) (ref. Nick Clegg) inside *(LACY)
  • 8dn: DOLEFUL: Wretched as an unemployed person's existence?: Double Definition (ref. Dole)
  • 13dn: YETIS: Old rock band keeping it up – hairy beasts: YES (ref. Yes) around IT reversed
  • 15dn: BALL BUSTER: Bullet bra's elastic is a real demoraliser of a bloke: *(BULLET BRAS)
  • 17dn: CARTOON: Strip, possibly showing oxygen-inflated ‘package’: O(xygen) inside CARTON (= package)
  • 18dn: SPINACH: Sailor man's muscle builder: orgasming in chaps: *(IN CHAPS) (ref. Popeye)
  • 19dn: BELLY UP: Bishop deployed pulley to achieve female missionary position: B + *(PULLEY)
  • 21dn: NUDITY: No cover-up with this state: Complete Definition
  • 24dn: PLOT: Plan to apply pressure to lashings: P(ressure) + LOT (?= lashings)
  • 25dn: YARN: Strand: any cruising to include river: *(ANY) outside R(iver)
generic function precompilation

Once upon a time, I wrote two blog entries about precompiling the dispatch for generic functions in order to improve the responsiveness at application startup, and obviously planned a third but didn't get around to writing it. Some time later, I asked the lazyweb about it, and unfortunately this time didn't get anything much back. On further reflection, I think I was going to discuss a couple of additional technical points.

First of all, there's the question about having this kind of precompilation on all the time; wouldn't it be nice if generic functions were always at their most efficient, ready to be called? Well, yes, but what we actually want is for functions to be at their most efficient when we want to call them, and we don't particularly care at other times. Why draw that distinction? Well, when we write code that uses generic functions, it is usually of the form:

(defgeneric foo (x y))
(defmethod foo ((x t) (y t)) ...)
(defmethod foo ((x integer) (y integer)) ...)
(defmethod foo ((x rational) (y float)) ...)

a sequence of method definitions; sometimes contiguous, as above; often distributed among various files. loading such code will execute the defmethod forms in series. And each of those executions will call add-method, changing the generic function's methods, so any aggressive dispatch precompilation scheme will kick in at every defmethod. That would be bad enough, being O(N) in the number of methods, but it's actually worse than that: the amount of work involved in precompilation will tend to be O(N) in the number of methods, or in the number of concrete classes applicable to those methods, so there's O(N2) or worse behaviour from having precompilation active all the time. SBCL is routinely mocked for having slow-loading FASL files (originally, FASL probably stood for FASt Load; we don't advertise that much); expensive and poorly-scaling computations at load-time probably won't win us any extra friends. (Though having written that, I now wonder if recording changed generic functions using the dependent-update protocol, and hooking load or asdf:load-system to precompile after loading a whole file or system, might be a tasteful compromise).

Secondly, there's the detail of exactly how to pre-fill the cache for a given generic function (given that its specializers are mostly or entirely classes). The simplest strategy, of filling the cache with exactly those classes used as specializers, fails as soon as abstract or protocol classes are used to organize the implementation of the generic function. The next simplest, which is to fill the cache with all possible subclasses of specializers at all argument positions, well, just reading that should set alarm bells ringing – it might not be too problematic for generic functions with one argument, but the cross-product effect of multiple arguments will probably cause the implementation to collapse under its own weight into a virtual black hole. It might be that the best approach is to allow the user to specify an exact set of classes: and to allow the user to introspect a running system to find out which class combinations have actually been seen and hence cached in a given session.

In fact, SBCL itself depends on a particular, specific version of this. The generic function print-object is used for printing almost everything; even without user specialization, it is called whenever a structure-object or standard-object needs to be output to any stream. Users can write methods on it, though, and that then invalidates any previous precomputed dispatch information. But there are times when it's really important that print-object just work, without any kind of extra computation to go on: in particular, when the debugger needs to inform the user that the Lisp is running out of storage space (heap or stack), it's pretty important that that reporting can happen without using more heap or stack than necessary. So for print-object, we never reset the cache state to empty; it is always prefilled with entries for control-stack-exhausted, binding-stack-exhausted, alien-stack-exhausted, heap-exhausted-error and restart for its first argument. (This strategy only works if there are no specializations of the second argument to print-object anywhere in the system; there's no good reason for a user to try to specialize the second argument in any case, so we emit a warning if they do that and hope that they read and learn from it.)

azed 2173

OK, when I said “that never happens”, I clearly didn't expect this.

  • 2ac: GLASNOSTIC: Signal cost ‘varied’ in the spirit of openness: *(SIGNALCOST); ref. Glasnost
  • 10ac: GOOROO: Schmaltz on jumper spelling ‘revered old teacher’: GOO (= schmaltz) + ROO (= kangaroo); variant of “guru”
  • 11ac: CORNEA: Cereal crop alongside river – it's regularly grafted: CORN + EA (= river); ref. Corneal grafting
  • 13ac: GORGON: What abstainers drink, reverting? It's appallingly ugly: NO GROG reversed
  • 14ac: QUAINT: Stop holding ‘amateur’ as meaning ‘lovably odd’: QUINT (organ stop) around A(mateur)
  • 15ac: SPIE: Not the actor? That is what Marlowe doubled as possibly: THESP (= actor) - THE + IE; ref. Christophe Marlowe
  • 16ac: PURPOSE: Turned up poser solved, the desired goal: UP reversed + *(POSER)
  • 20ac: EITHER: Solvent? I admitted it may introduce an alternative: ETHER (solvent) around I
  • 21ac: RINSES: Light bathes north in upward slopes: N(orth) in RISES
  • 22ac: NATANT: Swimming in Mediterranean at Antibes: mediterraneaN AT ANTibes
  • 24ac: ENTRAP: Snare with associate (all but red) made to backtrack: PARTNERED - RED
  • 28ac: DIAPASE: Old melody to fade with a dance coming in: DIE (= fade) around PAS (dance)
  • 30ac: GLEN: Depression, often extended, a feature of singleness: sinGLENess
  • 31ac: IMPARL: I unfolded map with both hands to hold a confab: I + *(MAP) + RL (both hands)
  • 32ac: SHALLI: Look at these delicate fibres – do you think I should?: Double definition
  • 33ac: CAPRID: E.g. angora bonnet, free?: CAP (= bonnet) + RID (= free); ref. Angora
  • 34ac: SAPELE: Fine wood from Peru got in auction: PE(ru) in SALE (= auction)
  • 35ac: METACENTRE: Mozart's first entr'acte composed containing English: a point of intersection: M(ozart) + *(ENTRACTE) around E(nglish): ref. Metacentre
  • 1dn: EGGS BENEDICT: Binges with ged etc cooked in tasty snack: *(BINGES GED ETC)
  • 2dn: GOOPY: Energy starting work with bit of yolk, glutinous: GO (= energy) + OP(us) + Y(olk)
  • 3dn: LORIOT: Pretty bird, one yet to pass test with love and loose living: L (ref. L-plate) + O (love) + RIOT; var. oriole
  • 4dn: SOOK: Aussie wimp, and accordingly not particularly good: SO (accordingly) + OK (not particularly good)
  • 5dn: NONPARTY: Parton active in New York, unattached politically: N(ew) Y(ork) outside *(PARTON)
  • 6dn: SOURCING: Finding origin of crab-apple? It includes Cuba: SOURING outside C(uba)
  • 7dn: TRAPUNTO: Type of quilting made of horsecloth until obsolete: TRAP (horsecloth) + UNTO (obs. until)
  • 8dn: INION: Lackey has head clipped causing lump thereon: (m)INION
  • 9dn: WATER SPANIEL: Gundog that's reliant with paws galloping round heart of fen?: *(RELIANT PAWS) around fEn
  • 12dn: ENSUE: To succeed it must be removed fom a bathroom: EN-SUITE - IT
  • 17dn: CHAMPART: Amateur page in carriage, what feudal lord was entitled to: HAM (amateur) + P(age) in CART (carriage)
  • 18dn: DENTARIA: Parasitic plant strangely radiant about end of tree: *(RADIANT) around treE
  • 19dn: BRETESSE: Gallery helping the beseiged – network's replacing one in foundation: BASSE (= foundation) with A (one) replaced by RETE (= network)
  • 23dn: ANIMA: A good mimic uplifted the feminine in a man: A MINA (var. myna, starling capable of imitations) reversed.
  • 25dn: ROLLER: Canary, one singing, about to fly off: CAROLLER - CA (= circa, about)
  • 26dn: NAPPE: Chump losing last sheet of sums: NAPPER (= head) - R
  • 27dn: SELLE: E.g. palfrey's saddle, partly tasselled: tasSELLEd
  • 29dn: KHAN: Dirk (Persian) pulling jar in eastern inn: KHANJAR (Persian dagger) - JAR

So there we go: two in a row.

guardian 21666

A cute puzzle by Otterden; the rubric reads:

Each of the four sides of the completed grid contains three proper nouns of a kind, not further defined

which added to the frisson when I started attacking it.

  • 9ac: KID GLOVES: Soft digital covers Boy George adores: KID + G(eorge) + LOVES
  • 10ac: TOTEM: Emblem of a form of betting – the first of many: TOTE (ref. The Tote) + M(any)
  • 11ac: SCANNER: Comes across critic after opening of show: CANNER (= critic) after S(how)
  • 12ac: RECLAIM: Recall miracle play: *(MIRACLE)
  • 13ac: SOU: Resources show very little money: reSOUrces
  • 14ac: FORESEE: Anticipate front view: FORE (= front) + SEE (= view)
  • 17ac: MISSION: Avoid one on delegation: MISS + I + ON
  • 19ac: ARTICLE: Credential unfortunately leaving out end item: *(CREDENTIAL - END)
  • 22ac: HENPECK: Continually criticise bird weight: HEN (bird) + PECK (? imperial unit of volume)
  • 24ac: NEY: Say no to old marshal: ref. Michel Ney
  • 25ac: ANISEED: A need is to provide liquid flavouring: *(ANEEDIS)
  • 26ac: PICKAXE: Choose to dispense with tool: PICK (= choose) + AXE (= dispense with)
  • 28ac: RAVEN: Hungrily devour bird: Double Definition
  • 29ac: SWEETENER: Come on in – coffee perhaps?: another Double Definition, and a good one; “Come on” and “in coffee perhaps”
  • 2dn: INDIA: Country favoured aid redistribution: IN (= favoured) + *(AID)
  • 3dn: UPLANDS: Good periods getting to secure higher areas: UPS (= good periods) around LAND (= secure)
  • 4dn: OBVERSE: Heads face up to old boy's poetry: O(ld) B(oy) + VERSE
  • 5dn: LUSTRUM: Crave spirit after five years: LUST + RUM
  • 6dn: SHTICKS: Supports include hospital attention-seeking devices: STICKS around H(ospital)
  • 7dn: OSTRACISE: Review riot cases ban: *(RIOTCASES)
  • 15dn: RETRIEVER: Trier somehow always one to get things back: *(TRIER) + EVER (= always)
  • 16dn: EEL: Devious one up to be a general: LEE reversed
  • 18dn: ICE: Reserve kitchen regularly: kItChEn
  • 20dn: CLEAN UP: Reduce mess list in vessel: LEAN (= list) in CUP (vessel)
  • 21dn: END USER: Final recipient endures being out of order: *(ENDURES)
  • 22dn: HYPHENS: They join up with egg producers who are after detailed promotional material: HENS (egg producers) after HYP(e); it took me a long time to work out that detailed was the indicator for de-tailing “hype”
  • 23dn: NECKTIE: Cheek to marry in formal wear: NECK (cheek) + TIE (marry)
  • 27dn: ANNUL: Cancel annual outing away: ANNUAL - A(way)

I guessed the perimeter theme fairly early on, but for some reason was working on LONDON/LISBON rather than BERLIN/TEHRAN for the bottom right-hand corner for rather too long.

inquisitor 1317

My word. I buy the Independent newspaper on Saturdays, largely out of the habit formed having discovered as a student that the crossword (then in the magazine supplement, not at that point called the Inquisitor) was achievable. I had a pretty good hit rate at the time; my (partial) recollection is that two weeks in three I would have a decent stab at it, often in Saturday morning maths lectures. (To any of my then lecturers I offer an apology, at the very least for my lack of discretion; any of my students coming across this should not use my behaviour then as an example now.)

One of my prouder moments was not only completing a crossword, but getting round to sending it in and receiving a dictionary through the post a couple of weeks later – not without some puzzlement, since at the time the prize advertised was a bottle of champagne or olive oil. Sadly the online archives of the crossword don't go back far enough for me to link to any evidence, but some of the puzzlement went away when I saw my name under lights; specifically, listed as a winner under crossword number 100 (which would make the one I won crossword number 97, though beware the renumberings in Inquisitor space – maybe it was really number 346).

In the (many) intervening years, I have continued to enjoy attempting crosswords; I eventually ‘discovered’ Araucaria), occasionally have a crack at Azed (never with success, I hasten to add), and generally enjoy my fortnightly dose of smut in Eye Crossword. I retain a particular soft spot for the themed crosswords; there's something satisfying about solving a puzzle which has taken considerable, conspicuous artistry to construct; what I generally lack is the time to do justice to an attempt at them (attack some clues, sit down with reference books, regret that the dictionary I won – Chambers 1993 – is no longer up to the minute).

I was therefore surprised to find that Inquisitor 1317 (Who's the Daddy by Ecologue, published Saturday 18th January 2014) yielded to a mildly sustained attempt over just one day: I think I was lucky to have solved as some of my early clues the ones interesecting with the theme light to give W.....MB.... as the first author: leaping to William Blake wasn't too hard, and since the corrected letters from clues towards the end fit with “symmetry”, I was fairly confident I had a match. The rest of the clues fell fairly straightforwardly; they're a lot easier to solve when you're not trying to guess the misprint, though even then I spent a fair amount of time trying to justify “orisons” (16ac) and “joe” (8dn) rather than “posions” and “jet”. I hadn't actually come across John Steinbeck's Burning Bright; The Tyger is apparently a sufficiently common source of title fodder for there to be a list of F&SF works with titles derived from it, but some more careful searching eventually turned up the Steinbeck.

Clue analysis:

  • 1ac: BEFRIENDED: Made buddlies with fine red drunk during sex: *(FINERED) in BED (sex);
  • 10ac: APEPSY: It may cause wilnd primate mind to be half abstracted: APE + PSY(che)
  • 12ac: POILU: In Paris soldant's two Hawaiian dishes chucked to the French: POI + LU(au)
  • 13ac: TATI: His Gallic thumour was evident from agitation: agiTATIon (Jacques Tati)
  • 14ac: LIMN: Initially lietter intricately majestic notation (initials; & lit.)
  • 15ac: ARGOT: Low-linfe language to work in deception: GO in ART
  • 16ac: GASES: Perhaps proisons have all sections behind government letter: A (all) SES (sections) behind G (government letter)?
  • 18ac: MORPHEW: Possibly wrash hack after easy romp: *(ROMP) + HEW (hack) = spots
  • 19ac: TAFT: Army on foot in former prieses according to Jock: TA (Territorial Army) + FT (foot) = William Howard Taft (preses = Scot. president)
  • 20ac: BOER: Neighbour around East farming wsort: BOR around E(ast).
  • 21ac: RETRO: Dazted before casualty comes round – about time!: OR (before) + ER (Emergency Room = casualty department) around T(ime)
  • 24ac: AFRAID: Fellow taking part in an attack. WShy?: F(ellow) in A RAID
  • 28ac: STATUE: It's cut from ruock, say, in astute turnover: *(ASTUTE)
  • 31ac: ETENS: Large bfellows set about round nut: *(SET) around EN = nut (printing)
  • 32ac: Martch for his queen ensconced with Home Secretary: H(ome) S(ecretary) around ER = queen
  • 33ac: NORN: In the early hours measure is halved for one of Tihree: ? = Norse analogue of Greek Fate
  • 34ac: POTSHOT: Bags stolen in random firme: POTS (= bags) + HOT (= stolen)
  • 37ac: EDGES: Thank God in butcher's returning fridnges: DG (Deo Gratias = thanks be to God) in SEE (= butcher's hook: look)
  • 40ac: EVITA: Raice lyrically wrote this part of The Return of the Native: in nATIVE reversed; Tim Rice wrote the book for the musical Evita.
  • 41ac: TRUE: HGood's time to regret: T(ime) + RUE
  • 42ac: ETAS: JHumble folk to stuff round: SATE reversed (lower-class Japanese menial workers)
  • 43ac: RENEW: Flying wren with energy to exptend: *(WREN + E(nergy))
  • 44ac: ERRANT: Engaged in pcrime tirade following hesitation: ER + RANT
  • 45ac: POPLINETTE: Daddy's clear of position being outside caorded stuff: POP + NETT (clear of) in LIE (position)
  • 1dn: BATATA: Tiuber essence cheers: BA (soul in ancient Egypt) + TATA (bye-bye)
  • 2dn: FETA: Product of minlk loudly chewed up: F + ATE reversed (= Feta cheese)
  • 3dn: ISLE: Lantd in water lives – headless eel rises: IS (= lives) + EL reversed
  • 4dn: NIMMER: Check up French gentlemen within litfter: MM (fr. Messieurs) + REIN (check) reversed (= thief)
  • 5dn: DONOR: A fellow has gold givenr: DON (university Fellow) + OR (gold)
  • 6dn: DORP: Poke e.g. with stick up isolated aldera: PROD reversed (= isolated village; aldea = Sp. village/hamlet)
  • 7dn: NIGHTMAN: BMinder of bricks, say, close to Mother and Nature at first: NIGH (= close) + MA + N(ature)
  • 8dn: SLOE: Perhaps joet seven down it's said: S (mediaeval Roman numeral 7) + LOE (homonym for low = down)
  • 9dn: OUTWORE: Dissolute roue has a couple lashted longer: *(ROUE) outside TWO
  • 11dn: PISTOL: Schooter on track mostly over line: PIST(e) + O(ver) + L(ine)
  • 16dn: GALIOT: Galledy by riot girl first without right: GAL + (r)IOT (var. galliot = small galley)
  • 17dn: SOFA: Might it provide basic compfort from the South of America: S(outh) + OF + A(merica)
  • 20dn: BAITS: Lurkes a bit suspiciously at top of shop: *(A BIT) + S(hop)
  • 21dn: RIBES: Curreants found in Carib estuary: caRIB EStuary (= saxifrage currant)
  • 22dn: RAKERS: They cleanr lawns, say, right old pain to scientists: R(ight) + AKE (obs. = ache) + RS (Royal Society)
  • 23dn: SLEETING: Having wintry pfall settling unsteadily over the east with temperature dropping: *(SETTLING + E(ast)) - T(emperature)
  • 25dn: WEEPERS: Those who blaubber: footballer's moving south: SWEEPER (= footballing position) with S(outh) moved
  • 26dn: MURE: Will's to walkl up with energy after a rise in spirit: RUM reversed + E(energy)
  • 27dn: LINGER: Palss in tedium in sector of Berlin, Germany: berLIN GERmany
  • 29dn: TETRYL: BYellow explosive let off during test? On the contrary: TRY in *(LET)
  • 30dn: SNASTE: RMiddle of candle that's burnt tin brewing teas: SN (tin) + *(TEAS) (= wick)
  • 32dn: HOT UP: Get wardmer excited on horseback
  • 35dn: OVER: Donae Trevor holds up: tREVOr reversed
  • 36dn: STEP: DTread note in special case: TE (leading note) in SP (special case?)
  • 38dn: DERN: Antiquarian danrk, perhaps, underneath clothes: unDERNeath (= archaic secret)
  • 39dn: ETAT: Loyon's rank is revoked in statement: sTATEment reversed (= Fr. rank)

and once all that's in, replacing WILLIAM BLAKE with JOHN STEINBECK allows highlighting of JOE SAUL and MORDEEN as well as FRIEND ED, leaving VICTOR as the Daddy of the title.

And what is it about synchronicity of obscure words? I had a go at the Sunday 19th January Azed, and what is 21dn? “Old cargo boat: flap about slick when it's capsized?” – OIL in TAG reversed, or GALIOT.

extended specializer responses

I got a few responses to my call for use cases for generalized specializers; I'll try to summarize them in this post. Before I do, it's perhaps worth noting that the work on generalized specializers that Jim Newton and I did was written up at about the same time as Charlotte Herzeel, Jorge Vallejos, Theo D'Hondt and Pascal Costanza's work on filtered dispatch, and so our Custom Specializers in Object-Oriented Lisp paper doesn't refer to it. I'll talk more about filtered dispatch in a later post, when I tie all these threads together, but for those who aren't aware of it it's worth looking at, along with ContextL for dispatch that varies based on dynamic program state.

Meanwhile, some other ideas for extended specializers: James Anderson suggested duck typing, and I think working along similar lines Patrick Stein thought of specializers that could dispatch on the dimensions of array arguments (or length of sequence arguments). That feels to me as though there should be a fairly compelling use case, but the toy example of

(defmethod cross-product ((x (length 3)) (y (length 3)))

doesn't generalize. I suppose there might be an example of selecting particular numerical algorithms based on overall matrix dimensions or properties, something like

(defmethod invert ((x (total-size> 1000000000)))
  ... invert big matrix ...)
(defmethod invert ((x (total-size<= 1000000000)))
  ... invert small matrix ...)

but it doesn't feel concrete yet.

On the other hand, just about everyone's first response to the question (Jan Moringen, and the crowded wisdom of #lisp IRC) was pattern specializers, usually through regular expressions or optima in particular; one example concrete use case given was to dispatch among handlers for urls. This is a very interesting case; I first considered this in the early days of extended specializers, and indeed mop-27.impure.lisp from the SBCL test suite captures one other use case, in a (toy) simplifier for arithmetic expressions, redolent of implementations of symbolic differentiation from a bygone age. At the time, I took that example no further, both because I didn't have a real use case, and also because I didn't fancy writing a full-strength pattern matching engine; now that optima exists, it's probably worth revisiting the example.

In particular, it's worth considering how to handle capturing of pattern variables. To continue with the toy simplifier, the existing dispatch and specializer implementation would allow one to write

(defmethod simplify1 ((p (pattern (+ x 0))))
  (cadr p))

but what would be really neat and convenient would be to be able to write

(defmethod simplify1 ((p (pattern (+ x 0))))

and in order to do that, we need to intercede in the generation of the actual method function in order to add bindings for pattern variables (which, helpfully, optima can tell us about).

The awkwardness in terms of the protocol – beyond the problems with metaprogramming at compile-time in the first place – is that make-method-lambda operates in ignorance of the specializers that will be applied to the method. In fact, this was the root cause of a recently-reported SBCL bug: SBCL itself needs to communicate the name and method lambda list from defmethod to make-method-lambda, and has no other way of doing that than special variables, which weren't being cleared to defend against nested calls to make-method-lambda through code-walking and macroexpansion.

But make-method-lambda is the function which generates the method body; it is exactly the function that we need to extend or override in order to do the desired automatic matching. So how can we expose this to the metaprogrammer? We can't change the signature of make-method-lambda; it might seem obvious to extend it by adding &optional arguments, but that changes the signature of the generic function and would break backwards compatibility, as methods must accept the same number of optional arguments as their generic function does; similarly, adding keyword arguments to the generic function doesn't work.

We could export and document the special variables that we ourselves use to propagate the information; in some ways that's cleanest, and has the virtue of at least having been tested in the wild. On the other hand, it feels unusual, at least in the context of the metaobject protocol; usually, everything of interest is an argument to a protocol function. So, we could define a new protocol function, and have make-method-lambda default to trampolining to it (say, make-method-lambda-with-specializer-specifiers; the basic idea, though not the horrible name, is due to Jan Moringen). But we'd need to be careful in doing that; there is some implementation-specific logic deep in PCL that performs extra optimizations to methods (the fast-method calling convention) if the system detects that only the standardized methods on make-method-lambda are applicable to a particular call; if we add any extra metaobject protocol functions in this area, we'll need to make sure that we correctly update any internal logic based on detecting the absence of metaprogrammer overrides or extensions...

career planning and networking

Helen Kempster from the Goldsmiths careers service came to talk about Career Planning and Networking – a full two-hour session, or maybe two one-hour sessions joined on to each other. The starting point for the Career Planning was the theoretical framework by Bill Law, a cycle of Self-awareness, Opportunity Awareness, Decision Making and Transition Learning. It was not too surprising to me to discover that very few students self-identified as still being at the self-awareness stage; in some ways it's natural for students doing their final projects and with one eye on the next thing being more interested in opportunities or decision-making, but given my own current issues around career Angst and (not) knowing what I want to do with my life, I felt the irony. But then, maybe that justifies the model in some small way: it is a cycle.

It was predictably difficult also to convince some of the students that the skills in biggest demand, even for Computing-related jobs, were generic (“transferrable”) skills; communication skills, team-working, time-management and so on. Yes, understanding of algorithmic analysis can be a competitive advantage, but just running the numbers (on workers in Computing-related fields vs graduates of Computer Science programmes) suggests that if specialist knowledge is required in a given role, there's at least a chance that it can be acquired on the job. This does raise the question of why we're even bothering teaching “industry-friendly” languages; there's probably some kind of economic argument around information asymmetry (to do with attractiveness to students, rather than good outcomes for employers) but this is probably not the place to analyse it; every time I get a bit disheartened about our syllabus, I remind myself that one possible Mission of the University is to help people to live well.

The exercise for people to self-rate their level of transferrable skills was interesting, in particular the difficulty that they had with providing evidence for their ratings. There's general lack of experience, but there's also the lack of experience of thinking of their whole life as a source for evidence of skills – paid work is not the only source of evidence for CV bullet points. I also amused myself (after the students had provided ratings) by telling them about the Dunning-Kruger effect.

The networking session was perhaps a bit more abbreviated, and we went into less depth: though we did talk a bit about whether Computing practitioners are particularly vulnerable to the lone wolf fallacy, because of their general ability to go from idea to prototype on their own. There was a nice moment when Helen mentioned Robert Kiyosaki's Rich Dad, Poor Dad, and one of the students volunteered that he'd gone bankrupt again.

azed 2172

I just finished an Azed.

That never happens. Ever. Generally I don't have the time, but even when I do have the time I don't have the vocabulary. But this time I mostly did. (Too late to send it in, sadly). I've noticed in the past some clue or answer synchronicity, where relatively obscure vocabulary shows up in several weekend crosswords at once; maybe this blog will help me understand whether it's down to coincidence or conspiracy; as well as sharing GALIOT, this and inquisitor 1317 both refer to ETAT meaning rank (in 8dn here).

  • 1ac: GARIBALDI: Blouse revealing girl's front, a loose one: G(irl) + A + RIBALD (= loose) + I
  • 12ac: COUPE: What it contains I disposed of – nothing amiss therein!: tricky, and the last clue I solved (just filling in the P). A coupe is a short-stemmed shallow glass dish, usually containing ice cream. So ICE (= what it contains) - I with O (nothing) + UP (amiss) inside
  • 13ac: AUTOCUE: A worked out before Q? It's a help for public speaker: A + *(OUT) + CUE
  • 14ac: AD MANUM: Chap embraced by Maud, excited and ready: MAN inside *(MAUD)
  • 15ac: ANURA: Amphibians inhabiting Caspian (Uralian): caspiAN URAlian (= order of amphibians including frogs and toads)
  • 17ac: TEAMSTER: The pet husband twice got rid of – he'll move animals along: THE HAMSTER (pet) - HH
  • 18ac: PILOW: Spicy dish, mixed type, of little value: PI (ramdomized printer's type) + LOW; variant of pilaf
  • 19ac: PETAR: Device for making explosive entrance is scrapped in Pirates misguidedly: *(PIRATES) - IS
  • 20ac: OPEN MARRIAGES: Male included, unconventional pairs agree on ... such?: *(PAIRS AGREE ON) outside M(ale); & lit.
  • 22ac: CREEL: Soak to soften line – it's used to catch lobsters: CREE (= soak to soften) + L(ine)
  • 24ac: STADE: Sports venue given support by the sound of it: homonym for “stayed”
  • 26ac: AROMATIC: Do a basmati rice? A bed is so fragrant, curried: *(DO A BASMATI RICE) = *(A BED IS AROMATIC)
  • 29ac: NAVEW: Root vegetable not available, and one not in prospect: N(ot) A(vailable) + VIEW (= prospect) - I; I so wanted this to be “navet” for old-time's sake
  • 30ac: NAUTICS: Anicut's adapted for water sports: *(ANICUTS)
  • 31ac: TREAGUE: Turbulent age contained by genuine cessation of war once: *(AGE) inside TRUE
  • 32ac: ATOKE: Return of ancient character accepting satisfactory part with no sex: OK inside ETA reversed
  • 33ac: INDAGATOR: One of yore searching out form of gradation: *(GRADATION)
  • 2dn: AO DAI: Women's tunic, excellent as wrap for concubine's pad: AI (A1 = excellent) outside ODA (room in harem)
  • 3dn: RUBMBLE: Grass on twig?: Double Definition
  • 4dn: BENT: Grass leaning: Double Definition
  • 5dn: ABUTTAL: An objection over pillar going up, one of two on which arch rests?: A + BUT + LAT (= pillar) reversed
  • 6dn: LAMEBRAINED: The crazed in bedlam are?: *(IN BEDLAM ARE) & lit.
  • 7dn: DUMA: Elected council identically trimmed boring shopping area: DU(ll) (= boring) + MA(ll) (identically trimmed)
  • 8dn: CONSTATE: Study ranks from below (pronounce in imitation of the French): CON + ETATS reversed
  • 9dn: SCUTA: Did they make a tortoise run away in South America: CUT (= run away) in S(outh) A(merica)
  • 10dn: CURER: Physician putting practice as of old in credit: URE (= [obs.] practice) in CR(edit)
  • 11dn: HEARTSEASE: Flower transformed seat in vehicle...: *(SEAT) in HEARSE
  • 12dn: CAMPO SANTO: ... Going to this, morning certain to be enveloped in melody: AM (= morning) + POS (= certain) in CANTO (= melody); it took me for ever to work out that the definition depended on the HEARSE from 11dn to indicate cemetery
  • 16dn: BONEMEAL: Fertilizer that has been mixed with loam: *(BEEN + LOAM)
  • 19dn: PRECAVA: Superior vessel in summary is missing narcotic drink: PREC(is) (summary) + AVA (drink from pepper plants); the answer brings back memories of Fantastic Voyage, which expanded my eight-year-old vocabulary considerably
  • 21dn: GALIOT: Old cargo boat: flap about slick when it's capsized: TAG (= flap) around OIL (= slick) reversed
  • 22dn: CRARE: Trader responsible for oversight, run in: CARE around R(un); another kind of ship
  • 23dn: ROVER: End of career past, he's ready to peg out!: careeR + OVER (= past); ref Croquet
  • 25dn: DECKO: Garland ring, look: DECK (= garland) + O (= ring)
  • 27dn: TOUN: Barrel with nothing in? You'll find Scotch aplenty here!: TUN (= barrel) around O (nothing)
  • 28dn: QUAG: Bog that shifts as ground on all sides falling away: QUA (= as) + GROUND - ROUND

The Fifteensquared analysis is pleasingly similar. I must be on the right track.

more prototypes with multiple dispatch

I received a couple of points in feedback regarding protoypes with multiple dispatch. Firstly, from Pascal Costanza, the observation that there's already a prototype-based dispatch system present in the quasi-standard CL MOP; the key is to view all the prototype objects as singletons, and to use introspection to manage delegation and the class-prototype as the actual object. Thus (any mistakes are mine, not Pascal's):

(defun clone (o)
  (let* ((scs (sb-mop:class-direct-superclasses (class-of o)))
         (new (make-instance 'standard-class :direct-superclasses scs)))
    (sb-mop:finalize-inheritance new)
    (sb-mop:class-prototype new)))

(defun add-delegation (o d)
  (let ((scs (sb-mop:class-direct-superclasses (class-of o))))
    (reinitialize-instance (class-of o) 
                           :direct-superclasses (list* (class-of d) scs))))

I don't think that this works in quite the same way as a traditional prototype-based system, mind you: here, a cloned object inherits all the behaviours provided by delegation, but not the behaviours from the cloned object itself. It's pretty close, though, and might help those to whom all this is new (but who are intimately familiar with the CLOS MOP – hm, actually, the intersection of those two sets might be a little small) get a bit of a handle on this.

Secondly, Lee Salzman was kind enough to get in touch, and we had a conversation about his thesis and his reflections on that work. I'll paraphrase, so again any misrepresentation is mine, but regarding the question at the end of my previous post about the semantics of redefinition, Lee was quite clear that the expectation was that previously-cloned objects should retain their previous behaviour, even if the original object is later updated (in prototypes and multiple dispatch terms, even if the original method is redefined). In other words, the ??? is indeed expected by prototype-oriented programmers to be (FOO[1] /B/).

Lee also said that, in his subsequent experience, he has come to believe that class-based systems are a necessity – particularly in image-based systems (where there is a persistent world). He's happy with prototypes as a mechanism in a world with static programme descriptions, where you can just edit the source code, then compile and run the programme, but believes that directly modifying running programmes makes classes or some equivalent abstraction necessary. I'm not sure I completely understand this point (though I'm fairly convinced by now that I don't much like the semantics of method redefinition in a world with long-lived objects), but I'm still waiting for a good example to highlight the advantages of prototype-oriented programmes.

Meanwhile, what does this imply for the implementation of prototypes with multiple dispatch within the extended CLOS metaobject protocol? Defining a new method specialized on an must override any existing behaviour for that object, but must leave unchanged the previous behaviour for any clones; that therefore implies that the generic function cannot discard the old method, as it must remain applicable to any clones. The consequences of this include: two prototype specializers, specialized on the same object, must not be considered to be the same specializers (otherwise the standard semantics of method definition would replace the old method with the new); and that the specializers must be annotated with the method that they belong to (so that it is possible to compute for a given specializer and argument whether that argument is acceptable to the specializer, remembering that the acceptability of an object to a prototype specializer in this world is primarily a function of the object itself). One piece of good news: the CLOS MOP already provides a mechanism for associating specializers with methods, through the specializer-direct-methods generic function, so that's at least some of the battle.

All of this merits a fuller description, a reference implementation, and some example code using it. I'm working on the first two, aiming to top up my all-important publications list by presenting at some conferences (and also helping to justifying my attendance); I'd love to hear about ideas for the third...

good student projects

For the first of my project and employability workshops this year, I aim to cover two principal discussion points. Firstly, the attributes of a good student project; we can do that in structured discussion form, which will almost certainly also highlight the need for a conscious effort in name recognition for networking:

  • pairs: introduce yourself and your project to your peer, and explain why you've chosen to do your particular project;
  • fours: introduce your peer and summarize their project to the other pair, and consider what makes a good student project.

Secondly, and taking as input the results of the previous discussion (and also comments from the project module leader, who can intervene at this stage), how can students best produce projects meeting the goals identified in the first discussion. I think this might be best delivered more from the front; there will be all sorts of practical advice that I can give (but I must make sure to ask the room for other experience).

So, before the session, what are the aspects of a good student project that I would highlight?

  • ambition: is the project hard? Does it carry an element of risk?
  • outcome: does the project work? How robust is it?
  • learning: has the student demonstrably learnt something while carrying out the project?
  • communication: is it clear to the assessors what the student has done?
  • demonstrability: can the project be used as an example in the student's future career?

I would expect the students as a group to come up with all of these, except perhaps ambition: I don't know if my expectations are wrong – I hope so, in fact – because if it doesn't show up I would wonder slightly whether we don't give enough emphasis through the programme to going beyond what is expected.

As for practical suggestions for them:

  • time-management and concentration: GTD, Remember the Milk; working habits and keeping logs; supervisor as project manager
  • learning and communication: communication to you now - what is interesting? What have you learnt?
  • demonstrability and outcome: minimum viable product
  • ambition: what would you like to do that you don't know how to do? What about things that no-one knows how to?
  • force multipliers: tools. Good compilers, good debuggers, good change management systems.

What am I forgetting? I hope I'll find out.

[ In the session, as it happened:

  • the standard “introduce your partner” bombshell when moving from pairs to fours had the expected effect, with many of the students unable to remember the name of their partner. I made the explicit link to networking.
  • “ambition” was in fact given by one group (but another group was contemplating “not too ambitious” as one of their criteria as I was walking around and listening, so there's still something to look at).
  • I didn't get on to anything technical; I spent more time on Minimal Viable Product, since the students claimed not to have heard the phrase before.


prototypes with multiple dispatch

On New Year's Day (did I have nothing better to do?) I asked the wider world real life uses of non-standard method selection, and I hinted that I had an real-life example of my own. This post does not discuss that example.

Instead, I'm going to discuss my limited understanding of another non-standard (at least, non-CLOS-standard) method selection system, and demonstrate what its use looks like in my current development world. I'm talking about a prototype-based object system: specifically, a mostly-faithful reimplementation of the Prototypes with Multiple Dispatch [1,2] system found in Slate and described by Lee Salzman and Jonathan Aldrich. I'll try to introduce the semantics as I understand them to an audience used to class-based multiple dispatch, but I'm left with some questions that I don't know how to answer, not being used to programming in this way myself.

So, first, what's going on? Well, the proper answer might be to read the linked papers, which have a relatively formal specification of the semantics, but the high-level idea could perhaps be summarised as finding what happens when you try to support prototype-based object systems (where the normal way to instantiate an object is to copy another one; where objects implement some of their functionality by delegating to other objects; and where single-dispatch methods are stored in the object itself) and multiple dispatch systems (where methods do not have a privileged receiver but dispatch on all of their arguments) simultaneously. The authors found that they could implement some reasonable semantics, and perform dispatch reasonably efficiently, by storing some dispatch metainformation within the objects themselves. The example code, involving the interactions between fish, healthy sharks and dying sharks, can be translated into my extended-specializer CL as:

(defpvar /root/ (make-instance 'prototype-object :delegations nil))
(defpvar /animal/ (clone /root/))
(defpvar /fish/ (clone /root/))
(defpvar /shark/ (clone /root/))
(defpvar /healthy-shark/ (clone /root/))
(defpvar /dying-shark/ (clone /root/))
(add-delegation /fish/ /animal/)
(add-delegation /shark/ /animal/)
(add-delegation /shark/ /healthy-shark/)
(defgeneric encounter (x y)
  (:generic-function-class prototype-generic-function))
(defmethod encounter ((x /fish/) (y /healthy-shark/))
  (format t "~&~A swims away~%" x))
(defmethod encounter ((x /fish/) (y /animal/))
(defgeneric fight (x y)
  (:generic-function-class prototype-generic-function))
(defmethod fight ((x /healthy-shark/) (y /shark/))
  (remove-delegation x)
  (add-delegation x /dying-shark/)
(defmethod encounter ((x /healthy-shark/) (y /fish/))
  (format t "~&~A swallows ~A~%" x y))
(defmethod encounter ((x /healthy-shark/) (y /shark/))
  (format t "~&~A fights ~A~%" x y)
  (fight x y))

(compare figures 4 and 7 of [1]; defpvar is secretly just defvar with some extra debugging information so I don't go crazy trying to understand what a particular #<PROTOTYPE-OBJECT ...> actually is.)

Running some of the above code with

(encounter (clone /shark/) (clone /shark/))



and returns


(and I'm confident that that's what is meant to happen, though I don't really understand why in this model sharks aren't fish).

The first question I have, then, is another lazyweb question: are there larger programs written in this style that demonstrate the advantages of prototypes with multiple dispatch (specifically over classes with multiple dispatch; i.e. over regular CLOS). I know of Sheeple, another lisp implementation of prototype dispatch, probably different in subtle or not-so-subtle ways from this one; what I'm after, though, probably doesn't exist: if there's no easy way of using prototype dispatch in Lisp, it won't be used to solve problems, and some other way will be used instead (let's call that the computer programmer's weak version of the Sapir-Whorf hypothesis). What's the canonical example of a problem where prototype-based object systems shine?

The second question I have is more technical, and more directly related to the expected semantics. In particular, I don't know what would be expected in the presence of method redefinition, or even if method redefinition is a concept that can make sense in this world. Consider

(defpvar /a/ (clone /root/))
(defgeneric foo (x)
  (:generic-function-class prototype-generic-function))
(defmethod foo ((x /a/)) `(foo[1] ,x))
(defpvar /b/ (clone /a/))
(foo /a/) ; => (FOO[1] /A/)
(foo /b/) ; => (FOO[1] /B/)
(defmethod foo ((x /a/)) `(foo[2] ,x))
(foo /a/) ; => (FOO[2] /A/)
(foo /b/) ; => ???

What should that last form return? Arguments from the prototype-oriented world would, I suspect, lead to the desired return value being (FOO[1] /B/), as the redefinition of the method on foo specialized to /a/ is completely irrelevant to /b/. This is my reading of the semantics described in [1], for what it's worth. Arguments from the world of generic functions and expected behaviour would probably argue for (FOO[2] /B/), because redefining a method is an action on a generic function, not an action on a set of application objects. And the argument of my implementation in practice (at present, subject to change) is to signal no-applicable-method, because method redefinition is the successive removal of the old method and addition of the new one, and removal of the old method of the generic function affects dispatch on all the objects, whereas adding the new one affects dispatch on just /a/.

why investigate databases of audio?

At the launch meeting for Transforming Musicology (I should perhaps say that technically this was in the pub after the launch meeting), Laurence Dreyfus asked me why I was experimenting with a collection of recordings when planning my Josquin/Gombert case study, as opposed to the perhaps more natural approach of investigating the motivic or thematic similarity of music starting with the notated works (manuscripts, contemporary or modern editions).

By this stage in the evening, I was not feeling at my best – a fact unrelated to the venue of the conversation, I hasten to add – and I wouldn't be surprised if my answer was not a model of clarity. Here's another go, then, beginning with an anecdote.

When I was a PhD student, a long, long time ago in a discipline far, far, away, the research group I was in had a weekly paper-reading club, where students and postdocs, and the occasional permanent member of staff, would gather together over lunch and try to work and think through an article in the field. Usually, the paper selected was roughly contemporary, maybe published in the last few years, and therefore (because of the publishing norms in Applied Mathematics and Theoretical Physics) the paper would be available on the arXiv, and so could be acquired, and printed, without leaving one's desk. (Picking up the printed copy could double up as one's daily exercise quota.) One week, however, some imp put down a paper that predated the arXiv's rise to prominence. I have two clear memories, which should nevertheless be treated as unreliable: one of myself, spending about an hour of elapsed time in the mathematical library, registering, finding the journal, finding the volume and photocopying the article; the other of the lunch meeting, discovering that those of us having seen copies of the article (let's not go as far as having read the article) being definitively in the minority.

There are several lessons to be drawn from this, beyond further evidence of the laziness and hubris of the typical PhD student. The availability of information, in whatever form, makes it more likely to be useful and more likely to be used. It's timely to reflect on the case of Aaron Swartz, who confronted this issue and faced disproportionate consequences; I'm pleased to say that I am involved in my institution's working group on Open Access, as we try to find a way of simultaneously satisfying all the requirements imposed on us as well as extending our ability to offer a transformative experience.

How does this affect musicology, and our particular attempt to give it the transformative experience? Well, one issue with music in general is its lack of availability.

That is of course a ridiculous statement, designed to be provocative: in this era of downloadable music, streaming music, ripped music, music in your pocket and on your phone, music has never been more available. But the music available in all those forms is audio; the story for other forms of musical data (manuscript notation, chord sequences, performing editions, even lyrics) is far less good. Sometimes that is for copyright reasons, as rights-holders attempt to limit the distribution of their artifacts; sometimes that is for economic reasons, as the resources to digitize and encode public-domain sources of notation are unavailable. But the net effect is that while musical audio is limited and expensive, it is often not possible to purchase large amounts of musical data in other forms at any price.

So, in my still-hypothetical case study of Josquin/Gombert attribution, I could use notation as my primary source of data if I had encodings of sources or editions of the pieces I'm interested in – the needles, if you will; if I had those, I could verify that some computational algorithm could identify the association between sections of the chansons, Lugebat and Credo. That encoding task would be tedious, but achievable in – let's say – a week or so for the three pieces. The problem with that is that that is only half the battle; it is the construction of the needle, but for a method like this to be properly tested it's not fair just to let it loose on the needle; a search system has to be given the chance to fail by hiding the needle in a suitable haystack. And constructing a haystack – encoding a few hundreds of other Renaissance vocal works – is the thing that makes it less than practical to work on search within databases of notation, at least for this long-tail repertoire.

And that's why at the moment I investigate databases of musical audio: I can get the data, and in principle at least so can other people. If you can do it better, or just differently, why not consider applying for one of the Transforming Musicology mini-projects? There will be a one-day exploratory event at Lancaster University on the 12th February to explore possible project ideas and collaborations, and having as wide a pool of interested parties as possible is crucial for the production and execution of interesting ideas. Please come!

slime has moved to github

This is probably not news to anyone reading this, but: SLIME has moved its primary source repository to github.

One of my projects, swankr (essentially, a SLIME for R, reusing most of the existing communication protocol and implementing a SWANK backend in R, hence the name), obviously depends on SLIME: if nothing else, the existing instructions for getting swankr up and running included a suggestion to get the SLIME sources from CVS, which as Zach says is a problem that it's nice no longer to have. (Incidentally, Zach is running a surveydirect link – to assess possible effects of changes on the SLIME userbase.)

So, time to update the instructions, and in the process also

  • hoover up any obvious patches from my inbox;
  • improve the startup configuration so that the user need do no explicit configuration beyond a suitable entry in slime-lisp-implementations;
  • update the and files for the current reality.

That's all safely checked in and pushed to the many various locations which people might think of as the primary swankr repository. But those org-mode files are exported to HTML to produce the minimal swankr web presence, and the script that does that was last run with org-mode version 7.x, while now, in the future, org-mode is up to version 8 (and this time the change in major version number is definitely indicative of non-backwards-compatible API changes).

This has already bitten me. The principles of reproducible research are good, and org-mode offers many of the features that help: easily-edited source format, integration with a polyglossia of programming languages, high-quality PDF output (and adequate-quality HTML output, from the same sources) – I've written some technical reports in this style, and been generally happy with it. But, with the changes from org-mode 7 to org-mode 8, in order to reproduce my existing documents I needed to make not only (minor) source document changes, but also (quite major) changes to the bits of elisp to generate exported documents in house style. Reproducible research is difficult; I suppose it's obvious that exact reproduction depends on exact software versions, platform configurations and so on – see this paper, for example – but how many authors of supposedly reproducible research are actually listing all the programs, up to exact versions of dynamically-linked libraries they link to, used in the preparation and treatment of the data and document?

In any case, the changes needed for swankr are pretty minor, though I was briefly confused when I tried org-html-export-as-html first (and no output was produced, because that's the new name for exporting to an HTML buffer rather than an HTML file. The swankr website should now reflect the current state of affairs.

the dangers of writing less

A year or so ago, I wrote a couple of blog entries on precompiling discriminating functions for generic functions, both given mostly class-based dispatch and also when most of the methods were specialized on individual objects (i.e. methods with specializers of class eql-specializer).

I signed off the second entry with

Next up, unless I've made further oversights which need correction: automating this process, and some incidental thoughts.

and of course never got back to this topic. Now it's over a year later, and I can no longer remember either my incidental thoughts, which I'm sure were fascinating, nor my strategy for automating this (because this was an attempt to address an actual user need, in – if I remember correctly – a CL implementation of protobuf). Suggestions for what I might have been thinking at the time gratefully received.

Write more: you know it makes sense.

transforming musicology launch meeting

Our AHRC-funded project, Transforming Musicology, has its first all-hands meeting today.

While it may seem odd that we are three months into the funded period of the project and we haven’t all met yet – and it is a bit odd, really – it was also pretty much unavoidable: the confirmation of funding only came through in August, at which point it is difficult to get things up and running in a University in time to advertise a PhD studentship, deal with application enquiries, shortlist, interview, appoint a good candidate and deal with all the associated paperwork in time for September enrolment. The timing meant that there was an inevitable built-in three-month lag before some key project personnel were available (the next PhD enrolment date is now, in January) so it made sense to delay the launch meeting until this point, too. Not to mention that many of the project partners are high-powered professors with many diary commitments; getting them all in one place on one day is a triumph of organization in itself, for which Richard deserves much credit.

I spent some time yesterday trying to cook up a demo of audioDB, an old, unlamented (but a bit lamentable in its current state) tool from an earlier project. I’m at the stage of identifying the low-hanging fruit in audioDB that would need picking to make it a tool useful to musicologists; as well as the UI issues – it is definitely a tool currently optimized towards use on the command-line – there is the existential question of whether what it does could be useful in any way to musicologists in the first place. I think so, but at the moment it’s more handwaving and faith than firm knowledge.

The case study I’m working on is based on a question of attribution of some chansons and motets. The issue in question has largely been settled; I think everyone these days accepts that the eight-part Lugebat David Absalon is by Nicolas Gombert, not Josquin Des Prez, and that some other works (Tulerunt Dominum, Je prens congies and J’ay mis mon cueur) are also likely to be of the same musical hand: Martin Picker wrote on this in 2001, but recordings that I have of these works which predate that also agree on the attribution. (The works were originally attributed to Josquin based partly on their high quality; it has been said that for a while Josquin composed more and better works after he was dead than while he was alive...)

A nice demonstration, then, would be to reproduce the similarity relationships between recordings of the works discussed in Picker’s article, and show that those similarities are stronger than acoustic similarities that arise by chance. This isn’t going to change anyone’s mind either way on the Lugebat question, of course, but if it works it can give some confidence to musicologists that audioDB could be used to investigate collections of recordings for acoustically similar material without already knowing the answer.

Does it work? Sort-of-not-quite-yet; the chansons and the Credo do bubble up near the top of the search results, but without the clear delineation between them and other, less related hits. Yesterday’s investigations revealed the need for a couple of improvements: firstly, some finer-grained filtering of “interesting” regions, as otherwise the strongest matches between audio of this era tends to be strong final G-minor chords; secondly, there needs to be some work done on audio feature design, to improve the machine listener’s ear, because at the moment the feature I'm using does not capture pitch perception reliably enough. The good news is that addressing these things are in scope for the Transforming Musicology project, so there's some chance that they'll get done.

more efficient hyperlinked blogging

How meta. To maintain my progress on my new year's resolution, I have written some code (*gasp!*) – yes, that counts as writing. And what does that code do? Why, it makes me more efficient at highly hyperlinked blogging: it is a certain amount of fairly trivial and mildly tedious elisp, which allows the easy insertion of markdown markup to create links to various authoritative sources of information about Lisp. Well, OK, to the Hyperspec and the MOP dictionary, but as an implementor that's all I really need, right? So, now I can talk about compute-effective-method or make-method-lambda and my eager readers can be taken to the relevant documentation at the speed of thought.

Questions that arose during the process:

  • why are all the fake packages in hyperspec.el created with (make-vector 67 0)?
  • has anyone in the 23 years since AMOP was published ever been glad that the MOP standardizes the extract-lambda-list and extract-specializer-names functions? (Fun fact: SBCL also has extract-parameters and extract-required-parameters functions, unexported and unused.)
bugs all the way down

There are times when being in control of the whole software stack is a mixed blessing.

While doing investigations related to my previous post, I found myself wondering what the arguments and return values of make-method-lambda were in practice, in SBCL. So I did what any self-respecting Lisp programmer would do, and instead of following that link and decoding the description, I simply ran (trace sb-mop:make-method-lambda), and then ran my defmethod as normal. I was half-expecting it to break instantly, because the implementation of trace encapsulates named functions in a way that changes the class of the function object (essentially, it wraps the existing function in a new anonymous function; fine for ordinary functions, not so good for generic-function objects), and I was half-right: an odd error occurred, but after trace printed the information I wanted.

What was the odd error? Well, after successfully calling and returning from make-method-lambda, I got a no-applicable-method error while trying to compute the applicable methods for... make-method-lambda. Wait, what?

SBCL's CLOS has various optimizations in it; some of them have been documented in the SBCL Internals Manual, such as the clever things done to make slot-value fast, and specialized discriminating functions. There are plenty more that are more opaque to the modern user, one of which is the “fast method call” optimization. In that optimization, the normal calling convention for methods within method combination, which involves calling the method's method-function with two arguments – a list of the arguments passed to the generic function, and a list of next methods – is bypassed, with the fast-method-function instead being supplied with a permutation vector (for fast slot access) and next method call (for fast call-next-method) as the first two arguments and the generic function's original arguments as the remainder, unrolled.

In order for this optimization to be valid, the call-method calling convention must be the standard one – if the user is extending or overriding the method invocation protocol, all the optimizations based on assuming that the method invocation protocol might be invalid. We have to be conservative, so we need to turn this optimization off if we can't prove that it's valid – and the only case where we can prove that it's valid is if only the system-provided method on make-method-lambda has been called. But we can't communicate that after the fact; although make-method-lambda returns initargs as well as the lambda, an extending method could arbitrarily mess with the lambda while returning the initargs the system-provided method returns. So in order to find out whether the optimization is safe, we have to check whether exactly our system-provided method on make-method-lambda was the applicable one, so there's an explicit call to compute-applicable-methods of make-method-lambda after the method object has been created. And make-method-lambda being traced and hence not a generic-function any more, it's normal that there's an error. Hooray! Now we understand what is going on.

As for how to fix it, well, how about adding an encapsulations slot to generic-function objects, and handling the encapsulations in sb-mop:compute-discriminating-function? The encapsulation implementation as it currently stands is fairly horrible, abusing as it does special variables and chains of closures; there's a fair chance that encapsulating generic functions in this way will turn out a bit less horrible. So, modify sb-debug::encapsulate, C-c C-c, and package locks strike. In theory we are meant to be able to unlock and continue; in practice, that seems to be true for some package locks but not others. Specifically, the package lock from setting the fdefinition from a non-approved package gives a continuable error, but the ones from compiling special declarations of locked symbols have already taken effect and converted themselves to run-time errors. Curses. So, (mapcar #'unlock-package (list-all-packages)) and try again; then, it all goes well until adding the slot to the generic-function class (and I note in passing that many of the attributes that CL specifies are generic-function SBCL only gives to standard-generic-function objects), at which point my SLIME repl tells me that something has gone wrong, but not what, because no generic function works any more, including print-object. (This happens depressingly often while working on CLOS).

That means it's time for an SBCL rebuild, which is fine because it gives me time to write up this blog entry up to this point. Great, that finishes, and now we go onwards: implementing the functionality we need in compute-discriminating-function is a bit horrible, but this is only a proof-of-concept so we wrap it all up in a labels and stop worrying about 80-column conventions. Then we hit C-c C-c and belatedly remember that redefining methods involves removing them from their generic function and adding them again, and doing that to compute-discriminating-function is likely to have bad consequences. Sure enough:

There is no applicable method for the generic function 
when called with arguments

Yes, well. One (shorter) rebuild of just CLOS later, and then a few more edit/build/test cycles, and we can trace generic functions without changing the identity of the fdefinition. Hooray! Wait, what was I intending to do with my evening?

seeking real life uses for generalized specializers

Some time ago (call it half a decade or so), Jim Newton of Cadence and I did some work on extensible specializers: essentially coming up with a proof-of-concept protocol to allow users to define their own specializers with their own applicability and ordering semantics. That's a little bit vague; the concrete example we used in the writeup was a code walker which could warn about the use of unbound variables (and the non-use of bindings), and which implemented its handling of special forms with code of the form:

(defmethod walk ((expr (cons (eql 'quote))) env call-stack)
(defmethod walk ((var symbol) env call-stack)
  (let ((binding (find-binding env var)))
    (if binding
        (setf (used binding) t)
        (format t "~&unbound: ~A: ~A~%" var call-stack))))
(defmethod walk ((form (cons (eql 'lambda))) env call-stack)
  (destructuring-bind (lambda lambda-list &rest body) form
    (let* ((bindings (derive-bindings-from-ll lambda-list))
           (env* (make-env bindings env)))
      (dolist (form body)
        (walk form env* (cons form call-stack)))
      (dolist (binding bindings)
        (unless (used (cdr binding))
          (format t "~&unused: ~A: ~A~%" (car binding) call-stack))))))

The idea here is that it's possible to implement support in the walker for extra special forms in a modular way; while this doesn't matter very much in Common Lisp (which, famously, is not dead, just smells funny), in other languages which have made other tradeoffs in the volatility/extensibility space. And when I say “very much” I mean it: even SBCL allows extra special forms to be loaded at runtime; the sb-cltl2 module includes an implementation of compiler-let, which requires its own special handling in the codewalker which is used in the implementation of CLOS.

So modularity and extensibility is required in a code walker, even in Common Lisp implementations; in Cadence Skill++ it might even be generally useful (I don't know). In SBCL, the extensibility is provided using an explicit definer form; sb-cltl2 does

(defun walk-compiler-let (form context env)
  #1=#<implementation elided>)
(sb-walker::define-walker-template compiler-let walk-compiler-let)

and that's not substantially different from

(defmethod sb-walker:walk ((form (cons (eql 'compiler-let))) context env)

So far, so unexpected, for Lisp language extensions at least: of course the obvious test for a language extension is how many lines of code it can save when implementing another language extension. Where this might become interesting (and this, dear lazyweb, is where you come in) is if this kind of extension is relevant in application domains. Ideally, then, I'm looking for real-life examples of patterns of selecting ‘methods’ (they don't have to be expressed as Lisp methods, just distinct functions) based on attributes of objects, not just the objects' classes. The walker above satisfies these criteria: the objects under consideration are all of type symbol or cons, but the dispatch partly happens based on the car of the cons – but are there examples with less of the meta nature about them?

(I do have at least one example, which I will keep to myself for a little while: I will return to this in a future post, but for now I am interested in whether there's a variety of such things, and whether the generalization of specializer metaobjects is capable of handling cases I haven't thought of yet. Bonus points if the application requires multiple dispatch and/or non-standard method combination.)

new year's resolution

So here we are: 31st December. Sitting here, with a film of dubious plot playing on the television, waiting for a sufficient approximation to the New Year to arrive, and thinking about the year just gone and the year that is yet to come, and making resolutions. And on looking back at 2013, one thing that I'm not happy about (while counting my many, many blessings) is the level of output: somewhat limited code, very limited publications – honestly, thanks to my PhD students Ray and Polina for any publications at all: they did the vast majority of the work on papers with my name listed as co-author. There are of course reasons for that, stemming both from personal and professional life, but it doesn't change my dissatisfaction with the end result: there's more that I can do, and more that I should be doing. And doing, as Yoda never said, leads to writing: if the act is sufficiently interesting, it should be natural to talk to people about it, to be enthusiastic about it, and to write about it.

I certainly feel that it's in my nature not to be enthusiastic. Friends, if I have any left, are likely to put my typical disposition somewhere near the cynical extreme – I like to think not without some provocation, but I can also acknowledge that an excess of cynicism can obscure those things that are in fact worth celebrating. I'm certainly not promising to be less cynical about those things which deserve it – many an academic will point at a number of things wrong with the current state of the profession, and I think we're not all harking back to a mislaid golden age when academics were free and academia was purely communitarian – but I think I need to act in positive ways as well as grind my teeth and bang my head against the wall when the negatives arise.

Baby steps first, though. The commitment: I will write more. I will write more about my ideas, my processes, my interactions and my achievements, and I'll do it publically where possible, as it's part of my self-image as an academic to educate by example. (And maybe at some point I will also make this website look pretty. If you're reading this through syndication, count yourself lucky.)

some sbcl wiki entries

One of the potential advantages of this new wiki/blog setup is that I should be able to refer easily to my notes in my blog. It's true that the wiki-like content will change, whereas a mostly-static blog entry will naturally refer to the content of the wiki as it is at the point of blogging; in theory I should be able to find the content at the time that the link was made through a sufficiently advanced use of the version control system backing the ikiwiki installation, and in practice the readers of the blog (if any) are likely to follow (or not) any links within a short time-window of the initial publication of any given entry. So that shouldn't prove too confusing.

In order to test this, and to provide some slightly-relevant content for anyone still reading this on planet lisp: I've transferred my non-actionable notes from my sbcl org-mode file to an sbcl wiki section. This new setup gets heaps of bonus points if someone out there implements the development projects listed there as a result of reading this post, but even if not, it's a useful decluttering...

new year new gpg key

I've been putting this off for long enough. My current PGP key was generated in a different age, with somewhat different threat models and substantially different computational power (let alone social and political power) available to those who might be interested in subverting the general ability of individuals to identify themselves and communicate in private. It's true that not much of what I do is likely to be of vast interest to security agents worldwide, but as SBCL release manager (on holiday for a month!) I do publish documents that purport to assure that the source code and binary that users can obtain are unmodified from my personal copies.

Therefore, I've created a new key, with 2013-era configuration settings: 2048-bit RSA encryption and signing, and SHA-1 hashes undesired (though still accepted, given the protocol requirement). I've created a transition document, signed with both keys, to help interested parties with the transition; while my older key will remain valid for a while (I have a stack of business cards with its fingerprint on to get through...) the new key should be preferred in all situations where that makes sense.


  1. This is all quite fiddly and complicated. There's a reason (well, several reasons: all those business cards!) I have been putting this off for a while. I understand why at least some of the knobs and levers are there; I've taught Computer Security at undergraduate level – but I'm still not confident that I have got everything right, and that's worrying both personally and systemically.
  2. epa-sign-region appears to malfunction for me when given a prefix argument: what it's meant to do is give a UI for selecting keys to use for signing, and the option of which kind of signature to generate. What it actually seems to do is generate an undecryptable PGP Message block. [edit: this was caused by epa/epg version skew, from an old installation of the Debian easypg package. ]
  3. I stumbled over the --escape-from-lines GnuPG configuration option while reading the documentation. It's on by default.
  4. GnuPG is crowdfunding a new website and some infrastructure.
new year's resolution more preparations

This new blogging setup, with the new domain, new feeds, and ideally new workflow, suggest the analysis of what was problematic with the old one.

Back when I was a PhD student, doing everything I could to avoid work on my thesis, I stumbled upon a culture of Lisp and Free Software developers – which may only come across as half ridiculous now but in 2001 seemed doubly counter-cultural. And so I joined a blogging platform for Free Software types that also happened to double as a testbed for trust metric research. And at the time, this offered neat features not easily available elsewhere: the word “blog” itself was only about two years old, and pertelote maintained a blog by hand for years, so having somewhere which would host my writing, and which moreover had some of the Social Network about it (the advogato userbase was responsible for certification of individuals' contribution to Free Software, and several of my then collaborators were on advogato before me) was ideal.

The principal downside for me, then as now, is the use of web browsers as the interface for text entry; a <textarea> just isn't rich enough in functionality. The situation is better now than it was – (mostly) gone are the days when an incautious C-q would instantly close all browser windows, including the one in which your meticulously-crafted draft was waiting for final polish before being submitted as a diary entry – but it's still a long way from using my text editor, moderately customized to my way of working, with all the comfort that that brings.

Meanwhile, for work, I've successfully used ikiwiki to document the processes and data around some of my academic administrative responsibilities over the years: postgraduate tutor, employability coordinator, director of postgraduate studies (the reality is even less glamorous than these titles); it's been really nice to be able to update documents while not at my desk, or while commuting, and even if I have never grown to love mercurial (ikiwiki's at-one-time default DVCS backend) it is at least worth knowing.

So when contemplating how to set up systems less likely to impede my productivity, it's perhaps natural that I should return to a recent success, and set up an ikiwiki instance. This time, though, I'm (obviously) explicitly using ikiwiki's support for blogging as well as its natural wiki nature; the idea is that the weblog is for timely, pertinent comments, while the wiki is for incrementally-refined notes on all subjects – probably replacing my org-mode note-taking habit, which while lovely relies too much on having a single emacs instance available at all times for when inspiration strikes. I fully expect to keep the org-mode setup I have for Getting (Some) Things Done, but will likely migrate the non-actionable notes I have to this wiki as and when.

Speaking of new feeds, the advogato tag will be punned into denoting content that should be syndicated to my diary on advogato. It's perhaps inevitable that something will go wrong, and including html tags in the text of the first syndicated entry is an excellent way to ensure that that happens, so if there's an editable textarea, escaped less-than and greater-than signs, or a sentence that seems to be missing something, then the Unix-Haters among the readers can reminisce about seeing >From in their e-mails oh wait that still happens...

new year's resolution preparations

One of the benefits of a robust process of evaluation is to be able to confront things that are not going as well as desired, as well as to identify strengths that can be exploited further. Universities don't necessarily have a robust performance evaluation process – it seems largely to depend on whether your Head of Department is a believer, or else on the individual to insist – but this year I did have the opportunity to have a Personal Development Review, and something that hit home as I was reviewing my own achievements was how little I've written recently. This is not ideal, either from the point of view of my academic career (where publish-or-perish might not be strictly true but it's at least a good first approximation) nor from the angle of my own self-esteem as a creative and productive individual.

So, steps must be taken, and it so happens that this is the time of year for turning over new leaves, cleaning up acts, and otherwise resolving to change behaviour. It's obviously not quite as simple as saying that I will write more; while there are some obstacles to creativity that can be straightforwardly addressed, I think the main thing necessary is to find that set of things such that when a spare fifteen minutes presents itself, the most enjoyable and least stressful thing to do with them is to write (or draft, or otherwise be productive).

Some of the things to arrange for this to happen are negative: to remove the temptation, or at least make it harder, to do unproductive things. So, apt-get purge crawl, for example, might be the single most effective productivity enhancer from my current state; I might have successfully avoided the trap of Candy Crush (and Angry Birds), but roguelikes are terribly easy to get sucked into, and anecdotally have the same addictive properties on me at least as the latest mobile game has on everyone else. But some steps are positive; I have used this as an excuse to set up a new computing environment for myself: a VPS running nginx, serving (among other things) a blog and a wiki using ikiwiki, which should make it easier to draft and publish than previously.