pages tagged algorithms-and-data-structures
http://christophe.rhodes.io/notes/tag/algorithms-and-data-structures/
notesikiwikiTue, 08 May 2018 21:21:14 +0200algorithms and data structures term2http://christophe.rhodes.io/notes/blog/posts/2018/algorithms_and_data_structures_term2/
http://christophe.rhodes.io/notes/blog/posts/2018/algorithms_and_data_structures_term2/
algorithms-and-data-structuresemacsgoldsmithshomoiconicityis52038blispussstrikeTue, 08 May 2018 21:17:43 +02002018-05-08T19:21:14Z<p>I presented some of the work on teaching algorithms and data
structures at
the
<a href="https://www.european-lisp-symposium.org/2018/index.html">2018 European Lisp Symposium</a></p>
<p>Given that I wanted to go to the symposium (and <a href="http://christophe.rhodes.io/notes/blog/posts/2018/els2018_reflections/">I’m glad I
did!</a>), the most economical method for going was
if I presented research work – because then there was a reasonable
chance that <a href="https://www.gold.ac.uk/">my employer</a> would fund the
expenses (spoiler: they did; thank you!). It might perhaps be
surprising to hear that they don’t usually directly fund attending
events where one is not presenting; on the other hand, it’s perhaps
reasonable on the basis that part of an academic’s job as a scholar
and researcher is to be creating and disseminating new knowledge, and
of course universities, like any organizations, need to prioritise
spending money on things which bring value or further the
organization’s mission.</p>
<p>In any case, I found that I wanted to write about the teaching work
that I have been doing, and in particular I chose to write about a
small, Lisp-related aspect. Specifically, it is now fairly normal in
technical subjects to perform a lot of automated testing of students;
it relieves the burden on staff to assess things which can be
mechanically assessed, and deliver feedback to individual students
which can be delivered automatically; this frees up staff time to
perform targeted interventions, give better feedback on more
qualitative aspects of the curriculum, or work fewer weekends of the
year. A large part of my teaching work for the last 18 months has
been developing material for these automated tests, and working on the
infrastructure underlying them, for my and colleagues’ teaching.</p>
<p>So, the more that we can test automatically <em>and meaningfully</em>, the
more time we have to spend on other things. The main novelty here,
and the lisp-related hook for the paper I submitted to ELS, was being
able to give meaningful feedback on numerical answer questions which
probed whether students were developing a good mental model of the
meaning of pseudocode. That’s a bit vague; let’s be specific and
consider the <code>break</code> and <code>continue</code> keywords:</p>
<pre><code>x ← 0
for 0 ≤ i < 9
x ← x + i
if x > 17
continue
end if
x ← x + 1
end for
return x
</code></pre>
<p>The above pseudocode is typical of what a student might see; the
question would be “what does the above block of pseudocode return?”,
which is mildly arithmetically challenging, particularly under time
pressure, but the conceptual aspect that was being tested here was
whether the student understood the effect of <code>continue</code>. Therefore,
it is important to give the student specific feedback; the more
specific, the better. So if a student answered 20 to this question
(as if the <code>continue</code> acted as a <code>break</code>), they would receive a
specific feedback message reminding them about the difference between
the two operators; if they answered 45, they received a message
reminding them that <code>continue</code> has a particular meaning in loops; and
any other answers received generic feedback.</p>
<p>Having just one of these questions does no good, though. Students
will go to almost any lengths to avoid learning things, and it is easy
to communicate answers to multiple-choice and short-answer questions
among a cohort. So, I needed hundreds of these questions: at least
one per student, but in fact by design the students could take these
multiple-chocie quizzes multiple times, as they are primarily an aid
for the students themselves, to help them discover what they know.</p>
<p>Now of course I could treat the above pseudocode fragment as a
template, parameterise it (initial value, loop bounds, increment) and
compute the values needing the specific feedback in terms of the
values of the parameters. But this generalizes badly: what happens
when I decide that I want to vary the operators (say to introduce
multiplication) or modify the structure somewhat (<em>e.g.</em> by swapping
the two increments before and after the <code>continue</code>)? The
parametrization gets more and more complicated, the chances of (my)
error increase, and perhaps most importantly it’s not any fun.</p>
<p>Instead, what did I do? With some sense of grim inevitability, I
evolved (or maybe accreted) an interpreter (in emacs lisp) for a
sexp-based representation of this pseudocode. At the start of the
year, it's pretty simple; towards the end it has developed into an
almost reasonable mini-language. Writing the interpreter is
straightforward, though the way it evolved into one gigantic <code>case</code>
statement for supported operators rather than having reasonable
semantics is a bit of a shame; as a bonus, implementing a
pretty-printer for the sexp-based pseudocode, with correct indentation
and keyword highlighting, is straightforward. Then armed with the
pseudocode I will ask the students to interpret, I can mutate it in
ways that I anticipate students might think like (replacing <code>continue</code>
with <code>break</code> or <code>progn</code>) and interpret that form to see which wrong
answer should generate what feedback.</p>
<p>Anyway, that was the hook. There's some evidence
in <a href="https://research.gold.ac.uk/23155/">the paper</a> that the general
approach of repeated micro-assessment, and also the the consideration
of likely student mistakes and giving specific feedback, actually
works. And now that the (provisional) results are in, how does this
term compare with <a href="http://christophe.rhodes.io/notes/blog/posts/2018/algorithms_and_data_structures_term1/">last term</a>?
We can look at the relationship between this term’s marks and last
term’s. What should we be looking for? Generally, I would expect
marks in the second term’s coursework to be broadly similar to the
marks in the first term – all else being equal, students who put in a
lot of effort and are confident with the material in term 1 are likely
to have an easier time integrating the slightly more advanced material
in term 2. That’s not a deterministic rule, though; some students
will have been given a wake-up call by the term 1 marks, and equally
some students might decide to coast.</p>
<p><a href="http://christophe.rhodes.io/notes/blog/posts/2018/algorithms_and_data_structures_term2/term2-vs-term1.png"><img src="http://christophe.rhodes.io/notes/blog/posts/2018/algorithms_and_data_structures_term2/200x-term2-vs-term1.png" width="200" height="150" alt="plot of term 2 marks against term 1: a = 0.82, R² = 0.67" class="img" /></a></p>
<p>I’ve asked R to draw the regression line in the above picture; a
straight line fit seems reasonable based on the plot. What are the
statistics of that line?</p>
<pre><code>R> summary(lm(Term2~Term1, data=d))
Call:
lm(formula = Term2 ~ Term1, data = d)
Residuals:
Min 1Q Median 3Q Max
-41.752 -6.032 1.138 6.107 31.155
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 3.18414 4.09773 0.777 0.439
Term1 0.82056 0.05485 14.961 <2e-16 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 10.46 on 107 degrees of freedom
(32 observations deleted due to missingness)
Multiple R-squared: 0.6766, Adjusted R-squared: 0.6736
F-statistic: 223.8 on 1 and 107 DF, p-value: < 2.2e-16
</code></pre>
<p>Looking at the summary above, we have a strong positive relationship
between term 1 and term 2 marks. The intercept is approximately zero
(if you got no marks in term 1, you should expect no marks in term 2),
and the slope is less than one: on average, each mark a student got in
term 1 tended to convert to 0.8 marks in term 2 – this is plausibly
explained by the material being slightly harder in term 2, and by the
fact that some of the assessments were more explicitly designed to
allow finer discrimination at the top end – marks in the 90s. (A note
for international readers: in the UK system, the pass mark is 40%,
excellent work is typically awarded a mark in the 70% range – marks of
90% should be reserved for exceptional work). The average case is,
however, only that: there was significant variation from that average
line, and indeed (looking at the quartiles) over 50% of the cohort was
more than half a degree class (5 percentage points) away from their
term 2 mark as “predicted” from their mark for term 1.</p>
<p>All of this seems reasonable, and it was a privilege to work with this
cohort of students, and to present the sum of their interactions on
this course to the audience I had. I got the largest round of
applause, I think, for revealing that as part of the peer-assessment I
had required that students run each others’ code. I also had to
present some of the context for the work; not only because this was an
international gathering, with people in various university systems and
from industry, but also because of the large-scale disruption caused
by
<a href="https://www.ucu.org.uk/strikeforuss">industrial</a> <a href="https://ussbriefs.com/">action</a> over
the <a href="https://www.uss.co.uk/">Universities Superannuation Scheme</a> (the
collective, defined benefit pension fund for academics at about 68
Universities and ~300 other bodies associated with Higher Education).
Perhaps most gratifyingly, students were able to continue learning
despite being deprived of their tuition for three consecutive weeks;
judging by their performance on the various assessments so far,</p>
<p>And now? The students will sit an exam, after which I and colleagues
will look in detail at those results and the relationship with the
students’ coursework marks (as I did <a href="http://christophe.rhodes.io/notes/blog/posts/2017/analysing_algorithms_and_data_structures_data/">last
year</a>). I will
continue developing this material (my board for this module currently
lists 33 todo items), and adapt it for next year and for new cohorts.
And maybe you will join me?
The <a href="https://www.doc.gold.ac.uk/computing/">Computing department</a>
at <a href="https://www.gold.ac.uk">Goldsmiths</a> is hiring lecturers and senior
lecturers to come and participate in research, scholarship and
teaching in computing:
a
<a href="https://jobs.gold.ac.uk/vacancy/lecturer-in-creative-computing-348799.html">lecturer in creative computing</a>,
a
<a href="https://jobs.gold.ac.uk/vacancy/lecturer-in-computer-games-348630.html">lecturer in computer games</a>,
a
<a href="https://jobs.gold.ac.uk/vacancy/lecturer-in-data-science-348531.html">lecturer in data science</a>,
a
<a href="https://jobs.gold.ac.uk/vacancy/lecturer-in-physical-and-creative-computing-348527.html">lecturer in physical and creative computing</a>,
a
<a href="https://jobs.gold.ac.uk/vacancy/lecturer-in-computer-science-348441.html">lecturer in computer science</a> and
a
<a href="https://jobs.gold.ac.uk/vacancy/senior-lecturer-in-computer-science-348301.html">senior lecturer in computer science</a>.
Anyone reading this is welcome
to <a href="mailto:c.rhodes@gold.ac.uk">contact me</a> to find out more!</p>
algorithms and data structures term1http://christophe.rhodes.io/notes/blog/posts/2018/algorithms_and_data_structures_term1/
http://christophe.rhodes.io/notes/blog/posts/2018/algorithms_and_data_structures_term1/
algorithms-and-data-structuresemacsgoldsmithsis52038blispteachingFri, 05 Jan 2018 16:52:50 +01002018-05-04T09:29:07Z<p>Another term, another cycle of Algorithms & Data Structures.</p>
<p>This year I am teaching the entire two-term course by myself, compared
with <span class="createlink">last academic year</span> when
I co-taught with another member of staff. The change brings with it
different challenges; notably that I am directly responsible for more
content. Or in this case, more multiple-choice quizzes!</p>
<p>This term, the students have been given nine multiple-choice quizzes,
of which four are new content and five are revisions of last year's
quizzes. The revisions are mostly to add more variety to particular
question types – finding axes of generalization that I hadn't
exploited last time – and more (and more punishing) distractor
answers, so that students cannot game the system <em>too</em> easily.</p>
<p>Some of this is my own adaptation to behaviours I see in the student
body; for example, one behaviour I started seeing this year was the
habit of selecting <em>all</em> the answers from a multiple-answer
multiple-choice question. This was probably a reaction itself to the
configuration of the quiz not telling the students the right answer
after the attempt, but merely whether they had got it right or wrong;
the behaviour of the quiz engine
(the
<a href="https://moodle.org/">Moodle</a> <a href="https://docs.moodle.org/en/Quiz_activity">quiz activity</a>)
was for each <em>selected</em> answer to indicate the correctness status, and
so students were exploiting this to see the right answers. This was
not in itself a problem – there were enough questions that in the
student's next attempts at the quiz they were unlikely to see the same
questions again – but it was being used as a substitute for actually
thinking and working at the problem, and so this was a behaviour that
I wanted to discourage. The next quiz, therefore, I adapted so that
it had many single-answer multiple choice with many more distractor
answers than usual: seven or eight, rather than the usual three or so.
(I do not know whether the message got through.)</p>
<p>The new quizzes address some weaknesses I saw in the student body of
knowledge last year, and indeed have seen in previous years too: a
general lack of a robust mental model (or possibly
“<a href="https://doi.org/10.1145/2483710.2483713">notional machine</a>” of
computation. To try to address this, I taught a specific dialect of
pseudocode in the introductory lecture (nothing particularly esoteric;
in fact, essentially what is provided by
the <a href="https://ctan.org/pkg/algorithmicx?lang=en">algorithmicx</a> LaTeX
package). I then also wrote a small interpreter in emacs lisp for
that pseudocode language (with a s-expression surface syntax, <em>of
course</em>) and a pretty-printer from s-expressions to HTML, so that I
could randomly generate blocks of pseudocode and ask students
questions about them: starting with the basics, with sequences of
expressions, and introducing conditionals, loops, nested loops, and
loops with break.</p>
<p>The results of this quiz were revealing; at the start of the cycle,
many students were unable to answer questions about loops at all –
perhaps unsurprising as the complete description of the loop syntax
was only given to the students in the <em>second</em> lecture. Even those
who had intuited the meaning of the loop form that I was using had
difficulty with nested loops, and this difficulty remained evident all
the way to the end of the quiz period. (By way of comparison,
students were able to deal with quite complicated chains of
conditionals with not much more difficulty than straight-line
pseudocode.) It will be <em>very</em> interesting to see whether the reforms
and extensions we have put in place to our first-year curriculum will
make a difference to this particular task next year.</p>
<p>Of course, once I had the beginnings of a pseudocode interpreter and
pretty-printer, I then got to use it elsewhere, to the point that it
has now grown (almost) enough functionality to warrant a static
analyser and compiler. I'm resisting, because fundamentally it's
about generating questions for multiple-choice quizzes, not about
being a language for doing anything else with – but with an even
fuller model for the computation, I could make something akin to Paul
F. Dietz' Common Lisp random tester
(<a href="https://twitter.com/PaulFDietz/status/948912322122272768">still active</a> after
<a href="https://web.archive.org/web/20170630015435/http://www.advogato.org/person/crhodes/diary/48.html">15</a> <a href="https://international-lisp-conference.org/2005/media/dietz-slides.pdf">years</a>)
which would probably have helped me spot a mistake I made when
generating multiple-choice answers to questions about recursive
algorithms, of the form “which of the following expressions replaces X
in order to make this function return the sum of its two arguments?”.</p>
<p>As well as the quizzes, the students have done six
automatically-marked lab exercises and one technical peer-assessment.
My direct involvement in assessment after setting all of these
exercises has been limited to checking that the results of the
peer-assessment are reasonable, by examining cases with outliers at a
rate of maybe 6 minutes per student. Indirect involvement includes
delivering lectures, answering questions face-to-face and on the
forum, system administration, and writing code that writes code that
writes summary feedback reports; this is probably a higher-value use
of my time for the students than individualized marking; in that time,
the students have received, on average: right-or-wrong judgments on
330 quiz questions (many of which have more than one possible right
answer); 45 individual judgments on a moderately open algorithmic
task; 50 marks and feedback on programming tasks; and a gentle helping
of encouragement and sarcasm, in approximately equal measure, from me.</p>
<p><a href="http://christophe.rhodes.io/notes/blog/posts/2018/algorithms_and_data_structures_term1/marks-hd.png"><img src="http://christophe.rhodes.io/notes/blog/posts/2018/algorithms_and_data_structures_term1/marks-hd.png" width="800" height="600" alt="2017-18 A&DS term one coursework marks" class="img" /></a></p>
<p>The coursework marks are encouraging; there are a cluster of students
at the top, and while the lower tail is disappointingly substantial it
is artificially enhanced by a number of students who have (for
whatever reason) dropped out. Limiting the analysis to those students
who missed at most one assessment gives a more pleasing distribution;
almost no-one who attempted everything received a failing mark, though
I should temper my satisfaction with that by saying that I need to be
careful that I'm not simply giving credit for being able to show up.
(There are also some marks from this picture in the middle range,
50-70, which are withheld while allegations of plagiarism and/or
collusion are resolved.)</p>
<p>And now, as I write, it is the last working day before term starts
again, when the students return to find this term's <s>lovingly
prepared</s> hastily assembled activities. Onward!</p>
analysing algorithms and data structures datahttp://christophe.rhodes.io/notes/blog/posts/2017/analysing_algorithms_and_data_structures_data/
http://christophe.rhodes.io/notes/blog/posts/2017/analysing_algorithms_and_data_structures_data/
algorithms-and-data-structuresgoldsmithsteachingSat, 17 Jun 2017 23:12:17 +02002017-06-17T21:13:43Z<p>As I wrote from a train, when
<a href="http://christophe.rhodes.io/notes/blog/posts/2017/karatsuba_multiplication_in_sbcl/">discussing Karatsuba multiplication</a>:
this academic year, I co-taught a module in Algorithms & Data
Structures, given to our second-year undergraduates
studying
<a href="http://www.gold.ac.uk/ug/bsc-computer-science/">Computer Science</a>
or
<a href="http://www.gold.ac.uk/ug/bsc-games-programming/">Games Programming</a>.
The syllabus covered a range of classic algorithms ranging from simple
recursive numerical or linked-list computations, through searching and
sorting, to string matching, dynamic programming
and
<a href="https://en.wikipedia.org/wiki/A*_search_algorithm">graph traversal</a>,
with some extra non-examinable stuff on the side; data structures
included standard linear structures, hash tables, and a general
discussion
of <a href="https://en.wikipedia.org/wiki/Suffix_tree">suffix trees</a>.</p>
<p>I put in a <em>lot</em> of effort in making coursework for the students. Ten
multiple-choice quizzes, each with about ten questions, each question
needing a hundred variants or so to prevent too much cheating; six
automatically-marked lab assignments, small implementations of
algorithms or data structures (each with unit tests written in both
Java and C++, because <em>of course</em> the two different degree programmes
use two different teaching languages); two peer-assessment exercises,
two independent essays (each on one of nine distinct topics, again to
prevent too much “sharing”...). Most of the coursework was
automatically graded – the students answered an average of 350
multiple-choice questions over the 10 weeks – but even so there were
200 two-page essays to grade and give feedback on, as well as guidance
for the peer assessments: and the need to write quiz questions often
kept me up on Sunday evenings well beyond the point of
<s>sanity</s>alertness.</p>
<p>However, all of that is now over and the results are in. (If you’re
one of my students reading this between the time of writing and the
time you receive your results: sorry!). One question might be: was
all that effort worth it?</p>
<p>I don’t really believe that it’s possible to measure the effectiveness
of teaching through student satisfaction scores or surveys. Indeed,
as is often pointed out
(<em>e.g.</em>
<a href="https://books.google.co.uk/books?id=gCQlDwAAQBAJ">Burke, Crozier and Misiasze</a>,
chapter 1), the process of learning is inherently uncomfortable,
involving as it does a period of uncertainty and doubt: and students
can’t necessarily judge how well they themselves have learnt. So
looking at numerical scores from module evaluation surveys is pretty
much doomed to meaninglessness, and even reading written feedback in
detail needs to disentangle entertainment by a lecturer from learning
facilitated by a lecturer.</p>
<p>But it’s also not really possible to measure the effectiveness
directly through marks. I can’t look at the marks students have
achieved in the coursework I set: their marks are directly related to
the pre-published marking schemes and assignment weightings, which
will influence the students’ allocation of effort: and also, since I
wrote the marking schemes and came up with the weightings, there’s a
clear conflict in using the results as a judgment on whether the
students have in fact learnt anything. Nevertheless, for the sake of
completeness, here’s how the cohort did in their courseworks:</p>
<pre><code>R> d <- read.csv("/tmp/is52038b.csv", na.strings=c("ABS", "NS"))
R> ggplot(d) + \
geom_histogram(aes(x=Final, y=..density..), color="black", fill="white", breaks=c(0, 34.5, 39.5, 49.5, 59.5, 69.5, 100)) + \
theme_bw() + \
theme(panel.grid.major=element_blank(), panel.grid.minor=element_blank()) + \
xlab("mark") + \
theme(axis.title.y=element_blank(), axis.ticks.y=element_blank(), axis.text.y=element_blank()) + \
stat_bin(aes(x=Final, y=..density..*0, label=..count..), breaks=c(0,34.5,39.5,49.5,59.5,69.5,100), geom="text", vjust=-1, size=2)
</code></pre>
<p><a href="http://christophe.rhodes.io/notes/blog/posts/2017/analysing_algorithms_and_data_structures_data/is52038b-final.png"><img src="http://christophe.rhodes.io/notes/blog/posts/2017/analysing_algorithms_and_data_structures_data/200x-is52038b-final.png" width="200" height="150" alt="histogram of second coursework marks" class="img" /></a></p>
<p>However, coursework should ideally have a formative effect.
(<a href="http://www.qaa.ac.uk/en">Quality practitioners</a> may wish to divide
all assessments into separate “formative” and “summative” categories,
but I believe that, with our students, having many notionally
“summative” assignments with small mark allocations to strongly
encourage students to actually do them is highly likely to help
students with their major assessments: in other words, performing the
function of formative assessment.) But formative for what?</p>
<p>Well, there’s an exam. And conveniently enough, my co-lecturer and I
wrote the exam (<em>and</em> its marking scheme) before the students engaged
in the coursework activities I described above. (They had done a
perhaps more traditional workshop- and paper exercise-based coursework
in the first term.) So we could examine whether there is any
correlation between the mark achieved in the courseworks, using that
as some kind of measure of engagement with the activities in question,
and the final exam...</p>
<pre><code>R> summary(lm(Exam~cwk1+cwk2, data=d))
Call:
lm(formula = Exam ~ cwk1 + cwk2, data = d)
Residuals:
Min 1Q Median 3Q Max
-33.465 -10.065 0.169 10.333 40.334
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 20.67228 6.64182 3.112 0.00252 **
cwk1 -0.04572 0.08655 -0.528 0.59870
cwk2 0.43467 0.07275 5.975 5.06e-08 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 14.3 on 86 degrees of freedom
(10 observations deleted due to missingness)
Multiple R-squared: 0.3088, Adjusted R-squared: 0.2928
F-statistic: 19.21 on 2 and 86 DF, p-value: 1.264e-07
</code></pre>
<p>This simple regression, ostensibly showing a strong link between
coursework 2 performance and exam performance, is far from conclusive.
There are many factors involved in a student’s performance in
assessment; students know their (provisional) coursework marks before
they sit the exam, and might therefore choose to prioritise other
activities if they are effectively guaranteed a passing grade anyway –
and in any case it’s not clear what the ideal correlation between
coursework and exam assessment might be: it’s probably larger than
zero, on the basis that there is probably some element of aptitude or
student engagement that remains constant – but it’s also probably
smaller than one, because with perfect correlation what would be the
point in separate assessments?</p>
<p>Nevertheless, I’m pretty pleased with how it all turned out, and I’ll
be repeating and extending the approach, with an even larger number of
small assessments with immediate feedback, in the next academic year.</p>
<p>(As well
as
<a href="https://jobs.gold.ac.uk/vacancy/lecturer-s-316041.html">lecturers</a>,
Goldsmiths Computing is currently recruiting
a
<a href="https://jobs.gold.ac.uk/vacancy/reader-in-data-science-316055.html">Reader</a> and
a
<a href="https://jobs.gold.ac.uk/vacancy/postdoctoral-researcher-in-data-science-316887.html">post-doctoral researcher</a> in
Data Science. If you think I’m being awfully naïve by modelling
marks, bounded between 0 and 100, with an unbounded error function,
tell me about it and join in the fun!)</p>
karatsuba multiplication in sbclhttp://christophe.rhodes.io/notes/blog/posts/2017/karatsuba_multiplication_in_sbcl/
http://christophe.rhodes.io/notes/blog/posts/2017/karatsuba_multiplication_in_sbcl/
advogatoalgorithms-and-data-structureseuropean-lisp-symposiumlispmultiplicationsbclSun, 02 Apr 2017 22:45:22 +02002017-04-02T20:48:17Z<p>Possible alternative title: I’m on a train!</p>
<p>In particular, I’m on the train heading to the
<a href="http://www.european-lisp-symposium.org/">European Lisp Symposium</a>,
and for the first time since December I don’t have a criticially
urgent piece of teaching to construct. (For the last term, I’ve been
under the cosh of attempting to teach Algorithms & Data Structures to
a class, having never learnt Algorithms & Data Structures formally,
let along properly, myself).</p>
<p>I have been giving the students some structure to help them in their
learning by constructing multiple-choice quizzes. “But
multiple-choice quizzes are easy!”, I hear you cry! Well, they might
be in general, but these quizzes were designed to probe some
understanding, and to help students recognize what they did not know;
of the ten quizzes I ran this term, several had a period where the
modal mark in the quiz was zero. (The students were allowed take the
quizzes more than once; the idea behind that being that they can learn
from their mistakes and improve their score; the score is correlated
to a mark in some minor way to act as a tiny carrot-bite of
motivation; this means I have to write <em>lots</em> of questions so that
multiple attempts aren’t merely an exercise in memory or screenshot
navigation).</p>
<p>The last time I was on a train, a few weeks ago, I was travelling to
and from Warwick to sing Haydn’s Nelson Mass (“Missa in angustiis”;
troubled times, indeed), and had to write a quiz on numbers. I’d
already decided that I would show the students the clever Karatsuba
trick for big integer multiplication, and I wanted to write some
questions to see if they’d understood it, or at least could apply some
of the results of it.</p>
<p>Standard multiplication as learnt in school is, fairly clearly, an
Ω(d<sup>2</sup>) algorithm. My children learn to multiply using the
“grid method”, where: each digit value of the number is written out
along the edges of a table; the cells of the table are the products of
the digit values; and the result is found by adding the cells
together. Something like:</p>
<pre><code> 400 20 7
300 120000 6000 2100
90 36000 1800 630
3 1200 60 21
</code></pre>
<p>for 427×393 = 167811. Similar diagrammatic ways of multiplying (like
[link]) duplicate this table structure, and traditional long
multiplication or even the online multiplication trick where you can
basically do everything in your head all multiply each digit of one of
the multiplicands with each digit of the other.</p>
<p>But wait! This is an Algorithms & Data Structures class, so there
must be some recursive way of decomposing the problem into smaller
problems; divide-and-conquer is classic fodder for Computer
Scientists. So, write a×b as
(a<sub>hi</sub>×2<sup>k</sup>+a<sub>lo</sub>)×(b<sub>hi</sub>×2<sup>k</sup>+b<sub>lo</sub>),
multiply out the brackets, and hi and lo and behold we have
a<sub>hi</sub>×b<sub>hi</sub>×2<sup>2k</sup>+(a<sub>hi</sub>×b<sub>lo</sub>+a<sub>lo</sub>×b<sub>hi</sub>)×2<sup>k</sup>+a<sub>lo</sub>×b<sub>lo</sub>,
and we’ve turned our big multiplication into four multiplications of
half the size, with some additional simpler work to combine the
results, and big-O dear! that’s still quadratic in the number of
digits to multiply. Surely there is a better way?</p>
<p>Yes there is.
<a href="https://en.wikipedia.org/wiki/Karatsuba_algorithm">Karatsuba multiplication</a>
is a better (asymptotically at least) divide-and-conquer algorithm.
It gets its efficiency from a clever
observation[<a href="http://cr.yp.to/papers/m3.pdf">1</a>]: that middle term in
the expansion is expensive, and in fact we can compute it more
cheaply. We have to calculate
c<sub>hi</sub>=a<sub>hi</sub>×b<sub>hi</sub> and
c<sub>lo</sub>=a<sub>lo</sub>×b<sub>lo</sub>, there’s no getting
around that, but to get the cross term we can compute
(a<sub>hi</sub>+a<sub>lo</sub>)×(b<sub>hi</sub>+b<sub>lo</sub>) and
subtract off c<sub>hi</sub> and c<sub>lo</sub>: and that’s then one
multiply for the result of two. With that trick, Karatsuba
multiplication lets us turn our big multiplication into three
multiplications of half the size, and that eventaully
<a href="https://en.wikipedia.org/wiki/Master_theorem">boils down</a> to an
algorithm with complexity Θ(d<sup>1.58</sup>) or thereabouts. Hooray!</p>
<p>Some of the questions I was writing for the quiz were for the students
to compute the complexity of variants of Karatsuba’s trick: generalize
the trick to cross-terms when the numbers are divided into thirds
rather than halves, or quarters, and so on. You can multiply numbers
by doing six multiplies of one-third the size, or ten multiplies of
one-quarter the size, or... wait a minute! Those generalizations of
Karatsuba’s trick are <em>worse</em>, not better! That was completely
counter to my intuition that a generalization of Karatsuba’s trick
should be asymptotically better, and that there was probably some
sense in which the limit of doing divide-bigly-and-conquer-muchly
would turn into an equivalent of FFT-based multiplication with
Θ(d×log(d)) complexity. But this generalization was pulling me back
towards Θ(d<sup>2</sup>)! What was I doing wrong?</p>
<p>Well what I was doing wrong was applying the wrong generalization. I
don’t feel too much shame; it appears that Karatsuba did the same. If
you’re
<a href="https://en.wikipedia.org/wiki/Toom%E2%80%93Cook_multiplication">Toom or Cook</a>,
you probably see straight away that the right generalization is not to
be clever about how to calculate cross terms, but to be clever about
how to multiply polynomials: treat the divided numbers as polynomials
in 2<sup>k</sup>, and use the fact that you need one more value than
the polynomial’s degree to determine all its coefficients. This gets
you a product in five multiplications of one-third the size, or seven
multiplications of one-quarter, and this is much better and fit with
my intuition as to what the result should be. (I decided that the
students could do without being explicitly taught about all this).</p>
<p>Meanwhile, here I am on my train journey of relative freedom, and I
thought it might be interesting to see whether and where there was any
benefit to implement Karatsuba multiplication in
<a href="http://www.sbcl.org/">SBCL</a>. (This isn’t a pedagogy blog post, or an
Algorithms & Data Structures blog post, after all; I’m on my way to a
Lisp conference!). I had a go, and have a half-baked implementation:
enough to give an idea. It only works on positive bignums, and bails
if the numbers aren’t of similar enough sizes; on the other hand, it
is substantially consier than it needs to be, and there’s probably
still some room for micro-optimization. The results?</p>
<p><a href="http://christophe.rhodes.io/notes/blog/posts/2017/karatsuba_multiplication_in_sbcl/fit.png"><img src="http://christophe.rhodes.io/notes/blog/posts/2017/karatsuba_multiplication_in_sbcl/fit.png" width="800" height="600" alt="Linear model fit for built-in and Karatsuba multiply" class="img" /></a></p>
<p>The slopes on the built-in and Karatsuba mulitply (according to the
linear model fit) are 1.85±0.04 and 1.52±0.1 respectively, so even
million-(binary)-digit bignums aren’t clearly in the asymptotic
régimes for the two multiplication methods: but at that size (and even
at substantially smaller sizes, though not quite yet at Juho’s
<a href="https://www.snellman.net/blog/archive/2004-08-22.html">1000 bits</a>) my
simple Karatsuba implementation is clearly faster than the built-in
multiply. I should mention that at least part of the reason that I
have even <em>heard</em> of Karatsuba multiplication is Raymond Toy’s
<a href="https://www.cons.org/cmucl/news/2002.html">implementation</a> of it in
<a href="http://cmucl.org/">CMUCL</a>.</p>
<p>Does anyone out there use SBCL for million-digit multiplications?</p>