I’ve written already about the type of proposal that is likely to be accepted to linux.conf.au, this is a discussion of how the process worked.
Our process aims to find a good set of talks. Past conferences have asked for written papers too, but we do not believe they are widely read and some authors have simply not sent them in, which is possibly unfair to people who believed the given requirements and wrote their paper. This year we didn’t ask. By not asking for papers, conferences like linux.conf.au are missing one opportunity to actually check that our speakers have had more than a paragraph worth of thoughts concerning their talk. Hence the emphasis on known good technical quality and known speaking ability in the criteria.
I’d like to make a quick comparison here with academic computing conferences. Firstly to clear up a common misconception about academic conferences: people don’t just read their papers out loud; or at least not in computing. I’m told they do in philosophy. It’s meant to be an engaging narrative about a problem and its solution, much like a technical conference talk. (Both types of conferences have speakers that fail at this.) The selection is very different though: for an academic conference you submit an abstract or a full academic paper, usually in the 8–15 pages range, and selection is usually based entirely on the quality of the research as demonstrated in the paper, rather than on your history as an engaging or hugely popular public speaker. And the papers are actually important, in computing they will contain (or ideally contain) enough details to allow people to replicate the research (in traditional experimental science, that stuff goes in journals, in computing journals tend to contain only very serious and really stellar work). People wanting to do serious critiques of the work or to extend it will refer extensively to the paper; the paper matters in the way that code does in Free Software. Reviewers will study the paper in detail: ten conference papers would be a very very high reviewing load for a single conference.
This year all program committee members were asked to review all proposals. We voted on them, literally, on a scale of 1–5, which I personally interpreted as please no
through to I will die if we reject this
, although other reviewers may have calibrated differently. We did not provide feedback that was intended for the authors. We did not, therefore, do what would be called peer review
, which is about extensive constructive criticism of the work suggesting ways to improve it, even if it is being rejected. That’s expensive for reviewers and would require drawing reviewers from a broader range of backgrounds: the kind of expertise required to say this talk is not terribly exciting
is not the same as the expertise required to write a letter to the author suggesting technical improvements to their work. I called the linux.conf.au process Am I hack or not?
initially, although since our acceptance rate is about 25% this turned out to be unfair to people who were rejected. Many were actually hack.
That acceptance rate does have certain effects when it comes to our criteria. We are not able to take many chances on people without a track record. We do not have the reviewing manpower to make any useful suggestions to people about their work or their talk proposal, although this would be possible with some other processes we could have used. The abstracts length for this conference makes proper peer review impossible (we could offer suggestions about making a better abstract, but not about doing better work as such even if we had the manpower). We can aim to possibly only select good or excellent talks.
I’ll be interested to compare the PyCon process, particularly since they’re pattern nuts and have found a series of patterns around which you can organise your committee meetings. I have to say an occupation hazard of doing these things is that you really want to go to the conference afterwards. I’d kill to go to PyCon now, if it wasn’t that that wouldn’t help me get a ticket to Texas one bit.
In other news, the linux.conf.au programme is available. Here’s talks I’m particularly looking forward to:
- The Kernel Report (Jonathan Corbet)
- Fixing suspend for fun and profit (Matthew Garrett)
- Digital Preservation – The National Archives of Australia, Open Standards and Open Source (Michael Carden) [although unfortunately this is up against Val Henson, who I’d also like to see]
- The OzDMCA: what it means for FOSS (Kimberlee Weatherall)
- Tutorial:GIMP Uncovered: Understanding Images and Image Editing (Akkana Peck) [I’ll have to catch either Kimberlee Weatherall or Akkana Peck on video though, another clash]
- Starting an Open Source business (Paul Fenwick)
- How to Herd Cats and Influence People (Jono Bacon)
- Concurrency and Erlang (André Pang)
- Making Sausage: How the OLPC Machine Was Designed (Jim Gettys)
Andrew has already put his hand up for the cricket match and he doesn’t even have permission to take the leave yet.
The linux.conf.au review process by Mary Gardiner is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.