So the way it works is there are a
number of big observatories around the
world and there are a number of
astronomers around the world, and
obviously they all want to use the big
telescopes and there's- you know more of
them wanting to use the telescopes than
there is time available so we have to
have some mechanism for deciding who
actually gets to use the telescopes,
what science gets done. It depends on the
telescope. So, for something like the
Hubble Space Telescope, which is one of
the premier facilities out there, then
you're probably talking over
subscription rates of about a factor of
10; so there's- people want ten times as
much time on the telescope as there is
available. For some of the littler
telescopes it gets down to a factor of
two or three, you have to fight off the
other astronomers before you get to use
it. We have this standard mechanism that
gets used for lots of different
things, whether it's assessing qualities
of grant applications or scientific
papers or whatever called peer review;
which essentially means that whatever
you propose to do, whatever you write,
gets sent to other astronomers who kind
of weigh it up against all the other
applications. In using telescopes, the way
this works is: probably twice a year, is
the usual way these things work, there's
a call for applications and then there's
a peer review panel - basically a
collection of astronomers who come from
the community - and the peer review panel
sifts their way through all these
applications, grades them all, figures
out which of them get the telescope time.
They're supposed to be these honest
brokers in this process, these
disinterested parties. And, you know, if
if there's an obvious conflict of
interest like, you know, if it's your own
telescope application obviously you
don't get to assess it. So there are ways
for dealing with very direct conflicts
of interest but really they're just
people that the community puts trust in
if you like. It varies a bit from panel
to panel, they all tend to work in
different ways. Some of them you actually
sort of give a grade to the to the
proposals before you even meet, and then
you get together and you compare your
grades and kind of refine them from
there; others you sort of start from
scratch in the meeting. In some of them
everyone assesses all the proposals in
detail, in others there's sort of one or
two members of the panel that are kind of the
lead for each application. You have to
give a score in the end, because you have
to decide, you know, which one's better
than which one; so you need some way of
actually ordering things to decide which
ones actually get the telescope time. The
problem is just the amount of work
involved. And this thing came to a head a
few years ago when I was on the ESO
telescope application panel, it's a thing
called the OPC. And so this is the- this
was the the pile of papers I received
which is just one set of proposals to
look at. It's about a hundred proposals,
slightly over 100 proposals. It's this huge mass of
papers and, as you can see, I've scribbled
on the fronts of them just because as I'm
going through I'm scoring them, trying to
decide you know which ones I like, which
ones I don't like even before I get to
the meeting. The problem with this is
it's such a big pile of papers that
actually even if- you know, it takes
what- it takes me about a week to go
through this lot, it's a weeks solid work
and then I go off and do the panel and
that's another weeks solid work. So it's
two weeks work just to do this process.
And the problem is even if I do as
conscientious a job as I can; I score the
first 1, by that time I've got to
number 97 I can't remember anything
about number 1 and so trying to score
number 97 on the same scale that I
scored number 1 - even if I do you know
really a conscientious job you just
can't because you just can't hold that
much stuff in your head at once. What got
me to thinking about this whole thing is
actually when a pile like this arrived
on my desk and it was one of the- you
know, it's the classic displacement
activity in that, you know, when something
like that comes along you find all sorts
of other things to do and so you know
you go tidy your office and do all those
kinds of things. But eventually I'd run out
of displacement activities and so rather
than actually get on with assessing it I
started trying to think, is there some
better way we could be doing this? The
scheme I came up with is kind of a way
of distributing the process amongst the
astronomers; instead of having a little
panel to do this let's have the whole
community do it. The people who apply
should be the people who actually do
the, do the assessment. And one of
the other issues is; because astronomers
do recognise this process is a little
bit of a lottery as to whether or not
you get time, there is this temptation to
say I'll put five proposals in because
that way I'm more likely to get
telescope time. And at the moment there's
no penalty really for doing that - you
annoy the panel a bit because they see
your name coming up lots of times and
you start irritating them, but actually
there isn't really a penalty. Whereas if
you could come up with a scheme where, if
you put a load of applications in you
get a load of work back to do the
assessment. The idea I came up with was
this: each application that comes in gets
sent out to say 5, 6, 7 other
people who've applied. So each person
will - who applies - will get a pile of say
seven or eight other people's
applications to look at. All they have to
do is actually go through that list, go
through that pile of applications
they've got. So they've only got- they haven't
got 100, they've only got seven or eight
to deal with so it's a manageable thing;
you can actually really read that seven
or eight proposals very carefully. The
only thing they then have to do
is rank them - saying this is the best one,
this is the worst one. They don't give
them scores, they don't give them anything
else; this is the one I like the most,
this is the one I like the least and this is
the order in between. And so everyone's
looked at a different combination of
seven or eight applications, because
they've been kind of randomly
distributed and obviously you don't get
your own one back and you have to be a
bit careful you make sure you don't get
anyone from the same institution as you;
so it's a little bit of care taken to
make sure that there's not you know no
chance for dishonesty in this process.
And it turns out this is a solved problem - how
you actually do that. It's a thing called
a board account. You've got a-
you've got these individual little ranked
lists and you've got to- somehow got to
compile into this global list; that
essentially means that you've got to
give every proposal a score that you
can then order so sort of the thing with
the highest score ends up at the top and
the thing with the lowest score ends up
at the bottom. And this modified board
account is actually very simple, in that
all you do is, supposing you had eight
proposals each person had eight
proposals to look at, all they do is they
rank the list they then give 7
points to the top one, 6 to the next,
5, 4, 3, 2, 1, 0. So
everything ends up with a score between
0 and n minus 1, where n's the number of
applications you looked at, okay? Each
application has been looked at by eight
different people, that means that there
are eight scores for each proposal
somewhere in that ranked list; so
somewhere between 0 and 7, you just add
them up. So then the sum of those then
gives you the score that each
applications get. And obviously if you're
near the top in everyone's little
ranked sub list you'll get something close
to 7 in each one, you'll end up with a nice
high score; if you're near the bottom
you'll end up with something close to 0.
That then gives you a score you can then
globally combine to create this master
list of all the applications. And
actually there's an astronomical
connection in this in that the guy
Borda who invented this was an
astronomer. And the reason why he
invented it is because he thought that-
he was, he was trying to get into the
National Academy in France and he
thought that the voting was being rigged
against him. So he got very interested in
electoral theory and how you can
actually come up with ways of dealing
with these kinds of things to actually
make sure that, you know that final vote
ends up honest. And so this this modified
Borda count that I ended up using to
rank all these things together actually
came from him but obviously applied to
something rather different here, how you actually
rank all these proposals together.
(Brady: Call me a cynic, but one of the first things)
(that comes into my head is that if I'm)
(in this contest and I see one that I)
(think is outstanding and definitely is)
(going to beat mine; I would have an)
(interest in penalising)
(that one and putting it at the bottom of)
(my list.) - So here's the trick; and in fact-
so this gets into this whole thing
called mechanism design, of how you stop
people gaming systems or at least how
you make them game the system in the way
you want them to do; which is the
following: that once you've created this
global list you then go back to
everyone's little sub lists, their own
little seven or eight that they've
assessed and you see whether the global
list, where those seven or eight things
appear in the global list, more or less
agree with where they- the order that
they put them in. Not completely because
as I say once in a while people will
disagree and all those kinds of things,
but actually the thing that you ranked the top
kind of appears somewhere near the top of
the global list; the thing that you ranked
at the bottom appears somewhere near the
bottom of the global list and so on.
Because that's an indication that
actually the way you ranked it more or
less agreed with everybody else. So
that's fine; but if you find somebody
where- you know, everybody else who looked
at that proposal said it was really good
but you said it was rubbish, you start
suspecting well maybe there's some game
that's been going on exactly along these
lines that maybe it was your competitor
and you were trying to do them down. So
if your little sub ranked list disagrees
with the global list, and it looks like
you've been playing their game, your
proposal moves down the pile. So you have
an incentive to be honest because
actually you're invested in this process,
you want your proposal to be in as near
the top of the pile as possible; if you
agree more or less with what everybody
else assessed your application moves up
a bit, if you disagree with every what
everybody else says about the proposals
your application moves down the list. So
you actually have a strong incentive not to cheat.
If you're interested in astronomy and would like to learn more
about deep space and the world's best
telescopes, why not check out our sister
channel called Deep Sky Videos. We go
inside all sorts of amazing
observatories there. And if you've got
more questions about this video - don't
worry, so did I - and you can hear some of
them answered in an extra film we've
already posted over on Deep Sky Videos.
So this in truth- this is the the main
criticism that's been levelled at this as
a way of doing things, and I- I'm
happy to concede it is a point...