(techno sound)
(techno music)
- If you ask, how does the brain work
from an engineering perspective, then you might ask,
what does the circuit diagram look like?
After all, if complex modern processors
are designed and understood by engineers,
then maybe a simple brain could be
likewise understood at that level.
So, let's consider exactly that, a simple processor.
And if you take the lid off, you'll find
underneath the die, the chip.
Looking at it a little bit closer
you'll see that it's broken up into various modules
for processing, for input from various sources
and for outputs, memory and likewise in the control.
If you'd look at a small region and zoom in on it
you'll find it's covered by a vast mesh of tiny, tiny wires.
This is about maybe a dozen layers thick,
underneath it all there are the actual switches,
the transistors, which do the switching on this.
Now if you want to make it, you just need
a few billion dollars, machines like this,
and, you know, you're good.
If you want to reverse engineer it,
then maybe you could use some machines like this
as one example, which I'll be discussing
in more detail, called a FIB SEM.
Now, on the other end, here's another processing unit.
Let's take the cover off and underneath there's a brain.
And it, too, has various modules in it, you know,
optical inputs, they're called optic lobes,
antenna inputs for smell, memory units,
they all have different names, mushroom bodies,
ellipsoid bodies, and so forth, studied by biologists.
So you can also zoom in and look very carefully
and presumably there's also going to be
a vast mesh of wiring in there,
and somehow buried in underneath all of that
should be synapses, which are switching
between all of these different wires.
Now if you want to make one of these
you need to invest in an old banana
(audience laughter)
and wait, not much cost and it'll self assemble.
And it's amazing and you've heard about this,
you know, in the first talk and it's truly amazing.
So now if you want to reverse engineer it
we really need, it's a tough problem
and we need tools from across the spectrum,
from biology, from genetics, and I'll talk
a little bit of the aspect that can come in
from the semiconductor industry, that FIB SEM.
So just to put a perspective, let's put some
dimensions on these and compare them side by side.
A current processor is sort of a
flattened structure, two and a half dimensions,
but the volume of it is under a millimeter.
The fruit fly, it's a nice little blob,
under a millimeter in size, but not
too different in the volume.
The number of transistors and switches
in the processor are actually pretty impressive,
about a billion transistors nowadays,
that's a bit more than the fruit fly.
If you go back 10 years, the Pentiums,
they had about a comparable number.
And the last part now is the wire size
and the transistor size and the dendrite size, synapse size,
they're all in the tens of nanometers, very small.
And the challenge is to bridge the gap
between the small size of tens of nanometers
all the way up to a good fraction of a millimeter
and catch that large dynamic range.
So to reliably resolve these transistors,
in all three dimensions, we need to sample,
probably instead of 50 nanometers,
maybe at five or 10 nanometer increments.
These are the voxels that you'd use to form images,
and you've seen before one can sort of collect
many such images in 3D, form a stack.
But if you just quickly look at the numbers,
how many voxels, how many pixels are there,
eight nanometers compared to half a millimeter,
there are lots, it's a good fraction of a petabyte.
And if you're acquiring the data at
a modest rate of a megahertz, then you'd be
acquiring data for eight years,
even for something as simple as a fruit fly plane.
Now I should also point out we do need
to really see the connectivity
one needs to go down to the EM level,
to the electron microscope level.
Light microscopy cannot quite resolve the details
and see the fine connectivity.
Light microscopy can do many other things,
image live, so all these techniques
are very complementary and support each other.
So, what is this FIB SEM?
This is basically an instrument
where you use FIB, which is a focused ion beam
and it basically abrades away the surface
a few nanometers at a time.
The advantage of an ion beam is the abrasion is
just a little bit finer, more controlled
in the Z dimension, otherwise it's very similar to
what Mark was talking about earlier,
the serial block face cutting.
After you abrade, or remove a little layer,
you image the surface with a scanning
electron beam and then you ablate again.
And you cycle through this about every minute.
And you keep going and going
for as long as you possibly can.
And in the end, you gradually reconstruct a circuit.
In this case, if you're looking at a semiconductor chip,
you can see all the metal layers inside.
If one's looking at a fly brain,
then you should get a three dimensional
reconstruction of all the details inside.
So this here is the outline of one of these synapses,
the black part is actually the membrane itself.
Ah, and there you can see.
So what you saw, you know time here is moving in in Z.
And you'll see on the left picture right there
the dark spots, those are actually
the synapses, they're pretty small.
And this is a typical fan in circuit,
there are multiple synapses going all onto one dendrite.
And you see the opposite in the fly, fan ins, fan outs,
there are just a variety of structures that are all visible.
It's more complex than it is for a mammalian brain.
So we got one of these machines and started out.
And so we ran it, three days later
it came to a grinding halt and stopped.
And so we imaged one one thousandths of a brain
and couldn't continue, there are problems.
And so we came to this conclusion.
(audience laughter)
FIBSEM's irrelevant because it can't image anything useful.
I continued to be employed.
(audience laughter)
And worked on the next problem,
five years of reliability engineering,
basically to handle all of these interrupts as they happen.
Can we anticipate things, make graceful stops
if something bad is going to happen,
mitigate damage control and we can
restart without loss of anything.
The bar is very high because if you have a gap
in this dense interconnect of wiring,
there's no way that you can connect the wires
from the left half to the right half
and your circuit diagram is just useless.
So you don't want to be there, imaging several years
into the project and then say oops.
So a few years later, here we are,
imaging now for about a three months project,
that is looking at the optical lobe.
You might have noticed three pieces,
the medulla, lobular and lobular plate.
And this is sort of what you see inside a fly brain.
This is, the motion you see is, again,
is moving in Z, deeper and deeper.
Initially the width was about 120 microns,
it's about the width of two human hairs.
And in zooming in about 10x and we can look at detail
and for example right there, that's a synapse.
And it's connecting this dendrite over to
several other guys on the other side,
you know, it's all very subtle.
And it's a, you know, it's a big challenge to try to
pull all of that information out and segment it.
So it took us three months to image this.
And then we gave it to some other people
and about three years later they processed the data.
Processing it and analyzing it
is by far the bigger challenge.
This is, this is not easy and it requires
a lot of hand effort or did require a lot of hand effort
to segment, curate, make sure there's no mistake in there.
And what we're doing here is dense segmentation.
We're getting pretty much all of the processes,
all of the neurons, identifying them.
And then you can pull them out one by one
and you find that they have their own unique shapes
and sizes, and compare them to what
is known from other experiments.
For example, you know, some of these neurons
are, you know, this is coming in from the retina,
some of them are sensitive to an on transition
or to an off transition or to a left
to right motion or front to back motion
and people can identify that based on optical imaging.
But optical imaging can't tell you
the detailed synaptic connectivity,
but it can give you the classes
of neurons, so one can connect the two
and bootstrap our way forward.
In addition, optical imaging can look at
live samples, too and tell which ones are firing
on on transitions and off transitions
and we can actually monitor that.
So all of that data can be put together
and you can begin to identify the
components of a circuit diagram.
And these are the three major parts
that you saw earlier, the medulla, the lobular
and the lobular plate, up here's the retina.
The thing is with the FIB SEM it allows you
to make the last little connection which is to ask,
here's one neuron which we know reacts
to a left to right on transition, what are its inputs?
And you can define the neurons that are responsible
for that, characterize those independently
and we make this last step.
And then you can begin to test models of the circuit.
Is this a coincidence detector based on excitatory inputs,
is it a different version of that,
a nor detector where you might have
an inhibitory input on one side?
And it turns out that it's a little bit
of a combination of both are present in the fly brain
and you can even sort of get estimates
of what might be causing the delay
based on length on the synapse.
So having this extra level of detail
helps close a lot of open gaps.
So so far we still haven't gotten to the full fly brain.
And we reverted to another technique
called hot knife sectioning to get all the way up.
Imagine the fly brain, you chop it up
into like a loaf of bread into about 30 pieces
each about 20 nanometers thick.
You put it onto a little laminated surface
and you have about 30 of these tabs
and feed them in to a bunch of machines.
And this puts us in the range of getting these,
this eight FIB SEM year effort maybe down to
a little bit more reasonable time.
And we are gearing up and trying to expand
toward this direction here.
And so far, for the past two years
we've come just a little ways along that path.
Looking in here we have what we've
accomplished called the hemi brain.
It aspires to be half a brain but
it's a little bit shy of it, it's on a female fly.
But it's unilateral over a good part of it,
misses some aspects, you know, too much on
the optic lobe, missing the parts below
which are associated with feeding and appetite.
But a lot of modules are in there,
the central complex, which is a sort of
an internal compass for the fly
and people are studying that independently,
olfactory learning and the mushroom body,
lateral horns, all these names are dear to biologists
studying different parts of the fly brain.
And here's what that looks like.
That big round circle, the doughnut
that you saw, that's the ellipsoid body.
And that's associated with its navigation,
sort of an internal compass of the fly.
And here we're just sort of flying through the data
and you can just sort of see the complexity,
here we're getting pretty close at this resolution,
it isn't as beautiful as what Mark has
with the dedicated hi res microscope.
What we're doing right here, as we fly in and out,
that's going across that section.
We have to make sure that section has
good enough quality that we can match the dendrites
from one side to the other without loss and that's critical.
And I'll just let this play a little bit.
(audience laughter)
It's amazing that this thing self wires and it looks
so disorganized, but in the end it really is organized.
And there are the different modules,
we're zooming back out, here you can see
the whole upper head of the fly brain there.
Remember these are several slabs
and we're just going to go through
six of them, we have 13 altogether.
This represents about two years of imaging
that we've done, we've just sort of completed this.
And what will come next after this,
we want to go over, try looking at a male fly,
but doing it now in its entirety, all the way
and that'll be a larger imaging effort.
And here we are back to the central complex.
So what's next?
That data is not good enough by itself,
it needs some processing and it goes off to experts,
computer science experts who use neural nets.
And there have been a lot of fascinating developments
in the past few years of what they can,
how quickly they can extract it.
So they are now maybe at the level where
instead of taking 10 times longer than imaging,
they might almost be able to keep up
and pull out the nerves and get these connectivity diagrams
of what nerve connects to what out of the thousands,
hundreds of thousands of different nerves in there.
And give those to biologists so that can be correlated
with all of the other information that they work with.
So I'd like to put this technology,
this FIB SEM, on a little diagram like this,
just to show where it might fit in into the world.
The bottom axis is size, we're looking at
relatively larger size structures,
it's getting close to a millimeter,
a little bit below that, for the fly brain.
And the vertical axis is resolution.
There's some techniques out there, as Mark talked about,
which can get much better resolution,
but the size is going to be a little bit more limited.
And we're playing this trade off between
size and resolution and I think FIB SEM has
a little bit of a sweet spot and fly brain
sort of demands some of those attributes.
Now the FIB SEM can also be used for,
in a different modality, where you can
operate it with lower current.
And in there one might be able to image,
let's say a whole cell, with high resolution.
The whole cell's smaller in volume
with a higher resolution and images instead of
being blurry like this, can be somewhat crisper.
And we can see more detail of the synapse
in the bottom slide over here, like right down here.
Post synaptic vesicles over here,
individual ribosomes can be imaged
and at the top we can even see microtubules.
This is just a random piece of mammalian cortex.
And one can take out a part of that,
segment it, similar to what Mark has shown,
and you can get all the intercellular organelles,
endoplasmic reticula, the mitochondria
and mine it for, you know, for various details.
Or at the bottom maybe even label all the vesicles,
you know, it's a little bit up to the biologist
how they begin to mine this kind
of information, but it is accessible.
So where do we go from here?
I think next, you know, we'll continue
on trying to acquire the fly brain atlas,
that's going to be a little bit of a longer term effort.
And, you know, that'll be then combined
with all the neural processing,
you know, to get the synaptic details
and the connectivity diagram and then
that'll be absorbed by other scientists to,
with light imaging and behavior to
really understand the circuitry.
I think going on to the next level beyond that,
after we have one nice, complete brain,
you begin to look at differences.
And I think this will be another exciting phase
for this field, where we look at fly to fly difference.
In the semiconductor industry if you want to do
failure analysis, you do die to die,
two computer chips, take two images,
subtract them from another, if there's a difference
that's an error, you throw it out,
or you know where the problem is.
Here we have to do a similar version, fly to fly.
And so and the differences could be
in development stage, not being overwhelmed
by all the background, or is there experience,
is there a memory, can we pull it out,
is it going to be represented in this EM data or not?
But I think it'll be just, you know, a fascinating time.
Thanks for your attention.
(applause)
(techno music)