[MUSIC]
I'm very excited here this afternoon to actually address
one of my favorite questions that are
coming up and I get asked this a lot.
What is the difference between
machine learning and artificial intelligence?
I'm actually not going to
answer this question directly today,
but hopefully after my talk,
every one of you will start
to form some opinion about it.
We live in the golden age of
machine learning and it's very
exciting to see how we
continue making new speaking boundaries.
Vision being the first domain data,
I would argue we conquered it.
No machine learning has achieved
supernatural abilities to track, identify,
and essentially just follow
any object that's it comes
to where it says overview of vision.
The next domain data we started
conquering is in language.
Now computer machine translations are capable of
translating texts across several hundreds of language.
We have machines capable of generating new content.
Also just generating live caption
as I speak my personal favorites of course is AlphaFold2
which was announced last year and not only did it
essentially solve for this grand
challenging biology of predicting protein folding,
it actually ushered in a exciting new age of
machine learning aided drug and material discovery,
which you can imagine will just accelerate all progress.
It's almost magical what
we can achieve in machine learning.
But I hate to say this.
Unfortunately, the magic here is very
very expensive for specialized hardware and the data,
lots, lots, lots of data as
my colleague Sujis already pointed at.
In fact, right now in this trend of what we
call the so called large data and large model trend.
As transformer models first
gets popular in the early 2018s,
you can see this trend shown here in this graph.
Last example being recently this week
announced the Megatron-Turing model by Microsoft.
This model have 500 billion parameters.
Just let that sink in for a little bit.
Also I want to draw your attention to this red line.
Don't let that deceive you.
It looks straight and linear,
if you pay attention to y-axis you realize
that it's exponential growth.
If you think about it,
the counter of resources,
goes into 21 of these models.
It's not achievable by just any entity.
If we continue this trend,
we'd literally going to run out of resources
on Earth training these large models.
As we invest so heavily in
all these machine learning models,
they're not always well.
It's not the very hard to actually find the examples
of where machine learning just fall flat to short.
In this example that came out by fuel in 2012,
they proposed a set of synthetic visual reasoning tests.
The test is very simple.
You can essentially break it into two categories
of essentially procedurally generated images.
You can think about the problems to come in two types.
One is the sameness difference.
On the first image on the left side,
you can see that
the two shapes are actually
the same shape, it's just translated.
This is the same as class.
The one below it,
it will be a difference class
at the two shapes that actually
different in variant of transformation.
The next two on the first row is
what we call spatial relationship.
The smaller image is inside the large shape,
and then the one next to it
it's also a spatial relationship.
They are lined in the straight line.
As it turned out, the machine learning
models that can perform perfectly identify
spatial relationships perform rather
poorly at some problems even at
the chance at identifying
sameness and difference type of problems.
Let this sink in for a little bit.
Sameness and difference,
this is the really basic intrinsic concept.
It's not new,
it's nothing like a concept that's only human can grasp.
In fact, even honey bee can.
In this behavioral study
done by Weber and Giurfa in 2013,
they actually tested the honey bee on
this thing called Delayed-Match on feminist.
They use a y-shaped maze that has three gates.
The honey bee would enter
the maze through the stem of the Y on
one gate and have to choose to
exit the maze on the arm of the Y.
Specifically these gates either coded by color or shapes.
In this case, they first trained
the honey bees to follow,
to enter the maze with say yellow color gates,
and then choose to leave the maze was
another yellow colored gate that would be to
prove so thus matching on sameness.
After they successfully trained
the honey bees to do that,
they switched the gates to
actually striations or patterns.
The same honey bee can be trained with very few examples.
Again, grasp on the sameness idea
of a following for example,
two horizontal striped gates to be different.
Honey bee can do this machine learning,
not that so great.
Well, what can we do since
the inception of a machine learning we always take
inspiration from biology and the perceptron
is after a model based on biological neurons.
To that extent, other group actually
collaborated with Gape's group in trying
to understand how dynamics and layout
or geometry really influences
a networks ability to learn and a capacity to learn.
To this extent, we implemented
some realistic dynamics, for example,
inhibition and excitatory neurons,
activations threshold, weight decay,
refractory period, internal propagation
delays and the spike-timing dependent plasticity.
Sparse network architecture goals
have a way of renting from random like Erdos Renyi,
stochastic block model to regular, clique,
grid/lattice to some of biological inspired layouts
like Caenorhabditis elegans and morphological lattice.
Other results both promising and puzzling.
Promising in them,
I'm going to show you the first video
is what the I categorized as promising.
In this video, we use
a randomly initialized the network under
the stimulus we use m-nest digits,
which are 28 by 28,
black and white to digitize the 100 digits.
We centered the stimulus through this network,
and then we tick the activation path
embedded using a method called the [inaudible].
Then we PCA them
to three dimensions just for visualization.
Here you can see that the activation path
actually corresponds to a stimulus classes very nicely.
In fact that while we visualize it,
you can see this dark brown patch
on the top that corresponds to class 1.
Then this for example,
purple patch corresponds to class 2, I believe.
However, what we mostly run into this actually this is
second scenario where we just
randomly stimulate the 20 percent of the notes.
There you will see both blue and the orange edges,
blue are excitatory orange are inhibitory.
As you can see, as
assimilation just goes through a couple iterations,
soon it gets to the state that
the network is saturated as very chaotic.
Though we don't actually know how to analyze this.
It's almost a keen to giving someone seizure.
As we explored one or more of
these neural dynamic inspired model,
what we realize is that,
the search parameter space is really
large and it's not linear,
which means that we can't have just hand
to one parameter and expect everything to work.
We really need somehow a way to guide us
to find how to two and even initialize these things.
That's where Ellison and
his wonderful organoids came into place.
I don't have to tell anyone in this panel
about how wonderful these organoids are.
Not only because after a certain level of development,
they start to manifested
these population a white synchronous behavior.
But this is really the first time
we have a brain miniature,
brain model that we can observe how it grows,
we can study, we can actually
start measuring its internal activities.
To this extent, actually,
Francesca in Allyson's lab start
a pioneer in this technique about constructing
an effective network based on activities measured at
a different sensor locations in
multi-electric array and using
this correlation and the delays to build a graph.
What we end up doing is
constructing a time series of
a graph based on these activities.
You see here on the right side are
some examples of this graph.
They actually represents a snapshot of
a growth in these organoid development period.
What is really awesome about these graphs sets,
it's actually a domain that as
computer scientists understand and they can study
and also what's also fascinating about these graphs is
it might not necessarily
represent the physical connection,
but in the effectiveness,
it's a way for us to start understanding
representations or embeddings to how they manifest.
Just a word about embeddings or representations.
Here, I actually showed
six different representations of
the same underlying graph structure.
In this case, is being use are search graph,
was specifically in electronic devices and we
actually even color-coded to
the type of the device that the beam user searching.
The data is the same,
and as you can see,
that the representation is completely different.
With this different type of representation,
they actually have a different purpose.
In a case of a being user search graph,
we use combining multiple of these representation,
actually using a method
called linear integer programming,
to help us better understand what are the user
searched for and respond with search queries.
But they're not just useful
actually in being a web search.
In the bottom example,
we actually have a connect them sets constructed
by Albert Claudes in
Cambridge prefers a drosophila larvae.
This is a very detailed to connect them,
that it's not just mapping,
alter all the cells or neurons to connect it,
but it goes to sub-cellular to map cell
to parts of the neurons that's connecting.
So you can see connections of axon
to dendrite or dendrite to dendrite, likewise.
Was that sub-neuronal level connections were able to
construct four different representation
or four different graphs.
Though, while we started trying to
figure out how or which representation is better,
that's when we couldn't tied you.
What we need to d, in the case of,
we're trying to find
all the mushroom body input motor neuron,
we actually found that by
combining all these representations,
that's what gave us the best results.
I talked about where
machine learning felt short and how we
take inspiration from biology and organoids,
for example, are really helpful for us
to understand computer and
the development changes and it's
effects on abilities to learn.
Then now, I'm going to circle back to this thing that
everything started with
machine learning and artificial intelligence.
I don't think I'm qualified to
define intelligence, artificial otherwise.
But what my colleague,
like Suji's and among others and I have
come up with is a list of desirable traits,
what we would like our AI to be able to do.
So Suji's already talked about for most.
We'd like to be efficient.
Biological organism has to function in very limited,
resource limited situation, so should our AI.
Then it also has to have the ability to generalize.
If I learn to identify cats inside my house,
I better be able to identify cats outside of my house.
Otherwise that skill is very limited.
If I understand about the sameness difference
by being able to visually tell two subjects apart,
I should be able to understand
the same concept of
sameless difference when it comes to auditory,
tactile, or olfactory.
Then some higher cognitive abilities reasoning.
Something that I never encountered I
should be able to say that I don't know,
which is also very basic than
relationships of the things that's larger,
or smaller, or even causal.
If I drop a pen, it will fall.
Last but not least,
if we want our AI to help
improve our life and guide us make decisions,
we better understand why is making those types of
suggestions and that's why it has to be explainable.
How do we achieve all that?
Hope by now I have convinced everyone that it's
not about building the most expensive cluster,
gathering the largest datasets,
and train the biggest model,
what I believe is that we have to
do this side-by-side with biology by
actually design a set of experiments that can be run
all the way from organoids to fly to zebra,
fish, and human.
This set of tests they have to be
invariant to language and also knowledge bias.
The purpose for this set of tests is to quantify
how biological capabilities run
the five access that I talked about.
But more importantly, is actually
forced to observe this learning process,
but also programmatically using
a measurement to quantify
learning and progress as Suji's already alluded to,
for example, using information bottleneck.
Only when we can understand how
learning occurs and quantify these things,
we can really start taking
inspiration from natural intelligence,
from biological occurrences to improve over AI.
This is what we proposed as
a collective alternate approach
to solving the intelligence problems.
[MUSIC]