From ChatGPT to Dall-E,
It might seem like the pace of progress in artificial intelligence is unstoppable.
But the artificial neural networks
that underpin these programs are coming up against some major limitations.
For one, they have a hard time reasoning.
Human brains are able to reason by analogy.
When we see something new, we don't have to grow new neurons.
We can generalize new concepts from existing knowledge,
But artificial neural networks struggle to reason by analogy.
In this way,
When confronting new information, they often require more artificial nodes
to scale up their statistical abilities, which enables them to learn new concepts.
This approach is called statistical AI.
Statistically, AI is at the heart of deep neural networks,
but there is another competing approach known as symbolic AI,
which uses logic-based programing
and makes use of symbols to represent concepts and rules.
So by their very nature they are incompatible.
And the big challenge
is that how we can combine them together to get the best of both worlds.
An emerging approach called hyperdimensional computing
aims to do just that.
It taps into the power of statistical AI, while also trying to emulate
symbolic computing.
It does this by using vectors, ordered lists of numbers
that can also represent arrows in space. Depending on where they're pointing,
these vectors can represent information in a highly complex, multi-dimensional way.
They can embody nuances and traits or concepts just by small changes
to their orientation.
These vectors can combine to represent
new concepts, then be pulled apart again to discern how they were formed.
And crucially,
these hyperdimensional vectors can encode information
without having to add more nodes to the network.
The moment that we bring hyperdimensional computing or better
symbolic architectures and marry it with the statistical methods
such as deep neural network, we can have a certain advantages.
In March 2023, computer scientists at IBM Research in Zurich
made a dramatic breakthrough by combining statistical and symbolic methods
to solve the Ravens progressive matrix, a puzzle that asks an
AI to predict an image to fill the final square of a three by three grid.
This is a classical problem. The first aspects is the perception part,
where we look at different visual objects and then we have to recognize
certain aspects of the objects.
For example, what is the shape of the object?
What is the color, what is the size? All this kind of the property.
And the second aspect is to perform some abstract reasoning on top of these
kind of attributes of the objects to be able to solve a puzzle.
So for the first time, we combine hyperdimensional computing
or better symbolic architectures machinery, with deep neural networks
to be able to solve abstract reasoning problems at a scale.
Essentially, we can build inside the neural network
a machinery that is able to emulate any symbolic computing task
that is hard for the neural network itself to do.
The researchers also broke new ground by solving the problem in record speed.
We were able to significantly accelerate
the inference times of some of abstract reasoning tasks
by 250 times, having a method that is able to search
in this kind of the space rapidly, it's quite crucial to be able
to make them quite practical for for larger scale and real world problems.
Although more work is needed, researchers like Rahimi
hope that hyperdimensional computing could be faster, more transparent,
and more energy efficient than current AI platforms.
We can substantially reduce energy consumption.
We are talking about, you know, ten times, 100 times lower energy consumptions
which can directly transfer to lowering the carbon footprint
once we are performing the inference of this kind of the models.
Solving driving progressive mattresses was just a prototype.
It was, so far amazing, but the fun is just beginning,
that's the beauty.
In the 1990s, mathematician Peter
Shor developed an algorithm that threatened to break the Internet.
He was trying to understand the power of quantum computers
and realized
that they would be able to rapidly break large numbers into their prime factors.
That would undermine one of the central methods of modern
cryptography that has safeguarded online privacy for decades.
So when I discovered the algorithm, I just called it the factoring algorithm,
the quantum factoring algorithm, and I published it.
And soon people started calling
it Shor's algorithm.
At the heart of Shor's algorithm is the act of repeatedly multiplying
one number with itself until the output of a certain function
starts repeating. The required number of multiplications called the functions
period can be used to find the prime factors of any large number.
For the last 30 years, this has been the fastest known algorithm
for factoring integers.
Then in August 2023, mathematician Oded Regev published
a groundbreaking new paper.
Shor's algorithm is is great in the sense that it allows
quantum computers to factor integers and do it quickly.
I was asking myself if it can be made even faster.
Regev suspected that he could speed up Shor's algorithm
by transforming the periodic function from one dimension to multiple dimensions.
That meant including more numbers, multiplying each with itself
many times over to find the period of repetition in the outputs.
In doing so, Regev established an improved method
to factor integers faster and more efficiently.
Shor's algorithm is about finding
how long you have to walk on the line to get back to where you started.
In the new algorithm, we no longer walk on the line.
We walk on the plane or even three dimensional space or more.
the idea is that you can walk
both in, right, left, but also top down of the plane.
So it could be, for instance, that if you walk four steps to the right
and five step up, you'll be back to where you started from.
Oded's improvement
to my algorithm came out of the blue.
There wasn't any, you know, work leading up to it.
It just had a really good idea.
And he sat down, figured it out
and published a paper.
Like Shor's algorithm.
Regev's new algorithm remains theoretical in nature.
Speaking to a potential future
when quantum computing becomes more practical.
I would be happy if if turns out that this new algorithm is more practical,
it has some of the components that could make it more practical.
But it's not clear yet if it's really there.
My algorithm is what, 30 years old now?
And Oded found an improvement on it that nobody had thought of before.
I wouldn't be surprised
if people came up with more innovative quantum algorithms. And its quite possible
that someone will improve factoring algorithm even further.
We already knew how to factor integers using Shor's algorithm.
So this new algorithm is kind of just a speedup.
What I would really like to know
is if quantum computers can do things that we just didn't know how to do.
Which movie do these emojis represent?
When researchers asked three
different large language models, they got very different answers.
The simplest
AI struggled to come up with a coherent answer.
A somewhat more complex model, spit out something reasonable but still wrong.
But the most complex system nailed it in just one try.
This impressive result can be explained by a range of exciting new abilities.
AI researchers are calling emergent behaviors.
Something that is not present in a small language model
that becomes present in a larger model can be said to have emerged.
We don't think that it will be there and then we scale it up
and all of a sudden this phenomenon is already there.
So it's not necessarily a predictable phenomenon.
Emergence happens
when a system behaves in ways its individual units don't on their own.
This phenomenon can be seen across nature,
such as when lifeless atoms give rise to living cells, water molecules
create waves and starlings swarm and swoop as one.
Researchers are seeing that if enough digital nodes combine in large
language models, surprising new behaviors can emerge.
These behaviors are a distinct departure from older neural networks
which process language word by painstaking word.
In 2017, a key piece of code known as a transformer was introduced
to LLMs, enabling neural networks to analyze words in parallel.
Now, LLMs can process strings of text in an instant.
The models we were training before seemed to struggle with a lot of, kind of
more abstract tasks, more generalization. And what we've seen since transformers
is that if we train much larger models, they're better at those tasks.
The magic of transformers is that they're extremely good
at solving sequence prediction problems.
So given a bunch of items in a sequence, predict the next thing.
And this is particularly good for building generative models
of, for example, language and also images.
This year, researchers have reported a range
of emergent behaviors that enable LLMs to solve problems
they've never seen before.
This is known as zero shot or few shot learning and it's long
been a goal in artificial intelligence.
But as exciting as the results are, researchers still have questions
about what exactly is underpinning these emergent abilities.
People are trying to figure out at what point is there a tipping point.
One theory suggests that emergence happens when large language models
break down complex tasks into smaller steps.
This process is called chain of thought reasoning.
But the true source of the tipping point into emergence remains a mystery.
The models get better as they get bigger and people are releasing
bigger models at a pretty rapid clip.
Meanwhile, the science of understanding what's happening under the hood progresses
more slowly than that.
So one of the challenges with emergence is that it's unpredictable.
Because on the one hand, you know, bigger models typically mean better models.
But on the other hand, bigger models also can demonstrate
novel behaviors that the developers didn't anticipate a priori.
These can be beneficial behaviors, but they can also be harmful behaviors.
So the ways in which they will be used or are used are still unfolding.
And therefore it's equally important to measure
the possible types of harms that might be espoused in these models
and how they emerge or do not emerge with scale.
And so things feel quite unpredictable right now.
But I don't think they will always be unpredictable.