hi imagine having partial measurements of a high dimensional complex dynamical
system say you have a few sensors submerged in a turbulent flow
or maybe you can track a few particles in a complex granular material flow
or maybe you have eeg measurements or fmri measurements of the brain and you'd like in
all these systems to understand the underlying dynamics the high dimensional dynamics
this sounds like an almost impossible task but this is exactly the problem we tackle in
our recent paper discovering governing equations with deep delay autoencoders
now this is work i've done with my colleagues kathleen champion nathan kutz and stephen brunton
the basic idea is you have some measurements let's say one-dimensional measurements y of t
and you can embed these measurements in a higher dimensional space
by time delay embedding and we know that this time delay embedding coordinates
with time series h of t is diffiomorphic to the original full state dynamics z of t
we know that because there's a theorem that says it token's embedding theorem
and we know that we can go from h to z back and forth using an invertible function v and c
some continuous nonlinear continuous function now we also know that given data in the full state
attractor or coordinates we can learn the dynamics z dot equals f of z now f is an unknown function
it's an analytical unknown function phi and c are also unknown functions and what we're proposing
here is to learn all these functions from data and the only given data is y of t of course this is
a highly imposed problem and here the the art is all about designing the loss function to optimize
for this and to recover the full state attractor along with the dynamical system that underlies it
so how does that work let's say you have one-dimensional measurements y of t
and here i'm showing these measurements being shifted in time like shifting a window towards
positive time and let's highlight four of these measurements as we're shifting them in time y of t
y of t plus tau y of t plus two tau y of t plus t tau so here i'm taking four but you can take more
what you can do is take these four points in one dimension and project them on a higher
dimensional space as one point in these in this four dimensional delay coordinates so now we have
this one one point and four dimensions that is an embedding of these four points in one d
now as i said earlier we know that the full state attractor from which this one dimensional
measurements came this is the higher dimensional system we know that these this coordinate system
is differentomorphic to this coordinate system because of token's embedding theorem
and we also know that given data in the full state coordinates we can discover the
underlying dynamics using sparse identification of nonlinear dynamics or also known as cindy
for example in the example illustrated here
this is the lorenz system it's a three-dimensional system with three unknown parameters so
now how do we actually develop this method it seems like there are a lot of unknowns the
theomorphism is unknown the sparse identification of solution dynamics is also a machine learning
algorithm with unknown coefficients and the delay embedding is also there is no unique way of doing
that there's um the number of dimensions that we choose is unknown and tau is unknown
so for validation we start with a known dynamical system high dimensional dynamic system
and we project it on 1d for example we can take z1 to be actually y of t and then therefore closing
the loop and developing and refining each one of these pieces so that we can at some point discover
a method that works for dynamical systems that we don't know so let's focus on each one of these
parts in this cycle so we know that time delay embedding is possible it's all a matter of
choosing the dimensions so we can in this case define the henkel matrix the henkel matrix
each column in the henkel matrix is one data point in the delay coordinates
so here we have q data points and we can write this time series in the delay coordinates as h1
h2 to hq here i'm showing a continuous attractor but in reality this will also be discrete
um why use time delay embedding this technique has been used extensively for example to show
to highlight the difference between chaos and noisy dynamics or noise and
recently my colleagues showed that chaos can be cast as an intermittently forced linear system
using time delay embedding or using the dominant modes of the henkel matrix i encourage you to
look at this paper it's very interesting um and on the other hand we have the sparse identification
of nonlinear dynamics method or cindy that takes data in full state coordinates or any coordinate
system and tries to discover the underlying dynamics f using a form of sparse regression
so if you don't know about this method i encourage you to look at this paper
but here i'm going to briefly illustrate how it works so you start with a data matrix
where you have time series z1 z2 to zq and you define a dictionary the dictionary will include
z1 z2 z3 z1 times z2 z1 cubed etc all these possibilities you can also have sine of z and
derivatives in other independent variables like if you have x and you you can get a pde in fact
um and at the end you cast this problem as the in the form of this differential equation where
c now is a matrix of unknown coefficients and what we're trying to do is optimize for these unknown
coefficients to fit the data so we're optimizing this loss function where we minimize the residual
squared of the differential equation and we also try to minimize the number of terms in
the differential equation therefore maximizing sparsity so cindy has been improved extensively
in recent years and there are of course it's an active area of research how do you discover the
best differential equation that underlies a given dynamics in any coordinate system
now the last piece of this puzzle is this differomorphism we know that it
exists but we don't know where it is so how do we go from this side to this side
we don't know exactly what it looks like but we know that's very non-linear and we know that this
coordinate system will probably be much higher than it will certainly be much higher
dimensional than this coordinate system in fact the tokens embedding theorem says
that this coordinate system should be at least um three uh two times this coordinate
the dimension of this coordinate system plus one so if this is three this has to be at least seven
and what better than a neural network to make that approximation more specifically a sort of encoder
now we know that a neural network can do the job because if we know both sides of this equation
we show in the paper that you can actually do this fitting really well and it generalizes well
and we also show that you can actually do this transform using a combination of svd decomposition
and a neural network so that you make a linear compression of that coordinate system
and then you make the rest with a non-linear transformation using an encoder
so now that we saw all of these puzzles
Let's summarize and highlight what we're looking for. So we have an unknown system here, we know the equations but in reality we don't...
and we're just making one-dimensional measurements y(t) let's say this is going to be x1 for example
so we time delay embed y into h and then we have an encoder that takes us from h to z.
Then we can map z to z-dot with this analytical function f therefore discovering a nonlinear dynamical system
and then we should be also able to go from z back to h this makes this
an encoder an auto encoder and f is of course an analytical function so that's the SINDy part
so it's a SINDy autoencoder and it uses delay coordinates so it's a delay SINDy autoencoder.
Now after doing that reconstruction you have of course to minimize all these three functions.
Now to show you our results for example concerning the Lorenz system, we discover something that
looks like the original system of course this is an ill-posed problem again and there are a lot of
diffeomorphiscs that are similar to the original Lorenz and what we're assuming here is that the
optimal solution given a certain loss function is similar at least has some invariance that
is shared with the original system and you can notice that the discovered system here
is even sparser than the original system and we can look at the system and make predictions
for at least the first dimension and gain some insight as to the structure of the attractor...
something that we cannot do with one dimension only. Now this work is based on previous work
this is very similar architecture to the cindy autoencoder developed in this paper done by my
colleagues so please look at this paper as well before you dive deeper into this work
um all right let's dive a little deeper how do we actually do that because it's
all about defining the loss function what is the loss function that allows us to go from
one-dimensional measurements to three-dimensional dynamical equation system equations
so we have y we delay embed y and h and we said we can go from h to z uh using combination of
svd and encoder or just an encoder and we can do that back to h thus having an autoencoder
now the cindy part is taking z and projecting it on a higher dimensional space
by defining this dictionary potential terms on in our in our differential equation and
then multiplying it by this unknown function c that we're also optimizing for to get z dot
now we can also go back to h to constrain our our model even further um you can see in the
original paper of delay autoencoders that it's useful to take the derivative of the input data
h dot and by automatic differentiation to get z dot the details of how you go from here to
here are explained in the paper now you can do the same thing to go from z dot to h dot
and you can notice that there are a lot of redundancies there's z dot z bar dot an h
dot in h bar dot and an h bar and h so we want all of these to be the same we don't want the
redundancies so this is what defines our loss functions so first we minimize the difference
between z dot and z bar dot and you can derive these as a function of the input
h and the functions v c and c um you can also we also want to minimize the difference between
h bar h dot and h bar dot as well as the reconstruction loss between h bar and h
now there's one last loss we can also add to constrain our model even further
which is to take the latent variable z and to integrate it using the discovered model like this
so now we have z would be this latent variable at the given epoch or at the given training step
is defined as z at t zero and then we integrate that in time using the cindy coefficient c and we
get these time steps or the solution z hat at t 1 until z hat at t n and we want the first dimension
of this solution to be also the input data which means that we would like our discovered model
to have the first dimension match the input data that we're giving it this is of course an
assumption but it constrains our model even further to get something that makes sense
So let's look at what the algorithm actually does. Here we initialize our neural network
to map to the dominant modes in the latent variable. And starting from that approximation we
try to train the algorithm to find the most optimal solution given those losses.
Of course there's something i didn't mention also that we're minimizing the sparsity of the solution
So here the full system is unknown of course we're just measuring the x dimension, the first dimension,
and the yellow curve is the latent variable and the red curve is the integrated SINDy model
so you can see that SINDy does a good job learning from the latent variable
but of course with time as the system is trying to minimize the sparsity of the solution
the attractor slowly drifts away from the original system because it has no information
as to the the the other dimensions... and here comes the important part about improving
method as you can see the model that's discovered here is different from the model that I showed you
before, and this is to highlight the fact that the solution is not going to be unique depending on
how you weigh your losses which losses you include and for example the architecture
of the neural network the dimension of the embedding there are so many hyper parameters
but given a system if you know for example that the energy is conserved you can bake in
those symmetries... physics informed symmetries... in the loss function
and therefore improve that discovery and get it closer and closer to the original system.
I think this method is a good start to start from somewhere to say that the origin system
has this kind of structure has these kind of terms in the differential equation
and as we try to improve our data-driven modeling methods we find more and more refined models that
are high dimensional from our measurements but have a structure that describes the physics
so we tried also we also tried this on the lorenz water wheel so we took this video from youtube
and from the harvard natural sciences lecture demonstrations and we took that video and we
wanted to extract one dimensional measurement which is the angle of this video ideally you would
take the video and pass it to some kind of encoder cnn encoder and immediately extract the latent
variable one latent variable which is the the angle but in this case we use a machine learning
algorithm to detect the lines and to cluster the lines and to discover the angle of those
lines and you can see this is a very noisy kind of measurement and when we extract those measurements
um with the angular velocity here we have noise so we have to kind of smooth the data but from that
smooth data we use our algorithm and we discover an attractor and here we're showing the model
solution using cindy and the latent variable of course this is kind of a butterfly shaped
uh attractor but there's a lot of improvements that have to be that can be done to get closer
to the original system and this is the discovered analytical model so um this is a i'm very excited
about this work to apply it to problems where we actually don't know the solution we don't know the
underlying dynamics and if you have that data i would be very excited to collaborate with you to
discover the underlying dynamics of your complex system and if you have ideas on improving this
method i encourage you to look at the paper and to give me ideas and to pitch ideas and
and to collaborate because this is what research is all about so thank you
you