everything's great I came back and you can see that she always breaks Sunday morning or
Monday morning you know late late at night what we don't know is we don't know the causal event
that would knock the system into that particular State something happened with cardano last night
seems like there's a lot of confusion even for cardano holders
the cardano community interacted with this tweet in droves I appreciate cardano its Community the
whole crypto space can learn from what happened with cardano last night so let's talk about it
did cardana go down is my investment safe to fully understand why 60 of all Ada nodes
went offline the other day this is a cardano developers take some of you will be aware of
an incident last night at approximately this time between these blocks which caused approximately 50
to 60 percent of cardano nodes to disconnect and restart this appears to have been triggered by a
transient anomaly causing one of two reactions in the Ada node some disconnected from appear others
through an exception and restarted I still don't understand what exactly happened listen to this
60-second clip Charles hoskinson on what exactly went down over the weekend today is January 23rd
2023 just a little bit of a postmortem on the weekend we had an interesting issue let me go
ahead and share my screen if you look over here a lot of people recognize that the network stalled
for about two minutes so everything's great I came back and you can see that [ __ ] always breaks
Sunday morning or Monday morning you know late late at night while everybody's sleeping that's
just how things work uh and anyway long and short is that it seems to be a transient issue and it's
probably a combination of a collection of things that happened at the same time which means that
the reproducibility is unlikely so we pulled the team together Sunday night it was a Sam Jared
um Arno and myself and we talked to the spos Marcus guffler and uh and a few others were
willing to come on and we took a look at all the logs we have and because we don't
have a rolling P CAP we couldn't actually see what the transaction or block that actually
caused this trip of the of the balance R code that was put in for optimization was
um we know where the air was called in the program but it's hard to actually as digging through to
actually figure out what was the triggering event for that uh that particular issue
um but it doesn't seem like it's reproducible so cardano was stalled and if I'm understanding the
problem correctly such transient issues even if they were to affect all cardano nodes were
considered in the design of the cardano node and consensus the system behaved exactly as expected
block production was only briefly impacted with a portion of the network falling out of sync before
the nodes restarted therefore impact was low akin to the delays that occurred during normal
operations and often seen as Epoch boundaries most nodes automatically recovered depending on
the stake pool operator deployment choice so the system behaved exactly as expected interesting
let's listen to Charles hoskinson expand on this the good news is cardano did exactly
what it was supposed to do when a stall occurs uh basically the system uh recovers itself heals so
the nose go back up and you'll notice you know everything's great and it's back you know and
that's exactly what we designed the nodes to do it's not very satisfying because I ideally every
time a distributed system has a blip like this or a stall like this you'd like to know an exact
cause but the problem is that distributed systems sometimes create what are called emergent bugs so
locally it's not reproducible but a collection of things create a collective Global state
that for some reason triggers something in the system basically stops uh for some people uh so
uh as I said there's a team working on it they're looking at it and they're chipping away at it uh
and we'll do a post-mortem inside the get issues and um obviously if there's a bug in either a
Haskell Library uh or a bug in the implementation of the balance R which was put in for optimizing
uh cardano uh then you know it'll get patched but the problem is bugs do require understanding of
what triggered them and that's the thing we're looking for right now and it's going to take a
little bit of time for us to get there some cases you can't actually figure these ones out and they
come up once every five years once every six years and and you just say okay you know the network
recovered we move on uh but uh sometimes you can uh but you never want to drive yourself nuts over
it listen in any distributed system stuff crashes all the time and that's okay listen why in any
distributed system whether it be the Googleplex or Microsoft Azure or Amazon web services so these
massive large-scale systems uh stuff crashes all the time systems go down all the time for
a variety of reasons it could be Hardware failures or software failures hacking attempts cosmic rays
all kinds of stuff they're even a famous cosmic ray issue with Google bigtable and that's okay
because the second thing you do when you build a large-scale distributed system is you build it for
resilience and self-healing so when these types of things happen the system recovers which is exactly
what occurred here it was really cool to see how the spos came together and how quickly uh they
were on top of this and they're still looking in fact it's a race condition everybody seems to be
looking for it and I'm really proud of uh Smog and Andrew Westberg and all the others that just
showed up literally moments after this occurred middle of the damn night and started collecting
large amounts of information and trying to debug the the issue so it just shows you how
decentralized and resilient cardano has become you give me your thoughts about this whole ordeal in
the comments section below and the final question we need to answer is my investment safe it's
important to remember this transactions were not lost blocks were not dropped no money was lost the
network actually didn't halt you know it stalled for a little bit and recovered but the network
was still moving forward it's still progressing so it's not like Panic calls had to go to exchanges
and they say up grade to this emergency node or all the consensus people had to get together and
Discord and figure out how to kick and restart cardano it's self-healed and that's the point of
a decentralized distributed resilient system is it should have this capability to self-heal and
rebuild itself in the event that it stalls from a transient event and you know I'm proud that
this happened get your tickets to bitcoin 2023 Miami May 18th through 20th Miami Beach this
year use code altcoin daily 10 off this is the biggest Bitcoin Conference of the Year ticket
prices continue to increase as we get closer to the event use code altcoin daily 10 off get
your tickets now if you're interested in making money in cryptocurrency Make sure you subscribe
to altcoin daily one video every day keeping you informed on the entire cryptocurrency market I'll
leave you with this cardano holders will this issue that affected 60 percent of cardano nodes
will this issue ever be a problem for cardano ever again this particular issue will never
come up again likely in this format especially if it's an emergent transient air and because
things are a moving Target when optimizations come in it may actually recover that part of
code when Genesis comes in there's a different operating model and network upgrades come in
there's a different operating model obviously input endorsers are going to completely change
everything so that's the other part of debugging is that as you'd like to find something you may
actually just change the code that had the problem and begin with while you upgrade the module to a
different state and not even know that you fixed the particular Puck that the issue is caused with