Sense.
React.
Confirm.
That’s video gaming in three words.
We see action on the screen and hear the sound,
react with a button press, and audio and video playback confirm what happened.
These words are not just a description; they are a timeline of events.
Players are responsible for the time between Sense and React.
Technology is responsible for the time between React and Confirm.
And if too much time passes between React and Confirm, then we have a problem.
We call it: lag.
If there is a small delay between these two,
we compensate for it – perhaps subconsciously. As the time period becomes longer, the problem moves
from our subconscious mind to our conscious mind, and compensation turns to frustration.
Wireless controllers, displays with slow image processing, and online multiplayer
contribute to lag in the modern era, but lag begins with the game code itself.
When it comes to using controller input to show what happens and play a sound, we are at the
mercy of the code’s priority for processing that user input and starting the response.
It lands somewhere between “yes, master” and “oh, it’s you. Wait here.”
Games of the past had the advantage of using a CRT television – a device
with almost zero lag – and that means once the controller input was read,
the results might be drawn to the screen after a period of a single frame.
Of course this also meant that developers could push the period of
time between receiving controller input and actually doing something with it.
Press button. Attack enemy. That’s the short version.
If we add more detail, two additional steps appear.
Process input. And… Begin Animation. Each of these two steps is a potential source of lag.
Players can feel the time elapsed between sending controller input and the start of processing
it and can see when the “wind-up” period of animation for an action takes too long to occur.
For Teenage Mutant Ninja Turtles on the NES,
BOTH of these two items provide a one-two punch of input lag.
Wrap a retro game in an emulator running on modern hardware, and it is quite likely that you extend
lag duration. While optimizing software emulation and the attached hardware can help minimize lag,
one of the best ways to reduce it is to modify the original game code itself.
Now this could be considered controversial as it fails to preserve the feel of the original game,
but then you know what? So does adding lag further down the chain.
So let’s talk a bit about game design, analyze the input lag and animation for
the attacks of our four turtles in TMNT, and find some ways to modify the code.
Let’s take this list and be a bit more specific.
Player presses the B button for an attack.
Address 4016 is polled to get 1st controller
input on the NES and provides data for that button press.
The results are saved into RAM for processing.
Sometime later, a function call sees that input value
in RAM that indicates an attack and therefore starts that attack.
Animation and collision detection begin.
Now - TMNT has some amount of lag for the time period between reading the controller
and the attack function call picking up the input. One reason is the game runs at 30 fps.
A second reason is because it uses the delta modulation channel (DMC) for drum playback.
The game also has some lag – either by design or not by design – for the animation of the
Turtles attack. Let’s tackle animation frames first since they are something we can see.
I am going to pick up at this moment in time – when the attack function
first sees that the player pressed the B button. We’ll cover the rest later.
For the sake of simplicity, let’s use a right-facing forward attack
with the primary weapon for each turtle. When the logic confirms the B button has
been pressed, it spawns the turtle’s weapon – Leo’s sword in this case.
One thing that is interesting about this is the sword is not assigned a position in this function.
It takes the next two CPU frames to complete initialization of the weapon,
and during the second of those two frames,
the game acknowledges a sword located at 0,0. You can see the hitbox in the top left corner.
There appears to be a priority issue between the functions that spawn the weapon’s numbers,
set the animation ID, and locate the weapon. Now you may consider this to be trivial,
but the game does roll collision detection
for all enemies on the screen vs. an invisible sword during this one frame.
I’d call it a bug – albeit inconsequential save for some lost CPU time.
Lookup tables help us know which animation frames, timers, offsets,
and more are required to make the attack happen.
By design in the code, the weapon’s location is first defaulted to the same location as the Turtle
when setting up the next frame of animation. After that, the sprite is relocated using offsets.
This is necessary because otherwise the sword swing would be left behind
while Leo keeps walking forward. It needs to stay with him as he moves.
Curiously, the first two frames of a located sword are in that shared,
default location without the use of an offset. Is this a bug or is it just wind-up time?
Even if this is by design so that it takes longer for Leo to start an attack,
the sword still exists in this location. That means that you could wait for an enemy to reach
point blank range and then defeat it as the sword sprite first appears behind Leo’s head.
As for the rest of the swing, we have 4, 4, and 4 video frames followed by 4 more in the same space
as the turtle. These last 4 frames are probably for a cooldown before the next swing is possible.
So this attack has a 2 frame initialization, 2 frame wind-up,
12 frame attack, and a 4 frame cooldown. That’s 20 frames not including input processing – which
is another 2 frames, basically. Again - We’ll… get to those details later.
In summary, Leo has a 22 frame attack from input acknowledgement to cooldown completion.
The NES runs at vertical refresh of 60.0988 Hz – just over 60 frames per
second. A single video output frame is therefore about 16.6ms. This multiplied
by 22 frames gives us an attack window of 366ms.
OK. This timing is an interesting piece of trivia, but what can it tell us?
For these enemies that are low to the ground, your sword does not reach their location until
after 4+2+4+4 = 14 frames! It is going to take over 232ms – close to a quarter
of a second - from the time you press the B button to Leo’s sword reaching the target.
Assuming you didn’t use that point blank strategy, of course.
Let’s move to Raphael.
Each position of his weapon is shown for two output frames. Unlike Leo,
there is no wind-up period. His Sai appears directly out in front immediately after its
position has been initialized. The weapon is rotated and then cools down for two frames.
If you add up input processing plus attack animation,
you get 22 frames – just like Leo. The primary difference is that Raph’s
weapon is “in play” for a longer percentage of time versus Leo’s.
Here are the two framesets side by side.
With two examples illustrated here, it is perhaps worth noting that the
duration of each attack frame is loaded from the same place in the ROM. For Leo’s frames,
that value is a two. For Raph’s frames, it’s a one.
Since the game runs at 30 frames per second and each game frame is output twice,
these numbers are effectively doubled for video output.
So without much thinking, we could cut Leo’s attack duration in half using
one game genie code to change the sword’s frame duration from 2 to 1.
Just like that, the sword swing speed doubles.
One minor problem… we didn’t change the attack speed of the turtle sprite,
so he lags behind. A second game genie code takes care of that.
The sword attack certainly feels faster. This was a really cheap hack to speed up Leonardo’s attack.
If we take a look at it frame by frame from the start, you’ll notice that the two frame
warm-up with the shared Turtle position has disappeared from the sequence. We
snap from the 0,0 weapon hitbox to the sword being right behind Leo’s head.
So not only did we speed up the swing, we also tightened up response
time. Animation occurs closer to the button press by two video frames now.
Is there anything else we can do?
The animation lookup table for the swing uses IDs 2, 3, and 4 for each swing position,
then a zero for the cooldown, and then an FF to signal the end of the animation.
If we change the cooldown value of zero to FF to end the sequence immediately after
the swing and cut out the cooldown frames, now the swing can end and begin again a lot faster.
Jerky animation aside, we reduced Leo’s attack duration from 22 frames to 10 frames.
It throws off the balance of attack speed between the four turtles, of course, but there it is.
Speaking of which, let’s check the timings of the two remaining turtles.
Michelangelo has the fastest attack at 18 frames. He uses a frame timer with a
value of 1 – the same as Raph – so we see each sprite for 2 frames of video
output. Four frames of input lag plus twelve frames of attack plus a two frame cooldown.
For both Raph and Mikey, we can’t reduce their animation timers as they are already set to 1.
Our only options for an even faster attack from an animation standpoint are to remove the cooldown
like we did for Leo and/or redesign the animation for the attack by changing the frame sequence.
Not that we need to make their attacks faster – we’re just discussing redesign options.
For the sake of messing with game design, we could also slow down the attacks by doubling the frames.
And of course in addition to the attack being noticeably slower,
there are now two video output starter frames with no positional offset – just like Leo.
This just leaves Donatello – the turtle with the slowest attack.
4 frame input lag + 8 frame wind-up + 20 frame
attack (10 back and then 10 forward) + 10 frame cooldown.
42 frames in total.
You’ll notice that Don’s wind-up appears to be two frames less than the attack frames.
So it would appear that the frame timer value for wind-up is the same
as the attack. The difference is that two frames are lost to attack initialization.
Since Raph and Mike’s attack counter is two frames of video,
that is the perfect amount of time to cover the setup time before starting the attack.
Leo and Don have to wait on the remainder of their attack
timer to finish this “pre-animation wind-up” before their attacks appear.
As for Don, we can, of course, drop his attack time to a smaller number
and/or remove his cooldown to speed up attack.
Were the speeds designed well? Were speeds slowed too much for Leonardo and Donatello?
Should the animation frames have been changed to make the movement more obvious?
For instance – would Leo have benefited from
a sprite with an animated sword arc baked into it like in Ninja Gaiden 3?
Maybe, but it would also increase the sprite count per line during an attack.
With a good grasp on attack animation, let’s fill in the gap that is the time
between receiving controller input and starting the attack.
While the “how” of code is important for the actions
performed, the “when” can also be very relevant.
For this reason, we’ll use the Event Viewer of the emulator.
Since the CPU does work as the PPU outputs to the screen, we can create a timeline of CPU execution
on top of the graphics for a given 1/60th of a second. What is the CPU doing to prepare the
next frame of video at the exact moment the PPU is drawing the video image behind it to the screen?
Konami elected to split work across the time of two video output frames. Instead of performing all
code inside 1/60th of a second, some logic is done in the first 1/60th of a second,
and the remaining logic in the second 1/60th. Meanwhile, the PPU just outputs the same
graphics for two consecutive frames of video. Therefore, the game runs at 30 frames per second.
And for our controller read, this comes into play immediately.
The controller port read, the save to RAM, and the NMI occur every 1/60th of a second – every
video output frame or “CPU frame” if you prefer. So even though the game runs at 30 frames per
second, we still clock-in the controller data every 60th of a second – every “CPU frame.”
But now let’s also mark the code that determines if Leo should attack.
OK. It occurs about a third of the way into this CPU frame and is followed by our trio.
Let’s move to the next CPU frame.
Oh. We jumped right down to the bottom here with the 4016
controller read. No attack check was executed this CPU frame. Why?
Well… TMNT’s logic is divided across two CPU frames,
and no attack check is performed this frame.
So what this means is that even when your controller read is picked up and
saved in this frame, the attack won’t be processed until the game passes all the
way through the next frame, and then the attack check is finally picked up here.
Now… there is another potential problem with the code that gets controller input.
Turtles has pretty good music, and it uses the delta modulation channel or DMC for drums.
Unfortunately, there is a possibility that reading a sample byte for the DMC
at the same time the controller is read will corrupt the controller data on the NTSC NES.
To get around this, developers would, for example, perform two consecutive controller
reads and compare the results for a match to verify controller data when the DMC was in play.
Super Mario Bros. 3 does this, and we even saw the double controller
read occur on an oscilloscope in our NES controller electronics episode.
So what do you do if the two reads don’t match? Mario 3 loops immediately to do
another controller read until two consecutive results match.
What about TMNT?
Should the DMC corrupt the controller read, controller code in Turtles will detect the
difference in two consecutive reads, invalidate the read and then… not loop
to try again. You have to wait until the NEXT FRAME before it tries again.
One real world possibility might be - a button is pressed on a frame prior to our attack read
frame but data corruption occurs, we miss the attack logic window, have to get clean
controller data sometime after that, pass through the next frame that does not handle attacks,
and then finally get the controller processed during the attack check on the fourth frame.
The only thing worse would be if you had bad luck
with consecutive frames of controller data corruption.
The emulator I am using now does emulate the DMC conflict, so how often does it happen in Turtles?
To find out, I modified the code executed after a failed read so it replaces the secondary weapon
slot count with failed controller reads to keep track of how many conflicts have occurred.
If we let the game play in real time, the counter moves up at a noticeable rate. Remember this code
is executed 60 times per second, so the number isn’t too high… but isn’t minimal, either.
And when you combine the fact the attack function is only executed every 1/30th of a second,
this combination of DMC corruption plus bad luck for frame timing can really add to input lag.
Meanwhile, if we modify the music code so that it bypasses the DMC work and effectively mutes
the bass drum, the counter stops going up as we have removed the possibility of a conflict.
So you can see here that any given button press
for TMNT could require a little extra time before the action is processed.
And when you combine this possibility with the required wind-up time for
Leonardo and Donatello’s attacks in particular,
we have inconsistency in controls that makes us “feel” like they could be more responsive.
It is important to recognize how input lag can stem from code design.
It can be deliberate – such as how many frames are used to start an attack,
but it can also have a random element – when was the button pressed versus when
an attack is processed (for a game running at less than 60 fps) as well
as if the controller read was corrupted thanks to the delta modulation channel.
You could argue that by design, Turtles has... “Input Timing RNG.”
The game is playable – many of us have played it and finished it – but it would
be nice to have a tighter response time between a button press and seeing the desired action.
Although we altered the attack animation frame count for Leonardo and Donatello
as well as removed their cooldowns, a more “design friendly” approach would
be to reduce the time between the controller press and the first frame of attack animation.
For the input logic itself, any emulator for the game should be
able to eliminate the simulation of the controller/DMC conflict. This
code would therefore always work and could likely be eliminated.
And remember this happens every frame. So you could trim your controller read logic here
and perhaps save enough cycles to reduce - some - frame lag. Even though Turtles is using two frames
to get its logic done, it still slows down at times – complete with a disappearing status bar.
A second option might be to eliminate the “dead time” for an attack for Leonardo and
Donatello. If it takes only one game frame – aka two video output frames - to get the
weapon setup for animation, then you could probably code a single game frame
of wait time to initialize the weapon and THEN start the weapon’s animation.
This would eliminate the two frame wind-up for Leo and the eight frame wind-up for Donatello.
I hope you have enjoyed this investigation on the timing of a standard attack for the
four Turtles. Please like and subscribe for more videos like this one. I also have
a Patreon available for those of you interested, and thanks for watching.