Hey guys, it’s Greg with Apple Explained.
And in this video we’re going to explore the story behind the iPhone X’s gesture
interface.
Now this may seem like a fairly straightforward topic, but creating a gesture-based interface
that feels natural is actually very complex and requires an understanding of how humans
perceive fluidity when interacting with a touch-based operating system.
Now this topic was the second place winner of last week’s voting poll, and if you didn’t
get to vote, make sure you’re subscribed.
That way the voting polls will show up right in your mobile activity feed and you can let
me know which video you’d like to see next.
Now before we begin I want you guys to know that this video is essentially a summary of
everything I learned from this session at the 2018 WWDC.
And I highly recommend watching it if you have the time, so I’ll leave a link to that
video in the description.
So when Apple decided to remove the home button starting with the iPhone X, it posed a problem.
How would users navigate iOS?
And how could Apple make that navigation experience even better than what the classic home button
offered?
Because the home button has been a signature feature of the iPhone since its introduction
in 2007.
And the home button eventually adopted much more functionality than simply returning you
to the home screen from wherever you’re at.
It eventually was capable of unlocking your phone with touch ID, helping to take screen
shots, triggering multi-tasking with a double click, summoning Siri with a long press, and
even activating accessibility features like reachability with a double tap.
So with all of this functionality in a single button, removing it raises a lot of questions.
Namely, how will Apple incorporate all of these capabilities into a strictly gesture
based operating system?
So let’s explore how exactly they did it.
Apple actually delegated the responsibilities of the home button to not only OS gestures
but also to other physical buttons the iPhone already had.
Touch ID was replaced by face ID, the side button helped take screen shots, multi-tasking
was triggered by a specific gesture, Siri was now triggered through a long press on
the side button, and reachability could be activated with its own gesture.
Now as you can see, these were a lot of changes Apple had to make simply from removing the
home button.
And that meant users had to fundamentally relearn a number of features that they had
been using for years.
But Apple did their best to take advantage of user’s existing muscle memory by mirroring
feature activation on the iPhone X as closely to previous iPhones as possible because Siri
was still activated by a long button press, screen shots were still taken by pressing
two buttons simultaneously, and your iPhone could be unlocked securely with just a glance
requiring virtually zero user interaction.
But that wasn’t even the hardest part in developing the iPhone’s gesture based OS.
The biggest challenge Apple faced was ensuring the OS felt natural, like an extension of
your hand.
Because your finger would be interacting with every element of the OS, those gestures had
to be as or even more responsive than a physical home button.
Which is difficult since other gesture based operating systems are known for feeling sluggish,
clunky, and frustrating.
So what did Apple have to consider to ensure the iPhone X’s interface was as fluid as
possible?
Well, there’s three major elements that needed to be optimized: the operating system’s
lag time, lightweight gesture detection, and the ability to overlap gestures.
And if these three elements can be perfected, the user’s experience in interacting with
their phone becomes incredibly quick, effortless, and even enjoyable.
So let’s start with the ability to overlap gestures, which is probably one of the most
important features when trying to make the OS feel fast.
Let’s say you’re in the middle of opening an app when you decide that you actually want
to access multitasking instead, so you swipe up and are able to summon multitasking while
the app is in the middle of opening.
So with iOS you can actually interrupt an action with another while still maintaining
the same fluidity as a complete, uninterrupted action.
Now this may appear to be an insignificant capability, but it’s critical in ensuring
the operating system keeps up with the speed of your thoughts.
Allowing you to change your mind in the middle of a gesture.
Because imagine how annoying it’d be to have to follow through with every gesture
you begin, and executing another gesture to undo that action and then navigating to where
you actually wanted to go in the first place.
So the technology that allows this feature is actually pretty impressive.
Because while you may assume a touch screen interface only needs to recognize vertical
and horizontal movements, a fluid gestural interface requires multi-axis sensing, therefore
enabling overlapping gestures.
That way your device responds to your stream of thoughts and actions, rather than you having
to adjust your workflow to your devices’ limited capabilities.
This also means you can discover new gestures while executing existing gestures.
Because as you’re swiping up to go home, you might discover the multitasking feature
along the way.
And this is all critical to ensuring the OS feels like it’s responding to you instantly.
Now let’s talk about latency or lag time.
Every iPhone has been known for being responsive, but it’s much more difficult for an operating
system to respond to an ambiguous gesture than a straightforward button press.
So how do you accomplish this type of interface?
Some people might assume Apple uses a timer.
For example, waiting to trigger multi-tasking until your finger travels below a certain
speed for a certain amount of time.
But it turns out, that creates a significant lag which is frustrating for users.
So how does iOS anticipate your desire to trigger multi-tasking before your finger even
stops moving?
Well, it turns out tracking the acceleration of your finger as it’s moving up, allows
iOS to anticipate the pause that will trigger multi-tasking, allowing the multi-tasking
pages to appear almost as soon as you think about it, instead of waiting for them to appear
after your finger has already stopped moving.
Because by that point, your mind has already moved on to the next task it wants to accomplish.
Another important characteristic of a fluid operating system is detecting lightweight
interactions.
This is what enables effortless swipes, scrolls and responsive taps.
But what is a light weight interaction?
Essentially, it means the OS can respond dramatically to the slightest and smallest of gestures.
Meaning you can accomplish more with less finger motion.
And it all begins with iOS understanding as much about your short gesture as possible
to calculate the energy and momentum needed to deliver an appropriate software response.
And these calculations include your fingers’ position, velocity, speed, and force, which
generates a fluid amplification of your movement.
And this allows your gestures to feel like an extension of your hand.
For example when you scroll through a page, your finger is only on the screen for a very
brief amount of time but the OS can calculate the correct momentum and speed of your gesture
and allow the page to scroll in a way that feels natural.
And if you remember back to the original iPhone, Steve Jobs demonstrated this feature which
is called inertial scrolling.
And it was a really big deal at the time because no other phone company had incorporated this
feature into their OS.
Instead, the phone would only scroll when your finger was moving across the screen.
Once your finger was no longer in contact with the display, it would stop scrolling
abruptly.
Which meant users had to make long strenuous gestures to scroll down something like a webpage.
And when it comes to the iPhone X, the home screen gesture operated in a very similar
way.
But one aspect of this gesture that some people may not consider is that you’re still using
the same muscle memory from pressing a physical home button, because your thumb still travels
to the bottom of the display to begin the home screen gesture, which is where the old
home button used to be.
And this an important consideration since it makes transitioning to the iPhone X’s
gestural interface more natural.
Now when it comes to making the OS feel smooth and fluid, you may assume it all has to do
with the frame rate, but that isn’t necessarily the case.
Because if a movement is not optimized to appear fluid, it will likely look choppy even
at a high frame rate.
For example, even at 30 frames per second, a movement can appear very smooth due to minimal
differences between each frame and other tricks like motion blur and motion stretching.
And you can see this in action when launching an app on the iPhone X.
Because as it stretches to take over the display, Apple actually added slight blurring between
each frame to give your eye extra information to perceive the motion as fluid.
But Apple didn’t only consider how smooth movements appeared but also how heavy they
felt.
For example, when you swipe between photos it takes an effortless flick because Apple
wanted the photos to feel light.
But when swiping between apps it takes a more deliberate drag since apps are conceptually
heavier than photos.
And although you may not have noticed this difference consciously, it’s inclusion enables
a more natural interaction with your phone.
Because maybe if iOS didn’t include these small details, your overall experience with
the OS would feel less natural.
Now today, this gesture-based operating system that began with the iPhone X is now used by
the XS, XS Max, and XR.
So it’s clear that this approach is the future of the iPhone and likely every other
smartphone on the market.
And I think this is definitely the right direction.
Because once you become acclimated to the changes made to iOS, navigating your iPhone
with gestures is at least twice as fast as with a physical home button.
And this is only the beginning.
I’m very excited to see what other ideas Apple has in store for this type of interface,
especially when it comes to the iPad.
So that was the story behind iPhone X’s gesture interface, and if you want to vote
for the next video topic, don’t forget to subscribe.
Thanks for watching, and I’ll see you next time.