It’s an honor to receive this award, even if it marks me as a bit of a gray beard.
My first serious work with computers began in 1974 when I started my PhD work in Cognitive
Psychology at the University of Oregon. Control of visual perception experiments got me hooked
on both real-time operating systems and the thrill of teasing out cognitive phenomena
at the millisecond level (and, more generally, on the interactions between humans and computers).
After graduating in 1978, I immediately took up a research position at IBM’s Thomas J.
Watson Research lab in Yorktown Heights NY, my home base for the next four decades (and
counting).
My first project at IBM was with the (small) team that created the world’s first Voice
Mail System. I’ll talk about that in a few minutes but will just mention now that fully
half that team were psychologists … a ratio of behavioral specialists to engineers I have
rarely seen since.
After a stint as one of the founders and senior manager of IBM’s User Interface Institute
(working with amazing colleagues like Jack Carroll and Mary Beth Rosson), I returned
to serious programming to prove a point … namely that pure object systems like Smalltalk were
quite up to the task of building high-performance systems. As testament to that, we built and
shipped the first bespoke system for K-12 Internet access, supporting up to 400 users
on a single PS-2 Mod 80 with probably less computing power than my iPhone. Importantly,
Smalltalk gave us the ability to rapidly iterate EVERYTHING from the user experience on Window
machines and Macs to the communication protocols with the server suite, also coded in Smalltalk.
It might seem odd that I focus so much on programming here. Why talk about that as opposed
to design work or experimental work? The answer, for me, is that programming has been my design
medium. I really can’t think clearly if I’m too far from the reality of making things
work in all their details. Plus, I’m a really lousy sketcher. So while I won’t recommend
this approach for everyone, I will suggest that it is important to find YOUR medium where
you can think most clearly and try out your ideas most efficiently.
Now it might also seem strange that after several years of working in Smalltalk, I would
dive into a project written in raw C for a device with only a 160 by 160 display and
a plastic stylus. I initially thought the project, home grocery shopping using a Palm
Pilot, was a dumb idea. But my now longtime colleague, Rachel Bellamy thought otherwise.
She was right. I’ll talk a bit about what became IBM’s first mobile eBusiness application
in a few minutes.
Another project I’ll talk about was co-led with my even longer-term collaborator, and
wife, Vicki Hanson. We were given the opportunity to make the World Wide Web more accessible
for older adults and those with visual, motor, and cognitive disabilities. By the time we
were done, the browser extensions we created were deployed around the world through IBM’s
program of corporate philanthropy.
In a dramatic shift of focus, I’ll talk about another project using what I’ve come
to call bi-directional time travel. In that work we compared programmer productivity in
parallel/scientific computing in the past with what it would be in the future.
And while I won’t talk about my more recent work in promoting AI transparency and governance
through what we call FactSheets, I list it here only to bring you all the way to the
present day. Overall, I’ve been really fortunate to be able to practice iterative, user-centered
design and development from the time I joined IBM to the present. User-centered development
is what I’ll turn to now by looking at a few of the barriers I’ve encountered along
the way.
So what are some of these barriers? Most will probably seem familiar to you. And while I
won’t claim that the way we overcame them will always work, they worked for me.
If you work in a corporate environment you’ve probably hit this barrier. To get to users
you need to convince the account team that you won’t mess up their relationship with
a customer. And then, you have to convince the customer that you won’t mess up their
relationships with their customers if you’re allowed to work directly with them..
But it’s really hard to do the kind of work that we do if we can’t work with end users
directly. So find a way to get to them.
Here’s a story that illustrates the point from the home shopping application I mentioned
earlier. When we were approached to build a front end on a Palm Pilot device to a terabyte
scale database of millions of loyalty card shoppers of a UK grocery chain, we said we
would need to work with a representative set of these shoppers to really understand what
would meet their needs. This request was not well received. And in a line I’ll never
forget, the account team asked us “What exactly will you learn?”. Of course, we
couldn’t tell them but we could be honest. We told them “We don’t know what we’ll
learn but it will be important” … and we drew a line in the sand saying that this
was the one condition that had to be met or we’d turn to other projects (by the way
you won’t always have the luxury of turning to other projects but working in a research
environment gave us a bit of freedom).
So what did we, where we was Wendy Kellog and myself, do? We flew to the UK and camped
out in one of the customer’s stores to talk with shoppers.
So what did we learn? We learned a lot of things, of course. Which got us started on
the path to building an app that leveraged people’s past purchases to select what to
buy. One cool feature of the Palm Pilots we were using was a built in bar code reader.
So in addition to the now familiar list of things you purchased before, shoppers could
also treat their pantry or the store itself as containing things to scan and buy.
This was all before Pea Pod and various other grocery shopping apps that are now ubiquitous.
But beyond being “a pioneer”, my favorite part of the story happened during the debrief
Wendy and I had after our initial field work. The customer executives asked “How did you
learn all this?”. And our answer was, “We talked with your customers”. Sometimes it
really is that simple.
I’m going to fast forward now to 2007 and a completely different domain, high performance,
parallel scientific computing. We were recruited into a VERY large project of over 1000 engineers
building what became the Power 7 peta scale cluster. But the government attached a very
interesting condition to the contract, namely that we had to demonstrate that programmers
working with the machine in 2012 would be 10 times more productive than they were in
2002. After agreeing to take up this challenge, I asked the obvious question, what was the
2002 productivity baseline against which we could compare the tools and system features
being created over the next 5 years. You can probably guess the answer … there was no
baseline. So, in what I called bi-directional time travel, we set out to compare what might
come to be against what had been years earlier. How did we do this?
Well, in a way, we simulated both what was and what was to be.
One of our team members, Jonathan Brezin, a crack programmer and all around good sport,
agreed to spend months programming a representative set of parallel programming tasks using Emacs
with C and MPI on the one hand, and the new X10 programming language that was part of
the to be delivered solution on the other hand. Another team, led by John Thomas working
with Bonnie John, started modeling future tasks at the level of keystrokes using Bonnie’s
CogTool. While there’s more to the story than can be told here, suffice it say that
using these techniques, and others, we successfully demonstrated between an 11x and a 13x productivity
gain from the combination of a new programming model, a new programming language, and a new
and much faster system. So if you can’t find users, or if you can’t create the environments
within which they will work, it is often possible to simulate what you need.
Now that we’re accustomed to bi-directional time travel, let’s take the simpler jump
back to the past when Windows 3 was on nearly every PC. As I mentioned earlier, we were
given support to make the Internet more accessible, starting with older adults but eventually
branching out to people with visual, motor, and cognitive disabilities. What I will show
here is how we approached the problem of simplifying a really large assortment of browser and system
transformations, adding in yet more capabilities through customized event handlers and programmatic
manipulations of the browser’s document object model (or DOM). The need for simplification
is exemplified by the example shown here. Internet Explorer allowed a user to fully
customize text background and foreground colors along with all the hover and visited links
colors etc. But this required up to 80 steps… way too many for people to do even if they
knew which colors best suited their particular visual difficulties and preferences.
So what did we do? We built a simplifying layer that hid all the underlying complexities
while optimizing browser behavior as needed for each individual user.
Here are two screen shots from our first release of Web Accessibility Technology (which later
became accessibilityWorks for Firefox). What you see is a band at the bottom of the browser
that cycles through around a dozen transforms that can be directly applied either alone
or in combination. On the left, the three most common color preferences are selectable
with a single click with all other colors (for example visited link) optimized for that
scheme. On the right is the ability to turn the page into a single column allowing large
text zooming without awkward horizontal scrolling. This is an example of a transformation performed
directly on the underlying DOM.
The last project I’ll talk about returns us to the beginning of my IBM career. We were
building the prototype of what became the world’s first voice mail system. But we
knew we didn’t know how to map all the functionality of that system to the 12-buttons of a telephone
keypad. This was going to take a lot of experimentation and a lot of re-programming in a relatively
low-level language. The team leader, Stephen Boies, himself a psychologist from the same
University of Oregon Cognitive Psychology program, and 10 years my senior, hit on the
idea of what is probably one of the first user interface management systems.
The UIMS we built separated the interface from the underlying functionality of recording,
playing back, sending, retrieving, and otherwise manipulating voice mail messages. Importantly
it allowed behavioral specialists to experiment and directly program changes.
So I’ve been practicing user-centered development for a long time, over 40 years at this point.
And I’ve learned a few things that are probably worth sharing. First, the approach works.
It doesn’t slow things down. It focuses the work and gets you to a better solution
faster than any other technique I know. Preaching to the choir here at CHI? Likely so
Second, there are lots of reasons you won’t be able to do this without a struggle. We’ve
looked at a few of the barriers you’ll likely encounter and sketched out how we overcame
them. You’ll probably encounter new ones too.
Third, structure your system so you can keep iterating as you learn more. There are lots
of ways to do this and lots of examples. But take the time to build your system to easily
change as you learn what works.
Fourth, structure your environment, most importantly the people you work with and the channels
you receive feedback from, so you can’t easily hide from the truth. Every project
I’ve been on has made mistakes. The trick is to detect them as quickly as possible.
One last thing I’ve learned, following this approach can be really satisfying. You’ll
build great systems.
If you want to learn more about these projects I’ve included a few references here.
I’ll end not only by thanking you, the viewer of this talk, but some of the collaborators
I’ve had over the years on the projects mentioned here. Nothing would have happened
without them and I have been more than lucky to know and work with them.