Hi, my name is Ishan Chatterjee, and I'm going to be presenting today on behalf of my co
authors, Olga to use Richard and Professor Shwetak Patel, at the University of Washington
UbiComp lab.
Over the next 10 minutes, I'll be taking you through our paper, Augmented Silkscreen.
In this paper, we looked at designing augmented reality interactions for electrical engineers
and technicians to assist them in debugging printed circuit boards.
We'll start with a little bit of background.
A printed circuit board or PCB is a sheet of fiberglass with buried conductive paths
to connect electrical components in the same way that wires connect components on a breadboard.
There are actually likely dozens of PCBs in the room you're in right now.
They're in everything from Instapots to Furbys to the phone or laptop you're watching this
on.
Here's a PCB I took out of an Apple Thunderbolt monitor.
As you can see, these PCBs can be exceedingly dense and complex.
Let's take a closer look at this one.
All of these items that you see on the surface are components such as resistors, capacitors,
and integrated circuits.
These boards can have tens, hundreds or even thousands of components on them, many smaller
than the breadcrumb or a head of a pin.
So how do electrical engineers design circuit board?
The electrical engineering design process typically starts with designing a circuit
to meet a set of functional requirements.
The logic of the circuit is formalized into a schematic diagram, which visualizes the
circuits components and symbols in the circuit interconnections is topological lines between
the components pins.
This logic is then transferred to a layout diagram, where components and the connections
between them are placed in physical coordinate space.
And finally, the design is then physically fabricated and assembled into a functional
PCB.
During the process of debugging a new PCB design electrical engineers with constantly
move between the circuit diagrams and board layout diagrams on their computer, and the
physical circuit board on the desk in front of them, in order to validate their design
and understand the nature of the design failure.
Augmented reality has been cited as an effective paradigm for reducing this overhead of tasks
with repeated context switching, particularly those with spatial associations, like the
ones that we see engineers must do during PCB debugging.
So that brings us to our central research question, can augmented reality assist electrical
engineers in their PCB debugging workflows, and also, how can it best support them?
We attack this problem in three parts.
We first ran a needsfinding study in which we collected opportunities and challenges,
pain points, that were offered by our users.
From this, we derived a set of design considerations that we could then apply to part two, which
was designing the interactions.
And then finally, in part three, we evaluated these interactions with our users, we recruited
a handful of participants from academic labs and industry settings.
From the first study, we derived a set of design considerations that we could then apply
as we designed our interaction techniques.
The first we've discussed already is to reduce context switching and facilitate pattern matching.
The second is to show relevant information without cluttering the view.
And here participants expressed interest in having context relevant information accessible
directly on the PCB, and through the design files as well, but we're wary of excess visual
clutter.
Third, we wanted to support the habitual and intuitive interaction with the PCB, while
participants each followed slightly different methods of localizing issues and taking measurements
that generally followed a common approach.
So any AR System should simplify methods of taking measurements and seeking information,
but it should not depart greatly from their current habits or their workflows.
And finally, we want to facilitate collaboration to a few of the participants noted that finding
design elements and following measurement procedures were exacerbated when working with
collaborators who are less familiar with a particular design.
So then we took these design considerations and applied them towards formulating interaction
techniques.
From our discussions, we found a few base fundamental tasks that we could target with
their interaction techniques.
The first that dealt with augmented visualization that is taking information from the design
files on the computer, and bringing them to the PCP such as element localization where
participants can select a component or other elements such as net and pin on the schematic
or layout and have that augmented for them directly on the board.
The second set dealt with augmented interaction, where information from the board was taken
and brought to the design file, just taking a measurement on the board and having that
measurement written back The design file it as you can see here in the inspiration flow
diagram, what might have taken multiple steps.
That's the traditional method in gray, as described to us by their participants can
now be done more efficiently with augmented reality interactions, what you see here in
blue, let's check out a video to help visualize some of these interactions.
Here's a mock up of what an augmented reality workbench for an electrical engineer might
look like.
A user has the view of the schematic and layout on the monitor, a benchtop multimeter connected
to the PC can obtain measurements, and an overhead camera projector system can provide
augmentations directly on the physical PCB sitting on the user's desk.
We work with engineers to design a few core interactions by selecting a component in the
schematic or layout view, the corresponding component on the board is identified.
Similarly, by selecting a pin or net in the schematic or layout view, the corresponding
pin or net on the board is identified.
Identifying components on schematic and layout view directly by pointing on PCB identifying
pins and that's on schematic and layout view directly by pointing on PCB.
Okay, now that we have these coordinate reactions, we can take these as building blocks and synthesize
workflows that reflect the workflows the participants discussed during the needs binding study.
Here, we'll take a look at some of those.
For example, here, a user takes a measurement on the PCB with a multimeter.
These measurement pin locations are recognized and the measurement is annotated on the schematic
and layout views.
During the process of board bring up the user wishes to confirm whether the power rails
are within design specifications.
The user specifies desired power rails and optionally adds test bounds.
The measurement points are indicated on the board sequentially as the user takes a measurement
with a multimeter the value is captured.
In a remote debugging scenario.
The electrical engineer has a view of the schematic and layout.
While the remote operator performing the debugging has an augmented visualization of the PCB.
The engineer asked the operator to identify whether a particular component is installed
properly.
The operator sees the highlighted component and confirms the component is installed.
The engineer asked the operator to take measurements on a particular net, the operator performs
the procedure, and the engineer finds the measurement to be out of bounds.
Okay, after synthesizing these core interactions and workflows, we took these back to the participants
in our last study, which consisted of three parts.
The first was to collect feedback on Korea interactions and workflows.
There's a lot of qualitative feedback discussion in detail in the paper.
So I'll just cover some of the highlights here.
Participants particularly liked the interactions that helped them find components and pro points
on the PCB.
They like the diagnostic measurement workflow, which aided in logging unstructured measurement
queries with associated pro points.
They liked the visual inspection workflow, which immediately provided element metadata
at the board without having to reference the design file.
And finally, they'd like to bring up workflow, which allowed for an ambiguous spatially co
localized and potentially automated pro point visualizations for directed measurement workflows.
Next, we evaluated varying certain attributes of these interactions, including preferences
on how to select items, how much information to display on the PCB and where to put it.
In the interest of time, I won't go into the results of this section, but encourage you
to see the paper if you're interested.
And finally, we took a subset of these interactions and performed a time task with the participants.
We developed an interactive prototype to facilitate this remotely, where the users could use their
own touchscreen device as a standard for the printed circuit board.
across two different tasks, participants were more efficient with air interactions seen
here in blue, as compared to traditional method that you see here in gray.
Additionally, they expressed higher ease and confidence with the AR interactions.
So what's next?
I think the natural continuation would be to work towards designing and developing a
real system that we can test in person as soon as the pandemic comes to a close.
From our discussions, we learned a number of specific practical expectations from the
user base, including items like the precision of augmentations, and the tools with which
the system would need to be linked.
And we describe some of these in the paper.
We also understand which workflows are seem to be specifically valuable and also the type
of workflows these interactions would serve.
We're also excited to see this space grow.
In this work, we found that despite professional populations having access to very powerful
tools, there are many gaps in the usability of their system.
We also see that the bulk of the work done by the HCI community has focused on supporting
maker educational and a hobbyist populations.
So we hope this paper will inspire more work towards supporting hardware workflow challenges
in professional populations as well.
Thank you