- [Pedro Lopez] Alright, good morning everyone.
Welcome to this circuit session.
My name is Pedro Lopez.
I'm from the Hasso Plattner Institute.
I'm very happy to chair this session,
because this is the kind of stuff where
I lost on my daily routine, you know to building prototypes,
So all of what you're going to see today is
fantastic systems that
actually help people solve reality problems in
prototyping hardware,
systems that sense how you're stacking things together,
systems that help you de-bug things on a hardware level.
It's really really fantastic work.
The first one is going to be Bifröst by Will McGrath,
Stanford University.
Enjoy.
- [Will McGrath] Great, thank you for the introduction.
Hello everyone. I'm Will McGrath and I'll be presenting our
Bifröst system for embedded debugging today.
So, to motivate this work,
consider a maker who wants to build a
noise-hazard warning indicator for their space to
protect everyone's hearing.
This Arduino-based device measures ambient sound level with
a small microphone, and displays the
current sound level using a ring of LEDs.
The LEDs will also change color based on the
average ambient sound level.
Like most embedded systems this device is implemented using
both code and electronic circuit;
however, if the system doesn't perform correctly,
it can be difficult to find out if the error lies in the
software, hardware, or on their interface.
Any potential error in this system could be in
any number of places.
Experienced debuggers developed an intuition for
where to start searching based on observable information.
But it can be difficult for inexperienced users to
know where to even start looking.
We believe that the fundamental cause of
this confusion is that much of the system state is
invisible and hard to examine.
In response to these challenges, we wrote Bifröst.
Bifröst unites data from multiple tools that
simultaneously measure the electrical and
program activity of an embedded system.
First I'm going to go through a couple of
existing tools used for embedded debugging.
Next, I'll walk through an example of
how Bifröst can be used in practice.
Then, I'll go into the technical details of
how Bifröst capture instrumentation and
checking infrastructure work.
Last, I'll go through the results of our user study.
So, when you're errors that you expect to
be electrical in nature,
an embedded designer might use a digital multimeter to
measure a single point voltage, or a logic analyzer to
measure multiple voltages changing simultaneously,
as in a a protocol box.
Last year at UIST, our group presented the toast board,
which incorporates voltage measurement into
every row of a smart breadboard.
Meanwhile, for software bugs, a programs first line of
defense is often print-statement debugging;
however, this approach quickly becomes tedious and
only provides basic ordering information about
code execution.
Breakpoint debugging as implemented in tools like JDB,
is a powerful tool for precisely manipulating program state,
and it can be useful for adjusting complex software bugs,
but it isn't necessarily supported by
all embedded platforms.
In contrast with tools that provide a limited view of the
hardware or software state of a system,
our goal with Bifröst was to combine the views provided by
multiple independent tools to display a
unified view of system state debugging.
So Bifröst provides a familiar view of analog and
digital signals provided by a tool like a logic analyzer,
but it also augments it by displaying the value of
program variables in a users program as
they change over time.
The UI also includes a code editor that's linked with the
other views to display the current line of code executing at
any time during a recording of device activity.
Last, Bifröst enhances ordinary print debugging by
automatically time stamping any user print statements.
So at a high level, Bifröst captures data about a
user's Arduino microcontroller based system by using an
off-the shelf logic analyzer,
a custom Arduino interface board,
and software instrumentation through code rewriting.
The logic analyzer measures the electrical activity of the
microcontroller's pins, and the
software instrumentation outputs a stream of data about the
program activity that is captured by the
same logic analyzer.
The data collected by that logic analyzer is then read by
Bifröst and displayed by the UI.
At this point the data is also made available to
Bifröst checking infrastructure.
Now I'm going to go through an example of
how you'd use Bifröst to debug the
noise-hazard monitor from earlier.
So, to begin, the user has written code to read the
ambient sound from the sensor,
and light an LED when the sound exceeds a given threshold;
however, after capturing the devices activity with Bifröst,
the system points out that the LED will never turn on
because the pin is not able to actually go high.
In this case the error is due to a mis-wiring.
After the mis-wiring is addressed,
the LED now behaves correctly.
So in this example, the LED ring that should measure the
sound level is behaving somewhat erratically,
so the user records a trace of it's activity while
simulating noise input by firmly tapping the table.
The trace value reveals that the
sensor value variable holding the value read from the
analog-to-digital converter is
overflowing and looping around.
The 10-bit ADC value is truncated when it is
stored in the 8-bit variable,
as you can see with the spikey pattern highlighted in
white here.
In this case the issue can be simply addressed by
changing the variable to a full amp,
which is 32-bits on this particular processor.
When the user re-records their device behavior with the
fix code, the LED ring behaves properly,
and the variable trace confirms that the error is now fixed.
So in this example, the user has extended the
previous state by adding noise averaging code,
but when they run it,
the ring starts out as yellow instead of green.
It correctly transitions to red and back to yellow in
response to sound, but never returns to green.
To investigate this issue,
the user records another trace with Bifröst.
The trace reveals that the state called the
average level in red here,
transitions from one to two but never actually assumes a
value of zero which would turn green in the users code.
Fortunately, this misunderstanding about sensor ranges is
easily addressed by changing the equals signs to
less than equals in the case statement.
When the user reloads their
program onto their microcontroller,
they verify that the ring starts out as green,
before correctly transitioning to yellow then red.
The trace reveals the progress of
the output state variable here in green compared to the
internal average that's being stored shown in red.
So when we were designing Bifröst, we attempted to make it
as easy as possible to jump from one view of the data to
other views to aid exploration and debugging.
I'm going to go through a couple of these interactions now.
First is the link between the time-series-trace view and
the user's code.
A click on the trace view highlights the
corresponding line of code in the editor.
The link between the circuit and variable trace view and
the program is simple, but powerful for
exploring causal relationships between code and the
inputs it reads or the outputs it causes.
Next, we'll link the program back to the trace-view.
A user can click on the line number of the editor to
quickly jump to the next time that
line was executed in the trace.
This is useful for following the behavior of a
specific function through time.
If the line was not executed, the system warns the
user with the blue bar up top.
This functionality can also be used to quickly test if
things like error handlers ran during a trace.
Last, we link the console back to the code and
trace through automatic timestamps.
When a user clicks on a timestamp next to a
print statement that they issued,
the system focuses the other two views on the
exact instant it was sent.
This can allow a user to easily step backwards to
examine the cause of the print or forwards to
see what happened afterwards.
Now I'll discuss how
Bifröst actually captures data traces at a low level.
So, when a user starts a new capture,
their code is first send from the UI to our
local server running on the same computer.
The server automatically configures the logic analyzer,
instruments the code, and programs the microcontroller.
As the user's code is running,
the logic analyzer captures both the electrical activity of
the microcontrollers pins,
and the debugging information generated by the
code instrumentation.
After the data is collected, it's read from the
logic analyzer as a CSV, and provided to the UI,
where its visualized and then processed by
Bifröst checking infrastructure.
Bifröst also supports a breakpoint debugging model that
wraps JDB and captures the electrical activity that
occurs during steps and for breakpoint debugging.
Now I'll discuss briefly our
software instrumentation approach.
So Bifröst automatically finds statements in
the user's program and adds a
print statement before each line that writes an op code to
the microcontroller's hardware serial box to mark the
bounds when that line was executed.
And now ill go into Bifröst checking infrastructure.
So Bifröst supports two distinct types of checks.
Automatic checks can determine everything they need to
run solely from the user's program.
The simplest example is that a write statement to a
pin should cause that pin to assume the value of
the write after it executes.
A more complex example is that
higher level protocols can be enforced in
hardware solely based on their specification.
User-parameterized checks on the other hand are a
small domain-specific language that can be used to
enforce relationships that aren't immediately obvious to the
system from the user's code.
For instance, this would specify a checker that if
a pulse error on D1 occurs, an interrupt function on
line 11 will run within a certain time limit.
So like for instance to match an interrupt handler.
So our evaluation sought to
validate whether Bifröst aids embedded debugging.
We trained ten subjects of
varying skill levels to use Bifröst and
had them perform six embedded debugging tasks during a
90 minute session in a within-subjects study.
The tasks consisted of pre-made circuits with
bugging code and included mis-configured pins,
thresholding errors, switch de-bouncing, and
timing and logic errors.
All of our users successfully made use of Bifröst,
but we noticed that some of the
novice users didn't fully utilize all of Bifröst features.
We attributed this to the fact that
novice users didn't have a fully developed mental model of
their system, and thus weren't able to fully make use of
all of the data provided by Bifröst.
Conversely, experienced users made most effective use of
our tool.
Interestingly, we observed that some of the
control participants made errors while
instrumenting their code, which helps validate our
approach of automatically instrumenting a
user system and dovetails with related work in the field.
So, we noted that the subjects in this
study used several recurring strategies when
addressing these hardware debugging problems.
For hardware interfacing issues at the boundary between the
software and the hardware,
users frequently compared variable electrical signals using
Bifröst trace-visualization.
They also made frequent use of the automatic checks.
Meanwhile, for debugging timing and logic errors,
users frequently used the
jump-to-next statement execution functionality,
as well as the trace-view for comparing multiple variables.
They also made use of the enhanced print statements for
detecting certain conditions.
So Bifröst, since we're rewriting and adding to your code,
there's naturally going to be a performance hit to that,
which varies depending on the application and
whether that matters or not also depends on the application,
but for many hobbyists applications it
shouldn't create a large impact.
Additionally, Bifröst only focuses on the code that the
user has written, and we're not really gearing to
allow people to debug library or
firmware code in this version.
So, in summary, Bifröst is a tool that helps debug and
understand embedded devices,
that collects, checks, and visualizes data about a
device's hardware and software state.
Thank you for listening and looks like we have some time for
questions as well.
(applause)
- [Pedro Lopez] The idea is that you guys come to
the microphone and ask really interesting questions.
- [Audience Question] Just two quick questions:
One, I'm curious what the timing precision and
resolution is of our time stamping and
also what types of logic analyzer hardware do you support?
- [Will McGrath] Sure. Thank you. Good question.
For the timing resolution we're primarily limited by the
rate at which our side-channel that
we're outputting instrumentation data to can output at.
In this case its about 2 megabod,
and we've experimented with other means of
pulling that data out of the processor,
but haven't been able to improve that a whole lot.
And, then for logic analyzer hardware,
currently we support the
Saleae or Saleae depending on who you ask,
which is a USB-3 logic analyzer with
16 channels that measures both analog and digital data,
but really we just need to be able to
programmatically control it and anything would work.
- [Audience Question] I'd like to ask a question as well.
We talked a little bit earlier before the talk started,
but I was wondering, I mean,
a lot of really nice features depend on the
specification of the hardware, so if we know that
those pins for that particular Arduino board are not in
use you can warn the user.
So I was wondering if there was some sort of
method description or something that
you can port easily new boards and
bring in new hardware and make the
tool more kind of grow up.
- [Will McGrath] Sure, thank you.
So, in the case of the checks,
the checks are actually very flexible.
We write them in javascript for the automatic checks,
so you could pretty easily extend it to different systems.
The rest of the tooling on the other hand is a
little more tied to the Arduino platform,
but the approach certainly extends to other types of
microcontrollers as well.
- [Audience Question] I have a question.
So right now in Bifröst you instrument each line of
code by adding a server out statement,
but Arduino runs on the Atmega platform,
which should have J-tag debugging features.
Is there a reason why you went with
manual instrumentation rather than reading the
registers directly?
- [Will McGrath] Sure. Thank you, first of all.
When I mentioned we explored a couple of other ways to
pull data out of the system, one of them was actually a
third party library that uses J-tag to,
rather than stopping execution of the device,
very quickly read data out of a ring-buffer in memory.
What we found in practice is though,
was that, that,
the system tends to be a lot more brittle when
built that way.
There's something actually very simple and robust and
appealing just putting a lot of print line statements.
If you have non-timing critical code,
you can imagine ways to exclude timing-critical code,
and tell the system hey,
don't mess with this or else you'll mess up my
bit-banging or something.
But when you have non-timing critical code,
it'll run and it'll output that data,
and then you can very easily capture it,
whereas with J-tag, you now have state in
debugging registers and a couple of different middle layers,
and on the PC to actually control J-tag,
and just in practice we found that kind of unweilding.
But, in general it's very powerful,
and if those issues were addressed it could be an
effective way. And we also support it partially in the
introspective mode, but good question. Thank you.
- [Pedro Lopez] If there is no further questions
we're going to move on to the next speaker.