[MUSIC PLAYING]
COMET is an experimental system developed
at Columbia University that generates
interactive multi-media instructions for equipment
maintenance and repair.
COMET combines research, and learning, and natural language
generation, and in knowledge-based graphics.
The form and content of all text and graphics
is created interactively.
Thus, instructions can be customized
on the fly for the individual user or situation.
In our work, we have concentrated
on coordinating multiple media.
This makes it possible for the text and graphics
that COMET generates to work together to communicate
information effectively.
We are using a radio modeled after the Army AN/PRC-119
communications radio as our experimental domain.
COMET has hierarchical models of the radio and its components
and of a number of actions that can be performed in maintaining
and repairing the radio.
These models are encoded using ISI's LOOM knowledge
representation system.
An additional graphics model keyed to the LOOM
representations allows 3D-shaded graphics,
such as that shown here, to be created efficiently.
COMET also has an underlying expert system
that enables it to diagnose user-specified problems
and to suggest tests and repair procedures.
Let's look at COMET's architecture.
On receiving a request for an explanation,
the content planner uses text plans,
or schemas, to determine which information
from the underlying knowledge sources should
be included in the explanation.
The Content Planner produces the full content
for the explanation, which is represented
as a hierarchy of logical forms.
Each logical form specifies what information
is to be communicated but doesn't
specify how the information is to be communicated.
The logical forms are passed to the Media Coordinator.
The Media Coordinator determines which pieces of information
are to be communicated in text and which in graphics.
It refines the logical forms by adding directives
that specify these assignments.
Each annotated logical form is passed
to a set of media generators.
COMET currently has a text generator and a graphics
generator.
Each generator creates material in its own medium
to express information that the Media Coordinator has assigned
to it.
The resulting text and graphics are passed to the Media Layout
component, which formats the final presentation, determining
where on the display information is to be presented.
Real-time rendering and typesetting components then
produce the finished display.
Each of COMET's major components runs
in parallel in its own process on five Sun and HP machines.
Text is displayed using the X11 Windows system,
while 3D-shaded graphics are displayed using
HP's StarBase graphics package.
Interaction with COMET is initiated through a menu.
A user can either invoke the expert system
to diagnose a problem or request an explanation
for how to carry out a specific repair or test procedure.
We'll start off by specifying a problem.
COMET displays a list of possible symptoms
that it knows about.
This time, we have a problem with loss of memory.
When we turn on the radio, the transmission frequency
that we stored previously is not displayed.
The first step is to install a new holding
battery whose job it is to maintain the stored frequency.
The text is generated by the Explanation component.
Graphics are not shown for this first step
since it is a high-level abstract step containing
many complex subactions.
The user can ask for a detailed explanation of the step
by pressing the More button.
Since additional detail has been requested,
COMET generates a sequence of displays
containing text and pictures.
In this case, there are seven displays,
one for each step in the installation procedure.
The user can skip around between steps,
go through them one by one, or can review previous steps
if desired.
In this first step, the user is instructed
to stay on the radio on its top side.
None of what you see is canned.
All text and graphics are designed and generated
after the user requests to see the display.
The user must now remove the battery cover plate,
loosening the screws and pulling the plate off the radio.
Note that COMET is capable of producing complex sentences.
Here, it uses a conjunction.
In designing the pictures, COMET decides
to use an arrow, seen in the right-hand picture,
to show how to remove the cover plate.
The user now removes the old holding battery.
Next, the user is asked to clean the compartment if it is dirty
and then to check the battery's polarity.
The new battery is placed in the compartment.
Finally, the user is instructed to replace the cover plate
by aligning it with the compartment
and tightening the screws.
Now we'll go on to step two in the troubleshooting procedure.
In this step, we are requested to load channel one's
transmission frequency.
We'll ask for more detail by pressing More.
The first three steps in loading the transmission frequency
involve adjusting various knobs on the radio's front panel.
The Media Coordinator has specified
that both text and graphics are to be
used to tell the user to turn the channel knob to position
one.
Here, we see how the text and graphics generators
use different resources to achieve the same communicative
goal.
For example, graphics uses an arrow
to illustrate the action of turning.
The arrows tip indicates the point to which
the knob is to be turned.
Text selects the verb to set since the knob has
discrete settings.
Here, the lexical chooser uses semantic information
and the underlying knowledge to influence its choice of verb.
The user is next requested to set two other knobs.
Finally, the user is asked to enter the frequency.
This task requires a sequence of actions
which are represented by a single complex logical form.
Many of these actions involve pushing buttons,
as shown in the pictures.
The Media Coordinator decides to express
much of the information in text only.
For example, the overview, enter the frequency,
is expressed only in text.
The next sentence contains several pieces of information
that are communicated in text alone.
The causal relation between pressing
the frequency button and the display
showing an arbitrary number is expressed only in text.
Note how COMET can use embedded sentences, as in, this
will cause the display to show an arbitrary number.
As well, the fact that the number
is an arbitrary one rather than the specific number shown
in the picture is communicated clearly in text
by using the word arbitrary.
Here, text is used to express concepts
that would be difficult to encode graphically
without requiring that the user learn
new graphical conventions.
In contrast, COMET expresses other information only
in graphics.
For example, the picture tells us where the button is located,
its size, and its shape, while the text does not.
These are all pieces of information
that the Media Coordinator has specified
should be communicated by graphics rather than by text.
At this point, we'll return from this explanation
to COMET's top level.
Now let's look at a few other examples of displays
from explanations produced by COMET.
This display helps explain how to remove
the radio's primary battery.
Here, comets graphics generator uses
a combination of techniques to indicate that the latches are
to be unsnapped.
Arrows specify the direction of unsnapping.
In addition, the latches are shown both before
and after they are unsnapped, with the before version shown
as a transparent ghost image.
The text generator is able to select words based
on the previous discourse.
Since the system has previously instructed the user
to remove the primary battery, it later
says, reinstall the primary battery,
return the primary battery to the radio,
instead of just using the word "install."
You have seen several examples of
integrated multi-media explanations generated
interactively by Columbia University's COMET system.
Alt text and graphics have been created on the fly
by COMET's text and graphics generators.
Fine-grained coordination between these generators
was accomplished through the use of a common content
description.
COMET's Media Coordinator annotated the content
description to indicate which generators were supposed
to express which information.
We're currently extending COMET in a variety of ways.
For example, we're making it possible for the Media
Coordinator to negotiate with the generators
and for the generators to communicate with each other.
This will allow even tighter coordination
between text and graphics.
Systems based on the techniques that we're developing for COMET
will someday make it possible for multi-media technical
documentation to be routinely created
on the fly, customized for the individual user and situation.
[MUSIC PLAYING]