Hello everyone, my name is Lihang Pan from Tsinghua University.
Today I am going to introduce our paper "A Human-Computer Collaborative Editing Tool
for Conceptual Diagrams".
Editing conceptual diagrams is a typical Office task, which involves complex and tedious operations
in current GUI interfaces.
For example, to increase the distance between four elements while keeping the position of
the second square unchanged, users need to manually drag and adjust each element one
by one.
This is because users have to convert their editing goals into complex operations and
manually perform them in the GUI.
To solve this problem, we propose a Human-Computer Collaborative Editing Tool (CET) that reduces
the interaction burden while ensuring interaction accuracy.
CET achieves this goal through task division and collaboration between humans and computers.
Users can describe high-level editing effects through multi-modal instructions, and the
computer calculates, recommends, and performs detailed and precise operations accordingly.
For example, instead of manually dragging a new element to a target position, the user
can simply draw a rough shape on the canvas and say "Create a circle here", and the computer
will automatically help the user achieve this editing effect.
In this paper, we focus on the editing of conceptual diagrams, which are important in
education, research, and software development.
Moreover, mobile devices have become increasingly popular for editing tasks, but compared to
desktop devices, mobile devices have fewer functions and more interaction-related problems.
Therefore, we narrow the interaction device down to smartphones and focus on the editing
problems of conceptual diagrams on mobile devices.
We first conducted a user behavior observation experiment to determine the specific task
division between humans and computers.
In the experiment, one participant acted as the end user and the other acted as the human
assistant.
The end user gave multi-modal conceptual diagram editing instructions to the human assistant,
who tried their best to execute the instructions.
The experimenters observed both participants' behavior.
The human assistant's operations were all the tasks that needed to be performed, while
the user's instructions only covered a small part of them.
The computer needed to fill the gap between the user's instructions and GUI operations.
Besides, we analyzed how the human assistant bridged the gap to determine how to implement
CET's functionality.
The results of Study 1 showed that there was a gap between the end user's behavior and
the human assistant's GUI operation.
The end user's instructions only involved the important features of the editing content
and usually ignored the unimportant features.
In diagram editing tasks, the most important feature is the position of the elements.
GUI operations require precise numerical values for all details, while the end user typically
specifies a fuzzy range through gestures or element comparisons.
In addition, the user's instructions may conflict with each other and do not indicate how to
resolve the conflicts.
All conflicts in GUI operations must be manually resolved.
Furthermore, we observed three strategies that human assistants used to bridge these
gaps.
They used experience-related prior knowledge to supplement missing or fuzzy parameters,
such as replacing "blue" with a better-looking "sky blue".
They also used content-related posterior knowledge, such as copying the properties of existing
elements or the relationship between properties.
To handle conflicts between instructions, human assistants would spontaneously modify
multiple diagram elements, even if these elements did not appear in the user's instructions.
For example, when the user requested to move square B, which is located between squares
A and C and has equal distances to both squares, the human assistant might adjust the positions
of both A and C to maintain the equal distance relationship.
Based on the results from Study 1, we designed CET's functionality.
CET has three main functionalities to "autocomplete" the user's vague inputs.
First, CET fills in missing or vague information in the user's instructions.
This is because user instructions are often imprecise and users usually care more about
the relationships between elements than specific attribute values.
Second, CET needs to handle conflicts between instructions in the most reasonable way.
Third, CET needs to provide users with multiple possible candidates because the vagueness
inherent in user instructions makes it difficult for even a human assistant to fully determine
the user's actual needs.
CET's system design is divided into five parts, with the most fundamental being mathematical
modeling of the editing content, which contains content-related knowledge.
During command parsing, CET parses the user's verbal instruction and aligns it with the
user's gesture.
Solution generation is the core of CET, in which CET calculates possible solutions based
on the user's instruction and recommends them to the user based on their scores.
Finally, CET predicts the user's subsequent behavior and waits for the user's next round
of operations.
Next, we introduce how we implemented this system design in SGDiag.
The most significant feature of our content modeling is the use of linear equations to
describe the positional relationships between elements.
During solution generation, we supplement missing positional information by performing
elementary transformations on the equations.
We can also use these linear equations to determine an optimization objective during
conflict resolution, allowing us to calculate different conflict resolution strategies.
The interaction design of CET consists of three parts.
Before the user gives instructions, CET predicts the user's subsequent actions and provides
corresponding prompts.
We make this prediction based on the user's behavior and the content being edited.
During instruction input, CET displays real-time understanding results of the instruction and
supports manual modification by the user.
After the user gives the instruction, CET proposes multiple possible solutions.
To help the user better understand, we also show the internal logic of these solutions
simultaneously.
We conducted user experiments to evaluate whether SGDiag can support users in accomplishing
diagram editing tasks with high efficiency, good results, and positive attitudes.
We compared SGDiag with PowerPoint on Android smartphones.
The experiment was divided into two stages.
In each stage, two groups of participants used different tools to complete the same
diagram editing tasks.
After all editing tasks were completed, we invited judges to score the editing results.
The results showed that only 16% of the instructions were accurate, and SGDiag needed to generate
solutions for the remaining instructions.
The top-1 accuracy rate of the generated solutions was 94.84%.
As shown in the figure, users achieved better results in less time with SGDiag.
Compared with the baseline, SGDiag can save 32.75% of time and improve the editing results
score by 21.89%.
In subjective evaluation, users also gave more positive feedback for SGDiag.
In conclusion, in this paper, we explored the possibility of combining AI and office
tasks.
This combination corresponds to a human-computer collaboration system, and the most important
aspect of this system is the task division between humans and computers.
In this paper, we determined the task division by observing the human-human collaboration
process.
The ultimate goal of task division is to bridge the significant gaps between multi-modal behaviors
and traditional GUI operations.
In this way, we can ensure both high accuracy and high interaction efficiency.
That's all for my presentation.
Thank you for your attention.