Hello, everyone, I'm Toby Li, I'm presenting our work on Screen2Vec, which is a semantic embedding technique for GUI screens and components.
This work was done with Lindsay Popowski from Harvey Mudd College, and Tom Mitchell and Brad Myers from Carnegie Mellon.
What we did in this work on a very high level summary is a technique for embedding the semantics of GUI screens into a vector space, so that screens that are semantically similar to each other can be closer to each other in this space.
Screen2cec is a part of a larger project called Sugilite, which is an interactive task learning agents we have been working on in the past four years that can learn tasks and the task relevant concepts interactively in a variety of domains from end users through a combination of natural language instructions and demonstrations on existing third party mobile apps.
An important research contribution of Sugilite is that it features a new mixed-initiative multimodal GUI-grounded approach, where the agent sits in between the user and the GUIs. This approach leverages existing app GUIs as a medium for the user to communicate with agent, the user demonstrates doing a task on GUIs while explaining the task in natural language with references to the GUIs.
The main motivation of the Screen2Vec project for us was to enable the agent to learn more generalizable and more robust task knowledge from users by having a better representation of the semantics of the GUI screens, you can learn more about the entire Sugilite project if you look up our previous papers.
We figure that with the available data, we can construct a user agnostic model of app UI screens and interactions. The representation we've done with knowledge learned from user instructions in Sugilite are mostly symbolic so far, but we see an opportunity in moving into neural network.
So as a first step towards this direction, we develop Screen2Vec, which is this self supervised approach for embedding the text contents, the visual layout, the app context and the design patterns of each app GUI screen into a vector space, so that screens that are semantically similar to each other can be closer in a space, which is useful for representing the task at a higher level of abstraction.
The approach we use for training the model is conceptually similar to word embedding methods, such as word Word2Vec, Glove, or ELMo. But we design the model structure to reflect the structure and the different types of information in GUI screens and components.
There are two levels for the model. The first level shown the light gray shade encodes individual GUI components within a screen. And the second level encodes GUI screens. Both models are trained in a continuous bag of words or CBOW task. We use a data set named RICO in training our model, RICO contains interaction traces of users interacting with over 66,000 screens from 9,000 apps. Each trace contains a sequence of app GUI screens. As shown on the slide. Let's first look at the first level of the model.
In the first level, the CBOW task predicts each of the GUI component using the context of other components near it on the same screen.
For example, for the screen on the right, let's say we want to predict a TextView item Hot Drinks using its nearby items. So this is similar to how you train a Word2Vec model using a CBOW task as shown on the left, where you have a sliding window and try to predict each target word in a sequence using the other words around it.
Then, in a second level, we used a similar CBOW task on the sequence of screens from the user interaction trace, the model encapsulates the individual UI components. The visual layout of the screen, and the app context represented in this App Store textual descriptions, you can find more details in the paper.
We have 79 Mechanical Turk workers evaluate more than 5000 pairs of nearest neighbor results generated using our Screen2Vec model, our model performed better than a two baselines.
Here are some example nearest neighbor results. You can see that for the request ride screen in Lyft, Screen2Vec is able to retrieve the equivalent screens in similar apps that have similar contents and layouts.
The Screen2Vec vectors also have composability. For example, if you take the search screen of the Marriott app, add the show result screen of a cheapoair app and minus the search screen in cheapoair. You can get approximately the show results screen of the Marriott app, as well as the show results screen of other similar travel booking apps.
For the future work we're currently looking at using a Screen2Vec representation for generalize task procedures across different apps in a Sugilite agent
Screen2Vec would also allow us to represent the mobile tasks at a higher level of abstraction, which helps improve the robustness of the agent. Lastly, we're also interested in exploring using Screen2Vec for helping with the GUI design process and for generative process.
And that brings us to the end of my talk today. Thank you very much.