Hello everyone. My name is Tom Veuskens, and today I'm presenting ...
... our work on Rataplan: Resilient Automation of User Interface Actions ...
... with Multi-modal Proxies, which was executed together with my ...
... advisors Kris Luyten and Raf Ramakers at Hasselt University.
We start by showing a video where Rataplan is being used to automate an action ...
... which plays every song from a specific artist in the top charts in Spotify.
First, Rataplan records every action the user demonstrates, and extracts ...
... the widgets the user interacted with.
After the demonstration is done, Rataplan asks human-readable ...
... follow-up questions to obtain the intention of the user.
For example, instead of playing exactly the demonstrated song, here, the user ...
... specifies Rataplan should play all songs of the demonstrated artist in a looping sequence.
After specification, the user executes the action by interacting with a ...
... proxy modal, in this case a tangible control.
Monitored changed in the interface are forwarded to output proxies.
To overcome the limitations of API based methods and macro recorders ...
... we present Rataplan, which combines the convenience of demonstrating ...
... actions, present in macro recorders, with the robustness and abilities ...
... of API based methods. We do this by improving upon ...
... state-of-the-art pixel-based reverse reverse engineering methods and acquiring additional ...
... information through human-readable follow-up questions.
Rataplan contributes a set of strategies to support resilient ...
... automation of modern interface that have a flat and minimal style, ...
... have frequent data and state changes, and have dynamic viewports.
We identified four challenges where current state-of-the-art ...
... approaches fail, and present a solution to each category.
The first issue are ambiguities, where one or more similar or identical ...
... looking UI elements are present. Current state-of-the-art systems ...
... are unable to differentiate between found matches. This is resolved ...
... in Rataplan by automatically identifying and comparing ...
... nearby UI elements during both demonstration and automation.
For example, to click the correct star icon, ...
... we expand the search region to eliminate matches. At the first ...
... iteration, the second mail is eliminated, as in the case, the sender ...
... of the mail does not match. The second iteration eliminates ...
... the last email, as the subject does not match, leaving only the correct match.
Even when there was no ambiguity present during demonstration, ..
... this approach ensures the correct one clicked during automation.
The second issue, hidden information, is bifold.
UI elements, visible during demonstration, can be completely hidden ...
... during automation, and only become available when hovering specific areas ...
... in the interface. For example, as the archive button is hidden ...
... until a mail is hover, state-of-the-art approaches won't be able to find it.
Rataplan identifies the relevant UI elements, in this case ...
... the combination of the mail sender and subject, and hovers them automatically to reveal the correct element
The second case is when UI elements have become partially hidden.
For example, when labels are cropped due to changes in the viewport or resolution.
This frequently occurs as a result of resizing different ...
... panels in an application. Combining optical character recognition ...
... with pixel-based recognition improves the accuracy when labels are cropped or resized.
The third issue are interface interruptions, such as pop-ups or ...
... advertisements, from the operating system or the automated application.
These can happen both during recording and execution.
If an interruption happens during recording, Rataplan won't be able to play ...
... it back, and hence automatically remove it. If an interruption happens ...
... during the automation phase, Rataplan will automatically ...
... try to dismiss the pop-up by looking for labels containing the ...
... text "cancel", "exit", or "later", or widgets containing a close icon.
If Rataplan if unable to find such a UI element, ...
... the application will be restarted and the sequence will executed from the beginning.
Thus, requiring no user intervention, in contrast to existing approaches.
The final category we identified, are applications being in a different state ...
... between demonstration and execution. To make sure ...
... the demonstration contains all necessary action to navigate to, and interact ...
... with the final widget, users are required to open applications during ...
... demonstration through the Rataplan interface.
When the user demonstrated clicking a play button for a song, by default, ...
... Rataplan assumes the user means exactly that song.
However, through follow-up questions, the user can reveal an intent to ...
... automate a parametric or looping variation of the demonstration.
For example, in this case the user clicks the first song of Ed Sheeran ...
... in the top charts. This demonstration can mean exactly that song, ...
... the first song of the artist in the playlist, ...
... or every song of the artist in the playlist.
The user is asked to identify relevant UI elements defining the intended behavior.
In this case, the header for the play button, and the artist name.
Rataplan then generates possible queries, from which the user can ...
... pick one, and visualizes the effect of the query in the current interface ...
... to facilitate selecting the correct one. When the playlist changes ...
... over time, the behavior of the specification is changed.
As we show in our studies comparing Rataplan to "Help, it looks confusing" and "Sikuli Slides" ...
... the current implementation of Rataplan provides significant benifits ...
... over the current state-of-the-art.
For details on these studies, we refer to our paper.
However, there remain some boundary cases, such as infinite scrolling lists ...
... and applications that keep their state when restarted.
Additionally, Rataplan can be extended to suggest more ...
... intricate queries such as lexical ordering, and "OR"-relations.
Thank you for your attention.