Hi, I am Damien Masson, a PhD student at the University of Waterloo in Canada and in this
talk, I will be presenting Chameleon, a project that started during my Master's at the University
of Lille, in France.
This project was done in collaboration with my supervisors Sylvain Malacria, Edward Lank
and Géry Casiez.
This project started as a simple observation that digital documents rarely incorporate
animated or interactive figures.
Instead, digital documents are often nothing more than a digitized paper version.
However, an interactive or animated figure can be of great help to understand a concept,
or a process.
To take an example from the HCI community, a task as common as describing an interaction
technique would be easy if we could let readers experience it.
Instead, when limited to static formats, authors often have to use careful wording to describe
the interaction technique.
To the question “Why is there no interactive figures in documents” we believe that the
answer lies in the lack of support from common tools and file formats.
As an example, the PDF Format made an attempt at supporting interactive figures by leveraging
Flash.
However, Flash is now discontinued, and no replacement solution has been offered.
Moreover, the process of including such a figure was tedious, and the feature was only
supported by Adobe Acrobat Reader and therefore, when using Preview on macOS, or even just
viewing PDFs from the web browser would not display these interactive figures.
Nonetheless, several solutions were adopted by authors to work around the limitations
of document formats.
For example, it is common to see comic strip figures describing the interaction in several
steps.
Another solution is to place a link to a website somewhere in the paper.
Authors are then freed from the limitations of the document by creating a dedicated web
page leveraging HTML5.
Finally, another more radical solution is to switch to a new, more permissive, file
format.
However, this solution is difficult to adopt in practice, existing file formats being firmly
anchored in users’ habits and replacing them requiring significant changes, both in
software and infrastructure.
Moreover, this last solution leaves behind all the already published documents.
A more ideal solution would be to preserve the existing formats and workflow, and, instead,
to provide tools to augment digital versions of the files such that both new and legacy
documents can become reactive documents.
Expanding on this point, support for the creation, dissemination, and viewing of reactive documents
should allow rich and flexible access to interactive features.
It should also preserve existing document formatting for visual correspondence with
paper versions, leverage existing file formats, and allow users to continue to use their current
document viewing tools rather than forcing them to adopt different software applications.
In that optic, we propose Chameleon, a tool that combines computer vision feature matching
algorithms with an open database format to allow the incorporation, and overlaying of
dynamic content over any type of document, without modifying existing applications or
the source document.
Chameleon works by first identifying files that contain augmentable figures . Then, Chameleon
looks for registered figures on-screen by analysing pixels
Finally, if a figure is found, the augmented version is overlaid on top of the static one.
Chameleon comprises two main parts: a Registration Tool to register a figure from a document
with its augmented version.
And an Augmentation Tool to analyse opened documents and augment them if augmentable
figures are found.
First, the registration tool.
Using the registration tool, an author can select a figure in a document, and then specify
the location of the corresponding HTML5 version of the figure.
The figure is then registered as an augmentable figure for the document.
The augmentation tool works in the background and monitors file access.
Documents containing augmentable figures are captured and their pixels are analysed using
a feature matching algorithm.
If a figure is located, then the HTML5 version of the figure is overlaid on top of the static
figure.
Finally, modifications to the document such as resizing it or moving it are tracked and
the augmented figure is modified accordingly to follow the document.
Put another way, when Chameleon sees a registered figure, Chameleon overlays the HTML5 version
in lieu of the static figure.
To support Chameleon, the pixel analysis algorithm locating figures has to be fast;
it also has to be accurate, especially with figures such as charts or sketches which are
common in digital documents, and even more common in research papers.
Finally, another important property of the algorithm is that it should be resilient to
large scale variations.
Documents being often resized or displayed on different screen sizes.
We chose to use a feature matching algorithm to fulfill these requirements.
Feature matching is a computer vision technique that was originally developed to locate objects
in natural images, even with the presence of slight variations in camera position, scale
or even color and therefore seemed particularly adapted to our use case.
However, feature matching algorithms were not conceived nor tested to locate digital
figures such as charts which have different characteristics than natural images.
Therefore it was not obvious which feature matching performed the best in our context.
To find out, we conducted 2 experiments.
In the first one, we compared a wide variety of algorithms on a dataset of more than 15
hundred figures extracted from scientific papers and presentations.
The algorithms were tasked to locate the figures in their original documents and with large
variations of scale.
In the second experiment, we evaluated the 2 best performing algorithms more in depth.
The goal was twofold: first, the goal was to tweak the parameters of these algorithms
to get the best possible performances; second, we wanted to test even larger variations of
scale.
Our result showed that BRISK and SURF offered the most promising results to first, find
scale independent features and then correctly match these features.
In the second experiment, we found that, even under large scale variaitons, BRISK and SURF
performed well when optimized to yield a high number of true positives while minimizing
false positives.
Over all scale variations, the proportion of true positives was 88% for BRISK and 94%
for SURF and less than 2% for false positives.
Finally, we also evaluated Chameleon through a first-use study.
We wanted to evaluate if users can use Chameleon; if they find it faster and more convenient
to use than existing solutions; and if they would use Chameleon in everyday life.
Participants were shown how to use Chameleon and then asked to augment documents through
3 scenarios.
In the first scenario, we asked participants to augment the original bubble cursor paper
with an interactive demo of the bubble cursor
In the second scenario, we asked participants to create an Explorable Multiverse Analyses
Reports by including an interactive figure showing different analyses of the data in
an existing PDF.
Finally, in the last scenario, we asked participants to augment a powerpoint presentation by embedding
a youtube video and an interactive chart in the presentation.
All 12 participants completed the scenarios without difficulty validating our first hypothesis.
They also all preferred Chameleon to any other alternative that they could think of, validating
our second hypothesis.
And finally, they all rated the integration of figures above 4 on a 5 points Likert-scale,
and stated that they would be willing to create interactive figures if they could integrate
them in their digital documents.
The current implementation of Chameleon is open source.
Our current version works for macOS but other operating systems can be supported by replacing
the operating system specific functions.
Chameleon could also be ported to mobile devices but its implementation is left as future work.
I presented Chameleon, a system-wide tool to support interactive and animated figures
in all digital documents.
If you have any questions, feel free to reach out.
Thank you for listening.