Hello, in this presentation I will introduce Marcelle,
an architectural model and toolkit for the design of human-machine learning interactions.
My name is Jules Francoise, I am a CNRS researcher at the LISN lab in Université Paris-Saclay,
and this work results from a collaboration with Baptiste Caramiaux, CNRS researcher at ISIR, Sorbonne université, and Téo Sanchez, PhD student at Inria.
Machine learning system development is still a matter for specialists.
App designers struggle to include machine learning in their design practice.
Domain experts have little control over the development of the systems assisting them with decision making.
With the growing use of machine learning models in applications with end-users, it is essential to include a broad spectrum of stakeholders in the process of building and assessing such models.
While significant research has been dedicated to interactive machine learning, to our knowledge, it still lacks general purpose toolkits that can be used by many stakeholders.
Our contributions in the work are twofold.
First, we propose an architectural model for toolkits dedicated to the design of applications where humans interact with machine learning systems.
Second, we present an implementation called Marcelle, which is an open source toolkit written in JavaScript.
Research in Interactive Machine Learning has highlighted the benefit of bringing human input in the training process of machine learning systems.
Interactive Machine Learning differs from conventional machine learning in involving shorter cycles of data edition, training and evaluation.
As reviewed by Dudley and Kristensson, users are engaged in a large spectrum of activities such as feature selection, model steering and quality assessment,
leading to workflows and interfaces that vary tremendously across projects.
Regarding existing tools, several machine learning frameworks have been widely adopted, such as Tensorflow, PyTorch or Scikit-Learn, along with visualization toolkits.
However, they are not designed for direct interaction.
Interactive machine learning software has been produced, that includes new interaction techniques and workflows, but it often remains domain-specific or task-specific
For instance, Wekinator or RunwayML are dedicated to creative applications.
General purpose toolkits are still missing.
Our architecture relies on 5 main design principles that come from our review of the literature and our own experience.
Here, we present these principles along with details on our implementation.
We propose a modular architecture where the building blocks for designing interactions are components embedding data, computation and interaction.
Components can typically be data sources (for instance a webcam capturing images), data structures (such as datasets), models, or visualization.
Most components provide a graphical user interface that can be displayed on demand.
In Marcelle, components are JavaScript objects created with factory functions. Marcelle uses Svelte internally, however any JavaScript framework can be used for custom components.
In their review of user interface design in IML, Dudley and Kristensson emphasize that while interfaces share common elements, their layout often varies according to the task and context.
We propose that toolkits should offer layout mechanisms facilitating user interface composition.
In our implementation, we provide two types of layouts:
dashboards composed of several pages displaying sets of components
and wizards where components are arranged into step by step guides.
Both can be created using a high-level API for organizing components on pages.
Interactive machine learning applications involve custom workflows where user interactions trigger various types of processing.
Therefore, there is a need for pipelines specifying complex relationships between the users actions and these operations.
We propose a design where developers can build reactive pipelines that propagate change.
Components expose arbitrary event streams that can be consumed by other components, and which can be processed outside components.
Our implementation relies on reactive programming, a paradigm that facilitates the creation, filtering, transformation and combination of event streams.
It enables encapsulated, loosely coupled components where streams act as an interface.
Our goal is to support collaboration between various stakeholders including machine learning engineers, end users such as domain experts, and designers.
We propose the use of flexible data stores supporting heterogeneous data (such as instances, models, results or logs).
Our implementation relies on the FeathersJS Framework, and enables developers to specify where each store’s data should be stored (be it locally or on remote servers).
Since Python has become a language of choice for machine learning, it is essential to provide interoperability with Python.
As a first step, our implementation provides partial support for exchanging models and data between Marcelle and Tensorflow.
With such a modular architecture, toolkits should enable rapid prototyping and extension, while providing support for collaboration.
Our proposed implementation for this architecture is called Marcelle.
It is written in JavaScript, and is available as open-source software under the MIT license.
You can find documentation and examples online.
We now present one of the two case studies described in the paper.
This case study focuses on facilitating the collaboration between ML engineers and clinicians for a task such as skin lesion diagnosis.
It illustrates how Marcelle can be used to create appropriate interfaces for different stakeholders and help them share data and models.
First, the machine learning engineer logs training information in data stores using Marcelle’s Python package for Tensorflow.
She creates a dedicated interface to monitor the training.
In particular, she creates a prediction pipeline to directly test her models with specific instances.
When drag'n dropping images, she gets instant predictions, and can also run batch predictions on selected data sets.
She can easily duplicate this pipeline to compare models side by side.
Finally, she can then share a particular model with the clinician using data stores, which provide real-time synchronization.
She creates a dedicated interface for the clinician by reusing her existing pipelines.
With this application, the clinician can test the classifier with new images, correct prediction, and save this additional data in a store to be shared with the developer.
To conclude, we have presented an architecture for toolkits dedicated to designing human-machine learning interactions, along with an implementation called Marcelle.
We provide two implemented case studies to support this work, with demos available online.
Our implementation still has limitations.
First, we plan to improve interoperability with Python and the main machine learning frameworks.
The goal is to facilitate Marcelle’s integration in the existing workflows of machine learning practitioners.
Second, we will keep extending Marcelle to include other data types, tasks and interaction techniques.
Our hope is that this tool could support research in interactive machine learning and human-AI interactions.
Marcelle is available as open source software and open to contributions.
We would be happy to answer any question, and we welcome you to join us for the demo.
Thank you