We introduce Marcelle, a toolkit for the design of human interactions with machine learning.
Marcelle's modular architecture allows developers to compose
custom workflows using interactive machine learning components.
As a toy example, we can easily build an image classifier from a webcam
and embed it in a musical application.
We illustrate Marcelle through two scenarios.
The first scenario involves a HCI researcher who prototypes a sketch recognition application to run a study.
The second scenario focuses on the collaboration between a machine learning expert and a clinician to build a skin lesion classifier.
In our first case study, we show how Marcelle can support rapid prototyping of interactive machine learning workflows.
Marcelle is a JavaScript library and only requires a simple script to define an app.
Components are instantiated using factory functions.
They provide views that can be composed into custom layouts such as dashboards.
In this example, we create a sketchpad, a dataset, and widgets to capture labeled sketches into the dataset.
While components are visualised, they do not yet communicate with each other.
Marcelle uses reactive programming for the design of pipelines
that link user interactions with machine learning processes.
In this example, we use the button’s stream of click events
to sample the sketchpad’s stream of images.
Combined with the label from the text field,
we obtain a stream of instances that can be fed to the dataset component.
With new JavaScript libraries such as Tensorflow.js,
it is possible to run training and inference in web browsers.
Marcelle provides models that can be trained from the user's data,
without need for a remote server.
Marcelle also provides visualizations that can be applied to components or streams,
for example to monitor the training, and to visualise predictions in real-time.
With this architecture, developers can rapidly experiment with various workflows.
Recomposing the app layout can be done by changing the components' order in the dashboard.
Pipelines can be modified for incremental learning:
predictions are updated at every stroke on the sketchpad,
and changes to the training set trigger the model’s re-training.
The second case study illustrates how Marcelle can support collaboration
between machine learning experts and domain experts such as clinicians.
Marcelle provides a Python package enabling machine learning developers
to record training logs and model checkpoints into a data store.
This data can then be accessed into a custom dashboard
to monitor and compare machine learning experiments.
Particular model checkpoints can be loaded in the web browser for direct interaction.
Custom pipelines and interfaces can be created to inspect a model's behavior
with particular instances, or with arbitrary datasets.
The creation of components and pipelines can be abstracted into functions.
Pipelines can therefore be reused, for instance to create side-by-side model comparisons.
Marcelle data stores can host data in browsers' local storage or on remote servers.
They provide real-time data synchronization, allowing machine learning experts
to instantly share their progress with clinicians.
Dashboards can be tailored for clinicians, allowing them to interact with a trained model
and to correct misclassifications.
By synchronising datasets, model developers can easily gather new annotated data from clinicians.