Welcome to the presentation 2 0:00:05.180,0:00:10.470 3 Towards Guidelines for Designing Human-in-the-Loop 4 Machine training interfaces, 5 This paper is concerned with interactive machine learning.
A field about which a 2014 paper by Amershi 7 0:00:20.650,0:00:24.160 8 the field s state of the art, 9 There are two main challenges mentioned here that motivated this pilot study.
The first is a call for more Principles and 11 0:00:32.829,0:00:36.670 12 The second is about Development of Techniques 13 and Standards for Appropriately Evaluating Interactive Machine-Learning Systems.
IML offers up a unique set of new problems 15 0:00:43.620,0:00:49.010 16 a system --> they can give a more detailed learning 17 input --> which will teach a system faster, However, to provide that more complex input
requires more from our human in the loop, 19 0:00:59.590,0:01:00.710 20 0:01:00.710,0:01:05.340 21 right balance and instead measure the combine 22 effectiveness of user and system working together on a training task.
We investigated this relationship by constructing 24 0:01:14.090,0:01:17.970 25 those interactions are used to construct a 26 linear function to denote interaction quality, Although this is not empirical evidence, it
is meant to provide an example of how we propose 28 0:01:29.370,0:01:30.370 29 0:01:30.370,0:01:33.780 30 front of furniture selection system --> with 31 the goal of getting a recommender system to recommend them a specific item; being a red
chair.
Users go through four different interfaces, 34 0:01:44.020,0:01:48.420 35 their effort using the task load index --> as 36 well as having them report interactivity and engagement on the same scale.
We also measure the efficiency with which 38 0:01:56.130,0:02:00.141 39 of inputs they gave as well as how long it 40 took them, 41 The first interface is the simplist interface --> with the user only having the option to like
or dislike an item, based on how similar it 43 0:02:09.830,0:02:15.720 44 Users can enable and disable different features 45 on the side of the screen and then like or dislike, telling it what to take into account
or not.
Interface 3 is a comparative interface, which 48 0:02:25.230,0:02:29.050 49 basically what they think to be the nearest 50 neighbour, 51 Interface 4 is a revision of the first interface, After liking or disliking the system asks
them on the basis of what feature their decisions 53 0:02:42.319,0:02:46.680 54 qualitative feedback showcasing avenues for 55 potential guidelines, The first of these is that room for richer
expression is likely to deliver a better experience, 57 0:02:55.730,0:02:58.530 58 Secondarily --> participants mention that recalling 59 information can be bothersome --> because they are not always aware of their own prior reasoning.
Rather they would prefer to supply information 61 0:03:06.660,0:03:11.650 62 mistakes and ruining their algorithm --> that 63 is why it is essential we provide a feeling that they can make no mistakes in returning
inputs. 65 0:03:19.990,0:03:25.290 66 an increase in interaction time is not a necessity 67 for an increase in quality of data, 68 We show how one can go about constructing a function to denote comparative interaction
quality based on experiment measurements. 70 0:03:42.110,0:03:46.360 71 with the efficiency --> we linearly relate those 72 factors to Q --> a representation of interaction quality.
It should be noted that the model provided 74 0:03:55.390,0:04:00.850 75 distinguishing interaction quality for a system, 76 We would like to inspire others to investigate the exact relationship between the function
parameters and the inclusion of more or different 78 0:04:11.810,0:04:17.979 79 Also some participants had a prior understanding 80 of ML --> which could have shaped their behavior, 81 In summary --> what we find is that interaction quality is crucial for training with a human
in the loop, especially long term. 83 0:04:31.060,0:04:36.690 84 of the interaction between system and trainer --> 85 and we propose a method of going about doing so, being an interaction quality model. Also,
the qualitative results gathered from participants 87 0:04:47.639,0:04:56.579 88 guidelines for designing ML,