Hi, I am Haiyue Yuan, I am presenting the paper ‘CogTool+: Modeling human performance
at large scale’.
This work is jointly done with Shujun Li from University of Kent, UK and Patrice Rusconi
from University of Messina, Italy.
Researches have been conducted to study human performance of human users interact with the
computer-based systems, which involves studying human cognitive process, model user interfaces
and model human behaviours.
Cognitive models are often used to facilitate this process.
Among all, GOMS, KLM and their variants have been proved to be effective and useful.
Based on these cognitive models, psychologists and computer scientists work together to develop
a number of cognitive modeling tools and they have been widely used in both academy and
industry.
As you may already noticed, ‘CogTool+’ is in the title of the paper, therefore, we
want to briefly introduce CogTool to the audience who are less exposed to cognitive modelling.
CogTool is developed at CMU, and it is an open source cognitive modeling tool based
on KLM.
The usage of CogTool consists of model User Interface, model user interaction, translating
high-level description into a low level ACT-R based model, and human performance prediction.
Despite the usefulness and popularity, similar to other cognitive modelling tools, there
are some challenges.
Challenges are raised when comes to model systems with dynamic tasks: one task is to
model dynamic UI, and another task is to model dynamic user interaction.
In addition, another challenge ls is to conduct large scale modelling tasks, which normally
requires lots of manual work and it is time consuming.
We summarise the following as the major challenges as well as the motivations to scale and extend
CogTool’s capabilities: The first one is to conduct large-scale modeling
tasks (semi-)automatically.
For instance, consider an example of modeling the 6-digit PIN entry task, which will require
producing up to 1 million models to cover all possible PINs.
The second one is to dynamically update/change default values of cognitive modeling operators
and parameters such as those related to Fitts’ law.
The third challenge is to build mixed probabilistic models through simple steps.
For instance, the ability to model a system that involves left-hand/right-hand users and
mouse/touch based interaction with respective weights or probabilities all together.
Having all these challenges in mind, we developed a new cognitive modeling software framework,
CogTool+, it consists of 4 main components Model generator, model interpreter, model
simulator, and offline analyzer.
The Model generator generates simulation models and meta models.
The simulation model is used to define simulation settings and parameters.
One meta model consists of a descriptive model and an algorithmic model, where the descriptive
model is for high-level UI and user interaction description, and the algorithmic models are
‘plug-and-play’ components written in JavaScript or CSV to support parameterization
and automation.
The Model interpreter consists of a number of interpreter workers, which can translate
high-level descriptions to low-level UI descriptions and atomic level user interaction scripts,
and eventually outputs intermediate cognitive models.
Including a scheduler, converter, and a simulator, the Model simulator coverts intermediate cognitive
models to a series of ACT-R based models and produces the simulation results saved in CSV
format.
Finally, the Offline Analyzer takes the simulation results and post-processes these data, followed
by streaming the final output to an interactive visualization platform.
Moreover, we would like to address another important feature of this framework.
That is to allow the software to work with external data to guide and help modeling and
simulation.
The framework allows users to link external data generated by either an algorithmic model
or direct input.
I can give you one example here, human behavioral patterns extracted from eye-tracking data
could be programmed as reusable behavioral templates to run within CogTool+ to facilitate
cognitive modeling tasks.
Now, let’s take an example to see CogTool+ in action:
The task is to model 50 6-digit PIN entries.
Why ? PINs remain as one of the most widely used user-authentication methods.
Some cyber attacks rely on inter-keystroke timing dictionaries built from user studies.
We would like to demonstrate that CogTool+ is cost-effective and accurate for modeling
6-digit PIN entries at a relative large scale.
What are we modelling ? First step is to link 50 PINs saved in an external file.
Then for each PIN, we need to model the user interaction of pressing 6 digits followed
by pressing the ‘Enter’ key.
Next part is to determine if all PINs have been modelled, if not, the program will assign
another PIN to model.
Let’s look at how the simulation model is used for this modelling task.
A simulation model is written in XML, first task is to define the number of simulations,
which is 50.
Next, as illustrated here, we create a global ‘ArrayList’ variable with ID of ‘externalPin’
to link with an external CSV file.
Then we will take a look at how the meta model is used to model the UI and user interactions.
A descriptive model is also written in XML, it can link with the global variable defined
in the simulation model, and is also responsible for model the high level UI and user interactions.
Here is an representation of the modeling process.
Now we have done the most important part and labour intensive part to define simulation
and meta model.
Then the CogTool+ model interpreter, model simulator, and offline analyzer will take
care everything else.
Let’s take a look at the results.
In this evaluation experiment, we compared the human data with the simulated data produced
by CogTool+.
As illustrated in this figure, the correlation coefficients for the selected PINs are all
close to 1, and the mean and standard deviation of correlation coefficient for all PINs suggesting
a strong association between the human timing data and the simulated timing data for all
PINs Now we have shown the capability and scalability
of CogTool+ to some extent.
However it does not clearly demonstrate the use of algorithmic models to model dynamic
UI and user interaction in more depth.
We also did another evaluation work to model a complex observer-resistant password systems,
Undercover.
Here we just give you a quick glance to illustrate the capability of CogTool+.
We can develop different JavaScript functions to be compiled using the model interpreter
at the run time, to deal with the dynamic interactions involved in this system.
In addition, we converted some visual search patterns as illustrated in these two videos
into JavaScript based behaviour templates to facilitate the modeling tasks.
More details can be found in our paper.
To conclude, we would like to address that CogTool+ can allow the integration of external
data, supports parameterization and automation, supports algorithmic components and mixed
probabilistic model.
By having these features, CogTool+ is compatible with CogTool, and can produce valuable simulated
data with less effort, last but not the least, it provides an alternative approach for large-scale
modeling.
We would like to see this as one contribution to the CogTool research community.
Finally, we welcome you to visit our github repository to give it a try.
Thank you very much for your time.
We would like to hear your feedback, comments, and questions.