My name is Magdalena Glas and I'm happy to present the paper Train As You Fight: Evaluating Authentic
Cybersecurity Training in Cyber Ranges.This work was carried out by me and my coauthors,
Manfred Vielberth and Günther Pernul from the University of Regensburg in Germany.
Given the ever-growing number of cybercrimes, the demand of skilled
cybersecurity professionals outpaces supply. In 2022, only 52% of this demand could be filled.
On the other hand, 3.7 million positions in cybersecurity remained open.This
is what's called the cybersecurity workforce gap. The consequences of
this workforce gap are drastic. For example, unpatched systems and unrecognized attacks.
Cybersecurity training equipping people with the skills and knowledge they need
to work in cybersecurity positions is crucial to address this problem.
A novel approach for cybersecurity training are exercises in cyber ranges. Cyber ranges are
representations of parts of organization, systems or applications that can be used for cybersecurity
training and testing. Like someone can practice to shoot on a shooting range, a cyber range provides
a safe and legal environment for cybersecurity training. In a cyber range exercise, short CRX,
trainees can experience real-world cyberattacks to learn and practice hands-on cybersecurity skills.
As for any learning environment or a software product which a cyber range is, it is necessary
to evaluate if it fulfills its intended purpose.
However, when CRXs are proposed in the literature, evaluation is often neglected.
We think that evaluation is crucial for a CRX to reach its full potential.
For this reason, we address the research question how can CRXs be evaluated?
To do so, we follow a seven-step method. First, we conducted a literature review to analyze the state
of the art in CRX evaluation. As a next step, we developed the TARGET framework consisting of the
TARGET taxonomy for CRX evaluation criteria and the TARGET evaluation process. Finally, we applied
the framework to evaluate the CRX - Iceberg- we developed at our university. This shows
the applicability of the framework and should serve as a guideline for other CRX designers.
We use the set of papers to analyzethe state of the art of crx evaluation.
Out of the 57 papers, 28 did not include any kind of evaluation.The
others differed highly in what was evaluated. Some only concentrated on learning parts of
the training, so "what did trainees learn"? Others only evaluated technical aspects.This
shows that best practices in evaluation that are applicable to any kind of CRX do not really exist.
This indicates that there is a need for a framework that facilitates evaluation and, thus, enables
more CRX designers to evaluate their concepts. As I said, CRXs are very heterogeneous.
For example, what types of attacks are deployed or what target groups they address. We wanted
that our framework provides the flexibility to evaluate all these kinds of different CRXes with
different kind of foci. To this end, we aimed for a goal-based evaluation approach. This means CRX
designers can choose which aspects of their CRX they want to put the evaluation focus on.
To make this decision - which aspects should be targeted by the evaluation - as informed
as possible, we extracted evaluation goals, or how we call it criteria from the literature.
To this end, we used our set of 57 papers to extract overall 75 criteria.These
criteria were classified in eight dimensions.Three
dimensions in the domain learning namely learning impact, learning outcome and learning experience.
The others are Scenario, Management, Monitoring and Environment.
The taxonomy is part of the TARGET process
which advises to first select criteria, then define metrics to measure this criteria.
On this basis, the evaluation procedure can be defined and designed and the evaluation conducted.
To make the evaluation framework more tangible, we instantiated it for a CRX
we recently developed for students in two cybersecurity courses at our university.We
developed the CRX Iceberg to introduce the students to incident response.This
means how to detect and handle cybersecurity incidents.To this end, we simulated an industrial
system and an attacker who tries to interfere with the system by conducting an attack. Through this
simulation, the trainees learn about practical incident response. They solve a sequence of
tasks in which they learn to identify the attack, eliminate the attacker, and restore the network.
We evaluated Iceberg to assess if it should be a permanent part of the two cybersecurity courses.
Following the TARGET process, we first selected those criteria of the taxonomy
that were most relevant for the purpose of our evaluation. Overall, these were 18 criteria.
For each criterion, we defined one or more metrics with which it should be measured.
As an example, we selected the criterion "Defensive cybersecurity skills" in the
Learning Outcome dimension. This criterion was measured by assessing
the trainee's skills with a multiple choice questionnaire before and after the training.
The evaluation consisted of a functional assessment of the CRX, a dry run to test
the technical infrastructure, and finally, a user study.
This user study was conducted with 50 students out of the two courses the CRX was developed for.
The TARGET framework helped us to identify that Iceberg showed very good results when
it came to how and what training is learned. However, if we want to conduct the training
with a larger number of students, a higher degree of automation will be very beneficial.
Thank you very much for listening. I hope I could raise your interest in our research
and I'm happy if you check out our paper and look at the GitHub repos. In this GitHub repos,
you will find the code for Iceberg CRX and also the steps of the
literature review and further material regarding the evaluation of Iceberg.
Of course, meet us here at CHI or contact us via email. My email is magdalena.glas@ur.de