- Hi. My name is Thomas Rindfuss.
I'm a Senior Solutions Architect with Amazon Lex.
Today I'll introduce a new feature
to you called Test workbench.
With Test workbench, you can create
run and analyze test sets against your Lex bots
making it easy to perform unit testing
and regression testing.
You can also use it for test-driven development
by defining your tests ahead of time
allowing your bot developers to validate the bot
as it's being built.
Test workbench will let you identify improvement areas,
highlighting intense slots
and utterances that need attention.
You can review performance metrics for transcription,
intent recognition and slot resolution
using test cases that can be generated
from your conversation logs or created manually.
This new feature saves you time
and effort by providing a no-code end-to-end approach
to generating, executing and analyzing test sets.
You can use the Gooey provided in the console
or implement the provided APIs to add test sets
to your CICD pipeline.
This feature will let you standardize your testing.
You'll be able to follow a consistent out-of-the-box
testing methodology for your bot roll outs
without the need for custom scripting, API integrations
and infrastructure to set up and manage.
Let me start by showing you how you can create a test set.
On the left hand side of the Amazon Lex console,
you'll now find a section for Test workbench.
Select Test sets, and create a new test set.
While I will create my test set using a CSV file,
I also first want to show you
that you can build a test set
based on the conversation logs from your bot.
Here I create a test set named Retail Convlog Test.
I select generate a baseline test
to generate a test set from my conversation logs.
Then I select the bot name, the alias and language
and specify a time range for the logs that I want to use.
I select an IAM role with the required permissions,
the modality voice or text,
and the location in S3 where I wanna store this test set.
Now let's get back to creating a test set using a CSV file.
I'll name this test set Demo Test Set.
I have a CSV file that I have based on the CSV template
that you can find on the create test set page.
This file specifies a number of conversations
that each have multiple turns.
We specify user or agent, the bot,
for each turn of the conversation,
what the input should be,
the expected input and slot values.
You can also specify things like input context,
request attributes, session attributes,
runtime hints, and error messages.
You also have corresponding fields
to define the expected outputs for each of these fields.
Let's continue creating this test set.
I've selected to upload my CSV file
and I've pointed it to my file.
I've also selected an IAM role that has permissions
to make runtime calls to Lex and store the results in S3.
You can find instructions
for how to create an appropriate IAM role
in the documentation for Test workbench.
I'm selecting text in this case.
You can also select audio.
Modality here indicates what type of data we'll be using
to create the test set.
I've selected an S3 bucket as the location
to store my test sets and select create.
If the test sets CSV had been incomplete
and lacking pertinent fields,
the test set generation process
would've thrown a validation error.
You would've then been able to download a file
that shows error logs against specific test set inputs.
Now that my test set has been created,
I can select execute to run it.
The test set can be run against any bot.
So we now choose the bot, alias,
and language to use for our test.
We also select if we're using text or audio,
and if we want to test the streaming or non-streaming APIs.
Generally, you'd use streaming for voice applications
and non-streaming for chat.
You can select the validate discrepancy button
to see if there are any discrepancies
between the test set and the selected bot.
You can now see if the test is referencing any intents
that are not present in the bot you've selected to test,
or if there are slots that don't exist
in the specified intents in the bot.
I'll now select execute to run the test set.
When the test set has completed,
we can select the test result ID to view the results.
Since this test only contained conversations,
and no single input tests,
let's first review the conversation results.
Here we can review which intents failed
and what percentage of failures they had.
We can see that cancel order failed one out of three times
and that track shipment failed one out of one times.
If we scroll down, we can see the failure metrics for slots.
Here we can see that the only slot
that failed is order number and that it fails every time.
At the top, we can see a table
showing the conversation pass rates.
Here we can see a line for each of the seven conversations
we had in our CSV file.
We can see if the conversation passed or failed overall,
and if it failed,
we can see if it was due to an intent not matching
or if one or more slots failed.
Under intent and slot results,
we get a list of the intents used,
the types of tests that they were part of,
single input or conversation, and the pass rate.
We can also see details for the slots
showing in which intents they were tested
and the success rate for each.
You have the option of providing audio files
to test the transcription accuracy.
This can be done automatically
if you create a test case from conversation logs.
With these audio files, you'll also get data
on how well the transcribed speech
matches your expected results.
Under detailed results,
you get a view similar to the CSV file
where you can review the test line by line
and review the expected output versus the actual result.
Here you also have an option
to download the detailed results as a CSV file.
Finally, let's have a look at the overall test results.
Here I've opened up a different test case
that has both single input and conversations.
You can see that we get a breakdown
of how many tests were of each type.
Next, we get a breakdown for the single input tests.
We can see how many tests failed overall
and how well the speech transcription performed
for those tests.
Below that, we get the same breakdown,
but this time for our conversation tests.
While the results for end-to-end
and speech transcription happen to be the same in this case,
they're not linked.
It's for example, possible that you could have test cases
that fail the end-to-end tests
even if the transcription was fully accurate.
Thank you for spending a few minutes
learning about this new feature.
Test workbench is available today
and it doesn't cost anything extra to use
other than the Amazon Lex resources you make use of
when running your tests.