In this video, you’ll see how to perform A/B testing with Amazon Personalize
and Amazon CloudWatch Evidently.
With this capability, you can orchestrate A/B tests to compare different
recommendation engine solutions and make informed design decisions.
Before we get started, let’s briefly define A/B testing.
With A/B testing, traffic is split in two
so that each user is presented with one of two variations of a user experience.
The goal is to see which variation performs better by measuring user engagement.
If a winner can be declared, that
variation becomes the new default user experience in your application.
A/B testing with Amazon CloudWatch Evidently involves setting up the application features
and experiments, defining traffic splitting rules, monitoring the
variations through a running experiment, and then evaluating the results of the
experiment against a business metric, such as conversion rate or clickthrough rate.
This is our full stack E-commerce demo application.
Currently, the homepage displays the same list of products to all users using a
popularity recipe from Amazon Personalize.
We’ll run an A/B test to see if recommendations from an Amazon
Personalize recipe that personalizes recommendations for each user based
on their past behavior improves the click through rate for this portion of the homepage.
To get started, let’s inspect the resources for the app in the Amazon Personalize console.
A dataset group is a container for your application's datasets, recommenders,
filters for buisiness rules, and more.
A dataset group for the retail demo store has already been created.
Let’s take a look.
Recommenders are machine learning based models trained on your data that
allow you to generate recommendations for a specific use case.
Two recommenders already exist.
The first one recommends popular products and is currently used by
the retail demo store homepage.
The second one is trained to recommend products based on each
user’s past interactions with the site.
These two recommenders will be the variations in our A/B test.
Now let’s navigate to CloudWatch Evidently to set up our A/B experiment.
A project has already been created for the retail demo store, which contains
several features representing different elements of the user interface.
We’ll use this feature to test the two variations of the retail demo store homepage.
Let’s view the configuration details.
The variations have been configured using the recommenders we looked at earlier.
In this case we are indicating the Personalize recommender ARN for each variation within a
JSON snippet as a string value.
When the app evaluates the feature for each user, it will use this value to
call the appropriate recommender.
CloudWatch Evidently provides sample
code for setting up the feature evaluation in your application.
For this example, the required setup has already been completed in the app.
We can also review an example of the API response when an experiment is not active for the feature.
Now let’s proceed with creating our experiment.
We’ll give our experiment a name and select the feature for the homepage.
We’ll use the default audience settings to split the traffic equally between the feature variations.
Let’s create a metric to measure the user clickthrough rate.
Our goal is to see this metric increase.
Let’s add another metric to track the price of the products users click on.
Again, the goal is to see this increase.
We can optionally set up CloudWatch alarms for each of the metrics.
Let’s create our experiment.
The experiment has been created.
Let’s start it.
We can schedule the duration of our experiment.
Let’s use the default seven-day duration.
Our experiment is now active.
Let’s return to the demo application.
Let’s refresh the page to see how the user experience changes now
that an experiment is running.
The experiment is now actively running, and the current user has been assigned variation B,
which displays recommendations personalized to the user.
Notice how recommendations now match the shopper's persona.
If the user clicks on a recommended product, conversion events are
sent to CloudWatch Evidently for the experiment's metrics.
Let’s switch shoppers to see how the experiment variation changes for different users.
For this user, the experiment is using variation A and popular products are displayed.
Now let’s return to the CloudWatch console and look at the experiment results.
This view is currently filtered to show the results for the click metric.
We can see the events and clicks for each variation as they come through.
Let’s view the price metric.
We can see that the price of the product that the first shopper clicked on was $69.
As our experiment runs, this data will continue to populate.
Our experiment has now been running for some time.
Let’s look at the results that have been generated so far.
The results show that the variation that offers user personalization is better for generating clicks.
Let’s look at the price metric.
For this metric, the results are inconclusive.
While variation B generated more click events and therefore a higher total value,
variation A had a slightly higher average price.
This tells us that popular products are higher priced, but users actually
engage with lower-priced products.
Additional testing could be performed to see how many of the products users
buy out of the ones they click on.
You’ve just seen how to perform A/B testing with Amazon Personalize and
Amazon CloudWatch Evidently.
You can learn more about this topic in the description and links for this video.
Thanks for watching. Now it’s your turn to try.