In this video, you'll see how you can run an experiment with Amazon CloudWatch Evidently.
With this capability, you can conduct controlled A/B experiments on new features before rolling them out for
general use, capture user metrics to evaluate feature performance, and interpret results with confidence.
In this demonstration, we'll be testing two promotions for our e-commerce bookstore website
to see which would result in a higher revenue.
The first offer is for 20% off any book purchase.
The second is to get 25% off the purchase of a second book.
To get started, let's navigate to Amazon CloudWatch Evidently.
We'll be working with a project, called “Bookstore,” that we've already created.
From here, we can create an experiment that we can conduct on a feature or application behavior.
Let’s give it a name.
Next, we'll define the feature we want to experiment on.
Let's add a new feature and name it.
Now we're ready to specify the variations we'll experiment.
We'll choose the String type for presenting the promotional offers as a banner on our website.
Let's name our first variation "twentyOff."
For the value, we'll copy the URL link to the promotion image.
This value will be used in our application code to show to customers who land with this variation.
We'll name the second variation "BuyOneGetOne25."
Again, we'll copy the URL for the corresponding promotion image.
CloudWatch Evidently prepares sample code that we can use in our application to render
the corresponding application behavior for the respective variations.
Next, we'll choose what percentage of our audience we want to test the variations on.
By default, Amazon CloudWatch Evidently equally distributes traffic across the variations,
which is a best practice for running experiments.
We'll retain this distribution.
Now we'll define the metrics that Evidently will use to evaluate the performance of our variations.
We can define our own metrics that we will emit from our application or use standard
Real-User Monitoring (RUM) metrics if our application uses the CloudWatch RUM feature.
Our goal will be to have the revenue increase.
Next, we’ll tell Evidently where to extract the metric.
We’ll specify the entity ID, which is a unique identifier associated with the end user,
as well as the value key and units for the metric we’re monitoring.
In this case, we’re tracking revenue in dollars.
We can copy the resulting code snippet to use in our application.
We can optionally implement a CloudWatch alarm that notifies us if the metrics meet specific conditions.
Let's create an alarm that measures whether our revenue is less than or equal to $10 over
a one-hour period, as this condition would indicate that something has gone wrong with our experiment.
We can use an existing Amazon Simple Notification Service (Amazon SNS) topic to receive alarm
notifications or create a new one.
Now we're ready to create the experiment.
Now that the experiment has been created, let's review how we've incorporated it in our application's code.
In the Hero component of our website, we've called for the project that defines the feature we're evaluating.
Then we use the Evidently client to call the Evaluate Feature operation, which will return the reason
behind the evaluation, the variation that the user was assigned, and the value assigned to that variation.
Then, when the Hero component is rendered, it will check whether the feature we called
for is being used as part of a feature launch or experiment.
If it is, the feature will be applied so that users will see one of the two promotional banners.
Before we can start the experiment, our application will need to be deployed with the implemented Evidently code.
Now let’s start the experiment.
We'll use the default schedule, which is to run our experiment for seven days.
The experiment now has a "LIVE" status.
Let's visit our e-commerce bookstore website to see the experiment in action.
We’ll refresh the page to simulate a new user coming to the site.
We can see that we have been assigned the "buyOneGetOne25" variation.
Since users have a 50% chance of seeing either, let's refresh again to see if we can view the "twentyOff" variation.
Let's return to the CloudWatch Management Console to review what results Evidently has captured so far.
Here are our results after running the experiment for several days.
The graph charts how many events have been recorded for the custom metric we set up to measure revenue.
In the chart below, we can see how many times each variation has been shown to users and
the total value and average of revenue accumulated by each variation.
The chart indicates that the “Buy One Get One 25” promotion is the better performer,
with an average checkout value that’s $5 greater than the value for the default “Twenty Off” variation.
This is a confidence interval improvement of nearly 34%, with a statistical significance
of 99.9%, so we have a clear winner and know which promotion to pursue.
You've just seen how you can run an experiment with Amazon CloudWatch Evidently.
You can learn more about this topic in the description and links for this video.
Thanks for watching. Now it's your turn to try.