In this video, you'll see how you can analyze experiment results with Amazon CloudWatch Evidently.
With this capability, you can monitor experiment
metrics in real-time and interpret results without advanced statistical knowledge
so that you can confidently launch the highest-performing application features.
For the purposes of this demonstration, we have already conducted an experiment using
Amazon CloudWatch Evidently in a previous video, which is linked in the description below.
The goal of this experiment was to measure
which of two promotions on our e-commerce sites drives higher revenue.
The 20% Off promotion is our default variation and Buy One get One 25% is the challenger.
We have a number of options for filtering the performance data.
First, we can choose which metrics to view results for.
In our case we only have one metric.
We can specify and filter our view based on specific time ranges.
CloudWatch Evidently also makes it easy to interpret experiment results using two statistical frameworks:
statistical significance, also known as the frequentist approach, and Bayesian inference.
We can also choose whether to view the improvements as a percentage, or absolute value.
We'll review our statistical significance results first.
Normally, peeking at an experiment's results before it finishes puts the experiment at risk for bias.
With Evidently, anytime valid p-values and confidence intervals account for this potential bias,
allowing the experiment owner to safely draw conclusions and end the experiment.
Let's refresh the output.
We can choose to represent the graph in four ways: events count over time,
average value over time, statistical significance, and improvement.
Let's view the plot of improvement over time, which has a 95% confidence interval (CI).
Below the graph, we can compare the results for each of the two variations we tested in our experiment.
We see the total event count for each of the variations, followed by the total value.
In this case, the total value is the revenue from all checkout transactions.
That column is followed by an average amount spent for each checkout event.
Next is the improvement of the variation in comparison to the default.
Next, the statistical significance describes how certain Evidently is
that the variation has an effect on the metric as compared to the default.
Based on these performance metrics, Evidently produces an overall result.
Since the goal of the experiment was to measure whether which promotion brings in more revenue,
we have a clear winner.
The Buy One Get One 25% promotion brought in nearly twice as much revenue
as the default promotion, and was determined to be better.
Let's review the Bayesian inference results.
Available only when the experiment is completed,
the Bayesian inference results show the probability of differences between the variations.
Here, we see there's a 100% probability that the Buy One Get One 25% promotion is going to yield an
increase in in total value when compared to the default variation.
Conversely, there is a 0% probability this metric will decrease.
CloudWatch Evidently has therefore determined that the 20% Off variation represents a worse result.
When we are ready to launch, we can do so right from here.
We'll name the launch for the higher performing promotion.
Next, we'll specify the feature to be launched, which is the promo banner.
We can use Evidently's granular scheduling capability to dial up traffic gradually.
In this case, however, we have such strong confidence in our Buy One Get One 25% variation
that we'll allocate 100% of our traffic to it.
We can continue to monitor key business and performance metrics
alarms, and tags for the new feature during the launch.
Let's create the launch.
Our launch has started, and the promotion is now visible to all site users.
Evidently will continue evaluating our feature until we choose to stop this experiment.
You've just seen how you can analyze experiment results with Amazon CloudWatch Evidently.
You can learn more about this topic in the description and links for this video.
Thanks for watching. Now it's your turn to try.