In this video, you'll see how you can get started with Amazon CloudWatch Evidently.
With this capability, you can reduce the risk of application disruption through controlled
launches and conduct experiments to validate new features before rolling them out for general use.
In this demonstration, we'll be adding a new module to the landing page of a fictional e-commerce
bookstore application to recommend books to users based on purchases their friends have made.
To get started, let's navigate to Amazon CloudWatch Evidently.
First, we'll need to create a project, which will be the container for the features we
want to launch or the experiment we want to conduct for the bookstore application.
We'll name the project “Bookstore.”
Next, we'll decide whether to have our evaluation events stored and, if so, where.
For this example, we’ll use an Amazon CloudWatch log group where we’re already storing Evidently metrics.
We can optionally assign tags.
Let's proceed and create the project.
Now we can add our feature to the Bookstore project.
We'll name the feature "Recommendations."
Next, we’ll define feature variations to expose different numbers of book recommendations
to subsets of users so we can compare the response.
First, we’ll need to select the variation type.
Boolean returns true or false depending on if the user is assigned to the variation or not.
Long and Double are numerical variation types, and String dynamically includes a string value.
Because we’re basing the experience of our application on the value of the variation, we’ll select Double.
Variation1, the default, will produce 5 book recommendations.
Variation2 will produce 10.
To set up the feature evaluation,
we need to allow Evidently's control and variation logic to hook into our application.
Sample code is provided in JavaScript, Java, and Python and can be leveraged in our application
to check the variation that’s assigned to the user at runtime by Evidently.
Here’s the code that evaluates the feature variation.
This code calls the EvaluateFeature method.
This is how the API response will appear in our application.
"Reason" states whether the feature was evaluated as a launch, experiment, or override.
We’ll be able to see which variation was assigned and the value associated with it.
We can use this response to code the application feature behavior for this variation.
We'll review the full application code later in this video.
We can also use overrides to specify that certain users always see a certain variation.
Overrides are useful for testing and beta releases.
Let's add an override now.
Now we can add this feature.
Let's quickly review the feature's evaluation rules.
Here we can see the overrides added for this feature.
This shows if the feature is part of any launch or experiment.
In our case, the feature is not yet part of either.
Now we’re ready to create a launch for the Recommendations feature.
Let’s return to the "Bookstore" project and create the launch.
We'll name the launch.
We can select an existing feature or add a new one.
We’ll select our "Recommendations" feature.
Next, we'll set up the launch configuration,
where we can choose to introduce the feature to customers immediately or incrementally.
We'll schedule the launch to incrementally introduce the feature to users.
We can select the amount or percentage of traffic to which we want to introduce this feature.
Let’s start with 10% of users seeing one variation, and 10% seeing the other.
We can add steps to this schedule and roll out the feature to a larger percentage of traffic over time.
Let’s raise the traffic percentages to 25% per variation the next day.
In a final step, we'll allow 50% of users to see each variation on the third day.
We have the option to add and monitor metrics that evaluate how our
Recommendations feature performs for customers.
We can define our own metrics that we will emit from our application or use standard
Real-User Monitoring (RUM) metrics if our application uses the CloudWatch RUM feature.
Let's set up a custom metric to measure how long it takes to load the "Recommendations" feature.
First, we’ll name the metric.
Next, we’ll add a rule that will be applied to data events emitted by our application to generate this metric.
We can copy and paste this code into our application at the appropriate place to emit a metrics event.
We can add up to three metrics.
Let's add a metric to measure the load time of our application's homepage.
We can also implement a CloudWatch alarm that will notify us if the metrics deviate from our expectations.
We won't use an alarm for this launch.
Now we're ready to create the launch.
When the launch is initiated, let's review the code we've prepared for our e-commerce bookstore application.
In our "Home" component, we use AWS Security Token Service (AWS STS) and Amazon Cognito
to return the credentials needed to call the Evidently API and initialize the Evidently event.
This is where we check to see whether or not a client exists.
Here, we create the Evidently client and define its endpoint,
region, and credentials so that we can pass it on to our "FriendsBought" component.
Here's where we’ll evaluate our feature flag for Evidently.
With this code, we've defined the request and called for the "Recommendations" feature.
We're using a randomly assigned ID for each user session.
Then we call the Evidently client prepared in the "Home" component and wait for a response.
We also save the feature state, variation name, and variation value.
Another set of code checks whether the "Recommendations" feature is being used for any
launches or experiments; if it's not, the "Recommendations" API won't be called.
Here, we see code for the Recommendations load time metric we established.
The post-event API we added from the sample code emits the metrics to Evidently.
Finally, we define the circumstances in which to render the feature, and then build and
deploy our application code into production.
Now that our feature launch has been active for some time, let's see how the application appears to various users.
Here we can see the module that recommends books purchased by our friends.
When we refresh the page, we'll see the feature variation for the 10 recommended books.
Let's return to the CloudWatch Management Console and review the launch's performance.
We'll select our launch and view the Monitoring tab.
The first graphic shows how many times each variation was assigned to the application's users.
The next two graphics display our custom metrics.
In this case, there’s little difference between Variation1 and Variation2 and the metrics are satisfactory.
If the feature had a bug or created a poor customer experience, we could reduce traffic
or even cancel the launch while we worked to improve the feature.
If we want to launch the better-performing feature variation earlier to all customers,
we can do so easily by updating the launch traffic to 100% for our chosen variation.
You've just seen how you can get started with Amazon CloudWatch Evidently, and launch a
feature with different variations incrementally to customers.
You can learn more about this topic in the description and links for this video.
Thanks for watching. Now it's your turn to try.