SPEAKER: Welcome to this episode of "Click to Deploy,"
a series where we show you how easy it
is to deploy a solution from the Google Cloud Marketplace.
In this episode, we'll showcase how
to set up an Elasticsearch and data flow integration
via the Cloud Marketplace.
Elastic Cloud now has native integration
with Google Cloud Dataflow, which
provides customers with easy and fast data ingestion
into Elastic Cloud.
Now data from BigQuery, Pub/Sub, and Google Cloud Storage
can be directly adjusted to Elastic via Google Dataflow.
Let's start by searching for Elastic in the marketplace.
We'll choose the solution labeled Elastic Cloud.
On the Solution page, make sure to read
through the overview, pricing, and support to learn more
about the solution.
Pricing for the solution is billed monthly or annually,
based on your subscription plan.
You'll notice the pricing calculator,
which will allow you to model your estimated monthly cost.
Start by selecting a plan.
We'll keep the default plan.
Next, choose the correct billing account.
Then, make sure to review the terms of the contract.
If you approve of them, agree to the additional terms,
which are about billing and complying
with the license agreement.
Finally, click the Subscribe button to launch the solution.
Now let's move on to the next step, which is managing
your subscription with Elastic.
You'll need to register with an Elastic account
to manage the solution.
Sign up or log in to your Elastic account.
On Elastic Cloud, you can create your own deployment.
A deployment includes Elasticsearch, Kibana,
and other Elastic Stack features,
allowing you to store, search, and analyze your data.
We'll name our deployment Test Deploy
and click the Create Deployment button
to launch the deployment.
Make sure to save your deployment credentials.
And when your deployment is ready, click Continue.
For the purposes of this tutorial,
we'll work to ingest data from BigQuery to Elasticsearch.
But please note that you can also
ingest data from Google Cloud Storage and Pub/Sub.
We'll go back to Google Cloud Marketplace and type
in Stack Overflow.
We'll start by ingesting the popular Stack Overflow data
set.
We'll use a table called Stack Overflow posts
under the BigQuery data set, Stack Overflow.
It has several structured fields as columns,
like post body, title, comment count, et cetera,
which we'll bring into Elasticsearch
to perform free text search and aggregation.
Next step is to create a Dataflow job from a template.
Visit Dataflow and then go to Jobs.
Choose Create A Job from the template.
Choose name, stackoverflowload1, and keep the default endpoint.
Choose the Dataflow template, BigQuery to Elasticsearch.
The BigQuery to Elasticsearch template
is one of Google's provided templates.
Now fill in the required parameters,
including your Cloud ID and base64-encoded API key.
[ELECTRONIC MUSIC]
For the Elasticsearch index field,
pick an index name where your data will be loaded.
We'll use the Stack Posts index.
Choose the table and BigQuery to read from.
We'll use BigQuery Public Data Stack Overflow Stack
Overflow posts.
Finally, set your max workers to 1.
Click Run Jobs to start the batch processing.
[ELECTRONIC MUSIC]
When you visit Kibana, within a few minutes,
you can see the data flowing into your Elasticsearch index.
And there you have it, BigQuery data
ingested into Elasticsearch with the click of a button.
If you enjoyed this video, please subscribe and hit
the Like button below.
We encourage you to check out the GCP Marketplace
and deploy the solution on your own.
Thanks for watching.
[ELECTRONIC MUSIC]