TONY LU: Hello, everyone.
Welcome to today's session.
My name is Tony, and I'm a product manager on BigQuery ML.
As most of you may already know, BigQuery ML
enables users to create machine learning models
and run inference against them within BigQuery,
all by using SQL queries.
Today, we're excited to show you two new features
for launching-- one, how you can now
import more types of models, train outside of BQ
into BigQuery ML for inference, and two,
how you can run inference with the remote models
hosted on Vertex Endpoints.
It's our pleasure to have Sarah from Semios
join us today in this customer case study.
Together, we'll explore how Semios
is using these two new features to improve in order temperature
predictions and simplify their existing workflow.
Now, to kick it off, I will Sarah
to give us a bit of background on herself and the company.
Over to you, Sarah.
SARAH OVERDUIN: Thanks, Tony.
Hello, I'm Sarah.
I'm a data scientist at Semios.
We are a company that offers precision agriculture
as a service, and our offering includes pest and disease
management, climate monitoring, water management,
as well as other services.
We have a network of IoT devices installed,
which includes weather stations, soil moisture
probes, pheromone dispensers, and camera trapped,
as well as other types of devices
that help us monitor the state of the orchard
and help growers make better decisions.
TONY LU: Now, the first feature we're going to look at
is our expanded support for imported models.
Previously, model import only worked with TensorFlow models,
but now, we're expanding this to support TensorFlow Lite,
XGBoost, and Onyx formats.
For example, it can convert models
from any common ML frameworks such as PyTorch, scikit-learn,
directly into Onyx and then import them into BigQuery ML.
This approach allows you to run predictions
on state of the art models developed elsewhere
directly within BigQuery without having to move your data.
From an inference inside BigQuery,
you also get better performance by leveraging
BigQuery's distributed travel query engine
for batch inference tasks.
Now, I'll hand it over to Sarah again
to show us how Semios is using this new feature
and share with us the results and learnings she
had from this experiment.
SARAH OVERDUIN: Thanks.
Just to set up the problem that we are dealing with,
so we have sensors that measure weather in the orchard,
and the temperature is a really important predictor
of crop, pest, and disease development,
and specifically, in-orchard temperature.
So that's why we have installed our devices everywhere.
However, we would like to be able to predict beyond where
we have our devices.
Sometimes, for whatever reason, we can't install the device,
or a grower wants to have a lower cost solution.
One thing that we've found out, though,
is that when we get third party weather data,
though, there is some discrepancy,
and this discrepancy is often in a particular direction.
For example, maybe at certain times of the day,
it's usually warmer than the temperatures that we measure,
or at other times of the day, it might
be cooler than the temperature that we measure.
And this poses a problem for us because the test development
and the crop development depend not only
on the temperature at a specific time,
but on the accumulated heat over time.
So if we have biases that continue for a longer
period of time, we end up predicting
when things are going to happen much more poorly.
So vegetation modifies the local temperature,
and it does it because of the evapotranspiration.
So as it takes water up, that energy
that would go into heating up the object instead
is dissipated by evaporation.
At the same time, different areas
have different local features, so some areas have a lot
of lush vegetation around them.
Other areas, we have a lot of bare areas, barren areas,
and how an orchard modifies the temperature
depends on these local features.
And these are often not captured in third party weather
services.
So the challenge for us is, how can we predict temperature
without sensor data?
And we would like to be able to account
for the influence of local land features and local orchard
features.
So in this example here, we have some parts
of the orchard that are near to very bare ground,
and other parts of the orchard which,
when it's in full season, will be very lush.
And there will be differences in how these parts of the orchard
will modify the temperature.
So what we want to do is we want to actually use
satellite imagery and extract features from them
and include them as part of a model to predict
the in-orchard temperature.
So we get third party weather data.
We combine that with our extracted features from images.
Create a machine learning model, and use in-orchard
temperature--
and predict in-orchard temperature.
So we currently have implemented just
using the third party weather data, as well
as some other local features that we have,
to predict the in-orchard temperature,
but using the imported model feature that is now available,
we can also use satellite imagery
and extract the features from those.
So this is the general workflow of how we upload a model,
create the model, and then make predictions.
So I'm just going to walk through a demo of how
we are using the new features in BigQuery.
So the first step with--
oops, I just need to reconnect here.
The first step with importing a model, of course,
we need a model, and we have a model that's
been trained with PyTorch.
And what we're going to do is we're going
to export it in Onyx format.
There were a few things that we needed to make
sure we included, which was--
because we're outputting a feature vector,
we need to make sure that we have specified the feature
vector when we export the model.
So we've done that, and then we need
to upload the model to GCS.
And once we have that, it's very simple to import the model.
We simply have to--
oh, sorry, I'm in the wrong one.
There we are.
We simply have to make a create or replace model statement here
and specify that we're importing an Onyx model.
So it's very similar to the previous TensorFlow
importing that was already supported.
And then once the model has been implemented, we can run it,
and so in this case, because we're working with images,
we have our images stored in object tables.
We are going to be predicting on images, which
we have to first decode the image,
and then we have to resize it to fit the model that we have.
And then we can get the predictions out,
and the predictions, in our case,
because we're getting a feature vector out,
we just have this list of numbers.
So in this case, we're expecting 768 of these values,
and we'll have that for each image in the data set
that we predict on.
Once we have those extracted features,
we want to combine them with our weather features
to train a model that will be able to predict
the in-orchard temperatures.
So for that, we are also going to use a BQ ML,
we're going to use the AutoML regressor in this example.
We've tried out, of course, different models,
and you can use any of the BQ ML models here.
So for that, it's very simple.
I should say that, in our case here,
we decided that we wanted to reduce the number of features,
so we decided to use principal component analysis
and extract some of the image features.
So we're running that model as well
to first extract those image features,
then combine them with our weather data,
and then train the model.
So that's what we have here.
So when that's finished, we have our model,
and now, we can use it to make predictions.
So I'll just go back to our presentation.
So in terms of the results, it turns out
that our image features seem to be fairly important.
So we have several different weather features in our model.
I think we have around, I think, close to 60 features
in this model, which includes both image and weather
features.
And we have-- actually, in the top 10,
we have two of the image-based features, which is interesting.
So we're excited about that, and we're
excited to see how this will help
us improve our in-orchard weather prediction.
TONY LU: That was excellent.
Thank you, Sarah.
Now, we'd like to move on and show you the second feature we
talked about earlier--
remote modeling inference.
With this new feature, you can now
grow inference with models hosted on Vertex AI Endpoints.
Again, Sarah's going to show us a demo of how
Semios is using this feature.
Over to you, Sarah.
SARAH OVERDUIN: Thanks.
So yeah, we're going to use the remote modeling feature.
So as you can see in this diagram,
you can import a model from Vertex AI
and then use it to make predictions.
So we have several models hosted right now on Vertex AI,
and using BQ for the inference will actually
simplify our pipelines quite significantly.
So I'm just going to walk through a little quick demo
here, and I'm just actually going
to show you, on the console here,
importing our remote model.
So again, it's very simple, very similar to importing an Onyx
model.
We just create our statement for creating the model,
and then we just have to specify the connection that we're
using, as well as the endpoint where the model is hosted.
The other thing that we need to specify
is the output that we're expecting from the model,
and once we have the model, we can make predictions with it.
And again, we just need to use ML predict because we're
predicting on images again.
We need to decode the images, so we
use this two Base64 for the image decoding,
and then we can get our predictions back.
So in this case, I'm just having--
this particular model here is only
predicting two classes and different and the same,
and I'm getting the confidences for each of them.
So that's how simple it is, and I'll just
return back to the slide.
So in terms of how that impacts us,
the big impact is that it simplifies our pipeline,
and it improves our maintainability.
So I think we're really excited to make
use of this in the future, and I think
it will help us iterate on models and help us run models
and keep our pipeline running quickly and easily.
TONY LU: That's really cool.
Thank you again to Sarah for walking us through these two
awesome demos.
So I do see these two new features
being helpful to Semios.
Just to recap what we learned today,
now you can, one, import more types of models
into BigQuery ML for inference, and two, run remote inference
with models hosted on Vertex AI Endpoints,
all within BigQuery ML.
Besides these two new features, we're
also adding support for inference
with unstructured data, such as images and text using
Vertex AI's state of the art pre-trained models.
We have the Vision API, natural language API,
and [INAUDIBLE] API.
We'll be sharing tutorials and user guides on this feature
as well.
For anyone who is interested in trying out these features,
please fill out the form at this link.
We look forward to working with you.