Hi, I'm Polita Paulus,
a software engineer at Snowflake.
Today, I want to show you a new type of Snowflake table, called an Iceberg Table,
one that pairs Snowflake's powerful performance engine and platform capabilities,
with open formats and storage managed by customers.
Let's check it out.
For this demo, I will play the role of a data engineer supporting the sales organization
at Bank Co., a financial services company already using Snowflake. Bank Co. recently acquired Finance One,
which has data in Parquet format in Amazon S3. Our sales team wants to know whether any of our
top Bank Co. accounts are also doing business with Finance One. This will be a great list to pursue
for joint products we have under development, but our company now has data in two places.
While Bank Co. wants to work with Finance One data, just as they would with any other Snowflake table,
we do not want to disrupt teams and other tools that are working with Finance One's Parquet data directly.
With Iceberg Tables and Snowflake, we can achieve both goals.
Iceberg Tables extend the features of Snowflake's platform
to Finance One's Parquet data, providing benefits such as governance
alongside the interoperability that an open table format provides. To create an Iceberg
Table in Snowflake, first I need to create an external volume, which is also a new concept.
External volumes hold file data and table metadata, all in customer-supplied storage.
Meaning there's no data movement and no storage billed by Snowflake.
During the due diligence process, before the acquisition closed,
Bank Co. had read-only access to the Finance One data, using a Snowflake external table to query it.
From this external table, I can easily create an Iceberg Table using "Create Iceberg Table" as select.
Notice that I used the qualifier Iceberg to create the table. Whether I create an Iceberg
Table from scratch or from a select statement, the Iceberg qualifier means that Snowflake
will write out the table's data in Parquet, along with Iceberg metadata
directly in Finance One's supplied S3 storage.
Now that I have Finance One's data in an Iceberg table, I can work with it
as I would any other Snowflake table. Let's query the table to make sure it has the data that I need.
This is using Snowflakes' query engine, even though it's querying Parquet files stored in Finance One's
S3 bucket. As you can see, this query returned results quickly, much like a standard Snowflake table.
Next, I need to mask personally identifiable information before I can share this data with my
sales team. As with any other Snowflake table, I can apply a native masking policy to the Iceberg Table.
Now let's answer the original question. Let's union the two tables, selecting customers who
live in Florida and Georgia, where we are focusing our product development.
I can quickly combine data from another Snowflake table with data from the Iceberg Table to generate
our top accounts list. I can now share these results with individual salespeople to begin outreach.
So some time has passed, and after our sales team began their outreach, we received a
request to delete data for the customer named April Smith. This would normally be hard because
it would require action in two systems, plus most systems that work with open table formats
do not support multi-table transactions. However, with Iceberg Tables and Snowflake, this is now
a single transaction. What's great about this is that one transaction updates multiple data sets
at once. What's even cooler is that the deletion request is applied to the data stored in S3.
As mentioned earlier, we want to make sure that teams could work with this data together in a
performant secure way, without disrupting any existing work underway at Finance One.
This means the Parquet data and Iceberg metadata need to be stored in Finance One's S3 storage.
We can tell Snowflake to create an Iceberg snapshot of the changes we've made,
which will be persisted to our external volume for other tools to use.
Let's take a look to see if this happened.
Here are the contents of the directory I used for the external volume.
You'll see Parquet and Iceberg metadata generated by Snowflake.
This is the same metadata Snowflake is using, and it's stored completely in Finance One's S3 bucket.
This matters because Finance One can continue querying this data using other tools, such as Trino
or Apache Spark, without disruption.
In this demo, you saw how the new Iceberg Tables enabled Bank Co.
to work with Finance One's data alongside other Snowflake tables. I was able to query the data,
apply a masking policy, and run a multi-table transaction, with no disruption to existing
tools using the data in S3. Iceberg Tables pairs Snowflake's ease-of-use, performance
and consistent governance to data in open formats ... in external storage.
Thanks for watching this demo of Iceberg Tables..
What are you excited to do with this new table type? Let us know in the comments.
And if you want to learn more, check out our blog, linked in the description.