- Perfect, so I'm gonna start with a short presentation
just give it some context
and then we'll go into a live Demo.
So, today I'm talking about Data Quality
and why it's especially important
in the medallion architecture.
So just a little bit of background,
so I'm currently working for Flash food
and what we do is we're a marketplace
that connects surplus food to buyers.
So we're offering food at a discount price
and we're keeping it out of the landfill at the same time.
So really it kinda has the two pieces to it.
Demo or me work Flash food now,
previously worked in IOT in manufacturing,
both with the Data bricks platform
and been using Delta for some time now.
So we've probably already all seen this graph
and it looks really nice and simple,
and it's just bronze, silver, gold,
and then you suddenly have it in where you want it.
But in the real world it gets a bit more messy
and a bit more complicated,
because usually the sources aren't just one-to-one.
(Speakers chattering in the background)
So in, in practice it's gonna look a bit more like this.
So you're gonna have all your different sources,
your Data lake, databases, your streams
or your external APIs.
You're gonna bring them in raw,
just to, just somewhere that almost a landing zone.
Maybe apply some schema to it in bronze
just so you know what you're working with.
But then you're gonna kinda bring it into silver,
and these could be more Ad hoc analytics tables
but they're gonna serve more of a particular use case.
So you're bringing things together,
you're taking out the fields you no longer need
and you're really making it presentable
to an analyst or a data scientist.
And then the final one's gonna be your gold.
So that's gonna be maybe your reporting tables.
So nice wide ones for your Tableau or your Power BI
or you're gonna have like a feature store
that's gonna run into some ML model on the end.
So, that looks all nice,
but the problem that comes out of that
is now you have all these connections
from upstream sources downstream
and you're trying to figure out, okay,
somebody has said there's an issue here,
but how do you actually find out where that issue happened?
And how can you be comfortable telling everybody that,
oh here's the issue, let me fix that right away
or hey, there's no issue,
it's supposed to look like that.
So that's kinda why I looked
into an automated testing framework
that you can at least have a sense on these tables
and just really good idea of if there's any problems
behind the scenes.
So some motivators for this is gonna be
know of the issue before the end user,
be able to debug where the source of the issue is happening.
Don't regress as modifications are made,
so maybe somebody goes in and switches
it to an inner join and suddenly half your data's gone
or they they're dropping nodes
but they did do a subset on which node they're dropping.
So suddenly it's, it's not as expected.
And you're really trying to find those silent
errors that you're not going to catch
in your typical code.
So a bit more of a Demo into how we been doing this
is we're gonna go through Data bricks connect
and this is gonna give us the PI test Environment
just as you would unit testing.
And the nice thing about this
as you're gonna have the ability
to run these varied from your ID,
and you're gonna be able to connect
to your Prod Environments.
So if we go into one of these silver
and we can look at temperature test
so the nice thing is we can actually
go in and talk to our Prod Data.
So in a more production, Non-Demo Environment
you might have like a read only connector,
read only cluster so that way you could give QA
or somebody else access
to write just even SQL queries, uh, expectations
on what they want that table to look like.
So in this sense, we're going to have a different file.
That's going to kind of manage the quality of each table.
So if we look in, in here, there's going to have a file
for each of our, the Delta tables that exist
and these are going to be responsible for, to keep in track.
So like a couple examples of that previously done
just to keep it in context is for instance, we were
we're bringing over a change feed from Mongo DB database
and you want to make sure the IDs match
on your Mongo and your new Delta
and you're not missing records or expiring bad records.
So you can actually in those test files, carry your Mongo
carry your Delta and do an accept all.
So then the nice thing too about using PI tests
is you get kind of the features out of it.
So if we go into our data Lake, we have the setup so same
as our test structure, and it's going to be the same
as our data structure and our Delta Lake.
So if we go in here, we can actually see our bronze tables.
So we have flights, weather, and then our different tables
underneath their flights (murmurs),
to the same thing for the tasks.
So, so what this is going
to do is in this conf test file here
it's going to give it some logic to add,
add any HTML output of the test that's run.
And it's going to tell it to copy it into an object store
just so it's accessible to everybody.
So the result of this is you end up having these tests
and it's going to mock the, the structure
of your test file, which should mock the, the Delta Lake.
So if we go into one of these
it's actually going to output a task each time it runs.
For the most recent one we can actually go see, Hey, how is
how is the quality of that data?
And, Oh, in a second
I'll show you how it's actually scheduled
and how you would be alerted on if these tests failed.
So here, we're going to see the result of that test running.
So here, we're going to say, we hide this
it's running two tests
it's checking his data recent and his data and range.
So if we open up this failed test
we can see the test that was running
and then some standard output that just kind of printed
out to the plan line there.
So here, it's going to say, Hey
the most recent 10 records are this,
so pretty old and these are actually
off the Data bricks damp sample data sets.
So they're not up-to-date
and my test is gonna fail because of that.
So it's going to say, it's checking records
as of, as of yesterday
and that's going to fail it because they don't exist.
And, and other things you can really extend these.
So it's going to be business specific logic.
So for instance, one time I had a
a gaps in Island solution.
So trying to find the ones
in the zero is of, of different, different dimensions.
So you're checking.
Okay. Are they alternating each time?
Cause you're, you're updating incrementally.
Is there, is there anything weird
in the data or does it match a business value?
So, so the final thing then is I want
it to be able to schedule these PI tests to run just
as I would my, my normal jobs.
So using this function here, I'm able to pass
in a table path and it's going to call a pitest.main
and it's going to run the
the test path that's packed in, passed in.
And then after this
we're going to package this up as a wheel
and then copy some things to the Data bricks Portal.
So here....
So here we can see just this project
packaged up as a wheel.
And then also just the test file brought test Biles.
So in here we can go in and see that actually
can call these from something like an Azure Data Factory
or really whatever orchestration tool you have
and you can click the bug
or I have them scheduled for instance, every three hours
and my actual more Production Environments
but you can see exactly, okay, gold.
The predict delays was wrong, but it's actually
getting caused by the silver weather temperatures
that doesn't have recent data, and we can look
at it and see exactly how these are going to be called.
So we have our main class, that's going
to call into our wheel given this job
and it's going to pass in this parameter.
So it's going to pass that test.
You could also extend it to just call them all in one test
run in one job, just loop
through the paths and then call them.
But I found this case gave a nice way
to see exactly where the failures were occurring.
Yeah. And that she covers it there.
- Awesome. So do we have any questions
also speakers are allowed to ask questions to each other.
(lady laughs)
All right cool
I don't see any in the zoom chats,
just checking the platform chat.
All right.
Looks like we're all set.
Well, I linked to all of the speakers contact details.
So feel free to reach out to them on LinkedIn.
And thank you so much for joining the community sessions.
The lightning talks.
Thank you to all of our speakers.
Thank you, Carter,
thank you Merryn, Michael and great to have you all.
And I think we're just about ready to
start the last keynote here for Summit.
So hope you all are enjoying.
I've enjoyed so far.
Sorry. Did you want to say Michael?
- I just want to say thanks everyone
for watching and enjoy the last keynote.
And I hope you had a good time at, at the summit as well.
- Awesome. Well with that,
I'll let everyone go and take care.
Thanks so much.
- [Carter] Thank you.
Bye
- [Micheal] Bye. Bye.
(Micheal sniffs)