Hi I’m Jared Hillam, If you’ve ever received a package in the
mail then you’re probably familiar with shipment tracking. We track our shipments
because we want to know where our stuff is and where it came from. That way if something
goes wrong (like say your package comes tangled in some bagpipes) you can trace back to the
possible source of the problem. Well the same is true for data. Every day organizations
move millions of records into different systems, data warehouses, reporting systems, staging
areas etc. And organizations need to know where this data is going. For example, lets
imagine that you’re looking at a report and the graph is showing a new division that
you know doesn’t exist. So you ask yourself, where did this data come from? For many organizations
this question can be a mess to answer. This is because the data movement process can be
a black box to business analysts. So this can kick off an expensive analysis effort
to document all the hops the data went through to land in that report. Now, if this need
only happened say once a year, maybe organizations could live with that analysis effort. But
in a data rich environment with lots of data consumers these question is going to be asked
regularly. And its not just the business users that will be asking these questions, but IT
team as well. This is because IT teams need to understand the impact of the changes they
make to the source systems on the downstream reports. A change to a single table could
impact hundreds of reports that eventually use that table.
And this is where I introduce you to something called Metadata Management. Pause for a second:
Notice I said METADATA Management and not MASTER DATA Management. It just so turns out
that within all these hops, each one of these systems keeps a log of the data that’s being
handled. These logs are metadata, or information about the data. By tying together the different
metadata sources you can begin to get a picture of where the data is moving and how it is
changing. Now if you had to tie this all together by hand you would have a massive hairball
of complexity to piece together. But luckily a number of the leading vendors in data integration
have prebuilt some of this for you. They’ve done this by packaging connectors to the metadata
which is stored in industry leading ETL tools, BI tools, databases, data modeling tools,
and other systems. So as long as you have a fairly industry standard tool in those categories,
there’s a good chance you’ll be able to draw metadata out of those tools without a
massive effort. If you have an obscure ETL tool or BI tool then you’re out of luck.
You’re left with the options of either hand integrating the metadata into a metadata management
tool or living without visibility into that step, and sometimes you can live without that
visibility. However, there is one source of metadata that you can’t or shouldn’t live
without, and that is your ETL tool. For a refresher on what an ETL tool does check out
my ETL video. The entire data movement process is driven by your ETL tool so you’ve got
to ensure that you have adequate visibility in that category.
The end result of an implemented Metadata Management tool is a full data tracing map.
Which gives you insight coming from either angle. You can look at it from a developers
perspective and ask which reports use this table I’m about to edit? Or you can look
at it from a business users perspective and ask how did this data end up in my report?
We call these Impact Analysis and Data Linage. Now in addition to Impact Analysis and Data
Lineage, Metadata Management tools offer another huge value add, which is called a Business
Glossary. Have you ever been in a report and seen a column title like “Shipment Date”
and wondered what that means? That term could mean a number of different things, wouldn’t
it be nice if there was a central location for storing business descriptions for such
data? That is exactly what Business Glossary intended to do. The leading Metadata Management
vendors provide this Business Glossary as an interface to enable business users to create
descriptions for the data. These descriptions can be collaborated on and run through an
approval process. Additionally many of these vendors provide the ability to combine terms
into calculations which are linked to each other. This is another area where it’s important
to understand that there is no pixie dust fiery that is going to provide these business
descriptions. It requires an engaged Data Governance organization to ensure those tasks
are taken care of by the business user community. You can get more information on the basics
of a Data Governance team by clicking on this video.
Intricity has an entire strategic offering around establishing a Metadata Management
practice. This includes engaging with a Data Governance team and helping the organization
build connectors to non-standard hops of metadata. I recommend you reach out to Intricity and
talk with one of our specialists. We can help you untangle that data movement mess into
a self-documenting map which business and IT can collaborate in.