Hi I’m Jared Hillam, There are a lot of parts and components that
go into accurately gathering data.
However, at the heart of any well-crafted solution is the data integration and query
logic.
This is the logic that tells the database what data is being requested and how to process
it.
Where that logic exists turns out to be a very important topic when all is said and
done as you’ll find in this video.
To illustrate this let me share with you an example.
Many years ago I worked for a software company that sought out to fix a common problem found
in Operational Reporting.
We developed a product that allowed you to open 1000s of operational reports and edit
all of them at once.
Why was this even necessary?
Well, operational reporting generally connects directly to operational databases or application
data, so that it has the most up to date information.
However, operational reports allow you to nest the data gathering logic as part of each
report template or file.
So in other words the logic is part of each individual report.
Eventually, organizations would find themselves with literally 1000s reports each with their
own nested logic.
The drawbacks of having logic spread out like this quickly revealed themselves.
Often the organizations would have a change to the underlying application which changed
how the data logic needed to be conducted.
This meant that 100s if not 1000s of reports had to be manually opened, and edited, and
saved and republished in order to account for the changes.
This issue still exists today.
However, Business Intelligence vendors over time have devised clever methods of getting
around this issue.
Instead of nesting the logic in the report template file, they came up with a virtual
modeling layer that allowed organizations to centralize all their logic in one place.
This way the reports generated from this layer would inherit all their logic from a single
location.
This layer is called a metadata layer.
So if changes needed to be made, they could be done once in the metadata layer.
However, there was a catch, these metadata layers would only allow you to use the reporting
tools provided by that BI vendor.
In other words, if you wanted to get access to that data, you were at the mercy of whatever
reporting tool the vendor gave you.
So you better hope they you don’t have more than one Business Intelligence tool.
To address this and other use cases, in the mid 2000’s the Data Integration industry
began introducing the concept of Data Federation.
These technologies essentially decoupled this virtual logic layer from the Business Intelligence
tools and exposed the access to the data as a connection.
So companies could connect any reporting tool to it and the data would be gathered for me
by issuing a standard SQL Query, and that query could be federated against multiple
sources of data.
At the time, Data Federation was lauded as the upcoming replacement to a Data Warehouse.
But this misconception was quickly debunked, as the speed of data access was ultimately
a function of how fast the data source is able to produce results.
Since a Data Warehouse stores all the history in an optimized model, it still proved to
be the most efficient method of delivering Business Intelligence data.
As the industry has matured however, the use cases for Data Federation have also matured.
Rather than considering Data Federation as a replacement to a Data Warehouse, it has
become a complementary solution to it.
Modern Data Federation tools are now called Data Virtualization tools.
This is because they are no longer simply for data connectivity.
But, rather they now generally represent a suite of virtual data acquisition tools that
include functions such as virtual data profiling, data quality, advanced caching, ETL tool integration
and business analyst friendly interfaces.
This means that Data Virtualization can now play the role of agile BI development.
One of the most common complaints in organizations is how long it takes to turn around a request
for information.
Often this is because the request requires changes to the underlying data warehouse which
will require changes all the way up the data gathering chain.
Edits will need to be made at ETL layer and possibly the model in the data warehouse,
as well as the BI metadata layer.
This might not sound like much, until you take into account that there are multiple
people managing each layer.
Then to add insult to injury, often the end result is not what the business wanted to
begin with.
Because of this slow and inconsistent process, business users often begin establishing “shadow
IT” departments and building their own data acquisition logic, in the hopes that they
can circumvent this lethargic process.
But then you have to ask yourself, didn’t you invest in data integration so you could
eliminate multiple version of the truth?
With Data Virtualization the process changes completely.
It does this by providing Business Analysts rapid access to the raw data, just as they
would normally see in their “shadow IT” but in an organized interface.
This doesn’t mean that IT is out of the picture; rather it means that the two can
collaborate on the same data integration logic necessary to satisfy the business.
All this is done without any physical data movement, so the number parties involved is
kept at a minimum.
When the Business Analyst is satisfied with their data acquisition, they can use that
very connection with their BI tools or if they have an advanced Data Virtualization
tool they will be able to push that data acquisition logic down to the ETL tool and enrich the
data warehouse.
To sum up, Data Virtualization can provide agility to an organizations data integration
and Business Intelligence process by: 1.
Enabling Operational Reports to be based on centralized logic
2.
Decoupling the Business Intelligence tool from the data acquisition logic, allowing
multiple BI tools to acquire from the same metadata layer
3.
Speeding up the information delivery process by engaging the Business Analysts and decreasing
the number of steps required I invite you to reach out to Intricity and
talk with one of our specialists about your current data integration needs.
We can set you up for future success without sacrificing your current most urgent needs.