This is Fede’s workshop with Libby’s assisted AI voice.
Our workshop focuses on Aligning your Software
and Resources to your Data Requirements.
Data Architecture, Software, Hardware and Cloud components must be governed and aligned to an overarching data capability framework.
The purpose of such alignment offers visibility as to cost allocation, required skills and workloads within all your data management knowledge areas.
A data capability framework spans across your Corporate Drivers. Your Corporate Drivers are your main business focus and also sum up a number of Challenges.
Your organisation has a business strategy, an IT & Sourcing Strategy and we need to develop the data strategy.
In order to evolve in your strategic and operational developments we have a baseline of governance and policies.
Your operational developments will evolve by adopting methods and standards
and acquiring, developing and retaining the required people, processes and technologies.
In relation to developing your data capabilities, it is necessary to build your foundational data architecture.
We have adopted a simple combination of data management methodologies that allow you to evolve in your transformation product developments.
We provide an outline as to how you map your foundational data architecture to data management knowledge areas.
Our Foundational Data Architecture component is mapped to eleven knowledge areas including Metadata, Master Data, Data Privacy and Data Quality.
For example, if you look at Microsoft SQL Stack On-Premise or on the Cloud,
their software architecture provides one or more Knowledge Areas that comply with best practices in Data Management.
To illustrate this, we review a number of knowledge areas and how they map to data architecture components.
Master Data:
MS SQL Master Data Services enables an organisation to amalgamate master data from various sources,
consolidate master data entities, validate all entities and redistribute them to their data sources for better compliance.
Metadata:
An example is MS SharePoint, a system that is able to establish relationships between structured and unstructured data
by classifying and linking the data and allow end-users to search and retrieve records through connected metadata.
Data Quality:
MS Data Quality Services allows an organisation to create rules to correct bad data to good data.
Data Warehousing:
With MS SQL Data Warehouse you can build modern data warehouses for specific business goals.
BI and Analytics:
With MS SQL Analysis Services your business users can consume Tabular Data Models through interactive reports in Ms PowerBI or MS Excel.
Data Science:
Azure Data Studio Notebooks in Python or R provide a combined solution to publish diagnostic, predictive or prescriptive machine learning models.
Choose your data architecture and assess whether all your technology components make a good fit and scale throughout your digital transformation projects.
Aligning your software or hardware components to your data requirements is important.
But it is equally important to determine the relevant skills and roles that will work with these technologies.
The most brilliant innovation is irrelevant if you don’t have skilled staff to use it.
Roles should be lined up to each knowledge area to support your data requirements.
Each of these knowledge areas require qualified staff that deals with the workloads and all your transformation projects.
Your organisation may have to upskill and reskill people so that they are better equipped to adjust to change.
Moreover, you need to retain the skills and the knowledge within your organisation to evolve in your digital maturity.
We have shortlisted an example of software components that can achieve the data architecture goals within each area.
Procuring technology is a coordinated effort between Finances, HR, CIO and CDO offices.
Make sure that technology and human resources budgets are aligned for all your short term or longer term projects.
Very often there is a disbalance between procurement, technology, workloads, projects, staff and business priorities.
Don’t draft a data strategy limited on decisions of CIO or CDO.
You may encounter rejections at the procurement stage
because Finances or Human Resources didn’t participate at an early stage to the scoping of resources and infrastructure.
Each organisation requires a Governance Board,
where all players are involved, and all investments in technology and people are vetted by all required decision makers.
A Data Governance Board should be formed by Subject Matter Experts
in Data, Technology, Human Resources, Finances and should be lead by the CIO and/or CDO offices.
Your digital maturity roadmap must avoid shortcomings in decision making
that can have negative impacts on performance and eventually lead to failing projects and wasted investments.
Let’s say, that you are now at the stage where you have your data architecture and human resources sorted out.
All your software components and skilled staff will provision quality data for all your reporting channels.
A Modern Data Provisioning Workflow support mainly five functions.
Legitimate Data Sources:
An organisation has its legitimate datasets across various physical and legal business units.
Data sharing agreements are in place to provision datasets across business units.
Data Security and Integrity:
An Data Integration function provision data connectivity to link datasets and make them available across designated reporting channels.
Modern Data Warehouse:
A modern data repository at the centre facilitates all business data between your structured and unstructured datasets.
Analytical Services:
The Analytical Function is the end point to facilitate legitimate and curated data across reporting channels.
Governance and Trust services:
Many organisations skip completely the Governance & Trust Function to the detriment of missing out on data quality KPIs.
As a result, stakeholders find it unreliable to evolve into predictive analytics when they cannot trust basic statistical reports.
Having a Governance and Trust function would provision Master Data Services, Data Quality Services,
Data Classification and Business Workflow features that help your stakeholders assert the quality of datasets over time.
Let’s say that you have implemented your Governance & Trust Services.
All organisations have data quality issues.
Each company takes its own approach in curating data.
The difference is at the speed at which they implement best practices in data quality management.
Correcting data is a continuous process that analyses and curates your data year on year.
Therefore it will be necessary that you have all required data quality components at an early stage of your digital transformation project.
This diagram illustrates the magnitude of running ETL workloads at various stages of the data provisioning workflow.
At the source, each Data Controller may produce a number of ETL iterations on their own datasets.
As you move data to the centre, you join up data from many Data Controllers
and therefore the ETL complexity increases substantially.
In other situations, and when the data is not curated at the source or at the centre,
you have business units dispersed across the organisation run their own ETL workloads.
Many times this is done at the Target Output stage when you produce the actual report.
Curating at the Target Output stage is bad practice and the last option to go for.
Our ETL magnitude ratio, illustrates that the ETL workload is reduced substantially if you perform ETL at the source or at the centre.
When you do it at the centre, ETL procedures change only when the Data Controller organisation or Business workflow of the dataset requires it.
While data curation should happen at the source, at the centre, data quality and master data processes
are required to correct and unify data entities and link data across many business units.
If your organisation is running many ETL procedures at the Target Output Stage,
make sure that all ETL for curation of your data is moved to the centre or back to its source.
Let’s talk about your Development environment for creating data connections and curating data.
If we look at the Microsoft Stack, you have Visual Studio as the most popular development environment.
You work with Visual Studio to create your databases,
your ETL integration scenarios and you design analytical models for reporting.
There is also a myriad of data engineers that have developed the required skills over the last decade.
When you choose a data engineering stack,
make sure it will help you scale and manage your ETL processes, databases and analytical models.
Using Data for Reporting.
This Diagram from Dustin Ryan illustrates the connectivity of most SQL components for analytics.
The dark blue background have on-premise components, while the lighter blue backgrounds run Azure cloud features.
We are looking at a Microsoft Hybrid model.
The gateway component moves data from on-premise to the cloud.
The solution offers analytics through PowerBI Services
and embedded analytics in MS Teams, SharePoint or your online applications.
The business users have the option to retrieve paginated
or interactive reports from the PowerBI service on the Cloud or the PowerBI Server On-Premise.
The Hybrid model is a good option when you keep your data on-premise
or you are planning a transition of your on-premise data to the cloud.
If your data is On-Premise and you have connectivity to Azure Cloud
you most probably require a number of Microsoft components.
Microsoft components are available as a Hybrid Model for your digital transformation.
We are providing this basic outline of Microsoft software tools that offer all features
to analyse and curate your data over time and provision an end-to-end workflow.
Microsoft has made an important jump forward
as they introduced new Ms SQL features a few years ago to integrate data of many sources and formats.
Your organisation should assess what architectural scenario works best to evolve your current architecture and skills towards digital maturity.
It can be sometimes overwhelming to work on a digital transformation project.
And this is especially true when you have to balance out all require people, processes and technologies.
An organisation requires a top down development of their data capabilities
and your data strategy should be aligned to your business-, IT- and sourcing strategies.
And these high-level developments will take you a few months to gain consensus.
But that won’t stop you from evolving with a number of bottom-up product developments that you can tackle with a small team.
The pieces in the centre of our data capability framework is the engine that will enable your digital transformation.
This is called the body of knowledge in data management.
Data Security and Privacy will be as relevant as having the correct levels of Master Data or Data Quality.
On the other hand, your data sources will remain in a dispersed location without
proper data integration, metadata and classification models.
It will be challenging to think “big data or advanced analytics” unless
we start classifying, securing, integrating and sharing your valuable structured and unstructured data.
All this transformative developments will build the confidence of all stakeholders and operations and help you evolve in your data maturity levels.
But why is it so important to choose the correct data architecture features?
What goes beyond maturing your data capabilities?
Trusting your data will be the most fundamental component to mature in your analytics capabilities.
It will be extremely difficult for an investor or stakeholder to move into advanced predictive
or prescriptive solutions when the fundamental Executive insights around data are not trusted.
When you look at this analytics graph your organisation naturally will evolve from foundational data capabilities
to augmented intelligence with the support of AI.
This will lead you into the analytics roadmap.
The speed at which you mature in your data capabilities,
will also determine the speed in which you mature in your analytics capabilities.
The more you know about your data, market and customers,
the more your organisation is factual, data-driven and will develop a competitive advantage.
This workshop focused on Aligning your Software, and Resources to your Data Requirements.
We know by now that Trusting your data will be the most fundamental component to mature in your analytics capabilities.
All players such as your CDO office, CIO, Human Resources, Finances, ICT and Data operations must develop one digital culture, one voice
and start retaining the knowledge at your organisation which will determine its future survival in the Digital Economy.
This was Fede Nolasco assisted AI voice, and we’ll see you in our next workshop.