>> Good afternoon.
I am Nigel Lewis and I am a member of the Protocol and Validation Team.
And today I will be discussing the process of Internal Data Validation, with you.
Our objectives for this session are to define internal data validation and quality data,
identify internal data validation resources developed by NHSN,
and describe recommended internal data validation activities.
First, we'll start by defining internal data validation
and quality data and discuss their importance.
There are two types of data validation.
Internal data validation is the active process of checking your data to ensure quality.
Internal data validation, or a validation of your own data, is generally built
in as a routine process at the individual facility level.
External validation involves an on site survey
and medical trial reviews conducted by an external agency.
This process is more research intensive
and accessess the accuracy of HAI data reported to NHSN.
Today, we will be focusing on internal validation.
What do we mean by quality data?
Quality data is data that's complete, timely, accurate, and consistent.
I'm sure many of you are wondering why Florence Nightingale is in the data presentation,
because in addition to her contributions to the nursing profession,
she is also known for her work in statistics and for using statistical data
to help reform lives, reforms that save lives.
Why is quality data important?
Quality data is important for helping you
to assess whether your prevention efforts are effective.
It allows you to benchmark your infection rates and progress against national data.
It ensures you are fulfilling reporting requirements of state, CMS, and other entities.
And it also gives you the confidence that your data will hold up against an external audit
from the State Health Department, or CMS.
Routine internal data validation leads to high quality data.
How does internal validation lead to quality data?
For routine internal data validation process identifies where the systematic weaknesses
in data collection and reporting are, and also how to correct them.
It assures you that the data reported by your facility is complete,
timely, accurate, and consistent.
It promotes coordination and partnership between stakeholders
and helps build confidence in your own facility data.
Why should you validate your own data?
The NHSN application has business rules to check if the data is sensible and reasonable.
These include validation errors that prevent you from entering implausible data and ensures
that you are entering all required data.
It also has alerts, which inform you of incomplete and missing data.
However, the system cannot check to see that all data was identified at the facility level
or the data was entered accurately.
Validating your data assures your stakeholders that your program is based on quality data.
Remember, this is your data.
You may be surprised at what you find,
especially if your data is submitted electronically-
We cannot emphasize enough that you really need to verify
that the electronically submitted data is accurate.
We understand that validating your HAI data can be an intimidating task,
so NHSN provides guidance for facilities to develop an annual plan
for surveillance and data validation.
The 2020 Internal Validation Guide, as a tool,
it can be found on the NHSN Data Validation webpage at the URL above.
You can also access the Data Validation webpage
by selecting data validation guidance from the navigation bar.
The Internal Validation document provides guidance and recommendations
for data validation activities to ensure that the data reported by NHSN is,
reported to an NHSN is of high quality.
Included are data quality survey tools that can be used to learn about your facilitys
current practices and identify needs for additional staff training.
There are also data quality checklists that provide a list of metrics recommended
by an NHSN SMEs for reviewing your data prior to the final submission.
We suggest that activity for data validation at the facility are a group by their frequency.
In the following slides, I will review the suggested activities for each time interval.
The suggestions include annual checks, monthly checks, weekly checks,
and also checking the data whenever necessary.
We recommend that the facility creates an annual surveillance and data validation plan,
a list of frequency and types of data quality checks,
and identify individual responsible for implementation.
You also want to ensure that you program adheres to NHSN protocols,
definitions, and surveillance methods.
Conduct the check of adequacy of your facility infrastructures, including the ability
for electronic reporting, ability to link systems, and the accuracy of the data obtained
from electronic sources.
Accuracies in these elements are necessary for the consistency of the data.
Make sure to review facility descriptors, and if additional locations were added,
make sure that they have been mapped correctly.
Recruit partners, because we know and understand IPs cannot do complete surveillance
of validation alone.
Annually, you should assess the knowledge and practices for data collection.
This helps to identify gaps in training needs.
An annual facility survey will be discussed in depth later on,
but I want to emphasize the importance of verifying the data for teaching hospital status,
number of beds, location, and location mapping, since this data is used
for the facility risk adjustment.
Incorrect data will impact your survey.
Additionally, you can administer the Data Quality Survey Tools.
These surveys include questions on numerator and denominator collection practices and rationales
for the correct answers to assure staff understands the correct NHSN procedures and definitions.
They are helpful for your facility to learn about the current surveillance practices
and identify any gaps in knowledge and used for training.
Determine which denominator counting method is more appropriate for your facility;
manual daily counting or the sampling method.
Here are a few examples of what you might want to look for.
Does your staff understand the definition
of central line and indwelling urinary catheter days and how to count them correctly.
Are there situations where data is missing or implausible data is reported?
For example, reporting total patient days rated in total beds in a facility.
It is vital to keep a log of how frequently these errors occur
and review these results with your staff.
Sometimes these things are just a matter of oversight.
We recommend that once a year you check the data collection practices
or validate a few locations for a week for assessment of continued accuracies of counts.
When switching from manual denominator counts
to an electronic denominator data collection system,
be aware that electronic data is often inflated unless they are validated.
Manual counts, which describes counting patient days and device days
at the same time every day for each recording location is considered the gold standard
against which other denominator, denominator counting methods should be validated.
NHSN requires every facility to validate their electronic denominator counts with manual counts
with every recorded location until the electronic counts are within 5%
of the manual counts for three consecutive months.
Once a reporting location achieves this goal,then you can begin reporting electronic counts
for the location.
If there is any deviation for any month in a reporting location,
then the three month counts start again.
If there are any problems identified, work IT to correct them.
There are also denominator validation template
in the Guidance document to assist with this process.
If you are switching from one electronic counting system to another,
we require that this same validation process be used.
Remember, manual counts are considered the gold standard,
so you should validate the electronic denominator counts
from the new electronic system with manual counts for every recording location
until the electronic counts are within 5% of the manual counts for 3 consecutive months.
Comparing a new electronic system to an old electronic method can magnify errors
and result in inaccurate denominator counts.
Again, this validation needs to be done only once, not every year.
However, we also highly recommend that you conduct spot checks
to assure continued accurate data captured by the electronic system.
For SSI denominator data, to ensure that the data procedure,
to see that the procedure data is not missed, identify all potential data sources
for catching procedures for ICD 10 PCS or CPT codes, such as hospital discharge data.
Cross referencing of multiple sources of data can reduce missing
or overreporting of the procedures data.
For example, you can compare your OR procedure schedule with discharge coding
to ensure complete procedure catching.
You will also want to perform spot checks to ensure complete procedure catching,
especially if you are using an electronic system to upload.
True story.
One hospital was electronically uploading their procedures and not validating the data.
One month the IP entered an SSI but could not link it to the procedure, because the procedure was not in NHSN.
Upon a further investigation,the IP discovered
that the electronic system they were using was programmed to capture procedures that occurred
from the 1st to the 30th of the month-
procedures performed on the 31st, were not being captured.
This had been occurring for a few years before the error was ever discovered.
So, you can only imagine how frustrated the facility may have been.
On a monthly basis, ensure that you have directly completed your monthly reporting plan.
Facilities are strongly encouraged to check the completeness and accuracy
of their data on a routine monthly basis.
This is very beneficial for maintaining data accuracy.
The samples of what to check for include missing data, implausible data, outliers that can result
in data exclusion, incomplete and inaccurate data,
and outstanding NHSN alerts for incomplete and missing data.
We spoke about clearing your alerts, and we want to emphasize
that if you don't clear your alerts, your data will not be complete or accurate,
and this data will not be included in your SIR or submitted to CMS.
Prior to your monthly data submission,
we recommend you review your data using the Data Quality Checklists
in the Validation Guidance document.
The data quality checklist in the Internal Validation Document provided recommended the data quality
checks, associated actions, and the analysis reports for checking the data.
There are three checklists available; one for CAUTI CLABSI, one for LabID, and one for SSI.
This is a snapshot of a portion of the SSI Event checklist.
The checklist provides step by step instructions for validating both denominator
and event (numerator) data for each HAI type.
The data quality checklist provides detailed instruction
on which analysis reports are available for checking each data element and step
by step instructions on how to access these reports.
For example, when verifying the denominator accuracy for LabID data, the checklist tells you how
to check the data and provides instructions
for assessing the three different analysis options available for checking the summary data.
We recommend you use the data quality checklist in conjunction with the information provided
to customize monthly data quality checks of your data.
In addition to monthly checks, you might also consider reviewing your data quarterly,
prior to CMS data submission deadlines.
On a daily or weekly basis, review positive laboratory specimens and conduct spot checks
to track accurate denominator data collection and conduct active case finding.
In addition to the recommended routine to have the quality checks,
what else would trigger additional data quality checks?
If there were new patient care locations added to your facility, are they correctly mapped?
You could have new staff collecting data.
You will want to verify that they are doing it correctly.
Have you implemented a new electronic medical record system
or has your current system been updated?
And you will want to validate the new system with manual counts.
And if you see any unusual data, verify that the data was collected
and entered completely incorrectly.
Here are some additional recommendations.
Besides annual and monthly checks, we recommend facilities
to conduct routine case-ascertainment checks.
CDC external validation guide provides medical records interaction tools
with case ascertainment.
Facilities could use these tools to identify your ability to accurately ascertain cases.
We would not want to dictate the frequency, but we highly recommend
that this process is conducted routinely.
You might consider doing this by using multiple reviews to check interrelated reliability.
And when cases are missed, investigate why and how to fix it.
Another piece of the analysis reports in NHSN is the ability to generate line lists
and run charts for analysis to assess the patterns of errors.
Longitudinal data can be run for numerators, denominators, and SIRs by location and overall.
This slide shows the CLABSI events and the LTAC award from January through December of 2017.
If you notice, the reported class is declined from 10 events in January to 1 in February.
It increases from 6 to 13 events between May and June.
Patient and Device days were similar between January and February,
and, again, between May and June.
The sudden decline and increase should trigger that equality checks
to determine whether these are true declines and increases.
However, you do not want to wait until after the CMS submission deadlines
to check longitudinal trends.
Check these trends periodically to identify, to identify potential data errors
so you'll have plenty of time to correct them before the deadlines.
So, what can you do at a minimum at your facility to ensure that your data is complete?
Here are some samples for completeness of numerator data.
The CLABSI and CAUTI review every positive blood specimen and urine culture as candidate events
to rule out the possibility of a case.
For potential SSI, identify and review all post op patients and hospital readmissions.
Reviewing just wound cultures might miss half of potential SSIs.
So, it's necessary to identify potential infections during daily rounds.
But LabID events, assess and review all lab test results from in patient,
emergency, and observation locations.
And to ensure the numerator data completeness, review the presence of central lines
or urinary catheters for every positive specimen or culture and ensure
that the collection of device days is complete.
For procedures being reported, ensure that you have gathered the complete count
of the procedures based on the ICD 10 PCS and CPT codes,
While reporting FacWideIN LabID denominators,
ensure that all patient admissions and encounters for every location have been included,
and that you have subtracted up IRF, IPF, and NICU/well baby counts appropriately.
It is important to have confidence in the quality of your facility's data.
Routine internal data validation ensures that your facility is up to date with NHSN methods
and surveillance definitions, and that you are applying them correctly.
In summary, quality data is vital to HAI prevention.
An ongoing internal data validation is key for improvement in prevention practice.
Routine internal data validation also lead
to improve [inaudible] your facility undergoes external data validation.
In conclusion, if you have been viewing this training video during March 2021,
please submit questions to be answered during a live Q&A session by selecting the title
of the presentation and submitting your question using the form located
on the 2021 NHSN training webpage.
Thank you.
And we look forward to hearing from you.