An unprecedented medical school exam scandal left both US and
international students wondering about the future of their medical education.
Here is everything you need to know about the Nepal USMLE score scandal.
Dr. Jubbal, MedSchoolInsiders.com
The United States Medical Licensing Examination, USMLE for short, has incredibly strict practices
that ensure every medical student taking their board exams is on an equal playing field.
They go much further than checking if photo ID matches test registration and proctors to monitor
unusual behavior. They go as far as to measure how long it takes to answer each question, in addition
to comparing your correct and incorrect answer choices with other test takers in your vicinity.
They are extreme with their anti-cheating tactics because the USMLE is designed to
determine who is qualified to be a physician and treat patients in the United States.
Not having strict anti-cheating measures in place could result in
patients being harmed if unqualified students become practicing doctors.
At the beginning of 2024, the USMLE announced that a large number of test takers from Nepal
had unusually high Step exam scores, and many had selected the exact same
answer choices for the vast majority of questions, regardless of whether it
was the correct answer or not. The odds of this occurring without cheating are near-impossible.
To catch everyone up to speed, we first need to look into how the Nepali test takers were
cheating to get a sense of the scale of their operation. The evidence the
USMLE released to the public suggests the examinees were most likely using “recalls”.
This is when examinees write down the questions they saw during their exam
to later sell to others. This has been occurring since the dawn of the USMLE
and will likely always continue. However, these recalls are usually fairly low quality,
as they’re based on the test taker’s memory, and memories are fallible.
The other important detail to note is that the USMLE likely has fewer test questions
than we think they do. Writing and validating questions that tens of thousands of people will
interpret in nearly the same way, regardless of their individual biases, is not a simple task.
There is a great chance questions will be recycled from one exam to
another, which makes recalls an unethical but lucrative business.
Some defending the Nepali test takers have argued these recalls are no different from
resources like First Aid and UWorld; however, those resources are used to teach students
medical concepts, whereas recalls are live USMLE test questions that are sold
to classmates and future test takers. Since recalls are exact exam questions,
they are often memorized rather than used as a tool to deeply understand the correct answer.
Naturally, creating, selling, and buying these recall questions is strictly against
the rules of taking board exams, as is even speaking about questions you saw
with friends. It’s part of the terms every test taker agrees to before they
step foot inside the testing center and again right before they take the test.
Nepali test takers joined WhatsApp groups designed to help others earn the highest
possible test score. The scale of this cheating operation is unprecedented,
and at the release of this video, 832 test takers have had at least
one of their board exams invalidated. The document, titled “Savior,” was over
a thousand pages long and included over 90% of the live USMLE test questions.
The people who bought these perfect recalls, which were unauthorized materials,
were able to memorize the questions and the correct answers before they
ever entered the testing center. Looking at this graph, you can see how Nepal’s
average score skyrocketed to well above any other country as a result of “Savior.”
Fortunately, the NBME was tipped off in January 2023. Due to the high number of
suspicious scores from Nepali students, NBME focused its initial investigation on
Nepal before launching a full investigation. They were able to join these WhatsApp groups
to discover first-hand how accurate and how widespread the cheating was. In the year since
those initial tip offs, the NBME notified individuals whose exams were invalidated.
And this is where things get interesting. One test taker chose
to sue the NBME for discrimination after being identified as a potential cheater,
claiming many Nepali students were being wrongfully targeted and that it violated
their civil rights. This case went to court, and the judge ruled in favor of the NBME,
saying the choice to look at Nepali students was based on data, not discrimination.
USMLE’s calculations found there was less than 1 in a hundred million chance
that these test takers from the same region would have answers that matched so similarly.
So, how did they get caught,
and why was the motion to restore scores denied? The simple answer is math.
The emergency motion to restore score validity was denied by a Washington,
D.C., district court judge due to the clear evidence of cheating.
Each USMLE question is designed to take 90 seconds, as that’s generally the amount of
time it takes to fully comprehend the question and answer. The data
shows this particular test taker got every single question right that they spent fewer
than 20 seconds on while performing much worse on questions they took longer on.
Additionally, the test taker claimed that the reason it only took them 20 seconds
was because they were quickly guessing. If this were true, they naturally would not
have guessed correctly 100% of the time when choosing between 5 different answer choices.
Due to the strict anti-cheating measures USMLE exams have in place,
the performance of test takers on stolen questions shared in WhatsApp groups can
be compared to their performance on the non-compromised test questions.
If they perform approximately the same across all questions, it suggests the test taker was
likely not cheating and raises the probability of a false positive in the USMLE’s system. However,
if the test taker performs much worse on the non-compromised test questions,
it’s a huge red flag that indicates access to past test questions.
With all of their data, it’s extremely easy for the USMLE to find what they called “a pattern of
anomalous test performance,” suggesting “prior unauthorized access to secure exam content.”
The students who were suspected of having unauthorized access to past test questions
have had their exams invalidated. Having their scores invalidated doesn’t mean they cheated;
instead, they now need to prove they didn’t cheat.
Those who received a message about an invalidated score were given three options to choose from.
1. They could request that NBME reconsider
its decision. This is a process that could take up to 10 weeks.
2. They could choose to retake the test(s), or
3. They could choose to do nothing.
For both options 1 and 2, the right to sue NBME would be waived.
If the identified cheater was aiming to match this year, they now will no longer be eligible for this
cycle as they no longer have a validated test score. If they retake their exams,
and if they legitimately pass, they will be allowed to apply to the Match again next year.
Of course, when they apply, program directors will be able to look at
their entire USMLE transcript and see the highly suspect, invalidated score,
indicating they likely cheated on their prior exam attempt.
However, the USMLE was able to go several years back to identify current residents and even
attendings from Nepal who they think had access to secure exam content before they took their board
exams. Because their ECFMG medical license, which is the licensing foreign medical graduates need to
be able to practice in the US, depends on having valid, passing scores on USMLE Step 1, Step 2,
and Step 3, these individuals will immediately lose their ability to practice in the US.
So, what does this all mean for future med students?
If you’ve already taken your Step exams, you don’t need to worry that you might
have received a worse score due to the abnormally high number of cheaters in
the past few years. The curve the USMLE uses is only based on MD students in the US and
Canada who are first time test takers. They set the curve for everyone else.
DOs, IMGs, FMGs, and even repeat test takers have no influence on
the curve. If you already took the test honestly, you can be rest assured that
the score you received is actually yours, and will not be invalidated.
Due to USMLE’s extremely high threshold and degree of certainty for catching cheating students,
there will likely be some false negatives, which means some cheaters will slip through
the cracks. However, this degree of certainly all but ensures no false
positives will occur—meaning no one will be falsely accused of cheating.
Lastly, the unfortunate victims of this scandal are the patients who have been
treated by unqualified residents and physicians,
as well as the IMGs applying to match this year who were not involved in the cheating scandal.
The USMLE will continue to heavily scrutinize test takers to ensure
there is no variation in score reports. They have already collected large amounts
of suspicious data in countries like Pakistan, India, and Jordan.
The organization will be aggressive in ensuring cheating is impossible on board exams,
as this is one of the best ways to make sure the physicians practicing in the US are qualified to
take care of patients here. In the wake of this scandal, it’s expected that they will create
a great deal of new test questions, along with removing the compromised ones from circulation.
At Med School Insiders, we always aim to bring you the unbiased facts,
but if you want my unfiltered take on this cheating scandal,
check out my reaction video over on the Kevin Jubbal M.D. channel.