Every quarter, somewhere in a pharma affiliate, a training report lands on a director's desk showing 100% completion. Everyone attended the product update. Everyone passed the end-of-module quiz. The report is green from top to bottom.
It is also nearly useless. Completion tells you who showed up. It doesn't tell you who can answer a specialist's question about the safety profile, who confuses two endpoints, or who is quoting last year's data. The analytics that matter after training are question by question and person by person, and they usually tell a very different story from the LMS. Below is what that looks like in practice, using one pharma program we ran, and what your training analytics should be able to tell you before the next cycle meeting.
Completion, attendance and hours measure effort, not knowledge
Most commercial organisations still measure training with three numbers: who attended, who completed, and how many hours were delivered. They are easy to collect and easy to report upward.
They share one flaw. All three measure the input. None of them measures whether anyone knows anything afterwards.
That matters more in Life Sciences than almost anywhere else. A rep who leaves a training session unsure of the contraindications, or who misremembers a primary endpoint, is not just less effective. They are a compliance risk every time they walk into a practice. Knowledge in this industry is both a commercial lever and a responsibility.
And knowledge doesn't stay put. Murre and Dros's 2015 replication of Ebbinghaus, a single-participant experiment with nonsense syllables, shows that unrehearsed material fades fast and that spacing helps. That is enough to say a completion certificate issued in week one tells you little about what is left in week six.
One pharma team, measured question by question
Here is a program that shows the gap between "trained" and "ready" better than any argument.
A multinational pharmaceutical group with a specialised prescription portfolio in highly regulated therapeutic areas had, by any conventional standard, a well-trained commercial team. It ran structured sales training, a formal onboarding program for new hires and regular scientific updates. Completion was tracked in the LMS and looked fine.
What it had never done was measure what the team actually knew, concept by concept. So it ran a gamified microlearning challenge with us across its critical medical and product topics.
The group was small, 23 people chosen by the company for this first program rather than a random sample of its field force. That makes the results easy to read, but it also means they describe this team, not the whole organisation:
- All 23 took part; the company did not make it mandatory.
- Each person answered around 240 questions on average, 5,543 answers in total.
- At first attempt, we found 1,263 knowledge gaps, 312 of them on content flagged as critical.
- On that critical, must-know content, 59% of responses were wrong (58.98% in the program records). The denominator is all responses on critical questions during the challenge, not first attempts only and not people; the records give the rate, not the count of critical responses.
Read that last line again. This was a trained team, with green completion reports, answering questions about the material they detail every week. On the content the company itself had marked as essential, most responses were wrong.
That is not a story about a bad team. It is a story about what completion reports hide.
On the content the company itself had marked as essential, most responses were wrong. The completion report was green.
From "I think they know" to "I know where we are failing"
The point of question-level analytics is not to catch people out. It's to replace impressions with a diagnosis.
A good analytics layer answers four questions precisely:
- Who has a gap?
- On which specific question?
- In which topic, and is that topic critical?
- Has that same person since answered it correctly, or is the gap still open?
The fourth question is the one most tools skip. A gap, in the way we measure it, is one person wrong on one question at first attempt. It only counts as closed when that same person answers it correctly later. That definition matters because it separates real learning from averages that move because different people happen to play on different days.
Once you have that level of detail, the conversation changes. A first-line manager no longer hears "your team scored well on the module". They hear "three of your reps confuse the two dosing regimens, and one of them has a meeting with the department head next week".
What happened after the gaps were found
Finding gaps is the start. The more interesting part is what you can do with them without pulling people out of the field.
Repetition closed the easy half
During the challenge phase, questions each person got wrong came back to them in short, spaced rounds on their phone. There were no extra workshops and no blocked sales days. The effect was significant. Total gaps fell from 1,263 to 731, a reduction of 42%. Critical gaps fell from 312 to 200, a reduction of 36%.
That second figure is lower for a reason we see often. Critical content tends to be the content that is conceptually harder, and some errors on it are beliefs rather than slips. Repetition alone fixes slips well. It is less good at beliefs.
Targeted gap fixing closed nearly all the rest
After the game, the program switched to gap fixing on the 200 remaining critical gaps. Instead of retraining everyone on everything, each person received the correction for their own specific mistakes, and was then tested on those questions again.
All 23 people with critical gaps took part. Of the 200 critical gaps still open at that stage, 190 were resolved, 95%. Ten remained, each one identified by person and by question. Keep the denominators apart: 95% is the gap fixing stage (190 of 200). Across both stages, 302 of the original 312 critical gaps had a later correct answer, 97% under this program's reporting definition. And each closure means the same person later answered correctly within the assessment; one later right answer does not rule out a lucky guess, and it is not proof of better calls.
That is the real turning point. The organisation went from "we don't know what we don't know" to "we know exactly which ten critical weaknesses remain, and who needs a conversation with their manager or with Medical".
Putting a number on risk, and why we treat it with care
The team went one step further. It linked each critical gap to an estimate of potential lost opportunity, based on its own assumptions about the value of a well-handled HCP conversation, and put a euro figure on the exposure. The estimate at the start was €485,484. After the program, the model counted €427,880 of that as resolved, about 88%, leaving €57,604 attached to specific, named gaps. To be plain about it: that is a modelled reduction in estimated exposure, not realised or incremental revenue. Nobody measured sales before and after.
We like this exercise, with a caveat. That figure is a model, not a measurement. It depends on assumptions about call value, prescribing behaviour and how much a single knowledge gap costs, and a different finance team would produce a different number. A fair objection is that putting euros on knowledge gives a false sense of precision.
Our answer is that the absolute number matters less than two things it does well. It forces a conversation between Training, Sales and Finance about which knowledge actually drives revenue, which is a conversation most organisations never have. And it makes the trend visible in a language senior leaders already use. If you build a model like this, make the calculation transparent: exposure per gap equals the number of affected calls, times the assumed share of those calls a gap would change, times the assumed value of a call. Publish each input next to the number, run it with a low and a high value for each assumption so the range is visible, and note what the model ignores (access, pricing, competitor activity). Then use it to rank priorities, not to promise returns.
How to start measuring in your next program
You don't need a new platform strategy to begin. You need a different set of questions, and a few weeks.
- Pick one real business moment: a cycle meeting, a label update, a congress, a new hire cohort. In our experience, programs tied to a moment get more play.
- With Medical and Marketing, mark which topics are critical. Usually it is the handful of things a rep must never get wrong: indication, key efficacy figures, main safety messages, the positioning line.
- Write questions around what is actually said in a call, and include wrong options that reflect the mistakes you suspect the field makes.
- Measure first attempts before anyone has seen the answers, so the baseline is honest.
- Bring each wrong answer back to the same person later, and count a gap closed only when they get it right.
- Hand the stubborn remainder to managers and Medical as a named list, not as a percentage.
A small team is a perfectly good place to start. The program above had 23 people, and it produced a clearer view of readiness than years of completion reports. What matters is the resolution of the data, not the size of the audience. Start with one team and one moment, show the results to the people who own the messaging, and let the findings make the case for going wider.
Why many LMS reports don't show this
None of this is a criticism of learning management systems. They do their job, which is to administer training: assign courses, record completion, keep the audit trail regulators expect. Many also run tests and quizzes.
What many LMS set-ups do not show, in the reports a commercial director actually receives, is:
- which specific question a specific person cannot answer
- how many critical gaps remain across your field force today
- which topics are fixed by repetition and which need coaching
- whether a gap found in January is still open in March
That requires measuring knowledge continuously, at the level of a single question and a single person, and it requires people to answer enough questions, often enough, to produce that data. That is why the mechanics matter. Short, competitive formats that people choose to play generate far more answers than a quiz at the end of a module, and more answers means a clearer picture.
The bigger lesson: invisible risk stays unmanaged
The real value of this program was not the pilot itself. It was what it revealed about this team's knowledge gaps: knowledge gaps exist whether or not you measure them, and without measurement they are invisible. Invisible risk doesn't get managed. It gets discovered, usually by an HCP.
The fix is not more training hours. In this case, the team didn't need a single extra workshop. It needed to know precisely where it stood, a mechanism that brought each person's mistakes back to them, and a targeted follow-up for what was left.
If you want one practical step from this, try it at your next quarterly review. Next to the completion rate, ask for three more numbers: how many critical gaps did we find, how many are verified closed, and who holds the ones still open? If nobody can answer, you're measuring activity, not knowledge.
That is the problem we built Atrivity to solve: short, gamified challenges that field teams actually play, gap fixing that sends each person the fix for their own mistakes, and analytics that show readiness by team, country and topic. But whatever tools you use, start by asking the question. Every commercial organisation has knowledge gaps. The ones that do well know where theirs are.