Picture a structural heart implanter who has done hundreds of tricuspid cases and is now looking at a CT of a young, low-risk patient with a bicuspid valve. The anatomy is asymmetric, there is a calcified raphe, and the annulus is not a neat circle. She turns to the clinical specialist in the room and asks the question that decides the case: what does the registry evidence actually say about this valve in this anatomy? The program we describe ran in 2021.
That question was the brief. A global MedTech company in transcatheter heart valves was about to put a bicuspid campaign in front of heart teams, built on registry data and on anatomy that the original randomised trials had largely excluded. Nobody knew which of the field could defend it. We ran a single four-day battle for 91 sales representatives and clinical specialists, and 49% of the 685 knowledge gaps we found at first attempt were verified closed. Readiness moved from 84% to 92%. The more useful result was where the other half of the gaps sat, and why most of them belonged to the clinical team rather than to another round of questions.
A campaign built on evidence the field had never had to defend
Most device campaigns rest on the randomised trials everyone in the field has presented a hundred times. This one did not.
Bicuspid aortic valve patients were, for years, kept out of the major transcatheter trials. The evidence for treating them comes mostly from national and international registries, observational cohorts and a growing body of single-centre and multi-centre publications. That kind of evidence is harder to present than a primary endpoint. It comes with confidence intervals, selection caveats, different follow-up windows and, crucially, comparisons against a competitor valve that were never randomised. A rep who says "our leak rate is lower" without the condition, the cohort and the timepoint is inviting an interventional cardiologist to take the claim apart in front of colleagues.
On top of the evidence sat anatomy. To talk credibly about bicuspid cases, the field had to be fluent in raphe and commissure orientation, the Sievers classification, how bicuspidity changes sizing, and why annular measurements behave differently when the valve complex is not round. Then there was the campaign itself: the claim, the CT screening message, lifetime management, the dates and the commitments the field had to log in the CRM.
Put simply, the stakes were three:
- An overclaim in front of a heart team is worse than silence. It damages trust for every case that follows.
- An understatement is a lost argument. Hesitant implanters ask for the reassuring numbers first.
- Campaign activity is easy to count and tells you nothing about whether the story holds up in the cath lab.
What we designed, and why a single battle was enough to start
The client did not need a months-long curriculum. It needed a read, fast, before the campaign went live, on who could carry which part of the story.
So we built one game with 48 questions and ran it as a battle across a four-day window. Participants competed in one-minute quiz challenges, five questions in 60 seconds, against real colleagues in the same global field force. Notifications pulled people back in across time zones. Every question someone got wrong came back to them later in the window, which is where the "verified closed" measure comes from: a gap is one person wrong on one question at first attempt, and it only counts as closed when that same person gets it right on a later attempt.
We split the 48 questions deliberately. Roughly half covered comparative evidence: pre and post-dilatation rates, paravalvular leak, pacemaker implantation and annular rupture, drawn from registry and low-risk cohort data for the client's valve and its main competitor. The other half covered bicuspid anatomy and typing, annular sizing, the registry and trial sources themselves, and the mechanics of the campaign.
Why a battle, and not a course? Three reasons.
- Speed of signal. A synchronised window gives you a clean baseline for the whole cohort in days, not a trickle of completions over a quarter.
- Honesty of answers. People answer faster and more instinctively under light time pressure, which is closer to how they answer a surgeon in a corridor than a leisurely quiz.
- Repetition built in. Because wrong answers return, the battle measures two things at once: what the field knew, and what it could learn by repetition alone.
That third point matters more than it sounds. If a gap closes on its own after two or three encounters, you don't need a clinical specialist's time for it. If it doesn't, you do.
Four days, 21,000 answers, and a clear split in the story
All 91 players took part, giving 236 answers each on average, about 21,000 in total.
At first attempt we logged 685 knowledge gaps. By the end of the window, 335 were verified closed, 49%, and 350 were flagged for follow-up coaching. Readiness, the share of questions answered correctly, rose from 84% on first attempt to 92% on last.
An 84% starting point looks reassuring. It hides the fact that the 16% the field got wrong was not spread evenly. It was concentrated precisely in the parts of the campaign that carried clinical risk.
Here is how the gaps and closure broke down by area. The five areas account for 676 of the 685 gaps; the program records do not assign the remaining nine to any of these areas, so we leave them out of the breakdown rather than guess:
- Procedural outcome comparison (dilatation, leak, pacemaker rate, rupture): 252 gaps, 47% closed.
- Bicuspid anatomy and typing: 140 gaps, 47% closed.
- Registry and trial evidence: 113 gaps, 40% closed.
- Campaign execution: 105 gaps, 70% closed.
- Annular sizing and method: 66 gaps, 39% closed.
Read that list twice. The campaign mechanics, the part the field owns and repeats every week, closed at 70%. The evidence and the sizing, the parts an implanter will actually probe, closed at around 40%. The field could run the campaign. It could not yet fully defend it.
A word on what "defend" means here. The battle tested knowledge of the approved evidence and anatomy, in multiple-choice form. A later correct answer is a verified correction within the assessment. It is not clinical competence, it does not show how someone performs in a case, and one later right answer does not rule out a lucky guess.
The field could run the campaign. It could not yet fully defend it. Activity metrics would never have told us which half was which.
Three questions that told the story
We tracked individual questions to see how specific misconceptions behaved. Three are worth describing, without reproducing the question text.
The first asked how widely the client's valve had been implanted across the main international bicuspid registries. Twenty-one people got it wrong at first attempt. Repetition fixed only three of them, and 18 were still wrong at the end. Most of those errors went in one direction: a dozen people understated the figure. That is the first number a hesitant implanter asks for.
The second asked for the 30-day moderate or severe paravalvular leak rate for bicuspid patients in a large national registry. Here the error ran the other way. Twenty-four people chose a figure a quarter of the true registry value at first attempt. Of the 26 wrong, repetition corrected ten and 16 were still wrong at close. Quoting a leak rate four times better than the registry in front of a heart team is exactly the kind of overclaim that ends a conversation, and it is also the kind of error that a compliance review would flag in a heartbeat.
The third asked for the post-dilatation rate of the competitor valve in low-risk patients. Twenty-three people inflated it, picking a number far higher than the published figure. Repetition fixed about half, 12 of 26, but 11 people still held the inflated number at the end. Once one number falls to an implanter's challenge, the others start to look shaky.
Notice the pattern. These were not random slips. They were directional beliefs, sometimes flattering to the brand and sometimes not, and they resisted simple repetition.
Five lessons we took from it
1. Starting readiness is a vanity number unless you break it down. An 84% average said "we're fine". The breakdown said "your campaign mechanics are fine, your evidence story has holes". Always look at closure and error by topic before you look at the headline.
2. Registry evidence needs more care than trial evidence. Trial and registry findings both require attention to study design, endpoints, comparators and limitations. Registry data has cohorts, timepoints and non-randomised comparisons, and the field is less used to it. The areas that closed worst, sizing at 39% and registry sources at 40%, were exactly the ones built on this kind of evidence.
3. Direction of error matters as much as the error. An understated figure costs you arguments. An overstated figure costs you trust and can put you on the wrong side of your own medical-legal review. We now report which wrong option people chose, not just that they were wrong, because the coaching conversation is completely different.
4. Repetition sorts gaps into two piles. Some gaps close on their own after a few encounters, and the campaign mechanics did exactly that. Others barely move. On the registry implantation question, only three of 21 people fixed it by repetition. Those gaps aren't a content problem. They are a belief, and beliefs need a person to explain the why.
5. Four days is enough to point your clinical specialists at the right people. The 350 open gaps were not a failure. They were a list: which person, which question, which direction. That list turned the clinical team's limited time into targeted conversations instead of another all-hands webinar.
A fair objection is that a four-day game cannot build real clinical fluency in bicuspid anatomy. True, and it wasn't the goal. The fluency comes from case reviews, proctoring and time with experienced implanters. What the battle does is tell you who needs that time most, and on which topic, before the campaign hits the cath lab rather than after.
There is a second boundary worth drawing. Knowing the registry numbers well is not permission to go beyond them. The campaign's approved claims were specific to the cohorts, timepoints and anatomy the data covered. A rep who knows a leak rate cold can still overreach by extending it to a patient group the registry did not study, or by turning a non-randomised comparison into a superiority claim. The questions tested the approved claims. Anything inferred from them belongs to Medical, not to the field.
What we would do next
Looking back at this program three years on, there are three things we would add.
First, a second wave. The natural follow-up to a baseline battle is a short reinforcement phase two or three weeks later, after the clinical team has worked through the flagged gaps. That second measurement would show whether the leak and registry misconceptions had actually moved, or only been talked about. Murre and Dros's 2015 replication of Ebbinghaus' forgetting curve, a single-participant experiment with nonsense syllables, is a reminder of the general point: unrehearsed material fades fast (savings of about 4% after 31 days in that study), and spacing helps. It tells us nothing precise about registry numbers, but it is reason enough not to trust a single sprint.
Second, sharper targeting of the fixes. Today we would send each person the explanation for their own mistakes, and route the stubborn directional errors (the overclaims especially) straight to a clinical specialist or manager with the person and question attached.
Third, we would split the audience. Sales representatives and clinical specialists have different jobs in the room. A specialist who cannot talk sizing is a bigger risk than a rep who cannot, and the readiness index should be read by role, not only across the whole cohort.
There is a broader point here for any campaign or launch team. McKinsey's analysis of 210 launches between 2003 and 2009, published in 2014, found that around two thirds of new drugs missed pre-launch sales expectations in their first year, and that most early underperformers kept underperforming. Devices are not drugs, but the logic travels: the first months of a campaign set the story heart teams remember. You want the field defending it correctly from the first conversation, not the tenth.
If you have a campaign coming up that rests on new or unfamiliar evidence, the question to take to your next planning meeting is simple: do we know which half of our story the field can defend? We are happy to show how we set up a battle like this one, and what the first four days usually reveal.