Phishing Simulations, Measured by People Who Do Not Sell Them

How CodoraTech is funded: CodoraTech is supported by advertising and, in some articles, by affiliate links. Where an article contains affiliate links, we say so at the top of that article.

Two field experiments, one following 14,733 employees for fifteen months and one following 19,500 for eight, have measured what simulated phishing campaigns and the training attached to them actually do to click rates. Both were presented at the IEEE Symposium on Security and Privacy, in 2022 and 2025 respectively. Neither found the improvement the security awareness industry advertises. The industry’s own benchmark reports describe reductions of 80 to 90 percent over a year. The two sets of figures are not measuring the same thing, and only one side of the argument owns the measuring instrument.

Key facts

• IEEE S&P 2022 study: 14,733 employees, 117,864 simulated phishing emails, 5.67 percent clicked per email
• 32.10 percent of those participants clicked at least one simulated phish over 15 months
• IEEE S&P 2025 study at UC San Diego Health: embedded training cut later clicking by about 2 percentage points
• 75 percent of UC San Diego Health staff spent one minute or less on the training page
• KnowBe4, which sells the training, publishes a fall from 33.1 percent to 4.1 percent over 12 months

Fifteen months, 14,733 employees, no measured resilience

Daniele Lain, Kari Kostiainen and Srdjan Capkun of ETH Zurich ran a simulated phishing programme inside a partner organisation from July 2019 to October 2020, publishing at the IEEE Symposium on Security and Privacy in 2022. The partner was a large public company with more than 56,000 staff in logistics, finance, transport and IT services. Its 14,733 participants received the simulations in their ordinary working context, not under lab conditions.

Per email, the figures look mild: of 117,864 messages sent, 6,680 were clicked, a rate of 5.67 percent. Cumulatively they do not. Across the fifteen months, 4,729 participants (32.10 percent) clicked at least one, and 3,747 of them (25.43 percent) performed what the researchers classed as a dangerous action, meaning entering credentials or enabling macros. Those actions spanned 4,885 emails, 4.14 percent of everything sent.

Failing more than once was ordinary. Some 1,448 participants, 30.62 percent of everyone who clicked, clicked two or more times, and 896 took a dangerous action on two or more occasions. One person fell for six of the eight simulations they were sent.

The authors’ summary of what the embedded training achieved is stated plainly in their abstract: embedded training during simulated phishing exercises, in their words, as commonly deployed in the industry today, does not make employees more resilient to phishing, and can have the opposite effect. Their raw counts run the same way, with more clicks among participants who received contextual training (3,593) than among those who did not (3,087).

Two percentage points at UC San Diego Health

The second study names its organisation, which matters, because most published phishing statistics do not. Grant Ho, Ariana Mirian, Stefan Savage, Geoffrey M. Voelker and colleagues examined UC San Diego Health, a large US healthcare system, and reported at the 46th IEEE Symposium on Security and Privacy in 2025. The scope was 19,500 employees and ten simulated campaigns across eight months.

In the first month, roughly 10 percent of employees clicked a phishing link. By the eighth, more than half had clicked at least one. Embedded training, the page shown immediately after someone fails a simulation, reduced subsequent clicking by roughly two percentage points. The annual mandated security awareness training that most large employers run showed no significant relationship with susceptibility at all.

The authors’ conclusion is quotable in full: anti-phishing training programs, in their current and commonly deployed forms, are unlikely to offer significant practical value in reducing phishing risks. That is a finding about the products as sold, not a claim that no intervention could work.

The lure decides the click rate, not the workforce

The same UC San Diego Health population produced a 30.8 percent click rate against a fake vacation-policy update and 1.82 percent against a fake Outlook password-update notice. Same employees, same eight months, roughly a seventeenfold difference between two emails.

That single comparison undermines most published simulation metrics, including those reported to boards. Any programme can manufacture a flattering trend by choosing gentler lures over time, or a damning one by doing the reverse, with no change in employee behaviour at all. A click rate measures the email, not the workforce.

It also explains how a simulation can become an incident in its own right. CBS News reported that GoDaddy sent staff a simulated phishing email in December 2020 promising a 650 dollar holiday bonus during the pandemic; roughly 500 employees received it and roughly 500 clicked. The company apologised publicly, saying that some employees were upset and felt it was insensitive, and that it needed to do better and be more sensitive to its employees.

What the vendors publish, and who runs the measurement

Against those two studies sits a much larger body of numbers, all of it published by companies selling the product being measured. KnowBe4 released its 2025 Phishing by Industry Benchmarking Report on 13 May 2025, drawn from 67.7 million simulated phishing tests across 14.5 million users at 62,400 organisations.

KnowBe4 reports a baseline phish-prone percentage of 33.1 percent before training, about 19.9 percent after 90 days and 4.1 percent after twelve months, an 86 percent reduction. Its published sector baselines run from 41.9 percent in healthcare and pharmaceuticals to 36.5 percent in retail and wholesale. KnowBe4 chief executive Stu Sjouwerman is quoted in the release saying the data speaks for itself and that security awareness training truly makes a difference.

The methodological gap is the whole story. That measurement is taken from KnowBe4’s customers on KnowBe4’s own platform, with no control group, no randomisation and obvious survivorship in a customer base that renews when the numbers improve. No independent replication of the 86 percent figure was located, and it and the two-percentage-point figure from UC San Diego Health are not competing estimates of the same quantity.

Proofpoint, which also sells awareness training, publishes findings that cut against its own category. Its 2024 State of the Phish surveyed 7,500 employees and 1,050 security professionals across 15 countries and reports that 68 percent of employees willingly undermined their organisation’s security and that 71 percent admitted a risky action, of whom 96 percent did so knowing the risk. The reasons given were convenience at 44 percent, saving time at 39 percent and urgency at 24 percent. If almost everyone who takes the risk already understands it, the deficit awareness training addresses is not the binding constraint.

An attention problem rather than a knowledge one

A 2024 follow-up at ACM CCS by Lain, Tarek Jost, Sinisa Matetic, Kostiainen and Capkun set out to find what benefit embedded training does deliver, and concluded that its effectiveness comes from its nudging effect, the periodic reminder of the threat, rather than from content that is rarely consumed because employees lack the time and rate its usefulness low.

Two of their results should change how programmes are run. Delaying the training page to the following day was as effective as showing it the instant someone fails, which removes the justification for the industry norm of immediate confrontation. Rewards and incentives did not improve secure behaviour. Their framing is that phishing is an attention problem rather than a knowledge one, even among the most susceptible employees, and that enforcing training therefore does not help.

The engagement measurements from UC San Diego Health show the same mechanism from the other end. Around 75 percent of users spent one minute or less on the training page, and roughly a third closed it immediately without engaging at all. A control whose content is not consumed cannot be evaluated on the quality of its content.

Reporting was the thing that worked

The ETH Zurich study did find one intervention that held up, and it was not training. Participants submitted 14,401 reports of suspicious email, of which 11,035 were the study’s own simulations, and 4,260 of the 14,733 participants reported at least once.

Speed was the useful property. About 10 percent of reports arrived within five minutes of the message being sent, 20 percent within fifteen minutes and 30 to 40 percent within thirty. The paper claims to be the first to demonstrate that using employees as a collective phishing detection mechanism is practical at large-organisation scale with an acceptable operational load, and engagement with the reporting button was sustained across the full fifteen months, which the training effects were not.

None of this makes simulations worthless; it makes their usual justification unsupported. Run to reduce click rates, they have two large independent studies against them. Run to build and measure a reporting channel, they have evidence behind them. Those are different programmes with different metrics.

Sources: arXiv (Lain, Kostiainen and Capkun, IEEE S&P 2022) · UC San Diego Today (Ho et al., IEEE S&P 2025) · arXiv (Lain et al., ACM CCS 2024) · KnowBe4 · Proofpoint