Measuring security awareness beyond click rates

TL;DR

• A low click rate feels like good news, but on its own it tells you little. The test email may simply have been easy to spot.
• Measure what people do: how many report, how fast, who keeps making the same mistake and what happens under time pressure.
• Weigh every number against how hard the scenario was. A generic fake email and a targeted spear phishing message are not the same yardstick.
• The most honest measure: does someone still dare to report after clicking? That is where you see whether your culture works.

October is European Cybersecurity Month, running from 1 to 31 October as an initiative of the European Commission with support from ENISA. In many companies that means a phishing test, a presentation and a number on a slide afterwards. Usually that number is the click rate. If it went down, it was a good year.

That is too simple. What security awareness is and how to build a program is already covered in our post on what security awareness is and in the security awareness training guide. This post asks a different question: how do you know it works? In a phishing test we measure reports as well as clicks, and in a security awareness program we pair training with realistic exercises to see what people do under pressure. That gives you numbers you can act on, rather than a click rate alone. These are the measures we use.

Why the click rate alone says too little

The click rate is the share of recipients who clicked the link in a test email. It is a real data point and we do not throw it away. But it depends entirely on the email you sent. A sloppy fake with an odd sender address and three typos almost always produces a low click rate. An email that looks like the message from your leasing company, sent on the day the new cars are allocated, produces a different result.

So a low click rate can mean your people are alert. It can also mean the test was too easy. The number itself does not tell you which. And it says nothing at all about the people who did not click: did they recognize the email and report it, delete it, or never see it because they were off that day? Three different situations, one and the same number.

Reporting rate and time to report

The reporting rate is the share of recipients who pass a suspicious email on to IT or hit the report button. It tells you what the click rate cannot: whether people actively do what you ask of them. Not clicking is passive. Reporting is an action, and that action protects everyone else, because after the first report IT can pull the email from every mailbox.

Time to first report belongs with it. A phishing email reported after three minutes does less damage than one that is noticed half a day later. In a phishing test we measure both: how many people report and how long it takes. If the reporting rate goes up and the time to report goes down, something is really changing, even if the click rate stays flat.

One note from practice: a report button that is hard to find pushes the reporting rate down. Then you do not know whether the problem is your people or your button. We sort that out first.

Repeat mistakes

An organization where different people click in every test has a different problem from one where the same ten people click every time. The first case is noise: everyone has a busy day now and then. The second case is a group that needs extra attention, and there is usually a reason behind it. A role with lots of external email, a shared mailbox, a team that never made it to the training.

That is why we look at results per team and per role, across several test waves. Not to put people in the pillory, but to see where repetition pays off most. Short, repeated practice works better there than another yearly presentation. That is what Swishing is for: sessions of five to ten minutes in which people decide for themselves whether an email is safe, with an explanation right after. Swishing is training, not measurement. The measurement is still the phishing test, and you run it again a few months later.

Behavior under pressure in realistic scenarios

A test email on a quiet Tuesday morning measures something different from a phone call from "the bank" while the quarter is being closed. Attackers pick their moment and play on haste, authority and helpfulness. If you want to know how your people respond then, your scenario has to do the same.

So we do not limit an awareness program to email. We combine training with phishing, quishing, smishing and vishing, with scenarios around AI use, and we test reporting behavior. Afterwards there is a live debrief by a hacker, and a few months later we measure again. What you see then is whether someone made the right call under pressure, and whether they dared to say "I will call you back on the number I know".

The Escape Truck belongs in the same corner. Teams of four to eight work on security puzzles against the clock and go through what they saw with an ethical hacker after every round. It is not a measuring instrument. It does show you who still thinks under pressure and who types in the first code that works.

How hard was the scenario?

Every number above needs context: how hard was the test? A generic email to everyone is a different level from a targeted spear phishing message to accounts payable using the name of a real supplier. In a phishing test we build campaigns in waves, from generic to targeted, and record the difficulty of each wave. Only then can you compare numbers.

A click rate that rises while the scenarios get harder is not a step backwards. A click rate that falls with scenarios of the same difficulty is progress. Without that second data point you are comparing apples with pears. That happens more often than you would think, especially when the same party picks the difficulty and reports the results.

Do people dare to report after a mistake?

This is the measure that is counted least often and says the most. Someone clicks, realizes a second later that it was wrong, and calls IT. Or does not. Real incidents grow out of that second scenario: a click nobody reports, because the person who clicked is afraid of the reaction.

Measuring it is simple: of the people who clicked, how many reported it themselves afterwards? That percentage says more about your culture than any other score. It tells you whether people dare, and daring comes from how you respond to mistakes. An organization that puts clickers on a list ends up with fewer reports and just as many clicks.

To us this is the heart of measuring awareness. A low click rate looks nice on a slide. A high reporting rate after a mistake means an incident gets stopped before it becomes one.

Frequently asked questions

Is a low click rate a bad sign?

No, but it is not automatically a good sign either. A low click rate can mean your people are alert, or that the test email was too easy to spot. Look at the difficulty of the scenario and at the reporting rate before you draw conclusions.

What is a reporting rate?

The share of recipients of a suspicious email who actively report it to IT or through a report button. Together with the time to first report, it is the measure that comes closest to real protective behavior.

Does Swishing replace a phishing test?

No. Swishing is training: short sessions in which people decide for themselves whether an email is safe and get an explanation right away. A phishing test measures what people do in their real mailbox. You use the test to measure and Swishing to improve, and then you measure again.

How often should you measure awareness?

More than once a year, because a yearly measurement says nothing about repeat mistakes or about the effect of training. Work in waves and measure again a few months after each training round, with scenarios of comparable difficulty.

Should you share results per employee?

Preferably not as a ranking. Results per team and per role show where repetition is needed. Use individual numbers only for targeted coaching, never to call someone out, or your reporting rate dries up.

Related services and resources

If you want to measure what your people really do, start with a phishing test (campaigns in waves, clicks and reports measured) inside a security awareness program that links training to realistic exercises and a retest. For repetition between two measurements there is Swishing, and for a day people keep talking about there is the Security Awareness Escape Truck. For background, read our posts on what security awareness is, the security awareness training guide, gamified phishing simulation, what Swishing is and stolen credentials and infostealers, which shows where a missed report can lead.