We recently did a phishing campaign for a large client. One major difference between phishing and regular penetration testing is that instead of testing IT equipment, software and/or configurations, we are actually testing human behaviour. This means that whenever a phishing email is successful, the reason is that someone (and not something) was at fault, often because of lack of security awareness or training.
When performing a phishing test, employees might interpret this as a way for the company to deceive them by sending them fake emails. Of course, in this case, the objective of the phishing email is to provide focused training for those who need it the most and not to blame the individuals.
A solution to that problem is to try to make results as anonymous as possible. For example, never record Personally Identifiable Information (PII), regroup the results in large groups so that it is impossible to retrace which phishing email was open and which individual actually click on a link. With that tactic, we can provide relatively anonymous data, but we are still able to know which group/department/location/etc. is most at risk . Additionally, using this approach, even OKIOK does not hold the PII information so there is even less risk of the information getting out. The client will never be able to know who exactly was “phished” (unless every single person in a group was “phished”).
This solution is often praised by the HR departments because it respects the privacy of their employees. But what is the impact of having anonymous data?
One of the problems with that approach is that we lose a lot of granularity. The larger the groups, the more anonymous, but also, the less granular the information we get is. Having a few individuals in a specific group perform poorly in a phishing test can cause the whole group to look bad. Another problem with anonymous data is that it is impossible to know if multiple events were caused by a single individuals clicking on the same link multiple times or by several individuals each clicking on the link a single time. This situation can creates false positives and affects the result of the analysis.
The question now is: “is that trade-off worth it?” The answer to that question may vary depending on who you ask it to. Some would say it is acceptable because it will prevent possible HR nightmares but they will have to target larger groups when performing additional security training and awareness. The results represent more of an estimate of the actual situation than a precise result. Other would just prefer to have the exact data and a more precise result in order to give targeted trainings.
In my opinion, having all the data is preferable, of course, if policies, laws and regulations allow it. This way, it is possible to ensure that results are 100% false-positive free, reliable and representative of the actual situation. In the case where it is a requirement to have tests performed anonymously, the Phishing Team at OKIOK developed a series of statistical analysis methods that correct the majority of the errors and false-positives that are usually generated by such a method.
In the end, it’s all about trade-off. Do you want 100% precise data while putting employees privacy at a certain risk or would you rather have estimates and overviews but make sure everyone is happy?