As a computer science major, I always knew security training was important but I had limited it to reducing human errors. I overlooked the treasure trove of data that the training and its metrics had to offer.  

In this digital age, small firms are becoming exposed to sophisticated cyber-attacks, very similar to the unwitting protagonist in the classic film “WarGames” who stumbles onto a military supercomputer, they too must try to understand their playing field and secure themselves. This can be accomplished by monitoring the results of their security training. 

The intersection of cybersecurity and data analytics has never been more critical as firms learn how to navigate this digital minefield. Security awareness training is an important component of a strong defensive plan, like the protagonist’s trip in “WarGames” where he learns about the consequences of hacking and responsible digital behavior. Small businesses too can learn and protect themselves.

But, with ever-changing dangers, how can small firms assess the effectiveness of their security awareness training? This blog post will explore the metrics associated with security training, helping small businesses through the tracking and analysis of user-based training, 

Understanding Security Awareness Training

What is Security Awareness Training?

At its core, security awareness training educates employees about the various cyber threats they face and the best practices to mitigate them. It’s not just about learning to recognize phishing emails; it’s about fostering a culture of security and mindfulness throughout the organization. My previous article goes more in-depth on the different types of cyber threats and the importance of cybersecurity training in the workplace. 

All this to say, security awareness training is imperative.

Why Data Analytics?

Data analytics plays a pivotal role in transforming raw training data into actionable insights. By analyzing the metrics of security awareness training, businesses can pinpoint areas of vulnerability, tailor their training programs, and ultimately fortify their defenses against cyber threats.

The Metrics That Matter 

Unless your company hires a team of security specialists to test your vulnerabilities like is done in the movie “Sneakers”, you don’t really know exactly what or where your vulnerabilities lie. You need to understand your system and understand why your data is important. Here are a few reasons to help you understand just how valuable your data really is:

• Your data:
  • helps you to identify and understand your enterprise vulnerabilities
  • guides your business where to prioritize strengthening security measures
  • allows you to analyze the effectiveness of your organization’s cybersecurity training efforts using measurable outcomes
  • allows your organization to adjust and fine-tune ongoing cybersecurity training programs based on data-driven insights, assuring continual improvement in employee knowledge

1. Click Rates in Simulated Phishing Campaigns

Phishing attacks remain one of the most common entry points for cybercriminals. Simulated phishing campaigns—where employees receive mock phishing emails—serve as a valuable training tool. But what do the click rates in these campaigns reveal?

• Click-Through Rate (CTR): This indicates the percentage of employees who clicked on a simulated phishing link. A high CTR suggests that employees are susceptible to phishing attacks, potentially putting the business at risk. Conversely, a low CTR demonstrates effective training and heightened vigilance.
  • Comparison to Industry Benchmarks and Thresholds
    • Industry benchmarks serve as a yardstick for evaluating CTRs. For instance, a 2022 report revealed that 32.4% of untrained users are likely to fail a phishing test. Comparing an organization’s CTR against such benchmarks can illuminate its relative risk level.

• Risk Assessment: Organizations face a fundamental challenge when it comes to analyzing cybersecurity risks: “How do you know when you should be worried?” While there is no universal threshold, industry norms and regulatory standards offer useful information. Let’s look at how these benchmarks can inform risk assessment:
  • GLBA Safeguard Rule: A Regular Mandate
    • The Gramm-Leach-Bliley Act (GLBA) mandates financial institutions to conduct periodic risk assessments. Specifically, the GLBA Safeguards Rule requires firms to assess the security of consumer information. This evaluation seeks to protect against unauthorized access, which could cause significant harm or inconvenience to customers. Key elements of the GLBA Safeguards Rule include:
  •  Industry Benchmarks 
    • Comparing an organization’s performance against industry benchmarks might yield significant insights. Consider the click-through rates (CTRs) in phishing testing. A 2022 report found that 32.4% of unskilled users failed such exams. Measure your organization’s CTR against this benchmark to determine its relative risk level. If your company frequently outperforms the average CTR for similarly sized firms, it’s time to rethink your training plan.
• Tailoring Training: Analyzing CTR data allows you to tailor training content. Focus on the types of phishing emails that fooled employees the most. Perhaps a deep-dive on identifying social engineering tactics or recognizing suspicious URLs is needed that’s tailored to that employee’s normal daily workflows.

Does this seem helpful so far? If so, keep reading!

A. Using Data Analytics to Mitigate Phishing Risks

Effective cybersecurity awareness training has a direct impact on an organization’s security posture, rather than just ensuring compliance. Businesses acquire significant insights into their strategic decisions by measuring the results of their training activities.

1. Baseline Metrics and Automation
   • Pre-Training Assessments: Establish baseline measures prior to initiating any training program. These may include:
     1. Phishing Click Rate: This is similar to the CTR mentioned in the previous section, determining how frequently employees fall for simulated phishing assaults.
     2. Security Knowledge Score: Evaluate employees’ understanding of security best practices.
     3. Incident Report Rates: Track the number of security incidents reported by employees.
2. Automated Risk Detection: Once these baselines have been established, data analytics can continue to track user behavior. Any deviations from the norm, such as unexpected increases in suspicious link clicks or attachment downloads, cause alarms. These discrepancies merit additional research. For example:
   • High-Risk Days or Times: Detect patterns that correspond to specific days of the week or times when users are more vulnerable.
   • Geographic Variations: Determine whether some places have higher click rates on suspicious links.
   • User Roles and Department: Compare behavior across roles (for example, executives, IT personnel, and general employees).
3. Implementing Tailored Training Methods
   • Record Department-Specific Metrics: Analyze training results by department or team
     1. IT vs Non-IT Staff: Compare the result and their security awareness levels
     2. High-Risk/ Administrator level Departments: Identify areas with vulnerabilities within staff with higher level access.
   • Geographic Variation: Consider the regional differences. Metrics may suggest that specific places require customized information owing to particular threats.
   • Module Effectiveness: If a curriculum or module within your training program consistently results in little to no results, revise it.
   • Frequent Refreshers: All the aforementioned metrics, if correctly collected and used will help you monitor and decide how often refresher courses are needed within your organization.

2. Training Failure Rates and Policy Evaluation

Small business owners confront distinct obstacles. However, these businesses can increase their security defenses by leveraging data-driven insights from training failure rates and policy evaluation as outlined below, 

• Training Completion Rate: How many employees complete their assigned security awareness training? Low completion rates may indicate disengagement or logistical issues. Regular reminders and incentives can boost participation.
• Post-Training Assessments: After training, conduct assessments to gauge knowledge retention. If employees consistently fail these assessments, it’s a red flag. Consider additional training sessions moving to a different method of training.
• Policy Effectiveness: High failure rates warrant policy evaluation. Are your policies clear? Do they align with industry best practices? Metrics can guide policy adjustments. For example, if employees struggle with password hygiene, revisit your password policy.

Remember, these metrics aren’t isolated numbers; they tell a story about your organization’s security posture. Small businesses should actively track and interpret these data points to stay ahead of cyber threats. 

Setting Benchmarks for Small Businesses

1. Defining Realistic Benchmarks

Small businesses often lack the resources of large enterprises, but that doesn’t mean they can’t measure their cybersecurity efforts effectively. Here’s how to set realistic benchmarks:

• Industry Comparisons: Research industry-specific benchmarks for security awareness metrics. For example:
  • Phishing Click Rates: The Anti-Phishing Working Group (APWG) reports average click rates for phishing emails. Compare your business’s rates to these industry standards.
  • Training Completion Rates: Look at similar-sized organizations. What completion rates do they achieve? Aim for parity or improvement.
• Company Culture and Context: Consider your business’s unique context. Factors like company size, industry, and employee demographics influence metrics. A small creative agency might have different benchmarks than a tech startup.

2. Interpreting Phishing Campaign Click Rates

• Thresholds for Concern: While there’s no one-size-fits-all answer, here are some guidelines that might be useful:
  • Low Risk: Click rates below industry averages indicate effective training.
  • Moderate Risk: Slightly above average—consider additional training or targeted modules.
  • High Risk: Well above average—urgent action needed. Revisit training content and reinforce policies.
• Context Matters: Consider the type of phishing campaign. If employees fall for basic scams, address fundamental awareness. If sophisticated spear-phishing succeeds, focus on advanced training.

3. Evaluating Training Failures

• Quantifying Failure: Calculate the percentage of employees who fail post-training assessments. This metric reveals knowledge gaps.
  • Thresholds: Define what constitutes an acceptable failure rate. For instance, if more than 20% of employees consistently fail, it’s time to investigate. After you’ve established this threshold, stick with it for a good amount of time before modifying it, so that you have enough supporting data to determine what the acceptable failure rate should be.
• Policy Adjustments: High failure rates signal policy weaknesses. Examples:
  • Password Policies: If employees struggle with password hygiene, revisit your password policy. Perhaps it’s too complex or lacks clarity.
  • Reporting Procedures: If employees fail to report incidents promptly, review your incident response policy. Also, review your reporting process – it may be that it’s overly burdensome or time-consuming to report incidents.

4. Reinforcing Policies

• Iterative Approach: Metrics drive policy improvements. Regularly assess training outcomes and adjust policies accordingly.
• Feedback Loop: Involve employees. Seek their input on training effectiveness and policy clarity.
• Scenario-Based Training: Use real-world examples like recent breaches to illustrate policy relevance.

Benchmarking Cybersecurity: Aligning with Best Practices

1. Understanding Industry Standards
   1. Industry standards like the CIS Benchmarks and the NIST Cybersecurity Framework provide detailed instructions for safeguarding IT systems and controlling cyber risks. These benchmarks reflect the opinion of global experts and provide a solid framework for defending systems from common risks.
2. The Role of Best Practices
   1. Best practices, supported by top IT security vendors and regulating bodies, serve as the foundation for a resilient cybersecurity strategy. They include several fundamental and advanced cybersecurity processes that, when implemented, can prevent the most prevalent threats.
3. Measuring Cybersecurity Excellence
   1. To gauge cybersecurity operations effectively, organizations can compare their performance against these widely accepted best practices. This comparison not only reveals security gaps but also highlights areas where the business exceeds industry norms, providing a clear path to continuous improvement.

By benchmarking against industry standards and best practices, small businesses can not only safeguard their assets but also gain a competitive edge. By doing so, they are proving their commitment to cybersecurity and enforcing their industry leadership.

Here are five steps to take that will help ensure your organization is able to effectively benchmark your security awareness program:

1. Use Industry Reports: Leverage the wealth of information available in industry reports to compare your organization’s performance against peers.
2. Track Incident Response Metrics: Incident response times and success rates are critical metrics that reflect an organization’s preparedness and agility.
3. Communicate Business Value: Articulate the value of security investments to stakeholders by linking them to business outcomes.

4. Assess Cyber Maturity: Evaluate the maturity of your cybersecurity practices about industry benchmarks.
5. Benchmark Digital Footprint: Monitor and compare your digital footprint with competitors to identify potential vulnerabilities.

Closing Remarks

Cybersecurity and data analytics are critical in this battle to protect and understand one’s information. Security awareness training, similar to the protagonist’s experience in “WarGames,” is essential. Assessing training efficacy is comparable to measuring the success of any educational program, as it involves evaluating whether the intended learning outcomes have been achieved and identifying areas for improvement.  Businesses can better adjust their security by measuring information such as phishing click rates and training failure rates. Benchmarking against industry standards drives continual improvement toward cybersecurity excellence. In today’s digital age, small firms can’t use data-driven strategies to succeed in the face of increasing dangers. Let us continue to strive for cybersecurity excellence while maintaining the trust of our companies and stakeholders.

References

Adams, M. (2019). How to Measure the Effectiveness of Cybersecurity Training and Awareness Programs. In National Institute of Standards and Technology.

Cybersecurity Best Practices. (2019, April 1). CIS.

Flare. (2023, September 22). Five Steps to Effective Security Benchmarking. Flare | Cyber Threat Intel | Digital Risk Protection.

Hodge, S. (2023, September 27). How to Measure and Benchmark Cybersecurity Excellence. CyberRiskInsight.com.

How to Track Phishing Resilience Using a Metrics Matrix. (2022, March 10). IANS.

Security Awareness Metrics to Benchmark Success US | Proofpoint US. (2021, June 8). Proofpoint.

Threat Trends Report: Crypto Malware, Phishing, Trojans and more. (2024, February 1). Cisco Umbrella.