How do you know if your security programme is actually working, or if you are just spending money and time on a false sense of safety? This chapter solves that exact problem by teaching you how to measure, check, and improve a security programme so you can prove its value to your boss, pass your audit, and pass the CISM exam.
Jump to a section
A simple way to picture Security Program Metrics, Evaluation, and Improvement
When a restaurant opens to the public, it first establishes its kitchen procedures and cleaning schedules. This leads to the need for a system that continuously monitors whether those procedures are actually being followed and whether they are effective at preventing food poisoning.
This is exactly what Security Program Metrics, Evaluation, and Improvement does for an organisation's cybersecurity. The restaurant has a health inspection checklist: fridge temperatures must be below 5°C, hand sanitiser must be in use, raw chicken must be stored separately from salad. Each of these checks is a metric – a specific measurement that tells you if a particular safety rule is being followed. The head chef doesn't just look at one temperature once and assume everything is fine. They review the temperature logs daily, taste the food, and watch the kitchen staff. This ongoing checking is the evaluation – is the security programme actually working as designed?
Finally, when the health inspector finds that the walk-in cooler is running warm, the restaurant doesn't just note the problem. It calls a repair technician, buys a new thermometer, and schedules a daily temperature check. That corrective action is the improvement phase. A security programme without metrics, evaluation, and improvement is like a restaurant that never checks its fridge temperatures, never inspects its kitchen, and then acts surprised when it gets shut down for a salmonella outbreak. The metrics tell you the temperature, the evaluation tells you the fridge is broken, and the improvement tells you how to fix it.
A security programme is not a one-time setup. You cannot install a firewall, create a password policy, and then walk away forever. Threats change, the business changes, and people make mistakes. If you never check whether your security controls are still working, you will eventually get breached.
The process of 'Security Program Metrics, Evaluation, and Improvement' is the answer to that problem. It is a continuous cycle, similar to the Plan-Do-Check-Act (PDCA) cycle from quality management. It ensures that the security programme stays effective over time.
Let's start by defining the core terms.
Metrics are numbers or measurements that tell you how well a security control is performing. Think of them as the dashboard in a car. You do not need to read every line of the engine manual, but you do need to look at the speedometer (are you going too fast?), the fuel gauge (are you running out?), and the temperature light (is the engine overheating?). In security, common metrics include the percentage of employees who completed security awareness training, the average time it takes to patch a critical vulnerability, or the number of phishing emails reported by staff last month. Metrics give you a quick, objective view of performance.
Key Performance Indicators (KPIs) are a specific type of metric that measures progress toward a critical business goal. Not every metric is a KPI. For example, 'number of firewall rules' is a metric, but it is not a KPI unless the business goal is to reduce complexity. A KPI for a security programme might be 'percentage of critical systems patched within 48 hours' if the business goal is to prevent ransomware outbreaks.
Key Risk Indicators (KRIs) are metrics that warn you about increasing risk before a bad event happens. For example, if you measure the number of open vulnerabilities on your internet-facing servers, a sudden spike in that number is a KRI. It tells you that the risk of a breach is rising. KRIs are proactive, while KPIs are more about current performance.
Evaluation is the step where you look at the metrics and decide what they mean. It is the difference between reading a number and understanding its significance. For example, if the metric says '87% of employees completed training', the evaluation asks: Is 87% good enough? Which departments are the 13% missing? Is there a reason they skipped it, such as a scheduling conflict or a broken computer? Evaluation turns raw data into actionable insight.
Improvement is what you do after the evaluation. You identify gaps or weaknesses and fix them. This might mean updating a policy, buying a new tool, retraining staff, or changing a process. The key is that improvement is systematic and documented, not just a random fix that might be forgotten next month.
Why does this matter for CISM? The exam tests whether you understand that a security programme is a living, breathing system. It is not enough to have controls in place; you must prove you are watching them and making them better. The exam questions will often set up a scenario where a manager is collecting data but not doing anything with it. The correct answer will be to analyse the data and make a recommendation for improvement.
The process replaces the old approach of 'install and forget'. In the past, companies bought a security tool, ran an annual audit, and called it a day. That does not work anymore. Cyber threats evolve every hour, and regulatory requirements get stricter every year. Continuous monitoring and improvement are now mandatory for compliance frameworks like ISO 27001, NIST, and GDPR.
Finally, there is the concept of 'maturity'. A security programme is not simply 'good' or 'bad'. It exists on a maturity scale. At the lowest level ('Initial'), you have no formal metrics or improvement process. At the highest level ('Optimising'), you have automated dashboards, predictive analytics, and a culture of continuous improvement. CISM wants you to know that the goal is to move up the maturity scale over time, not to achieve perfection overnight.
Identify Business Objectives
Meet with business leaders to understand what is most critical to protect: revenue, reputation, customer data, or compliance. These objectives will determine which security areas you must measure.
Select Relevant Metrics
Choose a small set of metrics (10-15 maximum) that directly map to the business objectives. Each metric must be specific, measurable, achievable, relevant, and time-bound (SMART). Avoid vanity metrics that look good but mean nothing.
Establish Baselines and Targets
Measure your current performance to create a baseline. Then set realistic targets for improvement. For example, if current patching time is 10 days, set a target of 7 days for the next quarter.
Collect and Visualise Data
Use automated tools (e.g., vulnerability scanners, SIEM systems) to collect data regularly. Build a dashboard that displays red, yellow, and green status so that the information is immediately understandable to non-technical stakeholders.
Analyse and Evaluate
Review the dashboard weekly or monthly. Do not just look at colours – ask why a metric is red. Investigate root causes using techniques like the '5 Whys' or fishbone diagrams. Document findings.
Implement Improvement Actions
Based on the evaluation, create a corrective action plan. Assign ownership, set deadlines, and track progress. This might involve updating policies, purchasing new tools, or retraining staff.
Report and Review
Present the results and improvements to the board or steering committee in a quarterly report. Use the report to justify budget requests and demonstrate the value of the security programme. Then repeat the cycle.
Imagine you are the new Information Security Manager at a mid-sized e-commerce company with 200 employees. The company sells custom furniture online and stores customer credit card numbers (which is a massive security risk, by the way). You walk in on your first day and find that nobody is tracking any security metrics. The previous manager kept everything in his head. This is a mess. How do you build a metrics and improvement programme from scratch?
Here is what you actually do, step by step, as an IT professional:
First, you identify what matters most to the business. You meet with the CEO and the head of the finance department. You ask them one question: 'What would hurt the business most if it stopped working?' The CEO says the website, because no website means no sales. The finance head says the payment system, because if customer credit cards get stolen, the company faces huge fines and lawsuits. Now you know your two top priorities: website uptime and payment card security. These become the focus of your metrics.
Next, you choose specific metrics. For website uptime, you pick 'percentage of uptime per month' and 'time to restore service after an outage'. For payment card security, you pick 'number of vulnerability scans with critical findings' and 'time to patch critical vulnerabilities in the payment server'. You set up automated tools to collect these numbers every week.
Then, you build a dashboard. You use a tool like Power BI or a simple spreadsheet. The dashboard shows green, yellow, and red lights. Green means the metric is within the target zone. Yellow means it is close to failing. Red means it is broken. You review this dashboard every Monday morning with your team.
When a metric turns red, you start the evaluation phase. For example, in your third month, the 'time to patch critical vulnerabilities' metric turns red. The target is 48 hours, but the actual average is 120 hours (5 days). You investigate and find out that the IT team is not getting notified when a new patch is released. The email alerts from the vulnerability scanner are going to a shared mailbox that nobody checks. The evaluation reveals a broken process, not a lazy team.
The improvement phase begins. You change the alert settings so that patches are sent directly to the IT team's phones via instant message. You also schedule a mandatory patching window every Thursday at 2 AM so that patches are applied automatically. Three weeks later, the metric turns green again. You document this improvement in a quarterly report to the board, showing that you reduced the average patching time from 120 hours to 14 hours. This report is how you prove the value of your security programme to the executives who control the budget.
In this real-world scenario, the IT professional spends their time on three things:
Collecting data (automatically where possible)
Analysing data to find root causes
Implementing and documenting improvements
They do not spend their time manually writing firewall rules or resetting passwords. They manage the system that ensures those tasks are done correctly and continuously improved.
The CISM exam tests 'Security Program Metrics, Evaluation, and Improvement' heavily because it is the 'management' part of the Certified Information Security Manager. The exam is not asking you to hack a system; it is asking you to run the department. Here is exactly what you need to know.
What they test: - The difference between KPIs and KRIs. This is a classic question. You will get a list of measurements and must identify which ones are leading indicators of risk (KRIs) versus trailing indicators of current performance (KPIs). - The PDCA cycle (Plan-Do-Check-Act) and how it maps to security programme management. For example, 'Which phase of PDCA involves reviewing metrics?' Answer: Check. - The balanced scorecard approach. CISM sometimes asks how to communicate security performance to the board. The answer is usually a balanced scorecard that includes financial, customer, internal process, and learning/growth perspectives. - Maturity models, specifically the Capability Maturity Model Integration (CMMI). You need to know the five levels: Initial, Repeatable, Defined, Managed, and Optimising. Questions will ask which level requires a formal metrics programme (the answer is 'Defined' or higher). - The difference between a metric and a measure. A measure is raw data (e.g., '5 incidents this week'). A metric is a measure compared to a baseline or goal (e.g., '5 incidents this week, which is better than the target of 10').
Traps the exam sets: - They will present a scenario where a manager has lots of data but no evaluation. The correct answer is never 'more data' – it is 'analyse the data and act on it'. - They will try to trick you into thinking that more metrics are always better. The correct answer is that metrics should be aligned with business objectives, and too many metrics cause confusion. - They will give you a KRI that looks like a KPI, or vice versa. For example, 'number of unpatched systems' is a KRI (it warns of rising risk). 'Time to patch a system' is a KPI (it measures current performance). - They will ask you what to do after you find a gap in the security programme. The trap answer is 'fire the responsible person'. The correct answer is 'perform root cause analysis and implement a corrective action plan'.
Key definitions to memorise: - Metric: A quantifiable measurement used to track and assess status. - KPI: A metric tied to a critical business objective. - KRI: A metric that signals increasing risk. - Balanced Scorecard: A strategic planning and management system used to align security activities with business goals. - Maturity Model: A framework for assessing the capability and effectiveness of a process. - Continuous Improvement: The ongoing effort to improve products, services, or processes.
The most common exam question pattern: 'An organisation has implemented a security awareness training programme. Which of the following is the BEST metric to determine if the programme is effective?' The correct answer is not 'number of employees who attended training' (that is a measure of activity). The correct answer is 'percentage reduction in successful phishing attacks' (that is a measure of effectiveness).
CISM loves to test the idea that a metric must measure effectiveness, not just activity. Memorise that distinction, and you will pick up easy points.
A metric is a measurement compared to a goal, not just raw data.
Key Risk Indicators (KRIs) predict rising risk before a breach occurs.
Key Performance Indicators (KPIs) measure how well current controls are performing.
The goal of evaluation is to identify root causes of gaps, not to assign blame.
Continuous improvement is a cycle, not a one-off project.
Metrics must be aligned with business objectives to be meaningful to executives.
A balanced scorecard includes financial, customer, internal process, and learning perspectives.
The highest level of security programme maturity is 'Optimising', which uses automated dashboards and predictive analytics.
These come up on the exam all the time. Here's how to tell them apart.
Key Performance Indicator (KPI)
Measures current effectiveness of a control
Is a lagging or real-time indicator
Example: 'Average time to patch a critical vulnerability'
Key Risk Indicator (KRI)
Signals an increase in risk exposure
Is a leading indicator of potential problems
Example: 'Number of unpatched critical systems this week'
Lagging Indicator
Shows what has already happened
Useful for historical analysis and reporting
Example: 'Number of security incidents last month'
Leading Indicator
Predicts what might happen in the future
Useful for proactive risk management
Example: 'Percentage of employees who failed phishing simulation'
Metric
A measure compared to a baseline or target
Provides context and meaning
Example: 'Incident response time of 45 minutes, exceeding the target of 60 minutes'
Measure
Raw, uncontextualised data
A single number without a comparison
Example: 'Incident response time of 45 minutes' (no target mentioned)
Mistake
A single metric, like 'uptime percentage', is enough to evaluate the entire security programme.
Correct
A security programme must be evaluated using a balanced set of metrics that cover multiple areas: prevention, detection, response, compliance, and awareness. Uptime tells you nothing about whether your data was exfiltrated.
Beginners often look for one 'magic number' that tells them everything is fine. In reality, security is a complex system that requires multiple indicators to see the full picture.
Mistake
If all the metrics are green, the security programme is perfect and no improvement is needed.
Correct
Green metrics only indicate that you are meeting your current targets. The business and threat landscape are constantly changing, so continuous improvement is always necessary, even when everything looks good.
People confuse 'meeting the target' with 'achieving the goal'. In security, the goal is to reduce risk over time, not just hit a static number.
Mistake
Metrics must be identical for every organisation to be valid.
Correct
Metrics must be tailored to the organisation's specific risks, size, industry, and business objectives. A bank cares about transaction fraud rates, while a hospital cares about patient data privacy metrics.
Beginners see frameworks like NIST or ISO and mistakenly think they are a checklist of required metrics. Frameworks provide categories, but you must customise the actual measurements.
Mistake
The most important metric is the number of security incidents that occurred last month.
Correct
While incident count is a metric, leading indicators like vulnerability patching speed or employee reporting rate are often more important because they predict future incidents.
It is natural to focus on what happened (lagging indicators) because they feel concrete. But CISM emphasises proactive management, which relies on leading indicators.
Mistake
Once you set your metrics and targets, you should not change them for at least a year.
Correct
Metrics and targets should be reviewed regularly (quarterly at minimum) and adjusted as risks evolve, new technologies are adopted, or business priorities shift.
People assume that metrics are like laws – permanent. In fast-moving fields like cybersecurity, rigidity leads to irrelevance.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
A KPI (Key Performance Indicator) measures how well a security control is performing right now, like 'average patch time'. A KRI (Key Risk Indicator) measures a condition that signals increasing risk, like 'number of unpatched critical systems'. KPIs are about current performance, KRIs are about future risk.
A good rule of thumb is to have no more than 15 to 20 meaningful metrics. Too many metrics create noise and confuse stakeholders. Focus on metrics that align with your most critical business risks.
No. Different departments have different risk profiles. The finance department might have metrics about bank transaction fraud, while the HR department might have metrics about employee background checks. Tailor metrics to the function.
PDCA stands for Plan-Do-Check-Act. In security metrics, 'Plan' is choosing what to measure, 'Do' is collecting the data, 'Check' is evaluating the metrics against targets, and 'Act' is implementing improvements based on the evaluation.
Operational metrics (like system uptime) should be reviewed daily. Tactical metrics (like patching speed) should be reviewed weekly. Strategic metrics (like overall risk posture) should be reviewed monthly or quarterly.
A balanced scorecard is a report that presents security performance from four perspectives: financial (cost savings), customer (user satisfaction), internal processes (efficiency), and learning/growth (training completion). It helps the board understand security in business terms.
You've finished Security Program Metrics, Evaluation, and Improvement. Continue through the CISM study guide to build a complete picture of the exam.
Done with this chapter?