data analysis
Based on the data we collect, we perform a variety of analyses each and every month. These analyses are then published in our Detailed Report for that month. These Detailed Reports are distributed only to our respondents; it is our way of saying “thank you” to them for the time they invest each month on this effort. The public website provides the aggregate Index and light commentary; the Detailed Report provides additional information beyond that.
Nevertheless, the content of the Detailed Report is not confidential. Our respondents are welcome to share it with their professional colleagues, which many do.
A review of the analyses that respondents receive each month appears below:
Top risks, and fastest rising risks for the month:
A subset of the above, we identify the top risks, as measured by the risks with the highest sub-index values at the end of the month, and the fastest rising risks. An example of this data table appears below:
Top trending risks:
We look back 3 months and 6 months so as to identify the risks that have increased the most during those periods. To do this, we reset the sub-index value 3 or 6 months prior to a base of 100, and see which ones end up as the top three over this period. An example of what then appears in the Detailed Report is like this: 
Question of the month:
Each month, we ask our respondents a question covering a topical issue that does not figure in the monthly Index calculation. In 2015, to pick a year, Questions of the Month covered encryption, safe harbor, ransomware, IPv6, affordability, change management, CEO involvement, regulation, worst case scenarios, security metrics, and offensive dominance. This is how a Question of the Month appears in a Detailed Report (using September 2016 as our example):
Diffusion index:
We calculate a straightforward “diffusion index” for each of the questions that we ask, in parallel with the calculation of the Purchasing Managers’ Index or the Consumer Confidence Index.
Diffusion indices are highly correlated to monthly increases in our sub-index values, in fact they would be identical but for our weighting scheme (which is explained in the calculation methodology for the sub-index, which is here).
.
Alpha-beta separation:
This is probably the most sophisticated of the statistical analysis we perform for the monthly report. We split the month’s change in each sub-index into the ‘expected’, and an ‘unexpected’ component. This allows us to identify the extent to which each risk has moved higher or lower than what were to be expected given the overall change in the index. But this again is hardly original thought, we stole the concept of alpha and beta separation as used in investment management and applied it to our sub-indices.
The returns earned by an investor in the public markets can be disaggregated and attributed to two sources of risk: the risk from being invested in the market as a whole, called systematic risk, and the risk from being invested in the specific securities that the investor has chosen, called specific, or unique risk. The underlying concept is that all security returns are correlated to the larger market, and just as a rising tide lifts all boats, all securities are affected by the general change in the market to which they belong. Some more so, and some less so: the precise extent determined by the correlation between market returns and the returns of the security in question. This return is called ‘beta’. If we subtract this ‘beta’ portion of the return from the total return earned by the security, we get the ‘alpha’, or what is uniquely attributable to just that security. Asset managers get rewarded for producing positive alpha by picking the right securities, for the beta is just the return from being in the markets.
We extended this idea to the sub-index values, and calculate the ‘beta’ for each sub-index against the overall index. So if the beta for a certain sub-index is 1.2, that means that based upon the historical correlation between the sub-index and the overall index, a 1% change in the overall index should create a 1.2% change in the value of the sub-index. If the sub-index has risen instead by 1.5%, the 0.3% ‘extra’ is the unexpected rise, or the alpha, that was not anticipated.
We have applied the same idea to our index as follows: the change in the total aggregate index is analogous to total market returns. Over the life of the index, each of the sub-indices has displayed a known correlation to the overall rate of change in the index, which in most cases is positive and often large. Yet each month the rate of change of an individual sub-index’s value is not identical to what can be correlated directly to the overall change in the aggregate index. The difference, or ‘alpha’, represents the surprise element for that month for that sub-index that is not explained by the change in the overall index value.
As an example, if the sub-index for the risk to the organization from adverse media/public perception has traditionally trended with a beta of ~2, we expect it to rise (or fall) twice as fast as the overall index. If however, it rises by 3% in a month where the overall index has risen by 2%, it has actually ‘underperformed’, borrowing investment terminology, by 1%. This analysis allows for outlier sub-index value changes to stand out better than a simpler analysis based on just the absolute value of changes.
For the mathematically minded, here is a fuller explanation:
We model the change in the value of a sub-index as a regression model as follows:
y = beta*x + alpha
where
– y is the rate of change in the value of the sub-index,
– x is the rate of change of the overall index,
– beta is the slope of the regression, and
– alpha is the remainder for the value of y that is not explained by x (technically the summation of the intercept and the error value, the latter in the absence of bias should be a random variable with mean zero)
We know the values of y, beta and x, and based on these can calculate for alpha.
This analysis provides us with the ‘unexpected’ change in the value of a sub-index, allowing us to identify the risks that changed in a way not predicted by their previous correlation to the overall cyber security risk as represented by the aggregate index.
A table of the type below appears in each month’s detailed report.
To sum up, we try to do what we can with the data. If you think something we are doing can be improved or added to, please do get in touch. We respond to everyone who writes to us. Being practitioners, we realize that data analysis has significant limitations, and the world is largely an unpredictable place. As always, past performance is not an indicator of future outcomes. Changes in our world are punctuated by random events that create unpredictable inflection points and discontinuities in what might look in the short term to be definitive and inviolable patterns.
That’s all folks!