Director General speaks to Financial Transparency Advisors about building risk assessment capacity

Purpose

I want to articulate some of the practical issues, from a managerial perspective, rather than from a technical perspective, that arise with moving towards an increasingly risk-focused approach to the risk assessment process.

Firstly, we will look at what I will not be covering. The fact that I don’t review a topic, however, doesn’t mean that we shouldn’t discuss it afterwards in conversation.
I also want to spend a good deal of time defining some key terms and activities. In my experience, time spent on clarifying concepts and building consensus around common understandings is rarely wasted within any complex build process.

Finally, I will elaborate some of the complex issues that have faced us and which I suspect will face any regulator trying to build a risk assessment process.

Areas not covered

I will focus on the risk assessment process to aid supervision within a financial regulator for AML purposes. I will not consider further the assessment of terrorist financing and I won’t consider further the kind of risk assessments that might be done in an FIU or some other body with a related mandate.

I’m also not looking at the usage by supervisors of a risk system and how to facilitate creative engagement with it. Before they act, supervisors must be allocated the task of supervising and it is that critical regulatory task of allocating supervisory resources in accordance with an assessment of relative riskiness that I want to focus on.

It is usually the case that this risk assessment process has to be interactive with supervisors’ other work; it needs to be plugged into the workflow management system that guides supervisors through their day-to-day work, so that they are inputting information as well as relying on its outputs. Ease of use by supervisors is an important criterion of success for a good risk assessment process, because if it is easy to use, it is more likely to be used.

I am also speaking about a risk assessment “process” to allocate resources. It is possible to have a process for risk assessment which is manual or spreadsheet-based, but it is more common to build or buy a risk system, made up of selected databases of relevant information and an analysis engine capable of generating one or more ratings or scores attaching to regulated entities. I won’t spend time on that choice.

I will also not examine the relative merits of off-the-shelf and self-build risk systems. On the basis that our experience in Jersey is of a self-build approach, I will address that, although many of the points also apply if you buy and customise an off-the-shelf system, which is a perfectly reasonable approach.

Finally, as to approach, although I will talk about our actual experience in Jersey, a lot of what I want to say is a stylised reflection arising from our experience, rather than a description of what we have achieved. I don’t know to what extent my experiences and thoughts are shared by others, but I suggest that the very question of whether our experiences are similar is itself a good focus for discussion.

Defining key concepts

Now that I have set out what I will not cover, let me turn then to the second of the three aspects of my presentation, which is defining key concepts.
I identify three key concepts that need definition:

• the nature of the system development process
• what the purpose of risk assessment inspections are and what they are meant to support and improve
• what defines progress in each iteration of the cycle of development that we need to go through repeatedly to evolve a good risk assessment system.

The system development process

To achieve an effective risk assessment system, you can think of yourselves, I suggest, as needing to go through a number of phases of development. In many ways, these are similar to classic IT project development phases. Roughly, I outline them as

• an initial design process, which is a collaboration between front line supervisors, AML/CFT specialists and risk system specialists, with business analyst input, intending to feed into developers in the next phase. The purpose of this design process is firstly to clarify a shared view on the purpose of risk assessment and, secondly, to conduct a GAP analysis between current resource allocation and approaches and a risk-focused approach and thirdly to outline, at a high level, the kind of process that would bridge that gap
• build and populate
• experimental first run
• model effectiveness review with supervisors’ feedback
• enhancement and recalibration design process.

You will note that I am assuming that this must be an iterative process. I suspect it is almost impossible to develop a satisfactory risk assessment process from scratch in one round of development. Even if you buy an off-the-shelf system, you will need to do work on calibrating it to your market.

I will focus most of my remarks on the initial design phase of the build-your-own approach. Put simply, if you can get the initial design process right, the likelihood of the remaining phases being manageable rises significantly.

This is hopefully helpful, even if you have already built or bought a risk system, because you are likely to go through that development cycle again and again to improve it and many of the challenges will recur.

The purpose of inspections

There is a slightly surprising challenge at the very beginning of the design process. This is the need to verify that the shared view between all of the risk system building team aligns with the goal or purpose of the inspection.

The particular context for raising this question is the need, when building the system, to have regard for the strong guidance out there from FATF and regional bodies on what should be achieved in the inspection process. There is good and bad in this situation.

Because FATF has moved so powerfully to guide and direct all of us towards a risk-focused approach, we can easily fall into a trap of building a risk assessment system simply to comply with FATF’s guidance.

I suggest you can’t build a good risk system with that kind of compliance focused culture. This is not a criticism of FATF, but an inevitable challenge arising from FATF having had to be quite directive in its guidance.

In Jersey, early on in the building of our approach, we were perhaps vulnerable to the accusation that we were building a risk-focused approach, just to comply with the FATF requirement and in the hope of doing well in a forthcoming MONEYVAL Assessment. However, we switched our articulated purpose from doing well in the assessment to making Jersey an even more inhospitable place for financial crime. While we pay close attention to FATF guidance, our purpose is to fight financial crime, not to get good marks in the assessment.

Being honest, that is a commitment that you need to constantly review and renew. It is all too easy to fall into the trap of only asking ‘what would MONEYVAL like to see’ rather than also asking ‘what will be effective in Jersey in the fight against financial crime?’ Of course, the two answers often lead to the same outcome, but sometimes they do not. Where they seem to lead a regulator in different directions, adopting a certain approach just because FATF seems to say ‘do it that way’ means you never really understand why you are doing what you do. Culturally, that would be a bad outcome; you would be a compliance-focused rather than a risk-focused organisation. But also, as I will cover in a moment, while FATF has plenty of helpful guidance, neither its guidance nor its assessors have the answers to all the questions you will have. Nor do they claim to have or aspire to have all the answers.

If we turn to the mutual evaluation reports (MERs), whether from MONEYVAL, FATF or one of the other regional bodies, what we see is clear descriptions of the negatives, the undesirable outcomes, and, to some degree, we can learn from those negatives – ‘what not to do’. The correct focus on negative results in MERs, however, leaves us with few positives to work with when we are looking to build our risk models.

I would also observe that, while it is feasible to build a shared commitment within a financial regulator for fighting financial crime effectively, it is harder to build a jurisdiction-wide consensus in pursuit of the goal of effectively fighting financial crime.

The inspection cycle issue and risk assessment

To illustrate the challenge, let me take one key point that is the extent of the ambition of your cycle of supervisory inspections. If I simplify a little, I can suggest that a review of the recent MERs will indicate that the high and medium risk entities should receive a regular cycle of inspections. Seeing this common theme in MERs, countries increasingly aspire to get a full cycle done between assessment rounds. 

FATF is aware of the limitations of this emerging practice and I suspect you can expect the body imminently to advise that supervisors should actively consider how to improve or augment the fixed cycle-based approaches, with more timely interventions to address significant changes or escalation of risks levels of regulated entities. That steps up what is demanded of your risk assessment system.

The caveat from FATF is well made, but the strong pressure to commit to a regular cycle for all higher risk rated entities remains.

Should you discuss internally whether this is the right target to aim for? Or should you just accept that this fixed-cycle is the implied target and get on and do it? Or perhaps there are other options as this emerging guidance suggests?

I suggest that what is more important than your particular answer to this is creating a process to review the issue and coming to a consensus on one approach. At least if we know clearly what we are aiming for and why, the development cycle will work better.

There are many views of the role of supervision under which an invariable fixed-cycle would certainly be the wrong target.   If, for example, you thought that we supervise to catch those who have already laundered money or have already put weak money laundering controls in place, you would go looking for them rather than adopting a fixed-cycle approach.

Alternatively, you could argue that we supervise not only, or even primarily, to catch those who have already broken the requirements, but to create an extra incentive for all to comply. If you took this view, a fixed-cycle where each medium or high-risk entity gets inspected could be justified. But, for some, that is not really a risk-based approach.

The first view, despite its common sense appeal, would confuse the role of the regulator with that of the police. It is good to weed this view out.

The second view of trying to create the best incentive for all through a fixed-cycle can be somewhat justified by the common-sense observation that people comply more often or better if they know they might be checked on. Research does bear this out. Humans tend to aspire to conformity to norms.  Perhaps the best way to describe this is that we inspect not just to create a credible fear of capture and punishment, but also to trigger potentially dormant, internal, self-regulatory emotional heuristics of responsibility and duty. This argument would be that there is a particular need to strengthen that sense of duty throughout the industry in circumstances where the financial incentives throughout the industry are mostly in the opposite direction. 

This view might justify a cyclical approach to inspections where risk rating was used to identify the medium and high-risk entities, but under which each entity received a similar level of attention – at least the first time round - and under which no further differentiation within these groups, based on their relative riskiness, was undertaken.  

It is useful to note that the risk ‘event’ in this approach is difficult to define.

This approach is a form of what is sometimes called ‘ex ante supervision’ as opposed to what is sometimes called ‘ex post supervision’ which is the kind of supervision which would have focused on finding actual money laundering or actual weak controls. 

It is an implication of this approach that an inspection can still be successful even if it finds no problems, because the conduct of an inspection has created or reinforced incentives to comply that only actual supervisory engagement can create or reinforce. It is a second implication of this that there is a strong case for an assured periodic cycle of inspections of high impact entities, even when the risk system is telling us that their controls are unlikely to be weak.

So why have I emphasised this point so much? There is scope for quite subtle differences of view, which can become embedded in the design process for a risk system and lead to unsatisfactory results. You can find yourselves with a mixture of risk indicators that point, potentially incoherently, at different risk events.

Put it to the test, I suggest, by creating a forum to articulate a shared vision for the purpose of supervision and to define value added by making that supervision risk-focused.

Defining progress

Let me now come to the third issue of definitions. What is progress? Each time you start a development cycle in evolving your risk system you need a clear target.
For many years now, speaking generally of regulatory risk, financial regulators have been claiming to operate in a risk-focused way. It is my impression that the claims significantly outpace the achievements. I say this not only in relation to the regulation of financial crime but more generally in relation to all financial regulation.

One pattern has been that prior to the building of risk systems, regulators allocated resources in proportion to the size of the entities in their supervisory portfolio. They then built a risk system. As a first iteration, they decided to allocate supervisory resources in accordance with impact and not to try to calculate risk event probability, because it’s too hard. (The language of AML/CFT risk guidance is a little different, but let me use the more standard language for a moment as it suits the point being made.) The result is they allocate resources in accordance with impact and, since size is the main proxy for impact, they end up concluding that the allocation of resources they already had is the best one and was, after all, risk-focused!

I do not want to be too hard on financial regulators, but this does suggest some difficulty in defining clear success targets. Without a doubt, the best regulators have continued to work on their risk systems and the best are now experimenting with big data and data analytics to crack the challenge of estimating probability. However, the challenge is there. The practices are widely varied.

What this does indicate is that the crucial question at any point in time is to define the ambition for the next step along a spectrum from impact assessment using size as a proxy, which is minimally disruptive to existing resource allocation, on to a more nuanced measurement of impact and on further towards the estimation of probability.

I have tried to sketch out a spectrum, which you might use to position yourself and challenge yourself on what the next step should be:

1.  Scale  of risk event with size of the entity as a proxy for scale = inherent risk (i)
2. Scale of risk event with a range of metrics = inherent risk (ii)
3. Scale metrics + simple proxy for probability = inherent risk (iii)
4. Scale metrics + proxy for probability + proxy for effect of controls = residual risk (i)
5. Scale of risk event metrics + proxy for probability + range of metrics for effect of controls = residual risk (ii)
6. Scale metrics + probability metrics + control metrics = residual risk (iii)

This is a lot and it’s quite theoretical. But it seeks to chart a probable path of progress, defined in terms of data complexity, in the potential development of risk systems for regulators.

As a manager, you need to be able to justify each step you take along this path. You should not assume that you need to keep moving up through these levels unless the increasing ambition is working for supervisors. Simple can be powerful. Complex can be expensive and disorientating, although it may also be powerful and enabling.

We in Jersey are somewhere around level three and in the process of defining our next step. In our model, the impact component has two elements. The first is the simple scale element, where we use size metrics like turnover and number of customers to provide a relative view between firms. Some of those scale measures go beyond impact in the AML context and also include jurisdiction-wide impacts, particularly those linked to safeguarding the economy. For example, we also use a size metric for employees, indicating how many people might lose their jobs if the firm was to fail. I will come back to this issue of combining AML and non-AML considerations in a moment.

The second aspect is to form and apply a view on the relative weighting of AML impact as a risk compared to the other risks we regulate. This effectively identifies a proportion of the maximum impact a firm presents, depending on how severe the risk, AML perhaps involves a higher level of total impact compared to loss of confidential data, but a lower level than terrorist financing.

We weight these different considerations based on judgement, but it is a strictly controlled judgement process, based on surveys and internal challenge.
The other issue highlighted by thinking about how you progress through these levels is the need to turn a single metric into either a probability or impact metric. Is having a relatively high number of Politically Exposed Persons (PEPs) an impact metric for AML because the scale of business involving PEPs is higher, or a probability metric as the more PEPs you have the higher the probability of facilitating money laundering? We have applied these metrics as a probability factor (after much circular discussion). You will need to form your own view, just making sure to minimise double counting.

Let me leave aside the potentially exciting world of data analytics where a regulator has not only a wide range of data, but also the capacity to establish key non-linear data relationships in order to do credible estimates of probability. This would be what would take us to Stage 6 of this schema. This is on the horizon. Should it become a reality, I think regulators will start to have serious debates about whether we should continue with hard-wired cycles of inspections covering all firms.

What is really interesting us in Jersey is the move from Step 3 to Step 4. This is the move from measuring inherent risk to also measuring residual risk. Let me talk first about data and then turn to that issue.

Risk system design issues: Four issues

Data

Once you have defined your next-step ambition, the immediate question then is data. The development of good risk assessment tools is hard. Most risks have to be assessed using data about proxies for that risk, rather than data about the kind of risk events which the supervisor is seeking to reduce. Much of the relevant information is hard to turn into data. Much of the data, if it exists, is in the private sector and must be collected. Many of the proxies are inadequate proxies.

So let me be very practical in telling you what we did. The data that we had, or that we routinely collected (the information about what our industry does, how many customers they have, what countries they deal with etc.) was pretty variable across our sectors. In some sectors (for instance banking) we had great data. We had a history of active interaction and engagement and could have a pretty good go at assessing AML risk in these institutions and sectors. In other sectors – less so.  

So we quickly steered our thinking towards some sort of standard improved, industry wide data collection exercise tailored to assess AML risk. For us, this was the first time that Industry was asked to do this kind of tailored, risk-focused reporting. A lot of our industry had not really thought about their client and transactional data in this way before; they didn’t necessarily collect or record or store their own data in a consistent way, when compared to each other. There were differences in definitions, in classifications – and we needed to get them all reporting in the same way so we could essentially compare apples to apples and oranges to oranges.

Our challenges in Jersey in this regard were not unusual. As regulators we know in practice that data management is proving hugely challenging for the whole financial industry. We should always recognise that in setting our ambition.

This is why we felt it was important to give Industry as much notice as possible about the data set we would be collecting. We needed to give Industry plenty of time to collect the data, to build the systems for storing the data and to build the processes for reporting to us.

It proved a good idea to launch this process at the same time as Jersey launched its National Risk Assessment process, in which the industry was also fully involved. Soon thereafter, we also launched a new specialist Financial Crime Inspection team. All this made it clear to Industry that the request for data was part of a step change in Jersey’s approach to fighting financial crime.

Leveraging this understanding and awareness, Industry “getting” the idea that we needed to assess AML risk across our sectors, really helped the success of the data collection exercise. We have achieved very high levels of satisfactory returns with only a few non-financials (around 5% of the total population) struggling.

As to the content of our expanded data requirements, we started with the data set that the World Bank methodology would require and added to that. As we have now built out our risk model, our requirements are likely to change a good deal.

We are now about to begin a new review of the data set to exclude data points we have not found useful and to seek new data that is suitable for the next stage in our development of the model, taking into account the options which emerged from Jersey’s National Risk Assessment.

One regulator/One risk system

A second issue is how integrated the risk assessment process for AML/CFT supervision should be with the regulators’ risk assessment process for other supervisory activities. Jersey like many other jurisdictions has other risks to supervise, we are also a prudential and conduct supervisor. Also like many other regulators, resources are always a challenge and we equally have to apply a risk-based approach to our other supervisory responsibilities.

Firstly let me note the huge advantage of placing the regulation of AML within the financial regulator. Supervisors of financial institutions will almost invariably have much better data, more historic engagement and a better understanding of compliance and culture than just about any other institution within a jurisdiction that might be a candidate for running any aspect of AML regulation. In Jersey we not only have all aspects of AML regulation in the financial regulator, but, quite unusually, we also have the jurisdictional company registration function within the financial regulator. From an anti-money laundering perspective, I would suggest this is an incredibly powerful position to be in.

This does present a challenge for MONEYVAL and FATF assessors, who need to see that the supervisory allocation of resources is appropriately driven by ML/TF risks. On the other hand, FATF recognises there are significant organisational synergies to be achieved by conducting financial crime risk assessments as part of wider risk assessments.

I don’t seek to resolve this issue here, just to note that the Jersey Financial Services Commission currently, as an interim fact of life, is partially adopting both approaches, having a separate AML/CFT system and a system which includes AML/CFT in its broader risk assessments. I will explain a little more about this, by turning to the question of residual risk.

Assessing residual risk

I said earlier that Jersey is somewhere between level 3 and level 4 in terms of its progress in developing its risk model. At some point, and this is the part of the development cycle we are working through, it becomes important to measure residual risk well. I define residual risk as the risk that remains after inspection and remediation.

Inherent risk models face what feels like a fundamental conflict at their core. They don’t account for the control environments that regulatory frameworks mandate, and in which we know many inherently high risk firms operate successfully. Any model that provides for ongoing supervision of firms who manage their risks perfectly well brings with it a call on resources that will always be tough to manage.

Assessing residual risk has two elements: firstly taking into account what you see in terms of controls when you go out to look and secondly how the situation has improved after your inspection report and any consequential improvements in controls. In process terms, factoring either of these in requires the option of ‘reassessment’ to be integrated into the risk model. There is also the option that you may be able to use proxies for the effectiveness of the control environment and therefore factor in an estimation of the effectiveness of the predicted control environment. This can be done even before inspection and as part of the initial assessment.

We know there was some discussion of this point in various FATF supervision fora over the last few years, with very vocal discomfort from some attendees about planning supervisory activity on assessed residual risk at all. That discomfort relates to the point I discussed earlier concerning the purpose of inspections and the fixed-cycle.

In Jersey, we have tried to manage these challenges, conflicts and needs by effectively building the two risk models I mentioned, at least as an interim approach.
We have an inherent financial crime model built on annually collected data, with its primary role being an entity selection tool that ranks, sorts and groups different sectors on the basis of inherent financial crime risk. It is this output that forms the basis of the cycles of examinations carried out by our specialist financial crime examiners.

The second model we have built shares the same starting point as our financial crime model, but takes it and expands it to cover prudential and conduct risks. It also uses an assessment of the controls in place in a firm to assess residual risk. In construction, this integrated model is much more sophisticated, being integrated into a technology platform that our supervisors use in their core supervisory processes. There is also automation built into the model, processing data to identify outliers, and spotting and flagging risk concentrations. It is this model that ultimately we hope will be the dynamic model that will drive much of our risk-based approach to supervision.

Even as we are setting ourselves the ambition of developing our capacity to assess residual risk, we have an open question as to what reliance we will place on that assessment. One option is that inherent risk is used to drive the frequency on an onsite inspection, and residual risk to drive the intensity and/or scope of the inspection. The inherently risky nature of an entity would lead to very frequent inspections, but the nature of the inspection (the breadth and depth of the control testing, etc.) could be less stringent where that entity has a demonstrated good track record of compliance (indicated by the lower residual risk). This approach would perhaps be close to that approach that FATF is moving closer to advocating.

Let me leave that difficult ‘philosophical’ question aside and, secondly, go back to the data issue. Data from firms is good enough to assess inherent risk. But for an assessment of residual risk, we need good compliance data (for instance the actual results of on-site and off-site testing). This is a challenge of a different order: collecting data from ourselves!

There are two potential reasons why you would not have good supervisory data. Firstly, because in relation to low inherent-risk sectors, you adopt a reactive approach to supervision and this means you don’t go on-site to many firms and don’t have the opportunity to collect data or rather to collect sufficient data to apply a robust, data-led “control” or “compliance” score to every firm.

The obvious option in these cases is to look for proxies.  If we have tested a small, random sample of, let’s say, accountants, perhaps we can assume that this is indicative of most accountants and simply apply those “control scores” to all accountants? In Jersey we have not done this, although it has some merit. So we have an outstanding issue of controls assessment in low inherent risk sectors.

Secondly, when we do go out on inspections, we have the issue of whether our inspection methodology is designed to facilitate the collection of comprehensive, comparative data on controls. Inspectors are used to the exercise of judgement rather than data collection; they do not naturally design their inspection to feed a risk system.

Going into this question might lead me into the topic of the next session. So let me leave that one there: just noting that when it comes to measuring residual risk, it is important to have designed a data set and recorded reasoned judgements that assess, for the record, the quality of controls in a regulated firm and then to have an inspection methodology which facilitates the efficient collection of that data and the articulation and challenge of those judgements. The more of these that are collected, the closer you get to having a control data set in relation to which you can benchmark subsequent on-site judgements.

We have also factored in some other data, for example ombudsman complaints which are helpful in this regard and we recognise that there may be many additional data sets that can be added. Some care should be taken with doing that, as weightings can be misjudged easily and throw off the system for its balanced judgement. 

Risk assessment of non-financial entities

Let me come then to the final area I will cover as part of my reflections on the design phase. I spoke earlier about the challenges involved in combining the AML risk assessment with the assessment of conduct and prudential risks. But even if you do not go down that route, you may still face the challenges of combining the assessment of financial institutions and non-financial businesses and professionals (DNFBPs).

This is less relevant to some of you because the practice in many countries is that the regulation of the non-financial business is done separately. From my perspective this is one of the great strengths of the Jersey approach, because it acts as an opportunity to prevent an internally inconsistent approach emerging to risk appetites with regard to the financial and non-financial sectors. It is my impression that this is a very real problem in many jurisdictions.

But the fact that we have the opportunity to develop a consistent approach to risk across financial and non-financial sectors does not guarantee that we will do so. There are challenges. The core of the challenge is the addition of a lot of non-financial sector entities into a regulatory framework and practice which is designed for the financial sector and which, as we have already discussed, tends to focus on the larger institutions. Non-financial entities will not have the accumulated investment in compliance and risk management functionality and they will tend to be small. This can mean that large populations of DNFBPs can be “lost” in the risk model and supervisory approach, particularly as they are added into a model which is in the process of an iterative development.

You may recall MERs over the last few years containing criticism of supervisors’ risk models being too obsessed or weighted towards size, when size is not necessarily the best indicator of risk. There are also substantially more money laundering and terrorist financing typologies; better and more widely accepted risk indicators in relation to financial institutions than for DNFBPs.

Most of you could explain fairly easily how a bank might be used for terrorist financing; but there is not quite the same level of understanding in relation to accountants, or lawyers, or other DNFBPs.

I said I wouldn’t talk about terrorist financing risk, but I can’t resist the temptation to observe that these problems arise particularly intensively in relation to terrorist financing. There are simply not good terrorist financing risk indicators and there are not sufficient terrorist financing investigations and prosecutions to underpin good risk assessment system development. Without good case examples or typologies, it’s often very difficult to develop meaningful risk indicators or red flags for some of these sectors. In Jersey, we have work to do in this area.

This is one area where my earlier discussion of the purpose of inspections comes into play. If non-financial entities are overwhelmingly small and with weak control frameworks and weak data, is the most reasonable thing to do to decide that they are all lower risk? In resource terms, that might seem like the only way to generate an achievable inspection ambition. 

To compound the issue even further, even if we could develop good indicators for the non-financial sector, we would then face the difficulty of comparing the results to the results from the financial sector. How do we ensure our risk model sufficiently considers and distinguishes between product risks across very different sectors (for instance a “high risk product” in a law firm vs a “high risk product” in a trust and company services provider vs a “high risk product” in a bank? Where there are separate regulators for accountants, trust companies, lawyers and financial institutions, this important issue is hidden. But it is still there. Even if it is done inadvertently, each jurisdiction is, by default, making relative judgements.

One way many risk systems seem to ‘fix’ this problem is by adding in some additional data fields which push up some of the smaller entities into the high risk category. This can appear to have corrected the problem but it is not clear that it is actually a risk-focused solution or whether designers are just playing with the weightings. Is it better than just randomly adding some small entities into the high-risk category? I confess I have some attraction to this idea of adding an element of non-risk assessed random inspections from the pool of entities, financial and non-financial, that are at risk of being lost in a highly structured, rational approach to risk assessment. 

I won’t try to bottom out this additional issue. For those who are strongly focused on the MER, it is interesting that this point rarely, if ever, comes up in the evaluation reports, perhaps because it is so difficult. The issue here is part of a much bigger issue of what is sometimes called ‘model risk’. Once you build a risk system and come to rely on it, you can miss an obvious problem even if it’s right in front of your eyes, because the risk system is driving your choices. Paradoxically, the more credible the risk system, the higher the chances of you becoming entranced by it.

Conclusion

So let me try to draw some conclusions:

• There is a well-established understanding of the stages in the development of any system which we can apply to the development of risk assessment systems; but we should accept that when it comes to risk systems, development is an iterative process and we will have to go through it repeatedly to evolve a good system
• We can define, I suggest, a path of improvement and map our position on that path and define a next step for ourselves; this allows us to define success at each stage
• FATF provides great guidance on what not to do, but understandably limited guidance on what to do. Supervisors need to take ownership of the underlying issue of fighting financial crime, rather than take a compliance-focused approach to the FATF standards
• The purpose of a risk system is for a contested issue and each organisation should allow itself to have that debate, perhaps multiple times, as it develops its risk system; the critical question is whether the focus is on the risk of money laundering happening or the risk of weak controls
• There are a number of critical issues around integration with non-AML risk systems, data collection (both from entities and, when it comes to measuring residual risk, inspectors) and reconciling risk assessments of financial and non-financial entities, which I have tried to explore a little.

Without doubt, I have not provided all the answers to the questions I have raised. We all have to approach this challenging area of regulation with great humility and recognition that many of the good ideas are scattered across the globe. These are some reflections on our attempts in Jersey to get better and better at this.