Message from British Rowing Chair on the Ranking Points Index
Hear from Chair, Mark Davies ahead of the launch of the SAS Ranking Points Index later today
(c) Nick Middleton
Read the technical paper relating to the data analysis that SAS has carried out following the introduction of the Competition Framework last April.
Message from Mark Davies, Chair of British Rowing.
Today, we are launching SAS’s naming rights of the Ranking Points Index (RPI) points system. Given that there has been a lot of chat around RPI, and given that I committed at the AGM in October 2018 to look at it, I wanted to give you an update on where we have got to with it and why I think that SAS’s formal backing of it at this point is a very positive thing.
For the immediate naysayers who fear that what follows is, therefore, a corporate whitewash, I’d like to stress at the outset a number of things. First is that I have no skin in this game, because the decision to introduce RPI was made before I came in as Chair. Second, that my first reaction to it was negative, because on my first day in the role a friend forwarded me media coverage that criticised the new system. And third, that part of my role as Chair of British Rowing is to let the executive team know when members point out things that they feel are wrong, so that they can either correct problems or explain their thinking and, on this topic, there has been plenty of that. I have had a number of people talk to me about RPI, and recently an old friend from College days sent me a number of examples of instances where he had seen surprising results – every one of which I forwarded to the team, who have included it in their analysis of the system. In short, I have not been afraid to push back on this issue (or indeed any other!) and, on this particular topic, I am about as independent in this matter as I can possibly get.
This article, therefore, simply lays out what work has been done on this topic, and what conclusions have been reached. Let me start by briefly re-capping the history, as I understand it.
A new competition framework including the Ranking Points Index was launched in April 2018 after many years of discussion and a pilot phase that had started in late 2016. It replaced the 0-12 point system where crews were sub-divided according to the number of events won, regardless of the experience or quality of the opposition and, although it operated for years, it was widely recognised not to meet the requirements of a wide and diverse racing community. It governed racing members from the very bottom to the very top on a scale that was too small and, as a result, middle tiers were too often grouped either with novice or Olympians.
So, a point on which I think we can all agree is that a better system was needed. But inevitably in a moment of change, there was some debate over what should replace it – and a number of suggestions were made that we should simply have taken other, proven, ranking systems from other areas. But there are a number of reasons why we created a bespoke one, and it is probably worthwhile laying those out.
One complication in creating a ranking system for rowing is that between summer and winter we have very different types of racing. While head races time crews down a course (making comparison relatively easy), the nature of regattas is that crews tend to do just enough to get through to the next round or win. The finish time, if it is even taken, is at least as much a reflection of the losing crew as it is the winning one.
Varying field entry size; differing crew compositions, boat sizes, and weather; and the existence (or not) of water current were additional complications that we sought to address in creating a new system that had as its primary aim a desire to help create closer, fairer and more exciting racing (by grouping crews of approximately the same standard to race each other). But that was not its sole objective. In addition, we also wanted to support head racing, and encourage new members to start and continue racing from early in the season. And, as a third objective, we wanted to allow people who had left the sport to return at a lower level. The old system, where points stuck once they had been awarded, meant returning only at a relatively high standard.
So, the ranking system that was created sought to enable crews to be banded (grouped) in competitions more accurately, and to encourage them to compete more regularly. These objectives could not have been met by adopting an existing system – particularly one that is primarily based on single one-on-one results. Widely-used ranking systems like ELO or GLICKO are particularly suited to very different and more simplistic competitions, such as chess. Although a case has been made by one mathematician that it should be used, plenty of other mathematicians I have spoken to have been rather less supportive of the argument made in its favour.
When the system created was rolled out, there was – as I said in my speech at the AGM – a lot of noise. I get collared about RPI on a regular basis and, as I said above, I listen to, and feed back to the team, the points that people make. Through feedback, and through the direct experience of members of the working group (in addition to the considerable amount of consultation that was carried out with the competitions, clubs and members of British Rowing during the initial investigation and discovery phases), concepts and principles have been refined. Changes have been implemented to points awarded at regattas (in order to create a system that is more reflective of the field variations at those competitions), and both the suitability of the framework and the underlying algorithm have been re-assessed. To be clear, this analysis has not been done by me and the staff at head office, it has been done by maths experts and consultants at SAS – a world-class analytics firm.
Since the AGM, using both historical data and a full season of results from 2018 (segregated as regattas and heads), a team comprising a SAS senior data scientist and analytics experts have sought to understand how well the developed ranking system is performing. In particular, they have sought to assess how accurately the results of a race can be predicted, based upon a crew’s ranking points index. That is, they have taken the sum of the individual crew member’s Personal Ranking Index and worked out whether the resulting Crew Ranking Index (CRI) gives a good indicator of likely placing ahead of an event. Using crew composition sourced from BROE2 (British Rowing’s online entry system), incoming competition results, and what subsequent points were awarded, SAS looked to establish trends across the season by club, competition or event. SAS used their Data Management and Visual Analytics platforms to benchmark the existing model, in order to be able to understand what changes, if any, need to be made.
The yardstick that was used to decide whether a CRI provided a good way to band crews accurately was taken from the old points system. That system split crews up into six levels (from Novice, through IM3 to IM1, Snr and Elite), five of which were commonly used however most rowers sat within either Novice or IM3 categories. On that basis, the analysts decided that if the CRI could predict placing to within 20%, that would be a good indicator that racing had been close – the intention of the system being, obviously, to have close racing, rather than to predict the result. If racing was close, then it follows that the system would be a good tool for banding crews accurately.
The results were as follows: the majority of crews racing in 2018 fell within the 20% band, with 70% of all Masters events meeting this target despite the smaller field sizes. Unsurprisingly, head races could be predicted more accurately (to within an average of 17% in 2018), while regattas had some outliers – as you might expect of events where overall placings are more complex (especially in two lane events where potentially high ranking crews may be unplaced after being knocked out in the first round). In the West region – where the number of British Rowing-affiliated competitions is low, and crews have limited exposure to gain points –results were consistently outside the 20%. We are keen to pick this up with the region themselves so that we can discuss ways of improving the situation.
Other issues have been thrown up, such as a problem in the placing of results for two-lane regattas because of BROE2 calculating ranking points allocation correctly but incorrectly calculating the overall position for two-lane competitions (which dominate the English calendar). The positioning calculation is being revised as a result and we are also considering other changes in relation to regatta placings and points. The positioning of regatta ‘zero’ point finishers is under review, so that more information can be gleaned from the results. Going forward, we are likely, in consultation with the National Competitions Committee, to see if there is any additional data we can use for regatta event results.
One very positive thing that is clear is that anomalies reduce over time. It is undoubtedly true that in a single race basis you can get all sorts of odd quirks, and the failure to take the field into account influences the outcome. But rarely is it true that this happens “to the extent that it may be, in practical terms, impossible to overturn that difference in the points through future racing”. An individual’s points are calculated over eight events, with a fade factor attributed to time, so crews that compete at least eight times per year across a full season with both head and regatta entries are seeing improving rank accuracy at each competition – whether their points are high, medium or low. As some individuals enter more than 15 events per year, the anomalies previously seen by hand-picking events either to avoid or gain points are being ruled out. A PRI that is heavily dependent on a single result from a single competition is more likely to throw up incongruous results but those that are normalised over a season much less so. Although there is always the risk that people try to “game” their score, that is true in pretty much any environment and under any system. As someone who once spent a summer successfully pot-hunting around the country, I can tell you with some confidence that it is much harder to game the new system than the old one.
One problem many people raise with me is the issue of heads providing many more points than regattas. This is undeniable but, at the moment, we wish to run with it to see how it averages out over time, for the very simple reason that our secondary objectives come into play: the analysis has confirmed that the loading of points to head races means that people start to race earlier, and that generally, we get closer races earlier in the season.
With the first stage of the review completed, the experts at SAS think that the framework is providing a prediction accuracy that means it does what it is designed to do. That prediction accuracy for regular racers will continue to improve year on year, and a period of stability (which brings with it more data over time, and potentially more insight from regattas) will mean that a crew’s quality will be further understood, assessed and incorporated as we progress. We could potentially weight points at different competitions on the basis of the quality of the entry field, but we will review that at the end of a second season. All of which means that following our review, we will be moving forward with the SAS RPI, rather than abandoning it or making wholesale changes (as some have suggested we should).
Having said which, there is always scope to improve, and it is obviously important that where anomalies exist, we iron them out. Equally, it is crucial that where you feel there is a problem, you know where to go with it. So we have set up a dedicated email to allow you to feed back examples of problems, and would ask you to send any queries to ([email protected]). We will endeavour to respond to your specific questions, and will commit to publishing data on how many queries we get, and how many of them suggest deeper problems. In our experience to date, many of the anomalies that have been thrown up have been issues of implementation rather than of the basic structure of the system but we will be able to explain those and hopefully reduce their number as a result. I would also direct you to the page of FAQs on this subject, in case your example has been raised before or indeed to the mechanism of how points are determined at each competition (Personal Ranking Reference Book). We will see, over time, whether the number of problems reduces, in addition to whether the quality of racing improves. Both will be reviewed.
I am conscious that not all will agree with our plan, as I am that change never occurs without its critics. But let me offer two reasons why moving onto a second full season is a reason to be optimistic.
The first is that another year’s data will allow us to consider further how crews hold their ranking points with time, and confirm the rate of convergence. Patterns are already emerging as to those clubs and crews that have moved up the ranking point tables quickly.
And the second is that we are also currently looking at how well GB Rowing Team, rowers fit within the system, given the allocation for international representation. We expect that they will fit well and, if we are proved right, then we may use movement and positioning of an individual’s points when we consider pathway recruitment. That opens the pathway to a far wider group of people than currently, which is hugely positive for the future of our national team, and for our sport generally.
Let’s see how we look a year from now.
Mark Davies
Chair, British Rowing [Bio]
