Discussion: Why the world ranking doesn't work?

At the Portugal O Meeting WRE day on 16th February, the men elite field was so large that it was split into M21SE and M21E.
The split was done by world ranking (eg if an athlete had a low overall score – whether just low scores or perhaps had only one or two scoring events – then they were placed on M21E).

M21SE and M21E were both 17.9km with 32 controls, so they were essentially two parallel/gaffled courses with a number of common controls.
This gives a very good opportunity to compare whether the allocation of world ranking points gives fair and balanced outcomes across different courses…..

M21SE was won by Gernot Kerschbaumer in 86:11, Pawlak Bartosz was 12th in 90:28
M21E was won by Alasdair McLeod in 90:28, exactly the same time as Pawlak Bartosz on M21SE.
Alasdair McLeod’s run scores 999 world ranking points.
Pawlak Bartosz’s run scores 1134 world ranking points.
(to score 999 points on M21SE required an athlete to run only 98mins, against the 90mins needed on M21E).
I could understand if the points were slightly different – but a difference of that size seems like nonsense in a system that is supposed to be designed to compare across courses!

There are people on here who understand this better than I do – but I think the following is the main reason?
- World ranking points gained at World Cup and World Championship races are artificially inflated to score higher than other world ranking races
- this means many of the athletes on M21SE will have world ranking scores that are high because some/all of their best scores will have been gained with that artificial inflation
- those artificially inflated scores ‘fool’ the world ranking system into thinking 90mins (or whatever times) is worth more points on 21SE than on 21E.

The world ranking system is used for several things. But it has a primary purpose of ranking athletes against each other, where they have run different courses against different athletes at different times.
And that ranking is then used to position athletes in start groups and start lists, and sometimes to decide who qualifies to run the top elite class.

It might be okay for the small top group of athletes, who have all run World Cups and World Champs against each other, and are pretty much all scoring the same artificially-inflated races (the top places in the rankings are filled by the expected top athletes, after all)
BUT it doesn’t seem like the rules have been designed to make the ranking system work too well for anyone else…..

Or am I missing something?

The rules have been designed in an attempt to increase participation at World Cups.

When the IOF decided to make WOC every year, they killed the World Cup.

Over the last decade it has just been an expensive procession across Scandinavia and Switzerland, with very few events in other countries (excluding championships that are given WC status). Attendance has become increasingly less international, with only the richer nations sending teams.

Since the IOF mauled the WOC format, the only way to get a good start position (and significantly increase your chances of a good result) is by having a good World Ranking. They have then given the World Cup inflated WRE points, meaning that the best way to get a good start position at WOC is to go and run the World Cups that would otherwise been unappealing.

They have essentially adopted the pay-to-win model from mobile games.

Nice analysis of the situation Jon. I'd be curious to see an official response to this.

@ JonX points gained at World ... are artificially inflated

Did it escape your notice that Brazilian ladies are also artificially inflated?

1284!!!

I think it is wrong to make conclusions based on a very unusual race where entry was restricted to runners with _low_ rankings. Normally, a race where all runners are ranked 700 and lower would be some insignificant and not particularly challenging race with mostly local participants and it would be wrong to give too many points to the winner of such a race.

Also, keep in mind that there are country quotas for World Cup races and they are only slightly different between countries. So many top 100 runners from, e.g., Scandinavian countries cannot even participate in World Cups and especially WOCs, yet their rankings are still high. So World Cup inflation cannot be a very significant factor. It could give an advantage to best runners from weaker countries, but I don't think there is any clear evidence of that.

Countries far away from Europe need artificial inflation since it is too costly to race there several times per year in WC events.

Incidentally I was bitterly disappointed with that link on artifically inflated Brazilian women.

I wouldn't dismiss inflation out of hand, in some disciplines inflation has an important role.

I think that was tRicky's point.

tRicky had a point?!? *falls over*

Nah - it was just the cut of his pants

@tricky.

Sorry about that. Did you think it had something to do with the IOF goal of
getting into the olympics?

I think graeme is making a bid for status as an honoury West Australian.

Except that it was pretty darn easy to get into M21SE. All you needed was ~2300pts. Alasdair would have only needed one more (3rd) WRE race to have gotten in.
http://arquivo.pom.pt/2015/Noticias/SUPERELITE.pdf

None of the top10 guys in M21E had run more than two WREs. It's too bad that these guys got fewer points than they could have in SE, given that the courses were essentially the same, but I don't see how you can feel bad for those who weren't fully ranked in the first place. I imagine these guys could have petitioned and gotten into SE as well.

I do agree with Nixon though that the system as a whole is wacky...

Yup wacky system. It means you shouldn't go and do something silly like mispunch when you go to the effort of attending a WRE...

The flaw with this event, as I see it, is that most of the runners in Elite had less than the full compliment of rankings so their points were deflated by running against people who are artificially worse than they actually are. I haven't done the maths, but if the same field went and ran the same race and got the same times today I think they would all get more points.

Just to note, people did petition, or at least enquire (an example being Alasdair) and no one got moved. The organisers published their reasons for not moving people up but I can't find the news item on their page any more.

Is it ever worked?
As I remembered there were always a few unsolved issues within the system. In such cases FOC credibility (when we talk about WRE) is lower than zero. I'm still waiting to hear why FOC didn't make a review after the last season..."There will be a post-season review of the World Ranking..."

@kofols I never paid it much attention, but some people think it used to work and predicted that IOF had broken it.

Meanwhile from the latest FOC minutes ... Special Rules for WOC Qualification – “protected rankings”. Discussion.
FOC supports the idea of protected ranking and sends a revised proposal for Council approval. AL to follow up.

for those of you who don't understand what, the FOC, they're saying, it means they've noticed the system isn't fit for purpose with respect to the WOC start list and they're going to fudge it. It probably helps that the guy who brought it in has quit.

'...and they're going to fudge it." In other words they're going to re-fudge a system that has already been fudged several times for several different purposes.

It seems to me it's time to start with a blank piece of paper with a fresh list of what the primary and secondary purposes of the system are and design it from there.

What the heck - don't go and start throwing logical solutions at a problem like this...

The system is not bad, some things are very good and better than before. The problem in my eyes is execution, interest to solve details, responsibility and promotion. After a year it is still impossible to see points calculation. This is a huge step backward. Maybe they did this on purpose but I just don't like a blackbox when it comes to the top elite level.

It's nothing to do with execution, it is philosophy. There was a good attempt to rank athletes by ability. Then it changed to a points system which ranks athletes by their strategy in attending certain races.

There's certainly a problem with philosophy, but that Brazilian race with the highest four scores ever obtained by Brazilians, by over 100 points, suggests a problem with the execution.

By contrast, in the UK system (same as old WRE system) pretty much everyone's ranking scores are within 50 points of their average.

Yes, philosophy has also changed but execution is still important and was not achieved or it's not very transparent, at least to me. There were mistakes in the past and sometimes FOC made adjustments to the final points calculation which have not been made according to the rules.

To illustrate the deficiencies, which can be classified under the execution.

1) HOW FORMULA WORKS?
I'm still confused about the rules and how formula really works. Is it linear with respect to time or is a statistical?

2.3 At each race, the point differences shall be linear with respect to time; each
minute shall be worth the same number of points from the fastest time down
to the slowest time.
7.4 Finally, calculate the ranking points RP for each competitor at the event as
follows:

2) INSIGHT INTO THE POINTS CALCULATION
We can not see the calculated "weighting factor IP for the race" as it's described in 7.3, 7.3.3, 7.3.4., formula used, average unweighted points of the runners, the number of ranked runners and points calculation. How should I be sure that responsible person (?) made calculation according to the rules? I believe that we're entitled to these information if we are paying a service fee and finance the operation of the IOF and their commissions.

3) MISTAKES AND CORRECTIONS
Mistakes happen and in other sports it's normal that coaches and athletes can be able to double check the final results and make a post-race complaint in case that they feel the points are not distributed according to the rules. We lost the opportunity to examine the points calculation so we're now like a blind sheep flocks and also according to the rules we can't make an official complaint and get the official notice. This was not possible in orienteering since the beginning of WRE and this I would say it's poor implementation and bad execution. Other sports are much more transparent when it comes to mistakes. It gives confidence in the system, if proceedings in respect of adjustment are clear and are also published. It is human to admit that mistakes are possible. Now we must suddenly believe that FOC has superb database and IT support which ensures that the points are 100% correct according to the rules. Even if that were true, I would still prefer to see a system that allows transparent two-way communication.

New rules for WOC start list
http://orienteering.org/updated-special-rules-for-...

It looks very simple and practical. The only thing which might be overlooked is that now WOC results up to 3 years (protected place) will have even greater priority over the current WR list. It's not written but we might have 9 or more different runners with WOC medal at previous years and I believe that in such case WR points are decisive.

On the other hand runners who are currently in WR Top10 and without a medal from the last 3 WOCs (8th Baptiste Rollier, 9th Magne Daehli, 10th William Lind) has lost their position. So at least two of them would have lost their position in group 1 if WOC would be tomorrow.

According to my sports knowledge present achievements should be worth more! I don't know any sport which guarantee TOP starting place based on 3 year old result. Top 10 WR runners should have priority over the WOC medalist if we want that WR retain at least some meaningful role and importance. In such case as we have today the two previous medalist with lowest WR points would be out of the group 1. Fair enough to all.

With the new rules WR has lost the last important use value for elite athletes. It's seems that WR has no luck.

good start position (and significantly increase your chances of a good result) is by having a good World Ranking

Not anymore! Now it is all about 3 year old bronze medal.

New rules relegate E. Bertuks (18th), V. Novikov (33th) and L. Novikov (85th) who are currently outside Top10 WR to WOC starting group 1.

Very unique rules.

You mean promote, not relegate

mistake, yes promote

Good news for the W35s ! In come Rabankyna, Rantanen and Billstam. And potentially Simone, though probably not.

@graeme
Where is possible to see WR position for any runner or all runners from one country who are outside top100? Search doesn't do the job.

Marathon majors will pay six figure appearance fees to start at the front for three year old results.

To me this makes it even harder to earn a good start position based on most recent form.

@kofols http://ranking.orienteering.org/Ranking

@Nixon Yes. The problem with the WR is the overweighting of WOC/WC results. Making WOC results more important is an odd way to fix it.

The problem is that the World Ranking does not work well, and with the qualification races removed from WOC, the IOF needs another way to make sure that the probable winners start towards the end of the start field. If we had a World Cup that worked, it would be possible to make a World Ranking that worked. Now that is tricky. On the other hand I think using the previous two WOCs would have been enough.

Thanks, graeme.

What is also very odd is that IOF asked Tove about the World Ranking and she gave them a first class knockdown answer but nobody really hear or listen what she said.

“Actually I have never cared much about the IOF’s Ranking List. I don’t look at it that often, so I didn’t know that I was on the top before someone told me.” Otherwise, to be the world number 1 is unimportant: “
http://orienteering.org/athlete-of-the-month/

This kind of publicity is very bad for all of us. It shows that IOF doesn't understand that artificial Rankings has no use for anyone. They haven't done their job in 17 years and it is frustrating that almost everyone at IOF are satisfied with the current status. On the other hand they want to have more visibility but in the same time they are willing to throw a promotional tool out of the window.

Being Nr.1 in sport should be recognized and respected. How can they expect that others will respect our sport if they can't do better than this. It's irresponsible to not act based on the answer from Nr.1 athlete.

@probable winners start towards the end of the start field

Why? There is also another option how to accommodate the media and at the same time ensure a fair play for all. First, it is not good that WOC start list play such an important role in elite sport and it's getting too complex with WR, WOC+SEA right to fix the start list. One solution would be to copy the rules from skiing (downhill); eg first starts group 2, then all the medalist from the previous WOCs who are currently not in group 1 or group 2 based on WR, then group 1, group 3 and then all the other groups. I believe that WOC goal should be to minimize the importance of the start list order and to give the recognition to all WR achievements.

The best placing runners would have time to give the first interviews already during the race and if someone from the rear set a good intermediate time it would be at least interesting for arena spectators to focus on him. It is always interesting to see a decent result from someone who is not a favorite. This concept has been tested in many sports.

Is such a starting order in conflict with the WOC media objectives?

Or maybe if they instated qualification races?

While there are certainly valid arguments against weighting WOCs and WCs higher, there are also arguments in favour:

1) Many top runners prepare specifically for WOC and WC races using other WREs as training. It would be wrong to give much weight to events where many athletes do not compete at full strength so the comparisons between them are not entirely objective.

2) WOCs and WCs are generally of a higher quality than other WREs (maps, terrain, general organization, etc.) Comparisons between athletes based on better quality races are more objective and should be weighted higher.

Comparisons between athletes based on better quality races are more objective and should be weighted higher.
This is rubbish.
Agreed, WOC may be a better comparison and more objective. Why then, if you come last at WOC, do you get extra points when the evidence of your mediocrity is more reliable?

An analogy: buying a better thermometer doesn't make it warmer.

(there are sensible ways to do the weighting - e.g. WOC could stay on the system for longer)

Being in first group is over-rated in my view, and even a disadvantage for anyone in the group who makes a small mistake and gets caught. Mironova 1st in Long in 2014 after starting early.

As for starting at the front in a marathon I can't see any advantage, just the danger of going fast too soon. Granted in a big field you might lose a bit of time starting from further back, but I reckon that many marathons are won by runners coming through the field after half-way.

Yes, but it's not just about the advantage to the runner.

The whole scheduling of WOC is twisted and compromised for live TV. Maybe you don't like that, but that's how it is so let's accept that for a moment. It is now essential that the likely contenders are running during the TV window. Using WR failed to deliver Mironova onto live TV. Using old WOC results would also have failed.

Nobody was surprised to see Mironova as a contender, she was, after all, the highest placed finisher from the European Champs. A race where the Gold medallist was also sent out early by WR, despite winning the (voided) qualifier, another blunder not fixed by using antique WOC results.

Use the Wimbledon solution... select a panel of judges to rank the contenders for the particular surface of play.

Funny how discretionary selection actually makes sense, eh? I'm with T.I.L.
Why bother with all this complex calculations when a group of three or four experts could likely pick a group of, say, 20 athletes they expect to win the medals and they would almost never be wrong.

Let's make a test for this year WOC. When is the last day to sign up for WOC and when it is expected that the organizer shall publish the list of competitors?

Still, when you expect that these 20 best should start? At the end, in the middle or at the beginning?