Discussion: US WOC Team Selection Process

Now that we're on the eve of the 2016 team trials I'd like to bring up a couple of pressing issues with the ESC-defined selection criteria that are of relevance to the trials scoring. The particular issues have to do with the "automatic" selections and with the makeup of the Review Panel which will consider any petitions.

For those of you who don't know, the establishment of this year's selection criteria started at a US Team meeting telecon on Nov 8, 2015 with ESC elections and a selection criteria proposal by the ESC, and ended Jan 27, 2016 when the (largely unchanged) selection criteria were sent to the team mailing list, one month past the rule-mandated deadline for publication. They were finally made public on the federation's team web pages in early April - three months later. A long discussion took place throughout Nov and Dec, in the relatively limited confines of the team mailing list. I personally spent a rather large amount of time and effort developing a set of recommended changes to those selection criteria which were sent to the ESC and team on Dec 22 of last year. You can read that document here (recommended_changes.pdf). There are links to the associated analyses at the top of the doc, which are still online.

A few team members commented, but the only response I received from the ESC was a note from the Chair stating simply that the "ESC office" is closed until after the holidays. The ESC subsequently made an unannounced request for a rules waiver at the Jan OUSA Board meeting, removing all restrictions on WOC team petitions, which the Board approved. The 6 specific points in my recommendation were basically ignored.

Point number two has important ramifications for the number of "automatic" (i.e. non-petition) selections based on the scores at the team trials. Specifically, the selection criteria wording leaves open the possibility of only two (or even fewer) "automatic" selections if the sprint winner is also one of the top two in the scoring list, or if any candidates decline spots on the team, which is uncommon but does happen. Back in Dec I didn't have the stats to show just how probable these scenarios might be, but I've now completed a re-analysis of all the available trials results from 2005-2015, applying the 2016 scoring methodology (all races count, no drops, no ranking scores) to all the years. There's an 80% chance that either the women's or men's sprint winner (or both) will also be one of the top two overall scoreres.

My concern is that the number of automatic, earned selections based directly on the team trials results might slip below 3 (less than half the team for a team of 5), and this would both undermine the significance and credibility of the trials results, and violate OUSA rule G1.7.1 that states the makeup of the team be based PRIMARILY on the results of a team selection competition (the Team Trials).

So last week I decided to make one final attempt to have the wording clarified ahead of the trials. I also wanted to request that the size of the Review Panel be increased from the current four to five members (odd number for tie-break decisions), and that no ESC members serve on the Review Panel. Given that the ESC has itself defined the selection criteria and is standing up the Review Panel, it is not appropriate for the ESC to name its own members to this Review Panel. It is particularly important given than there are several WOC team candidates that are themselves ESC members (Samantha, Giacomo, Cristina). The 2016 selection criteria even specifically says that the "majority" of the RP members not be ESC members, although it doesn't exclude them completely. Further, the conflict of interest issues with the WUOC team selection committee earlier this year make this even more important. The ESC member currently named to the WOC Review Panel (Peggy) was also on that WUOC committee, and has served on the previous four WOC Review Panels (2015, 2014, 2013, and 2012, although she ultimately left the 2014 panel before selections were made). This is entirely too long for one individual to serve on a committee that can subjectively name individuals to the WOC team. With the current subjective-heavy selection environment, I believe the Review Panels should be selected from a wider set of individuals, and at the very least be chosen in a less ad hoc manner.

I sent this letter to the OUSA Board on May 22. The Board responded, and they (and the ESC) have decided to take no action.

Needless to say, I've found this whole process rather distasteful. It began with false claims by the ESC chair of opinions about issues never discussed in the team meeting in the minutes back in Nov, deferred conflict of interest claims sent to the VP comp and ESC chair regarding WOC team candidates serving on the ESC, the Board approving an unannounced rules waiver in Jan without the membership having a chance to contact their reps on the issue, the selection criteria published too late (and provided to the standing team much earlier than the general community), a blatant rules violation by the ESC in the naming of the WUOC team in April, and culminating in a new team email list being formed in January without telling anyone on the old list that this had even taken place. I'm worried that the entire process next year will be limited to only the ESC and the 23 members of the standing team (a team named by the ESC's appointed committee), who themselves elected those very ESC members. All external oversight of this process is being squeezed out. The Board and General Assembly need to take a close look at this whole process.

Sometimes I feel as if I'm the only - or at least the biggest - proponent of team selection via objective selection races. Put the results in the hands of the athletes who are actually doing the hard work to get there, not in the hands of a committee. I can only hope that this year's RP does the right thing - the fair thing - with the scoring list.

By the way, there are some other interesting tidbits that came out of both sets of team trials results analyses, which are here:

Original trials scoring analysis (2005-2015):
team_trials_plots.pdf
team_trials_plots.html (better quality figures)
US_WOC_Team_Petition_Stats.pdf

Trials scoring analysis applying 2016 scoring methodology to the 2005-2015 results:
tt_sprint_score.html

With the 2016 methodology, although the possibility of the sprint winner also being in the top two in the scoring list is effectively a coin toss (two parallel coin tosses), the individuals actually winning the sprint from year to year are not random. Ali Crocker has won the last 4 TT sprints, and was also the top trials scorer in 5 of the past 6 years (and probably would have won all 6 but for having skipped the middle last year). Of the 11 sprints, Ali won 4 and Sam Saeger won 4. Saegers have won 5/11. On the men's side, Ross Smith has 4 TT sprint wins. No one else has more than one. A men's or women's sprint winner has been in the top three overall scorers all 11 years. Eric Bone had 3 consecutive overall wins 2008-2010. Apparently it helps to be named "Alison" - the last 6 trials winners were all named Alison. It is very difficult to objectively qualify for the team at the trials with an MSP or skipped race under the 2016 "sudden death" scenario. Ali came the closest in 2015, placing 6th on the scoring list after skipping the middle but winning decisively both the sprint and long.

Of course its possible to look further back in time, but the 2005-2015 results were the easiest to access online, with relatively consistent formatting. Perhaps Wyatt has results going back further.

The original trials results plots are also interesting. The difference in average slope between the women and men in the scoring lists in particular.

The most worrying result is this one: tt_attendance.gif

This is a US-centric problem: We have simply given the team coach the ability to pick whatever team he or she thinks is best. (We still get after-the-fact discussions of course, since the last couple of places are often not very obvious. :-) )

Given that the team trials start tomorrow, I don't really have the time to thoroughly respond, but I'd like to give a few cursory thoughts to each of your bulleted (and italicized) points:

1. My concern is that the number of automatic, earned selections based directly on the team trials results might slip below 3 (less than half the team for a team of 5), and this would both undermine the significance and credibility of the trials results, and violate OUSA rule G1.7.1 that states the makeup of the team be based PRIMARILY on the results of a team selection competition (the Team Trials).

Quoting my reply to you from December: I disagree with your interpretation that G.1.7.1 implies at least half of the team must come from the top of the trials list. For instance, I think it would be consistent with G.1.7.1 if selection were based on the RP with the mandate to weight trials results most heavily. This would be plausible e.g. if we had a less constrained assignment to start spots. I'd love for other people to weigh in on G.1.7.1 (zzzzzz). I suppose clarification from the ESC would be nice, but I don't see it as necessary. As a team trialer, my view is that if I do not either win the sprint or finish among the top two, I have not definitively made the team, and I'm fine with that.

2. I also wanted to request that the size of the Review Panel be increased from the current four to five members (odd number for tie-break decisions), and that no ESC members serve on the Review Panel.

Why? The current composition of the Review Panel satisfies G.1.7.5. Frankly, the set of people who are qualified, willing, and not on the ESC is painfully small.

3. The ESC member currently named to the WOC Review Panel (Peggy) was also on that WUOC committee, and has served on the previous four WOC Review Panels ... . This is entirely too long for one individual to serve on a committee that can subjectively name individuals to the WOC team.

Why is this "too long" for one individual to serve on a painfully thankless committee? I don't see that it violates any of the bylaws; your protest seems a subjective assessment of the committee. Given the amount of effort involved sifting through data and wrangling team members into replying to e-mails, it's laudable that Peggy (and everyone else involved in our team selection) has been as committed as she has. Also, I think you were the only one who found "conflict of interest issues" with the WUOC selection process.

4. Sometimes I feel as if I'm the only - or at least the biggest - proponent of team selection via objective selection races. Put the results in the hands of the athletes who are actually doing the hard work to get there, not in the hands of a committee. I can only hope that this year's RP does the right thing - the fair thing - with the scoring list.

I think this is generally true and not a bad thing. The objective of team selection by whatever method is to choose the best team for WOC. It's not even obvious what that means - for instance, the difference between an erratic runner with a small probability of a top half of the field result compared to a steady runner who is on average faster but has a smaller chance of a top result. In a perfect world, we would run a statistically significant number of trials races to generate the best estimate of performance; n=1 is a weak approximation of that. The trials might be objective, but they arguably have a higher variance than selection by committee.

I also categorically reject your assertion that objective selection is necessarily "the right thing."

Those in favor of objectivity should argue that the process in expectation will choose the best team; those in favor of subjectivity should argue that the process has less variance. My data is not comprehensive, but my impression is that most of the orienteering world uses more subjective selection than the US.

Rosstopher put it well.

My two cents is that it doesn't matter much how the last member (or two) of the team is selected; their value over a replacement runner is not high. I say this as someone who hasn't even been named an alternate yet - I aspire to that replacement level WOC runner. I haven't run the analysis, but in terms of the US's results at WOC, I don't expect it affects things much at all who finishes in the 80th or 90th percentile. The way to make the WOC team is to be unambiguously excellent, and for many reasons - chiefly the small size of our community, I think - it's not that hard in the US to be among the top few orienteers in the country. This isn't Sweden.

We should be spending much more effort figuring out how to constructively improve orienteering in the US and cultivating excellent and developing our athletes than quibbling about minutia that are ultimately irrelevant.

I couldn't agree more with Ian's last paragraph - let's have discussions on how to improve orienteering in the US and developing the athletes we have, rather than how to select our team for this one week of racing.

+1 Ian

I think you were the only one who found "conflict of interest issues" with the WUOC selection process.

Not the only one. Athletes who speak up publicly against a subjective selection process risk being further negatively affected by that process. It is what us economists might call "selection bias". Then again, we might call it something else much more sinister. At least one economist is on the ESC, and those so educated should put the 'right thing' ahead of 'go along to get along'. JMHO. (Then again, economics is a meta meta game :))

Look, I'm not gonna make a WUOC team. I'm not gonna make a WOC team. I'm not even gonna make a WC team (tho I did manage that feat a couple of times in the past). My son isn't either; (he does sports other than orienteering for reasons I've articulated ad nauseam on these pages in the past). I have no dog in this fight other than economic honesty (is that a phrase? I suppose it is now :))

I'm just someone who cares, and has donated anonymously to the team thru a third party, but the current state of affairs is a total buzzkill, at least for supporters like me. Someone spoke up on the team list, a list that donors and sponsors follow, assuming fandom for their contributions, and the team was moved to another list. Someone spoke up against this move, and the list was made private. Total buzzkill. Those who are supporting and donating to the team want and expect to be close to the team. Not to sound like an economist (sorry, can't help it); but we are paying for that experience, as well as a fair, transparent, and objective elite team selection process. Sort of like the NBA playoffs where if you want to dance, show up for the competition (yeah, I get that that is a hard concept for many readers here to absorb -- so look at it this way -- "you won by not showing up; that is, by cherrypicking the competition and races you wanted to and writing about it, and not alienating the ESC", well, nevermind ;))

Despite my donations, I held my tongue as an anonymous donor when I was called a "stingy curmudgeon" by a OUSA board member several years ago. But, what faces the team (and OUSA in general) is too grave now to remain silent. But, I don't care enough about it anymore other than to post this random message.

So, I suggest you donate to these people instead. They do great work, and I volunteer for them rather than OUSA these days, cuz they are just more worthy and they actually treat their volunteers and donors with respect and transparency and inclusion. This latest ESC train wreck has just yet again reinforced this contrast.

http://chestercountyfoodbank.org/

In any case, good luck to all athletes at WOC, regardless of how they were selected.

I would have confidence that the selection committee members are good people, are aware of their own possible biases and do their best to decide objectively.

If you want to have better, fairer trials and to require attendance, have them in Norway.

I know tough (and therefore questionable) decisions have been taken in the past, but also (unrelated case) forcing someone to run a long with mono and Lyme was not the smartest. Selection committees are necessary.

The key aspect for any selection process is that athletes preparing to try out know what they have to do to get on the team and do well at the event in question. Of course they need to know well in advance so they can train for it.

To me that seems much easier to achieve in an objective system.

Of course everyone wants to be the outstanding athlete in the top one or two places and from the outside it looks like the US petition process is designed to get those people on the team.

The big advantage of the objective system is that it should sort out the 3/4/5 places without the sorts of personality or bias issues that cause so much angst - for both athletes and selectors - in subjective processes.

The different issue of forced running with Lyme seems more like one for a Coach and the athlete rather than a selector.

@randy--

Two quibbles--I'm not sure that is a great example of a Selection Bias--it strikes me more as a "selection bias", but I'll let the economists figure that out.

As one of the very few people who directly witnessed that episode (the USOF board member insult), which really was at least a few years ago, and ironically, right around this time of year... was "stingy" really the modifier? Curmudgeon certainly was part of the epithet... I don't know--my memory fades. I do remember the exact location for what it's worth (the spatial memory of an orienteer, I suppose.)

I guess I'm not sure why you persist in keeping that wound raw... it would easily heal and the scar recede if you let it. After all, orienteering is full of interesting characters, who don't always hew to the norms of more polite society. If I held orienteers to the standards of some circles I travel, I'd long ago have purged the entire concept of the sport from my memory. I'd do the same if I bore a heavy cross. Those can always be put down.

Eddie obviously has a strong sense of fairness and that is to be commended.

I have wondered if it would be better for the selectors to choose the 2 or 3 individuals they definitely want on the team first, and than let the trials fill in the remainder of the slots. Instead, it works the other way around.

With respect to petitioning, it makes sense to keep it for instances of sickness, injury, or other circumstances. However, I do think that anyone wanting to be on the team should be at the trials unless they have a really good excuse and travel expense isn't one of them.

In looking at the US JWOC team, 40% don't reside in the US. I think this is fine but I do think those kids need to come to the trials and prove themselves (which they did). And I don't think it should be any different for the seniors though, it doesn't seem to be an issue since the serious seniors have all been at the trials the last several years anyway.

Relative to spending more time on how to improve, that should be a separate subject but I think the answer is obvious---though difficult. The obvious answer is that we need to spend more time orienteering--a lot, lot, lot more. The difficult part is that we need more maps, a lot more.

Hi Ian, yes you were one of the team members that commented on my recommended changes back in Dec - I did hear you. However I still disagree with your interpretation of rule G1.7.1, which I'll quote here for reference:

G.1.7.1 The makeup of the U.S. Team to the World Orienteering Championships (the WOC Team) is based primarily on the results of a team selection competition (the Team Trials). Each year the ESC will determine the time and place for the selection of the WOC team. The ESC will announce the date at least 4 months prior to the selection race. This competition can be held as part of a National Meet, or it can be a separate event, subject to the regulations of a normal Orienteering USA sanctioned meet, except that only the M-21+ and F-21+ categories are required.

Its states clearly that the primarily part is referring to "(the Team Trials)" It also mentions the announcement of the date of the "selection race," and
that it is subject to the regulations of a normal Orienteering USA sanctioned meet. This is clearly referring to a race or set of races, not a competition involving a combination of the weighted results of the trials and results of other races submitted by non-competing petitioners. The Review Panel is not mentioned at all. Why would we have a rule stating that the team will be primarily selected by the "team selection process?" What other way would there be? Otherwise you'd either leave out the word "primarily" or the words "(the Team trials)," or word it in some other way. I don't accept that rule G.1.7.1 is referring to anything other than the scoring of the team trials themselves, not the subjectively inserted petitions.

In the response I received from the Board on May 28, this point was specifically mentioned, since I had asked about it. Paraphrasing, I was reminded that the team trials are the "primary" means of selection (the word "primary" was in quotes).

Regarding the RP makeup, yes it does currently satisfy the rules, but the reasoning I gave (an odd number is better than an even number, and more is better than less) still stands. You mentioned a lack of qualified willing people to serve on the RP. A point similar to this was made at the team meeting back in November regarding elections to the ESC itself. It was also brought up in the WUOC thread. I'd like to point out that there has been very little advertising of these positions even being open. I've occasionally seen the open ESC positions advertised on the (old) team mailing list. This year the only mention of it was this single line in the meeting agenda sent on Oct 22, just a few days before the election itself:

"1. Terms are up for Peggy, Sam, and Clem. Peggy and Sam are willing to remain on the committee. Clem will decide"

In addition, I have never, ever seen a call for people to serve on any of the Review Panels. I've never even been asked personally about either, although given that I've been trying for the team I would have declined those positions anyways. You cannot justify putting people into potentially conflicted situations by simply saying "we had no other choice," particularly if there has been little or no effort made to actually ask for volunteers to fill these spots. I gather that the WOC RP is filled by Linda (or someone else on the ESC) asking a few people she thinks are qualified until she has a quorum. I'd much prefer to see an open call to fill these positions (both ESC and RP), and by open I mean community-wide, not just on the old or new team mailing lists.

I don't believe I'm the only one who found conflict of interest issues with the WUOC selection process. Actually I don't see how you could argue otherwise. The chair of the ESC named herself and two fellow ESC members to the review panel, which subsequently named yet another ESC member to the WUOC team entirely by petition. All based on selection criteria that the ESC itself established (partially in violation of the federation's own rules of competition), and didn't make public until after registration for the WUOC selection races had already closed. Whether it was or was not, it certainly looks like a conflict, and that's bad enough.

You made a mention of variance in individual performance in regard to the selection process, be it objectively at a limited-sample team trials event or by subjective hand-picking via committee. You said that the trials might be objective, but they arguably have a higher variance than selection by committee. To Randy's point, this *is* the very definition of selection bias. The first sentence in the wikipedia definition tells the story. Hand-picking your measurements to minimize the variance does not lead to a better determination of the desired quantity. Your point about the trials being n=1 is well taken (although its actually 3 races). I agree with that point, it would be better to have a larger sample, And before 2015 we had something to serve that purpose: the ranking score. But the ESC and team have chosen to eliminate that from the trials scoring system. The ESC was adamant about this. The falsification in the Nov team meeting minutes that I mentioned was exactly that - an implication that there was a strong feeling that there should not be a change to include the rankings - despite the fact that there was no discussion of the proposed selection criteria at all, either before or during that meeting. How could there be a strong feeling about something that wasn't even discussed?

In any case, the ranking score that used to be included in the team trials scoring was intended to provide just what you are asking for - some measure of individual performance over a longer period of time. It also served a couple of other purposes: 1) providing a "dropped score" to remove the threat of sudden death at the team trials, freeing athletes from running too conservatively, possibly affecting the results (point four, and discussed in the text of recommended_changes.pdf) , and 2) providing an incentive for athletes to attend more A-meets (incentives paragraph of that doc). I've been tracking the meet attendance and its not encouraging.

Finally, you made this point: "My two cents is it doesn't matter much how the last member (or two) of the team is selected; their value over a replacement runner is not high."

Not to pick on you specifically, because this sentiment has been expressed by others, but I want to speak to this point regarding the "value" of the last few runners selected to the team. I did quite a bit of analysis in Dec specifically looking at this issue (see the "Original trials scoring analysis (2005-2015)" links above). Sure, from a performance point of view the last few runners are all nearly equal, but these are not just pawns in a board game we're talking about here. They are real athletes - real people - who are working hard to earn these spots on the team. It really irks me when both team candidates and even the Review Panels take the attitude that "one is as good as another" at the bottom end. This is especially damaging in a combined objective-subjective selection system.

Every athlete that is inserted by petition onto the team is laying their burden (be that scheduling conflicts, travel problems, injury, exams, sickness, MP in the sprint, life in general...problems that *everyone* has to plan for and deal with) is laying that burden on another athlete - essentially asking them to give up their own spot and year or more of effort to accommodate the other. These decisions are being made subjectively, by opinion of committee, and are not without consequences. Although the rules do allow athletes to petition, that doesn't absolve the petitioners themselves from the conscience of placing this burden on their fellow athletes, no matter what anyone's opinions of the relative abilities of those athletes are.

Nice ideas, Carl.

Regarding more maps and more orienteering, sprint maps may be the easiest way to get both. When I lived in Calgary, the Vancouver club started making lots of sprint maps, holding sprint training camps, and in general doing more sprint events and training, in part due to some super keen university students. Calgary also did weekly street orienteering in the winter. This is a way to get a lot of orienteering in (events are closer) and do much more frequent training (evenings, rather than weekend trips to the forest). The Vancouverites came up with great ways to do sprint map training in pairs without even setting up any markers. Often detailed free city maps are available, or OSM, making the mapping task easier. Forest maps take a lot more work typically, and the best forest terrain can be far for training. Of course, for you and I Carl, the reverse is probably true :-), but for many Americans and Canadians this may be the case.

The Team Trials results (combined time) in the distant past determined the first three team members, and the rest were selected by a selection committee. Sometime in the 90s that was changed due to dissatisfaction with the process. The rules that were established at that point (after extensive discussion) are not the same as what's happening now, but might make for interesting reading for those unfamiliar with them.

This!

it would be better for the selectors to choose the 2 or 3 individuals they definitely want on the team first, and than let the trials fill in the remainder of the slots.

is so obvious. The first 2-3 pick should be uncontentious so you get the best people. The last few picks don't affect the quality of the team one way or the other - so they should be objective.

whether you agree with eddie's (non)selection or not, having a system which causes so much bad feeling is something you can't afford.

Anyone who thinks its none of my business can consider this a comment on the UK selection process.