[eDebate] Judge ratings at Kentucky
Fri Sep 29 09:07:53 CDT 2006
As folks prepare to do their judge ratings at Kentucky, I've received a
number of back-channels that worry about how various games that people
might play could affect the outcome. I'm forwarding one such
interaction. I still have a hope that folks will choose to give their
honest assessment of each judge. Regardless of the system we use, if
judge ratings/rankings don't reflect actual judgments but rather some
strategic game, any reporting of outcomes or evaluation of those
outcomes is a fool's errand.
>>You identify one of the dilemmas of the project. While all of us
critique current systems because they arguably don't reflect our
"natural" preferences, most of the e-mails I've gotten ask me to
speculate how one or another strategies to "game" the system are likely
to work. In these cases, some are asking advice on how they can do it
while most others are worried that everyone else will have figured out
how to do it while leaving them on the outside looking in.
I guess it says something about our game - whether or not we're all
willing to admit it.
With respect to your specific query - what happens if you have a
bimodal distribution with say 20-100's and 80-0's? Are you likely to
get all or mostly those judges ranked 100, thereby concentrating your
prefs even better than you could have with the various games played in
the categorical systems?
Perhaps - but at a significant risk. I suspect that you MIGHT get a
higher number of judges in your top 20 than you would if you just did
your normal distribution - but considering that over 75% of judges
assigned are typically 1's or 2's anyway, you might not do much better.
And since you have less mutuality with your opponents, you might
actually more quickly get stuck with one of your 0's.
So here's the risk and reward. You might get a higher percentage of
your top 20 judges. Even there you would have the case that the judges
would best match with one of your opponent's absolutely top judges
(based on the inflated z-scores of your 20 top judges since all others
are 0's). So if you don't think it's bad to get one of your top 20 who
might typically be one of your opponent's top five, the strategy MIGHT
But I can't tell you in advance that you will ALWAYS get your top 20,
and if not you are potentially getting your bottom judge. Part of the
reason I can't predict is that I can't assume that any one team would be
the only one to try to game the distribution. When two teams do exactly
what you propose and they have NO overlap of their top 20 or that judge
isn't available, or judged a team earlier, or is needed in a different
round, the two teams will essentially have a random match of their least
preferred 80. In that case, I can tell you that the hardest to place
judge in the tournament will probably get the call. So there are BIG
I am prepared for this experiment to succeed OR to be a canary that
tells us a great deal about what we really are trying to do with judge
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Mailman