[eDebate] Response to Kloster
Tue Nov 21 21:06:07 CST 2006
In the next few days I will weigh in on the mutual preference discussion
by discussing the experiment run at Kentucky and Wake, but while it is
fresh I thought I should comment on Kloster's suggestion for increasing
On one hand I agree with his observation that actual preferences might
be much more complicated than can be fully captured by a single
context-free rating or ranking that a team is permitted to enter for a
judge. It is probably true that a fully "accurate" judge rating would
include data about side and opponent (and to be honest also could be
different in presets, break rounds, elims, etc). But even if I just
encoded the variables Kloster identifies, at a tournament such as Wake a
team could assign 43,884 ratings to complete the ratings task (138
opponents, 159 judges, Aff and Neg).
But my principal concern is not that we couldn't manage the data or
that teams wouldn't be able to intelligently provide them. My real
concern is whether such a system could actually provide more mutuality
once implemented. If the only concern was that a single debate between
team X and team Y needs to find the most mutual judge among 159
possibilities, Kloster's suggestion would be a very good one. I might
discover that most of the mutual 1's aren't really mutual when I
consider side and opponent. Maybe none of them are. But presumably I
could find at least one BEST match among the 159 options.
But the task for judge assignment is not a local maximization problem
but rather a global maximization problem. A successful assignment
algorithm at Wake needs to simultaneously assign judges to 69 different
debates where the "ideal" judge for one debate might also be the ideal
or perhaps the only acceptable judge for a number of other debates as
well. At the same time, among the 159 judges, a significant number of
judges might be obligated to hear the round (meaning that they might
have to trump some more ideal judge placement). Additionally, among the
partial commitment judges, some lower preference judges need to hear
each early round to preserve committed rounds of higher preference
judges for later in the tournament. To complicate matters further,
among the rounds within the pairing some rounds might be break rounds
and hence considered more important for the maximization of pref.
Finally, the goal is to equalize the opportunity for high preference for
teams with the same record across all prelim rounds. In other words,
those teams that have been least fortunate in previous rounds should
have the maximum opportunity to receive their prefs in later rounds.
With the interaction of all of the global constraints, it is perhaps
unrealistic to expect that Kloster's increased mutuality could ever be
successfully balanced against all competing interests.
A much more critical concern impacts the proposal however. While
Kloster suggests a number of factors that might make a putative mutual
assignment less than mutual, he doesn't suggest any that would make them
MORE mutual. Given all of our competitive instincts, presumably equally
well informed teams would discover that they really had NO truly mutual
judges (unless by mutual I meant that the outcome would be totally
unpredictable). Say that I decide that such and such a judge is a
negative hack. I rate them highly on my neg sheet and low on my aff
sheet. But of course, my equally informed opponent rates them high as
neg and low as aff. If the whole tournament has the same knowledge, the
judge is never a match and we're worse off than we are now. At the end
of the day, every time my opponent knows what I know about a judge, the
more likely that I am to believe that the info matters makes it less
likely that the judge matches. So the only judges left are those that I
have no good intel on OR the really rare case where I just conclude that
the judge is absolutely "neutral" in the upcoming debate. But since we
all hate uncertainty, this genuinely "best" alternative might be
rejected by both of us.
Currently, judge preference selections are made without consideration
for the side of the resolution the participant will be debating, nor
participant's opponent. This leads to a situation where for a specific
debate if you look at the preference sheets a judge placement may be
exactly mutual (say a 1-1 placement) but in fact, one team believes
the assigned judge gives an advantage to their opponent.
The solution is to allow judge preference selections to be
differentiated based on which side of the resolution a participant
be on and who their opponent will be.
1) I may rank a judge 5 when I'm aff, but 1 when I'm neg.
2) I might rank a judge 1 in general but 8 if I'm debating against team
3) I might rank a judge 1 if I'm debating team Y on the neg, but 8 if
I'm debating against team Y on the aff.
By allowing consideration of side and opponent the judge placement
system can do a much better job of placing actually mutually preferred
judges in particular debates.
More information about the Mailman