[eDebate] rulebreaking doesn't win a lot of debates and there will
Thu Jun 21 12:07:02 CDT 2007
Two quick points to add to Gary's usual excellent analysis:
1. I did a similar analysis on a couple of tournaments I ran, and I found the same thing as Gary. When there was a difference in preferences (most often by one category), the team that preferred the judge LESS was actually very slightly more likely to win the round. This was not statistically significant, and I didn't control for other factors as Gary did, but I would hope that this would contribute to removing the angst involved when you get a judge who is a 2 for you and who you are convinced (though you don't even really know) is a 1 for your opponents.
2. On Gary's other point about clash of civilization debates in prelims:
Gary says: "But the real test is what happens when "clash of civilization" debates occur in prelims, something that is perhaps just as likely as in elims. In back-channels I get some interesting observations about how MPJ presumably impacts this situation. Some teams with statistically less chosen strategies complain that MPJ permits mainstream teams to freeze the most potentially sympathetic judges out of the debate because they can rank them low. Oddly enough, mainstream teams argue just as passionately that MPJ gives alternative perspectives TOO MUCH latitude or protection because debates will be judged by the judges in the middle rather than the more statistically prominent mainstream judges."
Obviously, as Gary indicated, this is an empirical question. I'm not sure his forthcoming analysis of the NDT will get at that particular question, but it would be interesting if it did. The point I want to make is that this is a function of assumptions about the judging pool. The first claim (that MPJ hurts the less-mainstream teams) could very well be true. Let's assume a judging pool and a tournament that is purely ideological. There could be a majority group (for fun, let's call them the Ellises) that makes up 90% of the pool and 90% of the teams in the tournament. The minority group (let's call them the Katsulases) makes up the remaining 10%. The ideological schism is (we assume) so large that nobody ever votes for a team from the opposite group in clash of civilization debates. The result of this would be:
1. The Ellises (remember, they're the majority!) can strike every Katsulas judge in the pool.
2. The Katsulases can only strike about 12% of the Ellis judges.
3. Every Katsulas-Katsulas debate (but there would be very few of them until they begin meeting with regularity at the bottom of the bracket--see below) would be judged by a Katsulas judge.
4. Every Ellis-Ellis debate would be judged by an Ellis judge (presumably chosen based on preferences derived from criteria other than ideology).
5. Every Clash of Civilizations (Ellis-Katsulas) debate would be judged by an Ellis judge. That's right: under this set of assumptions, every "Clash of Civilizations" debate would be judged by one side.
6. The only rounds that Katsulas teams could possibly win would be rounds against other Katsulas teams.
7. The Katsulas judges would not judge much if there was enough cushion in the judging pool.
Obviously, that is not quite the state of the community (though I certainly see elements of that distribution at some tournaments, and I think it is important for the majority to recognize that this is how a good chunk of the minority sees the SQ as functioning). You could construct other scenarios where the skew is less pronounced, where ideology is not the sole factor in judge preference (consider competence as well). And you could then construct a scenario where "Clash of Civilization" debates are likely to be heard by someone who is in the majority group but not strongly so, and who is generally viewed (especially within the majority group) as less skilled (we can call those judges Berches). The majority might pref such as judge as a 4 or 5 because "At least she isn't a wacko." The minority might pref such as judge as a 4 or 5 because:
a) they don't have enough judges in their own camp to fill almost half the sheet;
b) "He's pretty close to random."; that beats having a predictable judge who is in the majority.
Under that scenario, neither side is satisfied. The team from the majority gets what it views as a largely incompetent judge, and the team from the minority has to hope that the majority is right about the judge being incompetent (yes, we really can call those judges Berches!!).
The point is that there are numerous possible scenarios, they depend upon the actual ideological (and other) distributions among the teams in the tournament and the judging pool, and the equilibrium outcome may change from tournament to tournament. A study that examined this might be very valuable, but it would also be difficult to do (measuring ideology on an ad hoc basis is incredibly difficult, especially if there are multiple dimensions of ideology).
Just some thoughts to suggest that this is a very important question, and that it is complicated.
West Virginia University
----- Original Message -----
From: Gary Larson<mailto:Gary.N.Larson at wheaton.edu>
To: edebate at ndtceda.com<mailto:edebate at ndtceda.com>
Sent: Thursday, June 21, 2007 11:18 AM
Subject: [eDebate] rulebreaking doesn't win a lot of debates and there will
While I've been content to observe most of this discussion with polite amusement, I think it is important to address Scott's observation that MPJ has significant impact on how the whole topical vs. anti-topical, policy vs. K, traditional vs. performance etc. divide works itself out.
Scott appears to make a couple of claims. 1) MPJ makes it possible for teams to debate in their own cocoons in prelims without the concern that judges will punish their choice of strategies. 2) In elims, "conflict of civilization" debates are determined by the ideological predilections of whatever type of judges form the ideological majority on the panel. The suggestion was that some arbitrary tweak in the Larson algorithm could/would have changed the outcome of debates at the NDT.
Quite apart from the fact that the NDT does not use the "Larson algorithm" and, in fact, does not rely solely on automated machine placement of judges at all, both of Scott's claims deserve comment.
Regarding prelims, it is true on face that in rounds with two policy teams or two K teams or two performance teams ... the fact that they will likely have a judge sympathetic to their overall approach creates a cozier dynamic than if they were randomly assigned a critic that might not share their approach. But the real test is what happens when "clash of civilization" debates occur in prelims, something that is perhaps just as likely as in elims. In back-channels I get some interesting observations about how MPJ presumably impacts this situation. Some teams with statistically less chosen strategies complain that MPJ permits mainstream teams to freeze the most potentially sympathetic judges out of the debate because they can rank them low. Oddly enough, mainstream teams argue just as passionately that MPJ gives alternative perspectives TOO MUCH latitude or protection because debates will be judged by the judges in the middle rather than the more statistically prominent mainstream judges.
I actually suspect that MPJ has given at least some psychological comfort to teams that for whatever reason don't identify with the mainstream. That might be based on the choice of argumentation strategy, lack of program prestige, ethnicity, etc.
But the key observation - directly in conflict with Scott's prediction - is that MPJ differences appear to have NO statistically significant impact on the outcome of the debate, AT LEAST within the degree of difference permitted in tournaments using either Edwards' or my methodologies (editorial note - it would be a fascinating study to collect preference data at a tournament and then assign judges randomly to see if that would have any impact on outcomes :-). When I attempt to statistically predict the outcome of any individual prelim (or ELIM) debate, only ONE variable successfully explains almost all the variance. When all factors are controlled in a multiple regression, the winner of any individual debate is predicted by which team has the best aggregate record in the other seven prelim debates (and within that POINTS actually proves to be a slightly better predictor than RECORD in those debates where different predictions would be made). Although we worry a lot about side assignment, it has rather low predictive power. MPJ differences not only fail to have statistically predictive power, but somewhat ironically often have a very slightly NEGATIVE correlation with outcome. All things considered, the team that prefers the judge (or the panel) LESS is very slightly MORE likely to win the debate.
All in all, this is a novel concept. At the end of the day debates are likely to be won by the BETTER team (as measured by performance in all rounds at the tournament).
During my vacation this summer, I will be performing a number of analyses on the data from this year's NDT that will rigorously test the claims that have been made about MPJ impact on outcomes. I suspect that it will tell a similar story.
eDebate mailing list
eDebate at www.ndtceda.com<mailto:eDebate at www.ndtceda.com>
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Mailman