[eDebate] MPJ at Kentucky
Wed Sep 12 09:42:40 CDT 2007
As many are aware, significant research has been done on MPJ systems during the past 12 months to ensure that we can simultaneously address all of the competing objectives that impact the placement of judges in debate rounds. Last year at the Kentucky and Wake Forest tournaments we experimented with a 0-100 rating system that included no constraints on the distribution of ratings given by any one team. While the results were intriguing, the lack of constraints encouraged some teams to experiment with highly skewed distributions, either to maximize their advantage or to protect themselves from presumed attempts by others to "game" the system.
As a result, the second half of the year focused on either making modifications to the 4, 6 or 9 category systems or the adoption of ordinal ranking. The experiments culminated in an initiative to collect ordinal ranking data from every participant at the NDT. While the ordinal ranks were not used directly at the tournament, the data permitted extensive testing during the summer. Although the results of that research are still being evaluated, two conclusions can be reported.
First, the availability of a complete data set allowed me to make very significant improvements in the STA judge placement algorithm, providing a greater than 10% improvement over the best results previously available with my software. It is anticipated that this improved algorithm will make a significant difference at tournaments that I run and will eventually be available as an option within TRPC and Jim Hanson STA-XL Plus package as well.
Second, the research confirmed that for large tournaments and particularly for multi-judge panels (elims at all tournaments, prelims at the NDT), ordinal ranking provides a measurable improvement, particularly in terms of mutuality. The discussion about categorical systems last year focused on the anomalies created by within-category vs. between-category skews. For instance, if we have 30 judges per category, a 1-30 pairing is considered more mutual than a 30-31 pairing. In multiple judge panels, this anomaly is multiplied by the very real possibility (confirmed by actual data at the NDT) that the within-category skew can have the same polarity for all three judges. As a result, we could get an straight up AA/AA/AA match that was actually 1-28, 2-29 and 3-30 or off-81 in ordinal terms.
As a result, I have proposed to JW and Roger and they have approved that we use ordinal preferencing at the Clay. The one distinction between the system we will adopt and the data collection effort at the NDT is that we WILL permit ties - just like the system used at CEDA Nats last year.
As the tournament approaches, I will be glad to answer questions regarding the system. I will also be providing tools to facilitate easy ranking of the judges in the pool.
More information about the Mailman