[eDebate] Kentucky data

Gary Larson Gary.N.Larson
Thu Oct 15 11:50:58 CDT 2009


The discussion surrounding the 100 point scale, a theoretical 87 point average, and judge variance has been a valuable one.  Inevitably, any change creates a period of disequilibration (which is arguably a good thing in a learning system).  Most of the conversation about variance has focused on anecdotal anomalies where some judges consciously or unconsciously set different means.  The data from Kentucky confirms that these "anomalies" are indeed real and that judges did apparently approach the task with different interpretations of the mean.  The data below arrays the results for each judge in the pool that judged at least one prelim round.
 
But before we conclude too much, let me offer a couple of caveats.  First, the assumption of the comparison (and the one that z-scores are based on), is that every judge being compared viewed a statistically similar sample of all of the rounds in the tournament.  That is demonstrably false.  Second, many of the judges represented judged too small a sample to draw any reliable conclusions about their point assignment practices.  Third, distributions are just that.  A judge who consistently gives 27.5's on the 30-point scale is just as much an outlier as the judge who on average assigns 80's in the distribution below - even though it doesn't LOOK like as big an anomaly.
 
All that said, it really should be a goal that critics who are observing exactly the same event (although that only happens at the NDT) should be assigning comparable scores to that event.  In that case, inter-rater reliability is only achieved if we have some commonly held rubrics for how to reward and punish debate performances (in addition to shared interpretations of the scale) - but I suspect that this is a core issue about which we DON'T have a community consensus.  But we also need to remember that with single judge panels, critics NEVER observe the same events, even if two critics judge exactly the same subset of teams during the course of a tournament.  Every debate is formally an n of 1.  While we can and should agonize about whether scoring systems are better or worse about creating norms while preserving discrimination and necessary variability, there is no such thing as the Holy Grail. 
 
Avg    Std   n
 
75.75 3.30 4
77.88 4.63 16
78.94 4.28 16
79.25 4.09 12
79.38 6.48 16
79.50 5.04 8
79.50 5.15 8
80.13 3.05 16
80.50 2.94 12
80.54 3.78 24
80.58 4.56 12
81.00 3.66 8
81.00 1.83 4
81.50 6.45 16
81.75 6.85 16
81.94 4.46 16
82.50 2.67 8
82.75 2.55 8
83.33 3.52 12
83.50 3.96 8
83.75 6.42 24
83.75 4.69 20
83.75 2.05 16
83.88 3.05 16
83.88 3.76 8
84.13 7.55 8
84.25 2.02 16
84.25 1.16 8
84.29 4.03 24
84.50 2.53 16
84.50 8.72 12
84.63 4.90 8
84.63 2.97 8
84.75 5.17 16
84.75 0.71 8
84.87 4.91 15
85.00 0.00 4
85.08 3.60 12
85.29 6.02 24
85.31 5.65 16
85.32 2.88 28
85.45 4.95 20
85.50 5.80 4
85.50 9.47 4
85.58 2.91 12
85.65 4.55 20
85.65 5.32 20
85.88 3.52 16
85.88 1.55 8
86.13 4.32 16
86.25 4.39 28
86.25 1.49 8
86.33 3.14 12
86.42 6.19 12
86.44 7.79 16
86.45 2.21 20
86.50 2.20 8
86.50 1.73 4
86.56 3.54 16
86.63 7.23 8
86.67 1.83 12
86.69 3.36 16
86.75 4.88 16
86.86 3.33 28
86.95 2.50 20
87.00 4.88 28
87.13 3.58 16
87.14 3.68 28
87.15 4.42 20
87.33 3.31 12
87.38 2.72 8
87.42 3.06 12
87.44 3.14 16
87.44 4.72 16
87.50 4.08 16
87.50 1.68 12
87.50 0.58 4
87.58 3.68 12
87.68 2.71 28
87.70 3.46 23
87.75 2.77 24
87.75 3.77 4
87.88 2.94 16
87.92 5.12 24
87.92 4.62 12
87.92 1.16 12
88.00 2.95 12
88.00 2.45 4
88.08 3.37 12
88.13 2.75 8
88.18 3.80 28
88.40 2.33 20
88.45 5.25 20
88.50 1.29 4
88.55 2.63 20
88.65 3.10 20
88.75 1.29 12
88.75 6.90 4
88.83 4.47 12
88.88 3.40 16
88.89 4.83 28
88.92 1.08 12
88.94 3.26 16
88.94 1.69 16
89.00 2.36 28
89.00 2.61 28
89.00 2.60 20
89.00 4.24 8
89.13 2.17 8
89.50 3.20 16
89.55 2.37 20
89.56 2.10 16
89.67 1.67 12
89.89 2.13 28
90.00 0.00 4
90.50 0.58 4
90.83 2.29 12
91.50 3.32 24
92.00 2.16 4
92.75 2.71 20
92.85 3.23 20
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.ndtceda.com/pipermail/edebate/attachments/20091015/c555d61c/attachment.htm 



More information about the Mailman mailing list