Re: [AGA Contest] judge scoring...

[Roger Miller <roger@spinn.net>]

On Friday 21 May 2004 11:50, you wrote:

> Figures -- it's just me. But that is extraordinary when you
> think about it. With all the noise each year over how the
> judging occurs and how it should have occurred, no one
> considers how the scores might actually matter -- they
> assume, like voters in federal elections that a vote is a
> vote and a winner is a winner and it's only the subjective
> valuation of the voter that makes a diff.

I understand that our experience in the 2002 contest was not unique.  In one 
case two judges scored an entry so that it was at or near the top of the 
category and a third judge scored the same entry so that it was near the 
bottom of the category.  When the judge's ranks were averaged the entry was 
not in the top 3 -- despite the fact that 2 out of 3 judges believed the 
entry was one of the best in the category.

I suggested afterwords that the scores should be averaged using a harmonic 
mean rather than an arithmetic mean.  That way if one judge decides to give a 
low score to an entry the low score is underweighted in the final decision.  
I think this is a good choice because -- as Erik pointed out -- the low 
scores contain a lot of noise.  Low scores probably should not be equally 
weighted, which they are when using an arithmetic mean.   It also might help 
avoid having a judge (or judges) feel like a strongly-held opinion has been 
ignored.  I think most judges are going to have stronger opinions about the 
entries they award high scores than they do about the entry they award low 
scores.

Now I'm renewing that suggestion.  I think the judge's ranks should be 
combined using an harmonic mean.  That is a detail that can be changed in the 
procedure even without changing the documentation.


Roger
  ------------------
  To unsubscribe from this list, e-mail majordomo@thekrib.com
  with "unsubscribe aga-contest" in the body of the message.
  To subscribe to the digest version, add "subscribe aga-contest-digest"
  in the same message.
  Old messages are available at http://lists.thekrib.com/aga-contest
  When asked, log in as username is "aga-contest", and password "second".