Re: [AGA Contest] judge scoring...

[Erik Olson <erik@thekrib.com>]

Guys, this is all very subjective anyway.  All methods have their
drawbacks, and I guarantee that for every method one judge likes because
it emphasized their score, there's the other judge who is going to feel
that their input was just "thrown out".  

My solution to Roger's problem in 2002 is now to have four or five judges,
not three.  This removes the one-judge-fluke problem.  I would be happy to
try a harmonic mean calculation on last year's scores if I get the time.  
But seriously, if two of the five judges award an entry first place, it's 
going to probably get first.

Scott, if we awarded 1st place based on the number of 1sts awarded, there
would be a 5-way tie for 1st place in some categories.

None of these suggestions is going to solve the problem of entries that 
some judges think are brilliant and some think are awful.  

OK, on to my next delimma.  Judges.  I've now had several suggestions of
people... but they're the *same* people who are judging the AB contest and
now the DFWAPC contest.  They'll probably even be judging a lot of the 
same tanks.  I think we need some diversity here.

  - Erik

On Fri, 21 May 2004, S. Hieber wrote:

> Or you couldjust count first place rankings towards first
> place. Then if no one thought an entry should be first, it
> could not win first.
> 
> There are several ways to go about this. By applying
> rankings rather rather than scores -- essentially the
> durrent method as I understand it, the weighting that the
> judge had in mind is pretty much erased. The kicker in all
> this is that there's no one right way to do it. It's a
> matter of imposing outcomes after the scoring has occurred.
> 
> sh
> --- Roger Miller <roger@spinn.net> wrote:
> > On Friday 21 May 2004 11:50, you wrote:
> > 
> > > Figures -- it's just me. But that is extraordinary when
> > you
> > > think about it. With all the noise each year over how
> > the
> > > judging occurs and how it should have occurred, no one
> > > considers how the scores might actually matter -- they
> > > assume, like voters in federal elections that a vote is
> > a
> > > vote and a winner is a winner and it's only the
> > subjective
> > > valuation of the voter that makes a diff.
> > 
> > I understand that our experience in the 2002 contest was
> > not unique.  In one 
> > case two judges scored an entry so that it was at or near
> > the top of the 
> > category and a third judge scored the same entry so that
> > it was near the 
> > bottom of the category.  When the judge's ranks were
> > averaged the entry was 
> > not in the top 3 -- despite the fact that 2 out of 3
> > judges believed the 
> > entry was one of the best in the category.
> > 
> > I suggested afterwords that the scores should be averaged
> > using a harmonic 
> > mean rather than an arithmetic mean.  That way if one
> > judge decides to give a 
> > low score to an entry the low score is underweighted in
> > the final decision.  
> > I think this is a good choice because -- as Erik pointed
> > out -- the low 
> > scores contain a lot of noise.  Low scores probably
> > should not be equally 
> > weighted, which they are when using an arithmetic mean.  
> > It also might help 
> > avoid having a judge (or judges) feel like a
> > strongly-held opinion has been 
> > ignored.  I think most judges are going to have stronger
> > opinions about the 
> > entries they award high scores than they do about the
> > entry they award low 
> > scores.
> > 
> > Now I'm renewing that suggestion.  I think the judge's
> > ranks should be 
> > combined using an harmonic mean.  That is a detail that
> > can be changed in the 
> > procedure even without changing the documentation.
> > 
> > 
> > Roger
> >   ------------------
> >   To unsubscribe from this list, e-mail
> > majordomo@thekrib.com
> >   with "unsubscribe aga-contest" in the body of the
> > message.
> >   To subscribe to the digest version, add "subscribe
> > aga-contest-digest"
> >   in the same message.
> >   Old messages are available at
> > http://lists.thekrib.com/aga-contest
> >   When asked, log in as username is "aga-contest", and
> > password "second".
> > 
> 
> 
> =====
> -  -   -   -   -   -   -   -
> She Wrote the book on low maintenance aquatic gardening!
> Diana Walstad, author of _Ecology of the Planted Aquarium_
> Meet her at  - The Fifth AGA Annual Convention
> Details & Registration at www.aquatic-gardeners.org & www.gwapa.org
> 
> 
>       
>               
> __________________________________
> Do you Yahoo!?
> Yahoo! Domains ? Claim yours for only $14.70/year
> http://smallbusiness.promotions.yahoo.com/offer 
>   ------------------
>   To unsubscribe from this list, e-mail majordomo@thekrib.com
>   with "unsubscribe aga-contest" in the body of the message.
>   To subscribe to the digest version, add "subscribe aga-contest-digest"
>   in the same message.
>   Old messages are available at http://lists.thekrib.com/aga-contest
>   When asked, log in as username is "aga-contest", and password "second".
> 

-- 
Erik Olson
erik at thekrib dot com

  ------------------
  To unsubscribe from this list, e-mail majordomo@thekrib.com
  with "unsubscribe aga-contest" in the body of the message.
  To subscribe to the digest version, add "subscribe aga-contest-digest"
  in the same message.
  Old messages are available at http://lists.thekrib.com/aga-contest
  When asked, log in as username is "aga-contest", and password "second".