[AGA Contest] various replies, incl judging...

[Erik Olson <erik@thekrib.com>]

I don't have too much time to participate in these debates, especially 
when they're getting a little too personal in nature around the edges.  
But I will endevour to reply to some points while the kid sleeps:

Something to consider here.  In no other competition where the judging is 
done "in secred" have I ever seen the actual methods disclosed on how the
winners are picked.  In most aquarium competitions, the judges aren't even
known.  After seeing all this debate, I'm starting to get the feeling that
I should actually REMOVE any sort of disclosure on the website.
I am now, more than ever, convinced that there's no way to satisfy
everyone.  I think this was the same conclusion that *I* came to in 2000 
when we had this same durned debate, which is why I left what we use now 
in place.

I did some thinking yesterday about harmonic means: If you look at how
they are calculated, it's essentially that each Nth placing is given a
"weight" of 1/N, which means that a singlet first place award on one entry
(with all last places from the other judges) can only be beaten by another
entry with two second places or three third places.  If we have five
judges, then the fifth places are essentially meaningless.  If we had only
four judges, then the fourth place or below is meaningless.  This could be
an OK system if we really emphasize the 1st place ranking, but then it
doesn't address Carlos' issue (a particular point of view which I share)  
of "good tanks even at 4th and 5th place or below".

On the other hand, if we used the raw scores from the judges (which is
calculated to *help* with the selection of suggested candidates for best
in show and suggest tie-breakers), there is (as I mentioned in a previous
e-mail) a horrible bias because some judges use only a portion of the
point scale, let's say between 70 and 95 points -- with a varience of only
25 points, while other judges use the entire range of 0-100 points.  This
means that the judge who uses the entire range has essentially FOUR times
the weighting as the judge who only scores between 70 and 95.  OK, so we
could (and did, I think, play with in 2000) normalize each judge's scores
against their low and high so that we get the same range of 0-100.  But 
that too removes part of the judge's impression... if they thought all the 
tanks were actually mediocre, suddenly there's a fake spread induced 
(really, just as bad as the rankings, but more complicated!).

So in the end, we do the arithmetic mean.  It's, as I said, a compromise.  
If we had some kind of "judging school" beforehand and got everyone to 
come up with a consistent system, great.  I don't think we have that kind 
of time.

I guess the fundamental question is: If you are faced with a category in
which each judge awarded a different tank first place, but no two judges
agreed, but there was a third or fourth place tank which all judges 
thought was good but not as "striking" as their respective first place 
choice, which tank would you think should get the ribbon?  If you think 
it's one of the first-placers, then we should go with harmonic means.  If 
you think it should be the "common ground", then we should stick with what 
we have.

OK, this one is for Phil:  The Best of Show has never been determined by
calculation, except for 2001 when some of the judges were AWOL and the
decision had to be made the day of the presentation.  What actually
happens is that I flip a virtual switch on the site and all the judges get
to see each other's scores and comments, and I mail all the judges a list
of "candidates", based on the best in each category as well as best point
distributions.  The judges then debate amongst themselves and VOTE for a
best of show.  They don't even have to pick one in the list I send.  But
they must agree on a single winner.


Is Diana Walstad an aquascaper?  I didn't get that impression.  Recent
comments notwithstanding, we have tried to get judges who at least know
about or have experience with aquascaping.  Doesn't always turn out to
work, but I think most of the time they ARE pretty good.  I know, for
instance, not to ask Paul Krombholtz or Jan Bastmeijer to judge, because
they are both self-proclaimed "aquascaping idiots".  Christel, though... 
hmm, I think I will e-mail her -- excellent suggestion! 

  - Erik

-- 
Erik Olson
erik at thekrib dot com

  ------------------
  To unsubscribe from this list, e-mail majordomo@thekrib.com
  with "unsubscribe aga-contest" in the body of the message.
  To subscribe to the digest version, add "subscribe aga-contest-digest"
  in the same message.
  Old messages are available at http://lists.thekrib.com/aga-contest
  When asked, log in as username is "aga-contest", and password "second".