Author Topic: "Statistical Draw" after 3 out of 3 rounds with one victor  (Read 1351 times)

Offline Trafalgar

  • Bot Destroyer
  • ***
  • Posts: 122
    • View Profile
"Statistical Draw" after 3 out of 3 rounds with one victor
« on: July 10, 2007, 07:10:36 PM »
To test the bot I'm working on, I've been having it go through the F1 league. Since it was taking forever to do 5 matches with each bot, I restarted the program and changed it to only do 3 rounds.

In the first contest, it won all three rounds - but the program proclaimed that a "Statistical Draw" and began a fourth round. And after that one was won too, it began a fifth.  

(After the fifth round was also a win, it decided it wasn't a draw anymore)

Also, if there's ever a statistical draw requiring an extra round, every contest after that has the increased number of rounds too (This happens even when the number of rounds was originally 5, so you could get stuck at 6+ rounds per contest until you eventually lose a contest, for example).

Here's a screenshot from the Contest Results dialog after the third round:
« Last Edit: July 10, 2007, 07:25:49 PM by Trafalgar »

Offline abyaly

  • Bot Destroyer
  • ***
  • Posts: 363
    • View Profile
"Statistical Draw" after 3 out of 3 rounds with one victor
« Reply #1 on: July 10, 2007, 08:11:53 PM »
I think contest extention with 3 wins is intended behavior. There was an explanation somewhere on these forums, but I don't know where to find it ^^
It basically amounts to the idea that if you flip a coin 3 times and it comes up heads each time, that really isnt good evidence that its an unfair coin. Someone went through and did the math for the statistical draw calculations so that a bot needs pretty good evidence of superiority to be considered the winner.
« Last Edit: July 10, 2007, 08:13:06 PM by abyaly »
Lancre operated on the feudal system, which was to say, everyone feuded all
the time and handed on the fight to their descendants.
        -- (Terry Pratchett, Carpe Jugulum)

Offline Jez

  • Bot Overlord
  • ****
  • Posts: 788
    • View Profile
"Statistical Draw" after 3 out of 3 rounds with one victor
« Reply #2 on: July 11, 2007, 04:27:53 AM »
Here's a post where Nums explains how it's worked out; plus he links it to a Wikipedia entry explaining more about the method.

Abyaly is absolutely right, 5 matches is the minimum you need atm to get a statistically reliable result (to 95% confidence).

As long as you start the league with 5 matches it will keep the minimum amount of matches at 5 even if a pair of bots fight it out for longer. It's matches that aren't league but are under F1 conditions that causes the increased minimum match amount.
(Am so used to seeing that I had completely forgotten about it! Will add it to list though).

If you turn off the graphical display when running leagues you hopefully won't notice the time it takes to run those extra two matches.

Maybe one day we'll be able to adjust the confidence level so we can run quicker tests but that'd be a future project for a programmer.  
« Last Edit: July 11, 2007, 04:34:33 AM by Jez »
If you try and take a cat apart to see how it works, the first thing you have in your hands is a non-working cat.
Douglas Adams