Code center > Darwinbots3
DB3 Leagues
Peter:
A tournament like would have to repeat itself every time. That would drain cpu resources.
I like some parts of how it was done in the last google AI challenge.
They had a elo rating.
New versions were treated like new bots, i.e. elo rating is reset.
There was a (simple) test to be allowed at all to the league.
It was automatic.
spike43884:
--- Quote from: Peter on April 09, 2015, 05:40:31 PM ---A tournament like would have to repeat itself every time. That would drain cpu resources.
I like some parts of how it was done in the last google AI challenge.
They had a elo rating.
New versions were treated like new bots, i.e. elo rating is reset.
There was a (simple) test to be allowed at all to the league.
It was automatic.
--- End quote ---
Automatic is covered by floors...
New versions treated like new bots, not to complicated, you just need to check for 100% duplicate DNA
Simple test to be allowed at all to leagues, as I said...Check active DNA for league violation
Elo rating im still against, for reasons very clear in some of my bots that I wrote in DB2... Lets take a simple example of my 2 very iconic bots, Dizzy & All_hunter... All_hunter is superbly programmed, and can take out most bots, even some of the top F1 bots when i've pitted it in 1 species v 1 species scenario's but then when its pitted against Dizzy, its obliterated, because All_Hunter spreads out, and doesn't form colonies which allows it to target food across the map, and is a failsafe for cannibots evolving in evolutionary simulations and dizzy just spins it. Hence the name dizzy, it spins its prey... Brilliant against any bot that tries to go solo, because it spins the prey so quickly it actually breaks the eyes (some weird weakness in the code, and some sims they haven't even attacked a specific plant or bot but its eyes go as if its been spinned?) now, currently in Elo rating it'd put Dizzy above all_hunter probably, yet if we pitted Dizzy against a 3rd bot of mine, RBM (russian babies mobilised) it'd be obliterated, yet RBM is easily wiped out by all_hunter. So which one is better? You don't nessisarily save much more CPU cost than if did another type of league, or you'd get really inaccurate readings...
Your point of the entire tournament with floors having to be repeated is incorrect. A bot is entered in, then it starts only floor one, once floor one is finished it goes to floor 2 if it scored highly in floor one, and that process continues till it hits a floor which it struggles in... This stops the problem of 1 species v 1 species scenario's cropping up as well. Only once a bot is entered does a floor need to be repeated...which also allows all bots on that floor to be re-ranked...Maybe to conserve CPU resources we update the positions every 6 hours instead of every tournament, just storing scores between those intervals...
Thanks for the support botsareus :D
Panda:
I like the idea of using the Elo system.
--- Quote from: Numsgil on April 09, 2015, 02:51:18 PM ---There are possibly issues of elo inflation to contend with and bots doing better/worse on different versions, but those are similar to the issues real elo systems face, and we have the advantage that bots don't age or really change much over time (except for the issues around different versions).
--- End quote ---
The bots shouldn't be powered differently over shorter periods of time, only over a longer period of time. Perhaps the concepts of a season can be introduced (after an agreed number of versions or periodically) to help prevent problems between versions. This might reduce the effects of deflation. However, I don't believe that deflation is too much of a problem since it would probably be only used to compare current bots rather than bots between versions, since a bot is never removed from the rankings or never stops playing (unless the rules of the tournament are changed).
We wouldn't have a problem with bots "avoiding" playing, either.
Would it be sensible to have a round robin tournament (or something similar) periodically for the top n bots to give a clear ranking?
--- Quote from: spike43884 on April 10, 2015, 06:30:10 AM ---Elo rating im still against, for reasons very clear in some of my bots that I wrote in DB2... Lets take a simple example of my 2 very iconic bots, Dizzy & All_hunter... All_hunter is superbly programmed, and can take out most bots, even some of the top F1 bots when i've pitted it in 1 species v 1 species scenario's but then when its pitted against Dizzy, its obliterated, because All_Hunter spreads out, and doesn't form colonies which allows it to target food across the map, and is a failsafe for cannibots evolving in evolutionary simulations and dizzy just spins it. Hence the name dizzy, it spins its prey... Brilliant against any bot that tries to go solo, because it spins the prey so quickly it actually breaks the eyes (some weird weakness in the code, and some sims they haven't even attacked a specific plant or bot but its eyes go as if its been spinned?) now, currently in Elo rating it'd put Dizzy above all_hunter probably, yet if we pitted Dizzy against a 3rd bot of mine, RBM (russian babies mobilised) it'd be obliterated, yet RBM is easily wiped out by all_hunter. So which one is better? You don't nessisarily save much more CPU cost than if did another type of league, or you'd get really inaccurate readings...
--- End quote ---
You're forgetting that you're only really comparing these 3 bots together. Where RBM > Dizzy > All_hunter (RBM ? All_hunter) and you aren't taking into account all of the other bots for all of them 00. With enough competitions, the bot should arrive at an appropriate rating. All_hunters rating would be reduced by Dizzy slightly and (if it beats RBM) then its elo would be increased. Elo rating doesn't work well when you're comparing 3 bots that can beat each other but works better when more can be compared.
Peter:
I assume everyone naming Elo means a statistical model instead of just Elo. From a quick search Glicko, TrueSkill and Elo are the ones where libraries are plenty available.
Trueskill has the advantage it could rate "free for all" fights. Which may be handy for newer bots to quickly rise on the league table.
Also how about multiple fight maps, size, shapes and different physics. Means a bot need to survive in different environments. A all round bot.
A round robin-like for the top bots seems logically. Might enforce a higher confidence/reliability level in a rating model for the top bots to create it naturally. Not sure if it needs to happen often, once in place positions would be rather static. Which has the advantage to extend the round robin more down.
Panda:
--- Quote from: Peter on April 10, 2015, 09:10:48 AM ---I assume everyone naming Elo means a statistical model instead of just Elo. From a quick search Glicko, TrueSkill and Elo are the ones where libraries are plenty available.
Trueskill has the advantage it could rate "free for all" fights. Which may be handy for newer bots to quickly rise on the league table.
--- End quote ---
I suppose I do. I was reading about a few of them earlier and which statistical system we use would be a decision we'd have to make.
--- Quote from: Peter on April 10, 2015, 09:10:48 AM ---Also how about multiple fight maps, size, shapes and different physics. Means a bot need to survive in different environments. A all round bot.
--- End quote ---
A specific league for an "all round bot"? I think that's a brilliant idea but I foresee problems with computation times.
--- Quote from: Peter on April 10, 2015, 09:10:48 AM ---A round robin-like for the top bots seems logically. Might enforce a higher confidence/reliability level in a rating model for the top bots to create it naturally. Not sure if it needs to happen often, once in place positions would be rather static. Which has the advantage to extend the round robin more down.
--- End quote ---
Prehaps a system where, when a new bot reaches a certain rating (possibly above the worst bot in the "top league"), it can be entered into the "top league" and pitted against each one of the bots and then placed (or not placed) in the league accordingly? Unless that is what you were trying to say?
Navigation
[0] Message Index
[#] Next page
[*] Previous page
Go to full version