Challenge #3: Neural Network

[Elite]

I'm going to have another shot at these, but I'm going to do them in a more informal way. There will still be a hall of fame for those authors and their bots that suceed at the challenges. I'm just going to put out a bunch of open challenges out for bot programers to have a go at, and if you manage to program a bot that meets the criteria, post it.

Also, instead of pretty useless challenges like mazerunning and cliff-jumping, I'm going to be setting challenges at the cutting edge of bot programing.

Okay, so ...

CHALLENGE #3

Code a bot that with behaviors that do not come straight from the dna. Specifically, create a bot who's behavior is controled by an artificial neural network. You'll probably need to find out how they work before starting if you don't know already.

Use the following inputs and outputs only:

Input nodes:

*.eye5

*.refeye

Output nodes:

.up (continuous)

.aimdx (continuous)

-1 .shoot store (boolean)

Those are the only sysvars you can use. Don't bother with anything else, reproduction, body management, combat etc. Just the neural network and the above inputs + outputs.

The network must learn by reinforcement learning, using *.pleas averaged over the last 50 cycles as the reinforcement anti-cost. To get the average pleas over 50 cycles, please use this gene:

def avpleas 13

cond
start
*.pleas *.robage 50 mod 991 add store
*991 *992 add *993 add *994 add *995 add *996 add *997 add *998 add *999 add *1000 add *1001 add *1002 add *1003 add *1004 add *1005 add *1006 add *1007 add *1008 add *1009 add *1010 add *1011 add *1012 add *1013 add *1014 add *1015 add *1016 add *1017 add *1018 add *1019 add *1020 add *1021 add *1022 add *1023 add *1024 add *1025 add *1026 add *1027 add *1028 add *1029 add *1030 add *1031 add *1032 add *1033 add *1034 add *1035 add *1036 add *1037 add *1038 add *1039 add *1040 add *1041 add 50 div .avpleas store
stop

(In case you're wondering, no. I didn't do that by hand)

[Peter]

Code a bot that with behaviors that do not come straight from the dna. Specifically, create a bot who's behavior is controled by an artificial neural network. You'll probably need to find out how they work before starting if you don't know already.

Behavior that does not come straight from the DNA?  ? Controled by an artificial network, I am beginning to feel stupid, I have a slight idea what a neural network is, but can you give a slight example for what the meanig is, I haven't got a really clue.

To want the bot to learn itself something or something. Look here I can get nrg, and the bot has to learn to come and shoot or so. I think I totally miss the point.

Oh wait I missed this

Also, instead of pretty useless challenges like mazerunning and cliff-jumping, I'm going to be setting challenges at the cutting edge of bot programing.

I don't feel like I am really part of the cutting-edge of bot-programming, but if I understand the idea a little, I just want to see the result when it comes, could be interesting.

[Numsgil]

Neural network.

[Elite]

That's definitely going to be useful

What I mean by behavior that doesn't come straight from the dna, and by meta-coding, is behavior that isn't genetically deterministically generated by a list of if ... then triggers. Bots that can be affected by their environment, and find their own strategies, rather than reading off a rigid instruction manual.

(Endy has produced some bots which begin along these lines that use evolutionary algorithms made possible by epigenetics. Reproduction values, thresholds and the like are all free-floating and determined by epigenetic inheritance, with slight mutations each time)

An ANN is one such nongenetic approach. The way I think I'm going to start to go about it is to assign each node (~5 invisible nodes, if that?) to a gene, and reserve a bunch of non-sysvar memory locations for weightings (which can be made epigenetic in later functional versions).

Any advice Num?

[Numsgil]

I thought about implementing a neural net in DNA before.  However, I don't think you should try to code it straight.  Neural nets need alot of memory for the different weightings and such.  Because the free memory locations are scattered throughout the bot's memory structure, trying to hand code everything would lead to alot of insanity.  Instead, I'd program something in a scripting language and "compile" it to working bot DNA.

Neural nets are one of the few instances where 1000 memory locations is actually restrictive   Also, depending on how you look at it, a bot's gene structure is sort of like a complex neural net with non linear weightings.  Each gene takes some inputs, manipulates them, and produces some outputs.

Once upon a time I was playing with the idea of setting up codules in the new DNA system in a network like a neural net.  Each codule would take its input and feed it to another codule, which would either manipulate it some more or feed it to an output codule.  Each codule could have any DNA it wanted, so it could act as a simple additive weighting or a more complex evaluation.

You could also set up a neural net with several bots.  Have each bot act as a node, with either tieval or in/out (or some combination) as the method of communication.  Have most of the bots be blind ("hidden"), with a few designated as inputs (eyes) and a few as outputs (motion and shots).  You could theoretically have the large neuralbot be robust as to the number and connectivity of its bots, and have the connections be determined by some simple genetic code subject to mutation.

[MacadamiaNuts]

Mmmh...

This may be a starting point (ignore the stuff at the begining):

http://www.darwinbots.com/Forum/index.php?showtopic=2176

Instead of building the DNA through random evolution, it could go one gene at once, trying a set of values during a certain amount of time, then store the outcome, try a second set, compare both outcomes and save the good one, then loop until a solution for the first gene is found, then start building the second one.

Once it's done with 4-6 genes, they could keep checking small changes during small amounts of time, if a winner appears, check it again during a longer time, then adopt it. When it meets another "done" bot, they could match their overall outcomes and the loser use the knowledge from the winner.

Basically, I think we can do whatever we want within the current limits, as long as the bot does things one step at once. Heck, I even think we could try and do it with only one store... xD

[Elite]

I've got one! (See attached)

A really simple 2-input, 3-output (one of which is a boolean) 5-invisible-neuron network (or something very loosely resembling something that might be called a neural network  ) with 3 inputs per neuron. All the connectvity data, weightings, biases and node states take up 64 memory slots (902 - 965).

The biggest portion of the code is the declaring of custom variables (I don't know why I even bothered now ).

So far I have no way of actually assigning weights or connections. Its just a demonstrator.
(I'd be mildly surprised if this thing actually works when it does have a system of optimising the weights and connectivity)

Unfortunately, whenever I try and load it into DB the program crashes  

Comments?

-----------------------------------------------------

@ Macadamia

Looks interesting. Can't quite work out what's going on though. I'm guessing it's some sort of heuristic system using 'mutation' of variables that in turn control behavior. Am I close?

971 19 rnd add inc 999 ceil store

^ And what does the 999 ceil do there (or the store for that matter)?

-----------------------------------------------------

(And I've just realised that my .avpleas gene in the original post uses memlocs up to 1041! Doh! )

[Numsgil]

There's probably a limit on custom variables that's never been reached before.  Try limiting it to like 30 or so.  That should let you load it without crashing.

[Elite]

Yep, that does the trick

I've attached the bot with all custom variables removed. It's a bit harder to read, but it loads.

EDIT: And I've also altered the code so that the biases are added, like they are supposed to be, rather than multiplied

A quick tryout with completely randomised neural net settings shows that it also works! Completely scatterbrained, spinning and shooting and moving at random, but reacts to the presence of a something that it can see.

[EricL]

There's probably a limit on custom variables that's never been reached before.

FYI, its 50 in current builds.  I'll raise it to 100 in the next drop.

[Numsgil]

Neat, can't wait to see where you take it.

[Elite]

I haven't got a proper optimisation method yet, but I have got a way of retrieving useful neural network configurations, by loading hundreds of randomized bots at a time and letting them fight it out.

Four of the most interesting bots I found are attached. Look at them go!  

Some interesting consequences:
 - Bots can form 'memories', formed by loops in their archetecture
 - This can allow bots a certain degree of individuality depending on their experiences
 - Bots have a 'reaction time' during which stimulus' filter from input to output

I think I'm going to let epigenetic evolution (mutations generated between parent and child) handle the connection structuture; The weights and biases of the nodes I think I'd like to have adjusted during a bot's life in accordance with how much pain/pleas it has been getting (so a bot can learn by very dumb trial-and error), and then have the values passed on epigenetically.

Or that's my plan at least. Feel free to play around with my bots and neural network method, experiment, add your own features etc.

[Moonfisher]

Very nice
I just realized by looking at this that if I want to write a complex neural network bot then using a simple C program to train the fixed outputs of a league bot to get the weights will be the least of my problems. Gonna need a way to autogenerate the code for the network and insert the weights... I haven't looked at Sanger yet, hoping it can do it for me.
And code execution costs would be high if running F1 conditions...

But I'm not going to quit on this, even if Sanger doesn't help and I have to write a program to generate the bot myself.
I realy think tweeking the values in a neural network would be a far more stable way of evolving bot behavior.
Idealy it should use all inputs and outputs (I realize there may not be enough free mem locations for that), and just start off with random fixed weights, then let the weights change by mutation and possibly sexrepro. Then you could always train weights from a bots code in a seperate program, to create a base, you could also have different degees of training for the same bot (I only think you would be able to see how well trained the network is, and not how good it is at generalizing (Not sure that's an english word), but you could just have a fully trained network and some at different training steps).
I realize I can't have all inputs and outputs, would have to narrow it down to essentials...  
And I know this isn't something I can just whip up over night, I won't even have time to start on this any time soon...

But I realy think this could help produce faster evo results, since all the different mutations break genes and produce redundant code, and generaly have low chances of doing something usefull, or doing anything at all. I know this isn't unnatural, and I love how mutated code can actualy get better, but by mutating and swapping weights in a neural network the change will usualy have some effect, and it can be softer changes and overall tweeks in behavior. It will still be harmfull mutations most of the time, but I'm hoping it'll be less often and less radical changes.
It just hit me that it wouldn't need to take up more mem locations than the amount of hidden neurons since the weights would be fixed and only change by mutation, much more manageable... maybe I will go for all inputs and outputs then
(I realy hope I can use Sanger for most of this, but I kinda doubt it)

[Numsgil]

Keep us posted.  I'd really love to see a successful neural net bot in the leagues.

[Moonfisher]

Heh I'm already having problems determining how to convert certain inputs, like *.refxpos ...
And generaly some inputs seem harder for a bot to understand. So I would probably have to use *.refxpos *.refypos angle as an input, and possibly have no input for the raw values. (I can divide the angle by 1220, and if it can exceed I can always use mod, point is I know it's max value and it's easyer for a bot to put to use I think).
But it's a huge job, still not sure exactly how the values should be converted, but probably going to go for something simple and linear, since I get confused trying to write any complex equation in DB...
And there's probably plenty of values I haven't thought of which will be hard or impossible to convert, but I'll try to get as much raw data in it.
Also wondering how decimal values work... can I save 50 100 div into a location and have 0.5 saved ? If not, then can I do this : 50 100 div 100 mult and have 50 again (Will decimals work in the stack if not when saved)? Or would I risk ending up with 0 or 100 ?
Would make things easyer for me if I could just use dcimal values
As far as I can see all values should be converted to a value between -1 and 1, and I would probably start out with randoms in that range. Problem is I'm guessing mutations will change the weights more than intended, so I may have to scale everything to -100 to 100 anyway... still would be nice if it's possible to atleast use decimal values in the stack.

But this is going to take ages, will be a few months till I have time to put some serious effort into it, and then I need to find a way to convert as many inputs as possible, and I may also be underhestimating what I'll need to do with the outputs, then the program to genereate the code with all the weights, and a program to extract training values from a bot, and I'm proably completely overlooking something that will come back to haunt me.
Just hope I can get it done before DB3 comes out