Darwinbots Forum

Code center => Darwinbots3 => Topic started by: spike43884 on March 22, 2015, 08:21:13 AM

Title: DB3 progress and networking
Post by: spike43884 on March 22, 2015, 08:21:13 AM
(I've split the topic off from a discussion in this topic (http://forum.darwinbots.com/index.php/topic,2206.0.html)) - Numsgil.

DB3 - Could you give us a new wiki page which is a jargon free (or as jargon free as possible) list of the things done in DB3 or partially done...Especially things like the commands that are input to the DNA (so the language for the DNA). I do hope your using a relatively fresh language for the DNA, possibly with backwards compatibility to the old language so the old bots can still jump about in DB3. I mean, the DB2lang is really messy. A lot of the commands are illogical, like ABS being NOT, what is ABS meant to litterally mean? I thought abs were your chest muscles, not a not statement.
Title: Re: DB3 progress
Post by: Shadowgod2 on March 22, 2015, 09:55:05 AM
abs is the absolute value from 0. so if you have a positive number then it will stay the same, but if you have a negative number it will be turned to a positive number

0 abs = 0
x abs = 1
-x abs = 1

usually it is used for conditionless bots.

do you ever read the definitions and examples? the db code isn't messy, you just don't know everything that you should to be able to read it properly. that takes time but you have to read and test yourself a bit. why do you think i take so long on some things you give me, to give you time to come up with something yourself. it took me over a month of reading and testing to finally get it all, but i didn't give up on trying to understand.
Title: Re: DB3 progress
Post by: spike43884 on March 23, 2015, 12:32:08 PM
abs is the absolute value from 0. so if you have a positive number then it will stay the same, but if you have a negative number it will be turned to a positive number

0 abs = 0
x abs = 1
-x abs = 1

usually it is used for conditionless bots.

do you ever read the definitions and examples? the db code isn't messy, you just don't know everything that you should to be able to read it properly. that takes time but you have to read and test yourself a bit. why do you think i take so long on some things you give me, to give you time to come up with something yourself. it took me over a month of reading and testing to finally get it all, but i didn't give up on trying to understand.
And most of the things I try a lot on, especially before asking, it makes sense now the Abs, but elsewhere its been explained in ways, among other things in DB code, which make it seem messy. Plus, even messy code you can eventually understand, some of the biggest programming languages have inefficiencies and parts which are messy, take JAVA for example, when you run a program, it'll tell you that you have an error, but unlike python and such won't tell you where the error is. With java, good lord, everythings forced to be so long and thousands and thousands of lines of code are needed for simple programs, 1 error, not a clue where it is, not a clue what the error is. A mere inefficiency. The more efficient, and easy DB code is, the better for new people coming on. Plus, if the programming is easy then it could help people that don't know much about DNA and such understand it more, as even though its not perfect to real life, its probably one of the best, if not the best open source, free simulators out there. Then to top it off, the more simplistic the code for darwinbots the better, as DNA is made up of 4 bases, its that simple, of course we couldn't right in that easily (unless your steven hawkins maybe?) so it needs to be abbreviated into common, or relatively quick to work out, and well explained abbreviations, and commands. These commands then need multiple assets able to be given to them, thus when combined with other commands or operators, and intergers it can perform an almost endless array of tasks.

I mean, even numsigl has said that some bits of the code for DB2 is messy, and thats why he doesn't want to work on it, and instead wants to work on DB3. Both DNA side, and console side.

A FINAL POINT: Numsigl, considering your now hired by google, couldn't you get their interest in DB3 by stating that its an educational platform which can also help train skills, as it helps people understand programming more (Especially assembly language programming as DB2 is currently quite close to) and helps people understand biology, cellular life, the challenges of cellular life, methods that can be used by cellular life, the functions they can perform & how DNA works. then you could do multiple things, like you could get a small (very small :3) chunk of the google servers to run IM or such off (and because its google, they could solve port issues and such :3) and you could grab a few google nerds to help you on this (some other people at google must be interested).
Title: Re: DB3 progress
Post by: Numsgil on March 23, 2015, 09:53:26 PM
DB3 - Could you give us a new wiki page which is a jargon free (or as jargon free as possible) list of the things done in DB3 or partially done...

I'm still in day 1, creating the heavens and the earth.  Although it doesn't mention when God created the physics of the universe; that might be more like a day 0 thing.  I'm definitely not as far as separating the light from the darkness.

Also, the wiki is a pretty good resource if you don't understand what something is trying to do. eg: abs (http://wiki.darwinbots.com/w/Abs).
Title: Re: DB3 progress
Post by: spike43884 on March 24, 2015, 01:26:42 PM
Still, It'd be good to have a wiki page, just so we can see. If we've got a nice detailed jargon free list of things done, then we can help do solving of problems, and suggestions without even knowing the actual code.

--Or atleast some of us might be able to, It's like with java, I can't program in it, yet I can explain exactly what system to use for a complicated task to a java programmer? The world is a mysterious place indeed.


Anyway, you should look for some pals at google that want to help, then you can have 1 person doing physics, 1 person doing displaywork, 1 person doing option toggle stuff, 1 person writing the user-side code, 1 person putting in the direct bot functions. Splitting it into tasks makes it much easier :P What language are you doing it in?
Title: Re: DB3 progress
Post by: Numsgil on March 24, 2015, 10:33:55 PM
It's like with java, I can't program in it, yet I can explain exactly what system to use for a complicated task to a java programmer? The world is a mysterious place indeed.

Hmm, are you familiar with the Dunning-Kruger effect (http://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect)?

Quote
What language are you doing it in?

Let's play a little scavenger hunt type game and see if you can find the answer  :)  I'll give you a hint in the form of a limerick:

There once was a Darwinbots user
The wiki-- it could not confuse her
For in its bright pages
The words of the sages
Something Something read the wiki
Title: Re: DB3 progress
Post by: spike43884 on March 25, 2015, 12:38:34 PM
Hmm, are you familiar with the Dunning-Kruger effect (http://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect)?
I wasn't familiar with the Dunning-Kruger Effect, by that name, but most people would notice that. A biased viewpoint of ones self.
Anyway, Im not sure, It seems by every simple-level test that I should be able to make some magnificent program, yet give me something to write it in, and well, it goes kaput. Hmm, its a bit like a quote from jurassic park really...Do you remember "I think little boys are born wanting to be one of two things, astronomers or astronauts" where he says, some of them will want to watch from the safe, far away. Stating whats happening, or happened but never actually being there, or being there, in the danger actually being right in the action.

Let's play a little scavenger hunt type game and see if you can find the answer  :)  I'll give you a hint in the form of a limerick:

There once was a Darwinbots user
The wiki-- it could not confuse her
For in its bright pages
The words of the sages
Something Something read the wiki
Nice Limerick. C#
I've not done C# before, but some point later this year in computer science we should be learning it, so I'll have an early shot at it. -Just hoping you've programmed it modular.ly-

Also, random fact: You never actually go to www.darwinbots.com as you actually go to www.darwinbots.com.
Title: Re: DB3 progress
Post by: Numsgil on March 25, 2015, 03:55:56 PM
-Just hoping you've programmed it modular.ly-

You can browse the source here: svn.darwinbots.com/Darwinbots3 (http://svn.darwinbots.com/Darwinbots3) or there's a nice web interface here: http://websvn.darwinbots.com/listing.php?repname=Darwinbots3&path=%2FTrunk%2FModules%2F&#a6326297d44805def74d70f2afb501a33 (http://websvn.darwinbots.com/listing.php?repname=Darwinbots3&path=%2FTrunk%2FModules%2F&#a6326297d44805def74d70f2afb501a33).  Short answer: it's very modular.  There's a lot of "About This Folder.txt" files that you can read to get an idea of what the modules actually are supposed to do.

In terms of progress, I'm still pretty much where I was when I wrote this (http://forum.darwinbots.com/index.php/topic,6508.0.html) (life's been busy and all).
Title: Re: DB3 progress
Post by: spike43884 on March 26, 2015, 12:02:48 PM
-Just hoping you've programmed it modular.ly-

You can browse the source here: svn.darwinbots.com/Darwinbots3 (http://svn.darwinbots.com/Darwinbots3) or there's a nice web interface here: http://websvn.darwinbots.com/listing.php?repname=Darwinbots3&path=%2FTrunk%2FModules%2F&#a6326297d44805def74d70f2afb501a33 (http://websvn.darwinbots.com/listing.php?repname=Darwinbots3&path=%2FTrunk%2FModules%2F&#a6326297d44805def74d70f2afb501a33).  Short answer: it's very modular.  There's a lot of "About This Folder.txt" files that you can read to get an idea of what the modules actually are supposed to do.

In terms of progress, I'm still pretty much where I was when I wrote this (http://forum.darwinbots.com/index.php/topic,6508.0.html) (life's been busy and all).
I may a lot more give very technical details on systems to do some of the modules if I can't grapple C#
The web interface is a lot nicer (times new roman -> killed me)
Do you know any good tutorials etc. for C#
Title: Re: DB3 progress
Post by: spike43884 on March 26, 2015, 12:09:28 PM
or there's a nice web interface here: http://websvn.darwinbots.com/listing.php?repname=Darwinbots3&path=%2FTrunk%2FModules%2F&#a6326297d44805def74d70f2afb501a33 (http://websvn.darwinbots.com/listing.php?repname=Darwinbots3&path=%2FTrunk%2FModules%2F&#a6326297d44805def74d70f2afb501a33). 
I might ask, how can I edit things on that link, Im thinking of first adding some notes in the networking module. I might be able to devise something there. Primarily including port arrays hopefully.
[Definition] "Port Arrays" & "Port Array Teaming" - Port arrays are a specific pattern of ports, used to connect. This is beneficial over one port as if the specific port you connect via is blocked, a port array can divert to the unblocked ports, partially because it can be returned a ping to inform it of the successful ports. If multiple get through it can be teamed to allow stress on the ports to be alieviated so other programs can also get though.


That should help on what I mean.
Title: Re: DB3 progress
Post by: Numsgil on March 26, 2015, 01:34:12 PM
Do you know any good tutorials etc. for C#

I picked it up pretty easily just by trying to make it do stuff I needed, but then I already knew C++ very well.  If you don't really know a programming language yet it's going to be much harder.  I learned to program on the ancient C For Dummies series of books, but that was way back in the 90s.  I'm not sure what people use these days.  But if there's one question I'm sure the internet can answer it's where to find resources for learning to program.

Quote
I might ask, how can I edit things on that link

You don't; it's a read only look at the repository.  If you want to make changes, you have to grab the repository and tools and make SVN patches (again, that DB3 link in the wiki is pretty explicit about how to set it all up).

Quote
I'm thinking of first adding some notes in the networking module.

Any work around networking is going to be premature, as I don't have answers to even very basic questions like TCP or UDP.  (UDP is traditionally what games use, but our multiplayer model in DB2 is very asynchronous and involves mostly sending largish bot files sporadically, so a simple TCP or web service interface might be better, but then maybe I want to do something more clever than what DB2 does).  Or how I'm going to serialize anything.

As a beginner programmer the code base is going to be pretty frightening.  You have to learn about things like unit testing and SVN patches, and most of the code is numerical algorithms for linear algebra, computational geometry, or DirectX shader code.  That said, the Sunweaver module (the DNA basically) is self contained and pretty straightforward, so it might be a good place to poke around.

There's just not a lot of low hanging fruit left.  If it was easy, I've already done it, or it's so far in the future that it doesn't matter yet.  Not to be discouraging!  I like when people help.  Just don't expect to pick up the code and make meaningful contributions if you've never worked on a large (for some definition of large :)) code base before.
Title: Re: DB3 progress
Post by: spike43884 on March 27, 2015, 12:33:15 PM
Do you know any good tutorials etc. for C#

I picked it up pretty easily just by trying to make it do stuff I needed, but then I already knew C++ very well.  If you don't really know a programming language yet it's going to be much harder.  I learned to program on the ancient C For Dummies series of books, but that was way back in the 90s.  I'm not sure what people use these days.  But if there's one question I'm sure the internet can answer it's where to find resources for learning to program.

Quote
I might ask, how can I edit things on that link

You don't; it's a read only look at the repository.  If you want to make changes, you have to grab the repository and tools and make SVN patches (again, that DB3 link in the wiki is pretty explicit about how to set it all up).

Quote
I'm thinking of first adding some notes in the networking module.

Any work around networking is going to be premature, as I don't have answers to even very basic questions like TCP or UDP.  (UDP is traditionally what games use, but our multiplayer model in DB2 is very asynchronous and involves mostly sending largish bot files sporadically, so a simple TCP or web service interface might be better, but then maybe I want to do something more clever than what DB2 does).  Or how I'm going to serialize anything.

As a beginner programmer the code base is going to be pretty frightening.  You have to learn about things like unit testing and SVN patches, and most of the code is numerical algorithms for linear algebra, computational geometry, or DirectX shader code.  That said, the Sunweaver module (the DNA basically) is self contained and pretty straightforward, so it might be a good place to poke around.

There's just not a lot of low hanging fruit left.  If it was easy, I've already done it, or it's so far in the future that it doesn't matter yet.  Not to be discouraging!  I like when people help.  Just don't expect to pick up the code and make meaningful contributions if you've never worked on a large (for some definition of large :)) code base before.
I haven't worked on very large codebases, hence why its good that its modular :) I'll have a look at the sunweaver module, but the networking I can devise a basic outlining system of how to run it through. The speradic messages are terrible unideal at the moment, so is the connection via 1 single port (to be honest, peterIM in general is shoddy). It should be preparing for bots nearby the teleportation 'zone' as such, so that if they do go through not all the data is sent in that speradic moment. Alternatively a short limbo time where over 5 seconds maybe? a bot is taken through.
Title: Re: DB3 progress
Post by: Numsgil on March 27, 2015, 01:16:39 PM
Well if you seriously want to play with networking, you're going to need some background reading so you know what you're talking about (Dunning-Kruger effect and all that).  A good resource might be Gaffer on Games (http://gafferongames.com/networking-for-game-programmers/what-every-programmer-needs-to-know-about-game-networking/).  There's a whole slew of articles that talk about networking in the context that most video games have to worry about it.  Darwinbots is a little bit different than any of the networking archetypes typical for games, but it's a good place to start.  (For the record, I doubt the core engine for Darwinbots will be deterministic just because I haven't put any effort towards that.)

If you can do the grunt work of reading and getting up to speed with the technology and infrastructure and tradeoffs available I'm liable to seriously consider any contributions you want to make.  My knowledge of game networking is pretty superficial as it stands, and someone else with domain knowledge there would obviously be a great resource.
Title: Re: DB3 progress
Post by: spike43884 on March 28, 2015, 07:37:30 AM
Well if you seriously want to play with networking, you're going to need some background reading so you know what you're talking about (Dunning-Kruger effect and all that).  A good resource might be Gaffer on Games (http://gafferongames.com/networking-for-game-programmers/what-every-programmer-needs-to-know-about-game-networking/).  There's a whole slew of articles that talk about networking in the context that most video games have to worry about it.  Darwinbots is a little bit different than any of the networking archetypes typical for games, but it's a good place to start.  (For the record, I doubt the core engine for Darwinbots will be deterministic just because I haven't put any effort towards that.)

If you can do the grunt work of reading and getting up to speed with the technology and infrastructure and tradeoffs available I'm liable to seriously consider any contributions you want to make.  My knowledge of game networking is pretty superficial as it stands, and someone else with domain knowledge there would obviously be a great resource.
Checking that gaffer on games right now :) Thanks.
Title: Re: DB3 progress
Post by: spike43884 on March 28, 2015, 07:50:42 AM
The peer-to-peer part gave me an idea, it mentions how its primarily now only used for RTS games (or turnbased games) Darwinbots, is a turnbased program. Each cycle representing one turn. The main problem they state is desyncronisation but thats mainly stated because they're sharing the same view, as in darwinbots we use different views, connected via a teleporter. Of course in darwinbots the cycles in the currrent IM rarely match, so we don't have to match speeds, we just need to hold whatever is being sent in limbo until it can be transported to the other end. The limbo also allows it to switch ports whichever its going through, though if it includes a limbo that'd start to transfer it more into a client-server topology (though it'd still function relatively close to peer-to-peer).

Preferred System:If we went for a direct client-server model a different approach could be taken, which is one I very much like. Each client can hook their 'area' to the server, with an outgoing only portal (because then we can let the weakest computers join in to!) then per client theres an ingoing portal on a 'server-held area'. This area is a larger, more powerful simulation. Then via a webpage or such what is happening can be displayed by a video feed. Because its a server it can handle many more bots, and can last for much longer because of the additional space and power. All the user-side area's have to comply to the rules set by whichever server they're connecting to (so when they send their bots in they won't just be huge big bertha's). This benefically means that completely slow computers, won't damage the sim in any way, as they're signal is sent out, then when its received its directly fired into the simulation, and both sims have no need to be in-sync as its one-way and they're different ''area's''

Hopefully that made as much sense as it did to me :P
Title: Re: DB3 progress
Post by: Numsgil on March 30, 2015, 01:25:04 PM
The peer-to-peer part gave me an idea, it mentions how its primarily now only used for RTS games (or turnbased games) Darwinbots, is a turnbased program. Each cycle representing one turn. The main problem they state is desyncronisation but thats mainly stated because they're sharing the same view, as in darwinbots we use different views, connected via a teleporter. Of course in darwinbots the cycles in the currrent IM rarely match, so we don't have to match speeds, we just need to hold whatever is being sent in limbo until it can be transported to the other end. The limbo also allows it to switch ports whichever its going through, though if it includes a limbo that'd start to transfer it more into a client-server topology (though it'd still function relatively close to peer-to-peer).

If we just want to use teleporters like in DB2, a peer-to-peer architecture is probably overkill.  It would only make sense to me if we were trying to do something like patch the two simulations together (like, if bots go off the left of my screen they come in the right of yours), which is a neat idea but beyond what I think I know how to do easily, because once bots start forming larger multibot structures we'd have to either teleport the entire thing once it reached the edge of the screen or try to simulate it on two computers at once, and neither is very tractable.  If we're just sending bot files to each other we might as well make things easy and push to a central server.

Quote
This benefically means that completely slow computers, won't damage the sim in any way, as they're signal is sent out, then when its received its directly fired into the simulation, and both sims have no need to be in-sync as its one-way and they're different ''area's''

If the sims on the slower computers are outoging portals only, wouldn't it make more sense to just skip the simulation all together and just allow users to submit bots directly via something like a webpage?  But then you're not really talking about a distributed "internet mode" type thing anymore.  You're talking about a single computer running a simulation that others can add bots to.
Title: Re: DB3 progress
Post by: Peter on March 30, 2015, 05:24:45 PM
Without some knowledge of the other parts of the system. I'm not sure if you can get do much more than simple bot transmission.

It would be nice to have one universal system.
When one bots moves to another; how to integrate multibots between 2 sims, different sim speeds, different physics.
How feasible would it be to have a computer calculating a bot in a different simulation?
Extending the former, maybe not any simulation at all on a computer itself and just give computing power to a server, which gives a stream of the simulation back, or hosts a stream(svg animation?). Is that possible without overhauling the whole system?

A P2P system using DHT, through grid computing effectively creating a supercomputer which can't be turned off breeding super AI's sounds fun! Now we only need someone who has the knowhow to implement it properly.
Title: Re: DB3 progress
Post by: spike43884 on March 31, 2015, 12:13:53 PM
Without some knowledge of the other parts of the system. I'm not sure if you can get do much more than simple bot transmission.

It would be nice to have one universal system.
When one bots moves to another; how to integrate multibots between 2 sims, different sim speeds, different physics.
How feasible would it be to have a computer calculating a bot in a different simulation?
Extending the former, maybe not any simulation at all on a computer itself and just give computing power to a server, which gives a stream of the simulation back, or hosts a stream(svg animation?). Is that possible without overhauling the whole system?

A P2P system using DHT, through grid computing effectively creating a supercomputer which can't be turned off breeding super AI's sounds fun! Now we only need someone who has the knowhow to implement it properly.
With a central hub computer which we can access to the visualise everything...And of course things can monitor your CPU so it could integrate to not over-use slower computers...
Anyway, Without knowledge of other parts of the system, I could still get a simple bot transmission. The bot is a packet, packet transmissions are easy.

Actually, we probably should start treating bots as packets more, rather than a complex object...It'd be easier in theorywork. I mean, that'd solve the MB assembly problem, as you'd just recompile it like the packets in a message?
I think we certainly need to find what will be the central hub first. Something that doesn't go off (just incase) and can carry reasonable amounts of data (as it needs to cope with transmissions). Benefit of this system were devising, each computer could select whichever port it wishes to connect via? Then the hub just needs to have all ports open :)
Title: Re: DB3 progress
Post by: Peter on March 31, 2015, 12:27:57 PM
(For the record, I doubt the core engine for Darwinbots will be deterministic just because I haven't put any effort towards that.)
Why wouldn't it be deterministic?
Title: Re: DB3 progress
Post by: Peter on March 31, 2015, 12:40:28 PM
Actually, we probably should start treating bots as packets more, rather than a complex object...It'd be easier in theorywork. I mean, that'd solve the MB assembly problem, as you'd just recompile it like the packets in a message?
Nope, the issue is that the MB is in two simulations at one. Transmitting a MB has been possible in DB2.

I think we certainly need to find what will be the central hub first. Something that doesn't go off (just incase) and can carry reasonable amounts of data (as it needs to cope with transmissions). Benefit of this system were devising, each computer could select whichever port it wishes to connect via? Then the hub just needs to have all ports open :)
Most server software got only one port open, why would DB need all ports?
Title: Re: DB3 progress
Post by: Numsgil on March 31, 2015, 01:59:40 PM
(For the record, I doubt the core engine for Darwinbots will be deterministic just because I haven't put any effort towards that.)
Why wouldn't it be deterministic?

Determinism is one of those hard things that doesn't just happen.  There's no such thing as mostly deterministic.  If there's even one thing that's not bit-for-bit identical across two different machines, the whole thing is spoiled.  I'm not doing anything that shouldn't be deterministic in principle, I'm just not putting any effort in to it, which means almost certainly there's going to be something that is going to differ across machine architectures (maybe there's an acos call somewhere.  Those trig functions are notorious).  Not to say it couldn't be deterministic, but it's not a priority.
Title: Re: DB3 progress
Post by: Peter on March 31, 2015, 03:14:24 PM
Gotcha, I forgot about the floating points of hell. :blush:
Title: Re: DB3 progress
Post by: spike43884 on April 01, 2015, 11:35:43 AM
Hence why a central machine running one sim with visual accessible by the users, with then bots input via a one way teleporter would be good.
All ports being open (or most) is simply to relieve "port strain"
Its rare that port strain happens, but its best to avoid it. Its basically a port being overloaded, plus it increases compatibility (Everyone loves compatibility <3)
The floating points of hell wont be a problem with the central machine running it.
Title: Re: DB3 progress
Post by: Numsgil on April 01, 2015, 12:24:43 PM
Hence why a central machine running one sim with visual accessible by the users, with then bots input via a one way teleporter would be good.

Hmm, you know I wonder if you have some misconceptions that are driving how you see the program.  Do you think DB2 runs real time (30FPS+) for everyone but you, and you're stuck with .1 cyc/sec sims for some reason?  Because that's not true.  I was running simulations circa 2005 with a P4.  My phone has at least 4 times the horsepower I did back then.  The simulation just doesn't scale well.  That means as you double the size of the sim you need something like 4 times more resources.  Or worse, since it's not multithreaded and you can't really find single threaded performance gains anymore in hardware.

DB3 will be multithreaded, but what that gives I'm going to suck up with an even more aggressive simulation.  You could run it on some $10k "supercomputer" and a simulation might only run twice as fast as a dinky $500 machine.  If we want the sorts of bot-cycles that make a good ALife sim we're going to need to think wide (clusters) instead of tall (servers).

Quote
All ports being open (or most) is simply to relieve "port strain"
Its rare that port strain happens, but its best to avoid it. Its basically a port being overloaded, plus it increases compatibility (Everyone loves compatibility <3)
The floating points of hell wont be a problem with the central machine running it.

Hmm, I'm calling BS on this.  That's not really a thing.  Linux can handle tens of thousands of connections from a single source on a single port (http://stackoverflow.com/questions/2350071/maximum-number-of-concurrent-connections-on-a-single-port-socket-of-server).  Most webservers for popular websites have something like a "routing" server that accepts all the incoming http connections on port 80 for the entire world (or at least a region) and forwards them on to actual servers to send pages.

I don't understand why ports are such a hobby horse (http://www.phrases.org.uk/meanings/hobby-horse.html) for you.  Is there something I'm missing?
Title: Re: DB3 progress
Post by: spike43884 on April 02, 2015, 01:32:33 PM
Hence why a central machine running one sim with visual accessible by the users, with then bots input via a one way teleporter would be good.

Hmm, you know I wonder if you have some misconceptions that are driving how you see the program.  Do you think DB2 runs real time (30FPS+) for everyone but you, and you're stuck with .1 cyc/sec sims for some reason?  Because that's not true.  I was running simulations circa 2005 with a P4.  My phone has at least 4 times the horsepower I did back then.  The simulation just doesn't scale well.  That means as you double the size of the sim you need something like 4 times more resources.  Or worse, since it's not multithreaded and you can't really find single threaded performance gains anymore in hardware.

DB3 will be multithreaded, but what that gives I'm going to suck up with an even more aggressive simulation.  You could run it on some $10k "supercomputer" and a simulation might only run twice as fast as a dinky $500 machine.  If we want the sorts of bot-cycles that make a good ALife sim we're going to need to think wide (clusters) instead of tall (servers).
I don't have a misconception about this such things like the change in cycles per second. The thing is, we need a way around "floating points of hell" and the differentiating between the computers. In real life when a cell moves an inch, time doesn't slow down 10 times for it, then another inch speed up 5 times. What about thinking wide and tall. Running the simulation on the central machine giving visual feedback, but using the participating terminals (or personal computers) to calculate the bots. Remember, the simulation is split into 3 segments primarily, the area of the simulation, the bots then finally the projectiles etc. If you spread the load of the bots across the computer terminals then it can calculate the DNA triggers, then thats returned to the server which then treats the simulations (and maybe projectiles) then once a projectile has hit a bot it lets the computer terminals handle it as a 'bot event'. Thus because the actual simulation is on one centralised machine it doesn't suffer from differentiation, but the load is spread by the terminals to calculate the bots. Tall and wide. Then a simple little visual output is given to every terminal.
Title: Re: DB3 progress
Post by: Numsgil on April 02, 2015, 11:10:48 PM
Problem is that maybe 70% of the simulation cost is doing physics and vision and the like, 20% is the DNA execution, and 10% is random stuff.  Even if you offloaded everything that wasn't physics to other machines, you're only maybe going to make things 40% faster.  Not exactly compelling.

Even if you offloaded physics too, there's something like a 100ms latency between computers on the internet (more like 20-200ms, but on that order).  Even if there was a computer that could instantly give you the answer, if you have to wait 200 ms (for the round trip) for it, best case the simulation runs at 5 cycles/sec.  Again, not exactly compelling.

Finally, you have to consider the bandwidth of the server.  Let's say conservatively that a simulation has about 100K of state that you have to send and receive every cycle.  At a guess of $10/Mbps (https://blog.cloudflare.com/the-relative-cost-of-bandwidth-around-the-world/), that works out to a cost of about $10/month just for the bandwidth.  Which isn't a lot, but that's not really scaling the way I want either.

...

On point, having different parts of the "universe" for a bot run at different time rates isn't really a problem as I see it.  In fact, that's how are universe works anyway.  Even ignoring things like relativity, life in the tropics is more rapid than life in the arctic.  The more energy in an ecosystem the faster life there tends to evolve (Evolution and latinitudinal diversity (http://www.brown.edu/Research/Sax_Research_Lab/Documents/PDFs/Evolution%20and%20latinitudinal%20diversity.pdf)).  In a virtual landscape, faster simulations become valuable real estate, which adds texture.

I would say the biggest problem (or at least, area that could do with some thought) with the current method for IM in DB2 is the teleporters.  Not to say that the way they work now is bad, but the exact mechanism of transfer has a huge effect on what sort of simulation IM would be. 

For instance, it makes me think of Eve Online's wormhole space.  Basically there's a part of Eve Online where different star systems are connected with non-stable wormholes (so two different systems might be connected one hour and not connected the next).  The different star systems are tiered, with difficult but rewarding systems deeper in, and easier systems closer to "normal" space.  Generally the wormholes are two way, so if A connects to B, B connects to A, and the entrances and exits are next to each other in either system. 

A few months back they made a fairly innocuous change to how entering a system through a wormhole works which made heavier ships spawn further from the entrance back to their original system.  Which made it harder for people to scout out systems for "content" (ie: other people to fight) because once a heavier ship had jumped in they were pretty much committed to fighting ("slow boating" it back to the wormhole taking so long that you'd be dead before you got there if things went sour).  The small change made a fairly large change to the competitive landscape: it changed the playing field to encourage fewer, larger corporations, since they were the ones that could afford to lose expensive ships.

We might want to consider what sort of competitive landscape we want DB3 to look like, and then decide on a serving architecture that supports that goal.  For instance, do we want a single dominant species across all machines? (ala Pando (http://www.amusingplanet.com/2010/11/pando-single-largest-living-organism-on.html)?).  A fairly well connected universe with similar settings on machines would probably look like that.  I think that's what the current DB2 model encourages.  Or do we want regions of computers rules by the same species, but different species on the whole (ala Eve Online nullspace (http://evemaps.dotlan.net/map/Caldari_VS_Gallente))?  Sort of a chunky ownership map basically.  Machines connected so that there are a few choke points here and there would encourage that, I think.  Or do we want each machine to almost be an island with different species holding different machines?  Maybe something like the wormhole connectivity from Eve online would work for that.

Then there's issues of what can teleport out and what can teleport in and where.  If large multibot structures can be teleported in anywhere on the incoming sim, a species would be well advised to only allow its large multibots to teleport in to sims it's sure it can dominate.  If there's a single place to teleport in, then species can camp the entrance to their local sim to prevent incursion.  Or if only single bots can teleport, it might be more difficult for a species to invade another's machine, because they can't bring as much to the fight.
Title: Re: DB3 progress
Post by: spike43884 on April 03, 2015, 08:54:08 AM
Weve also got to decide how we want life to progress in it. I mean your making a point a lot of these systems might run at 5 cycles/sec but if its dedicated then it can run for a long time slowly.
Remember, not everyone runs at 100 cycles/sec or wants to have a complete mad evolution in 1 and a half minutes. Plus, your point of tropics having faster evolution, they still don't have faster time. The mutation rates check how fast our evolution goes...Not the speed of time.

Second, heres a possible solution to the multibot transportation problem. Make the simulations 'overlap'
The simulations have a 1st border, then 2nd border. The 1st border gives you visual output, its the edge of the overlap really...Its a region around the edge, then just after that the 2nd border. The 2nd border is the other ends 1st border, its more transfering infomation of what is going on there to the 1st border on your simulation. Thus you might not nessisarily transport the entire multibot at once, but it'll appear seamless still.

Finally, Compatibility is still something you need to prioritise, And latency you need to think of along with floating points. Irregularities in area's are fine, except for the most extreme ones. Plus everyone wants to equally enjoy IM, and if someone has a quantum computer running in their house, and end up creating a super-monsterous-cancerous-bot it might run fine on their computer, but on a computer that struggles it would obliterate. Now, I can understand your points that your coming from, but compatibility and everyones equal enjoyment are key factors to consider, which in DB2 weren't nessisarily as prioritised and should be one of the highest things on the IM agenda.
Title: Re: DB3 progress
Post by: Peter on April 03, 2015, 03:24:39 PM
We might want to consider what sort of competitive landscape we want DB3 to look like, and then decide on a serving architecture that supports that goal.  For instance, do we want a single dominant species across all machines? (ala Pando (http://www.amusingplanet.com/2010/11/pando-single-largest-living-organism-on.html)?).  A fairly well connected universe with similar settings on machines would probably look like that.  I think that's what the current DB2 model encourages.  Or do we want regions of computers rules by the same species, but different species on the whole (ala Eve Online nullspace (http://evemaps.dotlan.net/map/Caldari_VS_Gallente))?  Sort of a chunky ownership map basically.  Machines connected so that there are a few choke points here and there would encourage that, I think.  Or do we want each machine to almost be an island with different species holding different machines?  Maybe something like the wormhole connectivity from Eve online would work for that.
What would the wormhole connectivity do? Make it so only a minimal amount of bots arrive, weaken the bots before they can attack the native bots?

Added a image for DB2 and 'choke' points.
Assuming that DB2 random sends from any sim to any sim to any location. And choke points to sims 'nearby' at stuck locations.
Title: Re: DB3 progress
Post by: Numsgil on April 03, 2015, 04:16:20 PM
What would the wormhole connectivity do? Make it so only a minimal amount of bots arrive, weaken the bots before they can attack the native bots?

Well in Eve the wormholes collapse after a certain amount of mass goes through them.  So in that scenario you'd sever the outbound connection after some total number of bots, or their total mass, exceeds some threshold.  Not sure if that's a good idea or not, but something to consider and think through the implications.
Title: Re: DB3 progress and networking
Post by: spike43884 on April 04, 2015, 07:49:19 AM
If you used the overlaps instead of wormholes, shapes could drift as well, because its not a teleporter and more a solid connection...the shapes could move across, or if arranged solidly in  a simulation form a chokepoint, which would be more realistic as in real life objects get in the way ;3
Title: Re: DB3 progress
Post by: Peter on April 12, 2015, 03:55:51 PM
DB3 will be multithreaded, but what that gives I'm going to suck up with an even more aggressive simulation.  You could run it on some $10k "supercomputer" and a simulation might only run twice as fast as a dinky $500 machine.  If we want the sorts of bot-cycles that make a good ALife sim we're going to need to think wide (clusters) instead of tall (servers).
Kinda open question, but what amount of parallelization can be achieved in DB3? What's the maximum amount of processors it could fully use extra. 4, 12, 30, no clue yet? Is there a module that can hardly be parallelized?
Title: Re: DB3 progress
Post by: Numsgil on April 12, 2015, 04:33:32 PM
DB3 will be multithreaded, but what that gives I'm going to suck up with an even more aggressive simulation.  You could run it on some $10k "supercomputer" and a simulation might only run twice as fast as a dinky $500 machine.  If we want the sorts of bot-cycles that make a good ALife sim we're going to need to think wide (clusters) instead of tall (servers).
Kinda open question, but what amount of parallelization can be achieved in DB3? What's the maximum amount of processors it could fully use extra. 4, 12, 30, no clue yet? Is there a module that can hardly be parallelized?

I think the majority of CPU cycles are going to be going to optimization problems, which tend to parallelize pretty well up to maybe half the size of the problem?  So say the problem size scales with the number of panels in the world.  Say each bot has 16 panels on average and there are 100 bots in the world.  That works out to around 800 CPU cores I could probably utilize pretty well.  It's a good candidate for GPGPU work, for instance.  But the work scales up faster than the cores.  By which I mean, double the size of the problem, and you ~8x the amount of work while you can only really 2x the number of cores you have working on it.

Also, most of these optimization problems are sparse, so in the long run I expect to use some sparse algorithms instead, and get maybe a factor of 100x speed up there, but utilize far fewer CPU cores simultaneously.  Then maybe it scales to just dozens of CPUs regardless of the problem size?  Hard to say.

What are these optimization problems?  Things like fluid and bot deformation and physics to a certain limited extent.  For fluid especially, enough small things working in concert can create some fairly global effects.  Like if you had enough bots pushing the fluid leftwards you could get basically a river current going.  I'd like to allow that sort of thing, and the math for that ramps up pretty quick.
Title: Re: DB3 progress
Post by: spike43884 on April 13, 2015, 08:14:35 AM
DB3 will be multithreaded, but what that gives I'm going to suck up with an even more aggressive simulation.  You could run it on some $10k "supercomputer" and a simulation might only run twice as fast as a dinky $500 machine.  If we want the sorts of bot-cycles that make a good ALife sim we're going to need to think wide (clusters) instead of tall (servers).
Kinda open question, but what amount of parallelization can be achieved in DB3? What's the maximum amount of processors it could fully use extra. 4, 12, 30, no clue yet? Is there a module that can hardly be parallelized?

I think the majority of CPU cycles are going to be going to optimization problems, which tend to parallelize pretty well up to maybe half the size of the problem?  So say the problem size scales with the number of panels in the world.  Say each bot has 16 panels on average and there are 100 bots in the world.  That works out to around 800 CPU cores I could probably utilize pretty well.  It's a good candidate for GPGPU work, for instance.  But the work scales up faster than the cores.  By which I mean, double the size of the problem, and you ~8x the amount of work while you can only really 2x the number of cores you have working on it.

Also, most of these optimization problems are sparse, so in the long run I expect to use some sparse algorithms instead, and get maybe a factor of 100x speed up there, but utilize far fewer CPU cores simultaneously.  Then maybe it scales to just dozens of CPUs regardless of the problem size?  Hard to say.

What are these optimization problems?  Things like fluid and bot deformation and physics to a certain limited extent.  For fluid especially, enough small things working in concert can create some fairly global effects.  Like if you had enough bots pushing the fluid leftwards you could get basically a river current going.  I'd like to allow that sort of thing, and the math for that ramps up pretty quick.

About the river thing...Then you need particles, because to create a fluid with momentum, god that'd be really intensive nums. really intensive. I think it'd be better spent to be adding resources which the bots would utilise, such as oxygen, carbon, hydrogen, nitrogren, amino acids, glucose, fructose and starch. Also, an interesting thing to see would be an Z axis of some sort, maybe 2 layers on the Z axis which bots can go into, to drift over things. It'd create a new challenge of eyes but it'd surely be fun.
Title: Re: DB3 progress
Post by: Numsgil on April 13, 2015, 01:19:13 PM
About the river thing...Then you need particles, because to create a fluid with momentum, god that'd be really intensive nums. really intensive.

You mean simulate the fluid as individual particles bouncing around?  No, that's not a good way to do it.  Instead I'm simulating the fluid using "panel methods".  Specifically the fluid sim is a simplification of an incompressible potential flow, commonly used for simulating things like airfoils.  Basically each panel in a bot can pull or push the fluid so that the fluid won't pass through it.  This pulling or pushing takes the form of hills and valleys in the "potential" of the flow, and the velocity of the fluid at any point in space can be calculated as the sum of the slopes (gradients) of these hills and valleys at that point.  Which is a nicely linear problem.  Expensive to solve, but not that expensive.  Not entirely computationally dissimilar to if you connected each bot together using a spring, or played around with "planet eaters" mode in DB2.  But faster since I know what I'm doing :)

Have you seen the fluid demo (http://forum.darwinbots.com/index.php/topic,4866.0.html) I put up around two years ago? 

Quote
I think it'd be better spent to be adding resources which the bots would utilise, such as oxygen, carbon, hydrogen, nitrogren, amino acids, glucose, fructose and starch.

There'll be something along those lines certainly.  From what research I can find it seems that biodiversity (which is one of those holy grail goals I want for DB3) arise in nature from energy rich/resource poor areas.  So I need to simulate limiting nutrients, I think, and that implies I need to simulate nutrients.

Almost certainly it won't be things like oxygen or carbon dioxide.  That particular feed back loop is only really interesting on a global scale, so I don't think it's worth the effort for us.  That is, it's not like rain forests are oxygen rich because of all the plants.  The atmosphere is too well mixed for that.

But something analogous to phosphorus or potassium seems like a good idea.  Something necessary to build panels or other organelles probably.  Then bots can fight each other not just for energy to sustain themselves but essential nutrients they need to grow larger and reproduce.  Animals have an easy time of it, since they can rob from the plants. The plants have a harder time of it, since they can't go looking for more.

Quote
Also, an interesting thing to see would be an Z axis of some sort, maybe 2 layers on the Z axis which bots can go into, to drift over things. It'd create a new challenge of eyes but it'd surely be fun.

I'm thinking through having "leaves" in a third dimension for plants.  So plants can compete for space and access to light without preventing animals from running between them.  The other option for that I've been thinking about is having something like shapes emitting light from their surface.  It doesn't map well to how our universe works, but then it would encourage plants to crowd around the light giving shapes and the spaces between shapes would be mostly empty.  But even here I maybe want plants to be able to connect to each other using "roots", and I wouldn't want the roots to block animals from running around.

So while I like the simplicity of a 2D universe I might end up going with an "under" and/or "over" dimension to allow very specific sorts of things to not compete for space with the main 2D layer.  Or put another way, maybe physics between bots and some things they can build are disabled.
Title: Re: DB3 progress
Post by: spike43884 on April 14, 2015, 07:00:39 AM
About the river thing...Then you need particles, because to create a fluid with momentum, god that'd be really intensive nums. really intensive.

You mean simulate the fluid as individual particles bouncing around?  No, that's not a good way to do it.  Instead I'm simulating the fluid using "panel methods".  Specifically the fluid sim is a simplification of an incompressible potential flow, commonly used for simulating things like airfoils.  Basically each panel in a bot can pull or push the fluid so that the fluid won't pass through it.  This pulling or pushing takes the form of hills and valleys in the "potential" of the flow, and the velocity of the fluid at any point in space can be calculated as the sum of the slopes (gradients) of these hills and valleys at that point.  Which is a nicely linear problem.  Expensive to solve, but not that expensive.  Not entirely computationally dissimilar to if you connected each bot together using a spring, or played around with "planet eaters" mode in DB2.  But faster since I know what I'm doing :)

Have you seen the fluid demo (http://forum.darwinbots.com/index.php/topic,4866.0.html) I put up around two years ago? 

Quote
I think it'd be better spent to be adding resources which the bots would utilise, such as oxygen, carbon, hydrogen, nitrogren, amino acids, glucose, fructose and starch.

There'll be something along those lines certainly.  From what research I can find it seems that biodiversity (which is one of those holy grail goals I want for DB3) arise in nature from energy rich/resource poor areas.  So I need to simulate limiting nutrients, I think, and that implies I need to simulate nutrients.

Almost certainly it won't be things like oxygen or carbon dioxide.  That particular feed back loop is only really interesting on a global scale, so I don't think it's worth the effort for us.  That is, it's not like rain forests are oxygen rich because of all the plants.  The atmosphere is too well mixed for that.

But something analogous to phosphorus or potassium seems like a good idea.  Something necessary to build panels or other organelles probably.  Then bots can fight each other not just for energy to sustain themselves but essential nutrients they need to grow larger and reproduce.  Animals have an easy time of it, since they can rob from the plants. The plants have a harder time of it, since they can't go looking for more.

Quote
Also, an interesting thing to see would be an Z axis of some sort, maybe 2 layers on the Z axis which bots can go into, to drift over things. It'd create a new challenge of eyes but it'd surely be fun.

I'm thinking through having "leaves" in a third dimension for plants.  So plants can compete for space and access to light without preventing animals from running between them.  The other option for that I've been thinking about is having something like shapes emitting light from their surface.  It doesn't map well to how our universe works, but then it would encourage plants to crowd around the light giving shapes and the spaces between shapes would be mostly empty.  But even here I maybe want plants to be able to connect to each other using "roots", and I wouldn't want the roots to block animals from running around.

So while I like the simplicity of a 2D universe I might end up going with an "under" and/or "over" dimension to allow very specific sorts of things to not compete for space with the main 2D layer.  Or put another way, maybe physics between bots and some things they can build are disabled.

Do remember I wasn't here 2 years ago :P (also I just quickly scimmed over your fluid sim thing just in the past 45 seconds really) anyway, if your simulating panels for fluids, im presuming your working it more off surface tension. If you were to include surface tension via the panels oxygen could be more key than I thought, I wasn't originally thinking about oxygen for rich/poor area's and was more thinking of the bots spending some energy to take in oxygen, to then perform respiration etc. which would make them struggle in a CO2 rich sim. But if you included surface tension, you could create air bubbles and such...
Also, could you make something to help bots 'attach' to non-bot things more. I mean water droplets or shapes would be nice to be interacted with, I've always wanted to use shapes or such to make an amoeba with its little dwelling.
Title: Re: DB3 progress
Post by: Numsgil on April 14, 2015, 01:24:58 PM
im presuming your working it more off surface tension.

Surface tension is a different phenomenon.  If you want the closest analog that makes physical sense, I'm simulating pressure (only that's not really exactly true).

Quote
If you were to include surface tension via the panels oxygen could be more key than I thought, I wasn't originally thinking about oxygen for rich/poor area's and was more thinking of the bots spending some energy to take in oxygen, to then perform respiration etc. which would make them struggle in a CO2 rich sim. But if you included surface tension, you could create air bubbles and such...

It'll have a primitive form of respiration, but it'll work more like the "speed" of metabolism (how fast you can break things apart for nrg, or how fast you can build sugars from light using photosynthesis) is controlled by how many panels you have.  I really don't want to simulate anything that's supposed to be well mixed.

Quote
Also, could you make something to help bots 'attach' to non-bot things more. I mean water droplets or shapes would be nice to be interacted with, I've always wanted to use shapes or such to make an amoeba with its little dwelling.

Shapes will be made up of panels, too.  By which I mean the surface of shapes will also be composed of equilateral line segments.  Panels can get "sticky" and attach to other panels.  So in that way bots can connect to each other or to their surroundings.

Shapes will also be destructible.  So bots can remove material from a shape and carve out a little hole.  Bot corpses will tend to "clump" over time and form new shapes (partly to keep the environment interesting, partly to make the sim run faster).
Title: Re: DB3 progress
Post by: spike43884 on April 15, 2015, 07:29:03 AM
im presuming your working it more off surface tension.

Surface tension is a different phenomenon.  If you want the closest analog that makes physical sense, I'm simulating pressure (only that's not really exactly true).

Quote
If you were to include surface tension via the panels oxygen could be more key than I thought, I wasn't originally thinking about oxygen for rich/poor area's and was more thinking of the bots spending some energy to take in oxygen, to then perform respiration etc. which would make them struggle in a CO2 rich sim. But if you included surface tension, you could create air bubbles and such...

It'll have a primitive form of respiration, but it'll work more like the "speed" of metabolism (how fast you can break things apart for nrg, or how fast you can build sugars from light using photosynthesis) is controlled by how many panels you have.  I really don't want to simulate anything that's supposed to be well mixed.

Quote
Also, could you make something to help bots 'attach' to non-bot things more. I mean water droplets or shapes would be nice to be interacted with, I've always wanted to use shapes or such to make an amoeba with its little dwelling.

Shapes will be made up of panels, too.  By which I mean the surface of shapes will also be composed of equilateral line segments.  Panels can get "sticky" and attach to other panels.  So in that way bots can connect to each other or to their surroundings.

Shapes will also be destructible.  So bots can remove material from a shape and carve out a little hole.  Bot corpses will tend to "clump" over time and form new shapes (partly to keep the environment interesting, partly to make the sim run faster).
Can you make the interiors from the bots leak out so the materials can be eaten easily. Everyone likes some guts spilling out.
Title: Re: DB3 progress
Post by: Numsgil on April 15, 2015, 12:10:22 PM
Can you make the interiors from the bots leak out so the materials can be eaten easily. Everyone likes some guts spilling out.

Yes, that's on my feature list :)
Title: Re: DB3 progress
Post by: spike43884 on April 16, 2015, 08:13:47 AM
Can you make the interiors from the bots leak out so the materials can be eaten easily. Everyone likes some guts spilling out.

Yes, that's on my feature list :)
It'd be nice if we had the protein patterns on the surface of bots...Plus some sort of overhaul of the virus system...