Darwinbots Forum

Code center => Darwinbots3 => Topic started by: goffrie on May 29, 2008, 08:34:00 PM

Title: P2P Internet Mode
Post by: goffrie on May 29, 2008, 08:34:00 PM
It would be nice if there was a peer-to-peer network of sorts for Internet Mode in Darwinbots3, in which all capable clients would be able to connect to each other directly. Of course, there would still be a central server, but it would reduce the load on the central server, and a relay system could be set up to allow clients behind firewalls to connect to IM (maybe falling back to FTP). It would also be possible this way to have relatively real-time organism exchanges, as the server could tell the clients when a new organism was available and clients could download it as soon as they were told (the server could randomly choose a client to give it to). If the Darwinbots server can't run custom protocols, it could probably be managed over HTTP. The protocol itself should be fairly trivial, as it does not need to support a lot of features (maybe just compression), and efficiency is not a major issue. The only issue I see with this is that many people run behind routers and the system does not really work that way .

Of course, as Darwinbots3 is still early in development, it's not really time just yet to be implementing these kinds of things, but it's just an idea .
Title: P2P Internet Mode
Post by: Numsgil on May 29, 2008, 08:46:33 PM
I've played with the idea of having Darwinbots connect to the web server and run some sort of PHP script to upload and download bots.  If you could connect to the home page or forum, you could run IM.  But I'm about as green as it comes with PHP scripts.  I don't even know if what I want to do is possible (though I imagine it is...)
Title: P2P Internet Mode
Post by: goffrie on May 29, 2008, 09:03:14 PM
Hmm, I'm pretty good with PHP  
It definitely is possible, using HTTP POST with multipart/form-data. It could even work with ordinary POST (that wouldn't work with a web browser though). If you wanted, I could cook up a PHP script which would handle organism uploads. I have no idea how you would interface it with Darwinbots though, and you'd have to tell me where to store the files  .

Another thought, for P2P, perhaps it could interface with Tor and hidden services? That would solve any problems with NATs (firewalls would still be a problem though). It would be pretty finicky(sp?) and slow though.
Title: P2P Internet Mode
Post by: Numsgil on May 29, 2008, 09:25:16 PM
Yeah, I was looking at POST commands.  But I didn't understand it well enough and dropped the idea.  I would certainly appreciate any help writing the PHP script.  I can probably play with it if I had one that worked.  We could probably even set up the existing DB2 to use it...

What needs to happen is that I can call something like this:

www.darwinbots.com/IMMode/UploadScript.php?file="C:\Program Files\Darwinbots3\IM\out\Mybot256734532342344.dbo"

Or whatever the syntax would be.  It can't have a "choose the file" dialog pop up, since it needs to be entirely automated and behind the scenes.  UploadScript.php would then upload that file to some directory on the server that I could set.  It should also have a file size limit that I could set.

A similar script called DownloadScript.php?directory="C:\Program Files\Darwinbots3\IM\in\" would somehow download a binary file from the IM directory on the server and store it in the local directory specified, and then delete the file from the server.  It also can't have one of those "where do I download this too" dialogs pop up, since it also needs to be entirely behind the scenes and automatic.

I think I know how to call a URL from Darwinbots, so if you could cook up some PHPs like that, that I could put on the server and call, that would be very cool indeed.
Title: P2P Internet Mode
Post by: goffrie on May 29, 2008, 09:41:31 PM
You can't really tell the web client to upload a file like that. (It would be nice though ) You need to read the file and upload it as POST data, like "data=buibtuisnuiremhtr5398nf9324==" (base64 encoded or otherwise made 7-bit friendly). Then you could download the file from the server. I don't know if it's possible in VB, but it's almost certainly doable with C#.
Title: P2P Internet Mode
Post by: Numsgil on May 29, 2008, 10:02:11 PM
You mean encode the entire binary file in the url?  Speaking of which is there any sort of limit on the length of a url?  Like could you have a url that's thousands of characters?
Title: P2P Internet Mode
Post by: goffrie on May 30, 2008, 07:45:09 AM
Not quite, POST data is like GET data (which is in the URL) but it goes after the request headers. No idea how to do that in VB or C#, though.
You might be able to encode the data in the URL, but the web server would probably reject it as a buffer overflow attack, (or the server would have a buffer overflow).
Title: P2P Internet Mode
Post by: Numsgil on May 30, 2008, 01:20:50 PM
Now we're running in to my ignorance.  Perhaps a short example?
Title: P2P Internet Mode
Post by: EricL on May 30, 2008, 02:36:09 PM
Quote from: goffrie
It would be nice if there was a peer-to-peer network of sorts for Internet Mode...
This is exactly the way Terrarium worked.  The server did nothing but act as a rendevuez service and collect stats.  Bots moved between peers.  I agree with you in general, but....   Not only is it a lot of work, but there are serious security issues to overcome in such an architecture depending upon the protocol and the means by which the P2P communication is acheived.  Anytime you start running server code (meaning code that listens for new requests on a port and performs work based on those requests) on clients, you open up a serious security can of works not to mention isuses with NATs and firewalls, etc.

People should not confuse the client connectivity topology we choose to use with transport latency issues or with the connectivty problems inherent in the current FPT-based design.  The current FTP based thing is problematic to be sure, but the connectivity problems are not inherent problems with the design.  Those can be fixed by moving to a more reilable server and moving the server communication code out of proc, both of which I am working on (using in part, some code from Nums).  

But we will reach scalability limits on the current architecture.  Understand, there is no real server today.  No server that is, other than the FTP server itself.  There is no code authored by me running on any machine except the client simulators each of us run.  They cooperate through the FTP server by moving files, but in an incredibly ineffecient manner becuase they are forced to shoe horn their semantics into file movements and the semantics of the FTP protocol.  Some poeple call this a shared-file or file-sharing version of cleint-server.  Early corporate email systems (circa 1990) such as CC:Mail and Mcirosoft Mail used this archtiecture.   It is not true client-server (like POP/SMTP).  Were we to move to a true client-server model where we had special code runnnig on a server, implimenting our specific semantics, we could easily scale to several thousand sims connected through a single server.  Buidling this on top of HTTP or PHP or sockets or whatever is mostly a discussion of ease of implimentation.  The result is the same, a true cleint-server architecture.

Scaling beyond that would require mutliple servers with connection logic or partitioning or a P2P architecuture.
Title: P2P Internet Mode
Post by: Numsgil on May 30, 2008, 04:10:32 PM
As a followup to my last post...  There are there any problems having the download script save data to disk?  If there is, is it possible to embed binary data in the HTTP the PHP script returns?  If that wouldn't work either, an text based organism file would probably be possible.  I know for sure that would work...

Which leaves uploading the file, which is where I don't know how to proceed.

Quote from: EricL
Scaling beyond that would require mutliple servers with connection logic or partitioning or a P2P architecuture.

If we use a PHP sheme, we can leave the details of managing multiple servers to our web host.  As the number of users increase, it would just change the amount of bandwidth we would need to pay for (keeping in mind that .dbo files are small and our present bandwidth is only at like 2% utilization, I don't think there's going to be a problem).  If the file accesses start become a problem, it would be a simple matter to move from a file scheme to something in a database.
Title: P2P Internet Mode
Post by: goffrie on May 31, 2008, 08:55:05 PM
Quote from: EricL
there are serious security issues to overcome in such an architecture depending upon the protocol and the means by which the P2P communication is acheived.  Anytime you start running server code (meaning code that listens for new requests on a port and performs work based on those requests) on clients, you open up a serious security can of works not to mention isuses with NATs and firewalls, etc.
Security issues perhaps are a problem, but they are only moderately severe as long as code is kept well-managed. Issues with NATs can be overcome by relaying through the central server.

Quote
People should not confuse the client connectivity topology we choose to use with transport latency issues or with the connectivty problems inherent in the current FPT-based design.  The current FTP based thing is problematic to be sure, but the connectivity problems are not inherent problems with the design.  Those can be fixed by moving to a more reilable server and moving the server communication code out of proc, both of which I am working on (using in part, some code from Nums).
I'm not talking about latency issues or connectivity problems, I'm just talking about a peer-to-peer system. Moving to a more reliable server would help with thse issues, though, which is definitely appreciated.
"moving the server communication code out of proc" - I'm not too sure what you meant by this.
 
Quote
But we will reach scalability limits on the current architecture.  Understand, there is no real server today.  No server that is, other than the FTP server itself.
Which is a real server.

Quote
There is no code authored by me running on any machine except the client simulators each of us run. They cooperate through the FTP server by moving files, but in an incredibly ineffecient manner becuase they are forced to shoe horn their semantics into file movements and the semantics of the FTP protocol.  Some poeple call this a shared-file or file-sharing version of cleint-server.  Early corporate email systems (circa 1990) such as CC:Mail and Mcirosoft Mail used this archtiecture.   It is not true client-server (like POP/SMTP).  Were we to move to a true client-server model where we had special code runnnig on a server, implimenting our specific semantics, we could easily scale to several thousand sims connected through a single server.  Buidling this on top of HTTP or PHP or sockets or whatever is mostly a discussion of ease of implimentation.  The result is the same, a true cleint-server architecture. Scaling beyond that would require mutliple servers with connection logic or partitioning or a P2P architecuture.
... Why the lecture? I don't see what point you're trying to make.

Quote
As a followup to my last post... There are there any problems having the download script save data to disk? If there is, is it possible to embed binary data in the HTTP the PHP script returns? If that wouldn't work either, an text based organism file would probably be possible. I know for sure that would work...
Yes, you can put binary data into an HTTP stream - how do you think binary downloads work?

Quote
Which leaves uploading the file, which is where I don't know how to proceed.
Neither do I, since I'm not familiar with VB.
Title: P2P Internet Mode
Post by: Numsgil on May 31, 2008, 10:06:48 PM
Quote from: goffrie
Quote
As a followup to my last post... There are there any problems having the download script save data to disk? If there is, is it possible to embed binary data in the HTTP the PHP script returns? If that wouldn't work either, an text based organism file would probably be possible. I know for sure that would work...
Yes, you can put binary data into an HTTP stream - how do you think binary downloads work?  

Magic lawn leprachauns.  You ask for the file you want to download, and they use their magic to force their way in to a parallel dimension where I already have the file.  They then murder than alternate me, and steal his computer's hard drive.  Using their magics they teleport the bits from the stolen hard drive to my own.  And voila, I have the file I wanted to download

Quote
Quote
Which leaves uploading the file, which is where I don't know how to proceed.
Neither do I, since I'm not familiar with VB.

Well, let's ignore the whole VB thing for a moment.  If this were for a web site, and we wanted someone to click on a link that would automatically upload a certain file, how would that work?
Title: P2P Internet Mode
Post by: goffrie on June 01, 2008, 11:28:12 AM
Quote from: Numsgil
Magic lawn leprachauns.  You ask for the file you want to download, and they use their magic to force their way in to a parallel dimension where I already have the file.  They then murder than alternate me, and steal his computer's hard drive.  Using their magics they teleport the bits from the stolen hard drive to my own.  And voila, I have the file I wanted to download


Quote from: Numsgil
Well, let's ignore the whole VB thing for a moment.  If this were for a web site, and we wanted someone to click on a link that would automatically upload a certain file, how would that work?
It wouldn't - you can't force a user to upload a file without using client-side things like Javascript (and even then I don't think that would work.) Basically, it poses too much of a security risk to work.
The VB part comes in because you need to voluntarily put the data stream into the request.

Hmm, I found this information for VB6:
Quote
The transaction with Google happen via HTTP POST requests (http://www.jmarshall.com/easy/http/#othermethods). There is a control available in VS6 that allows you to perform these simply; the Microsoft Internet Transfer Control. It is basically a wrapper around the WinInet Windows API, making it nice and easy to use. To add this to your project, start a new standard .exe project in VB6. Go to the Project menu, then Components and pick the "Microsoft Internet Transfer Control". Click OK, and a component will be added to the components toolbox ready for use. Drop one of these onto your form.

(...)

In Visual Basic, you can do it as below this. Imagine Inet1 is the  name of the Internet Transfer Control you added above, and your email  address, password and application ID are stored in the string variables  "myEmail", "myPassword" and "mySource". Note the use of the  form-urlencoded Content-Type header.  
Code: [Select]
   strURL = "https://www.google.com/accounts/ClientLogin"
   strFormData  = "Email=" & myEmail & "&Passwd=" & myPassword &  "&source=" & mySource & "&service=cl"
   strHeaders = "Content-Type:application/x-www-form-urlencoded"
   Inet1.Execute strURL, "POST", strFormData, strHeaders
Your application needs to wait for Google to respond. Do this by  looking out for the icResponseCompleted event in the Inet1_StateChanged  event handler. When it does, store Google's reply in a variable, e.g.:

Code: [Select]
vtData = Inet1.GetChunk(1024, icString)
   </p>  <div class="codeblock"><code>      Do While LenB(vtData) > 0
           outputString = outputString + vtData      
            ' Get next chunk.
            vtData = Inet1.GetChunk(1024, icString)
         Loop      
      Response = outputString</code></div>


I suppose you could use this code (replacing the strFormData with "data=some-encoded-form-of-dbo" and the strURL to an appropriate address) to send a file to the server. On the server side, in PHP, $_POST['data'] would contain the encoded data.
Title: P2P Internet Mode
Post by: Numsgil on June 01, 2008, 04:15:23 PM
Hmm...  so bear with me if this is a stupid question.  I know some data can be embedded in the url (such as the forum url: index.php?showtopic=2808).  And other data can be sent behind the scenes, not in the URL.  For instance, when I write a really long, almost novel-like, post, all that text seems to get embedded in the stream somehow.  I think the two different types are called POST and GET, IIRC.  So... on the VB (or C#) side of things I can convert a file I want to upload in to a binary data stream.  Let's say, for the sake of argument, that the data stream is about 3MB large.  How would the web server running the PHP script expect to get that data?  What would the PHP script look like that took that data and saved it to a file somewhere?
Title: P2P Internet Mode
Post by: goffrie on June 01, 2008, 06:04:57 PM
Quote from: Numsgil
Hmm...  so bear with me if this is a stupid question.  I know some data can be embedded in the url (such as the forum url: index.php?showtopic=2808).  And other data can be sent behind the scenes, not in the URL.  For instance, when I write a really long, almost novel-like, post, all that text seems to get embedded in the stream somehow.  I think the two different types are called POST and GET, IIRC.  So... on the VB (or C#) side of things I can convert a file I want to upload in to a binary data stream.  Let's say, for the sake of argument, that the data stream is about 3MB large.  How would the web server running the PHP script expect to get that data?  What would the PHP script look like that took that data and saved it to a file somewhere?

Data embedded in the URL is GET data, and the data embedded "behind the scenes" as you put it is POST data.
Your 3MB data stream would be sent via POST "behind the scenes", and then the PHP script would receive it like this:
Code: [Select]
   <?php
   if (!isset($_POST['data'])) die("No data sent via POST!");
   $data = trim($_POST['data']); // now $data contains the encoded data - "trim" removes any excess spaces, newlines etc. around it
   $data = base64_decode($data); // assuming base64 encoding, $data now has the binary data. PHP is binary-safe in general but some functions aren't
   $file = @fopen('files/'.time().'.dbo', 'w'); // now open files/(current UNIX timestamp) for writing ('w'). time() returns an integer but it is implicitly converted to a string. the @ suppresses errors
   if (!$file) die("Error opening file ".time().".dbo for writing!"); // error checking
   fwrite($file,$data) or die("Error writing file!"); // write the binary data to the file. the 'or die("")' syntax is quite common for easy error checking, I could have used it for $file but meh
   fclose($file); // close the file. not mandatory
   ?>
This PHP script would take base64 encoded data from the POST stream as the "data" variable (like "data=encodeddata"), decode it, and write it to "files/(time).dbo". Of course, this script would have problems with multiple files being uploaded in the same second - a different file scheme would fix that.
A shortened version (it generates a few warnings/notices on error):
Code: [Select]
<?php
 fwrite(fopen('files/'.time().'.dbo','w'),base64_decode(trim($_POST['data']))) or die("Failed!");
 ?>
Title: P2P Internet Mode
Post by: Numsgil on June 01, 2008, 07:47:22 PM
Ooh, neat, I think this will work.  Why does the data need to be coded in base 64, though?
Title: P2P Internet Mode
Post by: goffrie on June 01, 2008, 08:47:27 PM
Quote from: Numsgil
Ooh, neat, I think this will work.  Why does the data need to be coded in base 64, though?

  Gah, you're supposed to use urlencode (with the %hexnumber goodness). The code I "post"ed (haha) doesn't handle this automatically, though, so you'll have to do it manually. I found this VB function at http://www.csie.nctu.edu.tw/~cdsheen/codin...b-urlencode.php (http://www.csie.nctu.edu.tw/~cdsheen/codings/vb-urlencode.php) :
Code: [Select]
<pre><font><font face="Georgia">Function UrlEncode(ByVal urlText As String) As String
     Dim I As Long
     Dim ansi() As Byte
     Dim ascii As Integer
     Dim encText As String
 
     ansi = StrConv(urlText, vbFromUnicode)
 
     encText = ""
         For I = 0 To UBound(ansi)
         ascii = ansi(I)
 
         Select Case ascii
         Case 48 To 57, 65 To 90, 97 To 122
             encText = encText & chr(ascii)
 
         Case 32
             encText = encText & "+"
 
         Case Else
             If ascii < 16 Then
                 encText = encText & "%0" & Hex(ascii)
             Else
                 encText = encText & "%" & Hex(ascii)
             End If
 
         End Select
     Next  I
    
     UrlEncode = encText
End Function

Using this function, you don't need base64 anymore (d'oh), so remove the "base64_decode" line.
Title: P2P Internet Mode
Post by: EricL on June 02, 2008, 10:52:18 AM
Quote from: goffrie
Security issues perhaps are a problem, but they are only moderately severe as long as code is kept well-managed.
As a guy who has spent a bunch of years working in software security, I respectfully disagree with you.  The moment you have applicaiton level code listening on a port on client machines, particularly code whose source is freely available as with DB, attacking you becomes relatively easy for someone with the right skills.  I am not going to spend my time code-reviewing and penetration testing a local DB server for every possible stack or buffer overflow and I would suggest other developer's time might be equally better spent elsewhere.  

Quote from: goffrie
Issues with NATs can be overcome by relaying through the central server.
If you're communicating through a server, by definition it's not P2P.


Quote from: goffrie
"moving the server communication code out of proc" - I'm not too sure what you meant by this.
It means pulling the client-side of the client-server communicaiton "out-of-process" and putting it in it's own process so that when it waits for the server to respond, it does so on a separate thread of execution (a separate process in this case) and thus does not block the simulator.

Quote from: goffrie
Which is a real server.
Forgive my shorthand.  By "real" server, I meant an applicaiton level server, specifically one which impliments our specific semantics.  Yes, FTP is a real server in the sense that it is code which listens on ports but the semanitcs it impliments are limited and not specific to our needs. It is not an applicaiton server.

Quote from: goffrie
... Why the lecture? I don't see what point you're trying to make.
That it is a hell of a lot easier, expediant and secure to build a real (excuse me.  I mean applciation level) server than to do P2P.  

Title: P2P Internet Mode
Post by: Vodalian on June 19, 2008, 01:51:38 PM
I think the current concept of Internet mode is pretty flawed right now for one reason. Bots are being teleported back and forth into constantly changing sims with different costs and physics, this really plays havoc on evolution. I think it would be nice if we could have a way of sharing settings, like individual 'ponds' that you can create or join.  Having shared settings would probably yield much more interesting results than the chaos they currently go through.
Title: P2P Internet Mode
Post by: Numsgil on June 19, 2008, 02:02:18 PM
Welcome!

Settings don't necessarily need to be exactly the same.  A little variation might help to create nooks and crannies.  And a bot that can successfully sweep through the sim is all the more impressive.  That said, yes, it would be a good idea to help users share settings for IM.  If they're all totally different, the sims might as well not even be connected.
Title: P2P Internet Mode
Post by: Cyberduke on September 27, 2008, 10:31:48 AM
Hi everyone, I just came across darwinbots.
I was originally just looking to see what was already about in the area of robot programming games (ala C-robots) with a view to creating my own in C#.
But having found Darwinbots have decided not to bother.

Anyway...

I have been reading allot about version 3 design decisions, in regards to networking what is the objective? Is this just for sharing bots in otherwise isolated simulations or are we talking multi-user simulation?

A multi-user simulation might be possible over a classic client-server sockets architecture, which would be the simplest way for networking this kind of game (relatively speaking).
The simulation would be run in a service to which one or more clients could connect and view the current simulation state.

It would better support:-

1)   Evolution Sims, You would be able to run your sim for months in the background as service and just load-up a client viewer every so-often to see what was going on. The service implementation could just save the sim when the service stopped and load it again when the service started.

2)   Different client views on the same data, you could have a basic win forms statistics overview client, a primitive 2D client and a swanky 3D client. These could be developed interpedently from the server and even independently from the server’s platform

3)   Independent simulation speed, it would enforce the total separation of rendering frame rate to simulation frame rate. The client can render as fast as it can, the simulation can simulate as fast as it can and the network can send and receive data as fast as it can, all Independently of each other.

4)   League and shared games, all participants could connect and view the simulation in real-time to check on the progress their entry was making. Users could just set up their own server (simulated environment) and other users can join and introduce their bots and see them interact with other user’s bots.

5)   Performance, it’s going to be the fastest and simplest way of any kind of multi-user simulation. Since this isn’t Counter strike there is no twitch user input or correlating feedback, you only really need to worry about raw bandwidth for sending updates. And in that regard you can just cap and rotate the number of bot updates you send in one tick, you would only need to send a reference ID and accumulated velocity delta’s as regular updates say about 32 bits per bot update, you should be able to manage an update rate of about 32 per tick (in a single UDP packet) with a tick rate of 5 per second, which would give you about 6.7kb/s (160 updates per second) per connection. Obviously for local connections you can achieve a display which approximates the status of the simulation much more tightly by using tick rates closer to that of the simulation frame rates or by updating a larger proportion of the bots in each tick.
Title: P2P Internet Mode
Post by: Peter on September 27, 2008, 10:55:04 AM
Quote
Is this just for sharing bots in otherwise isolated simulations or are we talking multi-user simulation?
Isolated that are shared trough a central server. Altrough that is what Eric is talking about and has been inplementated in DB2.
The title states a P2P-network as you see.

Quote
it would enforce the total separation of rendering frame rate to simulation frame rate
uhuh, seperation rendering/simulation, uhuh. If you mean about shutting the graphical interface down it is also in DB2. I am pretty sure there comes something like it in DB3.

Current way of IM. Simply, central server receivs bots, sends bots and updute IM-graphs. Does nothing else as far I know.
Title: P2P Internet Mode
Post by: Cyberduke on September 27, 2008, 11:28:31 AM
Ah ok I have misunderstood, after skimming though aload of posts about P2P networking I assumed the talk was of a system closer to the like of Terrarium.
Title: P2P Internet Mode
Post by: Cyberduke on September 27, 2008, 12:58:07 PM
And in fact after more reading (so far I have spent most of the day reading this forum) with ideas like complex dynamic cell shapes and joins that could well take a cell update into the realms of 128 bits or more which considering the number of cells and frequency of change would make viewing over the internet far too slow, it would still be fine locally, but then what’s the point!
Title: P2P Internet Mode
Post by: Numsgil on September 27, 2008, 04:24:18 PM
It's not like terrarium, where you have a server running things and you connect to view it (that's how it worked, right?)  Rather, each user is running their own local copy, and bots are periodically teleported to a central depository, where they sit in raw binary form until another user downloads them in to their own sim.  We don't have the money for a fancy central server, so the idea here is to leverage everyone's computing power to create a large universe for bots to roam around in.

BTW, if you're interested I could use a hand with DB3 (in C#), or Eric could use a hand with the current DB2 (in VB6).
Title: P2P Internet Mode
Post by: Cyberduke on September 27, 2008, 05:08:31 PM
Ok I would love to help, It looks like a really interesting project, It’s been like 7 years since I used any VB6 but my C# is current, I am experienced with sockets and GDI+ and a little bit of XNA, but I am sure TAO can't be that hard to learn.
Title: P2P Internet Mode
Post by: Numsgil on September 27, 2008, 05:13:11 PM
Actually I'm leaning towards XNA at the moment instead of TAO and OpenGL, since then I can get Darwinbots running on my XBox360.  Give me a few days (it's milestone week at work) and I'll update the pages on the wiki with instructions for the new source code I've been working on.
Title: P2P Internet Mode
Post by: Cyberduke on September 27, 2008, 05:42:36 PM
I have been wanting to do something for the Xbox for a while (I have one too) It would be great for running and displaying the simulation and even sharing bots about.
does the new v3 have any existing graphical concepts? I.e. to look like you’re looking under a microscope

http://www.youtube.com/watch?v=w8O4OolGcPg...feature=related (http://www.youtube.com/watch?v=w8O4OolGcPg&feature=related)

Is the DNA going to be executed in a single thread (each bots in turn) or in parallel via a thread pool?
Title: P2P Internet Mode
Post by: Numsgil on September 27, 2008, 05:46:20 PM
DNA execution is embarrassingly parallel, so I'm probably going to set up some threading in that regard.

Graphics wise I haven't given it as much thought as possible.  Right now my plans are to make a cleaner version of the current DB2 graphics.  Basically a vector graphic look, with lots of curves that tessellate to a proper level as you zoom in and out.
Title: P2P Internet Mode
Post by: Cyberduke on September 28, 2008, 05:43:48 PM
At the minute I am still suffering from 'so much to read so little time',
especially on all the biological side of things as I have no previous knowledge there.

The P2P networking... has there been any discussion on distributed computing?
As in when your client comes online your added to an existing web as an environment to the left, right, above or below someone else’s?  So Individual bots can wonder off the left of your screen, get taken out of your game and sent to the person on your left whereby it re-appears coming out from right side of their screen?

Would that even be desirable?
Title: P2P Internet Mode
Post by: Numsgil on September 28, 2008, 06:09:01 PM
Yes, we type a lot

I think there's been some discussion along those terms, but a P2P network still requires a server of sorts that can let people connect to each other even if they're behind different LANs.  We've tried to shy away from networking options that require a server other than our webserver.  Which is why we're using FTP right now.
Title: P2P Internet Mode
Post by: Cyberduke on September 28, 2008, 06:24:49 PM
Well yeah it’s really a different style of networking, The FTP route probably is the best for just plain DNA proliferation, and you could even add a database that tracked/maintained a species register for the DNA stored on the FTP server.

This would be a secondary option, and it would be even more important to have similar environments and even similar cycle speeds. But it would allow you to simulate huge worlds containing hundreds of thousands of organisms; although you would want to set limits on organism migration to lessen network traffic and to stop any individual environment from becoming flooded with them.

As for central servers, you just need to create the server program, I am sure someone out there would like to host their own mega simulation especially as it would be they that got to decide some of the common environmental parameters that would be needed to stabilise such a system.
Title: P2P Internet Mode
Post by: Numsgil on September 28, 2008, 06:39:43 PM
It'd be cool, I totally agree.
Title: P2P Internet Mode
Post by: Cyberduke on September 29, 2008, 06:50:40 AM
Also given the proposed modularity of the new system you could have a tertiary networking option that automated Leagues?

You could upload/post/e-mail your entry to the ftp/forum/central server where your entry’s eligibility was checked the DNA validated against the rule set, then passed on to the simulation module where it can run the match and return the results to be post back to the forum.
Title: P2P Internet Mode
Post by: Cyberduke on September 30, 2008, 06:16:17 AM
Ok I have been giving the networking some thought, now you may have to bear with me on this one as I have built a number of private n-tier client/server systems using .net sockets but never a public P2P one. (First time for everything eh.)

I believe I understand the principles though so let’s have a look...
How about a basic old style more or less unstructured P2P network?  I did look for some open source C# P2P libraries but I couldn’t find any decent generic ones (found a good Java one though)
We don’t really need allot of the extra features these libraries come with anyway which tend to focus on file sharing, indexed searches and the like.

Anyway moving on...
When a peer connects they ask the peer they have connected to for their list of known active peers and add that list to their own in a local cache, however each peer only releases a small random selection of known peers (My feeble security attempt at stopping any one peer knowing too much about the whole network at any one time, plus it helps more evenly distribute connections) (Note: Knowledge shared about peers consists of a hostname (using a reverse DNS lookup), a port and no other information)

Now most of these will be behind NAT/firewalls and/or use dynamic IP address etc (Part of the reason for the reverse DNS lookup) so this does rely on most users opening up a port in their firewall and/or a few kind soles (or unwitting victims depending on how you look at it) opting to become ‘proxy nodes’ or ‘hubs’ which will involve that peer maintaining a few more sockets to deal with peers that cannot or won’t accept new connections. (Same for all IP based P2P networks AFAIK) Of course we can also look at HTTP socket tunnels there are lots of them out there or just build one in as a feature.

Each peer will maintain 4 ‘environment sockets’ one for up, down, left and right and if they have an open firewall port  4 ‘network sockets’, one listing socket for new connections and 3 directory sockets for dealing with discovery requests.

Connections on directory sockets will timeout after 60 seconds to keep them clear for new requests, since they are just used to propagate knowledge of the network to new peers, and should probably not accept new connections for say another 60 seconds to lessen the impact of a rouge peer running a DOS attack on the network.

A peer will have a node cache which will come with some built in default host names like darwinbots.com (I don’t mind putting one of my own personal domains in this list) and again I don’t mind having my personal server run a lightweight dedicated peer directory service that just hosts the 4 ‘network sockets’ and known active peer list (without any proxy or environment sockets) I.e.  No heavy lifting.

These are only needed to seed entry to the network however since once a peer has connected it will now know about several more random peers that It can try to connect to (hence the cache approach) so if anyone leaves a peer running for any length of time they will become a good entry point for other peers, and in theory you could then withdraw the default entry points and the network would carry on... and then become self-aware, go by the name of ‘skynet‘ and take over the world with a species of bot it will call T-1000’s.

Anyway...
Upon connecting to the network it will walk though its list of fellow peers asking to hook up an environment socket, if the peer it has connected to is already full (or is not hosting an environment) it will proceed to the next in the list which of course will also transmit another set of random active peers which it can try.
When it does find a peer with an empty environment socket the connection will be transferred to that free socket and the two peers will be linked in one of the 4 directions.


The network will have 4 modes controlled by two flags 1) Evolution Mode and 2) Pond Mode
Don’t want too many as it will dilute the user base and only environments running the same mode can link. Organism migration rates, costs Brownian motion etc. can be set per peer to create diversity but some features will need to be locked to your environment mode such as gravity.

Standard enviroment (Petri dish style)
[attachment=1012:Networke..._Example.png]

Pond mode with gravity
[attachment=1013:Networke..._Example.png]

Err, don’t think I have forgotten anything... this post is getting far too long anyway.
Title: P2P Internet Mode
Post by: Cyberduke on September 30, 2008, 06:31:35 AM
By the way as the more astute of you may have already noticed I haven’t actually thought up a solution for propagating the grid structure yet.

Edit: And of course the reason for the structure is to give the effect of distance and separation of species for evolution, since if we just randomly pluck organisms from one environment and plonk them in another, then you are just keeping everything mixed up (like you are stirring it all about)
Title: P2P Internet Mode
Post by: Cyberduke on September 30, 2008, 09:28:04 AM
I really need to get back to doing some proper work, but before that I found there is information about for dealing with NAT rather than just resorting to tunnelling though it, which might be worth a look when I get some time.

http://www.codeplex.com/Wiki/View.aspx?Pro...Name=SharpStunt (http://www.codeplex.com/Wiki/View.aspx?ProjectName=SharpStunt)
http://www.codeproject.com/KB/IP/upnpnattraversal.aspx (http://www.codeproject.com/KB/IP/upnpnattraversal.aspx)
Title: P2P Internet Mode
Post by: EricL on September 30, 2008, 11:50:22 AM
Just FYI, I'm opposed to P2P on practical grounds.  Although I loved the way Terrarium worked (P2P bot movement with a coordinating rendevuez and statistics server) the security and network tunneling issues were and are a pain in the butt.  Besides, we don't need it.

The client-server semantics of DB internet mode are neither complex nor intensive.   They are similar to (actually simpler than) email systems I.e. small files (or packets of data if we design and build our own protocol/server) representing teleported bots or sim data get uploaded to a server periodically and then shortly downloaded again by another client.   This is neither network bandwidth intensive nor network server cpu intensive.   I claim we could easily achieve scalability of 1000+ connected sims per moderately sized server machine without heroic efforts.  For comparison, corporate emails servers such as Microsoft Exchange exceed this by perhaps an order of magnitude.  And because unlike email systems, the occasional loss of a bot is not serious problem, were we to write our own server, we could avoid server-side persistence of bots (they would flow through memory only with the risk of bot loss in the case the server crashes) and thus we would not be bottlenecked by the I/O capacity of the server machine (a common problem in client-server systems).

For scalability beyond this, multi-tier architectures would serve well.  This is how Hotmail, Google, etc. work.  Note that any topologies we wanted to build (e.g. connect to a specific IM mode such as F1 or Zerobot) could be easily built on top of this.

Anyway, my $0.02.


Title: P2P Internet Mode
Post by: Cyberduke on September 30, 2008, 01:34:53 PM
Quote from: EricL
The client-server semantics of DB internet mode are neither complex nor intensive.   They are similar to (actually simpler than) email systems I.e. small files (or packets of data if we design and build our own protocol/server) representing teleported bots or sim data get uploaded to a server periodically and then shortly downloaded again by another client.   This is neither network bandwidth intensive nor network server cpu intensive.   I claim we could easily achieve scalability of 1000+ connected sims per moderately sized server machine without heroic efforts.  For comparison, corporate emails servers such as Microsoft Exchange exceed this by perhaps an order of magnitude.  And because unlike email systems, the occasional loss of a bot is not serious problem, were we to write our own server, we could avoid server-side persistence of bots (they would flow through memory only with the risk of bot loss in the case the server crashes) and thus we would not be bottlenecked by the I/O capacity of the server machine (a common problem in client-server systems).

Ok better start by saying my comfort zone would actually lay in (just as I believe you have suggested) creating a lightweight TCP/IP(socket) based client/server system where the clients would connect and register themselves with the server and then they would then offer up bots for transfer under a particular conditions (a bot wondering off the screen, or a roaming teleporter, whatever) these are removed from the simulation, uploaded to the server and the server decides who amongst the other connected clients gets to receive this new bot(the instance not just the DNA) and pushes it down to them. (the advantage here being the centralized distribution logic) But I was trying to avoid someone needing to maintain a server with uptime demands upon it. Which the decentralized unstructured P2P route rather nicely solved “in theory” along with scalability, processor load and any bandwidth issues.

And yes just to confirm, under any TCP/IP based architecture we might build I would assume the bot is going to either be serialized and then compressed (quick and dirty) or written out by a specialized routine which can then apply optimizations such as bit-packing. Either way it ends up as a small package of data that needs to be sent to a particular instance of the simulation. I wouldn’t call either the client/server or P2P based approach bandwidth or cpu intensive (I am no longer talking about the single instance multi-user simulation which I first proposed a page or two back, which of course would be)

As far as I can see the P2P route is better in all but complexity and thus development time (which I guess is your point)
I just want to create an ‘Internet Mode’ that is less random and doesn’t mix everything up in one big pot. I actually don’t even mind hosting a dedicated server if we can keep the bandwidth low.

I am less concerned about speed as I am fairly confident that with the new modular and multithreaded approach there will be negligible impact on the simulation whatever we do, and I would expect transfer times via TCP/IP to be under a few seconds even for large multibot.
Title: P2P Internet Mode
Post by: Peter on September 30, 2008, 02:04:35 PM
Quote from: Cyberduke
As far as I can see the P2P route is better in all but development time, and complexity (which I guess is your point)
I just want to create an ‘Internet Mode’ that is less random and doesn’t mix everything up in one big pot. I actually don’t even mind hosting a dedicated server if we can keep the bandwidth low.

I am less concerned about speed as I am fairly confident that with the new modular and multithreaded approach there will be negligible impact on the simulation whatever we do, and I would expect transfer times via IP to be under a few second even for large multibot.
You can make a series of multiple 'controled' sims rather easy. With the inbound outbound teleporters you can rather easy setup multiple sims that are in a certain way connected. (one left out: other right in) Setup a VPN and you can link the different computers together rather easy.

I don't know how much bandwidth IM currently takes, but that can't be that much. So for performance it wouldn't matter that much would it.
Title: P2P Internet Mode
Post by: Cyberduke on September 30, 2008, 02:31:36 PM
Quote from: Peter
Quote from: Cyberduke
As far as I can see the P2P route is better in all but development time, and complexity (which I guess is your point)
I just want to create an ‘Internet Mode’ that is less random and doesn’t mix everything up in one big pot. I actually don’t even mind hosting a dedicated server if we can keep the bandwidth low.

I am less concerned about speed as I am fairly confident that with the new modular and multithreaded approach there will be negligible impact on the simulation whatever we do, and I would expect transfer times via IP to be under a few second even for large multibot.
You can make a series of multiple 'controled' sims rather easy. With the inbound outbound teleporters you can rather easy setup multiple sims that are in a certain way connected. (one left out: other right in) Setup a VPN and you can link the different computers together rather easy.

I don't know how much bandwidth IM currently takes, but that can't be that much. So for performance it wouldn't matter that much would it.

Firstly, darn you quoted me before my edit got published!    

Secondly, I am talking about the Internet Mode in DB3 which doesn’t exist yet, so there are no teleporters yet.

Thirdly, VPN?  Wouldn’t really be an Internet Mode then would it?
Title: P2P Internet Mode
Post by: Cyberduke on September 30, 2008, 02:39:51 PM
Sorry, I probably replied rather too hastily to that,

Yes an advantage to a client/server based approach would also be that lots of people could run their own servers including on a LAN or VPN etc  

And yes it would be my preference to see bots transferred between sims using the borders rather than at random too.
Title: P2P Internet Mode
Post by: Numsgil on September 30, 2008, 02:44:22 PM
A bit off topic, but what are you using to generate diagrams?
Title: P2P Internet Mode
Post by: EricL on September 30, 2008, 02:44:36 PM
Quote from: Cyberduke
Ok better start by saying my comfort zone would actually lay in (just as I believe you have suggested) creating a lightweight TCP/IP(socket) based client/server system where the clients would connect and register themselves with the server and then they would then offer up bots for transfer under a particular conditions (a bot wondering off the screen, or a roaming teleporter, whatever) these are removed from the simulation, uploaded to the server...
Yup, this is how I would do it.  It's basically how it works today, just using files over FTP.  It's more than just bots - you have sim data to aggregate - but a lightweight server wold be pretty easy to build and slot in in place of the current FTP plumbing.

Quote from: Cyberduke
and the server decides who amongst the other connected clients gets to receive this new bot(the instance not just the DNA) and pushes it down to them. (the advantage here being the centralized distribution logic)
Push is the wrong word.  I want no code on clients listening on open ports, otherwise you have NAT, firewall, proxy and security issues.  The client should poll, which achieves the same thing, just with a little latency.  I assume this is what you meant.  

On the subject of connected sim topology and the server making routing decisions, I'm not sure I agree.  We could do connection topologies today without changing the plumbing, but I would want to explore why we should.  Random mixing has certain advantages.

Quote from: Cyberduke
But I was trying to avoid someone needing to maintain a server with uptime demands upon it. Which the decentralized unstructured P2P route rather nicely solved “in theory” along with scalability, processor load and any bandwidth issues.
I host the bulk of IM today.  I would be happy to host a server or two tomorrow.

Quote from: Cyberduke
I just want to create an ‘Internet Mode’ that is less random and doesn’t mix everything up in one big pot.
Make a suggestion in the suggestions forum and I'll consider putting it on the list.

Quote from: Cyberduke
I am less concerned about speed as I am fairly confident that with the new modular and multithreaded approach there will be negligible impact on the simulation whatever we do, and I would expect transfer times via TCP/IP to be under a few seconds even for large multibot.
You are assuming the thing Nums is calling DB3 is a project that will someday supplant the current code base  That is a big assumption, one that is not shared by everyone.  It is much easier (and more likely to happen) that I simply port the current code to VB2008 to get threads and move the code base forward from there.  I have a version limping in VB2008 as we speak.

Title: P2P Internet Mode
Post by: Numsgil on September 30, 2008, 02:52:36 PM
Quote from: EricL
... It is much easier (and more likely to happen) that I simply port the current code to VB2008 to get threads and move the code base forward from there.  I have a version limping in VB2008 as we speak.

I'll believe it when I see it
Title: P2P Internet Mode
Post by: EricL on September 30, 2008, 03:02:08 PM
Quote from: Numsgil
I'll believe it when I see it


My feelings exactly regarding DB3...  


It compiles.  I had to stub out a lot of graphics stuff, which is where most of the differences are.  Now that it's raining in Seattle again, I'll probably get back to it in a week or two...
Title: P2P Internet Mode
Post by: Cyberduke on September 30, 2008, 03:08:00 PM
Quote from: EricL
Push is the wrong word.  I want no code on clients listening on open ports, otherwise you have NAT, firewall, proxy and security issues.  The client should poll, which achieves the same thing, just with a little latency.  I assume this is what you meant.
I was making the assumption that the clients would retain their connection to the server (which they instigated) and so by ‘push’ was simply referring to the server telling the client there was a new bot rather than the client polling the server to ask.

Quote from: EricL
On the subject of connected sim topology and the server making routing decisions, I'm not sure I agree.  We could do connection topologies today without changing the plumbing, but I would want to explore why we should.  Random mixing has certain advantages.
Well the problem I have had with a random mix is that 1 or 2 bots can very quickly sweep though the whole sim, being introduced very early on to each and every instance of the sim, rather than leaving isolated pockets of species to evolve and forcing the aggressive bots to actually expand and conquer rather than helpfully be seeded everywhere.

Quote from: EricL
You are assuming the thing Nums is calling DB3 is a project that will someday supplant the current code base  That is a big assumption, one that is not shared by everyone.  It is much easier (and more likely to happen) that I simply port the current code to VB2008 to get threads and move the code base forward from there.  I have a version limping in VB2008 as we speak.
Ah now that’s interesting you say that, Yes I made the assumption (again) that since the wiki points DB3 in the direction of Nums C# page and you said you where not going to be continuing the project in the long term that Nums C# version was going to eventually become the next generation of Darwinbots
Title: P2P Internet Mode
Post by: EricL on September 30, 2008, 03:24:51 PM
Quote from: Cyberduke
Well the problem I have had with a random mix is that 1 or 2 bots can very quickly sweep though the whole sim, being introduced very early on to each and every instance of the sim, rather than leaving isolated pockets of specials to evolve and forcing the aggressive bots to actually expand and conquer rather than helpfully be seeded everywhere.
You must run relatively small sims.  I run really large ones with shapes and physics such that it takes many thousands of cycles generally for a bot to make it from one side to the other, making colonization of immature sims be new-comers harder .  Additionally, unlike Terrarium, IM participants whose goal is to achieve the highest population for their bots can choose to grow their sim to maturity prior to connecting to IM, thus making it more difficult for any potential colonizer to take over the sim since it is already fully populated.

The fact that teleported-in bots are placed randomly instead of where the teleporter happens to be is deliberate on my part so as to make swarming of newcomers more difficult.    That said, I would be happy to discuss this area further on a different thread.   I'm not necessary opposed to adding sim topology to IM - but there may be better ways to achieve the things you want.

Quote from: Cyberduke
Ah now that’s interesting you say that, Yes I made the assumption (again) that since the wiki points DB3 in the direction of Nums C# page and you said you where not going to be continuing the project in the long term that Nums C# version was going to eventually become the next generation of Darwinbots
I'm not planning on going anywhere but neither can I spend large numbers of hours on DB.  I would welcome help on the current code base or related projects (like a server) but failing that, the community has to settle for me and my limited time commitment.
Title: P2P Internet Mode
Post by: Peter on September 30, 2008, 03:58:12 PM
Quote from: EricL
Quote from: Numsgil
I'll believe it when I see it


My feelings exactly regarding DB3...  
Imagine what the result would be if both of you where going on the same project.

Quote
It compiles.  I had to stub out a lot of graphics stuff, which is where most of the differences are.  Now that it's raining in Seattle again, I'll probably get back to it in a week or two...
I thought compiling to VB2008 had a low priority. Doesn't matter anyway, multiple treads, yeah.
Title: P2P Internet Mode
Post by: Cyberduke on September 30, 2008, 04:22:36 PM
Quote from: EricL
You must run relatively small sims.  I run really large ones with shapes and physics such that it takes many thousands of cycles generally for a bot to make it from one side to the other, making colonization of immature sims be new-comers harder .  Additionally, unlike Terrarium, IM participants whose goal is to achieve the highest population for their bots can choose to grow their sim to maturity prior to connecting to IM, thus making it more difficult for any potential colonizer to take over the sim since it is already fully populated.

The fact that teleported-in bots are placed randomly instead of where the teleporter happens to be is deliberate on my part so as to make swarming of newcomers more difficult.    That said, I would be happy to discuss this area further on a different thread.   I'm not necessary opposed to adding sim topology to IM - but there may be better ways to achieve the things you want.
My original thinking was really coming from the direction of combining multiple sim instances to achieve a distributed computing effect, in order to provide a larger more diverse environment for evolution to act upon. I can quite see that if you are not running mutations and/or are mainly pitting authored bot vs authored bot then random point to point transfers may even be desirable.


Quote from: EricL
I'm not planning on going anywhere but neither can I spend large numbers of hours on DB.  I would welcome help on the current code base or related projects (like a server) but failing that, the community has to settle for me and my limited time commitment.
I have no objection to helping on either project,  it’s the basic idea that interests me the most; I am a C# programmer by trade though so its Nums C# re-write that caught my attention.
Title: P2P Internet Mode
Post by: Numsgil on September 30, 2008, 04:40:40 PM
Quote from: Cyberduke
Quote from: EricL
You are assuming the thing Nums is calling DB3 is a project that will someday supplant the current code base  That is a big assumption, one that is not shared by everyone.  It is much easier (and more likely to happen) that I simply port the current code to VB2008 to get threads and move the code base forward from there.  I have a version limping in VB2008 as we speak.
Ah now that’s interesting you say that, Yes I made the assumption (again) that since the wiki points DB3 in the direction of Nums C# page and you said you where not going to be continuing the project in the long term that Nums C# version was going to eventually become the next generation of Darwinbots

Just to be clear, I don't think most users care what the program is coded in one way or another.  Eric has a great vested interest in the current code base, so the "That is a big assumption, one that is not shared by everyone" is probably code for "That is a big assumption, one that is not shared by me".  From my experience going from 2.37 to early 2.4, stability is more important to end users than features, so if the C# version is always buggier than the current 2.4X code, it'll never catch on.  If it's more stable, it'll catch on.  Part of the problem with the current version is that if something breaks it's not always immediately apparent, since most of the time people aren't using the program so much as writing bots to use the program.  So bugs can sit there for a loooong time.  Which gives the impression to users that the program is buggy, and is an impression that I think isn't too far off the mark.  So I think in this regard there's a lot of room for improvement and thus a lot of leverage for a hypothetical new C# version to convert current users.

With extensive unit testing and a highly modular design I'm hoping bugs will be fewer and further between, and less likely to crop up again in the future (a big problem with the current code base is that it's so brittle that the same bugs seem to get fixed again and again and again).  But I've been working on it a long time and progress is slow since I keep backtracking when something doesn't actually work (lots of work on physics that had to be abandoned, for instance), so it's entirely possible for DB3 just to be vaporware and never realized.  It's also possible it'll get to a somewhat finished state and it'll be too buggy to compete with the current, semi-stable version and all that work will be for naught (though I don't think this particular scenario is likely, again because of unit testing and proper design).

Another scenario is that, assuming Eric can successfully migrate the current code to VB 2005/8, and thus allow .NET interop, the two projects might blend, with newer parts slowly supplanting older parts (or not, depending on their individual merits, of course).  But I'm not holding my breath

Really, it's sort of appropriate, since those are the options for new species to supplant older species.  Either one dominates the other or they interbreed (or they form separate niches.  I guess that's possible, too).  Natural selection will eventually choose a program as the winner in whatever happens, as is entirely appropriate, I think.

So just to reiterate, there are really two versions you could work on.  The current version is in VB, and is used by lots of people.  You can do things and have feedback from end users in very short order, which is a neat experience.  A future speculative version is in C#, and while it might never actually see the light of day, it's far easier to work in and with (unit testing again, plus the C# is easier to understand than VB if you come from a C++ background), and you don't have to worry about backwards compatibility.  Any contributor is free to chose either (or both I suppose) to work on.  All so far, other than Eric, have wanted to work on the C# version.  Admittedly most burn out after a few days or maybe a week or two, with little to show for it, but there you are.  I have no ego to be bruised in this regard, you're free to chose as you see fit
Title: P2P Internet Mode
Post by: Cyberduke on September 30, 2008, 05:24:09 PM
Well yes ok I realise my wording of “going to eventually become the next generation” was a little strong, it was meant in the context of “has the intention of becoming...” due to the perceived wind-down of the current 2.4X code base (formed from comments on the wiki and forums).  
I am familiar with concept of vapourware having worked on a few Half-life mods in the past.

I didn’t appreciate however that both code bases had the intention of continued active development and that “DB3” was actually a branch. Not that it makes a huge difference I suppose, nothing wrong with a bit of healthy competition, but with only two main developers branching off does seem a bit of a waste.

I would be unsure of my competence editing/writing VB6 code and to be honest not that willing to learn, so I don’t mind helping with any .Net work though personally having no current investment in either project would be more tempted by the C# clean re-write, if there is support for one.
Title: P2P Internet Mode
Post by: Numsgil on September 30, 2008, 07:07:39 PM
There's a bit of a history here, actually, that will at the very least provide some backstory.

Originally I was the only active developer (I took over from PY.  There was very little overlap between the two of us working on the program at the same time).  After a handful of versions, I spent the summer after my sophomore year at University to implement some brand new things.  New physics based on springs that was a vast improvement over the old system, which involved lots of bail wire and duct tape.  (This is largely the physics the current version uses, I believe.  Eric has done some tuning and such, but the core is probably pretty much what I built that summer).  I also revamped the DNA to be less structured.  It used to be that genes were rigidly defined as a cond-start-stop triplet.  I spent some effort to allow for more complex or even nonsensical DNA.  There were lots of other changes, too, but those were the big two.  This new version was packaged as 2.4

2.4 had some problems.  Lots of problems, and lots of bugs.  It was really almost entirely unusable.  2.4 was really a failure on my part, having underestimated the amount of time it takes to go from a "feature complete" milestone to something that's stable and ready for end users (and lacking a lot of experience coding for an audience at all).  School had started back up and I was extremely busy and frustrated with the VB code.  Not just fixing bugs that I really should have caught in the first place, really the whole way the language worked.  It made trying to implement various "advanced" features exceptionally hard (there was an abortive attempt at a bucket system to sort bots by buckets (eg: sectors) for faster collision and vision code that was made much harder by the limitations of the language and my ability to adapt to them).

So I stopped working on the VB version, and most people continued to use the older 2.37 version.  Instead, I started porting sections of the code to C++.  This wasn't a rewrite, it was very close to a straight port.  I fixed a few bugs in the process as I worked through the code, but the C++ version would have looked very familiar to someone working on the VB source.  You can probably find the C++ version in some repository somewhere linked on the wiki.  It got pretty far, but I ran in to a brick wall with the physics.  I wasn't using any external libraries or anything, it was a rigid body simulator (rigid bodies being limited to circles exclusively) I'd implemented entirely on my own.  What I wanted to do with multibots just didn't work with the new version (or old version for that matter).  Plus there were some threading issues.  Also, the code, being almost a direct port of the VB code, was still a pain to work with.  Something like the robot class was thousands of lines of convoluted code, with lots of interdependencies between disparate classes (the shots class was a friend to the robot class, and vice versa).  So I gave up on the C++ version and spent some time working through how to actually achieve the sort of physics I wanted.  This was the end of my junior year at University.

Sometime during this process, Eric joined the community and decided he wanted to try and fix issues with the 2.4 source, since it did have some nice features even if it was a bit buggy.  After a few months of effort he was able to get even the hardliners to move from the 2.37 version to the new 2.4X version.  And he's been supporting that version ever since(roughly two years), adding features and fixing bugs as people suggest them/find them.

I, in the mean time, decided to start over with a complete rewrite from scratch.  Because of its similarity (in a good way) with VB, I chose to try again in C#.  To get the ball rolling I wrote about 75% of a DNA module in C# (actually wrote it twice.  The first time to get a taste of the language).  This was right when I first learned about unit testing (maybe a year and a half ago), so the vast majority of the DNA code is unit tested and, I think, pretty stable and bug free.

Then I spent about 8 months working on an entirely different project (A 3D version of something like SimEarth (http://www.gamedev.net/community/forums/topic.asp?topic_id=455477)).  Obviously not a lot happened with Darwinbots during that 8 months.  Eventually Darwinbots coding started to interest me and I put my other project aside.  After some abortive attempts at using existing libraries (most didn't last more than a few weeks) I thought I'd try my hand at building my own physics engine again.  Unfortunately, most of what I wanted to do had never been attempted before, so there was a lot of original research on my part.  That started about a year ago maybe and continued until maybe 3 months ago.  During the 4th of July weekend I scrapped the physics work I was doing and concentrated on getting re-acquainted with my code, reorganizing the file structure and fixing various annoyances with Visual Studio project files and temp directories and such.  Lots of meta programming kind of stuff.  I got my present job as a junior programmer for a video game company last March.  When you program 50-60 hours a week for a job it's hard sometimes to drum up the mental effort to work on part time coding projects, so progress has been... glaciatic.  I've put maybe 100 hours of work in during the last 3 months (though I finished a major milestone last week, so I can relax and go back to 40-50 hour weeks for a while.  Plus I have a 3 day weekend coming up which I'll probably spend, at least in part, working on Darwinbots).

During the time I was either not working on Darwinbots or working on physics, Eric's been supporting the 2.4 code.  He takes a few months off during the summer, but otherwise puts a lot of time and effort in to supporting the current version for users.  He tends to work slow and steady, where I've always been of the burn-out super nova explosion of coding school, where I spend about two weeks doing a massive sprint of thousands of lines of code then do nothing for several months   So Eric's understandably a little skeptical of any claims I make.

Heh, that's a little long winded, but that's how things ended up where they are.
Title: P2P Internet Mode
Post by: EricL on September 30, 2008, 07:49:46 PM
A very fair summary.

Understand that my skepticism regarding DB3 is not a statement about Nums coding skills or tenacity or anything like that.  It's just that I've spent 25 years in the software industry and in that time I've formed some opinions about how to move code bases forward.  Total re-writes never work.  It  sounds great to start clean and do things right from scratch "this time around" but the details and difficulty are always underestimated.  We tend to underestimate how much is really invested in the older code base, all the little things w.r.t. usability or features or hard-won lessons about the way things should work that are embedded in the code and present barriers to people moving.  It's not the rocket science core stuff that kills projects.  It's the million little simple things that add up to a usable program.  This is why is was so difficult for people to move to 2.4 even long after the basics were in place.   Sure, you can get there if you throw enough hours at it for long enough (I did) but in my experience, that is always the most expensive route to take.  

Understand that I've probably doubled the number of lines of code since I took over.   There are a lot of details, a lot of features, a lot of nuances.  Even were DB3 working today, it would take a long time to get to critical mass such that people could move.  I prefer the incremental approach.  One rule I learned the hard way and have followed in the large software projects I've managed is that you never, ever change more than 30% of the code from major release to major release.  You can do re-writes, but you do it of individual modules.  You move forward, incrementally, maintaining stability while you do.  Not doing so is the fastest path to vaporware.

So, I'm skeptical not only of "DB3" but of any project that does not start from the current code base and move forward incrementally.
Title: P2P Internet Mode
Post by: EricL on September 30, 2008, 08:04:32 PM
Quote from: Cyberduke
My original thinking was really coming from the direction of combining multiple sim instances to achieve a distributed computing effect, in order to provide a larger more diverse environment for evolution to act upon.
This is exactly why (well, one reason at least) I implemented private teleporters.  Unlike the IM teleporter, inbound private teleporters do not randomly distribute incoming bots across the field.  Bots stay where the teleporter is.  It's a doorway for incoming bots just like outbound teleporters are an exit.   They can be made stationary, perhaps surrounded by shapes, to create different fiefdoms or whatever other effect you want or they can be made to drift, ignoring or respecting shapes as you like.  They can filter bots to restrict or allow veggies or corpses and such.  You can have multiple pairs connecting multiple sims in whatever topology you want.   And, it is on my list to unify shapes and teleporters to provide greater flexibility on size including making entire field edges themselves act as teleporters.  So you see, we think very much alike here...
Title: P2P Internet Mode
Post by: Cyberduke on October 01, 2008, 05:07:27 AM
Quote from: EricL
It's just that I've spent 25 years in the software industry and in that time I've formed some opinions about how to move code bases forward. Total re-writes never work. It sounds great to start clean and do things right from scratch "this time around" but the details and difficulty are always underestimated. We tend to underestimate how much is really invested in the older code base, all the little things w.r.t. usability or features or hard-won lessons about the way things should work that are embedded in the code and present barriers to people moving. It's not the rocket science core stuff that kills projects. It's the million little simple things that add up to a usable program. This is why is was so difficult for people to move to 2.4 even long after the basics were in place. Sure, you can get there if you throw enough hours at it for long enough (I did) but in my experience, that is always the most expensive route to take.

Understand that I've probably doubled the number of lines of code since I took over. There are a lot of details, a lot of features, a lot of nuances. Even were DB3 working today, it would take a long time to get to critical mass such that people could move. I prefer the incremental approach. One rule I learned the hard way and have followed in the large software projects I've managed is that you never, ever change more than 30% of the code from major release to major release. You can do re-writes, but you do it of individual modules. You move forward, incrementally, maintaining stability while you do. Not doing so is the fastest path to vaporware.
Sounds like some very good advice there, *puts fingers in ears* 'laa la laaa la'    
You’re absolutely right; I know you are, its just from my point of view there would also be the overhead of learning a new...old... different language  and trying to get to grips with a large complex existing code base written in multiple styles by multiple developers over a long period of time.
On the other hand, a new start has an extra fun-factor about it and would give me the ability to understand it from the start as it was being laid down and for me to incorporate some of my own ideas into it, even if it does come with an increased risk of wasted effort.

Besides at work recently I am getting used to re-writing stuff, it's a constant stream of “here is our old legacy program, the vender went bust about 8 years ago and so there is no source available, Can you build us a new version that’s 10x better inside 6 months please”

Anyway...
It’s just a shame there would really be a duplication of effort reaching for a very similar goal which doesn’t quite sit right.

And for the record I probably sit closer to Nums school of coding, though maybe not quite so extreme, having had to earn a living as a programmer for 6 years has tempered that tendency somewhat.


Edit: Also even If the 2.4x code base wasn’t going to stay in active development I understand that it would take a lot of time before 3 could begin to compete for any percentage of the user base since the first number of releases would really by necessity have a greatly reduced feature set, and would then take a while to stabilize and for new features to be built on top of that.

2nd Edit: that and 3 looks like it just about decimates any chance of running existing DNA code on it.
Title: P2P Internet Mode
Post by: Cyberduke on October 01, 2008, 06:36:35 AM
Oh and Numsgil, that was a very long and informative post, which was greatly appreciated and congrats' on 6,000 posts!
Title: P2P Internet Mode
Post by: Numsgil on October 01, 2008, 01:32:44 PM
Woo 6K!

The current DNA (codenamed Sunweaver) for v3 isn't compatible with present bots, but being modular by nature it's possible a future module could be written to allow for current bots to run.  It just isn't my priority at the moment.