Author Topic: P2P Internet Mode  (Read 18407 times)

Offline Numsgil

  • Administrator
  • Bot God
  • *****
  • Posts: 7742
    • View Profile
P2P Internet Mode
« Reply #30 on: September 28, 2008, 06:09:01 PM »
Yes, we type a lot

I think there's been some discussion along those terms, but a P2P network still requires a server of sorts that can let people connect to each other even if they're behind different LANs.  We've tried to shy away from networking options that require a server other than our webserver.  Which is why we're using FTP right now.

Offline Cyberduke

  • Bot Builder
  • **
  • Posts: 88
    • View Profile
P2P Internet Mode
« Reply #31 on: September 28, 2008, 06:24:49 PM »
Well yeah it’s really a different style of networking, The FTP route probably is the best for just plain DNA proliferation, and you could even add a database that tracked/maintained a species register for the DNA stored on the FTP server.

This would be a secondary option, and it would be even more important to have similar environments and even similar cycle speeds. But it would allow you to simulate huge worlds containing hundreds of thousands of organisms; although you would want to set limits on organism migration to lessen network traffic and to stop any individual environment from becoming flooded with them.

As for central servers, you just need to create the server program, I am sure someone out there would like to host their own mega simulation especially as it would be they that got to decide some of the common environmental parameters that would be needed to stabilise such a system.

Offline Numsgil

  • Administrator
  • Bot God
  • *****
  • Posts: 7742
    • View Profile
P2P Internet Mode
« Reply #32 on: September 28, 2008, 06:39:43 PM »
It'd be cool, I totally agree.

Offline Cyberduke

  • Bot Builder
  • **
  • Posts: 88
    • View Profile
P2P Internet Mode
« Reply #33 on: September 29, 2008, 06:50:40 AM »
Also given the proposed modularity of the new system you could have a tertiary networking option that automated Leagues?

You could upload/post/e-mail your entry to the ftp/forum/central server where your entry’s eligibility was checked the DNA validated against the rule set, then passed on to the simulation module where it can run the match and return the results to be post back to the forum.

Offline Cyberduke

  • Bot Builder
  • **
  • Posts: 88
    • View Profile
P2P Internet Mode
« Reply #34 on: September 30, 2008, 06:16:17 AM »
Ok I have been giving the networking some thought, now you may have to bear with me on this one as I have built a number of private n-tier client/server systems using .net sockets but never a public P2P one. (First time for everything eh.)

I believe I understand the principles though so let’s have a look...
How about a basic old style more or less unstructured P2P network?  I did look for some open source C# P2P libraries but I couldn’t find any decent generic ones (found a good Java one though)
We don’t really need allot of the extra features these libraries come with anyway which tend to focus on file sharing, indexed searches and the like.

Anyway moving on...
When a peer connects they ask the peer they have connected to for their list of known active peers and add that list to their own in a local cache, however each peer only releases a small random selection of known peers (My feeble security attempt at stopping any one peer knowing too much about the whole network at any one time, plus it helps more evenly distribute connections) (Note: Knowledge shared about peers consists of a hostname (using a reverse DNS lookup), a port and no other information)

Now most of these will be behind NAT/firewalls and/or use dynamic IP address etc (Part of the reason for the reverse DNS lookup) so this does rely on most users opening up a port in their firewall and/or a few kind soles (or unwitting victims depending on how you look at it) opting to become ‘proxy nodes’ or ‘hubs’ which will involve that peer maintaining a few more sockets to deal with peers that cannot or won’t accept new connections. (Same for all IP based P2P networks AFAIK) Of course we can also look at HTTP socket tunnels there are lots of them out there or just build one in as a feature.

Each peer will maintain 4 ‘environment sockets’ one for up, down, left and right and if they have an open firewall port  4 ‘network sockets’, one listing socket for new connections and 3 directory sockets for dealing with discovery requests.

Connections on directory sockets will timeout after 60 seconds to keep them clear for new requests, since they are just used to propagate knowledge of the network to new peers, and should probably not accept new connections for say another 60 seconds to lessen the impact of a rouge peer running a DOS attack on the network.

A peer will have a node cache which will come with some built in default host names like darwinbots.com (I don’t mind putting one of my own personal domains in this list) and again I don’t mind having my personal server run a lightweight dedicated peer directory service that just hosts the 4 ‘network sockets’ and known active peer list (without any proxy or environment sockets) I.e.  No heavy lifting.

These are only needed to seed entry to the network however since once a peer has connected it will now know about several more random peers that It can try to connect to (hence the cache approach) so if anyone leaves a peer running for any length of time they will become a good entry point for other peers, and in theory you could then withdraw the default entry points and the network would carry on... and then become self-aware, go by the name of ‘skynet‘ and take over the world with a species of bot it will call T-1000’s.

Anyway...
Upon connecting to the network it will walk though its list of fellow peers asking to hook up an environment socket, if the peer it has connected to is already full (or is not hosting an environment) it will proceed to the next in the list which of course will also transmit another set of random active peers which it can try.
When it does find a peer with an empty environment socket the connection will be transferred to that free socket and the two peers will be linked in one of the 4 directions.


The network will have 4 modes controlled by two flags 1) Evolution Mode and 2) Pond Mode
Don’t want too many as it will dilute the user base and only environments running the same mode can link. Organism migration rates, costs Brownian motion etc. can be set per peer to create diversity but some features will need to be locked to your environment mode such as gravity.

Standard enviroment (Petri dish style)
[attachment=1012:Networke..._Example.png]

Pond mode with gravity
[attachment=1013:Networke..._Example.png]

Err, don’t think I have forgotten anything... this post is getting far too long anyway.
« Last Edit: September 30, 2008, 07:06:05 AM by Cyberduke »

Offline Cyberduke

  • Bot Builder
  • **
  • Posts: 88
    • View Profile
P2P Internet Mode
« Reply #35 on: September 30, 2008, 06:31:35 AM »
By the way as the more astute of you may have already noticed I haven’t actually thought up a solution for propagating the grid structure yet.

Edit: And of course the reason for the structure is to give the effect of distance and separation of species for evolution, since if we just randomly pluck organisms from one environment and plonk them in another, then you are just keeping everything mixed up (like you are stirring it all about)
« Last Edit: September 30, 2008, 08:44:46 AM by Cyberduke »

Offline Cyberduke

  • Bot Builder
  • **
  • Posts: 88
    • View Profile
P2P Internet Mode
« Reply #36 on: September 30, 2008, 09:28:04 AM »
I really need to get back to doing some proper work, but before that I found there is information about for dealing with NAT rather than just resorting to tunnelling though it, which might be worth a look when I get some time.

http://www.codeplex.com/Wiki/View.aspx?Pro...Name=SharpStunt
http://www.codeproject.com/KB/IP/upnpnattraversal.aspx

Offline EricL

  • Administrator
  • Bot God
  • *****
  • Posts: 2266
    • View Profile
P2P Internet Mode
« Reply #37 on: September 30, 2008, 11:50:22 AM »
Just FYI, I'm opposed to P2P on practical grounds.  Although I loved the way Terrarium worked (P2P bot movement with a coordinating rendevuez and statistics server) the security and network tunneling issues were and are a pain in the butt.  Besides, we don't need it.

The client-server semantics of DB internet mode are neither complex nor intensive.   They are similar to (actually simpler than) email systems I.e. small files (or packets of data if we design and build our own protocol/server) representing teleported bots or sim data get uploaded to a server periodically and then shortly downloaded again by another client.   This is neither network bandwidth intensive nor network server cpu intensive.   I claim we could easily achieve scalability of 1000+ connected sims per moderately sized server machine without heroic efforts.  For comparison, corporate emails servers such as Microsoft Exchange exceed this by perhaps an order of magnitude.  And because unlike email systems, the occasional loss of a bot is not serious problem, were we to write our own server, we could avoid server-side persistence of bots (they would flow through memory only with the risk of bot loss in the case the server crashes) and thus we would not be bottlenecked by the I/O capacity of the server machine (a common problem in client-server systems).

For scalability beyond this, multi-tier architectures would serve well.  This is how Hotmail, Google, etc. work.  Note that any topologies we wanted to build (e.g. connect to a specific IM mode such as F1 or Zerobot) could be easily built on top of this.

Anyway, my $0.02.


Many beers....

Offline Cyberduke

  • Bot Builder
  • **
  • Posts: 88
    • View Profile
P2P Internet Mode
« Reply #38 on: September 30, 2008, 01:34:53 PM »
Quote from: EricL
The client-server semantics of DB internet mode are neither complex nor intensive.   They are similar to (actually simpler than) email systems I.e. small files (or packets of data if we design and build our own protocol/server) representing teleported bots or sim data get uploaded to a server periodically and then shortly downloaded again by another client.   This is neither network bandwidth intensive nor network server cpu intensive.   I claim we could easily achieve scalability of 1000+ connected sims per moderately sized server machine without heroic efforts.  For comparison, corporate emails servers such as Microsoft Exchange exceed this by perhaps an order of magnitude.  And because unlike email systems, the occasional loss of a bot is not serious problem, were we to write our own server, we could avoid server-side persistence of bots (they would flow through memory only with the risk of bot loss in the case the server crashes) and thus we would not be bottlenecked by the I/O capacity of the server machine (a common problem in client-server systems).

Ok better start by saying my comfort zone would actually lay in (just as I believe you have suggested) creating a lightweight TCP/IP(socket) based client/server system where the clients would connect and register themselves with the server and then they would then offer up bots for transfer under a particular conditions (a bot wondering off the screen, or a roaming teleporter, whatever) these are removed from the simulation, uploaded to the server and the server decides who amongst the other connected clients gets to receive this new bot(the instance not just the DNA) and pushes it down to them. (the advantage here being the centralized distribution logic) But I was trying to avoid someone needing to maintain a server with uptime demands upon it. Which the decentralized unstructured P2P route rather nicely solved “in theory” along with scalability, processor load and any bandwidth issues.

And yes just to confirm, under any TCP/IP based architecture we might build I would assume the bot is going to either be serialized and then compressed (quick and dirty) or written out by a specialized routine which can then apply optimizations such as bit-packing. Either way it ends up as a small package of data that needs to be sent to a particular instance of the simulation. I wouldn’t call either the client/server or P2P based approach bandwidth or cpu intensive (I am no longer talking about the single instance multi-user simulation which I first proposed a page or two back, which of course would be)

As far as I can see the P2P route is better in all but complexity and thus development time (which I guess is your point)
I just want to create an ‘Internet Mode’ that is less random and doesn’t mix everything up in one big pot. I actually don’t even mind hosting a dedicated server if we can keep the bandwidth low.

I am less concerned about speed as I am fairly confident that with the new modular and multithreaded approach there will be negligible impact on the simulation whatever we do, and I would expect transfer times via TCP/IP to be under a few seconds even for large multibot.
« Last Edit: September 30, 2008, 02:22:43 PM by Cyberduke »

Offline Peter

  • Bot God
  • *****
  • Posts: 1177
    • View Profile
P2P Internet Mode
« Reply #39 on: September 30, 2008, 02:04:35 PM »
Quote from: Cyberduke
As far as I can see the P2P route is better in all but development time, and complexity (which I guess is your point)
I just want to create an ‘Internet Mode’ that is less random and doesn’t mix everything up in one big pot. I actually don’t even mind hosting a dedicated server if we can keep the bandwidth low.

I am less concerned about speed as I am fairly confident that with the new modular and multithreaded approach there will be negligible impact on the simulation whatever we do, and I would expect transfer times via IP to be under a few second even for large multibot.
You can make a series of multiple 'controled' sims rather easy. With the inbound outbound teleporters you can rather easy setup multiple sims that are in a certain way connected. (one left out: other right in) Setup a VPN and you can link the different computers together rather easy.

I don't know how much bandwidth IM currently takes, but that can't be that much. So for performance it wouldn't matter that much would it.
Oh my god, who the hell cares.

Offline Cyberduke

  • Bot Builder
  • **
  • Posts: 88
    • View Profile
P2P Internet Mode
« Reply #40 on: September 30, 2008, 02:31:36 PM »
Quote from: Peter
Quote from: Cyberduke
As far as I can see the P2P route is better in all but development time, and complexity (which I guess is your point)
I just want to create an ‘Internet Mode’ that is less random and doesn’t mix everything up in one big pot. I actually don’t even mind hosting a dedicated server if we can keep the bandwidth low.

I am less concerned about speed as I am fairly confident that with the new modular and multithreaded approach there will be negligible impact on the simulation whatever we do, and I would expect transfer times via IP to be under a few second even for large multibot.
You can make a series of multiple 'controled' sims rather easy. With the inbound outbound teleporters you can rather easy setup multiple sims that are in a certain way connected. (one left out: other right in) Setup a VPN and you can link the different computers together rather easy.

I don't know how much bandwidth IM currently takes, but that can't be that much. So for performance it wouldn't matter that much would it.

Firstly, darn you quoted me before my edit got published!    

Secondly, I am talking about the Internet Mode in DB3 which doesn’t exist yet, so there are no teleporters yet.

Thirdly, VPN?  Wouldn’t really be an Internet Mode then would it?

Offline Cyberduke

  • Bot Builder
  • **
  • Posts: 88
    • View Profile
P2P Internet Mode
« Reply #41 on: September 30, 2008, 02:39:51 PM »
Sorry, I probably replied rather too hastily to that,

Yes an advantage to a client/server based approach would also be that lots of people could run their own servers including on a LAN or VPN etc  

And yes it would be my preference to see bots transferred between sims using the borders rather than at random too.

Offline Numsgil

  • Administrator
  • Bot God
  • *****
  • Posts: 7742
    • View Profile
P2P Internet Mode
« Reply #42 on: September 30, 2008, 02:44:22 PM »
A bit off topic, but what are you using to generate diagrams?

Offline EricL

  • Administrator
  • Bot God
  • *****
  • Posts: 2266
    • View Profile
P2P Internet Mode
« Reply #43 on: September 30, 2008, 02:44:36 PM »
Quote from: Cyberduke
Ok better start by saying my comfort zone would actually lay in (just as I believe you have suggested) creating a lightweight TCP/IP(socket) based client/server system where the clients would connect and register themselves with the server and then they would then offer up bots for transfer under a particular conditions (a bot wondering off the screen, or a roaming teleporter, whatever) these are removed from the simulation, uploaded to the server...
Yup, this is how I would do it.  It's basically how it works today, just using files over FTP.  It's more than just bots - you have sim data to aggregate - but a lightweight server wold be pretty easy to build and slot in in place of the current FTP plumbing.

Quote from: Cyberduke
and the server decides who amongst the other connected clients gets to receive this new bot(the instance not just the DNA) and pushes it down to them. (the advantage here being the centralized distribution logic)
Push is the wrong word.  I want no code on clients listening on open ports, otherwise you have NAT, firewall, proxy and security issues.  The client should poll, which achieves the same thing, just with a little latency.  I assume this is what you meant.  

On the subject of connected sim topology and the server making routing decisions, I'm not sure I agree.  We could do connection topologies today without changing the plumbing, but I would want to explore why we should.  Random mixing has certain advantages.

Quote from: Cyberduke
But I was trying to avoid someone needing to maintain a server with uptime demands upon it. Which the decentralized unstructured P2P route rather nicely solved “in theory” along with scalability, processor load and any bandwidth issues.
I host the bulk of IM today.  I would be happy to host a server or two tomorrow.

Quote from: Cyberduke
I just want to create an ‘Internet Mode’ that is less random and doesn’t mix everything up in one big pot.
Make a suggestion in the suggestions forum and I'll consider putting it on the list.

Quote from: Cyberduke
I am less concerned about speed as I am fairly confident that with the new modular and multithreaded approach there will be negligible impact on the simulation whatever we do, and I would expect transfer times via TCP/IP to be under a few seconds even for large multibot.
You are assuming the thing Nums is calling DB3 is a project that will someday supplant the current code base  That is a big assumption, one that is not shared by everyone.  It is much easier (and more likely to happen) that I simply port the current code to VB2008 to get threads and move the code base forward from there.  I have a version limping in VB2008 as we speak.

Many beers....

Offline Numsgil

  • Administrator
  • Bot God
  • *****
  • Posts: 7742
    • View Profile
P2P Internet Mode
« Reply #44 on: September 30, 2008, 02:52:36 PM »
Quote from: EricL
... It is much easier (and more likely to happen) that I simply port the current code to VB2008 to get threads and move the code base forward from there.  I have a version limping in VB2008 as we speak.

I'll believe it when I see it