Author Topic: P2P Internet Mode  (Read 18409 times)

Offline goffrie

  • Bot Builder
  • **
  • Posts: 65
    • View Profile
P2P Internet Mode
« on: May 29, 2008, 08:34:00 PM »
It would be nice if there was a peer-to-peer network of sorts for Internet Mode in Darwinbots3, in which all capable clients would be able to connect to each other directly. Of course, there would still be a central server, but it would reduce the load on the central server, and a relay system could be set up to allow clients behind firewalls to connect to IM (maybe falling back to FTP). It would also be possible this way to have relatively real-time organism exchanges, as the server could tell the clients when a new organism was available and clients could download it as soon as they were told (the server could randomly choose a client to give it to). If the Darwinbots server can't run custom protocols, it could probably be managed over HTTP. The protocol itself should be fairly trivial, as it does not need to support a lot of features (maybe just compression), and efficiency is not a major issue. The only issue I see with this is that many people run behind routers and the system does not really work that way .

Of course, as Darwinbots3 is still early in development, it's not really time just yet to be implementing these kinds of things, but it's just an idea .

Offline Numsgil

  • Administrator
  • Bot God
  • *****
  • Posts: 7742
    • View Profile
P2P Internet Mode
« Reply #1 on: May 29, 2008, 08:46:33 PM »
I've played with the idea of having Darwinbots connect to the web server and run some sort of PHP script to upload and download bots.  If you could connect to the home page or forum, you could run IM.  But I'm about as green as it comes with PHP scripts.  I don't even know if what I want to do is possible (though I imagine it is...)

Offline goffrie

  • Bot Builder
  • **
  • Posts: 65
    • View Profile
P2P Internet Mode
« Reply #2 on: May 29, 2008, 09:03:14 PM »
Hmm, I'm pretty good with PHP  
It definitely is possible, using HTTP POST with multipart/form-data. It could even work with ordinary POST (that wouldn't work with a web browser though). If you wanted, I could cook up a PHP script which would handle organism uploads. I have no idea how you would interface it with Darwinbots though, and you'd have to tell me where to store the files  .

Another thought, for P2P, perhaps it could interface with Tor and hidden services? That would solve any problems with NATs (firewalls would still be a problem though). It would be pretty finicky(sp?) and slow though.
« Last Edit: May 29, 2008, 09:16:24 PM by goffrie »

Offline Numsgil

  • Administrator
  • Bot God
  • *****
  • Posts: 7742
    • View Profile
P2P Internet Mode
« Reply #3 on: May 29, 2008, 09:25:16 PM »
Yeah, I was looking at POST commands.  But I didn't understand it well enough and dropped the idea.  I would certainly appreciate any help writing the PHP script.  I can probably play with it if I had one that worked.  We could probably even set up the existing DB2 to use it...

What needs to happen is that I can call something like this:

www.darwinbots.com/IMMode/UploadScript.php?file="C:\Program Files\Darwinbots3\IM\out\Mybot256734532342344.dbo"

Or whatever the syntax would be.  It can't have a "choose the file" dialog pop up, since it needs to be entirely automated and behind the scenes.  UploadScript.php would then upload that file to some directory on the server that I could set.  It should also have a file size limit that I could set.

A similar script called DownloadScript.php?directory="C:\Program Files\Darwinbots3\IM\in\" would somehow download a binary file from the IM directory on the server and store it in the local directory specified, and then delete the file from the server.  It also can't have one of those "where do I download this too" dialogs pop up, since it also needs to be entirely behind the scenes and automatic.

I think I know how to call a URL from Darwinbots, so if you could cook up some PHPs like that, that I could put on the server and call, that would be very cool indeed.
« Last Edit: May 29, 2008, 09:26:37 PM by Numsgil »

Offline goffrie

  • Bot Builder
  • **
  • Posts: 65
    • View Profile
P2P Internet Mode
« Reply #4 on: May 29, 2008, 09:41:31 PM »
You can't really tell the web client to upload a file like that. (It would be nice though ) You need to read the file and upload it as POST data, like "data=buibtuisnuiremhtr5398nf9324==" (base64 encoded or otherwise made 7-bit friendly). Then you could download the file from the server. I don't know if it's possible in VB, but it's almost certainly doable with C#.
« Last Edit: May 29, 2008, 09:47:03 PM by goffrie »

Offline Numsgil

  • Administrator
  • Bot God
  • *****
  • Posts: 7742
    • View Profile
P2P Internet Mode
« Reply #5 on: May 29, 2008, 10:02:11 PM »
You mean encode the entire binary file in the url?  Speaking of which is there any sort of limit on the length of a url?  Like could you have a url that's thousands of characters?
« Last Edit: May 29, 2008, 10:03:43 PM by Numsgil »

Offline goffrie

  • Bot Builder
  • **
  • Posts: 65
    • View Profile
P2P Internet Mode
« Reply #6 on: May 30, 2008, 07:45:09 AM »
Not quite, POST data is like GET data (which is in the URL) but it goes after the request headers. No idea how to do that in VB or C#, though.
You might be able to encode the data in the URL, but the web server would probably reject it as a buffer overflow attack, (or the server would have a buffer overflow).

Offline Numsgil

  • Administrator
  • Bot God
  • *****
  • Posts: 7742
    • View Profile
P2P Internet Mode
« Reply #7 on: May 30, 2008, 01:20:50 PM »
Now we're running in to my ignorance.  Perhaps a short example?

Offline EricL

  • Administrator
  • Bot God
  • *****
  • Posts: 2266
    • View Profile
P2P Internet Mode
« Reply #8 on: May 30, 2008, 02:36:09 PM »
Quote from: goffrie
It would be nice if there was a peer-to-peer network of sorts for Internet Mode...
This is exactly the way Terrarium worked.  The server did nothing but act as a rendevuez service and collect stats.  Bots moved between peers.  I agree with you in general, but....   Not only is it a lot of work, but there are serious security issues to overcome in such an architecture depending upon the protocol and the means by which the P2P communication is acheived.  Anytime you start running server code (meaning code that listens for new requests on a port and performs work based on those requests) on clients, you open up a serious security can of works not to mention isuses with NATs and firewalls, etc.

People should not confuse the client connectivity topology we choose to use with transport latency issues or with the connectivty problems inherent in the current FPT-based design.  The current FTP based thing is problematic to be sure, but the connectivity problems are not inherent problems with the design.  Those can be fixed by moving to a more reilable server and moving the server communication code out of proc, both of which I am working on (using in part, some code from Nums).  

But we will reach scalability limits on the current architecture.  Understand, there is no real server today.  No server that is, other than the FTP server itself.  There is no code authored by me running on any machine except the client simulators each of us run.  They cooperate through the FTP server by moving files, but in an incredibly ineffecient manner becuase they are forced to shoe horn their semantics into file movements and the semantics of the FTP protocol.  Some poeple call this a shared-file or file-sharing version of cleint-server.  Early corporate email systems (circa 1990) such as CC:Mail and Mcirosoft Mail used this archtiecture.   It is not true client-server (like POP/SMTP).  Were we to move to a true client-server model where we had special code runnnig on a server, implimenting our specific semantics, we could easily scale to several thousand sims connected through a single server.  Buidling this on top of HTTP or PHP or sockets or whatever is mostly a discussion of ease of implimentation.  The result is the same, a true cleint-server architecture.

Scaling beyond that would require mutliple servers with connection logic or partitioning or a P2P architecuture.
Many beers....

Offline Numsgil

  • Administrator
  • Bot God
  • *****
  • Posts: 7742
    • View Profile
P2P Internet Mode
« Reply #9 on: May 30, 2008, 04:10:32 PM »
As a followup to my last post...  There are there any problems having the download script save data to disk?  If there is, is it possible to embed binary data in the HTTP the PHP script returns?  If that wouldn't work either, an text based organism file would probably be possible.  I know for sure that would work...

Which leaves uploading the file, which is where I don't know how to proceed.

Quote from: EricL
Scaling beyond that would require mutliple servers with connection logic or partitioning or a P2P architecuture.

If we use a PHP sheme, we can leave the details of managing multiple servers to our web host.  As the number of users increase, it would just change the amount of bandwidth we would need to pay for (keeping in mind that .dbo files are small and our present bandwidth is only at like 2% utilization, I don't think there's going to be a problem).  If the file accesses start become a problem, it would be a simple matter to move from a file scheme to something in a database.

Offline goffrie

  • Bot Builder
  • **
  • Posts: 65
    • View Profile
P2P Internet Mode
« Reply #10 on: May 31, 2008, 08:55:05 PM »
Quote from: EricL
there are serious security issues to overcome in such an architecture depending upon the protocol and the means by which the P2P communication is acheived.  Anytime you start running server code (meaning code that listens for new requests on a port and performs work based on those requests) on clients, you open up a serious security can of works not to mention isuses with NATs and firewalls, etc.
Security issues perhaps are a problem, but they are only moderately severe as long as code is kept well-managed. Issues with NATs can be overcome by relaying through the central server.

Quote
People should not confuse the client connectivity topology we choose to use with transport latency issues or with the connectivty problems inherent in the current FPT-based design.  The current FTP based thing is problematic to be sure, but the connectivity problems are not inherent problems with the design.  Those can be fixed by moving to a more reilable server and moving the server communication code out of proc, both of which I am working on (using in part, some code from Nums).
I'm not talking about latency issues or connectivity problems, I'm just talking about a peer-to-peer system. Moving to a more reliable server would help with thse issues, though, which is definitely appreciated.
"moving the server communication code out of proc" - I'm not too sure what you meant by this.
 
Quote
But we will reach scalability limits on the current architecture.  Understand, there is no real server today.  No server that is, other than the FTP server itself.
Which is a real server.

Quote
There is no code authored by me running on any machine except the client simulators each of us run. They cooperate through the FTP server by moving files, but in an incredibly ineffecient manner becuase they are forced to shoe horn their semantics into file movements and the semantics of the FTP protocol.  Some poeple call this a shared-file or file-sharing version of cleint-server.  Early corporate email systems (circa 1990) such as CC:Mail and Mcirosoft Mail used this archtiecture.   It is not true client-server (like POP/SMTP).  Were we to move to a true client-server model where we had special code runnnig on a server, implimenting our specific semantics, we could easily scale to several thousand sims connected through a single server.  Buidling this on top of HTTP or PHP or sockets or whatever is mostly a discussion of ease of implimentation.  The result is the same, a true cleint-server architecture. Scaling beyond that would require mutliple servers with connection logic or partitioning or a P2P architecuture.
... Why the lecture? I don't see what point you're trying to make.

Quote
As a followup to my last post... There are there any problems having the download script save data to disk? If there is, is it possible to embed binary data in the HTTP the PHP script returns? If that wouldn't work either, an text based organism file would probably be possible. I know for sure that would work...
Yes, you can put binary data into an HTTP stream - how do you think binary downloads work?

Quote
Which leaves uploading the file, which is where I don't know how to proceed.
Neither do I, since I'm not familiar with VB.

Offline Numsgil

  • Administrator
  • Bot God
  • *****
  • Posts: 7742
    • View Profile
P2P Internet Mode
« Reply #11 on: May 31, 2008, 10:06:48 PM »
Quote from: goffrie
Quote
As a followup to my last post... There are there any problems having the download script save data to disk? If there is, is it possible to embed binary data in the HTTP the PHP script returns? If that wouldn't work either, an text based organism file would probably be possible. I know for sure that would work...
Yes, you can put binary data into an HTTP stream - how do you think binary downloads work?  

Magic lawn leprachauns.  You ask for the file you want to download, and they use their magic to force their way in to a parallel dimension where I already have the file.  They then murder than alternate me, and steal his computer's hard drive.  Using their magics they teleport the bits from the stolen hard drive to my own.  And voila, I have the file I wanted to download

Quote
Quote
Which leaves uploading the file, which is where I don't know how to proceed.
Neither do I, since I'm not familiar with VB.

Well, let's ignore the whole VB thing for a moment.  If this were for a web site, and we wanted someone to click on a link that would automatically upload a certain file, how would that work?

Offline goffrie

  • Bot Builder
  • **
  • Posts: 65
    • View Profile
P2P Internet Mode
« Reply #12 on: June 01, 2008, 11:28:12 AM »
Quote from: Numsgil
Magic lawn leprachauns.  You ask for the file you want to download, and they use their magic to force their way in to a parallel dimension where I already have the file.  They then murder than alternate me, and steal his computer's hard drive.  Using their magics they teleport the bits from the stolen hard drive to my own.  And voila, I have the file I wanted to download


Quote from: Numsgil
Well, let's ignore the whole VB thing for a moment.  If this were for a web site, and we wanted someone to click on a link that would automatically upload a certain file, how would that work?
It wouldn't - you can't force a user to upload a file without using client-side things like Javascript (and even then I don't think that would work.) Basically, it poses too much of a security risk to work.
The VB part comes in because you need to voluntarily put the data stream into the request.

Hmm, I found this information for VB6:
Quote
The transaction with Google happen via HTTP POST requests. There is a control available in VS6 that allows you to perform these simply; the Microsoft Internet Transfer Control. It is basically a wrapper around the WinInet Windows API, making it nice and easy to use. To add this to your project, start a new standard .exe project in VB6. Go to the Project menu, then Components and pick the "Microsoft Internet Transfer Control". Click OK, and a component will be added to the components toolbox ready for use. Drop one of these onto your form.

(...)

In Visual Basic, you can do it as below this. Imagine Inet1 is the  name of the Internet Transfer Control you added above, and your email  address, password and application ID are stored in the string variables  "myEmail", "myPassword" and "mySource". Note the use of the  form-urlencoded Content-Type header.  
Code: [Select]
   strURL = "https://www.google.com/accounts/ClientLogin"
   strFormData  = "Email=" & myEmail & "&Passwd=" & myPassword &  "&source=" & mySource & "&service=cl"
   strHeaders = "Content-Type:application/x-www-form-urlencoded"
   Inet1.Execute strURL, "POST", strFormData, strHeaders
Your application needs to wait for Google to respond. Do this by  looking out for the icResponseCompleted event in the Inet1_StateChanged  event handler. When it does, store Google's reply in a variable, e.g.:

Code: [Select]
vtData = Inet1.GetChunk(1024, icString)
   </p>  <div class="codeblock"><code>      Do While LenB(vtData) > 0
           outputString = outputString + vtData      
            ' Get next chunk.
            vtData = Inet1.GetChunk(1024, icString)
         Loop      
      Response = outputString</code></div>


I suppose you could use this code (replacing the strFormData with "data=some-encoded-form-of-dbo" and the strURL to an appropriate address) to send a file to the server. On the server side, in PHP, $_POST['data'] would contain the encoded data.

Offline Numsgil

  • Administrator
  • Bot God
  • *****
  • Posts: 7742
    • View Profile
P2P Internet Mode
« Reply #13 on: June 01, 2008, 04:15:23 PM »
Hmm...  so bear with me if this is a stupid question.  I know some data can be embedded in the url (such as the forum url: index.php?showtopic=2808).  And other data can be sent behind the scenes, not in the URL.  For instance, when I write a really long, almost novel-like, post, all that text seems to get embedded in the stream somehow.  I think the two different types are called POST and GET, IIRC.  So... on the VB (or C#) side of things I can convert a file I want to upload in to a binary data stream.  Let's say, for the sake of argument, that the data stream is about 3MB large.  How would the web server running the PHP script expect to get that data?  What would the PHP script look like that took that data and saved it to a file somewhere?

Offline goffrie

  • Bot Builder
  • **
  • Posts: 65
    • View Profile
P2P Internet Mode
« Reply #14 on: June 01, 2008, 06:04:57 PM »
Quote from: Numsgil
Hmm...  so bear with me if this is a stupid question.  I know some data can be embedded in the url (such as the forum url: index.php?showtopic=2808).  And other data can be sent behind the scenes, not in the URL.  For instance, when I write a really long, almost novel-like, post, all that text seems to get embedded in the stream somehow.  I think the two different types are called POST and GET, IIRC.  So... on the VB (or C#) side of things I can convert a file I want to upload in to a binary data stream.  Let's say, for the sake of argument, that the data stream is about 3MB large.  How would the web server running the PHP script expect to get that data?  What would the PHP script look like that took that data and saved it to a file somewhere?

Data embedded in the URL is GET data, and the data embedded "behind the scenes" as you put it is POST data.
Your 3MB data stream would be sent via POST "behind the scenes", and then the PHP script would receive it like this:
Code: [Select]
   <?php
   if (!isset($_POST['data'])) die("No data sent via POST!");
   $data = trim($_POST['data']); // now $data contains the encoded data - "trim" removes any excess spaces, newlines etc. around it
   $data = base64_decode($data); // assuming base64 encoding, $data now has the binary data. PHP is binary-safe in general but some functions aren't
   $file = @fopen('files/'.time().'.dbo', 'w'); // now open files/(current UNIX timestamp) for writing ('w'). time() returns an integer but it is implicitly converted to a string. the @ suppresses errors
   if (!$file) die("Error opening file ".time().".dbo for writing!"); // error checking
   fwrite($file,$data) or die("Error writing file!"); // write the binary data to the file. the 'or die("")' syntax is quite common for easy error checking, I could have used it for $file but meh
   fclose($file); // close the file. not mandatory
   ?>
This PHP script would take base64 encoded data from the POST stream as the "data" variable (like "data=encodeddata"), decode it, and write it to "files/(time).dbo". Of course, this script would have problems with multiple files being uploaded in the same second - a different file scheme would fix that.
A shortened version (it generates a few warnings/notices on error):
Code: [Select]
<?php
 fwrite(fopen('files/'.time().'.dbo','w'),base64_decode(trim($_POST['data']))) or die("Failed!");
 ?>