Author Topic: P2P Internet Mode  (Read 18403 times)

Offline EricL

  • Administrator
  • Bot God
  • *****
  • Posts: 2266
    • View Profile
P2P Internet Mode
« Reply #45 on: September 30, 2008, 03:02:08 PM »
Quote from: Numsgil
I'll believe it when I see it


My feelings exactly regarding DB3...  


It compiles.  I had to stub out a lot of graphics stuff, which is where most of the differences are.  Now that it's raining in Seattle again, I'll probably get back to it in a week or two...
« Last Edit: September 30, 2008, 03:03:07 PM by EricL »
Many beers....

Offline Cyberduke

  • Bot Builder
  • **
  • Posts: 88
    • View Profile
P2P Internet Mode
« Reply #46 on: September 30, 2008, 03:08:00 PM »
Quote from: EricL
Push is the wrong word.  I want no code on clients listening on open ports, otherwise you have NAT, firewall, proxy and security issues.  The client should poll, which achieves the same thing, just with a little latency.  I assume this is what you meant.
I was making the assumption that the clients would retain their connection to the server (which they instigated) and so by ‘push’ was simply referring to the server telling the client there was a new bot rather than the client polling the server to ask.

Quote from: EricL
On the subject of connected sim topology and the server making routing decisions, I'm not sure I agree.  We could do connection topologies today without changing the plumbing, but I would want to explore why we should.  Random mixing has certain advantages.
Well the problem I have had with a random mix is that 1 or 2 bots can very quickly sweep though the whole sim, being introduced very early on to each and every instance of the sim, rather than leaving isolated pockets of species to evolve and forcing the aggressive bots to actually expand and conquer rather than helpfully be seeded everywhere.

Quote from: EricL
You are assuming the thing Nums is calling DB3 is a project that will someday supplant the current code base  That is a big assumption, one that is not shared by everyone.  It is much easier (and more likely to happen) that I simply port the current code to VB2008 to get threads and move the code base forward from there.  I have a version limping in VB2008 as we speak.
Ah now that’s interesting you say that, Yes I made the assumption (again) that since the wiki points DB3 in the direction of Nums C# page and you said you where not going to be continuing the project in the long term that Nums C# version was going to eventually become the next generation of Darwinbots
« Last Edit: September 30, 2008, 03:17:23 PM by Cyberduke »

Offline EricL

  • Administrator
  • Bot God
  • *****
  • Posts: 2266
    • View Profile
P2P Internet Mode
« Reply #47 on: September 30, 2008, 03:24:51 PM »
Quote from: Cyberduke
Well the problem I have had with a random mix is that 1 or 2 bots can very quickly sweep though the whole sim, being introduced very early on to each and every instance of the sim, rather than leaving isolated pockets of specials to evolve and forcing the aggressive bots to actually expand and conquer rather than helpfully be seeded everywhere.
You must run relatively small sims.  I run really large ones with shapes and physics such that it takes many thousands of cycles generally for a bot to make it from one side to the other, making colonization of immature sims be new-comers harder .  Additionally, unlike Terrarium, IM participants whose goal is to achieve the highest population for their bots can choose to grow their sim to maturity prior to connecting to IM, thus making it more difficult for any potential colonizer to take over the sim since it is already fully populated.

The fact that teleported-in bots are placed randomly instead of where the teleporter happens to be is deliberate on my part so as to make swarming of newcomers more difficult.    That said, I would be happy to discuss this area further on a different thread.   I'm not necessary opposed to adding sim topology to IM - but there may be better ways to achieve the things you want.

Quote from: Cyberduke
Ah now that’s interesting you say that, Yes I made the assumption (again) that since the wiki points DB3 in the direction of Nums C# page and you said you where not going to be continuing the project in the long term that Nums C# version was going to eventually become the next generation of Darwinbots
I'm not planning on going anywhere but neither can I spend large numbers of hours on DB.  I would welcome help on the current code base or related projects (like a server) but failing that, the community has to settle for me and my limited time commitment.
Many beers....

Offline Peter

  • Bot God
  • *****
  • Posts: 1177
    • View Profile
P2P Internet Mode
« Reply #48 on: September 30, 2008, 03:58:12 PM »
Quote from: EricL
Quote from: Numsgil
I'll believe it when I see it


My feelings exactly regarding DB3...  
Imagine what the result would be if both of you where going on the same project.

Quote
It compiles.  I had to stub out a lot of graphics stuff, which is where most of the differences are.  Now that it's raining in Seattle again, I'll probably get back to it in a week or two...
I thought compiling to VB2008 had a low priority. Doesn't matter anyway, multiple treads, yeah.
Oh my god, who the hell cares.

Offline Cyberduke

  • Bot Builder
  • **
  • Posts: 88
    • View Profile
P2P Internet Mode
« Reply #49 on: September 30, 2008, 04:22:36 PM »
Quote from: EricL
You must run relatively small sims.  I run really large ones with shapes and physics such that it takes many thousands of cycles generally for a bot to make it from one side to the other, making colonization of immature sims be new-comers harder .  Additionally, unlike Terrarium, IM participants whose goal is to achieve the highest population for their bots can choose to grow their sim to maturity prior to connecting to IM, thus making it more difficult for any potential colonizer to take over the sim since it is already fully populated.

The fact that teleported-in bots are placed randomly instead of where the teleporter happens to be is deliberate on my part so as to make swarming of newcomers more difficult.    That said, I would be happy to discuss this area further on a different thread.   I'm not necessary opposed to adding sim topology to IM - but there may be better ways to achieve the things you want.
My original thinking was really coming from the direction of combining multiple sim instances to achieve a distributed computing effect, in order to provide a larger more diverse environment for evolution to act upon. I can quite see that if you are not running mutations and/or are mainly pitting authored bot vs authored bot then random point to point transfers may even be desirable.


Quote from: EricL
I'm not planning on going anywhere but neither can I spend large numbers of hours on DB.  I would welcome help on the current code base or related projects (like a server) but failing that, the community has to settle for me and my limited time commitment.
I have no objection to helping on either project,  it’s the basic idea that interests me the most; I am a C# programmer by trade though so its Nums C# re-write that caught my attention.
« Last Edit: September 30, 2008, 04:52:13 PM by Cyberduke »

Offline Numsgil

  • Administrator
  • Bot God
  • *****
  • Posts: 7742
    • View Profile
P2P Internet Mode
« Reply #50 on: September 30, 2008, 04:40:40 PM »
Quote from: Cyberduke
Quote from: EricL
You are assuming the thing Nums is calling DB3 is a project that will someday supplant the current code base  That is a big assumption, one that is not shared by everyone.  It is much easier (and more likely to happen) that I simply port the current code to VB2008 to get threads and move the code base forward from there.  I have a version limping in VB2008 as we speak.
Ah now that’s interesting you say that, Yes I made the assumption (again) that since the wiki points DB3 in the direction of Nums C# page and you said you where not going to be continuing the project in the long term that Nums C# version was going to eventually become the next generation of Darwinbots

Just to be clear, I don't think most users care what the program is coded in one way or another.  Eric has a great vested interest in the current code base, so the "That is a big assumption, one that is not shared by everyone" is probably code for "That is a big assumption, one that is not shared by me".  From my experience going from 2.37 to early 2.4, stability is more important to end users than features, so if the C# version is always buggier than the current 2.4X code, it'll never catch on.  If it's more stable, it'll catch on.  Part of the problem with the current version is that if something breaks it's not always immediately apparent, since most of the time people aren't using the program so much as writing bots to use the program.  So bugs can sit there for a loooong time.  Which gives the impression to users that the program is buggy, and is an impression that I think isn't too far off the mark.  So I think in this regard there's a lot of room for improvement and thus a lot of leverage for a hypothetical new C# version to convert current users.

With extensive unit testing and a highly modular design I'm hoping bugs will be fewer and further between, and less likely to crop up again in the future (a big problem with the current code base is that it's so brittle that the same bugs seem to get fixed again and again and again).  But I've been working on it a long time and progress is slow since I keep backtracking when something doesn't actually work (lots of work on physics that had to be abandoned, for instance), so it's entirely possible for DB3 just to be vaporware and never realized.  It's also possible it'll get to a somewhat finished state and it'll be too buggy to compete with the current, semi-stable version and all that work will be for naught (though I don't think this particular scenario is likely, again because of unit testing and proper design).

Another scenario is that, assuming Eric can successfully migrate the current code to VB 2005/8, and thus allow .NET interop, the two projects might blend, with newer parts slowly supplanting older parts (or not, depending on their individual merits, of course).  But I'm not holding my breath

Really, it's sort of appropriate, since those are the options for new species to supplant older species.  Either one dominates the other or they interbreed (or they form separate niches.  I guess that's possible, too).  Natural selection will eventually choose a program as the winner in whatever happens, as is entirely appropriate, I think.

So just to reiterate, there are really two versions you could work on.  The current version is in VB, and is used by lots of people.  You can do things and have feedback from end users in very short order, which is a neat experience.  A future speculative version is in C#, and while it might never actually see the light of day, it's far easier to work in and with (unit testing again, plus the C# is easier to understand than VB if you come from a C++ background), and you don't have to worry about backwards compatibility.  Any contributor is free to chose either (or both I suppose) to work on.  All so far, other than Eric, have wanted to work on the C# version.  Admittedly most burn out after a few days or maybe a week or two, with little to show for it, but there you are.  I have no ego to be bruised in this regard, you're free to chose as you see fit

Offline Cyberduke

  • Bot Builder
  • **
  • Posts: 88
    • View Profile
P2P Internet Mode
« Reply #51 on: September 30, 2008, 05:24:09 PM »
Well yes ok I realise my wording of “going to eventually become the next generation” was a little strong, it was meant in the context of “has the intention of becoming...” due to the perceived wind-down of the current 2.4X code base (formed from comments on the wiki and forums).  
I am familiar with concept of vapourware having worked on a few Half-life mods in the past.

I didn’t appreciate however that both code bases had the intention of continued active development and that “DB3” was actually a branch. Not that it makes a huge difference I suppose, nothing wrong with a bit of healthy competition, but with only two main developers branching off does seem a bit of a waste.

I would be unsure of my competence editing/writing VB6 code and to be honest not that willing to learn, so I don’t mind helping with any .Net work though personally having no current investment in either project would be more tempted by the C# clean re-write, if there is support for one.
« Last Edit: September 30, 2008, 05:26:28 PM by Cyberduke »

Offline Numsgil

  • Administrator
  • Bot God
  • *****
  • Posts: 7742
    • View Profile
P2P Internet Mode
« Reply #52 on: September 30, 2008, 07:07:39 PM »
There's a bit of a history here, actually, that will at the very least provide some backstory.

Originally I was the only active developer (I took over from PY.  There was very little overlap between the two of us working on the program at the same time).  After a handful of versions, I spent the summer after my sophomore year at University to implement some brand new things.  New physics based on springs that was a vast improvement over the old system, which involved lots of bail wire and duct tape.  (This is largely the physics the current version uses, I believe.  Eric has done some tuning and such, but the core is probably pretty much what I built that summer).  I also revamped the DNA to be less structured.  It used to be that genes were rigidly defined as a cond-start-stop triplet.  I spent some effort to allow for more complex or even nonsensical DNA.  There were lots of other changes, too, but those were the big two.  This new version was packaged as 2.4

2.4 had some problems.  Lots of problems, and lots of bugs.  It was really almost entirely unusable.  2.4 was really a failure on my part, having underestimated the amount of time it takes to go from a "feature complete" milestone to something that's stable and ready for end users (and lacking a lot of experience coding for an audience at all).  School had started back up and I was extremely busy and frustrated with the VB code.  Not just fixing bugs that I really should have caught in the first place, really the whole way the language worked.  It made trying to implement various "advanced" features exceptionally hard (there was an abortive attempt at a bucket system to sort bots by buckets (eg: sectors) for faster collision and vision code that was made much harder by the limitations of the language and my ability to adapt to them).

So I stopped working on the VB version, and most people continued to use the older 2.37 version.  Instead, I started porting sections of the code to C++.  This wasn't a rewrite, it was very close to a straight port.  I fixed a few bugs in the process as I worked through the code, but the C++ version would have looked very familiar to someone working on the VB source.  You can probably find the C++ version in some repository somewhere linked on the wiki.  It got pretty far, but I ran in to a brick wall with the physics.  I wasn't using any external libraries or anything, it was a rigid body simulator (rigid bodies being limited to circles exclusively) I'd implemented entirely on my own.  What I wanted to do with multibots just didn't work with the new version (or old version for that matter).  Plus there were some threading issues.  Also, the code, being almost a direct port of the VB code, was still a pain to work with.  Something like the robot class was thousands of lines of convoluted code, with lots of interdependencies between disparate classes (the shots class was a friend to the robot class, and vice versa).  So I gave up on the C++ version and spent some time working through how to actually achieve the sort of physics I wanted.  This was the end of my junior year at University.

Sometime during this process, Eric joined the community and decided he wanted to try and fix issues with the 2.4 source, since it did have some nice features even if it was a bit buggy.  After a few months of effort he was able to get even the hardliners to move from the 2.37 version to the new 2.4X version.  And he's been supporting that version ever since(roughly two years), adding features and fixing bugs as people suggest them/find them.

I, in the mean time, decided to start over with a complete rewrite from scratch.  Because of its similarity (in a good way) with VB, I chose to try again in C#.  To get the ball rolling I wrote about 75% of a DNA module in C# (actually wrote it twice.  The first time to get a taste of the language).  This was right when I first learned about unit testing (maybe a year and a half ago), so the vast majority of the DNA code is unit tested and, I think, pretty stable and bug free.

Then I spent about 8 months working on an entirely different project (A 3D version of something like SimEarth).  Obviously not a lot happened with Darwinbots during that 8 months.  Eventually Darwinbots coding started to interest me and I put my other project aside.  After some abortive attempts at using existing libraries (most didn't last more than a few weeks) I thought I'd try my hand at building my own physics engine again.  Unfortunately, most of what I wanted to do had never been attempted before, so there was a lot of original research on my part.  That started about a year ago maybe and continued until maybe 3 months ago.  During the 4th of July weekend I scrapped the physics work I was doing and concentrated on getting re-acquainted with my code, reorganizing the file structure and fixing various annoyances with Visual Studio project files and temp directories and such.  Lots of meta programming kind of stuff.  I got my present job as a junior programmer for a video game company last March.  When you program 50-60 hours a week for a job it's hard sometimes to drum up the mental effort to work on part time coding projects, so progress has been... glaciatic.  I've put maybe 100 hours of work in during the last 3 months (though I finished a major milestone last week, so I can relax and go back to 40-50 hour weeks for a while.  Plus I have a 3 day weekend coming up which I'll probably spend, at least in part, working on Darwinbots).

During the time I was either not working on Darwinbots or working on physics, Eric's been supporting the 2.4 code.  He takes a few months off during the summer, but otherwise puts a lot of time and effort in to supporting the current version for users.  He tends to work slow and steady, where I've always been of the burn-out super nova explosion of coding school, where I spend about two weeks doing a massive sprint of thousands of lines of code then do nothing for several months   So Eric's understandably a little skeptical of any claims I make.

Heh, that's a little long winded, but that's how things ended up where they are.

Offline EricL

  • Administrator
  • Bot God
  • *****
  • Posts: 2266
    • View Profile
P2P Internet Mode
« Reply #53 on: September 30, 2008, 07:49:46 PM »
A very fair summary.

Understand that my skepticism regarding DB3 is not a statement about Nums coding skills or tenacity or anything like that.  It's just that I've spent 25 years in the software industry and in that time I've formed some opinions about how to move code bases forward.  Total re-writes never work.  It  sounds great to start clean and do things right from scratch "this time around" but the details and difficulty are always underestimated.  We tend to underestimate how much is really invested in the older code base, all the little things w.r.t. usability or features or hard-won lessons about the way things should work that are embedded in the code and present barriers to people moving.  It's not the rocket science core stuff that kills projects.  It's the million little simple things that add up to a usable program.  This is why is was so difficult for people to move to 2.4 even long after the basics were in place.   Sure, you can get there if you throw enough hours at it for long enough (I did) but in my experience, that is always the most expensive route to take.  

Understand that I've probably doubled the number of lines of code since I took over.   There are a lot of details, a lot of features, a lot of nuances.  Even were DB3 working today, it would take a long time to get to critical mass such that people could move.  I prefer the incremental approach.  One rule I learned the hard way and have followed in the large software projects I've managed is that you never, ever change more than 30% of the code from major release to major release.  You can do re-writes, but you do it of individual modules.  You move forward, incrementally, maintaining stability while you do.  Not doing so is the fastest path to vaporware.

So, I'm skeptical not only of "DB3" but of any project that does not start from the current code base and move forward incrementally.
Many beers....

Offline EricL

  • Administrator
  • Bot God
  • *****
  • Posts: 2266
    • View Profile
P2P Internet Mode
« Reply #54 on: September 30, 2008, 08:04:32 PM »
Quote from: Cyberduke
My original thinking was really coming from the direction of combining multiple sim instances to achieve a distributed computing effect, in order to provide a larger more diverse environment for evolution to act upon.
This is exactly why (well, one reason at least) I implemented private teleporters.  Unlike the IM teleporter, inbound private teleporters do not randomly distribute incoming bots across the field.  Bots stay where the teleporter is.  It's a doorway for incoming bots just like outbound teleporters are an exit.   They can be made stationary, perhaps surrounded by shapes, to create different fiefdoms or whatever other effect you want or they can be made to drift, ignoring or respecting shapes as you like.  They can filter bots to restrict or allow veggies or corpses and such.  You can have multiple pairs connecting multiple sims in whatever topology you want.   And, it is on my list to unify shapes and teleporters to provide greater flexibility on size including making entire field edges themselves act as teleporters.  So you see, we think very much alike here...
Many beers....

Offline Cyberduke

  • Bot Builder
  • **
  • Posts: 88
    • View Profile
P2P Internet Mode
« Reply #55 on: October 01, 2008, 05:07:27 AM »
Quote from: EricL
It's just that I've spent 25 years in the software industry and in that time I've formed some opinions about how to move code bases forward. Total re-writes never work. It sounds great to start clean and do things right from scratch "this time around" but the details and difficulty are always underestimated. We tend to underestimate how much is really invested in the older code base, all the little things w.r.t. usability or features or hard-won lessons about the way things should work that are embedded in the code and present barriers to people moving. It's not the rocket science core stuff that kills projects. It's the million little simple things that add up to a usable program. This is why is was so difficult for people to move to 2.4 even long after the basics were in place. Sure, you can get there if you throw enough hours at it for long enough (I did) but in my experience, that is always the most expensive route to take.

Understand that I've probably doubled the number of lines of code since I took over. There are a lot of details, a lot of features, a lot of nuances. Even were DB3 working today, it would take a long time to get to critical mass such that people could move. I prefer the incremental approach. One rule I learned the hard way and have followed in the large software projects I've managed is that you never, ever change more than 30% of the code from major release to major release. You can do re-writes, but you do it of individual modules. You move forward, incrementally, maintaining stability while you do. Not doing so is the fastest path to vaporware.
Sounds like some very good advice there, *puts fingers in ears* 'laa la laaa la'    
You’re absolutely right; I know you are, its just from my point of view there would also be the overhead of learning a new...old... different language  and trying to get to grips with a large complex existing code base written in multiple styles by multiple developers over a long period of time.
On the other hand, a new start has an extra fun-factor about it and would give me the ability to understand it from the start as it was being laid down and for me to incorporate some of my own ideas into it, even if it does come with an increased risk of wasted effort.

Besides at work recently I am getting used to re-writing stuff, it's a constant stream of “here is our old legacy program, the vender went bust about 8 years ago and so there is no source available, Can you build us a new version that’s 10x better inside 6 months please”

Anyway...
It’s just a shame there would really be a duplication of effort reaching for a very similar goal which doesn’t quite sit right.

And for the record I probably sit closer to Nums school of coding, though maybe not quite so extreme, having had to earn a living as a programmer for 6 years has tempered that tendency somewhat.


Edit: Also even If the 2.4x code base wasn’t going to stay in active development I understand that it would take a lot of time before 3 could begin to compete for any percentage of the user base since the first number of releases would really by necessity have a greatly reduced feature set, and would then take a while to stabilize and for new features to be built on top of that.

2nd Edit: that and 3 looks like it just about decimates any chance of running existing DNA code on it.
« Last Edit: October 01, 2008, 12:27:50 PM by Cyberduke »

Offline Cyberduke

  • Bot Builder
  • **
  • Posts: 88
    • View Profile
P2P Internet Mode
« Reply #56 on: October 01, 2008, 06:36:35 AM »
Oh and Numsgil, that was a very long and informative post, which was greatly appreciated and congrats' on 6,000 posts!
« Last Edit: October 01, 2008, 01:49:31 PM by Cyberduke »

Offline Numsgil

  • Administrator
  • Bot God
  • *****
  • Posts: 7742
    • View Profile
P2P Internet Mode
« Reply #57 on: October 01, 2008, 01:32:44 PM »
Woo 6K!

The current DNA (codenamed Sunweaver) for v3 isn't compatible with present bots, but being modular by nature it's possible a future module could be written to allow for current bots to run.  It just isn't my priority at the moment.