Darwinbots Forum

Code center => Darwinbots Program Source Code => Topic started by: Botsareus on April 19, 2017, 11:32:19 PM

Title: Use http get and set requests instead of ftp
Post by: Botsareus on April 19, 2017, 11:32:19 PM
Hey Numsgil I was thinking (well actually I got the idea from a guy I hang out with at school)

Can we replace the ftp with simple http get and set requests? That way we can really streamline everything and make it scale.

Problem is, do not how a clue how to write the server side portion of it. I may be able to hack the client side into vb6 though.

Also, wasn't this exactly what Shasta was working on?
Title: Re: Use http get and set requests instead of ftp
Post by: Numsgil on April 22, 2017, 06:06:10 AM
Yes, I think it's a reasonable approach and yes, it was one of the plans for a while.  It needs server side support, either PHP or python or whatever, and I'm not really a web programmer so I didn't know how to do that.
 There's also security concerns and I'm not enough of a web programmer or linux guy to know what is or isn't a good idea here.  I'd want to jail the custom server side code as much as possible so it can't break things and can't be used as an entry point to attack the server.
Title: Re: Use http get and set requests instead of ftp
Post by: spike43884 on April 22, 2017, 09:12:55 AM
I suspect PHP.
Though, I'm just too out of touch with web programming and haven't ever properly done PHP. You'd have to check really.

If we get a proper web programmer down here, it'd be nice to have an internet page where we can see what's happening on an internet sim.
Title: Re: Use http get and set requests instead of ftp
Post by: theblaze on July 18, 2017, 03:57:10 PM
Not a "proper" web programmer, so I'd be able to get it all put together securely etc (I'm more on the admin / security kinda stuff), but it won't have a pretty web page. Someone else can write the css if they'd like. Also, I'd suggest that the site should be moved to https using lets encrypt, really simple to set it up
Title: Re: Use http get and set requests instead of ftp
Post by: Numsgil on July 19, 2017, 04:02:00 AM
What do you need me to do?
Title: Re: Use http get and set requests instead of ftp
Post by: theblaze on July 19, 2017, 10:56:47 AM
RE https or the internet mode stuff?
Title: Re: Use http get and set requests instead of ftp
Post by: Numsgil on July 19, 2017, 07:16:53 PM
Yeah, sounded like you were maybe volunteering some expertise in the area?
Title: Re: Use http get and set requests instead of ftp
Post by: theblaze on July 20, 2017, 12:44:30 AM
Yeah, I'm happy to help out.
Title: Re: Use http get and set requests instead of ftp
Post by: theblaze on July 20, 2017, 04:41:43 AM
For the internet mode stuff I'll get the server side stuff running, then it's just adding code to Darwinbots. For the https it's pretty simple to set it up, but if you give me access to the server hosting it I can do it for you. Have multiple sites using lets encrypt, as well as all my own sites ( https://crt.sh/?q=%25theblazehen.com ), and it works really well. Would also be neat to have a matrix.org / riot.im server if you'd like me to set one up
Title: Re: Use http get and set requests instead of ftp
Post by: theblaze on July 20, 2017, 04:55:00 PM
For the internet mode, will the DB client ask for if there's a bot available to get regularly and always take it if there is one, or is the server side more intelligent and selectively chooses DB clients? I've got measures in place to prevent abuse (Draining all bots / pushing out a lot of your own bots),  but I'd need to know if I need to add more intelligence for the server

Eg, if there isn't any intelligence in the client then it may be worth, on receiving an uploaded bot assign it to another connected IP, rather than sending it back to the first client that issues a GET. I feel I may be over complicating things, and the simple method should work well enough:w
Title: Re: Use http get and set requests instead of ftp
Post by: Numsgil on July 22, 2017, 02:17:08 PM
For the https it's pretty simple to set it up, but if you give me access to the server hosting it I can do it for you.

Would you mind just walking me through setting it up?  I'd like to have at least some idea of what's changed so when it breaks I vaguely know what to poke :)  The server is just a linux box.

Quote
Would also be neat to have a matrix.org / riot.im server if you'd like me to set one up.

Can you give me the elevator pitch for how you think it'd work?  Is it just a matter of hooking up the Darwinbots client/desktop app, and then matrix would be the server?  Do we have/get to run matrix on our own server?

Quote
For the internet mode, will the DB client ask for if there's a bot available to get regularly and always take it if there is one, or is the server side more intelligent and selectively chooses DB clients?

At the moment/historically it's first-come-first-serve, and there's nothing to prevent either duplication of bots when two people download a file at once or getting your own bot back.  But we can certainly do something smarter, especially if it's only a little bit of code to enable.  But my first priority would just be getting something set up.

Quote
I've got measures in place to prevent abuse (Draining all bots / pushing out a lot of your own bots)

Can you issue login credentials to people?  That would prevent at least some sorts of abuses I'm concerned with.  If we could tie it in the forum authentication that would be even better, but that might be tricky.
Title: Re: Use http get and set requests instead of ftp
Post by: theblaze on July 22, 2017, 02:36:27 PM
Quote
Would you mind just walking me through setting it up?  I'd like to have at least some idea of what's changed so when it breaks I vaguely know what to poke :)  The server is just a linux box.

https://certbot.eff.org/ has a good quick start guide, otherwise we can chat on IRC or something if you have issues.

RE matrix, think of it as the IRC that we used to run just that it has scrollback / good web access etc. Was a bit off topic though.

Quote
At the moment/historically it's first-come-first-serve, and there's nothing to prevent either duplication of bots when two people download a file at once or getting your own bot back.  But we can certainly do something smarter, especially if it's only a little bit of code to enable.  But my first priority would just be getting something set up.
So what I've been thinking is to have a URL that you can use for different environments, eg im.darwinbots.com/zerobots or a private im.darwinbots.com/myownpersonalsim732187

Quote
Can you issue login credentials to people?  That would prevent at least some sorts of abuses I'm concerned with.  If we could tie it in the forum authentication that would be even better, but that might be tricky.
Should be possible, will be a bit tricky to do it with the forum details though
Title: Re: Use http get and set requests instead of ftp
Post by: theblaze on July 23, 2017, 05:18:56 AM
Quote
two people download a file at once or getting your own bot back.

So I'm thinking for each world (better word than environment in previous reply), each client needs to stay within +10 and -10 bots (client identified by IP, not that hard to beat the restriction, but if someone wants to they'll find a way) to prevent taking all / flooding the IM with their own bots, then we keep a pool of say 100 bots on the server. If a client requests to GET a bot and they're within their limit and we have 100+ bots on the server then we choose a random bot, send it to them, and delete it from the server. Won't guarantee that you won't get the same bot right back, but you'll have a good chance it isn't the same you sent out.
Title: Re: Use http get and set requests instead of ftp
Post by: theblaze on July 23, 2017, 09:24:11 AM
Woot, got it running. Will make it publicly available in 2-3 hours
Title: Re: Use http get and set requests instead of ftp
Post by: Numsgil on July 23, 2017, 12:28:25 PM
I got the https certificates but certbot is having a hard time installing them in to apache.  We have a lot of virtual hosts that are Include'd from other files, and I think that's confusing it.  I'll need to manually add the certificates to all the virtual host files.  Sometime in the next few days hopefully.
Title: Re: Use http get and set requests instead of ftp
Post by: theblaze on July 23, 2017, 01:55:24 PM
Alright. Yeah, certbot is mainly for the simple configurations. I use caddyserver.com to automatically get certs and renew them
Title: Re: Use http get and set requests instead of ftp
Post by: theblaze on July 23, 2017, 02:44:00 PM
Alright, not the best solution, but this should be temporary. You're gonna need https://www.cygwin.com/ with python3 and python3 requests, then go to the /cygdrive/c/.../Darwinbots/IM/outgoing in cygwin terminal, and

Code: [Select]
while :; do for bot in $(ls *dbo | shuf | head -n1); do curl -T$bot https://dbim.theblazehen.com/anythinggoes && rm $bot; done; rm *stats; sleep 1; done
Then save the following as im_incoming.py
Code: [Select]
import requests
import time
import random
while True:
time.sleep(0.5)
try:
print("getting bot")
r = requests.get('https://dbim.theblazehen.com/anythinggoes')
botData = r.content
if r.ok:
with open(str(random.randrange(0,1000)) + '.dbo', 'wb') as f:
f.write(botData)
f.close()
except:
print("failed to get bot")

Then cd to /cygdrive/c/Users/.../IM/incoming and
Code: [Select]
python3 /cygdrive/c/path/to/im_incoming.py
You can replace the anything goes with whichever world you want to connect to, so you can easily create your own. Note that per IP you need to stay within a range of -10 to +10 bots uploaded, so you'd be up to 10 quite quickly, then you get 1 point closer to 0 each minute. There needs to be a pool of 200 bots in the world before you can retrieve any, so it requires 200-10 minutes before you can receive anything back from the world.

@Numsgil will you be able to support using HTTP GET and PUT in the darwinbots client directly, or will it still use an external program with an incoming and outgoing directory?
Title: Re: Use http get and set requests instead of ftp
Post by: theblaze on July 23, 2017, 02:53:57 PM
Code: [Select]
from pprint import pprint
import threading
import uuid
import os
from collections import defaultdict
import re
import random
import time

from flask import request
from flask import Flask
app = Flask(__name__, static_url_path='')

botPath = 'bots/'

ipBotCounts = defaultdict(int)


# We allow adjustments back to zero slowly per IP, but not fast enough to cause abuse
def background_decay_ipCount():
    global ipBotCounts
    while True:
        time.sleep(60)
        for ip in ipBotCounts:
            if ipBotCounts[ip] > 0:
                print("removing 1 from " + ip)
                ipBotCounts[ip] -= 1

            if ipBotCounts[ip] < 0:
                print("adding 1 to " + ip)
                ipBotCounts[ip] += 1

decay_ip_thread = threading.Thread(target=background_decay_ipCount)
decay_ip_thread.start()


@app.route('/<world>', methods=['GET', 'PUT', 'POST'])
def handleRequest(world):
    pprint(ipBotCounts)
    path = request.path.split('/')[1]
    remoteIp = request.headers.get('X-Forwarded-For', request.remote_addr)

    if not re.match(r"[a-zA-Z0-9_]", path): #Someone is doing something dodgy
        return '', 400

    if not os.path.isdir(botPath + path):
        os.mkdir(botPath + path)

    if request.method == 'PUT':
        if ipBotCounts[remoteIp] >= 10:
            # Sent too many bots
            return '', 429

        ipBotCounts[remoteIp] += 1
        with open(botPath + path + '/' + str(uuid.uuid4()),'wb') as f:
            f.write(request.data)
        return ''

    if request.method == 'GET':
        if not len(os.listdir(botPath + path)) > 200: #not enough bots to give one out
            return '', 404

        if ipBotCounts[remoteIp] <= -10:
            # Sent too little bots to us
            return '', 429

        ipBotCounts[remoteIp] -= 1

        fileToSend = random.choice(os.listdir(botPath + path))
        fileToSend = botPath + path + '/' + fileToSend
        filedata = open(fileToSend, 'rb').read(100000)
        os.remove(fileToSend)

        return filedata

app.run(host='0.0.0.0', port=80, threaded=True)

Current server side code
Title: Re: Use http get and set requests instead of ftp
Post by: Numsgil on July 30, 2017, 09:56:04 AM
@Numsgil will you be able to support using HTTP GET and PUT in the darwinbots client directly, or will it still use an external program with an incoming and outgoing directory?

Although Visual Basic 6 is supposed to support http libraries, in practice we've had better success with external programs that read and write the files, with the DB2 app just reading and writing files.

...

For the server code, can you explain which abuses you're trying to protect against?
Title: Re: Use http get and set requests instead of ftp
Post by: theblaze on July 30, 2017, 03:11:01 PM
Quote
For the server code, can you explain which abuses you're trying to protect against?

Someone flooding the sim with one of their bots, or grabbing all the bots available in the pool. It tries to keep a balance between bots in / out for each client