Go to file
j 79b265dd61 mini distributed scraper 2013-06-05 12:26:29 +02:00
README mini distributed scraper 2013-06-05 12:26:29 +02:00
add_url.py mini distributed scraper 2013-06-05 12:26:29 +02:00
client.py mini distributed scraper 2013-06-05 12:26:29 +02:00
server.py mini distributed scraper 2013-06-05 12:26:29 +02:00

README

distributed scraper consists of 3 parts
server.py, client.py, add_url.py

run server, pass path to database as first arguemnt:

    python server.py urls.sqlite

run client, you have to pass server url, client name and job type:

    python server.py http://127.0.01:2623 client01 maps.google.com

now you can add urls by running 
    python add_url.py http://127.0.01:2623 maps.google.com "http://mt1.google.com/vt/lyrs=h@218000000&hl=en&src=app&x=3&y=6&z=4&s=Galileo"

REQUIREMENTS:
    you need:
        apt-get install python-ox, python-requests and python-twisted
    or:
        sudo easy_install ox requests twisted