No description
add_url.py | ||
client.py | ||
README | ||
server.py |
distributed scraper consists of 3 parts server.py, client.py, add_url.py run server, pass path to database as first arguemnt: python server.py urls.sqlite run client, you have to pass server url, client name and job type: python server.py http://127.0.01:2623 client01 maps.google.com now you can add urls by running python add_url.py http://127.0.01:2623 maps.google.com "http://mt1.google.com/vt/lyrs=h@218000000&hl=en&src=app&x=3&y=6&z=4&s=Galileo" REQUIREMENTS: you need: apt-get install python-ox, python-requests and python-twisted or: sudo easy_install ox requests twisted