20 lines
623 B
Plaintext
20 lines
623 B
Plaintext
distributed scraper consists of 3 parts
|
|
server.py, client.py, add_url.py
|
|
|
|
run server, pass path to database as first arguemnt:
|
|
|
|
python server.py urls.sqlite
|
|
|
|
run client, you have to pass server url, client name and job type:
|
|
|
|
python server.py http://127.0.01:2623 client01 maps.google.com
|
|
|
|
now you can add urls by running
|
|
python add_url.py http://127.0.01:2623 maps.google.com "http://mt1.google.com/vt/lyrs=h@218000000&hl=en&src=app&x=3&y=6&z=4&s=Galileo"
|
|
|
|
REQUIREMENTS:
|
|
you need:
|
|
apt-get install python-ox, python-requests and python-twisted
|
|
or:
|
|
sudo easy_install ox requests twisted
|