0x2620/distributed_scraper
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity

mini distributed scraper

Browse source
This commit is contained in:
j 2013-06-05 12:26:29 +02:00
commit 79b265dd61
4 changed files with 310 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

19
README Normal file
View file

@ -0,0 +1,19 @@
distributed scraper consists of 3 parts
server.py, client.py, add_url.py
run server, pass path to database as first arguemnt:
python server.py urls.sqlite
run client, you have to pass server url, client name and job type:
python server.py http://127.0.01:2623 client01 maps.google.com
now you can add urls by running
python add_url.py http://127.0.01:2623 maps.google.com "http://mt1.google.com/vt/lyrs=h@218000000&hl=en&src=app&x=3&y=6&z=4&s=Galileo"
REQUIREMENTS:
you need:
apt-get install python-ox, python-requests and python-twisted
or:
sudo easy_install ox requests twisted