## 在linux下安装并运行scrapyd
```
pip isntall scrapyd
mkdir /etc/scrapyd
vi /etc/scrapyd/scrapyd.conf
```
```
[scrapyd]
eggs_dir = eggs
logs_dir = logs
items_dir =
jobs_to_keep = 5
dbs_dir = dbs
max_proc = 0
max_proc_per_cpu = 10
finished_to_keep = 100
poll_interval = 5.0
bind_address = 0.0.0.0
http_port = 6800
debug = off
runner = scrapyd.runner
application = scrapyd.app.application
launcher = scrapyd.launcher.Launcher
webroot = scrapyd.website.Root
[services]
schedule.json = scrapyd.webservice.Schedule
cancel.json = scrapyd.webservice.Cancel
addversion.json = scrapyd.webservice.AddVersion
listprojects.json = scrapyd.webservice.ListProjects
listversions.json = scrapyd.webservice.ListVersions
listspiders.json = scrapyd.webservice.ListSpiders
delproject.json = scrapyd.webservice.DeleteProject
delversion.json = scrapyd.webservice.DeleteVersion
listjobs.json = scrapyd.webservice.ListJobs
daemonstatus.json = scrapyd.webservice.DaemonStatus
```
>制作软连接: ln -s /usr/local/python3/bin/scrapyd /usr/bin/scrapyd
>创建文件后执行命令启动scrapyd: (scrapyd > /dev/null &) 当想要记录输出日志时: (scrapyd > /root/scrapyd.log &)
- 工欲善其事必先利其器
- 请求库
- 解析库
- 数据库
- 存储库
- Web库
- app爬取相关库
- 爬虫框架
- 部署相关库
- ipython
- 基础
- 数学函数
- 随机函数
- 三角函数
- 字符串内建函数
- 列表方法
- 字典内置方法
- 正则表达式
- os
- 字符串及数字的判断
- 常用魔术方法
- db
- mongodb
- mysql
- redis
- ORM
- ODM
- mongodb操作方法
- sqlite3
- access
- files
- Excel
- xml文件
- Python环境
- anaconda
- pip常用命令
- virtualenv
- pyenv
- cmder
- 远程开发
- Jupyter
- crawler
- appium环境搭建
- adb工具
- uiautomator
- 运行Appium+Python Clinet + 夜神模拟器
- DesiredCapabilities参数大全
- requests
- scrapy
- gerapy
- scrapyd
- 请求头fake_useragent库
- 数据传递过程
- 数据清洗及入库pipelines.py
- scrapy调用阿布云代理
- 图片下载
- PyQt5
- pyinstaller
- 攻防
- xss
- xss反射
- Chrome模拟微信浏览器
- flask
- 注册app
- 蓝图Blueprint
- 表单验证wtforms
- Flask-SQLAlchemy
- 数据处理
- json
- tornado
- settings
- 工具
- fiddler
- ab压力测试工具
- 高阶
- 队列
- 多线程
- 消息队列
- 定时任务框架APScheduler
- Django
- 路由分离
- 模型
- admin
- Android
- apk逆向工程