本书介绍如何结合Python进行网络爬虫程序的开发，从Python语言的基本特性入手，详细介绍了Python网络爬虫开发的各个方面，涉及HTTP、HTML、JavaScript、正则表达式、自然语言处理、数据科学等不同领域的内容。全书共10章，包括Python基础知识、网站分析、网页解析、Python文件读写、Python与数据库、AJAX技术、模拟登录、文本与数据分析、网站测试、Scrapy爬虫框架、爬虫性能等多个主题。本书可作为高等职业院校计算机类专业的专业课教材，也可供计算机相关从业人员选用参考。

前言 在互联网大数据时代，海量数据爆炸式地出现在网络中，给人们的生活带来极大的便利。但同时，在海量的信息中，大多数信息是无效的垃圾信息。如何在海量的信息碎片中得到真正需要的信息，成为人们的迫切需求。 最简单的数据信息获取方式是人工操作浏览器搜索信息，但是单靠人工进行筛选不太现实，于是网络爬虫技术应运而生。通过该技术将相关的内容收集起来，再经过分析、筛选才能得到人们真正需要的信息。 网络爬虫（又被称为网页蜘蛛、网络机器人）是一种模拟浏览器发送网络请求、接收请求响应，按照一定的规则自动抓取互联网信息的程序。网络爬虫可以用来爬表格、爬图片、爬视频等，能通过浏览器访问的数据都可以通过网络爬虫获取。 本书以由浅入深、循序渐进的方式展开讲解，并通过经典的实例对Python 网络爬虫的功能进行详细介绍，具有极高的实用价值。通过本书的学习，读者可以掌握Python 网络爬虫的基本原理和应用方法。 一、本书特点 实例丰富 本书中的实例不管是数量还是种类，都非常丰富。本书结合大量的Python 网络爬虫实例，详细介绍 Python 网络爬虫的基本原理，让读者在学习实例的过程中潜移默化地掌握Python 网络爬虫的应用方法。 突出提升技能 本书从全面提升读者的Python 网络爬虫实际应用能力出发，通过深入剖析实例，使读者能够独立地完成各种Python 网络爬虫应用操作。 书中的大部分实例源自Python 网络爬虫项目案例，经过编者的精心提炼和改编，不仅能帮助读者学好知识点，而且能够提升读者的实际操作水平。 技能与思政教育紧密结合 本书在介绍Python 网络爬虫专业知识的同时，紧密结合思政教育主旋律，使读者在学好专业知识的同时，还能强化思政教育。 项目式教学，实操性强 本书的编者都是高校从事Python 网络爬虫教学与研究多年的一线教师，具有丰富的教学实践经验与教材编写经验，有一些编者出版过Python 相关书籍，这些图书经过市场的检验，很受读者欢迎。多年的教学工作使他们能够准确地把握学生的心理与实际需求。编者总结多年的开发经验及教学心得体会，力求在本书中全面、细致地展现Python 网络爬虫的基本原理和应用方法。 项目形式，实用性强 本书采用项目的形式组织内容，把Python 网络爬虫的理论知识分解并融入每个项目中，增强了本书的实用性。 二、本书的基本内容 本书共7 个项目，具体内容包括：Python 基础认知、网络爬虫基础认知、Urllib 请求模块库的应用、安装Urllib3 请求模块库并发送请求、Requests 请求模块库的应用、解析网页、Scrapy 网络爬虫框架等。 三、关于本书的服务 1．关于本书的技术问题或有关本书信息的发布读者若遇到有关本书的技术问题，可以将问题发到电子邮箱714491436@qq.com，我们将及时回复；也欢迎各位读者加入图书学习交流群（QQ：635068352）交流探讨。 2．配套资源 为满足教师的教学需求，本书配备了丰富的教学资源，包括电子课件、源文件等，读者可以登录华信教育资源网注册后免费下载相关资源。 本书由耿兴隆、胡钟月、周祥担任主编，曾俊、韦祥、闫金迎、周育辉、黄海华担任副主编。河北军创家园文化发展有限公司为本书的出版提供了必要的帮助，在此对他们的付出表示真诚的感谢。 由于编者水平有限，书中不妥之处在所难免，恳请广大读者批评指正。

目录 项目一 Python 基础认知 ····················································································.1 任务一 Python 概述 ·······································································································.1 一、Python 简介 ······································································································.1 二、安装Python ······································································································.2 三、安装PyCharm ···································································································.6 四、Python 语法规范 ·······························································································.11 任务二 Python 命令的组成 ·····························································································.13 一、基本符号 ·········································································································.14 二、常量与变量 ······································································································.16 三、数据类型 ·········································································································.19 四、功能符号 ·········································································································.24 任务三 程序结构 ·········································································································.26 一、表达式语句 ······································································································.26 二、顺序结构 ·········································································································.27 三、选择结构 ·········································································································.28 四、循环结构 ·········································································································.30 五、条件表达式 ······································································································.31 六、程序的流程控制 ································································································.32 项目实战 ·····················································································································.33 实战 输出百度网址 ································································································.33 项目二 网络爬虫基础认知 ················································································.35 任务一 网络爬虫概述 ···································································································.35 一、网络爬虫的基本原理 ··························································································.36 二、网络爬虫系统框架 ·····························································································.37 三、爬行策略 ·········································································································.37 四、网络爬虫的分类 ································································································.38 五、开源网络爬虫框架/项目 ······················································································.39 任务二 HTTP ·············································································································.41 一、HTTP 的工作原理 ·····························································································.41 二、Urllib 模块库 ···································································································.42 三、URL 定义 ·······································································································.43 四、URL 编码设置 ·································································································.47 任务三 网页请求过程 ···································································································.50 一、发送请求报文 ··································································································.51 二、返回响应 ········································································································.52 三、HTTP 消息 ······································································································.53 项目实战 ·····················································································································.54 实战一 搜索商品网址 ····························································································.54 实战二 搜索食品价格网址 ······················································································.56 项目三 Urllib 请求模块库的应用 ········································································.58 任务一 发送网页请求 ···································································································.58 一、基本HTTP 请求 ·······························································································.58 二、Request 网络请求 ·····························································································.66 三、设置请求头 ·····································································································.67 四、Handler 方法发送请求 ·······················································································.69 五、设置代理IP ····································································································.71 六、身份验证 ········································································································.73 任务二 网页下载 ·········································································································.77 一、网页结构 ········································································································.77 二、写入网页文件 ··································································································.77 三、网页文件下载 ··································································································.79 项目实战 ·····················································································································.82 实战一 下载Python 学习网址 ··················································································.82 实战二 下载公司网页HTML 文件 ············································································.85 项目四 安装Urllib3 请求模块库并发送请求 ··························································.87 任务一 安装Urllib3 请求模块库 ······················································································.87 一、安装Anaconda ·································································································.87 二、安装Urllib3 模块库 ···························································································.92 任务二 发送请求 ·········································································································.95 一、创建代理对象 ··································································································.96 二、请求方法 ········································································································.98 三、定义请求头 ·····································································································.99 四、设置代理IP ···································································································.101 五、自动重试 ·······································································································.102 六、重定向 ··········································································································.103 项目实战 ····················································································································.104 实战 发送请求访问淘宝 ························································································.104 项目五 Requests 请求模块库的应用 ·································································.106 任务一 网页请求 ·······································································································.106 一、标准的HTTP 请求 ···························································································.107 二、返回响应消息 ·································································································.109 三、JSON 格式数据 ·······························································································.114 任务二 发送请求方法 ·································································································.117 一、发送GET 请求方法 ·························································································.118 二、发送POST 请求方法 ························································································.120 三、其他请求方法 ·································································································.125 任务三 复杂网络请求 ·································································································.126 一、复杂请求头 ····································································································.126 二、上传文件 ·······································································································.129 三、Cookies 验证 ··································································································.131 四、会话保持 ·······································································································.131 任务四 异常处理 ·······································································································.133 一、try-except 语句 ································································································.133 二、Urllib 异常处理模块 ·························································································.134 三、Urllib3 异常处理模块 ·······················································································.135 四、request 异常处理模块 ·······················································································.135 项目实战 ···················································································································.138 实战 爬取豆瓣最受欢迎的影评网址 ·········································································.138 项目六 解析网页 ···························································································.141 任务一 正则表达式解析网页 ························································································.141 一、正则表达式模式 ······························································································.142 二、使用re 模块实现正则表达式 ··············································································.143 三、字符串查找 ····································································································.144 四、字符串替换 ····································································································.148 五、字符串分割 ····································································································.149 任务二 XPath 解析网页 ·······························································································.150 一、XPath 概述 ····································································································.150 二、XPath 网页解析 ······························································································.152 三、获取节点信息 ·································································································.154 四、节点关系 ·······································································································.160 五、查找节点信息 ·································································································.162 六、属性节点 ·······································································································.163 七、XPath 运算符 ·································································································.165 八、XML 节点轴 ··································································································.168 任务三 BeautifulSoup 解析网页 ······················································································.170 一、安装BeautifulSoup ···························································································.171 二、创建BeautifulSoup 对象 ····················································································.171 三、通过属性获取节点内容 ·····················································································.173 四、根据节点关系获取节点 ·····················································································.176 五、查找节点内容 ·································································································.178 六、通过CSS 选择器查找节点内容 ···········································································.182 项目实战 ····················································································································.183 实战一 获取查询网中河北省石家庄市的邮编区号 ·······················································.183 实战二 爬取销售热门图书名称 ···············································································.186 实战三 下载销售热门图书的图片 ············································································.188 项目七 Scrapy 网络爬虫框架 ···········································································.190 任务一 Scrapy 网络爬虫框架基础认知 ·············································································.190 一、Scrapy 网络爬虫框架基础 ··················································································.190 二、Scrapy 常用命令 ······························································································.192 三、创建Scrapy 项目 ·····························································································.193 任务二 使用模板创建Spider 文件 ··················································································.194 一、创建网络爬虫文件命令 ·····················································································.195 二、创建basic 模板文件 ·························································································.196 三、创建crawl 模板文件 ·························································································.197 四、创建csvfeed 模板文件 ······················································································.198 五、创建xmlfeed 模板文件 ······················································································.198 任务三 Scrapy 网络爬虫文件 ·························································································.199 一、Spider 类 ·······································································································.199 二、配置网络爬虫 ·································································································.201 三、启动网络爬虫 ·································································································.202 四、提取数据 ·······································································································.207 项目实战 ····················································································································.209 实战 提取景区名称 ······························································································.209