教育>高职高专>计算机类
HBase与Hive数据仓库应用开发  

HBase与Hive数据仓库应用开发  "

作者:唐美霞,许建豪,贾瑞民
ISBN:9787121461026
定价:¥49.0
字数:428千字
页数:268
出版时间:2023-08
开本:16开
版次:01-01
装帧:
出版社:电子工业出版社
简介

本书使用大数据存储常用工具与真实场景案例相结合的方式,以项目任务式为导向,较为全面地介绍了HBase分布式数据库与Hive分布式数据仓库的相关知识。全书共9个项目,内容包括认识数据库与数据仓库、安装与配置HBase列存储数据库、使用HBase Shell构建博客数据库系统、使用HBase Java API实现博客数据库系统的应用开发、安装与配置Hive结构化数据仓库、使用Hive实现数据定义操作、使用Hive Shell实现用户优惠券数据分析及处理、使用Hive Java API实现用户优惠券分析的应用开发、以及如何综合使用Hive与HBase存储技术实现电信运营商流失用户分析。本书大部分项目都设置了任务实训及课后习题,通过练习和操作实践,可以帮助读者巩固所学的内容,快速掌握书中所介绍的HBase与Hive存储工具的操作。本书可以作为高校大数据技术相关专业的教材,也可作为大数据技术或数据库爱好者的自学用书。希望通过学习本书内容,读者在提升大数据存储技术的应用能力的同时,也能够养成自主学习的意识,提高发现问题、分析问题和解决问题的能力,具备良好的问题分析素养和独立思考能力,并养成敬业、精益、专注的工匠精神。

前言

随着 5G、AI、云计算、区块链等新一代信息技术的蓬勃发展,大数据技术处于融合发展的关键阶段。随着大数据时代的到来,数据规模呈爆炸性增长,数据类型愈发丰富,数据应用快速深化,各行各业都积累了大量的数据。目前,数据要素市场化配置上升为国家战略,数据正式成为企业、产业乃至国家的战略性资源,得到了广泛关注。管理和分析数据是人类进入信息时代后推动社会进步的关键环节,数据的存储和分析在大数据技术领域占有越来越重要的地位。 大数据存储技术的迅猛发展带动着相关产业的快速发展,各高校对大数据技术相关专业的建设也十分重视。从企业人才需求的角度来看,具备扎实的理论基础又有业务实践经验的大数据人才缺口较大。大数据人才首先应具备存储和处理大数据的能力,能够根据实际业务需求利用各种工具对海量数据进行存储和处理,为后续相关操作做好数据准备。大数据存储是生产环境中必要的过程和阶段,如何掌握并运用大数据存储技术解决实际业务问题是本书的核心要义。本书是 HBase 和 Hive 两大存储工具的基础使用与应用开发实践教程,通过理论结合案例的方式带领读者快速提升 HBase 与 Hive 的基础操作能力和综合运用能力。 本书特色如下。 1. 以项目任务为导向,实现启发式教学。本书注重特色与创新,全书内容紧扣项目需求并拆分成多个任务。每个任务包括任务描述、任务要求、相关知识等内容,根据任务要求讲解知识点,着重启发思路和提供解决方案,让读者对实际项目的流程有初步认识。本书内容由浅入深,使读者明确如何利用所学知识解决问题,实现海量数据的存储和分析。 2. 体现思政育人。党的二十大报告指出,青年强,则国家强。本书的项目中融入了知识目标、技能目标、素养目标,让读者对该项目有初步了解,并帮助读者在学习过程中树立正确的观念,提升多方面的素养和能力;帮助读者提高认识问题、分析问题、解决问题的能力,引导读者在学习过程中培养大数据思维,树立正确的人生观和职业道德观,养成求真务实的科学精神和追求突破的工匠精神,加快推进党的二十大精神进教材、进课堂、进头脑。 3. 基于企业的真实场景和应用,实现理论与实践相结合。本书在介绍了 HBase 与 Hive的相关知识后,结合具体项目讲解了综合实战案例。本书的案例来自企业真实场景,对读者有一定的实践指导作用,能帮助读者提高大数据存储与分析的基本能力,同时培养良好的案例分析能力及思考能力。 本书设置了对应的任务实训及课后习题,帮助读者巩固、理解并应用所学知识。 本书适用对象如下。 1. 开设大数据存储相关课程的高校教师和学生。 2. 具有海量数据存储需求的技术人员。 3. 基于数据库或数据仓库应用的开发人员。 4. 进行大数据存储应用研究的科研人员。 本书由南宁职业技术学院唐美霞、许建豪、贾瑞民主编。唐美霞负责统稿、编写大纲、拟定体例以及定稿;许建豪负责设计本书框架、把关项目内容;贾瑞民负责整理实战案例。 具体分工如下。 南宁职业技术学院苏叶健:项目 1、项目 2; 广东泰迪智能科技股份有限公司张良均:项目 3; 广东泰迪智能科技股份有限公司周津:项目 4; 南宁职业技术学院黄伟:项目 5; 南宁职业技术学院梁凡:项目 6; 南宁职业技术学院唐美霞:项目 7、项目 8、项目 9。 本书编写人员包括高校教师、行业专家、企业人员。在编写的过程中,本书参考了实践中的诸多规律和经验总结,在此谨向这些活跃在大数据技术教学与企业一线的同仁们致以最诚挚的谢意!

目录

项目1 认识数据库与数据仓库 ············································································.1 【教学目标】 ······························································································.1 【背景描述】 ······························································································.1 任务 1 了解大数据 ·····················································································.2 【任务描述】 ···················································································.2 【任务要求】 ···················································································.2 【相关知识】 ···················································································.2 1.1.1 大数据的概念及发展历程 ····················································.2 1.1.2 大数据的数据类型 ·····························································.2 1.1.3 大数据的特点 ···································································.3 1.1.4 大数据的行业应用 ·····························································.4 1.1.5 大数据的技术体系 ·····························································.5 任务 2 了解大数据存储技术 ·········································································.7 【任务描述】 ···················································································.7 【任务要求】 ···················································································.7 【相关知识】 ···················································································.7 1.2.1 大数据存储简介 ································································.7 1.2.2 基于文件系统的数据存储 ····················································.8 1.2.3 基于数据库的数据存储 ·······················································.9 1.2.4 基于数据仓库的数据存储 ··················································.10 项目总结 ·································································································.10 课后习题 ·································································································.11 项目2 安装与部署HBase ················································································.12 【教学目标】 ····························································································.12 【背景描述】 ····························································································.12 任务 1 搭建完全分布式 Hadoop 集群 ····························································.13 【任务描述】 ·················································································.13 【任务要求】 ·················································································.13 【相关知识】 ·················································································.13 2.1.1 Hadoop 简介 ···································································.13 2.1.2 Hadoop 的核心组件 ··························································.14 2.1.3 Hadoop 生态系统 ·····························································.17 2.1.4 搭建 Hadoop 集群前的准备工作 ··········································.19 【任务实施】 ·················································································.19 【任务实训】 ·················································································.47 任务 2 安装 ZooKeeper 集群 ·······································································.47 【任务描述】 ·················································································.47 【任务要求】 ·················································································.47 【相关知识】 ·················································································.48 2.2.1 ZooKeeper 简介 ·······························································.48 2.2.2 ZooKeeper 的架构 ····························································.48 【任务实施】 ·················································································.49 任务 3 安装与配置 HBase 集群 ···································································.51 【任务描述】 ·················································································.51 【任务要求】 ·················································································.51 【相关知识】 ·················································································.52 2.3.1 HBase 简介 ····································································.52 2.3.2 HBase 的核心功能模块 ·····················································.53 2.3.3 HBase 的读/写流程 ··························································.54 【任务实施】 ·················································································.55 【任务实训】 ·················································································.58 项目总结 ·································································································.59 课后习题 ·································································································.59 项目3 使用HBase Shell 构建博客数据库系统 ······················································.60 【教学目标】 ····························································································.60 【背景描述】 ····························································································.60 任务 1 设计 HBase 表 ···············································································.61 【任务描述】 ·················································································.61 【任务要求】 ·················································································.61 【相关知识】 ·················································································.61 3.1.1 HBase 的数据模型 ···························································.61 3.1.2 HBase 表的结构设计原则 ··················································.62 3.1.3 HBase 的检索方式 ···························································.63 3.1.4 RowKey 设计原则 ····························································.63 3.1.5 热点问题 ·······································································.63 3.1.6 列族设计原则 ·································································.64 【任务实施】 ·················································································.65 任务 2 创建 HBase 表 ···············································································.66 【任务描述】 ·················································································.66 【任务要求】 ·················································································.66 【相关知识】 ·················································································.66 3.2.1 命名空间 ·······································································.66 3.2.2 创建表 ··········································································.67 3.2.3 查看表结构 ····································································.68 3.2.4 修改表 ··········································································.69 3.2.5 删除表 ··········································································.69 【任务实施】 ·················································································.69 【任务实训】 ·················································································.70 任务 3 查询 HBase 表数据 ·········································································.70 【任务描述】 ·················································································.70 【任务要求】 ·················································································.70 【相关知识】 ·················································································.71 3.3.1 插入数据 ·······································································.71 3.3.2 查询数据 ·······································································.72 3.3.3 扫描全表数据 ·································································.72 3.3.4 删除数据 ·······································································.73 3.3.5 清空数据 ·······································································.73 【任务实施】 ·················································································.73 【任务实训】 ·················································································.75 任务 4 查询符合指定条件的 HBase 表数据 ····················································.76 【任务描述】 ·················································································.76 【任务要求】 ·················································································.76 【相关知识】 ·················································································.77 3.4.1 HBase 高级查询 ······························································.77 3.4.2 HBase 的抽象操作符 ························································.77 3.4.3 HBase 的比较器 ······························································.77 3.4.4 HBase 的过滤器 ······························································.78 【任务实施】 ·················································································.79 【任务实训】 ·················································································.79 项目总结 ·································································································.80 课后习题 ·································································································.80 项目4 使用HBase Java API 开发博客数据库系统 ·················································.82 【教学目标】 ····························································································.82 【背景描述】 ····························································································.82 任务 1 搭建 HBase 开发环境 ······································································.83 【任务描述】 ·················································································.83 【任务要求】 ·················································································.83 【任务实施】 ·················································································.83 任务 2 插入并查询数据 ·············································································.95 【任务描述】 ·················································································.95 【任务要求】 ·················································································.95 【相关知识】 ·················································································.95 4.2.1 HBase Java API 的主要接口与类 ··········································.95 4.2.2 使用 HBase Java API 创建命名空间和表 ································.98 4.2.3 使用 HBase Java API 插入数据 ············································.99 4.2.4 使用 HBase Java API 查询数据 ···········································.100 4.2.5 使用 HBase Java API 进行全表查询 ·····································.101 【任务实施】 ················································································.102 【任务实训】 ················································································.111 任务 3 查询符合指定条件的数据 ································································.112 【任务描述】 ················································································.112 【任务要求】 ················································································.112 【相关知识】 ················································································.112 4.3.1 Hbase 过滤器API ···························································.112 【任务实施】 ················································································.114 【任务实训】 ················································································.120 任务 4 实现 MapReduce 与 HBase 表的集成 ··················································.120 【任务描述】 ················································································.120 【任务要求】 ················································································.121 【相关知识】 ················································································.121 4.4.1 Hadoop 集群运行 MapReduce 程序 ······································.121 4.4.2 将数据导入 Hbase 表中 ····················································.122 4.4.3 导出 HBase 表中的数据 ···················································.123 【任务实施】 ················································································.123 【任务实训】 ················································································.128 项目总结 ································································································.129 课后习题 ································································································.129 项目5 安装与配置Hive 结构化数据仓库 ···························································.131 【教学目标】 ···························································································.131 【背景描述】 ···························································································.131 任务 1 安装与配置 Hive ···········································································.132 【任务描述】 ················································································.132 【任务要求】 ················································································.132 【相关知识】 ················································································.132 5.1.1 Hive 的起源与发展 ·························································.132 5.1.2 Hive 与传统数据库的对比 ················································.132 5.1.3 Hive 的系统架构 ····························································.133 5.1.4 Hive 的工作原理 ····························································.134 5.1.5 安装前的准备工作 ··························································.135 【任务实施】 ················································································.135 【任务实训】 ················································································.142 任务 2 在 Hive CLI 界面执行 Shell 命令和 dfs 命令 ·············································.143 【任务描述】 ················································································.143 【任务要求】 ················································································.143 【相关知识】 ················································································.143 5.2.1 在文件中执行 Hive 查询 ···················································.143 5.2.2 在 Hive 中执行 Linux Shell 命令 ·········································.145 5.2.3 在 Hive 中使用 Hadoop 的 dfs 命令 ·····································.146 5.2.4 在 Hive 脚本中进行注释 ···················································.147 【任务实施】 ················································································.148 【任务实训】 ················································································.149 项目总结 ································································································.150 课后习题 ································································································.150 项目6 使用Hive 定义优惠券数据 ····································································.152 【教学目标】 ···························································································.152 【背景描述】 ···························································································.152 任务 1 创建 Hive 表 ·················································································.153 【任务描述】 ················································································.153 【任务要求】 ················································································.153 【相关知识】 ················································································.153 6.1.1 Hive 的数据类型 ····························································.153 6.1.2 创建与管理数据仓库 ·······················································.154 6.1.3 创建表 ·········································································.156 6.1.4 修改表 ·········································································.160 【任务实施】 ················································································.161 【任务实训】 ················································································.162 任务 2 向Hive 表中导入数据 ·····································································.163 【任务描述】 ················································································.163 【任务要求】 ················································································.163 【相关知识】 ················································································.163 6.2.1 导入数据 ······································································.163 6.2.2 导出数据 ······································································.167 【任务实施】 ················································································.168 【任务实训】 ················································································.169 项目总结 ································································································.170 课后习题 ································································································.170 项目7 使用Hive Shell 实现优惠券消费数据的分析及处理 ·····································.172 【教学目标】 ···························································································.172 【背景描述】 ···························································································.172 任务 1 查询领取了优惠券的用户信息 ··························································.173 【任务描述】 ················································································.173 【项目要求】 ················································································.173 【相关知识】 ················································································.173 7.1.1 select 基本查询 ······························································.173 7.1.2 limit 结果限制 ································································.175 7.1.3 distinct 去重查询 ····························································.176 7.1.4 where 条件查询 ······························································.176 7.1.5 Hive 内置运算符 ····························································.177 7.1.6 正则表达式 ···································································.179 【任务实施】 ················································································.180 【任务实训】 ················································································.182 任务 2 构建用户标签列 ············································································.183 【任务描述】 ················································································.183 【任务要求】 ················································································.183 【相关知识】 ················································································.183 7.2.1 case……when……语句的使用 ···········································.183 7.2.2 group by 分组查询 ··························································.184 7.2.3 having 条件筛选 ·····························································.185 【任务实施】 ················································································.185 【任务实训】 ················································································.186 任务 3 构建用户特征字段 ·········································································.187 【任务描述】 ················································································.187 【任务要求】 ················································································.187 【相关知识】 ················································································.187 7.3.1 Hive 内置函数 ·······························································.187 7.3.2 排序查询 ······································································.193 【任务实施】 ················································································.193 【任务实训】 ················································································.197 任务 4 连接用户特征字段 ·········································································.198 【任务描述】 ················································································.198 【任务要求】 ················································································.198 【相关知识】 ················································································.198 7.4.1 union 结果集合并 ···························································.198 7.4.2 join 连接表数据 ······························································.200 【任务实施】 ················································································.201 【任务实训】 ················································································.202 项目总结 ································································································.202 课后习题 ································································································.203 项目8 使用Hive Java API 开发优惠券消费数据分析应用 ······································.205 【教学目标】 ···························································································.205 【背景描述】 ···························································································.205 任务 1 搭建 Hive 开发环境 ········································································.206 【任务描述】 ················································································.206 【任务要求】 ················································································.206 【任务实施】 ················································································.206 任务 2 编写自定义函数统计优惠券折扣 ·······················································.212 【任务描述】 ················································································.212 【任务要求】 ················································································.212 【相关知识】 ················································································.213 8.2.1 Hive 自定义函数 ····························································.213 8.2.2 UDF 函数 ·····································································.213 8.2.3 UDAF 函数 ···································································.217 8.2.4 UDTF 函数 ···································································.218 【任务实施】 ················································································.220 【任务实训】 ················································································.221 任务 3 构建及合并特征字段 ······································································.222 【任务描述】 ················································································.222 【任务要求】 ················································································.222 【相关知识】 ················································································.222 8.3.1 Hive Java API 的主要类 ····················································.222 8.3.2 执行 SQL 语句的方法 ······················································.224 【任务实施】 ················································································.225 【任务实训】 ················································································.229 项目总结 ································································································.230 课后习题 ································································································.230 项目9 基于HBase 和Hive 的电信运营商用户数据分析实战 ··································.233 【教学目标】 ···························································································.233 【背景描述】 ···························································································.233 任务 1 案例背景和需求分析 ······································································.233 【任务描述】 ················································································.233 【任务要求】 ················································································.234 【任务实施】 ················································································.234 任务 2 数据预处理 ··················································································.236 【任务描述】 ················································································.236 【任务要求】 ················································································.236 【任务实施】 ················································································.236 任务 3 用户数据的基本查询 ······································································.240 【任务描述】 ················································································.240 【任务要求】 ················································································.240 【任务实施】 ················································································.240 任务 4 分析用户通话情况 ·········································································.241 【任务描述】 ················································································.241 【任务要求】 ················································································.242 【任务实施】 ················································································.242 任务 5 将 Hive 的数据导入 HBase 中 ···························································.244 【任务描述】 ················································································.244 【任务要求】 ················································································.244 【任务实施】 ················································································.244 项目总结 ································································································.253 附录 大数据组件的常用端口及其说明 ·······························································.254 参考文献 ······································································································.256

作者简介

编辑推荐

作者寄语

电子资料

http://www.hxedu.com.cn/hxedu/fg/book/bookinfo.html?code=G0461020

www.luweidong.cn

下一个