
时序数据库是一种新型技术,主要用于工业互联网软件建设中。近年来,伴随着物联网技术在智能制造、交通、能源、智慧城市等领域的发展,时序数据库也发展迅速,成为搭建应用的必备数据库之一。《深入理解 InfluxDB》从 InfluxDB 的安装开始,一步步详细介绍 InfluxDB 的功能及原理,带领读者深入理解以 InfluxDB 为代表的时序数据库。
据IDC预测,全球数据将从2018年的33ZB(泽字节)增长到2025年的175ZB,年均复合增长率为23%。到2025年,全球联网设备会有1500亿台,其中大多数设备将产生实时数据。届时,实时数据将超过其他类型的数据,成为全球第一大数据。
智能制造2025、工业互联网、工业4.0等政策推动了时序数据库的发展,全球对工业品的需求逐年上升。中国作为工业大国,2020年开始大力发展新基建,时序数据库作为智能制造的基础软件,其发展也同步受到关注。
和操作系统一样,数据库属于基础软件,更新迭代慢,一旦使用,可以在生产环境运行十几年,替换成本高,且企业替换意愿不强。企业初期选择数据库会非常慎重,使用后无论是管理层、执行层,还是技术人员都很难主动提出替换建议。
替换成本高、周期长、风险高造成了企业只相信头部数据库产品。毫无疑问,目前使用最多、技术生态最完善的时序数据库之一就是InfluxDB。InfluxDB是一个开源的、高性能的时序数据库,在时序数据库榜单DB-Engines Ranking上排名第一。正因为如此,熟练掌握InfluxDB已成为相关开发者非常重要的一项技能。本书将带领大家从InfluxDB的基础开始,一步一步理解InfluxDB,相信这些内容会对大家有帮助。
如何阅读本书
本书既是教程,又是参考指南,同时也适合作为高校相关专业教材。如果读者刚刚接触时序数据库开发,按照本书的章节顺序学习定会有所收获。
本书共14章,每章的内容简单介绍如下:
第1章主要介绍了什么是时序数据、InfluxDB是什么,以及InfluxDB的发展历史。
第2章主要介绍了InfluxDB的安装及配置。
第3章主要介绍了InfluxDB的可视化UI工具。
第4章主要介绍了InfluxDB的基本写入、查询操作。
第5章主要介绍了InfluxDB的常用函数和运算。
第6章主要介绍了InfluxDB的连续查询。
第7章主要介绍了InfluxDB的数据保留策略。
第8章主要介绍了InfluxDB的数据安全策略。
第9章主要介绍了InfluxDB的性能评估。
第10章主要介绍了InfluxDB集群相关内容。
第11章主要介绍了InfluxDB的数据备份与恢复。
第12章主要介绍了InfluxDB与程序设计。
第13章主要介绍了InfluxDB数据处理语言FLux。
第14章主要介绍了InfluxDB存储引擎知识。
读者对象
在本书的编写过程中,尽可能做到通俗易懂、由浅入深,不仅适用于初学者学习,也适用于专业人员学习。
本书可作为高校相关专业教材,也适合软件工程师、软件架构师、数据库工程师等从业人员阅读。
读者交流与图书反馈
本书的读者还可以访问InfluxDB专栏补充学习。该专栏搭建了一个供开发者交流学习的在线平台,阅读过程中如有疑问,也可以在网站上向作者提问,期待能够得到你们的真挚反馈。
由于作者水平有限,编写时间仓促,书中难免会出现一些错误或表达不准确的地方,恳请读者批评指正。我们也会将书中的勘误发布在专栏中,供大家参考。
编写说明
本书编写团队由业界一线研发人员组成,其中郑强负责全书的框架搭建及第1~9章的撰写,字数约28万字;张伟负责第10~13章的撰写,字数约6万字;刘爽负责第14章的撰写,字数约2万字。
致谢
感谢清华大学出版社的编辑,因为你们的帮助,这本书才得以问世。最后要感谢的就是你,我亲爱的读者,感谢你拿起这本书,你的认可是我们最大的快乐。
作者
第1章 时序数据库InfluxDB简介
1.1 数据简史 ······························································································2
1.1.1 什么是时序数据 ··················································································3
1.1.2 时序数据库的历史 ···············································································3
1.2 InfluxDB简介 ························································································4
1.2.1 什么是InfluxDB ··················································································4
1.2.2 InfluxDB的历史 ··················································································5
1.2.3 InfluxDB现状及未来 ············································································6
1.3 InfluxDB的基本概念 ···············································································7
1.4 InfluxDB的设计理念 ···············································································7
1.5 InfluxDB的核心特性 ···············································································8
1.5.1 内置REST接口 ··················································································8
1.5.2 数据Tag标记 ·····················································································8
1.5.3 类SQL的查询语句 ··············································································9
1.5.4 高性能 ······························································································9
1.5.5 开源 ·······························································································10
1.6 时序数据库的应用场景 ··········································································10
1.6.1 在工业环境监控中的应用 ····································································10
1.6.2 在物联网IoT设备采集存储中的应用 ·····················································11
1.6.3 互联网业务性能监控服务 ····································································11
1.6.4 在智能汽车中的应用 ··········································································12
1.7 小结 ··································································································12
第2章 InfluxDB的安装、配置、启动
2.1 在不同操作系统上安装InfluxDB································································14
2.1.1 硬件要求 ·························································································14
2.1.2 在Ubuntu和Debian系统中安装InfluxDB ···············································15
2.1.3 在RedHat和CentOS系统中安装InfluxDB ··············································20
2.1.4 在Windows系统中安装InfluxDB··························································21
2.1.5 在macOS系统中安装InfluxDB ·····························································22
2.1.6 使用Docker安装InfluxDB ···································································24
2.1.7 InfluxDB的端口设置 ··········································································25
2.1.8 InfluxDB 程序的使用 ··········································································25
2.1.9 InfluxDB的配置详解 ··········································································31
2.2 小结 ··································································································37
第3章InfluxDB UI数据可视化
3.1 InfluxDB UI ·························································································39
3.1.1 安装配置 ·························································································39
3.1.2 常用功能介绍 ···················································································41
3.2 Chronograf ·························································································47
3.2.1 安装配置 ·························································································48
3.2.2 常用功能介绍 ···················································································50
3.3 小结 ··································································································57
第4章InfluxDB基本操作写入与查询
4.1 客户端命令行方式 ················································································59
4.1.1 简介·······························································································59
4.1.2 使用InfluxDB ···················································································59
4.1.3 Influx基本命令·················································································61
4.2 数据样本 ····························································································62
4.2.1 空气传感器样本数据 ··········································································63
4.2.2 鸟类迁徙样本数据 ·············································································64
4.2.3 NOAA样本数据 ················································································64
4.2.4 美国地质勘探局地震数据 ····································································64
4.3 行协议 ·······························································································65
4.3.1 行协议案例 ······················································································65
4.3.2 行协议语法 ······················································································66
4.3.3 行协议要素分析 ················································································66
4.3.4 数据类型 ·························································································67
4.3.5 引号·······························································································68
4.3.6 特殊字符 ·························································································68
4.3.7 注释·······························································································69
4.3.8 重复数据 ·························································································69
4.4 桶操作 ·······························································································70
目 录
VII
4.5 写入操作 ····························································································71
4.5.1 写入数据 ·························································································71
4.5.2 文件数据导入 ···················································································72
4.6 查询操作 ····························································································73
4.6.1 select语句 ·······················································································74
4.6.2 类型转换 ·························································································77
4.6.3 where子句 ·······················································································78
4.6.4 函数 ·······························································································80
4.6.5 group by子句 ···················································································82
4.6.6 into子句 ··························································································85
4.6.7 排序 ·······························································································86
4.6.8 limit和slimit子句 ·············································································86
4.6.9 offset和soffset子句 ···········································································86
4.6.10 Time Zone子句 ················································································87
4.6.11 时间语法 ························································································87
4.7 小结 ··································································································88
第5章 InfluxDB的函数与运算
5.1 样本数据导入 ······················································································90
5.2 InfluxDB函数 ······················································································91
5.2.1 函数说明 ·························································································91
5.2.2 聚合类函数 ······················································································92
5.2.3 选择类函数 ······················································································94
5.2.4 转换类函数 ······················································································95
5.2.5 预测类函数 ······················································································98
5.3 数学运算 ····························································································99
5.3.1 常用运算符 ······················································································99
5.3.2 数学运算符的常见问题 ······································································101
5.4 小结 ································································································102
第6章 InfluxDB连续查询
6.1 连续查询 ··························································································104
6.2 样本数据导入 ····················································································104
6.3 创建连续查询 ····················································································106
6.3.1 <cq_query>语句 ···············································································106
6.3.2 连续查询运行时刻及查询的时间范围 ····················································107
6.3.3 连续查询举例 ··················································································107
6.4 复杂连续查询 ····················································································109
6.4.1 创建高级连续查询 ············································································109
6.4.2 高级连续查询的时间设置 ···································································109
6.4.3 GROUP BY time()、EVERY、FOR三者关系 ···········································111
6.4.4 高级连续查询举例 ············································································112
6.5 连续查询的管理 ··················································································113
6.5.1 查询所有连续查询 ············································································113
6.5.2 删除连续查询 ··················································································114
6.5.3 修改连续查询 ··················································································114
6.6 连续查询案例分析 ···············································································114
6.6.1 数据预处理 ·····················································································115
6.6.2 降低数据采样率 ···············································································115
6.6.3 HAVING功能 ··················································································116
6.6.4 替换嵌套函数 ··················································································116
6.7 小结 ·································································································117
第7章InfluxDB数据保留策略
7.1 保留策略 ···························································································119
7.2 创建保留策略 ·····················································································119
7.2.1 使用CREATE RETENTION POLICY创建保留策略 ························119
7.2.2 创建保留策略举例 ············································································121
7.3 查询保留策略 ····················································································121
7.4 修改保留策略 ····················································································122
7.5 删除保留策略 ····················································································123
7.6 综合实例 ··························································································123
7.6.1 样本数据 ························································································123
7.6.2 实验目标 ························································································124
7.6.3 实验过程 ························································································124
7.6.4 结论······························································································126
7.7 小结 ································································································126
第8章InfluxDB数据安全策略
8.1 认证技术 ··························································································128
8.1.1 简介······························································································128
8.1.2 认证方式 ························································································128
8.1.3 Token认证 ······················································································128
8.1.4 1.x版本兼容授权认证 ·······································································131
目 录
8.1.5 授权使用 ························································································137
8.2 权限类型 ··························································································142
8.3 用户管理 ··························································································143
8.4 权限控制实战 ····················································································146
8.4.1 创建与授权 ·····················································································146
8.4.2 测试效果 ························································································147
8.5 小结 ································································································149
第9章 InfluxDB性能评估
9.1 性能测试工具 ····················································································151
9.1.1 安装Go语言运行环境 ·······································································151
9.1.2 测试工具安装 ··················································································153
9.2 基准性能测试 ····················································································154
9.2.1 测试环境 ························································································154
9.2.2 写入性能测试 ··················································································154
9.3 性能优化 ··························································································157
9.3.1 数据写入优化方案 ············································································157
9.3.2 数据查询优化方案 ············································································157
9.4 性能报警 ··························································································159
9.4.1 创建检查 ························································································159
9.4.2 添加通知端点 ··················································································162
9.4.3 创建通知规则 ··················································································163
9.5 小结 ································································································164
第10章 InfluxDB集群
10.1 集群简介 ·························································································166
10.1.1 集群架构概述 ················································································166
10.1.2 数据所在的地方 ·············································································166
10.1.3 节点数量 ······················································································167
10.2 集群安装配置 ···················································································167
10.2.1 申请试用 ······················································································167
10.2.2 环境准备 ······················································································169
10.2.3 添加DNS条目 ···············································································170
10.2.4 Meta节点的安装配置 ·······································································171
10.2.5 Data节点的安装配置 ·······································································173
10.3 小结 ·······························································································176
第11章 备份管理
11.1 单机版备份管理 ················································································178
11.1.1 备份数据 ·······················································································178
11.1.2 恢复数据 ·······················································································179
11.2 集群版备份管理 ················································································180
11.2.1 备份数据 ·······················································································181
11.2.2 恢复数据 ·······················································································182
11.2.3 导出数据 ·······················································································184
11.2.4 导入数据 ·······················································································185
11.3 小结 ·······························································································185
第12章 InfluxDB与程序设计
12.1 Java SDK使用 ·················································································187
12.1.1 使用Java在InfluxDB中写入数据 ·······················································188
12.1.2 WriteApiBlocking ············································································192
12.1.3 使用Java查询InfluxDB中的数据 ·······················································193
12.1.4 queryApi ·······················································································195
12.1.5 使用Java为bucket添加权限 ·····························································196
12.1.6 连接InfluxDB的另一种方式—用户名和密码 ······································198
12.2 Python SDK使用 ··············································································200
12.2.1 使用Python在InfluxDB中写入数据 ····················································201
12.2.2 使用Python查询InfluxDB中的数据 ····················································202
12.3 小结 ·······························································································203
第13章 InfluxDB数据处理语言Flux
13.1 Flux概述 ························································································205
13.2 基本数据类型 ···················································································205
13.2.1 Boolean(布尔值) ·········································································206
13.2.2 Bytes(字节) ················································································206
13.2.3 Duration(持续时间) ······································································206
13.2.4 String(字符串类型) ······································································207
13.2.5 Time(时间类型) ··········································································207
13.2.6 Float(浮点类型) ··········································································208
13.2.7 Integer(整数类型) ········································································208
13.2.8 Null(空值) ·················································································209
13.3 查询数据源 ······················································································209
13.3.1 from()函数 ····················································································210
13.3.2 range()函数 ···················································································210
13.3.3 查询InfluxDB 1.x ············································································211
13.3.4 远程查询InfluxDB Cloud 或InfluxDB 2.x ··············································211
13.4 写入数据源 ······················································································212
13.5 小结 ·······························································································213
第14章 InfluxDB存储引擎
14.1 InfluxDB存储引擎历史 ·······································································215
14.2 LSM-Tree(LSM树)概述··································································215
14.2.1 LSM-Tree存储原理 ·········································································216
14.2.2 优势和问题 ···················································································220
14.3 InfluxDB数据格式 ·············································································220
14.4 TSM存储组件 ··················································································221
14.5 TSM file详解 ··················································································222
14.5.1 SeriesKey(时间序列关键字) ···························································223
14.5.2 Series Data Block ·············································································224
14.5.3 Series Index Block ············································································225
14.6 TSM数据写入 ··················································································227
14.6.1 写入总体框架 ················································································227
14.6.2 Shard路由 ·····················································································228
14.6.3 倒排索引引擎构建倒排索引 ······························································228
14.6.4 写入流程 ······················································································229
14.7 小结 ·······························································································230
《深入理解 InfluxDB》适合时序数据库工程师、物联网采集工程师、运维工程师阅读,也可供数据库爱好者、监控技术爱好者、工业互联网平台爱好者参考学习,还适合作为高校相关专业教材。