本书面向从事大数据存储系统设计及相关技术研究和开发工作的读者。既可以作为存储

系统架构师、软件开发工程师、产品或项目经理、数据中心运维人员等的实用工具书，还可

以作为普通高等院校计算机和大数据科学相关专业的教学或科研人员、研究生、高年级本科

生及相关培训机构学员的学习参考书。

目 录

 第1章 概 述 ·································································································1

1.1　大数据简介 ····································································2

1.1.1　大数据定义和维度 ···························································2

1.1.2　大数据管理挑战 ······························································6

1.2　高效能存储管理 ······························································9

1.2.1　存储虚拟化 ··································································10

1.2.2 自动分层存储 ·······························································13

1.2.3　自动精简配置 ·······························································17

1.2.4　数据缩减技术 ·······························································19

1.3　本章小结 ·····································································21

 第2章 存储技术基础 ····················································································23

2.1　存储介质 ·····································································24

2.1.1　磁存储介质 ··································································24

2.1.2　光存储介质 ··································································26

2.1.3　电子存储介质 ·······························································28

2.2　存储接口 ·····································································32

2.2.1　IDE接口 ······································································33

2.2.2　SATA接口 ···································································33

2.2.3　SCSI接口 ····································································34

2.2.4　SAS接口 ·····································································35

2.2.5　PCIE接口 ····································································36

2.2.6　FC接口 ·······································································37

2.3　存储系统架构 ·······························································38

2.3.1　存储器分层结构 ····························································38



2.3.2　RAID技术 ···································································40

2.4　网络存储系统 ·······························································44

2.4.1　直接连接存储 ·······························································44

2.4.2　网络附加存储 ·······························································46

2.4.3　存储区域网络 ·······························································47

2.4.4　基于对象存储 ·······························································48

2.4.5　几种存储结构的比较 ······················································49

2.5　本章小结 ·····································································51

 第3章 大数据管理技术 ·················································································52

3.1　分布式计算框架 ····························································53

3.2　分布式文件系统 ····························································56

3.2.1　HDFS ·········································································58

3.2.2　Ceph ··········································································59

3.2.3　Lustre　 ······································································61

3.3　NoSQL数据库 ······························································63

3.3.1　HBase ·········································································64

3.3.2　MongoDB ····································································66

3.4　大数据仓库 ··································································67

3.4.1　Hive ···········································································68

3.4.2　Pig ·············································································69

3.4.3　Phoenix ·······································································70

3.5　本章小结 ·····································································73

 第4章 重复数据删除存储系统 ······································································74

4.1　重复数据删除的概念及分类 ·············································75

4.1.1　基本概念 ·····································································76

4.1.2　技术分类 ·····································································78

4.2　重复数据删除存储原理 ···················································81

4.2.1　系统架构 ·····································································81

4.2.2　主要步骤 ·····································································82

4.2.3　衡量指标 ·····································································83

4.3　重复数据删除技术应用场景 ·············································84

4.3.1　数据备份 ·····································································84



XI

4.3.2　归档存储 ·····································································85

4.3.3　远程容灾 ·····································································85

4.3.4　虚拟化环境 ··································································86

4.3.5　主存储系统 ··································································86

4.3.6　新型存储介质 ·······························································87

4.4　相关产品及开源项目 ······················································87

4.4.1　企业产品 ·····································································88

4.4.2　开源项目 ·····································································91

4.5　本章小结 ·····································································94

 第5章 重复数据删除关键技术 ······································································95

5.1　数据划分方法 ·······························································96

5.1.1　全文件分块 ··································································97

5.1.2　静态分块 ·····································································97

5.1.3　基于内容分块 ·······························································98

5.1.4　基于应用分块 ······························································100

5.1.5　Delta编码 ···································································100

5.2　块索引查询优化技术 ·····················································102

5.2.1　基于块局部性的优化策略 ···············································102

5.2.2　基于分层消重的优化策略 ···············································104

5.2.3　基于固态存储的优化策略 ···············································105

5.3　可扩展数据路由技术 ·····················································107

5.3.1　基于分布式哈希表的块级数据路由技术 ·····························107

5.3.2　基于状态信息的超块级数据路由技术 ································108

5.3.3　基于相似性的文件级数据路由技术 ···································109

5.4　块指纹计算加速方法 ·····················································110

5.4.1　多核CPU加速方法 ························································110

5.4.2　GPGPU加速方法 ··························································111

5.5　数据还原方法 ······························································113

5.5.1　主存储还原方法 ···························································113

5.5.2　备份存储还原方法 ························································113

5.5.3　云存储还原方法 ···························································115

5.6　垃圾回收技术 ······························································116



5.6.1　引用计数法 ·································································116

5.6.2　标记清理法 ·································································117

5.7　高可靠数据配置策略 ·····················································118

5.7.1　纠错编码技术 ······························································118

5.7.2　副本策略 ····································································119

5.8　数据安全技术 ······························································120

5.8.1　加密冲突 ····································································121

5.8.2　旁路攻击 ····································································121

5.8.3　所有权证明 ·································································122

5.9　本章小结 ····································································123

 第6章 应用感知源端重复数据删除机制 ······················································124

6.1　研究背景 ····································································125

6.2　相关研究工作 ······························································127

6.2.1　应用感知存储研究 ························································127

6.2.2　基于源端重复数据删除的云备份研究 ································127

6.3　基本形式化模型 ···························································128

6.3.1　符号与基本定义 ···························································128

6.3.2　模型抽象与问题定义 ·····················································129

6.4　研究动机 ····································································132

6.5　高效应用感知源端重复数据删除的设计与实现 ·································139

6.5.1　ALG-Dedupe体系结构简介 ·············································139

6.5.2　文件大小过滤器 ···························································140

6.5.3　智能数据分块策略 ························································141

6.5.4　应用感知的消重器 ························································141

6.5.5　应用感知索引结构 ························································142

6.5.6　段和容器管理 ······························································143

6.6　实验评估 ····································································144

6.6.1　实验平台和数据集 ························································144

6.6.2　重复数据删除效果 ························································145

6.6.3　重复数据删除效率 ························································146

6.6.4　云备份窗口 ·································································147

6.6.5　能耗利用率 ·································································148



6.6.6　云存储代价 ·································································149

6.6.7　系统开销 ····································································150

6.7　本章小结 ····································································151

 第7章 高可扩展集群重复数据删除技术 ······················································152

7.1　研究背景 ····································································154

7.2　相关研究工作 ······························································155

7.3　基本模型与算法 ···························································157

7.3.1　超块相似性分析 ···························································158

7.3.2　基于手纹的数据路由算法 ···············································160

7.4　系统设计与实现 ···························································162

7.4.1　Σ-Dedupe系统架构 ······················································162

7.4.2　数据路由消息通信 ························································164

7.4.3　相似索引查询优化 ························································165

7.5　性能评估 ····································································166

7.5.1　实验平台和工作负载 ·····················································167

7.5.2　验证度量 ····································································167

7.5.3　单节点并行重复数据删除效率 ·········································168

7.5.4　集群重复数据删除效率 ··················································172

7.6　本章小结 ····································································176

 第8章 重复数据删除存储案例分析 ·····························································177

8.1　重复数据删除缩减率评估 ···············································178

8.2　主流厂商相关产品应用案例 ············································181

8.2.1　Dell EMC公司产品案例 ·················································181

8.2.2　IBM公司产品案例 ························································195

8.2.3　飞康软件公司产品案例 ··················································199

8.2.4　富士通公司产品案例 ·····················································204

8.2.5　NetApp公司产品案例 ····················································211

8.2.6　Quantum公司产品案例 ··················································218

8.3　本章小结 ····································································224



重复数据删除是提升大数据性能、节省成本的极高效方法。本书将各种前沿技术以及落地方案进行了讲解，对大数据从业者、存储从业者、院校师生都是一本极好的参考资料。

付印金 博士

军事科学院系统工程研究院博士后、陆军工程大学讲师，现为中国计算机学会高级会员、信息存储技术/系统软件专业委员会委员。长期从事网络存储、大数据管理与云计算方向的课题研究。已主持完成国家自然科学基金、江苏省自然科学基金等课题多项，发表学术论文50多篇，申请和授权国家发明专利10余项。主讲过“计算机组成原理”、“计算机网络”与“云计算”等本科与研究生专业课程。

 

肖侬 教授

中山大学国家超算计算广州中心教授、博士生导师，国防科技大学教授。获得教育部长江学者特聘教授、国家杰出青年科学基金资助。现任CCF Fellow、中国计算机学会大数据专家委员会副主任、信息存储技术专业委员会副主任。研究方向为网络计算与云计算、新型存储和系统结构，是我国高性能网络计算技术研究的先行开拓者之一。负责过863重大项目和国家自然科学基金重点项目等课题，发表学术论文200多篇。