本书围绕BFE 开源项目，介绍网络前端接入和网络负载均衡的相关技术原理，说明BFE开源软件的设计思想和实现机制，讲解如何基于BFE开源软件搭建网络接入平台。本书共17 章，分为四个部分。第一部分为原理篇，介绍BFE 开源项目的概貌，并对网络前端接入技术和网络负载均衡技术做简要介绍；第二部分为设计篇，介绍BFE 开源项目的设计细节，包括BFE 的设计思想、BFE 的转发模型、BFE 的关键机制和HTTPS 的优化等；第三部分为操作篇，说明如何安装和部署BFE、如何在各种使用场景下配置BFE；第四部分为实现篇，从处理流程、插件机制、协议实现等方面说明BFE的实现机制。

序 从2014年4月写下Go语言版本BFE的第一行代码起，7年多的时间过去了。从2015年1月Go语言版本BFE全量上线开始，BFE至今已经在百度稳定运行了6年多的时间，每天转发请求超过万亿次。 BFE（Baidu Front End，百度统一前端）是百度统一七层流量转发平台，当你访问百度的时候，很可能已经在使用BFE的服务了。 百度的BFE团队始建于2012年。当2012年年底我加入百度的时候，整个团队只有6~7个人。这个团队的创始人是夏华夏同学（现在在美团），他为BFE团队的工作方向做了奠基性规划。BFE初期的转发引擎是基于C语言的，听说是杨震原同学（现在在头条）的大作。 2014年年初，基于各种考虑，我们决定对转发引擎进行重构。这次重构前后花费了3个季度，投入了超过30个人月的资源。在面对多次失败的风险后，Go语言版本的BFE终于出炉了。 这里必须感谢部门领导李硕和团队经理贺锋的大力支持，感谢管理层的高度信任；感谢直接参与的几位同学（李炳毅、魏为、杨思杰、陶春华等），大家都是冒着失败离职的风险，硬着头皮把这个项目做下来的。 我必须要感谢百度。到目前为止，我仍然坚定地认为，百度是中国最适合做技术的公司。百度给了工程师最大的尊重和自由，也愿意为了技术研发承担最大的风险。BFE团队的另一个项目GTC(全局流量调度)，前后研发了5年时间。曾经有一个朋友告诉我，也就只有百度可以给团队这么多的时间，如果在其他公司，一年内做不出来，项目很可能就被取消了。能够在百度、在中国做全球最领先的技术，我感到无比骄傲。 2019年7月，BFE的转发引擎对外开源。项目名称仍保留英文缩写BFE，英文全称更名为Beyond Front End（中文意为“超越前端”）。我们希望通过BFE的开源推动负载均衡技术的发展。 从开源的那天起，BFE就已经开始了新的征程。BFE得到了各方的广泛关注，有不少新增的功能是由百度之外的开发者贡献的。BFE也被一些客户选择用于关键的业务场景，在度小满金融、央视网、招商银行等处都有BFE的身影。作为一个做技术的人，能够让自己所做的工作为社会创造价值，这是莫大的幸福和荣幸。 在BFE开源后，我们不断地收到大家提出的一些问题。网络负载均衡本身是一个比较专业和复杂的技术方向；BFE是为面向工业级使用场景而设计的，在模型和机制上和其他同类软件相比，会更加复杂。以上这些因素让一些使用者和开发者在理解BFE的机制方面遇到了困难。希望通过《深入理解BFE》这本书，能够帮助读者更好地了解网络负载均衡的相关技术，让读者更容易地理解BFE的设计机制和使用方法。 BFE项目，是一群技术人的汗水、梦想和追求。 BFE开源，是为了交流、共享，为全中国、全世界的同行赋能。 感谢各位读者的关注，欢迎大家使用BFE开源项目，并提出反馈或参与开发！ 章淼 博士 百度BFE团队技术负责人、百度代码规范委员会主席 2021年6月25日写于百度

原 理 篇 第1 章 BFE 简介·····························································································3 1.1 什么是BFE·························································································3 1.2 BFE平台介绍·····················································································4 1.2.1 为什么需要构建BFE 平台·····················································4 1.2.2 BFE平台的主要功能·····························································6 1.3 BFE开源项目介绍·············································································7 1.3.1 BFE平台的模块组成·····························································7 1.3.2 BFE开源项目中的内容··························································9 第2 章 网络前端接入技术简介·····································································10 2.1 什么是网络前端接入········································································10 2.2 网络前端接入面临的挑战································································11 2.3 百度的网络前端接入方案································································13 2.4 网络前端接入技术的发展趋势························································15 第3 章 网络负载均衡技术简介·····································································24 3.1 负载均衡的概念···············································································24 3.2 网络负载均衡功能的实现································································25 深入理解BFE ·VI· 3.2.1 机制说明···············································································25 3.2.2 两种方式对比········································································27 3.3 四层负载均衡和七层负载均衡························································28 设 计 篇 第4 章 BFE 的设计思想················································································33 4.1 BFE转发引擎重构的缘起································································33 4.2 BFE为什么要基于Go 语言·····························································34 4.3 BFE转发引擎的主要设计思想························································37 4.4 BFE和相关开源项目的对比····························································38 第5 章 BFE 的转发模型················································································42 5.1 转发模型概述···················································································42 5.1.1 基本概念···············································································42 5.1.2 转发过程···············································································43 5.1.3 对多租户实现机制的讨论····················································46 5.2 BFE的路由转发机制·······································································47 5.3 条件表达式·······················································································49 5.3.1 设计思想···············································································49 5.3.2 基本概念···············································································50 5.3.3 语法介绍···············································································51 5.3.4 条件原语匹配的内容····························································52 5.3.5 条件原语名称的规范····························································54 5.4 内网流量调度机制············································································55 5.4.1 内网流量调度背景介绍························································55 5.4.2 内网流量调度工作机制························································60 目 录 ·VII· 5.4.3 内网转发的其他机制····························································65 第6 章 与转发相关的关键机制·····································································68 6.1 健康检查机制···················································································68 6.1.1 健康检查的原理····································································69 6.1.2 主动健康检查和被动健康检查············································69 6.1.3 分布式健康检查和集中式健康检查····································73 6.1.4 BFE的健康检查···································································77 6.2 超时设置···························································································77 6.2.1 BFE和客户端间通信的超时················································78 6.2.2 BFE和后端实例间通信的超时············································81 6.3 BFE信息透传···················································································84 6.3.1 客户端IP地址的透传··························································84 6.3.2 其他信息的透传····································································85 6.4 限流机制···························································································89 6.4.1 限流的概念···········································································89 6.4.2 限流的配置···········································································91 6.4.3 分布式限流···········································································91 6.4.4 入口限流和目的限流····························································94 6.4.5 限流和内网流量调度····························································95 第7 章 运维相关机制····················································································97 7.1 监控机制···························································································97 7.1.1 日志监控及其问题································································98 7.1.2 BFE的内部状态输出···························································99 7.1.3 统计状态和日志的配合使用··············································101 深入理解BFE ·VIII· 7.2 Web Monitor基础库·······································································101 7.2.1 Web Monitor概述·······························································102 7.2.2 状态变量维护······································································102 7.2.3 延迟统计变量维护······························································104 7.2.4 建立专用的Web服务器·····················································106 7.2.5 注册回调函数······································································107 7.3 日志机制·························································································108 7.3.1 日志类型·············································································108 7.3.2 日志打印的注意事项··························································109 7.3.3 BFE的访问日志·································································110 7.4 配置管理························································································.111 7.4.1 BFE配置文件的分布·························································.111 7.4.2 常规配置和动态配置··························································112 7.4.3 动态配置的实现机制··························································113 第8 章 HTTPS 的优化················································································116 8.1 HTTPS优化背景及必要性·····························································116 8.2 HTTPS优化的挑战········································································117 8.3 HTTPS中的优化机制·····································································118 8.4 BFE中HTTPS 相关增强机制························································120 操 作 篇 第9 章 BFE 服务的安装部署······································································129 9.1 软件安装包下载安装······································································129 9.2 源代码编译方式安装······································································132 9.3 Docker方式安装·············································································133 目 录 ·IX· 9.4 BFE命令行参数·············································································134 9.5 查看BFE 服务的运行状态·····························································135 第10 章 BFE 服务的基础配置····································································136 10.1 场景说明·······················································································136 10.2 修改基础配置文件········································································137 10.3 转发的配置···················································································138 10.3.1 转发配置流程····································································138 10.3.2 具体案例···········································································139 10.3.3 服务访问验证····································································146 10.3.4 配置的重新加载································································146 第11 章 配置负载均衡算法及会话保持·····················································147 11.1 子集群间的负载均衡····································································147 11.2 子集群级别的会话保持································································149 11.2.1 配置实例···········································································149 11.2.2 参数的具体含义································································150 11.3 实例间的负载均衡········································································151 11.3.1 加权轮询配置示例····························································151 11.3.2 最小连接数的配置示例····················································153 11.4 实例级别的会话保持····································································154 第12 章 配置HTTPS 和更多协议·····························································155 12.1 设置HTTPS 基础配置··································································155 12.1.1 配置HTTPS 端口·····························································156 12.1.2 配置加密套件····································································156 12.1.3 配置服务端证书································································156 深入理解BFE ·X· 12.1.4 配置TLS 规则···································································157 12.2 配置TLS 会话重用·······································································158 12.2.1 配置会话缓存····································································159 12.2.2 配置会话票证····································································160 12.3 配置TLS 双向认证·······································································161 12.4 对不同安全等级的区分································································163 12.5 支持更多协议···············································································164 12.5.1 HTTP/2配置·····································································165 12.5.2 SPDY 配置········································································166 12.5.3 WebSocket配置·································································167 12.5.4 连接后端服务的协议························································169 第13 章 其他常用配置················································································171 13.1 配置重写·······················································································171 13.1.1 开启重写···········································································171 13.1.2 模块配置···········································································171 13.1.3 重写动作详细描述····························································173 13.2 配置重定向···················································································177 13.2.1 开启重定向········································································177 13.2.2 模块配置···········································································177 13.2.3 重定向动作详细描述························································178 13.3 配置限流功能···············································································180 13.3.1 开启限流模块····································································180 13.3.2 模块配置···········································································181 13.3.3 限制特定维度的流量························································182 13.3.4 设置限流门限····································································183 13.3.5 设置限流动作····································································184 目 录 ·XI· 实 现 篇 第14 章 其他常用配置················································································187 14.1 BFE的代码组织···········································································187 14.2 BFE的进程模型···········································································190 14.2.1 协程的分类·······································································190 14.2.2 并发模型···········································································191 14.2.3 并发能力···········································································192 14.2.4 异常恢复机制····································································192 14.3 请求处理流程···············································································193 14.3.1 连接的建立·······································································193 14.3.2 连接的处理·······································································194 14.3.3 请求的处理·······································································196 14.3.4 请求的结束·······································································198 14.3.5 连接的结束·······································································199 14.4 请求路由实现···············································································199 14.4.1 关键数据结构····································································200 14.4.2 目的租户路由····································································203 14.4.3 目的集群路由····································································204 14.5 负载均衡实现···············································································206 14.5.1 全局负载均衡····································································207 14.5.2 分布式负载均衡································································210 第15 章 模块插件机制················································································214 15.1 BFE的回调点设置·······································································214 15.2 BFE内置的扩展模块···································································216 15.3 模块框架的实现机制····································································217 深入理解BFE ·XII· 15.3.1 模块基础类型····································································217 15.3.2 连接/请求处理及回调函数的调用···································223 15.4 如何开发BFE扩展模块·······························································224 15.4.1 配置加载···········································································225 15.4.2 回调函数的编写和注册····················································226 15.4.3 模块状态的展示································································228 第16 章 核心协议实现················································································231 16.1 HTTP的实现················································································231 16.1.1 HTTP 代码的组织·····························································231 16.1.2 从用户读取HTTP 请求····················································233 16.1.3 向后端转发请求并获取响应············································236 16.1.4 向用户回复HTTP 响应····················································239 16.2 HTTP2的实现··············································································240 16.2.1 HTTP2 代码的组织···························································240 16.2.2 HTTP2 连接处理模块·······················································241 16.2.3 HTTP2 连接相关协程及关系···········································243 第17 章 BFE 的多进程GC 机制································································249 17.1 模型设计·······················································································249 17.1.1 多进程轮转机制································································250 17.1.2 子进程状态定义································································252 17.2 相关参数的确定············································································253 17.2.1 切换时间参数的选择························································253 17.2.2 子进程数量的计算····························································255 17.2.3 内存消耗的计算································································256