Apache Doris集群安装部署

装系统的sre运维

5 月前

Apache Doris 是一个基于 MPP 架构的高性能、实时的分析型数据库，以极速易用的特点被人们所熟知，仅需亚秒级响应时间即可返回海量数据下的查询结果，不仅可以支持高并发的点查询场景，也能支持高吞吐的复杂分析场景。基于此，Apache Doris 能够较好的满足报表分析、即席查询、统一数仓构建、数据湖联邦查询加速等使用场景，用户可以在此之上构建用户行为分析、AB 实验平台、日志检索分析、用户画像分析、订单分析等应用。
Doris整体架构如下图所示，Doris 架构非常简单，只有两类进程，这两类进程都是可以横向扩展的，单集群可以支持到数百台机器，数十 PB 的存储容量。并且这两类进程通过一致性协议来保证服务的高可用和数据的高可靠。这种高度集成的架构设计极大的降低了一款分布式系统的运维成本。
- Frontend（FE），主要负责用户请求的接入、查询解析规划、元数据的管理、节点管理相关工作。
- Backend（BE），主要负责数据存储、查询计划的执行。
Doris默认端口

实例名称	端口名称	默认端口	通讯方向	说明
BE	be_port	9060	FE --> BE	BE 上 thrift server 的端口，用于接收来自 FE 的请求
BE	webserver_port	8040	BE <–> BE	BE 上的 http server 的端口
BE	heartbeat_service_port	9050	FE --> BE	BE 上心跳服务端口（thrift），用于接收来自 FE 的心跳
BE	brpc_port	8060	FE <–> BE, BE <–> BE	BE 上的 brpc 端口，用于 BE 之间通讯
FE	http_port	8030	FE <–> FE，用户 <–> FE	FE 上的 http server 端口
FE	rpc_port	9020	BE --> FE, FE <–> FE	FE 上的 thrift server 端口，每个fe的配置需要保持一致
FE	query_port	9030	用户 <–> FE	FE 上的 mysql server 端口
FE	arrow_flight_sql_port	9040	用户 <–> FE	FE 上的 Arrow Flight SQL server 端口
FE	edit_log_port	9010	FE <–> FE	FE 上的 bdbje 之间通信用的端口
FE	broker_ipc_port	8000	FE --> Broker, BE --> Broker	Broker 上的 thrift server，用于接收请求

开发测试环境为手动部署，生产环境下K8S部署；
当前未使用hadoop，包括Spark，都不放hadoop，看情况吧，先轻量点；
部署Doris时，用的都是root用户，未使用admin；
开发&测试时我们部署了3BE，3FE；
TODO：当部署多个 FE 节点时，在多个 FE 之上部署负载均衡层来实现 Doris 的高可用。目前负载均衡有两种方式：ProxySQL 及 Nginx。主要步骤可以完全按照官网给的安装手册进行，写的很详细，完全参照即可： https://doris.apache.org/zh-CN/docs/dev/admin-manual/cluster-management/load-balancing

[root@middleware-2 src]# wget https://apache-doris-releases.oss-accelerate.aliyuncs.com/apache-doris-2.0.8-bin-x64-noavx2.tar.gz
  --2024-05-09 14:19:03--  https://apache-doris-releases.oss-accelerate.aliyuncs.com/apache-doris-2.0.8-bin-x64-noavx2.tar.gz
  正在解析主机 apache-doris-releases.oss-accelerate.aliyuncs.com (apache-doris-releases.oss-accelerate.aliyuncs.com)... 47.108.43.33
  正在连接 apache-doris-releases.oss-accelerate.aliyuncs.com (apache-doris-releases.oss-accelerate.aliyuncs.com)|47.108.43.33|:443... 已连接。
  已发出 HTTP 请求，正在等待回应... 200 OK
  长度：2684900152 (2.5G) [application/gzip]
  正在保存至: “apache-doris-2.0.8-bin-x64-noavx2.tar.gz”

  100%[==================================================================================================================================================>] 2,684,900,152 74.9MB/s 用时 33s

  2024-05-09 14:19:39 (78.2 MB/s) - 已保存 “apache-doris-2.0.8-bin-x64-noavx2.tar.gz” [2684900152/2684900152])

  [root@middleware-2 src]# ls
  apache-doris-2.0.8-bin-x64-noavx2.tar.gz  jdk-8u411-linux-x64.tar.gz  kafka-eagle-bin-3.0.1.tar.gz  redis-7.0.11
  apache-zookeeper-3.7.2-bin.tar.gz         kafka_2.12-2.8.2.tgz        nacos-server-2.0.2.tar.gz     redis-7.0.11.tar.gz
  
  [root@middleware-2 src]# tar -zxvf apache-doris-2.0.8-bin-x64-noavx2.tar.gz

  [root@middleware-2 src]# mv apache-doris-2.0.8-bin-x64-noavx2 /data/middleware/apache-doris-2.0.8

  #数据存储目录
  cd /data/middleware/apache-doris-2.0.8/
  mkdir data

  [root@middleware-2 apache-doris-2.0.8]# cd data

  [root@middleware-2 data]# mkdir metastorage
  [root@middleware-2 data]# mkdir datastorage

  [root@middleware-2 data]# ls
  datastorage  metastorage
  [root@middleware-2 data]# pwd
  /data/middleware/apache-doris-2.0.8/data

  # fe.conf
  priority_networks = 192.168.0.0/24

Doris & MySQL

mysql-5.7.43-linux-glibc2.12-x86_64.tar.gz 解压后不用安装，只用它来连接doris，进行安装时的初始操作。

配置Doris文件 be&fe

# fe.conf
  [root@middleware-2 conf]# pwd
  /data/middleware/apache-doris-2.0.8/fe/conf
  [root@middleware-2 conf]# ls
  fe.conf  ldap.conf  log4j2-spring.xml  ssl
  [root@middleware-2 conf]# vi fe.conf
  # 添加配置 实际本机IP地址/掩码长度
  priority_networks = 192.168.2.111/24
  meta_dir=/data/middleware/apache-doris-2.0.8/data/metastorage
  ##############################################
  # be.conf
  cd /data/middleware/apache-doris-2.0.8/be
  # 添加配置
  priority_networks = 192.168.2.111/24
  storage_root_path = /data/middleware/apache-doris-2.0.8/data/datastorage

  #storage_root_path配置存储目录，可以用;来指定多个目录，每个目录后可以跟逗号，指定大小默认GB storage_root_path=/opt/module/doris_storage1,10;/opt/module/doris_storage2

Doris 采用 MySQL 协议进行通信，用户可通过 MySQL client 或者 MySQL JDBC连接到 Doris 集群。选择 MySQL client 版本时建议采用5.1 之后的版本，因为 5.1 之前不能支持长度超过 16 个字符的用户名。 Doris 内置 root 和 admin 用户，密码默认都为空。启动完 Doris 程序之后，可以通过 root 或 admin 用户连接到 Doris 集群。

[root@middlware-1 middleware]# cd mysql-5.7.43-linux-glibc2.12-x86_64/
  [root@middlware-1 mysql-5.7.43-linux-glibc2.12-x86_64]# ls
  bin  docs  include  lib  LICENSE  man  README  share  support-files
  [root@middlware-1 mysql-5.7.43-linux-glibc2.12-x86_64]# cd bin
  [root@middlware-1 bin]# ls
  innochecksum    myisampack         mysqlcheck                  mysqld-debug   mysql_embedded    mysql_secure_installation  mysql_tzinfo_to_sql  resolveip
  lz4_decompress  my_print_defaults  mysql_client_test_embedded  mysqld_multi   mysqlimport       mysqlshow                  mysql_upgrade        resolve_stack_dump
  myisamchk       mysql              mysql_config                mysqld_safe    mysql_install_db  mysqlslap                  mysqlxtest           zlib_decompress
  myisam_ftdump   mysqladmin         mysql_config_editor         mysqldump      mysql_plugin      mysql_ssl_rsa_setup        perror
  myisamlog       mysqlbinlog        mysqld                      mysqldumpslow  mysqlpump         mysqltest_embedded         replace
  [root@middlware-1 bin]# ./mysql -h 192.168.2.156 -P 9030 -uroot
  Welcome to the MySQL monitor.  Commands end with ; or \g.
  Your MySQL connection id is 1
  Server version: 5.7.99 Doris version doris-2.0.8-d083188cdd

  Copyright (c) 2000, 2023, Oracle and/or its affiliates.

  Oracle is a registered trademark of Oracle Corporation and/or its
  affiliates. Other names may be trademarks of their respective
  owners.

  Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

  mysql> SET PASSWORD FOR 'root' = PASSWORD('1qaz!QAZ');
  Query OK, 0 rows affected (0.02 sec)

  mysql>

添加111、184两个FE

 #密码登录
  [root@middlware-1 bin]# ./mysql -h 192.168.2.156 -P 9030 -u root -p

  # 添加那两个FE
  mysql> ALTER SYSTEM ADD FOLLOWER "192.168.2.184:9010";
  Query OK, 0 rows affected (0.03 sec)

  mysql> ALTER SYSTEM ADD FOLLOWER "192.168.2.111:9010";
  Query OK, 0 rows affected (0.01 sec)

查看FE

mysql> show proc '/frontends';
  +-----------------------------------------+---------------+-------------+----------+-----------+---------+----------+----------+-----------+------+-------+-------------------+---------------------+----------+--------+------------------------+------------------+
  | Name                                    | Host          | EditLogPort | HttpPort | QueryPort | RpcPort | Role     | IsMaster | ClusterId | Join | Alive | ReplayedJournalId | LastHeartbeat       | IsHelper | ErrMsg | Version                | CurrentConnected |
  +-----------------------------------------+---------------+-------------+----------+-----------+---------+----------+----------+-----------+------+-------+-------------------+---------------------+----------+--------+------------------------+------------------+
  | fe_92aa1928_cf39_4c96_9727_db3b20925f5d | 192.168.2.111 | 9010        | 8030     | 9030      | 9020    | FOLLOWER | false    | 150231587 | true | true  | 1371              | 2024-05-09 16:56:42 | true     |        | doris-2.0.8-d083188cdd | No               |
  | fe_95994568_37ff_441b_9701_60d9d00bc748 | 192.168.2.156 | 9010        | 8030     | 9030      | 9020    | FOLLOWER | true     | 150231587 | true | true  | 1372              | 2024-05-09 16:56:42 | true     |        | doris-2.0.8-d083188cdd | Yes              |
  | fe_3c4542f0_4c33_4edd_8981_0d95eabe5ae5 | 192.168.2.184 | 9010        | 8030     | 9030      | 9020    | FOLLOWER | false    | 150231587 | true | true  | 1371              | 2024-05-09 16:56:42 | true     |        | doris-2.0.8-d083188cdd | No               |
  +-----------------------------------------+---------------+-------------+----------+-----------+---------+----------+----------+-----------+------+-------+-------------------+---------------------+----------+--------+------------------------+------------------+
  3 rows in set (0.04 sec)

添加BE

 mysql> ALTER SYSTEM ADD BACKEND "192.168.2.111:9050";
  Query OK, 0 rows affected (0.02 sec)

  mysql> ALTER SYSTEM ADD BACKEND "192.168.2.184:9050";
  Query OK, 0 rows affected (0.00 sec)

  mysql> ALTER SYSTEM ADD BACKEND "192.168.2.156:9050";
  Query OK, 0 rows affected (0.01 sec)

查看BE

  mysql> show proc '/backends'
  -> ;
  +-----------+---------------+---------------+--------+----------+----------+---------------+---------------+-------+----------------------+-----------+------------------+--------------------+---------------+---------------+---------+----------------+--------------------+--------------------------+--------------------------------------------------------------+---------+---------------------------------------------------------------------------------------------------------------+-------------------------+----------+
  | BackendId | Host          | HeartbeatPort | BePort | HttpPort | BrpcPort | LastStartTime | LastHeartbeat | Alive | SystemDecommissioned | TabletNum | DataUsedCapacity | TrashUsedCapcacity | AvailCapacity | TotalCapacity | UsedPct | MaxDiskUsedPct | RemoteUsedCapacity | Tag                      | ErrMsg                                                       | Version | Status                                                                                                        | HeartbeatFailureCounter | NodeRole |
  +-----------+---------------+---------------+--------+----------+----------+---------------+---------------+-------+----------------------+-----------+------------------+--------------------+---------------+---------------+---------+----------------+--------------------+--------------------------+--------------------------------------------------------------+---------+---------------------------------------------------------------------------------------------------------------+-------------------------+----------+
  | 12387     | 192.168.2.111 | 9050          | -1     | -1       | -1       | NULL          | NULL          | false | false                | 0         | 0.000            | 0.000              | 1.000 B       | 0.000         | 0.00 %  | 0.00 %         | 0.000              | {"location" : "default"} | java.net.ConnectException: 拒绝连接 (Connection refused)     |         | {"lastSuccessReportTabletsTime":"N/A","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} | 10                      |          |
  | 12443     | 192.168.2.156 | 9050          | -1     | -1       | -1       | NULL          | NULL          | false | false                | 0         | 0.000            | 0.000              | 1.000 B       | 0.000         | 0.00 %  | 0.00 %         | 0.000              | {"location" : "default"} | java.net.ConnectException: 拒绝连接 (Connection refused)     |         | {"lastSuccessReportTabletsTime":"N/A","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} | 7                       |          |
  | 12424     | 192.168.2.184 | 9050          | -1     | -1       | -1       | NULL          | NULL          | false | false                | 0         | 0.000            | 0.000              | 1.000 B       | 0.000         | 0.00 %  | 0.00 %         | 0.000              | {"location" : "default"} | java.net.ConnectException: 拒绝连接 (Connection refused)     |         | {"lastSuccessReportTabletsTime":"N/A","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} | 8                       |          |
  +-----------+---------------+---------------+--------+----------+----------+---------------+---------------+-------+----------------------+-----------+------------------+--------------------+---------------+---------------+---------+----------------+--------------------+--------------------------+--------------------------------------------------------------+---------+---------------------------------------------------------------------------------------------------------------+-------------------------+----------+
  3 rows in set (0.01 sec)

BROKER部署

可选,非必须部署，启动BROKER，我们暂时未部署BROKER

  # 以下内容只是记录，非真实部署过程
  # 三台集群都要启动 ./start_broker.sh--daemon

  # 添加broker节点
  ALTER SYSTEM ADD BROKER broker_name "master:8000","node1:8000","node2:8000";

  # 查询Broke状态
  SHOW PROC "/brokers";

Broker 以插件的形式，独立于 Doris 部署。如果需要从第三方存储系统导入数据，需要部署相应的 Broker ，默认提供了读取 HDFS 、百度云 BOS 及 Amazon S3 的 fs_broker 。 fs_broker 是无状态的，要每一个 FE 和 BE 节点都部署一个 Broker 。

BE&FE启动

 # FE
  # 156作为主节点，以上的配置中156主节点先启动
  [root@middlware-1 bin]# ./start_fe.sh --daemon
  
  #其他节点 使用 --helper 参数指向 Master FE。
  ./start_fe.sh --helper 192.168.2.156:9010 --daemon

  # BE
  [root@middleware-2 bin]# ./start_be.sh --daemon

  # 查看FE是否正常 curl http://fe_host:fe_http_port/api/bootstrap
  [root@middleware-2 bin]# curl http://192.168.2.111:8030/api/bootstrap
  {"msg":"success","code":0,"data":{"replayedJournalId":0,"queryPort":0,"rpcPort":0,"version":""},"count":0}[root@middleware-2 bin]#

  # 查询BE是否正常 curl http://be_host:webserver_port/api/health
  c[root@middleware-2 bin]# curl http://192.168.2.111:8040/api/health
  {"status": "OK","msg": "To Be Added"}[root@middleware-2 bin]#

BE到此都起来了，再查一下看（和添加BE的查询相比）

 mysql> show proc '/backends';
  +-----------+---------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------+------------------+--------------------+---------------+---------------+---------+----------------+--------------------+--------------------------+--------+------------------------+-------------------------------------------------------------------------------------------------------------------------------+-------------------------+----------+
  | BackendId | Host          | HeartbeatPort | BePort | HttpPort | BrpcPort | LastStartTime       | LastHeartbeat       | Alive | SystemDecommissioned | TabletNum | DataUsedCapacity | TrashUsedCapcacity | AvailCapacity | TotalCapacity | UsedPct | MaxDiskUsedPct | RemoteUsedCapacity | Tag                      | ErrMsg | Version                | Status                                                                                                                        | HeartbeatFailureCounter | NodeRole |
  +-----------+---------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------+------------------+--------------------+---------------+---------------+---------+----------------+--------------------+--------------------------+--------+------------------------+-------------------------------------------------------------------------------------------------------------------------------+-------------------------+----------+
  | 12387     | 192.168.2.111 | 9050          | 9060   | 8040     | 8060     | 2024-05-09 17:05:48 | 2024-05-09 17:21:24 | true  | false                | 14        | 0.000            | 0.000              | 88.046 GB     | 99.752 GB     | 11.74 % | 11.74 %        | 0.000              | {"location" : "default"} |        | doris-2.0.8-d083188cdd | {"lastSuccessReportTabletsTime":"2024-05-09 17:20:36","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} | 0                       | mix      |
  | 12443     | 192.168.2.156 | 9050          | 9060   | 8040     | 8060     | 2024-05-09 17:05:16 | 2024-05-09 17:21:24 | true  | false                | 14        | 0.000            | 0.000              | 85.591 GB     | 99.752 GB     | 14.20 % | 14.20 %        | 0.000              | {"location" : "default"} |        | doris-2.0.8-d083188cdd | {"lastSuccessReportTabletsTime":"2024-05-09 17:21:09","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} | 0                       | mix      |
  | 12424     | 192.168.2.184 | 9050          | 9060   | 8040     | 8060     | 2024-05-09 17:05:36 | 2024-05-09 17:21:24 | true  | false                | 14        | 0.000            | 0.000              | 88.933 GB     | 99.752 GB     | 10.85 % | 10.85 %        | 0.000              | {"location" : "default"} |        | doris-2.0.8-d083188cdd | {"lastSuccessReportTabletsTime":"2024-05-09 17:21:21","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false} | 0                       | mix      |
  +-----------+---------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------+------------------+--------------------+---------------+---------------+---------+----------------+--------------------+--------------------------+--------+------------------------+-------------------------------------------------------------------------------------------------------------------------------+-------------------------+----------+
  3 rows in set (0.00 sec)

Doris 内置的 Web UI http://192.168.2.156（111、184都可以）:8030/login 使用root及上面改过的密码（1qaz!QAZ）打开 Doris 内置的 Web 控制台。扩容缩容(TODO)

Doris可以很方便的扩容和缩容FE、BE、Broker实例。
FE 节点的扩容和缩容过程，不影响当前系统运行。
增加FE节点，FE分为Leader,Follower和Observer三种角色。默认一个集群只能有一个Leader,可以有多个Follower和Observer.其中Leader和Follower组成一个Paxos选择组，如果Leader宕机，则剩下的Follower会成为Leader，保证HA。Observer是负责同步Leader数据的不参与选举。如果只部署一个FE，则FE默认就是Leader.
第一个启动的FE自动成为Leader。

BE 扩容和缩容 Doris集群启停脚本

  待实现