官网下载
[root@server1 ~]# curl "http://ppa.moosefs.com/MooseFS-3-el7.repo" > /etc/yum.repos.d/MooseFS.repo 下载mfs仓库 [root@server1 ~]# cd /etc/yum.repos.d/ [root@server1 yum.repos.d]# cat MooseFS.repo [MooseFS] name=MooseFS $releasever - $basearch baseurl=http://ppa.moosefs.com/moosefs-3/yum/el7 gpgcheck=0 不需要校验 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-MooseFS enabled=1 [root@server1 yum.repos.d]# yum install moosefs-master moosefs-cgi moosefs-cgiserv moosefs-cli -y 安装 [root@server1 mfs]# vim /etc/hosts 添加解析，将server1设置为master 172.25.50.1 server1 mfsmaster [root@server1 mfs]# systemctl start moosefs-master 启动 [root@server1 mfs]# systemctl start moosefs-cgiserv.service 启动
访问：172.25.50.1:9425
在server2上安装moosefs-chunkserver
[root@server1 yum.repos.d]# scp MooseFS.repo server2:/etc/yum.repos.d/ [root@server2 ~]# yum install moosefs-chunkserver -y server2上安装chunkserver [root@server2 mfs]# vim /etc/hosts 添加解析 172.25.50.1 server1 mfsmaster [root@server2 ~]# cd /etc/mfs/ 指定存储点 [root@server2 mfs]# mkdir /mnt/chunk1 创建存储目录 [root@server2 mfs]# vim mfshdd.cfg
注意：在生产环境中此目录一定挂载的是一个分区
[root@server2 mfs]# chown mfs.mfs /mnt/chunk1 chunkserver是以mfs用户身份运行的，所以需要给存储目录mfs身份权限 server3进行同样操作 [root@server2 mfs]# systemctl start moosefs-chunkserver 启动
在server3上安装 moosefs-chunkserver
[root@server1 yum.repos.d]# scp MooseFS.repo server3:/etc/yum.repos.d/ [root@server3~]# yum install moosefs-chunkserver -y server2上安装chunkserver [root@server3 mfs]# vim /etc/hosts 添加解析 172.25.50.1 server1 mfsmaster [root@server3 ~]# cd /etc/mfs/ [root@server3 mfs]# mkdir /mnt/chunk2 [root@server3 mfs]# vim mfshdd.cfg
[root@server3 mfs]# chown mfs.mfs /mnt/chunk2 [root@server3 mfs]# systemctl start moosefs-chunkserver 启动
下载mfs客户端：
为了不多开虚拟机将宿主机作为客户端，客户端系统是redhat8
[root@foundation50 ~]# curl "http://ppa.moosefs.com/MooseFS-3-el8.repo" > /etc/yum.repos.d/MooseFS.repo 下载mfs仓库 [root@foundation50 ~]# yum install moosefs-client -y 安装客户端 [root@foundation50 ~]# cd /mnt/ [root@foundation50 mnt]# mkdir /mfs [root@foundation50 mfs]# cd /etc/mfs/ [root@foundation50 mfs]# ls mfsmount.cfg mfsmount.cfg.sample [root@foundation50 mfs]# vim mfsmount.cfg 编辑客户端配置文件
[root@foundation50 mfs]# vim /etc/hosts 添加解析 [root@foundation50 mfs]# 172.25.50.1 server1 mfsmaster [root@foundation50 mfs]# mfsmount 自动读取刚才我们配置的mfsmount.cfg文件 [root@foundation50 mfs]# df mfs#mfsmaster:9421 35622912 2855936 32766976 9% /mnt/mfs 客户端连接的是9421端口 9420端口连接的是chunkserver，9419连接的是原数据服务器 [root@foundation50 ~]# cd /mnt/mfs/ [root@foundation50 mfs]# mkdir dr1 创建目录 [root@foundation50 mfs]# mkdir dr2 创建目录 [root@foundation50 mfs]# mfsgetgoal dr1 查看副本数 dr1: 2 副本数为2 [root@foundation50 mfs]# mfsgetgoal dr2 查看副本数 dr2: 2 副本数为2 [root@foundation50 mfs]# mfssetgoal -r 1 dr1/ 设置dr1副本数为1 [root@foundation50 mfs]# cd dr1/ [root@foundation50 dr1]# cp /etc/passwd . 将passwd文件拷贝到dr1里 [root@foundation50 dr1]# cd .. [root@foundation50 mfs]# cp /etc/fstab dr2/ 将fstab文件拷贝到dr2里 [root@foundation50 dr1]# mfsfileinfo passwd 查看具体文件信息 passwd: chunk 0: 0000000000000001_00000001 / (id:1 ver:1) 一个副本，存储到server3上 copy 1: 172.25.50.3:9422 (status:VALID) [root@foundation50 mfs]# cd dr2/ [root@foundation50 dr2]# mfsfileinfo fstab 查看具体文件信息 fstab: chunk 0: 0000000000000002_00000001 / (id:2 ver:1) copy 1: 172.25.50.2:9422 (status:VALID) 有两个副本 copy 2: 172.25.50.3:9422 (status:VALID) [root@server3 ~]# systemctl stop moosefs-chunkserver 当我们停掉server3的chunkserver [root@foundation50 dr2]# mfsfileinfo fstab 发现两个副本的还可以用 fstab: chunk 0: 0000000000000002_00000001 / (id:2 ver:1) copy 1: 172.25.50.2:9422 (status:VALID) [root@foundation50 dr2]# cd .. [root@foundation50 mfs]# cd dr1/ [root@foundation50 dr1]# mfsfileinfo passwd 发现一个副本就不可以使用了 passwd: chunk 0: 0000000000000001_00000001 / (id:1 ver:1) no valid copies !!! 注：只要集群中master不出问题，chunkserver停止开启后数据不会丢失
扩容chunkserver
再开启一台虚拟机server4，作为chunkserver
[root@server2 yum.repos.d]# scp MooseFS.repo server4:/etc/yum.repos.d/ 将server2配置过的MooseFS软件仓库给server4拷贝一份 [root@server4 ~]# yum install moosefs-chunkserver -y 安装chunkserver [root@server4 ~]# vim /etc/hosts 添加解析 172.25.50.1 server1 mfsmaster [root@server4 ~]# mkdir /mnt/chunk3 创建存储目录 [root@server4 ~]# chown mfs.mfs /mnt/chunk3/ 给mfs用户写入权限 [root@server4 ~]# vim /etc/mfs/mfshdd.cfg 最后一行，添加存储路经 /mnt/chunk3/ [root@server4 ~]# systemctl start moosefs-chunkserver 启动
[root@foundation50 isos]# cd /mnt/mfs/ [root@foundation50 mfs]# cd dr1/ [root@foundation50 dr1]# dd if=/dev/zero of=bigfile bs=1M count=200 压测，每块chunk默认64M，200M需要四个 [root@foundation50 dr1]# mfsfileinfo bigfile 只有一个副本，分为四个块，读取数据需要四个快一起读取，其中一个坏了数据就不完整了 bigfile: chunk 0: 0000000000000003_00000001 / (id:3 ver:1) copy 1: 172.25.50.3:9422 (status:VALID) chunk 1: 0000000000000004_00000001 / (id:4 ver:1) copy 1: 172.25.50.4:9422 (status:VALID) chunk 2: 0000000000000005_00000001 / (id:5 ver:1) copy 1: 172.25.50.2:9422 (status:VALID) chunk 3: 0000000000000006_00000001 / (id:6 ver:1) copy 1: 172.25.50.4:9422 (status:VALID)
删除文件如何撤销
[root@foundation50 dr1]# rm -f passwd [root@foundation50 dr1]# mfsgettrashtime . 查看回收时间 .: 86400 [root@foundation50 ~]# cd /mnt/ [root@foundation50 mnt]# mkdir mfsmeta 创建原数据目录 [root@foundation50 mnt]# mfsmount -m /mnt/mfsmeta/ -m表示挂载原数据。挂载原数据到/mnt/mfsmeta/ 目录 [root@foundation50 mnt]# cd /mnt/mfsmeta/ [root@foundation50 mfsmeta]# ls sustained trash [root@foundation50 mfsmeta]# cd trash/ 删除的文件都在这个目录里面 [root@foundation50 trash]# find -name *passwd* 查找刚才删除的passwd ./004/00000004|dr1|passwd passwd 所在目录 [root@foundation50 trash]# cd 004/ [root@foundation50 004]# ls '00000004|dr1|passwd' undel [root@foundation50 004]# mv 00000004|dr1|passwd undel/ 将passwd文件移动到undel [root@foundation50 004]# cd /mnt/mfs [root@foundation50 mfs]# cd dr1/ [root@foundation50 dr1]# ls bigfile passwd 删除的passwd文件已经恢复2、mfs读写原理
【1】客户端向元数据服务器发出读请求。
【2】元数据服务器把所需数据存放的位置（Chunk Server 的 IP 地址和 Chunk 编号） 告知客户端。
【3】客户端向已知的 Chunk Server 请求发送数据。
【4】Chunk Server 向客户端发送数据。
【1】客户端向元数据服务器发送写入请求。
【2】元数据服务器与 Chunk Server 进行交互（只有当所需的分块 Chunks 存在的时候才进行这个交互），但元数据服务器只在某些服务器创建新的分块Chunks，创建成功后由Servers 告知元数据服务器操作成功
【3】元数据服务器告知客户端， 可以在哪个 Chunk Server 的哪些 Chunks 写入数据。
【4】客户端向指定的 Chunk Server 写入数据
【5】该 Chunk Server 与其他 Chunk Server 进行数据同步
【6】 数据同步成功
【7】同步成功后 Chunk Server 告知客户端数据写入成功。
【8】客户端告知元数据服务器本次写入完毕。
功后 Chunk Server 告知客户端数据写入成功。
【8】客户端告知元数据服务器本次写入完毕。
添加标签
[root@server2 ~]# vim /etc/mfs/mfschunkserver.cfg
[root@server2 ~]# systemctl reload moosefs-chunkserver.service 重启服务 [root@server3 ~]# vim /etc/mfs/mfschunkserver.cfg
[root@server3 ~]# systemctl reload moosefs-chunkserver.service 重启服务 [root@server4 ~]# vim /etc/mfs/mfschunkserver.cfg
[root@server4 ~]# systemctl reload moosefs-chunkserver.service 重启服务
创建存储类
[root@foundation50 ~]# cd /mnt/mfs [root@foundation50 mfs]# cd dr1/ [root@foundation50 dr1]# mfsscadmin create A,B classAB 创建存储类，是在A和B上创建的 storage class make classAB: ok [root@foundation50 dr1]# mfsscadmin list 查看 1 2 3 4 5 6 7 8 9 classAB 此为自己创建的存储类 [root@foundation50 dr1]# mfsscadmin create B,C classBC 创建存储类，在B和C上 storage class make classBC: ok [root@foundation50 dr1]# mfsscadmin list 1 2 3 4 5 6 7 8 9 classAB classBC 此为自己创建的存储类
数据迁移：数据存储A、B上，迁移到B、C上
[root@foundation50 ~]# cd /mnt/mfs [root@foundation50 mfs]# ls dr1 dr2 [root@foundation50 mfs]# cd dr2/ [root@foundation50 dr2]# mfsfileinfo fstab fstab: chunk 0: 0000000000000002_00000001 / (id:2 ver:1) copy 1: 172.25.50.2:9422 (status:VALID) 数据在A、B上 copy 2: 172.25.50.3:9422 (status:VALID) [root@foundation50 dr2]# mfssetsclass classBC fstab 把classBC存储类设置到fstab fstab: storage class: 'classBC' [root@foundation50 dr2]# mfsfileinfo fstab fstab: chunk 0: 0000000000000002_00000001 / (id:2 ver:1) copy 1: 172.25.50.3:9422 (status:VALID) 此时数据迁移到BC上 copy 2: 172.25.50.4:9422 (status:VALID)
多区域数据迁移：
[root@server2 ~]# vim /etc/mfs/mfschunkserver.cfg
[root@server2 ~]# systemctl reload moosefs-chunkserver.service 重载服务 [root@server3 ~]# vim /etc/mfs/mfschunkserver.cfg
[root@server3 ~]# systemctl reload moosefs-chunkserver 重载服务 [root@server4 ~]# vim /etc/mfs/mfschunkserver.cfg
[root@server4 ~]# systemctl reload moosefs-chunkserver 重启 [root@foundation50 dr2]# mfsscadmin create AS,BS class2S 创建存储类，在AS、BS上 storage class make class2S: ok [root@foundation50 dr2]# mfsscadmin list 1 2 3 4 5 6 7 8 9 classAB classBC class2S 创建存储类成功 [root@foundation50 dr2]# mfsfileinfo fstab fstab: chunk 0: 0000000000000002_00000001 / (id:2 ver:1) 查看当前数据在server3和server4上 copy 1: 172.25.50.3:9422 (status:VALID) copy 2: 172.25.50.4:9422 (status:VALID) [root@foundation50 dr2]# mfssetsclass class2S fstab 把class2S存储类设置到fstab fstab: storage class: 'class2S' [root@foundation50 dr2]# mfsfileinfo fstab 查看文件详细信息 fstab: chunk 0: 0000000000000002_00000001 / (id:2 ver:1) copy 1: 172.25.50.2:9422 (status:VALID) 数据由原来的server3、server4迁移到了server2和server3上 copy 2: 172.25.50.3:9422 (status:VALID) 将热数据迁移到温数据 [root@foundation50 dr2]# mfsscadmin create BH,CH class2H 创建存储类，在BH,CH上 storage class make class2H: ok [root@foundation50 dr2]# mfssetsclass class2H fstab fstab: storage class: 'class2H' [root@foundation50 dr2]# mfsfileinfo fstab fstab: chunk 0: 0000000000000002_00000001 / (id:2 ver:1) copy 1: 172.25.50.3:9422 (status:VALID) 由server2、server3，迁移到server3、server4 copy 2: 172.25.50.4:9422 (status:VALID)
[root@server3 ~]# vim /etc/mfs/mfschunkserver.cfg
[root@server3 ~]# systemctl reload moosefs-chunkserver.service 重启服务 [root@foundation50 dr2]# mfsscadmin create -C 2AS -K AS,BS -A BH,CH -d 7 class7d storage class make class7d: ok [root@foundation50 dr2]# mfssetsclass class7d fstab fstab: storage class: 'class7d' [root@foundation50 dr2]# mfsfileinfo fstab fstab: chunk 0: 0000000000000002_00000001 / (id:2 ver:1) copy 1: 172.25.50.2:9422 (status:VALID) copy 2: 172.25.50.3:9422 (status:VALID)5 、mfs集群热备份/高可用
实验前准备：server1、server2做高可用，先将erver2上数据移走
[root@foundation50 dr2]# mfssetsclass classBC fstab 将classBC存储类设置到fstab文件上 fstab: storage class: 'classBC' [root@foundation50 dr2]# mfsfileinfo fstab 查看文件信息 fstab: chunk 0: 0000000000000002_00000001 / (id:2 ver:1) copy 1: 172.25.50.3:9422 (status:VALID) 数据由server2、server3迁移到server3、server4上 copy 2: 172.25.50.4:9422 (status:VALID) [root@server2 ~]# systemctl reload moosefs-chunkserver.service 重载chunkserver [root@server2 ~]# systemctl stop moosefs-chunkserver.service 停掉server2上的chunkserver [root@foundation50 ~]# umount /mnt/mfs 手工卸载
如果内核崩溃，没有正常停止，该怎么办？
[root@server1 mfs]# kill -9 3857 强制删除master进程，没有正常停止 [root@server1 mfs]# cd /var/lib/mfs/ 进入原数据目录 [root@server1 mfs]# ll -rw-r----- 1 mfs mfs 120 May 28 00:08 metadata.crc -rw-r----- 1 mfs mfs 3969 May 28 00:08 metadata.mfs.back 一般正常停止back会被回收，此时back没有正常回收 -rw-r----- 1 mfs mfs 3969 May 27 23:53 metadata.mfs.back.1 一般正常停止back会被回收，此时back没有正常回收 -rw-r--r-- 1 mfs mfs 8 Aug 10 2021 metadata.mfs.empty -rw-r----- 1 mfs mfs 4984552 May 28 00:08 stats.mfs [root@server1 mfs]# systemctl start moosefs-master 此时再启动，无法启动报错 Job for moosefs-master.service failed because the control process exited with error code. See "systemctl status moosefs-master.service" and "journalctl -xe" for details. [root@server1 mfs]# mfsmaster -a 用此命令就可以恢复 [root@server1 mfs]# mfsmaster stop 停止master [root@server1 mfs]# ll -rw-r----- 1 mfs mfs 3969 May 28 00:33 metadata.mfs 此时back就被正常回收
如何防止内核崩溃，造成无法启动的情况？
[root@server1 mfs]# systemctl status moosefs-master.service 查看启动脚本路经 ● moosefs-master.service - MooseFS Master server Loaded: loaded (/usr/lib/systemd/system/moosefs-master.service; disabled; vendor preset: disabled) Active: active (running) since Sat 2022-05-28 00:37:24 CST; 2s ago
[root@server1 mfs]# systemctl daemon-reload 重载 [root@server1 mfs]# kill -9 3965 此时强制删除master进程 [root@server1 mfs]# systemctl start moosefs-master 此时再次启动就可以正常启动了
在server2上安装moosefs-master
[root@server2 ~]# systemctl stop moosefs-chunkserver.service 停止chunkserver [root@server2 ~]# yum install moosefs-master -y 安装master5、1要做双机热备份必须要让server1和server2上master数据同步
用iscs设备共享同步数据
内存有限我们不单独开虚拟机，在server3上添加一块磁盘
[root@server3 ~]# fdisk -l 查看磁盘
Disk /dev/vdb: 10.7 GB, 10737418240 bytes, 20971520 sectors 多出了一块我们刚才添加的磁盘
[root@server3 ~]# yum install targetcli -y 安装
[root@server3 ~]# systemctl start target.service 启动
[root@server3 ~]# targetcli
Warning: Could not load preferences file /root/.targetcli/prefs.bin.
targetcli shell version 2.1.fb46
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.
/> ls
o- / ..................................................................... [...]
o- backstores .......................................................... [...]
| o- block .............................................. [Storage Objects: 0]
| o- fileio ............................................. [Storage Objects: 0]
| o- pscsi .............................................. [Storage Objects: 0]
| o- ramdisk ............................................ [Storage Objects: 0]
o- iscsi ........................................................ [Targets: 0]
o- loopback ..................................................... [Targets: 0]
/> cd /backstores/
/backstores> cd block
/backstores/block> ls
o- block .................................................. [Storage Objects: 0]
/backstores/block> create mydisk /dev/vdb 创加存储名
Created block storage object mydisk using /dev/vdb.
/backstores/block> ls
o- block .................................................. [Storage Objects: 1]
o- mydisk ........................ [/dev/vdb (10.0GiB) write-thru deactivated]
o- alua ................................................... [ALUA Groups: 1]
o- default_tg_pt_gp ....................... [ALUA state: Active/optimized]
/backstores> cd ..
/> cd /iscsi
/iscsi> create iqn.2022-06.org.westos:target 创建外部共享
Created target iqn.2022-06.org.westos:target.
Created TPG 1.
Global pref auto_add_default_portal=true
Created default portal listening on all IPs (0.0.0.0), port 3260.
/iscsi> cd iqn.2022-06.org.westos:target/tpg1/
/iscsi/iqn.20...s:target/tpg1> ls
o- tpg1 ................................................. [no-gen-acls, no-auth]
o- acls ............................................................ [ACLs: 0]
o- luns ............................................................ [LUNs: 0]
o- portals ...................................................... [Portals: 1]
o- 0.0.0.0:3260 ....................................................... [OK]
/iscsi/iqn.20...s:target/tpg1> cd acls
/iscsi/iqn.20...get/tpg1/acls> ls
o- acls .............................................................. [ACLs: 0]
/iscsi/iqn.20...get/tpg1/acls> create iqn.2022-06.org.westos:client 创建访问控制
Created Node ACL for iqn.2022-06.org.westos:client
[root@server1 ~]# yum install -y iscsi-* 安装iscsi客户端
[root@server1 ~]# vim /etc/iscsi/initiatorname.iscsi 将创建的访问控制添加到此文件里
InitiatorName=iqn.2022-06.org.westos:client
[root@server1 ~]# iscsiadm -m discovery -t st -p 172.25.50.3
172.25.50.3:3260,1 iqn.2022-06.org.westos:target
[root@server1 ~]# iscsiadm -m discovery -t st -p 172.25.50.3
172.25.50.3:3260,1 iqn.2022-06.org.westos:target
[root@server1 ~]# iscsiadm -m node -l 登陆
Logging in to [iface: default, target: iqn.2022-06.org.westos:target, portal: 172.25.50.3,3260] (multiple)
Login to [iface: default, target: iqn.2022-06.org.westos:target, portal: 172.25.50.3,3260] successful.
[root@server1 ~]# fdisk -l 查看设备
Disk /dev/sda: 10.7 GB, 10737418240 bytes, 20971520 sectors 多出了一个设备，这个设备就是在srever3上输出的vdb10g的设备
[root@server1 ~]# fdisk /dev/sda 创建分区
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x360a447e.
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-20971519, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-20971519, default 20971519):
Using default value 20971519
Partition 1 of type Linux and of size 10 GiB is set
Command (m for help): p
Disk /dev/sda: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x360a447e
Device Boot Start End Blocks Id System
/dev/sda1 2048 20971519 10484736 83 Linux
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
[root@server1 ~]# mkfs.xfs /dev/sda1 格式化
[root@server1 mfs]# systemctl stop moosefs-master.service 停掉server1上master服务
[root@server1 ~]# mount /dev/sda1 /mnt/ 将设备挂载到mnt上
[root@server1 ~]# cd /var/lib/mfs/ master原数据目录
changelog.11.mfs changelog.16.mfs changelog.22.mfs metadata.mfs.back.1
changelog.12.mfs changelog.17.mfs changelog.8.mfs metadata.mfs.empty
changelog.13.mfs changelog.18.mfs changelog.9.mfs stats.mfs
changelog.14.mfs changelog.19.mfs metadata.crc
changelog.15.mfs changelog.21.mfs metadata.mfs.back
[root@server1 mfs]# cp -p * /mnt/ 需要把srevre1上的master原数据拷贝到/mnt里，来共享数据，达到server1、server2数据一致的目的 -p表示保留权限拷贝
[root@server1 mnt]# ll -d . 查看mnt目录的权限
drwxr-xr-x 2 root root 4096 May 29 02:11 .
[root@server1 mnt]# ll -d /var/lib/mfs/
drwxr-xr-x 2 mfs mfs 4096 May 29 02:07 /var/lib/mfs/ 由于mfs目录权限是需要mfs写入的，mnt写不进去
[root@server1 mnt]# chown mfs.mfs /mnt/
[root@server1 ~]# umount /mnt/ 卸载
如何把/dev/sda1 这个设备共享到server2上
[root@server2 ~]# yum install iscsi-* -y 安装相同的客户端
[root@server2 ~]# vim /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.2022-06.org.westos:client
[root@server2 ~]# iscsiadm -m discovery -t st -p 172.25.50.3
172.25.50.3:3260,1 iqn.2022-06.org.westos:target
[root@server2 ~]# iscsiadm -m node -l 登陆
[root@server2 ~]# fdisk -l 查看设备
Disk /dev/sda: 10.7 GB, 10737418240 bytes, 20971520 sectors sda已经在serevr2上查看了
Device Boot Start End Blocks Id System
/dev/sda1 2048 20971519 10484736 83 Linux sda1已经在server2上查看了
[root@server2 ~]# mount /dev/sda1 /var/lib/mfs 此时我们将sda1里的数据（也就是serevr1上master原数据）挂载到server2的master原数据目录，这样server1、server2 上的数据就保持一致了
[root@server2 ~]# vim /usr/lib/systemd/system/moosefs-master.service
[root@server2 ~]# systemctl daemon-reload [root@server2 ~]# systemctl start moosefs-master 启动，能启动说明数据同步成功5.2 安装高可用套件
[root@server1 mnt]# cd /etc/yum.repos.d/ 配置yum源 [root@server1 yum.repos.d]# vim dvd.repo [dvd] name=rhel7.6 baseurl=http://172.25.254.50/rhel7.6 gpgcheck=0 [HighAvailability] name=rhel7.6 HighAvailability baseurl=http://172.25.254.50/rhel7.6/addons/HighAvailability gpgcheck=0 [root@foundation50 isos]# cd /mnt/pub/docs/mfs/ [root@foundation50 mfs]# evince Pacemaker-1.1-Clusters_from_Scratch-en-US.pdf & 查看高可用文档 [root@server1 yum.repos.d]# yum install -y pacemaker pcs psmisc policycoreutils-python 安装集群套件 [root@server1 yum.repos.d]# scp dvd.repo server2:/etc/yum.repos.d/ 拷贝软件仓库到server2上 [root@server1 ~]# systemctl start pcsd.service 启动集群管理工具 [root@server1 ~]# systemctl enable pcsd.service 设置开机自启 [root@server2 ~]# systemctl enable --now pcsd.service 设置开机自启 为了方便在server1、serevr2上做免密 [root@server1 ~]# ssh-keygen [root@server1 ~]# ssh-copy-id server2 [root@server1 ~]# echo westos | passwd --stdin hacluster 给此用户设置密码 [root@server1 ~]# ssh server2 'echo westos | passwd --stdin hacluster' 在server2上同样给此用户设置密码 [root@server1 ~]# pcs cluster auth server1 server2 验证这两个节点 Username: hacluster Password: server1: Authorized server2: Authorized [root@server1 ~]# pcs cluster setup --name mycluster server1 server2 用验证成功的两个节点创建集群 [root@server1 ~]# pcs cluster start --all 启动集群所有节点 [root@server1 ~]# pcs cluster enable --all 设置集群开机自启 [root@server1 ~]# pcs status 查看集群状态 Cluster name: mycluster WARNINGS: No stonith devices and stonith-enabled is not false 有警告，需要把这个禁掉 [root@server1 ~]# pcs property set stonith-enabled=false 禁掉5.3 将mfs迁移到高可用集群平台上
[root@server1 ~]# pcs resource create vip ocf:heartbeat:IPaddr2 ip=172.25.50.100 cidr_netmask=32 op monitor interval=30s 创建vip资源 cidr_netmask=32表示子网掩码 op monitor interval=30s 表示每隔30s对资源进行监控 [root@server1 ~]# pcs status 查看集群状态
测试：
[root@server1 ~]# pcs node standby 让server1节点处于休息状态 [root@server1 ~]# pcs status
[root@server1 ~]# pcs node unstandby 让server1节点重新加入集群 将文件系统资源加入集群 [root@server1 ~]# pcs resource describe ocf:heartbeat:Filesystem 查看管理文件系统参数，那些参数是必须要添加的 [root@server1 ~]# pcs resource create mfsdate ocf:heartbeat:Filesystem device=/dev/sda1 directory=/var/lib/mfs fstype=xfs op monitor interval=60s fstype=xfs 表示文件系统格式 directory=/vae/lib/mfs 表示挂载点 op monitor interval=60s 表示每隔60s监控一次 [root@server1 ~]# pcs status 查看集群状态，mfsdate资源已经迁移到集群server1上
[root@server1 ~]# pcs resource create mfsmaster systemd:moosefs-master op monitor interval=60s 创建服务 [root@server1 ~]# pcs status
[root@server1 ~]# pcs resource group add hagroup vip mfsdate mfsmaster 添加一个hagroup组，然后把 vip mfsdate mfsmaster这三个都添加进来 [root@server1 ~]# pcs status ，全部都调整在server2上
[root@server2 ~]# pcs node standby 此时让server2休眠 [root@server2 ~]# pcs status
[root@server2 ~]# pcs node unstandby 当server2重新开启
[root@server1 ~]# vim /etc/hosts 172.25.50.100 mfsmaster 添加解析 [root@server2 ~]# vim /etc/hosts 172.25.50.100 mfsmaster 添加解析 [root@server3 ~]# vim /etc/hosts 172.25.50.100 mfsmaster 添加解析 [root@server4 ~]# vim /etc/hosts 172.25.50.100 mfsmaster 添加解析 [root@foundation50 dr2]# vim /etc/hosts 172.25.50.100 mfsmaster 添加解析
当内核崩溃，网卡出现故障就需要重启系统，但是集群没有这个重启功能
fence可以管理电源接口
[root@foundation50 isos]# fence_virtd -c fence配置
Module search path [/usr/lib64/fence-virt]:
Available backends:
libvirt 0.3
Available listeners:
tcp 0.1
serial 0.4
multicast 1.2
Listener modules are responsible for accepting requests
from fencing clients.
Listener module [multicast]: 回车
The multicast listener module is designed for use environments
where the guests and hosts may communicate over a network using
multicast.
The multicast address is the address that a client will use to
send fencing requests to fence_virtd.
Multicast IP Address [225.0.0.12]: 回车
Using ipv4 as family.
Multicast IP Port [1229]: 回车
Setting a preferred interface causes fence_virtd to listen only
on that interface. Normally, it listens on all interfaces.
In environments where the virtual machines are using the host
machine as a gateway, this *must* be set (typically to virbr0).
Set to 'none' for no interface.
Interface [virbr0]: 回车
The key file is the shared key information which is used to
authenticate fencing requests. The contents of this file must
be distributed to each physical host and virtual machine within
a cluster.
Key File [/etc/cluster/fence_xvm.key]: 回车
Backend modules are responsible for routing requests to
the appropriate hypervisor or management layer.
Backend module [libvirt]: 回车
The libvirt backend module is designed for single desktops or
servers. Do not use in environments where virtual machines
may be migrated between hosts.
Libvirt URI [qemu:///system]: 回车
Configuration complete.
=== Begin Configuration ===
backends {
libvirt {
uri = "qemu:///system";
}
}
listeners {
multicast {
port = "1229";
family = "ipv4";
interface = "virbr0";
address = "225.0.0.12";
key_file = "/etc/cluster/fence_xvm.key";
}
}
fence_virtd {
module_path = "/usr/lib64/fence-virt";
backend = "libvirt";
listener = "multicast";
}
=== End Configuration ===
Replace /etc/fence_virt.conf with the above [y/N]? y
[root@foundation50 isos]# mkdir /etc/cluster 创建
[root@foundation50 cluster]# dd if=/dev/urandom of=fence_xvm.key bs=128 count=1 表示取一个128字节的随机数当密钥用， urandom表示随机数设备
[root@foundation50 cluster]# systemctl start fence_virtd.service 启动fence_virtd
[root@foundation50 cluster]# netstat -anulp | grep :1229
udp 0 0 0.0.0.0:1229 0.0.0.0:* 30547/fence_virtd 监听udp的1229端口
server1、server2如何连接fence设备
[root@server1 ~]# stonith_admin -I 查看server1上是否有连接的fence设备
No devices found
[root@server1 ~]# yum install fence-virt -y 安装fence客户端
[root@server1 ~]# stonith_admin -I
fence_xvm
fence_virt
2 devices found
[root@server2 ~]# yum install fence-virt -y 安装fence客户端
[root@server2 ~]# stonith_admin -I 查看是否fence设备
fence_xvm 虚拟机fence
fence_virt
2 devices found
[root@server1 ~]# stonith_admin -M -a fence_xvm
需要校验，服务端已经设置好了/etc/cluster/fence_xvm.key 这个key来校验，不是谁都可以给电源接口发请求的
[root@server1 ~]# mkdir /etc/cluster 所以都需要创建key的目录
[root@server2 ~]# mkdir /etc/cluster 所以都需要创建key的目录
[root@foundation50 cluster]# scp fence_xvm.key server1:/etc/cluster 客户端需要读这个key，所以将服务端的key拷贝给客户端，路经保持一样
[root@foundation50 cluster]# scp fence_xvm.key server2:/etc/cluster
集群如何调用fence
[root@server1 cluster]# pcs stonith describe fence_xvm 查看描述
[root@server1 cluster]# pcs stonith create vmfence fence_xvm pcmk_host_map="server1:vm1;server2:vm2" op monitor interval=60s
pcmk_host_map="server1:vm1;server2:vm2" 表示映射，主机名在前，虚拟机名在后
[root@server1 cluster]# pcs resource cleanup vmfence 清理资源，再次添加 Cleaned up vmfence on server2 Cleaned up vmfence on server1 Waiting for 2 replies from the CRMd.. OK [root@server1 cluster]# pcs status
[root@server1 cluster]# pcs property set stonith-enabled=true
[root@server1 cluster]# pcs property set stonith-enabled=true 把之前禁掉的stonith激活
fence测试：
此时当我们对虚拟机server1使用命令---->echo c> /proc/sysrq-trigger (模拟内核崩溃)
的时候，它会进行重启并且添加在可用节点中。
