1. 概述

最新更新：2022年8月12日 21:16:00

本文使用kvm虚拟机node[19-24]在centos7.9操作系统，部署lustre文件系统2.12

集群规划如下：

节点名称	磁盘数目	服务
node19/node20	共享/dev/vbd	mgt/mgs
node21/node22	共享/dev/vd{b,c}	mdt/mds
node23/node24	共享/dev/vd{b,c,d,e,f,g}	oss/ost
node25		client

2. 虚拟机准备

说明：本篇使用kvm的克隆功能，实现快速部署。

也可以创建多个节点以后，使用批处理命令完成操作

2.1 创建一个lustre内核操作系统模板

virsh start node19

启动node19节点，创建yum源：vim /etc/yum.repo.d/whamcloud.repo

[lustre_latest_el7_client]
name=whamcloud_lustre_client
baseurl=https://downloads.whamcloud.com/public/lustre/lustre-2.12.8/el7.9.2009/client/
enabled=1
priority=5
gpgcheck=0

[lustre_latest_el7_server]
name=whamcloud_lustre_server
baseurl=https://downloads.whamcloud.com/public/lustre/lustre-2.12.8/el7.9.2009/server/
enabled=1
priority=5
gpgcheck=0

[e2fsprogs_latest]
name=whamcloud_e2fsprogs
baseurl=https://downloads.whamcloud.com/public/e2fsprogs/latest/el7/
enabled=1
priority=5
gpgcheck=0

执行安装yum插件：

yum -y install yum-plugin-priorities

执行安装lustre相关软件：

yum -y install e2fsprogs e2fsprogs-libs libss libcom_err kernel-*lustre kmod-lustre kmod-lustre-osd-ldiskfs kmod-lustre-tests lustre-osd-ldiskfs lustre-osd-ldiskfs-mount lustre-modules lustre-ldiskfs-dkms lustre

编辑内核配置文件：vim /etc/modprobe.d/lnet.conf

optinos lnet networks=tcp0(eth0)

编辑启动脚本vim /etc/rc.local

modprobe lnet
modprobe lustre
modprobe ldiskfs

chmod +x /etc/rc.local

关闭node19节点：virsh destroy node19

2.2 使用模板克隆生成新节点

 for node in {20..24}
 do
 	virt-clone -o node19 -n node${node} -f /var/lib/libvirt/images/node${node}.qcow2 -m 00:00:00:80:00:${node}
 done 

2.3 为虚拟机添加共享磁盘

本篇添加共享磁盘的方式为：virsh edit xxx。

也可以通过修改xyz.xml后，使用virsh define /etc/libvirt/qemu/xyz.xml导入修改

先创建虚拟磁盘：

# 1. 创建lustre-mgt-000.img，挂载给node19和node20,盘符/dev/vdb
qemu-img create -f raw lustre-mgt-000.img 10G
# 2. 创建lustre-mdt-000.img和lustre-mdt-001.img，挂载给node21和node22,盘符/dev/vd{b,c}
qemu-img create -f raw lustre-mdt-000.img 10G
qemu-img create -f raw lustre-mdt-001.img 10G
# 3. 创建lustre-ost-000.img - lustre-ost-006.img，挂载给node23,node24，盘符/dev/vd{b,c,d,e,f,g}
for i in {000..006}
do
qemu-img create -f raw lustre-ost-${i}.img 10G
done

为node19和node20添加共享磁盘

    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/lustre-mgt-000.img'/>
      <target dev='vdb' bus='virtio'/>
      <shareable/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x07' function='0x0'/>
    </disk>

为node21和node22添加共享磁盘

    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/lustre-mdt-000.img'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x07' function='0x0'/>
          <shareable/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/lustre-mdt-001.img'/>
      <target dev='vdc' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x07' function='0x0'/>
          <shareable/>
    </disk>

为node23和node24添加共享磁盘

    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/lustre-ost-000.img'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x07' function='0x0'/>
          <shareable/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/lustre-ost-001.img'/>
      <target dev='vdc' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x07' function='0x0'/>
          <shareable/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/lustre-ost-002.img'/>
      <target dev='vdd' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x07' function='0x0'/>
          <shareable/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/lustre-ost-003.img'/>
      <target dev='vde' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x07' function='0x0'/>
          <shareable/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/lustre-ost-004.img'/>
      <target dev='vdf' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x07' function='0x0'/>
          <shareable/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/lustre-ost-005.img'/>
      <target dev='vdg' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x07' function='0x0'/>
          <shareable/>
    </disk>

3.创建文件系统

3.1 mgs服务创建

在node19上执行：

# 格式化设备
mkfs.lustre --fsname=mylustre --reformat --mgs [email protected] [email protected] /dev/vdb
# 创建挂载点
mkdir -p /mnt/mgt000
# 挂载设备
mount.lustre /dev/vdb	/mnt/mgt000

node20上面不用创建，直接挂载即可

3.2 mds服务创建

在node21上执行：

# 格式化设备
mkfs.lustre --fsname=mylustre  --reformat --mdt --index=0  [email protected]:[email protected] [email protected] [email protected]  /dev/vdb
mkfs.lustre --fsname=mylustre  --reformat --mdt --index=1  [email protected]:[email protected] [email protected] [email protected]  /dev/vdc
# 创建挂载点
mkdir -p /mnt/mdt{000,001}
# 挂载设备
mount.lustre /dev/vdb /mnt/mdt000

在node22上执行：

# 创建挂载点
mkdir -p /mnt/mdt{000,001}
# 挂载设备
mount.lustre /dev/vdc /mnt/mdt001

3.3 oss服务创建

在node23上执行：

#!/bin/bash
#/dev/vdb
mkfs.lustre --fsname=mylustre --reformat --ost --index=0 [email protected]:[email protected] [email protected] [email protected] /dev/vdb
#/dev/vdc
mkfs.lustre --fsname=mylustre --reformat --ost --index=1 [email protected]:[email protected] [email protected] [email protected] /dev/vdc
#/dev/vdd
mkfs.lustre --fsname=mylustre --reformat --ost --index=2 [email protected]:[email protected] [email protected] [email protected] /dev/vdd
# mkdir
mkdir -p /mnt/{ost000,ost001,ost002,ost003,ost004,ost005}
# mount
mount.lustre /dev/vdb /mnt/ost000
mount.lustre /dev/vdc /mnt/ost001
mount.lustre /dev/vdd /mnt/ost002

在node24上执行：

#!/bin/bash
#/dev/vde
mkfs.lustre --fsname=mylustre --reformat --ost --index=3 [email protected]:[email protected] [email protected] [email protected] /dev/vde
#/dev/vdf
mkfs.lustre --fsname=mylustre --reformat --ost --index=4 [email protected]:[email protected] [email protected] [email protected] /dev/vdf
#/dev/vdg
mkfs.lustre --fsname=mylustre --reformat --ost --index=5 [email protected]:[email protected] [email protected] [email protected] /dev/vdg
# mkdir
mkdir -p /mnt/{ost000,ost001,ost002,ost003,ost004,ost005}
# mount
mount.lustre /dev/vde /mnt/ost003
mount.lustre /dev/vdf /mnt/ost004
mount.lustre /dev/vdg /mnt/ost005

4. 创建客户端虚拟机并挂载

node25执行yum -y install lustre-client

创建挂载节点：mkdir -p /home/export/online1

挂载文件系统:mount.lustre [email protected]:[email protected]:/mylustre /home/export/online1/

执行命令：lfs df -h

UUID                       bytes        Used   Available Use% Mounted on
mylustre-MDT0000_UUID        5.5G        2.0M        5.0G   1% /home/export/online1[MDT:0]
mylustre-MDT0001_UUID        5.5G        2.0M        5.0G   1% /home/export/online1[MDT:1]
mylustre-OST0000_UUID        9.2G        1.4M        8.7G   1% /home/export/online1[OST:0]
mylustre-OST0001_UUID        9.2G        1.4M        8.7G   1% /home/export/online1[OST:1]
mylustre-OST0002_UUID        9.2G        1.4M        8.7G   1% /home/export/online1[OST:2]
mylustre-OST0003_UUID        9.2G        1.4M        8.7G   1% /home/export/online1[OST:3]
mylustre-OST0004_UUID        9.2G        1.4M        8.7G   1% /home/export/online1[OST:4]
mylustre-OST0005_UUID        9.2G        1.4M        8.7G   1% /home/export/online1[OST:5]

filesystem_summary:        55.1G        8.1M       52.0G   1% /home/export/online1