创建 rbd imageceph iops 测试
iops 限制有两种方法 1 librbd 库 2 kernel 两种
测试区别及可行性
# rbd create -p volumes iotest --size 100G
# rbd info volumes/iotest
rbd image 'iotest':
size 100 GiB in 25600 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: d0a91b43d9968
block_name_prefix: rbd_data.d0a91b43d9968
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
op_features:
flags:
create_timestamp: Tue Jun 14 09:49:15 2022
access_timestamp: Tue Jun 14 09:49:15 2022
modify_timestamp: Tue Jun 14 09:49:15 2022
rbd map
# rbd feature disable volumes/iotest object-map fast-diff deep-flatten # rbd map volumes/iotest --id admin --keyring /etc/ceph/ceph.client.admin.keyringformat and test iops
# parted /dev/rbd0 mklabel gpt
信息: You may need to update /etc/fstab.
# parted /dev/rbd0 mkpart xfs 4 100%
信息: You may need to update /etc/fstab.
# mkfs -t xfs -i size=1024 /dev/rbd0p1
meta-data=/dev/rbd0p1 isize=1024 agcount=16, agsize=1638400 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=26212352, imaxpct=25
= sunit=1024 swidth=1024 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=12800, version=2
= sectsz=512 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
iops test1
# dd if=/dev/zero of=/mnt/test.img bs=1M count=4096 oflag=dsync 记录了4096+0 的读入 记录了4096+0 的写出 4294967296字节(4.3 GB)已复制，66.9778 秒，64.1 MB/秒iops setting
# umount /mnt
# rbd unmap /dev/rbd0
# rbd info volumes/iotest
rbd image 'iotest':
size 100 GiB in 25600 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: d0a91b43d9968
block_name_prefix: rbd_data.d0a91b43d9968
format: 2
features: layering, exclusive-lock
op_features:
flags:
create_timestamp: Tue Jun 14 09:49:15 2022
access_timestamp: Tue Jun 14 09:49:15 2022
modify_timestamp: Tue Jun 14 09:49:15 2022
# rbd config image set volumes/iotest rbd_qos_iops_limit 10
# rbd config image set volumes/iotest rbd_qos_bps_limit 10
# rbd config image set volumes/iotest rbd_qos_write_iops_limit 10
test iops again
iops limit not work
# rbd map volumes/iotest --id admin --keyring /etc/ceph/ceph.client.admin.keyring /dev/rbd0 # mount /dev/rbd0p1 /mnt # cd /mnt # dd if=/dev/zero of=/mnt/test.img bs=1M count=4096 oflag=dsync 记录了4096+0 的读入 记录了4096+0 的写出 4294967296字节(4.3 GB)已复制，64.9189 秒，66.2 MB/秒check for config
check image
# rbd config image list volumes/iotest | grep rbd_qos rbd_qos_bps_burst 0 config rbd_qos_bps_limit 10 image rbd_qos_iops_burst 0 config rbd_qos_iops_limit 10 image rbd_qos_read_bps_burst 0 config rbd_qos_read_bps_limit 0 config rbd_qos_read_iops_burst 0 config rbd_qos_read_iops_limit 0 config rbd_qos_schedule_tick_min 50 config rbd_qos_write_bps_burst 0 config rbd_qos_write_bps_limit 0 config rbd_qos_write_iops_burst 0 config rbd_qos_write_iops_limit 10 image
check pool
# rbd config pool ls volumes | grep rbd_qos rbd_qos_bps_burst 0 config rbd_qos_bps_limit 0 config rbd_qos_iops_burst 0 config rbd_qos_iops_limit 0 config rbd_qos_read_bps_burst 0 config rbd_qos_read_bps_limit 0 config rbd_qos_read_iops_burst 0 config rbd_qos_read_iops_limit 0 config rbd_qos_schedule_tick_min 50 config rbd_qos_write_bps_burst 0 config rbd_qos_write_bps_limit 0 config rbd_qos_write_iops_burst 0 config rbd_qos_write_iops_limit 0 configtry sysbench
yum install -y sysbenchprepare file
sysbench --test=fileio --file-total-size=50G preparerunning test
# sysbench --test=fileio --file-total-size=50G --file-test-mode=rndrw --init-rng=on --max-time=300 --max-requests=0 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 1
Initializing random number generator from timer.
Extra file open flags: 0
128 files, 400Mb each
50Gb total file size
Block size 16Kb
Number of random requests for random IO: 0
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Threads started!
Time limit exceeded, exiting...
Done.
Operations performed: 364380 Read, 242920 Write, 777263 Other = 1384563 Total
Read 5.56Gb Written 3.7067Gb Total transferred 9.2667Gb (31.63Mb/sec)
2024.33 Requests/sec executed
Test execution summary:
total time: 300.0009s
total number of events: 607300
total time taken by event execution: 18.0695
per-request statistics:
min: 0.00ms
avg: 0.03ms
max: 17.10ms
approx. 95 percentile: 0.04ms
Threads fairness:
events (avg/stddev): 607300.0000/0.00
execution time (avg/stddev): 18.0695/0.00
结论
bug ?无法实现 iops 限制
因为 dd 命令与 sysbench 都不带 librbd 库
当使用 nbd 时候，多次挂载，则可能会有 map 延时时间十分长
当使用 nbd 时候，在发生 IO 时候修改 iops 参数，会出现 ioerror 问题 并且一直持续无法恢复
