Setting up a Ceph Block Device

Creating a volume/image

rbd pool init test-pool
rbd create --size 1024 test-pool/image0
rbd ls test-pool

Mounting

find /lib/modules/ | grep rbd
modprobe rbd
rbd list test-pool
rbd device map test-pool/image0

old doc

Creating a volume/image

Create a volume/image,

rbd ls
rbd create datarepo --size 1T
rbd create datarepo2 --size 3T
rbd --image datarepo info
rbd --image datarepo2 info

parted /dev/rbd/rbd/datarepo print

Map an image to a device,

lsmod | grep rbd
modprobe rbd

tail -F /var/log/messages &

sudo rbd map datarepo

ls -lhF /dev/rbd*
ls -lhF /dev/rbd/rbd/*

Note. the RHEL/CentOS provided rbd module is missing some pieces. You might have to disable those,

0x38 fast-diff
0x28 object-map
0x20 deep-flatten

rbd feature disable datarepo fast-diff fast-diff object-map deep-flatten

To remove it,

rbd rm datarepo2

Mounting the volume

Get the key that will be used by the clients to receive the block device,

cat /etc/ceph/ceph.client.admin.keyring

e.g.,

[client.admin]
        key = KEY_HERE

On the clients receiving the block device (may be the ceph nodes themselfs eventhough unfortunately not recommended), load the block device kernel module and allow the monitors' addresses,

lsmod | grep rbd
modprobe rbd

ping -c1 192.168.0.1
ping -c1 192.168.0.2
ping -c1 192.168.0.3

echo "192.168.0.1,192.168.0.2,192.168.0.3 name=admin,secret=KEY_HERE rbd datarepo" > /sys/bus/rbd/add

ls -lhF /dev/rbd*
ls -lhF /dev/rbd/rbd/*

Resizing a block device

Grow/Shrink the disk (careful with the filesystem inside if you’re shrinking),

rbd resize datarepo --size 2T

and check,

sudo parted /dev/rbd/rbd/datarepo print