Nethence Newdoc Olddoc Lab Your IP BBDock  

Setting up a Ceph Block Device

Creating a volume/image

as ceph@admin

Create a volume/image,

rbd ls
rbd create datarepo --size 1T
rbd create datarepo2 --size 3T
rbd --image datarepo info
rbd --image datarepo2 info

sudo parted /dev/rbd/rbd/datarepo print

Map an image to a device,

lsmod | grep rbd
modprobe rbd

tail -F /var/log/messages &

sudo rbd map datarepo

ls -lhF /dev/rbd*
ls -lhF /dev/rbd/rbd/*

Note. the RHEL/CentOS provided rbd module is missing some pieces. You might have to disable those,

0x38 fast-diff
0x28 object-map
0x20 deep-flatten

rbd feature disable datarepo fast-diff fast-diff object-map deep-flatten

To remove it,

rbd rm datarepo2

Mounting the volume

as ceph@admin

Get the key that will be used by the clients to receive the block device,

sudo cat /etc/ceph/ceph.client.admin.keyring

e.g.,

[client.admin]
        key = KEY_HERE

as root@client1,2,3

On the clients receiving the block device (may be the ceph nodes themselfs eventhough unfortunately not recommended), load the block device kernel module and allow the monitors' addresses,

lsmod | grep rbd
modprobe rbd

ping -c1 192.168.0.1
ping -c1 192.168.0.2
ping -c1 192.168.0.3

echo "192.168.0.1,192.168.0.2,192.168.0.3 name=admin,secret=KEY_HERE rbd datarepo" > /sys/bus/rbd/add

ls -lhF /dev/rbd*
ls -lhF /dev/rbd/rbd/*

Resizing a block device

as ceph@admin or ceph@client1,2,3

Grow/Shrink the disk (careful with the filesystem inside if you’re shrinking),

rbd resize datarepo --size 2T

and check,

sudo parted /dev/rbd/rbd/datarepo print

Ready to go

as root@client1,2,3

Now do what ever you want with the block device e.g.,

mkfs.xfs /dev/rbd/rbd/datarepo

possibly shared,

GFS...

Home | GitHub | Donate | Contact