Converging QEMU and TCMU for Container Storage Huamin Chen Red Hat - - PowerPoint PPT Presentation

▶

Aug 29, 2023 2.57k likes •2.75k views

Converging QEMU and TCMU for Container Storage Huamin Chen Red Hat @root_fs https://github.com/rootfs https://huaminchen.wordpress.com/ Agenda Background Collaboration with Andy Grover (TCMU) and Fem Zheng (QEMU) Scope

SLIDE 1

Converging QEMU and TCMU for Container Storage

Huamin Chen Red Hat

@root_fs https://github.com/rootfs https://huaminchen.wordpress.com/

SLIDE 2

Agenda

Background

○ Collaboration with Andy Grover (TCMU) and Fem Zheng (QEMU) ○ Scope ■ Focused on on-premise Network Storage ○ Container Storage ■ Docker and Kubernetes ○ QEMU Block Drivers

Why Converging?
Related Works
Solutions and Status

SLIDE 3

Docker Volume

docker run -v /mnt:/data centos df
Container volumes are bind mount of those that mounted on the host

/data

Container Host OS Backend Storage (NFS, iSCSI, Gluster, Ceph ….) /mnt

SLIDE 4

Docker Volume Driver Model

Container Backend Storage (NFS, iSCSI, Gluster, Ceph ….) /data Docker Daemon Docker Volume Driver Bind mount /mnt Provisioning Mkfs mount

SLIDE 5

Kubernetes Volume Plugin Model

Container Backend Storage (NFS, iSCSI, Gluster, Ceph ….) /data /mnt Bind mount kubelet Provisioning Mkfs Fsck Mount

NFS iSCSI RBD Gluster EBS GCE PD Cinder Azure

SLIDE 6

Feel the Pain?

Last mile problem

○ Bringing easy storage access to clients is hard ○ No universal control or data plane exists

Storage vendors need to develop, maintain, and support all orchestration

frameworks

○ Kubernetes, Mesos, Docker, OpenStack, vSphere...

Orchestrators carry long lists of drivers

○ Most drivers do the same thing inside the orchestrators ○ Extensive integration tests ○ Framework level changes are often visible to and impact on all drivers

SLIDE 7

Why Converging with QEMU?

Easy data sharing.

○ There are plenty of qcow2 and vmdk images around. Enabling Containers directly to access these images eliminates data migration complexities.

Deduplicate Driver Development

○ QEMU already supports many storage drivers that Docker and Kubernetes try to support

Feature Enablement

○ Provisioning ○ Snapshot/restore ○ Resize ○ Encryption ○ Compression ○ Throttling ○ Replication

Performance

○ Some QEMU block drivers perform better than direct mount. ■ Gluster: no more FUSE bypassing overhead

SLIDE 8

QEMU Block Drivers

/dev/sdb /dev/sdc

Guest OS Host OS KVM QEMU

NFS iSCSI

Gluster RBD

…….

Backend Storage (NFS, iSCSI, Gluster, Ceph ….)

SLIDE 9

Related Works

Network Block Device (nbd): lack of drivers and features
Ploop (OpenVZ): Kernel-only model; also lack features

SLIDE 10

TCM(U)

TCM (aka LIO) is modularized SCSI target layer separating backend storage and

fabrics

TCMU handles userspace backend

SLIDE 11

Solution 1: TCMU Centric

Container Host OS Tcmu loopback

QEMU handler

NFS iSCSI

Gluster RBD

…….

Backend Storage (NFS, iSCSI, Gluster, Ceph ….) /dev/sdb /dev/sdc

TCMU Runner

TCMU Core TCMU User

SLIDE 12

Solution 2: QEMU Centric

Container Host OS Tcmu loopback

QEMU

NFS iSCSI

Gluster RBD

…….

Backend Storage (NFS, iSCSI, Gluster, Ceph ….) /dev/sdb /dev/sdc TCMU Core TCMU User libtcmu

SLIDE 13

QEMU-TCMU Integration Status

Solution 1

○ Decouple QEMU block layer from QEMU monolithic binary ■ Generate a qemu block shared library ○ Implement a QEMU block device handler for tcmu-runner ○ Create TCMU loopback device via targetcli ○ Bind mount to Docker Container

SLIDE 14

Future Work

A new standalone qemu-tcmu command
Support it in frameworks like Kubernetes, Mesos and Docker
Support it in fabrics like iSCSI, FC, NVMe-OF