LIO and the TCMU Userspace Passthrough: The Best of Both Worlds Andy Grover <agrover@redhat.com> @iamagrover March 11, 2015 1 LIO AND TCMU
What is LIO? Multi-protocol in-kernel SCSI target 2 LIO AND TCMU
Multi-protocol in - kernel SCSI target Unlike other targets like IET, tgt, and SCST, LIO is entirely kernel code. 3 LIO AND TCMU
targetcli rtslib Configuration User Kernel /sys/kernel/config/target Fabrics block /dev/vg0/vol0 iscsi LIO core tcm_fc file disk.img pscsi /dev/sdc iSCSI commands Backstores 4 LIO AND TCMU
Why add userspace command handling? ● Enable wider variety of backstores without kernel code ● Clustered network storage fjlesystems like Ceph, GlusterFS, & other things that have shared libraries available ● File formats beyond .img, such as qcow2 & vmdk for more interesting features & compatibility ● SCSI devices beyond mass storage ● Enable experimentation just like FUSE did for fjlesystems 5 LIO AND TCMU
Userspace handling challenges: perf ● I/O latency ● I/O throughput ● Parallelism within a dev (OoO cmd completion) ● Parallelism across devs, we're good. 6 LIO AND TCMU
Userspace handling challenges: usability ● Confjguration as simple as existing backstores ● Userspace daemon failure ● Userspace daemon activation/restart ● Balance ultimate fmexibility with common use ● Avoid “multiple personalities” ● Reasonable resource usage 7 LIO AND TCMU
SCSI Command processing cmd handling daemon targetcli rtslib Configuration uio0 uio1 User Kernel ! W ! W tcm-user E E N /sys/kernel/config/target N Fabrics block /dev/vg0/vol0 iscsi LIO core tcm_fc file disk.img pscsi /dev/sdc iSCSI commands Backstores 8 LIO AND TCMU
User/Kernel Communication Shared Memory Region Layout (not to scale) cmd handling daemon Mailbox cmdr_off Mailbox cmdr_size cmd_head ● read (wait for cmd) ● read to get ● write (cmds done) cmd_tail configuration dev ● mmap (get SMR) Command Ring add/remove Command Ring (netlink) cmd_entry opcode iovec[] /dev/uio0 /sys/class/uio/uio0 ... uio0 Data Area Data Area in/out data ... tcm-user from LIO core 9 LIO AND TCMU
tcm-user merged in 3.18 ● Initial patch as simple as possible ● Performance tuning not done, BUT an interface fmexible enough to not block likely perf optimization strategies ● Acceptance enabled phase 2: userspace usability pieces 10 LIO AND TCMU
Performance Opportunities for Later ● Larger vmalloc()ed shared mem region ● -> Demand-allocate pages to avoid bloat ● Block size == PAGE_SIZE preferred ● Complete commands out of order ● Must handle data area fragmentation ● Fabrics copy into already-mapped pages ● Just moves the cache misses? ● Fabrics don't have per-lun device queues anyway. Just IB reimplemented poorly? ● Userspace busywait on ring cmd_head 11 LIO AND TCMU
Our user-kernel API ended up fmexible, but fraught with danger! ● Ring operations easy to mess up ● Make every handler write daemon boilerplate? ● And support Netlink? ● And maybe D-Bus??? 12 LIO AND TCMU
tcmu-runner: A standard handler daemon ● Handle the messy ring bits ● Expose a C plugin API ● Implement library routines for common handler code, e.g. mandatory SCSI commands ● Permissively licensed: Apache 2.0 ● Doubles as sample code for do-it-yourself-ers ● Needed a prototype daemon in any case 13 LIO AND TCMU
tape vmdk mmc smc glfs ! W ! targetcli W tcmu-runner E E N N rtslib Configuration User uio0 uio1 Kernel tcm-user /sys/kernel/config/target Fabrics block /dev/vg0/vol0 iscsi LIO core tcm_fc file disk.img pscsi /dev/sdc iSCSI commands Backstores 14 LIO AND TCMU
Handlers Need Confjg Info ● All confjguration should still be through standard LIO mechanisms (i.e. targetcli) ● User backstore create includes a URL-like “confjgstring” that gives handler and per-device handler-specifjc stuf ● This is published in uio sysfs, and netlink add_device message ● Also has info to allow going from uio dev back to matching LIO backstore ● Handler needs attribs, block size, etc. 15 LIO AND TCMU
Confjg tools need Info on Handlers Too! ● Users should call “backstores/foo create x y z” not “backstores/user <confjgstring>” ● targetcli needs param and help strings for xyz ● Verify backstore params are correct before creating the device ● Must be loosely coupled – no hard dependencies on targetcli or tcmu-handler ● Solution: D-Bus! 16 LIO AND TCMU
! W ! W E E N N tape vmdk mmc smc glfs DBus targetcli tcmu-runner rtslib Configuration User uio0 uio1 Kernel tcm-user /sys/kernel/config/target Fabrics block /dev/vg0/vol0 iscsi LIO core tcm_fc file disk.img pscsi /dev/sdc iSCSI commands Backstores 17 LIO AND TCMU
The QEMU Question ● QEMU has great support for many image formats and other backstores ● Can we reuse or integrate somehow? ● How? ● Build qemu handler code separately and integrate as a tcmu-runner handler? ● Extend qemu to implement TCMU directly, and possibly also enable it to confjgure LIO exports? ● Deferred for now 18 LIO AND TCMU
Getting Involved ● Give feedback! ● agrover@redhat.com, target-devel@vger.kernel.org ● Check out tcmu-runner: https://github.com/agrover/tcmu-runner and its included sample handlers. ● Use github PRs and issue tracking. ● Much help needed, esp. QEMU hackers! ● Start doing some TCMU performance benchmarking ● Start thinking of interesting device types, userspace libraries to use, or weird things to do with a SCSI command sandbox 19 LIO AND TCMU
Thanks! Thanks! Questions? Questions? 20 LIO AND TCMU
lsmcli (libstoragemgmt) Remote Local targetcli targetd tcmu-runner rtslib liblvm User Kernel ! tcm-user W ! W E E N /sys/kernel/config/target N Fabrics block /dev/vg0/vol0 iscsi LIO core tcm_fc file disk.img pscsi /dev/sdc ISCSI commands 21 LIO AND TCMU
User/Kernel Communication Shared Memory Region Layout (not to scale) tcmu-runner Mailbox cmdr_off Mailbox cmdr_size cmd_head ● read (wait for cmd) ● read to get ● write (cmds done) cmd_tail configuration ● mmap (get SMR) Command Ring Command Ring cmd_entry opcode iovec[] /dev/uio0 /sys/class/uio/uio0 ... uio0 Data Area Data Area in/out data ... tcm-user from LIO core 22 LIO AND TCMU
SCSI A set of standards for physically connecting and transferring data between computers and peripheral devices* * http://en.wikipedia.org/wiki/SCSI 23 LIO AND TCMU
SCSI target Initiator sends commands, Target handles them 24 LIO AND TCMU
Multi-protocol SCSI target SCSI commands & data can be sent over many types of physical links and protocols. e.g: ● SCSI Parallel Interface (original) ● iSCSI (over TCP/IP) ● SAS (over SATA cables) ● Fibre Channel (over FCP) ● FCoE (over Ethernet) ● SRP (over Infiniband) ● SBP-2 (over Firewire) 25 LIO AND TCMU
tgtadm Configuration libglfs ! W ! W E tgtd E N N librbd User Kernel Backed by local files or block devices iSCSI commands from initiator disk.img /dev/vg0/vol0 26 LIO AND TCMU
g g n n i m i m ! o n ! o n C o C o o o S S qcow2 vmdk vdi rbd* glfs* others librbd libglfs qemu-lio-tcmu tcmu-runner User Kernel tcm-user /sys/kernel/config/target Fabrics block /dev/vg0/vol0 iscsi LIO core tcm_fc file disk.img pscsi /dev/sdc ISCSI commands 27 LIO AND TCMU
Recommend
More recommend
