CernVM-FS Graphdriver Plugin for Docker CernVM Workshop 2018 CERN Nikola Hardi hardi.nikola@gmail.com January 30th, 2018 Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 1 / 28
Containers get heavy! Joint Blog Post Mesosphere & CERN (03/2016) Network traffic gets congested as gigabytes worth of Docker downloads are moving across the pipe [. . . ]. Companies [. . . ] such as Twitter have already experienced this phenomenon. Red Hat, “Containers for Grownups” (02/2016) 10 things to avoid in docker containers: . . . 3) Don’t create large images. Medallia (10/2015, CERN KT Screening) The problem today with Docker is that distribu- tion of software is a mess, it is a “bottleneck” in our system. Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 2 / 28
CernVM and CernVM-FS The experiment software is delivered to the appliance just in time by means of a network file system (CVMFS) specifically designed for effi- cient software distribution and installation. — Predrag Buncic et al. “CernVM - a virtual appliance for LHC applications”, ACAT08 The virtual machine image is less than 20 megabyte in size. The actual operating system is delivered on demand by the CernVM File System. — Gerardo Ganis et al. “Status and Roadmap of CernVM”, J. Phys., 2015 To serve the needs of the HEP community, several iterations were made to create a scalable, user-level filesystem that delivers software worldwide on a daily basis. — Jakob Blomer et al. “Status and Roadmap of CernVM”, Comp. in Sci. and Eng., 2017 Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 3 / 28
The solution proposed by CernVM team Improved Docker Software Docker Software KT-Funded Project pull & push “apps” file-based transfer Docker “App Store” CernVM File System Our analysis shows that pulling packages accounts for 76% of container start time, but only 6.4% of that data is read. — T. Harter et al. “Slacker: Fast Distribution with Lazy Containers”, FAST16 Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 4 / 28
Container Engines Singularity Rocket Docker Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 5 / 28
About the Docker engine Main components: Docker client Docker daemon Registry: hub.docker.com Docker versions: 2017 - Jan: Docker 1.13.0 2017 - Feb: Docker 1.13.1 2017 - Mar: Docker 17.03 2018 - Mar: Docker 18.01 Host machine Internet HTTP HTTP Docker Docker Docker registry client daemon Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 6 / 28
About the Docker images Docker image is a stack of read-only layers. The list of read-only layers is specified in the image manifest. Layer is a filesystem diff between two snapshots. Layers are content addressable and reusable. Container is a Docker image in the state of execution. Each container has a dedicated read-write layer. nginx ubuntu OwnCloud Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 7 / 28
The CernVM-FS graphdriver plugin for Docker Host machine Remote server CernVM-FS Graphdriver This is CernVM-FS server plugin a container! Minio S3 S3 HTTP Docker Docker Docker client daemon registry $ docker plugin install cvmfs/graphdriver Restart Dockerd with −− experimental − s cvmfs/graphdriver $ docker run cvmfs/thin ubuntu echo ’’Hello CVM workshop!’’ Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 8 / 28
DEMO Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 9 / 28
Results Transferred Data [MB] Startup Time [s] 40 200 Just 3 MB! Constant startup time! 100 30 50 25 20 10 10 5 0 R OwnCloud Ubuntu R OwnCloud Ubuntu Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 10 / 28
Results 2 The cluster startup time reduced from 5 min to less than 5 s. Tomorrow @ 5:30pm, Ricardo Brito Da Rocha: Magnum and HNSciCloud Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 11 / 28
Results 3 Tomorrow @ 5:30pm, Ricardo Brito Da Rocha: Magnum and HNSciCloud Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 12 / 28
The CernVM-FS graphdriver — DETAILS — Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 13 / 28
The CernVM-FS graphdriver plugin for Docker Host machine Remote server CernVM-FS This is Graphdriver CernVM-FS server plugin a container! Minio S3 S3 HTTP Docker Docker Docker registry client daemon 1 Plugins run in containers. 2 Plugins communicate with the Docker daemon over HTTP. 3 Layer upload to CernVM-FS through a portal (Minio S3). Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 14 / 28
The thin image format Regular Docker Image Thin Image CernVM-FS graphdriver Scratch layer Thin image descriptor Local read-only layer CernVM-FS provided read-only layer Additional benefits comming from CernVM-FS 1 Files that are never accessed will never be transferred. 2 CernVM-FS controlled local cache size. 3 File based deduciplation / reuse for free. Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 15 / 28
Example: the thin image descriptor $ docker2cvmfs thin library/ubuntu:latest test.cern.ch { "version": "1.0", "origin": "library/ubuntu:latest@https://registry-1.docker.io/v2", "layers": [ { "digest": "1be7f2b886e89a58e59c4e685fcc5905a26ddef3201f290b96f1eff7d778e122", "url": "cvmfs://test.cern.ch/1be7f2b886e89a58e59c4e685fcc5905a26ddef3201f290b96f1eff7d778e122" }, { "digest": "6fbc4a21b806838b63b774b338c6ad19d696a9e655f50b4e358cc4006c3baa79", "url": "cvmfs://test.cern.ch/6fbc4a21b806838b63b774b338c6ad19d696a9e655f50b4e358cc4006c3baa79" }, { "digest": "c71a6f8e13782fed125f2247931c3eb20cc0e6428a5d79edb546f1f1405f0e49", "url": "cvmfs://test.cern.ch/c71a6f8e13782fed125f2247931c3eb20cc0e6428a5d79edb546f1f1405f0e49" }, { "digest": "4be3072e5a37392e32f632bb234c0b461ff5675ab7e362afad6359fbd36884af", "url": "cvmfs://test.cern.ch/4be3072e5a37392e32f632bb234c0b461ff5675ab7e362afad6359fbd36884af" }, { "digest": "06c6d2f5970057aef3aef6da60f0fde280db1c077f0cd88ca33ec1a70a9c7b58", "url": "cvmfs://test.cern.ch/06c6d2f5970057aef3aef6da60f0fde280db1c077f0cd88ca33ec1a70a9c7b58" } ] } Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 16 / 28
The docker2cvmfs 1 Inspect an image manifest. 2 Pull all layers of an image from a Docker registry. 3 Produce a thin image descriptor. 1 Manipulating Docker images is not an easy problem. 2 Special files, permissions, ownership, hardlinks, whiteout files... 3 Everyone has their own solution. Let’s work together! Checkout following presentations 1 Today @ 3pm, Tom Downes: Automated Conversion of Docker images to CVMFS for LIGO and the Open Science Grid 2 Tomorrow @ 10am, Michael Bauer: Building Reproducible Science with Singularity Containers Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 17 / 28
The pull scenario Pull cvmfs/thin ubuntu from the Docker registry This image contains just the layer with the thin image descriptor. Request the graphdriver to create the scratch layer. Mount the required CernVM-FS repositories. Creates the union mount of the scratch layer and the read-only layers mounted directly from CernVM-FS. The list of repositories and read-only layers is available in the thin image descriptor. Regular image Thin image Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 18 / 28 read-write layer thin image layer
The push scenario Regular image: 1 The changeset is stored in the r/w branch (r/w layer). 2 Publish this changes as new r/o layer on the Docker registry. 3 Create new Docker image with updated list or r/o layers. Thin image: 1 The changeset is stored in the r/w branch (r/w layer). 2 Publish this changes as new r/o layer on CVMFS repository. 3 Create new Docker image with updated thin image descriptor. + CVMFS Docker stratum 0 registry read-write layer read-only layer thin layer Note: this functionality is currently disabled. Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 19 / 28
Project status and roadmap Not production ready but can do useful work. Ready for evaluation. We already have few early adpoters. Improve stability and make workflows even smoother. CernVM-FS portals feature in CernVM-FS 2.6. CernVM team continues the development. Nikola Hardi hardi.nikola@gmail.com CernVM-FS Graphdriver Plugin for Docker January 30th, 2018 20 / 28
Recommend
More recommend
