Docker : devops, shared registries, HPC and emerging use cases - PowerPoint PPT Presentation

Docker : devops, shared registries, HPC and emerging use cases François Moreews & Olivier Sallou

P r e s e n t a t i o n “ Docker is an open-source engine to easily create lightweight, portable , self-sufficient containers from any application. The same container that a developer builds and test on a laptop can run at scale , in production , on Vms,[...], public clouds and more.”

P r e s e n t a t i o n chroot ● Each process/command on Linux has current working directory called root directory of a process/command. Chroot changes the root directory of a command, which ends up changing the root directory for both current running process and its children. ● A process/command that is run in such a modified environment cannot access files outside the root directory . This modified environment is known as " jailed directory " .

P r e s e n t a t i o n What are Docker “containers” ? Technically: ~chroot on steroids ● a container is a set of processes (running on top of common kernel) ● isolated* from the rest of the machine (cannot see/affect/harm host or other containers) ● using namespaces to have private view of the system (network interfaces, PID tree, mountpoints...) ● and cgroups to have metered/limited/reserved resources (to mitigate “bad neighbor” effect)

P r e s e n t a t i o n What are Docker “containers” ? From a distance: looks like a VM ● I can SSH into my container ● I can have root access in it ● I can install packages in it ● I have my own eth0 interface ● I can tweak routing table, iptables rules ● I can mount filesystems ...

P r e s e n t a t i o n What are Docker “containers” ? ● boot in milliseconds ● just a few MB of intrinsic disk/memory usage ● bare metal performance is possible Lightweight, fast, disposable...virtual environments : An efficient new way to build, ship, deploy & run your apps !

Wh y i t w o r k s — s e p a r a t i o n o f c o n c e r n s • t h e O p s ( a d mi n ) w o r r i e s a b o u t • t h e D e v e l o p e r w o r r i e s w h a t ’ s “ o u t s i d e ” t h e c o n t a i n e r a b o u t w h a t ’ s “ i n s i d e ” t h e • L o g g i n g c o n t a i n e r • R e mo t e a c c e s s • H i s c o d e • M o n i t o r i n g • H i s L i b r a r i e s • N e t w o r k c o n f i g • H i s P a c k a g e • A l l c o n t a i n e r s s t a r t , s t o p , M a n a g e r c o p y , a t t a c h , mi g r a t e , e t c . • H i s A p p s t h e s a me w a y • H i s D a t a • A l l L i n u x s e r v e r s l o o k t h e s a me

P r e s e n t a t i o n ● LinuX Containers (LXC) ● Control Groups & Namespaces ● AUFS ● Client – Server with an HTTP API

Build : Dependencies & Dockerfjle more api/Dockerfile FROM giltarchitecture/ ubuntu-openjdk-7-jre-headless:12.0.4 ADD . /apidoc ENTRYPOINT ["/apidoc/bin/apidoc-api"]

Run Docker docker -run --expose 80 -p 9000:80 mydockercontainer-1-2-3 –Denv.port=90 -Denv.conffile=file.conf deamon mode (-d) or interactif mode (-i)

DOCKER : Share Docker Registries

S h a r e d r e g i s t r i e s : y o u r s

S h a r e d r e g i s t r i e s : y o u r s

DOCKER : CLOUD & HPC Google Container Engine A Container based cloud architecture

G o o g l e C o n t a i n e r E n g i n e ● A l p h a ● t h e G o o g l e C o n t a i n e r E n g i n e i s i n s p i r e d b y G o o g l e ’ s e x p e r i e n c e w i t h b u i l d i n g a n d r u n n i n g c o n t a i n e r - b a s e d d i s t r i b u t e d s y s t e ms , C o n t a i n e r E n g i n e r e - i ma g i n e s s o me o f G o o g l e ’ s mo s t p o w e r f u l i n t e r n a l s y s t e ms , s o t h a t y o u c a n d e v e l o p a n d ma n a g e c o n t a i n e r s t h e w a y G o o g l e ’ s e n g i n e e r s d o . ● Wi t h c o n t a i n e r - b a s e d c o mp u t i n g , a p p l i c a t i o n d e v e l o p e r s c a n f o c u s o n t h e i r a p p l i c a t i o n c o d e , i n s t e a d o f o n d e p l o y me n t s a n d i n t e g r a t i o n i n t o h o s t i n g e n v i r o n me n t s . A t t h e s a me t i me , a p p l i c a t i o n s c a n b e b u i l t w i t h f e w c o n s t r a i n t s . O p e r a t i o n s c a n p r o v i d e a r o b u s t p l a t f o r m t h a t q u i c k l y p r o v i s i o n s c o mp u t e r e s o u r c e s a n d e a s i l y ma n a g e s a p p l i c a t i o n s . T h e t o o l s n e e d t o s u p p o r t t h e r i g h t c o n t r o l s f o r s u c h a p p l i c a t i o n a n d r e s o u r c e ma n a g e me n t . ● T h e f o c u s w i t h C o n t a i n e r E n g i n e i s o n b u i l d i n g t h e s e t o o l s a n d c o n t r o l s f o r o p e r a t i o n s . A t t h e s a me t i me , g o o g l e w a n t t o a l l o w f o r w o r k l o a d mo b i l i t y , w h e r e c o n t a i n e r i z e d a p p l i c a t i o n s c a n r u n mu l t i - c l o u d . T h e y h a v e , t h e r e f o r e , d e s i g n e d C o n t a i n e r E n g i n e t o s u p p o r t K u b e r n e t e s , t h e o p e n s o u r c e t e c h n o l o g y , s o t h a t c u s t o me r s c a n r u n o n mu l t i p l e c l o u d s .

DOCKER : CLOUD & HPC Google Kubernetes A Container based cloud architecture

G o o g l e K u b e r n e t e s ● K u b e r n e t e s i s a n o p e n s o u r c e c o n t a i n e r c l u s t e r ma n a g e r . I t s c h e d u l e s a n y n u mb e r o f c o n t a i n e r r e p l i c a s a c r o s s a g r o u p o f n o d e i n s t a n c e s . A ma s t e r i n s t a n c e e x p o s e s t h e K u b e r n e t e s A P I , t h r o u g h w h i c h t a s k s a r e d e f i n e d . K u b e r n e t e s s p a w n s c o n t a i n e r s o n n o d e s t o h a n d l e t h e d e f i n e d t a s k s . ● T h e n u mb e r a n d t y p e o f c o n t a i n e r s c a n b e d y n a mi c a l l y mo d i f i e d a c c o r d i n g t o n e e d . A n a g e n t ( a k u b e l e t ) o n e a c h n o d e i n s t a n c e mo n i t o r s c o n t a i n e r s a n d r e s t a r t s t h e m i f n e c e s s a r y . ● K u b e r n e t e s i s o p t i mi z e d f o r G o o g l e C l o u d P l a t f o r m, b u t c a n r u n o n a n y p h y s i c a l o r v i r t u a l ma c h i n e .

DOCKER : CLOUD & HPC GO Docker Batch Scheduler with Docker

Job/Interactive context If user request root access: ● mount job directory in container ● mount /softs, /db as read-only ● execute command ● chown job directory with user id after command completion If user does not request root access: ● mount ~user in container ● mount job directory in container ● mount /home/user ● mount additional user requested directories if user has read access (uid/gid) ● mount /softs, /db as read-only ● execute command as user (same uid/gid)