Current Status of CVMFS Fast, scalable, and reliable software - PowerPoint PPT Presentation

Current Status of CVMFS Fast, scalable, and reliable software distribution Jakob Blomer July 3rd, 2009 1 / 20

Outline 1 Overview 2 Recent Development 3 Benchmarks 4 Next Steps 2 / 20

1 Overview 2 Recent Development 3 Benchmarks 4 Next Steps 3 / 20

Software Distribution for Virtual Machines • “Ready to run” binaries, i. e. /mnt/cvmfs mirrors destination of make install • Read-only, public files 4 / 20

Repository Statistics Latest Release, CernVM x86 Platform • Many (many) small files • In the repository: significant amount of redundancy 10 th / 90 th # Files Size Median ZIP-Ratio Percentile ALICE 38 000 1.6 GB 3.3 KB 330 Byte / 24 KB 2.8 ATLAS 150 000 6.5 GB 2.2 KB 217 Byte / 36 KB 10.7 CMS 110 000 5 GB 2.1 KB 320 Byte / 20 KB 1.9 LHCb 166 000 4.5 GB 1.5 KB 32 Byte / 16 KB 2.6 5 / 20

Infrastructure Reference setup: Apache + Squid Key: Set of web caches serving static HTTP content from memory 6 / 20

1 Overview 2 Recent Development 3 Benchmarks 4 Next Steps 7 / 20

The CernVM File System Evolved from GROW-FS Fuse Modules CVMFS S CVMFS Components Quota / LRU Trace Capturing Catalog Cache Prefetcher Building Blocks SQLite libcurl libcrypto SHA1 MD5 zlib 8 / 20

Distinctive Features • Satisfied with outgoing HTTP(S) connection, i. e. works with practically every Internet connection • Every downloaded file is verified by SHA1 • Chain of forward proxy servers • Pre-built cache packages • Offline mode • Trace file system operations • Nested catalogs • Catalogs can be signed by X.509 certificate • Catalogs store TTL, thereby stay up to date 9 / 20

Distinctive Features • Satisfied with outgoing HTTP(S) connection, i. e. works with practically every Internet connection • Every downloaded file is verified by SHA1 • Chain of forward proxy servers • Pre-built cache packages • Offline mode • Trace file system operations • Nested catalogs • Catalogs can be signed by X.509 certificate • Catalogs store TTL, thereby stay up to date 9 / 20

Distinctive Features • Satisfied with outgoing HTTP(S) connection, i. e. works with practically every Internet connection • Every downloaded file is verified by SHA1 • Chain of forward proxy servers • Pre-built cache packages • Offline mode • Trace file system operations • Nested catalogs • Catalogs can be signed by X.509 certificate • Catalogs store TTL, thereby stay up to date 9 / 20

Distinctive Features • Satisfied with outgoing HTTP(S) connection, i. e. works with practically every Internet connection • Every downloaded file is verified by SHA1 • Chain of forward proxy servers • Pre-built cache packages • Offline mode • Trace file system operations • Nested catalogs • Catalogs can be signed by X.509 certificate • Catalogs store TTL, thereby stay up to date 9 / 20

Distinctive Features • Satisfied with outgoing HTTP(S) connection, i. e. works with practically every Internet connection • Every downloaded file is verified by SHA1 • Chain of forward proxy servers • Pre-built cache packages • Offline mode • Trace file system operations • Nested catalogs • Catalogs can be signed by X.509 certificate • Catalogs store TTL, thereby stay up to date 9 / 20

Distinctive Features • Satisfied with outgoing HTTP(S) connection, i. e. works with practically every Internet connection • Every downloaded file is verified by SHA1 • Chain of forward proxy servers • Pre-built cache packages • Offline mode • Trace file system operations • Nested catalogs • Catalogs can be signed by X.509 certificate • Catalogs store TTL, thereby stay up to date 9 / 20

Distinctive Features • Satisfied with outgoing HTTP(S) connection, i. e. works with practically every Internet connection • Every downloaded file is verified by SHA1 • Chain of forward proxy servers • Pre-built cache packages • Offline mode • Trace file system operations • Nested catalogs • Catalogs can be signed by X.509 certificate • Catalogs store TTL, thereby stay up to date 9 / 20

Distinctive Features • Satisfied with outgoing HTTP(S) connection, i. e. works with practically every Internet connection • Every downloaded file is verified by SHA1 • Chain of forward proxy servers • Pre-built cache packages • Offline mode • Trace file system operations • Nested catalogs • Catalogs can be signed by X.509 certificate • Catalogs store TTL, thereby stay up to date 9 / 20

Distinctive Features • Satisfied with outgoing HTTP(S) connection, i. e. works with practically every Internet connection • Every downloaded file is verified by SHA1 • Chain of forward proxy servers • Pre-built cache packages • Offline mode • Trace file system operations • Nested catalogs • Catalogs can be signed by X.509 certificate • Catalogs store TTL, thereby stay up to date 9 / 20

S CVMFS Catalog, Compression S CVMFS make install • S CVMFS provides a frontend to a repository directory • Missing part: atomic switch between repository versions (e. g. with ZFS) 10 / 20

Performance • Locally SHA1-Cache • Multi-Threaded • SQLite for catalogs instead of in-memory tree representation • Pure data cache, decoupled from catalog Protocol Overhead • Network Traffic Web Server • HTTP Keep-Alive T S F K E Y 200 OK I G C N N N SYN ACK A • Prefetching A , Y , P C , A S A N K T C • Compression C T I F K K H CernVM 70 MB repository = ⇒ 20 MB network traffic 11 / 20

Performance Prefetcher High predictability: ≈ 10-50 shared libraries per binary Construct prefetch hints: 1 Follow ldd -dependencies 2 Automatic construction from traces Analysis of open() sequence Automatic Construction (ROOT Extract) libGraf.so.5.22 (7) libGraf3d.so.5.22 (7) 100% 100% 100% 100% 100% libNet.so.5.22 (7) libHist.so.5.22 (7) 100% 100% 100% 18% 100% libCint.so.5.22 (7) libRIO.so.5.22 (7) 100% 100% 18% libMinuit.so (11) libCore.so.5.22 (7) 12 / 20

Performance Prefetcher High predictability: ≈ 10-50 shared libraries per binary Construct prefetch hints: 1 Follow ldd -dependencies 2 Automatic construction from traces Analysis of open() sequence Automatic Construction (ROOT Extract) libGraf.so.5.22 (7) libGraf3d.so.5.22 (7) 100% 100% 100% 100% 100% libNet.so.5.22 (7) libHist.so.5.22 (7) 100% 100% 100% 18% 100% libCint.so.5.22 (7) libRIO.so.5.22 (7) 100% 100% 18% libMinuit.so (11) libCore.so.5.22 (7) 12 / 20

1 Overview 2 Recent Development 3 Benchmarks 4 Next Steps 13 / 20

Setup We measure extra time for application benchmark stressHepix residing on CVMFS / AFS 14 / 20

High Latency Extra Running Time 800 AFS 700 CVMFS + Keep-Alive 600 + Prefetching 500 + Compression ∆ t [s] 400 300 200 100 0 LAN 25 50 100 150 Round Trip Time [ms] 15 / 20

WAN Simulation Extra Running Time (RTT 100ms) 800 AFS 700 CVMFS + Keep-Alive 600 + Prefetching, Compression 500 ∆ t [s] 400 300 200 100 0 0 5 10 15 20 25 iPerf-measured Throughput [mbit/s] 16 / 20

Gbit Ethernet 187 186 185 184 Running Time [s] 183 182 181 180 179 178 177 Local Storage AFS CVMFS Current 17 / 20

1 Overview 2 Recent Development 3 Benchmarks 4 Next Steps 18 / 20

Next Steps • Testing • Zero-Conf, in particular nearby proxy server • “Share My Cache”: Replace Squid by co-operative peers 19 / 20

Current Status of CVMFS Fast, scalable, and reliable software distribution Jakob Blomer July 3rd, 2009 20 / 20