SVN Pilot: CVS Replacement Manuel Guijarro Jonatan Hugo Hugosson - PowerPoint PPT Presentation

SVN Pilot: CVS Replacement Manuel Guijarro Jonatan Hugo Hugosson Artur Wiecek David Horat Jonathan Brugge Jonathan Brugge Michel Manent September 2008 CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/ i t

Outline • Introduction • Motivation • Subversion • Objectives • Performance Tests Performance Tests • Security • Implementation I l t ti • Questions CERN - IT Department CH-1211 Genève 23 Switzerland 2 www.cern.ch/ i t

Version Control Systems • Maintain current and historical versions of files and data (source code) files and data (source code) • There are many commercial and Open S Source VC Systems: VC S – (Centralised) CVS/SVN – (Distributed) GIT, Bazaar, Darcs, GNU arch, Mercurial, Monotone, etc – But subversion seems to be the most popular one (used by GCC, Phyton, PuTTY, Apache, GNOME KDE GNOME, KDE, etc) etc) • Physics User Community: (IN2P3, ROOT, Totem ) Totem..) CERN - IT Department CH-1211 Genève 23 Switzerland 3 www.cern.ch/ i t

CERN Central CVS Service • Hosts over 330 Software Projects – 29 for Atlas 29 f Atl – 46 for CMS – 8 for LHCb,….. 8 f LHCb • Over 3000 developers registered • Over 90 GBytes of source code • Creates 250 Remedy tickets per year Creates 250 Remedy tickets per year • Over 100000 commits per month CERN - IT Department CH-1211 Genève 23 Switzerland 4 www.cern.ch/ i t

CERN Central CVS Service CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/ i t

Central CVS service features • High Availability and Load Balancing • Web interface to repositories f • Usage Statistics • Repository Remote Replication + Mirroring • Daily archive of Repositories and DR Daily archive of Repositories and DR • Developers Mailing list • Pre/Post Commit Actions (such us e-mail P /P t C it A ti ( h il notification, etc) • Various access method (ssh/kerberos) • Role split (CVS Admin/Librarian/Developer) p ( p ) CERN - IT Department CH-1211 Genève 23 Switzerland 6 www.cern.ch/ i t

Motivation for SVN Pilot • Originally designed to host less than 100 projects projects • Requests to provide a central SVN service: – From CMS – From ATLAS (case study in 2006) – And from many others • CVS is over 20 years old while SVN is this y millennium technology • Requests for Read Access control Requests for Read Access control CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/ i t

SVN vs. CVS Feature SVN CVS Speed Speed Faster Faster Slower Slower Permission Full Limited File types All Limited Off line operations Yes No Repository format Database File system Locks No Yes Atomic commits Yes No CERN - IT Department CH-1211 Genève 23 Switzerland 8 www.cern.ch/ i t

New Features (SVN 1.5) • Automatic update of working copy • Merge tracking – Subversion keeps track of what changes have b been merged where d h • Sparse checkouts • Interactive conflict resolution CERN - IT Department CH-1211 Genève 23 Switzerland 9 www.cern.ch/ i t

Pilot Objectives • Provide current CVS service features • Add new features (available with SVN) • Add new features (available with SVN) – Control Read access per path (module) – Authenticated Web access Authenticated Web access – Binary files handling • Ease CVS to SVN migration g • Improved usage statistics (SVN Stats) • Handling of first line support via the Help Desk g pp p • Delegate administrative tasks to Software Librarians of each project • Prevent uncontrolled setup of SVN servers • Manpower: 1.2 FTE project p p j CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/ i t

Timetable • Preliminary study Feb 2008 2008 • CVS librarians feedback May y 2008 • SVN Pilot SVN Pilot July July 2008 • SVN service in production p Dec • CVS to SVN migration 2008 • CVS service close down Dec 2009 CERN - IT Department CH-1211 Genève 23 Switzerland 11 www.cern.ch/ i t

SVN Pilot study • Access methods – https htt – ssh • Shared storage – NFS 3/4 – AFS • Securing service g – Restricted Shell – Chrooted hooks (commit scripts) ( p ) • Infrastructure: – Librarian tools Statistics Web Interface – Librarian tools, Statistics, Web Interface,… CERN - IT Department CH-1211 Genève 23 Switzerland 12 www.cern.ch/ i t

Performance Tests • SVN check out of a 110 Mb project • Parameters – AFS/NFS3/NFS4 – HTTPS/SSH CERN - IT Department CH-1211 Genève 23 Switzerland 13 www.cern.ch/ i t

AFS vs NFS3 (1 server) CERN - IT Department CH-1211 Genève 23 Switzerland 14 www.cern.ch/ i t

AFS vs NFS4 (1 server) CERN - IT Department CH-1211 Genève 23 Switzerland 15 www.cern.ch/ i t

AFS vs NFS4 (3 servers) CERN - IT Department CH-1211 Genève 23 Switzerland 16 www.cern.ch/ i t

Preliminary Conclusions • AFS much faster than NFS • SSH much faster than https SS f • SSH scales very well with high load • … • New tests ongoing (with mixture of read and New tests ongoing (with mixture of read and write operations) CERN - IT Department CH-1211 Genève 23 Switzerland 17 www.cern.ch/ i t

Security • Project Isolation • Windows/Linux clients / • Worldwide access • Shared file system independent • Hooks executed on servers Hooks executed on servers • Librarians may put any script into the hooks • Librarians might need file system level • Librarians might need file system level access to repository – being studied Security risk!! Security risk!! CERN - IT Department CH-1211 Genève 23 Switzerland 18 www.cern.ch/ i t

Hooks (scripts) Client Server Svn commit Pre-commit hook Pre-commit hook is executed Post commit hook is executed SVN: Commit OK SVN: Commit OK Email notification recieved CERN - IT Department CH-1211 Genève 23 Switzerland 19 www.cern.ch/ i t

Hook scripts chrooted: Server svnserer Librarian hooks: jailed hooks/post-commit hook Usr-hooks/post- commit hook Repository (1) Repositories System files CERN - IT Department CH-1211 Genève 23 Switzerland 20 www.cern.ch/ i t

Architecture • svn.cern.ch (rw) – Secured subversion S d b i server (only ssh) – Read and write Read and write access to repository • svnweb cern ch (ro) • svnweb.cern.ch (ro) – User documentation – Project request P j t t – SVN web interface – Usage statistics U t ti ti CERN - IT Department CH-1211 Genève 23 Switzerland 21 www.cern.ch/ i t

Pilot Implementation Summary • SSH access for SVN clients – Restricted shell for all SVN clients R t i t d h ll f ll SVN li t – Hooks chrooted • SVN web (ro) – Web interfaces: websvn, trac – SSO Authenticated access • Administration delegated to librarian g – Access rights – Hooks – Admin tools CERN - IT Department CH-1211 Genève 23 Switzerland 22 www.cern.ch/ i t

Conclusions • Secure service • This will replace CVS by end of 2009 C S f • The service is supported (pre-production) • Pilot setup may differ from final setup – Access method, Web interface, shared file Access method, Web interface, shared file system, etc. – Changes will be transparent to the users g p CERN - IT Department CH-1211 Genève 23 Switzerland 23 www.cern.ch/ i t

Support http://cern.ch/svn –Try the pilot –Documentation Svn support@cern ch Svn.support@cern.ch CERN - IT Department CH-1211 Genève 23 Switzerland 24 www.cern.ch/ i t

Questions? Th Thanks For Listening…. k F Li t i M. Guijarro, A. Wiecek, David Horat, Jonathan Bugge, M. Manent, H. Hugosson CERN - IT Department CH-1211 Genève 23 Switzerland 25 www.cern.ch/ i t