Enabling CernVM for the Interactive Use Case Vasilis Nicolaou SFT - PowerPoint PPT Presentation

Enabling µ CernVM for the Interactive Use Case Vasilis Nicolaou SFT Group CERN vasilis.nicolaou@cern.ch Supervisor: Dr. Jakob Blomer Summer, 2013

Overview Web appliance agent for the µ CernVM 1 Merging user databases 2 Merging RPM databases 3 Reimplementation of the cernvm-update script 4 Vasilis Nicolaou (CERN) Enabling µ CernVM for IUC Summer 2013 2 / 25

Web appliance agent for the µ CernVM Motivation Motivation The µ CernVM will be used by physicists to access LHC data for analysis and µ CernVM architecture development being the successor of the CernVM which had an open source appliance agent designed by rPath that is no longer available. A new one needs to be developed to: Provide valuable information to reduce the users need of using shell commands Provide a user friendly layer for interacting with the system for simple In summary: tasks such as updating or powering off A read-only layer (R-O layer) and is mounted the system from the network Provide an easy mechanism for expanding the web application by A read/write layer (R/W layer) which is adding shell commands created on first boot Vasilis Nicolaou (CERN) Enabling µ CernVM for IUC Summer 2013 3 / 25

Web appliance agent for the µ CernVM Motivation Web appliance agent for the µ CernVM What is the web appliance agent? A web application that provides basic information related to the state of the µ CernVM such as memory usage, update notifications and cron jobs. Supports basic user interaction for altering the state of the machine such as updating the underlying operating system. Vasilis Nicolaou (CERN) Enabling µ CernVM for IUC Summer 2013 4 / 25

Web appliance agent for the µ CernVM User interaction User interaction with the appliance agent Vasilis Nicolaou (CERN) Enabling µ CernVM for IUC Summer 2013 5 / 25

Web appliance agent for the µ CernVM Extending the web appliance agent Extending the web appliance agent as a user API Management screen Information while API is enabled Helps the users with adding useful bash commands Supports deleting previously added bash commands Supports two formats of the generated HTML view; raw and table There is a technical report that contains more details on how to use the API and sample code Vasilis Nicolaou (CERN) Enabling µ CernVM for IUC Summer 2013 6 / 25

Web appliance agent for the µ CernVM Extending the web appliance agent Extending the web appliance agent as a user Vasilis Nicolaou (CERN) Enabling µ CernVM for IUC Summer 2013 7 / 25

Web appliance agent for the µ CernVM Extending the web appliance agent Extending the web appliance agent as a developer The framework of the web application was developed according to the model-view-controller architecture It does not matter how you develop your application as long as it directs its HTML output in a certain way Every embedded application should load the Configuration module and create the view. When the application has some output to display it should call the view.setContent in order to display the contents on the application reserved area Example code sys.path.append(os.environ[’MY_HOME ’]+ ’/ etc/config ’) sys.path.append(os.environ[’MY_HOME ’]+ ’/cgi -bin/chrome ’) from cern_vm import Configuration from view import View #[..]# config= Configuration () view = View(config.system.actions) view. setContent ("My application ", "Hello World ") Vasilis Nicolaou (CERN) Enabling µ CernVM for IUC Summer 2013 8 / 25

Web appliance agent for the µ CernVM Extending the web appliance agent Extending the web appliance agent as a developer Vasilis Nicolaou (CERN) Enabling µ CernVM for IUC Summer 2013 9 / 25

Merging user databases Merging user database Shadow, passwd and group Vasilis Nicolaou (CERN) Enabling µ CernVM for IUC Summer 2013 10 / 25

Merging user databases Merging user database The user database consists of three files: 1 /etc/passwd contains basic information about the users such as their username, id and login shell 2 /etc/group contains basic information about the groups such as their id, members and permissions 3 /etc/shadow is where the user passwords are stored The users can modify all the three files, which may cause conflicts Vasilis Nicolaou (CERN) Enabling µ CernVM for IUC Summer 2013 11 / 25

Merging user databases The problem of having two versions of user databases The problem of having two versions of user databases Conflicts with IDs, usernames, password inconsistency and changes in group members have to be resolved. Vasilis Nicolaou (CERN) Enabling µ CernVM for IUC Summer 2013 12 / 25

Merging user databases The current solution The current solution Merging /etc/group|passwd|shadow with the respective files from the R-O layer Vasilis Nicolaou (CERN) Enabling µ CernVM for IUC Summer 2013 13 / 25

Merging user databases Deploying the solution to the µ CernVM Deploying the solution to the µ CernVM Before any service starts, a series of scripts are executed to prepare the µ CernVM by mounting the cvmfs and merging it with the R/W layer on the local machine. User database is critical and it must be ready before services and daemons run since they often run under a system user account. Vasilis Nicolaou (CERN) Enabling µ CernVM for IUC Summer 2013 14 / 25

Merging RPM databases Merging RPM databases Vasilis Nicolaou (CERN) Enabling µ CernVM for IUC Summer 2013 15 / 25

Merging RPM databases Merging RPM databases Two RPM databases: One in The R/W and R-O RPM databases the R-O file system and one generated on the first boot and placed in the R/W layer The R/W can be modified When a new RPM database Example rpm R/W and R-O diff after installing gedit is available on the R-O layer, a mechanism must exist to merge the two into one new R/W database Vasilis Nicolaou (CERN) Enabling µ CernVM for IUC Summer 2013 16 / 25

Merging RPM databases The algorithmic solution The algorithmic solution simplified Vasilis Nicolaou (CERN) Enabling µ CernVM for IUC Summer 2013 17 / 25

Reimplementation of the cernvm-update script Reimplementation of the cernvm-update script The algorithm that merges the rpm databases is a vital part of the cernvm-update script which has the role of the update manager and it is responsible to leave the system in a state identical to what it would have been on a desktop Linux distribution. Usage Argument Action c check for new updates a Apply the new update if it exists p Patch the update with the user packages Vasilis Nicolaou (CERN) Enabling µ CernVM for IUC Summer 2013 18 / 25

Reimplementation of the cernvm-update script Fetching the update packs via mounting cvmfs Fetching the update packs via mounting cvmfs Configuration file CVMFS_CACHE_BASE = $tmplocation /cache The remote cvmfs repository is CVMFS_RELOAD_SOCKETS = $tmplocation /cache CVMFS_SERVER_URL =\ mounted http :// hepvm.cern.ch/cvmfs/cernvm -devel.cern.ch CVMFS_HTTP_PROXY =DIRECT Check latest update-pack CVMFS_KEYS_DIR =/ mnt /.rw/aux against the current system version Calling the cvmfs2 tool When a new update exists the cvmfs2 -o config=" $configuration "\ user may apply the update $(attr -qg fqrn /mnt /.ro)\ " $tmplocation / mountpoint "\ Sanity checks take place >/dev/null Once everything is OK, the Unmounting and clean up system will be updated on next reboot fusermount -u $tmplocation / mountpoint rm -rf $tmplocation Vasilis Nicolaou (CERN) Enabling µ CernVM for IUC Summer 2013 19 / 25

Reimplementation of the cernvm-update script Applying for the updates Applying for the updates ’Applying’ in this context does not mean that the updates are installed. The system is left in a state that when it will be rebooted the update process will be initiated. Remove the .cvmfs_snaphot Set in /mnt/.rw/ucontext/context file the version that the system is going to be updated to On reboot the rebase will take place meaning that new files will replace old ones. Vasilis Nicolaou (CERN) Enabling µ CernVM for IUC Summer 2013 20 / 25

Reimplementation of the cernvm-update script Patching the R/W layer with the system updates Patching the R/W layer with the system updates There are rules on which files are Process Summary replaced: The R-O RPM database is copied to a Normally most files under /var temporary directory and the RPM and /etc are kept replicas generated at pre-update stage are installed by referencing the However there is a problem temporary database with some of those files such The database then replaces the one under /var/lib/rpm as the user database consisting A new empty rpm database is created of /etc/passwd|group|shadow under a temporary directory and /var/lib/rpm/* . The post installation scripts that were chosen from pre-update stage are Such files cannot just be executed via calling rpm -i that points replaced or be left as they are, to the temporary database requesting a merging strategy is needed that no dependencies are checked Vasilis Nicolaou (CERN) Enabling µ CernVM for IUC Summer 2013 21 / 25

Reimplementation of the cernvm-update script User interaction with the update mechanism User interaction with the update mechanism Use the cernvm-update script Use a special button to press and apply the updates Get notifications from the desktop environment or the live feed inside the web interface Vasilis Nicolaou (CERN) Enabling µ CernVM for IUC Summer 2013 22 / 25