Grid Computing for I ndustry Grid Computing for I ndustry – Early Applications Early Applications – Hing-Yan LEE National Grid Office Singapore International Symposium on Grid Computing 2007
Preparing for I ndustry Adoption Preparing for I ndustry Adoption
Multi- -Organization Grid Organization Grid Multi Accounting System Accounting System Accounting Organisational level (MOGAS) (MOGAS) consumer-provider business relationship Bill for organization A = ∑ (usage of members of organization A at resources owned by other organizations) Metering Usage charging $ = f (CPU , memory, license,…) Ganglia Only stores & reports on information of resource Monitoring status, no information of users & their jobs Collection of raw data Hence, no metering & accounting mechanism Resource Resource … Usage Usage [Courtesy of A/Prof. Francis Lee, NTU]
PRAGMA MOGAS status PRAGMA MOGAS status (12/10/2006) (12/10/2006) KISTI, Korea AIST, Japan NCSA, USA CNIC, China TITECH, Japan SDSC, USA OSAKAU, Japan UoHyd, India NCHC, Taiwan CICESE, Mexico ASCC, Taiwan KU, Thailand UNAM, Mexico IOIT-HCM USM, Malaysia BII, Singapore MIMOS QUT, Australia UChile, Chile MU, Australia NGO, Singapore GT4 14 Organizations deployed with MOGAS, 5 of them with GT4 GT2 Cindy Zheng, GGF13, 12/10/06 modified by A/Prof. Bu-Sung Lee
Netrust Certificate Authority Certificate Authority Netrust • Commercial CA & Singapore’s only Licensed CA • Current Status: – Completed migration. All existing NGPP sites have migrated their host certificates to Netrust certificates. – Continual effort to issue certificates for • New users of NGPP resources • Additional hosts added to NGPP – Seeking accreditation from APGrid PMA (part of IGTF) • Temporary CA will continue to exist to issue certificates for testing & trials .
Meta Scheduler Rollout Meta Scheduler Rollout • Objective – To provide LSF Meta-Scheduler for seamless access to NGPP compute resources • Collaboration with Platform Computing • Meta-Scheduler inter-operates with – LSF, PBS Pro & N1GE • Completed: Q3 2005
NGPP Meta- -Scheduler Scheduler NGPP Meta The LSF Meta-Scheduler • interfaces with local workload schedulers (e.g. Sun’s N1GE, Platform’s LSF) of resources on NGPP, and schedules the job to the best available resources. • Current applications using or planning to use the Meta-Scheduler include: – MicroRNA project (BII) – Media Grid (AE@SG) – DMG Portal (NGPP) – Multipitch Speech project (I 2 R) NUS-SMA I MCB NGO-GOG I HPC Hydra3 4 CPUs Soursop Lime 60 CPUs 78 CPUs 6 CPUs
NGPP Solution NGPP Solution
-Concepts Concepts of- -of Proof- Proof
Goal of Proof-of-Concept For users to harness resources on NGPP to run FEMLAB on a pay-per-use basis resulting in cost-savings for companies, encouraging R&D activities, & improving utilization of resources.
Test- -bedding of Provisioning bedding of Provisioning Test • Objective – To assess viability of a spill-over facility using GOG clusters – To measure speed of provisioning – To explore appropriate cost model • Results – Accuracy of results confirmed – Mentor Graphics to review license conditions
Matlab POC POC Matlab • Aim: – To explore distributed version of MATLAB (Distributed Computing Toolbox) on GOG cluster
Digital Media Digital Media
Enterprise Grid for SMEs SMEs Enterprise Grid for • Objectives – Create awareness in SMEs – Provide assistance & guidance to SMEs to harness Grid Computing to exploit internal compute resources • Beneficiaries – MegaMedia’s digital media hub (dmh) – Ngee Ann Polytechnic – Singapore Polytechnic
Setting Up – – Setting Up • Set up & configure a 10-node cluster with mental ray software • Able to successfully check out licenses from NG’s license server to their compute cluster for rendering. Completed Jul 2006. • Following-up with teachers of Digital Media class for students to use the licenses for class assignments and projects. • Same effort for
Rendering Grid Rendering Grid • Objectives – To provide licenses and machines to Digital Media companies for their rendering needs, as a means to move them to utilizing services on the Grid – Nurturing the emergence of a Grid Service Provider • Mental ray licenses for 64 CPUs acquired – Free access & no charge to commercial & tertiary users – Floating licenses – Support from mental images GmbH • Compute resources from GOG machines
Use of mental ray Licenses Use of mental ray Licenses • Digital Media companies that have used the facilities • TheContentCompany • Omens-Studio • VHQ Productions • Companies in discussion • 3dSense (animation school – 3dsMax & Maya) • Big Communications Pte Ltd (animation - Maya) • ST Electronics (Digital Media) Pte Ltd formerly IV Labs (animation & games) • mega media (animation – Maya) • Visual Mediaworks (animation – 3dsMax) • Lights & Shadows (animation – 3dsMax) • 7cans (animation & games – 3dsMax) • 10tacle (games – 3dsMax) • Frontline Technologies Pte Ltd (Grid Service Broker)
Usage – – TheContentCompany TheContentCompany Usage • A 4-man company • Title: The Yellowman Shortfilm • Required resources for rendering a submission to “Cannes Lions 2006” 53 rd International Advertising Festival (18-24 June 2006)
Usage – – Usage “Smoking Fish – Hold onto Your Butt” Public Service Announcement • commissioned by the Surfrider Foundation of America to educate the public about how pollution is harming the country’s coastline . • Worked in tandem with Hollywood writer Adam McKay (“Anchorman - Legend of Ron Burgundy”, “Bewitched”) & director Ian O’Roarty • Omens Studios produced the CG animation for the project. The PSA was televised nationally in USA in 2006 on various networks.
Quotes Quotes • “It is now easier and faster to visualize our work, instead of waiting for days before the work is finished.” Tang Chi Sim MD, Omens Studios • “Grid Computing allows local digital media firms to focus on the creative side of things, rather than worrying over the lack of computing resources.” Yeo Chun Cheng CIO, Media Development Authority
Other Applications Other Applications
Land Data Hub - - 1 1 Land Data Hub • A statutory board whose main focus in land resource optimization • Maintains the national repository of land information (Land Data Hub or LDH) – Collated from 14 public & private agencies – Shared across the public sector • Project Objective – Replace aging IT infrastructure with a more efficient & flexible architecture
Land Data Hub - - 2 2 Land Data Hub • Motivated by – Need to cut costs – Achieve efficient database & system management – Ensure high scalability & availability – Allow fast & convenient data sharing among LDH members • Designed to be – A platform for inter-agency collaboration to develop cross-agency data products & services for the public – An inter-agency consultation platform
Land Data Hub - - 3 3 Land Data Hub • “Enterprise grid computing offers new capabilities on how geospatial data can be shared at a potentially lower cost” • “The Oracle 10g grid platform redefines how data is collected and shared among agencies while delivering greater efficiencies at lower costs” – Lewis Wu (Director, IT, SLA)
CrossWalk CrossWalk • Users – IT Department, National Library Board • Challenge – Reduce record conversion time from 3 full days to an 8-hour overnight run • Domain – Data processing for monthly reporting • Solution – Distribute the workload across as many idle computers as possible to convert the records in parallel using Condor workload management system – Instead of adding converted records t database residing on central server, each computer has its local database to store converted records – Removes bottleneck during records insertions into a central database, but requires an additional step of merging local databases subsequently • Benefits – Speedup of processing using idle compute resources within the enterprise, with no additional investment required.
CrossWalk Benchmark Benchmark - - 2 2 CrossWalk • Input – 50,000 records • Output – 50,000 records • Benchmarking Result – Original: 1 CPU takes about ~ 3 days – Grid: 4 CPUs take about ~ 3.5 hours • Grid resources are not dedicated for the benchmark & shared load with other jobs – Will require about ~ 29 CPUs to achieve target processing time
Recommend
More recommend
