wisent
play

WISENT Distributed processing of energy meteorologic data with - PowerPoint PPT Presentation

Jul 10, 2023 •302 likes •571 views

WISENT Distributed processing of energy meteorologic data with Condor Jan Ploski September 14th, 2006 Slide 1 Business Information Management OFFIS Outline Project data & goals Overview of applications Introduction to Condor Live

  1. WISENT Distributed processing of energy meteorologic data with Condor Jan Ploski September 14th, 2006 Slide 1 Business Information Management OFFIS

  2. Outline Project data & goals Overview of applications Introduction to Condor Live application demo Jan Ploski September 14th, 2006 Slide 2 Business Information Management OFFIS

  3. Project Data BMBF funding programme: e-Science and Networked Knowledge Management Funding agency: DLR NMB+F Project duration: 10/2005 till 09/2008 Awarded project budget: ca. 2 Million EUR 7½ man-years over 3 calendar years Integrated into German D-Grid initiative Participants: Jan Ploski September 14th, 2006 Slide 3 Business Information Management OFFIS

  4. Energy Meteorology An objective of energy meteorology is to obtain the information needed to characterize the fluctuating generation of solar and wind energy. Knowledge acquisition based on interdisciplinary cooperation: physical and meteorological methods transformation of wind and solar energy (physics) energy supply structures (electrical engineering, economics) monitoring methods (computer science, sensor technology) efficient, flexible distributed systems (computer science) Challenges of energy meteorology: preservation of future energy supply large amounts of data (terabytes) complex process chains (partially in real-time) Jan Ploski September 14th, 2006 Slide 4 Business Information Management OFFIS

  5. WISENT Objectives WISENT supports the scientific collaboration of the energy meteorology community by enabling access to Grid technology. WISENT aims to extend and renew today's processes to support new scientific methods and business segments. Jan Ploski September 14th, 2006 Slide 5 Business Information Management OFFIS

  6. Work Package Overview Jan Ploski September 14th, 2006 Slide 6 Business Information Management OFFIS

  7. Work Packages 1-3 WP1: Use of Grid technologies in Virtual Organizations • building a Grid infrastructure for scientific collaboration • tools for networked knowledge and information management • Virtual Institute of Energy Meteorology (vIEM) as an institutional framework WP2: Transfer and integration of data • automation, standardization and monitoring of data transfers • foundation for distributed computing WP3: Domain-specific standards and semantic interoperability • standardization of interfaces • evaluation of data quality • interoperability between software systems of the project partners Jan Ploski September 14th, 2006 Slide 7 Business Information Management OFFIS

  8. Work Packages 4-7 WP4: Efficient interaction with large data objects • visualization of large amounts of data • new forms of visualization • preemptive data distribution in the Grid WP5: Distributed processing of energy meteorologic data • Grid Computing: parallelizing of computations • scalability advantages for existing systems • enabling new application fields through Grid technology • flexible use of external Grid resources WP6: Cooperation with other Grid communities • D-Grid; establishment of the „Energy Meteorology“ community • EU project EGEE (Enabling Grids for E-sciencE) WP7: Project coordination Jan Ploski September 14th, 2006 Slide 8 Business Information Management OFFIS

  9. WP1: TikiWiki Collaboration System http://wisent.d-grid.de Jan Ploski September 14th, 2006 Slide 9 Business Information Management OFFIS

  10. WP2: Data Transfer meteocontrol <-> Uni OL Jan Ploski September 14th, 2006 Slide 10 Business Information Management OFFIS

  11. Photovoltaic Module Monitoring by meteocontrol Jan Ploski September 14th, 2006 Slide 11 Business Information Management OFFIS

  12. STEPS – Finding Locations for Solar Power Plants Source: DLR-TT Jan Ploski September 14th, 2006 Slide 12 Business Information Management OFFIS

  13. 3D Simulation of Solar Radiation Transfer Source: DLR-IPA Jan Ploski September 14th, 2006 Slide 13 Business Information Management OFFIS

  14. 3D Simulation of Solar Radiation Transfer, cont. Source: DLR-IPA Jan Ploski September 14th, 2006 Slide 14 Business Information Management OFFIS

  15. Cloud Index Computation Raw data, Source: Uni OL Jan Ploski September 14th, 2006 Slide 15 Business Information Management OFFIS

  16. Cloud Index Computation, cont. Ground albedo, Source: Uni OL Jan Ploski September 14th, 2006 Slide 16 Business Information Management OFFIS

  17. Cloud Index Computation, cont. Output, Source: Uni OL Jan Ploski September 14th, 2006 Slide 17 Business Information Management OFFIS

  18. Bottom-Up Introduction of Grid Middleware Jan Ploski September 14th, 2006 Slide 18 Business Information Management OFFIS

  19. Condor Workload management system for compute-intensive jobs Developed at University of Winsconsin-Madison In production for >15 years Deployed world-wide in >1600 pools managing >100,000 hosts Keywords: high-throughput computing (not HPC!) opportunistic scheduling cycle scavenging desktop Grids Jan Ploski September 14th, 2006 Slide 19 Business Information Management OFFIS

  20. Condor Features Condor balances the conflicting requirements of resource users resource owners Condor manages heterogenous computing resources different types: from workstations to dedicated clusters different platforms (Unix, Windows) Similar middleware products: Portable Batch System (PBS) Load Sharing Facility (LSF) Sun Grid Engine IBM LoadLeveler Jan Ploski September 14th, 2006 Slide 20 Business Information Management OFFIS

  21. Condor Features, cont. Job queuing Flexible scheduling based on user-defined priorities Resource monitoring Resource management Basic use scenario: Users submit one or more compute jobs to a Condor pool Condor decides when and where to execute jobs Condor monitors progress of each job's execution Users are notified upon their jobs' completion Jan Ploski September 14th, 2006 Slide 21 Business Information Management OFFIS

  22. Basic Architecture of a Condor Pool Jan Ploski September 14th, 2006 Slide 22 Business Information Management OFFIS

  23. Condor Jobs A job consists of a (batch) executable with associated input/output files and a description of the desired execution context. (Optional) Stage-in/stage-out for files Restricted communication among jobs no support for interactive jobs distributed memory model independent job execution (possibly with input/output dependencies) limited support for message passing ideal for data parallelization, little use for program parallelization emphasis on throughput, not execution time Jan Ploski September 14th, 2006 Slide 23 Business Information Management OFFIS

  24. Condor Jobs, cont. Condor universes (job categories) standard vanilla Java parallel ... The target universe for a job... determines the runtime environment in which the job is executed (e.g., sandboxing) imposes restrictions on what the executable may (not) do defines a specific set of features (e.g., process migration & checkpointing) Jan Ploski September 14th, 2006 Slide 24 Business Information Management OFFIS

  25. ClassAds Schedulers advertise job requirements from their queue. Resource owners advertise the type and availability of resources. Central mgr matches both types of ads and arranges job executions. Source: Thain et. al, 2004 Jan Ploski September 14th, 2006 Slide 25 Business Information Management OFFIS

  26. Condor and the Grid Run behind a higher-level Grid middleware (e.g., Globus, UNICORE) Use native support for inter-pool communication - „flocking“ Use the grid universe to submit Condor jobs to remote queues Interoperates with: Condor, Globus Toolkit, NorduGrid, Unicore, LSF, PBS Use the glidein mechanism for adding remote resources to the pool Jan Ploski September 14th, 2006 Slide 26 Business Information Management OFFIS

Recommend

More recommend