Exploiting Constraints to Build a Flexible and Extensible Data Stream Processing Middleware Workshop on Scalable Stream Processing Systems (SSPS) IPDPS 2010, Atlanta/Georgia, USA 19.04.2010 Nazario Cipriani, Carlos Lübbe, Alexander Moosbrugger
Data Stream Processing Systems – Some Foundations 2 Context-aware applications need context data to be processed Context-aware systems utterly heterogeneous Nearby sensors produce context data streams that needs to be processed online Not convenient to first store and then process offline due to high data volume Use Data Stream Processing Systems (DSPS) ! Data streams are handled by Data Stream Processing Systems A.k.a. Data Stream Management Systems ( DSMS ) Data streams are processed according to some processing definition Push-based processing paradigm instead of pull-based Today's DSPS offer a broad range of sophisticated and efficient processing schemes for online data stream processing Well suited for general purpose data stream processing Universität Stuttgart SFB 627
Agenda 3 Non-Trivial Application Scenario of a Distributed Visualization Pipeline Adaptation Problem for non-trivial Applications Gap Between Application’s and System’s Interests Constraints Classification Enhancing the NexusDS Platform by Constraints NexusDS Platform Overview Network Groups and Operator Model Constraint-Based Data Flow Graphs Enhanced Processing Model Supporting Constraints Conclusion and Future Work Universität Stuttgart SFB 627
Sample Scenario – Distributed Visualization Pipeline 4 Sources Operators Sinks Visualization-Pipeline User Location Updates Data Stream: Bus, Taxi, and User Locations Client: Image Stream Merge & Filtering: Mapping: Rendering: Selection of Rectangular Map Projection Nearby Objects with 3D Buildungs Rasterize e.g., Smartphones, PDAs, etc. Static Data: Buildings & Streets Points of Interest Parameter Updates Universität Stuttgart SFB 627
Sample Scenario – Distributed Visualization Pipeline 5 Sources Operators Sinks Visualization-Pipeline User Location Updates Data Stream: Bus, Taxi, and User Locations Client: Image Stream Merge & Filtering: Mapping: Rendering: Selection of Rectangular Map Projection Nearby Objects with 3D Buildungs Rasterize e.g., Smartphones, PDAs, etc. Static Data: Buildings & Streets Points of Interest Parameter Updates Built-In Operator can be Custom Operator to Dedicated hardware can used here define the Mapping improve performance Exploit already Exploit specialized Universität Stuttgart Process for Data to 5 SFB 627 existing operators ! Rendering Primitives ! hardware is reasonable !
Sample Scenario – Distributed Visualization Pipeline 6 Sources Operators Sinks Visualization-Pipeline User Location Updates Data Stream: Bus, Taxi, and User Locations Client: Image Stream Merge & Filtering: Mapping: Rendering: Selection of Rectangular Map Projection Nearby Objects with 3D Buildungs Rasterize e.g., Smartphones, PDAs, etc. Static Data: Buildings & Streets Points of Interest Parameter Updates Mobile devices still have limited capabilities (energy) Outsource processes Universität Stuttgart SFB 627 if possible!
Requirements for DSPS to Support Non-Trivial Applications? 7 Extensible operator base Complex domain specific operators, may require dedicated hardware Support for structured and unstructured data: “ Tuples ” vs. images , videos Heterogeneous system topology System with wide range of devices in mind – powerful computing servers vs. mobile devices Exploit available hardware for efficient execution Operator has deployment and runtime restrictions Influence deployment and runtime of operators in processing pipelines Universität Stuttgart SFB 627
Adaptation Gap between Applications and DSPS 8 Many different (distributed) context data streams Use distributed DSPS Applications or domains may ask for specific functionality But still depending on general stream processing principles Reuse existing components, “do not reinvent the wheel”! Data Stream Domain-specific Data Processing System Stream Application (DSPS) Adaptation Problem Universität Stuttgart 8 SFB 627
Adaptation Gap between Applications and DSPS 9 Many different (distributed) context data streams Use distributed DSPS Applications or domains may ask for specific functionality But still depending on general stream processing principles Reuse existing components, “do not reinvent the wheel”! Data Stream Domain-specific Data Processing System Stream Application (DSPS) Adapt DSPS to Application 1 2 Adapt Application to DSPS Adaptation Problem Universität Stuttgart 9 9 SFB 627
Adaptation Gap between Applications and DSPS 10 Many different (distributed) context data streams Use distributed DSPS Applications or domains may ask for specific functionality But still depending on general stream processing principles Reuse existing components, “do not reinvent the wheel”! Data Stream Domain-specific Data Domain-specific Data Processing System Stream Processing Stream Application (DSPS) System Adapt DSPS to Application 1 Adapt DSPS to Application 3 Adapt Application to DSPS 2 Adapt Application to DSPS Adaptation Problem Universität Stuttgart 10 10 SFB 627
Constraints Classification 11 System-Relevant Constraints Domain-Relevant Constraints Freely Application-Relevant Modifiable Constraints User-Relevant Constraints Bounded Modifiable Deployment Constraints Runtime Constraints Universität Stuttgart SFB 627
NexusDS Platform Overview 12 Nexus Applications & Extensions Nexus Applications & Extensions Nexus Applications and Extensions Application Operators Application Services Nexus Domain Extensions Nexus Domain Extensions Nexus Domain Extensions Domain Operators Domain Services Core Operators Core Query Service ( CQS ) Nexus Core Operator Repository Operator Execution Service ( ORS ) Service ( OES ) Service Publisher Communication and Monitoring Monitoring Service ( MS ) Service ( SPS ) Universität Stuttgart SFB 627
NexusDS Platform Overview 13 User-Relevant Constraints Application-Relevant Constraints Nexus Applications & Extensions Nexus Applications & Extensions Nexus Applications and Extensions Application Operators Application Services Domain-Relevant Constraints Nexus Domain Extensions Nexus Domain Extensions Nexus Domain Extensions Domain Operators Domain Services Core Operators Core Query Service ( CQS ) Nexus Core Operator Repository Operator Execution Service ( ORS ) Service ( OES ) System-Relevant Constraints Service Publisher Communication and Monitoring Monitoring Service ( MS ) Service ( SPS ) Universität Stuttgart SFB 627
User-Relevant Application-Relevant Classification of Runtime Environments 14 Domain-Relevant System-Relevant Organization scheme of available environments to reduce deployment complexity Access corresponding groups to get available execution environments x86 CPU Architecture Core Group Hardware ATI RV730 GPU Peer Properties Group … Nexus Group Software Windows Visualization Group Arbitrary Domain JVM 1.5 Arbitrary Domain Group Subgroup ..... … Universität Stuttgart SFB 627
User-Relevant Application-Relevant Operator Packages – Overview 15 Domain-Relevant System-Relevant Meta Data Defines operator’s properties Descriptor Requirements Deployment Meta Data Presets Runtime Program Code Program Code Actual operator implementation Dependencies Dependencies Third party libraries Operator Package Universität Stuttgart SFB 627
User-Relevant Application-Relevant Constraint-Based Data Flow Graphs 16 Domain-Relevant System-Relevant Deployment Deployment Name = Mapping NodeID = 007…009 Author = Me OperatorID = 123 Hardware = GPU Data Stream: Bus, Taxi, and User Locations Client: Image Stream Merge & Filtering: Mapping: Rendering: Selection of Rectangular Map Projection Nearby Objects with 3D Buildungs Rasterize e.g., Smartphones, PDAs, etc. Static Data: Buildings & Streets Points of Interest Runtime Runtime Buffer = 10 Resolution = 800x600 Buffer = 100 FPS = 20 Model_Detail = Medium Universität Stuttgart 16 SFB 627
User-Relevant Application-Relevant Domain-Relevant System-Relevant Deployment Model – Visualization Pipeline Sample Scenario 17 Extension Nexus Developer C 1 Application 1 Create the domain Nexus Domain Extension service and operators Constraints! Operator Repository Nexus Core Service (ORS) S S Communication and Monitoring GPU-enabled PC Buildings, Streets, GPU-enabled PC UMPC Client PC (AMD X2 @ 1,6GHz, POIs (AMD X2 @ 1,6GHz, Physical Layer (IntelA110 @ 800MHz, (AMD X2 @ 1,6GHz, 2GB RAM, 2GB RAM, 1GB RAM) 2GB RAM) Nvidia-GF GO 7200) Bus Locations Nvidia-GF GO 7200) Visualization Pipeline Core Query Operator Execution Monitoring Visual Client ( VC ) Service ( VPS ) Service ( CQS ) Service ( OES ) Service ( MS ) Universität Stuttgart SFB 627 Client Application Rendering Step Mapping Step Filtering & Merge Step Mobile Objects & C R M F S ( Sink ) ( Operator ) ( Operator ) ( Operator ) Buildings ( Sources )
Recommend
More recommend
