Interruptible Tasks: Treating Memory Pressure as Interrupts for - PowerPoint PPT Presentation

Interruptible Tasks: Treating Memory Pressure as Interrupts for Highly Scalable Data-Parallel Programs Lu Fang 1 , Khanh Nguyen 1 , Guoqing(Harry) Xu 1 , Brian Demsky 1 , Shan Lu 2 1 University of California, Irvine 2 University of Chicago SOSP’15, October 7, 2015, Monterey, California, USA Lu Fang et al. Interruptible Tasks SOSP’15, October 7, 2015 1 / 25

Motivation Data-parallel system ◮ Input data are divided into independent partitions ◮ Many popular big data systems Lu Fang et al. Interruptible Tasks SOSP’15, October 7, 2015 2 / 25

Motivation Data-parallel system ◮ Input data are divided into independent partitions ◮ Many popular big data systems � Memory pressure on single nodes Our study ◮ Search “out of memory” and “data parallel” in StackOverflow ◮ We have collected 126 related problems Lu Fang et al. Interruptible Tasks SOSP’15, October 7, 2015 2 / 25

Memory Pressure in the Real World Memory pressure on individual nodes ◮ Executions push heap limit (using managed language) ◮ Data-parallel systems struggle for memory Memory consumption Heap size OutOfMemoryError point Long and useless GC Execution time Lu Fang et al. Interruptible Tasks SOSP’15, October 7, 2015 3 / 25

Memory Pressure in the Real World Memory pressure on individual nodes ◮ Executions push heap limit (using managed language) ◮ Data-parallel systems struggle for memory Memory consumption Heap size OutOfMemoryError point Long and useless GC Execution time CRASH OutOfMemory Error Lu Fang et al. Interruptible Tasks SOSP’15, October 7, 2015 3 / 25

Memory Pressure in the Real World Memory pressure on individual nodes ◮ Executions push heap limit (using managed language) ◮ Data-parallel systems struggle for memory Memory consumption Heap size OutOfMemoryError point Long and useless GC Execution time CRASH OutOfMemory Error SLOW Huge GC effort Lu Fang et al. Interruptible Tasks SOSP’15, October 7, 2015 3 / 25

Root Cause 1: Hot Keys Key-value pairs Lu Fang et al. Interruptible Tasks SOSP’15, October 7, 2015 4 / 25

Root Cause 1: Hot Keys Key-value pairs Popular keys have many associated values Lu Fang et al. Interruptible Tasks SOSP’15, October 7, 2015 4 / 25

Root Cause 1: Hot Keys Key-value pairs Popular keys have many associated values Case study (from StackOverflow) ◮ Process StackOverflow posts ◮ Long and popular posts ◮ Many tasks process long and popular posts Lu Fang et al. Interruptible Tasks SOSP’15, October 7, 2015 4 / 25

Root Cause 2: Large Intermediate Results Temporary data structures Lu Fang et al. Interruptible Tasks SOSP’15, October 7, 2015 5 / 25

Root Cause 2: Large Intermediate Results Temporary data structures Case study (from StackOverflow) ◮ Use NLP library to process customers’ review ◮ Some reviews are quite long ◮ NLP library creates giant temporary data structures for long reviews Lu Fang et al. Interruptible Tasks SOSP’15, October 7, 2015 5 / 25

Existing Solutions More memory? Not really! ◮ Data double in size every two years, [ http://goo.gl/tM92i0 ] ◮ Memory double in size every three years, [ http://goo.gl/50Rrgk ] Lu Fang et al. Interruptible Tasks SOSP’15, October 7, 2015 6 / 25

Existing Solutions More memory? Not really! ◮ Data double in size every two years, [ http://goo.gl/tM92i0 ] ◮ Memory double in size every three years, [ http://goo.gl/50Rrgk ] Application-level solutions ◮ Configuration tuning ◮ Skew fixing Lu Fang et al. Interruptible Tasks SOSP’15, October 7, 2015 6 / 25

Existing Solutions More memory? Not really! ◮ Data double in size every two years, [ http://goo.gl/tM92i0 ] ◮ Memory double in size every three years, [ http://goo.gl/50Rrgk ] Application-level solutions ◮ Configuration tuning ◮ Skew fixing System-level solutions ◮ Cluster-wide resource manager, such as YARN Lu Fang et al. Interruptible Tasks SOSP’15, October 7, 2015 6 / 25

Existing Solutions More memory? Not really! ◮ Data double in size every two years, [ http://goo.gl/tM92i0 ] ◮ Memory double in size every three years, [ http://goo.gl/50Rrgk ] Application-level solutions ◮ Configuration tuning ◮ Skew fixing System-level solutions ◮ Cluster-wide resource manager, such as YARN We need a systematic and effective solution! Lu Fang et al. Interruptible Tasks SOSP’15, October 7, 2015 6 / 25

Our Solution I nterruptible Task : treat memory pressure as interrupt Dynamically change parallelism degree Lu Fang et al. Interruptible Tasks SOSP’15, October 7, 2015 7 / 25

Why Does Our Technique Help Heap size Memory consumption Program starts with multiple tasks Execution time Task Task Task Task Consumed Consumed Consumed Consumed Memory Memory Memory Memory Lu Fang et al. Interruptible Tasks SOSP’15, October 7, 2015 8 / 25

Why Does Our Technique Help Heap size Memory consumption Program pushes heap limit Execution time Task Task Task Task Consumed Consumed Consumed Consumed Memory Memory Memory Memory Lu Fang et al. Interruptible Tasks SOSP’15, October 7, 2015 8 / 25

Why Does Our Technique Help Heap size Memory consumption Long and useless GC Execution time Task Task Task Task Consumed Consumed Consumed Consumed Memory Memory Memory Memory Lu Fang et al. Interruptible Tasks SOSP’15, October 7, 2015 8 / 25

Why Does Our Technique Help Heap size Memory consumption OutOfMemory Error Execution time Task Task Task Task Consumed Consumed Consumed Consumed Memory Memory Memory Memory Lu Fang et al. Interruptible Tasks SOSP’15, October 7, 2015 8 / 25

Why Does Our Technique Help Heap size Memory consumption Long and useless GCs are detected Execution time Task Task Task Task Consumed Consumed Consumed Consumed Memory Memory Memory Memory Lu Fang et al. Interruptible Tasks SOSP’15, October 7, 2015 8 / 25

Why Does Our Technique Help Heap size Memory consumption Long and useless GCs are detected, start interrupting tasks Execution time Killed Killed Task Task Task Task Consumed Consumed Consumed Consumed Memory Memory Memory Memory Lu Fang et al. Interruptible Tasks SOSP’15, October 7, 2015 8 / 25

Why Does Our Technique Help Heap size Memory consumption Release the memory, memory pressure is gone Execution time Consumed Memory Killed Killed Task Task Task Task Local Data Structures Consumed Consumed Consumed Consumed Processed Input Memory Memory Memory Memory Unprocessed Input Output Lu Fang et al. Interruptible Tasks SOSP’15, October 7, 2015 8 / 25

Why Does Our Technique Help Heap size Memory consumption Release the memory, memory pressure is gone Execution time Consumed Memory Killed Killed Task Task Task Task Local Data Structures Released Consumed Consumed Consumed Consumed Processed Input Memory Memory Memory Memory Unprocessed Input Output Lu Fang et al. Interruptible Tasks SOSP’15, October 7, 2015 8 / 25

Why Does Our Technique Help Heap size Memory consumption Release the memory, memory pressure is gone Execution time Consumed Memory Killed Killed Task Task Task Task Local Data Structures Released Consumed Consumed Consumed Consumed Processed Input Released Memory Memory Memory Memory Unprocessed Input Output Lu Fang et al. Interruptible Tasks SOSP’15, October 7, 2015 8 / 25

Why Does Our Technique Help Heap size Memory consumption Release the memory, memory pressure is gone Execution time Consumed Memory Killed Killed Task Task Task Task Local Data Structures Released Consumed Consumed Consumed Consumed Processed Input Released Memory Memory Memory Memory Kept in memory, Unprocessed Input can be serialized Output Lu Fang et al. Interruptible Tasks SOSP’15, October 7, 2015 8 / 25

Why Does Our Technique Help Heap size Memory consumption Release the memory, memory pressure is gone Execution time Consumed Memory Killed Killed Task Task Task Task Local Data Structures Released Consumed Consumed Consumed Consumed Processed Input Released Memory Memory Memory Memory Kept in memory, Unprocessed Input can be serialized Final result: push Output out and released Lu Fang et al. Interruptible Tasks SOSP’15, October 7, 2015 8 / 25

Why Does Our Technique Help Heap size Memory consumption Release the memory, memory pressure is gone Execution time Consumed Memory Killed Killed Task Task Task Task Local Data Structures Released Consumed Consumed Consumed Consumed Processed Input Released Memory Memory Memory Memory Kept in memory, Unprocessed Input can be serialized Final result: push Output out and released Intermediate result: kept in memory, can be serialized Lu Fang et al. Interruptible Tasks SOSP’15, October 7, 2015 8 / 25

Why Does Our Technique Help Heap size Memory consumption Program executes without memory pressure Execution time Killed Killed Task Task Task Task Consumed Consumed Consumed Consumed Memory Memory Memory Memory Lu Fang et al. Interruptible Tasks SOSP’15, October 7, 2015 8 / 25