Post-Debugging in Large Scale Analytic Systems Eduard Bergen M.Sc. - PowerPoint PPT Presentation

Post-Debugging in Large Scale Analytic Systems Eduard Bergen M.Sc. BTW 2017 BigBIA Workshop

Context: Distributed systems log file analysis Distributed system A lot of log file information 2 BTW2017 BigBIA Workshop – Eduard Bergen – 06.03.2017

Exception domain Unchecked exceptions Source code example with exception 1 class SubText { Arithmetic, ArrayStore, 2 public static void main (String[] args) { ClassCast, IllegalArgument: 3 String s = “SubText”; [IllegalThreadState, NumberFormat] 4 System.out.println(s.substring(7,4)); IllegalMonitorState, IndexOutOfBound, 5 } NegativeArraySize, 6 } NullPointer, Security 3 BTW2017 BigBIA Workshop – Eduard Bergen – 06.03.2017

Cluster program execution 4 BTW2017 BigBIA Workshop – Eduard Bergen – 06.03.2017

Dataflow system on the JVM #1 User defined function public class Tokenizer extends RichtFlatMapFunction<String, Tuple2<String, Integer>> { @Overrride public void flatMap(String value, Collector<Tuple2<String, Integer>> out) { String[] subText = value.split(“\\W+”); for (String s: subText) { out.collect( new Tuple2<String, Integer>( s.substring( 7,4 ) )); } } } 5 BTW2017 BigBIA Workshop – Eduard Bergen – 06.03.2017

Dataflow system on the JVM #2 Job transformation 6 BTW2017 BigBIA Workshop – Eduard Bergen – 06.03.2017

JVM Bytecode instrumentation Operating system JVM Native Java agent attachment via operating system process id 7 BTW2017 BigBIA Workshop – Eduard Bergen – 06.03.2017

Current prototype development It is possible to record and Worker 1 S M R K replay manually, but current work is to generalize and F F Worker 2 M K S automate the manual process. F R (2) (1) (3) Linux process F 8 BTW2017 BigBIA Workshop – Eduard Bergen – 06.03.2017

Programming model 1. Create bytecode patch of transformed classes with invocations 2. Deploy desired patch to remote machines running JVM 3. Instrument bytecode via JVMTI using only ClassFileLoadHook with native client out-of-process 4. Copy serialized records to local machine 5. Start replay from deserialized records locally 9 BTW2017 BigBIA Workshop – Eduard Bergen – 06.03.2017

Considerations of problems during instrumentation 1. Consider of the order of loaded classes. 2. Use only wait-free code to avoid deadlocks. 3. Consider of state corruption of non-reentrant instrumentation code. 4. Beware of uninitialized objects usage in constructors. 10 BTW2017 BigBIA Workshop – Eduard Bergen – 06.03.2017

Conclusion 1. Instrumented code runs at full speed 1 , since inserted code (patch) is standard bytecode. 2. Transformed classes can be returned to their original state during the instrumented execution. 3. Analysis process takes place out of base process (JVM). 4. Instrumentation contains all files for execution. No waiting until transformation classes ready. 1 https://docs.oracle.com/javase/7/docs/platform/jvmti/jvmti.html#bci 11 BTW2017 BigBIA Workshop – Eduard Bergen – 06.03.2017

Future work 1. Declaration of Patches 2. Validation and provision of patches 3. Automate distribution of patches 12 BTW2017 BigBIA Workshop – Eduard Bergen – 06.03.2017

Related work Zeller, A (1999). Yesterday, My Program Worked. Today, It Does Not. Why? Proceedings of the 7th European Software Engineering Conference Held Jointly with the 7th ACM SIGSOFT International Symposium on Foundations of Software Engineering series, 253-267. Dave, A., Zaharia, M., & Stoica, I. (2013). Arthur: Rich Post-Facto Debugging for Production Analytics Applications. Technical report, University of California, Berkeley. Sen, T., & Mall, R. (2016). Extracting finite state representation of java programs. Software & Systems Modeling, Springer Berlin Heidelberg, 497–511. Gulzar, M., Interlandi, M., Yoo, S., Tetali, S., Condie, T., Millstein, T., & Kim, M. (2016). BigDebug: debugging primitives for interactive big data processing in spark. Proceedings of the 38th International Conference on Software Engineering, 784-795. 13 BTW2017 BigBIA Workshop – Eduard Bergen – 06.03.2017

Thank you for listening 14 BTW2017 BigBIA Workshop – Eduard Bergen – 06.03.2017