apache spark in einem raspberry cluster dozenten
play

Apache Spark in einem Raspberry Cluster Dozenten Burkhard - PowerPoint PPT Presentation

Der Funke springt ber: Apache Spark in einem Raspberry Cluster Dozenten Burkhard Hoppenstedt atr Software GmbH & Universitt Ulm burkhard.hoppenstedt@uni-ulm.de Nicolas Kuhaupt Universitt Ulm nicolas.kuhaupt@uni-ulm.de Gliederung


  1. Der Funke springt über: Apache Spark in einem Raspberry Cluster

  2. Dozenten Burkhard Hoppenstedt atr Software GmbH & Universität Ulm burkhard.hoppenstedt@uni-ulm.de Nicolas Kuhaupt Universität Ulm nicolas.kuhaupt@uni-ulm.de

  3. Gliederung Architektur Aufbau Code Snippets Spark Cluster Geschichte Konfigurierung Ergebnisse Algorithmen-Skalierung Clusterleistung

  4. Spark Geschichte 2002 MapReduce Forschung (Google) 2004 MapReduce Paper 2006 Hadoop sortiert 1.8TB in 47.9 h 2008 Konferenz: Hadoop Summit 2010 2010 Spark Paper 2012 Apache Hadoop 1.0 2014 Apache Spark top-level Projekt 2014 Spark sortiert 100TB in 23 min, Hadoop in 72 min

  5. Powered By Yahoo! eBay Inc. Spark Amazon IBM

  6. Dateiformate Format Strukturiert Textdateien Nein JSON Halb CSV Ja SequenceFile Ja Protocol buffers Ja Object files Ja

  7. Dateiformate Format Strukturiert Sequence File Header Textdateien Nein 3 Byte (SEQ) + 1 Byte Version JSON Halb Key Class Name CSV Ja Value Class Name isCompressed SequenceFile Ja IsBlockCompressed Protocol buffers Ja Class Name Object files Ja Meta Data SyncMarker

  8. Dateiformate Format Strukturiert message Person { required string name = 1; required int32 id = 2; Textdateien Nein optional string email = 3; } JSON Halb Person john = Person.newBuilder() .setId(1234) CSV Ja .setName("John Doe") .setEmail("jdoe@example.com") .build(); SequenceFile Ja output = new FileOutputStream(args[0]); john.writeTo(output); Protocol buffers Ja Person john; fstream input(argv[1], Object files Ja ios::in | ios::binary); john.ParseFromIstream(&input); id = john.id(); name = john.name(); email = john.email();

  9. Dateiformate Format Strukturiert Java Serialisierung Textdateien Nein • Nicht standardisiert mit Hadoop-Output JSON Halb • Langsam • Schnelle Verarbeitung CSV Ja beliebiger Objekte SequenceFile Ja Protocol buffers Ja Object files Ja

  10. Word count //Python Example rdd = textFile (“… path …“) words = rdd.flatMap(lambda x: x.split (“ “)) result = words.map(lambda x: (x,1)).reduceByKey(lambda x, y: x + y)

  11. Amazon JSON {'asin': '0078764343', 'description': 'Brand new sealed!', 'price': 37.98, 'imUrl': 'http://ecx.images- amazon.com/images/I/513h6dPbwLL._SY300_.jpg', 'related': {'also_bought': ['B000TI836G', 'B003Q53VZC', 'B00EFFW0HC', 'B003VWGBC0', 'B003O6G5TW', 'B0037LTTRO', 'B002I098JE', 'B008OQTS0U', 'B005EVEODY', 'B008B3AVNE', 'B000PE0HBS', 'B00354NAYG', 'B0050SYPV2', 'B00503E8S2', 'B0050SY77E', 'B0022TNO7S', 'B0056WJA30', 'B0023CBY4E', 'B002SRSQ72', 'B005EZ5GQY', 'B004XACA60', 'B00273Z9WM', 'B004HX1QFY', 'B002I0K50U'], 'bought_together': ['B002I098JE'], 'buy_after_viewing': ['B0050SY5BM', 'B000TI836G', 'B0037LTTRO', 'B002I098JE']}, 'salesRank': {'Video Games': 28655}, 'categories': [['Video Games', 'Xbox 360', 'Games']]}

  12. FP Growth Customer Also bought John Doe Banana, Unicorn, Sword Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Banana, Unicorn, Steam Boat Stephen Hawking Banana, Sword Nyan Cat Unicorn, Sword Cpt. Jack Sparrow Banana, Sword Steve Jobs Banana, Unicorn, Sword, Razor Mr. Nobody Banana, Unicorn, Sword Banana: 6 | Unicorn: 7 | Sword : 7 | Steam Boat: 2 | Razor: 1

  13. FP Growth - resort Customer Also bought John Doe Unicorn, Sword, Banana Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Hawking Sword, Banana, Nyan Cat Unicorn, Sword Cpt. Jack Sparrow Sword, Banana Steve Jobs Unicorn, Sword, Banana, Razor Mr. Nobody Unicorn, Sword, Banana Banana: 6 | Unicorn: 7 | Sword : 7 | Steam Boat: 2 | Razor: 1

  14. Customer Also bought John Doe Unicorn, Sword, Banana { } Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Sword, Banana, Hawking Nyan Cat Unicorn, Sword Cpt. Jack Sword, Banana Sparrow Steve Jobs Unicorn, Sword, Banana, Razor Mr. Nobody Unicorn, Sword, Banana

  15. Customer Also bought John Doe Unicorn, Sword, Banana { } Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Sword, Banana, Hawking Unicorn: Nyan Cat Unicorn, Sword 1 Cpt. Jack Sword, Banana Sparrow Sword: Steve Jobs Unicorn, Sword, Banana, Razor 1 Mr. Nobody Unicorn, Sword, Banana Banana: 1

  16. Customer Also bought John Doe Unicorn, Sword, Banana { } Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Sword, Banana, Hawking Unicorn: Nyan Cat Unicorn, Sword 2 Cpt. Jack Sword, Banana Sparrow Sword: Steve Jobs Unicorn, Sword, Banana, Razor 1 Mr. Nobody Unicorn, Sword, Banana Steam Boat:1 Banana: 1

  17. Customer Also bought John Doe Unicorn, Sword, Banana { } Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Sword, Banana, Hawking Unicorn: Nyan Cat Unicorn, Sword 3 Cpt. Jack Sword, Banana Sparrow Sword: Steve Jobs Unicorn, Sword, Banana, Razor 2 Mr. Nobody Unicorn, Sword, Banana Steam Boat:1 Banana: 1

  18. Customer Also bought John Doe Unicorn, Sword, Banana { } Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Sword, Banana, Hawking Unicorn: Nyan Cat Unicorn, Sword 4 Cpt. Jack Sword, Banana Banana: Sparrow 1 Sword: Steve Jobs Unicorn, Sword, Banana, Razor 2 Mr. Nobody Unicorn, Sword, Banana Steam Steam Boat:1 Boat:1 Banana: 1

  19. Customer Also bought John Doe Unicorn, Sword, Banana { } Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Sword, Banana, Hawking Unicorn: Nyan Cat Unicorn, Sword 4 Sword: Cpt. Jack Sword, Banana Banana: Sparrow 1 1 Sword: Steve Jobs Unicorn, Sword, Banana, Razor 2 Mr. Nobody Unicorn, Sword, Banana Steam Steam Banana: Boat:1 Boat:1 1 Banana: 1

  20. Customer Also bought John Doe Unicorn, Sword, Banana { } Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Sword, Banana, Hawking Unicorn: Nyan Cat Unicorn, Sword 5 Sword: Cpt. Jack Sword, Banana Banana: Sparrow 1 1 Sword: Steve Jobs Unicorn, Sword, Banana, Razor 3 Mr. Nobody Unicorn, Sword, Banana Steam Steam Banana: Boat:1 Boat:1 1 Banana: 1

  21. Customer Also bought John Doe Unicorn, Sword, Banana { } Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Sword, Banana, Hawking Unicorn: Nyan Cat Unicorn, Sword 5 Sword: Cpt. Jack Sword, Banana Banana: Sparrow 1 2 Sword: Steve Jobs Unicorn, Sword, Banana, Razor 3 Mr. Nobody Unicorn, Sword, Banana Steam Steam Banana: Boat:1 Boat:1 2 Banana: 1

  22. Customer Also bought John Doe Unicorn, Sword, Banana { } Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Sword, Banana, Hawking Unicorn: Nyan Cat Unicorn, Sword 6 Sword: Cpt. Jack Sword, Banana Banana: Sparrow 1 2 Sword: Steve Jobs Unicorn, Sword, Banana, Razor 4 Mr. Nobody Unicorn, Sword, Banana Steam Steam Banana: Boat:1 Boat:1 2 Banana: 2 Razor: 1

  23. Customer Also bought John Doe Unicorn, Sword, Banana { } Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Sword, Banana, Hawking Unicorn: Nyan Cat Unicorn, Sword 7 Sword: Cpt. Jack Sword, Banana Banana: Sparrow 1 2 Sword: Steve Jobs Unicorn, Sword, Banana, Razor 5 Mr. Nobody Unicorn, Sword, Banana Steam Steam Banana: Boat:1 Boat:1 2 Banana: 3 Razor: 1

  24. Customer Also bought John Doe Unicorn, Sword, Banana { } Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Sword, Banana, Hawking Unicorn: Nyan Cat Unicorn, Sword 7 Sword: Cpt. Jack Sword, Banana Banana: Sparrow 1 2 Sword: Steve Jobs Unicorn, Sword, Banana, Razor 5 Mr. Nobody Unicorn, Sword, Banana Steam Steam Banana: Boat:1 Boat:1 2 Banana: 3 Razor: 1

  25. Customer Also bought John Doe Unicorn, Sword, Banana { } Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Sword, Banana, Hawking Unicorn: Nyan Cat Unicorn, Sword 7 Sword: Cpt. Jack Sword, Banana Banana: Sparrow 1 2 Sword: Steve Jobs Unicorn, Sword, Banana, Razor 5 Mr. Nobody Unicorn, Sword, Banana Steam Steam Banana: Boat:1 Boat:1 2 Banana: 3 Razor: 1 Frequent Patterns Banana

  26. Customer Also bought John Doe Unicorn, Sword, Banana { } Mickey Mouse Unicorn, Steam Boat Donald Trump Unicorn, Sword John Cena Unicorn, Banana, Steam Boat Stephen Sword, Banana, Hawking Unicorn: Nyan Cat Unicorn, Sword 7 Sword: Cpt. Jack Sword, Banana Banana: Sparrow 1 2 Sword: Steve Jobs Unicorn, Sword, Banana, Razor 5 Mr. Nobody Unicorn, Sword, Banana Steam Steam Banana: Boat:1 Boat:1 2 Banana: 3 Razor: 1 Frequent Patterns Banana {Unicorn, Sword, Banana: 3}

Recommend


More recommend