Duy Hai DOAN @doanduyhai Who Am I ? Duy Hai DOAN Cassandra - PowerPoint PPT Presentation

Apache Zeppelin, the missing GUI for your BigData eco-system Duy Hai DOAN @doanduyhai

Who Am I ? � Duy Hai DOAN Cassandra technical advocate talks, meetups, confs • open-source devs ( Achilles , …) • OSS Cassandra point of contact • ☞ duy_hai.doan@datastax.com ☞ @doanduyhai @doanduyhai 2

Datastax � Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees • Headquarter in San Francisco Bay area • EU headquarter in London , offices in France and Germany • Datastax Enterprise = OSS Cassandra + extra features • @doanduyhai 3

What is Apache Zeppelin ? � Presentation � Architecture � �

Zeppelin Presentation � @doanduyhai 5

Demo https://github.com/doanduyhai/incubator-zeppelin/tree/ZeppelinPresentation @doanduyhai

Zeppelin Architecture � JVM Spark Interpreter Group JVM Spark SparkSQL R Zeppelin Server JVM E Tajo Interpreter S T JVM Zeppelin Engine WebSocket Flink Interpreter Zeppelin Interpreter Factory JVM Cassandra Interpreter @doanduyhai 7

What does Zeppelin provide ? � Front-end & display system for free Generic back-end with REST APIs & WebSocket Pluggable interpreters system Task scheduler (à la CRON) @doanduyhai 8

Zeppelin UI Layout � Notebook � Paragraph � UI elements �

Demo https://github.com/doanduyhai/incubator-zeppelin/tree/ZeppelinPresentation @doanduyhai

Zeppelin Display System �� Raw, Table, HTML � Available graphs � View modes � Dynamic form � Iframe export �

Demo https://github.com/doanduyhai/incubator-zeppelin/tree/ZeppelinPresentation @doanduyhai

Interpreter system �� Core interpreters � Third-parties interpreters � Interpreters conf & usage �

Interpreter processing lifecycle � Receive input commands/data ① as raw text • from form data • Process the input commands/data by the external back-end ② Format the response using Zeppelin display system ③ Send response back to the Zeppelin engine ④ @doanduyhai 14

Core interpreters �� Spark (Spark core, SparkSQL/DataFrame, PySpark) • Spark core = default (or %spark) • SparkSQL = %sql • Shell (%sh) • Markdown (%md) • � AngularJS (%angular) • @doanduyhai 15

Third-parties interpreters � Hive • Phoenix • Tajo • Flink • Ignite • Lens • Cassandra • Geode • PostgreSQL • Kylin • @doanduyhai 16

Interpreter conf & usage https://github.com/doanduyhai/incubator-zeppelin/tree/ZeppelinPresentation @doanduyhai

Writing An Interpreter �� How To � Simple interpreter example (AsciiDoc) � Complex interpreter example (Cassandra) �

Steps to write your own interpreter � Create a class that extends Interpreter base class • Register it in a static block • static { Interpreter.register("MyInterpreterName", MyClassName.class.getName()); } Optionnally define default config params • static { Interpreter.register("MyInterpreterName", MyClassName.class.getName(), new InterpreterPropertyBuilder() .add("property1", "default value", "Description of property1").build() ); } @doanduyhai 19

To register your interpreter as default � Edit the enum ZeppelinConfiguration.ConfVars • Add your interpreter FQCN in the property ZEPPELIN_INTERPRETERS • @doanduyhai 20

To register your interpreter in config files � Create conf/zeppelin-site.xml from conf/zeppelin-site.xml.template • Add your interpreter FQCN in the property zeppelin.interpreters • <property> <name>zeppelin.interpreters</name> <value>org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter, org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter, org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.shell.ShellInterpreter, org.apache.zeppelin.hive.HiveInterpreter, com.me.MyNewInterpreter </value> </property> @doanduyhai 21

Simple AsciiDoc Interpreter � JVM ① ② Raw Raw Zeppelin Server Text Text Block Block JVM Zeppelin Engine AsciiDoc Interpreter ④ ③ Converted HTML To Output HTML @doanduyhai 22

Simple interpreter (AsciiDoc) https://github.com/doanduyhai/incubator-zeppelin/tree/ZeppelinPresentation @doanduyhai

Cassandra Interpreter Architecture � ① ② ③ Raw Raw Text Text Block Block JVM JVM Async CQL statements Zeppelin Cassandra Interpreter Server Cassandra Java Cassandra Driver ⑥ ⑤ Display Render ④ Results as HTML HTML @doanduyhai 24

Cassandra Interpreter Commands � Native CQL statements Options Commands @consistency …; SELECT * FROM …; @retryPolicy …; INSERT INTO …; @fetchSize …; … Schema commands Prepared statements Commands DESCRIBE TABLE …; @prepare …; DESCRIBE KEYSPACE …; @bind …; … @remove_prepared …; Help command HELP; @doanduyhai 25

Complex interpreter (Cassandra) https://github.com/doanduyhai/incubator-zeppelin/tree/ZeppelinPresentation @doanduyhai

Zeppelin future �� Roadmap �

Roadmap & future � More graph options (Map viz ZEPPELIN-157 ) • Helium project, packaging Zeppelin view, logic (code) & resource into • Applications Interpreters packaging re-design • ship & compile core interpreters only • third-parties interpreters can be pulled from repository • which interpreter is core ? Who will maintain ? Community… . • Integrate security (Apache Shiro, ZEPPELIN-53 ) • @doanduyhai 28

Roadmap & future � Out of incubation state to become 1 st class Apache project • @doanduyhai 29

� � � Q & R @doanduyhai

Thank You @doanduyhai duy_hai.doan@datastax.com http://zeppelin.incubator.apache.org/ @doanduyhai

@doanduyhai