building dev tools at the right level of abstraction
play

Building dev tools at the right level of abstraction Ben Davis CTO - PowerPoint PPT Presentation

Sep 28, 2023 •420 likes •728 views

Building dev tools at the right level of abstraction Ben Davis CTO @BenCDavis ben@gatherdata.co The data engineering industry is very fragmented. Gather is a data integration tool for developers . It makes it really easy to build integration

  1. Building dev tools at the right level of abstraction Ben Davis CTO @BenCDavis · ben@gatherdata.co

  2. The data engineering industry is very fragmented.

  3. Gather is a data integration tool for developers . It makes it really easy to build integration pipelines that push and pull data from various SaaS APIs.

  4. Gather is a data integration tool for developers . It makes it really easy to build integration pipelines that push and pull data from various SaaS APIs. Your App

  5. But how did we get here?

  6. Initial motivating problem: building data pipelines is hard. We should fix it.

  7. Data pipelining is conceptual. It breaks down into many use-cases .

  8. Data pipelining is conceptual. It breaks down into many use-cases . Batch Streaming ETL …

  9. People will ask “Can I use it for this?” or “Oh I can you like this right?"

  10. No framework to answer those questions

  11. “Abstraction is amplification of the essential and elimination of the irrelevant .” –K.K Aggarwal

  12. Building companies and products requires choosing a set of abstractions

  13. The question is what use-cases are you abstracting away in your product? How many are there?

  14. Inspired by Cheng Lou (Facebook)

  15. Inspired by Cheng Lou (Facebook) EC2

  16. Inspired by Cheng Lou (Facebook) HEROKU EC2

  17. No man's land No man's land Abstraction level Multiple use-cases Concrete use-case

  18. DATA PIPELINES BATCH PROCESSING DATA COLLECTION AND INTEGRATION RECSYS REPORTS DATA PREP SAAS & API INTEGRATION IOT SENSORS CLICKSTREAM CUSTOMER SERVICE SYNC PAYMENT DATA ETC

  19. Value prop • No deployment from user • Not writing api adapters and glue code • Off the self connectors • Pre-built authentication • Not writing tests and worrying about fragile code

  20. The product should abstract away the complexities of those specific use-cases while maintaining flexibility and expressibility

  21. Options for the product

  22. Options for the product • UI for specific use-cases

  23. Options for the product • UI for specific use-cases • Python SDK

  24. Options for the product • UI for specific use-cases • Python SDK • Kubernetes-like declarative data flow?

  25. Kubernetes is the right inspiration because it operates at the same level of abstraction

  28. Conclusion • Starting at too higher level of abstraction • Building the tree is hard • Building a product that is misaligned with where you’ve position yourself on that tree

  29. THANKS FOR TALK TO ME. PLEASE LISTENING ben@gatherdata.co

Recommend

The Problem of Temporal Abstraction How do we connect the high level to the low-level? "

The Problem of Temporal Abstraction How do we connect the high level to the low-level? "

The Problem of Temporal Abstraction How do we connect the high level to the low-level? " the human level to the physical level? " the decide level to the action level? MDPs are great, search is great,

676 views • 31 slides
Are objects the right level of abstraction to enable the convergence between HPC and Big Data at

Are objects the right level of abstraction to enable the convergence between HPC and Big Data at

Are objects the right level of abstraction to enable the convergence between HPC and Big Data at storage level? Pierre Matri*, Alexandru Costan , Gabriel Antoniu , Jess Montes*, Mara S. Prez *, Luc Boug * Universidad

822 views • 44 slides
Models as the Basis for Visual Representation Level of abstraction: 1. Low: realistic 3D

Models as the Basis for Visual Representation Level of abstraction: 1. Low: realistic 3D

Models as the Basis for Visual Representation Level of abstraction: 1. Low: realistic 3D visualisation 2. High: insight at high abstraction level Link visualisation to model: 1. entity relationships (structure) 2. entity attributes

1.1k views • 7 slides
NOW Handout Page 1 CS258 S99 1 Relationship between Perspectives Back to Basics Parallel

NOW Handout Page 1 CS258 S99 1 Relationship between Perspectives Back to Basics Parallel

Meta-message today powerful high-level abstraction boils down to specific, simple low-level mechanisms each detail has significant implications Shared Memory Multiprocessors Topic: THE MEMORY ABSTRACTION sequence of reads and

388 views • 10 slides
Abstraction refinement and plan revision for control synthesis under high level specifications

Abstraction refinement and plan revision for control synthesis under high level specifications

Abstraction refinement and plan revision for control synthesis under high level specifications Pierre-Jean Meyer Dimos V. Dimarogonas KTH, Royal Institute of Technology July 12 th 2017 Outline Abstraction-based synthesis Global framework

603 views • 34 slides
Abstraction and OOP Tiziana Ligorio 1 Todays Plan Announcements Recap Abstraction OOP

Abstraction and OOP Tiziana Ligorio 1 Todays Plan Announcements Recap Abstraction OOP

Abstraction and OOP Tiziana Ligorio 1 Todays Plan Announcements Recap Abstraction OOP 2 Recap Minimize software size and interactions Simplify complex program to manageable level Break down into smaller problems Isolate

650 views • 54 slides
On the Soundness of Behavioural Abstraction in Hybrid Systems SIM@SYST.Level, 19 th of October,

On the Soundness of Behavioural Abstraction in Hybrid Systems SIM@SYST.Level, 19 th of October,

On the Soundness of Behavioural Abstraction in Hybrid Systems SIM@SYST.Level, 19 th of October, 2014, Cargse, France Simon Bliudze and Sbastien Furic Towards On the Soundness of Behavioural Abstraction in Hybrid Systems SIM@SYST.Level, 19

673 views • 40 slides
Point, Line, & Plane 1 Abstraction Abstraction is the act of considering something as a

Point, Line, & Plane 1 Abstraction Abstraction is the act of considering something as a

Point, Line, & Plane 1 Abstraction Abstraction is the act of considering something as a general quality or characteristic, apart from concrete realities, specific objects, or actual instances. 2 Abstraction Abstraction is

478 views • 21 slides
Cyclone: Safe Programming at the C Level of Abstraction Dan Grossman Cornell University Joint

Cyclone: Safe Programming at the C Level of Abstraction Dan Grossman Cornell University Joint

Cyclone: Safe Programming at the C Level of Abstraction Dan Grossman Cornell University Joint work with: Trevor Jim (AT&T), Greg Morrisett, Michael Hicks (Maryland), James Cheney, Yanling Wang A safe C-level language Cyclone is a

860 views • 83 slides
Predicate Abstraction with SATABS Existential Abstraction Predicate Abstraction for Software

Predicate Abstraction with SATABS Existential Abstraction Predicate Abstraction for Software

Outline Introduction Predicate Abstraction with SATABS Existential Abstraction Predicate Abstraction for Software Counterexample-Guided Abstraction Refinement Computing Existential Abstractions of Programs Version 1.0, 2010 Checking the

429 views • 12 slides
Flexible Hardware Design at Flexible Hardware Design at Low Levels of Abstraction Low Levels of

Flexible Hardware Design at Flexible Hardware Design at Low Levels of Abstraction Low Levels of

Flexible Hardware Design at Flexible Hardware Design at Low Levels of Abstraction Low Levels of Abstraction Emil Axelsson Hardware Description and Verification May 2009 Why low-level? Why low-level? Related question: Why is some software

908 views • 51 slides
From High to Low Level and Vice-Versa: A New Language for the Translation between Abstraction

From High to Low Level and Vice-Versa: A New Language for the Translation between Abstraction

From High to Low Level and Vice-Versa: A New Language for the Translation between Abstraction Levels in Robot Control Architectures Jos Carlos Gonzlez , Javier Garca, Raquel Fuentetaja, ngel Garca-Olaya and Fernando Fernndez Planning

637 views • 11 slides
CUB: A pattern of collective software design, abstraction, and reuse for kernel-level

CUB: A pattern of collective software design, abstraction, and reuse for kernel-level

CUB: A pattern of collective software design, abstraction, and reuse for kernel-level programming DUANE MERRILL, PH.D. NVIDIA RESEARCH What is CUB ? 1. A design model for collective kernel-level primitives How to make reusable software

1.14k views • 72 slides
CS 3330 introduction 1 layers of abstraction Higher-level language: C x += y

CS 3330 introduction 1 layers of abstraction Higher-level language: C x += y

CS 3330 introduction 1 layers of abstraction Higher-level language: C x += y Assembly: X86-64 add %rbx, %rax Machine code: Y86 6 0 03 SIXTEEN Hardware Design Language: HCLRS Gates / Transistors / Wires / Registers 2 layers of

1.18k views • 114 slides
Measurement Tools: RDM Common network abstraction Source: http://commons.wikimedia.org/wiki/

Measurement Tools: RDM Common network abstraction Source: http://commons.wikimedia.org/wiki/

Measurement Tools: RDM Common network abstraction Source: http://commons.wikimedia.org/wiki/ File%3ALautsprecherkabel_Makro_nah.jpg in models: Copper Wires Delay = d0 + PacketSize / Capacity (Access) Networks are stateful at layer 1 and/or

269 views • 6 slides
Instruction Set Design Instruction Set Architecture: to what purpose? ISA provides the level

Instruction Set Design Instruction Set Architecture: to what purpose? ISA provides the level

Instruction Set Design Instruction Set Architecture: to what purpose? ISA provides the level of abstraction between the software and the hardware One of the most important abstraction in CS Its narrow, well-defined, and mostly

401 views • 39 slides
Data Abstraction Announcements Data Abstraction Data Abstraction Programmers Compound

Data Abstraction Announcements Data Abstraction Data Abstraction Programmers Compound

Data Abstraction Announcements Data Abstraction Data Abstraction Programmers Compound values combine other values together All A date: a year, a month, and a day A geographic position: latitude and longitude Data abstraction

430 views • 19 slides
MultiNyx: A Multi-Level Abstraction Framework for Systematic Analysis of Hypervisors Pedro

MultiNyx: A Multi-Level Abstraction Framework for Systematic Analysis of Hypervisors Pedro

MultiNyx: A Multi-Level Abstraction Framework for Systematic Analysis of Hypervisors Pedro Fonseca, Xi Wang, Arvind Krishnamurthy Hypervisor correctness is critical Hypervisors need to virtualize correctly all the architecture details

388 views • 19 slides
Data Visibility Abstraction @stuartsierra Being abstract is something profoundly different

Data Visibility Abstraction @stuartsierra Being abstract is something profoundly different

Data Visibility Abstraction @stuartsierra Being abstract is something profoundly different from being vague... The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise.

824 views • 55 slides
DaMPL What is DaMPL? "Data Manipulation Programming Language High-level

DaMPL What is DaMPL? "Data Manipulation Programming Language High-level

DaMPL What is DaMPL? "Data Manipulation Programming Language High-level abstraction language Features tools to read, process and write data Translator generates efficient C code Quick-start guide Variables /* No need to

433 views • 19 slides
Functions Function Smaller, simpler, subcomponent of program Provides abstraction hide

Functions Function Smaller, simpler, subcomponent of program Provides abstraction hide

Chapter 14 Functions Function Smaller, simpler, subcomponent of program Provides abstraction hide low-level details give high-level structure to program, easier to understand overall program flow enables separable, independent

353 views • 20 slides
Tools and Behavioral Abstraction A Direction for Software Engineering K. Rustan M. Leino

Tools and Behavioral Abstraction A Direction for Software Engineering K. Rustan M. Leino

Tools and Behavioral Abstraction A Direction for Software Engineering K. Rustan M. Leino Research in Software Engineering (RiSE) Microsoft Research, Redmond Future of Software Engineering symposium ETH, Zurich, Switzerland 23 November 2010

244 views • 23 slides
Functional abstraction Readings: HtDP , sections 21-24. Language level: Intermediate Student With

Functional abstraction Readings: HtDP , sections 21-24. Language level: Intermediate Student With

Functional abstraction Readings: HtDP , sections 21-24. Language level: Intermediate Student With Lambda Topics: Anonymous functions Syntax & semantics Example: Transforming strings Abstracting: Map Abstracting: Foldr Abstracting: Foldl

502 views • 27 slides
Functional abstraction Readings: HtDP , sections 21-24. Language level: Intermediate Student With

Functional abstraction Readings: HtDP , sections 21-24. Language level: Intermediate Student With

Functional abstraction Readings: HtDP , sections 21-24. Language level: Intermediate Student With Lambda Topics: Anonymous functions Syntax & semantics Example: Transforming strings Abstracting: Map Abstracting: Foldr Abstracting: Foldl

1.11k views • 81 slides

More recommend