VAEX: 1 BILLION ROWS, 1 LAPTOP, SERIOUS DATA SCIENCE JOVAN - PowerPoint PPT Presentation

Sep 19, 2022 •312 likes •365 views

VAEX: 1 BILLION ROWS, 1 LAPTOP, SERIOUS DATA SCIENCE JOVAN VELJANOSKI Sr. Data Scientist @ XebiaLabs UNCOMFORTABLY LARGE DATA Working with %i samples without going to the cloud: < 1_000_000 samples ~10_000_000 samples

VAEX: 1 BILLION ROWS, 1 LAPTOP, SERIOUS DATA SCIENCE JOVAN VELJANOSKI Sr. Data Scientist @ XebiaLabs
UNCOMFORTABLY LARGE DATA Working with %i samples without going to the cloud: ➡ < 1_000_000 samples ➡ ~10_000_000 samples ➡ ~100_000_000 samples ➡ ~1_000_000_000 samples ➡ larger datasets
VAEX.IO: WHO ARE WE? Maarten Breddels Jovan Veljanoski Yonatan Alexander Former astrophysicist Former astrophysicist Head of Data Science at BuiltOn � jonathan@xdss.io Freelancer / consultant / data scientist Sr. Data Scientist @ XebiaLabs Core Jupyter-Widgets developer � https://www.linkedin.com/in/xdssio/ Co-founder of vaex.io � jovan.veljanoski@gmail.com Founder of vaex.io Principal author of vaex � https://www.linkedin.com/in/jovanvel/ � maartenbreddels@gmail.com � www.maartenbreddels.com � @maartenbreddels � github.com/maartenbreddels Mario Buikhuizen Freelancer / consultant Front-end / dashboards / widgets specialist � mbuikhuizen@gmail.com
THE NEED FOR VAEX The Gaia satellite: More than 1 billion observations of stars in our Galaxy! How do we work (explore, filter, visualize, analyze) with such data?
LIVE DEMO The Jupyter notebooks presented at the live demo can be found at: https://github.com/vaexio/vaex-talks

Recommend

Vaex: Out of core dataframes for Python Maarten A. Breddels & Jovan Veljanoski Article:

Vaex: Out of core dataframes for Python Maarten A. Breddels & Jovan Veljanoski Article: A&A 618, 2017 / Arxiv 1801.02638 PyParis - Nov 13/2018 Maarten Breddels Ex: astronomer (working on software for big data and visualization: vaex)

1.15k views • 54 slides

The First Billion Rows Alexander Zaitsev and Robert Hodges About Us Robert Hodges - Altinity CEO

ClickHouse Data Warehouse 101 The First Billion Rows Alexander Zaitsev and Robert Hodges About Us Robert Hodges - Altinity CEO Alexander Zaitsev - Altinity CTO 30+ years on DBMS plus Expert in data warehouse with virtualization and security.

578 views • 47 slides

1 Tabulation means putting data into tables. A table is a matrix of data in rows and

1 Tabulation means putting data into tables. A table is a matrix of data in rows and columns, with the rows and the columns having titles. 2 converting the set of numbers into the form of a grouped frequency table. This involves

833 views • 39 slides

Working with tidy data in R: dplyr Fundamental actions on data tables: choose rows

Working with tidy data in R: dplyr Fundamental actions on data tables: choose rows filter() choose columns select() make new columns mutate() arrange rows arrange() calculate summary statistics summarize()

341 views • 14 slides

Working with tidy data in R: tidyverse Fundamental actions on data tables: choose rows

Working with tidy data in R: tidyverse Fundamental actions on data tables: choose rows filter() choose columns select() make new columns mutate() arrange rows arrange() calculate summary statistics

599 views • 20 slides

Welcome to BASH! 1:1 Laptop Deployment Log Into Your Laptop Reaching IT The Student Helpdesk at

Welcome to BASH! 1:1 Laptop Deployment Log Into Your Laptop Reaching IT The Student Helpdesk at BASH is your first resource for any issues that you may have with your 1:1 laptop. We can be reached in a myriad of ways: Stopping by FLEX

391 views • 14 slides

Variable Shara Auty Problem Insurance data contains multiple rows of transactions per policy

Application of the INTO: Host - Variable Shara Auty Problem Insurance data contains multiple rows of transactions per policy holder. It is necessary to summarize the data or to view the data in a different manner. I needed to summarize

226 views • 18 slides

Research Data Managem ent: life beyond the laptop Library and Learning Services conference

Research Data Managem ent: life beyond the laptop Library and Learning Services conference Sunley Conference Centre 15 th June 2012 Miggie Pickton Outline Research data management drivers at Northampton Current RDM practice

510 views • 15 slides

One Laptop per Child One Laptop per Child USENIX: June 22, 2007 Mary Lou Jepsen This works are

One Laptop per Child One Laptop per Child USENIX: June 22, 2007 Mary Lou Jepsen This works are licensed under a Creative Commons Attribution 2.5 License. Acknowledgement Carla Gomez Monroy + all at OLPC + thousands of other organizations and

445 views • 43 slides

Fast and Secure H T Y T O H Laptop Backups F G R E U D B I N with Encrypted

I V N E U R S E I Fast and Secure H T Y T O H Laptop Backups F G R E U D B I N with Encrypted De-duplication Le Zhang <zhang.le@ed.ac.uk> Paul Anderson <dcspaul@ed.ac.uk> LISA 2010 Laptop Backup Options

390 views • 18 slides

Why IoT IoT Domain IoT Data Characteristics Massive data: 20.4 Billion connected Growing

Why IoT IoT Domain IoT Data Characteristics Massive data: 20.4 Billion connected Growing fast, impact on IoT Visualization our life thing by 2020 (Data Volume) Industries are putting e ff ort: Real-time integration of devices

146 views • 3 slides

About My New Laptop I ntroducing: HP OMEN 15-dc0051nr 15.6" Gaming Laptop Computer The Reason

About My New Laptop I ntroducing: HP OMEN 15-dc0051nr 15.6" Gaming Laptop Computer The Reason for Purchasing Performance Price/Value From Microcenter Website Overview of Specs From Microcenter Website More Spec Details: HP Omen 15

349 views • 30 slides

USING OBS STUDIO RECORD YOUR PRESENTATION WITH YOUR VIDEO IN LAPTOP USING OBS STUDIO What do you

RECORD YOUR PRESENTATION WITH YOUR VIDEO IN LAPTOP USING OBS STUDIO RECORD YOUR PRESENTATION WITH YOUR VIDEO IN LAPTOP USING OBS STUDIO What do you need PowerPoint Slide Laptop with Builtin Microphone & Camera You can either use the

452 views • 12 slides

One Laptop Per Child Exploring, Sharing, Reflecting, Learning Presentation at: Mona

One Laptop Per Child Exploring, Sharing, Reflecting, Learning Presentation at: Mona School of Business The University of the West Indies Sameer Verma, Ph.D. Mona, Jamaica sverma@sfsu.edu Outline Introduction Education Laptop

560 views • 36 slides

Modeling Highly Heterogeneous (Large) Data Sets: Towards a Billion Models Robert Grossman

Modeling Highly Heterogeneous (Large) Data Sets: Towards a Billion Models Robert Grossman University of Illinois at Chicago & Open Data Group Traditional Statistics One small data set A few attributes Vector-valued data Data

453 views • 18 slides

Understanding the CARES Act Programs for Employers The CARES Act $250 billion in $260 billion

April 1, 2020 Understanding the CARES Act Programs for Employers The CARES Act $250 billion in $260 billion in cash payments unemployment to families and insurance individuals benefits $377 billion in small business loans $500 billion

599 views • 29 slides

Networked World 1.3 billion users 700 billion minutes/month 280 million users 80%

Social Influence and Information Diffusion Jie Tang Department of Computer Science and Technology Tsinghua University 1 Networked World 1.3 billion users 700 billion minutes/month 280 million users 80% of users are

1.17k views • 82 slides

One Laptop Per Child Academy Program Graduate Students 1 The Organization Identity Global

One Laptop Per Child Academy Program Graduate Students 1 The Organization Identity Global Social Entrepreneurship Non Profit Mission Empower the children of developing countries to learn Highlights 2.7 million 45 countries XO laptop

369 views • 6 slides

Lyon INSA Team SIX BILLION DOLLARS SIX BILLION DOLLARS 24 000 Ferraris ! SIX BILLION

Lyon INSA Team SIX BILLION DOLLARS SIX BILLION DOLLARS 24 000 Ferraris ! SIX BILLION DOLLARS DISINFECTANTS WHY SO MUCH DISINFECTANT ? BIOFILM BIOFILMS : THE INDUSTRIAL ENEMY ! More pumping power / Pressure drops Source: N. Zelver,

624 views • 41 slides

Presentation For Higher Ed Employers Learners

Investor Presentation For Higher Ed Employers Learners Data Analytics People and Services Technology Key Takeaway $21 Billion + $12 Billion $33 Billion

793 views • 23 slides

Data Science: Statistics or Computer Science? 9/15/2015 DATA SCIENCE: STATISTICS OR COMPUTER

Data Science: Statistics or Computer Science? 9/15/2015 DATA SCIENCE: STATISTICS OR COMPUTER SCIENCE? DATA SCIE DA SCIENCE: ST STATIS ISTICS TICS OR OR CO COMPU MPUTER ER SCIEN SCIENCE? IMPLICATIONS FOR STATISTICS EDUCATION IMPLIC ICATION

608 views • 24 slides

Laptop Sales Presentation + eApp = Success First United American currently offers two great

Laptop Sales Presentation + eApp = Success First United American currently offers two great software programs to make Med-Supp sales easier: the Laptop Sales Presentation and the eApp. Both the Laptop Sales Presentation and the eApp can be used

291 views • 6 slides

There are four types:- 1. Desktop Computer 2. Laptop Computer 3. Palmtop Computer 4. Tablet

There are four types:- 1. Desktop Computer 2. Laptop Computer 3. Palmtop Computer 4. Tablet Computer It is big in size. All the parts are attached with wires. Laptop is a small computer which can easily put on your lap. It is

309 views • 11 slides

1:1 Digital Device User Policy K-2 Use your laptop for school only. You are responsible

1:1 Digital Device User Policy K-2 Use your laptop for school only. You are responsible for taking care of your laptop. Level 1 Device Usage Level 1 Device Maintenance Review questions Should you go on

419 views • 12 slides