what the heck is time series data
play

What the heck is time-series data (and why do I need a time-series - PowerPoint PPT Presentation

Feb 24, 2023 •175 likes •540 views

What the heck is time-series data (and why do I need a time-series database?) Ajay Kulkarni | Co-founder/CEO | ajay@timescale.com Fastest growing database category Source: DB Engines In this talk 1. What is time-series data? (hint: its

  1. What the heck is time-series data (and why do I need a time-series database?) Ajay Kulkarni | Co-founder/CEO | ajay@timescale.com

  2. Fastest growing database category Source: DB Engines

  3. In this talk 1. What is time-series data? 
 (hint: it’s not what you think) 2. Why do I need a time-series database ? 3. Is this just a fad?

  4. What is time-series data ?

  5. Q: Metrics and Logging? CPU, free memory, gc pauses, error reports, application instrumentation, etc. ✓

  6. Q: Financial data? Stock tick stream, payment records, transaction records ✓

  7. Q: Event data? Clickstreams, application events, outages, errors, system status ✓

  8. Q: IoT data? Sensor data, machine data, industrial monitoring, smart home, wearables ✓

  9. Q: Other data? Logistics tracking, environmental monitoring ✓

  10. A: All of the above

  11. So what is time-series data ?

  12. Time-series data has 3 characteristics • Capturing and analyzing 1. Time-centric data measurements/events over time. • Workloads generally 2. Primarily write new data. Rarely INSERTS update. • Data generally written to 3. Writes to recent most recent time interval interval (although delays possible).

  13. How is this different than having a time field? Treat changes as inserts, not overwrites.

  14. You can do more with time-series data PAST PRESEN FUTURE T • Analyze historical trends. • Real-time monitoring • Identify and fix problems before they • Look at the state of the • Troubleshoot occur, reducing system at any point in problems as they downtime. time. occur

  15. What does time-series data look like? (hint: it’s not what you think)

  16. What you have been told Name CPU Tags Host=Name,Region=West Data 1990-01-01 01:02:00 70 
 1990-01-01 01:03:00 71 1990-01-01 01:04:00 72 1990-01-01 01:04:00 73 1990-01-01 01:04:00 100

  17. What you have been told Name CPU FreeMem Tags Host=Name,Region=West Host=Name,Region=West Data 1990-01-01 01:02:00 70 
 1990-01-01 01:02:00 800M 
 1990-01-01 01:03:00 71 1990-01-01 01:03:00 600M 1990-01-01 01:04:00 72 1990-01-01 01:04:00 400M 1990-01-01 01:04:00 73 1990-01-01 01:04:00 200M 2 time-series? 1990-01-01 01:04:00 1990-01-01 01:04:00 0 100

  18. This is wrong

  19. Time-series data has a richer structure Tags Host=Name,Region=Wes t CPU MemFree Temp Data 70 800M 80 1990-01-01 01:02:00 
 71 600M 81 1990-01-01 01:03:00 72 400M 82 1990-01-01 01:04:00 73 200M 83 1990-01-01 01:04:00 100 0 120 1990-01-01 01:04:00

  20. Fewer queries Tags Host=Name,Region=Wes t CPU MemFree Temp Data 70 800M 80 1990-01-01 01:02:00 
 71 600M 81 1990-01-01 01:03:00 72 400M 82 1990-01-01 01:04:00 73 200M 83 1990-01-01 01:04:00 100 0 120 1990-01-01 01:04:00 select * where time = x

  21. Complex filters Tags Host=Name,Region=Wes t CPU MemFree Temp Data 70 800M 80 1990-01-01 01:02:00 
 71 600M 81 1990-01-01 01:03:00 72 400M 82 1990-01-01 01:04:00 73 200M 83 1990-01-01 01:04:00 100 0 120 1990-01-01 01:04:00 where temp > 100

  22. Complex aggregates Tags Host=Name,Region=Wes t CPU MemFree Temp Data 70 800M 80 1990-01-01 01:02:00 
 71 600M 81 1990-01-01 01:03:00 72 400M 82 1990-01-01 01:04:00 73 200M 83 1990-01-01 01:04:00 100 0 120 1990-01-01 01:04:00 avg(mem_free) group by (cpu/10)

  23. Correlations Tags Host=Name,Region=Wes t CPU MemFree Temp Data 70 800M 80 1990-01-01 01:02:00 
 71 600M 81 1990-01-01 01:03:00 72 400M 82 1990-01-01 01:04:00 73 200M 83 1990-01-01 01:04:00 100 0 120 1990-01-01 01:04:00 how does temperature correlate with mem_free?

  24. Leverage relations CPU Host Region Data 1990-01-01 01:02:00 
 70 1 91 1990-01-01 01:03:00 71 2 92 1990-01-01 01:04:00 72 3 93 1990-01-01 01:04:00 73 4 94 1990-01-01 01:04:00 100 5 95 Region stored in separate host metadata table

  25. How to store time-series data

  26. Can’t I use a “normal” database? You can, and some people do 42 % Non time-series 58 % Purpose-built for time-series 0 % 15 % 30 % 45 % 60 % Source: Percona

  27. Golden age of time-series databases

  28. Why do I need a specialized time-series database?

  29. Problem: Time-series data piles up very quickly data collected per hour by 25 GB connected cars (McKinsey) “Our Boeing 787s generate half a terabyte of data per flight” - Virgin Atlantic IT director

  30. Time-series databases introduce efficiencies by treating time as a first-class citizen.

  31. OLTP Time Series ✓ Primarily INSERTs ✗ Primarily UPDATEs ✓ Writes to recent time interval ✗ Writes randomly distributed ✓ Writes associated with a ✗ Transactions to multiple primary keys timestamp and primary key

  32. Time-series databases introduce efficiencies 1. Better write rates to handle ingest scale. 2. Query performance, even at scale. 3. Ease of use via common functions (e.g., interpolation)

  33. Is this just a fad? (No.)

  34. 
 Why time-series databases will continue to be popular Operational needs Business Tech trends needs • Managing increasingly • Constant need to make • Cheaper storage, complex systems better data-driven faster processors, requires: 
 decision faster. more bandwidth • More sources of data: • Better resources: real-time monitoring, new devices, old devices cloud computing, troubleshooting, 
 coming online, new data analysis tools better prediction. systems.

  35. Crazy idea: Is all data time-series data?

  36. https://github.com/timescale/timescaledb

Recommend

Time Series Representations for Better Data Mining What can we do with time series data?

Time Series Representations for Better Data Mining What can we do with time series data?

Time Series Representations for Better Data Mining What can we do with time series data? Clustering Anomaly (outlier) detection Forecasting What are the problems with time series data? Noise Concept-drift (trend-shift

122 views • 11 slides
Why do you care? Time-series data is all over the place. Time-Series Data Kaitlin Duck

Why do you care? Time-series data is all over the place. Time-Series Data Kaitlin Duck

Why do you care? Time-series data is all over the place. Time-Series Data Kaitlin Duck Sherwood CS 533c What is Time-Series Data? What is Time-Series Data? Lines. Usually periodic Source: van Wijk and van Selow, Cluster and

85 views • 4 slides
Introduction to Time Series Heino Bohn Nielsen 1 of 15 Outline (1) What is a time series? (2)

Introduction to Time Series Heino Bohn Nielsen 1 of 15 Outline (1) What is a time series? (2)

Econometrics 2 Fall 2005 Introduction to Time Series Heino Bohn Nielsen 1 of 15 Outline (1) What is a time series? (2) Important characteristics of time series data. (3) Data sampling and stochastic processes. Cross-sectional data vs. time

374 views • 8 slides
Time Series Data Visualization Class 2, Part A 2 1 Time Series Data Fundamental

Time Series Data Visualization Class 2, Part A 2 1 Time Series Data Fundamental

Large Scale Information Visualization Jing Yang Fall 2007 1 Time Series Data Visualization Class 2, Part A 2 1 Time Series Data Fundamental chronological component to the data set Random sample of 4000 graphics from 15 of

763 views • 31 slides
Plotting time - series data IN TR OD U C TION TO DATA VISU AL IZATION W ITH MATP L OTL IB

Plotting time - series data IN TR OD U C TION TO DATA VISU AL IZATION W ITH MATP L OTL IB

Plotting time - series data IN TR OD U C TION TO DATA VISU AL IZATION W ITH MATP L OTL IB Ariel Rokem Data Scientist Time - series data INTRODUCTION TO DATA VISUALIZATION WITH MATPLOTLIB Climate change time - series date,co2,relative_temp

627 views • 27 slides
Time Series Data Visualization 2 1 Time Series Data Fundamental chronological component to

Time Series Data Visualization 2 1 Time Series Data Fundamental chronological component to

Information Visualization Jing Yang Spring 2007 1 Time Series Data Visualization 2 1 Time Series Data Fundamental chronological component to the data set Random sample of 4000 graphics from 15 of worlds newspapers and magazines

315 views • 31 slides
Time Series Analysis and Mining with R Time Series Decomposi- tion Time Series Forecasting

Time Series Analysis and Mining with R Time Series Decomposi- tion Time Series Forecasting

R and Time Series Data Time Series Analysis and Mining with R Time Series Decomposi- tion Time Series Forecasting Yanchang Zhao Time Series Clustering Time Series RDataMining.com Classification http://www.rdatamining.com/ R Functions

1.91k views • 42 slides
Timeseries kinds and applications MAC H IN E L E AR N IN G FOR TIME SE R IE S DATA IN P YTH

Timeseries kinds and applications MAC H IN E L E AR N IN G FOR TIME SE R IE S DATA IN P YTH

Timeseries kinds and applications MAC H IN E L E AR N IN G FOR TIME SE R IE S DATA IN P YTH ON Chris Holdgraf Fello w, Berkele y Instit u te for Data Science Time Series MACHINE LEARNING FOR TIME SERIES DATA IN PYTHON Time Series

802 views • 37 slides
W HAT IS TIME SERIES D ATA ? W HAT IS TIME SERIES D ATA ? A value over time W HAT IS TIME

W HAT IS TIME SERIES D ATA ? W HAT IS TIME SERIES D ATA ? A value over time W HAT IS TIME

T IME S ERIES D ATA P APERS C OVERED Interactive Visualization of Serial Periodic Data John V. Carlis and Joseph A. Konstan Visualizing and Discovering Non-Trivial Patterns in Large Time Series Databases Jessica Lin, Eamonn Keogh,

764 views • 32 slides
CS6220: DATA MINING TECHNIQUES Mining Time Series Data Instructor: Yizhou Sun yzsun@ccs.neu.edu

CS6220: DATA MINING TECHNIQUES Mining Time Series Data Instructor: Yizhou Sun yzsun@ccs.neu.edu

CS6220: DATA MINING TECHNIQUES Mining Time Series Data Instructor: Yizhou Sun yzsun@ccs.neu.edu November 12, 2013 Mining Time Series Data Basic Concepts Time Series Prediction and Forecasting Time Series Similarity Search

649 views • 45 slides
Working w ith more than one time series VISU AL IZIN G TIME SE R IE S DATA IN P YTH ON Thomas

Working w ith more than one time series VISU AL IZIN G TIME SE R IE S DATA IN P YTH ON Thomas

Working w ith more than one time series VISU AL IZIN G TIME SE R IE S DATA IN P YTH ON Thomas Vincent Head of Data Science , Ge y Images Working w ith m u ltiple time series An isolated time series A le w ith m u ltiple time series

505 views • 29 slides
Mining Approximate Top-K Subspace Anomalies in Multi-Dimensional Time-Series Data Xiaolei Li,

Mining Approximate Top-K Subspace Anomalies in Multi-Dimensional Time-Series Data Xiaolei Li,

Mining Approximate Top-K Subspace Anomalies in Multi-Dimensional Time-Series Data Xiaolei Li, Jiawei Han University of Illinois at Urbana-Champaign VLDB 2007 1 Time Series Data Many applications produce time series data Intel stock 2

888 views • 60 slides
Introduction to Time Series Data and Analysis Simon Taylor Department of Mathematics and

Introduction to Time Series Data and Analysis Simon Taylor Department of Mathematics and

Introduction to Time Series Data and Analysis Simon Taylor Department of Mathematics and Statistics 20 April 2016 Contents What is Time Series Data? Analysis Tools Trace Plot Auto-Correlation Function Spectrum Time Series Models Moving

876 views • 18 slides
Time Series A time series is a collection of data obtained by observing a response variable at

Time Series A time series is a collection of data obtained by observing a response variable at

ST 430/514 Introduction to Regression Analysis/Statistics for Management and the Social Sciences II Time Series A time series is a collection of data obtained by observing a response variable at various times. Examples The monthly Case-Shiller

351 views • 8 slides
Query by Content for Time Series Data in RDBMS 1 I N E S F . V E G A - L O P E Z University

Query by Content for Time Series Data in RDBMS 1 I N E S F . V E G A - L O P E Z University

Query by Content for Time Series Data in RDBMS 1 I N E S F . V E G A - L O P E Z University of Houston, Computer Science Seminar 12/6/13 Roadmap 2 Querying non-text data Time series data ECG data ECG sequence

646 views • 36 slides
Merging time series MAN IP U L ATIN G TIME SE R IE S DATA W ITH XTS AN D ZOO IN R Je re y

Merging time series MAN IP U L ATIN G TIME SE R IE S DATA W ITH XTS AN D ZOO IN R Je re y

Merging time series MAN IP U L ATIN G TIME SE R IE S DATA W ITH XTS AN D ZOO IN R Je re y R y an Creator of x ts and q u antmod Introd u cing merge () Combine series b y col u mn cbind() and merge() Database st y le joins on inde x ( i . e

597 views • 18 slides
Time-Series Data Numerical data obtained at regular time intervals The time intervals can

Time-Series Data Numerical data obtained at regular time intervals The time intervals can

Time-Series Data Numerical data obtained at regular time intervals The time intervals can be annually, quarterly, daily, hourly, etc. Example: Year: 1999 2000 2001 2002 2003 Sales: 75.3 74.2 78.5 79.7 80.2

1.05k views • 55 slides
Data Mining Techniques CS 6220 - Section 3 - Fall 2016 Lecture 18: Time Series Jan-Willem van de

Data Mining Techniques CS 6220 - Section 3 - Fall 2016 Lecture 18: Time Series Jan-Willem van de

Data Mining Techniques CS 6220 - Section 3 - Fall 2016 Lecture 18: Time Series Jan-Willem van de Meent ( credit: Aggarwal Chapter 14.3 ) Time Series Data http://www.capitalhubs.com/2012/08/the-correlation-between-apple-product.html Time Series

589 views • 22 slides
How to use Dates & Times with pandas Manipulating Time Series Data in Python Date & Time

How to use Dates & Times with pandas Manipulating Time Series Data in Python Date & Time

MANIPULATING TIME SERIES DATA IN PYTHON How to use Dates & Times with pandas Manipulating Time Series Data in Python Date & Time Series Functionality At the root: data types for date & time information Objects for points in

279 views • 25 slides
Analysis Manuel Len Hoyos Overview What is Time Series Data? Index Prices: Crude Oil

Analysis Manuel Len Hoyos Overview What is Time Series Data? Index Prices: Crude Oil

Time Series Analysis Manuel Len Hoyos Overview What is Time Series Data? Index Prices: Crude Oil Gold Bitcoin What is Time Series Analysis? Uses Forecasting Time Series Data A collection of observations of a

738 views • 17 slides
Compare Time Series Growth Rates Manipulating Time Series Data in Python Comparing Stock

Compare Time Series Growth Rates Manipulating Time Series Data in Python Comparing Stock

MANIPULATING TIME SERIES DATA IN PYTHON Compare Time Series Growth Rates Manipulating Time Series Data in Python Comparing Stock Performance Stock price series: hard to compare at di ff erent levels Simple solution: normalize price

670 views • 39 slides
Time series Decomposing a series into meaningful components R.W. Oldford Time series data -

Time series Decomposing a series into meaningful components R.W. Oldford Time series data -

Time series Decomposing a series into meaningful components R.W. Oldford Time series data - Sunspot activity Recall the measurements of monthly mean relative sunspot numbers from 1749 to 1983 plotted as Monthly sunspot activity 200 Sunspot

860 views • 71 slides
Time-Series Data in MongoDB on a Budget Peter Schwaller Senior Director Server Engineering,

Time-Series Data in MongoDB on a Budget Peter Schwaller Senior Director Server Engineering,

Time-Series Data in MongoDB on a Budget Peter Schwaller Senior Director Server Engineering, Percona Santa Clara, California | April 23th 25th, 2018 TIME SERIES DATA in MongoDB on a Budget Click to add text What is Time-Series Data?

585 views • 43 slides
Slides from: Elena Tsiporkova What is Special about Time Series Data? Gene expression time series

Slides from: Elena Tsiporkova What is Special about Time Series Data? Gene expression time series

Dynamic Time Warping Algorithm Slides from: Elena Tsiporkova What is Special about Time Series Data? Gene expression time series are expected to vary not only in terms of expression amplitudes, but also in terms of time progression since

305 views • 13 slides

More recommend