NTTS 2015 - Session 6A Big data sources: web scraping and smart - PowerPoint PPT Presentation

Feb 18, 2023 •290 likes •396 views

Automatic price collection on Ingolf Boettcher the internet (web scraping) Brussels 10. March 2015 NTTS 2015 - Session 6A Big data sources: web scraping and smart meters www.statistik.at Wir bewegen Informationen Web scraping There is a

Automatic price collection on Ingolf Boettcher the internet (web scraping) Brussels 10. March 2015 NTTS 2015 - Session 6A – Big data sources: web scraping and smart meters www.statistik.at Wir bewegen Informationen
Web scraping There is a huge amount of data on the internet <HTML> How can we best collect/scrape/harvest <HEAD> data from there for <TITLE> DATA statistical purposes? </Title> </HEAD> </HTML> www.statistik.at Folie 2 | 10.03.2015
Web scraping Internet data collection – Minimum goal for (Price) Statistics: Turn website content into a spreadsheet www.statistik.at Folie 3 | 10.03.2015
Web scraping Internet data collection Options: 1. Manual price collection 2. Develop an API /Web scraper 2.1 by writing custom computer code 2.2 by using point and click web tools www.statistik.at Folie 4 | 10.03.2015
Web scraping Reasons for not writing an own web scraper IT-developer needed, therefore: • Expensive • Inflexible • Even maintenance cannot be handled by CPI staff www.statistik.at Folie 5 | 10.03.2015
Web scraping Reasons to use click and point webtools for web scraping: No IT-developer needed, therefore: • Cheap • Flexible • No programming skill required www.statistik.at Folie 6 | 10.03.2015
Web scraping How web scraping with click and point using import.io looks like: • web-platform that allows to structure and extract data from websites www.statistik.at Folie 7 | 10.03.2015
Webscraping with import.io www.statistik.at Folie 8 | 10.03.2015
Webscraping Web scraping with click and point on web- based platform offers solutions to: • extract data by point-and-click • record actions on a website • crawl all the data of a webpage More issues to be considered: • Legality to crawl on websites • Internal IT Security • Training of staff www.statistik.at Folie 9 | 10.03.2015
Automatic price Contact: Ingolf Boettcher collection on the internet Guglgasse 13, 1110 Wien Tel: +43 (1) 71128-7917 (web scraping) Fax: +43 (1) 7180718 Ingolf.boettcher@statistik.gv.at www.statistik.at Folie 10 | 10.03.2015

Recommend

Session 6A - Big data sources: web scraping and smart meters Using Internet as a Data Source for

NTTS 2015 Session 6A - Big data sources: web scraping and smart meters Using Internet as a Data Source for Official Statistics: a Comparative Analysis of Web Scraping Technologies Giulio Barcaroli(*) (barcarol@istat.it), Monica Scannapieco (*)

436 views • 12 slides

Web Scraping 1 / 9 Web Scraping Two ways to mine data from the web The hard way, by web

Web Scraping 1 / 9 Web Scraping Two ways to mine data from the web The hard way, by web scraping The easy way, using web service APIs Well see examples of both. 2 / 9 Web Scraping Web scraping, a.k.a. screen scraping, means getting

485 views • 9 slides

NTTS 2015 9 13 March 2015, Brussels Session 13C - Survey integration, coordination and

NTTS 2015 9 13 March 2015, Brussels Session 13C - Survey integration, coordination and alignment Avoiding duplicate collection of flow data: estimating intra-EU inbound tourism using partner data Christophe Demunter, Krista Dimitrakopoulou

697 views • 16 slides

Analysis of the Potential of Selected Big Data Repositories as Data Sources for Official

Analysis of the Potential of Selected Big Data Repositories as Data Sources for Official Statistics NTTS 2015 New Techniques and Technologies for Statistics Petrakos M., Santourian A., Farmakis G., Stavropoulos P., Oikonomopoulou G., Ntakou E.

221 views • 8 slides

A more visual dissemination for attracting new users NTTS 2015 Brussels Philippe BAUTIER,

A more visual dissemination for attracting new users NTTS 2015 Brussels Philippe BAUTIER, Bernard LE GOFF Session 5A The role of visualisation in data dissemination All the same challenge! How to build a dissemination and communication

491 views • 23 slides

video demo End-User Web Scraping: Google Scholar Edition Sarah Chasins data scraping tool

video demo End-User Web Scraping: Google Scholar Edition Sarah Chasins data scraping tool input demonstration of how to collect the first row of a relational dataset F r o m h i g h l y s t r u c t u r e d w e b p a

837 views • 63 slides

Katy Stokes Demographic Methods Centre, ONS NTTS Conference, March 2015 Uncertainty measures

Measuring uncertainty in ONS population estimates: capturing variability in statistics from combinations of census, administrative and survey sources Katy Stokes Demographic Methods Centre, ONS NTTS Conference, March 2015 Uncertainty measures

199 views • 8 slides

Using Web Scraped Data to Construct Consumer Price Indices Nigel Swier NTTS Conference, 10-12

Using Web Scraped Data to Construct Consumer Price Indices Nigel Swier NTTS Conference, 10-12 March 2015, Brussels Background One of 4 big data pilots in ONS Prices collection manually based Difficulties accessing retail

455 views • 11 slides

Mauro Michielon ICT Development NTTS conference, Brussels, 12 th March 2015 EEAs context EEA

Web tools for accessing and disseminating data of different formats Mauro Michielon ICT Development NTTS conference, Brussels, 12 th March 2015 EEAs context EEA needs to deal with a vast number of very heterogeneous datasets, coming from

478 views • 5 slides

from an integrated system of administrative microdata to an infrastructure for the users NTTS

Reversing the flow: from an integrated system of administrative microdata to an infrastructure for the users NTTS 2015 Session 6C - Enterprise architecture and integration Simone Ambroselli Istat ambrosel@istat.it Giuseppe Garofalo Istat

446 views • 10 slides

The Office for National Statistics Big Data Project Nigel Swier NTTS Conference, 10-12 March

The Office for National Statistics Big Data Project Nigel Swier NTTS Conference, 10-12 March 2015, Brussels Background Set up as a12 month project Jan-Dec 2014 Aims: Investigate the potential of big data for official statistics and

612 views • 12 slides

Modelling sample data from smart-type meter electricity usage Susan Williams NTTS Conference,

Modelling sample data from smart-type meter electricity usage Susan Williams NTTS Conference, 10-12 March 2015, Brussels Introduction Big Data Project in the UKs Office For National Statistics Practical pilot on the potential of

312 views • 12 slides

Web Scraping With P y thon W E B SC R AP IN G IN P YTH ON Thomas Laetsch Data Scientist , NYU B

Web Scraping With P y thon W E B SC R AP IN G IN P YTH ON Thomas Laetsch Data Scientist , NYU B u siness Sa vvy What are b u sinesses looking for ? Comparing prices Satisfaction of c u stomers Generating potential leads ... and m u ch more !

854 views • 29 slides

STATS 507 Data Analysis in Python Lecture 27: APIs Previously: Scraping Data from the Web We

STATS 507 Data Analysis in Python Lecture 27: APIs Previously: Scraping Data from the Web We used BeautifulSoup to process HTML that we read directly We had to figure out where to find the data in the HTML This was okay for simple things like

638 views • 26 slides

Scraping Distributed, Hierarchical Web Data with Programming by Demonstration! Sarah E.

Scraping Distributed, Hierarchical Web Data with Programming by Demonstration! Sarah E. Chasins 1 Maria Mueller 2 Rastislav Bodik 2 1 University of California, Berkeley 2 University of Washington The web: a rich source of data! 2008:

256 views • 24 slides

Data Sources; SCNL Data Sources Data sources producing waveform data can come from a remote

Data Sources; SCNL Data Sources Data sources producing waveform data can come from a remote source, via an import or a connection to a Seed Link server (ie: slink2ew), or from a local source, for example: a Quanterra digitizer (q3302ew) or

252 views • 22 slides

15-388/688 - Practical Data Science: Data collection and scraping J. Zico Kolter Carnegie Mellon

15-388/688 - Practical Data Science: Data collection and scraping J. Zico Kolter Carnegie Mellon University Fall 2019 1 Outline The data collection process Common data formats and handling Regular expressions and parsing 2 Outline The

605 views • 27 slides

Collecting Social Media Data Two different methods: 1. Screen scraping: extract data from source

Collecting Social Media Data Two different methods: 1. Screen scraping: extract data from source code of website 2. Web APIs (application programming interface): use a set of structured https requests that return JSON or XML files Collecting

121 views • 10 slides

Data with Alternative Data Sources Session 2 Session 2: Enhancing Health Survey Data with

Enhancing Health Survey Data with Alternative Data Sources Session 2 Session 2: Enhancing Health Survey Data with Alternative Data Sources Who gets it right? Using survey and administrative data to evaluate characteristics associated with

264 views • 7 slides

important banks NTTS March 11, 2015 www.jrc.ec.europa.eu Rationale people sometimes

Does web anticipate stocks? Analysis for a subset of systemically important banks NTTS March 11, 2015 www.jrc.ec.europa.eu Rationale people sometimes trade on noise as if it were information (Black, 1986) Where can we find the

126 views • 12 slides

USE OF GEOSPATIAL AND WEB DATA FOR OECD STATISTICS CCSA S PECIAL SESSION ON SHOWCASING BIG DATA 1

USE OF GEOSPATIAL AND WEB DATA FOR OECD STATISTICS CCSA S PECIAL SESSION ON SHOWCASING BIG DATA 1 O CTOBER 2015 Paul Schreyer Deputy-Director, Statistics Directorate, OECD OECD APPROACH OECD : Facilitator of discussion on new data sources

212 views • 20 slides

Recent Advances in the Measurement of Intangible Assets Presentation to NTTS 2015 Mary

Recent Advances in the Measurement of Intangible Assets Presentation to NTTS 2015 Mary OMahony, (KCL) Co-authors: Carol Corrado (The Conference Board ), Jonathan Haskel (Imperial College London) and Cecilia Joan Lasinio (LUISS and Istat)

252 views • 10 slides

Web Scraping 101 W OR K IN G W ITH W E B DATA IN R Charlo e Wickham Instr u ctor Selectors

Web Scraping 101 W OR K IN G W ITH W E B DATA IN R Charlo e Wickham Instr u ctor Selectors Li le bro w ser e x tensions Identif y the speci c bit ( s ) y o u w ant Gi v e y o u a u niq u e ID to grab them w ith Not u sed in this co

1.02k views • 16 slides

Using Research Data Centres (RDC) to access Big Data David Schiller (IAB) Anja Burghardt (IAB)

New Techniques and Technologies for Statistics NTTS 2015, Brussels (Belgium), March 11, 2015 Using Research Data Centres (RDC) to access Big Data David Schiller (IAB) Anja Burghardt (IAB) The Research Data Centre (FDZ) of the German Federal

377 views • 11 slides