Big Data Max Kemman University of Luxembourg October 11, 2015 - PowerPoint PPT Presentation

Big Data Max Kemman University of Luxembourg October 11, 2015 Doing Digital History: Introduction to Tools and Technology

Recap from last time What were aspects of an archive? What are the three steps of digitisation? What is the difference between data & metadata? What meta/data do we have of letters?

Today • Are digital libraries big data? • N=ALL • Messy data • From causality to correlation • Radical contextualisation • Next time

Are digital libraries big data? Last week we discussed digital libraries/archives Europeana contains about 53M digital objects Is this big data?

What is "data" anyway? Term has rhetorical function: "that which is given prior to argument" (Gitelman, 2014) Common description: "raw data" But creating data requires vast amount of work (as we saw last week) Interpretive work into creating data

What is "big data"? Metaphors used to describe big data give different interpretations (Awati & Shum, 2014) • Food: raw or cooked • Resource: oil, gold • Liquid: ocean, tsunami

What is "big data" anyway? 'Classic' definition by V's: • Volume: size • Velocity: accumulation • Variety: heterogeneous Another definition: too much data to handle

Is this new? Andrew Prescott (2015): • Domesday book • US Census 1890

What is "big data" anyway? What is the difference between "lots of data" and "big data"? (Lagoze, 2014) • "Large" is historical: computers change • Big data makes us rethink what science is

Are digital libraries big data? Or, does History have big data? From the definitions so far: • Size: not so much (compared to CERN) • Velocity: not so much • Variety: yes! • Too much data to handle: probably • Makes us rethink what science/scholarship is: maybe Is our collection of Hillary Clinton emails 'big data'? Some say History/Humanities do not have big data

Why is big data interesting BUT, why are we concerned with big data, but not with particle physics? (Wallach, 2014) What are the 2 reasons she gives? • Social: big data are about people • Granularity: individual people and their activities Here maybe History/Humanities do have interest in big data

Big data is a big topic Another definition of big data (Mayer-Schönberger & Cukier, 2014) • N=ALL • Messy • From causality to correlation Let's discuss these features

N=ALL "N" refers to the number of observations done as part of the sample size Sample: a group that represents the entire population So N=ALL refers to measuring everything, rather than a representative smaller group

All historical sources? A difference between "a lot of data" and "all data" Remember Rosenzweig from week 1: The injunction of traditional historians to look at “everything” cannot survive in a digital era in which “everything” has survived Rosenzweig (2003)

Is size that interesting? If big data is merely a quantitative difference, what's the interest? But, quantitive can lead to qualitative difference (Mayer-Schönberger, 2014)

Quantitative to qualitative

Longue durée Rather than focusing on a very short timespan, see development over ages (Manning, 2013)

Messy data Big data has Variety A heterogeneous dataset • Different data-types • Different variables Too much data to manually check

Can we use messy data? Mayer-Schönberger & Cukier: size makes up for messiness Exactness is from the age of spare information The noise can be smoothed out

Crowdsourcing One way of trying to get someone to look at the data Need to trust anonymous people

Does big data reflect the world? With N=ALL, big data = reality, right? But (big) data incorporates choices of what to measure Twitter/Facebook are biased reflections of the world

Biases in language Big data word-pairs (MIT Technology review) • Man - Woman • King - Queen • Brother - Sister • Computer programmer - Homemaker • Doctor - Midwife • Coward - Whore • etc

How big data is 'unfair' The average person is a fiction Hitchcock: it is the exceptions we are interested in!

Looking at the exceptions Wallach agrees: use the granularity of big data to study minorities & exceptions How do we discover the minorities & exceptions of interest? To repeat; cannot look at all cases individually Some statistical analysis is required

From causality to correlation Correlation: two variables show a statistical relation • Positive: when A increases, B increases • Negative: when A increases, B decreases Causation: one variable explains the second • Example: when it rains, more people take umbrellas with them

Correlation found A nice example is Google Flu Trends: • Took flu data from national health center for number of years • Investigated which keyword searches occurred shortly before or during flu outbreaks • Use keyword searches to predict outbreak of flu

Correlation and causation Important to remember: correlation does not equal causation The keyword searches do not cause the flu! Sometimes you don't know which variable comes first Maybe a third variable explains the two measured ones

Meaningful correlation Does the correlation mean anything? Google Flu Trends later found not to produce accurate results Spurious correlations

Spurious correlations

Spurious correlations http://www.tylervigen.com/spurious-correlations Find a correlation yourself: http://tylervigen.com/discover

Meaningful correlation We cannot only use the statistics, we need to interpret them But still we do not want to manually check all the possible correlations

Machine learning Wallach describes herself as machine learning researcher A simple introduction to machine learning (Geitgey, 2014) Rather than telling the computer what to do, it learns what to do • Supervised • Unsupervised

Supervised learning Provide enough answers to learn to give a new answer Computer figures out how to go from data to the answer

Supervised learning https://www.youtube.com/watch?v=SZ88F82KLX4 Or beat masters at chess or Go

Unsupervised learning No given answer Are there patterns? Outliers?

Train without knowing the rules What do pregnant women buy? How are sentences translated to different languages? (MIT Technology Review)

Biased algorithms? Issues of biased algorithms: • Diversity in job applications • School drop outs • Predictive profiling of criminality "We have no idea how these predictions are made" Often criticism of algorithm, but where does bias come from?

Rethinking science/scholarship How does this require a rethinking of scholarship? Ways of reasoning (Dixon, 2012) • Induction: from the specific to the general • Deduction: from the general to the specific • Abduction: patterns

Patterns Rens Bod: discovery of patterns with tools is Humanities 2.0 Hermeneutic interpretation of these patterns is Humanities 3.0 Fickers: context more interesting than the data

Radical contextualisation What is the context of each datapoint? Hitchcock - contextualize using the big data

Context If content is king, context is its crown Your search keywords make sense in your context

Radical context Remember from week 1: what does this tweet mean as part of 31M? Or actually: what does this tweet mean outside of Twitter?

Zooming Hitchcock describes the macroscope quoting Katy Börner Macroscopes provide a "vision of the whole," helping us "synthesize" the related elements and detect patterns, trends, and outliers while granting access to myriad details. Rather than make things larger or smaller, macroscopes let us observe what is at once too great, slow, or complex for the human eye and mind to notice and comprehend.

Zooming in on people If today we have a public dialogue that gives voice to the traditionally excluded and silenced – women, and minorities of ethnicity, belief and dis/ability – it is in no small part because we now have beautiful histories of small things. In other words, it has been the close and narrow reading of human experience that has done most to give voice to people excluded from ‘power’ by class, gender and race. Hitchcock

Close reading Hitchcock argues for interchange of close and distant reading Distant reading? That's the next lecture

For next time 18 October Distant Reading • Aiden, E. L., & Michel, J.-B. (2013). The sound of silence. In Uncharted (pp. 69–83). Penguin. • Moretti, F. (2009). Style, Inc. Reflections on Seven Thousand Titles (British Novels, 1740– 1850). Critical Inquiry, 36(1), 134–158.