Tie Vegan Data Diet Tie Vegan Data Diet How Wikipedia cuts down privacy issues while keeping data fit
Marcel Ruiz Forns Software Developer Analytics Team
Marcel Ruiz Forns Software Developer Analytics Team
Marcel Ruiz Forns Software Developer Analytics Team Anyone can edit!
Marcel Ruiz Forns Privacy Software Developer Analytics Team Why privacy ● What we do ● Implementation ● Pros and cons ● Anyone Questions ● can edit!
https://blog.wikimedia.org/2018/04/18/greece-legal-case-ended
By: Hugh D'Andrade, Senior Designer @ EFF https://commons.wikimedia.org/wiki/File:Laptop-spying.jpg
https://transparency.wikimedia.org
Privacy Privacy
https://foundation.wikimedia.org/wiki/Privacy_policy
● Read or edit without account. https://foundation.wikimedia.org/wiki/Privacy_policy
● Read or edit without account. ● Register account without name, email or any other info. https://foundation.wikimedia.org/wiki/Privacy_policy
● Read or edit without account. ● Register account without name, email or any other info. ● Never selling/sharing your info with third parties. https://foundation.wikimedia.org/wiki/Privacy_policy
● Read or edit without account. ● Register account without name, email or any other info. ● Never selling/sharing your info with third parties. ● Retaining your info for shortest time possible. https://foundation.wikimedia.org/wiki/Privacy_policy
Usage Data
Usage Data 500M web requests PER HOUR
Usage Data 500M 2000 web requests events PER HOUR PER SECOND
https://stats.wikimedia.org/v2/#/all-projects/reading/legacy-page-views
https://analytics.wikimedia.org/dashboards/browsers/#all-sites-by-os
username mforns ip_adress 31.214.189.167 user_agent Mozilla/5.0 (X11; Linux ... session_id 8c878625792be023 edit_count 4257 ui_skin minerva
username mforns s e n i l e d i u G n o i t n e ip_adress 31.214.189.167 t e R a t a D e s n e l i i d g u _ o n n t i e e t _ r t a D a i / w i k g / o r a . d i e m i k i user_agent Mozilla/5.0 (X11; Linux ... . w t a m e / s : / t p h t , d e t e l e d e b l l w i t i , s y a d 0 9 t s o session_id 8c878625792be023 m t ” a . d r e e fi t i f t A n “ e d i - e d r o d , e t a g e r g g a edit_count 4257 ui_skin minerva
Deleting Data D e l e t i n g D a t a
Deleting Data D e l e t i n g D a t a Are you sure? Cancel Delete
Data g n i t e l e D
--dry-run --execute undef -> execute undef -> dry-run --tables-to-delete --tables-to-delete undef -> all undef -> none * -> all
Executing tests … Tests passed. Starting DRY-RUN. Checking partitions to delete … Partitions that would be deleted by execution: - year=2019, month=1, day=1, hour=0, wiki=en.wikipedia - year=2019, month=1, day=1, hour=0, wiki=es.wiktionary - year=2019, month=1, day=1, hour=0, wiki=de.wikibooks - year=2019, month=1, day=1, hour=1, wiki=en.wikipedia - year=2019, month=1, day=1, hour=1, wiki=es.wiktionary - year=2019, month=1, day=1, hour=1, wiki=de.wikibooks - year=2019, month=1, day=1, hour=2, wiki=en.wikipedia - year=2019, month=1, day=1, hour=2, wiki=es.wiktionary - year=2019, month=1, day=1, hour=2, wiki=de.wikibooks DRY-RUN finished.
--database=event --tables=menuClicks --wikis=en.wikipedia --older-than=90 --skip-trash=true --execute=<checksum>
--database=event --tables=menuClicks --wikis=en.wikipedia --older-than=90 --skip-trash=true Executing tests … Tests passed. Starting DRY-RUN. Checking partitions to delete … Partitions that would be deleted by execution: - year=2019, month=1, day=1, hour=0, wiki=en.wikipedia - year=2019, month=1, day=1, hour=0, wiki=es.wiktionary --execute=<checksum> - year=2019, month=1, day=1, hour=0, wiki=de.wikibooks - year=2019, month=1, day=1, hour=1, wiki=en.wikipedia - year=2019, month=1, day=1, hour=1, wiki=es.wiktionary - year=2019, month=1, day=1, hour=1, wiki=de.wikibooks - year=2019, month=1, day=1, hour=2, wiki=en.wikipedia - year=2019, month=1, day=1, hour=2, wiki=es.wiktionary - year=2019, month=1, day=1, hour=2, wiki=de.wikibooks DRY-RUN finished. Parameter checksum: 57ca7987d987e9e98a6c79
#1 Dry-run --database=event --tables=menuClicks --wikis=en.wikipedia --older-than=90 --skip-trash=true Executing tests … Tests passed. Starting DRY-RUN. Checking partitions to delete … Partitions that would be deleted by execution: - year=2019, month=1, day=1, hour=0, wiki=en.wikipedia - year=2019, month=1, day=1, hour=0, wiki=es.wiktionary --execute=<checksum> - year=2019, month=1, day=1, hour=0, wiki=de.wikibooks - year=2019, month=1, day=1, hour=1, wiki=en.wikipedia - year=2019, month=1, day=1, hour=1, wiki=es.wiktionary - year=2019, month=1, day=1, hour=1, wiki=de.wikibooks - year=2019, month=1, day=1, hour=2, wiki=en.wikipedia #2 Eyecute - year=2019, month=1, day=1, hour=2, wiki=es.wiktionary - year=2019, month=1, day=1, hour=2, wiki=de.wikibooks DRY-RUN finished. Parameter checksum: 57ca7987d987e9e98a6c79
a t a D Sanit�ing
a t a D Sanit�ing Advanced
Unsanitized data 90 days
Unsanitized data 90 days S Sanitized data Kept indefinitely
Unsanitized data 90 days S Sanitized data Kept indefinitely
Unsanitized data 90 days S S Sanitized data Kept indefinitely
Unsanitized date 2019-01-01 ip 31.214.189.167 user_agent Mozilla/5.0 (X11; Linux ... wiki en.wikipedia action click target menu
Black-list Unsanitized date 2019-01-01 ip 31.214.189.167 user_agent Mozilla/5.0 (X11; Linux ... wiki en.wikipedia action click target menu
Black-list Unsanitized Sanitized date 2019-01-01 date 2019-01-01 ip 31.214.189.167 ip NULL user_agent Mozilla/5.0 user_agent NULL (X11; Linux ... wiki en.wikipedia wiki en.wikipedia action click action click target menu target menu
Black-list Unsanitized Sanitized date 2019-01-01 date 2019-01-01 ip 31.214.189.167 ip NULL user_agent Mozilla/5.0 user_agent NULL (X11; Linux ... wiki en.wikipedia wiki en.wikipedia action click action click target menu target menu cookie_id 724310 cookie_id 724310
White-list Unsanitized date 2019-01-01 ip 31.214.189.167 user_agent Mozilla/5.0 (X11; Linux ... wiki en.wikipedia action click target menu
White-list Unsanitized Sanitized date 2019-01-01 date 2019-01-01 ip 31.214.189.167 ip NULL user_agent Mozilla/5.0 user_agent NULL (X11; Linux ... wiki en.wikipedia wiki en.wikipedia action click action click target menu target menu
White-list Unsanitized Sanitized date 2019-01-01 date 2019-01-01 ip 31.214.189.167 ip NULL user_agent Mozilla/5.0 user_agent NULL (X11; Linux ... wiki en.wikipedia wiki en.wikipedia action click action click target menu target menu cookie_id 724310 cookie_id NULL
White-list Unsanitized Sanitized date 2019-01-01 date 2019-01-01 ip 31.214.189.167 ip Spain user_agent Mozilla/5.0 user_agent NULL (X11; Linux ... wiki en.wikipedia wiki en.wikipedia action click action click target menu target menu cookie_id 724310 cookie_id NULL
White-list Unsanitized Sanitized date 2019-01-01 date 2019-01-01 ip 31.214.189.167 ip Spain user_agent Mozilla/5.0 user_agent Linux (X11; Linux ... wiki en.wikipedia wiki en.wikipedia action click action click target menu target menu cookie_id 724310 cookie_id NULL
White-list Unsanitized Sanitized date 2019-01-01 date 2019-01-01 ip 31.214.189.167 ip Spain user_agent Mozilla/5.0 user_agent Linux (X11; Linux ... wiki en.wikipedia wiki en.wikipedia action click action click # target menu target menu cookie_id 724310 cookie_id 8d56ab209e10
P r i v a c y C u l t u r e
Unique visitor�
Unique visitor� UUID
Unique visitor� UUID, REQ UUID, REQ UUID
Unique visitor� UUID, REQ UUID, REQ UUID SELECT COUNT(DISTINCT uuid) FROM database.table WHERE date = ’2019-01-01’;
Unique visitor� UUID, REQ UUID
Unique visitor� LAST ACCESS
Unique visitor� LA, REQ LA, REQ LAST ACCESS
Unique visitor� LA, REQ LA, REQ LAST ACCESS SELECT COUNT(*) FROM database.table WHERE (la IS NULL OR la < date) AND date = ’2019-01-01’;
By: Victor Grigas https://commons.wikimedia.org/wiki/File:Papaul_Tshibamba-4.jpg
Tie Vegan Data Diet
Tie Vegan Data Diet ● Guarantee of privacy ● Less work related to data requests ● Easier to publicize
Tie Vegan Data Diet ● Guarantee of privacy ● Extra work ● Less work related to ● Privacy culture data requests needs time ● Easier to publicize ● Analysts have to compromise
QUESTIONS? QUESTIONS? By: Randall Munroe @ XKCD https://xkcd.com/285/
Recommend
More recommend