Real-Time Geo #rtgeo Who am i? Giving a real-time geo talk at - PowerPoint PPT Presentation

Real-Time Geo #rtgeo

Who am i?

Giving a real-time geo talk at @where20. How do you build stuff? #rtgeo. h o n e e r f o r i P v i a T w i t t 1 9 A p r C e n t e r n t i o n C o n v e a C l a r a m S a n t f r o y P a r k w a m e r i c a G r e a t A 5 0 0 1 5 0 5 4 a , C A 9 a C l a r S a n t a c e t h i s p l w e e t s a t V i e w T

Background [ ] r a f f i @ ~ / : c a t / e t w h c e r / s e e r h o v o i c e s | g r 5 e 8 5 p w 9 w h h e / u e r d p r e e h h o o o o # W H E R E 5 8 H O 5 O Wherehoo (2000) 9 / t c p # W H E R E H O O ⇢ “The Stuff Around You” ⇢ “Wherehoo Server: An interactive location service for software agents and intelligent systems” - J.Youll, R.Krikorian ⇢ In your /etc/services file! BusRadio (2004) ⇢ Designed mobile computers to play media while also transmitting telemetry ⇢ Looked and sounded like a radio - but really a Linux computer OneHop (2007) ⇢ Bluetooth proximity-based social networking

Background Twitter ⇢ Originally tech lead of API / Platform team ⇢ Built the first geo-based infrastructure before acquisition of Mixer Labs in December of 2009 ⇢ Now lead of the Application Services group ⇢ Runs five teams focused on scalable infrastructure around “core” data objects ⇢ Tweets, users, timelines, places, etc. ⇢ Delivery, authentication, APIs, etc.

Table of contents Background ⇢ Why are we interested in this? Twitter’s geo APIs ⇢ How do we allow people to talk about place? ⇢ Context around “place” Problem statement ⇢ What do we want our system to do? Infrastructure ⇢ How is Twitter solving this problem?

People want to talk about places

What’s happening here? Twitter’s Geo APIs

Original attempts Adding it to the tweet ⇢ Use myloc.me, et. al. to add text to the tweet ⇢ Puts location “in band” ⇢ Takes from the 140 characters Setting profile level locations ⇢ Set the user/location of a Twitter user ⇢ There’s an API for that! ⇢ Not a per-tweet basis ⇢ Not intended for high frequency alterations

Profile level changes [] raffi@~/: twurl -d location="San Francisco, California" \ http://twitter.com/account/update_location.xml <user> <id>8285392</id> <name>raffi</name> <screen_name>raffi</screen_name> <location>San Francisco, California</location> ... </user>

Geotagging API

Geotagging API Adding it to the tweet ⇢ Per-tweet basis ⇢ Out of band and pure metadata ⇢ Does not take from the 140 characters Native Twitter support ⇢ Simple way to update status with location data ⇢ Ability to remove geotags from your tweets en masse ⇢ Using GeoRSS and GeoJSON as the encoding format ⇢ Across all Twitter APIs (REST, Search, and Streaming)

status/update [] raffi@~/: twurl -d "status=hey-ho&lat=37.3&long=-121.9" \ http://api.twitter.com/1/status/update.xml <status> <text>hey-ho</text> ... <geo xmlns:georss="http://www.georss.org/georss> <georss:point>37.3 -121.9</georss:point> </geo> ... </user>

geocode parameter takes Search “latitude,longitude,radius” where radius has units of mi or km [] raffi@~/: curl "http://search.twitter.com/search.atom? geocode=40.757929%2C-73.985506%2C25km&source=foursquare" ... <title>On the way to ace now, so whenever you can make it I'll be there. (@ Port Imperial Ferry in Weehawken) http://4sq.com/ 2rq0vO</title> ... <twitter:geo> <georss:point>40.7759 -74.0129</georss:point> </twitter:geo> ...

geohose

location filtering [] raffi@~/: curl "http://stream.twitter.com/1/statuses/filter.xml? locations=-74.5129,40.2759,-73.5019,41.2759" locations is a bounding box specified by “long 1 ,lat 1 ,long 2 ,lat 2 ” and can track up to 10 locations that are most 1 degree square (~60 miles square and enough to cover most metropolitan areas)

Trends API

Trends API Global Trends ⇢ Analysis of “hot conversations” ⇢ Does not take from the 140 characters Location specific trends ⇢ Tweets being localized through a variety of means internally ⇢ Locations exposed over the API as WOEIDs and Twitter IDs ⇢ Can ask for available trends sorted by distnace

available locations [] raffi@~/: curl "http://api.twitter.com/1/trends/available.xml" <locations type=”array”> <location> <woeid>2487956</woeid> <name>San Francisco</name> <placeTypeName code=”7”>Town</placeTypeName> <country type=”Country” code=”US”>United States</country> <url>http://where.yahooapis.com/v1/place/2487956</url> n g o l n d a a t l a k e a </location> y t l l n a o t i o p n a C s d e n t r e a v ... h o t e r e t m a a r p s a d , t e o r </locations> s d , e r n u e t r n s o a t i c l o . o u y m o f r e n c a s t d i

L a Local trend o o k u p a t r e n d a t a g i v e n W O E I D [] raffi@~/: curl "http://api.twitter.com/1/trends/2487956.xml" <matching_trends type=”array”> <trends as_of=”2009-12-15T20:19:09Z”> ... <trend url=”http://search.twitter.com/search?q=Golden+Globe +nominations” query=”Golden+Globe+nominations”>Golden Globe nominations</ trend> <trend url=”http://search.twitter.com/search?q=%23somethingaintright” query=”%23somethingaintright”>#somethingaintright</trend> ... </trends> </matching_trends>

What’s in a name?

A place is a name 5001 Great America Parkway, Santa Clara, CA 95054 Great America Parkway and Tasman Drive The Bay Area Santa Clara convention center Twitter ID 3b7dd0d93e661e18

how do users what to share “where”?

Sharing coordinates More aptly named “geotagging” Good for sharing photos Possibly good for talking about a specific place (e.g. store, restaurant) People don’t understand numbers and without

Sharing polygons Privacy implications are potentially better If you thought sharing one pair of numbers was bad... Questions around polygon definition Still unable to visualize unless on a map

Sharing names Has the potential to make a connection with users Distinguishes a “named place” from simply a “place” Inverse relationship between granularity and connection Rather large internationalization / context implications

Geo-place API

Geo-place API Support for “names” ⇢ Not just coordinates ⇢ More contextually relevant ⇢ Positive privacy benefits Increased comlexity ⇢ Need to be able to look up a list of places ⇢ Requires a “reverse geocoder” ⇢ Human driven tagging and not possible to be fully automatic

Search [] raffi@~/: curl http://api.twitter.com/1/geo/search.json&lat=37.3&long=-121.9 ... "place_type":"neighborhood", "country_code":"US", "contained_within": [...] "full_name":"Willow Glen", "bounding_box": { "type":"Polygon", "coordinates": [[ [-121.92481908, 37.275903], [-121.88083608, 37.275903], [-121.88083608, 37.31548203], [-121.92481908, 37.31548203] ]] }, "name":"Willow Glen", "id":"46bc64ecd1da2a46", ...

Tweeting with a place [] raffi@~/: twurl -d "status=hey-ho&place_id=46bc64ecd1da2a46" \ http://api.twitter.com/1/status/update.xml <status> <text>hey-ho</text> ... <place xmlns:georss="http://www.georss.org/georss> <id>46bc64ecd1da2a46</id> <name>Willow Glen</name> <full_name>Willow Glen</full_name> <place_type>neighborhood</place_type> <url>http://api.twitter.com/1/geo/id/46bc64ecd1da2a46.json</url> <country code=”US”>United States</country> </place> ... </user>

Problem statement What do we want our system to do?

what do we need to build? Database of places ⇢ Given a real-world location, find places ⇢ Spatial search Method to store places with content ⇢ Per user basis ⇢ Per tweet basis

spatial lookup and index

as background... MySQL + Ability to index points and do a spatial query ⇢ For example, get points within a bounding rectangle ⇢ SELECT MBRContains(GeomFromText(‘Polygon(0 0, 0 3, 3 3, 3 0, 0 0))’), coord) FROM geometry Hard to cache the spatial query Possibly requires a DB hit on every query

options Grid / quad-tree ⇢ Create a grid (possibly nested) of the entire Earth Geohash ⇢ Arbitrarily precise and hierarhical spatial data reference Space filling curves ⇢ Mapping 2D space into 1D while preserving locality R-Tree ⇢ Spatial access data structure

Grid / Quad-Tree