Summary Summary Object Hyperlinking j yp g Mobile Visual Search - PDF document

MOBILE VISUAL SEARCH USING SMART M3 USING SMART-M3 Alessandro Franchi Luigi Di Stefano Tullio Salmon Cinotti Alessandro Franchi, Luigi Di Stefano, Tullio Salmon Cinotti First International Workshop on Semantic Interoperability June 22, 2010 for Smart Spaces (SISS 2010) Summary Summary � Object Hyperlinking j yp g � Mobile Visual Search � Smart-M3 platform Smart M3 platform � MVS Architecture � Sample Applications and Demo European Commission ARTEMIS JU SP3 SOFIA project (http://sofia-project org/) (http://sofia project.org/)

Object Hyperlinking Object Hyperlinking � Process of linking real-world objects to related digital Process of linking real world objects to related digital content by using some form of tag � Tags can then be read by a wireless mobile device � Tags can then be read by a wireless mobile device and information about objects and locations retrieved and displayed to the user 1 2 Object tag DB //nj4 3 RFID t RFID tags H Hardlinks dli k or SMS tags www www Virtual tags (GPS coordinates) Graphical tags Object Hyperlinking Object Hyperlinking � Applications: A li ti � Link an audio CD to an online website where the songs can be previewed g p SMS SMS � Enhancing a physical document (e.g. tag a newspaper) with multimedia content (e g a virtual tour of a property linked (e.g. a virtual tour of a property linked to a real estate advertising) � Cultural heritage and tourist information i f ti � … Virtual tag tag Graphical tags

Mobile Tagging Mobile Tagging Linking method: 2D barcodes Linking method: 2D barcodes � � Decoding… WWW Take a picture Decode Website P Pro: � � Fast recognition, even in mobile environments � Cheap to produce � The tag can directly encode an URL without the need of a central database to Th t di tl d URL ith t th d f t l d t b t store the link Cons: � � Requires to instrument the environment with artificial visual patterns � Requires to instrument the environment with artificial visual patterns Examples: � � Quick Response (QR), DataMatrix, Semacode, Microsoft Tag… Mobile Visual Search Mobile Visual Search � Linking method: “natural” visual tag � No artificial tagging required! gg g q … … www A Scanner Darkly Sending… Plot In a totalitarian socie ty in a near future, the undercover detective Bob Arctor is workin... Process Send picture to Display linked Take a picture image of an object processing server content � Examples: � Google Goggles, SnapTell, Nokia Point & Find iCandy, Doog, kooba…

MVS principles MVS principles � Conventional principles C ti l i i l � Usability : object identification should be fast, easy and reliable � Unobtrusiveness and productivity : deployment should be as unobtrusive and inexpensive as possible � Additional principle � Interoperability and information sharing: the � Interoperability and information sharing: the information deduced by the tag should be stored in a shared and interoperable information search extent to th the benefit of third party applications, which do not b fit f thi d t li ti hi h d t need to be aware of the connectivity nor of the identification technology used by the identification engine i Smart M3 Smart-M3 � Interoperability platform deployed within SOFIA I t bilit l tf d l d ithi SOFIA (Smart Objects For Intelligent Applications) � Purpose: � Enabling seamless interoperability between devices of many kinds and different manufacturer, operating in f ki d d diff f i i different business domains M ulti-domain M ulti-device M ulti-vendor � Main concepts: � Main concepts: � Shared tuple space mechanism for information exchange � The interpretation of information is based on common ontology models

Smart-M3 Architecture � The M3 functional architecture (Smart World) works on top of ( ) p one or multiple concurrent SOA service networks (the Service World) World) � Smart Spaces (SS) � Smart Spaces (SS) � Individual search extents KP Smart Space � Semantic Information Brokers (SIB) � Entities for storing the information � Knowledge Processors (KP) KP SIB SIB � Entities capable of inserting, removing and querying information from the d i i f ti f th KP Smart Space SIB MVS Architecture MVS Architecture � M3 Application: “ a scenario enabled by a set of collaborating KPs ” � Modular architecture � Base module � Basic recognition scenario � Uses computer vision algorithms to identify objects in pictures � Plugin modules Pl i d l � Address different visual search scenarios

Ontology Ontology Ontology ItemModelType ImageData Item classes classes rdf:type rdf:type rdf:type HasItemModelType HasImageData #Item_xxx HasRecognizedModelType #ItemModelType_1 #ImageData_yyy #ItemModelType_2 HasImageURL … “./ImageStore/Image01.png” HasAssociatedContent “http://...” MVS Architecture MVS Architecture MD MD Base Image g Mobile Device Mobile Device Station Plug-in KP Base Smart KP KP Space Space Image Store Recognized Base Object KP

Sample Application: Maintenance scenario M i i Take a picture of the Object is recognized …the application displays device that needs to be device that needs to be and… and a web page with related a web page with related repaired content such manual, most recent drivers etc… Sample Application: Mobile Shopping scenario M bil Sh i i …and automatically Take a picture p Object is j added to remote added to remote of an item recognized… shopping list Add another When finished, , confirm order item

Demo Demo Conclusions Conclusions � Novel MVS engine that uses Smart-M3 as interoperability platform � The adoption of Smart-M3 allows for higher interoperability between the interacting p y g entities, regardless of execution environments and implementation languages, making the and implementation languages, making the system easily extensible to previously unforeseen scenarios unforeseen scenarios � This system has been integrated in part of the Maintenance demo developed within SOFIA Maintenance demo developed within SOFIA

Sequence Diagram Initialization Image Smart MD Base MD BIS User Store Space KP Plugin KP 1: join Offline 2: subscribe 2: subscribe initialization initialization 3: app start 4: join j 5: app start 6: join and subscribe j Sequence Diagram Basic Object Recognition Scenario Image Smart MD Base MD BIS User Store Space KP Plugin KP 1: take photo 2: store

RDF (Resource Description Framework) � Information is represented using the RDF I f ti i t d i th RDF format (W3C standard) � Everything described in RDF is a “resource” � Each resource is described by one or more statement in the form of triples Subject- Predicate-Object � Subject: the resource � Object: a value or another resource � Predicate: a property that ties the subject to the object.