Postdoctoral fellowship: Sketchpad Development NATO Science Project (1996, UK) An AI Application for the Transformation of a 2D Sketch to a 3D Geometric Model Project Report: The NATO Science Fellowship Program for Post Doctoral Studies, NATO area code: 4301, NATO list code: 51/B96/TU
Ubiquitous System Development 2009-2012 Australian Research Council Discovery Grant, DP0988088 (Kavakli) A Gesture-Based Interface for Designing in Virtual Reality Research questions: “ How do we generate 3D models of real objects by sketching using VR in real- time?” and “ How can we support the design process using VR, design cognition, and gesture recognition?”
3D Sketchpad This project examines a novel environment in which a designer can define the contour of a sketch by controlling a pointer using a pair of cyber gloves and can interact with the design product by using a sensor jacket in 3D space. The sensor jacket, cyber gloves , and the pointer incorporate 3D position sensors so that drawing primitives entered are recreated in real time on a head mounted display worn by the user. Thus, the VR system provides a " 3D sketch pad " and the designer has the benefit of a stereo image. The interface to be developed will recognize hand-gestures of the designer, pass commands to a 3D modelling package via a motion recognition system, produce the 3D model of the sketch on-the-fly, and generate it on a head mounted display.
Frank Gehry, Guggenheim Museum, Bilbao, 1997
Hand gesture recognition 2 5 =32 possible combinations of gestures 5W:1 sensor per finger vs 16W:3 sensors Orientation trackers Switch tracking the motion of the hand in 3D Zoom in and out using mouse or keyboard Need motion trackers: SpacePad Flexure value x (0 ≤ x ≥ 1) Gesture Sketching Task ID Name Thumb Index Middle Ring Little ≤ 0.1 ≤ 0.1 ≤ 0.1 ≤ 0.1 ≤ 0.1 0 Fist Stop ≤ 0.1 ≥ 0.9 ≤ 0.1 ≤ 0.1 ≤ 0.1 1 Index Finger Draw Point ≥ 0.9 ≥ 0.9 ≥ 0.9 ≥ 0.9 ≥ 0.9 2 Open Hand Erase Gesture Definition Table VisoR: Virtual and Interactive Simulation of Reality Research Group 2008
DeSIGN VisoR: Virtual and Interactive Simulation of Reality Research Group 2008
DESIRE VisoR: Virtual and Interactive Simulation of Reality Research Group 2008
Gesture Recognition 52 individual piezzo resistive sensor strips located from wrist to shoulders on the right and left side of the t-shirt. The data is acquired by the National Instrument Data Acquisition Unit.
Findings Sparse Representation-based Classification (SRC). allows signals to be recovered with a few number of samples Using SRC and Compressed Sensing we obtained a gesture recognition rate of 100% for both sensor jacket and wii-mote based user-dependent tracking for 3D and 2D gesture sets 99.33% for user-independent 2D gesture sets 97.5% for user&time-independent 2D gesture sets The adapted SRC algorithm outperforms other methods SRC recognition rate in face recognition: 92.7 ile 94.7 Naïve Bayes recognition rate in sensor jacket apps: 65-97% HMM recognition rate 71.50-99.54%
We are still far from recognising Gehry’s sketches This means that explaining the 3D versions of these phenomena would require postulating a different mechanism and a different form of representation – one that itself could not take the form of a neural display since there are no known 3D neural displays that map space .
FUTURE PROJECT
Augmented Reality a live, copy, view of a physical, real-world environment whose elements are augmented by computer-generated sensory input such as sound, video, graphics or GPS data
iDesign A Multimodal Augmented Reality System for Spatial Design Using the GPS location, accelerometer and gyroscope of the smart tablet, and Google glasses, we will generate a mobile AR system. The AR system (I-DeSIGN) will facilitate design communication by using 3D architectural objects such as walls and windows to push and pull to shape and create a virtual built environment, whilst the architect has the benefit of having a superimposed image of the virtual world in physical reality.
iDesign A Multimodal Augmented Reality System for Spatial Design Linking this system with graphics packages such as Google Earth and Google Sketch Up, we will be able to use simple operations and digital libraries for creating quick space layouts. Thus, the AR system will facilitate communication between various stakeholders in the construction industry, while developing open standards to share data. I-DeSIGN will recognise speech and gestures of an architect , pass commands to a 3D modeling package via a gesture recognition system, generate the model of the 3D space, and superimpose it on physical reality displaying it on a smart tablet . We envisage I-DeSIGN as a ubiquitous interactive system and a vehicle to transfer design concept to a virtual built environment.
The 4Any Mobile AR Framework
ArcHIVE 4Any
ArcHIVE: Digital Heritage using Mobile Augmented Reality Aim of this project is to explore users’ interaction with digital heritage. Using the GPS location, accelerometer and gyroscope of the smart tablet, and Google glasses, we have generated a mobile AR system. The AR system (ArcHIVE) facilitates the communication between the user and a heritage site offering a CyberGuide at specific GPS locations. Future Projections: We aim at recreating a historical site as a virtual built environment in a 3D space, and completing the missing elements of the remaining section of the digital heritage using augmented reality and mapping the GPS locations provided by a smart tablet. The outcome will be a digitally completed heritage model projected on a see-through head mounted display with head tracking facilities.
CyberGuide: Learning Digital Heritage using Mobile Augmented Reality We have developed a prototype system for the CyberGuide using the historical buildings in a number of cities (such as, Chalon sur Saone, Istanbul, Safranbolu, Ephesus, and Gallipoli peninsula). The system is currently linked with Google Earth and Google maps. We have 3D video records of buildings in Chalon sur Saone , textual info and audio recordings and photographs of important historical buildings in other locations.
CyberGuide: Sketch (6 months) We have 3D video records of buildings in Chalon sur Saone, textual info and audio recordings and photographs of important historical buildings in other locations. In 2015 between Feb-August, our target is to link ArcHIVE system with a 3D modelling tool such as AutoCAD or Google Sketch Up to complete the front façade of a historical building and mapping this on the current facade in the videos or photographs. What is expected of the intern is: to get familiar with the prototype (1 month – Feb), to do scripting and coding to model a front façade (2 months – March-April), to conduct experimental studies to test the usability of the system with 20 university students in Sydney, Australia (1 month May) to analyse the collected data (1 month June), and to draw conclusions to improve the system in the next iteration and to write up an internship report (1 month July).
CyberGuide-Sydney: Digital Heritage using Mobile Augmented Reality (3 months) We have a collection of 3D video records, textual info, audio recordings and photographs of important historical buildings in many locations in the world, however, we have not done this for Sydney. Our goal in this 3 month project to add Sydney section to the Digital Heritage projects with appropriate links to video records and 3D models: http://www.youtube.com/watch?v=JCkRORnRIJc What is expected of the intern is: to get familiar with the prototype and add to this a collection of important landmarks in Sydney visiting these, recording GPS locations, taking photographs, and compiling the information to be presented by the CyberGuide (1 month – June), to conduct experimental studies to test the usability of the system with 20 people in Sydney, Australia (1 month -July) to analyse the collected data and to draw conclusions to improve the system in the next iteration and to write up an internship report (1 month -August).
Internship Project Topic: iDesign Sketch v1 A Multimodal Augmented Reality System for Spatial Design We have already developed a prototype system using the historical buildings in a number of cities. The system is already linked with Google Earth and Google maps . We have 3D video records of buildings. In 2015 between Feb-August, our target is to link this system with Google Sketch Up to design the front façade of a building next to the historical buildings mapped in the current prototype. What is expected of the intern is: to get familiar with the prototype (1 month – Feb), to do scripting to sketch a front façade (1 month – March), to conduct experimental studies with 20 university students in Sydney, Australia (2 months April-May) to analyse the collected data (2 months June-July), and to draw conclusions to improve the system in the next iteration and to write up an internship report (1 month August).
Expected Outcomes Investigation of how to design ubiquitous systems to turn the streets we live into an open museum with no walls. Implementation of a system model to support citizens’ communication with digital cultural heritage sites Integration frameworks for speech and gesture recognition technologies Development of models for Semantic Annotation Usability testing in a number of cultural heritage locations in Europe, Asia, and Australia Data Analysis and Evaluation Development of a cross-cultural cognitive framework Dissemination of results
Strategic Plan I To develop and share AR tools for 3D modelling To develop and share frameworks and methodologies Semantic coding and annotations for Digital Heritage Cognitive coding and protocol analysis To share PhD students Co-supervision of existing students Cotutelle scholarships (e.g., ENSAM, ParisTech & Polytech) Arranging student & researcher exchange Scientific Mobility Program http://ambafrance-au.org/Scientific-Mobility-Program-2014 Applications will re-open mid-December 2014 for travel in 2015 Visits Conferences
Strategic Plan II Joint grant applications ARC Discovery & Linkage grants with CNRS being an OI Industry support Smart Cities (e.g., contact Grand Chalon) BIM (e.g., contact Veolia and Bouygues) CRC Grants with CNRS being an OI National and EU Grants in France with MQ being an OI the Creative Europe and Horizon 2020 Programmes. Horizon 2020 is the new EU Framework Programme for Research and Innovation, with nearly € 80 billion available from 2014 to 2020. The EU Culture programme launched in 2013 has been funding a project titled “Cultural Heritage Counts for Europe: Towards an European Index for Valuing Cultural Heritage” to ensure that Europe’s cultural heritage is safeguarded and enhanced.
EU Framework #4 Content technologies and information management / 2014-2015 • Addresses: - Big Data with focus on both innovative data products and services and solving research problems - Machine translation to overcome barriers to multilingual online communication - Tools for creative, media and learning industries to mobilise the innovation potential of SMEs active in the area - Multimodal and natural computer interaction • Organised in eight topics: • Big data innovation and take-up • Big data research • Cracking the language barrier • Support to the growth of ICT innovative creative industries SMEs • Technologies for creative industries, social media and convergence • Technologies for better human learning and teaching • Advanced digital gaming/gamification technologies • Multimodal and natural computer interaction
What brings us together? While cultural heritage is central to the European Agenda, it is equally important for the Commonwealth of Australia, due to its significant contribution to the following objectives: promotion of cultural diversity and intercultural dialogue promotion of culture as a catalyst for creativity – heritage contributes through its direct and indirect economic potential, including the capacity to underpin our cultural and creative industries and inspire creators and thinkers promotion of culture as a vital element of the European Union's and Australia’s multi-cultural and multi-national dimension
What can we do? To strengthen our common position in the field of cultural heritage preservation, there is a need to: encourage the modernisation of the heritage sector, raising awareness and engaging new audiences apply a strategic approach to research and innovation, knowledge sharing and smart specialization; seize the opportunities offered by digitisation; to reach out to new audiences and engage young people in particular.
What else can we do? In particular: allow users to engage with their cultural heritage and contribute their own personal experiences, e.g. in relation to landmark historical events such as World War I. Therefore, inclusion of locations in Asia and Australia should be also considered. promote the development of sustainable, responsible and high- quality tourism, including products linked with cultural and industrial heritage and create cultural routes crossing several countries and joining them in a common narrative, such as the "EU sky route" aimed at putting Europe on the Worldwide Tour of Astro-Tourism or the "Liberation Route Europe" around 1944-45 events.
What else can we do? Audience development is a key priority of the programme. The heritage sector will be encouraged to experiment with new ways of reaching more diverse audiences, including young people and migrants . This may require the use of smart phones and tablets for a smart tour within a smart city context . The EU Commission, in cooperation with the Council of Europe, will also promote heritage-based and local-led development within the territory of the Union, by identifying new models for multi- stakeholder governance and conducting on-site direct experimentations . The Commission now invites all stakeholders to develop a more integrated approach .
Strategic Plan III Local grant applications NSW Multicultural Advantage Grants Program Multicultural Partnership Grants (up to 3 years $80K) To maximise linguistic and cultural assets of NSW population bringing together two or more organisations Unity Grants (up to $30K) to build relationships between multicultural and Aboriginal communities closes on 14 th Nov Community Inclusion Grants (up to $20K) with a particular focus on mentoring and inter-cultural activities that bring diverse groups of people together 2014/15 Community Applications will open later this year.
Thank you! We are all looking for an answer but in fact what drives us is the question. Future isn’t written. It is designed. Questions? manolya.kavakli@mq.edu.au
PAST PROJECTS & Findings How can we investigate user interaction with architectural/design objects? METHODOLOGIES & PILOT STUDIES on Speech & Gesture Recognition Cognitive Processing
Are speech and gesture processing Independent or Integrated systems? There are two main hypotheses relating to the relationships between speech and gestures: The ‘independent systems ’ framework hypothesis holds that gesture and speech are autonomous and separate communication systems. The alternative hypothesis “ integrated-system ” is that gesture and speech together form an integrated communication system. Finding 0: We found more empirical evidence for the “integrated system” hypothesis.
Can Speech and Gesture be integrated? Liu (2013) and Kavakli examined existing video and audio recordings and dissected their contents including the lexical, gestural but also the lexical categories.
David McNeil’s Gesture Classifications
McNeill’s gesture classification: iconic (resemble what is being talked about e.g. flapping arms when mentioning a bird), metaphoric (abstractedly pictorial, e.g. drawing a box shape when referring to a room), beat (gestures that index a word of phrase e.g. rhythmic arm movement used to add emphasis), deictic (gestures pointing to something, e.g. while giving directions).
ANVIL coding specification Prep preparation phase, bringing arm and hand to the stroke’s starting position. This means the limb moves away from a rest position into the gesture space where it will begin the stroke. Stroke the most energetic part of the gesture movement and also the requisite part of a gesture. A gesture is not said to happen with stroke phase absent. It is also the gesture phase with meaning and effort. Hold are optional still phases which can occur before and/or after the stroke, usually used to defer the stroke so that it coincides with a certain word. The hold can be a "post-stroke" hold or "pre-stroke" hold. Recoil directly after the stroke the hand may spring back so as to emphasise the harshness of the stoke. Retract returns the arms to a rest pose (i.e. arms resting on the chair, folded, in lap )- not always the same position as at the start. Partial-retract retraction movement that is stopped midway to open another gesture phase.
Speech Coding Each iconic and metaphoric gesture is related to at least one word. We coded words frequently used to identify which words were accompanied by gestures adjectives, parts of the chair, verbs, order and shapes.
Protocol Analysis The primary empirical method for studying design (Ericson and Simon, 1984) Design thinking is induced from the behaviour captured from the protocol including verbalisations (speech), drawings, and gestures . Critiques: PA does not address well the differences between internal and external representations (Chi, 1997) There is a gap between the levels of description and humans’ perception of what they are doing (Dorst, 1997) Designer mentally constructs a design world (Schon, 1988, Trousee and Christiaans, 1996) beyond the entitites, attributes and relations, including mental simulations beyond the parameters of a state space (Schon, 1992, Dorst, 1997)
Physical Actions D-actions: drawing actions M-actions: moves Dc: create a new depiction Moa: motion over an area Drf: revise an old depiction Mod: motion over a depiction Dts: trace over the sketch Mrf: move attending to relations or features Dtd: trace over the sketch on a different sheet Ma: move a sketch against the sheet beneath Dsy: depict a symbol Mut: motion to use tools Dwo: write words Mge: hand gestures Perceptual Actions P-actions: P-actions: P-actions: perceptual actions related to implicit perceptual actions related to features perceptual actions related to relations spaces Psg: discover a space as a ground Pfn: attend to the feature of a new Prn: create or attend to a new relation depiction Posg: discover an old space as a ground Pof: attend to an old feature of a Prp: discover a spatial or organizational depiction relation Pfp: discover a new feature of a new Por: mention or revisit a relation depiction
Functional Actions F-actions:Functional actions F-actions:Functional actions F-actions:Functional actions related to implementation related to new functions related to revisited functions Fn: associate a new depiction, feature Fo: continual or revisited thought of a Fi: implementation of a previous or relation with a new function function concept in a new setting Frei: reinterpretation of a function Fop: revisited thought independent of depictions Fnp: conceiving of a new meaning independent of depictions Conceptual Actions G-actions: Goals Subcategories of G1 type goals: G1: goals to introduce new functions G1.1: based on the initial requirements G2: goals to resolve problematic conflicts G1.2: directed by the use of explicit knowledge or past cases (strategies) G3: goals to apply introduced functions or arrangements in G1.3: extended from a previous goal the current context G4: repeated goals from a previous segment G1.4: not supported by knowledge, given requirements or a previous goal
Retrospective Protocol Analysis so I am going to have to segment this a little bit. Something has to be here and Segment something back here. And I am not going to bisect the main space. no: 248 Action type index class Description Dependency (where, of what, among what?) index On what Drawing Dc new Circle 3 Looking L1 old Line 67 Moves Perceptual New i-space The rest space Psg spatial rel (separate): the two spaces New l-relation New/ne Dc, Psg Prn1 spatial rel (included): the new space new g-relation w Dc, L1 Prn2 is on the side of the building New/old Functional Goals type content Source Trigger Seg/typ what? e I am not going to bisect the main space of the building Type 2 256 Type1.3 I am splitting the building on the side, not in the Type1.3 Prn1, Prn2 center
Experiment 1 Volunteers: 18 participants (9 males and 9 females) were filmed. Their ages varied from 20 to 50. not necessarily all native speakers, but spoke English fluently had at least 6 months experience of living in Australia.
Findings of E1 (similarities) Regarding the differences in the integration or alignment of speech and hand gestures, we found that, generally speech and hand gestures are tightly synchronised with each other. Males and females actually have similar integration patterns gestures precede the related speech within 2 seconds and have overlaps with corresponding lexical affiliates on the time axis. In our annotations for female participants, 81.15% of hand gesture strokes precede the related lexical affiliates. For male participants, it is even higher (89.39%).
Findings of E1 (differences) However, the temporal alignment of speech and hand gestures varies for males and females. The time lags between speech and co- occurring hand gestures are shorter for females than males. Also our findings showed that the duration of gesture strokes and related keywords are significantly different in males and females.
Findings of E1 These findings suggest that gender is a significant factor in the integration of speech and hand gestures for the design of MMIS. Adaptive integration strategies for different gender groups may improve the performance of systems.
Are Females’ Information Processing different from Males? In cognitive analysis, we found that females have more cognitive actions for same tasks. Females give more attention to details on different parts of the objects compared to males. More cognitive actions may indicate more frequent brain activities , which can cause strong brain waves with significant changes. The significant spectral moment in the brain for females may also imply faster brain activities associated with speech and hand gestures, which may be the reason for shorter integration time of speech and hand gestures for females.
Experiment 2 Fourteen (14) participants (7 females and 7 males) participated in our second experiment involving EEG signal collection. Each participant was required to speak a number of keywords extracted from the first experiment using hand gestures while they were wearing Emotiv Neuroheadset with their eyes closed. In total, 10 keywords were used.
Findings of E2 However, when they use speech and hand gestures coordinated together, we observe that beta spectral moment waves are stronger in females and the changes of spectral moment from alpha to beta bands are more significant for females. The significant spectral moment in brain waves may imply faster brain activities for females when use speech and hand gestures coordination, may be the reason for shorter integration time of speech and hand gestures for females .
Potential reasons for gender differences Gender differences in grey and white matter are also reported by others: ”In general, men have approximately 6.5 times the amount of gray matter related to general intelligence than women, and women have nearly 10 times the amount of white matter related to intelligence than men. Gray matter represents information processing centres White matter represents the networking of - or connections between - these processing centres”. Those connections may allow a woman’s brain to work faster than a man’s.
Experiment 3 8 males and 10 females 5+5 were chosen Total number of gestures: 157 25-30 years old Asian and Australian Professionals or university students
Differences in the use of gestures females use more gestures in a longer period (84 vs 72 gestures and 2:01 vs 1:28 seconds on average). frequency of gestures is higher in females (2.39 vs 1.78). males perform less number of gestures in a shorter time frame (25.6 sec vs 40.2 sec).
Findings of E3 There are gender differences in the use of gestures and the frequency and types of gestures used.
Gesture types Males Females There are no metaphoric gestures, no repetitions in males. Females use less number of beats and junk gestures.
Are there cultural differences in speech and gesture-based interaction? Culture We are locked into our cultural perspectives and mindsets, while using a language. Culture does not exist as a computational term in HCI. The software of the machines may be globalized, but the software of the minds that use them is not. Hofstede G. H., Hofstede G. J, Minkov Michael (2010) Cultures and organizations: software of the mind : intercultural cooperation and its importance for survival. (3rd ed) McGraw-Hill Professional ISBN 0071664181, 9780071664189
Experiment 4 The participants are asked to describe two chairs to the camera. We obtained approx 10 minutes of monologue object descriptions in a video footage. 10 participants divided in two groups. Anglo-Celtics with English as a first language English descendants (British or Irish ancestry). Latin Americans with English as a second language Mexican (3), Columbian (1) and Chilean (1) proficient bilinguals with English as their second language and all have been in living in an English speaking country (Australia) for the past 6 months.
The Hofstede’s model of dimensions of national culture Power Distance is the acceptance and expectation of power to be distributed unequally. Uncertainty Avoidance indicates the extent to which the members of society feel uncomfortable or comfortable in an ambiguous or abnormal situation. Individualism is the extent to which individuals are merged into groups. Masculinity refers to the distribution of emotional roles between the genders, and also serves to classify a culture as assertive/ competitive (masculine) or modest/caring (feminine). Long-Term Orientation . Countries with high Long-Term Orientation (LTO), foster pragmatic virtues oriented towards future rewards, in particular saving money, persistence, and adapting to changing circumstances.
HOSFTEDE’S 5D MODEL COMPARING ANGLO-CELTIC AND LATIN AMERICAN
Hall’s classification of cultures In a high context culture the Middle East, Asia, Africa, and South America, many things are left unsaid, letting the culture explain. There is more non-verbal communication, a higher use of metaphors, and more reading between the lines. In a low context culture including North America and much of Western Europe, the emphasis is on the spoken or written word. They have explicit messages, focused on verbal communication, and their reactions could be visible, external and outward.
Assumption Anglo-Celtic cultures (e.g. Australian, British, Irish, and New Zealanders) categorize as low context cultures Latin Americans (American countries where Spanish and Portuguese are primarily spoken) correspond to the high context cultures. Anglo-Celtic may predominantly use words, while the Latin Americans would use gestures.
3 metrics Gesture Type. certain types of gestures could be attributed to different cultures Frequency. the number of gestures performed by a participant divided by the period of the gesture of the same participant. Occurrence. If certain gestures are culture-oriented or task-oriented (i.e., related to the task being performed).
EXPERIMENTAL RESULTS Metrics Chair Avg gesture Total no of Avg gesture Sample Avg gestures SD Frequency duration gestures Time Chair 1 Anglo-Celtic 1.84 65 12.8 5.63 22.74 0.56 Latin Chair 1 1.49 59 11.8 2.16 17.81 0.66 American Chair 2 Anglo-Celtic 1.73 65 13 7.17 23 0.56 Latin Chair 2 1.67 43 8.6 2.88 14.22 0.60 American
Results Anglo-Celtics did not display too much variation between chair descriptions The standard deviation was again higher used more gestures on average to describe Chair 2 (Abstract chair) The reason behind this could be the degree of comfort in using a language when describing complexity. Latin Americans used less number of gestures to describe the same chairs as Anglo-celtics. had a smaller standard deviation and more frequent gestures in both chairs , shorter, concise, and common gestures by most of the participants. smaller count of gestures by Latin Americans is justified by less time in which they performed the gestures.
Latin Americans gesture frequency is higher in Chair 1 compared to Anglo-celtics, and increases in Chair 2 when the chair is more abstract. This could be because Latin Americans scored higher results in junk gestures in Chair 2. used more words for Chair 1 and less in Chair 2 the lack of vocabulary. The higher word count for Chair 1 must mean a higher degree of confidence, or more predictable and structured ideas Samples Words Anglo-Celtic Latin American Both Total Chair 1 9 13 6 28 Chair 2 13 10 5 30
Recommend
More recommend
