making sense of multimodal learning analytics mmla
play

Making Sense of Multimodal Learning Analytics (MMLA) Marcelo - PowerPoint PPT Presentation

Jun 02, 2023 •307 likes •721 views

Making Sense of Multimodal Learning Analytics (MMLA) Marcelo Worsley Assistant Professor, Learning Sciences & Computer Science Agenda - What is MMLA? - Why use MMLA? - How to use MMLA? - Examples of MMLA research - On-going challenges and

  1. Making Sense of Multimodal Learning Analytics (MMLA) Marcelo Worsley Assistant Professor, Learning Sciences & Computer Science

  2. Agenda - What is MMLA? - Why use MMLA? - How to use MMLA? - Examples of MMLA research - On-going challenges and future directions - Resources

  3. What is MMLA? Learning Analy9cs—a set of mul9-modal sensory inputs, that can be used to predict, understand and quan9fy student learning. (Worsley and Blikstein, 2011)

  4. What is MMLA? Mul9modal learning analy9cs (MMLA) (Blikstein & Worsley, in press.; Blikstein, 2013; Worsley, 2012) sits at the intersec9on of three ideas: mul9modal teaching and learning, mul9modal data, and computer-supported analysis. At its essence, MMLA u9lizes and triangulates among non-tradi9onal as well as tradi9onal forms of data in order to characterize or model student learning in complex learning environments. However, as we describe later, the ways that researchers u9lize mul9modal data vary widely. (Worsley, Abrahamson, Blikstein, Grover, Schneider and Tissenbaum, 2016)

  5. What is MMLA?

  6. What is MMLA?

  7. What is MMLA?

  8. VIDEO AUDIO TEXT EYE TRACKING DATA DEPTH CAMERA Pose Speaker Diariza9on Emo9on/Affect Fixa9ons Gestures Gaze Prosody Complexity Fixa9on Paeern Body Posi9on Gestures Spectrum Cohesion Pupil Dila9on Joint Tracking Object Manipula9on Facial Expressions Emo9on/Affect Seman9cs Eye loca9on Proxemics Intensity Syntac9cs Aeen9on Scene Understanding Speaking Rate Content Head Pose Movement Collabora9on/Turn-Taking Heartrate Voice Quality Object Tracking SMARTPHONES LEAP MOTION EDA WATCH OTHER MODALITIES EVENT LOGS Signg/Standing Gestures Arousal fMRI Ac9vity Loca9on Tool Usage Cogni9ve Load ELECTROCARDIOLOGY Noise Level Hand Movement Body Temperature ELECTROENCEPHALOGY Finger Movement Gestures ELECTROMYOGRAPHY Heartrate ELECTROGASTROGRAPHY Stress

  9. Why use MMLA? - Teaching and learning are mul9modal - Study and support complex learning environments - See the hard to see - Inform design of mul9modal technologies - Expand no9ons of learning to non- tradi9onal modali9es - Improve accessibility and inclusivity - Triangulate across modali9es

  10. Why use MMLA? - Visualizing/Represen9ng informa9on for human inference - Predic9on of indicators - Data-driven interven9ons - Evalua9ng conjecture-based learning designs

  11. How to use MMLA?

  12. How to use MMLA?

  13. How to use MMLA? DATA CAPTURE SOFTWARE Open Social Signal Interpreta9on Lab Streaming Layer Open Pipe Kit Open Broadcaster Sohware Mul9sense (AV Recorder) iMo9ons Aeen9on Tool

  14. How to use MMLA? DATA CAPTURE SOFTWARE Open Social Signal Interpreta9on Lab Streaming Layer Open Pipe Kit Open Broadcaster Sohware Mul9sense (AV Recorder) iMo9ons Aeen9on Tool

  15. How to use MMLA? DATA CAPTURE SOFTWARE Open Social Signal Interpreta9on Lab Streaming Layer Open Pipe Kit Open Broadcaster Sohware Mul9sense (AV Recorder) iMo9ons Aeen9on Tool

  16. How to use MMLA? DATA CAPTURE SOFTWARE Open Social Signal Interpreta9on Lab Streaming Layer Open Pipe Kit Open Broadcaster Sohware Mul9sense (AV Recorder) iMo9ons Aeen9on Tool

  17. How to use MMLA? DATA CAPTURE SOFTWARE Open Social Signal Interpreta9on Lab Streaming Layer Open Pipe Kit Open Broadcaster Sohware Mul9sense (AV Recorder) iMo9ons Aeen9on Tool

  18. How to use MMLA?

  19. How to use MMLA? Pre-processing is important for data synchroniza9on, accoun9ng for individual differences between par9cipants, and gegng data in the appropriate format for data extrac9on. It tends to vary by data type.

  20. How to use MMLA?

  21. AUDIO VIDEO TEXT EYE TRACKING DATA DEPTH CAMERA HTK OpenFace (gaze) Natural Language Toolkit Praat PyGaze OpenFace (face recogni9on) OpenNUI OpenSmile Lightside Ogama Emo9ent FACET OpenEar LIWC Kinect for Windows Covarep Noldus Face Reader EyeTrackingR ELAN Cohmetrix OpenCV QSRLib Tobii SDK ICSI Diarizer Affec9va Tone Analyzer Matlab SMI SDK Libfreenect Microsoh Face API Stanford Parser LIUM Diarizer iMo9ons Aeen9on Tool Microsoh Emo9on API CMU Sphinx Wordnet Google ASR Intraface Pupil Dila9on Sen9wordnet AT&T Watson ASR Mul9sense Bing Speech API Mallet Eularian Magnifica9on Code Audacity Word2Vec iMo9ons Emovoice DATA EXTRACTION EDA WATCH SOFTWARE Ledalab Open Social Signal Interpreta9on

  22. How to use MMLA?

  23. How to use MMLA? Human Coders

  24. Fusion Let’s say that you want to study engagement and have audio, video, bio-physiological and gesture data available for analysis. How do you use these to compute a measure of engagement?

  25. How to use MMLA?

  27. How to use MMLA?

  28. Examples

  30. Xbox Kinect RapidMiner Researcher(s) - Audio (Talk) - XMeans - Video (Pose/ Gaze) - Gestures (Hand Movement)

  31. Custom Xbox Kinect RapidMiner Researcher(s) - Audio (Talk) - XMeans - Video (Pose/ Gaze) - Gestures (Hand Movement)

  32. Challenges & Future Directions

  33. Simplifying Data Capture & Analysis

  35. Better Visualization/Inference Tools

  36. Best Practices Around Data Fusion and Data Analysis Pipelines

  37. Applications (especially as it relates to inclusive technology and providing feedback to learners and teachers)

  38. Resources Mul9modal Learning Analy9cs Special Interest Group - hep://sigmla.org CrossMMLA Workshops EC-TEL 2017, LAK 2018 (tenta9ve) MMLA Workshops @ ICMI, LAK, ICLS (2012 – Present) SOLAR LASI – Mul9modal Learning Analy9cs Tutorial Workshops Journal of Learning Analy9cs Special Sec9on on Mul9modal Learning Analy9cs CIRCL Cyberlearning Report

  39. Berland, M., Baker, R. S., & Blikstein, P. (2014). Educa9onal Data Mining and Learning Analy9cs: Applica9ons to Construc9onist Research. Technology, Knowledge and Learning , 19 (1–2), 205–220. Blikstein, P., & Worsley, M. (2016). Mul9modal Learning Analy9cs and Educa9on Data Mining : using computa9onal technologies to measure complex learning tasks, 3 (2), 220–238. heps://doi.org/ hep://dx.doi.org/10.18608/jla.2016.32.11 Fouse, A. S. (2011). ChronoViz : A system for suppor9ng naviga9on of 9me-coded data. Chi , 1–6. heps://doi.org/10.1145/1979742.1979706 Grafsgaard, J. F. (2014). Mul9modal Analysis and Modeling of Nonverbal Behaviors During Tutoring. In Proceedings of the 16th Interna=onal Conference on Mul=modal Interac=on (pp. 404–408). New York, NY, USA: ACM. heps://doi.org/10.1145/2663204.2667611 Leong, C. W., Chen, L., Feng, G., Lee, C. M., & Mulholland, M. (2015). U9lizing depth sensors for analyzing mul9modal presenta9ons: Hardware, sohware and toolkits. ICMI 2015 - Proceedings of the 2015 ACM Interna=onal Conference on Mul=modal Interac=on , (3), 547–556. heps://doi.org/10.1145/2818346.2830605 Luz, S. (2013). Automa9c iden9fica9on of experts and performance predic9on in the mul9modal math data corpus through analysis of speech interac9on. Proceedings of the 15th ACM on Interna=onal Conference on Mul=modal Interac=on - ICMI ’13 , 575–582. heps://doi.org/10.1145/2522848.2533788 Morency, L.-P., Oviae, S., Scherer, S., Weibel, N., & Worsley, M. (2013). ICMI 2013 grand challenge workshop on mul9modal learning analy9cs. Proceedings of the 15th ACM on Interna=onal Conference on Mul=modal Interac=on - ICMI ’13 , 373–378. heps://doi.org/10.1145/2522848.2534669 Oviae, S., & Cohen, A. (2013). Wrieen and mul9modal representa9ons as predictors of exper9se and problem-solving success in mathema9cs. … 15th ACM on Interna=onal Conference on Mul=modal … , 1–8. Retrieved from hep://dl.acm.org/cita9on.cfm?id=2533793 Scherer, S., Worsley, M., & Morency, L.-P. (2012). 1st interna9onal workshop on mul9modal learning analy9cs. In ICMI (pp. 609–610). Schneider, B., & Blikstein, P. (2015). Unraveling Students’ Interac9on Around a Tangible Interface using Mul9modal Learning Analy9cs. Journal of Educa=onal Data Mining . Spikol, D. (2017). Using Mul9modal Learning Analy9cs to Iden9fy Aspects of Collabora9on in Project-Based Learning Introduc9on PELARS system and context. Cscl , (June), 263–270. heps://doi.org/ 10.22318/cscl2017.37 Thompson, K. (2013). Using micro-paeerns of speech to predict the correctness of answers to mathema9cs problems: an exercise in mul9modal learning analy9cs. … 15th ACM on Interna=onal Conference on Mul=modal … . Retrieved from hep://dl.acm.org/cita9on.cfm?id=2533792 Worsley, M. (2012). Mul9modal Learning Analy9cs - Enabling the future of learning through mul9modal data analysis and interfaces. Interna=onal Conference on Mul=modal Interac=on , 12–15. heps:// doi.org/10.1145/2388676.2388755 Worsley, M., Scherer, S., Morency, L. P., & Blikstein, P. (2016). Exploring Behavior Representa9on for Learning Analy9cs. Proceedings of the 2015 Interna=onal Conference on Mul=modal Interac=on (ICMI) , 251–258. Xavier Ochoa, Marcelo Worsley, Katherine Chiluiza, & Saturnino Luz. (2014). MLA’14: Third Mul9modal Learning Analy9cs Workshop and Grand Challenges. Proceedings of the 16th Interna=onal Conference on Mul=modal Interac=on , 531–532. heps://doi.org/10.1145/2663204.2668318

  40. Marcelo Worsley Assistant Professor Electrical Engineering and Computer Science & Learning Sciences marcelo.worsley@northwestern.edu Marceloworsley.com 9ilt.northwestern.edu 39

Recommend

Using Multimodal Learning Analytics to Identify Aspects of Collaboration in Project-based Learning

Using Multimodal Learning Analytics to Identify Aspects of Collaboration in Project-based Learning

Using Multimodal Learning Analytics to Identify Aspects of Collaboration in Project-based Learning Daniel Spikol, Malm University, Sweden Emanuele Ruffaldi, Scuola Superiore Sant'Anna,Italy Mutlu Cukurova, University College London, UK CSCL

259 views • 22 slides
Making Sense of Performance in Data Analytics Frameworks Authors: Kay Ousterhout, Ryan Rasti,

Making Sense of Performance in Data Analytics Frameworks Authors: Kay Ousterhout, Ryan Rasti,

Making Sense of Performance in Data Analytics Frameworks Authors: Kay Ousterhout, Ryan Rasti, Sylvia Ratnasamy, Scott Shenker, Byung-Gon Chun Presenter: Zi Wang Why? Commonly Accepted mantras Network IO/disk Straggler Takeways

362 views • 22 slides
MAKING SENSE OF WEB ANALYTICS Why website data is so important What data should you track

MAKING SENSE OF WEB ANALYTICS Why website data is so important What data should you track

MAKING SENSE OF WEB ANALYTICS Why website data is so important What data should you track How to track and report What to do with the data #1 SOURCE OF MARKETING MEASUREMENT 57% of global data and analytics

374 views • 18 slides
The quantity of a small set You perceive the parts and put together the whole can be intuitively

The quantity of a small set You perceive the parts and put together the whole can be intuitively

Making Sense of Number Sense Definition Number sense is .good intuition about numbers Making Sense of Number Sense and their relationships . It develops gradually as a result of exploring numbers, visualizing them in a variety of contexts

339 views • 9 slides
Mobile, Metrics & Mayhem Mobile Analytics Making sense from data your users give you

Mobile, Metrics & Mayhem Mobile Analytics Making sense from data your users give you

Mobile, Metrics & Mayhem Mobile Analytics Making sense from data your users give you Presentation by: Vincent Baskerville | @whoisvince vince baskerville | @whoisvince realize your brand nation prove the business value guide

585 views • 44 slides
Making Sense of Word Sense Variation Rebecca J. Passonneau and Ansaf Salleb-Aouissi Nancy Ide

Making Sense of Word Sense Variation Rebecca J. Passonneau and Ansaf Salleb-Aouissi Nancy Ide

Making Sense of Word Sense Variation Rebecca J. Passonneau and Ansaf Salleb-Aouissi Nancy Ide Center for Computational Learning Systems Department of Computer Science Columbia University Vassar College New York, NY, USA Poughkeepsie, NY, USA

324 views • 8 slides
Multimodal Machine Learning Louis-Philippe (LP) Morency CMU Multimodal Communication and Machine

Multimodal Machine Learning Louis-Philippe (LP) Morency CMU Multimodal Communication and Machine

Multimodal Machine Learning Louis-Philippe (LP) Morency CMU Multimodal Communication and Machine Learning Laboratory [MultiComp Lab] 1 CMU Course 11-777: Multimodal Machine Learning 2 Lecture Objectives What is Multimodal? Multimodal:

2.48k views • 71 slides
Multimodal Machine Learning Louis-Philippe (LP) Morency CMU Multimodal Communication and Machine

Multimodal Machine Learning Louis-Philippe (LP) Morency CMU Multimodal Communication and Machine

Multimodal Machine Learning Louis-Philippe (LP) Morency CMU Multimodal Communication and Machine Learning Laboratory [MultiComp Lab] 1 CMU Course 11-777: Multimodal Machine Learning 2 Lecture Objectives What is Multimodal? Multimodal:

1k views • 72 slides
Sense Making in an IOT World: Sensor Data Analysis with Deep Learning Natalia Vassilieva, PhD

Sense Making in an IOT World: Sensor Data Analysis with Deep Learning Natalia Vassilieva, PhD

GTC 2016 Sense Making in an IOT World: Sensor Data Analysis with Deep Learning Natalia Vassilieva, PhD Senior Research Manager Deep learning proof points as of today Vision Speech Text Other Search & information Interactive voice

351 views • 33 slides
MAKING SENSE OF MEDIA Dr Idil Osman MAKING SENSE OF MEDIA; ENGAGING VULNERABLE COMMUNITIES

MAKING SENSE OF MEDIA Dr Idil Osman MAKING SENSE OF MEDIA; ENGAGING VULNERABLE COMMUNITIES

MAKING SENSE OF MEDIA Dr Idil Osman MAKING SENSE OF MEDIA; ENGAGING VULNERABLE COMMUNITIES Minority communities particularly from refugee and diaspora backgrounds engage with media spaces at varying levels of engagement and for different

271 views • 5 slides
TUFF TUFF TUFF TUFF TUFF TUFF TUFF TUFF MAKING MAKING MAKING MAKING SENSE OF SENSE OF

TUFF TUFF TUFF TUFF TUFF TUFF TUFF TUFF MAKING MAKING MAKING MAKING SENSE OF SENSE OF

TUFF TUFF TUFF TUFF TUFF TUFF TUFF TUFF MAKING MAKING MAKING MAKING SENSE OF SENSE OF SENSE OF SENSE OF THE FRAUD THE FRAUD THE FRAUD THE FRAUD JACK WRAITH MBE JIGSAW! JIGSAW! JIGSAW! JIGSAW! Chief Executive Officer TUFF

537 views • 16 slides
Multimodal Machine Learning Main Goal Define a common taxonomy for multimodal machine learning

Multimodal Machine Learning Main Goal Define a common taxonomy for multimodal machine learning

Multimodal Machine Learning Main Goal Define a common taxonomy for multimodal machine learning and provide an overview of research in this area Introduction: Preliminary Terms Modality : the way in which something happens or is experienced

575 views • 26 slides
Making Sense of Word Sense 24 February, 2011 Deutschen Gesellschaft fr Sprachwissenschaft (DGfS)

Making Sense of Word Sense 24 February, 2011 Deutschen Gesellschaft fr Sprachwissenschaft (DGfS)

Making Sense of Word Sense 24 February, 2011 Deutschen Gesellschaft fr Sprachwissenschaft (DGfS) Gottingen Rebecca J. Passonneau Nancy Ide Vikas Bhardwaj Vassar College Ansaf Salleb Aouissi Outline The word sense conundrum The MASC

716 views • 47 slides
Novartis benchmarking initiative: making sense of AI Mark Baillie (with Conor Moloney &

Novartis benchmarking initiative: making sense of AI Mark Baillie (with Conor Moloney &

10/29/2019 AMDS Clinical Development and Analytics Novartis benchmarking initiative: making sense of AI Mark Baillie (with Conor Moloney & Janice Branson) BBS, Basel November 01, 2019

540 views • 20 slides
Making Sense of Life Psalm 73 Learning to see life from Gods perspective 1 Psalm

Making Sense of Life Psalm 73 Learning to see life from Gods perspective 1 Psalm

Making Sense of Life Psalm 73 Learning to see life from Gods perspective 1 Psalm 73:1 What he said and sang in the temple! God is good! He blesses His own! Truly God is good to Israel, To such as are pure in heart. The

424 views • 16 slides
Making Sense of the World Wide Web Jon Margerum-Leys The University of Michigan

Making Sense of the World Wide Web Jon Margerum-Leys The University of Michigan

Making Sense of the World Wide Web Jon Margerum-Leys The University of Michigan www-personal.umich.edu/~jmargeru/websense/ Making Sense of the Web Origin of this work University of Michigan Digital Library Kathleen Hamel, Tracy

548 views • 22 slides
LEARNING ANALYTICS What is learning analytics? The practice of developing actionable insights

LEARNING ANALYTICS What is learning analytics? The practice of developing actionable insights

LEARNING ANALYTICS What is learning analytics? The practice of developing actionable insights through the collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and

327 views • 13 slides
Japan Making sense of the cyclical and the structural Webcall November 2017 Paul Chesson

Japan Making sense of the cyclical and the structural Webcall November 2017 Paul Chesson

Japan Making sense of the cyclical and the structural Webcall November 2017 Paul Chesson This presentation is for Professional Clients only and is not for consumer use. Head of Japanese Equities 14566/SC/NR/IPJ std UK/Jul17 Making sense

393 views • 27 slides
Reform! Research on learning shows: the entire standards-based Some benefits of

Reform! Research on learning shows: the entire standards-based Some benefits of

Signs of Change Making Sense of Math Makes Sense New standards Focus, Coherence, Rigor Randall I. Charles College and career readiness Carmel, CA Higher expectations rcharles155@icloud.com New curricula New assessments

380 views • 24 slides
Making Sense of It All Keynote Speaker: Joe

Making Sense of It All Keynote Speaker: Joe

Making Sense of It All Keynote Speaker: Joe Lazzaro DANIELS COLLEGE OF BUSINESS | UNIVERSITY OF DENVER DIGITAL MARKETING SUMMIT Social

717 views • 16 slides
MMLA M ARATHI S HALA 2010 1 A GENDA Progress Until Now Objective Background

MMLA M ARATHI S HALA 2010 1 A GENDA Progress Until Now Objective Background

MMLA M ARATHI S HALA 2010 1 A GENDA Progress Until Now Objective Background Bharati Vidyapeeth University, Pune Scope Syllabus Course Levels General Guidelines Class Structure Teachers Survey Student

776 views • 18 slides
Making sense of event attribution: the decision-making and communication context Robert S. Webb

Making sense of event attribution: the decision-making and communication context Robert S. Webb

Making sense of event attribution: the decision-making and communication context Robert S. Webb NOAA Earth System Research Laboratory Physical Sciences Division Boulder, Colorado September, 2012 Knowing and Unknowing Collaborators: John Stein

606 views • 25 slides
The Greatness of God Examining Gods CV The difficulties of making sense of tangled ideas

The Greatness of God Examining Gods CV The difficulties of making sense of tangled ideas

The Greatness of God Examining Gods CV The difficulties of making sense of tangled ideas Drawing metaphors and analogies can help make sense of difficult material PPARs play essential roles in the regulation of cellular

753 views • 74 slides
December 2005 Current Sense Circuit Collection Making Sense of Current Tim Regan, Jon Munson

December 2005 Current Sense Circuit Collection Making Sense of Current Tim Regan, Jon Munson

Application Note 105 December 2005 Current Sense Circuit Collection Making Sense of Current Tim Regan, Jon Munson Greg Zimmer, Michael Stokowski INTRODUCTION Sensing and/or controlling current flow is a fundamen- sensing, or negative supply

1.75k views • 118 slides

More recommend