Introduction to Vocareum Virtual Labs Sanjay Srivastava David Lin - PowerPoint PPT Presentation

Webinar Introduction to Vocareum Virtual Labs Sanjay Srivastava David Lin David Joyner, PhD CEO Business Development Georgia Tech March 26, 2020

Agenda • Vocareum Introduction - Sanjay Srivastava - 10 mins • Teaching Computer Science at Scale - David Joyner - 30 mins • Q&A

Mission ( incl. skills-based exams) Deliver virtual labs and assessments (infrastructure, tools, content) (~ 500K learners) securely, on budget, and at scale (user roles, policies, spend)

Academic Partners Delft University Indiana University RMIT University University of Illinois … over 600 institutions using Vocareum Labs today …

Supporting broad range of subject areas Data Science Programming Engineering Fully hosted notebook solutions; Broad range of languages; Stream desktop & applications; Pre-installed packages Standard IDE, spyder, eclipse. License flexibility, Raspberry Pi C/C++ Big Data Machine Learning Cloud Computing Hadoop processing; Any ML framework; Policy controlled, budgeted; Run MapR on Clusters Dedicated GPU Aggressive resource mgmt Database Full Stack DevOps & Cyber Security MySQL, PostgreSQL; Front- and back-end services; Range of tools and infrastructure; Neo4j, DynamoDB Business logic; Databases Linux; Sudo/Root/Admin

Assessment Options Automated Manual In-Context Grading Grading Feedback Competition Mastery Peer Leaderboard Learning Review

Using diverse architecture to support labs • Web Applications • Streaming Linux Applications • Batch Compute Clusters • Cloud Infrastructure • BYO Applications + Infrastructure • Network of Docker Containers • Network of Virtual Machines • AppStream

Open source web applications/frameworks • Jupyter Notebook • Hadoop applications, including Spark • R Studio • ML frameworks - TensorFlow, • Terminal PyTorch • MySQL • Eclipse Theia • MongoDB • Cloud 9 • Neo4J • Node • Postgres Supporting a variety of languages: • C, C++, Python, R, Java, Scala, Go, PHP , etc.

Open source desktop applications • MySQL Workbench • Eclipse • Rattle (for R) • Spyder • Firefox browser • pgAdmin Others can be easily added upon request

Learning Management & Assessment • Enrollment • Hosting Course Material • Grading • LTI integrations

Supporting Exams • Synchronous exams with 500 students • Asynchronous exams (90 minutes over 3 days) • Scheduled proctored certification tests • Exams with a passcode provided by proctor • Check whether learners came from a specific IP “I can see how it is very challenging to scale up the system so all students can access and work range on the exam at the same time. • Deliver random tests from a bank of problems In previous years, we used our own online platform to handle students’ submissions. But, • Built in plagiarism detection tests. we found Vocareum to be much more maintainable and has better features. Using Vocareum really helps us: we know who submitted, at what time they submitted, and then we can run an auto-grading job immediately.” Dr. Oka Kurniawan, SUTD

CONFIDENTIAL Student Lab Interfaces

Supporting Advanced Coursework & Research • Long running jobs • Large data sets • Additional resources • Cloud - AWS • Databases - MySQL, Mongo, Neo4j • Aggregated labs + tools

Deploying Vocareum Hands-on Labs / MOOC Skills-based Exams Labs for learners to use tools, technology and services Administer proctored skills-based certification in managed environment (policy, budget, etc) exams, with automatic scoring Training / Bootcamps / Hack-a-Thons Research Course material + lab environments to support Team-based lab environment with data security, instructional training events grant management, etc. Virtual Campus Labs Syndication Course material + labs packaged and distributed to Upgrade on-prem computer labs with Universities, partners, etc. purpose-built cloud labs and tools

Security Customer VPC Public Subnet Private Subnet servers proxy server internet gateway distributed file system NAT gateway object storage

REST API

CONFIDENTIAL Thank you!

CS1 Online & At Scale Vocareum Webinar Series David Joyner May 28, 2020

Goal: Scale up the availability of intro-level computer science education. Solution: Georgia Tech's online CS1301: Introduction to Computing in Python course, available as both a massive open online course (MOOC) and a for-credit offering.

Course Videos Integrated Exercises Programming Exercises CS1301 CS1301 Adaptive Textbook Introduction to Introduction to Computing Computing On Online Problem Sets Proctored Exams Recitations & TAs

~500 2-minute videos Course Videos ~500 integrated multiple choice and text entry exercises Integrated Exercises ~200 integrated programming exercises with live feedback Programming Exercises Textbook with ~1000 multiple choice and text entry exercises Adaptive Textbook ~150 programming and ~500 multiple choice/ text entry problems Problem Sets Proctored exams occurring within a browser-based IDE Proctored Exams Traditional in-person recitations, graders, and teaching assistants Recitations & TAs

Course Videos Integrated Exercises Free MOOC Programming Exercises Verified MOOC Adaptive Textbook Certificate MOOC Series Problem Sets Proctored Exams Georgia Tech Course Recitations & TAs

Opened July 2018, Free Free MOOC Series Opened February 2017, Free Free MOOC Opened February 2017, $100 Verified MOOC Opened July 2018, $400 Certificate MOOC Series Opened January 2017, Tuition Georgia Tech Course Opened August 2017, Tuition Honors Program Course Opened January 2018, Free* Dual Enrollment Course

Course Content Video Course Adaptive Textbook Partnerships Platform Platform Proctoring Platform Assignment Platform

Video Content 20% Headshot 80% Screencapture All Course Content Available at bit.ly/CS1301PC

Exercises Programming Problems Multiple Choice & Text Entry All Course Content Available at bit.ly/CS1301PC

Integrated Exercises All Course Content Available at bit.ly/CS1301PC

Random Test Cases All Course Content Available at bit.ly/CS1301PC

Sample Answers All Course Content Available at bit.ly/CS1301PC

Timed, Proctored Tests All Course Content Available at bit.ly/CS1301PC

Total Volume: Over 500 coding problems All Course Content Available at bit.ly/CS1301PC

Automated Evaluation Student Code Phineas Ferb (Local (Global Autograder) Autograder) Grade Feedback Results All Course Content Available at bit.ly/CS1301PC

2018 2017 2019 J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D Georgia Tech Course Honors Program Course DE Course Free MOOC Free MOOC (Series) Verified MOOC Certificate MOOC (Series)

Enrollment Over Time (Georgia Tech) 1800 1600 1400 1200 1000 800 600 400 200 0 2017 2018 2019

Enrollment Over Time (All Versions) Georgia Tech Free MOOC Verified MOOC Free MOOC Series Verified MOOC Series 250000 200000 150000 100000 50000 0 2017 2018 2019

Total Course Completers by Version Georgia Tech, 2069 Free MOOC, 678 Certificate MOOC Series, 5564 Verified MOOC, 239 Free MOOC Series, 3300

Research Findings rm as Students in CS1301 online class pe perf rform well as 1 or better than 2 students in a traditional version of CS1301 in a pre- test/post-test setup. 1: Joyner, D. A. (2018). Toward CS1 at Scale: Building and Testing a MOOC-for-Credit Candidate. In Proceedings of the Fifth Annual ACM Conference on Learning at Scale. London, United Kingdom. ACM Press. 2: Joyner, D. A. & McDaniel, M. (2019). Replicating and Unraveling Performance and Behavioral Differences between an Online and a Traditional CS Course. In Proceedings of the ACM Global Computing Education Conference (CompEd). Chengdu, China. ACM Press.

Research Findings Students who previously failed or withdrew prefer the online version, from a CS class pre and perform better in the online version than the traditional version 1 . 1: Joyner, D. A. (2018). Intelligent Evaluation and Feedback in Support of a Credit-Bearing MOOC. In Proceedings of the 19th International Conference on Artificial Intelligence in Education. London, United Kingdom. Springer.

Research Findings Students rate the online version of CS1301 quality than other college courses as higher qu they have taken 1 . 1: Joyner, D. A. (2018). Toward CS1 at Scale: Building and Testing a MOOC-for-Credit Candidate. In Proceedings of the Fifth Annual ACM Conference on Learning at Scale. London, United Kingdom. ACM Press.

Awards Class Central Top 100 Course of Bronze Award: Computing & IT Discipline All-Time (2019) 2019 Reimagine Education Awards

Questions? For more: bit.ly/CS1301PC (the whole course in MOOC format) LucyLabs.gatech.edu (research on the course) DavidJoyner.net (me!)