Crowdsourced Conformance Testing via Remote Sandboxing Joe Gibbs - PowerPoint PPT Presentation

Crowdsourced Conformance Testing via Remote Sandboxing Joe Gibbs Politz

http://www.batchgeo.com

“Remote Sandboxing?”

Students' interpreters g r a d e . r k t 1 6 0 0 t i m e s ?

Students' interpreters g r a d e . r k t

# l a n g r a c k e t / b a s e ( r e q u i r e ' i n t e r p . r k t ' ) ( c o m m a n d - l i n e “ - - t e s t ” t e s t - d i r ( r u n - t e s t s i n t e r p t e s t - d i r ) )

# l a n g r a c k e t / b a s e ( r e q u i r e ' i n t e r p . r k t ' ) ( c o m m a n d - l i n e “ - - t e s t ” t e s t - d i r ( r u n - t e s t s i n t e r p t e s t - d i r ) “ - - r e p o r t ” ( s i g n ( r u n - t e s t s i n t e r p o u r - t e s t s ) “ s o o p e r - s e e k r e t ” ) )

# l a n g r a c k e t / b a s e ( d e f i n e i n t e r p ( d y n a m i c - r e q u i r e “ i n t e r p . r k t ” ' i n t e r p ) ) ( c o m m a n d - l i n e “ - - t e s t ” t e s t - d i r ( r u n - t e s t s i n t e r p t e s t - d i r ) “ - - r e p o r t ” ( s i g n ( r u n - t e s t s i n t e r p o u r - t e s t s ) “ s o o p e r - s e e k r e t ” ) )

Students' interpreters g r a d e . b i n 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 1 1 1 1 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 1 1 1 1 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 1 1 1 1 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 Remote Sandboxing

“Crowdsouced Conformance Testing”?

ParselTongue Testing Assignment • tests/ • loop-tests/ correct-interp ✓ • for-test1.psl • for-test2.psl ✓ for-update-after-test • operator-tests/ • subtract-test1.psl ✓ plus-coerces-strings • subtract-test2.psl • add-test1.psl arity-before-evaluate • function-tests/ • apply1.psl discard-if-effects • apply2.psl . • closure1.psl • closure2.psl . .

> 10,000 tests from 250 people That's as many tests as JavaScript's conformance suite. Crowdsourced Conformance Testing

Lessons ● Racket's dynamic code support is awesome, and Racket's build tools for distribution just work ● Value proposition for teaching PL online – get crowdsourced tools and tests