Exams Gerth Stlting Brodal Algorithms and Data Structures Retreat, - PowerPoint PPT Presentation

Autogenerating Algorithms and Data Structures Exams Gerth Stølting Brodal Algorithms and Data Structures Retreat, Sandbjerg, Denmark, March 3, 2020

Algorithms and Data Structures Cource ADS 10 ECTS, 2 nd semester 1991 - 2003 Written exam ADS 1 5 ECTS, 2 nd semester 2004 - 2014 Multiple choice + short answers 2015 - 2017 Multiple choice ADS 2 5 ECTS 2 nd semester 2004 - 2017 Written ADS 10 ECTS, 1 st semester 2017 Multiple choice + written question 2018 - Multiple choice 0 𝑥 = question weight no answer 𝑥/𝑟 correct 𝑟 = # subquestions score = ቐ 𝑙 = # answer options −𝑥/(𝑟 𝑙 − 1 ) wrong

Previously exercises made manually with LaTeX + xfig

xfig  Vector based drawing program  First version 1985  Early version for Unix, later Linux  Runs on Windows 10 using Cygwin X-server  Allows LaTeX in text fields PostScript / pdf  Transfig: fig + latex (same font in figures + text)  Due to lack of support, for a period numbers should be type in right-to- left, e.g. “125” as “5” “2” “1”

Compiling LaTeX document – perl script A.fig B.fig perl script transfig ∙∙∙ main.tex list of figures .pdf A.ps .pdf B.ps latex A.tex B.tex pdflatex main.dvi dvips main.ps pstopdf main.pdf

Fall 2019 Is Is it it possible to autogenerate ADS exams ?  Essentially all questions are standard question, many of the form “ What does algorithm X output on input Y ? ”  Must come up with two exams every year (ordinary + reexam)  Provide students with (extremely) representative test exam to train on  Special in Fall 2019, could not make exam due to COI (thanks Kasper)  Excuse for doing some Python programming

Status  Can currently autogenerate ≈ 2/3 of an ADS exam  Missing topics: • dynamic programming • loop invariants • amortized analysis • augmented search trees  3.000+ lines of Python code pdflatex ads-exam.tex ads-exam.pdf ads-solutions.tex ads-solutions.pdf main.py ads-pdf-form.tex ads-pdf-form.pdf ads-selftest.tex ads-selftest.pdf (embeded JavaScript)

The autogeneration of a question repeat if input “non - interesting”  Generate random input or two identical answer options  Run correct algorithm or two identical subquestions  Run “obvious wrong” algorithms or ... raise question.Retry('no valid coloring') radix-sort Retries: Q32:1 Q34:1 Q35:1 Q36:1 Q38:1 Q40:1 Q41:1 Q42:4 Q47:1 Q50:1 Q52:5 Q53:7 Q54:6 Q55:6 Q56:3 Q57:2 Q58:15 Q59:3 Q60:5 Q68:1 Q71:1115 Q72:4129 Q73:3772 Q74:240 Q75:589 Q76:1696 Q77:719 Q78:388 Q79:1030 Q80:407 Q81:40 Q82:19 Q83:34 Q84:69 Q85:7 Q86:24 Q87:38 Q88:16 Q89:11 Q90:5 Q91:4 Q92:10 Q93:15 Q94:4 Q95:15 Q96:6 Q97:2 Q98:9 Q99:6 Q100:6 Q101:5 Q102:13 Q103:5 Q104:22 Q105:22 Q106:16 Q107:5 Q108:15 Q109:18 Q110:17 Q111:17 Q112:3 Q113:43 Q114:20 Q115:4 Q116:25 Q118:8 Q119:13 Q120:28 Q121:34 Q122:29 Q123:29 Q124:12 Q125:35 Q126:20 Q127:6 Q128:13 Q129:24 Q130:3 Q131:2 Q132:22 Q133:151 Q134:33 Q135:13 Q136:38 Q137:9 Q138:39 Q139:80 Q140:6 Q143:1 Q144:1 Q148:1 Q150:1 Q151:22 Q152:12 Q154:12 Q155:2 Q156:5 Q159:9 Q160:7 Q161:1406 Q162:22 Q163:777 Q164:2807 Q165:2739 Q166:52 Q167:221 Q168:831 Q169:20 Q170:1921 Q171:3 Q172:4 Q174:2 Q175:2 Q176:3 Q177:1 Q180:2 Q181:25 Q182:36 Q183:165 Q184:42 Q185:3 Q186:15 Q187:104 Q188:9 Q189:3 Q190:30 Q191:380 Q192:92 Q193:172 Q194:21 Q195:82 Q196:16 Q197:58 Q198:19 Q199:55 Q200:127 Q201:2 Q203:7 Q204:1 Q205:2 Q206:4 Q208:1 Q209:3 Q210:1 Q219:1

Figures – LaTeX “ tikz ” and “ tikz-qtree ”

PDF score evaluation (requires AcroRead)

 List of increasing complexity classes  Random O, Ω or Θ  Multiply/divide with random constant  Added minor terms (terms permuted)  Weight parameters  Flip left/right expression uniformly

 Manually made 71 loop codes  Simple ASCII-> LaTeX  Hardness parameter (2, 1, 1)  Favourite:

 Generate random skewed tree  Fun with tikz

 Max-Heap-Insert options • correct Max-Heap-Insert • apply Build-Max-Heap • insertion order • decreasing order • buggy Build-Max-Heap (wrong loop)  Build-Max-Heap question • same options  Heap-Extract-Max • correct ExtractMax • buggy ExtractMax (leave hole at leaf) • buggy ExtractMax (shift left to the right of hole) • buggy ExtractMax (swap last into leaf hole) • apply BuildMaxHeap after extraction

 Input • random numbers • must include max digit • sorted suffix should have an appropriate number of repetitions  Options • correct stable sorted • unstable • sort on prefix • reversed suffix sorting • totally sorted  Requires many retries

 Input requirements • must have initial collisions • some updates must warp around • updates not all identical hash values • at least 2 different answers  Quadratic and double hashing • same

 Generate all colorings  Options • valid colorings • root red • red color of a node in a valid solution moved to children

 Random tree  Compute random colorings  Select random insert  Options • correct restructuring • red leaf without rebalancing • black leaf without rebalancing • random valid red-black tree

 Random union tree  Options • correct union find • no path compression during union • meld path compresses to new root • link left below right, ignoring rank

 Random graph • predefined set of families of graphs • at least 6 nodes • bidirectional 20 %  Options • BFS • buggy BFS, push random order • buggy BFS, pushed reversed 2 nd half • DFS

 Compute all valid BFS  Check all nodes reachable  Check at least 3 solutions  For each valid solution add random spanning tree  Select 5 random from above

 Graph • 6-10 nodes • bidirectional 20 % • check all nodes reachable  50-50 if finishing or discovery order  4 random edges • check at least three edge types

 Options • Dijkstra • Dijkstra with missing distance updates for the 2 nd half • BFS • DFS

 Options • correct MST by Prim • Dijkstra • buggy Prim (DFS version of Prim) • Dijkstra on MST tree

 Random graph with max 5 nodes  Compute all topological orders  Check if DAG  At most 3 correct topological orders  Incorrect = correct + random swap

 Random graph • edges bidirectional = 15 % • If only 1 SCC, retry with 90 %