Exams Gerth Stlting Brodal Algorithms and Data Structures Retreat, - - PowerPoint PPT Presentation

Autogenerating Algorithms and Data Structures Exams

Gerth Stølting Brodal

Algorithms and Data Structures Retreat, Sandbjerg, Denmark, March 3, 2020

Algorithms and Data Structures Cource

ADS 10 ECTS, 2nd semester

1991 - 2003 Written exam

ADS 1 5 ECTS, 2nd semester

2004 - 2014 Multiple choice + short answers 2015 - 2017 Multiple choice

ADS 2 5 ECTS 2nd semester

2004 - 2017 Written

ADS 10 ECTS, 1st semester

2017 Multiple choice + written question 2018 - Multiple choice

𝑥 = question weight 𝑟 = # subquestions 𝑙 = # answer options score = ቐ no answer 𝑥/𝑟 correct −𝑥/(𝑟 𝑙 − 1 ) wrong

Previously exercises made manually with LaTeX + xfig

xfig

• Vector based drawing program
• First version 1985
• Early version for Unix, later Linux
• Runs on Windows 10 using

Cygwin X-server

• Allows LaTeX in text fields

PostScript / pdf

• Transfig: fig +

latex (same font in figures + text)

• Due to lack of support, for a period numbers should be type in

right-to-left, e.g. “125” as “5” “2” “1”

Compiling LaTeX document – perl script

main.tex A.fig B.fig list of figures perl script A.ps A.tex B.ps B.tex ∙∙∙ transfig main.dvi latex dvips main.ps pstopdf main.pdf .pdf .pdf pdflatex

Fall 2019 Is Is it it possible to autogenerate ADS exams ?

• Essentially all questions are standard question, many of the form

“What does algorithm X output on input Y ?”

• Must come up with two exams every year (ordinary + reexam)
• Provide students with (extremely) representative test exam to train on
• Special in Fall 2019, could not make exam due to COI (thanks Kasper)
• Excuse for doing some Python programming

Status

• Can currently autogenerate ≈ 2/3 of an ADS exam
• Missing topics:
• dynamic programming
• loop invariants
• amortized analysis
• augmented search trees
• 3.000+ lines of Python code

main.py ads-exam.tex ads-pdf-form.tex ads-solutions.tex ads-selftest.tex (embeded JavaScript) ads-exam.pdf ads-pdf-form.pdf ads-solutions.pdf ads-selftest.pdf pdflatex

The autogeneration of a question

• Generate random input
• Run correct algorithm
• Run “obvious wrong” algorithms

Retries: Q32:1 Q34:1 Q35:1 Q36:1 Q38:1 Q40:1 Q41:1 Q42:4 Q47:1 Q50:1 Q52:5 Q53:7 Q54:6 Q55:6 Q56:3 Q57:2 Q58:15 Q59:3 Q60:5 Q68:1 Q71:1115 Q72:4129 Q73:3772 Q74:240 Q75:589 Q76:1696 Q77:719 Q78:388 Q79:1030 Q80:407 Q81:40 Q82:19 Q83:34 Q84:69 Q85:7 Q86:24 Q87:38 Q88:16 Q89:11 Q90:5 Q91:4 Q92:10 Q93:15 Q94:4 Q95:15 Q96:6 Q97:2 Q98:9 Q99:6 Q100:6 Q101:5 Q102:13 Q103:5 Q104:22 Q105:22 Q106:16 Q107:5 Q108:15 Q109:18 Q110:17 Q111:17 Q112:3 Q113:43 Q114:20 Q115:4 Q116:25 Q118:8 Q119:13 Q120:28 Q121:34 Q122:29 Q123:29 Q124:12 Q125:35 Q126:20 Q127:6 Q128:13 Q129:24 Q130:3 Q131:2 Q132:22 Q133:151 Q134:33 Q135:13 Q136:38 Q137:9 Q138:39 Q139:80 Q140:6 Q143:1 Q144:1 Q148:1 Q150:1 Q151:22 Q152:12 Q154:12 Q155:2 Q156:5 Q159:9 Q160:7 Q161:1406 Q162:22 Q163:777 Q164:2807 Q165:2739 Q166:52 Q167:221 Q168:831 Q169:20 Q170:1921 Q171:3 Q172:4 Q174:2 Q175:2 Q176:3 Q177:1 Q180:2 Q181:25 Q182:36 Q183:165 Q184:42 Q185:3 Q186:15 Q187:104 Q188:9 Q189:3 Q190:30 Q191:380 Q192:92 Q193:172 Q194:21 Q195:82 Q196:16 Q197:58 Q198:19 Q199:55 Q200:127 Q201:2 Q203:7 Q204:1 Q205:2 Q206:4 Q208:1 Q209:3 Q210:1 Q219:1

radix-sort

repeat if input “non-interesting”

• r two identical answer options
• r two identical subquestions
• r ...

raise question.Retry('no valid coloring')

Figures – LaTeX “tikz” and “tikz-qtree”

PDF score evaluation (requires AcroRead)

• List of increasing complexity classes
• Random O, Ω or Θ
• Multiply/divide with random constant
• Added minor terms (terms permuted)
• Weight parameters
• Flip left/right expression uniformly

• Manually made 71 loop codes
• Simple ASCII-> LaTeX
• Hardness parameter (2, 1, 1)
• Favourite:

• Generate random skewed tree
• Fun with tikz

• Max-Heap-Insert options
• correct Max-Heap-Insert
• apply Build-Max-Heap
• insertion order
• decreasing order
• buggy Build-Max-Heap (wrong loop)
• Build-Max-Heap question
• same options
• Heap-Extract-Max
• correct ExtractMax
• buggy ExtractMax (leave hole at leaf)
• buggy ExtractMax (shift left to the right of hole)
• buggy ExtractMax (swap last into leaf hole)
• apply BuildMaxHeap after extraction

• Input
• random numbers
• must include max digit
• sorted suffix should have an

appropriate number of repetitions

• Options
• correct stable sorted
• unstable
• sort on prefix
• reversed suffix sorting
• totally sorted
• Requires many retries

• Input requirements
• must have initial collisions
• some updates must warp around
• updates not all identical hash values
• at least 2 different answers
• Quadratic and double hashing
• same

• Generate all colorings
• Options
• valid colorings
• root red
• red color of a node in a valid

solution moved to children

• Random tree
• Compute random colorings
• Select random insert
• Options
• correct restructuring
• red leaf without rebalancing
• black leaf without rebalancing
• random valid red-black tree

• Random union tree
• Options
• correct union find
• no path compression during union
• meld path compresses to new root
• link left below right, ignoring rank

• Random graph
• predefined set of families of graphs
• at least 6 nodes
• bidirectional 20 %
• Options
• BFS
• buggy BFS, push random order
• buggy BFS, pushed reversed 2nd half
• DFS

• Compute all valid BFS
• Check all nodes reachable
• Check at least 3 solutions
• For each valid solution add

random spanning tree

• Select 5 random from above

• Graph
• 6-10 nodes
• bidirectional 20 %
• check all nodes reachable
• 50-50 if finishing or discovery order
• 4 random edges
• check at least three edge types

• Options
• Dijkstra
• Dijkstra with missing distance

updates for the 2nd half

• BFS
• DFS

• Options
• correct MST by Prim
• Dijkstra
• buggy Prim (DFS version of Prim)
• Dijkstra on MST tree

• Random graph with max 5 nodes
• Compute all topological orders
• Check if DAG
• At most 3 correct topological orders
• Incorrect = correct + random swap

• Random graph
• edges bidirectional = 15 %
• If only 1 SCC, retry with 90 %