Class and program design: SCOOBI perspective Whats difficult in - PowerPoint PPT Presentation

Class and program design: SCOOBI perspective ● What’s difficult in developing scoobi/scoobido? ➤ Always do the hard part first. If the hard part is impossible, why waste time on the easy part? Once the hard part is done, you’re home free. ➤ Always do the easy part first. What you think at first is the easy part often turns out to be the hard part. Once the easy part is done, you can concentrate all your efforts on the hard part Al Schapira, Bell Labs ➤ Make the hard part easy by developing an interface that supports a fast-to-develop prototype, but allows you to switch in a class that does all the hard stuff right and fast Owen Astrachan, Duke CS ➤ corollary to: It is faster to make a four-inch mirror then a six- inch mirror than to make a six-inch mirror Ken Thompson’s rule for first-time telescope makers 14. 1 Duke CPS 108

Time and space: a classic tradeoff ● What’s more important, speed of scoobi or scoobido? ➤ What are the issues here? Who determines importance? ● What’s more important, speed of scoobido or size of index? ➤ What does it depend on? ● What information needs to be/should be stored to facilitate searching with scoobido? ➤ What is minimal information, what is maximal information, what’s a a reasonable compromise? ➤ What other forces are there in making this decision? 14. 2 Duke CPS 108

Glimpse, Scoobi, Indexing Store full-paths of files, /u/ola/data/poe.txt ● /u/ola/data/twain.txt what other info? /u/ola/data/melville.txt ➤ optimizations? /u/ola/data/shakespeare/romeo.txt ➤ plain text? the 1 2 3 25 Store words and file ● ship 3 5 numbers vapid 3 23 97 ➤ trouble with numbers? ➤ issues? How are .scoobi files read ● and written? ➤ what classes have responsibility? scoobi.readFrom(dir); What about searching? scoobi.writeTo(dir); Optimizations? 14. 3 Duke CPS 108

Where are the bottlenecks in scoobi/scoobido? ● Don’t guess, profile ● Key issues here are searching and reading ➤ hashtables for searching are very fast, my HMap class may be faster than SGI/STL hash_map class ➤ the g++ stream I/O libraries are horrendously slow, using raw C-style, low-level reads make program faster, less clear (solution?) ● What about regular expressions w.r.t. searching and indexing? ● What’s different in indexing/searching web vs indexing/searching directories and files? ● Supporting different sizes/speeds from scoobi/scoobido ➤ isolating user/client from these issues 14. 4 Duke CPS 108