cse 7 5337 information retrieval and web search
play

CSE 7/5337: Information Retrieval and Web Search Dictionaries and - PowerPoint PPT Presentation

Dec 21, 2022 •437 likes •1.52k views

CSE 7/5337: Information Retrieval and Web Search Dictionaries and tolerant retrieval (IIR 3) Michael Hahsler Southern Methodist University These slides are largely based on the slides by Hinrich Sch utze Institute for Natural Language

  1. Exercise Google has very limited support for wildcard queries. For example, this query doesn’t work very well on Google: [gen* universit*] ◮ Intention: you are looking for the University of Geneva, but don’t know which accents to use for the French words for university and Geneva. According to Google search basics, 2010-04-29: “Note that the * operator works only on whole words, not parts of words.” But this is not entirely true. Try [pythag*] and [m*nchen] Exercise: Why doesn’t Google fully support wildcard queries? Hahsler (SMU) CSE 7/5337 Spring 2012 31 / 108

  2. Processing wildcard queries in the term-document index Problem 1: we must potentially execute a large number of Boolean queries. Most straightforward semantics: Conjunction of disjunctions For [gen* universit*]: geneva university or geneva universit´ e or gen` eve university or gen` eve universit´ e or general universities or . . . Very expensive Problem 2: Users hate to type. If abbreviated queries like [pyth* theo*] for [pythagoras’ theorem] are allowed, users will use them a lot. This would significantly increase the cost of answering queries. Somewhat alleviated by Google Suggest Hahsler (SMU) CSE 7/5337 Spring 2012 32 / 108

  3. Outline Recap 1 Dictionaries 2 Wildcard queries 3 Edit distance 4 Spelling correction 5 Soundex 6 Hahsler (SMU) CSE 7/5337 Spring 2012 33 / 108

  4. Spelling correction Two principal uses ◮ Correcting documents being indexed ◮ Correcting user queries Two different methods for spelling correction Isolated word spelling correction ◮ Check each word on its own for misspelling ◮ Will not catch typos resulting in correctly spelled words, e.g., an asteroid that fell form the sky Context-sensitive spelling correction ◮ Look at surrounding words ◮ Can correct form / from error above Hahsler (SMU) CSE 7/5337 Spring 2012 34 / 108

  5. Correcting documents We’re not interested in interactive spelling correction of documents (e.g., MS Word) in this class. In IR, we use document correction primarily for OCR’ed documents. (OCR = optical character recognition) The general philosophy in IR is: don’t change the documents. Hahsler (SMU) CSE 7/5337 Spring 2012 35 / 108

  6. Correcting queries First: isolated word spelling correction Premise 1: There is a list of “correct words” from which the correct spellings come. Premise 2: We have a way of computing the distance between a misspelled word and a correct word. Simple spelling correction algorithm: return the “correct” word that has the smallest distance to the misspelled word. Example: informaton → information For the list of correct words, we can use the vocabulary of all words that occur in our collection. Why is this problematic? Hahsler (SMU) CSE 7/5337 Spring 2012 36 / 108

  7. Alternatives to using the term vocabulary A standard dictionary (Webster’s, OED etc.) An industry-specific dictionary (for specialized IR systems) The term vocabulary of the collection, appropriately weighted Hahsler (SMU) CSE 7/5337 Spring 2012 37 / 108

  8. Distance between misspelled word and “correct” word We will study several alternatives. Edit distance and Levenshtein distance Weighted edit distance k -gram overlap Hahsler (SMU) CSE 7/5337 Spring 2012 38 / 108

  9. Edit distance The edit distance between string s 1 and string s 2 is the minimum number of basic operations that convert s 1 to s 2 . Levenshtein distance: The admissible basic operations are insert, delete, and replace Levenshtein distance dog - do : 1 Levenshtein distance cat - cart : 1 Levenshtein distance cat - cut : 1 Levenshtein distance cat - act : 2 Damerau-Levenshtein distance cat - act : 1 Damerau-Levenshtein includes transposition as a fourth possible operation. Hahsler (SMU) CSE 7/5337 Spring 2012 39 / 108

  10. Levenshtein distance: Computation f a s t 0 1 2 3 4 c 1 1 2 3 4 a 2 2 1 2 3 t 3 3 2 2 2 s 4 4 3 2 3 Hahsler (SMU) CSE 7/5337 Spring 2012 40 / 108

  11. Levenshtein distance: Algorithm LevenshteinDistance ( s 1 , s 2 ) 1 for i ← 0 to | s 1 | 2 do m [ i , 0] = i 3 for j ← 0 to | s 2 | 4 do m [0 , j ] = j 5 for i ← 1 to | s 1 | 6 do for j ← 1 to | s 2 | 7 do if s 1 [ i ] = s 2 [ j ] 8 then m [ i , j ] = min { m [ i -1 , j ]+1 , m [ i , j -1]+1 , m [ i -1 , j -1] } 9 else m [ i , j ] = min { m [ i -1 , j ]+1 , m [ i , j -1]+1 , m [ i -1 , j -1]+1 } 10 return m [ | s 1 | , | s 2 | ] Operations: insert (cost 1), delete (cost 1), replace (cost 1), copy (cost 0) Hahsler (SMU) CSE 7/5337 Spring 2012 41 / 108

  12. Levenshtein distance: Algorithm LevenshteinDistance ( s 1 , s 2 ) 1 for i ← 0 to | s 1 | 2 do m [ i , 0] = i 3 for j ← 0 to | s 2 | 4 do m [0 , j ] = j 5 for i ← 1 to | s 1 | 6 do for j ← 1 to | s 2 | 7 do if s 1 [ i ] = s 2 [ j ] 8 then m [ i , j ] = min { m [ i -1 , j ]+1 , m [ i , j -1]+1 , m [ i -1 , j -1] } 9 else m [ i , j ] = min { m [ i -1 , j ]+1 , m [ i , j -1]+1 , m [ i -1 , j -1]+1 } 10 return m [ | s 1 | , | s 2 | ] Operations: insert (cost 1), delete (cost 1), replace (cost 1), copy (cost 0) Hahsler (SMU) CSE 7/5337 Spring 2012 42 / 108

  13. Levenshtein distance: Algorithm LevenshteinDistance ( s 1 , s 2 ) 1 for i ← 0 to | s 1 | 2 do m [ i , 0] = i 3 for j ← 0 to | s 2 | 4 do m [0 , j ] = j 5 for i ← 1 to | s 1 | 6 do for j ← 1 to | s 2 | 7 do if s 1 [ i ] = s 2 [ j ] 8 then m [ i , j ] = min { m [ i -1 , j ]+1 , m [ i , j -1]+1 , m [ i -1 , j -1] } 9 else m [ i , j ] = min { m [ i -1 , j ]+1 , m [ i , j -1]+1 , m [ i -1 , j -1]+1 } 10 return m [ | s 1 | , | s 2 | ] Operations: insert (cost 1), delete (cost 1), replace (cost 1), copy (cost 0) Hahsler (SMU) CSE 7/5337 Spring 2012 43 / 108

  14. Levenshtein distance: Algorithm LevenshteinDistance ( s 1 , s 2 ) 1 for i ← 0 to | s 1 | 2 do m [ i , 0] = i 3 for j ← 0 to | s 2 | 4 do m [0 , j ] = j 5 for i ← 1 to | s 1 | 6 do for j ← 1 to | s 2 | 7 do if s 1 [ i ] = s 2 [ j ] 8 then m [ i , j ] = min { m [ i -1 , j ]+1 , m [ i , j -1]+1 , m [ i -1 , j -1] } 9 else m [ i , j ] = min { m [ i -1 , j ]+1 , m [ i , j -1]+1 , m [ i -1 , j -1]+1 } 10 return m [ | s 1 | , | s 2 | ] Operations: insert (cost 1), delete (cost 1), replace (cost 1), copy (cost 0) Hahsler (SMU) CSE 7/5337 Spring 2012 44 / 108

  15. Levenshtein distance: Algorithm LevenshteinDistance ( s 1 , s 2 ) 1 for i ← 0 to | s 1 | 2 do m [ i , 0] = i 3 for j ← 0 to | s 2 | 4 do m [0 , j ] = j 5 for i ← 1 to | s 1 | 6 do for j ← 1 to | s 2 | 7 do if s 1 [ i ] = s 2 [ j ] 8 then m [ i , j ] = min { m [ i -1 , j ]+1 , m [ i , j -1]+1 , m [ i -1 , j -1] } 9 else m [ i , j ] = min { m [ i -1 , j ]+1 , m [ i , j -1]+1 , m [ i -1 , j -1]+1 } 10 return m [ | s 1 | , | s 2 | ] Operations: insert (cost 1), delete (cost 1), replace (cost 1), copy (cost 0) Hahsler (SMU) CSE 7/5337 Spring 2012 45 / 108

  16. Levenshtein distance: Example f a s t 0 1 1 2 2 3 3 4 4 1 1 2 2 3 3 4 4 5 c 1 2 1 2 2 3 3 4 4 2 2 2 1 3 3 4 4 5 a 2 3 2 3 1 2 2 3 3 3 2 3 2 4 3 3 3 2 t 4 4 3 2 3 2 3 3 2 4 3 3 3 4 4 4 3 2 s 5 5 4 3 3 4 4 3 2 Hahsler (SMU) CSE 7/5337 Spring 2012 46 / 108

  17. Each cell of Levenshtein matrix cost of getting here from cost of getting here my upper left neighbor from my upper neighbor (copy or replace) (delete) the minimum of the cost of getting here from three possible “move- my left neighbor (insert) ments”; the cheapest way of getting here Hahsler (SMU) CSE 7/5337 Spring 2012 47 / 108

  18. Levenshtein distance: Example f a s t 0 1 1 2 2 3 3 4 4 1 1 2 2 3 3 4 4 5 c 1 2 1 2 2 3 3 4 4 2 2 2 1 3 3 4 4 5 a 2 3 2 3 1 2 2 3 3 3 2 3 2 4 3 3 3 2 t 4 4 3 2 3 2 3 3 2 4 3 3 3 4 4 4 3 2 s 5 5 4 3 3 4 4 3 2 Hahsler (SMU) CSE 7/5337 Spring 2012 48 / 108

  19. Dynamic programming (Cormen et al.) Optimal substructure: The optimal solution to the problem contains within it subsolutions, i.e., optimal solutions to subproblems. Overlapping subsolutions: The subsolutions overlap. These subsolutions are computed over and over again when computing the global optimal solution in a brute-force algorithm. Subproblem in the case of edit distance: what is the edit distance of two prefixes Overlapping subsolutions: We need most distances of prefixes 3 times – this corresponds to moving right, diagonally, down. Hahsler (SMU) CSE 7/5337 Spring 2012 49 / 108

  20. Weighted edit distance As above, but weight of an operation depends on the characters involved. Meant to capture keyboard errors, e.g., m more likely to be mistyped as n than as q . Therefore, replacing m by n is a smaller edit distance than by q . We now require a weight matrix as input. Modify dynamic programming to handle weights Hahsler (SMU) CSE 7/5337 Spring 2012 50 / 108

  21. Using edit distance for spelling correction Given query, first enumerate all character sequences within a preset (possibly weighted) edit distance Intersect this set with our list of “correct” words Then suggest terms in the intersection to the user. → exercise in a few slides Hahsler (SMU) CSE 7/5337 Spring 2012 51 / 108

  22. Exercise 1 Compute Levenshtein distance matrix for oslo – snow Hahsler (SMU) CSE 7/5337 Spring 2012 52 / 108

  23. s n o w 0 1 1 2 2 3 3 4 4 1 o 1 2 s 2 3 l 3 4 o 4 Hahsler (SMU) CSE 7/5337 Spring 2012 53 / 108

  24. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 o 1 2 ? 2 s 2 3 l 3 4 o 4 Hahsler (SMU) CSE 7/5337 Spring 2012 54 / 108

  25. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 o 1 2 1 2 s 2 3 l 3 4 o 4 Hahsler (SMU) CSE 7/5337 Spring 2012 55 / 108

  26. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 o 1 2 1 2 ? 2 s 2 3 l 3 4 o 4 Hahsler (SMU) CSE 7/5337 Spring 2012 56 / 108

  27. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 o 1 2 1 2 2 2 s 2 3 l 3 4 o 4 Hahsler (SMU) CSE 7/5337 Spring 2012 57 / 108

  28. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 o 1 2 1 2 2 3 ? 2 s 2 3 l 3 4 o 4 Hahsler (SMU) CSE 7/5337 Spring 2012 58 / 108

  29. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 o 1 2 1 2 2 3 2 2 s 2 3 l 3 4 o 4 Hahsler (SMU) CSE 7/5337 Spring 2012 59 / 108

  30. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 ? 2 s 2 3 l 3 4 o 4 Hahsler (SMU) CSE 7/5337 Spring 2012 60 / 108

  31. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 s 2 3 l 3 4 o 4 Hahsler (SMU) CSE 7/5337 Spring 2012 61 / 108

  32. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 s 3 ? 2 3 l 3 4 o 4 Hahsler (SMU) CSE 7/5337 Spring 2012 62 / 108

  33. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 s 3 2 1 3 l 3 4 o 4 Hahsler (SMU) CSE 7/5337 Spring 2012 63 / 108

  34. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 2 3 s 3 2 ? 2 1 3 l 3 4 o 4 Hahsler (SMU) CSE 7/5337 Spring 2012 64 / 108

  35. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 2 3 s 3 2 1 2 2 3 l 3 4 o 4 Hahsler (SMU) CSE 7/5337 Spring 2012 65 / 108

  36. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 2 3 3 3 s 3 3 ? 2 1 2 2 3 l 3 4 o 4 Hahsler (SMU) CSE 7/5337 Spring 2012 66 / 108

  37. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 2 3 3 3 s 3 2 1 2 2 3 3 3 l 3 4 o 4 Hahsler (SMU) CSE 7/5337 Spring 2012 67 / 108

  38. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 2 3 3 3 3 4 s 3 4 ? 2 1 2 2 3 3 3 l 3 4 o 4 Hahsler (SMU) CSE 7/5337 Spring 2012 68 / 108

  39. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 2 3 3 3 3 4 s 3 4 2 1 2 2 3 3 3 3 l 3 4 o 4 Hahsler (SMU) CSE 7/5337 Spring 2012 69 / 108

  40. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 2 3 3 3 3 4 s 3 4 2 1 2 2 3 3 3 3 3 2 l 4 ? 3 4 o 4 Hahsler (SMU) CSE 7/5337 Spring 2012 70 / 108

  41. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 2 3 3 3 3 4 s 3 4 2 1 2 2 3 3 3 3 3 2 l 4 3 2 4 o 4 Hahsler (SMU) CSE 7/5337 Spring 2012 71 / 108

  42. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 2 3 3 3 3 4 s 3 4 2 1 2 2 3 3 3 3 3 2 2 3 l 4 3 ? 3 2 4 o 4 Hahsler (SMU) CSE 7/5337 Spring 2012 72 / 108

  43. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 2 3 3 3 3 4 s 3 4 2 1 2 2 3 3 3 3 3 2 2 3 l 4 3 3 2 2 4 o 4 Hahsler (SMU) CSE 7/5337 Spring 2012 73 / 108

  44. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 2 3 3 3 3 4 s 3 4 2 1 2 2 3 3 3 3 3 2 2 3 3 4 l 4 3 3 ? 3 2 2 4 o 4 Hahsler (SMU) CSE 7/5337 Spring 2012 74 / 108

  45. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 2 3 3 3 3 4 s 3 4 2 1 2 2 3 3 3 3 3 2 2 3 3 4 l 4 3 3 2 2 3 3 4 o 4 Hahsler (SMU) CSE 7/5337 Spring 2012 75 / 108

  46. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 2 3 3 3 3 4 s 3 4 2 1 2 2 3 3 3 3 3 2 2 3 3 4 4 4 l 4 3 4 ? 3 2 2 3 3 4 o 4 Hahsler (SMU) CSE 7/5337 Spring 2012 76 / 108

  47. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 2 3 3 3 3 4 s 3 4 2 1 2 2 3 3 3 3 3 2 2 3 3 4 4 4 l 4 3 3 2 2 3 3 4 4 4 o 4 Hahsler (SMU) CSE 7/5337 Spring 2012 77 / 108

  48. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 2 3 3 3 3 4 s 3 4 2 1 2 2 3 3 3 3 3 2 2 3 3 4 4 4 l 4 3 3 2 2 3 3 4 4 4 4 3 o 5 ? 4 Hahsler (SMU) CSE 7/5337 Spring 2012 78 / 108

  49. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 2 3 3 3 3 4 s 3 4 2 1 2 2 3 3 3 3 3 2 2 3 3 4 4 4 l 4 3 3 2 2 3 3 4 4 4 4 3 o 5 4 3 Hahsler (SMU) CSE 7/5337 Spring 2012 79 / 108

  50. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 2 3 3 3 3 4 s 3 4 2 1 2 2 3 3 3 3 3 2 2 3 3 4 4 4 l 4 3 3 2 2 3 3 4 4 4 4 3 3 3 o 5 4 ? 4 3 Hahsler (SMU) CSE 7/5337 Spring 2012 80 / 108

  51. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 2 3 3 3 3 4 s 3 4 2 1 2 2 3 3 3 3 3 2 2 3 3 4 4 4 l 4 3 3 2 2 3 3 4 4 4 4 3 3 3 o 5 4 4 3 3 Hahsler (SMU) CSE 7/5337 Spring 2012 81 / 108

  52. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 2 3 3 3 3 4 s 3 4 2 1 2 2 3 3 3 3 3 2 2 3 3 4 4 4 l 4 3 3 2 2 3 3 4 4 4 4 3 3 3 2 4 o 5 4 4 ? 4 3 3 Hahsler (SMU) CSE 7/5337 Spring 2012 82 / 108

  53. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 2 3 3 3 3 4 s 3 4 2 1 2 2 3 3 3 3 3 2 2 3 3 4 4 4 l 4 3 3 2 2 3 3 4 4 4 4 3 3 3 2 4 o 5 4 4 4 3 3 2 Hahsler (SMU) CSE 7/5337 Spring 2012 83 / 108

  54. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 2 3 3 3 3 4 s 3 4 2 1 2 2 3 3 3 3 3 2 2 3 3 4 4 4 l 4 3 3 2 2 3 3 4 4 4 4 3 3 3 2 4 4 5 o 5 4 4 3 ? 4 3 3 2 Hahsler (SMU) CSE 7/5337 Spring 2012 84 / 108

  55. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 2 3 3 3 3 4 s 3 4 2 1 2 2 3 3 3 3 3 2 2 3 3 4 4 4 l 4 3 3 2 2 3 3 4 4 4 4 3 3 3 2 4 4 5 o 5 4 4 4 3 3 2 3 3 Hahsler (SMU) CSE 7/5337 Spring 2012 85 / 108

  56. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 2 3 3 3 3 4 s 3 4 2 1 2 2 3 3 3 3 3 2 2 3 3 4 4 4 l 4 3 3 2 2 3 3 4 4 4 5 4 4 3 3 3 2 4 o 3 3 5 4 4 4 3 3 2 Hahsler (SMU) CSE 7/5337 Spring 2012 86 / 108

  57. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 2 3 3 3 3 4 s 3 4 2 1 2 2 3 3 3 3 3 2 2 3 3 4 4 4 l 4 3 3 2 2 3 3 4 4 4 4 3 3 3 2 4 4 5 o 5 4 4 4 3 3 2 3 3 How do I read out the editing operations that transform oslo into snow ? Hahsler (SMU) CSE 7/5337 Spring 2012 87 / 108

  58. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 2 3 3 3 3 4 s 3 4 2 1 2 2 3 3 3 3 3 2 2 3 3 4 4 4 l 4 3 3 2 2 3 3 4 4 4 4 3 3 3 2 4 4 5 o 5 4 4 4 3 3 2 3 3 cost operation input output 1 insert * w Hahsler (SMU) CSE 7/5337 Spring 2012 88 / 108

  59. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 2 3 3 3 3 4 s 3 4 2 1 2 2 3 3 3 3 3 2 2 3 3 4 4 4 l 4 3 3 2 2 3 3 4 4 4 4 3 3 3 2 4 4 5 o 5 4 4 4 3 3 2 3 3 cost operation input output 0 (copy) o o 1 insert * w Hahsler (SMU) CSE 7/5337 Spring 2012 89 / 108

  60. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 2 3 3 3 3 4 s 3 4 2 1 2 2 3 3 3 3 3 2 2 3 3 4 4 4 l 4 3 3 2 2 3 3 4 4 4 4 3 3 3 2 4 4 5 o 5 4 4 4 3 3 2 3 3 cost operation input output 1 replace l n 0 (copy) o o 1 insert * w Hahsler (SMU) CSE 7/5337 Spring 2012 90 / 108

  61. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 2 3 3 3 3 4 s 3 4 2 1 2 2 3 3 3 3 3 2 2 3 3 4 4 4 l 4 3 3 2 2 3 3 4 4 4 4 3 3 3 2 4 4 5 o 5 4 4 4 3 3 2 3 3 cost operation input output 0 (copy) s s 1 replace l n 0 (copy) o o 1 insert * w Hahsler (SMU) CSE 7/5337 Spring 2012 91 / 108

  62. s n o w 0 1 1 2 2 3 3 4 4 1 1 2 2 3 2 4 4 5 o 1 2 1 2 2 3 2 3 3 2 1 2 2 3 3 3 3 4 s 3 4 2 1 2 2 3 3 3 3 3 2 2 3 3 4 4 4 l 4 3 3 2 2 3 3 4 4 4 4 3 3 3 2 4 4 5 o 5 4 4 4 3 3 2 3 3 cost operation input output 1 delete o * 0 (copy) s s 1 replace l n 0 (copy) o o 1 insert * w Hahsler (SMU) CSE 7/5337 Spring 2012 92 / 108

  63. Outline Recap 1 Dictionaries 2 Wildcard queries 3 Edit distance 4 Spelling correction 5 Soundex 6 Hahsler (SMU) CSE 7/5337 Spring 2012 93 / 108

  64. Spelling correction Now that we can compute edit distance: how to use it for isolated word spelling correction – this is the last slide in this section. k -gram indexes for isolated word spelling correction. Context-sensitive spelling correction General issues Hahsler (SMU) CSE 7/5337 Spring 2012 94 / 108

  65. k -gram indexes for spelling correction Enumerate all k -grams in the query term Example: bigram index, misspelled word bordroom Bigrams: bo, or, rd, dr, ro, oo, om Use the k -gram index to retrieve “correct” words that match query term k -grams Threshold by number of matching k -grams E.g., only vocabulary terms that differ by at most 3 k -grams Hahsler (SMU) CSE 7/5337 Spring 2012 95 / 108

  66. k -gram indexes for spelling correction: bordroom bo aboard about border boardroom ✲ ✲ ✲ ✲ or border lord morbid sordid ✲ ✲ ✲ ✲ rd aboard ardent border boardroom ✲ ✲ ✲ ✲ Hahsler (SMU) CSE 7/5337 Spring 2012 96 / 108

  67. Context-sensitive spelling correction Our example was: an asteroid that fell form the sky How can we correct form here? One idea: hit-based spelling correction ◮ Retrieve “correct” terms close to each query term ◮ for flew form munich : flea for flew , from for form , munch for munich ◮ Now try all possible resulting phrases as queries with one word “fixed” at a time ◮ Try query “flea form munich” ◮ Try query “flew from munich” ◮ Try query “flew form munch ” ◮ The correct query “flew from munich” has the most hits. Suppose we have 7 alternatives for flew , 20 for form and 3 for munich , how many “corrected” phrases will we enumerate? Hahsler (SMU) CSE 7/5337 Spring 2012 97 / 108

  68. Context-sensitive spelling correction The “hit-based” algorithm we just outlined is not very efficient. More efficient alternative: look at “collection” of queries, not documents Hahsler (SMU) CSE 7/5337 Spring 2012 98 / 108

  69. General issues in spelling correction User interface ◮ automatic vs. suggested correction ◮ Did you mean only works for one suggestion. ◮ What about multiple possible corrections? ◮ Tradeoff: simple vs. powerful UI Cost ◮ Spelling correction is potentially expensive. ◮ Avoid running on every query? ◮ Maybe just on queries that match few documents. ◮ Guess: Spelling correction of major search engines is efficient enough to be run on every query. Hahsler (SMU) CSE 7/5337 Spring 2012 99 / 108

  70. Exercise: Understand Peter Norvig’s spelling corrector import re, collections def words(text): return re.findall(’[a-z]+’, text.lower()) def train(features): model = collections.defaultdict(lambda: 1) for f in features: model[f] += 1 return model NWORDS = train(words(file(’big.txt’).read())) alphabet = ’abcdefghijklmnopqrstuvwxyz’ def edits1(word): splits = [(word[:i], word[i:]) for i in range(len(word) + 1)] deletes = [a + b[1:] for a, b in splits if b] transposes = [a + b[1] + b[0] + b[2:] for a, b in splits if len(b) gt 1] replaces = [a + c + b[1:] for a, b in splits for c in alphabet if b] inserts = [a + c + b for a, b in splits for c in alphabet] return set(deletes + transposes + replaces + inserts) def known_edits2(word): return set(e2 for e1 in edits1(word) for e2 in edits1(e1) if e2 in NWORDS) def known(words): return set(w for w in words if w in NWORDS) def correct(word): candidates = known([word]) or known(edits1(word)) or known_edits2(word) or [word] return max(candidates, key=NWORDS.get) Hahsler (SMU) CSE 7/5337 Spring 2012 100 / 108

Recommend

CSE 7/5337: Information Retrieval and Web Search Introduction and Boolean Retrieval (IIR 1)

CSE 7/5337: Information Retrieval and Web Search Introduction and Boolean Retrieval (IIR 1)

CSE 7/5337: Information Retrieval and Web Search Introduction and Boolean Retrieval (IIR 1) Michael Hahsler Southern Methodist University These slides are largely based on the slides by Hinrich Sch utze Institute for Natural Language

456 views • 35 slides
CSE 7/5337: Information Retrieval and Web Search Web Search (IIR 19) Michael Hahsler Southern

CSE 7/5337: Information Retrieval and Web Search Web Search (IIR 19) Michael Hahsler Southern

CSE 7/5337: Information Retrieval and Web Search Web Search (IIR 19) Michael Hahsler Southern Methodist University These slides are largely based on the slides by Hinrich Sch utze Institute for Natural Language Processing, University of

791 views • 63 slides
CSE 7/5337: Information Retrieval and Web Search Web Search (IIR 19) Michael Hahsler Southern

CSE 7/5337: Information Retrieval and Web Search Web Search (IIR 19) Michael Hahsler Southern

CSE 7/5337: Information Retrieval and Web Search Web Search (IIR 19) Michael Hahsler Southern Methodist University These slides are largely based on the slides by Hinrich Sch utze Institute for Natural Language Processing, University of

754 views • 72 slides
CSE 7/5337: Information Retrieval and Web Search Evaluation & Result Summaries (IIR 8)

CSE 7/5337: Information Retrieval and Web Search Evaluation & Result Summaries (IIR 8)

CSE 7/5337: Information Retrieval and Web Search Evaluation & Result Summaries (IIR 8) Michael Hahsler Southern Methodist University These slides are largely based on the slides by Hinrich Sch utze Institute for Natural Language

562 views • 52 slides
CSE 7/5337: Information Retrieval and Web Search Web crawling and indexes (IIR 20) Michael

CSE 7/5337: Information Retrieval and Web Search Web crawling and indexes (IIR 20) Michael

CSE 7/5337: Information Retrieval and Web Search Web crawling and indexes (IIR 20) Michael Hahsler Southern Methodist University These slides are largely based on the slides by Hinrich Sch utze Institute for Natural Language Processing,

636 views • 27 slides
CSE 7/5337: Information Retrieval and Web Search The term vocabulary and postings lists (IIR 2)

CSE 7/5337: Information Retrieval and Web Search The term vocabulary and postings lists (IIR 2)

CSE 7/5337: Information Retrieval and Web Search The term vocabulary and postings lists (IIR 2) Michael Hahsler Southern Methodist University These slides are largely based on the slides by Hinrich Sch utze Institute for Natural Language

708 views • 46 slides
CSE 7/5337: Information Retrieval and Web Search Document clustering I (IIR 16) Michael Hahsler

CSE 7/5337: Information Retrieval and Web Search Document clustering I (IIR 16) Michael Hahsler

CSE 7/5337: Information Retrieval and Web Search Document clustering I (IIR 16) Michael Hahsler Southern Methodist University These slides are largely based on the slides by Hinrich Sch utze Institute for Natural Language Processing,

786 views • 55 slides
CSE 7/5337: Information Retrieval and Web Search Index construction (IIR 4) Michael Hahsler

CSE 7/5337: Information Retrieval and Web Search Index construction (IIR 4) Michael Hahsler

CSE 7/5337: Information Retrieval and Web Search Index construction (IIR 4) Michael Hahsler Southern Methodist University These slides are largely based on the slides by Hinrich Sch utze Institute for Natural Language Processing, University

556 views • 54 slides
CSE 7/5337: Information Retrieval and Web Search Scoring, term weighting, the vector space model

CSE 7/5337: Information Retrieval and Web Search Scoring, term weighting, the vector space model

CSE 7/5337: Information Retrieval and Web Search Scoring, term weighting, the vector space model (IIR 6) Michael Hahsler Southern Methodist University These slides are largely based on the slides by Hinrich Sch utze Institute for Natural

715 views • 67 slides
Information Retrieval Introducing Information Retrieval and Web Search Information Retrieval

Information Retrieval Introducing Information Retrieval and Web Search Information Retrieval

Introduction to Information Retrieval Introducing Information Retrieval and Web Search Information Retrieval Information Retrieval (IR) is finding material (usually documents) of an unstructured nature (usually text) that satisfies an

1.01k views • 57 slides
Web Information Retrieval Lecture 9 Information Retrieval in the Web Search use (iProspect

Web Information Retrieval Lecture 9 Information Retrieval in the Web Search use (iProspect

Web Information Retrieval Lecture 9 Information Retrieval in the Web Search use (iProspect Survey, 4/04) Without search engines the web wouldnt scale 1. No incentive in creating content unless it can be easily found other finding

814 views • 60 slides
Information Retrieval CS276: Information Retrieval and Web Search

Information Retrieval CS276: Information Retrieval and Web Search

Introduction to Information Retrieval Introduction to Information Retrieval CS276: Information Retrieval and Web Search Christopher Manning and Prabhakar Raghavan Lecture 10: Text

736 views • 57 slides
Information Retrieval CS276: Information Retrieval and Web Search

Information Retrieval CS276: Information Retrieval and Web Search

Introduction to Information Retrieval Introduction to Information Retrieval CS276: Information Retrieval and Web Search Christopher Manning, Pandu Nayak, and Prabhakar Raghavan

1.49k views • 50 slides
Information Retrieval Introducing Information Retrieval and Web Search

Information Retrieval Introducing Information Retrieval and Web Search

Introduction to Information Retrieval Introducing Information Retrieval and Web Search Information Retrieval Information Retrieval (IR) is finding material (usually documents) of

585 views • 57 slides
Information Retrieval CS276: Information Retrieval and Web Search Pandu

Information Retrieval CS276: Information Retrieval and Web Search Pandu

Introduction to Information Retrieval Introduction to Information Retrieval CS276: Information Retrieval and Web Search Pandu Nayak and Prabhakar Raghavan Lecture 3: Dictionaries

489 views • 48 slides
Information Retrieval CS276: Information Retrieval and Web Search Pandu

Information Retrieval CS276: Information Retrieval and Web Search Pandu

Introduction to Information Retrieval Introduction to Information Retrieval CS276: Information Retrieval and Web Search Pandu Nayak and Prabhakar Raghavan Lecture 4: Index

649 views • 49 slides
Information Retrieval CS276 Information Retrieval and Web Search Christopher

Information Retrieval CS276 Information Retrieval and Web Search Christopher

Introduction to Information Retrieval Introduction to Information Retrieval CS276 Information Retrieval and Web Search Christopher Manning and Prabhakar Raghavan Lecture 9: Query

864 views • 51 slides
1 What is multimedia information retrieval? 1.1 Information retrieval 1.2 Multimedia 1.3

1 What is multimedia information retrieval? 1.1 Information retrieval 1.2 Multimedia 1.3

1 What is multimedia information retrieval? 1.1 Information retrieval 1.2 Multimedia 1.3 Semantic Gap? 1.4 Challenges of automated multimedia indexing 2 Basic multimedia search technologies 2.1 Meta-data driven retrieval 2.2 Piggy-back text

902 views • 56 slides
Information Retrieval CS276: Information Retrieval and Web Search Pandu Nayak and Prabhakar

Information Retrieval CS276: Information Retrieval and Web Search Pandu Nayak and Prabhakar

Introduction to Information Retrieval Introduction to Information Retrieval CS276: Information Retrieval and Web Search Pandu Nayak and Prabhakar Raghavan Lecture 4: Index Construction Introduction to Information Retrieval Plan Last

611 views • 49 slides
Information Retrieval CS276: Information Retrieval and Web Search Text Classification 1 Chris

Information Retrieval CS276: Information Retrieval and Web Search Text Classification 1 Chris

Introduction to Information Retrieval Introduction to Information Retrieval CS276: Information Retrieval and Web Search Text Classification 1 Chris Manning, Pandu Nayak and Prabhakar Raghavan Introduction to Information Retrieval Ch. 13

658 views • 47 slides
Search and Information Retrieval Search on the Web 1 is a daily activity for many people

Search and Information Retrieval Search on the Web 1 is a daily activity for many people

Search and Information Retrieval Search on the Web 1 is a daily activity for many people throughout the world Search and communication are most popular uses of the computer Applications involving search are everywhere The field of

1.06k views • 38 slides
Three types of information systems: Information-Retrieval Systems (IR) Search large bodies

Three types of information systems: Information-Retrieval Systems (IR) Search large bodies

Three types of information systems: Information-Retrieval Systems (IR) Search large bodies of information which are not specifically formatted as formal data bases. Web search engine Keyword search of a text base Typically

352 views • 12 slides
Three types of information systems: Information-Retrieval Systems (IR) Search large bodies

Three types of information systems: Information-Retrieval Systems (IR) Search large bodies

Three types of information systems: Information-Retrieval Systems (IR) Search large bodies of information which are not specifically formatted as formal data bases. Web search engine Keyword search of a text base Typically

479 views • 12 slides
Three types of information systems: Information-Retrieval Systems (IR) Search large bodies

Three types of information systems: Information-Retrieval Systems (IR) Search large bodies

Three types of information systems: Information-Retrieval Systems (IR) Search large bodies of information which are not specifically formatted as formal data bases. Web search engine Keyword search of a text base Typically

305 views • 13 slides

More recommend