Speech Processing 11-492/18-492 Spoken Term Detection/Key Word - PowerPoint PPT Presentation

Speech Processing 11-492/18-492 Spoken Term Detection/Key Word Spotting

Listening for Keywords Listening for Keywords  No need to use push-to-talk No need to use push-to-talk  Always on Always on

Examples Examples

Uses Uses  Activate a computer/task Activate a computer/task – “Computer, locate Commander Riker” Computer, locate Commander Riker” “  Robot/device control Robot/device control – “Next slide” Next slide” “  Broadcast News, Meetings Broadcast News, Meetings – Tell me when “Microsoft” is mentioned Tell me when “Microsoft” is mentioned  “ “Triggers” vs “(General) Keyword spotting” Triggers” vs “(General) Keyword spotting”

Google Now Google Now  “ “Okay Google schedule” Okay Google schedule”  Always on Always on  Hands free Hands free  Uses battery all the time Uses battery all the time  But “Okay Google” is only said when its meant But “Okay Google” is only said when its meant

How to do it How to do it  Full ASR Full ASR – Run full ASR all the time Run full ASR all the time – Post process it to find keyword Post process it to find keyword – Very computationally expensive Very computationally expensive  Model for Keyword Model for Keyword – Build an acoustic model just for keyword Build an acoustic model just for keyword – Run DTW (or similar) on the acoustics Run DTW (or similar) on the acoustics

How to measure its success How to measure its success  False Positives False Positives – Find examples that aren't there Find examples that aren't there  False Negatives False Negatives – Miss examples that are there Miss examples that are there  What is the relative cost of the error What is the relative cost of the error – FN: FN: • Trigger: person will say it again Trigger: person will say it again • KWS: its lost KWS: its lost – FP FP • Trigger: an extra command will be interpreted Trigger: an extra command will be interpreted • KWS: time wasted in looking at example to discard it KWS: time wasted in looking at example to discard it  Change your thresholds Change your thresholds – Trigger: less FP Trigger: less FP – KWS: less FN KWS: less FN

Hot Spots Hot Spots  Only look in good places Only look in good places  Speech vs non-speech Speech vs non-speech  Target Speaker vs Other speakers Target Speaker vs Other speakers  “ “Long” speech vs (very) short speech Long” speech vs (very) short speech  Prosodically interesting parts Prosodically interesting parts

Noise Cancellation Noise Cancellation  Remove known (irrelevant) channels Remove known (irrelevant) channels – Remove TV feed from ASR stream Remove TV feed from ASR stream – Remove Others from conference call Remove Others from conference call

Boosting Boosting  (For Keyword Spotting) (For Keyword Spotting)  Words are defined by the company they keep Words are defined by the company they keep  Words will typically appear more than once Words will typically appear more than once – Near to each other Near to each other  Recognition with lattices (i.e. choices) Recognition with lattices (i.e. choices) – If a document has one occurrence If a document has one occurrence – boost others boost others  If related words in document If related words in document – Boost others Boost others

Choose your Trigger Word Choose your Trigger Word  Something unlikely to appear elsewhere Something unlikely to appear elsewhere  Something easy to recognize Something easy to recognize  Something not confusable Something not confusable  Something easy to remember Something easy to remember  Something relevant Something relevant  Good examples Good examples – Affirmative and negative (vs yes and no) Affirmative and negative (vs yes and no) – “Okay Google” Okay Google” “ – “Nebuchadnezzar” Nebuchadnezzar” “  Bad examples Bad examples – “huh” “sass” huh” “sass” “

IARPA Babel Project IARPA Babel Project  4 teams 4 teams – CMU/JHU CMU/JHU – BBN BBN – IBM IBM – ICSI (and others) ICSI (and others)  35 languages over 5 years 35 languages over 5 years – Low resource languages Low resource languages – Pashto, Bengali, Vietnamese, Cantonese,... Pashto, Bengali, Vietnamese, Cantonese,...  100 hours, 10 hours, and 0 hours 100 hours, 10 hours, and 0 hours

0 Data Case 0 Data Case  No labeled data in “unknown” language No labeled data in “unknown” language – So can't build initial ASR engine So can't build initial ASR engine – Build index in the audio domain Build index in the audio domain  Keywords are spoken Keywords are spoken – “Look for 'apple computers' Look for 'apple computers' “  Issues Issues – Cross speaker mapping Cross speaker mapping – (Use of synthesis – but need data) (Use of synthesis – but need data)

Spoken Term Detection Spoken Term Detection  “ “Old” goal but popular again Old” goal but popular again – Not in fact much easier than full ASR Not in fact much easier than full ASR – You can constrain the problem though You can constrain the problem though • Limited keywords, train people Limited keywords, train people  Can be used for search Can be used for search – But need good ASR in language But need good ASR in language