Mixed-Initiative Application for Equipment Diagnostics Bill - PowerPoint PPT Presentation

Mixed-Initiative Application for Equipment Diagnostics Bill Cheetham Bill Cheetham General Electric Global Research General Electric Global Research Outline: • Appliances Help Desk Application • Comments on Human and Software Agent Initiative • Self-Confidence in a Diagnostic System

General Electric Overview Aircraft Engines Aircraft Engines Capital Services Capital Services Consumer Products Consumer Products Industrial Systems Industrial Systems Medical Systems Medical Systems Plastics Plastics Power Systems Power Systems Specialty Materials Specialty Materials Transportation Systems Transportation Systems NBC NBC

Research Locations Niskayuna, NY – World Headquarters Bangalore, India Munich, Germany Shanghai, China

Appliances Help Desk Increase the ability of an Factory Service (FS) Calltaking Rep to solve a consumer product complaint on phone without scheduling service. Save The Call Challenges Limited supply of qualified people (currently 240 - 260 FS reps) High turnover of ASI reps ( ~70% yearly) Little time for training of reps Increasing demands on call reps Corporate Image depends on service

Inference (eGain) k-commerce tool Case-Based Reasoning (CBR) with Question Answering and Rule-Based System Case Title Refrigerator - No Ice Case Description Refrigerator doesn’t make ice when lever arm is up Questions Refrigerator What type of appliance is it? No Can you make ice? Yes Is the lever arm in the up position? Lower lever arm Action

Developer Interface - Rules Rules answer questions based on keywords or questions. Keywords: If keyword Model_number = **7***** then question “What is the type of your refrigerator?” = Monogram Questions: If question “Where is the freezer on your refrigerator?” = Top_Mount then question “Does your refrigerator have an ice maker?” = False These rules specify when to take initiative

User Interface • Unlimited number of cases Desktop launches CBR and sends brand, Product line and Problem statement to CBR. • Only relevant cases displayed • Attachments / Graphics Calltaker collects information on call Calltaker attempts to save the call using CBR using Calltaking 32

User Interface Suggest Attribute Suggest Feature

Outline: • Appliances Help Desk Application • Comments on Human and Software Agent Initiative • Self-Confidence in a Diagnostic System

Human Initiative What initiative do you want from a person? • Go beyond just doing requested tasks • Example: Outstanding employees show initiative Requirements for initiative • Understanding of goals and priorities • Understanding of current situation • Identification of task from current situation that can help goals • Identification of potential problems from doing task • Confidence that task should be performed • high – do task • low – don’t do task • medium – ask if task should be done • Ability to perform task • Inform others task has been done

Software Agent Initiative

Bad Software Agent Initiative Questions about software agent initiative • Is a person with initiative but no common sense dangerous? • Do software agents have common sense? • Can a software agent with initiative be dangerous? Dangers of software agent initiative • Bothering the user • Locking the user out (stealing cycles) • Acting in an unknown way (possibly undoing users actions) • Making the user undo the computers actions • Keeping the user from doing tasks (2001’s HAL) • World domination - Terminator / Matrix

Good Software Agent Initiative

Good Software Agent Initiative Attributes of grammar/spelling checking software agent • Advice only given when rules violated (grammar, spelling) • Unobtrusive • User retains control • User can ignore, accept, or teach Requirements for software agent initiative • Understanding of goals and priorities • Understanding of current situation • Identification of task from current situation that can help goals • Identification of potential problems from doing task • Confidence that task should be performed • high – do task • low – don’t do task • medium – ask if task should be done • Ability to perform task • Inform others task has been done

Human and Software Agent Initiative Start action action initiative User Agent initiative End Agent has High confidence

Outline: • Appliances Help Desk Application • Comments on Human and Software Agent Initiative • Self-Confidence in a Diagnostic System

The Problem GE is using CBR to automate real world decision tasks that are currently being done by people. > Equipment Diagnosis (Appliances, Power Turbines) > Approving Financial Applications (Mortgage, Insurance) These automated systems need high accuracy They do not have to automate every problem New Process Old Process Automate Action Take CBR Human Problem Problem Action Assist Systems User

Solution CBR system produces solution and confidence The confidence specifies if the predicted accuracy in solution is high enough to automate the action or not Automate the action when confidence is high yes Automate Is Solution Action CBR Confidence Problem Systems high? Confidence Assist User no

Solution Problem RETRIEVE RETRIEVE Similar Cases REUSE REUSE RETAIN RETAIN And Predict And Predict Accuracy Accuracy CASE-BASE Solution Solution REVISE REVISE And Confidence And Confidence Predict the accuracy of the solution in the REUSE phase Output both a solution and the systems confidence that the solution is accurate

Confidence in People I was gratified to be able to answer promptly, and I did. I said I didn't know. - Mark Twain The mean as concerns fear and confidence is courage: those that exceed in confidence are foolhardy, while those who lack confidence are cowardly. - Aristotle from “The Doctrine of the Mean”

Related Work in CBR Bruce McLaren and Kevin Ashley (ICCBR 2001) SIROCCO – provides advice on engineering ethics Proposed rules for when the program can not suggest advice (i.e. help it know what it knows) 1) If the best superficial matching case is a weak match 2) If enough of top N cases differ in their solution They found these rules improve system’s performance

Creating a Confidence Measure 1) Identify potential confidence indicators. 2) Use statistics about the case base to determine which indicators work best for calculating confidence. 3) Create an algorithm that takes the indicators and produces a confidence value • Value is a term (high, low) for automation • Value is number [0 – 1] for assisting users

Potential Confidence Indicators Using k nearest neighbors to determine one best solution from a set • Sum of similarities for retrieved cases with best solution • Maximum similarity of a case with best solution • Number of cases retrieved with best solution • Sum of similarities for all other solutions (not best) • Difference between sum of similarity for best and all other • Number of cases retrieved with second best solution • Percent of cases retrieved with best solution • Sum of similarities for second best solution • Average similarity of cases with best solution • Average similarity of cases with second best solution • Standard deviation of numeric solutions suggested • Rules created from domain knowledge

Determine Which Indicators are Best Run leave-one-out testing, printing out all indicators, plus the real solution, and if CBR’s best solution is correct Use C4.5 to determine best indicators Case Number 1 2 3 4 5 6 7 8 9 1) Sum of similarity for best 16.9 20.2 19.5 16.6 20.6 12.8 14.4 16.7 19.1 2) Max similarity of best 2.2 2.8 2.6 2.5 2.8 2.2 2.5 2.6 2.6 3) Number of cases with best 8 8 8 7 8 6 6 7 8 4) Sum of similarity for all other 0 0 0 2.3 0 4.2 4.6 2.3 0 5) Difference (1 - 4) 16.9 20.2 19.5 14.3 20.6 8.6 9.8 14.4 19.1 6) CBR’s second solution NA NA NA F3 NA F2 F0 F3 NA 7) CBR’s suggested solution F0 F1 F0 F0 F0 F0 F4 F0 F0 Real solution F0 F1 F0 F3 F0 F2 F0 F0 F0 Is CBR’s solution correct? YES YES YES NO YES NO NO YES YES

Creating Confidence Algorithm C4.5 output: Start Legend Difference (1 – 4) > 15.1 <= 15.1 2 nd _Solution 377 Count 9% F0 F1 F2 F3 F4 Error Rate Max_Similarity 25 16 6 1 16% 21% 66% 0% <= 8.9 >8.9 Want a high 7 12 count and 84% 8% low error rate Confidence Equation: If Difference (1-4) > 15.1 or 2nd_solution = F2 and Max_Similarity_for_Best > 8.9 then confidence is High else confidence is Low

Conclusion Initiative is good when system is confident

end