How to Evaluate Accuracy of Biometric Systems Peter Vojtek - PowerPoint PPT Presentation

How to Evaluate Accuracy of Biometric Systems Peter Vojtek peter.vojtek@innovatrics.com

Intro ● PeWe member ○ 2006 - 2010 ● Innovatrics ○ fingerprint-based biometrics ■ SDKs ■ large-scale fingerprint matching (AFIS) ■ end-products ○ 300M+ people

Typical Biometric Modalities ● Face ● Iris ● Retina ● Signature ● … ● Fingerprints

Examples of Biometric Systems ● National ID ● Social Insurance ● Health Insurance ● Border Control ● Driving Licenses ● Voter’s Lists ● Ghost Workers Identity Management Systems

Accuracy ● Enrollment: ○ FTE: Fail to Enroll ● Verification: ○ FMR: False Match Rate ○ FNMR: False Non-Match Rate ● Identification: ○ FPIR: False Positive Identification Rate ○ FNIR: False Negative Identification Rate

AFIS ● Automated Fingerprint Identification System ○ CAFIS, MegaMatcher, ExpressID AFIS ● 100 000 000+ fingerprint comparisons / second / CPU

How to Compute Accuracy 1. Enroll 1000 different records for the first time Every record must be unique, we label them A = {a1, a2, … , a1000} 2. Enroll same people again We label them B = {b1, b2, … , b1000}. We know that a and b with the same index are from the same person 3. Perform verification of all records from A against all records from B In total we will have 1 million matching results for every pair. 4. Analyze 1000 scores having the same index This is called genuine distribution. 5. Analyze 999 000 scores having different index This is called impostor distribution. 6. Calculate FNR and FNMR for different scores to get ROC curve

How to Compute Accuracy Should: Accept Should: Reject Reality: Accepted TA FA Reality: Rejected FR TR

How to Compute Accuracy Should: Accept Should: Reject Reality: Accepted TA (1000) FA (0) Reality: Rejected FR (0) TR (999 000) The false non-match rate is the expected probability that A i will be falsely declared not to match to B i . FNMR = FR / (FR + TA) = 0 : 1000

How to Compute Accuracy Should: Accept Should: Reject Reality: Accepted TA (999) FA (0) Reality: Rejected FR ( 1 ) TR (999 000) The false non-match rate is the expected probability that A i will be falsely declared not to match to B i . FNMR = FR / (FR + TA) = 1 : 1000

How to Compute Accuracy Should: Accept Should: Reject Reality: Accepted TA (1000) FA ( 0 ) Reality: Rejected FR (0) TR ( 999 000 ) The false match rate is the expected probability that a sample will be falsely declared to match a single randomly-selected “non-self”. FMR = FA / (FA + TR) = 0 : 999 000

Similarity score FNMR FMR

FNMR ROC Curve FMR

Examples of Real-life Accuracies ● iPhone 5S ○ Verification, 1 finger, FMR 1:50 000 ● Time Attendance System ○ Verification, population 10-1000, 1 finger, FMR < 1:1000 ● Population 4.5M, 6 fingers ○ Identification, FPIR < 1:100 000, FNIR < 2% (1:50)

How to Influence Accuracy ● Threshold ○ Security vs. comfort ● Fingerprints ○ how many, which positions ○ quality ○ position anonymization ● Template extractor ● Matching speed ● Discriminative ability of bio. modality ○ Dataset size ~ FMR

Customer and Accuracy ● Not aware ● Aware, but ignoring ● Cooperating ● Demanding

Datasets ● physical access ● huge difference in accuracy due to quality of fingerprints ● annotated datasets

Independent Accuracy Tests ● NIST PFT ○ Proprietary Fingerprint Template Evaluation ○ Verification ● NIST FpVTE ○ Fingerprint Vendor Technology Evaluation ○ Identification ● NIST MINEX ○ Minutia Exchange

NIST PFT II

Resources ● INDIA UID ● Introduction to Biometrics ○ Springer, 2011 ● Best Practices in Testing and Reporting Performance of Biometric Devices ○ http://ftp.sas.ewi.utwente.nl/open/courses/intro_biometrics/Mansfield02.pdf

Other Keywords ● Deterrence effect ● Fingerprint quality (NFIQ) ● Speed ● Template extraction ○ basic pattern, minutiae points, pattern ● Segmentation ● ABIS ● Positive/Negative identification ● Criminal/Civil AFIS ● India UID, Indonesia eKTP ● iPhone ● FAR, FRR