Assessing System Efficiency in Providing Public Benefits Supportive Parents Information Network (SPIN) Caring Council February 9, 2012
Our Goals Pr Proposing oposing a pr a process ocess – no no a pr a product oduct To De Develop elop an an aut authe hent ntic ic pa partn tner ership ship amo among ng all all key ey play player ers: s: HHS HHSA A Mana Manage gemen ment, t, Wor orker ers, s, Appli licants/ ts/Recipien ipients ts To successfully address and remove the barriers to the efficient and compassionate delivery of public benefits While we are critical – we don’t wish to be adversarial We want the system to work and we assume the County does as well
Concerns The here e is n is no sys o systemic, o temic, objec bjecti tive mea e means o ns of monitoring of the system’s effectiveness It seems that all “outputs” are assumed to have a positive outcome – no assessment is made to test that assumption Data that are reported on the system’s functioning only provide a partial picture – an overly positive picture The he f fun unda damen mental tal app pproa oach h is flaw is flawed ed “Public assistance eligibility determination processes and operations needed to be refreshed and aligned with CalWIN System” Kim Forrester System designed to serve the tool – rather than the tool designed to serve the system
Examples Enrollment v. Participation Rate The County consistently refers to its enrollment as “participation rate” and points to the significant rise in cases. While enrollment has increased there is no evidence that the participation rate has increased. Compliance Rate & Pending The County consistently reports its Monthly Compliance Rate at above 90% but fails to mention that this number is based on less than half the monthly cases. San Diego continues to have an unusually high pending rate in comparison to other large counties and the state
Enrollment v. Participation Rate Estimates of Participation v Actual Participation 250,000 225,000 200,000 Individual Enrollment 175,000 150,000 125,000 100,000 75,000 50,000 25,000 0 9 9 9 0 0 0 1 1 1 9 9 9 9 9 9 9 9 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 9 0 1 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 1 1 1 1 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - n b r y n l g t v c n b r y n l g t v c n b r y n l g t v c r u p r u p r u p p c o p c o p c o a e a a u u e e a e a a u u e e a e a a u u e e J O J O J O J M A M N D J M A M N D J M A M N D F J A S F J A S F J A S Months Participation 3% change 9% change Linear (Participation) Linear (3% change) Linear (9% change) Using Unemployment as indicator of demand – shows County is keeping up with demand – not increasing its participation rate
Compliance Rate & Pending Pending Rate Across Largest Counties 6 5 6 0 5 5 5 0 % Pending End of Month 4 5 4 0 3 5 3 0 2 5 2 0 15 10 5 0 Jan-09 Jul-09 Nov-09 Jan-10 Jul-10 Nov-10 Jan-11 Jul-11 Nov-11 Feb-09 Apr-09 May-09 Jun-09 Aug-09 Sep-09 Oct-09 Dec-09 Feb-10 Apr-10 May-10 Jun-10 Aug-10 Sep-10 Oct-10 Dec-10 Feb-11 Apr-11 May-11 Jun-11 Aug-11 Sep-11 Oct-11 Dec-11 Mar-09 Mar-10 Mar-11 Months San Diego State Los Angeles Orange Riverside San Bernadino The County has consistently had a pending rate of over 50% since August of 2009 and peaked at 61% in November 2011
Technical v Political It is important to remember - because this is a public agency this is as much a technical problem as a political problem Politics is – The translation of the ideal into the real This definition raises the questions: • Who gets to define the “Ideal” • Who gets to create the “real” Our hope is that we can create a process where the three major players [HHSA Mgt, Workers, Recipients] work together to define the ideal and create the real
A Framework Outputs Outcomes Impact HHSA Feedback The Evaluation must be FORMATIVE & SUMMATIVE: It must examine the process (formative) and the outcomes (summative). Quality feedback mechanisms need to be developed and institutionalized: The efficiency of any system is related to the quality of the feedback it gets and its ability to incorporate that feedback. Must expand whose feedback is accepted & incorporated – Management, Workers, Clients.
Scope of the Evaluation Needs to examine the Business Process Re-Engineering (BPR) The Heart of BPR: The ACCESS Call Center Task-Based Operation Needs to assess Outcomes Is it achieving its intention? [Requires Management Perspective] How well is it being implemented? [Requires Worker Perspective] What is the experience of the recipient? [Requires Client Perspective]
ACCESS Call Center Intelegy Report provides assessment of ACCESS Call Center Valida alidate tes s some some of of ou our r co conc ncer erns: ns: “There is no evidence of a joint techanical planto support the current service delivery model, operational needs of ACCESS or the FRC‟s or requirement of a future „online‟ service delivery model.” Suppor Supp orts some of ts some of ou our asse r assessmen ssment t & & rec ecommen ommenda dation tions: s: • Migrate work force to flexible work assignments based on customer need vs. task specialization • Revise processes to minimize customer handoffs • Staffing levels not sufficient • Upgrade FRCs
Task Based Operation: Summative Assessment Evaluation begins with operationalizing Outcomes Operationalized outcomes provide measureable indicators of how well the system is functioning Examples of Outcome Measures: • Percent of cases disposed of within 30 days • Percent of cases pending each month in relation to other large counties • Changes in the participation rate [not enrollment] • Denial rates in relation to other large counties • Negative error rate [both false positives and false negatives] • Number of appeals – percent reversed • Incidence of lost paperwork
Environment What are the conditions in the FRCs - how well are they functioning? In particiular: • What are the physical conditions? • What resources are available with in the Center? • How long does it take to complete one‟s business? • How many trips does it take? What is effect of Early Fraud Detection on client experience & what is its value? • 25 to 30% of all food stamps recipients also receive CalWORKs, requiring them to submit to Project 100% • Anecdotal evidences indicates Project 100% is a deterrent – not for fraud, but fear of losing children • DA reports millions in “Cost Avoidance” but has no legitimate data to support claim [See State Auditor‟s report on Anti -Fraud activities]
Task Based Operation: Formative Assessment Formative Evaluation needs to examine both the efficiency of the process and its impact on the recipient Evaluation of internal processes must include workers & ability of clients to provide feedback Examples of things that make the work harder: • Barriers to correcting errors in CalWIN • No means for tracking misfiled documents • Multiple & conflicting notices of Action sent to client in one day • No single person has overall knowledge of any case
Recommendations SSAB from a workgroup made up of HHSA Management, workers, and clients for the purpose of: • Developing a set of measurable outcomes • Creating a system for monitoring the impact of internal processes on those outcomes • Developing an ongoing evaluation/monitoring system that includes regular reporting to the SSAB
Cautions Numbers don’t lie, people do The value of data is in its interpretation Numbers are only indicators of a process or an outcome – they are not the process or the outcome. No indicator is a perfect reflection of what it is measuring. The best interpretation comes when all perspectives come to consensus. You get what you measure The way outcomes are operationalized and measured will affect worker behavior If an ACCESS work is assessed on length of call, s/he will focus on ending calls in the prescribed time. If an ACCESS worker is assessed on the percent of successful resolutions, s/he will focus on finding resolutions to a caller‟s issues
Recommend
More recommend
