Getting America Back to Work

Predictive Modeling and Behavioral Analytics in Unemployment Insurance

Meet Carol. She recently got
UI Overpayments Cost $3.2B Annually laid off from her job and has Data from Internal and External Sources
decided to apply for UI benefits
Objective: Identify and prevent improper payments in Data Sources:
unemployment insurance benefits administered • Historical claims data (2012 – 2013) —
Client’s Unemployment Framework for Claims
Background: According to the USDOL, improper payments cost
and Tax Services (uFACTS) database
more than $3.2B to Unemployment Insurance (UI) trust funds in
2013, over 8% of the $40B in benefits paid out. Currently,
This is a pointer style
• ZIP code level demographic data (2013) —
overpayments are often identified after payments have already been Easy Analytic Software, Inc. (EASI)
made – potentially months later – at which point it can be difficult to Number of claims at the address,
recover the costs. Data Types
Whereand Sample Variables
Claimants Live geographic area, Past fraud in
the region
UI Overpayments by Cause
What Claimants Report Gender, Age, Occupation

To initiate her benefits, Carol Timing of filing claims, Method of

19% Benefit Year Earnings How Claimants File Claims filing, Changes in established
fills out the
34% Separation Issues Initial Application
behavior
Number of base period
Work Search Where Claimants Work employers, Past fraud at the
33% employer, Employer attributes
14% Other
Past fraud and overpayment
As she fills out the application, Past History on Claimants behavior
Carol’s risk of getting overpaid
is assessed and her
application is customized
Models Predict Fraud in Real-Time Benefits Realized through Model Applications
Three logistic regression models were developed for different Prioritization of Investigatory Efforts
overpayment types. Predictive strength, data availability/reliability, Carol completes and submits to maximize staff efficiencies and identify improper payments
and business interpretation were all considered during variable the application faster. Workload on adjudication staff is reduced by 30-40% with
selection. Higher Lower minimal impact on overpayments by: Applicant Information
Font Color: Risk Font Size: Importance
State agency completes fact- Applicant Name: Carol Jones
finding and adjudication • Prioritizing high risk claims for Weeks Certified: 10
Benefits Paid: $5,000
investigation Risk Assessment: 95

To receive benefits, each

• Auto-adjudicating low risk Priority level: HIGH
week Carol fills out the claims
Weekly Certification
Survival Curve: Audited vs. Non-audited Claimants Enhanced Work Search Audits
to help claimants get back to
work up to 10 weeks faster.
Based on behavioral changes Non-audited claimants stay on
Separation Overpayments Model Benefit Year Earnings Fraud Model and new information reported,
(Initial Application) (Weekly Certification)
Carol’s risk of committing
benefits longer – decrease
fraud is assessed and her average time on benefits by
Model Performance application is customized conducting automated work
5% Highest Predicted Overpayment search audits, especially for high
300% • 2x higher rate of fraud at initial claim
• >3x higher rate of separation overpayment risk claimants.
• Weekly certs: >4x higher rate of fraud
250%

Carol completes and submits

Behavioral Messaging
Actual Relative Fraud / Overpayment

200% the certification

to prevent improper payments by promoting honest behavior.
150% 10% Lowest Predicted Overpayment Serving dynamic and customized messages to 10% of the highest
• 89% lower rate of fraud at initial claim
• 71% lower rate of separation overpayment
risk claims responsible for 49% of earnings fraud can change
100%
• Weekly certs: 89% lower rate of fraud claimants’ behavior by the time they hit “Submit”:
State Agency completes fact-
Reminder
50% finding and adjudication • Certification Boxes —
9 out of 10 people in your county report earnings accurately. If you
0%
“I certify that the information worked between 2/24/2013 and 3/2/2013, please ensure you
report these earnings. Please report all information accurately.
For each certification week I am providing is accurate.” Edit Responses Continue

-50% that Carol meets the eligibility • Persuasive Messages —

Earnings Fraud (Initial Application) requirements, she is paid
Separation Overpayment (Initial Application) “We verify your employment and earnings information.”
-100% Earnings Fraud (Weekly Certification) Benefits
Top
• Educational or clarifying messages —
1 2 3 4 5 6 7 8 9 10
5% “To qualify for benefits, you must have been physically able to
Low Highest In a few weeks, Carol finds a
Model Risk Assessment from the
work, meaning you were not injured, ill, or otherwise physically
Baseline new job and is able to roll off
Overall Average Fraud /
Moment Claim/Certification Submitted UI benefits! unable to accept work.”
Overpayment Rate

Anomaly Detection Identifies ~50% of Hard Results Measured with RCTs

Fraud at Time of Application After implementation of the solution, a series of randomized control
trials (RCTs) will be run to quantify the following expected effects:
An ensemble of anomaly detection algorithms was developed to
combat organized fraud schemes like identity theft and fictitious This is a pointer style

employers. • Improved ease-of-use and user experience

• Back to work, faster
• Sentinel Effect – increased honesty under evaluation

• Reduced overpayment dollars

Cluster Analysis Association Rules
• Decreased time to detection and resolution of overpayments

STATE • Efficient agency resource allocation

Predicted Age

• Earlier identification of emerging fraud schemes

As results develop over 10-12 weeks, continous monitoring of the

Benford’s Law
Reported Age
Likelihood Methods results is necessary in order to optimize risk-based business rules.