AIRS

INFORMATION & REFERRAL OUTCOMES REPORT

AIRS OUTCOMES PROJECT REPORT

Revised August 10, 2011

July 15, 2011

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

Copyright 2011 Alliance of Information and Referral Systems and Integer Research & Consulting, LLC, except as otherwise stated. All rights reserved. No part of this document may be reproduced in any form or by any means without express written permission, except for the non-profit purposes of education and scientific advancement. For information, please contact the Alliance of Information & Referral Systems, P.O. Box 33095, Portland, OR 97292.

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

TABLE OF CONTENTS
EXECUTIVE S UMMARY .............................................................................................................................................5 I. BACKGROUND .......................................................................................................................................................7 II. P ROJECT C OMMITTEE .........................................................................................................................................8 III. P ROJECT L ITERATURE ........................................................................................................................................9 IV. C ONCEPTUAL F RAMEWORKS ...........................................................................................................................10
1. Intervention Framework .......................................................................................................................................................10 2. Outcomes Framework ...........................................................................................................................................................12 3. Process Framework ...............................................................................................................................................................15 4. Measurement Framework ....................................................................................................................................................18

V. M EMBER O UTCOMES S URVEY ..........................................................................................................................22 VI. S UMMARY ........................................................................................................................................................29 VII. M OVING A HEAD .............................................................................................................................................30
Key Concerns ................................................................................................................................................................................30 Project Strategy ............................................................................................................................................................................31

APPENDICES ............................................................................................................................................................33
Appendix 1: References ............................................................................................................................................................34 Appendix 2: Logic Models .........................................................................................................................................................37 Appendix 3: 211 LA Developmental Screening Project Fact Sheet ........................................................................................40 Appendix 4: AIRS Outcomes Survey .........................................................................................................................................41

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

EXECUTIVE SUMMARY
In December of 2010, the AIRS Board commissioned Integer Research & Consulting of Chicago to develop a funding prospectus for I&R/A outcomes research that would be suitable for presentation to potential funders, participants, and the public. 1 Integer conducted a review of literature pertinent to I&R/A outcomes, populated a project website with the results, and began working with a Project Team in January, 2011. Within the first month, it was apparent that the project would have to focus at a foundational level as a first step toward developing a funding prospectus. This report defines and describes these foundations and highlights factors to consider about moving the work of I&R/A outcomes forward. It also provides ample material with which to build a prospectus in the future. The following observations are offered for the consideration of the AIRS Board. The literature on I&R/A outcomes is somewhat sparse, but several seminal sources were identified and used in the project. These include a comprehensive text on I&R/A written by Risha Levinson, one of AIRS founders, and three research papers that provide a general framework for I&R/A outcomes. Solid conceptual foundations for I&R/A outcomes exist and offer excellent development potential. In exploring these, we iden- tified four key structural components that require further development for our purposes. 1. Intervention Framework: Risha Levinson developed an intervention framework that encompasses all the major varia- tions of I&R/A services. She provides important support for the idea that a single, flexible model can be used to char- acterize outcomes across the range of I&R/A interventions, whether they are for for aging and disabled populations, the community at large, other special populations, or people in crisis. 2. Outcomes Framework: The research of Matthew Saxton in the US and Ann Netten in the UK provides a robust model for understanding the impact of I&R/A on three levels: on clients, on community organizations, and on society. Out- comes data from all three levels will offer the strongest possible evidence of the far-reaching value of I&R/A.

1 Throughout this report, I&R/A is used to designate the full spectrum of interventions that AIRS members offer the public: information, referral,

and/or assistance.

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

3. Process Framework: We began developing an end-to-end, client-centered process framework. The process starts with the clients request for help from the I&R/A organization and encompasses all the key steps he or she takes until the needs or issues are addressed by the service system. This process framework is essential for defining the full range of I&R/A outcomes, because they extend beyond the I&R/A organization to other organizational and individual actors in the community. 4. Measurement Framework: Finding a means of measuring outcomes that is understandable, affordable, practical, and applicable across the range of AIRS member organizations and services is a tremendous challenge. We achieved sig- nificant insights by examining several measurement and feedback models. These offer excellent development poten- tial, but there is a long way to go. We decided to assess the interest of the AIRS membership in the subject of I&R/A outcomes and related development work. We mounted an online survey in April; the compiled survey data suggests that: - The vast majority of respondents feel that I&R/A outcomes are very important today and are essential to securing fund- ing from all the major channels. - They rate three factorsestablishing clear definitions, methods of measuring outcomes, and information on best prac- ticesas very important in I&R/A outcomes development work. - Many respondents stated that having an industry standard for I&R/A outcomes is very important, and they voiced their appreciation for AIRS considering such an initiative. At the end of this preliminary phase of the project, important quesaons remain about how to conanue the work. These princi- pally concern gathering buy-in and support from key member consatuencies, dening a way to manage extended development work, and securing the necessary human and nancial resources.

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

I. BACKGROUND
November of 2010, Integer Research and Consulting made its final presentation to the AIRS Board on the Environmental Scan Re- search Project. In the course of the presentation, Integer made four suggestions (Figure 1, left). The Board decided to act on the third suggestion, Formulate a universal and measurable I&R process (Figure 1, right). Integer was retained by AIRS to develop a funding prospectus for outcomes research suitable for presentation to potential funders, participants, and the public.
Figure 1: Slides from Integers Presenta\on to the AIRS Board in November, 2010

During December, 2010 and January, 2011, Integer conducted a review of literature pertinent to I&R/A outcomes and evidence- based practice. The research was pursued in several online academic databases, in Google Scholar and Microsoft Academic, and in general Internet searches using Google. Preliminary research material was compiled and loaded to a private website that was de- veloped for the project. 7

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

II. P ROJECT COMMITTEE
During January, the AIRS Board and staff were canvassed for suggestions on Outcomes Committee membership. Thought leaders in I&R/A, health and human services research, and library science were invited to join the Committee, with twelve accepting. Five teleconferences were conducted from February through April, with members participating as frequently as their schedules and interests allowed. In the intervals between teleconferences, workpapers and research materials were added to the website and team members were invited to review and comment upon them. The agenda and notes for each teleconference were provided to the team several days in advance, and served as a guideline for each discussion.
Table 1: Outcomes Project Commi`ee
Commi`ee Members DR. SHERRY BAME, Professor, Health Planning & Policy Texas A&M University LINDA DAILY, Director 2-1-1 and Disaster Services United Way Worldwide HELEN ELTZEROTH, Chief Programs & CommunicaZons Ocer NaZonal AssociaZon of AAA (n4a) ANGELA FISHER, Director of Client Services AAA for Southwest Florida/ADRC MARY HOGAN, Library Director Cora J. Belden Library DAVID JOBE, Director of 2-1-1 United Way of Greater Houston AMY LATZER, Chief OperaZng Ocer 211 LA County RANDY NICKLAUS, President 2-1-1 Big Bend JOHN OHANIAN, Chief ExecuZve Ocer 211 San Diego JACKY RODDY Inform Canada PATRICK ROGERS, ExecuZve Director InsZtute for Human Services MICKI THOMPSON, ExecuZve Director 2-1-1 Tampa Bay Cares, Inc. CHARLENE HIPES, Chief OperaZng Ocer Alliance of InformaZon and Referral Systems CLIVE JONES, Consultant Alliance of InformaZon and Referral Systems MARK H. NEUFFER, President Integer Research & ConsulZng, LLC Teleconference Dates

February 3, 2011 February 17, 2011 March 3, 2011 March 17, 2011 April 14, 2011

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

III. P ROJECT LITERATURE
Several hundred articles and publications were scanned by the project consultant during December, 2010 and January, 2011. The search produced over 20 books, articles, and presentations published between 1998 and 2011 relevant to I&R/A outcomes, per- formance measurement, quality, and cost/benefit analysis. The Committee supplied several useful additions to the literature. Lit- erature was uploaded to the project website and made available to the team. A list of references appears in Appendix 1. Four resources were identified as having essential importance to the project: Risha W. Levinson is an AIRS founder and a retired professor of social work. Her second book on the I&R/A field, New Routes to Human Services: Information and Referral, was published in 2002 and is the most authoritative and comprehensive overview writ- ten to date. Her depiction of I&R/A interventions as a continuum has been important to our project. Matthew Saxton is a professor at the Information School at the University of Washington. He and several colleagues at the Informa- tion School were commissioned to review the states 2-1-1 system and to suggest an impact framework that would extend beyond the usual measures of outputs into the realm of user outcomes. In 2-1-1 Informa@on Services: Outcomes Assessment, BenetCost Analysis, and Policy Issues, Saxton and his colleagues present a model of I&R/A outcomes as they relate to the individual, to commu- nity organizations, and to society in general. Ann Netten and Julien Forder, both from the University of Kent (United Kingdom) received government funding to review Informa- tion and Advice (I&A) services and to recommend how outcomes could be measured. Using the Saxton outcomes model as a foun- dation, Netten and Forder extended and detailed the framework, and developed an approach for measuring I&A outcomes at the level of the individual client. Measuring the Outputs of Information and Advice Services: Initial Report was published in 2008. In the two years following, Karen Windle and Ann Netten conducted a detailed study and authored Measuring the Outcomes of In- forma@on and Advice Services: Final Report in 2010. To date, this is the most comprehensive and relevant research found on I&R/A outcomes. The conceptual underpinnings of the outcomes logic model from Netten (and earlier from Saxton) are reviewed and amplified. All the details of the study are presented, and the study closes with five recommendations for expanding I&R/A out- comes research.

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

IV. CONCEPTUAL FRAMEWORKS
1. Intervention Framework The AIRS Environmental Scan underscored the diversity of AIRS membership, operations, and services. This raises the question of whether it is possible to evaluate outcomes of the various I&R/A services with a single outcomes model that uses standardized defi- nitions, measures, and methods. This question is addressed by Risha Levinsons work in New Routes to Human Services. As shown in Figure 2, she maps a wide range of I&R/A interventions. In our understanding, the horizontal axis shows the degree of intervention activity, with less at the left of the matrix and more at the right. On the first line, for example, the most basic intervention is providing in- formation (information assistance). This is ex- tended by adding a referral, and further extended with a follow-up. The vertical axis shows the degree of an interven- tions professional involvement, with the least in- volvement at the top and the most at the bottom. For example, there is minimal or no involvement in providing information (information assistance) in the top row. By contrast, case advocacy (third row) requires in-depth knowledge of the client and signifi- cant professional involvement.
Figure 2: Risha Levinsons Model of I&R/A Interven\ons

This model makes it possible to characterize I&R/A services as a family of related interventions. It en- ables I&R/A professionals to depict, analyze, and quantify their work within an extensible framework that defines the common fea- tures of, and the differences between, the various service permutations of I&R/A. 10

From New Routes to Human Services: Informa@on and Referral (p. 63), by R.W. Levinson, 2002, New York, NY: Springer Publishing Company, Inc., ISBN 0-8261-2393-7. 2002 by Springer Publishing Company, Inc. Reprinted with permission.

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

Figure 3 adapts the Levinson model to depict the I&R/A interventions practiced today by AIRS members. The diagram aims to show the es- sential continuity of the interventions; the actual content and place- ment of the intervention types would be more accurately supplied by I&R/A professionals. The top row, I&R Basic Components, shows the standard information, referral, and follow-up services provided to the general public.
Figure 3: Adapta\on of the Levinson Model

Crisis Intervention2 requires greater professional involvement with the caller (placing it farther down the vertical axis as a support serv- ice) and is also more extensive in terms of process, time, and re- sources (thus extending it farther along the horizontal axis). Amy Latzer introduced the Care Coordination intervention in our project work. Callers with a particular problem are guided by the I&R/A pro- fessional to pre-qualified community service providers suited to deal with that problem. The extent of the intervention and professional From New Routes to Human Services: Information and Referral (p. 63), by R.W. Levinson, 2002, New York, NY: Springer Publishing Company, Inc. , involvement are somewhat broader and deeper than the Basic I&R ISBN 0-8261-2393-7. 2002 by Springer Publishing Company, Inc. model. 3 The Assistance intervention (I&R/A) encompasses the Adapted with permission. broadest set of intervention transactions and potentially the greatest professional involvement. It includes all transactions that support the client on an ongoing basis through the clients search for help; it therefore encompasses services such as advocacy, assistance with transportation, help getting an appointment, motivational support, and case management. The model shows that I&R, Crisis Intervention, Care Coordination, and Assistance are closely-related members of the same family of professional practice which demand different extents of intervention and professional involvement. These I&R/A interventions may be characterized and assessed with a common outcomes framework that provides additional definitions, measures, and methods applicable to specific variations.
2 Crisis Intervention refers to assisting callers with mental health issues. 3 The Care Coordination intervention is described on the 211 LA County website at http://www.211la.org/?page_id=150 and in the Fact Sheet for 211 LA

Countys Developmental Screening Project in Appendix 3 of this report.

11

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

2. Outcomes Framework Matthew Saxton and his colleagues have made an important contribution to the I&R/A field by conceptualizing a three-level out- comes framework for individual service recipients, community organizations, and the community or society at large. Although Sax- ton developed the model for a 2-1-1 system study, we feel it is entirely applicable to the full continuum of I&R/A interventions. Fig- ure 4 provides a high-level representation that is further detailed on page 13. Figure 4: Schema\za\on of Saxtons Three-Level Outcomes Framework The terminology is probably familiar to most readers: inputs include funding, information, other resources, and clients. Processes include the full range of I&R/A interventions outlined on the previous page. Outputs consist of the quantifiable results of the processes: total cli- ent calls, client contacts, crisis interventions and so on. Outcomes are the measurable im- pacts of these I&R/A interventions: crises de- escalated; service provider contacts made by the client after he or she received a referral; and service uptake by the client after receiving a referral and/or assistance.

From 2-1-1 InformaZon Services: Outcomes Assessment, Benet-Cost Analysis and Policy Issues, by M.L. Saxton, C.M. Naumer, and K.E. Fisher, 2007, Government Informa@on Quarterly, 24, pp. 202-203. 2007 by Elsevier Inc. Adapted with permission.

This framework provides a means of broadly quantifying the outcomes and benefits of I&R/A for the individual, community organizations, and society at large; it invites further elaboration by I&R/A professionals. I&R/A organizations and the I&R/A profession would then have a robust and rigorous frame- work for demonstrating the help they provide to all the people and organizations they touch.

12

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

The following page shows the Saxton framework with details later supplied by Netten. Here are a few notes to help the reader re- view the diagram more easily. The Horizontal Axis shows the complete span of I&R/A acavity from inputs to outcomes. The inputs are shown in one large box to suggest that they are shared by the three levels of the individual, organizaaons, and community. The Ver\cal Axis encompasses the three levels of I&R/A impact: the individual, community organizaaons, and society. Individual-Level Processes: this box includes mulaple types of I&R/A intervenaons. This is Nekens conceptualizaaon based on Informaaon & Advice services in the United Kingdom. For the purposes of our Outcomes Project, this box would be populated with the types of intervenaons provided by I&R/A organizaaons in North America. Outcomes are characterized as short-term, intermediate, and long-term; the model aords the exibility to dene the length of these intervals on a case-by-case basis. The Outcomes Committee noted that it is important to distinguish outcomes that are more directly in the chain of I&R/A cause and effect (for example, a service connection enabled by an accurate and appropriate referral) from outcomes that are farther down the chain and which are not directly impacted by I&R/A activity (for example, the clients problem is resolved after he receives services). The red triangle symbol in the top right corner of some boxes indicates that the original Saxton content has been replaced with subsequent renements by Neken. The Outcomes Committee recognized that the Saxton-Netten outcomes framework invites further elaboration; much remains to be added in specific detail. Yet, as already noted, the framework provides an excellent basis for broadly characterizing the beneficial impact of I&R/A and is thus a way of both deepening and broadening the discourse on the value of I&R/A to all stakeholders.

13

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

Figure 6: Combined Saxton and Ne`en Outcomes Frameworks

From 2-1-1 InformaZon Services: Outcomes Assessment, Benet-Cost Analysis and Policy Issues, by M.L. Saxton, C.M. Naumer, and K.E. Fisher, 2007, Government Informa@on Quarterly, 24, pp. 202-203. 2007 by Elsevier Inc.; and Measuring the Outputs of InformaZon and Advice Services: IniZal Report, by A. Nemen and J. Forder, 2008, PSSRU Discussion Paper 2543/2, pp. 17-18. Personal Social Services Research Unit, University of Kent, Canterbury. Adapted with permission.

14

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

3. Process Framework The Saxton-Netten model supplies a general definition of I&R/A outcomes for individuals, organizations, and society. In order to de- fine and measure outcomes at an operational level, a detailed process and outcomes framework is needed. This framework would: Provide a standardized approach to describing processes, outcomes, and metrics usable for all I&R/A intervenaon types. Dene the key process steps for I&R/A interacaons with individuals, community organizaaons, and society. Dene the cause-and-eect relaaonships between process steps and their outcomes (also known as a logic model.) Provide a diagnosac capability to reveal where the process is not working as desired (e.g., due to barriers) or should be redesigned because the underlying process or logic models do not produce desired results. The Project Team took several approaches to detailing the process/outcomes models. Our first effort was to bring the Saxton- Netten model to a finer level of definition (see Appendix 2). It was then decided to focus this preliminary thinking on outcomes at the individual level. The team experimented with several different approaches, including a tabular format, flowcharts, and hybrids of these. Several key design principles were crystallized: The process needs to be dened as an end-to-end, client-centered framework. The process starts with the clients request for help from the I&R/A organizaaon and encompasses all the key steps he or she takes unal the needs or issues are addressed by the serv- ice system. This framework denes all the I&R/A outcomes from the start of the process to its compleaon. The model needs to distinguish clearly between outcomes that are directly facilitated by I&R/A activities (for example, contact with a service provider enabled by an accurate and appropriate referral) and outcomes for which the client or service organiza- tions have a direct responsibility (for example, the outcomes resulting from the client receiving services). The primary aim of the I&R/A process model is to improve the effectiveness in I&R/A as a distinct, but integral, part of the larger human service system. I&R/A processes and I&R/A outcomes contribute to, but are distinct from, those of the larger sys- tem which are assessed through program and system evaluations. Clearly defining and communicating the I&R/A process/ outcomes model, its terminology, and its practices would underscore its compatible role in the long-established field of health and human service outcomes evaluation.

15

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

Figure 7 on the following page was created after the last Project Team teleconference and reflects a synthesis of the thinking when the project concluded. The illustration shows the framework at the next level of detail below the Saxton-Netten model. It is still very simplified and is presented to exemplify current thinking, not to provide a definitive or detailed model. The top section of the framework includes a high-level block diagram of the end-to-end process. Note that the model includes three client states: the client before getting information, the client after receiving I&R/A help, and the client after receiving help from the direct service provider. These are placed in the diagram to show explicitly the clients before and after condi- tionsthat is, milestones in the process where outcomes should be defined and measured. The bottom portion of the framework shows a high-level logic model of the processes with representative outcomes. For ex- ample, an outcome of providing information is client knowledge: the clients knowledge will either be increased or will remain the same after contact with the I&R/A specialist. Other outcomes have a more contingent cause-and-effect relationship with I&R/A activities. Whether the client contacts the service provider after receiving a referral depends upon multiple factors, some of which are within the I&R/As control and oth- ers of which are not. If a client fails to reach a service provider because the referral information is deficient, this is a negative outcome over which the I&R/A has control. On the other hand, a client may or may not contact the provider based on factors (such as the clients motivation, family responsibilities, and access to transportation) that are in many cases outside the control of the I&R/A. Defining responsibility for outcomes enables I&R/As to correct negative outcomes for which they are accountable, and also to identify client, process, service provider, and system shortcomings which can be corrected or improved to foster better results. Outcomes in the model have been characterized as short-term, intermediate, and long-term based on the elapsed time, degree of control, and number of process steps and actors involved. For example, the I&R/A has significant control over the out- come of increasing a callers knowledge at the time of the contact; hence it is a short-term outcome. The I&R/A has partial control over the client contacting a service provider and it may take time for the outcome to be realized, hence the outcome is considered intermediate. Finally, the I&R/A has no direct control over the clients course of service, which may be temporally remote from the initial contact. To the extent, however, that the I&R/A facilitated the accession to services by providing infor- mation, a referral, or additional assistance, this long-term outcome is essential to record.

16

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

Figure 7: End-to-End, Client-Centered Human Service Model (Individual Level of Impact)

2011 Integer Research & ConsulZng, LLC

17

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

4. Measurement Framework The Project Committee discussed the significant challenges associated with capturing and measuring I&R/A outcomes data: In many cases, intelligence gathered about process eecaveness and outcomes is limited to ad-hoc feedback from clients or serv- ice partners reporang a maker on their own iniaaave. Such feedback is useful on a case-by-case basis, but not for general process or outcome improvement. In some I&R/A organizaaons, a quality assurance follow-up process has been designed and implemented (Standard Assessment and Referral with Follow-Up). Local pracace determines how follow-up paracipants are selected, the number and aming of follow- ups, the nature of the informaaon gathered, and the use of the informaaon. Some I&R/A intervenaons incorporate assistance services that extend beyond informaaon, assessment, and referral. In interven- aons like Care Coordinaaon and I&A for the elderly and disabled, conanued client contact and awareness of process outcomes are intrinsic to the service model. Consequently, outcomes data collecaon and analysis may actually be incorporated in the service model, as it is in the 211 LA Developmental Screening Project.4 David Jobe, a member of the Project Commikee, described a public health research project that uses a Navigator who provides moavaaonal support to the client and helps the client contact and engage needed services. The Navigator also develops informa- aon about and insights into the psychological, personal, and pracacal barriers the client faces in the service process. The Project Team discussed the potenaal of a navigator-like funcaon in the context of I&R/A outcomes. A hypotheacal system navigator could assemble informaaon on process volumes, characterisacs, and cycle ames for each segment of the end-to-end process. The funcaon could determine the root causes of process failure and could assist in improving process performance, de- ning best pracaces, and seong standards. As well, the funcaon could highlight client characterisacs that foster posiave service engagement or process failure. A sketch of the various I&R/A intervenaons, with their potenaal to measure outcomes, is provided on the next page in Figure 8. It was apparent that considerable effort would be required to develop an outcomes measurement approach that would be under- standable, affordable, and applicable to the wide variety of I&R/A practices among the AIRS members. AIRS members are exploring different features of possible new approaches, but there is a long way to go.

4 The Developmental Screening Project is described on the 211 LA County website: http://www.211la.org/?page_id=150

18

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

Figure 8: Measurement Poten\al of I&R/A Interven\ons (Individual Level of Impact)

2011 Integer Research & ConsulZng, LLC

19

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

Figure 9 (next page) did not appear in the original report to the AIRS Board, but was included in the Board presentation of June 4, 2011. It brings the thinking of the Report to a next stage of possible development. Like the preceding diagrams, it is intended to represent concepts rather than literal detail. The process block diagram appears in a yellow rectangle at the top of Figure 9. Below it are high-level outcomes for a standard assessment and referral intervention (green rectangle). For each outcome, there is a failure/success symbol in a cascading arrangement. For example, the client attempts to reach an I&R/A specialist and if he or she is successful [O-1], the client receives a standard assessment and referral from the I&R/A spe- cialist. A successful outcome [O-2] takes place when the clients need is accurately assessed and a referral appropriate to the clients need can be made to an existing service provider. A failed outcome might occur when no resource exists to meet the clients need and therefore no referral can be made. After a standard referral is successfully provided, the clients knowledge is enhanced successfully [O-3] if he or she is able to assimilate sufficient information to make contact with an appropriate pro- vider. A failed outcome might occur if the information is not assimilated due to a language barrier, or is not successfully passed on to another person who needs the services. Every salient activity in the end-to-end process has identifiable and measurable outcomes. If there are feedback mechanisms that capture these events, the data about outcome successes and failures may be gathered and analyzed. This is illustrated in the diagram with the dotted vertical lines from the failure/success symbols to the drum symbols ( ), which represent data, and to the summation symbols (), which represent analysis of the data. This would make it possible to gauge where a particu- lar activity needs to be redesigned to increase its effectiveness; where clients need additional help to operationalize the infor- mation they are given; or where other factors, such as available service capacity, require attention. As shown in Figure 9, feedback data and analysis facilitate better process and outcomes management for I&R/A organizations, for service provider organizations, and for the service system as a whole. This approach recognizes that communities, govern- ment, and funders are most interested in the end-to-end process of serving the needs of the public. Gathering and analyzing data on the end-to-end process enables this objective; finding a practicable and valid means of accomplishing this is key. The incorporation of client and community data into the feedback process (shown at the left of Figure 9) would also have a transformative effect on process and outcomes management. The ability to analyze service system performance segmented by particular demographics, service needs, and access channels would make it possible to calibrate the system to best meet the service objectives for specific problems, needs, and populations.

20

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

Figure 9: Process and Outcome Measurement and Management for a Standard Assessment and Referral Interven\on

2011 Integer Research & ConsulZng, LLC

21

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

V. MEMBER OUTCOMES SURVEY
The Project Committee agreed that it was important to determine AIRS member interest in developing I&R/A outcomes concepts and applications. An initial draft was reviewed by the Committee and the finished survey was mounted online from April 27th through May 5th. The following text reviews response rates and the answers to each survey question. Response Rates The survey gathered 117 responses over the course of one week, amounting to a response rate of 9% of the membership (top figure, left pie). Responses came from members in 60 percent of the states, provinces, and overseas locations (top figure, right pie). An analysis of 37 AIRS surveys during 2009 and 2010 showed a median response rate of 99, placing the response rate for the outcomes survey within the third quartile for this pe- riod (bar chart). As another basis of comparison, the 2010 AIRS Board elections gathered the same number of responses (117) as the outcomes survey. SURVEY QUESTION 2:
2. Please describe the service focus of your organiza\on. Check more than one only if appropriate.

The majority of respondents (81%) indicated that they have a single service focus (blue pie slice); 19% indicated that they have multiple service focuses (green pie slice). The largest segments of respondents came from AIRS members providing I&R/A to the aging/ disabled population (40%) and to the general community (37%), with smaller segments serving special populations and multiple populations, and providing crisis intervention services. These reflect a fair cross-section of the membership.

22

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

SURVEY QUESTION 3:
For these audiences, how important is it today to create a framework for dening and measuring I&R outcomes?

The vast majority of respondents answered that it was very im- portant to create an I&R outcomes framework at the present for the purposes of the individual respondent (74%), the I&R organization (75%), and funders (76%). Smaller fractions saw it as very important to establish such a framework for the pur- poses of service providers (57%) and local government (56%). Fractions of 5 percent or less of the respondents felt that creat- ing an outcomes framework was unimportant.

SURVEY QUESTION 4:

For these audiences, how important will it be two years from now to have a framework for dening and measuring I&R outcomes?

The answers were very similar to the previous question. Across the five audiences with a possible stake in I&R outcomes, the fraction of respondents deeming an outcomes framework to bevery important increased over Question 3 from between 5% and 13%.

23

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

SURVEY QUESTION 5:
For these purposes, how important is it to your organiza\on to create a framework for dening and measuring I&R outcomes?

This question sought to determine the importance of establishing an outcomes framework for securing funding, for system trouble- shooting, and for demonstrating the value of I&R to the public. The majority of respondents deemed it very important to have a framework for each of these purposes. Fractions of more than 70% felt the framework was very important to secure funding from state government (78%), federal government (77%), and foundations (74%). Respondents ranging between 55% and 69% deemed it very important for the purposes of getting funding from local government and the United Way, for troubleshooting system barri- ers, and for demonstrating the value of I&R to the public.

24

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

SURVEY QUESTION 6:
How helpful to your organiza\on would it be to for AIRS to develop an Outcomes Framework that includes...

The vast majority of respondents felt it to be very important to develop a framework that includes clear definitions of outcomes (87%), methods for measuring outcomes (90%), and information on best practices and improving outcomes (87%).

SURVEY QUESTION 7:
What is the best way of describing the subject of "outcomes" in the context of Informa\on and Referral? Do you think "outcomes" should stand alone as a dis\nc\ve concept and word, or does it logically fall under the umbrella of I&R "Quality Assurance"?

The majority of respondents (58%) felt that I&R Outcomes was the clearest way of describing the subject in the context of I&R, with sub- stantially smaller fractions voting for I&R Quality Assurance (21%) or being uncertain (18%).

25

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

Narra\ve Responses Respondents were able to provide comments for each question, and their answers were recorded and analyzed. Five themes emerged from a total of 159 narrative responses. The most-frequently voiced of these (#1 at right) is a desire to have a set of common I&R outcomes measures that can be used to com- pare the performance of I&R programs to each other or to a service benchmark. The second group (#2) warrants a follow-up be- cause the respondents indicate they are already using an outcomes protocol of some kind. Valu- able insights or practices may be revealed by inves- tigating these further. The third theme (#3) encompassed five smaller response clusters that expressed support for outcome measures as a way of improv- ing service, demonstrating value, gaining funding, and evaluating effectiveness. Two lesser themes were the desire to incorporate outcome measures with existing I&R software (#4) and support for such measures provided they didnt burden staff that are already stretched to the limit (#5). A Sampling of Responses 1. Quantitative and Qualitative Evaluation is critical...we need to know now more than ever that our efforts are truly making a difference - a documentable difference. We can't afford to spend time and money on things that aren't really getting the information to the people who need it. Thanks for taking up this issue of evaluation. (United Way Agency/2-1-1) 2. It takes our service to the next level - helps us be more accountable and is critical to program evaluation and im- provement... (Public Health Association)

26

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

3. Two years from now the funding available is going to be even more difficult to access. All the more reason to do it today... (Community Action Partnership) 4. Our state is covered by a consortium of providers and we need this framework to drive service uniformity and demonstrate a true system. (United Way Agency/2-1-1) 5. Are we just proposing developing a framework for I&R per se (as an industry tool) or are we proposing a model to be used to determine the effectiveness of I&R AND the services to which we refer? The latter being preferable as it would provide for a more 'world-view' model that funders can use to better serve people/communities/as well as to better communicate the value of our services. (Comprehensive Community I&R/2-1-1) [italics added] 6. We definitely need a framework to measure I&R outcomes. We have been trying to develop a process for years without success. (Hospital Medical Center) 7. I'm constantly on grants to explain the outcomes and I find it very difficult especially when you're not following through to find out if people called the referrals or were helped. (Senior Information & Assistance Agency) 8. I would like to see a few different tools developed for consideration; then possibly allow AIRS members to vote on utilization. I would think it would be important for all to use the same tools to ensure credibility and uniformity for reviewers/funding sources. There should also be sufficient training provided free of charge to all I&R providers re- garding the tool and its function/utilization. Thank you for allowing us to participate in this survey! (County Coun- cil on Aging) 9. We are continuously asked by funders for outcomes other than the activity performed (referrals and calls handled). The interest is in knowing results of the activity, such as the percentage of callers who connected with services or whose lives were improved in definable ways. AIRS standards for outcome-based follow-up questions and adequate sample sizes would lend validity as we propose outcomes to funders, in that we could point to nationally recognized achievement standards. (Comprehensive Community I&R/2-1-1)

27

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

The survey comments express strong support for developing a standardized I&R/A outcomes framework. They also raise some trenchant questions. For example, comment #5 asks about the goal of an I&R outcomes framework: is it to improve the delivery of information within the scope of I&R/A, or is it, more broadly, to enable system-wide improvement capability for the larger human service system? This echoes a question raised by University of Nebraska Professor and AIRS member Nancy C. Shank in her paper Examining the Potential Benefits of a 2-1-1 System: Quantitative and Other Factors: Additional exploration of the role that 2 -1-1s play in linking callers to appropriate services needs to be undertaken. Who determines what kind of service is appropriate and efficacious and, therefore, should be included in a 2-1-1 database? Should 2 -1-1s be in the business of measuring outcomes, such as clients achieving self-sufficiency? If so, what is their role in evaluating programs? (Shank & Rosenbaum, 2003, p. 21; italics added ). Comment #8 suggests a development process that involves the AIRS membership and which includes a vote on outcomes measure- ment alternatives in the process of deciding a national standard. This kind of enthusiasm would be instrumental in moving this pro- ject ahead successfully. This comment, and others like it, also remind us of the variety of I&R/A organizations in the AIRS membership. Suppose for a mo- ment that an inclusive development process produced a core set of I&R/A outcomes practices and standards that was accepted by the majority of AIRS members; and further suppose that well-designed variations could be built into this model to accommodate the wide range of the members institutional capacities, technologies, and funder demands. Perhaps developing basic, intermedi- ate, and advanced outcomes measurement models would go a long way to securing the support of the membership for this im- portant work.

28

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

VI. SUMMARY
The Outcomes Project started in December of 2010 and ran for six months. With the participation of the Project Committee, five teleconferences were held, ideas were exchanged, and this first-round report was created. The original intention of the project was to produce a finished funding prospectus for I&R/A outcomes research. However, it was apparent within the first month of work that the project would have to focus at a foundational level to define the possible directions for I&R/A outcomes development work. Here are the essential findings: The conceptual foundaaons for I&R/A outcomes exist in robust, if incomplete, form as the result of work by Levinson, Saxton and colleagues, Neken and colleagues, and other contributors. Much remains to be done to complete these foundaaons, but the work done to date is conceptually sound and can be built upon with condence. Our project adds to these foundaaons. In paracular, an end-to-end process framework such as that sketched by our study (and illustrated in Figures 7, 8, and 9) could serve as an excellent way for dening, measuring, and analyzing I&R/A process outcomes. It seems highly likely that an end-to-end, client-centered perspecave on services will become increasingly important in the human services eld. Measuring outcomes appears to be an area in need of greatest akenaon. As noted by the report, some of the intervenaons prac- aced today in I&R/A operaaons and research suggest possible development pathways for outcomes measurement approaches. In paracular, a navigator-like funcaon seems to oer potenaal for data gathering and for performing process diagnosacs. Although it is not clear at present how such a funcaon could pracacably be incorporated into rouane I&R/A operaaons, this quesaon could be taken as a catalyst, not a bar, to innovaaon. AIRS survey respondents voiced wholehearted support for the creaaon of I&R/A outcomes deniaons, measurements, and meth- ods. Some responses araculated cauaonary notes about the complexity and overhead associated with incorporaang an outcomes framework into everyday I&R/A pracace. Work on this Outcomes Project may point to an interesang way of addressing both the promise and the pracacal concerns voiced about Outcomes implementaaon. This might involve creaang a unied and standardized I&R/A outcomes foundaaon that could support several operaaonal models to meet dierent member needs and capaciaes.

29

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

VII. M OVING A HEAD
Key Concerns As the project unfolded, questions naturally emerged about whether the work should proceed and, if so, how. Here are several of the key concerns that were raised. Feasibility from a Member Perspec\ve: The Project Team compiled and generated ideas of potenaal value to an I&R/A Outcomes Framework. Given the memberships general support of an I&R/A outcomes framework (as reected in survey responses), how would it appraise the Teams specic ideas? Are these ideas worthy of further development? Would focus groups be a logical next step in answering this question? Member Commitment: How commi[ed are AIRS members to developing an I&R/A Outcomes Framework? The online survey shows that there is strong interest in an Outcomes iniaaave, and that members have good ideas to contribute. The project strat- egy presented in the next few pages makes it clear that a commitment of member ame, experase, and resources is needed if the project is to conanue. The Role of AIRS: It is clear that the scope of this project is signicant in terms of its impact on I&R/A, its need for paracipaaon, resources and funding, and the ame required for development and implementaaon. What role can AIRS take in this process? As the leadership, training, and credenaaling body for I&R/A professionals, AIRS holds a preeminent place in any Outcomes Project for I&R/A. Should AIRS be the development engine to drive this process? Project Strategy: As noted above, an Outcomes Project is signicant in terms of resource, funding, and ame requirements. What strategy would best be suited for such a project? In some ways, this is the most readily answered of these important questions. A variation of the classic systems analysis and design model would provide an excellent means of conceptualizing and managing an Outcomes Project. The following two pages present an outline of a project strategy which could be refined and expanded to meet the needs of an Outcomes Project.

30

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

Project Strategy

Status Indicators: = complete;

= started but not complete; = not started

Phase 0: Idea\on Develop preliminary ideas for the Outcomes Framework. Gauge member interest in the Outcomes Framework. Review ndings and make recommendaZons to the AIRS Board. 1 2 3 4 5 Project Team generates iniZal Outcomes Framework ideas Project Team gauges member interest in Outcomes with online survey AIRS sta provides Outcomes Project Report to Board (5/25/11) Integer presents Outcomes Project Report to Board (6/4/11) Make Outcomes Project Report available to AIRS Members (July, 2011)

Phase 1: Feasibility The Feasibility Phase establishes whether developing and implemenZng an Outcomes Framework is feasible from the perspecZves of the AIRS membership and external consZtuents. If the rened Outcomes Framework is determined to be feasible and funding is available, the project proceeds to the Design Phase. Representa\ve Ac\vi\es: AIRS Board approves Feasibility Phase proposal, funding, and project plan; appoints Project Team Project Team denes key internal and external consZtuent segments Project Team conducts Focus Groups and Outreach to internal consZtuents; revises Outcomes Framework based on input Project Team conducts Focus Groups and Outreach to external consZtuents; revises Outcomes Framework based on input Project Team canvasses potenZal funders, human service organizaZons, and other parZcipants, and compiles salient informaZon Project Team presents ndings and recommendaZons to AIRS Board AIRS Board approves Design Phase proposal, funding, and project plan; adds resources as needed to Project Team

31

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

Status Indicators: = complete;

= started but not complete; = not started

Phase 2: Design The Design Phase transforms the ndings from previous phases into one or more designs that incorporate key deniZons, metrics, and processes. The designs are reviewed with internal and external consZtuencies, revised as necessary, and presented to the AIRS Board. If the AIRS Board approves the designs and funding is available, the project proceeds to the Development Phase. Phase 3: Development The Development Phase produces working prototypes from the design(s) of the previous phase. Each prototype is reviewed with mulZple representaZves of each potenZal user audience. The user audience includes I&R/A organizaZons and may include stakeholders, such as funders, that are willing to parZcipate. A detailed walkthrough of each prototype is conducted, and revisions are made on the basis of user feedback. I&R/A eld test agencies are selected from each audience and prepared for tesZng. For each test site, representaZves from local funding organizaZons, service providers, and other stakeholders are idenZ- ed, contacted, and engaged in reviewing the prototypes. If the AIRS Board approves the prototypes and funding is available, the project proceeds to the Field TesZng Phase. Phase 4: Field Tes\ng The Field TesZng Phase engages I&R/A agencies in tesZng the Outcomes Framework and Methods suitable for a parZcular audience. A detailed account of each eld test is created and used to guide correcZon and renement of deniZons, metrics, and processes. Stakeholders provide detailed feedback on fac- tors that will enable or prevent successful use of the new Framework. A cost/benet model is developed to jusZfy the investment in the new methods. Based on the Field Test results, the Project Team develops an implementaZon strategy that includes phasing by audience or other member segment, training requirements, resource requirements, and other factors. If the AIRS Board approves the results of eld tesZng and the implementaZon strategy, and if funding is available, the project proceeds to the ImplementaZon Phase. Phase 5: Implementa\on The AIRS Outcomes Framework is implemented in the phases specied by the implementaZon strategy. Ongoing monitoring and quality assurance are under- taken to maximize the ease and eecZveness of the transiZon.

32

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

APPENDICES

33

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

Appendix 1: References5 Flowers, A. (2009, June 23). I&R Innovations: Discovering the foundations of customer service. Presentation for National Aging I&R Support Center Lecture Series, National Association of States United for Aging and Disabilities, Washington, D.C. Retrieved from http://www.nasuad.org/documentation/I_R/PP_IRSupportCenter_LectureSeries.pdf Harris, E. (2008, May 21). ABCs of satisfaction & impact surveys. Presentation at the Alliance of Information & Referral Systems 30th Annual Training and Education Conference, Houston, TX. Retrieved from www.airs.org/files/public/Conference08_Management_SatisfactionImpactSurveys.ppt Lampkin, L. M. & Hatry, H. P. (2003). Key steps in outcome management. Series on Outcome Management for Nonprofit Organiza- tions. Washington, DC: The Urban Institute. Retrieved from http://www.urban.org/publications/310776.html Latzer, A. (2008, May 21). 211 Centers: Doing it, the how of metrics; what to collect and how to use it! Presentation at the Alliance of Information & Referral Systems 30th Annual Training and Education Conference, Houston, TX. Retrieved from www.airs.org/files/public/Conference08_211_HowOfMetrics.ppt Latzer, A. (2010, May 24). Quality, training, and coaching Best practices for I&R contact centers. Presentation at the Alliance of In- formation & Referral Systems 32nd Annual Training and Education Conference, Rochester, NY. Retrieved from www.airs.org/files/public/Conference2010/ServiceDelivery_QualityTrainingCoaching.pptx Levinson, R. W. (2002). New routes to human services: Information and referral. New York, NY: Springer.
[Springer Publishing Company offers a 20% discount to AIRS members for purchase of Levinson's book. It is available at the Springer Publishing Company website. AIRS members should use Promo Code AIFS. Shipping within the United States is free when ordering through the Springer website. Used copies may be available at amazon.com, Powells Books, or other booksellers.]

Marin, M. (2007, May 7). Metrics for 2-1-1 centers and systems: A policy dialogue. presentation at the Alliance of Information & Referral Systems 29th Annual Training and Education Conference, Jacksonville, FL. Retrieved from www.airs.org/files/public/Metrics_for_211_Centers_and_Systems.ppt Michigan 211, Inc. (2011). Michigan 211 business plan. Lansing, MI: Author.

5 Where possible, links to articles, books, and presentations have been provided. The links were operative as of July 15, 2011, but may subsequently change.

34

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

Miloff, M. & Associates (2007). Tapping the 211 Information Goldmine. [Report prepared for 211 Ontario]. Toronto, Ontario, Can- ada. Municipal Benefits Committee, Ontario 211 Team, United Ways of Ontario (2007). Who benefits from Ontario 211? [Report pre- pared for Association of Municipalities of Ontario]. Toronto, Ontario, Canada. Nayyar-Stone, R., & Hatry, H. P. (2003). Finding out what happens to former clients. Series on Outcome Management for Nonprofit Organizations. Washington, DC: The Urban Institute. Retrieved from http://www.urban.org/publications/310815.html Netten, A., & Forder, J. (2008). Measuring the outputs of information and advice services: Initial report. (PSSRU Discussion Paper 2543/2) Retrieved from Personal Social Services Research Unit, University of Kent, Canterbury website:
http://www.pssru.ac.uk/pdf/dp2543_2.pdf

Neuffer, M. H. (2011). AIRS research atlas. Retrieved from http://integerconsult.com/airs/ Reisz, I., & Goodman, G. (2010). Why elders do not follow up on suggestions they requested: An exploratory study by Care for Elders funded by the City of Houston, Department of Health and Human Services [Report for City of Houston, Department of Health and Human Services]. Saxton, M. L., Naumer, C. M., & Fisher, K. E. (2007). 2-1-1 Information services: Outcomes assessment, benefit-cost analysis and policy issues. Government Information Quarterly, 24, 186-215. Purchased at:
http://www.sciencedirect.com/science/article/pii/S0740624X06000463

Shank, N. C., & Rosenbaum, D. I. (2003). Examining the potential benefits of a 2-1-1 system: Quantitative and other factors. Infor- mation and Referral, 25, 1-26. Retrieved from: http://digitalcommons.unl.edu/publicpolicyshank/2/ Sheppard, V. & Christopher, J. (2001). Measuring outcomes in the delivery of information and referral services: Maternal and child health example. Information and Referral, 23, 189-235. Retrieved from AIRS archives. Sim, S. C., & Rocha, K.A. (1998). A qualitative and quantitative approach in evaluating a telephone based information and referral program. Information and Referral, 20, 1-14. Retrieved from AIRS archives.

35

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

Thompson, M., & Luther, J. (2009, June 1). Quality assurance, customer satisfaction and follow-up: Definition and practical experi- ence. Presentation at the Alliance of Information & Referral Systems 31st Annual Training and Education Conference, Reno, NV. Retrieved from AIRS archives. United Way of America. (2000). Agency experiences with outcome measurement: Survey findings. Retrieved from http://www.unitedwayslo.org/ComImpacFund/10/Experiences_Outcome_Measure.pdf U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, Office of the Director, Office of Strat- egy and Innovation. (2005). Introduction to program evaluation for public health programs: A self-study guide. Retrieved from
http://www.cdc.gov/eval/evalguide.pdf

Windle, K., Netten, A., Caiels, J. Masrani, R., Welch, E., & Forder, J. (2010). Measuring the outcomes of information and advice services: Final report. (PSSRU Discussion Paper 2713). Retrieved from Personal Social Services Research Unit, University of Kent, Can- terbury website: http://www.pssru.ac.uk/pdf/dp2713.pdf Wood, C., & Fay, P. (2010, May 24). Quality not quantity: A focus on the resource database. Presentation at the Alliance of Informa- tion & Referral Systems 32nd Annual Training and Education Conference, Rochester, NY. Retrieved from AIRS archives. Woods, D., & Eyre, J. (2003). 211 for all Ontario: Bringing people and services together. Final Report July 2003. Part 2: Torontos 211 experience. [Report prepared for InformOntario, United Ways of Ontario, Community Information Toronto, and United Way of Greater Toronto]. Toronto, Ontario, Canada.

36

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

Appendix 2: Logic Models
DETAILED LOGIC MODELS BASED ON SAXTON-NETTEN FRAMEWORK

Because of its size, the model has been divided into the three levels of outcomes as dened by Saxton and Neken. This poraon of the model shows shows the individual client seeking and receiving help. Posiave and negaave outcomes are supplied for each stage; the detail of process steps has been omiked. (A}er compleang this model, the Outcomes Commikee decided that working at a higher level of abstracaon would be more helpful, and that this degree of detail would have to be elaborated at a later ame.)

37

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

DETAILED LOGIC MODELS BASED ON SAXTON-NETTEN FRAMEWORK

The second poraon of the model deals with the organizaaonal level of outcomes. To conserve space, only the posiave outcomes have been shown in this diagram.

38

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

DETAILED LOGIC MODELS BASED ON SAXTON-NETTEN FRAMEWORK

The third poraon of the model deals with the societal level of outcomes very briey. To conserve space, only the posiave outcomes have been shown in this diagram.

39

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

Appendix 3: 211 LA Developmental Screening Project Fact Sheet

40

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

Appendix 4: AIRS Outcomes Survey
1. PLEASE IDENTIFY YOURSELF AND YOUR ORGANIZATION 2. PLEASE DESCRIBE THE SERVICE FOCUS OF YOUR ORGANIZATION. CHECK MORE THAN ONE ONLY IF APPROPRIATE.
I&R and I&R?A for Aging and/or Disabled PopulaZons I&R for Other Special PopulaZons (e.g., Youth, Families) Crisis IntervenZon Services Other (specify) I&R for General Community

3. FOR THESE AUDIENCES, HOW IMPORTANT IS IT TODAY TO CREATE A FRAMEWORK FOR DEFINING AND MEASURING I&R OUTCOMES?
(a) You as an I&R or I&R/A Professional (d) Your Funders (b) Your OrganizaZon (e) Your Local Government (c) Human Service Providers in Your Community

Scale: No Opinion Not Important Somewhat Important Very Important

4. FOR THESE AUDIENCES, HOW IMPORTANT WILL IT BE TWO YEARS FROM NOW TO HAVE A FRAMEWORK FOR DEFINING AND MEASURING I&R OUTCOMES?
(a) You as an I&R or I&R/A Professional (d) Your Funders (b) Your OrganizaZon (e) Your Local Government (c) Human Service Providers in Your Community

Scale: No Opinion Not Important Somewhat Important Very Important

5. FOR THESE PURPOSES, HOW IMPORTANT IS IT TO YOUR ORGANIZATION TO CREATE A FRAMEWORK FOR DEFINING AND MEASURING I&R OUTCOMES?
(a) Gezng funding from local government (city, county) (b) Gezng funding from state government (d) Gezng funding from United Way (e) Gezng funding from other private sources (e.g., foundaZons) (g) DemonstraZng the value of I&R to the general public (h) Other (specify) Scale: No Opinion Not Important Somewhat Important Very Important (c) Gezng funding from federal government (f) TroubleshooZng service system barriers and problems

6. HOW HELPFUL TO YOUR ORGANIZATION WOULD IT BE TO FOR AIRS TO DEVELOP AN OUTCOMES FRAMEWORK THAT INCLUDES:
(a) Clear deniZons of what i&R outcomes are (b) Guidelines and methods for measuring I&R outcomes (c) InformaZon on best pracZces, methods of improving outcomes, etc. Scale: No Opinion Not Important Somewhat Important Very Important

7. WHAT IS THE BEST WAY OF DESCRIBING THE SUBJECT OF "OUTCOMES" IN THE CONTEXT OF INFORMATION AND REFERRAL? DO YOU THINK "OUTCOMES" SHOULD STAND ALONE AS A DISTINCTIVE CONCEPT AND WORD, OR DOES IT LOGICALLY FALL UNDER THE UMBRELLA OF I&R "QUALITY ASSURANCE"?
Scale: No Opinion Not Certain I&R Outcomes I&R Quality Assurance

8. WHAT KINDS OF OUTCOMES MEASUREMENT TOOLS WOULD BE USEFUL TO YOUR I&R ORGANIZATION?

41

AIRS INFORMATION & REFERRAL OUTCOMES REPORT

42