Benefits Evaluation Survey Process

System & Use Assessment Survey This document is designed to detail the process of adapting and administering the System & Use Assessment Survey (S&U Survey). For further information please contact the Infoway Clinical Adoption team at clinicaladoption@infoway-inforoute.ca. Evaluation Objectives The customizable System & Use Assessment Survey is intended to be administered soon after a project has gone live. Its purposes are to: 1. 2. 3. 4. 5. Provide benefits statements to drive adoption in later implementations Assist in identifying barriers to adoption so that remedial action may be initiated Identify additional functionality which could be provided in future releases Provide analysis of the viability of communication and training strategies Provide stakeholders with the assurance that their adoption of the solution is important.

Background The S&U Survey was created with the intention of being used to assess quality and use components of health IT systems, and to flag obstacles to adoption and the realization of net benefits from the system. The questions that make up the survey were developed by evaluation Subject Matter Experts and Infoways Benefit Evaluation team. To date, a number of projects such as those listed in the following domain areas have successfully customized the System and Use Survey: drug information systems, diagnostic imaging, EMR pharmacy integration, clinical information systems, and clinical outcomes data projects. A copy of the original survey template can be found in Appendix A. Survey Process The S&U Survey process is divided into four phases. The activities that make up each the phase are outlined below in Table 1. More detailed descriptions of select activities that require additional explanation are discussed following the table below.

Table 1: Survey Activities Activity Plan and Adopt Survey

Identify opportunity for survey use and assess appropriate timing of administering survey Survey template shared with Project team and questions are customized to suit project (please see appendix B for survey template). The inclusion

Involvement
Project Project Evaluation Lead Infoway Benefits Evaluation (BE) Project Project Evaluation Lead Infoway Benefits Evaluation

Timeframe
Ideally during planning stages of project

Ideally prior to implementation stages of project

of Infoways Core Questions is discussed with Project team (Appendix D). Process agreed upon for distribution for survey i.e. survey instructions, timelines, survey leads etc. Project approves final version of survey Electronic, web-based version of survey is created 2. Data Collection Project compiles a list of potential survey participants and decide on method to distribute survey (both electronic and paper versions) Link to electronic S&U Survey enabled

Project Project Evaluation Lead Infoway Benefits Evaluation Project Project Evaluation Lead Infoway Benefits Evaluation Infoway Benefits Evaluation Information Management and Technology (IMT) Project Project Evaluation Lead Infoway Benefits Evaluation Project Project Evaluation Lead Infoway Benefits Evaluation Information Management and Technology (IMT) Infoway Information Management and Technology (IMT) Project Project Evaluation Lead Project Project Evaluation Lead Infoway Benefits Evaluation Project Project Evaluation Lead Infoway Benefits Evaluation Project Project Evaluation Lead Project Project Evaluation Lead

See above

See above

See above

Survey software automatically collects participants responses Reminder email #1 sent out to non-responders Discussion of need to administer paper-based survey to non-responders Infoway provides Project team with copy of paper-based survey and Project team prepares to distribute it Reminder email #2 sent out to non-responders Paper-based survey is distributed by Project team

1-2 weeks after initial email

2 weeks after initial email

2 weeks after initial email

3 weeks after initial email 3 weeks after initial email

3. Analysis and Report Project evaluation lead provided with raw survey data results throughout survey and at completion (see Appendix C for sample reports)

Project may choose to conduct additional analysis of raw data

Project Project Evaluation Lead Infoway Benefits Evaluation Information Management and Technology (IMT) Project Project Evaluation Lead Evaluator AND/OR Third party organization (responsibility of Project)

Frequency to be determined with project

Once collection of responses is complete

4. Data Storage Both Project and Infoway retain data sets with no personal identifiers. Infoway may aggregate the survey data with results from other evaluation projects and use it in pan-Canadian studies

Project Project Evaluation Lead Infoway Benefits Evaluation

Sample Size Prior to administering the survey, it is important to determine the number of potential survey participants. In order to calculate a response rate for the survey it is necessary to track the number of people invited to participate in the survey, and how many people complete the survey. A high response rate is important to legitimize a survey's results. When a survey elicits responses from a large percentage of its target participants, the findings are seen as more accurate. The survey sponsor needs to determine who in their organization will take the survey. Some choices are: the whole company, only permanent and/or full-time employees, or only certain job positions (i.e. physicians, nurses, pharmacists, support staff). Administering paper based version of survey While the S&U Survey is designed to be administered electronically, the electronic method can sometimes result in low response rates. Distributing a paper copy of the survey to participants in addition to the electronic version is possible, but will require additional labour from the project team. Infoway will provide the project evaluation team with a paper version of the electronic survey; however, the project evaluation team will be responsible for distributing the paper version, collecting the completed surveys, and inputting the responses into the electronic version of the survey.

Possible methods of distributing the paper survey include: 1. Sending out survey attached to employees pay stubs 2. Organizing sessions in which groups of employees complete the survey together (either in regular group-meeting times or in larger groups at a special event) 3. Project evaluation team personally handing out surveys to individuals or departments and arranging a convenient drop off location Results Infoways survey software will collect and store the raw data collected from electronic survey participants. The collection and storage of the paper version will be the responsibility of the project evaluation team. Infoway will provide the project evaluation lead with data result reports periodically. Please see Appendix A for examples of the two formats that data is presented in. Privacy and data storage Confidentiality The survey is anonymous. The survey does not ask for participants to provide any personal identification such as name or employee ID number, or any other information that is potentially identifying. To ensure confidentiality compliance, all responses inputted into the electronic survey will be stored on Infoways secure server. The data will not have any personal identifiers. The server administrator will require a password to access the data. The project evaluation team will be responsible for ensuring the paper copies of the survey are kept confidential. Data retention Infoway will maintain the results from the electronic survey in accordance with stringent data protection and management requirements to protect the confidentiality of the data and to prevent unauthorized use or access.

Appendix A: Original Survey Template Dear [insert name], The [project name] and Canada Health Infoway are conducting a benefits evaluation study in order to improve the quality of the information provided by the health information systems, as well as assess the level of satisfaction amongst end-users. Your feedback and assistance with this survey will help [project name] and Infoway to better understand the general benefits of systems and deliver better services. The following online survey consists of specific questions on: the ease and functionality of the health information systems, information quality, and service quality related to the systems implemented at your Centre. The survey will take approximately 10 minutes to complete. Please choose the response that best represents your opinion. Please note the following: Information that is collected during this survey will be kept anonymous and confidential. We will know what clinic a survey response comes from, but will not know which individual member of the clinic completed the survey. Information obtained in this survey will be analyzed in aggregate and shared with Canada Health Infoway. Your participation in this survey is entirely voluntary and you are in no way obligated to complete this survey. Your employment will not be affected by your choice to complete or to not complete this survey. We will not reveal your answers to your employer or fellow employees.

If you do participate, completion of the survey implies your awareness of and consent to the above information. Any questions or concerns should be directed to [contact information]. Thank you in advance for your participation. Sincerely yours, [name of evaluation lead]

SECTION 1. OVERALL USER SATISFACTION

1.

In general, how satisfied are you overall with the system you are currently working with? By system we mean, the ease and functionality of the system itself, the quality of the information given and the quality of the services provided for the system. Highly satisfied Moderately satisfied Neither satisfied nor dissatisfied Moderately dissatisfied Not at all satisfied

2.

Please indicate your level of agreement or disagreement with each of the following statements below. Strongly Agree a.) The system improves my productivity b.) The system improves the quality of care I can provide c.) The system makes my job easier d.) The system enhances our ability to coordinate the continuity of care e.) The system improves our sharing of patient information amongst providers f.) The system enhances the efficiency of ordering lab tests, X-rays, prescriptions, etc. g.) The alerts, reminders and order set features (i.e. support tools) improve the quality of my decision-making Moderately Agree Moderately Disagree Strongly Disagree Not Sure Not Applicable

3.

Are there aspects of the system that you would change, and if so, which ones would they be? Please describe your comments.

4.

Do you have any experiences with the system where it has supported the provision of care? Please describe your comments.

SECTION 2. SYSTEM QUALITY

5.

Based on your experiences to date with the system, how acceptable is the quality of the system itself (as described by the specific characteristics listed below)? Would you say it is; Highly acceptable Moderately acceptable Neither acceptable nor unacceptable Moderately unacceptable Not at all acceptable

6.

Please indicate your level of agreement or disagreement with each of the following statements below. Strongly Agree Moderately Agree Moderately Disagree Strongly Disagree Not Sure

a.) The system is easy to use b.) The response time is acceptable c.) The system is integrated with my workflow d.) The system security is acceptable e.) The system features enable me to perform my work well f.) The system is reliable in its performance g.) Overall, the quality of the system is excellent

SECTION 3. INFORMATION QUALITY

7.

In general, when thinking about the quality of the information provided by the system, do you find the quality of the information to be; Highly acceptable Moderately acceptable Neither acceptable nor unacceptable Moderately unacceptable Not at all acceptable

8.

Please indicate your level of agreement or disagreement with each of the following statements below. Strongly Moderately Moderately Strongly Agree Agree Disagree Disagree The information is complete The information is quickly provided The information is accurate

Not Sure

a.) b.) c.)

d.) The information is relevant e.) f.) The information is available when I need it The format and layout of the information is acceptable

SECTION 4. SERVICE QUALITY

9. In general, when thinking about the quality of the services (i.e. technical support and training services) provided for the system, do you find the quality of these services to be; Highly acceptable Moderately acceptable Neither acceptable nor unacceptable Moderately unacceptable Not at all acceptable

10.

Please indicate your level of agreement or disagreement with each of the following statements below. Strongly Agree Moderately Agree Moderately Disagree Strongly Disagree Not Sure

a.) The implementation process at this Hospital or Centre was acceptable b.) The current level of training is acceptable c.) The level of on-going support provided is acceptable

SECTION 5.

PUBLIC HEALTH SURVEILLANCE SPECIFIC

-TO BE COMPLETED BY PUBLIC HEALTH SURVEILLANCE PERSONNEL ONLY -

11. Please indicate your level of agreement or disagreement for each of the following statements below. Strongly Agree a.) The system improves the detection and management of reportable diseases b.) The system improves the management of immunization process Moderately Agree Moderately Disagree Strongly Disagree Not Sure

SECTION 6. SYSTEM USAGE

12.

In a typical day, how many times do you use the system?

_____________________ Number

of times, a day

Always . 13.

Rarely .

In a typical week, please indicate the number of days in which you use the system.
_____________________ Number

of days, a week

14.

Please estimate what percent of your patients do you use the system? ______________% patients (FILL IN) Dont know .

15.

How likely are you to recommend the system to other healthcare providers at other Hospitals or Centres?

Definitely

Probably

May or may not

Probably Not

Definitely not

16. Given a choice, would you like to increase or decrease your future use of the system that you are currently working with? Would that be a significant or moderate increase / decrease, or would you like your future use to stay the same?

Significant Increase

Moderately Increase

Moderately Decrease

Significant Decrease

REMAIN THE SAME

SECTION 7. OTHER COMMENTS

17. Do you have any other comments you would like to make regarding the system?

SECTION 8. DEMOGRAPHIC INFORMATION

18. What is your profession? Administrative support staff .............................. Imaging technologist .......................................... Laboratory technician .......................................... Nurse ...................................................................... Pharmacist Family physician ................................................... Specialist physician (please specify below) .......

Other (please specify below) ...................................................

19. How would you describe your use of the system? (Ch eck all that apply) I use the system for clinical decision making .................................................................. I use the system to access patient information and support the clinical decision maker ...................................................... 20. How long have you been using the system? Less than a month 1-3 months 4-6 months 7-12 months 1-2 years 3-5 years I use the system to both access patient information and in clinical decision making ....................................................................

21. Currently, how do you receive your patient results? ______ % FAX write below) ________ % SYSTEM ________ % OTHER ( please specify /

22. How would you rate your computer proficiency? None Basic Average Advanced Expert

23. Please check the response(s) that best describe the settings where you work. Academic / Teaching Hospital ......................... Community Clinic / Health Center .................. Community Hospital .......................................... Nursing Home / Long Term Care Facility ....... Private Office / Clinic ......................................... Other (please specify/write answer below) ..... 24. Where are you located? Alberta ................................................................. British Columbia ................................................ Manitoba................................................................ New Brunswick ................................................... Newfoundland .................................................... Northwest Territories .......................................... Nova Scotia ........................................................... Nunavut ................................................................. Ontario ................................................................. Prince Edward Island ......................................... Quebec.................................................................... Saskatchewan ........................................................ Yukon ..................................................................... a. Do you work within the emergency department?

Yes ................................................... No ...................................................