Running head: INSTRUCTOR’S GUIDE FOR DECISION MAKING

Instructor’s Guide for How to Use a Data Evaluation Job Aid for Decision-making:

Design Project Report

A Framework for Decision-making

IST626 Instructional Design

Dr. Jeanne Farrington
Nuong Nguyen, Tawn Gillihan, Jeff Kitts, and Corey Wallace
July 23, 2018

Instructor’s Guide for How to Use a Data Evaluation Job Aid for Decision-making: Design

Project Report
INSTRUCTOR’S GUIDE FOR DECISION MAKING

The Design Project Report is the opportunity to report on Returning Team One’s

experiences in working with a client to create a real-world design project. For this project, our

client requested a job aid to provide learners with a stand-alone resource for evaluating data for

effective decision-making. Our assignment was two-fold; create a project to meet the client’s

needs, and gain practical experience for the full-life cycle of project management. While it was

important for our team to create a job aid and instructor’s guide for our client to use in their

consulting practice, the heart of this project was to work through the process of determining the

project scope, creating the deliverables, and establishing strong communication practices.

We learned a great deal from the assignment and had a positive experience with the

client. Dr. Thomas is not just a client in search of a training resource; she is also an expert in

project management and instructional design. As we moved through the steps of project

management and design, she provided guidance, feedback, and recommendations to improve our

effectiveness. The takeaway from this experience for many of us is that we should challenge the

project description when appropriate. Dr. Thomas stated more than once that our role as

designers is to assess the client’s request and recommended improvements. We learned a great

deal from this experience and are both excited for future opportunities to work with clients. We

now have a better understanding of the challenges of project management. Overall, this design

project has been an essential step in our professional and academic development.

Client and Organizational Goals

The client is The Fleming Group, LLC, an executive consulting firm that applies

evidenced-based practices to produce measurable outcomes of value for commercial, non-profit,

and government markets. Dr. Mary Thomas, Chief Executive Officer of The Fleming Group, is

the co-author of the data evaluation framework that forms the basis of the project. The client’s
INSTRUCTOR’S GUIDE FOR DECISION MAKING

goal is to expose entry-level supervisors and managers to effective decision-making practices

early in their career development.

Effective decision-making practices, as defined in Guerra-Lopez & Thomas (2011), are

based on data that is evaluated to be relevant, valid, reliable, and complete. By instructing

leaders in sound data evaluation methods, The Fleming Group adds value to the clients and

society as the clients make important decisions. The project fits the Fleming Group’s mission of

performance improvement through evidence-based practices.

Learner Analysis

The target audience for this instruction is entry-level supervisors and managers who

would benefit from professional development in sound decision-making practices. The learners

will be clients of The Fleming Group, LLC in the public and private sectors.

Many professionals working in leadership positions lack understanding of how data are

used in decision-making and may be reluctant to make decisions that may have significant

consequences. Likewise, some professionals make hasty decisions based on ‘gut instinct’ that

also put the organization at risk. These target learners are often unfamiliar with the concepts of

data relevance, reliability, validity, and completeness, and lack a data evaluation framework. This

lack of understanding hinders their ability to distinguish between information that is relevant

versus extraneous data. Therefore, the learners will be highly motivated to learn this data

evaluation framework to add value to their workplace. The learners will take away concepts,

definitions, resources, and awareness of valid and relevant information in the decision-making

process, and its impact on organizational goals.

Project Description
INSTRUCTOR’S GUIDE FOR DECISION MAKING

The purpose of this project is to provide an engaging instructional packet that familiarizes

the learner with a data evaluation method, outlined in Dr. Ingrid Guerra-López and Dr. Mary

Norris Thomas’s article “Making Sound Decisions: A Framework for Judging the Worth of Your

Data” (2011). Training will be offered as a ninety minute, face-to-face, instructor-led session

using materials developed by the design team. The materials include an instructor’s guide and a

job-aid for stand-alone the application of the data evaluation framework.

The deliverables for this project are distribution-ready documents formatted in Microsoft

Word, and designed to fit a variety of public and private sector’s needs. The scope of this project

is to design a concise instructional guide and job aid to enable an instructor to work with learners

of various backgrounds, positions, and levels of readiness. Learners are intended to become

familiar with the data evaluation framework written by Guerra-Lopez and Thomas and the job

aid designed by Returning Team One. Using the framework will result in improved abilities to

determine the quality of available data and whether the data are sufficient to support sound

decision-making.

Design Decisions

The design process involved multiple client meetings with reviews of sample

approximations. The team’s view of the product design significantly changed as we learned

about the subject matter, client expectations, and as we also benefited from the client’s

instructional design expertise. Below are examples of how our design process changed the

delivered product.

The original design included an instructional guide, job aid, and a learner workbook for

practice activities. The team envisioned the learner working through practice problems from the

workbook as part of the instructional session. Dr. Thomas, the client, helped the team realize the
INSTRUCTOR’S GUIDE FOR DECISION MAKING

job aid should be the main focus of the training. Using the job aid during instruction will help the

learner use the job aid in subsequent real-world activities. The team considered including basic

lessons on statistics as part of the instruction for data evaluation. Both Dr. Thomas and Dr.

Farrington reinforced the view that lessons on statistics are out of scope.

The team also struggled with trying to fit the instruction within a one-hour timeframe.

Given how much content is covered in this training, the team and Dr. Thomas agreed that sixty

minutes would not do the content justice. Therefore, the time was extended to ninety minutes,

which gives the learners time to familiarize themselves with the content.

To ensure the students have adequate practice, every module in the lesson has the learners

using the job aid. The repetitive use of the job aid reinforces the terminal learning objective for

real-world data evaluation for sound decision-making. This also allows the instructor to assess

the learners’ knowledge.

Early in the project, there was a question of whether using the data evaluation job aid

should result in one of a set of specific outcomes. Currently, the data evaluation job aid has only

the following outcome, “Derive useful information from the data and proceed with the decision.”

Other outcomes are possible, such as “Not sufficient data, but proceed with a qualified or

experimental decision, while recognizing the risks.” The proposed change in the number of

outcomes did not fit within the scope of this project, but might be a future direction for the job

aid.

The design of the job aid is centered on the principles of data evaluation for sound

decision making as stated in the paper “Making Sound Decisions: A Framework for Judging the

worth of Your Data.” Designing the job aid involved addressing concerns about readability, flow,

and proximity of text and diagrams.

INSTRUCTOR’S GUIDE FOR DECISION MAKING

Design of the Instructor’s Guide

The design choices included in the Instructor’s Guide are based on a sample guide in

previously used by The Fleming Group. The sample offered the team a chance to assess the look

and function preferred by the client so that early design decisions could be aligned with known

preferences. In laying out the guide, the designers strove for a clean document with a simple

layout that would be easily understood by an instructor unfamiliar with the topic. The skill level

and experience of potential instructors is unknown, and so the team chose to include background

information on the use of job aids and familiarization training.

The guide includes recommendations for supplies, safety and accessibility guidelines,

group discussion and instructor feedback techniques, and instructional methods for

familiarization training, frequently asked questions and answers, sample scenarios, discussion

topics for introducing each step, and definitions of terms. The team’s goal in laying out the

instructor’s guide in this format was to provide a stand-alone tool for instructors who are not

familiar with the decision-making framework and who may not be professional trainers.

As a publication, the guide is not an exact match with The Fleming Group’s current

materials, but it does provide a similar style in the selection of margin, fonts, colors, table

elements, and headings. We formatted Headings and lists using Microsoft Word’s Styles feature

to provide consistency and to support an automated Table of Contents. These formatting choices

are not simply professional elements that a client should expect from a design team, the Styles-

based headings and document elements support accessibility for users who rely on assistive

technology.

Branding in the instructor’s guide is limited to use of the client’s logo, herein used by

permission of Dr. Thomas. The final version of the document has been provided to the client in
INSTRUCTOR’S GUIDE FOR DECISION MAKING

an editable Microsoft Word format as this was an essential element of the design request. The

design team understands the final version of the Instructor’s Guide may be revised by the client

for professional use.

Evaluation

Given the time constraints of the course and with the client, evaluating the training

materials with test subjects was informally performed. A graphic designer and engineer were able

to follow the decision flowchart and understand the examples of the terms relevance, validity,

reliability, and completeness. Based on everyone’s feedback, the team revised the time, the order

of the instruction, and clarify the last two sections “Demonstration and Practice” and the “Wrap

Up.”

Assessing the learner’s transfer of the declarative and conceptual knowledge for

evaluating data for sound decision-making is integrated into each lesson format. The instruction

is closely guided with one instructor and one or few learners. This type of training is considered

to be familiarization training. The consulting situation, the scope of the content covered, and

closely guided instruction allows for personalized learning and immediate corrective feedback.

The desired outcome is for the learners to use the job aid in their workplace. If the learner’s data

evaluation skills are not sufficient for a subsequent job-related needs, follow-on consulting for

specific data evaluation situations is appropriate.

Teamwork

Early in the project our team determined roles and responsibilities and shared these with

Dr. Thomas. Our original team assignments were as follows:

Tawn Gillihan - Meeting facilitator, copy editor, accessibility reviewer, design and

creative editor.
INSTRUCTOR’S GUIDE FOR DECISION MAKING

Nuong Nguyen - Documentarian, team facilitator, and scheduler

Jeff Kitts - Product developer

Corey Wallace - Product developer

Performance of these roles generally aligned with the original assignment. When there

was variation from the assigned role it was due to the positive teamwork and our long

collaboration in the MIST program. We have capitalized on our individual strengths throughout

our time in MIST, but we have also alternated roles to ensure we each develop the skills we need

to succeed after graduation.

For this project, Tawn acted as Project Manager to establish scope, reformatted the

Instructional Guide, facilitated client communication during meetings, and created the final draft

of documents to ensure the team spoke with a single voice. Tawn’s skill in creating accessible

documents in Microsoft Word was useful in creating a polished final document.

Nuong acted as Documentarian to lead the team in creating course deliverables,

coordinating each team member’s assigned tasks, documenting meetings, tracking action items,

keeping the project moving forward, and meeting deadlines. Nuong is critical in ensuring project

deliverables happen on time and that all members were participating. She also helped to revise

and edit the final document and made sure to consider the client’s feedback in all the

assignments.

Jeff acted as a Project Developer and provided technical and research leadership for

Returning Team One. He analyzed the research document and created the job aid in Adobe

Illustrator. The job aid is the stand-alone final product; interpreting the original research to

synthesize the job aid was the essential team assignment.

INSTRUCTOR’S GUIDE FOR DECISION MAKING

Corey acted as another Product Developer and as “Client Whisperer” in meetings.

Quickly recognizing the client’s intent and desired outcome is critical for the success of any

design team. Corey is the first member of the team to grasp project requirements and

communicates them to the team. In meetings with Dr. Thomas he amplified communication by

checking for understanding and reframing the ideas and requests. This improved our team

performance and client satisfaction.

The client shared preferences with the team for the function of the document with the

understanding that the IST 626 assignment was intended as an instructional design experience in

the context of working with clients. Skill in Microsoft Word and the ability to provide a well-

formatted final document was not the primary goal, but Dr. Thomas did express a preference for

a polished final project. Each team member took responsibility for writing at least two sections

of the instructor’s guide, and the final report, to evenly distribute content-development. The

entire process was a true collaboration for our team.

Challenges

One of the challenges we faced was staying within the scope of the project. At first, we

tried to cover all the content from the article “Making Sound Decisions: A Framework for

Judging the Worth of your Data” into the training. We soon realized that this would be an

information overload for the learners and would be impossible to teach in one hour. We narrowed

the training subject to the use of the decision-making flowchart, which meets the objectives and

the needs of the client within the time constraints.

The next challenge was designing the instruction. The team had to decide on the

complexity of the examples for each data qualities. We decided to use simple examples since we

were going to introduce new terms: relevance, reliability, validity, and completeness. We
INSTRUCTOR’S GUIDE FOR DECISION MAKING

included a more complex example at the end of the training when the learners have already been

exposed to the terms. To aid the learners with the terms and the process of evaluating data, we

made a job aid that has all the information needed on one page. Doing all of this in sixty minutes

of training time was impractical since learners needed to practice using the job aid with each

step. We resolved on extending the instructional time to ninety minutes and received approval

from the client.

The final challenge was communicating our differences of opinion. Our team addressed

this early in our planning sessions so that we had guidelines in place to work through our

creative differences. Also, our method of dealing with the lively back and forth in meetings with

all team members and the client was to trade off the conversational lead. Returning Team One

has worked together throughout our MIST enrollment, and we support and rely on each other’s

strengths quite well.

Advice for Future Students

Our advice is to contact the client right away to set up a meeting and complete the “Kick-

Off Meeting Agenda” with the given information. Further, the entire team should be included in

all communications and attend all meetings. This is important to ensure all team members have

an opportunity to develop skill in instructional development. Our entire team met with our client

almost every week to discuss our progress, review design updates, and plan deliverables. We also

found it helpful to meet as a team before each meeting with the client. We created the agenda,

decided who would lead the discussion of each agenda item, review the next deliverable, and our

goal for the client meeting. This helped to keep everyone on the same page and made the best use

of the client’s time and expertise. It is important to respect the client’s time by being prepared

and meeting regularly. We also sent the meeting agenda to the client in advance, kept meeting
INSTRUCTOR’S GUIDE FOR DECISION MAKING

notes, and shared these with the client for her comments. This communication kept the client

informed and confirmed the team’s understanding of the client’s requests.

An important part of our process was the timing of client communications. We chose to

review documents with our client following submission to the instructor. This choice allowed our

team to submit our own writing for grading and instructor feedback, then gain additional

feedback from the client on the materials. The client received all final versions of project

deliverables.

We recommend the following actions:

 Assign team roles and responsibilities.

 Plan in advance for conflict and offer support to team members who may struggle with

roles and responsibilities.

 Establish early and regular contact with the client.

 Document meetings and include assigned action items.

 Share notes with the client and asks for feedback.

 Track changes and maintains document versioning.

 Be very clear on the scope of the project and do not exceed the project definition. If

necessary, designate areas “to be developed.”

 Leave sufficient time for revision and final polishing of the deliverables for the client.

 Understand that the process will be frustrating at times and your team will gain and lose

ground. This is normal.

 The project is about learning to work with a client in a design project. Remember to

nurture your client relationship.

INSTRUCTOR’S GUIDE FOR DECISION MAKING

 At the end of your project, ask your client for feedback on the entire process. This advice

is invaluable.

 Ask for help when you need it.

 Remember to thank your client for his or her time and effort!

Summary

Returning Team One worked well together and were happy with our choice of project and

client. We communicated frequently via text, email, Zoom, and in chat windows. We used

Google Docs to allow everyone to edit the documents, tracked revisions and document

versioning, and exported the final content to Word for advanced formatting. We took advantage

of each team member’s strengths but allocated a variety of tasks so that each of us gained full

benefit from the experience. We also tried to remain positive, and support and listen to each

other’s ideas.
INSTRUCTOR’S GUIDE FOR DECISION MAKING

References

Guerra-Lopez, I. & Thomas, M. (2011). Making sound decisions: A framework for judging the

worth of your data. Performance Improvement, 50 (5).