SOFTWARE EVALUATION MODEL

Software Evaluation Model:

A Condensed Tool for Assessing Educational Apps
Authored by BC Boys:
John Munro
BrentonClose

EDUC 5101:91
(Assessment of Software and Information Technology Applications for Education)
Cape Breton University
Prof. David Lloyd
July, 2015

SOFTWARE EVALUATION MODEL

Introduction
As technology plays an increasingly central role in 21st century education, selection
of software for schools is critical to the success of technology enhanced learning. It
is inconceivable to expect the selection process to be effective without the help of
some kind of evaluation tool, but choosing from the plethora of evaluation models
can be as confusing as choosing the software itself. Broadly, software can be
assessed through experimental study, by expert opinion, or by user surveys.
(Stirling, 1997), and in the case of education, decisions around purchasing software
are almost always made by teachers and administrators in schools. This
recognition might seem to narrow the process (to expert opinion), but in reality the
number of evaluation tools and the scope and variation among them only complicate
the issue further.
Our intention in creating the following software evaluation tool is to offer a
model that is best suited to the particular needs of science and mathematics
teachers interested in testing software applications for middle and secondary
classrooms. The model does contain enough content and functionality criteria to be
somewhat generic, but our purpose is not to create a one size fits all evaluation
tool. The curricular connections in this tool are specific to the British Columbia
Education Plan and the competencies contained there (BC Education Plan, 2015),
and we hope that teachers faced with the dual challenge of new curriculum and
abundant software will find this software evaluation model useful.

SOFTWARE EVALUATION MODEL

Part A: Who Will Use the Software Evaluation Model?

The software evaluation tool created by our group is intended for use by teachers who are
interested in introducing iPAD and iPOD applications into their classrooms. The number of free
or almost free application offerings already makes selection a daunting task, and the pool of
availability continues to grow at what seems an alarming rate. This tool could potentially be
applied to a wider range of software selection, but is intended primarily to assist teachers in
filtering through the huge number of available applications that promise to deliver educationally
sound and challenging learning experiences for students.
In particular, teachers of middle school and high school science or mathematics courses
in the province of British Columbia will find this evaluation tool useful. This is true for a couple
of reasons. First, the model is tailored to software titles of particular interest to teachers at this
level. Secondly, the categories in the model align with the emerging curriculum in British
Columbia, the new BCED Plan (BC Education Plan, 2015), which focuses on 21st century skills
and reflects the continuum of student learning through the use of competencies rather than
learning outcomes as in previous curricula. We consider both core competencies and
curricular competencies from this government document, and it is expected that a teacher will at
least be familiar with these standards before using the evaluation tool. The Depth of Knowledge
(DOK) model developed by Webb governs large scale assessment in BC as well and is therefore
an important consideration for secondary teachers (Webb, 1997). We have included this element
in our preliminary survey of the software title.
We expect that individual teachers would use this quick evaluation tool to determine
whether an application meets their needs, and in the event that a more expensive piece of
software is under consideration for purchase, the tool can be used to demonstrate due diligence in

SOFTWARE EVALUATION MODEL

evaluating the software before dedicating school budget to it. For this reason we have included a
section to explain the intended use of the software. We would consider this a standard
procedural step to be undertaken in the process of seeking administrative approval for a
purchase.
With regard to the process of evaluating software, the research findings of Jadhav &
Sonar (2008), suggest using a seven-step procedure:
1. Determining the need for purchasing the system and preliminary investigation of the
availability of packaged software that might be suitable candidate.
2. Short listing of candidate packages.
3. Eliminating most candidate package that do not have required feature or do not work with
the existing hardware, operating system and database management software or network
4. Using an evaluation technique to evaluate remaining packages and obtain a score or
overall ranking of them
5. Doing further scrutiny by obtaining trial copy of top software packages and conducting
an empirical evaluation. Pilot testing the tool in an appropriate environment.
6. Negotiating a contract specifying software price, number of licenses, payment schedule,
functional specification, repair and maintenance responsibilities, timetable for delivery,
and options to terminate any agreement.
7. Purchasing and implementing the most appropriate software package. (p. 557)
Though most of these steps are commonly assumed and would occur automatically in a normal
software selection, we recognize that our evaluation model represents only part of the selection
process. In relation to the work of Jadhav & Sonar, our model would be most appropriate for the

SOFTWARE EVALUATION MODEL

completion of step 4: evaluating and ranking the operable and potentially useful software that
survive the obvious eliminations of the third step.

Our software evaluation tool is intended for use by classroom teachers, particularly in the
areas of science and mathematics. Most teachers have little extra time for reviewing titles but
are responsible for answering to curricular requirements, have concern that their students are
engaged in meaningful ways, and are intimately familiar with their own classes and classroom
routines. With this in mind our model is as brief as possible while still highlighting the essential
considerations for software in classrooms, especially the large numbers of inexpensive or free
applications available.

Part B: Criteria Used in the Software Evaluation Model

A study conducted by Hardin & Patrick in 1998 (as cited in Kara, 2007) states, One of
the major reasons for poor software in schools deals with the evaluation of that software by
education professionals. Despite the increasing number of software titles in education, the area of
educational software evaluation has been increasingly more and more muddled because of a lack
of consensus among software evaluators. This dated study unfortunately remains totally
relevant to this day. There is little consensus on the subject among both developers and users of
todays more advanced and progressive software titles. Issues that are paramount to one person
are seemingly insignificant to another. This makes it nearly impossible to develop one set of
criteria by which a software tool would be evaluated. Nevertheless we intend to use a
streamlined evaluation tool to aid in this determination. Further, we hope to embrace the
challenge imposed by this lack of consensus, and to develop a tool not intended primarily for

SOFTWARE EVALUATION MODEL

general use, rather for specific application in middle and secondary science and mathematics
classes.
We recognize that applications have a variety of uses and this in particular was a driving
concern in our development process. It is our feeling that a teacher using this tool should have a
clear understanding of the various types of applications that are available. It is necessary to
understand what a software title is intended to deliver before it can be evaluated, so our
evaluation tool includes the type of application along with the early identifying characteristics of
the software. Application categories as suggested by the article 11 Types of Education
Technology Available to Schools (Technology in Education, 2014), include: authoring systems,
desktop publishing, graphic software, reference software, drill & practice software, tutorial
software, educational games, simulations, special needs software, math problem solving
software, and utility software. With a few modifications for clarity we have chosen categories
that include:
Simulations: Tools that provide a virtual experience such as dissections or laboratory
procedures.
Tutorial: Programs that are useful in reinforcing basic skills and in sharing new content with
students, often used in lower grades to reinforce learning. Tutorials allow self-paced learning.
Drill: Programs that focus on on-going review of essential/basic skills. Drills utilize repeated
practice with on-going feedback and progression of skill level. They are useful in test
preparation.
Utility: Used to prepare tests, quizzes, flash cards etc. or also as a grading system.
Game: Combines learning with gaming to improve student motivation.

SOFTWARE EVALUATION MODEL

Logic tool: Used to strengthen problem-solving skills through critical thinking/mind building
challenges.
Development tool: Programs such as word processors used by students to produce and display
their work, also known as desktop publishing tools.
Reference: Applications used for gathering information and research. References include
encyclopedias, thesauruses and other sources of information.
Graphic: Used to capture, edit and create images, often used for presentations.
Special Needs: Addresses specific needs of the learner. These include speech synthesizers that
read text or multi-media tools that address specific learning disabilities.
Communication: Programs that use telecommunications networks to share ideas and allow
students to collaborate with others.
It is our expectation that anyone choosing to use our evaluation tool would apply it to software
titles that fall into one of these categories.
It is also assumed that users would become familiar with aspects of large scale
assessment which includes the use of Webbs Depth of Knowledge categories. The Depth of
Knowledge (DOK) model is employed to analyze the cognitive expectation demanded by
standards, curricular activities and assessment tasks (Webb, 1997), and we have included an
indicator of the level of cognitive expectation demanded by the software being evaluated.
Webbs levels are elaborated as:
Level 1: Recall and Routine - Curricular elements that fall into this category involve basic tasks
that require students to recall or reproduce knowledge and/or skills. A student answering a Level
1 item either knows the answer or does not; that is, the answer does not need to be figured out
or solved.

SOFTWARE EVALUATION MODEL

Level 2: Skills and Concepts - At this level students are asked to transform/process target
knowledge before responding. Example mental processes that often denote this particular level
include: summarize, estimate, organize, classify, and infer.
Level 3: Short-term Strategic Thinking - Items falling into this category demand a short-term
use of higher order thinking processes, such as analysis and evaluation, to solve real-world
problems with predictable outcomes. Key processes that often denote this particular level
include: analyze, explain and support with evidence, generalize, and create.
Level 4: Extended Strategic Thinking - Curricular elements assigned to this level demand
extended use of higher order thinking processes such as synthesis, reflection, assessment and
adjustment of plans over time. Key strategic thinking processes that denote this particular level
include: synthesize, reflect, conduct, and manage.
In addition to the focus on type of software tool, we considered elements of the 21st
century learning experience. We direct our attention at core competencies and curricular
competencies as laid out in the BCED plan. This initiative will drive our classroom practice
when implemented in British Columbia, so we felt it prudent to move in this direction in advance
of the shift. As such, this tool would be best used by someone familiar with the BCED Plan and
the move toward standard based assessment. (BC Education Plan: Focus on Learning, 2015).
Critical thinking and problem solving involves making judgements based on reasoning. This
includes considering options; analyzing criteria; drawing conclusions and making judgements.
Collaboration and leadership emphasizes skills that promote and generate cooperation among
peers in order to achieve a collective outcome

SOFTWARE EVALUATION MODEL

Communication and digital literacy encompasses a students ability to impart and exchange
information, experiences and ideas, explore the world around them and to effectively engage
with digital technology.
Personal and social responsibility refers to an individuals awareness, understanding and
appreciation for the factors that contribute to a healthy sense of oneself.
Creativity and understanding emphasizes new approaches to situations. Additionally, it
questions conventional approaches and encourages new ideas, innovations, designs and attempts
new cutting edge approaches.
Global and cultural understanding explores an individuals ability to relate within the broader
social context of their world as it is reflected in identifiers such as ethnicity, nationality,
language, ability, sex/gender, age, geographic region, sexuality and religion.
Within the realm of science education, the more subject-specific curricular competencies
include: evaluating, communicating, planning and conducting, questioning and predicting,
processing and analyzing data/information, applying and innovating.
The second page of our evaluation tool deals with content and functionality of an
application. With respect to the categories chosen in this section of our model, all types of
schemes have been created to evaluate software. Some of these schemes are very simple and
only involve three major items to examine: solid instructional design, flexibility, and results
(WD&S, 2000). Still other assessment tools used eight different categories: accuracy, learning
style, user experience, curriculum, pedagogical philosophy, learning objectives, format, and
instructor preferences (Hardin & Patrick, 1998). A paper by Jason Wrench at USC Rossier
School of Education suggests six basic areas: curriculum content, instructor use, student use,
program content, program operation, and publisher information. (Jason S. Wrench, 2001) This

SOFTWARE EVALUATION MODEL

last structure is more appealing to us, since it includes some aspects of the software applications
usability as well as various aspects of student learning.
For our purposes we have elected to frame our evaluation tool loosely around these areas,
choosing to explore some areas in slightly more detail. In recognition that these categories
require more than simple checkmark responses, we have used a four-point scale for a more
graduated form of assessment, and have included an N/A option in the event that a particular
category is irrelevant to the software being evaluated. The scale we have used is similar to a
Likert scale (McLeod, 2008), but without a collective scoring system. We have chosen to
emphasize distinction between software on the strength of individual categories rather than
relying on the collective score of a true Likert scale. The categories found on the second page of
the evaluation tool represent a variety of aspects of both student use and instructor use of the
software.
Curriculum content and curriculum connection are two important elements of any
educational software design and therefore essential in any evaluation. While it seems common
for teachers to use software simply because it is there and it fills time, it is not preferable for this
to occur. Instead, a software title should be selected and implemented with a clear focus on the
integration and collaboration with the course curriculum. A parallel concern, especially in the
area of scientific study which can change quickly and dramatically with new discoveries, is that
the content be accurate and up-to-date. Well-designed software might be combined with great
pedagogy to reinforce learning, but if the content of the software is inaccurate or dated, the
learning might actually be the seed for significant ignorance and misinformation. A strong
educational resource will have clear objectives, and be age-appropriate, closely aligned with all
curriculum content, and contain current and accurate information.

10

SOFTWARE EVALUATION MODEL

In the area of student use, we feel that student engagement is essential to the
effectiveness of any software title. For this reason we have emphasized engagement by placing it
near the top of our criteria and have used less ambiguous language for this category. Closely
linked to engagement, it is also essential that the user experience authentic and relevant
content through the learning activities. In a study conducted by Howe (1989), it was found that
many children who mastered spelling vocabulary words on the computer were unable to define
and spell those same words in other situations. Context of learning is clearly relevant to
retention, and it is the authenticity of students experience that will enable them to transfer their
learning beyond the digital environment of the computer workstation, to real-life situations. An
effective software tool should engage students in the most authentic activities possible, thereby
maximizing retention and the ability to apply learning outside the digital environment. In some
instances, teacher support materials could assist in off-line activities and assessment of the skills
learned through the digital application.
On-going feedback for students motivates further effort and becomes a tool for
improvement through formative assessment, so is included in our evaluation tool as a distinct
category. It is further expected that the tool provide graduated levels of challenge to maintain
learner engagement and to ensure growth in the intended learning. Balance is essential to the
graduation through increasing levels of challenge; if the difficulty increases too rapidly students
may become frustrated and disengage from the activity, but if the top level of a software
application is reached too quickly then learning is limited.
Increasingly, as software tools become more elaborate, they are expected to provide
differentiation for learners, both in learning ability and in learning modality; a good software
title should utilize several ways of reinforcing a students learning. Other areas emphasized in

11

SOFTWARE EVALUATION MODEL

relation to student use include the ability to collaborate with other users; concerns about social
bias; and usability.
Instructor use addresses ways in which student interaction with the software is directed or
supplemented by instruction, rather than simply allowing students to use the program without
direction or guidance. This might include supporting documentation with suggestions for the
implementation of the software in a classroom setting. Follow-up activities, student worksheets,
group work or projects may also be a part of this information. Manuals and instructor materials
should clearly outline prerequisite skills and curriculum connections. In addition, teachers often
need to produce reports of student progress. The ability to produce these reports and the ability to
share them easily are often an important factor in the successful use of a software tool.
Accordingly, we have included categories for instructions, reporting, and clarity of reporting.
An essential component to any software is the support that continues after the purchase.
In many cases this is simply a user manual or online forum, but in many cases (especially when
the investment is significant) users demand personal tech support from the software developer.
Troubleshooting, which includes the availability of tech support, is the final category in our
evaluation tool.
Finally, our evaluation model closes with a space for the reviewers recommendation for
or against the software being assessed, and a space for evaluator comments. In the event that
several applications are being reviewed at once, we feel that a short personal comment or
anecdotal reflection on the software is what will later differentiate it from others, and can
sometimes summarize overall impression in a single comment.

12

SOFTWARE EVALUATION MODEL

PART C: Rationale for the Software Evaluation Model

Software evaluation involves many criteria as they relate to personal preferences of the end
user. As such there is little agreement on a universal tool for this purpose. There are some
common strategies used in the development of these tools however. Three of these are evaluated
by Jadhav, A., & Sonar, R. (2008):
The analytical hierarchy process (AHP) is one widely used organizational strategy. Users
of the AHP first reduce a complex problem into a hierarchy of simpler sub-problems. Each subproblem is then evaluated independently. Tools designed according to this infrastructure can be
utilized to address any aspect of the decision making process. These factors can be carefully
measured or approximated, well understood or poorly understood, tangible or intangible. AHP is
widely used for evaluation of the software packages. The highest level emphasizes the goal of
the selection process. Next the major factors are further broken down so that at the lowest level
each alternative can be analyzed by comparison. Ultimately a relative final score is tabulated for
comparison of each alternative.
The weighted scoring method establishes a weighted ranking scale for the evaluation
criteria. The predetermined weighting reflects the relative importance of each of the criteria as
determined by the designer of the evaluation tool. A rating scale determines the degree to which
each criteria achieves the evaluators expectations. The rating scale and weighted value are then
multiplied and then added to achieve a score.
Finally there is the fuzzy based approach. This technique is most valuable when
performance ratings are less precise. This method recognizes the uncertainty inherent in human
judgement. Criteria are listed, but the evaluator is directed to apply a personal standard to each of
these criteria based on their experience and interest.

13

SOFTWARE EVALUATION MODEL

Our tool utilizes a hybrid of these types. The challenge of choosing a software title is first
simplified by determining whether it meets the initial requirements of covering curriculum and
core competencies. Then the specific criteria that we view as essential are listed in detail. The
complication of a weighted average is avoided by using sub-categories to further specify areas
that we view as more important such as student use. The evaluation tool is our effort to develop a
somewhat generic framework that can be used for selecting from a wide range of software titles,
though we do focus on curricular connections to science and mathematics. This framework is
designed to help decision makers in evaluation and selection of the software packages while
reducing the time and effort required for software selection.
We are intentional in making this software evaluation tool as concise as possible. In
practice if a teacher needs to evaluate several titles it is unlikely that (s)he will undergo a lengthy
assessment of each. The criteria we have chosen are very deliberate and cover a broad range of
potential concerns. In the article, Educational Software: Criteria for Evaluation, the author
suggests a context for software evaluation, stating that both the software and the medium of
delivery should be evaluated. Jadhav & Sonar (2008) explicate a detailed seven-step procedure
for software selection beginning with Determining the need and ending with Purchasing
and implementing (p. 557). It is our view that most of these steps are already implicit in the
normal process of selection, so the focus of our evaluation tool is mostly on step 4 of this
process, Using an evaluation technique to evaluate remaining packages and obtain a score or
overall ranking of them (p. 557) It should also be noted that in many cases schools make
hardware purchasing decisions prior to departmental or individual decisions around software. In
these cases limitations on software choice are pre-determined (Geissinger, 1997).

14

SOFTWARE EVALUATION MODEL

It is worth noting that software evaluation is a subjective process even when numerical
values are assigned. With this in mind the end-user should be the person evaluating the software
rather than a coordinator or administrator. Teachers who intend to implement the considered
software must be part of the decision-making process.

Part D: The Software Evaluation Tool

Please see the second attachment for our evaluation tool, which is created using a
horizontal format and different margins than this paper.

15

SOFTWARE EVALUATION MODEL

References
11 Types of Education Software Available to Schools - BRIM Anti-Bullying Software. (2014, April
30). Retrieved July 18, 2015, from https://antibullyingsoftware.com/blog/technology-ineducation/11-types-of-education-software-available-to-schools

BC Education Plan: Focus on Learning. (2015). Retrieved July 18, 2015, from
http://www.bcedplan.ca/assets/pdf/bcs_education_plan_2015.pdf

Finding the Right Tool for the Task -- 4 Categories of Educational Technology. (2001, February 5).
Retrieved July 18, 2015, from http://www.sun-associates.com/resources/categories.html

Geissinger, H. (1997, December 7). Educational Software: Criteria for Evaluation. Retrieved July
25, 2015, from
http://www.ascilite.org.au/conferences/perth97/papers/Geissinger/Geissinger.html

Hardin, L., Patrick, T. B. (1998). Content review of medical educational software assessments.
Medical Teacher, 20, 207-212.

Howe, J. A. (1988). Assessment of skill generalization taught by computer software. Unpublished

Masters Thesis, Russell Sage College.

Jadhav, A., & Sonar, R. (2008). Evaluating and selecting software packages: A review. Information
and Software Technology, (51), 555-563. Retrieved July 25, 2015, from
http://www.ic.unicamp.br/~wainer/outros/systrev/29.pdf

16

SOFTWARE EVALUATION MODEL

Kara, Y. (2007). Educational Software Form for Teachers. Paper Presented at the International
Educational Technology (IETC) Conference. Retrieved July 26, 2015, from
http://eric.ed.gov/?id=ED500133

McLeod, S. A. (2008). Likert Scale. Retrieved July 26, 2015, from

www.simplypsychology.org/likert-scale.html

Stirling, D. (1997). Evaluating Instructional Software. Retrieved July 24, 2015, from
http://www.stirlinglaw.com/deborah/software.htm

Tokmak, H., Incikabi, L., & Yelken, T. (2012). Differences in the educational software evaluation
process for experts and novice students. Australasian Journal of Educational Technology, 28(8),
1283-1297. Retrieved July 26, 2015, from http://www.ascilite.org.au/ajet/ajet28/sancartokmak.pdf

WD&S. (2000). How to evaluate educational software. Curriculum Review, 39, 15. Zane, T., &
Frazer, C. G. (1992). The extent to which software developers validate their claims. Journal of
Research on Computing in Education, 24, 410-420.

Webb, N. (1997). Criteria for alignment of expectations and assessments on mathematics and
science education. Research Monograph, 6.

17

SOFTWARE EVALUATION MODEL

Wrench, J. (2001). Educational Software Evaluation Form: Towards a New Evaluation of
Educational Software. The Source, 3(1). Retrieved July 24, 2015, from
http://www.cin.ufpe.br/~case/artigos/Avaliacao%20e%20Classificacao/Educational%20Software
%20Evaluation%20Form%20-%20Towards%20a%20New.pdf

18