A critique of the document evaluation framework in

Designing Public Documents

by David Sless (2004)
Victor Diacono

Introduction: benchmarking the design of public documents

In this paper, David Sless lays strong emphasis on the democratic process
and on consumer pressure as drivers for the good design of public
documents. To achieve this aim, he sets a series of four criteria as a
benchmark: the socially desirable, the current performance of the document,
the practically achievable and the socially acceptable.
The application of benchmarks in itself is of course highly desirable as it
preempts Windahl’s (2009) concern about “correct and unbiased reception
[being] taken for granted”. Assumption and bias on an author’s part will
always creep in but, if passed through a pre-compilation sieving process,
residual bias is bound to be less damaging than if it were introduced
unfiltered from the start, thereby compromising the document’s whole
framework.
However, Sless’s hierarchical filtering of benchmark categories is not
always convincing for two reasons. There is the ‘current performance of
documents’ which raises the question, not dealt with by the author, of what
procedure to adopt with new documents. Does one simply remove the
second criterion and reduce the checklist to three? But perhaps more
critically, the sequence of the fourth criterion seems debatable. Is the
practically achievable being sieved (and sacrificed in part) for the sake of
political correctness and as an offering on the altar of democratisation?
Because the implication is that what passes the test of the previous criteria is
to be submitted for the imprimatur of the Socially Acceptable and axed if not
found up to measure.

Quantitative versus Qualitative

Is document quality only to be gauged by the attributes going into Sless’s set
of strictly quantitative benchmarks? The qualitative is completely absent
from the Sless criteria as highlighted by the emphatic percentile hit-rates of
literates that should be able to locate information quickly and easily. One is
led to query whether the application of such strictly and exclusively
quantitative criteria is in fact socially acceptable in itself. None of the
benchmarks takes into consideration lucidity or pleasantness of diction, for
instance. These two attributes impact strongly on Attractiveness and
Credibility, in the latter case insofar as it holds that documents have not only
to be authoritative but also perceived as such. It even impacts on
Respectfulness shown to the potential readership when those not fully
literate have to struggle to comprehend.
And here lies one deficiency in Sless’s paper: the attributes overlap to
such degree that one could justifiably feel the need to flowchart them if they
were to be appreciated and understood comprehensively! While they do
provide a reasonably strong framework upon which to fashion the design of
documents, further decomposition of the individual attributes (particularly
the first three of the Socially Desirable class) would lead to more
unambiguous benchmarking.

Emphasis on the democratic process – a contradiction?

Another arguable area, and a critical one, lies in the proposed framework
targeting only the fully literate. Fair enough, the illiterate cannot be
addressed through literature but the semi-literate can, if aided. This social
group is not given any meaningful consideration by Sless. In fact, non-
literates are explicitly excluded even though the medicinal application
adopted by the author is of obvious import to all. Moreover, this approach
appears to be a contradiction of the espoused aim of democratisation… it can
hardly be deemed respectful of democratising processes when no specific
effort is made to make critical information available to anybody but the fully
literate!
However, there cannot be much argument over Sless’s prerequisites for
both the Physically Appropriate and the Socially Appropriate. The need for
Physical Appropriateness stands out not only where inappropriateness should
be patently obvious to the designer, such as in the context of road maps to
be read in a moving car (page 26). It is often also evident in the mundane,
such as in restaurant menus where the graphical component of menu design
stands out to an extent which, especially when exacerbated by an ambience
of dim lighting or dark decor in the establishment, does not allow for easy
reading of the overlying type. And the socially inappropriate does not only
come over as disrespectful when, for instance, a patronising conversational
dynamic is adopted, but can also take the chatty form which can be both
hilarious and indicative of lack of professionalism on the commissioning
organisation’s part (Stokes, 2010b).

Reader sustainability
While the above five attributes, deemed by Sless to be necessary for the
enlistment of readership, are somewhat open to debate, his attributes for
reader sustainability are clearly delineated and concise.
The gauge of Productivity is deemed as being the document’s usefulness
both in the imparting of information and in the reassurance it can provide the
reader in search of confirmation of knowledge already known. But its most
productive attribute must be the opening up of the possibility of building
upon knowledge through further action or dialogue.
Usability and Efficiency may not appear to be clearly distinct attributes,
and Philip Stokes (2010c) laments their seeming inseparability. But after
further reflection it becomes evident these two attributes do not overlap as
heavily as they appear to at face value. While one can spend either a long or
a reasonable time looking up information in a document, depending on its
level of efficiency, there is no reason why inefficiency at look-up stage should
impinge on ease of use of the same information once located.

Conclusion
The principal area of debate in Sless’s argumentation lies in the ambiguity of
the five attributes (in terms of the crossing-over of categories) when one
attempts to apply them, as well as in the emphasis on democratization. The
former have been expounded on above and, with the latter, while it is all well
and good that democratic pressure for literature espousing the said attributes
is heeded, should the Socially Desirable always be the governing criterion? It
is definitely socially desirable to have brochures or manuals that can be read
from cover to cover in a short time and understood immediately, but not all
applications can conform to such desirability.
There are applications, such as in the case of industrial tools and
machinery, or indeed the very same medicinal application to which Sless
applies his argumentation, where it could be dangerous to look at social
desirability from such an angle. It is especially from this angle that Social
Desirability should not be the governing criterion, but an attribute apart to be
accommodated where it does not impact on critical or professionally desirable
criteria.

Bibliography
Levi, Michael C., Conrad Frederick G. (2002). A Heuristic Evaluation of a
World Wide Web Prototype. Washington, DC. U.S. Bureau of Labor
Statistics (Office of Survey Methods Research).
Schriver, K.A. (1997). Chapter 7: What document designers can learn from
readers. In: Dynamics in Document Design, New York, Wiley, 443-496.
Stokes, P. (2010b). Blackboard Document Evaluation blog post (Evaluating
the OmniOutliner User Manual), 7th November 2010.
http://shuspace.shu.ac.uk/webapps/portal/frameset.jsp?tab_tab_group_id
=null&url=/webapps/blackboard/content/launchLink.jsp%3Fcourse_id%3D_
228115_1%26tool_id%3D_144_1%26tool_type%3DTOOL%26mode%3Dvi
ew%26mode%3Dreset
Stokes, P. (2010c). Blackboard Document Evaluation blog post (in Evaluation
of an instruction Manual by Eunice Mannion), 9th November 2010 to 10th
November 2010).
http://shuspace.shu.ac.uk/webapps/portal/frameset.jsp?tab_tab_group_id
=null&url=/webapps/blackboard/content/launchLink.jsp%3Fcourse_id%3D_
228115_1%26tool_id%3D_144_1%26tool_type%3DTOOL%26mode%3Dvi
ew%26mode%3Dreset
Windahl, S., Signitzer, B. & Olson, J.T. (2009). Using Communication Theory:
an Introduction to Planned Communication. London: Sage, 205.
In: Stokes, P. (2010a). Blackboard Document Evaluation discussion (in
Evaluation – Online Help System by Uchit Luktuke), 3rd November 2010 to
21st November 2010.
http://shuspace.shu.ac.uk/webapps/portal/frameset.jsp?tab_tab_group_id
=null&url=/webapps/blackboard/content/launchLink.jsp%3Fcourse_id%3D_
228115_1%26tool_id%3D_144_1%26tool_type%3DTOOL%26mode%3Dvi
ew%26mode%3Dreset