See

discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/286459229

Social Semiotics

Chapter · January 2009

DOI: 10.1075/hop.13.soc5

CITATIONS READS

0 4,939

2 authors:

Jeff Bezemer Carey Jewitt

University College London University College London
58 PUBLICATIONS 800 CITATIONS 86 PUBLICATIONS 2,077 CITATIONS

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

In-hospital resuscitation under the microscope: Analyzing inter-professional communication using

video View project

IN-TOUCH: Digital touch communication View project

All content following this page was uploaded by Jeff Bezemer on 11 December 2015.

The user has requested enhancement of the downloaded file.

This is a pre-print of Bezemer, J. & C. Jewitt (2009). Social Semiotics. In: Handbook
of Pragmatics: 2009 Installment. Jan-Ola Östman, Jef Verschueren and Eline
Versluys (eds). Amsterdam: John Benjamins

Social Semiotics

Introduction
Social semiotics is concerned with meaning makers and meaning making. It studies
the media of dissemination and the modes of communication that people use and
develop to represent their understanding of the world and to shape power relations
with others. It draws on qualitative, fine-grained analysis of records of meaning
making, such as ‘artifacts’, ‘texts’, and ‘transcripts’, to examine the production and
dissemination of discourse across the variety of social and cultural contexts within
which meaning is made. Different ‘versions’ of social semiotics have emerged since
the publication of Michael Halliday’s Language as Social Semiotic in 1978. The
account we offer in this paper is focused on the version proposed by Gunther Kress,
Robert Hodge, Theo van Leeuwen, and others. Following a historical overview we
discuss its connections with Pragmatics and other approaches; key concepts;
analytical focus; and fields of application.

Historical overview
In Language as Social Semiotic (1978) Michael Halliday proposes that the semiotic
resources of language are shaped by how people use them to make meaning-the social
functions they are put to. He holds that every sign serves three functions
simultaneously: they express something about the world (‘ideational metafunction’),
position people in relation to each other (interpersonal metafunction) and form
connections with other signs to produce coherent text (‘textual metafunction’). His
ideas were taken up by Robert Hodge and Gunther Kress, who developed a critical
account of language and society derived from Marx in Language as Ideology (Kress
& Hodge, 1979). In Social Semiotics (1988) they adopted a similar stance to explore
any set of semiotic resources that people use in everyday life, the resources of
language as much as the resources of image, and of other modes. Hodge & Kress
point to a number of distinctive features of a social account of semiotics. First, they
problematize essentialist notions of meaning:

“Traditional semiotics likes to assume that the relevant meanings are frozen and
fixed in the text itself, to be extracted and decoded by the analyst by reference
to a coding system that is impersonal and neutral, and universal for users of the
code. Social semiotics cannot assume that texts produce exactly the meanings
and effects that their authors hope for: it is precisely the struggles and their
uncertain outcomes that must be studied at the level of social action, and their
effects in the production of meaning.” (Hodge & Kress, 1988:12)

Like Pragmatics, Social Semiotics argued against some of the working hypotheses of
traditional linguistics and semiotics, and in favour of a situated perspective on
communication. Social semiotics aims to account for ‘context’, not based on “a naive
text-context dichotomy”, but rather based on the assumption that “context has to be
theorized and understood as another set of texts.” (Hodge & Kress, 1988:8) Crucially,
that involves breaking with the assumption that verbal language is always dominant
and autonomous. As Hodge (2009) puts it: “Social meanings cannot be tracked only

1
in one code, even in verbal language as the dominant one. The supposed dominance
and autonomy of the verbal code is indeed an ideological assumption whose taken-
for-granted truth needs to be questioned by social semiotics.”
Hodge & Kress discuss examples from a range of social contexts involving a
range of different modes, but writing and image in ‘print’ media (e.g., magazines,
billboards) are discussed more than speech, gesture, gaze and other modes operating
in the social encounters with which Pragmatics is concerned. In the late 1980’s, when
Gunther Kress and Theo van Leeuwen began to develop a social semiotic approach to
the visual, the focus was still on print media, culminating in their book Reading
Images. The Grammar of Visual Design (1996). In Reading Images they propose a
framework for the analysis of image, which draws on the broad semiotic aspects of
Halliday’s social semiotic theory and made use of the system networks as a heuristic
framework for theorizing meaning as choice. At the same time, Michael O’Toole
applied Halliday’s systemic functional grammar and the tools it offered to examine
the visual in his book The Language of Displayed Art (1994). Both accounts of image
as a set of meaning making resources were also informed by the insights of film
studies, iconography and art history.
The issue of mode and multiple modes became, perhaps inevitably,
foregrounded in the 1990s. With the focus now on the co-operation of modes rather
than the study of modes in isolation the term ‘multimodality’ became a key term. It
provided an alternative to terms such as ‘verbal’ and ‘non-verbal’, which position the
verbal as the unmarked, dominant resource for making meaning, and the non-verbal
as the marked, auxiliary resource. In Multimodal Discourse. The Modes and Media of
Communication (2001), Kress & van Leeuwen describe the ‘multimodal’ perspective
as follows.

“We aim to explore the common principles behind multimodal communication.

We move away from the idea that the different modes in multimodal texts have
strictly bounded and framed specialist tasks [...]. Instead we move towards a
view of multimodality in which common semiotic principles operate in and
across different modes.” (Kress & van Leeuwen 2001:2).

The framing of the book as being about ‘discourse’ rather than ‘semiotics’ marks the
connection between social semiotics and discourse (in a Foucauldian sense as a social
category). Until then, (Critical) Discourse Analysis, while sharing an interest in
reconstructing ideologies (cf. Fairclough, 2003), had not attended to modes other than
speech and writing, or to their material, cultural and social affordances; that was the
realm of social semiotics (see Analytical Focus). As ‘discourse’ and ‘critical’ became
more ‘speakable’ (Hodge 2009) publishers preferred book titles featuring those terms
and they became important carriers of social semiotics. Connections with Discourse
Analysis and other approaches are also emphasized in the Routledge quarterly Social
Semiotics (founded in 1990), which presents itself as “a journal for discourse and
critique looking for high quality, politically engaged papers that use textual analysis,
discourse analysis, political economy, ethnography or combinations of these and/or
other methods, to say something concrete about the nature of life in our societies.”

Connections with other approaches

Like Pragmatics, Social Semiotics is concerned with meaning in context. Whether
they are seen as ‘disciplines’ or ‘perspectives’ they both adopt a functional
perspective on meaning making. They are both reactions to traditions which largely

2
ignored the social and cultural situatedness and power implications of meaning
making. Some other approaches have played a major role in the development of both
Pragmatics and Social Semiotics, such as Sociolinguistics, Linguistic Anthropology,
Hallidayan Linguistics and Critical Discourse Analysis. Other disciplines have
impacted on one more than the other. Social semiotics for instance also draws on Film
Theory and Iconography; Pragmatics also draws on Pyscholinguistics.
A key difference between the two disciplines is their approach to multimodality.
To varying degrees, Pragmatics has been and is increasingly concerned with modes
other than speech, such as gesture or gaze, especially in work drawing on
conversation analysis, interactional sociology, interactional sociolinguistics, linguistic
anthropology, micro-ethnography and linguistic ethnography. Some of this work is
based on the working hypothesis that speech or writing is always dominant, carrying
the ‘essence’ of meanings, and that other, simultaneously operating modes can merely
expand, exemplify or modify these meanings. This is reflected by fine grained,
moment-to-moment analysis of, e.g., lexis, intonation, rhythm and tone, hesitations
and restarts, alongside more occasional discussion of, for instance, hand movements
or shifts in direction of gaze in talk.
The methodological privileging of particular linguistic resources is also reflected in
notions like ‘non-verbal’, ‘paralinguistic’ or ‘context’. Gumperz (1999), for instance,
defines a ‘contextualisation cue’ as ‘any verbal sign which when processed in co-
occurrence with symbolic grammatical and lexical signs serves to construct the
contextual ground for situated interpretation’ (p. 461), thus treating lexis and grammar
as ‘text’, other ‘verbal’ signs, such as intonation, rhythm and tone as ‘paralinguistic’
or ‘context’, and any other ‘non-verbal’ sign is either treated as context or placed
beyond the scope of the analysis. Other studies foreground the significance of
particular modes (e.g. pointing in linguistic anthropology), and transcribe them in
conjunction with speech or writing. Yet other studies attend to a wide range of
different modes and their mutually modifying effect, emphasizing their different
potentials and constraints and essentially moving towards a semiotic perspective on
representation and communication.
The multimodal scope of social semiotics means that it can attend to forms of
meaning making which remain largely unattended in Pragmatics, notably in contexts
where people are not physically co-present and where speech is not involved, such as
a child drawing. Where the empirical domains of Pragmatics and Social Semiotics
overlap, the two provide distinctive analyses, using concepts derived from their
different perspective on meaning making and multimodality. For Social Semiotics,
these concepts include the sign, semiotic resource, mode, affordance and
orchestration. These concepts are discussed below.

Key Concepts
This section gives an overview of key concepts in social semiotics. Using illustrative
examples where appropriate, we discuss the notion of the sign, semiotic resource,
mode, affordance and multimodal orchestration.

Sign
The notion of the sign is borrowed from traditional semiotics. Signs are elements in
which the signified (‘meaning’) and signifier (‘form’) have been brought together.
Social semiotics holds that the process of sign-making is subject to the interest of
sign-makers, their availability of semiotic resources and the aptness of those

3
resources to the meanings which they wish to realize. That is to say, the relation
between ‘form’ and ‘meaning’ is not arbitrary but motivated (Kress, 1993).
In (mediated) communication, signs are made and remade, and from a social
semiotic perspective signs are always newly made in a specific environment and
according to the interests of the sign makers interests. In other words, “Signs are
made – not used – by a sign-maker who brings meaning into an apt conjunction with a
form, a selection/choice shaped by the signmaker’s interest’, interests that are shaped
by the environment and circumstances of use” (Kress, forthcoming).

Semiotic resource
As social semiotics evolved there was a growing emphasis on how the context of
communication and the sign maker shaped signs and meaning. It moved towards a
more flexible notion of grammar, with a focus on people’s situated choice of
resources rather than emphasizing the system of available resources. Semiotic
resource is central to multimodality although it features slightly differently within
particular approaches. Kress and van Leeuwen suggest that a semiotic resource can be
thought of as the connection between representational resources and what people do
with them. Van Leeuwen describes semiotic resource as follows:

Semiotic resources are the actions, materials and artifacts we use for
communicative purposes, whether produced physiologically – for example, with
our vocal apparatus, the muscles we use to make facial expressions and gestures
– or technologically – for example, with pen and ink, or computer hardware and
software – together with the ways in which these resources can be organized.
Semiotic resources have a meaning potential, based on their past uses, and a set
of affordances based on their possible uses, and these will be actualized in
concrete social contexts where their use is subject to some form of semiotic
regime (van Leeuwen, 2005:285).

The emphasis on rules within social semiotics is on rules as socially made and
changeable through social interaction (van Leeuwen, 2005). Van Leeuwen also notes
that the notion of ‘resource’ has begun to replace the notion of ‘sign’.
The concept of semiotic resource offers a different starting point for thinking about
semiotic systems and the role of the sign maker in the process of making meaning. In
this perspective signs are a product of a social process of sign making. A person (sign
maker) ‘chooses’ a semiotic resource from an available system of resources. They
bring together a semiotic resource (a signifier) with the meaning (the signified) that
they want to express. In other words people express meanings through their selection
from the semiotic resources that are available to them in a particular moment:
meaning is choice from a system. But this choice is always socially located and
regulated, both with respect to what resources are made available to whom, and the
discourses that regulate and shape how modes are used by people. There are various
kinds of normative discourses for how we use semiotic resources – sometimes more,
sometimes less binding, and of different kinds, but nonetheless they do provide ‘rules’
for their use. Discourses of gender, social class, race, generation, institutional norms
and other articulations of power shape and regulate people’s use of semiotic
resources. These are not ‘codes’ in the sense that they cannot be changed and that
they ‘are simply there’ – but they are social rules (van Leeuwen, 2005).

Mode

4
A mode is often defined as a set of socially and culturally shaped resources for
making meaning. A number of different criteria can be considered to establish
whether ‘a set of resources’ counts as a mode or not. The most common ‘test’ is,
following Halliday, the meta-function test. That is, if it can be shown that the set can
serve all three meta-functions, it is regarded a mode. For instance, the resources of
‘colour’ can be used to represent what the world is like (cf. the gardener’s green
dress), to establish social relations (cf. the doctor’s white dress) and to create
coherence (cf. the background colour against which different images are set in a
magazine). However this ‘test’ only works if a particular community of users of the
mode have been identified. ‘Typeface’, for instance, may not qualify as a mode among
archeologists; that is, they may not have access to all the resources of ‘font’ and
therefore be unable to use font to serve all three metafunctions. Typesetters, on the
other hand, or their contemporary counterparts, graphic designers, are likely to have a
sense of the potentials of ‘type’. Indeed, their professional identity rests on that sense.
Becoming a member of a community, social group, or ‘school’ means becoming an
apt user of the modes used within that community. The identification of a community
of users suggests that in order for a form of meaning to be treated as a mode there
needs to be a shared understanding of those forms of meaning making. That leads to
another ‘test’, the community test: the words, or gestures, produced by an individual
may be meaningful to the individual, but they are not necessarily shared by a
particular, socially and culturally situated group. Inasmuch as they are, they can be
treated as belonging to a particular ‘mode’; inasmuch as they are not, they can be
treated as idiosyncratic –yet no less meaningful and therefore of no less interest to the
social semiotician- forms of meaning making. In other words, in order for a set of
resources to count as a ‘mode’ its resources must have come to display regularities.
These regularities are the outcome of the historical, social and cultural use of material
forms (sounds, parts of the body, canvas, et cetera).

Material, social and cultural affordance

The term affordances is contested and continuously debated. It has particular
emphasis and currency in social semiotic approaches to multimodality (see Jewitt,
2009). Modal affordance is used to refer to what it is possible to express and represent
easily with a mode (Kress 1993). He positions affordance as a concept connected to
both the material and the cultural, and social historical use of a mode. In other words,
the affordance of a mode is shaped by what it has been repeatedly used to mean and
do its ‘provenance’), and the social conventions that inform its use in context. Each
mode (as it is realized in a particular social context) possesses a specific logic and
provides different communicational and representational potentials. The logic of
sequence in time is unavoidable for speech: one sound is uttered after another, one
word after another, one syntactic and textual element after another. This sequence
becomes an affordance or meaning potential: it produces the possibilities for putting
things first or last, or somewhere else in a sequence. The mode of speech is therefore
strongly governed by the logic of time. In contrast, (still) images are more strongly
governed by the logic of space and simultaneity. Like all governing principles they do
not hold in all contexts and are realized through the complex interaction of the social
as material and vice versa – in this sense the material constitutes the social and vice
versa.

Multimodal orchestration

5
The meaning of any message is distributed across different modes and not necessarily
evenly. The different aspects of meaning are carried in different ways by each of the
modes in the ensemble. Any one mode in that ensemble is carrying a part of the
message only: each mode is therefore partial in relation to the whole of the meaning
and speech and writing are no exception. Multimodal research attends to the interplay
between modes to look at the specific work of each mode and how each mode
interacts with and contributes to the others in the multimodal ensemble. At times the
meaning realized by two modes can be ‘aligned’, at other times they may be
complementary and at other times each mode may be used to refer to distinct aspects
of meaning and be contradictory, or in tension. As Lemke has stated (2002: 303):
No [written] text is an image. No image or visual representation means in all
and only the same ways that some text can mean. It is this essential
incommensurability that enables genuine new meanings to be made from the
combinations of modalities.
The relationships between modes as they are orchestrated in interactions (and texts)
may realize tensions between the aspects of meaning in a text. This kind of tension
can itself be meaningful and a means for encouraging reflection and critique. The
structure of a text and hyperlinks realize connections and disconnections between
screens. These contribute to the expansion of meaning relations between elements.
The question of what to attend to, what to ‘make meaningful’ is a significant aspect of
the work of making meaning. In other words, the task of what to attend to and to
select as salient to the task at hand is amplified by a multimodal focus.

Analytical Focus
Social semiotics aims to understand how people use modes available to them in
particular social situations. That involves the collection of records of meaning making
-texts, artifacts, video recordings- which capture the semiotic work involved in those
situations. The analysis proceeds by attending to the following aspects.

Meaning makers and modes

The meaning makers and the modes and the resources of those modes that they have
used need to be identified. In print media, they are likely to include resources of
image (e.g., pictorial detail, depth), writing (e.g., lexis, clause structure), typography
(e.g. type, letter fit), and layout (e.g., spread, grid). In embodied interaction, they are
likely to include speech (e.g. intonation, tone), gesture (e.g. direction, span), gaze
(direction, fixation), and posture. These resources are not necessarily equally
available to all sign makers involved. Speech, for instance is often available to only
one sign maker at the time; but gaze is not. In other contexts the use of modes is
distributed along professional lines: the sign makers involved in the production of
print media, for instance, have specialized in writing (‘authors’), image (‘illustrators’,
‘picture editors’) and layout (‘graphic designers’). A growing body of literature is
available detailing the use of these resources in a variety of social contexts. Crucially,
social semiotics aims to understand how sign makers exploit the potentials of these
resources to articulate the meanings they wish to express.

Chains of semiosis
In representing the world, ‘translations’ are constantly made of meanings made in one
mode or ensemble of modes to meanings made in another mode or ensembles of
modes. Such ‘translations’ are inevitable because on the one hand social

6
environments are changed in recontextualization and on the other hand the available
modes and media and their affordances are constrained. Transduction inevitably
brings profound changes in the move from one mode to the other. Given the
difference in material and histories of social and cultural work, there can never be a
‘perfect’ translation from one mode to another. Assume two characters were
described in writing. The author might have given a written description like, “Sitting
in the late autumn sunshine, Sam and Bill share a bench in the park”. An illustrator or
designer might have been asked to ‘draw across’, to transduct the written description
into the mode of image. Now the illustrator has to ask ‘how close to each other were
they sitting?’ ‘was Bill to the left or to the right of Sam?’ The translator/transductor
has to become precise, whether she or he wishes to do so or not.
While transduction describes changes involving a change in mode, transformation
describes changes in arrangement within one mode. Transformations are operations
on structures within the one mode, in which entities remain the same while structures
change. In a transformation, say within the mode of writing, words remain,
syntactic/grammatical categories remain those of the mode, as do textual
arrangements. What changes is their arrangement. In transduction, the change from
one mode to another, brings with it a change of entities. There are no words in image,
there are depictions; semiotic/semantic relations which in speech or writing are
expressed in clauses and as verbs are realized through ‘vectors’ or lines. Other
semiotic relations between lexical-syntactic elements –prepositions for instance (on,
over, by, etc)– are realized by spatial means in images; and so on. Transformation and
transduction are ubiquitous semiotic processes. It is the task of the analyst to identify
them and ask what was ‘gained’ and what was ‘lost’ as signs are remade to form
chains of semiosis.

Inter-semiotic relations and multimodal orchestration

The relationships across and between modes in multimodal texts and interaction are a
central area of interest for multimodal research. Where the modes of image, writing
and layout are involved this modal co-operation is often captured by the notion of
‘design’ and ‘designing’. Where the modes of speech, gesture and gaze are involved
this is often captured by the notion of ‘ensemble’ and ‘orchestrating’. The task of the
analyst may be seen to be to reconstruct what the distinctive contributions are of the
various modes to a modal configuration.
Linguistic routes through social semiotics and multimodality, modal relations have
been investigated by applying inventories from Hallidayan linguistics. That leads to
modal relations being defined in terms of their ‘iterative’ relation (such as synonymy,
antonymy, meronymy), ‘logico-semantics’ (e.g. one clause expanding another) or
cohesion (e.g. reference). There are some difficulties attached to this approach. First,
it assumes that the relations which can be drawn between signs made in one mode,
namely speech, can also be found in other modes; in other words that the types of
relations between signs is stable across modes. Second, it assumes that the levels at
which the semiotic features of modes operate in the different modes are the same.

Fields of application
Despite the close association between social semiotics and multimodality strictly
speaking these are two distinct and differently orientated terms - social semiotics is a
theory of meaning, and multimodality is a field of application, or a perspective on
resources for making meaning. Multimodality opened the door to forms of
collaboration with a range of disciplines also concerned with looking at

7
communication and interaction beyond language, including pragmatics, sociology,
anthropology and psychology. In the first decade of the 21st century, this joint
enterprise was focused on technology mediated interaction, pedagogy, ‘literacy’,
design, and increasingly, fields such as literary studies, language teaching, and
identity studies. The Routledge Handbook of Multimodal Analysis (Jewitt, 2009)
provides a detailed account of the history of multimodality as a field of research.
Multimodal research has been conducted on the technologization of practices and
communication and interaction more generally. Much of this work explores and
theorizes the nature of image and writing relations in narratives, relationships between
book and computer based versions of texts, and the role of on-line communities of
various kinds in the critique, as well as the interpretation and generation of new forms
of multimodal and digital narratives and literacy practices. This work often describes
new forms of literacy in an attempt to remap the territory of communication in a
contemporary context and the kinds of practices that help move across it. The visual
character of writing comes to the fore on screen, for instance, to function as objects of
literacy in fundamentally different ways than it does on the page and this has been a
topic of multimodal research.
From early 2000, after the New London Group (1996) made the call to understand
knowledge and pedagogy as multimodal, there has been an explosion of interest in
multimodality within research and this perspective has been actively taken up by
educational researchers leading to substantial work that looks at multimodal meaning
making across a wide range of sites. Substantial multimodal research has been
undertaken in pre-school and early years contexts, with a focus on multimodal
meaning making practices. Science education has proven to be a productive site for
multimodal investigations into the construction of knowledge across a range of
resources, as has mathematics education, Music education and school English and
Media education. Multimodal studies in Higher Education and in Professional settings
have also examined pedagogic strategies and learning. Taken as a whole, these
multimodal studies show that significant pedagogic work is realized through a range
of modes. The need to rethink what it means to learn and to have capacities to engage
fully in contemporary communication be is a thread that runs through much
multimodal research.
How identities are articulated through multimodal means is an area that has
attracted some attention within multimodal research. Much of the work on what might
broadly be called literacy practices from a multimodal perspective is concerned with
the production of identities.

References
Fairclough, N. (2003). Analysing Discourse. Textual analysis for social research.
London: Routledge.
Gumperz, J.J. (1999), ‘On interactional sociolinguistic method’, in S. Saranghi and C.
Roberts (eds), Talk, work and institutional order. Discourse in medical, mediation
and management settings. Berlin, New York: Mouton de Gruyter, pp. 453-471.
Halliday, M. (1978), Language as Social Semiotic. London: Edward Arnold.
Hodge, R. and Kress, G. (1988), Social Semiotics. Cambridge: Polity Press.
Hodge, R. (2009). Social Semiotics. In: P. Bouissac (ed.), Semiotics Encyclopedia
Online. http://www.semioticon.com/seo/S/social_semiotics.html#. Retrieved 15
February 2009.
Jewitt, C. (ed) (2009). The Routledge Handbook for Multimodal Analysis. London:
Routledge.

8
 Kress, G. (1993). Against Arbitrairiness: the social production of the sign as a
foundational issue in critical discourse analysis. Discourse and Society 4, 2, 169-
193.
Kress, G. & R. Hodge (1979), Language as Ideology. London: Routledge.
Kress, G and van Leeuwen, T (1996), Reading Images: The Grammar Of Visual
Design. London: Routledge.
Kress, G. and van Leeuwen, T. (2001), Multimodal Discourse. The Modes and Media
of Contemporary Communication. London: Edward Arnold.
Kress, G. (forthcoming). Multimodality. London: Routledge.
Lemke, J. (2002). Travels in Hypermodality. Visual Communication 1, 3, 299-325.
New London Group (1996). A pedagogy of multiliteracies. Designing social futures.
Harvard Educational Review 66, 60-92.
O’Toole, M. (1994). The Language of Displayed Art. London: Leicester University
Press.
van Leeuwen, T. (2005), Introducing Social Semiotics. London: Routledge.

View publication stats