Professional Documents
Culture Documents
Monitoring and
Evaluation
Evidencing Effective
Communications
within an Organisation
NB This article is about evaluating client and stakeholder (internal and external)
communications. Take into consideration that M&E techniques for these purposes are
different to research about how customers/consumers perceive the service they have
received from the company. (Consumer research article to follow shortly).
Part one in the Communications Strategy series already outlines some of the
techniques you can use to establish what M&E your company may already have
in place. Monitoring and Evaluation can be a lengthy process requiring
considerable amounts of time and resources, however there are ways in which
you can minimise cost and time by looking at what you currently have and using
existing data to build a foundation or baseline. From this point, you can look at
how you present the data and plan how you will compare data in the future.
Depending on the software and the expertise of your digital/web team, (if you
have one) it should be fairly easy to look at what people are reading on your
internal and external channels. For example, you may have an external blog in
place and want to know how many people have read it over a certain
period of time. Or you have a regular e-newsletter that you send to a certain
stakeholder group and you would like to assess whether it is being read in order
to consider changes to the channel or the content. You may want to show how
readership is rising on a particular channel and monitor whether there are any
particular trends and the potential reasons for this.
You may also want to conduct a review of your channels and present information
in a different way and M&E will help to inform this process. If a blog on your
external channel is not being read, is it because it hasnt been promoted
effectively? How are people reaching the blog? And are you optimising the
digital space by linking in from other areas (if appropriate). If the newsletter you
are disseminating is only being read by 25% of people on the database, is your
stakeholder database up to date? How many bounce backs are you receiving
and is the information still relevant for the target group? This kind of data paints
a picture and is invaluable in looking at how you plan communications and what
changes you will need to make to ensure that you are optimising your channels
and content.
Qualitative V Quantitative
There is no battle between these 2 types of research. Each is beneficial in its own
way and the type of research you choose depends on your M&E planning and
the objectives of your research. I like to use a combination of both, however this
may not be necessary and depends on how you choose to evaluate marketing
and communications in your organisation. Lets break down the 2 different types
of research and explore how you would use each type to support your Internal
and External communications planning. For the purpose of comparison, the
definitions given on the snap survey website give a comprehensive explanation
of both research types. NB In the case study example on the right, quantitative research
has been used as a baseline for further exploration using qualitative research - The definition
below suggests that this order is reversed, but (in my opinion) the order of research - if you are
using both- is interchangeable.
So, when would you use Quantitative and Qualitative research to inform your
communications planning? A few examples are provided in the following
case studies.
Case Study 1
Research Type - Quantitative
Marketing and Communications role (Internal)
You want to run a quick survey to gain an initial impression about
perceptions of employee communications. You want to find out how well
people feel they are being informed and if they understand the objectives of
the business. You decide that you will create a 4 question survey with a view
to gathering responses and following up at a later date to observe whether
perceptions have improved after you have made some changes to internal
channels and content. Due to limited resources and time, you decide to use
outlook software to create a questionnaire and send to all staff. You pose
the following questions;
1. On a Scale of 1-10, rate the last internal newsletter content
(1 being poor and 10 being optimal)
2. Rate how well you feel informed about the organisations values;
Very Informed
Quite Informed
Could be better informed
Not informed at all.
3. How often do you access the intranet?
More than once a day
Once a day
Once every couple of days
Once a week
Never
4. On a scale of 1-10 how do you rate Internal communications?
(1 being poor and 10 being optimal )
2. Rate how well you feel informed about the organisations values;
Very Informed
Quite Informed
Could be better informed
Not informed at all.
If you answered could be better informed or not informed at all, what could we do
better to keep you informed
Looking at case study 1, we can see that the initial 4 question survey would give
us a snapshot of perceptions, opinions and preferences, but wouldnt tell us
much more. Allowing respondents to give reasons for their answers, we gain
further insight into motivational factors that encourage people to access certain
channels and a deeper understanding about the motivations behind the answers
to the quantitative questions. This gives your research another layer and added
depth.
Quick surveys can be beneficial if you have recently conducted a larger
evaluation project and want to follow up on areas of interest. They are also
useful as a starting point for deeper evaluation (especially if time is pressing).
The next case study observes different data gathering techniques using
quantitative and qualitative research methods in a new scenario.
Case Study 2
Research Type - Qualitative
Marketing and Communications Role - Social
Media Strategist
You have been asked to conduct a social media review in order to find out
how people are using social media channels in your company, what
comments they are making, where they are from and how the organisation
is responding. Your organisation wants to make sure that they are using all
the channels appropriately and optimising social media in order to engage
with stakeholders effectively. Your organisation would also like to know how
other organisations are using their social media channels and would like you
to write a document rationalising and comparing social media use in your
organisation. You have a digital team who you lead and manage and you
have decided to draw on their direct knowledge and expertise in order to
gather data for analysis to inform your paper.
You establish a time limited working project group inviting members of your
digital team to give their initial impressions of the current use of social
media in the organisation. In the first meeting, you ascertain what channels
are being used and note your teams opinions about what improvements the
business could make to enhance their social media presence. You ensure that
the meeting is structured and have a clear idea about what questions you
want to ask. (The qualitative data comes from the notes you make from your
project group). You also decide to follow up using quantitative
questions via a survey sent to your working group for further analysis.
You begin to make decisions about how to structure the data and start by clearly
showing what current social media channels your business uses, differentiating
between internal and external channels. You create a table showing the following
information.
Social Media Channels -External e.g.
Twitter
Facebook
Blogs (embedded into website)
Social Media Channels -Internal e.g.
Yammer
Forums
Blogs
You breakdown how often people are accessing the channel and ask your digital
team to provide you with the relevant data over a specific time frame. You also
request historical data in order to chart the progression of social media in your
business and look for trends in how it has been accessed over a period of time.
You also want to know what people are saying on these channels and ask your
team to compile data on all social media channels highlighting what people are
saying over a period of time.
Taking a thematic approach for ease of reference, you keep a note of all comments
using a spreadsheet. You decide to create a bar chart with the themes you have
developed in order to clearly show dominant topics and areas of interest. You also
make note of responses to comments given by the business you work for.
You find a comparable business (or ask you digital team to suggest one) and
observe their social media channels, looking at how they are being used in order to
draw a comparison using the same baseline criterion you have set for observing
your companies social media.
You review the notes from your digital working group about perceptions of social
media and look at the quantitative data you have gathered in order to write your
paper.
(All the above are quantitative data gathering techniques using existing and historical data
through systematic observation
What Software
There are many types of software you can use to develop a survey and again
depends on the size of the organisation, the amount of time you have to dedicate to M&E and your budget. If you want to conduct a quick survey, it is likely
that you have the tools in-house to do so.
Outlook
Dependent on your version of outlook,
you can create surveys using your email
application. This is beneficial if you use this
program as the database of respondents is
most likely embedded into the same
application. If you are pressed for time
and have limited resources. Outlook is
useful in developing a basic template.
Creating Polls in Outlook
Survey Monkey
Survey Monkey is simple to use and comes with a number of customisable options for survey and questionnaire creation. Dependent on your organisation,
you may find that a particular department already has the license to use this
software. You should perhaps do a little research and ask around before you
purchase a plan. There are tips on how to create a survey, as well as clear
instructions on how to use the software.
Survey Monkey
Share Point
Ive worked in organisations who are moving over to SharePoint. Im not sure
how prevalent this is, but it does come with a lot of integrated features which
allow you to create surveys.
Sharepoint
External agencies and consultants (software)
If you are working with external agencies or consultants to develop your M&E
plan, they are likely come with their own software for survey creation and
analysis. Ill outline how to work effectively with external agencies and
consultants further on in the article.
There are many more options and some of them come with a free trial, or have
certain features that you can use for free if you dont require everything the
software has to offer.
Gaining Feedback
Most of the software mentioned above gives you the option of developing your
own template and criterion and its important that you plan what questions you
want to ask in line with your objectives.
You should also take your audience into consideration when planning your
survey and developing your template. It is perhaps a lot easier to garner interest
in completing surveys with your internal audience. You should think about
approaching your external audience in a different way.
Questionnaires and surveys can be obtrusive and time consuming and whilst you
can gain the support of your internal audience by effective engagement techniques, think carefully about how you conduct research with your
external stakeholders. Again, dependent on what type of organisation you work
for, questionnaires can be embedded in different channels.
If you are evaluating communications, e.g. how effective are your channels,
marketing material, website etc, take into consideration that this area of
research is different to how customers/clients perceive the service they have
received from the company and you will need to use a different approach. This
is out of the scope of this article. (Consumer Research article to follow)
Anonymity Assurance
Write a brief.
Your brief should be clear and succinct, letting the agency know what you want
to find out through your research and how this will add value to communications
and development in your business. If your external agency is going to analyse
the data for you, make it clear how you want this to be presented, e.g. do you
want bar charts, graphs, inforgraphics, interactive elements, film and/or a
summary of findings to illustrate the data? Let them know if you intend to
embed the data into your communications planning and ensure that there is
scope to compare findings at a later stage.
Meetings
Try to conduct a face to face meeting with your external agency in the first
instance. Its been my experience that establishing a relationship in this way sets
a good foundation for collaboration and putting a name to a face is invaluable.
Make sure you discuss how you both intend to monitor progression and
development of the M&E plan and make sure you are happy with the
format of any questionnaires.
Contact
Ensure that the agency know the right people to contact at various stages of
development. You may be the initiator or lead for the project, but have trusted
members of a team that will carry the work forward whilst you focus on other
activities. Be clear about who is involved from the outset and ensure that a
process is put in place so you are kept updated at key stages throughout
development of the M&E plan.
Networks, forums and working groups developed specifically for the purpose of
gaining insight into employee perceptions are a nice way to engage with people
in your organisation.
You have reached the stage where you have developed your plan, created a
questionnaires and/or surveys template, gained feedback from responders and
now have the raw data ready to structure and present. What next?
Depending on whether you have conducted a deep dive or have decided to gain
an initial first impression by running a quick survey, if your organisation has not
really focussed on M&E for communications, or started it at some point and not
really kept up with the process, remember that you will want to set a baseline for
further research and exploration.
If you are continuing M&E efforts and already have a baseline for the purpose of
data comparison, then the task ahead is simply a matter of comparing historical
data with new data. If you are starting from scratch, its important that you build
a solid foundation.
Charts
Perhaps the simplest way to present quantitative data is by using charts and
graphs to show comparisons. A majority of survey software will actually
generate this for you depending on what you input. If you have integrated your
research techniques and have a combination of qualitative and quantitative
methods, you may want to look at taking a thematic approach to analysing your
qualitative data.
If you have asked people to respond with an opinion rather than scale, you can
look through the answers you have received and search for denominators.
People may have expressed a concern about knowing or understanding the
mission of the business or have requested new content in your employee
newsletter. The responses may be similar, but people will have expressed the
same opinion in different ways using different language.
If you can make your presentation engaging and specific, you will find that the
data you have collected and subsequent analysis comes to life. If you can then
connect this with your plans for communications and highlight why the data will
inform changes you intend to make based on the targets and responses, you will
find that a picture begins to take shape.
If you have planned your research well, you should find that there is a seamless
integration between your qualitative and quantitative data. How you choose to
present this depends on what you want to show, but remember to make
connections that support your strategy. Make sure that you create clear headings,
state the purpose of you M&E plan and highlight the standards, targets and
objectives you are aiming to achieve. Remember to set a specific time frame for
further comparisons to evidence progress.
You may want to give a presentation showing your findings and highlighting the
most important or relevant section of the analysis. Statistics can be interesting
and it all depends on how you present them. If you are reading from a sheet and
simply regurgitating dry statistics with no context, whats the point?