Effective Vaccine Management Initiative

EVM Assessment Tool

Country: Indonesia
Facility Name: DINAS KESEHATAN SUMENEP (1LD01)

Release notes

416188405.xls 1 of 23 04/11/2019
DINAS KESEHATAN SUMENEP
Note to assessors: Obtain as much of this information as possible before the on-site inspection begins.

L1 Assessment scope: Full 1LD

L2 Facility details

L2.01 Supply chain level: Lowest distribution level L2.02 Administrative area: SUMENEP

L2.03 Facility name: DINAS KESEHATAN SUMENEP L2.04 Unique code for this facility: 1LD01

L2.05 Facility address: Jl. Dr Cipto No 33

L2.06 Telephone: (0328) 66122-672270 L2.07 Fax number: 665373-672270

L2.08 Email: Dinkes.Sumenep@telkom.net L2.09 Primary contact: 82139494920

L2.10 Maximum temperature at site °C: 32 L2.11 Minimum temperature at site °C: 25

L2.12 Facility assessment start date: 07 may 2018 L2.13 Facility assessment finish date: 08 May 2018

L2.14 Assessed records start date: 07 may 2018 L2.15 Assessed records finish date: 08 May 2018

L2.16 Assessment team members: Team Dinas Kesehatan

L2.17 Target eligibles: 14,222 L2.18 Target population: 108,255

L3 Permanently employed staff in the facility

List permanently employed members of staff, including managers, storekeepers, administrators, drivers, cleaners, etc.

Name Job title Qualifications (see note 1) Experience (see note 2)

L3.01 Imam Muttaqin,S,KM,M.Kes Kasi Survim S2 Kasi Survim

L3.02 Wishesa Arya Pratama Wasor Imunisasi S1 Wasor

L3.03 Sjamsul Arifin Staf Imunisasi SMA Korim

L3.04 Ratnawati Yuliana Staf Imunisasi D3 Bidan dan Korim

L3.05

1) Qualifications: Specify level of

education: e.g. technical college
vocational qualification, university first
degree etc.
2) Experience: In particular record
previous experience in the field of
vaccine management.

416188405.xls 2 of 23 04/11/2019
 Location Data

On-line climate database

http://www.wolframalpha.com
http://www.weatherbase.com

Years in post
11 Tahun

5 Tahun

20 Tahun

5 Tahun

416188405.xls 3 of 23 04/11/2019
EVM # EVM REQUIREMENTS Q no. EVM QUESTION Result Score Max Guidance notes for assessors C
a
t

E0 Complete these facility-specific questions before continuing s

c
o
0.1 Answers to these questions switch indicators r
below on and off. e
E0:01a What type of cold chain equipment is used in the store? Answers to this question switch later indicators on and off.
[Select one or more options]:

A Freezer room(s) and/or cold room(s) are used for storing N

vaccine [Y, N]
B Freezers and/or refrigerators are used for storing vaccine [Y, Y Including solar refrigerators.
N]
C Freezer room(s), cold room(s), freezers or refrigerators are N Select 'Y' if there is equipment in the store used exclusively for freezing or cooling icepacks.
used for freezing icepacks or cooling cool water packs [Y,N] Ignore fridge/freezers with icepack compartments or freezer rooms/cold rooms that are also used
to store vaccine.
Notes:
E0:03a What type of equipment is used to deliver and/or collect
vaccine at this store?
[Select one or both options]:

A Cold boxes, vaccine carriers and/or other passive containers Y Other passive containers' includes insulated roll containers or palletized containers. These may
[Y,N] be used at primary and sub-national levels.
B Refrigerated vehicles [Y,N] N
Notes:
E0:04a What organization is responsible for vehicle operations at this Enter all options that apply. 'Vehicle' includes motorcycles, cars, trucks, boats and aircraft.
store?
[Select either A, A+B or C only and explain in notes box]:

A Government agency [Y, N] N Transport' means any type of vehicle, including motorcycles, cars, trucks, boats and aircraft.

B Parastatal, PPP or 3PL contractor. [Y, N]
C There is no official transport at this store [Y, N]
Notes:
E0:05a Specify all transport types used for vaccine delivery or
collection at this level. [Select at least one option]:
PPP = public-private partnership. 3PL= third party logistics contractor.
0.00 Official' transport is one or both of A and B
List all transport types: government-operated, parastatal or private.

A Motorcycles [Y, N] Y
B Cars, vans, trucks, refrigerated trucks and other road vehicles
[Y, N]
C Aircraft [Y, N]
D Railway [Y, N]
E Boat [Y, N]
F Other [Y, N. Enter description in notes box]
Notes:

E2 All vaccines and diluents are stored within WHO-recommended temperature ranges.

2.2 Continuous temperature records must be kept,

and these records must demonstrate that
vaccine has been stored correctly in both
permanent and temporary cold stores.

2.2.1 Store all vaccines and diluents at the correct E2:03a Can the storekeeper(s) or health worker(s) give the correct Y 5.00 5.00 Knowledge of correct storage temperatures is vital. If the respondent gives any incorrect ###
temperature storage temperature range for each of the vaccines on the answers, score 'No'. Note also:
schedule? [Y, N]. 1) If s/he says that freeze-dried vaccines should be stored at -20°C record this as 'Yes'. Current
WHO advice is to store these vaccines at +2° to + 8°C, but storage at -20°C will not damage
these vaccines.
2) If s/he say that freeze-dried vaccines packed with diluents can be stored at -20°C, this answer
MUST be scored as 'No'.
3) At service point level, it is correct to say that all vaccines can be stored at +2° to +8°C

Notes:
E2:04a Do the storekeeper(s) or health worker(s) know which Y 5.00 5.00 It is essential that respondents know which of the vaccines should not be exposed to ###
vaccines on the schedule can be damaged by temperatures temperatures below 0°C. It is not important that they know the precise freezing temperature of
below 0°C? [Y, N] each of these vaccines.
Notes:

416188405.xls 4 of 23 04/11/2019
EVM # EVM REQUIREMENTS Q no. EVM QUESTION Result Score Max Guidance notes for assessors C
a
t
E2:05a Has the storekeeper received classroom, or on-the-job, 0.00 1.00 Training should cover the following minimum topics: 1) correct storage temperatures; 2) vaccines###
training in how to look after vaccines? [Y, N] damaged by freezing; 3) vaccines which can be frozen; 4) vaccines which must be frozen; 5) s
VVM reading; 6) shake test; 7) temperature monitoring using the range of available devices; 8) c
storage of diluents; 9) how to arrange vaccines in cold rooms, freezer rooms, refrigerators and o
freezers; 10) icepack conditioning; 11) cold box/vaccine carrier packing. r
e

2.2.3 Maintain complete temperature records for every
freezer room, cold room, vaccine freezer, vaccine
refrigerator and refrigerated vehicle.
Notes:
E2:07a CONDITION: If there is refrigeration equipment
Is there a complete set of manual temperature records for
each and every cold room, freezer room, vaccine refrigerator
and vaccine freezer throughout the review period? [Y, N]
0.00 1.00 This question applies all in health facilities with active refrigeration. It does not apply in health ###
facilities where cold boxes or vaccine carriers are used to store vaccine temporarily.
Manual records are necessary because this is the only way to be sure that temperatures are
checked at least twice a day. 'Complete' means that there should be no missing' record sheets or
electronic temperature traces and that the entire review period should be covered for each
appliance.

Notes:
2.2.4 Keep temperature records in a safe place for a E2:11a CONDITION: If there is refrigeration equipment 0.00 1.00 Countries should be encouraged to retain such records for tracing purposes. If records are not ###
minimum of three years. Have temperature records been kept in a safe place for at available for three years, review the records kept since the immunization programme adopted
least three years? [Y, N] EVSM or EVM.

Notes:
2.2.5 Carry out an internal review of the temperature E2:12a CONDITION: If there is refrigeration equipment There must be a written record of these reviews which identifies problems and records actions
records every month. Temperature review checklist. taken. A verbal assurance is not sufficient.

A Are temperature records formally reviewed at least once a Y Check whether records are reviewed and recorded. 'Excursions' are temperatures below +2°C or
month in order to identify temperature excursions and their above +8°C for cold rooms and vaccine refrigerators, and above -5°C for freezer rooms and
causes? [Y, N] vaccine freezers.
B Have correct remedial actions been taken? [Y, N, n/a if there Y 5.00 5.00 Interview the responsible person. Discuss and evaluate remedial actions taken. ###
are no excursions or breakdowns]
Notes:
Criteria score E2: 15.00
Maximum score: 18.00
Percentage: 83%
E3 Cold storage, dry storage and transport capacity is sufficient to accommodate all vaccines and supplies needed for the programme.

3.1 Storage capacity must be sufficient for

maximum stock levels of routine vaccines and
related consumables. Where supplementary
vaccines and/or other products are also held in
the store it must be able to accomodate these.

3.1.1 Carry out volume calculations for all vaccines, E3:01a CONDITION: If there is refrigeration equipment Collect site data, then use the sidebar tool to perform these calculations.
diluents and droppers that are kept in the vaccine Using data from the annual vaccine arrival schedule (primary
store. level) or the annual vaccine delivery schedule (lower levels),
establish whether the capacity of the +2°C to + 8°C vaccine
store is sufficient over a one year cycle. Where relevant,
include other products and supplementary vaccines.

A Measure the net storage capacity of the +2°C to + 8°C 100.0

store(s) in litres or m³.
B Enter maximum estimated vaccine volume in litres or m³, plus 75.0 5.00 5.00 The store capacity, taking account of utilisation factor, is acceptable if it is greater than the ###
10%. maximum estimated volume of vaccine to be stored, +10%.
Notes:
3.4 Capacity of passive containers and capacity to
produce icepacks/chilled water packs must be
sufficient to meet maximum demand.

3.4.1 Calculate whether the available icepack
freezers/chilled water pack coolers have sufficient
capacity to meet the maximum daily demand for
icepacks/chilled water packs.
E3:09a CONDITION: If cold boxes or other passive containers are Collect site data, then use the sidebar tool to perform these calculations.There must be sufficient
used. freezing or cooling capacity to produce an adequate flow of icepacks/chilled water.
Using data from the vaccine delivery schedule or outreach
session size data, establish whether coolant capacity is
sufficient to meet maximum demand levels

416188405.xls 5 of 23 04/11/2019
 Calculate whether the available icepack
freezers/chilled water pack coolers have sufficient
capacity to meet the maximum daily demand for
icepacks/chilled water packs.
EVM # EVM REQUIREMENTS Q no. EVM QUESTION Result Score Max Guidance notes for assessors C
a
t
A Is there sufficient freezing or cooling capacity to meet Y Icepack freezing or cooling may sometimes take place in another facility. If this is the case, the
maximum demand? [Y, N]. capacity of the other facility needs to be checked. s
B Is there sufficient storage capacity for icepacks/chilled water Y 5.00 5.00 Peak demand' is the maximum weight of icepacks/chilled water packs required in a day. ###
c
packs storage to meet peak demand? [Y, N]. o
Notes: r
e
3.4.2 Calculate whether sufficient passive container E3:10a CONDITION: If cold boxes or other passive containers are Check total vaccine storage capacity of available passive containers and compare against
capacity is available to meet maximum daily used. delivery schedule. If containers are always supplied by the receiving store this question is not
demand. Using data from the vaccine delivery schedule, establish applicable here but must be assessed at the next level down in the supply chain.
whether there are sufficient passive containers to meet
maximum demand levels.

3.5 There must be a contingency plan to protect the
vaccine in an emergency.
3.5.1 Develop a contingency plan designed to deal with
forseeable emergencies at each store.
A Does this store stock passive containers for delivery and/or
collection? [Y, N]
B Are there sufficient passive containers? [Y, N, n/a if containers
are supplied by the receiving store].
Notes:
E3:11a CONDITION: If there is refrigeration equipment.
Vaccine store contingency planning checklist:
N Answer 'Y' if the store keeps a stock of passive containers. If delivery and collection dates are
staggered, the same containers may be used for both purposes.
0.00 0.00 If some containers are in use, check the container inventory to establish how many there should###
be.
Contingency plans are context-specific, but should include the following minimum elements:
1) Identify the major sources of risk (e.g. power failure, flood, earthquake, etc.).
2) Prepare a written contingency plan.
3) Post emergency contact details on the noticeboard in the vaccine store.
4) Staff must know that the universal safe storage temperature for ALL vaccines in an emergency
is +2°C to +8°C.

A Is there a satisfactory SOP which sets out a contingency plan Y 5) Identify at least two alternative cold storage locations. Inspect them to make sure they are
in the event of equipment failure or other emergency? [Y, N]. large enough, well-managed, and can maintain correct temperatures.

B Are emergency contact details posted in the vaccine store? Y 6) Ensure that there is a written agreement with the selected emergency providers. If payment
[Y, N]. will be required, make sure that this can be funded.
C Do staff know what to do in the event of an emergency? [Y, Y 5.00 5.00 7) Regularly rehearse the contingency plan with store staff and the emergency providers. ###
N].
Notes:
Criteria score E3: 15.00
Maximum score: 15.00
Percentage: 100%
E4 Buildings, cold chain equipment and transport systems enable the vaccine and consumables supply chain to function well.

4.1 The store building(s) must be conveniently sited

and the standard of construction must meet
minimum requirements.
4.1.2 The quality of the vaccine store building(s) must E4:03a Vaccine store or health facility building checklist: Vaccine store' in this context includes service point health facilities.
meet minimum requirements.
A Is the vaccine store building suitable for the climate? [Y, N]. N Suitable' means one of the following:
1) Cold or temperate climates: Thermally insulated and heated.
2) Hot, wet climates: Naturally ventilated or air-conditioned. Well protected against rain
penetration.
3) Hot, dry climates: Naturally ventilated or air-conditioned and/or passively protected against
excessive heat build up. Protected against dust penetration.

B Is the roof free of leaks? [Y, N].
C Are the external walls free of severe cracks or other major
damage? [Y, N].
D Are windows and external doors in good condition and secure
(grilles and/or locks)? [Y, N].
E Are floors dry and reasonably level? [Y, N].
F Are there any fire extinguishers and have they been tested in
the past 12 months? [Y, N, n/a].
G Is the electrical system satisfactory? [Y, N, n/a].
Y Roof leaks': Check for damp stains on ceilings.
Y
Y
Y Reasonably level': The floor should not be visibly sloping and it should be smooth enough for
small wheeled pallet trucks.
N/A Fire extinguishers should be tested and certified annually. There should be a label on each one
showing the test date. Fire extinguishers are unlikely to be found in small health facilities, so
enter 'n/a'.
Y Satisfactory' means that wiring, circuit breakers, electrical sockets and light fittings are in good
condition and securely fixed in place. All circuits should be properly earthed. In a full inspection
the earthing of sockets can be checked with a ring main tester. If there are any doubts, a
professional inspection should be carried out.
Service point facilities may have no electricity. If this is the case, score 'n/a'.

H Is the drainage system working (both rainwater and foul Y

drainage)? [Y, N].
416188405.xls 6 of 23 04/11/2019
EVM # EVM REQUIREMENTS Q no. EVM QUESTION Result Score Max Guidance notes for assessors C
a
t
J Is the air-conditioning system working? [Y, N. Score 'n/a' in Y
climates where a/c is not required]. s
K Is the heating system working? [Y, N. Score 'n/a' in climates N/A 4.38 5.00 ###
c
where heating is not required]. o
Notes: r
e
4.3 Equipment must be satisfactory in both
permanent and temporary vaccine stores.
4.3.3 Freezers and refrigerators must comply with the E4:13a CONDITION: If there are refrigerators and/or freezers
listed minimum standards. Vaccine freezer/refrigerator checklist.

A Do all vaccine freezers comply with the WHO specifications Y Check that the equipment's climate rating is correct for the site.
that were in force at date of purchase, including correct
climate zone? [Y, N. Score n/a if no freezers].

B Do all vaccine freezers have continuous temperature Y Continuous temperature recording in freezers is only mandatory at primary store level.
recorders? [Y, N. Score n/a if no freezers].
C Do vaccine freezers have a working thermometer stored with Y All freezers should have a stem or dial thermometer.
the vaccine? [Y, N. Score n/a if no freezers].
D Do all vaccine refrigerators comply with the WHO Y Check that the equipments's climate rating is correct for the site. PIS and PQS pre-qualified
specifications that were in force at date of purchase, including equipment should have a sticker to show the climate rating.
correct climate zone? [Y, N. Score n/a if no refrigerators].

E Are all ice-lined refrigerators fitted with the correct vaccine
storage baskets? [Y, N, n/a]
F Do all vaccine refrigerators have continuous temperature
recorders or refrigerator loggers or freeze indicators? [Y, N,
n/a].
G Do all vaccine refrigerators have a working thermometer
stored with the vaccine? [Y, N. Score n/a if no refrigerators].
Y ILRs must always be fitted with baskets so that vaccine does not come into contact with the
bottom or walls of the unit because this can freeze vaccine.
Y Continuous temperature recorders or refrigerator loggers (FridgeTag type) are essential at
primary level. Refrigerator loggers are desirable at SN, LD and SP levels.
Y

H Where applicable, are there sufficient reserve supplies of N/A Sufficient' reserve supplies means at least one spare jerrycan of kerosene or at least one spare
kerosene and/or gas? [Y, N, n/a]. full gas bottle.
J Where applicable, is kerosene or gas always available? [Y, N, N/A
n/a].
K Cold climates only. Do vaccine refrigerators EITHER have N/A 5.00 5.00 ###
low temperature protection OR are they located in a
permanently heated room? [Y, N. Score 'n/a' if low
temperature protection is not required].

4.3.4 The use of CFC gases in refrigeration equipment
must be phased out in accordance with
UNICEF/WHO policy.
Notes:
E4:14a CONDITION: If there is refrigeration equipment
Inspect refrigeration equipment:
A How many refrigeration units are there (dual CR and FR units
count as 2, refrigerators and freezers count as 1 each)?

B How many of these are CFC-free? 0.00 1.00 ###

4.3.5 In primary and large sub-national vaccine stores
there must be a standby generator, with automatic
start-up. Preferably the generator should serve the
vaccine store alone.
Notes:
E4:15a Standby generator. There should always be a standby generator at the primary store unless mains electricity is
completely reliable and/or has a backup supply from a separate transformer. The same
requirement applies at sub-national stores equipped with cold rooms. Generators are not
required in small sub-national stores, lowest distribution level and service point stores where ice-
lined refrigerators are used and there is at least 8 hours electricity per day. Generators are also
not required where kerosene, gas or solar refrigerators are used or where a health facility uses
cold boxes or vaccine carriers for storing vaccine.

A Is a generator required at this store? [Y, N] Y

B Is a standby generator installed? [Y, N] N 0.00 5.00 If a generator is installed but is not required then the generator questions are switched off and ###
need not be answered.
Notes:
E4:16a CONDITION: If a generator is required and installed. The sidebar tool can be used to help with these calculations.
Standby generator checklist:

A Is the generator in working order? [Y, N]. Y

416188405.xls 7 of 23 04/11/2019
EVM # EVM REQUIREMENTS
4.3.6 Provide voltage regulators for all refrigeration
equipment wherever voltage fluctuations exceed +/-
15% of rated voltage (or the refrigeration equipment
manufacturer's voltage tolerance, whichever is
lower).
4.3.8 National telecommunications links (including mobile E4:21a
networks) must be sufficient to manage vaccine
arrivals, delveries and other vaccine management
issues.
4.4 Transport and transport containers used to
distribute vaccines down to the next level in the
cold chain must be satisfactory.
4.4.1 Provide a reliable and sufficient transport
infrastructure at all locations where vaccines and
consumables need to be distributed down the
supply chain.
4.4.2 Provide suitable transport containers to protect
vaccines during distribution.
416188405.xls
Q no. EVM QUESTION Result Score Max Guidance notes for assessors C
a
t
B Can the generator start all the connected equipment in the N The starting load for a refrigeration compressor is much higher than the running load. Ask for a
vaccine store? [Y, N]. demonstration to prove that the generator is able to cope with the combined starting load of all s
connected refrigeration units. Well-designed systems may have 'smart switching' which reduces c
the starting load by automatically phasing in the individul refrigeration units. o
r
e
C Is the fuel tank large enough? [Y, N]. 72 hours continuous running is recommended, but this is context-dependent. If power cuts are
short and/or fuel is readily available, a smaller capacity will be acceptable.
The way to assess this is as follows: 1) Establish the longest power cut experienced during the
review period. 2) Check the generator fuel consumption in litres per hour. 3) Check the capacity
of the fuel tank. If tank capacity/consumption per hour is less than the longest power cut then
the tank capacity is too small.
D Are there sufficient reserve supplies of fuel? [Y, N].
E Is the generator in a secure compound? [Y, N]. 0.00 0.00 ###
Notes:
E4:17a Is the voltage fluctuation sufficiently severe to require voltage N Voltage regulation equipment on all refrigeration equipment is essential if voltage fluctuations
regulation equipment? [Y, N, n/a if solar refrigeration or cold exceed ±15%. Obtain information from an electrical engineer, or from the local supply company.
boxes/vaccine carriers are used to store vaccine].
Notes:
E4:18a CONDITION: If voltage regulation is required. If there is no refrigeration equipment in a particular category - e.g. 'freezer room', enter 0 for the
Voltage regulator checklist. equipment number and 0 for the voltage regulator number.
A How many freezer rooms are there?
B How many are attached to a functioning voltage regulator? There may be a single central voltage regulator for the vaccine store or some sharing
arrangement across multiple appliances. In this case score on the basis that every piece of
equipment has a regulator and make a note in the comment box.
C How many cold rooms are there?
D How many are attached to a functioning voltage regulator of
the correct type?
E How many vaccine freezers and refrigerators are there?
F How many are attached to a functioning voltage regulator of 0.00 0.00 Note: Solar refrigerators do not require voltage regulators. Absorption refrigerators require a ###
the correct type? different type of voltage regulator to those used for compression refrigerators. This needs to be
checked during the inspection.
Notes:
Are telecommunications links working? [Y, N Score 'N' if there N 0.00 1.00 Telecommunications links include telephone, internet and radio. At least one of these must be ###
is no provision]. working well.
Notes:
E4:22a CONDITION: If vaccine transport is directly operated by this This checklist is written with road vehicles in mind. Some programmes may own and operate
store. boats. If this situation is encountered, evaluate boats using this indicator. If transport is provided
General vehicle checklist (includes refrigerated vehicles): by a parastatal or private sector contractor try to inspect their vehicles, especially if refrigerated
vehicles are used.
A Is every vehicle, for which the programme is responsible, in N Ignore vehicles that are scheduled for disposal.
good mechanical condition? [Y, N].
B Does every vehicle, for which the programme is responsible, N
have its own user logbook? [Y, N].
C Does every vehicle, for which the programme is responsible, N 0.00 5.00 ###
have an up to date maintenance and service record (including
a record of refrigeration unit servicing where applicable)? [Y,
N].
Notes:
E4:24a Transport container checklist:
A CONDITION: If passive containers are used. N Passive containers include cold boxes, vaccine carriers and palletised or roll containers. Identify
Do passive containers used at this level comply with WHO products by using the PQS or PIS catalogue.
specifications? [Y, N]
B CONDITION: If refrigerated vehicles are used. 0.00 1.00 No standard set. Use judgement to assess quality. Most important is the way in which cartons or###
Refrigerated vehicles: are the disposable cartons or reusable crates are packed and restrained.
crates used to transport vaccine fit for purpose. [Y, N]
8 of 23 04/11/2019
EVM # EVM REQUIREMENTS Q no. EVM QUESTION Result Score Max Guidance notes for assessors C
a
t
Notes:
Criteria score E4: 9.38 s
Maximum score: 23.00 c
Percentage: 41% o
r
E5 Maintenance of buildings, cold chain equipment and vehicles is satisfactory. e

5.1 Planned preventive maintenance to buildings,

equipment and transport must be carried out.

5.1.1 Buildings: Establish a planned preventive
maintenance programme and provide evidence that
this plan is being followed.
5.1.2 Equipment: Establish a planned preventive
maintenance programme and provide evidence that
this plan is being followed.
E5:01a Planned preventive maintenance checklist for buildings: This question is primarily about planned preventive maintenance (PPM). However, it is possible
that the building is in good condition because it is being informally maintained. 01:C covers this
option.
A Is there a written multi-year planned preventive maintenance Y An effective planned preventive maintenance programme for buildings should cover several
(PPM) programme. [Y, N] years into the future and should set target dates for routine maintenance such as re-decoration,
electrical safety inspections, fire-extinguisher maintenance, etc.
B Is there documentary evidence that the programme is being If the score to question A is 'N' then the score for Question B must also be 'N'. PPM activities
followed. [Y, N] should be monitored and recorded.
C Is there visual evidence that maintenance is taking place. [Y, N 1.67 5.00 If an effective maintenance programme is in place, the building should be in good condition. ###
N] Indicators of poor building maintenance include broken light bulbs, leaks and damp stains,
blocked drains, flaking paintwork, etc. See answers to E4:04a.
Notes:
E5:02a CONDITION: If there is refrigeration equipment In this context, 'planned preventive maintenance' (PPM) is as set out in the cold chain equipment
Planned preventive maintenance checklist for refrigeration manufacturers' service manuals. Both the need for and the timing of PPM can be foreseen.
equipment:

A Is there a written planned preventive maintenance (PPM) N

programme. [Y, N]
B Is there documentary evidence that the programme is being
followed. [Y, N]
C Is somebody assigned to carry out routine maintenance [Y, N]
D Is there evidence that cold rooms/refrigerators/ freezers have
recently been cleaned and defrosted? [Y, N]
E Is there evidence that kerosene refrigerator wicks have been
trimmed and chimneys cleaned? [Y, N, n/a]
N PPM should be recorded in a maintenance record book or worksheet.
Y A permanent member of the facilty staff must be formally assigned to carry out the tasks below.
They should not be left to ancillary staff such as cleaners.
Y Check for excessive ice accumulation on evaporator units or the inside lining. Ice accumulation
or condensation along the panel joints in cold rooms and freezer rooms is a sign that panel joints
are leaking or poorly designed.
N/A Frequent trimming and replacement of wicks in kerosene refrigerators is absolutely essential.
Chimneys also need to be cleaned regularly, especially if kerosene quality is poor.

F Is there evidence that solar refrigerator battery electrolyte has N/A Some flooded batteries are not sealed and require regular checking of electrolyte levels and
been checked and solar panels cleaned? [Y, N, n/a] topping up with battery acid. All solar panels should be cleaned from time to time to maintain
efficiency. This is particularly important in dusty areas and in countries where there is snow.

5.1.3 Transport: Establish a routine maintenance and
overhaul programme and provide evidence that this
plan is being followed.
G Are all solar panels completely unshaded by buildings, trees N/A 2.50 5.00 Shading seriously degrades solar panel performance. Shading problems can increase over the ###
and overhead cables? [Y, N, n/a] years following the original installation as trees grow, new buildings are built and new overhead
cables are erected.
Notes:
E5:03a CONDITION: If vaccine transport is directly operated by this This indicator only applies to government-operated vehicles - see E0:04a.A.
store.
Transport maintenance and overhaul checklist:

A Are vehicles maintained in accordance with the The maintenance regime in the manufacturer's service manual must be followed.
manufacturer's service manual? [Y, N]
B Is there documentary evidence that the service manual is Documentary evidence' may be a service log book or a stamp in the service manual supplied
being followed. [Y, N] with the vehicle.
C Is there a written planned overhaul programme. [Y, N] Overhaul' means the dismantling or replacement of major components, such as clutch linings,
engines, transmissions, etc. The timing may vary between similar vehicles, but the need can be
foreseen.
D Is there documentary evidence that the overhaul programme 0.00 0.00 ###
is being followed. [Y, N]
Notes:
5.2 Arrangements must be made to carry out
prompt repairs in the event of equipment or
vehicle breakdown.

416188405.xls 9 of 23 04/11/2019
EVM # EVM REQUIREMENTS Q no. EVM QUESTION Result Score Max Guidance notes for assessors C
a
t
Equipment: make arrangements to ensure that cold E5:05a CONDITION: If vaccine is stored at the facility AND there are Include all vaccine refrigerators and freezers in the store, unless it is clear that broken units are
chain equipment is repaired within 7 days of refrigerators/freezers genuinely being kept for spare parts or disposal. s
breakdown. Record these repairs. Vaccine refrigerator and vaccine freezer maintenance c
checklist: o
r
e
A How many vaccine refrigerators and freezers are there (all 9 Ignore equipment that has been officially decommissioned. Include any equipment that is
types, including solar)? temporarily switched off because it is not needed. Check that it operates.
B How many were fully operational at the time of inspection? 9 5.00 5.00 Check if the unit has been out of operation for more than 7 days. If so, record this in the notes ###
box. With kerosene units, check that the burner has been assembled correctly.
Notes:
E5:06a CONDITION: If there are dedicated icepack freezers/cool Include all units in the store unless it is clear that broken units are genuinely being kept for spare
water pack coolers parts or disposal.
Maintenance checklist for freezer rooms, cold rooms, freezers
and refrigerators only used for freezing icepacks or cooling
cool water packs:

A How many icepack/cool water pack units are there? 100 Ignore equipment that has been officially decommissioned. Include any equipment that is
temporarily switched off because it is not needed. Check that it operates.
B How many were fully operational at the time of inspection? 100 0.00 0.00 Check if the unit has been out of operation for more than 7 days. If so, record this in the notes ###
box.
Notes:
5.2.2 Transport: make arrangements to ensure that E5:07a CONDITION: If vaccine transport is directly operated by this As a general rule, under EVM recommendations, distribution vehicles should not be out of
planned vaccine distributions are not affected by store. operation for more than 7 days. This question helps establish whether transport availability is
vehicle breakdown or poor maintenance planning. Transport availability checklist: high or low.

A During the review period, how many vaccine collections, Outreach immunization sessions' in this context means sessions that required a motorcycle, car
distributions or outreach immunization sessions were or 4WD.
planned?
B How many were cancelled or seriously delayed because of 0.00 0.00 Cancelled' means that the distribution or immunization session did not occur during the planned ###
mechanical failure in the vehicle fleet? working week. 'Vehicle' includes motorcycles, cars, 4WD, trucks and small boats. Air and rail
links (if used) are assumed to be outside the programme's control.
Notes:
Criteria score E5: 9.17
Maximum score: 15.00
Percentage: 61%
E6 Stock management systems and procedures are effective.

6.1 Standardized recording and reporting of all

stock transactions must be carried out.
Preferably this should be computerized at
primary level and in all other large stores.

6.1.1 Where a computerized stock control system is used, E6:01a Is a computerized stock control system in use? [Y, N]. Y
the software and computer equipment must be
suitable for the task and well-maintained and staff Notes:
must know how to use the system.
E6:02a CONDITION: If computerized stock control is used.
Computerized stock control checklist:

A Is the software fit for purpose [Y, N]
B Is the software package supported by the software developer
[Y, N]
C Is there up-to-date anti-virus software on the computer [Y, N]
Y Fit for purpose' means that it can handle the full range of vaccines and commodities, has
sufficient data capacity for the country's needs, and records the minimum data set listed in
E6:04a and E6:05a
Y
Y Anti-virus software is essential.

D Has the storekeeper received training in its use [Y, N] Y

E Is the computer suitable for its purpose and in working order? Y
[Y, N].
F Is the printer in working order and supplied with paper? [Y, N] Y

G Is the computer equipment connected to the standby Y

generator? [Y, N]
H Are stock records printed out and filed as a permanent record Y Paper records are needed as a backup and for quality assurance and review purposes.
at least once a month? [Y, N].
J Are stock records backed up at least once a week? [Y, N]. Y 1.00 1.00 Backup should be on a flash drive,secondary hard drive, CD/DVD or on a secure web-based ###
backup site. Ideally the device should be kept off site.
Notes:
416188405.xls 10 of 23 04/11/2019
EVM # EVM REQUIREMENTS Q no. EVM QUESTION Result Score Max Guidance notes for assessors C
a
t
6.1.2 Ensure that all stock transactions are recorded in E6:03a Are all vaccine arrivals and vaccine dispatches recorded and Y 5.00 5.00 The practice of batch processing arrival and delivery slips should be discouraged because it ###
the stock records by the end of the working day. stock balances updated within one working day of the leads to out of date stock records. s
transaction? [Y, N]. c
Notes: o
6.1.3 Arrival. Accurately record incoming vaccines, E6:04a Do the stock records record the following information for all r
diluents, and other consumables. vaccines: e
A Type of vaccine [Y, N] Y
B Vaccine presentation (vial size) [Y, N] Y
C Quantity received in doses [Y, N] Y
D Vaccine manufacturer [Y, N] N
E Manufacturing batch or lot number [Y, N] Y
F Expiry date of each vaccine batch [Y, N] Y
G VVM status where applicable [Y, N, n/a] Y
H Location in the store [Y, N, n/a] N 3.75 5.00 A record of location in the store is essential at primary level and in larger sub-national stores. It ###
is unnecessary in small sub-national stores and at health facility level.
Notes:
E6:05a Do the stock records record the following information for all
diluents that are packed separately from the vaccine to which
they belong:

A Type of diluent [Y, N] N

B Diluent presentation (vial size) [Y, N] N
C Quantity received in doses [Y, N] N
D Diluent manufacturer [Y, N] N
E Manufacturing batch or lot number [Y, N] N
F Expiry date of each batch [Y, N] N
G Location in the store [Y, N, n/a] N 0.00 5.00 A record of location in the store is essential at primary level and in larger sub-national stores. It ###
is unnecessary in small sub-national stores and in health facilities
Notes:
E6:07a Are vaccine requisition forms used for ordering and receiving Y 1.00 1.00 See model SOP on requisitioning for guidance on good practice. Assess content of existing ###
vaccine? [Y, N]. form(s) against the content of the model forms.
Notes:
6.1.5 Dispatch: Establish a pre-delivery or pre-collection E6:08a Is there an effective pre-delivery, or pre-collection, notification Y 1.00 1.00 Effective' means that the system is understood and used by the receiving stores and that ###
notification system. system in place? [Y, N] deliveries or collections are reliably made on or near a day that has been arranged well in
advance.
Notes:
6.1.6 Dispatch. Issue and use vaccines, diluents and E6:09a Do stock records and/or stock in hand demonstrate that Y 1.00 1.00 Unless VVMs on later-dated stock show Stage 2 exposure, vaccines should always be issued in###
other date-limited products in Earliest Expiry - First vaccine is issued according to the 'earliest-expiry-first-out' EEFO order. Health worker use of EEFO is covered by question E8:09.
Out (EEFO) order. If VVM status indicates that (EEFO) principle? [Y, N]. In PR, SN and LD stores, the stock records should contain the data needed to demonstrate
some vaccine vials should be used ahead of its compliance. Otherwise physically check the stock. In health facilities it may be easier to check
correct EEFO order, this may be done, but the the stock in hand.
reason for doing so must be recorded.

Notes:
E6:10a Can the storekeeper make exceptions to the EEFO rule (e.g. Y 1.00 1.00 If VVMs on later-dated stock show Stage 2 exposure whilst early-dated stock is still at Stage 1, ###
because of VVM status)? [Y, N]. the heat-exposed later-dated stock should generally be issued first.

6.1.8 Arrival: When vaccines and consumables arrive at
the receiving store, carry out arrival checks and
record results on the arrival section of the issue
voucher. Return a copy of the arrival section to the
issuing store.
Notes:
E6:13a Does the issuing store have a completed arrival voucher for Y 1.00 1.00 Use judgement. If the review period has just ended, the arrival slip for recent deliveries may not###
every delivery which took place during the review period? [Y, yet have been received by the issuing store. There may be other legitimate reasons for missing
N]. forms. The important point is that the incomming goods must be checked and the form must be
signed by the receiving store.

6.1.9 Disposal. Safely dispose of damaged or expired
stock in accordance with standing orders.
Notes:
E6:14a Does a representative sample of these completed arrival Y 1.00 1.00 As with temperature records, suspiciously 'correct' forms may conceal lax practices. If vaccines ###
sections indicate that arrival checks were carried out are put in quarantine for any reason, these vaccines should be ignored.
correctly? [Y, N].
Notes:
E6:15a CONDITION: If vaccine is stored at the facility Question does not apply if there is no refrigerated storage at the facility. For example, at health
Checklist for management of damaged or expired stock: facilities using cold boxes and vaccine carriers.

A Is the stock control system designed to record vaccine and Y If vaccine has to be discarded, any associated diluent must also be discarded. Failure to do this
diluent wastage due to expiry, freezing or heat-exposure? [Y, can lead to a stockpile of 'orphaned' diluent.
N]

416188405.xls 11 of 23 04/11/2019
 stock in accordance with standing orders.

EVM # EVM REQUIREMENTS Q no. EVM QUESTION Result Score Max Guidance notes for assessors C
a
t
B If 'yes', do the records show that the system is being used? [Y, N/A If the stock records show that there have been no losses during the review period, ask the
N. Score n/a if there have been no losses in the review period] storekeeper to confirm whether or not this is true. s
c
C Do staff know that expired and damaged vaccine should be Y Damaged vaccine must be clearly labelled and stored outside the cold chain to prevent it being o
clearly labelled and stored out of the cold chain until final issued in error. r
disposal? [Y, N] e
D If 'yes', is there evidence that the procedure is followed? [Y, N. N/A 5.00 5.00 Check that there is no damaged/expired vaccine in the store. ###
Score n/a if there have been no losses in the review period]

Notes:
E6:16a CONDITION: If there is a functioning system for recording Carry out this check even if the storekeeper reports that no vaccine has been lost.
damaged or expired vaccine. A score of 5 is only achieved if the percentage of vaccine lost is no more than 1% of the doses
Select a freeze-sensitive vaccine. Establish how many doses issued during the review period. Exposure to freezing must have been confirmed by a shake test.
have been discarded during the review period due to freezing Exposure to high temperature must have been confirmed by VVM colour changes.
or exposure to high temperature.

A Enter choice of vaccine:

B Record number of doses of chosen vaccine issued during the
review period.
C Record number of doses of chosen vaccine discarded 0.00 5.00 This indicator can only be assessed if stock records are accurate, informative and truthful. If ###
because of incorrect storage temperatures. there is no method to record vaccine damage, the score is 0.
Notes:
E6:17a Are disposal facilities and procedures in accordance with 0.00 1.00 Refer to Guidelines for Safe Disposal of Unwanted Pharmaceuticals in and after Emergencies. ###
WHO and/or national norms? [Y, N]. WHO/EDM/PAR/99.2

Notes:
E6:18a Have records of discarded vaccine been kept for at least three 0.00 1.00 ###
years, or, if for a lesser period, since the immunization
programme adopted the EVSM or EVM? [Y, N]

Notes:
E6:19a Are internal reviews of the vaccine loss/damage records N 0.00 5.00 ###
carried out at least twice a year? [Y, N]

Notes:
6.2 Stocks of vaccines and consumables must be
maintained between the safety stock level and
the maximum stock level.
6.2.1 Establish a maximum stock level, a reorder level E6:20a Assess stock level policy: Use the stock records to assess whether the store held excessive and/or insufficient stock during
and a safety stock level for each vaccine and for the review period.
each consumable. A Is a maximum stock level set for each vaccine and Y A maximum stock level should be set to control over-stocking. Overstocking makes store
commodity? [Y, N] management difficult and increases the risk of expiry.
B Is a reorder level set for each vaccine and commodity? [Y, N] Y A reorder level is needed to take account of delivery lead-time so that safety stocks are not
breached
C Is a safety stock level set for each vaccine and commodity? Y A safety stock level needs to be set to reduce the risk of stockouts.
[Y, N]
D During the review period, did the stock of each and every Y Note: The EVM maximum recommended stock level for each vaccine is its defined maximum
vaccine remain within its EVM recommended maximum? [Y, stock level plus 10%.
N]
E During the review period, did the stock of each and every Y 5.00 5.00 Note: The EVM minimum recommended stock level for each vaccine is its defined safety stock ###
vaccine remain within its EVM recommended minimum? [Y, level minus 10%.
N].
Notes:
6.2.2 There must be no stockouts E6:21a Check stockouts and related events during the review period: Note: If no records are available, interview the storekeeper and make a judgement.

A Were vaccine stocks sufficient throughout the review period Y A stockout occurs when the demand or requirement for an item cannot be fulfilled from the
(no stockouts)? [Y, N]: currently available inventory. It does not necessarily mean that there is zero stock. Cases may
occur where there is a stockout, but this does not affect deliveries because a new vaccine arrival
occurs before the next delivery has to be made. Question B covers this situation.

B Was the facility always able to supply all the lower level Y 5.00 5.00 Deliveries to lower level stores are not applicable at health facility level. ###
stores? [Y, N, score n/a if this is a health facility]
Notes:
6.3 Periodic physical inventories must be
conducted.

416188405.xls 12 of 23 04/11/2019
EVM # EVM REQUIREMENTS
6.3.1 Carry out a physical inventory of vaccine, diluent
and dropper stocks at least once every three
months at primary level and at least equal to the
planned supply frequency at lower levels.
6.3.2 Carry out a physical inventory of other consumables E6:24a
(AD syringes, safety boxes, consumables, spare
parts, etc.) at least once every three months at
primary level and at least equal to the supply
frequency at lower levels.
6.4 Good warehousing practices must be followed.
6.4.1 Storage: store all vaccines, diluents and droppers
securely and correctly in the vaccine store.
6.4.2 Storage: store all syringes and safety boxes
securely and correctly in the dry store.
416188405.xls
Q no. EVM QUESTION Result Score Max Guidance notes for assessors C
a
t
E6:22a Physical inventory frequency checklist: Question does not apply if there is no refrigerated storage at the facility. For example, at health
facilities using cold boxes and vaccine carriers. s
c
o
A Enter the planned vaccine supply period, in months 4,821.0
B Enter number of recorded physical counts of vaccine stocks 717 5.00 5.00 Primary stores should carry out a physical count at least 4 times a year. Lower level stores and ###r
carried out during the 12 month review period health facilities should carry out a count at the time of each incoming delivery. The scoring logic e
assumes this frequency.
Notes:
E6:23a CONDITION: If vaccine is stored at the facility This question does not apply if there is no refrigerated storage at the facility. For example, at
Choose a sample freeze-dried vaccine and diluent. Enter health facilities using cold boxes and vaccine carriers.
name. 338 The following checks are carried out:
A Carry out a physical count of the sample vaccine. Enter 338
number of doses counted:
B Carry out a physical count of the sample diluent. Enter 338 1) If the diluent count differs from the vaccine count by >±1% a 'Mismatch' message appears.
Match
number of doses counted: Otherwise a 'Match' message appears.
C Check stock records for sample vaccine. Enter number of 338 2) If the vaccine stock records = the vaccine count an '=' sign appears. If they are different, the
Equal
doses recorded as currently in stock: warning 'Not =' appears.
D Check stock records for sample diluent. Enter number of 338 3) If the diluent stock records = the diluent count an '=' sign appears. If they are different, the
Equal
doses recorded as currently in stock: warning 'Not =' appears.
E Scoring: Score 4 if stock count and records match exactly 4 5.00 5.00 Scoring: Ideally, the physical count and the stock records should always match at the end of the###
AND vaccine and diluent quantities are within 1% of one working day. However, there may be discrepancies during the day if the stock records are not
another (see results). Use your judgement to adjust the score updated immediately; check with the storekeeper. There are legitimate reasons why diluent
between 0 and 4. quantities may differ from vaccine quantities. One of these is that vaccine manufacturers
sometimes over-supply diluent. Therefore, if there is slightly more diluent than vaccine, there
may be no problem. If there is much more, or much less diluent, than vaccine, there must be a
problem.
Notes:
CONDITION: If consumables are held in this store.
Choose a sample consumable. Enter it in the box below :
A Choose a sample consumable. Enter name. Safety Box
B Carry out a physical count of the sample consumable. Enter 25
number of items counted:
C Check stock records for sample consumable. Enter number 25
Equal
of items recorded as currently in stock:
D Scoring: Score 4 if stock count and records match exactly. 4 1.00 1.00 Scoring: See notes to E6:22a. ###
Use your judgement to adjust the score between 4 and 0 for
mismatches.
Notes:
E6:25a CONDITION: If vaccine is stored at the facility Question does not apply if there is no refrigerated storage at the facility. For example, at health
Checklist for vaccine storage arrangements: facilities using cold boxes and vaccine carriers.
A Is the vaccine stock secure? [Y, N] Y
B Are contents labels fixed to all cold chain equipment indicating Y Check for labels on the lid of feezers and refrigerators, or on the edge of the shelves in freezer
vaccine type, lot no. and expiry date [Y, N, n/a]. rooms and cold rooms. Score 'n/a' where labelling is not needed. Labels are not required at
health facility level unless there are several refrigerators.
C Are vaccines correctly stored? [Y, N]. Y No freeze-sensitive vaccines stored close to cold room evaporators, stored close to the ice lining
in ILRs or stored close to the evaporator plate of service point refrigerators. No vaccines should
be kept in the door shelves of domestic refrigerators.
D Is vaccine laid out in in EEFO order, by type and by lot Y Ask which vaccines of a particular type are going to be picked first and check whether these
number? [Y, N] items have the shortest expiry date.
E Is the vaccine store clean, dry and pest-free? [Y, N] Y 1.00 1.00 ###
Notes:
E6:26a CONDITION: If consumables are held in this store.
Checklist for dry storage arrangements:
A Are dry goods secure? [Y, N] Y
B Are dry goods correctly stored? [Y, N]. Y Correct' means that stocks should be neatly stacked on pallets so that they are accessible and
are not affected by floor washing.
13 of 23 04/11/2019
 Storage: store all syringes and safety boxes
securely and correctly in the dry store.

EVM # EVM REQUIREMENTS Q no. EVM QUESTION Result Score Max Guidance notes for assessors C
a
t
C Are dry goods laid out in an orderly fashion and in EEFO Y Syringes have expiry dates and must therefore be organised in EEFO order. Ask which stock is
order where applicable? [Y, N] going to be picked first and check whether these items have the shortest expiry date. s
c
D Is the dry store clean, dry and pest-free? [Y, N] Y 1.00 1.00 o
###
Notes: r
e
6.4.3 Data security: keep all records secure. E6:27a Are the records secure? [Y, N] Y 1.00 1.00 Records should be stored in lockable filing cabinets. ###

Notes:
Criteria score E6: 44.75
Maximum score: 63.00
Percentage: 71%
E7 Distribution between each level in the supply chain is effective.

7.1 Distribution reports must demonstrate

compliance with the planned delivery schedule.

7.1.1 Maintain a programme for the distribution of vaccine E7:01a During the review period, did the issuing store send a N 0.00 5.00 Throughout this section, the issuing store is the store which is being assessed. ###
from the issuing store to the each receiving store. distribution programme to each receiving store setting out
The programme must be flexible enough to dates for the delivery and/or collection of vaccines? [Y, N]
accommodate variations in demand from the
receiving stores.
Notes:
E7:02a CONDITION: If a distribution programme exists. Scoring: Actual deliveries must be within 80% to 120% of planned deliveries. If no distribution
Assess actual deliveries/collections against the programme: programme exists, the score is 0.

A Enter number of deliveries/collections (to all receiving stores)

that were scheduled during the review period?
B Record the number of deliveries/collections made during the 0.00 1.00 ###
review period.
Notes:
E7:03a CONDITION: If a distribution programme exists. Sample period is context dependent. For example, if deliveries are made four times a year over
Assess the timeliness of a sample of deliveries/ collections: two week periods, the chosen sample should be one of these two week periods.

A How many deliveries/collections were scheduled during the

sample period?
B How many deliveries/collections actually took place during the 0.00 1.00 Scoring: Algorithm ensures that score is always <1 if actual deliveries/ collections do not match ###
sample period? the number scheduled.
Notes:
7.1.2 Establish an effective reporting system which E7:04a CONDITION: If a distribution programme exists. 0.00 1.00 ###
monitors actual vaccine distributions and compares During the review period, was there a reporting system which
these with anticipated distributions. monitored actual vaccine distributions and compared these
with planned distributions? [Y, N]

Notes:
7.2 A system for managing short shipments must
be in place.
7.2.1 Establish an effective system for managing short E7:05a Assess number of short shipments during the review period Compare requisition quantities with actual delivery quantities. Question the storekeeper on the
shipments to receiving stores. and check if they were followed up and corrected. actions s/he has taken to reduce the impact of short shipments when they occur.

A Where there any short shipments? [Y, N]

B CONDITION: If there were short shipments. 0.00 0.00 ###
Evaluate the effectiveness with which short shipments were
managed and score on a scale of 0-4].
Notes:
7.3 Vaccines must be correctly packed for
transport, either in passive containers or in
refrigerated vehicles
7.3.1 Transport vaccine in passive containers which have E7:06a CONDITION: If cold boxes or other passive containers are Ask whether a systematic temperature monitoring, or journey profiling, study has been carried
a cold-life (or in cold climates, a warm-life) sufficient used. out. Coolant preparation includes icepack conditioning, preparation of cool water packs and
for the longest expected journey. Assess knowledge of coolant preparation and packing of cold preparation of coolant for palletized or wheeled passive containers.
boxes, vaccine carriers and other passive containers:

A Icepack conditioning or cooling of chilled water packs is in Y For palletized or wheeled containers, follow the manufacturer's instructions.
accordance with WHO guidelines? [Y, N]

416188405.xls 14 of 23 04/11/2019
 a cold-life (or in cold climates, a warm-life) sufficient
for the longest expected journey.

EVM # EVM REQUIREMENTS Q no. EVM QUESTION Result Score Max Guidance notes for assessors C
a
t
B Is packing in accordance with appliance labelling or SOP? [Y, Y 5.00 5.00 Cold boxes, vaccine carriers and other passive containers are designed to suit a particular size ###
N] of icepack. Sometimes the correct size is not available. In such circumstances it is essential s
that store staff know what to do. c
Notes: o
E7:08a Cold climates only. Do staff know how to prevent vaccine N/A 0.00 0.00 Refer to SOP on freeze prevention for details. r
###
freezing during transport? [Y, N, n/a]. e

Notes:
7.4 Freeze indicators and VVMs must be used
correctly to monitor the quality of the
distribution system, and data must be recorded.

7.4.1 Dispatch: Record VVM status for each batch of
vaccine. If a freezing risk exists, insert a freeze
indicator in every shipment of freeze-sensitive
vaccine sent from the issuing store to the receiving
stores.
E7:09a Are freeze indicators required for the transport of freeze- N Freeze indicators are NOT required if cool water packs are always used in passive containers
sensitive vacines? [Y, N ] and there is no risk of freezing in winter months. Freeze indicators MAY be required when
vaccine is transported in refrigerated vehicles with poor temperature control or limited monitoring
equipment. Freeze indicators ARE required when freeze-sensitive vaccines are packed with
frozen ice packs, even if these have been conditioned. They are also recommended in very cold
climates where warm water packs are needed to prevent vaccine freezing during transport.

Notes:
E7:10a CONDITION: If freeze indicators are required. 0.00 0.00 If freeze indicators are used, there must be an effective zero-reporting system to record the state###
Were freeze indicators packed with deliveries of freeze- of the freeze indicator on arrival at the receiving store.
sensitive vaccines during the review period?
Scoring: Score on a scale of 0-4 where 4 indicates that freeze
indicators were used on all deliveries and 0 indicates that they
were never used.

Notes:
7.4.2 Arrival: The receiving store must return the arrival E7:11a Select a one month sample of issue vouchers. If no system is in place, leave all cells blank. The score will be zero. Note absence of system in
section of the issue voucher and must record VVM notes box.
and freeze indicator status (where applicable) at the A How many vouchers were issued by the issuing store?
time of arrival.
B How many arrival sections of the sampled issue vouchers
have been returned to the issuing store?
C How many of the returned vouchers record VVM status? (If
vaccines do not have VVMs enter n/a)
D CONDITION: If freeze indicators are required. 0.00 1.00 Scoring algorithm ensures that only returned and fully completed vouchers count towards the ###
How many of these vouchers also record freeze indicator score.
status?
Notes:
7.5 In case of cold chain failure during distribution,
damage must be reported and effective action
taken.
7.5.1 Replace damaged vaccine as soon as possible. E7:12a Assess vaccine losses caused by freezing or heat exposure This is a difficult indicator to assess, unless the stock records are transparent and honest. 100%
during transport from the issuing store: score is achieved if there are no losses. If losses occurred and damaged vaccine was replaced
in a 'reasonable' time, the score is 50%.

A The EVM target is a total loss of less than 1%. (Score Y if this Y Reasonable period' depends upon the level in the cold chain and the established supply interval
target was achieved, otherwise score N). and the extent of the vaccine damage. The country should have a clear policy to cover this and
the assessment should be made against this policy.
B CONDITION: If 1% loss target was exceeded. N 5.00 5.00 If there is no fixed policy, use judgement. For example, at service point level, if the set supply ###
Was the damaged vaccine replaced within a reasonable interval is 4 weeks, then 2 weeks would be reasonable. If the supply interval is 2 weeks, then
period of notification to the issuing store? [Y, N] . one week would be reasonable.
Notes:
Criteria score E7: 10.00
Maximum score: 19.00
Percentage: 53%
E8 Appropriate vaccine management policies are adopted and implemented.

8.1 Knowledge of vaccine handling must be good.

8.1.2 Knowledge of the shake test is good. E8:01a CONDITION: If vaccine is stored at the facility This question is not applicable at health facility level in situations where there is no refrigerator
Assess knowledge of the shake test:

416188405.xls 15 of 23 04/11/2019
 Knowledge of the shake test is good.
EVM # EVM REQUIREMENTS Q no. EVM QUESTION Result Score Max Guidance notes for assessors C
a
t
A Does the storekeeper or health worker know how to conduct Y Check knowledge against the Shake test learning guide
the shake test? [Y, N] s
B Does the storekeeper or health worker know when to conduct Y Check knowledge against the Shake test learning guide c
the shake test? [Y, N] o
C Has the storekeeper of health worker carried out a shake test Y 5.00 5.00 Only score n/a if you are sure that there were no events in the review period that required a r
###
in the last 12 months? [Y, N. Score n/a if there were no events shake test. e
that required a shake test]
Notes:
8.2 VVM knowledge and practice must be good.
8.2.1 The VVM policy is correctly implemented by national E8:05a Are written instructions on the use of VVMs, such as posters Y 1.00 1.00 ###
EPI. and stickers, available to storekeepers and health workers?
[Y, N].
Notes:
E8:06a Do storekeepers/health workers know how to read VVMs? [Y, Y 5.00 5.00 Use dummy VVMs and/or sticker samples to check knowledge. To score 'Y', all those ###
N]. responsible in the facility must answer correctly.
Notes:
8.4 Monitoring and data collection must take place
at all levels and the data must be used
effectively.
8.4.1 There is a vaccine wastage monitoring system. E8:12a Review the periodic immunization reports and/or any other 0.00 1.00 At national level this question should be adressed to the EPI Manager. At lower level stores ###
reporting forms that are used to monitor vaccine wastage. wastage rates may not be calculated. However, the information needed for the calculation
[Score 0 if there is no reporting. Score on a scale of 1-4 if should be collected and reported up the supply chain. For example, number of children
formal reporting exists. Use commentary to elaborate]. immunized, doses used and doses lost, destroyed or expired.

Notes:
8.4.2 When vaccines are ordered, wastage rate E8:13a Wastage calculation checklist: At national level this question should be adressed to the EPI Manager.
information is used to establish the quantities
required. A Do you calculate wastage rates for each vaccine? [Y, N] N
B Is there a standard form used to record wastage? [Y, N] N
C Is there a complete set of wastage records for the review N
period? [Y, N]
D Are these data used to monitor vaccine management 0.00 5.00 ###
performance? [Y, N]
Notes:
8.4.3 Regular supportive supervision takes place. E8:14a Does the facility receive regular supportive supervision? [Y, N] Y 5.00 5.00 Ask staff when the last supervisory visit took place and how regular these visits are. Answer yes ###
if the level of support is suitable, taking account of the general knowledge of the facility staff.

Notes:
Criteria score E8: 16.00
Maximum score: 22.00
Percentage: 73%
E9 Information systems and supportive management functions are satisfactory.

9.1 Standard operating procedures must be in

place.
9.1.1 Standard operating procedures must be presented E9:01a Is there a Standard Operating Procedures (SOP) manual? [Y, N
in a form which can be easily understood by all staff N].
who carry out the procedures. Notes:
E9:04a CONDITION: If there is an SOP manual. 0.00 1.00 Check that relevant SOPs are available. For example there is no need to have an SOP for ###
Does the facility have a copy of the SOP manual? [Y, N] looking after cold rooms at lowest distribution level.

9.3 Field data must be collected and used to
support management decision-making.
9.3.1 Use an evidence-based method to forecast the
need for vaccines and consumables.
Notes:
E9:07a Checklist for vaccine needs forecasting: Use one of the options described in the Mid-level Management training series. Namely:
1) At primary and sub-national levels: Target population or previous consumption data.
2) At service point level: Session size.
Check to ensure that there is compatibility/convergence with the method used in developing the
country's comprehensive multi-year plan (cMYP).

A Is a standard method used to estimate annual vaccine need? Y Standard method' means the UNIICEF vaccine forecasting tool or equivalent. 'Evidence-based'
[Y, N] means that data have been systematically collected - for example national population census,
updated as necessary.

416188405.xls 16 of 23 04/11/2019
EVM # EVM REQUIREMENTS Q no. EVM QUESTION Result Score Max Guidance notes for assessors C
a
t
B Is evidence-based target population data used in the Y Calculations can be based on target population data and assumed vaccine wastage rates, but it
calculation? [Y, N] is much better to use evidence-based wastage data collected from the field. s
C Is evidence-based coverage data used in the calculation? [Y, Y Coverage data will generally be taken from the routine reporting system. Ideally these data c
N] should be verified by periodic coverage surveys. o
D Is evidence-based vaccine wastage data used in the Y 5.00 5.00 Calculations can be based on target population data, coverage data and assumed vaccine r
###
calculation? [Y, N] wastage rates, but it is much better to use evidence-based wastage data collected from the field. e

Notes:
E9:08a Is an evidence-based method used to forecast need for Y 5.00 5.00 See previous guidance note. ###
syringes and safety boxes? [Y, N]
Notes:
Criteria score E9: 10.00
Maximum score: 11.00
Percentage: 91%

416188405.xls 17 of 23 04/11/2019
% score
Target %
E1: Vaccine arrival
E1

0%
80%
E2: Temperature
E2
Summary data

E3: Storage capacity

E3

83% 100%
80% 80%
E4: Buildings, equipment, transport
E4

41%
80%
E5: Maintenance
E5

61%
80%
Criteria

E6: Stock management

E6

71%
80%

E7: Distribution
E7

53%
80%

E8: Vaccine management

E8

73%
80%

E9: MIS, supportive functions

E9

91%
80%
B

Buildings
C

Capacity
88% 100%
80% 80%
E

Equipment
43%
80%
M

Management
67%
80%
Categories
 Categories
R
Repairs/maintenance

61%
80%
T
Training

96%
80%
V

Vehicles

0%
80%
Charts
0 20%
Criteria Scores 0 40%
1 0 60%
0 80%
0 100%
0.8

0.6

0.4

0.2

0
E1 E2 E3 E4 E5 E6 E7 E8 E9

% score Row 101 Row 101 Row 101 Target % Grid

Category Scores
1

0.8

0.6

0.4

0.2

0
B C E M R T V

% score Row 101 Row 101 Row 101 Target % Grid

Charts

Criteria Scores

E1: Vaccine arrival

E9: MIS, supportive functions E2: Temperature
8
0.
4
0.
E8: Vaccine management E3: Storage capacity
0

E7: Distribution E4: Buildings, equipment, transport

E6: Stock management

E5: Maintenance

Row 101 Row 101 Row 101 % score Target %

Category Scores

Buildings

1
Vehicles 0.8 Capacity
0.6
0.4
0.2
0
Training Equipment

Repairs/maintenance Management

Row 101 Row 101 Row 101 % score Target %