IMAGE PROCESSING FOR VISUAL PROSTHESES: A CLINICAL PERSPECTIVE

Lauren N. Ayton
1,2
, Chi D. Luu
1,2
, Sharon A. Bentley
1,3
, Penelope J. Allen
1,2
, Robyn H. Guymer
1,2


1
Centre for Eye Research Australia, Royal Victorian Eye and Ear Hospital
2
Department of Ophthalmology, University of Melbourne
3
School of Medicine (Optometry), Deakin University

ABSTRACT

Recent advances in the field of visual prostheses or bionic
eyes have shown that it is possible to use electrical
stimulation to produce basic phosphenised vision to patients
who are profoundly vision impaired or blind. In particular,
retinal prostheses have been implanted in a number of
clinical trials for a degenerative eye disease known as
retinitis pigmentosa.

To date, the visual improvements in these trials have been
small and not easily quantified. The aim of this paper is to
highlight the inherent complexities in the assessment of
visual function in the profoundly vision impaired, and
discuss the potential for improvement in outcomes using
image processing technology.

Index Terms visual prosthesis, functional vision
outcomes, clinical assessment measures

1. INTRODUCTION

Until recently, there were no treatment options available for
people who had become profoundly vision impaired from
retinal degenerations. The most common inherited retinal
degeneration is retinitis pigmentosa (RP, fig. 1), which
affects approximately 1.5 million people worldwide [1]. RP
causes a selective loss of outer retinal cells (photoreceptors),
with results from morphometric and retinal imaging studies
showing that the inner retinal neurons and posterior visual
pathway are relatively preserved [2-6]. As such, it is
possible to use electrical stimulation from a retinal implant
to stimulate the remaining inner retinal cells, optic nerve and
visual cortex.
This theoretical basis for retinal prosthetic
technology has been validated through a number of recent
clinical trials, which have shown that retinal prostheses are
able to induce visual phosphenes in patients with end stage
retinitis pigmentosa [7-11]. However, at this stage the visual
improvement possible with such devices is small, with the
best visual acuities reported to date being 1.69 [9] to 1.8
logMAR [7]. In practical terms, this represents an
improvement in vision from only being able to see light and
dark, to a level where the patient is able to identify objects



and very large letters (>15 cm at arms length). Previous
studies have also shown that in prior clinical trials, the
dynamic range of brightness has been limited to less than 10
levels of just noticeable difference [12, 13].



Figure 1: Example ocular fundus image of a normal eye (A) and a
patient with retinitis pigmentosa (B), with the characteristic
pigmentation clumps (for which the disease is named) to the left of
the second image.

The advantage with using an electronic biomedical
device such as a retinal prosthesis for vision restoration is
that there is the potential to use image manipulation and
signal processing to improve the visual outcomes for
patients. This is especially important with the current
devices, which only offer limited number of electrodes on
the retinal array. Whilst some retinal prostheses use
photovoltaic cells for direct light stimulation of the
electrodes [9], the majority of approaches will have an
external spectacle mounted camera to capture the original
image (fig. 2). A visual prosthesis that is implanted further
back in the visual pathway (such as optic nerve and cortical
implants) will also require a camera to obtain the raw image.
The image sent from the camera can then be processed in a
number of ways en-route to the electrode array, in order to
optimise certain parameters and improve patient function.
Hence, image processing will be a vital component in
improving the potential outcomes of many visual prosthesis
implant designs.

A B
1540 978-1-4799-2341-0/13/$31.00 2013 IEEE ICIP 2013


Figure 2: Schematic representation of the Bionic Vision Australia
retinal prosthesis design.


2. CLINICAL ASSESSMENT OF THE PROFOUNDLY
VISION IMPAIRED PATIENT

Prior to discussion of the potential uses for image processing
in visual prosthesis implant recipients, it is important to
understand the inherent difficulties in the measurement of
outcomes in such clinical trials.
Profound vision loss is defined as vision of less
than 6/120 [14], which means that a patient is unable to see
any letters on a visual acuity chart in the clinic, even if they
stand twice as close to the chart as normal. At this early
stage of research, potential candidates for visual prosthesis
surgery will have even poorer vision, such that can only see
light and dark. This bare light perception vision is often
very hard to measure, as it can vary from day to day due to
environment, patient fatigue and other factors. As such, it is
difficult to obtain a baseline measure of vision without
repeated measures over a period of time.
There have been attempts to develop more
objective measures of vision and residual retinal function at
this very poor level, including various low vision visual
acuity charts [15-18] and advanced analysis of objective
measures such as the electroretinogram [19]. These new
measures have improved our ability to estimate visual
performance at this very low level of function, but are
limited by observer differences and inter-session variability
due to factors such as fatigue, internal and external
motivations and psychological wellbeing at the time of
evaluation.

3. FUNCTIONAL VISION OUTCOMES

Due to the above limitations in assessment of clinical visual
performance and the need to understand the patients
perspective, it has been proposed by both researchers and
clinicians that studies of visual prosthesis implants should
also include real-world functional vision outcome
measures [20, 21]. The advantage of such evaluations is that
they can tell us more information about the usefulness of a
device in a persons day-to-day life. Whilst clinical
measurements such as visual acuity and visual fields are
more easily standardized and quantified, they do not always
reflect the complexities of real-world functioning.
Functional vision outcomes cover aspects such as
orientation and mobility and activities of daily living.
Orientation and mobility (O&M) relates to a persons ability
to move around their environment in a safe and efficient
manner. Whilst other senses are important for O&M (such as
hearing, touch and smell), sight leads to a strong
improvement in performance, allowing people to localize
and avoid obstacles, and decide on the most efficient route
of travel. Generally, once people have profound vision loss,
they benefit from external mobility devices for assistance,
such as a long cane, sighted human guide or a guide dog. An
O&M instructor usually teaches these skills.
The other main aspect of functional performance
that plays a large role in the assessment of visual prostheses
is activities of daily living (ADLs, fig. 3). ADLs are basic
tasks that people perform to live a safe, healthy and fulfilling
life. The tasks include identification of food and common
household items, money handling, self-care (showering,
dressing, toileting) and cooking. ADLs are often assessed in
the community by occupational therapists, who play a large
role in determining rehabilitation and training protocols.

A)

B)

Figure 3: Examples of activities of daily living that may be
assessed in visual prosthesis studies; pouring a drink (A) and
finding an object on a table (B)

The importance of these two areas of functional
vision (O&M and ADLs) is evident by their inclusion in
preliminary guidelines set out by the US Food and Drug
Administration for the Investigational Device Exemption
(IDE) of retinal prostheses [22]. As such, there is strong
interest internationally in measuring real world performance
on O&M and ADL tasks in patients using a visual
prosthesis. Currently there are no established methods of
assessment, and the challenge is to develop standardized
measures of functional vision performance. Hence, it is
important that the broad range of researchers involved in
vision restoration, such as engineers and basic science
1541
researchers who work on image processing have an
understanding of the types of conditions in which we hope
the prostheses will be of use.

4. IMAGE PROCESSING REQUIREMENTS (FROM A
CLINICIANS VIEWPOINT)

With respect to image processing, one of the main
differences between clinical measures of visual performance
and the more real-world functional vision assessments is in
the complexity of visual stimulus. In a clinical visual acuity
measurement, the letter optotypes are presented at 100%
contrast with black letters on a white chart. As such, edge
detection and letter recognition is simpler than it would be in
a low contrast, high noise environment. Whilst it is
important to show efficacy in clinical measures, the real use
of the device will occur in complex environments, requiring
more sophisticated image processing approaches.
From a clinical viewpoint, one of the most
important parameters to consider is dynamic range. When
people have low vision, they often rely on graduations in
light and shadows to help delineate objects (including
potential dangers such a stairs, curbs and drop-offs). As
such, it is important that shadows be seen as a different
percept to a solid object, to enable people to move around
the environment safely. In the same way, it is important to
optimize the just noticeable difference levels in the image,
to enable people to differentiate small variations in light,
colour and texture.
Whilst on the theme of safety, obstacle localization
is of high importance when designing a device to be used
while mobile. Our colleagues at NICTA have investigated a
number of mechanisms for enhancing navigational
performance around obstacles, and have found that
augmented depth algorithms give the best performance on a
maze task (compared to simple brightness or contrast- based
images) [23-25]. Interestingly, they reported a strong
learning effect, which supports expectations that patients
who receive visual prosthesis implants will require
substantial rehabilitation and training to optimize their
performance..
Enhancement of navigation in an environment can
also be achieved by improving the detection and recognition
of signs. It is worth remembering that the majority of people
who receive a visual prosthesis will have had useful vision in
the past, and so they will know where to look to find a useful
sign. As such, the ability to incorporate automatic fixation of
particular optotypes (letters, numbers etc.), including
segmentation algorithms [26], will enable quick recognition
of important landmarks and signs. For many low vision
patients, the ability to see the number on an approaching
bus, or see the logo on the shop sign, would lead to a
considerable increase in independence and efficiency.
A common remark from patients who have lost
their vision is that they wish to see the faces of their friends
and family again. Face recognition is important not only for
identification of people, but also enables people to observe
facial expressions and body language that is important for
social interaction. It is important that face recognition
algorithms enable people to quickly locate and recognise
faces, with real time tracking a required attribute [27]. Face
recognition remains an ambitious goal at this point in time,
due in part to restrictions on the number of electrodes in
visual prosthesis arrays, but will lead to large improvements
in quality of life once achieved.
One of the key objectives with the image
processing for visual prostheses should be to develop
systems that are flexible. It is likely that people will desire
different visual functions in different environments, and so
the ability to change modes is an attribute desired both by
patients and their clinicians.

5. CONCLUSION

The research field of vision prostheses is rapidly growing,
and the potential therapeutic benefit for patients is
promising. The strength of this research lies in the
multidisciplinary collaborations between scientists,
engineers, clinicians, surgeons and patients, and the role of
image processing will be significant.
When designing visual prosthesis systems, it is vital
to consider the baseline visual status of a patient, and also
their wants and needs. A device will only be as useful as the
patient makes it, and so it is important that we consider their
requirements at every stage of development. With
collaboration and inventive thought, the future of vision
restoration technology looks bright.

6. ACKNOWLEDGEMENTS

The work of Bionic Vision Australia is supported by the
Australian Research Council (ARC) through its Special
Research Initiative (SRI) in Bionic Vision Science and
Technology grant to Bionic Vision Australia (BVA).

Support was also received from the Centre for Clinical
Research Excellence 529923 (Translational Clinical
Research in Major Eye Diseases) and NH&MRC
Practitioner Fellowship 529905 (RHG).

The Centre for Eye Research Australia receives Operational
Infrastructure Support from the Victorian Government.

The authors acknowledge our collaboration with Assoc/Prof
Nick Barnes and the NICTA Computer Vision Research
Group.

1542
7. REFERENCES

[1] M. Chizzolini, A. Galan, E. Milan, et al., "Good
epidemiologic practice in retinitis pigmentosa: from
phenotyping to biobanking," Curr Genomics, vol.
12, pp. 260-6, Jun 2011.
[2] J. L. Stone, W. E. Barlow, M. S. Humayun, et al.,
"Morphometric analysis of macular photoreceptors
and ganglion cells in retinas with retinitis
pigmentosa," Arch Ophthalmol, vol. 110, pp. 1634-
9, Nov 1992.
[3] A. Santos, M. S. Humayun, E. de Juan, Jr., et al.,
"Preservation of the inner retina in retinitis
pigmentosa. A morphometric analysis," Arch
Ophthalmol, vol. 115, pp. 511-5, Apr 1997.
[4] D. C. Hood, C. E. Lin, M. A. Lazow, et al.,
"Thickness of receptor and post-receptor retinal
layers in patients with retinitis pigmentosa
measured with frequency-domain optical coherence
tomography," Invest Ophthalmol Vis Sci, vol. 50,
pp. 2328-36, May 2009.
[5] M. S. Humayun, M. Prince, E. de Juan, Jr., et al.,
"Morphometric analysis of the extramacular retina
from postmortem eyes with retinitis pigmentosa,"
Invest Ophthalmol Vis Sci, vol. 40, pp. 143-8, Jan
1999.
[6] B. W. Jones and R. E. Marc, "Retinal remodeling
during retinal degeneration," Exp Eye Res, vol. 81,
pp. 123-37, Aug 2005.
[7] M. S. Humayun, J. D. Dorn, L. da Cruz, et al.,
"Interim results from the international trial of
Second Sight's visual prosthesis," Ophthalmology,
vol. 119, pp. 779-788, Jan 11 2012.
[8] A. K. Ahuja, J. D. Dorn, A. Caspi, et al., "Blind
subjects implanted with the Argus II retinal
prosthesis are able to improve performance in a
spatial-motor task," Br J Ophthalmol, vol. 95, pp.
539-543, 2010.
[9] E. Zrenner, K. U. Bartz-Schmidt, H. Benav, et al.,
"Subretinal electronic chips allow blind patients to
read letters and combine them to words," Proc Biol
Sci, vol. 278, pp. 1489-1497, 2011.
[10] J. F. Rizzo, 3rd, J. Wyatt, J. Loewenstein, et al.,
"Perceptual efficacy of electrical stimulation of
human retina with a microelectrode array during
short-term surgical trials," Invest Ophthalmol Vis
Sci, vol. 44, pp. 5362-9, Dec 2003.
[11] J. F. Rizzo, 3rd, J. Wyatt, J. Loewenstein, et al.,
"Methods and perceptual thresholds for short-term
electrical stimulation of human retina with
microelectrode arrays," Invest Ophthalmol Vis Sci,
vol. 44, pp. 5355-61, Dec 2003.
[12] M. S. Humayun, J. D. Weiland, G. Y. Fujii, et al.,
"Visual perception in a blind subject with a chronic
microelectronic retinal prosthesis," Vision Res, vol.
43, pp. 2573-81, Nov 2003.
[13] S. H. Greenwald, A. Horsager, M. S. Humayun, et
al., "Brightness as a function of current amplitude
in human retinal electrical stimulation," Invest
Ophthalmol Vis Sci, vol. 50, pp. 5017-25, Nov
2009.
[14] A. Colenbrander, "Visual Standards - Aspects and
Ranges of Vision Loss with Emphasis on
Population Surveys," International Council of
Ophthalmology, Sydney, Australia, April
20022002.
[15] I. L. Bailey, A. J. Jackson, H. Minto, et al., "The
Berkeley Rudimentary Vision Test," Optom Vis
Sci, vol. 89, pp. 1257-64, Sep 2012.
[16] A. K. Bittner, P. Jeter, and G. Dagnelie, "Grating
Acuity and Contrast Tests for Clinical Trials of
Severe Vision Loss," Optom Vis Sci, 2011.
[17] M. Bach, M. Wilke, B. Wilhelm, et al., "Basic
quantitative assessment of visual performance in
patients with very low vision," Invest Ophthalmol
Vis Sci, vol. 51, pp. 1255-60, Feb 2010.
[18] K. Schulze-Bonsel, N. Feltgen, H. Burau, et al.,
"Visual acuities "hand motion" and "counting
fingers" can be quantified with the freiburg visual
acuity test," Invest Ophthalmol Vis Sci, vol. 47, pp.
1236-40, Mar 2006.
[19] L. N. Ayton, R. H. Guymer, and C. D. Luu, "How
Blind Is A Flat ERG? - An Improved Method Of
Assessing Retinal Function In Patients With
Extremely Poor Vision," in Association for
Research in Vision and Ophthalmology, Ft
Lauderdale, USA, 2012.
[20] G. Dagnelie, "Psychophysical evaluation for visual
prosthesis," Annu Rev Biomed Eng, vol. 10, pp.
339-68, 2008.
[21] G. Dagnelie, Visual Prosthetics: Physiology,
Bioengineering, Rehabilitation. New York:
Springer, 2011.
[22] US Food and Drug Administration, "Investigational
Device Exemption (IDE) Guidance for Retinal
Prostheses," ed. USA, 2009.
[23] N. Barnes, P. Lieby, H. Dennett, et al., "Mobility
Experiments With Simulated Vision and Sensory
Substitution of Depth " presented at the Association
for Research in Vision and Ophthalmology, Fort
Lauderdale, FL., 2011.
[24] P. Lieby, N. Barnes, C. McCarthy, et al.,
"Substituting depth for intensity and real-time
phosphene rendering: visual navigation under low
vision conditions," Conf Proc IEEE Eng Med Biol
Soc, vol. 2011, pp. 8017-20, 2011.
[25] C. McCarthy, N. Barnes, and P. Lieby, "Ground
surface segmentation for navigation with a low
1543
resolution visual prosthesis," Conf Proc IEEE Eng
Med Biol Soc, vol. 2011, pp. 4457-60, 2011.
[26] L. Horne, N. Barnes, C. McCarthy, et al., "Image
segmentation for enhancing symbol recognition in
prosthetic vision," Conf Proc IEEE Eng Med Biol
Soc, vol. 2012, pp. 2792-5, Aug 2012.
[27] X. He, J. Kim, and N. Barnes, "An face-based
visual fixation system for prosthetic vision," Conf
Proc IEEE Eng Med Biol Soc, vol. 2012, pp. 2981-
4, Aug 2012.


1544