Holistic and timely monitoring of a Japanese science and technology innovation

system through an annual panel survey of experts and researchers

Masatsura Igami (igami@nistep.go.jp)
National Institute of Science and Technology Policy,
Ministry of Education, Culture, Sports, Science, and Technology,
Kasumigaseki, Tokyo 100-0013, Japan

ABSTRACT
The National Institute of Science and Technology Policy (NISTEP) expert survey on Japanese
science, technology and innovation (STI) system is an annual panel survey administered to Japanese
experts and researchers at universities, public research institutions, and private firms. It intends to
track the status of STI in Japan through 57 questions related to Japanese STI system.

The survey

provides a holistic view of the STI system in timely manner and qualitative information such as
diversity in basic research and usability of research funds, which is generally difficult to gauge based
on research and development statistics.

Owing to the originality of the data, various governmental

councils and committees have referenced the surveys results in their official documents, including
planning the fifth Science and Technology Basic Plan (STBP). This paper provides an overview of
the surveys methodology and design, and discusses changes that have occurred in Japanese STI
during the fourth STBPs implementation period between 20112015.
surveys future development are discussed.

Finally, directions for the

1. Introduction
Japans science and technology policies are promoted based on a Science and Technology Basic
Plan (STBP) that is formulated every five years.

The fourth STBP (20112015) focused on the

period between FY2011 and FY2015, and served as the basis upon which a variety of policies was
implemented.

The fifth STBP was launched in 2016.

The effective implementation of the

plan-do-check-act cycle in science, technology, and innovation (STI) policies requires evidence
regarding how Japanese STI has changed over time.
Monitoring the status of STI is a challenging undertaking.

First, its status does not change

overnight, thereby necessitating continuous monitoring in order to identify changes (i.e., continuity).
Second, assessing the national STI system requires a comprehensive point of view (i.e.,
comprehensive understanding).

Third, there are often instances wherein quantitative data

documenting STIs status are unavailable (i.e., measurability).

Fourth, the timely monitoring of the

impact of various policies is needed (i.e., timeliness).

To determine the status of Japanese STI while taking continuity, comprehensive understanding,
measurability, and timeliness into consideration, the National Institute of Science and Technology
Policy (NISTEP) conducted a panel survey that targeted Japans leading researchers and experts,
whose views were assessed by means of a set of questions intended to gauge their levels of
satisfaction regarding issues related to STI. This survey can be likened to a health monitoring system,
wherein the status of Japanese STI is the subject being monitored, and the sensor used to monitor
this subject is the recognition of the countrys leading researchers and experts (i.e., combined
wisdom).
A well-known and similar survey is the Bank of Japans Tankan survey (also known as the
Short-Term Economic Survey of Enterprises in Japan), which examines how companies perceive
their business performance, status, and prospects in relation to economic conditions.

The surveys

results are critical, and greatly impact the stock market and policies related to it.

The Cabinet

Offices Economy Watchers Survey is similar.

Hence, the methodology adopted in the present

studys survey is widely used (i.e., the qualitative assessment of phenomena through cooperation
with individuals capable of observing those phenomena).
This paper provides an overview of NISTEP Teiten survey (also known as the NISTEP Expert
Survey on Japanese S&T and Innovation System), and discusses changes in the Japanese STI system
during the fourth STBP. Potential directions are then discussed for the future development of the
NISTEP Teiten survey.

2. Overview of NISTEP Teiten surveys conducted during the fourth STBP

2-1. Questions
The NISTEP Teiten surveys conducted during the fourth STBP attempted to determine the status
of Japanese STI and its situational changes according to 57 questions asked annually, in addition to
questions specific to certain years.

The 57 questions can be categorized into either of research

personnel; research environment; industry-academia-government collaboration; basic research; and

innovation policy (see Figure 1). The survey was designed so that respondents would answer each
question based on a six-point scale ranging from insufficient to sufficient.

Responses in the

previous year were fed back to respondents. When respondents changed their answer from the
previous year, we asked them to provide a short description indicating why they changed their
position.
Figure 1: Composition of the 57 NISTEP Teiten survey questions

2-2. Respondents
The surveys respondents comprised two groups.

The first included approximately 1,000

individuals who were presidents of universities or public research institutions (PRIs), principal
investigators of large funding programs, or researchers nominated by department heads at
universities or PRIs. To obtain an intergenerational perspective, we asked each department head to
nominate three candidates: a professor, associate professor, and assistant professor (or their
equivalents).

Additionally, respondents were nominated to monitor how conditions differed

according to university size and department field.

The second group, the innovation overview group, comprised approximately 500 industry experts
(e.g., members of science and technology policy-related councils or subcommittees, executives in
charge of research and development at private firms, representatives of small or medium-sized
enterprises), individuals who bridge research and development with innovation, and those involved
in science and technology think tanks or mass media.

Based on the responses of the aforementioned researchers and experts, how did the status of
Japanese STI change between 20112015? The section 3 focuses on items wherein there were
significant increases or decreases in satisfaction when compared to the 2011 NISTEP Teiten survey.
It should be noted that the yearly response rates for these surveys were extremely high, averaging
86%.
2-3. Examples of questions and aggregate results
Figure 2 shows an example of a question and their aggregations.

This example asks about the

status of the baseline funding for executing research and development at universities and PRIs
(Q118).
Figure 2(a) includes aggregation results for a given question broken down according to attribute;
the values at the top and bottom are for 2011 and 2015, respectively.

The results show that the

satisfaction indices (i.e., values denoting satisfaction on a scale of 110) declined for many attributes,
especially in the public research institute.
satisfaction changed between years.

In Figure 2(b), respondents indicated why their levels of

By analyzing their responses, it is possible to understand the

specific contexts in which respondents attitudes changed.

The NISTEP Teiten survey report contains an analysis of all 57 questions.

More detailed

aggregations according to attribute, as well as all open-ended responses, have been published in the
form of supplementary materials1.
Figure 2: Examples of questions and their aggregate results
(a) Aggregate results according to attribute

Q1-18: Status of the baseline funding for executing research and development at universities and
PRIs
Index

-0.47
(-0.1)
-1.57
(-0.77)

2.9(148)
2.6(134)
2.3(134)
2.1(134)
2.0(121)
2.2(237)
2.1(230)
2.0(235)
1.9(227)
1.7(221)

G3

2.2(160)
2.1(153)
2.1(153)
2.0(158)
1.7(143)

Sufficient

Insufficient

University group

2.4(109)

G2

-0.97
(-0.15)

Agricultural
sciences

Medical
sciences

Index
change

3.0(108)
2.9(105)
2.6(105)
2.4(101)

Natural
sciences
Engineering

-1.06
(-0.46)

2.0(96)
3.1(248)
2.9(236)
2.8(235)
2.5(239)
2.6(218)
1.7(83)
1.5(78)
1.4(82)
1.6(75)
1.5(75)

2.5(236)
2.3(221)
2.3(223)
2.3(227)
2.3(219)

-0.53
(0.07)
-0.22
(-0.08)
-0.24
(-0.06)

-0.55
(-0.2)
-0.44
(-0.23)
3.7(203)

G4

4.0(122)
3.8(117)
3.4(117)
3.2(114)

Public research
institute

Note:

Sufficient

University

G1

3
2.7(748)
2.6(713)
2.5(720)
2.4(725)
2.3(689)

Attributes

Insufficient

Index
Index
change

Field of department

Attributes

3.5(196)
3.5(196)
3.4(206)
3.5(204)

-0.17
(0.07)

Categorization of university group was done by scientific publication share in Japan. Group 1: 5% or more; Group 2: 1%
or more and less than 5%; Group 3: 0.5% or more and less than 1%; Group 4: 0.05% or more and less than 0.5%.

http://www.nistep.go.jp/research/scisip/nistep-teiten-data (in Japanese)

(b) Reasons for change in opinion

Reasons for increased satisfaction

Reasons for decreased satisfaction

Increased funding made available by the president or

dean of university
Initiatives by the universitys executive directors.
Changes in circumstances owing to a respondent
transferring to a different institution.

President of university forced to decrease spending,

thereby resulting in less funds being allocated to
departments and academic staff for research
purposes.
Management expenses grants decreased considerably
over time, necessitating budget cuts.
Sufficient money for research cannot be procured
through facility maintenance funds alone.
Insufficient funding for conducting experiment based
research activities.
Increased cost of electricity and consumption taxes
have forced researchers to bear an increasing
proportion of expenses related to the use of common
facilities.
Funds are being depleted to cover just printing and
postage costs.

3. Situational changes of Japanese STI system during the fourth STBP

3-1. Questions for which respondents felt circumstances had improved
Table 1 shows the top 10 questions in terms of positive change with respect to satisfaction when
compared to the 2011 NISTEP Teiten survey.
must be tackled.

The fourth STBP identified important issues that

The integrated promotion of science and technology/innovation policies intended

to resolve such issues is a fundamental objective of the basic plan.

Increases or positive trends were observed in the satisfaction indices for questions related to
innovation policy (e.g., cooperation beyond the boundaries of natural sciences to address technical
issues in resolving important problems [Q34]; unified public-private efforts to implement Japanese
technology and systems overseas [Q312]; concentration on core competence in government-led
research and development [Q3-3]; implementation of strategies and national projects to resolve
important issues through cooperation between industry, academia, and government [Q32];
introduction or relaxation of regulations and/or enhancement or establishment of systems [Q37]).
With respect to reasons for higher levels of satisfaction when compared to previous years,
respondents mentioned specific national research and development programs (e.g., the
Cross-ministerial Strategic Innovation Promotion Program (SIP), the Impulsing Paradigm Change
through Disruptive Technologies Program (ImPACT), and the Center of Innovation (COI)
Program).

Concerning the introduction or relaxation of regulations, specific initiatives were

mentioned (e.g., the New Regenerative Medicine Act, Revision of the Pharmaceutical Affairs Act,
and the easing of regulations related to fuel-cell vehicles).
From these results, it is apparent that respondents felt that some progress was made in resolving

important issues during the fourth STBP.

However, the satisfaction indices absolute values

indicate that there were many questions for which a strong feeling of insufficiency remained, thereby
suggesting that further improvement is needed.
Table 1: Top 10 questions in terms of positive change with respect to satisfaction between 20112015
Rank

Question
No.

Category

Q1-19

Research
environment

Usability of research expenses in Grants-in-Aid for Scientific Research

(KAKENHI)

0.79
(0.13)

5.4

Q1-22

Research
environment

Fostering and securing of specialized personnel to handle operations

necessary for the smooth execution of research activities (URAs) (in
universities and PRIs)

0.35
(0.09)

2.4

Q3-04

Innovation
policy

Cooperation beyond the boundaries of the natural sciences to address

technical issues for resolving important issues

0.34
(0.07)

3.6

Q3-12

Innovation
policy

Unified public-private efforts to implement Japanese technology and systems

to overseas

0.32
(0.04)

2.8

Q3-03

Innovation
policy

Concentration on core competence in government-led research and

development

0.30
(0.10)

3.9

Innovation
policy

Implementation of strategies and national projects to resolve important issues

through cooperation between industry, academia, and government

0.24
(0.03)

3.6

Research
personnel

Number of foreign researchers (in universities and PRIs)

0.23
(0.09)

2.8

0.23
(0.04)

7.3

Introduction or relaxation of regulations and/or enhancement or establishment

of systems

0.16
(-0.04)

2.8

Interests for the needs of private companies (technical issues, etc.) (in
universities and PRIs)

0.15
(0.03)

10

Note 1:
Note 2:

Note 3:

Q3-02

Q1-13

Q1-20

Research
environment

Q3-07

Innovation
policy

Q2-02

Industry-academiagovernment
collaboration

Question

Effectiveness of the multi-year funds for executing of R&D

Index

Index change value 2015

4.8
Darker shades in the index change column indicate greater change in satisfaction. The top and bottom figures in each
row indicate the degree of change in satisfaction between 20112015 and 20142015, respectively.
The satisfaction index values are on a scale of zero (insufficient) to 10 (sufficient). Values of 5.5 or greater are
considered unproblematic (
), 4.55.4 not generally problematic (
), 3.54.4 insufficient (
), 2.53.4
generally insufficient (
), and less than 2.5 extremely insufficient (
).
KAKENHI is the largest competitive fund for academic research in Japan.

3-2. Questions for which respondents felt circumstances had worsened

Although progress was achieved in some areas during the fourth STBP, there were also questions
indicative of growing concerns when compared to the 2011 NISTEP Teiten survey (see Table 2).
Questions that exhibited the largest satisfaction index declines were those regarding baseline funding
for executing research and development at universities and PRIs (Q118). Based on the Basic
Policies for Economic and Fiscal Policy Management and Structural Reform 2006, the management
expenses grants of the national university corporations were reduced by 1%, and have been
consistently reduced for about ten years.

Respondents expressed that the reduction of the

management expenses grants had large impact of research and development at university.

Table 2: Top 10 questions in terms of negative change with respect to satisfaction between 20112015
Rank

Question
No.

Q1-18

Research
environment

Baseline funding for executing research and development at universities and

PRIs

-0.62
(-0.19)

2.3

Q1-06

Research
personnel

Whether or not individuals with the required capabilities are applying to

doctoral programs

-0.57
(-0.17)

3.0

Q1-24

Research
environment

Research facilities and equipment for innovative and advanced R&D; and the
training of high-quality human resources (in universities and PRIs)

-0.49
(-0.07)

4.4

Basic research

Degree of diversity in Japanese basic research as the source of future

innovation

-0.43
(-0.14)

3.0

Basic research

Degree of originality in Japanese basic research as the source of future

innovation

-0.40
(-0.16)

3.0

Research
environment

Amount of indirect funding related to the government's public appeal-based

research funds (competitive research funds, etc.) (in universities and PRIs)

-0.36
(-0.07)

4.0

Research
personnel

Multi-faceted evaluation of researchers, rather than a single indicator related to

scientific publications (in universities and PRIs)

-0.35
(-0.03)

4.5

Research
environment

Efforts to secure researchers' time allocated to R&D activities (in universities

and PRIs)

-0.31
(-0.06)

2.2

Research
environment

Circumstances of intellectual and research information infrastructure in Japan

(in universities and PRIs)

-0.30
(-0.03)

4.2

Research
environment

Government's S&T budget, taking account of current S&T situation in Japan

-0.28
(-0.16)

10

Note 1:
Note 2:

Q2-22

Q2-23

Q2-17

Q1-16

Q1-21

Q2-19

Q2-16

Category

Question

Index

Index change value 2015

2.7
Darker shades in the index change column indicate greater change in satisfaction. The top and bottom figures in each
row indicate the degree of change in satisfaction between 20112015 and 20142015, respectively.
The satisfaction index values are on a scale of zero (insufficient) to 10 (sufficient). Values of 5.5 or greater are
considered unproblematic (
), 4.55.4 not generally problematic (
), 3.54.4 insufficient (
), 2.53.4
generally insufficient (
), and less than 2.5 extremely insufficient (
).

There was also growing concern regarding a lack of diversity and/or originality in basic research.
Detailed questions were conducted in the 2014 NISTEP Teiten survey to examine changes in
research activities and behaviors among researchers at universities and PRIs (see Figure 3 and Figure
4). According to the results, respondents believed that there was growth in research for (a) the
direct purposes of resolving social issues and creating economic value, (b) to fulfill organizational
goals (e.g., regional or social contributions), and (c) to achieve integration between different
disciplines. In examining the behavior of researchers, there was a growing awareness that they
were producing output other than scientific publications, such as patents and prototypes.

Given that

the resolution of important issues was emphasized in the fourth STBP, these changes can be
considered the result of various policies that were promoted under the plan.
The results likewise show that respondents were increasingly aware that research pursuing
temporary trends was growing, whereas exploratory research to find new research themes and
pioneering research toward producing new research fields were in declinealbeit at a minimal rate.
With respect to the behavior of researchers, respondents were aware that most researchers were
strongly inclined to (a) produce short-term results, (b) conduct research with a high probability of

success, and (c) emphasize the number of scientific publications as research output (in response to
evaluation). In contrast, there was a decline in terms of researchers addressing themes emphasizing
long-term research strategies. None of these changes is desirable with respect to ensuring research
diversity.
Generally speaking, it is clear from the 201115 NISTEP Teiten survey results that there is
growing concern regarding fundamental research activities at universities and PRIs.

Figure 3: Changes in research activities at universities and PRIs over the past ten years categorized
according to respondent group and content

Research on basic technologies required by industry (material testing,

etc.)
Research that grasps fragmented knowledge from a comprehensive
viewpoint
Pioneering research toward producing new research fields

2.3

3.0
3.0

2.4
2.9
2.0
-0.6

2.6

1.0
0.0

Increasing

Research in pursuit of temporary trends

2.4

Decreasing

Research for the direct purposes of resolving social issues and creating
economic value
Research to fulfill organizational goals (e.g., regional or social
contributions)
Research to achieve integration between different disciplines

0.2

-1.0
-0.8
-1.4
-0.9

Exploratory research to find new research themes

-5.0 -4.0 -3.0 -2.0 -1.0 0.0 1.0 2.0 3.0 4.0 5.0

Innovation overview group

University and PRI group

Figure 4: Changes in research activities at universities and PRIs over the past ten years categorized
according to respondent group and researchers behaviors
Researchers who are strongly inclined to produce short-term results

4.2
4.6

Researchers who conduct research with a high probability of success

Researchers who produce research output other than scientific

publications (e.g., patents and prototypes)

2.0

3.0
2.7
2.8

Increasing

Researchers who announce results on a piecemeal basis (in response to

evaluation)

Decreasing

Researchers who place emphasis on the number of scientific

publications papers as research output (in response to evaluation)

3.3
3.7

1.9
1.8

Researchers who address research themes with emphasizing long-term -4.5

-4.7
research strategies
-5.0 -4.0 -3.0 -2.0 -1.0 0.0 1.0 2.0 3.0 4.0 5.0

Innovation overview group

Note:

University and PRI group

Respondents were asked to indicate the degree of change compared to 2005 by selecting decreased greatly, decreased,
no change, increased, and increased greatly. The results obtained after indexation were as follows: decreased
greatly (-10 points), decreased (-5 points), no change (0 points), increased (5 points), increased greatly (10
points).

4. In-depth analyses of selected questions

4-1. Monitoring the breadth of the effects of policy measures implementation
Breaking down the NISTEP Teiten survey results according to attribute widens an understanding
of the breadth of the effects of policy measures implementation.

Figure 5 shows the aggregate

results by attribute for the usability of research expenses in Grants-in-Aid for Scientific Research
(KAKENHI), the item for which improvement of the satisfaction indices were highest.
As the results show, satisfaction indices increased for each attribute.

The stated reasons for this

change included smoother annual carryover, and increased convenience owing to the introduction of
multi-year funds. Grants-in-Aid for Scientific Research (KAKENHI) is the largest competitive
fund in Japan; hence, alterations to it influence many of the countrys researchers. However, it
requires elaborative efforts.
Figure 5: Usability of Grants-in-Aid for Scientific Research (KAKENHI)

Q1-19: Usability of research expenses in Grants-in-Aid for Scientific Research (KAKENHI)

Index

G4

Note:

4.7(149)
5.3(133)
5.6(134)
5.6(133)
5.7(121)
4.3(235)
4.7(228)
5.0(232)
5.0(226)
5.1(219)

4.8(156)
5.1(149)
5.3(150)
5.6(156)
5.6(141)
4.5(191)
4.8(189)
5.0(190)
5.1(199)
5.4(197)

Attributes

Index
change

5.0(108)

0.44
(0.13)

0.91
(0.04)

Natural
sciences
Engineering

Agricultural
sciences

Medical
sciences

0.92
(0.09)

5.7(103)
5.9(103)
5.8(101)
5.9(96)
5.1(239)
5.4(228)
5.4(230)
5.5(233)
5.6(214)
4.1(78)
4.6(76)
5.0(80)
5.2(74)
5.3(72)

3.8(233)
4.0(219)
4.5(220)
4.7(223)
4.9(216)

Easy to use

0.85
(0.13)

Easy to use

Not easy to use

University group

G3

4.7(106)
4.8(101)
4.9(103)
5.0(103)
5.1(98)

Public research
institute

G2

4.5(731)
4.9(699)
5.2(708)
5.3(714)
5.4(678)

University

G1

Not easy to use

Field of department

Attributes

Index
Index
change

0.5
(0.09)
1.15
(0.09)
1.12
(0.21)

0.78
(0.11)
0.85
(0.08)
0.9
(0.26)

Categorization of university group was done by scientific publication share in Japan. Group 1: 5% or more; Group 2: 1%
or more and less than 5%; Group 3: 0.5% or more and less than 1%; Group 4: 0.05% or more and less than 0.5%.

An in-depth analysis of a question regarding university research administrators (URAs), an item

with the second-highest satisfaction index rise, was conducted. URA universities were compared to
other universities in terms of whether differences existed in circumstances related to Q122 (i.e., the
fostering and securing of specialized personnel to handle operations necessary for the smooth
execution of research activities (URAs)).
For this analysis, URA universities were defined as those that adopted the 20112014 Initiative for
the Developing a System for Fostering and Recruiting URAs or the 20132023 Initiative for the
Strengthening of Research Universities, as well as members of the Association of Research
Administrators. Of the 36 universities that fell into these categories, 32 were targeted in the

NISTEP Teiten survey2.

Figure 6(a) shows the difference between URA universities and other universities regarding
changes in satisfaction indices with respect to fostering and securing of URAs (Q122). The two
sets of universities showed similar satisfaction indices in 2011; however, the satisfaction index for
URA universities increased each year, such that by 2015 it was 0.79 points higher than in 2011.

In

contrast, the satisfaction index for the other universities barely changed.
Furthermore, the change in the satisfaction index varied according to respondents duties.
6(b) contains the aggregate results for respondents whose duties were management related.

URA universities, the satisfaction index was 1.9 for the 2011 NISTEP Teiten survey.
between 20112013 the index increased sharply, reaching 4.4 in 2015.

Figure
Among

Nevertheless,

Given that between

20112013 these universities adopted the Initiative for Developing a System for Fostering and
Recruiting URAs and/or the Initiative for the Strengthening of Research Universities, the
commencement and adoption of these programs were reflected in the form of changes in the
satisfaction index.
The satisfaction index for respondents whose duties were not managerial (see Figure 6(c)) rose
between 201115, albeit markedly less than for their manager counterparts.

This finding suggests

that although universities are progressing in fostering and securing URAs, these advances are not
having a substantial impact within institutions.

Indeed, in this question, one of the reasons cited for

a decline in satisfaction level was the benefits have not yet been felt at a department level.

Data from the following websites (each accessed on February 27, 2016) were used to prepare the list of URA universities:
Initiative for Developing a System for Fostering/securing URAs (http://www.mext.go.jp/a_menu/jinzai/ura/detail/1315871.htm),
Initiative for the Strengthening of Research Universities (http://www.mext.go.jp/a_menu/kagaku/sokushinhi/1338460.htm),
Association for Research Administrators (http://www.rman.jp/aboutus/memberlist.html).

10

Figure 6: Comparison between URA universities and other universities

Satisfaction index about fostering and

securing URAs (Total)

(a) All respondents

5.0
4.5
4.0
3.5
3.0

2.7

2.5
2.0 1.9
1.9
1.5
2011

2.0

URA universities
Other universities

2012

2013

2014

2015

Year of survey

Satisfaction index about fostering and

securing URAs (Respondents whose
duties were management related)

(b) Respondents whose primary duties are management related

5.0
4.4

4.5
4.0
3.5
3.0

2.5

2.5

2.3

URA universities

2.0 1.9

Other universities

1.5

2011

2012

2013

2014

2015

Year of survey

Satisfaction index about fostering and

securing URAs (Respondents whose
duties were not managerial)

(c) Respondents whose primary duties are not management related

5.0
4.5
4.0
3.5
3.0
2.5

2.5
2.0 1.9
1.9
1.5
2011

1.9

URA universities
Other universities

2012

2013

2014

Year of survey

11

2015

4-2. Monitoring attitudinal differences between actors

Analyzing the aggregate results by attribute makes it possible to ascertain attitudinal differences
between actors.

Respondents at the private firm, university, and government all expressed

dissatisfaction regarding the utilization of intellectual property by private firms obtained from
research and development conducted at universities and PRIs (see Figure 7).

Individuals in the

innovation overview group were particularly dissatisfied: their satisfaction index was 0.8 and 1.2
points lower than university and PRI respondents, respectively.
Respondents from private firms indicated that the quality of universities granted patent was low,
and furthermore that universities were unable to fund international applications pursuant to the
Patent Cooperation Treaty; consequently, the most of patent applications were domestic rather than
international.

In contrast, university respondents pointed out that there was significant

apprehension and an unwillingness to take risks in product development at private firms, which
prevented knowledge transfer from universities to private firms.
Figure 7: Example of a question showing attitudinal differences between actors

Q2-8: Utilization of intellectual property by private firms obtained from research and development
conducted at universities and PRIs
Index

Note2:

2.8(414)
2.8(392)
2.8(371)
2.8(384)
2.8(371)

-0.21
(0.13)
-0.05
(-0.02)

4.1(103)
4.2(91)
3.9(95)
4.0(92)
4.1(81)
3.5(170)
3.4(159)
3.3(163)
3.4(164)
3.5(156)

Sufficient

Insufficient

University group

Note1:

G2

-0.09
(0.05)

4.2(106)
4.0(104)
3.9(105)
3.8(104)
4.0(101)

Public research
institute

G1

3.7(536)
3.7(513)
3.6(520)
3.6(527)
3.6(495)

University

Innovation
overview group

-0.02
(0.1)

Attributes

Agricultural
sciences

Medical
sciences

Index
change

3.6(62)
3.5(61)
3.4(61)
3.6(59)
3.7(54)

Natural
sciences
Engineering

4.1(179)
4.0(172)
4.0(171)
3.9(174)
4.1(156)
3.8(54)
3.6(46)
3.5(53)
3.8(49)
3.8(51)

3.3(172)
3.2(165)
3.0(165)
3.1(168)
3.0(161)

0.06
(0.14)

Sufficient

Insufficient

Field of department

Attributes

Index
Index
change

0.03
(0.18)
0.03
(0.05)
-0.27
(-0.1)

-0.02
(0.12)

G3

3.6(119)
3.7(110)
3.7(110)
3.7(117)
3.8(104)

0.15
(0.07)

G4

3.8(144)
3.7(153)
3.5(150)
3.5(154)
3.5(154)

-0.35
(-0.04)

Categorization of university group was done by scientific publication share in Japan. Group 1: 5% or more; Group 2: 1%
or more and less than 5%; Group 3: 0.5% or more and less than 1%; Group 4: 0.05% or more and less than 0.5%.
Innovation overview group comprised approximately 500 industry experts (e.g., members of science and technology
policy-related councils or subcommittees, executives in charge of research and development at private firms,
representatives of small or medium-sized enterprises), individuals who bridge research and development with innovation,
and those involved in science and technology think tanks or mass media.

4-3. Complementary analysis along with quantitative data

Analyzing the NISTEP Teiten survey results in conjunction with qualitative data facilitates an
understanding of the underlying causes of changes in the quantitative data.
Respondents from universities and PRIs were of the strong opinion that individuals with the
required capabilities were not applying to doctoral programs (see Figure 8).

In examining changes

since the 2011 NISTEP Teiten survey, the satisfaction indices for all attributes either declined or

12

exhibited a declining trend. With respect to individual university groups, the satisfaction indices
for groups 1 and 4 declined by more than 0.6 points; as for various university departments, the
satisfaction indices for natural sciences, agricultural sciences, and medical sciences declined by
nearly 0.7 points. Engineering showed a smaller decline when compared to other departments,
although its dissatisfaction level was already relatively high in 2011.
Figure 8: Aggregate results by attribute for the quality of applicants to doctoral programs

Q1-6: Whether or not individuals with the required capabilities are applying to doctoral programs
Index

-0.73
(-0.06)

Applying

Not-applying

University group

-0.56
(-0.18)

4.2(77)
3.9(76)
3.7(83)
3.5(82)
3.5(77)

Public research
institute

3.7(148)
3.3(132)
3.1(134)
3.2(133)
3.0(120)
3.3(238)
3.2(228)
3.0(229)
2.9(222)
2.8(220)

G2

Agricultural
sciences

Index
change

3.6(104)
3.3(97)
3.1(101)
3.2(96)
2.9(94)

Natural
sciences
Engineering

-0.68
(-0.29)

3.0(243)
2.8(229)
2.8(228)
2.7(228)
2.7(214)
3.2(80)
3.3(78)
3.0(81)
2.8(75)
2.5(73)

3.7(224)
3.3(216)
3.2(215)
3.2(223)
3.0(214)

Medical
sciences

-0.26
(0.02)
-0.7
(-0.28)
-0.67
(-0.16)

-0.45
(-0.19)

3.7(185)
3.3(185)
3.4(185)
3.3(195)
3.0(193)

G4

-0.65
(-0.19)

-0.5
(-0.14)

3.4(152)
3.2(146)
3.1(148)
3.2(153)
3.0(140)

G3

Note:

3.5(723)
3.2(691)
3.2(698)
3.1(703)
2.9(673)

University

G1

Applying

Attributes

Not-Applying

Index
Index
change

Field of department

Attributes

-0.66
(-0.23)

Categorization of university group was done by scientific publication share in Japan. Group 1: 5% or more; Group 2: 1%
or more and less than 5%; Group 3: 0.5% or more and less than 1%; Group 4: 0.05% or more and less than 0.5%.

Figure 9: Number of students enrolled in doctoral programs

(a) Change in the number of enrollees by course
(b) Change in proportion of working enrollees
10,000 people

10,000 people

2.0

2.0

The number of enrollment

1.8

50%

Other
students

Others

1.6
Social
science

1.4

40%
1.5

37.7%

Humanities

1.2

30%

1.0

Medical
sciences

0.8

Agricultural
sciences

0.6

Engineering

0.4

Working
students

Percentage
of working
students

1.0
21.7%

20%

0.5
10%

Natural
sciences

0.2
0.0
1981 84

87

90

93

96

99

02

05

08

11 2014 FY

0.0
2003

10

2014

0%
FY

Other includes courses that do not fall under the umbrella of engineering, agriculture, health, the humanities, or
social/physical sciences.
Source: National Institute of Science and Technology Policy (2015). Japanese science and technology indicators 2015. Research
Material No. 229.
Note:

13

Figure 9 shows the change in the number of students entering doctoral programs according to the
Ministry of Education, Culture, Sports, Science, and Technologys Basic School Survey. After
peaking in 2003, the number of enrollees entered a declining trajectory; by 2014, the number had
fallen to a level nearly identical to the late 1990s (see Figure 9(a)). The temporary rise that
occurred in 2010 could be attributable to the 2008 financial crisis.
The proportion of working students enrolled in doctoral programs rose from 21.7% in 2003 to
37.7% in 2014. Whereas the number of working students has remained relatively flat since the late
2000s, the number of non-working students has declined. As for why satisfaction dropped with
respect to the question regarding the quality of doctoral program applicants (Q16), respondents
cited the absence of reliable financial support or feasible career paths. Making such statements
suggest that the result of NISTEP Teiten survey likely reflects the declining number of enrollees
among non-working students (Figure 9(b)).
5. Conclusion and future works
It can be deduced that the NISTEP Teiten surveys are effective in comprehensively monitoring the
STI systems status.

The data provided by these surveys are useful in formulating science and

technology policies, and have been referenced in official documents by various governmental
councils and committees.

Likewise, the surveys results were referenced in science and technology

white papers, in planning the fifth STBP, and in media such as newspapers. The fifth STBP
outlines initiatives to strengthen fundamental capacities, particularly with respect to fostering and
promotion of young researchers, as well as reforming and strengthening the functions of universities.
When policymakers were establishing the direction of these policies, they referred to the NISTEP
Teiten survey results as one of evidence. Hence, it is clear that the NISTEP Teiten survey provide
useful information that others cannot.

Looking forward, it should be possible to build data that are

even more useful in policy formation and evaluation by implementing surveys and related analyses
wherein the following three points are emphasized.
5-1. Understanding the relationships between questions
As mentioned previously, the NISTEP Teiten survey results can be employed in various ways;
nevertheless, they are often used in a partial manner.
examined in an interrelated fashion.

The surveys 57 questions should be

For instance, in attempting to improve upon one item, its

relationship to others ought to be considered. This requires a comprehensive understanding of the

relationships between all questions.
5-2. Further analysis of the open-ended responses
The NISTEP Teiten surveys include open-ended questions regarding to various STI issues. The
total responses to these questions between 20112015 exceeded 2.5 million Japanese characters.

14

Through text mining or machine learning, it is possible to extract words used in specific contexts.
Currently, this is a manual and time-consuming process.

Hence, a methodology based on

computerized text mining or machine learning should be developed in order to identify new issues,
as well as those that have not been discussed previously.
5-3. Analysis of the extent to which the effects of policies spread
There are some questions for which no significant changes were observed in the satisfaction
indices between 20112015 that means the effects of the policies were not felt by researchers,
perhaps because of a lack to implement relevant ones, or to do so on a limited scale.

Accordingly,

continuous monitoring is necessary that goes beyond the STBPs five-year framework, since a
sufficient amount of time is needed for conditions to improve.

However, the NISTEP Teiten survey

includes identical respondents each year, and their ages obviously increase.

Consequently, the use

of respondent groups and surveying methods that facilitate long-term chronological monitoring
while simultaneously maintaining respondent continuity should be examined.
5-4. Determining question content and identifying target respondents
The NISTEP Teiten surveys are effective in assessing the state of STI in Japan. Nevertheless, the
results depend largely on the content of questions and to whom they are directed.

For instance,

some individuals argue that because the NISTEP Teiten surveys comprise many questions related to
research environment, that the results tend to emphasize dissatisfaction among researchers.

It

should be possible to solve this problem by analyzing attitudinal differences between actors and
setting questions that examine researchers own efforts.
This paper presented a method for qualitatively determining the status of STI through a panel
survey administered to researchers and experts. Of course, determining the status of STI based on
qualitative data alone is insufficient.

Hence, utilizing qualitative data from the NISTEP Teiten

surveys in conjunction with various research and development statistics would promote a fuller
understanding of STI systems.
Currently, preparation for the new NISTEP Teiten survey, which will be conducted during fifth
STBP, is undergoing. Fifth STBP identified numerical targets/indicators that should be monitored
during the basic plan; therefore, we believe that original evidence that NISTEP Teiten survey
provides is getting more important in order to understand a context of changes of the indicators.
To authors knowledge, the NISTEP Teiten surveys are unique in their focus on monitoring the
status of STI systems, we hope that our experience will be useful for the development of new STI
indicators and widens our collective knowledge for the measurement of STI activities.

15

References
National Institute of Science and Technology Policy (2016). Analytical report for 2015 NISTEP expert survey on
Japanese S&T and innovation system (2015 NISTEP TEITEN survey). NISTEP Report No. 167 (in Japanese).
National Institute of Science and Technology Policy (2015). Japanese science and technology indicators 2015.
Research Material No. 229.

16