2017 3rd International Conference on Science in Information Technology (ICSITech)

Software Development Evaluation Process Using

CMMI-Dev on Limited Resources Company
I Made Sugi Ardana, Suharjito
Computer Science Department,
BINUS Graduate Program – Master of Computer Science, Bina Nusantara University,
Jakarta, Indonesia
sugiardana@gmail.com, suharjito@binus.edu

Abstract—The quality of products is very important for a
software development industry. The product quality is strongly
influenced by the software development process. Software
development company with limited resources is hard to improve
the quality of the software produced. The method to be taken is
to improve the software development process to achieve maturity
level 2. In this study, CMMI-Dev (CMMI for Development)
version 1.3 with continuous representation is used as a reference.
SCAMPI-C with PST Tool was used to conduct the assessment of
the 5 (five) process areas of CMMI level 2. Data were collected by
interviews and observations of project documentation and the
process of software development. Unapplied practice from
assesment then be analyzed using Ishikawa Analysis to find root
of the problem refers to the SPF (Software Process Framework)
and subsequently use Pareto Analysis to determine the
improvements priority. The process which have already good are
in the process area PP (Project Planning), CM (Configuration
Management), and most of the processes in the area of PMC
(Project Monitoring and Control). Software development
processes which are need to be improved in the process area with
priority PPQA (Process and Product Quality Assurance), REQM
(Requirements Management), then PMC (Project Monitoring
and Control). The prior improvement from the SPF category
were in the order of procedure category and then process
category.
Keywords—Software Process Framework; CMMI-Dev;
SCAMPI; Ishikawa Analysis; Pareto Analysis
I. INTRODUCTION
As the time goes by, the software development will
continue to evolve as the complexity of feature additions,
change business processes, error recovery, and application of
new technologies. Thus, the high quality of the software
become a vital need [1]. To get a high-quality software, a good
process quality of making software is required. Software
Engineering Institute (SEI) said that the quality of a system or
product determined by the quality of the process and the
maintenance [2]. The most important thing that affect the
success or failure of software development is the process of
making the software [3].
Software development industry that has limited resources
continuously strives to improve the quality and delivery time of
software being made. This is done by improving software
development processes. But apparently there are still some
projects that are not in accordance with the expectation of the
company and the client. For example, there is a delay of
implementation and there are many changes in the requirement
without being followed by increased prices resulting in
increased production costs and reduce the company's revenue.
There are some guidelines to help software developer results in
a process in accordance with needs. CMMI (Capability
Maturity Model Integration) is one of the guides that focus on
the software development process.
To survive in the competition of making software, software
development company creates software development strategies
by optimizing processes so that software development is not
too dependent on people, but rather on processes that have
been standardized. If related to the maturity level of CMMI,
then the level that wants to be achieved is a level 2 (Managed)
where at this level has been ascertained that the process is
planned and executed according to the policy (policy), the
project team consists of people who are competent so that the
results can be measured, involving relevant stakeholders, the
project is monitored, controlled, reviewed, and evaluated, as
well as job status can be determined at any time [2, 4].
In this study, CMMI will be used to evaluate the software
development process in limited resources company. From the
results of the evaluation can be suggested any
recommendations regarding the process that needs to be
enhanced and which processes are already good and should be
maintained.
II. RELATED WORKS
A. Software Development
Software development is a series of activities aimed to
produce a software product that suits the requirement. There
are four basic activities that are common to the development of
software, namely, software specification, software
development, software validation and software evolution [5].
For different type of systems require different development
processes. Increasing the hardware performance resulted the
need for software development to be increasingly large and
complex.

978-1-5090-5864-8/17/$31.00 ©2017 IEEE 319

 2017 3rd International Conference on Science in Information Technology (ICSITech)
By using an informal approach to face an increasingly
complex development, the development of software gets
increasingly ineffective. This resulted in greater costs, more
time, as well as lower quality. To overcome the ineffectiveness,
of that sparked an idea to do software development with
engineering approach, resulting a more efficient, standardized
and scalable software development.
B. SDLC (System Development Life Cycle)
SDLC is a cyclical process of system development. The
entire process of system development is done through many
steps from the beginning until the formation of the product
through the analysis, design, implementation and maintenance.
Along with the development of software engineering, many
models have been developed to assist the software
development process, such as the waterfall, spiral, prototyping,
XP, and RAD. Although many models will evolve, but it
generally follows the basic stages of the SDLC in waterfall
development model, which consists of the analysis and
definition of requirements, system and software design,
implementation and unit testing, integration and system testing,
operation and maintenance [5] as in Fig. 1.
Fig. 1. Basic Stages of SDLC.
C. Software Development Process Framework
To obtain good results when developing software, the SEI
issued an operational framework that refers to the Capability
Maturity Model, called SPF (Software Process Framework).
The SPF stated that the framework forms the link between the
elements of the operation, namely policy, standard, process,
procedure, training, tools [6]. Framework that is formed will
be used to define and design the information, where it can be
used to help establish the necessary documents. Policy is used
as a rule directing, guiding, or limit the operation. Standard is
to provide operational definitions or criteria for acceptance of
the product or process. Process is to describe what is done in
the operation in accordance with the policy and standards.
Procedure describing instruction steps in implementing a
process. Training is to give individuals a knowledge and skills,
including training for the organization's policies, standards,
processes, procedures, and tools. Tool is to provide the
required assistance in supporting policy, standards, processes,
procedures, and training. The relation of these elements, in
establishing an operational framework is shown in Fig. 2.
320
Fig. 2. Operational Framework.
D. CMMI (Capability Maturity Model Integration)
CMMI published by the SEI (Software Engineering
Institute) of Carnegie Mellon University, is a model approach
in the assessment of the level of maturity and ability of an
organization [2]. CMMI is based on three concepts: Process
Area (PA), goal, and practice. There are two types of goal,
which are Specific Goals (SG) and the Generic Goal (GG).
There are two types of practice, which are Specific Practices
(SP) and the Generic Practice (GP). Specific Goals are a series
of activities or goals of the process area. Specific Practice is the
steps that are used to achieve goals [2]. Achievements and
fulfillment of the SG on the PA is very influential in the
maturity level achieved by the organization. Each PA will
consist of several SG and SG will consist of several SP [2].
E. CMMI-Dev (CMMI for Development)
CMMI-Dev is a model framework issued by the SEI
(Software Engineering Institute) of Carnegie Mellon University
in late 2001 and published in August 2006 to replace a similar
concept that is CMM that has been used for the assessment
process since the 1990s. CMMI-Dev was made to avoid the use
of various CMM models separately. Therefore, the model that
has been integrated in the CMMI for Development is the
codification of CMM models [2]. CMMI-Dev consists of 22
Process Areas which are: CAR (Causal Analysis and
Resolution), CM (Configuration Management), DAR (Decision
Analysis and Resolution), IPM (Integrated Project
Management), MA (Measurement and Analysis), OPD
(Organizational Process Definition), OPM (Organizational
Performance Management), OPF (Organizational Process
Focus), OPP (Organizational Process Performance), OT
(Organizational Training), PI (Product Integration), PMC
(Project Monitoring and Control), PP (Project Planning),
PPQA (Process and Product Quality Assurance), QPM
(Quantitative Project Management), RD (Requirements
Development), REQM (Requirements Management), RSKM
(Risk Management), SAM (Supplier Agreement Management),
TS (Technical Solution), VAL (Validation), VER
(Verification) [2].
There are four categories of process area which are: Process
Management, Project Management, Engineering, and Support.
The relationship between process areas, categories, and
maturity level can be seen in Table I [2]. In the organizations
with maturity level 2, the job status can be known by
 2017 3rd International Conference on Science in Information Technology (ICSITech)

management at certain points, for example, on major milestone TABLE II. COMPARISON OF SCAMPI CLASS
(major milestones) and on completion of a major task (major Class A Class B Class C
tasks) [2]. Evaluation team size 3–4 1–2
8 – 10
Evaluation time 10 days 3-4 days 1-2 days
TABLE I. PROCESSS AREA, CATEGORY, AND MATURITY LEVEL Min data source 3 2 1
Process Maturity Reliability and success High Medium Low
Category
Area Level Effort and cost needed High Medium Low
CAR Support 5 Direct interview Yes No
CM Support 2 Yes
SCAMPI A B C
DAR Support 3
IPM Project Management 3
MA Support 2
OPD Process Management 3 III. METHODOLOGY
OPF Process Management 3
OPM Process Management 3 A. Research Object
OPP Process Management 5 Based on the comparison table class of SCAMPI, for
OT Process Management 4
PI Process Management 3
SCAMPI C, the minimum number of data sources is 1 (one)
PMC Engineering 3 [1]. In this study the data source will be taken from two (2)
PP Project Management 2 software development project that has been completed.
PPQA Project Management 2
QPM Support 2 To select the projects that will be evaluated, which
RD Project Management 4 represents a typical software development project of the
REQM Engineering 3 enterprise, interviews will be conducted the Director of the
RSKM Project Management 2 company.
SAM Project Management 3
TS Project Management 2
VAL Engineering 3
B. Framework of Thinking
VER Engineering 3 Based on a literature review, in order to increase the
success of the project development, we should improve the
process used. Increasing development process can refer to the
F. SCAMPI best practices of CMMI-Dev.
SCAMPI stands for Standard CMMI Appraisal Method for Operations of the software development refers to the
Process Improvement. SCAMPI is composed of three classes software development process framework. Framework is
namely class A, class B, and class C, which is distinguished by already adapted to the implementation of the process areas of
the level of accuracy and the effort produced. SCAMPI-A is the CMMI. To determine which parts need to be improved, we
the most rigorous method and the only method that can need a method to measure it.
generate value (score) and judgement necessary to lead a
certified appraisal. Usually costs incurred for scampi A pretty To perform these measurements, will be using SCAMPI
big. SCAMPI-B is a method that is less formal than the method. In this research, the measurement is only to determine
SCAMPI-A, the activities are fewer than SCAMPI-A. By which parts need to be improved and not to measure the score
applying the SCAMPI-B, organization can predict the results of maturity level. As explained respectively, then SCAMPI
obtained when doing SCAMPI-A. This method does not need a class C to be chosen [7].
certified appraisal lead. SCAMPI-C is shorter, flexible, and less Based on the results of these measurements, it determined
expensive with SCAMPI SCAMPI-A and-B. Decision from the weaknesses and what improvements need to be done in
SCAMPI-C is usually used to measure the readiness of the accordance with the reference CMMI-Dev and SPF. Pareto
organization before applying CMMI. By doing SCAMPI analysis then will be used to make the priority of improvements
appraisal-C, it can easily get a gap of a process that has been [10].
carried out by the organization in comparison with the best
practices of the CMMI. Scope of SCAMPI-C can also be C. Research Stage
customized with the objective appraisal [7, 8, 9].
Stages of research in Fig. 3, started by defining the issues
For SCAMPI class C, the need to do the appraisal reduces that will be examined along with the boundary and scope. Then
the amount of effort and cost, but it reduces the accuracy and determine the objective and benefits of the research. After that
reliability as well but the accuracy and reliability are also doing study literature and taking case to be investigated.
lower. SCAMPI-C even allows reducing the number of
projects that will be investigated to zero and just do the Data collection is done after preparing the data collection
standard documented appraisal process, not the implementation instruments. With reference to the CMMI-Dev level 2, there
of the project [7]. The comparison between the three SCAMPI are 7 Process Areas had to be evaluated. But in this study only
classes can be seen in Table II. 5 Process Area from collected data will be examined, which
relates directly to the software development process, namely
the PP (Project Planning) which consists of a total of 3 Specific
Goal and 14 Specific Practice, PMC (Project Monitoring and

321
 2017 3rd International Conference on Science in Information Technology (ICSITech)
Control) , consists of a total of 2 Specific Goal and 10 Specific
Practice, REQM (Requirement Management), consists of a
total of 1 Specific Goal and 5 Specific Practice, CM
(Configuration Management), consisting of a total of 3 Specific
Goal and 7 Specific Practice, PPQA (Process and Product
Quality Assurance), consists of a total of 24 Specific Specific
Goal and Practice.
After sufficient data has available, then conduct CMMI
assessment by performing data processing of the results of
these observations by using PST Tool. From this assessment
will be obtained CMMI process that appropriate and the
process that needs to be improved. For the processes that need
to be improved, will be analyzed using Ishikawa diagrams and
refer to the SPF to find the root of the problem. To determine
the priority of process improvements, will be used Pareto
diagram, with reference to the SPF. Finally, establish
recommendations based on the results of data processing as a
whole.
Start
Research Problem CMMI Assessment Assesment
Result
Research Objective
Ishikawa Analysis Assesment
Result
Study Literature
Pareto Analysis Assesment
Result
Setup Data
Collection
Instrument Establish
Recommendation
Data Collection
End
Fig. 3. Research Stage.
D. Data Collection Method
Data collection methods in this study consist of interviews,
observation of the software development process, and analysis
of the documents associated with software development
process at software development company with limited
resources.
E. Data Processing Method
Data obtained from interviews, observation and document
analysis will be used as a reference to perform data analysis.
The analysis will be done based on the procedures used to
carry out an appraisal of the process area by using the
SCAMPI-C method. By doing the assessment will be obtained
which practice is in a good state and practice that need to be
improved. For practice that needs to be improved will look for
the root of the problem by using the Ishikawa Analysis.
Furthermore, for the prior of improvements will be analyzed
using Pareto analysis. Finally, will be made recommendations
for improvement.
322
IV. RESULT AND DISCUSSION
A. Data Collection
Interviews were conducted for the selection of projects that
will be evaluated. Based on the results of interviews regarding
the selection of projects to be evaluated, then taken software
development projects for a group insurance in the insurance
company ABC and insurance company XYZ. The data being
collected in the form of project documents and observations
results on the process of software development on these
projects.
1) Profile of Software Development Company
Software development company being used in this study
specializes in making software for insurance company. Initially
the software development process is highly dependent on the
employee so that when there are employees who resigned, the
company desperate. Likewise, the status of the project, known
only by employees who manage the project. Learning from this
experience, the company started making policies to planning,
reporting and monitoring of the project. Thus, the development
of a project can be seen at regular intervals, including when
there are issues or risks that must be escalated to management
level.
The management of the company hope the future of
software development is not too dependent on people, but
rather on processes that have been standardized. By referencing
to the maturity level CMMI, then the level that want to be
achieved is level 2 (Managed). At this level has been
ascertained that the process is planned and executed according
to the policy, the project team consists of people who are
competent so that results can be measured, involving relevant
stakeholders, the project is monitored, controlled, reviewed,
and evaluated, as well as job status can be determined at any
time.
Currently, an employee could have a different role
according to the phases of the project and its involvement in
the project. After the software project closing, an employee is
still engaged to support, because there are no support division,
which is a different team with software developers team. An
employee who became a systems analyst in a project could also
be a programmer in another project. Likewise, a system analyst
in a phase of defining requirements, then will be a programmer
when development phase and to provide guidance to the user
during implementation phase. This resulted in limited resources
working on a project.
2) Software Development Project for Insurance Group in
PT. ABC
PT. ABC previously had experienced failure in
implementing insurance software group from the previous
vendor. This causes a very high in overseeing the work of
vendors. Software being created integrating the 12 departments
that conduct insurance group operations.
Often each department which is the owner of each module
have diametrically opposed interests. Sometimes the
requirements that have been approved by a department could
be changed when discussion on other related modules from
other departments. Replacement of user in each department as
 2017 3rd International Conference on Science in Information Technology (ICSITech)

well be the next obstacle. If one user resigned and his TABLE IV. FINDING ON EACH SPF CATEGORY
replacement had worked at another insurance company before,
usually brings flow and business processes from the previous SPF Category Number of Findings
company. Procedures 10
3) Software Development Project for Insurance Group in Processes 8
PT. XYZ Policies 6
PT. XYZ also had a bad experience with a previous vendor, Standards 1
where the previous software cannot accommodate the needs of
the business of PT. XYZ. Besides, due to vendor’s dispersion,
a lack of adequate support arises in the event of a software D. Results of Pareto Analysis
problem. Conflict of interests between departments are also
Based on the analysis Ishikawa for each category of SPF,
faced by the software development project in PT. XYZ
the number of finding for each category can be summarized to
although not as hard as in PT. ABC, as previously it has been
be used for pareto analysis based on SPF category. The finding
using desktop software with integrated database. At the time of
is sorted from the most to the least finding, then each category
software development at PT. XYZ, the company had an
is calculated of count, cumulative and cumulative percentage
integrated web-based software, so it does not create from the
finding and can be seen as in Table V.
scratch. Defining the software requirement has also become a
bit easier. It done by gap analysis between existing software
and user needs. TABLE V. PARETO ANALYSIS BY CATEGORY SPF

Number of
B. Results of Process Area Assessment using SCAMPI-C SPF Category
Findings
Cumulative Cumulative %
To assess the process area using PST Tool from Dr. Procedures 10 10 40
Kneuper, artefacts on selected projects is used to replenish the Proocesses 8 18 72
tools. In this study, the artefacts are available in the form of Policies 6 24 96
project documents. For process area with its practice that no Standards
documents as object evidence, the practice has not been 1 25 100
successfully applied to the project and will be found with
yellow background. If it is not yet implemented, will be found Based on data in Table V then can be made Pareto diagram
with red background. All finding will be counted for each as shown in Fig. 4.
process area. Number of finding for each process area can be
seen in Table III. The most finding is in the PPQA (11
findings) and the least finding in the CM (2 findings).

TABLE III. FINDING ON EACH PROCESS AREA

Number of
Process Area
Findings
Process and Product Quality Assurance (PPQA) 11
Requirement Management (REQM) 6
Project Monitoring and Control(PMC) 3
Project Planning (PP) 3
Configuration Management (CM) 2 Fig. 4. Pareto Diagram By SPF Category.

Based on the Pareto diagram in Fig. 4, the priority of

C. Results of Ishikawa Analysis
Using Ishikawa Analysis for each finding that was
associated with the SPF, each finding is analyzed whether in
the category of SPF Procedures, Processes, Policies, or
Standards. All finding will be counted for each category.
The number of finding for each category are summarized in
Table IV. The most findings are in the procedures category (10
findings) and the least finding in the standards category (1
finding).
process improvement by SPF category is procedures category
and then followed by process category.
Based on the Ishikawa analysis for each process area, the
number of finding for each process area can be summarized to
be used for pareto analysis based on process area. The finding
is sorted from the most to the least finding, then each process
area is calculated of count, cumulative and cumulative
percentage finding and can be seen as in Table VI.

323
 2017 3rd International Conference on Science in Information Technology (ICSITech)

TABLE VI. PARETO ANALYSIS BY PROCESS AREA V. CONCLUSIONS

Number After evaluation of the software development process at the
Cumulative
Process Area of Cumulative
% company with limited resources using SCAMPI-C based on
Findings
Process and Product 11 11 44 CMMI-Dev version 1.3 with level 2 and analyzed with
Quality Assurance Ishikawa analysis and Pareto analysis, it can be concluded that
(PPQA) some software development process already good and some
Requirement 6 17 68 need to be improved. Software development processes which
Management (REQM) already good were process contained in the area of PP (Project
Project Monitoring and 3 20 80
Control(PMC)
Planning), CM (Configuration Management), and PMC
Project Planning (PP) 3 23 92 (Project Monitoring and Control).
Configuration 2 25 100 Software development process which need to be improved
Management (CM) were processes contained in the area of Process and PPQA
(Product Quality Assurance), REQM (Requirements
Based on data in Table VI then can be made pareto diagram Management), and a small portion in the area of the PMC
as shown in Fig. 5. (Project Monitoring and Control).
Priority of software development process improvement is
recommended using the following order. Start with establish
procedures of recording module testing, procedures of testing
results sign off, procedure of testing planning. Then conduct
training on how to perform QA, using best practice of
experience accomplished. Then establish procedure of defining
software requirement, procedure of defining software
requirements planning, procedure of definition software
requirements evaluation. Then conduct training on software
requirements definition, bi-direction software requirements
tracking. The last is establish procedures of project monitoring
and controlling and conduct training on project monitoring and
controlling.
Fig. 5. Pareto Diagram By Process Area.

Based on the Pareto diagram in Fig. 4, the priority of REFERENCES

process improvement by process area sequentially are PPQA
(Process and Product Quality Assurance), REQM
(Requirement Management), and PMC (Project Monitoring
and Control).
E. Recommendation
By combining pareto analysis by SPF category and pareto
analysis by process area, can be establish recommendation on
process improvements that can be done in the company.
The first recommendation is to establish procedures of
recording module testing, procedures of testing results signoff,
and procedure of testing planning. Then conduct training on
how to perform QA, using best practice of experience
accomplished. After that establish procedure of defining
software requirement, procedure of defining software
requirements planning, and procedure of definition software
requirements evaluation.
Next recommendation is to conduct training on software
requirements definition, bi-direction software requirements
tracking. The last recomendation is establish procedures of
project monitoring and controlling then conduct training on
project monitoring and controlling.
[1] W. Widodo, "Evaluasi Proses Pengembangan Perangkat Lunak pada
Virtual Team Development Menggunakan CMMI Versi 1.3," Jurnal
Informatika, vol. 10, no. 1, pp. 1140-1148, Jan 2016.
[2] CMMI Product Team, "CMMI for Development Version 1.3," Carnegie
Mellon University, Hanscom, 2010.
[3] J.-C. Liou, "On Improving CMMI in an Immature World of Software
Development," Journal of Information Science and Engineering, 2011.
[4] Kautsarina, "Penilaian Tingkat Kematangan Tiga Proses Area Level 2
CMMI Versi 1.2 pada Small Independent Software Vendor di Indonesia
(Studi Kasus: Inovasia)," Widyariset, vol. 14, no. 3, pp. 665-674, 2011.
[5] I. Sommerville, Software Engineering Ninth Edition, Boston: Addison-
Wesley, 2011.
[6] T.G. Oslon et.al, "A Software Process Framework for the SEI Capability
Maturity Model: Repeatable Level," Software Engineering Institute,
Pittsburgh, 1993.
[7] R. Kneuper, CMMI Inproving Software and System Developmen
Processing Using Capability Maturity Model Integration (CMMI-Dev).,
Santa Barbara: Rockynook, 2008.
[8] W. Hayes, G. Miluk, L. Ming and M. Glover, Handbook for Conducting
Standard CMMI Appraisal Method for Process Improvement (SCAMPI)
B and C Appraisals, Version 1.1, Hanscom: Carnegie Mellon University,
2015.
[9] "PIID and SCAMPI Tool (PST)," 2016. [Online]. Available:
http://www.kneuper.de/English/PIID-SCAMPI-Tool/. [Accessed 12
May 2016]
[10] D. Haughey, "Pareto Analysis Step by Step," 2016. [Online]. Available:
https://www.projectsmart.co.uk/pdf/pareto-analysis-step-by-step.pdf.
[Accessed 01 August 2016].

324