Professional Documents
Culture Documents
Muhammad Sulayman
PhD Student Department of Computer Science The University of Auckland msul028@aucklanduni.ac.nz
Supervised by:
Abstract
It is observed that in recent years small and medium Web companies have emerged very rapidly and thousands of such companies are in existence all over the globe. To cater the needs of such companies, a new field of research was created Web Engineering, given than Web engineering differs from traditional software engineering in numerous ways, which include the need of agile process models, extended modelling techniques (WebML), Navigational development techniques, different architectures and rapid application process along with different testing techniques. [12] [13] [15] [16 [25] [27]]. It has been observed that Software process improvement emerges as one of the biggest challenges for such companies [12]. A systematic literature review (SLR) has been conducted to identify and discuss the existing models and techniques used by small and medium Web companies. Important phases of our SLR included identification of the research questions to be investigated; primary and secondary database searches to identify relevant literature; data extraction from selected studies; data synthesis to formulate answers; and formal discussion to identify trends and research gaps. A total number of 88 studies were selected, after being filtered using an initial inclusion and exclusion criteria. Surprisingly, further inspection revealed only 4 relevant studies on the topic. A careful evaluation of studies was performed using qualitative as well as quantitative checklists; extracted data were further synthesized to answer the probed research questions. The identification of research gaps and possibilities of further research were explored.
1 Table of Contents
Table of Contents
Table of Contents.............................................................................................................................. 1 1. 2. Rationale of the Research ......................................................................................................... 2 Systematic Literature Review Process ....................................................................................... 3 2.1 2.2 2.3 3. 4. Formal Definition .................................................................................................................... 3 Motivation & Benefits ............................................................................................................. 3 The Process ............................................................................................................................. 4
Formulation of Research Questions .......................................................................................... 5 Identification of Relevant Literature ......................................................................................... 6 4.1 4.2 4.3 4.4 4.5 Searching Procedure ............................................................................................................... 7 Searching Stages ..................................................................................................................... 8 Primary Search Phase.............................................................................................................. 8 Primary Search Execution ..................................................................................................... 10 Inclusion & Exclusion Criteria................................................................................................ 11
5. 6.
Selected Studies ...................................................................................................................... 12 Study Quality Assessment ....................................................................................................... 12 6.1 6.2 6.3 Checklist for Qualitative studies ........................................................................................... 12 Checklist for Quantitative studies ......................................................................................... 13 Studies Quality Assessment Results...................................................................................... 13
7. 8.
Data Extraction ....................................................................................................................... 14 Data Synthesis & Results ......................................................................................................... 14 8.1 8.2 8.3 8.4 8.5 8.6 Research Question 1 ............................................................................................................. 15 Research Question 2 ............................................................................................................. 19 Research Question 3 ............................................................................................................. 22 Research Question 4 ............................................................................................................. 23 Research Question 5 ............................................................................................................. 23 Drawbacks Found among the Existing Studies ..................................................................... 24
9.
A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies
3 Systematic Literature Review Process the traditional systems and revolves around different quality dimensions [34]. RIA (Rich Internet Applications) is using the above specified Web Engineering practices [35]. The fact that the engineering of Web applications differs from the engineering of software applications motivated this work. As previously illustrated, many development methodologies and techniques were proposed specifically to tackle issues associated with Web applications development and project management. Therefore SPI for small and medium Web enterprises also seemed a relevant research topic to be investigated, which is the objective of this systematic literature review and automatically also the objective of this research. We focus explicitly on Web companies, which are characterized by companies that only provide Webrelated services such as Web application development, Web hosting and Web data management etc. The above mentioned studies and facts laid the foundation of our investigation. We observed different approaches for the various artefacts of engineering Web. In addition, a large number of Web applications are developed in small and medium-sized development companies. Therefore, the purpose of this systematic review (SR) is to gather evidence about process improvement initiatives observed for small or medium sized web companies.
A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies
Largely benefits researchers, PhD students and industry practitioners by identifying the state of practice/research and gaps in the existing research used to plot new research efforts.
A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies
5 Formulation of Research Questions Fig.1 Systematic Literature Review Process Flow Chart
6 Identification of Relevant Literature Table 1: Summary of PICOC Population Intervention Small & Medium Web Development Organizations Using Software Process Improvement Various Practices of Software Process Improvement, Corporate Software Process Improvement Models i.e. CMMI, ISO, SPICE etc., Quality Assurance Activities Used by Small and Medium Web Organizations Measure of success showing how successful SPI approaches have been to small and medium Web organizations Industry, practitioners and consultants from Small & Medium Web Development Organizations, small & large-scale tasks
Outcomes Context
The question identification process was iterative. Below are the research questions that we finalized and investigated after various revisions. Research Question 1 Which software process improvement models/techniques are followed by small and medium Web development organizations? Research Question 2 Which software process improvement models/techniques were successful to small and medium Web development organizations and how success is being measured? Research Question 3 Are there any software process improvements models that have been specifically made to measure for small and medium Web companies? Research Question 4 What are the important characteristics of small and medium Web organizations that follow software process improvement activities and practices? Research Question 5 What constitutes a small or medium Web organization for the studies investigated?
A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies
Table 2: Terms Derived from PICOC Population Intervention Small & Medium Web Development Organizations, Software Process Improvement Software Process Improvement, Software Quality Assurance, Small and Medium Web Development Organizations/Companies, Software Process improvement Models, Software Process Improvement Techniques SPI Success, Measures of SPI success Industry, practitioners and consultants, small & large-scale tasks
Outcomes Context
Table 3: Terms Derived from Synonyms Basic Term Software Process Improvement Alternate Term Software Process Enhancement, software Process Enrichment, Software Maturity Attitude Excellence Setting / Business / Organisation/ Enterprise/Company Evaluation, Appraisal, Review Estimation, Capacity, Capability CMMI, SPICE, ISO-15504, PRINCE II Internet / WWW / World Wide Web Development Small & Medium Web Companies/ Internet Companies
Quality Organization Assessment Measurement CMM / SW-CMM Web Small & Medium Organizations
Web
Table 4: Concatenation of Similar Terms with Boolean OR (Software Process Improvement OR Software Process Enhancement OR Software Process Enrichment OR Software Maturity Attitude ) (Software Process Assessment OR Software Process Evaluation OR Software Process Appraisal OR Software Process Review) (Software Process Estimation OR Software Process Measurement OR Software Process Capacity
A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies
8 Identification of Relevant Literature OR Software Process Capability) (Software Process Improvement Goals OR Software Process Improvement Objectives OR Software Process Improvement Targets OR Software Process Improvement Purpose) (Software Process Improvement Goals OR Software Process Improvement Objectives OR Software Process Improvement Targets OR Software Process Improvement Purpose ) (CMMI OR SW-CMM SPICE OR ISO-15504 OR PRINCE II) (Web OR Internet OR WWW OR World Wide Web)) (Small and Medium Web Development Organizations OR Small & Medium Web Companies OR Internet Companies)
Table 5: Concatenation of Similar Terms with Boolean AND (Software Process Improvement OR Software Process Enhancement OR Software Process Enrichment OR Software Maturity Attitude) AND (Software Process Assessment OR Software Process Evaluation OR Software Process Appraisal OR Software Process Review) AND (Software Process Estimation OR Software Process Measurement OR Software Process Capacity OR Software Process Capability) AND (Software Process Improvement Goals OR Software Process Improvement Objectives OR Software Process Improvement Targets ) AND (Measurement OR Estimation OR Assessment) (CMMI OR SW-CMM) AND (Web OR Internet OR WWW OR World Wide Web)) AND (Small & Medium Web Development Organizations OR Small & Medium Web Companies OR Small & Medium Web Development Organisations)
ACM Digital library IEEE Xplore ISI Web of Science INSPEC Science Direct
A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies
Journal of Systems and Software (JSS) Empirical Software Engineering (EMSE) IEEE Transactions on Software Engineering (TSE) Journal of Software Process Improvement and Practice Information and Software Technology Journal of IEEE Software Software Quality Journal Journal of Software Maintenance and Evolution (SME) ACM Transactions on Software Engineering and Methodology (TOSEM)
4.3.4 Conferences:
International Conference on Software Engineering (ICSE) IEEE International Symposium on Empirical Software Engineering WOSQ (International Workshops on Software Quality) EuroSPI (European Conferences on Software Process Improvement) International SPICE Conferences on Process Assessments and Improvement Australian Software Engineering Conferences
The primary search phase used the above mentioned sources as they were accessible through the University resources and a previously conducted SR on the status of Web engineering research has used most of those sources [39].
A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies
10 Identification of Relevant Literature Fig. 2 Process of Identifying Relevant Literature from the Primary Search Phase
11 Identification of Relevant Literature which abstracts suggested the presence of Web companies were considered. The PhD student Mr. Muhammad Sulayman and his research supervisor Dr. Emilia Mendes both read the titles and abstracts of the found studies and agreement was reached on the inclusion or exclusion of studies. For the remaining studies a detailed inspection of their full text was conducted. The aim is to discard bias and improve internal as well as external validity [41]. In relation to the studies whenever we were unsure they were applied to the Web companies or not we contacted their authors for clarification. Authors of 61 studies were contacted, out of which 39 replied and 3 studies were included in the SR based on the positive replies of the authors. Very surprisingly, of the 88 studies initially shortlisted and finally only 4 studies met the inclusion criteria. Table 6 summarizes the complete search process. ACM and Springer Databases have demonstrated very interesting behaviour. ACM initially retrieved 255 studies as a result of our search string and similarly Springer retrieved 187 relevant studies. On detailed investigation, 238 studies from ACM and 182 studies from Springer did not satisfy our inclusion criteria and therefore were not considered. The authors of the remaining studies i.e. 17 from ACM and 5 from Springer were contacted by E-mail for clarification on whether their studies were applied to Web companies. 39 contacted authors that form 63.93% of the contacted authors responded to our queries. The studies for which we did not receive any clarifications were unchecked on our SR. None of the studies retrieved using Springers database were included. None of the studies retrieved using the ACM database were included in our SR. Author of only one study from ACM confirmed that his study was applied on Web companies along with software companies but he was unable to provide any separate data for Web companies which made us not to select his study.
Table 6: Summary of Primary Search Results Serial No. 1 2 3 4 5 6 7 8 9 10 Database Name IEEE Explore INSPEC Scopus ISI Web of Science Pro Quest Computer Database ACM Springer Google Scholar Grey literature Number of Publications Found 15 23 13 0 3 1 17/255 (Initially 255 found) 5/187 (Initially 187 found) No new results 1 Relevant 2 1 0 0 0 0 0 0 0 1 Irrelevant 13 22 13 0 3 1 17 5 0 0
12 Selected Studies or medium Web companies, or did not investigate the use of SPI techniques and frameworks were excluded.
5. Selected Studies
After careful investigation, the study selection process short listed the following 4 studies that satisfied the inclusion criteria. S1 L. Scott, R. Jeffery, L. Carvalho, J. D'Ambra, P. Rutherford, Practical software process improvement - the IMPACT project, Proceedings of Australian Software Engineering Conference, 2001. S2 A. El Sheikh, H. Tarawneh, A survey of web engineering practice in small Jordanian web development firms, Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering, 2007 S3 P. Allen, M. Ramachandran, H. Abushama, PRISMS: an approach to software process improvement for small to medium enterprises, Proceedings of Third International Conference on Quality Software, 2003. S4 R. Naidu, Software Process Improvement of Small & Medium Organizations, MSc thesis, Department of Computer Science, University of Auckland, 2003.
1 2 3 4 5 6
Is the methodology used suitable to address the stated research questions? Does the article target the ideal population? Does the article use the research methodology adequately? Does the article discuss any of the previous work/literature? Is the study process specified in the article repeatable? Is the article biased towards one SPI framework model or
A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies
13 Study Quality Assessment technique? Do the findings address the original research questions? Yes/No/Partially Does the article document any assumptions taken? Yes/No/Partially Does the article document the procedure used to validate its Yes/No/Partially findings?
7 8 9
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Are the aims of the research clearly stated? Is the research methodology used suitable to address the research questions? Does the article target the ideal population? Was the sample used random? Was the SPI technology/framework used clearly defined? Did the study account for confounding factors? Are the measures used in study fully defined? Are the measures used in the study relevant to answer the research questions? Are the data collection methods adequately defined? If different groups are treated, are they treated equally in the study? Was only relevant data used in the study? Are any of the statistical methods used for analysis of data described? Has the use of statistical methods been motivated? Are all main findings relevant to answer the research questions? Are the negative findings presented? Has the research ignored any significant factors, either methodology or measures? Are the results compared with previous results or is it clear that there were no previous results? Does the result adequately answer the research questions?
Yes/No/Partially Yes/No/Partially Yes/No/Partially Yes/No/Partially Yes/No/Partially Yes/No/Partially Yes/No/Partially Yes/No/Partially Yes/No/Partially Yes/No/Partially Yes/No/Partially Yes/No/Partially Yes/No/Partially Yes/No/Partially Yes/No/Partially Yes/No/Partially Yes/No/Partially Yes/No/Partially
A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies
14 Data Extraction From the four studies selected, three were quantitative (S2 to S4) and one was qualitative (S1). Table 9 shows the summarized scores of each study, measured based on the quality assessment checklists.
Table 8: Assessment Scores of Selected Studies Sr. No. S1 S2 S3 S4 Quality Assessment Score 6.5/9 10/18 8.5/18 13/18 Study Type (Qualitative) (Quantitative) (Quantitative) (Quantitative)
Studies S1 and S4 presented the highest quality based on our quality assessment. Average score found for study S2, however study S3 did not score very well. These scores represent our assessment regarding the relevance of each study to support evidence to be used to answer our research questions.
7. Data Extraction
The purpose of the data extraction phase is to extract the relevant data, later to be used to prepare summary tables and quality scores, later to be used to answer SRs research questions. Data extraction was performed using two extraction forms created to extract the data needed to answer the SRs research questions and assesses the quality of each study. One of the forms stored the data extracted from for the qualitative study and the other stored data extracted from quantitative studies. Both data extraction forms are available in the appendix. The PhD student read all the related studies and filled out all the data extraction forms. The quality of data extraction was validated by the students main supervisor, who also read a portion of the selected studies. Results were compared and points of conflict were also discussed. The purpose of this activity was to remove the bias related to the understanding of the studies.
A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies
Table 9: Models and Techniques used by Small and Medium Web Development Organizations
Sr. No. SPI Model SPI Techniques No. of Companies Salient Features
S1
IMPACT Project A Generic (Plan-DoAct-Check) Model Best practices are considered as techniques such as : Web metrics and automated tools for post project analysis, Control functions for the development process models, Project Management (PM) best practices, SPI & quality management initiatives, Change management & Project Tracking. PRISMS Model based on Dynamic CMM
Iterative Approach, Model has two stages: project level & process level. Findings from the project level are applied at the process level for improvements. Provides some key practices and measures companies against them.
S2
18
S3
Iterative Approach, Definition of business goals, Key process areas (KPA) for improvement, Plan improvement, Post project Evaluation, Continuous Project
A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies
The three different SPI models and techniques used by the studies in our SR will each be briefly described below. The IMPACT Model The IMPACT model was based on the methodology of continuous learning and improvement and uses the PLAN-DO-ACT-Check principle of project management. The model is based on the IDEAL model of software process improvement [44]. Fig. 3 provides a detailed illustration of the model. IMPACT is a two staged model that comprises two cycles, are process and project cycles. These cycles drive each other and observations from one cycle are used to continuously improve the other. The Project cycle is advocated by QIP (quality improvement paradigm) which has six stages. Since the model is iterative, so there is a strong focus on understanding the process and applying improvements. The processes and improvements are measured and results are stored in process guides that are a motivation for the next cycle or iteration. Fig. 3 The IMPACT Project Model [55]
A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies
17 Data Synthesis & Results The PRISMS Model The PRISMS model is illustrated in Fig. 4.The model focuses on the relationship between requirements goals and business goals. This model is also iterative and focuses on prioritization of key process areas with the help of metrics. It also encourages the involvement of stakeholders. This model is flexible and can be tailored to any model of software process improvement. The model demonstrates the development of key process areas (KPA) and its improvement plan is based on them. Key process areas for improvement are devised after consultation with the developers, process improvement initiatives and business goals. Process improvement plans are always evaluated and are revised on feedback. Plans are always influenced by the metrics.
The Project Post-Mortem System Project post-mortem system of software process improvement for small and medium software organizations is generic to small and medium software companies but can also be adapted to Web companies. This model is motivated by the IDEAL model [44]. This approach is also iterative in nature and it uses a five steps. According to Naidu software organizations are classified into different categories according to their nature and in the first step the organization is to be identified according to a category e.g. software, Web, Embedded Systems etc. [56]. In the second step the focus is on the current state of practice and the analysis of the current software process improvement status of the organization. Based on the state of practice report this model identifies the improvement opportunity and development of the focus area that needs immediate attention. Focus areas can be prioritized according to the requirement of the organization. Process innovation is acquired in the next step of the model and it is assessed in the final step of the model. Then the post-mortem meeting is conducted and the next iteration is
A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies
18 Data Synthesis & Results planned based on the assessment of the data gathered from several questionarries. The model is very flexible in nature and more practical due to the support of generalizations, as illustrated in fig. 5.
SPI Techniques Mentioned in Survey of Jordanian Web Companies The survey of Web Engineering practices in small Jordanian Web firms is the study that focuses on some known general techniques of software process improvement and surveys them over a number of companies. These techniques include:
Change Control Management Development of Standards and Procedures Application of Web metrics Development Process Control Proper Tools and Techniques for Configuration, Process and Change Management The survey also demonstrates that the use of Web metrics is the neglected area in most of the surveyed Web companies whereas organizations are focusing on the use of automated tools and techniques for improvement.
A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies
19 Data Synthesis & Results Discussion on Research Question 1 All the four considered studies propose a certain Model or Techniques for SPI of small and medium Web development companies and there are various commonalities found among them which are detailed in Table 10. Post project analysis and process management and measurement are the techniques that have been emphasized in all four studies; followed by project tracking, feedback analysis, change management, configuration management as well as mapping of business goals with SPI. All the suggested SPI models are using an iterative approach and influenced by CMM/CMMI and there is strong tendency for use of the IDEAL model. Similarly, all the studies are suggesting a measurement and management program for processes that govern the use of Web metrics.
Table 10: Common Techniques/Features used by Small and Medium Web Development Organizations
Studies S1 S2 S3 S4
Sr. No 1 2 3 4 5 6 7 8 9 10 11
SPI Techniques/Model Features Iterative Approach Automated tools for SPI Post project analysis Best Project Management Practices Mapping of Business Goals or GQM with SPI Motivation for the use of CMM/CMMI Configuration/Change Management Project Tracking and Feedback Requirements Management Process Management and Measurement Use of IDEAL Model
A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies
20 Data Synthesis & Results Table 11: Success & Measures of Success of Different Models and Techniques found useful for Small and Medium Web Development Organizations Sr. No. S1 Successful SPI Model Successful SPI Techniques IMPACT Project A Generic (Plan-DoAct-Check) Model Success & Measures of Success Comparison of current practices with standards, PLAN-DO-ACTCHECK approach, process guides or accepted practices like CMM, RUP or ISO, Use of GQM. Success is measured by Increase in productivity, decrease in cycle time, feedback and experiences from discussion forums. Best practices are Does not state any measure of considered as techniques success. such as: commitment by the organization, Web metrics and automated tools for post project analysis, Control functions for development process models, Project Management (PM) best practices, SPI & quality management initiatives, Change management & Project Tracking
S2
S3
Assessment interviews from the customer and developers at the end of a business case workshop, Customized versions of CMM, GQM for assessment are measures of success. Project monitoring, Feedback from customers and developers and managers. The use of metrics for SPI coordination with business goals also contribute as success measures.
A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies
21 Data Synthesis & Results S4 Project Post-mortem System Questionnaire based for each AOI (Area of Interest) derived from customized CMM to scale the known outputs. Measures of success are Feedback from developers, customers, managers, Strategic value proposition, customer satisfaction, operational excellence, Reviewbased project tracking and monitoring.
All the three SPI models mentioned in studies S1, S4 & S5 were considered successful, despite the use of different measures of success. Study S1 of IMPACT model measures success as: Use of Plan-Do-Act-Check Iterative Approach for Software Development Comparison of Current Practices with Known Standards, Process Guides or Accepted Practices like CMM, RUP or ISO Effective Linkage of GQM with Collected Measures Feedback and Experience of Process Participants Decrease in Cycle Time Increase in Productivity Study S4 of PRISMS model measures success as: Proper Definition of Business Goals Motivated by GQM Definition of Key Process Areas for Improvement Preparation of SPI Plan Metrics for SPI Mapping with Business Goals and their Evaluation on Feedback Project Tracking Activities Creation of Measurement Baselines Study S5 of Project Post-Mortem model measures success as: Creation of Focus Points based on Organizational Goals and Strategy Focus Pints Determine Areas of Interest (AOI). They include Process Management, Requirements Management, Requirements Development, Verification& Validation, Configuration Management, Relationship Management, and Product Innovation. Questionnaire Based Measurement Feedback from Developers, Customers, Managers Operational Excellence Review based Project Tracking and Monitoring
A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies
22 Data Synthesis & Results Studies S2 considered various techniques to be successful that include: Commitment by the Organization Towards the SPI Initiatives Definition of Set Standards and Procedures and their Application Use of Web Metrics Change Control and Configuration Management Use of Proper Automated Tools & Techniques However, Study S2 does not indicate any specific measures of success with respect to SPI. The four studies demonstrate certain measures of success and also exhibit certain common factors that are present in different studies. Table 12 exhibits the various measures of success found across the five studies. Increase in productivity, reduced time for development, client and development team satisfaction, operational excellence and feedback from discussion are considered as the most important measures of success by four of the five studies included in SR. Table 12: Comparisons of measures of success across selected studies
Studies S1 S2 S4 S5
Sr. No. 1 2 3 4 5 6 7 8 9 10
Measures of Success Increase in Productivity Reduced Development Time Client Satisfaction Development Team Satisfaction Operational Excellence Compliance with Existing Standards Feedback from Discussion Statistical Analysis/Use of Metrics Alignment with Business Goals Project Monitoring Through Reviews
23 Data Synthesis & Results engineering, some of them are also mentioned in Section 1 of this SR. Therefore, it is our argument that the way in which an organization identifies process improvement opportunities needs to be specialized for the Web.
Table 13: Characteristics of Small and Medium Web Development Companies that follow SPI
Sr. No. Web Projects Type Companys Age Target Market/s Total Turnover Average No. of Employees in Project Average Project Duration Average Project Cost Process Model (s) Used
A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies
24 Data Synthesis & Results Study S4 stated that small organizations constitute from three to twenty employees whereas medium constitutes from twenty to fifty employees. The average duration of a project for a small company is around six months and for a medium company is around a year. Table 14 summarizes the findings of the selected studies against Question 5. It indicates certain commonalities like, small and medium Web companies operate under tight budget constraints and with short deadlines. They like their strategies to be less risk based and they always demand quick results. Table 14: Characteristics of Small and Medium Web Development Companies
Sr. No. S1
Characteristic Budget Deadline Commitment Time Frame Strategy Visibility Number of Employees Financial Position Strategy Results Number of Employees Project Duration
Small Company Low Short Short Term Tight Intensive Increased for Customers < 10 Tight Low Risk Based Quick 3-20 6 Months
Medium Company
S2
S3
S4
A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies
25 Research Gaps
9. Research Gaps
The number of studies that met our inclusion criteria is very small, therefore the gaps identified are very wide and consequently represent a large research potential. Web Engineering is comparatively a new emerging discipline [17] where its technology also changes very rapidly as new standards, tools and protocols are being introduced [22] [49]. Therefore, investigation of SPI in the context of Web application development companies formulates an interesting research case. Selected studies S1, S3 & S4 indicate integration of existing SPI models for Web Companies but the integration phenomenon is not completely narrated and the studies do not indicate how to tailor new Web standards and procedures along with the existing SPI models. The systematic review did not identify any specific SPI model or technique made to measure for Web companies; therefore there is a clear research gap on the proposal of a specific SPI model for Web companies keeping in view the constraints and challenges. This can be achieved by extending some existing model or by proposing one from scratch. The evidence comes from the recent research where new development models have been proposed to deal specifically with Web projects e.g. new size measures for Web cost estimation [57], UML based Web engineering [51], OOHDM for Hypermedia Web applications [26], navigational design techniques(NDT) for Web [30] etc. Therefore, we argue that Web specific models or practice may also be applied to SPI frameworks that are specific to the Web. Therefore, there is a visible research gap to investigate result oriented, cheaper and lesser time consuming SPI strategy for small and medium Web companies. Selected studies S1 & S3 have indicated that most Web companies have tight financial position and they are relatively low budgeted companies. S2 and S3 have indicated that their strategies are intensive and they demand immediate results within limited time. Corporate SPI giants like SEI and ISO are also formulating focus groups to make their models more practicable and feasible for smaller sized companies [45] [46] but their explicit focus so far is not focused solely on Web companies. Therefore, there is a visible research gap to investigate the success factors for the SPI programmes targeted towards Web companies. The different context and nature of Web projects makes an interesting case to investigate how SPI should be tailored to them and what factors can be influential regarding its success. The outcomes of above research gaps would be some bridges that might help Web companies in addressing their immediate concerns and goals regarding the processes which will enhance the possibilities for longer term SPI commitment. In this case they might be able to reap the benefits of an SPI program since their very inception.
10. Conclusion
Conducting a systematic literature review is a more exhaustive task as compared to traditional literature reviews [36]. It is performed by following the provided guidelines. The overall effort for conducting a systematic review is always very planned and disciplined in its case. Development of the review protocol for this systematic review was the most important and time
A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies
26 References consuming task as all other activities are based on the review protocol which streamlines the overall approach. Systematic reviews are an excellent source to investigate and synthesize the existing literature in order to obtain validated research gaps [39]. In this systematic review, we have investigated the current evidence of software process improvement in the context of Web companies. Due to our strict inclusion criteria, the number of relevant studies found was very small but the overall search process was very comprehensive and it was performed by following the recommended guidelines according in [36], to the best of our knowledge. Similarly data extraction and synthesis phase is also performed as prescribed by the practitioners with proper validation and quality assurance. Four studies were included in this SR and the number of investigated research questions was also five. Main objective of the SR was to investigate specific SPI models or techniques for small and medium Web companies. Found studies suggested did not suggest any specific model or technique made to measure for the SPI of Web companies. The SR also revealed the characteristics of some small and medium companies and suggested that they have tight budget constraints, have tight deadlines and they have a short term strategy. Investigated measures of success for small and medium Web companies include development team and client satisfaction, increase in productivity, compliance with standards and overall operational excellence. The review as mentioned earlier helped us in identifying the possible research gaps and directions. One of the research gaps lies in proposing a specific SPI model for Web companies, which keeps in view their characteristics and aims to help them measure their success and improve continuously. This can be achieved by either enhancing some existing SPI model or by proposing one from scratch. This is the line of our future work.
11. References
[1] R.L. Glass, Software Creativity, Englewood Cliffs, New Jersey: Prentice-Hall, 1995. [2] L. Harjumaa et. al. Improving Software Inspection Process with Patterns Proceedings of the Fourth International Conference on Quality Software, 2004 [3] M. Lepassaar, T. Makinen, Integrating Software Process Assessment Models using a Process Meta Model, Proceedings of IEMC: Volume I, IEEE, 2002. [4] G. Cugola, and C. Ghezzi, Software Processes: A Retrospective and a Path to the Future. Software Process Improvement and Practice, 4, pp. 101-123,1998. [5] H.Thompson, and P.Mayhew, Approaches to Software Process Improvement. Software Process Improvement and Practice, 3 (1), 3-17,1997.
A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies
27 References
[6] S. Zahran, Software process improvement: practical guidelines for business success. AddisonWesley Pub. Co., Reading, Mass., 1998. [7] W. Florac, R. Park, and A.Carleton, Practical Software Measurement: Measuring for Process Management and improvement, CMU/SEI-97-HB-003, The Software Engineering Institution, Pittsburgh, 1997. [8] P. Abrahamsson, Rethinking the Concept of Commitment in Software Process Improvement, Scandinavian Journal of Information Systems , 13(1), 2001. [9] J. Kuilboer, and N. Ashrafi, Software process and product improvement: an empirical assessment. Information and Software Technology, 42 (1), pp. 27-34, 2000. [10] M.Tortorella, and G. Visaggio, Empirical Investigation of Innovation Diffusion in a Software Process. International Journal of Software Engineering and Knowledge Engineering, 9 (5), pp. 595-621, 1999. [11] M. Fayad, M. Laitinen, & R.Ward, Software engineering in the small. Communications of the ACM, 43(3), pp. 115118, 2000. [12] P. Allen, M. Ramachandran, H. Abushama, PRISMS: an approach to software process improvement for small to medium enterprises, Proceedings of Third International Conference on Quality Software, 2003. [13] S. Vasudevan, D. Wilemon, Rapid application development: major issues and lessons learned," Innovation in Technology Management - The Key to Global Leadership. PICMET '97: Portland International Conference on Management and Technology , pp.484-, 1997. [14] L. Rising, N. Janoff, The Scrum software development process for small teams, Software, IEEE , 17(4), pp.26-32, 2000. [15] R. Ahamd Web engineering: a new emerging discipline, Proceedings of the IEEE Symposium on Emerging Technologies, 2005. [16] A.Ginige & S.Murugesan, Web Engineering: An Introduction, IEEE MultiMedia, 8 (1,. 2001. [17] Y. Deshpande, S. Murugesan, A. Ginige, Hansen et al., Web Engineering, Journal of Web Engineering, 1(1), 2002. [18] R. Pressman, Software Engineering: a Practitioner's Approach (6th ed.), McGraw-Hill Publishing Company, 2005. [19] Y. Deshpande, M. Gaedke, Web Engineering: Developing Successful Web Applications In A Systematic Way, 14th International World Wide Web Conference, Chiba, Japan, 2005.
A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies
28 References
[20] A. Schauerhuber, , M. Wimmer, , and E. Kapsammer, Bridging existing web modeling languages to model-driven engineering: a metamodel for WebML, Proc. of Model Driven Web Engineering, Palo Alto, CA, 2006. [21] C. Gnaho, and F. Larcher, A User Centered Methodology for Complex and Customizable Web Applications Engineering, Proceedings of First ICSE Workshop on Web Engineering, ACM, Los Angeles, 1999. [22] Y. Deshpande, S. Hansen, Web engineering: creating a discipline among disciplines Multimedia, IEEE , 8(2), pp.82-87, 2001. [23] D. Schwabe and G. Rossi, An Object Oriented Approach to Web-Based Application Design, Wiley and Sons, New York, ISSN 1074-3224,Theory and Practice of Object Systems 4(4), 1998. [24] G. Hebbron, L. Oates, A Simple Method and Tool for Web Engineering, ACM Press , Web Engineering Workshop, SEKE 02, Ischia, Italy, 2002. [25] Object Management Group, UML 2.0 Superstructure http://www.omg.org/cgi-bin/apps/doc?formal/05-07-04.pdf, 2005. Specification,
[26] J. Gomez, and C.Cachero, Information modeling for internet applications, Chapter OO-H method: extending UML to model web Interfaces (Idea Group Publishing, Hershey, PA, USA, 2003. [27] M. Lockyer, G. Hebbron, B. Oates, A teaching method & tool for Web Engineering, Proceedings of the 3rd IEEE International Conference Learning Technologies Advanced, on , pp. 284-285, July 2003. [28] C. Fraternali and F. Bongio, Web Modelling Language (WebML): a modelling language for designing web sites,WWW9 conference proceedings, 2008. [29] D. Bolchini, J. Mylopoulos, From task-oriented to goal-oriented Web requirements analysis, Web Information Systems Engineering, 2003. WISE 2003. Proceedings of the Fourth International Conference on , pp. 166-175, 2003. [30] J. Escalona, , A. Gustavo, NDT. A Model-Driven Approach for Web Requirements, IEEE Transactions on Software Engineering, 34(3), pp.377-390, 2008. [31] G. Griffiths, CASE in the third generation, Software Engineering Journal, 1994. [32] E. Potts, Proceedings of Software Process Workshop;, IEEE Computer Society Press, 1984. [33] A. Jones, M. Birtle, An Individual Assessment Technique for Group Projects in Software Engineering, SoftwareEngineering Journal, 4, (4), p.226-232. , 1989.
A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies
29 References
[34] H. Nguyen, Web application testing beyond tactics, Proceeding of Sixth IEEE International Workshop on Web Site Evolution, WSE 2004.., pp. 83-, 2004. [35] J. Preciado, , M. Linaje, S. Comai, Designing Rich Internet Applications with Web Engineering Methodologies, 9th IEEE International Workshop on Web Site Evolution WSE 2007, pp.23-30, 2007. [36] B. Kitchenham, Guidelines for Performing Systematic Literature Review in Software Engineering, EBSE Technical Report, Keele University, Version 2.3, 2007. [37] G. Noblit, and R. Hare, Meta-Ethnography: Synthesizing Qualitative Studies. Sage Publications, 1988. [38] M. Petticrew, and H. Roberts, Systematic Reviews in the Social Sciences: A Practical Guide, Blackwell Publishing, 2005. [39] E. Mendes, A Systematic Review of Web Engineering Research, International Symposium on Empirical Software Engineering, 2005. [40] B. Kitchenham, S. Charters, Procedures for Performing Systematic Literature Reviews in Software Engineering, EBSE Technical Report, Software Engineering Group, School of Computer Science and Mathematics, Keele University, UK and Department of Computer Science, University of Durham, UK, 2007. [41] C. Cochrane . Cochrane Reviewers Handbook. Version 4.2.1. December 2003. [42] I. Crombie, The Pocket Guide to Appraisal, BMJ Books, 1996. [43] A. Fink, Conducting Research Literature Reviews. From the Internet to Paper, Sage Publication, 2005. [44] B. McFeeley, IDEALSM: A User's Guide for Software Process Improvement. Handbook CMU/SEI-96-HB-001. Software Engineering Institute, Carnegie Mellon University, Pittsburgh, PE, USA, 1996. [45] Improving processes in small settings (IPSS project). http://www.sei.cmu.edu/iprc/ipssbackground.html . October, 2006. [46] ISO/IEC JTC1/SC7 Working Group 24. Available sc7wg24.gelog.etsmtl.ca/Webpage/iso-iec-sc7wg24_english.html . on: Available on:
http://www.iso-iec-
A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies
30 References
[47] Google Search, http://www.google.co.nz/search?hl=en&defl=en&q=define:model&sa=X&oi=glossary_definition &ct=title [48] Google Search, http://www.google.co.nz/search?hl=en&defl=en&q=define:Technique&sa=X&oi=glossary_definit ion&ct=title [49] J. Offutt ,Quality attributes of Web software applications. IEEE Software, March/April, 19(2):pp 2532, 2002. [50] M. Albert, V. Pelechano, J. Fons, et. al., Extracting knowledge from association relationships to build navigational models, Proceedings of First Latin American Web Congress, pp. 2-10, 2003. [51] D. Carvalho, A. Silva, Extending UWE to improve Web navigation project - a case study, IEEE International Conference on Systems, Man and Cybernetics , pp. 2608-2613, 2005. [52] IEEE Std. 20012002 Recommended Practice for the Internet Web Site Engineering, Web Site Management, and Web Site Life Cycle, IEEE, 2003. [53] M. Taylor, H. McWilliam, Methodologies and website development: a survey of practice. Information and Software Technology, 44(6):381391, 2002. [54] P. Fraternali, P. Paolini, Model-driven development of Web applications: the AutoWeb system. ACM Transactions on Information Systems(TOIS), 18(4):pp 135, 2000. [55] L. Scott , R. Jeffery, L. Carvalho, J. D'Ambra, P. Rutherford, Practical software process improvement - the IMPACT project, Proceedings of Australian Software Engineering Conference, 2001. [56] R. Naidu, Software Process Improvement of Small & Medium Organizations, MSc thesis, Department of Computer Science, University of Auckland, 2003. [57] E. Mendes, N. Mosley, S. Counsell, A replicated assessment of the use of adaptation rules to improve Web cost estimation, Proceedings of International Symposium on Empirical Software Engineering, pp. 100-109, 2003 . [58] O. Salo, Improving Software Development Practices in an Agile Fashion, Agile Newsletter 2, Agile-ITEA, pp. 8 , 2005. [59] R. Solingen, Measuring the ROI of software process improvement, IEEE Software, 21(3), pp. 32-38, 2004
A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies
31 References
[60] A.McDonald, R.Welland , Agile Web Engineering (AWE) Process, Department of Computing Science Technical Report TR-2001-98, University of Glasgow, Scotland, 2001 [61] A.McDonald, R.Welland, Agile Web Engineering (AWE) Process: Multidisciplinary Stakeholders and Team Communication, Book Chapte( Web Engineering)r, Springer, pp. 253-312, 2003.
A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies
32 Appendix I
Appendix I
Data Extraction form for Quantitative Studies
Data Item Value Additional Information
Identification of Study Year Author/s Title Reference Type Publisher Country of Study Setting Type of Study Article Peer Reviewed?
Unique Number
Journal/Conference/Report
What SPI model/technique is used by the organization? What SPI activities are considered successful to the organization being studied? How does the organization measure success and what are its indicators and how success is measured? What are the important characteristics of small and medium web organizations? Does the article propose any framework or model or technique that is specific to small and medium Web development organizations?
Quantitative Study Analysis
Are the aims of the research clearly stated? Is the research methodology used suitable to address the research questions? Does the article target the ideal population? Was the sample used random? Was the SPI technology/framework used clearly defined? Did the study account for confounding factors? Are the measures used in study fully defined?
A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies
33 Appendix I Is the methodology used suitable to address the stated research questions? Does the article target the ideal population? Does the article use the research methodology adequately? Does the article discuss any of the previous work/literature? Is the study process specified in the article repeatable? Is the article biased towards one SPI framework model or technique? Do the findings address the original research questions? Does the article document any assumptions taken? Does the article document the procedure used to validate its findings? Is the methodology used suitable to address the stated research questions? Does the article target the ideal population?
A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies
34 Appendix I
Data Extraction form for Qualitative Studies
Data Item Value Additional Information
Identification of Study Year Author/s Title Reference Type Publisher Country of Study Setting Type of Study Article Peer Reviewed? What SPI model/technique is used by the organization? What SPI activities are considered successful to the organization being studied? How does the organization measure success and what are its indicators and how success is measured? What are the important characteristics of small and medium web organizations? Does the article propose any framework or model or technique that is specific to small and medium Web development organizations?
Qualitative Study Analysis
Unique Number
Journal/Conference/Report
Is the methodology used suitable to address the stated research questions? Does the article target the ideal population? Does the article use the research methodology adequately? Does the article discuss any of the previous work/literature? Is the study process specified in the article repeatable? Is the article biased towards one SPI framework model or technique? Do the findings address the original research questions? Does the article document any assumptions taken? Does the article document the procedure used to validate its findings?
A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies
35 Appendix II
Appendix II
Study Protocol
A Systematic Literature Review of Software Process Improvement for Small and Medium Web Companies