Categorized | Practices

ITSM Process Assessments using ISO/IEC 15504 for CSI

 This article was originally published in the itSMF Australia Bulletin

ITSM Process Assessments for CSI

While the continual service improvement (CSI) journey embarked upon by each organisation is unique, one aspect of service improvement that should be standardised is the measurement of service improvements. In other words, units of service measurement should be based on standards that communities can use for common benchmarks. Classic examples include hours and minutes as the units of measurement for time and kilometres to measure distance. Unfortunately such quantitative units of measurement are not easy to determine for IT service improvements.

The University of Southern Queensland initiated an Australian Research Council (ARC) Linkage project in 2011 to develop a standard approach to determine process capabilities for service improvement in ITSM. The research team includes ITSM practitioners and International Standards committee members. This article proposes the use of the international standard of process assessment (ISO/IEC 15504) in measuring ITSM improvements.

Measuring IT service quality is challenging since it requires a mix of quantitative and qualitative metrics. ITIL defines three types of metrics for CSI: service, technology and process metrics. The three metric categories can be extended with a list of specific metrics and common measurement methods as illustrated in Table 1.

Table 1: Key Service Improvement Metrics and Measurement Methods

Diagram 1

IT service processes define standard sets of activities to meet ITSM objectives. It makes sense to also have a standard measurement mechanism for such processes. It is important to realise the scope of ITSM process assessments in CSI. ITSM process capability is one of several metrics that can inform IT service improvements. Therefore conducting standards-based ITSM processes assessments contributes to CSI by consistently measuring improvements in IT service process quality.

Existing ITSM Process Assessment Frameworks

The existing ITSM process assessment frameworks available to the ITSM community are listed in Table 2.

Table 2: Existing ITSM Process Capability Assessment Frameworks

Diagram 2

Besides the assessment frameworks listed in Table 2, there are several proprietary and/or vendor-specific assessment services or commercial process assessment tools available in the ITSM market. However, there are strong arguments against the value of conducting ITSM process assessments in the ITSM community. Two major criticisms are prominent and discussed next.

Fallacy of “ITIL Compliance”

A review of the existing ITSM process assessment frameworks suggests that ITIL is the most used reference model for assessments. However there is a fundamental misjudgement in adopting this approach. ITIL is not designed as a unit of measurement for service process improvements. Instead ITIL is developed to provide best practice guidelines which when followed could lead to service improvements.

Recent discussions in the ITSM community suggest that there is probably a negative impact of trying to fully comply with ITIL. A more pragmatic approach would be to use ITIL where it matters, i.e. where ITIL can deliver business value by minimising risks, cutting costs, meeting service levels or increasing customer satisfaction. If ITIL is only applied when and where relevant, the metric of “ITIL compliance” can be misleading. “ITIL compliance” driven process assessments can lead to providing a capability score that provides little value. Such assessments measure how well processes align to ITIL rather than how well processes are improving in response to business requirements.

Ambiguity in Assessment Methods

The good news about the existing ITSM process assessment frameworks is uniformity in the measurement framework used. Process assessments are largely based on two similar measurement frameworks: Capability Maturity Model Integration (CMMI) and ISO/IEC 15504. Moreover these two frameworks have similar origins from the software engineering discipline and are harmonized in their structure. However, there is ambiguity due to different assessment methods based on these measurement frameworks. Achievement of a particular capability level for a process can have different meanings or interpretations when using different assessment frameworks. The lack of a transparent and consistent assessment framework hinders repeatability and uniformity in process assessment activities. This in turn tarnishes the integrity of the assessment outcome.

Lack of a transparent and uniform process assessment framework to measure IT service processes has negative implications towards establishing a widely supported industry benchmark. Consider booking a hotel: transparency of service quality in hotels is provided by the universally popular star rating system (e.g. a three-star vs. five-star hotel). Such transparency in service quality helps customers make an informed choice about accommodation based on service expectations. Unlike the tourism industry, the IT service industry lacks a similar widely accepted industry benchmark for quality and process capability.

Towards a Solution using ISO/IEC 15504 Standards

Considering the value of ITSM process assessments in CSI and the prevalent issues of the existing ITSM process assessment frameworks, this article proposes an approach of using the ISO/IEC 15504 standard and translating the assessment indicators from the standard into assessment questions that are relevant to the ITSM industry. These assessment questions are then loaded into an online survey tool with automation in the calculation of responses to determine process capability and provide improvement recommendations. This approach is termed as “Software-mediated ITSM Process Assessments using ISO/IEC 15504”.

Figure 1 demonstrates a model of the ISO/IEC 15504 standard for ITSM process assessments. The measurement framework of ISO/IEC 15504 presents five levels of process capability which are similar to the levels in CMMI. ISO/IEC 15504 provides a concentrated focus on specific aspects of process capability which is defined by nine process attributes. For example, at capability level 3 (CL3), process attribute 3.1 (PA3.1) examines the standard definition of a process and PA3.2 looks at the implementation of a process based on the defined standard process.

figure 1

Figure 1: ITSM Process Assessment Model based on ISO/IEC 15504

A very important feature of this assessment framework is the use of a consistent reference model, measurement framework and assessment model all based on the same standard, i.e. ISO/IEC 15504. The process reference model is provided by ISO/IEC 20000 part 4 which is designed to provide a consistent unit of measurement to determine capability level 1. The measurement framework is presented in part 2 of ISO/IEC 15504. Finally, part 8 of ISO/IEC 15504 provides an exemplar process assessment model for ITSM with all the necessary indicators to assess achievement of the process attributes in all capability levels.

The ISO/IEC 15504 standard has been adopted by ISACA in its COBIT assessment program. Therefore use of ISO/IEC 15504 can bring consistency in an organisation’s assessment of overall IT governance and ITSM processes. An extensive literature search of the academic and industry publications suggests that the assessment framework presented by ISO/IEC 15504 provides the most transparent and detailed process assessment model available for ITSM today. Given its transparent orientation, this framework has a potential to achieve industry wide acceptance for consistent IT service process quality measurement and benchmarking.

Moreover, software-mediated ITSM Process Assessment proposes the use of software as a survey tool in performing process assessments. Besides conducting surveys based on the standard assessment questions and performing standard-based calculations to determine process capability levels, the software tool is embedded with a knowledge management database to store process improvement recommendations to improve service processes in order to fully achieve a particular capability level. Such gap analysis can be conducted with the help of a knowledge base populated from the ITIL guidelines. Rather than using ITIL for an unreliable measurement, this is the right place to use ITIL to determine process improvement recommendations.

Even though software-mediated process assessment tools using the ISO/IEC 15504 standard in ITSM can be a promising prospect for CSI, this tool cannot be expected to provide a turn-key solution. There is still a need to understand the organisational context to determine business value and improvement priorities for service improvements. Discussions among ITSM process management team members and external consultants is important for ITSM process improvements. Software-mediated ITSM process assessments can trigger and facilitate such discussions.

In summary, use of the ISO/IEC 15504 standard as a process assessment framework in ITSM overcomes two key challenges in the existing ITSM process assessment arena: use of ITIL as a false unit of measurement and lack of transparency in assessment methods. Furthermore, a software tool can be implemented following the ISO/IEC 15504 standard guidelines that can assist assessors to conduct ITSM process assessments in a more efficient manner cutting costs and resource requirements. Using online surveys to capture information relating to process implementation and operation allows a broader community to be polled and together with algorithms that analyse the “patterns” of data captured, the software tool can also present accurate and reliable assessment outcomes. It is important to understand the value of ITSM process assessments using ISO/IEC 15504 for CSI: this combination is not specifically a process improvement framework but a measurement framework for ITSM process improvements which is one of several initiatives that can drive CSI.


Anup Shrestha
PhD Candidate, School of Management & Enterprise
University of Southern Queensland

Paul Collins
Chief Technology Officer
Assessment Portal

Aileen Cater-Steel
Assoc. Prof., School of Management & Enterprise
University of Southern Queensland

About Guest Author

This post has been contributed by a Guest Author, please see details above

2 Responses to “ITSM Process Assessments using ISO/IEC 15504 for CSI”

  1. An interesting and thought provoking article. Thank you.


Leave a Reply