Evaluation

Evaluation 

Evaluation is a generic process that considers whether the performance of something is acceptable, value for money etc. – and whether it will be proceeded with, accepted into use, paid for, etc.

Purpose, goal and objective 

The purpose of evaluation is to provide a consistent and standardized means of determining the performance of a service change in the context of existing and proposed services and IT infrastructure. The actual performance of a change is assessed against its predicted performance and any deviations between the two are understood and managed. The goal of evaluation is to set stakeholder expectations correctly and provide effective and accurate information to Change Management to make sure changes that adversely affect service capability and introduce risk are not transitioned unchecked.
The objective is to:

• Evaluate the intended effects of a service change and as much of the unintended effects as is reasonably practical given capacity, resource and organizational constraints 
• Provide good quality outputs from the evaluation process so that Change Management can expedite an effective decision about whether a service change is to be approved or not.

Scope

Specifically in this section we consider the evaluation of new or changed services defined by Service Design, during deployment and before final transition to service operations. The importance of evaluating the actual performance of any service change against its anticipated performance is an important source of information to service providers to help ensure that expectations set are realistic and to identify that if there are any reasons that production performance does not meet what was expected.

Value to business

Evaluation is, by its very nature, concerned with value. Specifically effective evaluation will establish the use made of resources in terms of delivered benefit and this information will allow a more accurate focus on value in future service development and Change Management. There is a great deal of intelligence that Continual Service Improvement can take from evaluation to analyse future improvements to the process of change and the predictions and measurement of service change performance.

Policies, principles and basic concepts 

Policies 

• The following policies apply to the evaluation process: 
• Service Designs or service changes will be evaluated before being transitioned. 
• Any deviation between predicted and actual performance will be managed by the customer or customer representative by accepting the change even though actual performance is different to what was predicted; rejecting the change; or requiring a new change to be implemented with revised predicted performance agreed in advance. No other outcomes of evaluation are allowed. 
• An evaluation shall not be performed without a customer engagement package

Principles 

The following principles shall guide the execution evaluation process: 

• As far as is reasonably practical, the unintended as well as the intended effects of a change need to be identified and their consequences understood and considered. 
• A service change will be fairly, consistently, openly and, wherever possible, objectively evaluated. 

Basic concepts 

The evaluation process uses the Plan–Do–Check–Act (PDCA) model to ensure consistency across all evaluations.

Evaluation report

The evaluation report contains the following sections. 

Risk profile 

A representation of the residual risk left after a change has been implemented and after countermeasures have been applied. 

Deviations report 

The difference between predicted and actual performance following the implementation of a change. 

A qualification statement (if appropriate) 

Following review of qualification test results and the qualification plan, a statement of whether or not the change has left the service in a state whereby it could not be qualified. 

A validation statement (if appropriate) 

Following review of validation test results and the validation plan, a statement of whether or not the change has left the service in a state whereby it could not be validated. 

A recommendation 

Based on the other factors within the evaluation report, a recommendation to 

Change Management to accept or reject the change: 

• Review and close transition
• Knowledge Management. 

Triggers, inputs and outputs and inter-process interfaces 

Triggers: 

• Request for Evaluation from Service Transition manager or Change Management 
• Activity on Project Plan. 

Inputs: 

• Service package
• SDP and SAC 
• Test results and report. 

Outputs: 

• Evaluation report for Change Management. 

Information management

• Service Portfolio
• Service package 
• SDP, SAC 
• Test results and report 
• Evaluation report. 

Key performance indicators and metrics 

The customer/business KPIs are: 

• Variance from service performance required by customers (minimal and reducing) 
• Number of incidents against the service (low and reducing). 

The internal KPIs include: 

• Number of failed designs that have been transitioned (zero) 
• Cycle time to perform an evaluation (low and reducing). 

Challenges 

Challenges include: 

• Developing standard performance measures and measurement methods across projects and suppliers 
• Projects and suppliers estimating delivery dates inaccurately and causing delays in scheduling evaluation activities 
• Understanding the different stakeholder perspectives that underpin effective risk management for the evaluation activities 
• Understanding, and being able to assess, the balance between managing risk and taking risks as it affects the overall strategy of the organization and service delivery 
• Measuring and demonstrating less variation in predictions during and after transition
• Taking a pragmatic and measured approach to risk 
• Communicating the organization’s attitude to risk and approach to risk management effectively during risk evaluation 
• Building a thorough understanding of risks that have impacted or may impact successful Service Transition of services and releases 
• Encouraging a risk management culture where people share information.