Executive Education's Impact Assessed

Imperial College Business School conducts Impact Studies to monitor the effectiveness on their executive development programmes and to see how learning is being applied by participants when they return to their workplaces. This is a synopsis of their findings:

What is an Impact Study?

It’s notoriously difficult to measure the benefits of executive development programmes. The Impact Study investigates changes in participant behaviour that can be ascribed to the effect of the development programme they have attended. Rather than trying to produce hard, numerical evidence of impact, it creates a series of stories – all evidence-based, that point to individual and organisational benefits. Our Impact Studies are mixed method interviews conducted with programme participants, their line managers and occasionally the organisation’s customers, to identify changes in behaviour and demonstrated skills that can be ascribed to the particular development programme

Why do Impact Studies?

The main driver behind conducting Impact Studies is to uncover the extent to which learning from Imperial College Business School development programmes is being applied back within the workplace.

Why is this important? It helps the organisation to understand what value it has gained from the investment in the development programme, and what it can do in terms of organisation, consultation and communication to improve the impact (and return on investment) of future programmes. It confirms for participants what they have gained from the programme and finally helps us, the learning partner, to understand how we can work further as a trusted advisor to help increase the value of our customers’ investments.

To assess this, the focus of attention is on programme participants’ perceptions and actions since the conclusion of their programme, backed up by observations gathered from their line managers. This synopsis reports on findings from two Impact Studies conducted following programmes delivered to a large multi-national construction company.

How did we do our latest Impact Study?

Employing a mixed-methods approach, just under 100 participants and their line managers were invited to participate in an online survey. Following this, a subset of participants and their line managers were invited to participant in one-to-one interviews to obtain data more specific than that obtained via the online survey. The following information was sought:

• What participants recall from the programme (main highlights)
• How the programme impacted upon the participants’ work activities, including specific examples to evidence that impact
• How participants managed to apply or implement course content
• The extent of manager awareness and understanding of, and support for, the programme
• The facilitators and barriers to implementation

Participant and line manager suggestions for improving the programme, particularly how it was ‘set up’ and ‘set down’.

Main Findings and Themes

To date, the main findings from our Impact Studies can be grouped into the following four themes (see below for further details): 

1. Specific benefits of programme from participants’ and line managers’ perspectives. Our mixed-methods approach helped identify specific examples of benefits coupled with frequency analyses to demonstrate how prevalent those benefits are.
2. Greater understanding regarding the role of the line manager in maximising value from the programme
3. Insight into set-up and set-down factors that can impact the success of the programme and the application of learning
4. Additional benefits and outcomes resulting from implementation of Impact Studies

Ongoing reiterations of the Impact Studies are helping to refine a standardised process for their development and use. Concurrently, clients are valuing the Impact Studies as a device that helps them develop their organisation’s learning culture, as opposed to an organisation that sends employees on learning activities.

Main findings:

1) ROI - Specific benefits of programme from participants’ and line managers’ perspectives. Our mixed-methods approach helped identify specific examples of benefits coupled with frequency analyses to demonstrate the prevalence of those benefits.

• Changes in participants’ cognitions and behaviours
• Concrete examples of application of learning and skills
• Tangible business outcomes
• Specific benefits to clients
• Reputational benefit (working with business school like Imperial)

Participants came out of programme more rounded and able to develop throughout entire company, not just within single business unit. They report having more understanding and awareness of whole company and their role in it.

All of these identified benefits help the HR and Learning & Development teams to justify such investment in development.

2) Greater understanding regarding the role of the Line Manager in maximising value from the programme.

• Can be facilitators of learning translation and application – making space for participants to practise new skills; giving the participant new authority to act
• At best, an unengaged line manager has no impact but at worst they can have a negative impact upon their participant – being alienated from the programme and its aims they often see a contradiction between how the participant has developed and daily business (e.g. “get back to your real work”)
• A negative impact can also occur for the line manager (i.e. if their participant begins using language they are not familiar with) – making them feel isolated or even ‘stupid’
• General recognition that line managers cannot make up for lack of personal initiative. Both are required! – so experiential learning that developed self-confidence was often identified as valuable (and we’d agree)

3) Insight for the Organisation - into the set-up and set-down factors that can impact the success of the programme and the transfer and application of learning

• Identified barriers to transfer and application of learning
• Line manager engagement essential – method helps identify logistical issues associated with engaging line managers
• Importance of robust/appropriate participant selection
• Importance of managing participant expectations (i.e. job retention, pay, promotion, option to participate on further courses). Not doing so can alienate and/or disengage line managers
• Defining success criteria; value in participants/line managers/companies identifying desired and projected outcomes and (SMART) implementation plans
• Importance of company providing opportunities or mechanisms for participants to apply learning
• Potential further role for the provider of development to provide mechanisms to facilitate retention and application of course learning as well as enthusiasm from course
• For large multi-national companies, this method can help identify implementation/follow-up challenges for certain sectors (i.e. site-based participants)
• This Impact Study was itself identified as a mechanism to facilitate knowledge retention and implementation – it brought the themes of the programme back to mind
• Highlights importance of co-ordinating input from ALL stakeholders (participants, line mangers, business unit directors, etc.) in order to maximise value from course

Mixed-methods approach helps generate customer- and site-specific suggestions for how to maximise value from course. Rather than us providing generic recommendations, this method allows us to identify and suggest ‘best-fit’ solutions.

4) General findings

Our method allows us to generate a level of awareness and understanding in different stakeholders beyond that which is typically gained via standard means

• Importance of conducting assessment at appropriate time: Long enough to have had time to apply learning, not too long so as to have forgotten content
• Process and questions from Impact Studies can be designed in such a way so that it is not just specific benefits that are identified (i.e. increased confidence) but also the way these are being expressed and demonstrated. This makes the findings more evidence-based rather than quite so anecdotal.
• The Impact Studies can highlight potential interactions between participant characteristics (i.e. proactive initiative) and use of content within the workplace

Impact Studies can also shed light upon organisational culture issues and how those issues can impact perceived course value and performance outcomes.