Articles archive

Does Management Development Add Value?

Author: Keith Patching, Cranfield School of Management
First published: 1999

Both here in the UK and elsewhere, organisations are becoming more anxious to measure the value of executive development. It seems that the halcyon days of executive education for its own sake have passed, and those of us in the business of providing development are being called to task. The message is, 'Prove your worth, or we'll find another way.'

It seems perfectly reasonable to move beyond the simplistic approach to measurement of value: a sign above the desk which says, 'If you think education is expensive, try ignorance,' no longer constitutes 'proof. There is, in consequence, a growing body of research into ways of testing the hypothesis that executive development does add value.

One of the key considerations in measuring the value of executive development is to recognise that development is not education. By education we refer to the more or less speculative investment in the dissemination of new knowledge in the hope that it will come in useful some time in the future. It still has a role, but measuring its value is very difficult.

By executive development we refer to a much more focused set of learning activities which address issues facing individuals and organisations. This enables us to work with executives to define the specific outcomes the development activity is designed to help achieve.

Measuring ROI

Amongst the most visible of these ways is the return on investment approach, best exemplified in the work of Jack Phillips (Phillips 1997). This approach demonstrates how to put a firm financial figure on the benefit of training and performance improvement programmes. It is a thorough and rigorous approach which has a great deal to commend it.

For one thing, it cites numbers of programmes which have been subjected to this kind of analysis, and which have delivered returns on investment as high as 1,031 per cent and 741 per cent (op.cit.:187,195). For another, it attempts to address most of the caveats which, over the years, have been raised about the difficulties of measuring things like the monetary value of improved morale, and lower staff turnover.

Many organisations have adopted the ROI approach, and are demanding of their providers of education and development that they apply these kinds of measures of worth to their activities. For some of these organisations, the debate is over. It's ROI or nothing.

One reason why others shy away from this kind of evaluation is that it can also produce gloomy results. In a series of studies cited by Goleman a large proportion of programmes 'were found to be utterly worthless' (Goleman, 1998: 247). For those that promised some kind of financial pay back, some estimates suggested that the payback period would be up to seven years.

One problem with these kinds of financially based measures is that the formulae contain a number of elements which are based upon estimates of the value of both costs and benefits. Given the ease with which statistics can be manipulated, it is not surprising that the same kinds of programmes can be seen to produce staggeringly high returns, while others seem to deliver no value at all.

Part of the problem is that there is no universally accepted way of gathering review data. The many methods include:

• end of programme evaluations (happy sheets);
• productivity measures;
• staff and customer satisfaction surveys;
• post-programme questionnaires;
• one to one interviews;
• 360º appraisals;
• focus groups;
• follow up workshops;
• critical incident recording analysis;
• personal strategy implementation plans;
• cultural audits/comparative snapshots.

This is not an exhaustive list. But it does illustrate that, given the wide range of choice, comparing like with like is difficult. It also raises the question of which methods will be best suited to the task. Such choices provide freedom to tailor evaluation to specific needs, but they make any attempt at a 'science' of evaluation the more tenuous.

Methodological issues

However rigorous we may want our evaluations to be, we do need to take into account that we face a number of methodological issues which will challenge that rigour. The choices of data-gathering methods are just the tip of the iceberg. Evaluating interventions is a complex social scientific activity (Chen, 1990).

One of the most significant issues lies in the fact that there are likely to be multiple stakeholders involved in any executive development activity. Although the 'bottom line' (the goal of the ROI approach) should be of interest to most of these stakeholder groups, there may be other factors of greater or more immediate interest. Some stakeholders may be comfortable with a pay back period of several years, if they believe that they are laying the foundations for a change of culture. Others may be content to take a loss over the short term for the sake of developing knowledge, skills and attitudes which will enable the organisation to be ahead of the game as markets change.

There is also the perennial problem of potential bias in the methods. of data collection. Deciding what to focus on, what questions to ask, how to .ask those questions, who to involve in both the gathering of data and the provision of answers, and so on, are non-trivial issues which can influence results enormously. One also needs to be clear about the mood and motivations of those providing data. If questionnaires are used, for example, researchers need to take it on more. than simple trust that respondents will answer truthfully. Why should they? What's in it for them?

There is also the problem of 'unintended consequences'. The more rigorously a research instrument is designed to weed out 'noise' and achieve comparative data, the more it becomes blind to what may be highly significant outcomes which fall outside the scope of the investigation. But if the research is totally unstructured, it is unlikely to produce data which can be useful to any 'rigorous' ROI analysis.

Finding the link

Not all writers are convinced of the efficacy of the search for ROI. Citing a number of studies of links between a range of HR investments and business financial results, Ulrich says, 'The questions of why this relationship exists and how it operates. "remain unanswered. Unless and until the path of intermediate steps linking these two factors can be traced, such research may be more academic than useful' (Ulrich 1998: 5).

Knowing whether a programme has delivered a good return on investment is one thing. Understanding why one programme appears to deliver results while another does not is a more daunting challenge.

Lying at the heart of many pseudo-scientific methods of measuring the value of executive development is an unspoken assumption, which can be illustrated like this:

The unspoken assumption is that the 'raw materials' of the education process do not affect the process itself. But executive development is not something which is done to executives; it is an activity in which executives are intimately involved. Evaluating executive development demands a much stronger focus on the process itself. Otherwise the risk is to play the 'post hoc ergo propter hoc' game. This is trial and error rather than effective learning design based upon good learning theory.

Truly understanding value depends upon the underpinning (often unspoken) theory of learning. The 'educational black box' is underpinned by an unconscious model of passivity on the part of the learner:

Far better is to explore the learning process itself and accept that learning is more than simply the transfer of knowledge. It involves at least four distinct stages:

This model, which is explored in depth in Management and Organisation Development (Patching, 1999: 293-308), takes a learner-centred approach to executive development. Used for both learning design and evaluation, it recognises that learning and development involve different aspects of the personality, and produces different kinds of outcome.

Effective learning involves:

• the acquisition of new knowledge (what to do);
• intellectual understanding of the underlying processes of the topic (why do it like this);
• alignment with personal values (a commitment to doing things differently);
• the capability to make it happen (breaking old habits).

Executive development programmes can be very good in some of these areas, but poor in others. And this is as much a function of the personalities, histories, current pressures, expectations and motivations of learners as it is of the 'programme' itself. In other words, what (if anything) needs to be measured is not an independent 'thing' (an executive development programme), but a series of interactions, each of which will vary widely according to the specific psychodynamics of each iteration of that programme.

Assessing value

In our experience at Cranfield, few sponsors of management development interventions, and few participants in those interventions undertake executive development simply to achieve a positive ROI. The purposes of interventions are many and varied. The clue to successful evaluation lies in this sense of purpose.

Post hoc measurement fails not only because it is based upon an irrelevant set of assumptions about what goes on during a learning event, but also because it is rarely aligned with this sense of purpose. Effective evaluation starts with clarity about outcomes.

Our approach to evaluation starts when we work with sponsors and participants on the desired outcomes from an event. The process informs both evaluation and design. We work together to define what people will be doing differently if the intervention is to be judged as successful. We ask, 'How will you know if the programme has achieved its aims?' We also ask, 'What will happen if you don't undertake this executive development?' As we refine the picture of the behaviours, skills and attitudes the intervention is expected to help bring about, we also identify those aspects of this picture which are beyond the scope of the intervention itself. We identify factors such as reward systems, structures and internal processes which will either enhance or get in the way of progress towards those goals.

Finally, we ask, 'If we can help bring about this set of behaviours, would this be of value to you?' If the answer is 'yes', then achieving these behaviours is considered to have added value in the eyes of the buyer.

This simple approach masks more detailed discussions. We have to agree over what period of time the changes in behaviour are to be assessed. Value for money is considered by the buyer as he or she recognises how much the intervention will cost, in terms of price, opportunity cost, and other factors. But at all times, it is up to the buyer to determine the value.

Value is not inherent in an executive education intervention, but in the perceptions of the buyer of the worth of the outcomes. If, at a later date, ROI measures are to be imposed, that is for the buyer to decide. If he or she. thinks that they cannot in any other way be sure about the value to them of the outcomes from an intervention, then they are free to undertake whatever confirmation takes their fancy.

Does executive development add value?

Few of our clients undertake ROI analyses. For them, such a detailed and complex investigation is unnecessary, since it is up to them, as senior managers, to make judgements of value. By taking an outcomes-oriented approach, we help clients make those judgements.

The overwhelming majority of such outcomes-oriented judgements are positive. Executive development can, and frequently does, achieve those desired outcomes. New skills, knowledge, attitudes, and behaviours are seen as valuable by the buyers.

Any attempt to replace this apparently subjective concept of value with ROI analysis or other quantitative means would be rigorous but, in the end, pointless.

Value is not an objective 'thing' to be measured in the abstract. It is deeply rooted in the visions of the future in which executive development plays a significant, and valuable part.

References

Chen, Huey-Tsyh (1990) Theory-driven Evaluations, Sage Publications, Newbury Park, California.

Goleman, Daniel (1998) Working with Emotional Intelligence, Bloomsbury Publishing, London.

Patching, Keith (1999) Management and Organisation Development: Beyond Arrows, Boxes and Circles, Macmillan, London.

Phillips, Jack J. (1997) Return on Investment in Training and Performance Improvement Programs, Gulf Publishing Company, Houston Texas.

Ulrich, Dave (1998)Delivering Results: A New mandate for Human Resource Professionals, Harvard Business Review Books, Boston, Mass.