80.Follow the instruction to write 2pages activity

Stakeholder Training Evaluation ActivityFred Nickols provides a perspective that considers multiple stakeholders in
training evaluation. Reflecting upon his perspective complete the following
activity by following the steps below and the form/table on the following page.
STEP 1:
Think of a training with which you have been involved.
STEP 2:
Write a brief description of the training.
STEP 3:
Consider all of the possible stakeholders involved in the training and
list those in the center column labeled “Stakeholders” in the table
below. Give consideration to all those in the organization (and outside
of it) who could benefit in some way from the training.
STEP 4:
List the contributions made to the training by each group of
stakeholders.
STEP 5:
List the inducements taken away by each group of stakeholders.
Type of Training Evaluated (Provide a Brief but Specific Description Below)
Stakeholder Model of Evaluation
Contributions (Put In)
Stakeholders
Inducements (Take Out)
Evaluation
Designs
Unit 9
There are several related topics in
this unit…
Training as an Independent Variable
Causal Relationships
Control Groups
Matched Groups
Full Evaluation Designs
Time Series Evaluation Designs
Partial Evaluation Designs
Training—an Independent Variable
Evaluation of Training
In research terms, training is an independent variable. That is something
thought to affect an outcome or dependent variable. The dependent variable in
terms of training could be any number of things ranging from better customer
service, better employee communication, or better adherence to the mission of
the company. Whatever workplace issue we have arranged the training to
address ends up as the dependent variable.
In very simple terms, then, we want to evaluate whether or not the training had
an impact on a particular outcome. That is, whether or not or to what degree
the independent variable affected the dependent variable.
But how can we show that training was successful in bringing about a
particular outcome? To answer this question we need an evaluation design.
Establishing a Causal Relationship
Evaluation of Training
In order to establish a causal relationship—that is one in which
we can say ‘this caused that to happen’ or ‘the training led to this
specific outcome’ three conditions must be met…
1. The training must precede the observed outcome in time. We
can not attribute training to behavior that exists before training
occurs.
2. The training must relate to the expected behavior in some
meaningful way. Training that does not relate to the expected
behavior will not produce the intended results.
3. Changes due to training must be the result of the training and
not some other factor (known as a confounding variable).
Control Groups
Evaluation of Training
Control refers to the need to control as many
factors as possible so that we can isolate the
causal relationship between training and the
expected outcomes.
With regard to training, we exercise control by
having a control group or a group that does not
receive training to compare the training group
against.
Matched Groups
Evaluation of Training
When comparing groups we want to begin with groups that are as evenly
matched as possible. If we train a group that already has some proclivity to
behave a particular way (say being polite to customers) then we can not be
sure that training produced any notable differences between the trained group
and the control group.
There are two primary strategies for creating matched groups:
1. Assign people randomly to either the trained group or the control group so
that differences end up spread out and dispersed among the groups.
2. Use pretests to determine where people reside with regard to the outcomes
of interest and then assign them to groups so that the groups contain
equally matched participants (that is people that are better and/or worse in
the area of training end up in both the training and the control group).
Evaluation Designs
Evaluation of Training
Evaluation designs vary in the degree to which they are more or
less sophisticated. More sophisticated designs demonstrate the
greatest degree of control through the use of pretests, control
groups, and random assignment of people to groups. Weaker
designs lose some of these features.
Understand though that we can not always set up an evaluation
design that is completely robust. Thus, it is helpful to know and
understand the range of possibilities available and to choose
which is best given the circumstances and possible constraints.
Full Designs
Evaluation of Training
Full designs include random assignment, control and
training groups, and at times a pretest. There are several
full designs from which to choose.
The Pretest-Posttest Control Group Design
Random Assignment
Random Assignment
Pretest
Pretest
Training
Posttest
Posttest
This is the basic model for full designs, it includes both
random assignment and pretests, as well as a control
and training group. Outcomes are assessed with a
posttest after training.
Full Designs
Evaluation of Training
The pretest however may not be necessary if we
randomly assign people to groups. Thus, if it proves
cumbersome or problematic to have a pretest we can use
the design below, which is essentially the same, with the
exception of not having a pretest.
The Posttest-Only Control Group Design
Random Assignment
Training
Posttest
Random Assignment
Posttest
Full Designs
Evaluation of Training
Similarly, random assignment may not be necessary if we use
pretests and match groups well. In this case we can use the design
below.
The Pretest-Only Control Group Design
Pretest
Training
Pretest
Posttest
Posttest
This design is the same as the pretest-posttest design, except it
does not use random assignment. It does though have pretests, a
control group, and a posttest.
Full Designs
Evaluation of Training
It may be helpful to see the three full designs together to better understand how
they compare to and differ from one another.
The Pretest-Posttest Control Group Design
Random Assignment
Pretest
Training
Random Assignment
Pretest
Posttest
Posttest
The Posttest-Only Control Group Design
Random Assignment
Training
Random Assignment
Posttest
Posttest
The Pretest-Only Control Group Design
Pretest
Training
Pretest
Posttest
Posttest
The first has both a pretest and random assignment, whereas the second and
third have one or the other. All three have training and control groups and of
course a posttest.
Time Series Designs
Evaluation of Training
Sometimes it is important to establish a baseline for training
outcomes before training occurs. That is, to have evidence of the
training outcome of interest over time rather than simply from
one single pretest.
When this is the case a time series design is appropriate. Time
series designs rely on multiple pretests to establish a baseline
score for training participants.
Time Series Designs
Evaluation of Training
Time series designs rely on a set of pretests that help to establish a “baseline”
or average to compare the posttest score against. Time series designs include
the simple time series design and the multiple time series design.
Time-Series Design
Multiple Pretests Over Time
Training
Posttest
The simple time-series design lacks a control group, which can be added to
create what is called a…
Multiple Time Series Design
Multiple Pretests Over Time
Multiple Pretests Over Time
Training
Posttest
Posttest
Partial Designs
Evaluation of Training
The final set of evaluation designs exercise the least
amount of control because they lack several key
features. Partial designs are scaled down versions of
full designs. They have only the very basic structure of
full designs.
Although not preferable, they are appropriate when
organizational constraints limit the degree to which we
can conduct fuller evaluation designs.
Partial Designs
Evaluation of Training
There are three common partial designs:
One Shot Case-Study Design
One Group Pretest-Posttest Design
Pretest
Training
Posttest
Training
Posttest
Training
Posttest
Posttest
Static Group Comparison Design
As you can see, the simplest partial design of all is the one shot case-study design
which simply involves a posttest of a training group.
There are two possible ways to improve on this design without creating a full design.
First, to add a pretest (the one group pretest-posttest design). Second, to add a control
group (the static group comparison design). Adding both would lead to a full design.
A Stakeholder Approach to
Evaluating Training
Fred Nickols
www.nickols.us
fred@nickols.us
A Stakeholder Approach to Evaluating Training
Introduction
There are probably no more widely accepted “realities” or truisms in the world of
training than the following:
This Article’s Key Points

Training, whether it is a particular offering or the entire function, must satisfy multiple constituencies
known as “stakeholders.”

A stakeholder is a group or an individual with an interest in seeing a particular endeavor succeed.

A stakeholder’s interest in the success of the endeavor in question is rooted in a quid pro quo (i.e., a
stakeholder puts something into the endeavor with
the expectation of getting something out of it).
There is widespread interest in
evaluating training, particularly at
the higher levels of TKM (i.e., onthe-job behavior change and business results) and in going beyond
TKM (e.g., in determining the ROI
of training or even its societal impact).

What stakeholders put in are known as “contributions” and what they take out are known as “inducements.”

Although various stakeholder groups readily agree
in general about the kinds of results expected from
training, they hold very different views about what is
important when it comes to evaluating training.
Their inducements are different.

There exists adequate if not abundant knowledge and an available
supply of viable tools for evaluating
training at all levels of TKM (and
beyond).

To adequately evaluate training (or any other endeavor having multiple constituencies), it is necessary to assess the extent to which all stakeholder
groups are satisfied with what they receive from the
training.

Yet, despite the above-mentioned
interest in and availability of tools
for more robust efforts, evaluations
of training remain mired in TKM
Level 1.

The only way to ensure that all training stakeholder
groups are satisfied is to factor in their various requirements during the design, development and delivery of the training.

When it comes to evaluating training, the dominant model is “The
Kirkpatrick Model” (TKM).

TKM is rarely implemented in its
entirety and training evaluations
are usually confined to the “smiles
test” (TKM Level 1: Trainee Reactions).

Why is this? If evaluation is so important and if the means of carrying it out exist, why do evaluations typically consist of little more than the famous “smiles test”? Is it because the interest in evaluating training is
feigned? Is it because the costs of evaluating training outweigh the benefits? Is it a case of diminishing
returns, that is, the higher up TKM an evaluation goes, the more costly the evaluation and the less valuable
the information? Or is it perhaps the case that trainers are the only ones interested in TKM – and in going
beyond it?
It is my view that the training community is committed to an approach to evaluating training that, after more
than 40 years, has failed to capture the commitment and support of other important constituencies, most
especially, that of the trainees, their managers and the senior managers of the organizations in and for
which training is conducted. If this is true, then the issue isn’t one of figuring out how to apply TKM – or
even of extending it – instead, the issue is one of finding some other approach to evaluating training.
It is also my view that there is a better approach to evaluating training – a stakeholder-based approach. Although the focus of this paper is on evaluating training, a stakeholder approach can be applied to evaluating
HRD and other functional areas as well, especially those considered as having “internal customers” or constituencies to be satisfied.
The basic premise of the stakeholder approach is that several groups within an organization have a stake in
training conducted for organization members and any effort to design, develop, deliver and evaluate training
must factor in the needs and requirements of these stakeholder groups or the results of any subsequent
evaluation are bound to fall short of expectations. The approach proposed here has two theoretical roots:
stakeholder theory (Donaldson & Preston, 1995; Freeman, 1984) and the contributions-inducements view of
organizational membership (Barnard, 1947; March & Simon, 1958).
© Fred Nickols 2003
(DRAFT: For Review Only – Do Not Cite without Permission)
1
A Stakeholder Approach to Evaluating Training
Finally, it should be noted that this is a proposed approach, a new approach; it speaks to what could and
should be done, not what is currently being done. There are, then, no cases to point to, no testimonials to
present, no data to manipulate. There is simply a proposal to go about evaluating training in a very different
way and some suggestions as to how to do that. But first, some measurement and evaluation basics.
Measurement & Evaluation
There is a difference between measurement and evaluation. Measurement focuses on obtaining information
as a result of comparing a given against a standard (e.g., information about the length of a board can be
determined by comparing it against the standard provided by a tape measure). Evaluation concerns itself
with making judgments based on the information provided by measurement (e.g., the board in question is
too long or too short or just right). Judgments are usually about value and can be couched in terms of utility
or economics or even aesthetics. In organizations, the “givens” typically consist of information about actual
performance and the “standards” consist of the goals and objectives established for performance. Value
judgments come into play in deciding whether the performance is “good enough” or whether improvement is
required.
To evaluate anything is to determine its value. From a transaction perspective, the value of anything derives
from its importance or worth in an exchange. Whether you are bartering or using money as a medium of
exchange, value is measured by the amount of one thing that can be exchanged for another. Ultimately,
value is a highly individual matter; it boils down to how much of one thing a person is willing to exchange for
another. I might be willing to give up time with my family to put in long hours at work in return for the chance
of advancing my career. You might not. You might be willing to pay $45,000 for an automobile; I might not.
You might be willing to burn the midnight oil to acquire an advanced degree; I might not. I might be willing to
travel extensively as part of my work; you might not. In ascertaining the value or worth of anything, including
training, one must always ask, “Ascertain its value to whom?”
To evaluate training, then, is to ascertain its value or importance or worth; however, and this is extremely
important, the question that usually goes begging is, “To whom?” It is one thing to ascertain the value of
training to the trainees. It is something else to determine its value to management. And, it is yet a third matter to fix the value of training to trainers, be they instructors or developers. Trainees, trainers and management, these are just three of several groups with a stake in training. Other stakeholders include training
vendors (whether selling off-the-shelf or custom-developed materials) and, of course, the managers of the
trainees. Let us return now to TKM and the added notion of ROI.
TKM & ROI
As noted at the outset of this article, current thinking about the evaluation of training is dominated by what
most call “The Kirkpatrick Model” (TKM). TKM focuses on four “levels” of evaluation: Reactions, Learning,
Behavior and Results (Kirkpatrick, 1975a, 1975b, 1975c, 1975d). TKM is widely known and widely accepted, even if it is rarely fully implemented. Another, more recent addition to TKM, what some call a fifth
level, is the notion of determining the financial return on investment (ROI) of training (Philips, 1997). And,
there are those who suggest that it is possible and desirable to go beyond TKM and ROI to societal impact
(Watkins, Leigh, Foshay & Kaufman, 1998).
It is not the intent in this paper to engage in lengthy critiques of TKM or efforts to determine the ROI of training. That has been done elsewhere (Alliger & Janak, 1989; Holton, 1996, Kaufman & Keller, 1994; Nickols,
2000). Instead, this paper uses TKM as a point of departure, a launch pad for introducing a stakeholderbased approach to the evaluation of training. We will, however, take a brief look at what typically happens in
evaluating training.
Evaluating Training: What Typically Happens
What typically happens is that the interests of most of the stakeholders are subordinated to the interests of
the trainers and their managers.
Trainers and their managers are understandably anxious to demonstrate the value of what they do. While it
is entirely conceivable that a funding manager will want to know something about the ROI of the training, it is
equally conceivable that the trainees could care less. The instructors and the developers are probably very
interested in the nature and extent of learning that has taken place and, perhaps, in the degree of transfer to
the work place. However, unless they’re hoping for a promotion into management or a transfer to a performance consulting unit, their interest in the ROI of the training is apt to take a back seat. The trainees are
likely to care mainly about two things: the applicability or relevance of the subject matter (concepts, prin-
© Fred Nickols 2003
(DRAFT: For Review Only – Do Not Cite without Permission)
2
A Stakeholder Approach to Evaluating Training
ciples, methods, tools, techniques, etc.) and the extent to which the training makes good use of their time.
Training vendors want to know if their client, the training department, is happy with the training they bought.
Everyone wants to know what the trainees think – and for good reason. Why? Because if the trainees are
sharply and uniformly critical of the training, very little else matters.
So, most of the time, efforts to evaluate training takes the form of the required “smiles test,” a measure of
trainee reaction, perhaps some assessment of the learning that has taken place, occasionally an attempt to
determine the extent of transfer of training or behavior change on the job and job performance impact, and a
rare effort to quantify the bottom-line impact of training and use it to establish the ROI of the training.
An interesting and useful question to ask about the four (or five) levels of training evaluation is this: “Who is
interested in this particular evaluation?” In other words, who is the audience for the information obtained at
each level? Further: What judgments are to be based on this information? Who will make them?
As one considers the various audiences for training evaluations and the judgments these audiences will
make about training, it becomes apparent that there are many constituencies with an interest in training.
Trainee reactions, TKM Level 1, are obtained from the trainees but they are of interest to many in the organization, not the least of which are the trainers and the trainees’ managers. Learning (i.e., skills or competencies acquired) is clearly of interest to the trainees and trainers and perhaps of importance to others as well.
Behavior change on the job is no doubt of interest to the trainees’ managers – and to trainers as well, especially if they are interested in demonstrating the impact of training. Results, too, are of interest to trainers
and to management, albeit for different purposes. Managers want results from training for the sake of the
results themselves; trainers are more likely to want results more for the purpose of demonstrating the value
of training than for the value of the result itself. As for the ROI of training, the only ones likely to be interested in that are those who are under pressure to demonstrate it or those who have a need for it. If such pressure exists, it most likely focuses on trainers, not the trainees or their management.
There are, then, several constituencies implied by TKM: trainers, trainees, the trainees’ managers, managers of the training function or department and, perhaps, senior managers throughout the organization.
These constituencies all have a vested interest in having things go well in training; none of them want it to
be a waste; all want it to add value. In short, they have a stake in the training, an interest in having it succeed, and that makes them stakeholders.
Stakeholder Defined
Freeman (1984, p.46) defined a stakeholder as “any group or individual who can affect or is affected by the
achievement of an organization’s objectives.” This is a very broad definition; too broad, perhaps, because it
would include competitors as stakeholders. Neely and Adams (2003), in developing their “Performance
Prism,” took care to point out that any look at stakeholders must include stakeholder contributions as well as
stakeholder satisfaction. In their view, stakeholders put in something and they take out something. This
transaction view of a stakeholder is quite similar to the contributions-inducements theory of organizational
membership articulated over a period of several decades by the likes of Chester Barnard, James March and
Herbert Simon (more on contributions and inducements in a moment).
For the purposes of this paper, a stakeholder is defined as a person or group with an interest in seeing an
endeavor succeed. For example, most employees have an interest in seeing their companies succeed. So
do that company’s suppliers, its customers and the community in which the company is embedded. Similarly, most trainers have an interest in seeing that the training they develop and deliver is successful. There
are others who want training to be successful, too. Chief among them are the managers who sponsor or
fund the training, the managers who manage the training department and last, but not least, the trainees. A
list of typical training stakeholders follows:








Trainees
The Trainees’ Managers
Funding Managers
Training Developers
Instructors
Training Managers
Training Vendors
The Training Community
In cases wherein the training is expected to have a fairly direct and substantial impact on some critical aspect of the organization’s performance, senior managers and executives are also important stakeholders.
© Fred Nickols 2003
(DRAFT: For Review Only – Do Not Cite without Permission)
3
A Stakeholder Approach to Evaluating Training
There are even situations in which the community as well as state and federal regulators become stakeholders (e.g., as is likely the case when training nuclear power plant operators).
Stakeholder Contributions and Inducements
As the definition of stakeholder provided earlier implies, stakeholders are people with an interest in seeing
an endeavor succeed; they expect to get something out of the endeavor or effort in question. That something might be a return on their investment, as is the case with investors. But, and this is extremely important, stakeholders must also put something into the endeavor.
Stakeholders put something in and they take something out. Investors put their money at risk in hopes of a
return just as the managers who fund training do so in hopes of a positive impact on performance or costs or
productivity or some other payoff. Trainees contribute their time, attention, energy and other forms of input
(e.g., participating in discussions and exercises) and they hope to take out useful knowledge and skills, methods, techniques and tools. Instructors put in their time and energy, too, along with their skills at leading or
facilitating discussions, presenting subject matter in interesting, relevant ways and handling the occasionally
difficult trainee. They hope to walk away with a return in the form of a sense of accomplishment, a reputation
maintained or enhanced and high marks from the trainees. Developers invest a great deal of time and
energy in designing, developing and field-testing instructional materials and most of them hope to receive in
return a decent paycheck, a modicum of recognition and a sense of satisfaction with a job well done. In the
formal language of organizational theory, stakeholders exchange contributions in return for inducements.
The contributions-inducements schema has a long history and has been observed and commented upon by
noted management and organizational theorists starting with Chester Barnard (1947) and continuing through
James March and Herbert Simon (1958). Its essence is that the various participants or stakeholders must
perceive value in the exchange. Generally speaking, inducements must be seen as having equal or greater
value than contributions. From the stakeholders’ perspective, what they receive is of equal or greater value
to them than what they contribute. That is why they are in the relationship. And if that relationship does not
offer them inducements of equal or greater value to them than the contributions expected of them, they
leave the relationship. That is why employees, customers and suppliers go elsewhere and it is also why
training departments are periodically cut back or even eliminated. They are not perceived as contributing or
adding value that is equal to or greater than their cost.
The importance of this contributions-inducements relationship cannot be overstated. As James Burke, CEO
of Johnson & Johnson during its Tylenol crisis, once remarked, “The ultimate measure of an organization’s
success is the extent to which it serves all of its constituencies better than its competition” (PBS Video,
1995). It falls to management, then, to manage stakeholder or constituent relationships. This is as true for
the training department and its management as it is for the larger organization.
To meaningfully evaluate training one must assess the nature of the contributions-inducements relationship
between each of the stakeholder groups and the training. What are they putting in? What are they getting
out? Are they putting in what they should? Are they getting out of it what they want or need? Do they view
the transaction as balanced or unbalanced (i.e., are they putting in more than they’re getting out)?
Typical Training Stakeholder Contributions & Inducements
The table that follows identifies some of the typical contributions and inducements that could be involved for
the various stakeholder groups with respect to a particular training course or training in general. It does not
and cannot represent all such contributions and inducements. These will vary with the course and the
people involved. A stakeholder “scorecard” must be constructed to fit the situation. However, the table below does serve as a model and a starting point.
Other groups who might be stakeholders and who might have to be added include senior managers and
executives, the community and government regulators.
© Fred Nickols 2003
(DRAFT: For Review Only – Do Not Cite without Permission)
4
A Stakeholder Approach to Evaluating Training
Table of Stakeholder Contributions and Inducements
Contributions (Put In)
Stakeholder
Groups
Inducements (Take Out)
Their time, energy, skills and knowledge,
manifested in individual training events.
Trainers
Pay, recognition, personal satisfaction in accomplishment, new insight and knowledge,
professional development, continued employment.
Resource commitments, direction, support,
leadership.
Training
Managers
Pay, pride in accomplishment and status or
standing in the organization, influence (e.g., a
seat at the table), both for themselves and
their unit.
Money, sanction, support.
Funding
Managers
Operational and financial impact of greater
value.
Opportunity costs of releasing the employee for the training, sanction, support.
Using Managers
Improved performance on the job.
Their time, attention, energy and knowledge, participation.
Trainees
Useful information and knowledge, tools and
job aids, good use of their time, improved
skills, improved standing.
Courses and course materials, development costs and their reputation.
Vendors
Money, repeat business, enhanced reputation, referrals.
The courses, materials and their time,
energy, skills and knowledge.
Developers
Pay, recognition, personal satisfaction in accomplishment, new insight and knowledge,
growth and development, improved standing.
A Process for Applying a Stakeholder Approach
At this point it is probably prudent to remind the reader that this paper presents a proposed approach to evaluating training. So far as the author knows, no one has yet done so. Stakeholder-based approaches, evaluations and scorecards have been developed for general business use but not for evaluating training. Consequently, the process outlined below is a conceptual view of how one might go about evaluating training
using a stakeholder-based approach. It is not a detailed plan. Conceptually, at least, the process is very
simple:










Identify the key stakeholder groups
Identify the contributions and inducements for each group
Prioritize these contributions and inducements
Ruthlessly reduce them to a short list for each stakeholder
Devise simple ways of measuring the satisfaction of the various stakeholders with their inducements
Devise simple ways of measuring the value of the contributions made by the various
stakeholders
Incorporate them into a Stakeholder Contributions-Inducements Scorecard
Check frequently; communicate often; use to drive productive conversations
Incorporate into meetings, the training itself and training post-mortems
Modify, modify, modify
Practically and politically, however, it will likely prove to involve a lot of hard work.
There are those who will duck accountability and shirk responsibility. Some of them are trainers. The last
thing many managers want is someone else to whom they have to be accountable, especially when they
see little coming their way in return. So, the place to begin is always with the value expected from training,
© Fred Nickols 2003
(DRAFT: For Review Only – Do Not Cite without Permission)
5
A Stakeholder Approach to Evaluating Training
be it a single offering or the entire training function. If that value proposition cannot be made clear and compelling there is little hope for the training let alone a stakeholder or any other approach to evaluating it.
Mutual Accountability and Shared Responsibility
A stakeholder approach leads to mutual accountability and shared responsibility. Trainers are not and cannot be solely responsible for the success of training. The trainees have something to do with that, too. So do
developers and vendors and managers and clients. The ROI of training is neither the sole nor the paramount
measure of training. The “smiles test” provides some useful information but it also allows trainees to criticize
the training without any accompanying assessment of their behavior and performance as trainees. There is
such a thing as “a responsible trainee” and the evaluation of training rarely takes stock of that ingredient, yet
it is essential to the success of training. The managers who fund the training have a right to expect something for the money they spend but they also have an obligation to contribute to the success of that training
(even if it’s only to sit still and be interviewed regarding their expectations of the training or to explain the
rationale that led them to conclude that training is the solution to some problem of performance). There is,
then, the notion of a “responsible client” as well. Under a stakeholder approach, the various stakeholders
are accountable to one another and they share the responsibility for success.
What Value Is Added by A Stakeholder Approach?
 It focuses everyone on the value proposition, that is, the value to be provided by training.
 It shifts the emphasis in training from transforming trainees to providing value to stakeholders.
 It elucidates and clarifies the “stakes” – the value expected from the training by the various stakeholders.
 It focuses trainers and the training on the needs and requirements of its many constituencies.
 It accommodates all the various stakeholder perspectives; it offers a balanced view.
 It accommodates the Kirkpatrick model and the ROI approach – when and as they are relevant to
the stakeholder groups.
 It comes to grips naturally with the politics of evaluation.
 It takes evaluation out of the realms of special exercises and specialized expertise.
 It forces training design to focus on results and thus moves evaluation “up front” where it belongs.
 It encourages and supports mutual accountability and shared responsibility.
 It opens up some long ignored success factors (e.g., the “responsible trainee” and the “responsible
client”).
Implications & Conclusion
If one accepts the notion that training has multiple constituencies or stakeholders whose needs, wants, requirements and preferences must be taken into account, one must also accept that the only effective way of
doing so is take them into account during the design, development and delivery of the training. Anything
else is bound to come up short at evaluation time. Moreover, it is well to keep in mind that, although training
providers and their constituencies might agree in general about the results to be obtained from training, they
also hold very different perceptions regarding the criteria to be used in evaluating training programs (Michalski, G., 1997). For this reason, evaluation issues belong on the front-end of training endeavors as well
as on the back-end. The real question, then, is how does one design, develop and deliver training so as to
meet all the stakeholders’ needs and requirements? Do this and do it well and any subsequent evaluation is
certain to be favorable. How does one do that? Well, that’s beyond the purview of this paper but a few principles to keep in mind are listed below:





Find the shared interest.
Involve the stakeholders.
Think optimization, not maximization.
Be willing to make trade-offs and take shortcuts.
Remember, design is as much or more art as it is science: Trust your gut.
The stakeholder view, though not without its flaws (Key, 1999) and critics (Jennings, 1999), is gathering
momentum in management thinking (Donaldson & Preston, 1995) and is increasingly reflected in managerial
tools and actions aimed at assessing organizational and managerial performance (Atkinson, Waterhouse &
Wells, 1997; Fraser & Zarkada-Fraser, 2003; Neely, Adams & Crowe, 2003). As one group of observers
writes, “The days when companies could survive and prosper by focusing on the wants and needs of one
stakeholder – the shareholder – are long gone” (Neely, Adams & Kennerly, 2002). Trainers, too, must satisfy multiple constituencies. Adopting a stakeholder approach to evaluating training is a step in the right direction.
© Fred Nickols 2003
(DRAFT: For Review Only – Do Not Cite without Permission)
6
A Stakeholder Approach to Evaluating Training
References
1. Alliger, G.M. & Janak, E.A. (1989). Kirkpatrick’s levels of training criteria: Thirty years later. Personnel Psychology, 42(2), 331-342.
2. Atkinson, A. A., Waterhouse, J. H., and Wells, R. B. (1997). A stakeholder approach to strategic
performance measurement. Sloan Management Review (Spring).
3. Barnard, C. A. (1947). The functions of the executive. Cambridge, MA: Harvard University Publishing.
4. Burke, J. (1995). Remarks made during a PBS video [author’s notes].
5. Donaldson, T. and Preston, L. (1995). The stakeholder theory of the corporation: concepts, evidence, and implications. Academy of Management Review 20:1 65-91.
6. Fraser, C. and Zarkada-Fraser, A. (2003). Investigating the effectiveness of managers through an
analysis of stakeholder perceptions. Journal of Management Development 22:9 762-783.
7. Freeman, R. (1984). Strategic management: a stakeholder approach. Boston, MA: Ballinger
8. Holton, E.F. (1996). The flawed four-level evaluation model. Human Resources Development Quarterly, 7(1), 5-21.
9. Jennings, M. (1999, April). Stakeholder theory: letting anyone who’s interested run the business –
no investment required. Paper presented at a conference titled Corporate Governance: Ethics
Across the Board, hosted by the Center for Business Ethics at the University of St. Thomas, Houston,
TX.
Retrieved
December
6,
2003
from
http://www.stthom.edu/cbes/conferences/marianne_jennings.html
10. Kaufman, R., and Keller, J. M. (1994). Levels of evaluation: beyond Kirkpatrick. Human Resource
Development Quarterly, 5, 371-380.
11. Key, S. (1999). Toward a new theory of the firm: a critique of stakeholder “theory.” Management
Decision 37:4, 317-328.
12. Kirkpatrick, D. L. (1975a). Techniques for evaluating training programs, part 1: reaction. In Evaluating Training Programs, compiled by D. L. Kirkpatrick, 1-5. Madison, WI: ASTD.
13. Kirkpatrick, D. L. (1975b). Techniques for evaluating training programs, part 2: learning. In Evaluating Training Programs, compiled by D. L. Kirkpatrick, 16-9. Madison, WI: ASTD.
14. Kirkpatrick, D. L. (1975c). Techniques for evaluating training programs, part 3: behavior. In Evaluating Training Programs, compiled by D. L. Kirkpatrick, 10-13. Madison, WI: ASTD.
15. Kirkpatrick, D. L. (1975d). Techniques for evaluating training programs, part 4: results. In Evaluating Training Programs, compiled by D. L. Kirkpatrick, 14-17. Madison, WI: ASTD.
16. March, J. G. and Simon, H. A. (1958). Organizations. New York, NY: John Wiley & Sons.
17. Michalski, G. V. (1997, November). Stakeholder variation in perceptions about training program
results and evaluation: a concept mapping investigation. Paper presented at American Evaluation
Association Conference, San Diego, CA.
Retrieved December 6, 2003 from
http://www.conceptsystems.com/papers/paperusr/michalsk/aea51.htm
18. Neely, A., Adams C. and Kennerly, M. (2002). The performance prism: the scorecard for measuring and managing success, 1. London: Financial Times Prentice-Hall
19. Neely, A., Adams, C. and Crowe, P. (2003). The performance prism in practice. Retrieved December 6, 2003 from http://www.som.cranfield.ac.uk/som/cbp/PrismInPractice.pdf
20. Nickols, F. W. (2000). Evaluating training: there is no “cookbook” approach. In J. Woods & J. Cortada (Eds.), The 2001 ASTD Training & Performance Yearbook (pp. 322-333). New York, NY:
McGraw-Hill.
21. Phillips, J. (1997). Return on investment in training and performance improvement programs.
Houston, TX: Gulf Publishing Company.
22. Watkins, R., Leigh, D., Foshay, R., & Kaufman, R. (1998). Kirkpatrick plus: evaluation and continuous improvement with a community focus. In Educational Technology Research and Development, 46:4 90-96
© Fred Nickols 2003
(DRAFT: For Review Only – Do Not Cite without Permission)
7
A Stakeholder Approach to Evaluating Training
Author Bio & Contact Information
Fred Nickols is a senior management consultant and executive with almost 50 years of experience in the
workplace, much of it associated with training and development and with other efforts to improve performance and productivity in organizational settings. For many years he was an executive director with Educational Testing Service. His career began in the United States Navy where he served on active duty for 20
years, retiring in 1974 with the rank of Chief Petty Officer. While in the Navy he received his training and
early experiences as an instructor, a writer of programmed instructional materials, an instructional systems
specialist and an internal management and organizational development (OD) consultant. His consulting
career spans more than 30 years and his clients include many well-known corporations, non-profit organizations and government agencies. He has published dozens of articles in a wide variety of professional journals and trade magazines. Currently, he is the managing partner of Distance Consulting LLC.
Fred Nickols
www.nickols.us
fred@nickols.us
Author’s Note
A revised version of this article with a slightly different title appears in Advances in Developing Human Resources. Citation information is as follows:
Nickols, F.W. (2005). Why a stakeholder approach to evaluating training. Advances in Developing
Human Resources. 7(1), 121-134.
© Fred Nickols 2003
(DRAFT: For Review Only – Do Not Cite without Permission)
8
TRAINING IMPACT QUESTIONNAIRE
DeWine, S. (1987). Evaluation of organizational communication competency: The
development of the communication training impact questionnaire. Journal of
Applied Communication Research, 15(1-2), 113-127.
Purpose: The Training Impact Questionnaire (Training IQ) is a post-training
instrument that measures employees’ perception of their capabilities to utilize a
tool taught in a training program.
Theory/Background: DeWine found that training and development programs
conducted in-house of organizations lack an effect evaluation process for the
training. Other evaluation tools focus mostly on employee reaction to training and
do not look at the long term impact that training has on job performance. This
instrument was created to fill the need for a training evaluation form that also looks
at the benefits of training, the perceived skills and appropriate application of skills
in the work place.
Description: The Training IQ is a 20-item questionnaire that uses declarative
statements and asks respondents to respond using a 5-point Likert scale that ranges
from (5) strongly agree to (1) strongly disagree. Items 2, 3, 8, 9, 11-19 are reversecoded. This questionnaire is meant to be given out two to four weeks after a
training session. This delay provides enough time for employees to utilize the new
skill on the job.
There are two factors in this questionnaire. The first factor is called “Relationship
of training to job,” and measures the association between an employees job
requirements and the information that was taught during the training session. The
second factor is called “Skilled performance,” and measures the extent to which
the new skill is used by the employee in their job.
TRAINING IMPACT QUESTIONNAIRE
This series of statements are possible perceptions of an employee regarding
previously conducted training and its impact on his or her ability to apply skills
taught during training to the job. Please respond to each statement by placing the
appropriate number in the blank to the left of each item.
5 = strongly agree
4 = agree
3 = neutral
2 = disagree
1 = strongly disagree
1. After attending this training program, I am interested in attending other training
programs.
2. I don’t perform the skill on the job because the skill is too difficult for me.
3. I use this skill regularly on the job.
4. Because of learning this skill I feel more comfortable about doing my job.
5. Because of attending this training program, I feel better about the company.
6. I learned to perform the tasks well in the training program, but I could have
learned it just as easily from a manual or an instruction sheet.
7. I think my participation in this training program will help me to advance in the
company.
8. I didn’t learn this skill in the training program, so I had to learn it on the job.
9. Work conditions don’t allow me to perform the skill the way I learned it in
training, so I do the task differently on the job.
10. After training I would perform this skill with practicing.
11. I don’t perform the skill on the job because the skill comes up so rarely that I
forget how to do it.
12. I don’t perform the skill on the job because I didn’t learn the skill in the
training program, so I get help to do the skill.
13. I had trouble learning the skill because the training program was confusing.
14. I never perform this skill on the job.
15. The skill isn’t part of my job.
16. I don’t perform the skill because I was assigned a different job.
17. I had trouble learning the skill in the training program because there wasn’t
enough reference material.
18. I perform the skill differently on the job because the skill doesn’t work the way
I learned it in training.
19. I perform the skill differently on the job because my supervisor told me to do it
differently.
20. I learned to perform the task well in the training program because the program
was effective.