NEED IN 20 hours: 6 page Article Critique Assignment – Human Resources

  

NEED IN 20 hours: 6 page Article Critique Assignment – Human Resources

Article Critique Assignment:

6 double-spaced typed pages in length excluding cover page based on the attached file marked instructions.

The article you are critiquing is attached too. (File named Holton)

Article Critiques

Objectives

This assignment is designed for students’ self-directed learning regarding the subject of program evaluation and analysis. Among others, the objectives include

1. Conduct research oriented learning

1. Practice analytical and critical thinking

Assignments

You need to identify a peer-reviewed published paper on the subject of HRD/Tech measurement and evaluation and following the requirement below. You may choose any article in the reading list of this course. You may also select articles outside the reading list as long as they are relevant to the topic of this course. (Note: Please do not use articles from general or commercial websites. It has to have a journal title and volume/issue and page numbers.)

Requirements:

1. Begin the Critiques with a complete bibliographic citation in proper APA style [Author, (year of publication). Article title. Journal Title, volume (issue), page range]. APA style is established by American Psychological Association for all psychology related fields. For detailed APA requirements, please visit www.apastyle.org.

1. Briefly summarize why the article is important for students measurement and evaluation (M&E) in HRD 5307. (e.g., How does it relate to M&E in HRD? Why is it important to enhance our understanding in M&E? How important is the article to the field of HRD?)

1. Summarize the article’s content: No more than 2 pages and use your own words to paraphrase. Please avoid copying from the article abstract.

1. Discuss the practical applications (if any) of the article for practitioners. What should they be able to do or to learn regarding M&E after reading the article?

1. Be sure to critique the article, discuss any weaknesses or any occasions when you think the author’s theory, model, process or ideas won’t work and explain why with literature support. Note that a major portion of your grade will depend on the quality of your critiques.

1. The paper should be at least 6 double-spaced typed pages in length excluding cover page. 20 percent points will be deducted for late submissions (all critiques must be completed to receive a grade for the assignment).

1.

You are strongly encouraged to reference additional research or articles for a high quality paper. Please structure your writing with headings and subheadings.

10.1177/152342230427208

0

Advances in Developing Human Resources February 200

5

Holton / HOLTON’S EVALUATION MODEL

Holton’s Evaluation Model:
New Evidence and
Construct Elaborations

Elwood F. Holton, III

The problem and the solution. Holton proposed the HRD
Evaluation and Research Model as a comprehensive framework
for diagnosing and understanding the causal influences of HRD
intervention outcomes. Unfortunately, a full test of Holton’s
model has not been possible because tools to measure the con-
structs in the model did not exist. This article reviews recent
studies relevant to the constructs in Holton’s model and
updates it by delineating specific constructs that should be mea-
sured in each of the conceptual categories proposed

.

Keywords: HRD evaluation; evaluation models; HRD outcomes; HRD
theory; Learning Transfer System Inventory; LTSI

Holton (1996) sharply criticized Kirkpatrick’s (1959) four-level evaluation
model and proposed the HRD Evaluation and Research Model as a more
comprehensive framework for diagnosing and understanding the causal
influences of HRD intervention outcomes. The original model (see Figure
1) was theoretically derived and more conceptually comprehensive than
Kirkpatrick’s simple four-level taxonomy. Three outcome levels are
hypothesized in the model: learning, individual performance, and organiza-
tional performance. Following Noe and Schmitt (1986), the macro-
structure of that model hypothesizes that HRD outcomes are a function of
factors in three construct domains: ability, motivation, and environmental
influences. The model further specified conceptual constructs in each
domain that are hypothesized to influence each of the three outcome levels.
Secondary influences are also included, particularly those that affect moti-
vation to learn.

The model addressed one of the biggest risks of the four-level model,
specifically, that any failure to achieve outcomes from an intervention
would be attributed to the intervention itself when it could well be due to
moderating variables. Perhaps the best example of this is the situation that

Advances in Developing Human Resources Vol. 7, No. 1 Februar y 2005 37-5

4

DOI: 10.1177/1523422304272080
Copyright 2005 Sage Publications

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

http://adh.sagepub.com/

arises when learning outcomes (level 2) from a training intervention are
positive but no on-the-job behavior change occurs (level 3) because the
transfer climate is poor. Unless the transfer climate is evaluated, the deci-
sion derived from the four-level model would be that the training interven-
tion had failed and needed to be changed. The correct evaluation decision
derived from the Holton model would be that the training intervention did
not need to be changed but the organization did not have the transfer climate
to support it, so an organization development intervention would be needed.

Unfortunately, a full test of my model has not been possible because
many of the tools to measure the constructs in the model did not exist. How-
ever, since I first proposed my model in 1996, research evidence has contin-
ued to accumulate and generally supports it, although some research sug-
gests that modifications are needed. This article reviews selected recent
studies relevant to the constructs in my model and updates the model by
delineating specific constructs that should be measured in each of the con-
ceptual categories proposed. In certain instances, the model is modified
based on new research or theory. The result is an updated version of the
model that is more appropriate for empirical testin

g.

38 Advances in Developing Human Resources Febr uar y 2005

Learnin

g

Individual

Performance

Organizational
Performance

Outcom

es

Environment

Elemen

ts

Ability/Enabling
Elements

Motivation

Elements

Secondary
Influences

Motivation to
Learn

Perceptions of
Training

Transfer Climate

Motivation to
Transfer

Learning Design

Ability

Transfer
Design

Linkage to
Organization Goals

Individual
Characteristics

Intervention
Readiness

Job Attitudes

Intervention
Fulfillment

External Events

Expected
Utility/Return
on

Investment

FIGURE 1: HRD Evaluation Research and Measurement Model From Holton (1996

)

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

http://adh.sagepub.com/

Influences on Learning
This section reviews recent research on factors influencing individual

learning outcomes. These are the first domain of outcomes in the model
moving sequentially from left to right.

Secondary Influences—Individual Learner Characteristics

Some of the most intriguing research in recent years has been focused on
learner dispositional influences on learning, primarily through motivation
to learn. Dispositional influences are a general category of individual traits
that are relatively stable and enduring and that predispose a person toward
certain tendencies or patterns. The best-known example of dispositional
characteristics is personality, although there are many others.

Much of the work on personality traits has used the Big Five framework.
The Five Factor Model dominates the current view of personality and pro-
vides a unifying structure to its study. This model has, in fact, garnered so
much support that the FFM “has now become an almost universal template
for understanding the structure of personality” (Ferguson & Patterson,
1998, p. 789). As the name implies, this model suggests that there are five
broad categories of traits at the top of the personality trait hierarchy. The rel-
atively orthogonal five-factor taxonomy resulted from decades of research
on the structure of human personality (Costa & McCrae, 1992) and has
gained the support of numerous researchers (e.g., Costa & McCrae, 1995;
Digman, 1990; Goldberg, 1990). As highlighted by Costa and McCrae
(1992), the five dimensions of the Five Factor Model of personality are
neuroticism (or emotional stability), extraversion, openness to experience,
agreeableness, and conscientiousness.

Colquitt, LePine, and Noe’s (2000) seminal meta-analysis on training
motivation research examined research on a variety of influences on learn-
ing and transfer outcomes of training. There were a number of important
findings in their work relevant to this model. First, two dimensions of the
Big Five personality measures were found to influence motivation to learn
through pretraining self-efficacy. They were conscientiousness and anxiety
(a component of neuroticism). Conscientiousness refers to the extent to
which someone is dependable, persevering, hardworking, disciplined,
deliberate, and achievement oriented (Herold, Davis, Fedor, & Parsons,
2002, p. 854). It has consistently been linked to motivation to learn and
training outcomes (Barrick & Mount, 1991; Colquitt & Simmering, 1998;
Naquin & Holton, 2002).

Emotional stability, also known by its opposite trait of neuroticism,
reflects the absence of feelings of anxiety, insecurity, and nervousness. It
reflects to some degree positive psychological adjustment. People scoring

Holton / HOLTON’S EVALUATION MODEL 3

9

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

http://adh.sagepub.com/

high on this dimension are generally less anxious and more upbeat in their
outlook. Emotional stability has been found to be related to training out-
comes, including transfer outcomes (Herold et al., 2002).

Several studies also found that openness to experience, another of the Big
Five personality dimensions, is related to both learning and transfer out-
comes (Barrick & Mount, 1991; Herold et al., 2002; LePine, Colquitt, &
Erez, 2000), although Naquin and Holton (2002) did not find a significant
effect for this construct. Openness to experience is associated with intellect,
curiosity about one’s environment, and a willingness to explore new things.
Because training requires persons to embrace new things, this trait should
be helpful to trainees (Herold et al., 2002, p. 855).

Naquin and Holton (2002) found support for agreeableness as a predic-
tor, whereas most studies have not. Agreeableness is an interpersonal trait
and reflects the degree to which a person is generally a cooperative, compas-
sionate, and trusting person in interpersonal situations. Only limited sup-
port has been found, perhaps because its effect is limited to training related
to interpersonal skills training.

In sum, three of the Big Five factors—conscientiousness, neuroticism,
and openness to experience—have received strong support in the literature,
whereas agreeableness has received weak support. Extraversion appears to
have no relationship with training outcomes. Thus, the three personality
traits with strong support are recommended for inclusions in the model as
personality trait measures.

Moving beyond the Big Five, the latest trait to emerge in literature has been
goal orientation. Goal orientation posits that individuals are of two
types—learning oriented versus performance oriented.

A learning orientation is characterized by a desire to increase one’s competence by developing
new skills and mastering new situations. In contrast, performance orientation reflects a desire to
demonstrate one’s competence to others and to be positively evaluated by others. (p. 498)

Research has shown that individuals with a learning orientation tend to pursue
difficult learning challenges and persist in the face of failure of learning difficul-
ties. Persons with a performance orientation tend to see the same situations as
threatening and to withdraw from them. Thus, a learning orientation is associ-
ated with more positive learning outcomes, whereas a performance orientation
is associated with negative or neutral learning outcomes (Bell & Kozlowski,
2002; Chen, Gully, Whiteman, & Kilcullen, 2000; Colquitt & Simmering, 1998;
Ford, Smith, Weisbein, & Gully, 1998). Most recently, goal orientation has been
shown to have an interactive effect with cognitive ability (Bell & Kozlowski,
2002). Because of the compelling evidence in the literature, goal orientation is
recommended as another individual difference variable for the model.

Another dispositional variable that has received recent support in the lit-
erature is locus of control (Colquitt et al., 2000). Persons with an internal
locus of control tend to have more positive attitudes and motivation toward

40 Advances in Developing Human Resources Febr uar y 2005

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

http://adh.sagepub.com/

training because they are more likely to believe that they can change their
abilities and motivation through their own actions. Persons with an external
locus of control are more likely to believe that changes in performance are
only possible through changes in factors external to themselves. Logically,
then, persons with an internal locus of control are more likely to believe that
they can improve their skills and performance by exerting effort in training.
Locus of control has been shown to be a significant predictor of motivation
to learn in Colquitt et al.’s (2000) meta-analysis and thus is recommended
for inclusion in the model.

In summary, recent research points to five variables as measures of the
individual characteristics category in the Holton Evaluation Model. They
are (a) conscientiousness, (b) neuroticism (emotional stability), (c) open-
ness to experience, (d) goal orientation, and (e) locus of control. The origi-
nal theory posited that the effect of these variables would be fully mediated
by motivation to learn. However, Colquitt et al.’s (2000) meta-analysis
showed convincingly that these learner characteristics had both an indirect
effect through motivation to learn as well as a direct effect on learning.
Thus, it seems appropriate to add a path directly from these individual
characteristics to learning.

Secondary Influences—Job Attitudes

The second category of secondary influences posited in the Holton
model was job attitudes. The theory posits that job attitudes should affect
both motivation to learn and motivation to transfer learning. Logically, it
presumes that individuals who have more positive attitudes toward their
organization would be more likely to engage in learning that will benefit the
organization through improved performance. Unfortunately, as Colquitt
et al. (2000) note in their meta-analysis, job attitude variables have not
received sufficient research attention in the training literature. In fact,
most of the job attitude variables could not even be included in their
meta-analytic path analysis because of the paucity of studies.

The limited research on job attitudes has focused on several variables.
First is job involvement, which has been the most frequently researched jo

b

attitude variable and has consistently been shown to be a significant predic-
tor of motivation. Second is organizational commitment. Naquin and
Holton (2002) provide compelling evidence of the importance of these vari-
ables in their study of the effects of dispositional variables and work atti-
tudes on a construct they identified as Motivation to Improve Work Through
Learning (MTIWL). MTIWL is a higher order construct combining motiva-
tion to learn and motivation to transfer. In their study, work commitment,
which included both job involvement and two dimensions of organizational
commitment, was the second strongest predictor of motivation after positive

Holton / HOLTON’S EVALUATION MODEL 4

1

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

http://adh.sagepub.com/

affectivity. Work commitment also mediated the effects of conscientious-
ness on motivation. Thus, work commitment was shown to have a strong and
direct effect on both dimensions of motivation. However, Colquitt et al.’s
(2000) meta-analysis did not find job involvement to be a significant predic-
tor and they were unable to test organizational commitment due to an
insufficient number of studies.

Given the limited evidence, it is hard to know definitively which job atti-
tudes to measure. It is clear that more research is needed in this area. Given
that there is some evidence supporting job involvement and organizational
commitment as predictors of motivation, these two variables are included in
the model primarily because they need to be more carefully studied to
understand their effect on learning and transfer outcomes.

Perceptions of Training

Perceptions of training, also known as reactions, have been the most con-
troversial domain of evaluation. Kirkpatrick (1998) sanctioned reactions as
legitimate outcomes, calling them level 1 outcomes. Holton (1996) demoted
them to a moderator variable in the HRD Evaluation and Research Model,
based largely on meta-analysis research that has shown low or no relation-
ship between reaction outcomes and higher level outcomes of learning and
performance (Alliger & Janak, 1989; Alliger, Tannenbaum, Bennett, Trave,
& Shotland, 1997). Swanson and Holton (1999) included them as an out-
come but advocated not trying to achieve high levels of reaction results in
order not to overemphasize reaction outcomes because doing so often
underemphasizes learning and performance outcomes.

Recent research has shed new light on the multidimensionality of reac-
tions and their relationship to learning and performance outcomes. Morgan
and Casper (2000) conducted a factor analytic study of a large database of
responses to a multidimensional reaction instrument. They demonstrated
that reaction measures were, in fact, multidimensional, including a factor
they labeled as “Utility Reactions” along with other more typical affective
reaction measures. These results were cross-validated with confirmatory
factor analysis in a split-sample design. They suggested that models of
training effectiveness such as the Holton model might benefit from incorpo-
rating a multidimensional treatment of participant reactions.

Two studies have suggested that utility reactions may have some incre-
mental validity in predicting learning or performance outcomes. Tan, Hall,
and Boyce (2003) created a utility reaction scale that also incorporated
behavioral intentions and tested the effect of the affective and utility reac-
tion measures on learning and performance. Their results showed that one
utility reaction scale was a significant predictor over and beyond a pretest in

42 Advances in Developing Human Resources Febr uar y 2005

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

http://adh.sagepub.com/

predicting learning. However, no reaction measures were significant in
predicting performance.

Ruona, Leimbach, Holton, and Bates (2002) added support to a role for
utility reactions in predicting motivation to transfer learning. In their analy-
sis, they first entered transfer system perceptions into a hierarchical regres-
sion model and then forced the utility reaction measure to enter last. Utility
reactions were a significant predictor and increased the model R2 from .41 to
.46, suggesting that utility reactions added predictive power over and
beyond transfer system perceptions.

Together, this research suggests that perceptual measures of training
completed by trainees may have a role in predicting learning and perfor-
mance outcomes, albeit a small one. It is clear that the research is inconclu-
sive, but it suggests several tentative conclusions. First, there is no support
for including affective reactions in the model. No research is emerging that
shows that affective reactions have any role in predicting important out-
comes from training. Second, research suggests that utility reactions offer
small but significant predictive power to the model. Finally, an intriguing
new dimension, behavioral intentions, has emerged with a strong theoretical
base that warrants inclusion in the model and further testing. Based on these
findings, two constructs are recommended for the Holton model for the per-
ceptions of training domain: utility reactions and behavioral intentions.

Influences on Individual Performance
The next domain of outcomes to consider is individual performance. In

this section, research on factors influencing individual performance is
examined and enhancements to the Holton model are proposed.

Learning Transfer Research

Immediately after publishing the model shown in Figure 1, I began work
to move toward a comprehensive test of the model. Unfortunately, there
were glaring gaps in both theory and measurement tools that needed to be
addressed before any full test was possible. Perhaps the most glaring gap
was in the area of transfer climate. At that time, there was no validated
framework to define transfer climate nor any generally accepted and vali-
dated instrument to measure what was identified in my model as transfer
design, transfer climate, and motivation to transfer. My associates and I
quickly began to work on a conceptual framework and measurement instru-
ment. Few people realized that it was not learning transfer per se that drove
our interest but rather our desire to work toward a comprehensive test of the
Holton model.

Holton / HOLTON’S EVALUATION MODEL 4

3

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

http://adh.sagepub.com/

This program of research now focuses on what we call the learning trans-
fer system, instead of transfer climate (Holton, Bates, & Ruona, 2000). The
learning transfer system is defined as all factors in the person, training, and
organization that influence transfer of learning to job performance. Transfer
climate, the more common term, is actually but one subset of factors that
influences transfer, although the term is sometimes incorrectly used to refer
to the full set of influences. Other influences on transfer include training
design, personal characteristics, opportunity to use training, and motiva-
tional influences. Thus, the transfer system is a broader construct than trans-
fer climate but includes all factors traditionally referred to as transfer cli-
mate. Transfer can only be completely understood and influenced by
examining the entire system of influences.

The instrument that resulted from this program of research is called the
Learning Transfer System Inventory (LTSI). It has been well tested with
strong evidence of construct validity (Bookter, 1999; Holton et al., 2000),
initial evidence of criterion validity (Bates, Holton, Seyler, & Carvalho,
2000; Ruona et al., 2002; Seyler, Holton, Bates, Burnett, & Carvalho, 1998),
and good cross-cultural validity (Chen, 2003; Khasawneh, 2004; Yamnill,
2001). The LTSI program of research has resulted in a framework that
defines 16 constructs that make up the learning transfer system. Table 1
shows the constructs measured by the LTSI, their definition, and a sample
item along with scale reliabilities. These 16 factors are believed to measure
the full system of factors that influence learning transfer, although further
criterion validation studies are still under way.

The LTSI defines and measures all the variables that should be measured
in the transfer design, transfer climate, and motivation to transfer boxes of
the Holton model. Figure 2 shows how the constructs of the LTSI map to
these three conceptual categories.

Motivation to Improve Work Through Learning
One of the most important developments in the Holton model has been

the reconceptualization of the motivation constructs for both learning and
transfer. In the original model, motivation was conceptualized in the tradi-
tional way, namely, as the two separate construct domains of Motivation to
Learn (MTL) and Motivation to Transfer (MTT). As shown in Figure 1, a
relationship between the two was posited in that MTT was hypothesized to
influence MTL, but they were still conceptualized as separate constructs.

Naquin and Holton (2003) completely reconceptualized motivation by
creating the construct MTIWL. The work improvement process in HRD,
they argue, is not just a function of learning or training. Rather, it requires
learners to acquire knowledge and transfer that knowledge into improved
work outcomes or productivity. They further argue that using MTL is too

44 Advances in Developing Human Resources Febr uar y 2005

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

http://adh.sagepub.com/

45

TA
B
LE

1
:

L
e
a
rn

i

n
g

T

ra
n

sf
e
r

S
ys

t

e
m

I
n
ve

n
to

ry
(

LT
S

I)
S

ca
le

D
e
fi

n
it

i

o
n

s
a
n

d
S

a
m

p
le

I
te

m
s

#
o

f
F

a
ct

o
r

D
e
fi
n
it

io
n

S
a
m

p
le
I
te

m
It

e
m

s
α

T
ra

i

n
in

g-
sp

ec
ifi

c
sc

al
es

Le
ar

n
er

re
ad

i

n
es

s
ex

te

n
t

t

o
w

h
ic

h
in

d
iv

id
u
al

s
ar

e
p
re

p
ar

e

d
to

e

n
te

r
an

d
p
ar

ti
ci

p
at

e
in

tr
ai

n
in

g
B

ef
o

r

e
th

e
tr

ai
n
in

g,
I

h
ad

a
go

o
d

u
n
d
er

–
st

an
d

in

g
o

f

h
o

w
it

w
o

u
ld

fit
m

y
jo

b
-r

el
at

ed
d
ev

el
o

p
m

en
t.

4
.7

3

M
o

ti
va

ti
o

n
to

tr
an

sf
er

d
ir

ec
ti

o
n
,
in

te
n
si

ty
,
an

d

p
er

si
st

e

n
ce

o
f

ef
fo

r

t
to

w
ar

d
u
si

n
g

in
a

w
o

rk
se

t

t
in

g
sk

ill
s

an
d

k

n
o

w
le

d
ge

le
ar

n
ed

I
ge

t
ex

ci
te

d
w

h
en

I
th

in
k

ab
o

u
t

tr
yi

n
g

to

u
se

m
y

n
ew

le
ar
n
in
g
o

n
m

y
jo

b
.

4
.8

3

P
o

si
ti

v

e
p
er

so
n
al

o

u
tc

o
m

es

d
eg

re
e

to
w

h
ic

h
ap

p
ly

in
g

tr
ai
n
in
g
o

n
th

e
jo

b
le

ad
s

to
o

u
tc
o
m

e

s
th

at
ar

e

p
o

si
ti

ve
fo

r
th

e
in
d
iv
id
u
al

E
m

p
lo

ye
es

in
th

i

s
o

rg
an

iz
at

io
n

re
ce

iv
e

va
ri

o
u
s

“p
er

ks
”

w
h
en

th
ey

u
se
n
ew

ly
le

ar
n
ed

sk
ill

s
o
n
th
e
jo
b
.

3
.

6

9

N
eg

at
iv

e
p
er
so
n
al

o
u
tc

o
m

es
ex

te
n
t

to
w
h
ic
h
in
d
iv
id
u
al

s
b
el

ie
ve

t

h
at

n
o

t
ap

p
ly
in
g
sk
ill

s
an

d
kn

o
w

le
d
ge

le
ar
n
ed

in
tr

ai
n
in

g
w

ill
le

ad
to

n
eg

at
iv
e
p
er
so
n
al
o
u
tc
o
m
es

If
I

d
o

n
o

t
u
se

m
y
tr
ai
n
in
g,
I

w
ill

b
e

ca
u
–

ti
o
n
ed
ab
o
u
t

it
.

4
.7
6

P
er

so
n
al

ca
p
ac

it
y

fo
r

tr
an
sf
er

ex
te

n
t
to
w
h
ic
h
in
d
iv
id
u
al

s

h
av

e
th

e
ti

m
e,

en
er

gy
,
an

d
m

en
ta

l
sp

ac
e

in
th

ei
r

w
o

rk
liv

e

s
to

m
ak

e
ch

an
ge

s
re

q
u
ir

ed
to
tr
an
sf
er
le
ar
n
in

g
to

th
e

jo
b

M
y

w
o

rk
lo

ad
al

lo
w

s
m

e
ti

m
e

to
tr

y
th

e
n
ew

th
in

g

s
I

h
av

e
le

ar
n
ed

.
4

.6
8

P
ee

r
su

p
p
o

rt
ex

te
n
t
to
w
h
ic

h
p
ee

rs
re

in
fo

rc
e

an
d

s

u
p

p
o

rt
u
se

o
f
le
ar
n
in
g
o
n
th
e
jo

b
M

y
co

lle
ag

u
es

en
co

u
ra

ge
m

e
to

u
se
th
e
sk
ill
s
I
h
av
e
le
ar
n
ed
in
tr
ai
n
in

g.
4

.8
3

(c
o
n
ti

n
u
ed

)
at AHRD on January 26, 2016adh.sagepub.comDownloaded from

http://adh.sagepub.com/

46

Su
p
er

vi
so

r
su
p
p
o
rt
ex
te
n
t
to
w
h
ic

h
su

p
er
vi
so

rs
/m

an
ag

er
s

su
p
–

p
o

rt
an

d
re

in
fo
rc
e
u
se
o
f
tr
ai
n
in
g
o
n
th
e
jo
b
M
y

su
p
er

vi
so

r
se

ts
go

al
s

fo
r
m
e

th
at

en
co
u
ra
ge
m
e
to

ap
p
ly

m
y
tr
ai
n
in
g
o
n
th
e
jo
b
.

6
.9

1
Su
p
er
vi
so

r
sa

n
ct

io

n
s

ex
te
n
t
to
w
h
ic
h
in
d
iv
id
u
al

s
p
er

ce
iv

e
n
eg

at
iv

e
re

sp
o

n
se

s
fr

o
m
su
p
er
vi
so
rs
/m
an
ag
er
s
w
h
en
ap
p
ly
in
g
sk
ill

s
le

ar
n
ed
in
tr
ai
n
in
g
M
y
su
p
er
vi
so

r
o

p
p
o

se
s

th
e
u
se
o
f
th
e

te
ch

n
iq

u
es

I
le

ar
n
ed
in
tr
ai
n
in

g.
3

.6
3

P
er
ce
iv

ed
co

n
te
n
t

va
lid

it
y
ex
te
n
t
to
w
h
ic

h
tr

ai

n
ee

s
ju

d
ge
tr
ai
n
in

g
co

n
–

te
n
t

to
ac

cu
ra

te
ly

re
fl
ec

t
jo

b
re

q
u
ir

em
en

ts
W

h
at

is
ta

u
gh

t
in
tr
ai
n
in

g
cl

o
se

ly
m

at
ch

es
m

y
jo
b
re
q
u
ir
em
en

ts
.

5
.8

4
T
ra

n
sf

er
d
es

ig
n

d
eg
re
e
to
w
h
ic

h
(a

)
tr

ai
n
in

g
h
as

b
ee

n
d
es

ig
n
ed

an
d

d
el

iv
er

ed
to

gi
ve

tr
ai
n
ee
s
th

e
ab

ili
ty

to
tr

an
sf

er
le

ar
n
in

g
to
th
e

jo
b
,
an

d
(b

)
tr
ai
n
in

g
in

st
ru

ct
io

n
s

m
at

ch
jo

b
re
q
u
ir
em
en
ts

T
h
e

ac
ti

vi
ti

es
an

d
ex

er
ci

se
s
th
e
tr
ai
n
er

s
u
se

d
h
el

p
ed

m
e

kn
o

w
h
o

w
to

ap
p
ly
m
y
le
ar
n
in
g
o
n
th
e
jo
b
.
4
.8
5

O
p
p
o

rt
u
n
it

y
to

u
se
ex
te
n
t
to
w
h
ic
h
tr

ai
n
ee

s
ar

e
p
ro

vi
d
ed

w
it
h
o
r
o

b
ta

in
re

so

u
rc

es
an

d
ta

sk
s

o
n
th
e
jo
b

e

n
ab

lin
g

th
em

to
u
se

tr
ai
n
in
g
o
n
th
e
jo
b
T
h
e

re
so

u
rc

es
I

n
ee
d
to
u
se

w
h
at

I
le
ar
n
ed
w
ill
b
e

av
ai

la
b
le

to
m

e
af

te
r

tr
ai
n
in
g.
4
.7
0

G
en

er
al

sc
al

es
T

ra
n
sf

er
ef

fo
rt

–
p
er

fo
rm

an
ce

ex

p
ec

ta
ti

o
n
s

ex
p
ec

ta
ti
o
n
th
at
ef
fo

rt
d
ev

o
te

d
to
tr
an
sf
er

–
ri

n
g
le
ar
n
in
g
w
ill
le
ad
to

ch
an

ge
s

in
jo

b
p
er

fo
rm
an
ce
M
y
jo
b
p
er
fo
rm
an
ce

im
p
ro

ve
s

w
h
en

I
u
se

n
ew
th
in

gs
th

at
I

h
av
e
le
ar
n
ed
.
4

.8
1

TA
B
LE

1
(c

o
n

ti
n

u
e
d

)
#
o
f
F
a
ct
o
r
D
e
fi
n
it
io
n
S
a
m
p
le
I
te
m
It
e
m
s
α
(c
o
n
ti
n
u
ed
)
at AHRD on January 26, 2016adh.sagepub.comDownloaded from

http://adh.sagepub.com/

47

P
er
fo
rm
an
ce

-o
u
tc

o
m
es
ex
p
ec
ta
ti
o
n
s
ex
p
ec
ta
ti
o
n
th
at
ch
an
ge
s
in
jo
b
p
er
fo
rm
an
ce
w
ill

le
ad

to
va

lu
ed

o
u
tc
o
m

es
W

h
en

I
d
o

th
in

gs
to

im
p
ro

ve
m

y
p
er

fo
r –

m
an

ce
,
go

o
d
th
in

gs
h
ap

p
en

to
m

e.
5

.8
3

R
es

is
ta
n
ce

/

o
p
en

n
es
s
to
ch
an

ge
ex

te
n
t
to
w
h
ic

h
p
re

va
ili

n
g

gr
o

u
p
n
o

rm
s

ar
e

p
er
ce
iv

ed
b
y

in
d
iv

id
u
al
s
to

re
si

st
o

r
d
is

–
co

u
ra

ge
th

e
u
se

o
f
sk
ill
s
an
d
kn
o
w
le
d
ge

ac
q
u
ir

ed
in

tr
ai
n
in
g

P
eo

p
le

in
m

y
gr

o
u
p

ar
e
o
p
en

to
ch

an
gi

n
g
th
e

w
ay

th
ey
d
o
th
in

gs
.

6
.8

5
P
er
fo
rm
an
ce

se
lf-

ef
fic

ac
y

an
in

d
iv
id
u
al

’s
ge

n
er

al
b
el

ie
f

th
at
th
ey
ar
e

ab
le

to
ch
an
ge

th
ei

r
p
er

fo
rm
an
ce
w
h
en
th
ey

w
an

t
to

I
am

co
n
fid

en
t

in
m

y
ab

ili
ty
to
u
se
n
ew
ly
le
ar
n
ed
sk
ill
s
o
n
th
e
jo
b
.
4
.7
6
P
er
fo
rm
an
ce

co
ac

h
in

g
fo

rm
al

an
d
in
fo
rm
al

in
d
ic

at
o

rs
fr

o
m

an
o

rg
a-

n
iz

at
io

n
ab

o
u
t

an
in
d
iv
id
u
al

’s
jo

b
p
er
fo
rm
an
ce

A
ft

er
tr

ai
n
in
g,
I

ge
t

fe
ed

b
ac

k
fr

o
m

p
eo

p
le
ab
o
u
t
h
o

w
w

el
l
I

am
ap

p
ly
in
g
w
h
at
I
le
ar
n
ed
.
4
.7
0
at AHRD on January 26, 2016adh.sagepub.comDownloaded from

http://adh.sagepub.com/

limiting for organizational learning environments. What employees are
really engaged in is the process of improving work through the learning pro-
cess that necessarily entails transferring learning into job application.

MTIWL was defined as

Motivation to Improve Work Through Learning (MTIWL) =
ƒ(Motivation to Train, Motivation to Transfer)

Naquin and Holton (2003) demonstrated the initial construct validity of this new
construct using confirmatory factor analysis.

Although on the surface it might appear that this construct only combines
two existing constructs in an additive fashion, the effects are likely to be
more significant on outcomes in the Holton model. That is, persons entering
a learning situation with high levels of MTIWL are likely to have greater
motivation to engage in work-relevant learning experiences offered with
strong transfer designs that stress practice and job application than persons
with high levels of simple MTL. Thus, the nature of the MTIWL motivation
in the learning environment is expected to be substantively different from
simple MTL and will demand different types of learning experiences. Fur-
ther, they can be expected to exhibit higher rates of transfer to individual
performance.

48 Advances in Developing Human Resources Febr uar y 2005

Outcomes

Environment
Motivation

Secondary
Influences

Ability

Learning
Individual

Performance

Content Validity
Transfer Design
Personal Capacity for Transfer
Opportunity to Use

Motivation to Transfer
Transfer Effort –> Performance
Performance–> Outcomes

Feedback
Peer Support
Supervisor Support
Openness to Change
Personal Outcomes-Positive
Personal Outcomes-Negative
Supervisor Sanctions

Performance Self-Efficacy
learner Readiness

Organizational
Performance

FIGURE 2: Learning Transfer System Inventory (LTSI) Conceptual Map of Constructs

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

http://adh.sagepub.com/

Obviously, this new construct remains untested and has unknown crite-
rion validity. However, it is promising enough that I am changing the Holton
model to incorporate it and spur research on its validity. The constructs pre-
viously included in the MTL and MTT boxes in the model are now inte-
grated into one motivation domain, MTIWL.

Also unknown is whether the relationships found in the literature as dis-
cussed above between secondary influences such as dispositional factors,
job attitudes, and MTL will also hold true for MTIWL. However, I suggest
that it is a short theoretical leap to hypothesize that they will, although only
further research will definitively validate them.

Influences on

Organizational Performance

Unfortunately, there has been almost no research on factors influencing

the transfer of individual performance into organizational performance
results. This reflects the almost exclusive focus in training research on the
individual level of analysis. It also highlights a serious shortcoming of train-
ing research in that the greatest effect on organizations obviously occurs
through attaining organizational performance, yet this domain receives the
least attention in the literature.

One notable exception is a study by Montesino (2002) in which he exam-
ined the linkage between training, the strategic direction of the organiza-
tion, transfer enhancing behaviors, and usage of training on the job. Several
important findings emerged. First, a low to moderate correlation was found
between managers’ and trainees’ engagement in transfer enhancing behav-
iors and their perceptions that training was congruent with the strategic
direction of the firm. This directly supports the relationship in the HRD
Evaluation and Research Model between Expected Return/Utility of Train-
ing and MTT in that training perceived to be congruent with the firm’s strat-
egy is expected to have higher utility and therefore to increase trainee MTT.

Second, trainees who reported very high usage of training perceived a signifi-
cantly higher alignment of training with the strategic direction of the firm
(Montesino, 2002, p. 102). Montesino concludes that

apparently those trainees who saw more clearly the connection of the training program with the
strategic direction of the organization were able to apply on the job the skills they learned in the
training program in greater proportion than were the trainees who did not see that connection
clearly. (p. 103)

This clearly supports the relationship between Linkage to Organizational Goals
and individual performance in the ability domain of the Holton model.

Although this is only one study, it offers clear evidence that the relation-
ships and constructs in my model are promising. Equally evident is that
there is a dearth of research in this area and a rich opportunity exists for

Holton / HOLTON’S EVALUATION MODEL 49

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

http://adh.sagepub.com/

additional research into factors affecting the strategic effect of training on
organizational performance.

Bates and Khasawneh (2004) provided some additional insight into how
this linkage might operate. In their study, they examined the effects of a
learning organization culture and selected dimensions of transfer climate on
organizational innovation, an important organizational outcome from learn-
ing organization interventions. Their results suggested that a learning orga-
nization culture had both direct and indirect effects on organizational inno-
vation, with transfer climate constructs partially mediating the effects of the
learning organization culture. Thus, this study provides additional evidence
that Expected Utility/Return on Investment and Linkage to Organizational
Goals are important organizational factors as hypothesized in this model.
Learning organizations typically include a strong linkage between organi-
zational goals and learning, presumably resulting in higher perceived return
on investment, which should lead to greater motivation to learn. In addition,
they have indirect influences through transfer climate as shown in this
model by the relationships hypothesized between organizational influences
and transfer climate factors at the individual performance level.

Conclusion
This article accomplished three goals. First, it reviewed recent research

that supports the Holton Evaluation and Research Model. Second, it pro-
vided an update to the model by modifying it to reflect new theory, particu-
larly in the area of motivation. Third, it presented an elaboration of the
model by identifying the specific variables that should be measured within
each of the conceptual construct domains specified in Holton (1996). By
doing so, it moved the model one step closer to empirical testing and
validation.

Figure 3 presents the revised model with complete construct definitions,
where possible. The obvious challenge that remains is to validate the model.
Although complex, I believe that the model can one day be validated, at least
in part if not in its entirety. This article and the work that has transpired in the
past 8 years have made it now feasible to consider a validation study where it
was impossible to even consider 8 years ago. Most likely, the initial valida-
tion study will include just the learning and individual performance out-
come domains, as much work remains to be done in the organizational per-
formance domain. Nonetheless, tremendous progress has been made and
validation looks more feasible than ever before.

A validation effort would likely need to be conducted in steps. First, vali-
dation studies could be conducted on a single level of outcomes. For exam-
ple, researchers would measure all the intervening variables affecting learn-
ing as an outcome and test their effect on learning. Or, the intervening

50 Advances in Developing Human Resources Febr uar y 2005

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

http://adh.sagepub.com/

variables affecting transfer to job performance would be measured and
tested for their effect on transfer.

Second, a multilevel analysis that combines the two approaches dis-
cussed above could be conducted. Specifically, the intervening variables
affecting learning and those affecting performance would be measured and
tested for their ability to predict transfer to job performance.

Ultimately, the goal would be to test the entire model. That will require
considerable work to define and operationalize constructs influencing orga-
nizational results. Montesino’s (2002) study is a promising example of the
kind of research that can be conducted to further define intervening vari-
ables at the organizational level and how they influence both individual
transfer and organizational results.

It is clear that these analyses are complex and will require the use of
advanced statistical analysis techniques and partners willing to engage in
extensive data collection efforts. The validation studies will require struc-
tural equation modeling analysis to study the causal relationships hypothe-
sized among the construct. Because the model is a multilevel one, hierarchi-
cal linear modeling would likely be employed to analyze the cross-level
relationships.

Holton / HOLTON’S EVALUATION MODEL 51

Outcomes
Environment
Motivation
Secondary
Influences
Ability

Learning Individual Performance

Content Validity
Transfer Design
Personal Capacity for Transfer
Opportunity to Use

Motivation to Improve Work Through Learning (MTIWL)
Motivation to Learn
Motivation to Transfer
Transfer Effort –> Performance
Performance –> Outcomes

Feedback
Peer Support
Supervisor Support
Openness to Change
Personal Outcomes -Positive
Personal Outcomes -Negative
Supervisor Sanctions

Performance Self -Efficacy
Learner Readiness

Organizational Performance

Personality Traits
Conscientiousness
Neuroticism
Openness to Experience
Goal Orientation
Locus of Control

Job Attitudes
Organizational Commitment
Job Involvement

Perceptions
Utility Perceptions
Behavioral Intentions

Linkage to Organization
Goals

External Events

Expected
Utility/Return on

Investment
Learning Design
Ability

FIGURE 3: Revised HRD Evaluation and Research Model

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

http://adh.sagepub.com/

In conclusion, the validation of this model will clearly be ambitious and
demanding research. Nonetheless, if the network of causal influences on
important HRD intervention outcomes is to be understood, then this type of
research is necessary and should be undertaken.

References

Alliger, G. M., & Janak, E. A. (1989). Kirkpatrick’s levels of training criteria: Thirty
years later. Personnel Psychology, 42, 331-342.

Alliger, G. M., Tannenbaum, S. I., Bennett, W., Trave, H., & Shotland, A. (1997). A
meta-analysis of the relations among training criteria. Personnel Psychology, 50(2),
341-358.

Barrick, M. R., & Mount, M. K. (1991). The big five personality dimensions and job per-
formance: A meta-analysis. Personnel Psychology, 44, 1-26.

Bates, R. A., Holton, E. F., III, Seyler, D. L., & Carvalho, M. B. (2000). The role of inter-
personal factors in the application of computer-based training in an industrial setting.
Human Resource Development International, 3, 19-42.

Bates, R. A., & Khasawneh, S. (2004). Organizational learning culture, transfer climate,
and perceived innovation in Jordan. In T. M. Egan & M. L. Morris (Eds.), Proceedings
of the 2004 Academy of Human Resource Development Conference (514-521). Bowl-
ing Green, OH: Academy of Human Resource Development.

Bell, B. S., & Kozlowski, S.W.J. (2002). Goal orientation and ability: Interactive effects
on self-efficacy, performance and knowledge. Journal of Applied Psychology, 87,
497-505.

Bookter, A. I. (1999). Convergent and divergent validity of the learning transfer ques-
tionnaire. Unpublished doctoral dissertation, Louisiana State University, Baton
Rouge.

Chen, G., Gully, S. M., Whiteman, J. A., & Kilcullen, R. N. (2000). Examination of rela-
tionships among trait-like individual differences, state-like individual differences,
and learning performance. Journal of Applied Psychology, 85, 835-847.

Chen, H.-C. (2003). Cross-cultural construct validation of learning transfer system
inventory in Taiwan. Unpublished doctoral dissertation, Louisiana State University,
Baton Rouge.

Colquitt, J. A., LePine, J. A., & Noe, R. A. (2000). Toward an integrative theory of train-
ing motivation: A meta-analytic path analysis of 20 years of research. Journal of
Applied Psychology, 85, 678-707.

Colquitt, J. A., & Simmering, M. J. (1998). Conscientiousness, goal orientation and moti-
vation to learn during the learning process: A longitudinal study. Journal of Applied
Psychology, 83, 654-665.

Costa, P. T., & McCrae, R. R. (1992). Revised NEO personality inventory and the
five-factor inventory professional manual. Odessa, FL: Psychological Assessment
Resources.

52 Advances in Developing Human Resources Febr uar y 2005

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

http://adh.sagepub.com/

Costa, P. T., & McCrae, R. R. (1995). Solid ground in the wetlands of personality: A reply
to Block. Psychological Bulletin, 117, 216-220.

Digman, J. M. (1990). Personality structure: Emergence of the five factor model. Annual
Review of Psychology, 41, 417-440.

Ferguson, E., & Patterson, F. (1998). The five factor model of personality: Openness a
distinct but related construct. Personality and Individual Differences, 24, 789-796.

Ford, J. K., Smith, E. M., Weisbein, D. A., & Gully, S. M. (1998). Relationship of goal ori-
entation, metacognitive activity, and practice strategies with learning outcomes and
transfer. Journal of Applied Psychology, 83, 218-233.

Goldberg, L. R. (1990). An alternative “description of personality”: The big-five factor
structure. Journal of Personality and Social Psychology, 59, 1216-1229.

Herold, D. M., Davis, W., Fedor, D. B., & Parsons, C. K. (2002). Dispositional influences
on transfer of learning in multistage training programs. Personnel Psychology, 55,
851-869.

Holton, E. F., III. (1996). The flawed four-level evaluation model. Human Resource
Development Quarterly, 7(1), 5-21.

Holton, E. F., III, Bates, R. A., & Ruona, W.E.A. (2000). Development of a generalized
learning transfer system inventory. Human Resource Development Quarterly, 11(4),
333-360.

Khasawneh, S. A. (2004). Construct validation of an Arabic version of the learning
transfer system inventory for use in Jordan. Unpublished doctoral dissertation, Loui-
siana State University, Baton Rouge.

Kirkpatrick, D. L. (1998). Evaluating training programs: The four levels (2nd ed.). San
Francisco: Berrett-Koehler.

Kirkpatrick, D. (1995). Techniques for evaluating training programs. Journal of the
American Society for Training and Development, 13, 3-9.

LePine, J. A., Colquitt, J. A., & Erez, M. (2000). Adaptability to changing task contexts:
Effects of general cognitive ability, conscientiousness, and openness to experience.
Personnel Psychology, 53, 563-593.

Montesino, M. U. (2002). Strategic alignment of training, transfer-enhancing behaviors,
and training usage: A post training study. Human Resource Development Quarterly,
13, 89-108.

Morgan, R. B., & Casper, W. J. (2000). Examining the factor structure of participant reac-
tions to training: A multidimensional approach. Human Resource Development
Quarterly, 11(3), 301-317.

Naquin, S. S., & Holton, E. F., III. (2002). The effects of personality, affectivity, and work
commitment on motivation to improve work through learning. Human Resource
Development Quarterly, 13, 357-376.

Naquin, S. S., & Holton, E. F., III. (2003). Motivation to improve work through learning
in human resource development. Human Resource Development International, 6,
355-370.

Noe, R. A., & Schmitt, N. (1986). The influence of trainee attitudes on training effective-
ness: Test of a model. Personnel Psychology, 39, 497-523.

Holton / HOLTON’S EVALUATION MODEL 53

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

http://adh.sagepub.com/

Ruona, W.E.A., Leimbach, M., Holton, E. F., III, & Bates, R. A. (2002). The relationship
between learner utility reactions and predicted learning transfer among trainees.
International Journal of Training and Development, 6(4), 218-228.

Seyler, D. L., Holton, E. F., III, Bates, R. A., Burnett, M. F., & Carvalho, M. B. (1998).
Factors affecting motivation to transfer training. International Journal of Training
and Development, 2, 2-16.

Swanson, R. A., & Holton, E. F., III. (1999). Results: How to assess performance, learn-
ing and perceptions in organizations. San Francisco: Berrett-Koehler.

Tan, J. A., Hall, R. J., & Boyce, C. (2003). The role of employee reactions in predicting
training effectiveness. Human Resource Development Quarterly, 14, 397-411.

Yamnill, S. (2001). Factors affecting transfer of training in Thailand. Unpublished doc-
toral dissertation, University of Minnesota, Twin Cities.

Elwood F. Holton, III, is the Jones S. Davis Distinguished Professor of Human
Resource, Leadership and Organization Development at Louisiana State University.
He has authored 19 books and more than 200 articles in HRD. He was the founding
editor of Human Resource Development Review as well as the president of the
Academy of Human Resource Development. He received the AHRD Outstanding
Scholar Award in 2001.

Holton, E. F., III. (2005). Holton’s evaluation model: New evidence and construct
elaborations. Advances in Developing Human Resources, 7(1), 37-54.

54 Advances in Developing Human Resources Febr uar y 2005

at AHRD on January 26, 2016adh.sagepub.comDownloaded from

http://adh.sagepub.com/