MGMT20135 Critical thinking and managerial decision-making.

Task: need to select an article from the list of articles and write a critical review report .(check the attachments and select one among the list of articles)

   The review must be 1500 words (+/-10%) in length. 

Cover page: The essay must include a cover page that contains your name, student number, resident campus, assessment title, and lecturer. 

Minimum number of peer-reviewed journal articles to pass:  10  

Both on-campus students and Distance students must write a review article and submit online via Moodle by the due date. 

Format: Text should be word-processed, with appropriate layout and use of headings/sub-headings. Times New Roman, 12 size font and line spacing (1.5). 

Referencing: The list of references should form the last page or two, at the end of the assessment. Referencing should be in a consistent APA style. 

Please save/upload your file in either a Word format ( or x) or pdf. format. 

Task Description: The purpose of this task is for you to demonstrate your ability to critically analyze, synthesize and evaluate information and present that information in a written format. A list of articles will be available on the unit’s Moodle site for you to choose from. You will then write a critical article review based around your responses to the following questions. 

Critical Analysis 

o What is the background of the research? (Background) 

o What is the research problem? What do we know about the research problem? What do we not know? (Research objective and literature review) 

o What are the core research questions the article addresses? Why are they important to address? (Research questions) 

o How was the research conducted? (Methodology) 

o What where the major findings? (Findings) 

o What are the significance of the findings for future research or management practice? (Significance/implications) 

Critical evaluation 

o In your view, are there any limitations with this research? 

o What are some of the assumptions that the authors have made in this research? Do you agree with them? 

o What is your overall evaluation of the article’s effectiveness and credibility? 

Please do not simply list the questions and your response to them. The questions are to be used simply as a means for you to critically think about your chosen article. Your review needs to contain answers to these questions, but they need to be in essay format, that is, containing introduction section, body paragraphs and a conclusion section.  

Assessment Due Date   

Week 4 Monday (30 Jul 2018) 5:00 pm AEST

You must submit your essay with a cover page on Moodle by the due date

Return Date to Students   

Week 6 Monday (20 Aug 2018)

The marks will be released with feedback by this date.

Weighting                               30%                                                              Assessment Criteria                

Introduction *3 marks    Clearly outlines the purpose, scope, structure of the essay   

Body—critical analysis *10 marks    The following list of questions are effectively addressed: o What is the background of the research? o What is the research problem? What do we know about the research problem? What do we not know? o What are the core research questions the article addresses? Why are they important to address? o How was the research conducted? o What where the major findings? o What are the significance of the findings for future research or management practice? Demonstrate ability to critically analyze, synthesize and present information in a written format 

Body—critical evaluation *10 marks    The following list of questions are effectively addressed: o In your view, are there any limitations with this research? o What are some of the assumptions that the authors have made in this research? Do you agree with them? o What is your overall evaluation of the article’s effectiveness and credibility? Demonstrate ability to critically evaluate information and present argument in a convincing manner   Conclusion *3 marks    All aspects drawn together in a brief, concise summary. Consistent with findings, no new material introduced but highlights implications or a comment on the future of the issue.   

Presentation *2 marks    High quality of expression, grammar, spelling, punctuation and proofreading. Format and layout in professional manner ( i.e. 1.5 spacing, 12-size font, Times New Roman). Assessment cover page included, within the word count.   

Referencing *2 marks    Use of APA referencing system in a consistent and correct manner in the essay itself. Inclusion of an accurate reference list on a separate page listing only the sources that actually have been used. The reference list is arranged in alphabetical order according to the authors’ last names.   

Further breakdown of each criterion (From Fail to High Distinction) is available on Moodle.

Learning and Individual Differences

2

1 (2011)

6

7–77

Contents lists available at ScienceDirect

Learning and Individual Differences

journal homepage: www.elsevier.com/locate/lindif

Epistemic beliefs and critical thinking of Chinese students

Ngai-Man Chan a, Irene T. Ho a,⁎, Kelly Y.L. Ku b

a Department of Psychology, The University of Hong Kong, Pokfulam Road, Hong Kong
b Department of Education Studies, Hong Kong Baptist University, Kowloon Tong, Hong Kong

⁎ Corresponding author. Tel.: +8

5

2 2859 2296; fax:
E-mail addresses: h015

3

802@graduate.hku.hk (N.-M

kellyku@hkbu.edu.hk (K.Y.L. Ku).

10

4

1-6080/$ – see front matter © 2010 Elsevier Inc. Al
doi:10.1016/j.lindif.2010.11.001

a b s t r a c t

a r t i c l e i n f o

Article history:
Received 18 October 2009
Received in revised form 23 August 2010
Accepted 3 November 2010

Keywords:

Epistemic belief

Critical thinking
Chinese student
Epistemology
Individual difference

Two studies were carried out to examine the relationship between epistemic beliefs and critical thinking. In
the first study, 138 Chinese undergraduates completed the adapted Epistemic Beliefs Inventory (EBI) and the
Halpern Critical Thinking Assessment Using Everyday Situations. Their cognitive ability and thinking
dispositions were also measured. Results showed that other than cognitive ability, the belief that knowledge is
certain was most related to thinking performance. In the second study, 111 undergraduates read a passage on
a controversial issue and were asked to indicate their position, explain the reasoning behind, and rate their
confidence in judgment. They were then presented relevant counterarguments and asked to respond to them.
Participants were also administered the adapted EBI. There was clear evidence that those believing
knowledge to be certain exhibited poorer two-sided thinking and a stronger tendency to devaluate or ignore
counterarguments. Implications for the enhancement of critical thinking are discussed.

+852 2858 3518.
. Chan), itfho@hku.hk (I.T. Ho),

l rights reserved.

© 2010 Elsevier Inc. All rights reserved.

1. Introduction

With the advent of the Information Age and increased complexity
of society, critical thinking has been receiving attention as an ability
that enables one to deal with myriad information, to make reasonable
judgments, and to participate in society as competent citizens
(Halpern, 2003; Paul, 1984). Schools have been given the important
responsibility of helping students learn to think deeply and rationally
so that they can evaluate different arguments and views effectively
(Brown, 1997). How critical thinking may be fostered is a question
that educators and psychologists have been asking.

Among the various factors affecting critical thinking performance,
personal epistemology, defined as one’s views about the nature of
knowledge and knowing, has been proposed to be relevant (Brabeck,
1983; Gallagher, 1998; Jones, Merritt, & Palmer, 1999; Kuhn, 1999;
Kurfiss, 1988; Mines, King, Hood, & Wood, 1990; Schraw, 2001). A
better understanding of how personal epistemology affects thinking
performance would help improve related instructional practices.

1.1. Epistemic beliefs

Personal epistemology refers to “beliefs about the definition of
knowledge, how knowledge is constructed, how knowledge is
evaluated, where knowledge resides, and how knowing occurs”
(Hofer, 2002, p. 4). As these beliefs influence how people handle and
use knowledge, they play an important role in students’ thinking,

learning, and motivation (Hofer, 2001; Hofer & Printrich, 1997; Muis,
2007; Schommer, 1994; Schraw & Sinatra, 2004).

Researchers have differed in their approaches to the study of
personal epistemology. One approach, initiated by Schommer (1990,
1994), has conceptualized personal epistemology as a system of more or
less independent beliefs. A five-factor model of epistemological beliefs
has been proposed, consisting of beliefs about simple knowledge
(knowledge as an accumulation of discrete and unambiguous facts),
certain knowledge (knowledge is absolute and unchanging), omniscient
authority (authorities have best access to knowledge), quick learning
(learning occurs quickly or not at all), and innate ability (the ability to
acquire knowledge is fixed).

The measurement of these beliefs started with Schommer’s
Epistemological Beliefs Questionnaire (Schommer, 1990), which
utilizes a Likert-type scale to assess the strength of various epistemic
beliefs. This instrument spurred efforts by other researchers to
develop similar measures for the study of epistemic beliefs (e.g.,
Bendixen, Schraw, & Dunkle, 1998; Jehng, Johnson, & Anderson,
1993; Wood & Kardash, 2002). One example was the development of
the 32-item Epistemic Belief Inventory (EBI) by Bendixen and
colleagues (Bendixen et al., 1998). The EBI was modeled on Schommer’s
questionnaire but was found to have better predictive validity and test

–

retest reliability (Schraw, Bendixen, & Dunkle, 2002).

Quantitative instruments of this kind allow us to differentiate
individuals on the level of sophistication or maturity in their handling
of knowledge during thinking and learning. Based on the five-
dimension conceptualization, sophisticated beliefs embrace “evolving
knowledge, multiple approaches to the justification of knowledge,
integration of knowledge,…, gradual learning, and ever growing
ability to learn” (Schommer-Aikins, 2002, p.113). For example, if a

http://dx.doi.org/10.1016/j.lindif.2010.11.001

mailto:h0153802@graduate.hku.hk

mailto:itfho@hku.hk

mailto:kellyku@hkbu.edu.hk

http://dx.doi.org/10.1016/j.lindif.2010.11.001

http://www.sciencedirect.com/science/journal/10416080

68 N.-M. Chan et al. / Learning and Individual Differences 21 (2011) 67–77

student tends to believe that knowledge is absolute and knowable
with certainty, we refer to him or her as having a naïve belief in
knowledge being certain. In contrast, a student who appreciates the
tentative nature of knowledge would be considered having a more
sophisticated or mature belief in knowledge being uncertain. More
sophisticated beliefs in the uncertain, relative, and complex nature of
knowledge are associated with the ability and readiness to take in
new ideas or change old ones, therefore supporting flexible thinking.

Closely associated with the development of epistemic belief
measures have been efforts to delineate more clearly the multi-
dimensional nature of these beliefs. While some researchers have
found evidence for Schommer’s original five-dimensional theory of
epistemic beliefs (Schraw et al., 2002), others have been unable to
replicate the proposed dimensions at the item level. For example,
Schommer herself did not generate a clear omniscient authority
dimension in her own study (1990), while Nussbaum and Bendixen
(2003) found that omniscient authority and quick learning failed to
emerge as distinct factors.

Related studies in non-Western contexts have also suggested
cross-cultural differences in the structure, level, and effects of
epistemic beliefs (Khine, 2008). For example, researchers studying
Chinese populations often highlight the emphasis on respect for
authority and conformity in the Confucian culture in their prediction
or interpretation of related results (Chan & Elliot, 2004a; Lin, 2001;
Qian & Pan, 2002). However, findings have been inconsistent and so
far no clear patterns of the structure and effects of Chinese people’s
epistemic beliefs have emerged. As these studies are still very limited
in number and have generated diverse findings, more investigations
in relation to a wider range of learning and thinking processes are
needed (for reviews of related issues see Chan & Elliot, 2004a; Duell &
Schommer-Aikins, 2001; Hofer, 2008). The accumulation of empirical
findings would help generate clearer patterns of universal and
culture-specific aspects of epistemic beliefs and their effects.

1.2. Epistemic beliefs and critical thinking

Critical thinking is “reasonable reflective thinking that is focused
on deciding what to believe and do” (Ennis, 1991, p.7). It involves an
active evaluation of one’s thinking process, which would provide
useful and accurate feedbacks to improve the thinking (Halpern, 1998,
1999). Although authors vary in the breadth of their definition of
critical thinking, overall there is agreement about judgment, reasoning,
meta-cognition, and reflective thinking being the major components
(see review by Fischer & Spiker, 2000). For example, Halpern (1998)
proposed a taxonomy of five major critical thinking skills comprising
verbal reasoning, argument analysis, hypothesis testing, assessing
likelihood, and decision-making/problem-solving. These are skills
frequently used in a wide range of everyday evaluative thinking.

In situations requiring critical thinking, the adequacy of individuals’
epistemological theories will in some way determine how they
approach the task and what meaning they make of the given
information. As illustrated above, sophisticated beliefs underlie flexible
thinking, which is essential in the process of thinking critically. Indeed,
many researchers share the notion that advanced epistemological
awareness is a prerequisite for one to engage in critical thinking
(Gallagher, 1998; Jones et al., 1999; Kurfiss, 1988), with related
empirical evidence available. For example, Brabeck (1983) found a
significant positive correlation between epistemological understanding
and critical thinking, whereas Mines and colleagues (Mines et al., 1990)
noted that “students who reason using the assumptions of the higher
stages of reflective judgment demonstrate better critical thinking skills
than do those who use lower stage assumptions” (p. 545). Other
researchers also explicitly assert that personal epistemology has the
potential to inform educators about ways to enhance students’ thinking
(Kuhn, 1999; Schraw, 2001).

1.3. Effects of epistemic beliefs on argumentation

While researchers have argued for a relationship between
epistemic beliefs and critical thinking performance, these beliefs
have been particularly observed to produce impacts on argumenta-
tion skills and argument analysis, which constitute a specific and
salient aspect of critical thinking ability (see for example, Halpern’s
1998 taxonomy of critical thinking skills). Indeed, of the many studies
linking epistemic beliefs to thinking performance, most examine this
relationship in contexts that involve the evaluation of two-sided
arguments or multiple solutions in relation to controversial or
ambiguous issues.

Oneimportant observationfrom these studies is that naïve epistemic
beliefs would induce simplistic approaches to complex issues, often
manifested in an inadequate consideration of opposing views. For
example, it was shown that individuals adopting naïve epistemic beliefs
tended to draw oversimplified or absolute conclusions after reading
passages highlighting two opposing views on controversial issues in the
psychology and nutrition domains (Schommer, 1990). They would also
write conclusions that failed to reflect the inconclusive nature of mixed
evidence when considering controversial issues such as HIV–AIDS
relationship, abortion, and communication styles of men and women
(Kardash & Scholes, 1996; Schommer-Aikins & Hutter, 2002).

Furthermore, it has been observed that people with naïve beliefs
would treat information that does not support their existing beliefs in a
biased manner. For example, they would use fewer cognitive strategies
when reading dual-position text, reflecting a tendency to ignore certain
information, and distort contradictory information in order to make it
consistent with preexisting beliefs during free recall of text information
(Kardash & Howell, 2000). In another study, naïve epistemic beliefs
were found to be associated with the tendency to decline scientific
explanation when learning about the potentially controversial topic of
human evolution (Sinatra, Southerland, McConaughy, & Demastes,
2003). Also the conceptual change of people holding naïve epistemic
beliefs was found to be much more limited after reading refutation text
that directly confronted their misconceptions about a physics topic
(Qian & Alvermann, 1995).

Generally, people with naïve epistemic beliefs have been found to
avoid argumentative situations (Nussbaum & Bendixen, 2003) and
perform poorly in ill-defined tasks with multiple, non-guaranteed
solutions (Schraw, Dunkle, & Bendixen, 1995). In other words, the
level of sophistication of an individual’s epistemic beliefs is particu-
larly associated with performance on open-ended and ill-structured
thinking tasks which involve comparing alternative viewpoints, the
strength of evidence, and the adequacy of arguments in order to arrive
at a judgment. It is in these situations that an absolutist tendency
appears to produce the greatest impact on thinking and information
processing.

1.4. Epistemic beliefs and other critical thinking factors

Researchers have largely conceptualized critical thinking in terms
of two major factors: cognitive skills or ability and thinking
dispositions. There is general consensus that although some people
may have excellent cognitive abilities which are necessary for
thinking critically, they may not actively or effectively engage in
such thinking if they lack the disposition to do so (Ennis, 1991;
Facione, Facione, & Giancarlo, 2000; Halpern, 1998; Perkins, Jay, &
Tishman, 1993). If epistemic beliefs have also been found to be related
to critical thinking performance, it would be necessary to investigate
how these beliefs relate to ability and dispositions and to examine
their relative effects on thinking performance.

Cognitive ability and epistemic beliefs are likely to have reciprocal
influences on each other. On the one hand, people with limited
cognitive ability may find it difficult to process information from
multiple perspectives, as in the comparison of arguments, therefore

69N.-M. Chan et al. / Learning and Individual Differences 21 (2011) 67–77

discouraging them from endorsing more sophisticated epistemic
beliefs that acknowledge the tentative and complex nature of
knowledge. On the other hand, persons with an absolutist inclination
or naïve personal epistemology, with their belief in knowledge being
facts that are either right or wrong, may not see the need to engage in
complex intellectual activities such as argument (Hofer & Printrich,
1997; Kuhn, 1999, 2001; Weinstock & Cronin, 2003), thus limiting
their practice and development of higher level reasoning skills.

Thinking dispositions that have been suggested to facilitate critical
thinking include having an interest in thinking (e.g., need for
cognition and inquisitiveness), a mature approach to making judg-
ments (e.g., openness and truth-seeking), and being careful and
systematic in thinking (e.g., conscientiousness and systematicity)
(American Philosophical Association, 1990; Ennis, 1991; Facione et al.,
2000; Perkins et al., 1993). More advanced development in episte-
mological understanding not only fosters thinking skills but also
enhances the value for critical thinking. As individuals with more
sophisticated epistemic beliefs see the need to evaluate assertions
based on criteria of argument and evidence, they would be more
willing to engage in effortful thinking and be open-minded about new
evidence or alternatives. In return, these thinking inclinations will
also fortify sophisticated epistemic beliefs, as the habit of engaging in
complex reasoning would reinforce the experience of knowledge
being complex and tentative.

Thus, cognitive abilities, thinking dispositions and epistemic
beliefs probably develop in synchrony. Given that these individual
difference variables are likely to be correlated, the extent to which
personal epistemology provides unique contribution to critical
thinking on top of the contribution of cognitive abilities and thinking
dispositions requires investigation.

1.5. The present research

Current evidence as reviewed above has pointed to the phenom-
enon that individuals with more sophisticated epistemic beliefs tend
to be better critical thinkers. However, more research is needed to
gain a better picture of how naïve beliefs about knowledge hinder
good thinking. In particular, there has not been research with a focus
on the simultaneous effects of epistemic beliefs, cognitive ability, and
thinking dispositions on critical thinking. Furthermore, extending
related research beyond Western contexts would throw light on the
extent to which the hypothesized relationships are universally
applicable.

The present research, consisting of two studies, aimed at extending
the empirical basis for the epistemic belief-critical thinking relation-
ship by examining it in groups of Chinese students in Hong Kong. The
first study was an examination of general relationships that constitute
basic understanding, including: (i) What is the factor structure of
epistemic beliefs among Chinese students and which aspect(s) of
these beliefs produce most salient effects on critical thinking?
(ii) What is the pattern of relationships between epistemic beliefs
and cognitive ability as well as thinking dispositions? (iii) What are
the unique and relative contributions of beliefs, ability, and disposi-
tions to critical thinking performance? Study 2 was a more focused
investigation of how epistemic beliefs affect argumentation, which is
an essential aspect of critical thinking.

2. Study 1

There were two focuses of examination in Study 1. Firstly, we
explored the factor structure of epistemic beliefs as well as the
relationships between these beliefs and other critical thinking factors,
namely cognitive ability and thinking dispositions. Secondly, we
examined whether epistemic beliefs had a unique contribution to
students’ critical thinking performance beyond their effects through
associations with cognitive ability and thinking dispositions. Based on

previous findings from Western research, the following hypotheses
were tested:

(i) More sophisticated epistemic beliefs are associated with better
cognitive ability.

(ii) More sophisticated epistemic beliefs are associated with higher
levels of desirable thinking dispositions.

(iii) More sophisticated epistemic beliefs are associated with better
critical thinking performance.

(iv) Epistemic beliefs have unique effects on critical thinking
performance over and above those contributed by cognitive
ability and thinking dispositions.

2.1. Method

2.1.1. Participants
Participants were138 Chinese undergraduate students (40 males

and 98 females) at a comprehensive university in Hong Kong,
recruited through campus-wide advertising. Ages ranged from 19 to
32 (M=20.72, SD=1.60) and their average GPA was 2.92 (SD=.51).
Participants consisted of students in their first (47.8%), second (37.0%)
and final (15.2%) years of study with major subjects in social science,
arts, science, business, and others. Participation was on a voluntary
and anonymous basis.

2.1.2. Measures
A pilot study was carried out to examine the applicability of the

Epistemic Beliefs Inventory (EBI) (Bendixen et al., 1998) for use with
the present participants, in particular its reliability and factorial
structure. The Inventory consists of 32 items falling under five factors,
namely Certain Knowledge, Simple Knowledge, Innate Ability, Quick
Learning and Omniscient Authority. The Inventory was found to have
better test–retest reliability (stability coefficients ranging from .62 to
.81) and predictive validity than earlier instruments measuring
similar constructs (Schraw et al., 2002).

In the pilot study, the Chinese version of the 32 items was derived
through a translation–back-translation procedure. Then 366 univer-
sity and high school students in Hong Kong were asked to rate the
items on five-point Likert scales (“Strongly agree” to “Strongly
disagree”). Results from exploratory factor analysis showed that
consistent with the findings from an earlier study by Nussbaum and
Bendixen (2003), the Quick Learning and Omniscient Authority
dimensions did not emerge as distinct factors. Instead, three
interpretable and distinct factors with satisfactory internal consisten-
cies (Cronbach’s alphas above .65) resulted, namely Innate Ability,
Simple Knowledge and Certain Knowledge. Based on the results from
this pilot study, three scales, each with four items (a total of 12 items)
selected from the EBI, were used in the present study. Examples of
items are: “Smart people are born that way” (Innate Ability), “If a
person tries too hard to understand a problem, they will most likely
end up confused” (Simple Knowledge), and “Absolute moral truth
does not exist” (Certain Knowledge; reverse coded). Higher scores on
these scales indicate more naïve beliefs.

The Halpern Critical Thinking Assessment Using Everyday Situations
(HCTAES; Halpern, 2007) was used to measure participants’ critical
thinking performance. The HCTAES consists of 25 everyday scenarios,
each followed by open-ended and multiple-choice questions. The test
provides an overall index of critical thinking performance covering all
major critical thinking skills: verbal reasoning, argument analysis,
hypothesis testing, assessing likelihood and decision-making/problem-
solving. The following is a hypothetical item similar in length and
presentation to those in the HCTAES:

Results from a recent study indicated that female adolescents who
perceive themselves as being unpopular among peers are more likely to
be overweight. The researchers suggested that running social skills

Table 1
Factor loadings of epistemic belief items and factor correlations in Study 1 (N=138).

Item Innate
ability

Certain
knowledge

Simple
knowledge

1. Smart people are born that way .85
2. People can’t do too much about how
smart they are

.62

3. Some people will never be smart no
matter how hard they work

.51

4. How well you do in school depends
on how smart you are

.40

5. Absolute moral truth does not exista .73
6. Truth means different things to

different peoplea

.59

7. Sometimes there are no right answers
to life’s problemsa

.59

8. What is true today will be true tomorrow .40
9. Too many theories just complicate things .84
10. The best ideas are often the most simple .64
11. If a person tries too hard to understand a

problem, they will most likely end up
confused

.53

12. Things are simpler than most professors
would have you believe

.46

Factor correlations
Innate ability 1
Certain knowledge −.09 1
Simple knowledge .08 .05 1

a Item reversed coded.

70 N.-M. Chan et al. / Learning and Individual Differences 21 (2011) 67–77

training programs for female adolescents who are overweight would help
solve their weight problems.

Open-ended question:
Based on this information, would you support this idea as a way of

solving overweight problems for female adolescents? Type “yes” or “no”
and explain why or why not.

Forced-choice question:
Based on this information, which of the following is the best answer?
(Four choices provided).
Sample choice: Social skills training will probably reduce overweight

problems among female adolescents because the researchers found that
girls who perceive themselves as being unpopular among peers are more
likely to be overweight.

The HCTAES have been used with different samples of American
students with convergent and divergent validity evidence reported,
including high positive correlations (in the .50 to .70 range) with a
number of achievement or ability tests (e.g., SAT-Verbal, SAT-Math
and GRE-Analytic), and moderate correlations in the .30 to .40 range
with measures of need for cognition and conscientiousness. Reliability
coefficients (Cronbach’s alphas) of over .80 have been reported for
different groups of students (Halpern, 2007). Preliminary studies on
the appropriateness of the test for use with Chinese students were
conducted (Hau et al., 2006). Analyses of the results from these
studies helped to establish the validity and reliability of the Chinese
version of the test, with Cronbach’s alphas for different samples of
secondary and university students ranging from .65 to .74. All items
were scored according to the standards provided in the original test
(Halpern, 2007) and a total score was derived by summing all the item
scores.

The Verbal Comprehension Index (VCI) of the Chinese version of the
Wechsler Adult Intelligence Scale—Third Edition (WAIS-III, Chinese
Edition, 2002) was used to measure participants’ cognitive–verbal
ability. The VCI is derived from scores on the Vocabulary, Similarities
and Information subtests of the WAIS-III. Previous studies have shown
verbal abilities to be the strongest cognitive predictor of critical
thinking performance (Clifford, Boufal, & Kurtz, 2004).

Three dispositional variables that are considered important
personal traits that contribute to critical thinking and are commonly
examined in related studies (e.g., Clifford et al., 2004; Halpern, 2007;
Sá, West, & Stanovich, 1999; Spector, Schneider, Vance, & Hezlett,
2000; Toplak & Stanovich, 2002; West, Toplak, & Stanovich, 2008)
were measured. These included need for cognition, open-mindedness,
and conscientiousness. All items of the scales were rated on five-point
Likert scales, ranging from “Strongly agree” to “Strongly disagree”. The
Chinese version of the Need for Cognition Scale-Short Form (NCS-SF;
Cacioppo, Petty, & Kao, 1984; Hui, 2003) measures the tendency to
engage in and enjoy effortful thinking. It consists of 18 items and a
sample item is “The notion of thinking abstractly is appealing to me”.
This scale was found to be valid and reliable for use with Chinese
students in previous studies (Kao, 1994; Ku & Ho, 2010; Kuang, Shi,
Cai, & Wang, 2005). The Openness to Experience Subscale and the
Conscientiousness Subscale of the NEO Five Factor Inventory (NEO-FFI,
Costa & McCrae, 1992) were used as measures of open-mindedness
and conscientiousness. They each consist of 12 items. Sample items
include “I believe controversial topics will only confuse students”
(reverse coded) for openness and “I work towards my goal
systematically” for conscientiousness. The Chinese version of the
NEO-FFI was found to have good reliability for use with Chinese
students in previous studies (e.g., Ku & Ho, 2010; Zhang, 2002).

2.1.3. Procedure
Participants were tested in two sessions. In the first session, they

completed the HCTAES. Within a week, participants came back for the
second session, where they completed the various self-report scales
and were individually administered the WAIS-III subtests by school
psychology trainees who had received training on administering the

tests. Background information on participants’ gender, age, GPA, and
socio-economic status (SES – family income, fathers’ education,
mothers’ education) was also gathered.

2.2. Results and discussion

2.2.1. Preliminary analyses
The factorial structure of the adapted three-dimensional EBI was

further examined by confirmatory factor analysis with the data from this
group of students. The fit of the data to the a priori three-factor model
was reasonably good as indicated by the various goodness of fit indexes;
χ2(51)=71.28, RMSEA=.051, NNFI=.92, CFI=.94, with all standard-
ized factor loadings greater than .40 (see Table 1). Internal consistency
as measured by Cronbach’s alphas for the Innate Ability Scale, Certain
Knowledge Scale and Simple Knowledge Scale were .67, .66, and .71
respectively. The inter-factor correlations were all less than .10,
suggesting that the three were quite distinct belief dimensions.

Internal consistency coefficients for the other scales as measured
by Cronbach’s alphas were .69 for the HCTAES, .87 for the NCS-SF, .67
for the Openness to Experience Scale, and .76 for the Conscientious-
ness Scale. Means and standard deviations of the variables are given in
Table 2. Zero-order correlations among the variables are given in
Table 3.

2.2.2. Relations between epistemic beliefs and other critical
thinking factors

To examine how students’ epistemic beliefs were related to
demographic variables (gender, age, SES composite derived from
family income, father’s education and mother’s education), cognitive
ability and dispositions, a set of regression analyses with each of the
beliefs as criterion variables were conducted (Table 4). Results
showed that being younger in age and a lower need for cognition
would significantly predict a stronger belief in ability being innate.
Being male, a lower verbal ability and being less open to experience
predicted a higher tendency to believe in knowledge being certain.
None of the variables examined were related to the belief in
knowledge being simple. Among the three epistemic beliefs, Certain

Table 2
Means, standard deviations, and ranges of scores (Studies 1 and 2a).

Variable Mean SD Potential
range

Actual
range

Study 1
Cognitive ability (VCI) 105.47 7.91 Mean=100 61–131
Openness (NEO-FFI-O) 42.20 5.39 12–60 29–56
Conscientiousness (NEO-FFI-C) 41.71 5.69 12–60 25–53
Need for cognition (NCS-SF) 58.56 9.52 18–90 37–81
Epistemic beliefs (EBI)

Innate ability 12.77 3.04 4–20 4–19
Certain knowledge 8.41 2.82 4–20 4–17
Simple knowledge 12.83 2.80 4–20 4–19

Critical thinking (HCTAES) 115.79 13.06 0–194 73–147

Study 2
Epistemic beliefs (EBI)

Innate ability 11.57 2.72 4–20 5–19
Certain knowledge 8.77 2.53 4–20 4–18
Simple knowledge 13.37 2.83 4–20 7–19

Two-sided argument skills

4.25 1.54 1–6 1–6
Confidence 2.52 .82 1–5 1–4
Convincingness of counterargument 3.19 .84 1–5 1–4
Tackling of counterargument 1.92 .78 1–3 1–3

Note. VCI=Verbal Comprehension Index of WAIS-III. NEO-FFI-O=NEO-FFI, Openness
to Experience Subscale. NEO-FFI-C=NEO-FFI, Conscientiousness Subscale. NCS-
SF=Need For Cognition Scale—Short Form. EBI=Epistemic Beliefs Inventory.
HCTAES=Halpern Critical Thinking Assessment Using Everyday Situations.

a NStudy 1=138, NStudy 2=111.

Table 4
Relations between epistemic beliefs and other critical thinking factors (Study 1,
N=138).

Criterion variable (standardized beta)

Epistemic belief

Predictor Innate
ability

Certain
knowledge
Simple
knowledge

Demographics
Gender −.02 −.30*** −.10
Age −.25** .10 −

.12

SES .08 .01 .07

Cognitive ability
GPA −.17 .13 .03
VCI .09 −.18* .03

Thinking dispositions
Openness (NEO-FFI-O) .21 −.38*** −.20
Conscientiousness (NEO-FFI-C) .10 −.13 −.10
Need for cognition (NCS-SF) −.38** .07 −.14

Multiple R2 .15 .27 .13

Note. VCI=Verbal Comprehension Index of WAIS-III. NEO-FFI-O=NEO-FFI, Openness
to Experience Subscale. NEO-FFI-C=NEO-FFI, Conscientiousness Subscale. NCS-
SF=Need For Cognition Scale—Short Form.
*pb .05, **pb .01, ***pb .001.

71N.-M. Chan et al. / Learning and Individual Differences 21 (2011) 67–77

Knowledge appeared to have the strongest relationship with
thinking-related constructs. Cognitive ability and dispositions, to-
gether with gender, accounted for 27% of the variance in this belief.

2.2.3. Epistemic beliefs as predictors of critical thinking performance
Stepwise regressions were conducted to examine the effects of

various variables on critical thinking. Demographic variables were
entered in the first step, followed by cognitive ability indexes,
dispositional factors, and finally epistemic beliefs. The results
(Table 5) indicated that neither demographics nor dispositional factors
significantly predicted thinking performance. Rather, cognitive–verbal
ability (VCI) was the most significant predictor. Moreover, the naïve
beliefs that ability is innate, and more importantly that knowledge is
certain, predicted poorer critical thinking, after controlling for their

Table 3
Zero-order correlations among variables (Study 1, N=138).

1 2 3 4 5 6 7 8

1. Cognitive ability
(VCI)

–

2. Openness
(NEO-FFI-O)

.17* –

3. Conscientiousness
(NEO-FFI-C)

−.02 −.10 –

4. Need for cognition
(NCS-SF)

.15 .55** .14 –

5. EBI-Innate ability .10 −.02 −.05 −.20* –
6. EBI-Certain

knowledge
−.21* −.34** .03 −.15 −.09 –

7. EBI-Simple
knowledge

−.02 −.27** −.11 −.26** .07 .04 –

8. Critical thinking
(HCTAES)

.46** .12 −.07 .17 −.13 −.29** −.12 –

Note. VCI=Verbal Comprehension Index of WAIS-III. NEO-FFI-O=NEO-FFI, Openness to
Experience Subscale. NEO-FFI-C=NEO-FFI, Conscientiousness Subscale. NCS-SF=Need
For Cognition Scale—Short Form. EBI=Epistemic Beliefs Inventory. HCTAES=Halpern
Critical Thinking Assessment Using Everyday Situations.
*pb.05. **pb .01.

possible joint effects with cognitive or dispositional factors. Together
cognitive ability and epistemic beliefs contributed to 37% of the variance
in critical thinking performance, of which beliefs contributed 11%
unique variance.

To sum up, results from Study 1 provided evidence for three major
dimensions of epistemic beliefs, with omniscient authority and quick
learning not emerging as distinct factors. This converged with findings
from an earlier study by Nussbaum and Bendixen (2003). There was
also support for all the stated hypotheses to various extents, with the
relationships between epistemic beliefs and cognitive ability, thinking
dispositions, and thinking performance being specified as follows:

(i) The more sophisticated belief in knowledge being tentative
(uncertain) was associated with better cognitive ability.

(ii) The more sophisticated belief in ability being malleable (non-
innate) was associated with a higher level of need for cognition,
whereas the more sophisticated belief in knowledge being
uncertain was associated with greater open-mindedness.

Table 5
Stepwise regression on critical thinking performance (Study 1, N=138).

Predictor Critical thinking performance (standardized
beta)

Step 1 Step 2 Step 3 Step 4

Demographics
Gender −.07 −.01 −.03 −.14
Age −.15 −.10 −.07 −.10
SES −.03 .00 .01 .03

Cognitive ability
GPA .08 .11 .12
VCI .46*** .45*** .41***

Thinking dispositions
Openness (NEO-FFI-O) −.05 −.16
Conscientiousness (NEO-FFI-C) −.09 −.12
Need for cognition (NCS-SF) .14 .07

Epistemic beliefs (EBI)
Innate ability −.19*
Certain knowledge −.32***
Simple knowledge −.14

Multiple R2 .03 .24 .26 .37
Increase in R2 .21 .02 .11

Note. VCI=Verbal Comprehension Index of WAIS-III. NEO-FFI-O=NEO-FFI, Openness to
Experience Subscale. NEO-FFI-C=NEO-FFI, Conscientiousness Subscale. NCS-SF=Need
For Cognition Scale—Short Form. EBI=Epistemic Beliefs Inventory.
*pb .05, ***pb .001.

72 N.-M. Chan et al. / Learning and Individual Differences 21 (2011) 67–77

(iii) The more sophisticated belief in knowledge being uncertain,
and to a lesser extent also the belief in ability being malleable,
was associated with better critical thinking performance.

(iv) There were unique effects of epistemic beliefs on thinking
performance, other than the possible effects exerted through
their association with cognitive and dispositional factors.

3. Study 2

Results from Study 1 provided evidence that personal epistemology
is related to cognitive ability and thinking disposition, and it produces
unique effects on thinking performance. Study 2 aimed at further
examining how, or the mechanisms through which epistemic beliefs
exert influence on critical thinking performance, focusing on their
effects on the argumentation process. More specifically, we investigated
students’ reasoning process in face of a controversial issue where
reasonable arguments and counterarguments were both available. This
kind of tasks is typical of those calling for critical thinking. A more fine-
grained examination of where individuals with naïve epistemic beliefs
differ from their counterparts holding more sophisticated beliefs in this
thinking process would throw further light on the exact nature of the
epistemic belief-critical thinking relationship.

As mentioned in the literature review, findings from previous studies
have suggested a close-minded thinking style to be an important link
between naïve epistemic beliefs and poor argumentation, epitomized in
the tendency to draw simplistic or absolute conclusions based on
inadequate considerations of opposing views or mixed evidence as well
as the inclination to treat information that challenges preexisting beliefs
in a biased manner. Results of our Study 1 also suggested that beliefs in
ability and knowledge being fixed were associated with a reluctance to
engage in thinking and be open-minded, at the same time predicting
poorer performance on critical thinking tasks.

When facing a controversial issue, which usually involves
reasonable arguments supporting both sides of the debate, the
drawing of sound conclusions would depend on an adequate
consideration of arguments on both sides, i.e., two-sided thinking. A
close-minded or absolutist thinking style will likely hamper this
process by fixating only on one viewpoint and focusing on evidence
that support their adopted position while ignoring counterarguments
that do not. This explains why individuals who hold naïve epistemic
beliefs tend to perform poorly in critical thinking, as critical thinkers
should be able to effectively consider alternative viewpoints (Halpern,
1998; Lewis & Smith, 1993). Based on this rationale the following
hypotheses regarding the relationship between epistemic beliefs and
argumentation performance were proposed and tested:

(i) Participants with naïve epistemic beliefs are weaker in two-sided
thinking, i.e., they are more likely to focus on the arguments
supporting their position while overlooking counterarguments
refuting their position.

(ii) Participants with naïve epistemic beliefs exhibit a greater
tendency to devaluate counterarguments, i.e., naïve beliefs will
bias the participants’ thinking so that they will fail to objectively
evaluate reasonable counterarguments, thus finding them less
convincing than they are.

(iii) Participants with naïve epistemic beliefs will be less efficient in
tackling counterarguments, i.e., the lower tendency to consider
and understand counterarguments would weaken one’s ability
to consider arguments in a reasonable manner.

3.1. Method

3.1.1. Participants
Participants were 111 Chinese undergraduates (35 males and

76 females) in Hong Kong. Recruitment was done through making an
announcement in Introductory Psychology courses. The age range was

from 18 to 23 (M=19.85, SD=1.13) and their average GPA was
2.81 (SD=.50). Participants consisted of students in their first
(77.5%), second (16.2%) and final (6.3%) years of study with various
major subjects. Participation was on a voluntary and anonymous
basis.

3.1.2. Thinking task
A controversial issue was developed as the stimulus for the

assessment of participants’ argumentation performance. In order to
choose an issue that was appropriate for the study, a number of
controversial issues were subject to several rounds of discussions and
pilot-tested for their appropriateness. There were two criteria for
judging appropriateness. Firstly, the number of people voting for and
against the issue should be comparable. Secondly, there should be
equally convincing arguments for and against the issue. Based on the
results of the pilot study, an issue about whether long-term drug
patent is acceptable was selected. Drug patents allow pharmaceutical
companies to be the sole producer and freely set the price for a newly
developed drug for 20 years. This arrangement safeguards the profit
of the company and encourages the development of new drugs. Yet it
also results in high drug prices which are not affordable by many who
need the medicine. Therefore there has been a debate on whether this
patent arrangement should be kept.

In the pilot study, a group of graduate students (N=15) read a
summary of the background information and the opposing views in
the debate. Then they read two passages arguing for or against the
patent arrangement. Participants were asked to rate the reasonable-
ness of both passages on a 10-point Likert scale ranging from −5
(“Totally unreasonable”) to 5 (“Totally reasonable”).Then they
indicated their position for the issue, i.e., whether the drug patent
arrangement should be kept. It was found that the two argumentative
passages were both rated as reasonable by all the participants. Both
the arguments for (M=3.47, SD=.92) and against (M=3.33,
SD=1.11) the arrangement were rated as reasonable, with non-
significant difference between the mean ratings for reasonableness of
the two argumentative passages; t(14)=.358, n.s. For the final
position, there were 53.33% participants indicating support for and
46.67% indicating opposition of the specified patent arrangement.
Thus the controversial nature of this issue was ensured, with the two
passages arguing for or against it being comparable in reasonableness.

3.1.3. Measures
The adapted Epistemic Beliefs Inventory (EBI) used in Study 1 was

employed in the present study to measure participants’ epistemic
beliefs. It consists of three dimensions (innate ability, certain
knowledge, simple knowledge) and a total of 12 items (see Study 1
for details).

Participants’ performance in argumentation was measured
through essay tasks. They were assessed on (i) their ability to provide
two-sided arguments in response to the controversial issue and
(ii) their ability to tackle counterarguments presented to them.

Participants first read the passage that summarized the debate on
the issue. They were then asked to indicate their position regarding
the patent arrangement (for or against) and write a short essay
providing the reasoning behind their position (“Please explain your
thinking and the relevant arguments behind your position”). They
were also asked to rate their confidence in the correctness of their
conclusion on a 5-point Likert scale (from 1 “Totally unconfident” to 5
“Totally confident).

After that, participants were given an argumentative passage
which opposed their stated position and were asked to rate the
convincingness of the counterarguments on a 5-point Likert scale (from
1 “Totally unconvincing” to 5 “Very convincing”). Finally they were
asked to re-consider their position and explain their thinking in
writing (“After reading the article, would you change your stand on
the issue? Explain why you would or would not change your stand”).

73N.-M. Chan et al. / Learning and Individual Differences 21 (2011) 67–77

A scoring rubric was developed to assess participants’ two-sided
argument skills in their initial essays explaining the reasoning behind
their position, with reference to the scheme developed by Ferretti,
Macarthur, and Dowdy (2000) in their study of persuasive essays.
Responses were rated from 1 to 6 according to their ability to
recognize and address arguments for both sides of the debate and
provide clear reasoning for their final conclusion (see Appendix A).
Higher ratings indicated better two-sided argument skills. A random
sample of one-third of the scripts was coded by another independent
rater and the inter-rater reliability was found to be very good (Cohen’s
kappa=.80). As for the tackling of counterarguments, another scoring
rubric with a 3-point scale for rating responses was derived (see
Appendix A). Higher scores indicated more competent tackling of
counterarguments. Again an independent rater coded a random
sample of one-third of the scripts and inter-rater reliability (Cohen’s
kappa) was .71.

3.1.4. Procedure
All participants completed the procedures in one session, being

administered the adapted EBI first and asked to provide background
information on gender, age, and GPA. Then they were given the
thinking task as described above.

3.2. Results and discussion

3.2.1. Preliminary analyses
We started the analyses by again examining the factorial structure

and classification of the 12 items for the adapted EBI, using
confirmatory factor analysis. As indicated by the various goodness of
fit indexes, the empirical data fitted the a priori 3-factor structure very
well, χ2(51)=68.00, RMSEA=.055, NNFI=.90, CFI=.92, with all
standardized factor loadings greater than .40. Internal consistency as
measured by Cronbach’s alphas for Innate Ability, Certain Knowledge
and Simple Knowledge was .68, .64, and .67 respectively. Again none
of the inter-factor correlations were significant at the .05 level, with rs
ranging from −.04 to .22. Thus further evidence was provided for the
reliability of the inventory measuring three distinct dimensions of
epistemic beliefs. Means and standard deviations of the variables are
given in Table 2.

3.2.2. Relationship between epistemic beliefs and argumentation
Regression analyses were conducted to examine the extent to which

epistemic beliefs predicted specific features in the argumentation
process, namely (i) two-sided argument skills, (ii) confidence about

Table 6
Predicting argumentation performance with epistemic beliefs (Study 2, N=111).

Criterion variable (standardized beta)

Argumentation performance

Predictor Two-sided argument skills Con

Background
Gender −.02 .0
Age −.04 .1
GPA .04 −.1

Epistemic beliefs (EBI)
Innate Ability −.03 .0
Certain Knowledge −.35*** .0
Simple Knowledge −.22* −.0

Argumentation performance
Two-sided argument skills .0
Confidence

Addressing counterargument

Convincingness rating
Tackling of counterargument

Multiple R2 .17 .0

Note. EBI=Epistemic Beliefs Inventory.
*pb.05, ***pb .001.

own conclusion, (iii) attitude towards reasonable counterarguments
(convincingness rating), and (iv) tackling of counterarguments. Gender,
age and GPA as an index of general ability were also included in the
regression equations so as to control for their possible effects on various
aspects of argumentation.

Results (Table 6) showed that the certainty dimension of
epistemological beliefs, and to a lesser extent also the simplicity
dimension, significantly predicted weaker two-sided argumentation.
Therefore Hypothesis 1 for the present study was supported. The
belief in knowledge being certain also predicted a greater tendency to
devaluate reasonable counterarguments, thus providing support for
Hypothesis 2. However, these beliefs did not seem to be related
to how well students tackled counterarguments, apparently not
supporting Hypothesis 3. Instead, it was found that students who
were more confident about their viewpoint were slightly more likely
to tackle counterarguments with good reasoning.

In order to further explore possible differences in the treatment of
counterarguments between people holding naïve beliefs and those
holding sophisticated beliefs, more detailed analyses were conducted
with selected groups of participants. Specifically, participants were
divided into three groups based on their scores on the Certain
Knowledge Scale, and comparisons were made between the group
with highest scores (top one-third constituting the naïve group,
N=44) and the group with lowest scores (bottom one-third making
up the sophisticated group, N=35). Group membership was based on
scores on the Certain Knowledge dimension because this dimension
was found to be most salient in influencing critical thinking in
previous and the present studies. The distribution of the types of
responses given by these two extreme groups in their treatment of
counterarguments is given in Table 7.

A chi-square test showed a significant uneven distribution of
the types of responses given by these two groups; Pearson chi-square
(2)=6.33, pb.05; Fisher’s Exact Test=6.32, pb .05. On closer
examination, there was a greater percentage of people among the
naïve group (43%) ignoring or failing to address counterarguments
than among the sophisticated group (17%). In contrast, students
holding more sophisticated beliefs were more likely to address
counterarguments, whether adequately or inadequately (a total of
83%), than their naïve counterparts (a total of 57%). Worthy to note is
that the proportion of people being able to address counterarguments
adequately was actually quite comparable between the naïve group
and the sophisticated group (23% versus 29%, respectively). This
pattern of results suggested that the naïve group and the sophisti-
cated group differed more in attention to counterarguments rather

Addressing counterargument

fidence Convincingness rating

Tackling counterargument

2 −.14 −.08
1 .04 .10
1 .02 −.10

1 .01 .13
1 −.24* .06
9 .11 .10

9 .09 .13
−.02 .19*

.12

4 .11 .15

Table 7
Rebuttals generated by the naïve and sophisticated groups (frequencies and percentages).

Tackling of counterargument Belief in certain knowledge

Naïve group Sophisticated group

1. Failing to address counterargument 19 (43%) 6 (17%)
2. Underdeveloped rebuttal 15 (34%) 19 (54%)
3. Sensible rebuttal 10 (23%) 10 (29%)
Subtotal 44 (100%) 35 (100%)

74 N.-M. Chan et al. / Learning and Individual Differences 21 (2011) 67–77

than in their ability to tackle them. In view of this, the regression
analysis result of a non-significant relationship between epistemic
beliefs and performance in the tackling of counterarguments could
perhaps be attributed to the lack of differentiation between attention
and ability factors in the measure of this performance. This analysis of
the distribution of different treatments of counterarguments and the
corresponding chi-square test results did provide some support for
Hypothesis 3.

4. Summary and concluding discussion

The present studies aimed at examining the relationship between
epistemic beliefs and critical thinking among Chinese students, and
the results largely conformed to related theorizations in the Western
context. In particular, the unique contribution of these beliefs on
thinking was highlighted. Results from Study 1 showed that these
beliefs had unique effects on critical thinking performance, over and
above those produced by cognitive ability. Specifically, the belief in
knowledge being fixed and absolute (certain knowledge), to a lesser
extent also the belief that our ability to process knowledge is fixed
(innate ability), significantly predicted poorer performance on
everyday evaluative thinking. It was also found that this belief in
the certainty of knowledge was associated with closed-mindedness
and poorer cognitive ability, whereas the belief in fixed ability was
related to a lower need for cognition. It appears that while poor
cognitive skills contribute to poor thinking performance, the lack of
recognition of the tentative nature of knowledge, together with the
lack of a conviction that effort and practice would improve our ability
to think, further hampers critical thinking performance.

In Study 2, it was further demonstrated that when thinking about
controversial issues, beliefs about knowledge being certain and simple
would deter one from considering issues from multiple perspectives
and make him/her dismiss counterarguments more readily. This
tendency to engage in one-sided thinking would produce detrimental
effects on argumentation performance. In conclusion, the belief in
knowledge being certain has been highlighted in two studies as the
most significant aspect of personal epistemology affecting everyday
evaluative thinking and the more specific process of argumentation.

These results throw light on the nature of epistemic effects on
critical thinking. Naïve beliefs about knowledge predispose one to a
fixated and single-perspective approach to issues, thus affecting
the integration and generation of knowledge during thinking. In
particular, the lack of multiple perspectives hampers the ability to
generate and judge alternatives in decision-making and problem-
solving, which characterizes good critical thinking (Ennis, 1991;
Halpern, 1998). Moreover, the tendency to devalue reasonable
counterarguments reflects a poor ability to reason independently of
one’s prior belief bias, which has also been emphasized to be an
important critical thinking skill (Sá et al., 1999; Stanovich & West,
1997).

The present results also provide support for the postulation of
reciprocal relationships among epistemic beliefs, cognitive ability, and
thinking dispositions. While sophistication in epistemology was found
to be related to cognitive ability and desirable thinking dispositions in
Study 1, the fine-grained analysis of argumentation processes in Study 2

pointed further to the nature of these relationships. People holding
absolutist beliefs (knowledge being certain and simple) tend to reach a
position on controversial issues quickly rather than spending effort on
the consideration and objective evaluation of opposing arguments. This
tendency would reinforce a close-minded thinking habit. Moreover,
although individuals with naïve beliefs may have the ability to pro-
vide sensible rebuttals for counterarguments, the inclination of not
addressing counterargument would prevent them from exercising and
refining this ability. Handling problems in these ways is not adaptive as
it makes the revision of one’s reasoning difficult. As a result of these
cyclical processes, epistemic beliefs, thinking ability and thinking
dispositions would develop in synchrony.

Overall, the present findings highlight an important link between
naïve epistemic beliefs and poor critical thinking, namely the
tendency to draw quick conclusions without adequate consideration
of alternative views. This observation is consistent with findings about
the cognitive characteristics of people holding naïve epistemic beliefs
in previous research, in which other thinking performances were
examined with different methodologies (Kardash & Howell, 2000;
Kardash & Scholes, 1996; Qian & Alvermann, 1995; Schommer, 1990;
Schommer-Aikins & Hutter, 2002; Sinatra et al., 2003). It also
corroborates other findings about the relationship between cognitive
dispositions and personal epistemology: an open-minded approach is
positively correlated with the belief that decision-making takes time
and effort (Kardash & Sinatra, 2003), and a mature and systematic
thinking style is associated with the belief in knowledge being
tentative and evolving (Valanides & Angeli, 2008).

While the present findings indicate that the postulated relation-
ships between epistemic beliefs and critical thinking largely apply in
both Western and Chinese contexts, the fact that beliefs in knowledge
being certain are of particular importance warrants attention. In the
Chinese–Confucian culture, high values are placed on respect for
authority, tradition and social order, predisposing members of the
culture to cognitive conservatism (Ho, 1994). For an individual to put
aside preconceptions and consider issues from alternative perspec-
tives, a strong belief in the tentative nature of knowledge is needed. In
other words, in a society where following established views is
encouraged and arguments are discouraged, the importance of a belief
in the tentative nature of knowledge as a precursor to good critical
thinking is highlighted, as the present results show.

4.1. Implications for practice

Programs designed for the enhancement of critical thinking
typically focus on the teaching of related thinking skills. The present
results suggest that fostering sophistication in students’ epistemic
beliefs is also conducive to critical thinking development. Specifically,
students should be encouraged to recognize the tentative and
complex nature of knowledge as well as the fact that practice and
effort could help improve our ability to think. This has special
significance for the nurturing of critical thinking in Chinese and other
societies sharing the Confucian cultural heritage. As mentioned above,
the tendency of cognitive conservatism in this culture may pose
significant hindrance to people’s engagement in analytical thinking.
Without the simultaneous cultivation of sophisticated epistemic
beliefs, the effects of critical thinking skills training may be limited.

Western research has shown that epistemological development
could be fostered by the experience of epistemic doubt, where one
questions the existence of absolute knowledge (Bendixen, 2002). An
effective way to induce epistemic doubt is to have students reflect and
judge on ill-structured problems so that they become more aware of
alternative modes of thought (King & Kitchener, 2002; Kuhn, 2001).
For example, Kuhn, Shaw, and Felton (1997) found that students’
metacognitive awareness of the co-existence of multiple viewpoints
could be enhanced through dyadic discussions of controversial issues.

75N.-M. Chan et al. / Learning and Individual Differences 21 (2011) 67–77

Having students debate over controversial issues would also
allow opportunities for practicing the generation and evaluation of
counterarguments. As our results show, the tendency to devalue
counterarguments that may be reasonable is a major hindrance for
students holding naïve beliefs to think critically. At the end of the
debate, students should draw their own conclusion based on the
supporting arguments as well as their explanations for the counter-
arguments being inferior to the supporting arguments. Such activities
would not only foster the development of multiple perspectives but
also promote the essential critical thinking skill of assessing subjective
viewpoints based on objective criteria of good reasoning.

An additional implication for practice is that Chinese educators,
who may succumb more to the knowledge-transmission conception
of teaching and learning (Chai, Khine, & Teo, 2006; Chan & Elliot,
2004b), should be more aware of instructional practices that foster
naïve beliefs in the certainty of knowledge. For example, questioning
that focuses too much on soliciting factual information would
reinforce the belief that knowledge is absolute (Kurfiss, 1988). In
the same vein, “textbooks that present subject matter as non-
problematic reinforce dualistic thinking” (Kurfiss, 1988, p. 64). The
common use of assessment methods with restrictive answers might
also cue students to search for model answers when they face
problems and regard those answers as absolute. For example,
multiple-choice questions implicitly reinforce the concept that only
one correct answer exists. Where appropriate, using open-ended
methods to assess students’ learning progress might be more desirable
for the promotion of epistemological development. Generally speaking,
constructivist learning approaches characterized by the active con-
struction of knowledge through critical analysis would have more
positive impacts on students’ epistemological and cognitive develop-
ment (Marra, Palmer, & Litzinger, 2000; Pascarella & Terenzini, 1991;
Stephenson & Hunt, 1977; Valanides & Angeli, 2005).

4.2. Limitations and further research

The present studies provide clear evidence that epistemological
beliefs are related to critical thinking performance, and the results
largely converge with findings from related studies in Western
contexts. However, more research is needed to examine how
epistemic beliefs interact with cognitive ability and thinking disposi-
tions to produce effects on critical thinking. Clarifying these relation-
ships would help build a more precise and comprehensive theory of
critical thinking, at the same time generating insights into the
effective enhancement of critical thinking from multiple perspectives.
In particular, conducting similar research in different cultural contexts
would enable comparisons to be made and advance our understand-
ing of cultural factors in thinking.

In view of the limitations of quantitative studies using Likert scale
measures, which assume that the variables could be captured on a
continuous scale, it would be desirable to also conduct qualitative
research (e.g., in-depth interviews) in an attempt to capture more
clearly the complexity of changes in thinking and beliefs under
different conditions. Discourse analysis using natural data on how
students treat knowledge during thinking would also be fruitful.
Employing mixed-method designs in research would enable broader
perspectives in investigation and cross-validation of findings.

As the ultimate value of understanding thinking-related processes
is to inform about how thinking performance may be enhanced, more
direct investigations on effective epistemological intervention would
be desirable. Although implications for practice could be drawn based
on available evidence such as those generated in the present studies,
the conclusions drawn are largely only inferential in nature based on
correlational data. Direct empirical evidence generated from exper-
imental or longitudinal studies would help derive more definite
assertions about effective intervention. For example, tracing how
students’ thinking and beliefs develop during intervention or over

time would provide valuable information. Issues that are worth
exploring include the direction of causality of effects, the malleability
of epistemic beliefs, and other possible intervening factors.

To conclude, the present studies have provided evidence for the
significant and unique role of personal epistemology in critical
thinking among Chinese students. More specifically, naïve beliefs
about knowledge limit one’s ability to consider alternatives from
multiple perspectives and to objectively evaluate arguments that are
not consistent with prior conceptions. These beliefs likely develop in
synchrony with cognitive ability and desirable thinking dispositions,
jointly producing effects on critical thinking. Future research should
attempt to clarify the relative roles and relationships of these
thinking-related factors and directly investigate intervention issues
so as to specify more clearly the kind of epistemological intervention
that would be effective for the enhancement of critical thinking.

Acknowledgement

Thepreparation of this manuscript was partially supported bya grant
from the Research Grants Council of Hong Kong (Project no. 443809).

Appendix A

Scoring rubrics for argumentation performance in Study 2.

Score

Description
Two-sided argument skills
1

No valid argument

The arguments used to support the position are either unrelated
to the topic or invalid.

2
Partially developed one-sided argument

Although some valid arguments are stated to support the position,
the exposition also contains other supporting arguments which are
either unrelated to the topic or invalid.

3
Well-developed one-sided argument

The exposition states supporting arguments which are valid and
elaborated clearly.

4
Two-sided argument: only recognizing counterargument

The exposition states supporting arguments which are elaborated
clearly. In addition, it also mentions counterarguments but fails to give
any refutations against them.

5
Two-sided argument: minimally addressing counterargument

The exposition states supporting arguments which are elaborated
clearly. Counterarguments are stated clearly but not addressed
adequately with sensible rebuttal or alternative solutions.

6
Two-sided argument: well-developed argument addressing counterargument

The exposition states supporting arguments which are elaborated
clearly. Moreover, counterarguments are addressed either with sensible
rebuttal or alternative solutions.

Tackling counterargument
1

Failing to address counterargument

Participant largely only restates own position and argument, without
addressing the counterargument.

2
Underdeveloped rebuttal

Participant attempts to address the counterargument but the reasoning
or rebuttal is not well developed or is of limited plausibility.

3
Sensible rebuttal

Participant provides a sensible evaluation of the counterargument and
provides a reasonable conclusion.

References

American Philosophical Association. (1990). Critical thinking: A statement of expert
consensus for purposes of educational assessment and instruction, (“The Delphi
Report”). ERIC Document Reproduction No. ED 315 423.

Bendixen, L. D. (2002). A process model of epistemic belief change. In B. K. Hofer, & P. R.
Pintrich (Eds.), Personal epistemological: The psychology of beliefs about knowledge
and knowing (pp. 191−208). Mahwah, NJ: Erlbaum.

Bendixen, L. D., Schraw, G., & Dunkle, M. E. (1998). Epistemic beliefs and moral
reasoning. The Journal of Psychology, 13, 187−200.

Brabeck, M. (1983). Critical thinking skills and reflective judgment development:
Redefining the aims of higher education. Journal of Applied Developmental Psychology,
4, 23−34.

76 N.-M. Chan et al. / Learning and Individual Differences 21 (2011) 67–77

Brown, A. L. (1997). Transforming schools into communities of thinking and learning
about serious matters. The American Psychologist, 52(4), 399−413.

Cacioppo, J. T., Petty, R. E., & Kao, C. F. (1984). The efficient assessment of need for
cognition. Journal of Personality Assessment, 48(3), 306−307.

Chai, C. S., Khine, M. S., & Teo, T. (2006). Epistemological beliefs on teaching and
learning: A survey among pre-service teachers in Singapore. Educational Media
International, 43, 285−298.

Chan, K. W., & Elliot, R. G. (2004a). Epistemological beliefs across cultures: Critique and
analysis of beliefs structure studies. Educational Psychology, 24, 123−142.

Chan, K. W., & Elliot, R. G. (2004b). Relational analysis of personal epistemology and
conceptions about teaching and learning. Teaching and Teacher Education, 20,
817−831.

Clifford, J. S., Boufal, M. M., & Kurtz, J. E. (2004). Personality traits and critical thinking
skills in college students. Assessment, 11(2), 169−176.

Costa, P. T., & McCrae, R. R. (1992). Revised NEO Personality Inventory and NEO Five Factor
Inventory: Professional Manual. Odessa, FL: Psychological Assessment Resources.

Duell, O. K., & Schommer-Aikins, M. (2001). Measures of people’s beliefs about
knowledge and learning. Educational Psychology Review, 13(4), 419−449.

Ennis, R. H. (1991). Critical thinking: A streamlined conception. Teaching Philosophy, 14(1),
5−25.

Facione, P. A., Facione, N. C., & Giancarlo, C. A. (2000). The disposition toward critical
thinking: Its character, measurement, and relationship to critical thinking skill.
Informal Logic, 20(1), 61−84.

Ferretti, R. P., Macarthur, C., & Dowdy, N. (2000). The effects of an elaborated goal on
persuasive writing of students with learning disabilities and their normally
achieving peers. Journal of Educational Psychology, 92(4), 694−702.

Fischer, S. C., & Spiker, V. A. (2000). A framework for critical thinking research and training.
Report prepared for the US Army Research Institute. Santa Barbara, California: Anacapa
Sciences Inc.

Gallagher, S. A. (1998). The road to critical thinking: The Perry scheme and meaningful
differentiation. NASSP Bulletin, 82(595), 12−20.

Halpern, D. F. (1998). Teaching critical thinking for transfer across domains:
Dispositions, skills, structure training, and metacognitive monitoring. The American
Psychologist, 53(4), 449−455.

Halpern, D. F. (1999). Teaching for critical thinking: Helping college students develop
the skills and dispositions of a critical thinker. New Directions for Teaching and
Learning, 80, 69−74.

Halpern, D. F. (2003). Thought and knowledge: An introduction to critical thinking (4th ed.).
Mahwah, NJ: Erlbaum.

Halpern, D. F. (2007). Halpern Critical Thinking Assessment: Background and scoring
standards. Unpublished manuscript. Claremont, CA: Claremont McKenna College.

Hau, K. T., Halpern, D. F., Marin-Burkhard, L., Ho, I. T., Ku, K. Y. L., Chan, N. M., & Lun, V. M. C.
(2006). Assessment of Chinese and U.S. students’ critical thinking: Open-ended versus
forced-choice items. Unpublished manuscript. Hong Kong, China: The Chinese
University of Hong Kong.

Ho, D. Y. F. (1994). Filial piety, authoritarian moralism, and cognitive conservatism in
Chinese societies. Genetic, Social, and General Psychology Monographs, 120,
347−365.

Hofer, B. K. (2001). Personal epistemology research: Implications for learning and
teaching. Educational Psychology Review, 13(4), 353−383.

Hofer, B. K. (2002). Personal epistemology as a psychological and educational construct:
An introduction. In B. K. Hofer, & P. R. Pintrich (Eds.), Personal epistemological: The
psychology of beliefs about knowledge and knowing (pp. 3−14). Mahwah, NJ: Erlbaum.

Hofer, B. K. (2008). Personal epistemology and culture. In M. S. (Ed.), Knowing,
knowledge, and beliefs: Epistemological studies across diverse cultures (pp. 3−22).
Dordrecht, the Netherlands: Springer.

Hofer, B. K., & Printrich, P. R. (1997). The development of epistemological theories:
Beliefs about knowledge and knowing and their relation to learning. Review of
Educational Research, 67(1), 88−140.

Hui, N. H. H. (2003). Nomological network of need for cognition: A study with Chinese
college students. Unpublished manuscript, Department of Psychology, Chinese
University of Hong Kong, Hong Kong.

Jehng, J. J., Johnson, S. D., & Anderson, R. C. (1993). Schooling and students’ epistemological
beliefs about learning. Contemporary Educational Psychology, 18, 23−35.

Jones, P. C., Merritt, J. Q., & Palmer, C. (1999). Critical thinking and interdisciplinarity in
environmental higher education: The case for epistemological and values
awareness. Journal of Geography in Higher Education, 23(3), 349−357.

Kao, C. (1994). The concept and measurement of need for cognition. Chinese Journal of
Psychology, 36, 1−20.

Kardash, C. M., & Howell, K. L. (2000). Effects of epistemological beliefs and topic-
specific beliefs on undergraduates’ cognitive and strategic processing of dual-
positional text. Journal of Educational Psychology, 92(3), 524−535.

Kardash, C. M., & Scholes, R. J. (1996). Effects of preexisting beliefs, epistemological
beliefs, and need for cognition on interpretation of controversial issues. Journal of
Educational Psychology, 88(2), 260−271.

Kardash, C. M., & Sinatra, G. (2003). Epistemological beliefs and dispositions: Are we
measuring the same construct? Poster paper presented at the Annual General
Meeting of the American Educational Research Association, Chicago, IL.

Khine, M. S. (Ed.). (2008). Knowing, knowledge, and beliefs: Epistemological studies across
diverse cultures. Dordrecht, the Netherlands: Springer.

King, P. M., & Kitchener, K. S. (2002). The reflective judgment model: Twenty years of
research on epistemic cognition. In B. K. Hofer, & P. R. Pintrich (Eds.), Personal
epistemological: The psychology of beliefs about knowledge and knowing (pp. 37−61).
Mahwah, NJ: Erlbaum.

Ku, K. Y. L., & Ho, I. T. (2010). Dispositional factors predicting Chinese students’ critical
thinking performance. Personality and Individual Differences, 48(1), 54−58.

Kuang, Y., Shi, J. Q., Cai, Y. Q., & Wang, L. (2005). The Chinese version of need for
cognition scale. Chinese Mental Health Journal, 19(1), 57−60.

Kuhn, D. (1999). Metacognitive development. In L. Balter, & C. S. Tamis-LeMonda (Eds.),
Child psychology: A handbook of contemporary issues (pp. 259−286). Philadelphia,
PA: Psychology Press.

Kuhn, D. (2001). How do people know? Psychological Science, 12(1), 1−8.
Kuhn, D., Shaw, V., & Felton, M. (1997). Effects of dyadic interaction on argumentative

reasoning. Cognition and Instruction, 15(3), 287−315.
Kurfiss, J. G. (1988). Critical thinking: Theory, research, practice, and possibilities.

Washington, DC: Association for the Study of Higher Education.
Lewis, A., & Smith, D. (1993). Defining higher order thinking. Theory into Practice, 32(3),

131−137.
Lin, C. H. (2001). Epistemological development and academic performance among

elementary students. Journal of National Taichung Teachers College, 15, 191−206.
Marra, R. M., Palmer, B., & Litzinger, T. (2000). The effects of a first-year engineering

design course on student intellectual development as measured by the Perry
scheme. Journal of Engineering Education, 89(1), 39−45.

Mines, R. A., King, P. M., Hood, A. B., & Wood, P. K. (1990). Stages of intellectual
development and associated critical thinking skills in college students. Journal of
College Student Development, 31(6), 538−547.

Muis, K. R. (2007). The role of epistemic beliefs in self-regulated learning. Educational
Psychologist, 42, 173−190.

Nussbaum, E. M., & Bendixen, L. D. (2003). Approaching and avoiding arguments: The
role of epistemological beliefs, need for cognition, and extraverted personality
traits. Contemporary Educational Psychology, 28(4), 573−595.

Pascarella, E. T., & Terenzini, P. T. (1991). How college affects students. San Francisco, CA:
Jossey-Bass.

Paul, R. W. (1984). Critical thinking: Fundamental to education for a free society.
Educational Leadership, 42(1), 4−14.

Perkins, D. N., Jay, E., & Tishman, S. (1993). Beyond abilities: A dispositional theory of
thinking. Merrill-Palmer Quarterly, 39(1), 1−21.

Qian, G., & Alvermann, D. (1995). Role of epistemological beliefs and learned
helplessness in secondary school students’ learning science concepts from text.
Journal of Educational Psychology, 87(2), 282−292.

Qian, G., & Pan, J. (2002). A comparison of epistemological beliefs and learning from
science text between American and Chinese high school students. In B. K. Hofer, &
P. R. Pintrich (Eds.), Personal epistemology: The psychology of beliefs about knowledge
and knowing (pp. 365−385). Mahwah, NJ: Erlbaum.

Sá, W. C., West, R. F., & Stanovich, K. E. (1999). The domain specificity and generality of
belief bias: Searching for a generalizable critical thinking skill. Journal of Educational
Psychology, 91(3), 497−510.

Schommer, M. (1990). Effects of beliefs about the nature of knowledge on
comprehension. Journal of Educational Psychology, 82(3), 498−504.

Schommer, M. (1994). An emerging conceptualization of epistemological beliefs and
their role in learning. In R. Garner, & P. A. Alexander (Eds.), Beliefs about text and
instruction with text (pp. 25−40). Hillsdale, NJ: Erlbaum Associates.

Schommer-Aikins, M. (2002). An evolving theoretical framework for an epistemological
belief system. In B. K. Hofer, & P. R. Pintrich (Eds.), Personal epistemology: The
psychology of beliefs about knowledge and knowing (pp. 103−118). Mahwah, NJ:
Erlbaum.

Schommer-Aikins, M., & Hutter, R. (2002). Epistemological beliefs and thinking about
everyday controversial issues. The Journal of Psychology, 136(1), 5−20.

Schraw, G. (2001). Current themes and future directions in epistemological research: A
commentary. Educational Psychology Review, 13(4), 451−464.

Schraw, G., Bendixen, L. D., & Dunkle, M. E. (2002). Development and validation of the
Epistemic Belief Inventory (EBI). In B. K. Hofer, & P. R. Pintrich (Eds.), Personal
epistemology: The psychology of beliefs about knowledge and knowing (pp. 121−141).
Mahwah, NJ: Erlbaum.

Schraw, G., Dunkle, M. E., & Bendixen, L. D. (1995). Cognitive processes in well-defined
and ill-defined problem solving. Applied Cognitive Psychology, 9(6), 523−538.

Schraw, G., & Sinatra, G. M. (2004). Epistemological development and its impact on
cognition in academic domains. Contemporary Educational Psychology, 29(2),
95−102.

Sinatra, G. M., Southerland, S. A., McConaughy, F., & Demastes, J. W. (2003). Intentions
and beliefs in students’ understanding and acceptance of biological evolution.
Journal of Research in Science Teaching, 40(5), 510−528.

Spector, P. E., Schneider, J. R., Vance, C. A., & Hezlett, S. A. (2000). The relation of
cognitive ability and personality traits to assessment center performance. Journal of
Applied Social Psychology, 30, 1474−1491.

Stanovich, K. E., & West, R. F. (1997). Reasoning independently of prior belief and
individual differences in actively open-minded thinking. Journal of Educational
Psychology, 89(2), 342−357.

Stephenson, B., & Hunt, C. (1977). Intellectual and ethical development: A dualistic
curriculum intervention for college students. Counseling Psychologist, 6, 39−42.

Toplak, M. E., & Stanovich, K. E. (2002). The domain specificity and generality of
disjunctive reasoning: Searching for a generalizable critical thinking skill. Journal of
Educational Psychology, 94, 197−209.

Valanides, N., & Angeli, C. (2005). Effects of instruction on changes in epistemological
beliefs. Contemporary Educational Psychology, 30(2), 314−330.

Valanides, N., & Angeli, C. (2008). An exploratory study about the role of epistemological
beliefs and dispositions on learners’ thinking about an ill-defined issue in solo and
duo problem solving contexts. In M. S. Khine (Ed.), Knowing, knowledge, and beliefs:
Epistemological studies across diverse cultures (pp. 197−218). Dordrecht, the
Netherlands: Springer.

Wechsler Adult Intelligence Scale-Third Edition (Chinese Version) (2002). San Antonio,
TX: The Psychological Corporation.

77N.-M. Chan et al. / Learning and Individual Differences 21 (2011) 67–77

Weinstock, M., & Cronin, M. A. (2003). The everyday production of knowledge:
Individual differences in epistemological understanding and juror-reasoning skill.
Applied Cognitive Psychology, 17(2), 161−181.

West, R. F., Toplak, M. E., & Stanovich, K. E. (2008). Heuristics and biases as measures of
critical thinking: Associations with cognitive ability and thinking dispositions.
Journal of Educational Psychology, 100, 930−941.

Wood, P., & Kardash, C. (2002). Critical elements in the design and analysis of studies of
epistemology. In B. K. Hofer, & P. R. Pintrich (Eds.), Personal epistemological: The
psychology of beliefs about knowledge and knowing (pp. 231−260). Mahwah, NJ:
Erlbaum.

Zhang, L. F. (2002). Measuring thinking styles in addition to measuring personality
traits? Personality and Individual Differences, 33(3), 445−458.

Epistemic beliefs and critical thinking of Chinese students
Introduction
Epistemic beliefs
Epistemic beliefs and critical thinking
Effects of epistemic beliefs on argumentation
Epistemic beliefs and other critical thinking factors
The present research
Study 1
Method
Participants
Measures
Procedure
Results and discussion
Preliminary analyses
Relations between epistemic beliefs and other critical �thinking factors
Epistemic beliefs as predictors of critical thinking performance

Study 2
Method
Participants
Thinking task
Measures
Procedure
Results and discussion
Preliminary analyses
Relationship between epistemic beliefs and argumentation

Summary and concluding discussion
Implications for practice
Limitations and further research
Acknowledgement
Appendix A
References

Contents lists available at ScienceDirect

International Journal of Educational Research

journal homepage: www.elsevier.com/locate/ijedures

Understanding and teaching critical thinking—A new approach

Kristoffer Larsson
University of Gothenburg, Sweden

A R T I C L E I N F O

Keywords:
critical thinking
learning
teaching
pedagogy
phenomenography

A B S T R A C T

Developing students’ critical thinking is a major educational goal in societies around the world.
In spite of this, the research community has had serious problems handling this highly prized
goal. In reference to these problems, several issues have been discussed, one being the theory
issue, where the theoretical development has been pointed out as insufficient, especially theory
concerning the learning experience connected with the development of critical thinking. This
article introduces the use of the phenomenographic theoretical approach in the field of critical
thinking. Taking an empirical study as an illustrative example, the article shows how phenom-
enographic theory, in a promising way, could be used for understanding concrete expressions of
critical thinking and designing teaching to develop students’ critical thinking.

1. Introduction

Developing students’ critical thinking skills is regarded as a highly important educational goal in many societies around the world,
as it is seen as promoting such disparate qualities as democracy and personal development (Behar-Horenstein & Niu, 2011; Beyer,
1995; Facione, 2006; Martin, 2005; Tsui, 1998). Despite the importance of critical thinking as an educational goal, the research
community has had serious problems in tackling questions related to this topic. In 2004, Kuhn and Dean described critical thinking as
one of the “major unsolved problems of pedagogy” (Kuhn & Dean, 2004:269) and over the years there have been several research
reviews concluding that we know very little about critical thinking and how to promote it (Brunt, 2005; McMillan, 1987; Tsui, 1998).
One of the more recent of these ended by appealing to the readers to “critically analyse the substance of empirical studies on teaching
critical thinking” (Behar-Horenstein & Niu, 2011:

38

).

Against this background, several issues have been debated in the critical-thinking research community. Three of the more pro-
minent ones have been the definition issue, the measuring issue and the theory issue.

The definition issue concerns how to define critical thinking. It has been said that the phenomenon of critical thinking is an
elusive and complex one, and that it is futile to attempt capture it in any definition (Brodin, 2007; Tsui, 2007). As a result of this,
there has been an extensive debate on what critical thinking really is (Brodin, 2007; Johnson & Hamby, 2015; Mason, 2007; Moore,
2013; Petress, 2004; Phillips & Bond, 2004; Tsui, 1998). One area of debate has concerned what to include and exclude in the
definition. Is critical thinking just a set of skills or does it have dispositional aspects as well? Is creativity an important element or not
and should emotions, intuition and societal change be brought into the definition (Brodin, 2007; Mason, 2007)? Another important
area of debate has been whether critical thinking is subject specific or general (Mason, 2007; Tsui 1998, 2007). McPeck has argued
that critical thinking is something subject specific, depending heavily on the subject that is to be critically thought about, and that
every subject therefore has to find its own definition (McPeck 1990a, 1990b). Although admitting that the subject matter is involved,
the strongest voices in the field, such as Ennis (1987, 1993) and Paul (1992), both long-term authorities in the field (Menssen, 1993),
and Facione (1990), leader of the group that developed the APA/Delphi definition, have argued that there are indeed general

http://dx.doi.org/10.1016/j.ijer.2017.05.004
Received 5 December 2016; Accepted 14 May 2017

E-mail address: kristoffer.larsson@gu.se.

International Journal of Educational Research 84 (2017) 32–

42

Available online 14 June 2017
0883-0

35

5/ © 2017 Elsevier Ltd. All rights reserved.

MARK

http://www.sciencedirect.com/science/journal/08830355

http://www.elsevier.com/locate/ijedures

http://dx.doi.org/10.1016/j.ijer.2017.05.004

http://dx.doi.org/10.1016/j.ijer.2017.05.004

mailto:kristoffer.larsson@gu.se

http://dx.doi.org/10.1016/j.ijer.2017.05.004

http://crossmark.crossref.org/dialog/?doi=10.1016/j.ijer.2017.05.004&domain=pdf

principles of critical thinking that have wide usability and go beyond specific subjects. They have consequently advocated generic
definitions of critical thinking. Having failed to reach full closure on any of these questions, the debate still continues to this day.

The measuring issue concerns how critical thinking skills are measured when studied empirically. Over the years, there has been a
heavy reliance on standardised multiple-choice tests (Brunt, 2005; McMillan 1987; Norris 1986; Tsui 1998), such as the Cornell
Critical Thinking Test and the Watson-Glaser Critical Thinking Appraisal (Behar-Horenstein & Niu, 2011; Tsui, 1998). There are
indeed advantages to these kinds of tests. For example, they give clear-cut answers that do not allow for different interpretations
when assessed, and so have high inter-rater reliability, and there is also the possibility of quite easily conducting large-scale studies,
for example, measuring critical-thinking skills in big cohorts of compulsory school students, thus aiming at generalizability. The
heavy reliance on these kinds of test, however, has also been criticised. One criticism has been that these tests can only handle what
are called well-structured problems with logical solutions. That is not an unimportant part of critical thinking, but it leaves out what
are called ill-structured problems, in short, problems where there is no definite answer and it is up to the individual to make
judgements based on reasoning (King & Kitchener, 1994; Kuhn, 1991). Another criticism has been that these tests only measure
product (right or wrong x-marks on multiple-choice questions) while neglecting process (how the application of critical-thinking
skills led to the answers) (Norris, 1985; Tsui, 1998, 2002; Norris, Leighton, Phillips, 2004; Newmann, 1991; Larsson, In press).
Against this background, suggestions have been made to increase the use of alternative measuring methods, such as open-ended essay
tasks, when measuring critical thinking empirically (Brunt, 2005; Tsui, 1998). This kind of essay task has been said to make the
students reveal their critical-thinking skills, i.e. the essay format forces the students to make their reasoning processes explicit. These
kinds of tasks are also better suited to ill-structured problems, because they can be formulated to allow for uncertainty, postponed
conclusions etc. (King & Kitchener, 1994, 2004; Kuhn, 1991). However, from a measuring point of view, one problem with such tasks
is that they allow the possibility of different interpretations when assessed, making the inter-rater reliability somewhat troublesome.
This, coupled with the fact that they are more suited to small-scale studies, because the empirical data for analysis soon becomes
voluminous, makes for studies with a lower degree of generalizability.

The theory issue was raised by McMillan, in 1987. In his research review, he observed a lack of theoretical development within
the field, especially concerning the “theoretical description of the nature of the learning experience” in connection to the develop-
ment of critical thinking (McMillan, 1987:14). Over the years, there have been some attempts to meet his criticism. Kuhn is one
researcher who has connected her studies to theories from a cognitive development tradition (Kuhn & Dean, 2004; Kuhn, 1991,
1999). In short, her view rests on an assumption that an individual’s skill in thinking critically rests on that individual’s mental
models on a meta-level, the level where models are selected and their use is monitored (Kuhn & Dean, 2004). To become more
qualified in critical thinking, a person has to change her/his mental models to more advanced ones. This could be, for example, a
change in the mental model of causality, from a co-occurrence causality model to a multivariable model of causality. Kuhn affirms
that this change is dependent both on maturity and on learning experiences. In designing learning experiences that promote this kind
of change, it is important not to focus merely on the direct performance level, for example describing what a multivariable model of
causality is and how to use it. Instead the learning experience should be directed towards the mental models on a meta-level. For
example, with regard to the causality models, the learning experience should be designed to focus on why the multivariable model of
causality is preferable and why the other model is less effective, so that the multivariable model becomes the preferred choice for the
individual (Kuhn & Dean, 2004). Another pair of researchers who have connected their studies on critical thinking to a cognitive
development tradition are King and Kitchener (King & Kitchener, 1994, 2004). Though close to Kuhn, King and Kitchener focus
purely on the individual’s assumptions about knowledge and how knowledge is acquired, and link this to an individual’s ability to
think critically. In their model, there are seven different hierarchically ordered views of knowledge, linked to seven hierarchically
ordered skills in thinking critically, where only the two most advanced views of knowledge actually allow for critical thinking.
According to King and Kitchener, a change in one’s view of knowledge (and consequently critical thinking) is due to both maturity
and learning experiences. In order to create learning experiences that facilitate more advanced views of knowledge, King and
Kitchener propose that the experience should be designed to challenge the individual’s current view of knowledge. If this is suc-
cessfully done, the individual becomes aware of the inadequacy of his/her current view of knowledge and over time changes his/her
view to a more advanced one. This change concurrently makes his/her critical thinking more complex (King & Kitchener, 1994).

Though attempts, such as the ones mentioned above, have been made to give a theoretical description of learning experiences
connected to the development of critical thinking, many studies (Sendag & Ferhan Odabasi, 2009; Tsui, 2002; Wheeler & Collins,
2003; Chowning, Griswold, Kovarik, & Collins, 2012; Fung, 2014) still lack a clear-cut and thoroughly developed theory to describe
changes in critical thinking due to different types of learning experiences. On the whole, McMillan’s observation from three decades
ago still holds true.

Using an empirical study as an illustrative example, this article focuses on the theory issue. The article introduces phenomeno-
graphy (Marton & Booth, 1997; Marton, 1981) as a new approach in the field, with a theory that can be used both to understand
manifestations of critical thinking and, building on such understandings, to describe and explain learning experiences that can
enhance critical thinking among students. In this way, the article aims to contribute to theoretical development within the field and
to suggest an alternative approach, rooted in the phenomenological and hermeneutical traditions, to those approaches that have been
tried so far (Sandberg, 1996). Grounded in interpretative traditions, the phenomenographic approach uses qualitative methods to
obtain data for its analysis. In the study presented in this article, open-ended essay tasks were used. As such, the article also indirectly
addresses the measuring issue, showing how these kinds of essay tasks could be used and so gives one alternative to the predominant
usage of standardised multiple-choice tests within the field.

In the article, the phenomenographic approach and its conceptual framework will be developed alongside the empirical study and
the implications of the results.

K. Larsson International Journal of Educational Research 84 (2017) 32–42

33

2. The empirical study

2.1. Aim

The study I will take as my point of departure revolves around lower secondary-school students solving an essay task designed to
elicit critical thinking about the philosopher Robert Nozick’s views of justice.

Within the phenomenographic approach it is an axiom that there is only a limited number of distinctly different ways in which
people are capable of understanding, seeing or experiencing any specific phenomenon they meet (Marton & Booth, 1997). Those
different ways of understanding a phenomenon are logically related and as a rule they can also, in an educational setting, be
hierarchically ordered in relation to a norm or an educational goal (Marton & Booth, 1997). If the phenomenon consists of a task, like
the one on Nozick’s views on justice, that calls for critical thinking, then according to this assumption, there can only be a limited
number of hierarchically ordered different ways of understanding the task among people at a collective level. Furthermore phe-
nomenography makes the assumption that an ability to act in a certain way in relation to a phenomenon is delimited by a certain way
of understanding that specific phenomenon (Marton & Booth, 1997; Marton, 1996; Marton, Runesson, Tsui, 2004). This means that
each of the different ways of understanding a particular phenomenon delimits a specific way of acting in relation to it. If the
phenomenon consists of a task that calls for critical thinking it could then be said that each of the different ways of understanding the
task delimits a specific way of handling it, or more precisely, a specific type of critical thinking in relation to the task. In the light of
the hierarchical order among the different ways of understanding such a task, it can also be said that a more powerful way of
understanding delimits a more powerful type of critical thinking in relation to the task. Similarly, a less powerful way of under-
standing the task delimits a less powerful type of critical thinking in relation to the task.

With this as a background, the aim of the study was:

• To reduce and describe the different hierarchically ordered ways of understanding the task and the types of critical thinking they
delimit.

• To use the findings from the first aim to discuss how to improve students critical thinking using phenomenographic learning and
teaching theory.

2.2. Method

2.2.1. Definition of critical thinking
As stated in the introduction, there has been a great deal of debate on how to conceptualise critical thinking. This study was

designed to connect to Robert Ennis’s generic definition of critical thinking (Ennis, 1987, 1993). This decision was based on three
factors. Firstly, the article argues for a use of phenomenography at a general level when conducting research and designing teaching
on critical thinking, using the presented study, with its particular content, only as an example of how to use the approach. Such an
intent makes the generic property of the Ennis definition suitable Secondly, the Ennis definition is one of the most recognised
definitions of critical thinking in the field, and therefore connecting the study to this definition links the study to the overall body of
work within the field (Masson, 2007; Menssen, 1993; Moore, 2013). Thirdly, most other well-established definitions within the field
are generic in nature, for example, the Rickard Paul definition (1992) and the APA/Delphi definition (Facione, 1990), the latter
having been formulated by 46 of the USA’s and Canada’s leading critical-thinking experts.

Ennis defines critical thinking as “reasonable reflective thinking focused on deciding what to believe or do”. It is indeed a very
dense definition and Ennis makes more than 150 different specifications on what this definition actually means (Ennis, 1987).
However, Ennis also makes an abridgment consisting of only 10 specifications (Ennis, 1993). This study was designed to be in
reasonable accordance with 4 of the specifications in the abridgment, namely a person’s ability to:

1. Identify conclusions, reasons, and assumptions.
2. Judge the quality of an argument, including the acceptability of its reasons, assumptions, and evidence.
3. Develop and defend a position on an issue.
4. Draw conclusions when warranted, but with caution (Ennis, 1993:180).

2.2.2. The task
The task began with the students reading a text introducing some of Robert Nozick’s thoughts on justice (see Appendix A). Based on

this text the students were then asked to scrutinize Nozick’s standpoints. In this they were urged to reason about strengths and weak-
nesses in his way of arguing for his standpoints and to explain why they saw a strength as a strength and a weakness as a weakness.

2.2.3. Task design
Several considerations were made when designing the task. For example, the text, as well as the wording of the actual task, were

designed to be age-appropriate. Moreover, the task was created to be in reasonable accordance with the four specifications from the
Ennis definition. The task was also formulated as an essay task, giving access to the student’s reasoning process and so making it
possible to analyse from a phenomenographic point of view (Newmann, 1991; Norris et al., 2004; Norris, 1985; Tsui, 1998, 2002;
Larsson, In press). Finally, on a continuum between well-structured and ill-structured problems, the task was designed to be close to

K. Larsson International Journal of Educational Research 84 (2017) 32–42

34

the ill-structured end of the continuum (King & Kitchener, 1994; Kuhn, 1991).

2.2.4. Participants
The students participating in the study were aged 15–16 and were in their last year in the Swedish compulsory school. There was a

total of 19 participants (labelled S1-S19), 11 girls and 8 boys: 6, 8 and 5 of them had respectively the lowest, medium and highest
grades in civics; and they came from 3 different schools. The number of students participating was decided upon using empirical
saturation, i.e. new students were added until there was no more relevant variation within in the targeted group. In order to provide a
theoretical sample of a broader population, the composition of the students group was based on a desire to have a variation in sex,
prior achievement and school attended (Ojo & Booth, 2009). Because the task relied heavily on reading and writing ability, students
who had recently immigrated to Sweden and students with dyslexia were excluded from participating.

2.2.5. Implementation
The empirical study was implemented in such a way that each student worked alone on separate occasions, without any time

limit, to solve the task. Each student wrote down their response to the task in a word file on a computer that was blocked for internet
access. There were no other tools available to solve the task except for a dictionary and pencil and paper if the student wanted to
make notes. While the student solved the task, I sat in an adjoining room and from time to time I looked in to check that everything
was alright.

2.2.6. Empirical analysis
To derive the different ways in which the students understood the task about Robert Nozick’s views on justice a phenomeno-

graphic analysis of the students’ written answers was made (Sandberg, 1994). The possibility of doing this rests on the theoretical
assumption mentioned above, namely that the act − i.e. the answer or, so to say, the critical thinking in relation to the task − is
delimited by a certain way of understanding the task (Marton & Booth, 1997; Marton et al., 2004; Marton, 1996). In other words, the
different answers to a task reflect the limited number of ways of understanding the task. The analysis set out to discover these
different ways of understanding the task, to group them together in a category system of logically related hierarchically ordered ways
of understanding and to present them in what, in phenomenographic terms, is called an outcome space (Marton & Booth, 1997). At a
global level, the key question in deriving the categories was: “For the student to arrive at this answer, how would he or she have
understood the task?” (Marton & Säljö 1997 [1984]; Svensson 1997 [1984]). The analysis in itself involves an iterative process of
reading the essay-answers repeatedly, interpreting and reinterpreting, looking at similarities and differences, grouping and re-
grouping them in order to finally put what emerges as the same understanding of the task in the same group (Booth & Ingerman,
2002; Marton & Booth, 1997; Sandberg, 1994). As this is an interpretive process, the results are not seen as incontrovertible, but
should provide a fair and well-founded picture of the different ways of understanding the task represented in the empirical data
(Booth & Ingerman, 2002).

2.3. Empirical results

From the data analysis, an outcome space was established with an internal structure that can be illustrated as in Fig. 1. In total,
five different ways of understanding the task were found, ranging from what are here labelled A to E, each of which leads to a
particular type of critical thinking, also labelled A to E, where E is more complex than D, D is more complex than C, and so on, to the
least complex, A.

To give a better understanding of the outcome space, I will now turn to a more detailed description of the five different ways of
understanding the task, linking them to empirical extracts from the students’ answers, i.e. a certain type of critical thinking.

To be able to do this, it is first necessary to make some further theoretical distinctions.
From a phenomenographic point of view, the most important characteristics of a specific way of understanding a phenomenon

relate to what is called the internal horizon (Marton & Booth, 1997). The internal horizon is constituted of the components or parts of
the phenomenon that are being discerned and how these components are organized (Marton & Booth, 1997). Thus a certain way of
understanding a phenomenon is constituted of a whole of discerned components and their internal organization. In view of the
hierarchical structure of the category system, this ordinarily means that a more complex way of understanding the phenomenon
consists of a whole of more relevant discerned components with a higher degree of internal organization, in relation to an educational
norm, usually equivalent to the scientifically established norm of understanding, for the phenomenon in question. Similarly, a less
complex way of understanding the phenomenon is seen as a whole with fewer relevant components discerned and with a less
integrated organization in relation to the norm (Marton & Booth, 1997).

In line with these theoretical claims, the analysis to derive the different ways of understanding the Nozick task targeted the
internal horizon of a way of understanding and ordered its constituents hierarchically according to a norm. This means that each
different way of understanding the task is described in terms of a whole of discerned components and their organization. This also
means that a more complex way of understanding the task consists of a whole of more relevant discerned components with a higher
degree of internal organization in relation to the norm, as compared to a less complex way of understanding the task. Each specific
way of understanding the task also delimits a certain type of critical thinking, made explicit in the empirical extracts from which the
specific understanding was educed. In the case of this study, the preferred norm was found among the 19 students participating and
equates to the way of understanding the task that delimits the most elaborate critical thinking in relation to the task within this group
of students. It should be noticed that using the participating students as a standard for establishing the norm could mean that there

K. Larsson International Journal of Educational Research 84 (2017) 32–42
35

might be other more advanced ways of understanding the task (F, G and so on) to be found, for instance, if older age groups had been
included in the study. As indicated above, this way of establishing the norm is not the most common one within phenomenography.
However, it was used due to the fact that there is no pre-established, undisputed scientific norm regarding how the Nozick task is to
be understood.

What now follows is a description of each different way of understanding the task, starting with the least complex way of
understanding the task, i.e. Way of understanding A. Each way of understanding is described in terms of its components, and for
details on these, see Table 1.

Way of understanding A
In Way of understanding A, the task is understood as one of describing some of Nozick’s standpoints on justice. The central

components are here limited to the standpoint component and the describing component. A typical student answer, i.e. the type of
critical thinking from which this way of understanding was derived, looked like this:

“Robert Nozick’s argument is that if we don’t let all individuals fight for their own welfare, then individuals will not make an effort
and live on others instead. According to him, it’s not fair if you’re not allowed to keep what you worked for (for example, if the
state takes your property).” (S4)1

Way of understanding B
In way of understanding B, the task is understood as one of scrutinizing one or more of Nozick’s standpoints by using one’s own

values or standpoints.
In comparison to Way of understanding A, this represents a significant shift in the quality of the understanding of the task from

simply describing standpoints to in fact scrutinizing them. If it is linked to the definition of critical thinking used in the study, one
could even say that this shift in understanding between A and B marks the borderline between a way of understanding the task that
allows no actual critical thinking and a way of understanding the task that does allow for some basic kind of critical thinking and not
just passive description. The central components in Way of understanding B are limited to the standpoint component, the scrutinizing

Way of
understanding E

Way of
understanding D

Way of
understanding C

Way of
understanding B

Way of
understanding A

Critical thinking
type E

Critical thinking
type C

Critical thinking
type B

Critical thinking
type A

Critical thinking
type D

Fig. 1. Illustration of the internal structure of the outcome space concerning the task about Robert Nozick’s view on justice. To accentuate that it is the way of
understanding the task that delimits the type of critical thinking in relation to the task, the different types of critical thinking are shown in dotted outlines.

1 This empirical extract, and those presented below, have been translated from Swedish to English by the author. An effort has been made to stay as close to the
original texts as possible. For distribution of individuals across categories, see Appendix B.

K. Larsson International Journal of Educational Research 84 (2017) 32–42

36

component and the value component. A typical answer, i.e. the type of critical thinking from which this way of understanding was
derived, looked like this:

“The strength in Robert Nozick’s argumentation is the right to own things and the right to life. I think it’s a strength because it’s
important to be able to have a life of your own and the right to own what you legally bought, etc. I think the weakness in his
argumentation is that the state must never take taxes from the rich and give to the poor. I think it’s a weakness because the rich
hardly need as much money as they have so why not help someone who does not have it so good.” (S8)

Way of understanding C
Way of understanding C shows the same kind of understanding as the previous one (B) but makes one further distinction within

understanding the scrutinizing part of the task. This distinction is characterised by an understanding that opens up for examining one
broader implication, other than the ones expressed in the text on Nozick (see Appendix A), of one of Nozick’s standpoints. In addition
to discerning the same components as Way of understanding B, Way of understanding C then also discerns one additional component
delimited as the broader implication component. A typical answer from which this way of understanding was derived manifested
critical thinking connected to all four components of the understanding. The extract below is a part of such an answer. The underlined
segment in the answer shows where the broader implication component was derived from.

“Nozick’s strength is that if you own something then it’s yours and nobody can take it away from you, which of course makes it

Table 1
Summarized illustration of the different ways of understanding the task concerning Robert Nozick’s view on justice, complemented with derived educationally critical
differences (ecds). The different ways of understanding are hierarchically ordered with regard to the internal horizon. “Way of understanding E” is the most complex
way of experiencing the task. “Way of understanding A” is the least complex. The italic heading points to the most significant characteristic of the specific way of
understanding in relation to the others.

Way of Understanding E
“To scrutinize by drawing attention to an inner contradiction between two of Nozick’s standpoints”
The discernment of direct standpoints of Nozick’s view in line with the spirit of Nozick’s standpoints (standpoint component). The discernment of the main objective

of the task as one of scrutinizing one/some of Nozick’s standpoints (scrutinizing component). The discernment of personal values/standpoints in relation to one/
some of Nozick’s standpoints and offering opinions on it/them based on personal values/standpoints (value component). The discernment that a specific
standpoint may have broader implications for society/groups/individuals (in addition to those directly treated in the text), if society were organized according
to Nozick’s standpoints (broader implication component). Regarding such a specific broader implication, the discernment that this broader implication can be
explained by argument and conclusion (argument/conclusion component). The discernment that (at least) two of Nozick’s standpoints may be set against one
another and that there may therefore be an internal contradiction in Nozick’s reasoning (contradiction component).

Educationally critical differences (ecds): the contradiction component

Way of Understanding D
“To scrutinize by conclusion and argument regarding a broader implication of some of Nozick’s standpoints”
The discernment of direct standpoints of Nozick’s view in line with the spirit of Nozick’s standpoints (standpoint component). The discernment of the main objective

of the task as one of scrutinizing one/some of Nozick’s standpoints (scrutinizing component). The discernment of personal values/standpoints in relation to one/
some of Nozick’s standpoints and offering opinions on it/them based on personal values/standpoints (value component). The discernment that a specific
standpoint may have broader implications for society/groups/individuals (in addition to those directly treated in the text), if society were organized according
to Nozick’s standpoints (broader implication component). Regarding such a specific broader implication, the discernment that this broader implication can be
explained by argument and conclusion (argument/conclusion component).

Educationally critical differences (ecds): the argument/conclusion component

Way of Understanding C
“To scrutinize by paying attention to one broader implication of one of Nozick’s standpoints”
The discernment of direct standpoints of Nozick’s view in line with the spirit of Nozick’s standpoints (standpoint component). The discernment of the main objective

of the task as one of scrutinizing one/some of Nozick’s standpoints (scrutinizing component). The discernment of personal values/standpoints in relation to one/
some of Nozick’s standpoints and offering opinions on it/them based on personal values/standpoints (value component). The discernment that a specific
standpoint may have broader implications for society/groups/individuals (in addition to those directly treated in the text), if society were organized according
to Nozick’s standpoints (broader implication component).

Educationally critical differences (ecds): the broader implication component

Way of Understanding B
“To scrutinize one/some of Nozicḱs standpoints by using onés own values/standpoints”
The discernment of direct standpoints of Nozick’s view in line with the spirit of Nozick’s standpoints (standpoint component). The discernment of the main objective

of the task as one of scrutinizing one/some of Nozick’s standpoints (scrutinizing component). The discernment of personal values/standpoints in relation to one/
some of Nozick’s standpoints and offering opinions on it/them based on personal values/standpoints (value component).

Educationally critical differences (ecds): the scrutinizing component
and the value component

Way of Understanding A
”To describe some of Nozick’s standpoints”
The discernment of direct standpoints of Nozick’s view in line with the spirit of Nozick’s standpoints (standpoint component). The discernment of the main objective

of the task as one of describing some of Nozick’s standpoints (describing component).

K. Larsson International Journal of Educational Research 84 (2017) 32–42

37

better for those who already have a lot. That he thinks it should be safe for everyone is also good. On the other hand he forgets
everything else − how fair is it to be born rich and be able to live a good life without any effort while others are struggling and
never succeed?” (S5)

Way of understanding D
Way of understanding D encompasses the same basic understanding as the previous way of understanding. But, in addition, Way

of understanding D makes even more distinctions within understanding the scrutinizing part of the task, adding to the possibility of
paying attention to one broader implication of one of Nozick’s standpoints, providing an opening for also explaining and drawing
conclusions around such an implication. Apart from discerning the same components as Way of understanding C, Way of under-
standing D then also discerns one additional component delimited as the argument/conclusion component. A typical answer from
which this way of understanding was derived manifested critical thinking connected to all five components of the understanding. The
extract below is a part of such an answer. The underlined segment in the answer shows where the argument/conclusion component
was derived from.

“… his idea that the State only exists to give people security so they won’t be afraid of someone stealing from them, does not
include anything that is related to income support, assistance with standard of living, health-care, or the like/…/However, he
forgets something very important − democracy. When people do not have food in their stomachs, they will not cope with
engaging in the democratic and political process, and so democracy will fall. When your primary occupation is to take care of your
children and find bread for tomorrow, there will be few who have the strength to vote or to get involved in party politics. That’s
what I find to be the weakest with Robert Nozick’s arguments.” (S3)

Way of understanding E
Way of understanding E, the most complex way of understanding found within the student group, is characterised by the same

understanding as Way of understanding D, but adds yet another distinction within understanding the scrutinizing part of the task.
This distinction revolves around opening up for examining (at least two of) Nozick’s standpoints in relation to each other by looking
for the possibility that there may be a contradiction between them. This distinction also poses a conceivable reorganisation of the way
of discerning Nozick’s standpoints compared to all the previous ways of understanding the task (D-A). In the previous ways of
understanding, there is a linear organization of the standpoints, where each standpoint is discerned as individual and discrete
elements. In Way of understanding E, Nozick’s standpoints are instead (also) discerned as an integrated whole, where the standpoints
can be checked against each other to test their internal congruence. Besides discerning the same components as Way of understanding
D, Way of understanding E then also discerns one additional component delimited as the contradiction component. A typical answer
from which this way of understanding was derived manifested critical thinking connected to all six components of the understanding.
The extract below is an example of such an answer showing only the part of the answer from which the contradiction component was
derived.

“Robert thinks that it doesn’t matter how much each of us owns, it’s how they got it, i.e. if they purchased it legally, then it’s fair.
But he also says that everyone’s got a right to life. But a life of extreme poverty is almost considered as no life. I think in this
argumentation I can see a weakness in that he contradicts himself.” (S10)

3. Using the empirical results together with phenomenographic learning and teaching theory

Next I turn to a discussion on how the above phenomenographic empirical results can be used together with phenomenographic
learning and teaching theory in educational situations to, in some respects, improve critical-thinking skills among students.

Phenomenography views learning as a change in a person’s way of understanding a specific phenomenon (Marton & Booth, 1997).
In an educational setting, this is usually viewed as a change in a person’s way of understanding the phenomenon in the direction of, or
preferably in accordance with, the educational norm of understanding the phenomenon (Marton & Booth, 1997). What is to be
learned builds on the hierarchically ordered ways of understanding a specific phenomenon and targets the differences between the
various ways of understanding the phenomenon with the aim of obtaining access to what are called educationally critical differences
or ecds (Marton & Booth, 1997). Each ecd is equivalent to a certain component and emerges at the intersection points between the
different ways of understanding the phenomenon and against the background of the norm. More specifically, this means that an ecd
emerges when two successive ways of understanding in the hierarchy are compared with each other. The ecd, or possibly the ecds, are
determined and delimited by what is discerned in the more complex way of understanding and is missing in the less complex one and
at the same time is a necessary condition to understand the phenomenon in line with the norm. When a deduction is made throughout
the category system in order to find the ecds, you finally end up with a set of ecds, i.e. components. These ecds are what it is critical
for a person who initially understands the phenomenon in the least complex way to discern in order to understand the phenomenon
in accordance with the norm (Marton & Booth, 1997). If we apply this reasoning to the outcome space of the Nozick task, the ecds are:
the scrutinizing component, the value component, the broader implication component, the argument/conclusion component and the
contradiction component (see Table 1).

If we now know the phenomenographic view on learning and the notions of educational norm and ecds, how then can a teacher
give a person the opportunity to discern the ecds in relation to a norm of a certain phenomenon in an educational setting? According
to the phenomenographic viewpoint, this can be done by using a set of different patterns of variation, the most important ones being
called contrast and separation (Lo & Marton, 2012; Lo, 2012; Marton et al., 2004). The idea behind the pattern of variation called

K. Larsson International Journal of Educational Research 84 (2017) 32–42
38

contrast is that a person will only be able to discern a particular ecd if they can at the same time discern what the ecd is not.
Phenomenography says that a dimension of variation must be opened up in the teaching situation where the ecd constitutes one of at
least two contrasting values within that dimension. Through a contrasting comparison between the ecd and the other value or values,
it becomes possible to discern the specific distinctions of the ecd. Implicitly and simultaneously, it also becomes possible to discern
both the distinctions of the other value or values that are brought into the dimension and the dimension in itself (Lo & Marton, 2012;
Lo, 2012; Marton & Pang, 2006; Marton et al., 2004). Thus, in cases where several ecds could be said to belong to the same dimension
of variation, all of them become possible to discern at the same time in the teaching situation (Lo, 2012). This is because they should
all be brought in as contrasting values in the same dimension of variation.

The idea of the pattern of variation separation is built on two basic assumptions. Firstly, if a specific ecd of a phenomenon is to be
made discernible in a teaching situation, it must be separated from the phenomenon itself through the pattern of variation contrast.
Secondly, when this happens, everything else concerning the phenomenon must be kept invariant or closed. In a teaching situation,
this means that the ecds for the particular phenomenon belonging to the same dimension of variation are treated at the same time
through the pattern of variation contrast, while ecds belonging to other dimensions of variation are kept invariant by their di-
mensions of variation being kept closed. As a final remark, it is also important to be aware that although the two patterns − contrast
and separation − are analytically separable, they are created and occur simultaneously in an educational situation (Lo & Marton,
2012; Lo, 2012; Marton & Pang, 2006; Marton et al., 2004).

Going back to the Nozick task, let’s see how the five ecds could be made possible to discern in an educational setting, i.e. giving
the students an opportunity to improve their way of understanding the task, and in doing so, improving their critical thinking in
relation to it.

When phenomenography describes a teaching sequence in which the ecds for a phenomenon are made discernible, it normally
follows a certain logic, meaning that if one ecd can be assumed to be the base for another, the former is presented before the latter. If
this line of reasoning is followed in the case of the task, it could be argued that discerning the scrutinizing component is considered a
base for discerning the other four ecds. This is because they are all different discernments of how to scrutinize Nozick’s standpoints
and if one is to be able to make these different discernments, one first has to discern that the task is about scrutinizing, and not, for
example, describing or reading out loud. It follows from this argument that the first ecd to be addressed in the suggested teaching
sequence is the scrutinizing component.

So how then do we go about giving the student the opportunity to discern this crucial component? The core of this component can
be said to revolve around ways of handling one of Nozick’s standpoints, where discernment of the scrutinizing component is char-
acterised by handling the standpoint by scrutinizing it. To give the students the opportunity to discern this component, it could be
separated from its original task by opening a dimension of variation concerning [ways of handling one of Nozick’s standpoints] with
one value consisting of the ecd [to scrutinize one of Nozick’s standpoints] and the contrasting value consisting of, for example, [to
describe one of Nozick’s standpoints]. In an actual teaching situation, the opening of this dimension could be carried out by letting
the students experience two different texts, one where a specific standpoint is described and one where the same standpoint is
scrutinized. The emphasis in such a teaching situation should be to clarify the differences between scrutinizing a standpoint and
describing a standpoint. The object of this is to specify, with the help of contrast, what distinguishes the scrutinizing component so it
can be discerned by the students. As supposed above, it is crucial that both the chosen example texts are directed towards the same
standpoint, so the separation does not become unclear by opening up a parallel dimension of variation concerning [Nozick’s
standpoints].

If we now have given the students the opportunity to discern the base-component how do we continue with the other ecds? As
indicated above, it is possible to claim that the four remaining ecds all have one common core: they are all different types of
scrutinizing one of Nozick’s standpoints (or two standpoints, in the case of the contradiction component). One way of offering the
students the opportunity to discern all four of these critical differences, in terms of different types of scrutinizing, is for the teacher to
open up the dimension of variation [ways to scrutinize one/two of Nozick’s standpoints] with the four different scrutinizing types as
contrasting values within that dimension. In an actual teaching situation, this could be done in several ways. One way is to let the four
different types of scrutinizing contrast with each other by means of comparing four different answers (to the task), each characterised
by one of the four types. The emphasis of the comparison should be directed towards the unique distinction for each type of
scrutinizing. Further, it is important that the presented answers concerning the value component, the broader implication component
and the argument/conclusion component all are directed towards one of the two standpoints that are a part of the presented answer
for the contradiction component. This is in order to minimize the risk of blurring the separation opening up a dimension of variation
concerning [Nozick’s standpoints].

When it comes to these four particular ecds, the opening of the dimension [ways of scrutinizing one/two of Nozick’s standpoints]
is most likely not enough to allow the students to fully discern them. The teacher also has to address the organizational differences
between the contradiction component (standpoint seen as an integrated whole containing the potential for scrutinizing the internal
congruence of the standpoints) and the other three components (standpoint seen as linear and separated into parts, limiting the
scrutinizing to one standpoint at the time). This could be done by the teacher opening up a dimension of variation in [ways of
organizing Nozick’s standpoints] with the contrasting values [Nozick’s standpoints organized as integrated whole] and [Nozick’s
standpoint organized as linear separate parts]. In an actual teaching situation, this could be done by choosing, for example, four of
Nozick’s standpoints and organizing them against each other as an integrated whole. It is crucial that it is made clear that this way of
organizing the standpoints is what distinguishes, and is a prerequisite for, the type of scrutiny that characterizes the contradiction
component. This can then be contrasted in relation to an example where the same four standpoints are organized as linear and
individual parts. As with the example concerning the “integrated whole” way of organizing, it is of equal importance that it is made

K. Larsson International Journal of Educational Research 84 (2017) 32–42

39

evident that this way of organizing Nozick’s standpoints distinguishes the three other types of scrutinizing, and limits them to being
directed towards just one standpoint at a time. To avoid opening a dimension of variation concerning [Nozick’s standpoints], making
the separation fuzzy, it is important to underline that the same four standpoints should be used in both of the contrasting examples.

We have now seen an example of a full teaching sequence where all of the ecds for the Nozick task are afforded to the students to
discern. In other words, we have seen how phenomenographic results could be used together with phenomenographic learning and
teaching theory to make it possible for the students to understand the task in line with the norm, i.e. in this case the most complex
way found in the empirical data. As the most complex way of understanding the task delimits the most complex critical thinking in
relation to the task, this also means that the students, by this sequence, would be given an opportunity to develop the most complex
type of critical thinking in relation to the task.

This being said, this possibility of improving the students’ critical thinking has to be seen as quite narrow, as the improvement is
only in relation to the Nozick task. Phenomenography, however, does offer an opportunity to go beyond this limitation and use the
empirical results from the Nozick task in a wider context of improving the students’ critical thinking.

In this regard, let us first take a look at the phenomenographic notion about the possibility of general characteristics in the
different content-specific ways of understanding a phenomenon. This notion rests on the assumption that if the content-specific ways
of understanding a phenomenon are deprived of its specific content, there could be more general characteristics that might appear as
components in ways of understanding another phenomenon of the same kind, or be generalised across several different phenomena of
the same kind (Dahlgren, 1997[1984]; Wenestam, 1980). With this notion as a backbone, the scrutinizing component from the
Nozick task can be taken as an illustrative example. The core of the content-specific scrutinizing component as it appears in the task
could be articulated as ‘discerning that one of Nozick’s standpoints should be scrutinized’. If it is then content-deprived, losing its
“Nozickness”, it could be articulated as ‘discerning that a standpoint should be scrutinized’. As such, it ought to be generalisable to
other similar content consisting of one or more standpoints, preferably within the subject of civics.

If we continue, using this reasoning together with phenomenographic learning and teaching theory, the content-deprived scru-
tinizing component can be introduced to a new content consisting of one or more standpoints and again be made content-specific and
discernible for students in relation to this new content by using the patterns of variation, contrast and separation. This could be done
in the same manner as in the example I used above about how the scrutinizing component could be made discernible in relation to its
original task. The only difference is that the content would now be something other then Nozick’s standpoints, for instance some
political party’s standpoints on some issue. The dimension of variation that is opened in such a potential teaching situation could then
be described in terms of [ways of handling one of the political party X’s standpoints on the issue Z] with one value being the, now
once more content-specific, scrutinizing component [to scrutinize one of the political party X’s standpoints on issue Z] and the
contrasting value being [to describe one of the political party X’s standpoints on issue Z].

Without going any further, this kind of manoeuvre could be carried out for every ecd from the Nozick task, making each of them
discernible in relation to new content consisting of one or more standpoints. In this way, the empirically derived ecds from the Nozick
task could be used in an educational situation to give students the opportunity to improve their critical thinking in a wider sense.

4. Concluding remarks

The article has focused on what has been called the theory issue. It has introduced the phenomenographic approach to the critical
thinking scene and shown how this approach can be used as a strong theoretical frame both for understanding differences in
manifestations of critical thinking and, based on such an understanding, designing learning experiences to develop students’ critical-
thinking skills. As such, the article provides an alternative to the (scarce) theoretical framing that has been tried so far. In relation to
the approaches used by Kuhn and by King and Kitchener, phenomenography offers an alternative to their cognitive developmental
tradition, by bringing in the experiential tradition, with roots in phenomenology and hermeneutics. Most importantly, phenomen-
ography offers a teaching theory to go along with the learning theory, something that is more or less absent in Kuhn’s, as well as King
and Kitchener’s, theoretical framing (King & Kitchener, 1994, 2004; Kuhn & Dean, 2004; Kuhn, 1991, 1999). Their perspective fo-
cuses on a theoretical description of the learning, while not giving any clear-cut and detailed theoretical description of the teaching
that makes that learning possible. For example, Kuhn (Kuhn & Dean 2004) does not give any real theoretical description of the
teaching in relation to changing students’ mental model for causality to a more complex one with regard to the why feature. She only
describes the learning (in theoretical terms) that needs to take place for such a change, i.e. the shifting of mental models. Phe-
nomenography can, on the other hand, by means of, for instance, the theory about patterns of variation and how they should be used
in teaching, present a teaching theory to go along with the learning theory. In this way, phenomenography does not simply provide
another approach, but offers a distinctly new theoretical aspect to the field, which has a substantial bearing on what McMillan has
asked for, i.e. a “theoretical description of the nature of the learning experience” that would help to develop critical thinking
(McMillan, 1987:14). From a more general perspective, the article, through its use of phenomenography, addresses the overall lack of
theory within the field and so makes a valuable contribution to the development of this crucial area.

Though the article has shown how phenomenography could act as a solid and cohesive theoretical frame for understanding and
teaching critical thinking, it is important to notice that the study the article uses to introduce the phenomenographic approach is just
an example of how the approach can be used. Should phenomenography be used in relation to other types of contents, subjects or
problems, or if another definition of critical thinking is to be used, appropriate adjustments must be made. And, as stated earlier,
when it comes to the ecds and the teaching sequences emanating from the Nozick task in the example study, they are, when stripped
of their Nozickness, to some extent applicable to other contents and subjects where one or more standpoints are to be scrutinized.
However, there is no theoretical ground to go further than that.

K. Larsson International Journal of Educational Research 84 (2017) 32–42

40

In sum, the article confirms that phenomenography seems to provide a feasible way to grapple with the theory issue and gives a
strong foundation for developing the theoretical aspects of understanding and teaching critical thinking. Even so, to more thoroughly
examine the feasibility of using phenomenography, further exploration of the approach within this domain of research is necessary.
This can be done in several ways. One of the studies which most urgently needs to be conducted is to test the teaching sequences
suggested above in real classroom situations and evaluate their success. This would presumably be best carried out with a researcher
leading and evaluating the project, and teachers (who would receive instruction in phenomenographic theory) conducting the actual
teaching sequences. Other projects to test the feasibility of the phenomenographic approach could aim at finding ecds emanating
from other types of contents, subjects and problems, and design, test and evaluate teaching sequences based on these investigations.
Such efforts could be made in the action-research inspired learning study practice, which has evolved within the phenomenographic
tradition (Pang & Ki 2016; Pang & Lo 2012; Pang & Marton 2003). In this kind of research, one or more researchers work together
with a group of teachers to develop a specified skill among a group of students using phenomenographic theory as a base.

As cited at the beginning of the article, Kuhn and Dean described critical thinking as one of the “major unsolved problems of
pedagogy” (2004:269). It is not reasonable to think that one single research endeavour or one specific theoretical approach could
solve this problem. This article, however, argues that the phenomenographic approach could be one path worth further exploring on
such a quest.

Appendix A. Text on Robert Nozick

Robert Nozick
In his book “Anarchy, State and Utopia”, the philosopher Robert Nozick writes that we always have the right to keep what we

acquired legally. If everybody originally acquired their property in a just way − for example, by inheritance, gift, purchase or
exchange − then the distribution that arises in a society is also fair.

According to Nozick, a state may never go in and redistribute by taxing the rich and giving to those who are worse off. It is unfair,
even if it were true that a small group owns everything and the majority are starving.

The state should only guarantee that people can live safely and in security, and that they are not subjected to violence or anyone
trying to steal their property. Nozick calls such a state the minimal state.

By far the most important rights a man has, according to Nozick, is the right to property and the right to life. The state may never
restrict these rights. Then, what the allocation of resources looks like in practice is irrelevant from an equity point of view, that is, as
long as everyone has acquired what they own in a just way. On the other hand, if someone acquired their property incorrectly − for
example, by threats or theft − it must be corrected, no matter how far back in time the error was made.

(The text is a modified version of a text from the book “Dilemma – om rätt och fel, etik och moral” [“Dilemma – right and wrong,
ethics and moral”] by Uriel Hedengren. Stockholm: Natur & Kultur, 2002.)

Appendix B. Distribution of individuals across categories

Way of understanding A: S1, S10, S14, S19
Way of understanding B: S3, S6, S7, S11, S13
Way of understanding C: S5
Way of understanding D: S2, S8, S9, S12, S15, S17, S18
Way of understanding E: S4
Not categorised: S16 (due to absence of answer)

References

Behar-Horenstein, L., & Niu, L. (2011). Teaching critical thinking skills in higher education: A review of the literature. Journal of College Teaching & Learning,
8(February (2)), 25–

41

.

Beyer, B. K. (1995). Critical thinking. Bloomington, Indiana: The Phi Delta Kappa Educational Foundation.
Booth, S., & Ingerman (2002). Making sense of Physics in the first year of study. Learning and Instruction, 12(2002), 493–507.
Brodin, E. (2007). Critical thinking in scholarship. Meanings, conditions and development. Lund: Lund University. Department of Education.
Brunt, A. (2005). Critical thinking in nursing: An integrated review. The Journal of Continuing Education in Nursing, 36(March/April (2)), 60–67.
Chowning, J., Griswold, J., Kovarik, D., & Collins, L. (2012). Fostering critical thinking, reasoning, and argumentation skills through bioethics education. PLoS One,

7(May (5)).
Dahlgren L-O. (1997) [1984]. Learning Conceptions and Outcomes. In F. Marton, D. Hounsell, & N. Entwistle (ed.) The Experience of Learning. Edinburgh: Scottish

Academic Press Limited.
Ennis, R. H. (1987). A taxonomy of critical thinking dispositions and abilities. In J. K. Baron, & R. J. Sternberg (Eds.), Teaching thinking skills: Theory and practice (pp. 9–

26). .
Ennis, R. H. (1993). Critical thinking assessment. Theory into Practice, 32(Summer (3)), 179–186.
Facione, P. A. (1990). Critical thinking: A statement of expert consensus for purposes of educational assessment and instruction. Executive Summary The Delphi ReportMillbrae:

The California Academic Press.
Facione, P. A. (2006). Critical thinking: What is it and why it counts. California: Academic Press.
Fung, D. (2014). Promoting critical thinking through effective group work: A teaching intervention for Hong Kong primary school students. International Journal of

Educational Research, 66, 45–62.
Johnson, R. H., & Hamby, B. (2015). A meta-Level approach to the problem of defining ‘Critical thinking’. Argumentation, 4, 417–430.
King, P. M., & Kitchener, K. S. (1994). Developing reflective judgment. Understanding and promoting intellectual growth and critical thinking in adolescents and adults. San

Francisco: Jossey-Bass Publishers.
King, P. M., & Kitchener, K. S. (2004). Reflective judgment: Theory and research on the development of epistemic assumptions through adulthood. Educational

K. Larsson International Journal of Educational Research 84 (2017) 32–42
41

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0005

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0005

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0010

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0015

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0020

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0025

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0030

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0030

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0040

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0040

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0045

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0050

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0050

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0055

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0060

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0060

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0065

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0070

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0070

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0075

Psychologist, 39(1), 5–18.
Kuhn, D., & Dean, D., Jr. (2004). Metacognition: A bridge between cognitive psychology and educational practice. Theory Into Practice, 43(4), 268–273.
Kuhn, D. (1991). The skills of argument. New York: Cambridge University Press.
Kuhn, D. (1999). A developmental model of critical thinking. Educational Researcher, 28(2), 16–46.
Larsson K. (in press) Critical thinking in students’ ethical reasoning: A reflection on some examples from the Swedish national tests in religious education. In O. Franck

(ed.) Assessment in Ethics Education −A Case of National Tests in Religious Education. Dordrecht: Springer.
Lo, M. L., & Marton, F. (2012). Towards a science of the art of teaching: Using variation theory as a guiding principle of pedagogical design. International Journal for

Lesson and Learning Studies, 1(1), 7–22.
Lo, M. L. (2012). Variation theory and the improvement of teaching and learning. Göteborg: Acta Universitatis Gothoburgensis.
Martin, D. S. (2005). Critical thinking for democracy and social justice. In N. M. Michelli, & D. L. Keiser (Eds.), Teacher education for democracy and social justice (pp.

209–228). New York: Routledge.
Marton, F., & Booth, S. (1997). Learning and awareness. Mahwah: Lawrence Erlbaum Associates.
Marton, F., & Pang, M. F. (2006). On some necessary conditions of learning. The Journal of the Learning Sciences, 15(2), 193–220.
Marton F., Säljö R. (1997) [1984]. Approaches to Learning. In F. Marton, D. Hounsell, & N. Entwistle (eds.) The Experience of Learning. Edinburgh: Scottish Academic

Press Limited.
Marton, F., Runesson, U., & Tsui, A. B. M. (2004). The space of learning. In F. Marton, & A. B. M. Tsui (Eds.), Classroom discourse and the space of learning (pp. 3–40).

Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
Marton, F. (1981). Phenomenography −Describing conceptions of the world around us. Instructional Science, 10, 177–200.
Marton, F. (1996). Cognosco ergo sum −Reflections on reflections. In G. Dall’Alba, & B. Hasselgren (Eds.), Reflections on phenomenography. Toward a methodology? (pp.

163–187). Göteborg: Acta Universitatis Gothoburgensis.
Mason, M. (2007). Critical thinking and learning. Educational Philosophy and Theory, 39(4), 339–349.
McMillan, J. H. (1987). Enhancing college students’ critical thinking: A review of studies. Research in Higher Education, 26(1), 3–29.
McPeck, J. E. (1990a). Teaching critical thinking. Dialogue and dialectic. New York: Routledge.
McPeck, J. E. (1990b). Critical thinking and subject specificity: A reply to ennis. Educational Researcher, 19(4), 10–12.
Menssen, S. (1993). Critical thinking and the construction of knowledge. The American Behavioral Scientist, 37(1), 85–93.
Moore, T. (2013). Critical thinking: Seven definitions in search of a concept. Studies in Higher Education, 38(4), 506–522.
Newmann, F. M. (1991). Classroom thoughtfulness and students’ higher order thinking: Common indicators and diverse social studies courses. Madison, Wisconsin: National

Center on Effective Secondary Schools.
Norris, S. P., Leighton, J. P., & Phillips, L. M. (2004). What is at stake in knowing the content and capabilities of children’s minds? A case for basing high stakes tests on

cognitive models, 2. Theory and Research in Education, 2(3), 283–308.
Norris, S. P. (1985). Synthesis of research on critical thinking. Educational Leadership, 1985(May), 40–45.
Ojo, E., & Booth, S. (2009). Internationalisation of higher education in a South African university: A phenomenographic study of students’ conceptions. Education as

Change, 13(December (2)), 309–323.
Pang, M., & Ki, W. (2016). Revisiting the idea of critical aspects. Scandinavian Journal of Educational Research, 1–14.
Pang, M., & Lo, M. L. (2012). Learning study: Helping teachers to use theory, develop professionally, and produce new knowledge to be shared. Instructional Science,

40(3), 589–606.
Pang, M., & Marton, F. (2003). Beyond lesson study: Comparing two ways of facilitating the grasp of some economic concepts. Instructional Science, 31(3), 175–194.
Paul, R. (1992). Critical thinking: What, why, and how. New Directions for Community Colleges, 1992(77), 3–24.
Petress, K. (2004). Critical thinking: An extended definition. Education, 124(3), 461–466.
Phillips, V., & Bond, C. (2004). Undergraduates’ experiences of critical thinking. Higher Education Research & Development, 23(August (3)), 277–294.
Sandberg, J. (1994). Human competence at work. An interpretative approach. Göteborg: BAS.
Sandberg, J. (1996). Are phenomenographic results reliable? In G. Dall’Alba, & B. Hasselgren (Eds.), Reflections on phenomenography. Toward a methodology? (pp. 129–

140). Göteborg: Acta Universitatis Gothoburgensis.
Sendag, S., & Ferhan Odabasi, H. (2009). Effects of an online problem based learning course on content knowledge acquisition and critical thinking skills.

Computers & Education, 53(1), 132–141.
Svensson L. (1997) [1984]. Skills in learning and Organising Knowledge. In F. Marton, D. Hounsell, N. Entwistle (eds.) The Experience of Learning. Edinburgh: Scottish

Academic Press Limited.
Tsui, L. (1998). A review of research on critical thinking. Paper presented at the annual meeting of the association for study of higher education, 5–8 november 1998.
Tsui, L. (2002). Fostering critical thinking through effective pedagogy: Evidence from four institutional case studies. The Journal of Higher Education, 73(6), 740–763.
Tsui, L. (2007). Cultivating critical thinking: Insights from an elite liberal arts college. The Journal of General Education, 56(3 & 4), 200–227.
Wenestam, C.-G. (1980). Qualitative differences in retention. Göteborg: Acta Universitatis Gothoburgensis.
Wheeler, & Collins (2003). The influence of concept mapping on critical thinking in baccalaureate nursing students. Journal of Professional Nursing, 19(6), 339–346.

K. Larsson International Journal of Educational Research 84 (2017) 32–42
42

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0075

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0080

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0085

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0090

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0100

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0100

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0105

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0110

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0110

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0115

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0120

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0130

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0130

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0135

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0140

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0140

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0145

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0150

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0155

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0160

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0165

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0170

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0175

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0175

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0180

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0180

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0185

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0190

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0190

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0195

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0200

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0200

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0205

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0210

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0215

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0220

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0225

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0230

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0230

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0235

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0235

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0245

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0250

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0255

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0260

http://refhub.elsevier.com/S0883-0355(16)31257-5/sbref0265

Understanding and teaching critical thinking—A new approach
Introduction
The empirical study
Aim
Method
Definition of critical thinking
The task
Task design
Participants
Implementation
Empirical analysis
Empirical results
Using the empirical results together with phenomenographic learning and teaching theory
Concluding remarks
Text on Robert Nozick
Distribution of individuals across categories
References

lable at ScienceDirect

Learning and Instruction 29 (2014) 31e42

Contents lists avai

Learning and Instruction

journal homepage: www.elsevier.com/locate/learninstruc

Improving critical thinking: Effects of dispositions and instructions
on economics students’ reasoning skills

Anita Heijltjes a,*, Tamara van Gog b, Jimmie Leppink c, Fred Paas b,d

a Learning and Innovation Centre, Avans University of Applied Sciences, Hogeschoollaan 1, Postbus 90116, 4800RA Breda, The Netherlands
b Institute of Psychology, Erasmus University Rotterdam, The Netherlands
c Department of Educational Development and Research, Maastricht University, The Netherlands
d Interdisciplinary Educational Research Institute, University of Wollongong, Australia

a r t i c l e i n f o

Article history:
Received 15 February 2012
Received in revised form
10 July 2013
Accepted 17 July 2013

Keywords:
Critical thinking instructions
Dispositions
Biased reasoning

* Corresponding author.
E-mail address: aeg.heijltjes@avans.nl (A. Heijltjes

0959-4752/$ e see front matter � 2013 Elsevier Ltd.
http://dx.doi.org/10.1016/j.learninstruc.2013.07.003

a b s t r a c t

This experiment investigated the impact of critical thinking dispositions and instructions on economics
students’ performance on reasoning skills. Participants (N ¼ 183) were exposed to one of four conditions:
critical thinking instruction, critical thinking instruction with self-explanation prompts during subse

–

quent practice, critical thinking instruction with activation prompts during subsequent practice, or no
critical thinking instruction or prompts (control). In all conditions, practice was a within-subjects factor,
some task categories present in the test were practiced on a business case, others were not. Participants
in the instruction conditions significantly outperformed participants in the control condition on th

e

immediate and delayed post-test, but only on the practiced task categories e with the exception of the
self-explanations condition, which also showed a better performance than the control condition on not-
practiced categories, though only on the immediate post-test. Dispositions (i.e., Actively Open-minded
Thinking and Need for Cognition) predicted reasoning skills at pre-test but did not interact with in-
structions on post-tests performances.

� 2013 Elsevier Ltd.

All rights reserved.

1. Introduction

Against the background of complex and rapidly changing busi-
ness environments, economics students are expected to become
critical thinkers (Klebba & Hamilton, 2007; Smith, 2003). Critical
thinking enables students to make sound logical and unbiased de-
cisions, and in educational situations it has been shown to lead to
better learning and transfer outcomes (e.g., Facione, 2009; Halpern,
1998; Helsdingen, Van Gog, & Van Merriënboer, 2011). Therefore, it
is surprising that critical thinking is rarely explicitly taught in eco-
nomics curricula (Jones, 2007). This might be due to the prevailing
view that immersion in business methods and strategies will lead to
the spontaneous development of critical thinking skills (Jones, 2007

).

However, there is little evidence that such skills develop spontane-
ously as a consequence of instruction in a discipline (Halpern, 1999).
On the contrary, research has shown that critical thinking seems to
increase only if taught explicitly (e.g., Abrami et al., 2008). It is
difficult, though, for educators in economics (and other domains for

).
All rights reserved.

that matter) to derive guidelines from existing research regarding
when, where and how to foster critical thinking throughout the
curriculum (Smith, 2003). According to Abrami et al. (2008), existing
studies on critical thinking instruction often lack a powerful empir-
ical design (see also McMillan, 1987; Ten Dam & Volman, 2004;
Wolcott, Baril, Cunningham, Fordham, & St.Pierre, 2002). Moreover,
the definition of critical thinking or the aspects of critical thinking
that are investigated tend to differ (Abrami et al., 2008).

The present studyaims to empiricallyexamine the effects of explicit
critical thinking instructions on an essential aspect of critical thinking,
whichis highly important todecision-making incomplex andhigh-risk
situations that often occur in the field of economics: the avoidance of
biased reasoning by means of more controlled rational judgment and
decision-making (Evans, 2003; Smith, 2003; Stanovich & Stanovich,
2010; Tversky & Kahneman, 1983; West, Toplak, & Stanovich, 2008).
Because such biases seem inherent to human reasoning and decision-
making, economists are no exception and are also prone to violating
the principles of rationality (Sanfey, Loewenstein, McClure, & Cohen,
2006). This study investigates whether declarative critical thinking in-
struction followed by practice, either by itself or combined with self-
explanation or activation prompts during practice, would enhance

Delta:1_’

Delta:1_given name

Delta:1_surname

Delta:1_given name

mailto:aeg.heijltjes@avans.nl

http://crossmark.crossref.org/dialog/?doi=10.1016/j.learninstruc.2013.07.003&domain=pdf

www.sciencedirect.com/science/journal/09594752

http://www.elsevier.com/locate/learninstruc

http://dx.doi.org/10.1016/j.learninstruc.2013.07.003

http://dx.doi.org/10.1016/j.learninstruc.2013.07.003

http://dx.doi.org/10.1016/j.learninstruc.2013.07.003

A. Heijltjes et al. / Learning and Instruction 29 (2014) 31e4232

economics students’ learning of reasoning skills. Moreover, it is inves-
tigated whether individual differences in thinking dispositions are
associated with initial performance on those skills and with learning.

Before describing the research on critical thinking instructions
and the rationale behind our study, we will first address the prin-
ciples of critical thinking (especially in relation to biased reasoning)
and its underlying cognitive mechanisms in more detail.

1.1. Critical thinking

The ‘American Philosophical Association Delphi Panel’ (Facione,
1990), characterized critical thinking for educational purposes (i.e.,
to support the instruction and assessment of critical thinking) as an
extensive concept including both cognitive skills and dispositions.
A critical thinker must, for example, be skilled at reasoning, which
refers to the cognitive process of drawing conclusions from given
information (Facione, 1990). Although the critical thinking litera-
ture in educational research has focused on a wide variety of topics
in the past decades (see e.g. Angeli & Valanides, 2009), little
attention has been paid to the avoidance of biased reasoning (West
et al., 2008). Critical thinking, in terms of avoiding biased reasoning,
may be classified as a subspecies of rational thinking (Facione,
2009; Stanovich, 2011; West et al., 2008). The dual processing
framework unraveled the underlying cognitive processes of
rational thinking (e.g., Evans, 2003, 2008, 2011). According to the-
ories of dual processing, two distinct types of reasoning processes
are at work. Type 1 processes have a rapid, automatic nature and
involve little reflection. Decision-making is based on past experi-
ences and requires little effort, which is useful and efficient in many
routine situations. However, because of its automaticity, it might
also result in biased thinking in other situations, unless Type 2
processes overrule these automatic responses by explicit reasoning
efforts. Type 2 processes are slow, sequential in nature, and require
the exclusion of attention to other matters, and therefore draw
more heavily on working memory capacity.

Stanovich (2009) distinguishes Type 2 processes into reflective
and algorithmic operations. The reflective mind operates at an
intentional level based on dispositions such as beliefs, cognitive
style, goals, and epistemic values, which affect the algorithmic
mind. Research has shown that dispositions such as ‘actively open-
minded thinking’ (AOT, i.e., the active search for evidence against
one’s own beliefs, plans, or goals and the ability to weigh available
evidence fairly; Baron, 2008) and ‘need for cognition’ (NFC, i.e., “the
tendency for an individual to engage in and enjoy thinking”;
Cacioppo, Petty, & Kao, 1984, p. 306) predict performance on tasks
associated with rational thinking such as syllogisms, statistical
reasoning, and framing (Stanovich & West, 1997, 2007; for a review
on NFC, see Cacioppo, Petty, Feinstein, & Jarvis, 1996). Individuals
with higher AOT scores performed better on argument evaluation
(Stanovich & West, 1997) and co-variation judgment (Sa, Kelley, Ho,
& Stanovich, 2005; West et al., 2008), considered more alternative
possibilities (i.e., other possibilities than initially favored), asked
themselves more frequently what possibility would produce the
best expected outcomes, and showed less overconfidence in hasty
conclusions than individuals with lower AOT scores (Baron, 2008).
NFC is characterized by cognitive motivation that predicts perfor-
mance on cognitive tasks. For example, individuals with higher NFC
scores have been found to recall more of the information to which
they are exposed, are more focused on substantive merits of the
information (e.g., are more influenced by the quality of arguments
of persuasive messages), generate more task-relevant thoughts
which reflect the quality of arguments, make more thoughtful
judgments (i.e., scrutinize and elaborate material more) and put
more effort into cognitive tasks than individuals with lower NFC
scores.

The algorithmic mind performs analytic and inhibitory processes
that enable a person to process information in such a way that the
correct actions are taken (Stanovich, 2009). Thus, the algorithmic
mind has the ability to override Type 1 processes by applying
knowledge of inferential rules and strategies of rational thought (e.g.
probabilistic reasoning, causal reasoning and logic). Failures to
override Type 1 reasoning often occur on classical heuristics and
biases tasks, which tend to evoke an automatic response, while they
require causal and probabilistic reasoning, assessment of covaria-
tion, a tendency to think statistically, and to think of alternative
explanations. Forexample Tverskyand Kahneman (1983) illustrated
poor probabilistic reasoning on a classical conjuntion task: “Linda is
31 years old, single, outspoken, and very bright. She majored in phi-
losophy. As a student, she was deeply concerned with issues of
discrimination and social justice, and also participated in anti-nuclear
demonstrations. Which of the two alternatives are more probable: 1.
Linda is a bank-teller or 2. Linda is a bank teller and is active in the
feminist movement.” (p. 297). Most people are inclined to choose
option 2, but indicating that option 2 is more probable than option 1
violates the conjunction rule because a conjunction cannot be more
probable than one of its ingredients (P(A&B) � P (B)).

Although heuristics and biases tasks are largely unexploited in
the traditional critical thinking literature and measurement (West
et al., 2008), these tasks allow for the assessment of the degree of
rationality in terms of reflective and algorithmic mechanisms
(Stanovich, Toplak, & West, 2008). Failures to override Type 1 pro-
cesses, demonstrated on heuristics and biases tasks (Stanovich,
2009), might occur due to lack of declarative knowledge of
reasoning skills or insufficient strategies to use available knowledge.
Both might presumably be counteracted by instructions.

1.2. Critical thinking instruction: avoiding biased reasoning

Relatively few studies have focused on the questions of how to
avoid biased reasoning in favor of more rational thinking
(Stanovich & Stanovich, 2010), of whether rational thinking is
trainable, of what type of instructions would be most helpful, and
of whether instructions have a persistent impact on learners’
thinking beyond the period of instruction (for reviews see Ritchart
& Perkins, 2005; Stanovich, 2011).

Research that has been conducted on educational interventions
revealed some evidence that explicit rule training on the law of
large numbers fostered performance on base-rate problems (Fong,
Krantz, & Nisbett, 1986), that explicit debiasing instructions (i.e.,
instructions to decouple prior beliefs and opinions from evaluation
of evidence and arguments such as prompting subjects with ‘give
both reasons for and reasons against’) improved performance on
argument generation tasks and syllogistic reasoning tasks
(Macpherson & Stanovich, 2007). Fong et al. (1986) found that both
teaching the rules and guidance to apply these rules in particular
domains by examples, were both effective; however, providing
examples significantly improved the use of abstract rule systems.
According to Nisbett, Fong, Lehman, and Cheng (1987), on some
tasks (e.g. causal and conditional tasks) reasoning improvement
only had an enduring effect if abstract rule training and example
training were both provided.

In the academic domains of psychology, medicine, and law,
Lehman, Lempert, and Nisbett (1988) showed that teaching infer-
ential and logical rules improved students’ reasoning performances
in domain-related tasks as well as on everyday life problems. In the
financial domain, Larrick, Morgan, and Nisbett (1990) found that
participants who had been instructed on cost-benefit principles,
followed by using these principles on examples and indicating
whether they agreed or disagreed with the reasoning in the ex-
amples, became more rational on reasoning (i.e., responded more

A. Heijltjes et al. / Learning and Instruction 29 (2014) 31e42 33

normatively) compared to untrained participants and applied
normative principles on a different type of problem, both imme-
diately and after a full month, and in different contexts (i.e., transfer
took place).

These findings on debiasing instructions are consistent with
studies in the broader critical thinking literature (e.g. Abrami et al.,
2008; Angeli & Valanides, 2009; Bangert-Drowns & Bankert, 1990),
which show that general critical thinking instruction combined
with the integration of critical thinking principles into subject
matter instruction (e.g., thought provoking activities) works best
for improving critical thinking.

Moreover, these findings suggest that two key-factors appear to
play a role in critical thinking instruction: explicit teaching of
reasoning principles and room for practicing these principles. Ac-
cording to Bransford, Sherwood, Vye, and Rieser (1986), combining
explicit instructions with practice allows students to integrate
general rules into a domain specific context, which helps them to
integrate, combine, and cluster information in a meaningful way.
This would result in a conceptual and practical organization of
knowledge that facilitates future performance on such tasks.

When both explicit instruction and practicing would be
required for acquiring reasoning skills, then the question arises
whether additional prompts during practice might have added
value and if so, which kind of prompts? Two potential prompting
methods might be useful and will be elaborated upon below: self-
explanation and activation prompts.

1.2.1. Self-explanation prompts
Regarding self-explanation, Austin, Gregory, and Chiu (2008)

found that prompting students to self-assess and reflect
improved their performances on reasoning tasks. Participants who
provided a brief written explanation of how they arrived at the
rating and a rationalization of why they selected a particular
answer, demonstrated a significantly better performance on critical
thinking (i.e., less heuristic and more algorithmic reasoning), than
participants who did not receive these prompts. Angeli and
Valanides (2009) examined the impact of critical thinking teach-
ing methods in performances on an ill-defined problem (e.g. to
discuss an issue and to produce an outline for a paper) and found
the highest effect size for the condition in which students discussed
an issue, and reflected on their thinking combined with a short
lecture about critical thinking. These findings are consistent with
studies about the effectiveness of self-explanation.

Self-explanation fosters the proper use of available knowledge
and skills, which engages students in active and meaningful
learning while effectively monitoring their understanding (Roy &
Chi, 2005). The underlying cognitive mechanisms of self-
explaining have been described as “generating inferences to fill
missing information, integrating information within study mate-
rials, integrating new information with prior knowledge and
monitoring and repairing faulty knowledge” (Roy & Chi, 2005, p.
272). According to Lombrozo (2006), explaining is especially useful
for approaches to reasoning as it promotes learning by the inte-
gration of new information with prior beliefs. Explanations help to
assess the probability of for example claims in light of prior beliefs,
evoke beliefs-revision, encourage learners to re-describe materials,
and allow them to identify relevant principles (Williams &
Lombrozo, 2010).

Moreover, self-explanation fosters generalizations. Aleven and
Koedinger (2002) found that students who explained their steps
during problem solving learned more effectively and generalize
readily to novel situations, than students who did not explain their
steps. Hence self-explanation might be a particularly effective
strategy to promote both learning and transfer (e.g., Lombrozo,
2006; Renkl, 2005; Rittle-Johnson, 2006), even at a delay: Rittle-

Johnson (2006) found that the effects of direct instruction com-
bined with self-explanation facilitated learning and transfer addi-
tively and that the benefits persisted over a 2-week delay. Finally,
these beneficial effects of self-explanations occur even though
participants rarely receive feedback on the quality of their expla-
nations (Matthews & Rittle-Johnson, 2009). Based on these studies
it can be hypothesized that prompting self-explanation during
practice of reasoning tasks would have an additional impact on
acquisition of reasoning skills, as it helps not only to integrate ac-
quired knowledge and skills (which only practising would also do),
but helps to examine gaps, evoke belief-revision, and foster gen-
eralizations during practice.

1.2.2. Activation prompts
Activation prompts relate to the assumption that reasoning

biases are due to a inhibition failure in working memory and not to
a lack of understanding of probabilistic principles or a lack of
deductive competence (Moutier, Angeard, & Houdé, 2002; Moutier
& Houdé, 2003). Moutier et al. showed that inhibition training led
to better performance on Wason selection tasks and conjunction
tasks. For instance, they used an adaptation of the classical ‘Linda
task’ described above, which requires inhibition of the misleading
scheme (the conjunction) and activation of the relevant scheme
(the probability). A control group (no training) was compared to a
strictly logical training group (in which the experimenter explained
the misleading scheme and admonished subjects not to fall in a
trap), and an inhibition-training group. The inhibition-training
group received the logical training and in addition, participants
were encouraged to redirect their attention toward logically rele-
vant information. However, even though it can be effective, inhi-
bition training may have the unintended side effect of sensitizing
the mind to other thoughts that have to be avoided (Wenzlaff &
Bates, 2000). Stanovich and Stanovich (2010) also stated that sup-
pressing the initial response (Type 1 reasoning) is only helpful
when a better response is available to substitute for it. In line with
findings on inhibition training, an alternative means of increasing
Type 2 reasoning might be the activation of the ‘weaker’ cues
instead of the suppression of dominant ones (i.e., evoke an atten-
tion shift from irrelevant to relevant information; Miyake,
Friedman, Emerson, Witzki, & Howerter, 2000).

Thus, based on previous studies it would be reasonable to expect
that activation prompts during practice would encourage an
attention shift to relevant tasks aspects, which would have an
additional impact on the acquisition of reasoning skills.

In sum, an important question for education, which is addressed
in the present study, is whether critical thinking in terms of unbi-
ased reasoning can be enhanced by explicit instructions in combi-
nation with practice, and whether self-explanation or the
activation prompts during practice can further enhance reasoning
skills.

1.3. The present study

The present study addresses the following questions: a) What is
the impact of individual differences in dispositions on economics
students’ critical thinking as measured by their performance on
judgment and reasoning tasks? b) What are the effects of in-
structions, practice, and prompts during practice on economics
students’ critical thinking as measured by their performance on
reasoning tasks immediately and after a three-week delay?

Regarding dispositions, we hypothesized in line with previous
research that those participants with higher scores on dispositions
(i.e., AOT and NFC) would score better on an initial assessment of
critical thinking skills than participants with lower scores on these
dispositions (Hypothesis 1). An interesting related question is

A. Heijltjes et al. / Learning and Instruction 29 (2014) 31e4234

whether students with higher scores on dispositions would also
benefit more from the critical instruction than students with
lower scores on dispositions, or whether instruction is equally
effective for all participants; this question is explored here
(Question 1).

Regarding instructions, it is hypothesized that critical thinking
instruction enhances performance on critical thinking skills
compared to no instruction, both immediately (Hypothesis 2a) and
after a three-week delay (Hypothesis 2b); however, based on prior
research it can be expected that this beneficial effect would only
arise when instruction is combined with practice. Therefore,
practice is taken into account as a within-subjects factor, and it is
hypothesized that only those tasks which will be instructed and
practiced, would enhance reasoning skills (Hypothesis 2c).

If critical thinking instruction has a general effect, then perfor-
mance on both practiced and not-practiced tasks should be
enhanced compared to the control (i.e., no instruction) condition;
however, if instruction only has a beneficial effect when combined
with practice, only performance on practiced tasks should be
enhanced compared to the control condition. Secondly, it is hy-
pothesized that after instruction, combining practice with prompts
(either self-explanation or activation prompts) would foster
acquisition of critical thinking skills compared to instruction and
practice only (i.e., without prompts) both immediately (Hypothesis
3a) and after a three-week delay (Hypothesis 3b).

2. Method

2.1. Participants and design

Participants were 195 part-time Economics students of a Dutch
University of Applied Sciences with various educational back-
grounds (in the Netherlands, Universities of Applied Sciences can
be entered via various secondary education trajectories). Twelve
participants had to be excluded because of missing data, leaving a
final sample of 183 students (121 men, 62 women; age M ¼ 29.3,
SD ¼ 6.5). Participants were randomly assigned to one of four
conditions: 1) no critical thinking instruction (control; n ¼ 40), 2)
critical thinking instruction (n ¼ 46), 3) critical thinking instruc-
tion plus self-explanation prompts (n ¼ 47), and 4) critical thinking
instruction plus activation prompts (n ¼ 50). The experiment
consisted of 5 phases: 1) pre-test, 2) instructions (critical thinking
or unrelated topic depending on condition), 3) practice of
reasoning skills on a business case, 4) immediate post-test, 5)
delayed post-test. Note regarding phase 3 that in each condition,
practice was a within-subjects factor (i.e., some task categories
were practiced during the practice phase, others were not). In the
prompting conditions, the prompts were provided during practice
(phase 3).

2.2. Materials and procedure

2.2.1. Critical thinking disposition tests
We used a Dutch translation of the 41-item Actively Open-

minded Thinking test (AOT; Stanovich & West, 2007) and the
18-item (short form) Need for Cognition questionnaire (NFC;
Cacioppo et al., 1984) to measure critical thinking dispositions.
Both consist of items requiring a response on a 6-point Likert scale
(AOT: as in Stanovich & West, 2007; NFC: as in West et al., 2008;
note that Cacioppo et al., 1984 used a 9-point scale) ranging from
(1) strongly disagree to (6) strongly agree. Scores on the items are
summed for AOT and for NFC separately (after reverse scoring
items that are formulated negatively). Higher scores on AOT
characterize a greater tendency toward open-minded thinking.
The reliability of the AOT was good: the internal consistency

(Cronbach’s alpha) was 0.82. Higher scores on the NFC represent a
greater tendency to engage in and enjoy effortful cognitive en-
deavors. Reliability of the NFC was also good: the internal con-
sistency (Cronbach’s alpha) was 0.80.

2.2.2. Critical thinking skills tests
The critical thinking skills tests consisted of sixteen tasks (see

Appendix A for an example task in each category). The tasks in the
pre-test, immediate, and delayed post-test were structurally
equivalent, but surface features (cover stories) differed.

The reasoning tasks examined the heuristic tendency: 1) to be
influenced by intense personal and case evidence in favor of more
representative statistical evidence (two causal base-rate tasks
adapted from Fong et al., 1986), 2) to base judgments on prior
belief and intuition without taking sufficient account of the base-
rate frequency (two non-causal base-rate tasks adapted from De
Neys & Glumicic, 2008), 3) to neglect simple and fundamental
qualitative rules of probability in conjunction problems in which a
conjunction cannot be more probable than one of its ingredients
(two conjunction tasks, adapted from Tversky & Kahneman, 1983),
4) to shift outcomes when the same information is framed in
different ways as in cases of gains and risky options (two framing
tasks adapted from Tversky & Kahneman, 1981), 5) to evaluate the
information given in a 2 � 2 contingency table unequally, that is,
to base estimations on already experienced evidence and disre-
gard some of the presented evidence (two covariation tasks
adapted from Wasserman, Dorner, & Kao, 1990), 6) to examine the
tendency to verify rules rather than to falsify them (two Wason
selection tasks adapted from Stanovich, 2009, and Wason &
Shapiro, 1971), and 7) to examine the tendency to evaluate the
logical validity of arguments on the basis of one’s prior beliefs
about the truth of conclusions on syllogistic reasoning tasks
(Evans, 2003; Markovits & Nantel, 1989; Sa et al., 1999). In syllo-
gistic reasoning tasks a conclusion is drawn from two given pre-
mises or assumed propositions and participants have to indicate
whether or not the conclusion follows logically from the premises.
Different types of inconsistent (i.e., the validity of the conclusion is
in conflict with its believability) syllogistic tasks were included
(one of each type): affirming the antecedent or modus ponens (if p
then q, p therefore q; valid), affirmation of the consequent (if p
then q, q therefore p; invalid), denial of the antecedent (if p then q,
not p therefore not q; invalid) and denial of the consequent, or
modus tollens (if p then q, not q therefore not p; valid).

The content of the surface features (cover stories) of the tasks
was adapted to the interests of students in the economics domain.
The format of the tasks differed; a multiple-choice format with two
to five answer options (depending on task characteristics) was used
(the correct answer based upon rational reasoning strategies and
incorrect answers related to biased reasoning). The internal con-
sistency (Cronbach’s alpha) on the three tests was 0.50, 0.70 and
0.73, respectively. Modest reliability for the pre-test composite
score could be expected and is in line with previous research (de
Bruin, Parker, & Fischhoff, 2007; West et al., 2008). It should be
noted that reliability on the post-tests was much higher. It is not
uncommon for pre-tests to show a lower reliability as performance
prior to instruction is often more random or variable than perfor-
mance on post-tests after instruction.

2.2.3. Critical thinking instructions
The critical thinking instructions (see Table 1 for an overview of

the study design) in the experimental conditions consisted of
computer-based presentation (visual and auditory information) of
15 min. in which the features of critical thinking, its importance, the
required reasoning skills, the dispositions, and the risk of biased
thinking and fallacies in thinking were explained. Examples and

Table 1
Overview of study design.

Experimental conditions Video-based critical thinking
instruction

Practice 4 out of 7
categories of tasks

Additional prompts
on practiced tasks

Control No (unrelated video) Yes No
Critical thinking instruction Yes Yes No
Critical thinking þ self-explanation prompts Yes Yes Yes
Critical thinking þ activation prompts Yes Yes Yes

A. Heijltjes et al. / Learning and Instruction 29 (2014) 31e42 35

demonstrations of all task categories were provided, referring back
to the tasks seen in the pre-test, which could have allowed par-
ticipants to mentally correct initially erroneous responses. As an
example, the base-rate fallacy was demonstrated by a slide pre-
senting an image of a person, not to identify as male or female, with
a guitar, beer, and some engineering tools. Next to the image the
visual text appears: base-rate fallacy, group: 990 women, 10 men.
At the same time the following auditory information was given:
“The base-rate fallacy is a thinking failure that occurs when the sta-
tistical distribution of a population is ignored. For example when we
select a participant randomly out of 1000 participants with 990
women and 10 men, and we will tell you that this person is called Sam,
loves drinking beer and listening to hardrock music, and has graduated
as mechanical engineer, then most people tend to find it most likely
that this person is a man. In that case they ignore that the total group
contains of 10 men only” (NB: a different name was used in the
video, a Dutch name that can also be given to both men and
women, but for the sake of clarity we used an English name here).
The computer controlled the sequence and pace of the video, but
participants could pause, forward and rewind, although observa-
tions during the experiment suggested they hardly made use of
these options The control group received a 15 min. digital video on
an unrelated topic (i.e., what happens in your brain when you are in
love).

2.2.4. Practice phase
After the video-instruction all participants were exposed to a

business-case from an economics course containing a description
of a coffee manufacturer who had to decide about marketing,
quality control, extending the assortment, and the sustainability
measures. Four categories of tasks (i.e., a contingency task, a
conjunction task, a non-causal base-rate task, and a Wason selec-
tion task) were practiced with a similar format as the tasks used in
the tests, but the cover stories of the practice tasks were derived
from the business case. Participants did not receive feedback on the
quality of their performance on the practice tasks.

Participants in the control condition and the critical thinking
instruction only condition performed the practice tasks without
additional prompts. In the instruction plus self-explanation con-
dition, prompts were given after each task to self-explain how the
answer was obtained: ‘Explain by using keywords how you’ve
come to the answer’. In the instruction plus activation prompts
condition, prompts were given prior to the task to focus on the
relevant factors. Participants were prompted with hints: ‘search for
confirmation and refutation’ on the contingency task, ‘think of the
logical probability’ on the conjunction task, ‘think of the statistical
distribution’ on the non-causal base-rate task, and ‘violation?’ on
the Wason selection task.

To measure whether mental effort invested in practicing tasks
differed between conditions, a 9-point subjective rating scale
ranging from (1) very, very low effort to (9) very, very high effort
(Paas, 1992) was applied after each task in the practice phase.
Mental effort is an indicator of actual cognitive load, and this scale

is widely used in educational research (for a review see Van Gog &
Paas, 2008).

2.2.5. Procedure
The experiment was run in 11 sessions in a computer room at

the university with 15e22 participants per session and all condi-
tions represented in each session; participants had been randomly
assigned to one of the conditions prior to the experiment. All of the
materials were delivered in a computer-based environment that
was created for this experiment and participants could work at
their own pace. They first completed the pre-test and dispositions
tests. Then, participants in the experimental conditions received
the critical thinking instruction, while participants in the control
condition watched the video on an unrelated topic. Subsequently,
all participants read the business case and performed the practice
tasks, with or without prompts depending on their assigned con-
dition. Finally, they completed the immediate post-test. Three
weeks later, all participants were requested via e-mail to complete
the delayed post-test online.

2.2.6. Data analysis
For each correct answer on the critical thinking skills pre-test,

immediate post-test, and delayed post-test, 1 point was assigned,
resulting in a maximum score of

8

points for practiced tasks, and

8

points for not-practiced tasks, on each test.

Ratings on the AOT and the NFC were summed after reverse
scoring negatively formulated items, resulting in a maximum
score on the AOT of 246 and on the NFC of 108. Partial Eta-squared
(hp

2) values were computed to estimate the magnitude of group
differences prior to treatment (i.e., pretest) in test performance
and critical thinking dispositions, and Cramer’s V was computed
for group differences with regard to educational background.
Further, Odds Ratios (ORs) were computed to estimate the
magnitude of dichotomous variables that might predict dropout
from immediate to delayed post-test, and Cohen’s d-values were
computed to estimate the standardized difference in average age,
AOT, NFC, pre-test performance, immediate post-test performance,
and invested mental effort during instruction between dropouts
and non-dropouts. R2 values were computed to estimate age as a
predictor of post-test scores and AOT and NFC as predictors of pre-
test scores. Finally, standardized regression coefficients (betas)
were computed for predictor variables in the multilevel model for
immediate and delayed post-test performance. To take the hier-
archical structure of the data into account, not the raw test per-
formance standard deviation (as linked to the standardized beta
formula calculated in single-level multiple regression analysis) but
the residual (adjusted) standard deviation of test performance was
used to calculate the betas. The lowest level variance is then “the
amount of variation in the outcome measure attributable to the
individual observation after appropriate controls have been made”
(Schagen & Elliot, 2004, p. 13). Although effect sizes tend to be
slightly larger in the residual standard deviation approach as
compared to a raw standard deviation approach, “such

A. Heijltjes et al. / Learning and Instruction 29 (2014) 31e4236

calculations are considered appropriate because they explicitly
model the extent and impact of clustering in the data” (Schagen &
Elliot, 2004, p. 13).

5.00

6.00

7.00

8

3. Results

For all analyzes, a significance level of 0.05 was used, except for
post-hoc comparisons between conditions for which a significance
level of 0.05/6 was used (note that the uncorrected p-values are
reported). Random assignment of participants to conditions had
been successful; there were no significant differences between
conditions in critical thinking pre-test performance on practiced
tasks, F(3, 179) ¼ 1.36, p ¼ 0.258, hp2 ¼ 0.02, not-practiced tasks, F(3,
179) ¼ 0.44, p ¼ 0.723, hp2 ¼ 0.01, critical thinking dispositions (AOT:
F(3, 179) ¼ 0.33, p ¼ 0.803, hp2 ¼ 0.01; NFC: F(3, 179) ¼ 1.2, p ¼ 0.311,
hp
2 ¼ 0 02), and educational background, c2(6) ¼ 8.38, p ¼ 0.212,

V ¼ 0.15. Regarding the following analyzes, it is worth noting that
the response rate on the delayed post-test was approximately 45%
(N ¼ 85): control (n ¼ 16), critical thinking instruction (n ¼ 25),
critical thinking instruction plus self-explanation prompts (n ¼ 21),
and critical thinking instruction plus activation prompts (n ¼ 23).
Logistic regression to investigate whether participants who
completed the immediate post-test only (i.e., dropouts) differed
from participants who completed both the immediate and the
delayed post-test (i.e., persisters) reveal that dropouts could not be
predicted based on AOT scores (p ¼ 0.284, dropouts M ¼ 179.88,
SD ¼ 17.09, persisters M ¼ 177.36, SD 14.45, d ¼ 0.16), NFC scores
(p ¼ 0.432, dropouts M ¼ 77.98, SD ¼ 9.11, persisters M ¼ 79.05, SD
9.44, d ¼ �0.12), educational background (p ¼ 0.851, OR ¼ 1.04),
gender (p ¼ 0.188, OR ¼ 0.65), pre-test performance (p ¼ 0.513,
dropouts M ¼ 7.68, SD ¼ 2.31, persisters M ¼ 7.92, SD ¼ 2.55,
d ¼ �0.01), immediate-post-test performance (p ¼ 0.283, dropouts
M ¼ 9.11, SD ¼ 3.31, persisters M ¼ 9.60, SD ¼ 2.76, d ¼ �0.16), in-
struction conditions (p ¼ 0.862, OR ¼ 0.95) or invested mental
effort during instructions (p ¼ 0.342, dropouts M ¼ 3.65, SD ¼ 1.30,
persisters M ¼ 3.83, SD ¼ 1.20, d ¼ �0.14). Only age was a significant
predictor of dropout (p ¼ 0.004, dropouts M ¼ 30.6, SD ¼ 7.35,
persisters M ¼ 27.9, SD ¼ 5.12, d ¼ 0.43), indicating a small to me-
dium effect (Cohen, 1992); however this effect seems of minor
importance as age was not a predictor of immediate post-test
scores (p ¼ 0.089, R2 ¼ 0.02) or delayed post-test scores
(p ¼ 0.076, R2 ¼ 0.04). All in all, the p-values in combination with
the measures of effect size indicate that the groups who did and did
not complete the delayed posttest were comparable on the re-
ported variables prior to treatment.

To explore whether invested mental effort during practice
differed among instruction conditions an ANOVA was conducted,
which showed no significant differences among conditions, F(3,
179) ¼ 1.30, p ¼ 0.276, hp2 ¼ 0.02.

Note. Group A: control, Group B: critical thinking instruction, group C: critical thinking instruction + self-
explanation prompts, Group D: critical thinking + activation prompts. Covariates in the model kept constant on
their mean value: Scores on pre-test reasoning tasks, invested mental effort during instruction, Actively Open-
minded Thinking, and Need for Cognition.

0.00

1.00

2.00

3.00

4.00

Immediate Post-test Delayed Post-test

S
co

re
s

ra
ng

e
0

–

Group A Group B Group C Group D

Fig. 1. Estimated marginal means of practiced tasks on immediate and delayed post-
test by instruction conditions.

3.1. Dispositions and pre-test performance

To test the hypothesis that participants with higher scores on
dispositions (i.e., AOT and NFC) would score better on the pre-test
of critical thinking skills than participants with lower scores on
these dispositions (Hypothesis 1), a multiple regression analysis
was performed. On the dispositions tests, data from 11 participants
were lost due to a technical error. Their scores were replaced with
the average sample scores. In line with our hypothesis, the dispo-
sitions AOT and NFC significantly predicted pre-test critical
thinking skills, F(2, 180) ¼ 11.55, p < 0.001, R2 ¼ 0.11, indicating a medium effect (Cohen, 1992). Regarding the standardized co- efficients the AOT scores showed a slightly higher impact, b ¼ 0.22, t(180) ¼ 2.95, p ¼ 0.004, than the NFC scores, b ¼ 0.20, t(180) ¼

2.64, p ¼ 0.009, indicating a small to medium effect (Lipsey &
Wilson, 2001).

3.2. Effects of instructions and dispositions on immediate and
delayed post-test performance

To include not only participants who completed the immediate
and the delayed test but to use as much information as possible, a
multilevel regression model was used. The following fixed factors
were included in the model consistently: pre-test scores (as co-
variate), dispositions scores (i.e., AOT and NFC), invested mental
effort scores during instruction, instruction conditions, practice
(i.e., practiced tasks versus not practiced tasks), and test moment
(i.e., immediate post-test scores and delayed post-test scores). A
student-level random intercept and random slope for practice
were included in the model. No abnormal departures from
normality or outliers were found. Figs. 1 and 2 display the mean
reasoning scores of practiced and not-practiced tasks in each
condition on the immediate and delayed post-test. Table 5 pre-
sents the adjusted means along with their standard errors and 95%
confidence intervals for every combination of condition by prac-
tice by test moment, keeping all quantitative covariates constant
at their mean value.

Regarding the explorative question of whether students with
higher scores on dispositions would also benefit more from the
critical instruction than students with lower scores on disposi-
tions, or whether instruction would be equally effective for all
participants (Question 1), Table 2 shows that the dispositions and
critical thinking performance were correlated significantly even
after instruction (i.e., mainly between AOT and post-test scores),
however the multilevel regression analysis, using Satterthwaite
approximation for the degrees of freedom in SPSS version 21
(Table 3), revealed that no interactions were found between scores
on dispositions (i.e., AOT and NFC) and instructions, between
dispositions and practice, or between dispositions and test
moment.

The hypotheses that instruction would have an effect on post-
test performance (Hypothesis 2a) when combined with practice
(Hypothesis 2c), that prompts during practice would further
enhance the effects of instructions and practice (Hypothesis 3a),
and that these effects remained the same after a three-week delay
(Hypothesis 2b and 3b), were also tested by means of the

Table 3
Fixed effects for predictors of reasoning scores.

Parameter df F p

Intercept 1186.06 43.63 <0.001 Conditions 3289.58 3.94 0.009 Practice 1302.88 3.17 0.076 Test moment 1255.56 51.95 <0.001 Conditions � practice 3302.20 10.85 <0.001 Conditions � test moment 3254.98 1.10 0.351 Conditions � practice � test moment 3259.66 0.28 0.842 Practice � test moment 1260.12 60.93 <0.001 Pre-test reasoning scores 1166.67 82.39 <0.001 Invested mental effort 1173.95 4.62 0.033 NFC 1262.90 0.11 0.747 AOT 1267.42 1.96 0.163 NFC � test moment 1255.12 1.04 0.309 AOT � test moment 1270.45 0.85 0.358 NFC � conditions 3166.92 0.96 0.414 AOT � conditions 3178.35 1.95 0.124 NFC � practice 1217.96 1.50 0.223 AOT � practice 1235.33 2.55 0.112

Note. Intercept based on not-practiced tasks on the immediate post-test; Practice:
practiced tasks (code ‘1’) versus not-practiced tasks (code ‘0’); Test moment: im-
mediate post-test (code ‘0’) versus delayed post-test (code ‘1’). AOT: Actively Open-
minded Thinking, NFC: Need for Cognition, Conditions: control, critical thinking
instruction, critical thinking instruction þ self-explanation prompts, critical thin-
king þ activation prompts.

Note. Group A: control, Group B: critical thinking instruction, group C: critical thinking instruction + self-
explanation prompts, Group D: critical thinking + activation prompts. Covariates in the model kept constant on
their mean value: Scores on pre-test reasoning tasks, invested mental effort during instruction, Actively Open-
minded Thinking and Need for Cognition.

0.00
1.00
2.00
3.00
4.00
5.00
6.00
Immediate Post-test Delayed Post-test
S
co
re
s
ra
ng

e
0-

8
Group A Group B Group C Group D

Fig. 2. Estimated marginal means of not-practiced tasks on immediate and delayed
post-test by instruction conditions.

A. Heijltjes et al. / Learning and Instruction 29 (2014) 31e42 37

multilevel regression analysis (Table 3). This analysis showed
significant main effects of instruction condition and test moment,
and significant interaction effects between instruction condition
and practice, and between test moment and practice. Table 4
shows the B-values for the main and interaction effects and the
effect sizes (betas). Note that the intercept is based on not-
practiced tasks of the immediate post-test in the control condi-
tion (i.e., group A). The betas for pre-test reasoning scores and
invested mental effort indicate that pre-test reasoning score has a
strong positive effect on immediate and delayed post-test per-
formance, while invested mental effort during instruction appears
to have a small to moderate negative effect on performance. In
line with the significance tests presented in Table 3, the signifi-
cance tests and betas in Table 4 appear to indicate that NFC and
AOT hardly influence post-test performance. Further, the condi-
tion by practice interaction appears to be a strong effect; in the
control condition, immediate post-test performance is consider-
ably lower on practiced tasks (notice the negative beta), while in
the other three conditions immediate post-test performance is
much higher on the practiced tasks than on the not practiced
tasks.

To investigate the interaction effect between condition and
practice further, simple effect analyzes taking into account the
factors practice (i.e., practiced versus not-practiced tasks) and test
moment (i.e., immediate and delayed post-test), revealed that
participants in the instruction conditions outperformed the con-
trols on practiced tasks on the immediate post-test (all p < 0.001) and on the delayed post-test (all p smaller than 0.002). LSD post-

Table 2
Correlations between critical thinking performance and dispositions (AOT and NFC) on pre

1 2

1. AOT e
2. NFC 0.321*** e
3. Pre-test practiced tasks 0.220** 0.173*
4. Pre-test not-practiced tasks 0.244** 0.263***
5. Immediate post-test practiced-tasks 0.323*** 0.192**
6. Immediate post-test not-practiced tasks 0.259*** 0.114
7. Delayed post-test practiced tasks 0.330** 0.123
8. Delayed post-test not-practiced tasks 0.222* 0.106

Note. AOT: actively open-minded thinking scores, NFC: need for cognition scores, *** Signi

hoc tests on practiced tasks on the immediate and delayed
post-test revealed no significant differences between instruction
conditions (immediate test: all p larger than 0.484, delayed test: all
p larger than 0.053). On not-practiced tasks, only participants in the
instruction condition with self-explanation prompts outperformed
those in the control condition on the immediate post-test
(p < 0.001) but not on the delayed post-test (p ¼ 0.199). LSD post- hoc tests showed that other comparisons between conditions on not-practiced tasks were not significant on either the immediate post-test (all p larger than 0.020) or the delayed post-test (all p larger than 0.030). Note that the uncorrected p-values are reported, but these are no longer significant after correction for multiple tests (i.e., 0.05/6).

Finally, the practice by test moment interaction (Table 4) in-
dicates that there is a decrease in performance on the practiced
tasks from immediate to delayed post-test whereas there is no
change in performance on not-practiced tasks; the beta of
practice by test moment suggests that this is a strong effect. Note
though (Table 3), that there was no significant interaction with
condition.

4. Discussion

The aim of this study was to examine a) the impact of individual
differences in dispositions and b) effects of instructions, practice,

-test, immediate post-test and delayed post-test of practiced and not-practiced tasks.

3 4 5 6 7

e

0.364*** e
0.379*** 0.317*** e
0.341*** 0.502*** 0.440*** e
0.240* 0.198 0.653*** 0.434*** e
0.283** 0.430*** 0.355** 0.655*** 0.428***

ficant at the 0.001 level, ** Significant at the 0.001 level, * Significant at the 0.05 level.

Table 4
Fixed effects estimates and covariance estimates of reasoning scores.

Parameter B (SE) df t p Effect size (beta)

Level 1 Fixed effects
Intercept 1.98 (0.43) 208.38 4.62 <0.001 Group B 0.52 (0.29) 289.85 1.81 0.072 0.48 Group C 0.99 (0.29) 289.48 3.44 <0.001 0.91 Group D 0.55 (0.28) 288.78 1.94 0.054 0.51 Practice �1.09 (0.29) 302.01 �3.74 <0.001 �1.01 Test moment 0.92 (0.35) 262.03 2.66 0.008 0.85 Practice � test moment �1.49 (0.48) 267.24 �3.11 0.002 �1.38 Group B � practice 1.86 (0.40) 302.10 4.66 <0.001 1.72 Group C � practice 1.41 (0.40) 301.15 3.53 <0.001 1.30 Group D � practice 2.07 (0.39) 302.64 5.31 <0.001 1.91 Group B � test moment 0.63 (0.45) 256.05 1.40 0.161 0.58 Group C � test moment �0.02 (0.46) 259.51 �0.05 0.962 �0.02 Group D � test moment �0.11 (0.45) 258.31 �0.24 0.807 0.10 Group B � practice � test moment 0.01 (0.62) 259.53 0.02 0.983 0.01 Group C x Practice � Test moment �0.28 (0.64) 264.55 �0.44 0.664 �0.26 Group D � practice � test moment �0.44 (0.63) 264.58 �0.70 0.487 �0.40 Pre-test reasoning scores 0.32 (0.04) 166.67 9.08 <0.001 0.71 Invested mental effort �0.14 (0.07) 173.95 �2.15 0.033 �0.16 NFC 0.01 (0.02) 198.80 0.52 0.602 0.09 AOT �0.01 (0.01) 213.56 �1.18 0.241 �0.18 Group B � AOT 0.03 (0.02) 169.30 1.78 0.076 0.40 Group C � AOT 0.04 (0.01) 186.69 2.07 0.040 0.45 Group D � AOT 0.03 (0.01) 177.66 1.91 0.058 0.40 Group B � NFC �0.02 (0.03) 163.57 �0.60 0.552 �0.14 Group C � NFC �0.04 (0.03) 175.93 �1.46 0.146 �0.33 Group D � NFC �0.00 (0.03) 167.16 �0.09 0.932 �0.02 AOT � practice 0.01 (0.01) 235.33 1.60 0.112 0.19 NFC � practice 0.02 (0.01) 217.96 1.22 0.223 0.15 AOT � test moment 0.01 (0.01) 270.45 0.92 0.358 0.11 NFC � test moment �0.01 (0.01) 255.12 �1.02 0.309 �0.11 Level 2 Random parameters

cov (SE) Wald Z p
Intercepts participants 0.53 (0.13) 4.25 <0.001 Practice participants 1.05 (0.26) 4.02 <0.001 Residuals 1.17 (0.11) 10.21 <0.001

Note. Intercept based on not-practiced tasks on the immediate post-test; Practice: practiced tasks (code ‘1’) versus not-practiced tasks (code ‘0’); Test moment: immediate
post-test (code ‘0’) versus delayed post-test (code ‘1’). Conditions: group A: control condition, group B: CT instruction, group C: CT instruction þ self-explanation prompts,
group D: CT-Instruction þ activation prompts; NFC: Need for Cognition; AOT: Actively Open-minded Thinking.
Effect sizes: 0.1 small; 0.25 medium, 0.40 large (Lipsey & Wilson, 2001).

A. Heijltjes et al. / Learning and Instruction 29 (2014) 31e4238

and prompts on economics students’ critical thinking as measured
by their performance on reasoning tasks on an immediate and
delayed post-test.

In line with our first hypothesis (Hypothesis 1), the results
clearly indicate that those participants with higher scores on dis-
positions (i.e., AOT and NFC) scored significantly better on the

Table 5
Estimated marginal means, standard errors and 95% confidence intervals of prac-
ticed and not-practiced task categories on the immediate and delayed post-test by
groups.

Group Test-moment

Immediate post-test Delayed post-test

M SE CI M SE CI

Practiced tasks A 4.83 0.34 [4.17, 5.50] 3.98 0.23 [3.53, 4.43]
B 5.95 0.27 [5.42, 6.49] 4.64 0.21 [4.04, 4.88]
C 5.87 0.29 [5.29, 6.44] 4.94 0.21 [4.52, 5.36]
D 5.60 0.28 [5.05, 6.16] 4.50 0.21 [4.09, 4.90]

Not-practiced
tasks

A 2.29 0.34 [1.63, 2.96] 2.88 0.23 [2.43, 3.33]
B 5.31 0.27 [4.77, 5.84] 5.23 0.21 [4.81, 5.65]
C 4.40 0.29 [3.82, 4.98] 5.26 0.21 [4.83, 5.68]
D 4.55 0.28 [3.40, 5.11] 5.47 0.21 [5.07, 5.87]

Note. Group A: control, Group B: instruction only, Group C: instruction and self-
explanation prompts, Group D: instruction and activation prompts. Covariates in
the model kept constant on their mean value: Scores on pre-test reasoning tasks,
invested mental effort during instruction, AOT: Actively Open-minded, Thinking
NFC: Need for Cognition.

initial assessment of critical thinking skills than participants with
lower scores on dispositions. These results converge with findings
in other studies on argument evaluation (Stanovich & West, 1997),
syllogistic reasoning (Macpherson & Stanovich, 2007), and covari-
ation judgment (Sa et al., 2005; West et al., 2008). Regarding the
question of whether students with higher scores on dispositions
would benefit more from the critical thinking instruction than
students with lower scores on dispositions (Question 1), our results
showed no significant interaction effects between instruction
conditions and dispositions (i.e., AOT and NFC). Thus, it seems that
despite the relationship between dispositions and pre-test perfor-
mance, all students benefitted equally from instructions in terms of
post-test performance; students who score higher on dispositions
did not benefit more from instructions than students who score
lower on dispositions or vice versa. These results indicate that the
active search for evidence against one’s own beliefs, plans, or goals,
and the ability to weigh available evidence fairly (i.e., AOT; Baron,
2008), and the intrinsic cognitive motivation (i.e., NFC; Cacioppo
et al., 1996), which are both important features of the reflective
mind (Stanovich, 2011), appear to regulate reasoning, but not
learning to reason. These explorative findings suggest that dispo-
sitions might be overruled by extrinsic factors in the educational
context; declarative instruction to search for general and under-
lying principles and the instruction to give considerable thoughts
to the instructions seems to have canceled out any influence of

A. Heijltjes et al. / Learning and Instruction 29 (2014) 31e42 39

AOT and NFC on learning. These results indicate that instructions
were effective for all participants regardless of their disposition
scores.

Our second hypothesis was that critical thinking instruction
would enhance performance on critical thinking skills compared to
no instruction, both immediately (Hypothesis 2a) and after a three-
week delay (Hypothesis 2b), though only when combined with
practice (Hypothesis 2c). Indeed, it seems that the combination of
instructions with practice is crucial and has a large effect: on both
the immediate and the delayed post-test the improvements on
reasoning skills compared to the control condition were found on
task categories that were practiced with the business case, but not
on task categories that had not been practiced. There was one
exception: on the immediate test, the self-explanation condition
performed better than the control condition on not-practiced tasks.
We will return to this finding below, when discussing effects of
prompting (Hypothesis 3). Our results are in line with and extend
findings from previous studies in which it was found that explicit
instructions improved critical thinking (e.g., Abrami et al., 2008;
Angeli & Valanides, 2009), that rational thinking to override Type
1 processes is trainable through explicit instruction (e.g., Nisbett
et al., 1987) and that even short instructions can have a persistent
impact on becoming more rational at reasoning (Larrick et al.,
1990). On the other hand, our results also show that critical
thinking instruction without practice was not sufficient to lead to
sustained effects and that practice had an added value for reasoning
instruction. Practice seems to play a key role and presumably leads
to better learning by allowing participants to integrate and elabo-
rate information from the instructions (e.g., Bransford et al., 1986)
with tasks in the economics context. Interestingly, mental effort
during practice did not differ between conditions, and combined
with the differences in post-test scores on practiced tasks, this
suggests that the processes engaged in during practice differ as a
function of instructions, but that the cognitive demands imposed
by those processes do not differ (i.e., practice becomes more
efficient).

The results only lend partial support to our third hypothesis
that after instruction, combining practice with prompts, would
foster acquisition of critical thinking skills compared to instruction
and practice only both immediately (Hypothesis 3a) and after a
three-week delay (Hypothesis 3b). Activation prompts, which we
expected to be able to affect critical thinking performance by
redirecting attention to relevant cues, did not lead to better test
performance than instructions only. This finding might indicate
that instructions were sufficient to redirect attention and inhibit
initial automatic responses during practice. Self-explanation
prompts, which we expected to foster the proper use of avail-
able knowledge (Roy & Chi, 2005) also did not foster performance
on practiced reasoning tasks compared to instructions only.
Interestingly, however, participants in this condition performed
better on not-practiced reasoning tasks on the immediate post-
test than the control condition, suggesting that prompting self-
explanations established a kind of transfer from practiced to
not-practiced tasks, at least in the short run (this effect was no
longer present on the delayed test). This result converges with
studies on the benefits of self-explanation of problem solving
tasks that showed that self-explanation can foster transfer (e.g.,
Aleven & Koedinger, 2002; Lombrozo, 2006; Renkl, 2005; Rittle-
Johnson, 2006). However, given that the difference with the
other instruction conditions was not significant, and that the effect
was no longer present on the delayed test, this effect should be
interpreted with caution and should be replicated in future
research. It might also be worthwhile to attempt to deepen
instructional effects of self-explanation in future studies by

explicitly teaching it as a meta-strategy or by providing feedback
on self-explanations, as this might further enhance the effects of
self-explanation on critical thinking and potentially on transfer to
not-practiced tasks.

In sum, based on these findings, we can conclude that dis-
positions only have an impact on pre-test reasoning perfor-
mance but not on learning of reasoning skills. Secondly we can
conclude that explicit reasoning instructions should be com-
bined with opportunities for practice to have an effect on crit-
ical thinking performance. Thirdly, prompting self-explanation
during practice might be an interesting instructional method as
we found some indications of transfer in terms of performance
benefits on not-practiced reasoning tasks compared to the
control condition; however, further research would be necessary
to establish the merits of this method for teaching reasoning
skills.

This study has some limitations. First, for practical reasons it
was not possible to administer the delayed post-test at the
institute as the rest of the experiment, however the web-based
environment that was used for the delayed post-test was the
same as the one used during the session at the institute. Sec-
ondly, despite the fact that students could complete the delayed
test wherever they wanted, there was a rather high attrition rate
on the delayed test. However, it should be noted that participants
who did complete the delayed test did not differ from partici-
pants who did not on pre-test performance, dispositions,
educational background, gender, invested mental effort during
instruction, and performance on the immediate test. Third, we
did not include conditions that received the prompts during
practice but no critical thinking instruction. The reason for that
was that it can be questioned whether prompts would have a
beneficial effect for students lacking knowledge about the tasks
(i.e., without instructions first). As Stanovich and Stanovich
(2010) stated, on the tasks we used, suppressing the initial
response (Type 1 reasoning) is only helpful when a better
response is available to substitute for it. And regarding self-
explanations it has been shown that without any knowledge, it
would be unlikely that students would be able to provide deep,
principled explanations, which have been shown to be the most
effective (Renkl, 1997). Under conditions of very low prior
knowledge, instructional explanations have been shown to be
most effective (Renkl, 2002). The instructions provided students
with a basis for responding to the prompts, which they would
not have had otherwise. Given the low effectiveness of prompt-
ing even after instructions, it seems highly unlikely that
prompting without instructions first would have had an effect.
Nevertheless, we did not directly test this, and future research
might resolve the question of whether self-explanation or acti-
vation prompts have an impact on reasoning improvements
without prior critical thinking instruction. Finally, it should be
kept in mind that this study defined critical thinking in terms of
reasoning skills, and that the findings therefore may not gener-
alize to other definitions of critical thinking.

Despite these limitations, this study provided promising
results for (economics) educators who wish to enhance their
students’ critical thinking skills, by showing that such skills
can be enhanced through relatively simple explicit in-
structions eprovided they are combined with practice. The
findings are especially promising given that this experimental
study was of relatively short duration; teaching critical
thinking throughout the economics curriculum might help
students to learn to avoid biased reasoning and better prepare
them for decision-making in dynamic and complex business
environments.

Appendix A. Example of each category of tasks used on critical thinking tests (* [ right option; 1 [ category practiced;
2 [ category not practiced).

Conjunction task
(1)

A kitchen manufacturer wants to reposition itself in the kitchen industry. The goal is to
increase the market share to 10%. Based on a strength/weakness analysis, measures have
been taken to improve market share, pay more attention to the entrepreneurship of the
employees, and conduct a more value-oriented communication campaign towards
customers.
Which option is most likely?
Option 1. The market share of the kitchen manufacturer will increase by 3%.*
Option 2: The market share of the kitchen manufacturer will increase by 5%.
Option 3: The market share of the kitchen manufacturer will increase by 3%, and the
satisfaction of customers and employees will improve.
Option 4: The market share of the kitchen manufacturer will increase by 5%, and the
satisfaction of customers and employees will improve.

Explanation: Options 2, 3 and 4 violate the conjunction rule as a conjunction cannot be
more probable than one of its constituents.

Contingency task
(1)

An insurance company claimed that too much time was spent on advising customers
without any results. Costs and benefits were not balanced. Applying a new advising
system should improve this situation. The new system was used on a part of the
customers. A student was assigned to evaluate the effect of the new system to determine
the systems’ time efficiency. The student obtained the information below and concludes
that the application of the new system resulted in saving of time.

New system used New system not used
Saving time 325 260
No time saving 90 55

Table: number of customers on which the system was applied or not and time was saved
or not.
Do you agree with the student?
Option 1. fully agree
Option 2. agree
Option 3. neither agree nor disagree
Option 4. disagree*
Option 5. fully disagree*

Explanation: Option 1, 2 and 3 refer to the tendency to evaluate the information given in
a 2 x 2 contingency table unequally.

Non-causal

base-rate task
(1)

A renowned regional company has two vacancies for junior economists. This company
has very good experiences with economics master graduates from a specific University
for over 15 years. The company management will continue this policy of hiring graduates
from this University. A new personnel manager, however, suggests attracting economics
graduates of a Polytechnic University. The manager argued that he gained outstanding
experiences in a business where he worked formerly with a polytechnic graduate who
functioned at an excellent level from the outset. The personnel manager believes that
polytechnic graduates work equally well as master graduates from the specific
University.
What should the management of the company best decide?
Option 1: Definitely chose for master graduates of the specific University.*
Option 2: Probably chose for master graduates of the specific University.*
Option 3: Probably chose for graduates of the Polytechnic University.
Option 4: Definitely chose for graduates of the Polytechnic University.

Explanation: People who choose option 3 or 4 neglect the base-rate, for example
motivated by personal and case evidence in favor of more representative statistical
evidence.

Causal

base-rate task
(2)

A study had 1000 participants. Among the participants there were 25 men and 975
women. Sam is a randomly chosen participant in this study. Sam is 23 years old,
graduated as mechanical engineer and enjoys going out with friends, likes drinking beer
and is a fan of hard rock music.
Which option is most probable?
Option 1: Sam is a man.
Option 2: Sam is a woman.*

Explanation: Option 1 demonstrates the tendency to base judgments on prior belief and
intuition rather than on logical reasoning (i.e., taking into account the prior probability).

Framing task
(2)

Imagine that you face the following pair of concurrent decisions: First examine both
decisions, then indicate the options you prefer.

A. A sure gain of 480 euro.
B. 25% chance to gain 2000 euro and 75% chance to gain nothing.

C. A sure loss of 1500 euro.
D. 75% chance to lose 2000 and 25% chance to lose nothing.

Options: AC, BC*, AD, BD

Explanation: The paired choices were presented together but the problem was ‘framed’
as a pair of separate choices. The combination B&C is superior: 25% chance to win 500
and 75% chance to lose 1500. For example, with A&D there is 25% chance to win 480
and 75% chance to lose 1520. In case of gains people tend to chose risk aversive (option
A) and in case of losses to chose risk seeking (option D).

Wason selection
task
(1)

Each of the tickets below has a destination on one side and an airline on the other side.
Here is a rule: If ‘Barcelona’ is on one side of the ticket, then ‘Ryanair’ is on the other
side of the ticket. Your task is to decide which tickets you would need to turn over in
order to find out whether or not the rule is being violated.

Destination
Barcelona

Destination
Madrid

Airline
Ryanair

Airline
Transavia

A B C D
Options: AC, AD*, BC, BD

Explanation: People who chose other options than AD probably fail to apply logical
principles, verify rules rather than to falsify them, or demonstrate matching bias by
selecting options explicitly mentioned in the conditional statement.

Syllogistic
reasoning tasks
(2)

The category syllogistic reasoning tasks exist of 4 types of tasks.
In the following assignments, you will be given two premises, which you must assume
are true. A conclusion from the premises then follows. You must decide whether the
conclusion follows logically from the premises or not.

Premises:
All oil companies are quoted companies.
Shell is a quoted company.
Conclusion: Shell is an oil company.
Option 1. Conclusion follows logically from premises.
Option 2. Conclusion does not follow logically from premises.*

Explanation:

Syllogism type: Affirmation of consequent, invalid

Premises:
All mammals walk.
Dolphins are mammals.
Conclusion: Dolphins walk.
Option 1. Conclusion follows logically from premises.*
Option 2. Conclusion does not follow logically from premises.

Explanation:

Syllogism type: Affirmation of antecedent/Modus Ponens, valid

Premises:
All things that move love water.
Cats do not love water.
Conclusion: Cats do not move.
Option 1. Conclusion follows logically from premises.*
Option 2. Conclusion does not follow logically from premises.

Explanation:

Syllogism Type: Denial of consequent/ Modus Tollens, valid

Premises:
All oil countries are rich.
Belgium is not an oil country.
Conclusion: Belgium is not rich.
Option 1. Conclusion follows logically from premises.
Option 2. Conclusion does not follow logically from premises.*

Explanation:

Syllogism type: Denial of antecedent, invalid

A. Heijltjes et al. / Learning and Instruction 29 (2014) 31e42 41

References

Abrami, P. C., Bernard, R. M., Borokhovski, E., Wade, A., Surkes, M. A., Tamim, R., &
Zhang, D. (2008). Instructional interventions affecting critical thinking skills
and dispositions: a stage 1 meta-analysis. Review of Educational Research, 78,
1102e1134. http://dx.doi.org/10.3102/0034654308326084.

Aleven, A. W., & Koedinger, K. R. (2002). An effective meta-cognitive strategy:
learning by doing and explaining with a computer-based cognitive tutor.
Cognitive Science, 26, 147e179. http://dx.doi.org/10.1207/s15516709cog2602_1.

Angeli, C., & Valanides, N. (2009). Instructional effects on critical thinking: perfor-
mance on ill-defined issues. Learning and Instruction, 19, 322e334. http://
dx.doi.org/10.1016/j.learninstruc.2008.06.010.

Austin, Z., Gregory, P. A., & Chiu, S. (2008). Use of reflection-in-action and self-
assessment to promote critical thinking among pharmacy students. American
Journal of Pharmaceutical Education, 72, 1e8.

Bangert-Drowns, R. L., & Bankert, E. (1990). Meta-analysis of explicit instruction for
critical thinking. Boston, MA: Paper presented at the annual meeting of the
American Educational Research Association.

http://dx.doi.org/10.3102/0034654308326084

http://dx.doi.org/10.1207/s15516709cog2602_1

http://dx.doi.org/10.1016/j.learninstruc.2008.06.010

http://dx.doi.org/10.1016/j.learninstruc.2008.06.010

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref4

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref4

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref4

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref4

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref5

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref5

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref5

A. Heijltjes et al. / Learning and Instruction 29 (2014) 31e4242

Baron, J. (2008). Actively open-minded thinking. In J. Baron (Ed.), Thinking and
deciding (pp. 199e232). New York: Cambridge University Press.

Bransford, J., Sherwood, R., Vye, N., & Rieser, J. (1986). Teaching thinking and
problem-solving. American Psychologist, 41, 1078e1089. http://dx.doi.org/
10.1037/0003-066X.41.10.1078.

de Bruin, W., Parker, A. M., & Fischhoff, B. (2007). Individual differences in adult
decion-making competence. Journal of Personality and Social Psychology, 92,
938e956. http://dx.doi.org/10.1037/0022-3514.92.5.938.

De Neys, W., & Glumicic, T. (2008). Conflict monitoring in dual process theories of
thinking. Cognition, 106, 1248e1299. http://dx.doi.org/10.1016/j.cognition.
2007.06.002.

Cacioppo, J. T., Petty, R. E., Feinstein, J. A., Blair, G., & Jarvis, W. (1996). Dispositional
differences in cognitive motivation: the life and times of individuals varying in
need for cognition. Psychological Bulletin, 119, 197e253. http://dx.doi.org/
10.1037/0033-2909.119.2.197.

Cacioppo, J. T., Petty, R. E., & Kao, C. F. (1984). The efficient assessment of need for
cognition. Journal of Personality Assessment, 48, 306e307. http://dx.doi.org/
10.1207/s15327752jpa4803_13.

Cohen, J. (1992). Quantitative methods in psychology; a power primer. Psychological
Bulletin, 112, 155e159. http://dx.doi.org/10.1037/0033-2909.112.1.155.

Evans, J. St. B. T. (2003). In two minds: dual-process accounts of reasoning. Trends in
Cognitive Sciences, 7, 454e459. http://dx.doi.org/10.1016/j.tics.2003.08.012.

Evans, J. St. B. T. (2008). Dual processing account of reasoning, judgment and social
cognition. Annual Review of Psychology, 59, 255e278. http://dx.doi.org/10.1146/
annurev.psych.59.103006.093629.

Evans, J. St. B. T. (2011). Dual-process theories of reasoning: contemporary issues
and developmental applications. Developmental Review, 31, 86e102. http://
dx.doi.org/10.1016/j.dr.2011.07.007.

Facione, P. A. (1990). Critical thinking: a statement of expert consensus for purposes of
educational assessment and instruction. Millbrae, CA: The California Academic
Press.

Facione, P. A. (2009). Critical thinking: what it is and why it counts. Retrieved from
http://www.insightassessment.com/9articles%20WW.html.

Fong, G. T., Krantz, D. H., & Nisbett, R. E. (1986). The effect of statistical training on
thinking about everyday problems. Cognitive Psychology, 18, 253e292. http://
dx.doi.org/10.1016/0010-0285(86)90001-0.

Halpern, D. F. (1998). Teaching critical thinking for transfer across domains: dis-
positions, skills, structure training, and metacognitive monitoring. American
Psychologist, 53, 449e455. http://dx.doi.org/10.1037/0003-066X.53.4.449.

Halpern, D. F. (1999). Teaching for critical thinking: helping college students
develop the skills and dispositions of a critical thinker. New Directions for
Teaching and Learning, 80, 69e74. http://dx.doi.org/10.1002/tl.8005.

Helsdingen, A., Van Gog, T., & Van Merriënboer, J. J. G. (2011). Effects of practice
schedule and critical thinking instruction on learning and transfer of a complex
judgment task. Journal of Educational Psychology, 103, 383e398. http://
dx.doi.org/10.1037/a0022370.

Jones, A. (2007). Multiplicities or manna from heaven? Critical thinking and the
disciplinary context. Australian Journal of Education, 51, 84e103.

Klebba, J. M., & Hamilton, J. G. (2007). Structured case analysis: developing critical
thinking skills in a marketing case course. Journal of Marketing Education, 29,
132e139. http://dx.doi.org/10.1177/0273475307302015.

Larrick, R. P., Morgan, J. N., & Nisbett, R. E. (1990). Teaching the use of cost-benefits
reasoning in everyday life. Psychological Science, 1, 362e370. http://dx.doi.org/
10.1111/j.1467-9280.1990.tb00243.x.

Lehman, D. R., Lempert, R. O., & Nisbett, R. E. (1988). The effects of graduate training
on reasoning. American Psychologist, 43, 431e442. http://dx.doi.org/10.1037/
0003-66X.43.6.431.

Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. London: Sage.
Lombrozo, T. (2006). The structure and function of explanations. Trends in Cognitive

Sciences, 10, 464e470. http://dx.doi.org/10.1016/j.tics.2006.08.004.
Macpherson, R., & Stanovich, K. E. (2007). Cogitive ability, thinking dispositions, and

instructional set as predictors of critical thinking. Learning and Individual Dif-
ferences, 17, 115e127. http://dx.doi.org/10.1016/j.lindif.2007.05.003.

Markovits, H., & Nantel, G. (1989). The belief-bias effect in the production and
evaluation of logical conclusions. Memory & Cognition, 17, 11e17. http://
dx.doi.org/10.3758/BF03199552.

Matthews, P., & Rittle-Johnson, B. (2009). In pursuit of knowledge: comparing self-
explanations concepts and procedures as pedagogical tools. Journal of Experi-
mental Child Psychology, 102, 1e21. http://dx.doi.org/10.1016/j.jecp.2008.08.004.

McMillan, J. H. (1987). Enhancing college students’ critical thinking: a review of
studies. Research in Higher Education, 26, 3e29. http://dx.doi.org/10.1007/
BF00991931.

Miyake, A., Friedman, N. P., Emerson, M. J., Witzki, A. H., & Howerter, A. (2000). The
unity and diversity of executive functions and their contributions to complex
’frontal lobe’ tasks: a latent variable analysis. Cognitive Psychology, 41, 49e100.
http://dx.doi.org/10.1006/cogp.1999.0734.

Moutier, S., Angeard, N., & Houdé, O. (2002). Deductive reasoning and matching-
bias inhibition training: evidence from a debiasing paradigm. Thinking and
Reasoning, 8, 205e224. http://dx.doi.org/10.1080/13546780244000033.

Moutier, S., & Houdé, O. (2003). Judgement under uncertainty and conjunction
fallacy inhibition training. Thinking and Reasoning, 9, 185e201. http://dx.doi.org/
10.1080/13546780343000213.

Nisbett, R. E., Fong, G. T., Lehman, D. R., & Cheng, P. W. (1987). Teaching reasoning.
Science, 238, 625e631. http://dx.doi.org/10.1126/science.3672116.

Paas, F. (1992). Training strategies for attaining transfer of problem solving skills in
statistics: a cognitive-load approach. Journal of Educational Psychology, 84, 429e
434. http://dx.doi.org/10.1037/0022-0663.84.4.429.

Renkl, A. (1997). Learning from worked-out examples: a study on individual differ-
ences. Cognitive Science, 21, 1e29. http://dx.doi.org/10.1207/s15516709cog2101_1.

Renkl, A. (2002). Worked-out examples: instructional explanations support
learning by self-explanations. Learning and Instruction, 12, 529e556. http://
dx.doi.org/10.1016/S0959-4752(01)00030-5.

Renkl, A. (2005). The worked-out principle in multi-media learning. In R. E. Mayer
(Ed.), The Cambridge handbook of multimedia learning (pp. 229e246). Cam-
bridge: Cambridge University Press.

Ritchart, R., & Perkins, D. N. (2005). Learning to think: the challenges of teaching
thinking. In K. J. Holyoak, & R. G. Morrison (Eds.), The Cambridge handbook of
thinking and reasoning (pp. 775e802). New York: Cambridge University Press.

Rittle-Johnson, B. (2006). Promoting transfer: effects of self-explanation and direct
instruction. Child Development, 77, 1e15. http://dx.doi.org/10.1111/j.1467-
8624.2006.00852.x.

Roy, M., & Chi, M. T. (2005). The self-explanation principle in multimedia learning.
In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 271e
286). Cambridge: Cambridge University Press.

Sa, W., Kelley, C., Ho, C., & Stanovich, K. E. (2005). Thinking about personal theories:
individual differences in the coordination of theory and evidence. Personality
and Individual Differences, 38, 1149e1161. http://dx.doi.org/10.1016/j.paid.2004.
07.012.

Sa, W. C., West, R. F., & Stanovich, K. E. (1999). The domain specificity and generality of
belief bias: searching for a generalizable critical thinking skill. Journal of Educa-
tional Psychology, 91, 497e510. http://dx.doi.org/10.1037/0022-0663.91.3.497.

Sanfey, A. G., Loewenstein, G., McClure, S. M., & Cohen, J. D. (2006). Neuro-
economics: cross-currents in resaerch on decision-making. Trends in Cognitive
Sciences, 10, 108e116. http://dx.doi.org/10.1016/j.tics.2006.01.009.

Schagen, I., & Elliot, K. (2004). But what does it mean? The use of effect sizes in
educational research. London: National Foundation for Educational Research.

Smith, G. F. (2003). Beyond critical thinking and decision making: teaching business
students how to think. Journal of Management Education, 27, 24e51. http://
dx.doi.org/10.1177/1052562902239247.

Stanovich, K. E. (2009). Distinguishing the reflective algorithmic and autonomous
minds: is it time for a tri-process theory? In J. Evans, & K. Frankish (Eds.), In two
minds: Dual processess and beyond (pp. 55e88) Oxford: Oxford University Press.

Stanovich, K. E. (2011). Rationality & the reflective mind. New York: Oxford Univer-
sity Press.

Stanovich, K. E., & Stanovich, P. J. (2010). A frame-work for critical thinking, rational
thinking and intelligence. In D. D. Preiss, & R. J. Sternberg (Eds.), Educational
psychology; Perspectives on learning, teaching and human development (pp. 195e
237). New York: Springer Publishing Company.

Stanovich, K. E., Toplak, M. E., & West, R. F. (2008). The development of rational
thought: a taxonomy of heuristic and biases. Advances in Child Development and
Behavior, 36, 251e285. http://dx.doi.org/10.1016/s0065-2407(08)000062.

Stanovich, K. E., & West, R. F. (1997). Reasoning independently of prior belief and
individual differences in actively open-minded thinking. Journal of Educational
Psychology, 89, 342e357. http://dx.doi.org/10.1037/0022-0663.89.2.342.

Stanovich, K. E., & West, R. F. (2007). Natural myside bias is independent of
cognitive ability. Thinking and Reasoning, 13, 225e247. http://dx.doi.org/
10.1080/13546780600780796.

Ten Dam, G., & Volman, M. (2004). Critical thinking as a citizenship competence:
teaching strategies. Learning and Instruction, 14, 359e379. http://dx.doi.org/
10.1016/j.learninstruc.2004.01.005.

Tversky, A., & Kahneman, D. (1981). The framing of decisions and the psychology of
choice. Science, 211, 453e458. http://dx.doi.org/10.1126/science.7455683.

Tversky, A., & Kahneman, D. (1983). Extensional versus intuitive reasoning: the
conjunction fallacy in probability judgment. Psychological Review, 90, 293e315.
http://dx.doi.org/10.1037/0033-295X.90.4.293.

Van Gog, T., & Paas, F. (2008). Instructional efficiency: revisiting the original
construct in educational research. Educational Psychologist, 43, 16e26. http://
dx.doi.org/10.1080/00461520701756248.

Wason, P. C., & Shapiro, D. (1971). Natural and contrived experience in a reasoning
problem. Quarterly Journal of Experimental Psychology, 23, 63e71. http://
dx.doi.org/10.1080/00335557143000068.

Wasserman, E. A., Dorner, W. W., & Kao, S. F. (1990). Contributions of specific cell
information to judgments of interevent contingency. Journal of Experimental
Psychology: Learning, Memory, and Cognition, 16, 509e521. http://dx.doi.org/
10.1037/0278-7393.16.3.509.

Wenzlaff, R. M., & Bates, D. E. (2000). The relative efficacy of concentration and
suppression strategies of mental control. Personality and Social Psychology
Bulletin, 26, 1200e1212. http://dx.doi.org/10.1177/0146167200262003.

West, R. F., Toplak, M. E., & Stanovich, K. E. (2008). Heuristics and biases as measures
of critical thinking: associations with cognitive ability and thinking disposi-
tions. Journal of Educational Psychology, 100, 930e941. http://dx.doi.org/10.1037/
a0012842.

Williams, J. J., & Lombrozo, T. (2010). The role of explanation in discovery and
generalization: evidence from category learning. Cognitive Science, 34, 776e806.
http://dx.doi.org/10.1111/j.1551-6709.2010.01113.x.

Wolcott, S. K., Baril, C. P., Cunningham, B. M., Fordham, D. R., & St.Pierre, K. (2002).
Critical thought on critical thinking research. Journal of Accounting Education,
20, 85e103. http://dx.doi.org/10.1016/S0748-5751(01)00029-X.

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref6

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref6

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref6

http://dx.doi.org/10.1037/0003-066X.41.10.1078

http://dx.doi.org/10.1037/0003-066X.41.10.1078

http://dx.doi.org/10.1037/0022-3514.92.5.938

http://dx.doi.org/10.1016/j.cognition.2007.06.002

http://dx.doi.org/10.1016/j.cognition.2007.06.002

http://dx.doi.org/10.1037/0033-2909.119.2.197

http://dx.doi.org/10.1037/0033-2909.119.2.197

http://dx.doi.org/10.1207/s15327752jpa4803_13

http://dx.doi.org/10.1207/s15327752jpa4803_13

http://dx.doi.org/10.1037/0033-2909.112.1.155

http://dx.doi.org/10.1016/j.tics.2003.08.012

http://dx.doi.org/10.1146/annurev.psych.59.103006.093629

http://dx.doi.org/10.1146/annurev.psych.59.103006.093629

http://dx.doi.org/10.1016/j.dr.2011.07.007

http://dx.doi.org/10.1016/j.dr.2011.07.007

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref16

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref16

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref16

http://www.insightassessment.com/9articles%2520WW.html

http://dx.doi.org/10.1016/0010-0285(86)90001-0

http://dx.doi.org/10.1016/0010-0285(86)90001-0

http://dx.doi.org/10.1037/0003-066X.53.4.449

http://dx.doi.org/10.1002/tl.8005

http://dx.doi.org/10.1037/a0022370

http://dx.doi.org/10.1037/a0022370

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref22

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref22

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref22

http://dx.doi.org/10.1177/0273475307302015

http://dx.doi.org/10.1111/j.1467-9280.1990.tb00243.x

http://dx.doi.org/10.1111/j.1467-9280.1990.tb00243.x

http://dx.doi.org/10.1037/0003-66X.43.6.431

http://dx.doi.org/10.1037/0003-66X.43.6.431

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref26

http://dx.doi.org/10.1016/j.tics.2006.08.004

http://dx.doi.org/10.1016/j.lindif.2007.05.003

http://dx.doi.org/10.3758/BF03199552

http://dx.doi.org/10.3758/BF03199552

http://dx.doi.org/10.1016/j.jecp.2008.08.004

http://dx.doi.org/10.1007/BF00991931

http://dx.doi.org/10.1007/BF00991931

http://dx.doi.org/10.1006/cogp.1999.0734

http://dx.doi.org/10.1080/13546780244000033

http://dx.doi.org/10.1080/13546780343000213

http://dx.doi.org/10.1080/13546780343000213

http://dx.doi.org/10.1126/science.3672116

http://dx.doi.org/10.1037/0022-0663.84.4.429

http://dx.doi.org/10.1207/s15516709cog2101_1

http://dx.doi.org/10.1016/S0959-4752(01)00030-5

http://dx.doi.org/10.1016/S0959-4752(01)00030-5

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref39

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref39

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref39

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref39

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref40

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref40

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref40

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref40

http://dx.doi.org/10.1111/j.1467-8624.2006.00852.x

http://dx.doi.org/10.1111/j.1467-8624.2006.00852.x

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref42

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref42

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref42

http://dx.doi.org/10.1016/j.paid.2004.07.012

http://dx.doi.org/10.1016/j.paid.2004.07.012

http://dx.doi.org/10.1037/0022-0663.91.3.497

http://dx.doi.org/10.1016/j.tics.2006.01.009

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref46

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref46

http://dx.doi.org/10.1177/1052562902239247

http://dx.doi.org/10.1177/1052562902239247

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref48

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref48

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref48

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref48

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref49

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref49

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref50

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref50

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref50

http://refhub.elsevier.com/S0959-4752(13)00058-3/sref50

http://dx.doi.org/10.1016/s0065-2407(08)000062

http://dx.doi.org/10.1037/0022-0663.89.2.342

http://dx.doi.org/10.1080/13546780600780796

http://dx.doi.org/10.1080/13546780600780796

http://dx.doi.org/10.1016/j.learninstruc.2004.01.005

http://dx.doi.org/10.1016/j.learninstruc.2004.01.005

http://dx.doi.org/10.1126/science.7455683

http://dx.doi.org/10.1037/0033-295X.90.4.293

http://dx.doi.org/10.1080/00461520701756248

http://dx.doi.org/10.1080/00461520701756248

http://dx.doi.org/10.1080/00335557143000068

http://dx.doi.org/10.1080/00335557143000068

http://dx.doi.org/10.1037/0278-7393.16.3.509

http://dx.doi.org/10.1037/0278-7393.16.3.509

http://dx.doi.org/10.1177/0146167200262003

http://dx.doi.org/10.1037/a0012842

http://dx.doi.org/10.1037/a0012842

http://dx.doi.org/10.1111/j.1551-6709.2010.01113.x

http://dx.doi.org/10.1016/S0748-5751(01)00029-X

• Improving critical thinking: Effects of dispositions and instructions on economics students’ reasoning skills

1 Introduction
1.1 Critical thinking
1.2 Critical thinking instruction: avoiding biased reasoning
1.2.1 Self-explanation prompts
1.2.2 Activation prompts
1.3 The present study
2 Method
2.1 Participants and design
2.2 Materials and procedure
2.2.1 Critical thinking disposition tests
2.2.2 Critical thinking skills tests
2.2.3 Critical thinking instructions
2.2.4 Practice phase
2.2.5 Procedure
2.2.6 Data analysis

3 Results
3.1 Dispositions and pre-test performance
3.2 Effects of instructions and dispositions on immediate and delayed post-test performance
4 Discussion
Appendix A Example of each category of tasks used on critical thinking tests (*=right option; 1=category practiced; 2=categ …
References

H

E

a

b

a

A

R
R

A
A

K
C
C
C
L
L

1

c

W

i

o
i

M

c
e

1

Thinking Skills and Creativity 21 (2016) 41–49

Contents lists available at ScienceDirect

Thinking Skills and Creativity

j o u r n a l h o m e p a g e : h t t p : / / w w w . e l s e v i e r . c o m / l o c a t e / t s c

ow might language affect critical thinking performance?

mmanuel Manalo a,∗, Chris Sheppard b

Graduate School of Education, Kyoto University, Kyoto, Japan
Center for English Language Education, Faculty of Science and Engineering, Waseda University, Tokyo, Japan

r t i c l e i n f o

rticle history:
eceived 27 July 2015
eceived in revised form 29 March 2016
ccepted 11 May 2016
vailable online 12 May 2016

eywords:
ritical thinking skills instruction
ritical evaluation
ognitive cost
anguage proficiency
anguage structure

a b s t r a c t

This study examined whether language structure or language proficiency might affect
students’ critical thinking performance. Previous research has claimed that many non-
Western students struggle with the demands of demonstrating critical thought. Two
language-related causes have been suggested: one concerning structural limitations in the
non-Western students’ first language, and the other concerning their second language pro-
ficiency. In Study 1 described here, reports written by 110 Japanese second year university
students, who had received instruction in academic discourse for critical evaluation (which
is one aspect of critical thinking), were analyzed for use of evaluative statements. No dis-
advantage was found for use of the Japanese language, which is considered as having a
more indirect structure that may make critical evaluation more difficult. Measurements of
language proficiency in English and Japanese, however, were found to correlate with pro-
duction of evaluative statements in those respective languages suggesting that language
proficiency could affect critical evaluation use. In Study 2, the same task was given to 4

3

first year students who had not yet received the same instruction. Analysis revealed similar
patterns in their written work but at a lower level, suggesting that the second year students
had benefitted from the skills instruction. Furthermore, unlike the second year students,
the first year students evidenced no correlations between their language proficiency scores
and their production of evaluative statements, suggesting that proficiency on its own is
inadequate: students need instruction on the specific language forms and structures to use
to demonstrate critical thinking in their written work.

© 2016 Elsevier Ltd. All rights reserved.

. Introduction

In modern societies, the ability to evaluate the credibility of information that one encounters is an essential skill, especially
onsidering the proliferation of unvetted information through the Internet and other forms of mass media (e.g., Glassner,

einstock, & Neuman, 2005; Thomm & Bromme, 2011). However, educational development of critical thinking skills, which
ncludes critical evaluation, is not a straightforward matter. There are, for example, disagreements about appropriate meth-
ds for critical thinking skills instruction and assessment, and various individual and situational factors are believed to
nfluence the development and use of such skills (e.g., Davies, 2006; Halpern, 1998; Ku, 2009; Manalo, Kusumi, Koyasu,

ichita, & Tanaka, 2013, 2015; Ten Dam & Volman, 2004). One of the main areas of contention is whether cultural background
ontributes to differences in critical thinking performance (e.g., Ennis, 1998; Manalo et al., 2013, 2015). Some authors, for
xample, have portrayed Asian students as being deficient in critical thinking compared to Western students (e.g., Atkinson,

∗ Corresponding author at: Graduate School of Education, Kyoto University, Yoshida-Honmachi, Sakyo-ku, Kyoto 606-8501, Japan.
E-mail address: manalo.emmanuel.3z@kyoto-u.ac.jp (E. Manalo).

http://dx.doi.org/10.1016/j.tsc.2016.05.005
871-1871/© 2016 Elsevier Ltd. All rights reserved.

dx.doi.org/10.1016/j.tsc.2016.05.005

http://www.sciencedirect.com/science/journal/18711871

http://www.elsevier.com/locate/tsc

http://crossmark.crossref.org/dialog/?doi=10.1016/j.tsc.2016.05.005&domain=pdf

mailto:manalo.emmanuel.3z@kyoto-u.ac.jp

dx.doi.org/10.1016/j.tsc.2016.05.005

42 E. Manalo, C. Sheppard / Thinking Skills and Creativity 21 (2016) 41–49

1997; Fox, 1994), and it has been found that many instructors at the tertiary level subscribe to such a view (e.g., Lee &
Carrasquillo, 2006; Robertson, Line, Jones, & Thomas, 2000).

Two language-related explanations have been proposed for the apparent differences in critical thinking performance
manifested by students from different cultural backgrounds. One of those explanations concerns the structure of the student’s
native language (L1). This explanation posits that, due to their structure, some languages may present constraints in the ease
with which certain thinking skills can be undertaken or expressed. This explanation is sometimes referred to as the “Sapir-
Whorf hypothesis” or the “linguistic relativity hypothesis” (see, e.g., Au, 1983; Hill & Mannheim, 1992; Hockett, 1954; Whorf,
1956). An example of a claim of this kind is Bloom’s (1981) proposal that counterfactual thinking (i.e., thinking about what
might have been, contrary to facts) may be more difficult in Chinese compared to English.

Some observations of linguistic differences, such as “indirectness” being a feature more prevalent in some languages,
particularly Asian languages (e.g., Kong, 2005), would appear to support the idea that language structure could affect the ease
with which certain modes of thinking could be undertaken or expressed. In a study by Itakura and Tsui (2011), for example,
evidence was found that book reviewers use different strategies to convey critical evaluation when writing in Japanese
compared to English: in Japanese, criticism is usually indirectly conveyed and is frequently preceded by an apology. Previous
studies, however, have failed to provide convincing evidence that language structures could actually impose constraints in
what users of the language can communicate. Although the earlier-mentioned study by Bloom (1981) claimed to have found
evidence for such constraints where counterfactual thinking in the Chinese language is concerned, subsequent investigations
failed to replicate or support Bloom’s results (Au, 1983; Liu, 1985; Takano, 1989; Vorster & Schuring, 1989). Even where
the weaker version of the linguistic relativity hypothesis is concerned (i.e., that language can merely influence people’s
thought patterns, rather than constrain their thought processes), the findings of the few studies that claim to have found
supporting evidence (e.g., Boroditsky, 2001) have not been replicated in subsequent independent investigations (e.g., Chen,
2007; January & Kako, 2007; Tse & Altarriba, 2008).

The other language-related explanation for the apparent culture-based differences in critical thinking performance con-
cerns students’ second language proficiency (e.g., Floyd, 2011; Lun, Fischer, & Ward, 2010; Paton, 2005). This explanation is
based on the observation that the students who are usually reported as having lower levels of critical thinking competence
are international students in English speaking countries. It suggests that, because most of those students have to use a sec-
ond language (L2, which is usually English) in their host environment, and they may lack adequate proficiency in that L2,
they would likely manifest lower competence when performing tasks that require use of that L2. Tasks that are likely to get
affected include cognitive tasks like critical thinking.

One way of understanding this possible influence of language proficiency on critical thinking performance is in terms
of cognitive cost (i.e., the mental resources cost associated with task performance). Language processing entails the use of
cognitive resources in working memory (Baddeley, 1986, 1998), and lower proficiency in a language would require the use
of more resources. Thinking critically would likewise require the use of working memory resources. There are, however,
limited resources available in working memory (Baddeley, 1986, 1998) and, if a considerable amount of those resources has
already been expended on utilizing a language in which proficiency is low, there may not be adequate resources remaining
for the satisfactory execution of critical thinking.

The negative impact of the higher cognitive cost entailed in using a language in which proficiency is low, on the execution
of other cognitive tasks, has been demonstrated in previous research. Takano and Noda (1993, 1995) showed that the use
of a foreign language detrimentally affects performance in concurrently undertaken non-linguistic tasks like arithmetic
calculation and mental imagery, and Manalo and Uesaka (2012, 2014) reported evidence indicating that students’ lower
proficiency in an L2 limits their ability to use diagrams when explaining information in that L2. Where critical thinking
is concerned, both Lun et al. (2010) and Floyd (2011) reported indications that lower proficiency levels in English could
detrimentally affect Asian students’ performance in critical thinking tests administered in English. However, neither of those
studies used appropriate, objective measures of L2 proficiency to reliably confirm the connection between L2 proficiency
and critical thinking skills performance.

1.1. Overview of the present study

The present study examined whether there might be evidence to support either or both (i) the language structure expla-
nation, and/or (ii) the language proficiency explanation, in students’ manifestation of critical thinking in their written work.
The study was not intended to be a comprehensive test of the language structure hypothesis: it examined only whether, in
the written work of Japanese university students, there might be observable differences in the presence of critical thinking
qualities depending on the language used, Japanese or English. Critical thinking was operationalized as students’ use of
evaluative statements. This decision was based on the fact that use of evaluative statements comprises a salient expression
of critical evaluation, which in turn is central to the notion of critical thinking application (e.g., Fisher & Scriven’s, 1997, p.
21, definition of critical thinking as “skilled and active interpretation and evaluation of observations and communications,
information and argumentation” – italics added).

In the present study, Japanese was deemed an appropriate language to examine because, like a number of other Asian
languages, it employs patterns of expression that make it more indirect and inductive compared to English (e.g., Itakura &
Tsui, 2011; Scollon & Wong-Scollon, 1991). Evaluation, however, requires precision and directness in conveying judgments
about the quality or value of the subject being referred to. Thus, structural features of the Japanese language could make

t
l

m

O

r
t
w
a
b
t
a
h
b

e
t
–
a

h
t
p
r
l
n
t

u
i
a
l
s
l
t
a

2

E
o

2
2

F
c
d

2

e
r
c

t
w
a
t

E. Manalo, C. Sheppard / Thinking Skills and Creativity 21 (2016) 41–49 43

he production of evaluative language relatively more difficult. If so, it should be possible to detect lower rates of evaluative
anguage use in the students’ written work in Japanese compared to English.

It was equally important to consider whether using an L2 might detrimentally affect students’ critical evaluation perfor-
ance. If this was the case, one would expect that critical evaluation performance in the L1 would exceed that in the L2.
ne would also expect to find a relationship between the critical evaluation performance and L2 proficiency. Thus, possible

elationships between students’ L2 proficiency and their production of evaluative statements were investigated. The ques-
ion was whether L2 proficiency would manifest as a limiting factor because lower proficiency entails higher cognitive cost
hen using the L2, leaving insufficient resources in working memory for critical evaluation. If this explanation is supported,

relationship should be found between the students’ L2 proficiency and their evaluative statements production in the L2,
ut not in the L1. A relationship in both the L1 and the L2 would suggest that general language or intellectual abilities – rather
han L2 proficiency – affect critical evaluation performance. The reason is that language abilities, and intellectual abilities
nd performance, are generally considered as being related (e.g., Ackerman, 1986; Neisser et al., 1996). Thus, a student with
igher language and intellectual abilities could be expected to score higher in measurements of L2 proficiency, and evidence
etter performance in tasks like critical evaluation – in both their L1 and L2.

Furthermore, it is conceivable that this kind of relationship between language proficiency and critical evaluation also
xists in the L1. In other words, that the more proficient or skilled a student might be in using his or her native language,
he less cognitive resources such use might demand, and the more resources might be made available for critical evaluation

which could then be better executed. Thus, in the present study, the possible existence of this relationship in the L1 was
lso investigated.

When considering the development of critical evaluation skills through instruction, if the language structure explanation
olds, one would expect that skills instruction provided in the L2 would not result in any discernible skills improvement in
he L1 (i.e., since the L1 has inherent structural constraints on the use or expression of those skills). In contrast, if the language
roficiency explanation holds, one could expect that skills instruction provided in the L2 – where proficiency is lower – would
esult in at least some discernable transfer to the L1, where proficiency is higher and more cognitive processing resources are
ikely to be available for the application of those skills. Critical evaluation skills are thinking skills and its acquisition should
ot be bound to the language of instruction. The important question is whether constraints or limitations in the language
o use could detrimentally affect critical evaluation performance.

The research reported here comprised two related studies. In Study 1, evaluative statements that second year Japanese
niversity students produced in Japanese (their L1) and in English (their L2) were examined. These students had received

nstruction on academic discourse. Thus, they were not naïve as to the requirements of expressing evaluative language,
nd any differences in the writing they produced in L1 and L2 could be attributed to either the inherent structure of the
anguage they were using or their proficiency in using that language. In Study 2, the same writing task was given to first year
tudents who had received little instruction on academic discourse, and nothing explicit on the production of evaluative
anguage. The purpose of this second study was to find out if the characteristics of L1 and L2 written work produced by
he first year students, compared to their second year counterparts, differed. Hence, the aim was to find out whether the
dditional instruction that had been received by the more advanced second year students might have made a difference.

. Study 1

Study 1 tested two hypotheses. The first was that students’ production of evaluative statements in Japanese and in
nglish would differ. The second was that the students’ proficiency in a particular language would be related to the amount
f evaluative statements they produce in that language.

.1. Method

.1.1. Participants
The participants were 110 Japanese university students in their second year of study in science and engineering disciplines.

or these students, Japanese is L1 and English is L2. They were taking a compulsory English communication skills development
ourse that covers oral and written academic discourse using a task-based learning approach. The students came from four
ifferent classes in that course.

.1.2. Materials and procedure
As part of the communication skills course, the students were provided class instruction, textbook explanations and

xamples (Anthony, Rose, & Sheppard, 2010), and practice in the use of language appropriate for critical evaluation, including
anking and debating different reasons and other forms of alternatives (e.g., clearly stating the premises, and then drawing
onclusions). These materials and the instruction were all provided in English.

For the purposes of the present investigation, the students were additionally provided with a single page Japanese

ranslation of the part of the textbook dealing with how to make valid arguments. They were also supplied brief (one page)
ritten examples (one in English and one in Japanese) of how alternative reasons could be ranked according to judgments

bout their relative importance. The example texts conveyed someone’s opinion about the most important reason for learning
he English language, among four possible reasons. The texts provided examples of evaluative statements and provision of

44 E. Manalo, C. Sheppard / Thinking Skills and Creativity 21 (2016) 41–49

support for claims, although those were not labeled or overtly identified in any way in the texts. The equivalence and
appropriate use of language in the English and Japanese versions were checked by several bilingual teachers of the course.
Although all materials provided in the course are usually in English, the Japanese versions were supplied in this case to avoid
possible disadvantage to the students’ production of evaluative language in Japanese (i.e., without the Japanese versions, it
could be argued that the students might have simply been unfamiliar with the equivalent Japanese expressions for critical
evaluation).

During two consecutive 90-min class sessions of the course, the students were introduced to the Titanic and Space Shuttle
Challenger disasters, including four basic causes that have been proposed for the occurrence of each of those disasters. During
the class sessions, the students participated in guided exercises to explore and discuss the disasters and their corresponding
possible causes.

For homework, the students were asked to write two brief reports to explain what they considered to be the most
important cause of each of the disasters. To avoid any possible misunderstandings about the requirements of the homework
task, written instructions were provided in Japanese. The students were randomly assigned to write one report in English
and the other in Japanese (i.e., if they were asked to write the Titanic report in English, they had to write the Challenger
report in Japanese, and vice versa).

2.1.3. Analyses
The total number of sentences [Total], and the number of evaluative sentences (i.e., sentences where some evaluation

of the relative value of the topic is made) [Evaluative], were counted and scored in the analysis of the students’ written
work.

Operational criteria were drawn up for determining what data counted as “evaluative sentences”. Examples of such
requirements included: the sentence must explicitly say something about the worth or value of the subject, and that worth
or value must be in comparison to something else. Conditional statements that explicitly convey a relative evaluation of the
subject were counted. The following examples, in contrast, did not count: the use of simple adjectives or adverbs to describe
something, prescriptive statements not explicitly expressing a relative evaluation or judgment, and conditional statements
in general. Inter-rater reliability was checked by asking an independent coder to score a randomly selected sample of 25% of
the data. Reliability coefficients obtained (Cronbach’s alphas) were deemed to be satisfactory: they were 0.922 in English,
and 0.960 in Japanese.

Language proficiency was measured (i) by using the students’ TOEIC-IP (Test of English for International Communication
– Institution Program) scores, and (ii) in terms of the complexity of the sentences the students produced in English and in
Japanese (explained below). TOEIC is a norm-referenced test of English listening comprehension and reading skills, widely
used as a measure of students’ English language proficiency levels in Japan (http://www.ets.org/toeic). Although there is
a speaking and writing test version, it is not as widely used as the listening and reading test version, which is what the
TOEIC-IP contains. Students in the faculty where the students came from are required to sit the TOEIC-IP test at regular
intervals during their period of enrolment, and their scores on that test are made available to their teachers. The TOEIC test
has been used in numerous previous studies as a measure of L2 proficiency (e.g., Dornyei & Katona, 1992; Eggly, Musial,
& Smulowitz, 1999), and various studies have reported that it has at least adequate reliability and validity (e.g., In’nami
& Koizumi, 2011; Perkins, 1987; Powers, Kim, & Weng, 2008). The TOEIC is also considered a good measure of non-native
English speakers’ ability to use English in real life situations rather than just in strictly academic contexts (Eggly et al., 1999).
For these reasons, the authors of the present study decided to use the students’ TOEIC-IP scores as one measurement of their
English proficiency.

The other measure of proficiency used – complexity – was selected because, along with accuracy and fluency, it is
generally considered as one of the key dimensions of proficiency in language production – both oral and written (e.g.,
Ellis, 2009; Ishikawa, 1995; Ortega, 2003; Skehan & Foster, 1999; Wolfe-Quintero, Inagaki, & Kim, 1998). However, unlike
accuracy and fluency, for which it is difficult to come up with meaningful and equivalent/comparable measurements
for English and Japanese, the authors of the present study were able to determine a viable way to measure complexity.
Usually, analysis of production complexity in English entails calculating the ratio of words over sentences or terminable
units (T-units; e.g., Ishikawa, 1995; Ortega, 2003; Richards, Platt, & Platt, 1992). However, what counts as a word in
Japanese is not exactly the same as in English, and thus, such a method for calculating complexity was deemed inap-
propriate. Instead, the number of verbs that the students used were counted and divided by their total number of
sentences to generate a measure of complexity. Verbs were chosen as they are comparable in English and Japanese,
and the use of more verbs can be considered a good indicator of more complex sentence structures (i.e., simple sen-
tences usually contain only one verb, and inclusion of more verbs within one sentence usually requires the use of a
more complex structure). It should also be noted that even though this measurement of complexity and the measure-
ment of the students’ use of critical evaluation were derived from the same source (i.e., the writing that the students
produced), these measurements were not dependent on each other. In other words, students can produce complex
writing without being critically evaluative, and vice versa: they can write evaluative sentences that are low in complex-

ity.

Analyses of variance were conducted to compare the students’ production of sentences (Total, and Evaluative) in English
and in Japanese. Correlational analyses were carried out to examine possible relationships with the students’ most recent
TOEIC test scores and their complexity scores in English and in Japanese.

http://www.ets.org/toeic

E. Manalo, C. Sheppard / Thinking Skills and Creativity 21 (2016) 41–49 45

Table 1
Means (with standard deviations in brackets) of total and evaluative sentences that the second year students produced in English and in Japanese.

Total Evaluative

English 20.31 (5.59) 3.46 (1.76)
Japanese 18.68 (5.92) 3.75 (1.80)

Table 2
Correlation coefficients between second year students’ TOEIC and writing complexity scores, and numbers and proportions of evaluative sentences they
produced, according to the language used (effect sizes, R2 shown in brackets).

Scores English Evaluative Sentences Japanese Evaluative Sentences

Number Proportion Number Proportion

TOEIC 0.23* (0.053) 0.14 (0.020) 0.18 (0.032) −0.06 (0.004)
English Complexity 0.19* (0.036) 0.24* (0.058) 0.15 (0.023) 0.17 (0.029)
Japanese Complexity 0.02 (0.000) 0.09 (0.008) −0.07 (0.005) 0.33*** (0.109)

*

*
*
2
i

c
e
t

m
J
l
1
t

p
t
i
t

E
t
b
o
c
s
o
b
t

E
e

2

E
h
w

l
i

p < 0.05. * p < 0.01. ** p < 0.001.

.2. Results

Table 1 shows the means, and standard deviations (in brackets), obtained for Total and Evaluative categories of sentences
n the students’ written work in English and in Japanese.

Repeated measures 2 × 2 ANOVAs demonstrated that there were no significant effects due to the task (i.e., the Titanic
ompared to the Challenger reports), hence demonstrating task equivalence. The analysis however revealed significant
ffects due to language in both the total number of sentences written [Total], F(1, 108) = 11.47, p = 0.001, �p2 = 0.020; and
he number of evaluative sentences [Evaluative], F(1, 108) = 4.81, p = 0.030, �p2 = 0.006.

These results indicate that the students wrote significantly more sentences in English compared to Japanese, but produced
ore of the target evaluative language in Japanese. The ANOVA results also revealed that the students wrote more verbs in

apanese (mean = 48.52, SD = 14.01) than in English (mean = 38.77, SD = 12.54), F(1, 109) = 84.56, p < 0.001, �p2 = 0.119. The anguage produced was also significantly more complex in Japanese as measured by the number of verbs per sentence (F(1, 09) = 100.8, p < 0.001, �p2 = 0.269), indicating that the reason for the production of fewer sentences in Japanese was because he students’ greater proficiency in that language enabled them to produce more complex sentences.

Because the total number of sentences that the students produced differed considerably between the two languages, the
roportions of evaluative sentences (i.e., Evaluative as a proportion of Total) were also calculated and compared according
o the language used. The comparison revealed a significant difference: F(1, 109) = 20.32, p < 0.001, �p2 = 0.039. This result ndicates that the proportion of evaluative sentences was higher in the reports that the students wrote in Japanese compared o those they wrote in English.

For the correlational analysis, the correlations between the students’ TOEIC scores and their writing complexity scores in
nglish and in Japanese, and the evaluative sentences they produced in the respective languages (actual and proportions of
otal), were examined. The results are shown in Table 2. The students’ TOEIC scores correlated significantly with the num-
er of evaluative sentences they produced in English. The correlation between the students’ TOEIC scores and the number
f evaluative sentences they produced in Japanese was not significant. Where the complexity scores were concerned, the
omplexity of the students’ writing in English significantly correlated with both the number and proportion of evaluative
entences they produced in English. The complexity of their writing in Japanese significantly correlated with the proportion
f evaluative sentences they produced in Japanese. All cross language correlations between writing complexity and num-
ers/proportions of evaluative sentences were not significant. Taken together, these results lend support to the argument
hat language proficiency influences students’ ability to evidence critical evaluation in their writing.

A significant correlation was also found between the proportions of evaluative sentences the students produced in
nglish and in Japanese, r = 0.72, p < 0.001, R2 (effect size) = 0.518. This result suggests that students who were more critically valuative in one language also demonstrated greater critical evaluation in the other language.

.3. Discussion

Differences were found in both actual numbers and proportions of evaluative sentences that the students produced in
nglish and in Japanese. The direction of the differences, however, was opposite to the language structure-based prediction:
igher proportions of evaluative sentences were found in Japanese instead of English. This result suggests that the students

ere better at producing evaluative language in their L1.

The significant relationship found between the students’ language proficiency and their critical evaluation performance
ends support to the notion that language proficiency could explain apparent cultural differences in students’ critical think-
ng performance. When using an L2 (such as English) in which they may lack adequate proficiency (e.g., less automated

46 E. Manalo, C. Sheppard / Thinking Skills and Creativity 21 (2016) 41–49

processing, inadequate linguistic knowledge), Asian students in Western countries may not be able to demonstrate critical
thinking to the same degree as their native English-speaker counterparts because of the higher cognitive processing resource
requirements of that L2, which depletes resources available in working memory for critical thinking. The important find-
ing in the present study is that proficiency in both languages (English and Japanese) significantly correlated with critical
evaluation performance in the respective language.

There still remains the possibility that the significant correlations between the two languages in the proportions of the
target language produced, are a result of other factors, which are correlated to both first and second language proficiency,
and which might affect the production of evaluative language – irrespective of the language being used. In other words, in
addition to language proficiency, people’s critical evaluation performance may also be influenced by some other individual
factors – such as general intelligence, working memory capacity, or communication skills – that are independent of the
language being used for expression. It would be useful to identify and examine those other factors in future research.

As noted earlier, the student participants in this first study had already received instruction in academic discourse that
included the use of evaluative language. Therefore, an important next question to address was, “To what extent had that
instruction affected the relative production of evaluative language in English and in Japanese?” – which was pursued in
Study 2.

3. Study 2

Study 2 tested three hypotheses. The first was that first year students would manifest lower use of the target evalua-
tive language compared to the second year students from whom data were gathered in Study 1. The second was that the
differences in evaluative language use would be consistent across English and Japanese. In other words, the prediction was
that, like the second year students, the first year students would evidence greater use of evaluative statements in their L1
(Japanese) compared to their L2 (English). The third hypothesis was that the first year students’ proficiency in both languages
would not be related to the proportions of evaluative language they produce. This hypothesis was based on the assumption
that, without receiving instruction about how to undertake and convey critical evaluation, students’ language proficiency
would not make a difference to their production of the target evaluative language as they would have very limited knowledge
about the structures necessary for producing the required language.

3.1. Method

3.1.1. Participants
Forty-three Japanese university students who were in their first year of studies in the same science and engineering

faculty as the students in Study 1 were added to the participants of Study 1, for a total of 153 participants. The additional
students came from two classes of a compulsory first year English communication skills course which deals with various
aspects of oral and written academic discourse, but nothing explicit about evaluative language (which is not covered until
the second year course).

3.1.2. Materials, procedure, and analysis
For one of their homework assignments, the students were given brief reading materials (in English and in Japanese)

about the Titanic and Space Shuttle Challenger disasters, including the proposed causes of those disasters. These materials
were drawn from the textbook used in the second year course. The Japanese translations were provided to these first year
participants to ensure that their subsequent writing performance would not have been compromised by possible difficulties
in understanding the English versions. The contents of those materials were not covered in class.

The homework task that the students had to do was the same as that given to the second year students: to produce
two brief reports to explain what they considered to be the most important cause of each of the disasters, after reading the
materials provided. Like the second year students, they were randomly assigned to write one report in English and the other
in Japanese. Also, like the second year students, they were provided with the one-page examples (one in English and one in
Japanese) of how alternative reasons (for learning the English language) could be ranked according to judgments about their
relative importance. The crucial difference was that the first year students were not provided class instruction and exercises
on the use of academic discourse specifically pertaining to evaluative language.

The written reports that the students produced were analyzed and scored in the same manner described in the first study.
The first and second year students’ data were then compared using repeated measures ANOVAs. Correlational analyses were
carried out to examine possible relationships between the students’ language proficiency scores and the proportions of
evaluative language they produced.

3.2. Results

Table 3 shows the means, and standard deviations (in brackets), obtained for the Total and Evaluative categories of
sentences in the first year students’ written work in English and in Japanese.

Analyses of variance revealed significant effects due to year of enrolment (first year compared to second year) in the stu-
dents’ scores for the total number of sentences [Total], F(1, 152) = 22.90, p < 0.001, �p2 = 0.113; and the number of evaluative

E. Manalo, C. Sheppard / Thinking Skills and Creativity 21 (2016) 41–49 47

Table 3
Means (with standard deviations in brackets) of total and evaluative sentences that the first year students produced in English and in Japanese.

Total Evaluative

English 15.70 (8.20) 2.09 (1.71)
Japanese 12.84 (6.28) 2.26 (1.24)

Table

4

Correlation coefficients between first year students’ TOEIC and writing complexity scores, and proportions of evaluative sentences they produced, according
to the language used (effect sizes, R2 shown in brackets).

Scores Proportions of Evaluative Sentences

English Japanese

s
1
s

f
m
t
a
i
fi
E

w
t

3

o
l
c
s
t
p

i
p
i
i

4

p
s
fi
f
a
r

l
o
r
s
t

TOEIC 0.03 (0.000) −0.02 (0.000)
English Complexity 0.16 (0.026) −0.18 (0.032)
Japanese Complexity 0.01 (0.000) 0.14 (0.020)

entences [Evaluative], F(1, 152) = 27.20, p < 0.001, �p2 = 0.130. A significant language effects were also found for Total, F(1, 52) = 26.67, p < 0.001, �p2 = 0.026. No significant interaction effects between language and year were found; nor were any ignificant effects found due to the task (Titanic versus Challenger).

These results indicate that, compared to the second year students, the first year students wrote fewer sentences in total
or their reports, and fewer evaluative statements. Like the second year students, the first year students wrote significantly

ore sentences in English than in Japanese. Analysis of the complexity of the sentences they produced revealed that the
otal number of verbs they used was likewise higher in Japanese compared to English, F(1, 43) = 14.80, p < 0.001, �p2 = 0.032; nd the index of complexity (total number of verbs divided by total number of sentences) was higher for what they wrote n Japanese compared to what they wrote in English, F(1, 43) = 75.58, p < 0.001, �p2 = 0.409. This finding indicates that the rst year students, like their second year counterparts, were producing more complex sentences in Japanese compared to nglish.

The results of the correlational analysis are shown in Table 4. None of the correlations were significant – suggesting that,
ithout instruction, the students’ proficiency in both languages generally did not make a difference to their production of

he target evaluative language.

.3. Discussion

The results of Study 2 showed that the second year students wrote more sentences in their reports, and produced more
f the target evaluative language, compared to the first year students. This finding suggests that instruction on appropriate
anguage to use – which had been provided to the second year students – can improve students’ abilities in manifesting
ritical evaluation in their written work. Although as noted the instruction was provided almost entirely in English, the
ignificant language effects found were all in favor of the Japanese language, which suggests that there is transfer across
he languages in skills acquisition. In other words, skills taught and learned in English also produce improvements in the
roduction of evaluative language in Japanese.

The results of the correlational analysis underline the importance of providing instruction to cultivate student competence
n critical evaluation. No significant correlations were found between the proportions of evaluative language the students
roduced and their proficiency scores in English and in Japanese. This therefore suggests that general proficiency on its own

s not adequate: for students to evidence critical evaluation in their written work, it is necessary to also provide them with
nstruction on the specific language forms and structures to use.

. General discussion

The findings of this study provide evidence that, at least for Japanese students, using the Japanese language (their L1)
resents no disadvantage compared to English (their L2) in the production of evaluative language (i.e., the Japanese language
tructure is not a limiting factor). How Japanese students’ evaluative language use might compare to that of students whose
rst language is structured differently (e.g., native English speakers responding to the same tasks), or students who are

ully bilingual in Japanese and English, would need to be examined in future research. However, in the present study, there
ppeared to be no obvious deficits in evaluative language production in Japanese among the second year students who had
eceived instruction in the necessary academic discourse.

There is evidence in the present study, however, that language proficiency can influence the production of evaluative
anguage. Although the conceptualization of critical thinking is likely to be independent of the language being used, the use

f a language that one is not so proficient in requires greater cognitive processing resources and thus limits the remaining
esources that could be utilized for the expression of critical thinking. In this study, the significant correlations between the
tudents’ English proficiency scores and the amount of evaluative language they evidenced in their English writing indicate
hat performance varied with L2 proficiency. This provides useful evidence to corroborate previously made claims (e.g., Floyd,

48 E. Manalo, C. Sheppard / Thinking Skills and Creativity 21 (2016) 41–49

2011; Lun et al., 2010; Paton, 2005) that some of the shortcomings in critical thinking skills manifested by international
students can be attributed to their having to use an L2 in which they may not be as proficient compared to their native
speaker counterparts. Although those significant correlations were not high, results were also obtained in the present study
showing that the students’ Japanese writing complexity significantly correlated with the proportion of evaluative language
they produced in Japanese. This finding provides further support for the argument that language proficiency can influence
the production of evaluative language, this time in the L1.

4.1. Conclusion

The finding that L2 proficiency is a potential limiting factor in students’ use of critical thinking skills suggests that
educational strategies for improving non-native speaker students proficiencies in English (or the language used in the host
country) would be beneficial. Successful implementation of such strategies would likely reduce the perceived deficiencies
in Asian and other foreign students’ critical thinking competencies (cf. Lee & Carrasquillo, 2006; Robertson et al., 2000).
The findings of this study also indicate that appropriate classroom instruction is essential for facilitating the development
of students’ abilities to demonstrate critical thinking competencies in the work they produce. The second year students
evidenced similar writing profiles to those of first year students; however, having received instructions in academic discourse
relevant to critical evaluation, they also produced more of the target evaluative language. They did this in both languages,
L1 and L2, even though academic discourse instruction was primarily provided in the L2 – suggesting some transfer of skills
across languages. Furthermore, the lack of significant correlations between the first year students’ L1 and L2 proficiency and
their use of the target evaluative language underline the necessity of providing critical evaluation instruction to students.
Without receiving explicit instruction, most students − irrespective of their language proficiency – would not likely know
how to effectively demonstrate critical evaluation in the work they produce.

List of funder and grant number

This research was supported by a grant-in-aid (23243071) received from the Japan Society for the Promotion of Science.

Conflict of interest

There is no conflict of interest known to the authors.

Acknowledgements

The authors would like to thank Kyoko Watanabe and Rongting Shen for their invaluable assistance during various stages
of this study.

References

Ackerman, P. L. (1986). Individual differences in information processing: an investigation of intellectual abilities and task performance during practice.
Intelligence, 10, 101–139.

Anthony, L., Rose, R., & Sheppard, C. (2010). Concept building and discussion: foundations. Tokyo: DTP Publishing.
Atkinson, D. (1997). A critical approach to critical thinking in TESOL. TESOL Quarterly, 31, 9–37.
Au, T. K. F. (1983). Chinese and English counterfactuals: the Sapir-Whorf hypothesis revisited. Cognition, 15, 155–187.
Baddeley, A. D. (1986). Working memory. Oxford: Oxford University Press.
Baddeley, A. (1998). Human memory. Boston: Allyn & Bacon.
Bloom, A. (1981). The linguistic shaping of thought. Hillsdale, NJ: Erlbaum.
Boroditsky, L. (2001). Does language shape thought?: Mandarin and English speakers’ conceptions of time. Cognitive Psychology, 43, 1–22.
Chen, J. Y. (2007). Do Chinese and English speakers think about time differently? Failure of replicating Boroditsky (2001). Cognition, 104, 427–436.
Davies, W. M. (2006). An ‘infusion’ approach to critical thinking: Moore on the critical thinking debate. Higher Education Research & Development, 25,

179–193.
Dornyei, Z., & Katona, L. (1992). Validation of the C-test amongst Hungarian EFL learners. Language Testing, 9, 187–206.
Eggly, S., Musial, J., & Smulowitz, J. (1999). The relationship between English language proficiency and success as a medical resident. English for Specific

Purposes, 18, 201–208.
Ellis, R. (2009). The differential effects of three types of task planning on the fluency, complexity, and accuracy in L2 oral production. Applied Linguistics,

30, 474–509.
Ennis, R. H. (1998). Is critical thinking culturally biased? Teaching Philosophy, 21, 15–33.
Fisher, A., & Scriven, M. (1997). Critical thinking: its definition and assessment. Norwich, UK: Centre for Research in Critical Thinking.
Floyd, C. B. (2011). Critical thinking in a second language. Higher Education Research and Development, 30, 289–302.
Fox, H. (1994). Listening to the world. Urbana, IL: National Council of Teachers of English.
Glassner, A., Weinstock, M., & Neuman, Y. (2005). Pupils’ evaluation and generation of evidence and explanation in argumentation. British Journal of

Educational Psychology, 75, 105–118.
Halpern, D. F. (1998). Teaching critical thinking for transfer across domains. American Psychologist, 53, 449–455.

Hill, J. H., & Mannheim, B. (1992). Language and world view. Annual Review of Anthropology, 21, 381–406.
Hockett, C. E. (1954). Chinese versus English: an exploration of the Whorfian thesis. In H. Hoijer (Ed.), Language in culture (pp. 106–123). Chicago:

University of Chicago Press.
In’nami, Y., & Koizumi, R. (2011). Factor structure of the revised TOEIC test: a multiple-sample analysis. Language Testing, 29, 131–152.
Ishikawa, S. (1995). Objective measurement of low-proficiency EFL narrative writing. Journal of Second Language Writing, 4, 51–69.

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0005

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0005

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0005

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0005

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0005

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0005

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0005

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0005

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0005

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0005

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0005

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0005

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0005

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0005

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0005

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0005

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0005

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0005

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0005

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0005

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0005

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0005

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0010

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0010

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0010

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0010

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0010

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0010

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0010

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0010

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0010

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0015

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0015

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0015

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0015

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0015

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0015

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0015

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0015

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0015

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0015

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0015

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0015

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0015

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0015

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0015

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0015

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0020

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0020

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0020

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0020

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0020

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0020

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0020

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0020

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0020

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0020

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0020

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0020

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0020

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0020

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0020

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0025

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0025

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0025

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0025

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0025

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0025

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0025

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0030

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0030

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0030

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0030

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0030

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0030

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0030

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0035

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0035

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0035

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0035

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0035

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0035

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0035

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0035

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0035

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0040

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0040

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0040

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0040

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0040

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0040

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0040

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0040

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0040

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0040

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0040

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0040

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0040

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0040

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0040

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0040

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0040

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0040

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0040

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0045

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0045

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0045

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0045

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0045

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0045

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0045

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0045

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0045

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0045

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0045

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0045

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0045

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0045

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0045

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0045

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0045

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0045

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0045

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0045

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0045

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0050

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0050

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0050

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0050

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0050

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0050

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0050

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0050

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0050

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0050

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0050

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0050

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0050

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0050

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0050

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0050

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0050

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0050

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0050

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0050

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0050

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0050

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0050

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0055

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0055

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0055

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0055

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0055

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0055

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0055

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0055

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0055

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0055

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0055

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0055

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0055

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0055

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0055

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0055

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0060

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0060

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0060

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0060

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0060

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0060

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0060

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0060

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0060

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0060

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0060

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0060

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0060

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0060

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0060

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0060

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0060

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0060

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0060

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0060

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0060

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0060

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0065

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0065

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0065

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0065

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0065

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0065

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0065

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0065

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0065

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0065

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0065

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0065

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0065

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0065

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0065

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0065

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0065

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0065

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0065

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0065

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0065

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0065

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0065

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0065

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0065

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0065

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0065

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0070

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0070

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0070

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0070

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0070

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0070

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0070

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0070

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0070

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0070

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0070

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0070

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0070

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0075

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0075

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0075

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0075

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0075

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0075

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0075

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0075

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0075

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0075

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0075

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0075

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0075

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0075

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0075

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0080

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0080

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0080

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0080

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0080

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0080

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0080

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0080

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0080

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0080

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0080

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0080

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0080

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0080

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0080

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0080

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0080

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0085

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0085

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0085

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0085

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0085

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0085

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0085

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0085

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0085

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0085

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0085

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0085

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0085

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0090

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0090

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0090

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0090

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0090

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0090

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0090

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0090

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0090

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0090

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0090

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0090

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0090

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0090

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0090

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0090

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0090

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0090

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0090

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0090

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0090

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0095

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0095

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0095

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0095

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0095

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0095

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0095

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0095

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0095

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0095

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0095

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0095

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0095

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0095

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0095

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0100

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0100

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0100

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0100

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0100

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0100

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0100

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0100

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0100

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0100

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0100

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0100

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0100

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0100

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0105

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0105

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0105

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0105

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0105

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0105

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0105

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0105

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0105

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0105

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0105

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0105

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0105

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0105

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0105

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0105

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0105

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0105

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0105

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0105

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0105

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0105

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0105

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0105

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0105

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0110

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0110

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0110

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0110

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0110

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0110

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0110

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0110

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0110

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0110

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0110

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0110

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0110

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0110

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0110

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0110

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0110

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0110

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0115

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0115

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0115

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0115

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0115

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0115

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0115

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0115

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0115

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0115

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0115

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0115

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0115

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0115

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0115

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0115

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0115

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0115

I

J
K
K

L
L
L

M
M
M
M

N

O

P

P
P
R
R

S
S

T

T

T
T
T

T

V

W
W

E. Manalo, C. Sheppard / Thinking Skills and Creativity 21 (2016) 41–49 49

takura, H., & Tsui, A. B. M. (2011). Evaluation in academic discourse: managing criticism in Japanese and English book reviews. Journal of Pragmatics, 43,
1366–1379.

anuary, D., & Kako, E. (2007). Re-evaluating evidence for the linguistic relativity hypothesis: response to Boroditsky (2001). Cognition, 104, 417–426.
ong, K. C. C. (2005). Linguistic resources as evaluators in English and Chinese research articles. Multilingua, 24, 275–308.
u, K. Y. L. (2009). Assessing students’ critical thinking performance: urging for measurements using multi-response format. Thinking Skills and Creativity,

4, 70–76.
ee, K. S., & Carrasquillo, A. (2006). Korean college students in United States: perceptions of professors and students. College Student Journal, 40, 442–456.
iu, L. G. (1985). Reasoning counterfactually in Chinese: are there any obstacles? Cognition, 21, 239–270.
un, V. M. C., Fischer, R., & Ward, C. (2010). Exploring cultural differences in critical thinking: is it about my thinking style or the language I speak?

Learning and Individual Differences, 20, 604–616.
analo, E., & Uesaka, Y. (2012). Elucidating the mechanism of spontaneous diagram use in explanations: how cognitive processing of text and

diagrammatic representations is influenced by individual and task-related factors. In P. Cox, P. Rodgers, & B. Plimmer (Eds.), Diagrammatic
representation and inference. Lecture notes in artificial intelligence (LNAI) 7352 (pp. 35–50). Berlin, Heidelberg: Springer-Verlag.

analo, E., & Uesaka, Y. (2014). Students’ spontaneous use of diagrams in written communication: understanding variations according to purpose and
cognitive cost entailed. In T. Dwyer, H. Purchase, & A. Delaney (Eds.), Diagrammatic representation and inference. Lecture notes in artificial intelligence
(LNAI) 8578 (pp. 78–92). Berlin, Heidelberg: Springer-Verlag.

analo, E., Kusumi, T., Koyasu, M., Michita, Y., & Tanaka, Y. (2013). To what extent do culture-related factors influence university students’ critical
thinking use. Thinking Skills and Creativity, 10, 121–132.

analo, E., Kusumi, T., Koyasu, M., Michita, Y., & Tanaka, Y. (2015). Do students from different cultures think differently about critical and other thinking
skills? In M. Davies, & R. Barnett (Eds.), Palgrave handbook of critical thinking in higher education (pp. 299–316). New York: Palgrave Macmillan.

eisser, U., Boodoo, G., Bouchard, T. J., Jr., Boykin, A. W., Brody, N., Ceci, S. J., & Urbina, S. (1996). Intelligence: knowns and unknowns. American
Psychologist, 51, 77–101.

rtega, L. (2003). Syntactic complexity measures and their relationship to L2 proficiency: a research synthesis of college-level L2 writing. Applied
Linguistics, 24, 492–518.

aton, M. (2005). Is critical analysis foreign to Chinese students? In E. Manalo, & G. Wong-Toi (Eds.), Communication skills in university education: the
international dimension (pp. 1–11). Auckland: Pearson Education.

erkins, K. (1987). Review of the ‘test of English for international communication’. In J. C. Alderson, K. J. Krahnke, & C. W. Stansfield (Eds.), Reviews of
English language proficiency tests (pp. 81–83). Washington, DC: TESOL.

owers, D. E., Kim, H.-J., & Weng, V. Z. (2008). The redesigned TOEIC (listening and reading) test: relations to test-taker perceptions of proficiency in English
(ETS Research Rep. No. RR-08-56). New Jersey: ETS; Princeton.

ichards, J., Platt, J., & Platt, H. (1992). Longman dictionary of language teaching and applied linguistics. Harlow, UK: Longman.
obertson, M., Line, M., Jones, S., & Thomas, S. (2000). International students, learning environments and perceptions: a case study using the Delphi

technique. Higher Education Research & Development, 19, 89–102.
collon, R., & Wong-Scollon, S. (1991). Topic confusion in English-Asian discourse. World Englishes, 10, 113–125.
kehan, P., & Foster, P. (1999). The influence of task structure and processing conditions on narrative retellings. Language Learning, 49, 93–120.
akano, Y., & Noda, A. (1993). A temporary decline of thinking ability during foreign language processing. Journal of Cross-Cultural Psychology, 24, 445–462.
akano, Y., & Noda, A. (1995). Interlanguage dissimilarity enhances the decline of thinking ability during foreign language processing. Language Learning,

45, 657–681.
akano, Y. (1989). Methodological problems in cross-cultural studies of linguistic relativity. Cognition, 31, 141–162.
en Dam, G., & Volman, M. (2004). Critical thinking as a citizenship competence: teaching strategies. Learning and Instruction, 14, 359–379.
homm, E., & Bromme, R. (2011). It should at least seem scientific! Textual features of scientificness and their impact on lay assessments of online

information. Science Education, 96, 187–211.
se, C.-S., & Altarriba, J. (2008). Evidence against linguistic relativity in Chinese and English: a case study of spatial and temporal metaphors. Journal of

Cognition and Culture, 8, 335–357.
orster, J., & Schuring, G. (1989). Language and thought: developmental perspectives on counterfactual conditionals. South African Journal of Psychology,

19, 34–38.
horf, B. L. (1956). The relation of habitual thought and behavior to language. In J. B. Carroll (Ed.), Language, thought, and reality: selected writings of

Benjamin Lee Whorf (pp. 134–159). Cambridge, MA: MIT Press.
olfe-Quintero, K., Inagaki, S., & Kim, H.-Y. (1998). Second language development in writing: measures of fluency, accuracy, and complexity. Technical report

no. 17. Honolulu, H awai’i: University of Hawai’i, Second Language Teaching and Curriculum Center.

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0120

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0120

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0120

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0120

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0120

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0120

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0120

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0120

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0120

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0120

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0120

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0120

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0120

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0120

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0120

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0120

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0120

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0120

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0120

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0120

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0120

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0125

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0125

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0125

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0125

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0125

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0125

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0125

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0125

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0125

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0125

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0125

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0125

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0125

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0125

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0125

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0125

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0125

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0125

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0130

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0130

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0130

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0130

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0130

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0130

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0130

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0130

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0130

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0130

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0130

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0130

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0130

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0130

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0130

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0130

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0130

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0135

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0135

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0135

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0135

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0135

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0135

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0135

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0135

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0135

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0135

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0135

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0135

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0135

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0135

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0135

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0135

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0135

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0135

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0135

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0135

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0135

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0140

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0140

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0140

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0140

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0140

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0140

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0140

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0140

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0140

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0140

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0140

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0140

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0140

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0140

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0140

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0140

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0140

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0140

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0140

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0140

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0145

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0145

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0145

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0145

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0145

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0145

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0145

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0145

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0145

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0145

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0145

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0145

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0145

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0145

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0145

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0150

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0150

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0150

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0150

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0150

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0150

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0150

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0150

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0150

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0150

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0150

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0150

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0150

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0150

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0150

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0150

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0150

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0150

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0150

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0150

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0150

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0150

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0150

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0150

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0150

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0150

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0150

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0155

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0165

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0165

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0165

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0165

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0165

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0165

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0165

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0165

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0165

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0165

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0165

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0165

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0165

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0165

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0165

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0165

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0165

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0165

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0165

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0165

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0165

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0165

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0175

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0175

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0175

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0175

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0175

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0175

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0175

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0175

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0175

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0175

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0175

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0175

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0180

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0180

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0180

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0180

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0180

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0180

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0180

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0180

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0180

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0180

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0180

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0180

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0180

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0180

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0180

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0180

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0180

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0180

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0180

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0180

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0180

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0180

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0180

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0180

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0195

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0195

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0195

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0195

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0195

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0195

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0195

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0195

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0195

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0195

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0195

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0195

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0195

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0195

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0195

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0195

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0195

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0195

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0195

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0195

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0195

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0195

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0195

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0195

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0195

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0200

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0200

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0200

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0200

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0200

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0200

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0200

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0200

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0200

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0200

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0200

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0200

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0205

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0205

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0205

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0205

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0205

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0205

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0205

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0205

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0205

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0205

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0205

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0205

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0205

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0205

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0205

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0205

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0205

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0205

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0205

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0205

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0205

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0205

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0205

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0205

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0210

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0210

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0210

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0210

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0210

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0210

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0210

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0210

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0210

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0210

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0210

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0210

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0210

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0215

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0215

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0215

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0215

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0215

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0215

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0215

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0215

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0215

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0215

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0215

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0215

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0215

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0215

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0215

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0215

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0215

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0215

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0215

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0220

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0220

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0220

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0220

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0220

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0220

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0220

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0220

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0220

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0220

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0220

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0220

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0220

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0220

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0220

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0220

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0220

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0220

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0220

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0220

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0225

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0225

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0225

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0225

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0225

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0225

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0225

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0225

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0225

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0225

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0225

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0225

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0225

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0225

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0225

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0225

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0225

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0225

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0225

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0225

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0230

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0230

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0230

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0230

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0230

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0230

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0230

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0230

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0230

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0230

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0230

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0230

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0230

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0230

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0230

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0235

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0235

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0235

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0235

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0235

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0235

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0235

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0235

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0235

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0235

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0235

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0235

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0235

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0235

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0235

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0235

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0235

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0240

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0240

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0240

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0240

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0240

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0240

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0240

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0240

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0240

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0240

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0240

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0240

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0240

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0240

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0240

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0240

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0240

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0240

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0240

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0240

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0240

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0240

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0240

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0240

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0240

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0240

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0240

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0245

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0245

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0245

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0245

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0245

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0245

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0245

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0245

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0245

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0245

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0245

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0245

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0245

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0245

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0245

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0245

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0245

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0245

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0245

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0245

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0245

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0245

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0245

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0245

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0245

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0245

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0245

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0250

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0250

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0250

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0250

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0250

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0250

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0250

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0250

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0250

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0250

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0250

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0250

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0250

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0250

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0250

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0250

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0250

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0250

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0250

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0260

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0260

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0260

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0260

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0260

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0260

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0260

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0260

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0260

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0260

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0260

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0260

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0260

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0260

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0260

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0260

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0260

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0260

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0260

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0260

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0260

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0260

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0260

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0260

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0260

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0260

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0260

http://refhub.elsevier.com/S1871-1871(16)30025-6/sbref0260

• How might language affect critical thinking performance?

1 Introduction
1.1 Overview of the present study
2 Study 1
2.1 Method
2.1.1 Participants
2.1.2 Materials and procedure
2.1.3 Analyses
2.2 Results
2.3 Discussion
3 Study 2
3.1 Method
3.1.1 Participants
3.1.2 Materials, procedure, and analysis
3.2 Results
3.3 Discussion
4 General discussion
4.1 Conclusion
List of funder and grant number
Conflict of interest
Acknowledgements
References

Thinking Skills and Creativity 24 (2017) 140–15

1

Contents lists available at ScienceDirect

Thinking Skills and Creativity

j o u r n a l h o m e p a g e : h t t p : / / w w w . e l s e v i e r . c o m / l o c a t e / t s

c

Understanding critical thinking in Chinese sociocultural
contexts:

A

case study in a Chinese college

Lei Chen
Shaanxi Normal University, 199, Changan Nan Road, Yanta District, Xi An, Shaan Xi, China

a r t i c l e i n f o

Article history:
Received 1 November 2015
Received in revised form 6 February 2017
Accepted 22 February 2017
Available online 27 February 2017

Keywords:
Critical thinking
Chinese students
Chinese indigenous philosophy
Case study

a b s t r a c t

The presented research examines Chinese students’ conceptualization of critical thinking
through a qualitative study interviewing 46 Chinese college students. This study finds that
Chinese students tend to conceptualize critical thinking as a multi-faceted concept. Cog-
nitive thinking skills, intellectual autonomy, and the omnipresence of positive and negative
aspects are commonly mentioned in the students’ definitions of critical thinking (among
other concepts). The study finds that there are unique qualities in Chinese students’ con-
ceptualizations of critical thinking that need to be understood in Chinese sociocultural
contexts. This project proposes ways to understand and study Chinese students not previ-
ously considered in critical thinking literature.

© 2017 Elsevier Ltd. All rights reserved.

1. Introduction

When an English-speaking professor asks a Chinese student to think critically, does the Chinese student understand what
he or she is required to do?

The term critical thinking is one of several “fuzzy” constructs in education frequently used by researchers and practitioners,
yet the definition is vague to the point of being regarded as problematic or even unnecessary by some researchers (Atkinson,
1997; Kumaravadivelu, 2003; Vandermensbrugghe, 2004). To make the issue even more complex, critical thinking has been
applied to classrooms and studies in other languages and cultures (Davidson, 1995; Shao, 2013;

Z

hang & Lambert, 2008).
With the increasing number of Chinese students studying abroad in Western universities, there are a growing number of
anecdotal accounts of some Western professors complaining about a lack of critical thinking among Chinese students (Clark
& Gieve, 2006; Heng, 2016; Paton, 2005). Some professors and educators question if critical thinking can be applied to a
Chinese educational culture which values rote-learning and respects authority and harmony (Li, Rao, & Tse, 2012; Ryan &
Slethaug, 2010; Turner, 2006). Some scholars, however, found that Chinese students are not alien to the critical thinking
concept and can demonstrate critical thinking when teaching is effectively conducted (Dong, Anderson, Kim, & Li, 2008;
Durkin, 2008; Yang, 2016).

At the same time, in several documents and public speeches released by the Chinese government Chinese educators
announced that it is highly important to innovate Chinese education to increase the focus on critical thinking (Dong, 2015).

Educational researchers are experimenting with various methods that could potentially improve Chinese students’ critical
thinking (Chan, Ho, & Ku, 2011; Yang, 2016). Thus, projects contextualizing critical thinking in Chinese educational contexts
are greatly in demand.

E-mail address: elaine chenlei@sina.com

http://dx.doi.org/10.1016/j.tsc.2017.02.015
1871-1871/© 2017 Elsevier Ltd. All rights reserved.

dx.doi.org/10.1016/j.tsc.2017.02.015

http://www.sciencedirect.com/science/journal/18711871

http://www.elsevier.com/locate/tsc

http://crossmark.crossref.org/dialog/?doi=10.1016/j.tsc.2017.02.015&domain=pdf

mailto:elaine_chenlei@sina.com

dx.doi.org/10.1016/j.tsc.2017.02.015

c

1

2

3

s
r
a

1

E
t
b
t
e

a
&
a
r
o
p
I
t

v
s
a
T

w

L

p

a
s
t
r
(
l
P

t
c
U
c

a
e
a
d
c

o
c

L. Chen / Thinking Skills and Creativity 24 (2017) 140–151 141

This paper brings literature on the critical thinking concept into conversation with literature on Chinese learner’s socio-
ultural contexts through consideration of these three research questions:

. Can Chinese students (who have no experience studying in Western universities) provide a definition of critical thinking?

. How do Chinese college students conceptualize critical thinking in their academic context?

. How is such conceptualization affected by their sociocultural backgrounds?

I draw from interviews with 46 Chinese college students to answer these questions. This paper suggests that Chinese
tudents have unique and varied conceptualizations of critical thinking based on their sociocultural contexts. This study
aises questions around the practice of regarding critical thinking as a predetermined and universal concept which can be
pplied in all contexts.

.1. The construction of critical thinking in Western literature

The ways in which critical thinking is defined and constructed in current literature are virtually endless (Atkinson, 1997).
ducators, philosophers, psychologists, and many others have proffered varied definitions of critical thinking. At the same
ime, some scholars regard the construction of critical thinking as dictated by Western philosophical tradition which could
e viewed as inherently biased against Eastern cultures. In this section, I review some widely-cited definitions of critical
hinking and analyze how current definitions cannot be applied to educational practices which attempt to improve or
valuate students’ critical thinking in non-Western backgrounds.

Frequently, critical thinking is defined as a set of cognitive thinking skills including: self-directed reasoning, reflection,
nd making decisions about what to believe (Ennis, 1996); analyzing and evaluating thinking with a view to improve it (Paul

Elder, 2009); and problem finding as well as problem solving (Wineburg, 1998). Bloom’s Taxonomy (Bloom, 1956) includes
hierarchical scale of skills – knowledge, comprehension, application, analysis, synthesis, and evaluation – which has been
ecognized as connected to critical thinking (Johnston, Ford, Mitchell, & Myles, 2011). Some more recent conceptualizations
f critical thinking are illustrated in Dwyer, Hogan, and Stewart (2014) which defines critical thinking as a metacognitive
rocess that may purposefully increase the chances of producing a logical conclusion to an argument or solution to a problem.

n Dwyer et al.’s framework, aside from the analytical skills and inference skills generally mentioned when defining critical
hinking, memory, metacognition, and reflective judgment were also essential to the critical thinking concept.

Other scholars have pointed out that some skills in Bloom’s Taxonomy, for instance, analysis and problem-solving, are
ague and can include a number of different skills (Wegerif, Li, & Kaufman, 2015). Furthermore, what constitutes problem-
olving skills is not universal for all people in all cultures. Cultural psychology has demonstrated that in various tasks, Asians
pply problem-solving skills differently than Western participants (Ji, Peng, & Nisbett, 2000; Miyamoto & Wilken, 2013).
o give one example, Ji et al. (2000) found that East Asians are more likely to incorporate contexts in problem-solving
hile European-Americans are more likely to focus on the problem alone, ignoring contexts, when completing a Framed-

ine Task (FLT). Cultural psychology studies remind critical thinking scholars that Asian cultures may demonstrate or value
roblem-solving skills and higher-order thinking skills differently from Western cultures.

Atkinson (1997), Johnston et al. (2011) and Thayer-Bacon (1992) critiqued a universal evaluative standard that encour-
ges students to be adventurous and ask questions. Confucian teaching, for instance, believed good students to be
elf-reflective rather than simply inquisitive. Here is a quote from The Analects of Confucius: “The Master said, I have
alked with Hui for a whole day, and he has not made any objection to anything I said – as if he were stupid. When he
etired, and I have examined him in private, and found him able to elucidate. Hui! – He is not stupid. “(Analects, 2:9)

”) According to Confucius, the exemplary student does not chal-
enge the teacher with words. Instead, the student should reflect on himself and practice the learned philosophy with action.
ondering the problems quietly is valued more highly than asking the teacher many questions in Confucian cultures.

Atkinson (1997) proposed critical thinking is a social practice. Wider sociocultural contexts, for instance, demonstrate
hat what is regarded as “good thinking” in a society influences how that culture defines critical thinking. Specifically, three
ultural presumptions underlie the Western concepts of critical thinking: Notions of the Individual, Self-Expression, and
sing Language to Learn. Based on these three assumptions, Atkinson further proposed that the Western definitions of
ritical thinking may not be compatible to other cultures.

In more recent studies of the critical thinking concept, scholars are growingly aware that critical thinking is multi-faceted
nd multi-dimensional. For instance, Ku (2009) wrote that multiple assessments (multiple-choice response format and open
nded format) need to be used to assess students’ critical thinking. Moore (2013) explored the definitions of critical thinking
mong 13 university professors and found at least seven different definitions of critical thinking. Howe (2004) compared
ifferent definitions of critical thinking from Canadian and Japanese teachers and found that more Japanese teachers defined

ritical thinking as intellectual engagement than Canadian teachers.

In sum, in agreement with Atkinson’s approach, I argue that critical thinking tends to be embedded in the Western ideals
f thinking and learning as well as the minds of the practitioners. Thus, it is highly important to examine the definitions of
ritical thinking espoused by students and teachers, as Howe (2004) and Moore (2013) demonstrated. Examination of how

142 L. Chen / Thinking Skills and Creativity 24 (2017) 140–151

Chinese students define critical thinking may provide significant insights for critical thinking researchers and educators of
Chinese students around the globe.

1.2. Critical thinking concept in Chinese educational literature

Critical thinking in the Chinese language is a translated phrase about which Chinese scholars have yet to reach an
agreement. Interestingly, through the scholars’ struggle some underlying cultural assumptions can be observed. The first
translation of the word “critique” in Chinese is believed to derive from Renyuan Hu’s translation of the English version of
Kant’s essay Critique of Pure Reason in 1935. The Chinese word for critical – (Pi Pan) – is similar to the word criticize which
has the negative connotation of finding fault in something (Wu, 2011). As a result, Chinese scholars worried that translating
critical thinking as (Pi Pan Xing Si Wei) may create unnecessary objections from parents, students, and education
officials when this concept was promoted in China. Thus they recommended that instead of critical ( , Pi Pan), the Chinese
translation of critical thinking should place more emphasis on the process of logical thinking and making decisions (Wu,
2011). This brief history of translation reveals that Chinese scholars and researchers are concerned that the Western concept
of critical thinking might not be easily accepted in Chinese contexts due to a variety of social, cultural, and historical reasons.

There is also disagreement whether and how the critical thinking concept is compatible with Chinese traditional cultures,
specifically Confucian culture. Kim (2003) suggests that traditional Chinese culture entails reflexivity, which is essential to
critical thinking concepts. Nevertheless, a more commonly accepted judgment on the nature of Confucian teaching is from
Graham (1989): Confucius valued learning but not thinking. Or as Kim restated, Confucius was more interested in how people
should conduct their personal, social, and political life than the decision making process of what kind of life to live. Confucius
did not believe that one can reach the truth through individual thinking. Unlike some Western definitions of critical thinking
that emphasize the students’ ability to make judgments (Ennis, 1996), critical thinking in Confucian contexts seems to
emphasize self-reflexivity, but not judgment of what kind of life to live.

Chinese literature on the critical thinking concept demonstrates that considering Chinese contexts when defining critical
thinking is important as this concept is foreign, borrowed, and might be incompatible with Chinese students’ sociocultural
contexts. Thus, it is highly important to understand how Chinese students perceive this topic through empirical studies.

1.3. Sociocultural theories of thinking and learning

In sociocultural theories of thinking, individual efforts and sociocultural institutions and practices are constituted by and
of each other (Rogoff, 2003). Learning is affected by the interactive, institutional, social, cultural, and historical contexts of
the learning activity (Lave & Wenger, 1991). According to Rogoff (2003), a child’s development is affected by the norms,
regulations, and lifestyles of the sociocultural environment in which the child lives. For instance, the concept of freedom is
different in East and West in terms of childrearing. While for Caucasian Americans, freedom of self-expression is valued, for
Chinese immigrants in the U.S., the ability to become self-sustaining and to make contributions to family and society is more
important (p. 194). Sociocultural scholars also examined school cultures and minority students’ home cultures in the U.S.
Minority students who were once regarded as incapable of effective learning were the victims of cultural mismatch between
school and home cultures (Caffarella & Merriam, 2000). Thus, from a sociocultural approach, to understand Chinese students’
critical thinking it is important to examine how students’ social, cultural, and educational settings shape their understanding
and conceptualization of critical thinking.

2. Methods

2.1. Participants

This project features interview data collected at a Chinese college campus during the 2012 spring semester. The research
setting was a university in Guangzhou, a Southern metropolitan city. The participants of the study were from a variety of
majors including Business, Literature, Education, and Engineering. Among the 46 student participants, 22 students were
freshmen, 18 students were sophomore, 6 students were juniors; 33 were females, 13 were males. All of the participants
were in their early twenties and came from middle to upper middle class families in cosmopolitan or suburban areas. All
of the students were of Han ethnicity (the majority ethnicity in the People’s Republic of China). None of the students had
studied abroad. The examination score required to enter this college is above the first-tier college entrance standard ( ),
meaning that student motivation and the standard of learning in this university is above average in the nation. This is a large
and comprehensive university. Most of the courses conducted in the university follow and meet national standards. In the
English language courses I observed in the university, the faculty used government approved textbooks and urged students
to prepare for the College English Test Band Four (CET-4). During the time this study was conducted in the university, no
specific courses about critical thinking or other thinking skills were offered. However, most students were able to mention

a few faculty members who used innovative methods to teach based on the faculty’s personal interests in teaching.

As this is an exploratory case study, the participants were volunteers. Five faculty members and approximately 200
students in the freshman, sophomore, and junior classes in the university were contacted for the purpose of participant
recruiting. Among the 200 students, 46 students volunteered. The variety in gender and fields of study of participants

a
u
t
a
i

2
w

i
s

a
i
t
d
o

i
e
o
i

2

t
t
d
m
c
b
a
i

p
(
a
d
t
m

h
c
p
u
p
f
c

c
t
d
i
d
i

L. Chen / Thinking Skills and Creativity 24 (2017) 140–151 143

llowed me to examine a range of opinions that might exist among Chinese college students. Studies demonstrate that
nderstanding of critical thinking may be varied among different majors (Johnston et al., 2011). Therefore, it is important
hat students of four different areas of studies were presented. With that said, since all 46 students are from one college
nd are volunteers, their answers may not represent all college students around China. This is a case study that proposes
nteresting perspectives rather than providing generalized claims about how all Chinese college students think.

.2. Interviews

All interviews were conducted in the Chinese language and recorded, then translated and transcribed. The interviews
ere semi-structured. The interview protocol is attached as Appendix A.

Prior to the interviews, I spent several weeks auditing classes, eating, and talking with students in dining halls to famil-
arize myself with the students. All interviews were conducted in classrooms and student dorm rooms to ensure that the
tudents were comfortable talking with me.

In the interviews, students were invited to discuss their perception of critical thinking. I used both English, critical thinking,
nd Chinese, (Pi Pan Xing Si Wei), when I first mentioned the concept in every interview. In my first one or two
nterviews where I explicitly used the word define ( ), the students assumed I was asking for a formal definition of critical
hinking and replied that they did not know how to define it. When I switched to more informal wording and asked “What
o you guess critical thinking means?” or “What do you suppose critical thinking means?” students appeared to be more
pen to sharing their personal opinions on critical thinking.

After the Chinese students offered their definitions of critical thinking, I asked them to offer an example of critical thinking
n their life and in their study. I also asked the students’ opinions on how critical thinking influenced their educational
xperiences and might influence their future careers. While the students’ definitions of critical thinking usually consisted
f a few concise words and phrases, the specific descriptions provided by the Chinese students offered further reference to
nterpret their definitions of critical thinking.

.3. Analysis

In this study the purpose of the data analysis is to explore and identify patterns of conceptualizing and discussing critical
hinking among Chinese college students. I adopted a multi-step open-coding strategy to identify potential patterns in
he data. In the initial analysis process, I used open-coding strategy and developed in-vivo codes that summarize student
efinitions (Strauss, 1987). Examples of in-vivo codes include: logical thinking, problem-solving, my ideas are unique, I have
y own judgment. In the second round of analysis, using the constant comparative method (Strauss, 1987), I categorized

oding into a few different categories and ensured that I captured all the possible concepts of critical thinking discussed
y the interviewed Chinese students. Then I read the coded excerpts selectively to develop themes about Chinese social
nd cultural backgrounds which characterize the interview data. The themes, examples of in-vivo codes, and examples of
nterview excerpts are presented in Appendix C.

To ensure that my interpretations of the data were valid, I communicated with participants and confirmed my inter-
retations through email. Member checking is an important technique for establishing credibility in qualitative research
Cho & Trent, 2006; Hammersley & Atkinson, 1995; Lincoln & Guba, 1985). In this study, each participant was contacted
t least twice during the analysis. The first time, I contacted my participants to check the categories I assigned for their
efinitions. The second time, I contacted them to discuss my analysis of the way in which their reflections about critical
hinking are influenced by Chinese social, cultural, historical, and educational contexts. While students agreed with most of

y interpretations, their disagreements concerning some of the interpretations are discussed in Section 3.
In addition, two teachers were interviewed to triangulate the data analysis (Creswell & Clark, 2007). Both Chinese teachers

ad more than five years of college teaching experience and received graduate degrees in foreign studies and intercultural
ommunication. These two interviews were conducted after the student interviews were coded. In these two interviews, I
resented the teachers with Table 2 and asked if they agreed with my interpretations of the students’ definitions. This project
sed teacher interviews to triangulate the interpretations of students’ interviews since the teachers were familiar with the
articipants as well as the Chinese cultural contexts. This dialogic method of conducting intercultural research is adopted
rom Tobin, Hsueh, & Karasawa (2009) who recorded pre-school scenarios in three cultures and invited local teachers to
omment on each scenario as well as other people’s comments.

I was aware that I enjoyed some privileges as a Chinese graduate student in a U.S. university during the time I was
onducting the interviews. My status was respected by the students. However, there are problems being intimate with
he participants. The fact that I was interested in the topic of critical thinking could have increased their interest and

rawn their attention to the topic. My personal knowledge about Chinese education and Confucian culture might assist my

nterpretation of the students’ experiences because I had an insider point of view. At the same time, such knowledge could
elude my interpretation of the participants’ experiences and interfere with interpretation of the data. The limitations and

mplications of my analysis are discussed in the conclusion section.

144 L. Chen / Thinking Skills and Creativity 24 (2017) 140–151

Table 1
Chinese students’ abilities to provide a definition of critical thinking.

Number of students

Frequency

Were able to provide one or more definitions of critical thinking and provide examples 43 93%
Did not provide any definitions of critical thinking 2 5%
Were able to provide a definition but were unable to support it with an example 1 2%

Table 2
Themes in Chinese students’ definitions of critical thinking.

Themes Codes Frequency

Cognitive thinking skills Quality reasoning and sound judgment; 36 students
Logical thinking; 75%
Problem-solving;
Evidence based thinking;
Deliberate and controlled thinking strategies.

Intellectual autonomy Having their own ideas; 27 students
Unique ideas; 56%
Freedom to make own judgment;
Conduct own research;
Freedom to use imagination.

Omnipresence of the opposite point of view Positive and negative; 11 students
Pros and cons. 23%

Multiple perspectives (More than two, and
aware of conflicts)

Thinking in a variety of perspectives; 8 students
Torn between different perspectives. 17%

Revolutionary thoughts in art, science, and
other fields.

Revolutionary ideas; 7 students
People who bring revolutions to their fields. 15%

Knowledge and skills outside the scope of
standard examination

The knowledge and skills that are
useful in real life, but cannot be tested.

1 student
2%

3. Results

3.1. The Chinese college students’ abilities to define critical thinking

The majority of the students in this project did not have difficulty providing definitions of critical thinking and were
able to provide examples from their own experiences to illustrate critical thinking. However, in two interviews, students
explicitly told me that they were not familiar with the critical thinking concept and refused to offer any definitions of
critical thinking. Both of the students were freshmen and believed they had never heard about the critical thinking concept
in their previous educational experiences. In both interviews, I asked the students why they would volunteer if they had
no idea what the critical thinking concept meant. The students expressed that they were interested to learn the concept
of critical thinking and would like me to define the concept for them. I offered a few definitions of critical thinking such
as higher-order thinking, intellectual autonomy, argumentation, etc. Both students understood the explanations and were
able to give examples of such based on their own understanding. However, they refused to tell me which definitions most
aligned with their conceptions of critical thinking. One Chinese student provided a definition but could not provide any
supporting examples. After I repeatedly prompted her to try, she said she wasn’t sure about the definition and would rather
that I provided a definition and an example for her. The students’ abilities to provide a definition of critical thinking is listed
in Table 1.

The Chinese students’ ability to define critical thinking demonstrates that critical thinking is not alien to Chinese college
students. Although two students could not define critical thinking, they did understand the concepts related to critical
thinking. Note again that all the participants were volunteers and the percentage provided in this study might not represent
the ability to define critical thinking by all Chinese college students.

3.2. The multiplicity of students’ definitions

In the interviews, most students mentioned several different themes. For instance, a student said in her interview, “First
of all, I have my own judgment, and also, I have a reason to support my own judgment.” According to my analysis, two themes
– cognitive thinking skills and intellectual autonomy – were mentioned in her short, one sentence answer. Sometimes, a

student would define critical thinking in one way when asked to provide a definition and another way when giving an
example of critical thinking or discussing real life cases in which critical thinking is important. During the member checking
process, I checked with students if all definitions in their answers were valid conceptualizations of critical thinking. I also
verified that there were no other ways to define critical thinking other than what they had mentioned.

“
i
w

3

d
c
s
c
s
s
fi
c

i
t
c

3

t
c
d
a
a
c

(
d
s
o
i
t

c
c
n
t
s

3

w
s
a
t
f
K

p
b
o
a
c
c
t
i

L. Chen / Thinking Skills and Creativity 24 (2017) 140–151 145

In one interview, the student specified that her conceptualization was the only way to define critical thinking. She stated
Only great writers have critical thinking.” When I asked her if a college student could have critical thinking skills, she noted
n very rare cases, when a college student produced publishable work, that would be an example of critical thinking. She

as the only student that conceptualized critical thinking with one absolute definition.

.3. The popular themes among Chinese students’ definitions of critical thinking

The themes and frequency of Chinese students’ definitions are presented in Table 2. Note that most of the Chinese stu-
ents mentioned more than one theme in the interviews so the total frequency is greater than 100%. The most popular
onceptualization is critical thinking as a cognitive skill. 75% of the students mentioned some cognitive terms such as rea-
oning, evidence, or backing up claims in their definitions of critical thinking. 27 students conceptualized critical thinking
oncept as intellectual autonomy, making it the second most commonly mentioned definition among the Chinese college
tudents in this project. 22 students mentioned the omnipresence of the opposite point of view, and 10 students mentioned
ome concept of thinking with multiple perspectives. 7 students mentioned revolutionary thoughts in art, science, and other
elds as critical thinking, and one student mentioned knowledge and skills that are useful in life, but cannot be tested in schools as
ritical thinking. I will explain in the following sections each concept used to characterize the Chinese students’ definitions.

As this is a case study conducted with volunteers, the various ways critical thinking was defined by the Chinese students
n this project does not represent how critical thinking would be defined by all Chinese students across the nation. However,
he analysis of the unique features of Chinese students’ definitions of critical thinking may indicate how Chinese culture
ould influence Chinese students’ understanding of critical thinking.

.3.1. Cognitive thinking skills
Cognitive thinking skills, including logical thinking, problem-solving, and other higher-order thinking strategies, were men-

ioned by 36 students in the interviews making it the most commonly mentioned way of conceptualizing the critical thinking
oncept. The Chinese students offered a variety of examples demonstrating the use of some cognitive thinking skills in aca-
emic settings as critical thinking. For instance, five students regarded logical thinking and problem-solving in math studies
s examples of critical thinking. Six students described decision making using certain thinking strategies in extra curriculum
ctivities, student groups, internship, selecting their majors, and various other contexts to illustrate their understanding of
ritical thinking.

Critical thinking defined as cognitive thinking skills is consistent with the definition provided by many educational scholars
Ennis, 1996; Paul & Elder, 2009). This similarity demonstrates that despite the Confucian cultural influence which values
ifferent methods of reasoning compared to Western reasoning, the concept of logical reasoning is not alien to Chinese college
tudents. Chinese college students do not live in a bubble of Chinese traditional culture. They understand the importance
f logical thinking, problem-solving, and reasoning based upon evidence, at least theoretically. Similar results have been found
n other studies; in Jones (2004), Chinese students taking an economic class in Australia demonstrated an understanding of
he faculty’s critical thinking requirements similar to the Australian students.

However, there is disagreement among Chinese students whether cognitive thinking skills can capture the critical thinking
oncept in this project. While in member checking, most students confirmed that the thinking skills they mentioned are
ritical thinking; however, one student explicitly said, “No, reasoning is like thinking according to certain logic, there’s
othing critical in that.” When I showed this case to the Chinese teachers, both teachers said they were surprised but found
he student’s position understandable. This example demonstrates that defining critical thinking as higher order thinking
kills is not agreed upon by every Chinese student in this project.

.3.2. Intellectual autonomy
In this project, 27 students spoke of their conceptions of critical thinking as having their own ideas. They provided examples

hen their opinions disagreed with their parents or teachers to emphasize the originality of their own ideas. This is the
econd most common definition that Chinese students provided in this project after cognitive thinking skills. Intellectual
utonomy is essential in Kantian philosophy, which proposes that man should be capable of using his own understanding in
ruth seeking without the guidance of another person (Kant, 1996). Students’ confidence to disagree with authority, to want
reedom, and to believe in their own ability to make judgments is a demonstration of the intellectual autonomy valued by
ant and many other Western educators (Facione, Sánchez, Facione, & Gainen, 1995; Howe, 2004; Paul & Elder, 2009).

Intellectual autonomy shares some similarities and differences with the critical thinking disposition (Ennis, 1996) –
eople’s willingness to seek “scientific” conceptualizations of a subject or problem. A critical thinking disposition includes
eing “broad and adventurous,” the desire to “clarify and seek understanding” (p.169), and to “seek alternatives and be
pen to them” (p. 165). In my analysis, there are similarities between critical thinking as a disposition and the intellectual
utonomy emphasized in Chinese students’ definitions of critical thinking. Both concepts emphasize students’ willingness,

ourage, and motivation to question preexisting assumptions. However, there are differences in the emphasis of the two
oncepts. Whereas in Chinese students’ definitions the emphasis lays in students’ ownership of their ideas, in the critical
hinking disposition the emphasis is on using scientific methods to seek truth. The Chinese students emphasized that their
deas of critical thinking are set against an authority’s perspective, which can be scientific or not.

146 L. Chen / Thinking Skills and Creativity 24 (2017) 140–151

Following is an excerpt from a female student, Jin. Jin was the first student who volunteered to be interviewed and very
outspoken. Throughout the interviews, she expressed her personal opinions on a few different issues, including the Chinese
educational system, the management of her university, and the economic development of her hometown. Jin defined critical
thinking as follows:

Author: What do you think the term critical thinking means to your understanding?

Jin: The first thing [in defining critical thinking] is critical, to critique or criticize. Critique, to me, means I have my own
opinions. I have my own opinions towards a particular topic, and my ideas are different from other people, from my
teachers and my parents. And then, I have evidence. I have a reason to support my opinions. That’s it pretty much.
[05/15/2012]

In Jin’s interview, she emphasized that having her own opinions different from the authority is crucial to the critical thinking
concept, and then she had to support her own idea with evidence, which is a demonstration that she valued higher order logical
thinking as part of critical thinking at the same time. When I asked Jin to give an example of what she meant by her own
ideas, she gave an example in which she disagreed with the school policy of mandatory internship at the end of senior year.

Jin: . . .You know that, when the college students do the internship, they [college students] are treated as cheap labor.
A lot of them [students] are not doing major-related work. They [students] are working for better resume, or school
credit. I think this is not worth it. [05/15/2012]

Jin emphasized the words “I,” “my,” and “I think” with intonation in the interview. Those words demonstrate the freedom
of thought that she valued as critical thinking. Her opinions about mandatory internship were against the opinion of the
authority.

Another female student, Hui, expressed similar ideas in her interview.

Author: . . .. . .What do you suppose critical thinking means?

Hui: Critical thinking emphasizes having opinion of my own. The teacher does not put restrictions on me. I have the
freedom to make my own judgment. [05/28/2012]

In Hui’s answer, she regarded her teacher as the authority and critical thinking as the freedom to make her own judgment
without teacher or authority interference, scientifically or not. Hui later gave an example of answering open-ended questions
in social studies classes as an example of critical thinking. In this sense, unlike Jin whose concept of intellectual freedom is
against authority, Hui’s freedom was granted by authority.

In the following example, Shipeng, a male student, discusses his idea of critical thinking. He expressed suspicion of the
teachers’ professional competence to illustrate his conceptualization of critical thinking, although from his point of view,
this spirit of suspicion might not help him in his studies.

Author: What do you think critical thinking means?

Shipeng: Critical thinking is, when your teacher taught something, you don’t take things for granted, but think about
it· · ·Sometimes it may interfere with your study.
Author: Why is that?

Shipeng: Sometimes, I have doubt in what the teacher is talking about in class, I will not listen to the rest of it; I will ignore
it. I would rather search on the Internet for it and learn the stuff myself. Sometimes, maybe the teacher was right, (in that
case) I would miss it (what the teacher has to say). . .Those things may be tested. So to some extent critical thinking
may interfere with my study. But not absolutely. If I can learn everything after the class, then that’s fine. [05/30/2012]

Shipeng questioned his teachers in class and developed his own method to test his hypothesis. Instead of regarding critical
thinking as the desire to seek a scientific conceptualization of a problem (Ennis, 1996), Shipeng emphasized discovering the
conceptualization of a problem in his own way. In the interview, it seems that Shipeng believed his pursuit of critical thinking
might be punished with low scores on tests which value reproduction of the detailed knowledge the teacher imparted rather
than the process of finding out information (Dong, 2015).

During member checking, interviewed students reiterated that they emphasized “having their own ideas” as critical
thinking, while some students reminded me that they defined critical thinking as a multi-faceted concept and “being able
to offer evidence for their own ideas” is also important.

In the interviews with Chinese teachers, one commented on Jin’s definition, “this (students’ ownership of ideas) can
be critical thinking, but this is too simple. It is not enough for students to have their own opinions to demonstrate critical
thinking. The students’ opinions need to be properly defended, reasoned, and examined” (Interview, 09/02/2012). After
some discussion, we agreed that it was not for the researcher or the teachers to judge the students’ definitions, but to seek

to understand them. Thus, the Chinese students’ understanding of critical thinking that emphasizes having their own ideas
was theorized by this project as intellectual autonomy.

L. Chen / Thinking Skills and Creativity 24 (2017) 140–151 147

3.3.3. The omnipresence of opposite point of views
19 (40%) Chinese students in the interviews defined critical thinking as consideration of diverse aspects of an issue when

making a judgment. Those students explicitly mentioned “considering multiple aspects of an issue when making a judgment”
or “being aware that there is (a) con coming with every pro.”

On the surface, such definitions are in agreement with some Western scholars of critical thinking. Blodgett-McDeavitt
(1995) as cited in Johnston et al. (2011) defines critical thinking as “thinking about other alternatives” and “looking at the
bigger scheme of things” (p. 38). However, in my interpretation there is something special about the Chinese version of
“thinking about other alternatives”. Among the 19 students who mentioned considering multiple perspectives/angles, 11 (23%
of overall sample) specifically indicated or offered examples of “positive and negative aspects” ( ) when defining
critical thinking. This description of pros and cons is influenced by the Chinese indigenous philosophy of dialectics in my
analysis.

For example, Pingmei described the process of making a decision about whether to join an extracurricular student group
as an example of critical thinking. The positive side of joining a student group is practicing social skills, and the negative side
is wasting time that should be spent on study. I asked her if, with the co-existence of positive and negative sides, she could
make up her mind. She said the decision could not be made because there was always cons. Eventually, she said the wise
decision could only be made a year later when she became more experienced with college life. Thus, in my analysis, critical
thinking as omnipresence of pros and cons helps the students to realize their own limitations while not necessarily assisting
in decision making.

Shan gave another example where she made a decision based on her need, while critical thinking as the omnipresence of
pros and cons helped her realize her limitations:

Shan: I think critical thinking is a way of thinking, you don’t always think about things in positive ways, there is
negative side in everything too· · ·For example, people around me talk about going to graduate school after college.
With critical thinking, I think about the negative things about it, I won’t be able to earn income for long.

Author: So, how are you going to make the decision based on the pros and cons?

Shan: Actually, I am preparing to apply to graduate school· · ·I think in college, you only learned some surface infor-
mation about a particular field. I want to learn more in-depth things.

Author: So you made a decision against your critical thinking?

Shan: That’s right. I made a decision according to my need. Critical thinking helps me be aware of the negative side of my
decision. [05/28/2012, The emphasis is the author’s.]

In this example, Shan tried to make a decision for herself instead of following the people around her. This can be understood
as intellectual autonomy. Shan also sees critical thinking as an awareness that she needs to think about both the pros and
cons of going to graduate school. However, after weighing the pros and cons, her decision was made based on other factors.

In another example, Meiru explicitly stated that the term she used for defining critical thinking, the theory of two sides
( ), is adopted from her Chinese high school philosophy curriculum.

Author: What do you think critical thinking means?

Fig. 1. Yin-Yang symbol.

148 L. Chen / Thinking Skills and Creativity 24 (2017) 140–151

Meiru: It is like in the politics and philosophy class. . .[We learned that] when thinking about something, you cannot
just look at it from one side, you have to think about it according to the theory of two sides ( ). For one single
matter, it is possible that we would see it from different positions, and perspectives. [05/23/2012]

Meiru adopted a key concept from her Chinese high school philosophy curriculum to illustrate critical thinking. The key
concept was “considering a problem by dividing it into two ( )” (Philosophical Textbook, 2010).

According to Peng, Spencer-Rodgers, and Nian (2006), the Chinese high school philosophy curriculum covers Chinese
Indigenous Philosophy, which was widely misattributed and politically propagated under the cover of Marxism. Meiru’s
understanding of critical thinking was a reflection of Chinese Indigenous Philosophy, although she remembered it as part of
philosophy class.

Nisbett (2003) described the Chinese dialectic (Chinese Indigenous Philosophy) and its difference with Western philos-
ophy in the following passage:

[In Western philosophy], thesis is followed by antithesis, which is resolved by synthesis, and which is “aggressive”
in the sense that the ultimate goal of reasoning is to resolve contradiction. The Chinese dialectic instead uses contra-
diction to understand relations among objects or events. . . In the Chinese intellectual tradition there is no necessary
incompatibility between the belief that A is the case and the belief that not-A is the case. . .A can actually imply that
not-A is also the case, or at any rate soon will be the case. (¨ )̈ . . .It is the Middle Way that is the goal of reasoning.”

Chinese Indigenous Philosophy was adopted by some Chinese students to illustrate their conceptualization of critical
thinking in this project. Chinese Indigenous Philosophy states that in life, matter, and the world, there are always contradic-
tions. In everything black, there is something white. In everything white, there is something black, as illustrated by Yin-Yang
symbol. The balanced harmony is emphasized rather than contradictions. Chinese Indigenous Philosophy has been cited by
many cultural psychology studies to illustrate and understand the thinking process of Chinese lay people (Boucher, Peng,
Shi, & Wang, 2009; Needham, 1954; Peng et al., 2006) (Fig. 1).

There is a similar concept about contradictions in Western philosophy. For instance, Janusian thinking, named for the
Roman god Janus who has two faces, each looking at the opposite direction. However, as Nisbett (2003) described, the
Western idea of contradiction emphasizes the conflict and reasoning process necessary to resolve the contradiction. Janusian
thinking is regarded as a source of creativity by many Western scholars (Rothenberg, 1971). This is quite different from
Chinese Indigenous Philosophy’s emphasis on the harmonious state of two opposites. Chinese Indigenous Philosophy has
been said to have a negative influence on creativity (Heng, 2016) – since there are always going to be cons, it is better to
make peace rather than resolve them. This may explain why interviewed students in this study did not find Indigenous
philosophy to be very useful in decision making.

It is important to point out that not all the interviewed students thought about critical thinking in the context of Chinese
Indigenous Philosophy. Eight noted that they understood the complexities and contradictions between the “positive” and
“negative” sides. Many of them mentioned “one question leads to another” and “there is no answer” in defining critical
thinking. These answers demonstrate an effort to explore and resolve the contradiction, similar to the Western ideas of
critical thinking such as “to be broad and adventurous,” to “clarify and seek understanding” (Ennis, 1996, p. 169), and to
“seek alternatives and be open to them” (p. 165).

In member checking, the Chinese students and both Chinese teachers confirmed that critical thinking can be defined
as considering a problem by dividing it into two ( ), the principle learned from philosophy classes and Chinese
Indigenous Philosophy.

4. Conclusion

Scholars who write about critical thinking concepts rarely consider the voices of students, especially Chinese students, and
even those scholars that do focus on Chinese students’ learning habits rarely problematize the concept of critical thinking.
This project fills this gap and hopes to make Chinese student voices available to researchers. The case study of 46 Chinese
students increases our knowledge of Chinese students’ understanding of the critical thinking concept and how it may be
influenced by sociocultural contexts. This study raises questions around the practice of regarding critical thinking as a
predetermined and universal concept that can be applied in all contexts.

The students in this project tend to mention several ways to conceptualize critical thinking simultaneously in one inter-
view. Very few students were unable to define critical thinking or provide suitable definitions in the interviews. Furthermore,
those students were interested to learn about the concept of critical thinking and were able to understand different concepts
related to critical thinking such as evidence-based reasoning and logical thinking.

When asked to define critical thinking, cognitive thinking skills, intellectual autonomy and the omnipresence of positive and
negative sides were commonly mentioned by the interviewed students. The emphasis on ownership of ideas against authority
is characterized as intellectual autonomy in this project. The students’ understanding of pros and cons and the significance of

harmony was also discussed. Intellectual autonomy and the omnipresence of positive and negative sides reflect the influences
of Chinese Indigenous Philosophy and the Western concept of individualism, among many other influences.

This study challenges the stereotype that Chinese students tend to be obedient, which has been refuted in other studies
such as Grimshaw (2007), an ethnographic study in a university in China. Grimshaw found some Chinese college students

r
s
t
i
a

s
t
s
m
b
e

d
a
a
m
c
(

c
p
c
c

s
s
T
c

A

L
A

•
•
•
•
•
•
•
•
•
•

A

L. Chen / Thinking Skills and Creativity 24 (2017) 140–151 149

ead books on their own instead of paying attention to lectures when they disliked the professor’s teaching. The result of this
tudy echoes previous research conducted on Japanese teachers’ conceptualization of critical thinking (Howe, 2004); critical
hinking defined as individualistic thinking was highly valued in East Asian contexts. Further studies are needed to examine
n more detail how Asian social and cultural contexts affect/promote students’ intellectual autonomy. Furthermore, studies
re needed to analyze how to meet the need of teachers and students to express their authentic opinions in the classroom.

Another feature of the Chinese student definitions of critical thinking is the omnipresence of opposite points of views. It
eems that, influenced by Chinese culture and the high school philosophical curriculum, Chinese students have a tendency
o understand that problems and issues can be analyzed in terms of positive and negative sides. However, some Chinese
tudents appeared to lack the understanding of the complexities involved in such dichotomy, leading to reluctance in decision
aking since there would always be some negative result. This is discussed as a limitation of Chinese Indigenous Philosophy

y some researchers (Dong, 2015). Further studies are needed to explore how to promote thinking beyond merely the
xistence of positive and negative sides when teaching Chinese students.

This finding highlights the complexities and conflicts in defining critical thinking. Some concepts were regarded as a
emonstration of critical thinking for some students, but a lack of it for others. Some students regarded logical reasoning
s critical thinking while others dismissed the idea. It is possible that a Chinese student stating their personal opinions on

controversial topic may be regarded as a demonstration of critical thinking in the student’s perspective. However, faculty
ay not consider this statement critical thinking because it lacks sufficient evidence and reasoning. The ambiguity and

omplexities in defining critical thinking often found by scholars can also be found in the opinions of students and teachers
Kumaravadivelu, 2003; Vandermensbrugghe, 2004).

This project is aware that overemphasizing the differences between Chinese and Western students with regards to
ritical thinking may contribute to stereotyping Chinese students, especially in international education. There are important
arallels in how critical thinking is defined by Chinese students and the Western definitions of critical thinking. The most
ommonly mentioned concept in all the interviews was critical thinking as cognitive thinking skills. This is similar to how
ritical thinking is defined by Western scholars (Bloom, 1956; Ennis, 1996; Paul & Elder, 2009).

There are limitations in this study. The sample in this project is too small to make generalized claims about how Chinese
tudents think across China. The project was conducted by a single author instead of a team. Further study with a larger data
et is needed to examine if there are other prominent patterns in the definitions of critical thinking among Chinese students.
his project is a teaser proposing methods to study critical thinking from student perspectives rather than establishing
ritical thinking as a standardized and pre-determined concept.

cknowledgement

I would also like to show my gratitude to my colleagues from University of Delaware, including Eugene Matusov, Anselmo
ima, Katherine von Duyke, and Rosalie Rolón Dow, for their assistance, guidance and comments.

ppendix A. Interview protocol.

What is your major of study?
How far are you in your study?
Have you heard the term “critical thinking” before?
Have you taken any courses concerning critical thinking or other thinking skills?
What do you think “critical thinking” means?
Can you give an example of “critical thinking”?
Can you give an example when you applied “critical thinking” in your work?
Do you think “critical thinking” is useful for your study? How?
Can you give an example when your teacher nurtured your “critical thinking”?
Do you think critical thinking might be useful for your future career?

ppendix B. Chinese transcript of the selected interview.

150 L. Chen / Thinking Skills and Creativity 24 (2017) 140–151

Appendix C. Example of codes and interview transcripts.

Themes Examples of Students’ definitions
(in-vivo codes)

Examples of critical thinking given by the students in the
interviews

Frequency

Cognitive thinking
skills

Quality reasoning and sound judgment
Logical thinking
Problem-solving
Evidence based thinking
Deliberate and controlled thinking
strategies
Argumentative writing

“Try to look at an issue from different perspectives so you
can be more objective.”
“Solving a math problem with your own method.”
“I have a reason to support my opinion.”
“When I make a decision, it is not just according to my
feeling, but I write it down, think (it) over carefully, and
evaluate different choices.”
“To solve a math problem, I ask myself, what do I need to
know to answer this question? It’s a controlled process,
step by step.”
“In writing class, the teacher tried to make us argue with
him about a variety of issues, like, is using Weibo (Chinese
Twitter) a positive or negative thing for a college student?”
“I think my writing teacher really wants us to develop
critical thinking and he looks for that when he evaluates
students’ work.”
“Critical thinking skill is used a lot in debate competition.”

36 students.
75%

Intellectual autonomy Having my own ideas.
My ideas are unique.
I have the freedom to make my own
judgment.
I have an idea that is different than the
teachers or the parents.
I do my own research.
The students have freedom to use their
imagination. There is no suppressing
the students.

“I think mandatory internship is hurtful to students’
development.”
“I want to apply to graduate school but my parents want
me to find a job. I will continue with my own plan despite
their disagreement.”
“If I have doubts about what the teacher is talking about in
class, I would stop listening to her and do my own
research.”
“When the teacher introduces us to something, especially
some opinions, I should not just accept it, but also think
about it.”

27 students.
56%

Omnipresence of the
opposite point of
view

Not always thinking something as
positive, but also seeing the negative
side in it.
You can agree with something, and
disagree with it at the same time. Not
one-sided opinion.
There is something good and
something bad about everything.
Critique the things that are bad, and
continue with the things that are good.
Evaluate the pros and cons

“For instance, in the history class, whenever you are asked
to evaluate something, you always say the positive side
and the negative side of the issue. The positive side of the
revolution is that it leads the country to a better society;
the bad side is many people are sacrificed in the wars.”
“I can only give an example of my action that lacks critical
thinking. During the first semester in freshman year, I
planned to join a student group. I thought about what kind
of benefits those student groups can offer me. I didn’t
really pay attention to the disadvantages of joining them.”

11 students.
23%

Multiple perspectives
(More than two, and
aware of the
conflicts)

Thinking in a variety of perspectives.
Torn between different perspectives

“Some movies or novels made me think critically,
like· · ·Fight Club?”
“In some novels, there are no bad people but there’s no
real solution to the problem in the novel. I am torn
between different perspectives.”
“Sometimes, the teacher in social studies may present us
with a social problem that just inspires us to think, like the
vast difference between cities and countryside in China,
what caused the problem, what is the solution, one
question leads to another? No easy answer.”

8 students.
17%

Revolutionary thoughts
in art, science and
other fields

Writers.
Scientists.
Philosophers.
People who revolutionize their fields.

“Only great writers have critical thinking, like Lu Xun. He
critiqued the society of his time with deep insight.”
“I think what Copernicus did, that was an example of
critical thinking.”
“Critical thinking makes me think of philosophers like
Kant, Descartes, and Marx.”

7 students
15%

Self-explorative study I did my own research “Sometimes, we will be asked to do a project, a research
project on a certain topic. There’s no definite answer. We
just looked for different sources. It is true inquiry.”

2 students.
4%

Knowledge and skills
outside the scope of
standard

The knowledge and skills that are
useful in real life, but cannot be tested.

“I didn’t come to college to pass the exams. University is
not just about studying, or exams. The stuff that is useful in
real life but cannot be tested, the extra stuff, that is critical

1 students
2%

examination thinking.”

R

A
B
B

C
C
C
C

C

D

D
D

D

D
E
F
G
G

H
H

H

J

J
J

K
K
K

K
L
L

L
M

M
N
N
P

P
P

R
R
R

S

S
T
T
T

V

W
W

W
Y

Z

L. Chen / Thinking Skills and Creativity 24 (2017) 140–151 151

eferences

tkinson, D. (1997). A critical approach to critical thinking in TESOL. TESOL Quarterly, 31(1), 71–94.
loom, B. S. (1956). Taxonomy of educational objectives. Vol. 1: Cognitive domain. pp. 20–24. New York: McKay.
oucher, H. C., Peng, K., Shi, J., & Wang, L. (2009). Culture and implicit self-esteem: Chinese are good and bad at the same time. Journal of Cross-Cultural

Psychology, 40(1), 24–45.
affarella, R., & Merriam, S. B. (2000). Linking the individual learner to the context of adult learning. Handbook of Adult and Continuing Education, 55–70.
han, N., Ho, I. T., & Ku, K. Y. (2011). Epistemic beliefs and critical thinking of Chinese students. Learning and Individual Differences, 21(1), 67–77.
ho, J., & Trent, A. (2006). Validity in qualitative research revisited. Qualitative Research, 6(3), 319–340.
lark, R., & Gieve, S. N. (2006). On the discursive construction of ‘the Chinese learner’. Language Culture and Curriculum, 19(1), 54–73.

http://dx.doi.org/10.1080/07908310608668754
reswell, J. W., & Clark, V. L. P. (2007). Designing and conducting mixed methods research. California: Sage Publications, Inc.
avidson, B. W. (1995). Critical thinking education faces the challenge of Japan. Inquiry: Critical Thinking Across the Disciplines, 14(3), 41–53.
ong, T., Anderson, R. C., Kim, I., & Li, Y. (2008). Collaborative reasoning in China and korea. Reading Research Quarterly, 43(4), 400–424.
ong, Y. (2015). Critical thinking education with Chinese characteristics. In M. Davies, & R. Barnett (Eds.), The Palgrave handbook of critical thinking in

higher education (pp. 351–368). New York: Springer.
urkin, K. (2008). The adaptation of East Asian masters students to western norms of critical thinking and argumentation in the UK. Intercultural

Education, 19(1), 15–27.
wyer, C. P., Hogan, M. J., & Stewart, I. (2014). An integrated critical thinking framework for the 21 st century. Thinking Skills and Creativity, 12, 43–52.
nnis, R. H. (1996). Critical thinking dispositions: Their nature and assessability. Informal Logic, 18(2), 165–182.
acione, P. A., Sánchez, C. A., Facione, N. C., & Gainen, J. (1995). The disposition toward critical thinking. The Journal of General Education, 44(1), 1–25.
raham, A. C. (1989). Disputers of the Tao: Philosophical argument in ancient China. La Salle: Open Court.
rimshaw, T. (2007). Problematizing the construct of ‘the Chinese learner’: insights from ethnographic research. Educational Studies, 33(3), 299–311.

http://dx.doi.org/10.1080/03055690701425643
ammersley, M., & Atkinson, P. (1995). Ethnography: Practices and principles. New York: Routledge.
eng, T. T. (2016). Different is not deficient: Contradicting stereotypes of Chinese international students in US higher education. Studies in Higher

Education, 1–15.
owe, E. R. (2004). Canadian and Japanese teachers’ conceptions of critical thinking: A comparative study. Teachers and Teaching, 10(5), 505–525.

i, L., Peng, K., & Nisbett, R. E. (2000). Culture, control, and perception of relationships in the environment. Journal of Personality and Social Psychology,
78(5), 943.

ohnston, B., Ford, P., Mitchell, R., & Myles, F. (2011). Developing student criticality in higher education: Undergraduate learning in the arts and social sciences.
London: Bloomsbury Publishing.

ones, A. (2004). Teaching critical thinking: An investigation of a task in introductory macroeconomics. Higher Education Research & Development, 23(2),
167–181.

ant, E. (1785/1996). The metaphysics of morals: Cambridge texts in the history of philosophy. Cambridge: Cambridge University Press.
im, H. (2003). Critical thinking, learning and confucius: A positive assessment. Journal of Philosophy of Education, 37(1), 71–87.
u, K. Y. (2009). Assessing students’ critical thinking performance: Urging for measurements using multi-response format. Thinking Skills and Creativity,

4(1), 70–76.
umaravadivelu, B. (2003). Problematizing cultural stereotypes in TESOL. TESOL Quarterly, 37(4), 709–719.
ave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge: Cambridge University Press.
i, H., Rao, N., & Tse, S. K. (2012). Adapting western pedagogies for Chinese literacy instruction: Case studies of Hong Kong, Shenzhen, and Singapore

preschools. Early Education & Development, 23(4), 603–621.
incoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage.
iyamoto, Y., & Wilken, B. (2013). Cultural differences and their mechanisms. In D. Reisberg (Ed.), Oxford Handbook of Cognitive Psychology (pp. 970–985).

Oxford: Oxford University Press.
oore, T. (2013). Critical thinking: Seven definitions in search of a concept. Studies in Higher Education, 38(4), 506–522.
eedham, J. (1954). . Science and civilization in China (Vol. I) Cambridge: Cambridge University Press.
isbett, R. E. (2003). The geography of thought: How Asians and Westerners think differently and why. New York: Free Press.
aton, M. (2005). Is critical analysis foreign to Chinese students. In E. Manalo, & G. Wong-Toi (Eds.), Communication skills in university education: The

international dimension (pp. 1–11). Auckland, New Zealand: Person Education New Zealand.
aul, R., & Elder, L. (2009). Critical thinking: Ethical reasoning and fairminded thinking, part I. Journal of Developmental Education, 33(1), 36.
eng, K., Spencer-Rodgers, J., & Nian, Z. (2006). Naïve dialecticism and the Tao of Chinese thought. In U. Kim, K. Yang, & G. Huang (Eds.), Indigenous and

cultural psychology: Understanding people in contexts (pp. 247–262). New York: Springer.
ogoff, B. (2003). The cultural nature of human development. Oxford: Oxford University Press.
othenberg, A. (1971). The process of Janusian thinking in creativity. Archives of General Psychiatry, 24(3), 195–205.
yan, J., & Slethaug, G. (Eds.). (2010). International education and the Chinese learner (Vol. 1). Hongkong: Hong Kong University Press.

hao, W. (2013). . A brief review of the core value of nurturing critical
thinking and self-directed knowledge construction in current higher education. University Education, 2013, 14.

trauss, A. L. (1987). Qualitative analysis for social scientists. Cambridge: Cambridge University Press.
hayer-Bacon, B. (1992). Is modern critical thinking theory sexist? Inquiry: Critical Thinking Across the Disciplines, 10(1), 3–7.
obin, J., Hsueh, Y., & Karasawa, M. (2009). Preschool in three cultures revisited: China, Japan, and the United States. Chicago: University of Chicago Press.
urner, Y. (2006). Students from mainland china and critical thinking in postgraduate business and management degrees: Teasing out tensions of culture,

style and substance. International Journal of Management Education, 5(1), 3–11.
andermensbrugghe, J. (2004). The unbearable vagueness of critical thinking in the context of the Anglo-Saxonisation of education. International

Education Journal, 5(3), 417–422.
egerif, R., Li, L., & Kaufman, J. C. (2015). The Routledge international handbook of research on teaching thinking. London: Routledge.
ineburg, S. (1998). Reading Abraham Lincoln: An expert/expert study in the interpretation of historical texts. Cognitive Science, 22(3), 319–346.

u H. (2011). (Critical thinking: study of the semantics) Journal of Yanan University (Social Science), 33(1), 5–17.
ang, Y. (2016). Lessons learnt from contextualising a UK teaching thinking program in a conventional Chinese classroom. Thinking Skills and Creativity,

19, 198–209.
hang, H., & Lambert, V. (2008). Critical thinking dispositions and learning styles of baccalaureate nursing students from china. Nursing & Health Sciences,

10(3), 175–181.

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0005

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0005

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0005

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0005

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0005

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0005

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0005

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0005

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0005

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0005

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0005

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0005

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0005

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0005

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0005

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0005

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0010

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0010

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0010

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0010

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0010

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0010

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0010

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0010

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0010

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0010

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0010

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0010

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0010

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0010

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0010

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0010

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0015

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0015

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0015

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0015

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0015

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0015

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0015

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0015

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0015

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0015

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0015

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0015

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0015

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0015

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0015

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0015

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0015

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0015

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0015

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0015

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0015

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0015

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0015

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0020

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0020

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0020

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0020

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0020

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0020

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0020

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0020

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0020

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0020

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0020

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0020

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0020

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0020

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0020

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0020

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0020

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0020

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0020

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0020

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0025

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0025

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0025

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0025

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0025

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0025

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0025

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0025

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0025

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0025

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0025

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0025

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0025

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0025

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0025

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0025

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0025

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0025

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0030

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0030

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0030

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0030

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0030

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0030

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0030

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0030

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0030

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0030

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0030

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0030

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0030

dx.doi.org/10.1080/07908310608668754

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0040

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0040

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0040

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0040

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0040

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0040

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0040

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0040

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0040

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0040

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0040

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0045

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0045

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0045

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0045

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0045

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0045

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0045

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0045

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0045

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0045

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0045

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0045

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0045

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0045

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0045

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0045

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0045

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0045

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0045

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0045

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0050

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0050

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0050

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0050

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0050

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0050

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0050

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0050

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0050

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0050

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0050

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0050

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0050

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0050

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0050

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0055

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0055

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0055

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0055

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0055

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0055

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0055

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0055

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0055

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0055

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0055

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0055

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0055

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0055

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0055

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0055

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0055

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0055

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0055

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0055

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0055

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0055

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0055

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0055

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0055

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0055

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0055

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0055

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0055

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0060

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0060

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0060

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0060

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0060

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0060

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0060

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0060

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0060

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0060

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0060

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0060

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0060

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0060

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0060

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0060

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0060

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0060

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0060

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0060

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0060

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0060

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0060

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0060

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0060

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0060

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0065

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0065

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0065

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0065

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0065

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0065

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0065

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0065

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0065

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0065

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0065

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0065

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0065

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0065

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0065

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0065

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0065

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0065

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0065

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0065

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0070

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0070

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0070

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0070

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0070

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0070

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0070

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0070

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0070

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0070

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0070

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0070

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0070

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0070

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0070

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0075

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0075

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0075

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0075

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0075

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0075

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0075

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0075

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0075

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0075

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0075

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0075

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0075

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0075

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0075

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0075

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0080

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0080

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0080

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0080

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0080

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0080

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0080

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0080

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0080

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0080

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0080

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0080

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0080

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0080

dx.doi.org/10.1080/03055690701425643

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0090

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0090

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0090

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0090

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0090

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0090

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0090

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0090

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0095

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0095

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0095

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0095

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0095

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0095

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0095

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0095

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0095

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0095

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0095

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0095

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0095

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0095

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0095

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0095

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0095

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0095

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0095

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0095

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0095

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0095

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0100

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0100

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0100

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0100

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0100

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0100

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0100

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0100

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0100

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0100

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0100

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0100

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0100

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0100

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0100

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0100

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0100

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0100

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0100

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0100

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0105

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0105

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0105

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0105

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0105

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0105

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0105

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0105

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0105

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0105

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0105

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0105

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0105

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0105

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0105

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0105

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0105

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0105

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0105

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0110

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0110

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0110

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0110

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0110

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0110

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0110

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0110

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0110

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0110

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0110

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0110

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0110

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0110

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0110

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0110

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0110

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0110

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0115

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0115

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0115

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0115

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0115

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0115

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0115

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0115

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0115

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0115

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0115

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0115

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0115

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0115

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0115

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0115

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0115

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0115

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0115

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0115

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0115

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0115

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0125

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0125

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0125

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0125

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0125

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0125

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0125

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0125

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0125

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0125

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0125

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0125

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0125

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0125

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0125

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0125

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0125

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0125

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0125

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0130

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0130

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0130

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0130

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0130

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0130

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0130

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0130

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0130

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0130

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0130

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0130

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0130

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0130

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0130

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0130

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0130

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0130

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0130

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0130

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0130

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0135

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0135

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0135

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0135

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0135

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0135

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0135

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0135

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0135

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0135

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0135

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0135

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0135

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0140

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0140

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0140

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0140

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0140

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0140

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0140

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0140

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0140

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0140

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0145

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0145

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0145

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0145

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0145

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0145

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0145

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0145

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0145

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0145

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0145

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0145

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0145

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0145

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0145

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0145

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0145

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0145

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0145

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0145

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0145

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0145

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0145

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0145

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0145

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0145

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0150

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0150

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0150

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0150

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0150

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0150

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0150

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0155

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0155

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0155

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0155

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0155

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0155

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0155

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0155

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0155

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0155

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0155

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0155

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0155

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0155

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0155

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0155

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0155

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0155

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0155

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0155

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0155

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0155

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0160

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0160

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0160

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0160

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0160

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0160

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0160

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0160

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0160

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0160

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0160

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0160

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0160

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0160

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0160

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0160

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0160

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0160

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0160

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0165

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0165

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0165

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0165

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0165

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0165

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0165

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0165

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0165

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0165

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0165

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0165

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0170

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0170

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0170

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0170

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0170

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0170

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0170

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0170

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0170

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0170

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0170

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0170

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0170

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0170

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0170

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0170

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0170

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0175

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0180

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0180

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0180

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0180

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0180

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0180

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0180

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0180

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0180

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0180

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0180

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0180

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0180

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0180

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0180

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0180

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0180

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0185

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0190

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0190

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0190

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0190

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0190

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0190

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0190

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0190

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0190

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0190

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0190

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0195

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0195

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0195

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0195

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0195

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0195

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0195

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0195

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0195

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0195

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0195

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0195

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0195

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0195

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0195

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0195

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0195

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0200

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0200

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0200

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0200

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0200

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0200

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0200

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0200

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0200

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0200

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0200

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0200

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0200

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0210

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0210

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0210

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0210

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0210

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0210

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0210

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0210

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0210

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0210

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0215

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0215

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0215

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0215

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0215

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0215

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0215

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0215

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0215

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0215

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0215

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0215

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0215

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0215

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0215

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0215

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0215

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0215

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0220

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0220

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0220

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0220

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0220

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0220

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0220

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0220

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0220

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0220

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0220

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0220

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0220

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0220

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0220

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0220

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0220

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0225

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0230

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0230

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0230

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0230

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0230

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0230

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0230

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0230

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0230

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0230

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0230

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0230

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0230

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0230

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0230

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0230

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0230

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0230

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0230

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0230

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0230

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0230

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0230

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0235

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0235

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0235

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0235

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0235

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0235

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0235

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0235

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0235

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0235

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0235

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0235

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0240

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0240

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0240

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0240

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0240

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0240

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0240

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0240

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0240

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0240

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0240

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0240

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0240

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0240

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0240

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0240

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0240

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0240

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0240

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0240

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0250

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0250

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0250

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0250

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0250

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0250

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0250

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0250

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0250

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0250

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0250

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0250

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0250

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0250

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0250

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0250

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0250

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0250

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0250

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0250

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0250

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0250

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0250

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0250

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0255

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0255

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0255

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0255

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0255

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0255

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0255

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0255

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0255

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0255

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0255

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0255

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0255

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0255

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0255

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0255

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0255

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0255

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0255

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0255

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0255

http://refhub.elsevier.com/S1871-1871(17)30054-8/sbref0255

• Understanding critical thinking in Chinese sociocultural contexts: A case study in a Chinese college

1 Introduction
1.1 The construction of critical thinking in Western literature
1.2 Critical thinking concept in Chinese educational literature
1.3 Sociocultural theories of thinking and learning
2 Methods
2.1 Participants
2.2 Interviews
2.3 Analysis
3 Results
3.1 The Chinese college students’ abilities to define critical thinking
3.2 The multiplicity of students’ definitions
3.3 The popular themes among Chinese students’ definitions of critical thinking
3.3.1 Cognitive thinking skills
3.3.2 Intellectual autonomy
3.3.3 The omnipresence of opposite point of views

4 Conclusion
Acknowledgement
Appendix A Interview protocol.
Appendix B Chinese transcript of the selected interview.
Appendix C Example of codes and interview transcripts.
References

Learning and Individual Differences 20 (2010) 604–616

Contents lists available at ScienceDirect

Learning and Individual Differences

journal homepage: www.elsevier.com/locate/lindif

Exploring cultural differences in critical thinking: Is it about my thinking style or the
language I speak?

Vivian Miu-Chi Lun a,b,⁎, Ronald Fischer a, Colleen Ward a

a School of Psychology and Centre for Applied Cross-Cultural Research, Victoria University of Wellington, PO Box 600, Wellington, New Zealand
b Department of Management, City University of Hong Kong, Tat Chee Avenue, Kowloon Tong, Kowloon, Hong Kong, China

⁎ Corresponding author. Department of Management
Tat Chee Avenue, Kowloon Tong, Kowloon, Hong Kong,
fax: +852 3442 0309.

E-mail addresses: vivian.lun@cityu.edu.hk (V.M.-C. L
(R. Fischer), colleen.ward@vuw.ac.nz (C. Ward).

1041-6080/$ – see front matter © 2010 Elsevier Inc. Al
doi:10.1016/j.lindif.2010.07.001

a b s t r a c t

a r t i c l e i n f o

Article history:
Received 8 January 2009
Received in revised form 29 June 2010
Accepted 4 July 2010

Keywords:
Critical thinking
Cultural differences
English proficiency
Dialectical thinking
International education

Critical thinking is deemed as an ideal in academic settings, but cultural differences in critical thinking
performance between Asian and Western students have been reported in the international education
literature. We examined explanations for the observed differences in critical thinking between Asian and
New Zealand (NZ) European students, and tested hypotheses derived from research in international
education and cultural psychology. The results showed that NZ European students performed better on two
objective measures of critical thinking skills than Asian students. English proficiency, but not dialectical
thinking style, could at least partially if not fully explain these differences. This finding holds with both self-
report (Study 1) and objectively measured (Study 2a) English proficiency. The results also indicated that
Asian students tended to rely more on dialectical thinking to solve critical thinking problems than their
Western counterparts. In a follow-up data analysis, students’ critical thinking was found to predict
their academic performance after controlling for the effects of English proficiency and general intellectual
ability, but the relationship does not vary as a function of students’ cultural backgrounds or cultural adoption
(Study 2b). Altogether, these findings contribute to our understanding of the influence of culture on critical
thinking in international education.

, City University of Hong Kong,
China. Tel.: +852 3442 5679;

un), ronald.fischer@vuw.ac.nz

l rights reserved.

© 2010 Elsevier Inc. All rights reserved.

1. Introduction

Cultivation of critical thinking has been considered an important
goal of tertiary education, especially in today’s learning environment
where students are exposed to tremendous amount of information
which requires effective cognitive strategies to process. Despite the
apparent consensus of the importance of critical thinking (Pithers &
Soden, 2000), the influence of culture on critical thinking and its
instruction is not clear (ten Dam & Volman, 2004). In particular, there
are debates about whether Asian students show lower level of critical
thinking in comparison to their Western counterparts, and conse-
quentially whether critical thinking instruction can be applied to
students of diverse cultural backgrounds (e.g., Atkinson, 1997; Ennis,
1998; Kumaravadivelu, 2003). The present research seeks to
empirically address whether there is a difference in critical thinking
between Asian and Western students, and examines the possible
explanations and impact if such a difference exists.

1.1. Do Asian students think less critically?

The current trend of tertiary education involves an increasing
number of Asian students studying at a Western English-speaking
institution (OECD, 2009). With regard to the development of critical
thinking in such a context, concern about whether Asian students
tend to think less critically in contrast to their Western counterparts
has gained much attention (Atkinson, 1997; Kumaravadivelu, 2003).

Research showed that academic staffs experienced in teaching
international students are dissatisfied with the international students’
poor critical thinking and analytical skills (Robertson, Line, Jones, &
Thomas, 2000). Academics often express that Asian students do not
naturally take part in critical thinking because they do not overtly
participate in classroom discussions (cf. Paton, 2005). Lee and
Carrasquillo (2006) found that college professors perceived their Korean
students as having “difficulty in openly expressing critical thinking”
(p. 451). The perceptions of Asian students being less overt or less
expressive in classrooms are often interpreted as lack of critical thinking,
especially when Western academic standards of critical thinking
involving overt argumentation and debate are applied (Durkin, 2008).

The claim that Asian students think less critically has been based
predominantly on the observations by teaching professionals of
English-second-language (ESL)learners. Nevertheless, there iscurrently
little empirical research directly examining differences in critical
thinking between Asian students and their Western counterparts.

http://dx.doi.org/10.1016/j.lindif.2010.07.001

mailto:vivian.lun@cityu.edu.hk

mailto:ronald.fischer@vuw.ac.nz

mailto:colleen.ward@vuw.ac.nz

http://dx.doi.org/10.1016/j.lindif.2010.07.001

http://www.sciencedirect.com/science/journal/10416080

605V.M.-C. Lun et al. / Learning and Individual Differences 20 (2010) 604–616

1.2. Critical thinking — a multi-dimensional construct

At this point, the term critical thinking requires clarification. In the
international education literature, the term is mostly used to refer to a
set of behaviors which is assumed to truly reflect critical thinking
skills (e.g., Atkinson, 1997). These behaviors often include overt
questioning or debating of ideas in a classroom setting (Durkin, 2008).
Teaching professionals observe these behaviors and infer whether the
students show critical thinking. These observations then lead the
educators to conclude whether the students have critical thinking or if
they are critical thinkers.

However, besides behavioral expressions, critical thinking, as a
form of thinking, involves important cognitive skills and dispositions.
A critical thinker is willing, persistent, flexible, open-minded and
confident in using certain cognitive skills when it is necessary and
appropriate (Ennis, 1987; Facione, 1990; Halpern, 1998, 2003).
Pascarella and Terenzini (2005) summarized various definitions of
critical thinking in the literature and suggested that college-level
critical thinking skills include identifying assumptions behind an
argument, recognizing important relationships, making correct
references from data, drawing conclusions from the information or
data provided, interpreting the merit of a conclusion based on
available information, evaluating the credibility of a statement and its
source, and making self-corrections. The ultimate goal of critical
thinking is to make decisions or solve problems in different situations
with the appropriate use of these critical thinking skills (Ennis, 1987;
Halpern, 1998). The term critical thinking, as theorized by Ennis
(1987) and Halpern (1998), represents a set of cognitive skills and
dispositions which are conducive to decision making and problem
solving in different situations.

The concepts of critical thinking and intelligence appear similar
and are indeed closely related, but critical thinking is more often
accepted as teachable and improvable with appropriate instruction
(Nickerson, Perkins, & Smith, 1985; Halpern, 2007). Nickerson et al.’s
(1985) view of thinking skills as good strategies helps to distinguish
between critical thinking and intelligence. Their idea was that
thinking skills are a consequence of education, while intelligence
relates more to the “raw power” of an individual’s mental equipment:
“raw power is one thing and the skilled use of it (is) something else”
(p. 44). Because it is more probable to teach skills rather than the
supposedly unchangeable raw mental power, critical thinking has
drawn much attention in psychology and education (e.g., Halpern,
1998; Pithers & Soden, 2000; ten Dam & Volman, 2004).

1.3. Focus of the present research

The debate in the international education literature has mainly
been focused on the behavioral expression of critical thinking (e.g.,
Atkinson, 1997; Kumaravadivelu, 2003). Nevertheless, observation of
classroom behaviors is susceptible to the influence of different
cultural rules and assumptions underlying behaviors such as talking
(Kim, 2002) or debating and questioning authorities (Tweed &
Lehman, 2002). Here we focus on critical thinking as a skill because
(a) the cultivation of critical thinking skills is a principal goal of
university education (Pithers & Soden, 2000), and (b) these skills can
be objectively measured.

1.4. Cultural differences in critical thinking

To examine if Asian students really think less critically compared
with their Western counterparts, one of the best ways is to test and
compare the critical thinking skills between these two groups of
students. Nevertheless, as ten Dam and Volman (2004) observed,
empirical research which addresses the role of culture in relation to
critical thinking is currently limited. To the best of our knowledge, there
is no published study involving direct comparison of critical thinking

skills betweenAsian and Westernstudents in aninternational education
setting. To fill this gap in the literature, the present study directly tests
whether there are objective differences in critical thinking skills
between Asian and Western students who are studying at a Western
institution.

Based on the assumption that the observations made by university
teaching professionals (e.g., Lee & Carrasquillo, 2006; Robertson et al.,
2000) reflect a kernel of truth, we hypothesize that Western students
would perform better than Asian students in an objective measure of
critical thinking (Hypothesis 1).

1.5. Dialectical thinking

The literature on dialectical thinking proposed one causal
mechanism to explain why critical thinking performances may be
lower among Asian students compared to Western students (Nisbett,
Peng, Choi, & Norenzayan, 2001; Peng & Nisbett, 1999). According to
this paradigm, the Asian way of information processing can be
summarized by three principles: 1) reality is dynamic and change-
able; 2) opposing propositions may exist in the same object or event;
and 3) everything in life and nature is related (Nisbett et al., 2001).
Asians, compared with Westerners, tend to perceive more changes,
are more tolerant to contradictions, and perceive things as more
interrelated. These three characteristics involved in the Asian way of
thinking are suggested to be incongruent with the formal logical
tradition of thinking which is dominant in Western cultures (Peng &
Nisbett, 1999).

These cultural thinking styles have been shown to have important
influence on certain psychological and behavioral differences be-
tween Asian and Westerners. In a seminal study, Norenzayan, Smith,
Kim, and Nisbett (2002) found that Westerners were more likely to
use formal logical rules in reasoning, whereas East Asians used more
intuitive and experience-based reasoning when there was a conflict
between intuitive and formal reasoning strategies. In one of their
experiments, European American and Korean students were pre-
sented with a set of arguments and asked to evaluate whether or not a
conclusion followed logically from the premises for each argument.
When faced with a conclusion that logically follows the arguments but
is intuitively unbelievable, Korean students tended to make more
mistakes in judging the conclusion as invalid than the American
students, indicating that they relied more on their own experience but
less on formal logical rules in deductive reasoning than their
American counterparts. Norenzayan et al. (2002) speculated that
the pedagogical emphasis on critical thinking in Western classrooms
compared with the experience-based approach in Asian classrooms
may be one of the reasons of the different modes of thinking in the
two cultural systems. Therefore, these laboratory experiments
suggested that dialectical thinking and critical thinking are linked,
but in an opposite direction. A preference for dialectical thinking may
prompt Asians students to seek a “middle-way” between apparent
contradictions more than their Western counterparts (Peng & Nisbett,
1999, 2000) and to choose intuitive reasoning over formal reasoning
(Norenzayan et al., 2002), which then may explain the noted
difference on critical thinking performance between Asian and
Western students. Therefore, we expect that dialectical thinking
style mediates the differences in critical thinking skills between Asian
and Western students (Hypothesis 2).

This hypothesis is based on the assumption that dialectical thinking is
applicable to both Western and Asian students. Based on the different
historical developments of philosophical thoughts in the East and the
West, dialecticalthinking is suggested to be more typicalofAsiancultures
compared to Western contexts, where analytical thinking styles are more
typical (Nisbett et al., 2001). Individuals within each of these cultural
contexts differ in terms of their propensity to engage in each of these
thinking styles. In other words, dialectical thinking is treated as an
individual difference variable that shows variability across cultural

606 V.M.-C. Lun et al. / Learning and Individual Differences 20 (2010) 604–616

contexts and can be used to explain differences in objective measures of
critical thinking (Matsumoto & Yoo, 2006).

Using individual difference variables to account for observed cultural
differences has become increasingly popular because of its effectiveness
to account for the active ingredients of cultural differences in
psychological variables (Leung & van de Vijver, 2008). However, this
approach assumes that the mediating variable is similarly related to the
target variable in all cultural groups involved. Although this rationale
underlies the development of individual difference measures of
dialectical thinking, and these scales have been used in past research
to explain cultural differences in other psychological variables and
behaviors (e.g., Hamamura, Heine, & Paulhus, 2008; Spencer-Rodgers,
Peng, Wang, & Hou, 2004), it is uncertain whether dialectical thinking
style is similarly associated with critical thinking skills in both European
and Asian student populations. Peng and Nisbett (1999, 2000)
suggested that dialectical thinking style reflects philosophical tradition
prevalent in Eastern cultures, whereas Western culture is characterized
by analytical thinking style, which has a root of Greek philosophical
tradition. Therefore, these thinking styles may be qualitatively different
and may not relate to critical thinking in one cultural group or another.
Such qualitative differences in their functioning can weaken the
usefulness of thinking styles in accounting for cultural differences in
critical thinking. Therefore, a secondary aim of our analysis is to explore
whether dialectical thinking shows the same relationship with critical
thinking in both Asian and European student populations. The findings
will have important implications for the conceptualization of dialectical
thinking across cultural groups.

1.6. English proficiency

An alternative hypothesis for explaining cultural difference in
critical thinking relates to Asian students’ is the use of English as a
second language (Paton, 2005). In an international education setting,
Asian international or immigrant students usually possess lower level
of English proficiency than their native English-speaking counter-
parts. As a result, these Asian students may be more “cognitively
overloaded” when using English in tasks requiring critical thinking.
English proficiency then accounts for their lower critical thinking
performance. Campbell, Adams, and Davis (2007) suggested similar
processes of increased cognitive demand for ESL students in solving
mathematical problems. According to the cognitive load theory (Paas,
Renkl, & Sweller, 2003), only limited amount of information can be
stored and processed in working memory. Using this theoretical
framework, Campbell et al. (2007) illustrated that ESL learners
required some of that limited working memory to process informa-
tion related to the unfamiliar language and structure of word
problems, which overloaded their working memory and prevented
them from effectively processing information pertinent to the
question per se. Similar ideas can be applied to ESL learners who
are required to think critically about problems presented in English.

In fact, the use of a second language has been shown to have
detrimental effect on one’s performance in cognitive tasks. Takano and
Noda (1993) observed that native-Japanese speakers who had to use
English as their second language to work on a linguistic task performed
less well in a concurrent calculation task than they would in using
Japanese on the linguistic task. The same deteriorating effect was also
found among native English speakers who had to use Japanese on the
linguistic task. In light of these findings, Takano and Noda (1993)
suggested that the use of a foreign language would cause temporary
decline of thinking ability as a result of heavier cognitive processing load.

Language proficiency has been found to play an important role in
students’ critical thinking performance. Previous research on the
predictors of university students’ critical thinking test performance
showed that verbal scores in the Scholastic Aptitude Test (SAT) were
significantly and positively related to critical thinking skills as well as
grade point average (Taube, 1997). Clifford, Boufal, and Kurtz (2004)

also observed college students’ scores on the verbal comprehension
subtest of the Wechsler Adult Intelligence Scale-Third Edition (WAIS-III;
Wechsler, 1997) being significant predictor of their critical thinking test
scores. Although these studies did not address the issue of culture, the
findings suggested a positive relationship between language proficiency
and critical thinking performance, which may be used to explain the
difference in critical thinking between Asian and Western students.
Accordingly, we hypothesize that English proficiency explains cultural
differences in critical thinking (Hypothesis 3).

1.7. Possible consequence of cultural differences in critical thinking

An important implication of observing cultural difference in
critical thinking concerns the practice of critical thinking in interna-
tional education. If Asian students were indeed less capable of critical
thinking than their Western counterparts, instructions emphasizing
critical thinking might become something undesirable to Asian
students. The increasing number of Asian students studying in
Western institutions (OECD, 2009) might then undermine the values
of cultivating critical thinking in university education (see Atkinson,
1997; Davidson, 1998; Ennis, 1998; Gieve, 1998; and Kumaravadivelu,
2003 for details on this line of debate).

If a course has been designed to promote critical thinking,
students’ critical thinking skills should be reflected in their perfor-
mance in the course. That is, students should be able to practice their
critical thinking in course assessments. Previous research in the
United States showed that students’ critical thinking skills measured
by standardized instruments were positively related to their academic
achievement in courses which emphasize the practice of critical
thinking (e.g., Bowles, 2000; Collins & Onwuegbuzie, 2000). If such
positive relationship between critical thinking skills and academic
performance is equally applicable to both Asian and Western student
samples, the emphasis of critical thinking can then be considered
equally useful to both groups of students.

Assuming Asian students are indeed less capable of critical thinking
than Western students, in a course which emphasizes the use of critical
thinking, it would be possible to observe that Asian students showing
lower level of academic performance (Hypothesis 4). Nevertheless, ample
research evidence showed that Asian students outperform their Western
counterparts in academic achievements in different national and
international research, even though certain Asian learning practices
have been considered unfavorable to good learning outcomes according
to Western standards (i.e., “the paradox of the Asian learners”, see
Watkins & Biggs, 2001 for details). Therefore, it would also be possible
that Asian students can achieve high grades in courses emphasizing
critical thinking despite their apparently lower level of critical thinking
skills. Instead of observing systematic cultural difference in academic
achievement, the relationship between critical thinking and academic
performance might be found to be weaker among Asian students than
Western students (Hypothesis 5). In such case, the relevance of
emphasizing critical thinking and the usefulness of related educational
practices in international tertiary education would still require
reexamination.

1.8. Overview of the present research

In the present research, we are testing whether there are differences
between Asian and Western students on two different objective tests of
critical thinking. Second, we explore the roles of English language ability
and dialectical thinking styles, based on the propositions of the
Cognitive Load Theory (Paas et al., 2003) and the dialectical thinking
paradigm (Nisbett et al., 2001; Peng & Nisbett, 1999), in explaining the
observed difference. The relationship between dialectical thinking and
critical thinking is also explored. Finally, to understand how a teaching
programme emphasizing critical thinking may be influenced by culture,
we examine the relationship between critical thinking and academic

607V.M.-C. Lun et al. / Learning and Individual Differences 20 (2010) 604–616

performance in both Asian and Western student samples. The finding
from this investigation is expected to provide important empirical data
about the potential impact of culture on the emphasis of critical thinking
in university education.

The research was conducted in a large New Zealand University. New
Zealand provides a good case study due to its cultural diversity. In the
past 20 years, the growth in Asian population has been the highest
among all immigrating ethnicities in New Zealand (Friesen, 2008). In
the educational context, New Zealand attracts students mostly from
Asian countries and communities such as China, Korea, and Japan (New
Zealand Ministry of Education, 2008). This characteristic of the students’
cultural composition in New Zealand offers an interesting avenue to test
out the aforementioned research hypotheses.

Two studies were conducted in this research. Study 1 was designed
to provide an initial account of whether there is a difference in critical
thinking abilities between the Chinese and New Zealand European
students. A newly-developed critical thinking instrument, the Halpern
Critical Thinking Assessment using Everyday Situations (HCTAES;
Halpern, 2006), was used to assess the critical thinking skills of a
sample of Chinese and European students. The roles of dialectical
thinking and English proficiency in the observed cross-cultural
difference were also explored.

Study 2 included an assessment with a more popular measure of
critical thinking (the Watson–Glaser Critical Thinking Appraisal Short
Form; Watson & Glaser, 1994, 2006). The study replicated the design
and tested the same set of hypotheses as in Study 1, except that a
different instrument was used to enhance the generalizability of the
results. Following the research strategies employed in previous studies
on the predictors of critical thinking performance (e.g., Clifford et al.,
2004; Taube, 1997), an objective measure which assesses English
proficiency and general intellectual ability was also included. A follow-
up data analysis of a subset of study 2 focusing on the academic
performance of students in a management class was conducted to test
hypotheses 4 and 5, addressing concerns about the possible conse-
quence of cultural difference in critical thinking in international
education.

2. Study 1: pilot study with the HCTAES

This pilot study was conducted to provide preliminary evidence on
the observed cross-cultural differences in critical thinking between
Asian and Western students. To explore the effect of dialectical thinking
and English proficiency in explaining the differences, self-report
measures of these variables were included in the assessment.

2.1. Method

2.1.1. Participants
Seventy students of a university in New Zealand participated in the

study. Twenty-four students identified themselves as ethnically
Chinese, 35 as New Zealand European, and 11 participants were of
other ethnic identities such as New Zealand Maori and Samoan. Because
our research focus was on Asian and New Zealand European students,
only the data from the Chinese and New Zealand European students
were subjected to further analysis. All international students had to
provide evidence of sufficient proficiency in written and spoken English
upon enrolling to the university. Among the 59 students included, there
were 47 females, 11 males, and one unidentified. The average age of
these students was 21.95 years (SD=1.99).

2.1.2. Materials
Participants were instructed to complete a set of instruments as

described below:

2.1.2.1. Halpern Critical Thinking Assessment using Everyday Situations
(HCTAES). The HCTAES (Halpern, 2006) is an instrument designed for

assessing critical thinking abilities on five dimensions, namely, verbal
reasoning skills, argument analysis skills, skills in thinking as
hypothesis testing, using likelihood and uncertainty, and decision
making and problem-solving skills, which are the five important
college-level critical thinking skills according to Halpern (1998). The
test consists of 25 everyday-life scenarios each with one open-ended
item and one forced-choice item.

According to the information in the test manual, the HCTAES was
shown to correlate positively with the Arlin Test of Formal Reasoning
(Arlin, 1984) within a high-school student sample and a college
student sample (r=.32 in both samples). The Arlin Test is a test of
formal reasoning based on Piaget’s model of cognitive development.
In addition, the HCTAES was found to have a positive correlation
(about .60) with the Analytic subtest of the Graduate Record Exam,
showing convergent validity with other tests which measured related
cognitive abilities.

The whole test takes about 90 min to finish, so to keep the testing
time reasonably brief for the participants, only the forced-choice
portion of the test was used. The use of both open-ended and close-
ended portions together was intended for educational and training
purpose. However, the forced-choice portion alone would be
sufficient for a brief assessment of the critical thinking abilities of
the samples, which was intended to show if test-takers are able to use
the skill when they are provided with more clues in the close-ended
items (Halpern, 2006).

Therefore in the final test, there were altogether 25 forced-choice
scenario based items, with five items tapping each skill category. In
addition to the more common multiple-choice format which involves
choosing the best answer among a number of alternatives (nine
items), there were also items requiring the testees to give ratings on
different options pertinent to a particular scenario (seven items), to
choose multiple correct answers among a number of alternatives (five
items), and to identify the characteristics of a list of statements (four
items). Given the diversified response formats of the items, the final
score was calculated using the standardized scores of every item; a
similar procedure was employed by Hau, Ho, Lai, Ku, and Hui (2008).

2.1.2.2. Dialectical Self Scale (DSS). The DSS was developed by Spencer-
Rodgers et al. (2001) for measuring dialectical thinking in the domain
of self-perception. It has been utilized to explain differences in
psychological well-being (Spencer-Rodgers et al., 2004) and response
styles (Hamamura et al., 2008) between Asian and Western cultures.
The scale was constructed using a 7-point format with higher score
indicating higher level of dialectical self-concept. Sample items
included “When I hear two sides of an argument, I often agree with
both”, “I often find that my beliefs and attitudes will change under
different contexts”. Cronbach’s alpha was .78 in the Chinese sample
and .71 in the New Zealand European sample.

2.1.2.3. Perceived proficiency in reading and writing English (English).
The participants were asked to rate their proficiency in English on a 7-
point scale on “How proficient are you in reading in English” and
“How proficient are you in writing in English”, with higher scores
indicating higher level of proficiency. The correlation between these
items was .82 (pb.01) in the Chinese sample and .92 (pb.01) in the
New Zealand European sample. The scores on these items were
averaged to provide an estimate of the participants’ perceived
proficiency in reading and writing English.

All measures were administered in English because English is the
medium of instruction in the New Zealand tertiary education system.
Entry requirements of the universities in New Zealand include
providing evidence of the students’ English language proficiency,
either in the form of the NZ Bursary Examination or standardized
international English proficiency tests. Therefore we can confidently
assume that the participants possess a reasonable level of English
proficiency to take part, and no participant from either sample

Table 1
Descriptive statistics and independent sample t-tests of the target variables in Study 1.

Chinese New Zealand
European

t-test

M SD M SD

HCTAES (standardized) −1.26 1.70 0.87 1.13 −5.78⁎⁎⁎
Perceived English proficiency 4.58 1.10 6.71 0.61 −8.63⁎⁎⁎
DSS 4.00 0.52 3.87 0.46 1.25

⁎⁎⁎ pb.001.

608 V.M.-C. Lun et al. / Learning and Individual Differences 20 (2010) 604–616

reported difficulty in understanding the instructions or the items. The
average time required for completing the session was about one hour.

2.1.3. Analytical strategy
To examine the effect of perceived English proficiency and the DSS

in the relationship between culture and critical thinking skills,
mediation analysis was used (MacKinnon, Fairchild, & Fritz, 2007).
In testing the mediation hypotheses, the procedures proposed by
Preacher and Hayes (2004, 2008) on testing indirect effects were used
instead of the more commonly adopted Baron and Kenny’s (1986)
causal-step procedures with regressions or Structural Equation
Modeling (SEM).

According to Preacher and Hayes (2004), the method described by
Baron and Kenny (1986) is likely to suffer from low statistical power,
especially in small samples (MacKinnon, Lockwood, Hoffman, West, &
Sheets, 2002). For example, the Baron and Kenny procedure requires
significance of the X–Y relation. However, the coefficient may be
nonsignificant due to low statistical power even though a nonzero
effect in the population is actually present. In this situation, mediation
model cannot be tested based on the Baron and Kenny’s procedures
which then results in a Type II error. Instead, Preacher and Hayes
(2004) suggested that testing the significance of the indirect effect
between X and Y through the mediator requires one fewer hypothesis
test, which reduces the likelihood of Type II error in mediation
analysis. The Sobel test (Sobel, 1982) has been commonly used for
testing significance of indirect effect (MacKinnon et al., 2002).
However, the test requires the estimates of indirect effect to be
normally distributed which is rarely the case in small samples. To
circumvent the problem associated with small sample sizes, Preacher
and Hayes (2004) recommended a bootstrapping procedure which
makes no assumption about the sampling distribution of the
estimates of effects.

Bootstrapping involves repeated sampling from the data set with
replacement and estimates the indirect effect in each resampled data
set (Preacher & Hayes, 2004). An approximation of the sampling
distribution of the indirect effect will be built by repeating the
resampling process for thousands of times and then be used to
construct confidence intervals for the indirect effect. The procedures
of testing indirect effects with bootstrapping have been extended to
multiple mediator models, with syntaxes and macros being designed
for related analysis with common statistical software such as SPSS,
SAS, and LISREL (Preacher & Hayes, 2008). The bootstrapping
procedure generates estimates of the total effect of X on Y, the direct
effect of X on Y after entering the list of mediators, the indirect effects
of X on Y via the list of mediators, and the specific indirect effects of X
on Y via each particular mediator. The total indirect effect of X on Y is
the sum of all specific indirect effects and is equal to the difference
between total and direct effects.

The SPSS macro offered by Preacher and Hayes (2008) was used in
the present analysis. Each of the total and specific indirect effects
generated through the procedure were assessed by three 95%
bootstrap confidence intervals (CIs), namely, the percentile, the
bias-corrected (BC), and the bias-corrected and accelerated (BCa)
intervals. An indirect effect is shown to be significantly different from
zero at 95% confidence if zero is not included in the range of CIs.

2.2. Results and discussion

2.2.1. Cultural differences in critical thinking skills and other self-report
measures

Table 1 shows the descriptive statistics and independent sample t-
tests results of the three variables under consideration. Significant
difference in critical thinking skills was noted between the two
samples, with New Zealand European students performing better
than the Chinese students in the HCTAES. This result supports
Hypothesis 1 which was about the difference in critical thinking
between Asian and Westerns students.

The two samples were also significantly different from each other
in terms of their perceived English proficiency, again with New
Zealand European students scoring significantly higher than the
Chinese sample. However, the two samples did not significantly differ
from each other on the DSS.

2.2.2. Mediation between culture and critical thinking skills
Multiple mediation analysis was conducted following the proce-

dures suggested by Preacher and Hayes (2008). Perceived English
proficiency and the DSS were tested as mediators of the relationship
between culture and critical thinking skills. The effect of gender was
controlled for in the analysis.

The total effect of culture on critical thinking skills was significant
b=2.45, SE=0.34, pb .001. The partial effect of gender on critical
thinking skills was not significant (b=0.38, SE=0.41, p=.35),
indicating that there is no significant gender effect on the measure
of critical thinking skills. With consideration of the two mediators in
the model, the direct effect of culture on critical thinking skills became
only marginally significant, b=1.00, SE=0.53, p=.06. Using the logic
of Baron and Kenny’s (1986) causal-step procedures, the reduction of
the coefficient between culture and critical thinking skills indicated
that at least some of the proposed mediating variables in the model
were potent.

Table 2 summarizes the result of the bootstrapping mediation
analysis of the total and specific indirect effects as well as contrasts
between the specific indirect effects. As zero is not contained in the
confidence intervals of the total indirect effect, the total indirect effect
was significantly different from zero, indicating that the relationship
between culture and critical thinking skills was mediated by at least
some of the proposed mediators. The specific indirect effect of
perceived English proficiency was the only one significantly different
from zero at 95% confidence, showing that perceived English
proficiency was the significant mediator in the model. These results
provided support to Hypothesis 3 but not Hypothesis 2.

The results of this pilot study showed that there was a difference
between Chinese and New Zealand European student samples on
critical thinking performance as assessed by an objective measure,
with the latter sample performing better than the former in the test.
Self-report English proficiency could, at least to a certain extent,
explain this difference. Nevertheless, dialectical thinking style could
not explain the cultural difference in critical thinking as we proposed.

3. Study 2a: assessment with the WGCTA-SF

To test the research hypotheses more fully, an assessment was
conducted with another instrument — the Watson–Glaser Critical
Thinking Appraisal Short Form (WGCTA-SF; Watson & Glaser, 1994,
2006). The WGCTA is one of the most popular measures of critical
thinking. Similar to the HCTAES (Halpern, 2006), the WGCTA employs
everyday situations to test critical thinking in a general sense (Renaud
& Murray, 2008). Following the research strategy employed in Taube
(1997) and Clifford et al. (2004), English proficiency and general
intellectual ability are assessed with an objective measure: The
Shipley Institute of Living Scale (SILS; Shipley, 1940; Zachary, 1991).
We expected that an objective measure of English proficiency would

Table 2
Mediation analysis in Study 1.

Bootstrapping results for indirect effects

Point
Estimate

Percentile
95% CI

BC
95% CI

BCa
95% CI

Lower Upper Lower Upper Lower Upper

Indirect effects
English 1.41 0.74 2.22 0.74 2.22 0.72 2.19
DSS 0.02 −0.08 0.21 −0.05 0.28 −0.05 0.26
Total 1.44 0.75 2.28 0.73 2.24 0.70 2.22

Contrasts
English vs. DSS 1.39 0.69 2.21 0.70 2.23 0.68 2.19

5000 bootstrap samples; significant indirect effects or contrasts in bold.

609V.M.-C. Lun et al. / Learning and Individual Differences 20 (2010) 604–616

provide more solid evidence of the role of language in explaining
cultural difference in critical thinking. We controlled for general
intellectual ability, to rule out the possibility that differences in critical
thinking are due to the difference in basal general intellectual
competence between the two samples.

Apart from these additional control measures, we acknowledge
that self-report English proficiency, apart from representing the
intended perceived English proficiency, reflects also a person’s
adoption to an English-speaking cultural environment (e.g., Gaudet
& Clément, 2009). Because of this possible linkage between perceived
language proficiency and acculturation, it is logical to argue that the
findings in Study 1 was essentially showing that adoption of an
English-speaking culture, rather than proficiency in using English, was
what explained the cultural difference in critical thinking skills.1 In
such case, critical thinking might alternatively be interpreted as
something specifically favorable to Western culture, thereby under-
mining the usefulness of practicing critical thinking in an interna-
tional education context. To rule out the possibility of adoption to a
Western culture was the actual factor leading to cultural differences in
critical thinking, a measure of cultural adoption was also included in
the analysis.

Around half of the participants in this study were recruited
through a Management course as a part of the accreditation project of
the Management school. Data on the final course grade was available
for this subsample. As suggested above, cultural differences observed
in critical thinking have important implication in evaluating the
suitability of promoting critical thinking in an international class-
room. If teaching professionals attempt to promote critical thinking in
their courses but students of different cultural backgrounds cannot
equally apply their critical thinking as intended, the usefulness of
promoting critical thinking in education would then be questionable.
Therefore, this data set with students’ course grades enables the
possibility of testing the relationship between students’ academic
performance and critical thinking abilities, and thereby provides
information about the practical implication of promoting critical
thinking in higher education.

3.1. Method

3.1.1. Participants
Three hundred and sixty-six university students participated in

the study, among these participants 102 identified themselves as
ethnically Asian, 210 identified as ethnically New Zealand European,
and 54 of other ethnicities, including New Zealand Maori, Samoan,
African, and mixed ethnicities. Again, because the major focus of this
research was on the comparison between Asian and New Zealand
European students, the data from those of the other ethnicities were

1 We would like to thank an anonymous reviewer of an earlier version of this
manuscript for pointing out this possible alternative explanation.

not included in the subsequent analysis. The majority of the Asian
sample consisted of Chinese (68.6%), followed by Indian (9.8%),
Vietnamese (6.9%), and Filipinos (3.9%), with the rest being
Cambodian, Indonesian, Japanese, Korean, Malay, Sri Lankan, and
Thai. This pattern closely resembles the actual situation of interna-
tional education in New Zealand, where Chinese migrants form the
majority of Asian students (New Zealand Ministry of Education,
2008). The average age of the final sample of 312 participants was
20.09 years (SD=4.83), with 124 males, 187 females and one
unidentified gender.

It is noted that the Asian student sample appeared to be more
diverse than the NZ European sample. Nevertheless, because we are
looking for the active ingredient of cultural difference in critical
thinking by using individual differences variables (Matsumoto & Yoo,
2006), the differences in terms of within-group diversity between the
two samples would be accounted for in the analysis. Therefore, the
conclusion drawn would not be affected by the seemingly more
diversified Asian sample.

3.1.2. Materials
The English version of the following instruments was administered

to the participants. The average time required for completing the
whole session was about one hour.

3.1.2.1. Watson–Glaser Critical Thinking Appraisal Short Form (WGCTA-
SF). The WGCTA was designed to measure critical thinking abilities
along five dimensions, namely, inferences, recognition of assump-
tions, deduction, interpretation, and evaluations of argument. The
short form version with 40 items was employed in this study (Watson
& Glaser, 1994, 2006). The WGCTA-SF was derived by shortening the
original 80-item Form A of the WGCTA (Watson & Glaser, 1980) in an
attempt to reduce the administration time. Reliability of the test was
shown by Cronbach’s alpha of .81 based on a sample of 1608
participants in the original development of the short form, and alphas
between .66 and .85 in the other studies reported in the Manual.
Validity of the test was demonstrated by the significant correlation
between test scores and criterion-related measures such as effective-
ness in clinical decision making (Shin, 1998) and cognitive problem-
solving skills (Spector, Schneider, Vance, & Hazlett, 2000). In each
subscale, directions and sample questions were provided in front of
the actual test items. All participants were instructed to read carefully
before answering the questions.

Although the scores on the WGCTA-SF might be interpreted at the
subscale level, a meta-analytic review on the psychometric properties
of the WGCTA showed that it would be better to treat the scale as a
measure of general critical thinking competency but not to interpret
the subscales individually (Bernard et al., 2008). This suggestion was
based on the principal component analysis of the subscale means of
the original versions of WGCTA (Watson & Glaser, 1980) which
yielded a one-factor solution. In view of the empirical evidence and for
the sake of parsimony, only the total score of WGCTA-SF is interpreted
in the present research.

3.1.2.2. Shipley Institute of Living Scale (SILS). Developed by Shipley in
1940, the SILS was intended for assessing general intelligence in
adults and adolescents. The revised version by Zachary (1991) was
employed in this study. The SILS consists of two subtests, a 40-item
vocabulary test and a 20-item abstraction test. In the vocabulary test,
participants were asked to choose among four alternative words that
mean the same or nearly the same to a specific target word. In the
abstraction test, a logical sequence was presented and the participants
were required to complete the sequence by filling in the appropriate
numbers or letters. Although the test scores could be used together to
estimate IQ scores based on more popular intelligence testing
instruments such as the Wechsler Adult Intelligence Scale-Revised
(WAIS-R; Wechsler, 1981) by means of various sophisticated

Table 3
Descriptive statistics and independent sample t-tests of the target variables in Study 2a.

Asian New Zealand
European

t-test
M SD M SD

WGCTA total 24.07 5.64 27.05 6.19 −4.02⁎⁎⁎
SILS vocabulary test 24.15 5.90 29.08 4.13 −7.55⁎⁎⁎
SILS abstraction test 16.15 3.00 16.53 2.24 −1.14
DSS 4.02 0.46 3.92 0.56 1.74
Cultural adoption 4.20 1.02 5.26 0.67 −9.29⁎⁎⁎
Perceived English Proficiency 5.17 1.45 6.62 0.68 −9.56⁎⁎⁎

⁎⁎⁎ pb.001.

Table 4
Mediation analysis in Study 2a.

Point
Estimate
Bootstrapping results for indirect effects
Percentile
95% CI
BC
95% CI
BCa
95% CI
Lower Upper Lower Upper Lower Upper

Indirect effects
SILS vocabulary 2.21 1.39 3.15 1.42 3.18 1.42 3.18
DSS 0.02 −0.11 0.18 −0.07 0.24 −0.07 0.24
Cultural adoption −0.57 −1.54 0.19 −1.51 0.21 −1.42 0.27
Perceived English −0.47 −1.50 0.56 −1.51 0.56 −1.53 0.56
Total 1.19 0.16 2.24 0.21 2.33 0.22 2.34

Contrasts
SILS vocabulary vs.

DSS
2.19 1.38 3.11 1.42 3.17 1.42 3.17

SILS vocabulary vs.
cultural adoption

2.78 1.67 4.10 1.65 4.09 1.61 4.06

SILS vocabulary vs.
Perceived English

2.67 1.15 4.32 1.20 4.39 1.20 4.39

DSS vs. cultural
adoption

0.59 −0.16 1.56 −0.18 1.53 −0.26 1.43

DSS vs. Perceived
English

0.49 −0.57 1.56 −0.56 1.57 −0.55 1.58

Cultural adoption vs.
Perceived English

−0.11 −1.87 1.39 −1.85 1.41 −1.76 1.49

5000 bootstrap samples; significant indirect effects or contrasts in bold.

610 V.M.-C. Lun et al. / Learning and Individual Differences 20 (2010) 604–616

conversion procedures (e.g., Paulson & Lin, 1970; Zachary, Paulson, &
Gorsuch, 1985), the raw scores of each test were used as measures of
the participants’ English proficiency (vocabulary score) and general
intellectual competence (abstraction score), respectively.

3.1.2.3. Dialectical Self Scale (DSS). The same instrument employed in
Study 1 was used. Cronbach’s alpha was .65 in the Asian sample and
.78 in the New Zealand European sample.

3.1.2.4. Perceived English proficiency (English). The two items used in
Study 1 were again used to indicate perceived English proficiency.
Correlation between the two items was .93 (pb.01) in the Asian
sample and.76 (pb.01) in the New Zealand European sample. The
scores on those items were averaged to estimate the participants’
perceived proficiency in reading and writing English.

3.1.2.5. Cultural adoption of behavioral norms in New Zealand. The
Behavioral Acculturation Scale (BACS-16) was a 16-item scale
developed by Groenvynck, Beirens, Arends-Toth, and Fontaine
(2006) to measure two independent dimensions of acculturation,
namely, cultural maintenance and cultural adoption. The term
acculturation is used to describe the cultural change that a person
undergoes during cross-cultural transitions (Ward, 1996). Based on
the two-dimensional model of acculturation (Berry, 1997), the BACS-
16 was designed to capture an individual’s cultural change in terms of
their actual knowledge, actions, and behaviors related to their home
and host cultures. In essence, the scale consists of eight pairs of item.
Within each pair, one item focuses on the maintenance of home
culture, while the other item focuses on the adoption of host culture.

In the present study, eight items from the cultural adoption
subscale were chosen to capture the participants’ adoption of
behavioral norms in New Zealand. All participants were instructed
to rate the items concerning their knowledge of and behaviors within
New Zealand culture using a 6-point Likert scale, ranging from fully
disagree (1) to fully agree (6). Sample items include “I know the New
Zealand culture and traditions well” and “I live according to rules that
apply in the New Zealand culture”. Cronbach’s alpha was .87 in the
Asian sample and .86 in the New Zealand European sample.

3.2. Results and discussion

3.2.1. Cultural differences in WGCTA, SILS scores, and other self-report
measures

Table 3 shows the descriptive statistics of the target variables in
the Asian and New Zealand European samples. It was observed that
the New Zealand European sample scored higher on all variables
except the DSS. Independent sample t-tests on the variables revealed
that the differences between the two samples were statistically
significant in terms of the WGCTA score, the SILS vocabulary score,
cultural adoption, and perceived English proficiency. The differences
observed in SILS abstraction score and DSS were not statistically
significant, indicating that the two cultural groups are not signifi-

cantly different from each other in general intellectual competence
and self-report dialectical thinking style.

3.2.2. Mediation between culture and critical thinking skills
Similar to the pilot study, multiple mediation analysis was

conducted using the procedures proposed by Preacher and Hayes
(2008). SILS abstraction score was included as a covariate to control
for the effect of general intellectual competence on critical thinking
skills. Gender was also controlled for in the analysis.

The total effect of culture on the WGCTA score was significant
b=2.70, SE=0.72, pb.001. The partial effect of gender on the WGCTA
was not significant (b=−0.19, SE=0.67, p=.78), indicating that
there was no significant gender effect on critical thinking skills as
measured by the WGCTA. However, the partial effect of SILS
abstraction score was significant (b=0.53, SE=0.13, pb .001),
suggesting that general intellectual competence was positively
related to critical thinking skills. After taking into account the
mediators, the direct effect of culture on critical thinking skills
became statistically nonsignificant, b=1.51, SE=0.86, p=.08. Again,
based on Baron and Kenny’s (1986) logic, the reduction in the effect of
culture on the WGCTA indicated that at least some of the proposed
mediators were potent.

Table 4 summarizes the results of the bootstrapped mediation
analysis of the total and specific indirect effects as well as contrasts
between the specific indirect effects. Consistent with the observation
on the difference between the total effect and the direct effect of
culture on critical thinking, zero was not contained in the confidence
intervals of the total indirect effect, which showed that the total
indirect effect was significantly different from zero. In other words,
the relationship between culture and critical thinking skills was
mediated by at least some of the proposed mediators.

Inspection of the CIs of the proposed mediators revealed that only
the specific indirect effect of SILS vocabulary was significantly
different from zero. Tests of contrasts between the specific indirect
effects showed that the specific indirect effect of SILS vocabulary was
significantly different from those of the other variables, suggesting
that it was the most important mediator in the relationship between
culture and critical thinking skills among the variables considered.

Table 5
Correlation among the variables in the two samples in Study 2a.

WGCTA SILS
vocabulary

SILS
abstraction

DSS

Cultural
adoption

SILS vocabulary (.43⁎⁎)
.44⁎⁎

SILS abstraction (.43⁎⁎) (.33⁎⁎)
.32⁎⁎ .36⁎⁎

DSS (.26⁎) (.20) (.29⁎⁎)
−.04 .05 .03

Cultural adoption (.04) (.32⁎⁎) (.05) (−.09)
−.12 −.05 −.07 .02

Perceived English
proficiency

(.13) (.51⁎⁎) (.12) (−.04) (.64⁎⁎)
.06 .20⁎⁎ .19⁎ −.04 .12

Numbers in parentheses are correlations in the Asian sample, n=94; Numbers without
parentheses are correlations in the New Zealand European sample, n=169; listwise
deletion of missing data.
⁎ pb.05.
⁎⁎ pb .01.

611V.M.-C. Lun et al. / Learning and Individual Differences 20 (2010) 604–616

Therefore, hypothesis 3 was supported with an objective measure of
English proficiency.

It is noted that the specific indirect effect of perceived English
proficiency, unlike that in Study 1, was not significant in the current
analysis. A subsequent analysis using only DSS and perceived English
proficiency as mediators revealed that the specific indirect effect of
self-report English proficiency was significant without the SILS
vocabulary score in the list of mediators.2 This finding suggested
that an objective measure of English proficiency might be more
powerful than a self-report measure, so that the effect of perceived
English proficiency was taken up by the SILS vocabulary score when
they are analyzed simultaneously.

Table 6
Moderating effect of culture on the relationship between critical thinking and DSS.

Steps β t ΔR2 F Change

1. SILS vocabulary .49 9.60 .24 92.11⁎⁎

2. SILS vocabulary .47 8.39⁎⁎ .001 .172
DSS −.01 −.15
Culture .03 .55

3. SILS vocabulary .47 8.24⁎⁎ .01 3.17a

DSS .16 1.50
Culture .04 .74
DSS×culture −.19 −1.78a

3.2.3. Relationship between DSS and WGCTA across cultures
Table 5 shows the correlations among the target variables in each

group. Similar correlation pattern among the three test scores were
observed in both cultural groups, where WGCTA was both positively
related to SILS vocabulary score and SILS abstraction score, and the
two SILS scores were moderately positively related to one another.
This pattern is consistent with those observed in previous research
(Clifford et al., 2004; Taube, 1997).

It is worth to note that in the Asian sample, perceived English
proficiency was significantly related to both SILS vocabulary score and
cultural adoption, indicating that perceived English proficiency could
be interpreted both in terms of actual English language abilities and
behavioral adoption of New Zealand culture.

Dialectical self-concept was found to be significantly positively
related to WGCTA score in the Asian sample, whereas its correlation
with the WGCTA was close to zero in the New Zealand European
sample. The correlation pattern implicated that the relationship
between dialectical self-concept and critical thinking vary as a
function of culture, although dialectical self-concept could not help
explain the cultural difference in critical thinking performance.
Moderation analysis using multiple regressions (Baron & Kenny,
1986) was conducted to reveal the possible cultural difference in how
dialectical self-concept predicts critical thinking performance. The
effects of English proficiency (SILS vocabulary scores) was controlled
for in the analysis because it was found to account for the cultural
difference in critical thinking performance.

The results of the regression analyses are reported in Table 6. The
interaction between culture and dialectical self-concept was margin-
ally significant (p=.08), suggesting a trend of cultural difference in
the relationship between DSS and critical thinking skills. It was likely

2 The point estimate of the specific indirect effect of perceived English proficiency
was .96, with percentile 95% CI ranged from.11 to 1.94; BC 95% CI ranged from .11 to
1.95; and BCa 95% CI ranged from .11 to 1.94.

that if the sample size was larger, the effect would become statistically
significant because of higher statistical power. The interaction effect is
depicted in Fig. 1 using the internet version of Modgraph (Jose, 2008).
It showed that dialectical self-concept was positively related to critical
thinking skills in the Asian sample, but the relationship between the
two variables was slightly negative in the New Zealand European
sample.

These findings basically replicated the results obtained in Study 1.
Asian students were again found to perform less well on a critical
thinking skills assessment than their Western counterparts, which
provided support to Hypothesis 1. This cultural difference was again
explained by English proficiency but not dialectical thinking style,
thereby showing support to Hypothesis 3 but not Hypothesis 2. It is
also important to note that cultural adoption did not mediate the
relationship between culture and critical thinking, which ruled out
the possibility that adoption to Western culture is what explains the
observed cultural difference. In other words, the result does not
support the idea that critical thinking favors Western over Asian
cultural group.

Interestingly, the relationship between dialectical thinking and
critical thinking was found to be culturally different. Specifically,
dialectical thinking was only positively related to critical thinking
among Asian students but not New Zealand European students,
suggesting that different cognitive processes might be involved in
critical thinking between these two samples. Since the current
analysis revealed that such cultural difference might exist, and
given that dialectical thinking involves a complex array of cultural
differences in cognitive tendencies and philosophical traditions (Peng
& Nisbett, 1999), further research will be needed to unravel the exact
influence of dialectical thinking style on critical thinking.

4. Study 2b: critical thinking skills and academic performance

Study 1 and Study 2a consistently showed that Asian students
performed less well on two objective measures of critical thinking
skills than their New Zealand European counterparts. This finding
might challenge the current enthusiasm on critical thinking in the
context of international education, because students of different
cultural backgrounds might show different preferences regarding
engaging in critical thinking and it may disadvantage some students.
In such case, the value of promoting critical thinking in university
education would be undermined as its usefulness varied as a function
of students’ cultural backgrounds.

University education in New Zealand showed relatively stronger
emphasis on critical thinking than that in Asia (Lun, Fischer, & Ward,
2009). Academic conventions such as writing literature reviews,
critical reviews, essays, and research proposals are commonplace in
New Zealand university education, but Asian students often report
difficulties in fulfilling the requirement of these assessment methods
because of lack of knowledge about these academic conventions

Culture was dummy coded as “Asian students=0” and “New Zealand European
students=1”; DSS was mean centered in the analysis (Aiken & West, 1991).

a p=.08.
⁎⁎ pb .01.

Fig. 1. Visual depiction of the interaction between DSS and culture on critical thinking in Study 2a.

612 V.M.-C. Lun et al. / Learning and Individual Differences 20 (2010) 604–616

(Campbell & Li, 2008). If students’ use of critical thinking is
encouraged and assessed by these tasks, Asian students might find
it harder to express their critical thinking as required despite their
actual ability. Consequently, the relationship between critical thinking
skills and academic performance might be weaker among Asian
students. The use of these assessment methods to develop students’
critical thinking may then be considered inappropriate to Asian
students.

In view of these forms of possible cultural influence on the
relationship between critical thinking and academic performance,
Hypothesis 4 and Hypothesis 5 as outlined in the Introduction were
tested in a management course at the university.

4.1. Participants

A subgroup of the sample in Study 2a was included in this analysis.
This sample was drawn from an introductory management course,
that explicitly focused on critical thinking and the final course grades
were available for analysis. The subsample comprised 110 New
Zealand European students and 52 Asian students.

4.1.1. Course grades in the introductory management course
The course under investigation was an introductory management

course which explicitly emphasizes the development of critical
thinking skills. The course outline stated that the major objective of
the course was to give an introduction of the trends, issues and
challenges of the business environment in New Zealand, and students
were expected to develop and apply their critical thinking in the
course.

The assessment components were designed with the aim of
developing students’ critical thinking through various assignments
and final examination. These included a journal of the student’s
personal experience of a business topic (20% of final course grades),
essays with critical discussion (40% of final course grades), a final
examination which required critical discussion on issues related to
business development (30% of final course grades), and satisfactory
completion of an essay writing and referencing course (10% of final
course grades). With this explicit emphasis of critical thinking, the
course offered an interesting avenue for testing the present research
hypotheses.

The final course grades of the participants in the introductory
management course were obtained with approval from the manage-
ment school as a part of the school’s accreditation project. Ethics
approval for the accreditation project was granted to the management
school by the university’s Human Ethics Committee. The grades were
transformed into a continuous numeric variable based on this scale:
1=E, 2=D, 3=C, 4=C+, 5=B-, 6=B, 7=B+, 8=A−, 9=A,
10=A+. According to the grading system of the university, grade C or
above denotes a pass in the course (i.e., achieving 50% of final course
grades).

4.2. Results and Discussion

4.2.1. Cultural difference in academic performance
Independent sample t-test revealed that there was a marginally

significant difference in the academic performance between the two
groups [t(160)=−1.83, p=.07]: New Zealand European students
[M=6.41, SD=1.53] performed slightly better than Asian students
[M=5.92, SD=1.68] in the course which provides initial support to
Hypothesis 4.

4.2.2. Relationship between critical thinking and academic performance
across cultures

Table 7 shows the correlation among the target variables in the
two samples. WGCTA was significantly correlated with SILS vocabu-
lary and SILS abstraction test scores, indicating that critical thinking
are positively related to one’s English proficiency and general
intellectual competence. The correlation between course grade and
the WGCTA score was also positive in both samples. Students’ course
grades were significantly correlated with SILS abstraction score,
showing that general intellectual competence is positively associated
with students’ academic achievement. The correlation between
course grades and SILS vocabulary score was significant only in the
Asian sample, but statistical analysis using Fisher z-transformation
revealed that the difference in the correlation coefficients between
the two samples was only not significant, z=1.49, p=.14. This
finding suggested that English proficiency may be a factor in
predicting students’ academic performance in both samples.

With regard to Hypothesis 5, moderation analysis using multiple
regressions (Baron & Kenny, 1986) was used to examine the effects of
critical thinking skills, cultural adoption, culture, and their interac-
tions on course grades. SILS vocabulary score, SILS abstraction score,

Table 7
Correlation among the variables in the two subsamples in Study 2b.

Course grade WGCTA SILS vocabulary SILS abstraction DSS Cultural adoption

WGCTA (.29a) –
.34⁎⁎

SILS vocabulary (.45⁎⁎) (.50⁎⁎) –
.19 .44⁎⁎

SILS abstraction (.32⁎) (.38⁎) (.38⁎⁎) –
.30⁎⁎ .30⁎⁎ .28⁎⁎

DSS (−.09) (.23) (.21) (.37⁎) –
−.21 .07 .10 .07

Cultural
adoption

(.16) (.13) (.26) (−.13) (−.21) –
.14 −.14 −.07 −.03 −.12

Perceived
English proficiency

(.20) (.13) (.59⁎⁎) (.26) (.02) (.53⁎⁎)
.19 .01 .28⁎⁎ .07 −.14 .15

Numbers in parentheses are correlations in the Asian sample, n=45; Numbers without parentheses are correlations in the New Zealand European sample, n=85; listwise deletion
of missing data.

a p=.06.
⁎ pb.05.
⁎⁎ pb .01.

613V.M.-C. Lun et al. / Learning and Individual Differences 20 (2010) 604–616

perceived English proficiency, and gender were entered in the first
block of the regression analysis to control for their effects to enable a
clearer picture about the cultural influence on the relationship
between critical thinking and academic performance. The result of
the regression analysis is presented in Table 8.

The first step of the regression analysis showed that course grade
was significantly predicted by SILS vocabulary score and SILS
abstraction score, but the effect of gender was not significant. In the
second step of the regression analysis, it was noted that WGCTA
significantly predicted course grades, indicating that critical thinking
was positively related to students’ academic achievement even after
controlling for the effects of English proficiency and general
intellectual competence. The significant effect of WGCTA supports
that the course assessment was positively related to students’ skills in
practicing critical thinking.

In the subsequent steps of the regression analysis, the effects of all
two-way and three-way interaction terms were statistically nonsig-
nificant, indicating that the relationship between critical thinking and
academic performance was neither moderated by the cultural back-
grounds nor the level of cultural adoption of the students. Therefore,
the results did not support Hypotheses 5. Critical thinking has a
positive effect on academic performance independent of the cultural
background of students.

These results are important in showing that critical thinking skills,
even after considering the effects of language proficiency and general
intellectual competence, are predictive of students’ academic perfor-
mance in a course where the practice of critical thinking is
emphasized. Though Asian students were found to perform less well
on critical thinking as measured by the WGCTA, they were not any
different from their New Zealand European counterparts in using
critical thinking when it is required in a course. The nonsignificant
interaction between critical thinking and cultural adoption also
indicated that acquisition of Western cultural norms did not seem
to affect the relationship between critical thinking and academic
performance.

5. General discussion

Critical thinking is an important skill for many university students
(Halpern, 1998; ten Dam & Volman, 2004). We empirically examined
cultural differences in critical thinking between Asian and Western
university student samples, addressing an important issue debated in
the international education literature. We found that New Zealand
European students performed significantly better than Asian students
in two objective measures of critical thinking. Investigation of the
underlying mechanisms revealed that English proficiency, both self-

report (Study 1) and objectively measured (Study 2a), at least
partially if not fully account for this observed difference. Although
dialectical thinking style did not explain the observed cultural
difference in critical thinking, results in Study 2a suggested that this
thinking style may be beneficial for Asian students but not New
Zealand European students in the practice of critical thinking.
Moreover, despite the observed cultural difference, the relationship
between critical thinking and academic performance did not vary as a
function of students’ cultural backgrounds or their level of adoption of
Western cultural norms (Study 2b).

The present research confirms with objective assessments the
cultural differences in critical thinking as observed by ESL teaching
professionals. While those observations cannot be simply discounted
as cultural stereotypes of Asian students (e.g., Kumaravadivelu, 2003),
we should pay attention to the fact that English language proficiency,
but not dialectical thinking style, explained the difference. In other
words, the difference in critical thinking appears to be more of a
linguistic issue rather than a cultural issue. The finding that cultural
adoption does not mediate the cultural difference in critical thinking
provided further support to this argument.

5.1. Dialectical thinking

Dialectical thinking style did not mediate the observed difference
in critical thinking skills in the present investigation. In the
examination of the relationship between DSS and WGCTA scores, it
was observed that the relationship between the two variables was
only positive in the Asian sample, but not the New Zealand sample.
The difference in the direction of correlations might explain why
dialectical thinking could not mediate the effect of culture on critical
thinking skills. It should be highlighted, however, that the interaction
effect was only marginally significant. The result should be considered
as indicating a possible cultural difference in cognitive processes
related to critical thinking.

Dialectical thinking style was a self-report measure in the present
research. It is possible that individual’s evaluations about various
dialectical thinking principles might influence the association be-
tween critical thinking and dialectical thinking in different cultures. In
a Western culture such as that in New Zealand, the principles involved
in dialectical thinking might be interpreted as incongruent to the
formal logical rules which have been commonly endorsed as the
preferred mode of thinking (Peng & Nisbett, 1999). In this situation, it
is reasonable to find the principles of dialectical thinking showing a
negative relationship with critical thinking performance, because the
endorsement of the principles of dialectical thinking could be seen as
illogical to critical thinkers. On the other hand, it is possible that the

Table 8
Moderating effects of culture and cultural adoption on the relationship between critical
thinking skills and academic performance.

Steps β t ΔR2 F Change

1. SILS vocabulary .22 2.15⁎ .15 5.67⁎⁎⁎

SILS abstraction .23 2.60⁎⁎

Perceived English proficiency .06 0.65
Gender −.07 −0.83

2. SILS vocabulary .13 1.15 .05 2.59a

SILS abstraction .20 2.31⁎

Perceived English proficiency .02 0.23
Gender −.08 −0.94
WGCTA .20 2.11⁎

Cultural adoption .17 1.79b

Culture −.02 −0.25
3. SILS vocabulary .17 1.49 .02 1.22

SILS abstraction .19 2.08⁎

Perceived English proficiency .02 0.19
Gender −.09 −1.08
WGCTA .16 0.98
Cultural adoption .21 1.35
Culture −.02 −0.19
WGCTA X Culture .06 0.39
Cultural adoption X Culture −.01 −0.05
Cultural adoption X WGCTA −.15 1.48

4. SILS vocabulary .17 1.56 .01 2.22
SILS abstraction .18 2.00⁎

Perceived English proficiency .02 0.14
Gender −.08 −0.90
WGCTA .06 0.33
Cultural adoption .13 0.78
Culture .00 0.03
WGCTA X Culture .11 0.71
Cultural adoption X Culture .05 0.36
WGCTA X Cultural adoption −.02 −0.11
WGCTA X Cultural adoption X Culture .19 1.49

Culture was dummy coded with “Asian students=0” and “New Zealand European
students=1”; WGCTA score and cultural adoption were mean centered (Aiken & West,
1991).

a p=.06.
b p= .08.

⁎ pb .05.
⁎⁎ pb .01.
⁎⁎⁎ pb .001.

614 V.M.-C. Lun et al. / Learning and Individual Differences 20 (2010) 604–616

endorsement of dialectical thinking was deemed as reflecting an
individual’s wisdom or intellectual capacities in Asian cultures (see
Buchtel & Norenzayan, 2008). So, critical thinkers who are skillful in
the use of different cognitive strategies might prefer the principles of
dialectical thinking, which resulted in the positive relationship
between DSS and WGCTA scores.

The present findings indicated some interesting pattern between
dialectical thinking style and cognitive tasks such as critical thinking.
Further examination of the concept of dialectical thinking and
refinement of its measurement would be beneficial to understand
the practical implication of the concept to areas such as designing
appropriate cognitive instruction to students of diverse cultural
backgrounds.

5.2. English proficiency

The present findings confirm Paton’s (2005) suggestion that Asian
student’s apparent lack of critical thinking is a consequence of the
need to use English as a second language in academic discourse.
Without sufficient English proficiency and/or enough confidence in
using the language, Asian students are discouraged from overtly
expressing their critical thinking in classrooms even if they want to do
so.

According to the CLT (Paas et al., 2003), cognitive load can be
reduced by forming schemas in the long-term memory that can be

brought to the working memory for cognitive processing. Increased
English proficiency resembles formation of schemas about the
language in the long-term memory so that less cognitive load will
be created by processing information of the language. It is important
to note that this principle applies to every student regardless of their
cultural backgrounds. If a critical thinking task requires information
processing in a language, students who are proficient in that
particular language would be able to spare more cognitive capacity
in the working memory to think critically.

Critical thinking skills such as verbal reasoning and argument
analysis demand a certain level of language proficiency. Even the
mere act of reading a scenario or understanding a problem presented
in a language requires certain levels of verbal abilities. While it is
almost impossible to avoid the use of language in critical thinking,
educators may try to structure assessments in a way that the required
cognitive load of language proficiency would be kept at a reasonable
level for all students. For example, overuse of jargon or culture-
specific slangs or words should be avoided in designing course
materials or tests that aim at promoting critical thinking among
students. If the use of such language is deemed necessary, it would be
important to explain fully and familiarize the students with the
associated usage.

The framework proposed by Campbell et al. (2007) for mathe-
matical instruction for second language learners also provided a
useful reference of designing critical thinking instruction. One
element of the framework includes structuring a problem statement
in a way that students may infer the meanings of certain unfamiliar
words from the context of the problem. That involves a lot of effort in
structuring the problem but close coordination between university
teaching professionals and ESL teachers could help to identify the
areas of concern and the appropriate solutions.

5.3. Implication of the practice of critical thinking in international
education

It was intriguing to note in Study 2b that academic performance
was predicted by critical thinking skills after controlling for the effects
of language and intellectual competence. This finding suggests that
critical thinking could indeed play a unique role, apart from the more
commonly emphasized language proficiency and abstract thinking
ability, in university education. More importantly, our findings
showed that the relationship between critical thinking and academic
performance did not seem to be susceptible to the influence of culture.
We may argue that cultural difference in critical thinking does not
necessarily constitute a cultural unfairness which undermines the
value of emphasizing critical thinking in international education
(Atkinson, 1997). However, it would be important to pay attention to
the discussed linguistic issues while designing and implementing
instructional strategies which aim to promote critical thinking among
students.

5.4. Limitations and future research

The present research empirically addressed the claim that Asian
students lack critical thinking by objectively comparing the critical
thinking skills between Asian and Western students. However, some
issues remain to be answered in future research.

It is important to be aware of the multi-dimensional nature of
critical thinking. Beside behavioral expressions such as debating or
questioning ideas (Durkin, 2008), critical thinking also consists of
different cognitive skills and dispositions (e.g., Ennis, 1987; Facione,
1990; Halpern, 2003). The present research has been focused on the
skill aspect of critical thinking and how culture affects this dimension.
A logical follow-up would be to examine the exact relationship
between skills, dispositions, and behaviors, and how culture is related
to the relationships among these variables. Such investigation enables

615V.M.-C. Lun et al. / Learning and Individual Differences 20 (2010) 604–616

a more comprehensive understanding of the relationship between
culture and critical thinking so that proper instructional strategies
may be designed to cater for the need of promoting critical thinking in
multicultural classrooms.

Moreover, the method employed in the present research only
involved paper-and-pencil form of assessment of critical thinking
skills. This is only one of the many possible ways for measuring one’s
critical thinking. After all, critical thinking is not only about test
performance, it is also the application of different cognitive skills to
everyday problems in participants’ lives (e.g., Ennis, 1987; Halpern,
1998). To be able to apply critical thinking to real-life situations is the
ultimate goal of cultivating critical thinking among students. Research
on the application of critical thinking to real-life issues, especially
with reference to the influence of culture, would provide useful
information about the transfer of critical thinking skills in university
education.

Acknowledgements

We would like to thank the Victoria Management School, Victoria
University of Wellington, for their generous help with data collection.
We are also grateful to Prof. Diane Halpern for her permission to
administer the HCTAES and Pearson Assessment and Information
Australia and New Zealand for the research sponsorship on the
WGCTA-SF. This research was supported by grants from the School of
Psychology, Victoria University of Wellington, and the New Zealand
International Doctoral Research Scholarship administered by Educa-
tion New Zealand. Thanks also to Melanie Vauclair, Katja Hanke, and
Diana Boer for their comments on earlier versions of this paper.

References

Aiken, L., & West, S. (1991). Multiple regression. Newbury Park, CA: Sage.
Arlin, P. K. (1984). Arlin test of formal reasoning. East Aurora, NY: Slosson.
Atkinson, D. (1997). A critical approach to critical thinking in TESOL. TESOL Quarterly, 31

(1), 71−94.
Baron, R. M., & Kenny, D. A. (1986). The moderator–mediator variable distinction in

social psychological research: Conceptual, strategic, and statistical considerations.
Journal of Personality and Social Psychology, 51(6), 1173−1182.

Bernard, R. M., Zhang, D., Abrami, P. C., Sicoly, F., Borokhovski, E., & Surkes, M. A. (2008).
Exploring the structure of the Watson–Glaser Critical Thinking Appraisal: One scale
or many subscales? Thinking Skills and Creativity, 3, 15−22.

Berry, J. W. (1997). Immigration, acculturation, and adaptation. Applied Psychology: An
International Review, 46(1), 5−34.

Bowles, K. (2000). The relationship of critical-thinking skills and the clinical-judgment
skills of baccalaureate nursing students. Journal of Nursing Education, 39(8),
373−376.

Buchtel, E. E., & Norenzayan, A. (2008). Which should you use, intuition or logic?
Cultural differences in injunctive norms about reasoning. Asian Journal of Social
Psychology, 11, 264−273.

Campbell, A. E., Adams, V. M., & Davis, G. E. (2007). Cognitive demands and second-
language learners: A framework for analyzing mathematics instructional contexts.
Mathematical Thinking and Learning, 9, 3−30.

Campbell, J., & Li, M. (2008). Asian students’ voices: An empirical study of Asian
students’ learning experiences at a New Zealand university. Journal of Studies in
International Education, 12(4), 375−396.

Clifford, J. S., Boufal, M. M., & Kurtz, J. E. (2004). Personality traits and critical thinking
skills in college students: Empirical tests of a two-factor theory. Assessment, 11,
169−176.

Collins, K. M., & Onwuegbuzie, A. J. (2000). Relationship between critical thinking and
performance in research methodology courses. Paper presented at the Annual
Meeting of the Mid-South Educational Research Association, Lexington, Kentucky.

Davidson, B. W. (1998). Comments on Dwight Atkinson’s “A Critical Approach to
Critical Thinking in TESOL”: A case for critical thinking in the English language
classroom. TESOL Quarterly, 32(1), 119−123.

Durkin, K. (2008). The adaptation of East Asian masters students to western norms of
critical thinking and argumentation in the UK. Intercultural Education, 19(1),
15−27.

Ennis, R. H. (1987). A taxonomy of critical thinking dispositions and abilities. In J. Baron,
& R. Sternberg (Eds.), Teaching thinking skills: Theory and practice (pp. 9−26). New
York: Freeman.

Ennis, R. H. (1998). Is critical thinking culturally biased? Teaching Philosophy, 21(1),
15−33.

Facione, P. A. (1990). Critical thinking: A statement of expert consensus for purposes of
educational assessment and instruction. American Philosophical Association (ERIC
Document Reproduction Services no. ED315 423).

Friesen, W. (2008). Diverse Auckland: The face of New Zealand in the 21st century?
Wellington, New Zealand: Asia New Zealand Foundation.

Gaudet, S., & Clément, R. (2009). Forging an identity as a linguistic minority: Intra- and
intergroup aspects of language, communication and identity in Western Canada.
International Journal of Intercultural Relations, 33, 213−227.

Gieve, S. (1998). Comments on Dwight Atkinson’s “A Critical Approach to Critical
Thinking in TESOL”: A case for critical thinking in the English language classroom. A
reader reacts. TESOL Quarterly, 32(1), 123−129.

Groenvynck, H., Beirens, K., Arends-Tóth, J. V., & Fontaine, J. J. R. (2006). Factorial
Validity of the Behavioral Acculturation Scale: Evidence for a Bi-Factorial Model.
Ghent University, Belgium and Tilburg University, The Netherlands.

Halpern, D. F. (1998). Teaching critical thinking for transfer across domains:
Dispositions, skills, structure training, and metacognitive monitoring. American
Psychologist, 53, 449−455.

Halpern, D. F. (2003). Thought and knowledge: An introduction to critical thinking, 4th
ed. . Mahwah, NJ: Lawrence.

Halpern, D. F. (2006). Halpern critical thinking assessment using everyday situations:
Background and scoring standards. Unpublished manuscript. Claremont, CA:
Claremont McKenna College.

Halpern, D. F. (2007). The nature and nurture of critical thinking. In R. J. Sternberg, H. L. I.
Roediger, & D. F. Halpern (Eds.), Critical thinking in psychology (pp. 1−14). New York:
Cambridge University Press.

Hamamura, T., Heine, S. J., & Paulhus, D. L. (2008). Cultural differences in response
styles: The role of dialectical thinking. Personality and Individual Differences, 44,
932−942.

Hau, K. T., Ho, I. T., Lai, Y., Ku, K. Y. L., & Hui, N. H. H. (2008). Chinese students’ critical
thinking: Validation of the factorial structure of a critical thinking assessment.
Paper presented at the American Educational Research Association Annual Meeting,
New York, NY.

Jose, P. E. (2008). ModGraph-I: A programme to compute cell means for the graphical
display of moderational analyses: The internet version, Version 2.0. Victoria
University of Wellington, Wellington, New Zealand. Retrieved May 28, 2009, from
http://www.victoria.ac.nz/psyc/staff/paul-jose-files/modgraph/modgraph.php

Kim, H. S. (2002). We talk, therefore we think? A cultural analysis of the effect of talking
on thinking. Journal of Personality and Social Psychology, 83, 828−842.

Kumaravadivelu, B. (2003). Problematizing cultural stereotypes in TESOL. TESOL
Quarterly, 37(4), 709−719.

Lee, K. S., & Carrasquillo, A. (2006). Korean college students in United States:
Perceptions of professors and students. College Student Journal, 40(2), 442−456.

Leung, K., & van de Vijver, F. J. R. (2008). Strategies for strengthening causal inferences
in cross cultural research: The consilience approach. International Journal of Cross
Cultural Management, 8, 145−169.

Lun, V. M. C., Fischer, R., & Ward, C. (2009). Teaching critical thinking across cultures: A
study of the university course syllabi in New Zealand and Hong Kong. In R. Ismail,
M. E. J. Macapagal, N. M. Noor, J. Takai, & T. Hur (Eds.), Global issues and challenges in
a changed world: Psychological, cultural and group relationships (pp. 131−147). Kota
Kinabalu, Malaysia: Center for Research and Innovation, Universiti Malaysia Sabah.

MacKinnon, D. P., Fairchild, A. J., & Fritz, M. S. (2007). Mediation analysis. Annual Review
of Psychology, 58, 593−614.

MacKinnon, D. P., Lockwood, C. M., Hoffman, J. M., West, S. G., & Sheets, V. (2002). A
comparison of methods to test mediation and other intervening variable effects.
Psychological Methods, 7(1), 83−104.

Matsumoto, D., & Yoo, S. H. (2006). Toward a new generation of cross-cultural research.
Perspectives in Psychological Science, 1, 234−250.

New Zealand Ministry of Education. (2008). International enrolments in New Zealand.
Retrieved October 24, 2008, from http://www.educationcounts.govt.nz/publica-
tions/international/15260/28332/24707.

Nickerson, R. S., Perkins, D., & Smith, E. E. (1985). Teaching thinking. Hillsdale, NJ:
Erlbaum Associates.

Norenzayan, A., Smith, E. E., Kim, B. J., & Nisbett, R. E. (2002). Cultural preferences for
formal versus intuitive reasoning. Cognitive Science, 26, 653−684.

Nisbett, R. E., Peng, K., Choi, I., & Norenzayan, A. (2001). Culture and systems of thought:
Holistic versus analytic cognition. Psychological Review, 108(2), 291−310.

OECD. (2009). Education at a Glance 2009: OECD Indicators. Retrieved February 25,
2010, from http://www.oecd.org/dataoecd/41/25/43636332

Paas, F., Renkl, A., & Sweller, J. (2003). Cognitive load theory and instructional design:
Recent developments. Educational Psychologist, 38, 63−71.

Pascarella, E. T., & Terenzini, P. T. (2005). How college affects students: A third decade of
research. San Francisco: Jossey-Bass.

Paton, M. (2005). Is critical analysis foreign to Chinese students? In E. Manalo, & G.
Wong-Toi (Eds.), Communication skills in university education: The international
dimension (pp. 1−11). Auckland, New Zealand: Pearson Education New Zealand.

Paulson, M. J., & Lin, T. (1970). Predicting WAIS IQ from Shipley–Hartford scores. Journal
of Clinical Psychology, 22, 294−296.

Peng, K., & Nisbett, R. E. (1999). Culture, dialectics, and reasoning about contradiction.
American Psychologist, 54, 741−754.

Peng, K., & Nisbett, R. E. (2000). Dialectical responses to questions about dialectical
thinking. American Psychologist, 55, 1067−1068.

Pithers, R. T., & Soden, R. (2000). Critical thinking in education: A review. Educational
Research, 42(3), 237−249.

Preacher, K. J., & Hayes, A. F. (2004). SPSS and SAS procedures for estimating indirect
effects in simple mediation models. Behavior Research Methods, Instruments, &
Computers, 36, 717−731.

Preacher, K. J., & Hayes, A. F. (2008). Asymptotic and resampling strategies for assessing
and comparing indirect effects in multiple mediator models. Behavior Research
Methods, 40, 879−891.

616 V.M.-C. Lun et al. / Learning and Individual Differences 20 (2010) 604–616

Renaud, R. D., & Murray, H. G. (2008). A comparison of a subject-specific and a general
measure of critical thinking. Thinking Skills and Creativity, 3, 85−93.

Robertson, M., Line, M., Jones, S., & Thomas, S. (2000). International students, learning
environments and perceptions: A case study using the Delphi technique. Higher
Education Research & Development, 19(1), 89−102.

Shin, K. R. (1998). Critical thinking ability and clinical decision-making skills among
senior nursing students in associate and baccalaureate. Journal of Advanced Nursing,
27, 414−418.

Shipley, W. C. (1940). A self-administering scale for measuring intellectual impairment
and deterioration. Journal of Psychology, 9, 371−377.

Sobel, M. E. (1982). Asymptotic intervals for indirect effects in structural equations
models. In S. Leinhart (Ed.), Sociological methodology 1982 (pp. 290−312). San
Francisco: Jossey-Bass.

Spector, P. E., Schneider, J. R., Vance, C. A., & Hazlett, S. A. (2000). The relation of
cognitive ability and personality traits to assessment center performance. Journal of
Applied Social Psychology, 30(7), 1474−1491.

Spencer-Rodgers, J., Peng, K., Wang, L., & Hou, Y. (2004). Dialectical self-esteem and
East–West differences in psychological wellbeing. Personality and Social Psychology
Bulletin, 30(1), 1416−1432.

Spencer-Rodgers, J., Srivastava, S., & Peng, K. (2001). The dialectical self scale.
Unpublished data, as cited in Spencer-Rodgers, J., Peng, K., Wang, L., & Hou, Y.
(2004). Dialectical self-esteem and East–West differences in psychological well-
being. Personality and Social Psychology Bulletin, 30, 1416–1432.

Taube, K. T. (1997). Critical thinking ability and disposition as factors of performance on
a written critical thinking test. Journal of General Education, 46, 129−164.

Takano, Y., & Noda, A. (1993). A temporary decline of thinking ability during foreign
language processing. Journal of Cross-Cultural Psychology, 24(4), 445−462.

Ten Dam, G., & Volman, M. (2004). Critical thinking as a citizenship competence:
Teaching strategies. Learning and Instruction, 14, 359−379.

Tweed, R. G., & Lehman, D. R. (2002). Learning considered within a cultural context:
Confucian and Socratic approaches. American Psychologist, 57(2), 89−99.

Ward, C. (1996). Acculturation. In D. Landis, & R. S. Bhagat (Eds.), Handbook of
intercultural training (pp. 124−147)., 2 ed. Thousand Oaks, CA: Sage.

Watkins, D. A., & Biggs, J. B. (2001). The Paradox of the Chinese Learner and Beyond. In
D. A. Watkins, & J. B. Biggs (Eds.), Teaching the Chinese learner: Psychological and
pedagogical perspectives (pp. 3−23). Hong Kong: Comparative Education Research
Centre, The University of Hong Kong.

Watson, G., & Glaser, E. M. (1980). Watson–Glaser Critical Thinking Appraisal: Forms A
and B. San Antonio, TX: The Psychological Corporation.

Watson, G., & Glaser, E. M. (1994). Watson–Glaser Critical Thinking Appraisal, Form S
manual. San Antonio, TX: The Psychological Corporation.

Watson, G., & Glaser, E. M. (2006). Watson–Glaser Critical Thinking Appraisal-Short
Form Manual. San Antonio, TX: Harcourt Assessment.

Wechsler, D. (1981). Manual for the Wechsler Adult Intelligence Scale-Revised. New
York: Psychological Corporation.

Wechsler, D. (1997). Wechsler Adult Intelligence Scale-Third Edition. San Antonio, TX:
Psychological Corporation.

Zachary, R. A. (1991). Shipley Institute of Living Scale Revised Manual. Los Angeles, CA:
Western Psychological Services.

Zachary, R. A., Paulson, M. J., & Gorsuch, R. (1985). Estimating WAIS IQ from the Shipley
Institute of Living Scale using continuously adjusted age norms. Journal of Clinical
Psychology, 41(6), 820−831.

• Exploring cultural differences in critical thinking: Is it about my thinking style or the language I speak?

Introduction
Do Asian students think less critically?
Critical thinking — a multi-dimensional construct
Focus of the present research
Cultural differences in critical thinking
Dialectical thinking
English proficiency
Possible consequence of cultural differences in critical thinking
Overview of the present research
Study 1: pilot study with the HCTAES
Method
Participants
Materials
Halpern Critical Thinking Assessment using Everyday Situations (HCTAES)
Dialectical Self Scale (DSS)
Perceived proficiency in reading and writing English (English)
Analytical strategy
Results and discussion
Cultural differences in critical thinking skills and other self-report measures
Mediation between culture and critical thinking skills

Study 2a: assessment with the WGCTA-SF
Method
Participants
Materials
Watson–Glaser Critical Thinking Appraisal Short Form (WGCTA-SF)
Shipley Institute of Living Scale (SILS)
Dialectical Self Scale (DSS)
Perceived English proficiency (English)
Cultural adoption of behavioral norms in New Zealand

Results and discussion
Cultural differences in WGCTA, SILS scores, and other self-report measures
Mediation between culture and critical thinking skills
Relationship between DSS and WGCTA across cultures

Study 2b: critical thinking skills and academic performance
Participants
Course grades in the introductory management course
Results and Discussion
Cultural difference in academic performance
Relationship between critical thinking and academic performance across cultures
General discussion
Dialectical thinking
English proficiency
Implication of the practice of critical thinking in international education
Limitations and future research
Acknowledgements
References

Contents lists available at ScienceDirect

Thinking Skills and Creativity

journal homepage: www.elsevier.com/locate/tsc

A meta-analysis on critical thinking and community college student
achievement☆

Carlton J. Fonga,⁎, Yughi Kima, Coreen W. Davisa, Theresa Hoanga, Young Won Kimb

a Department of Curriculum and Instruction, Texas State University, USA
b Department of Educational Psychology, University of Texas at Austin, USA

A R T I C L E I N F O

Keywords:
Critical thinking
Academic achievement
Community college
Two-year college
Meta-analysis

A B S T R A C T

The purpose of this meta-analysis was to examine the relationship between student levels of
critical thinking (as established via critical thinking tests) and community college student suc-
cess. We conducted a meta-analysis to synthesize the extant literature on critical thinking and
community college student success. After systematically searching the relevant literature through
electronic databases using an array of search terms, we screened studies and reviewed them for
inclusion. Included studies were reliably coded using a protocol to extract correlational effect size
data and study characteristics. These effect sizes were aggregated meta-analytically. From a total
of 23 studies (27 samples, N =

82

33), we found that critical thinking was moderately related to
community college student success. The relationship between student achievement and levels of
critical thinking (as skills or dispositions) was constant for both nursing and non-nursing students
as well as for grades and individual test outcomes; however, students’ levels of critical thinking
was more strongly associated with longer-term outcomes compared with those at the shorter
term. Meta-regression results indicated some evidence that effects were weaker for male and
minority community college students. Implications for this study include the importance of
cultivating critical thinking for all community college students and the exploration of under-
studied areas for future research.

1. Introduction

Critical thinking is arguably one of the most valued learning goals in postsecondary education (Gellin, 2003; Pithers & Soden,
2000; Wilson & Wagner, 19

81

). In an expanding world of information, there is growing need for individuals to process data, evaluate
ideas, and reason through arguments (Pascarella & Terenzini, 2005). Critical thinking is particularly critical for community college
students engaged in vocational training or preparing for transfer to four-year institutions. Community college enrollment has rapidly
expanded over the last century, affording greater participation in higher education, especially for individuals with limited oppor-
tunities (Goldrick-Rab, 2010; Stevens & Kirst, 2015; Zientek, Yetkiner Ozel, Fong, & Griffin, 2013). In fact, recent statistics indicated
that nearly 40% of students in higher education are enrolled in community colleges (Shapiro, Dundar, Yuan, Harrell, & Wakhungu,
2014). However, despite the recent rise of community colleges, failure to match students to credentials is a persistent issue. Research
has consistently shown low degree completion rates (Fike & Fike, 2008); in fact, only one third of community college students earned
a credential within six years (Calcagno, Bailey, Jenkins, Kienzl, & Leinbach, 2006). Moreover, some reports indicated that almost half

http://dx.doi.org/10.1016/j.tsc.2017.06.002
Received 20 October 2016; Received in revised form 10 April 2017; Accepted 6 June 2017

☆ Parts of this paper were presented at the annual meeting of the Council for the Study of Community Colleges. This project has been funded through an American
Psychological Association Educational Psychology Early Career Grant.
⁎ Corresponding author at: Department of Curriculum and Instruction, Texas State University, San Marcos, TX,

78

666, USA.
E-mail address: carltonfong@txstate.edu (C.J. Fong).

Thinking Skills and Creativity 26 (2017) 71–

83

Available online 15 June 2017
1871-1871/ © 2017 Elsevier Ltd. All rights reserved.

MARK

http://www.sciencedirect.com/science/journal/18711871

http://www.elsevier.com/locate/tsc

http://dx.doi.org/10.1016/j.tsc.2017.06.002

http://dx.doi.org/10.1016/j.tsc.2017.06.002

mailto:carltonfong@txstate.edu

http://dx.doi.org/10.1016/j.tsc.2017.06.002

http://crossmark.crossref.org/dialog/?doi=10.1016/j.tsc.2017.06.002&domain=pdf

of community college students dropped out within their first year (Xu & Jaggars, 2011).
The preponderance of the literature examining factors affecting community college students’ academic success and persistence

has tended to concentrate on variables such as first-generation status, socioeconomic status, and prior school achievement. Although
it is vital to identify such background factors, the use of prescriptive measures which assess students’ cognitive, motivational, and
behavioral variables affecting access, success, and retention cannot be overlooked (Fong et al., 2017). By identifying variables which
can be enhanced through educational interventions, such as students’ critical thinking skills, educators and practitioners can design
and implement interventions to help students improve in these areas.

1.1. Critical thinking

Critical thinking is an elusive construct to define and understand, partly due to a variety of definitions in the field (Moore, 2013).
For our study, a working definition of critical thinking is “purposeful, self-regulatory judgment that results in interpretation, analysis,
evaluation, and inference, as well as explanations of the considerations which that judgment is based”. Critical thinking has been
conceived of in two ways: skills and disposition. Critical thinking skills involve a set of capacities including interpreting, predicting,
analyzing, and evaluating (Abrami et al., 2008). On the other hand, dispositions refer to an individual’s proclivities and personality
characteristics related with aspects of critical thinking such as curiosity, inquisitiveness, open-mindedness, and prudence in decision-
making (Facione, Facione, & Sanchez, 1994). Some of the most common measurements of critical thinking skills and/or dispositions
are the Watson-Glaser Critical Thinking Appraisal tool (Watson & Glaser, 1964), Cornell Critical Thinking Test (Ennis,
Millman, & Tomko, 1985), Motivated Strategies for Learning Questionnaire (Pintrich, Smith, Garcia, & McKeachie, 1993), and Cali-
fornia Critical Thinking Disposition Inventory (Facione, Facione, & Giancarlo, 2001).

Critical thinking has been used as an appropriate framework for adult education (Garrison, 1992). Applying critical thinking to
adult learners, Brookfield (1987) outlined five phases: a triggering event, appraisal of the situation, exploration and explanation,
development of alternatives, and integration of perspectives. Particularly important for adult learners who prefer self-directed
learning, critical thinking places on the learner the responsibility of making sense of a situation or integrating new ideas with
previous knowledge. In a variety of educational settings, there is evidence that supports how critical thinking increases learning
outcomes (Facione, 2009; Halpern, 1998; Heijltjes, van Gog, Leppink, & Paas, 2014). Evidence from a prior meta-analytic review has
illustrated the positive impact of the college experience on developing critical thinkers (Huber & Kuncel, 2016; McMillan, 1987).
Moreover, another recent meta-analysis has documented effective strategies that target critical thinking. Despite recent syntheses on
critical thinking, the context of the community college and the role of critical thinking in student achievement have yet to be explored
meta-analytically.

1.2. Critical thinking and community college students

In the community college setting overall, critical thinking is considered an ideal learning outcome (Bers, McGowan, & Rubin,
1996). However, there is inconsistent evidence regarding the boon of critical thinking for community college students. For example,
Thompson (2009) examined two groups of two-year college students: a successful group (grades of A/B) and an unsuccessful group
(grades of C or worse). There were significantly different levels of critical thinking disposition between the two groups with the
successful group reporting high levels of critical thinking. Alternatively, in a distance-learning class, Puzziferro (2006) also compared
successful and unsuccessful students on their course grades and critical thinking but found no significant differences. Similarly,
Singleton-Williams (2009) found a non-significant correlation between students’ levels of critical thinking and final course grades in
an introductory computer application class. Whereas some studies may show negative effects (e.g., Foust, 2005), some studies show
no effects or positive relations with achievement (e.g., Thompson, 2006). Due to the discrepancies in the literature as well as the
prevalence of critical thinking as a proposed goal of the community college experience, a research synthesis to aggregate previous
findings is timely and important.

One subpopulation among community college students in which critical thinking is particularly salient is nursing students.
Because of the rapidly changing healthcare environment and the need to quickly process and interpret multiple sources of in-
formation, critical thinking is an indisputable characteristic nursing educators desire to cultivate within their students
(Scheffer & Rubenfeld, 2000). Moreover, because community colleges produce a significant amount of nurses in our society
(Jacobs & Dougherty, 2006), cultivating critical thinking at the community college level is becoming increasingly important.

1.3. Factors related to critical thinking and college student achievement

A number of factors may influence the relationship between students’ levels of critical thinking and their academic achievement.
Student background characteristics, such as demographic information like gender and ethnicity, have been examined in the literature
on critical thinking; however, the evidence is mixed. Regarding gender, some studies have reported no differences on critical thinking
measures, differences favoring women, or differences favoring men (Giancarlo & Facione, 2001; King, Wood, & Mines, 1990). Re-
garding ethnicity, Terenzini, Springer, Pascarella, and Nora (1995) found that ethnicity (White vs. non-White) was not a significant
predictor of students’ levels of critical thinking skills. Furthermore, a comparison between traditional college students and first-
generation students, who tend to consist of historically underrepresented minorities, yielded no significant differences in critical
thinking skills (Terenzini, Springer, Yaeger, Pascarella, & Nora, 1996). Although ethnicity may not play a direct role in students’
levels of critical thinking skills, there is compelling evidence regarding the influence of diversity and diversity-related experiences

C.J. Fong et al. Thinking Skills and Creativity 26 (2017) 71–83

72

that enhance students’ cognitive and intellectual development (Mayhew et al., 2016). For example, in a meta-analysis by Bowman
(2010), the strongest positive associations between diversity experiences and college student cognitive development were found
when collegians had interpersonal, interracial experiences. To further clarify how gender, ethnicity, and diversity moderate the
relation between students’ levels of critical thinking and their achievement, we planned moderator tests to assess these student
characteristics.

Another important college student characteristic to consider is students’ academic discipline or major. Similarly, the literature
regarding the influence of academic major on critical thinking is mixed (Mayhew et al., 2016). A large number of studies have
demonstrated that students in the physical sciences, life sciences, engineering, and math generally had significantly higher critical
thinking scores than students in the humanities and social sciences (Arum & Roksa, 2011; Brint, Cantwell, & Saxena, 2012; King et al.,
1990). On the other hand, research has indicated no significant differences among majors (Li, Long, & Simpson, 1999). In particular,
Pike, Kuh, and McCormick (2011) found disciplinary differences on higher order thinking among college freshmen, yet these dif-
ferences were not apparent among college seniors. Categorizing majors into practice disciplines (i.e., nursing, education, business)
and nonpractice disciplines (i.e., English, history, psychology), Walsh and Hardy (1999) found that in general, students in non-
practice disciplines had more favorable levels of critical thinking dispositions. In the community college literature, critical thinking
has been frequently studied in the nursing field. Therefore, the current meta-analysis will measure disciplinary differences as a
moderator, namely between nursing fields and non-nursing fields, mirroring the contrast of practice discipline versus nonpractice
discipline, used by Walsh and Hardy (1999).

1.4. Research questions

Our meta-analytic study was guided by two research questions: 1) What is the relationship (direction and magnitude) between
critical thinking and community college student success in the existing literature? 2) What factors explain variation in the re-
lationship between critical thinking and community college student success?

2. Method

Research syntheses aggregate the findings of studies that address the same research question. Bowman (2012) encouraged the use
of quantitative research synthesis, or meta-analysis, in higher education research. Despite the rise of meta-analytic research in
education, use of this methodology in the community college field is scant. In the following sections, we briefly outline the meth-
odological and analytic approaches used in our meta-analysis, which is closely aligned with the methods from Cooper, Hedges, and
Valentine (2009) and Cooper (2010).

2.1. Literature search and inclusion criteria

The first step in conducting the research synthesis was to exhaustively search the literature for studies on critical thinking and
community college success. In order to do so, we used an array of search strategies to comprehensively uncover relevant studies (see
Table 1). We opted to choose a broad set of keyword terms because researchers may not explicitly mention or describe critical
thinking. Retrieved studies must have included at least one search term in the following three domains: predictors related to critical
thinking, populations related to community college, and outcomes related to achievement and persistence. We searched the following
electronic databases: ERIC, PsycINFO, and Proquest Dissertation and Theses Full Text. The literature search did not have an explicit start
date and ended in January 2015. In addition, we conducted a number of hand searches in the most prominent community college
journals, namely, Community College Journal of Research and Practice and Community College Review.

After all citations were retrieved, we screened titles and abstracts of each of the citations for potential relevance to the topic. The
full-text documents of the resulting pool of studies were located and screened using the following inclusion criteria (Fig. 1 depicts the
search retrieval and screen flow chart). Within each full-text document, we also conducted an ancestry search, examining the

Table 1
Search Strategy.

Domain Keywords

Predictor “critical thinking” OR “creative thinking” OR “study habits” OR “study skills” OR “problem solving” OR “conflict resolution” OR “decision
making” OR “psychosocial factors” OR “time management” OR adjustment OR “communication skills” OR adaptability OR “career development”
OR “vocational maturity” OR self-efficacy OR “student attitudes” OR planning OR conscientiousness OR “work attitudes” OR involvement OR
determin* OR factor* OR variable* OR parameter* OR reason* OR caus* OR correlat* R antecedent* OR predictor*

Population “community college” OR “two year college” OR “junior college” OR “two-year college” OR undergraduate OR freshman OR sophomore OR
“junior student” OR “senior student” OR “transfer student” OR “vocational school” OR “vocational college” OR “technical school” OR “technical
college” OR “associate’s degree” OR “city college”

Outcome “academic achievement” OR “academic performance” OR grade* OR scholastic OR grade point average OR GPA OR degree OR “college
performance” OR “college achievement” OR mark OR graduation OR completion OR attainment OR “retention” OR “school holding power” OR
“academic persistence” OR attendance OR dropout OR “dropping out” OR “enrollment management” OR “student attrition” OR truancy OR
withdrawal

Note. Asterisks denote truncation, and quotation marks surround a phrase. Each domain of keywords is linked by the AND Boolean.

C.J. Fong et al. Thinking Skills and Creativity 26 (2017) 71–83

73

reference lists to uncover any other potentially relevant studies.
To be included in our meta-analysis, a study must have had the following components: a) community college sample, b) critical

thinking variable, c) student success outcome, and (d) effect size information to estimate a correlational relationship between critical
thinking and a success outcome. For the sample, students had to be enrolled in a community college, two-year vocational school, or
two-year technical school. For the critical thinking variable, any measure of critical thinking skills or dispositions must have been
used. Measurements of critical thinking can be a formal assessment of critical thinking skills (i.e., Watson-Glaser Critical Thinking
Assessment) or a self-reported measure regarding the efficacy for or use of critical thinking skills as well as dispositions regarding
critical thinking.

For the outcome criteria, we included studies with persistence- and achievement-related outcomes. Achievement-related out-
comes consisted of course grades, GPA, and test scores such as standardized achievement measures (Persistence outcomes were not
reported in relation to critical thinking in our pool of included studies because we only found one study with persistence outcomes:
Eason, 1986). For the final criterion, effect size information was required for a study to be included, which mainly consisted of
Pearson r correlations.

2.2. Information retrieval and coding

Once full-texts were deemed to meet our inclusion criteria, a team of five coders retrieved information on various characteristics
for each of the research reports. Because every study report differs in how much detail it provides, we inferred a number of codes
when necessary by using pre-established definitions to code ambiguous characteristics. The codes were categorized into four do-
mains: research report, sample, predictor variable, and outcome variable.

Research report characteristics. First, we coded characteristics regarding the report, such as the author name, year of the
report, and type of report. We were mainly interested in the type of report to ascertain whether a work was published in a peer-
reviewed journal or not. This allowed us to assess publication bias, an important feature when conducting a meta-analysis (Polanin,
Tanner-Smith, & Hennessy, 2016).

Sample characteristics. Second, we coded for sample characteristics in each report. We coded for the community college setting
as well as demographic characteristics such as age, gender, ethnicity, and socioeconomic status. We also coded for educational
characteristics related to the sample, including program/major, full-time status, and hours worked per week.

Predictor characteristics. Third, we coded aspects of the predictor variable such as the type of psychosocial variable and study
authors’ descriptions of the predictor. We also noted if there was a scale or instrument name, reliability of the measure, and domain.
Two aspects of interest were whether the critical thinking measure described skills or a disposition and whether the measure was self-
reported (i.e., self-efficacy for or use of skills) versus a formal assessment of their critical thinking abilities.

Outcome characteristics. Fourth, we coded characteristics of the outcome. Namely, we were interested in student success
outcomes, which comprise of either retention in community college, degree attainment, and course completion or achievement-
related outcomes such as grades, GPA, or tests. We also coded for the duration of the persistence measure (one semester or beyond
one semester) to compare the shorter-term impacts with longer-term ones.

Coder reliability. All reports were coded independently by five trained coders. The coders had experience in meta-analytic
coding and were extensively trained on each code using the previously mentioned coding frame. For a reliability check, we compared
all pairs of codes for each study between two coders. We calculated a reliability measure by dividing the number of matched codes by
the total number of codes. Disagreements were noted and resolved by another coder. Coder interrater reliability was high with an

Fig. 1. Search retrieval and screening flow diagram.

C.J. Fong et al. Thinking Skills and Creativity 26 (2017) 71–83

74

agreement rate of 96%.

2.3. Effect size calculation and data integration

Effect sizes were computed as Pearson’s r. We converted the correlations to Fisher’s z, conducted our analyses, then converted
back to r. When possible, we extracted correlations and sample sizes from correlation tables or text in the reports. If data were only
available from means, standard deviations, and sample sizes of two groups (i.e., high achievers and low achievers), we estimated a
correlation. When this information was not reported in a study, corresponding inference test statistics (e.g., t-statistic, F-statistic, p-
values, chi-squared test) were used to derive an effect size. If statistical significance was denoted yet both raw data and inferential test
statistics were unavailable, a conservative effect size was derived with an assumed p-value of 0.05.

We used a shifting-unit-of-analysis approach (Cooper, 2010) in order to deal with the issue of determining what constitutes an
independent estimate of effect. This approach involves coding as many effect sizes from each study that exist as a result of variations
within the study. Then, we averaged effects appropriately in analyses to prevent violating the assumption of independence at various
data points.

Accounting for sample size, we used weighted procedures to calculate average effect sizes across all comparisons. The weights
were calculated by multiplying each independent effect size by the inverse of its variance and then the sum of these products is then
divided by the sum of the inverses (see Cooper et al., 2009). We also calculated 95% confidence intervals (CI) for each effect size.
Synthesis procedures were conducted on Comprehensive Meta-Analysis software (Borenstein, Hedges, Higgins, & Rothstein, 2005).

Because of inherent measurement error in instruments (in our case, critical thinking measures), we also calculated unattenuated
correlation effect sizes. Using psychometric methods recommended by Borenstein, Hedges, Higgins, and Rothstein (2009), we
converted observed or attenuated correlations to adjusted or unattenuated correlations, using the reliability coefficient of each
critical thinking measure. We then conducted a second overall meta-analysis using the adjusted effect sizes. Using estimated total
variance of the observed effect estimates and the estimated variance of the true effects, we calculated a proportion of variance in the
observed or attenuated effects explained by artifacts of measurement error or sampling error in the critical thinking variable.

In a research synthesis, there is a theoretical possibility of not obtaining all studies that have investigated the relationship between
critical thinking and community college student success, either due to failure on the part of the meta-analyst to retrieve all relevant
reports or censoring on the part of individual authors. Therefore, we employed Duval and Tweedie’s (2000) trim-and-fill procedure to
assess whether the effect size distribution differed from normally distributed estimates. Trim-and-fill methods impute missing values
that would be present to achieve an approximately normal distribution of effect sizes. Thus, we provided corrected effect sizes when
effect sizes were deemed missing on the right or left size of the distribution to indicate the impact of data censoring on the effect size
distribution.

Analyses were conducted using both random error (RE) assumptions and fixed error (FE) assumptions. Random error assumptions
account for variance at the study level (Hedges & Vevea, 1998), which is assumed to be an additional source of random variation,
whereas a fixed of error does not but rather, assumes that differences among participants are the sole source of error. However,
neither model is perfect. Fixed error assumptions do not take into consideration how study-level features contribute random error to a
set of correlational data. Overton (1998) cautioned that random effects can overestimate error variance and produce overly con-
servative confidence intervals when assumptions are violated. Because both models can be flawed, we provide our results using both
FE and RE as a way of sensitivity analysis as recommended by Greenhouse and Iyengar (1994). If there are discrepancies between
both sets of error, i.e., a significant result using FE and a nonsignificant result using RE, then we will interpret our findings presuming
that a significant result using both models of error is considered the most robust (Cooper, 2010). This approach follows other meta-
analytic studies in the educational and psychological literature (Dent & Koenka, 2016; Patall, Cooper, & Robinson, 2008).

Moderator analyses. Effect sizes may vary even if they estimate the same underlying population value. Therefore, homogeneity
analyses were necessary to examine whether sampling error alone accounted for this variance compared to the observed variance
caused by study features. We tested homogeneity of the observed set of effect sizes using a within-class goodness-of-fit statistic (Qw).
A significant Qw statistic suggests that sampling variation alone could not adequately explain the variability in the effect size esti-
mation, so moderators should then be examined (Cooper et al., 2009). Similarly, homogeneity analyses can be used to determine
whether multiple groups of average effect sizes vary more than predicted by sampling error. In this case, statistical differences among
different categories of studies were tested by computing the between-class goodness-of-fit statistic, Qb. A significant Qb statistic
indicates that average effect sizes vary between categories of the moderator variables more than predicted by sampling error alone.
For continuous moderators, we used meta-regression to assess the moderation of continuous variables on the correlation between
critical thinking and student success.

3. Results

After employing our search for relevant studies, we uncovered 11,832 unique reports. Upon reviewing titles and abstracts, we
selected 697 studies for full-text retrieval in order to evaluate them using the inclusion criteria. Our final pool included 27 samples (k)
from 23 studies spanning the years 19

76

–2014. The total sample size of participants across all studies was 8233 (N) with an average
sample size of 316.7 per study. After using Grubbs’ test to identify outliers within that set of effect sizes, no outliers were detected.

C.J. Fong et al. Thinking Skills and Creativity 26 (2017) 71–83

75

T
ab

le
2

C
h
ar
ac
te
ri
st
ic
s
of

In
cl
u
d
ed

St
u
d
ie
s.

A
u

th
or

(Y
ea
r)

P
u
bl
ic
at
io
n
T
yp

e
Sa

m
p
le

Si
ze

(F
em

al
e%

)
M
in
or
it
y
(%

)
M
aj
or
/
P
ro
gr
am

C
T
T
yp

e
C
T
M
ea
su
re

(r
el
ia
bi
li
ty
)

O
u
tc
om

e
D
u
ra
ti
on

r
r

ru

B
ac
h
m
an

(1
9
9
8
)

D
is
se
rt
at
io
n

7
8
(9
2
%
)

N
R

N
u
rs
in
g

D
C
C
T
D
I
(0
.9
0
)

G
P
A
/G

ra
d
es

>

1
se
m
es
te
r

0
.2
6

0
.2
7

0
.2
9

S
C
C
T
ST

(0
.7
8
)

0
.2
8

C
or
n
el
iu
s
(2
0
1
1
)

D
is
se
rt
at
io
n

1
7
2
(9
3
%
)

6
%

N
u
rs
in
g

S
C
T
A

(

0
.6
9

*)

G
P
A
/G
ra
d
es

>
1
se
m
es
te
r

0
.2
2

0
.2
2
0
.2
6

C
ri
n
er

(1
9
9
2
)

D
is
se
rt
at
io
n

4
1
(7
3
%
)

N
R

N
on

-n
u
rs
in
g

S
N
JT

R
S
(0
.7
0
*)

G
P
A
/G
ra
d
es
>
1
se
m
es
te
r

0
.4
2

0
.4
2

0
.5
0

E
as
on

(1
9
8
6
)

D
is
se
rt
at
io
n

1
8
7
(6
2
%
)

N
R
N
on
-n
u
rs
in
g

S
W
G
C
T
A

(0
.7
7
)

G
P
A
/G
ra
d
es
1
se
m
es
te
r

−

0
.0
5

0
.0
6

0
.0
0

T
es
t
sc
or
e

>
1
se
m
es
te
r

0
.1
6

Fo
u
st

(2
0
0
8
)

D
is
se
rt
at
io
n

2
1
(6
2
%
)

2

0
%

N
on
-n
u
rs
in
g

S

M
SL

Q
(0
.1
6
)

G
P
A
/G
ra
d
es
1
se
m
es
te
r

−
0
.3
5

−
0
.3
5

−
0
.8
8

G
ay

th
w
ai
te

(2
0
0
6
)

D
is
se
rt
at
io
n

5
7
(

6
7
%

)

2
3
%

N
on
-n
u
rs
in
g
S
M
SL

Q

(0
.8
0
)

G
P
A
/G
ra
d
es
1
se
m
es
te
r

0
.1
4

0
.1
4

0
.1
5

G
u
st
er

an
d
B
at
t
(1
9
8
9
)

A
rt
ic
le

5
0
(5
8
%
)

N
R
N
on
-n
u
rs
in
g
S
W
G
C
T
A

(0
.8
3
)

T
es
t
sc
or
e
1
se
m
es
te
r
0
.1
4
0
.1
4
0
.1
5

H
ea
rr
on

(1
9
9
1
)

D
is
se
rt
at
io
n

2
9
4
(7
7
%
)

1
6
%

N
on
-n
u
rs
in
g

S
T
O
LT

(0
.8
3
)
G
P
A
/G
ra
d
es
1
se
m
es
te
r

0
.4
7

0
.4
7

0
.5
1

H
u
ro
v
(1
9
8
7
)

R
ep

or
t

1
1
8
(8
0
%
)

1
0
%

N
u
rs
in
g
S
W
G
C
T
A
(0
.8
0
)
G
P
A
/G
ra
d
es
1
se
m
es
te
r

0
.2
5

0
.2
5
0
.2
8

K
u
zn

ar
(2
0
0
9
)

D
is
se
rt
at
io
n

1
3
6
(5
3
%
)

1
0
%
N
u
rs
in
g
S
M
SL

Q
(0
.7
9
)

T
es
t
sc
or
e
1
se
m
es
te
r

−

0
.0
9

−
0
.0
9

−

0
.1
0

Le
e

(2
0
0
1
)

D
is
se
rt
at
io
n

8
4
(8
5
%
)

N
R
N
u
rs
in
g
S
W
G
C
T
A

(0
.7
3
)

G
P
A
/G
ra
d
es
>
1
se
m
es
te
r

0
.1
3

0
.1
3
0
.1
5

M
on

ey
(1
9
9
7
)

D
is
se
rt
at
io
n

6
8
(5

9
%

)

0
%
N
u
rs
in
g

S
C
C
T
T
(

0
.6
2

*)

G
P
A
/G
ra
d
es
1
se
m
es
te
r

−
0
.0
1

−
0
.0
1

−
0
.0
2

5
6
(5
9
%
)

0
%
N
on
-n
u
rs
in
g
0
.1
3
0
.1
3

0
.1
7

5
7
(5
9
%
)

0
%

0
.3
0

0
.3
0

0
.3
8

P
ar
le
tt

(2
0
1
2
)

D
is
se
rt
at
io
n

4
7
(6
7
%
)

9
%
N
u
rs
in
g
S
M
SL

Q
(0
.7
0
*)

G
P
A
/G
ra
d
es
>
1
se
m
es
te
r

0
.4
9

0
.4
9

0
.5
8

P
it
ts

(2
0
0
1
)
D
is
se
rt
at
io
n

1
6
1
(8
5
%
)

2
5
%

N
u
rs
in
g
S
C
C
T
ST

(0
.6
2
)

G
P
A
/G
ra
d
es
1
se
m
es
te
r

0
.4
4

0
.3
0

0
.3
7

>
1
se
m
es
te
r

0
.3
9

6
7
(9
0
%
)

2
6
%

D
C
C
T
D
I
(0
.8
8
)

1
se
m
es
te
r
0
.1
6
>
1
se
m
es
te
r
0
.1
7
S
C
C
T
ST
(0
.6
2
)
1
se
m
es
te
r
0
.1
3
0
.1
4
0
.1
5
>
1
se
m
es
te
r
0
.3
0
D
C
C
T
D
I
(0
.8
8
)
1
se
m
es
te
r
−
0
.0
1
>
1
se
m
es
te
r
0
.1
5

2
2
8
(8
6
%
)

2
5
%
S
C
C
T
ST
(0
.6
2
)
1
se
m
es
te
r
0
.3
0

0
.2
3

0
.3
0
>
1
se
m
es
te
r

0
.3
6

D
C
C
T
D
I
(0
.8
8
)
1
se
m
es
te
r
0
.0
9
>
1
se
m
es
te
r
0
.1
7

4
9
(8
8
%
)

4
8
%

S
C
C
T
ST
(0
.6
2
)
1
se
m
es
te
r

0
.3
1

0
.3
0

0
.3
2

>
1
se
m
es
te
r

0
.3
4

D
C
C
T
D
I
(0
.8
8
)
1
se
m
es
te
r
0
.2
8
>
1
se
m
es
te
r
0
.2
8

P
u
zz
if
er
ro

(2
0
0
6
)
D
is
se
rt
at
io
n

8
1
5
(8
0
%
)

2
6
%
N
on
-n
u
rs
in
g
S
M
SL

Q
(0
.8
0
)

G
P
A
/G
ra
d
es
1
se
m
es
te
r
0
.0
0
0
.0
0
0
.0
0

R
ei
d
(2
0
0
0
)

D
is
se
rt
at
io
n

4
1
7
(8
6
%
)

2
4
%

N
u
rs
in
g
S
C
C
T
ST

(0
.7
1
)

G
P
A
/G
ra
d
es
>
1
se
m
es
te
r
0
.2
2
0
.2
2
0
.2
6

Si
lv
er

(1
9
9
9
)

D
is
se
rt
at
io
n

3
9
8
(7
2
%
)

N
R
N
on
-n
u
rs
in
g

S
SS

SE
S
(0
.8
8
)

G
P
A
/G
ra
d
es
>
1
se
m
es
te
r

0
.6
5

0
.6
5
0
.6
9

Si
n
gl
et
on

-W
il
li
am

s
(2
0
0
9
)

D
is
se
rt
at
io
n

8
5
(6
8
%
)

6
7
%
N
on
-n
u
rs
in
g
S
M
SL
Q
(0
.8
0
)
G
P
A
/G
ra
d
es
1
se
m
es
te
r
0
.0
9
0
.0
9
0
.1
0

Si
su
n
g
(2
0
0
5
)

D
is
se
rt
at
io
n

1
0
1
5
(8
3
%
)

6
%
N
on
-n
u
rs
in
g

S
C
C
A
P
(0
.8
2
)

T
es
t
sc
or
e
1
se
m
es
te
r
0
.3
0
0
.3
0

0
.3
3

T
h
om

p
so
n
(2
0
0
6
)

D
is
se
rt
at
io
n

1
8
5
(8
6
%
)

N
R
N
u
rs
in
g

S
N
E
T
(0
.9
3
*)

G
P
A
/G
ra
d
es
1
se
m
es
te
r

0
.4
8

0
.4
8
0
.5
0
T
h
om

p
so
n
(2
0
0
9
)

D
is
se
rt
at
io
n

3
2
(N

R
)

N
R
N
u
rs
in
g

D
C
C
T
D
I
(0
.9
3
)

G
P
A
/G
ra
d
es
1
se
m
es
te
r
0
.6
2
0
.6
2

0
.6
4

Y
os
t
(2
0
0
3
)

D
is
se
rt
at
io
n

4
0
(9
0
%
)

N
R
N
u
rs
in
g
S
M
SL
Q
(0
.8
0
)
G
P
A
/G
ra
d
es
1
se
m
es
te
r
0
.0
6
0
.0
5
0
.0
5
T
es
t
sc
or
e
1
se
m
es
te
r

0
.1
1

N
ot
e.

A
u
th
or

n
am

es
ar
e
li
m
it
ed

to
ju
st

th
e
su
rn
am

e
of

th
e
fi
rs
t
au

th
or
;
fo
r
th
e
fu
ll
au

th
or

li
st
,
p
le
as
e
re
fe
r
to

th
e
re
fe
re
n
ce

li
st
.
C
T
=

C
ri
ti
ca
l
T
h
in
ki
n
g;

N
R
=

n
ot

re
p
or
te
d
;
D
=

D
is
p
os
it
io
n
;
S
=

Sk

il
ls

;
C
C
A
P
=

C
ol
le
gi
at
e

A
ss
es
sm

en
t
of

A
ca
d
em

ic
P
ro
fi
ci
en

cy
;
C
C
T
D
I
=

C
al
if
or
n
ia

C
ri
ti
ca
l
T
h
in
ki
n
g
D
is
p
os
it
io
n
In
ve

n
to
ry
;
C
C
T
ST

=
C
al
if
or
n
ia

C
ri
ti
ca
l
T
h
in
ki
n
g
Sk

il
ls

T
es
t;
C
C
T
T
=

C
or
n
el
l
C
ri
ti
ca
l
T
h
in
ki
n
g
T
es
t;
C
T
A
=

C
ri
ti
ca
l
T
h
in
ki
n
g
A
ss
es
sm

en
t;

M
SL

Q
=

M
ot
iv
at
ed

St
ra
te
gi
es

fo
r
Le

ar
n
in
g
Q
u
es
ti
on

n
ai
re
;
N
E
T
=

N
u
rs
e
E
n
tr
an

ce
T
es
t;
N
JT

R
S
=

N
ew

Je
rs
ey

T
es
t
of

R
ea
so
n
in
g
Sk

il
ls
;
SS

SE
S
=

St
u
d
y
Sk

il
ls

Se
lf
-e
ffi
ca
cy

Su
rv
ey

(T
ex
t-
ba

se
d
C
ri
ti
ca
l
T
h
in
ki
n
g)
;
T
O
LT

=
T
es
t
of

Lo
gi
ca
l
T
h
in
ki
n
g;

W
G
C
T
A
=

W
at
so
n
G
la
se
r
C
ri
ti
ca
l
T
h
in
ki
n
g
A
p
p
ra
is
al
s.

R
el
ia
bi
li
ti
es

of
th
e
sc
al
es

ar
e

p
ro
vi
d
ed

in
p
ar
en

th
es
es

af
te
r
sc
al
e
ab

br
ev

ia
ti
on

s.
W
h
en

re
li
ab

il
it
ie
s
fo
r
th
e
in
cl
u
d
ed

st
u
d
y
sa
m
p
le

w
er
e
n
ot

p
ro
vi
d
ed

,
re
li
ab

il
it
ie
s
fr
om

th
e
ci
te
d
li
te
ra
tu
re

w
er
e
in
cl
u
d
ed

as
es
ti
m
at
es

(t
h
es
e
ar
e
m
ar
ke

d
by

an
*)
.

r
=

w
ei
gh

te
d
co

rr
el
at
io
n
fo
r
ea
ch

u
n
iq
u
e
sa
m
p
le
;
ru

=
u
n
at
te
n
u
at
ed

w
ei
gh
te
d
co
rr
el
at
io
n
fo
r
ea
ch

u
n
iq
u
e
sa
m
p
le
.

C.J. Fong et al. Thinking Skills and Creativity 26 (2017) 71–83
76

3.1. Characteristics of included studies

In this section, we want to highlight salient characteristics of our pool of included studies (see Table 2 for the comprehensive list
of characteristics of included studies). The first notable characteristic is the large majority of unpublished studies, namely doctoral
dissertations, in our database of studies. Second, regarding student characteristics, there was a large proportion of nursing majors in
the included samples. Third, measures of critical thinking mostly assessed skills, and only a few studies assessed dispositions. Some
studies used measures that asked students to self-report their perceptions of their confidence to employ critical thinking skills (e.g.,
Motivated Strategies Learning Questionnaire, MSLQ). In contrast, other instruments consisted of test questions (e.g., California
Critical Thinking Skills Test), which judge their ability to perform on multiple-choice items. All instruments were established scales,
validated by previous research. Lastly, academic achievement outcomes presented in the included studies were course grades or GPA,
with a smaller segment of studies using individual exams as the dependent variable. Outcomes were assessed at the end of a semester
(the duration of one course) or beyond one semester.

3.2. Overall analysis of the relationship between community college and achievement

First, we examined the overall relationship between critical thinking and community college student achievement (see Table 3).
Under a fixed effects model, the weighted average r was 0.26 with a 95% CI of 0.23–0.28. Under a random effects model, the
weighted average r was 0.24 with a 95% CI of 0.15–0.33. Therefore, the hypothesis that the relationship between critical thinking

Table 3
Overall Meta-Analysis Results of Critical Thinking and Community College Achievement.

Fixed Effects Random Effects

k r 95% CI k r 95% CI Qw

Critical Thinking 27 0.26*** 0.23, 0.28 27 0.24*** 0.15, 0.33 247.12***
Trim-and-Fill (FE) 30 0.27*** 0.25, 0.30 30 0.28*** 0.19, 0.36 2

79

.03***
Trim-and-Fill (RE) 32 0.32*** 0.29, 0.34 32 0.30*** 0.21, 0.38 368.79***

Note. Trim-and-fill procedures were conducted using both fixed effects (FE) and random effects (RE); trimmed studies were to the right of the mean. ***p < 0.001.

Fig. 2. Forest plot of studies. The size of points reflects weight in meta-analysis.

C.J. Fong et al. Thinking Skills and Creativity 26 (2017) 71–83

77

and achievement is equal to zero was rejected under both FE and RE models (p < 0.001). See Fig. 2 for forest plot. When meta- analyzing the unattenuated correlations (the adjusted effect sizes correcting for measurement error using scale reliabilities), the weighted average effect size was 0.28. The psychometric analysis would estimate the proportion of explained variance is 0.74, indicating that 74% of the total variance in the attenuated effect sizes is explained by sampling and measurement error.

Using trim-and-fill analyses to look for asymmetry using both fixed- and random-error models (Duval & Tweedie, 2000), we
searched for possible missing effects on the right side of the distribution, which would increase the size of the slightly positive average
r for community college students on achievement. Using the fixed-effects model, we found evidence that three effect sizes might have
been missing on the right side. Imputing these values would change the correlation between critical thinking and achievement to
r = 0.27 (95% CI = 0.25, 0.30) under fixed effects and r = 0.28 (95% CI = 0.19, 0.36) under random effects. Using the random-
effects model, we found evidence that five effect sizes might have been missing on the right side. Imputing these values would change
the correlation between critical thinking and achievement to r = 0.32 (95% CI = 0.29, 0.34) under fixed effects and r = 0.30 (95%
CI = 0.21, 0.38) under random effects. Thus, when accounting for possible data censoring, the relationship between critical thinking
and community college student achievement is slightly stronger.

In addition, the tests of the distribution of the effect sizes revealed that we could reject the hypothesis that the effects were
estimating the same underlying population value, Q(26) = 247.12, p < 0.001; therefore, there was sufficient heterogeneity among the effect sizes that could be explained by moderator variables, which we will examine in the following section.

3.3. Moderator analyses

Next, we examined two categories of moderators: categorical moderators, which include publication status (published vs. un-
published), student program/major (nursing vs. non-nursing), predictor type (skills vs. disposition), and outcome type (test score vs.
GPA) and duration (one semester vs. beyond one semester); and continuous moderators which include percent female composition
and percent minority student composition (see Table 4 for categorical moderator results). These continuous moderators were cal-
culated because effect sizes were not often provided by gender or ethnicity. Therefore, we created continuous variables of the
percentage of the sample that was female or minority status as a proxy for these demographic variables.

For the first categorical moderator, we found no significant differences between published and unpublished studies, indicating
that there does not seem to be a publication bias in this literature under both fixed and random effects (FE: Q = 0.35, p = 0.55; RE:
Q = 0.11, p = 0.75). See Fig. 3 for funnel plot. Second, we assessed differences between nursing (k = 14) and non-nursing students
(k = 13); non-nursing students consisted of collegians from business, music, computer science, and biology. Under both fixed and
random effects, we did not observe a significant difference between the two groups of students (FE: Q = 1.86, p = 0.17; RE:
Q = 0.13, p = 0.72). Third, we assessed whether the relationship between critical thinking and achievement would significantly
differ if critical thinking was measured as skills (k = 24) or a disposition (k = 4). Moderator tests revealed no significant differences,
under both fixed (Q = 0.62, p = 0.43) and random effects (Q = 0.76, p = 0.38). Fourth, we assessed if self-reported measures versus
formal assessments moderated the relationship between students’ levels of critical thinking and academic achievement. Under fixed
effects only, there was a significant difference between self-reported measures of critical thinking (r = 0.21) and ability tests of
critical thinking (r = 0.28), Q = 6.48, p = 0.011. A significant difference was not detected under random effects, Q = 0.43,

Table 4
Categorical Moderator Analyses for Critical Thinking and Community College Achievement.

Fixed Effects Random Effects

k r Qb 95% CI r Qb 95% CI

Publication Status 0.35 0.11
Published 2 0.21*** 0.07, 0.35 0.21*** 0.07, 0.35
Unpublished 27 0.26** 0.23, 0.28 0.24*** 0.15, 0.34

Critical Thinking Type 0.62 0.76
Skills 24 0.25*** 0.23, 0.28 0.23*** 0.13, 0.32
Disposition 4 0.31*** 0.16, 0.45 0.33** 0.11, 0.53

Type of Measure 6.48* 0.54
Self-reported 11 0.21*** 0.16, 0.25 0.18 −0.06, 0.40
Ability test 16 0.28*** 0.25, 0.31 0.27*** 0.20, 0.34

Major/Program 1.86 0.13
Nursing 17 0.23*** 0.19, 0.28 0.22*** 0.14, 0.31
Non-nursing 13 0.27*** 0.24, 0.30 0.26*** 0.09, 0.40

Outcome Type 0.03 0.79
GPA/Grades 25 0.26*** 0.23, 0.29 00.25*** 0.05, 0.20
Tests/Exams 3 0.25*** 0.20, 0.31 0.10 −0.21, 0.40

Outcome Duration 34.05*** 0.65
One semester 22 0.18*** 0.14, 0.22 0.21*** 0.11, 0.31
> one semester 11 0.33*** 0.30, 0.37 0.28*** 0.14, 0.42

Note. Sample sizes varied per analysis due to the shifting unit of analysis approach and the existence of multiple effect sizes per study; * p < 0.05, ** p < 0.01, ***p < 0.001.

C.J. Fong et al. Thinking Skills and Creativity 26 (2017) 71–83
78

p = 0.46, but mean point estimates were trending in a similar direction to the fixed effects model.
Fifth, we compared the effects of critical thinking on individual test scores such as nursing licensure exams (k = 3) with overall

GPA (k = 24), which also resulted in nonsignificant differences (FE: Q = 0.03, p = 0.87; RE: Q = 0.79, p = 0.38). Lastly, we tested
outcome duration as a moderator variable, one semester (k = 22) compared to beyond one semester (k = 11). Under fixed effects
only, there was a significant difference with a larger average effect size for achievement outcomes beyond one semester (r = 0.33)
and a smaller average effect size for outcomes at one semester (r = 0.18), Q = 34.06, p < 0.001. A significant difference was not detected under random effects, Q = 0.65, p = 0.41, but the observed mean effect sizes were in a similar direction to the fixed effects model. Forest plots for categorical moderator analyses are present in supplementary materials.

For the continuous moderators, we found significant moderation of gender (k = 25) and minority status (k = 17) under fixed
effects only. Specifically, using the percentage of females in the student sample composition, we found a slope value for gender on
effect sizes significantly different from zero (FE: β = 0.25, p = 0.02, RE: β = 0.29, p = 0.42). This result suggests that as female
percentage increases, the relation of critical thinking and achievement is higher. In addition, using the percentage of historically
underrepresented minority students (non-White, non-Asian) in the student sample composition, we found a slope value for minority
status on effect sizes significantly different from zero (FE: β = −0.27, p = 0.02, RE: β = −0.09, p = 0.68). Thus, as minority status
percentage increases, the relation of critical thinking and achievement was lower. However, due to incomplete data reporting in the
primary studies, the sample sizes of these moderator tests were reduced, and results should be interpreted with caution.

In sum, critical thinking had a moderate influence on community college achievement across a variety of student programs and
outcome types with no sign of publication bias. Under fixed effects only, it appeared that critical thinking was more strongly related
to longer-term achievement outcomes beyond the one semester mark. Examining demographic characteristics as moderators revealed
that critical thinking seemed more impactful for women and majority students but under fixed effects only.

4. Discussion

In sum, the results of our meta-analysis indicated positive relationships between critical thinking and community college student
success. According to Cohen’s (1992) benchmarks for evaluating effect sizes, the magnitude of the relationships was small to
moderate. Additionally, when conducting sensitivity analyses to impute missing effect sizes and correct for attenuated correlations
affected by measurement error and internal consistency of critical thinking scales, the truer weighted average effect sizes was even a
bit larger. Given the low rates of degree completion that plague U.S. two-year colleges and the complexity of community college
persistence and achievement, identifying positive factors, such as critical thinking, that are malleable and can bolster student success
is surely worthwhile. Not accounting for other background characteristics, critical thinking skills and dispositions seem to positively
relate with community college student success.

Comparing the relationship between critical thinking and achievement from previous meta-analytic reviews on four-year college
or university students, we observed that the average weighted correlations in our meta-analysis were markedly higher. In a meta-
analysis by Robbins, Lauver, Le, Davis, and Langley (2004), the observed mean correlation between academic-related skills (aca-
demic and cognitive strategies) and GPA was 0.129. Similarly, in an updated review by Richardson, Abraham, and Bond (2012), the
observed average correlation between critical thinking and achievement (correcting for sampling error) was 0.15. The effects in our
sample of community college studies ranged from 0.24 to 0.32. In our community college sample, the critical thinking-achievement
relationship was twice the magnitude. One may argue that compared to the four-year university setting, attaining higher achievement

Fig. 3. Funnel plot of studies.

C.J. Fong et al. Thinking Skills and Creativity 26 (2017) 71–83
79

at community colleges requires students to develop higher levels of critical thinking because of the increased demands of work and
family responsibilities. Terenzini et al. (1996) suggested that college students who work longer hours off-campus and juggle multiple
responsibilities (like many community college students) can cultivate better time management skills, self-discipline, and academic
focus, which in turn lead to greater cognitive development and higher levels of critical thinking.

Furthermore, because critical thinking development has been found because of the significant vocational focus at the community
college level, perhaps critical thinking is more relevant to success as students directly engage in workforce preparation. For instance,
in community colleges, nurses-in-training, which comprised a large portion of the meta-analytic student sample, are taught to reason
critically about the judgments they make in their day-to-day responsibilities and the life-and-death decisions in nursing practice
(Colucciello, 1997). Thus, the demand for high levels of critical thinking for the success of community college nursing students may
explain the larger associations in our meta-analysis compared to previous reviews.

Moderator results. Results from our moderator tests revealed interesting patterns regarding the demographic characteristics of
the community college samples as well as measurement and methodology issues. Before interpreting our moderator results, one
caveat to understanding the significance level of the contrasts is when there are discrepant findings between fixed and random effects.
When this occurs, the random effects model is considered more robust, so significant findings solely under fixed effects should be
interpreted with caution. In general, moderator results in meta-analysis are exploratory given the nature of across-study synthesis.
Also, we reiterate the caution required when interpreting results from these exploratory tests given the small number of studies that
comprise moderator categories.

Regarding student characteristics, critical thinking was not moderated by nursing student status, indicating that critical thinking
influences achievement equally for nursing and non-nursing disciplines. In spite of the emphasis of critical thinking in nursing
education, critical thinking seemed to be equally important across majors. Under fixed effects, taking into consideration gender and
minority status, evidence that the relationship between students’ levels of critical thinking and achievement decreased with samples
of greater male representation and minority representation is noteworthy. With regards to small correlations for students with greater
numbers of minority students, this result leads to an interesting discussion on the impact of racial diversity and diversity experiences
on cognitive development. From the literature, one may expect that as minority composition increases, critical thinking would be
further cultivated and lead to increased achievement due to the positive influence of diversity experiences on students’ levels of
critical thinking (Bowman, 2010). One possible explanation to disentangle this effect comes from research that demonstrated di-
versity experiences to be more impactful for White students’ levels of critical thinking skills compared to non-White students (Loes,
Pascarella, & Umbach, 2012). Although samples with a greater minority student composition, in theory, have more opportunities for
interpersonal diversity experiences, the minority students themselves may not have as a large of a benefit on their cognitive de-
velopment and thus academic achievement.

Although these findings were only apparent under fixed effects, it is interesting that the achievement of minority males did not
benefit as much from critical thinking, especially given the attention focused on their remediation in the community college lit-
erature. Harris and Wood (2014) developed the Socio-Ecological Outcomes model to capture the variety of factors that come into play
regarding male minority student success in community colleges. Factors from the individual, academic and environmental contexts,
and campus ethos (and the interactions among them) are proposed to increase student success. Thus, a multitude of other factors
apart from critical thinking skills and dispositions are required to understand all community college students and their success,
particularly minority male collegians.

From a methodological perspective, moderator tests indicated that skills and dispositions were equally related with community
college achievement, under both fixed and random effects. However, given the few studies that measured dispositions, further
research in this area is required. One significant methodological moderator under fixed effects only involved whether critical thinking
was measured via a self-report survey or ability test, with the latter having stronger associations with academic achievement. This
finding suggests that ability tests are potentially more valid when related with achievement, as self-reported surveys may introduce
more measurement error. Regarding outcome characteristics, the relationship between critical thinking and achievement was not
moderated by achievement type (GPA/grades or tests/exams); similarly, the small number of studies contributing effect sizes for tests
and exams warrants additional scholarship. Another significant moderator result in our study (under fixed effects) revealed that effect
sizes were larger when achievement was measured beyond one semester compared to at the one semester mark. This suggests that
critical thinking had a stronger relationship with achievement in the long run, which can be possibly explained through the cu-
mulative effects of psychological strategies which develop and strengthen over time (Dent & Koenka, 2016).

Publication bias. Although the moderator analysis comparing published and unpublished studies did not indicate a publication
bias (given the few published studies in our sample), the more notable result is the large majority of unpublished studies in our
dataset. This is indicative of the field of community college research. Crisp, Carales, and Nuñez (2016) observed how despite the long
history of community colleges in the U.S., only since the 1990s have research in this area emerged. They recognized that the majority
of this research may be unpublished, and our study corroborates such findings along with previous meta-analyses (Fong et al., 2017).
Whereas some scholars may scrutinize the heavy use of unpublished data in a meta-analysis citing lack of quality as the primary
criticism, the accepted and encouraged practice in contemporary meta-analysis in the education field is to include all viable data
regardless of publication status (Polanin et al., 2016). This can aid in mitigating the threat of publication bias.

4.1. Implications for research and practice

This study has several important implications for research and practice, but for the purposes of this study, we focus on a few
salient points. First, regarding implications for future research, our systematic synthesis revealed a few gaps in the literature. For

C.J. Fong et al. Thinking Skills and Creativity 26 (2017) 71–83

80

example, our search through the literature exposed a dearth of studies that focus on other important community college outcomes
such as persistence, degree attainment, transfer, and job placement and the role of critical thinking. Only one study (Eason, 1986)
measured credits earned and course completion as an outcome in relation to critical thinking with small or negative correlations
ranging from −0.04 to 0.05. Further research is needed to examine persistence-related outcomes beyond achievement and test
outcomes.

By affirming the positive relationship between critical thinking and community college success, this study offers administrators
and practitioners accumulated and synthesized evidence for informed decision-making and practices. In particular, our findings can
be used to build programs, initiatives, orientations, success courses, and trainings that focus on bolstering critical thinking in
community college students (Piergiovanni, 2014). Given the substantive relationship between students’ levels of critical thinking and
community college student success, one suggestion is to teach students the importance of critical thinking and its relevance for
academic achievement. Previous research has underscored the use of critical thinking courses to further cognitive development
(Mayhew et al., 2016). These courses along with other opportunities can serve as the context for faculty to discuss with students their
understanding of critical thinking, its bearing on their academic performance, and how it can be developed further. Additionally,
critical thinking can also be infused in instructional practices as recommended by meta-analysis, which emphasized two proven
strategies to increase critical thinking. First, opportunities of discussion or dialogue where teachers pose questions in groups or
whole-class instruction were helpful. Second, exposing students to authentic tasks with role playing or applied problem solving
promoted critical thinking. Third, in combination with these two strategies, including mentorship as an additional strategy appeared
to augment the critical thinking process. Furthermore, when measuring students’ levels of critical thinking, formal ability tests like
the Watson Glaser Critical Thinking Appraisal may be more preferable given the stronger associations between tests and achievement
compared to using self-reported surveys.

However, from our moderator results, extra supports beyond critical thinking training need to be offered for collegiate men and
students of color who may not benefit as much from boosts in critical thinking. In general, the small to moderate correlations within
this study, especially the smaller correlations for samples with greater proportions of male and minority students, demonstrate the
need to examine other factors that influence community college achievement.

Finally, we hope our study provides an example and encouragement for the use of meta-analysis as a methodological tool to
understand the landscape of higher education research. Systematic review and synthesis techniques can afford researchers, admin-
istrators, educators, and practitioners the ability to confidently evaluate, measure, target, and change the most important factors
related to community college student outcomes.

4.2. Limitations to generalizability

There were a number of limitations to this study. The first limitation of all research syntheses in general consist of synthesis-
generated evidence, which should not be interpreted as supporting causal relationships (Cooper, 2010). For example, it is not ne-
cessarily the case of our meta-analysis that critical thinking causes higher achievement because high achieving students may possess
greater critical thinking. Therefore, when significant differences were found within a research synthesis, results should be interpreted
with caution and used to direct future research of these factors in a controlled design to appropriately infer causal impact. It is also
important to recognize that the findings were based on small numbers of effect sizes, making it difficult to place great confidence in
the magnitude of the estimated effects.

In addition, meta-analysts depend on the authors of the primary studies to provide all relevant data and characteristics. Thus,
there were many important variables that could not be examined as moderators due to incomplete reporting. Sample characteristics
such as age, full-time status, hours working, and distance versus traditional learning modes (which are very significant factors when
studying community college students) could not be assessed. One important research design issue that was not inferable from the
included studies was the timeframe of critical thinking assessment. A researcher could foreseeably assess critical thinking at the
beginning of community college as a predictor for success or towards the end of their schooling. The latter situation would be
confounded with the influence of postsecondary instructive over the duration of the college experience (Huber & Kuncel, 2016).
Future research is encouraged to systematically examine these factors to see their moderating influence on the relationship between
students’ levels of critical thinking and postsecondary student success.

Another limitation of the study is the psychometric approach towards operationalizing critical thinking taken in this study. Liu,
Frankel, and Roohr (2014) argued for a parsimony between authenticity and psychometric quality, citing that measures that rely on
multiple-choice tests are not fully capturing the complexities of critical thinking. Rather, measures that prioritize psychometric
quality may be tapping into other related constructs such as reading comprehension or general linguistic competencies. Although the
current study has solely focused on these types of measures, we encourage future research to examine other ways of understanding
and measuring critical thinking that reflects the authenticity of both the construct and the contexts in which students use their critical
thinking skills.

Appendix A. Supplementary data

Supplementary data associated with this article can be found, in the online version, at http://dx.doi.org/10.1016/j.tsc.2017.06.
002.

C.J. Fong et al. Thinking Skills and Creativity 26 (2017) 71–83
81

http://dx.doi.org/10.1016/j.tsc.2017.06.002

http://dx.doi.org/10.1016/j.tsc.2017.06.002

References1

Abrami, P. C., Bernard, R. M., Borokhovski, E., Wade, A., Surkes, M. A., Tamim, R., et al. (2008). Instructional interventions affecting critical thinking skills and
dispositions: A stage 1 meta-analysis. Review of Educational Research, 78, 1102–1134. http://dx.doi.org/10.3102/0034654308326084.

Arum, R., & Roksa, J. (2011). Academically adrift: limited learning on college campuses. University of Chicago Press.
*Bachman, M. L. (1998). Anxiety, critical thinking and age as performance predictors of community college nursing students. CO: Colorado State University Fort Collins

[Unpublished doctoral dissertation].
Bers, T. H., McGowan, M., & Rubin, A. (1996). The disposition to think critically among community college students: The California critical thinking dispositions

inventory. The Journal of General Education, 45, 197–223.
Borenstein, M., Hedges, L., Higgins, J., & Rothstein, H. (2005). Comprehensive meta-analysis (Version 2.1) [Computer software]. Englewood, NJ: BioStat.
Borenstein, M., Hedges, L., Higgins, J., & Rothstein, H. (2009). Introduction to meta-analysis. West Sussex, UK: Wiley & Sons.
Bowman, N. A. (2010). College diversity experiences and cognitive development: A meta-analysis. Review of Educational Research, 80, 4–33. http://dx.doi.org/10.

3102/0034654309352495.
Bowman, N. A. (2012). Effect sizes and statistical methods for meta-analysis in higher education. Research in Higher Education, 53, 375–382. http://dx.doi.org/10.

1007/s11162-011-9232-5.
Brint, S., Cantwell, A. M., & Saxena, P. (2012). Disciplinary categories majors, and undergraduate academic experiences: Rethinking Bok’s underachieving colleges

thesis. Research in Higher Education, 53, 1–25.
Brookfield, S. D. (1987). Developing critical thinkers. San Francisco, CA: Jossey-Bass.
Calcagno, J., Bailey, T., Jenkins, D., Kienzl, G., & Leinbach, D. T. (2006). IIs student right-to-know all you should know? An analysis of community college graduation

rates. Research in Higher Education, 47, 491–519. http://dx.doi.org/10.1007/s11162-005-9005-0.
Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155–159. http://dx.doi.org/10.1037/0033-2909.112.1.155.
Colucciello, M. L. (1997). Critical thinking skills and dispositions of baccalaureate nursing students: A conceptual model for evaluation. Journal of Professional Nursing,

13, 236–245.
Cooper, H., Hedges, L. V., & Valentine, J. C. (Vol. Eds.), (2009). The handbook of research synthesis and meta-analysis. 1Russell Sage Foundation.
Cooper, H. (2010). Synthesizing research: A guide for literature reviews (4th ed.). Thousand Oaks, CA: Sage.
*Cornelius, S. T. (2011). Prior grade point average, social and financial support, and standardized test scores as predictors of success among associates degree nursing students

(Unpublished doctoral dissertation). Ann Arbor, MI: Walden University.
*Criner, L. A. (1992). Teaching thinking and reasoning: A study of critical thinking in adults (Unpublished doctoral dissertation). Fayetteville, AR: University of Arkansas.
Crisp, G., Carales, V. D., & Núñez, A. M. (2016). Where is the research on community college students. Community College Journal of Research and Practice, 40, 767–778.
Dent, A. L., & Koenka, A. C. (2016). The relation between self-regulated learning and academic achievement across childhood and adolescence: A meta-analysis.

Educational Psychology Review, 28, 425–474. http://dx.doi.org/10.1007/s10648-015-9320-8.
Duval, S., & Tweedie, R. (2000). A nonparametric trim and fill method of accounting for publication bias in meta-analysis. Journal of the American Statistical Association,

95, 89–98. http://dx.doi.org/10.1080/01621459.2000.10473905.
*Eason, L. E. (1986). The relationship of critical thinking skills and psychological type in community college students’ responses to science instruction (Unpublished doctoral

dissertation). Gainesville, FL: University of Florida.
Ennis, R. H., Millman, J., & Tomko, T. N. (1985). Cornell critical thinking tests level X & level Z: Manual. Pacific Grove, CA: Midwest Publications.
Facione, N. C., Facione, P. A., & Sanchez, C. A. (1994). Critical thinking disposition as a measure of competent clinical judgment: The development of the California

Critical Thinking Disposition Inventory. Journal of Nursing Education, 33(8), 345–350.
Facione, P. A., Facione, N. C., & Giancarlo, C. A. F. (2001). California critical thinking disposition inventory: CCTDI. California Academic Press.
Facione, P. A. (2009). Critical thinking: What it is and why it counts. [Retrieved from http://www.insightassessment.com/9articles%20WW.html].
Fike, D. S., & Fike, R. (2008). Predictors of first-year student retention in the community college. Community College Review, 36, 68–88. http://dx.doi.org/10.1177/

0091552108320222.
Fong, C. J., Davis, C. W., Kim, Y., Kim, Y. W., Marriott, L., & Kim, S. (2017). Psychosocial factors and community college success: A meta-analytic investigation. Review

of Educational Research, 87, 388–424. http://dx.doi.org/10.3102/0034654316653479.
*Foust, R. A. (2008). Learning strategies, motivation, and self-reported academic outcomes of students enrolled in web-based coursework (Unpublished doctoral dissertation)

Detroit, MI: Wayne State University.
Garrison, D. R. (1992). Critical thinking and self-directed learning in adult education: an analysis of responsibility and control issues. Adult Education Quarterly, 42(3),

136–148.
*Gaythwaite, E. S. (2006). Metacognitive self-regulation, self-efficacy for learning and performance, and critical thinking as predictors of academic success and course retention

among community college students enrolled in online, telecourse, and traditional public speaking courses (Unpublished doctoral dissertation). Orlando, FL: University of
Central Florida.

Gellin, A. (2003). The effect of undergraduate student involvement on critical thinking: A meta-analysis of the literature 1991–2000. Journal of College Student
Development, 44, 746–762. http://dx.doi.org/10.1353/csd.2003.0066.

Giancarlo, C. A., & Facione, P. A. (2001). A look across four years at the disposition toward critical thinking among undergraduate students. Journal of General
Education, 50, 29–55.

Goldrick-Rab, S. (2010). Challenges and opportunities for improving community college student success. Review of Educational Research, 80, 437–469. http://dx.doi.
org/10.3102/0034654310370163.

Greenhouse, J. B., & Iyengar, S. (1994). Sensitivity analysis and diagnosis. In H. Cooper, & S. Iyengar (Eds.), Handbook of research synthesis (pp. 383–398). New York:
Russell Sage Foundation.

*Guster, D., & Batt, R. (1989). Cognitive and affective variables and their relationships to performance in a lotus 1–2-3 class. Collegiate Microcomputer, 7, 151–156.
Halpern, D. F. (1998). Teaching critical thinking for transfer across domains: Dispositions, skills, structure training, and metacognitive monitoring. American

Psychologist, 53, 449–455. http://dx.doi.org/10.1037/0003-066X.53.4.449.
Harris, F., III, & Wood, J. L. (2014). The socio-ecological outcomes model: A framework for examining men of color’s experiences and success in community colleges. San Diego,

CA: Minority Male Community College Collaborative.
*Hearron, M. C. (1991). The predictive effect of logical thinking, prior knowledge, and learning style characteristics on academic achievement in an anatomy and physiology

course (Unpublished doctoral dissertation). Commerce, TX: East Texas State University.
Hedges, L. V., & Vevea, J. L. (1998). Fixed-and random-effects models in meta-analysis. Psychological Methods, 3, 486–504.
Heijltjes, A., Van Gog, T., Leppink, J., & Paas, F. (2014). Improving critical thinking: Effects of dispositions and instructions on economics students’ reasoning skills.

Learning and Instruction, 29, 31–42. http://dx.doi.org/10.1016/j.learninstruc.2013.07.003.
Huber, C. R., & Kuncel, N. R. (2016). Does college teach critical thinking? A meta-analysis. Review of Educational Research, 86, 431–468. http://dx.doi.org/10.3102/

0034654315605917.
*Hurov, J. T. (1987). A study of the relationship between reading, computational, and critical thinking skills and academic success in fundamentals of chemistry (Report)St.

Louis, MO: Saint Louis Community College.
Jacobs, J., & Dougherty, K. J. (2006). The uncertain future of the community college workforce development mission. New Directions for Community Colleges,

2006(136), 53–62.

1 Denoted with an * were included in the meta-analysis.

C.J. Fong et al. Thinking Skills and Creativity 26 (2017) 71–83
82

http://dx.doi.org/10.3102/0034654308326084

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0010

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0015

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0015

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0020

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0020

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0025

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0030

http://dx.doi.org/10.3102/0034654309352495

http://dx.doi.org/10.3102/0034654309352495

http://dx.doi.org/10.1007/s11162-011-9232-5

http://dx.doi.org/10.1007/s11162-011-9232-5

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0045

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0045

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0050

http://dx.doi.org/10.1007/s11162-005-9005-0

http://dx.doi.org/10.1037/0033-2909.112.1.155

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0065

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0065

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0070

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0075

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0080

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0080

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0085

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0090

http://dx.doi.org/10.1007/s10648-015-9320-8

http://dx.doi.org/10.1080/01621459.2000.10473905

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0105

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0105

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0110

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0115

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0115

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0120

http://www.insightassessment.com/9articles%20WW.html

http://dx.doi.org/10.1177/0091552108320222

http://dx.doi.org/10.1177/0091552108320222

http://dx.doi.org/10.3102/0034654316653479

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0140

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0140

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0145

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0145

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0150

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0150

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0150

http://dx.doi.org/10.1353/csd.2003.0066

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0160

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0160

http://dx.doi.org/10.3102/0034654310370163

http://dx.doi.org/10.3102/0034654310370163

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0170

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0175

http://dx.doi.org/10.1037/0003-066X.53.4.449

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0185

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0190

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0195

http://dx.doi.org/10.1016/j.learninstruc.2013.07.003

http://dx.doi.org/10.3102/0034654315605917

http://dx.doi.org/10.3102/0034654315605917

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0210

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0210

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0215

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0215

King, P. M., Wood, P. K., & Mines, R. A. (1990). Critical thinking among college and graduate students. The Review of Higher Education, 13, 167–186.
*Kuznar, K. A. (2009). Effects of high-fidelity human patient simulation experience on self-efficacy, motivation and learning of first semester associate degree nursing students.

Minneapolis, MN: University of Minnesota.
*Lee, G. J. (2001). An investigation of the development of critical thinking skills for traditional program and non-traditional program associate degree nursing students

(Unpublished doctoral dissertation). Moscow, ID: University of Idaho.
Li, G., Long, S., & Simpson, M. E. (1999). Self-perceived gains in critical thinking and communication skills: Are there disciplinary differences. Research in Higher

Education, 40, 43–60. http://dx.doi.org/10.1023/A:1018722327398.
Liu, O. L., Frankel, L., & Roohr, K. C. (2014). Assessing critical thinking in higher education: Current state and directions for next-generation assessment. ETS Research

Reports Series, 2014, 1–23. http://dx.doi.org/10.1002/ets2.12009.
Loes, C., Pascarella, E., & Umbach, P. (2012). Effects of diversity experiences on critical thinking skills. Journal of Higher Education, 83, 1–25.
Mayhew, M. J., Rockenbach, A. N., Bowman, N. A., Seifert, T. A. D., Wolniak, G. C., Pascarella, E. T., et al. (2016). How college affects students: 21st century evidence that

higher education works, Vol. 3. San Francisco, CA: Jossey-Bass.
McMillan, J. H. (1987). Enhancing college students’ critical thinking: a review of studies. Research in Higher Education, 26, 3–29.
Money, S. M. (1997). The Relationship Between Critical Thinking Scores, Achievement Scores, and Grade Point Average in Three Different Disciplines (Unpublished Doctoral

Dissertation). Lansing MI: Michigan State University.
Moore, T. J. (2013). Critical thinking: Seven definitions in search of a concept. Studies in Higher Education, 38, 506–522. http://dx.doi.org/10.1080/03075079.2011.

586995.
Overton, R. C. (1998). A comparison of fixed-effects and mixed (random-effects) models for meta-analysis tests of moderator variable effects. Psychological Methods,

3(3), 354–379.
*Parlett, D. K. (2012). A comparison of associate and bachelor degree nursing students’ motivation (Unpublished doctoral dissertation). Minneapolis, MN: Walden University.
Pascarella, E. T., & Terenzini, P. T. (2005). In K. A. Feldman (Vol. Ed.), How college affects students. Vol. 2. San Francisco, CA: Jossey-Bass.
Patall, E. A., Cooper, H., & Robinson, J. C. (2008). The effects of choice on intrinsic motivation and related outcomes: A meta-analysis of research findings.

Psychological Bulletin, 134, 270–300. http://dx.doi.org/10.1037/0033-2909.134.2.270.
Piergiovanni, P. R. (2014). Creating a critical thinker. College Teaching, 62, 86–93. http://dx.doi.org/10.1080/87567555.2014.896775.
Pike, G. R., Kuh, G. D., & McCormick, A. C. (2011). An investigation of the contingent relationships between learning community participation and student en-

gagement. Research in Higher Education, 52, 300–322.
Pintrich, P. R., Smith, D. A., García, T., & McKeachie, W. J. (1993). Reliability and predictive validity of the Motivated Strategies for Learning Questionnaire (MSLQ).

Educational and Psychological Measurement, 53(3), 801–813.
Pithers, R. T., & Soden, R. (2000). Critical thinking in education: A review. Educational Research, 42, 237–249. http://dx.doi.org/10.1080/001318800440579.
*Pitts, L. N. (2001). Critical thinking skill and disposition as predictors of success in associate degree nursing education (Unpublished doctoral dissertation). Gainesville, FL:

University of Florida.
Polanin, J. R., Tanner-Smith, E. E., & Hennessy, E. A. (2016). Estimating the difference between published and unpublished effect sizes: A meta-review. Review of

Educational Research, 86, 207–236. http://dx.doi.org/10.3102/0034654315582067.
*Puzziferro, M. (2006). Online technologies self-efficacy, self-regulated learning, and experiential variables as predictors of final grade and satisfaction in college-level online

courses (Unpublished doctoral dissertation). New York, NY: New York University.
*Reid, H. V. (2000). The correlation between a general critical thinking skills test and a discipline-specific critical thinking test for associate degree nursing students (Unpublished

doctoral dissertation). Denton, TX: University of North Texas.
Richardson, M., Abraham, C., & Bond, R. (2012). Psychological correlates of university students’ academic performance: A systematic review and meta-analysis.

Psychological Bulletin, 138, 353–387. http://dx.doi.org/10.1037/a0026838.
Robbins, S. B., Lauver, K., Le, H., Davis, D., Langley, R., & Carlstrom, A. (2004). Do psychosocial and study skill factors predict college outcomes? A meta-analysis.

Psychological Bulletin, 130, 261–288. http://dx.doi.org/10.1037/0033-2909.130.2.261.
Scheffer, B. K., & Rubenfeld, M. G. (2000). A consensus statement on critical thinking in nursing. Journal of Nursing Education, 39(8), 352–359.
Shapiro, D., Dundar, A., Yuan, X., Harrell, A., & Wakhungu, P. K. (2014). Completing college: A national view of student attainment rates – fall 2008 cohort (Signature report

No. 8)Herndon, VA: National Student Clearinghouse Research Center.
*Silver, B. B. (1999). Indicators of academic achievement: A structural equation model (Unpublished doctoral dissertation). Storrs, CT: The University of Connecticut.
*Singleton-Williams, S. D. (2009). Motivational and cognitive learning strategies as predictors of academic success in economically disadvantaged community college students

(Unpublished doctoral dissertation). Minneapolis, MN: Capella University.
*Sisung, J. (2005). Relationship between standardized critical thinking test scores and earned grades in courses purported to teach critical thinking at Kellogg Community College

(Unpublished doctoral dissertation). Lincoln, NE: The University of Nebraska.
Remaking college: The changing ecology of higher education. In M. Stevens, & M. Kirst (Eds.), Stanford University Press.
Terenzini, P. T., Springer, L., Pascarella, E. T., & Nora, A. (1995). Influences affecting the development of students’ critical thinking skills. Research in Higher Education,

36, 23–39.
Terenzini, P. T., Springer, L., Yaeger, P. M., Pascarella, E. T., & Nora, A. (1996). First-generation college students: Characteristics, experiences, and cognitive de-

velopment. Research in Higher Education, 37, 1–22.
*Thompson, M. (2006). A correlational study on critical thinking as a predictor of success in the associate degree nursing program at Seminole Community College (Unpublished

doctoral dissertation). Minneapolis, MN: Capella University.
*Thompson, J. (2009). To question or not to question: The effects of two teaching approaches on students’ thinking dispositions, critical thinking skills, and course grades in a

critical thinking course (Unpublished doctoral dissertation). Minneapolis, MN: Capella University.
Walsh, C. M., & Hardy, R. C. (1999). Dispositional differences in critical thinking related to gender and academic major. Journal of Nursing Education, 38, 149–155.
Watson, G., & Glaser, E. M. (1964). Watson-Glaser Critical Thinking Appraisal manual: Forms. New York, NY: Harcourt, Brace, and World.
Wilson, D. G., & Wagner, E. E. (1981). The Watson-Glaser Critical Thinking Appraisal as a predictor of performance in a critical thinking course. Educational and

Psychological Measurement, 41(4), 1319–1322.
Xu, D., & Jaggars, S. S. (2011). Online and hybrid course enrollment and performance in Washington State community and technical colleges (Unpublished manuscript) (2011,

March)NY: Community College Research Center, Columbia University New York.
*Yost, H. D. (2003). Motivational orientation, self-regulated learning, and academic performance in associate degree nursing students (Unpublished doctoral dissertation).

Greensboro, NC: The University of North Carolina.
Zientek, L. R., Yetkiner, Z. E., Fong, C. J., & Griffin, M. (2013). Student success in developmental mathematics. Community College Journal of Research and Practice, 37,

990–1010. http://dx.doi.org/10.1080/10668926.2010.491993.

C.J. Fong et al. Thinking Skills and Creativity 26 (2017) 71–83
83

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0220

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0225

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0225

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0230

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0230

http://dx.doi.org/10.1023/A:1018722327398

http://dx.doi.org/10.1002/ets2.12009

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0245

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0250

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0250

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0255

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0260

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0260

http://dx.doi.org/10.1080/03075079.2011.586995

http://dx.doi.org/10.1080/03075079.2011.586995

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0270

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0270

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0275

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0280

http://dx.doi.org/10.1037/0033-2909.134.2.270

http://dx.doi.org/10.1080/87567555.2014.896775

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0295

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0295

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0300

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0300

http://dx.doi.org/10.1080/001318800440579

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0310

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0310

http://dx.doi.org/10.3102/0034654315582067

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0320

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0320

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0325

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0325

http://dx.doi.org/10.1037/a0026838

http://dx.doi.org/10.1037/0033-2909.130.2.261

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0340

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0345

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0345

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0350

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0355

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0355

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0360

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0360

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0365

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0370

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0370

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0375

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0375

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0380

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0380

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0385

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0385

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0390

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0395

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0400

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0400

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0405

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0405

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0410

http://refhub.elsevier.com/S1871-1871(16)30150-X/sbref0410

http://dx.doi.org/10.1080/10668926.2010.491993

• A meta-analysis on critical thinking and community college student achievement

Introduction
Critical thinking
Critical thinking and community college students
Factors related to critical thinking and college student achievement
Research questions
Method
Literature search and inclusion criteria
Information retrieval and coding
Effect size calculation and data integration
Results
Characteristics of included studies
Overall analysis of the relationship between community college and achievement
Moderator analyses
Discussion
Implications for research and practice
Limitations to generalizability
Supplementary data
References1