group project

broaden and fulfill based on the proposal to final. Attached is the part that I am responsible for.

Sample Size and Statistical PowerProvide a clear indication of the study’s sample size and the reasons for selecting this size.  If the sample size may be freely chosen by the organization, then a calculation of the minimal sample size to achieve the suggested effect size based on considerations of statistical power should be performed.  If there are reasons why the organization cannot select a sample size, then a calculation of the statistical power for the chosen sample size should be performed.  If the design of your study does not match a setting discussed in class, then you may utilize your simulations in Part 2 to estimate the statistical power for your selected sample size.For studies with more than one outcome that will be generated together, the sample size may be selected by designating a primary outcome corresponding to the most highly prioritized research question.

Research Design: Lecture 9
Outline
Part 1: Statistical Power
Part 2: Sample Size Calculations
Part 3: Simulations
Part 1: Statistical Power
How can we know if an experiment is well designed?
Are we likely to demonstrate an effect?
How can we quantify the power of an experiment?
False Positives, False Negatives
False Positives
Hypothesis tests are usually designed to limit the likelihood of a False Positive (Type I Error) to no more than the significance
level α when the null hypothesis H0 is true.
False positives are typically considered worse mistakes than false negatives.
False positives can correspond to poor outcomes like jailing an innocent person or providing cancer treatments
to a healthy patient.
Proving an Effect
When H0 is false, there is a real effect that takes place – e.g. a relationship between the independent and dependent variables.
Rejecting H0 would conclude that the effect is reasonably supported by the data.
When H0 is false, rejecting H0 would draw the correct conclusion.
The Effect Size
The effect size is the amount of change in the dependent variable that is associated with changes in the independent variable.
This effect size is stated in terms of the parameters used in the hypotheses. For instance, in the two-sample t test, our null
hypothesis is:
H0 : μT − μC = 0
for treatment group T and control group C .
In this case, the actual difference in means would be the effect size associated with the treatment.
In other settings, the effect size could be a regression coefficient, a percentage improvement, a degree of
dissimilarity, or any number of other measures.
Effects Sizes and Interpretations
The effect size tells you the impact of a course of action that changes the independent variable.
An effect size needs to be large enough to be considered meaningful in the domain.
Demonstrating a large effect size can state the case for taking a course of action.
The effect size is an important consideration that goes beyond statistical significance. In practice, the best studies will
achieve both.
Examples: Effect Size
An additional dollar of spending on online advertising is associated with 1.43 dollars of additional revenue.
A new medication will increase a patient’s 5-year probability of survival by 10%.
Adding a video series to an educational curriculum will lead to a 3% improvement in quiz scores.
Statistical Power
When H0 is false, the probability of a false negative is β .
The power of a stastical test is the probability of rejecting H0 when it is false.
The equation for the power of a test is:
Power = 1 − β
A Framework for Calculating the Power
Select an effect size.
Assuming the effect size is true, compute the probability of rejecting H0 . This is the power for the test.
The resulting power will apply to any effect size at least as extreme as the selected effect size.
Example: A Normal Z Test
H0 : μ = 0 versus HA : μ > 0.
We will assume μ = 1 is the real effect size.
Likewise, we will assume that the standard deviation of the data is σ = 2 .
With a sample size of n = 50, what is the statistical power of the test with significance level α = 0.05?
The Power Calculation
The test statistic is:
X̄ −0
Z = σ/
.
n
√
With a one-sided test and α = 0.05, we will reject H0 if Z > z1−α , where z0.95 = 1.645 , the 95th percentile of the standard
Normal Distribution corresponding to α = 0.05.
Filling in what we know, we then say that we’ll reject H0 if:
X̄ −0
X̄
Z = σ/
=
> z1−α
n
2/ 50
√
√
We can then rearrange this equality to reject H0 if:
X̄ > ( σn ) z1−α =
√
2
1.645 = 0.465
50
√
We’ll continue this calculation on the next slide.
Solving for the Power
We need to determine:
P(X̄ > 0.465) .
By the Central Limit Theorem, X̄ ≈ N (𝚖𝚎𝚊𝚗 = 1, 𝚜𝚍 =
2
.
√50 )
Note that the mean of X̄ is based on the assumed effect size, while the standard error of X̄ is based on the assumed standard
deviation and sample size.
Then we can calculate this quantity directly in R:
pnorm(q = 0.465, mean = 1, sd = 2/sqrt(50), lower.tail = FALSE)
[1] 0.9707219
This is the statistical power of the test.
What About Smaller Effect Sizes?
μ = 0.5:
pnorm(q = 0.465, mean = 0.5, sd = 2/sqrt(50), lower.tail = FALSE)
[1] 0.5492409
μ = 0.25 :
pnorm(q = 0.465, mean = 0.25, sd = 2/sqrt(50), lower.tail = FALSE)
[1] 0.2235855
μ = 0.1:
pnorm(q = 0.465, mean = 0.1, sd = 2/sqrt(50), lower.tail = FALSE)
[1] 0.09844378
The Power Depends on the Effect Size
It is easier to detect larger effects.
If the other parameters are held constant, smaller effect sizes will have less statistical power than larger effect sizes would have.
Proving a very small effect can become quite difficult.
The Power Also Depends on the Sample Size
The standard error of X̄ is
σ
.
√n
The difference between the true effect size and the assumed value in the null hypothesis (e.g. zero) can be written in terms of
the number of standard errors.
As n grows larger, this standard error shrinks.
As n grows larger, the true effect becomes a larger number of standard errors away from the null value.
Therefore, the likelihood of rejecting H0 when it is false (the power) increases as n increases.
Part 2: Sample Size Calculations
How can we establish an appropriate sample size for a study?
What kind of assumptions do we have to make?
How do these calculations change with more complicated statistical tests?
Calculating a Sample Size
We previously assumed a sample size in order to calculate the power of a test.
We could just as easily assume a level of power in order to calculate the sample size.
However, we still have to assume an effect size and standard deviation to make this calculation.
Calculating the Sample Size: Framework
Returning to our earlier equation, we had:
X̄ −0
Z = σ/
> z1−α
n
√
and therefore:
X−0
P(Z > z1−α) = P ( σ/
> z1−α).
n
√
We will continue this calculation on the next slide.
Then we can subtract zero in a creative way, with μA being the assumed effect size:
X−μA +μA
X−0
P ( σ/ n > z1−α) = P ( σ/ n
√
√
> z1−α)
X−μA
μA
X−μA
= P ( σ/ n + σ/ n > z1−α) = P ( σ/ n
√
√
√
μA
μA
μA
> z1−α − σ/ n ) = P (W > z1−α − σ/ n ) = 1 − P (W ≤ z1−α − σ/ n ) ,
√
√
√
where W is a standard Normal random variable with mean 0 and standard deviation 1. We’ll continue this calculation on the next slide.
Calculating the Sample Size: Power Assumptions
Now we can assume a value for the power as being 1 − β, which could be a value like 0.9. Then:
μA
1 − β = 1 − P (W ≤ z1−α − σ/ n ), so
√
μA
β = P (W ≤ z1−α − σ/ n )
√
The inverse Normal function Φ−1 converts a standard Normal probability P(W ≤ k) into a quantile value. For example,
Φ−1 (0.1) = −1.28 .
With this in mind, we can solve the above equation for n :
Φ−1 (β) = Φ−1 (P (W ≤ z1−α − σ/ A n )) = z1−α − σ/ A n .
μ
μ
√
√
Rearranging this to solve for n , we have:
2
z1−α −Φ−1 (β)
n = ( μ /σ )
A
Plugging in the Values
z1−α −Φ−1 (β) 2
1.645−(−1.28) 2
n = ( μ /σ ) = (
= 34.22
)
1/2
A
In practice, this means we would need 35 sampled records.
What this means: We would have a 90% power for a test to detect a true effect size of 1 if at least 35 subjects are sampled
when the standard deviation is 2.
Assumptions
We are assuming an effect size in a power or sample size calculation. In practice, this is the minimal effect size that would
constitute a meaningful effect.
We also have to make assumptions about additional parameters such as the standard deviation of the data. This can impact the
results, too.
Then, by assuming either a sample size or a level of power, we can solve for the other quantity.
R’s pwr Library
The pwr library has standardized the calculations of sample sizes for a number of statistical tests. This includes calculations for the
following tests:
Proportions (1 and 2 sample);
t Tests (1 and 2 sample);
Correlation Test;
Chi Squared Test;
ANOVA.
Example: Power for a t Test
Calculating the power in a two-sample t test:
library(pwr)
pwr.t2n.test(n1 = 50, n2 = 35, d = 0.5, sig.level = 0.05,
alternative = “greater”)
t test power calculation
n1 = 50
n2 = 35
d = 0.5
sig.level = 0.05
power = 0.7275115
alternative = greater
Example: Sample Size for a t Test
Here we are calculating the sample size for one of the two groups:
library(pwr)
pwr.t2n.test(n1 = 50, d = 0.5, sig.level = 0.05, power = 0.9,
alternative = “greater”)
t test power calculation
n1 = 50
n2 = 111.7928
d = 0.5
sig.level = 0.05
power = 0.9
alternative = greater
With this method, any single parameter can be solved for by specifying all of the others.
Sensitivity Analyses
What would happen if your assumptions about the effect size, power, or the other parameters are not correct?
It can be useful to run sensitivity analyses that explore a range of values for each parameter.
This can help you to determine how robust your study’s design might be – and what you may need to plan for if your
assumptions don’t fully account for the real setting.
Why Calculate a Sample Size?
We want to design our experiment to have a reasonable chance of demonstrating the intended effect if it exists.
We also don’t want to needlessly add subjects to the study, especially when doing so would be costly in time and
resources.
These calculations ask us to directly address the assumptions about the effect size and other parameters.
All of these considerations intertwine the technical and strategic elements of pursuing the research.
It’s About Planning
We do all of this analysis before conducting the research study.
This planning helps us to anticipate the limitations and design flaws of a study.
Better yet, we can make adjustments to the plans before we invest in the actual work.
Part 3: Simulations
What might the actual data look like once the study is conducted?
How can we create a simulation for the experiment?
How can simulation methods help us in planning studies?
Hypothetical Data
Before you run the experiment, you don’t really know what values of the data will result.
Having an example of what the data might look like can help you create your plans for analysis.
A simulation is a method of using random number generation and scenario planning to create hypothetical data.
Example Data
Does a medication reduce average blood pressure? A randomized controlled trial was conducted.
The data for this trial might look like this:
Show
10
entries
Search:
Group
1
Treatment
BP
160.2
2
Control
164.5
3
Treatment
148.5
4
Treatment
146
5
Control
166
6
Treatment
163.9
7
Control
156.2
8
Control
169.7
9
Control
171.4
10
Control
147.9
Showing 1 to 10 of 10 entries
Previous
1
Next
Plan for Simulation
We’ll set a sample size for the experiment.
We’ll randomly assign subjects to the treatment and control groups.
We’ll assume an effect size.
We’ll make other assumptions about the distribution of blood pressure (Normal) and the other parameters (means, standard
deviations, and others as needed).
Example Simulation
n