Chapter 8: Hypothesis Testing
Learning Objectives
- Explain the logic of a hypothesis test using the courtroom analogy.
- State null and alternative hypotheses for a business question, and choose between a one-tailed and a two-tailed test.
- Compute a one-sample t statistic and its p-value in Excel cells, and make a decision at a chosen significance level.
- Interpret a p-value correctly, and recognize what it does not mean.
- Describe Type I and Type II errors and power in terms of business costs.
- Distinguish statistical significance from practical significance, and explain why "not significant" doesn't prove the null hypothesis.
- Ask an AI to help state hypotheses, run a one-sample t-test in R for Excel, and verify the result with
T.DIST.RT.
8.1 Is the Wait Too Long?
Ridgeline's service goal is simple: customers should wait 4 minutes, on average, from joining the line to getting their order. Rosa, the Metro regional manager, doesn't think the chain is meeting it. "Our customers wait more than four minutes," she tells the owners. "We need more staff at the busy hours."
Marcus Bell is sympathetic, but extra staff are expensive. He wants evidence before he spends the money. He asks you: do the survey data show that Ridgeline's average wait is longer than 4 minutes?
You already know the survey's average wait: 4.45 minutes, from Chapter 4. Describing Data. That's more than 4. Is that the end of it? Not quite. In Chapter 7. Confidence Intervals, you saw that a sample mean is only an estimate. A different 600 customers would have given a different average. So the real question is whether 4.45 is far enough above 4 that it can't reasonably be explained by which customers happened to be surveyed. If the true average were exactly 4 minutes, could a sample of 600 easily come out at 4.45?
A hypothesis test is a formal way to answer that kind of question. This chapter teaches its logic, which every test in the rest of the book shares, using the simplest test there is: the one-sample t-test.
8.2 The Logic of a Test: A Courtroom
A hypothesis test works much like a criminal trial.
- The starting assumption. A defendant is presumed innocent. The court doesn't ask the defendant to prove innocence. It assumes it, and asks whether the evidence is strong enough to overturn that assumption.
- The burden of proof. The prosecution has to bring the evidence. If the evidence is weak or ambiguous, the presumption stands.
- The standard of proof. "Beyond a reasonable doubt" sets how strong the evidence must be before the jury convicts. It's deliberately demanding.
- Two possible verdicts. "Guilty" or "not guilty." Notice there's no verdict of "innocent." A "not guilty" verdict means the evidence wasn't strong enough. It doesn't mean the court has proven the defendant didn't do it.
- Two possible mistakes. The jury can convict an innocent person, or acquit a guilty one. A justice system can't eliminate both. A very high standard of proof protects the innocent but lets more guilty people go.
A hypothesis test has every one of these parts:
| Courtroom | Hypothesis test | Rosa's question |
|---|---|---|
| Presumed innocent | The null hypothesis | The average wait is 4 minutes (the goal is being met) |
| The prosecution's claim | The alternative hypothesis | The average wait is more than 4 minutes |
| The evidence | The data, summarized by a test statistic | 600 survey responses, mean 4.45 |
| Beyond a reasonable doubt | The significance level α | Usually 5% |
| Guilty | Reject the null hypothesis | Evidence that waits exceed 4 minutes |
| Not guilty | Fail to reject the null hypothesis | Not enough evidence; the goal might be met |
| Convicting an innocent person | Type I error | Concluding waits are too long when they aren't |
| Acquitting a guilty person | Type II error | Missing a real wait problem |
8.3 Null and Alternative Hypotheses
A hypothesis test starts with two competing statements about a population parameter, here μ, the average wait of all Ridgeline customers.
Null hypothesis (H₀) — The default claim that a test assumes is true unless the data give strong evidence against it. It usually says there is no effect, no difference, or that a parameter equals a specific value. It always contains an equals sign.
Alternative hypothesis (Hₐ) — The claim the test looks for evidence for. It's what you would conclude if the evidence against the null hypothesis is strong enough. It usually says there is an effect, a difference, or that a parameter is above or below a specific value.
For Rosa's claim:
- H₀: μ = 4 (the average wait for all customers is 4 minutes)
- Hₐ: μ > 4 (the average wait for all customers is more than 4 minutes)
Some textbooks write the null as μ ≤ 4, which says "the goal is being met." The calculation is the same either way, because the hardest case for Rosa's claim to beat is a true mean of exactly 4.
Four rules help you write hypotheses well.
- Hypotheses are about the population, never the sample. You already know the sample mean is 4.45. There's nothing to test about it. The question is about μ, which you don't know.
- The claim you want evidence for goes in the alternative. Rosa is making a claim and wants Ridgeline to act on it. Like a prosecutor, she carries the burden of proof. Putting her claim in Hₐ means Ridgeline acts only if the evidence is strong.
- The null hypothesis has the equals sign. The test needs a specific value to calculate what samples would look like if H₀ were true. "μ = 4" gives one. "μ > 4" doesn't.
- Write the hypotheses before you look at the data. Hypotheses should come from the business question. Choosing them after peeking at the data, to get a result you like, undermines the whole test. You'll see why in the section on p-hacking.
8.4 Prepare the Data in Excel
Open the Ridgeline workbook and go to the Survey Table. The column you need is wait_minutes: minutes from joining the line to getting the order, a continuous numeric variable.
Check it before you test it:
=COUNT(Survey[wait_minutes])should return 600, so there are no blanks or text values.=MIN(Survey[wait_minutes])and=MAX(Survey[wait_minutes])return 0.5 and 12.0. Both are believable waits.- A histogram (Insert > Insert Statistic Chart > Histogram) shows a single-peaked distribution with a slightly longer right tail. In Chapter 4. Describing Data, the 1.5 × IQR rule flagged 3 waits above 11.05 minutes as outliers. They're plausible long waits on busy mornings, not data errors, so they stay.
Later in the chapter you'll test each region separately, so add a region column now if you didn't in Chapter 7. Confidence Intervals. Click the first empty cell to the right of the Survey Table's header row, type region, press Enter, and in the first data cell of the new column enter:
=XLOOKUP([@store_id],Stores[store_id],Stores[region])
The Table fills it down all 600 rows.
Finally, insert a new worksheet for the test (right-click a sheet tab, Insert… > Worksheet) and name it WaitTest.
8.5 How the Test Works
8.5.1 The test statistic: how many standard errors away?
Start with the question the null hypothesis lets you ask: if the true average wait were exactly 4 minutes, what kind of sample means would surveys of 600 customers produce?
You know the answer from Chapter 6. Sampling and the Central Limit Theorem. Sample means would follow a sampling distribution that is close to normal, centered on 4, with a standard deviation equal to the standard error. The standard error of the mean is s / √n:
2.347 / √600 = 2.347 / 24.49 = 0.0958 minutes
So if H₀ were true, sample means would typically land within about 0.1 minutes of 4. Most would fall between about 3.8 and 4.2. Ridgeline's sample mean is 4.448. How unusual is that?
The natural way to measure "unusual" is to count standard errors. The sample mean is 0.448 minutes above 4, and one standard error is 0.0958 minutes, so the sample mean is
(4.448 − 4) / 0.0958 = 4.68 standard errors above the hypothesized value.
That number is the test statistic.
Test statistic — A number calculated from the sample that measures how far the data are from what the null hypothesis predicts, usually in standard errors. The farther it is from 0, the stronger the evidence against the null hypothesis.
In words: the test statistic is the distance between what you observed and what the null hypothesis says, measured in standard errors. In symbols, for one mean:
t = (x̄ − μ₀) / (s / √n)
where μ₀ ("mu-naught") is the value in the null hypothesis. It's called t because, for the same reason as in Chapter 7. Confidence Intervals, it follows a t distribution with n − 1 degrees of freedom: the standard error uses the sample standard deviation s, not the unknown σ.
Why count in standard errors instead of minutes? Because "0.45 minutes above 4" means different things in different samples. With 10 customers, the standard error would be about 0.74 minutes, and 0.45 minutes would be well within the normal bounce of sample means. With 600 customers, it's 4.7 standard errors. Standard errors put every sample on the same scale.
Your intuition from the empirical rule already tells you something: about 95% of values in a normal distribution lie within 2 standard deviations of the center, and almost all lie within 3. A sample mean 4.7 standard errors from the hypothesized value would be extraordinarily rare if H₀ were true.
8.5.2 The p-value
The p-value puts an exact number on "extraordinarily rare."
p-value — The probability, calculated assuming the null hypothesis is true, of getting a test statistic at least as extreme as the one actually observed. A small p-value means the data would be surprising if the null hypothesis were true.
Read that definition slowly. It has three parts, and each matters.
- "Assuming the null hypothesis is true." The p-value lives in an imaginary world where μ really is 4. It asks how the data look from inside that world.
- "At least as extreme." It's not the probability of getting exactly t = 4.677. It's the probability of getting 4.677 or anything even further in the direction of the alternative. For Rosa's one-sided alternative (μ > 4), that's the area under the t distribution to the right of 4.677.
- "As the one actually observed." It's about your data, from your one sample.
For Rosa's question, that right-tail area is about 0.0000018, or 1.8 in a million. Statisticians would report it as p < 0.001. In words: if Ridgeline's true average wait were 4 minutes, a survey of 600 customers would produce an average this far above 4 less than once in a thousand tries. Actually, less than twice in a million.
So you face a choice between two explanations. Either the average wait really is 4 minutes and Ridgeline got a roughly one-in-half-a-million fluke of a sample, or the average wait isn't 4 minutes. The second explanation is far more believable. That's the reasoning behind every hypothesis test: if the data would be very surprising under H₀, doubt H₀.
A t distribution curve with 599 degrees of freedom, centered at 0, with the horizontal axis labelled "t (standard errors from 4 minutes)" from −5 to 5. A vertical line at t = 4.68 is labelled "Ridgeline's sample." The area to its right is shaded (it's barely visible) and labelled "p-value ≈ 0.0000018." A second, lighter shading marks the area beyond t = 1.65 to show where a 5% one-tailed test would start rejecting.
What a p-value is NOT. These misreadings are common, even in business reports and the news.
- It is not the probability that the null hypothesis is true. p < 0.001 does not mean "there's less than a 0.1% chance the average wait is 4 minutes." The p-value is calculated assuming H₀ is true, so it can't also tell you how likely H₀ is.
- It is not the probability that the results are "due to chance." That phrase sounds like the definition, but it quietly flips the conditional into the first mistake.
- 1 − p is not the probability that the alternative is true. A p-value of 0.03 doesn't mean "97% chance Rosa is right."
- It doesn't measure how big or important the effect is. A tiny, unimportant difference can have a tiny p-value if the sample is big enough. You'll see this shortly.
- A large p-value doesn't show the null hypothesis is true. It means the data are consistent with H₀. They may also be consistent with many other values.
A safe sentence: "If the null hypothesis were true, results at least this extreme would happen with probability p."
8.5.3 The significance level and the decision
How small does a p-value need to be before you reject H₀? You decide that in advance, by choosing a significance level.
Significance level (α) — The threshold for rejecting the null hypothesis, chosen before looking at the data. If the p-value is less than α, you reject H₀. α is also the probability of rejecting H₀ when it is actually true. The most common choice is 0.05.
The decision rule:
- If p < α, reject H₀. The result is called statistically significant. The data give convincing evidence for the alternative.
- If p ≥ α, fail to reject H₀. The result is not statistically significant. The data don't give convincing evidence for the alternative.
The wording "fail to reject" is clumsy on purpose. Like the verdict "not guilty," it avoids claiming that H₀ has been proven. You never "accept" the null hypothesis on the strength of a test.
The significance level is the "reasonable doubt" standard, and 0.05 is a convention, not a law. A test that could trigger an expensive, hard-to-reverse decision might call for α = 0.01. An early, exploratory look might use 0.10. What matters is that you choose it before you see the p-value, so the data can't influence the standard they're judged by.
For Rosa's question, p ≈ 0.0000018 is far below 0.05, so you reject H₀. The survey gives strong evidence that Ridgeline's average wait is longer than 4 minutes.
8.5.4 The worked example, cell by cell
Now build the whole test on the WaitTest sheet, so you can see each piece. Labels go in column A and formulas in column B.
| Cell | Label (column A) | Formula (column B) | Result |
|---|---|---|---|
| B1 | Hypothesized mean μ₀ | 4 (typed in) |
4 |
| B2 | Sample mean | =AVERAGE(Survey[wait_minutes]) |
4.448 |
| B3 | Sample SD | =STDEV.S(Survey[wait_minutes]) |
2.347 |
| B4 | n | =COUNT(Survey[wait_minutes]) |
600 |
| B5 | Standard error | =B3/SQRT(B4) |
0.0958 |
| B6 | t statistic | =(B2-B1)/B5 |
4.677 |
| B7 | Degrees of freedom | =B4-1 |
599 |
| B8 | p-value, one-tailed (Hₐ: μ > 4) | =T.DIST.RT(B6,B7) |
1.8E-06 |
| B9 | p-value, two-tailed (Hₐ: μ ≠ 4) | =T.DIST.2T(ABS(B6),B7) |
3.6E-06 |
| B10 | Significance level α | 0.05 (typed in) |
0.05 |
| B11 | Decision (one-tailed) | =IF(B8<B10,"Reject H0","Fail to reject H0") |
Reject H0 |
Walk through it:
- B5 is the standard error: how much sample means of 600 bounce around.
- B6 is the test statistic. The sample mean is 4.68 standard errors above 4.
- B8 uses
T.DIST.RT, which returns the area under the t distribution to the right of a value: the "at least as extreme" area for a "greater than" alternative. Excel shows the result in scientific notation: 1.8E-06 means 1.8 × 10⁻⁶, or 0.0000018. - B9 uses
T.DIST.2T, the two-tailed version, which adds the matching area in the left tail. You'll see when to use it in the next section. - B11 applies the decision rule.
The WaitTest sheet with labels in A1:A11 and formulas in B1:B11 as in the table above. Cell B8 is selected so the formula bar shows =T.DIST.RT(B6,B7), and B8 displays 1.8E-06. B11 shows "Reject H0."
The critical-value view. Some textbooks and the ToolPak describe the same decision with a cutoff instead of a p-value.
=T.INV(1-B10,B7)returns 1.647: for a one-tailed test at α = 0.05, any t above 1.647 leads to rejection. Ridgeline's t of 4.677 is well past it. The p-value approach and the critical-value approach always reach the same decision. The p-value is more informative, because it says how far past the line you are.
T.DIST.2Tgives an error for a negative t. It expects a positive value, which is why B9 wraps the statistic inABS. When the sample mean is below the hypothesized value, as it will be for the Mountain region later,ABSkeeps the formula working.
8.5.5 One-tailed and two-tailed tests
Rosa's claim has a direction: waits are longer than 4 minutes. So the alternative is μ > 4, and only large values of t count as evidence for it. That's a one-tailed test.
Often you don't have a direction in mind. Marcus might ask, "Is our average wait different from 4 minutes, either way?" Then the alternative is μ ≠ 4, and a sample mean far below 4 would count as evidence too. The p-value has to include both tails: the chance of a t at least 4.677 above 0 or at least 4.677 below it. That's a two-tailed test, and its p-value is twice the one-tailed one: about 0.0000036.
One-tailed and two-tailed tests — A one-tailed test has a directional alternative (μ > μ₀ or μ < μ₀), and its p-value counts only the tail in that direction. A two-tailed test has the alternative μ ≠ μ₀, and its p-value counts both tails. For the same data, the two-tailed p-value is twice the one-tailed one when the result is in the predicted direction.
How do you choose? Use the business question, decided before you see the data.
- Use a one-tailed test only when a result in the other direction would lead to the same action as no result at all. If waits turned out shorter than 4 minutes, Ridgeline wouldn't add staff, just as it wouldn't if waits were exactly 4. Rosa's question passes that check.
- Use a two-tailed test otherwise. It's the default in most software, including R's
t.testand Excel'sT.TEST, and it's the safer choice whenever you're unsure.
Never pick the tail after seeing which way the data point. A one-tailed p-value is half the two-tailed one, so switching after the fact is an easy way to turn a "not significant" result into a "significant" one. You'll see a real example in the Suburban region below.
8.5.6 What the one-sample t-test assumes
You've just run a one-sample t-test.
One-sample t-test — A test of whether a population mean equals a specific value μ₀. Its test statistic is t = (x̄ − μ₀) / (s / √n), and its p-value comes from a t distribution with n − 1 degrees of freedom.
Its assumptions are the same as the t confidence interval's in Chapter 7. Confidence Intervals:
- A random, or at least representative, sample. No test can fix a biased sample. A survey run only at quiet times would understate waits, and the test would be precise about the wrong thing.
- Independent observations. With responses spread over a year and 12 stores, one customer's wait doesn't determine another's.
- A roughly normal sampling distribution of the mean. With 600 mildly skewed waits, the central limit theorem takes care of this. With a small sample, check a histogram first.
8.5.7 Type I and Type II errors
A test can reach the wrong conclusion in two ways, just like a jury.
Type I error — Rejecting the null hypothesis when it is actually true: a false alarm. Its probability is the significance level α.
Type II error — Failing to reject the null hypothesis when it is actually false: a missed effect.
| H₀ is true (average wait is 4) | H₀ is false (average wait is above 4) | |
|---|---|---|
| Reject H₀ | Type I error | Correct decision |
| Fail to reject H₀ | Correct decision | Type II error |
In statistics both are just "errors." In business they have prices, and the prices are rarely equal.
- A Type I error here means concluding waits are too long when they really average 4 minutes. Ridgeline would hire extra staff for busy hours, at real cost, to fix a problem it doesn't have.
- A Type II error means missing a real wait problem. Customers keep waiting too long, satisfaction slips, and some of them stop coming. In Chapter 12. Correlation and Simple Regression, you'll find that each extra minute of waiting goes with about a third of a point lower satisfaction, so this error has a cost too.
You can make Type I errors rarer by lowering α, say from 0.05 to 0.01. But a stricter standard makes it harder to detect a real effect, so Type II errors become more common. As with a court's standard of proof, you can't push both down at once with the same data. Thinking about which mistake costs more helps you choose α. If a false alarm would trigger an expensive, permanent change, use a smaller α. If missing a real problem is the bigger danger, a larger α can be reasonable.
The other way to reduce Type II errors is to collect more data. That's the idea of power.
Power — The probability that a test rejects the null hypothesis when it is false, that is, the chance of detecting an effect that really exists. Power equals 1 minus the probability of a Type II error. It increases with sample size, with the size of the true effect, and with α.
To get a feel for power, imagine the true average wait were 4.25 minutes: a real but small problem, 15 seconds over the goal. With a standard deviation around 2.35 minutes and a one-tailed test at α = 0.05:
- A survey of 600 customers would detect it (reject H₀) about 83% of the time.
- A survey of 100 customers would detect it only about 28% of the time.
With 100 customers, the standard error is about 0.23 minutes, so a 0.25-minute problem is barely bigger than the normal bounce of sample means, and the test would usually miss it. A study with low power is like a smoke detector with a weak battery: when it stays silent, that doesn't tell you much. This is one more reason why "not significant" isn't the same as "no effect."
8.5.8 Confidence intervals and two-sided tests
Tests and confidence intervals are two views of the same calculation. They use the same estimate, the same standard error and the same t distribution. So they agree:
A two-sided test at significance level α rejects H₀: μ = μ₀ exactly when μ₀ falls outside the (1 − α) confidence interval.
The 95% confidence interval for the average wait, built as in Chapter 7. Confidence Intervals, is 4.26 to 4.64 minutes. The value 4 lies outside it, so a two-sided test at α = 0.05 rejects μ = 4. The interval tells you more, though. It says not only that the average wait isn't 4, but how far from 4 it plausibly is: somewhere between about 16 and 38 seconds over the goal.
This connection works for any hypothesized value. Average spend has a 95% interval of $8.27 to $8.68 and a 99% interval of $8.20 to $8.75. A test of "average spend = $8.25" would reject at α = 0.05, because $8.25 is outside the 95% interval, but not at α = 0.01, because $8.25 is inside the 99% interval. You'll confirm that with the actual p-value below.
Report the interval alongside the test. A p-value says whether there is evidence of a difference. A confidence interval says how big the difference plausibly is. Managers almost always need the second.
8.6 Statistical vs. Practical Significance
You rejected H₀: Ridgeline's average wait is longer than 4 minutes. Should Marcus hire more staff?
Look at the size of the difference. The sample average is 4.45 minutes, about 0.45 minutes or 27 seconds over the goal. The confidence interval says the true excess is between about 16 and 38 seconds. The result is overwhelmingly statistically significant because 600 responses estimate the mean very precisely. But is half a minute, averaged across the whole chain, a problem worth paying to fix? That's a business judgment, not a statistical one.
Statistical vs. practical significance — A result is statistically significant if the data give convincing evidence that an effect is not zero (p < α). It is practically significant if the effect is large enough to matter for a decision. With a large sample, a tiny, unimportant effect can be statistically significant. With a small sample, an important effect can fail to reach significance.
Statistical significance answers "is there anything there?" Practical significance answers "is it big enough to care about?" A good analyst always answers both. For Rosa's question, the chain-wide answer is "yes, there's something there, and it's modest." The more useful finding may be hiding inside the average, which is where the next section looks.
8.7 Breaking It Down by Region
Rosa manages Metro stores, and her complaint was really about them. A chain-wide average can hide very different stories underneath. Test each region against the 4-minute goal, this time with two-tailed tests, because for regions the question is "is this region on target, or off it in either direction?"
On the WaitTest sheet, starting in D1, enter headers Region, n, Mean, SD, SE, t, p (two-tailed). In D2:D4, type Metro, Mountain and Suburban. In row 2:
| Column | Formula in row 2 |
|---|---|
| n (E) | =COUNTIFS(Survey[region],D2) |
| Mean (F) | =AVERAGEIFS(Survey[wait_minutes],Survey[region],D2) |
| SD (G) | =STDEV.S(FILTER(Survey[wait_minutes],Survey[region]=D2)) |
| SE (H) | =G2/SQRT(E2) |
| t (I) | =(F2-$B$1)/H2 |
| p (J) | =T.DIST.2T(ABS(I2),E2-1) |
Copy row 2 down to row 4. The $B$1 points at the hypothesized 4 minutes, so it stays fixed as you copy.
| Region | n | Mean wait | SD | t | p (two-tailed) | Decision at α = 0.05 |
|---|---|---|---|---|---|---|
| Metro | 217 | 5.81 | 2.31 | 11.50 | < 0.001 | Reject: above 4 |
| Mountain | 190 | 3.09 | 1.80 | −6.94 | < 0.001 | Reject: below 4 |
| Suburban | 193 | 4.26 | 2.00 | 1.78 | 0.076 | Fail to reject |
Now the picture is much clearer, and much more useful.
- Metro customers wait about 5.8 minutes on average, nearly two minutes over the goal. The evidence is overwhelming, and the difference is large enough to matter. Rosa is right about her region.
- Mountain customers wait about 3.1 minutes, nearly a minute under the goal. Its t is negative, and the two-tailed test catches it. (Had you used Rosa's one-tailed "greater than" test here, this region would never have shown up as different.)
- Suburban customers wait about 4.26 minutes. The p-value is 0.076, which is above 0.05, so the result is not statistically significant.
The Suburban result is where people most often go wrong. "Not significant" does not mean Suburban stores are meeting the goal, or that their average wait is 4 minutes. It means this sample doesn't give convincing evidence that the average differs from 4. The 95% confidence interval for the Suburban average runs from about 3.97 to 4.54 minutes. The data are consistent with an average right at 4, and equally consistent with an average half a minute over it. The honest summary is "we can't tell yet." If the Suburban stores matter for a staffing decision, the remedy is more data, not a confident conclusion either way.
Absence of evidence is not evidence of absence. Failing to reject H₀ never proves H₀. In the courtroom, "not guilty" isn't "innocent." When a test isn't significant, say "the data don't show a difference," report the confidence interval, and let it show how large a difference the data can't rule out.
8.8 Two More Tests: Satisfaction and Spend
The same test answers any question of the form "does the population mean equal this value?" Here are two more from Ridgeline.
Satisfaction against a goal. The owners want average satisfaction of at least 7.5 on the 1–10 scale. Is the chain beating that goal? The hypotheses are H₀: μ = 7.5 and Hₐ: μ > 7.5. The sample mean is 7.70 with an SD of 1.37, so the standard error is 1.370 / √600 = 0.056, and t = (7.702 − 7.5) / 0.056 = 3.61. The p-value is below 0.001, so the data give strong evidence that average satisfaction exceeds the 7.5 goal. Recall from Chapter 4. Describing Data that Metro averages only 7.0, so the chain-wide success, like the chain-wide wait, hides a weaker region.
Spend against a budget assumption. Dana's budget assumed customers spend $8.25 per visit. Is actual spend different, in either direction? Use a two-tailed test: H₀: μ = 8.25, Hₐ: μ ≠ 8.25. With mean 8.475 and standard error 0.1055, t = 2.13, and =T.DIST.2T(2.134,599) gives p = 0.033.
- At α = 0.05, reject H₀: average spend is different from $8.25 (higher).
- At α = 0.01, you would not reject. That matches the confidence intervals from the previous section: $8.25 is outside the 95% interval but inside the 99% interval.
- Practically, the difference is about 22 cents per visit. The budget assumption was a little conservative. Whether that matters depends on what Dana does with it, but it's good news, not a crisis.
p-hacking and multiple testing. Suppose Rosa, unconvinced, has you test each of Ridgeline's 12 stores against 4 minutes, then each store by day of week, then each store by order type, until something comes out significant. Something will. At α = 0.05, each test of a true null hypothesis has a 5% chance of a false alarm. Run 12 independent tests of true nulls, and the chance of at least one false alarm is 1 − 0.95¹², about 46%. Run 100, and false alarms are practically guaranteed.
The same problem comes in quieter forms: switching to a one-tailed test after seeing the data (the Suburban two-tailed p of 0.076 becomes a one-tailed 0.038), dropping "inconvenient" data points, or trying several measures and reporting only the one that worked. Each makes a p-value look smaller than the evidence deserves. This is called p-hacking, and it's a major reason published findings sometimes fail to hold up.
The defenses: decide your hypotheses, tail, α and data before you test; report every test you ran, not just the significant ones; and treat a surprising result from many tests as a lead to check with new data, not a conclusion. Chapter 10. Comparing Many Groups: ANOVA shows formal methods for comparing many groups at once.
8.9 Describe the Data to AI
8.9.1 First, talk through the hypotheses
Before asking an AI for any code, use it as a thinking partner on the hardest part of a test: turning a business question into hypotheses. Start a new conversation and describe the situation in plain words.
Needs: a reply that states H0: mu = 4 (or mu <= 4) and Ha: mu > 4 about the population mean wait, recommends a one-tailed one-sample t-test because the claim is directional, explains the choice of tail in terms of the business decision, and describes a Type I error (adding staff when waits average 4) and a Type II error (missing a real wait problem). Ideally it also mentions checking assumptions or suggests reporting a confidence interval. The text after this block compares the reply with the chapter.
Compare the reply with what you've learned. Does it put Rosa's claim in the alternative? Are the hypotheses about the population mean? Does the choice of tail come from the business question rather than the data? A two-tailed suggestion is a defensible, more cautious choice. Something like "H₀: the sample mean is 4.45" is a mistake worth catching: hypotheses are about the population. Talking through hypotheses before running anything keeps the business question, not the data, in charge of the test.
8.9.2 Then, ask for the analysis
Open R for Excel in JOLT. In 1. Data, click Add a table… and add Survey. Because you'll also test by region, add Stores too. In 2. Ask your AI, write your data description:
Each row of Survey is one customer's response to Ridgeline Coffee's 2025 survey
(600 responses, one per customer visit). wait_minutes is the minutes from joining
the line to getting the order. store_id links to the Stores data frame, where each
row is one store and region is Metro, Suburban or Mountain.
Question: Ridgeline's goal is an average wait of 4 minutes. Is the average wait of
ALL customers more than 4 minutes? Then, for each region separately, does the average
wait differ from 4 minutes in either direction?
I'm in an intro business statistics course and have learned confidence intervals and
hypothesis testing (null and alternative hypotheses, p-values, one- and two-tailed tests).
Please run a one-sample t-test of wait_minutes against 4 with the alternative "greater"
for all customers, and two-sided one-sample t-tests against 4 for each region, send
each result to Excel with excel_out(), and explain how to read the output.
Click Copy prompt, paste the prompt into your AI assistant, and send it.
Needs: a reply with a base R script that runs t.test(Survey$wait_minutes, mu = 4, alternative = 'greater'), merges Survey with Stores by store_id (or otherwise looks up region), runs a two-sided t.test against mu = 4 for each region, and sends every result to Excel with excel_out(). Its explanation should point to the t statistic, df, p-value and confidence limits. The text after this block checks the script against the example below, especially the mu value, the alternative for each test, and how region was attached.
8.10 Read the Script
A script for this usually looks like the one below.
# One-sample t-test: is the average wait longer than 4 minutes?
wait_test <- t.test(Survey$wait_minutes, mu = 4, alternative = "greater")
excel_out(wait_test, "Wait vs. 4 minutes (one-tailed)")
# Attach each response's region from the Stores table
sv <- merge(Survey[, c("store_id", "wait_minutes")], Stores, by = "store_id")
# Two-tailed test against 4 minutes for each region
for (r in c("Metro", "Suburban", "Mountain")) {
region_test <- t.test(sv$wait_minutes[sv$region == r], mu = 4)
excel_out(region_test, paste("Wait vs. 4 minutes:", r))
}
In plain language:
- The first block runs
t.teston thewait_minutescolumn ofSurvey.mu = 4sets the hypothesized mean: this is the "4" in H₀.alternative = "greater"makes it a one-tailed test of Hₐ: μ > 4. Thenexcel_outsends the result to an output card. - The second block matches each survey response to its store's row in
Storesbystore_id, so each response gets a region. The survey itself doesn't record region, which is why the script needsStores. - The third block repeats a test for each of the three regions. For each one, it keeps only that region's waits, tests them against 4 minutes, and sends the result to its own card. No
alternativeis given, so these tests are two-tailed, the default.
When you check the AI's version, the most important details are the ones that change the answer without causing an error:
- Is
muset to 4? If it's missing,t.testtests against 0, which answers a meaningless question ("is the average wait zero?") and gives a huge t and a tiny p-value that look impressive. - Is the alternative right for each test?
"greater"for Rosa's chain-wide claim, two-sided for the regions. An AI might use"greater"everywhere, which would hide Mountain's short waits, or two-sided everywhere, which would double the chain-wide p-value. The second is a defensible choice; the first isn't. - Does it use
wait_minutes? Notsatisfactionorspend.
8.11 Run It in R for Excel
- Paste the AI's whole reply into 3. Run R code. The app keeps just the code.
- Click Run.
- Four output cards appear: one for the chain-wide test and one for each region. Click New sheet on each. (Or click Send everything to a new sheet to put them all on one sheet.)
If you see an error, click Copy a prompt to fix it, paste it back into your AI, and run the corrected code.
The R for Excel task pane after running the script, with four output cards titled "Wait vs. 4 minutes (one-tailed)," "Wait vs. 4 minutes: Metro," "Wait vs. 4 minutes: Suburban" and "Wait vs. 4 minutes: Mountain." Behind the pane, a new worksheet shows the first card's two-column block, headed "One Sample t-test," with rows Data, mean of x, Hypothesized mean, t, df, P-value, Alternative, Lower 95%, Upper 95%.
8.12 Interpret and Verify
8.12.1 Reading the output block
The chain-wide test lands on its sheet as a two-column block headed One Sample t-test. Here's what each row means:
| Row | Value | What it means |
|---|---|---|
| Data | Survey$wait_minutes | The column that was tested. Check it's the right one. |
| mean of x | 4.448 | The sample mean, the point estimate of μ. |
| Hypothesized mean | 4 | μ₀, the value in H₀. Check it's 4, not 0. |
| t | 4.677 | The test statistic: 4.68 standard errors above 4. |
| df | 599 | Degrees of freedom, n − 1. |
| P-value | about 1.8e-06 | The one-tailed p-value. Report it as p < 0.001. |
| Alternative | greater | The direction of Hₐ. Check it matches your question. |
| Lower 95% | 4.290 | The lower end of a one-sided 95% confidence bound. |
| Upper 95% | Inf | Infinity: a one-sided interval has no upper end. |
The last two rows look odd at first. A one-tailed test comes with a one-sided confidence interval: "we are 95% confident the average wait is at least 4.29 minutes." It has no upper limit, which R writes as Inf. If you want the usual two-sided interval (4.26 to 4.64), run the test without alternative = "greater". The region sheets, which are two-sided, show ordinary lower and upper limits.
8.12.2 Verify in Excel
Check the R result against your WaitTest sheet:
- t: the block's 4.677 matches B6.
- P-value: the block's p-value matches B8,
=T.DIST.RT(B6,B7). You can also check it from the output sheet itself: pointT.DIST.RTat the cells holding R's t and df, and it returns the same 1.8E-06. - Regions: each region sheet's t and P-value should match your region table: Metro t = 11.50, Mountain t = −6.94, Suburban t = 1.78 with p = 0.076.
Why not
T.TESTor the ToolPak? Excel'sT.TESTfunction and the Data Analysis ToolPak's t-tests are built to compare two groups, which is the topic of Chapter 9. Comparing Two Groups. Neither has a one-sample option. For a one-sample test, the cell-by-cell calculation withT.DIST.RTorT.DIST.2Tis Excel's check.
If R and Excel ever disagree, look for a different column, hypothesized value, alternative or filter. Here they match: two tools, one answer.
8.12.3 The conclusion, in business language
Marcus doesn't need t statistics. He needs a recommendation he can act on:
Chain-wide. Surveyed customers waited 4.45 minutes on average. That's strong evidence that the true average wait is above the 4-minute goal (p < 0.001), but the gap is modest: we're 95% confident it's between about 16 and 38 seconds.
By region. The chain average hides a large difference. Metro customers wait about 5.8 minutes, close to two minutes over the goal (p < 0.001). Mountain customers wait about 3.1 minutes, under the goal. Suburban customers wait about 4.3 minutes, and the survey can't tell whether that's above the goal or not (p = 0.076).
Recommendation. Rosa's concern is justified for Metro, and that's where extra staffing at busy hours should be considered first. There's no evidence for a chain-wide staffing change. For Suburban stores, collect more wait data before deciding.
Notice how the conclusion separates "significant" from "big enough to act on." That's the habit to carry into every test that follows.
8.13 Summary
A hypothesis test asks whether sample data give convincing evidence against a default claim about a population. Like a trial, it starts by assuming the null hypothesis, which states a specific value such as μ = 4, and looks for evidence for the alternative hypothesis, which holds the claim you want to support. The test statistic measures how many standard errors the sample result is from the null value. The p-value is the probability, assuming the null hypothesis is true, of a result at least that extreme. It isn't the probability that the null hypothesis is true. If the p-value is below the significance level α, chosen in advance, you reject the null hypothesis. Otherwise you fail to reject it, which never proves it true. A one-tailed test fits a directional question decided before seeing the data, and a two-tailed test fits everything else. A two-sided test and a confidence interval always agree.
Tests can err in two ways: a Type I error is a false alarm, with probability α, and a Type II error is a missed effect. Power, the chance of detecting a real effect, grows with sample size. In Ridgeline's survey, the one-sample t-test gives t = 4.68 and p < 0.001: the average wait exceeds 4 minutes, but only by about half a minute chain-wide. Broken down by region, Metro waits are well over the goal, Mountain waits are under it, and the Suburban result (p = 0.076) is inconclusive. Statistical significance says an effect is there. Practical significance, judged from the size of the effect and the business context, says whether it matters. Decide your hypotheses before testing, report every test you run, verify R's output with T.DIST.RT or T.DIST.2T, and end with a sentence a manager can act on.