Chapter 9: Comparing Two Groups

In Chapter 8. Hypothesis Testing you compared one average with a target, such as whether the average wait is longer than 4 minutes. Most business questions are comparisons between two groups instead. Did the customers who got a coupon spend more than those who didn't? Do promo days bring in more customers than ordinary days? Do trained baristas earn better ratings than untrained ones?

Dana Whitfield and Marcus Bell have asked you all three of these questions this week. They look alike: two groups, one number to compare. They turn out to need three different kinds of thinking. The coupon question comes from a planned experiment, so it can show cause and effect. The promo question involves the same stores measured twice, so the data come in pairs. The training question comes from records nobody designed for a test, so a difference might have more than one explanation. Learning to see those differences is as important as learning the tests.

9.1 Two Kinds of Two-Group Data

Before you pick a test, ask how the two groups were formed. There are two possibilities.

In the coupon test, Ridgeline took 1,000 loyalty customers and randomly split them. 517 got a mobile coupon and 483 didn't. Each customer is in exactly one group, and nothing links a particular coupon customer to a particular control customer. These are independent samples.

The promo question is different. Each of Ridgeline's 12 stores had some promo days and some ordinary days in 2025. You can compare each store's promo days with that same store's ordinary days. Downtown is compared with Downtown, Summit with Summit. The two measurements on each store form a pair. These are paired samples.

A quick way to tell them apart: could you draw a line connecting each value in one group to one value in the other, and would that line mean something? For the stores, yes: a line from Downtown's promo average to Downtown's ordinary-day average. For the coupon test, no: there's no reason to connect customer C0002 to any particular control customer. Other common paired designs are before-and-after measurements on the same people, and two products rated by the same customers.

The difference matters because pairing removes a big source of noise. Stores differ a lot from each other. Comparing each store with itself takes those differences out of the comparison, so a smaller effect becomes visible. You'll see exactly how much later in this chapter.

9.2 Question 1: Did the Mobile Coupon Work?

Last spring, Ridgeline ran a test with its loyalty program. A computer randomly chose about half of 1,000 loyalty customers to receive a mobile coupon. Over the next 30 days, Ridgeline recorded how many times each customer visited and how much each one spent. Dana's question is simple: did the coupon make customers spend more?

9.2.1 Prepare the data in Excel

Open the Ridgeline workbook and go to the CouponTest sheet. The CouponTest Table has one row per customer, with these columns:

• customer_id: an identifier.
• group: Coupon or Control.
• redeemed: Yes if the customer used the coupon (always No in the Control group).
• visits_next_30: visits in the 30 days after the coupon was sent.
• spend_next_30: dollars spent in those 30 days.

Add a new sheet (click + next to the sheet tabs) and name it CouponCalc. Build a small summary block, one column per group. Type Coupon in B1 and Control in C1, then enter these formulas in column B. Copy them to column C and change "Coupon" to "Control".

Row	Label (column A)	Formula in B (Coupon)	Coupon	Control
2	n	=COUNTIFS(CouponTest[group],B$1)	517	483
3	mean spend	=AVERAGEIFS(CouponTest[spend_next_30],CouponTest[group],B$1)	25.75	21.94
4	SD of spend	=STDEV.S(FILTER(CouponTest[spend_next_30],CouponTest[group]=B$1))	15.35	13.62

Because the formulas refer to B$1, you can copy them across without editing, if you prefer: the $ keeps the row fixed while the column changes from B to C.

FILTER returns just the spend values for one group, and STDEV.S takes their standard deviation. You'll use FILTER this way often in the rest of the book. It's the quickest way to pull one group out of a Table without sorting or copying.

The coupon group spent $25.75 on average over 30 days and the control group spent $21.94, a difference of $3.81. That's the descriptive answer. The inferential question is whether a gap of $3.81 is bigger than you'd expect from the luck of the random split alone.

9.2.2 How the two-sample t-test works

Imagine Ridgeline had sent the coupon to nobody, but still split the customers randomly into two groups and labeled them "Coupon" and "Control." The two groups' average spends would not be exactly equal. Some heavy spenders would land in one group by chance, and the averages would differ by some amount. So the question isn't "are the averages different?" They always are. The question is "is the difference bigger than random splitting typically produces?"

You answered a question with the same shape in Chapter 8. Hypothesis Testing. There you divided the distance between a sample mean and a target by the standard error of the mean. Here you divide the difference between two sample means by the standard error of the difference:

t = (difference in sample means) ÷ (standard error of the difference)

The top of the fraction is the signal: how far apart the groups are. The bottom is the noise: how much the difference would bounce around from one random split to another. A t of 1 means the gap is about the size chance produces all the time. A t of 4 means the gap is four times what chance typically produces, which chance rarely manages.

Where the standard error comes from. Each group's mean has its own standard error, s ÷ √n, as you saw in Chapter 6. Sampling and the Central Limit Theorem. When you subtract one mean from the other, both sources of uncertainty count, and their variances add:

SE = √( s₁²/n₁ + s₂²/n₂ )

Here s₁ and n₁ are the standard deviation and size of group 1, and s₂ and n₂ those of group 2.

In words: square each group's standard deviation, divide by that group's size, add the two results and take the square root. Bigger samples shrink the standard error. More spread inside the groups enlarges it.

The hypotheses. Write μ(Coupon) and μ(Control) for the average 30-day spend of all loyalty customers who would get, or not get, the coupon.

• H₀: μ(Coupon) = μ(Control). The coupon makes no difference to average spend.
• Hₐ: μ(Coupon) ≠ μ(Control). It makes some difference, in either direction.

This is a two-tailed test. Ridgeline would want to know if the coupon somehow lowered spending, too.

The worked example. Continue on the CouponCalc sheet, below the summary block:

Row	What	Formula	Result
6	difference in means	=B3-C3	3.81
7	standard error	=SQRT(B4^2/B2+C4^2/C2)	0.916
8	t	=B6/B7	4.16
9	degrees of freedom (Welch)	=(B4^2/B2+C4^2/C2)^2/((B4^2/B2)^2/(B2-1)+(C4^2/C2)^2/(C2-1))	995.4
10	two-tailed p-value	=T.DIST.2T(ABS(B8),B9)	< 0.001
11	margin of error (95%)	=T.INV.2T(0.05,B9)*B7	1.80

(Put the formulas in column B, next to their labels in column A.)

Read it from the top. The coupon group spent $3.81 more on average. Random splitting alone would typically move that difference by about $0.92. The observed gap is 4.16 standard errors from zero. If the coupon truly did nothing, a gap that large would happen less than once in a thousand random splits (the p-value is about 0.00003). That's strong evidence against H₀.

The degrees of freedom formula in row 9 looks intimidating. You don't need to memorize it. Its job is to tell the t distribution how much information the two standard deviations carry. With hundreds of customers in each group the result is close to n₁ + n₂ − 2, and the t distribution is almost identical to the normal curve. With small groups, the formula matters more.

The 95% confidence interval for the difference is 3.81 ± 1.80 dollars, or $2.01 to $5.61. It's built just like the intervals in Chapter 7. Confidence Intervals: estimate, plus or minus a t value times a standard error. Because the whole interval is above zero, it agrees with the test: the coupon raised spending, by somewhere between about $2 and $5.60 per customer over the month.

Check it with one function. Excel's T.TEST does all of this in one step and returns the p-value:

=T.TEST(FILTER(CouponTest[spend_next_30],CouponTest[group]="Coupon"),
        FILTER(CouponTest[spend_next_30],CouponTest[group]="Control"),2,3)

The third argument, 2, asks for a two-tailed test. The fourth argument chooses the type of test: 1 is paired, 2 assumes equal variances, and 3 is Welch's unequal-variance test. The result is about 0.0000346, the same p-value you built by hand.

9.2.3 Welch's test or the pooled test?

There's an older version of the two-sample t-test, the pooled (or equal-variance) t-test. It assumes the two populations have the same standard deviation, combines ("pools") the two sample SDs into one estimate, and uses n₁ + n₂ − 2 degrees of freedom. In Excel it's T.TEST(..., 2, 2).

For the coupon data the two tests nearly agree: Welch's t is 4.16, the pooled t is 4.14, and both p-values are below 0.001. They agree because the groups are large and similar in size, and their SDs ($15.35 and $13.62) aren't far apart.

The two tests can disagree when the group sizes are quite different and the SDs are too. If the smaller group has the larger SD, the pooled test underestimates the noise and gives p-values that are too small. That makes it too eager to declare a difference. Welch's test protects you in that situation and costs almost nothing when the SDs happen to be equal. That's why most statisticians now use Welch's test by default, and why this book does too.

9.2.4 The assumptions

Every test rests on assumptions. Here's what the two-sample t-test needs, and how the coupon data measures up.

1. Independence. Observations must be independent within each group and between the groups. Random assignment of separate customers handles this. It would fail if, say, members of the same household were split between groups and shared one coffee budget.
2. Roughly normal sample means. The t distribution is exact when the data are normal. Thanks to the central limit theorem, it works well for any reasonably shaped data once each group has about 30 or more observations. Spending is right-skewed (a few customers spend a lot), but with about 500 per group, that isn't a concern. With small groups and strongly skewed data, it would be.
3. No extreme outliers that dominate the means. Make a quick box plot of each group (Insert > Charts > Statistic Chart > Box and Whisker, as in Chapter 4. Describing Data). The largest 30-day spend is under $90, which is plausible for a loyal customer. Nothing looks like a data-entry error.

Welch's test doesn't need equal standard deviations, so that isn't on the list.

9.2.5 How big is the difference?

The p-value says the coupon's effect probably isn't zero. It doesn't say whether the effect is large. With a big enough sample, even a trivial difference produces a tiny p-value. This is the difference between statistical and practical significance you met in Chapter 8. Hypothesis Testing.

There are two good ways to describe size. The first is in the business's own units: $3.81 more per customer over 30 days, with a confidence interval of $2.01 to $5.61. Managers understand dollars, so lead with this. The second is an effect size that doesn't depend on units, which lets you compare effects across different studies and measures.

Cohen's d tells you how many standard deviations apart the two group means are. The pooled SD for the coupon data is about $14.54, so

d = 3.81 ÷ 14.54 ≈ 0.26

In Excel: =B6/SQRT(((B2-1)*B4^2+(C2-1)*C4^2)/(B2+C2-2)). A common rule of thumb calls d around 0.2 small, 0.5 medium and 0.8 large. The coupon's effect is small: the two groups' spending overlaps heavily, and plenty of control customers outspent coupon customers. It's still a real and useful effect. Across thousands of loyalty members, $3.81 each adds up. Whether it's worth the coupon's cost is a question you'll come back to in Chapter 14. Communicating Results: Capstone.

9.2.6 Experiments and cause

There's one more reason the coupon result is strong: how the groups were formed. A computer, not a manager or the customers themselves, decided who got the coupon. That makes the coupon test a randomized experiment.

Random assignment is what lets you say "the coupon caused the extra spending." Heavy spenders, light spenders, morning regulars and weekend visitors were all scattered across both groups by chance. The only systematic difference between the groups is the coupon. If the difference is bigger than chance produces, the coupon is the explanation left standing. Keep this in mind; the third question in this chapter doesn't have this protection.

9.2.7 Comparing two proportions

Spending is one outcome. Marcus also wants to know about visits. A simple way to measure loyalty is the share of customers who came in at least four times in the 30 days, about once a week. That's a Yes/No outcome for each customer, so you compare two proportions instead of two means.

The logic is the same as before: difference divided by its standard error. Under H₀ the two groups have the same true proportion, so you estimate that common proportion by pooling both groups together, and use it to compute the standard error.

In symbols, with p₁ and p₂ the two sample proportions and p the pooled proportion:

z = (p₁ − p₂) ÷ √( p(1 − p)(1/n₁ + 1/n₂) )

Build it on the CouponCalc sheet, a few rows below the t-test:

Row	What	Formula (column B)	Result
14	Coupon customers with 4+ visits	=COUNTIFS(CouponTest[group],"Coupon",CouponTest[visits_next_30],">=4")	232
15	Control customers with 4+ visits	=COUNTIFS(CouponTest[group],"Control",CouponTest[visits_next_30],">=4")	172
16	Coupon proportion	=B14/B2	0.449
17	Control proportion	=B15/C2	0.356
18	pooled proportion	=(B14+B15)/(B2+C2)	0.404
19	standard error	=SQRT(B18*(1-B18)*(1/B2+1/C2))	0.0311
20	z	=(B16-B17)/B19	2.98
21	two-tailed p-value	=2*(1-NORM.S.DIST(ABS(B20),TRUE))	0.003

44.9% of coupon customers visited four or more times, compared with 35.6% of control customers, a gap of about 9 percentage points. The z of 2.98 says that gap is about three standard errors from zero, and the p-value of 0.003 says chance alone would rarely produce it. A 95% confidence interval for the gap (computed with each group's own proportion rather than the pooled one) is about 9.3 ± 6.1 percentage points, roughly 3 to 15 points.

The test is reliable when each group has at least about 10 "successes" and 10 "failures." Here every count is in the hundreds.

Now try a second definition of returning: did the customer come back at all (one or more visits)? Change ">=4" to ">=1" in rows 14 and 15. 95.2% of coupon customers returned, compared with 94.0% of control customers. The z is only 0.82 and the p-value is 0.414, so there's no evidence of a difference. Almost every loyalty member comes back within a month anyway.

Put the two results together and you get a clearer business story than either gives alone. The coupon didn't bring back customers who would otherwise have stayed away. It made customers who were coming anyway come more often and spend more. If Dana's goal is to win back lapsed customers, this coupon isn't the tool. If the goal is to increase visits from active members, it works.

9.3 Question 2: Do Promo Days Bring In More Customers?

On about one day in seven, each Ridgeline store runs a promotion: a discounted drink, posted on the app and on a sign outside. Marcus wants to know whether promo days really bring in more customers, or whether they just give away margin on customers who would have come anyway.

9.3.1 A first attempt, and its problem

The DailySales Table has 4,380 rows, one for each store on each day of 2025. 649 of those store-days were promo days. A quick comparison takes all of them:

=T.TEST(FILTER(DailySales[transactions],DailySales[promo_day]="Yes"),
        FILTER(DailySales[transactions],DailySales[promo_day]="No"),2,3)

Promo store-days averaged 388.1 transactions and ordinary store-days 349.5, a difference of 38.7. Welch's t is 9.81, and the p-value is far below 0.001.

That looks convincing, but the design has two problems.

• The observations aren't independent. The 4,380 rows are 12 stores, each measured 365 times. Riverside is busy on most days, promo or not, and Aspen Falls is quiet on most days. Rows from the same store are related to each other. The two-sample t-test assumes every row is a separate, unrelated observation, so it overstates how much information the data contain.
• The groups mix stores in different proportions. Promo days aren't spread evenly: in 2025, the number of promo days per store ranged from 45 to 66. Stores differ enormously in size, from about 250 to about 470 transactions on an average day. So part of the 38.7 could come from which stores happened to run more promos, not from the promos themselves.

Both problems have the same fix. Compare each store with itself.

9.3.2 Prepare the data in Excel: a PivotTable of store averages

You'll build a table with one row per store and two columns: the store's average transactions on promo days, and its average on ordinary days.

1. Click any cell in the DailySales Table.
2. Choose Insert > PivotTable. In the dialog, the Table/Range box should read DailySales. Choose New Worksheet and click OK.
3. In the PivotTable Fields pane, drag store_id to Rows, promo_day to Columns and transactions to Values.
4. The values show "Sum of transactions." Click that field in the Values box, choose Value Field Settings…, select Average, and click OK.
5. Optional, for tidiness: open Value Field Settings… again, click Number Format, choose Number with 1 decimal place, and click OK twice.
6. Rename the sheet PromoPairs.

If the PivotTable starts in cell A3 (the default), the store IDs are in A5:A16, the ordinary-day averages (No) in B5:B16 and the promo-day averages (Yes) in C5:C16. Row 17 holds the grand totals.

Now add a column of differences. In E4 type diff, in E5 type =C5-B5, and fill it down to E16. (Type the cell references rather than clicking the PivotTable cells; clicking inserts a GETPIVOTDATA formula that doesn't fill down the way you expect.) Here are the results:

Store	Ordinary days	Promo days	Difference
Downtown	420.8	459.3	38.5
University	446.8	498.8	52.0
Riverside	467.5	511.1	43.6
Old Town	389.2	428.7	39.5
Maple Grove	367.3	396.2	28.9
Lakeside	381.8	401.2	19.4
Westfield	355.6	394.6	39.0
Cedar Park	302.2	343.3	41.1
Pine Ridge	257.1	288.6	31.5
Summit	281.6	306.8	25.3
Aspen Falls	251.1	270.4	19.4
Timberline	281.0	307.8	26.8

Look at it before you run any test. Every one of the 12 stores did better on promo days. The gains range from about 19 to 52 transactions a day.

9.3.3 How the paired t-test works

Here's the key idea: once you have the differences, the paired t-test is just the one-sample t-test from Chapter 8. Hypothesis Testing, applied to the difference column. The question becomes: is the average difference zero?

• H₀: the mean promo-day gain across stores is 0.
• Hₐ: the mean gain isn't 0.

The test statistic is the mean difference divided by its standard error:

t = (mean of the differences) ÷ (SD of the differences ÷ √n)

where n is the number of pairs. Add these formulas below the difference column on the PromoPairs sheet:

What	Formula	Result
number of pairs	=COUNT(E5:E16)	12
mean difference	=AVERAGE(E5:E16)	33.76
SD of differences	=STDEV.S(E5:E16)	10.13
standard error	=STDEV.S(E5:E16)/SQRT(12)	2.92
t	=AVERAGE(E5:E16)/(STDEV.S(E5:E16)/SQRT(12))	11.55
two-tailed p-value	=T.DIST.2T(11.55,11)	< 0.001
check with T.TEST	=T.TEST(C5:C16,B5:B16,2,1)	< 0.001

The degrees of freedom are n − 1 = 11: you have 12 differences, and you're estimating one mean from them. T.TEST with type 1 runs the paired test directly on the two columns and gives the same p-value as building it from the differences.

Why pairing is so much more powerful here. Compare the noise in the two designs. The daily transaction counts have an SD of about 89, mostly because the stores are so different in size. The SD of the 12 store-level differences is only about 10. Pairing removed the store-to-store variation, which had nothing to do with promotions, and left the variation that does. The paired design uses only 12 numbers, yet its t is larger than the all-days comparison's.

A 95% confidence interval for the average gain is 33.76 ± T.INV.2T(0.05,11) × 2.92, which is about 27 to 40 more transactions per store per promo day. That's a lift of about 9.6% over an ordinary day.

Assumptions for the paired test. The pairs must be independent of each other (one store's results don't drive another's), and the differences should be roughly normal with no wild outliers. With only 12 pairs, look at them: the differences run from 19.4 to 52.0, with no extreme value. That's fine.

9.3.4 More customers, smaller tickets

Promo days have a cost. The average ticket (dollars per transaction) was about $0.28 lower on promo days, because customers buy the discounted drink. Marcus's worry, "do we just give away margin?", is about exactly this trade-off.

You can test the average ticket the same way: add avg_ticket to the PivotTable values (as an average) and repeat the paired steps. Every store shows a lower ticket on promo days. Then do the same for revenue. Paired by store, revenue was about $138 higher per store on a promo day than on an ordinary day (t = 8.66, p < 0.001).

Put those numbers together and the business picture is clear. A promo day brings in roughly 10% more transactions at a ticket about 4% lower, and the extra customers more than make up for the discount. Promo days increase revenue. Whether they increase profit depends on the cost of the discounted drinks and the extra staffing, which the sales data don't include. That's a good question to send back to Marcus.

9.4 Question 3: Does Barista Training Improve Ratings?

Last year Ridgeline offered a barista training course, and 63 of its 104 employees completed it. Customers rate employees through the app on a 1-to-5 scale, and each employee's average rating is in the Staff Table's customer_rating column. Dana wants to know whether the training worked before paying for another round.

The comparison is the same as the coupon's: two independent groups (trained and untrained employees) and a numeric outcome. Build it the same way:

=AVERAGEIFS(Staff[customer_rating],Staff[completed_training],"Yes")
=T.TEST(FILTER(Staff[customer_rating],Staff[completed_training]="Yes"),
        FILTER(Staff[customer_rating],Staff[completed_training]="No"),2,3)

Trained employees average 4.44 and untrained employees 4.07, a difference of 0.37 points. Welch's t is 7.23 and the p-value is below 0.001. The 95% confidence interval for the difference is 0.27 to 0.48. Cohen's d is about 1.4, a large effect.

Should Dana conclude that training raises ratings by about 0.37 points? Not yet. The data came from a different kind of study.

9.4.1 Observational studies and confounding

Nobody randomly assigned employees to training. Employees signed up, or their managers sent them. That makes this an observational study.

When groups form themselves, they can differ in ways that also affect the outcome. Look at who completed the training:

• Trained employees have been with Ridgeline for 26.5 months on average. Untrained employees average 14.1 months.
• Half the baristas completed the training, compared with 75% of shift leads and 92% of managers.

Experienced employees probably get better ratings anyway: they're faster and they know the regulars. So part of the 0.37-point gap might be an experience gap, not a training effect. Tenure here is a confounding variable.

A simple way to reduce confounding is to compare like with like. Baristas are the biggest group, and in this data set trained and untrained baristas have almost the same tenure (about 11 months each). Restrict both groups to baristas by adding a condition to FILTER:

=T.TEST(FILTER(Staff[customer_rating],(Staff[completed_training]="Yes")*(Staff[role]="Barista")),
        FILTER(Staff[customer_rating],(Staff[completed_training]="No")*(Staff[role]="Barista")),2,3)

Multiplying two conditions inside FILTER means "both must be true." Among baristas only (34 trained, 34 untrained), trained baristas average 4.40 and untrained 4.04. The difference is 0.36, with p < 0.001. The gap barely shrank, which suggests tenure and role aren't the main explanation.

That's encouraging, but it still isn't proof. Something else might differ between baristas who chose training and those who didn't. The most motivated baristas may have signed up, and motivated baristas may earn better ratings with or without a course. Only random assignment rules out explanations you haven't thought of. An honest summary for Dana:

Trained employees earn customer ratings about 0.37 points higher (on a 5-point scale). Part of that could reflect experience, since trained staff have been here longer, but among baristas with similar tenure the gap is almost the same (0.36). The evidence is consistent with training helping. To be sure, the next round could be offered to a randomly chosen half of untrained baristas, with ratings compared after three months.

In Chapter 13. Multiple Regression you'll learn a more general way to handle confounders: estimating the training difference while holding tenure and role constant, all in one model.

9.5 Describe the Data to AI

You've worked all three questions by hand in Excel. Now you'll run the coupon and promo analyses in R for Excel, the way you would for a larger or messier analysis. The steps are the same as in Chapter 2. Working with AI and R for Excel.

1. Open R for Excel from the JOLT add-in.
2. In 1. Data, click Add a table… and choose CouponTest. Do the same for DailySales.
3. In 2. Ask your AI, type the data description below.
4. Click Copy prompt, paste it into your AI assistant, and send it.

The app's prompt already lists each data frame's columns, their types, and their ranges or levels. Your job is to add what the app can't know: what a row is, what the columns mean, how the data were collected, and what you want back.

CouponTest: each row is one Ridgeline loyalty customer in a randomized experiment.
A computer randomly assigned each customer to group "Coupon" (got a mobile coupon)
or "Control" (no coupon). spend_next_30 is dollars spent in the 30 days after the
coupon was sent; visits_next_30 is the number of visits in those 30 days.
redeemed is Yes if the customer used the coupon.

DailySales: each row is one store on one day of 2025 (12 stores x 365 days).
transactions is the number of transactions that day; promo_day is "Yes" on
promotion days. The same 12 stores appear on both promo and non-promo days.

Questions:
1. Did the coupon change average 30-day spend? Did it change the share of
   customers who made 4 or more visits?
2. Do promo days have more transactions? Because each store has both kinds of
   day, I want to compare each store with itself: average each store's
   transactions on promo days and on other days, then run a paired t-test
   on the 12 stores.

I'm in an intro business statistics course and have learned Welch's two-sample
t-test, the paired t-test and the two-proportion z-test.
Please use Welch's t-test for spend, a two-proportion test without continuity
correction for the 4+ visits share, and a paired t-test on the store averages.
Send each test result and the table of store averages to Excel with excel_out(),
and explain how to read each result.

Notice three things in this description. It says the coupon groups were randomly assigned, which tells the AI that causal language is appropriate. It explains why the promo comparison should be paired, so the AI doesn't fall back on the all-days comparison. And it names the tests you want, including "without continuity correction," a detail explained in the Interpret section below.

Read the reply before you run anything. Check that it uses the right data frames and columns, that the spend test is Welch's (not var.equal = TRUE), that the proportion test uses correct = FALSE, and that the promo test is paired on store averages, not run on all 4,380 rows. If any of these is wrong, say so in the same conversation and ask for a fix.

9.6 Read the Script

A script for this usually looks like the one below. Yours will differ in its details, but it should do the same things.

# 1. Coupon test: average 30-day spend, Coupon vs. Control (Welch's t-test)
spend_test <- t.test(spend_next_30 ~ group, data = CouponTest)
excel_out(spend_test, "Coupon test: spend next 30 days (Welch t-test)")

# 2. Coupon test: share of customers with 4 or more visits
four_plus  <- as.vector(tapply(CouponTest$visits_next_30 >= 4, CouponTest$group, sum))
group_size <- as.vector(table(CouponTest$group))
visit_test <- prop.test(four_plus, group_size, correct = FALSE)
excel_out(visit_test, "Coupon test: share with 4+ visits (two-proportion test)")

# 3. Promo days: average transactions for each store, promo days vs. other days
store_means <- tapply(DailySales$transactions,
                      list(DailySales$store_id, DailySales$promo_day), mean)
excel_out(store_means, "Average daily transactions by store and promo day")

# 4. Paired t-test: each store compared with itself
promo_test <- t.test(store_means[, "Yes"], store_means[, "No"], paired = TRUE)
excel_out(promo_test, "Promo vs. other days (paired t-test, 12 stores)")

Here's what each part does.

• Part 1 compares spend_next_30 between the two levels of group in the CouponTest data frame. The ~ reads as "by": spend by group. t.test runs Welch's test unless told otherwise (var.equal = TRUE would switch it to the pooled test). The result goes to an output card.
• Part 2 counts, for each group, how many customers had 4 or more visits, and how many customers each group has. Then prop.test compares the two proportions. correct = FALSE turns off a small adjustment R otherwise makes, so the result matches the z-test you built in Excel.
• Part 3 is the R version of your PivotTable. tapply takes the transactions, splits them by store and by promo day, and averages each piece. The result is a grid with 12 rows (stores) and two columns (No, Yes), which goes to Excel so you can compare it with your PivotTable.
• Part 4 runs the paired t-test on the two columns of that grid. paired = TRUE is what makes it a paired test. Without it, R would treat the 12 promo averages and 12 ordinary averages as independent groups.

The checks that matter: the right data frame in each test, the right outcome column, Welch for spend, correct = FALSE for the proportions, and paired = TRUE on store-level averages for the promo question.

9.7 Run It in R for Excel

Paste the AI's whole reply into 3. Run R code. The app keeps just the code. Click Run.

When the run finishes, the Results area shows the console output and one output card for each excel_out() call. On each card, click New sheet. (Or click Send everything to a new sheet to put them all on one sheet.)

If the run stops with an error, click Copy a prompt to fix it, paste that prompt back into the same AI conversation, and run the corrected code. The most common error here is a misspelled column or data frame name. R is case-sensitive, so couponTest and CouponTest are different names.

9.8 Interpret and Verify

Each test result arrives on the sheet as a two-column block headed by the test's name. The Welch block for the coupon looks like this, with values rounded:

Welch Two Sample t-test
Data	spend_next_30 by group
mean in group Control	21.94
mean in group Coupon	25.75
Hypothesized difference	0
t	−4.16
df	995.4
P-value	0.0000346
Alternative	two.sided
Lower 95%	−5.61
Upper 95%	−2.01

Notice the signs. R puts the groups in alphabetical order, so "Control" comes first and the difference is Control minus Coupon: negative. The t is −4.16 instead of 4.16, and the interval is −$5.61 to −$2.01 instead of $2.01 to $5.61. Same result, opposite direction. Always check which group was subtracted from which before you write "higher" or "lower."

Now verify each result against Excel:

• Welch t-test: the means, t, df and interval should match your CouponCalc sheet (ignoring the sign), and the P-value should match =T.TEST(…,2,3).
• Two-proportion test: R reports X-squared of about 8.90 instead of a z. That's not a different test: 8.90 is $2.98^2$, the square of your z. The P-value, 0.003, matches your Excel calculation. The two proportions are listed as "prop 1" (Control, 0.356) and "prop 2" (Coupon, 0.449).
• Store averages: the grid should match your PivotTable value for value.
• Paired t-test: the mean difference should be 33.76, the t 11.55 with 11 df, and the P-value should match =T.TEST(C5:C16,B5:B16,2,1).

You can also check with the Data Analysis ToolPak (on the Data tab, Data Analysis; if you don't see it, turn it on under File > Options > Add-ins). It offers three t-tests: t-Test: Two-Sample Assuming Unequal Variances (Welch), t-Test: Two-Sample Assuming Equal Variances (pooled) and t-Test: Paired Two Sample for Means. Each needs each group in its own column. For the coupon test, put =FILTER(CouponTest[spend_next_30],CouponTest[group]="Coupon") in one column and the Control version in the next, with a label in the row above each. For the paired test, point it at the two columns of your PivotTable. The ToolPak's p-value appears as "P(T<=t) two-tail." It may show the Welch degrees of freedom rounded to a whole number. The ToolPak isn't available in Excel for the web.

9.8.1 The conclusions, in business language

Statistics are only finished when someone can act on them. Here are the three answers, as you might write them to Dana and Marcus.

Coupon. In a randomized test of 1,000 loyalty members, customers who got the mobile coupon spent $3.81 more over the next 30 days than those who didn't (95% CI: $2.01 to $5.61), and 45% of them visited four or more times, compared with 36% of the control group. Because the groups were chosen at random, the coupon caused this increase. It did not bring back customers who weren't returning: about 95% of both groups came back at least once.

Promo days. All 12 stores had more transactions on promo days. On average a store gained about 34 transactions on a promo day (95% CI: about 27 to 40), a lift of roughly 10%. The average ticket was about $0.28 lower, but revenue per store was still about $138 higher on promo days. Promotions grow revenue; whether they grow profit depends on the cost of the discount, which we should look at next.

Training. Trained staff earn customer ratings about 0.37 points higher than untrained staff. Trained staff also tend to be more experienced, but among baristas with similar tenure the gap is nearly the same. This supports the training, though because staff chose whether to train, a randomized rollout would give a firmer answer.

9.9 Summary

To compare two groups, first ask how the groups were formed. Independent samples, like the coupon and control customers, call for a two-sample t-test. This book uses Welch's version, which divides the difference in means by √(s₁²/n₁ + s₂²/n₂) and doesn't assume equal standard deviations. Paired samples, like each store on promo and ordinary days, call for a paired t-test, which is a one-sample t-test on the differences. Pairing removes variation between units, such as the large differences between stores, and so detects smaller effects. When the outcome is Yes/No, the two-proportion z-test compares two percentages with the same signal-over-noise logic. In Excel, T.TEST runs all three t-tests (type 1 paired, 2 pooled, 3 Welch), and a few formulas build the z-test.

A p-value says whether a difference is likely to be real. An effect size, in dollars or as Cohen's d, says whether it's big enough to matter. How the data were collected decides what you may conclude. A randomized experiment supports cause-and-effect language. An observational study doesn't, because a confounding variable, such as tenure in the training data, may explain part of the difference. When you use AI and R for Excel, describe how the groups were formed, ask for the specific test you need, and check the output's heading, the direction of the difference and the p-value against Excel before you write the conclusion.