Why I stopped A/B testing cold email copy and started A/B testing audience segments instead. The reply rate difference was not close
segment_tester_hans · 2026-08-21 · 88 views
For 8 months I ran copy A/B tests on every campaign. Subject line variations. Opening sentence variations. Call to action variations. I got very good at running tests and very frustrated with the results. Most tests moved reply rate by 0.2 or 0.3 percentage points. Statistically real. Operationally meaningless.
Four months ago I stopped testing copy and started testing audience segments instead. The results were not close.
My ICP is HR leaders at B2B software companies. That is a broad ICP. Within it I ran three segment tests.
Segment A: HR Directors at companies with 50 to 200 employees. No additional signal filter. Reply rate: 2.4 percent.
Segment B: HR Directors at companies actively hiring for more than 5 roles simultaneously, as a proxy for growth pressure. Same copy. Reply rate: 4.1 percent.
Segment C: HR Directors who had been promoted into the Director role within the last 90 days. Same copy. Reply rate: 5.6 percent.
Same copy. Same PuzzleInbox infrastructure. Same Instantly sending setup. The only variable was which HR Directors I contacted and what signal indicated they were worth contacting right now.
The 90-day new-hire signal I built in Clay using LinkedIn join dates. The active hiring filter used Apollo's job posting data. Both cost under $50 per month at my contact volume.
My old copy tests delivered 0.3 percentage points of lift after weeks of work. Segment testing delivered 3.2 percentage points of lift in the first test. The math on where to invest time is not complicated once you see these numbers side by side.
Keep a baseline copy that works well enough. Then spend every extra hour figuring out which 15 percent of your total addressable market is in active buying mode right now. That is where the reply rates actually live.