Home/Work/Tools & Automations/A/B Test Kitchen

Tools & Automations

A/B Test Kitchen

An agency dashboard that pulls A/B tests from Optimizely, Convert, VWO, Intelligems, Shoplift and Shopify Rollouts into one place, judges them all by the same rules, and shows which are ready to call.

ReactTypeScriptViteHonoPostgreSQLA/B Testing
  • Pulls tests from six platforms into one sheet and one board
  • Recomputes every result from raw visitors and conversions, so all tests are judged the same way
  • Flags a test as Ready after 14 days and 90% confidence or its full planned sample
  • Peeking switch blurs results until a test has earned it, with click-to-peek for a single test
  • Call a test a winner, learner or inconclusive and keep notes on what you learned

I run A/B tests for clients across a pile of different platforms, and every one of them reports results its own way. Figuring out which tests were actually ready to call meant opening six tabs and doing a lot of squinting.

So I built a kitchen. A/B Test Kitchen pulls every test from Optimizely, Convert, VWO, Intelligems, Shoplift and Shopify Rollouts into one place, recomputes the stats from the raw counts, and sorts each test into Draft, Running, Ready or Called. Same rules for every client and every platform: at least two weeks, plus 90% confidence or the full planned sample.

My favorite part is the peeking switch. Checking results too early is how you end up calling a false winner, so with peeking off, the numbers stay blurred until the test has earned them. You can still click to peek at a single test. It's a speed bump, not a wall.

Right now it runs on fictional clients and simulated traffic, but the connectors are built against each platform's real API, so plugging in keys is the next step, not a rewrite. Next up: an MCP server so Claude can answer "what's ready to call?" and, eventually, start and stop tests right from the dashboard.

6 reviews

What people say.

5.0
★★★★★
6 written reviews
5★
6
4★
0
3★
0
2★
0
1★
0
★★★★★
LinkedIn Review
A collaborator you keep coming back to

Intuitive, collaborative, and helpful — Katy knows how to solve your problem and has consistently provided numerous sources of knowledge and personal growth. Katy wants everyone around her to excel and goes out of her way to help people find their strengths. I would jump at the opportunity to work with her again.

★★★★★
LinkedIn Review
Positivity, wit, and creativity

Few people I've worked with in my professional career have brought as much positivity, wit, and creativity to the job. I've found her to be driven, adaptable, passionate, and able to focus on the details while not losing sight of the mission. Her professionalism and empathy make her a true joy to work with.

★★★★★
Co-Worker @ The Good
Curiosity and ambition on every project

...is an incredible collaborator, always bringing curiosity, ambition, and a can-do attitude to every project.