FAQ

Why is my experiment not receiving impressions?

Work down this list:

  • The experiment isn't live. The SDK only assigns variants and sends impressions for a live experiment. Launch it from the detail page.
  • Wrong apiKey or experimentId. A bad key or id makes the config request fail, and the SDK falls back to control silently — by design, so your page never breaks. Double-check both against Settings and the experiment page.
  • init() isn't running on that page, or runs after the code that reads getVariant(). It must be called on the client, early.
  • You're refreshing the same browser. Impressions are deduplicated per session, so reloads don't add to the count. Use a fresh session (new private window, cleared cookies) to test.
  • An ad blocker or privacy extension is blocking the request to your API origin.
  • Your account is past its trial or unpaid. Events from inactive accounts are accepted by the API and then discarded — see below.

How much traffic do I need?

There is no minimum. Absolutely Butter has no detectable-effect floor and won't stop you launching a low-traffic test. Low traffic just means wider credible intervals and a longer — or unavailable — forward projection. The result stays honest the whole way; it just takes longer to sharpen. Bigger true differences need far fewer visitors to show up than small ones.

Why is the result still uncertain after two weeks?

Usually one of two things:

  • Not enough data yet. With small samples the two distributions still overlap, so P(variant > control) sits near 50%. Check the planning estimate for a rough time to a clearer result.
  • The real difference is small. If relative lift is only a percent or two, the probability will climb slowly no matter how long you wait. Decide whether a change that size is worth shipping at all.

A low number after two weeks is information, not a malfunction.

What happens when my trial expires?

The 30-day trial ends and, until you add a card, the account is inactive. While inactive:

  • You can't create, launch, or reactivate experiments.
  • Impression and conversion events are never rejected with an error — the API accepts them and silently discards them, so a forgotten SDK integration on a live site never starts throwing.
  • Your experiments and all their historical counts stay exactly as they were.

Start the $19/month subscription from Settings and data collection resumes immediately — no reconfiguration.

Can I run multiple experiments at the same time?

Not in v1. The SDK holds one experiment's assignment in memory and init() is meant to be called once per page. You can conclude one experiment and launch another, but concurrent experiments on the same page aren't supported yet. It's on the roadmap.

What does "may not produce a clear result within 90 days" mean?

That's the forward projection telling you it couldn't produce a useful planning estimate. Extrapolating from your traffic rate since launch and the conversion rates so far, the experiment isn't projected to reach a clear result inside a 90-day window — so rather than show a number you can't plan around, it says so plainly. The usual causes are low traffic or a very small difference between the arms. Options: send more traffic to the tested surface, accept a directional read, or conclude on the relative lift if it's already big enough to act on.

What does the early-data warning mean?

The warning appears while there are fewer than 100 total impressions or fewer than 30 impressions in either arm. It does not block a conclusion; it reminds you that the posterior distributions and credible intervals are still being shaped by limited exposure. Treat the probability and interval together, and wait for more data when the practical decision is sensitive to a small difference.

Is any of this personally identifiable?

No. The SDK sets two cookies — the assigned variant and a random session id — and neither is tied to a person. Events carry only the experiment id, the arm, the event type, a timestamp, and that random id. There is no user identifier, and the per-session rows are deleted when an experiment is archived.

Can I change the control or variant after launching?

The descriptions and the conversion goal lock when you launch, so the data can't drift against a moving definition. Name and hypothesis stay editable. If you need to change what you're testing, conclude the experiment and start a new one. See Experiment lifecycle.