Holdouts

Holdouts exclude a percentage of users from experiments. You can exclude these users from specific experiments or even all of your experiments.

Holdout membership is not a stored list of users. It is computed every time a feature flag is evaluated: PostHog hashes the user's ID and places them in the holdout if the hash falls below the holdout percentage. The same user always gets the same result, and new users are bucketed the same way on their first visit, so the holdout always covers the configured percentage of your current traffic.

Holdout testing is useful for understanding long term effects of product changes, typically over a few months. For example, a new onboarding flow might show great initial conversion rates, but a holdout group could reveal whether those users remain active in 3 months from now.

How to create a holdout

To create a new holdout:

1. Navigate to Experiments and click on the Holdout groups tab.
2. Click New holdout.
3. Enter a descriptive name for your holdout, fill out the optional description to help you remember its purpose, and set the percentage of users you want to include in the holdout.

The holdout should be large enough to be statistically significant, but small enough as to not significantly reduce the pool of users available for experiments. We recommend between 1% and 10%.

Once created, you can assign a holdout to a draft experiment in the Manage distribution section of the Variants tab. The holdout becomes locked once any experiment using it is launched and you will be unable to modify it. This is to prevent data inconsistencies that could affect results.

How to analyze holdout impact

Holdout users are not included in experiment results, since they were never exposed to the experiment.

To measure the impact of the holdout, compare holdout users against other users in insights. Holdout users receive the feature flag variant holdout-{holdout_id} (see below), so you can filter or break down any metric by that flag value.

How holdouts affect feature flags

When a user is assigned to a holdout group, the feature flags associated with experiments using that holdout return a special variant value instead of the experiment's control or test variants.

Holdout variant format

Users in a holdout receive a variant value of holdout-{holdout_id}, where holdout_id is the internal database ID of the holdout group. For example, if your holdout has ID 727, users in that holdout receive the variant holdout-727.

You can find the holdout ID in the URL when viewing the holdout in PostHog (e.g., /experiments/holdouts/727).

Detecting holdout users in code

When evaluating a feature flag for an experiment with a holdout, you can check if a user is in the holdout by looking for the holdout- prefix in the variant:

const variant = posthog.getFeatureFlag('experiment-feature-flag')

if (typeof variant === 'string' && variant.startsWith('holdout-')) {
    // User is in a holdout group - show them the default experience
} else if (variant === 'test') {
    // User is in the test variant
} else {
    // User is in control or flag is disabled
}

flags = posthog.evaluate_flags("user-id")
variant = flags.get_flag("experiment-feature-flag")

if isinstance(variant, str) and variant.startswith("holdout-"):
    # User is in a holdout group - show them the default experience
elif variant == "test":
    # User is in the test variant
else:
    # User is in control or flag is disabled

Holdout evaluation priority

Holdout conditions are evaluated before regular flag conditions. If a user matches the holdout's rollout percentage, they receive the holdout variant regardless of any other targeting rules on the flag.