Software

cmrdesign

Use pilot data to decide how to allocate the next wave of an experiment. cmrdesign implements the methods in When and How to Pilot, from a treatment–control split to designs with several arms, strata, or outcomes. It also includes tools for planning pilot sizes.

Documentation GitHub Paper

Citation

@misc{yamin2026pilot,
  title = {When and How to Pilot: Design Rules for Two-Wave Experiments},
  author = {Yamin, Juan C.},
  year = {2026},
  doi = {10.48550/arXiv.2607.16982},
  url = {https://arxiv.org/abs/2607.16982}
}

@manual{cmrdesign2026,
  title = {cmrdesign: Conditional Minimax Regret Design Rules},
  author = {Yamin, Juan C.},
  year = {2026},
  note = {R and Python software, version 0.1.0},
  url = {https://juancyamin.github.io/cmrdesign/}
}

Worked examples

Install once install.packages("cmrdesign") Two-arm quick start ↗

Choose an example to see its code and results.

Two arms

One treatment, one control

Start with the standard setting: a pilot with 200 observations in each arm and an outcome known to lie between 0 and 1. Use the pilot to choose the treatment–control split for 1,000 main-wave participants.

Example data · 400 pilot observations

The illustrative treatment outcomes are more variable than the control outcomes. The 0–1 bounds are known in advance.

z <- rep(c(0.05, 0.15, 0.35, 0.65, 0.95), 40)
d <- rep(c(1, 0), each = 200)
y <- c(z, 0.4 + 0.2 * z)

import numpy as np

z = np.tile([0.05, 0.15, 0.35, 0.65, 0.95], 40)
d = np.repeat([1, 0], 200)
y = np.r_[z, 0.4 + 0.2 * z]

library(cmrdesign)

fit <- cmr_two_arm(
  y, d, alpha = 0.05, method = "bounded"
)

allocation <- realize_allocation(
  fit, n_main = 1000
)
allocation$counts

import cmrdesign as cmr

fit = cmr.cmr_two_arm(
    y, d, alpha=0.05, method="bounded"
)

allocation = cmr.realize_allocation(
    fit, n_main=1000
)
print(allocation.counts)

Main-wave allocation

Number of participants · 1,000 total

Assignment	Participants
Treatment	692
Control	308

The rule allocates more observations to the more variable arm while accounting for uncertainty in the pilot variance estimates.

For an outcome on another bounded scale, supply its known bounds when normalizing.

Two-arm documentation ↗

Binary outcomes

When the outcome is zero or one

The pilot records a binary outcome, such as take-up. Exact Bernoulli confidence bounds use the outcome’s structure to inform the main-wave allocation.

Example data · 400 pilot observations

100 successes out of 200 in treatment, and 20 out of 200 in control.

d <- rep(c(1, 0), each = 200)
y <- c(rep(1, 100), rep(0, 100),
       rep(1, 20), rep(0, 180))

import numpy as np

d = np.repeat([1, 0], 200)
y = np.r_[np.ones(100), np.zeros(100),
          np.ones(20), np.zeros(180)]

library(cmrdesign)

fit <- cmr_two_arm(
  y, d, alpha = 0.05, method = "bernoulli"
)

allocation <- realize_allocation(
  fit, n_main = 1000
)
allocation$counts

import cmrdesign as cmr

fit = cmr.cmr_two_arm(
    y, d, alpha=0.05, method="bernoulli"
)

allocation = cmr.realize_allocation(
    fit, n_main=1000
)
print(allocation.counts)

Main-wave allocation

Number of participants · 1,000 total

Assignment	Participants
Treatment	625
Control	375

The confidence set uses exact bounds for the binary success probabilities. With outcomes coded 0/1, method = "auto" selects this method too.

Binary-outcome guide ↗

Unbounded outcomes

Work with outcomes on their original scale

For a raw outcome without known support bounds, use a kurtosis assumption to construct the variance confidence set. Here we assume a kurtosis bound of 3 in each arm and plan a main wave of 10,000 observations.

Example data · 2,000 pilot observations

Fixed normal-quantile values create reproducible illustrative outcomes, with different scales across arms.

u <- (((0:999) * 137) %% 1000 + 0.5) / 1000
z <- qnorm(u)
d <- rep(c(1, 0), each = 1000)
y <- c(50 + 13 * z, 50 + 8 * rev(z))

import numpy as np
from statistics import NormalDist

u = ((np.arange(1000) * 137) % 1000 + 0.5) / 1000
z = np.array([NormalDist().inv_cdf(p) for p in u])
d = np.repeat([1, 0], 1000)
y = np.r_[50 + 13 * z, 50 + 8 * z[::-1]]

library(cmrdesign)

fit <- cmr_unbounded(
  y, d, psi = 3, alpha = 0.05
)

allocation <- realize_allocation(
  fit, n_main = 10000
)
allocation$counts

import cmrdesign as cmr

fit = cmr.cmr_unbounded(
    y, d, psi=3, alpha=0.05
)

allocation = cmr.realize_allocation(
    fit, n_main=10000
)
print(allocation.counts)

Main-wave allocation

Number of participants · 10,000 total

Assignment	Participants
Treatment	6,241
Control	3,759

The median-of-means construction replaces known support bounds with a supplied kurtosis bound, psi. That bound needs an independent justification.

Small pilots can return balanced assignment without a finite regret certificate.

Unbounded-outcome documentation ↗

Multiple outcomes

Choose one allocation for several outcomes

An experiment has two co-primary binary outcomes. Give them weights of 0.6 and 0.4, then choose a treatment–control split that accounts for both.

Example data · 400 units, two outcomes

Rows are pilot participants and columns are outcomes; each participant has one treatment indicator.

d <- rep(c(1, 0), each = 200)
y1 <- c(rep(1, 100), rep(0, 100),
        rep(1, 20), rep(0, 180))
y2 <- c(rep(1, 40), rep(0, 160),
        rep(1, 80), rep(0, 120))
y <- cbind(y1, y2)

import numpy as np

d = np.repeat([1, 0], 200)
y1 = np.r_[np.ones(100), np.zeros(100),
           np.ones(20), np.zeros(180)]
y2 = np.r_[np.ones(40), np.zeros(160),
           np.ones(80), np.zeros(120)]
y = np.column_stack([y1, y2])

library(cmrdesign)

fit <- cmr_multiple_outcomes(
  y, d, weights = c(0.6, 0.4),
  estimand = "coprimary",
  alpha = 0.05, method = "auto"
)

allocation <- realize_allocation(
  fit, n_main = 1000
)
allocation$counts

import cmrdesign as cmr

fit = cmr.cmr_multiple_outcomes(
    y, d, weights=[0.6, 0.4],
    estimand="coprimary",
    alpha=0.05, method="auto"
)

allocation = cmr.realize_allocation(
    fit, n_main=1000
)
print(allocation.counts)

Main-wave allocation

Number of participants · 1,000 total

Assignment	Participants
Treatment	545
Control	455

The co-primary setting weights the variances of separate outcome estimators. Choose the outcome weights to reflect the study’s priorities.

For a single weighted outcome index, use the package’s separate estimand = "index" option.

Multiple-outcome documentation ↗

Proxy outcomes

Plan before the primary outcome is observed

Only a short-run proxy is available in the pilot. Assume the primary and proxy outcomes lie in [0, 1], and their standard deviations differ by at most 0.05 in each arm.

Example data · 400 proxy observations

The binary proxy has 100 successes in treatment and 20 in control, each out of 200 observations. The bridge bound is supplied separately.

d <- rep(c(1, 0), each = 200)
proxy_y <- c(rep(1, 100), rep(0, 100),
             rep(1, 20), rep(0, 180))

import numpy as np

d = np.repeat([1, 0], 200)
proxy_y = np.r_[np.ones(100), np.zeros(100),
                np.ones(20), np.zeros(180)]

library(cmrdesign)

fit <- cmr_proxy(
  proxy_y, d, zeta = 0.05,
  alpha = 0.05, method = "auto"
)

allocation <- realize_allocation(
  fit, n_main = 1000
)
allocation$counts

import cmrdesign as cmr

fit = cmr.cmr_proxy(
    proxy_y, d, zeta=0.05,
    alpha=0.05, method="auto"
)

allocation = cmr.realize_allocation(
    fit, n_main=1000
)
print(allocation.counts)

Main-wave allocation

Number of participants · 1,000 total

Assignment	Participants
Treatment	613
Control	387

The bridge radius, zeta, widens the uncertainty set before choosing an allocation for the primary outcome. Its value needs substantive or external evidence.

Proxy-outcome documentation ↗

Multiple arms

Several treatments, one control

A pilot compares two treatments with a shared control. Use its binary outcomes to allocate 1,000 main-wave participants across the three arms.

Example data · 600 pilot observations

Illustrative 0/1 data: 20, 60, and 100 successes out of 200 observations in control, treatment 1, and treatment 2.

arm <- rep(0:2, each = 200)
y <- unlist(lapply(c(20, 60, 100), function(k) {
  c(rep(1, k), rep(0, 200 - k))
}))

import numpy as np

arm = np.repeat([0, 1, 2], 200)
y = np.concatenate([
    np.r_[np.ones(k), np.zeros(200 - k)]
    for k in [20, 60, 100]
])

library(cmrdesign)

fit <- cmr_multiarm(
  y, arm, control_arm = 0,
  alpha = 0.05, method = "auto"
)

allocation <- realize_allocation(
  fit, n_main = 1000
)
allocation$counts

import cmrdesign as cmr

fit = cmr.cmr_multiarm(
    y, arm, control_arm=0,
    alpha=0.05, method="auto"
)

allocation = cmr.realize_allocation(
    fit, n_main=1000
)
print(allocation.counts)

Main-wave allocation

Number of participants · 1,000 total

Control308

Treatment 1331

Treatment 2361

The rule accounts for uncertainty in each arm’s variance. realize_allocation() turns the recommended shares into whole participant counts.

Multi-arm documentation ↗

Stratified experiments

Allocate across groups as well as treatments

The target population is 60% urban and 40% rural. With pilot outcomes from both groups, choose how to allocate 1,000 main-wave observations across the four treatment-by-stratum cells.

Example data · 480 pilot observations

Each cell has 120 binary observations. Success counts are 6 and 30 in urban treatment/control, and 54 and 12 in rural treatment/control.

strata <- rep(c("Urban", "Rural"), each = 240)
d <- rep(rep(c(1, 0), each = 120), 2)
y <- unlist(lapply(c(6, 30, 54, 12), function(k) {
  c(rep(1, k), rep(0, 120 - k))
}))

import numpy as np

strata = np.repeat(["Urban", "Rural"], 240)
d = np.tile(np.repeat([1, 0], 120), 2)
y = np.concatenate([
    np.r_[np.ones(k), np.zeros(120 - k)]
    for k in [6, 30, 54, 12]
])

library(cmrdesign)

fit <- cmr_stratified(
  y, d, strata,
  strata_share = c(Urban = 0.6, Rural = 0.4),
  alpha = 0.05, method = "auto"
)

allocation <- realize_allocation(
  fit, n_main = 1000
)
allocation$counts

import cmrdesign as cmr

fit = cmr.cmr_stratified(
    y, d, strata,
    strata_share={"Urban": 0.6, "Rural": 0.4},
    alpha=0.05, method="auto"
)

allocation = cmr.realize_allocation(
    fit, n_main=1000
)
print(allocation.counts)

Main-wave allocation

Number of participants · 1,000 total

Stratum	Treatment	Control	Total
Urban	193	357	550
Rural	274	176	450
Total	467	533	1,000

Population shares define the target. The recommended sample shares can differ: here, 55% of observations go to the urban stratum and 45% to the rural stratum.

Stratified-design documentation ↗

Pilot planning

How much of the sample should go to a pilot?

Before collecting data, suppose you have a budget of 3,000 observations. Prior studies suggest treatment and control standard deviations of 0.18 and 0.28 for an outcome bounded between 0 and 1.

library(cmrdesign)

plan <- cmr_plan(
  n = 3000, sigma1 = 0.18, sigma0 = 0.28,
  alpha = 0.05, method = "bounded",
  accounting = "design_only",
  desired_pilot = 120
)

plan$recommendation
plan$desired_status

import cmrdesign as cmr

plan = cmr.cmr_plan(
    n=3000, sigma1=0.18, sigma0=0.28,
    alpha=0.05, method="bounded",
    accounting="design_only",
    desired_pilot=120
)

print(plan["recommendation"])
print(plan["desired_status"])

Candidate pilot sizes

72–134

observations, in even increments

A proposed pilot of 120 is inside this range.

These sizes pass necessary conditions for adaptation to be worthwhile. The range does not establish an optimal size or guarantee a gain.

This example counts pilot observations as a cost to the main wave. Pooled analysis uses different accounting.

Pilot-planning guide ↗