Allocation¶
The allocation module provides some utils to be used before running A/B test experiments. Groups allocation is the process that assigns (allocates) a list of users either to a group A (e.g. control) or to a group B (e.g. treatment). This module provides functionalities to randomly allocate users in two or more groups (A/B/C/…).
Let’s import first the tools needed.
[1]:
import numpy as np
import pandas as pd
from abexp.core.allocation import Allocator
from abexp.core.analysis_frequentist import FrequentistAnalyzer
Complete randomization¶
Here we want to randomly assign users in n groups (where n=2) in order to run an A/B test experiment with 2 variants, so called control and treatment groups. Complete randomization does not require any data on the user, and in practice, it yields balanced design for large-sample sizes.
[2]:
# Generate random data
user_id = np.arange(100)
[3]:
# Run allocation
df, stats = Allocator.complete_randomization(user_id=user_id,
ngroups=2,
prop=[0.4, 0.6],
seed=42)
[4]:
# Users list with group assigned
df.head()
[4]:
| user_id | group | |
|---|---|---|
| 0 | 0 | 1 |
| 1 | 1 | 1 |
| 2 | 2 | 1 |
| 3 | 3 | 1 |
| 4 | 4 | 1 |
[5]:
# Statistics of the randomization: #users per group
stats
[5]:
| group | 0 | 1 |
|---|---|---|
| #users | 40 | 60 |
Note: Post-allocation checks can be made to ensure the groups homogeneity and in case of imbalance, a new randomization can be performed (see the Homogeneity check section below for details).
Blocks randomization¶
In some case, one would like to consider one or more confounding factor(s) i.e. features which could unbalance the groups and bias the results if not taken into account during the randomization process. In this example we want to randomly assign users in n groups (where n=3, one control and two treatment groups) considering a confounding factor (‘level’). Users with similar characteristics (level) define a block, and randomization is conducted within a block. This enables balanced and homogeneous groups of similar sizes according to the confounding feature.
[6]:
# Generate random data
np.random.seed(42)
df = pd.DataFrame(data={'user_id': np.arange(1000),
'level': np.random.randint(1, 6, size=1000)})
[7]:
# Run allocation
df, stats = Allocator.blocks_randomization(df=df,
id_col='user_id',
stratum_cols='level',
ngroups=3,
seed=42)
[8]:
# Users data with group assigned
df.head()
[8]:
| user_id | level | group | |
|---|---|---|---|
| 0 | 0 | 4 | 1 |
| 1 | 1 | 5 | 2 |
| 2 | 2 | 3 | 2 |
| 3 | 3 | 5 | 1 |
| 4 | 4 | 5 | 0 |
[9]:
# Statistics of the randomization: #users per group in each level
stats
[9]:
| group | 0 | 1 | 2 |
|---|---|---|---|
| level | |||
| 1 | 70 | 70 | 70 |
| 2 | 64 | 63 | 63 |
| 3 | 62 | 64 | 64 |
| 4 | 69 | 69 | 68 |
| 5 | 68 | 68 | 68 |
Multi-level block randomization
You can stratify randomization on two or more features. In the example below we want to randomly allocate users in n groups (where n=5) in order to run an A/B test experiment with 5 variants, one control and four treatment groups. The stratification will be based on the user level and paying status in order to create homogeneous groups.
[10]:
# Generate random data
np.random.seed(42)
df = pd.DataFrame(data={'user_id': np.arange(1000),
'is_paying': np.random.randint(0, 2, size=1000),
'level': np.random.randint(1, 7, size=1000)})
[11]:
# Run allocation
df, stats = Allocator.blocks_randomization(df=df,
id_col='user_id',
stratum_cols=['level', 'is_paying'],
ngroups=5,
seed=42)
[12]:
# Users data with group assigned
df.head()
[12]:
| user_id | is_paying | level | group | |
|---|---|---|---|---|
| 0 | 0 | 0 | 6 | 2 |
| 1 | 1 | 1 | 1 | 1 |
| 2 | 2 | 0 | 1 | 0 |
| 3 | 3 | 0 | 1 | 3 |
| 4 | 4 | 0 | 5 | 1 |
[13]:
# Statistics of the randomization: #users per group in each level and paying status
stats
[13]:
| group | 0 | 1 | 2 | 3 | 4 | |
|---|---|---|---|---|---|---|
| level | is_paying | |||||
| 1 | 0 | 19 | 17 | 19 | 18 | 19 |
| 1 | 15 | 17 | 18 | 18 | 18 | |
| 2 | 0 | 17 | 17 | 14 | 17 | 17 |
| 1 | 18 | 17 | 16 | 18 | 17 | |
| 3 | 0 | 16 | 16 | 16 | 15 | 16 |
| 1 | 19 | 19 | 19 | 19 | 19 | |
| 4 | 0 | 12 | 12 | 12 | 12 | 11 |
| 1 | 15 | 15 | 15 | 14 | 15 | |
| 5 | 0 | 18 | 18 | 17 | 16 | 17 |
| 1 | 17 | 18 | 19 | 18 | 19 | |
| 6 | 0 | 18 | 19 | 19 | 18 | 18 |
| 1 | 16 | 15 | 16 | 16 | 15 |
Homogeneity check¶
Complete randomization does not guarantee homogeneous groups, but it yields balanced design for large-sample sizes. Blocks randomization guarantees homogeneous groups based on categorical variables (but not on continuous variable).
Thus, we can perform post-allocation checks to ensure the groups homogeneity both for continuous or categorical variables. In case of imbalance, a new randomization can be performed.
[14]:
# Generate random data
np.random.seed(42)
df = pd.DataFrame(data={'user_id': np.arange(1000),
'points': np.random.randint(100, 500, size=1000),
'collected_bonus': np.random.randint(2000, 7000, size=1000),
'is_paying': np.random.randint(0, 2, size=1000),
'level': np.random.randint(1, 7, size=1000)})
df.head()
[14]:
| user_id | points | collected_bonus | is_paying | level | |
|---|---|---|---|---|---|
| 0 | 0 | 202 | 6580 | 1 | 4 |
| 1 | 1 | 448 | 4075 | 0 | 5 |
| 2 | 2 | 370 | 2713 | 1 | 6 |
| 3 | 3 | 206 | 3062 | 0 | 3 |
| 4 | 4 | 171 | 3976 | 0 | 5 |
Single iteration
In the cell below it is shown a single iteration of check homogeneity analysis.
[15]:
# Run allocation
df, stats = Allocator.blocks_randomization(df=df,
id_col='user_id',
stratum_cols=['level', 'is_paying'],
ngroups=2,
seed=42)
[16]:
# Run homogeneity check analysis
X = df.drop(columns=['group'])
y = df['group']
analyzer = FrequentistAnalyzer()
analysis = analyzer.check_homogeneity(X, y, cat_cols=['is_paying','level'])
analysis
[16]:
| coef | std err | z | P>|z| | [0.025 | 0.975] | |
|---|---|---|---|---|---|---|
| user_id | -3.000000e-04 | 0.000000 | -1.505000e+00 | 0.132 | -0.001000 | 0.0001 |
| points | 2.000000e-04 | 0.001000 | 3.660000e-01 | 0.714 | -0.001000 | 0.0010 |
| collected_bonus | 6.935000e-05 | 0.000044 | 1.559000e+00 | 0.119 | -0.000018 | 0.0000 |
| C(is_paying, Treatment('1'))[T.0] | 8.000000e-03 | 0.127000 | 6.300000e-02 | 0.950 | -0.240000 | 0.2560 |
| C(level, Treatment('3'))[T.1] | -1.180000e-02 | 0.215000 | -5.500000e-02 | 0.956 | -0.433000 | 0.4090 |
| C(level, Treatment('3'))[T.2] | 1.440000e-02 | 0.226000 | 6.400000e-02 | 0.949 | -0.429000 | 0.4580 |
| C(level, Treatment('3'))[T.4] | -1.646000e-16 | 0.213000 | -7.740000e-16 | 1.000 | -0.417000 | 0.4170 |
| C(level, Treatment('3'))[T.5] | -1.628000e-16 | 0.215000 | -7.570000e-16 | 1.000 | -0.422000 | 0.4220 |
| C(level, Treatment('3'))[T.6] | -1.628000e-16 | 0.214000 | -7.590000e-16 | 1.000 | -0.420000 | 0.4200 |
The check_homogeneity function performs univariate logistic regression per each feature of the input dataset. If the p-value (column P>|z| in the table above) of any variables is below a certain threshold (e.g. threshold = 0.2), the random allocation is considered to be non homogeneous and it must be repeated. For instance, in the table above the variable collected_bonus is not homogeneously split across groups p-value = 0.119.
Multiple iterations
[17]:
# Generate random data
np.random.seed(42)
df = pd.DataFrame(data={'user_id': np.arange(1000),
'points': np.random.randint(100, 500, size=1000),
'collected_bonus': np.random.randint(2000, 7000, size=1000),
'is_paying': np.random.randint(0, 2, size=1000),
'level': np.random.randint(1, 7, size=1000)})
df.head()
[17]:
| user_id | points | collected_bonus | is_paying | level | |
|---|---|---|---|---|---|
| 0 | 0 | 202 | 6580 | 1 | 4 |
| 1 | 1 | 448 | 4075 | 0 | 5 |
| 2 | 2 | 370 | 2713 | 1 | 6 |
| 3 | 3 | 206 | 3062 | 0 | 3 |
| 4 | 4 | 171 | 3976 | 0 | 5 |
In the cell below we repeatedly perform random allocation until it creates homogeneous groups (up to a maximum number of iterations). The groups are considered to be homogeneous when the p-value (column P>|z|) of any variables is below a certain threshold (e.g. p-values < 0.2).
[18]:
# Define parameters
rep = 100
threshold = 0.2
analyzer = FrequentistAnalyzer()
for i in np.arange(rep):
# Run allocation
df, stats = Allocator.blocks_randomization(df=df,
id_col='user_id',
stratum_cols=['level', 'is_paying'],
ngroups=2,
seed=i + 45)
# Run homogeneity check analysis
X = df.drop(columns=['group'])
y = df['group']
analysis = analyzer.check_homogeneity(X, y, cat_cols=['is_paying','level'])
# Check p-values
if all(analysis['P>|z|'] > threshold):
break
df = df.drop(columns=['group'])
analysis
[18]:
| coef | std err | z | P>|z| | [0.025 | 0.975] | |
|---|---|---|---|---|---|---|
| user_id | -1.000000e-04 | 0.000000 | -5.640000e-01 | 0.573 | -0.001000 | 0.000 |
| points | 2.000000e-04 | 0.001000 | 3.200000e-01 | 0.749 | -0.001000 | 0.001 |
| collected_bonus | 2.449000e-05 | 0.000044 | 5.520000e-01 | 0.581 | -0.000063 | 0.000 |
| C(is_paying, Treatment('1'))[T.0] | 1.570000e-02 | 0.127000 | 1.240000e-01 | 0.901 | -0.232000 | 0.264 |
| C(level, Treatment('3'))[T.1] | -1.180000e-02 | 0.215000 | -5.500000e-02 | 0.956 | -0.433000 | 0.409 |
| C(level, Treatment('3'))[T.2] | -1.440000e-02 | 0.226000 | -6.400000e-02 | 0.949 | -0.458000 | 0.429 |
| C(level, Treatment('3'))[T.4] | -9.064000e-17 | 0.213000 | -4.260000e-16 | 1.000 | -0.417000 | 0.417 |
| C(level, Treatment('3'))[T.5] | -9.236000e-17 | 0.215000 | -4.290000e-16 | 1.000 | -0.422000 | 0.422 |
| C(level, Treatment('3'))[T.6] | -9.237000e-17 | 0.214000 | -4.310000e-16 | 1.000 | -0.420000 | 0.420 |