{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Analysis Bootstrap"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This tutorial shows how to perform analysis after A/B test experiments using bootstrapping. This technique makes \n",
"inference about a certain estimate (e.g. sample mean) for a certain population parameter (e.g. population mean) by \n",
"resampling with replacement from the observed dataset. It does not make any assumption on the samples distribution.\n",
"\n",
"Let's import first the tools needed."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"from abexp.core.analysis_frequentist import FrequentistAnalyzer\n",
"from abexp.visualization.analysis_plots import AnalysisPlot"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Simple bootstrap"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here we want to compare a specific metrics of the control group versus the treatment group (e.g. average revenue per \n",
"user). We will perform bootstrapping on the kpi metrics (revenue) of each group."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"# Generate random data for revenue control group\n",
"revenue_contr = np.random.randint(low=50, high=500, size=100)\n",
"\n",
"# Generate random data for revenue treatment group\n",
"revenue_treat = np.random.randint(low=50, high=600, size=100) "
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"# Define the analyzer\n",
"analyzer = FrequentistAnalyzer()"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"# Define the aggregation function that will be applied on the sample\n",
"aggregation_func = np.mean\n",
"\n",
"# other possibles aggregation functions might be:\n",
"# - standard deviation = np.std, \n",
"# - sum = np.sum\n",
"# - median = lambda x: np.median(x, axis=0)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Bootstrapping will generate a sequence of ``N`` values (where ``N`` is the number of repetitions). The bootstrap \n",
"function returns a table with the median, 2.5 percentile and 97.5 percentile of this sequence."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" median \n",
" 2.5 percentile \n",
" 97.5 percentile \n",
" \n",
" \n",
" \n",
" \n",
" 282.475 \n",
" 255.092 \n",
" 307.1285 \n",
" \n",
" \n",
"
\n",
"
\n",
"\n",
"
\n",
" \n",
" \n",
" median \n",
" 2.5 percentile \n",
" 97.5 percentile \n",
" \n",
" \n",
" \n",
" \n",
" 305.5 \n",
" 276.192 \n",
" 339.1205 \n",
" \n",
" \n",
"
\n",
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"# Plot results with confidence interval\n",
"fig = AnalysisPlot.barplot(bars, [ci_contr, ci_treat], \n",
" groupslabel=['control group', 'treatment group'], \n",
" ylabel='average revenue per user', xlabel='')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the barplot above we see that there is no difference between empirical means because the confidence intervals \n",
"overlap. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Time series bootstrap"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here we want to compare a specific metrics of the control group versus the treatment group (e.g. average revenue per \n",
"user) across time. We will perform bootstrapping on the kpi metrics (revenue) of each group per each day. Note that the \n",
"bootstrap function maintains the correlation across days."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"# Generate random data for revenue control group\n",
"revenue_contr_ts = pd.DataFrame({'day1': np.random.randint(low=1, high=500, size=1000),\n",
" 'day2': np.random.randint(low=1, high=500, size=1000),\n",
" 'day3': np.random.randint(low=1, high=500, size=1000),\n",
" 'day4': np.random.randint(low=1, high=500, size=1000),\n",
" 'day5': np.random.randint(low=1, high=500, size=1000),\n",
" 'day6': np.random.randint(low=1, high=500, size=1000),\n",
" 'day7': np.random.randint(low=1, high=500, size=1000)})\n",
"\n",
"# Generate random data for revenue treatment group\n",
"revenue_treat_ts = pd.DataFrame({'day1': np.random.randint(low=1, high=600, size=1000),\n",
" 'day2': np.random.randint(low=1, high=600, size=1000),\n",
" 'day3': np.random.randint(low=1, high=600, size=1000),\n",
" 'day4': np.random.randint(low=1, high=600, size=1000),\n",
" 'day5': np.random.randint(low=1, high=600, size=1000),\n",
" 'day6': np.random.randint(low=1, high=600, size=1000),\n",
" 'day7': np.random.randint(low=1, high=600, size=1000)})"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" median \n",
" 2.5 percentile \n",
" 97.5 percentile \n",
" \n",
" \n",
" \n",
" \n",
" day1 \n",
" 246.6630 \n",
" 237.597475 \n",
" 255.912150 \n",
" \n",
" \n",
" day2 \n",
" 248.0410 \n",
" 239.164575 \n",
" 256.451925 \n",
" \n",
" \n",
" day3 \n",
" 250.4535 \n",
" 241.068275 \n",
" 259.431525 \n",
" \n",
" \n",
" day4 \n",
" 252.0625 \n",
" 244.145850 \n",
" 261.326350 \n",
" \n",
" \n",
" day5 \n",
" 246.4465 \n",
" 237.647800 \n",
" 255.376150 \n",
" \n",
" \n",
" day6 \n",
" 252.0445 \n",
" 243.933075 \n",
" 261.501475 \n",
" \n",
" \n",
" day7 \n",
" 249.0605 \n",
" 240.451950 \n",
" 257.918600 \n",
" \n",
" \n",
"
\n",
"
\n",
"\n",
"
\n",
" \n",
" \n",
" median \n",
" 2.5 percentile \n",
" 97.5 percentile \n",
" \n",
" \n",
" \n",
" \n",
" day1 \n",
" 305.8540 \n",
" 295.303525 \n",
" 315.689025 \n",
" \n",
" \n",
" day2 \n",
" 297.1785 \n",
" 287.122225 \n",
" 308.437950 \n",
" \n",
" \n",
" day3 \n",
" 311.1690 \n",
" 300.329400 \n",
" 322.258075 \n",
" \n",
" \n",
" day4 \n",
" 297.0245 \n",
" 286.180500 \n",
" 307.432525 \n",
" \n",
" \n",
" day5 \n",
" 302.7850 \n",
" 292.530300 \n",
" 313.642875 \n",
" \n",
" \n",
" day6 \n",
" 300.4425 \n",
" 289.563675 \n",
" 311.556775 \n",
" \n",
" \n",
" day7 \n",
" 299.9155 \n",
" 288.822375 \n",
" 310.642600 \n",
" \n",
" \n",
"
\n",
"
"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"fig = AnalysisPlot.timeseries_plot(y, [ci_contr_ts, ci_treat_ts])"
]
}
],
"metadata": {
"hide_input": false,
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.6"
},
"pycharm": {
"stem_cell": {
"cell_type": "raw",
"metadata": {
"collapsed": false
},
"source": []
}
}
},
"nbformat": 4,
"nbformat_minor": 4
}