Contextual Multi-Armed Bandit

For the contextual multi-armed bandit (cMAB) when user information is available (context), we implemented a generalisation of Thompson sampling algorithm (Agrawal and Goyal, 2014) based on NumPyro.

The following notebook contains an example of usage of the class Cmab, which implements the algorithm above.

import numpy as np

from pybandits.cmab import CmabBernoulli
from pybandits.model import BayesianNeuralNetwork, BnnLayerParams, BnnParams, FeaturesConfig, StudentTArray

/home/runner/.cache/pypoetry/virtualenvs/pybandits-vYJB-miV-py3.10/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

n_samples = 1000
n_features = 5

First, we need to define the input context matrix \(X\) of size (\(n\_samples, n\_features\)) and the mapping of possible actions \(a_i \in A\) to their associated model.

# context
X = 2 * np.random.random_sample((n_samples, n_features)) - 1  # random float in the interval (-1, 1)
print("X: context matrix of shape (n_samples, n_features)")
print(X[:10])

X: context matrix of shape (n_samples, n_features)
[[ 0.20698801 -0.68642079  0.24808328 -0.98492321 -0.78317524]
 [ 0.53372548  0.47914972 -0.13256975  0.03150264  0.16970786]
 [ 0.11977016 -0.36082668 -0.21225983 -0.63033428  0.74349276]
 [-0.56757527  0.5986273  -0.7686507  -0.19767006 -0.65528173]
 [ 0.33695884  0.73023901 -0.39399004  0.02049374 -0.59060345]
 [-0.05371841 -0.43400499 -0.48727424 -0.95101026 -0.16336443]
 [-0.39535699 -0.22588124 -0.65061813 -0.51245238  0.36723538]
 [-0.54937539  0.81721037 -0.53163048  0.10816275  0.90441919]
 [-0.59223737 -0.62222561  0.39051145 -0.69407987  0.56443256]
 [ 0.74410902  0.84970739 -0.6244633   0.94331482 -0.05040833]]

# define action model
bias = StudentTArray.cold_start(mu=1, sigma=2, shape=1)
weight = StudentTArray.cold_start(shape=(n_features, 1))
layer_params = BnnLayerParams(weight=weight, bias=bias)
model_params = BnnParams(bnn_layer_params=[layer_params])
feature_config = FeaturesConfig(n_features=n_features)

update_kwargs = {"num_steps": 100, "batch_size": 128, "optimizer_type": "adam"}

actions = {
    "a1": BayesianNeuralNetwork(
        model_params=model_params,
        feature_config=feature_config,
        update_kwargs=update_kwargs,
    ),
    "a2": BayesianNeuralNetwork(
        model_params=model_params,
        feature_config=feature_config,
        update_kwargs=update_kwargs,
    ),
}

We can now init the bandit given the mapping of actions \(a_i\) to their model.

# init contextual Multi-Armed Bandit model
cmab = CmabBernoulli(actions=actions)

The predict function below returns the action selected by the bandit at time \(t\): \(a_t = argmax_k P(r=1|\beta_k, x_t)\). The bandit selects one action per each sample of the contect matrix \(X\).

# predict action
pred_actions, _, _ = cmab.predict(X)
print("Recommended action: {}".format(pred_actions[:10]))

Recommended action: ['a2', 'a1', 'a2', 'a2', 'a1', 'a2', 'a2', 'a2', 'a2', 'a1']

Now, we observe the rewards and the context from the environment. In this example rewards and the context are randomly simulated.

# simulate reward from environment
simulated_rewards = np.random.randint(2, size=n_samples).tolist()
print("Simulated rewards: {}".format(simulated_rewards[:10]))

Simulated rewards: [0, 0, 0, 1, 1, 0, 0, 0, 0, 1]

Finally, we update the model providing per each action sample: (i) its context \(x_t\) (ii) the action \(a_t\) selected by the bandit, (iii) the corresponding reward \(r_t\).

# update model
cmab.update(context=X, actions=pred_actions, rewards=simulated_rewards)