Composing an Offer with a Contextual Bandit: Item Portions (that must sum to 1) + Price
We run a store that sells a bundled offer made of three items. For every customer we must decide two things at once:
The mix — what portion of the bundle each of the three items takes. The three portions must add up to 1 (it is a single bundle).
The price — a normalized price in
[0, 1]for the whole offer.
Both decisions are continuous and both depend on context (who the customer is). This is a job for a contextual multi-armed bandit with a BNN-based quantitative model: a Bayesian Neural Network maps (context, offer parameters) -> P(purchase), and Thompson sampling explores the continuous offer space while exploiting what it has learned.
The catch: a structural equality constraint
portion_1 + portion_2 + portion_3 = 1 is an equality constraint. The quantitative optimizer in pybandits searches the hyper-cube [0, 1]^d and treats a constraint callable g(x) as feasible where g(x) >= 0 — i.e. it supports inequalities, not exact equalities. An exact equality carves out a measure-zero surface that a differential-evolution optimizer has nothing to descend on.
So we turn the equality into geometry the model and optimizer both like. The quantity vector is [p_1, p_2, price]: the first N_ITEMS - 1 = 2 coordinates are the item portions directly (so the BNN reasons in real portion space), and the last portion is the leftover p_3 = 1 - p_1 - p_2. Keeping every portion non-negative reduces to a single inequality, p_1 + p_2 <= 1, which we hand to the optimizer as a forbidden region. The feasible set is a triangle (half the cube) — a
full-measure region, far friendlier than the measure-zero equality.
This deliberately avoids two worse options: an exact equality on [p_1, p_2, p_3] (measure-zero for the optimizer, and a redundant third input the BNN cannot use), and a stick-breaking re-parameterization (valid by construction, but it warps the space and privileges one item, making the reward surface harder to learn).
[1]:
import numpy as np
import pandas as pd
from pybandits.cmab import CmabBernoulli
from pybandits.quantitative_model import QuantitativeBayesianNeuralNetwork
rng = np.random.default_rng(seed=42)
%load_ext autoreload
%autoreload 2
/home/runner/.cache/pypoetry/virtualenvs/pybandits-vYJB-miV-py3.10/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
The offer parameterization and its constraint
The quantity vector the bandit optimizes is [p_1, p_2, price]. split reads it back into the three portions (last = leftover) and the price. portions_sum_over_one is the forbidden-region margin: pybandits treats a region as forbidden where region(x) > 0, so returning p_1 + p_2 - 1 forbids exactly the corner of the cube where the portions would exceed 1 (i.e. where p_3 would go negative).
[2]:
N_ITEMS = 3 # items in the bundle; their portions must sum to 1
def split(quantity):
"""Read a quantity vector [p_1, ..., p_{N-1}, price] into (portions, price).
The first N_ITEMS - 1 coordinates are the item portions; the final
portion is the leftover so the portions sum to 1. The BNN sees these
coordinates directly, so it learns the reward in real portion space.
"""
free = np.asarray(quantity[: N_ITEMS - 1], dtype=float)
portions = np.append(free, 1.0 - free.sum())
price = float(quantity[N_ITEMS - 1])
return portions, price
def portions_sum_over_one(quantity):
"""Forbidden-region margin: > 0 where the free portions exceed 1 (invalid)."""
return float(np.sum(quantity[: N_ITEMS - 1]) - 1.0)
# Passed to predict(): forbids the p_1 + p_2 > 1 corner for the 'offer' arm, in
# both the optimized (exploit) and Thompson-sampled (explore) branches.
forbidden_actions = {"offer": portions_sum_over_one}
A quick check of the feasible region: about half the cube is feasible, and every feasible point yields non-negative portions that sum to 1.
[3]:
samples = rng.random((10000, N_ITEMS))
feasible = np.array([portions_sum_over_one(q) <= 0 for q in samples])
portions = np.array([split(q)[0] for q in samples[feasible]])
assert np.allclose(portions.sum(axis=1), 1.0), "portions must sum to 1"
assert (portions >= 0).all(), "feasible portions must be non-negative"
print(f"{feasible.mean():.0%} of the cube is feasible; all feasible offers have portions >= 0 summing to 1")
50% of the cube is feasible; all feasible offers have portions >= 0 summing to 1
Simulated environment: what makes a customer buy
Context is three features in [0, 1]: [affluence, preference_item_1, preference_item_2].
Each customer has a hidden ideal offer:
an ideal portion mix that reflects their item preferences (item 3’s preference is the leftover), and
an ideal price that rises with affluence.
The purchase probability is high when the offer’s mix and price are both close to the customer’s ideal, and decays with distance (a bell curve on each). The bandit has to discover this per-context sweet spot from binary purchase feedback alone.
[4]:
def make_ideal(context):
"""The customer's hidden sweet-spot offer, given their context."""
affluence, pref1, pref2 = context
raw = np.array([pref1, pref2, 1.0 - 0.5 * (pref1 + pref2)]) + 0.1 # keep every share positive
ideal_portions = raw / raw.sum()
ideal_price = 0.2 + 0.6 * affluence
return ideal_portions, ideal_price
def reward_function(quantity, context):
portions, price = split(quantity)
ideal_portions, ideal_price = make_ideal(context)
mix_fit = np.exp(-np.sum((portions - ideal_portions) ** 2) / 0.05)
price_fit = np.exp(-((price - ideal_price) ** 2) / 0.03)
prob = float(np.clip(mix_fit * price_fit, 0.0, 1.0))
return rng.binomial(1, prob), prob
def get_optimal_reward(context):
# The ideal offer hits mix_fit = price_fit = 1, so the best achievable prob is 1.
return 1.0
Build the bandit
A single quantitative action, "offer", of dimension N_ITEMS (two free portion coordinates + price). The BNN receives [quantity, context] and outputs P(purchase).
With one action the arm choice is trivial (you’ll see a “MAB will be deterministic” warning) — the real decision here is the continuous offer composition, which the quantity optimizer still explores. Add more actions (e.g. distinct bundle templates) if you also want the bandit to choose between offers.
[5]:
n_features = 3 # [affluence, preference_item_1, preference_item_2]
dimension = N_ITEMS # 2 free portion coordinates + 1 price
update_kwargs = {"epochs": 100, "optimizer_type": "adam", "batch_size": 64, "optimizer_kwargs": {"step_size": 0.001}}
dist_params_init = {"mu": 0, "sigma": 2}
actions = {
"offer": QuantitativeBayesianNeuralNetwork.cold_start(
dimension=dimension,
n_features=n_features,
base_model_cold_start_kwargs=dict(
hidden_dim_list=[32],
update_kwargs=update_kwargs,
dist_params_init=dist_params_init,
activation="gelu",
bias_std=0.1,
),
),
}
cmab = CmabBernoulli(actions=actions, epsilon=1) # full exploration for the training batch
/home/runner/work/pybandits/pybandits/pybandits/meta_model/base.py:209: UserWarning: Only a single action was supplied. This MAB will be deterministic.
warnings.warn("Only a single action was supplied. This MAB will be deterministic.")
Train the bandit
We collect a single exploration batch of 4096 offers with epsilon=1 (random, constraint-respecting offers — no optimizer on the cold model), then update the BNN once. predict is called on the whole batch at once — no loop. We pass forbidden_actions so every sampled offer respects p_1 + p_2 <= 1.
[6]:
current_context = rng.uniform(0, 1, (4096, n_features))
# Single exploration batch: one batched predict, one update.
pred_actions, _, _ = cmab.predict(context=current_context, forbidden_actions=forbidden_actions)
chosen_actions = [a[0] for a in pred_actions]
chosen_quantities = [list(a[1]) for a in pred_actions]
rewards_and_probs = [reward_function(q, ctx) for q, ctx in zip(chosen_quantities, current_context)]
rewards = [r for r, _ in rewards_and_probs]
probs = [p for _, p in rewards_and_probs]
regret = float(np.mean([get_optimal_reward(ctx) for ctx in current_context]) - np.mean(probs))
cmab.update(actions=chosen_actions, rewards=rewards, context=current_context, quantities=chosen_quantities)
print(f"Explored and updated on {len(current_context)} offers. Avg exploration regret: {regret:.4f}")
Explored and updated on 4096 offers. Avg exploration regret: 0.9523
Inspect the learned policy
We rebuild the bandit with epsilon=0 to exploit the trained model, then ask it for the chosen offer at a handful of representative customers and compare to the hidden ideal. The portion_sum column is 1 and every portion is non-negative — guaranteed by the p_1 + p_2 <= 1 forbidden region.
[7]:
cmab = CmabBernoulli(actions=actions, epsilon=0) # exploit the trained model
test_contexts = np.array(
[
[0.9, 0.9, 0.1], # affluent, loves item 1
[0.9, 0.1, 0.9], # affluent, loves item 2
[0.2, 0.4, 0.4], # budget, balanced taste
[0.5, 0.1, 0.1], # mid, leftover preference -> item 3
]
)
pred_actions, _, _ = cmab.predict(context=test_contexts, forbidden_actions=forbidden_actions)
rows = []
for ctx, (_, quantity) in zip(test_contexts, pred_actions):
portions, price = split(quantity)
ideal_portions, ideal_price = make_ideal(ctx)
rows.append(
{
"context": np.round(ctx, 2),
"chosen_portions": np.round(portions, 3),
"portion_sum": round(float(portions.sum()), 6),
"chosen_price": round(price, 3),
"ideal_portions": np.round(ideal_portions, 3),
"ideal_price": round(float(ideal_price), 3),
}
)
pd.DataFrame(rows)
/home/runner/.cache/pypoetry/virtualenvs/pybandits-vYJB-miV-py3.10/lib/python3.10/site-packages/scipy/optimize/_differentiable_functions.py:552: UserWarning: delta_grad == 0.0. Check if the approximated function is linear. If the function is linear better results can be obtained by defining the Hessian as zero instead of using quasi-Newton approximations.
self.H.update(delta_x, delta_g)
[7]:
| context | chosen_portions | portion_sum | chosen_price | ideal_portions | ideal_price | |
|---|---|---|---|---|---|---|
| 0 | [0.9, 0.9, 0.1] | [0.119, 0.0, 0.881] | 1.0 | 0.000 | [0.556, 0.111, 0.333] | 0.74 |
| 1 | [0.9, 0.1, 0.9] | [0.0, 0.0, 1.0] | 1.0 | 0.046 | [0.111, 0.556, 0.333] | 0.74 |
| 2 | [0.2, 0.4, 0.4] | [0.565, 0.278, 0.158] | 1.0 | 0.000 | [0.294, 0.294, 0.412] | 0.32 |
| 3 | [0.5, 0.1, 0.1] | [1.0, 0.0, 0.0] | 1.0 | 0.000 | [0.143, 0.143, 0.714] | 0.50 |
Continued example: discrete prices as separate arms
Suppose price is not a free continuous knob but a discrete choice — say -10%, 0%, +10% around a reference price. The natural model is one quantitative arm per price level: three arms that each optimize only the portion mix (dimension N_ITEMS - 1 = 2), while the bandit’s arm choice picks the price. Now Thompson sampling does real work across arms and optimizes the continuous mix within the chosen arm.
Everything else carries over: the p_1 + p_2 <= 1 forbidden region applies to every arm.
[8]:
PRICE_LEVELS = {"price_down": 0.45, "price_same": 0.50, "price_up": 0.55} # -10%, 0%, +10% of a 0.50 base
def portions_from(quantity):
"""Portions from a portions-only quantity (all coords are free portions; last = leftover)."""
free = np.asarray(quantity, dtype=float)
return np.append(free, 1.0 - free.sum())
def reward_price_arm(arm, quantity, context):
portions = portions_from(quantity)
price = PRICE_LEVELS[arm]
ideal_portions, ideal_price = make_ideal(context)
mix_fit = np.exp(-np.sum((portions - ideal_portions) ** 2) / 0.05)
price_fit = np.exp(-((price - ideal_price) ** 2) / 0.03)
prob = float(np.clip(mix_fit * price_fit, 0.0, 1.0))
return rng.binomial(1, prob), prob
def get_optimal_reward_discrete(context):
# Best achievable: perfect mix (mix_fit = 1) at the closest available price level.
_, ideal_price = make_ideal(context)
return max(np.exp(-((p - ideal_price) ** 2) / 0.03) for p in PRICE_LEVELS.values())
# One quantitative arm per price level; each optimizes portions only (dimension
# N_ITEMS - 1), under the same p_1 + p_2 <= 1 forbidden region.
forbidden_actions_multi = {arm: portions_sum_over_one for arm in PRICE_LEVELS}
actions_multi = {
arm: QuantitativeBayesianNeuralNetwork.cold_start(
dimension=N_ITEMS - 1, # portions only; the price is the arm
n_features=n_features,
base_model_cold_start_kwargs=dict(
hidden_dim_list=[32],
update_kwargs=update_kwargs,
dist_params_init=dist_params_init,
activation="gelu",
bias_std=0.1,
),
)
for arm in PRICE_LEVELS
}
Train the multi-arm bandit
Same single-batch recipe, but now predict also chooses among the three price arms. We explore one batch of 4096 (epsilon=1), update every arm from its share of the data, and measure regret against the best achievable reward on the discrete price grid (a perfect mix at the closest price level, generally below 1).
[9]:
cmab_multi = CmabBernoulli(actions=actions_multi, epsilon=1)
current_context = rng.uniform(0, 1, (4096, n_features))
pred_actions, _, _ = cmab_multi.predict(context=current_context, forbidden_actions=forbidden_actions_multi)
chosen_arms = [a[0] for a in pred_actions]
chosen_quantities = [list(a[1]) for a in pred_actions]
rewards_and_probs = [
reward_price_arm(arm, q, ctx) for arm, q, ctx in zip(chosen_arms, chosen_quantities, current_context)
]
rewards = [r for r, _ in rewards_and_probs]
probs = [p for _, p in rewards_and_probs]
regret = float(np.mean([get_optimal_reward_discrete(ctx) for ctx in current_context]) - np.mean(probs))
cmab_multi.update(actions=chosen_arms, rewards=rewards, context=current_context, quantities=chosen_quantities)
arm_counts = {arm: chosen_arms.count(arm) for arm in PRICE_LEVELS}
print(f"Explored and updated on {len(current_context)} offers. Avg regret: {regret:.4f}. Arm counts: {arm_counts}")
Explored and updated on 4096 offers. Avg regret: 0.5800. Arm counts: {'price_down': 1421, 'price_same': 1330, 'price_up': 1345}
Inspect the learned price + mix
Rebuild with epsilon=0 to exploit the trained arms. For each test customer the bandit now returns a price arm and a portion mix; it should lean toward the price level nearest the customer’s ideal price and a mix near their ideal portions.
[10]:
cmab_multi = CmabBernoulli(actions=actions_multi, epsilon=0) # exploit the trained arms
pred_actions, _, _ = cmab_multi.predict(context=test_contexts, forbidden_actions=forbidden_actions_multi)
rows = []
for ctx, (arm, quantity) in zip(test_contexts, pred_actions):
portions = portions_from(quantity)
ideal_portions, ideal_price = make_ideal(ctx)
rows.append(
{
"context": np.round(ctx, 2),
"chosen_price_arm": arm,
"chosen_price": PRICE_LEVELS[arm],
"chosen_portions": np.round(portions, 3),
"portion_sum": round(float(portions.sum()), 6),
"ideal_portions": np.round(ideal_portions, 3),
"ideal_price": round(float(ideal_price), 3),
}
)
pd.DataFrame(rows)
/home/runner/.cache/pypoetry/virtualenvs/pybandits-vYJB-miV-py3.10/lib/python3.10/site-packages/scipy/optimize/_differentiable_functions.py:552: UserWarning: delta_grad == 0.0. Check if the approximated function is linear. If the function is linear better results can be obtained by defining the Hessian as zero instead of using quasi-Newton approximations.
self.H.update(delta_x, delta_g)
[10]:
| context | chosen_price_arm | chosen_price | chosen_portions | portion_sum | ideal_portions | ideal_price | |
|---|---|---|---|---|---|---|---|
| 0 | [0.9, 0.9, 0.1] | price_down | 0.45 | [0.0, 0.0, 1.0] | 1.0 | [0.556, 0.111, 0.333] | 0.74 |
| 1 | [0.9, 0.1, 0.9] | price_down | 0.45 | [1.0, 0.0, 0.0] | 1.0 | [0.111, 0.556, 0.333] | 0.74 |
| 2 | [0.2, 0.4, 0.4] | price_same | 0.50 | [1.0, 0.0, 0.0] | 1.0 | [0.294, 0.294, 0.412] | 0.32 |
| 3 | [0.5, 0.1, 0.1] | price_up | 0.55 | [0.0, 0.0, 1.0] | 1.0 | [0.143, 0.143, 0.714] | 0.50 |
Conclusion
We used a contextual bandit with a BNN quantitative model to choose both the item mix and the price of an offer, conditioned on customer context — a fully continuous, multi-dimensional decision learned from binary purchase feedback.
The key idea for the sum(portions) == 1 requirement:
Optimize the portions directly and reduce the equality to one inequality. The first
N_ITEMS - 1coordinates are the actual portions (so the BNN learns in un-warped portion space), the last portion is the leftover, andp_1 + p_2 <= 1is enforced as a forbidden region — a full-measure triangle, far friendlier than a measure-zero equality.
Contrast with the alternatives: an exact equality on [p_1, p_2, p_3] gives the optimizer a measure-zero feasible set and the model a redundant input; a stick-breaking encoding is always valid but warps the space and privileges one item. Reach for the forbidden-region / constraint= callables whenever feasibility is a genuine inequality (“price must exceed cost”, “item 1 below 0.5”); reduce a structural equality to the smallest inequality you can, as we did here.
And when a dimension is discrete rather than continuous (a fixed set of prices, tiers, or templates), don’t force it into the quantity vector — model it as separate quantitative arms, one per level, and let the bandit choose the level while each arm optimizes the continuous remainder, as in the discrete-price example above.