We exploit the computational structure of convolutions to re-parametrize the weight uncertainty (under the Bayesian framework) in an incremental learning setting. We validate our hypothesis "structured adaptability bias" through empirical experimentations. Our method results in both parameter efficiency and accuracy improvements over state-of-the-art.