Normalization¶
AFN: Adaptive Feature Norm¶

class
tllib.normalization.afn.
AdaptiveFeatureNorm
(delta)[source]¶ The Stepwise Adaptive Feature Norm loss (ICCV 2019)
Instead of using restrictive scalar R to match the corresponding feature norm, Stepwise Adaptive Feature Norm is used in order to learn taskspecific features with large norms in a progressive manner. We denote parameters of backbone \(G\) as \(\theta_g\), parameters of bottleneck \(F_f\) as \(\theta_f\) , parameters of classifier head \(F_y\) as \(\theta_y\), and features extracted from sample \(x_i\) as \(h(x_i;\theta)\). Full loss is calculated as follows
\[\begin{split}L(\theta_g,\theta_f,\theta_y)=\frac{1}{n_s}\sum_{(x_i,y_i)\in D_s}L_y(x_i,y_i)+\frac{\lambda}{n_s+n_t} \sum_{x_i\in D_s\cup D_t}L_d(h(x_i;\theta_0)+\Delta_r,h(x_i;\theta))\\\end{split}\]where \(L_y\) denotes classification loss, \(L_d\) denotes norm loss, \(\theta_0\) and \(\theta\) represent the updated and updating model parameters in the last and current iterations respectively.
 Parameters
delta (float) – positive residual scalar to control the feature norm enlargement.
 Inputs:
f (tensor): feature representations on source or target domain.
 Shape:
f: \((N, F)\) where F means the dimension of input features.
Outputs: scalar.
Examples:
>>> adaptive_feature_norm = AdaptiveFeatureNorm(delta=1) >>> f_s = torch.randn(32, 1000) >>> f_t = torch.randn(32, 1000) >>> norm_loss = adaptive_feature_norm(f_s) + adaptive_feature_norm(f_t)

class
tllib.normalization.afn.
Block
(in_features, bottleneck_dim=1000, dropout_p=0.5)[source]¶ Basic building block for Image Classifier with structure: FCBNReLUDropout. We use \(L_2\) preserved dropout layers. Given mask probability \(p\), input \(x_k\), generated mask \(a_k\), vanilla dropout layers calculate
\[\begin{split}\hat{x}_k = a_k\frac{1}{1p}x_k\\\end{split}\]While in \(L_2\) preserved dropout layers
\[\begin{split}\hat{x}_k = a_k\frac{1}{\sqrt{1p}}x_k\\\end{split}\]

class
tllib.normalization.afn.
ImageClassifier
(backbone, num_classes, num_blocks=1, bottleneck_dim=1000, dropout_p=0.5, **kwargs)[source]¶ ImageClassifier for AFN.
 Parameters
backbone (torch.nn.Module) – Any backbone to extract 2d features from data
num_classes (int) – Number of classes
num_blocks (int, optional) – Number of basic blocks. Default: 1
bottleneck_dim (int, optional) – Feature dimension of the bottleneck layer. Default: 1000
dropout_p (float, optional) – dropout probability. Default: 0.5
StochNorm: Stochastic Normalization¶

class
tllib.normalization.stochnorm.
StochNorm1d
(num_features, eps=1e05, momentum=0.1, affine=True, track_running_stats=True, p=0.5)[source]¶ Applies Stochastic Normalization over a 2D or 3D input (a minibatch of 1D inputs with optional additional channel dimension)
Stochastic Normalization is proposed in Stochastic Normalization (NIPS 2020)
\[ \begin{align}\begin{aligned}\hat{x}_{i,0} = \frac{x_i  \tilde{\mu}}{ \sqrt{\tilde{\sigma} + \epsilon}}\\\hat{x}_{i,1} = \frac{x_i  \mu}{ \sqrt{\sigma + \epsilon}}\\\hat{x}_i = (1s)\cdot \hat{x}_{i,0} + s\cdot \hat{x}_{i,1}\\ y_i = \gamma \hat{x}_i + \beta\end{aligned}\end{align} \]where \(\mu\) and \(\sigma\) are mean and variance of current minibatch data.
\(\tilde{\mu}\) and \(\tilde{\sigma}\) are current moving statistics of training data.
\(s\) is a branchselection variable generated from a Bernoulli distribution, where \(P(s=1)=p\).
During training, there are two normalization branches. One uses mean and variance of current minibatch data, while the other uses current moving statistics of the training data as usual batch normalization.
During evaluation, the moving statistics is used for normalization.
 Parameters
num_features (int) – \(c\) from an expected input of size \((b, c, l)\) or \(l\) from an expected input of size \((b, l)\).
eps (float) – A value added to the denominator for numerical stability. Default: 1e5
momentum (float) – The value used for the running_mean and running_var computation. Default: 0.1
affine (bool) – A boolean value that when set to
True
, gives the layer learnable affine parameters. Default:True
track_running_stats (bool) – A boolean value that when set to True, this module tracks the running mean and variance, and when set to False, this module does not track such statistics, and initializes statistics buffers running_mean and running_var as None. When these buffers are None, this module always uses batch statistics in both training and eval modes. Default: True p (float): The probability to choose the second branch (usual BN). Default: 0.5
 Shape:
Input: \((b, l)\) or \((b, c, l)\)
Output: \((b, l)\) or \((b, c, l)\) (same shape as input)

class
tllib.normalization.stochnorm.
StochNorm2d
(num_features, eps=1e05, momentum=0.1, affine=True, track_running_stats=True, p=0.5)[source]¶ Applies Stochastic Normalization over a 4D input (a minibatch of 2D inputs with additional channel dimension)
Stochastic Normalization is proposed in Stochastic Normalization (NIPS 2020)
\[ \begin{align}\begin{aligned}\hat{x}_{i,0} = \frac{x_i  \tilde{\mu}}{ \sqrt{\tilde{\sigma} + \epsilon}}\\\hat{x}_{i,1} = \frac{x_i  \mu}{ \sqrt{\sigma + \epsilon}}\\\hat{x}_i = (1s)\cdot \hat{x}_{i,0} + s\cdot \hat{x}_{i,1}\\ y_i = \gamma \hat{x}_i + \beta\end{aligned}\end{align} \]where \(\mu\) and \(\sigma\) are mean and variance of current minibatch data.
\(\tilde{\mu}\) and \(\tilde{\sigma}\) are current moving statistics of training data.
\(s\) is a branchselection variable generated from a Bernoulli distribution, where \(P(s=1)=p\).
During training, there are two normalization branches. One uses mean and variance of current minibatch data, while the other uses current moving statistics of the training data as usual batch normalization.
During evaluation, the moving statistics is used for normalization.
 Parameters
num_features (int) – \(c\) from an expected input of size \((b, c, h, w)\).
eps (float) – A value added to the denominator for numerical stability. Default: 1e5
momentum (float) – The value used for the running_mean and running_var computation. Default: 0.1
affine (bool) – A boolean value that when set to
True
, gives the layer learnable affine parameters. Default:True
track_running_stats (bool) – A boolean value that when set to True, this module tracks the running mean and variance, and when set to False, this module does not track such statistics, and initializes statistics buffers running_mean and running_var as None. When these buffers are None, this module always uses batch statistics in both training and eval modes. Default: True p (float): The probability to choose the second branch (usual BN). Default: 0.5
 Shape:
Input: \((b, c, h, w)\)
Output: \((b, c, h, w)\) (same shape as input)

class
tllib.normalization.stochnorm.
StochNorm3d
(num_features, eps=1e05, momentum=0.1, affine=True, track_running_stats=True, p=0.5)[source]¶ Applies Stochastic Normalization over a 5D input (a minibatch of 3D inputs with additional channel dimension)
Stochastic Normalization is proposed in Stochastic Normalization (NIPS 2020)
\[ \begin{align}\begin{aligned}\hat{x}_{i,0} = \frac{x_i  \tilde{\mu}}{ \sqrt{\tilde{\sigma} + \epsilon}}\\\hat{x}_{i,1} = \frac{x_i  \mu}{ \sqrt{\sigma + \epsilon}}\\\hat{x}_i = (1s)\cdot \hat{x}_{i,0} + s\cdot \hat{x}_{i,1}\\ y_i = \gamma \hat{x}_i + \beta\end{aligned}\end{align} \]where \(\mu\) and \(\sigma\) are mean and variance of current minibatch data.
\(\tilde{\mu}\) and \(\tilde{\sigma}\) are current moving statistics of training data.
\(s\) is a branchselection variable generated from a Bernoulli distribution, where \(P(s=1)=p\).
During training, there are two normalization branches. One uses mean and variance of current minibatch data, while the other uses current moving statistics of the training data as usual batch normalization.
During evaluation, the moving statistics is used for normalization.
 Parameters
num_features (int) – \(c\) from an expected input of size \((b, c, d, h, w)\)
eps (float) – A value added to the denominator for numerical stability. Default: 1e5
momentum (float) – The value used for the running_mean and running_var computation. Default: 0.1
affine (bool) – A boolean value that when set to
True
, gives the layer learnable affine parameters. Default:True
track_running_stats (bool) – A boolean value that when set to True, this module tracks the running mean and variance, and when set to False, this module does not track such statistics, and initializes statistics buffers running_mean and running_var as None. When these buffers are None, this module always uses batch statistics in both training and eval modes. Default: True p (float): The probability to choose the second branch (usual BN). Default: 0.5
 Shape:
Input: \((b, c, d, h, w)\)
Output: \((b, c, d, h, w)\) (same shape as input)

tllib.normalization.stochnorm.
convert_model
(module, p)[source]¶ Traverses the input module and its child recursively and replaces all instance of BatchNorm to StochNorm.
 Parameters
module (torch.nn.Module) – The input module needs to be convert to StochNorm model.
p (float) – The hyperparameter for StochNorm layer.
 Returns
The module converted to StochNorm version.
IBNNet: InstanceBatch Normalization Network¶

class
tllib.normalization.ibn.
InstanceBatchNorm2d
(planes, ratio=0.5)[source]¶ InstanceBatch Normalization layer from Two at Once: Enhancing Learning and Generalization Capacities via IBNNet (ECCV 2018).
Given input feature map \(f\_input\) of dimension \((C,H,W)\), we first split \(f\_input\) into two parts along channel dimension. They are denoted as \(f_1\) of dimension \((C_1,H,W)\) and \(f_2\) of dimension \((C_2,H,W)\), where \(C_1+C_2=C\). Then we pass \(f_1\) and \(f_2\) through IN and BN layer, respectively, to get \(IN(f_1)\) and \(BN(f_2)\). Last, we concat them along channel dimension to create \(f\_output=concat(IN(f_1), BN(f_2))\).

class
tllib.normalization.ibn.
IBNNet
(block, layers, ibn_cfg=('a', 'a', 'a', None))[source]¶ IBNNet without fully connected layer

property
out_features
¶ The dimension of output features

property
Modified from https://github.com/XingangPan/IBNNet @author: Baixu Chen @contact: cbx_99_hasta@outlook.com

tllib.normalization.ibn.
resnet18_ibn_a
(pretrained=False)[source]¶ Constructs a ResNet18IBNa model.
 Parameters
pretrained (bool) – If True, returns a model pretrained on ImageNet

tllib.normalization.ibn.
resnet18_ibn_b
(pretrained=False)[source]¶ Constructs a ResNet18IBNb model.
 Parameters
pretrained (bool) – If True, returns a model pretrained on ImageNet

tllib.normalization.ibn.
resnet34_ibn_a
(pretrained=False)[source]¶ Constructs a ResNet34IBNa model.
 Parameters
pretrained (bool) – If True, returns a model pretrained on ImageNet

tllib.normalization.ibn.
resnet34_ibn_b
(pretrained=False)[source]¶ Constructs a ResNet34IBNb model.
 Parameters
pretrained (bool) – If True, returns a model pretrained on ImageNet

tllib.normalization.ibn.
resnet50_ibn_a
(pretrained=False)[source]¶ Constructs a ResNet50IBNa model.
 Parameters
pretrained (bool) – If True, returns a model pretrained on ImageNet

tllib.normalization.ibn.
resnet50_ibn_b
(pretrained=False)[source]¶ Constructs a ResNet50IBNb model.
 Parameters
pretrained (bool) – If True, returns a model pretrained on ImageNet
MixStyle: Domain Generalization with MixStyle¶

class
tllib.normalization.mixstyle.
MixStyle
(p=0.5, alpha=0.1, eps=1e06)[source]¶ MixStyle module from DOMAIN GENERALIZATION WITH MIXSTYLE (ICLR 2021). Given input \(x\), we first compute mean \(\mu(x)\) and standard deviation \(\sigma(x)\) across spatial dimension. Then we permute \(x\) and get \(\tilde{x}\), corresponding mean \(\mu(\tilde{x})\) and standard deviation \(\sigma(\tilde{x})\). MixUp is performed using mean and standard deviation
\[\gamma_{mix} = \lambda\sigma(x) + (1\lambda)\sigma(\tilde{x})\]\[\beta_{mix} = \lambda\mu(x) + (1\lambda)\mu(\tilde{x})\]where \(\lambda\) is instancewise weight sampled from Beta distribution. MixStyle is then
\[MixStyle(x) = \gamma_{mix}\frac{x\mu(x)}{\sigma(x)} + \beta_{mix}\]
Note
MixStyle is only activated during training stage, with some probability \(p\).
@author: Baixu Chen @contact: cbx_99_hasta@outlook.com

tllib.normalization.mixstyle.resnet.
resnet18
(pretrained=False, progress=True, **kwargs)[source]¶ Constructs a ResNet18 model with MixStyle.

tllib.normalization.mixstyle.resnet.
resnet34
(pretrained=False, progress=True, **kwargs)[source]¶ Constructs a ResNet34 model with MixStyle.