Ranking

H-score

tllib.ranking.hscore.h_score(features, labels)

H-score in An Information-theoretic Approach to Transferability in Task Transfer Learning (ICIP 2019).

The H-Score \(\mathcal{H}\) can be described as:

\[\mathcal{H}=\operatorname{tr}\left(\operatorname{cov}(f)^{-1} \operatorname{cov}\left(\mathbb{E}[f \mid y]\right)\right)\]

where \(f\) is the features extracted by the model to be ranked, \(y\) is the groud-truth label vector

Parameters

• features (np.ndarray) – features extracted by pre-trained model.

• labels (np.ndarray) – groud-truth labels.

Shape:

• features: (N, F), with number of samples N and feature dimension F.

• labels: (N, ) elements in [0, \(C_t\)), with target class number \(C_t\).

• score: scalar.

LEEP: Log Expected Empirical Prediction

tllib.ranking.leep.log_expected_empirical_prediction(predictions, labels)

Log Expected Empirical Prediction in LEEP: A New Measure to Evaluate Transferability of Learned Representations (ICML 2020).

The LEEP \(\mathcal{T}\) can be described as:

\[\mathcal{T}=\mathbb{E}\log \left(\sum_{z \in \mathcal{C}_s} \hat{P}\left(y \mid z\right) \theta\left(y \right)_{z}\right)\]

where \(\theta\left(y\right)_{z}\) is the predictions of pre-trained model on source category, \(\hat{P}\left(y \mid z\right)\) is the empirical conditional distribution estimated by prediction and ground-truth label.

Parameters

• predictions (np.ndarray) – predictions of pre-trained model.

• labels (np.ndarray) – groud-truth labels.

Shape:

• predictions: (N, \(C_s\)), with number of samples N and source class number \(C_s\).

• labels: (N, ) elements in [0, \(C_t\)), with target class number \(C_t\).

• score: scalar

NCE: Negative Conditional Entropy

tllib.ranking.nce.negative_conditional_entropy(source_labels, target_labels)

Negative Conditional Entropy in Transferability and Hardness of Supervised Classification Tasks (ICCV 2019).

The NCE \(\mathcal{H}\) can be described as:

\[\mathcal{H}=-\sum_{y \in \mathcal{C}_t} \sum_{z \in \mathcal{C}_s} \hat{P}(y, z) \log \frac{\hat{P}(y, z)}{\hat{P}(z)}\]

where \(\hat{P}(z)\) is the empirical distribution and \(\hat{P}\left(y \mid z\right)\) is the empirical conditional distribution estimated by source and target label.

Parameters

• source_labels (np.ndarray) – predicted source labels.

• target_labels (np.ndarray) – groud-truth target labels.

Shape:

• source_labels: (N, ) elements in [0, \(C_s\)), with source class number \(C_s\).

• target_labels: (N, ) elements in [0, \(C_t\)), with target class number \(C_t\).

LogME: Log Maximum Evidence

tllib.ranking.logme.log_maximum_evidence(features, targets, regression=False, return_weights=False)

Log Maximum Evidence in LogME: Practical Assessment of Pre-trained Models for Transfer Learning (ICML 2021).

Parameters

• features (np.ndarray) – feature matrix from pre-trained model.

• targets (np.ndarray) – targets labels/values.

• regression (bool, optional) – whether to apply in regression setting. (Default: False)

• return_weights (bool, optional) – whether to return bayesian weight. (Default: False)

Shape:

• features: (N, F) with element in [0, \(C_t\)) and feature dimension F, where \(C_t\) denotes the number of target class

• targets: (N, ) or (N, C), with C regression-labels.

• weights: (F, \(C_t\)).

• score: scalar.