Ranking¶
H-score¶
-
tllib.ranking.hscore.
h_score
(features, labels)[source]¶ H-score in An Information-theoretic Approach to Transferability in Task Transfer Learning (ICIP 2019).
The H-Score \(\mathcal{H}\) can be described as:
\[\mathcal{H}=\operatorname{tr}\left(\operatorname{cov}(f)^{-1} \operatorname{cov}\left(\mathbb{E}[f \mid y]\right)\right)\]where \(f\) is the features extracted by the model to be ranked, \(y\) is the groud-truth label vector
- Parameters
features (np.ndarray) – features extracted by pre-trained model.
labels (np.ndarray) – groud-truth labels.
- Shape:
features: (N, F), with number of samples N and feature dimension F.
labels: (N, ) elements in [0, \(C_t\)), with target class number \(C_t\).
score: scalar.
LEEP: Log Expected Empirical Prediction¶
-
tllib.ranking.leep.
log_expected_empirical_prediction
(predictions, labels)[source]¶ Log Expected Empirical Prediction in LEEP: A New Measure to Evaluate Transferability of Learned Representations (ICML 2020).
The LEEP \(\mathcal{T}\) can be described as:
\[\mathcal{T}=\mathbb{E}\log \left(\sum_{z \in \mathcal{C}_s} \hat{P}\left(y \mid z\right) \theta\left(y \right)_{z}\right)\]where \(\theta\left(y\right)_{z}\) is the predictions of pre-trained model on source category, \(\hat{P}\left(y \mid z\right)\) is the empirical conditional distribution estimated by prediction and ground-truth label.
- Parameters
predictions (np.ndarray) – predictions of pre-trained model.
labels (np.ndarray) – groud-truth labels.
- Shape:
predictions: (N, \(C_s\)), with number of samples N and source class number \(C_s\).
labels: (N, ) elements in [0, \(C_t\)), with target class number \(C_t\).
score: scalar
NCE: Negative Conditional Entropy¶
-
tllib.ranking.nce.
negative_conditional_entropy
(source_labels, target_labels)[source]¶ Negative Conditional Entropy in Transferability and Hardness of Supervised Classification Tasks (ICCV 2019).
The NCE \(\mathcal{H}\) can be described as:
\[\mathcal{H}=-\sum_{y \in \mathcal{C}_t} \sum_{z \in \mathcal{C}_s} \hat{P}(y, z) \log \frac{\hat{P}(y, z)}{\hat{P}(z)}\]where \(\hat{P}(z)\) is the empirical distribution and \(\hat{P}\left(y \mid z\right)\) is the empirical conditional distribution estimated by source and target label.
- Parameters
source_labels (np.ndarray) – predicted source labels.
target_labels (np.ndarray) – groud-truth target labels.
- Shape:
source_labels: (N, ) elements in [0, \(C_s\)), with source class number \(C_s\).
target_labels: (N, ) elements in [0, \(C_t\)), with target class number \(C_t\).
LogME: Log Maximum Evidence¶
-
tllib.ranking.logme.
log_maximum_evidence
(features, targets, regression=False, return_weights=False)[source]¶ Log Maximum Evidence in LogME: Practical Assessment of Pre-trained Models for Transfer Learning (ICML 2021).
- Parameters
- Shape:
features: (N, F) with element in [0, \(C_t\)) and feature dimension F, where \(C_t\) denotes the number of target class
targets: (N, ) or (N, C), with C regression-labels.
weights: (F, \(C_t\)).
score: scalar.