Validation Metrics

Some more validation metrics no included in sklearn.metrics

Some validation functions.

revrand.metrics.lins_ccc(y_true, y_pred)

Lin’s Concordance Correlation Coefficient.

See https://en.wikipedia.org/wiki/Concordance_correlation_coefficient

Parameters:
  • y_true (ndarray) – vector of true targets
  • y_pred (ndarray) – vector of predicted targets
Returns:

1.0 for a perfect match between y_true and y_pred, less otherwise
Return type:

float

Example

>>> y_true = np.random.randn(100)
>>> lins_ccc(y_true, y_true) > 0.99  # Should be good predictor
True
>>> lins_ccc(y_true, np.zeros_like(y_true)) < 0.01  # Bad predictor
True
revrand.metrics.mll(y_true, y_pred, y_var)

Mean log loss under a Gaussian distribution.

Parameters:
  • y_true (ndarray) – vector of true targets
  • y_pred (ndarray) – vector of predicted targets
  • y_var (float or ndarray) – predicted variances
Returns:

The mean negative log loss (negative log likelihood)
Return type:

float

Example

>>> y_true = np.random.randn(100)
>>> mean_prob = - norm.logpdf(1e-2, loc=0)  # -ve log prob close to mean
>>> mll(y_true, y_true, 1) <= mean_prob  # Should be good predictor
True
>>> mll(y_true, np.random.randn(100), 1) >= mean_prob  # naive predictor
True
revrand.metrics.msll(y_true, y_pred, y_var, y_train)

Mean standardised log loss under a Gaussian distribution.

Parameters:
  • y_true (ndarray) – vector of true targets
  • y_pred (ndarray) – vector of predicted targets
  • y_var (float or ndarray) – predicted variances
  • y_train (ndarray) – vector of training targets by which to standardise
Returns:

The negative mean standardised log loss (negative log likelihood)
Return type:

float

Example

>>> y_true = np.random.randn(100)
>>> msll(y_true, y_true, 1, y_true) < 0  # Should be good predictor
True
>>> msll(y_true, np.random.randn(100), 1, y_true) >= 0  # naive predictor
True
revrand.metrics.smse(y_true, y_pred)

Standardised mean squared error.

Parameters:
  • y_true (ndarray) – vector of true targets
  • y_pred (ndarray) – vector of predicted targets
Returns:

SMSE of predictions vs truth (scalar)
Return type:

float

Example

>>> y_true = np.random.randn(100)
>>> smse(y_true, y_true)
0.0
>>> smse(y_true, np.random.randn(100)) >= 1.0
True