Package {imv}

Cross-validated IMV for comparing two models

Description

S3 generic that computes the InterModel Vigorish (IMV) between a baseline model m0 and an enhanced model m1 via k-fold cross-validation. For each fold, both models are refit on the training partition and evaluated on the held-out partition; the IMV is computed from those out-of-fold predictions.

imv.default provides an escape hatch for unsupported model types via predict_fn: in that case the original fitted models are used (without refitting) to obtain predictions on each test fold.

Usage

imv(m0, m1, ...)

## S3 method for class 'glm'
imv(m0, m1, data = NULL, nfold = 4, predict_fn = NULL, y = NULL, ...)

## Default S3 method:
imv(m0, m1, data = NULL, nfold = 4, predict_fn = NULL, y = NULL, ...)

Arguments

Value

A named list with four elements:

folds

Numeric vector of per-fold IMV values (length nfold).

mean

Mean IMV across folds.

sd

Standard deviation of per-fold IMVs.

ci

Named numeric vector of length 2: a 95% interval computed as mean +/- 1.96 * (sd / sqrt(nfold)).

References

Domingue, B. W., Rahal, C., Faul, J., Freese, J., Kanopka, K., Rigos, A., Stenhaug, B., & Tripathi, A. S. (2025). The InterModel Vigorish (IMV) as a flexible and portable approach for quantifying predictive accuracy with binary outcomes. PloS one, 20(3), e0316491. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0316491

See Also

imv.binary, imv.glmerMod, imv.SingleGroupClass

Examples

## --- glm ------------------------------------------------------------
set.seed(1)
x  <- rnorm(100)
y  <- rbinom(100, 1, plogis(x))
df <- data.frame(x = x, y = y)
m0 <- glm(y ~ 1, df, family = "binomial")
m1 <- glm(y ~ x, df, family = "binomial")
result <- imv(m0, m1, nfold = 2)
result$mean
result$ci

## --- custom predict_fn (escape hatch for unsupported model types) ---
pfn    <- function(model, newdata) predict(model, newdata, type = "response")
result <- imv(m0, m1, data = df, y = "y", predict_fn = pfn, nfold = 2)
result$mean


## --- glmer (requires lme4) ------------------------------------------
if (requireNamespace("lme4", quietly = TRUE)) {
  data(sleepstudy, package = "lme4")
  sleepstudy$slow <- as.integer(sleepstudy$Reaction > 300)
  m0 <- lme4::glmer(slow ~ (1 | Subject),        sleepstudy, family = binomial)
  m1 <- lme4::glmer(slow ~ Days + (1 | Subject), sleepstudy, family = binomial)
  imv(m0, m1)
}

## --- mirt (requires mirt) -------------------------------------------
if (requireNamespace("mirt", quietly = TRUE)) {
  resp <- mirt::expand.table(mirt::LSAT7)
  mod1 <- mirt::mirt(resp, 1, "Rasch", verbose = FALSE)  # 1PL
  mod2 <- mirt::mirt(resp, 1, verbose = FALSE)            # 2PL
  imv(mod1, mod2)
}

Cross-validated IMV for mirt IRT models

Description

Computes the InterModel Vigorish (IMV) for item response theory models fit with the mirt package via response-level k-fold cross-validation. Fold splits are at the individual response level: each held-out observation is a single person-by-item pair, and ability is estimated from the remaining responses for that person.

When a single model is supplied (m1 = NULL), predictions from m0 are compared to item-level prevalence rates (the null model). When two models are supplied, m0 serves as the baseline and m1 as the enhanced model.

Only dichotomous response models are supported.

Usage

## S3 method for class 'SingleGroupClass'
imv(m0, m1 = NULL, data = NULL, nfold = 5,
     predict_fn = NULL, y = NULL,
     fscores.options = list(method = "EAP"),
     whole.matrix = TRUE,
     remove.nonvarying.items = TRUE,
     remove.allNA.rows = TRUE, ...)

Arguments

Value

A named list with four elements:

folds

Numeric vector of per-fold IMV values (length nfold).

mean

Mean IMV across folds.

sd

Standard deviation of per-fold IMVs.

ci

Named numeric vector of length 2: a 95% interval computed as mean +/- 1.96 * (sd / sqrt(nfold)).

References

Domingue, B. W., Rahal, C., Faul, J., Freese, J., Kanopka, K., Rigos, A., Stenhaug, B., & Tripathi, A. S. (2025). The InterModel Vigorish (IMV) as a flexible and portable approach for quantifying predictive accuracy with binary outcomes. PloS one, 20(3), e0316491. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0316491

See Also

imv, imv.binary

Examples

if (requireNamespace("mirt", quietly = TRUE)) {
  set.seed(1)
  resp <- mirt::expand.table(mirt::LSAT7)

  # Single model vs prevalence baseline
  mod1 <- mirt::mirt(resp, 1, "Rasch", verbose = FALSE)
  imv(mod1)

  # Two models
  mod2 <- mirt::mirt(resp, 1, verbose = FALSE)
  imv(mod1, mod2)

  # Priors specified as a variable are handled correctly
  my_prior <- list(a1 = c(0, 1, 0.25, 3))
  mod3 <- mirt::mirt(resp, 1, prior.list = my_prior, verbose = FALSE)
  imv(mod3)
}

Compute IMV for binary outcomes

Description

Computes the InterModel Vigorish (IMV) comparing baseline predictions p1 to enhanced predictions p2 for binary outcomes y. A positive value indicates that p2 predicts better than p1 out of sample; a negative value indicates the reverse.

Usage

## S3 method for class 'binary'
imv(m0, m1, p2, sigma = 1e-04, ...)

Arguments

Value

A scalar IMV value. Positive values favour p2; negative values favour p1.

References

Domingue, B. W., Rahal, C., Faul, J., Freese, J., Kanopka, K., Rigos, A., Stenhaug, B., & Tripathi, A. S. (2025). The InterModel Vigorish (IMV) as a flexible and portable approach for quantifying predictive accuracy with binary outcomes. PloS one, 20(3), e0316491. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0316491

See Also

imv

Examples

set.seed(1)
x  <- rnorm(1000)
y  <- rbinom(length(x), 1, plogis(x))
df <- data.frame(x = x, y = y)
m  <- glm(y ~ x, df, family = "binomial")
pr <- predict(m, data.frame(x = x), type = "response")
imv.binary(y, mean(y), pr)

Cross-validated IMV for binomial mixed-effects models

Description

Computes the InterModel Vigorish (IMV) for binomial mixed-effects models fit with lme4::glmer() via k-fold cross-validation. Both models are refit on each training fold; predictions on the held-out fold use allow.new.levels = TRUE to handle random-effect levels not seen during training.

Only binomial family models are supported.

Usage

## S3 method for class 'glmerMod'
imv(m0, m1, data = NULL, nfold = 4, predict_fn = NULL, y = NULL, ...)

Arguments

Value

A named list with four elements:

folds

Numeric vector of per-fold IMV values (length nfold).

mean

Mean IMV across folds.

sd

Standard deviation of per-fold IMVs.

ci

Named numeric vector of length 2: a 95% interval computed as mean +/- 1.96 * (sd / sqrt(nfold)).

References

Domingue, B. W., Rahal, C., Faul, J., Freese, J., Kanopka, K., Rigos, A., Stenhaug, B., & Tripathi, A. S. (2025). The InterModel Vigorish (IMV) as a flexible and portable approach for quantifying predictive accuracy with binary outcomes. PloS one, 20(3), e0316491. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0316491

See Also

imv, imv.binary

Examples

if (requireNamespace("lme4", quietly = TRUE)) {
  data(sleepstudy, package = "lme4")
  sleepstudy$slow <- as.integer(sleepstudy$Reaction > 300)

  m0 <- lme4::glmer(slow ~ (1 | Subject), sleepstudy, family = binomial)
  m1 <- lme4::glmer(slow ~ Days + (1 | Subject), sleepstudy, family = binomial)
  imv(m0, m1)
}

IMV for a GLM compared to a prevalence baseline

Description

Legacy function. Computes the IMV for a fitted glm model against a null model (intercept only) via k-fold cross-validation. Both the full and null models are refit on each training fold and evaluated on the held-out fold.

For new code, prefer imv(m0, m1) with an explicit null model as m0.

Usage

imv0glm(m, nfold = 5)

Arguments

Value

A numeric vector of length nfold containing the per-fold IMV values.

References

Domingue, B. W., Rahal, C., Faul, J., Freese, J., Kanopka, K., Rigos, A., Stenhaug, B., & Tripathi, A. S. (2025). The InterModel Vigorish (IMV) as a flexible and portable approach for quantifying predictive accuracy with binary outcomes. PloS one, 20(3), e0316491. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0316491

See Also

imv, imv.binary

Examples

set.seed(1)
x  <- rnorm(1000)
y  <- rbinom(length(x), 1, plogis(x))
df <- data.frame(x = x, y = y)
m  <- glm(y ~ x, df, family = "binomial")
imv0glm(m)

IMV for a GLM versus the same model with one variable removed

Description

Legacy function. Computes the IMV comparing a full glm model to a reduced model with var.nm dropped from the formula, via k-fold cross-validation. Both models are refit on each training fold and evaluated on the held-out fold.

For new code, prefer constructing both models explicitly and calling imv(m0, m1).

Usage

imvglm.rmvar(m, nfold = 5, var.nm)

Arguments

Value

A numeric vector of length nfold containing the per-fold IMV values. The reduced model (without var.nm) serves as the baseline; the full model serves as the enhanced model. A positive mean indicates that var.nm improves out-of-sample prediction.

References

Domingue, B. W., Rahal, C., Faul, J., Freese, J., Kanopka, K., Rigos, A., Stenhaug, B., & Tripathi, A. S. (2025). The InterModel Vigorish (IMV) as a flexible and portable approach for quantifying predictive accuracy with binary outcomes. PloS one, 20(3), e0316491. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0316491

See Also

imv, imv.binary

Examples

set.seed(1)
x  <- rnorm(1000)
z  <- rnorm(1000)
y  <- rbinom(length(x), 1, plogis(x))
df <- data.frame(x = x, z = z, y = y)
m  <- glm(y ~ x + z, df, family = "binomial")
imvglm.rmvar(m, var.nm = "z")