MDL: Lower-bound N for FIA

Model Selection with FIA

Based on Minimum Description Length (MDL / FIA) multiTree allows to compute the Fisher information approximation (FIA), a model selection criterion based on the minimum description length principle similar to AIC or BIC. Essentially, FIA allows to make a trade-off between the fit and complexity of the models under consideration. FIA has the advantage that it takes the functional complexity of the models into account (e.g., how the parameters in the MPT model are connected and whether order constraints such as Do>Dn are included). However, FIA is only an approximation that can fail if the number of observations is too small, leading to severely biased model selection. As a remedy, it should only be used if the total number of observations (usually the numer of responses times the number of participants) exceeds a lower bound. The following Excel sheet allows to compute this lower-bound sample size for FIA:

Reference

For more details see:

  • [PDF] Heck, D. W., Moshagen, M., & Erdfelder, E. (2014). Model selection by minimum description length: Lower-bound sample sizes for the Fisher information approximation. Journal of Mathematical Psychology, 60, 29–34. https://doi.org/10.1016/j.jmp.2014.06.002
    [Abstract] [BibTeX] [Preprint] [GitHub]

    The Fisher information approximation (FIA) is an implementation of the minimum description length principle for model selection. Unlike information criteria such as AIC or BIC, it has the advantage of taking the functional form of a model into account. Unfortunately, FIA can be misleading in finite samples, resulting in an inversion of the correct rank order of complexity terms for competing models in the worst case. As a remedy, we propose a lower-bound N’ for the sample size that suffices to preclude such errors. We illustrate the approach using three examples from the family of multinomial processing tree models.

    @article{heck2014model,
title = {Model Selection by Minimum Description Length: {{Lower-bound}} Sample Sizes for the {{Fisher}} Information Approximation},
author = {Heck, Daniel W and Moshagen, Morten and Erdfelder, Edgar},
date = {2014},
journaltitle = {Journal of Mathematical Psychology},
volume = {60},
pages = {29--34},
doi = {10.1016/j.jmp.2014.06.002},
abstract = {The Fisher information approximation (FIA) is an implementation of the minimum description length principle for model selection. Unlike information criteria such as AIC or BIC, it has the advantage of taking the functional form of a model into account. Unfortunately, FIA can be misleading in finite samples, resulting in an inversion of the correct rank order of complexity terms for competing models in the worst case. As a remedy, we propose a lower-bound N' for the sample size that suffices to preclude such errors. We illustrate the approach using three examples from the family of multinomial processing tree models.},
arxivnumber = {1808.00212},
github = {https://github.com/danheck/FIAminimumN},
keywords = {heckfirst}
}