Effect of the shape of distribution and mean in Huynh’s po index

Authors

  • Gonzalo Almerich Universitat de València
  • Rosa M. Bo Bonet Universitat de València

DOI:

https://doi.org/10.7203/relieve.12.1.4248

Keywords:

Evaluation, Assessment, criterion-referenced evaluation, reliability, criterion-referenced reliability, classification consistency indexes, po index, Huynh’s method, mean, shape of distribution

Abstract

In the field of Educational Evaluation, the Criterion-Referenced Evaluation is a very relevant, though incipient approach. In this paper, we present a simulation study oriented to the analysis of a reliability index: the index po calculated by means of the Huynh’s method for criterion-referenced tests. The simulation study has been carried out with the aid of computer software developed ad hoc. The aim of this paper is to describe the influence of two variables – the distribution shape and the mean- on that index. The conclusion is that both variables consistently affect the reliability index. Furthermore, the obtained evidence brings relevant information about desirable metrical characteristics for applied use of the index in the criterion-referenced tests.

References

Algina, J. & Noe, M. J. (1978). A Study of Accuracy of Subkoviak's Single-Administration Estimate of the Coefficient of Agreement Using two True-Score Estimates. Journal of Educational Measurement. Vol. 15, nº 2, pp. 101-110.

Almerich, G. (2000). Indicadores de fiabilidad en pruebas de referencia criterial. Tesis doctoral. Universidad de Valencia. No publicada.

Berk, R.A. (1984). Selecting the Index of Reliability. En Berk, R. A. (Ed.) A Guide to Criterion-Referenced Test Construction. The John Hopkins University Press. Baltimore. Pp. 231-266.

Brennan, R. L. (1980). Applications of Generalizability Theory. En Berk, R. A. (Ed.) Criterion-Referenced Measurement: The State of the art. Baltimore. John Hopkins University Press. Pp. 186-232.

Brennan, R. L. & Kane, M. T. (1977). An Index of Dependability for Mastery Test. Journal of Educational Measurement, vol. 14, nº. 3, pp. 277-289.

http://dx.doi.org/10.1111/j.1745-3984.1977.tb00045.x

Glaser, R. (1963). Instructional technology and the measurement of learning outcomes: Some questions. American Psychologist, 18, pp. 519-521.

http://dx.doi.org/10.1037/h0049294

Glaser, R. & Klaus, D.J. (1962). Proficiency measurement: Assessing human performance. En Gagne, R. M. (Ed.) Psychological Principles in systems development. New York. Holt, Rinehart and Winston. Pp. 419-474.

Hambleton, R.K. & Novick, M. R. (1973). Toward an integration of theory and method for criterion-referenced tests. Journal of Educational Measurement, vol. 10, pp. 159-171.

http://dx.doi.org/10.1111/j.1745-3984.1973.tb00793.x

Hambleton, R.K. & Slater, S.C. (1997). Reliability of credentialing examinations on the impact of scoring models and standard-setting policies. Applied Measurement in Education, vol. 10, Nº 1, pp. 19-38.

http://dx.doi.org/10.1207/s15324818ame1001_2

Huynh, H (1976). On the reliability of decisions in domain-referenced testing. Journal of Educational Measurement, vol. 13, pp. 253-264.

http://dx.doi.org/10.1111/j.1745-3984.1976.tb00016.x

Huynh, H. & Saunders, J. C. (1980). Accuracy of two Procedures for Estimating Reliability of Mastery Tests. Journal of Educational Measurement. Vol. 17, nº. 4, Pp. 351-358.

http://dx.doi.org/10.1111/j.1745-3984.1980.tb00836.x

Jornet, J. M. y Suárez, J.M (1994). Evaluacion Referida al criterio. Construcción de un test criterial de clase. En Garcia Hoz, V.(Dir.) Problemas y métodos de investigación en educación personalizada. Rialp. Madrid. Pp. 419-443.

Keats, J. A. & Lord, F. M. (1962). A Theoretical Distribution for Mental Test Scores. Psychometrika. Vol 21, nº. 1, Pp. 59-72.

http://dx.doi.org/10.1007/BF02289665

Lee, W., Hanson, B. A. and Brennan, R.L. (2002). Estimating Consistency and Accuracy Indices for Multiples Classifications. Applied Psychological Measurement. Vol. 26, nº 4, Pp. 412-432.

http://dx.doi.org/10.1177/014662102237797

Livingston, S. A. (1972). Criterion-Referenced Applications of Classical Test Theory. Journal of Educational Measurement, vol. 9, nº. 1, pp. 13-26.

http://dx.doi.org/10.1111/j.1745-3984.1972.tb00756.x

Marshall, J. L. & Haertel, E. H. (1976). The Mean Split-Half Coefficient of Agreement: A Single Administration Index of Reliability for Mastery Tests. Manuscrito no publicado. University Of Wisconsin.

Peng, C-Y & Subkoviak, M. J. (1980). A Note on Huynh's Normal Approximation Procedure for Estimating Criterion-Referenced Reliability. Journal of Educational Measurement. Vol. 17, nº. 4, pp. 359-368.

http://dx.doi.org/10.1111/j.1745-3984.1980.tb00837.x

Popham, W.J. & Husek, T.R. (1969). Implications of criterion-referenced measurement. Journal of Educational Measurement, vol. 6, pp. 1-9.

http://dx.doi.org/10.1111/j.1745-3984.1969.tb00654.x

Spray, J. A. & Welch, C. J. (1990). Estimation of Classification Consistency when the Probability of a Correct Response Varies. Journal of Educational Measurement, vol. 27, nº 1, pp. 15-25.

http://dx.doi.org/10.1111/j.1745-3984.1990.tb00731.x

Subkoviak, M (1976). Estimating reliability from a single administration of a criterion-referenced test. Journal of Educational Measurement, vol. 13, pp. 265-275.

http://dx.doi.org/10.1111/j.1745-3984.1976.tb00017.x

Subkoviak, M. J. (1978). Empirical Investigation of Procedures for Estimating Reliability for Mastery Tests. Journal of Educational Measurement, vol. 15, nº. 1, pp. 111-116.

http://dx.doi.org/10.1111/j.1745-3984.1978.tb00062.x

Subkoviak, M.J. (1984). Estimating the Reliability of Mastery-Nonmastery Classifications. En Berk, R. A. (Ed.) A Guide to Criterion-Referenced Test Construction. The John Hopkins University Press. Baltimore. Pp. 267-291.

Subkoviak, M. J. (1988). A Practitioner's Guide to Computation and Interpretation of Reliability Indices for Mastery Tests. Journal of Educational Measurement, vol. 25, nº 1, pp. 47-55.

http://dx.doi.org/10.1111/j.1745-3984.1988.tb00290.x

Swaminathan, H., Hambleton, R. K. & Algina, J. (1974). Reliability of Criterion-Referenced Tests: A Decision-Theoretic Formulation. Journal of Educational Measurement. Vol. 11, nº. 1, Pp. 263-267.

http://dx.doi.org/10.1111/j.1745-3984.1974.tb00998.x

Downloads

Issue

Vol. 12 No. 1 (2006)

Section

Research Articles

License

The authors grant non-exclusive rights of exploitation of works published to RELIEVE and consent to be distributed under the Creative Commons Attribution-Noncommercial Use 4.0 International License (CC-BY-NC 4.0), which allows third parties to use the published material whenever the authorship of the work and the source of publication is mentioned, and it is used for non-commercial purposes. 

    The authors can reach other additional and independent contractual agreements, for the non-exclusive distribution of the version of the work published in this journal (for example, by including it in an institutional repository or publishing it in a book), as long as it is clearly stated that the Original source of publication is this magazine.  

    Authors are encouraged to disseminate their work after it has been published, through the internet (for example, in institutional archives online or on its website) which can generate interesting exchanges and increase work appointments. 

   The fact of sending your paper to RELIEVE implies that you accept these conditions.