Evaluating Assessment. Validation with PLS-SEM of ATAE Scale for the Analysis of Assessment Tasks

Authors

  • María Soledad Ibarra-Sáiz UNESCO Chair on Evaluation and Assessment, Innovation and Excellence in Education. EVALfor Research Group. University of Cadiz, Spain. http://orcid.org/0000-0003-4513-702X
  • Gregorio Rodríguez-Gómez  UNESCO Chair on Evaluation and Assessment, Innovation and Excellence in Education. EVALfor Research Group. University of Cadiz, Spain. http://orcid.org/0000-0001-9337-1270

DOI:

https://doi.org/10.7203/relieve.26.1.17403

Keywords:

Assessment task, Assessment as learning, Empowerment, PLS-SEM, Partial Least Squares, Structural Equation Modeling, PLSpredict

Abstract

One of the essential functions of university teachers lies in the decision-making process regarding the various components included in assessment process design, where the quality of assessment tasks is a key aspect. This study presents both validation of an instrument for students to evaluate the assessment tasks and the model that upholds the relationships between constructs that characterise the assessment tasks. Working from a review of the literature, a theoretical model has been devised featuring the characteristics of the assessment tasks and the relationships between them. The Analysis of the Assessment and Learning Tasks questionnaire (ATAE) has been designed to check them, based on a formative measurement model. Using a cohort design, a total of 1,166 questionnaires were obtained, completed by students from the Business Administration and Management (BAM) and Finance and Accounting (F&A) degree courses. The measurement model and the structural model were evaluated by means of the Partial Least Squares Structural Equation Modeling (PLS-SEM) technique using SmartPLS_3 software. The results show no collinearity problems plus high levels of absolute and relative importance for each questionnaire item. From the students’ perception, it should be highlighted that the challenging aspect of an assessment task is related to transfer of learning, and that this is measured by use of communication strategies and demonstration of in-depth understanding.

Author Biographies

María Soledad Ibarra-Sáiz, UNESCO Chair on Evaluation and Assessment, Innovation and Excellence in Education. EVALfor Research Group. University of Cadiz, Spain.

María Soledad Ibarra Sáiz.  Senior Lecturer in Educational Assessment and Evaluation at University of Cadiz. Director of UNESCO Chair on Evaluation and Assessment, Innovation and Excellence in Education. Director of EVALfor Research Group–SEJ509  Assessment & Evaluation in Training Contexts  from the Andalusian Programme on Research, Development and Innovation (PAIDI). She develops her research mainly in the field of assessment and evaluation in higher education. She has been principal researcher of more than 10 European, international and national projects, whose results have been published in various articles, book chapters and contributions to international conferences. She is currently the main co-researcher of the FLOASS Project - Learning outcomes and learning analytics in higher education: An action framework from sustainable assessment (RTI2018-093630-B-I00) in which 6 Spanish universities participate.

Gregorio Rodríguez-Gómez,  UNESCO Chair on Evaluation and Assessment, Innovation and Excellence in Education. EVALfor Research Group. University of Cadiz, Spain.

Gregorio Rodríguez Gómez.  Professor of Educational Research Methods at the University of Cadiz. He is the coordinator of the strategic area "Studies and research in assessment and evaluation" of the UNESCO Chair on Evaluation and Assessment, Innovation and Excellence in Education. Founding member of the EVALfor Research Group –SEJ509  Assessment and evaluation in training contexts . His research interest is focused on research methods and assessment and evaluation in higher education. He is currently the main co-researcher of the FLOASS Project - Learning outcomes and learning analytics in higher education: A framework for action from sustainable assessment (RTI2018-093630-B-I00). Author of articles, book chapters and contributions to international conferences. He has been President of the  Interuniversity Association for Pedagogical Research  (AIDIPE). He is currently President of the RED-U  Spanish University Teaching Network.

References

Ashford-Rowe, K., Herrington, J., & Brown, C. (2014). Establishing the critical elements that determine authentic assessment. Assessment & Evaluation in Higher Education, 39(2), 205–222. https://doi.org/10.1080/02602938.2013.819566

Ashwin, P., Boud, D., Coate, K., Hallet, F., Keane, E., Krause, K.-L., … Tooher, M. (2015). Reflective teaching in higher education. Bloomsbury. https://doi.org/10.1142/S0129183114500405

Bearman, M., Dawson, P., Bennett, S., Hall, M., Molloy, E., Boud, D., & Joughin, G. (2017). How university teachers design assessments: a cross-disciplinary study. Higher Education, 74(1), 49–64. https://doi.org/10.1007/s10734-016-0027-7

Bearman, M., Dawson, P., Boud, D., Bennett, S., Hall, M., & Molloy, E. (2016). Support for assessment practice: developing the Assessment Design Decisions Framework. Teaching in Higher Education, 21(5), 545–556. https://doi.org/10.1080/13562517.2016.1160217

Bearman, M., Dawson, P., Boud, D., Hall, M., Bennett, S., Molloy, E., & Joughin, G. (2014). Guide to the Assessment Design Decisions Framework. Retrieved from http://www.assessmentdecisions.org/guide/

Biggs, J., & Tang, C. (2011). Teaching for quality learning at university. What the students does (4th ed.). McGraw-Hill-SRHE & Open University Press.

Boud, D. (2014). Shifting views of assessment: from secret teachers’ business to sustaining learning. In C. Kreber, C. Anderson, N. Entwistle, & J. McArthut (Eds.), Advances and inovations in university assessment and feedback (pp. 13–31). Edinburgh University Press Ltd. https://doi.org/10.3366/edinburgh/9780748694549.003.0002

Boud, D. (2020, en prensa). Challenges in reforming higher education assessment: a perspective from afar. RELIEVE - Electronic Journal of Educational Research, Assessment and Evaluation.

Cepeda-Carrión, G., Nitzl, C., & Roldán, J. L. (2017). Mediation Analyses in Partial Least Squares Structural Equation Modeling: Guidelines and Empirical Examples. In H. Latan & R. Noonan (Eds.), Partial Least Squares Path Modeling: Basic concepts, methodological issues and applications (pp. 173–195). Springer. https://doi.org/10.1007/978-3-319-64069-3

Chin, W. W. (1998). The partial least squares approach to structural equation modeling. In G. A. Marcoulides (Ed.), Modern methods for business research (pp. 295–398). Lawrence Erlbaum.

Cubero-Ibáñez, J., & Ponce-González, N. (2020). Aprendiendo a través de tareas de evaluación auténticas: Percepción de estudiantes de Grado en Educación Infantil. Revista Iberoamericana de Evaluación Educativa, 13(1), 41–69. https://doi.org/10.15366/riee2020.13.1.002

Dawson, P., Bearman, M., Boud, D. J., Hall, M., Molloy, E. K., Bennett, S., & Joughin, G. (2013). Assessment Might Dictate the Curriculum, But What Dictates Assessment? Teaching & Learning Inquiry: The ISSOTL Journal, 1(1), 107–111. https://doi.org/10.2979/teachlearninqu.1.1.107

Dochy, F. (2009). The Edumetric Quality of New Modes of Assessment: Some Issues and Prospect. In G. Joughin (Ed.), Assessment, Learning and Judgement in Higher Education (pp. 85–114). Springer. https://doi.org/10.1007/978-1-4020-8905-3_6

Dochy, F., & Gijbels, D. (2006). New learning, assessment engineering and edumetrics. In L. Verschaffel, F. Dochy, M. Boekaerts, & S. Vosniadou (Eds.), Instructional Psychology: Past, present and future trends. Sixteen essays in honour of Erik De Corte. Elsevier.

Entwistle, N., & Karagiannopoulou, E. (2014). Perceptions of Assessment and their Influences on Learning. In C. Kreber, C. Anderson, N. Entwistle, & J. McArthut (Eds.), Advances and innovations in university assessment and feedback (pp. 75–98). Edinburgh University Press Ltd. https://doi.org/10.3366/edinburgh/9780748694549.003.0005

Garson, G. D. (2016). Partial Least Squares: Regression & Structural Equation Models. Statistical Publishing Associates.

Glofcheski, R. (2017). Making Assessment for Learning Happen Through Assessment Task Design in the Law Curriculum. In D. Carless, S. M. Bridges, C. K. Y. Chan, & R. Glofcheski (Eds.), Scaling up Assessment for Learning in Higher Education (pp. 67–80). Singapore. https://doi.org/10.1007/978-981-10-3045-1_5

Gore, J., Ladwig, J., Eslworth, W., & Ellis, H. (2009). Quality assessment framework: A guide for assessment practice in higher education. Callaghan, NSW Australia: The Australian Learning and Teaching Council. The University of Newcastle. Retrieved from http://www.olt.gov.au/system/files/resources/QAF FINAL doc for print.pdf

Gulikers, J. T. M., Bastiaens, T. J., Kischner, P. A., & Kester, L. (2006). Relations between student perception of assessment authenticity, study approaches and learning outcomes. Studies in Educational Evaluation, 32, 381–400. https://doi.org/10.1016/j.stueduc.2006.10.003" target="_blank">https://doi.org/10.1016/j.stueduc.2006.10.003

Gulikers, J. T. M., Bastiaens, T., & Kirschner, P. A. (2004). A five-dimensional framework for authentic assessment. Educational Technology Research and Development, 52(3), 67–85. https://doi.org/10.1007/BF02504676

Hair, J. F., Hult, G. T. M., Ringle, C. M., & Sarstedt, M. (2017). A Primer on Partial Least Squares Structural Equation Modeling (PLS-SEM) (2nd ed.). Sage.

Hair, J. F., Risher, J. J., Sarstedt, M., & Ringle, C. M. (2019). When to use and how to report the results of PLS-SEM. European Business Review, 31(1), 2–24. https://doi.org/10.1108/EBR-11-2018-0203

Hair, J. F., Sarstedt, M., Ringle, C. M., & Gudergan, S. P. (2018). Advanced Issues in Partial Least Squares Structural Equation Modeling. Sage.

Herrington, J., & Herrington, A. (2006). Authentic conditions for authentic assessment: Aligning task and assessment. Proceedings of the 29th HERDSA Annual Conference, 146–151.

Horst, S. J., & Pyburn, E. M. (2018). Likert scaling. In B. B. Frey (Ed.), The SAGE encyclopedia of educational research, measurement, and evaluation (pp. 974–976). Sage.

Ibarra-Sáiz, M. S. y Rodríguez-Gómez, G. (2019). Una evaluación como aprendizaje. En J. Paricio, A. Fernández e I. Fernández (Eds.), Cartografía de la buena docencia. Un marco para el desarrollo del profesorado basado en la investigación (pp. 175–196). Narcea.

Ibarra-Sáiz, M. S., Rodríguez-Gómez, G., & Boud, D. (2020). Developing student competence through peer assessment: the role of feedback, self-regulation and evaluative judgement. Higher Education. https://doi.org/10.1007/s10734-019-00469-2

Jiménez-Cortés, R. (2019). Aprendizaje de las mujeres en las redes sociales: Validación de la escala MAIA con PLS. Revista de Investigación Educativa, 37(2), 431–449. https://doi.org/http://dx.doi.org/10.6018/rie.37.2.325721

Johnson, R. L., & Morgan, G. B. (2016). Survey scales. A guide to development, analysis, and reporting. The Guilford Press.

Muñoz-Cantero, J.-M., Rebollo-Quintela, N., Mosteiro-García, J., & Ocampo-Gómez, C.-I. (2019). Validación del cuestionario de atribuciones para la detección de coincidencias en trabajos académicos. RELIEVE - Revista Electrónica de Investigación y Evaluación Educativa, 25(1), 1–16. https://doi.org/10.7203/relieve.25.1.13599

Nitzl, C., Roldán, J. L., & Cepeda, G. (2016). Mediation Analysis in Partial Least Squares Path Modeling: Helping researchers discuss more sophisticated models. Industrial Management & Data Systems, 116(9), 1849–1864. https://doi.org/10.1108/IMDS-07-2015-0302

O’Donovan, B. (2016). How student beliefs about knowledge and knowing influence their satisfaction with assessment and feedback. Higher Education, 74(4), 617–633. https://doi.org/10.1007/s10734-016-0068-y

OECD. (2013). Methodological considerations in the measurement of subjective well-being. In OECD Guidelines on measuring subjective well-being (pp. 61–138). Paris: OECD Publishing. https://doi.org/10.1787/9789264191655-en

Panadero, E., Andrade, H., & Brookhart, S. (2018). Fusing self-regulated learning and formative assessment: a roadmap of where we are, how we got here, and where we are going. The Australian Educational Researcher, 45(1), 13–31. https://doi.org/10.1007/s13384-018-0258-y

Panadero, E., Klug, J., & Järvelä, S. (2016). Third wave of measurement in the self-regulated learning field: when measurement and intervention come hand in hand. Scandinavian Journal of Educational Research, 60(6), 723–735. https://doi.org/10.1080/00313831.2015.1066436

Pereira, D., Niklasson, L., & Flores, M. A. (2017). Students’ perceptions of assessment: a comparative analysis between Portugal and Sweden. Higher Education, 73(1), 153–173. https://doi.org/10.1007/s10734-016-0005-0

Ringle, C. M., Wende, S., & Becker, J.-M. (2015). SmartPLS 3. Bönningstedt: SmartPLS. Retrieved from http://www.smartpls.com

Rodríguez-Gómez, G., & Ibarra-Sáiz, M. S. (2015). Assessment as learning and empowerment: Towards sustainable learning in higher education. In M. Peris-Ortiz & J. M. Merigó Lindahl (Eds.), Sustainable learning in higher education. Developing competencies for the global marketplace (pp. 1–20). Springer. https://doi.org/10.1007/978-3-319-10804-9_1

Roldán, J. L., & Sánchez-Franco, M. J. (2012). Variance-Based Structural Equation Modeling: Guidelines for Using Partial Least Squares in Information Systems Research. In Research Methodologies, Innovations and Philosophies in Software Systems Engineering and Information Systems (pp. 193–221). IGI Global. https://doi.org/10.4018/978-1-4666-0179-6.ch010

Sadler, D. R. (2016). Three in-course assessment reforms to improve higher education learning outcomes. Assessment & Evaluation in Higher Education, 41(7), 1081–1099. https://doi.org/10.1080/02602938.2015.1064858

Sambell, K., McDowell, L., & Montgomery, C. (2013). Assessment for Learning in Higher Education.   Routledge.

Sharma, P. N., Shmueli, G., Sarstedt, M., Danks, N., & Ray, S. (2018). Prediction-Oriented Model Selection in Partial Least Squares Path Modeling. Decision Sciences. https://doi.org/10.1111/deci.12329

Shmueli, G., Ray, S., Velasquez Estrada, J. M., & Chatla, S. B. (2016). The elephant in the room: Predictive performance of PLS models. Journal of Business Research, 69(10), 4552–4564. https://doi.org/10.1016/j.jbusres.2016.03.049

Shmueli, G., Sarstedt, M., Hair, J. F., Cheah, J. H., Ting, H., Vaithilingam, S., & Ringle, C. M. (2019). Predictive model assessment in PLS-SEM: guidelines for using PLSpredict. European Journal of Marketing, 53(11), 2322–2347. https://doi.org/10.1108/EJM-02-2019-0189

Smith, J. K., & Smith, L. F. (2014). Developing Assessment Tasks. In C. Wyatt-Smith, V. Klenowski, & P. Colbert (Eds.), Designing Assessment for Quality Learning (pp. 123–136). Springer. https://doi.org/10.1007/978-94-007-5902-2

Strijbos, J., Engels, N., & Struyven, K. (2015). Criteria and standards of generic competences at bachelor degree level: A review study. Educational Research Review, 14, 18–32. https://doi.org/10.1016/j.edurev.2015.01.001

Tai, J., Ajjawi, R., Boud, D., Dawson, P., & Panadero, E. (2018). Developing evaluative judgement: enabling students to make decisions about the quality of work. Higher Education, 76(3), 467–481. https://doi.org/10.1007/s10734-017-0220-3

Thomas, T., Jacobs, D., Hurley, L., Martin, J., Maslyuk, S., Lyall, M., & Ryan, M. (2019). Students’ perspectives of early assessment tasks in their first-year at university. Assessment & Evaluation in Higher Education, 44(3), 398–414. https://doi.org/10.1080/02602938.2018.1513992

Wren, J., Sparrow, H., Northcote, M., & Sharp, S. (2009). Higher Education Students’ Perceptions of Effective Assessment. International Journal of Learning, 15(12), 11–23. https://doi.org/10.18848/1447-9494/CGP/v15i12/46037

Zhao, X., Lynch, J. G., & Chen, Q. (2010). Reconsidering Baron and Kenny: Myths and truths about mediation analysis. Journal of Consumer Research, 37(2), 197–206. https://doi.org/10.1086/651257

Downloads

Published

2020-10-20

Issue

Vol. 26 No. 1 (2020): Monográfico: Evaluación en la Educación Superior

Section

Special Section

License

The authors grant non-exclusive rights of exploitation of works published to RELIEVE and consent to be distributed under the Creative Commons Attribution-Noncommercial Use 4.0 International License (CC-BY-NC 4.0), which allows third parties to use the published material whenever the authorship of the work and the source of publication is mentioned, and it is used for non-commercial purposes. 

    The authors can reach other additional and independent contractual agreements, for the non-exclusive distribution of the version of the work published in this journal (for example, by including it in an institutional repository or publishing it in a book), as long as it is clearly stated that the Original source of publication is this magazine.  

    Authors are encouraged to disseminate their work after it has been published, through the internet (for example, in institutional archives online or on its website) which can generate interesting exchanges and increase work appointments. 

   The fact of sending your paper to RELIEVE implies that you accept these conditions.