AI Literacy in Higher Education: A Systematic Approach to Questionnaire Development and Validation

This paper presents the development, refinement, and validation of the Critical Artificial Intelligence Literacy Scale, an instrument designed to measure artificial intelligence literacy across four dimensions: knowledge-related, operational, critical, and ethical. The initial version of the questionnaire, based on a robust theoretical framework and expert consultation, included 40 items and was tested with 57 doctoral students. It demonstrated strong psychometric properties (comparative fit index = 0.946, Tucker-Lewis index = 0.92) but showed limitations such as item redundancy (α = 0.947) and low performance of general items. To address these issues, the questionnaire was refined to a concise 24-item version. The revised instrument was evaluated using a sample of 314 first-year student teachers. Exploratory and confirmatory factor analyses confirmed a four-factor structure, with each dimension demonstrating strong reliability (Cronbach’s alpha ranging from 0.838 to 0.912) and excellent model fit indices (comparative fit index = 0.960, root mean square error of approximation = 0.0441). The results validate the Critical Artificial Intelligence Literacy Scale as a reliable and efficient tool for assessing artificial intelligence literacy in educational settings.