Psychometric Evaluation of A Creative Thinking Performance Test for Science Education

Participants

The sample was selected using purposive sampling, a technique based on certain criteria relevant to the research objectives (Creswell, 2012). Purposive sampling was chosen because it allows researchers to obtain participants per the research context, such as academic background, university of origin, and semester level, so the data obtained can be more in-depth and representative according to research needs (Creswell, 2012). The study involved 138 students as participants.

According to gender ( Table 1), there were 14 male students (10.14%) and 124 female students (89.86%). Based on university affiliation, 63 students came from University A (45.65%) and 75 from University B (54.35%). Regarding the semester, 46 students were in semester 2 (33.33%), 47 students in semester 4 (34.06%), and 45 students in semester 6 (32.61%). The distribution of domicile location is almost balanced, with 65 students from urban areas (47.10%) and 73 students from rural areas (52.90%). Participation in campus organizations indicated that 60 students were actively involved (43.48%), while 78 students did not participate in organizations (56.52%). Based on academic achievement, most students have a GPA of 3.1–4.0 (101 students, 73.19%), while 37 students (26.81%) are in the GPA range of 2.1–3.0, and no one has a GPA below 2.0.

Theoretical construction

The literature review identified four dimensions of creative thinking skills that are relevant to explore in the context of science learning (Batlolona et al., 2019; Haim & Aschauer, 2024; Kholid et al., 2024; Suradika et al., 2023; Torrance et al., 1992). The study refers to Torrance’s classic framework to formulate an adequate construct, strengthening it with the results of recent research emphasizing the context of science education. The four dimensions (see Table 2) are positioned as the main factors, which are sensitivity, which refers to the ability of students to be sensitive in detecting problems and generating adaptive ideas; flexibility, which refers to the skill of generating varied ideas from various perspectives and categories; novelty, which emphasizes the ability to create unique and original ideas in offering new solutions; and elaboration, which describes the ability to expand and develop ideas into more detail and quality. The definition of the four dimensions becomes the basis for the operationalization of the instrument, where each dimension will be derived into indicators and items that represent students’ creative thinking performance in science.

Blueprint construction

The blueprint of the creative thinking skills instrument was developed based on four main dimensions determined through theory synthesis 1: 1) Sensitivity, 2) Flexibility, 3) Novelty, and 4) Elaboration. The research referred to a variety of sources to identify relevant concepts for each dimension, including classic literature on creativity studies (e.g., Torrance et al. (1992)), recent academic publications in reputable journals (e.g., Batlolona (2020)), and research reports focusing on the context of science education (e.g., Ernawati et al. (2023)). The decision to combine classic and contemporary sources was intended to ensure that the instrument’s construction was grounded in basic theories of creativity and relevant to recent developments in science learning. The blueprint development process was carried out through three stages, which are: 1: first, researchers independently reviewed relevant documents and articles to record indicators of creative thinking skills in each dimension; second, the results of the review were compiled and grouped so that more concise and representative indicators were obtained; and third, the indicators that had been compiled were then validated by a panel of experts in the field of science education and instrument development to ensure the representativeness of the concept and suitability for the research context.

Item construction

Constructing the items began by aligning each item with the concept of creative thinking skills formulated in the blueprint (see Table 2). The alignment was done to ensure coverage of the four dimensions: sensitivity, flexibility, novelty, and elaboration. Two items represented each dimension, so there were eight essay items. The essay format was chosen because it allows students to express ideas freely, display originality, and provide a more in-depth description of the cognitive processes underlying creative thinking skills in science (Kartini et al., 2021; Mafinejad et al., 2017). The design stage developed each item to encourage students to provide argumentative, detailed, and contextual answers according to the science problems presented. The initial draft of the items underwent several revisions to enhance clarity of wording, appropriateness of context, and level of cognitive demands. Two researchers independently reviewed each item to minimize ambiguity and ensure alignment with indicators in each dimension. Furthermore, an expert panel consisting of science education and educational assessment experts reviewed each item to provide input related to clarity, relevance, and conformity to theoretical constructs. Joint discussions were held until consensus was reached on the necessary modifications.

Scoring

The scoring guidelines in this instrument were developed based on a performance rubric approach that refers to Torrance et al. (1992) creativity theory as well as developments in current research in science education (e.g., Batlolona (2020)). The theory emphasizes that creative thinking skills are not only seen from the number of ideas, but also the quality of the ideas produced, including relevance, realism, uniqueness, and level of elaboration. Therefore, scoring was done in the range of 0–4, with the criteria: score 4 for ideas that are relevant, realistic, contextual, and expressed clearly and completely; score 3 for ideas that are adaptive, realistic, and contextual, but less clear or less complete; score 2 for ideas that are adaptive but less realistic; score 1 for ideas that are not adaptive to the problem; and score 0 if no ideas are given.

Delphi validation

Validation of the instrument items in the current study was conducted using the Delphi technique, which is a method that involves a panel of experts to obtain a consensus of judgment through a systematic and structured process (Linstone et al., 2002). Delphi validation is needed to ensure that the instrument items are not only theoretically valid but also in accordance with the substantive context being measured, thus increasing the instrument’s content validity (Aiken, 1985). The validation results suggested that all items obtained an Aiken index between 0.93–0.96 with a V table value 0.74. Because all index values exceeded the critical value, each item was declared substantially valid (Aiken, 1980). Therefore, the eight essay items developed have met the criteria of content suitability based on expert consensus and are suitable for the next stage of instrument testing.

Pilot test

Conducting a pilot test is an important stage before the instrument is widely used, because it ensures that respondents can understand the items well, have clarity of wording, and can represent the abilities to be measured. Patel & Patel, (2019) state that pilot testing helps researchers identify instrument weaknesses in terms of language, substance, and technicality, while Creswell, (2012) emphasizes the role of pilot tests in providing an initial overview of instrument reliability and validity. The pilot test conducted on 35 students showed that all items could be answered well and did not cause significant confusion. However, some suggestions regarding the wording of certain items needed to be simplified to make them clearer. The results suggest that the instrument is generally feasible to use, but still requires minor revisions to the linguistic aspects to be more optimal in the main research.

Data analysis

Data analysis was conducted through several stages, starting with Exploratory Factor Analysis (EFA) to explore the factor structure, considering factor loading ≥0.40 as the minimum limit (Hair et al., 2019). Furthermore, Confirmatory Factor Analysis (CFA) was used to test the model fit, with cut-off criteria such as CFI and TLI ≥ 0.90 (Kline, 2015), RMSEA ≤0.08, and χ²/df ≤ 3 (Lin & Tsau, 2013). Discriminant validity was tested using the Fornell-Larcker approach, where the AVE value must be greater than the squared correlation between constructs (Kline, 2015), while criterion validity was determined from the presence of significant correlations with relevant external measures (Shahzad et al., 2025). The Rasch Measurement Model was used to check item quality with the criteria of item fit on infit and outfit MNSQ 0.5–1.5 (Andrich & Marais, 2019), item and person reliability ≥0.70 (Cronbach, 1951), and person-item distribution analysis to see the balance of item difficulty levels with respondents’ abilities.

Descriptive results

According to the results of descriptive analysis, there are variations in creative thinking skills scores in the participant categories (see Table 3). Gender-wise, female students (N = 124) exhibited higher scores than males (N = 14), for example in sensitivity (M = 64.97, SD = 11.91 vs M = 56.69, SD = 15.12) and flexibility (M = 67.78, SD = 10.45 vs M = 60.13, SD = 8.73). By university, students from University A (N = 63) scored higher on flexibility (M = 65.16, SD = 8.49) than University B (M = 54.33, SD = 13.10), while University B was superior on elaboration (M = 65.82, SD = 14.88). Based on semester, semester 2 students (N = 46) stood out in sensitivity (M = 71.55, SD = 9.64) and novelty (M = 61.99, SD = 9.37), while semester 6 students (N = 45) were relatively higher in flexibility (M = 63.54, SD = 12.07). Within the location category, urban students (N = 65) scored better on almost all aspects, such as sensitivity (M = 63.79, SD = 10.48) and elaboration (M = 59.93, SD = 10.15), compared to rural students (M = 62.04, SD = 11.65; M = 57.24, SD = 9.44). Involvement in campus organizations also has an effect, where students who are active in organizations (N = 60) are higher in flexibility (M = 66.23, SD = 10.12) than those who are not active (M = 60.47, SD = 11.28). Lastly, based on GPA, the 3.1–4.0 group (N = 101) performed better on novelty (M = 55.87, SD = 10.53) and elaboration (M = 59.84, SD = 9.91) than the 2.1–3.0 GPA group (M = 52.75, SD = 9.80; M = 57.36, SD = 10.44). The findings suggest that demographic and academic factors have different roles in influencing variations in students’ creative thinking skills.

Exploratory Factor Analysis (EFA)

The Exploratory Factor Analysis (EFA) results suggested that the instrument was worthy of further analysis and in accordance with the theoretical construction. The KMO value of 0.821 indicated an excellent level of sample feasibility, while Bartlett’s Test of Sphericity yielded Chi-Square = 459.593, df = 28, and p = 0.000, indicating that the correlation matrix was significant and the data met the assumptions for factor analysis (see Table 4 and Figure 1). According to the extraction results, four main factors with an eigenvalue of more than 1 cumulatively explained 82.24% of the total variance. The first factor explained 28.77%, the second factor 25.40%, the third factor 14.96%, and the fourth factor 13.11% of the overall variance after rotation. The component matrix shows that each instrument item has a loading factor above 0.70 on its respective factor, which indicates the consistency and representativeness of the item. Consistent with the theoretical framework, the first factor is interpreted as Sensitivity (Item_1 and Item_2), the second factor as Flexibility (Item_3 and Item_4), the third factor as Novelty (Item_5 and Item_6), and the fourth factor as Elaboration (Item_7 and Item_8). Consequently, the EFA results suggest that the empirical structure of the instrument supports the four dimensions of creative thinking skills that have been theoretically established. Hence, the instrument has good initial construct validity.

Figure 1. Exploratory factor analysis results on test instrument.

Confirmatory Factor Analysis (CFA)

Confirmatory Factor Analysis (CFA) results (see Table 5) suggest that the four-dimensional model of creative thinking skills has an excellent fit with the data. The value of Chi-Square/df = 1.87 is below the threshold of <3.0, which indicates a good fit. Other indices also supported the model fit, including RMSEA = 0.056 (< 0.08), SRMR = 0.041 (< 0.08), CFI = 0.954 (> 0.90), TLI = 0.942 (> 0.90), NFI = 0.918 (> 0.90), GFI = 0.931 (> 0.90), and AGFI = 0.905 (> 0.90). Each index is within the recommended cut-off criteria, indicating that the four-dimensional construct model-Sensitivity, Flexibility, Novelty, and Elaboration-empirically supports the established theoretical structure. Therefore, the CFA results confirmed that the instrument has good construct validity and can be used to measure students’ creative thinking skills performance reliably.

Discriminant validity

The results of the discriminant validity analysis using the Fornell-Larcker criterion show that each dimension of creative thinking skills has good discrimination ability. The Average Variance Extracted (AVE) value in each dimension is 0.78 for Sensitivity, 0.81 for Flexibility, 0.76 for Novelty, and 0.79 for Elaboration (see Table 6). Some of the AVE values are greater than the squared correlation between factors, for example, the correlation between Sensitivity and Flexibility is 0.54, Sensitivity and Novelty is 0.49, and Sensitivity and Elaboration is 0.52. It indicates that each dimension explains more of its own item variance than the variance explained by other dimensions, so that each construct can be distinguished theoretically and empirically. Therefore, the instrument has sufficient discriminant validity, ensuring that the four dimensions are distinct yet conceptually related constructs.

Criterion validity

Criterion validity results suggest significant relationships between several demographic variables and the dimensions of students’ creative thinking skills (see Table 7). Location variable had the most consistent and significant effect on all dimensions, with β = 0.28 (p = 0.004) on sensitivity, β = 0.25 (p = 0.007) on flexibility, β = 0.22 (p = 0.012) on novelty, and β = 0.24 (p = 0.008) on elaboration, indicating that students from urban areas tend to have higher creative thinking scores than students from rural areas. The semester variable also significantly affects all dimensions, for example, β = 0.21 (p = 0.022) on sensitivity and β = 0.20 (p = 0.029) on elaboration, indicating that increasing academic experience with each semester contributes to creative thinking ability. Moreover, GPA displayed significant effects on flexibility (β = 0.20, p = 0.034), novelty (β = 0.19, p = 0.039), and elaboration (β = 0.18, p = 0.041), indicating a positive relationship between academic achievement and the quality of ideas generated by students. The variables gender, university, and campus organisation participation display a more limited effect, with some p-values close to significant (e.g. gender on sensitivity β = 0.18, p = 0.041; university on flexibility β = 0.17, p = 0.038). Overall, the results confirm that the instrument can reflect differences in creative thinking ability related to students’ demographic and academic characteristics, thus supporting the criterion validity of the instrument.

Model fit

Rasch Measurement results demonstrate that the creative thinking skills instrument has good measurement quality at the person and item levels. The mean score for the person was 20.2 with a standard deviation of 4.6, a score range of 6 to 30, and a mean measure of 0.98 with a standard error of 0.55 (see Figure 2). The MNSQ infit and MNSQ outfit values averaged 1.00 each, indicating a good fit of the data to the Rasch model. In contrast, the person reliability = 0.84 and separation = 2.27 values indicated the instrument’s ability to differentiate the levels of students’ creative thinking skills adequately. RAW SCORE-TO-MEASURE CORRELATION reached 0.98, confirming the consistency of measurement.

Figure 2. Person fit model analysis results.

Among the items, the mean score was 352.4 with a standard deviation of 31.2, and the mean measure was 0.00 with a standard error of 0.13 (see Figure 3). The range of item sizes was between −0.90 to 0.70, and the MNSQ infit and outfit values averaged 1.00 each, indicating that all items fit the Rasch model. Item reliability = 0.94 and separation = 4.03 indicated that the instrument could distinguish the difficulty level of each item well. Overall, the Rasch results indicate that this eight-item essay instrument is internally valid, reliable, and has an adequate balance between item difficulty and student ability, making it feasible to measure creative thinking skills performance in the target population.

Figure 3. Results of person fit model.

Internal consistency

The results of the instrument’s internal consistency indicate a good reliability level in measuring students’ creative thinking skills. The correlation between the raw score and the measure reached 0.98, indicating a strong relationship between the students’ scores and the measured construct. In addition, the Cronbach’s Alpha (KR-20) value of 0.84 showed high internal reliability, indicating that the eight essay items have sufficient internal consistency and can be trusted to assess overall creative thinking performance. These results support using the instrument in the main study, as it provided stable and consistent measurements across participants.

Item characteristic curves

The analysis results of item 1.1SS (see Figure 4), which measures students’ ability to formulate adaptive solutions based on local potential related to the electrical energy crisis in eastern Indonesia, show that scores are concentrated in the medium to high category. Seventy-one students (51%) scored 3 with an average ability of +1.15 logit, indicating their ability to generate adaptive and contextual ideas is quite good. Thirty-five students (25%) were at a score of 2 with an ability of +0.40 logits, indicating a basic understanding but still need to develop ideas to be more realistic. Twenty-two students (16%) achieved the maximum score of 4 with +3.29 logit, demonstrating full mastery in designing creative and contextual solutions according to local potential. However, only a small number of students, namely 6 people (4%) at score 1 with −1.44 logit ability and 4 people (3%) at score 0 with −1.55 logit ability, failed to show adequate understanding of the concept of alternative energy and utilisation of local resources.

Figure 4. Example of student response patterns on item 1.

The Item Characteristic Curve (ICC) visually demonstrates that the expectation curve of the Rasch model (red line) is in line with the empirical data pattern (black-blue dots), with most of the dots falling within the 95% confidence interval. A good fit of the model is indicated, although there is a slight deviation in the low to medium ability range (around −2 to 0 logits). Therefore, item 1.1SS proved empirically valid, has sufficient discrimination power, and effectively assesses students’ ability to design adaptive solutions based on science and local potential, distinguishing low, medium, and high ability students.

Following the analysis of students’ ability to formulate local potential-based adaptive solutions related to the energy crisis in item 1.1SS, the next step is to evaluate their ability to design more specific innovative solutions in the context of science and technology. Item 5.5NY emphasises the application of creative thinking skills to produce innovative ideas in the form of simple tools that can convert kitchen waste into energy, so that it can illustrate the extent to which students can integrate the concepts of science, creativity, and local contexts in a more practical and applicable manner.

The analysis results of item 5.5NY (see Figure 5), which measures students’ ability to design innovative ideas for simple tools to convert kitchen waste into energy, indicate that most students are in the middle ability category. A total of 45% of students obtained a score of 2 with an average ability of 0.93 logits, indicating an initial ability to generate adaptive ideas, but not fully realistic or detailed. Meanwhile, 34% of students obtained a score of 3 with an average ability of 1.50 logits, reflecting a more mature ability to develop contextualised innovative solutions to household organic waste problems. Only 7% of students achieved the maximum score of 4 with an ability of 4.75 logits, showing full mastery in designing creative, functional and science-based tools. Students with the lowest score of 0 were only 1%, indicating a small proportion who could not generate ideas related to renewable energy from waste.

Figure 5. Example of student answer response patterns on item 5.

The pattern is in line with the Category Probability Curve (CPC), where category 2 has the highest probability at ability around 0–1 logits, category 3 is dominant at ability 1–3 logits, and category 4 only appears at ability above 3 logits, consistent with the low proportion of students who reach the maximum score. The Item Characteristic Curve (ICC) also indicates that students’ expected scores follow the Rasch model well, although there is a slight deviation in the middle to high ability range (1–3 logits). Therefore, item 5.5NY can be valid and reliable, effective in assessing students’ ability to integrate creativity, science, and local context to produce innovative energy-based solutions from household waste. However, its discrimination capacity at highly proficient (>4 logits) is still limited.

Person-item histograms

The person-item map results demonstrate the distribution of respondents’ ability and item difficulty on a single logit scale. The person part (above) shows that the majority of respondents are distributed around logit 0 to +2, with a peak frequency of about 20–22 respondents at logit 0 (green colour) and about 20 respondents at logit +2 (red colour). It indicates that most of the respondents have moderate to above-average ability. There are still some respondents with low ability, indicated by about 2–3 respondents at logits −3 to −4, but the number is relatively small compared to the moderate ability group.

The items (below) are all clustered around logits 0 to +1, with no items that are either extremely difficult (logits > +2) or extremely easy (logits < −2) (see Figure 6). The items tended to be of medium difficulty, so they were reasonably well balanced with the average ability of the participants. Taken together, the distribution shows that the instrument is adequate in measuring the skills of respondents with moderate to high ability. However, it is less able to distinguish between respondents with very low or very high ability due to the limited variation in item difficulty.

Figure 6. Person-item histograms of person and item results on the test instrument.

Measurement of creative thinking subscale on the topic of renewable energy

Analysis of the creative thinking skills subscale on Renewable Energy indicated that the four dimensions of the instrument had varying measures with low standard errors, indicating stable and reliable estimates. The Sensitivity dimension has a measure of −0.515 with a standard error of 0.14, INFIT MNSQ of 0.875 (ZSTD -1) and OUTFIT MNSQ of 0.87 (ZSTD -1.1), and point-measure correlation of 0.695 (see Table 8), indicating that students are quite sensitive in identifying science problems related to renewable energy and can generate ideas that are adaptive and relevant to the local context. Flexibility dimension (measure −0.465; SE 0.135; INFIT MNSQ 1.23; OUTFIT MNSQ 1.25; point-measure correlation 0.615) signifies students’ ability to generate diverse ideas from various perspectives, for example, considering various alternative energy sources or innovative ways to utilise waste into energy, thus demonstrating divergent thinking skills that are important in problem-based science learning.

The Novelty dimension (measure 0.42; SE 0.13; INFIT MNSQ 0.86; OUTFIT MNSQ 0.845; point-measure correlation 0.72) emphasises students’ ability to create original and unique ideas, such as designing a simple device to convert kitchen waste into energy, reflecting scientific creativity in the context of science experiments. The Elaboration dimension (measure 0.56; SE 0.13; INFIT MNSQ 1.03; OUTFIT MNSQ 1.035; point-measure correlation 0.67) indicates students’ ability to develop ideas in detail and systematically, for example, designing a complete renewable energy utilisation procedure, tool, or strategy, which is relevant to critical and analytical thinking competencies in science learning. Based on the overall validity and reliability of all subscales, with INFIT and OUTFIT MNSQ within the range of 0.5–1.5 and point-measure correlation >0.6, the instrument is effective in differentiating students’ ability to think creatively and apply science concepts on the topic of renewable energy.

Limitations and future directions

The main limitation of the study rests on the scope and context of the instrument developed. The CTPT instrument has only been validated on elementary school teacher education program students, so the generalisation of the results is still limited to this group. The instrument has not been tested at other levels of education, such as secondary schools or non-primary teacher education higher education. It has not been used in international contexts with different learner characteristics and cultural backgrounds. Furthermore, the items in the CTPT are still dominated by science-based contexts, so their applicability is relatively limited when used in other fields that require creativity, such as arts, technology or social sciences. To strengthen the external validity and ensure the instrument’s flexibility, further research needs to be directed at expanding the trials across educational levels, scientific fields, and cultures so that the CTPT can become a more universal and adaptive instrument in measuring creative thinking skills.

Besides limitations in scope and context, this study also has methodological and technical limitations. The psychometric analysis is still limited to using Rasch models and SEM, so it does not include other, more comprehensive analytical methods to enrich validity evidence. The instrument also needs to be updated regularly as theories and conceptual frameworks regarding creative thinking develop to remain relevant to research and educational practice needs. Future research directions include the integration of longitudinal tracking to monitor the development of creativity over time and the use of learning analytics to link test scores with students’ actual performance in completing science-based and cross-cutting creative tasks.