Keywords
Validations, Strategy, Research instrument, Factor Analysis, Brazil.
Scientific investigations utilizing the questionnaire methodology for data acquisition mandate a rigorous validation process for their research instruments. This imperative arises from the need to mitigate biases and uphold impartiality in the derived results. Regrettably, a lack of awareness persists among researchers regarding this pivotal requirement within the methodological framework. Although comprehensive validation typically unfolds across multiple stages, the present study is dedicated exclusively to the structural validation phase.
A questionnaire with 18 closed questions (observable variables) was administered across 12 units in different regions of Brazil. Data were classified according to a Likert scale, which ranges from 1 to 5, representing the participants’ degree of agreement in relation to 8 characteristics proposed for the model (latent variables). The research involved 75 participants who answered the research instrument between March and August 2023. To define the structure of the model, the data were subjected to an exploratory factor analysis, with the support of the Factor Analysis software.
Findings suggested a structure was organized into two factors, called internal and external factors, according to the degree of governability that the SPH has in relation to the proposed items.
The validation evidence indicates a structure composed of two factors for the better organization of the items of a strategy development model in Schools of Public Health, in Brazil.
Validations, Strategy, Research instrument, Factor Analysis, Brazil.
Scientific works that use the questionnaire technique to collect data must have their research instruments subjected to a validation process, even if similar instruments already exist in a foreign language. The search for validation evidence can contribute to the publication of the research in higher impact journals, although not all researchers are aware of this methodological process requirement, which can compromise the results found and, consequently, the entire research.1–4
The validation of a research instrument must at least go through the stages of the tripartite method or Content, Construct and Criteria (CCC), which is composed of three validation activities: a) item content; b) construct structure; c) external criteria.5 Content validation aims to evaluate the clarity, pertinence and relevance of the instrument’s items; of the construct, the structure of the instrument; and the external validation, the extent to which the constructed instrument is aligned with other instruments already developed for the same purpose. Item Response Theory (IRT) and consequential validation have recently been incorporated into the method validation, addressing the ethical criteria of the research.6,7
Initially, the content validation was carried out using the Content Validation Coefficient (CVC), one of the most used techniques in the literature, validating the contents of the instrument’s items proposed in this research that obtained a coefficient above 0.8.8,9 As the exploratory research carried out in the Web of Science database, Scopus and Scielo did not identify works published in the last five years that contained instruments ready to be used in strategy development models in schools of public health in Brazil, the external validation that compares the proposed instrument with others already consolidated was compromised; however, the difficulty in identifying ready-made instruments may suggest an innovative approach to the research. To obtain evidence of consequential validation, the instrument used was submitted for consideration and approval by the Research Ethics Committee (REC) of the proposing institution and co-participants, when necessary, which assessed, among other aspects, the risks that could impact the research participants. However, consequential validation is not within the scope of this research.
Therefore, this work’s scope is exclusively to evaluate the validation stage of the structure of the strategy development models in Schools of Public Health (SPH), in the context of contemporary Brazil. The objective of this study was to define the structure of the strategy development model aimed at Brazilian SPHs.
In order to achieve the proposed objective, it was initially necessary to define the characteristics of the proposed model, as well as define what an SPH is in a Brazilian context.
In Brazil, there is no single concept of what an SPH is in the literature.10 An example of the lack of clarity is the definition of “School of Government in Health”, which aims to train health managers, such as health secretaries. This concept ends up being confused with the definition of SPH.11 Despite such conceptual problems, the SPH have in common a role in training staff focused on the Brazilian Unified Health System (UHS).12 If, on the one hand, it implements the National Policy on Permanent Health Education, on the other, it also contributes to the development of this and other policies, at a regional level, relating to the topic of health education.13,14 The participation of the SPHs, whether developing or implementing public health policies, contributes to reinforcing the relevance of research.
Since every model is an abstract representation of reality, it needs to be defined through a series of characteristics. The search for these characteristics was carried out through exploratory research. From the theory of professional bureaucracy, which mainly contains typified institutions such as schools and hospitals, some proposed characteristics emerged, such as the democratic and participatory character.15 The other characteristics unfold from the basic theory, with the application of the snowball technique.16 The proposed characteristics of the SPH strategy development model were:
• Systemic: as SPHs implement the National Policy for Permanent Education in Health and contribute to the development of policies related to the topic of health education,13,14 it is assumed that schools must remain open to the environment, receiving and exerting influences. For Gandin, planning must be a technical process that produces a political process to contribute to social development,17 which is only possible when schools operate as an open system, receiving and generating environmental stimuli through the proposed model18;
• Prescriptive: as a survey carried out in 19 schools suggests that SPH managers do not have management training,10 it is assumed that the prescriptive model can help managers with strategy development. However, despite being prescriptive, the model also provides for a certain flexibility and adaptability19;
• Innovative: the consolidation of the contingency approach to administration, showing that appropriate models require context, has led to the assumption that the strategy, in public health schools, must be developed by models specific to the characteristics of a professional bureaucracy,20–22 associated with the difficulty of implementing strategies,23–26 suggests that top-down models consolidated from the 1980s onwards, with the works of Porter,27,28 for strategy development, mainly in industrial organizations, are not suitable for the SPHs.17 Furthermore, according to França et. al, SPHs are heavily involved in research, in addition to teaching, which requires models that can not only be innovative, but also capable of fostering innovation10;
• Pragmatic: Mintzberg, Ahlstrand and Lampel29 present criticisms of the Design school, such as the separation between strategy development and action. The separation between the elaboration and execution stages is observed in models that address only one of these variables at a time, but it is not suitable for complex models, which must include both stages at the same time. Thus, it is assumed that a model that predicts the strategy implementation phase can generate value for the SPHs;
• Dynamic: because planning is thought of in the long term, four or five years, it is assumed that the ability to admit strategies that emerge even after the end of the planning preparation phase, called “emerging strategies”, can be incorporated periodically, in smaller and more frequent cycles, which add up to the deliberate strategies, this is a characteristic that suggests adding value to the proposed model15,30;
• Complex: only a complex model is capable of being bureaucratic and prescriptive, in order to comply with the principle of public administration legality; and be at the same time innovative and dynamic, characteristics necessary for the development of research in these schools. In addition to containing apparently antagonistic characteristics, which nevertheless prove to be complementary, complexity is also suggested by Mintzberg, in professional bureaucracy theory.15
Thus, it was expected that the model could reflect the two initial characteristics, plus the six proposals, totaling eight characteristics.
In light of the lack of a unified approach in the literature for strategy development models in the context of Brazilian public health schools, an exploratory study was undertaken. The study design details the data collection instrument, the data collection technique, the participant selection strategy and the data analysis techniques, which are described below.
As the eight identified characteristics represent latent variables, which cannot be observed directly, there was a need to propose items or questions, which represented the observable variables. The characteristics and items or questions that were part of the instrument to evaluate models for developing strategies in SPH are presented in Table 1.
Following the proposal of the items, they were distributed to participants in the SPH study in Brazil between March and August 2023. A questionnaire was individually sent via email, presented as an invitation, consisting of 18 closed questions. The purpose was to assess the factorial load of the items and analyze their structural organization into distinct factors.
The data were classified according to a Likert scale,32 which ranges from 1 to 5, representing the participants’ degree of agreement. Therefore, the closer to 5, the greater the participant’s agreement in relation to the proposed variables or items.
The content of the survey instrument was first validated using the CVC, which recommends including experts and users as participants. A similar approach to the previous technique was used to validate the structure. Participants were selected based on two requirements. Firstly, they had to be experts in a school of public health, holding the position of director in the school. Secondly, they had to be potential users of the proposed model for the school’s strategic planning. In addition, they all had to work in an SPH.
To choose the schools that participated in the research, the following requirements were adopted: a) they are part of the Brazilian Network of Public Health Schools, coordinated by the Sergio Arouca National School of Public Health, of the Oswaldo Cruz Foundation; b) they are called public health schools; c) are in the capital of the Brazilian states. By applying the established criteria, 18 SPH were identified.
In order for the participants to receive the research instrument, it was first necessary for the school to authorize the research by signing the Institutional Consent Term (ICT); then the authorization of the institution proposing the research’s REC; and lastly, the authorization of the participant themselves, in the Free and Informed Consent Term (FICT).
Finally, two types of criteria were used to determine the saturation of the sample: a) qualitative, that there should be at least 1 SPH per region of Brazil to guarantee a minimum regional representativeness; b) quantitative, that the number of the sample (n) should be sufficient to apply the technique used to obtain evidence of validation of the structure.
Data analysis was carried out using factor analysis, which consists of a set of statistical techniques, initially developed by Charles Spearman, in 1904.33,34 The methodological path containing the set of techniques used in this research were summarized in Table 2, and detailed below.
To achieve the objective of obtaining validation evidence, a Factor Analysis was carried out and, as it is a new research instrument, the Exploratory (AFE type) was used, with the support of the Factor Analysis software, version 12.4.335 [1], which used the Robust Diagonally Weighted Least Squares (RDWLS) extraction algorithm and the Polychoric correlation matrix,36 indicated when the univariate distributions of ordinal items are asymmetric.37
The unidimensionality assessment was carried out using the Unidimensional Congruence (UniCo) and Explained Common Variance (ECV) measures. The index values must be less than 0.95 and 0.85, respectively, so that the model can be considered one-dimensional.38
The number of factors was decided through the optimized implementation of Parallel Analysis, with random permutation of the observed data, carried out with a resampling procedure.39 Aiming to achieve simplification of the factor and, consequently, of the structure, the Robust Promin rotation was used.40 The cutoff point in the factorial matrix is, at least, a factorial load of 0.300,41 represented in a module, to consider the proposed items. Items with a factorial load lower than the cutoff point were excluded.42
The H index was used to evaluate the stability of the factors. This index evaluates how well a set of items represents a common factor. H values range from 0 to 1. High H values are greater than 0.80, and suggest a well-defined latent variable, which is more likely to be stable across studies. Low values of H suggest a latent variable that is poorly defined and probably unstable between different studies.38 Trustworthiness or reliability was measured by Composite Reliability (CR), being acceptable above 0.70 for each of the factors.43
Pratt’s importance measure was also implemented to complement the interpretation of factor loadings. Pratt’s measure indicates, in percentage terms, how much each factor explains the items.44 Finally, the discrimination parameter and item thresholds were evaluated using the Reckase parameterization, which is the Item Response Theory procedure.45,46
Finally, the adequacy of the model was assessed using the Comparative Fit Index (CFI) and the Tucker-Lewis Index (TLI). CFI and TLI values must be above 0.90.47
The institution linked to the National Commission for Ethics in Research, the Research Ethics Committee (REC) of the Integrated Center for Manufacturing and Technology, of the National Service for Industrial Learning, as the proposing institution, approved the ethical aspects of the research with the number of Certificate of Presentation of Ethical Assessment (CPEA) n° 59519522.5.0000.9287, on July 12, 2022. When required, the research was also submitted and approved by the REC of Brazilian Schools of Public Health (SPH), as a co-participant.
Twelve SPHs, or 66.6% of the target audience, participated in the validation of the factorial structure of the strategy development model, distributed across the five regions of the Brazilian territory as shown in Figure 1, and included 75 participants who answered to the research instrument.
Source: Ref. 31.
The parallel analysis suggested two factors as being the most representative for the analyzed data, as shown in Table 3.
The number of factors to be retained is 2*, as two real data have a percentage of explained variance greater than the random data average. Unidimensionality is permissible when considering the 95% confidence interval, with random data coming from the resampling technique. However, it is noteworthy that the unidimensionality indicators “Unidimensional Congruence” (UniCo) of 0.841 and Explained Common Variance (ECV) of 0.734, were below the cutoff points and, therefore, did not support the unidimensionality of the model.
The two factors were named internal and external. What differentiates them is the governance of the school in relation to the variables. For example, the school has greater governance in relation to the participation of workers in the strategy development, than of representatives of civil society, users and government bodies.
The factor loadings of the items can be seen in Table 4, omitting factor loadings with values below 0.300, in modulus. Trustworthiness or Composite Reliability indices are also reported, as well as replicability estimates of factor scores, given by the latent H-index.38
The measure of replicability of the latent H-index factor structure38 suggested that the factors may be replicable in future studies (H > 0.80). Trustworthiness or reliability, measured by Composite Reliability (CR), proved to be acceptable, being above 0.70 for all factors. The Factor Determinacy Index - FDI, above 0.800; and the Overall Reliability of fully-Informative prior Oblique N-EAP scores (ORION), above 0.70048 corroborate the reliability of the model, according to the FDI indices of 0.905 for factor 1 and 0.902 for factor 2; and ORION, of 0.818 for factor 1 and 0.814 for factor 2.
The items presented adequate factor loadings, with high factor loadings in their respective factors. No item had cross-loading patterns (items with factor loadings above 0.30 on more than one factor).
However, two items were excluded as they had factor loading below 0.30. Item 14 suggests that the model should only have strategic level participation in the strategy development, it is an item with inverted loading, where a response closer to the levels of disagreement on the Likert scale used was expected. This item contradicts the theory of professional bureaucracy, on which the schools are based.49 However, it was initially included in the model to measure acquiescence bias.50 Item 18, which indicates the participants’ degree of agreement on the relationship between strategic objectives and the budget, was included in the research instrument as it is considered a classic weakness of strategic planning processes51,52; however, it is assumed that the low factor loading occurred due to the need to link budget and strategy, is not an exclusive requirement of SPHs and, therefore, does not contribute to differentiating them from other organizational typologies.
Item 17, which measures the participation of interdisciplinary groups in strategy development, was the one with the lowest factor loading, 0.377, associated with factor 2, which groups factors external to the school. At first, what seems out of place is justified by the composition of these interdisciplinary groups, formed by representatives of workers, users, civil society and other government bodies. The school only has governance over the workers’ segment in this group’s formation.
Pratt’s measurement44 provides a more understandable data interpretation, when compared to the factorial structure of the model, as its indicators are expressed as a percentage. It demonstrated that the items were more strongly explained by his factor than by the other. Pratt’s measurement also helps to confirm the absence of cross-loading.
Table 5 shows that the instrument’s items 1, 6, 7 and 10 have 100% adherence to the internal factor, for example. And that items 5, 8, 11 and 12 are those that have full adherence to the external factor. It can be seen that item 14, which was excluded from the instrument due to its low factor loading, had a percentage of adherence relatively balanced between the two factors, which is also not a good indicator.
Item thresholds and discrimination parameters were evaluated using Item Response Theory, and are presented in Tables 6 and 7, respectively. Regarding item thresholds, no unexpected response pattern was found, so the higher the response category of the scale, the higher the level of latent trait required to endorse it.
The most discriminative item or the one that best represents the internal factor is 10, which addresses the strategies developed before and during the planning process (a = 1,114). For the external factor, the most discriminative item was 5, which addresses user participation (a = 1,000), according to the values in Table 7.45
Finally, the fit indices of the instrument Comparative Fit Index (CFI) of 0.951 and Non-Normed Fit Index (NNFI) or Tucker & Lewis Index (TLI) of 0.936 were close to 1 and, therefore, adequate to the parameters established in the literature.53
Evidence was obtained to validate the model structure that aims to produce strategies aimed at SPHs in Brazil, which organized the items into two factors, called internal and external. The internal factors contain nine items; and external ones, seven, which deserve greater attention, as schools have less governance over and, therefore, are those that may present a greater risk when developing their strategies. Two proposed items were excluded, as they did not have a minimum acceptable factor loading.
The trustworthiness indicators (Composite Reliability) and replicability (H-index) used were satisfactory, indicating that the items are capable of measuring the proposed characteristics and that they can be used in future studies. Item response theory indicates an expected pattern of response. According to the item thresholds, the higher the response category on the Likert scale, the higher the latent trait level required to respond to it. The adjustment indices CFI and NNFI or TLI were also adequate to the parameters established in the literature.
Thus, the instrument used for strategy development models aimed at SPHs in Brazil, went through all possible planned stages, with emphasis on structure validation, validation based on Item Response Theory and consequential validation, through risks identified by the Research Ethics Committee. However, the article only investigated the reality of SPHs in Brazil, and to be used in other countries, the instrument must go through a new process of validation evidence search.
Arca Dados: Evidence of validation of the structure of a model for developing strategies in Schools of Public Health. https://doi.org/10.35078/JIDZ3L. 54
This project contains the following extended data:
• Figure 1. jpeg (Participating schools by Brazilian region).
• Data file 1. (Table 1_Survey Instrument Items).
• Data file 2. (Table 2_Steps and Techniques of Factor Analysis).
• Data file 3. (Table 3_Parallel Analysis Results).
• Data file 4. (Table 4_Factor Structure of the Strategy Development Model).
• Data file 5. (Table 5_Pratt’s Measurement of Items by factor).
• Data file 6. (Table 6_Item thresholds).
• Data file 7. (Table 7_Most discriminative item for each factor).
Data are available under the terms of the Creative Commons Attribution 4.0 International License (CC BY 4.0).
I would like to thank Psicometria Online Academy for the teachings on exploratory factor analysis and the Innovation Laboratory “Pólen” for supporting this work.
1 The Factor is a freeware program developed at the Rovira i Virgili University. Download: https://psico.fcep.urv.cat/utilitats/factor/Download.html.
Views | Downloads | |
---|---|---|
F1000Research | - | - |
PubMed Central
Data from PMC are received and updated monthly.
|
- | - |
Provide sufficient details of any financial or non-financial competing interests to enable users to assess whether your comments might lead a reasonable person to question your impartiality. Consider the following examples, but note that this is not an exhaustive list:
Sign up for content alerts and receive a weekly or monthly email with all newly published articles
Already registered? Sign in
The email address should be the one you originally registered with F1000.
You registered with F1000 via Google, so we cannot reset your password.
To sign in, please click here.
If you still need help with your Google account password, please click here.
You registered with F1000 via Facebook, so we cannot reset your password.
To sign in, please click here.
If you still need help with your Facebook account password, please click here.
If your email address is registered with us, we will email you instructions to reset your password.
If you think you should have received this email but it has not arrived, please check your spam filters and/or contact for further assistance.
Comments on this article Comments (0)