Exploring Sample Size Determination in Educational Research: A Comprehensive Review

Introduction

The term “sample size” describes the number of subjects or observations that make up a study ‘n’ is typically used to represent this number. The size of a sample affects two statistical properties:1) the accuracy of estimates and 2) the study’s ability to draw inferences.¹

Surveys, experiments, observational studies, and other types of clinical research studies can all be categorized. Many different factors are involved in excellent research planning. The first step is to define the practical issue. Choosing the relevant participants and controls, as well as the experimental or observational units, was the second stage.

The inclusion and exclusion criteria must be carefully defined and should account for any potential variables that could affect the measurements and units being observed. The study design must be precise, and the procedures must follow the best technique currently available. Based on these considerations, the study’s sample size needs to be appropriate for its goals and potential variability. The sample must be “large enough” for the effect to be statistically significant and have the expected size of scientific significance. At the same time, it is crucial that the study sample not be “too big,” where a statistically significant effect of minor scientific import could still be found.² Additionally, the sample size was economically significant. Resources may be wasted in an insufficient study because it may not yield valuable results, whereas an excessively large study consumes more resources than is required. The sample size of a study involving human or animal subjects is a crucial ethical concern, because a poorly planned experiment exposes participants to potentially hazardous procedures without contributing to information.^3,4 Therefore, calculating the power and sample size is crucial in the design of clinical research. Unaccountable studies printed in national and intercontinental journals have found that sample size estimation were incorrectly disclosed or had smaller samples than necessary, which reduced their power.^1,2

There is still much confusion despite the fact that unnumbered studies clarifying the methods of sample size computation have been published in the existing literature. It is crucial to realize that there is no single universal formula for calculating the sample sizes for all study designs. Instead, different study designs require different methods to calculate sample sizes.^3,4 The study aims to explore existing literature on sample size determination in educational research, examining the methodologies employed and providing insights into associated challenges.

Method

Search Strategy: To conduct this systematic review, Preferred Reporting Items for Systematic review and meta-analysis (PRISMA-S) criteria were used to guide the search.¹ The search strategy was aimed to identify relevant articles addressing sample size determination in educational research. Researches searched the following databases such as PubMed, Embase, and the Cochrane Library, additionally; hand searching of reference lists and citation tracking were conducted to identify additional relevant articles. The search term or keywords for accessing the data from the databases related to sample size (e.g. sample size calculation, sample size determination) and educational research (educational studies, pedagogic studies). To perform search effectively and refine search results, the search strings were constructed using BOOLEAN operator (AND, OR).

Inclusion and Exclusion criteria: Articles from the year of 2010 to 2022 were screened; free full-text, unlocked articles, pertinent terminology and information, and English language usage were required for inclusion of the data. The exclusion criterion was considered and they excluded abstracts, locked articles and journals, no relevance in the data, and languages other than English. Duplicate articles and those not directly related to sample size determination in educational research was excluded.

Screening Process: Two authors independently screened the titles and data of the extracted articles to determine the eligibility of the included articles. The screening process involves reviewing the titles and abstracts of retrieved articles to assess their relevance to the research question. Any discrepancies or disagreements between the two authors regarding the eligibility of articles were resolved through discussion and consensus.

Documentation and Standardization: Data from the extracted articles were documented into standardized format into Microsoft excel sheet. This documentation process ensured consistency and organization in recording relevant information from each article.

Assessment of the included articles: The included articles were assessed by using appropriate criteria such as methodological rigor, relevance to the research question, and transparency of reporting.

Synthesis of data: In the end all the data were synthesized for common themes, relevancy, challenges, and recommendation related to the sample size assessment. The principal investigator carried out the entire review planning process, which was authorized by the other authors. The presentation of the entire search is shown in Figure 1.

Figure 1. The schematic presentation of the data by using updated PRISMA guidelines.

Results

Seven studies were selected from the 928 articles by using Google Scholar and PubMed with the applicable of stringent inclusion and exclusion criteria. All the information related to the articles were shown in the Table 1.

Discussion

Research in health science education is expanding. Emerging educational research relies on relevant conceptual frameworks, reliable research techniques, and important discoveries.^2,3 Prior reviews have shown that many educational research articles employ small sample sizes, despite the fact that researchers rarely take into account the expected impact size, intend the sample size in before, or describe the actual precision in evaluating the results.^4,5

Hence, the determination of the sample size in the research is the utmost criteria before performing any research studies. McConnell et al. emphasize accurate sample size estimation in medical research, especially with novel interventions like programmed intermittent epidural bolus analgesia. They stress considering effect size, categorized as small, medium, or large respectively, for values of 0.20, 0.50, and 0.80 by Cohen’s criteria, indicating the practical significance of outcomes.^6,7 Understanding expected effect size aids in determining the minimum sample size for sufficient statistical power.⁸ Power analysis ensures studies can detect true effects, preventing statistically insignificant results. In educational research, reporting effect sizes and confidence intervals is common, aiding in comparing results across studies.⁷ Hattie’s synthesis of 1,200 meta-analyses provides mean effect sizes for various educational activities.⁹ Researchers should avoid simplistic categorizations of effect size and instead uses validated instruments, and consider factors like attrition rates. Confidence intervals provide a range of possible effect values, emphasizing magnitude and precision.⁶ The criteria for forecasting the sample group, such as the relevance factor, preferred statistical significance, predicted difference in score, and approximate evaluation variation, which may be estimated from previous studies, were discussed in order to determine the number of participants required to assess the effects of an intervention on a specific outcome or the association between variables.^6,10 Interventions in education frequently concentrate on changing latent conceptions, which are theoretical and cannot be readily seen or quantified. This causes the validated scales to vary, changing how the outcome measures are calculated.¹⁰

Staffa et al. presented a five-step method for selecting sample sizes with therapeutic significance in pediatric surgery. Their approach emphasized defining primary outcomes, expected effect sizes, and appropriate statistical tests. They highlighted the importance of statistical power, which is the likelihood of rejecting the null hypothesis if the observed effect in the population matches the effect size.¹¹ Studies with higher power are preferred to avoid missing important connections. A power of 90% is ideal, with 80% considered the minimum. Sample size, effect size, and risk of type I errors all influence power. Sample sizes should be justified using power calculations to ensure adequate detection of clinically meaningful effects.¹² Historical control groups should be comparable to research groups to effectively compare outcomes. The suggested approach is applicable to various data and study designs, although examples may not cover every research circumstance.¹³ Overall, the study advocates for a rigorous approach to sample size determination to improve the quality and reliability of paediatric surgery research.

Dreyhaupt et al. shed light on cluster randomization in educational research, comparing it with individual randomization and highlighting the need for larger sample sizes and more complex calculations in cluster-randomized studies. Understanding the implications of study design choices on sample size requirements is crucial. Their objective was to establish optimal subject numbers for studies, ensuring sufficient statistical power to address clinically important questions.¹⁴ The evaluation focused on achieving standard power levels (80% or 90%) in randomized controlled trials with parallel group designs. Essential themes include the need for a control arm, statistical comparability, and structural equality between study arms.¹⁵ Cluster randomization is commonly used for non-therapeutic interventions, with clusters ranging from two to thousands of individuals. Contamination bias reduction is a primary reason for conducting cluster-randomized studies, considering the statistical dependency within clusters. This dependency affects sample size computation, necessitating specific methods tailored for cluster-randomized investigations.¹⁴

Cook et al.’s systematic review underscores the significance of adequate statistical power in simulation-based education research. They observed considerable variability in sample sizes among studies, with many only capable of detecting moderate to large effect sizes, while others could only detect immensely large effects. Moreover, studies not identifying statistically significant differences often had broad confidence intervals, hinting at potential large and crucial differences not captured by the analysis.⁹ Importantly, the absence of statistically relevant outcomes in these studies doesn’t imply the superiority or equivalence of interventions being compared. This highlights the necessity of considering both statistical significance and effect size when interpreting educational research findings. Their re-analysis of meta-analysis data further accentuates the challenge of achieving sufficient power, particularly in studies with smaller effect sizes. Most included research could only detect moderate to large standardized mean differences, while some could only detect immensely large effects.⁸ This study underscores the importance of carefully addressing statistical power and effect size considerations in educational research to ensure meaningful and reliable conclusions.

A comprehensive overview by Agnihotram highlights the steps involved in estimating sample sizes for various research objectives. Clear study aims, appropriate design selection, and meticulous statistical analysis planning are emphasized, underscoring the need for a systematic approach to sample size determination. The study’s findings underscore the multifaceted nature of sample size calculation, considering factors from research objectives to statistical methodologies.¹⁶ In research, statisticians play a crucial role in ruling out the number of subjects and analyzing final results to ensure rational and trustworthy implications applicable to the sample population. Understanding analytical methods is crucial for investigators to conduct well-defined studies producing reliable outcomes. Statistics play a vital role in improving patient care by extracting crucial information from empirical data. Sample size estimation can be categorized into two main types: for estimation studies and for hypothesis testing or comparison studies. In estimation studies, researchers estimate parameters like mean hemoglobin levels or disease prevalence, while in hypothesis testing studies, they compare population characteristics or outcomes before and after interventions. Precision in estimation studies requires larger sample sizes, as the margin of error decreases. For hypothesis testing, sample size calculation aims to achieve the desired power to detect clinically significant differences at a predetermined significance level.

Ferreira et al. investigated hypotheses and sample sizes for assessing physical activity in pediatric populations, comparing subjective and objective methods. They emphasized the importance of adequate sample sizes for achieving stable agreement between these methods.¹⁷ Three sample size determination strategies were discussed: interval strategy, hypothesis-related approach, and “indifference zone.” A priori hypotheses, fundamental to the scientific method, were inferred based on assumed assumptions.¹⁸ The systematic review suggested modest to moderate agreement between subjective and objective measures of physical activity intensity and duration. However, currently, there’s no data to support a priori assumptions regarding agreement between assessment methods. Robust a priori hypotheses are crucial for sample size selection, precision, and power. The study recommended a sample size of 50–90 subjects for assessing physical activity parameters, noting stable agreement within this range. Variability in findings was observed in studies with small samples, potentially due to inferior design compared to studies with larger samples.¹⁸ Overall, the study highlights the importance of careful consideration of sample size and hypothesis formulation in research on physical activity assessment in paediatric populations.

Guo et al. used two different types of hypotheses, taking into account sample size planning factors such superiority/non-inferiority and equivalence of two means. When population variances are unknown, no exact sample can be found through traditional sample size formula and resulting sample size must be suitable enough to meet the required level of significance and probability of correct decision and power. The cost constraint depends on the two experimental goals for given level of αand power 1-β i.e. allocation of having minimal total cost and ratios are a function of unit cost ration and standard deviations.¹⁹ Their study highlighted the importance of selecting appropriate statistical tests and considering the implications of unknown and unequal population variances on sample size calculations. The findings provided valuable insights for researchers planning studies with cost considerations.

In general, the challenges in sample size determination encompass various factors, including effect size consideration, methodological complexity, discrepancies between subjective and objective measures, achieving sufficient statistical power, balancing power requirements with cost considerations, and adapting methodologies to specific study designs.^11,12,14,16 Overcoming these challenges requires a systematic approach and careful consideration of the unique aspects of each research study such as determining sample size in research, a systematic approach is needed. Validating measurement tools and techniques ensures consistency and reliability between subjective and objective measures. Multiple assessment methods can strengthen findings. Power analyses help determine the required sample size, balancing budget and feasibility constraints. Tailoring methodologies to study objectives and design optimizes calculations. Clear study aims, design selection, and meticulous statistical analysis planning align with research objectives and methodological requirements. Consulting methodological experts or statisticians helps navigate complex methodologies. Prioritizing understanding of expected effect size through pilot studies or literature reviews informs accurate sample size calculations. These strategies help researchers overcome challenges and detect meaningful effects while considering methodological complexities and practical constraints.

Researchers can use various strategies to determine sample size, including the interval strategy, hypothesis-related approach, and the “indifference zone”. The interval strategy maintains a high confidence interval while limiting sampling error, while the hypothesis-related approach specifies null and alternative hypotheses to detect significant differences. The “indifference zone” places populations performing better in a zone more likely to be chosen correctly. Each strategy offers unique advantages and considerations for researchers.

According to the statistics, there are various methods, test and formula for estimation of the sample size required to perform the research and other relevant studies. But the lack of research regarding the appropriate and whole number needed for performing any research is not established yet, like pilot study confirms the 12 participants for each group enrolling for the particular trial.²⁰

This review highlights the diverse approaches and procedures used in many studies to determine the appropriate sample sizes for educational research. It also emphasized the difficulties in defining the main outcome, choosing the right statistical tests, and taking effect size and statistical power into consideration. This article fosters a critical comprehension of the results and their relevance in various research contexts. Along with the recommendations, key considerations include defining research objectives, selecting appropriate study designs, and ensuring adequate statistical power. Through adherence to PRISMA-S guidelines for systematic reviews, the article emphasizes the importance of transparency and rigor in the review process. This commitment to methodological rigor enhances the credibility and trustworthiness of the insights presented.

Conclusion

The review underscores the critical importance of considering sample size early in the research phase to gather comprehensive background insights, ultimately enhancing the application of pedagogical findings. It emphasizes that all types of research investigations necessitate careful determination of sample size, highlighting the essential role of selecting the appropriate formula. The choice of sample size formula is guided by various factors including the study’s objectives, outcome variables, planned statistical analyses, study groups, and allocation procedures. Determining the sample population involves considerations such as study feasibility, statistical power, precision of calculations, analytical relevance, confidence level, ability to detect clinically significant differences, as well as practical constraints like financial resources, workforce availability, subject recruitment, and time constraints. Studies employing cluster randomization methodologies typically require larger sample sizes and more intricate calculation methods. The cited studies suggested that a sample size of approximately 24.24 participants per group is typically required for conducting new methods, while the median sample size for simulation-based educational research is around 30. However, it acknowledges the need for further research to establish appropriate sample sizes and develop a universal formula applicable to various study designs.

Data availability

All data underlying the results are available as part of the article and no additional source data are required.