Bergen social media engagement and experiences scale (Be-SMEE): A short questionnaire covering important aspects of social media use among adolescents. Development and association with symptoms of depression and anxiety.

Measures of social media use

In a specific section of the survey, adolescents answered statements about their use, beliefs, experiences, and attitudes toward social media. The items were based on findings from focus group interviews of adolescents (27 adolescents across five groups; for more information, see for instance¹³). All the statements had five response options, and for the present study, the following domains were of interest (original number of items in parentheses):

Social comparison and self-presentation (e.g. “I spend a lot of time and energy on what I post on social media”, 7 items), Negative acts and exclusion (e.g. “Others say/post bad things about me on social media”, 5 items), Unwanted attention from others (e.g. “I receive unwanted nude photos or sexualized content from others”, 3 items), Subjective overuse (e.g. “I spend too much time on social media”, 5 items), Social obligations (e.g. “I feel that I must respond to all messages, “streaks” and similar things I receive”, 8 items), and Source of concern (e.g. “There is so much happening on social media that I often feel overwhelmed”, 3 items). The specific statements and response options are presented in Table 1.

Symptoms of anxiety

Symptoms of anxiety were assessed using the General Anxiety Disorder 7 (GAD-7) questionnaire.²⁵ The GAD-7 consists of seven questions about general anxiety symptoms, scored from 1 (“not at all”) to 4 (“almost every day”). It can be used as a continuous measure (total score ranging from 0 to 28) or as a dichotomous variable with a cut-off score of 10 to define case-level anxiety. For this study, the GAD-7 was used as a continuous variable.

Symptoms of depression

Symptoms of depression were measured using the Short Mood and Feelings Questionnaire (SMFQ²⁶). The SMFQ includes thirteen statements about depressive symptoms, with response options of 0 (“not true”), 1 (“sometimes true”), and 2 (“true”). It can be used as a continuous measure (total score ranging from 0 to 26) or as a dichotomous variable with a cut-off at the 90^th percentile to define case-level depression. For this study, the SMFQ was used as a continuous variable.

Statistical analyses

The “LifeOnSoMe”-study proper was designated as the model development dataset, while the pilot study data was designated as the external validation sample. First, descriptive statistics of self-reported age and gender, as well as mental health variables, were presented across the model development and the external validation sample. Frequencies and proportions were estimated for age and gender, while the median and interquartile range were estimated for symptoms of depression and anxiety. Potential differences between the two datasets were estimated using Pearson’s Chi-squared tests for age and gender and Wilcoxon rank sum tests for the mental health variables. Next, item reduction was done using Ant Colony Optimization (ACO). ACO is a metaheuristic algorithm inspired by ants’ foraging behaviour, in this case applied to factor analysis for optimizing measurement scales.²⁷ In our context, artificial ants construct solutions by selecting items, guided by pheromone trails that represent the quality of previous solutions.^28–30 The algorithm iteratively updates these trails, reinforcing paths leading to psychometrically sound scales and factor structure.²⁷ ACO has previously been successfully used to develop short scales, such as for assessing personality²⁹ and the beforementioned short social media scale for depression.²² ACO-analysis was performed using the model development data set. Model fit and factor loadings for the suggested model were estimated across both datasets using confirmatory factor analysis with the Diagonally Weighted Least Squares (DWLS) estimator. DWLS was employed to handle the ordinally scaled items included, providing more accurate parameter estimates and standard errors. Model fit was assessed using the Comparative Fit Index (CFI, good fit: ≥ 0.95), Tucker-Lewis Index (TLI, good fit: ≥ 0.95), Root Mean Square Error of Approximation (RMSEA, good fit ≤ 0.06) and Standardized Root Mean Square Residual (SRMR, good fit ≤ 0.08). Configural and scalar measurement invariance³¹ were also tested across the two datasets, as well as across gender and age (see Table 5). As per recommendations for ordinal indicators, we bypassed metric invariance testing and directly assessed scalar invariance by simultaneously constraining factor loadings and thresholds.^32,33 This approach is more appropriate and parsimonious for ordinal data, as both loadings and thresholds jointly determine response probabilities. For measurement invariance, we jointly considered ΔCFI ≤ -0.01 and ΔRMSEA ≤ 0.015 as evidence of invariance across groups. Finally, Bayesian linear regression models³⁴ adjusted for age and gender were separately estimated between each of the suggested domains (summed average score for each domain) and symptoms of anxiety and depression as dependent variables across both data sets. The following estimates were obtained from the regression models; the coefficient and the corresponding 95% credible interval and the probability of direction. For the regression models, the dependent variables were standardised (Z-scored; mean of 0 and standard deviation of 1) in each data set. Additionally, the Bayes factor and the error percentage were estimated when comparing an age- and gender-only model (baseline) versus a model that also included the social media domains. Bayes factor estimates the relative evidence for one statistical model over another by comparing their predictive performance.³⁵ Potential differences in regression estimates across the two datasets were investigated in moderation analyses in a combined dataset with a grouping variable term: dependent variable×dataset. Moderation by dataset was considered present when the credible interval of the interaction term did not cross zero. For the development dataset, a total of 3,285 participants had valid responses for all variables of interest and were included in the analytical sample. In comparison, the external validation dataset included 509 participants with valid responses. Missing data was handled by case-wise deletion, with a maximum of 7.7% of the total number of participants excluded in any of the analyses. All analyses were done using R Studio.³⁶ ACO-analyses were performed using the ‘ShortForm’-package,³⁷ while Bayesian linear regression models were computed using ‘rstanarm’³⁸ and Bayes factor estimates were derived from the ‘BayesFactor’-package.³⁹ Confirmatory factor analyses were done using the ‘lavaan’-package.⁴⁰ Tables were produced using the packages ‘gtsummary’⁴¹ and ‘flextable’.⁴²

Implications

The suggested questionnaire could potentially have public health relevance as a reliable and efficient tool for assessing social media use and its association with mental health among adolescents. The questionnaire can potentially help identify adolescents at risk of anxiety and depression related to social media use, enabling early intervention and support. The relatively high level of explained variance suggests that the questionnaire captures a substantial portion of the social media aspects related to depression and anxiety symptoms, highlighting its potential relevance in understanding these mental health issues in a modern technology-oriented society.

Public health professionals can use scores from the questionnaire to design targeted interventions aimed at addressing potential links between social media use and mental health outcomes in adolescents. Findings from research employing the questionnaire can inform policymakers about the relationship between social media and adolescent mental health, helping to shape regulations and guidelines that promote healthier social media habits. For research purposes, the suggested questionnaire provides a comprehensive framework for understanding how different aspects of social media use relate to important outcomes such as mental health problems and well-being. By exploring the associations between specific domains of social media use and mental health outcomes, researchers can gain deeper insights into the mechanisms underlying these relationships. It can also help ongoing research to monitor trends in social media use and the effects of social media use, and this knowledge could potentially be used in intervention studies, helping to adapt public health strategies over time. Ultimately, the proposed questionnaire has the potential to enhance understanding of mental health challenges associated with social media use, contributing to improved health outcomes for adolescents.

Strengths and limitations

The present study has several notable strengths. Firstly, by using the ACO approach to reduce the number of items, we identified a relatively short questionnaire that effectively captures six different domains related to social media use.^46,47 Secondly, we leveraged two independent yet comparable datasets, enabling robust external validation of the questionnaire in terms of both factor structure and its relationship with symptoms of anxiety and depression. This also meant that we were able to do measurement invariance testing across the two datasets. Although the practical relevance of (especially higher order) measurement non-invariance has been debated,^31,48,49 it is a strength that we were able to test for configural and scalar invariance in two independent samples. It is also a strength in itself that we were able to investigate the convergent validity and relevance of the suggested domains against frequently used and validated scales focusing on symptoms of depression and anxiety.^25,26 Thirdly, the items included in the suggested questionnaire are less platform-dependent and likely more robust to changes in functionality and mere usage patterns on social media. This is especially important as social media is often thought of as a moving target in research,⁵⁰ and we believe that the suggested items are less prone to being outdated by changes to the underlying technology or user interface.

However, several limitations should be acknowledged. Firstly, although the suggested questionnaire covers key domains related to social media use, it does not encompass all relevant dimensions.^9,22 Social media use and aspects of social media is complex and multidimensional, and depending on the focus of interest, other domains may be more or less important to assess than those included here. For example, the questionnaire primarily captures risk-related aspects, leaving out positive dimensions of social media use that might buffer against mental health problems. However, we believe that the suggested questionnaire covers six domains that are likely to play a crucial role in our understanding of how social media use may be a health determinant, especially for factors related to mental health and well-being. Secondly, the study is based on cross-sectional data, which limits our ability to infer causality or temporality relationships. Future research should investigate changes over time for the suggested domains and longitudinal associations with for instance mental health. Thirdly, as both datasets rely on data collected from upper secondary schools, the age-range is quite limited (range 16-21 years). Future research should investigate how well the suggested questionnaire performs in both older and younger cohorts. Lastly, our study population was geographically restricted to Vestland County, Norway, and both datasets were collected in a Norwegian context. Future research should assess the questionnaire’s relevance, psychometric properties, and cultural adaptability in diverse populations and linguistic contexts.