[version 1; peer review: 1 approved with reservations]
No competing interests were disclosed.
Statistical tests are numerous in conducting a meta-analysis goes beyond the simple calculation of a group estimate. Combining these diverse results (heterogeneity, bias, sensitivity, subgroups, etc.) into a coherent whole suitable for various groups of people (e.g., academic papers, plain language summaries) is a major challenge.
We created
The
Meta-analysis is a valuable method of quantitative data synthesis across studies. For continuous outcome measures, it usually entails calculation and combining either Standardized Mean Differences (SMD) or Mean Differences (MD) (
The R Shiny application (
The most distinctive feature is interoperability with Google’s Gemini large language model (LLM). After completing the analyses, the app then gathers all the results and sends them to the LLM to generate draft reports in various formats—plain language summaries, for example. This allows researchers to produce written descriptions of their results quickly without wasting time and improving readability.
The app is built using R (version 4.4.0) and the Shiny web framework (version 1.8.1.1). It uses the
A full suite of analyses is run using functions from established R packages:
Users upload a CSV file via
The helper function
After the statistical analyses are completed reactively, their textual outputs are collected using
The LLM’s raw response is cleaned of basic markdown formatting using the
The dashboard interface includes two main tabs:
The
The
You can run MetaLLM Reporter locally with R (version 4.0.0 or higher) and the required packages, or use a hosted version in your browser. To access LLM features, you’ll need an internet connection and a valid Google Gemini API key (user to provide).
In the “Inputs” tab, upload a CSV with mean, SD, and sample size for both groups, plus the author name. Optional columns (
Go to the “Meta-Analysis Text” tab to see results like heterogeneity, publication bias, and regression. The app runs all analyses automatically.
Check the final four tabs (“LLM Cochrane”, “LLM NEJM”, “LLM Lancet”, “LLM Plain”) for AI-generated summaries tailored to different reporting styles.
MetaLLM Reporter is designed to assist researchers who have completed a meta-analysis, or possess the requisite data, in consolidating diverse statistical outputs into comprehensive written summaries or reports.
These scenarios demonstrate the tool’s capacity to automate the generation of initial text drafts from complex statistical results, potentially expediting the reporting process.
Though a vast number of tools exist for doing meta-analysis, briefing results from heterogeneity tests, bias calculations, sensitivity analyses, and subgroup/meta-regression analyses into intelligible written description relics a massive undertaking.
MetaLLMReporter referrers to this problem by performing not just a plethora of average meta-analysis actions using already existing R packages (meta, metafor, dmetar, etc.) but also by mixing a large language model (LLM) to instinctively produce reports in different styles.
The main revolution is the usage of Google’s Gemini LLM to produce statistical yields into organized texts that are catered for a diverse audience, going from recognized Cochrane-style reports to simple language abstracts. This intends to drastically decrease the writing effort that is expected by researchers, providing them with drafts, including results from various analytical components. The program executes a wide range of appropriate analyses robotically on data upload, making sure the components are readily accessible for the LLM prompt.
Nonetheless, the dependence on LLM-generated texts needs serious deliberation. Whilst possibly saving time, the output must be cautiously reviewed as a draft needing thorough analysis and proof reading by experts. LLMs may mistake the context, misunderstand statistical distinctions, or quite possibly “hallucinate” information. The superiority of the LLM is seriously reliant on the cohesive and extensiveness of the prompt given and with the abilities of the LLM used. The present implementation also necessitates implanting API keys, leading to potential security risks thus making it inappropriate for large scale sharing without changes made. Moreover, the app centres on text production and lacks the interactive plotting aspects found in other meta-analysis tools.
Changes down the line could potentially give users the ability to choose which analysis to incorporate in the LLM prompt, fine tuning the prompts for more precision and style adherence, including visuals together with texts, accompanying further data types, presenting options for diverse LLMs, and applying additional safe API key management.
This study did not involve human participants, human data, or human tissue. Therefore, ethical approval and consent to participate were not required.
Source code available from:
Archived software available from:
License: Apache 2.0 (OSI-approved open license)
This software is freely available and can be used without restriction. Use of the LLM features (e.g., Gemini API) requires a valid API key provided by the user. The demonstration dataset and all analysis functionality are included in the GitHub repository for full replication of results.
No new data were generated or analyzed in support of this research. All data and materials supporting the results or analyses presented in this article are openly available. A sample dataset used for demonstration is included in the GitHub repository and can be used to replicate all study findings.
These datasets include:
The values behind the means, standard deviations, and other measures reported;
The values used to build graphs and figures;
The points extracted from images for analysis.
No new external data were generated during this study. All provided data are openly licensed under the Apache 2.0 license, permitting unrestricted reuse.
Dataset available from:
Persistent identifier: Zenodo DOI
License: Apache 2.0
If any dataset cannot be shared due to ethical, privacy, or security concerns, this will be noted in future updates. At present, no such restrictions apply.
None.
The article presents MetaLLMReporter, an R Shiny application that runs standard meta-analysis for continuous outcomes and then generates draft narrative reports using a large language model (LLM). The app reads a user CSV to calculate aggregated means, standard deviations, and sample sizes, fits fixed-effect or random-effects models using established R packages, prints textual outputs for common diagnostics such as heterogeneity, leave-one-out analysis, publication bias tests, trim-and-fill, meta-regression, subgroup and cumulative analyses, and an optional Bayesian model. It then sends the captured text to an LLM to produce prose in several editorial styles. The paper provides a public GitHub repository and a Zenodo archive.
The motivation is clear: investigators often have many numerical outputs from a meta-analysis and need coherent text for reports or manuscripts. The paper frames the tool as a bridge between analytic output and narrative reporting.
To strengthen the motivation further, the authors can add one concrete use case that maps each displayed analysis block to a common reporting requirement, and a short note that LLM text must be checked by a human.
The top-level architecture is sound and maps well to standard practice in R. However, several details in the current code can yield incorrect or fragile results.
First, cumulative meta-analysis is called without enforcing a chronological or pre-specified ordering. The output will depend on input row order. A meta-analysis usually orders cumulative results by year or another meaningful variable. The app should require a study-level ordering column and sort before calling the cumulative procedure.
Second, if the uploaded CSV lacks the intervention sample size, the server fills one default value from a numeric input and uses it for all studies. This can bias effect size variances and weights. The app should reject inputs that miss required counts for either group or ask for per-study values.
Third, in the meta-regression and Bayesian analysis, the code recomputes study effects with metafor::escalc() using the UI’s effect measure, while the main model uses meta::metacont() with a configurable SMD method. This can produce a mismatch if the SMD correction in the two paths differs. The regression should reuse the study-level effects and standard errors from the fitted meta object or enforce the exact same SMD specification in escalc().
Fourth, the “Study Weights” text pastes names (m$w.random) with the numeric weights. In typical meta::metacont fits the weight vector is not named, so the display can be unlabeled.
Finally, there are plaintext LLM API keys in the code and the key is passed in the URL. Keys must not be hardcoded. They should be read from environment variables or a secrets store and never logged. Also, sampling from multiple keys seems unnecessary.
These issues do not change the overall concept, but they affect correctness, security, and maintainability.
The article gives repository and archive links, the tool is small, and package names are stated, which makes basic replication feasible.
For stronger reproducibility across systems, the repository should include an renv.lock with exact package versions or a minimal Dockerfile and instructions to run the app. The app should also document the exact CSV schema with a minimal valid example and validation rules, and explain how to set the LLM API key at runtime. Without these pieces, users may encounter version drift or silent data issues.
The tool prints many standard summaries, but it does not guide users on when a diagnostic is meaningful or how to read it. For example, Egger’s test can be unreliable with few studies or strong heterogeneity; p-curve has assumptions that should be stated; cumulative meta-analysis requires a justified ordering. The app should add short, in-context notes that state applicability conditions and common pitfalls, and it should display a per-study table of computed effects and variances so users can verify inputs. Adding forest and funnel plots would also help users cross-check the prose.
It is reasonable to conclude that the app can run standard analyses and produce draft narratives. However, claims about reporting support and workflow gains would be stronger with a small worked example that starts from the sample CSV, shows the key numeric outputs, and compares the LLM drafts with a hand-written baseline. If any performance or quality claim is made, it should be backed with either expert review or a simple user study.
(i) Enforce a valid ordering for cumulative meta-analysis by requiring an ordering column and sorting before computation.
(ii) Avoid filling in default values and require complete per-study counts, or explicitly document and implement a principled missing-data strategy that does not bias variances.
(iii) Align the meta-regression inputs with the primary model by reusing study-level effects and standard errors from the meta fit or by ensuring escalc() applies the same SMD correction as the main model.
(iv) Replace the unlabeled weight printout with a labeled table that pairs each studlab with its weight.
(v) Remove hardcoded LLM API keys from the code and read a single key from the environment. Do not pass keys in query strings. Document setup steps for users.
(vi) Add input validation and short interpretive notes next to each diagnostic to warn about small-k limits, heterogeneity effects, and assumptions.
(vii) Provide an renv.lock or Dockerfile and a minimal end-to-end script or vignette that reproduces the example in the article.
Are the conclusions about the tool and its performance adequately supported by the findings presented in the article?
Partly
Is the rationale for developing the new software tool clearly explained?
Yes
Is the description of the software tool technically sound?
Partly
Are sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others?
Yes
Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool?
Partly
Reviewer Expertise:
Data mining, measurement
I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.