ALL Metrics
-
Views
-
Downloads
Get PDF
Get XML
Cite
Export
Track
Research Article

A pilot study: Can heart rate variability (HRV) be determined using short-term photoplethysmograms?

[version 1; peer review: 2 approved with reservations]
PUBLISHED 22 Sep 2016
Author details Author details
OPEN PEER REVIEW
REVIEWER STATUS

Abstract

To date, there have been no studies that investigate the independent use of the photoplethysmogram (PPG) signal to determine heart rate variability (HRV). However, researchers have demonstrated that PPG signals offer an alternative way of measuring HRV when electrocardiogram (ECG) and PPG signals are collected simultaneously. Based on these findings, we take the use of PPGs to the next step and investigate a different approach to show the potential independent use of short 20-second PPG signals collected from healthy subjects after exercise in a hot environment to measure HRV. Our hypothesis is that if the PPG--HRV indices are negatively correlated with age, then short PPG signals are appropriate measurements for extracting HRV parameters. The PPGs of 27 healthy male volunteers at rest and after exercise were used to determine the HRV indices: standard deviation of heartbeat interval (SDNN) and the root-mean square of the difference of successive heartbeats (RMSSD). The results indicate that the use of the $aa$ interval, derived from the acceleration of PPG signals, is promising in determining the HRV statistical indices SDNN and RMSSD over 20-second PPG recordings. Moreover, the post-exercise SDNN index shows a negative correlation with age. There tends to be a decrease of the PPG--SDNN index with increasing age, whether at rest or after exercise. This new outcome validates the negative relationship between HRV in general with age, and consequently provides another evidence that short PPG signals have the potential to be used in heart rate analysis without the need to measure lengthy sequences of either ECG or PPG signals.

Keywords

Mobile health, Point-of-care device, Affordable healthcare, Moving averages

Introduction

The most commonly accepted heart rate variability (HRV) metric to measure both instantaneous heart rate (HR) and RR intervals is derived from electrocardiogram (ECG) signals. Metrics used to describe HRV in literature include heart period variability, cycle length variability, RR interval tachogram, and RR variability.

HRV measurement captures HR variations of the mean HR, while providing information on the stability of the sympathetic–parasympathetic autonomic system. This in turn provides information on the risk of sudden cardiac death. An example of this can be seen when continuous subjection to dry heat occurs with subjects. Dry heat exposure provokes a stress response that is designated by increased HR, which occurs due to a significant reduction in parasympathetic control of the HR, which is signified by reduced HRV1.

The traditional method of identifying heartbeats in ECGs is by detecting R peaks. In almost every study, comparisons are made between HRV calculated from ECG signals and those calculated from photoplethysmogram (PPG) signals. In these studies, the feasibility of using PPGs as an alternative simple, inexpensive, and convenient diagnostic tool was explored in parallel with the use of ECGs. The overall results showed that PPG is potentially an ideal stand-alone alternative but it has not been explored independently, possibly explaining why there are no known investigations into the alternate use of PPG signals to measure HR and HRV.

It is noted that accurately detecting inter-beat intervals from fingertip PPG signals is challenging24. Bernston et al.2, noted these challenges and recommended using RR intervals from ECG signals to determine inter-beat intervals. Even with the use of a peak detection algorithm that is sophisticated, using intra-arterial pressure pulses is likely acceptable but indirect PPG signal measures need further validation. The contour of the PPG can measure HRV through the detection of the heartbeat, and thus HRV can be measured5.

Giardino et al.4 demonstrated that it is sufficient to use the distal pulse pressure, under resting conditions, to determine the heart rate as shown in Figure 1(a). The authors also recommended further investigations into the test-retest reliability evaluation for different data collection techniques.

00c6e344-0b97-47b4-a41c-b2f7ef51cdec_figure1.gif

Figure 1.

Two successive beats in (a) fingertip photoplethysmogram (PPG) signal (b) second derivative wave of photoplethysmogram (APG) signal.

These cautious evaluations may explain the reason for limited investigation into the use of PPG signals (rather than ECG) to measure HR and HRV.

The contour of the PPG can be utilized to detect the heart beat and therefore HRV can be measured from the contour5. In Figure 1(a) two circles, with the smallest positive PPG amplitudes, represent two consecutive heartbeats. Reliable detection of the heartbeats using the PPG contour is, however, challenging due to noise and the nature of its associated interference with hemodynamic variables6. These identified challenges can be addressed by using the second derivative of the photoplethysmogram waveform, also known as the acceleration plethysmogram (APG). In Figure 1(b), two circles with the largest positive amplitudes, represent two consecutive heartbeats of the APG signal. The heart rate is more accurately detected using the APG signal due to the clearly defined peaks in the APG, when compared to the peaks in the PPG contour (more specifically the a wave), as shown in Figure 2(b).

00c6e344-0b97-47b4-a41c-b2f7ef51cdec_figure2.gif

Figure 2. Fingertip photoplethysmogram signal measurement7.

(a) Fingertip photoplethysmogram. (b) Second derivative wave of photoplethysmogram. The photoplethysmogram waveform consists of one systolic wave and one diastolic wave, while the second derivative photoplethysmogram waveform consists of four systolic waves (a, b, c, and d waves) and one diastolic wave (e wave).

We therefore sought to validate the usefulness of PPGs using an alternative method without the collection of ECGs based on the already established negative correlation between HRV and age8. We hypothesized that if the PPG–HRV indices calculated from short PPG signals are negatively correlated with age then a short PPG signal is an appropriate measurement for extracting HRV parameters. In this study, only the standard deviation of the heartbeat interval (SDNN) and the root-mean square of the difference of successive heartbeats (RMSSD) indices are investigated, as they are suitable for short recordings based on the recommendation in 9. Herein we investigate if the SDNN and RMSSD calculated from short PPG signals measured before and after exercise can potentially be correlated with age.

Materials and methods

Ethics statement

Charles Darwin University (CDU) has one annotated PPG database available with data collected, in 2006. The dataset includes participant recordings during rest (before exercise) and after one hour of exercise (walking) on a treadmill in the climate control chamber at the Northern Territory Institute of Sport (Darwin, Australia). PPG data was collected on three separate occasions during each rest period as shown in Figure 3.

00c6e344-0b97-47b4-a41c-b2f7ef51cdec_figure3.gif

Figure 3. Measurement Protocol.

The duration of the whole experiment was approximately 4 hours, each exercise consumed approximately 1 hr while the PPG signals collected during the 5 minutes break of each exercise at a sampling rate of 200 Hz. The length of each PPG recording is 20 seconds.

Exercise was set to moderate intensity, with the treadmill speed set at 5 km/h with a 1% incline increment (which corresponds to effort required to walk on 8 kg of webbing). Details on the project are found at 10. Written consent was obtained from all subjects before participating in the Charels Darwin University Ethics Committee approved study and was conducted in accordance with applicable guidelines and regulations.

Database used

A photoplethysmography device (Salus APG, Japan) was used to measure PPG recordings from 27 healthy male volunteers (mean ± SD age of 27 ± 6.9). Sensors were placed on the second digit of the left hand on the cuticle. Measurements were recorded for each subject at rest on a chair. Data were collected at a sampling rate of 200 Hz, with recording duration of 20 seconds.

We used one PPG oximeter to collect the data from all subjects measured after each exercise. It was not plausible to collect data for a longer period of time as we needed to measure other subjects waiting in line. The longer the waiting time, the more the heat stress impact due to cooling of the body is lost. Therefore, it was convenient to collect data for a consistent recording length of 20 seconds. Preliminary feasibility was also tested with this method, since the collected of a shorter recording length is preferred in a clinical setting.

The beat annotations were performed by a cardiologist, and each beat was labeled as an a wave after processing the PPG signals with a second-order bandpass Butterworth filter from 0.5–8 Hz of the unfiltered PPG signal, based on the recommendation in 11. There were a total of 584 heartbeats founds in signals measured before exercise (during rest). A total of 885 heartbeats were founds in the PPG signals collected post one hour of exercise and also contained fast rhythm PPG signals.

Methodology

Detection of a wave. In this study, the a wave detection algorithm published in 12 will be used. Elgendi proposed an algorithm in 201311 that was used to detect systolic waves in PPG signals, QRS complexes in ECG signals13, and to detect c, d, and e waves in APG signals14. The algorithm is adapted for this paper and the same approach will be utilized to detect the a waves, which has four stages: bandpass filtering, squaring, generating potential blocks, and thresholding and uses five parameters F1, F2, MApeak, MAbeat, and β. Each stage is explained in more detail as follows.

  • Bandpass filter: A zero-phase second-order Butterworth filter, with bandpass 0.5–15 Hz, was implemented to remove frequency components that do not contribute to the a wave, such as the baseline wander and high frequencies. A filtered S[n] signal was produced by applying the output of the zero-phase Butterworth filter the PPG signal—at rest and post exercise—produced a filtered signal Therefore, F1 = 0.5 Hz and F2 = 15 Hz based on a brute force search that is discussed in the parameter optimization section in 12.

  • Second derivative: The second derivative was used on the filtered PPG to obtain and analyze the APG signals. Non-causal filters are represented by the Equation 1 and Equation 2. Using a delay of only two samples, the three-point centre derivative was created.

S[n]=dSdt|t=nT=12T(S[n+1]S[n1]),(1)

Z[n]=dSdt|t=nT=12T(S[n+1]S[n1]),(2)

  • T is the sampling interval and equals the reciprocal of the sampling frequency and n is the number of data points.

  • Cancellation of b wave: Emphasis of the APG a wave is necessary to make it distinct for detection, which can be accomplished by cutting out the negative parts of the APG signal (Z[n] = 0, if Z[n] < 0).

  • Squaring: Large differences that result from a wave are emphasized by squaring. Small differences from the diastolic wave and noise are suppressed, resulting in the output.

  •                                              y[n] = Z[n]2,           (3)

  • This is important for improving the accuracy in distinguishing the a wave segment in APG signals.

  • Generating blocks of interest: Using two event-related moving averages, blocks of interest are created, that demarcate the a wave and heartbeat areas.

  • In this procedure, the first moving average (MApeak) is used to emphasize the a wave area, and is given by

MApeak[n]=1W1(y[n(W11)/2]+...+y[n]+...+y[n+(W11)/2]),(4)

  • where W1 represents the window size of the systolic-peak duration. The resulting value is rounded to the nearest odd integer. The exact value for W1 of 175 ms is determined based on a brute force search and is further discussed in the parameter optimization section in 12.

  • The beat area is emphasized by the second moving average (MAbeat) that will be used as a threshold for the first moving average, and is given by

MAbeat[n]=1W2(y[n(W21)/2]+...+y[n]+...+y[n+(W21)/2]),(5)

  • where W2 represents a window size of approximately one beat duration and is rounded to the nearest odd integer. A brute force search is used to determine the exact value of W2 of 1000 ms and is further in the parameter optimization section in 12.

  • Thresholding: Determining the offset level is done using the following equation: (α) is β z, where β = 0 based on a brute force search and is discussed further in the parameter optimization section in 12, while z is the statistical mean of the squared filtered PPG signal. The first dynamic threshold value was calculated by shifting the MAbeat signal with an offset level α, as follows:

  •                  THR1 = MAbeat[n] + α.             (6)

  • By comparing the MApeak signal with THR1 blocks of interest were generated, resulting in the generation of many blocks of interest. Some will contain the APG feature (a wave), while others will mainly contain noise. Thus, block rejection for blocks that result from noise must be carried out next and is based on the anticipated systolic-peak width. For the purpose of this paper, rejection of undesired blocks using THR2, which is a threshold that rejects the blocks containing diastolic wave and noise. Applying the THR2 threshold, will result in only accepting blocks that contain a waves only,

  •                                   THR2 = W1.             (7)

  • As previously mentioned, the threshold THR2 corresponds to the anticipated a wave duration. Classification of an a wave takes place when a block is wider than or equal to THR2, otherwise it will be classified as noise. The final step in this process is to identify the maximum absolute value within each block to detect the a wave. Note that not all the blocks hold potential a waves, since some blocks are created by noise and need to be eliminated. Rejection of blocks will also take place when a block is smaller than the expected width for the a wave duration and are considered as noisy blocks. Accepted blocks are considered to hold an a wave. Detected a waves are compared against the annotated a waves to determine whether they were correctly detected.

Performance of the a wave detector. We used two statistical measures to evaluate the a wave detection performance: sensitivity (SE) and positive predictivity (+P); whereas SE = TP/(TP + FN) and +P = TP/(TP + FP). TP represents the total number of true positives (a wave detected as an a wave), FN represents the total number of false negatives (a wave has not been detected), and FP represents the total number of false positives (non-a wave detected as an a wave). The percentage of true, correctly detected a waves by the algorithm is reported by SE. The +P reports the percentage of the detected a waves that were true a waves. MATLAB 2012a (The MathWorks, Inc., Natick, MA, USA) was used to implement the a wave detector.

Calculation of HRV indices. The detected a waves (heartbeats) are used to calculate the duration of each consecutive aa interval, as follows: aa[i] = A[i + 1] − A[i], where A represents the annotated a waves in each PPG signal, and aa represents the aa intervals. Note that the main interest is to analyze the aa duration rather than the amplitude, no preprocessing is needed. It is known that HRV decreases with normal based on the analysis of R peaks in ECG signals1517. Therefore, based on using a waves in PPG signals, if the correlation between HRV and age is decreasing, PPG signals can potentially measure HRV. The correlation between age and HRV is found by calculating and comparing two time-domain HRV parameters and are typically used with ECG signals. The first parameter, SDNN, is the SD of heartbeat duration; here, the RR interval is replaced by aa intervals. The SDNN is calculated, as follows:

SDNN=(1/N)i=1N(aa[i])2{(1/N)i=1N(aa[i])}2.(8)

The second parameter is RMSSD, which is calculated as follows:

RMSSD=(1/N)i=1N(aa[i])2.(9)

Analysis of trend As we have a small sample size, there is a need to examine both the correlation coefficient (r) and the slope. Note that it is common to obtain a small correlation with a small sample size, and therefore the slope can be used as an alternative to determine the relationship. Here, r is calculated as follows: r=Cov(u,v)σuσv, where Cov(u, v) is the covariance between data u and data v, σu is the SD of data u and σv is the SD of data v. Here, u and v refer to the HRV indices.

Significance of trend. To test the significance of the slope, we performed a multilinear regression of the responses in y on the predictors in x for all lines in Figure 4 and Figure 5. The null hypothesis states that the slope is equal to zero, and the alternative hypothesis states that the slope is not equal to zero. MATLAB 2012a (The MathWorks, Inc., Natick, MA, USA) was used to calculated the significance of trend.

Results and discussion

To date, several studies have evaluated the agreement between HRV calculated from PPG and ECG signals; however, some results remain controversial18,19. This disagreement is due to applying inappropriate methodologies or inefficient experimental settings. A recent study by Chen et al.20 showed that the accuracy of HRV obtained from PPG is mostly incommensurable across 26 unhealthy subjects. However, it has been proven that the HRV calculated from PPG signal is sufficient in its accuracy for only healthy (and mostly younger) subjects at rest21. Therefore, in our study, fit and healthy subjects were only considered. Because of this selection, it is expected that there will be no apparent differences between subjects. Moreover, we can focus on the age impact— without any dependencies such as arrhythmia, transvenous cardiac pacing, and heart transplant—as an indirect assessment for the usefulness of HRV calculated from PPG signals.

It is worth noting that the heat stress PPG data were collected for this study as a part of the project funded by the Australian Department of Defence. As mentioned in the ’Database used’ subsection, there was no exercise test with the same subjects in a cooler (or normal) environment included in the main project protocol. Therefore, we are unable comment, compare or discuss the HRV calculation in two different conditions, hot and cool environments.

The designed exercise length for the heat stress test was four hours; however, only 16 out of 27 subjects completed the full duration. All subjects were able to complete the first 1-hr exercise, and therefore we analyzed all PPG signals collected only after 1-hr exercise.

After collecting the PPG signals, perhaps the question was how can we calculate the HRV from these PPG signals more accurately. In the literature, we found that the second derivative of the PPG signal enhances its frequencies and improves waveform characteristics especially in heat stressed PPG signals22. Moreover, we found that detection of a waves in APG signals slightly increases the accuracy of detecting heart beats compared to systolic peaks in PPG signals. For example, the overall a wave detection rate was 99.9%12, while the overall systolic wave detection rate was 99.8%11. Therefore, we applied the second derivative to the PPG signal to obtain APG signals and then applied the a wave detection algorithm described in the 'Methodology’ section.

The a wave detection algorithm was evaluated using 27 records, containing 1,540 heartbeats (584 heartbeats measured at rest and 956 heartbeats measured after 1-hr exercise), with an overall SE of 99.8%, and the overall +P of 100%. The overall accuracy was sufficient enough to calculate the HRV indices automatically.

After the automatic detection of a waves, SDNN and RMSSD indices are calculated for 27 healthy subjects using PPG recordings each of 20 seconds duration during rest and after exercise. Figure 4(a) & (b) show the relationship between age and the SDNN index at rest and after exercise, respectively. The SDNN index at rest is more negatively correlated with age (r = −0.271) and has a steeper negative slope (−0.004) than after exercise (r = −0.12 and slope = −0.001). The slope significance for SDNN before exercise was p = 0.001, and the slope significance for SDNN after exercise was p = 0.04.

00c6e344-0b97-47b4-a41c-b2f7ef51cdec_figure4.gif

Figure 4. Correlation between age and SDNN index.

(a) Age and SDNN calculated from PPG signals for all subjects measured at rest, (b) age and SDNN calculated from PPG signals for all subjects measured after exercise. It is clear that the SDNN index is more negatively correlated with age for 20-second PPG signals measured at rest compared to after-exercise measurements. Here, the p-value shows the significance for testing the slope of the linear regression.

Figure 5(a) shows the relationship between the age and the RMSSD index at rest and Figure 5(b) shows the relationship between the age and the RMSSD index after exercise. The RMSSD index at rest is more negatively correlated with age (r = −0.217) and has a more negative slope (−0.004) than the RMSSD index after exercise (r = −0.091 and slope = −0.001). The slope significance was achieved by RMSSD before exercise with p = 0.04, while it failed for RMSSD after exercise as p = 0.18.

00c6e344-0b97-47b4-a41c-b2f7ef51cdec_figure5.gif

Figure 5. Correlation between age and RMSSD index.

(a) Age and RMSSD calculated from PPG signals for all subjects measured at rest, (b) age and RMSSD calculated from PPG signals for all subjects measured after exercise. It is clear that the RMSSD index is more negatively correlated with age for 20-second PPG signals measured at rest compared to after-exercise measurements. Here, the p-value shows the significance for testing the slope of the linear regression.

It is worth mentioning that the remainder of the correlations are not as strong, specifically the correlation between SDNN and age and that between RMSSD and age. Nevertheless, the slope demonstrates significance between HRV indices measured at rest and after exercise. The combination of the correlation coefficient and the slope provides a more precise evaluation for the trend analysis. Note that PPGs measured at rest have a greater negative slope compared to those measured after exercise. Although it was known that long-term exercise exerts significant effects on the HRV23, it has not been investigated over short PPG signals. This study is important because it reports the effect of exercise on HRVs calculated from short PPG signals and compared their results in subjects measured at rest and after exercise.

The results of various cross-sectional studies have shown a linear decrease in HRV during exercise with increasing age using only ECG signals8,15,24. Interestingly, our results confirm the inverse linear relationship between HRV measures (SDNN and RMSSD) and age. Moreover, SDNN is statistically significant with age for PPG measured before and after exercise. This new outcome shows that HRV can potentially be measured using short PPG signals.

Healthy subjects were the focus of this study and the proposed method was successfully implemented only on these subjects. Subject physiology changes significantly depending on the current health status, and this was typically reflected in the PPG signal. Thus, due to these changes, the robustness of the proposed algorithm needs to be tested on and verified on recordings from unhealthy subjects. The next step, based on results from this study, is to examine the HRV indices calculated from PPG signals in the diagnosis and monitoring of abnormalities, such as arrhythmia, diabetes, hypertension, and hyperlipidemia. The HRV indices are usually calculated over a period of five minutes from the ECG signals; however, the PPG recordings in this study were very short (20 seconds). Additional studies to demonstrate the 20-second recordings are equivalent in terms of usefulness to the standard length of time (5 minutes) are needed to further validate our findings. Studies that evaluate the extracted HRV indices as a function of PPG duration are recommended. The PPG database contains signals collected from subjects of approximately the same age (about 22 years old), which creates imbalance in the age distribution. Therefore, a larger and more diverse sample size with a balanced age distribution is needed to generalize the findings of this study.

Conclusion

The findings of this preliminary study build on previous studies that discussed the potential use of only PPG to measure HRV (i.e., without collecting ECG data). Suitable indices for short duration signals include SDNN and RMSSD, which can be used with 20-second PPG recordings. A negative correlation with age at rest has been demonstrated for both indices. Interestingly, only SDNN shows significance for after exercise measurements. Since long signal measurement can be challenging due to perspiration, calculating SDNN using short PPG measurements from subjects after exercise in hot and humid weather is an appealing approach that needs to be explored on a larger scale. The overall result of this study indicates that an alternative potential modality for HRV analysis and the identification of individuals at risk, is the use of short PPG signals.

Data availability

In agreement with the Australian Department of Defence, the raw dataset could not be made available because the data could not be sufficiently anonymised to protect subject confidentiality. Upon request and review, the dataset is available at CDU (http://www.cdu.edu.au/ehse). The PPGs are human subject data and will be made available upon request from appropriate investigators after approval by Charles Darwin University Human Research Ethics Committee (http://www.cdu.edu.au/research/ori/human-ethics).

Comments on this article Comments (0)

Version 1
VERSION 1 PUBLISHED 22 Sep 2016
Comment
Author details Author details
Competing interests
Grant information
Copyright
Download
 
Export To
metrics
Views Downloads
F1000Research - -
PubMed Central
Data from PMC are received and updated monthly.
- -
Citations
CITE
how to cite this article
Elgendi M, Norton I, Brearley M et al. A pilot study: Can heart rate variability (HRV) be determined using short-term photoplethysmograms? [version 1; peer review: 2 approved with reservations]. F1000Research 2016, 5:2354 (https://doi.org/10.12688/f1000research.9556.1)
NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.
track
receive updates on this article
Track an article to receive email alerts on any updates to this article.

Open Peer Review

Current Reviewer Status: ?
Key to Reviewer Statuses VIEW
ApprovedThe paper is scientifically sound in its current form and only minor, if any, improvements are suggested
Approved with reservations A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.
Not approvedFundamental flaws in the paper seriously undermine the findings and conclusions
Version 1
VERSION 1
PUBLISHED 22 Sep 2016
Views
13
Cite
Reviewer Report 10 Oct 2016
Albert Avolio, Department of Biomedical Sciences, Macquarie University, Sydney, NSW, Australia 
Approved with Reservations
VIEWS 13
The authors propose a means of detecting heart rate variability (HRV) from the photoplethysmographic (PPG) signal. As presented, the study is only marginally informative and the following issues require consideration.
 
  1. From the references cited
... Continue reading
CITE
CITE
HOW TO CITE THIS REPORT
Avolio A. Reviewer Report For: A pilot study: Can heart rate variability (HRV) be determined using short-term photoplethysmograms? [version 1; peer review: 2 approved with reservations]. F1000Research 2016, 5:2354 (https://doi.org/10.5256/f1000research.10293.r16498)
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.
Views
19
Cite
Reviewer Report 03 Oct 2016
Mariusz Oszust, Department of Computer and Control Engineering, Rzeszów University of Technology, Rzeszów, Poland 
Approved with Reservations
VIEWS 19
In the paper, heart rate variability (HRV) is determined using the photoplethysmogram (PPG) signal. The demonstrated findings are based on the calculation of the standard deviation of heartbeat interval (SDNN) and the root-mean square of the difference of successive heartbeats ... Continue reading
CITE
CITE
HOW TO CITE THIS REPORT
Oszust M. Reviewer Report For: A pilot study: Can heart rate variability (HRV) be determined using short-term photoplethysmograms? [version 1; peer review: 2 approved with reservations]. F1000Research 2016, 5:2354 (https://doi.org/10.5256/f1000research.10293.r16497)
NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article.

Comments on this article Comments (0)

Version 1
VERSION 1 PUBLISHED 22 Sep 2016
Comment
Alongside their report, reviewers assign a status to the article:
Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested
Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.
Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions
Sign In
If you've forgotten your password, please enter your email address below and we'll send you instructions on how to reset your password.

The email address should be the one you originally registered with F1000.

Email address not valid, please try again

You registered with F1000 via Google, so we cannot reset your password.

To sign in, please click here.

If you still need help with your Google account password, please click here.

You registered with F1000 via Facebook, so we cannot reset your password.

To sign in, please click here.

If you still need help with your Facebook account password, please click here.

Code not correct, please try again
Email us for further assistance.
Server error, please try again.