MSL: Facilitating automatic and physical analysis of published scientific literature in PDF format

Published scientific literature contains millions of figures, including information about the results obtained from different scientific experiments e.g. PCR-ELISA data, microarray analysis, gel electrophoresis, mass spectrometry data, DNA/RNA sequencing, diagnostic imaging (CT/MRI and ultrasound scans), and medicinal imaging like electroencephalography (EEG), magnetoencephalography (MEG), echocardiography (ECG), positron-emission tomography (PET) images. The importance of biomedical figures has been widely recognized in scientific and medicine communities, as they play a vital role in providing major original data, experimental and computational results in concise form. One major challenge for implementing a system for scientific literature analysis is extracting and analyzing text and figures from published PDF files by physical and logical document analysis. Here we present a product line architecture based bioinformatics tool ‘Mining Scientific Literature (MSL)’, which supports the extraction of text and images by interpreting all kinds of published PDF files using advanced data mining and image processing techniques. It provides modules for the marginalization of extracted text based on different coordinates and keywords, visualization of extracted figures and extraction of embedded text from all kinds of biological and biomedical figures using applied Optimal Character Recognition (OCR). Moreover, for further analysis and usage, it generates the system’s output in different formats including text, PDF, XML and images files. Hence, MSL is an easy to install and use analysis tool to interpret published scientific literature in PDF format.


Introduction
There has been an enormous increase in the amount of the scientific literature in the last decades 1 . The importance of information retrieval in the scientific community is well known; it plays a vital role in analyzing published data. Most published scientific literature is available in Portable Document Format (PDF), a very common way for exchanging printable documents. This makes it all-important to extract text and figures from the PDF files to implement an efficient Natural Language Processing (NLP) based search application. Unfortunately, PDF is only rich in displaying and printing but requires explicit efforts in the extraction of information, which significantly impacts the search and retrieval capabilities 2 . Due to this reason several document analysis based tools have been developed for physical and logical document structure analysis of this file type.
PubMed and some other publishing platforms (e.g. DOAJ, Google Scholar) provide search options to locate relevant published manuscripts but do not claims to search over the full-text literature and images. The recently, provided basic information retrieval (IR) system by PubMed is efficient in extracting literature based on published text (titles, authors, abstracts, introduction etc.), with the application of automatic term mapping and Boolean operators 3 . The normal outcome of a successful NLP and text based query brings a maximum of 20 relevant results per page; however, user can improve the search by customizing the query using the provided advanced options. So far, the current PubMed system, as well many other related system are unable to completely implement an efficient information retrieval system, capable of extracting both text and figures from published PDF files. One of the major and technical challenges is the availability of structured text and figures. To our limited knowledge, there still is no single tool available which can efficiently perform both physical and logical structure analysis of all kinds of PDF files and can extract and classify all kinds of information (embedded text from all kinds of biological and scientific published figures). Different commercial and free downloadable software applications provide support in extracting the text and images from PDF files: A-PDF (http://www.a-pdf.com/image-extractor/), PDF Merge Split Extract (http://www.pdf-technologies.com/pdflibrary-merge-split.aspx), BePDF (http://haikuarchives.github.io/BePDF/), KPDF (https:// kpdf.kde.org), MuPDF (http://mupdf.com), Xpdf tool (http://www.foolabs.com/ xpdf/), Power PDF (http://www.nuance.com/for-business/imaging-solutions/ document-conversion/power-pdf-converter/index.htm) However, these software applications do not provide text and images in a form where they could be considered for further logical analysis e.g. mining text in reading order from double or multiple columns documents (the text of first column followed by the text of second column, and so on), searching marginal text using keywords, removing irrelevant graphics and extracting embedded text inside single and multi-panel complex biological images.
So far, the current PubMed system as well many other related orthodox NLP approaches e.g. 4-13, are unable to completely implement an efficient information retrieval system, capable of extracting both text and figures from published PDF files.
To meet the technological objectives of this challenge, we took a step forward in the development of a new user friendly, modular and client based system (MSL) for the extraction of full and marginal text from PDF files based on the keywords and coordinates ( Figure 1). Since MSL provides a module for the extraction of figures from PDF files and applies Optical Character Recognizer (OCR) to extract text from all kinds of biomedical and biological Images. MSL comprises three modules working in productline architecture: Text, Image and OCR ( Figure 2). Each module performs its task independently and its output is used as an input for the next module. It can be configured on Microsoft Windows platforms following a simple six-step installation process.

Methods
MSL extracts text and figures from the published scientific literature and helps in analyzing embedded text inside figures. The overall methodological implementation and workflow of the MSL is divided into two processes: (I) Text mining and (II) Image analysis. MSL is a desktop application, designed and developed following the scientific software engineering principles of three-layered Butterfly 14 software development model.

Text mining
Physical and logical document analysis is one of the living challenges. To the best of the authors' knowledge, there is no solution available which can perform efficient physical and logical structural analysis of PDF files, implement completely correct rendering order and classify text in all possible categories e.g. Tile, Abstract, Headings, Figure Captions, Table Captions, Equations, References, Headers, Footers etc.
However, there are some tools available which are helping in this regard e.g. PDF2HTML towards contextual modeling of logical labelling 15 , PDF-Analyzer for object level document analysis 16 , XED for hidden structure analysis 2 , Dolores for the logical structure analysis and recovery 17 automatic conversation from PDF to XML 18 and PDF to HTML 19 , Layout-aware 20 etc.

Figure 1. Graphical user interfaces of MSL and modular workflow.
This figure shows the graphical user interface and modular workflow of three main components: Text, Image and OCR. A PDF document 21 is input and processed by MSL. Text module provides extracted, searched and marginalized text in reading order, and file attributes.
Image component provides the preview of extracted images from the document. OCR component provides extracted text from selected and processed image.
We developed MSL's Text module, which is capable of processing PDF files with single, double or multiple columns. It divides the system's text based output in four sub-modules: full text, marginal text, keyword based extracted text and file attributes. Full text gives the complete text from PDF file, marginal allows user to give the coordinates (Lower Left X, Lower Left Y, Upper Right X and Upper Right Y) and extract the desired portion of the text from the PDF file. The keyword based text allows user to extract the information from PDF file based on keywords and respective coordinates (Left, Top, Width, Height) e.g. if a user is only interested in getting the figure caption or references, this kind of search will be helpful. The last sub module, File attributes gives the information about input file including title, author, creator, producer, subject, creation date, keywords, modified, number of pages and number of figures.
While implementing Text module, we researched and tried different available commercial and freely downloadable libraries with a focus on full text extraction, marginal text extraction, keyword based text extraction and text extraction from embedded images from PDF files. We tried different implemented systems and libraries (Table 1) e.g. iTextSharp Bytescout, Spire PDF Sautinsoft PDF Focus Dynamic PDF, PDFBox, iText PDF, QPDF, PoDoFo, Haru PDF Library, JPedal, SVG Imprint, Glance PDF Tool Kit, BCL SharpPDF etc.
One of the common problems in almost all libraries is merging and mixing of text, using double or multiple columns. Our developed system is the combination of different libraries, useful for different purposes. We have used Spire PDF to remove the Book-marks, iTextSharp for the extraction of full and marginal text, Bytescoute for the keyword based marginalized text search and producing output in the form of XML file ( Figure 2). The generated XML file contains structured (tagged) text along with the information about its coordinates (placement in the file), font (Bold, Italic etc.) and size, which can be used for mapping and pattern recognition tasks.

Image processing
Image-based analysis is a versatile and inherently multiplexed approach as it can quantitatively measure biological images to detect those features, which are not easily detectable by a human eye. Millions of figures have been published in scientific literature that includes information about results obtained from different biological and medicinal experiments. Several data and image mining solutions have been already implemented, published and are in use in the last 15 years 22 . Some of the mainstream approaches are towards the analysis of all kinds of images (flow charts, experimental images, models, geometrical shapes, graphs, images of thing or objects, mixed etc.). There are not many approaches proposed for specific kinds of image-analysis e.g. towards the identification and quantification of cell phenotypes 23 , prediction of subcellular localization of proteins in various organism 24 , analysis of gel diagrams 25 , mining and integration of pathway diagrams 26 .
While implementing a new data-mining tool, one of our goals was to extract images from published scientific literature and try to extract embedded text as well. We analyzed different freely available and commercial OCR systems and libraries including Aspose, PUMA, Microsoft OCR, Tesseract, LEADTOOLS, Nicomsoft OCR, MeOCR OCR, OmniPage, ABBYY, Bytescout claiming to be able to extract embedded text from figures. During our research we found LEADTOOLS ( Figure 2) as one of the best available solutions for this purpose. MSL is capable of automatically extracting images from the PDF files and allowing the user to apply OCR to any extracted image by clicking and enlarging it for a better view (using Windows default image viewer).

Results and discussion
We tested MSL with similar parameters on randomly selected scientific manuscripts ( Here, in the case of text extraction we observed that the text was in reading order when using manuscripts from F1000Research and IEEE but text was without spaces in the manuscript from PLOS and with additional lines and extra spaces in the manuscript from Hindawi. In the case of figure extraction we observed one common problem among the four manuscripts from these publishers; along with the manuscript images (Figures), embedded journal or publishers' logos and images were also extracted. Additionally, while analyzing the manuscript from F1000Research, we observed that the images were broken into many pieces and it was not possible to find one single complete image. As we did not test all manuscripts from the mentioned publishers, we cannot claim that the results will be the same for all papers from a publisher, as the output may vary in Raw dataset is attached to this manuscript, which categorically provides all images and text in XML format, extracted from manuscripts (from different publishers (included in file names)) using MSL 37 .
To apply MSL, published scientific literature has first to be downloaded in the form of a PDF file, from any published source. The validation process using MSL consists of three major steps: 1) Text mining, 2) Image extraction, and 3) Application of OCR to extract text from selected images as shown in Figure 1, following the implemented workflow as shown in Figure 2. Example results and graphics are shown in Figure 1, Figure 3 and Figure 4. Representation includes the extraction of text and images from one of the randomly selected papers 21 , and application of OCR to one of the extracted images from another randomly picked publication 27 . Figure 1 shows that one randomly selected published article's PDF file 21 is inputted to the MSL's text, the extracted text is divided into three categories (i) complete text in excellent rendering order (ii) marginalized text and (iii) keyword based searched text. Two figures (Figure 1 and Figure 2) are extracted and displayed in the image section, and one of those is selected to apply OCR.
The applied OCR extracts textual information, which is displayed in and can be exported in a PDF file.
To further validate the application of OCR and discuss different results, Figure 3 show another example of embedded text extraction from a complex figure 38 , which includes three panels of images (i) colorful pie and circle charts, (ii) biological images and (iii) tabular information. Similar to our prior application of OCR, results are displayed in textual form as well as generated PDF file of extracted text. A noticeable difference between both outputs is that the textual information is presented in line-byline order whereas in the PDF file the information is displayed in margins with respect to the original image.
The last resultant example is based on the validation of MSL by extracting the textual information from image based PDF files. We produced an image form of one of the randomly selected article 27 and then processed one of pages. As Figure 4 shows, the obtained results were comprehensive in both textual as well as the PDF form. This kind of textual extraction can be very helpful, especially when the literature is available in only images e.g. in the case of old published literature in print only format but electronically available in scanned form. MSL produces several files as system output in the parent folder of the files. These files are: XML files (which include structured or tagged information), an Images File (extracted from the PDF file) and PDF files for all analyzed images using OCR. Additionally, extracted text can be split into different IMRAD parts and results can be searched and categorized e.g. based on abstract, introduction, methods, discussion, conclusion and references.
We mentioned earlier that we have tried and implemented different libraries for text and image extraction and analysis. The best text based outcome was observed using iTextSharp, better image extraction was observed using Spire and OCR from LEADTOOLS was the most promising. While validating the implemented solution, other than the expected results (text and images), we observed some limitations in the used libraries: unexpected and irrelevant images were also extracted e.g. journal, publisher's logos and header-footer images embedded inside document (e.g. images added by the publishers, to provide publishing details), text was not always in good rendering order, especially when there were text-based mathematical equations with super and subscripts; and in case of double or multicolumn PDF files, most of the libraries' rendering order is not correct. During extracting text, we found that some important symbols were missed and spaces were generated for some paragraphs. We found that it was not possible to extract particular images that are created as a combination of different sub-images and text objects in the manuscript. In these cases, text is found in extracted text area and all extracted sub-images are image sections, with the possibility of missing some sub-images as well. Moreover, when we applied OCR to different images (extracted or loaded), we found that its performance does vary with respect to the complexity of inputted images. In case of special characters (e.g. Greek delta, alpha, beta etc.), it does not perform well unless these are hard wired in the software.
In comparison to earlier mentioned tools; MSL possess some advantages as well as limitations. For instance, Dolores help user in adding custom tags to the PDF document and create semantic model associated to the processed class of documents, PDF2HTML implements conditional random fields (CRF) based model to learn semantics from processed PDF page's content, PDF-Analyzer devised a model based on rectangular objects for the analysis on PDF documents, XED applies method to combine PDF symbol analysis with traditional document image processing technique. MSL does not apply any of these methods and support such features. However, MSL does support segmentation of text, provides text in correct reading order, enable users with keywords based search and provide extraction of embedded text from figures (using OCR), which none of these tools does. To enhance the functionality of the MSL program (e.g. our standard version available here for download), we give a table of the most often used special symbols in biomedical literature (Table 3). Depending on your application in mind, you thus simply extend the MSL parser by considering also these special characters occurring often in your texts.
One in-house example is the DrumPID database 39 , where different types of data and images are warehoused by us and an improved separation and retrieval of text versus figure legends, image descriptions etc. is highly useful and currently applied. The latest version of DrumPID allows understanding and screening of compounds for their effects in protein interaction networks. It is helpful in exploring potential antibiotic lead structures, studing individual pathways and potential targets in various organisms.  page of one of the randomly selected papers 21 is processed using OCR (LEADTOOLS). Text is extracted from the image and a new PDF is generated, which is based on the text, placed with similar margins to the image file. We compared (Table 4) MSL with with different, earlier discussed related software applications: A-PDF, PDF Merge Split Extract, BePDF, KPDF, MuPDF, Xpdf tool, Power PDF, PDF2HTML, PDF-Analyzer, XED, Dolores and Layout-aware. The chosen and compared tools were picked based on the following criteria: 1) able to extract text from single, double and multiple column PDF files, 2) able to extract images from from single, double and multiple column PDF files, 3) provide options to search text and image based elements, 4) help in segmentation of text and images, 4) open source, 5) commercial, 6) freely available, 7) easily customizable, and 8) additionally, capable of analyzing embedded images with the application of OCR. There is not mathematical or statistical ranking of these tools were performed as most of these shared common features and obtained results were close. The reason to draw comparison is not at all to undermine the importance of any other available software applications but tried to justify the importance of MSL, as it's the only one among which is fully open source, capable of extracting text and images from PDF files, applies OCR to get text from extracted chosen images, allow users to search, easily customizable (users can replace libraries) and freely available.

Implementation & operation
MSL architecture is based on the Product Line Architecture (PLA) and Multi-Document Interface (MDI) developmental principles, and it is designed and developed (using C-Sharp programming language, Microsoft Dot NET Framework) following the key principles of Butterfly paradigm 14,38 . The work-flow of MSL is divided into two processes: (I) extraction and marginalization of text with respect to the division and placement of text in PDF file and keyword based search by using the iTextSharp, Bytescoute, Spire PDF libraries, and (II) extraction and analysis of figures by using the Spire PDF library and LEADTOOLS OCR.
It takes Portable Document Format (PDF) based literature files as input, performs partial physical structure analysis, and exports output in different formats e.g. text, images and XML files. It allows user to extract keywords and marginal (X and Y coordinates) information based text, have PDF file's metadata information (title, author, creator, producer, subject, creation date, keywords, modified, number of pages and number of figures) and save extracted full and marginal text in text files.  Biomedical image extraction and analysis is one of the most complex tasks from the field of computer sciences and image analysis. Some of the mainstream approaches [40][41][42][43][44][45] have been proposed towards the analysis of all kinds of images (e.g. flow charts, experimental images, models, geometrical shapes, graphs, imageof-thing, mix etc.). MSL allows user to automatically extracting images from the PDF files, let any selected image viewed via Windows default image viewer and apply implemented OCR.
Other than extract images from PDF file, MSL allow user to load any image, apply OCR and export output in readable PDF file.
MSL produces several out files in the parent folder including XML files (which include structured or tagged information), Images File (extracted from PDF file) and PDF files for all analyzed images using OCR ( Figure 5).
MSL application is very simple to install and use. It was tested and can be well configured on a Microsoft Windows platform (preferred OS version: 7). MSL follows a simple six steps installation process ( Figure 6). After installation, it can be run by either clicking on the installed application's icon at the desktop or execute application following sequence of steps: Start → All Programs → MSL 1.0.0 → MSL.
Regarding using the MSL application, one important point to remember is that it is based on different PDF text extraction, marginalization and figure extraction libraries, which are automatically configured during installation but used OCR by the LEAD-TOOLS is not a freely available library, which we have used upon academic research (free) license. The OCR library is also automatically configured during installation but its performance at different (non-licensed) machines is not confirmed. Moreover, the recommended display screen resolution size is 1680×1050 with landscape orientation.

Conclusions
As most of the publishers are publishing in simple HTML and PDF formats, its not possible to segment and analyze raw form literature using available commercial and open-source software applications, as those are helpful in mainly text and image extraction. It will be helpful to have literature available in semantically tagged formats (e.g. XML, OWL etc.), so literature can be efficiently parsed, categorized and searched.
The development of a virtual research environment to store and link molecular data, can be well achieved and established if first the mixture of text, protocols and omics data is properly separated from images, figures and figure legends -a task for which our tool can be well suited. There are a number of databases (e.g.

Alzheimer's Disease Neuroimaging Initiative (ADNI); Breast Cancer Digital Repository (BCDR); BiMed; Public Image Databases; Cancer Image Database (caIMAGE); COllaborative Informatics and Neuroimaging Suite (COINS); DrumPID; Digital Database for Screening Mammography (DDSM); Electron Microscopy Data Bank (EMDB); LONI image data archive; Mammography Image Databases (MID); New Database Provides Millions of Biomedical Images; Open Access Series of Imaging Studies (OASIS); Stanford Tissue Microarray Database (TMA); STRING; The Cancer
Imaging Archive (TCIA); Whitney Imaging Center etc.) which can directly profit from MSL by fast, automatic and rapid separation of text and text description from images and figure legends describing the images is important for further improvement of the database and its content.
The latest available and easy to use version of MSL has been tested and validated in-house. The advancements in information retrieval techniques for text and figure analysis combined with this sophisticated computational tool can support various studies.

Software access
The software executable is freely available at the following web link: https://zenodo.org/record/30941#.Vi0PtmC5LHM The software download section provides one executable: MSL, setup to be installed on the Microsoft Windows platform.
MSL has been NOT been developed for any commercial purposes but as a non-commercial prototype application for academic research, analysis and development purposes.

License
All associated files are licensed under the Academic Free License 3.0 (AFL 3.0).

Author contributions
ZA: developed the complete solution (including research, software designing, programming, testing, deployment and technical documentation). TD guided the study. All authors participated in writing of the manuscript and approved the final manuscript for publication.

Competing interests
No competing interests were disclosed. The manuscript has been moderately improved by last two revisions. My main concern was that the work was not significantly new although it was claimed so. Authors tried to tone down their claim of novelty during the revisions. Authors added more comparisons and improved the readability of figures. While I am not fully convinced on the current version of the manuscript, I believe that the tool still could be useful for the community as it combines several features together.

Grant information
No competing interests were disclosed. Competing Interests: Unfortunately I remain unconvinced about the contribution of the work presented in this manuscript. The results are presented through case studies rather than formal evaluation.

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
Claims are made relating to various capabilities of the system that are not justified; for instance, the statement that the tool can split the text into "various IMRAD parts" is neither explained (in terms of Methodology --does this simply involve searching for specific key words such as "Abstract" or "Introduction"?) nor directly evaluated. There are no quantitative evaluations of performance that support the claims that are made, only statements that the tool has been "validated in-house". This does not constitute a scientific evaluation of the tool.
Closer inspection of the Text box of Figure 1 suggests that the text is not in natural reading order; I see several lines from separate columns juxtaposed. The XML file produced for this text retains the columnar structure and shows The Lipid-Pro application can be configured on Microsoft Windows The Lipid-Pro application can be configured on Microsoft Windows immediately followed by supported by Waters qTOF instrument platforms, acquires intact in the next column field without any straightforward way to extract the linear order of the text; the second sentence here clearly does not follow on from the first. I don't see how this format would be useful at all to anyone wanting to do natural language processing on this text, as the authors state. (NB: information retrieval and natural language processing are not the same and the textual representation assumed for NLP is very different than for IR; it is very important to retain the natural text flow and sentence structure for NLP tasks.) The premise of the work is still not adequately justified; the journals examined no longer (if they ever did; I doubt PLoS or PeerJ, for instance, ever did) publish work as images requiring OCR of the whole document. Lumping HTML and PDF together (as "simple formats") is clearly inappropriate; they have very different characteristics and indeed in HTML the text is nearly always in straightforward linear structure. However, not all PDFs are created equal and most PDFs have perfectly extractable (non-image/non-scanned) text these days through tools such as PDFbox. The authors have not responded to my prior comment that many publishers are already making documents available as nicely structured XML, in a format far preferable to what the authors are producing, due to richer provided semantic structure.
The extraction and analysis of images themselves may be useful; doing OCR of text in an included image could be valuable. The authors claim that this is unique to their tool. It may be, and this may be the most useful part of their tool. However it is unclear how well the tool does this; a close look at Figure 1 identifies some clear errors, e.g. "Coiled" rather than "Collect"; "Nud" rather than "Nucl"; etc. What level of error is tolerable?
No competing interests were disclosed.

Competing Interests:
Referee Expertise: Biomedical natural language processing, text mining I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. This manuscript addresses the analysis of scientific articles published in PDF form, and introduces a software tool that essentially wraps a number of other tools to build a single tool that integrates a number of features.
There is no novel methodological contribution in the tool itself that I could determine; the value is primarily in integrating the other tools into a user-facing tool. However, the value of a user-facing tool (as opposed to an automated tool that proceeds without human input) is not made clear or evaluated (are users happy to do the work expected of them?).
As a very high level point on motivation, many scientific articles --particularly in the open access literature which form the majority of the studied papers --are already available as raw XML and it does not make sense to me to try to parse the PDF to infer structure (in a way that first requires user interaction, and second may introduce errors) that can be read directly from the publisher-produced XML. The PubMed Central repository is a repository of full-text, available online as structured HTML, and the open access collection which is downloadable in its entirety uses XML ( ). So in my opinion several of the key objectives of this https://www.ncbi.nlm.nih.gov/pmc/tools/openftlist/ tool are not needed for a large and growing proportion of the scientific literature. The argument could be made that older articles and articles from certain publishers are not available as "raw" XML, but the author neither make this argument nor specifically test this question. In addition, this XML resource provides a fantastic opportunity to automatically test (some aspects of) the performance of the authors tool. This is not done. The authors should consider the contribution of their tool in the context of such resources. The Introduction focuses on PubMed, but PubMed does not claim to search over the full-text literature (it is limited to abstracts by design), and PubMed Central is not mentioned here.
There is one more tool that I am aware of --and I suspect there are others --that the authors do not mention; the Layout-Aware PDF tool . In general, I'd be interested to understand more deeply how the authors' tool differs from the various tools they mention, and why they selected the tools they did for their final system.
The Results presented by the authors are not framed in any well-defined evaluation framework. While they mention "best overall performance" no indication of either specific quantitative or qualitative criteria used to assess the tool have been provided. What were the criteria that were used? Were there guidelines for how to "score" systems on various criteria? What is presented is largely ad hoc qualitative observations. On what basis was performance of the tool ranked?

Comments on presentation:
It is poor practice to cite papers that are purely studied as artifacts --a citation implies that you are referencing scientific content which you are not. Please remove citations to the papers that were used to test the system; they should be listed (preferably with DOIs) in Table 2, with a paper ID, and the paper IDs should be referenced in the article.
There are a number of phrases in the manuscript that are awkward. "marginalization" does not mean "to find margins". Queries in PubMed are not "NLP quer[ies]" but rather queries processed by an IR user system (NB: arguably, IR systems don't even use NLP). The authors talk about "orthodox NLP approaches"; I have no idea what makes an NLP approach "orthodox". I suspect the authors mean "pipeline" rather than "product line". What is an "inherently multiplexed approach"?
The long list of databases in the Conclusions doesn't add much to the manuscript; perhaps the example of the DrumPID should be pulled into a discussion section, and elaborated to clearly demonstrate the practical application and value of the tool.
Thank you so much for your recommendations.

Reply:
This manuscript addresses the analysis of scientific articles published in PDF form, and introduces a software tool that essentially wraps a number of other tools to build a single tool that integrates a number of features. There is no novel methodological contribution in the tool itself that I could determine; the value is primarily in integrating the other tools into a user-facing tool. However, the value of a user-facing tool (as opposed to an automated tool that proceeds without human input) is not made clear or evaluated (are users happy to do the work expected of them?). As a very high level point on motivation, many scientific articles --particularly in the open access literature which form the majority of the studied papers --are already available as raw XML and it does not make sense to me to try to parse the PDF to infer structure (in a way that first requires user interaction, and second may introduce errors) that can be read directly from the publisher-produced XML. The PubMed Central repository is a repository of full-text, available online as structured HTML, and the open access collection which is downloadable in its entirety uses XML (https://www.ncbi.nlm.nih.gov/pmc/tools/openftlist/).
Thank you so much for your views.

So in my opinion several of the key objectives of this tool are not needed for a large and growing proportion of the scientific literature. The argument could be made that older articles and articles from certain publishers are not available as "raw" XML, but the author neither make this argument nor specifically test this question. In addition, this XML resource provides a fantastic opportunity to automatically test (some aspects of) the performance of the authors tool. This is not done. The authors should consider the contribution of their tool in the context of such resources. The Introduction focuses on PubMed, but PubMed does not claim to search over the full-text literature (it is limited to abstracts by design), and PubMed Central is not mentioned here.
We thank you so much for nice suggestion and have tried to revise manuscript accordingly.

Reply:
There is one more tool that I am aware of --and I suspect there are others --that the authors do not mention; the Layout-Aware PDF tool 1. In general, I'd be interested to understand more deeply how the authors' tool differs from the various tools they mention, and why they selected the tools they did for their final system.
We thank you so much for nice suggestion and have tried to revise manuscript accordingly.

Reply:
The Results presented by the authors are not framed in any well-defined evaluation framework. While they mention "best overall performance" no indication of either specific quantitative or qualitative criteria used to assess the tool have been provided. What were the criteria that were used? Were there guidelines for how to "score" systems on various criteria? What is presented is largely ad hoc qualitative observations. On what basis was performance of the tool ranked?
We thank you so much for nice suggestion and have tried to answer question in revision.

Reply:
It is poor practice to cite papers that are purely studied as artifacts --a citation implies that you are referencing scientific content which you are not. Please remove citations to the papers that were used to test the system; they should be listed (preferably with DOIs) in Table 2, with a paper ID, and the paper IDs should be referenced in the article.
We thank you so much for nice suggestion and have added DOIs.

Reply:
We thank you so much for nice suggestion and have added DOIs.

Reply:
There are a number of phrases in the manuscript that are awkward. "marginalization" does not mean "to find margins". Queries in PubMed are not "NLP quer [ies]" but rather user queries processed by an IR system (NB: arguably, IR systems don't even use NLP). The authors talk about "orthodox NLP approaches"; I have no idea what makes an NLP approach "orthodox". I suspect the authors mean "pipeline" rather than "product line". What is an "inherently multiplexed approach"?
We thank you so much for nice suggestion and have tried to revise manuscript accordingly.

Reply:
The long list of databases in the Conclusions doesn't add much to the manuscript; perhaps the example of the DrumPID should be pulled into a discussion section, and elaborated to clearly demonstrate the practical application and value of the tool. later (Results). Problems of other tools mentioned in the Introduction, such as "removing irrelevant graphics" are not solved by MSL either.
I think the comparison with other tools should be presented in terms of what these fail to detect while MSL does not, and vice versa. E.g. For a particular article "pdftotext" was not able to recognize the columns while MSL does, etc.
In summary, better putting this system into the context of existing ones.
The system is implemented as an interactive tool intended for analyzing a single article at a time, even presenting some of the final results (e.g. text extracted from images) back as PDF. For a single article (or a small number) human inspection will be better than any automated system. I guess the real potential of this system is in the automated parsing of large collections of articles. In my opinion the authors should focus the manuscript more on that.

Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool? Yes
Are the conclusions about the tool and its performance adequately supported by the findings presented in the article? Partly No competing interests were disclosed.

Competing Interests:
I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
Author Response 29 Mar 2018 , University of Connecticut Health Center, USA Zeeshan Ahmed Thank you so much for your recommendations.

Reply:
direction, such as that presented in this work, is of potential interest.
Thank you so much for your views and we agree with you.

Reply:
My main concern with this work is that most of the features described are already available in existing software, even those described as "new". For example, tools like "pdftotext" can parse multi-column PDFs, "pdfimages" extract the images within a PDF file, "OCRFeeder" detects the image elements and extract the texts, tables, etc, … Thank you so much and we respect your points that many competing applications do exist Reply: but that still doesn't not negatively impact our contributions.
On the contrary, some potentially newer features of MSL, such as recognizing the article parts (Abstract, References, … -page 3-) mentioned in Methods are not described later (Results).
We thank for nice suggestion and have tried to revise manuscript accordingly.

Reply:
Problems of other tools mentioned in the Introduction, such as "removing irrelevant graphics" are not solved by MSL either.
Thank you so much and we respect your point that it's not a complete solution, as future Reply: research and development is also recommended but still helps at some good levels.
I think the comparison with other tools should be presented in terms of what these fail to detect while MSL does not, and vice versa. E.g. For a particular article "pdftotext" was not able to recognize the columns while MSL does, etc.
We thank for nice suggestion and have added comparison.

Reply:
In summary, better putting this system into the context of existing ones.
We thank for nice suggestion and have tried to revise manuscript accordingly. Reply: The system is implemented as an interactive tool intended for analyzing a single article at a time, even presenting some of the final results (e.g. text extracted from images) back as PDF. For a single article (or a small number) human inspection will be better than any automated system. I guess the real potential of this system is in the automated parsing of large collections of articles. In my opinion the authors should focus the manuscript more on that.
We agree with you and thank so much for nice suggestion. We mainly tried to focus on that Reply: part but as one of the examples to show the strength of system, we have included that.
We thank you so much for your time and excellent suggestions, which have helped us in improving the manuscript.
With best wishes, Authors  (BIOKDD 2003) No competing interests were disclosed.

Competing Interests:
I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
Reader Comment 11 Apr 2017 , University of Connecticut Health Center, USA Zeeshan Ahmed Thank you so much for your recommendations.

Reply:
In this manuscript authors presented a computational tool that extracts text and images from PDF files. In general the manuscript is interesting considering that it can analyze various types of PDF files from different scientific areas based on the keywords and coordinates. Thanks.

Reply:
However, it lacks technical novelty over the published literatures and needs additional input on the image analysis section before indexing.
Thanks for raising this point, we have revised and tried to make it more clearer Reply: . Extraction of texts and images from scientific publications has been presented in various domains: computer science1, biomedical2-4, chemistry5, proteomics6 and so on. The manuscript by Zeeshan Ahmed and Thomas Dandekar presents an incremental innovation without providing clear technological advancement in the field. The objective of performing both physical and logical structure analysis of all kinds of PDF files as mentioned in the manuscript has not been sufficiently supported by technological contribution described in Methods section.
Thanks for raising this point and please accept apologies for the confusion. Its true that Reply: there have been many efforts from the past towards the same problem, and from different disciplines. We have taken a facile step by combining some of available technologies and tried to present a method, which can be adapted and further enhanced. To make our point more clearer, we have revised manuscript. The image processing section the manuscript has been very brief. It does not provide any advanced image analysis technique as mentioned in the abstract. Authors should mention how exactly segmentation of figures and labels are performed and how they are represented to make logical connection between different entities in order to perform further analysis and customized visualization.
Thanks for raising this point and please accept apologies for the confusion. We agree with Reply: your point. We didn't discuss and go in to details of algorithmic image presenting because we didn't implement any algorithm for this work but tested and adopted some pre-existing OCR based libraries. We choose to go for commercial and licensed libraries because open source libraries and methods (we found) were unable to meet the developmental objectives this software. Without such details it is not possible to draw comparative (algorithmic, metrics based) conclusions. However, we give now a feature-based comparison in the results and discussion section of the manuscript. The framework has been tested with a very small set of PDF files and no qualitative/quantitative result reporting the accuracy with respect to manually annotated files was presented. It would be good to increase the number test files and include the results of qualitative/quantitative analysis. result reporting the accuracy with respect to manually annotated files was presented. It would be good to increase the number test files and include the results of qualitative/quantitative analysis.
Thanks for raising this point and we have revised with additional details. We tested our Reply: system with similar parameters on randomly selected scientific manuscripts (ten PDF files) from different open access ( F1000Research, Frontiers, PLOS, Hindawi, PeerJ, BMC) and restricted access ( Oxford University Press, Springers, Emerald, Bentham Science, ACM) publishers. Some of the figures (Figures 1, 3, 4 and 6) in the manuscript are hard to see the details in both online and print format. These figures could be reformatted.
Thanks for raising this point and we have revised Figures.

Reply:
No competing interests.
No competing interests were disclosed.

Competing Interests:
I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
Reader Comment 11 Apr 2017 , University of Connecticut Health Center, USA Zeeshan Ahmed Thank you so much for your recommendations.

Reply:
The manuscript titled "MSL: Facilitating automatic and physical analysis of published scientific literature in PDF format" addresses an important issue of extracting information from published manuscripts. Thanks.

Reply:
However, the following issues must be clarified before indexing. Sure.

Reply:
In the text mining section authors say that there is no tool to perform physical and logical structural analysis of PDF files. However, in the next paragraph they describe "Dolores" for logical structure analysis. Authors should describe how their method is different than Dolores.
Thanks for the nice suggestion. We have provided a brief comparison with Dolores, as well Reply: as some other mentioned tools in the paper. This includes Dolores, PDF2HTML, XED, and PDF-Analyzer, too and is now mentioned first time when mentioning Dolores. Legends of all the figures should be more descriptive so that figures are understandable on their own. Each component of the figure should be described in the legend.
Thanks for pointing this out, we have revised the manuscript and added more details to the Reply: figure legends explaining the symbols used. The results section is missing. Is it integrated in the discussion section? It is unclear what exactly the results were.
Yes, we have an integrated results and discussion section, please accept apologies for this Reply: confusion. To further clarify it, we have revised the heading titles and stressed the results achieved and subsequently discussed by subtitles. The article will be much clear if all the libraries (described on page 4 second paragraph) are described in the form of a table.
Thanks for this important suggestion. We have added a table to the manuscript showing all Reply: libraries we tested for MSL as well as whether they are partly or completely integrated in MSL. (i) Authors should include a clear metric to estimate performance of the algorithm. This can be achieved by comparison with existing tools or through comparative analysis. (ii) A clear example showing the information extracted from several PDF files to address a biologically relevant example will be useful.
Thanks for these valuable suggestions. Reply: (i). Initially, we also aimed to perform such comparative analysis, but most of the used and tested libraries are from different commercial and licensed sources, and algorithmic details were not given. We choose to go for commercial and licensed libraries because open source libraries and methods (we found) were unable to meet the developmental objectives this software. Without such details it is not possible to draw comparative (algorithmic, metrics based) conclusions. However, we give now a feature-based comparison in the results and discussion section of the manuscript. This also clearly shows the advantages of the MSL software. Furthermore, we discuss also the limitations and possible extensions of our MSL software, again referring to other existing software.