MSL: Facilitating automatic and physical analysis of published scientific literature in PDF format

Introduction

There has been an enormous increase in the amount of the scientific literature in the last decades¹. The importance of information retrieval in the scientific community is well known; it plays a vital role in analyzing published data. Most published scientific literature is available in Portable Document Format (PDF), a very common way for exchanging printable documents. This makes it all-important to extract text and figures from the PDF files to implement an efficient Natural Language Processing (NLP) based search application. Unfortunately, PDF is only rich in displaying and printing but requires explicit efforts in the extraction of information, which significantly impacts the search and retrieval capabilities². Due to this reason several document analysis based tools have been developed for physical and logical document structure analysis of this file type.

PubMed and some other publishing platforms (e.g. DOAJ, Google Scholar) provide search options to locate relevant published manuscripts but do not claims to search over the full-text literature and images. The recently, provided basic information retrieval (IR) system by PubMed is efficient in extracting literature based on published text (titles, authors, abstracts, introduction etc.), with the application of automatic term mapping and Boolean operators³. The normal outcome of a successful NLP and text based query brings a maximum of 20 relevant results per page; however, user can improve the search by customizing the query using the provided advanced options. So far, the current PubMed system, as well many other related system are unable to completely implement an efficient information retrieval system, capable of extracting both text and figures from published PDF files. One of the major and technical challenges is the availability of structured text and figures. To our limited knowledge, there still is no single tool available which can efficiently perform both physical and logical structure analysis of all kinds of PDF files and can extract and classify all kinds of information (embedded text from all kinds of biological and scientific published figures). Different commercial and free downloadable software applications provide support in extracting the text and images from PDF files:

A-PDF (http://www.a-pdf.com/image-extractor/),

PDF Merge Split Extract (http://www.pdf-technologies.com/pdf-library-merge-split.aspx),

BePDF (http://haikuarchives.github.io/BePDF/), KPDF (https://kpdf.kde.org),

MuPDF (http://mupdf.com), Xpdf tool (http://www.foolabs.com/xpdf/),

Power PDF (http://www.nuance.com/for-business/imaging-solutions/document-conversion/power-pdf-converter/index.htm)

However, these software applications do not provide text and images in a form where they could be considered for further logical analysis e.g. mining text in reading order from double or multiple columns documents (the text of first column followed by the text of second column, and so on), searching marginal text using key-words, removing irrelevant graphics and extracting embedded text inside single and multi-panel complex biological images.

So far, the current PubMed system as well many other related orthodox NLP approaches e.g. 4–13, are unable to completely implement an efficient information retrieval system, capable of extracting both text and figures from published PDF files.

To meet the technological objectives of this challenge, we took a step forward in the development of a new user friendly, modular and client based system (MSL) for the extraction of full and marginal text from PDF files based on the keywords and coordinates (Figure 1). Since MSL provides a module for the extraction of figures from PDF files and applies Optical Character Recognizer (OCR) to extract text from all kinds of biomedical and biological Images. MSL comprises three modules working in product-line architecture: Text, Image and OCR (Figure 2). Each module performs its task independently and its output is used as an input for the next module. It can be configured on Microsoft Windows platforms following a simple six-step installation process.

Figure 1. Graphical user interfaces of MSL and modular workflow.

This figure shows the graphical user interface and modular workflow of three main components: Text, Image and OCR. A PDF document²¹ is input and processed by MSL. Text module provides extracted, searched and marginalized text in reading order, and file attributes. Image component provides the preview of extracted images from the document. OCR component provides extracted text from selected and processed image.

Figure 2. Conceptual architecture of MSL and component’s workflow.

This figure shows the conceptual architecture of the MSL application, which consists of three main components: Text, Image and OCR, and nine sub-components: Text File, Image File, Visualize Image, PDF File, LEADTOOLS, XML File, iTEXTSharp, Bytescout, Spire. As figure shows, Text component applies iTEXTSharp, Bytescout, Spire to extract the text from PDF document and write output in XML file. Image components applies Spire to extract images from the PDF document and visualize that using Visualize Image. OCR component applied LEADTOOLS to extract text from images and export that to PDF format.

Methods

MSL extracts text and figures from the published scientific literature and helps in analyzing embedded text inside figures. The overall methodological implementation and workflow of the MSL is divided into two processes: (I) Text mining and (II) Image analysis. MSL is a desktop application, designed and developed following the scientific software engineering principles of three-layered Butterfly¹⁴ software development model.

Results and discussion

We tested MSL with similar parameters on randomly selected scientific manuscripts (ten PDF files) from different open access (F1000Research, Frontiers, PLOS, Hindawi, PeerJ, BMC) and restricted access (Oxford University Press, Springers, Emerald, Bentham Science, ACM) publishers, including some of the authors’ published papers, details are given in Table 2. While testing MSL on the selected manuscripts, we observed best overall performance for the manuscripts^27–32, with satisfactory results from almost all publishers (including Oxford University Press, BMC, Frontiers, PeerJ, Bentham Science, ACM) in terms of both extracting text in reading order and extracting images. An observed poor performance involved manuscripts from PLOS³³, Hindawi³⁴, F1000Research³⁵ and IEEE³⁶ publishers. Here, in the case of text extraction we observed that the text was in reading order when using manuscripts from F1000Research and IEEE but text was without spaces in the manuscript from PLOS and with additional lines and extra spaces in the manuscript from Hindawi. In the case of figure extraction we observed one common problem among the four manuscripts from these publishers; along with the manuscript images (Figures), embedded journal or publishers’ logos and images were also extracted. Additionally, while analyzing the manuscript from F1000Research, we observed that the images were broken into many pieces and it was not possible to find one single complete image. As we did not test all manuscripts from the mentioned publishers, we cannot claim that the results will be the same for all papers from a publisher, as the output may vary in different papers. Our observed results using MSL are given in attached supplementary material (Supplementary Table S1 and Dataset 1).

To apply MSL, published scientific literature has first to be downloaded in the form of a PDF file, from any published source. The validation process using MSL consists of three major steps: 1) Text mining, 2) Image extraction, and 3) Application of OCR to extract text from selected images as shown in Figure 1, following the implemented workflow as shown in Figure 2. Example results and graphics are shown in Figure 1, Figure 3 and Figure 4. Representation includes the extraction of text and images from one of the randomly selected papers²¹, and application of OCR to one of the extracted images from another randomly picked publication²⁷.

Figure 3. Example: Publication, Figure 1 of (YY et al., 2015).

This figure shows document image analysis, text extraction and PDF conversion. A figure (based on three panels; including two charts, one image and a table) is selected from one of the randomly selected papers 2⁶. OCR (LEADTOOLS) is applied to extract and report the text from the figure in simple text form (section: Extracted Text from Figure) and in PDF file with similar margins to the original figure (section: Exported text in PDF format).

Figure 4. Example: Publication, Page 1 (Ahmed et al., 2015).

This figure shows document image analysis, text extraction and PDF conversion. First, scanned image based page of one of the randomly selected papers²¹ is processed using OCR (LEADTOOLS). Text is extracted from the image and a new PDF is generated, which is based on the text, placed with similar margins to the image file.

Figure 1 shows that one randomly selected published article’s PDF file²¹ is inputted to the MSL’s text, the extracted text is divided into three categories (i) complete text in excellent rendering order (ii) marginalized text and (iii) keyword based searched text. Two figures (Figure 1 and Figure 2) are extracted and displayed in the image section, and one of those is selected to apply OCR. The applied OCR extracts textual information, which is displayed in and can be exported in a PDF file.

To further validate the application of OCR and discuss different results, Figure 3 show another example of embedded text extraction from a complex figure³⁸, which includes three panels of images (i) colorful pie and circle charts, (ii) biological images and (iii) tabular information. Similar to our prior application of OCR, results are displayed in textual form as well as generated PDF file of extracted text. A noticeable difference between both outputs is that the textual information is presented in line-by-line order whereas in the PDF file the information is displayed in margins with respect to the original image.

The last resultant example is based on the validation of MSL by extracting the textual information from image based PDF files. We produced an image form of one of the randomly selected article²⁷ and then processed one of pages. As Figure 4 shows, the obtained results were comprehensive in both textual as well as the PDF form. This kind of textual extraction can be very helpful, especially when the literature is available in only images e.g. in the case of old published literature in print only format but electronically available in scanned form. MSL produces several files as system output in the parent folder of the files. These files are: XML files (which include structured or tagged information), an Images File (extracted from the PDF file) and PDF files for all analyzed images using OCR. Additionally, extracted text can be split into different IMRAD parts and results can be searched and categorized e.g. based on abstract, introduction, methods, discussion, conclusion and references.

We mentioned earlier that we have tried and implemented different libraries for text and image extraction and analysis. The best text based outcome was observed using iTextSharp, better image extraction was observed using Spire and OCR from LEADTOOLS was the most promising. While validating the implemented solution, other than the expected results (text and images), we observed some limitations in the used libraries: unexpected and irrelevant images were also extracted e.g. journal, publisher’s logos and header-footer images embedded inside document (e.g. images added by the publishers, to provide publishing details), text was not always in good rendering order, especially when there were text-based mathematical equations with super and subscripts; and in case of double or multicolumn PDF files, most of the libraries’ rendering order is not correct. During extracting text, we found that some important symbols were missed and spaces were generated for some paragraphs. We found that it was not possible to extract particular images that are created as a combination of different sub-images and text objects in the manuscript. In these cases, text is found in extracted text area and all extracted sub-images are image sections, with the possibility of missing some sub-images as well. Moreover, when we applied OCR to different images (extracted or loaded), we found that its performance does vary with respect to the complexity of inputted images. In case of special characters (e.g. Greek delta, alpha, beta etc.), it does not perform well unless these are hard wired in the software.

In comparison to earlier mentioned tools; MSL possess some advantages as well as limitations. For instance, Dolores help user in adding custom tags to the PDF document and create semantic model associated to the processed class of documents, PDF2HTML implements conditional random fields (CRF) based model to learn semantics from processed PDF page’s content, PDF-Analyzer devised a model based on rectangular objects for the analysis on PDF documents, XED applies method to combine PDF symbol analysis with traditional document image processing technique. MSL does not apply any of these methods and support such features. However, MSL does support segmentation of text, provides text in correct reading order, enable users with keywords based search and provide extraction of embedded text from figures (using OCR), which none of these tools does. To enhance the functionality of the MSL program (e.g. our standard version available here for download), we give a table of the most often used special symbols in biomedical literature (Table 3). Depending on your application in mind, you thus simply extend the MSL parser by considering also these special characters occurring often in your texts.

One in-house example is the DrumPID database³⁹, where different types of data and images are warehoused by us and an improved separation and retrieval of text versus figure legends, image descriptions etc. is highly useful and currently applied. The latest version of DrumPID allows understanding and screening of compounds for their effects in protein interaction networks. It is helpful in exploring potential antibiotic lead structures, studing individual pathways and potential targets in various organisms.

We compared (Table 4) MSL with with different, earlier discussed related software applications: A-PDF, PDF Merge Split Extract, BePDF, KPDF, MuPDF, Xpdf tool, Power PDF, PDF2HTML, PDF-Analyzer, XED, Dolores and Layout-aware. The chosen and compared tools were picked based on the following criteria: 1) able to extract text from single, double and multiple column PDF files, 2) able to extract images from from single, double and multiple column PDF files, 3) provide options to search text and image based elements, 4) help in segmentation of text and images, 4) open source, 5) commercial, 6) freely available, 7) easily customizable, and 8) additionally, capable of analyzing embedded images with the application of OCR. There is not mathematical or statistical ranking of these tools were performed as most of these shared common features and obtained results were close. The reason to draw comparison is not at all to undermine the importance of any other available software applications but tried to justify the importance of MSL, as it’s the only one among which is fully open source, capable of extracting text and images from PDF files, applies OCR to get text from extracted chosen images, allow users to search, easily customizable (users can replace libraries) and freely available.

Implementation & operation

MSL architecture is based on the Product Line Architecture (PLA) and Multi-Document Interface (MDI) developmental principles, and it is designed and developed (using C-Sharp programming language, Microsoft Dot NET Framework) following the key principles of Butterfly paradigm^14,38. The work-flow of MSL is divided into two processes: (I) extraction and marginalization of text with respect to the division and placement of text in PDF file and keyword based search by using the iTextSharp, Bytescoute, Spire PDF libraries, and (II) extraction and analysis of figures by using the Spire PDF library and LEADTOOLS OCR.

It takes Portable Document Format (PDF) based literature files as input, performs partial physical structure analysis, and exports output in different formats e.g. text, images and XML files. It allows user to extract keywords and marginal (X and Y coordinates) information based text, have PDF file’s metadata information (title, author, creator, producer, subject, creation date, keywords, modified, number of pages and number of figures) and save extracted full and marginal text in text files.

Biomedical image extraction and analysis is one of the most complex tasks from the field of computer sciences and image analysis. Some of the mainstream approaches^40–45 have been proposed towards the analysis of all kinds of images (e.g. flow charts, experimental images, models, geometrical shapes, graphs, image-of-thing, mix etc.). MSL allows user to automatically extracting images from the PDF files, let any selected image viewed via Windows default image viewer and apply implemented OCR. Other than extract images from PDF file, MSL allow user to load any image, apply OCR and export output in readable PDF file.

MSL produces several out files in the parent folder including XML files (which include structured or tagged information), Images File (extracted from PDF file) and PDF files for all analyzed images using OCR (Figure 5).

Figure 5. Screenshot of the all extracted images and generated files (XML and PDF).

This figure shows different files generated during analysis of PDF document. PDF file (top, left) is the actual document, XML file is the structured (tagged) form of extracted text, second PDF file (top, right) is the extracted text from image (see Figure 3) and all other files are extracted image from PDF document.

MSL application is very simple to install and use. It was tested and can be well configured on a Microsoft Windows platform (preferred OS version: 7). MSL follows a simple six steps installation process (Figure 6). After installation, it can be run by either clicking on the installed application’s icon at the desktop or execute application following sequence of steps: Start → All Programs → MSL 1.0.0 → MSL.

Figure 6. MSL six steps installation process.

Regarding using the MSL application, one important point to remember is that it is based on different PDF text extraction, marginalization and figure extraction libraries, which are automatically configured during installation but used OCR by the LEADTOOLS is not a freely available library, which we have used upon academic research (free) license. The OCR library is also automatically configured during installation but its performance at different (non-licensed) machines is not confirmed. Moreover, the recommended display screen resolution size is 1680×1050 with landscape orientation.

Conclusions

As most of the publishers are publishing in simple HTML and PDF formats, its not possible to segment and analyze raw form literature using available commercial and open-source software applications, as those are helpful in mainly text and image extraction. It will be helpful to have literature available in semantically tagged formats (e.g. XML, OWL etc.), so literature can be efficiently parsed, categorized and searched.

The development of a virtual research environment to store and link molecular data, can be well achieved and established if first the mixture of text, protocols and omics data is properly separated from images, figures and figure legends – a task for which our tool can be well suited. There are a number of databases (e.g. Alzheimer’s Disease Neuroimaging Initiative (ADNI); Breast Cancer Digital Repository (BCDR); BiMed; Public Image Databases; Cancer Image Database (caIMAGE); COllaborative Informatics and Neuroimaging Suite (COINS); DrumPID; Digital Database for Screening Mammography (DDSM); Electron Microscopy Data Bank (EMDB); LONI image data archive; Mammography Image Databases (MID); New Database Provides Millions of Biomedical Images; Open Access Series of Imaging Studies (OASIS); Stanford Tissue Microarray Database (TMA); STRING; The Cancer Imaging Archive (TCIA); Whitney Imaging Center etc.) which can directly profit from MSL by fast, automatic and rapid separation of text and text description from images and figure legends describing the images is important for further improvement of the database and its content.

The latest available and easy to use version of MSL has been tested and validated in-house. The advancements in information retrieval techniques for text and figure analysis combined with this sophisticated computational tool can support various studies.

Data availability

F1000Research: Dataset 1. Extracted Images and Text from Papers tested using MSL, 10.5256/f1000research.7329.d108739³⁷

Software availability