Problematizing Digital Research Evaluation using DOIs in Practice-Based Arts, Humanities and Social Science Research

ORCID, Definitions of Data in AHSS Research

At Goldsmiths, University of London, not many researchers have created their individualised ORCID numbers yet and neither do they, by and large, label their data with digital object identifiers (DOIs), apart from possibly their journal articles, which of course contain processed data. This is because many researchers active in arts, humanities and social science (AHSS) research do not identify their research outputs (e.g. field notes, monographs, art work sketches, film rushes, interview materials etc.) as research data. The term ‘data’, to them has a much narrower definition, belonging only to the professional jargon used by researchers working in (bio) medical and other scientific fields (e.g. quantitative, numerical data sets, graphs and pie charts). Other AHSS researchers see ‘data’ as only being synonymous with personal data: personal information about research participants, such as names and dates of birth, subject to safeguarding by the data protection act.

The definition of what might be classified as research data, however, could be much broader if conceived of creatively. The first mission of any digital advocate therefore, is to clarify with the help of researchers themselves, what might count as research data in their disciplines and how this could be labelled appropriately, digitised, archived and maintained as necessary, to document research processes and methods. This process of defining and scoping research data must meet the needs of project-specific research enquiries without letting administrative requirements take the upper hand or imposing non-AHSS data collection models where this is not appropriate.

To some extent the question of what counts as data in visual arts research was already addressed at Goldsmiths. Between 2011–2013, Goldsmiths participated in a Joint Information Systems Committee (JISC) funded project called KAPTUR together with Glasgow School of Art, University for the Creative Arts and University of the Arts London. KAPTUR built on the work undertaken by the Digital Curation Centre and was led by the Visual Arts Data Service. The KAPTUR project sought to: “investigate the current state of the management of research data in the arts; develop a model of best practice applicable to both specialist arts institutions and arts departments in multidisciplinary institutions; and to apply, test and embed the model with four institutional partners (http://www.vads.ac.uk/kaptur/about.html)”. The project helped question definitions of data and provide examples of what might count as data and how it might be managed.

KAPTUR’s findings, useful toolkits, reports and outcomes have not filtered through to most visual arts researchers however and few therefore have embraced the idea that the definition of research data can be broadened. Those that have, however: are struggling with IP and copyright issues for digitised work; have no time to digitise analogue research outputs to be able to create digital object identifiers; or may not publish in the digital domain, meaning that digital identification and metrics cannot currently be used effectively to assess any potential impact being created.

Consequently, many AHSS researchers feel ORCID numbers and digital object identifiers are simply not relevant to what they currently do. As much AHSS research remains unfunded there are also no contractual or funder obligations to which researchers must adhere which stipulate that research data must be open access or discoverable. This reluctance to engage with digital dissemination and citation practices is especially in evidence when researchers work in disciplines that still rely on the production of practice-based outputs and the publication of monographs as the most important esteem indicators and research outputs in their fields.

Monographs in AHSS Research

Goldsmiths’ researchers publish many monographs. Monographs contain research data: images of art works; creative writing outputs; and (auto) biographical details for example. The author is not always the copyright owner of this data or indeed its creator, and will often have to gain permission to use information for the purposes of publishing their monograph. What is owned by the author is the intellectual theory and often text.

Academic monographs are still predominantly issued in non-open access formats. If published in the digital domain, proprietary formats and specific software are used. These are unlikely to promote sustainability of digital research data in the open access domain. The challenge is compounded by the current lack of digital research data for monographs that might easily be labelled and referenced electronically. Here it is worth referring to the January 2015 Crossick report on open access monographs generated by the Higher Education Funding Council England (Hefce).

Whilst recognising the importance of monographs to AHSS scholars, the Hefce report identified the limitations of hard copy formats. Those relevant to us here include: a) the fact that video, audio and other examples cannot be embedded in hard-copy monographs; b) text-mining options and easy ways of measuring citation levels and impact rates are absent; c) the fact that hard copy monographs are not ‘living’ documents. Comments and reviews cannot be easily shared, updates require new editions and comparisons of passages and ideas are not quickly communicated. It is these limitations of the monograph that researchers at Goldsmiths are wishing to explore in collaboration with publishers, computing and legal experts.

The challenges identified by the Hefce report with regards to labelling research data and making monographs open access are several. Those highlighted by the report are, for example, that academically authored exhibition catalogues are part of business models for Independent Research Organisations (IROs) and galleries. Making exhibition catalogues and the research data openly accessible will reduce the vital income received by IROs such as the Tate Galleries. Additionally, data in catalogues and creative writing outputs have often been generated by people other than the catalogue author. This raises copyright, intellectual property and revenue challenges impacting on the licensing and sub-licensing of research data such as images and musical examples if researchers wanted to include certain materials in their open access monograph using digital object identifiers. Careful consideration must therefore be given to labelling research data before joining it to researcher ORCID numbers and making it open access if in monograph or catalogue format.

The analogue, licencing, authorship and ethics

Questions of research ethics, integrity and licencing also come in to play when labelling practices are considered. Pictures, images and text may constitute to a representation of something or someone. Not the actual object or person. Practice-based researchers often argue that a (digital) image or recording of their analogue, and possibly temporaneous art work is a different object epistemologically to the actual, physical work and therefore cannot be labelled as being the same item. Similarly, where open access is an option, anthropologists and ethnographers frequently opt for non-derivative licences meaning no materials based data/text can be created derived from the original text. This is important where non-academic research collaborators agree to participate on the basis that they are represented fairly and where these collaborators often have a say in how their interview excerpts, musical materials and images (that is to say, research data collected by the researcher) are used and placed in texts. Re-using research data uncritically, or labelling it with object identifiers often runs contrary to the highly personalised material that is being explored, which belongs to both the researcher and his/her participants at the very least and in some cases to the research participant alone, who is sharing it with the researcher in good faith. AHSS authors therefore choose the most restrictive licensing options in order to do no harm. This ethical priority reduces the re-use and therefore citation options available for their data and as a result the potential for metrical impact.

Questions of authorship and citation surface as well. In the sciences definitions of what constitutes ‘authorship’ vary across disciplines and between journals. Citation practices are also not standardised. Guidelines do exist, however. By way of contrast, in AHSS research very few, if any, definitions and guidelines of what constitutes to authorship exist. If one were to apply certain biomedical models of authorship definition to AHSS research, many non-academic research participants would technically qualify as co-authors of research papers as they helped shape research data via their active, sometimes non-anonymous, participation in the research enquiry, particularly in applied, bottom-up, process-oriented research enquiries.

Whilst acknowledging research participants’ input and possible co-authorship of research papers and data may be a more accurate and ethical reflection of their role in the research, it could also potentially create logistical challenges in the domains of IP, copyright, ethics and citation. Ethical considerations need to inform citation and author definition practices. Whilst it might seem like a good idea to measure impact through non-academic authorship, citation and engagement in the way hinted at above, this approach should not be recommended without careful ethical screening addressing questions of anonymity and equitable data sharing and ownership that are likely to arise, amongst many other hurdles.

Metrics, impact and digital dissemination in AHSS

Another challenge to be overcome is that of the use of metrics in assessing research quality and impact. ORCIDs and DOIs will be especially helpful in collecting statistical information quickly and digitally on how often research is being cited, and accessed and may go some way, so the argument goes, to showing how much impact is being achieved. Metrics however, is only one way in which impact becomes measurable and for AHSS researchers, it is thought to be misleading and ineffective. In a response to Hefce’s consultation on the use of metrics in assessment of research quality and impact, many Goldsmiths staff remained unconvinced that metrics could be used to conclusively prove research excellence or impact. Implicitly therefore, they had little faith in the use of DOIs and ORCID numbers as a way of improving impact analyses through metrics, although some did agree it would improve the visibility of research outputs and data. Most felt though that discoverability should not be equated with quality or impact per se.

Metrics such as citation rates and journal impact factors, they observed, operate differently not only according to discipline (e.g., between psychology and literature), but also differ significantly between varying branches of the same discipline. Psychology, for example, is arguably a more diverse discipline than some, so indices like citation factors need to be interpreted very carefully even within different sub-disciplines to allow for meaningful 'like for like' comparisons (e.g., neuroscience journals typically have much higher impact factors than social psychology journals).

The Media, Communication and Cultural Studies Association’s (MeCCSA) REF consultation, compiled by Prof Golding and reiterated in the Goldsmiths’ response to the Hefce metrics consultation by MeCCSA members employed at Goldsmiths, yielded various anonymised comments from researchers. It was pointed out that evidence “suggests that variations in citation practices occur within disciplines as much as across disciplines, so the issue of calibration cannot simply be to the average for the subject as is proposed in science subjects. Staff felt it difficult to envisage a reliable way in which to develop disciplinary citation norms in interdisciplinary areas against which to compare individual counts. This would be especially true in fields such as media and communications which encourage publication across a very wide range of outlets in the arts, humanities, and social sciences. The usual suggestion of Web of Science (Thomson Scientific) as the database to be used for calibration of citation counts is acknowledged to be problematic. Web of Science is demonstrably incomplete in many areas (http://www.meccsa.org.uk/pdfs/REF-Consultation.pdf)”. Colleagues were unconvinced any database was complete and therefore figures not necessarily accurate or useful.

Others Goldsmiths’ colleagues felt that in AHSS research the number of citations was not a conclusive indicator of research merit, value or impact. A colleague commented that in the humanities (literary studies in particular), no publication is ever really superseded or made obsolete, nor is the author-critic irrelevant to the argument; the argument is very often his/her interpretation and appreciation of certain phenomena; in the sciences the assumption is that scientists report hard facts/results of experiments, not their idiosyncratic and poetic take on the set of data. Once a set of data or theory is superseded, scientific research tends no longer be cited or becomes part of what ‘everybody already knows’ (see Latour & Woolgar, 1986). This is not true for a lot of AHSS research and so the citation of data or the linking of outputs with ORCIDs might be of limited use it was felt.

The intellectual debates surrounding digital dissemination that are flourishing at Goldsmiths will inform practical digital dissemination strategies that the University as a whole will adopt in future. Intellectually these same debates seek to influence emerging theory in digital scholarship and dissemination in AHSS subjects more generally.

Interdisciplinarity, impact, Goldsmiths and digital dissemination

Despite this scepticism and caution, new developments have begun to flourish. During recent data management scoping exercise, it became apparent that Goldsmiths researchers are engaging with digital dissemination practices quite effectively. This is especially the case where researchers are working on projects that are interdisciplinary, well-funded and usually include an element of non-AHSS research.

For example, Goldsmiths hosts a large Arts and Humanities, Research Council (AHRC) grant. Its research team actively seek to use and create scientific computing tools to collect and analyse large amounts of data to help further musicological analyses and practice. One such undertaking is the ‘Transforming Musicology’ project (Box 1).

The project relies heavily on the digital labelling of musical units to help catalogue and identify compositional structures and musical pieces. The research has the potential to inform debates on musical performance, copyright, composition and musical analysis amongst other areas. To this end it will develop open source software tools as well. The process of identifying and labelling musical units with URIs means that this project has a large number of data sets and individual data items which might potentially be cited and accessible in audio format in the planned monograph for this large grant. The grant’s research team are presently considering the possibility of engaging with publishers to explore open access monograph formats so that they might include digital data sets. The creation of digital data sets and an open access monograph, in turn, provide a significant impetus for the research team to consider adopting ORCID numbers so that URIs and DOIs might be linked to their names and the grant. If data sets and digital tools were made available they would also benefit non-academic and amateur music groups, such as the lute-players with which the principal researcher works. For open access sharing to be made a reality however, careful consideration will need to be given to how research data sets are shared in the monograph and whether or not this might contravene existing copyright legislation, for example, as data is compiled from existing musical pieces. Until adequate sharing mechanisms are explored, it may not be possible to freely share the data accumulated during the project’s lifespan.

Another initiative taken by Goldsmiths’ researcher Joanna Zylinska and her team (Professor Joanna Zylinska, Dr Kamila Kuc, Jonathan Shaw, Ross Varney, Dr Michael Wamposzyc. Project advisor: Professor Gary Hall), includes the creation of an open book, Photomediations. The project redesigns a coffee-table book as an online experience to produce a creative resource that explores the dynamic relationship between photography and other media. Photomediations: An Open Book uses open (libre) content, drawn from various online repositories (Europeana, Wikipedia Commons, Flickr Commons) and tagged with the CC-BY licence and other open licences. In this way, the book showcases the possibility of the creative reuse of image-based digital resources.

Through a comprehensive introduction and four specially commissioned chapters on light, movement, hybridity and networks that include over 200 images, Photomediations: An Open Book tells a unique story about the relationship between photography and other media. The book’s four main chapters are followed by three ‘open’ chapters, which will be populated with further content over the next 18 months. The three open chapters are made up of a social space, an online exhibition and an open reader. A version of the reader, featuring academic and curatorial texts on photomediations, will be published in a stand-alone book form later in 2015, in collaboration with Open Humanities Press.

Photomediations: An Open Book’s online form allows for easy sharing of its content with educators, students, publishers, museums and galleries, as well as any other interested parties. Promoting the socially significant issues of ‘open access’, ‘open scholarship’ and ‘open education’, the project also explores a low-cost hybrid publishing model as an alternative to the increasingly questioned traditional publishing structures. Photomediations: An Open Book is a collaboration between academics from Goldsmiths, University of London, and Coventry University. It is part of Europeana Space, a project funded by the European Union's ICT Policy Support Programme under GA n° 621037. It is also a sister project to the curated online site Photomediations Machine: http://photomediationsmachine.net. This example provides a good model of how AHSS researchers in the visual arts might approach the production of open access monographs and arts catalogues, where licencing and copy right issues are very much foregrounded.

A third example of Goldsmiths engagement with open access dissemination and data is that of work led by Dr Jennifer Gabrys and her team on an ERC funded project called Citizen Sense in the Sociology department (Box 2).

As part of their studies the research team on Citizen Sense collect live scientific data on for example air quality, using sensor devices. The team has now developed a website which visualises air quality data so that the general public can view the crowd sourced results. The interdisciplinary scope of this large project therefore means that the research data generated comes in a format that is more akin to data generated in science environments as opposed to AHSS disciplines. Therefore the collection, labelling, storage and archiving of this data using DOIs and attaching these to ORCIDs might therefore usefully draw on practices established in non-AHSS domains. This in turn could potentially enhance the visibility and impact of this research both environmentally and academically.