Ask a clearer question , get a better answer

Many undergraduate students struggle to engage with higher order skills such as evaluation and synthesis in written assignments, either because they do not understand that these are the aim of written assessment or because these critical thinking skills require more effort than writing a descriptive essay. Here, we report that students who attended a freely available workshop, in which they were coached to pose a question in the title of their assignment and then use their essay to answer that question, obtained higher marks for their essay than those who did not attend. We demonstrate that this is not a result of latent academic ability amongst students who chose to attend our workshops and suggest this increase in marks was a result of greater engagement with ‘critical thinking’ skills, which are essential for upper 2:1 and 1 class grades. The tutoring method we used holds two particular advantages: First, we allow students to pick their own topics of interest, which increases ownership of learning, which is associated with motivation and engagement in ‘difficult’ tasks. Second, this method integrates the development of ‘inquisitiveness’ and critical thinking into subject specific learning, which is thought to be more productive than trying to develop these skills in isolation. This article is included in the Innovations and best channel. practices in undergraduate education Dominic Henri ( ) Corresponding author: d.henri@hull.ac.uk Henri D, Morrell L and Scott G. How to cite this article: Ask a clearer question, get a better answer. [version 1; referees: 1 approved, 1 2015, :901 (doi: ) approved with reservations] F1000Research 4 10.12688/f1000research.7066.1 © 2015 Henri D . This is an open access article distributed under the terms of the , which Copyright: et al Creative Commons Attribution Licence permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Data associated with the article are available under the terms of the (CC0 1.0 Public domain dedication). Creative Commons Zero "No rights reserved" data waiver The work was unfunded and produced while employed by the University of Hull. Grant information: The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Competing interests: No competing interests were disclosed. 25 Sep 2015, :901 (doi: ) First published: 4 10.12688/f1000research.7066.1 Referee Status:


Introduction
Supporting the development of critical thinking skills in students can be considered to be one of the key goals of most higher education institutions (ten Dam & Volman, 2004).Critical thinking skills such as analysis, evaluation and synthesis represent the highest levels of learning and literacy capabilities, and are highly sought after by employers (CBI, 2009;Krathwohl, 2002;Miller & Tanner, 2015).Despite the focus on teaching critical thinking skills at university, only ~2/3 of UK graduates (lower than the global average) were capable of exhibiting them during a recent literacy skills survey, which is disappointing given the strong correlation between high level skills and employment among graduates (OECD, 2013;OECD, 2015).This paper evaluates a simple method of encouraging students to engage with these higher level skills in their written assessments.Chanock (2010) outlines five goals that an essay should fulfil.The first is "presenting a question/problem to the reader" and forms the focus of this study.In our experience, students who fail to achieve high grades on written assignments do so because they write descriptive essays lacking a question or problem to solve; i.e they do not understand goal one (Cottrell, 2011).By defending a position or hypothesis using understanding drawn from wider literature, students can provide evidence of high-level literacy and critical thinking (Kellogg & Raulerson, 2007;Miller & Tanner, 2015).Previous studies support the idea that encouraging questioning behaviour promotes the exhibition of critical thinking by students at a range of levels (Commeyras & Summer, 1998;Keeley et al., 1998;Tsui, 2002).In fact, lack of practice performing critical thinking is thought to be a particularly important barrier to the development of higher-level literacy skills (Cottrell, 2011;Kellogg & Raulerson, 2007).We hypothesise that by making the first objective of an essay more obvious to students, and by encouraging them to approach written assignments as questions that need to be answered, students are more likely engage with higher level learning outcomes (Krathwohl, 2002).
A second cited barrier to the development of higher-level literacy skills is reticence on the part of the student, as essays containing evaluation and synthesis are more difficult to write than descriptive essays (Cottrell, 2011;Keeley et al., 1995).An important aspect of our method involves allowing the students to choose what question they are most interested in within the defined subject area.In this study each student was asked to explore a broader concept associated with a specific animal behaviour they chose to study earlier in the year (for details see Methods section).We believe that the sense of ownership of the task could help to improve engagement with 'difficult', higher-learning outcomes, as motivation to engage with studying is thought to be positively associated with personal interest in the topic (Pintrich, 2003).
We propose that coaching students to pose a question in their essay title can integrate 'inquisitiveness' development into written assessments.Integration of the development of these skills into the context of the course has been argued as being more effective than trying to teach these skills in a separate course (ten Dam & Volman, 2004;Wingate, 2006).We suggest that students who start by posing a question in the title are more likely to understand the first of Chanock's (2010) aims of an essay as well as being more likely to exhibit higher-level literacy skills throughout their writing.Thus, we hypothesise that students who pose a question in their title will obtain higher assessment scores than those who do not, as evidence of higher-level skills are essential in obtaining higher marks.

Participants
Project participants were students enrolled on a second year undergraduate module Behavioural Ecology (UK level 5, 20 credits).We believe this group of 55 individuals to be typical of the wider population of UK undergraduate students enrolled on Honours Degree Programs in the Biological Sciences.23 of the students were male and 32 were female.
Assessment of the module was by an end of module written examination (50%) and summative coursework (50%).This coursework comprised three tasks (A, B and C) worth 10%, 10% and 30% respectively.Task A required pairs of students to work together to find a short (3 minute) video clip of animals performing a behaviour that interested them and to complete a written assessment in the form of briefing notes for a film crew interested in recreating the video as part of a wildlife documentary.This task encourages observation and description.Task B required individual students to self-assess task A and reflect (in writing) upon their use of the assessment criteria in doing so.This task encourages students to think about the assessment criteria and the way in which they are applied.Task C required students to write a detailed essay exploring the underlying principles and wider context of the behaviour chosen for assessment A. The notes provided to students to explain these tasks are available as supplementary material (Appendix 1).This project investigates the impact of an optional workshop-based intervention that took place after the students had received grades and feedback on assessments A & B and before they completed assessment C. All students were invited to attend a workshop led by DH and LM as preparation for assessment C with a focus on improving essay writing skills.All students were provided equal opportunity to attend; multiple timeslots were available for students with other commitments.At the workshop, DH & LM explained the function of a good essay in that it should outline a problem that needs to be solved, then present and evaluate the various solutions using wider literature.We suggested that in order to help the students do this they should present a question that needs answering as the essay title, and then use the essay to answer that question with reference to the broader literature (see Appendix 2 for essay titles).We then helped students create a relevant question to ask, suggesting they avoid descriptive 'how' questions, and focused on evaluative 'why' or 'to what degree' type questions.This activity was not conceived as a research project and because attendance at the workshop was optional student attendance was not monitored.For the purpose of this study we assumed that students who posed a question in the title of their essay had attended the workshop and understood the underlying concepts of the workshop, and this has been used as the independent factor in our analysis.We acknowledge that this lack of certainty in the allocation of students to the did/did not attend category does need to be borne in mind when interpreting our results.Another possible confounding factor is that voluntary workshop attendance may be skewed towards individuals who are more engaged or motivated with the module; and these individuals are more likely to obtain higher grades because of this higher engagement with the module content (Pintrich, 2003).We have controlled for inherent capability or engagement of the student in this study by including the previous mark on Assessment A of the student as an independent factor in our statistical analyses (see Statistics).Students' essays were marked by an assessor who was not involved in the delivery of the module or aware of the purpose of the workshops but who does have the relevant disciplinary expertise (GS), so as to not influence student grades.

Ethics
Ethical approval for publication of our study was obtained from the University of Hull, SoBBEs ethics board (Code H038).As the significance of the results presented here was only noted after marking had taken place, it was not possible to obtain student approval.However, students cannot be identified individually from the study results or data set, which was deemed sufficient by the ethics board.

Statistics
A generalised linear mixed effect model (GLMM) was used to test for an effect of posing a question as the essay title on the percentage mark awarded to the essay.The student's mark on assignment A was included as a second independent variable to control for the effect of inherent capability.As assignment A was written in pairs, the pair groups were included as a random factor (random intercepts) to control for non-independence of the marks.An observation level, random factor was included to account for overdispersion (Harrison, 2014).As the dependent factor was a percentage, the GLMM was run with a Binomial error structure.All statistics were performed in R using the glmer function in the lme4 package (Bates, 2010) of R v3.1 (R Development Core Team, 2014).The Minimum Adequate Model was established via log-likelihood ratio comparisons using Maximum Likelihood approximation, for which X 2 results indicating significance are reported (Bates, 2010).

Results
Essays with a question in the title scored significantly higher than those without (X 2 1 = 4.62, P= 0.03; Figure 1 & Table 1).There was no significant effect of score of previous assignment (X 2 1 = 3.02, P= 0.10), or interaction between the two independent variables (question in title:previous score, X 2 1 = 0.81, P= 0.36).

Discussion & Conclusion
Our results support our original hypothesis that students who posed a question in the title of their essay would obtain higher grades than those who did not.We suggest that this is because the process of coaching students to use questions to think 'inquisitively' improves the likelihood they will engage with critical thinking skills, such as analysis, evaluation and synthesis.Our results support this because evidence of these skills is necessary for work to be awarded 1 st class grades (70% or higher), and we note the much higher proportion of students posing a question who obtained a 1 st class grade (44% [asked a question] vs. 22% [did not ask a question]).Given the importance of critical thinking skills for obtaining higher degree classifications, better literacy scores and gaining employment following graduation, we suggest this outlook may be added to the methods of developing student essay skills (CBI, 2009;OECD, 2013;OECD, 2015).Our method is particularly advantageous because it can be easily integrated into the curricula; as opposed to needing to be taught separately (ten Dam & Volman, 2004;Wingate, 2006).Furthermore, the workshop method in our study focused on helping students develop their own questions to answer, encouraging student ownership and motivation in order to overcome any reticence to engage in 'difficult' higher-level literacy skills (Cottrell, 2011;Keeley et al., 1995).
It is important to state that we believe the whole process of teaching students to think in terms of questions/problems and how to answer them is important; as opposed to merely the act of placing a question in the title.We also do not suggest that this is a blanket method of encouraging students to develop high-level literacy/critical thinking skills; evidently, some of our students who attended the workshop and used the method did poorly (hence did not grasp the underlying concept) and other students did well despite not using the method detailed herein (Figure 1).This is to be expected where students construct their knowledge base and its application individually and thus respond differently to instruction based on their prior experiences and learning preferences, and does not undermine its validity as a potential tool for broader teaching strategies (ten Dam & Volman, 2004).We concede that a more extensive study, including more students across multiple assessments, is required to resolutely confirm the trends found herein.Further work should focus on helping students to distinguish between descriptive 'how' questions and evaluative 'why' questions to see if this further improve the efficacy of the method.Remember to carefully read the assessment criteria for the task -they provide you with a lot of potentially useful guidance.

Assessment A marking criteria Upper 1st
As for 85, with impeccable descriptions, written clearly and concisely showing very strong evidence of ability to sort and order information into a logical and coherent way.Summary of wider scientific principle shows overwhelming evidence of understanding of concepts at the forefront of the discipline.
As for 85, but summary of wider scientific principles may fall very slightly short of excellence. 1st Highly suitable choice of video clip, which displays wild animals carrying out a particular behaviour.Accurate and clear summary of the species, insightful identification of the behaviour, and clear description of it such that film crew can identify behaviour for further filming.Video chosen also allows exploration of clearly identified wider scientific principles.Summary of wider scientific principles display excellent knowledge and understanding.Information is drawn from a wide range of primary sources, clearly and accurately referenced.Language is highly scientific and concise and presentation is excellent.
As for 85, but may not be as accurate in behaviour description or identification.
As for 80, but may be less well presented, or choice of video may be less suitable.

2.1
Suitable choice of video clip, which displays wild animals carrying out a particular behaviour.Clear summary of the species, sound identification of the behaviour, and clear description of it such that a film crew can identify behaviour for further filming.Video chosen also allows exploration of clearly identified wider scientific principles, summarised clearly.Information is drawn from a range of primary and secondary sources.
As for 68 but behavioural description may contain some inaccuracies, or wider scientific principles are not summarised clearly.
As for 65 but video clip may not be ideal, or some species details may be missing.

2.2
Mostly suitable choice of video clip, which displays wild animals carrying out a particular behaviour.Mostly sound description of the species and identification of the behaviour, and clear description of it such that a film crew can identify behaviour for further filming.Video chosen may be less suitable for exploration of clearly identified wider scientific principles, such principles are slightly unclear in explanation.Information is drawn from a range of primary and secondary sources As for 58 but maybe a few inaccuracies in behavioural description, or species details, or referencing may be less sound.
As for 55 but some less important elements may be missing. 3rd Video clip chosen, but maybe unsuitable because it is not of an animal in the wild, or cannot be used to illustrate a wider scientific principle, or contains a range of different behaviours.Identification of the species and behaviour is mostly sound, but work may lack a clear description of it such that film crew can identify behaviour for further filming.Wider scientific principles are not clearly explained, or may not be relevant.References drawn from a limited range of sources.
As for 48, but some elements may demonstrate misunderstandings.
As for 45 but some elements may be missing or demonstrate misunderstandings.

Just fail
Fails to identify behaviour in a largely unsuitable video.Makes some attempts at most parts of the task, but some evidence of misunderstandings.
fail Fails to identify behaviour in a largely unsuitable video.Makes some attempts at some parts of the task, but evidence of many misunderstandings.Contains important misunderstandings and inaccuracies that mean that the most important concepts surrounding the task have not been understood.Little evidence of engagement with the task, and contains many misunderstandings and inaccuracies.
Negligible evidence of engagement with the task.No relevant material.2. Write a short justification of that mark.In doing so it is important that you refer to the assessment criteria for assessment A and that you provide evidence to support your claim, (you might for example include extracts from the submitted piece as evidence, but pasting in large chunks of your assignment A is a waste of words).In particular, you should focus on the differences between the classification boundaries and explain why your assignment falls into a particular category.We expect you to make extensive use of the published literature (preferably articles from peer reviewed journals) in writing this essay.

Upper 1st
As for 90, but excellent and comprehensive coverage of the subject, showing deep understanding and including good evidence of critical analysis.

90
Covers all the aspects of the subject thoroughly, showing overwhelming evidence of understanding.Uses and collates information from a wide number of appropriate sources.Impeccable use of English, extremely well presented and structured using appropriate academic conventions.Written clearly and concisely showing very strong evidence of ability to sort and order information into a logical and coherent way.85 1st As for 80, but covers nearly all of the topics in the subject.

80
Uses and collates information from a good variety of prescribed and appropriate non-prescribed sources, covers the most important aspects of the subject accurately and without omissions.Shows strong and convincing evidence of understanding of fundamental concepts and some evidence of critical analysis.Well presented using appropriate academic conventions.Written clearly and concisely, showing strong evidence of ability to sort and order information into a logical line of argument.

75
As for 80 but may use fewer appropriate sources, contain a small number of omissions, or lack excellence in presentation/referencing.68

2.1
As for 65, but shows one or more of: analysis, wider breadth and depth of subject coverage, or better evidence of ability to sort and order information into a logical line of argument.

65
Uses and collates information from a sufficient variety of prescribed sources, covers the majority of the most important aspects of the subject well, shows evidence of understanding and some critical analysis.Mostly well presented using appropriate academic conventions and written clearly and concisely.62 As for 65, but shows less breadth and depth of subject coverage, or less understanding, or is less well presented.58

2.2
As for 55, but may include more sources, more clarity, or fewer omissions.

55
Uses and collates information from a small number of appropriate sources.Covers the most important aspects of the subject and shows understanding of the fundamental concepts, but with omissions.Writing mostly clear, presentation shows some effort.52 As for 55 but may not be as well presented, clearly written, or may contain more omissions.48 3rd As for 45, but may include more sources, more clarity, or fewer omissions.

45
Contains some evidence of understanding of a few of the fundamental concepts surrounding the subject, but with many omissions.Draws information from a very small number of mostly appropriate prescribed sources.Limited evidence of ability to sort and order information into a logical line of argument.Writing reasonably clear, presentation shows minimum of effort.42 As for 45 but may not be as well presented, clearly written, or may contain more omissions.

38
Just fail Fails to show evidence of understanding of the most important concepts surrounding the subject, although may include descriptions of other aspects of the subject.

fail
Fails to show evidence of understanding of the most important concepts surrounding the subject, although may include descriptions of other aspects of the subject and contains some misunderstandings and inaccuracies.
25 Contains important misunderstandings and inaccuracies which mean that the most important concepts surrounding the subject have not been understood.

15
Little evidence of coverage of any topics surrounding the subject, perhaps one or two points which are relevant, little evidence that the most important concepts have been understood, and contains many misunderstandings and inaccuracies.5 Negligible evidence of coverage of topics surrounding the subject, with omissions in knowledge and understanding.0 No relevant material.Critical thinking is certainly a key skill which we would like our students to develop during their time in Higher Education.The paper points to the OECD studies showing that students tend to lack these critical thinking skills.I think it would also be good to highlight the evidence for this in the education literature.

Gender
The paper looks at an intervention through an optional workshop to coach students to think about framing essay titles into questions.The hypothesis being that those who formulate questions achieve a higher mark when compared against those who did not.
In the methods it would be good to clarify if the gender ratio of that particular module is typical of the wider cohort, it is suggested, but not implicitly stated.It would also be good to know if there were any mature students in this cohort (maturity being another confounding factor).The workshop was optional, giving rise to a group without the intervention, and a group with the intervention (control).It would be good to have a table here showing the numbers in each group, and the gender and age split.The authors suggest a confounding factor in that more motivated students (possibly achieving higher grades) were more likely to opt in.It would be interesting to look at gender and maturity as factors as well within the self-selecting group vs control group.I would like to see more information about the intervention workshop, how was it run?How long was it?Did the workshop only cover the framing of titles as questions, or did it also cover the structured argument required to answer the question?Did students who attend this follow up conversations within the workshop with further questions?If so, how would this have influenced their final grade?Do you have any qualitative data from student feedback to show you what was valued within the workshop?The paper states that the independent assessor who marked the essays was unaware of the study being conducted.Were the essay titles removed prior to marking?I think it would be good for the discussion to examine the recent rise in the number of A-levels students taking the extended project qualification (approx.33,000-data can be obtained from the Joint Councils for Qualifications).This dissertation requires students to formulate a research question, and then to investigate and critically analyse sources.This is becoming an increasingly popular and important qualification, and is set to rise with the removal of the AS examination.It would be good for the authors to discuss what future impact this might have on the quality of critical thinking of our students as they enter into higher education.I think this paper hints at straight forward intervention within the curriculum which could help develop critical thinking skills, but I think the needs more evidence over at least another cohort of students on the F1000Research I think this paper hints at straight forward intervention within the curriculum which could help develop critical thinking skills, but I think the needs more evidence over at least another cohort of students on the same module before the influence of the workshop can really be shown on student attainment.
I have read this submission.I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
No competing interests were disclosed.This paper considers a workshop conducted to support students preparing a summative written assignment, which had the unplanned benefit of improving critical thinking skills, as evidenced by higher assignment grades.The development of higher-order thinking skills in undergraduate students is of considerable relevance in relation to the employment prospects of the increasingly diverse student body; this study is therefore a useful contribution to the literature.I have some suggestions below, consideration of which may improve the paper.
There is overlap between hypotheses 1 ("…by encouraging [students] to approach written Hypothesis 1. assignments as questions that need to be answered, students are more likely to engage with higher level learning outcomes") and 2 ("…students who pose a question in their title will obtain higher marks, as evidence of higher-level skills").Hypothesis 1 is not tested by any analyses, and I can't think of an appropriate approach to do.Since demonstrating higher-order learning outcomes and gaining higher marks are so closely related, perhaps no further analysis is warranted and H1 should be lost.If H1 is retained, the analyses and the will need expansion.

Results
Achieving higher-level learning outcomes by demonstrating critical thinking skills is Learning outcomes.at the heart of this paper.However, whilst the allude to higher-level Assessment C marking criteria learning outcomes (e.g."shows convincing evidence of understanding" compared to "fails to show evidence of understanding"), no specific words (e.g. compared to ) are used.Therefore, evaluate describe evidence that higher learning outcomes have been achieved could be more explicit; perhaps some descriptive text (including essay extracts) could be added to the to provide qualitative evidence Results that higher level learning outcomes were met.
More detail of the cohort characteristics would be useful to justify the "We believe this Participants.group" statement, in particular in relation to ethnicity.At my own institution, BME groups are poorly represented on the Ecology pathway of BSc (H) Biological Sciences (who would take a Behavioural module) compared to, for example, those studying Biomedical Sciences.In the context of Ecology 'Narrowing the Gap' initiatives, the ethnic composition of the cohort could usefully be clarified.
The Methods are clear and supplementary material is very useful.More detail could be Methods.provided on some aspects to meet the journal guideline that the work should be repeatable by others.For example, clarify how the "function of a good essay" was "explained" (could further supplementary material usefully be provided)?The statistical approach used is robust -would a brief description of / introduction to GLMMs (or perhaps a reference to further information) be useful?Expanding on my

Figure 1 .
Figure 1.Box plots comparing % scores across essays that included a question in the title and those that did not.
You should write a short individual reflection that explains to the assessor what mark you think your briefing notes (assessment a) will be awarded:1.State what grade (% mark) you think the work is worth (refer to the assessment criteria for this task).
of Science and Technology, Nottingham Trent University, Nottingham, UK Your video clip should show a single behaviour, not a series of different behaviours performed by the same animal.In choosing your clip, you should think particularly how you would develop your work on the topic for Assessment C. Provide a link to the footage (if it is a short section of a longer piece remember to explain where the section of interest starts/stops).Write a brief description of the species involved, geographical location and the best time/location to film the behaviour in question b.Description: Write a description of the footage -what does the viewer see?This section should describe what is going on ("the squirrel is burying nuts under dry leaves"), not explain what is happening.Here, you should explain what the behaviour is and why the animal is carrying it out ("the squirrel is caching nuts ready for the winter", and then explain why the squirrel does this) 1. Locate a short piece of video footage (<3 min) that clearly shows wild animals carrying out a particular behaviour.a. Background: c. Explanation: d.Wider concepts: Summarise the wider scientific principles underlying the behaviour that you will write about in Assessment C. Think about the general area in which the behaviour lies, that could form the focus of Assessment C. Examples might include optimal foraging, or intersexual selection, or animal migration.This section is not about identifying as many concepts as you can, but about highlighting the topic that you will develop further in Assessment C.

Submission: electronic submission only Word count (or equivalent): 2000 Assessment overview: Individually write
Use the assessment criteria for assessment B to guide you in the types of areas you should include in your self-assessment.a detailed essay on the scientific principles underlying the behaviour you chose to write about in assessments A and B. You should not simply expand the work you have already done, but focus on explaining the wider scientific concept underlying the behaviour you chose.While your essay should not focus on the particular species in your video clip, you may of course use it as an example within the essay.