The proportions of device and clicker users giving particular responses were compared using proportion tests (prop.test in Rv2.13.0; R Development Core Team, 2011). Exact binomial tests were used to assess whether the proportion of participants giving a particular response was greater than 0.5.

The School of Biological, Biomedical and Environmental Sciences and the Faculty of Science and Engineering ethical approval committees approved the project and questionnaires prior to any work commencing. Questionnaire data (including student contact details) was input and anonymised by student interns, who also carried out and transcribed focus group interviews. Lecturing staff were unaware of student identity in data analysis. Consent information ( Supplementary Material) was included with the questionnaire: participation was voluntary and students were free to withdraw at any time.

Seventy-eight students completed the questionnaire. Of these, 44 (56.4%) used their own device, while 34 (43.6%) used only clickers. Throughout, we refer to those that used their own devices as ‘device users’ and those that used the clickers we provided as ‘clicker users’ (device users used clickers in the clicker-only lectures). Age and gender of the students reflected the make-up of the cohort: 53.9% male, 43.6% female (two students did not answer), and 65 students (88.3%) were aged 18–23, with five (6.4%) aged between 24 and 29, five (6.4%) between 30 and 35 and three (3.9%) students aged 36 or over.

Almost every student owned at least one type of device that could potentially be used, with a large proportion owning multiple device types. Sixty-seven students (85.9%) owned a smartphone, with 76.9% owning a laptop, 32% owning a tablet, and 9% owning a netbook or ‘other’ technology (including iPods and non-smart mobile phones). While 21 students (26.9%) owned only one type of technology, the remainder owned more than one (2:42.3%, 3:26.9%), with two students declaring ownership of four types of devices that could potentially be used as part of an audience response system (a smartphone, laptop, netbook and tablet). One student did not possess any type of technology that could be used.

As clickers are used in other modules the students may have taken (notably a level 4 module Ecology and Evolution), students were asked if they had previous experience of using clickers in lectures: 67.9% (53 students) had, while 32.1% (25 students) had not.

Of the 44 students that used their own device, 37 (84.1%) used a smartphone and five (11.4%) used a tablet, with the remaining two students using either a laptop or other type of technology. In terms of operating system, 59.1% used iOS, 38.6% used Android, and one student (2.3%) used Windows (on a laptop). Of the 34 students who did not use their own device to participate, 23 (67.7%) already had a device that they would be willing to use, while eight (23.5%) did not have a device that they would be willing to use (although all owned at least one device that could potentially be used). The remaining three students (8.8%) did not currently possess a device, but were thinking of getting one. These three students included the one who did not currently own any types of technology and two possessing only a laptop.

We asked students which type of interactivity was more enjoyable, informative, understandable, and made lectures seem to pass more quickly. Figure 1 summarises the responses. In general, device users felt that interactive lectures (either using their own devices or using clickers) enhanced their enjoyment and understanding slightly more than clicker users did, although these differences were not significant. Device users who preferred interactive lectures were divided in whether they preferred using clickers, their own devices or had no preference between the two types of technology (see Figure 1). These three preference categories have been grouped together as ‘interactive lectures’ in the analysis that follows. Overall, very few students felt the no-technology lectures scored most highly on any of the four measures.

In terms of enjoyment ( Figure 1a), 77.3% of device users (34 students) preferred lectures with interaction via either clickers or their own technology, compared to 55.9% of clicker users (proportion test: χ ² = 3.107, df = 1, p = 0.078) preferring interactive lectures over the no-technology lectures. Significantly more than half of the device users preferred interactive lectures (binomial test: p < 0.001), but this was not true for clicker users (p = 0.608). None of the device users and only 5.9% of the clicker users (2 students) felt that the no-technology lectures were the most enjoyable ( Figure 1a).

For levels of understanding ( Figure 1b), 70.5% of device users (31 students) and 52.9% of the clicker users (18 students; χ ² = -1.825, df = 1, p = 0.179) felt that the interactive lectures were more understandable. This represents significantly more than half of the device users (p = 0.010) but not the clicker users (p = 0.864). Two students (4.5%) who used their own technology and two students who used clickers (5.9%) felt that the no-technology lectures were more understandable ( Figure 1b).

Students expressed less clear overall preferences in terms of the informativeness of the lectures ( Figure 1c) or the speed at which they passed ( Figure 1d). 61.4% of device users (27 students) and 55.9% of clicker users (19 students; χ ² = 0.598, df = 1, p = 0.439) felt that the interactive lectures were more informative than the non-interactive ones, but this was not significantly more than half (device users: p = 0.174, clicker users p = 0.608). Again, a few students (four device users and three clicker users) felt the no-technology lectures were the most informative ( Figure 1c). In terms of the speed at which the lecture appeared to pass ( Figure 1d), almost half the students (45.5% of device users and 44.1% of clicker users) had no preference with two device users and four clicker users feeling the no-technology lectures passed more quickly. The remaining 50% of device users (22 students) and 44.1% of clicker users (15 students) felt that the interactive technology lectures passed more quickly (proportion test: χ ² = 0.083, df = 1, p = 0.774, binomial tests, device users: p = 1, clicker users: p = 0.608; Figure 1d).

When asked whether having their device ready to answer questions distracted them from the lecture, 36 of the device users (81.8%) answered in the negative, while seven (16.9%) said that it did. A mixed response was highlighted in the focus groups:

“ It was easier to me like, ah, I’ll just reply to this text quickly”

“ I think I’d be more tempted because it’s like well I could just say I’m doing it for this purpose [answering the questions] when really I wasn’t”

“ I don’t think it [having your device ready] makes a difference at all”

However, when asked which, if any of a number of web-based and interactive activities (including checking email, using social media, browsing the web and texting or sending a message) they carried out during each type of lecture, much higher numbers reported using their devices. Overall, 70.5% of device users admitted to using their devices for at least one other activity during the device lectures, compared to only 25% in clicker lectures and 43.2% in the no-technology lectures ( Figure 2a). Students were significantly more likely to self-report using their device for other purposes in the device lectures than the other two lecture types (proportion tests: device lectures vs clickers lectures, χ ² = 16.443, df = 1, p < 0.001, device lectures vs no technology lectures, χ ² = 5.604, df = 1, p = 0.018), but there was no difference in self-reported usage between clicker lectures and no technology lectures (χ ² = 2.478, df = 1, p = 0.115). Thus, students are most likely to use their devices for other purposes when they are also using them as audience response devices.

Students used their devices for checking email, using social media, browsing the web, sending and receiving messages as well as other activities ( Figure 2b), with the majority of students reporting using their device for multiple purposes. When asked in the focus groups about the sort of activities they used their devices for during lectures, students in the first focus group suggested that it was mostly socialising rather than study-related, although the second focus group highlighted the academic activities that they might engage in, such as checking the Virtual Learning Environment (VLE) for assignments, going through the slides or using Google to search for information about unfamiliar topics. Both focus groups identified Facebook as being the primary “ culprit”. Other activities included online shopping for groceries and other items.

Twenty seven device users (61.4%) and 21 clicker users (61.8%) reported being aware of others using their devices for other purposes during a normal lecture. In the device lectures, this proportion increased slightly among clicker users (to 79.4%) but not device users (61.4%). When students in the focus groups were asked whether they noticed others around them using their devices for other purposes, students again answered affirmatively, but suggested that this was no more than usual. They did find that it could be distracting:

“ I did notice a few people like on Facebook and things”

“ I know a few times I’ve been in lectures where people next to me have been on a game and I’ll be like oh I’ll just go on that for a minute and then you kind of get into it and I’ve spent like a full 2 hours on a game”

“ If they’re on the phone it’s not distracting. But if they’re talking then it will distract me”

“ People are sometimes playing games and that’s distracting. You can see it out of the corner of your eye.”

Interestingly, two students mentioned that they actively avoided taking their laptops and tablets to lectures because they find them much more distracting than their smartphones:

“ I never take my laptop in… got an increased chance of getting distracted, because it’s there with a little button that says ‘internet’ on”

However, despite admitting that they were more likely to get distracted, students felt that the interactive element of the lecture meant it held their attention more:

“ I’m actually trying to think about the questions and actually answer it whereas before I’d just be sat there like… [tails off]”

“ I was actually paying more attention to try and get the right [answer]”

“… we actually had a discussion about the question, before we pressed it, so we were all a bit more attentive to the lecture”

They also highlighted that factors other than having their devices available made them more likely to use them. Primarily, they suggested that they would be more likely to use their devices once they were already disengaged with the lecture:

“… I’m getting bored and I have to go on my phone to keep awake”

“ I don’t think I am distracted by it. I tend to turn to it when I’m bored or not engaged with lectures”

They were also more likely to get distracted in environments (lecture theatres) that are perceived as being of lower quality:

“ I’m more likely to use my phone in there” [a lecture theatre identified by other students in the group as being “ awful” and having issues with echoes].

Of the 44 students who used their own device (device users), 39 answered the question “were you happy to use your own devices in this way?” with a positive response (88.6%) while five students (11.4%) would prefer not to, but would if it was the only option. No students responded “no” to this question. Of the device users 32 (72.7%; significantly more than half: p = 0.004) preferred the lectures using student-owned technology as response devices, while 12 (27.3%) preferred using clickers ( Figure 3). Of the 34 students who did not use their own devices (clicker users), 76.5% (26 students) preferred the clicker lectures (significantly more than half: p = 0.003), while 14.7% (five students) preferred the lectures with no interactive technology ( Figure 3; two students had no preference and one did not answer). No student who was unable to participate using their own device reported feeling disadvantaged in those lectures.

Some other general themes emerged from the interviews and free text comments. Students mentioned that they would be unwilling to use their own devices if they had to pay for text messages or use up their data allowances, and highlighted software glitches as a potential concern for the wider use of the technology. Several students mentioned that battery usage would act as a barrier to their participation, particularly (as seemed to be the case with the software we used) if the device remained active rather than going into standby when the app was active, and required participants to log in again if the app was minimised. One student highlighted that their battery reserve decreased by 20–25 percentage points during a 1-hour lecture.

Although all the students who agreed to participate in the focus groups used their own devices, they identified a number of barriers to participation in their peers, including app availability (not available on Windows phones), the fact that those who did not have suitable technology would feel disadvantaged, particularly if compatible clickers were not available. One student suggested that this may lead to disengagement with the lecture:

“… you put some people at a disadvantage if they don’t have a smartphone or whatever…”

“… a few people said they didn’t feel involved in the lecture they just sit there and they’re like, well I can’t do anything so I’m just going to sit there, I’m not going to pay attention and then they’re probably more likely to go on their phones and actually like do something completely different just because like they couldn’t interact properly”

Student participation is likely to vary widely depending on the software used, and as alternatives become available, some of the barriers highlighted here may disappear.