Well, we did it. Seventeen psychology faculty from six post-secondary institutions in British Columbia came together on July 18 & 19 and worked intensively for two days to create a testbank designed to accompany open textbooks for introductory psychology. As I have previously written about, the absence of ancillary materials (a testbank most of all) presents a significant challenge to instructors who wish to adopt an open textbook. Although some observers will turn their noses up at the thought of faculty not writing every single one of the questions on their exams, the reality is that many (overworked) instructors of large, multi-section survey courses do start working from the publisher-supplied testbank, writing, revising, and editing as necessary over time. The need for this crutch becomes even more pronounced for new faculty that do not have years of classroom and assessment experience to draw on, sessional faculty assigned to a course only weeks ahead of the first class meeting, and large sections with limited teaching assistant support. For those of us that advocate for the adoption of open educational resources in the classroom, ignoring or criticizing this reality is self-defeating.

Fast forward to this May when, following a series of productive conversations with Mary Burgess (Director of Open Education at BC Campus) and Peter Lindberg (NOBA Project), I was able to secure the financial support of both organizations to coordinate a “testbank sprint.” Modelled on the efforts of textbook sprints that, for example, bring together a number of faculty members for a few days to write a textbook from scratch, the idea was to invite faculty from a number of psychology departments in BC to come together for two days for a retreat-style sprint.

At the best of times, writing questions is not enjoyable work, so I knew that in order to recruit enough participants and to create a productive environment the sprint would have to be interspersed with opportunities for socializing and rejuvenation. Another goal was to gather sprinters from a variety of institutions (the wider the representation, the more natural the sense of ownership and buy-in). Finally, the challenge of producing a resource like this for Introductory Psychology is the sheer number of topic areas (15) that need coverage. Thus we would have to find a way of collecting data about potential participants’ areas of expertise and mapping these onto the topic areas in order to ensure adequate coverage.

Enter the Organizing Team

Following a couple of discussions with members of my department at Kwantlen Polytechnic University, three wonderful souls in Dianne Crisp, Cory Pedersen, and Levente Orban stepped forward to offer their assistance with organizing the sprint. Together we dealt with the questions of when to hold the sprint (mid-July, in time for Fall adoptions), where to hold it (the Cheam Golf Course in Chilliwack with accommodations at the nearby Coast Chilliwack hotel – central enough for those travelling from Victoria or Kamloops but far enough away to feel like a retreat), how to recruit faculty (the Articulation listserve + direct emails to those with past involvement in open education projects), how to map availability & areas of relative expertise onto the 15 topic areas of Introductory Psychology (enter the technical wizardry of Levente with custom online forms and formulae-laden Excel worksheets), and how to structure the event (see below).

Nineteen faculty signed up and, although two dropped out just 48 hours before the sprint was to begin, seventeen faculty representing the psychology departments at Kwantlen Polytechnic University, Thompson Rivers University, Camosun College, Northern Lights College, Capilano University, and the University of the Fraser Valley met in Chilliwack on July 18. The amazing Clint Lalonde from BC Campus was also on hand to provide logistical and technical support for both days of the sprint.

Sprint Structure

The sprint was structured around 3 90-minute working sessions each day for two days, interspersed with breakfast, coffee breaks, and lunch. Following an early breakfast we began on both days with an orientation at 8:30am. The working sessions ran from 9-10:30am, 11am-12:30pm, and 1:30-3pm, with a final progress update & wrap up from 3-3:30pm. Dragon boating and a hosted dinner were scheduled for the afternoon/evening of July 18 while the sprinters were free to play 18 holes of golf on the afternoon of the 19th.

The seventeen faculty were assigned to one of 7 working groups of 2-3 faculty for each session, with each group responsible for writing questions for a specific topic area. The composition of the groups and the topics changed every session such that by the end of the second session on Day 2 of the sprint each faculty member had participated in 5 working groups and each topic area had been visited by two working groups. From an organizational point of view this was a bit like playing with a Rubik’s cube with 17 squares and 15 sides. Only the squares had feelings and the saturation of the colours was not always the same.

The target for each working group during their 90 minute session was to write an average of 10 high-quality multiple-choice questions per group member (~20-30 questions per topic per session and ~50 questions per topic or ~750 questions in total by the end of the sprint). To help with this we circulated the topic assignments and a document outlining some best practices for good multiple-choice questions ahead of the sprint (the sprinters were encouraged to think about or even write questions ahead of time and to bring their own laptops).

The final session on Day 2 was devoted to the vetting and categorization (Applied, Conceptual, or Factual) of questions. I assigned one faculty member to go through and vet all of the questions written for one topic area in which they had intermediate-to-high expertise. Minor problems could be fixed immediately but larger problems had to be flagged. Flagging a question meant that another faculty member with overlapping expertise would have to be consulted to see if the question could be resuscitated or whether it had to be deleted (deletion was always a two-person decision). I should note here that we did not have an easy template to follow for any of this so the structure and rules of the sprint really resulted from my sense of what would work well. Mercifully it all went to plan and when I solicited feedback and suggestions at the end of the first day the group appeared very happy with how things were going.

The Finish Line

So how did we do? By the end of the fifth session we had collectively written 870 questions across the 15 topic areas (comfortably surpassing my initial target of 750 questions). Following the vetting process in the final session on Day 2, we finished with 851 questions (296 Applied, 219 Conceptual, & 336 Factual). Very soon this entire testbank will be made available by BC Campus to any recognized psychology faculty, in a format that can imported into any LMS. In other words, mission accomplished, under budget and on schedule.

Perhaps most importantly though, we truly enjoyed ourselves. Both process and outcome, journey and destination, were gold.

Some Observations

• Although some working groups began writing questions collaboratively, most found that they worked more productively when they divided the learning objectives/sub-topics and worked in parallel. Having said this, some truly enjoyed writing questions collaboratively.
• The target of 10 questions per faculty member per 90 minute session was found to be reasonable and well-calibrated.
• Although we had installed testbank creation software on a KPU-issued laptop at each of the seven stations Clint and I decided ahead of the sprint that it would be easier and a time-saver to have everyone enter questions (using the correct format) into a more familiar tool (MS Word) instead. The software permitted the easy import of questions from Word.
• Our hosts at the golf course remarked several times at how quiet we were while working. This was generally true as the soundtrack of each session was a steady chorus of fingers on keyboards punctuated with laughter (one of the sources of laughter was the amusement we spontaneously derived from embedding one another’s first names into many of the applied questions!).
• Given the nature of the work we were there to perform I remain amazed by the level of enjoyment we derived. The dominant and universal theme in the feedback I have received since the end of the sprint has been how much fun it was.
• As noted by someone during our dinner at the end of Day 1, dragon boating on Harrison Lake on the first evening proved to be the ideal metaphor for the sprint. We were in it together, we had to coordinate our efforts well, we had one person (me) providing continual feedback to the group while staying mindful of our pace and progress, we rested in between sessions of hard work, we laughed a lot, truly cherished one another’s company, and experienced a rather unforgettable adventure.

The Heroes of the Sprint

My thanks to my organizing team (Dianne Crisp, Cory Pedersen, & Levente Orban from Kwantlen Polytechnic University), other fellow sprinters (Betty Rideout, Carla Maclean, David Froc, Farhad Dastur, John Marasigan, Kurt Penner, Kyle Matsuba, and Richard LeGrand from Kwantlen Polytechnic University, Jacqueline Kampman from Thompson Rivers University, Randal Tonks from Camosun College, Istvan Geczy from Northern Lights College, Hammond Tarry from Capilano University, Wayne Podrouzek from the University of the Fraser Valley, and  Clint Lalonde from BC Campus), ed-tech whiz Meg Goodine from Kwantlen Polytechnic University, and project sponsors (Mary Burgess from BC Campus and Peter Lindberg from the NOBA Project). Thanks also to the Mavericks Dragon Boat Team for hosting us on the evening of July 18.

See Also

Clint’s blog post from Day 1 of the sprint

Press coverage of our work by the Chilliwack Times