The University Challenge of using BlackBoard Test Tool

“Magna Carta! Disraeli! Debussy!” are usually my exclamations when watching University Challenge as I desperately attempt to get any question correct over anyone else watching with me.

I’d never get anywhere near a team for University Challenge, I’d like to think I’ve retired.  So I was delighted when Jane Coles, Societies Coordinator for the Guild, contacted our team for help with the trails for the University of Liverpool team.

Jane was looking for a way to run the trial through our VLE BlackBoard and to use the Test assessment tool.  The issues Jane had was:

  • To move a paper test into an online format.
  • Avoid manual marking each attempt.
  • Analyse the results of each student.

After meeting to discuss the issues, we went into the Test tool and looked at the different question types.  To get automatic marking to work, the best option was Fill in the blank option.

One of the benefits of Fill in the Blank option is it’s one of the few question types in Blackboard that allows you to set the correct answer to text that is either an exact match or contains that text.

uni challenge1

So for example, if the question was “Name colour is a ripe banana?”- you could either set the correct answer to be an exact match of ‘yellow’ being correct or contains ‘yellow’ so the student would still get the correct answer if they submitted ‘bright yellow’.

The option would also allow automatic marking, which can be viewed as soon as the student submitted their exam in either the Grade Centre or Retention Centre.

Jane went away and built a test consisting of 127 questions, varying from subjects covered on University Challenge.  Jane also manually enrolled the students onto the module and invited them to attend one of three time slots, Monday-Wednesday, where the trials would be taking place.

To avoid students potentially cheating by searching the web, Jane used NetSupport School software and locked down all the computers in the computer lab to only access the student intranet and Blackboard.

The students sat the trials and were encouraged to answer as many questions as they could during a two-hour slot.  They could leave anytime they liked but could not return once they had submitted their answers.

The trials themselves were successful, with the system not crashing or students having issues navigating through Blackboard and the Quiz.  I attended the final day and everything went smoothly.

Afterwards, Jane and I met to push the results through to the Grade Centre as exclamation marks were appearing.  This was due to the first few questions being of Essay type to simply ask availability, contact details and year studies finished.

Using the ‘Grade Questions’ option in the Grade Centre allowed us to quickly mark the Essay type questions for all student attempts on one page and finally push through the students’ score.

The other issues we had was if the answer was set as a plural answer (e.g. eggs) but the student set their answer as singular (egg) – the software identified the answer as wrong.

In the future we would need to set the answer as singular, but thankfully the score could be overwritten to be correct before marks are finalised.  Other issues to be aware of are to avoid capital lettering, set numbers as figures and text, as well as ensure apostrophes are correct or remove them.

uni challenge2

Also any questions that required two answers, if the student put “Atlantic and Pacific” but the correct answer set was “Pacific and Atlantic” it would be marked as incorrect.  So for future use, we would need to put both combinations in as the correct answer, as the Quiz tool allows more than one set of correct answers.

I found it to be a positive learning experience.  I enjoyed working with Jane and the Guild to get this project completed. I also learnt a lot of about the capabilities and restrictions the Blackboard Test tool offers.

In the time of writing this, the team has been selected to represent the University and are going through the first stages of getting onto the TV show.  I wish them the best of luck and hope Jeremy Paxman goes easy on them.

Ben McGrae

Advertisements

Bridge Over Troubled Water – Turnitin Summit 19th October

As I arrived in Newcastle, I took a walk around its famous Quayside.

Stepping onto one of the iron bridges to take in the River Tyne, Simon & Garfunkel’s “Bridge Over Troubled Water” started playing through my headphones.

This song couldn’t be more ironic or apt for attending the Turnitin Summit.
I planned ahead with a list of questions I wanted to ask about the problems experienced with Feedback Studio, I suspected other attendees came with a similar agenda of grilling Turnitin.

But this was billed as a conference about Academic Integrity and how Turnitin will be working more closely with UK institutions. Marc Daubach of Turnitin, introducing the agenda of the day, reiterated this at the opening.

The keynote speaker was Kerr Gardiner (https://www.kerrgardiner.co.uk/) who is a consultant in Learning Technology. He talked about his past role as Head of Learning Technology at the University of Glasgow, of how they pulled out of using Turnitin within their Moodle integration.

The anecdote was building up to how in the past, he worked in partnership with Turnitin to try to help meet Glasgow’s demands. That now that partnership has strengthened in his work with HeLF (https://helfuk.blogspot.co.uk/p/about-helf.html). Both HeLF and Turnitin are discussing customer service, future problems in learning & teaching, smoothing out workflows and improving partnerships with UK institutions.

A report developed by HeLF outlined that Turnitin need to work on improving:
• Trust – new features appearing late or not working
• Transparency – if updates and new features are coming, to know in advance
• Relations – better communication and more targeted for the UK

Themes that Turnitin need to be addressing were:
• Reliability of the system, as outage times have had a big impact for a number of institutions.
• Data access has been an issue for a number of years being locked down and not accessible to users.
• Marking – still no double/multiple marking or access for external staff
• Workflows and integration – better support for the variety of different VLE systems, accessibility plans and archiving assignments.

Kerr outlined that Turnitin are looking into the future requirements we might have such as detecting ghost writing, and dealing with other forms of assessment like video. He was strong in saying Turnitin need to deliver and that dialogue will continue with HeLF.

Ron Park of Turnitin then addressed the conference with how Turnitin would meet these problems.
• Turnitin have invested £5.2 million in hardware and software to improve customer relations and dealing with calls.
• Turnitin are improving their cloud server to deal with more assignments and to be more stable.
• The Turnitin website is currently in Beta, which allows users to create folders, and users can drag and drop files onto the website. Still unsure of how these files are used.
• Viewer is being upgraded within Feedback Studio so will be turning on features to improve the magnification of a paper and get around the blurred text issue.
• New API will be available in November 2017

Bill Loller of Turnitin did a talk about the Academic Integrity vision and discussed how it is important for UK institutions, and that there is a need to raise awareness and educate teaching staff.

The conference was then broken up into break out areas called Discovery Sessions, which we signed up in advance of attending. The first was ‘Feedback, marking and moderation’ with Ron Park, which tried to address the issues Kerr Gardiner outlined.

This descended into a Q&A session, issues raised were:
• How were Turnitin going to support the variety of VLE integrations?
• Promises made today (double marking) were promised a few years ago with Feedback Studio and we’re still waiting.
• Other assessment tools are getting better, which allow for double marking, group submissions, other file formats like videos.

Turnitin diverted answers to these issues, threatening that if we moved away from Turnitin to a competitor that we’d be going back 5 years. Customer relations and transparency were questionable here.

After lunch it was the remaining two discovery sessions. I attended Bill Loller’s ‘Ghost Writing’ which was an interesting discussion on how it works. Bill talked about how the service is advertised to students, which type of students it is targeted to, how institutions should deal with students caught and how to educate staff about the issue.

The final session was ran by Gill Powell, ‘Preparing to tackle all elements of academic integrity’. Was a discussion of how different intuitions are educating their staff and students, and the challenges they’ve met. A member of staff from the University of Huddersfield shared his academic integrity report.

The rest of the afternoon was a number of talks around the issue of academic integrity. Simon Bullock from the QAA talked about raising student awareness and using technology to help educate students.

Irene Glendinning from the University of Coventry discussed her Academic Integrity scoring model developed through the IPPHEAE project, which involved scoring a number of universities across the European Union.

Cath Ellis from the University of New South Wales did an interesting talk about the global Academic Integrity picture. She’s been investigating students buying essays from online websites, and looking at student behaviour and how they view cheating.

The final talk was a video presentation by Phil Newton from the University of Swansea, titled ‘Pouring ALE on Contract Cheating’. ALE stands for Assessment Design, Law and Education. He described how the websites work for students wanting to have their essay written by someone else and how institutions can tackle the problem.

As I left the summit and boarded the train home, I plugged myself into music and shuffled back to Simon & Garfunkel. There’s a line at the beginning of “Bridge Over Troubled Water” which says “I’m on your side, Oh when times get rough.”

It seems Turnitin are willing to work more closely with UK institutions to improve support and trust in the system. I feel there are still issues in the current Feedback Studio that need to be addressed such as the app and user issues in Grademark. I felt encouraged there is more effort being made to listen to our concerns with the system and want to see action being taken, to be on our side and resolve them over the next academic year.

Ben McGrae

Blackboard or Turnitin – our EMA decision-making flowchart

The team are fresh back from the now-legendary 2017 ALT conference, and bursting to share and connect with this  fantastic community. One of the things presented by Alex, Tünde and Dan which caught people’s eye during our session on the sometimes arduous road to EMA was Tünde’s ‘Blackboard or Turnitin’ decision flow chart.

It was designed to help staff decide what were the most important factors when they first come to choose whether BB or Tii is the best fit for their context. Behind it is the usual mountain of documentation, guides, caveats, sub-clauses and subtleties, but we had a think about what were the most-often asked questions from our staff and the priority they gave different requirements and used this to help map an easier route through the terrain.

As it looked useful to a few people, we’ve tidied it up, and given it a CC licence so please feel free to grab it from here and try it out with your own colleagues. If you want to know any more about our approach to guiding staff through the BB and Tii tangle then please do get in touch.

Creative Commons LicenseBlackboard or Turnitin Assignment by Centre for Innovation in Education, University of Liverpool is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

You can also catch up with our slides from the session here.

Something else that caught people’s attention, quite literally, was our percussion department for bringing the breakout discussion parts of our workshop presentations to order. ALT delegates are incredibly eager to get stuck in to a good conversation about any aspect of learning technologies, so you definitely need some serious help on your side as a presenter if you want to get a word in again! Here’s our kit, level 1 – the bell, level 2 – the castanets, and level 3 – our last resort – the slide whistle. We went all the way to level 3.

Alex, Tünde and Dan

Black Sabbath to Busted – Report on Blackboard Enhanced Assessment and Feedback Event

Leaving home under an early-morning starry sky and up over the misty, snowy Pennines to Sheffield, I wasn’t sure what to expect from the Blackboard Enhanced Assessment and Feedback day to which I was travelling. I had a sketched outline of the themes for the day ahead but not much more, namely that we would engage in some way with:

Assessment and feedback – an institutional perspective

  • Examination of key drivers and challenges (reputations, quality of process, quality of data, legal, business efficiencies, risk etc.)
  • Placing use cases on a confidence/effectiveness model
  • Highlight relevant Blackboard solutions

 Assessment and feedback – innovation and organisational change

  • Identifications for drivers for change
  • Gap analysis of practice and stakeholder experiences
  • Prioritisation of opportunities for change
  • Highlight relevant Blackboard solutions

This kind of workshoppy day from Blackboard was something I hadn’t experienced before, so I was propelled by curiosity as much as that it seemed relevant to the work that our team are currently leading on e-submission and e-feedback at Liverpool. What I got was a useful day of frank discussion and sharing of experiences, ideas and commonalities, which was mostly reassuring, with colleagues from other institutions in the kind if depth you don’t often get. This is especially useful for a sense of the bigger picture in HE, to talk about the differently badged or described but largely similar activities, structures and strategies that are top of our agendas at the moment, e-submission and e-marking being one of Liverpool’s current strategic TEL focus. One universal and rapidly-emerging area of concern that became evident on the day was a need for a variety of programme-level views of assessment activity in the VLE for academic and administrative staff and students. This is a long-requested feature from Blackboard usergroups that’s time has come with the adoption of e-submission and e-marking policies across the sector of late and I hope this was the main takeaway message for the Blackboard team.

The event was run by our regional Blackboard Customer Success Team, in partnership with the BB North user group, recognising a need for a more extensive exploration of particular issues that get raised at user group meetings, where the format doesn’t allow fuller discussion. Whilst advertised as intended for senior leaders, learning technologists, TEL managers and academic staff, the majority attending today were learning technologist types. It was instructive to hear that on the previous day at Edinburgh a couple of PVCs had attended, sending some very positive signals about the depth of an institutions’ engagement and intent with the actual tools that students, academic and professional service staff use as a part of their everyday life at the University.

Music, sweet music…

Our first activity was to introduce ourselves telling the room the first piece of music that we had ever bought. An astonishing array of formats and first loves was paraded, from Now compilation tapes to Avril Levigne downloads, from Osmonds vinyls to Busted CDs. Blackboard’s Alan Masson and Gillian Fielding all admitted to their first purchases, but I’ll spare their blushes here. Top-trumping all these, however, was Blackboard’s Steve Hoole, whose overnight Novotel stay featured a vinyl deck (remember these, kids?) and a selection of Sheffield synth heaven albums to spin the night away.

Structure of the sessions

The morning and the afternoon were structured in a similar way so that we would first ‘brainstorm’ our thoughts in groups on a set of e-assessment themes, then work together on some specific ideas from those and bring something interesting back to the room. We’d finally end with a discussion of the potentially useful tools in Blackboard that could be a part of the thinking for some of these. I initially thought this last element was going to be a sales pitch but it was pleasingly nuanced in that the Blackboard team wanted to hear stories of how people are using these tools, where they’re working well and what the gaps are. A very clear point made more than once was that the recent Blackboard activity in developing the assignment tool to offer dual marking, moderation and anonymous marking had been excellent but it now seemed that the Blackboard focus had moved on from this, whilst the toolset still needs work, that there had only been one iteration of the process. Also that this process of close consultation should be constantly repeated for other areas as our needs are constantly evolving, not just for assessment.

Morning Session – what’s needed to enhance assessment and feedback practice?

For the morning, we’d thought about quality, processes and workflows and where the opportunities for enhancement lay. We chucked all of our ideas at the first Padlet below. You’ll find all of the issues that we have encountered in the course of our work as learning technology developers and as part of the University’s EMA project, from how to handle video submission and feedback, to combining some functionality of Blackboard assignments (group tools, double marking, staged release of feedback, etc) with that of Turnitin assignments (Originality Checking, GradeMark, offline marking). In fact, this was another big ask on the day, that Turnitin and Blackboard align/integrate their products in ways that will help us, as you will gather from some of the posts in this Padlet.

Made with Padlet

What leapt out at me was a so-far un-encountered issue at Liverpool of needing a read-only external examiner access to modules. In some institutions administrative staff are packaging up content and assignments into a special section of a module, which only the external has access to, and making the rest of the module inaccessible to them. This is to meet an anxiety around externals potentially changing grades and altering content, but it costs hours of administrative time, essentially duplicating what’s already in the module, so re-introducing at a later stage of the assessment cycle serious administrative burden that the electronic submission process had originally taken away from the front end. What’s needed is a read-only access enrollment level, which is another development idea for the Blackboard team to add to the suggestions box.

Moving on to the next activity in groups again, we were tasked with listing and describing up to five assessment and feedback enhancements that would have significant benefits for the listed stakeholders and the degree to which it would require resource. This photo shows our effort…

dsc01104

…and for those who aren’t adept at reading the handwriting of people who spend their whole day attached to a keyboard our five enhancements (all pretty standard) were:

  1. External Examiner Access – read-only access for external examiners or a similar idea.
  2. Student Assessment Journey – programme level views of student assessment activity for students and staff
  3. Flexible innovative assessment – making the assessment and feedback tools at all points in Blackboard, not just for assignments, so that you can start thinking about using any tool for assessment purposes.
  4. Double marking – further work on the current functionality to take it to a robust, fully-usable level.
  5. Programme Level Assessment – looking at assessment practice across entire programmes and thinking about programme-level learning outcomes.

Hearing back from the rest of the room we discussed in more depth some of the already described above (external examiner access, programme-level views of assessment) and the Blackboard team promised to send round some case study examples of good practice for external examiner processes using Blackboard tools. In a discussion around whether and how institutions were using the Delegated Grading functionality, which was designed for UK HEIs, again the Blackboard team said they would gather together some case studies of where these are being used well. The feeling from the room was that this kind of functionality should be available across all assessment tools rather than locked to a single tool.

A few other interesting discussion points to end the morning session were that many institutions are thinking at programme level about replacing traditional assignment assessment. Video assignments and feedback are rapidly on the rise but also causing headaches as infrastructure and policy isn’t keeping up.

Afternoon Session

As I said, the afternoon session followed the same structure. So our post-lunch digestif activity was another Padlet , this time thinking about innovation and new practices that would enhance assessment and feedback in our institutions. I’ll let the Padlet do the talking so scroll around to see the ideas. I was interested in things like students being able to select the kinds of assessment that they wanted to do, and learners and academics developing assessment literacies through feedback dialogue and feed-forward as a continuous process.

Made with Padlet

As in the morning, next was a group task, where we were asked to think “aspirationally” about how we imagined assessment could look, if we had a free rein. What change or innovation in assessment and/or feedback would have significant impact and how would it benefit learners, tutors, courses and institutions? Essentially we were encouraged to go wild in the aisles of transformative assessment practice.

Our group went Back to Basics and offered the transformative potential of programmes where learning outcomes were mapped to assessment.  Well, someone had to. Other groups had some tidy, Tomorrow’s World ideas including:

  • An assessment wizard which built the kind of assessment you wanted with one view for staff and students (no more multiple systems or at least hiding these from you).
  • A tool that surfaced programme level assessment data.
  • A tool for personalised feedback and assessment routes – feedback raises flags on further help students can get and other staff can see that in later assessments.

And as per the morning session the Blackboard team led a discussion on how their products could work to do some of these things. One thing they did bring back to my attention was the Goals and Outcomes system which has a new dashboard view of the data and I think it would be opportune to review this in the light of programme development work that is heading the way of our team, as this could present an opportunity for offering programme-level views of progress through modules.

The end

So not a sales day, not your regular product roadmap/roadshow day, this represented a deeper dive into Electronic Management of Assessment, including the Blackboard tools that can be a part of that that environment. The Blackboard team wanted the day to be about sharing practice, raising awareness of what Blackboard tools we have already and encouraging us to get the best we can out the Blackboard tools and products that you have and I think this was more than achieved on the day. Thanks to the team and to the Bb North UserGroup for arranging and hosting. I had some very useful conversations (which is pretty much usual for the BB North User Groups meetings) and plenty from all of the above to take back for the project board overseeing the implementation of an e-submission policy at Liverpool.

What music did I first buy? The Muppet Show album. And I’ve never needed any other in my life…

Dan

Winter School 2017 – Week 2 Workshops – 16th – 20th January

The Winter School began today with a great session on engaging students visually in lectures with loads of ideas for presentation materials. There are a number of other sessions in the first week , which you can read about here and still book the last few places here.

This year we are also running a second week of sessions (Monday 16th to Friday 20th January) looking at technologies for learning and teaching beyond VITAL and a number of which are designed to tie-in with the online Bring Your Own Device for Learning 2017 events. All sessions listed below and bookable at: https://www.liverpool.ac.uk/cll/booking/

  • Monday 16th Jan – An introduction to Twitter (2 – 4pm)
  • Tuesday 17th Jan – GoMobile user group meeting (1 – 3pm)
  • Wednesday 18th Jan – e-submission and e-marking workshops
    • An Introduction To Electronic Submission Of Coursework (9:30 – 11am)
    • The Turnitin Assignment Tool For e-submission (Part 1) And GradeMark For Feedback (Part 2) (12 – 2pm)
    • The Blackboard Assignment Tool For e-submission (Part 1) And Feedback (Part 2) (2:30 – 4pm)
  • Thursday 19th Jan – Student as co-creators (11am – 1pm)
  • Friday 20th Jan – Advanced Twitter (2 – 4pm)

Finally, after these two weeks are a couple of events which might interest you.

  • Wednesday 25th Jan – Building good VITAL modules – a practical session looking at ways of building on the VITAL Baseline to create well-designed modules.
  • Thursday 26th Jan – eLearning Network meeting. In this meeting we will be getting a first look at the new Turnitin Feedback Studio.

All sessions bookable at: https://www.liverpool.ac.uk/cll/booking/

winter-school

Winter School 2017 diary dates

winter-school

A quick post to let you know about the eLearning Unit’s 2017 Winter School. All sessions bookable at: https://www.liverpool.ac.uk/cll/booking/

We’ve scheduled the first full week of sessions as listed here.

  • Monday 9th Jan – Engaging Learners visually in lectures – tools, tips and tricks (1 – 3pm)
  • Tuesday 10th Jan – VITAL Essentials (10am – 12pm)
  • Tuesday 10th Jan – An introduction to Technology Enhanced Learning (13:00 – 16:00)
  • Wednesday 11th Jan – Running webinars and online classrooms with Adobe Connect (10am – 12pm)
  • Thursday 12th Jan – 2nd annual Pedagogic Research Conference (all day)
  • Thursday 12th Jan – Turnitin GradeMark (2 – 3:30pm)
  • Friday 13th Jan – Stream Capture for lecture capture and screencasting (1 – 3pm)

This year we are also running a second week second week of sessions (Monday 16th to Friday 20th January) a number of which are designed to tie-in with the online Bring Your Own Device for Learning 2017 events. All sessions bookable at: https://www.liverpool.ac.uk/cll/booking/

  • Monday 16th Jan – An introduction to Twitter (2 – 4pm)
  • Tuesday 17th Jan – GoMobile user group meeting (1 – 3pm)
  • Wednesday 18th Jan – e-submission and e-marking workshops
    • An Introduction To Electronic Submission Of Coursework (9:30 – 11am)
    • The Turnitin Assignment Tool For e-submission (Part 1) And GradeMark For Feedback (Part 2) (12 – 2pm)
    • The Blackboard Assignment Tool For e-submission (Part 1) And Feedback (Part 2) (2:30 – 4pm)
  • Thursday 19th Jan – Student as co-creators (11am – 1pm)
  • Friday 20th Jan – Advanced Twitter (2 – 4pm)

Finally, after these two weeks are a couple of events which might interest you.

  • Wednesday 25th Jan – Building good VITAL modules – a practical session looking at ways of building on the VITAL Baseline to create well-designed modules.
  • Thursday 26th Jan – eLearning Network meeting. In this meeting we will be getting a first look at the new Turnitin Feedback Studio.

All sessions bookable at: https://www.liverpool.ac.uk/cll/booking/

E-submission and e-marking workshops and support 2017

esubmission long

In support of the University’s e-submission policy for the academic year 2016-17 the eLearning Unit continue to run e-submission and e-marking workshop days throughout the academic year. The days consist of three sessions described below and you are welcome to attend any or all, in whole or in part. We also run tailored school and departmental workshops and if you are considering training or consultancy from the team for e-submission and marking processes and systems, please do get in touch as soon as possible. You will also find dedicated online support resources from the eLearning Unit in the VITAL module VITAL Baseline and support, on which all staff are enrolled.

To search for and book onto any session please see the CLL booking page here. Scheduled e-marking workshop days (all on Wednesdays) for the rest of 2017 are:

  • January 18th
  • February 22nd
  • March 15th
  • April 26th
  • June 14th

Session 1: An introduction to electronic submission of coursework

(Please note this is not a lab-based session). The University is moving to online submission for coursework assignments (see this blog post for more detail on policy). To help staff get an overview of the available tools, understand associated current issues, and plan your own approach to e-submission the eLearning Unit offer this session comparing and discussing the two tools in VITAL for electronic submission, the Blackboard assignment tool and Turnitin.

In this two hour session we will look at:

  • An overview of e-submission in Higher Education and at Liverpool, including the student view
  • A discussion of attendees’ current practice
  • Analysis and examples of the two available assignment submission tools, Blackboard Assignment and Turnitin Assignments and the linked tools available for each.
  • Institutional guidance and key issues for planning assignment submission ‘workflows’.
  • Follow-up support and guidance.

Session 2: Turnitin Assignment tool for e-submission (part 1) and GradeMark for feedback (part 2)

This practical session runs in two parts. After a quick recap of the key differences between Turnitin and Blackboard we will firstly look at the different ways in which the Turnitin Assignment tool can be set up for online assignment submission (part 1 – 45 minutes approx). Then in the optional second part (45 minutes approx) we will try out the GradeMark feedback facilities available, from Criteria marking sheets to inline comments, both in desktop and mobile versions. If you would like to try out the mobile version of the grading tool please could you bring your device. This is available for iOS devices only at the moment and can be found here: https://itunes.apple.com/gb/app/turnitin/id657602524?mt=8

Session 3: The Blackboard Assignment tool for e-submission (part 1) and feedback (part 2)

This practical session runs in two parts. After a quick recap of the key differences between Turnitin and Blackboard we will firstly look at the different ways in which the Blackboard Assignment tool can be set up for online assignment submission (part 1 – 45 minutes approx). Then in the optional second part (45 minutes approx) we will try out the grading facilities available, from Criteria marking sheets to inline comments, both in desktop and mobile versions.

Please contact the team if you have any questions about these sessions.

Dan