Black Sabbath to Busted – Report on Blackboard Enhanced Assessment and Feedback Event

Leaving home under an early-morning starry sky and up over the misty, snowy Pennines to Sheffield, I wasn’t sure what to expect from the Blackboard Enhanced Assessment and Feedback day to which I was travelling. I had a sketched outline of the themes for the day ahead but not much more, namely that we would engage in some way with:

Assessment and feedback – an institutional perspective

  • Examination of key drivers and challenges (reputations, quality of process, quality of data, legal, business efficiencies, risk etc.)
  • Placing use cases on a confidence/effectiveness model
  • Highlight relevant Blackboard solutions

 Assessment and feedback – innovation and organisational change

  • Identifications for drivers for change
  • Gap analysis of practice and stakeholder experiences
  • Prioritisation of opportunities for change
  • Highlight relevant Blackboard solutions

This kind of workshoppy day from Blackboard was something I hadn’t experienced before, so I was propelled by curiosity as much as that it seemed relevant to the work that our team are currently leading on e-submission and e-feedback at Liverpool. What I got was a useful day of frank discussion and sharing of experiences, ideas and commonalities, which was mostly reassuring, with colleagues from other institutions in the kind if depth you don’t often get. This is especially useful for a sense of the bigger picture in HE, to talk about the differently badged or described but largely similar activities, structures and strategies that are top of our agendas at the moment, e-submission and e-marking being one of Liverpool’s current strategic TEL focus. One universal and rapidly-emerging area of concern that became evident on the day was a need for a variety of programme-level views of assessment activity in the VLE for academic and administrative staff and students. This is a long-requested feature from Blackboard usergroups that’s time has come with the adoption of e-submission and e-marking policies across the sector of late and I hope this was the main takeaway message for the Blackboard team.

The event was run by our regional Blackboard Customer Success Team, in partnership with the BB North user group, recognising a need for a more extensive exploration of particular issues that get raised at user group meetings, where the format doesn’t allow fuller discussion. Whilst advertised as intended for senior leaders, learning technologists, TEL managers and academic staff, the majority attending today were learning technologist types. It was instructive to hear that on the previous day at Edinburgh a couple of PVCs had attended, sending some very positive signals about the depth of an institutions’ engagement and intent with the actual tools that students, academic and professional service staff use as a part of their everyday life at the University.

Music, sweet music…

Our first activity was to introduce ourselves telling the room the first piece of music that we had ever bought. An astonishing array of formats and first loves was paraded, from Now compilation tapes to Avril Levigne downloads, from Osmonds vinyls to Busted CDs. Blackboard’s Alan Masson and Gillian Fielding all admitted to their first purchases, but I’ll spare their blushes here. Top-trumping all these, however, was Blackboard’s Steve Hoole, whose overnight Novotel stay featured a vinyl deck (remember these, kids?) and a selection of Sheffield synth heaven albums to spin the night away.

Structure of the sessions

The morning and the afternoon were structured in a similar way so that we would first ‘brainstorm’ our thoughts in groups on a set of e-assessment themes, then work together on some specific ideas from those and bring something interesting back to the room. We’d finally end with a discussion of the potentially useful tools in Blackboard that could be a part of the thinking for some of these. I initially thought this last element was going to be a sales pitch but it was pleasingly nuanced in that the Blackboard team wanted to hear stories of how people are using these tools, where they’re working well and what the gaps are. A very clear point made more than once was that the recent Blackboard activity in developing the assignment tool to offer dual marking, moderation and anonymous marking had been excellent but it now seemed that the Blackboard focus had moved on from this, whilst the toolset still needs work, that there had only been one iteration of the process. Also that this process of close consultation should be constantly repeated for other areas as our needs are constantly evolving, not just for assessment.

Morning Session – what’s needed to enhance assessment and feedback practice?

For the morning, we’d thought about quality, processes and workflows and where the opportunities for enhancement lay. We chucked all of our ideas at the first Padlet below. You’ll find all of the issues that we have encountered in the course of our work as learning technology developers and as part of the University’s EMA project, from how to handle video submission and feedback, to combining some functionality of Blackboard assignments (group tools, double marking, staged release of feedback, etc) with that of Turnitin assignments (Originality Checking, GradeMark, offline marking). In fact, this was another big ask on the day, that Turnitin and Blackboard align/integrate their products in ways that will help us, as you will gather from some of the posts in this Padlet.

Made with Padlet

What leapt out at me was a so-far un-encountered issue at Liverpool of needing a read-only external examiner access to modules. In some institutions administrative staff are packaging up content and assignments into a special section of a module, which only the external has access to, and making the rest of the module inaccessible to them. This is to meet an anxiety around externals potentially changing grades and altering content, but it costs hours of administrative time, essentially duplicating what’s already in the module, so re-introducing at a later stage of the assessment cycle serious administrative burden that the electronic submission process had originally taken away from the front end. What’s needed is a read-only access enrollment level, which is another development idea for the Blackboard team to add to the suggestions box.

Moving on to the next activity in groups again, we were tasked with listing and describing up to five assessment and feedback enhancements that would have significant benefits for the listed stakeholders and the degree to which it would require resource. This photo shows our effort…

dsc01104

…and for those who aren’t adept at reading the handwriting of people who spend their whole day attached to a keyboard our five enhancements (all pretty standard) were:

  1. External Examiner Access – read-only access for external examiners or a similar idea.
  2. Student Assessment Journey – programme level views of student assessment activity for students and staff
  3. Flexible innovative assessment – making the assessment and feedback tools at all points in Blackboard, not just for assignments, so that you can start thinking about using any tool for assessment purposes.
  4. Double marking – further work on the current functionality to take it to a robust, fully-usable level.
  5. Programme Level Assessment – looking at assessment practice across entire programmes and thinking about programme-level learning outcomes.

Hearing back from the rest of the room we discussed in more depth some of the already described above (external examiner access, programme-level views of assessment) and the Blackboard team promised to send round some case study examples of good practice for external examiner processes using Blackboard tools. In a discussion around whether and how institutions were using the Delegated Grading functionality, which was designed for UK HEIs, again the Blackboard team said they would gather together some case studies of where these are being used well. The feeling from the room was that this kind of functionality should be available across all assessment tools rather than locked to a single tool.

A few other interesting discussion points to end the morning session were that many institutions are thinking at programme level about replacing traditional assignment assessment. Video assignments and feedback are rapidly on the rise but also causing headaches as infrastructure and policy isn’t keeping up.

Afternoon Session

As I said, the afternoon session followed the same structure. So our post-lunch digestif activity was another Padlet , this time thinking about innovation and new practices that would enhance assessment and feedback in our institutions. I’ll let the Padlet do the talking so scroll around to see the ideas. I was interested in things like students being able to select the kinds of assessment that they wanted to do, and learners and academics developing assessment literacies through feedback dialogue and feed-forward as a continuous process.

Made with Padlet

As in the morning, next was a group task, where we were asked to think “aspirationally” about how we imagined assessment could look, if we had a free rein. What change or innovation in assessment and/or feedback would have significant impact and how would it benefit learners, tutors, courses and institutions? Essentially we were encouraged to go wild in the aisles of transformative assessment practice.

Our group went Back to Basics and offered the transformative potential of programmes where learning outcomes were mapped to assessment.  Well, someone had to. Other groups had some tidy, Tomorrow’s World ideas including:

  • An assessment wizard which built the kind of assessment you wanted with one view for staff and students (no more multiple systems or at least hiding these from you).
  • A tool that surfaced programme level assessment data.
  • A tool for personalised feedback and assessment routes – feedback raises flags on further help students can get and other staff can see that in later assessments.

And as per the morning session the Blackboard team led a discussion on how their products could work to do some of these things. One thing they did bring back to my attention was the Goals and Outcomes system which has a new dashboard view of the data and I think it would be opportune to review this in the light of programme development work that is heading the way of our team, as this could present an opportunity for offering programme-level views of progress through modules.

The end

So not a sales day, not your regular product roadmap/roadshow day, this represented a deeper dive into Electronic Management of Assessment, including the Blackboard tools that can be a part of that that environment. The Blackboard team wanted the day to be about sharing practice, raising awareness of what Blackboard tools we have already and encouraging us to get the best we can out the Blackboard tools and products that you have and I think this was more than achieved on the day. Thanks to the team and to the Bb North UserGroup for arranging and hosting. I had some very useful conversations (which is pretty much usual for the BB North User Groups meetings) and plenty from all of the above to take back for the project board overseeing the implementation of an e-submission policy at Liverpool.

What music did I first buy? The Muppet Show album. And I’ve never needed any other in my life…

Dan

Winter School 2017 – Week 2 Workshops – 16th – 20th January

The Winter School began today with a great session on engaging students visually in lectures with loads of ideas for presentation materials. There are a number of other sessions in the first week , which you can read about here and still book the last few places here.

This year we are also running a second week of sessions (Monday 16th to Friday 20th January) looking at technologies for learning and teaching beyond VITAL and a number of which are designed to tie-in with the online Bring Your Own Device for Learning 2017 events. All sessions listed below and bookable at: https://www.liverpool.ac.uk/cll/booking/

  • Monday 16th Jan – An introduction to Twitter (2 – 4pm)
  • Tuesday 17th Jan – GoMobile user group meeting (1 – 3pm)
  • Wednesday 18th Jan – e-submission and e-marking workshops
    • An Introduction To Electronic Submission Of Coursework (9:30 – 11am)
    • The Turnitin Assignment Tool For e-submission (Part 1) And GradeMark For Feedback (Part 2) (12 – 2pm)
    • The Blackboard Assignment Tool For e-submission (Part 1) And Feedback (Part 2) (2:30 – 4pm)
  • Thursday 19th Jan – Student as co-creators (11am – 1pm)
  • Friday 20th Jan – Advanced Twitter (2 – 4pm)

Finally, after these two weeks are a couple of events which might interest you.

  • Wednesday 25th Jan – Building good VITAL modules – a practical session looking at ways of building on the VITAL Baseline to create well-designed modules.
  • Thursday 26th Jan – eLearning Network meeting. In this meeting we will be getting a first look at the new Turnitin Feedback Studio.

All sessions bookable at: https://www.liverpool.ac.uk/cll/booking/

winter-school

Winter School 2017 diary dates

winter-school

A quick post to let you know about the eLearning Unit’s 2017 Winter School. All sessions bookable at: https://www.liverpool.ac.uk/cll/booking/

We’ve scheduled the first full week of sessions as listed here.

  • Monday 9th Jan – Engaging Learners visually in lectures – tools, tips and tricks (1 – 3pm)
  • Tuesday 10th Jan – VITAL Essentials (10am – 12pm)
  • Tuesday 10th Jan – An introduction to Technology Enhanced Learning (13:00 – 16:00)
  • Wednesday 11th Jan – Running webinars and online classrooms with Adobe Connect (10am – 12pm)
  • Thursday 12th Jan – 2nd annual Pedagogic Research Conference (all day)
  • Thursday 12th Jan – Turnitin GradeMark (2 – 3:30pm)
  • Friday 13th Jan – Stream Capture for lecture capture and screencasting (1 – 3pm)

This year we are also running a second week second week of sessions (Monday 16th to Friday 20th January) a number of which are designed to tie-in with the online Bring Your Own Device for Learning 2017 events. All sessions bookable at: https://www.liverpool.ac.uk/cll/booking/

  • Monday 16th Jan – An introduction to Twitter (2 – 4pm)
  • Tuesday 17th Jan – GoMobile user group meeting (1 – 3pm)
  • Wednesday 18th Jan – e-submission and e-marking workshops
    • An Introduction To Electronic Submission Of Coursework (9:30 – 11am)
    • The Turnitin Assignment Tool For e-submission (Part 1) And GradeMark For Feedback (Part 2) (12 – 2pm)
    • The Blackboard Assignment Tool For e-submission (Part 1) And Feedback (Part 2) (2:30 – 4pm)
  • Thursday 19th Jan – Student as co-creators (11am – 1pm)
  • Friday 20th Jan – Advanced Twitter (2 – 4pm)

Finally, after these two weeks are a couple of events which might interest you.

  • Wednesday 25th Jan – Building good VITAL modules – a practical session looking at ways of building on the VITAL Baseline to create well-designed modules.
  • Thursday 26th Jan – eLearning Network meeting. In this meeting we will be getting a first look at the new Turnitin Feedback Studio.

All sessions bookable at: https://www.liverpool.ac.uk/cll/booking/

E-submission and e-marking workshops and support 2017

esubmission long

In support of the University’s e-submission policy for the academic year 2016-17 the eLearning Unit continue to run e-submission and e-marking workshop days throughout the academic year. The days consist of three sessions described below and you are welcome to attend any or all, in whole or in part. We also run tailored school and departmental workshops and if you are considering training or consultancy from the team for e-submission and marking processes and systems, please do get in touch as soon as possible. You will also find dedicated online support resources from the eLearning Unit in the VITAL module VITAL Baseline and support, on which all staff are enrolled.

To search for and book onto any session please see the CLL booking page here. Scheduled e-marking workshop days (all on Wednesdays) for the rest of 2017 are:

  • January 18th
  • February 22nd
  • March 15th
  • April 26th
  • June 14th

Session 1: An introduction to electronic submission of coursework

(Please note this is not a lab-based session). The University is moving to online submission for coursework assignments (see this blog post for more detail on policy). To help staff get an overview of the available tools, understand associated current issues, and plan your own approach to e-submission the eLearning Unit offer this session comparing and discussing the two tools in VITAL for electronic submission, the Blackboard assignment tool and Turnitin.

In this two hour session we will look at:

  • An overview of e-submission in Higher Education and at Liverpool, including the student view
  • A discussion of attendees’ current practice
  • Analysis and examples of the two available assignment submission tools, Blackboard Assignment and Turnitin Assignments and the linked tools available for each.
  • Institutional guidance and key issues for planning assignment submission ‘workflows’.
  • Follow-up support and guidance.

Session 2: Turnitin Assignment tool for e-submission (part 1) and GradeMark for feedback (part 2)

This practical session runs in two parts. After a quick recap of the key differences between Turnitin and Blackboard we will firstly look at the different ways in which the Turnitin Assignment tool can be set up for online assignment submission (part 1 – 45 minutes approx). Then in the optional second part (45 minutes approx) we will try out the GradeMark feedback facilities available, from Criteria marking sheets to inline comments, both in desktop and mobile versions. If you would like to try out the mobile version of the grading tool please could you bring your device. This is available for iOS devices only at the moment and can be found here: https://itunes.apple.com/gb/app/turnitin/id657602524?mt=8

Session 3: The Blackboard Assignment tool for e-submission (part 1) and feedback (part 2)

This practical session runs in two parts. After a quick recap of the key differences between Turnitin and Blackboard we will firstly look at the different ways in which the Blackboard Assignment tool can be set up for online assignment submission (part 1 – 45 minutes approx). Then in the optional second part (45 minutes approx) we will try out the grading facilities available, from Criteria marking sheets to inline comments, both in desktop and mobile versions.

Please contact the team if you have any questions about these sessions.

Dan

New Turnitin features – August 2015

In this summer’s upgrade to VITAL there were a couple of enhancements to Turnitin worth knowing about.

1. New 40MB file upload size limit. The previous limit was 20MB. Students can now submit one file of up to 40MB to a Turnitin assignment.

2. ‘Email late submitters’ facility.​ Send an email to all students who have not yet submitted to an assignment. This feature also works with anonymised assignments.

And as a refresher, there have been a number of useful new features introduced into Turnitin over the last couple of years which are listed below along with some links to more details. Any questions about any of the below please do contact the eLearning Unit (elearning@liv.ac.uk).

1. Submit PowerPoints, Excel and Google docs for originality checking. The range of file types that can be submitted by students and which Turnitin will scan for originality and create an Originality Report includes PowerPoint, Excel and Google Docs as well as Microsoft Word, WordPerfect, PostScript, PDF, HTML, RTF, OpenOffice (ODT), Hangul (HWP), and plain text. They should be less than 40MB and contain at least 20 words of text. When creating the assignment, on the page where dates and titles are entered select the option “Allow only file types that Turnitin can check for originality”.

2. Submit images. Students can submit images in the JPEG, GIF, PNG, TIFF, BMP, or PICT formats. These are not originality checked so will not generate an Originality Report but can be opened directly in GradeMark for comment, feedback and grading. They should be less than 40MB. When creating the assignment, on the page where you add in the dates and title select the option “Allow any file type”.

3. Submit any file type. A new option at the assignment set-up stage will let students submit any file type they wish to the Turnitin assignment. (When creating the assignment, on the page where you add in the dates and title select the option “Allow any file type”). Originality checking cannot be guaranteed here. The Originality Check facility will try to scan student submissions but will not return a value or report if it cannot. GradeMark will also try to display the file directly in the Turnitin document viewer but again, if it cannot do this for a particular filetype then it will display the submission as a link to download the file to open in the relevant application. You can still use all of the GradeMark feedback facilities and a blank page for QuickMark comments is available in the case where the file is only available as a download. Files must be less than 40MB.

4. Grade without submissions. You can use the GradeMark facilities with no need for the students to submit work. An example context of this kind of use is an assessed presentation – GradeMark could be used during the presentation on an iPad or laptop to make grading notes, notes on any criteria marking, and offer the overall comment, grade, criteria marks and/or audio comment afterwards.

5. Link inline comments and marks with Rubric card criterion. If using a rubric card and also the QuickMarks to comment inline on student work you can choose to link a particular QuickMark/comment with one of your criterion which is then visible to the student on the rubric – extremely useful for evidencing the levels attained by students in the criterion. For more details of this feature see this blogpost from Turnitin.

6. ‘Grading Forms’ (simple criteria marking sheets). Essentially a scaled-down rubric/criteria electronic marking sheet, as exemplified in the screen shot below. For full details on this new feature see this blogpost from Turnitin.

An example of the new grading form for use with Turnitin Grade Mark

7. Upload criteria from Excel. Use the Turnitin Excel template to import your criteria from Excel which can make creating Turnitin criteria cards much faster when you have the cards existing in other digital formats. See this Turnitin item for details.

8. Student assignment submission ‘workflow’. The student submission workflow is a three stage process as it always has been. The stage where students check their work before submission is a paginated and expandable window on the submission confirmation page (stage two) containing the fully formatted work as submitted by the student (it used to be an unformatted view of the submitted text only). Students click on the image of their work if they need to enlarge it, and click through it using the on screen arrow icons to check their work.

This is an important detail as students need to confirm that the content displayed here corresponds to what they expect to submit (i.e. there have been no glitches etc in the upload process) and any submission guidance should be updated to reflect this. Students can also now submit directly from DropBox and GoogleDocs, as well as upload from their desktop machine. For more details please read this Turnitin blogpost.

9. Quantitative % grading schemes – do not need to add up to 100%. They can be less than 100% if required.

Turnitin continually seek ideas and feedback from users on how to improve the system (which includes GradeMark and PeerMark also). You can see the current list of ideas, vote for them, add you own etc on the Turnitin Roadmap forum from any Turnitin assignment classlist. See this blog post for more details on the Turnitin Roadmap.

We’re offering workshops on GradeMark and Turnitin this academic year. Keep an eye on the CLL Booking site or our twitter feed for dates.

Dan

Turnitin GradeMark on your iPad…

…and hurrah! it includes a facility for working offline.

This summer the iPad app for Turnitin GradeMark was generally released after a long period of Beta-testing and a lot of anticipation.

Available free from iTunes (https://itunes.apple.com/us/app/turnitin/id657602524) this app will let you carry out all of the grading and feedback functions you are used to with GradeMark.

Before you can use the app you will need to make a settings change which is detailed in this FAQ from our VITAL Self Service tool (click the link). The FAQ also explains how you create a class code so that you can link from the VITAL module containing the assignments to the iPad app.

We’d be very interested in hearing any reports from staff trying this app out so please do get in touch with the eLearning Unit to discuss your experiences.

Augmented Reality

Imagine hovering your smartphone camera display over a poster on a wall and seeing a video appear on screen related to the content of the poster. Augmented reality (AR) works in this way. See the video below for an example. It merges content created virtually (videos, images, animations, graphics) with real world environments. This virtual content is layered over a real-life object, whether that is a picture, a person or even a building, the options are endless. Viewed through the camera display on a smartphone or tablet device, these AR ‘overlays’ can be triggered by a GPS location, a sound, or through the recognition of an image (usually a photograph, graphical display, painting or poster). This is dependent on the application used but if content is synchronised, using an AR app, then content can ‘overlay’ information, for example, onto a famous landmark, or help someone to navigate around an a city area.

Newspapers, magazines and advertising agencies have already made use of AR. I’ve seen one app used to animate logos and text displays in the headlines of newspapers. It is fun seeing an advertisement in a newspaper come to life as an animation, however, this use of AR appears, in my opinion, rather gimmicky and the ‘fun’ effect was one that quickly waned. In terms of using two forms of media together, I think that comes down to preference. Certainly in my own experience I’m unlikely to use my phone or tablet for AR purposes whilst simultaneously reading a newspaper. AR in the context of entertainment will inevitably face practical questions as to why audiences should use AR in conjunction with other firmly established forms of media. Will the technological development of AR echo the anticipated rise and now more recent fall of the 3D TV? I do believe AR has a place that, once identified, will integrate it into certain elements of everyday practice. My hope is that this practice includes education and the interactivity found in learning environments.

AR in Education

It’s hard to know where to start when choosing an app to work with AR. A brief search through the Google Play and online App stores reveals an explosion of AR apps all focusing on different types augmented experiences. This ranges from live virtual views of someone’s location, using map software, to producing music using a hand-drawn piano layout.

After trying a lot of different AR apps, I came across Aurasma. Aurasma is a smart device app that was demonstrated by Judy Bloxham at the e-Assessment Association hosted by Manchester Metropolitan University in November 2012. Judy’s involvement in an AR project, supported by JISC and their RSC Northwest Network, is powered by Aurasma and has utilised AR for the purposes of enhancing educational content. The project compiled a series of posters that aided formative assessment and became interactive when students used the Aurasma app alongside them. This helped students to engage in a reflective thinking process drawing upon some of the key principles they had covered in their learning of the subject. Using AR and interactive content like this can also provide an ideal entry point for a learning activity to begin – with options to break down content into a series of options that students can choose to follow.

A good example is outlined in the poster below (provided by Judy Bloxham), allowing the user has to use Aurasma to interact with a series of buttons and videos to see which is the correct shelf for storing raw meat in the fridge.

fridge[Link to the original website for this poster:  http://tinyurl.com/dxhnbhm]

You can use Aurasma’s image recognition tool and see how this poster can trigger interactive learning content. This can be tested on a computer screen or using a printed version of one of the posters linked below. There are two ways to do this depending on the smart device that you are using. Some instructions are outlined below.

Android devices:

1. Visit the Google Play store to search and download Aurasma
2. Open Aurasma and press the ‘A’  icon at the bottom of the screen
3. This takes you to the Explore section. Press the magnifying glass at the bottom of the screen to go to the Search page.
4. Type in RSC Northwest to find the RSC Northwest Channel. Click the image and then click the ‘Following’ button on the next screen.
5. Now press the bracket icon [ ] on the bottom line to return to the camera view. Then view over one of the posters below.

Apple devices:

1. You can download Aurasma in the same way as above (for the Android devices) and follow the options to connect to the RSC Northwest channel.
2. Alternatively, search RSC Northwest in the App Store and install it to your Apple device.
3. Use the camera view and hover one of the posters below. Make sure the whole of the poster is displayed on your smart device.

hazards-v2.1

[Link to the original website for this poster:  http://tinyurl.com/dxhnbhm]

hearing

[Link to the original website for this poster:  http://tinyurl.com/dxhnbhm]

These are examples of how an image can trigger an Aurasma ‘aura’. Open one of the PDF files to full screen and hover your smartphone device over the image. If the app has joined the channel correctly the image should trigger the interactive ‘aura’ content. Do let us know in the comment box below about your experiences using this app.

To read more about the use of AR used in education then visit the following sites below. These were recommended by Judy Bloxham, who I would like to thank on behalf of the eLearning Unit here at the University of Liverpool, for allowing us permission to make use of the above posters, demonstrating the use of Augmented Reality in education. Her work in raising awareness about the potential benefits of AR in education is ongoing and well worth following. Below are some further links related to the information above and to other AR projects, if you would like to delve further.

https://sites.google.com/a/jiscadvance.ac.uk/augmented-reality/home

Guardian Blog Post – http://www.guardian.co.uk/higher-education-network/blog/2013/feb/11/augmented-reality-teaching-tool-trend

Scarlet Project – http://teamscarlet.wordpress.com/

cARe Project http://blogs.city.ac.uk/care/

Living Learning: Plumbing from Kendal College https://itunes.apple.com/gb/app/living-learning/id549156508?mt=8

written by Phil Walker