Image 1 - Vice Chancellor Professor Mark E. Smith (front-centre) with the Managed Learning Environment team (from left to right: Louis Pliskin, Graham Robinson, Dean Wall, Matthew Deeprose, Sam Cole.

Photographs taken by Sofy Bazzini. Words by Matthew Deeprose.

Whatever stage we are at in our lives, it’s very likely that we can remember when we last took an exam. Almost one fifth of current University of Southampton students have been taking their exams on a computer.

The University of Southampton is leading the way in the uptake of computer-based examinations. At the heart of this innovation is the Managed Learning Environment (MLE) Team who have been developing and supporting the computer-based exams experience at UoS since the early 2000s. Last week, our President and Vice-Chancellor Professor Mark E. Smith visited the MLE team to find out about their vital work in supporting several key pillars of the University’s strategy.

Computer-based exams (CBE)

With almost two decades of experience of running computer-based exams and developing the software and infrastructure, the MLE team have made the current exam experience as smooth and stress-free for students as possible. The MLE team took Mark through the CBE process.

Starting the exam

Any University public workstation can be used to take an exam. The exams office will block book workstation rooms and arrange for invigilators to be present in each room. The invigilators, exams office, and the MLE team communicate using two-way radios. The invigilator will prepare a computer by simply logging into University PC.

Once the student sitting the exam enters their ID number their student card and id photo appears, allowing the invigilator to ensure the student sitting the exam is the same person that appears on their ID card.

Image 2 - Accessing ExamStart

Taking exams can be a nerve wracking experience. In earlier years students would forget their password and need IT support to start their exam. The latest improvement to the ExamStart system, developed over the past ten years by Graham Robinson and Dr Pei Zhang, is that students can start an exam even if their password has been reset or they use a password manager.

Exams may use OnDemand which is used for question types such as multiple response and matching questions, or a stripped-down version of Word, with no aids such as grammar and spelling checks, for essay-based exams, or even niche tools such as SPSS, Jupyter Notebook, or Authentic World.

A solid and supported experience

As students take their exam backups are taken every minute. This means that if a computer crashes or fails, the student can be relocated to a vacant computer and continue taking their exam. At the end of the exam, the papers are saved to a secure server. Essays can be printed or viewed on-screen. In their office based on Highfield campus, the MLE team are able to keep a vigilant eye over this essential activity.

Before an exam, workstations run self tests to identify any issues before a student uses the computer. During exams, workstation screens and rooms are monitored remotely from the MLE office at Highfield.

Image 3 - The MLE team can monitor multiple workstations being used to take exams.

Image 4 – During the exam period the MLE team may monitor activity in the workstation areas using the lecture capture cameras.

When Mark arrived the BIOL1020 exam was about to start. Graham and Louis demonstrated to Mark how they could ensure that exams in multiple rooms around our campuses started effectively. With communication between the Exams Office, Exam Invigilators, and the MLE team by two-way radio, if any issue cannot be resolved remotely, the team can attend in person. Not only can the exams experience be stressful for students, invigilators rely on the technology working and to be able to call for and receive assistance within minutes when they need it.

Image 5 - Graham Robinson and Louis Pliskin give an overview of the computer based exams process

“It’s all about improving the staff and student experience and maximising our effectiveness,'' said Graham. As the exam began, Mark saw an overview of ExamStart and then tried taking an exam himself. With a background in Materials Physics, Mark was able to try an example exam put together with five questions randomly chosen from a first year physics exam to demonstrate the exam taking experience.

Image 6 - Mark begins an example exam. Louis points out elements of the user interface.

Reflecting on assessment practices

Using computer-based exams can improve efficiency and effectiveness. Exams can be generated from a larger pool of questions, and students given questions at random from a pool. Fixed response questions can be marked immediately, and statistics generated to help determine whether the question was a “good” question.

Mark shared with the team his own journey with assessment and how he used it in his teaching. He reflected with the team on changes he had made in his own assessment practice to improve the retention of students on his modules.

A consistent process

Exams typically take place at the end of each semester, with supplementary exams taken before the start of the next academic year. As such the MLE team need to carefully manage their presence in the office to ensure that the right number of staff are always available to support this vital practice. Just in the academic year 2018/19, 18,528 exam papers were sat at a computer.

This includes those students with “Additional Exam Requirements” who have been given dispensation to use a computer to write an exam that their fellow students are still expected to use pen and paper to complete. In many ways the process behind a computer-based exam is similar to paper based exams. Academics must submit their final papers to the exams office by about six weeks before the exam, and this is no different for computer-based exams.

While the technology would allow changes up to just before an exam is taken, it is vital the same level of quality checking and assurance as is expected with paper based exams is applied to those taken on a computer.

Image 7 - Mark reviews his results with Graham

Mark was very impressed with what he saw of ExamStart and the processes and procedures that the MLE team has developed and optimised for CBE over many years and said, “It is really interesting to learn about what capability we have developed at Southampton in CBE. In becoming a digitally-enabled university making real use of such an approach can set us apart. It was very impressive to see first-hand what the team have already achieved”.

Toward the end of their meeting with Mark the MLE team raised some other important aspects of their work.

Turnitin

Lecture Capture

Lecture capture is highly valued by University of Southampton students. The MLE team introduced lecture capture technology a decade ago already. Mark was pleased to hear that 179 teaching spaces are equipped for recording lectures and 104 whiteboards can now be captured. There is no technical reason preventing the further uptake of lecture capture. The reliability of the cameras and other infrastructure is vitally important here as well.

Following continuous improvement practices put in place by the MLE team, cameras used for lecture capture automatically alert the team when faults are detected.

MySouthampton mobile app

The University’s MySouthampton app is going from strength to strength. The team demonstrated to Mark how usage of the app has increased year on year. They outlined the most popular features, which are timetable and email checking, campus maps, and Unilink bus times.

Mark is keen for students to be able to find available study spaces on campus wherever they are located and the MLE team look forward to taking part in a project to monitor available study space and make it identifiable within the app. The MLE team work closely with Communications on the app.

Asuccessful development has been the use of the app to support Open Days. Prospective students and their parents can use the app before the event to identify and set reminders for which sessions in particular they wish to attend, such as workshops on preparing their personal statement. During the Open Day the app reminds attendees when it’s time to attend their preferred sessions.

Blackboard

Blackboard is probably the best known of the applications that MLE support. In 2018/19 members of UoS spent more than 70 combined years using Blackboard, which has more than 6500 actively used courses. Not only students use Blackboard, as well as running their own modules on Blackboard, staff will find themselves taking health and safety and diversity training, as well as other professional development activities.

This year the MLE team will be working closely with HR to implement the “Leading for the Future” line manager training which will be hosted on Blackboard. The MLE team gave Mark an overview of some key improvements made to Blackboard of late and showed that statistics on usage demonstrate the effectiveness of these improvements.

Accessibility

The MLE team has also been leading the way in the understanding of The Public Sector Bodies (Websites and Mobile Applications) (No. 2) Accessibility Regulations 2018 .

These regulations apply to public sector bodies and require that their web-based services, documents, videos and mobile apps meet an accessibility requirement. What is this requirement? In simple terms it is to follow these four steps:

Understand the impact of the regulations.
Check the accessibility of institutional websites, media, and apps, including those provided by third parties
Make a plan to fix any ways the above do not fulfil the relevant requirements of the European standard on the accessibility requirements suitable for public procurement of ICT products and services in Europe (EN 301 549).
Publish an accessibility statement for each relevant service.

Toward the end of Mark’s visit to the MLE office, Matthew Deeprose demonstrated how the MLE team had been making an impact in raising the awareness and importance of these vital regulations.

“The services we support within the MLE team are among the most used at the University by the widest range of staff and students. Therefore, it is essential not only to understand our responsibilities regarding these regulations, but to align our services with them.”

Timeline of MLE activities related to accessibility regulations

A key moment in this journey was the work to implement a new responsive theme for the Blackboard VLE. “The responsive theme provides a much better experience, but to implement it we had to configure the theme to use the University brand colour scheme. When we choose colours from the University brand, we must ensure that the colours have enough contrast to ensure their legibility. In the regulations, this means that when text is presented, the colour of the text has a minimum contrast ratio of 4.5:1 with the colour behind it.”

The University colour scheme has 21 colours, making for 441 possible colour combinations, but through testing each one the MLE team found that only 19% of those are at or above the minimum threshold to be used for the presentation of text. Matt created a “Brand Palette Accessibility Matrix”, where each possible colour combination and its accessibility rating can be looked up. It has already been used hundreds of times to ensure that content is both on brand and accessible.

Section of Brand Palette Accessibility Matrix. AA is the minimum rating and refers to colours having a 4.5:1 or higher contrast ratio. AAA is the enhanced rating where colours have a 7:1 or higher contrast ratio. G is the minimum allowed for graphics elements such as icons and diagrams.

The Accessibility Statement allows us to describe the accessibility of services and explain any issues and what we are doing to resolve them. They should also provide further information about how users can make effective use of assistive technologies such as those built into browser and operating systems.

A recent review of Accessibility Statements published on Electronics and Computer Science’s Lexdis site found that in October 2019 less than 4% of organisations covered by the regulations had compliant accessibility statements.

This is another area where the MLE team are leading the way. Back in April 2019 they published the first ever accessibility statement for a VLE, and shared it with the wider community. After sharing practice on the Blackboard community site, and JISC Digital Accessibility Regulations Mailing list, Matt has been invited to write blog posts, and present at conferences on the theme of accessibility.

Image 9 - Matthew Deeprose talks Mark through the Colour Accessibility Matrix

“Regardless of the legal mandate, which some institutions are debating ,'' concluded Matthew, “it’s what is morally right to do: to remove barriers and ensure that every member of our community can get the most out of the services and platforms that we work hard to develop and maintain, and that they rely on for their success whilst at Southampton.”

Saying goodbye

The meeting between our Vice-Chancellor and the MLE team was tinged with sadness. Sam Cole, a member of the MLE team since its creation in 2003 is leaving the University to take up a new post at Falmouth University. The team wishes Sam all the best in this next step in his journey.

Valued team member Dr Pei Zhang was also unable to attend due to illness.

Conclusion

Vice-Chancellor Professor Mark E. Smith was very impressed with the team and their activities, he concluded, “The range of activity covered by the team from the VLE, to lecture capture, to computer-based examinations, to website accessibility is remarkable. The team deals with issues that play centrally into our strategy of making our student experience outstanding. It was hugely encouraging to see the enthusiasm and commitment of the team”.