Learning Analytics - Opportunities and Challenges

  

Introduction

As the use of learning analytics broadens within the education sector I thought it would be useful to describe some of its use cases within the following contexts: predictive analytics, student profiling, learner support, student services, performance management, external facing websites, teaching and instructional design, value propositions and codes of practice. The opportunities and challenges for each use case are also highlighted. For the purpose of this article, I have decided to take a neutral position when describing the use of learning analytics across these contexts.

If you are new to analytics the following video from Educause is a useful starting point. The video describes the use of analytics within the higher education sector but many of its messages are also applicable to schools and colleges regardless of their size. It defines analytics as the use of data, statistical analysis, explanatory and predictive models to gain insights and act on complex issues. [1] These insights can deliver greater clarity into complex issues such as student retention and completion; enhancing the quality of teaching, learning and assessment; improving business operations; and managing the many services that are used by the student body. One of the most interesting elements that is highlighted in the video is the use predictive analytics and this is described in more detail below.

Analytics works better in organisations that have digitised core operations. This means that organisations have enabled prospective students to search, find, apply and pay for courses online; their customer facing websites are tagged and search engine data is used to inform the placement of current and future courses that are available at the institution; their definition of student data is very broad and is shared and supported by their student body; student data is used to enhance all student facing services; back office functions are digitised and they take advantage of customer relationship systems, content and document management systems and all workflows and processes are digitised.

Schools, colleges and universities can be placed into three distinct stages of development in their use of learning analytics. The first stage is were educational institutions focus on the governance, expertise, data, process, policies and the widening use of data and metrics to improve their business operations. The majority of schools, colleges and universities fall into this group.

Institutions in the second stage of development use analytics to leverage their strategic and operational capability to forecast, plan and to take action. The use of analytics leads to improved performance within faculties and operational service teams. Institutions make use of machine learning to enhance their suite of education services. The services that are accessed by students become increasingly segmented, personalised and contextualised to meet the myriad of needs and requirements for each student.

The third and final stage of development sees schools, colleges and universities using business intelligence and analytics tools to move themselves beyond enrollment, student progress and resource optimisation. It becomes an indispensible service that enables institutions to differentiate themselves from others so that they can secure a digital advantage over their competitors. The use of analytics in these institutions results in fundamental revisions or changes to existing services and leads to a new service paradigm within the education sector.

Analytics

Predictive Analytics

The use of predictive analytics by schools, colleges and universities is an area that remains underdeveloped and is not widespread. Despite this, the Open University in the UK and its work through the Knowledge Media Institute have for a number of years explored the use of analytics to enhance the quality of teaching, learning and assessment. The following video from the Open University dates back to 2012, but Professor Josie Taylor clearly explains the University's rationale for using predictive analytics and these principles are still applied today by the University. [3]

There a numerous use cases for predictive analytics for schools, colleges and universities. Firstly, institutions can use predictive analytics to profile students when they apply for courses. In this instance, institutions can analyse data and use predictive algorithms to identify those students who are most likely to be placed on an at risk register during their studies; they can predict anticipated grades and outcomes for all students; predict student retention and completion rates; predict progression pathways; and the projected cost and revenue for any given course. There are issues regarding the ethics of using algorithms and student data in this fashion; but the general consensus is that the use of predictive analytics in this manner fosters improved support for all new students starting their studies at the institution; it improves student retention and completion rates; it improves operational planning and efficiency; and it supports the institution to improve its short, medium and long term forecasts and plans.

Secondly, colleagues can use predictive analytics to inform and develop appropriate intervention strategies for each student, course, faculty and business support team within their institution. For instance, course teams and student support teams can utilise analytics to determine which students are at risk of not completing a component of their course. Colleagues use variables such as attendance, punctuality, submission times for assignments, the number of occasions a student has logged onto the institution's virtual learning environment to access course materials, activities completed on the virtual learning environment; and support interventions that have been recorded by colleagues to identify the students who may not succeed in a particular part of their studies or those who are falling short of anticipated grades and outcomes.

Thirdly, colleagues can use predictive analytics to plan for anticipated demand for courses and services that are offered by their institutions. For instance, faculties may ascertain from their records that their is a high probability of students taking a business studies module also taking a module in business start ups. If this is the case the faculty can prepare for this anticipated demand. The use of clustering can be particularly helpful for faculties and marketing teams when promoting courses and they can better differentiate services when communicating with each cluster group. You may also wish to look up the use of behavioural clustering as part of your predictive analytics and business intelligence toolset. The use of analytics in this manner means that institutions are better informed when designing courses and services to meet the needs of their students.

And fourthly, institutions can use predictive analytics to help them identify the courses that their current students are likely to enroll on or purchase next. The use of propensity modeling by institutions means that colleagues can examine student profiles, purchase and enrollment histories to identify the exact course(s) each student is likely to purchase or enroll on next. The use of propensity modeling can also be used in a college's or university's enrollment centre by enabling colleagues to make course suggestions to potential students.

One of the services that I find particularly helpful is the use of predictive analytics to anticipate student actions and queries. For example, a student entering a university campus receives a notification from the university library which states 'the accounting journal that you ordered is now available for collection. Students who borrowed this journal also borrowed the journal on tax law. This is also waiting for you at the library collection desk' or 'it's great to see that you are interested in the postgraduate course in accounting, but you only have two weeks left to submit your application. One of our student service reps is free to meet you tomorrow at 3pm after your afternoon lecture. Please click confirm to accept the invitation to see the student services rep'. In another scenario, a college student who is interested in applying for an undergraduate course is presented with services on her home page that will support her university application. The use of this service increases the number of university applications made by students at the college.

Student Profiling

Analytics can be used to support and enhance student profiling with two regards. Firstly, it enables institutions to identify, quantify and statistically represent broader characteristics of the student body such as age, gender, ethnicity, entry qualifications etc. And secondly, it enables institutions to quantify and identify students who require additional and specific support during and after their studies. Student profiling can be perceived as contentious if the variables in the algorithms are used inappropriately or if generalised assumptions are applied to a particular group of students. Institutions can review variables such as the student's prior qualifications; if the student has declared a learning support need or the academic record of the student if he or she has studied at the institution before. The use of student profiling is far from new; with many schools using data to stream students into specific classes or tiers.

The variables that are contentious may include the use of postal code, zip code, address and the ethnicity of students to determine their profiles. The use of these variables and many more is also legally dubvious as well. For example, the assumptions or generalisations that are used by the institution may state that students from a particular area of the town or those who belong to a particular ethnic group tend to have lower retention and completion rates than other student groups. Other variables that may be applied could include details such as: if the student's parents are married, separated or divorced; the occupations of the student's parents; the success of the student's siblings who previously studied at the institution or the student's household income. If institutions apply these variables they may assume that students who come from families that are not divorced do better than students whose parents are separated or divorced; students who come from wealthier families perform better than students who come from less wealthy families and so on. Please note that the use of these variables in this manner is not advocated.

Learner Support Services

The most common reason to date for using analytics and business intelligence tools is to identify students at risk of not completing their studies and implementing support mechanisms that will improve the likelihood of the student achieving his or her programme of study. Schools, colleges and universities have ample datasets that routinely collect data such as attendance, punctuality to classes, absenteeism, which classes are being missed, grades, qualifications and grades achieved prior to joining the course, whether or not the student has been submitting coursework on time, the number of coursework referrals, whether or not the student is falling below expected outcome levels, the pattern of behaviour on the institution's virtual learning environment or learning platform, the frequency and type of interactions with course teams or with support teams via the home page or face-to-face and so on and so forth.

The Student Service Centre

One of my favourite scenarios that describes the use of analytics plays out as follows. A prospective student telephones a college and is prompted for her name, post code and the first line of her address. The individual taking the call checks her details on the computer terminal and brings up the caller's profile. The employee at that College is presented with a complete history of the caller; her contact details, education history, her employment details, a record of previous study and qualifications achieved at the college, her long term career goals and so on. The caller asks if she is qualified to apply for a new course that supports her career plans. The employee already has the list of courses that the caller is qualified to take on the screen. After confirming and asking for some additional information the caller is in a position to be enrolled on the course. The employee provides the caller with options about how to pay for the course. The caller wishes to set up a monthly direct debit to pay for the course. The employee already has the caller's banking details on her screen and a direct debit order is agreed and arranged in that instance. The caller receives confirmation of her new course, along with billing and start of course information via SMS. An email later confirms her username and password for the college's student portal. When she logs on she sees her course listed on her home page and she can access a detailed payment schedule for her new course. The duration of the call is only 5 minutes in length.

The college has made the whole process for the prospective student as smooth and effortless as possible. The college's core operations, its internal processes and how it captures, analyses and distributes data and information are all prerequisites to delivering an exemplary customer experience. In this scenario the college's ability to utilise analytics, business intelligence tools and other systems such as finance, e-commerce and its learner management system means that it can deliver customer service in a manner that is efficient, seamless and effortless for the employee and more importantly for the prospective student. I must also mention that the prospective student could have achieved the same outcome if she had accessed the college via her smartphone's college app or via the college's website. The ability for college's to deliver exemplary customer service via multiple channels is a must because the college's customers expect that they can enquire, apply and pay for courses on their smartphones, tablet devices, laptops and desktop computers.

In the above scenario the algorithms used by the college identify suggested course(s) that the prospective student is eligible to undertake. These suggestions are based on a number of variables such as the caller's prior qualifications, the last course undertaken, the caller's long term career goals, work experience, whether or not the caller wishes to opt for a part-time or a full-time course at the college, the probability of the caller opting for a specific course and more. If the caller is a current student at the college and wishing to progress onto further studies at the college, the probability of selecting the likeliest course(s) rises because additional data or variables can be taken into account by the algorithms that are used by the college. These include the browsing history of the applicant on the student home page; the information, advice and guidance or progression articles read by the student on the student home page; the interactions between the student and the employees at the college; the list of college courses that the student has been searching for; her recent search history on the student home page; even the use of keywords that have been used by the student when messaging her tutor or the careers team at the college.

Performance Management

Leadership teams in schools, colleges and universities are increasingly using data and analytics to support performance management in the workplace. Performance management is broad since it includes how the institution performed financially; the number of students enrolled, retention rates by course-department and institution; the value added score for specific students, courses and departments; the percentage of students progressing onto further study, training or employment; student attendance by student, course, department or institution and many more.

The use of open system architecture enables individuals to view data and information from across their school, college or university. Managers can compare the performance of different courses, teams and departments and take appropriate action to spread good practice or to take action in areas that are under performing.

One of the scenarios could play out as follows. A departmental manager has access to a dashboard on her desktop monitor. She wishes to review the performance of two of her teams. The key performance indicators for Team A show a growth in student numbers, improved student retention, a rise in success rates on their courses and the team has achieved a growth in revenue from the courses it sells. Team B is not doing so well. The team has seen a fall in student numbers, the student retention rate continues to deteriorate, predicted success rates are lower than last year and revenues have subsequently fallen.

The manager has access to further information to support her overall assessment of both teams. She is able to review the record of lesson observations for each teacher across both teams. She notices that the records for Team B indicate room for further improvement in the quality of teaching, learning and assessment. The records indicate that one of the teachers is in a middle of a formal performance and capability procedure. The manager also sees that the student satisfaction survey for the courses that are delivered by Team B show poor feedback on the team's use of the college's virtual learning environment. The records show that the virtual learning environment is not used as extensively by the teachers in Team B and students are not engaging with the college's online platform. The manager also notes that Team B's students are not progressing onto further studies and the proportion of students applying for a university course has fallen from the previous year.

The use of analytics and business intelligence tools in this context allows managers to identify areas for improvement and it allows managers to devise intervention strategies that are well informed. In addition, analytics and business intelligence tools empower teachers to monitor key performance indicators so that they can take timely and informed actions. In this scenario the manager utilises service oriented architecture to access datasets and information to support the institution's performance management processes.

External Facing Website

A college's or university's website is a unique platform from which to gather, collate and analyse data. Institutions that use analytics effectively will use this data to analyse what keyword searches are trending on their websites. For instance, are prospective students searching for accounting courses, computing courses, are they searching for a combination of courses or are they searching for courses that the institution is not providing yet? If the organisation's operational processes mean that it can take a year or more to validate a new course the college or university will not be flexible, agile or adaptable enough to meet the demand for new courses from prospective students.

The use of analytics, business intelligence and data visualisation tools need to be utilised in a manner that 'can make insights easier to understand and to act on at every point in the organisation, and at every skill level. They transform numbers into information and insights that can be readily put to use instead of relying on further interpretation or leaving them to languish due to uncertainty about how to act'. [4] The need to present information in an accessible manner is vital if institutions are to secure traction and greater interaction amongst teachers, students and support teams.

Informing Teaching and Instructional Design

One of the major challenges facing teachers is their ability and capacity to deliver differentiated instruction and assessment. If we review a definition of differentiated learning I have to conclude that virtual learning environments has so far failed in enabling teachers to deliver differentiated instruction, learning and assessment.

'Differentiated instruction and assessment (also known as differentiated learning or, in education, simply, differentiation) is a framework or philosophy for effective teaching that involves providing different students with different avenues to learning (often in the same classroom) in terms of: acquiring content; processing, constructing, or making sense of ideas; and developing teaching materials and assessment measures so that all students within a classroom can learn effectively, regardless of differences in ability.' [5]

For instance teachers typically post an homogeneous set of learning and assessment materials for a specific course on their virtual learning environment. All students on a given course, regardless of ability and personal learning styles, receive the same content when they log onto their institution's virtual learning environment. All students on the course are asked to complete the same assessment activities and they are all asked to submit their coursework on the same date and time.

One of my favourite scenarios to describe the use of learning analytics to support differentiated instruction, learning and assessment plays out as follows: Students at a college have noticed that the content and assessment activities on the college's virtual learning environment responds and adapts to each student on the course. They have noticed that all of the students on the course start off with the same initial course content, but once the unit gets underway, the students who are confident with the content receive extension activities whilst students who are struggling with the content are presented with content and assessment activities that are more gradual. Students also note that the instructional material that they are each presented with on their virtual learning environment also differs. In this instance, the coupling of an adaptive learning environment with learning analytics allows the college to deliver differentiated content and assessment activities to each student. Students welcome the ability to submit coursework at a date and time that suits them. For instance, a student wishing to submit coursework early so that he can progress onto the next element of the course; or a student who requires additional time to complete a module on her course both can do so. The use of analytics enables teachers and student support teams to confidently track, monitor and support this group of students as they progress through their studies.

At a structural and institutional level, the education sector has some way to go before all students are presented with a truly differentiated learning programme; one that enables them to start and complete a course at a date and time that suits them; one that enables them to undertake formal assessment and examinations according to their timetable rather than the institutions; and institutions (particularly schools) who encourage students to undertake courses according to their abilities and interests regardless of their age.

Value Propositions

The success or failure of a learning analytics project in a school, college or university will be dependent on the value that students and employees derive from the learning analytics service. The value proposition is also dependent on the student's knowledge and awareness about the data that is collected on them and how the data is put to use to support and enhance the services that they access from the school, college or university.

The use of analytics can benefit students in multiple ways. Here are a few examples. Firstly, students gain access to information, advice and guidance that is tailoured to meet their needs and requirements. For instance, a student who is wishing to pursue an undergraduate course in dentistry will see information pertaining to that course or links to institutions who offer that course on his or her home page. Employees who work in the school or college's careers team are aware of the student's desire to pursue an undergraduate dentistry course and can offer suitable information to support the student's application to university.

Secondly, a student is struggling to secure the required grades to apply for a university course. Colleagues at the college are made aware of the situation via the weekly student at risk register that is automatically generated by the pool of algorithms that the college has authored. The student's profile indicates that he needs to raise the grades on two of his units to meet the entry qualifications for the university course. Colleagues design a support plan for the student and this is to be discussed at his next one-to-one meeting with his course tutor. The student receives a notification on his smartphone regarding the meeting.

And thirdly, services should be device agnostic; thereby ensuring that students experience a rich, vibrant and tailoured service regardless of the device being used. Today, students expect that their institution's home page responds not only to the device that they are using but also to multiple contexts such as their location on the campus, the time of day-week-term-year, what they have already read on the site and the numerous events that are happening in real time around the campus. If services are used in this manner schools, colleges and universities can provide a meaningful and engaging experience to every student, in every moment and on every device. [6]

The use of learning analytics means that colleagues have an opportunity to deliver a comprehensive and more cohesive set of services to each student at the school, college or university. The old silos have disappeared and colleagues from various teams across the institution now work more closely to support the student body. Teachers are able to access a dashboard or profile on each student. They are able to identify students at risk. They are able to devise intervention and support strategies that are well informed and which are shared and supported by colleagues and teams from across the institution. Managers are able to identify strategies to improve performance on specific courses. The use of a analytics informs the short, medium and long term actions, responses and strategies of the school, college or university.

Code of Practice

Schools, colleges and universities need to be completely transparent and open with their student body and their wider stakeholder group about what data they collect; how its stored, protected, distributed and archived; how and why the data is analysed and the uses the data is put to.

In June 2015, Jisc produced a Code of Practice on Learning Analytics. You may also wish to review an Educause (April 2014) article on the ethics of learning analytics by colleagues at Purdue University. The reading lists that accompany both of these articles are particularly useful if you wish to explore the ethics of learning analytics in greater detail. If you are after an example of a code of practice on learning analytics I recommend looking at the Open University's Ethical use of Student Data for Learning Analytics Policy. It's a great starting point if you are planning to write a similar policy document for your school, college or university.

Institutions who adopt the use of analytics will need to address various challenges. These include the use of machine learning to support institutions to identify correlations, patterns and trends and to make predictions and decisions based on these inputs. The use of machine learning is particularly useful when an institution has thousands of students who collectively fulfill a few hundred thousand transactions each day on the institution's student home page or via its apps. The volume of transactions that are carried out and the variety of services that are accessed by the student body each day are so large that they are impossible to manage without the use of machine learning and its suite of algorithms; particularly when each student receives an increasingly personalised service from the college or university.

The use of machine learning will translate itself into a number of forms which may come to challenge an institution's code of practice on learning analytics. For example, the myriad of articles and pieces of information that make up an institution's information, advice and guidance (IAG) database are presented to students in a structured and contextualised manner only through the aid of machine learning. The institution's algorithms determine which article or piece of information is presented to a student via his or her home page or via an app on a smartphone. One of the scenarios that could play out is described as follows: A teacher who is presented with a group of students who are sat at their laptops will see that each student is presented with a segmented, differentiated and personalised student home page that is unique to each student. The teacher will have some knowledge but will not possess the total sum of knowledge behind the reasons for presenting specific services and information to each of these students. In fact, the colleagues who authored each article or who entered information into the IAG database may not be able to fully describe how each student home page has been compiled. Is such a scenario acceptable to the institution's careers team? Is it acceptable to the teacher who does not know what is being presented to each student? Is it acceptable to the student body who appear to be presented with services and information that are accurate, responsive and contextualized to fulfill their needs and requirements?

In another example, an institution's use of analytics, machine learning and its suite of algorithms impacts directly on the instructional and assessment materials that are presented to each student via the institution's virtual learning environment. For instance, a school or a college has a large group of students who are enrolled on a mathematics course. This course has a syllabus that is common across all the schools and colleges in the country. Teachers on the course have access to an abundance of scorm compliant materials to support the delivery of the maths syllabus. These materials have been produced by numerous educational publishers who service the course nationwide. The use of learning analytics at the institution has enabled it to transform its traditional virtual learning environment into an adaptive learning environment which means that each maths student is presented with instructional and assessment materials that adapt to their learning needs. Once again we are presented with numerous questions. Are teachers happy to accept the use of analytics in this manner? Are they happy to have an algorithm determine the instructional and assessment materials that are presented to a student? Are students happy to be instructed and assessed in this manner? What is the role of the teacher in this scenario?

Policy Implications of Analytics

The use of analytics will invariably inform and shape educational policy; especially with regard to funding. The education sector could see the introduction of a performance based funding system; were schools, colleges and universities get funding that is proportional and directly related to the achievement and success rates achieved at the institution. Institutional funding may be based partly on the actual destination of each student once he or she has completed a given course.

Funding agencies may incentivise or make it mandatory for schools, colleges and universities to deliver wholly online or part online courses. The use of learning analytics will underpin this development and reduce the barriers of entry to this area of work.

Conclusion

On a final note one must remember that a student stands in front of all these datasets. He or she arrives at the school, college or university with a history that is unique to them; a personality that is unique to them; with needs, ambitions and aspirations that are also unique to that individual student. Similarly teachers and support teams cannot be placed into a homogeneous group who act in an assumed and predictable manner. Hence, institutions need to exercise a degree of flexibility that allows individuals to act on standardised reports but also enables them to act in a manner that better suits their unique circumstances and requirements.

Footnotes:

  1. What is Analytics? (Educause YouTube Channel)
  2. Analytics: Making the Case (Educause)
  3. Learning Analytics at the Open University (KMi, The Open University)
  4. Analytics: The new path to value (IBM, 2010)
  5. Wiki definition of differentiation
  6. Segmented services (Aftab Hussain, January 2015)