Reflections on Learning Analytics

Week 11 builds on the previous week’s theme of big data, providing a spotlight on the use of data specifically for learning. Unsurprisingly, there are many links and reference points from topics throughout the rest of IDEL.

The core readings seem to indicate that the field of learning analytics is still very immature, and when compared with the use of other technologies within education, could be considered to be lagging.

It seems, on the whole, learning analytics operate at surface level at present. Gasevic et al (2015) highlight the complexities around learner agency and pedagogical approaches that can provoke glaring holes in the interpretation of any data. Ultimately in any scenario, educational or not, data needs context to have any meaning (and therefore actionable value), and it seems to be falling short in this area at present.

I enjoyed reading about Purdue University’s Course Signals project in the core readings. The intention behind this project seems to empower the teacher, rather than simply ‘measure’ the student. While the positivity around the results should be taken with a pinch of salt in Clow (2013) (indeed Gasevic et al (2015) does proffer further critique of this), it would seem that involving the teacher in the choice of interactions recognises the absence of relationship and emotion that perhaps these analytics struggle to encompass. However, it does appear that the aggregation and understanding of quantitative data that could bridge this gap is improving (Gasevic et al, 2015).

I particularly liked Clow’s (2013) description of learning analytics as a “‘jackdaw’ field of enquiry, in that it uses a fragmented set of tools and methodologies – it could be seen to be using a rather less cohesive approach that would be required. This is certainly one of Gasevic et al’s (2015) key points – that the field needs to be underpinned with a more robust foundation to really allow it to develop in a useful way.

I wonder if the lack of maturity in this field is an implication of the nature of the field. The learning analytics cycle used by Clow (2012) identified that learning analytics are gathered and used after a learning activity or intervention has taken place. As has become even more apparent to me throughout this course, the pace of technological change is significant and rapid and the impacts on education are quite far-reaching.

If technology and tools are being developed, trialled, integrated, ditched and recycled so rapidly, inevitably it must be a challenge to assess with any rigour. Indeed Gasevic et al (2015) highlight the lack of empirical studies available to attempt to interpret this area. It’s interesting to hear in Clow (2013) that the use of proprietary systems impedes’ this too, with the lack of data available. This is particularly pertinent given their prevalence across the educational sector, and in turn impacts the assessments that can be made across domains and contexts (Gasevic et al, 2015).

A pervading theme across IDEL has been the discourse around the educational ‘voice’ in the development and use of technology, for example, Bayne (2015). Quite rightly this academic world wants to scrutinise, assess and challenge, but it seems the pace of change makes this less and less possible to take place.

For me, the spectre of the private sector is raised in Perrotta & Williamson (2016). It argues that that the use of analytics and interventions are “partially involved in the creation of the realities they claim to measure”. The challenge here is the increasing commercial influence taking place in the field of learning analytics. It cites Pearson as an example, as they have the large-scale distribution of their products to gather learner data, the resources to interpret and mine this, as well as the size to influence wider policy-making. Given the rhizomatic nature of the development of learning analytics, it seems to be that there are many reasons to be fearful of this development, particularly as it looks to be self-perpetuating.

Of course, I’m keen to keep in mind that this is one side of the argument, and I’m sure the likes of Pearson see themselves as helping to push things forward. Certainly, there are areas that the commercial world can help ‘lift the lid’ on learner behaviour, and empower teachers to make interventions – I guess the issue is how much those outside the corporation are at the discussion table. The stark truth is that Pearson’s core responsibility, above all, is to its shareholders, not it’s students.

My own personal experience has been at a ‘macro’ level, or what Clow (2015) refers to as ‘Business Intelligence’. As a commercial training provider, we used learner analytics (at a rather shallow level) to understand learner behaviour, and help us understand the product performance. Given the commercial nature of the people around me, however, there was probably an unhealthy bias or interest in how these can be used to improve commercial metrics. I certainly recognised some of the observations raised by Gasevic et al (2015) around data visualisation, and the pitfalls these can cause.

I think given this week’s reading I’ll certainly be more aware of some of the challenges in this area, particularly around providing metrics without some context. There almost needs to be some ‘priming’ of the viewer before they gain access, just to reduce the risk of mis-interpretation. I think I’ll also be keen to trial the use of analytics data to empower tutors, rather than simply automating prompts which has been the norm in the past. Alongside this, providing students with their own ‘performance’ data would be something I’d be keen to explore.

Last week’s discussions on big data raised concern about the skills needed within institutions to use big data, and I would suggest these are not solely limited to the educational world. The same issues occur in the commercial world, and can oft have quite dramatic implications if not used with care and forethought. It seems like if you are a data analyst with an understanding of educational methodologies you would be able to choose your job!

References:

Data Visualisation

This week in IDEL we communication on the course ventured out of the forums and onto Twitter. As we were investigating the concepts of big data, and its role within education, we were challenged to answer some questions around this on the social media platform using the hashtag #mscidel as the identifier.

We then used a visualisation tool to summarise the week’s activity, and this is the result:

To add to this data, there were 54 nodes, and 151 edges as a result of the week’s conversations.

It seems to me there was a handful of vocal participants, indicated by the increased size of the handle. I don’t think this is a huge surprise, the names that crop up seem to be those that have been more active on Twitter over the last few months. Naturally this does raise a question around the choice of tool, naturally those more comfortable with a tool – or those where the tool is more ingrained in their daily activities – perhaps are more likely to involve themselves in the conversation. This could be as much effect as the cause though, it could be argued those more active on Twitter are more active for a reason.

Although there are 54 nodes, I’m actually a little surprised it’s not gone further than this. The visualisation, to me, shows that the conversation was held within a rather tight group. I think this is interesting for a conversation to occur on an open and public platform not to have more interjections, particularly given the alumni of the MSc programme that is likely to have connections with those leading the conversations.

In terms of an experience, it’s certainly been one of the more intriguing activities.

It felt throughout that there was minimal tutor intervention, and this was something that was actively discussed. There was speculation that this could be by design, simply as a way to avoid colouring the tag output and keeping this ‘organic’. More simply, it was also argued that the conversation was free-flowing anyway, and the tutors’ role was to tee up the conversation and leave it to the students, it’s not as if the programme leaders aren’t busy!

All the activities so far on the course have been private. The blogs have been locked down, and directed conversations have been held within closed environments, for example, Minecraft, skype chats, and the forums. Given we’re now at week 10, this is indicative of our increased confidence in the subject area and lack of nerves about being ‘on show’.

Given Twitter became the central place for conversation, I’ve inevitably found myself making comparisons with the forums, and the relative merits with each. Twitter has some particular strengths of note as a conversational tool.

  • It was easier to keep abreast of the conversations happening, given it’s accessible on our mobile devices more readily, and doesn’t require us to log in. I think with the forums I always set down to ‘do some work’, whereas found myself checking and contributing to the conversations more frequently on Twitter. An interesting point to this is that there is an increased incentive to contribute, as you ultimately know this is going to feed into the visualisation. It could be argued that this links into the notion of badges looked at in week 9. Although this is largely used as a reward for activity or completion, it’s another influence on learning behaviour and motivation.
  • Twitter caps posts to 280 characters (it would have been interesting to do this exercise with the until-recent 140 character limit). I think this encourages brevity and more salient points. Of course, the trade-off here is the depth of response and consideration put into it. There’s probably a place for both.
  • It seemed to me that the conversations on brought in more informal articles and blog postings, whereas the forums seem to be more aligned with journals and papers. Perhaps there’s an aspect of formality here, and how the community is expected to behave in each of the mediums. In the forums, it could be seen as more within the ‘virtual walls’ of the university (week 6 and 7 flags here!) so there’s a perception of what’s expected. Outside of these walls, there are less concerns. Perhaps we feel less observed and on display to the ‘guardians’, even if this is completely false.

On the flip side, there are some considerable strengths to the forums.

  • I felt at times on Twitter that it was difficult to ensure I’d seen all the key conversations. Although we were using an understood hashtag, it’s easy to miss this from your posts, and I find Twitter’s inbuilt search facility problematic at times. With the forums, it’s easy to see all the threads and make sure you’ve not missed an important one.
  • Despite our increased confidence as the course has progressed, everyone may not be in the same position within the group. Ultimately Twitter is open, and this could be a discouragement to some. Given the granular options to be involved in Twitter (e.g. ‘liking’ posts), this may provide lurkers with the opportunity to stay involved without contributing. This doesn’t concern me personally, but sometimes that push can help vocalise your thoughts.

I felt the activity also deepened some of my relationships with the peers, and the increased contact was probably well overdue since the Minecraft interactions. For me, Twitter blurs the line between synchronous and asynchronous activity. There were times when some of the conversations were happening in real time. It was also interesting that in this format the timezone factor did seem to come in. Some of the students are based in North America, so you’d often wake to a flurry of tweets on your timeline – I guess I should count myself lucky that I’m in the ‘right’ timezone for the course. It’d be interesting to hear those views from those who had a different experience as a result of this.

Sandra Flynn’s early blog post about the visualisation raised some interesting thoughts about the nature of data, and almost an inherent implication to start comparing or competing. I’ve struggled to find any research into this area, but it’s an interesting notion – once you can start to measure something, does it change the nature of what is being measured?

The visualisation also became a useful social object once these were produced. However, this didn’t spark off the conversations I was expecting it to. This could be a simple case of it being towards the end of the week, but given one of the activities was to blog about this, I suspect many of us (myself included) are using our observations to add to our blog.

It turned out useful to have done the bulk of the core reading early in Week 10 to give the Twitter conversations more meaning. For me, the more pertinent discussion areas during the week revolved around:

Looking forward to exploring this in more detail next week – learning analytics!

Big Data

As IDEL progressed we seem to have moved away from perhaps some of the more theoretical and human topics, towards a more technical focus. Week 9 was focused on digital badges and blockchaining, and week 10 looks at the concept of ‘big data’ and its implications for education.

Like blockchaining, I’m aware of the concept of big data, but have not got to grips on what this means, or what it can do for us. I think there’s a danger some of these initiatives can be seen as simply buzzwords or part of the zeitgeist, without any longevity to them. But it’s apparent by digging into both blockchaining and big data that these are unlikely to be fleeting developments and are likely to underpin many technical changes over the next few years.

Unsurprisingly, and in line with the rest of the course so far, the focus on the readings has been to contrast the opportunities big data provides with some of the pitfalls. There’s also been a focus on some of the blind spots in this area.

So starting with the positive aspects, with Selwyn (2015) highlighting three areas:

  • Increased ability to use data to measure goals, targets, benchmarks, performance indicators etc
  • An ability to harmonise and standardise across borders, whether these be institutional or geographic
  • To provide a basis for an infrastructure for education to be understood and organised

But naturally, there are some challenges. Eynon (2013) puts the spotlight on three aspects for concern:

  • The ethics behind the sourcing, mining, interpretation and ultimately use of the data
  • The scope of the data, what can be measured as a results, and the questions it can (and can’t) help us answer
  • Inequalities linked to the sourcing and accessing of any data.

The area of ethics is one discussed in detail across several of the recommended readings, as it seems to be a grave cause for concern. A wonderful example is given by Williamson (2015) on use of data provided by Facebook, and the criticism afterward about the permission (or lack thereof) around the use of the data (the defence being that it was already in the ‘public realm’).

To provide an example related to university admissions, at present applications are (largely) based on Academic results at an undergraduate level. However ‘big data’ could provide the ability to forecast degree completion, and perhaps future earning potential, and even link this to social backgrounds and family history. Naturally, on the one hand, this could be empowering – providing institutes with more insight into how to support their students to succeed. The rather dystopian view is that this could prejudice entry requirements, and given some of the metrics being forced upon universities (e.g. ‘satisfaction ratings) and the subsequent ‘slap on the wrists’ as a result of this, it doesn’t seem impossible that big data will be used in this way at some point.

Selwyn (2015) extrapolates these issues further to explore what they may mean for institutes and their students. He argues that this increase in performance metrics may create an “intensification of managerialism within education”, which suggests a move towards a workplace more typical of a commercial organisation.

This commercial influence is unsurprising, given the history of big data. It seems to me that using the processes and themes of big data in education comes with it with ‘baggage’, because of its very design. Because of the commercial background too, it’s important to critique the strength and role of the educational or academic voice within technical developments such as this. This harks back to one of IDEL’s earliest topics, about the role of technology in education, and who is at the table when it comes to the discussions and implementation of this. This seems to be another example of where education could be perceived as a recipient of this technology, rather than helping to shape it.

This is of particular concern when you read about Pearson’s developments in Williamson (2015). Some of the criticism of Pearson could be seen as a reaction against their developments, but when Williamson argues that Pearson could be using big data to create new “models of cognitive development and learner progression”, then this would be a major red flag. Pearson’s main responsibility is to their shareholders, to their students, so it’s important that the progression of technical initiatives like big data is not left to commercial educational companies to drive.

A common issue picked up by Eynon (2013), Selwyn (2015) and WIlliamson (2015), is that of an institution’s capability, or more specifically the personnel within it, to use big data. ‘Use’ in this context, is quite a broad term, from sourcing and mining the data, through to combining with different sources and interpreting it. Williamson argues that there are “several competencies for education data science”, and that there is a significant deficit in the numbers of those equipped with the necessary skills. The skills are a blend of the technical (computational and statistical skills), the educational and an understanding of the ethical and social concerns in this area. As such, Williamson argues that educational data science is very much a field in its own right, rather than an appendage to statistical analysis. Naturally, if this is an area that is significantly under-resourced, then this reduces the impact education can have in shaping big data.

This may also be more difficult to fix than Eynon envisages. The demand for talent – given the nature of the role – is spread across both commercial and educational organisations, meaning commercial companies may be able to outbid educational institutes for their services. It may be one thing to recognise the issue, but fixing it may be increasingly difficult.

I picked up on several themes across the papers that have been discussed earlier in IDEL.

Given the rise of commercial influence in this area (in particular), there seemed to be a ‘call to action’ to the wider educational crowd to become more vocal, and come more centre-stage in these discussions. Selwyn (2013) argues that “the opportunity now exists for educational research to develop nuanced approaches to understanding, and then offering alternatives to, the dominant data conditions that are being established across educational contexts”. This reminded me of Biesta (2013), in his call for teachers to teach, and Bayne (2015) to ensure academia has a role to play in wider technological developments.

Biesta’s references to a neo-liberalistic agenda also pop up in Selwyn – “expanded access to data allows institutions and individuals to operate more efficiently, effectively and equitably”, and Eynon also references themes of efficiency and cost-effectiveness in big data.

Selwyn (2013) also uses the metaphor of water in his discussions around big data. ‘Deluge’, ‘flow’, and ‘flood’ are terms used, and I think this possibly inevitable. The comparisons between data and water are natural – is it aplenty, can travel at speed (rivers) or not (lakes), comes from many different sources and directions, and requires real skill to manage. It’s also a fundamental of life, and you can argue data is the bedrock of economies now (it’s even been termed as more valuable than oil). The dystopian view is that it can also be a dangerous force of huge power, and like recent devastating floods all over the planet, can pose an immediate danger to us through years of mismanagement.

I thought it was interesting that Selwyn (2015) points out that the sociological approach to data is to assume that there are already some inherent issues with it. This admission of lack of neutrality is quite refreshing, and makes a lot of sense. It’s a battle that’s difficult to fight – it’s probably a better use of time to acknowledge this and work out how to deal with it than try and fix at source. The rhizomatic metaphor is also apparent here, in that Selwyn argues that “this approach is careful to acknowledge that data are profoundly shaping of, as well as shaped by, social interests”.

As a final thought, I liked this quote from Eynon (2013) – “We must not get seduced by Big Data”. I think if you were to replace ‘Big Data’ with ‘technology’, you’ve probably got the core theme of IDEL in a nutshell.

References: