Making the invisible, visible: An engagement score for virtual learning platforms

Summary

Switching from traditional classroom and face to face instructor training to computer-based training in a virtual classroom makes the learning experience entirely different for learners. While passive listening and notes taking are expected in a traditional classroom, learners with a “traditional” mindset find it difficult to adapt to an online learning environment that lacks the personal touch offered by peers or instructors that most learners take comfort in.

An Engagement Score measures and tracks a user's engagement on the iGOT Karmayogi platform. It recognises the dynamic nature of online interactions by categorising a user’s behaviour into three key relationships - relationship with the self, relationship with others, and relationship with the content. The score is a reflection of various activities located within each relationship. These activities are carried out by the user and measured through pre-determined indicators. The work on Engagement scores draws on previous academic research but distinguishes itself by focussing on adult learners. It provides visibility to their online interactions and thus, acknowledges learning dynamics outside the classroom and provides the foundation for future work on user engagement.

Rationale

In a traditional classroom-based setting, teacher-student interactions are often at the core of measuring student engagement. For example, externally visible indicators like attendance and a student’s willingness to participate and interact in the classroom are often rendered invisible in an online setting. Nevertheless, it remains important to unpack and identify such indicators on VLPs for two reasons: 

1. Differentiated needs for adult learners: VLPs include a significant population of adult learners, many of whom have different needs, expectations, and limitations with regard to why, how, and what they want to learn.
2. Platform acceptability: It is useful to track user behaviours encompassed within digital learning environments because they measure platform acceptability by helping understand, predict, and boost engagement.

Solution

The Engagement Score is a comprehensive solution for unpacking low engagement amongst adult learners. It measures how well users are connected to the iGOT Karmayogi platform by identifying and tracking behaviours on VLPs through three key relationships – relationship with the self, others, and the content. The final score is based on an algorithm that prioritises high and low engagement behaviour to ensure each user has a unique score: the higher the score, the more engaged a user is. However, to account for the dynamic nature of engagement and interactions, the scores are also used to classify users into three categories - beginning, intermediate, or advanced i.e. an advanced user who has not engaged with the platform over a long period of time can see a drop in their category to an intermediate, and subsequently a beginner user. On the iGOT Karmayogi platform, the Engagement Score has four key uses: (i) track user behaviours, (ii) compare departments and ministries, (iii) make changes in real-time, and (iv) guide content providers towards the best-designed courses. In essence, by understanding, predicting, and boosting user engagement through tangible yet simple components on a VLP, the Engagement score makes the invisible, visible.