Club Admiralty

v7.0 - moving along, a point increase at a time

Multilitteratus Incognitus

Traversing the path of the doctoral degree

MOOC Completion...according to whom?


The other day I had an interesting (but brief exchange) with Kelvin Bentley on twitter about MOOC completion.  This isn't really a topic that I come back to often, given that completion-rates for MOOCs, as a topic, seems to have kind of died down, but it is fun to come back to it. To my knowledge, no one has come up with some sort of taxonomy of the different degrees of completion of a MOOC†.

But let me rewind for a second.  How did we get to the topic of MOOC completion?  Well, I've been attempting to make my extended CV more accessible (to me).  In the past, I used a WYSIWYG HTML publishing platform to manage my extended CV‡.  The idea was that I could easily export it and just push it on the web.  In practice, I never did this, and when I changed computers it became a hassle to maintain. So, I moved everything over to google docs for cleanup (and easier updates).  In cleaning up my CV sections (I am not done, btw!), I did make a startling self-discovery. In the time-period 2013-2016, I binged on a lot of xMOOCs!😅  The most notable platforms were Coursera, Edx, Udacity, but there were others such as the now-defunct Janux (Oklahoma University) and Open2Study (Australia Open University), as well as overseas platforms like MiriadaX and FutureLearn.  In the time period 2011-2012 I didn't have a lot of MOOCs, mostly because during this period a lot were cMOOCs and xMOOCs hadn't really spread like wildfire.

This realization now begs the question: "How many did you complete?" (and you guessed it, Kelvin asked it...).  My answer comes in the form of a question "based on whose metrics and measures?".  When you sign up for a paid course (e.g., professional development seminar, college course, certification prep course, etc.) I think that there is an unspoken assumption that the goals of the course mirror, to a greater or lesser extent, the goals of the learner♠.  Can this assumption be something that transfers over into the world of a free MOOC?  I personally don't think so.  I've long said that the course completion metric (as measured by completing all assignments with a passing grade) is a poor metric.  One very obvious reason to me was that people simply window-shop; and since there is no disincentive to unenroll, people don't take that extra step to leave the course formally, as they would with a paid course where they could receive a refund. I've been saying this since xMOOC completion rates were touted as an issue, but few people listened. Luckily it seems that people are changing their minds about that (or just don't care 😜). I guess George Siemen's advice to Dave Cormier holds true for my own rantings and ravings: publish those thoughts in a peer-reviewed journal or they don't exist 🤪 (paraphrased from a recent podcast interview with Dave).

Assuming that we exclude window-shoppers from our list of completion categories♣, what remains?  Well, instead of thinking of distinct categories (which might give us a giant list), let's think of completion in terms of whose perspective we are examining.  On the one extreme, we have the learner's perspective.  The extreme learner's perspective is characterized by total control by the learner as to what the goals are. In this perspective, the learner can be in a course and complete a certain percentage of what's there and still consider the course as done. Why?  The learner might have prior knowledge, and what they are looking for is to supplement what they already know without going through the hoops of any or all assessments in the course. They've probably evaluated the materials in the course, but if they already know something, why spent a lot of time on something already known? Or, an item that should be done to obtain 100% completion is only available in the paid version (some FutureLearn courses are like this), and are inaccessible to learners on the free tier.

On the other extreme, we have the perspective of the course designer. This is the perspective that most research studies on completion seem to adopt. The course designer is working with an abstracted learner population, with abstracted goals.  The outcomes of the course might be based on actual research into a learner group, they might be based on the intuition of the course designer, or they might just be whatever the course designer has an interest in preparing (sort of like the Chef's soup of the day, it's there, you can have it, but it doesn't mean that this is what you came into the restaurant for).  In a traditional course (the ones you pay and get credentialed for) it makes sense that a learner could simply go along for the (educational) ride because they are paying and (presumably) they've done some research about the course, and it meets their goals. In a free offering, why would a learner conform to the designer's assumptions as to what the learner needs? Especially when a free offering can (and probably does) gather the interest of not just aspiring professionals, but people in the profession (who presumably have some additional or previous knowledge), as well as hobbyists who are free-range learning?

Given those two extremes of the spectrum, I would say that there is a mid-point.  The mid-point is where the power dynamic between the learner and the designer is at equilibrium.  The educational goals (and what hoops the learner is willing to jump through) 100% coincide with what the designer designed. Both parties are entering the teaching/learning relationship on equal footing.  If you lean over a little to one side (learner side), the designer might consider the course incomplete, and if you lean over to the other side (the designer side) the learner might start to feel a bit annoyed because they have to jump through hoops that they feel are not worth their while. Some might begrudgingly do it, others not, it really depends on what the carrot is at the end of that hoop.  For me, a free certificate or badge did the trick most times. The threat of being marked as a non-completer (or more recently the threat of losing access to the course altogether 😭) however does not motivate me to "complete" the course on the designer's terms.

That said, what about my experience?  Well... my own behaviors have changed a bit over the years.  When xMOOCs first hit the scene I was willing to go through and jump through all the hoops for the official completion mark.  I did get a certificate at the end; and even though it didn't really carry much (or any?) weight, it was a nice memento of the learning experience. Badges were custom made (if there were badges), and the certificates were each unique to the MOOC that offered them.   Back in the day, Coursera had certificates of completion (you earned the minimum grade to pass), and certificates of completion with distinction (you basically earned an "A").  It was motivating to strive for that, even though it didn't mean much. It was also encouraging when MOOC content was available beyond the course's official end, so you could go back and review, re-experience, or even start a bit late.  As we know, things in the MOOC world changed over the years.  Certificates became something you had to pay for.  Sometimes even the assessment itself was something you had to pay for - you can see it in the MOOC but you can't access it.  Peer essay grading on coursera wasn't something that I found particularly useful, but I was willing to jump through the hoops if it meant a free moment at the end of the course (achievement, badge, certificate, whatever). Once things started having definitive start- and end- dates♪ , and content disappeared after that when certificates (which still we're worth much to the broader world) started costing money, the jumping through the same silly hoops (AES, CPR, MCEs, etc.) it just didn't feel worthwhile to go above my own learning goals and jump through someone else's hoops.

So, did I complete all those MOOCs?  Yup, but based on my own metrics, needs, and values.

What are your thoughts on MOOC completion?  Do you have a different scale? Or perhaps defined categories?

† There may be some article there somewhere that I've missed, but in my mission to read all of the MOOC literature that I can get access to, I haven't found anything.

‡ What's an extended CV?  It's something that contains everything and the kitchen sink.  That workshop I did back in 1999 for that defunct software?  Yup, that's there...because I did it, and I need a way to remember it. It's not necessarily about the individual workshops, but about the documenting of the learning journey.  The regular CV is somewhat cleaner.

♠ Maybe this assumption on my part is wrong, but I can't really picture very many reasons (other than "secret shopper") that someone would pay money to sign-up for a course that doesn't meet their goals.

♣ Window-shoppers I define as people who enroll to have a look around, but either have no specific educational goals they are trying to meet (e.g., lookie-loos), or have goals to meet, but they deem the MOOC to not meet them (e.g., "thanks, but not what I am looking for"). Either way, they don't learn anything from the content or peers in the MOOC, but at the same time, they don't unenroll since there is no incentive to do so (e.g., a refund of the course course).

♪ e.g., module tests deactivating after the week was over and you couldn't take them - AT ALL if you missed that window

xMOOCs as on-demand documentary viewing

For the past semester I've mostly ignored synchronous learning on coursera.  Instead of consuming materials as they are released, I log in once a week, download the videos for the course, and I keep them in my video library.  If there are textual materials available as well, I donwload those, but I tend to focus more on video materials. When inspiration (or curiosity) strikes, I dive into the specific course of interest and have a video play.  At the moment I tend to play lectures in chronological order, the order that they were listed in - in the course I got them from.

So, why not use coursera as "intended"?  Well, the predominant reason is the lack of time. These past two semesters have been quite busy for me and I don't have the time or inclination to do things as they are released by content providers (yes, I know - the noun used was quite deliberate).  There are also a lot of interesting courses being offered, not just on coursera, but also other MOOC providers. This means that my time, little as it was before, now becomes less when you think of the plethora of stuff that's out there.  My initial tactic was to sign up for many xMOOCs and view things later, but as I discovered, some MOOCs, once the course if "over", have adopted another annoying approach of traditional education - making content in-accessible. So, when I sign up for MOOCs, I download everything I can.  I probably contribute to the great number of 'non completers', but 'completing' a MOOC means many things to many people.  I'll complete the MOOC on my own terms ;-)

I know that some MOOCs are 'on demand' on coursera, which means that you can access them anytime, however those don't easily allow for video downloads. At least, I have not found a way to download for offline viewing thus far, so I usually don't sign up for those.

So, in thinking about my own, recent, use of xMOOC materials, I was thinking of public television, things like the national geographic television channel, documentaries. I was also thinking about  on-demand viewing venues like amazon prime, hulu, and netflix (and to a much lesser degree things like torrenting).  I think over the past 10 years (probably a little more) we've seen a huge change in how entertainment media is accessed and consumed.  Initially we had time-shifiting with devices like TiVo, so you didn't have to watch something on the day that it was on - you could watch it at another time.  Yes, VHS tapes also accomplished the same thing but I had a hard time programming my VCR to do this, whereas TiVo seems to have made it user friendly (even though I never owned a TiVo).

We also experienced place-shifting with things like Sling players which allows you to view things that are outside of your georgraphic region. So, if you are an mid-westerner living abroad, you can watch your local football team abroad if you want. Provided that you have mobile bandwidth, you could watch your local TV locally on your mobile as well. Finally, we have providers such as Hulu, Netflix, Amazon, Apple (and the list goes on) that allow you to stream materials to your device.  Some, like Amazon recently, allow you to download a copy for offline use. So, if you are on a plane, or away from the web, you can still watch what you've paid for. So, in the end, you aren't tied to a broadcast model, one where the preferences are set by external interests, but by your time, availability, and mood.

Along with these revolutions in how we consume media, I was thinking how much I liked documentaries as a kid, and I was thinking back to high school where we were required to keep a journal in some science classes (watch a documentary relevant to your science class topic and write about what you learned). And then it hit me.  I am using xMOOC videos in a similar manner as I do television shows these days.  The heuristics of xMOOC videos (at least the social science and history ones I tend to follow) are such that they encourage this type of viewing.  I can just add them to my iPad and I can view them while I am on the train, or while shaving (most of them seem to be talking heads, so no worries about cutting myself), or while gardening (ditto on the talking head).  Learning, then, becomes embedded into other activities.  This is not because the learning isn't engaging - if it were boring I wouldn't be doing it - but rather because the heuristics of materials provided as the bread and butter of the course seem to point to that usage.  If a learner doesn't have the need to engage socially with fellow co-learners, and if the materials don't connect in ways that encourage such social aspects, then nothing is lost. Now...if only those videos were all OERs...

Compare this with cMOOCs. More specifically a cMOOC called creativity for learning in higher ed, The collaboration and discussion component is a large part of the MOOC.  To get the most out of this MOOC you'd have to be engaged in the course, via discussions and exchanges with other co-learners, because that's what the design heuristics dictate. Since I am currently too busy with school and work for this MOOC, I probably won't participate in it as it exists when many are working in that space. I do however think that I will most likely come back in December once the semester is over and I can take a breather.

To wrap up, back to my xMOOC videos as documentary viewing - what do you think?  Should we encourage such asynchronous view of  learning and engaging in the material? What does this do to our current notions of 'completion' and 'engagement'?  How about assessment of learning?   Your thoughts?


Lurk on, dude, lurk on!


The other day, while catching up on my (ever growing) pocket reading list, I came across a post from, friend and fellow MobiMOOC colleague, Inge on MOOCs.  It was a rather on-the-nose post about MOOCs, learning, assessment, and the discourse used in MOOCs about learners. Concurrently I am working with a Rhizo team on a social network analysis post where the topic of 'completion' came up, and we started discussing  the real connotations of completion.  How does one measure 'completion' in a MOOC? Is it a worthwhile metric? and what about engagement?  Finally, to add to this volatile mix of intellectual ideas, I am working on a conferece presentation, with fellow lifelong learner and MOOCer Suzan†.

These raw materials made me think back to the early discussions on MOOCs (before the 'x' ones came out) and discussions about lurkers in MOOCs.  Before the xMOOC came out we didn't seem to frame non-visible members of the community as 'dropouts' but rather as lurkers.  There were probably people who quit the MOOC, as in they came, they saw, it wasn't for them, they left - but we left the door open for them to be lurkers if they wanted to.

Early on I viewed MOOC participation sort of similar to the participation patterns in a community of practice (at least those that I had learned in school) which are visually depicted by the image in this post.  ~90% are lurkers, ~9% contribute, and ~1% contribute a lot‡.  In one of my earlier MOOCs, #change11, I engaged more with the idea of lurkers, and the main thesis I had (at least in retrospect) was that at most they were harmless onlookers, at worst they didn't contribute to the continued well being of the community.  I viewed (and still view) learning as a communal activity, so the more people participate in the network of learning the better the outcome for everyone.  It allows depth  of conversation, different discussions to take place, and diversity of opinion.  When a lot of people lurk, my concern was, that a critical mass for community purposes would not be available so that a experience learning could either not get off the ground, or it would not be possible to sustain it.

Fast forward to 2015.  After more than 100 xMOOCs, cMOOC, pMOOC, rMOOC, αMOOC, βMOOC, γMOOC, and other free online learnign experiences I am not really sure where I stand on the subject of lurkers.  Well, I do, but I am also conflicted.  See, learner choice is one big aspect of learning.  You cannot really force anyone to learn something, or participate in some experience.  This holds true for open learning experiences like MOOCs, and for closed experiences (paid courses, seminars, workshops, etc.). Intrinsic motivation is important in learning, and it's what pulls the learner through times, both easy and difficult.  In this aspect, if what motivates learners is for them to lurk, or just participate in certain weeks or modules, then that is not only perfectly OK, it should be encouraged.

The point of conflict, however, comes in kicstarting and sustaining the learning community. Let's say I am an open learning designer♠ and I have this awesome course I am thinking of designing for a certain demographic.  Sort of like hosting a party I don't want it to fail. I want people to attend, be engaged, and have fun (and learn something in the process).  What can I do to make sure that there is a minimum mass to sustain the course through it's x-week duration?  Do I do anything to recruit and tend to the learning garden? Or do I let it run wild, and if it succeeds - then great, and it fails (like a lame party), then that's OK too?

I guess what I am asking (and proposing a discussio on) what are some #altMetrics for MOOC success other than visible participant engagement, or 'course completion', or any one of the traditional success factors?  By de-coupling attendance from success metrics, I think we can be quite fine with having a ton of lurkers in our MOOC, and still having a MOOC be a success.  Lurkers get what they need, active participants get what they need, course designers get what they need.  It's a win-win.  But - how do you get there?


† You know, when I tell people (who already have a PhD) that I am pursuing my EdD online through Athabasca University I get a bit of a sour face. They can't fathom how you can develop academic relationships that lead to stimulating discussions (and papers) at a distances.  Between my cohort and the people I've met in MOOCs I think I have had more mental stimulation that people in residential programs - just saying ;-)

‡ Wonder if this triangle is a distortion, sort of like Edgar Dale's corrupted cone...

♠ Mark my words, Open Learning Designer will be a job title soon enough if it's not already. Prbably a type of instructional or learning designer ;-)

MOOC Completion rates matter?

A while back I came across a post by Martin Weller titled MOOC Completion rates DO matter. Because my Pocket account was overflowing with some great content (including this one), I thought it was high time that I read this article ;-).  In this short post Martin writes that completion rates do matter in MOOCs, taking the opposite view of some cMOOC folks. 

He goes on to tackle the analogy that MOOCs are like newspapers and that people don't necessarily read all sections, or even visit all sections.  As someone who doesn't really read physical newspapers that often any more (I only bought one last summer so I could get the Asterix comic that came with it), the analogy only gets me so far.

I don't know what sort of analogy is valid for MOOC participant behaviors.  After pondering this a bit, I don't think that there is one analogy that will encompass all behaviors.  At the beginning of the MOOC I think the analogy of window shopping is most appropriate for everything leading up to the conclusion of the first (or maybe even the second) week of the course.  I think that there are many people who are curious about the course and join it, regardless of their intents for the course.  Coursera and EdX have gotten a little smarter these past few months and query their users to see what their intents are for the course right at the beginning of the course.  Are they interested in just browsing the content? browsing the content and taking some quizzes? doing the assignments? going full force into the course?  I think this is smart, because at the onset of the course you have idea of what the participants in the course are intending to do (assuming that they are truthful in their answers).

This only goes so far, however.  Back in the days of Change11 I had pondered about having a nag-system (for lack of a better term) that would keep track of user logons in the system, and track their patterns of participation.  If they weren't active the system would send them a reminder email about the course, and see if they would like to unenroll, so there would be a perpetual Opt-In function for people who are not active. I think at this point I would also augment the system with highlights of upvoted posts and interesting discussions happening so as to entice people to come in and say something (or at least if they don't contribute something, they will be seen by the system as being active). Lurking isn't bad, everyone lurks from time to time, it's not possible to fully engaged all the time, in each course you take.  You do need time to let things steep.  Especially in MOOCs you might have multiple conflicting priorities and need to put the MOOC on the back burner for a few days. That said, the point of the system is to separate those who are passive participants (aka lurkers), from those who are checked out (and not coming back), and from those who are on leave (not active for a week, but intend on coming back).

Then, at the end, it would be useful to have participants self-assess.  Those who took the initial survey should be surveyed again to see if they met their own expectations, or if they did or did not meet their own levels of engagement in the course.  The other points to ponder are theses:
  • Under whose rubric do we measure completion? The course designer certainly has some ideas about what it means to complete a course, however in an open course you don't control for enrollment, so you might have some people who are very under-prepared for the course, and some where the course is too basic for them.
  • Does completion equate with learning?  These two aren't necessarily the same. Going back to the case where the participants already have the knowledge, but don't know it here's something from my own experience:  I was in two courses, one in coursera, and one in edx.  I signed up to learn something new, only to find out at the end of Week 1 that things seemed a bit basic.  I ended up taking the quizzes for these two courses in advance for subsequent weeks (they were available) and I passed with high 90s without viewing any subsequent lectures past week 1.  I completed the courses, but did I learn anything (new)?  The answer is no.
  • Who does completion matter to? I know institutions who subsidize these want to see some ROI, but ultimate, in an open course (where there is no formal grade or credit for that grade), doesn't the participant sit in the drive seat to determine whether they've completed the course or not (to their own satisfaction)?

Do completion rates matter?  I think they do, however in certain specific contexts.  Not all contexts are the same!  What do you think?

comic from:
 Comments (1)

MOOC on vacation: what does "completing a MOOC" mean?

View from Itea, Greece

Some people bring a book on vacation (which I have) and others immerse themselves in the local culture (which I am also doing to some extent), but since I find myself lucky enough to be vacationing somewhere with fast wifi access I decided to continue to MOOC while I am on vacation from the day job. I honestly don't know how well the experiment will go, but I decided to follow three MOOCs. One on FutureLearn, focusing on management (making those connections with my MBA), and two MOOCs on MiriadaX, the Spanish MOOC platform: one focusing on social media and marketing, and another focusing on media studies called "the 3rd golden age of television."

I didn't really think about my MOOCing until I had a back and forth with a friend and colleague on a Facebook about "completing MOOCs" and whoever completes one gets a unicorn ;). I guess the point he was trying to make is that completion is mission impossible. This made me think of my presentation at last January's NERCOMP symposium on online and international education where I spoke about MOOCs, and as part of that presentation I discussed that I had completed so many MOOCs (around 30 at the time) and had dropped out of two. I also went to class central (a MOOC aggregator) to mess around with it and decided to start counting the specific MOOCs I've completed. Class central has a variety of categories for MOOCs such as: interested (in taking the MOOC), completed, partially completed, dropped, audited and taking right now.

Of course, the system relies on the user to self identify which label applies to which MOOC the learner/user has taken (or wants to take). This got me thinking about how people self identify their role, as learners, in MOOCs. I think the "dropped" and "partially completed" are a little problematic because learners may have some issues with identifying that they quit a MOOC. Issues of self-efficacy, morale, and anxiety may come up if people deem that not completing a MOOC is a failure of some sort. So, I won't spend a lot of time discussing that now. Perhaps in a future post.

The big question, for me, is how do we define "completed" or even "audited". In a regular college course the line between completion of a course and an auditor is clearly demarcated: the auditor gets an "aud" on their transcript and the completer gets a grade and some GPA points on their transcript. The amount of effort between the two categories may be the same, but the line of demarcation is the transcript. In a MOOC, I know that there are some MOOCs, like the Rhizo14 MOOC where I've clearly out in much more effort than an xMOOC (let's say something on EdX or coursera), but I count both the Alexander the Great MOOC (which I completed this spring) and rhizo as "completed" regardless of the effort (measured in products and time sunk in the course) put into the MOOC.

Even with xMOOCs there are different levels of engagement. On open2study I attempted to engage in the forums, whereas On coursera I don't. Sometimes it's about the platform, other times about the course design, others about the ration if chaos vs usefulness of the forum. The bottom line is that the forum tool is not used that often (at least by me). So, coming back to our original focus, what does it mean to complete a MOOC? If a participant in a MOOC complete 3/4 of the MOOC and gets what he needs out of it, can this constitute completeness? If a participant comes in and views the videos and does the quizzes, does his constitute completeness? In a traditional classroom we have a notion of what completeness is because it's designed to be facilitated by a faculty member, who grades assignments, which aren't optional. In a MOOC where sign up, participation, and engagement are optional, is the model of instructor-defined completeness rubrics still meaningful? If yes, why? If not, why not? If somewhere in the middle of yes and no, what does your conception of completeness look like?

Leave a comment and chime in :-)



 Comments (1)