Club Admiralty

v7.2 - moving along, a point increase at a time

Multilitteratus Incognitus

Pondering what to learn next 🤔

Course beta testing...


This past weekend a story came across my slashdot feed titled Software Goes Through Beta Testing. Should Online College Courses? I don't often see educational news on slashdot so it piqued my interest. Slashdot links to an EdSurge article where Coursera courses are described as going through beta testing by volunteers (unpaid labor...)

The beta tests cover things such as:

... catching mistakes in quizzes and pointing out befuddling bits of video lectures, which can then be clarified before professors release the course to students.

Fair enough, these are things that we tend to catch in developing our own (traditional) online courses as well, and that we fix or update in continuous offering cycles.   The immediate comparison, quite explicitly, in this edsurge article is the comparison of xMOOCs to traditional online courses.  The article mentions rubrics like Quality Matters and SUNY's open access OSCQR ("oscar") rubric for online 'quality'. One SUNY college is reportedly paying external people $150 per course for such reviews of their online courses, and the overall question seems to be: how do we get people to do this beta test their online courses?

This article did have me getting a bit of a Janeway facepalm, when I read it (and when I read associated comments). The first reason I had a negative reaction to this article was that it assumes that such checks don't happen.   At the instructional design level there are (well, there are supposed to be) checks and balances for this type of testing. If an instructional designer is helping you design your course, you should be getting critical feedback as a faculty member on this course.  In academic departments where only designers do the design and development (in consultation with the faculty member as the expert) then the entire process is run by IDs who should see to this testing and control. Even when faculty work on their own (without instructional designers), which happens to often be the case in face-to-face courses, there are checks and balances there.  There are touch-points throughout the semester and at the end where you get feedback from your students and you can update materials and the course as needed. So, I don't buy this notion that courses aren't 'tested'.†

Furthermore, a senior instructional designer at SUNY is cited as saying that one of the challenges "has been figuring out incentives for professors or instructional designers to conduct the quality checks," but at the same time is quoted as saying “on most campuses, instructional designers have their hands full and don’t have time to review the courses before they go live.”  You can't say (insinuate) that you are trying to coax someone to do a specific task, and then say that these individuals don't have enough time on their hands to do the task you are trying to coax them to do. When will they accomplish it?  Maybe the solution is to hire more instructional designers? Maybe look at the tenure and promotion processes for your institutions and see what can be done there to encourage better review/testing/development cycles for faculty who teach. Maybe hire designers who are also subject matter experts to work with those departments.‡

Another problem I have with this analogy on beta testing is that taught courses (not self-paced courses, which is what xMOOCs have become) have the benefit of a faculty member actually teaching the course, not just creating course packet material. Even multimodal course materials such as videos, podcasts, and animations, are in the end, a self-paced course packet if there isn't an actual person there tutoring or helping to guide you through that journey.   When you have an actual human being teaching/instructing/facilitating/mentoring the course and the students in the course there is a certain degree of flexibility.  You do want to test somewhat, but there is a lot of just-in-time fixes (or hot-fixes) as issues crop up.  In a self-paced course you do want to test the heck out of the course to make sure that self-paced learners aren't stuck (especially when there is no other help!), but in a taught course, extensive testing is almost a waste of limited resources.  The reason for this is that live courses (unlike self-paced courses and xMOOCs) aren meant to be kept up to date and to evolve as new knowledge comes into the field (I deal mostly with graduate online courses),  Hence spending a lot of time and money testing courses that will have some component of the course change within the next 12-18 months is not a wise way to use a finite set of sources.

At the end of the day, I think it's important to critically query our underlying assumptions.  When MOOCs were the new and shiny thing they were often (and wrongly) compared with traditional courses - they are not, and they don't have the same functional requirements.  Now that MOOCs are 'innovating' in other areas, we want to make sure that these innovations are found elsewhere as well, but we don't see a stop to query if the functional requirements and the environment are the same.   Maybe for a 100 level (intro course) that doesn't change often, and that is taken by several hundred students per year (if not per semester) you DO spend the time to exhaustively test and redesign (and maybe those beta testers get 3-credits of their college studies for free!), but for some courses that have the potential change often and have fewer students, this is overkill.  At the end, for me, it comes down to local knowledge, and prioritizing of limited resources.  Instructional Designers are a key element to this and it's important that organizations utilize their skills effectively for the improvement of the organization as a whole.

Your thoughts?




NOTES:
† Yes, OK, there are faculty out there have have taught the same thing for the past 10 years without any change, even the same typos in their lecture notes! I hope that these folks are the exception in academia and not the norm.

‡ The comparison here is to the librarian world where you have generalist librarians, and librarians who also have subject matter expertise in the discipline that they are librarians in. Why not do this for instructional designers?
 Comments
Stacks Image 20

Archive

 Jan 2023 (6)
 Dec 2022 (1)
 Oct 2022 (1)
 Sep 2022 (2)
 Aug 2022 (3)
 Jul 2022 (1)
 Jun 2022 (1)
 May 2022 (2)
 Mar 2022 (2)
 Dec 2021 (2)
 Oct 2021 (1)
 Sep 2021 (1)
 Aug 2021 (2)
 Jul 2021 (1)
 Jun 2021 (1)
 Nov 2020 (1)
 Oct 2020 (1)
 Sep 2020 (1)
 Jun 2020 (2)
 May 2020 (2)
 Mar 2020 (1)
 Jan 2020 (3)
 Nov 2019 (2)
 Sep 2019 (1)
 Aug 2019 (1)
 Jun 2019 (1)
 May 2019 (1)
 Apr 2019 (1)
 Jan 2019 (5)
 Dec 2018 (1)
 Nov 2018 (2)
 Oct 2018 (2)
 Jul 2018 (1)
 May 2018 (1)
 Apr 2018 (2)
 Mar 2018 (2)
 Feb 2018 (1)
 Jan 2018 (1)
 Dec 2017 (2)
 Nov 2017 (1)
 Oct 2017 (2)
 Sep 2017 (2)
 Aug 2017 (2)
 Jul 2017 (4)
 Jun 2017 (7)
 May 2017 (3)
 Mar 2017 (4)
 Feb 2017 (5)
 Jan 2017 (5)
 Dec 2016 (9)
 Nov 2016 (1)
 Oct 2016 (6)
 Sep 2016 (4)
 Aug 2016 (7)
 Jul 2016 (8)
 Jun 2016 (9)
 May 2016 (10)
 Apr 2016 (12)
 Mar 2016 (13)
 Feb 2016 (7)
 Jan 2016 (11)
 Dec 2015 (10)
 Nov 2015 (7)
 Oct 2015 (5)
 Sep 2015 (8)
 Aug 2015 (9)
 Jul 2015 (7)
 Jun 2015 (7)
 May 2015 (15)
 Apr 2015 (2)
 Mar 2015 (10)
 Feb 2015 (4)
 Jan 2015 (7)
 Dec 2014 (5)
 Nov 2014 (13)
 Oct 2014 (10)
 Sep 2014 (8)
 Aug 2014 (8)
 Jul 2014 (5)
 Jun 2014 (5)
 May 2014 (3)
 Apr 2014 (4)
 Mar 2014 (8)
 Feb 2014 (10)
 Jan 2014 (10)
 Dec 2013 (4)
 Nov 2013 (8)
 Oct 2013 (6)
 Sep 2013 (10)
 Aug 2013 (6)
 Jul 2013 (4)
 Jun 2013 (3)
 May 2013 (2)
 Apr 2013 (8)
 Mar 2013 (4)
 Feb 2013 (10)
 Jan 2013 (11)
 Dec 2012 (3)
 Nov 2012 (8)
 Oct 2012 (17)
 Sep 2012 (15)
 Aug 2012 (16)
 Jul 2012 (19)
 Jun 2012 (12)
 May 2012 (12)
 Apr 2012 (12)
 Mar 2012 (12)
 Feb 2012 (13)
 Jan 2012 (14)
 Dec 2011 (19)
 Nov 2011 (21)
 Oct 2011 (31)
 Sep 2011 (12)
 Aug 2011 (8)
 Jul 2011 (7)
 Jun 2011 (3)
 May 2011 (2)
 Apr 2011 (8)
 Mar 2011 (5)
 Feb 2011 (6)
 Jan 2011 (6)
 Dec 2010 (3)
 Nov 2010 (2)
 Oct 2010 (2)
 Sep 2010 (4)
 Aug 2010 (9)
 Jul 2010 (8)
 Jun 2010 (5)
 May 2010 (4)
 Apr 2010 (2)
 Mar 2010 (3)
 Feb 2010 (7)
 Jan 2010 (9)
 Dec 2009 (5)
 Nov 2009 (9)
 Oct 2009 (13)
 Sep 2009 (13)
 Aug 2009 (13)
 Jul 2009 (13)
 Jun 2009 (15)
 May 2009 (15)
 Apr 2009 (14)
 Mar 2009 (13)
 Feb 2009 (10)
 Jan 2009 (12)
 Dec 2008 (6)
 Nov 2008 (8)
 Oct 2008 (2)
 Jul 2008 (1)
 Jun 2008 (6)
 May 2008 (1)
Stacks Image 18