Club Admiralty

v7.2 - moving along, a point increase at a time

Multilitteratus Incognitus

Pondering what to learn next 🤔

Badge MOOC Challenge 4: Accreditation and Validation Frameworks for a Badge Ecosystem

Value Map Badge
It's Saturday, so it must be #OpenBadgesMOOC time :)  The thing that I just noticed about these badges on the #OpenBadgesMOOC is that if you look closely enough they look stitched.  Maybe there is an easter egg hidden somewhere, whereby if you earn all #OpenBadgeMOOC badges they send you a sash with all of them stitched on - LOL :)

In any case, it's the end of Week 4 on the MOOC (2 more weeks to go) and this week we are talking about validation.  It's interesting.  One of the things that comes to mind as I progress through these is that the initial "levels" were a little easier to articulate, at least for my #ESLMOOC project, but as the weeks progress it's getting harder since I don't have all of the information. This also draws a parallel to Kirkpatrick's Level's of Evaluation where Levels 1 and 2 are easier to measure, at least in the short term, but Levels 3 and 4 (and if you look at Philip's 5th level) it gets harder.

In any case, we trot on.  As with weeks before, the prompt comes first, and the brainstorming follows.  Any assumptions will be footnoted (I wish blogger had some good footnoting tools)

Challenge Assignment 4: Accreditation and Validation Frameworks for a Badge Ecosystem
At the next level of complexity, we add an additional component:
  • Validation
Because badges are agnostic as to the mode of learning that learners employed to gain competencies, learning providers can innovate with new ways to engage learners. While these new strategies may be as effective, if not more so, than more traditional approaches, postsecondary institutions seeking and maintaining accreditation are concerned about whether these new strategies conform to accreditation standards. Accreditors are themselves catching up to what rapidly evolving innovations in teaching and learning mean.

Learning providers have traditionally relied on academic accreditation and reputation as validation of the value of their targeted learning outcomes. Badge systems open the field for non-traditional learning providers as well as innovative learning methodologies in traditional institutions. With this opening up comes the need for clear, transparent validation of learning providers and their methodologies; without it, the value of their badges is questionable. Based on the badge system you described in the prior challenge, describe the validation framework that’s needed.
  1. Will traditional accreditation or other existing frameworks suffice?
  2. Are there relevant standards bodies that can provide review and validation?
  3. What types of reputation frameworks are already in place, and do they apply to the badge system?
  4. What peer organizational and institutional networks can be leveraged for peer review, evaluation, and endorsement?
  5. Will new research or evidence be needed, for example evaluation of employment outcomes (e.g, how many job seekers get jobs)?
  6. Write one or more “after badges” user stories depicting the value of the badge ecosystem for learning provider personas.

BRAINSTORMING for this week:
The live session this past Monday was pretty interesting, especially since there was a presenter from NEASC, who happens to accredit the school I work for.  Of course, that being said, since the #ESLMOOC is going to be more of a pet project (something that hopefully will lead to a dissertation - fingers crossed), NEASC won't have a ton to do with this type of accreditation.

Now, thinking of this week, I honestly can't say that existing "accreditation" will suffice for the #ESLMOOC.  What we have, in terms of "evaluation" for English Language Learners is standardized testing by the TOEFL by way of ETS, the CPE by way of Cambridge University ESOL, the MET by way of CaMLA,  and finally the IELTS test. At least these are the four big ones that people seem to know about.  From a framework perspective, we have the the CEFR developed by the council of Europe, and national tests tend to map onto this framework.  That said, the framework, while a good start, seems quite broad to me. I wouldn't be able to say with a high degree of certainty that two individuals, both at the C1 level (penultimate level) have the same exact skill set with the language. I am sure that they can adapt, but they won't necessarily be a perfect fit from the start.

Knowing this, I think that the council of Europe and the ACTFL, on the broad level, as well as national standards bodies for language would be able to come up with validation criteria for specific badges if tasked with this undertaking - the question is: are the on-board with badges? The other thing that comes to mind is perhaps a concern about over-regulation of badges. For example, in a Higher Education classroom, if a colleague wants to institute badges for both demonstrable knowledge (artefact based) and demonstrated behavior (no visible artefacts from the learner to be digitized and available to the accreditor), how much latitude does this person have from their department, college, university or accreditor to determine what sort of behavioral badges are appropriate for his classroom? I guess there is a Goldilocks Zone here, in badging,  as well.

Now, as far as reputation frameworks go, I am actually not really sure what already exists. I suppose, from people I know, the reputation is the fact that they have passed x-exam with y-grade.  For instance, last summer when I was talking to a number of friends of friends, or friends of family and they told me with pride that they had passed the "Lower" exam, or had passed their "Proficiency" exam.  This was usually tied to discussions about work, and working abroad, so for them this was a credential.  When I was asked if I had passed any of these exams, I was a bit perplexed, because even though I grew up in Greece, I also spent the last 20 years in the US, and have completed all of my tertiary education here. So, it would seem to me that these exams have a built-in reputation system because it is something that people can compare apples-to-apples to; whereas when they were asking me about my exams, since I had none (but I had alternate experiences), they did not know how to compare their achievements to mine.

Thinking about Peer Assessment and Endorsement, I would conceive this as three concentric circles.  In hte inner most circle, I would say that ESL departments in campuses around the New England area (NEASC's territory) could get together to peer review each other.  It seems that ESL, at least in my institution, is non-credit.  The #ESLMOOC, similarly, would also be non-credit; after all, it will be open, online, and free as MOOCs are.   The next level up is a circle at the national, or North American level.  Perhaps ESL departments from around North America can get together and peer review these badging system so determine efficacy, validity and fidelity.  Finally, the top-level circle would be standards bodies like the European Council and ACTFL looking at the broad strokes and reviewing this work.  So, the outer levels of this circle are more broad, and the closer you get to the core, the more details you get.

As far as research goes, I think that some research will be needed, but I don't think, at this point in time, that employment is the right rubric to use.  While the #ESLMOOC is mostly going to be targeted toward learners in higher education settings (maybe those who are looking toward switching the language of instruction to English), and there is some aspect of employability, I think that there are probably other areas that can be examined through research.  One area that really comes to mind, for me, is tying into Learning and Learner Analytics when it comes to classroom performance.  I think that as badges become more central (and assuming that there is no grade inflation with badges), it would be interesting to see how students with different sets of badges do in different classes.  Right now courses have generic pre-requisites for upper level courses such as "COURSE 601, 605, 612" or the nebulous "permission of instructor." The idea of pre-requisites is that you have gained, supposedly, some knowledge in those previous courses that is a foundation block for this upper level course.  Well, what about transfer credit students.  Sometimes transfer students meet 80% of the course requirements so they get to waive that intro course, but are they missing a certain key skill or knowledge that they will need to succeed in an upper level course?  With badges we can get more granular as to what students know, and can do, and therefore can better place them in courses.

With that, let me lead you into an "after" badges scenario.  In this scenario we have two actors: Councilor Troy.  She works at the University Advising Office at Big State University (BSU).  She is one of the people responsible for advising students for course sign-up each semester.  Based on student interests, courses completed, and University policies, Troy tries to help students sign up for the courses they need in order to get to their desired goals.

Right across from her is Jonas Quinn, who has recently transferred in from Regional Community College (RCC) after completing his Associates Degree in Computing Technology.  Jonas now had taken some Sociology courses while he was completing his AS degree and wants to major in Sociology now that he is a Big State University.  All of his general education requirements transferred in, so he is hopeful that he doesn't have to repeat any courses that he has already taken in order to pursue a degree in Sociology.

In years past, Troy's job was considerably harder, in that upper level courses, like SOC315 only had other SOC courses as pre-requisites, like SOC215. In turn, SOC201 had a pre-requisite of SOC101 and 115. Since Troy wasn't a member of the sociology department, it was at times hard to figure out what skills were needed to be successful in upper level courses.  Now that badges had been implemented regionally, Jonas had come in with a variety of badges from the sociology courses he had taken.  As it turns out, BSU and RCC use some common badges in their respective sociology departments.

It seems that Jonas has 9 out of 10 pre-requisite badges for his first BSU sociology course.  Since Jonas doesn't have all 10 necessary to join the course, but is over-qualified for lower courses, she send him to Professor Smith, department chair of Sociology, to seek permission to join the course, and to also see what he (Jonas) can do to demonstrate knowledge or ability for that 10th badge.

** Addendum Section Based on Feedback Received **
1. How does the badge ecosystem accredit and validate the learning provider and/or assessor and it's program(s)?
The badge ecosystem for #ESLMOOC has been envisioned to go along the same lines as accreditation of higher education programs in the US. Since #ESLMOOC is most likely going to be offered with a University partnership, any sort of accreditation stemming from a peer reviewer entity such as NEASC will be used to accredit the #ESLMOOC.  Also, #ESLMOOC badges should be interoperable with the CEFR that I have mentioned before.  This combination of regional accreditation as well as the European Framework (which is known at a much broader level) will serve as a way to accredit the learning and/or assessing in this MOOC.  That said, I don't think NEASC, or the Council of Europe have had to deal with badges yet, so foundations will have to be laid in order to get this Accreditor/Standards Body/Learning Organization partnership going :)

2. How can a learner/badge seeker determine the legitimacy of the provider, or the suitability of the provider's methodology to the learner's needs?
I have to say, this is quite a difficult thing to think about because even "accredited" programs (traditional courses and programs that is) seem to be peer reviewed through regional bodies that have some sort of endowment from the government. Just because a program is accredited in the country-x it doesn't mean that their degree is valid in country-y without some sort of additional work (or having to do the whole thing over again).  Thus, I would say that the criteria for determining the legitimacy of the learning provider would be the same as determining the legitimacy of an accredited program in the US since #ESLMOOC will run through a regular college program. If the school is accredited, then the course is accredited to some extend. It would be great to have the council of Europe backing it too (at least as a test-case), but it is not highly likely.

As far as the methodology goes - the MOOC format is really something that still has a lot of unknowns, which is why it's also an interesting topic of study for a dissertation. I don't think badges can really tell you much about the methodology used, at least if we go for badges for achievement. So, I don't see a concrete link between badge and methodology.  That said, since the MOOC format is so new, it would be really useful to provide some learning scaffolds and some up front information to let the learners know what they can expect in a MOOC.

3. In your ecosystem, could a well-structured reputation framework be a supplement or stand-in for peer review?
I think that Peer Review is important when learning a language.  Some learners will pick up on things that others will not. Therefore there is this collective scaffolding that goes on, and learners help each other fill the gaps in their knowledge and capabilities.  That said, peer review isn't the only thing that's important.  I know in xMOOCs (Edx, Coursera, et al) Peer Review seems to be the main form of assessment (along with machine graded quizzes), but this type of peer review isn't taken seriously, so I wouldn't give badges just for doing peer review.  I think that an instructor would have to review  the peer review comments and then badges can be provided to peer reviewers.  The better your reviews towards peer's work, the more you can "level up" in the badges you get.

4. How you think badges might result in grade inflation (which pre-supposes that badges exist along side grades, rather than in place of grades?
The #ESLMOOC won't be graded - unless of course the sponsoring organization wants to also give people the option for credit.  That said, graded assignments are not going into the design of the course.  Badges, however, are planned to be part of the design.  My concern with grade inflation is the subjectivity of the assessor.  In theory letter and numeric grades should be based on grading rubrics.  Sometimes people are more generous than others in awarding a grade or marking the rubric than others. If we look at this from a qualitative research perspective, if we draw parallels between grading and coding qualitative data, I am essentially worried about interrater reliability - or in my case "intergrader reliability".  Does this make sense? :)

** End of Addendum **

That's all for this week on badges.  Tune in next week for more ;-)  Your ideas for #ESLMOOC badges?
 Comments (3)
Stacks Image 20


 Nov 2023 (1)
 Aug 2023 (1)
 Jul 2023 (1)
 May 2023 (1)
 Apr 2023 (4)
 Mar 2023 (5)
 Feb 2023 (2)
 Dec 2022 (6)
 Nov 2022 (1)
 Sep 2022 (1)
 Aug 2022 (2)
 Jul 2022 (3)
 Jun 2022 (1)
 May 2022 (1)
 Apr 2022 (2)
 Feb 2022 (2)
 Nov 2021 (2)
 Sep 2021 (1)
 Aug 2021 (1)
 Jul 2021 (2)
 Jun 2021 (1)
 May 2021 (1)
 Oct 2020 (1)
 Sep 2020 (1)
 Aug 2020 (1)
 May 2020 (2)
 Apr 2020 (2)
 Feb 2020 (1)
 Dec 2019 (3)
 Oct 2019 (2)
 Aug 2019 (1)
 Jul 2019 (1)
 May 2019 (1)
 Apr 2019 (1)
 Mar 2019 (1)
 Dec 2018 (5)
 Nov 2018 (1)
 Oct 2018 (2)
 Sep 2018 (2)
 Jun 2018 (1)
 Apr 2018 (1)
 Mar 2018 (2)
 Feb 2018 (2)
 Jan 2018 (1)
 Dec 2017 (1)
 Nov 2017 (2)
 Oct 2017 (1)
 Sep 2017 (2)
 Aug 2017 (2)
 Jul 2017 (2)
 Jun 2017 (4)
 May 2017 (7)
 Apr 2017 (3)
 Feb 2017 (4)
 Jan 2017 (5)
 Dec 2016 (5)
 Nov 2016 (9)
 Oct 2016 (1)
 Sep 2016 (6)
 Aug 2016 (4)
 Jul 2016 (7)
 Jun 2016 (8)
 May 2016 (9)
 Apr 2016 (10)
 Mar 2016 (12)
 Feb 2016 (13)
 Jan 2016 (7)
 Dec 2015 (11)
 Nov 2015 (10)
 Oct 2015 (7)
 Sep 2015 (5)
 Aug 2015 (8)
 Jul 2015 (9)
 Jun 2015 (7)
 May 2015 (7)
 Apr 2015 (15)
 Mar 2015 (2)
 Feb 2015 (10)
 Jan 2015 (4)
 Dec 2014 (7)
 Nov 2014 (5)
 Oct 2014 (13)
 Sep 2014 (10)
 Aug 2014 (8)
 Jul 2014 (8)
 Jun 2014 (5)
 May 2014 (5)
 Apr 2014 (3)
 Mar 2014 (4)
 Feb 2014 (8)
 Jan 2014 (10)
 Dec 2013 (10)
 Nov 2013 (4)
 Oct 2013 (8)
 Sep 2013 (6)
 Aug 2013 (10)
 Jul 2013 (6)
 Jun 2013 (4)
 May 2013 (3)
 Apr 2013 (2)
 Mar 2013 (8)
 Feb 2013 (4)
 Jan 2013 (10)
 Dec 2012 (11)
 Nov 2012 (3)
 Oct 2012 (8)
 Sep 2012 (17)
 Aug 2012 (15)
 Jul 2012 (16)
 Jun 2012 (19)
 May 2012 (12)
 Apr 2012 (12)
 Mar 2012 (12)
 Feb 2012 (12)
 Jan 2012 (13)
 Dec 2011 (14)
 Nov 2011 (19)
 Oct 2011 (21)
 Sep 2011 (31)
 Aug 2011 (12)
 Jul 2011 (8)
 Jun 2011 (7)
 May 2011 (3)
 Apr 2011 (2)
 Mar 2011 (8)
 Feb 2011 (5)
 Jan 2011 (6)
 Dec 2010 (6)
 Nov 2010 (3)
 Oct 2010 (2)
 Sep 2010 (2)
 Aug 2010 (4)
 Jul 2010 (9)
 Jun 2010 (8)
 May 2010 (5)
 Apr 2010 (4)
 Mar 2010 (2)
 Feb 2010 (3)
 Jan 2010 (7)
 Dec 2009 (9)
 Nov 2009 (5)
 Oct 2009 (9)
 Sep 2009 (13)
 Aug 2009 (13)
 Jul 2009 (13)
 Jun 2009 (13)
 May 2009 (15)
 Apr 2009 (15)
 Mar 2009 (14)
 Feb 2009 (13)
 Jan 2009 (10)
 Dec 2008 (12)
 Nov 2008 (6)
 Oct 2008 (8)
 Sep 2008 (2)
 Jun 2008 (1)
 May 2008 (6)
 Apr 2008 (1)
Stacks Image 18