Gazing at Learning

In the grand scheme of things, this feels normal for a graduate course (from a design point of view), but it feels easy having gone through the experience of completing a doctorate. I think classmates who were in their first master's program and may have just returned to school either right from undergraduate studies or after a hiatus, probably found it a bit more challenging.  This reminded me of the many times that colleagues don't put themselves in the student's shoes when designing (or teaching) a course.  They think that something is easy because they can produce a 6,000-word paper in a month (not a term) so they might think that this is acceptable to ask of learners who are just learning the ropes. It seems like we often superimpose our current skills and capabilities on our younger (less experienced) selves, and that doesn't help serve our learners.

Last term I completed the negotiation course (yay!), so I saw that experience through. Now, as I am pondering what else to learn (I did say I was a lifelong learner), I am wondering if others have experienced that kind of shift in mental gears, whereby if they have completed a higher degree (e.g., Master's) they find undergrad courses (e.g., a sophomore level course) too boring.  Do you roll with it and enjoy the change of pace? Do you challenge yourself with your own challenges?  Do you not sign up for those courses?  And, how does being a learner again impact your ways of being in the teaching (or designing) role you have?

Yesterday evening I came across a post by Tony Bates where he shared some findings from some recent reports on faculty (and student) preferences for learning in a post-pandemic world. I haven't read the reports yet, but I had some initial thoughts based on Tony's high-level overview of those three reports.  

Two big things jumped out at me. The first is that students and faculty aren't on the same page (for the most part), and faculty still overwhelmingly prefer on-campus.  Tony notes that, at least in the EDUCASE case, compared to pre-covid numbers, more faculty have expressed an interest in something other than purely face-to-face (about 79% pre-covid, with 50% now).

Tyton Partner's Analysis of Modality Preferences

The first thing I noticed above is where students and faculty fall in their respective preferences.  If taken together, something other than purely face-to-face is a clear winner for student preferences.  Now, what that something else actually is...well, that's up for debate. I honestly don't like how they've broken up Blended and Hybrid. In their definition of Hybrid we seem to be using a kind of Allen & Seaman definition of 30-80% online (the traditional definition of blended/hybrid), whereas the Blended definition here appears to be somewhat of an inverse: 80-90% on-site with some online. This seems like the worst of both worlds, sort of like having a job that's "Hybrid" but requiring you to go into the office four days per week 😅.  The numbers on Hyflex surprise me on the student side, but not the faculty side.  The various *flex approaches require A LOT of work for the instructor without requiring a lot of skin in the game for students: they can choose to be virtual at any time without much warning to the instructional staff to allow them to prep for the changing classroom dynamics.  I sometimes feel like *flex courses are students responding "maybe" to an invitation when they clearly want to decline it but feel bad about doing so 😂

Back in 2014, when I started my EdD at Athabasca, I had a chat with Pat Fahy (now retired) during one of our breaks from orientation. This was about the reason why students choose distance education over physically proximal education, and his response was "It's always the parking, stupid!"  Of course, it goes beyond parking.  In Boston parking is one thing ($15 per park on-campus!), plus traffic, plus coming after work, plus gas money, plus tolls for some people coming from farther away. The monetary and time costs add up.  Of course, pre-pandemic not everyone thought about this.  We were all in a routine, so when the idea of DE came up, the usual (and very automated) response was "Oh, Online is fine, but I learn better on campus"🤷‍♂️. I think what we are seeing in a post-pandemic world is that when students have had a break from their routines, and they can step back and do a compare/contrast critically, they may choose other options.  In fact, we see from the data above that most students picked some option other than purely face-to-face, which is telling IMO.

EDUCAUSE analysis of faculty preferences

In EDUCAUSE's report,  one of the reasons why faculty prefer on-campus is to have a connection with their learners. Again, having taught folks how to teach online (for over 12 years now... oh how does time fly...) this is something I've heard frequently: I prefer teaching on-campus because I can see my students, get a temperature read in the classroom, connect better (whatever that means), etc.  I think this may be true for some folks, but it's not true for all, and certainly not in all cases.  It definitely privileges students who are extroverts and engage more in those social spaces. Assuming you can read people's facial expressions can also bring you to incorrect conclusions: I often went to class, for example, after work - so by that time I was away from home for about 10 hours already.  Any apathy or boredom on my face was probably fatigue, otherwise, I wouldn't be in class. I am not convinced that we can accurately read people's faces and determine their level of engagement (maybe a lit review I should dive into one of these days).  

Ultimately what this need to connect points me to is this: instructional staff are replicating their own classroom experiences in their own teaching and they have never learned new ways of being.  Most folks who've been active in virtual spaces (chatrooms, BBS, IMs, Social Media) can tell you that they've felt those kinds of closeness in virtual spaces that people express having felt in physically-proximal spaces (yes, there are probably a lot of asterisks in this claim). It's something that's learned and experienced, but it requires practice (just as F2F requires practice...and most of us have been practicing for a long time). I think it's important to cultivate additional ways of being that are geared toward virtual spaces.  This doesn't mean that F2F is bad, but it also doesn't mean that F2F is the only way.

Bing Image Creator: A Syllabus

I think it's important to start with a course that you know so that you can both critique the design "decisions" that GPT makes and also leave yourself open to being pleasantly surprised by what an LLM produces.  The three AI services I tested were SMARTIE, ChatGPT 3.5, and Gamma. I won't comment on SMARTIE extensively because I chose to run another course that I used to teach frequently (INSDSG 601: Foundations in Instructional Design and Learning Technology) through it so I could do a compare/contrast, but I got a bit bogged down enough to just junk that attempt. If you're curious as to how far I got, here's the unannotated Google doc of the SMARTIE 601 syllabus/outline.

So, back to 684.  Here's the ChatGPT syllabus with my comments, and here's the Gamma PowerPoint output (which is still in beta). Feel free to go through and make your own annotations.

Some big-picture things I've noticed:

• It took me over 90 minutes to prompt CGPT for everything, copy it into the Google Doc, and format it. This doesn't delete duplicates, critically analyze the syllabus, and address duplications and inconsistencies.  If I were called upon to teach this course again, with no current syllabus or list of learning objectives from the department, it would take less time than prompting CGPT to produce something for me.
• CGPT produced a list of required and optional texts for the course, and while most were legitimate (i.e., they exist), it really did incorporate some made-up stuff in that list, which means that anyone using this really, really, really needs to fact-check those books and articles.
• It's worthwhile noting that GAMMA did the same thing with the course text. It totally made up a textbook. How do I know? I couldn't find it on Google, on IGI's website, and it's not listed on the faculty profile of the supposed author.
• CGPT produced a plan which made for both an uneven, and an unreasonable workload for learners. For example, just for Week 1, the assigned readings were around 450 pages in two assigned books. This pattern continued through Week 10, while Week 11 only had 2 articles (maybe 40 pages between both). Alignment between learning objectives for the course, the weekly topic, and the readings was there in a way, but boy was it a lot!
• Speaking of additional resources, CGPT recommended videos, podcasts, and articles (at my prompting), but didn't include any URLs.  I know that this is an issue, but when specific resources are given, resources that have defined URLs and DOIs (e.g., Teaching in Higher Ed Podcast and Sal Khan's TED talk - Let's Use Video to Reinvent Education) should be no brainers
• It seems like CGPT has a weird fascination with pairs.  If you skip down to the Activities section of the syllabus, you'll see that every week basically has two activities for students to complete. While CGPT gets points for keeping things interesting and varied, I am having a hard time seeing how the learning community gels if they're jumping from task to task.
• The criteria points for graded assignments were basic and repetitive for all assignments.  For what it's worth, I felt the same way about rubrics produced in SMARTIE for the other class.
• Going back to student overload for a moment, I don't think the activities and the course deliverables were really all that integrated (or at least it seems like CGPT didn't consider what had come before from past prompts when it provided me with activities) because the grading breakdown for the course not only didn't consider the activities and assessments that came before, it added to them!
• In terms of course policies, was pleasantly surprised that the camera policy for live sessions is "camera optional"
• Also, in policies, I think that many things that were presented as a "policy" seemed more like classroom norms to discuss and engage with learners about
• GAMMA did produce some headscratchers - such as "screen recording" being used in the design of the course. I don't know how germane that is to learners in the course.
• Finally, with the exception of a hiccup in CGPT, it seems like a lot of final projects for this course are an online course designs, which is to be expected.  In fact, I think both all 5 instructors for this course (going back to 2008) have had this as a final deliverable.

Another me in a Star Trek setting, by NightCafe

So...with that in mind, here are some use-cases that I can think of (some of which have been borrowed and adapted from the first #etmooc session).

Use Case 1: Getting your biography starter pack from ChatGPT. 

I like writing.  I don't like writing about myself.  It feels very toot your own horn-like, and I've always never liked those people.  I acknowledge that to get ahead in life, and in academia, you have to do some of that self-promotion. Still, I don't like writing about myself. So, I can foresee using something like ChatGPT to give me an author biography for me, something that is between 200-300 words because I hate writing those.  I could feed it my academic CV, and it could trawl the internet to see what's been said about me (including LinkedIn testimonials), and it could produce something that I could use whenever I need a bio.  This is something that doesn't work well at present time.  I have asked ChatGPT to give me a bio and it just makes stuff up. I've asked it a few times over the last two months, and it still gets things wrong about me. Could I train it to have the correct info? Probably.  Do I want to? No!  The most recent wrongisms about me will be posted in Part III of Session 1 ponderings.

Use Case 2: Generating a character sheet for an RPG. 

I can foresee AI being used as a way to create a randomized character sheet though (or a character sheet based on a made-up bio from example 1). This kind of character sheet can be used in gaming, but it can also be used by students throughout the term if we want to make the classroom a kind of RPG experience.   Another use, blending use-case 1 and use-case 2 is the creation of a CCG (collectible card game) card with you as the character.  Think of this as a kind of playful business card.

Use Case 3: Using AI to change the format of a paper.

I usually write with APA in mind. Yes, I know that I can use a citation manager like zotero to update all my citations on the fly, but that doesn't account for other elements (like spacing, use of footnotes, making all footnotes endnotes, and vice versa, and so on).  Citation managers also ask you to do A LOT of work up front to make sure that your citation data isn't garbage, whether you end up using those sources or not. I could foresee a future where AI processors are "smart" enough to take a paper written in one format (APA, MLA, IEEE, ACM, etc) and convert the entirety of the paper into another format.

Use Case 4: De-escalating writing

We sometimes write emails when we're mad or upset.  It's usually not a good idea to send those. Maybe you need some time to cool down.  Perhaps a use of AI could be to put some of those emails in a penalty box for review after some time has passed.  GMail gives me nudges for follow-up emails, so why not a nudge to check to see if I really really really wanted to send that nastygram? Or, perhaps Grammarly (or ChatGPT) can help me rephrase it or edit it to make it sound more professional but still register one's discontent with the situation.

Me in a Star Trek-themed anime AI image

Part of the session was really dedicated to identifying ways in which this kind of technology can help with what we do.  Essentially flipping the script and going from "ZOMG! ChatGPT is used for cheating" getting to "how to use ChatGPT to help us with learning?" 

There were a number of examples used in this brainstorming session which present for red flags for me.  I did think of a few examples of my own that may (or may not) be good examples of what you could use tech like this for.  I'll start with my concerns though.

Example 1: Using ChatGPT to grade. This is a use case of having a kind of machine-human collaboration. It was acknowledged that the machine can't really accurately grade everything, and the instructor should have a look-over to correct or supplement the machine output, but this would be a potentially revolutionary use of this tech.  I'm not convinced. First, I have issues with both feeding student submissions into such a system without appropriate guardrails for student submissions. We've seen, from past actors in this space, that they take student submissions and appropriate them for their own use.  Students should not have to consent to having a "smart" machine grade their submission as part of their learning experience.  My second issue here is that such machine grading takes away Teacher Agency to some extent, and it may be taken away as a means to being more "efficient" or "less burned out." Teachers and course facilitators are in a classroom for a reason. If grading submissions is becoming an issue it's important to interrogate why they are becoming an issue instead of throwing some LLM at it.

Example 2: Continuing on with that thread of human-machine collaboration, when working with ChatGPT it's like you're working in a group, but instead of other humans, your team-mate is an AI.  Maybe if AIs were like Mr. Data on Star Trek, I might have a different opinion.  Right now LLMs are like dumb appliances.  They can "learn" but they are essentially machines. Collaboration requires agency, scope definition, goals, and drive, which machines simply do not have.  In Connectivism, you can have interactions between human and non-human/appliance nodes, but I would not go so far as to say that they are collaborating. It's not even a one-sided "collaboration" for the human in that equation.  When you're collaborating in a team you don't have to fact-check your team-mates submissions.  You can have sufficient overlap between areas of expertise so as to have more than one pair of eyes on claims made, and people who are more expert at something can ELI5 things to other team members, but ultimately there is a back and forth.  In a human-machine "collaboration" you have the issue whereby the human needs to be an expert in the subjects to be able to know where the machine goes wrong and correct it.  In a learning context, I think that this is potentially detrimental to the learning process.  It's not the knowledge navigator future we've dreamed of - at least not yet.

The question that came to my mind is this:  why are some folks thinking of LLMs as a "collaborator" and not looking at Google search as a collaborator?

Example 3: OK, final critique here.  One of the things I've heard over the last few weeks is something along the lines of: if you are a good prompt engineer you can get some amazing information, which you have to fact-check."   There are just too many conditionals here to be that kind of study buddy that was mentioned above in example 2. Now, this reminds me of my undergraduate days when I learned about library databases and how to search for resources using Boolean logic. Yes, you needed to play around with your logic and your search terms (and sometimes you needed to learn controlled vocabulary), but you got actual sources that you could read and evaluate (and cite).  I think prompt engineering is less of a sign of things that learners need to learn and more of a sign of a system that is still half-baked ;-). That said, I come back to the fact that you need to know how things work in order to assess whether the output is of any use (or even factual).  An example that comes up is people learning another language.  You would write something in English (assuming English is not your native language) and pop it into an LLM to have it convert that output to something more "native sounding." Amongst other issues, it's useful to know why something sounds more correct than other options when you're learning a language. An LLM could do it for you, but that doesn't help you progress as a learner. We had an example of why it's important to know your stuff (even if machines help) in Star Trek Picard this season. The short version is that the ship's captain is brought to sick bay with some symptoms. The veteran doctor realizes that he has internal bleeding that the younger doctor's medical imaging devices failed to catch. If the veteran doctor didn't know her stuff, the captain would be dead 🤷🏻‍♂️

Anyway, this post is getting too long, so I'll save my ideas for using ChatGPT/AI for another post ;-)

Thoughts? 

~~~~~~

Just for documentation purposes, here are the objectives of the first session:

By participating in the synchronous Zoom session and any additional activities you pursue as part of your own learning experience, you will see how your colleagues are responding to ChatGPT. By the end of the live session and completion of any other activities you pursue, you will be able to:

• Identify at least three ways ChatGPT might be of benefit to you and those you serve in your section of our lifelong learning environment
• Anticipate at least three challenges ChatGPT may pose to you and those you serve
• Describe at least one way you may begin incorporating ChatGPT into your work or describe at least one step you can take to overcome a challenge you face in incorporating ChatGPT into your lifelong learning efforts

I think you should go and read Dave's post, since I'm just going to reflect and react on a few select points. Dave introduced me to an article by Schinske and Tanner (2014) where they describe four purposes for assessments, those purposes being feedback, motivation, student ranking, and the objective evaluation of knowledge. 

There were two things that jumped out at me: (1) the objective evaluation of a learner's knowledge and (2) ranking learners.  

From a philosophical perspective, I don't think that it's the instructor's job to rank learners. As an instructor, I am not there to indicate whether Tom is marginally better at X than Dick. My job is to help learners go get through a particular corpus of knowledge that they should be able to use to do something with. This type of assessment only really exists on a one-to-one basis. As an undergrad I had a professor, let's call him Hans, who really believed in the bell curve. 🙄 On the first day of class, he announced that there would be so many As and most people would fall in the B/C range.  I don't know what his feelings or beliefs were about the other end of the bell curve (the D and F), but I don't know if we ever found out. The knowledge that no matter how well you do, you are ranked against others is demotivating. If I know that my grade is most likely going to be in the B-range, I'll most likely nope out of most things in class and strategically choose what to participate in. If I were a student in an AI-world (assuming the AI generation was worth anything) I'd most likely be tempted to just autogenerate whatever garbage since assessments were more about a belief that anything is actually useful. As an aside, I still, to this day, wonder what a belief in a statistical probability chat is 🤷‍♂️. As an aside, as an instructional designer, I also must have missed the cID lass where it was my job to help devise assessments to rank people, instead of actually...assessing their knowledge and application of that knowledge 🤣

The other thing that jumped out at me was the objective evaluation bit. The more time I've spent teaching, the more I've come to the conclusion that I cannot objectively evaluate the entire corpus of the class I teach.  Well, I could, but it would take a very (very very) long time.  Instead, what I've observed happening is that we use snapshots of concept evaluation as a proxy measure for the entirety of the corpus of knowledge that we try to cover in our classes. We pick concepts that may be more "important" than others, or concepts that can be used like key puzzle pieces so that students can fill in that negative space with concepts and knowledge that's adjacent to what we're testing. Ultimately one cohort of CLASS101 is not going to be evaluated the same was as another cohort from CLASS101.

This reminded me a little bit of a conversation I had with one of my doctoral professors at Athabasca.  We were discussing comprehensive exams at the Master's level.  He was telling me that AU moved from a comp exam to a portfolio because, ultimately, their comp exam was neither comprehensive nor an exam.

In any case, back to course design. Dave writes that the internet (over the past 30 years) has changed the educational landscape. The way  I see it, these represent some different eras of the web. Here's what Dave wrote (a bit paraphrased and expanded) - learners have...

• The availability of connections to other people to support learners in their education journey - examples being SMS, group chats, and various IM chat clients (Yahoo, ICQ, MSN, etc.) and so on. I would say that this was my generation of college undergrad. It wasn't everyone who did this (there is a certain amount of access privilege associated with this), but it seemed like classmates were a good source of peer learning whenever we got stuck.
• The availability of pre-created content available through sites like Chegg and also through google searches. Content that can be used to respond to any existing assessments.  This is just a digitalized version of the ol' sneaker net of years past where an someone who had taken the course before could share an exam with others. This was the mode of concern up until the ChatGPT paranoia hit in early 2023. 
• The availability of generative systems that can create responses to assessments, whether they are "correct" or not. This is where we are now with things like ChatGPT

While reading Dave's original post, I was reminded of conversations about connectivism over the past 10+ years. This is, in fact, an instantiation of connectivism. We connect to human and non-human nodes, "learning" seems to be residing in non-human appliances, and decision-making is itself a learning process. This last point I want to focus a little bit on because I think it has implications for design, teaching, and of course - assessment. If we take decision-making as the center of our learning experience, what kind of content is our sine qua non (SQN) content? These are the minimum elements that we need to begin to make decisions, and also allow us (as learners) to unravel connections to other learning that we need to do. Dave writes that "with the plethora of options for students to circumvent the intent of our assessments this require the rethinking of the way we design our courses."  I agree.  The question is what is that the core of that learning experience?  Not necessarily content (although it is important to some extent), but rather the ability to be lifelong learnings, get new inputs, assessment, and perhaps use them to make decisions for an ever-evolving ill-formed set of problems that come our way in our personal and professional lives.

Whaddyathink?

 Another AI-based image tool shared in ETMOOC that I thought I would try out.  This one is called NightCafe and it creates images based on a prompt and a particular style from its styles list.    My prompt for this one was: Show me a small group of Greek students huddled around a cafe table, drinking caffeinated beverages, while vigorously discussing philosophy.

It's interesting that in my mind I had "college students" but these images remind me a lot of Rennaissance Italy (Assassin's Creed II time period) rather than a more modern Athenian Cafe with Freddo Cappucino and Frappe coffees...

Anyway - I thought I'd share some of my reactions to the article which I jotted down as I was reading it...(sometimes  I feel like these would be better as TikTok Takes 😂)

"I shared with colleagues that “All we have to do is ask ‘Did you write this?’” and then copy and paste the student work into the prompt box."

My first question is: did you actually research how ChatGPT works? With follow-up questions about the ethics of using a student's submission to feed to the system? And...why did you share with your colleagues that you're doing this experiment as if it's a done deal? I mean, I'd personally run the experiment first and see what happened before broadcasting something potentially embarrassing...🙄

"My inkling was that somebody created a shared study guide using ChatGPT and then, one by one, students took pieces word for word that they included in their short essays. Was this an issue of plagiarism or crowdsourcing—and is there a difference?"

If you don't have a good and solid definition of academic dishonesty, then that's your first problem. Crowdsourcing is (most likely) not the same as plagiarism (depending on what your definition of plagiarism is).  You really should know for yourself what you're looking for before you go looking for it because you also need to clue in your learners as to what is and is not academically dishonest in your setting.

"While I was wrestling with the haste of my initial response, I received six frantic student emails, each professing their honesty, fear and dismay. Another student stopped by in tears. She rushed out of her house to catch me early in order to explain that she’d worked really hard on her study guide and the subsequent essay question. Yet another student sent me an example of how ChatGPT took credit for writing an article that we read as a class. By now, I knew that the tools at my disposal were flawed. Those 20 students in my class did not cheat on their essays, despite my confidence in my sleuthing skills."

Congrats my man! You managed to instill in students the fear of God, for no reason. You brought students to tears.  I. Hope. You're. Happy! 🙄 You have no sleuthing skills, I think that's now a given.  Listen, I don't mean to be harsh on this person, but before you accuse anyone of anything, you best get your facts straight (unless of course, you're Fox "news" and have money to burn in courtrooms and settlements). I genuinely feel bad for the students, and I hope you do too!

"I lamented to the students that I had made a mistake. Instead of focusing on student growth and innovation, I invested too much time into surveillance strategies. Instead of focusing on my commitment to developing strong personal relationships with students, I let a robot tell me what I should believe. Instead of trusting my students, I had trusted an online tool."

I think hindsight is 20/20, and I am glad you've learned from this experience, but did you really have to put students through this so that you could learn? Why not start with care and believing students, and working on mentoring rather than trying to catch supposed plagiarists?🤔

"The scenario in my class, however, also reminded me of some basic principles of critical pedagogy that we must consider as we are ushered into the new AI era. First, we must trust students, foremost because they are real people. They bring to the classroom their values, livelihoods and traumas. We must be mindful of the ways in which academic surveillance dehumanizes the very people we signed up to nurture"

We've been through this (at least) once before with remote proctoring "solutions" during the three years we were in the height of COVID and people were not in classrooms.  Why did we need to re-learn this lesson so soon?  It begs the question - did we really learn it the first time around? or did we just pay lip-service?

"if I were to continue to subscribe to the belief that it’s an educator’s job to catch and punish plagiarists, then everyone in the class would lose."

I don't get paid to be on the lookout for potential plagiarists and cheaters.  That's not the job of the faculty member. Yes, I think if you voluntarily find yourself in that position (because you'd have to consent to doing it), it would be a lose-lose proposition. So...why bring yourself into that position to begin with? 🤔