What it took and how we did it

This is how we created and ran an online course. It’s not the best way, it’s not how I would do it next time, but it’s what we did and I thought I would share it with you.

The process was not a single concerted push. The materials for this course were created and accumulated over five years of efforts from half a dozen postdocs (though only three at a time), perhaps a dozen undergraduates (one or two at a time), and one professor. These efforts were scattered, but often intense. If I wanted to create a complete course all in one go, I think I could probably do it with three people working full-time for a year.

Before The Launch Day

There was no central textbook for the course, so we needed to create something that not just tested the students, but taught them as well. We started with a lot of assessment materials – questions from homework, worksheets, exams, etc. that had been collected over the years. Most teachers and professors have this kind of material lying around in one form or another. We also had a good amount of instructional material – worked examples, textbook-style descriptions of concepts in physics, and videos that we were able to repurpose (see note at bottom).

We didn’t have much of a method to creating these things or choosing what to adapt. This was a mistake, and one that I don’t intend to replicate if I do this again. When I was hired I was told to “make some videos about rotation,” and maybe “write some stuff about center-of-mass.” We didn’t have a lot of structure just because we didn’t realize we needed it. Few people who get into this industry realize how difficult it is.

Our course was much more text-based than some online courses. A lot of courses use video lectures; we suspect that they’re not any more effective than text, though perhaps more enjoyable for some folks. The advantage was that we could edit text fairly easily, to remove errors and fix equations. Editing video in the same way is near-to impossible. We still used shorter videos to target a few specific concepts.

System mismatch errors were infrequent but persistent. The grading scheme we used on-campus and in LON-CAPA didn’t match anything that edX could accomplish. edX changed its codebase on a regular basis, which occasionally broke many of our problems. There were problems that we had to reconceptualize because the problem type we wanted wasn’t available – though I should give edX a lot of credit for adding features fairly quickly. Most of the problem types we wanted are now available, so we can do (for example) a “check all that applies” question instead of 7 individual true/false pull-down menus. The real lesson here is, “be flexible.” If we had an “our way or the highway” approach, well, we’d be out on the highway. When you run an online course, you’re not really the one in charge – the system is. A little willingness to change goes a long way.

We assembled these materials into the “course sequence”: the order in which students would see them if they went through the recommended path. At that point, the course is nominally ready – we could set the due dates and we’re ready to go, right?

After The Launch Day

No.

We needed testing.

Lots of testing. More testing than you think. Keep testing. Don’t stop now, or ever. You will never test your course enough. You especially need people who are not you and novices in your field to test it. The more people who can test and check your material, the stronger your course will be for it, and once you open, you will still have issues from minor typos to major conceptual errors that pop up every day. From the day we opened beta testing to the end of the course, I would say we easily had a thousand issues, about 600-700 of which made it to our tracking system and many of which were fixed by course staff before they made it to that point. That’s for a course that had been offered before, where dozens or hundreds of people at varying levels had actually taken the course, read the material, watched the videos, and answered the questions. Keep testing. Prepare for bugs.

Ok. Deep breath. Sorry, I get sidetracked by that issue easily. We actually did very well. 1000 typos for 1000+ pages of text and problems is a very good ratio. If we hadn’t  run the course once or twice before it would have been a lot more.

While the course was running we had a bunch of staff helping out. I’ll talk more about this next week; there’s a whole structure to our wonderful support staff. For now, suffice it to say that when I send e-mails out to the “all staff” list, there are about 80 people on that list. Most of them are volunteers, working for cheap or for free. We could run the course without them, but it wouldn’t be as good, and it would have eaten up all my weekends. Their main contributions were in the course’s discussion boards, but they also were the front line when it came to detecting and reporting errors in the course.

Last time the course ran, our materials were still being assembled while the course was in action, only a week or two ahead of the oncoming storm of students. This time everything was in place, but our professor was still making edits.

Final Notes

You can see more detail about the history of the course, as well as some initial research findings, in the paper (PDF link) that we wrote for this summer’s LINC conference.

I’ll talk more about support staff next Friday. If there’s anything else that you as a reader would like to know about our process, definitely let me know and I’ll be glad to answer. I’ve assumed some foreknowledge here, like the classic question, “How do you grade the students?” and the like, so feel free to ask about that as well.

Note from above on repurposing:

We thought at one point that repurposing freely available educational materials for our own course would make things easier. In the end, I would say that it didn’t make things harder, but it didn’t make them much easier, either. It’s certainly difficult and time-consuming to create professional-looking lecture videos. (It is much more difficult and time-consuming than most people think at first glance.) However, it’s also very tough to find exactly the piece you’re seeking in someone else’s video, cut it to the right length, and not leave out anything important or accidentally keep something that you don’t want to cover. You end up watching a lot of video, often multiple times, just to get something that sort-of-mostly-fits into your course. You end up throwing out some videos and rewriting some pieces of text simply because they use a different sequence than you’d like. You end up rewriting problems because they don’t use your terminology. The issue of repurposing materials actually grew to be a major project for our research group for the future.

Advertisements

About Colin Fredericks

By day I help to create online courses at HarvardX. By night I write roleplaying games.

Posted on August 9, 2013, in Uncategorized and tagged . Bookmark the permalink. Leave a comment.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: