Blog Archives

Notes from LINC

This year I went to the LINC conference at MIT. This year’s theme was “Realizing the Dream: Education Becoming Available To All. Will the World Take Advantage?” I was officially there to present a paper about our course, but also to learn about other parts of education. I love conferences; you get such a mixing of ideas. Sadly I had a lot of work to do while I was there (this was in the early days of the online course, where more attention was needed) so I didn’t get to see as many talks as I would have liked.

The parts of the conference I did get to see were a mixed bag.

On the positive side, I saw a lot of efforts to improve education and make it more accessible. It was also great to see that this was coming from many different places in the world. For example, there was actually a talk where an audience member had to ask what a lahk was, because all the numbers in the talk were in that unit.

On the negative side, I saw a lack of actual research results. People were talking about the changes they had made, but rarely about the measurable impacts of those changes. Those who did have results rarely talked about them in the language of statistics.

One really excellent talk was at lunch the second day, from Cliff Missen of (among other things) the eGranary project. Missen started off the talk by saying that online courses are great when you can worry about whether or not people have internet access, but he more often has to worry about whether they have water. It really put things in perspective for me.

Overall I preferred the AAPT conference, but LINC’s more international view was definitely something I appreciated.


What it took and how we did it

This is how we created and ran an online course. It’s not the best way, it’s not how I would do it next time, but it’s what we did and I thought I would share it with you.

The process was not a single concerted push. The materials for this course were created and accumulated over five years of efforts from half a dozen postdocs (though only three at a time), perhaps a dozen undergraduates (one or two at a time), and one professor. These efforts were scattered, but often intense. If I wanted to create a complete course all in one go, I think I could probably do it with three people working full-time for a year.

Before The Launch Day

There was no central textbook for the course, so we needed to create something that not just tested the students, but taught them as well. We started with a lot of assessment materials – questions from homework, worksheets, exams, etc. that had been collected over the years. Most teachers and professors have this kind of material lying around in one form or another. We also had a good amount of instructional material – worked examples, textbook-style descriptions of concepts in physics, and videos that we were able to repurpose (see note at bottom).

We didn’t have much of a method to creating these things or choosing what to adapt. This was a mistake, and one that I don’t intend to replicate if I do this again. When I was hired I was told to “make some videos about rotation,” and maybe “write some stuff about center-of-mass.” We didn’t have a lot of structure just because we didn’t realize we needed it. Few people who get into this industry realize how difficult it is.

Our course was much more text-based than some online courses. A lot of courses use video lectures; we suspect that they’re not any more effective than text, though perhaps more enjoyable for some folks. The advantage was that we could edit text fairly easily, to remove errors and fix equations. Editing video in the same way is near-to impossible. We still used shorter videos to target a few specific concepts.

System mismatch errors were infrequent but persistent. The grading scheme we used on-campus and in LON-CAPA didn’t match anything that edX could accomplish. edX changed its codebase on a regular basis, which occasionally broke many of our problems. There were problems that we had to reconceptualize because the problem type we wanted wasn’t available – though I should give edX a lot of credit for adding features fairly quickly. Most of the problem types we wanted are now available, so we can do (for example) a “check all that applies” question instead of 7 individual true/false pull-down menus. The real lesson here is, “be flexible.” If we had an “our way or the highway” approach, well, we’d be out on the highway. When you run an online course, you’re not really the one in charge – the system is. A little willingness to change goes a long way.

We assembled these materials into the “course sequence”: the order in which students would see them if they went through the recommended path. At that point, the course is nominally ready – we could set the due dates and we’re ready to go, right?

After The Launch Day


We needed testing.

Lots of testing. More testing than you think. Keep testing. Don’t stop now, or ever. You will never test your course enough. You especially need people who are not you and novices in your field to test it. The more people who can test and check your material, the stronger your course will be for it, and once you open, you will still have issues from minor typos to major conceptual errors that pop up every day. From the day we opened beta testing to the end of the course, I would say we easily had a thousand issues, about 600-700 of which made it to our tracking system and many of which were fixed by course staff before they made it to that point. That’s for a course that had been offered before, where dozens or hundreds of people at varying levels had actually taken the course, read the material, watched the videos, and answered the questions. Keep testing. Prepare for bugs.

Ok. Deep breath. Sorry, I get sidetracked by that issue easily. We actually did very well. 1000 typos for 1000+ pages of text and problems is a very good ratio. If we hadn’t  run the course once or twice before it would have been a lot more.

While the course was running we had a bunch of staff helping out. I’ll talk more about this next week; there’s a whole structure to our wonderful support staff. For now, suffice it to say that when I send e-mails out to the “all staff” list, there are about 80 people on that list. Most of them are volunteers, working for cheap or for free. We could run the course without them, but it wouldn’t be as good, and it would have eaten up all my weekends. Their main contributions were in the course’s discussion boards, but they also were the front line when it came to detecting and reporting errors in the course.

Last time the course ran, our materials were still being assembled while the course was in action, only a week or two ahead of the oncoming storm of students. This time everything was in place, but our professor was still making edits.

Final Notes

You can see more detail about the history of the course, as well as some initial research findings, in the paper (PDF link) that we wrote for this summer’s LINC conference.

I’ll talk more about support staff next Friday. If there’s anything else that you as a reader would like to know about our process, definitely let me know and I’ll be glad to answer. I’ve assumed some foreknowledge here, like the classic question, “How do you grade the students?” and the like, so feel free to ask about that as well.

Note from above on repurposing:

We thought at one point that repurposing freely available educational materials for our own course would make things easier. In the end, I would say that it didn’t make things harder, but it didn’t make them much easier, either. It’s certainly difficult and time-consuming to create professional-looking lecture videos. (It is much more difficult and time-consuming than most people think at first glance.) However, it’s also very tough to find exactly the piece you’re seeking in someone else’s video, cut it to the right length, and not leave out anything important or accidentally keep something that you don’t want to cover. You end up watching a lot of video, often multiple times, just to get something that sort-of-mostly-fits into your course. You end up throwing out some videos and rewriting some pieces of text simply because they use a different sequence than you’d like. You end up rewriting problems because they don’t use your terminology. The issue of repurposing materials actually grew to be a major project for our research group for the future.

“Teaching” Online

This past year I’ve been heavily involved with Mechanics ReView, an online course through edX. I’ve polished the course after its transfer from LON-CAPA, replaced about 50% of the problems with new ones, recruited TAs, and actually run the course. I’ve also taught a lot of on-campus courses at both the college and high school levels, and I’d like to talk a little about the difference between teaching on-campus and “teaching” in an online course.

The quotes are there because I can’t really consider what I do online to be teaching.

There are strong similarities between online and on-campus teaching in the preparation stages. I and the research group I worked with still had to choose an order for the topics, come up with descriptive material (textbooks, web pages, videos, demonstrations, whatever you like), find a way to assess the students (quizzes, homework, projects, etc.), and put it all together in a well-tested package. Creating an on-campus course is much less work than creating an online course, but it’s not fundamentally different work.

Where the rubber hits the road, however, the two approaches are so different as to be almost unreconcilable.

As a teacher in the classroom, I spend my time talking with students, pulling out their current understandings and helping them weave those into new and stronger understandings. I tell stories about physics and physicists. I ask students to tell me stories about their own experiences, and about the things they imagine would happen in a little-toy-physics world that we construct together. I ask questions frequently; I rarely answer them. When there is ambiguity, we explore and understand it. When there is uncertainty, we use dialogue, consideration, and (rarely, in targeted situations) authority to clear it up.

As a course coordinator online, I rely heavily on my TAs. When I go to the discussion boards for the course, almost every question has already been answered, sometimes by another student, more often by a TA. (I consider this a minor failing of the course – more on that another day.) My interactions with the students are primarily limited to answering highly technical questions, either about more advanced physics than the course offers, or about the system on which the course runs. I spend more time fixing technical issues, rewording questions to remove ambiguity, and coordinating the wonderful staff that I have, than doing anything that feels like classroom teaching.

I think that a major component of this difference comes from bandwidth. Text, when compared to face-to-face interaction, is painfully slow and essentially void of nuance. This is doubly the case when typing back and forth to non-native speakers. Over 70% of our students come from outside the USA, and about 50% are from outside India, the UK, and Australia.

Another component is scale:  there are nearly 16,000 registrants for Mechanics ReView. Most of them are just here to take a peek, so a better measure is that there are over 1200 serious students in the course at this point. If I tried to do this all on my own, I would have no time to sleep. We have nearly 80 support staff who are the student-facing part of the operation, and I’ll spend a whole entry on them later, because they’re amazing people. They handle the day-to-day talking, answering, and interacting that presents the human side to the course. My job is at the top of a pyramid, where in the classroom I was the entire pyramid.

It doesn’t feel much like teaching at all. It feels like what I call it – being a course coordinator.

There is definitely still education taking place – you can see people learn; there is a semi-permanent record of students’ struggles with the material and their minor epiphanies on the discussion boards. I just don’t feel like I’m contributing to that understanding after the course begins. Research will show the relative effectiveness of classroom and online learning. My gut feeling is that there won’t be much difference, on average, between a course of well-delivered lectures and an online course with pre-recorded well-delivered lectures. What I want is a way to bring things beyond just lectures into the online environment. More about that on Monday.

If you want to take a look at the course, it should be visible through at least the end of September 2013, and hopefully archived well beyond that for anyone to see.