Blog Archives

Crowdsourced Grading

The beautiful dream of every teacher: never having to grade. After many years of course design, I think that’s one of my main concerns when I create a new course. I say to myself, “I know I can teach the students well, but how can I do it with the minimum amount of grading?”

Grading is a pain. Grading is why a teacher’s day does not end when school ends. Grading is what makes it feel like your days just. Do. Not. End. Oh, classroom management can be a pain, and there’s a lot to be said for teaching students who genuinely want to learn, but six hours of grading on a Sunday can wear you down just as bad as a non-compliant classroom. I’ll look at basically any way to decrease that load.

The Crowdgrader tool mentioned in the above link is fairly simple – teachers create “assignments” (which means they write some instructions), and students upload their completed submissions, and then grade each other’s work, and then rate each other’s feedback. A nice touch is the ability to do group submissions.

I’ve done a few peer grading activities in the past, all of them in the classroom. The less structured these activities were, the worse they came out. When I set up a clear rubric, peer grading generally went very well. Students especially seem to appreciate rubric-based activities when they know that you’ll be using a similar rubric to grade exams. It’s always nice to open up the black box of the classroom so that students know how things work. In more technical or creative fields it can also be nice to see how your classmates approached their work.

In fact, I think that might be the most useful thing about all of this. Forget using it for courses where most things can be boiled down to right and wrong answers. That’s just as big a waste of student time as it is a waste of teacher time. Let’s push for this in more creative, more demanding courses where the approach can be just as important as the final answer. Let’s use this when there are multiple paths to the right answer, or when we’re in that fruitful area between purely objective and purely subjective, where justification and rational argument really count.

Using this in a MOOC would be another thing entirely. The ratio of participants to lurkers in MOOCs is so low that if you assigned everyone’s work to five other people during the first week of class, you’d get 1/2 of a person responding. However, later in the course, or with courses that have better “attendance” rates, there’s a lot of potential in this tool.

Turning Students into Teachers, Online

I want to talk today about what I consider one of the major failings of our online course. Other people probably won’t consider it that way, because this wasn’t an agreed-on goal for our course (or even a discussed goal), but I was always hoping that it would happen.

(By the way, this is what happens when your “design” for something it “I sure hope it happens.”)

What I was hoping would happen was that, as the term went on, our students would help each other out more. I was hoping that our course staff – both on-campus and volunteer – would be able to pull back on our involvement in the forums, and that we would have students from the course who talked about physics, discussed concepts and problems with each other, and generally helped each other out.

This did not, as a whole, happen. Certainly there are some folks who helped one another, but for the most part our discussion board was hierarchy-driven and expert-reliant.

Part of this was the fact that we intentionally recruited Community TAs (see my previous discussion of the course staff), and they were seen as part of the staff. They weren’t viewed by the majority of forum-goers as regular students who were just friendly and helpful, these students became elevated in the eyes of others. Suddenly they had a green border around their posts, while others did not. They had a little tag that practically announced, “I KNOW PHYSICS so you should LISTEN TO ME!” I would have loved to change some students’ epistemologies from “receive knowledge from experts” to “learn and help to learn skills as part of a group.” I don’t think this happened at more than a background level.

Part of it was also the size of the group. There were about 300 people who introduced themselves in the forums, maybe 1200 who got certificates (we’ll know an exact number in two weeks), and 15000+ who registered for the course. In a group that size, you’re almost guaranteeing that most people don’t get a chance to shine. The folks who really are experts are likely to hand out answers when asked. The folks who aren’t experts (yet) are likely to receive and use them. That gets them points, which is what they want, so everything continues in that manner.

We didn’t create a system that encouraged expertise, collaboration, and mastery to the extent that I would have liked. I think that our pedagogy within the course materials has a better-than-background chance of pointing people towards expertise, but the way the students were able to interact didn’t reinforce that.

I think it’s a neat course design challenge.  As I mentioned, we didn’t intentionally design for it this time – the closest that happened was some encouragement I sent out to our volunteers, and that was only once or twice. I think it would be worth trying some intentional design in this area. We know that student interaction works as a teaching method. Bringing it to the online arena is clearly not just a matter of letting students interact. If we want to be more effective, we need to be more intentional.

Reflections on MIT

In two days I leave MIT.

I wanted to take a little time to reflect on what the place is like. There’s a lot of mythology and legend built up around the place – you can tell from the fact that we have tourist groups from around the world here every day or two. I wanted to give a little insight into the Institute beyond the “we number everything” and “LOL hacks” level.

MIT is a highly “siloed” institution. When you’re working in your group, it’s common to not hear about other groups. Who are doing the same thing. In the next building over. Part of this is due to workload. You spend a lot of time working rather than making connections, which is fairly understandable. Unfortunately, it’s endemic to the culture of the institute. For instance, the network on my floor is run by the same group that runs the network on half of the 2nd floor, but the other half of the 2nd floor and the 1st floor are run by a different group. I think it’s only in the last year that the various education research groups have taken notice of each other. I sort of worry that when I go, one of the big links between our group and the Bio Education group will disappear.

The “not made here” ethic is also big at MIT. People would rather create something of their own than use existing materials. I can’t say that we end up with shoddy products because of it – instead, we end up with products that lack functionality.

It was stunning to see how technologically conservative the school is. Only last year did they start using online course registration, something that RPI implemented when I was an undergrad and UMass implemented when I was a grad student. Naturally, the “not made here” problem was present in that as well. I was also surprised that when I got here, I, who just came from teaching high school for four years, was the tech guy in my group. The stereotype that everyone at MIT is a technological wizard is a little overblown. (There are tech wizards here, just not everyone.)

On the more positive side, it’s very encouraging to see the motivations of the people who work here. Sure, there are curmudgeons who are only in it for their own fame, but the majority – the vast majority – of professors and staff at MIT do what they do because they want to improve the world. These are people motivated by compassion, by their conscience, and by consideration for others. I appreciate their pursuit of the truth, but more than that I admire their humanitarian goals.

MIT is most definitely filled with smart people. It’s also filled with very hard-working people, and the latter who get more respect. Smart people are a dime a dozen around here; you can’t swing a cat (why would you swing a cat?) without hitting one. Most folks are also hard-working, but there are some who really stand head-and-shoulders above when it comes to work ethic.

MIT definitely gets a high level of performance out of its students. The Institute’s basic method for this is as follows:

  1. Recruit the best
  2. Work them like dogs
  3. Provide safety nets

Most people wouldn’t expect the third one. MIT provides a lot of safety nets for its students, from summer programs for underprepared students to “recovery” courses to tutoring groups to special-interest dorms and affinity groups. If someone claims that the institute ignored them, they probably didn’t reach out. There’s no doubt that the workload is intentionally overwhelming, but there is support – free support! – for people who are willing to make the time for it.

When I was a high-school student, I wanted to go to MIT for college. It wasn’t my greatest dream or anything, but it was an aspiration. I ended up at RPI instead, which was a good school for me (at least as an undergraduate – I shouldn’t have stayed for graduate school). In the end I was ok that I hadn’t gone to MIT, because I heard about the level of competition and stress that pervades the institute, and I had enough difficulty dealing with stress as it was. I think I could have handled it here, but I don’t think I would have been happy. I’m glad when I see students who are happy here, because I know that the dream of making the world a better place is alive in them as well.

Notes from LINC

This year I went to the LINC conference at MIT. This year’s theme was “Realizing the Dream: Education Becoming Available To All. Will the World Take Advantage?” I was officially there to present a paper about our course, but also to learn about other parts of education. I love conferences; you get such a mixing of ideas. Sadly I had a lot of work to do while I was there (this was in the early days of the online course, where more attention was needed) so I didn’t get to see as many talks as I would have liked.

The parts of the conference I did get to see were a mixed bag.

On the positive side, I saw a lot of efforts to improve education and make it more accessible. It was also great to see that this was coming from many different places in the world. For example, there was actually a talk where an audience member had to ask what a lahk was, because all the numbers in the talk were in that unit.

On the negative side, I saw a lack of actual research results. People were talking about the changes they had made, but rarely about the measurable impacts of those changes. Those who did have results rarely talked about them in the language of statistics.

One really excellent talk was at lunch the second day, from Cliff Missen of (among other things) the eGranary project. Missen started off the talk by saying that online courses are great when you can worry about whether or not people have internet access, but he more often has to worry about whether they have water. It really put things in perspective for me.

Overall I preferred the AAPT conference, but LINC’s more international view was definitely something I appreciated.

Support Staff

If you’re going to run an online course, the first, most important thing that you need, and I cannot stress this enough, is to have a team.

Running an online course can take as much of your time as you let it. I’ve seen people try to run classes ten times smaller than ours (in terms of enrollees) who ended up with no free time. They spent not just most of their work day, but hours every night and most of their weekend trying to handle the course. Those folks had a small team (or no team), and were working just a few days ahead of the deadlines. We didn’t do that this summer. By having a strong team and having the material ready before the course started, we had a stronger course and I had most of my weekend free.

I think of the organization as sort of a pyramid structure (though the bottom layer isn’t actually the largest). At the bottom of the pyramid were Community TAs, then Master Teachers, then On-Campus Staff, then me at the top.

The Community TAs were recruited from the course as it ran. This is a common practice in edX courses. We did two recruitment drives, one in the second week and one later on. The TAs have no official obligations except to be professional and courteous. Most of them are the kind of folks who would be contributing a ton to the discussions anyway. They get little green badges around their names in the discussion boards, so that people can tell who they are. They have some powers to moderate the discussion boards, such as editing or deleting others’ posts, and they had access to the system that we used to track issues. We had about 15-20 of them by the end of the course.

Our Master Teachers were a stroke of genius on the part of one of my co-workers. What we did is e-mail the top 15% of our students from last year’s course and say, essentially, “Hey, you folks did awesome last year. We’re running the course again this year and if you have 20 hours to help out over the course of the summer, we’d love to have help. What can we do to make it worth your time?” And from that we got 30 people, many of whom were public school high school teachers (thus the name). The most common thing they asked for was Continuing Education Units, which we were glad to provide.

I cannot say enough about how awesome these folks were to us, day in and day out. Many donated far more than 20 hours of their time in the discussion forums. They helped us scope out the unreleased material for bugs and typos. They were supportive and friendly, and I would never want to run a course without this kind of help.

Our On-Campus Staff consisted of four people: one high-school intern, two undergraduates, and a postdoc. The intern checked through the entire course from front to back, and caught a ton of errors. The undergrads helped out on the forums, but their main job was handling issues that were reported through our tracking system. The postdoc was my right-hand woman, and she tackled just about everything, from checking the course to writing new problems to being our Spanish voice on the forums. These folks had two major things that separated them from the rest of the course staff: First, they were paid full-time or part-time employees, and second, they had access to change any item in the course. They could actually make changes to the text or the homework problems.

I was the Course Coordinator. My primary job was executive decision-making. I handled issues that no one else felt they had the authority to respond to, or alterations to the course structure (removing bad problems, swapping the order of things, etc.). I also spent time on the forums and checked the problem tracker, but that’s because I had time, not because I was needed to handle day-to-day items. I could spend my time on higher-level considerations and weighing what was best for the course.

The setup might not have been ideal, but it was very good. People with  If I were to run another online course, I’d definitely try to have this sort of setup again.

Demographic Divide

Physics has a few issues in terms of demographics. Look at any college department and you’ll see it.

  • There are almost no poor students.
  • There are very few women, either as students or as faculty.
  • Most students and faculty are white, asian, or indian. Other ethnicities are much less common.

Online education is often touted as “education for everyone” or “education for the masses.” Unfortunately, the demographic divide remains.

  • Every student could afford an internet connection, which in many countries is a dear expense.
  • Our course had only 17% female students. The percentage was fairly constant across countries. Women are being repelled from physics in droves, everywhere people speak English (and likely elsewhere). I won’t speculate on why; there’s plenty of solid research on that.
  • The edX intro survey asks about race and ethnic group,  but no one has analyzed it for minority status yet. The fact that we reach other countries does not mean that we reach minorities in those countries. Let’s say “not enough data” here.

It seems that simply providing education online (or in any other forum) isn’t enough to break down barriers. It’s clear that online education is not “education for everyone” yet, but it also seems that many of the barriers are put in place at younger ages. If an online physics course attracts roughly the same low percentage of women as an on-campus physics course, we must be doing something wrong before students even get to that point.

Job Prep

What I learned as a high school student:

  • How to write well
  • How to edit other peoples’ work
  • How to prioritize my time
  • How to interact with people
  • How to do basic calculus
  • How to solve a few well-known physics problems
  • Work ethic and responsibility

What I learned as an undergraduate in physics:

  • To start my homework the second it’s assigned
  • How to use basic matrices
  • How to solve a large variety of well-known physics problems
  • A tiny amount of lab work
  • How to oversee people working toward a common goal

What I learned as a graduate student in physics:

  • How to grade two hundred labs a week
  • How to solve more complex well-known physics problems
  • How to do tensor calculus
  • How to teach courses
  • How to carry out research

What physics professors do:

  • Propose and carry out research
  • Oversee graduate students
  • Run a lab
  • Write grants
  • Serve on committees
  • Design and teach courses
  • Write papers
  • Review papers


  1. My high school education did as much to prepare me for a professorial job as my graduate and undergraduate education combined.
  2. I learned as many useful things outside of my coursework as I did inside.

Depth and breath of physics knowledge is a worthy prerequisite for being a professor. Unfortunately, it is a “necessary but not sufficient condition” for doing well as one. If we’re not preparing our physics majors to be professors, what are we preparing them for?

That’s a genuine question, not a rhetorical one. If you can spot the job for which physics students are ideally prepared with their current curriculum, not counting electives, I’d love to hear what it is.

Not every physics major becomes a professor, nor should they. However, given that it’s the current “expected route” for which we prepare people, it may be worth considering a revision to the undergraduate and especially graduate-level curricula.


No, not a spitball, I mean spitballing.

(Man, I can’t believe I just linked to Urban Dictionary. There goes my librarian cred.)

When I was working at Hyde last summer, I was given a small job (you know, one that could sit next to my big jobs and get them drinks when they needed it). I was asked to lead a brainstorm to figure out how to do study hall better. I had six people to work with, all other teachers. I knew that if we started with what we usually did and worked from there, we’d be more likely to end up with something that looked like what we had, so I decided to work from the opposite direction. So here’s what I passed on to the team:

“Pair up and grab a sheet of paper. Your job in the next 10 minutes is to come up with as many creative ideas as you can of how we could do study hall. You are only allowed to write down sensible ideas after you run out of ridiculous ones.”

And let me tell you, some of these ideas kept me chuckling for days. The idea of doing study hall on the football field, with the whole school lined up by weight, still cracks me up.

Once we got down to the more sensible ideas, however, they were really interesting. Study hall in the morning rather than the evening. Putting everyone in the dining hall and doing homework during an extended dinner period. Study hall right after (or before) each class period, with the teacher present. These folks came up with some really creative ideas, including the obvious “no study hall.”

That phase of spitballing – being willing to talk seriously about crazy ideas, or even admit that we have them – is too rarely seen. Many institutions are quick to throw out anything truly new. Even more rare is being willing to accept and try out such ideas; sadly, I think study hall ended up more or less as it had in previous years. I’m still glad that we spent time coming up with some bizarre ways to do homework, because it gives me hope that some day someone might actually try them.

Elbow Grease

One of my favorite aspects of the boarding schools at which I worked, one of which I went to myself, was the work program. When my wife and I were considering starting a school of our own, this was one of the key components I wanted. Here’s how it worked and why I liked it.

At both places, students have to put in some physical labor. At Hyde everyone has to work some time in the dining hall. Crews are organized by sports team, so one day you might have the football offense helping with lunch, and girls’ soccer helping with dinner. Jobs include dish room, sweeping, keeping the juice bar stocked, scrubbing pots and pans, etc. No actual food service or cooking; that’s left to the professionals. At NMH it’s a much broader program. Everyone has one job per trimester, which is about 45 minutes a day or a big 4-hour chunk on the weekend. Everyone seems to spend at least one term in the kitchen, but lots of students clean classrooms, work on the farm (yes, there’s a farm attached to campus), vacuum the dorms, and so forth. Many juniors and seniors also have the chance to work in other ways, such as being a band manager, running the computer lab or help desk, working at the pre-school, or being a student leader (more on that another day).

I liked this program even when I was a student. I’m sure I complained about it once in a while, but only in the “now I have to leave my friends and go do work” sort of way. I saw a lot of benefit in the work, and I especially saw benefit in the fact that everyone had to work. This wasn’t college-style work study that was part of a financial aid package. You couldn’t buy your way out. It was good work, it needed to be done, and you could see the positive effects.

Most of my work ethic came from my parents and especially my grandparents, but the rest of it came from a school that demanded both intellectual and physical work. It’s something I’m still proud of to this day.

The Laptop Program

I mentioned the other day, in talking about online high schools and Idaho’s plan for online courses, that my alma mater ran a laptop program while I was there. A fellow alum suggested that I write about that, so why not?

They started the mandatory program in 1999 – every freshman had to have a laptop, and the school offered one at fairly steep discount (especially considering the software on it; these days it includes Maple, Photoshop, a CAD program, and a few other mid-to-high-cost items). What drove the idea is not something I’m going to get into; there are probably as many motivations as there were people pushing for the initiative, and as many counter-arguments as there were people pushing back. Either way, it happened, and so what the faculty did with it is more important at this point.

Adoption in the freshman courses, especially the more technically oriented ones, was fairly good. The physics program switched over its MBL design to the laptops rather than the existing (frankly ancient) desktops. Math recitations were computer-oriented before this even started, so the change was fairly easy. I can’t speak to the engineering courses personally, never having taken them, but I heard that many of the intro courses used the laptops well. Humanities and social science courses were slower to adapt. I assume they faced little pressure to do so. It was an engineering school, after all.

Where the program suffered was in the upperclass years. Computer  usage in sophomore-level classes was scattered. Professors in Junior- and Senior-level courses ended up saying things like “Oh that’s right, you all have laptops now.” Graduate courses? Forget it. Building technology into the curriculum of the institute as a whole seems not to have been a major goal; if it was, it’s one that was never achieved or even approached while I was there.

In terms of outside reputation, the laptop program was effective. Newspaper articles were written, the school showed up on “most-wired” lists, and the school’s PR umbrella generally expanded. Student responses to opinion surveys of the system were generally favorable. In terms of educational effectiveness, there seems to be little evidence either way. I’m fairly sure there was no consideration at the institutional level of doing any educational effectiveness studies on this matter. I was involved with the physics education research group and I hope that we would have heard about something like that if it happened.

Summary: In terms of providing a low-cost and fairly decent laptop, replacing some of the older computer labs, and getting the school some press, I have to say that the program seems to have worked very well. In terms of educational impact, I’m not sure there was any.


For those interested in how sausage is made (or just finding out the name of the school):