Would You Want Your Students’ Essays Graded by Computers?


NPR recently did a report on “robo-grading” of student essays via computer for standardized tests and other constructed responses. It’s a growing field in which proponents have touted advancements in artificial intelligence and savings of time and money.

Developers of so-called “robo-graders” say they understand why many students and teachers would be skeptical of the idea. But they insist, with computers already doing jobs as complicated and as fraught as driving cars, detecting cancer, and carrying on conversations, they can certainly handle grading students’ essays.


He writes,

…but the basic problem, beyond methodology itself, was that the testing industry has its own definition of what the task of writing should be, which more about a performance task than an actual expression of thought and meaning. The secret of all studies of this type is simple– make the humans follow the same algorithm used by the computer rather than he kind of scoring that an actual English teacher would use. The unhappy lesson there is that the robo-graders merely exacerbate the problems created by standardized writing tests.

The point is not that robo-graders can’t recognize gibberish. The point is that their inability to distinguish between good writing and baloney makes them easy to game. Use some big words. Repeat words from the prompt. Fill up lots of space. Students can rapidly learn performative system gaming for an audience of software. And the people selling this baloney can’t tell the difference themselves. That’s underlined by a horrifying quote in the NPR piece. Says the senior research scientist at ETS, “If someone is smart enough to pay attention to all the things that an automated system pays attention to, and to incorporate them in their writing, that’s no longer gaming, that’s good writing.” 

In other words, rather than trying to make software recognize good writing, we’ll simply redefine good writing as what the software can recognize.

Computer scoring of human writing doesn’t work. In states like Utah and Ohio where it is being used, we can expect to see more bad writing and more time wasted on teaching students how to satisfy a computer algorithm rather than develop their own writing skills and voice to become better communicators with other members of the human race. We’ll continue to see year after year companies putting out PR to claim they’ve totally got this under control, but until they can put out a working product, it’s all just a dream.

He’s right.

You just can’t automate voice.

Makes one think if this is the direction for North Carolina on a large scale because there are many in Raleigh who do not want people to develop voice.

One thought on “Would You Want Your Students’ Essays Graded by Computers?

  1. Six years ago, one of our assistant principals having attended a summer workshop for administrators, approached me with an offer from “Robo-grading” program (online) that was in testing. Would be interested in beta-testing the system and then talking with the developers.

    I said yes.

    After six months, I quit explaining that other than counting words as the means of determining if a paragraph was sufficiently developed, spelling, punctuation, and few simple style elements, the program evaluation was useless.

    It could tell the difference between active and passive voice , but had no idea which mode was appropriate to the the subject/ genre. The same was true of lexile.

    But when it came to determining the most advantageous POV, subtleties of irony, or humor, the program was “tone deaf.”

    The highest values were awarded to utterly wooden, formulaic written expression devoid of any insightful thinking.

    When I attempted to explain this to a student who received the grade, my explanation was met with hurt feelings because the teacher was overriding the A with a C-.

    That was not an experience I wanted to repeat.

    Has anyone looked to how much money Pearson is giving to the legislators supporting mechanical grading. I suspect therein lies another story.


Comments are closed.