Les Perelman’s “Construct Validity, Length, Score, and Time in Holistically Graded Writing Assessments: The Case Against Automated Essay Scoring (AES)”



Perelman, Les. “Construct Validity, Length, Score, and Time in Holistically Graded Writing Assessments: The Case Against Automated Essay Scoring (AES).” International Advances in Writing Research: Cultures, Places, Measures. Ed. Charles Bazerman, Chris Dean, Jessica Early, KAren Lunsford, Suzie Null, Paul Rogers, and Amanda Stanswell. Fort Collins: WAC Clearinghouse and Parlor, 2012. 121-32. Web.



Perelman makes four points regarding automated essay scoring systems and why they should not be used to score student writing. First, the scores mostly assess the length of the writing piece rather than actual elements as it simply counts. Then, because the timeframe for writing these timed pieces is so short, length correlates with score. These timed essays are abnormal not only because of the correlation between length and score, but because they do not have authentic prompts. Next, the validity of AES scoring has been proven to be false. Last, MS Word has better grammar checking software than AES, and MS Word is very limited itself (128).

Works Cited

Attali, Y., & Powers, D. (2008). A developmental writing scale (ETS Research Report RR-08-19). Princeton, NJ: Educational Testing Service.

Council of Writing Program Administrators, National Council of Teachers of English, and National Writing Project. (2011). Framework for Sucess in Post- secondary Writing. Retrieved from http://wpacouncil.org/ les/framework- for-success-postsecondary-writing.pdf

Elliot, N. (2005). On a scale: A social history of writing assessment in America.New York: Peter Lang.

Huot, B. (2002). (Re)articulating writing assessment. Logan, UT: Utah State University Press.

White, E. M. (1984). Holisticism. College Composition and Communication, 35(4), 400-409.



  • “Although White (1995) has made a case for the timed-impromptu for certain assessment decisions, it is a genre of writing that has no real analogue in real human communication and therefore is invalid as a measure. Indeed, the timed impromptu exists in no activity system except for mass-market writing assessments and education geared towards mass-market writing assessments.” (Perelman 122)
  • “When students have one hour to write, the shared variance predicted by length decreases to approximately 20%, and when students are given 72 hours, length predicts 10% or less of the shared variance of the holistic score.” (Perelman 124)
  • “They do not understand meaning, and they are not sentient. They do not react to language; they merely count it.” (Perelman 125)


If the grammar checking software is as limited as Perelman claims it is, how can it truly assess students’ grammar usage? How many students have been rated lower because of the limitations of its software?

Perelman explains that in 1966, AES emerged because human scorers were not reliable. What was first tried to make human scorers more reliable? Did norming sessions occur?

I have seen World War II referenced in the history of composition in a few articles. What happened during or just after World War II to make composition change as much as it did? Was there a paradigm shift during this time? How did World War II affect other disciplines?


“The Rise of Technical Writing Instruction in America” Robert J. Connors

Connors, Robert J. “The Rise of Technical Writing Instruction in America.” Ed. Stuart A. Selber. Central Works in Technical Communication. Ed. Johndan Johnson-Eilola. New York: Oxford UP, 2004. 3-19. Print.

History of Technical Writing

Early Years (1895-1939): relatively new field being taught in college; Civil War had more people going to college–especially for engineering–that merited these courses; English teachers were soon to be in demand, but they felt they needed to “humanize” engineers, and so they received literary instruction along with writing instruction; no cooperation between English and engineering departments and it becomes more severe as time wears on

A Discipline Comes of Age (1940-1980): “engineering English” was gone from academia during WWII 1940-46; technical writing was an actual job because of its demand during WWII;  rhetoric and technical writing marry in late 1950’s; “reader-writer relationship” p 14;  other disciplines take advantage of technical writing; still, people who taught techwrit still wanted to teach lit because that was their background; 1970s finally found ground for technical writing as its own field

“Sypherd complained bitterly that literature courses were too few and too little to matter, that freshman courses were ineffective, and that lack of writing in other engineering courses, bad student attitudes, and no interdepartmental cooperation had brought engineering English to a critical pass.” p 11
“During the fifties the importance of the profession of technical writing became apparent to industry, and colleges gave more serious consideration to turning out trained technical writers.”  p 13
“Technical writing teachers were not always rewarded by their departments, but many found freedom and credit in the 1970’s that had previously only been dreamt of.” p 17
Can the digital humanities study this timeline and make any inferences as to its own fate?
What major event on the scale of World War II could make the digital humanities boom by need?
Will or is technical writing its own department at some colleges and universities?
Works Cited
H.E. Hand, An Attempt to Measure Success in Technical Writing, Proceedings, ASEE, 72,  pp. 70-72, 1964.
J.A. Walter, Confessions of a Teacher of Technical Writing, The Technical Writing Teacher, 1,  pp. 5-6, 1973.
J.H. Wilson, Jr., Our Colleges Can Teach Writing–If They Are Made To, Proceedings, ASEE, 62,  pp. 431-435, 1955.

“The Status of Composition and Rhetoric in American Colleges, 1880-1902: An MLA Perspective” Donald C. Stewart


Stewart, Donald C. “The Status of Composition and Rhetoric in American Colleges, 1880-1902: An MLA Perspective.” Norton Book of Composition Studies. Ed. Susan Miller. First ed. New York: W.W. Norton, 2009. 129-140. Print.


Attitude that teaching composition is wearisome and not worth professors’ while. p 129

Some considered composition and/or rhetoric boring and just about grammar and spelling, while others considered one of the most important subjects as it teaches people to speak and write intelligently.

Bizarre that Pedagogical Section of MLA was removed because it wanted writing to be “natural and sincere, not phony and artificial” (138). Awesome!

MLA presidents from 1899-1904 were of the mind that the English is a living, breathing language, and that there were several misconceptions about the language p 132

Pro-teaching of rhetoric teachers advised teaching the history of rhetoric, history of English syntax, different ways to use language and how expert writers use it, and history of literary criticism. p 134

Committee’s report on The Century’s experiment on how test argument validity in writing had three findings: (1) there is much room to wander in unchartered territory, (2) the report was faulty, and (3) you must teach composition and not hope that students will absorb what they read: they need “structure.” p. 135-136

Teachers adapt their methods to that of their students’ abilities. This would include how and what is taught at the elementary, secondary, and post-secondary levels. p. 137


“The association [MLA] was most certainly not a teacher’s agency nor was it centrally concerned with pedagogical problems.” p 132 James Bright’s papers.

“After reviewing the many responses they received [from the survey], the committee concluded that “Rhetoric” should be defined much more broadly than as ‘practical composition’ and ‘that the field thus opened will afford abundant opportunity for investigation by the serious student.'” p 135


What sections are current with the MLA?

What are the attitudes towards these findings in today’s MLA membership?

How does MLA and NCTE differ in their fundamental beliefs?

Works Cited

Hart, James Morgan. “The College Course in English Literature, How It May Be Improved.” PMLA os (1884-85): 84-95.

Mead, William E. “Conflicting Ideals in the Teaching of English Composition.” Report of the Pedagogical Section of the MLA, Proceedings of the MLA (1902): vviii-xxiii.

“Where Do English Departments Come From?” William Riley Parker


Parker, William Riley. “Where Do English Departments Come From?” Norton Book of Composition Studies. Ed. Susan Miller. First ed. New York: W.W. Norton, 2009. 3-16. Print.



  • English comes from a “broken home” of its parents Public Speaking and Linguistics.
  • English studies are older than the teaching of English and date back to the Renaissance and Reformation.
  • Cambridge didn’t have a professor of Anglo-Saxons until 1878 (empty position for ~230 years) and no literature professor until 1911.
  • The notion of English studies is fairly recent and is the convergence of English literature, linguistics, English grammar, elocution and oratory skills,
  • UK English ed adopted modern Eng. much more readily than US
  • John Hopkins university tried to bring specialization of teachers to education after it opened in 1876. Before then, there were no teachers specifically trained for English as there were for history per se.
  • MLA was founded for several reasons: change in attitudes towards traditional education p 11
  • English departments “overreached” themselves to take up unclaimed spaces between departments.
    • Because of this, we (English) have never bothered to stop and determine what we are and are not.
  • How professors teach the upper level grad courses or speciality courses and grad students teach comp…yeah, that started in the 1890s 



“So let us begin with the recognition of a simple fact: the teaching of English, as a constituent of college or university education, is only about 100 years old, and departments of English are younger still.” p. 3

“Although Emerson’s famous ‘American scholar’ address was delivered in 1837, it is important to remember that this was not a time that produced in America and literary or linguistic scholarship of real substance, and the professor of English language and literature did not immediately emerge. In the United States before 1860 only a very few colleges ventured to mention English literature as a subject in their catalogues or announcements.” p. 8

“But these facts do not add up to the conclusion that the professor of rhetoric and oratory should disappear, to be supplanted by the teacher of English language and literature.” p 12. Talking about composition classes and their popularity, who was deemed most qualified to teach them.

“You may think me unfitted to be a chairman when I say, now, that the history of our profession inspires in me very little respect for departments of English; their story is one of acquisitiveness, expediency, and incredible stupidity. I care a lot about liberal education, and I care a lot about the study of literature in English, but it seems ot me that English departments have cared much less about liberal education and their own integrity than they have about their administrative power and prosperity.” p 15


Where will the English department go in the future? Will mini-departments form and deviate from “English” much like composition studies has at some schools?

Why are the effects of literary studies methodology, scientific linguistics, and how liberal education was to be run different between the US and the UK? Does it remain in today’s educational cultures?

p 7-8 talks about how Cambridge and Oxford graduates had to subscribe to the Thirty-nine Articles to prove their allegiance to the Church of England, but why? What consequences were they if they didn’t or did not remain? Since there were many non-conformist institutions established soon after, I am guessing that the consequences were not awful.

Rhetoric & DH Replacing Computer Classes in K12

Douglas Walls’ chapter “In/Between Programs: Forging a Curriculum between Rhetoric and the Digital Humanities” had me wondering how this could translate in K12 classrooms.

At my high school, we no longer require or offer an introductory computer class because administration and staff assume that these “digital natives” know enough to consider hem digitally literate. When I started high school in 2005 (shh…don’t judge me), computer classes were no longer required but were still there. What we are finding is that there is still a divide between students and their abilities to manipulate technology. Cue digital humanities and rhetoric…

I think that DH could have its place in a “computer” class in K12 schools. Students do not need to learn how to type X amount of words per minute if they are engaging with programs that include this, but do not necessarily focus on speed. Besides, not all jobs exactly care about that anymore, or typing is something that can be acquired. What students could really stand to gain is how to be creative and mindful when making choices in using technology.

The easiest way would be to start with social media and analyze the rhetoric behind others’ profiles and pages while simultaneously exploring ways to set up students’ pages and profiles. After listening to my peers’ book reviews, I know there is much research behind this and many directions a teacher may go.

After getting an idea of rhetoric and a brief introduction to DH, students could move into disciplinary projects that would help them to understand what writing looks like in that discipline or what it could look like.


The only thing that would be difficult, I think, is how the DH teacher would collaborate with the disciplinary teachers to determine what genres would be appropriate. Initially, I think it would launch well, but I wonder how it would continue down the road as it needs updating. Like Walls, I also wonder if certain teachers would seek to get more screen time on their projects. Regardless, I think this would be worth looking into for schools who want to incorporate technology classes. but not refer back to the traditional computer classes.

Learning How to Learn

The texts from last class make a serious argument for why all students should learn how to code, but I honestly cannot see how or why I would include coding regularly into my classroom.

What I do value in my classroom is the question, “What is writing?” but I do not think that having my students code for a project would be the best use of our very limited amount of time and resources. What I wish I had time for is “learning how to learn,” which can be fostered in learning coding.

While I don’t see its relevancy to my classroom, I think it has been a humbling experience as a teacher. I am positive that my students have to struggle with class material from time to time, and getting to be back in that position again is good for me (but why does it have to be coding?!).

The Creators Project website has poetry written in HTML code, considering it its own “language.” “Ishac Bertran saw this linguistic beauty in code and sought to treat creative coding text as what he believes it is: poetry,” which I completely agree with. Again, I just  do not see its place in my English classroom. If it is its own language, then I feel it should have its own class. Many public schools are cutting computer classes believing that these “digital natives” coming into secondary schools do not need it. If coding is its own lanuage, then we could treat it as we do Spanish or French to give it its own amount of time.

Like Meg said, DH needs its own place rather than tapping into different disciplines.




Archiving To Kill a Mockingbird

My biggest struggle in this class and with DH (whatever it is right now) is how can I use this in my class of 14-15 year old students. Archiving is something that I could use with my students, much like Rice and Rice describe in their chapter.

I am thinking of framing the archive in this way:

Purpose: to inform your audience of how To Kill a Mockingbird is relevant today by finding the theme in today’s life and metaphors for characters and events.

Audience: next trimester’s students who will then discuss ethos, pathos, and logos in the videos, pictures, articles, etc. in the archive as a pre-reading activity.

Since To Kill a Mockingbird has this “timeless” quality to it, it would be interesting to see if over time, the things students associate with it also have temporal endurance. I think that the archive they build will also show the book’s temporal endurance, and having students argue the rhetoric behind the archive contributions would be a great experience for them. It’s more than words on page, though that will still be a component of the project.

I definitely want students to be conducting interview with people, especially regarding the themes behind the book. My goal is that students do not go and look for things that have already been associated with TKAM, but to make new connections. For example, students would be interviewing people about courage and then arguing the connections between the interview and TKAM.

I am so excited about this project, and have my work cut out for me. I need to figure out a platform that could sustain a longterm project, has enough storage, and is still easy enough for students to figure out how to edit the pages. I also need to figure out how I want to organize the archive.

Another awesome thing about this project: I have the other 10B teachers on board, like really supportive with this project.

Privacy Has Its Place

**Happy snow day to me! I’m actually getting two days worth of free days from work AND school!**

Last week’s readings about cultural rhetoric threw me back into my Native American literature classes (hey, Michelle!). One of the things we often talked about was how tribes’ traditions have been stomped on repeatedly by outsiders, watered down into simple trinkets (think dreamcatchers), or secret rites have been exposed.

I get that in some instances spreading traditions around to the general public is a good thing. By educating people outside the tribe, they are less ignorant to what goes on inside the tribe, generating a culture of understanding (ideally). In Withey’s presentation “On Not Looking: Ethics and Access in the Digital Humanities, she touches on the fine lines and the detailedness that DHists need to be mindful of. In her own experience, she talks about how she and her team work with an indigenous group in Australia to share some of their values and traditions.

She notes that they come to their exhibit (how odd to have an exhibit on yourself) to tape over tribe members’ faces that are deceased since posting the pictures. They also dictate what can be posted online or how large the pictures can be. The relationship between this group of people and the DH team seems to be a good one: they communicate what can or cannot be shared, how the site content should look and how it operates, and more. However, we are not too naive to know that this is the best case scenario.

I always wondered if there is a “code of conduct” for anthropologists, and this seems like one of those cases that would merit a code for DHists too. DH is such a new field that it is worrisome that someone who calls himself or herself a digital humanist might screw up the field’s reputation by not abiding by the code or invading an indigenous group’s privacy. I guess I shouldn’t be too worried about it since obviously other academic fields have had to navigate this before, but if you look into some Native American departments, they are managed by white people who only know the academic side. Some have misconceptions of their own studies because they only stick to the academic side of NA lit because they cannot really truly know the culture from reading a white person’s perspective of a tribes’ traditions.

That also brings up the question: who is qualified to look at these groups of people? Are people within the groups the ones who should be able to speak about their cultures? Or to what extent should others get to speak on their behalf without it getting to be a long chain of telephone with incorrect deliverances?

I have so many questions that I feel are going to nitpick the boundaries, especially when I start comparing it to other academic fields. It


Be cautious, DH friends.

According to Whom: Defining Digital Humanities

Disciplines need a home. But what do you do with the ones that do not have a home?

Why can’t disciplines get along? And take from each other?

And then disciplines get away from rhetoric


(my sad attempt at figuring out where DH fits in the sign system and this “third space”)

I had so many questions when reading Alexander Reid’s chapter “Digital Humanities Now and the Possibilities of a Speculative Digital Rhetoric.” I’m not sure why, but this one frustrated me the most because I didn’t know what digital humanities was or speculative theory or how everything fits together.

I did get that the humanities needs to be rethought, so I guess I should figure out what humanities are considered now. Steve’s general test of whether or not a discipline is a humanity rings true for most: if it needs a stats class, then it is not a humanity.

After deciding which is which, Reid offers that the digital humanities can be identified only by using methods that math and science cannot support. The issue that is in flux at the moment is “the identification of cultural objects and practices as knowable only through a limited set of humanistic methods” (16).

Everyday advocacy

Changing the Narrative about Literacy Education

Teachers, Profs, Parents: Writers Who Care

A blog advocating for authentic writing instruction

charles french words reading and writing

An exploration of writing and reading

Citizen Teacher

education for a more humane & democratic society

Research in Theory and Practice of Writing

English 621 at Eastern Michigan University

Thomas Passwater

Putting a queer shoulder to the wheel


rhetoric is everywhere

Teaching Composition: Breaking Out of Traditions

Laura Kovick - Eastern Michigan University - lkovick@emich.edu

Thinking In Play

Playing with big ideas

Broke-Ass Parents

Raves, rants, tips, and stories about our life as broke parents.