Les Perelman’s “Construct Validity, Length, Score, and Time in Holistically Graded Writing Assessments: The Case Against Automated Essay Scoring (AES)”

 

Citation

Perelman, Les. “Construct Validity, Length, Score, and Time in Holistically Graded Writing Assessments: The Case Against Automated Essay Scoring (AES).” International Advances in Writing Research: Cultures, Places, Measures. Ed. Charles Bazerman, Chris Dean, Jessica Early, KAren Lunsford, Suzie Null, Paul Rogers, and Amanda Stanswell. Fort Collins: WAC Clearinghouse and Parlor, 2012. 121-32. Web.

 

Summary

Perelman makes four points regarding automated essay scoring systems and why they should not be used to score student writing. First, the scores mostly assess the length of the writing piece rather than actual elements as it simply counts. Then, because the timeframe for writing these timed pieces is so short, length correlates with score. These timed essays are abnormal not only because of the correlation between length and score, but because they do not have authentic prompts. Next, the validity of AES scoring has been proven to be false. Last, MS Word has better grammar checking software than AES, and MS Word is very limited itself (128).

Works Cited

Attali, Y., & Powers, D. (2008). A developmental writing scale (ETS Research Report RR-08-19). Princeton, NJ: Educational Testing Service.

Council of Writing Program Administrators, National Council of Teachers of English, and National Writing Project. (2011). Framework for Sucess in Post- secondary Writing. Retrieved from http://wpacouncil.org/ les/framework- for-success-postsecondary-writing.pdf

Elliot, N. (2005). On a scale: A social history of writing assessment in America.New York: Peter Lang.

Huot, B. (2002). (Re)articulating writing assessment. Logan, UT: Utah State University Press.

White, E. M. (1984). Holisticism. College Composition and Communication, 35(4), 400-409.

 

Quotations

• “Although White (1995) has made a case for the timed-impromptu for certain assessment decisions, it is a genre of writing that has no real analogue in real human communication and therefore is invalid as a measure. Indeed, the timed impromptu exists in no activity system except for mass-market writing assessments and education geared towards mass-market writing assessments.” (Perelman 122)
• “When students have one hour to write, the shared variance predicted by length decreases to approximately 20%, and when students are given 72 hours, length predicts 10% or less of the shared variance of the holistic score.” (Perelman 124)
• “They do not understand meaning, and they are not sentient. They do not react to language; they merely count it.” (Perelman 125)

Questions

If the grammar checking software is as limited as Perelman claims it is, how can it truly assess students’ grammar usage? How many students have been rated lower because of the limitations of its software?

Perelman explains that in 1966, AES emerged because human scorers were not reliable. What was first tried to make human scorers more reliable? Did norming sessions occur?

I have seen World War II referenced in the history of composition in a few articles. What happened during or just after World War II to make composition change as much as it did? Was there a paradigm shift during this time? How did World War II affect other disciplines?

“The Rise of Technical Writing Instruction in America” Robert J. Connors

Citation

Connors, Robert J. “The Rise of Technical Writing Instruction in America.” Ed. Stuart A. Selber. Central Works in Technical Communication. Ed. Johndan Johnson-Eilola. New York: Oxford UP, 2004. 3-19. Print.

Summary

History of Technical Writing

Early Years (1895-1939): relatively new field being taught in college; Civil War had more people going to college–especially for engineering–that merited these courses; English teachers were soon to be in demand, but they felt they needed to “humanize” engineers, and so they received literary instruction along with writing instruction; no cooperation between English and engineering departments and it becomes more severe as time wears on

A Discipline Comes of Age (1940-1980): “engineering English” was gone from academia during WWII 1940-46; technical writing was an actual job because of its demand during WWII;  rhetoric and technical writing marry in late 1950’s; “reader-writer relationship” p 14;  other disciplines take advantage of technical writing; still, people who taught techwrit still wanted to teach lit because that was their background; 1970s finally found ground for technical writing as its own field

Quotations

“Sypherd complained bitterly that literature courses were too few and too little to matter, that freshman courses were ineffective, and that lack of writing in other engineering courses, bad student attitudes, and no interdepartmental cooperation had brought engineering English to a critical pass.” p 11

“During the fifties the importance of the profession of technical writing became apparent to industry, and colleges gave more serious consideration to turning out trained technical writers.”  p 13

“Technical writing teachers were not always rewarded by their departments, but many found freedom and credit in the 1970’s that had previously only been dreamt of.” p 17

Questions

Can the digital humanities study this timeline and make any inferences as to its own fate?

What major event on the scale of World War II could make the digital humanities boom by need?

Will or is technical writing its own department at some colleges and universities?

Works Cited

H.E. Hand, An Attempt to Measure Success in Technical Writing, Proceedings, ASEE, 72,  pp. 70-72, 1964.

J.A. Walter, Confessions of a Teacher of Technical Writing, The Technical Writing Teacher, 1,  pp. 5-6, 1973.

J.H. Wilson, Jr., Our Colleges Can Teach Writing–If They Are Made To, Proceedings, ASEE, 62,  pp. 431-435, 1955.

“The Status of Composition and Rhetoric in American Colleges, 1880-1902: An MLA Perspective” Donald C. Stewart

Citation

Stewart, Donald C. “The Status of Composition and Rhetoric in American Colleges, 1880-1902: An MLA Perspective.” Norton Book of Composition Studies. Ed. Susan Miller. First ed. New York: W.W. Norton, 2009. 129-140. Print.

Summary

Attitude that teaching composition is wearisome and not worth professors’ while. p 129

Some considered composition and/or rhetoric boring and just about grammar and spelling, while others considered one of the most important subjects as it teaches people to speak and write intelligently.

Bizarre that Pedagogical Section of MLA was removed because it wanted writing to be “natural and sincere, not phony and artificial” (138). Awesome!

MLA presidents from 1899-1904 were of the mind that the English is a living, breathing language, and that there were several misconceptions about the language p 132

Pro-teaching of rhetoric teachers advised teaching the history of rhetoric, history of English syntax, different ways to use language and how expert writers use it, and history of literary criticism. p 134

Committee’s report on The Century’s experiment on how test argument validity in writing had three findings: (1) there is much room to wander in unchartered territory, (2) the report was faulty, and (3) you must teach composition and not hope that students will absorb what they read: they need “structure.” p. 135-136

Teachers adapt their methods to that of their students’ abilities. This would include how and what is taught at the elementary, secondary, and post-secondary levels. p. 137

Quotations

“The association [MLA] was most certainly not a teacher’s agency nor was it centrally concerned with pedagogical problems.” p 132 James Bright’s papers.

“After reviewing the many responses they received [from the survey], the committee concluded that “Rhetoric” should be defined much more broadly than as ‘practical composition’ and ‘that the field thus opened will afford abundant opportunity for investigation by the serious student.'” p 135

Questions

What sections are current with the MLA?

What are the attitudes towards these findings in today’s MLA membership?

How does MLA and NCTE differ in their fundamental beliefs?

Works Cited

Hart, James Morgan. “The College Course in English Literature, How It May Be Improved.” PMLA os (1884-85): 84-95.

Mead, William E. “Conflicting Ideals in the Teaching of English Composition.” Report of the Pedagogical Section of the MLA, Proceedings of the MLA (1902): vviii-xxiii.

“Where Do English Departments Come From?” William Riley Parker

Citation

Parker, William Riley. “Where Do English Departments Come From?” Norton Book of Composition Studies. Ed. Susan Miller. First ed. New York: W.W. Norton, 2009. 3-16. Print.

 

Summary

• English comes from a “broken home” of its parents Public Speaking and Linguistics.
• English studies are older than the teaching of English and date back to the Renaissance and Reformation.
• Cambridge didn’t have a professor of Anglo-Saxons until 1878 (empty position for ~230 years) and no literature professor until 1911.
• The notion of English studies is fairly recent and is the convergence of English literature, linguistics, English grammar, elocution and oratory skills,
• UK English ed adopted modern Eng. much more readily than US
• John Hopkins university tried to bring specialization of teachers to education after it opened in 1876. Before then, there were no teachers specifically trained for English as there were for history per se.
• MLA was founded for several reasons: change in attitudes towards traditional education p 11
• English departments “overreached” themselves to take up unclaimed spaces between departments.
  • Because of this, we (English) have never bothered to stop and determine what we are and are not.
• How professors teach the upper level grad courses or speciality courses and grad students teach comp…yeah, that started in the 1890s 

   

Quotations

“So let us begin with the recognition of a simple fact: the teaching of English, as a constituent of college or university education, is only about 100 years old, and departments of English are younger still.” p. 3

“Although Emerson’s famous ‘American scholar’ address was delivered in 1837, it is important to remember that this was not a time that produced in America and literary or linguistic scholarship of real substance, and the professor of English language and literature did not immediately emerge. In the United States before 1860 only a very few colleges ventured to mention English literature as a subject in their catalogues or announcements.” p. 8

“But these facts do not add up to the conclusion that the professor of rhetoric and oratory should disappear, to be supplanted by the teacher of English language and literature.” p 12. Talking about composition classes and their popularity, who was deemed most qualified to teach them.

“You may think me unfitted to be a chairman when I say, now, that the history of our profession inspires in me very little respect for departments of English; their story is one of acquisitiveness, expediency, and incredible stupidity. I care a lot about liberal education, and I care a lot about the study of literature in English, but it seems ot me that English departments have cared much less about liberal education and their own integrity than they have about their administrative power and prosperity.” p 15

Questions

Where will the English department go in the future? Will mini-departments form and deviate from “English” much like composition studies has at some schools?

Why are the effects of literary studies methodology, scientific linguistics, and how liberal education was to be run different between the US and the UK? Does it remain in today’s educational cultures?

p 7-8 talks about how Cambridge and Oxford graduates had to subscribe to the Thirty-nine Articles to prove their allegiance to the Church of England, but why? What consequences were they if they didn’t or did not remain? Since there were many non-conformist institutions established soon after, I am guessing that the consequences were not awful.