Reassessing the Rubric

. 2 comments.

What I’m about to share with you is probably an incredibly dangerous thing to do with assessment coordinators, MEBD pathway specialists, administrators, and others watching. Deep breath. Here goes.

My name’s Jeremy Fiebig. I teach theatre in the Department of Performing and Fine Arts and I’m a recovering rubric addict.

I think rubrics serve a useful function in a variety of teaching and administrative settings. In the past couple of years, I’ve played a small role in looking at and developing rubrics for writing, for an information literacy project, and for more than a handful of other projects and assignments, big and small, in my classes. I find rubrics, in many cases, to be useful tools that help clarify for me and for my students exactly what the expectations for a project are, how a given student performed against those expectations, and what that student’s grade should be given their performance. Rubrics save time for professors and administrators who need to score whatever’s being assessed in an efficient way. Rubrics provide some semblance of an objective standard, which is important. Rubrics enable you to really spend time considering what it is you’re looking for from students (or faculty members who are being assessed). Rubrics serve an important function in terms of assessment — they help provide a performance baseline so that grades, instead of just being grades, are able to be mined for meaning, for “closing the loop” in continuous improvement, and for providing evidence to administrators and legislators who need to see that we’re doing something. But it is far past time to admit we have a rubric problem. A big one.

Yes, rubrics are useful. Yes, they do some things really, really well. But there are some things they don’t do. And there are some things they do very poorly. And there are some ways in which they lie to us. And there are some ways in which they are abused. And there are some ways in which they are letting faculty members, students, and administrators off the hook when it comes to some of the core values of our institution and of the academy as a whole. Here are some examples:

1) Rubrics lose information. Rubrics are almost by definition narrow. A really, really full rubric might measure 6 or 8 or 10 criteria, which don’t necessarily provide a complete picture of whatever it is that is being assessed. Not everything gets measured.

2) Rubrics give the illusion of objectivity. Here’s the thing: our brains tell us that if the standards and criteria are in a rubric, that our measurement of student performance against that rubric is objective. And we can sleep better at night believing we’ve been fair and impartial. This is a lie. Just because the criteria are explicit and the standards are fair does not mean that professors or administrators are being explicit or fair about how they score things on the rubric.

3) Rubrics make us substitute parts for the whole. Let’s say you’ve just finished a meal and are thinking to yourself that you really, really liked it. Why? Because it tasted good? Because you were hungry? Because the plating of the food was exceptional? Because of the music playing in the background? Because of your dinner date? Because of your dietary preferences? Because it was a good value? Some combination of these things? If so, how much does does the plating score in your mind versus how hungry you were? How does taste score against value? Does service get a score? And how does that figure in?  When we think about our meal, we may be thinking of all of these things — some of them consciously, some unconsciously, but rarely do we bring a scorecard with us to dinner. When thinking about assessment, the rubric asks us to do precisely that: to predetermine whether and how we will evaluate the assignment. Not only do we potentially lose information here, but we substitute evaluation of parts of the assignment for the whole.

4) Rubrics don’t happen most places in the way they happen here. Rubrics do happen in real life. When I apply for a credit card or home loan, some computer or number cruncher somewhere scores my credit history against a rubric and assesses what kind of risk I am. Same with insurance. Same when the IRS determines whether I should be audited. There are hard-and-fast criteria and standards out there that we all have to deal with. And I can get a pretty good sense of what those criteria and standards are if I pay attention. Some of them are even legislated and regulated.  But some criteria and standards are not so published. If I’m about to hire you as an employee, you need to have experience, you need to meet some basic qualifications, and your references need to vouch for you (all of these are criteria — and therefore ARE, essentially, rubrics). These are things I can evaluate — maybe even somewhat objectively — and that you will know are part of the deal. But there are also a billion little assessments I make: your dress, your appearance, your hair, your vocal quality, your confidence, your spirit, your personal drama, your determination and drive, your language, your grammar, and so on. My grandfather, who was CFO of a company back in the 1980s, had a hidden criteria at all his interviews: if you salted your food before you tasted it, the interview was over (so to speak) and you weren’t getting the job. It can be argued that this was, in fact, an undisclosed rubric. Judgment is made on a host of things that are both consciously and subconsciously scored on rubrics — both those for which the standards are known and those which go on undisclosed, silent, or even unknown. In short, not everywhere we go on planet earth are the expectations clearly laid out before us — and when they are, our performance is not always evaluated in an open or objective way.  In the face of these realities, we still are compelled to figure out what we can, to do our best, and to try.  In the classroom, and in assessment, the use of rubrics is often couched in language about being fair, clear, forthright, and objective. My question is simply this: by using rubrics, are we training students to expect that they will be graded or judged in a way that will always be clear, fair, and forthright? Are we making students who simply “perform to the rubric” while disregarding some of the intangible and intrinsic and unspoken criteria at play all around us? Are we providing incentive for students not to think creatively, or to apply themselves, or to “figure it out on their own” with some of our rubrics?

5) Rubrics risk substituting measurement for judgment and instruments for expertise. We hire quality faculty because of their expertise, but then risk essentially pigeon-holing that expertise when it comes to rubrics — making faculty become, in essence, extremely overqualified graders and not experts who are in tune with the nuances of performance. While outcomes assessment has drifted more and more to the use of rubrics in recent years, the academy is of at least two minds when it comes to the use of rubrics. On the one hand, we use language about rubrics and assessment when we are assessing students, learning outcomes, and programs that suggests fairness, consistency, objectivity, and forthrightness. On the other hand, we shape our processes of tenure and promotion around experts sitting in a room making what we hope are informed judgments. Why do we treat the two forms of assessment differently? I don’t think the answer is necessarily sinister. We value both kinds of assessment — the role of instruments and the role of experts. Is it possible that we seek to strike this balance in the classroom and in program assessment and in other arenas where the pendulum has swung far to the side of the rubric?

6) Rubrics do not always speak to value. Let’s go back to the meal analogy. The question is not whether the plate looked pretty or whether the flavors fused together in a balanced way. The question is: was it good? or am I satisfied? or would I recommend this to my friends? Our ability to value (or evaluate) performance is, fundamentally, about these kinds of value questions. Now, it will be (and should be) argued that, in our role as educators, we are not your run-of-the-mill restaurant patrons — we are professional chefs, food critics, and gastronomes — and this isn’t a restaurant, it’s a cooking school. And whomever argues that point will be correct. It is to these people, and to myself, that I say this: in our mad scramble to evaluate whether students can fry an egg or bake a souffle in the hopes that they will someday know enough to become a chef, are we evaluating the goodness, the artfulness, the sensibility of their egg frying or souffle making? Are we making people who can manage spreadsheets, but not account? Who can film commercials, but not engage in commerce? Who can paint canvasses but not be artists? Who can memorize lines but not be actors? Do we — can we — value and evaluate things *besides* mere performance? What about evaluating a student’s disposition? Or their role as citizens?

7) Rubrics can be massaged. This is the story of a professor who looks at a project and says to himself or herself, “This seems like a C to me,” and then massages how he or she score the project on the rubric to get at the C he or she feels the project deserves.

8) Rubrics can be abused in the same ways not using a rubric can be abused. This is the story of a professor who, up against a midterm deadline, brain fried from several projects to grade, says, “the student who did this project seems to be with it and has been earning As so far. Rather than looking closely at the project or making substantive notes or giving worthwhile feedback, I’m just going to circle some things on this rubric that seem to make sense to me.”

9) Rubrics standardize assessment, which can standardize teaching and learning.  I won’t rehash the debate of the last couple of decades over standardized testing. What I will do is suggest that regardless of what you think about standardized testing and its value is that it risks institutions, schools, professors, and programs to begin engaging in standardized teaching and learning. In the face of scholarship about multiple intelligences and in light of thinking like that presented here (http://teachingatfsu.com/ken-robinson-changing-education-paradigms-a-discussion-starter/), I wonder if we might engage in an honest discussion about the implications of the rubrickization of our classrooms, outcomes assessment, and program assessment at our institution.

This is a starter list. As mentioned, I’m a recovering rubric addict — and one who is relapsing in several instances across several of my courses this semester. I wonder if you are, too, and if we might engage in a discussion about healthy rubric use, about making better rubrics where possible, and about exchanging rubrics for different kinds of assessment tools (and not just tests).

I’m sitting down now. Anyone else want a chance to talk?

Good read: Talking about religion in the classroom

. No comments.

Our editorial board member, Emily Lenning, came across this article about University of Central Florida professor, Charles Negy, who encountered religious-oriented hostility in the classroom. The article includes an email from Professor Negy to his class after a discussion in his cross-cultural psychology course became out of hand.

We think there are some lessons here for FSU faculty, particularly as we see a student population heavily influenced by evangelical Christianity and the influx of students from a variety of backgrounds with Fort Bragg’s presence so nearby.

http://www.huffingtonpost.com/2012/08/16/charles-negy-reddit-letter-to-students_n_1789406.html

What do you think?

A Lesson from Canada

. No comments.

Student uprisings at some universities in Quebec have made news in this country, including The Chronicle of Higher Education, and the analysis at the link below, by Justin E. H. Smith, explores several topics especially relevant for our experience at FSU. Read the full article at the link below. Meantime, check out these two excerpts:

 

“To put this last point a bit more pessimistically: it is growing ever harder to study, to teach, and to do philosophy in North American universities. One has the sense, in fact, that from a high administrator’s point of view, the humanities in general must appear as a sort of vestigial organ upon the university, something it hasn’t yet lost fully through the process of evolution, yet something that no longer contributes to the central mission of the university. That mission is, increasingly, to ‘partner’ with the private sector. Among other things, this partnering requires that academics learn to speak, or to imitate, the language of the men and women of business. And this in turn requires that the short-term pay-off of inquiry be immediately evident, even, or especially, to those who are not participating in the inquiry.”

“Ostensibly in the name of ‘better service’, universities are cutting corners and streamlining in a way that is jarringly similar to the recent upheavals in, say, the airline industry. Like almost every industry in recent years, more and more of the university experience is moving online. One often hears that the fate of the newspaper industry is the best guide to what will happen, with perhaps a decade’s delay, to the university. I recently learned that my own university will soon be offering students the choice of submitting course evaluations (that is, assessments of my ability in introducing them to philosophy) by mobile phone. The student evaluation, once a necessary counterweight to the sometimes arbitrary authority of faculty members, is now reduced to a convenient iPhone app, a streamlining of the educational product that directly parallels the appearance of mobile-phone-based boarding passes. Both the educational and the travel technologies are meant to be used with equal casualness and insouciance.

One way to understand the current movement would be to suppose that there is a dawning awareness that being given handy new apps for submission of course evaluations is not at all the same thing as being listened to. At every turn, the Quebec government has reacted to the students’ requests to open up lines of communication with something approaching contempt. The government’s line has been that there is nothing to discuss, since the budget is already a fait accompli. But what this misses are the broader concerns about the fate of education. It also misses the value of the students’ sincere effort to think about radically different –and, yes, unrealistic– models of the fate of universities in an era in which the consumer model threatens to infect nearly all varieties of human social interaction.”

Full article: http://www.jehsmith.com/1/2012/04/on-the-quebec-student-strike.html

Your thoughts?

 

Reframing the Documentary

. 2 comments.

This is the second post in which I am doing some public brainstorming about my plans for a new syllabus (again, this is republished from my personal blog). I’m getting closer to mapping out this particular course, but I would appreciate any responses you might have–including suggestions of favorite documentaries, especially those that test the limits of what we mean by the term.

For the first time in several years, I will be teaching Fayetteville State University’s senior seminar in the spring. The course fell into my lap late in the semester, so I haven’t had the time to prep that I would normally want, but I’m excited to have the opportunity to explore a set of research questions in detail with my students. The last time I taught this course–way back in 2007 (!)–I focused on the theme of “Documenting Injustice,” a phrasing that I hoped would encompass a wide range of activist, narrative, non-fiction texts in a wide range of media. Because the course is taught in our fairly traditional English department, I wanted not only to include a focus on literary texts but also to respect approaches based in textual analysis. Thus, while I’d enjoy teaching a course on the political economy of digital cinema (say), I don’t think my students would receive the “capstone” experience this course is supposed to represent.

That being said, the students who have signed up for the course know that I am the “film guy” in our department and know a little about my interests. So, with that in mind, I have decided to do an updated version of that course, which I am tentatively calling “Reframing the Documentary,” in part to entertain some slightly different questions about various forms of non-fiction. For the previous course, I sought to discuss a wide range of media forms–written non-fiction, photography, and film–and I’ll maintain that cross-media focus this time. Once again, I will require my students to read Jacob Riis’s How the Other Half Lives and Agee and Evans’ Let Us Now Praise Famous Men, texts that explore different aspects of documentary and observation with the goal of some sort of social change. But unlike last time, I am going to add David Eggars’ category-defying narrative, Zeitoun, in which he tells the story of a Syrian-born Katrina survivor, writing from Zeitoun’s voice.

In addition, I want to introduce some significant case studies on photography, such as the Dorothea Lange “Migrant Mother” photographs, Robert Capa’s “Falling Soldier,” and others. I’m planning to avoid directly studying most of the recent controversial photographs (especially the Abu Ghraib photos), although I may teach at Errol Morris’s Standard Operating Procedure. Instead, I’d like to look at some of Morris’s essays on older photographs, possibly including this essay on whether photographs lie (new to me: Morris’s New York Times essays on photography have been anthologized in a book). Or, more likely, that I will be teaching The Thin Blue Line, Morris’s blog posts on documentary re-enactments. I’ll supplement this discussion of Morris with screenings of either Strange Culture or Road to Guantanamo (or both).

From there, I ams till trying to decide how to engage with some of the “limits” of documentary. By that, I mean definitional limits, rather than where documentary itself is limited. I will likely include the animated Israeli documentary, Waltz with Bashir, and I’m thinking about doing a couple of mock documentaries, most likely Confederate States of America, and, thanks to some recent research by a former student, the boundary-defying film, The Watermelon Woman. One other area of emphasis will likely be autobiography, and I’m leaning towards Jonathan Caouette’s Tarnation, if only because I am familiar with it and because its production history is pretty catchy.

Finally, given some of my own recent work on “transmedia documentary,” I will likely finish with a couple of recent documentaries that used online media to expand the limits of what counts as a documentary, including The Age of Stupid. Following up on this idea of “transmedia documentary,” I’d very much appreciate any suggestions about online videos, photography series, or articles that depict aspects of the Occupy Wall Street movement. One “text” that I would certainly like to discuss would be the “We are the 99%” tumblr blog, but I may also set up a discussion of how iconic images of #OWS, such as the macing incident at UC Davis, have been remixed or repurposed.

I recognize that this post is all over the map–and mostly consists of a list of possible texts–but I am still brainstorming to some extent, trying to decide how, exactly, I want to frame this course. I am looking forward to doing an in-depth study of documentary, activism, and narrative, but I’d welcome any reminders about texts that I’ve neglected, including short essays, short stories, or other explorations of how we document our lives and how we use non-fiction images, sounds, and narratives to represent significant social and political events. Feedback (on Twitter, Facebook, or in the comments) is welcome.