Inter Rater Assessment

As my regular readers know, last week I was at an Assessment Institute. What has always been odd to my mind is how much effort is often spent selling Assessment.

I refuse to sell, market, or otherwise carpetbag for Assessment.  Note the capital A.

Do not get me wrong; Victoria College does assessment and Assessment, will continue to do both, and I will be proud of my record as a coordinator of the latter and a practitioner of the former.

The problem with selling something, whether a set of professional linguistics, rote spells or recipes sold off by master practitioners (or charlatans), or any other facet of Assessment, is that you cheapen your product.

In my opinion, assessment does not need selling. Do you believe in self-improvement? Do you believe you have the power, the innate ability, to make your part of the world, no matter how large or small, a better place? If you answer yes to any of these questions, then the instant follow up is: “How do you know if you’re making a difference?” And in that case, there’s no need to sell assessment. Because assessment is just a buzzword for the scientific process of research. And I dare say science is humanity’s boldest effort to utterly master the world around us.

If you answered ‘no’ to any of those questions, I do not need to sell you assessment. I need to give you hope. Hope that no matter how dreary the circumstance, no matter how ghastly the storm, you do in fact have extraordinary abilities to make changes. I also have a book I need to give you to read. So, if you did answer no, please come see me.

Back to Assessment. It is a sometimes not-so-well explained, not-quite-perfectly executed set of trending-towards-actionable numbers designed to appease legislatures and accreditation agencies. I think this is where the apologetic sales pitch comes into being. People realise that far too much of Assessment is summative. Thus, the apologists leap into sales mode because they intuitively understand assessment is required.

On the other hand, there are professors who do assessment all the time, but, when they see the trappings of Assessment, like good iconoclasts, they have natural scepticism to the obvious herd mentality, and the, ahem, rubrics, one tends to see in such activities.

I want to close off this week’s post with two important thoughts that I bring back from this Assessment Institute.

The first is that Assessment, that legalese bureaucracy that does help us tell our story yet is quite tedious, is going to change. There are some new words that will be required. There’s going to be a change from formative to summative. This is driven by state and federal legislation. I feel better prepared to make the case for Victoria College now. It may not be glamorous; all the same, this is important work. And Victoria College is worth it. But I am not selling it.

The second is a very cool bit of psychometric validation technology. For the first time, I met a fellow user of Shiny at an education conference (although I’m sure there must be more of us out there). He developed some quite nice code that makes it very easy to measure and rate inter-rater scoring (think the type of scoring that happens when one uses a rubric).  Dr David Eubanks gave an amazing talk about these inter-rater facets, and I was quite glad I picked his session. Plus, since he’s an applied mathematician, I may have just found my new hero for assessment.

Do you use rubrics with multiple raters? Are you curious how reliable your rubrics or raters are? If you’d like to talk, drop me a line or come visit me in CAPE 110!



Matt Wiley is a tenured, associate professor of mathematics with awards in both mathematics education and honour student engagement. He earned an assortment of degrees in computer science, business, and pure mathematics from the University of California and Texas A&M systems. He is the director of quality enhancement at Victoria College, assisting in the development and implementation of a comprehensive assessment program to enhance institutional performance outcomes. A programmer, a published author, a mathematician, and a transformational leader, Matt has always melded his passion for writing with his joy of logical problem solving and data science. From the boardroom to the classroom, he enjoys finding dynamic ways to partner with interdisciplinary and diverse teams to make complex ideas and projects understandable and solvable.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s