I have been teaching at UNSW now for over two years, and in that time I have become well acquainted with the UNSW Course and Teaching Evaluation and Improvement (CATEI) reports. These ubiquitous evaluations are completed by students at the end of each course and constitute a snapshot of their overall “satisfaction” with the course and the lecturer. Happily, I have received uniformly positive CATEI evaluations (90%+ satisfaction) in my courses thus far, and have paid them scant attention except in so far as it seems to please my superiors. More entertaining, from my perspective, has been the direct (anonymous) feedback from students. Some examples:

Question: What were the best features of this lecturer’s teaching?
Answer: Useful examples in explaining mathematical concepts… his use of a hyper-potato in 3 dimensions to explain the Lagrangian multipliers is something that will stick with me for years.

Question: What were the best features of this lecturer’s teaching?
Answer: The way he sometimes wore jeans and joggers (adorable).

Question: How could this course be improved?
Answer: Strippers… Otherwise flawless.

N-dimensional tubers, fashion advice, and exotic dancers aside, the overwhelmingly positive feedback has been a great boost to my confidence and has allowed me to be myself in lectures, something I was reluctant to do at first. But other than that I have not really found the CATEI evaluations to be much use in my teaching practice; indeed, I have tried not to take them too seriously, for fear that I become complacent in the classroom.

So I was quite interested to learn more about the role of evaluation in University Learning and Teaching in the fourth module of FULT, and in particular to discover new tools for evaluating my development as an educator. It has been argued (by John Hattie, for example), that the primary role of a teacher is as an evaluator: that we need to be constantly assessing how well we are reaching each student. I think this is overly reductionist and provocative — educators are many things, including communicators, mentors, facilitators, and even “change agents” (good grief). But certainly we need to be aware of our efficacy if we are to develop as teachers.

In this context, we were asked to perform a “mini-evaluation” of our teaching as preparation for this FULT module. As I teach nearly 500 students (and administer nearly 1000), there wasn’t going to be much that was “mini” about any sort of student feedback. But I nonetheless set up a Moodle feedback poll on the course webpage and asked students to submit anonymous comments. Within a week I had over 100 replies — more than enough to keep me busy.

The evaluation tool I used was “Stop, Start, Continue”: students were asked to suggest something the lecturer should STOP doing, something I should START doing, and something I should CONTINUE doing (there was also room for additional comments). As with the CATEI evaluations, the responses I received were overwhelmingly positive. (The “median” response to the questions Stop? Start? Continue? could be summarized as “nothing”, “nothing”, and “everything”, respectively). And in this sense they were actually quite disappointing — or, if not disappointing, then not very useful.

There were specific things that I could act on, to be sure. Some students requested that I work more slowly, or give more examples, or adjust the brightness on the projector. All small tweaks that I have since implemented in my teaching, demonstrating, if nothing else, the great advantage of a mid-semester evaluation rather than an evaluation at the end of the course. But overall it raised a critical issue that we discussed at length in the Face-to-Face session this week: what is the evaluation for?

One way that we explored this question was through a survey of our own views of four concepts: “Student experience”, “Student engagement”, “Student evaluation of teaching”, and “Student satisfaction”. We were asked to define each of these concepts in our own words, and, while I was able to wax lyrical about what constituted the “Student experience” (“the whole of University experience, from sweating over a mid-term, to chatting with professor after class, to late-night conversations over beers with friends”), and could hazard a try at “Student engagement” (“focused interaction with a given learning experience”) and “Student evaluation of teaching” (“a student’s perception of the ‘quality’ of a teaching interaction, compared to what they have experienced before”), I had great difficulty in making sense of the idea of “Student satisfaction”.

The problem here is that “satisfaction” is a squirrelly concept. Students assess learning experiences against all sorts of experiences, and not just educational ones: they will also compare learning experiences with popular entertainment, movies, games, and so on. So the idea of “student satisfaction” starts to shade into a general sense of consumerist enjoyment — five stars on an Amazon.com review — and loses its utility.

During our table discussions, I came to realize the problem with implements like CATEI evaluations and Stop/Start/Continue is that they assess student satisfaction, student engagement, and student’s own perception of teaching. And while these can be considered to be proxies for the quality of a learning interaction, what we are really interested in is the actual student experience.

But how do we measure such a nebulous concept? As we learned in FULT, there are tools that we can use to try to dig more deeply into the student experience. But I am not convinced that they are practicable for classes of up to 1000 students. I am also not entirely sure that we have a duty or a right to try to assess the very idea of “experience” — something that takes in all aspects of University life. The student experience is one of transformation and growth. It can be painful; it is usually challenging; it is not always enjoyable. Implicit in the idea of an “evaluation” is that there are preferred outcomes. But what happens when the outcome is just the jarring, messy, and ultimately deeply enriching process of growing into an adult? How much of that is my responsibility?

I don’t know the right way to evaluate my teaching. Maybe there isn’t one. In the meantime, I will do the best I can to to pay heed to student evaluations when they are useful and ignore them when they are not. Oh, and adjust the brightness on the projector.