M7 - Collecting evidence about teaching

Introduction

"One of the hardest things teachers have to learn is that the sincerity of their intentions does not guarantee the purity of their practice" (Brookfield, 1995, p.1)

Faced with a class of students, you can be safe in assuming that they don't all learn the same way you do - or the same way as each other. Your planning for teaching will take into account the diverse learning styles and interests of your students, but how do you know that what you have planned and implemented in your teaching is actually promoting student learning? This module will consider ways of evaluating student engagement and student learning during a unit and formalise the concept of reflective practice which was introduced in Module 6. We will consider how you can collect evidence about your teaching; and how you might use this evidence to both enhance learning and teaching, and to demonstrate your teaching competence through a teaching portfolio.


Learning Outcomes

At the end of this module, you will be able to:

  • Describe how critical reflection can be used to enhance teaching and learning outcomes.
  • Plan and implement activities to obtain feedback from students, yourself, peers and the literature in relation to your teaching and learning practice.
  • Critically reflect on your teaching and learning practice.
  • Collate feedback as part of a professional teaching practice portfolio.

Module Structure

The module covers the following topics about collecting evidence about your teaching; and using this evidence in other ways:

Part 1: Collecting evidence about your teaching

Part 2: How might you use the data you collect about your teaching in other ways?


How do we know that our teaching is promoting student achievement of the intended learning outcomes?

It is important that we evaluate our teaching as we progress through a unit to ensure that our teaching is ‘hitting the mark': achieving the targeted student learning outcomes. Evaluating our teaching enables us to respond to experience and student characteristics and preferences, adjust the approach for future units, and to adjust our approach for the balance of the unit. Evaluating our teaching in a formal, recorded way also enables us to demonstrate externally our teaching competence: the collection and management of feedback on our teaching is an important dimension of a professional teaching portfolio.

Task 7.1 What do you notice when you are teaching?

  • Spend a few minutes noting down the things that you take notice of when you are teaching. These may be to do with what the students do (or don't do), about timing and sequencing, about your own performance as a teacher, about the content etc. It may be helpful to recall a situation where you think your teaching went very well and perhaps another where you think your teaching did not work so well (how did you know?).
  • Do you keep a record of any of these things? If so, how (and where?).
  • Keep this list handy as we work through this module.

Part 1: Collecting evidence about your teaching

Stephen Brookfield (1995), a prominent academic in the field of reflection on teaching, suggests that there are four ways that we can collect evidence about our teaching. These four ways are:

  • from students
  • from yourself
  • from your peers
  • from the literature

We will talk about each of these ‘lenses' for collecting evidence as we progress through the module. However, it is important that we not only collect the evidence, but also that we reflect on it and then respond in a constructive way to enhance our practice. By employing reflection, we learn from our experiences (good, bad and interesting).

Critical reflection is the process of ‘analysing, reconsidering and questioning experiences within a broad context of issues' ("Critical Reflection", n.d.). In the case of university teachers, this may be in light of learning theories, use of technology, curriculum directions, employer expectations etc.)..

Collecting evidence from students

There are several situational contexts in which we may obtain feedback from students to gauge whether our teaching is effective. This may occur during, or at the end of, a unit. Evidence may be collected formally or informally, directly or by observing students and their learning, and in different teaching contexts. These are explored below.

Task 7.2 Cross-checking your own evidence about teaching

As we go through, cross-check your own list compiled in Task 7.1.

Lectures

There are a wide range of techniques that can be used to obtain feedback about your teaching in the context of a lecture, including:

  • Body language of the students (Are they engaged, or disengaged? Actively participating or passively writing notes? Do they ask questions?).
  • Questions to the whole group (You'll get the clearest responses to those where you give 2 choices and ask them to raise a hand to indicate a choice selection).
  • Think, pair, share (Give students time to think about the question themselves, then with a partner, and then with a group - the group response may or may not be shared with the whole class (University of New South Wales, n.d.)).
  • Minute paper (Give students some paper and ask them to note down the most important thing that they have learnt in the class, and one unanswered question).
David Bressoud describes how he used a Minute Paper to check his students' understanding of the concept of finding derivatives of exponential functions.
For example, after a class in which I introduced exponential functions of the form bx and explained how to find their derivatives, I discovered that there was still a lot of confusion about how the derivative of 3x was obtained. For many students, using (3x+.01 - 3x-.01)/.02 as an approximation to the derivative of 3x was more confusing than useful because they thought that an approximation to the derivative should look like (3x+h - 3x)/h. Some students were thrown by my use of the phrase "in terms of x" when I spoke of "the derivative in terms of x." They knew about derivatives, but had never heard of derivatives in terms of x. This provided an opening the next day to come back to some of their continuing misunderstandings about functions. More than one student thought that the most important point of this class was how to find the derivative of e. This forewarned me that some of my students considered e to be the name of a function. Only four students picked out what I thought I had been emphasizing: that the significance of e is that it provides a base for an exponential function that is its own derivative. Almost as many thought that the identification of ln with loge was the most important thing said all period. Knowing the principal points of confusion about derivatives of exponentials, I was able to start the next class by clearing up some of them and using selected questions to motivate the new material I wanted to introduce." Bressoud (n.d.)
  • Muddiest point: ask students to write down what idea/concept/technique has been the least clear to them. You can then consider whether you then provide a list of FAQs based on the identified questions or issues by students.
  • Student response systems (‘clickers') (Bruff, 2009). Emails from students asking for clarification.
  • Survey of students, asking students them about issues that are problematic and their preferences for learning and teaching.
  • Numbers of students attending lectures (if the numbers are low this might be that the students believe they can get the material in other ways. If you do have the opportunity to put lecture notes online - what are you value-adding in your lectures for those students who attend?).

Tutorials

Many of the activities suggested for lectures will also work in tutorials. However, with smaller groups you can use additional feedback strategies. As described in Module 6, as you give (informal) feedback to students on their learning, you find out about your teaching as well. Some suggestions are:

  • working through a problem in groups or on a whiteboard, with students making suggestions of alternative strategies
  • students working through example questions and circling or annotating their solutions to denote any areas of problem
  • through sharing examples of work as they are working through problems (consider discussion of sharing the cognitive processes involved and explaining the thinking and the problem-solving approach)
  • asking students to report back on key points from lectures
  • asking students to anonymously write on a slip of paper at the beginning of a tutorial any things that they are confused by or are having trouble with, which can then inform the conduct of the tutorial. An alternative is that this is emailed to the teacher or posted on a discussion board in advance of the class; this enables the teacher to consider the material in the course of preparing the lesson. This technique can also be used in relation to getting feedback on various aspects of the tutorial experience including pace, clarity, utility etc
  • asking students to annotate sample answers to explain the steps used
  • asking students to formulate questions about the material for others to answer
  • measure and record the percentage of students who can complete target formative assessment tasks in the tutorial group at the beginning and end of the tutorial or series of classes

Partway through a unit

Many of the suggestions above will have a fairly immediate impact on your teaching. For example, if you notice that students are having a particular problem in a tutorial, then you will spend some time addressing this. There are other more formal methods for obtaining information about how well students are tracking towards the learning outcomes as the course progresses. These include:

  • quizzes (in class or online)
  • formative assessment tasks where students are required to mark their own work against sample answers and comment on where they approached the problem differently, or where and why they went wrong
  • surveys of students asking for feedback on areas that they would like covered in more depth, anything that they feel is missing in the course so far etc.
  • getting feedback from tutors about how students are performing in tutorials
  • assessment tasks.

Assessment tasks not only give you the opportunity to pick up common issues and misunderstandings they can also give the students valuable feedback that they can use to assist them in their learning. This idea was explored extensively in Module 6.

Task 7.3 Gaining feedback from assessment tasks

Consider this example of student work. It is an example of a student's solution to a question testing their understanding of mathematical induction. If this is part of the formative assessment material, you will need to provide concise statements of where their mistakes are, what is the likely misconception which led to each mistake and indicate how they can move beyond these misconceptions.

Look at the example. Where has their mistake been made? What feedback has been given to the student - and what might you cover (or re-cover in class) to assist student understanding if this was a common error made by the class?

There are two mistakes in the attached proof. The annotations on the solution are to provide feedback to the students.However, you can also obtain important information from this material. The student's solution tends to indicate that the student didn't fully understand summation notation and the difference between the term of a sequence and the position in the sequence. In the above proof the student attempted to take the (k+1)th term out of the summation. But they simply added k+1 and not the value of the sequence i2 at i=k+1. This indicates that the student may just be implementing a recipe for solution and has not grasped fully the overall concepts. However they have taken the inductive step of using P(k) in the substitution from line 2 of the proof to line 3. Given the previous mistake, it would be worthwhile checking that they do understand this important step and if not, go back through this with the class. You could use this example as a prompt for students; however, you should make sure that it is not recognisable as any particular student's work.

Task 7.4 Collecting evidence from your students

Choose two of the above forms of evidence collected part

way through a course. In your classes, collect examples of these and keep them in a folder for reference in the next section of the module. Annotate these to indicate whether you shared any of this information with your colleagues and/or students, and what, if any, changes you made to your teaching as a result of this evidence.

At the end of the unit

There are also several ways of collecting summative evidence about your teaching:

  • students' performance on examination or summative assessment tasks
  • student Evaluation of Teaching (SET, SETL etc.)
  • survey of student opinions
  • feedback from other stakeholders (for example teachers of the units that follow on from yours, or employers etc.)
  • focus groups of students

Task 7.5 Using student feedback to enhance teaching

Read the student comments in relation to a great teaching experience in a quantitative discipline at Stanford (Stein, 2010).

Think about the themes from the comments, and how the teacher engaged and got feedback from the students in relation to their understanding. Consider also his teaching and how he improved it during the class (after the early interactions), rather than only leaving it to the end.

A fairly ubiquitous source of student feedback is student evaluations (SET, or the equivalent at your university) that usually have both quantitative (Likert scale) and qualitative (open-ended comments) components. Whilst the quantitative results can be of some use, particularly for comparison, the qualitative comments can give an excellent insight into students' impressions of your teaching.

While this evidence is important, it is essential that you view this summative feedback in conjunction with other evidence (such as comments from your peers or co-teachers), and the context of the unit. Deal with student evaluations as soon as you get them, while you still remember the nuances of the unit. It is recognised that change is often difficult for students, so you may get a lower SET score on some questions when you have implemented change - this does not necessarily mean that the change was bad. By considering what the students have said in the comments, you can determine if it may have been introduced or supported differently. Also consider whether, in fact, they performed better on assessment tasks after the change.

Collecting evidence from yourself

There are ways that you can collect self evidence about your teaching. These include:

  • Annotating lectures. (For example, note if you needed to add information or supplementary examples, where you thought students didn't understand or were lacking prior knowledge; additional examples that were raised in class; suggested changes in timing or sequencing of material - post-it notes can be good for this!)
  • Video your lecture or tutorial and critique the performance with recommendations for further improvement.
  • Play the role of a student in one of your lectures that you have recorded - sit through the video, write notes etc. How could the experience of the student be improved to drive better outcomes?
  • Create and use a standard form for self assessment at the end of each lecture or tutorial, with questions assessing the level of student engagement, the pace through the material, demonstrated student learning, achievement of outcomes. Store and reflect on these self-assessment pieces at the end of the unit. This can be done as an online survey response which then collects and analyses the responses for trends.
  • Watch other teachers, either by observation in classes or watching recordings (and complete a structured assessment and analysis instrument).
  • Think about how you would teach something if you could not use your usual methods, but wanted to engage the students.

Some specific question your might like to use are:

  • How was the diversity of students managed?
  • Did I ask a variety of questions?
  • What was the level of student engagement?
  • Has there been a change in student beliefs about mathematics?
  • Do I have overly high expectations of my students?
  • Were the examples used relevant to my students?
  • Did I deliver praise and feedback where relevant?

Task 7.6 Self-reflection on a teaching experience

After a lecture or tutorial, spend some time to annotate your notes in the ways suggested above.

Systematic filing of annotated lecture and tutorial notes is very useful not only in informing teaching in the current semester, but can and should be used for planning when you are teaching the material for a second (or subsequent time).

Collecting evidence from your peers

There are a number of ways you can get feedback from your peers - in what ways do you think you (or could you) interact with your peers to enhance your teaching?

Benchmarking

Asking a colleague, either within your department or from another institution, to review your unit outline, your unit readings or texts and your assessment tasks can be very useful. This is a method of benchmarking - and with this you can be confident that your unit is on par with those in similar degree programs. Connecting with your peers from other institutions can be done through conferences, professional associations or personal contact.

Your colleagues

Most universities require you to write a unit review, reporting on the student feedback and the pass rate, and other issues. If your department then holds meetings to discuss or approve these reports, you can obtain peer feedback; if they are reviewed only by your Head, he or she may suggest someone to talk to. Experienced colleagues may have advice; some problems require a collective or co-ordinated response; good practice can be shared or adapted for other units.

You may have experienced tutors teaching in your unit. They can be a valuable source of feedback on student learning, and on the materials you have provided for the tutorials.

Peer observation

Peer observation of teaching can also be arranged to give you feedback. This can be initiated by you, or can be facilitated through your Head of school/department or your academic development unit.

  • Arrange a pre-meeting: Peer observation works best when a relationship is established - meeting with your reviewer before the observation, and agreeing on aspects to be reviewed assists in developing this. You may have something specific to raise as a result of student feedback, or from your own self-reflection. There are also many different sources you can use as prompts, a good one is: Murdoch University ASD Peer Feedback on Teaching (Murdoch University, n.d.). You may also like to discuss some pragmatics, where the reviewer will sit, what you will tell the class (if this is relevant), and when and how feedback will be given.
  • Observation: Once the topic for the peer observation has been established, and any concerns discussed, the observation can occur at a mutually agreed time or times. This resource for observation templates and checklists from the University of Minnesota (University of Minnesota, n.d.) and the comprehensive user guide from Macquarie University (Rowe, Solomonides & Handal, 2010) can get you started.
  • Post-observation feedback: Feedback can then be given verbally, or in written format (or preferably a combination of both). Again, pro formas are available for this purpose, or you can construct one to meet your purposes.
  • Follow up and action plan: During the post-observation dialogue, and in the subsequent reflection, you can make a plan for enhancement or ideas that you may like to trial.

A great Australian resource for peer observation is the HERDSA guide, Peer Observation partnerships in higher education (2005).

Task 7.7 Looking at observable characteristics of good teachers or following up on peer observation.

  • Look at this resource for observation templates and checklists (University of Minnesota, n.d.) and find the list of 'observable characteristics of effective teachers'. Choose five that you would be interested in getting feedback on.
  • If you have engaged in peer observation in the past, find the feedback you were given. How have you engaged with this feedback to change your teaching? Make a note about these changes.

Collecting evidence from the literature

In addition to general research on learning and teaching, there are journals, professional associations and scholarly conferences that focus on mathematics-specific teaching and learning. A comprehensive list is found under the Further Reading section in Module 12. In this literature you will find case studies of practice that can also inform your work.

Task 7.8 (optional) Looking at case studies

Choose one of the sources listed in the Further Reading section in Module 12 and find an article that relates to a unit or topic that you teach, or where an author has tested a strategy that you would be interested in trialling in your classes. How might you use what you have read in this article to enhance your practice?

Example: Students in a service unit, who were accustomed to a passive style of tutorial in their own discipline, were very critical on formal surveys of the amount of thinking they were expected to do for themselves in an active-learning mathematics tutorial and expressed a preference for the tutor to work problems on the board. Though it was tempting to give the students what they asked for, the lecturer was supported by his colleagues instead to develop better ways to explain the rationale for the tutorial style, and to check that the tutorial activities aligned with the assessment. He explained in the unit review for the partner discipline the action he proposed in response to this ‘problem’. The fourth lens (the literature) clearly supports active learning activities which align with the intended learning outcomes.

What does reflection look like in practice?

The table below describes a practical hierarchy of reflection. It was adapted from the work of Zeichner and Liston (1996, pp.41-42, as cited in University of Birmingham, 2006) by the Professional Development for Academics Involved in Teaching website (University of Birmingham, 2006). We have further adapted it for our purposes, with our additions in italics.

Rapid reflection Immediate and automatic Ongoing decision-making while teaching, happens very fast, almost constantly, often privately.This is often ‘spur of the moment' based on picking up almost imperceptible cues from the class, or responding to a ‘teachable moment'.
Repair Thoughtful Teacher makes a decision to alter behaviour in response to cues from students. This may be as a result of cues from students indicating the material is not pitched correctly, or is not engaging e.g. questions or comments that indicate misunderstandings. If you are able to observe students working through problems, for example, you may pick up issues that need to be thought through and addressed.
Review Less formal At a particular point in time Teacher thinks about, writes about or discusses some element of teaching or students' learning; often interpersonal and collegial. This may include notes added to lesson plans or minor changes to units or sequencing of units.
Research More systematic Over a period of time Thinking and observation become more sharply focussed around particular issues; involves collecting data over time e.g. Action Research, keeping a teaching journal.Further information on Action Research is found in Module 12
Re-theorising and Reformulating Long term Informed by public academic theories More abstract and more rigorous; teachers critically examine their practical theories, and consider these in the light of academic theories. This may involve writing, or contributing, a scholarly article or disseminating the work to your peers.

Table 1. Practical hierarchy of reflection (adapted from Zeichner & Liston, 1996, pp. 41-42, as cited in University of Birmingham, 2006)

Task 7.9 Levels of reflection

Referring once again to your own list created in Task 7.1, perhaps augmented as you have gone through the module; which of the pieces of evidence would you use at each of the levels of reflection?

Extension reading on the four lenses for evaluating your teaching: summary one (The University of Sydney, n.d.) and/or summary two (Miller, 2010).

Part 2: How might you use the data you collect about your teaching in other ways?

As well as using your data to enhance your practice, this evidence can be used to compile a teaching portfolio to demonstrate professional competence, or high levels of professional practice.

Why teaching portfolios?

Teaching Portfolios allow academics to make a persuasive case for their effectiveness as teachers and can be used for:

  • Demonstration of learning (e.g. Graduate Certificate Assessment)
  • Enhancement of career (e.g. to support application for academic positions, confirmation of probation, or promotion)
  • Professional development (e.g. to support applications for teaching and learning grants or professional learning opportunities, citations and awards, to enhance a scholarly approach to teaching, for performance management and performance planning purposes)
  • Academic audit purposes (e.g. internal/external reviews of degree programs, AUQA audits).

As with a research portfolio, a Teaching Portfolio should be developed from a firm academic base, in this case the body of research into effective teaching and learning at tertiary level.

The foundations of a teaching portfolio: the archive

Material for a teaching portfolio is drawn from a broader collection of material variously known as a "Teaching Record" or "Teaching Portfolio Archive". The material in a Teaching Portfolio archive may include:

  • Quantitative, factual and descriptive material (e.g. lists of units taught and the courses they contribute to; unit outlines; assessment tasks; teaching resources developed; innovations you have introduced; lists of teaching development responsibilities, teaching and learning grant applications)
  • Qualitative material (e.g. student feedback SETLs - with annotations/reflections and other evaluations, unsolicited student comments, peer feedback - from teaching observations or review of resources, reflective journal, critiques of performance, case study accounts)
  • Scholarship of teaching/professional learning material (e.g. results of research into your teaching, publications about teaching, courses or workshops you have attended together with any reflection, invitations to present about teaching, grants and awards gained.

(Adapted from Lublin, 2002, pp.2-4)

Task 7.10 Compiling a list of your teaching evidence

What type of teaching-related records have you kept? List all the different evidence you have in your archive. Remember that you have collected some evidence in your tasks assigned for the first part of this module. What other information should you keep? Make a list of other possible material to collect for the remainder of this semester/year.

You may like to consider using an e-Portfolio software such as PebblePad for the collation of an electronic teaching archive that can be used as the source of evidence for particular purposes such as those listed above.

What does a teaching portfolio look like?

A teaching portfolio is a concise document (a maximum of 5 written pages plus appendices) that summarises an academic's approach to teaching and learning. It is designed to be reflective and discursive and to contain evidence to support claims. The data for the Portfolio is selectively drawn from the teaching portfolio archive, depending upon the purpose for which it is being constructed.

It is usual for a portfolio to contain an introduction that includes a teaching philosophy statement (with references to relevant literature, the fourth lens), a context (what is being taught and to whom), and then statements written against a series of criteria. Many universities have adopted criteria that reflect the Australian Learning and Teaching Council criteria for teaching citations. A possible outline for a teaching portfolio is:

Outline of a teaching portfolio - and exemplar evidence

1. Introduction and teaching philosophy

  • A general statement about your teaching aims, philosophy, and approach.
  • Demonstration of your knowledge of effective practice.
  • General description of the context in which teaching and learning takes place.

2. Evidence of teaching responsibilities

  • Analysis of units taught, roles and responsibilities.

3. Evidence to support claims against the relevant criteria (e.g. the ALTC criteria)

  • Design and plan for approaches to teaching that influence, motivate and inspire students to learn.
  • Develop curricula and resources that support teaching and learning.
  • Recognise the roles of assessment; plan and implement approaches to assessment and feedback that foster independent learning.
  • Recognise student diversity and apply this to the planning of teaching and learning experiences.
  • Evaluate and improve teaching and student learning outcomes through gathering and evaluating data and employing critical reflection.

4. Appendix

Relevant supporting documentation (that is referenced in the text of the portfolio).

Task 7.11 Matching your evidence with criteria and categorising by source

  • Using your list of evidence compiled in Task 7.9, fill out the following grid to make a note of what type of evidence you have collected for each of the criteria:
Criterion Evidence
Design and plan for approaches to teaching that influence, motivate and inspire students to learn.  
Develop curricula and resources that support teaching and learning.  
Recognise the roles of assessment; plan and implement approaches to assessment and feedback that foster independent learning.  
Recognise student diversity and apply this to the planning of teaching and learning experiences.  
Evaluate and improve teaching and student learning outcomes through gathering and evaluating data and employing critical reflection.  
  • To check you have a range of sources of evidence, use highlighter pens to designate each of the four different angles for collecting evidence: students, yourself, your peers and the literature. Have you gathered evidence from a range of sources, are there any gaps that you can identify, and how will you address these in your future collection of evidence?
  • Make a teaching portfolio plan for the remainder of the academic year. What evidence will you collect and when will it be collected? What arrangements, if any, do you need to put in place to collect the evidence (e.g. requesting peer observations, constructing student surveys). Post this plan to the discussion board .

Review and conclusion

In this module you should have gained an awareness of the different types of evidence that can be collected about your teaching and how it can be used to support critical reflection and ongoing improvement to your teaching, and importantly to student learning. Each of Brookfield's 4 lenses for evaluating teaching (your students, yourself, your peers, and the literature) has been discussed.

The collecting of evidence about teaching is not only needed to inform ongoing enhancement of student learning; it can also be used to demonstrate competence and highly developed practice in teaching through a portfolio. This module has also outlined how you might go about compiling an archive from which a portfolio can be drawn.

The next module begins the second part of this unit - Co-ordinating Units. The starting module 8, Planning and designing units will build on what has been covered in the modules so far, in terms of student learning, teaching and assessment. Preparing a unit description and the component learning, teaching and assessment activities are also sources of evidence that you can use for a teaching portfolio (as evidence of curriculum development and assessment for example), so as you progress through the next series of modules be mindful of how you can add to your teaching archive.



References

Further reading

  • Committee on Recognizing, Evaluating, Rewarding, and Developing Excellence in Teaching of Undergraduate Science, Mathematics, Engineering, and Technology. (2003). Evaluating teaching in science, technology, engineering, and mathematics: principles and research findings. In M.A. Fox & N. Hackerman (Eds.), Evaluating and improving undergraduate teaching in science, technology, engineering and mathematics (pp.51-68). Washington, DC: National Academies Press.
  • Committee on Recognizing, Evaluating, Rewarding, and Developing Excellence in Teaching of Undergraduate Science, Mathematics, Engineering, and Technology. (2003). Evaluation methodologies. In M.A. Fox & N. Hackerman (Eds.), Evaluating and improving undergraduate teaching in science, technology, engineering and mathematics (pp.69-99). Washington, DC: National Academies Press.
  • Graves, G.H. , Sulewski, C.A., Dye, H.A., Deveans, T.M., Agras, N.M. and Pearson, J.M.. How are you doing? Assessing effectiveness in teaching mathematics, PRIMUS, 19(2), pp.174–193. doi: 10.1080/10511970802409222
  • Simonson, M.R. (1997). Evaluating teaching and learning at a distance. New directions in teaching and learning, 71(Fall), pp. 87-94. Retrieved from: http://www.grossmont.edu/don.dean/pkms_ddean/ET690/documents/EvaluatingDistanceEd.pdf

Updated: 24 Feb 2014
Feedback