M12 - Evaluating units

Introduction

This Module discusses why and how you might evaluate a unit you are teaching, and how you would go about making improvements. The idea of using action research for researching your own learning and teaching practice is introduced, together with information on sources of scholarly literature and conferences that focus on tertiary mathematics education research.

Learning Outcomes

By the end of the module you should be able to:

explain why units should be evaluated and how this aligns with your obligations for unit evaluation within the quality assurance framework at your university
identify and collect evidence that can be used to evaluate a unit, and outline how this evidence may be used to enhance a unit
apply the principles of action research to plan for a small-scale intervention aimed at enhancing teaching in your unit, or a unit into which you teach
be more familiar with sources of research into undergraduate teaching of mathematics and professional development that can both inform your practice and give you an opportunity to share your practice with your peers.

Module Structure

The module covers the following topics:

Why evaluate units?
What evidence should you collect to evaluate units?
Making changes to units
Action research.

Why evaluate units?

Evaluation of units is very important in maintaining the quality of university courses. Most universities have a quality assurance framework that expects unit co-ordinators to undertake reviews of their units on a regular basis.

Continual review and enhancement of units is important to ensure that the unit is:

current in content (what is taught) and delivery methods (how it is taught)
aligned with the course within which it is situated (does it prepare students for later years, does it progress from earlier units without undue repetition, does it contribute to the program and/or course level learning outcomes?)
meeting the needs and expectations of students
taking into account feedback from external stakeholders
comparable to similar units in other universities.

The data that are collected to evaluate this are very similar to the data you collect to inform your own teaching (see Module 7), though here the emphasis is on the unit as a whole.

Task 12.1. Locate a weblink to your university's Quality Assurance Framework

Find the link to the Quality Assurance Framework at your university and bookmark it for future reference. Can you find a reference to unit review within the framework? What are the obligations for unit co-ordinators in terms of unit evaluations?

Hint: If you cannot find this with a search for ‘unit review' or ‘quality assurance', you may find reference to this under pages that are headed ‘curriculum review' or ‘course review'. If this is not easy to find on your university web site or in documentation, find out from your program co-ordinator if any such documentation exists.

What evidence should you collect to evaluate units?

Task 12.2 Categorise evidence

Referring to your work in Module 7, where you considered evaluating your own teaching, fill out the table below to indicate the types of evidence you could (in principle) collect to evaluate a unit. Once you have done this you can cross-check with the example provided below.

Self	Peers
Student (reactions and learning)	Literature

Here are some suggestions for evidence that you could collect:

Table 1. Suggestions for evidence for evaluating units

Self

Annotations on lecture notes/tutorial planning/unit outlines etc
Reflective journal
Notes taken on ideas from conferences, meetings with stakeholders, discussions with peers, observations of peers etc
Recordings (eg video, audio)

Peers

Formal and informal reviews of teaching, teaching materials, assessment tasks and unit outlines
Units, tasks, course documentation obtained from colleagues working in similar units
Ideas from teaching interest groups, discussions with peers etc
Use of a peer review checklist Review checklist (Word)

Student (reactions and learning)

Student Evaluation of Teaching and Learning Informal student feedback
Online feedback
Student work samples
Questionnaires
Focus groups
Student journals
Student questions in class or to tutors
Student self-reports of learning and skills
Student assessment outcomes
Attrition and progression rates
External surveys (e.g. CEQ and AUSSE)
Minute papers
Employer/workplace feedback on student skills/performance/readiness

Literature

Discipline-specific journals - teaching and learning in relevant disciplines
Conference presentations regarding teaching in your discipline (e.g. AustMS, MERGA)
(A list of relevant journals and conferences - where research related to the teaching and learning of tertiary mathematics and statistics is shared - is provided under 'Further reading'.)

Task 12.3 Planning to collect your own evidence

What types of evidence do you (or intend to) collect to inform an evaluation of your own unit (or the unit that you teach into)? Highlight this on the table above (or on your own table from Activity 12.2) and keep this in your teaching portfolio file. How many different sources have you used? How could you systematise the collection of evidence? Have a look at an example from the University of Western Australia of how to do this (Centre for the Advancement of Teaching and Learning, 2009).

Making changes to units

Once evidence has been collected, it can be analysed to suggest where improvement may need to be made. In some cases it will be relatively clear what course of action can or should be taken; in others it may be a good idea to devise a possible course of action that can be trialled and evaluated. It may be useful for you to ask a peer to help you review the evidence with a view to developing an action plan if needed.

Negative evaluations from students do not necessarily mean that you should make changes. Instead, better communication of your goals and outcomes and why you have chosen a particular teaching paradigm may change negative evaluations into positive ones.

Remember that, unlike your own teaching, ‘your’ unit actually belongs to your department, and there are many stakeholders in it. You should discuss your proposed action plan, including your evidence, with your Head or in a department meeting before you act on changes you have identified.

Task 12.4: Identifying the need for change

Collect and consider evidence from at least three different sources about your unit (or the unit that you teach into). Do you see any issues arising from this data that would prompt you to implement or investigate a change in the unit? Write a brief summary of what you have found and possible future actions (no more than two paragraphs).

This activity will provide you with some evidence that you can include in your teaching portfolio.

How do you go about effectively communicating a change to your unit to the students?

When you make a change to a unit, it is good practice to make this publicly known. In some universities there is an expectation that any changes made as a result of student feedback are reported to the students. Below are some examples that have been published on a website specifically for students.

Table 2. Examples of changes as reported to students (University of Tasmania, 2009)

Data Handling and Statistics 2	The assessment items have been made shorter and more frequent and a presence on the Learning Management System is being developed.
Probability Models	Detailed online material has been prepared for each week of lectures, also a decision to make the unit flexible has been introduced.
Bridging Maths	Revised assignment content to ensure links to unit objectives are established. Assessment value of assignments revised to reflect relative degree of difficulty.
Banking and Financial Institutions	Changes to the textbook have been made. Students agree with the decision to introduce continuous assessment in the form of in-class tests.
Statistics for Business	The values of test 1 and test 2 have been changed to 10% and a practice project worth 20% (investigating the use of statistics in a business setting) has been introduced as a key learning tool.
Economics for Managers	Examples of assignment questions and solutions are provided on the LMS and discussed in class, with reference to the assessment criteria sheet to provide students with an understanding of the depth of economic analysis required for a pass or higher grade in this unit.

Task 12.5 What evidence would point to the need for change?

In two of the units abovethe co-ordinators made the following changes:

Adopting an authentic assessment task that will enable early feedback to be given to students (Statistics for Business), and;
Providing early feedback and communicating clear criteria and standards for assessment tasks (Economics for Managers).

Post your responses to the following question on the discussion board :

Can you suggest what type of evidence may have been collected to indicate the need for changes such as these? What might have been found in the data that suggested these changes?

In Statistics for Business, it may have been found that students had failed assessment tasks due to an inability to apply their knowledge into authentic contexts in examination questions; students giving formal or informal feedback about needing to have assessment that was more authentic; and/or feedback or input from employers or other stakeholders that students could be better prepared to tackle real-life problems.

For Economics for Managers some ideas you may have had could have been large numbers of students who were finding it difficult to understand the requirements of a pass (or higher grade), or who had indicated in student evaluations that they did not understand assessment criteria.

As the assessment approaches in each of the units have undergone change, it would be a good idea to follow up on the effectiveness of the changes and whether they have in fact enhanced student learning. Note down how the co-ordinators might go about this (what evidence to collect, what to do once it was obtained) in each of these units.

What you suggest will depend on the reasons for the change,. It is likely that a review of overall student results might be considered - in general have the changes resulted in more students passing the unit? You may also have suggested that the co-ordinator could include a pertinent question in the student evaluations or conduct informal evaluation by using a survey, minute paper or inviting email feedback. It may also be possible to hold student focus groups to get feedback.

Action research

What you have started to describe is a form of Action Research. This is a type of research ‘applicable in situations in which participants wish to improve their own practice' (Kember, 1998). Unit co-ordinators who are interested in improving their unit can use action research principles to investigate changes to the unit. Examples of areas you might like to change could be teaching tasks used in the unit, modes of delivery or assessment tasks. Action research may be different from other types of research you have undertaken, however it is well established and has the following characteristics:

concerned with social practice (such as learning and teaching)
participative (the researcher is involved in the research)
participants decide on the topic (what is researched arises from the experience of the researcher and the participants in the research)
aims towards improvement (the research is not about finding an answer to a question, but aiming to improve practice)
cyclical process (there is a continual cycle of intervention, collecting evidence, analysis and refinement of intervention)
involves systematic inquiry (evidence is collected systematically and from a variety of sources)
reflective process (the participants review and reflect on findings in an ongoing way).

More detailed information on action research can be found here: Kember, D. (1998). Action research: Towards an alternative framework for educational development. Distance Education. 19(1), pp. 43-63.

Notice that action research is not just ‘trial and error’ or ‘hit and miss’ – it is targeted and systematic. Action research is only one type of education research, but it is particularly suitable for practitioners.

Task 12.6 An example of action research in tertiary mathematics

Read this conference paper that is an example of action research in mathematics (Nelson, 2004). Consider the issues that prompted the intervention; the systematic collection of evidence to evaluate the intervention; and the implications of what was found.

Task 12.7 Planning your own action research

Returning to your own data collected in task 12.4, is there a small action research project that you could implement in your teaching? Some ideas might be:

Making a change to lecture delivery by including a: 'minute paper'; a think, pair, share activity; using electronic voting technology (see Module 7 for a more detailed description of these tools).
Making a change to your tutorials: by introducing some self or peer assessment of problems, giving students a choice of problems to solve, asking students to explain their solutions to problems to a peer.
Making a change to the technology you use: introducing the use of tablet PCs to your classes.
Introducing a peer study-group session for your class.

In your own work, a more formal approach may be to work through the project plan below:

Introduction to the unit	Background to the unit What is the unit about? How many students are enrolled in the unit? How is it delivered? How does it fit in with the course of study? What are the characteristics of the students?
The issue	What is an issue that you have identified and how have you identified it? e.g. student feedback, employer feedback, SET results, self review, peer review
The intervention	What do you intend to do to address the issue?What has led you to choosing this intervention? e.g. benchmarking with another unit, brainstorming with students/colleagues, literature
Evidence	What evidence will you collect to evaluate the success of your intervention? When will you collect this? How will you analyse the evidence?
Results(after you have collected the evidence)	What have you found out from the evidence that you have collected?
Reflection	Has the intervention been successful, what other actions are suggested (e.g changes to assessment structure, introduction of new teaching tasks, collection of further evidence for evaluation, undertake external peer review).

Note that if you wish to publish an account of an action research project (depending on the type of evidence you collect) you may require ethics approval which you have to obtain before you carry out the research. For instance, ethics approval by the university is often required for situations where you are directly involving students in the research. This is not an area within the expertise of most mathematicians, so you should be aware of this and not assume that it is not required. If you need some information about this you could contact the academic development centre at your university, and many departments have a staff member who is available for advice.

Don’t forget to check out the further reading (below) for this module, which lists places where you may access, or even publish your own, research into the teaching of mathematics.

Review and conclusion

Evaluating your unit leads to the following actions and outcomes:

You have evidence about whether your teaching is achieving the learning outcomes required.
If the unit is successful in achieving the learning outcomes, you could communicate this to students and to your peers – perhaps even in the Gazette of the AustMS or the Statistical Society Newsletter.
If the learning outcomes are not met, make changes and communicate these to students. Again, collect evidence so that you know whether the changes have improved the outcomes.
Encourage students to give you feedback. Close the loop – communicate to students about the changes you have made as a result of previous feedback.Please communicate your unit development – positive and negative – to others through conferences and journals. Evidence of results will make these publishable in peer reviewed journals and will help other mathematics educators.
Good evidence on your units will help with promotion.
The main advantage of collecting and acting on evidence is the benefit to student learning.

You have now finished the last module. Congratulations! We will conclude the course with some of our final thoughts and questions. Please continue to the final wrap-up.

References

Centre for the Advancement of Teaching and Learning. (2009). The Framework. Retrieved February 22, 2011, from http://www.catl.uwa.edu.au/tcf/the_framework
Kember, D. (1998). Action research: Towards an alternative framework for educational development. Distance Education. 19(1), pp. 43-63.
Nelson, M. A. (2004). Reforming the Teaching of Calculus: Oral Exams. Paper presented at the annual meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education, Toronto, Canada. Retrieved 26 May, 2009, from http://citation.allacademic.com/meta/p_mla_apa_research_citation/1/1/7/7/0/pages117708/p117708-1.php
University of Tasmania. (2009). Feedback to Students. Retrieved February 24, 2011, from http://www.studentcentre.utas.edu.au/setl/Students/feedbackstudents.html