Trialling the use of a mathematics diagnostic assessment task
Keith McNaught |
Category: Refereed Research [ Refereed papers ] |
Mathematics units often present the greatest difficulty for enabling pathway ('bridging course') students for a wide range of reasons. Many of these students have limited prior mathematics content knowledge, and report, like much of the general population, of feeling phobic towards mathematics and lacking confidence. At the University of Notre Dame Australia (UNDA), Fremantle campus, a student's final mark in their mathematics enabling unit often results in failing to meet the institutionally required benchmarks to move into undergraduate study. As staff experimented with a diagnostic mathematics assessment for teaching and learning purposes, a new proposition emerged: Could diagnostic assessment be used to help students make an informed decision about their choice of enabling program, and select a less intense course, without mathematics in the first semester? The benefits in having a student in the 'right course' should improve retention, and increase the number of students successfully transitioning to undergraduate study. UNDA is piloting the use of the diagnostic mathematics assessment to help students select the specific enabling program (either the "Tertiary Enabling Program" (TEP) or "Foundation Year" program) best suited to their background and experience.
The TEP program attracts approximately 200 entrants in first-semester, and approximately 90 in a smaller mid-year intake. Many entrants come from backgrounds of disadvantage (e.g. low socio- economic background, first in family to university, educational disadvantage within schooling) and are accordingly recognised as a skewed entry group, requiring intensive academic input and strong pastoral support (Levy & Murray, 2005). EP001 Learning Skills, the first coursework unit, is delivered as week-long intensive course prior to all other units commencing, and orientates students to university level study and expectations, and negotiates understandings around independent adult learning principles (Knowles, 1982). Staff and students alike understand that an enabling course is a bridge to future undergraduate study; with few exceptions, it is not a course completed 'for its own sake'. However, some entrants express a lack of clarity about their vision for the future; the TEP may be viewed by them as experiential. Some entrants acknowledge that their parents have 'strongly encouraged' them to undertake this course, as they have not gained undergraduate entry, and are not in full time employment.
All TEP students complete a total of seven units, including the following six core units.
Students at the University of Notre Dame Fremantle campus who complete the TEP are required to achieve a minimum of 65% in every unit of study in order to be considered for approval for undergraduate course admission. The 'minimum 65%' was determined on the basis of tracking TEP students over a period of five years, and ascertaining the point at which these students would be expected to make a smooth transition to successful undergraduate study. Such success was measured in terms of academic progress, via grade point average (GPA), and retention data. Since 2005, the percentage of students achieving the required benchmark (65% in all units) has varied considerably, from 25% of students in semester one, 2009, to 65% in semester two, 2010. Students who complete the course are regularly offered places at other universities, who accept a pass (50%) in all units, rather than the institutionally required benchmark (65%). It is impossible to track the success of students moving to other institutions to complete undergraduate study, other than incidentally and anecdotally. UNDA has data which demonstrates that TEP graduates perform at least as well as other entrants, as undergraduate students. Retention rates for TEP graduates, in undergraduate courses, are markedly higher than other alternative entry pathways, including mature age students and Certificate IV graduates.
The problematic nature of lacking fundamental mathematics, as an undergraduate student, is noted throughout the research: "Students who lack the basic and fundamental skills, especially in mathematics and writing, are finding it difficult to cope with the normal course workload" (Lau, 2003, p. 2).
In 2010, The University of Tasmania investigated the role of numeracy within all courses offered at the institution, noting the importance of numeracy when it was both explicit and embedded (Skalicky et al, 2010).
All TEP students complete EP005 Mathematical Competency, a three hour per week, 13 week unit, which covers the areas of Number, Measurement, Geometry, Algebra and Data. The content of EP005 does not exceed that usually expected of students in 'general mathematics' courses at Year 10, and is thus described as a 'low cognitive load' course.
The TEP is fairly described as an 'intense course' with multiple units which traverse English, Mathematics, and Information Technology. Understandably, many students find this challenging. There are several likely reasons why the students in TEP have such difficulty with this mathematics unit. Firstly, many students come to the TEP without having completed a high level of, or in some cases any, mathematics at Year 12. In Western Australia students choose subjects in stages 1, 2 or 3 of various curriculum offerings; however, as an example, in Semester Two, 2011, not one EP005 student had completed a stage 3 mathematics course, and less than 40% had completed a stage 2 mathematics course. Tracking demonstrates those students without at least mathematics at stage 2 (or its equivalent) in Year 12 struggle with the mathematical content which is positioned within this course. Secondly, many students who have undertaken lower level mathematics courses in secondary schools have made extensive use of calculators for all computational work. The entire EP005 mathematics program is positioned around non-calculator use, and for many students who enter this course, their dependence on calculators appears, over time, to leave them lacking skills with straightforward mental mathematics. The errors in questions on the diagnostic test, which require students to multiply whole and decimal numbers by 100, illustrate the limited mental mathematics skills of participants. That large numbers of students have trouble with such questions alludes to broader issues of teaching and learning experiences during 12 years of formal schooling.
As the TEP is an intense course, EP005 is completed simultaneously with six other units of study and so, whilst the mathematics content itself is not onerous, there is a considerable concurrent work load for a student. Students in many cases choose to focus their study time on the subjects they enjoy more or the subjects that they think might be more beneficial to them, if they underestimate the importance of numeracy to any university course.
Lastly, the first task completed in EP005, a written personal reflection on mathematics, provides evidence every semester that many students enter this course with negative prior mathematical experiences, and at least some students are reluctant to engage with mathematics, or phobic, or anxious, as they commence this unit. A student with an unconstructive approach to mathematics may have reduced confidence, and such negativity can impact on self-esteem, inhibit functioning, and interfere with performance (Shapka & Keating, 2003). That anxiety and confidence levels predict mathematics performance better than standardised measures of quantitative ability, demonstrates the significance of psychological factors inherent in mathematics teaching and learning (Ironsmith, Marva, Harju & Eppler, 2001).
TEP course coordinators and EP005 teaching staff wrestled with the issues. Students were finding this unit the most problematic, and in many cases it was the unit that ultimately prevented them moving to undergraduate study, as they failed to achieve the benchmark in this unit alone. Many students were ill prepared for mathematics, in particular within a 13 week program, and at least some students needed time to begin to address their negative attitudes towards mathematics.
Every profession recognises the importance of early identification and intervention (Gale, et al, 2010). The principle is that the earlier intervention occurs, the more effective it will be (Rogers, 1996). Moreover, early intervention is associated with reduced costs over time (McCain & Mustard 1999; Gauntlett, Hugman, Kenyon & Logan 2000). Early intervention in an academic setting is ethically responsible behaviour, and there is solid evidence that proactive work is more effective than remediation (Tumen, Shulruf & Hattie, 2008). In interpreting the results of the University of Auckland's diagnostic screening tool, Elder and von Randow (2008) note that students identified as 'mainly satisfactory' are unlikely to obtain high grades in their first-year subjects. Furthermore, they note that students identified as being 'at risk' are "at risk of failure in one or more subjects" (p. 176).
In the case of EP005, all the content and assessment for each semester is prescribed before any knowledge of the participants is known. This is a conundrum for higher education - where units are tightly planned, with all content and assessment prescribed, without reference to the incoming student cohort and their particular strengths and needs. It might be logically implied that the results of a diagnostic assessment could potentially necessitate a significant change to the teaching and learning plan. However, University regulations around the pre-approval of unit outlines prevent changes without a rigorous and time-consuming process being followed. Whilst best teaching practice might be to change content sequencing, topics to be covered, or the dates that assessments are completed, none of these changes are permitted. A rationale for diagnostic testing for a unit such as EP005, whilst educationally and ethically sound, designed to improve the quality of teaching and learning, might well be thwarted by university processes which reduce flexibility. Academic staff need to be able to use the results of diagnostic testing within the set parameters, and to maximise the flexibility available to them professionally. The innate complexities are pronounced by the very specific time constraints within tertiary units. In the case of EP005, a 13 week timeframe to address mathematics skills, content and attitudes, perhaps reflective of 12 years of formal schooling, is an immediate challenge.
Diagnostic assessment mechanisms can be powerful for the more 'able' students (Harrington, 2005), not just those flagged 'at risk' (Elder & von Randow, 2008). In practice, it may be easier to modify teaching and learning for more able students (Alderson, 2005), as they are more able to self pace through the required content. Moreover, they are likely to become dissatisfied if held back in their work, in order to work at a pace more suited to less skilled peers. The mathematics text aligned to EP005 has the advantage of having a parallel set of tasks for each topic, but at an 'extended' level; however, staff discussions demonstrated that these were seldom used or recommended, without logical explanation being available. This lack of use might indicate that staff were more focused on the needs of 'at risk' students, rather than the provision of a differentiated curriculum (Forster, 2004), for more capable learners.
At the end of EP005, semester two, 2010, in an analysis of the final results of students occurred, again, largely informally and through collegial discussions, but it was clear that a 'critically low score' in the diagnostic assessment was linked to students failing to achieve the minimum benchmark of 65% in the unit. No student who achieved a score less than 40% achieved an overall EP005 result of 65% or greater. Of the 6 students who achieved the score of 40% in the diagnostic test, only 2 (33%) achieved the necessary 65% at the end of the unit. Essentially, a cathartic change of thinking occurred with the team working with the diagnostic assessment. Instead of thinking about such an assessment as a way to inform teaching and learning, and increase student self-awareness, a new hypothesis had emerged: Could this diagnostic assessment be used to help students make an informed decision about their choice of enabling program, and select a less intense course, without mathematics in the first semester? Moreover, could delaying a mathematics course be a better option for the transition of some enabling program students? An effective enabling program should assist students to become self-regulating, independent learners, and to use reflective mechanisms about their own skill set, to ensure the likelihood of future success (Wright, 2010).
Students entering UNDA to complete a bridging course have the option of undertaking an alternative enabling program to TEP, the year-long Foundation Year Program. The Foundation Year Program does not have a mathematics unit in first semester and is structured very differently to the TEP. Foundation Year students complete four units linked to academic literacy only, in semester one: Learning Skills, Literacy Competency, Academic Writing and Research Skills. These are four of the seven units also completed by the TEP students. In their second semester, Foundation Year students complete four undergraduate units, in a chosen stream, which may later be used for advanced standing in their undergraduate degree. Foundation Year students will first encounter mathematics units as undergraduate students, if so required in their course of study. The advanced standing option, from their second semester program, will potentially permit them to complete less units in the semester in which they complete a mathematics course, thus enabling more time to be devoted to this area, if so required. The AESC offers primer courses for undergraduate mathematics units, and also free tutoring services, creating additional support mechanisms for students wishing to engage with these services. On a superficial judgement, many students choose the TEP Program, over the Foundation Year, because it can be completed in one semester. However, students who are unlikely to achieve success in an intense course, such as the TEP, may be better advised to complete the Foundation Year.
Prior to the enrolment of students within the TEP and Foundation Year, for semester two, 2011, a flowchart was collaboratively created for use by the University's Admissions Office. All applicants who do not meet the requirements for direct entry to undergraduate study are usually offered a place within an enabling program, and many have placed this as a 'second preference' in their application to the University. This flowchart was designed to place students in the course deemed best suited to their background. Essentially, the flowchart used stage two mathematics courses as the discriminator; students who had completed stage two mathematics were recommended for TEP, those without stage two mathematics were recommended for the Foundation Year. resulted in more than 30 students who had applied for TEP being offered, and accepting, a place in the Foundation Year program. The mathematics skills of those 30 students were not tested, but all were offered the opportunity to complete the diagnostic assessment, in order to make a fully informed decision. None chose to do so.
Figure 1: Marks of all students, diagnostic test, semester one, 2011
The students were told, prior to completing the assessment, that the task was designed primarily to give them an indication of their current skill set in the topics that would be covered, and to use this information to predict the time, work and effort that would be required to achieve the necessary benchmark. The students were informed that additional, optional, no cost tutoring would be available for interested students, but that it would be provided contingent on student attendance. The students were informed that students from semester two, 2010, who had achieved a mark of less than 40% (in the diagnostic test), had not achieved the benchmark of 65% in EP005. Staff were explicit with students that the information was useful, but not conclusive - hypothetically, a student with a low score could do very well in the unit, and to use the data as a guide, not a guarantee.
Figure 2: Student performance in EP005 compared with Diagnostic assessment performance
No student who achieved a mark less than 40% in the semester one, 2011, diagnostic test, achieved the required benchmark of 65% for EP005 (see Figure 2). This replicated the same results from the Semester Two, 2010 data.
Proportionately, males performed better than females in the diagnostic test. Of the 59 students who had a result of less than 50% in the Diagnostic Test, 86.4% are females, 13.5% are males; of the 65 students who achieved >65% in the Diagnostic Test, 55.3% are males and 44.6% are females. Given that the gender balance of EP005 overall (176) is 68.2% female and 31.8% male, a disproportionately high number of females achieved low results in the diagnostic test. However, although females perform poorly, compared to males, in the diagnostic test, females are more than likely to perform well (achieve >65%) in the EP005 unit than males. Of the 82 students to achieve an EP005 final mark >=65%, 65.8% are female and 34.1% are male. Reasons for female students making significant gains in their mathematical skill levels warrant future examination. It is hypothesised that the pedagogical approach, focused on both procedure and conceptual understandings, and an intentionally explicit teaching approach, linked to linear progression within topics, is a contributing reason, based on the student feedback contained within unit evaluation forms, and subsequent discussions with the staff involved. As noted earlier, EP005 also has a strong focus on building personal confidence with mathematics (Rutter, 2001), which warrants further investigation as a gender issue.
The semester one, 2011, TEP cohort was a high achieving group, across all units, with the exception of EP005. The fail rate for all courses, other than mathematics, was less than 3%, yet 18.3% (six times the rate) in EP005. The percentage of EP005 students to achieve the minimum of 65% was significantly lower than all the other units. It is of note that this particular cohort, with 54.5% achieving the minimum benchmark, was the highest rate of any semester since 2006. Whilst staff working within EP005 were pleased to see the increased success across the cohort, and noted the gains made by female students, they remained disappointed that EP005 still had a significant performance tail, and only 54% of students met the required 65% benchmark.
A diagnostic assessment may serve several purposes, including being a predictor of future success. It is apparent that scores, on a diagnostic mathematics test, within a low range, indicate that TEP students would be unlikely to achieve the required benchmark for the unit. Students complete enabling programs as a bridge to undergraduate studies and the success of an enabling program can be measured, to a large degree, on future undergraduate success. Unrealistic expectations or naivety are likely to result in both failure and disappointment. The use of diagnostic assessment has the potential to assist enabling students to make informed choices, and to create a study plan based on realistic understandings, which demonstrate a course's suitability, and the level of workload anticipated for future success. Given that many universities offer a range of enabling programs, diagnostic assessment could be a useful tool to assist students to select the program which provides the best chance of successful completion.
Arem, C. (2003). Conquering Math Anxiety (2nd ed.). Pacific Grove, CA: Brooks/Cole.
Elder, C., & von Randow, J. (2008). Exploring the utility of a web-based English language screening tool. Language Assessment Quarterly, 5(3), 173-194.
Forster, J. (2004). Quality practice: Implementing differentiated teaching and learning. Australasian Journal of Gifted Education, 13(1), 28-37.
Gale, T., Tranter, D., Bills, D., Hattam, R., & Comber, B. (2010). Interventions early in school as a means to improve higher education outcomes for disadvantaged (particularly low SES) students. Canberra: DEEWR.
Galligan, L. & Taylor, J.A. (2008). Adults returning to study mathematics. In H. Forgasz, A. Barkatsas, A. Bishop, B, Clarke, S. Keast, W. Seah & P. Sullivan (Eds.), Research in Mathematics Education in Australasia. (pp. 99-118). Rotterdam: Sense.
Galligan, L. (2004). Preparing international students for university: Mathematics as part of an integrated program. Senior Mathematics Journal, 18(1), 28-41.
Gauntlett, E., Hugman, R., Kenyon, P. & Logan, P. (2001). A meta-analysis of the impact of community-based prevention and early intervention. Commonwealth Department of Family and Community Services, Canberra, ACT.
Harrington, S. (2005). Learning to Ride the Waves: Making Decisions about Placement Testing. WPA: Writing Program Administration, 28(3), 9-29.
Ironsmith, M., Marva, J., Harju, B., & Eppler, M. (2003). Motivation and Performance in College Students Enrolled in Self-Paced Versus Lecture-Format in Remedial Mathematics Courses. Journal of Instructional Psychology, 30(4), 276-284.
Knowles, M. (1982). The Modern Practice of Adult Education: From Pedagogy to Andragogy (2nd ed.), Cambridge Books, New York.
Lau, L. K. (2003). Institutional factors affecting student retention. Education, 124(1), 126-136.
Levy, M., & Murray, J, (2005) Tertiary Entrance Scores Need Not Determine Academic Success: An analysis of student performance in an equity and access program. Journal of Higher Education Policy and Management, 27(1), 129-140.
McCain, M. N. & Mustard, J. F. (1999). Reversing the Real Brain Drain: Early Years Study Final Report. Toronto: Ontario Children's Secretariat.
McNaught, K. & Hoyne, G. (2011) Mathematics for first year success. Proceedings of the 14th Pacific Rim First Year in Higher Education Conference, Fremantle June 29 - July 1st, 2011.
Parsons, S. & Bynner, J. (2005). Does Numeracy Matter More? National Research and Development Centre for Adult Literacy and Numeracy, Institute of Education, London.
Rogers, S. (1996). Brief report: Early intervention in autism. Journal of Autism and Developmental Disorders, 26(2), 243-246.
Rosenshine, B. & Meister, C. (1992). The use of scaffolds for teaching higher-level cognitive strategies. Educational Leadership, 49(7), 26-33.
Rutter, M. (2001). Psychosocial adversity: Risk, resilience and recovery. In J. M. Richman & M. W. Fraser (Eds.), The context of youth violence: Resilience, risk, and protection (pp.13-41). Westport, CT: Praeger Publishers.
Shapka, J.D. & Keating, D.P. (2003). Effects of a girls-only curriculum during adolescence: Performance, persistence, and engagement in mathematics and science. American Educational Research Journal, 40, 929-960.
Skalicky, J., Adam, A., Brown, N., Caney, A. & Lejda, A. (2010). Tertiary Numeracy Enquiry. Centre for the Advancement of Learning and Teaching (CALT), University of Tasmania.
Tumen, S., Shulruf, B., & Hattie, J. (2008). Student Pathways at the University: Patterns and Predictors of Completion. Studies in Higher Education, 33(3), 233-252.
Please cite as: McNaught, K. (2012). Trialling the use of a mathematics diagnostic assessment task. In Creating an inclusive learning environment: Engagement, equity, and retention. Proceedings of the 21st Annual Teaching Learning Forum, 2-3 February 2012. Perth: Murdoch University. http://otl.curtin.edu.au/tlf/tlf2012/refereed/mcnaught.html |
Copyright 2011 Keith McNaught. The authors assign to the TL Forum and not for profit educational institutions a non-exclusive licence to reproduce this article for personal use or for institutional teaching and learning purposes, in any format, provided that the article is used and cited in accordance with the usual academic conventions.