September 2009 — Volume 3, Issue 5

Student and graduate satisfaction - tensions and synergies

The Course Experience Query (CEQ) is a useful tool for exploring gaps in student perceptions, especially for probing differences in quantitative outcomes produced by internal institutional surveys, according to Professor Beverley Oliver and Julie-Ann Pegden.

Professor Oliver, Director of Teaching and Learning at Curtin University of Technology and the institution’s evaluation analyst Ms Pegden presented their paper ‘External and Internal Indicators of Student and Graduate Satisfaction: Tensions and Synergies’ at the Australian Universities Quality Forum in June.

Based on a case study conducted in a large undergraduate degree program at Curtin, the investigation was prompted by the very different results gained from internal and external surveys of student and graduate experiences. The study sought to explore anomalies in overall satisfaction, investigating why positive overall satisfaction in Curtin’s internal survey by final year students does not always translate into similarly positive overall satisfaction in the CEQ and the eVALUate Graduate Survey.

Curtin implemented eVALUate in 2006 with two surveys in the online suite of particular interest to this study – the eVALUate graduate survey and an eVALUate unit survey which has 11 quantitative and two qualitative items seeking feedback on what aspects of a unit helped and hindered students achievement of learning outcomes as well as their engagement, motivation and overall satisfaction. The eVALUate graduate survey, however, is aimed at on and offshore graduates of up to five years and invites respondents to indicate their perceptions of how their course helped them achieve Curtin’s graduate attributes in addition to preparing them for their profession.

This survey has 13 quantitative items that map Curtin’s graduate attributes as well as one item that asks respondents to indicate overall satisfaction. Both eVALUate surveys use response scales ranging from “strongly agree, agree, disagree, strongly disagree to unable to judge” while the CEQ offers five points on a scale from “strongly agree” to “strongly disagree”.

The three surveys sometimes reveal very different aggregated quantitative results, for example, in measures of overall satisfaction. For the case study, Professor Oliver and Ms Pegden triangulated the results from the three surveys to report on one quantitative item (overall satisfaction) and then attempted to account for apparent differences through an analysis of qualitative comments.

The authors say the comparison of results is imperfect and not intended to be a definitive solution while also noting that each instrument captures comments from respondent groups that are not necessarily identical because they are targeted at different cohorts in different contexts.

Notwithstanding these caveats, Professor Oliver and Ms Pegden say it has been interesting to compare emphases in comments. Satisfaction with units in final semester, as revealed through the eVALUate unit survey, is around 80 per cent, while four months later, about two-thirds of graduates from the same cohort say they are satisfied with the quality of their course. The one-off eVALUate graduate survey shows that about 80.5 per cent of graduates agree that overall they are satisfied with their preparation as a result of the course and believe it prepared them for employment.

Across the three surveys, the authors observed that final year students and graduates comment most on methods of teaching and learning, quality of staff and flexibility in course design. As time passes there is less emphasis on staff accessibility, learning resources and the teaching skills of staff. There is, however, a growing emphasis over time, particularly from graduates five years out, on intellectual outcomes, knowledge, skills and application. In these subdomains the odds of a comment being “best aspect” are lower with graduates of up to five years less likely to make that comment. Subdomains recording a steep decline in positivity are staff accessibility, relevance in course design, intellectual outcomes, work application and practical and theoretical links. These aspects become more pronounced in the “needs improvement” comments which are an indication of student dissatisfaction. Areas attracting increasing emphasis and higher negativity relate to three domains – staff, outcomes and course design which the authors say
can be further explored by drawing on the specific definitions of CEQuery subdomains.

They say these pronounced changes in attitudes to staff may relate to the surveys with research on the CEQuery tool revealing that the good teaching scale has a particularly strong association with all the CEQuery domains. “This means that positive comments on staff reliably predict high ratings on the majority of the CEQ scales and negative comments predict low ratings,” according to Professor Oliver and Ms Pegden.

Hotspots in students’ comments in the CEQuery include the staff domain where more negative remarks around teaching skills and quality and attitudes are recorded. Recurring themes highlighted in the Curtin case study as needing improvement include graduates’ increasing perception that their course needed more real world application which prepared them for the world of work, was more flexible and more engaging. “A closer reading of individual comments also show an emphasis on needing career development skills and assistance.”

Overall, the authors argue that the Curtin case study shows the usefulness of the CEQuery tool in determining emphases and levels of positivity and negativity in student comments. It also allows academic leaders to map their courses results against broad indicators provided by field of education based on the CEQuery research project which analysed in excess of 160,000 comments made by graduates from 14 Australian universities.

The paper on the Curtin case study is part of the 2009 AUQF proceedings which can be accessed at www.auqa.edu.au

*Professor Oliver was recently appointed an ALTC Teaching Fellow.