ALTC Grants Scheme -
Evaluating Projects

6. What are the criteria for making judgements about the findings of the evaluation?

Examples of criteria questions might include:

• To what extent have the intended student learning outcomes been achieved?
• How well have the needs of staff been met?
• How appropriate were the project activities in relation to staff capabilities and the institution's ICT structures?

The making of judgments lies at the core of evaluation. For project evaluation, this may involve, for example, making judgments on the project outcomes, identifying strengths and weaknesses of the project's processes, determining how well the project has met stakeholder needs, or deciding the extent to which the project outcomes are sustainable. Such judgments will play a key role in any decision-making that the evaluation is intended to inform. In general, judgments will be required for each key evaluation question.

In planning an evaluation, it is important to identify the standards by which such judgments will be made, i.e. the evaluation criteria. Examples of evaluation criteria include:

• achievement of the project goals, objectives or intended outcomes
• needs of stakeholders such as students, staff and the funding body
• set standards in the specific field of the project
• best or good practice
• ideals or social/political values and expectations
• the quality of alternatives
• relevance
• effectiveness
• efficiency
• appropriateness
• sustainability
• potential usability for others
• dissemination among stakeholders.

Often these criteria are expressed in terms of extent of achievement or performance, for example,

To what extent have the intended student learning outcomes been achieved?
How well have the needs of staff been met?
How appropriate were the project activities in relation to staff capabilities and the institution's ICT structures?

Often more than one criterion will be adopted for an evaluation, depending on the nature and range of judgments required. Once the particular criteria have been selected, further specification is normally needed in order to clarify how they will be applied. Thus, for example,

Extent to which the needs of staff, students and the funding body have been met – specification of particular groups of staff and students, and of the specific needs of each group that provide the focus.

Set standards in the specific field of the project – identification of particular standards and the actual levels that are regarded as acceptable.

Best or good practice – naming of the locations or source(s) of the practices adopted as the benchmark, and of the particular dimensions or aspects of those practices that will be used for comparison purposes.

Relevance or effectiveness or efficiency, in terms of specific project processes and outcomes.

Sustainability – identification of what aspects are deemed worthy of sustaining, e.g. particular outcomes or project structures or associated changes in approaches.

Ideally, the level of specification would be such as to enable direct measurement, using either qualitative or quantitative information or both, and judgment. Not all criteria are necessarily amenable however to this level of specification. At times it will be a matter of identifying a number of indicators of performance that together will enable the evaluator to make the judgment.