ALTC Grants Scheme -
Evaluating Projects
 

7. What resources and skills are required to conduct the evaluation?

  • Who will conduct the evaluation?
  • Should the evaluation be undertaken by an individual or team?
  • Should the evaluation be undertaken by insiders or outsiders?
  • What are the key issues to consider in engaging an evaluator?
  • At what stage of the project should the evaluator become involved?
  • What is the budget required for the evaluation activities?
In planning a project, consideration needs to be given to the human, material and other resources needed both for the overall project and for its evaluation component. Planning the budget in detail for the evaluation component will enable the insertion of an evaluation line item in the overall budget. Careful costing of each of the proposed project activities and of the project's operation and management needs to be carried out to ensure that the project is feasible, and that its budget components can be justified. Reference should be made to the current ALTC Grants Program Guidelines that list the items for which the budget can provide, along with certain specified maximum amounts and rates, plus items for which grants may not be used. A draft budget template is provided in Figure 4 at the end of this section.

Who will conduct the evaluation?

This is a key question underpinning the budget. This may be considered in terms of two sub-questions –

i. Should the evaluation be undertaken by an individual or by a team?

There are advantages and disadvantages for each option. Having a team means finding common times for meetings and other activities, developing shared understanding on procedures, data analysis and findings, pulling together possibly different writing styles, and meeting joint deadlines. Working as an individual removes these constraints. A team however may enable the evaluation to tap into a range of specialist skills relating to planning, data gathering and analysis, and reporting, which the individual may not necessarily have to the same levels. The range of skills may also enable allocation of specific tasks to team members, thereby sharing the load and individual time commitment. A team may also provide a variety of perspectives that can be brought to the evaluation, providing an inbuilt 'sounding board'.

ii. Should the evaluation be undertaken by insiders or outsiders?

An insider is defined here as anyone who is directly involved in the development or operation of the project being evaluated or who has a direct stake in the evaluation's outcomes. An outsider is defined as anyone who is not directly involved in the development or operation of the project being evaluated or who does not have a direct stake in the evaluation's outcomes.

Insiders may be located externally as well as internally. An example of an external insider would be an outside organisation providing professional services of a type that would be needed if the project were deemed to be a success and its operation extended eg a professional association. By the same token, outsiders may be located internally as well as externally. An example of an internal outsider would be an individual in the project institution from another Faculty or School who has no actual, potential or perceived stake in the project or its evaluation outcomes.

Insiders, particularly internal insiders, carry advantages in that they often will have detailed understanding of the context in which the project is operating, along with ready-made points of contact for information gathering. Being known to the parties involved, they may be seen as less threatening and thus find it easier to elicit information. Their involvement may moreover be costed at a relatively low level, to the extent that it is seen as a part or an extension of their continuing substantive role in the institution.

Having a direct stake in the project or the evaluation's outcomes raises however the issues of actual, potential and perceived conflict of interest. Essentially the insider cannot be seen as providing an evaluation that is independent. This raises questions of credibility and reliability, which in turn may diminish the usefulness of the evaluation's findings in informing future decision-making.

Involving an outsider will help in establishing an independent process. An independent process is desirable in any project evaluation, and for projects receiving ALTC funding of more than $150,000 it stands as a formal requirement 1.

Involving an outsider also carries other advantages. It may enable the acquisition of specialist evaluation expertise and experience that may not be present to the same extent among insiders, or at least not readily available at the time when it is needed. Outsiders can bring new perspectives and a sense of impartiality to the evaluation, coming with fresh eyes to the project and its operation.

Consideration might also be given to establishing an evaluation team that includes both insider and outsider members, to capitalise on the advantages of both sources – detailed understanding of the project's operating context, ease of access to information, a range of skills and perspectives, and a degree of impartiality.

Finally, in relation to gaining access to outsider expertise, the role of the Project Reference Group warrants close consideration. The Grants Program Guidelines state that

'All project teams should appoint a reference group. The reference group should include some external participants who have appropriate expertise to ensure there is constructive advice on the design, development and ongoing evaluation of the project and to ensure the project has maximum impact within the institution/s engaged in the project and beyond those institutions.'

While the Reference Group will not carry out the evaluation, careful selection of its outsider members should provide a useful sounding board and sources of expert advice in relation to the evaluation's development and implementation.

What are the key issues to consider in engaging an evaluator?

A first step in engaging an evaluator is to prepare an evaluation brief or terms of reference. This is a statement that should give a prospective evaluator sufficient information to prepare an evaluation proposal. It is important not to be too specific in relation to the evaluation methodology in the statement, leaving some freedom for the prospective evaluator to use his/her expertise and experience to propose detailed ways of proceeding.

The evaluation brief or terms of reference would normally include at least the following –

  • a brief outline of the project (background, processes, outputs, timeline and intended outcomes);
  • the purpose(s), focus and scope of the evaluation;
  • any preferred or required information gathering sources and techniques;
  • the roles of the Project Manager, Project Team and the Project Reference Group in relation to the evaluation and to the evaluator;
  • reporting requirements and timelines;
  • the budget available for the evaluation (generally 5-10% of the project budget); and
  • qualities expected of the evaluator.

The qualities expected of the evaluator include those expected of evaluators in general as well as those that relate to the particular project. The qualities would normally include –

  • project evaluation experience in higher education, and ideally in the discipline or area of the project;
  • broad understanding of the discipline or area of the project;
  • skills in quantitative and/or qualitative data analysis, as appropriate to the project;
  • high level oral and written communication skills;
  • independence;
  • capacity to meet the project's evaluation timelines; and
  • willingness and capacity to work with the Project Manager, Project Team and the Project Reference Group, as required.

At what stage of the project should the evaluator become involved?

The evaluator should be involved in discussions with the Project Team as early as possible, and ideally before the project is underway. An internal evaluator may be already known and could be involved in the grant application. This allows for critical examination and shared understanding and endorsement of the details of the evaluation. It also enables timely planning of how the evaluation procedures can dovetail with the project and become an integral part of it rather than an add-on. The evaluator at this stage can act as a sounding board for the team, asking questions that will enable greater clarity and precision to be attached to planned processes and outcomes. Early and continuing involvement of the evaluator along these lines should also deepen the evaluator's understanding of the project context, thereby further developing the potential of the evaluation to generate rich insights into the factors influencing the project outcomes.

Figure 4. Project Evaluation Budget Worksheet

Budget/cost item

Budget estimate

Salaries/wages

  • External consultant(s)
  • Internal staff
    • Project leader
    • Project manager
    • Other staff
  • Casuals

 

Training & staff development

  • Workshop facilities
  • Materials
  • Catering
  • Other

 

Data collection

  • Evaluation tools
    • Purchases/development
    • Printing
    • Distribution & collection
  • Other collection costs
    • Hire of equipment/facilities

 

Travel & accommodation

  • Travel costs
  • Accommodation
  • Meals etc.
  • Other

 

Data processing & analysis

  • Hire/use of equipment/facilities
  • Purchase of software
  • Other

 

Printing of reports

  • Printing
  • Mail out

 

Miscellaneous office costs

  • Telephone, fax
  • Meeting room hire
  • Stationery

 

Total

 

Adapted from University of Tasmania, Centre for the Advancement of Learning and Teaching Project Evaluation Toolkit: http://www.utas.edu.au/pet/index.html

1 Current ALTC Grants Program Guidelines state that 'recipients of grants in excess of $150 000 are required to commission a formal independent evaluation of the project. This may be funded from the ALTC grant and should be included in the project proposal budget.'