External mid term evaluation of the project Partnerships for Quality and Relevance in ICT Vocational Education in Moldova, implemented in partnership with National Association of Private ICT Companies, funded by ADA, USAID and Romanian Government.

Project title: Partnerships for Quality and Relevance in ICT Vocational Education in Moldova
Country: Republic of Moldova
Project number: 8295-00/2015
Project duration: 01.12.2015 – 30.11.2018

1. Background

Since its independence, the Republic of Moldova economy has suffered from a lack of efficiency, partly due to serious decline in quality of education. The Education Strategy 2020, adopted in 2014, sets out as main priorities for the Government improving the quality, relevance and efficiency of the education system. The Vocational Education Strategy 2013-2020, adopted in 2013, is the main policy document in the field vocational/technical education, that aims by 2020 that VET will become more attractive, quality, relevant for the labor market, accessible, career oriented, flexible, allowing nonformal and informal learning validation and mobility in the European space in line with Copenhagen Process and Bruges Communique provisions.

The current project is a second phase of the project „Advancing quality in ICT vocational education in Moldova“ that supported the qualitative improvement of content and process of ICT professional education in 5 institutions (colleges and vocational schools), implemented in the period December 2012 – May 2015. Given the growing importance of the ICT for the economic development, it was decided to continue with raising the quality of ICT programs in VET schools. Thus, in December 2015, the second project with similar aim, but with extended objective, covering 18 VET schools and focusing on building the capacity of the Centre of Excellence in ICT. The new component on quality assurance was added to the project, as a result of the grant awarded by the Romanian Government and the Embassy of Romania/ADA agreement to implement the project according to the ADA rules.

The project Partnerships for quality and relevance in ICT vocational education in Moldova aims at assuring the quality and the relevance of ICT vocational education, through the support in launching and in qualitative functioning of the Centre of Excellence in VET ICT and harmonization of ICT education programs with labour market demands. Another component of the project is a support to the initial activity and functioning of the National Agency for Quality Assurance in Professional Education as an independent, functional and professional body.

Educational Centre PRO DIDACTICA (CEPD) is implementing the project in partnership with National Association of Private ICT Companies (ATIC).

The expected results by November 2018 of the project are:
1. An ICT Centre of Excellence for Professional Education established and functions as a resource hub for teachers and an attractive education provider to students in cooperation with the ICT sector;
2. The content and teaching methods of the ICT programs within VET institutions are modernized;
3. Information about career paths in ICT is made available to the public in a sustainable manner by VET institutions;
4. Basic standards, organizational procedures and work processes of the National Agency for Quality Assurance in Professional Education (ANACIP) developed to create a base for ANACIP to fulfill its public mandate.

The project aims at improving several aspects in the sphere of vocational education and training in Moldova by creating/improving:
• a functional ICT Centre of Excellence that will serve as model for other VET institutions
• a functional National Agency for Quality Assurance in Professional Education with development of organizational procedures and regulations
• the creation of new opportunities for VET students
• a more inclusive presence of students in the ICT field (by attracting more females to technical specialties and by attracting representatives of vulnerable groups such as minorities and persons with disabilities)
• the development of market driven curricula, with involvement of ICT companies
• sustainable cooperation models with the private sector to decrease the shortage in the ICT field
The target group for the project is the ICT Centre of Excellence and 18 institutions assigned to it, as well as the National Agency for Quality Assurance in Professional Education (ANACIP).

2. Purpose of the mid- term evaluation

The evaluation will focus on assessing the extent to which the activities, achievement of the results and objectives are on track of being implemented/achieved and to identify main barriers in their fulfilment/realisation.
• The evaluation is intended to contribute to improvement of policies, processes and methods of a development intervention.
• The evaluation contributes to the accountability towards the stakeholders and taxpayers, including clear statements if the objective and the results have been achieved.
• The evaluation makes recommendations for the last year of the project.

3. Objectives

The main objective of the evaluation is to assess and present results (output, outcome), conclusions, lessons learnt and recommendations.

The evaluation’s main goal is to document project achievements towards the results and targets set out in the initial project proposal and project developments. Furthermore, the evaluation should review and assess the relevance of the project activities, the progress made towards the achievement of the expected results and their progress towards sustainability. It should include a reflection about lessons learnt and provide recommendations for the project counterparts for the remaining project period.

The specific objectives of evaluation are:
a) the design and coherence of the project/programme including the design of the log frame matrix/programme theory and present the underlying theory of change and its assumptions.
b) the extent to which the project/programme has already achieved its objectives and results or is likely to achieve them, including the extent to which the lives of the project/programme beneficiaries (women, men, girls, boys) has already been improved. Also the extent to which supported institutions have already benefitted people.
c) Identifying potential risks or constraints that may affect the effective implementation of the project;
d) Provide recommendations on how to further improve the project implementation for the next months to maximize the impact of the project;
e) the extent to which cross-cutting issues (gender and environment mainstreaming) were applied..

4. Subject and focus

The subject of the evaluation is the project “Partnerships for Quality and Relevance in ICT Vocational Education in Moldova”, implemented in Chisinau and several regions of the Republic of Moldova, where VET schools are located.
The evaluator(s) will review and summarise the available evidence of the relevance, quality, efficiency, effectiveness, sustainability and impact of the project activities and will raise and address all relevant issues that may emerge during the evaluation.
The evaluation will consist of both desk and field study (few VET schools visits in other regions). ADA’s cross-cutting issues: gender, social inclusion and environment aspects have to be taken into consideration throughout the evaluation and reporting process.

5. Main evaluation questions

Relevance:

• To what extent are the objectives of the project/programme still valid for the partner country, the partner organization and the beneficiaries?
• Are the expected results/outputs of the project/programme consistent with the outcome, immediate impact and overall goal/impact (as part of the analysis of the logframe matrix/programme theory and the presentation of the theory of change and its underlying assumptions)?
• To what extent the activities implemented to date within the project was relevant and met the beneficiary needs?
• Will the curriculum developed within the project be applied nationwide?

Efficiency:

• Was the project/programme implemented in the most efficient way (time, personnel resources)? Have any issues emerged, if so which ones and why?
• If applicable, to what extent were all items/equipment purchased and used as planned under this project/programme?
• Are there any alternatives for achieving the same results with less inputs/funds?

Effectiveness:

• To what extent has the project/programme already achieved its expected results/outputs or will be likely to achieve them?
• What were the major factors influencing the achievement or non-achievement of the outcome(s)/expected results/outputs? (Also consider any which were possibly beyond the control of the project)?
• Have the project stakeholders worked together as planned? Are there any issues that need to be solved? If yes, which ones?
• To what extent was gender, environmental, social inclusion mainstreaming included in the project/programme and to what extent were recommendations from the ADA gender-assessment considered and implemented? Have any issues emerged, if so which ones and why?
• How likely will the project achieve its expected results by November 2018?

Sustainability:

• If the project/programme continues will it be integrated in local structures and/or funded by other sources?
• What were the major factors which influenced the achievement or non-achievement of sustainability of the project/programme?
• If applicable, what needs to be done and/or improved to ensure sustainability?

6. Evaluation approach and methods

The evaluation/review consists of several phases:

Contract and Kick-off meeting: Contract is signed and a discussion of the assignment takes place. First documents, including available data, are provided to the evaluation team.

Desk Study: The evaluation/review team studies all necessary project/programme documents; re-construct and analyse the intervention logic/programme theory and theory of change and its assumptions. Existing data needs to be analysed and interpreted.

Inception-Phase: In the inception report the evaluators will describe the design of the evaluation and will elaborate on how data will be obtained and analysed. The use of a data collection planning worksheet or a similar tool is required. First interviews take place.

Data triangulation and quality control are very important and need to be discussed in the inception report.

The field trip will only take place upon official approval of the inception report by the contractor.

Field-phase: Data needs to be gathered, analysed and interpreted. It is expected that the evaluation will include quantitative and qualitative data disaggregated by sex.

Presentation: Presentation of key findings (feedback workshop) at the end of the field trip.

Final Draft Report: Submission and presentation of final draft report, inclusion of comments from partners and contractor.

Final Report: Submission of final report, see reporting requirements under point 9).

For the different phases it is expected that data and information will be obtained through different methods such as: analysis of documents, structured interviews, semi-structured interviews face-to face or by phone, group discussions, online-survey (if applicable), others.

All data collected needs to disaggregated by sex.
It is expected that the evaluation/review team will present concrete recommendations which are addressed to the specific stakeholders.

It is currently estimated that 30 number of people need to be interviewed, representatives of the implementing team – CE ProDidactica, ATIC; donor agencies – ADA, the Embassy of Romania in Moldova (on behalf of the Government of Romania), USAID; beneficiaries – Centre of Excellence in Informatics and Informational Technologies, 3-4 assigned VET schools, ANACIP; partners – the Ministry of Education, Ministry of Information Technology and Communications.

The ADA Evaluation Quality Standards are to be applied and compliance needs to be comprehensible in the evaluation: http://www.entwicklung.at/fileadmin/user_upload/Dokumente/Evaluierung/Evaluierungs_Leitfaeden/EN_Leitfaden_Evaluierung.pdf

The results-assessment form (in Annex) needs to be filled in by the evaluator as part of the reporting requirement.

7. Timetable

Optional: A total of 15 working days is currently estimated for this assignment

1. The Evaluation Team

The evaluation team will consist of 2 members.

Key Qualifications in the team should be:

• Relevant academic degree (master level) in social sciences. economics or other relevant field;
• A minimum of three years’ experience and expertise in the field/sector of education, project management or other related fields
• Team leader has conducted at least three evaluations in the last five years ideally in the relevant field
• Team member has participated in at least three evaluations ideally in the relevant field
• Knowledge of the Republic of Moldova with focus on topics such as education, labor force development;
• Working experience in Eastern European countries
• Experience in project cycle management
• Experience and expertise in evaluating cross-cutting issues
• Experience in social science methods
• Excellent oral and written Romanian and English skills
• Sound MS Office and IT skills

The consultants must not have been involved in the design, implementation or monitoring of this project/programme.

CVs, reference evaluations and evaluation methodology need to be provided.

9. Reports

The consultants will submit the following reports:

• an inception report (10-15 pages without annexes);
• a final draft evaluation report (about 25-30 pages without annexes), including a draft executive summary and the results-assessment form (part of the reporting requirement);
• and the final evaluation report (25-30 pages without annexes), the final executive summary and the results-assessment form (part of the reporting requirement).

All reports need to be written in English. If another language is selected please, add that the executive summary needs to be in the other language and also in English.

The executive summary should summarize key findings and recommendations (three to five pages) and needs to be submitted as part of the final draft report.

The findings and recommendations of the draft final report and final report have to be structured according to the evaluation questions. An outline of the report’s structure needs to be agreed upon during the inception phase.

The quality of the reports will be judged according to the following criteria:

• Is the results-assessment form submitted as part of the report?
• Does the report contains a comprehensive and clear executive summary?
• Were the Terms of Reference fulfilled and is this reflected in the report?
• Is the report structured according to the OECD/DAC criteria?
• Are all evaluation questions answered?
• Are the methods and processes of the evaluation sufficiently documented in the evaluation report?
• Does the report describe and assess the intervention logic (e.g. logframe, program theory) and present/analyze a theory of change and its underlying assumptions?
• Are cross-cutting issues analyzed in the report?
• Are the conclusions and recommendations based on findings and are they clearly stated in the report?
• Does the report clearly differentiate between conclusions, lessons learnt and recommendations?
• Are the recommendations realistic and is it clearly expressed to whom the recommendations are addressed to?
• Were the most significant stakeholders involved consulted?
• Does the report present the information contained in a presentable and clearly arranged form?
• Is the report free from spelling mistakes and unclear linguistic formulations?
• Can the report be distributed in the delivered form?

 10. Co-Ordination/Responsibility

Contact details:

CE Pro Didactica, str. Armeneasca 13, Chisinau, MD 2012,

phone, E-mail: + (373) 22 541994, 542556, 079592525

rbezede@prodidactica, odraguta@prodidactica.md, Ms. Rima Bezede, Ms. Oxana Draguta

ATIC, str. Studentilor 9/11, executive@ict.md, (+373) 78 498 834, Ms. Ana Chiriţa.

The applications (CV(s) of expert(s) proposed for the assignment, reference evaluations, statement on availability of the proposed expert(s) during the planned evaluation period, description of the proposed methodology, information on fee rate in EUR, calculation of number of work days, as well as information on travel costs) have to be delivered to Educational Centre PRO DIDACTICA no later than 11 December, 2017. The applications have to be delivered in English language.

Any application received after the deadline will not be considered.

Contracting Authority:                                  Educational Centre PRO DIDACTICA
Contact person:                                              Ms. Rima Bezede
Address:                                                             Educational Centre PRO DIDACTICA
13, str. Armeneasca, mun. Chişinău, MD-2012

11. Annexes:

1. Results-Assessment Form for Mid-Term Evaluations
2. Project Logframe,
3. Other annexes will be provided as requested or needed