TERMS OF REFERENCE

Consulting services for final external evaluation 

EXTENDED DEADLINE: 12 February

1. BACKGROUND INFORMATION

   1.1. GENERAL BACKGROUND

Education plays a key role in creating conditions for sustainable human development and building a knowledge based society. The success of an individual depends on one’s ability to adapt to change and life-long learning, and the educational system must provide an appropriate environment for the development of these capacities. The key factor to achieve these goals is teachers’ professional development.

The increasing pace of changes in society, job market (growing ICT-sector, more engineers needed and data analysts, digital skills a key competence for 21st century jobs) and in the technological landscape (IoT, AI, big data) puts a pressure on educational systems around the world. The Republic of Moldova, as other countries, faces an increasing ICT skills gap and a low level of digital literacy. These drawbacks caused by the underuse of the ICT advantages, prevent many citizens from participating in the ICT-based society and economy. A large part of the population does not have sufficient knowledge and digital skills throughout their lives which can also reduce their access to public and social services, that will be increasingly available online, as well as will hinder Moldova’s opportunity to participate in the global digital economy. This can be changed only via the education system and by providing tailored training programs for teachers.

Educational innovation is difficult to implement. Modern technologies are underused in the classroom. The biggest risk today is a society that is ill-prepared for the future. If education is to be the backbone of growth and inclusion, a key task is preparing citizens to make the most of the opportunities and meet the challenges of a fast-moving, globalized and interconnected world.

This task cannot be fulfilled by the institutions which do not use the opportunities offered by ICT and that operate in an old-fashioned way – with a limited application of ICT devices and interactive methods in teaching and learning. According to the Republic of Moldova’s Education Strategy 2020, communication in institutions is still dominated by classic paper-based methods, despite developing the management platforms that should be promoted and maintained. Most teachers have limited access to teacher training courses in ICT pedagogy. The existing professional training programs on digital skills for teachers are of poor quality. The majority of courses are outdated, and the final exam is formal. The missing element is the flexible, diverse, staged and adapted model of training to each individual’s needs. Teachers will have to choose the modules that respond best to their needs and subjects that will help them teach by using ICT to communicate with their students, to manage their classrooms and for professional development. The module on educational software will be focused on available software and on how to use it in teaching, learning and evaluation.

Despite some progress at a policy level (digital standards for general education teachers, strategies, investments in infrastructure and curriculum development for IT specialties (VET and general education), there have been minor advancements in the use of technologies for the improvement of management processes, teaching and learning in institutions, including VET institutions. Despite the fairly good access to IT infrastructure in almost all general education institutions (according to MoECR, half of them are outdated, but that will be checked in the VET system), IT technologies are used less in teaching and learning.

1.2. OVERALL PROJECT DESCRIPTION

Austrian Development Agency (ADA) is supporting the Republic of Moldova to overcome these long-lasting problems, through the implementation of the ”Digital skills for employment in modern economy” project (ADA project number 8362-00/2018).

EC PRO DIDACTICA is the implementing partner ADA for ”Digital skills for employment in the modern economy” Project. The Project Management Unit is created by ECPD, led by the Project Director. There is a Project Steering Committee and an Advisory Group within the Project framework.

Implementation period: December 2018 – November 2021

Total budget: 690.000 Euro

The direct beneficiaries are the teachers and the management of 7 VET institutions including the Centre of Excellence in Informatics and Communication Technologies, as well as other 6 VET institutions (around 15 managers and around 100 teachers in total).

Around 3,000 students, boys and girls (all students of 5 VET institutions and of CEITI), and 500 teachers, women and men (all teachers of 6 VET institutions and of CEITI), will indirectly benefit from the project measures in the short term. In the long term all students of the VET system will benefit (as of today there are around 86,000 students).

6 VET institutions were selected on a competitive basis and will be further equipped with IT devices, for establishing a modern classroom, redesigned and equipped with educational IT digital devices, and are receiving intensive training for managers and teachers. The capacities (trainers, IT experts) and resources (training rooms) of CEITI developed in the phase II of the VET TIC project, will be used for the proposed project.

In order to boost digital skills for employment in the modern economy, the project is being implemented throughout a 3-year period and shall result in the following 4 outputs:

Output 1: Digital skills are acknowledged and continuously supported by vocational education and training (VET) institutions at strategic level as part of teaching and organizational culture.

Activities to achieve this expected output includes: Capacity development of VET institutions’ manager of 6 beneficiary institutions through training, coaching visits and consultation in the process of IT infrastructure procurement, IT management systems development and maintenance; design of modern spaces.

Output 2: VET teachers and instructors have better digital skills for an efficient teaching, learning, assessment and class management.

Activities to achieve this expected output includes: Develop, include in the Excellence Center in IT (CEITI) in-service training offer and implement ToT programs, mentoring program, digital competence training program for teachers; organise one study visit.

Output 3: Digital content for VET trades and specialties is developed and open education resources are increasingly used in teaching, learning and assessment.

Activities to achieve this expected output includes: Developing guidelines on implementation of curricula from ICT integration perspective, developing digital content for 3 trades and specialties; implementing training programs on the development of digital content and copyrights and OER promotional campaign;

Output 4: Modern learning spaces created to foster the integration of ICT in the education process as well as the student centred learning.

Activities to achieve this expected output includes: Procurement of modern digital devices in 5 VET institutions and CEITI; organising training on modern teaching methods using spaces and IT equipment; organising 6 online and offline network meetings for teachers.

The project’s intended impact is to substantially increase the number of VET students, both girls and boys, who have relevant skills, including technical and vocational skills, for employment, decent jobs and entrepreneurship (based on SDG 4.4.).

The project’s expected outcome is that the Information and Communication Technologies are effectively integrated into the vocational education and training system (based on Moldova Education Strategy 2020) thus enhancing the access to qualitative and relevant education (based on SDG 4).

2. EVALUATION PURPOSE AND OBJECTIVES

2.1.  Purpose

The main purpose of this final project review and evaluation is to facilitate a process, which will document project outputs and impact. Eventually, the process should also mobilize the various stakeholders to take action based on this documentation.

• The evaluation is intended to contribute to improvement of policies, processes and methods of embedding digital skills in TVET;
• The main users of the evaluation will be: Austrian Development Cooperation, EC. PRO DIDACTICA, Ministry of Education, Culture and Research;
• The evaluation contributes to the accountability towards the stakeholders and taxpayers, including clear statements if the objective and the results have been achieved.

2.2.  Objectives

a)     PERFORMANCE ASSESSMENT

To assess the project’s performance and achievements vis-à-vis the project’s overall objectives and to conduct impact assessment on the various sub-national level beneficiaries. Each of the detailed key questions and issues will be analyzed in a participatory, collaborative and systems-based approach using appropriate key review criteria. This assessment will also include an analysis of the capacity of the management structures of the implementing partner and target communities to implement the project activities as well as the monitoring and evaluation system.

b)     LESSONS LEARNED

To generate lessons learned from the implementation of the project’s activities and the outcomes achieved that will be useful for similar projects in the future for the same sector. Based on the findings and conclusions from the assessment of the project’s achievements, the review will identify lessons learned.

c)      RECOMMENDATIONS

To develop specific recommendations for major stakeholder groups anchored on the conclusions the different stakeholder groups will develop based on their own recommendations and insights. An action plan for major stakeholder groups shall be developed to promote sustainability and long-term impact to the beneficiary communities.

3.  SCOPE

EC PRO DIDACTICA is seeking a company or a team for carrying out the Final External Evaluation of the project.

3.1. Time frame

The selected team will evaluate the following project activity timeframe December 2018 – May 2021

3.2.  Geographical area to be covered

The assessment must cover the following geographical areas of Moldova: Chișinău, Bălți, Cahul, Țaul.

3.3.  Thematic focus

The level of digital skills of TVET teachers and pupils.

4.  EVALUATION QUESTIONS

Effectiveness:

1. To what extent has the project/programme already achieved its expected results/outputs or will be likely to achieve them? How can the quality of results be assessed?
2. What were the major factors influencing the achievement or non-achievement of the outcome(s)/expected results/outputs? (Also consider any which were possibly beyond the control of the project)?
3. Have the project stakeholders worked together as planned? Are there any issues that need to be solved? If yes, which ones?
4. To what extent was gender, environmental, social inclusion mainstreaming included in the project/programme and to what extent were recommendations from the ADA appraisal on gender, environment and social standards considered and implemented? Have any issues emerged, if so which ones and why?
5. How likely will the project achieve its expected outputs and outcome by November 2021?

Sustainability:

6. If the project/programme continues will it be integrated in local structures and/or funded by other sources?
7. What were the major factors which influenced the achievement or non-achievement of sustainability of the project/programme?
8. If applicable, what needs to be done and/or improved to ensure sustainability?

5.  DESIGN AND APPROACH 

1. Contract and Kick-off meeting: The contract is signed and a discussion about the assignment takes place in the EC PRO DIDACTICA office. First documents, including available data, are provided by the project team to the Consultant.
2. Presentation of the Work plan and the Methodology, the Consultants/company will describe the design of the evaluation and will elaborate on how the data will be obtained and analysed. The use of a data collection planning worksheet, questionnaires, Survey monkey tools or any similar tool is required. Bidders are invited to submit a detailed evaluation methodology that is considered appropriate to answer the evaluation questions.
3. Desk Study: The evaluation/review team studies all necessary project/programme documents; re-construct and analyses the intervention logic/programme theory and theory of change and its assumptions. Existing data needs to be analysed and interpreted.
4. Inception-Phase: In the inception report the evaluators will describe the design of the evaluation and will elaborate on how data will be obtained and analysed. The use of a data collection planning worksheet or a similar tool is required. First interviews take place. The report will follow the q uality c hecklist (Annex 5, pag.46). An outline of the report’s structure needs to be agreed upon during the inception phase.

Data triangulation and quality control are very important and need to be discussed in the inception report.

The evaluation will follow ADC and O ECD/DAC norms and standards as well as ethical guidelines for evaluations.

The field trip/online interviews will only take place upon official approval of the inception report by the contractor.

5. Field-phase: Data needs to be gathered, analysed and  interpreted.  The data collection and analysis methods suggested are sufficiently rigorous to allow for a complete, fair and unbiased assessment. It is expected that the evaluation will include quantitative and qualitative data disaggregated by sex.

6. Presentation: Presentation of key findings (feedback workshop) at the end of the field trip/online interviews.

7. Final Draft Report: Submission and presentation of final draft report, inclusion of comments from donor and contractor.

8. Final Report: Submission of final report. The reports must be written in English and Romanian. The consultant can choose the original language of reporting and state this in their proposal. . The structure of all reports, including the structure of the executive summary should follow the structure provided in the ADA.

Guidelines – Annex 6 Quality Checklist for Evaluation Report (pag. 48) and Annex 7 – Template for Evaluation Matrix Austrian Development Cooperation Director approves the reports.

The quality of the reports will be judged according to the following criteria:

• Is the results-assessment form submitted with the report?
• Does the report contain a comprehensive and clear executive summary?
• Were the Terms of Reference fulfilled and is this reflected in the report?
• Is the report structured according to the OECD/DAC criteria?
• Are all evaluation questions answered?
• Are the methods and processes of the evaluation sufficiently documented in the evaluation report?
• Does the report describe and assess the intervention logic (e.g. logframe, program theory) and present/analyze a theory of change and its underlying assumptions?
• Are cross-cutting issues analyzed in the report?
• Are the conclusions and recommendations based on findings and are they clearly stated in the report?
• Does the report clearly differentiate between conclusions, lessons learnt and recommendations?
• Are the recommendations realistic and is it clearly expressed to whom the recommendations are addressed to?
• Were the most significant stakeholders involved consulted?
• Does the report present the information contained in a presentable and clearly arranged form?
• Is the report free from spelling mistakes and unclear linguistic formulations?
• Can the report be distributed in the delivered form?

For the different phases it is expected that data and information will be obtained through different methods such as: analysis of documents, structured interviews, semi-structured interviews face-to face or by phone, group discussions, online-survey (if applicable), others.

The data collection and analysis methods will be sufficiently rigorous to allow for a complete, fair and unbiased assessment. All data collected needs to disaggregated by sex.

It is expected that the evaluation/review team will present concrete recommendations which are addressed to the specific stakeholders.

The results-assessment form needs to be filled in by the evaluator as part of the reporting requirement. The RAF template (Excel format) can be downloaded from here (Annex 9, pag. 53).

6.  WORKPLAN

 The intended start date is February 15, 2021 and the period of implementation of the contract will be until May 28, 2021.  

7.  EVALUATION MANAGEMENT ARRANGEMENTS

All the teams involved will respect the ethical standards and guiding principles for evaluation, including impartiality and independence.

7.1.  PROJECT MANAGEMENT TEAM

The Project Management Team (EC. PRO DIDACTICA) is involved in the project’s implementation. The team will provide information for the review process and will support the Consultant in conducting the review. The main roles and responsibilities are:

• Support the Consultant with the organization of the review process
• Generation and sharing of information
• Facilitate field activities, as needed
• Analyze results and develop recommendations through the assistance of the Consultant
• Implement recommendations as agreed in the review process

7.2.   AUSTRIAN DEVELOPMENT COOPERATION TEAM

• provide the criteria of evaluation according to the Guidelines for Programme and Project Evaluations ADA 2020
• Provide feedback for the inception report
• Approve the final draft of the report
• Help disseminate review results

7.3.   STAKEHOLDER TEAM (MINISTRY OF EDUCATION, CULTURE AND RESEARCH)

The Stakeholder Team represents all stakeholders in addition to the Implementation Team involved in the review. It is the responsibility of the stakeholders to designate representatives of each group that will work with the Consultant. This group will:

• Generate and share information at their stakeholder group level;
• Conduct preliminary analysis of findings, lessons learned and recommendations;
• Provide feedback and help disseminate review results

7.4.  FACILITIES TO BE PROVIDED BY THE CONTRACTING AUTHORITY AND/OR OTHER PARTIES

EC PRO DIDACTICA can provide, as appropriate, room facilities for meetings of stakeholders, discussions and presentation of the report.

8.  REQUIREMENTS FOR EVALUATORS

8.1.  GENERAL

• The evaluation team will consist of 2 members;
• Note that civil servants and other staff of the public administration, of the partner country or of international/regional organisations based in the country, will only be approved to work as experts if they are well justified. The justification should be submitted with the tender and will include information on the added value the expert will bring, as well as proof that the expert is seconded or on personal leave.
• Evaluators must not have been involved in the design or implementation of the programme;
• Gender balanced and diverse team is required.

8.2.  KEY EXPERTS

 8.2.1. Key expert 1: Team Leader – evaluation

 Qualifications and skills

•  Advanced university degree in education, social sciences or related field;
• Minimum of 7 years of relevant professional experience in capacity development needs assessment, policies, analysis, research and evaluation;
• At least 5 (five) years of experience in the designing or implementation, monitoring and evaluation systems of different social policies, labour market analysis or VET institutions;
• At least 5 (five) years of experience in international development projects in related areas;
• Proven experience in quantitative and qualitative research in education, economics or another relevant field;
• Knowledge and experience of VET developments and trends in the EU and OECD countries;
• Experience in implementing projects with national coverage;
• Familiarity with international, European policies and benchmarks in education(vocational education) and best European practices in the field;
• Demonstrated experience of work with the Government of the Republic of Moldova in undertaking research/evaluations/reviews in the baseline studies is an asset;
• Advanced command of English, Romanian and Russian both oral and writen.

8.2.2. Key expert 2: IT & Innovation

 QualifIcations and skills

•  At least 5 years of experience implementing ICT hands-on projects in educational settings;
• At least 5 years of experience of education/VET developments and trends in the EU and OECD;
• At least 3 years of experience in promoting distance learning or e-learning solutions;
• Knowledge and experience of the ICT sector in Moldova or integration of digital skills in education;
• Knowledge of emerging technologies used in education;
• Knowledge of DigiCompEdu framework, Future Classroom concept;
• Ability to combine and assess data, knowledge of statistical and data analysis software;
• Knowledge of English would be an asset;
• University degree in the field of computer science, ICT in education or closely related areas will be an asset;

9.  SPECIFICATIONS FOR THE SUBMISSION FOR THE OFFERS

The company/consultants have to provide detailed CVs, estimated number of working days’ with total budget and a timeline for the assignment.

The weight given to the assessment of the technical and financial offer is specified (as a percentage).

Guidance notes on expert inputs:

Working days: performance of the contract (and therefore payment) will be based solely on working days. The Consultant will only be paid for days actually worked on the basis of the daily fee rate contained in the budget breakdown.

The fee rates for all experts must include all the „administrative costs of employing the relevant experts, such as relocation and repatriation expenses including travel costs upon mobilisation an accommodation, expatriation allowances, leave, medical insurance and other employment benefits given to the experts by the Consultant”.

DEADLINE for submissions: All proposals should be sent to EC PRO DIDACTICA at the address provided below by 31 January 2021 at 18:00.

miliev@prodidactica.md and odraguta@prodidactica.md

ANNEX 1– REFERENCES

1.  ADA Guidelines for Programme and Project Evaluations
2. Evaluation policy
3. DAC – Evaluation Criteria: Adapted Definitions and Principles for Use
4. Baseline study
5. Logframe Matrix

  PROJECT: DIGITAL SKILLS FOR EMPLOYMENT IN THE MODERN ECONOMY