The CACAS project was approved in December 2003, as a successor for the Som/96/002, which was active from 1996 to 2003. The development objective of the project is the restoration and rehabilitation of the civil aviation services in Somalia. This objective should be accomplished by providing the basic facilities and services to ensure safe, effective and reliable air service through the full operationalisation of the CACAS, the development of a nucleus civil aviation administration, and the establishment of the minimum requirements to render functional the airports used by the UN-Agencies. This objective is in support of the Secretary General’s April 1991 decision to provide emergency aid to Somalia, and the March 1995 authorization for ICAO to act in matters of civil aviation with regard to Somalia.
The Mission of the Authority has been defined as:
To provide, under the supervision of the Director of the Technical Cooperation Bureau of ICAO, for the operation and maintenance of essential facilities, equipment and services for international air transport operations, including humanitarian and relief flights within the Mogadishu Flight Information Region (FIR – Airspace and ground facilities in Somalia), as far as feasible on a self-financing basis, in order to meet immediate requirements for safety and to plan, programme and develop an essential nucleus for the establishment of a functional Civil Aviation Administration infrastructure for future government of Somalia.
Purpose of the evaluation:
This evaluation aims at assessing the status of the implementation of the five objectives of the project. The evaluation should also establish if the objectives of the project and the mission of CACAS are adequate and relevant in the current context; special consideration should be given to the authorization that the Secretary General UN assigned to the ICAO.
Duties and Responsibilities
The projects concentrate on five key functions:
* The provision of Air Traffic Services to aircraft operating within and through the airspace of Somalia (FIR) and the collection of revenue through the application of aeronautical charges
* Provision of technical and operational assistance at designated airports and to local administrations in Somalia
* Establishment and operation of a nucleus civil aviation administration for the functioning of CACAS;
* Formulation and implementation of a training programme for national personnel;
* Formulation of procedures and draft regulations required for the operation and maintenance of civil aviation activities.
Issues to be covered by the Evaluation:
The Mission is expected to review progress and impact achieved to date by CACAS against its objectives and mission statement, and specifically to asses following issues:
Project Concept and Design
The evaluation mission is requested to evaluate the institutional arrangements of the project, at present and those deemed appropriate when the project was approved. The evaluation should also determine if the project could benefit from other sources of funds to respond to the development requirements identified by the national officials (local authorities) and how to fit it into the national (Northwest, Northeast, Somalia) sector plans.
The evaluation is also expected to provide advice and proposals, if appropriate, for the next phase of the project and how to ameliorate, if necessary, the present project document. This should include a review of the legal context, as current operations rely on the 1995 authorization.
The evaluation is required to check the extent to which the project’s main activities have been implemented as planned and as envisaged in the project objectives, and to determine, in instances where activities may have not been implemented as envisaged, the reasons and consequences of such deviations.
The mission shall address itself to the systems performance as established by the project in terms of its primary and secondary functions and in light of the conditions and usage applied to air navigation charges.
The evaluation should asses whether the local administrations and the national staff were involved in the implementation and took responsibilities, taking into account Somalia’s geopolitical environment and CACAS’ mandate in terms of safety.
Other issues to be looked at carefully relate to operations and procedures in light of the ‘caretaker’ role of the project, as CACAS carried out these functions in lieu of a government authority; determine if major problems were encountered during the implementation stage. The functions include the establishment of an organizational structure, maintaining facilities, service provision, and establishment of a training programme in support of the services provided and to facilitate integration of Somali civil aviation personnel, formulation of regulations, agreements and operating procedures for the maintenance of the essential elements of an air transport system.
The mission is also expected to review the results of previous evaluations of the project and if and how these have contributed to successful project implementation.
Detailed scope of the evaluation
The evaluation will be carried out in three phases:
* The inception phase, consisting of a) background study prior to arrival in Nairobi (some materials will be provided by ICAO and UNDP); b) development of an evaluation work plan and travel schedule; and c) briefing sessions with key stakeholders.
* The implementation phase, consisting of information and data gathering, meetings with key partners, field visits and other necessary evaluation methodologies, to address all issues described in the evaluation scope.
* The closing phase, in which the team presents the evaluation findings and prepares the draft and final reports.
In conducting the evaluation, the mission team will cover at least the following points:
Project Concept and Design
* Is the project mission and ‘caretaker’ role of the authority sufficiently clarified and explained in the project document?
* Did the project document clearly define the problem to be addressed by the project, taking into account the institutional, social-political, economic and environmental context?
* Did the project document provide a clearly defined project approach and strategy?
* Did the project document provide for implementation and management arrangements, as well as clear linkages among different project objectives, inputs, activities, expected outcomes and impacts?
Relevance of Project
* How relevant is the project to the activities of the international community in Somalia and to the international community as beneficiaries of flight safety over the Somali airspace?
* How relevant is the project to the development priorities of Somalia? (The mission should asses the needs of local authorities)
To what extent has the project responded to its functions in terms of:
* Available services and their potential impact on safety, efficiency and economy of international air transport operations within the Mogadishu FIR
* Existing equipment installed, its functioning and reliability, and the provision of regular maintenance and required repairs;
* Proficiency in air traffic services personnel in all functions associated with the operation of the FIC, AIS Aerocom and Aeromet;
* Any other activities of the project associated with the provision of services in the Mogadishu FIR;
* Procedures for quality control, including training and the investigation of incidents and follow-up action;
* Letters of procedure with other contracting States and their implementation;
* Existing local operating procedures and instructions.
* To what extent has the project facilitated the transfer of expertise from international staff to national staff?
* What is the impact of training programmes and training activities currently undertaken by the project staff?
* Are plans for upgrading the skills of national staff and improving their experience adequate? (In particular the issue of licenses for the Somali air traffic controllers should be analyzed)
* What course of action, if any, is recommended for the early takeover of senior positions by national staff, and what should the timeframe be?
* How well has the project used its resources to produce the target outputs?
* To what extent have Somali expertise and resources been used? Is the project owned by stakeholders and how?
* What is the project status with respect to target outputs in terms of quantity, quality and timeliness? What factors impede and facilitate the production of such outputs?
* How useful are the outputs of the project to the specific needs of Somalia? Is there a general acceptance of the outputs by the beneficiaries?
* How appropriate are the execution and implementation modalities? How is the project managed? How extensive is the involvement in management of Somali staff, of local Somali authorities, of ICAO HQ?
* How adequate is the support provided by the ICAO ESAF Regional office? Should the collaboration be improved or strengthened; and how?
* How adequate is the support provided by UNDP country office? Should the collaboration be improved or strengthened; and how?
* Assess current involvement of stakeholders, particularly local administrations. (the evaluators should also assess the present relationship with local authorities and, if necessary, suggest ways to improve it)
* Define successes and problems, if any, and how to strengthen/address them
* Identify possible gaps in matters of civil aviation that the project does currently not address
* Review the work plan and training schedule
* Assess resource mobilization potential and advise on suitable donor strategies.
The mission will be required to draw any significant lessons learned from the implementation of this project in the past and how these lessons could alter the objectives of the project. The mission should advise any changes required and whether further extension of the project activities is justified.
In particular the evaluation should advise on present and future amelioration strategies to strengthen the elements of capacity building, ‘caretaker’ role, objectives and missions, relations with the Country Office and management.
Composition of the Team
The evaluation team will be composed of three external evaluators. One evaluator (team leader) will be selected by UNDP. S/he will posses in-depth knowledge of UNDP policies and procedures and should have some experience in crisis countries (preferably Somalia). The second evaluator will be selected by ICAO. S/he will have specific technical knowledge of civil aviation and safety requirements as well as project management principles. The third evaluator will be selected by UNDP and s/he should be a Somali independent national consultant. The evaluators shall be ready to travel to Somalia for approximately half the time of their assignment. Knowledge of Somali and Arabic will be an advantage.
The team leader is expected to have the following qualifications:
* Extensive and proven experience in conducting evaluations in a team leader role.
* Experience in transport and aviation sector is a required.
* Substantive experience in post-conflict/crisis countries. Prior experience in East Africa, particularly Somalia.
* Knowledge of UNDP’s results-based management system and evaluation guidelines.
* Strong writing skills and ability to deliver in a timely manner.
Detailed qualifications for the evaluation team members are in Annex 1.
Monitoring and progress controls
* The team will work under the supervision of the UNDP Programme Manager for Recovery and Sustainable Livelihoods. Progress will be measured according to the expected outputs/deliverables. The final report will utilize UNDP evaluation report guidelines:
Principles and quality
* The report is expected to follow UN evaluation standards as specified by the UN Evaluation Group and DAC Evaluation Network.
* Transparency on data sources used, including strengths and weakness which may affect findings
* Information should be cross-checked
* Methodology should be specified and followed as planned
* Assumptions, risks and external contributing factors should be addressed
* Constraints and limitations on information gathering and the assessment phase should be detailed in the report
* Views presented and considered in the findings should be inclusive, involving all stakeholders
Itinerary and outputs
* The evaluation will last four weeks, of which at least two will be spent in Somalia. The final itinerary will be agreed upon with the team itself, it should nonetheless include interviews with all stakeholders in Nairobi, ICAO regional office, IATA, UNHAS, ECHO, UN-Agencies, UNDP, project staff, NGO’s, Kenyan and local authorities in the field.
The outputs will be:
* A brief Inception Report (Max 4 pages with a refined TOR, work plan, key issues to address with the various stakeholders, evaluation methodology, evaluation framework, roles and responsibilities and report structure and format) to be delivered no later than 5 days working days after the start of the assignment. A working group consisting of UNDP and ICAO staff will review and approve the work plan.
* Debriefing meeting with a PowerPoint presentation with relevant stakeholders in Nairobi to address and discuss key results of the review. This will take place no later than 19 working days after the start of the assignment.
* A preliminary draft report, to be handed to UNDP no more than one week after completion of the evaluation mission, which addresses all the issues listed in the scope of work. The report shall be no more than 20 pages including an executive summary but excluding annexes.
* A final in-depth evaluation report, including clear recommendations to guide decision making on future project operations. The report should identify critical issues to be addressed, recommendations, a proposed division of responsibility to address these issues, and a proposed time line. The final report shall be delivered no later than two weeks after the completion of the assignment.
UNDP will provide office space, security briefing and radio equipment for travels to Somalia, and necessary secretarial services to support all logistical arrangements for such 1:
Required Skills and Experience
The team is to be comprised of three evaluators:
* Evaluation Team Leader
* Aviation Expert
* Somali National Evaluator
Evaluation Team Leader
* Extensive and proven experience in conducting evaluations in a team leader role;
* Postgraduate degree in Economics, Development Studies, Political Science or related field;
* Experience in transport and aviation sector is required;
* At least 10-12 years experience in project design, implementation and evaluation;
* Experience in conducting project evaluations and knowledge of M&E principles and results-based management and project management principles;
* At least 5 years experience in post-conflict/crisis countries. Prior experience in East Africa, particularly Somalia, preferred;
* Experience and knowledge of the socio-political context of Somalia would be an asset, in particular in relation to institutional and capacity development issues;
* In-depth knowledge of UNDP policies and procedures;
* Knowledge of UNDP’s results-based management system and evaluation guidelines;
* Good command of written and spoken English is essential
* Strong report writing skills and ability to deliver in a timely manner.
* Knowledge of Somali or Arabic would be an asset;
Aviation and Evaluation Expert
* Advanced degree or equivalent in one of the fields relevant to civil aviation, preferably in air traffic services;
* Specific technical knowledge of civil aviation and safety requirements;
* Management experience of a Civil Aviation Authority or equivalent is desirable
* Experience with ICAO Technical Cooperation requirements and practices; Field experience is desirable
* Proven experience in evaluation and appraisal of civil aviation services;
* Knowledge of results-based management and project management principles;
* Familiarity with UNDP evaluation standards and principles is desirable;
* Strong writing skills and the ability to work in a team
* Experience in a post-conflict context would be an asset;
* Good command of written and spoken English is essential
* Knowledge of Somali or Arabic would be an asset
Somali National Evaluator
* University degree in political science, public administration or relevant field;
* Extensive experience in conducting project evaluations
* Familiarity with Somalia civil aviation and air services
* Familiarity with evaluation standards and principles;
* Good English language skills and fluency in Somali and preferably Arabic
* Excellent knowledge of the Somalia Context