Detailed Information on the
Support for East European Democracy/Freedom Support Act Assessment

Freedom Support Act evaluation is being conducted to increase the effectiveness of border security programs in Eurasia. We are determining the effectiveness of FREEDOM Support Act (FSA) -funded border security programs implemented between FY 2004 to FY 2006. Explain why progress towards the planned results of these programs has been achieved or not; and identify lessons learned and best practices to be incorporated into both current and future border security programs in the region.

Conduct independent evaluation of this office in its role as a coordinator of assistance or the impact it has on the effectiveness of these programs and the achievement of the FSA and SEED Act's purposes.

New guidance created to address assistance coordination, performance & resource allocation oversight. They are Interagency Country Assistance Review to set country assistance strategies & Phase-Out Guidance to have sector-level phase-out plans.

In FY 2007 EUR/ACE participated fully in the new Office of the Director of U.S. Foreign Assistance review and approval process of FY 2007 Operation Plan submissions by USAID Missions. Interagency Country Assistance Reviews completed country assessment strategies in Albania, Belarus, and Russia.

Measure: Assistance Coordinator for Europe and Eurasia (ACE) administrative costs as a percent of all assistance coordinated by ACE.

Explanation:The targets and baselines are by % of SEED/FSA funds directly managed and as a % of all assistance managed. SEED/FSA=S/F

Measure: Improve scores on the monitoring Country Progress Index for Economic Reform. Scores range from 1-5 with 5 being optimal.

Explanation:Improve scores on the economic reform index. Indicates the transition to a market economy is sustainable. Scores range from 1-5 with 5 representing standards of advanced industrial market economies). Targets for SEED=S, Targets for FSA=F

Measure: Improve scores on the monitoring Country Progress Index for Democratic Reform. Scores range from 1-5 with 5 being optimal.

Explanation:Improve scores on the democratic reform index. Indicates the transition to a democratic society is sustainable. Scores range from 1-5 with 5 representing standards of advanced democratic institutions). Targets for SEED=S, Targets for FSA=F

Measure: Number of countries that phased out of democracy assistance in established timeframes.

Explanation:Democracy assistance phased out as transitions become irreversible in each country.

Measure: Number countries that phased out of economic assistance in established timeframes.

Explanation:Economic assistance phased out as transitions becomes reversible in each country.

Measure: % annual reports and MPPs using performance data consistent with FSA and SEED goals and standards (fully consistent, mostly consistent, and needs work)

Explanation:Ensures that performance data reported in Annual Reports and MPPs is sufficient quality and relevance to make phase out decisions. The goal is to have all MPPs and annual reports move from need work [NW] to mostly consistent [MC].

Measure: % country portfolios with expanded pipeline greater than 24 months as of September 30 (and 30 months as of March 31) not justified by events or implementation requirements

Explanation:Ensure efficient use of assistance. Pipeline review of May 17 2004 will establish baseline and targets will be developed.

Is the program purpose clear?

Explanation: ACE holds two sets of partners accountable for cost, schedule & performance results; specifically 1)Posts and their program partners are held accountable for phasing out strategic assistance areas on schedule as they have been held accountable for phasing out the programs in the eight countries where we provided assistance that have now joined the European Union. They are also held accountable for improving their own use of performance information to improve the efficiency and effectiveness of US assistance and relevance to Department goals. Annual reports and MPPs will be used to monitor accountability. Question 2.5 provided an example of the GRASP program. 2)Agency partners are held accountable through the annual meetings as in bullet three of question 3.1 with the performance data they submit. Each agency pipeline is reviewed twice a year. Agencies are required to submit financial data to get new funding.

Evidence: SEED and FSA Acts; Charter for Coordinator of U.S. Assistance to Europe and Eurasia; ACE Mission Statement

Does the program address a specific and existing problem, interest or need?

Explanation: The Office of the Coordinator ensures targeted, relevant and efficient assistance to SEED and FSA countries. With 40 agencies playing a role in assistance to these countries, it is necessary to have one point of coordination: that is the Assistance Coordinator. Country strategies balance strategic interests; while individual agencies are focused on agency specific missions. ACE has initiated and is chairing interagency Country Phase-Out Assessment meetings to establish timeframes for phase out of strategic assistance areas in each country where we still provide assistance. It has been clear in those meetings that the Coordinator's office supports more ambitious timeframes than the implementers. ACE reviews budgets quarterly across all agencies (program performance & pipeline). ACE is the central point of donor coordination e.g. counternarcotics work with the EC: we coordinate INL, NP & DEA. Overall, the SEED & FSA administrative structure and regional accounts were intended to be temporary and to facilitate the relatively short-term transition of these countries. Over ten years later, they are still in existence as the transitional period in these countries has taken much longer than predicted.

Evidence: List of Agencies coordinated; Country Phase Out Assessment Guidelines; Sample country budget report; FSA and SEED Acts; ACE Charter

Is the program designed so that it is not redundant or duplicative of any other Federal, state, local or private effort?

Explanation: ACE plays a unique role of coordination required by statute and is expanding its ability to deal with issues of efficiency and effectiveness. ACE is the interface between the policy and political environment of the State Department and the foreign assistance role of USAID, USDA, and other agencies and parts of the DOS that implement programs in the regions as well as other global donors. An example is ACE coordination of energy reform in the Caucasus collaborating with the World Bank and the European Commission: ACE coordinated the experience of the Department of Energy, USAID and the State policy people. No one else makes the policy and budget tradeoffs. The Country Phase-Out Assessment process is also indicative of the role not taken by anyone else: systematic objective assessment to inform the recommendations made by an interagency group of policy makers and implementers.

Evidence: Country Phase Out Assessment Guidelines; ACE Charter

Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

Explanation: The office has a clear overarching goal to assure that democracy and market economies are irreversible. There is an established process of working with posts to develop country strategies There is also a process to assess the timeframe for phase out and the priorities to allow that to happen. Budget performance and management of pipelines are incroporated into the budget allocation process. Country goals and indicators for assistance have been extended to MPPs this year. SEED/FSA Annual Reports provide the mechanism to report on performance and are timed to modify budget plans. The Coordinator's Office has excellent engagement with implementers, policy staff, the Hill and stakeholders.

Evidence: (ACE Mission Statement) (Country Assistance Review Guidelines) (Country Phase Out Guidelines) (Minutes of JPC Meeting ' March 24, 2003 & Pipeline guidances Oct 2003 and May 30,2004) (Guidance cable on Annual reports and ACE addendum to guidance cable on MPPs. US share of development assistance.

Is the program effectively targeted, so that resources will reach intended beneficiaries and/or otherwise address the program's purpose directly?

Explanation: The Coordinator's Office ensures that resources reach intended beneficiaries and address the purposes of the SEED and FSA Acts. ACE directs resources to those posts and agencies with a comparative advantage in implementing the priority programs. The Coordinator's office: Works with posts to develop country strategies; Allocates country budgets based on strategic importance, need, ability to have impact, effective use of funds; Solicits posts' recommendations for tradeoffs within their country between strategic priorities and appropriate implementers to achieve goals and reach end beneficiaries. Has introduced performance data into MPPs and SEED/FSA Annual Reports which institutionalizes the assessment of impact on goals and beneficiaries. Finalizes budget allocations only after confirming pipeline analysis showing the agency's utilization of funds.

Evidence: Guidance on strategy reviews; Agency Budget Review memo dated Sept. 16, 2003; Country Phase out Guidance; MPPs and Annual Reports

Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

Explanation: The purpose of SEED and FSA assistance is to ensure that the transition to democracy and market economies is irreversible in each of the countries. The Coordinator's office has established performance measures (economic and democracy reforms matrix and the economic performance and needs matrix) that are based on independent and highly credible sources. Goal lines have been established that help us determine the phase out timeframes. The process to implement these assessments is that, when the timeframe is five or fewer years, the post will develop a phase out strategy. Assessments will be reported in the BPP process. Indicator: Number of countries that phase out of democracy assistance in established timeframes. Number of countries that phase out of economic assistance in established timeframes.

Evidence: Country Phase Out Guidelines; USAID, Monitoring Country Progress 2004, Guidance cable on the annual report

Does the program have ambitious targets and timeframes for its long-term measures?

Explanation: Proposed phase out timeframes are very ambitious because these are changes which the USG can influence but cannot directly cause to be achieved. The Phase Out process has led to ambitious, but realistic phase out timeframes (see Chart for economic and democratic phase out timeframes). This year, the timeframes are being established. In future years, the target would be to phase out in the timeframes established. These were developed from "goal lines" that represent the average of the progress made by Bulgaria and Romania when they were invited into NATO. The goal lines represent systematic country performance information from highly credible international sources to estimate when the economic and democratic reforms in SEED and FSA countries will most likely be irreversible. Assumptions and progress will be reviewed annually to determine wehther targets remain accurate. The causal links between USG assistance programs and the ability to impact the indicators used to measure progress towards the long-term goals are tenuous.

Evidence: Draft Country Phase Out Timeframe Chart

Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

Explanation: Yes the Coordinator's Office has four types of indicators: Effectiveness: Countries monitored annually for achievement of their MPP targets and ACE takes actions as necessary. Quality Management: Ensure that performance data reported in Annual Reports and MPPs is sufficient quality and relevance to make phase out decisions. Progress toward graduation: Whether and when to phase out economic and democratic assistance Review Phase out timeframes annually as part of the Annual report and MPP review processes to reconfirm that the established phase out timeframe is still accurate. Efficiency of the allocation of assistance: Percentage of country programs with pipeline red flags (as defined in question 2.4)

Evidence: Guidance to ACE staff for review of MPPs and Annual reports

Does the program have baselines and ambitious targets for its annual measures?

Explanation: 100% annual reports and MPPs reviewed by ACE for performance and consistency with policies and priorities, 100% annual reports & MPPs using performance data consistent with FSA & SEED goals & standards (fully consistent, mostly consistent, consistent, and needs work), 100% review of established phase out timeframes, 10 % country pipelines with red flags (expanded pipelines greater than 30 months) not adequately justified

Evidence: Pipeline review on September 30, 2003 data; Program to Strengthen Performance Measurement in Europe and Eurasia

Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

Explanation: All partners working on economic and democratic reforms are working toward the achievement of the reforms represented by the goal lines and therefore the phase out of assistance on the established timeframe. Although security and law enforcement do not have explicit 'goal lines', resources are directed toward preparing those countries to be partners with the USG in international security and law enforcement efforts and timeframes are established on that basis. Posts are held accountable through review of the Annual Reports and MPPs. For example, funds for the GRASP program in Romania were put on hold for nonperformance and reprogrammed to INL. Agencies are held accountable in the annual budget review meetings ACE holds with each agency to discuss proposed funding. ACE screens proposals for that agency's contribution to ensuring democratic and economic transitions. Agencies must present performance information to justify their budget.

Evidence: SEED and FSA Annual Reports; Agency annual budget review memo "EUR/ACE Review of FY 2004 Agency Program Proposals"

Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

Explanation: Each agency is required to evaluate the effectiveness of their programs. For example in FY03, Management Systems International conducted an eight country Rule of Law Evaluation. Lessons from that evaluation are being fed back into democracy and, and to some extent, law enforcement programs in the region. In an example of a program managed in ACE the Stability Pact has initiated an external evaluation of the Investment Compact. For FY 04, the Coordinator's office has identified three areas where insufficient evaluation work has occurred previously and planned evaluations: 1) Land Grant University staff have been contracted by USDA assistance in Armenia (team in the field) 2) an evaluation of customs programs implemented by Dept. Homeland Security (scope of work being drafted) and if funds allow, 3)a region-wide comparative assessment of Democracy Commission grants. For FY 05, the Coordinator's Office will identify those areas with multiple agencies addressing the same objectives and plan a multi-country comparative program evaluation.

Evidence: USAID Evaluation and Analytic Agenda; MAP scope; AT Kearney prelinary report for the Investment Compact; TTF scope of work on customs and the scope for the Democracy Commission grants are currently being drafted.

Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

Explanation: The country phase out data links performance quite explicitly to the timeframe of continued assistance. In the BPP, we will present results of the inter-agency assessments for phase out and link the results directly to the SEED and FSA resource requests. For example, one priority area that has come up in many country phase out assessment meetings is that of creating jobs through small and medium enterprise programs. The FY 2006 budget request identifies jobs as one of four priorities for budget increases. When determining country levels, performance is a factor. Pipeline Performance is factored into resource decisions: e.g. ACE held up Macedonia & Albania programs until pipeline anomalies were corrected. By 2006, a database to link performance to budget will be built. However, there are no documents that hold implementing agencies accountable to the Coordinator for results to be achieved with each tranche of funds, or that specify the consequences of failing to achieve results.

Evidence: draft EUR Investment Prospectus and BPP tables; EUR/ACE Investment Strategy, Agency annual budget review memo "EUR/ACE Review of FY 2004 Agency Program Proposals"

Has the program taken meaningful steps to correct its strategic planning deficiencies?

Explanation: ACE has identified annual goals and measures for the Coordinator's office as a management unit as well as strengthening the performance measures for the long-term goals. The new Coordinator has demonstrated strong interest in and attention to using long term and annual measures in management. ACE has hired a specialist in strategic planning and performance monitoring to improve these in ACE and to work with the posts to strengthen theirs. To strengthen that link, this person has been given the budget responsibility as well.

Evidence: Coordinator's testimony before SFRC March 2, 2004 (not presented) ; ACE Mission Statement; Position Description for director of Program Budget Unit

Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

Explanation: Generally in our relationships with our partners as well as these three key ways: 1) The Phase out Process is based on systematic, objective data from highly credible sources. In order to be transparent, this data is used in interagency meetings to make initial determinations of phase out timeframes. In addition, the Coordinator will chair interagency meetings in which he will communicate the timeframes for phase out as well as identify certain program priorities necessary to achieve phase out. 2) ACE required posts to submit program impact and effectiveness information in Annual reports and MPPs and is working with posts to be sure this information is timely and based on credible sources. 3) The Coordinator's Office also meets with partner agencies to discuss their program proposals for the coming year. ACE requests "performance measurement plans and targets."

Evidence: Annual report and MPP guidance; "Memo Re: "EUR/ACE Review of FY2004 Agency Program Proposals" dated September 16, 2003

Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

Explanation: ACE holds two sets of partners accountable for cost, schedule & performance results; specifically 1)Posts and their program partners are held accountable for phasing out strategic assistance areas on schedule as they have been held accountable for phasing out the programs in the eight countries where we provided assistance that have now joined the European Union. They are also held accountable for improving their own use of performance information to improve the efficiency and effectiveness of US assistance and relevance to Department goals. Annual reports and MPPs will be used to monitor accountability. Question 2.5 provided an example of the GRASP program. 2)Agency partners are held accountable through the annual meetings as in bullet three of question 3.1 with the performance data they submit. Each agency pipeline is reviewed twice a year. Agencies are required to submit financial data to get new funding.

Evidence: Phase Out ' Time Frame Chart;MPPs and Annual Report guidance; "Memo Re: "EUR/ACE Review of FY2004 Agency Program Proposals" dated September 16, 2003

Are funds (Federal and partners') obligated in a timely manner and spent for the intended purpose?

Explanation: Most SEED and FSA funds are obligated in a timely manner. Eighty-five percent of this two year money is moved out to agencies within the first year. This year, the Coordinator's office has included "budget not yet obligated" in the pipeline analysis to emphasize the importance of timely obligation and to be able to shift funds to other purposes or countries if blocks to obligation continue. The Coordinator's office continues to monitor pipelines twice yearly for both the SEED and FSA accounts. In FY 04 for example, ACE held up the Macedonia and Albania programs until pipeline anomalies were corrected. SEED and FSA assistance are spent for the intended purposes. The Annual Reports and the MPPs are the source of information on whether funds are spent for the intended purpose supported by the Inspector General offices of the various agencies.

Evidence: NOA Tracker Pipeline analysis; Annual Reports and MPPs

Does the program have procedures (e.g. competitive sourcing/cost comparisons, IT improvements, appropriate incentives) to measure and achieve efficiencies and cost effectiveness in program execution?

Explanation: In terms of managing partners towards efficiencies and cost effectiveness, the Coordinator's office relies on: agencies' pipeline analysis which reflects not only their efficiency in moving money but can also reflect the efficiency and effectiveness of using the money; and the annual meetings with agencies where their performance is discussed in the context of their request for additional resources factoring in the data provided by posts on performance as a check. ACE's own efficiency is being measured through the indicator: ACE admin costs as % of all assistance coordinated. (NB funding cuts for exchanges in SEED & FSA levels for FY 2004 with the same staff caused the percentage to increase in 2004.) Staff are still required to coordinate budget levels & impementation with ECA.

Evidence: Pipeline analysis; "Memo Re: "EUR/ACE Review of FY2004 Agency Program Proposals" dated September 16, 2003, Democracy Evaluation Matrix

Does the program collaborate and coordinate effectively with related programs?

Explanation: The primary mission of the office is coordination. Examples of coordination are given above such as the interagency process of Phase Out Assessments, coordination of budget and strategy reviews with partner agencies, participation in the Joint Policy Council, etc. This year, the democracy unit in ACE initiated monthly meetings on programs and policies with DRL and USAID that are being held up as a model of coordination. The ACE mission statement explicitly includes : "Solicit the expertise of U.S. field missions and implementing agencies as a foundation for strategies and budgets" showing the value placed on working collaboratively with the field. ACE routinely coordinates with EU and the OECD. Norway and Estonia recently expressed interest in working together in Georgia asking our thoughts on how best to engage there.

Evidence: ACE Mission statement; Summary of 4-28-04 Democracy Coordination Meeting

Does the program use strong financial management practices?

Explanation: ACE uses strong financial management practices. For example, there were two ways in which financial management practices have been further improved this year: (1) The standards and processes for the budget function used in FSA are being applied to SEED as well as all budgeting is being done in the Program and Budget Unit; (2) We have agreed with USAID on standardized quarterly reporting with twice yearly pipeline reporting. To strengthen financial management practices, ACE does one lump sum apportionment, an OMB suggestion, with adjustments as necessary. The ACE current year budget analyst uses a tracking sheet to focus on keeping the funds moving.

Evidence: NOA FY 04/05 SEED Funds

Has the program taken meaningful steps to address its management deficiencies?

Explanation: The coordinator is building on the reorganization that combined SEED and FSA and has added a Program and Budget Unit that will manage an integrated budget and explicitly link it with performance measurement. The new Program and Budget Unit will be able to build on the best practices of the SEED and FSA offices. Statutorily, the SEED and FSA budgets must be presented separately as they are separate accounts, but the Coordinator is pursuing the application of standards and processes for both SEED and FSA as indicated in question 3.6. ACE has addressed multiple PART recommendations from last year.

Evidence: Functional Statement for the Program and Budget Unit in ACE; Tracker

Are grants awarded based on a clear competitive process that includes a qualified assessment of merit?

Explanation: The Coordinator's office directly manages a limited number of grants and contracts itself. In FY 2003, of the $14.3 million the Coordinator's Office directly managed, 41% was competitive: Humanitarian assistance and the PVO Grants ($8.4 million) are not competitive; Transportation ($5.86 million) is competitive. The transport grants have very clear procedures for competition using a tender process managed by a transportation agent. For the PVO grants, EUR/ACE uses "notwithstanding authority" to meet emergency needs on a real-time basis, providing critically needed emergency commodities to displaced or suffering population. The remaining 98% of SEED & FSA funds are subject to Federal Procurement Regulations as applied by each agency. Their competitiveness processes are reviewed by the agencies' IG offices and not by the Coordintor's office.

Evidence: Transportation tender process; Democracy Commission grants approval memo.

Does the program have oversight practices that provide sufficient knowledge of grantee activities?

Explanation: ACE provides oversight in coordination with the field posts. The Humanitarian grants are overseen by the Humanitarian unit in ACE in close phone and email communication with posts on a real time basis. Commodities are tracked to their final location to be sure that help arrives as fast as it can. PVOs are required to submit reports on their assistance. All grantees are required to submit financial and program reports upon the conclusion of their projects. The Coordinator's office provides little direct oversight of its implementers' grantees. Instead, once allocations are made, implementors (such as USAID) manage their own programs directly. All agencies receiving FSA and SEED funds are required to submit to periodic program and financial audits through their Inspector General.

Evidence: Sample Democracy Commission grant agreement

Does the program collect grantee performance data on an annual basis and make it available to the public in a transparent and meaningful manner?

Explanation: The SEED and FSA Annual Reports are publicly available but data is not disaggregated at the grantee level. While USAID/E&E receives quarterly performance reports from grantees and contractors, and aggregates the data reported in the Congressional Budget Justification, this performance information does not provide evidence for all FSA and SEED competitive grants and is not compiled by the Coordinator, the program manager for SEED and FSA.

Evidence: SEED and FSA Annual Reports

Has the program demonstrated adequate progress in achieving its long-term performance goals?

Explanation: On May 1, 2004, Eastern Europe celebrated the inclusion of ten new countries in the EU ' eight of them had received assistance from the U.S. If you exclude the countries of the former Yugoslavia, this means that eight of the twelve SEED countries have already phased out of both economic and democratic assistance(see the Country Phase Out Timeframe Chart). Two more countries are scheduled for phase out in 2006: Bulgaria and Croatia; with Russia scheduled for economics to phase out in 2006. The proposed schedule for the remaining SEED and FSA countries is being developed. It should be noted that the data used to establish the phase out timeframes come from the monitoring Country Progress indices for democracy and economics that were reported in last year's PART. ACE is still using and monitoring this data but now using it in a way more tied to the management of our programs.

Evidence: Draft Country Phase Out Timeframe Chart

Does the program (including program partners) achieve its annual performance goals?

Explanation: Review of Phase out timeframes: the baseline is established (May 2) and programs will be reviewed in FY05 for consistency with phase out timeframes. Effectiveness ' 100% of annual reports and MPPs were reviewed by ACE staff this past year. 100% of annual reports used performance data consistent with FSA and SEED goals and standards (the lowest acceptable rating with targets of increased consistency over time) before the report was approved for printing. 100% of MPPs had objectives and performance data at least consistent with FSA and SEED goals and standards (the lowest acceptable rating). Percentage of country programs with pipelines greater than 30 months with inadequate justification' data pending.

Evidence: Baseline for phase out timeframes will be proposed to Armitage June 10; Pipeline report from Sept 03 and March 31, 04 (pending)

Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

Explanation: The efficiency indicator for the Coordinator's office is: ACE administrative costs as a percent of all assistance coordinated by ACE. The FY 2004 value is 0.45%. In the development of the FY 06 budget request, the Coordinator requested deputies to identify savings as part of the process for developing their requests. Savings included: the decreased need for humanitarian assistance in, for example, Azerbaijan, Belarus & Ukraine; in response to progress in economic reform, there is reduced need for economic assistance in Russia and Ukraine, for example. For this year, the major efficiency role played by the coordinator's office is to assure that funds are allocated to agencies that can spend them in a timely manner. This is why the pipeline analysis is such a key tool in program management.

Evidence: Administrative costs analysis; pipeline analysis; EUR investment proposal (shows the savings in humanitarian and economic as well as other assistance.)

Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

Explanation: There are no other comparable examples of this kind of coordination mechanism for a broad geographic region to compare to ACE.

Evidence:  

Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?

Explanation: Several independent evaluations focused on the impact of cross-cutting and specific programs directly at the FSA & SEED Act purposes have been completed or are in the process of being contracted. However, there has been no direct evaluation of ACE in its role as a coordinator or the impact this office has on the effectiveness of these programs and the achievment of the FSA & SEED Act's purposes. EUR/ACE is planning an independent evaluation to begin this year and is drafting a scope of work.

Evidence: SEED and FSA Annual Reports