The Right Way to Improve Efficiency at the State Department

Foreign policy must commit to developing strategic-level monitoring and evaluation capacity.

This article is the third in a series of articles exploring how Monitoring and Evaluation might be applied to high-level policy and strategy questions at the Department of State. [Part 1: Can We Monitor and Evaluate US Foreign Policy?, and Part 2: The Benefits and Challenges of M&E for Strategy].

Efficiency is defined as “achieving maximum productivity with minimum wasted effort or expense.” Thus, if an organization’s understanding of its own productivity is ambiguous, then efficiency is impossible to measure. This is the situation in which US foreign policy finds itself.

An essential task for advocates of efficiency and effectiveness in foreign policy is to build a robust system to measure and evaluate success.

Foreign policy can learn a lot from other fields. Key performance indicators (KPIs) is a common approach in the business world. All companies evaluate themselves (and are evaluated by others) on topline metrics like profits and stock price, but also more subtle measures such as capacity utilization rate, lead conversion rate, customer satisfaction, or R&D performance. Monitoring and Evaluation (M&E) is the public sector’s equivalent approach to KPIs. The M&E toolkit is widely used for foreign assistance expenditures, but virtually never for high-level policy and strategy questions. Certainly, the public sector is different from the private sector, and foreign assistance is only one component of foreign policy, but the State Department (and foreign policy in general) has a lot to learn about measuring productivity and efficiency.

With USAID being swallowed into the State Department, foreign assistance will be managed by policy-focused regional bureaus for the first time. If this is to work, policymakers will need to master new skills of program design and management, especially measuring effectiveness and efficiency. This is no easy task, but it provides an opportunity to unify the work of foreign policy and foreign assistance and improve the quality of both fields.

That the effectiveness and efficiency of US foreign policy should be measured is obvious to some. But the dominant paradigm within the State Department rejects this idea. Prominent foreign policy leaders suggest there is no use for evaluation of foreign policy strategies — that they should be exempt, judging the field as too complex and unpredictable for such scrutiny. Others claim that “evaluation” is already baked into the day-to-day work of foreign policy—that it is inseparable from decision-making. The result is that the State Department and the broader foreign policy establishment does not invest in systematically evaluating their work.

Avoiding discussions of evaluation (and thus efficiency and effectiveness) makes sense if one subscribes to the common belief that “foreign policy is an art, not a science.” But I believe this sentiment has undermined the quality of US foreign policy and sown distrust of foreign policy decision-makers among the American public.

It’s time for foreign policy to mature as a field of practice and overcome its resistance to evaluation. M&E for strategy offers a path forward.

Beyond Compliance

Congress has pushed the State Department to do more to evaluate its strategy processes, even before the merger of State and USAID.

The Government Performance and Results Act (GPRA) laid the foundation for performance-based accountability across federal agencies when it passed in 1993. The law requires agencies to establish performance goals, measure outcomes, and report on their progress. In 2010, the GPRA Modernization Act strengthened existing law by introducing requirements for cross-agency priority goals and emphasizing the use of performance information in decision-making. “Performance evaluation” is a technical term referring to whether a program did the things it said it was going to do, not whether those actions caused any particular external impact.

The Foundations for Evidence-Based Policymaking Act, passed in 2018, expanded the framework for federal M&E by mandating that agencies take a systematic approach to building and using evidence. It directed agencies to integrate evidence-building activities, such as data analysis, program evaluations, and performance assessments, into their policy and management decisions. The legislation induced the State Department to author its first Learning Agenda in 2022. While the questions posed in the Learning Agenda were excellent – “What is the impact of high-level diplomatic engagement?” – the Agenda languished without resources or expertise to advance its work.

The State Department’s Managing for Results framework (18 FAM 300) serves as the Department’s tailored approach to planning and learning. Guidelines for program design, monitoring, and evaluation are spelled out, but these are rarely practiced for strategic or policymaking settings (as opposed to foreign assistance, which tends to be more rigorously evaluated).

The State Department hosts a hierarchical strategy process to set the goals of U.S. foreign policy. These strategy processes provide a natural starting point for implementing high-level M&E. Following the White House-led National Security Strategy, the State Department articulates quadrennial high-level priorities in the Joint Strategic Plan. The JSP helps the State Department develop subordinate Joint Regional Strategies for geographic bureaus and Functional Bureau Strategies for cross-cutting bureaus. Embassies and missions overseas develop Integrated Country Strategies to support the relevant regional and functional strategies. Ad hoc strategies are also sometimes developed, such as by the Bureau of Policy and Planning.

But it is bewildering to discover how little investment is made to formally evaluate the efficacy of these hundreds of strategies. The JSP, Regional, and Functional strategies each receive some basic performance management feedback, but this is not the same thing as M&E. Progress towards the JSP goals, for instance, is measured in the State and USAID Annual Performance Report, but the goals are abstract and not falsifiable. Performance management tends to be useful for tracking whether money was spent and tasks were completed, but unhelpful for understanding impact. It amounts to a compliance exercise.

Here's an example. An objective articulated in the 2022-2026 JSP is Promoting a Stable Cyberspace: “By September 30, 2026, sustain and enhance international cooperation to promote the U.S. vision of an open, interoperable, reliable, and secure Internet and a stable cyberspace.” The goal, typical of the dozens of other objectives in the JSP, is non-falsifiable; phrased in such a way that makes it impossible to know what failure might look like. Virtually any activity can qualify as “sustaining and enhancing.” Indeed, the Department dutifully reports having “met” this objective in 2023 by conducting 175 new or sustained diplomatic engagements on cyber issues, an increase of 171% over the previous year. This figure is almost meaningless; it says nothing about whether Cyberspace is more open or secure.

While the Department boasts that it met or exceeded most of its JSP goals (67 out of 99!), its performance measures say little. It is unclear whether any of the work achieved US interests, or what was learned along the way.

Some suggest – incorrectly – that the intelligence community (IC) might be well-positioned to evaluate the quality of US foreign policy. This misunderstands the IC’s role. A “firewall” has been constructed between policymakers and analysts to prevent the politicization of intelligence analysis. The result is that the IC is excellent at assessing international challenges but unable to analyze the quality of US policies.

Four Essential Principles

As I said previously, M&E is easiest in highly controlled, static environments, but the world of international affairs rarely provides laboratory-like conditions: policy challenges are often “wicked,” featuring a high degree of complexity and uncertainty; strategies often deploy many concurrent activities intended to be mutually reinforcing, and; precise identification of cause and effect is impossible. Some skeptics therefore argue that M&E is a waste of time. This conclusion is wrong.

The goal of M&E is not to seek perfect knowledge in an infinitely complex world. Instead, the goal of M&E is to help policymakers. It achieves this by improving our collective understanding of how policy choices affect the world, mitigating uncertainties, and increasing the likelihood of policy success. Feedback is, after all, the essential ingredient in the development of expertise.

M&E is not effective when it simply suctions up a ton of useless information like a vacuum cleaner. This is a giant waste of time and money. The right approach focuses on collecting knowledge that will improve the policy. A few principles that M&E practitioners working in strategic environments must consider:

1. The challenges we face are highly complex. Most challenges in foreign policy will feature a dizzying array of factors that can affect the outcome. We must remember that policies are dynamic and interrelated. The world is not linear, nor are our policy responses. We must avoid conceptualizing problems or evaluating solutions in overly simplistic or stove-piped ways. M&E practitioners must seek to understand the multivariate nature of the world and apprehend the connections between issues.

2. At the same time, one must not become overwhelmed by complexity. Developing an M&E plan requires parsimony, a prioritization of the issues that matter most. Policy environments are like maps: one can draw a map that shows all the roads, the topography, or the sewer system, but not all three at the same time. One must make choices about what to include and exclude. Policymakers can practice parsimony by prioritizing the most important policy actions in their strategy that require M&E.

3. Systems inherently involve uncertainty. Policymakers are always operating in a state of partial blindness. But uncertainty is not ignorance. Policymaking is like playing a game of cards: one wins by identifying the strategy most likely to succeed, recognizing that failure is a possibility even with the best strategy. The most successful policymakers are experts in uncertainty, taking calculated risks with incomplete information to maximize long-term success. M&E plans should help policymakers mitigate the most meaningful sources of uncertainty throughout the policy cycle.

4. Evidence and feedback from M&E processes are most helpful when directly policy-relevant. Gathering endless data is counterproductive and wasteful. Feedback should seek to deepen our knowledge of both the causes of the problems we face, and the efficacy of the tools we use to address them. When developing an M&E plan, one should always ask: How would confirming or disconfirming evidence change our strategy or behavior? How exactly will this evidence be used to support decision-makers? This helps ensure M&E efforts are useful.

M&E systems that take these four considerations into account – complexity, parsimony, uncertainty, and relevance – will likely be most successful in improving the efficacy of the policy process.

Getting the Details Right: An M&E Primer

M&E is a technical skill that benefits from training and practice. There is a healthy and growing literature from which M&E practitioners and reformers should benefit. At the same time, the particularities should be tailored to the needs of policymakers for each strategic setting. That said, most M&E work follows a similar framework:

The first step – develop the policy or strategy goals – is the foundation upon which the M&E plan will rest. I have written about this elsewhere and will not go into extensive detail here except to say that the purpose of this step is to eliminate ambiguity around what the policy or strategy is attempting to accomplish. Goals should be SMART – specific, measurable, achievable, relevant, and time-bound. Tools like the theory of change, logic maps, and other strategy frameworks help map out all the small steps necessary to achieve one’s goals.

Developing the monitoring plan is the second step. There are few better ways to sharpen a strategy than by interrogating exactly how success will be measured. In complex settings, it can be helpful to begin by identifying the largest uncertainties in the policy environment, and what kind of information would be helpful to mitigate those uncertainties. The information gathered should be useful for improving the policy, not merely scorekeeping purposes.

Selecting the right metrics to track is tricky, but remember that perfection is not the goal. Develop metrics that are both confirming (evidence that would improve confidence that the policy is working) and disconfirming (evidence that would suggest the policy is failing). Track both outputs (the direct results of one’s activities, e.g., how many schools were built) and outcomes (the longer-term impacts, e.g., are the graduates getting good jobs). Consider metrics for performance, context, and resources. Performance monitoring informs you whether implementation is on track (whether your team is doing the tasks they committed to doing), context helps understand external factors that will affect the policy, and resources center on the necessary people and money to get the job done. Quantitative metrics tend to be more objective, though mixing in qualitative metrics can be helpful as well.

The State Department already has standards for data quality, which are useful:

(1) Validity: the data accurately represent the intended measure;

(2) Integrity: safeguards are in place that minimize risk of data manipulation or error to ensure data are accurate and consistent throughout their lifecycle;

(3) Precision: data have a sufficient level of detail for decision making;

(4) Reliability: data collection processes and data analyses are consistent over time; and

(5) Timeliness: data are as current as possible and available at a useful frequency for decision making

The third step is to develop a plan to collect all the data painstakingly identified as necessary in the previous step. Consider investing in surveys, requesting specific field reporting or intelligence collection, or deploying more technical tools such as satellite photos or geo-located data. A reasonable guideline is to invest 5 percent of one’s budget into M&E and requisite staff time.

The fourth step is policy adaptation. Schedule regular evidence review check-ins to evaluate the data collected so far and evaluate the progress of policy. Notice the ways in which the policy is seems to be working, and where unexpected obstacles are arising. Use this feedback to make intentional policy adaptations, including refining the M&E plan itself, based upon what is learned.

The final step – learning – typically occurs after a policy effort or strategy is completed. The goal of this stage is to reflect on the entire policy process and capture lessons about success and failure. These lessons will be useful for future policymaking efforts that make take on similar challenges or attempt to use similar strategies.

Conclusion

I admit to being a zealot when it comes to M&E of foreign policy. I believe it is negligent that the State Department – and the field of foreign policy more generally – does not prioritize M&E for policies and strategies. While the precise discernment of success and failure of diplomatic interventions is often impossible, robbing oneself of feedback altogether is self-imposed ignorance. It is hard to think of another profession with a more tenuous relationship with feedback than foreign policy.

The result is that conversations about efficiency (and productivity, and effectiveness) are grounded almost entirely in subjective perceptions rather than any sort of real evidence. This creates fertile ground for the politicization of foreign policy that is antithetical to expertise.

Meanwhile, the Department of State is struggling to wield influence within the national security establishment. The National Security Council continues to grow in size and influence, not to mention the Department of Defense and Intelligence Community, often relegating the State Department to little more than messengers and tour guides.

Diplomacy needs to up its game. Implementing a new M&E system for policy and strategy is perhaps the most powerful and necessary challenge facing the State Department. The integration of USAID into the Stae Department makes this process more urgent. Nevertheless, it may take years to develop an effective M&E system. It will require investment, trial and error, and dedicated leadership. Ultimately, the quality of the United States’ foreign policy decision-making process may be a more important source of geopolitical power than all of the missiles and fighter jets money can buy. It will be worth the investment of time and money to get M&E for diplomacy right.