The Benefits and Challenges of M&E for Strategy

Part 2 in a series about M&E at the State Department

Image produced by DALL-E

Read part 1 of the M&E series first:

Can We Monitor and Evaluate US Foreign Policy Strategy?

The Opportunities

A mentor once explained that there are two ways for an organization to achieve success: A) achieve its objectives; or, B) fail to achieve its objectives but learn something important in the attempt.

But there is currently little formal feedback to assess the effectiveness of diplomatic strategies and foreign policy, resulting in a culture with a weak understanding of its own tools. Today’s State Department is not dedicated to learning.

Monitoring and evaluation (M&E) practices (or something analogous) could be pivotal for improving the health of the State Department culture.

Creating an effective system of feedback and learning at the State Department would improve its reputation for solving problems, build a culture dedicated to expertise, and help return diplomacy to its rightful role in the center of US national security.

A rigorous sense of “what works” is an essential ingredient for designing useful training the next generation of diplomats, and perhaps even starting to construct diplomatic doctrine. Investments in understanding diplomacy’s successes and failures could also strengthen the merit evaluation system, which is currently derided as a creative writing contest.

One of M&E's strengths is that it forces conversations about exactly what success looks like before the policy is deployed. This is an antidote to the scourge of watered-down, wishy-washy policies that often emerge from the State Department’s maligned clearance process — policies that are calculated to offend nobody rather than create real change.

Strong M&E practices also enhance transparency and accountability. Whereas the State Department has a hard time demonstrating the value of diplomacy to Congress and the American people, M&E can help provide discrete evidence of diplomacy’s impact.

And yet, these benefits are largely theoretical. M&E systems are largely untested for high-level strategy and policymaking at the State Department. While such tools are tried and tested for foreign assistance and program-level work, high-level M&E work presents many new challenges and complications.

The Challenges of Complex, Strategic Environments

M&E is easiest in highly controlled, static environments, but the world of international affairs rarely provides laboratory-like conditions. Diplomacy is not a widget-making factory in which the foreman can easily observe each employee’s productivity. As the scholar James Q. Wilson explains in his book, Bureaucracy, settings where impact is difficult to observe tend to leave managers grasping for control over subordinates by imposing compliance-oriented performance management procedures.

Performance management is not all bad. It is a subtype of M&E that focuses inward on compliance issues: How many hours have we worked? How much money have we spent? Are we doing the activities we said we’d do? Are we following proper procedures? Answering these questions can be useful in certain circumstances, but performance management says nothing about whether a team actually achieved anything.

This situation describes the State Department’s sprawling system of offices and embassies. For example, one of the goals articulated in the 2022-2026 Joint Strategic Plan for the State Department is Promoting a Stable Cyberspace: “By September 30, 2026, sustain and enhance international cooperation to promote the U.S. vision of an open, interoperable, reliable, and secure Internet and a stable cyberspace.”

The Department dutifully reported that it “met” this goal in 2023 because it conducted 175 new or sustained diplomatic engagements on cyber issues – an increase of 171% over the previous year! I’m sure Congress and the Secretary of State were pleased that the team kept busy, but this figure is functionally meaningless; it says nothing about whether all this activity made cyberspace more open and secure.

As fp21 fellow Toby Weed wrote in an excellent article about measurement in foreign policy,

Many officials worry about the unintended consequences of measurement in foreign policy. The scholar Kishan Rana suggests that data-driven management “may misdirect diplomatic activity to the measurable forms and de-valorize the more subtle, long-term work of relationship building.” For instance, a 1983 joint resolution of Congress asked the State Department to produce a report counting how often other countries voted with the United States in the UN General Assembly. The goal was to nudge embassies to invest more in multilateral diplomacy, but embassies were observed “equally weighting ‘throw-away’ policies with real impactful U.S. policy goals.” As the management guru Peter Drucker famously observed, “What gets measured gets managed – even when it’s pointless to measure and manage it, and even if it harms the purpose of the organization to do so.”

It is little wonder, then, that policymakers at State are resistant to such measurement activities. Good policy judgment can become distorted by incentives to maximize meaningless performance metrics. Rigid adherence to top-down directives and compliance frameworks can actually erode the quality of programs and policies. The expertise and judgment of frontline staffers to adapt to new challenges and contexts is an important ingredient for success. This may be especially true in complex strategic settings.

Some go even further, suggesting it is not possible to usefully measure diplomacy. “Diplomacy is fuzzy,” explained a former Ambassador and Dean of the Department’s Foreign Service Institute, “it's almost impossible to quantify what we do, and in fact I think that there's a great danger in trying to quantify it." Again, from Weed’s article:

A core challenge is that “the results of the work of foreign ministries are typically indirect, difficult to measure and not easy to influence directly,” and “whether an expected result is obtained does not depend completely on the actions of the foreign ministry.” (Mathiason, 2007, p. 226). That is, it can be hard to determine causality in diplomacy. Did a leader withdraw their armed forces because of skilled negotiation, or was it due to the opposition’s resolve? Did a civil war end because of the efforts of third party mediators, or because the warring parties simply grew tired of the violence?

A related methodological challenge is that many strategies are often deployed at the same time in a single policy environment. For example, there may be dozens of separate China strategies and policies with complex interdependencies. This confounds methodological approaches that seek to isolate a single, linear cause and effect.

The promise of AI and big data leads many M&E practitioners to surmise that more data is always better. Some programs today may proliferate hundreds of metrics to be tracked. But, more often than not, this attitude is naïve and amounts to little more than a big waste of time. This “M&E overkill” problem may be exacerbated in increasingly complicated strategic settings.

Even if the previous challenges are surmountable, the State Department’s culture strenuously resists critical feedback. If a negative result can harm one’s career, robust M&E practices will rightly be perceived as threatening. This resistance to feedback is political, exacerbated by a climate of distrust between Congress and the State Department. I sometimes joke that “diplomacy is the art of setting ambiguous goals and always claiming success.” A culture that forbids failure is incompatible with effective M&E.

A final challenge is about optics: M&E may also have a branding problem, associated with the scars of past failures rather than as an instrument for success. M&E may also be too closely linked to programmatic work to be accepted by policymakers who see themselves as distinct from (or superior to) foreign assistance work.

What’s Next?

I hope my review of the challenges of strategy-level M&E has not convinced you that I am wasting my time. Instead, my goal here is to be clear-eyed about the challenges facing proponents of high-level M&E for policy and strategy. I am interested in hearing from you: What have I missed? Have I inappropriately minimized any of these challenges? Please send me your thoughts.

My next article will start to flesh out some ideas for how these challenges might be surmounted, and to advance the conversation about creating a more effective culture of policymaking at the State Department.

Stay tuned!