Link to the University of Pittsburgh Homepage
Link to the University Library System Homepage Link to the Contact Us Form

Improving the Analysis of Foreign Affairs: Evaluating Structured Analytic Techniques

Coulthart, Stephen (2015) Improving the Analysis of Foreign Affairs: Evaluating Structured Analytic Techniques. Doctoral Dissertation, University of Pittsburgh. (Unpublished)

Primary Text

Download (2MB)


Research suggests that foreign affairs analysis is weak—even the best analysts are accurate less than 35 percent of the time (Tetlock 2005). To compensate for analytic weaknesses, some have called for the use of structured analytic techniques, that is, formalized judgement-driven methods. This imperative was enshrined in the Intelligence Reform and Terrorism Prevention Act (2004), which mandates that analysts use these techniques. This research investigates how the techniques have been applied in the U.S. intelligence community (IC) while making a modest attempt to evaluate 12 core techniques.

The investigation of how the techniques are applied is based on semi-structured interviews with 5 intelligence experts and a survey of 80 analysts at an IC agency, along with follow-up interviews with 15 analysts. Interestingly, 1 in 3 analysts reported never using the techniques. Two factors were related to the use of the techniques: analytic training (p=0.001, Cramer's V=0.41) and the perception of their value (p=.049, Cramér's V= 0.23). There was not a statistically significant relation between the time pressure under which analysts work and their use of the techniques (p=0.74).

Questions about the effectiveness of the techniques were answered in part by employing a “systematic review,” a novel methodology for synthesizing a large body of research. A random sample of more than 2,000 studies, suggests that there is moderate to strong evidence affirming the efficacy of using three techniques: Analysis of Competing Hypotheses, Brainstorming, and Devil’s Advocacy. There were three main findings: face-to-face collaboration decreases creativity, evidence weighting appears to be more important than seeking disconfirming evidence, and conflict tends to improve the quality of analysis. This research also employed an experiment with 21 graduate intelligence studies students, which confirmed the first two findings of the systematic review.
The findings of the dissertation represent a contribution to “evidence-based intelligence analysis,” the systematic effort to develop a robust evidence-base linking the use of specific analytic techniques to the improvement of analysis in foreign affairs. Future research might build on the evidence-base presented here to improve intelligence analysis, one of the most important areas of judgment in foreign affairs.


Social Networking:
Share |


Item Type: University of Pittsburgh ETD
Status: Unpublished
CreatorsEmailPitt UsernameORCID
Coulthart, Stephensjc62@pitt.eduSJC62
ETD Committee:
TitleMemberEmail AddressPitt UsernameORCID
Committee ChairDunn, Williamdunn@pitt.eduDUNN
Committee MemberKenney, Michaelmkenney@pitt.eduMKENNEY
Committee MemberBird,
Committee MemberWilliams, Phillipridgway1@pitt.eduRIDGWAY1
Date: 29 September 2015
Date Type: Publication
Defense Date: 4 June 2015
Approval Date: 29 September 2015
Submission Date: 26 August 2015
Access Restriction: No restriction; Release the ETD for access worldwide immediately.
Number of Pages: 238
Institution: University of Pittsburgh
Schools and Programs: Graduate School of Public and International Affairs > Public and International Affairs
Degree: PhD - Doctor of Philosophy
Thesis Type: Doctoral Dissertation
Refereed: Yes
Uncontrolled Keywords: Intelligence analysis, intelligence, foreign affairs analysis
Date Deposited: 29 Sep 2015 14:36
Last Modified: 15 Nov 2016 14:30


Monthly Views for the past 3 years

Plum Analytics

Actions (login required)

View Item View Item