Friday, February 22, 2013

More on Mandiant's APT1 Report: Guilt by Proximity and Wright Patterson AFB

The blog post that I wrote earlier in the week "Mandiant Report APT1 Has Some Critical Analytic Flaws" was based upon my history of interacting with some Mandiant folks online and in person as well as my interpretation of the facts as they were presented in the report. Thanks to some feedback that I received from readers as well as a teleconference that I had with three Mandiant executives yesterday, I've learned some new things that color my earlier article.

1. Mandiant has expanded their original definition of APT

Yesterday, I spoke with three Mandiant executives and learned that their meaning of the term has evolved with the times and it no longer represents a Who, but a What; or more precisely, a well-documented multi-staged process that attackers from multiple nation states have adopted. Mandiant has not formally announced this change (although they probably will later this year) so when I wrote my article on their APT1 report, I was referencing their former definition which I know now is no longer in use. While Mandiant often sees Chinese hackers at work stealing trade secrets and intellectual property, they also acknowledge that other countries may be doing the same thing. I'm happy to report this change because it's been a point of contention between myself and some folks at Mandiant ever since 2010. I'm glad that we're closer to being on the same page.

2. Mandiant did some negative analysis before publishing their report

Another thing I learned from that phone meeting was that there was an effort made to look at alternative  scenarios that might explain the facts that Mandiant had before them. Mandiant isn't a part of the Intelligence Community (even though they have some ex-IC folks working there) and they don't have the time, resources, or manpower to do the same type of analysis that is performed at Langley. It's also not their mission to do nation state attribution so I want to give them at least some credit for the counter-analysis that they did do, even though the significance of their conclusion demanded a more rigorous methodology in my opinion.

Thanks to input from my readers, I've also learned some additional negatives about the report.

1. Mandiant's reliance on proximity to prove its claim that PLA Unit 61398 is Comment Crew aka APT1 is harmed by simple geographical mistakes such as:
  • p.10 of Mandiant's report refers to Hebei as a borough in Shanghai. Hebei is actually a province about 600 miles and 3 provinces away from Shanghai.
  • NEC and Intel along with many other high tech companies operate less than 8 miles from PLA Unit 61398 and all would be served by the same fiber optics cable provided by China Unicom.
  • There are more free proxy servers in China than anywhere else in the world and some of those proxy servers overlap with the IP blocks identified in the Mandiant report. 
  • An IP registration for UglyGorilla was described by Mandiant as being "across the river" from Unit 61398. In fact, it was 33 kilometers away.
2. Speaking of guilt by proximity, one of the "obviously false" IP address registrations according to Mandiant was for an address in Yellow Spring, Ohio. It should have been spelled "Yellow Springs". However, a cursory check shows that the address is real except for that one missing "s". Even more interesting is that it is located 13 miles from Wright-Patterson Air Force Base which is the Air Force's "boot camp for cyber warriors".

Directions via Google Maps
Either this is a bizarre coincidence or someone on the Comment Crew has a wicked sense of humor. As it turns out, Michael Murphy is a real person who lives in Yellow Springs, Ohio and who used to be the director of admissions at Antioch College whose office is located at 795 Livermore St., Yellow Springs, OH - the address that Mandiant assumed was fake.

3. (UPDATED 23 FEB 13)  On page 11 of the report, under "Size and Location of Unit 61398's Personnel and Facilities", Mandiant wrote "public sources confirm that in early 2007, Jiangsu Longhai Construction Engineering Group completed work on a new building for Unit 61398 located at Datong Road 208 within the Pudong New Area of Shanghai. At 12 stories in height and offering 130,663 square feet of space, we estimate that this building houses offices for approximately 2,000 people." In reality, it's the Unit's pre-school:

English translation via Google Translate

And this isn't all of the errors. It's just a fraction. While each may seem minor, collectively they call into question Mandiant's final conclusion and, at the very least, should serve as a lesson to policy makers not to rush to judgment on matters of attribution. There's plenty of evidence that China engages in cyber espionage without upping the ante by trying to claim the Peoples Liberation Army is involved. 

At the end of the day it's important to remember that Mandiant isn't a U.S. government agency nor are they trained to do intelligence collection and analysis at the same level that it's done at Langley. They're a group of highly skilled professionals who serve their customers as incident responders and have a well-deserved reputation for excellence. 

Tuesday, February 19, 2013

Mandiant APT1 Report Has Critical Analytic Flaws

Mandiant's APT1 report is the latest infosec company document to accuse the Chinese government of running cyber espionage operations. In fact, according to Mandiant, if a company experiences an APT attack, then it is a victim of the Chinese government because in Mandiant-speak, APT equals China.

"We tend to perceive what we expect to perceive" 
- Richard J. Heuer, "The Psychology of Intelligence Analysis

The fact that Mandiant refuses to acknowledge that other nation states engage in cyber espionage when the facts show otherwise demonstrates what Heuer calls an "expectation bias", but it's much worse than that.

Mandiant's alleged proof is summarized in Table 12 (pp. 59-60): "Matching characteristics between APT1 and Unit 61398". Mandiant's entire premise that APT1 is PLA Unit 61398 rests on the connections made in that table and that no other conclusion is possible:
"Combining our direct observations with carefully researched and correlated findings; we believe the facts dictate only two possibilities: Either a secret, resourced organization full of mainland Chinese speakers with direct access to Shanghai-based telecommunications infrastructure is engaged in a multi-year, enterprise scale computer espionage campaign right outside of Unit 61398’s gates, performing tasks similar to Unit 61398’s known mission or APT1 is Unit 61398." (APT1, p. 60)
If this report were written by a professional intelligence analyst at CIA, it would most likely undergo a vetting process known as ACH (Analysis of Competing Hypotheses):
"Analysis of competing hypotheses, sometimes abbreviated ACH, is a tool to aid judgment on important issues requiring careful weighing of alternative explanations or conclusions. It helps an analyst overcome, or at least minimize, some of the cognitive limitations that make prescient intelligence analysis so difficult to achieve."
In other words, ACH forces the intelligence analyst to look for all alternative hypotheses and assess them one at a time to see which best fits the data collected. This is rarely if ever done by information security companies, and it's the single biggest objection that I have when it comes to individuals making claims of attribution to nation states. Heuer's iconic "Psychology of Intelligence Analysis" explains why ACH is so important:

"The way most analysts go about their business is to pick out what they suspect intuitively is the most likely answer, then look at the available information from the point of view of whether or not it supports this answer. If the evidence seems to support the favorite hypothesis, analysts pat themselves on the back ("See, I knew it all along!") and look no further. If it does not, they either reject the evidence as misleading or develop another hypothesis and go through the same procedure again. Decision analysts call this a satisficing strategy. (See Chapter 4, Strategies for Analytical Judgment.) Satisficing means picking the first solution that seems satisfactory, rather than going through all the possibilities to identify the very best solution. There may be several seemingly satisfactory solutions, but there is only one best solution." 
"Chapter 4 discussed the weaknesses in this approach. The principal concern is that if analysts focus mainly on trying to confirm one hypothesis they think is probably true, they can easily be led astray by the fact that there is so much evidence to support their point of view. They fail to recognize that most of this evidence is also consistent with other explanations or conclusions, and that these other alternatives have not been refuted."

If Mandiant or another organization were to use ACH on this evidence, here's how Heuer recommends it be done. It's an 8-step process:

1. Identify the possible hypotheses to be considered. Use a group of analysts with different perspectives to brainstorm the possibilities.
2. Make a list of significant evidence and arguments for and against each hypothesis.
3. Prepare a matrix with hypotheses across the top and evidence down the side. Analyze the "diagnosticity" of the evidence and arguments--that is, identify which items are most helpful in judging the relative likelihood of the hypotheses.
4. Refine the matrix. Reconsider the hypotheses and delete evidence and arguments that have no diagnostic value.
5. Draw tentative conclusions about the relative likelihood of each hypothesis. Proceed by trying to disprove the hypotheses rather than prove them.
6. Analyze how sensitive your conclusion is to a few critical items of evidence. Consider the consequences for your analysis if that evidence were wrong, misleading, or subject to a different interpretation.
7. Report conclusions. Discuss the relative likelihood of all the hypotheses, not just the most likely one.
8. Identify milestones for future observation that may indicate events are taking a different course than expected.

I don't have the time to run Mandiant's evidence through an ACH process but I'd like to propose that a volunteer group of intelligence students at Mercyhurst Institute of Intelligence Studies do that very thing. My friend Professor Kris Wheaton who teaches there and writes the outstanding Sources and Methods blog is an expert in this area and I'm hopeful that he'll pick up the challenge.

In the meantime, the following table has four columns. The first three are from Mandiant's table 12. The "Other" column contains a partial group of alternatives that I've provided for each of Mandiant's "characteristics". These alternatives need to be analyzed and ruled out using a rigorous analytic process like ACH before Mandiant or anyone else can claim that APT1 is a part of China's Peoples Liberation Army.




In summary, my problem with this report is not that I don't believe that China engages in massive amounts of cyber espionage. I know that they do - especially when an executive that we worked with traveled to Beijing to meet with government officials with a clean laptop and came back with one that had been breached while he was asleep in his hotel room.

My problem is that Mandiant refuses to consider what everyone that I know in the Intelligence Community acknowledges - that there are multiple states engaging in this activity; not just China. And that if you're going to make a claim for attribution, then you must be both fair and thorough in your analysis and, through the application of a scientific method like ACH, rule out competing hypotheses and then use estimative language in your finding. Mandiant simply did not succeed in proving that Unit 61398 is their designated APT1 aka Comment Crew.

UPDATE (22 FEB 2013): I've published a follow up to this article: "More on Mandiant's APT1 Report: Guilt by Proximity and Wright-Patterson AFB"

Monday, February 11, 2013

Personal Reflections on Suits and Spooks DC 2013

Now that I've had a chance to decompress from and reflect upon the terrific Suits and Spooks DC conference last weekend, I want to share some surprising shifts in thinking that occurred for me during those two days.

The speakers were all terrific, but some topics triggered a lot of passionate debate amongst the speakers and the attendees. You can get a sense of what transpired by reading the live Twitter stream from the event. How those passions were channeled and the manner in which some speakers conducted themselves in the heat of the moment really impressed me. Keep in mind that the speaker to attendee ratio was 1:4. That's unheard of at most conferences. In fact, I don't know of another event where it's that low, which is too bad because I believe that it makes for a much more valuable experience for both the attendees and the speakers.

Some of the areas in which my thinking has shifted includes:

International Cooperation. The international speakers that I invited to attend did a phenomenal job. I particularly want to commend Marco Obiso of the ITU. He was on the receiving end of a lot of heated debate and pointed comments and parried them all without loosing his temper (I can't say the same about some of his opponents). Marco did an excellent job of explaining the ITU's sometimes controversial platform while always responding to his critics in a balanced and informed way. The lesson for me was in watching how he wants engagement while his critics don't. Obiso and the ITU came out ahead because of that. In an adversarial debate, the side which has a deep expertise and is confident in their ability to engage can do so in a balanced way. Some of the ITU opponents weren't able to do that and they lost the debate as far as I was concerned.

Kaspersky. I take a lot of shots at Eugene Kaspersky, but his employee Roel Schouwenberg did a terrific job in explaining Red October. He provided some new information - that Kaspersky's client who brought ROCRA to their attention was from the European Union. Despite Kaspersky's contractual and non-contractual relationships with the Russian government, they are the world's fourth largest security software vendor and they arguably do the best work in writing reports that describe important malware attacks. Roel will always be a welcome speaker at future Suits and Spooks events.

Hack-Back and Active Defense. Some of the speakers who favored hack-back were successful in describing scenarios that made sense and seemed possible to implement without causing unfortunate blow-back. Other speakers took "hack-back" off the table when describing other active defense practices, particularly deceptive techniques. My take-away was that active defense including hack-back could probably be implemented responsibly by a few private parties but certainly would be taken advantage of by less responsible ones so I think that law enforcement oversight is a requirement. Also, the CFAA definitely needs to be modified from its out-dated current language.

Opinions Derived From Online Interactions. One of the most refreshing things that happened to me was how much I enjoyed interacting with people whom I had previously only known online. We all form opinions about people based upon limited interactions. In today's networked world of social media, many of those opinions are formed without the benefit of personal interactions. And sometimes those opinions conflate individuals with the companies that they were formerly employed by. Last week's Suits and Spooks was a joy for me to participate in because I was newly impressed by some people who I had previously only known from the news or social media. Those newly positive impressions came about precisely because of the extended interaction (two days), low attendee:speaker ratio, and heated discussions. Just meeting someone in "real life" often isn't enough to change perceptions. Extended interaction in combination with engagements or arguments over heated issues makes all the difference.

Feedback. In closing, I'm happy to share some of the feedback that I received from speakers and attendees of Suits and Spooks DC 2013:

"SNS provides a first-class forum to openly (and professionally) debate cyber security policy issues.  Everyone benefits from hearing all sides of the issues and, correspondingly, leave with new perspectives." - Robert Bigman, former CISO, Central Intelligence Agency

"One of those rare conferences where even the speakers learn something new."
- Stewart A. Baker, former General Counsel, National Security Agency; former Ass't Secretary for Policy, Department of Homeland Security

"Suits and Spooks provided a unique forum for discussing the hard, unanswered questions with leading technical and policy experts."  - Jim Denaro, founder of CipherLaw

"SNS provided a spotlight into the evolving edge of cyber." - Greg Hoglund, former founder, CEO of HBGary, Inc.

"Suits & Spooks brought together that right mix of backgrounds that allowed for informed discussion on the challenges of employing offensive techniques in support of defensive measures.  The networking alone made this conference worth being there." - Jim Butterworth, Commercial Chief Security Officer, HBGary, Inc.

"The most interesting, provocative, lively discussion of cyber conflict issues I’ve seen. And that’s my layman’s view." - Tom Gjelton, National Public Radio journalist

If you attended SNS DC 2013 and want to send me a quote to use, please do so via Twitter or email. If you didn't attend, but you want to be informed about upcoming events, you can follow Suits and Spooks on Twitter. Our next event will be announced shortly.