Disclaimer: The views expressed in this paper are those of the author only. They have not been approved by the Federal Bureau of Investigation (FBI), the Department of Justice (DOJ), or the IIT Research Institute (IITRI).
Motivation: I was one of the members of the Independent Review team for Carnivore, and one of the coauthors of the report. I do not agree with much of the press coverage regarding the final report. Many of these reports indicate that the review team did not find very much wrong with Carnivore. That is simply not the case, and I would like to address some of the issues raised by those reports.
1. Many news articles have been written which indicate that the IITRI report was nothing but a "whitewash", or at the very least, extremely favorable of Carnivore. A selection of these are: 1) A Reuters wire report, "FBI Electronic Wiretap Needs Few Changes". It goes on to say, "They recommended minor modifications to the existing system ...", 2) A Computerworld article quotes, "a generally favorable draft report by the independent team that reviewed it.", 3) An Infoworld article's headline is "Study supports Carnivore". 4) A story from the Associated Press on CNN has a headline of "Critics blast report that supports FBI's Carnivore." 5) An article on the Wired website says, "FBI Gets Carnivore Approval." 6) A report from Applelinks is titled, "FBI Email Snooper Technology Whitewash?" 7) An article from Newsfactor network says, "Panel OKs FBI Internet Spy Service." 8) An article on C|Net's News.com is titled, "Report finds risk but supports Carnivore email surveillance."
Response: The IITRI report notes numerous problems with Carnivore. A selection of problems noted are:
The Final Report makes 10 recommendations for improvements to Carnivore. The report, in Section 5.4 recommends, "Provide individual accountability and audit for all Carnivore actions." In the rationale for this recommendation it describes re-writing the whole program so it runs as a service rather than an application. In that way, the Windows NT Identification and Authentication features, and Auditing capabilities could be used. The report also suggests changing the way access to the advanced features is obtained. In addition, in both Section 5.4 and in Section 5.6, there are suggestions for major changes to both auditing and data logging. Clearly, these are not "minor modifications." The implementation of a good many of those recommendations is anything but trivial.
How can a report that found so many problems could be described as "supporting Carnivore?"
Finally, several of the news reports indicated that IITRI supports Carnivore, or has approved Carnivore. IITRI was not given approval authority, thus could not do so, even if it wanted to. IITRI took no sides in this review. That's the whole idea of being independent. The report simply states the facts about what was found, and makes recommendations for improvement. It is up to DOJ or Congress, or some other U.S. Government entity to decide if Carnivore is "approved."
2. In another story, "Researchers fault independent review of Carnivore." Among the complaints in the researcher's paper, is "There is inadequate discussion of audit and logging (both of logs maintained by Carnivore itself and of logs maintained by the host operating system and supporting tools)."
Response: The discussion must not have been too inadequate, as two paragraphs later, the same paper says, "We also urge that the report's recommendations with regard to logging and audit be considered carefully and made a high priority."
Many of the other criticisms of their paper have been addressed in the IITRI final report, and I in particular am appreciative for the time taken by these reviewers (and others, especially the Privacy Foundation) to suggest items that we may have missed. In particular, the researcher's are correct that there is more evaluation that could have been done (i.e. analysis for string buffer overflows, RADIUS usage, etc.), but there simply were not sufficient time or resources to complete such analysis. Finally, IITRI agreed completely with their observation that, "we must emphasize that no single review can ever capture every potential problem with critical software of this complexity, especially when it must be run under a wide range of operational environments. Furthermore, as the software is enhanced and the environment under which it runs evolves, existing reviews may well be rendered obsolete. As such, the Department of Justice must consider an on-going process to maintain confidence in the system." In Section 5.9 of the Final Report, the recommendation is, "Work toward public release of Carnivore source code by eliminating exploitable weaknesses. Until that can be done, continue independent evaluation of each Carnivore version to assess effectiveness and risks of over- and under-collection."
3. The ACLU made a press release entitled, "ACLU Slams Biased Review Team Thumbs-Up for Government Snoopware Program "Carnivore" ." It it, they state, "This report is, at best, a fuzzy snapshot of Carnivore, and it will be obsolete in two months when the FBI comes out with the next version of Carnivore."
Response: Please note the point above which agrees that there needs to be ongoing independent evaluation of future versions. While IITRI was given a copy of the alpha-test Version 2.0 system code, a conscious decision was made not to evaluate it for several reasons. First was the limited amount of time and resources at hand. Second, was that version 1.3.4 is what is currently being used. And third, alpha-test code is notoriously bug filled, and subject to major changes. Thus, the limited resources and time we had were devoted to the current version.
4. The ACLU press release goes on to state, "Despite the review team's assurances in news stories today that Carnivore does not "overcollect" evidence, documents obtained through a Freedom of Information Act (FOIA) request by the ACLU clearly state that Carnivore could "reliably capture and archive all unfiltered traffic to the internal hard drive."
Response: The ACLU is correct that Carnivore could "reliably capture and archive all unfiltered traffic to the internal hard drive." However, as noted in the Final Report, "Carnivore does not come close to having enough power 'to spy on almost everyone with an e-mail account.' In order to work effectively it must reject the majority of packets it monitors. It also monitors only the packets traversing the wire to which it is connected. Typically, this wire is a network segment handling only a subset of a particular ISP’s traffic. The main limitation is the amount of storage. For example, if Carnivore were collecting all traffic on a link that has a steady 25-Mbps traffic rate, the 2-Gbyte Jaz disk will be full in about 11 minutes. In the time needed to change disks the input buffers would likely overflow and data would be missed. Even if collecting to fast hard drives, the amount of data to be recorded would quickly overflow the amount of storage available. A 60-Gbyte hard disk could be filled in about 5–6 hours. If traffic were faster than 25 Mbps, then the storage would fill even faster."
5. The ACLU press release, many news stories, and politicians (most notably House Majority Leader Dick Armey, R-Texas) have accused IITRI of being biased in favor of Carnivore because of previous Government contracts, donations by some members of the review team to various political campaigns, and security clearances held by some of the review team members.
Response: Again, speaking only for myself, my time of employment with NSA was a tremendous training ground in the field of Information Security. While I still hold an active security clearance, I would not let that compromise my professional integrity. As a Certified Information Systems Security Professional (CISSP), I have a certain code of ethics to live up to as well. In addition, if I were biased in favor of Carnivore, I would not have posted this rebuttal of news articles (without permission from my employer, the FBI, or DOJ) that indicate IITRI has "approved" Carnivore. Finally, the Electronic Privacy Information Center (EPIC) in it's comments to the DOJ, says that, "While DOJ and FBI spokespersons have attempted to characterize the Draft Report as a vindication of the Carnivore system, a close reading of the reviewers' conclusions in fact validates much of the public and Congressional criticism that has been expressed since the existence of the surveillance system was revealed earlier this year." If this review panel was biased in favor of Carnivore, I don't think EPIC would have been able to make this claim. I would welcome the opportunity to discuss Carnivore with Congress if asked.
6. In an article on C|Net's News.com titled, "Report finds risk but supports Carnivore email surveillance" it says, "We are overall pleased with the findings," said FBI spokesman Paul Bresson. "They are quite positive and substantiate the contentions that we've made all along--that when the Carnivore system is performed accurately it only provides investigators with the information it is designed for under a court order."
Response: Please see point number 1 above. Yes, if Carnivore is used accurately, it will only provide the information allowed by the court order. However, note that there is little in the way of accountability to assure that it will be used accurately. There are also some technical problems that were noted. To say that the report is "quite positive and substantiate(s) the contentions that we've (the FBI) made all along" is a significant reach.
While I speak only for myself and not for the whole IITRI team in this posting, I think they would agree that we tried very hard to do an objective independent review that was not biased either in favor of Carnivore, nor against it. We simply wanted to do as thorough an evaluation as we could in the approximately 5 weeks that we were given (including report writing time), and present the facts as we saw them.
Much has been said in the press about the recommendation to, "Continue to use Carnivore ..." This has been taken out of context. The complete recommendation reads, "Continue to use Carnivore rather than less-precise, publicly available sniffer software, such as EtherPeek, when precise collection is required and Carnivore can be configured to reflect the limitations of a court order." Clearly, any tools which provides some level of filtering is better to be used for surveillance than the other tools referenced, which simply grab everything. That recommendation does not say, "Continue to use Carnivore as is, for it is a perfect device with absolutely no privacy concerns." If that was the recommendation, there would not have been need for the other 9 recommendations that were made.
Carnivore should ultimately be a useful tool for the FBI to use for internet surveillance. In order to protect the privacy of the public, the recommendations of the report should be implemented as soon as possible. In particular, without providing the Identification, Authentication, and Audit controls that are recommended, I suspect that most smart defense attorneys will be able to have Carnivore's evidence excluded due to not having a solid chain of custody. However, the FBI should do everything it can to provide full disclosure of Carnivore so that the public can convince itself that Carnivore will not invade the privacy of individuals not subject to court authorized wiretaps.
Repeat of Disclaimer: The views expressed in this paper are those of the author only. They have not been approved by the Federal Bureau of Investigation (FBI), the Department of Justice (DOJ), or the IIT Research Institute (IITRI).