How destroying 100 computers cost the taxpayer $2.7 million. A lesson in communication.

by theintrospectivemaster

If your computers get a virus…The best option is to pay someone to destroy it, that is until you run out of money to pay someone to destroy it.

How $2.7 million of taxpayer collected revenues were essentially “wasted” as a result of poor communication and overacting.

Image

In December of 2011, the Department of Homeland Security contacted two federal agencies about possible malware infections being identified in their networks. Those two agencies were  the National Oceanic and Atmospheric Administration (NOAA) and the Economic Development Administration (EDA).

One of these agencies cleared the problem up within a few weeks, while the other took nearly a year and some additional “destructive measures”.

The agency that took over  a year to correct the identified malware infection was the EDA.

What happened over the course of a year is a strong case example of when communication lines breakdown between levels of the hierarchy within the agency organization and the damage of poor senior leadership.

The EDA responded to the malware warning by first removing its connections to the Hubert Hoover buildings network connections (essentially disconnecting itself from the world). The agency had thought that nearly half of their computers had become infected (146 out of 250). This prompted the agency to contact an IT contractor (Commerce department incident response team) to come to the agency and inspect how many machines were actually infected by malware.

The response team initially reported that 146 machines had been infected, however, this was a false analysis. The incident response team then communicated to the EDA that it had revised its analysis and found only two machines were actually infected.

Unfortunately, the revised analysis was not made clear by the incident response team and the EDA IT team did not understand nor get confirmation about the revised number of actually infected machines.

This failure of communication between the two departments was not discovered until a year later after an audit report was conducted that discovered this lapse in adequate notification.

Under the impression that half of its machines were infected with malware, “the chief information officer, who was relatively new to the bureau, thought the agency was under [a percieved threat of a cyber attack] based on what he knew about EDA’s long-standing cyber challenges. The IG said EDA had outstanding cyber vulnerabilities, which auditors first highlighted four years ago.”

Under the impression that the agency was under a possible attack (thinking half of its hardware was infected) decided on a policy action which physically destroyed (uninfected) desktop computers, printers, cameras, keyboards, and even mice. The destruction stopped when the agency had run out of money to pay for destroying the hardware.

In total, over $170,000 of hardware was physically destroyed before the agency had run out of money.

The total cost to the taxpayer was $2.7 million:

-$823,000 for the contractor to investigate
-$1,061,000 for the acquisition of temporary hardware (requisitioned from the Census Bureau since they destroyed half of their own equipment)
-$4,300 to destroy $170,500 in IT equipment (service paid to destroy the hardware)
-$688,000 paid to contractors to assist in development of a long-term response.

Additional cost unaccounted for was the disruption of the agencies provided services as it dealt with this incidence.

All of the details that led up to this unnecessary and “overacted” destruction was detailed by the Commerce Departments Office of the Inspector General Report released June 26th, 2013.

$2.7 million was not a total lost as it allowed the agency to “learn” from its mistake in ensuring accurate and confirmed information sharing between agencies and that when in doubt, you do not have to throw the baby out with the bathwater.

As Joseph Beal, the chief information security officer at CCSi stated:
“We learned in this and many other cases, you just can’t throw people and technology at a problem. More so, you need to sometimes just pick up the phone and say, ‘This is exactly what we are seeing and this is what we have. We need to at least try to define some way of containing what we have and what is the impact of our business mission,'” said Beal, who has worked with several agencies to address cyber-security challenges, including DHS, Transportation and the Marine Corps. “I think that is where this situation fell short. You see there were multiple emails that went back and forth between multiple constituents. After four or five emails, there’s a rule of thumb for my guys that says if you have four or five emails from someone, you need to pick up the phone so you have clear understanding of the issue and you know what’s going on.”

Courtesy of Arstechnica
http://arstechnica.com/information-technology/2013/07/us-agency-baffled-by-modern-technology-destroys-mice-to-get-rid-of-viruses/

Courtesy of the Office of the Inspector General at the Department of Commerce
http://www.oig.doc.gov/OIGPublications/OIG-13-027-A.pdf

Courtesy of Federal news radio
http://www.federalnewsradio.com/241/3382009/EDAs-overreaction-to-cyber-attack-highlights-every-agencys-challenge

Advertisements