The financial sector’s disaster recovery effort has come a long way in a short time. A 2003 report (PDF) from the General Accounting Office found that at least four major Wall Street firms maintained no backup data centers for disaster recovery at that time, while six other firms’ facilities were located within 10 miles of their primary site. Of those six, four were “critical organizations” with less than 5 miles separating their primary and secondary data centers, including some as close as two miles. Numerous firms maintain backup sites in northern New Jersey or Brooklyn.

The GAO didn’t publicly name the culprits, but regulators urged firms handling critical market data to “establish back-up facilities a significant distance away from their primary sites,” according to a 2003 SEC white paper on disaster recovery. Industry groups and lobbyists moved swiftly to deflect any effort to fix a minimum distance between primary and secondary data centers – which would have made many New York-area backup facilities obsolete, stranding millions of dollars in IT infrastructure.

Instead, Wall Street firms became motivated buyers of surplus data centers from bankrupt telcos and web hosts. Chapter 11 filings by WorldCom, Exodus, Metromedia Fiber Network and Global Crossing flooded the market with surplus data centers and telecom assets. Financial firms that bought or leased data centers outside New York in the past two years include the Bank of New York, Wachovia, Deutsche Bank, MBNA Corp., New York Life, MasterCard and Goldman Sachs. Dallas, Kansas City and St. Louis became the hottest markets for mission-critical facilities.

The investment goes beyond data centers. The New York Stock Exchange, which was among the buildings mentioned in the August terror alert, said this week that it has spent more than $100 million to bolster physical and cyber security and improve business continuity. That investment includes the Secure Financial Transaction Infrastructure (SFTI), a highly-redundant private network built atop self-healing fiber-optic rings. More than 600 financial firms are connected to the network, which can be accessed through multiple data centers throughout the country.

One challenge was real-time mirroring technology, which historically has limited the distance between primary and secondary data centers to about 60 miles. Software advances have helped overcome these distance limitations, improving the speed of mirroring technology and recovery times.

While concerns about disaster recovery have boosted activity in the US data center market, some Wall Street firms appear to be looking abroad to outsource their backup facilities. In its final white paper, regulators noted that several firms “indicate that they are exploring overseas locations as part of their recovery and resumption solutions, and ask for some assurances that domestic and foreign financial authorities will permit such arrangements,” the white paper said.