Profile of Brian Clark of the New York Stock Exchange, for CIO Connect magazine, summer 2008 issue. By Sue Tabbitt

* ALL FEATURES ARE COPYRIGHT PROTECTED AND BELONG TO THE MAGAZINE THAT COMMISSIONED THE WORK. UNDER NO CIRCUMSTANCES MUST THIS CONTENT BE USED ELSEWHERE BY ANY OTHER PARTY.

Not being afraid to admit when something isn't working is one of the hallmarks of a good leader. And so it was when Brian Clark, chief software architect at the New York Stock Exchange, and his team were faced with the hinderingly slow performance of a data management system they'd implemented just a couple of years earlier.

Unexpected acceleration in the growth of the volumes of data it had to handle meant the original Oracle 10g system, which was being used as the data tier of the NYSE's trading platform, was struggling to cope. "It could take 12 hours to load, and 24 hours to complete queries, if at all," Clark recalls.

The story began in 2004 when the NYSE's CIO mandated the move to a shared database, to reduce data duplication and improve performance against all of its trading activities. Today, the data management system handles 140Tb of production data, serving two different environments: the NYSE trading floor, and Arca Equities and Arca Options.

By the middle of 2006, NYSE was experiencing a 'very large data management problem'. The merger of NYSE Group with Euronext NV meant a surge in volumes of data which was being managed across hybrid platforms, and the existing system couldn't cope.

Arca Equities is growing at a rate of 100% a year, and Arca Options at 200%, he explains. "The old system just couldn't cope with this high growth. The problem was that if data was not readily available, this would affect our compliance with industry regulations."

There was no avoiding it; a huge data management project was needed. Clark and his team replaced Oracle with Netezza, a specialist data warehouse appliance that combines server, storage and database in a single unit, to enable speedier access to business-critical data, delivered in a more cost-effective way. Whereas previously it could take up to 26 hours to retrieve data, this now takes a matter of minutes. As well as ensuring compliance (NYSE must keep data for seven years, in a readily accessible form), NYSE, which collects, stores and manages trading data, is now able to sell more data.

Ensuring the new system was right for the job involved comprehensive pilots, initially based on 35 regular queries for the proof of concept, and then opening the test phase to more demanding users and ad-hoc users with more specialist queries.

By the time the case was made at CEO level, the arguments were watertight, easing the new system through the approvals process. "Previously, we had to keep buying servers and storage and increasing the number of DBAs to support the installation; the numbers were getting very large," Clark notes. "If we could manage the data intelligently, we knew we could save millions of dollars."

"Excellent internal and external support" resulted in a speedy implementation (just eight weeks). "Teamwork was key," Clark says. "There's nothing like a crisis and a burning platform to get people's attention."

Every team in the IT organisation was represented at weekly or twice-weekly meetings, while business users played a key role in testing the environments so the new system could be modified. "The unpredictable queries of ad-hoc users provided particularly good feedback," Clark notes. "After what happened with Oracle, we only had one chance to get this right, so it was imperative we involved everyone in defining our requirements."

This is the biggest lesson the team took away from the experience with Oracle, he says. "Then, we bypassed the hands-on proof-of-concept assessment between different database platforms. Even though we weren't dealing with the same volumes of data at the time, it would have helped us get a better feel for the potential challenges."

All's well that end's well, which must come as a relief given that, in capital cost terms, this is so far the biggest deployment of Clark's 25-year career.

NYSE Euronext, the holding company created by the combination of NYSE Group Inc and Euronext NV, was launched in April 2007. It operates the world's largest and most liquid exchange group, offering a highly diverse array of financial products and services. It brings together six cash equities exchanges in five countries and six derivatives exchanges, and is a world leader for listings, trading in cash equities, equity and interest rate derivatives, bonds and the distribution of market data.