Back-up system is lifeline of NSE

The National Stock Exchange handles more data in a day than many organisations do in a year, so the back-up system must necessarily be as world-class as the exchange itself. Shipra Arora finds out how the NSE does it

SAN-based direct connect technology and the use of automated tape back-up has resulted in a more reliable back-up process for NSE, says g m shenoy

The National Stock Exchange (NSE) has a 60-65 percent share of the equity market and a 99 percent share of the derivatives market in terms of turnover. Almost 1.8 million transactions are conducted every day over 10,000 online trading terminals (at brokers’ offices) which are connected to the NSE from across 350 locations in the country. Its average daily turnover is Rs 8,000-9,000 crore.

Critical asset

According to G M Shenoy, vice-president of information technology at NSE, the most critical asset of the exchange is its transaction data. The total volume of data residing in NSE’s systems runs into 2-3 terabytes, and is expected to grow to 7.5 terabytes over the next four years.

Any downtime is unacceptable at NSE since the business lost due to data loss can be tremendous. Given the nature of its business, data protection and availability are crucial. A centralised automated tape back-up system (including online back-up of Oracle databases) has enabled the organisation to ensure data management, protection and availability. The back-up system of the exchange is a heterogeneous SAN-based LAN-free back-up solution provided by StorageTek.

Challenges

Till about three years ago NSE had a distributed back-up system in place using direct-attached tapes. The processes required 400 GB of data to be backed up daily and 900 GB at the weekend over multiple servers and across different platforms such as Unix, NT and Linux. However, the changing external market and internal growth dynamics warranted a centralised back-up solution. With the Indian economy growing, the data volumes within the exchange were also growing, leading to growth in data back-up needs as well.

Coping with a growing back-up need and a shrinking back-up window (time period within which data has to be backed up) was becoming increasingly difficult with a distributed system. Shenoy says that this brought in its wake many problems in terms of management and control. As a result, back-up was a tedious and laborious process requiring logs to be maintained. In addition to this, heavy manual intervention was required since the jobs were manually queued and cartridges manually mounted. Besides, the consumption of cartridges also increased significantly.

The time required for back-up activity also kept increasing as data increased, so there was concern that the NSE would not able to meet its deadlines in the future. The back-up time window available to NSE is between 9 pm and 7 am Shenoy points out that considering the rate at which the volume of data was growing, the exchange would not have been able to meet the 7 am deadline if it had continued with its earlier system. (Back-up activity has to wind up latest by 7 am and stop if it goes beyond this deadline.)

Way forward

These existing and looming requirements led NSE to realise that it was time to graduate to a more robust back-up solution. Centralisation was the way forward since it improves the utilisation of resources as well as makes control and monitoring easier.

NSE had two options. First, to go for back-up on the network, that is, LAN. Second, back-up using SCSI. Since it had a back-up window of just a few hours, neither of these two could meet NSE’s requirements. A LAN-based back-up would lead to network congestion, while connecting the tape libraries to the server using a SCSI cable would mean that storage resources could not be shared. In order to enable sharing of resources, and efficient transfers within the back-up window, NSE therefore opted for a SAN-based solution. With a SAN-based LAN-free solution, the SAN functions on its own communication path independent of the LAN, and the main network is also relieved of data transfer loads and congestion. "A separate network for back-up means that there are no chokes in the network; the back-up, being on a Fibre Channel, is also faster. This was required for meeting the back-up time window, which remained constant," explains Shenoy.

Solution

NSE considered various competitive options before zeroing in on StorageTek’s SAN-based LAN-Free solution. The evaluation before arriving at the decision was made on the basis of factors like speed, connectivity, back-up software, recovery and support. "Since these solutions are not plug-and-play, some amount of initial support is required," says Shenoy. Both StorageTek and its distributor, CMS Computers (which also provides support) jointly pitched the proposal, with back-up software from Veritas and BMC.

The solution implemented at NSE in May 2000 comprised a StorageTek L700 tape library (400 slots) with four DLT tape drives, Brocade switches, Storage Net Fibre Channel routers/bridges and back-up software. NSE’s servers are now connected to the Brocade switches using Fibre Channel. The tape drives are connected to the Brocade switches via bridges to implement a heterogeneous SAN. The SAN is specific only to the centralised tape solution for back-up purposes.

Veritas NetBackup Data Centre has been installed on the servers. The tape library is located at the data centre in Mumbai. At the end of the day, the back-up is taken on the tape library. The solution allows the pool of tape drives to be shared among multiple connected servers, hence there is no need to back-up the servers individually.

Early challenges

NSE did face some teething problems with its new back-up system during the implementation stage as well during as the early stages of post-implementation. A few of the servers gave problems, and some software patching had to be done. During the early stages a lot of testing of procedures was carried out.. It took almost six months for the system to stabilise. However, once the back-up system steadied, NSE was comfortable with it.

System benefits

SAN-based direct connect technology and the use of automated tape back-up has resulted in a more efficient, consistent, disciplined and reliable back-up process for NSE even while its data size is growing and the back-up window shrinking. Today, even with 600 GB of data back-up per day, NSE is easily able to meet the 7 am deadline. Since the entire process is now automated with no manual intervention required, NSE can back-up in half the time, and, according to Shenoy, the manpower requirement has not increased though data volumes and applications have.

Centralisation of the process has made control and monitoring tasks easier so that back-ups made are more reliable. Today the entire back-up happens in the Mumbai office, as opposed to back-ups being taken at multiple locations. Availability of a single management console for heterogeneous platforms provides for seamless and more efficient management. Restoration is easier, more systematic and faster. It also provides the scalability needed to increase storage capacity as NSE’s requirements grow.

In terms of financial benefits, the solution has reduced the number of cartridges consumed daily. Having a SAN-based solution has helped NSE avoid issues like LAN network bandwidth bottlenecks and collision of data packets. With all these benefits, NSE is able to meet the demand for high availability and business continuity.

Roadmap

NSE is now in the process of implementing an enterprise SAN solution from HP for its growing storage needs. This will require augmentation of its existing tape library. The exchange will be migrating from DLTs to SDLTs (Super DLT) for the SAN.