Safeguarding critical applications at Greenply

Greenply Industries’ comprehensive DR and BCP helps secure its critical ERP application along with other digital assets, says Abhinav Singh

For Greenply Industries, a company that manufactures decorative veneers and plywood, applications and customers are its two biggest assets. The company’s operations have expanded to remote locations such as Nagaland. Since Greenply has deployed a standardized SAP ERP system to run its transactional systems, it has become vital for the company to protect SAP data from any downtime, as it would mean incurring a huge loss of revenue and market opportunity, and worse, losing business to competitors.

Additionally, the company also has applications such as centralized messaging, Internet access through the Firewall and Web-based SAP for remote access. All of these are now clustered with online backup and automatic failover.

The need for DR

Once the company went ahead with SAP ECC 6.0, it was completely dependent upon its primary SAP server (SAP runs on IBM p520 servers at Greenply and it uses a DB2 database on AIX). The ERP system has been used to integrate all critical business functions. There have been numerous instances that have compelled the company to look towards deploying a comprehensive disaster recover (DR) and business continuity (BCP) solution. Dabinder Singh, Head, IT, Greenply Industries, explained, “We experienced unexpected problems due to abnormal shutdown of the server, which was because of a simple power failure. The database server went down for some time at the month end when dispatches were at their peak, and there was tremendous pressure from end users and top management to meet their targets. We started the server again, but unfortunately some unforeseen circumstances prevented the database from running.

“After persistent efforts, the database was up after five hours. However, downtime of the database meant a substantial lose of time and money for the organization. Since it happened soon after the implementation of SAP, we convinced the management to deploy a disaster recovery solution for business continuity which was crucial for our operations.”

The company also needed DR because its data center was located at a place where servers were left unattended at night. The data was primarily compiled from the corporate office at Delhi. The plywood industry being part of the unorganized sector, any delay in delivering material to a customer results in a loss of market share. Since the company handled premium products, to keep the market share in focus, it became mandatory for Greenply to have a DR system and BCP in place. Singh said, “One cannot think of DR and BCP in isolation; they have to run simultaneously. If an organization is having multiple points of sale and all of its locations are connected through VPNs, it becomes mandatory to put in serious thought. In an age of competition where customers do not have time to wait for material, the organization stands to lose face in terms of image and revenue, in the case of a disaster. Here, the image and brand loss is more significant than the monetary loss. Money can be recouped in a short timeframe, but not the brand and image.”

Different manufacturing plants and operations of Greenply were using their own anti-virus solutions and there was no foolproof data backup system in place. There were many unwanted outages due to unavailability of proper anti-virus and backup systems. Hard disk crashes and failures, massive data losses and virus-infected systems plagued the company. Yet another challenge was to move control of all anti-virus systems and their regular critical patch management as well as data backup of the ERP system to the head office in Delhi. Hence, a foolproof backup of critical data generated by application users across the company’s locations using a standardized solution was the only way to prevent the company from suffering major business losses.

Singh said, “SAP does not permit you to deal with unstructured business and it requires an adequate amount of discipline in the business process. Our business is completely governed by a disciplined business process in SAP, and hence we had a pressing need to protect the data through a proper DR system.” The Recovery Time Objective (RTO) of the company was minimal. It was decided that the backup and recovery should take place at the rate of 4 GB per minute; this requirement was met by the HP Ultrium 3. The tape library was incorporated as a medium for backup and recovery.

The company made a detailed study of different hardware and software architectures and found that a combination of IBM hardware and CA Xosoft best met its requirements and was in line with its budget. Singh said, “We evaluated CA and found it to be a stable and cheap alternative. The main challenge for us was the DB2 database, which SAP was using. We took support from external resources on DB2 to start working on DR and BCP.” Initially, the company gave a different host name for the DR server. However, due to low or negligible maintenance in a similar host name, the company opted for using the same host name with a different IP and started replicating the database through the file system. The evaluation of the DR solutions took the company around two weeks time and the complete project cost came to around Rs. 40 lakhs.

The DR setup

Greenply chose two IBM p520 servers and a IBM System Storage DS4700. Earlier the company had used an IBM SCSI storage box, which was six times cheaper than the DS4700. Since storage plays a vital role in DR, the company did not compromise; storage capacity used today amounts to 2.5 TB; the storage box is presently attached to both servers that are clustered; the database runs on the primary server and the application on the secondary one. Greenply has opted for servers in a clustered mode to avoid downtime. Earlier, the company was using a SCSI storage box as DR insurance. It found that data replication was working out well but as soon as it switched over to a proper DR setup, performance took a dive and users were unable to do their routine work. The DR system was taking a long time to save a simple voucher. Consequently the company decided to replace its SCSI storage with the fiber channel DS 4700.

The company has deployed a Storage Area Network (SAN); it has created two LUNs (Logical Units) on the DS4700 Express SAN box which happens to contain 16 146 GB Fiber Channel hard disks. The two LUNs have been created to take care of data load balancing and minimize I/O contention. This kind of storage network topology is designed to minimize the risk of downtime while maximizing performance. Since SAP is resource-hungry software, the utmost care has to be taken while designing the storage box. Additional 4 Gbps SAN switches and host bus adapter (HBA) have also been deployed. The DR deployment covered multiple aspects such as choosing the right software for backup and recovery at a high speed, to the creation and implementation of a DR site. Greenply has chosen CA Xosoft for backup and recovery, and also for data replication between the primary and the DR site.

During the setting up of the DR site, the company decided to also set up a virtual DR system at the primary site to minimize the chances of switching over to the DR site in the eventuality of a disaster. Greenply planned things in such a way that the primary server was clustered and multiple application servers were added, so that in the case of the database server going down, the cluster application server could be replicated as the database server. The company wanted complete backups of the database and the ERP application on a daily basis. An online backup is conducted daily and an offline backup is done weekly. It is mandatory to take the backup of the root folders where the OS resides on a fortnightly basis. As a thumb rule, the DR site hardware configuration has to be equal to the configuration of the primary site servers.

The company used to replicate data from primary storage to DR at an interval of 15 minutes, which meant that if something went wrong with the primary server, the company will lose only the data of the last 15 minutes and within no time the users can switch over to the DR system. It was decided to keep the host id common to both the locations—primary and DR; the IP addresses being different.

Protection against disasters

The company has deployed HA storage to protect against any hardware failure. It was also decided to cluster the hardware, controller, HBA, storage box, SAN switches and UPS. Singh explained, “Clustering is the second layer of security. Performance is also an important factor keeping in mind the faster execution of data and MIS. DR itself is a layer of security as it came into action when everything had come to a standstill at the primary site. It was also factored in that having multiple layers of DR is equivalent to having multiple layers of security and that the primary DR site should be equipped with an adequate fire control system.

Improving BCP with virtualization

The company decided to go ahead with virtualization to improve business continuity and eliminate downtime due to failure of the storage array. Normally, DR comes into the picture only when the primary site is completely down. Interestingly, the company found that if only the storage goes down at the primary site there is no need to switch over to the DR site. Singh said, “Instead of moving to DR, we may use virtual storage at the primary site itself, saving time and increasing BCP. In virtualization, an additional storage box is attached to the existing storage box of the same configuration, at the primary site. The data is replicated in real time using the virtualization box, which is a transporter between the primary and secondary storage. Post the DR implementation, things are better as the DR system is up and running across two locations and all redundancies are in place to keep the company’s vital business seamlessly functioning.”

abhinav.singh@expressindia.com