HPC

Tux fuels the HPC market

Penguin-powered servers are increasingly being deployed for HPC and driving this trend are oil & gas, government funded research labs, EDA and pharmaceuticals says Nivedan Prakash

Large government funded research work in seismic and weather forecasting are setting benchmarks in HPC and vendors are using these benchmarks to their advantage. Broadly there are two things worth mentioning here—that they are using Linux servers in clustered mode in HPC and second is that open source applications software used in oil & gas exploration, research (seismic data and weather forecasting) and EDA are additional drivers for the HPC market in India. The open source applications have really taken off with the advent of more choices in the OS—variants of Linux (Red Hat, SUSE) and OpenSolaris. With the availability of these OSs a lot of applications have been developed in-house or made available on the Internet for HPC customers to use.

Today, a majority of applications are available on these open source OS for HPC. So much so that ISVs who are specializing in HPC have shifted from proprietary OS towards open source ones. This would be incomplete if the commodity computing components were not touched upon. The capability of commodity x86 processors to deliver substantial compute performance has also been a catalyst for this growth and acceptance of open source for HPC requirements.

Justifying the role of open source in HPC, Dr P K Mishra, Consulting Architect, Novell India, said, “In the late 1990s the HPC environment was crowded by expensive proprietary hardware and there were software components in almost every HPC deployment. Every year, since then, Linux, open source compilers and tools have been increasing their share in this space due to their commodity nature, lower cost factor and reasonable performance.

“Linux as server OS technology fits in very well with the HPC segment. Overwhelmingly the majority of today’s HPC clusters are based on Linux. Some of the key benefits which are driving this adoption are high performance, scalability, flexibility, affordability and of course, an increasing ecosystem of hardware and software vendors supporting Linux for the HPC market,” explained Nandu Pradhan, President and Managing Director, Red Hat India.

On the other hand, Vivek Mansingh, Country Manager, Dell India R&D Center, is of the view that the role of open source software in the HPC segment can be analyzed by understanding users and their needs. Large corporations are usually willing to pay to get support for mission-critical applications. As the sizes of applications grow so do their complexities.

The market for HPC applications is still in a nascent stage and there are not many products in this space for parallel computing. This, in turn, has lead to the lack of commercial products for those who need them. Most parallel computing customers tend to be universities or national labs and they can effectively work together to develop software for e.g. Open MPI. The potential for open source’s success in a HPC environment is also due to the fact that many of the users in this area already know how to program. Therefore, development and testing along with effective sharing is easy. It is, therefore, understood that until demand for HPC software becomes widespread, most of the parallel computing software may remain open source.

Viswanath Ramaswamy, Project Lead, STG, IBM India/South Asia, however, believed that open source software plays a big role in HPC only as far as small clusters are concerned "If we talk about the large applications, then it does not have that big a role to play because it is the known players who provide the compilers and work on the file systems that customers use."

OSS vs. proprietary software

Linux has dominated the HPC segment but there are some niche special purpose applications, which may require a proprietary OS. Research communities and national labs have typically preferred open source rather than using their limited funds to purchase proprietary OSs and associated support. Hence over decades, these communities have learned to live with any shortcomings that may exist in open source. As the proprietary OSs found various ways to reduce the costs of license, deployment and support, there is a tendency to continue using them. The newer business models such as SaaS and virtualization-based clusters may also make a difference in the HPC environment.

Karthik Ramarao, Director- Technologies, Systems Practice, Sun Microsystems India, commented, “While open source OS is an accepted platform there are several specialized applications in a wide variety of both traditional, such as oil and gas or CAD/CAM/CAE, and new markets like rendering and animation where proprietary software is still preferred for the reasons of features, performance and support.”

It is said that for lower TCO, you need architecture that supports open source. This has become the principal driver behind Linux’s penetration into the HPC space. However, in the past, customers relied on a costly UNIX infrastructure that met their application requirements for HPC, but were far too expensive. Today Linux platform from vendors such as Red Hat are preferred for HPC as they meets the application requirements as well as the performance and scalability benchmarks.

Pradhan added, “Red Hat does not charge for software licenses. Our customers buy subscriptions, which provide them updates and support. So today customers are deploying Red Hat Enterprise Linux and Red Hat GFS with industry-standard server hardware and storage area networking. This has allowed them to reduce their infrastructure costs significantly while increasing throughput. They have been able to deploy an easy-to-manage scalable cluster solution to replace their existing proprietary infrastructure.”

The overall TCO consists of many components—the cost of acquisition of hardware and software, installation, managing the same, cost of down time etc. For an organization to lower TCO it depends on which of these costs are relevant. For example, the cost of downtime may not be a key component for a particular organization. For such an organization, it may be just the cost of hardware and software and a bit on management. So Linux or any other open source software may be the best solution here. However, there could be an organization which needs 24x7 support. There the dynamics could be different.

“Cost is just one of the factors. Openness, simplicity, vendor-independence and above all the proven track record are the reason for success of Linux in a HPC environment,” said Mishra.

If we talk about the Linux servers, it seems that they are pushing into the Windows server space in the HPC market as well. It is even seen that companies that were running their clusters on a Windows platform are now using Linux. Ramaswamy explained, “Linux is more approachable from the HPC point of view and it is a more robust operating system at the moment, and a lot of applications running on HPC have been developed on Linux. Windows has entered the market very late and though the Windows cluster servers are just coming into this pace they have not embraced the HPC market in a big way. Additionally Linux is more cost-effective.”

Linux in HPC is gaining ground at the cost of branded UNIX too. A few years back, UNIX variants such as AIX, HP-UX, Tru64 UNIX, Solaris, Digital UNIX, Irix, etc., ruled. Linux is now displacing a majority of them in HPC space. Microsoft never had a great position in this space. In recent years, Microsoft is pushing Windows based HPC solutions harder than before. It is pushing into the HPC space through its CCS edition of Windows server. Even the large business and research agencies are using Linux for their HPC requirement and it is simply because Linux on a cluster of servers (based on x86) is a lot more economical. This cluster has also become easy to set-up, simple to manage and more importantly there are a lot of resources available—many of them free of cost.

“The psychology of customers of these large business establishments and research agencies is completely different. They generally require a large amount of computing source at a low cost and hence open source fits the bill,” explained Ramaswamy.

Driven by medium businesses

While it has been seen that HPC had been largely limited to enterprises, R&D firms, and academic institutions, there is also a broad swath of mid-market companies adopting HPC due to the availability of economical solutions. The solutions, which were previously costly and proprietary in the past, have become affordable and open.

In the mid-market, pharmaceutical and biotech firms are hot markets for HPC Linux clusters. Recent work on the human genome has triggered growth in Genomics and Bioinformatics. Bioinformatics research requires HPC and open source applications are utilized in the core of this kind of research. HPC is also growing in the animation industry for computing, visualization and storage.

In India the IT and the ITeS have been leaders in such adoption whether for structural analysis, crash simulation or financial analysis. Even small life sciences companies that want to understand protein-folding characteristics have been adopting HPC. Apart from traditional applications used by customers of technical background, there has been a traction of HPC adoption amongst commercial customers for applications such as financial analysis and portfolio management, digital security and surveillance as well as decision support computing.

Now even smaller organizations with small budgets that need HPC are in a position to afford it using commodity-class technology. Ramaswamy explained, “Primarily if we look at HPC, to a very large extent in Asia, it is currently restricted to huge organizations in the oil and gas segment, financial services segment, government establishments and academic institutions. Today we are also seeing a lot of smaller animation studios, small time engineering colleges and computer labs using HPC.”

In addition to the R&D firms and educational institutes, manufacturing companies have used HPC clusters for large simulations. Oil and gas companies are doing reservoir simulations and in the past have used proprietary SMP systems primarily, but are starting to use Linux clusters.

Vendors optimistic

“While HPC started in the research organizations, it is now being used by many mid-market companies such as banks, investment firms, in the video and animation field to ensure that they are able to do crunch their computing workloads quickly and at a much lower cost of ownership from a capex and opex perspective,” commented Pradhan. HPC clusters are playing an increasingly significant role in IT. As organizations depend on these clusters more, there is increasing value in a consistent environment across the data center. “Red Hat has been committed to developing open source technology for more integrated HPC cluster deployments. We are also actively building a strong portfolio of certified software applications from the leading vendors that deliver applications for HPC workloads,” added Pradhan.

Seeing this kind demand for HPC in the mid-market segment, HP is also offering HPC platforms designed to help mid-sized companies to effectively take advantage of the technologies that have transformed engineering and research at some of the world’s largest enterprises. One such platform is HP Cluster Platform Workgroup System, which deals in HPC applications in CAE, oil and gas, financial services, and life and material sciences environments. The HP CP Workgroup System enables customers to run larger and more difficult compute-intensive workloads faster—providing seven times as much simulation performance as a single-node system.

“HP is extending the power of high-performance computing to mid-sized businesses by delivering easy-to-use and affordable solutions that allow customers to stay ahead of the competition,” said Faisal Paul, Country Manager, High Performance Computing and Linux Business, HP India. Further expanding its mid-sized business technology offerings, HP also introduced mid-sized Solution Blocks for material science and CAE, and a high-density x86 server ideal for HPC environments.

Given the above developments the growth of HPC (both x86 and RISC) in the oil & gas, EDA, research labs, pharma and mid-sized companies will continue to drive deployments in India in 2008 and open source will play an increasingly large role.

nivedan.prakash@expressindia.com