|
Lead
Is x64 the answer for HPC?
High performance computing is becoming increasingly popular
amongst media and entertainment companies and their platform of choice is none
other than 64-bit x86, writes Varun Aggarwal
 Gone
are the days when Symmetric Multiprocessing (SMP) used to rule the high performance
computing (HPC) space. IBM, Sun Microsystems and HP all support clustered HPC
environments built around industry standard x64 (64-bit x86) gear. HPC grids
are popular in various industry verticals such as IT/ITES, manufacturing (crash
analysis, electronic design automation), BFSI (financial risk assessment), engineering
& scientific research and media & entertainment.
One of the biggest adopters of HPC is the media and entertainment industry;
for this vertical, HPC is the backbone of a core aspect of the business, viz,
rendering. Before going further, let us first understand how rendering works.
Rendering is the process of creating special effects laden images for a movie
or TV show and it is a compute-intensive task. Individual images are sequenced
together to form a scene. One second of a movie scene is made up of 25 high-resolution
images (frames). Depending upon the complexity of a scene, rendering a single
frame may take several hours.
Fortunately, this process can be parallelized. If you have sufficient resources,
each frame can be rendered by different computers. For example, if one has 100
Computers, four seconds or 100 frames of the scene can be rendered in the same
time that it takes to render one frame. Such an array of computers used for
rendering is known as a ‘Render Farm’. Managing a Render Farm is a
challenge, as one has to deal with multiple jobs (scenes) and frames being rendered
simultaneously. This is where HPC Grids come into the picture. HPC Grid software
(Sun Grid Engine, Microsoft WCCS or Microsoft HPC 2008) provides the infrastructure
for these activities.
Proprietary vs. Open source
|
 "M&E
companies have not adopted HPC primarily on account of its complexity
with regard to installing, configuring, integrating and deployment"
- Amit Srivastava
Chairman & CEO, DUX Soft
|
|
 "Linux
has come a long way and become a robust server OS that not only reduces
costs but also gives you a better opportunity to customize tools for specific
needs"
- Viswanath Ramaswamy
Country Manager - Projects, Systems & Technology Group, IBM India
South Asia
|
The fight for the top slot exists even in the HPC space, where
both Linux and Microsoft claim their superiority. Each of these platforms can
help companies in the media and entertainment (M&E) industry.
Microsoft has a HPC-specific server OS, Windows Compute Cluster
2003, which is based on Windows Server 2003. This 64-bit OS has been specifically
designed to cater to HPC requirements and is best suited for 64-bit applications.
Microsoft plans to launch Windows HPC server 2008 soon, which is currently in
its beta. Microsoft has tied up with various software services companies (DUX
Soft and Wipro are two) to offer tailor-made solutions for customers in the
M&E space.
On the other hand, IBM and Sun Microsystems are quite bullish
about open source software (read Linux and solutions running on the open source
OS). With plenty of choice on the OS front—variants of Linux (Red Hat,
SUSE) and OpenSolaris, a lot of applications have been developed in-house or
made available on the Internet for HPC customers to use. Viswanath Ramaswamy,
Country Manager - Projects, Systems & Technology Group, IBM India/South
Asia, commented, “Linux has come a long way and become a robust server
OS that not only reduces costs but also gives you a better opportunity to customize
tools for specific needs. There are loads of applications available on Linux
for HPC environments and many ISVs that specialize in HPC are shifting towards
open source, giving customers sufficient choice. Moreover, users are starting
to realize the power of commodity x86 servers in the HPC space.”
Customized solutions
M&E Companies require a lot of computing power for rendering. There are
quite a few render management tools in the market. The biggest drawback of these
tools is that they have all been designed from a work management perspective
and not from a compute load management perspective. This is where HPC Tools
have a part to play as these have been natively designed for optimizing computing
activity like number crunching, memory management and I/O (Input / Output of
data or instructions) management. These tools support the grouping of computer
resources on the basis of performance, configuration, availability and proximity
giving you better control of resources. They help track each activity on every
computer node in detail, giving a end-user complete control not only over work
management, but also over resource management.
For example, a highly compute intensive task such as the
final rendering for a movie format can be given to compute nodes that have the
latest, highest configuration while intermediate review work can be alloted
to older or comparatively lower-end machines in a render farm. It is this ability
of HPC tools that also allows one to salvage free CPU cycles from workstations
so that they double up as compute nodes for a render farm. Small and mid-sized
studios can save on hardware, infrastructure and software licenses in this manner.
| Maya Entertainment was struggling from exploding
data and computing requirements with the advent of large video formats like
High Definition (HD). Jai Natarajan, EVP, Business Head, Maya Entertainment,
felt that the increasing amount of data and the exponentially high computing
needs could only be addressed with high performance computing. Moreover,
the company was moving everything on a 64-bit platform and was thus looking
for a server class native 64-bit OS. The previous architecture was a 32-bit
cluster that was running desktop class windows and the grid had to be manually
managed.
While looking out for solutions, Natarajan was
introduced to Windows Compute Cluster 2003, a 64-bit server OS meant designed
specifically for HPC environments. Natarajan decided to do a Proof of
Concept and a pilot run with Microsoft for their new platform. R&D
was done to test all the software running at Maya and to ensure that everything
worked fine. Tests for performance and compatibility were done for about
four months during which bugs were found in old software, which were fixed
with updates from the respective vendors.
Challenges
As the deployment began there were different projects
running at Maya that were in different stages. Changing the platform at
such a stage could translate into failure for many projects that were
underway. “That’s the reason why we spent so much time on testing,”
commented Natarajan.
Another major challenge was that of change management.
Many employees of Maya Entertainment had apprehensions about the new platform
and resisted change. However, when tests confirmed a performance boost
of 15 to 20%, people were compelled to buy into the new platform.
The deployment was complete with render farms running
on blade servers powered by Intel dual core Xeon processors.
The way ahead
The work is not over yet for Natarajan. After the
successful implementation of HPC on a 64-bit computing platform, he wants
to squeeze out more performance from the system. In this endeavor, he
is using DUX Soft’s expertise as this was the company that helped
in the successful deployment of Maya’s 64-bit platform. Once optimum
performance is achieved from the existing platform, the company will look
at deploying quad core processors up from the current dual core processors.
|
Amit Srivastava, Chairman & CEO, DUX Soft, commented, “M&E companies
have not adopted HPC primarily on account of its complexity with regard to installing,
configuring, integrating and deployment. Moreover, finding the right talent
for setting up such systems, is also difficult. Our tool SPARX tries to bridge
this gap, allowing studios to concentrate on their core competencies i.e. developing
mind blowing animation, visual effect or games.”
What DUX Soft has done is to get SPARX to easily integrate with a studio’s
production pipeline. The pipeline here refers to the flow of data files from
one department to the next, during the process of creating an animation scene.
Rendering is the logical culmination of this flow where complex floating point
computations generate output image frames. Then again, studios may require HPC
resources at early stages for creating effects involving ‘Particle Dynamics’
namely waves, storms etc. Such computations are done at an early stage so as
to give animators a better perspective on how the remaining elements in a scene
should react. SPARX, customization for the M&E industry, ensures that HPC
resources are available at any stage of the production pipeline. “One customization
that we are currently working on is to store context information of HPC utilization
at each stage so that it can easily be available at later stages of the project.
This will help keep track of the exact HPC-related happenings (errors, corrections,
changes etc.) in a project,” further added.
Render Wranglers (folks who manage the Rendering activity) are generally not
techies. SPARX has been built keeping ease-of-use in mind. The learning curve
on a Render Wrangler’s part is minimal.
Sun’s M&E HPC solutions address the complete workflow requirements
in animation, gaming and post-production studios. They are easy-to-deploy, maximize
open-source technology and can be scaled on-demand to accommodate changing M&E
business requirements.
Karthik Ramarao, Director-Technology Team, Systems Practice, Sun Microsystems
India, added, “Depending on their workflow environment and on-going projects,
M&E users need to optimize resources at-hand. Sun’s HPC solutions contain
a customizable software stack. M&E users tailor these software solutions
to manage their workloads, provision hardware resources, repurpose render-farms,
manage storage capacities and so on. Sun’s M&E solutions offer complete
flexibility to users to adopt them in their studios.”
IBM started a separate business unit to cater to the M&E vertical. The company
offers a wide array of services focusing on HPC for this segment. Unlike some
other architectures, IBM provides all high performance computing at the server
end where all the rendering, compute work etc is done on blade servers. The
user has a choice of selecting the front-end OS, which could be anything from
Windows, to Mac OS or even Linux. The back-end is Linux. For this, IBM recommends
using thin clients that also bring down the cost of client machines.
While M&E companies enjoy the benefits of high performance computing, x64
is an ideal platform, with a plethora of services available on it that are not
only fast but also highly cost-effective. As far as others are concerned, they
too get to enjoy the high performance computing benefits by the means of high
quality animation and effects in their favorite movies, which are a result of
the use of HPC in media and entertainment industry.
varun.aggarwal@expressindia.com
|
