CONTENTS

Cloud Based Test & Development

Software testing using the Cloud is a multibillion dollar opportunity for testing companies. However, the issues here include diagnostics, interoperability, and security as well as the age old issue of having an engineer on site when things go wrong and these continue to dog organizations preventing them from moving this activity completely into the Cloud. By Venkatesh Ganesh

Software testing in a client organization has traditionally been managed at the project level and there has been a lack of tight processes, methodology, tools and reporting. Almost all testing has been performed on-site, making it a costly affair. Today, with mobility and the Cloud being the buzzwords of the day, CXOs have more choices.

According to CMU, 50% of the software development costs in an organization go towards testing. Research firm NelsonHall has estimated that the software testing market would reach $37.6 billion by 2012.

Chicken and egg scenario

The Cloud is starting to have an impact on software testing practices. Organizations are figuring out that it can help in this area particularly when they go in for a process-driven approach that enables stringent process monitoring.

“Corporations can proactively manage the testing process,” said Michael Azoff, Principal Analyst, Ovum. Typically, testing work is outsourced to independent testers or, in many cases, work is done out of the client’s premises using its systems.

The economic slowdown in 2008-09 coupled with increasing competition made companies look at different ways of reducing software development costs. The financial sector, which was the foremost when it came to software testing, has been under severe pressure to cut costs. According to Forrester, on an average, 27% of the spending done by companies happens to be on custom software development. The likes of Pegasystems are encouraging IT organizations to build and test software on the Cloud by addressing connectivity issues and enabling shared infrastructure that’s accessible from multiple locations.

The Cloud environment forces you to test non functional parameters such as availability, scalability and performance by equating their importance with functionality testing. In the traditional mode, dedicated hardware, tools, software and human resources are used to carry out testing. In a standalone environment, these would typically have a lower priority, felt Kumudha Sridharan, VP Testing Services, Wipro Technologies.

Earlier, applications were mostly tested for functionality and other parameters were not given the same importance. The Cloud, however, offers infrastructure on demand or on an ‘as needed’ basis. The fact is that you do not need regular access to software testing infrastructure as it is required only when new software is developed or old stuff is modified.

“There are times when a dedicated test environment may sit idle and there are times when it would be ideal to have multiple test environments for running different types of tests,” commented Sumanth Tarigopula, Director – India Center Best Shore Application Services, HP.

The Cloud gives you access to infrastructure whereby virtual images that reflect your test environment can be provisioned in minutes, used for testing and then decommissioned once you are done with the virtual infrastructure. Companies only pay for the resources that are used and, therefore, there is no need for tying up CAPEX in testing infrastructure. Cloud-based infrastructure is commonly used to deliver application development and application test platforms as a service.

The Cloud can remove uncertainty to a certain extent depending upon whether or not a company’s application can scale from a testing perspective, said Paddy Padmanabhan, MD and CEO, Appnomic Systems.

Another area where the Cloud comes in handy is with regards to load distribution, which, according to most testing companies is similar to the real world scenario. Such virtualized services can be provisioned from anywhere in the world and easily moved, unlike dedicated infrastructure, which must be physically shifted, averred Tarigopula.

"The high upfront CAPEX has been a primary consideration. In this environment, companies are desperately looking to cut costs." Vishnu Bhat VP & Global Head – Cloud, Infosys

Sensing this new paradigm, Infosys has set up a new line of business consisting of developers, testers and consultants that specifically work on Cloud-related services. Already, the company has started testing for a railroad company based out of North America, said Vishnu Bhat, Vice President and Global Head – Cloud, Infosys.

“Some of a client’s primary considerations have been the high upfront CAPEX and, in this environment, companies are desperately looking to cut costs,” reasoned Bhat.

Similarly, in December 2010, Smartesting, a provider for model-based testing announced that global supply chain optimization solution provider, Manhattan Associates Inc., would deploy its solution in its Quality Assurance (QA) organization across India and the US with a view to boost the value delivered to its clients. The first phase of the collaboration enabled Manhattan to verify the scalability of the solution in areas such as collaborative, incremental work and integration with the existing processes that were in place. Manhattan reduced test cycle time by 35% and raised the level of automated execution above 80%. Additionally, with the Smartesting model-based approach, Manhattan was able to get a quick alignment of the test assets developed by the QA organization with the business needs expressed by the operational teams, despite a gap of two continents and ten time zones, said Michel Guez, MD, Smartesting India.

":Predicting application behavior in various environments such as social or mobile can be done better on the Cloud. Demand for Cloud testing is growing 20% every quarter." Vineet Arora Mng. Director, Aditi Technologies

The concept is catching on. “Demand from our clients for testing in the Cloud is growing 20% every quarter,” said Vineet Arora, MD, Aditi Technologies.

Companies like iViz said that although adoption was slow to begin with, they were seeing a surge of late.

Cloud testing allows dynamic server applications such as Web sites powered by .NET, J2EE or LAMP to be quickly scaled up. This has had a considerable impact for one of Infosys’ Japanese clients that ran a testing application (for a financial software algorithm) that used to take six-seven hours, which has been brought down to a minute.

This brings to the fore another trend in the industry, which is of Cloud platforms being used beyond server-based applications. Any testing services can be performed using a Cloud platform that uses the processing power of the Cloud to simulate anywhere from dozens to hundreds of thousands of users per hour, stated Govindarajan Muthukrishnan, Testing Practice Leader, Capgemini India.

From the perspective of building a software testing lab (which is the bread and butter of independent software testers), the Cloud offers some advantages. According to Anand Bagmar, Lead Consultant – Quality Analyst and Sharath Kumar, Lead Consultant, ThoughtWorks, there are usually many combinations of software, OS and hardware resources. The Cloud helps tailor these combinations for setting up test labs.

As companies are moving towards Cloud-based solutions, it is vital that they test the applications in their end state environments. With different environments cropping up such as mobile phones and tablets, analysts pointed out that it was feasible to vary the environment and the resources in order to enable different types of performance testing.

Arora of Aditi Technologies agreed with this contention adding that predicting the behavior of an application in different kinds of environments such as social or mobile could be done better on the Cloud.

The usage of dedicated hardware and software, upgrades and maintenance of hardware all add to the CAPEX. “The human resource pool to carry out testing and maintain the environment is also tied up to a particular client. This reduces agility and makes responding to changing business needs a sluggish affair,” pointed out Dr. Neelesh Kumbhojkar, Head of Performance Engineering CoE, Tech Mahindra.

Apart from cost effectiveness and a reduction in product life cycle, development teams want more options when it comes to cost reduction and the optimal use of compute power. The advantages of sharing the talent pool across the global development centers become more prominent with Cloud-based testing.

“Customers need to buy tools or go to consultants; in the alternative, they can simply go online and get their applications tested. What Salesforce.com did for CRM, we are trying to do for testing,” said Bikash Barai, CEO and Co founder, iViZ Security.

"Generating loads of virtual users on Cloud VMs lets you address the temporary increase in hardware and network bandwidth required for large performance tests." Ashwin Palaparthi VP - Innovation, AppLabs

Companies can also have a scalable pay-per-use model of generating loads of virtual users from several different machines running in the Cloud. This addresses the temporary increase in hardware and network bandwidth that is required for a large performance test, averred Ashwin Palaparthi, Vice President, Innovation, AppLabs.

This can prove particularly handy when an online retail venture is in the design phase. “If you need to invest in four servers with load balancing for testing a new Web site and only use this infrastructure for the final round of testing in the last couple of weeks before release, investing on infrastructure ends up being wasteful,” said Nitin Vyakaranam, AVP Global Marketing and Alliances, Prithvi Information Solutions.

The Cloud gives you access to infrastructure whereby virtual images that reflect your test environment can be provisioned in minutes, tested and completed.

Testing on mobile devices

The workforce in most enterprises is making greater use of mobile devices for accessing enterprise applications, at least messaging to begin with. Testing applications for use on devices such as tablets and smartphones is starting to gain momentum. Enterprise applications need to be tested across myriad devices for compatibility and security.

The Cloud has a role to play in mobile application testing and, for this, HP has partnered with solution providers like Perfecto Mobile to offer a range of devices for testing accessibility through the Cloud. Perfecto makes these devices available through the Cloud and through software tools that integrate with the HP Quality Center allowing for application testing on multiple devices.

Diverse handsets, platforms, form factors, roaming, operator, device configuration, memory size, etc. all contribute to the complexity. Moreover, each mobile OS is different. “This makes it difficult for developers and testers to maintain a decent level of performance for the end application,” said Suresh Samuel Jayaseelan, Director, Testing Services, CSC India.

“One can test out a particular device for a particular browser and a particular application,” said Moorthy Uppaluri, GM, Developer and Platform Evangelism, Microsoft India.

There are frameworks like WebDriver that attempt to provide common APIs to enable ease in testing on these platforms. Nevertheless, until these frameworks become more stable and rich in the functionality that they offer, this will continue to be a specialized domain.

"Test harnesses and automation frameworks can access applications to setup and execute distributed test cases enabling diagnostics to trace execution." Ramesh Loganathan Mng. Director & VP - Products, Progress Software India

Other industry watchers felt that all of this complexity was a part of the distributed application landscape. “One will need test harnesses and automation frameworks that can access multiple applications to setup and execute a distributed test case and enable the necessary diagnostics to trace the execution of the test,” explained Ramesh Loganathan, Managing Director and Vice President, Products, Progress Software India.

As devices, platforms and technologies keep changing, it brings about a host of challenges and testers are learning this the hard way.

Weak points to be addressed

While the companies that Express Computer spoke to were gung-ho about Cloud-based testing, the question remains whether or not the detection of software bugs is better while using a Cloud environment. The answer remains unclear as Cloud testing hasn’t quite matured yet.

There are two key areas wherein the Cloud can address the weak points in software applications namely minimizing configuration errors and in running meaningful performance load tests. This rationale stems from the fact that, in a traditional test and development environment, different departments running, say, middleware software, could be using older versions of the software. This could result in configuration errors in test or production environments.

These issues are better addressed by mimicking volume peaks, scalability and drilling down to the root cause as this allows developers and testers to interact with each other.

Another process that is a part of testing is to run meaningful performance load tests. A real time test analytics dashboard provides for analyzing the results in a live environment as the test happens.

The real-time analytics engine could provide actionable information as the test is running, regardless of its size, felt Jayaseelan. The dashboard could be customized to provide insight into the infrastructure that had typically been available only after sifting through log files after a test was completed and a dashboard could be customized to view the average response time, error analysis, error rate, send rate etc.

The test environment is traditionally a smaller environment than the production one, so the results of a performance load test need to be extrapolated in order to determine performance failure points in production. Systems do not necessarily scale linearly, so this approach is often flawed. By using the Cloud, we can ensure that the test environment is identical to the production environment so that any performance load test provides meaningful results for the production environment without having to extrapolate.

Security concerns

The principal difference between a conventional test and development environment and doing it in the Cloud is that of multi-tenancy and data storage.

“The client’s Cloud security posture will be characterized based on the completeness of the risk-adjusted security controls that have been implemented, pointed out Kumbhojkar. These controls are implemented ranging from physical to network to application security.

In addition, controls need to be implemented at the people and process levels. These include the separation of duties and change management, virtualization security etc. That’s the inherent risk.

Azoff felt that, as confidence in the Cloud continued to grow, mission-critical apps would end up being tested on the Cloud. The fundamentals of modern day business applications are functional quality, performance and security.

Companies have to bear in mind that improper usage of Cloud-based test environments can actually result in higher costs. Although some vendors have offered pay-as-you-go Cloud-based testing services, this approach could work out to be expensive or out of sync with requirements, particularly if user estimates are too conservative or wildly overblown. Companies that have applied pay-as-you-go approaches have had to perfect their cost models or apply process-driven estimates rather than utilizing projections that are unsupported by data.

As the Cloud gathers steam, procedures are being developed to improve security and performance in the public Cloud. For instance, service providers like Capgemini are developing virtual private Clouds and client partitions.

Another important aspect that tends to be overlooked is that of performance. “As public Clouds are shared by numerous users, there may be cases where a company might have to wait for the required bandwidth, taking you back to square one,” said Muthukrishnan.

Uptime is an important consideration when testing on the Cloud and clients insist on five or six nines uptime depending upon the applications that are being tested.

Enterprises are apprehensive about moving their applications or infrastructure into the Cloud. Without a trusted environment, it will be difficult to win CIO mindshare for Cloud adoption.

Non-profit organizations like Cloud Security Alliance (CSA) are promoting the use of best practices for securing the Cloud. Cloud providers should adopt these best practices to provide a secured Cloud environment. Even testing houses need to follow these guidelines while testing customer infrastructure/applications.

Moving testing to the Cloud should be viewed as a strategic initiative rather than as a tactical objective. To gain confidence, providers should first create a proof of concept. Organizations that do not have sufficient testers can seek on-demand testing services, wherein service providers take complete responsibility for testing in the Cloud.

Interoperability issues

Interoperability is an issue that most vendors are reluctant to address. With a lot of existing investments in proprietary systems that, in a lot of cases, are unable to talk to each other, all these grandiose Cloud plans could fall flat.

Tarigopula said that there were two aspects to this. Firstly, interoperability would become an increasing need with mobility solutions as employees began to choose their own devices and organizations had to keep pace with that in order to minimize the incidence of employees owning multiple devices while delivering a secure, consistent service. Secondly, the industry would see the evolution of open standards, which as service providers and application providers started implementing and evolving, would reduce the need for significant interoperability testing.

Also, many enterprises wanted to mix and match their Cloud requirement and getting different kinds of services from Cloud vendors was not a problem as long as you did not want them to interact with each other. However, if these services had to interact, then you ended up dealing with interoperability issues.

Bagmar and Kumar pointed out that, in terms of interoperability, Cloud services gave greater flexibility around software and hardware.

Like the maturing telecommunications industry, all the stakeholders in the Cloud arena have to come up with interoperability guidelines. Cloud vendors have formed the Cloud Computing Interoperability Forum (CCIF), which aims to create a Unified Cloud interface. The purpose of this forum is to create common interfaces for interaction between remote platforms, networks, systems, applications, services, identity and data. All of the vendors that are members have to adhere to the guidelines and follow the standards and compliance requirements before they go live.

“Interoperability is our selling proposition and, whether you are testing on Python or .NET, the same level of importance is given and unlike the perceptions that are going around, we do not afford second class treatment to non .NET platforms,” said Uppaluri.

This raises the question of SLAs in the Cloud. Terms and conditions of Cloud services are sometimes difficult to understand and misleading in areas such as governance, data integrity, data preservation, data location and transfer. Some Cloud infrastructure providers offer only limited types of configurations, technology, servers and storage, networking and bandwidth, making it difficult to create real-time test environments.

According to Peter Coffee, VP and Head of Platform Research, Salesforce.com, when any service provider makes a claim of four or five nines uptime, it’s vital to check the fine print that defines the denominator of the fraction.

Some software testing companies that undertook outsourced work were confronted with issues surrounding legacy systems testing on the Cloud.

“Connecting to legacy systems from the Cloud has not been addressed properly as of yet,” averred Palaparthi. As long as Cloud apps use industry standards like Web services and XML, they address the interoperability issues by design.

One of the biggest issues that Cloud operators are trying to tackle is that of providing the ability to integrate with existing systems and companies are working to accelerate this integration.

Diagnostics aren’t quite there

For a solution that is merely hosted on the Cloud, the diagnostics will be as good or bad as those on any other environment. These platforms are similar to other regular servers and can be diagnosed using conventional utilities. “However, if the application is a distributed one spanning multiple servers on the Cloud or accessing multiple applications through Web services, then the diagnostics can be a little tricky,” cautioned Loganathan. This is especially the case when tracing specific flows through multiple applications.

To this end, HP has established a service called Cloud Assure that leverages the vendor’s tools and expertise for monitoring applications running in the Cloud from a performance, security and availability perspective. “It is important to be able to determine if a problem exists in the application or the Cloud infrastructure when diagnosing a problem and taking corrective action.

Vendors insisted that every application had to provide a robust logging and tracing mechanism for troubleshooting. Additionally, the Cloud provider would generally host and maintain diagnostic tools in-house. As an example, Sridharan said that, Windows Azure began offering a full-featured logging and tracing infrastructure, based on the Event Tracing for the Windows (ETW) framework with the PDC09 release.

In the final analysis, cost and security would be the determining factors that could see large scale adoption of the Cloud being used for software testing.

venkatesh.ganesh@expressindia.com