Different types of performance risks:

The typical risks involved while doing performance testing includes:
Application speed Risks,
Scalability Risks,
Stability risks.

Application Speed risks:
Application Speed risks can be defined in this way.
Check whether the application is responding with enough speed so that the end users are satisfied with its response time.
Check if the application in enough fast to process the data on time. For example if a customer request data needs to be processed with in 1 hour after submission, but the application takes 1.3 hrs?
Check if the application gives updated and accurate data every time when needed? I.e. data related to share quotes, flight timings, currency values etc..

To avoid the above mentioned speed related risks make sure you take the below precautions before releasing the product.

Compare the finished application/product with the existing competitors product and see if your product speed is more than or equal to that.

Compare your application’s speed with the older versions of the same applications to check if the new built is faster than the old version.

Conduct performance testing parallel to some downloads, uploads from and to the application or while some virus scanning is going on.

Conduct performance testing by putting heavy loads on the application just as in real time.

Know the feed back of the application end users about your application.

Scalability Risks:
Scalability Risks include the below.
Check whether the application performs as desired under peak load and at maximum loads at peak hours.
Check if the application can process all the data under peak loads applied to it.
Check if the application functionality is as desired even when heavy loads applied.

Stability Risks:

Base line testing

Base lines are commonly used as metrics to compare the results with future test results to know the applications performance. Base lines can be prepared for individual or entire layers of the application, i.e. for single component, a group of components or entire application. Base lines can be prepared independently for application, database servers, application servers etc.

Before capturing the baselines for the application, one should know in and out of the application. Then only baselines can be captured properly. After every change done to the application, it will undergo a series of performance testing again and compared the results with baselines to see if there are any considerable changes. Base line results can be normally response times, throughput, no of transactions per sec, no of web pages delivered per sec etc…

You can not always take the base lines as a standard comparison every time. It is because when the application changes over time, you need to take new base lines again as a standard. It happens mainly when there is lot of change in the application at the time when you first captured base lines and the application in today’s time.

Stress Testing:

Stress testing is also a part of performance testing which is done to find out the max load that the application can handle during production operations. This kind of testing will be done in such a way that the gradual increase of load will be applied until system breaks down. The point where the application/system breaks down will be noted as the breaking point of the application.

Stress testing is alike Load testing but the only difference is in load testing we apply varying load with a gradual increase and observe the response times, throughput and other statistics just to know how the application is responding at peak loads in peak hours.

Where as stress testing aims at finding the breaking point of the application server/ web server / database server for a particular load. In stress testing we increase the load until the system breaks down. This is mainly useful to know the future statistics, i.e if in future the expected new customers for the application increases, then this calculated stress testing measures will help us to know if the application will work fine with the increase of new customers.

Load Testing:

Load testing is a part of performance testing. It is used to validate the system performance under varying loads at different time levels that are anticipated during production operations.
Normally Load is nothing but the no of requests which will attack the server. In simple terms, if 100k people from all over the globe are using www.mail.yahoo.com at the same time, and trying to upload their photos in through attachment option in yahoo.mail.com, then it is obvious that the system will slow down and it takes couple of mins more to finish the transaction than it takes normally. It is because 100k client requests attacked the server at the same time and since the server can not process 100k requests at a time and probably it may put some for the requests in queue until it finishes the previous requests. So, performance engineers observe and note the time it takes to finish the upload attachment transaction. Now we apply a load of 200k requests at the same time to the server for the same transaction of upload attachment in mail.yahoo.com. So, this time the server takes some more extra time to process the requests and give response back. In this way, performance engineers apply different varying load gradually at frequent intervals and observe the response times of server and analyze the results and prepare a report of the behavior of the server at different loads. Then if they found that the response times are degraded too high, then they will investigate the problem along with the database administrators, web server & application server admistrators together and analyze the logs which are generated in web servers, application servers & database servers to resolve the issue and find out the bottlenecks.

Performance testing:

Performance testing normally done to determine the application speed in terms of response time, throughput (bytes/sec from server), proper utilization of resources of the application/product under test. In simple terms Functional testing is done to check whether the application works as desired functionally (in terms of creativeness of results).
Where as performance testing is done to check whether the application responds at desired speed with the user who is using the application/product. So, it checks whether the user request goes to server as desired & servers responses back to user as desired(in terms of speed). To say a layman’s example for why we need performance testing at all? The answer goes like this.
If you are a customer of ‘ABC Bank’, and you will do online banking to transfer money every Monday morning and Friday evening. Monday mornings when you do this transaction, you will not face any delays in processing and you observe that the website works very fast and you finish the work in 2 mins. But the same transaction when you do on Friday evenings, it will take 10 mins for you to complete the transaction. As a layman, you do not understand why this happened you simply think that your internet connection got some problem. But the problem is in the ABC bank website. Monday mornings the no of customers who does online transactions are comparatively less that that of Friday evenings. So, the application is not able to take the load that is coming on Friday evenings. To avoid this kind of issues and there by loosing valued customers, every application needs to undergo performance testing before it is available for customers.

Pre requisites before doing performance testing:

1)Complete understanding of the project and the 2) main objectives of the performance testing, 3) performance success strategy, 4) SDLC process, 5) The scheduling of the project, 6) Assigned budget of the project, tools and necessary soft wares for doing the performance testing etc….7) End client expectations, 8) Compliance strategy, 8) Security issues, etc.. Performance Tuning Process: Tuning is basically a combined effort of all the wings who directly or indirectly responsible for the application (under test) to perform well. I.e. it includes Developers, Database administrators, Network administrators, System administrators Testers, Vendors, etc... With out the combined effort of all these members, it is impossible to make the application/Software to perform efficiently. When the application is ready, the performance characteristics are gathered and analyzed with that of the original values at the starting and if it is noticed that these values are unacceptable then performance testers & tuning team will enter in the scene and drill down to find out where is the problem. Normally the tuning team and the performance engineers will have full/limited access to the test environment to see if the new changes to the environment will result in better performance of the application. After each change done in test environment by tuning team, the performance engineers will run the different tests to see and compare the performance results with the production bench marks. If performance seems to be improved then the changes will be made permanently and if any changes to test environment results in poor performance then the changes are reverted back to its original pace.

why performance testing

Performance testing is mainly conducted to address various problems/risks that are close to expense, cost and company’s reputation. Performance testing gives us a chance to estimate the performance of the application under test. To avoid the chances of end users not willing to use the application because of its slow response times, which may lead to high revenue loss, loss of reputation, loss of existing and new customers for the company which own the application? So, performance testing gives us a chance to find out the application health in terms of its behavior for high loads/ different load levels / peak loads etc.. Also we can find out the how the application runs if the load increases in future because of newly added customers. So forecasting can be done easily using performance testing. I.e how much load the current infrastructure can handle? If the load increases in future, whether the current infrastructure can handle it? So, depending upon these statistics the company can set up the infrastructure that is needed.

Performance Testing essentials for beginners

A) Identify the complete testing Environment: Find the testing environment before starting anything that includes Network, software and hardware configuration details at production environment and the environment which is available for testing team. This kind of study enables you to do more efficient and in depth testing challenges and there by finding the critical bottle necks. B) Identify the Performance production bench mark criteria. Find out the production bench mark criteria (also called SLA- service level agreement) i.e. desired response time, throughput and system measurements from the respected personnel (users, business users, system administrators at clients place). Some times as a performance engineer, it is your responsibility to suggest the users for better combinations of the results in order to built more efficient end user application. In a simple word you should be in a position to explain what configurations of the existing production system (network band width, data base configurations, application servers’ configuration, web server configurations, and load balancers etc). C) Planning and Designing of the Tests to be done. Planning and designing of test includes creating the more critical, common and unusual scenarios that may occur when the application reaches the end users. A scenario should be created thinking if you were the end user, what activities you will do most often while using the application. For a while assume that your application is www.bankofamerica.com. Consider scenario1, you wanted to transfer some money to your friend. For this scenario you may need to do more than one transaction to complete your goal. i.e first you will login the site with your user name and password (this will be one script), second, you will click on ‘transfer funds’ link and then add a new user(to whom you are going to transfer the money) to your recipients list(this will be second script), thirdly you will transfer the amount by giving your account details and your friends account details(this will be one more script). So, this whole thing of ‘transferring money’ is considered as one scenario. By now you might have understood that each scenario may consist of one or more scripts. You have to design such scenarios along with the proper documentation of what kind of system metrics / server metrics to be collected and consolidated so as to compare the with the Production benchmarks/SLAs. D) Creating / configuring the test environment. Testing environment has to be prepared before conducting the test. The test environment includes the tools, soft wares, hard wares and monitoring tools for resource measurement. A proper study is very important before creating the testing environment, otherwise some times if you run all the scenarios and gathered the results, you realized that the environment is not designed to match the production environment, it will consume more time, resources and money as it includes re testing of all the scenarios all together. E) Implementing the test design. This is nothing but developing the performance tests as required by the test design. This includes developing the scripts for all scenarios as described in the test designs. All the scripts have to be tested independently before executing the scenarios (which includes one or more scripts together). F) Executing tests in test design. Create the scenarios, collect the scripts which are required to create the scenarios into controller and load necessary load generators, add all the server details(IP addresses) i.e application servers, web servers, database servers to the monitoring resources. Once it is completed then schedule the scenario(time to run/start the scenario). Once the scenario is started running then observe the response times, throughputs, system resources of different servers to see if there are any abnormal behavior of the server. Once completed then collect the measurements, other performance data and analyze it and make a detailed report/suggestion note. G) Finally Analyzing reporting/suggestions of test results. Here collect all the results of above step(executing the test) and see individual results and cross functional results of different measures (response time, throughput, system resources, cpu usages etc) and take the ave times for combinations of these and compare all the results with the Production benchmark / SAL(service level agreement) data. If the analyzed results are with in the limits of production bench marks, then your scenarios is passed for those system requirements. If it does not meet the production bench mark results then you need to rerun the test with some changes and again do the whole process to compare with production bench mark data. Do this until it meets the SLA requirements. Many times you need to discuss with the development team, server administrators with the gathered results to find out the bottle necks and solve the problem.

In the analysis what do you understand by Correlated graphs and Overlay graphs?

Both the graphs result in merging of two graphs and showing the results in one graph. But there is a main difference between them. In overlay graphs: The X-axis will be shared by both the graphs. I.e X-axis will be same for both. Where as left hand side Y-axis shows the current graphs results, and right hand side Y-axis shows the results of the graphs that is merged.
In Correlated graphs, Only Y-axis of both the graphs will be compared with each other. The current graphs Y-axis will become the X-axis of the current graph. And the merged graphs Y-axis will become the Y-axis of the current graph.
Explain vuser_init() function?
Vuser_init() contains the statements related to logon to the server. It contains the steps to be performed only once at the beginning, out of all iterations of the script execution.
Explain vuser_end() function?
Vuser_end() contains the statements related to logoff from the server. It contains the steps to be performed only once at the end, out of all iterations of the script execution.

How do you find out application server bottlenecks / issues?

Application server bottlenecks or issues can be found by adding application server monitors and by analyzing the application server related graphs in the controller. These include transaction response time, transactions/sec, total transaction/sec passed, total transactions/sec failed etc…

How can you find webserver related bottlenecks?

Webserver related bottle necks can be found by adding webserver monitors from the controller and analyzing the webserver graphs. Here you can monitor hits/sec, throughput, http response time/sec, pages downloaded per sec, etc… You can also find out if the load generators in the web servers are distributing the load(requests) equally to the application servers? Some times more requests will be send to one application server and the 2nd application server(if you system has) will be sent only few requests etc.. this may lead to long response times.

How can you find Database related bottle necks? Database related issues?

Database related bottle necks/issues can be found by using database monitors. You can add database related graphs from the controller and analyze them by seeing the database resource graph and there by analyzing the database server logs and find out how the sql queries taking time? How much time the database server is taking to send a response back to application server etc…

What is the first step in finding the performance bottle necks? How do you find them?

Performance bottlenecks can be found by using monitors in the controller. By seeing the metrics of the monitors we can find out if there are any performance issues. Normally by seeing the transaction response time, throughput, no of hits per sec etc... We can find out if any performance bottleneck issues. Normally these monitors might include webserver monitors, application server monitors, database server monitors or network monitors.

What do you understand by configuration of your system?

Configuration includes software, hardware, network resources etc.. The configuration of the client system where is vuser is running. It includes any software tools that are running in the system, virtual memory, physical memory, hard disk capacities etc..

How do you compare Response time and Throughput?

Transaction response time is in secs. The time taken by the server to process a request is normally measured by transaction response time. Where as Throughput is measured in bytes. Throughput is number of bytes that a server sends to vuser. So, it is obvious that if the throughput decreases then response time also decreases. If throughput increases then transaction response time also increases.

How can you stop the script while running?

You can stop your script by using a load runner function lr_abort (). There is a default function vuser_end (). But it will stop the script at the end last line. But if you want to stop the script in the middle after some error occurred, then use lr_abort ();.

What is Run VUSER as a thread and Run VUSER as a process?

If you select run vuser as a thread the driver program is shared for every vuser. I,e if 100 vusers are running at the same time, then all the 100 vusers will point and share the same driver program so that lot of memory will be saved. Where as if you select run vuser as a process, then the driver program will be loaded 100 times so that each vuser will use the driver program independently thus taking lot of memory.

Explain Ramp up setting in controller?

Ramp is nothing but increasing the load(number of vusers) gradually over the time. This settings will be done in controller—design tab—edit schedule.

What is the importance of Run time settings? And what changes you can implement in run time settings?

There are few important settings you need to know in run-time settings. 1)Run logic: Here we give number of iterations that a script has to run. 2)In pacing we give the intervals between each iteration(in secs). 3) In Log: we have standard log and Extended log. Extended logs are useful when you need to capture parameter, correlation logs as well. 4)Think time settings mostly either ignore think time or replay think time. 5) In preferences you can configure time out options, you can change from 120 secs(standard) to 1000 secs.

What are the main metrics you need to watch while running tests in order to do a proper performance tuning?

You need not see all of the 100 metrics your servers have. Webserver metrics, application server metrics & database server metrics. You can not judge many time by seeing all the metrics at the same time and getting confused your self and ending up with nothing to do and nothing to understand.
So you have to start from high end metrics like CPU utilization, Memory, Input/Output (I/O), Average load etc...
Understand the bottle necks; see the scalability of the environment.Look for other metrics only if you need additional information

What is Performance tuning? What is its importance? At what stage performance tuning is required?

In very layman words, Performance tuning means changing the system configurations, change in some settings, change in the application etc..., to improve the performance of the application.
But when this needs to be done? At what stage performance tuning is required? Before preparing the test plan? Or after writing the test plan or after running the tests (Load test etc...)? Many people would probably say they need to do performance tuning before or after the test plan is prepared. But in reality performance tuning is required after running a load test, benchmark test, stress test etc… It is because only after performing few tests you will compare the response times and other metrics and if you think any of these metrics are abnormal, then you can suggest for some changes in system configuration, settings etc… so this is the exact time that you need performance tuning.

How to find Virtual memory bottle necks in Load runner?

Normally in any java application this virtual memory issues can be found by using the operating system utility called VMSTAT. If the memory in JVM machine is low then the application will take too long for GC (garbage collection). There is one more alternative to find out the reason if your application is very low in memory is Java.Lang.OutofMemory exceptions. Systems will through java.lang.outofmemory exceptions if the application is low in memory. So memory related problems can be found by using ‘Heap size’, ‘verbose garbage collection’, or by ‘VMSTAT’.

Memory Bottle necks: How do you find Physical memory issues? Physical memory related bottle necks?

Physical memory: In UNIX environment physical memory can be found by using VMSTAT command. Normally people thing if there is a paging indication, then there is a problem with physical memory, but in fact it is not. But you have to see the ‘Scan Rate’ metrics to find out the physical memory issue. Because this metric will tell you when ever the OS will scan for memory pages to page out.
Normally all OS will maintain the memory to cache the input/output pages to improve the performance by saving disk memory. Paging is only a problem when the memory used by the application/product is paged to the physical disc. But before this happens, you will normally look for page scanning activity by OS in the VMSTAT.

What is CPU related Bottle necks? And how to fix them?

As many of us think, CPU bottle neck is the simplest among all to find out and of course to solve. In UNIX environment VMSTAT, IOSTAT will give the CPU resources and by analyzing you can confirm if there is a CPU problem. Normally if system resources are 90% or more then you can confirm that it’s a CPU bottleneck. In windows environment ‘Perfmon’ will give the CPU resources utilization. Simply got to command prompt and type ‘perfmon’, press enter.
Once you are decided that there is an issue, and then there are two ways to solve. The simplest way is to increase the CPU capacity either by buying additional memory or by buying physical hard disk. And the other alternative is to tune the application so as to utilize min CPU resources.

What are bottle necks in Load runner? Did you face any bottle necks in your previous projects? What is your approach in finding bottlenecks?

In simple words, Bottle neck is nothing but poor performance of the application. Example: you are trying to upload an image through your yahoo mail, and it is taking long time to upload, you (user) are getting frustrated with this delay of the application response time. This is clearly a bottle neck. But this bottle neck need not be yahoo applications problem, but depends on the internet speed your system is using.
There are normally different types of bottle necks, eg: memory bottle necks, network bottle necks, hard ware bottle necks, server bottle necks(Server bottle necks includes Application server, webserver & database server) and also application(code related, complex sql queries, nested queries etc..) bottle necks.
Memory bottle necks include Physical memory and or Virtual memory problems, memory leakage problems. Memory leakage can be found normally by Heap size. If the heap size is decreased gradually over time, then it is clear that there is some problem. If you are running a load test for long time, and you observe that the heap size is getting decreased then one of the problems could be memory leakage. Memory leakage means there might be a memory leak in the application. If garbage collection (GC) is not handled properly in the application (developers job), this memory leakage problems occurs.
Bottle necks mainly fall under any of the 4 areas. 1) CPU, 2) Memory, 3) I/O, 4) Software. This very important to know because then only you can start finding the bottle necks and solve them in a systematic order.

Did you write any user-defined functions in your previous project? How to write and implement user defined functions in Load runner?

If you want to use user defined functions in your load runner script, first you need to define external libraries also called DLLs (dynamic link library) and store them in the Bin directory. Now create a user defined function in vugen script and pass this function name as a parameter to the DLL which is there in the Bin directory.
There is a function called lr_load_dll (), by which you can load DLLs and there by create your own functions in the script. You can normally create a DLL either using C or C++. And have to include some .h includes also. Normally one should have good C/C++ programming knowledge to write DLLs. If you are good in programming before then you can concentrate more on DLLs, else just understand the basics and leave it for programmers. As a performance engineer you need to concentrate more on analysis to find out the bottle necks.

Can you debug a loadrunner script?

Yes, Loadrunner has an option of debugging. Click on any line in the script you want to start debug, right click, and select ‘toggle break point’. And the other option is from the menu bar, Edit—breakpoint... Once the breakpoint is set, and when you run the script it will stop at the place when you kept break point. Here if can press F10 to execute line by line or press F5 to execute rest of the line at a time. You can see the debug values in the output window. ‘Lr_set_debug_message’ function is used to insert in the script to debug a portion of the script.

What are different types of Logs in Loadrunner? What to use when?

There are three main options to capture logs in Loadrunner. Standard logs and Extended logs and Errors only. Standard logs will capture logs of each and every functionality and how the server responses for every request etc. Extended logs will capture everything including error messages and warning messages from the server including what it logs in standard. ‘Errors only’ will capture only when an error occurred while running the script. Once we the functionality of the script is correct, then we use normally errors only option to reduce the log size. Because if you select ‘extended logs’ for larger scripts, they will capture 1000s of lines of logs which will take more disk space. When we call a script in a scenario, the logging will be automatically disabled.

What is LB, RB & ORD in correlation?

‘LB’ is left boundary. We normally take few words left to the value which we want to correlate and will be assigned to LB. And few words right to the value will be assigned to RB. Since every time when we run the script, the correlated value will change by the server but not left and right side values. So, when correlation is done, the server recognizes the correlation value by reading the LB & RB values. ORD is ordinance. i.e. the occurrence of the correlated value in the entire script, when it changes in the 1st occurrence or 2nd, 3rd etc..Depending on this we keep ORD =1, ORD=2 etc..If we are not sure of which occurrence then we put ORD = ALL.

What is LB, RB & ORD in correlation?

‘LB’ is left boundary. We normally take few words left to the value which we want to correlate and will be assigned to LB. And few words right to the value will be assigned to RB. Since every time when we run the script, the correlated value will change by the server but not left and right side values. So, when correlation is done, the server recognizes the correlation value by reading the LB & RB values. ORD is ordinance. i.e. the occurrence of the correlated value in the entire script, when it changes in the 1st occurrence or 2nd, 3rd etc..Depending on this we keep ORD =1, ORD=2 etc..If we are not sure of which occurrence then we put ORD = ALL.

Explain the need of parameters in Loadrunner?

The functionality of parameters are same every where ( whether in loadrunner, c, java etc..). Parameters act as a variable in the code, and they help in passing different values to the servers every time when it talks to the server. i.e we wat to run the load test on “Login” functionality with 5 users simultaneously. So, we record the script first time by logging into the application by using a user id / pwd. To conduct load test with 5 users, we need 5 sets of user ids/pwds to simulate 5 different users logging at the same time. In this situation, we create parameters with two variables ex ‘username’ & ‘password’, and we pass 5 sets of userids/pwds to the script so that the parameters will replace with a new value every time when the script runs. Parameters will reduce the length of the code

What does recording mode in vugen will do?

Recording mode in vugen will help record the actions that a user will do on the application/product to test its functionality. While testing the functionality of the application, the requests are sent from the client (user machine) to server (database machine) and vise-versa. All these transactions will be recorded in vugen and it will generate a script automatically. Now we can run the script any time we want to simulate the same actions that we did while recording the script. We can even modify the script by adding some functions when we need to add some extra functionality.

Explain a Scenario? in Loadrunner

A scenario is a series of events that occur during a session. A scenario may include but not limited to no of vusers to work in the session, no of load generators to work and how the vusers are assigned to each load generator, which script to run after / before another script etc...

Explain the importance of a Rendezvous point?

Rendezvous point is normally inserted in between the vuser scripts to emulate the heavy/peak loads on the servers. This is achieved by gathering vusers to wait until some time and release all at a time to put load on the server all at once. For example stop all your users up to certain time until they are 200 in count and then release all at a time so that every one logs in the same time to give more load on the server.

Name the different components of a Loadrunner?

Load runner has the following components. Vuser generator, Analysis, Monitoring, Controller, Loadrunner online books and the agent process. There is so much to learn about each component and its importance. You can see loadrunner books online, to study each of these components thoroughly.
Vuser generator: is used for creating scripts, debugging, running the scripts.
Controller: is used to create a scenario, schedule the scenario and running the scenario.
Analysis: is used to analyze the results after running the scenario.

performance test plan activities

Performance Testing Activities:Identify the Test environment.Identify the Performance acceptance/Test Criteria.Plan the test and design it.Prepare the test environment.Executing the test.Analyzing and reporting of the test.Identify the Test environment:Identify the production and test environments. Compare them if they are scaled properly.Compare the production environments that may include network band width, server configurations, Resources i.e hardware, software and tools to that of the test environment.Identify the Performance acceptance criteria:Consider response time and through put of the different transactions in an application. Find out what configurations may lead to best response time and as well throughput.Plan and design the Test:Identify the critical/important scenarios, detail about the metrics you are going to calculate during the tests, Simulate the test data which should match more or less production data.Configuring the test environment:Now configure the test environment that is similar to the production environment.This includes software, Hardware, tools, resources etc.. Scalability should be done carefully from production to test environment, because the test results will be taken as benchmarks to compare the production environments performance.Execute/Implement the test design:Implement the test design by executing the tests you have designed in the test design phase. Run the scripts and run the different tests that are mentioned in the design phase.Analyzing and reporting the test results:Analyze the test results that are generated by running the tests in test design. Analyze the response times, throughputs, and cpu utilization, other network resources etc.. and make a detailed report with what you have noticed and give your suggestions to improve the performance of the application. Performance, testing, performance testing, loadrunner, load runner, mercury load runner, performance activities, load tets

Difference between Schedule by Group and Schedule by scenario:

Difference between Schedule by Group and Schedule by scenario:Schedule by Group:Schedule by group allows to delaying a test script on the remote host. i.e we can select the order of scripts to run on remote host.Ex: Run script2 after script 1,Run script3 after script 2Schedule by scenario:Schedule by scenario will allow you to run N number of virtual users to run exactly at equal intervals. Say forex: “run 10 v users every 10 mins”Note: Ramp up, Duration, Ramp down will be same options for both of these.Schedule by Scenario will have extra option of “start time” to make the order of scripts to run on remote host.

Important Performance Test Activities:

Important Performance Test Activities:A) Identify the complete testing Environment:Find the testing environment before starting anything that includes Network, software and hardware configuration details at production environment and the environment which is available for testing team. This kind of study enables you to do more efficient and in depth testing challenges and there by finding the critical bottle necks.B) Identify the Performance production bench mark criteria.Find out the production bench mark criteria (also called SLA- service level agreement) i.e. desired response time, throughput and system measurements from the respected personnel (users, business users, system administrators at clients place). Some times as a performance engineer, it is your responsibility to suggest the users for better combinations of the results in order to built more efficient end user application. In a simple word you should be in a position to explain what configurations of the existing production system (network band width, data base configurations, application servers’ configuration, web server configurations, and load balancers etc).C) Planning and Designing of the Tests to be done.Planning and designing of test includes creating the more critical, common and unusual scenarios that may occur when the application reaches the end users.A scenario should be created thinking if you were the end user, what activities you will do most often while using the application. For a while assume that your application is www.bankofamerica.com. Consider scenario1, you wanted to transfer some money to your friend. For this scenario you may need to do more than one transaction to complete your goal. i.e first you will login the site with your user name and password (this will be one script), second, you will click on ‘transfer funds’ link and then add a new user(to whom you are going to transfer the money) to your recipients list(this will be second script), thirdly you will transfer the amount by giving your account details and your friends account details(this will be one more script). So, this whole thing of ‘transferring money’ is considered as one scenario. By now you might have understood that each scenario may consist of one or more scripts. You have to design such scenarios along with the proper documentation of what kind of system metrics / server metrics to be collected and consolidated so as to compare the with the Production benchmarks/SLAs.D) Creating / configuring the test environment.Testing environment has to be prepared before conducting the test. The test environment includes the tools, soft wares, hard wares and monitoring tools for resource measurement. A proper study is very important before creating the testing environment, otherwise some times if you run all the scenarios and gathered the results, you realized that the environment is not designed to match the production environment, it will consume more time, resources and money as it includes re testing of all the scenarios all together.E) Implementing the test design.This is nothing but developing the performance tests as required by the test design. This includes developing the scripts for all scenarios as described in the test designs. All the scripts have to be tested independently before executing the scenarios (which includes one or more scripts together).F) Executing tests in test design.Create the scenarios, collect the scripts which are required to create the scenarios into controller and load necessary load generators, add all the server details(IP addresses) i.e application servers, web servers, database servers to the monitoring resources. Once it is completed then schedule the scenario(time to run/start the scenario). Once the scenario is started running then observe the response times, throughputs, system resources of different servers to see if there are any abnormal behavior of the server. Once completed then collect the measurements, other performance data and analyze it and make a detailed report/suggestion note.G) Finally Analyzing reporting/suggestions of test results.Here collect all the results of above step(executing the test) and see individual results and cross functional results of different measures (response time, throughput, system resources, cpu usages etc) and take the ave times for combinations of these and compare all the results with the Production benchmark / SAL(service level agreement) data. If the analyzed results are with in the limits of production bench marks, then your scenarios is passed for those system requirements. If it does not meet the production bench mark results then you need to rerun the test with some changes and again do the whole process to compare with production bench mark data. Do this until it meets the SLA requirements. Many times you need to discuss with the development team, server administrators with the gathered results to find out the bottle necks and solve the problem

Need of Performance testing for any application:

Need of Performance testing for any application:Performance testing is mainly conducted to address various problems/risks that are close to expense, cost and company’s reputation. Performance testing gives us a chance to estimate the performance of the application under test.To avoid the chances of end users not willing to use the application because of its slow response times, which may lead to high revenue loss, loss of reputation, loss of existing and new customers for the company which own the application?So, performance testing gives us a chance to find out the application health in terms of its behavior for high loads/ different load levels / peak loads etc..Also we can find out the how the application runs if the load increases in future because of newly added customers. So forecasting can be done easily using performance testing. I.e how much load the current infrastructure can handle? If the load increases in future, whether the current infrastructure can handle it? So, depending upon these statistics the company can set up the infrastructure that is needed.Pre requisites before doing performance testing:1)Complete understanding of the project and the 2) main objectives of the performance testing, 3) performance success strategy, 4) SDLC process, 5) The scheduling of the project, 6) Assigned budget of the project, tools and necessary soft wares for doing the performance testing etc….7) End client expectations, 8) Compliance strategy, 8) Security issues, etc..

Performance tuning

Performance Tuning Process:Tuning is basically a combined effort of all the wings who directly or indirectly responsible for the application (under test) to perform well. I.e. it includesDevelopers, Database administrators, Network administrators, System administrators Testers, Vendors, etc...With out the combined effort of all these members, it is impossible to make the application/Software to perform efficiently. When the application is ready, the performance characteristics are gathered and analyzed with that of the original values at the starting and if it is noticed that these values are unacceptable then performance testers & tuning team will enter in the scene and drill down to find out where is the problem.Normally the tuning team and the performance engineers will have full/limited access to the test environment to see if the new changes to the environment will result in better performance of the application. After each change done in test environment by tuning team, the performance engineers will run the different tests to see and compare the performance results with the production bench marks. If performance seems to be improved then the changes will be made permanently and if any changes to test environment results in poor performance then the changes are reverted back to its original pace.

Performance testing:

Performance testing:Performance testing normally done to determine the application speed in terms of response time, throughput (bytes/sec from server), proper utilization of resources of the application/product under test. In simple terms Functional testing is done to check whether the application works as desired functionally (in terms of creativeness of results).Where as performance testing is done to check whether the application responds at desired speed with the user who is using the application/product. So, it checks whether the user request goes to server as desired & servers responses back to user as desired(in terms of speed). To say a layman’s example for why we need performance testing at all? The answer goes like this.If you are a customer of ‘ABC Bank’, and you will do online banking to transfer money every Monday morning and Friday evening. Monday mornings when you do this transaction, you will not face any delays in processing and you observe that the website works very fast and you finish the work in 2 mins. But the same transaction when you do on Friday evenings, it will take 10 mins for you to complete the transaction. As a layman, you do not understand why this happened you simply think that your internet connection got some problem. But the problem is in the ABC bank website. Monday mornings the no of customers who does online transactions are comparatively less that that of Friday evenings. So, the application is not able to take the load that is coming on Friday evenings. To avoid this kind of issues and there by loosing valued customers, every application needs to undergo performance testing before it is available for customers.

Load testing

Load Testing:Load testing is a part of performance testing. It is used to validate the system performance under varying loads at different time levels that are anticipated during production operations.Normally Load is nothing but the no of requests which will attack the server. In simple terms, if 100k people from all over the globe are using www.mail.yahoo.com at the same time, and trying to upload their photos in through attachment option in yahoo.mail.com, then it is obvious that the system will slow down and it takes couple of mins more to finish the transaction than it takes normally. It is because 100k client requests attacked the server at the same time and since the server can not process 100k requests at a time and probably it may put some for the requests in queue until it finishes the previous requests. So, performance engineers observe and note the time it takes to finish the upload attachment transaction. Now we apply a load of 200k requests at the same time to the server for the same transaction of upload attachment in mail.yahoo.com. So, this time the server takes some more extra time to process the requests and give response back. In this way, performance engineers apply different varying load gradually at frequent intervals and observe the response times of server and analyze the results and prepare a report of the behavior of the server at different loads. Then if they found that the response times are degraded too high, then they will investigate the problem along with the database administrators, web server & application server admistrators together and analyze the logs which are generated in web servers, application servers & database servers to resolve the issue and find out the bottlenecks.

Stress Testing:

Stress Testing:Stress testing is also a part of performance testing which is done to find out the max load that the application can handle during production operations. This kind of testing will be done in such a way that the gradual increase of load will be applied until system breaks down. The point where the application/system breaks down will be noted as the breaking point of the application.Stress testing is alike Load testing but the only difference is in load testing we apply varying load with a gradual increase and observe the response times, throughput and other statistics just to know how the application is responding at peak loads in peak hours. Where as stress testing aims at finding the breaking point of the application server/ web server / database server for a particular load. In stress testing we increase the load until the system breaks down. This is mainly useful to know the future statistics, i.e if in future the expected new customers for the application increases, then this calculated stress testing measures will help us to know if the application will work fine with the increase of new customers.

base line testing

Base linesBase lines are commonly used as metrics to compare the results with future test results to know the applications performance. Base lines can be prepared for individual or entire layers of the application, i.e. for single component, a group of components or entire application. Base lines can be prepared independently for application, database servers, application servers etc.Before capturing the baselines for the application, one should know in and out of the application. Then only baselines can be captured properly. After every change done to the application, it will undergo a series of performance testing again and compared the results with baselines to see if there are any considerable changes. Base line results can be normally response times, throughput, no of transactions per sec, no of web pages delivered per sec etc…You can not always take the base lines as a standard comparison every time. It is because when the application changes over time, you need to take new base lines again as a standard. It happens mainly when there is lot of change in the application at the time when you first captured base lines and the application in today’s time.

different types of testing

Different Types of Testing:

Smoke test:
Smoke test is conducted before conducting performance test to find out whether the application runs smoothly under basic load. This is mainly to find out whether the application runs as desired functionally. i.e to check if each and every component of the product works well.Performance testingPerformance test is normally done to validate/investigate the performance of the application in terms of its speed (response time), and other characteristics like through put, of the application under test.

Load testing:

Load testing is done to find out the application / product behavior under peak load situations. Load testing is done to know whether the product meets the expected requirements as mentions in production base lines or as represented in SLA (service level agreement). Load testing is done to know up to what load system can handle with out failure. It is assumed that below that level system fails for any further requests/load.

Stress testing:

Stress testing is a subset of Load testing where it is also done to see the performance of the application under peak loads. But here the load is applied to such extent that the system completely crashes and will not respond to any requests. The load will be applied continuously until the system fails. These metrics also used for future reference when the load increases in future.Capacity testingCapacity testing is done to find out the measurements of the servers in terms of CPU utilization, memory usage, other server resources etc… This is mainly to determine the health of the environment, not the application. Environment means the servers where the application is installed and other tools that that are used to distribute the loads to the servers and the servers that run the application etc…

Spike testing:

Spike testing is similar to stress testing. In spike testing load is applied beyond its peak load point at frequent intervals in a zigzag way to find out the applications performance.Endurance testingEndurance testing is done for long hours to check if the application works well in a consistent manner for long duration of runs. When a test is run for long hours normal errors may found is Garbage collection , memory leakage etc..

Unit testing:

Unit testing is done normally for a single peace of code out of the whole application. Normally unit testing is conducted by developers when they finished their coding for the assigned unit. Developers will prepare their own test cases and do the unit testing.

Different types of performance risks:

The typical risks involved while doing performance testing includes:
Application speed Risks, Scalability Risks, Stability risks.

Application Speed risks:

Application Speed risks can be defined in this way.Check whether the application is responding with enough speed so that the end users are satisfied with its response time.Check if the application in enough fast to process the data on time. For example if a customer request data needs to be processed with in 1 hour after submission, but the application takes 1.3 hrs?Check if the application gives updated and accurate data every time when needed? I.e. data related to share quotes, flight timings, currency values etc..To avoid the above mentioned speed related risks make sure you take the below precautions before releasing the product.Compare the finished application/product with the existing competitors product and see if your product speed is more than or equal to that.Compare your application’s speed with the older versions of the same applications to check if the new built is faster than the old version.Conduct performance testing parallel to some downloads, uploads from and to the application or while some virus scanning is going on.Conduct performance testing by putting heavy loads on the application just as in real time.Know the feed back of the application end users about your application.Scalability Risks:Scalability Risks include the below.Check whether the application performs as desired under peak load and at maximum loads at peak hours.Check if the application can process all the data under peak loads applied to it.Check if the application functionality is as desired even when heavy loads applied.Stability Risks:

importance of load testing

What is the use of Load testing (performance testing)? Why it is important?Load testing is important to find out the application / products performance in terms of its speed, response time and how it handles when large number of users uses the application at the same time. Performance or Load testing is important because it plays a vital role in deciding the usability / reliability of the application in present and future times. Load testing forecasts whether the application can handle the future growth of the users to the application. It gives a clear picture whether the current application’s set up / configuration / environment will support if after 2 years the number of users who uses the application will double.So, for every organization / enterprise which cares for its reputation or brand value, must undergo performance / Load testing of their application.There is little some difference between Load testing and Performance testing.Load testing aims to find out if, more and more number of users trying to use the application at the same time. To find out at what loads the system will crash?Performance testing cares about so many other areas like system metrics, response times of transactions, through put of the servers etc.. It aims at finding out the over all performance of the application in terms of user friendly, customer(end user) satisfaction while browsing the application(if it is web application) irrespective of the peak loads or not.

What versions of LoadRunner did you use in your latest project?

What versions of LoadRunner did you use in your latest project?Yes, I have used version 8.1 for the entire loadrunner suit which includes (controller, vugen, Analysis etc...).

Can you explain the entire Load testing process in detail?

Can you explain the entire Load testing process in detail?Planning of the Test:Creating Vusers.Creating scenario.Running the created scenarios.Monitoring /observing the scenarios.Finally analyzing the test results.A: Planning of the Test: here we plan the test in such a way that it servers the objective of the load testing. I.e. creating the scenarios that will server the load testing process in order to reach out objective of testing.B: Creating Vusers: Creating vusers are nothing but creating the scripts by recording different functionality of the application. These scripts will have the tasks / transaction that will be run by vusers (virtual users).C: Creating the scenario: Creating the scenario is the important step in load testing process. Creating scenario includes number vusers, vusers scripts, The transactions that made the scenario, and load generation machines etc…. We will create a scenario in controller. There are normally 2 types of scenarios. 1. Manual scenario 2. Goal oriented scenario.D: Running the scenarios: Running the scenario means allowing vusers to run the scripts so that the load will be applied simultaneously on the server. After creating the scenario in controller, we will schedule the scenario and then run the scenario.E: Monitoring the scenarios: Monitoring the scenario means we will observe the different resources while the scenario is running. These resources include but not limited to System resources, Web resources, Network resources, Server resources, Application server resources, Data streaming resources, Database server resources, security resources etc…. After monitoring these resources if we notice any abnormal metrics then contact the related server administrators or the personnel who is responsible for that and they will investigate on this issue.F: Analyze the test results: After running the scenario, we will go to Loadrunner analyzer and create the respective graphs for different servers, and analyze the metrics with the benchmark results/metrics to see if the results are near to those benchmarks or they are degraded. If they are degraded, then analyze why this degradation happened? This you can do by analyzing the server logs and find out if database server is taking long time, then send those logs to database administrator to work on this. If application server is taking long time to process requests then contact application server administrator to work on this issue. The same way if requests are in queue for a long time at web server, i.e load distribution is not handled properly by load generator then contact the appropriate personnel to solve this issue.

What is the difference between Manual scenario and Goal-oriented scenario?

What is the difference between Manual scenario and Goal-oriented scenario?In Manual scenario, load test will be managed by number of vusers that are assigned to the test. In Manual scenario the numbers of vusers are distributed among vuser scripts depending upon the settings we made in load generators percentage.In Goal-Oriented scenario, the controller will create a scenario based on the goals we specify. In goal oriented scenario the goal will be automatically created by controller based on number of transactions per sec, number of hits per sec.

What is the max no of vusers you have used in your recent project?

What is the max no of vusers you have used in your recent project?We have used 1000 vuser, because we have license only for 1000 users, our company is in the process of buying 500 more vusers from mercury. Normally you try to give the figure in between 200 to 1000 vusers. Most of the companies will have up to 500 vusers license. Merely do not say that you have used 5000 vusers at a time.