5
Test Considerations
This chapter provides an overview of test considerations. It includes the following sections:
Time Based Testing
This section describes how to run WebHammer for a specific period of time. Normally, WebHammer runs a workload using the information contained in the scenario file. When all the entries in the scenario file are processed, the test is complete. There may be situations, however, when you would rather repeatedly run a set workload for a specific period of time. For example, you can set up a scenario file to contain all the possible URL requests and operations that users perform on the Web site. This test workload runs continuously 12 or 24 hours and the information is recorded to the log file.
The -f option runs the WebHammer test continuously. The following is an example of a hammer command with a request for continuous operation:
iesun8 > hammer -f -u 10 -p 80 -r edtcsun33 -o log scenario
Continuous tests are stopped by pressing CTRL-C on your keyboard or with the interrupt character. On receipt of the interrupt character, WebHammer outputs results to the designated log file. The fourth column which, without the -f option, is never greater than 1, displays large values such as 2191. This value represents the number of times that a response was received for that URL operation. The fifth column represents the number of times no response was received for a particular URL operation. This may indicate that the remote server is under stress. The following is a sample of the results:
0:185150:3553802:2191:4
1:180707:3553802:2191:0
2:186445:3625170:2235:0
3:183125:3625170:2235:3
4:178547:3433774:2117:0
5:184293:3432152:2116:0
6:181792:3437018:2119:0
7:181118:3437018:2119:3
8:177248:3498654:2157:0
9:185576:3498654:2157:0
10:176634:3407822:2101:0
Running Multiple Tests Simultaneously
This section describes how to run multiple WebHammer tests simultaneously and how to stop them. There may be certain situations in which it is desirable to run several instances of WebHammer simultaneously, with each invocation running a different scenario or workload. This is shown in the following example, in which each test is running a different scenario:
hammer -f -u 10 -p 80 -r edtcsun33 -o log1 scenario1 &
hammer -f -u 10 -p 80 -r edtcsun33 -o log2 scenario2 &
hammer -f -u 10 -p 80 -r edtcsun33 -o log3 scenario3 &
hammer -f -u 10 -p 80 -r edtcsun33 -o log4 scenario4 &
If the -f option is used as in the preceding example, these tests run continuously in the background.
The tests can be stopped if needed. To terminate the tests running in the background, use the kill command to send a SIGTERM signal as follows:
kill -s TERM <pid>
where <pid> is the process ID of the hammer command.
In this way, tests are terminated cleanly and the results are written to the appropriate log files.
High Volume Testing
This section describes some elements that you need to consider on the client system when you perform tests simulating a large number of users, or greater than 1000 users.
If you are running a large test simulating a large number of users, WebHammer modifies system resources such as the maximum number of file descriptors that are opened by a process.
Run WebHammer as root. You can also set the User ID on execution bit to allow the system modification. The following is an example of how a super user performs this task:
iesun8 > chown root hammer
iesun8 > chmod 4755 hammer
Alternatively, you can use the ulimit command from the shell to increase the maximum number of file descriptors that are opened by a process. You can calculate the value for the maximum number of file descriptors to equal the number of users to simulate plus four.
The following is an example of how a super user performs this task using ulimit:
iesun8 > ulimit -n 1200
Workflow Scenarios
If you would like to test your Web server with a series of POST or GET operations, recreating a user moving through a procedure such as a credit card transaction, you can do so with a workflow scenario file.
WebHammer supports the concept of workflow and transaction support. This is the result of a URL operation parsed for specific content, for example, an order id. That content is used to dynamically build the next URL operation in the sequence.
In order to specify that the results of a URL operation should be parsed, you need to include the following line directly after the URL operation whose results you want to parse in the scenario file:
FIND start_delimiter1 end_delimiter1 start_delimiter2 end_delimiter2...start_
delimter<n> end_delimiter<n>
A delimiter is any character or string. You can specify any number of delimiter pairs. Start and end delimiters can be different, for example:
FIND ~ #
The FIND local operation searches for the tilde (~) , selecting the information from that point. The selection continues until the FIND local operation detects the pound mark(#) the end delimiter.
To use a workflow scenario effectively, you must know what pattern results are returned from your Web server. For example, if you enter a credit card number in a form, the results from your Web server indicate the delimiters around the credit card number.
To create a workflow scenario file, perform the following:
- Create a scenario file. A scenario file is a workload file.
- Add a FIND local operation to the scenario file directly after the URL operation, either GET or POST.
- After the FIND operation, you need another POST or GET operation. Within this POST or GET operation section of your scenario file, indicate token replacement with { }.
The POST or GET operation is then dynamically built by replacing the content between the { } with the parsed values.
{1,1} locates the first delimiter pair, replacing the token with the first occurence of text for that pair.
{1,2}locates the first delimiter pair, replacing the token with the second occurrence of text for that pair.
{2,1} locates the second delimiter pair, replacing the token with the first occurrence of text for that pair.
- Save the scenario file.
- Run the
hammer command.
The following example shows a credit card payment. When the results are received back for the first POST operation, a search is performed for the text enclosed between &ord= and &ad. The text string between these two delimiters is the order id. This order id is used in the next POST operation and dynamically replaced {1,1} with the order id.
POST /acme/olsr/1200 HTTP/1.0
CA-Fields: 0102030405060708090A0B0C0D0E0F101112131415161718191A
User-Agent: OpenTV 1.2
Accept: application/octet-stream
Content-Type: application/octet-stream
X-OpenTV-Appl-id: 1000-400000
X-OpenTV-Appl-ver: 1
X-OpenTV-Appl-time: 26 Oct 1999 00:00:00
Content-Length:
tid=1200&msi=614408&con=1000000005&lin=1&itm=100115&qty=1&val=141
FIND &ord= &ad
POST /acme/olsr/1201 HTTP/1.0
CA-Fields: 0102030405060708090A0B0C0D0E0F101112131415161718191A
User-Agent: OpenTV 1.2
Accept: application/octet-stream
Content-Type: application/octet-stream
X-OpenTV-Appl-id: 1000-400000
X-OpenTV-Appl-ver: 1
X-OpenTV-Appl-time: 26 Oct 1999 00:00:00
Content-Length:
tid=1201&msi=614408&con=1000000005&ord={1,1}&pid=871&nam=VisaCard&acc=9200000000
00002948&iss=1&exp=0201&str=0198&spm=
&sav=N
Troubleshooting and Errors
The following table lists the errors that can occur and the corresponding actions to performe to resolve them.
Monitoring System Performance on the Remote System
This section describes how to monitor the remote system during a workload test. This is useful when running a stress test to determine how the remote system is coping with the workload.
When performing high volume testing with a large number of users, the remote system might be under stress. If the remote system is a UNIX-based system, as in our test example, you can examine the network statistics using the following command:
edtcsun33 > netstat -f inet
The output of this command shows information relating to TCP connection between the Apache Web server on edtcsun33 and WebHammer on iesun8. The Web server is listening on port 80 and the corresponding connections that were established from the WebHammer client on iesun8.
The following table describes states that appear in the testing example:
TCP: IPv4
Local Address Remote Address Swind Send-Q Rwind Recv-Q State
-------------------------------------------------------------------------------------------------
edtcsun33.ie.oracle.com.80 iesun8.ie.oracle.com.42666 8760 0 24820 0 TIME_WAIT
edtcsun33.ie.oracle.com.80 iesun8.ie.oracle.com.42667 8760 0 24820 0 TIME_WAIT
edtcsun33.ie.oracle.com.80 iesun8.ie.oracle.com.42668 8760 0 24820 0 TIME_WAIT
edtcsun33.ie.oracle.com.80 iesun8.ie.oracle.com.42669 8760 0 24820 0 TIME_WAIT
edtcsun33.ie.oracle.com.80 iesun8.ie.oracle.com.42670 8760 0 24820 0 TIME_WAIT
edtcsun33.ie.oracle.com.80 iesun8.ie.oracle.com.42671 8760 0 24820 0 TIME_WAIT
edtcsun33.ie.oracle.com.80 iesun8.ie.oracle.com.42672 8760 0 24820 0 TIME_WAIT
edtcsun33.ie.oracle.com.80 iesun8.ie.oracle.com.42673 8760 0 24820 0 TIME_WAIT
edtcsun33.ie.oracle.com.80 iesun8.ie.oracle.com.42674 8760 0 24820 0 TIME_WAIT
edtcsun33.ie.oracle.com.80 iesun8.ie.oracle.com.42675 8760 0 24820 0 TIME_WAIT
edtcsun33.ie.oracle.com.80 iesun8.ie.oracle.com.42676 8760 0 24820 0 FIN_WAIT_2
edtcsun33.ie.oracle.com.80 iesun8.ie.oracle.com.42677 8760 0 24820 0 FIN_WAIT_2
edtcsun33.ie.oracle.com.80 iesun8.ie.oracle.com.42678 8760 0 24820 0 FIN_WAIT_2
edtcsun33.ie.oracle.com.80 iesun8.ie.oracle.com.42679 8760 0 24820 0 FIN_WAIT_2
edtcsun33.ie.oracle.com.80 iesun8.ie.oracle.com.42680 8760 0 24820 0 ESTABLISHED
edtcsun33.ie.oracle.com.80 iesun8.ie.oracle.com.42681 8760 0 24820 0 ESTABLISHED
edtcsun33.ie.oracle.com.80 iesun8.ie.oracle.com.42682 8760 0 24820 0 ESTABLISHED
If the remote system is UNIX, you can use the following commands to monitor the system.
iostat - report I/O statistics
vmstat - report on virtual memory statistics
sar - system activity reporter
See your system documentation for more information on these commands.
Using Trace
This section describes how to set and interpret trace information. Trace information is enabled by using the verbose or -v <level> option., where <level> specifies the level of trace required. There are five levels of trace, and you can combine these if required. The five levels are as follows:
- Level 1 - print URL data received from Web server in response to a URL operation
- Level 2 - print the actual URL operation that is sent to the Web server
- Level 4 - print the number of bytes received from the Web server in response to a URL operation
- Level 8 - URL transmission rate information
- Level 16 - General trace information
You can combine trace levels, for example, 1+2+4+8+16= 31 includes all debug information.
Trace information is useful when you are setting up your workload initially or are experiencing problems and need extra information in order to troubleshoot.
Level 1
Level 1 displays the actual response that is read from a Web server as a result of a URL request being sent to the Web server. This is useful when validating a response from the Web server to a URL operation.
The following is an example of the hammer command using Level 1 trace and the corresponding trace text, with bold text representing actual text received from the Web server:
iesun8 > hammer -v 1 -n 1 -u 1 -p 80 -r edtcsun33 -o log scenario_get
Trace level=1
Parsing scenario file "scenario_get"
Started 1 users, start staggered by 10 msec, divided workload, each user
executes 1 URLs
Please wait while workload completes...
Starting user number [0]
READ response data=[<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<HTML>
<HEAD>
<TITLE>Test Page for Apache Installation on Web Site</TITLE>
</HEAD>
<!-- Background white, links blue (unvisited), navy (visited), red (active) -->
<BODY
BGCOLOR="#FFFFFF"
TEXT="#000000"
LINK="#0000FF"
VLINK="#000080"
ALINK="#FF0000"
>
<H1 ALIGN="CENTER">
It Worked! The Apache Web Server is Installed on this Web Site!
</H1>
<P>
If you can see this page, then the people who own this domain have just
installed the <A HREF="http://www.apache.org/">Apache Web server</A>
software successfully. They now have to add content to this directory
and replace this placeholder page, or else point the server at their real
content.
</P>
<HR>
<BLOCKQUOTE>
If you are seeing this page instead of the site you expected, please
<STRONG>contact the administrator of the site involved.</STRONG>
(Try sending mail to <SAMP><Webmaster@<EM>domain</EM>></SAMP>.)
Although this site is
running the Apache software it almost certainly has no other connection
to the Apache Group, so please do not send mail about this site or its
contents to the Apache authors. If you do, your message will be
<STRONG><BIG>ignored</BIG></STRONG>.
</BLOCKQUOTE>
<HR>
<P>
The Apache
<A
HREF="manual/index.html"
>documentation</A>
has been included with this distribution.
</P>
<P>
The Webmaster of this site is free to use the image below on
an Apache-powered Web server. Thanks for using Apache!
</P>
<DIV ALIGN="CENTER">
<IMG SRC="apache_pb.gif" ALT="">
</DIV>
</BODY>
</HTML>
]
READ response data=[]
1 url processed, 0 responses timed out
Processing results to "log"
Level 2
Level 2 displays the actual URL string that is sent to the Web server and the type of operation, for example, GET, SEND, POST.
The following is an example of the hammer command using Level 2 trace and the corresponding trace text, with bold text representing actual text received from the Web server:
iesun8 > hammer -v 2 -n 1 -u 1 -p 80 -r edtcsun33 -o log scenario_get
Trace level=2
Parsing scenario file "scenario_get"
Started 1 users, start staggered by 10 msec, divided workload, each user
executes 1 URLs
Please wait while workload completes...
Starting user number [0]
WRITE request data=[GET /testdir/test1.html]
1 url processed, 0 responses timed out
Processing results to "log"
Level 4
Level 4 displays the number of bytes read from the Web server in response to a URL request being sent to the Web server.
The following is an example of the hammer command using Level 4 trace and the corresponding trace text, with bold text representing the number of bytes received from the Web server:
iesun8 > hammer -v 4 -n 1 -u 1 -p 80 -r edtcsun33 -o log scenario_get
Trace level=4
Parsing scenario file "scenario_get"
Started 1 users, start staggered by 10 msec, divided workload, each user
executes 1 URLs
Please wait while workload completes...
Starting user number [0]
READ response bytes=[1622]
READ response bytes=[0]
1 url processed, 0 responses timed out
Processing results to "log"
Level 8
Level 8 trace information indicates whether URLs are being submitted too fast or too slow for the required throughput. Throughput information is displayed only when the -v 8 option and -t option are used together and information is displayed only when the throughput is either faster or slower than the desired throughput.
Two examples of the hammer command using Level 8 trace and the corresponding trace text is displayed below.
The first example shows a throughput of -t 1 (submit 1 URL/sec has been specified). The information shows that WebHammer needs to wait or slow itself down so that the required throughput is obtained.
iesun8 >hammer -t 1 -v 8 -n 10 -u 1 -p 80 -r edtcsun33 -o log scenario_get
Trace level=8
Parsing scenario file "scenario_get"
Started 1 users, start staggered by 10 msec, divided workload, each user
executes 10 URLs
Please wait while workload completes...
Starting user number [0]
Actual URL throughput exceeding desired pace, need to wait: 0 sec 996 msec
Actual URL throughput exceeding desired pace, need to wait: 0 sec 996 msec
Actual URL throughput exceeding desired pace, need to wait: 0 sec 996 msec
Actual URL throughput exceeding desired pace, need to wait: 0 sec 996 msec
Actual URL throughput exceeding desired pace, need to wait: 0 sec 996 msec
Actual URL throughput exceeding desired pace, need to wait: 0 sec 996 msec
Actual URL throughput exceeding desired pace, need to wait: 0 sec 996 msec
Actual URL throughput exceeding desired pace, need to wait: 0 sec 996 msec
Actual URL throughput exceeding desired pace, need to wait: 0 sec 996 msec
Actual URL throughput exceeding desired pace, need to wait: 0 sec 996 msec
10 url processed, 0 responses timed out
Processing results to "log"
The second example shows a throughput of -t 350 submitting 350 URLs per second. The information shows that WebHammer cannot keep up with the throughput and the rate at which it is falling behind. The throughput information has been placed in bold text for emphasis.
iesun8 > hammer -t 350 -v 8 -n 10 -u 1 -p 80 -r edtcsun33 -o log scenario_get
Trace level=8
Parsing scenario file "scenario_get"
Started 1 users, start staggered by 10 msec, divided workload, each user
executes 10 URLs
Please wait while workload completes...
Starting user number [0]
URL throughput falling behind desired pace by a factor of 1.32
URL throughput falling behind desired pace by a factor of 1.45
URL throughput falling behind desired pace by a factor of 1.20
URL throughput falling behind desired pace by a factor of 1.25
URL throughput falling behind desired pace by a factor of 1.26
10 url processed, 0 responses timed out
Processing results to "log"
Level 16
Level 16 displays general trace information, such as the following:
- number of URLS to be processed
- information on threads spawned
- when a new TCP/IP connection is established
The following is an example of the hammer command using Level 16 trace and the corresponding trace text:
hammer -v 16 -n 1 -u 1 -p 80 -r edtcsun33 -o log scenario_get
Trace level=16
scenario_file=[scenario_get]
log_file=[log]
Parsing scenario file "scenario_get"
urlcount=[1]
URL=[GET /testdir/test1.html]
Spawning threads
Started 1 users, start staggered by 10 msec, divided workload, each user
executes 1 URLs
tid=5
Please wait while workload completes...
Starting user number [0]
Connection fd=4
1 url processed, 0 responses timed out
Processing results to "log"
Bookmark
