Handling Chunked Requests in Sun Java System Web Server 7.0

In the HTTP/1.0 protocol, the length of a message body is determined by the content-length header. A key difference between HTTP/1.0 and HTTP/1.1 is the introduction of chunked transfer coding in the latter. With that coding, clients that understand HTTP/1.1 can break a message body into chunks of arbitrary lengths through the header Transfer-Encoding: Chunked, which specifies that the message contains a chunked body. HTTP/1.1 clients can take advantage of chunking in these scenarios:

• When the request data length is generated by a dynamic script and therefore not known in advance
  
  
• When the request data is too long for convenient transfer and can be broken into smaller chunks of arbitrary lengths

Typically, chunking is used in responses but sometimes in requests, too. This article explains how Sun Java System Web Server 7.0 (henceforth, Web Server) handles chunked requests through Netscape Server API (NSAPI) plug-ins.

A properly formatted chunked request from a client contains chunked borders (size indicators) a data chunk with the length of the chunk. To mark the end of the message, the last chunk has a zero-length chunked border. Since all HTTP/1.1 requests with a message body must contain either a content-length or a transfer-encoding header, a properly formatted chunked request always contains a transfer-encoding header.

Here is an example of a client request with chunked transfer coding:

PUT /dav/file.html HTTP/1.1
Content-type: text/xml; charset="utf-8"
Host: host.domain
Transfer-encoding: chunked
Connection: close

5
abcde
0

For more information on chunked transfer coding, see the section Chunked Transfer Coding in the Hypertext Transfer Protocol--HTTP/1.1 specification.

To handle chunked requests, Web Server allocates a buffer whose size is specified in the max-unchunk-size parameter of the http element in the server.xml file. The default value of that parameter is 8,192 bytes with a maximum of 2 Gbytes.

When it receives a request with a chunked message body, Web Server allocates a buffer whose size is as specified in max-unchunk-size. If the message body size is less than the limit specified, Web Server performs the following tasks:

1. Deletes the chunked borders.
   
   
2. Reads the request body into the buffer—that is, the sn->inbuf data structure—and calculates a content length.
   
   
3. Sets the transfer-encoding header to identity in rq->headers.

If you send a message whose body size exceeds the limit specified, Web Server cancels the request and throws a 411 error to alert you that it cannot fit the entire request into the buffer. As a resolution, edit server.xml file and specify a larger limit for max-unchunk-size.

On occasion, some message bodies might exceed the 2-Gbyte maximum limit for max-unchunk-size. In that case, set max-unchunk-size to 0. Web Server then performs two tasks:

1. Deletes the chunked borders.
   
   
2. Sets the transfer-encoding header to identity in the rq->headers.

Web Server does not calculate the content-length header, hence the Server Application Function (SAF) must read the data from sn->inbuf until the end of the file (EOF).

Web Server 6.1 and later releases support input filtering. While processing the internal filter, Web Server deletes the chunked borders, so the SAF does not see them but sees only the request data.

Note: You can override max-unchunk-size for a particular SAF with the ChunkedRequestBufferSize obj.conf parameter.

To verify whether a message body exists, Service SAFs or input filter plug-ins check whether a content-length or transfer-encoding parameter is present in rq->headers.

Note: HTTP header names are case-insensitive. However, header names in NSAPI data structures are case-sensitive and are stored in those structures in lowercase by default.

If either parameter is present, the entity body is available in the sn->inbuf data structure. If Web Server sets a content-length header, then the plug-in reads the content-length bytes from sn->inbuf. However, if the transfer-encoding header is set with no content length, the plug-in must read the entity body until EOF.

If a plug-in works as a reverse proxy to an application server and forms a new HTTP message to a back-end application server, then the plug-in can calculate the content length and generate a content-length header to pass in the request to the application server. Alternatively, the plug-in can choose to apply transfer coding and generate a transfer-encoding header of its own to pass back to the application server.

Note: Because transfer-encoding is a hop-by-hop header, any plug-in that serves as a reverse proxy to an application server must not automatically pass that header to the application server.

You can edit the value for the max-unchunk-size parameter with the wadm command set-http-prop. The following example sets max-unchunk-size to 16,384 bytes.

wadm> get-http-prop -config=test
...
max-unchunk-size=8192
...

wadm> set-http-prop --config=test max-unchunk-size=16384
CLI201 Command 'set-http-prop' ran successfully
wadm> deploy-config test
CLI201 Command 'deploy-config' ran successfully

Take a look at this sample NSAPI plug-in, which processes request data. The plug-in checks for the content-length and transfer-encoding request headers. If the content-length header exists, the plug-in reads the content-length bytes from sn->inbuf. Otherwise, it reads the data from sn->inbuf until EOF.

To run the sample plug-in:

1. Compile the plug-in.
   
   For example, on the Solaris platform, type:
   
   % cc -G -I Web_Server_install_dir /include -c source_filename -o libreadpost.so
   
   
2. Copy the sample library to the plug-in directory, for example, Web_Server_install_dir /plugins/.
   
   
3. Configure Web Server:
   
   
   1. Add the following line at the end of the magnus.conf file:
      
      

      Init fn="load-modules" shlib="libreadpost.so" funcs="read_post"

       
      
   2. Add the following lines to the obj.conf file:
      
      

      <Object name="default"> Other code NameTrans fn=assign-name name="posttest" from="/posttest/*" Other code </Object> Other code <Object name="posttest"> Service fn="read_post" method="(POST)" </Object>

       
      

Finally, test the NSAPI plug-in. Here are the command and example output:

% telnet testhost.domain
POST /posttest/ HTTP/1.1
Host: testhost.domain
Transfer-Encoding: chunked
Connection: close

4
abcd
0

HTTP/1.1 200 OK
Server: Sun-Java-System-Web-Server/7.0
Date: Wed, 05 Sep 2007 02:25:08 GMT
Content-type: text/html
Connection: close

rq->headers contains : host="host.domain" transfer-encoding="identity"
accept="*/*" connection="close" content-length="4"
content-length in rq->headers : 4
transfer-encoding in rq->headers : identity
Trying to read content-length(4) bytes from sn->inbuf
Number of bytes read successfully = 4

• Sun Java System Web Server
  • Main page
  • Developer home
  • Software Forum
  
  
• Related documentation
  • Hypertext Transfer Protocol
  • NSAPI Developer's Guide
  • Administrator's Configuration File Reference
  • Optional Parameters for Service Directives ( ChunkedRequestBufferSize)