Oracle Storage Cloud Service: Uploading Files Larger than 5GB Using the Java Library

Purpose

This tutorial provides steps to upload files larger than 5GB to Oracle Storage Cloud Service using a Java library.

This tutorial was developed in Oracle JDeveloper, but you can use any Java IDE or text editor.

Time to Complete

Approximately 1 hour

Introduction

The Oracle Storage Cloud Service enables businesses to store and manage digital content in the cloud. Stored objects can be retrieved directly by external clients or by applications running within Oracle Cloud Service instances.

The Oracle Storage Cloud Service can support objects of virtually any size, but there is a maximum upload size of 5GB. To upload an Object larger than 5GB, the file must first be locally segmented into multiple files smaller than 5GB.

You can access Oracle Storage Cloud Service through a REST API or through an official Java library. This tutorial shows how to upload files larger than 5GB to the Oracle Storage Cloud Service by using the Java library provided in the Oracle Storage Cloud Service Java SDK.

For a simpler tutorial that uses FTM API to upload files larger than 5 GB to Oracle Storage Cloud Service, see Uploading Files Larger than 5 GB Using the File Transfer Manager API.

Scenario

To upload an object larger than 5GB, the file must first be locally segmented into multiple files smaller than 5GB. The individual segments can then be uploaded individually. After all segments have been uploaded, the Oracle Storage Cloud Service must be told to create a new object from the previously uploaded segments.

In this tutorial, you will:

• Create a Java program that will:
• Upload individual file segments
• Map the uploaded file segments together into a single object

This tutorial will not cover how to:

• Segment a single file into sequential segments 
• Calculate the MD5 checksum of a file

Software Requirements

• Access to Oracle  Storage Cloud Service
• A text editor or a Java IDE such as Oracle JDeveloper
• One of the supported browsers listed in the following table:



Browser	Version
Internet Explorer	10 (Desktop), 9, 8
Firefox	Latest two production releases
Chrome	Current and previous Chrome versions at time of certification
Safari	6 or later
Safari Mobile	IOS 6 native Safari, certified on iPad 3
Android browser	4.0 or later

Prerequisites

Before starting this tutorial, you should:

• Have familiarity with the Java programming language, specifically experience with reading and writing files to disk and generating MD5 checksums.
• Have familiarity with implementing your own Java classes that implement a given Java interface.
• Have downloaded and unpacked the Oracle Storage Cloud Service Java SDK. The SDK contains the Jar files of the Java Client. These Jars should be available on your classpath. Click here to download this SDK.  
• Have a file to upload. It does not have to be larger than 5GB. A file of any size will work.
• Create a container in which to store objects. Below, we have used myfiles-java as an example.
• Have familiarity with Oracle Cloud services. 

Before starting this tutorial, consider a typical workflow for Oracle Storage Cloud Service. The focus of this tutorial is on the fifth step.

1. Define users.

2. Define roles.

3. Create containers.

4. Assign users specific access privileges to specific containers by using Access Control Lists.

5. Upload objects (the focus of this tutorial is uploading files larger than 5GB).

6. Download objects.

This tutorial is one of two tutorials about uploading files larger than 5GB to the Oracle Storage Cloud Service. Another tutorial, Oracle Storage Cloud Service: Uploading Files Larger than 5GB Using the REST API, shows how to perform the same tasks using the cURL editor with the REST API.

Each of the following topics of this tutorial examines a section of the Java program; these sections are not intended to be executed separately. The entire executable program is displayed at the end of the tutorial, in the "Examining the Full Example Java Program" section. You can test this tutorial's functionality by substituting your credentials, connection details, and data files in the example code at the end of the tutorial.

After downloading the Oracle Storage Cloud Service Java Client SDK, the Java program must import the following classes into your Java program:

              import oracle.cloud.storage.*;
              import oracle.cloud.storage.model.*;
              import oracle.cloud.storage.exception.*;

For this tutorial's example, the Java program must also import these Java packages:

              import java.io.*;
              import java.util.*;
              import java.net.*;

• The local file must be split into sequential segments.
• The sequential segments must be alphabetically sortable. The sequential segments cannot be numerically sortable.
  • For example:
    • This works:
      • Original File: hello-world.txt
      • Segments: hello-world-a, hello-world-b, … hello-world-f
    • This doesn't work:
      • Original File: hello-world.txt
      • Segments: hello-world-1, hello-world-2, … hello-world-6
• The sequential segments must share the same prefix. A segment name's only differing quality should be its ordering suffix.
• The sequential segments must concatenate to exactly match the original file.
• Splitting a local file into sequential segments with the Java programming language is outside the scope of this tutorial.
  • There are numerous ways to accomplish this from the Java programming language, with each way providing its own benefits.
• For demonstration purposes this tutorial assumes that a separate class which implements the LocaFileSegmenter interface exists.
  • The LocalFileSegmenter interface is defined as:

import java.io.*;

import java.util.*;

public interface LocalFileSegmenter {

    List<File> segmentFile(File localFile,
int segmentSize, String segmentPrefix);

}          

• This is just one method for segmenting a local file. Other methods can be used, provided the segmentation rules are followed.
  

• Using your own implementation of the LocalFileSegmenter interface, segment the local file and print the details of the generated segments:

String prefix = "hello-world-segment-";
LocalFileSegmenter myFileSegmenter = new SampleFileSegmenter();
List<File> listOfSegments = myFileSegmenter.segmentFile(new File("./hello-world.txt"), 2048, prefix);
              
for (File segment : listOfSegments) {
    System.out.println("Segment Name: " + segment.getName() + 
                       "\t Segment Size: " + segment.length());
}

1. Returning to the Java program illustrated at the end of this tutorial, the program creates a connection to your Oracle Storage Cloud Service instance.

CloudStorageConfig myConfig = new CloudStorageConfig(); 
myConfig.setServiceName("myService-myIdentityDomain")                
              
  .setUsername("john.doe@oracle.com")
  .setPassword("xUs8M8rw".toCharArray()) 
  .setServiceUrl("https://storage.us2.oraclecloud.com"); 

               
CloudStorage myConnection = CloudStorageFactory.getStorage(myConfig); 

To identify the service name and service URL (for the setServiceName and setServiceUrl methods)  of your Oracle Storage Cloud Service instance:

• Sign in to the Oracle Cloud My Services application. The My Services Dashboard is displayed. It lists the services that are assigned to your account.
• Look for Oracle Storage Cloud Service.
• From the Actions menu, select View Details. Alternatively, click the Oracle Storage Cloud Service link in the Dashboard page.

On the resulting page, look for the REST Endpoint field. You can see the details of the service name and service URL (for the setServiceName and setServiceUrl methods) of your Oracle Storage Cloud Service instance, as highlighted in the following screenshot.

1. Each of the uploaded segments need to be in the same Container, have the same prefix, and be sequentially ordered. We have already created the sequential segments and named them correctly.

   FileInputStream segmentFis;  
   
   for (File segment : listOfSegments) {
       segmentFis = new FileInputStream(segment);
       myConnection.storeObject("myfiles-java", segment.getName(), "text/plain", segmentFis);
       System.out.println("Uploaded Segment: " + segment.getName());
       segmentFis.close();
   }

1. Once all the segments have been uploaded to the same Container (myfiles-java), the Java program creates a special type of object called a Manifest Object which will map the sequential segments into a single Object named hello-world.txt.

   myConnection.storeObjectManifest("myfiles-java", "hello-world.txt", "text/plain", "myfiles-java", prefix);
   System.out.println("\nUploaded Segments have been mapped together");
   

1. Now that the Manifest Object has been created successfully, the Java program downloads it and saves it as a local file. When downloading the Manifest Object, the Oracle Storage Cloud Service will concatenate all of the sequential segments together in the HTTP response. This will allow us to verify that the concatenated file is identical to the original file.

   StorageInputStream manifestObjectFis = myConnection.retrieveObject("myfiles-java","hello-world.txt");
   FileOutputStream manifestObjectFos = new FileOutputStream("hello-world-concatenated.txt");
   
   byte[] buffer = new byte[10*1024];
   
   int bytesRead;
   while ((bytesRead = manifestObjectFis.read(buffer)) != -1) {
       manifestObjectFos.write(buffer, 0, bytesRead);
   }
   manifestObjectFos.close();
   
   System.out.println("\nManifest Object has been downloaded");

• Oracle Storage Cloud Service identifies Objects using an MD5 checksum of the Object's contents. Two Objects are considered identical if their MD5 checksums are identical.
• In order to verify that our concatenated file is identical to our original local file, we must calculate the MD5 hash of both the local file and the downloaded concatenated file.
• Calculating the MD5 checksum of a local file in the Java programming language is outside the scope of this tutorial.
• For demonstration purposes this tutorial assumes that you have written your own class that implements the following interface:

  import java.io.*;
  
  public interface FileMd5Calculator {
      public String calculate(File localFile);
  } 

• Using your own implementation of the FileMd5Calculator interface, calculate the MD5 checksums of the original local file and the downloaded concatenated file:
  

  
  FileMd5Calculator myMd5Calculator = new SampleFileMd5Calculator();
  
  String originalFileMd5 = myMd5Calculator.calculate(new File("hello-world.txt"));
  String concatenatedFileMd5 = myMd5Calculator.calculate(new File("hello-world-concatenated.txt"));
                    
  System.out.println("\nThe original file's MD5 is: " + originalFileMd5);
  System.out.println("The concatenated file's MD5 is: " + concatenatedFileMd5);              

• Compare the original file's MD5 checksum with the downloaded concatenated file's checksum. They should be equal.
  

  
  if (originalFileMd5.equals(concatenatedFileMd5)) {
      System.out.println("\nSUCCESS: The original file and the concatenated file are identical!");
  } else {
      System.out.println("\nERROR: The original file and the concatenated file are NOT identical!");
  }

Here is the full source code for the Java program. You can swap in your own credentials, connection details, and data files to test this functionality:

import oracle.cloud.storage.*;
import oracle.cloud.storage.model.*;
import oracle.cloud.storage.exception.*;  
import java.io.*;
import java.util.*;
import java.net.*;  

public class UploadingSegmentedObjects {

    public static void main(String[] args) {
        try {
            String prefix = "hello-world-segment-";
            LocalFileSegmenter myFileSegmenter = new SampleFileSegmenter();
            List<File> listOfSegments = myFileSegmenter.segmentFile(new File("./hello-world.txt"), 2048, prefix);

            for (File segment : listOfSegments) {
                System.out.println("Segment Name: " + segment.getName() + 
                                   "\t Segment Size: " + segment.length());
            }
            CloudStorageConfig myConfig = new CloudStorageConfig();

            myConfig.setServiceName("myService-myIdentityDomain")
                .setUsername("john.doe@oracle.com")
                .setPassword("xUs8M8rw".toCharArray())
                .setServiceUrl("https://storage.us2.oraclecloud.com");

            CloudStorage myConnection = CloudStorageFactory.getStorage(myConfig);


            System.out.println("\nUploading Segments ...\n");

            FileInputStream segmentFis;  

            for (File segment : listOfSegments) {
                segmentFis = new FileInputStream(segment);
                myConnection.storeObject("myfiles-java", segment.getName(), "text/plain", segmentFis);
                System.out.println("Uploaded Segment: " + segment.getName());
                segmentFis.close();
            }
            myConnection.storeObjectManifest("myfiles-java", "hello-world.txt", "text/plain", "myfiles-java", prefix);
            System.out.println("\nUploaded Segments have been mapped together");
            StorageInputStream manifestObjectFis = myConnection.retrieveObject("myfiles-java","hello-world.txt");
            FileOutputStream manifestObjectFos = new FileOutputStream("hello-world-concatenated.txt");

            byte[] buffer = new byte[10*1024];

            int bytesRead;
            while ((bytesRead = manifestObjectFis.read(buffer)) != -1) {
                manifestObjectFos.write(buffer, 0, bytesRead);
            }
            manifestObjectFos.close();

            System.out.println("\nManifest Object has been downloaded");
            FileMd5Calculator myMd5Calculator = new SampleFileMd5Calculator();

            String originalFileMd5 = myMd5Calculator.calculate(new File("hello-world.txt"));
            String concatenatedFileMd5 = myMd5Calculator.calculate(new File("hello-world-concatenated.txt"));

            System.out.println("\nThe original file's MD5 is: " + originalFileMd5);
            System.out.println("The concatenated file's MD5 is: " + concatenatedFileMd5);
            if (originalFileMd5.equals(concatenatedFileMd5)) {
                System.out.println("\nSUCCESS: The original file and the concatenated file are identical!");
            } else {
                System.out.println("\nERROR: The original file and the concatenated file are NOT identical!");
            }
        } catch (Exception e) {
            e.printStackTrace();
        }        
    }
}

*******************************************

In this tutorial, you learned to:

• Create a Java program that will:
• Upload individual file segments
• Map the uploaded file segments together into a single object

Resources

• See Uploading Files Larger than 5 GB Using the File Transfer Manager API OBE

Credits

• Lead Curriculum Developer: Richard Green
• Key Contributor: Rehan Iftikhar