Create Supporting Artifacts for a Pipeline

Before You Begin

This 15-minute tutorial shows you how to create supporting artifacts for a Stream Analytics pipeline. This pipeline lets you track financial losses that are attributed to specific products from various suppliers. This tutorial takes approximately 15 minutes to complete.

In this series, you learn how to use Stream Analytics application for tracking financial losses from various product suppliers. This is the first tutorial in the Tracking Financial Losses from Various Suppliers by Using a Stream Analytics Application series. Read them sequentially.

• Creating Supporting Artifacts for a Pipeline
• Creating a Pipeline
  
• Publishing and Deploying a Pipeline

Access to Stream Analytics

To complete the tutorials of this course, you need access to Oracle Integration Cloud, which has the Stream Analytics included in it.

Background

Stream Analytics is a feature of Oracle Integration Cloud. It is an intuitive, web-based interface, powered by Spark Streaming and Apache Kafka Messaging runtime, and the interface enables you to explore, analyze, and manipulate streaming data sources in real time.

What Do You Need?

• Access to an instance of Stream Analytics
• Chrome browser
• Data and events pushed to Kafka
• Installed Java Runtime Environment and JDK (8 or later)
• Any SQL Client or IDE that supports a connection to Oracle Database such as SQL Plus
• Host and credential details to access the database
• The resourcesfortutorials ZIP file from the Product page.
• Keep the utility running throughout your work on this tutorial.
  

Push Sample Data

1. Extract the resourcesfortutorials.zip file to a folder on your machine.  The folder contains the following files:
   • BusEvents.json
   • SalesTransactions.json
   • OsacsEventFeeder.jar
   • suppliers.sql
2. To create and populate the SUPPLIERS table in your database, in SQL*Plus, run the suppliers.sql script. The sample data in this table is required to enrich the stream in this tutorial.
3. Open a command prompt on your machine.
4. Navigate to the location where you extracted the resourcesfortutorials.zip file.
5. To push events, run the following command:
   java -jar OsacsEventFeeder.jar -r <RESTURL> -j SalesTransactions.json -u <username> -p <password>
Note: RESTURL is the the REST Proxy endpoint URL for the Kafka topic that you provisioned in your Event Hub Service. See Provisioning Oracle Integration Cloud for reference. For example, the RESTURL may look like: https://<IPAddress>:1080/restproxy/topics/salestransactions. The topic name used in this tutorial is salestransactions. The following response confirms that data is being seeded to the salestransactions topic:
RESPONSE:{"offsets" : [{"partition" :0, "offset" :28, "error_code" :null, "error" :null}], "key_schema_id" :null, "value_schema_id" :null}

Create a Database Connection

1. Sign in to the Stream Analytics instance.
2. In the left navigation tree, click Catalog.
3. In the Create New Item menu, select Connection.
4. In the Create Connection dialog box, on the Type Properties tab, enter the following and click Next.
   • In the Name field, enter SuppliersDatabaseConnection.
   • In the Tags field, enter tutorial, retail.
   • From the Connection Type drop-down list, select Database.

   

• In the Connection Details section, enter the following details:
• Database service name: The service name for the database service name. (For example, XE)
• Host name: The host where the database resides and is running
• Port: The port where the database is running
• Username
• Password
5. Click Test Connection.
6. Click Save.

Create a Kafka Connection

1. In the left navigation tree, click Catalog.
2. From the Create New Item menu, select Connection.
   
3. In the Create Connection dialog box, do the following and click Next.
• In the Name field, enter SupplierAnalysis.
• In the Tags field, enter tutorial, retail.
• From the Connection Type dripSelect Kafka.
4. Enter the Zookeepers URL you have saved earlier at the time of provisioning.
5. Click Test Connection.
   
6. Click Save.
   

Create a Reference

1. Click Catalog.
2. From the Create New Item menu, select Reference.
   
3. In the Create Reference dialog box, do the following and click Next:
• In the Name field, enter ListOfSuppliers.
• In the Tags field, enter tutorial, retail.
• In the Reference Type drop-down list, select the Database Table.

  

4. On the Source Details tab, in the Connection drop-down list, select the SuppliersDatabaseConnection and click Next.
5. On the Shape tab, do the following and click Save.
   •  In the Shape Name drop-down list, select SUPPLIERS.
   • Change the UNITCOST type to Number.

     

Create a Stream

1. Click Catalog.
2. From the Create New Item menu, select Stream.
   

   

3. In the Create Stream dialog box, on the Type Properties tab, do the following and click Next.
• In the Name field, enter SalesTransactions.
• In the Tags field, enter tutorial, retail.
• From the Stream Type drop-down list, select Kafka.
4. On the Source Details tab, do the following and click Next:
• In the Connection field, select SupplierAnalysis.
• From the Topic name drop-down list, select salestransactions.
5. On the Shape tab, click Infer Shape. The fields are auto populated.

   

6. Click Save.
   

Create a Target

1. Click Catalog.
2. In the Create New Item menu, select Target.
   
3. In the Create Target dialog, do the following and click Next.
• In the Name field, enter SupplierAnalysis.
• the Tags field, enter tutorial, retail.
• From the Target Type drop-down list, select Kafka.
4. On the Target Details page, do the following information and click Next:
• In the Connection drop-down list, select SupplierAnalysis.
• In the Topic name field, enter supplieranalysis.
5. On the Shape page, click the Manual Shape option, enter the following, and click Save:

   Field Path	Field Type
   TxnTime	Text
   ProductLine	Text
   TxnLoss	Number
   TxnCost	Number
   TxnRevenue	Number
   ProductType	Text
   Product	Text
   QuantitySold	Number
   CountrySold	Text
   Channel	Text
   Supplier	Text

   

Next Tutorial

Creating a Pipeline