How to enable OAuth using Oracle Endeca?

Purpose

In this tutorial, you will learn how to implement OAuth using Oracle Endeca, and how to create to create a sample graph to fetch tweets from Twitter. This tutorial is based on EID 3.0 using clover etl v3.3.x.

Time to Complete

Approximately 25 minutes

Overview

Authorization is mostly commonly used information security when it comes to access rights on any information. OAuth is one such open standard implementation, most know popular websites such as Twitter, Google, LinkedIn, and Flickr. It allows client access to the information using REST API. This tutorial illustrates how to implement OAuth. It is just an alternative and not a best practice. It is recommended you understand how to implement OAuth on any client application. This is detailed in http://oauth.net/. For our case we will consider the use of client credentials and token credentials and 2-legged process.

Getting Started

In this example, let’s consider the Twitter REST API 1.1 to demonstrate how Oracle Endeca can be used to fetch tweets using Endeca integrator and OAuth protocol. Twitter search api will be used to fetch tweets, with the search term “guardianangel”.

  1. As a first step prepare the API URL. The URL should consider using search API 1.1, with result in JSON format, and with the query “guradianangel”. Various other parameters such as geocode, lang, locale, count, etc can also be added to make search linear. Please refer to https://dev.Twitter.com/ on usage.

Prepared URL: https://api.Twitter.com/1.1/search/tweets.json?q=guardianangel.

  1. On your web browser, open the Twitter home page (https://Twitter.com) and register for a new account if you don’t already have one. Go to Twitter developers page (https://dev.Twitter.com/apps) and sign in with your credentials. Create a new sample application on MY Application page. The new sample application will have the OAuth information as shown in fig 1. This will be used in integrator to create client application that interacts with REST API. To understand how Twitter authorizes api using OAuth information please refer the interaction information here: https://dev.Twitter.com/docs/auth/implementing-sign-Twitter.

Figure 1: The OAuth settings page of Twitter application. The information above is private to one’s profile.

  1. Open Endeca Integrator as a next step in building graph. In the new graph, add DataGenerator and Reformat component. We will use the data generator to input the consumerKey, consumerSecret, accessToken, accessTokenSecret and the API url.

Figure 2: DataGenerator and Reformat component in the integrator with user defined metadata.

  1. Create an edge from the DataGenerator component to the Reformat component. On the edge, select and create new user defined metadata with fields below and all other to default values, as shown in fig 3.

Figure 3: User defined metadata that transforms data from DataGenerator to Reformat component.

  1. Edit the DataGenerator component, click on the generator property, and then on Source tab. Input the following information from your Twitter apps, as shown in fig 4.

Figure 4: Source code used in the DataGenerator.

  1. In DataGenerator component, set the property “Number of records to generate” to value 1, as shown in fig 5.

Figure 5: Edit tab of the DataGenerator component in the integrator.

  1. In the next step edit the Reformat component, and click on the Transform property. On the transformation tab to the extreme right top click “Convert to Java” (as shown in fig 6), and input the below java code.

Figure 6: This option is available in the Transformation tab of edit Reformat, Transform property.

import com.opensys.cloveretl.ctl.CTLUtils;

import static com.opensys.cloveretl.ctl.CTLUtils.DECIMAL_MAX_PRECISION;


import java.io.IOException;

import java.net.HttpURLConnection;

import java.net.URL;


import org.jetel.exception.ComponentNotReadyException;


import oauth.signpost.OAuth;

import oauth.signpost.OAuthConsumer;


import oauth.signpost.basic.DefaultOAuthConsumer;


public final class OAuthHeaderCreator extends org.jetel.component.CTLRecordTransform {


/**Code from CTL function globalScopeInit()*/

public final void globalScopeInit() throws org.jetel.exception.ComponentNotReadyException {

}


/**Code from CTL function transform()*/

protected final int transformDelegate() throws org.jetel.exception.TransformException {

try {

OAuthConsumer consumer2 = new DefaultOAuthConsumer(CTLUtils.toString(getInputRecord(0).getField(0).getValue()), CTLUtils.toString(getInputRecord(0).getField(1).getValue()));

consumer2.setTokenWithSecret(

CTLUtils.toString(getInputRecord(0).getField(2).getValue()),

CTLUtils.toString(getInputRecord(0).getField(3).getValue()));

URL url2 = new URL(CTLUtils.toString(getInputRecord(0).getField(4).getValue()));

System.setProperty("http.proxyHost", "www-proxy.us.oracle.com");

System.setProperty("http.proxyPort", "80");

HttpURLConnection conn3 = new HttpURLConnection(url2) {

@Override

public void connect() throws IOException {}

@Override

public boolean usingProxy() {return false;}

@Override

public void disconnect() {}

@Override

public void setRequestProperty(String key, String value) {

if(key==OAuth.HTTP_AUTHORIZATION_HEADER) {

getOutputRecord(0).getField(1).setValue(value);

}

super.setRequestProperty(key, value);

}

};

consumer2.sign(conn3);


String string = CTLUtils.toString(getInputRecord(0).getField(4).getValue());

String[] parts = string.split(":");

String part1 = parts[0];

String part2 = parts[1];

getOutputRecord(0).getField(0).setValue(part1+":(http://www-proxy.us.oracle.com)"+part2);


return 2147483647;


} catch (Exception e) {

return 2147483647;

}

}

}


  1. The above code completes the remaining parameters in OAuth header which are required for authorization such as oauth_nonce, oauth_signature, oauth_singature_method. The code also uses proxy www-proxy.us.oracle.com and port 80 which should be modified depending on the network connection.

  2. As a last step in adding components to the graph, add the HTTP connector and map all the components as shown in fig 8. The Trash component can optionally be replaced with JSON reader component to process them to Endeca server which is not part of this example. The user defined metadata that transforms data from Reformat to Http connector is shown in fig 7.

Figure 7: User defined metadata that transforms data from Reformat to Http connector component.

Figure 8: Components mapped in a graph in integrator.

  1. Now edit the HTTP connector and set the method to ‘GET’, and input the additional HTTP header with ‘Authorization’ and empty value (Fig 9). Select any appropriate output file in the property, and in the input mapping complete the map as shown in fig 10.

Figure 9: The Edit tab of HTTP Connector component in integrator

Figure 10: Inputmapping of the HTTP connector in the integrator.

  1. Run the graph and view the output in the json format such as in fig 11. Once you have the data in json format, the data can be considered for loading into Oracle Endeca server which is not part of this example.

Figure 11: The output file containing the tweets in json format.


Summary

This tutorial has covered how OAuth can be implemented using Oracle Endeca integrator, and to use Twitter REST API 1.1 to fetch tweets.

Credits

Kapil Angamuthu, Lubos Imriska, Rajith Raghu




Copyright © 2012, Oracle and/or its affiliates. All rights reserved