Java SE 8: Using Threads in an HTTP Link Checker

In this section, you use the Java SE 8 regular expression to validate the correctness of the URL format. The HTTPClient.java file has a verifyUrl method, which accepts the URL as the input parameter.

1. Import the following packages:

import java.net.HttpURLConnection;
import java.net.URL;
import java.util.logging.Level;
import java.util.logging.Logger;
import java.util.regex.Matcher;
import java.util.regex.Pattern;


2. Implement the Runnable interface in the HTTPClient.java file.
   
   public class HTTPClient implements Runnable{

   The Runnable interface is implemented by HTTPClient.java, whose instance is executed by a thread. The class defines a method of no arguments called run.

3. Add the following method to the HTTPClient.java file to verify the URL format:

   private boolean verifyUrl(String url) {
        String urlRegex = "^(http|https)://[-a-zA-Z0-9+&@#/%?=~_|,!:.;]*[-a-zA-Z0-9+@#/%=&_|]";
        Pattern pattern = Pattern.compile(urlRegex);
        Matcher m = pattern.matcher(url);
        return m.matches();
 }


The verifyUrl method verifies the url parameter passed as the input parameter by matching it with the regular expression. If the match is successful, then it returns true; otherwise, it returns false.

4. Review the code. Your code should look like the following:

    
   

In this section, you validate the URLs listed in the url-list.txt file.

1. Modify HTTPClient.java and initialize it by using a constructor.

   private final String urlToBevalidated;

public HTTPClient(String url) {
this.urlToBevalidated = url;
}

2. Invoke the verifyUrl method in the validateUrl() method.

   public void validateUrl(String url) throws Exception {
    if (verifyUrl(url)) {
        try {

   The validateUrl method receives the URL as an input parameter. In the if condition, you verify the URL by passing url as an input parameter to the verifyUrl method. The verifyUrl method returns true for a valid URL format; otherwise, it returns false. If the verifyUrl method returns true, then the if condition is true, and the code is executed in the try block.
   

3. Create the HttpURLConnection connection.

URL myURL = new URL(url);
HttpURLConnection myConnection = (HttpURLConnection) myURL.openConnection();


Next, you create a myURL instance and open the connection with the HttpURLConnection class.

4. Validate the URL with the response code.

   if (myConnection.getResponseCode() == URLStatus.HTTP_OK.getStatusCode()) {
         Results.getInstance().appendSuceeded("\n" + url + "****** Status message is : "
         + URLStatus.getStatusMessageForStatusCode(myConnection.getResponseCode()));
       } else {
        Results.getInstance().appendFailed("\n" + url + "****** Status message is : "
         + URLStatus.getStatusMessageForStatusCode(myConnection.getResponseCode()));
       }


   The myConnection instance receives the URL's response code and verifies the status. If the status code is 200 (HTTP_OK), then the URL is classified as succeededURLS; otherwise, it's classified as failedURLS.

5. Close try with the catch block.

   } catch (Exception e) {
          Results.getInstance().appendFailed("\n" + url + "****** Status message is : " + e.getMessage());
  }

   The catch block is executed when an exception is thrown while  HttpURLConnection is created or opened.

6. Verify the incorrect URLs.

   }else {
        Results.getInstance().appendIncorrect("\n" + url);
      }
  }


   The else block is executed when the verifyUrl method returns false because the URL validation failed.

7. Implement the run()method.

   @Override
public void run() {
    System.out.println("Started thread " + Thread.currentThread().getName() + " Verifying URL " + this.urlToBevalidated);
    try {
            this.validateUrl(this.urlToBevalidated);
     } catch (Exception e) {
            Logger.getLogger(HTTPClient.class.getName()).log(Level.SEVERE, null, e);
     }
     Results.getInstance().latchCountDown();
   }
}


   The executor.execute method calls the run()method defined in HTTPClient.java.

8. Review the code. Your code should look like the following:

    
   

In this section, you create a Java class named HTTPClientTest.java, which includes the main() method where the execution starts.

1. Create HTTPClientTest.java and import the following packages:

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.List;
import java.util.concurrent.CopyOnWriteArrayList;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.logging.Level;
import java.util.logging.Logger;


2. Add the following code to the main()method:

   public class HTTPClientTest {
  public static void main(String[] args) {
   
    ExecutorService executor = Executors.newFixedThreadPool(5);
    List<String> urlList = new CopyOnWriteArrayList<>();
    try {
      urlList = Files.readAllLines(Paths.get("src/url-list.txt"));
    } catch (IOException ex) {
      Logger.getLogger(HTTPClient.class.getName()).log(Level.SEVERE, null, ex);
    }
   
    Results.getInstance().setLatch(new CountDownLatch(urlList.size()));
    urlList.forEach(url -> executor.execute(new HTTPClient(url)));
    try{
      Results.getInstance().getLatch().await();
    } catch (InterruptedException ex){
      System.out.println("Interrupted: " + ex.getMessage());
    } finally {
      if (!executor.isShutdown()) {
        executor.shutdown();
      }
    }  
    System.out.println("\n------------------------------------------------\n");
    System.out.println("***** Invalid URLS: *****");
    System.out.println(Results.getInstance().getIncorrect());
    System.out.println("\n------------------------------------------------\n\n");
    System.out.println("** Valid URLS that have successfully connected : **");
    System.out.println(Results.getInstance().getSuceeded());
    System.out.println("\n------------------------------------------------\n\n");
    System.out.println("** Broken URLS that did not successfully connect : **");
    System.out.println(Results.getInstance().getFailed());
  }
}

   In the main()method, ExecutorService uses the executor instance through which threads in the thread pool execute the URL validation. The Executors class creates a thread pool with five threads.
   The CopyOnWriteArrayList()method creates an empty list and copies the URLs stored in the file. The CountDownLatch()method retrieves the size of the urlList and sets that value to the latch variable.
   
   In this application, you create a urlList collection of type string where you store the URLs in url-list.txt. You launch a thread for each URL through the For-Each with lambda expression:
   urlList.forEach(url -> executor.execute(new HTTPClient(url)));
   
   A new thread is spawned and validates each URL. If the number of URLs is more than the number of threads in the thread pool, then executor waits until a thread is available to validate the next URL.
   
   The lambda expression uses the Consumer functional interface. This interface has the abstract accept(T t) method, which accepts an object and doesn't return anything.
   Alternatively, you could replace the original lambda expression with one assigned to a variable, as follows, and also import the import java.util.function.Consumer; package.
   
   Consumer <string> processItem = (String url) -> executor.execute(new HTTPClient(url));
   urlList.forEach(processItem);
   
   The await() method causes the current thread to wait until the latch counts down to zero. If it's interrupted during wait, then it throws InterruptedException; otherwise, it displays the invalid URLs, the valid URLs that connected, the broken URLs that didn't connect, and the status code of all URLs. The shutdown() method allows previously submitted tasks to execute before terminating, and then ExecutorService shuts down.

3. Review the code. Your code should look like the following:

    
   

In this section, you run the application by passing a set of URLs as input, and then verify and validate the URLs. Depending on the correctness of the URL, they will be classified and displayed in the console.

1. On the Projects tab, right-click HTTPClientTest.java and select Run File.



2. Review the set of URLs displayed in the url-list.txt file. 



For the given set of URLs, the application retrieves each URL, verifies it, validates it, and classifies it accordingly.

3. Verify the output.



You successfully used the URLs listed in the url-list.txt file and classified them as valid, broken, and incorrect URLs. The status codes of the broken URLs are displayed in the console. You also see the assignment of each URL to a random thread chosen among five threads present in the thread pool.