Java proxy http requests

Blog

How to Configure Proxy Authentication for Java HTTP Client

• Post author: Chinna
• Post published: March 6, 2023
• Post category: Java
• Post comments: 0 Comments

In our previous tutorial, we implemented a thin REST API client application to consume secured CRUD REST APIs. However, the client does not support proxy authentication. Therefore, we are going to configure the client with proxy authentication.

Introduction

Often, corporate networks provide internet access via proxy servers, and at times they require authentication as well. Hence, applications must do proxy authentication to connect to the internet.

What You’ll Build

A REST client that supports proxy authentication to consume REST APIs with JWT authentication.

What You’ll Need

• Spring Tool Suite or any IDE of your choice
• JDK 11
• MySQL Server
• Apache Maven

Configuring HTTP Client with Proxy Authentication

Let’s see how we can create an HTTP client v2 with proxy authentication in step-by-step.

1. Firstly, create an HTTP Client v2 builder with 10 seconds of connection timeout.
2. Secondly, If the proxy host and port are provided, then
   1. Create a ProxySelector using ProxySelector::of method. It provides a ProxySelector which uses a single proxy for all requests. The system-wide proxy selector can be retrieved by ProxySelector.getDefault() .
   2. Invoke the builder.proxy() method to use this ProxySelector . Note: If this method is not invoked prior to building, then newly built clients will use the default proxy selector, which is usually adequate for client applications. The default proxy selector supports a set of system properties related to proxy settings. This default behavior can be disabled by supplying an explicit proxy selector, such as NO_PROXY , or one returned by ProxySelector::of , before building.
   3. Get the default Authenticator which will be used for connection without authentication. If the proxy username and password are provided, then it means the proxy server requires authentication. Hence, In this case, we have to create a new Authenticator and override the getPasswordAuthentication() method to create a new PasswordAuthentication object using these credentials.
   4. Invoke the builder.authenticator() method to use this authenticator.

   public ProductManagerImpl(ProductManagerConfig config) < this.productManagerConfig = config; HttpClient.Builder builder = HttpClient.newBuilder().version(HttpClient.Version.HTTP_2) .connectTimeout(Duration.ofSeconds(10)); if (config.proxyHost() != null && config.proxyPort() != null) < builder.proxy(ProxySelector.of(new InetSocketAddress(config.proxyHost(), config.proxyPort()))); Authenticator authenticator = Authenticator.getDefault(); if (config.proxyUser() != null && config.proxyPass() != null) < authenticator = new Authenticator() < @Override protected PasswordAuthentication getPasswordAuthentication() < return new PasswordAuthentication(config.proxyUser(), config.proxyPass().toCharArray()); >>; > builder.authenticator(authenticator); > this.requestHelper = new HttpRequestHelper(builder.build()); >

   Modifying REST Client API

   Now, let’s see the complete changes required to support proxy authentication in our REST client that we developed earlier.

   ProductManagerConfig.java

   ProductManagerConfig class holds the REST API base URL and credentials required for authentication.

   package com.javachinna.rest.client.config; public record ProductManagerConfig(String baseUrl, String username, String password, String proxyHost, Integer proxyPort, String proxyUser, String proxyPass)

   Note: We have changed this class to a record class since it is intended to serve as a simple “data carrier”. A record class declares a sequence of fields, and then the appropriate accessors, constructors, equals , hashCode , and toString methods are created automatically. All the fields are final.

   ProductManagerImpl.java

   We have modified this implementation to use the new HttpRequestHelper class with the dynamically created HTTP Client instance.

   package com.javachinna.rest.client; import com.javachinna.rest.client.config.ProductManagerConfig; import com.javachinna.rest.client.helper.HttpRequestHelper; import com.javachinna.rest.client.model.*; import lombok.RequiredArgsConstructor; import java.net.Authenticator; import java.net.InetSocketAddress; import java.net.PasswordAuthentication; import java.net.ProxySelector; import java.net.http.HttpClient; import java.time.Duration; import java.util.HashMap; import java.util.List; import java.util.Map; @RequiredArgsConstructor public class ProductManagerImpl implements ProductManager < private static final String AUTH_API = "auth/signin"; private static final String PRODUCT_API = "products"; private final ProductManagerConfig productManagerConfig; private final HttpRequestHelper requestHelper; public ProductManagerImpl(ProductManagerConfig config) < this.productManagerConfig = config; HttpClient.Builder builder = HttpClient.newBuilder().version(HttpClient.Version.HTTP_2) .connectTimeout(Duration.ofSeconds(10)); if (config.proxyHost() != null && config.proxyPort() != null) < builder.proxy(ProxySelector.of(new InetSocketAddress(config.proxyHost(), config.proxyPort()))); Authenticator authenticator = Authenticator.getDefault(); if (config.proxyUser() != null && config.proxyPass() != null) < authenticator = new Authenticator() < @Override protected PasswordAuthentication getPasswordAuthentication() < return new PasswordAuthentication(config.proxyUser(), config.proxyPass().toCharArray()); >>; > builder.authenticator(authenticator); > this.requestHelper = new HttpRequestHelper(builder.build()); > public Product getProduct(Integer productId) throws Exception < String url = productManagerConfig.baseUrl() + PRODUCT_API + "/" + productId; return requestHelper.get(url, getAccessToken(), Product.class); >@SuppressWarnings("unchecked") public List> getAllProducts() throws Exception < String url = productManagerConfig.baseUrl() + PRODUCT_API; Mapresponse = requestHelper.get(url, getAccessToken(), Map.class); return (List>) response.get("content"); > public ApiResponse createProduct(ProductRequest request) throws Exception < return requestHelper.post(productManagerConfig.baseUrl() + PRODUCT_API, request, ApiResponse.class, getAccessToken()); >public ApiResponse updateProduct(Integer productId, ProductRequest request) throws Exception < return requestHelper.put(productManagerConfig.baseUrl() + PRODUCT_API + "/" + productId, request, ApiResponse.class, getAccessToken()); >public ApiResponse deleteProduct(Integer productId) throws Exception < return requestHelper.delete(productManagerConfig.baseUrl() + PRODUCT_API + "/" + productId, ApiResponse.class, getAccessToken()); >public String getAccessToken() throws Exception < LoginRequest loginRequest = new LoginRequest(productManagerConfig.username(), productManagerConfig.password()); JwtAuthenticationResponse jwtResponse = requestHelper.post(productManagerConfig.baseUrl() + AUTH_API, loginRequest, JwtAuthenticationResponse.class); return jwtResponse.getAccessToken(); >>

   HttpRequestHelper.java

   HttpRequestHelper contains generic methods used to make HTTP GET, POST, PUT, and DELETE requests.

   package com.javachinna.rest.client.helper; import com.fasterxml.jackson.core.JsonProcessingException; import com.fasterxml.jackson.databind.ObjectMapper; import java.io.IOException; import java.net.URI; import java.net.http.HttpClient; import java.net.http.HttpRequest; import java.net.http.HttpRequest.BodyPublisher; import java.net.http.HttpRequest.Builder; import java.net.http.HttpResponse; /** * * @author Chinna [javachinna.com] * */ public class HttpRequestHelper < private final String APPLICATION_JSON = "application/json"; private final String CONTENT_TYPE = "Content-Type"; private final String AUTHORIZATION = "Authorization"; private final String BEARER = "Bearer "; private final ObjectMapper objectMapper = new ObjectMapper().findAndRegisterModules(); private final HttpClient httpClient; public HttpRequestHelper(final HttpClient httpClient) < this.httpClient = httpClient; >public T get(String url, String token, Class valueType) throws IOException, InterruptedException < Builder builder = HttpRequest.newBuilder().GET().uri(URI.create(url)); return send(valueType, token, builder); >public T post(String uri, Object request, Class valueType) throws IOException, InterruptedException < return post(uri, request, valueType, null); >public T post(String uri, Object request, Class valueType, String token) throws IOException, InterruptedException < Builder builder = HttpRequest.newBuilder().uri(URI.create(uri)).POST(getBodyPublisher(request)).header(CONTENT_TYPE, APPLICATION_JSON); return send(valueType, token, builder); >public T put(String uri, Object request, Class valueType, String token) throws IOException, InterruptedException < Builder builder = HttpRequest.newBuilder().uri(URI.create(uri)).PUT(getBodyPublisher(request)).header(CONTENT_TYPE, APPLICATION_JSON); return send(valueType, token, builder); >public T patch(String uri, Class valueType, String token) throws IOException, InterruptedException < Builder builder = HttpRequest.newBuilder().uri(URI.create(uri)).method("PATCH", HttpRequest.BodyPublishers.noBody()).header(CONTENT_TYPE, APPLICATION_JSON); return send(valueType, token, builder); >public T delete(String uri, Class valueType, String token) throws IOException, InterruptedException < Builder builder = HttpRequest.newBuilder().uri(URI.create(uri)).DELETE(); return send(valueType, token, builder); >private BodyPublisher getBodyPublisher(Object request) throws JsonProcessingException < return HttpRequest.BodyPublishers.ofString(objectMapper.writeValueAsString(request)); >private T send(Class valueType, String token, Builder builder) throws IOException, InterruptedException < if (token != null) < builder.header(AUTHORIZATION, BEARER + token); >HttpResponse response = httpClient.send(builder.build(), HttpResponse.BodyHandlers.ofString()); if (response.statusCode() != 200) < throw new RuntimeException(response.body()); >return objectMapper.readValue(response.body(), valueType); > >

   Note: We have removed the HttpRequestUtils static class and created this helper class instead to create the HttpClient dynamically based on the configured proxy properties.

   Source Code

   Conclusion

   That’s all folks. In this article, we have added support for proxy authentication to our lightweight Java HttpClient.

   Источник

   HTTP Requests in Java with Proxies

   Accessing data over HTTP is more common every day. Be it APIs or webpages, intercommunication between applications is growing. And website scraping. There is no easy built-in solution to perform HTTP calls in Java. Many packages offer some related functionalities, but it’s not easy to pick one. Especially if you need some extra features like connecting via authenticated proxies. We’ll go from the basic request to advanced features using fluent.Request , part of the Apache HttpComponents project.

   Direct Request

   The first step is to request the desired page. We will use httpbin for the demo. It shows headers and origin IP, allowing us to check if the request was successful. We need to import Request , get the target page and extract the result as a string. The package provides methods for those cases and many more. Lastly, print the response.

   import org.apache.hc.client5.http.fluent.Request; public class TestRequest  public static void main(final String. args) throws Exception  String url = "http://httpbin.org/anything"; String response = Request .get(url) // use GET HTTP method .execute() // perform the call .returnContent() // handle and return response .asString(); // convert response to string System.out.println(response); > > 

   We are not handling the response nor checking for errors. It is a simplified version of a real-use case. But we can see on the result that the request was successful, and our IP shows as the origin. We’ll solve that in a moment.

   Proxy Request

   There are many reasons to add proxies to an HTTP request, such as security or anonymity. In any case, Java libraries (usually) make adding proxies complicated. In our case, we can use viaProxy with the proxy URL as long as we don’t need authentication. More on that later. For now, we’ll use a proxy from a free list. Note that these free proxies might not work for you. They are short-time lived.

   import org.apache.hc.client5.http.fluent.Request; public class TestRequest  public static void main(final String. args) throws Exception  String url = "http://httpbin.org/anything"; String proxy = "http://169.57.1.85:8123"; // Free proxy String response = Request.get(url) .viaProxy(proxy) // will set the passed proxy .execute().returnContent().asString(); System.out.println(response); > > 

   Proxy with Authentication

   Paid or private proxy providers — such as ZenRows — frequently use authentication in each call. Sometimes it is done via IP allowed lists, but it’s frequent to use other means like Proxy-Authorization headers. Calling the proxy without the proper auth method will result in an error: Exception in thread «main» org.apache.hc.client5.http.HttpResponseException: status code: 407, reason phrase: Proxy Authentication Required . Following the example, we will need two things: auth and passing the proxy as a Host. Proxy-Authorization contains the user and password base64 encoded. Then, we need to change how viaProxy gets the proxy since it does not allow URLs with user and password. For that, we will create a new HttpHost passing in the whole URL. It will internally handle the problem and omit the unneeded parts.

   import java.net.URI; import java.util.Base64; import org.apache.hc.client5.http.fluent.Request; import org.apache.hc.core5.http.HttpHost; public class TestRequest  public static void main(final String. args) throws Exception  String url = "http://httpbin.org/anything"; URI proxyURI = new URI("http://YOUR_API_KEY:@proxy.zenrows.com:8001"); // Proxy URL as given by the provider String basicAuth = new String( Base64.getEncoder() // get the base64 encoder .encode( proxyURI.getUserInfo().getBytes() // get user and password from the proxy URL )); String response = Request.get(url) .addHeader("Proxy-Authorization", "Basic " + basicAuth) // add auth .viaProxy(HttpHost.create(proxyURI)) // will set the passed proxy as a host .execute().returnContent().asString(); System.out.println(response); > > 

   Ignore SSL Certificates

   When adding proxies to SSL (https) connections, libraries tend to raise a warning/error about the certificate. From a security perspective, that is awesome! We avoid being shown or redirected to sites we prefer to avoid. But what about forcing our connections through our own proxies? There is no security risk in those cases, so we want to ignore those warnings. That is, again, not an easy task in Java. The error goes something like this: Exception in thread «main» javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target . For this case, we will modify the target URL by switching it to https . And also, call a helper method that we’ll create next. Nothing else changes on the main function.

   public class TestRequest  public static void main(final String. args) throws Exception  ignoreCertWarning(); // new method that will ignore certificate warnings String url = "https://httpbin.org/anything"; // switch to https // . > > 

   Now to the complicated and verbose part. We need to create an SSL context and fake certificates. As you can see, the certificates manager and its methods do nothing. It will just bypass the inner working and thus avoid the problems. Lastly, initialize the context with the created fake certs and set it as default. And we are good to go!

   import java.security.cert.X509Certificate; import javax.net.ssl.*; public class TestRequest  // . private static void ignoreCertWarning()  SSLContext ctx = null; TrustManager[] trustAllCerts = new X509TrustManager[]  new X509TrustManager()  public X509Certificate[] getAcceptedIssuers() return null;> public void checkClientTrusted(X509Certificate[] certs, String authType) <> public void checkServerTrusted(X509Certificate[] certs, String authType) <> > >; try  ctx = SSLContext.getInstance("SSL"); ctx.init(null, trustAllCerts, null); SSLContext.setDefault(ctx); > catch (Exception e) <> > > 

   Conclusion

   Accessing data (or scraping) in Java can get complicated and verbose. But with the right tools and libraries, we got to tame its verbosity — but for the certificate. We might get back to this topic in the future. The HttpComponents library offers attractive functionalities such as async and multi-threaded execution.

   Источник

   Читайте также:  Python executable one file