Proxy that support java

HTTP Requests in Java with Proxies

Accessing data over HTTP is more common every day. Be it APIs or webpages, intercommunication between applications is growing. And website scraping. There is no easy built-in solution to perform HTTP calls in Java. Many packages offer some related functionalities, but it’s not easy to pick one. Especially if you need some extra features like connecting via authenticated proxies. We’ll go from the basic request to advanced features using fluent.Request , part of the Apache HttpComponents project.

Direct Request

The first step is to request the desired page. We will use httpbin for the demo. It shows headers and origin IP, allowing us to check if the request was successful. We need to import Request , get the target page and extract the result as a string. The package provides methods for those cases and many more. Lastly, print the response.

import org.apache.hc.client5.http.fluent.Request; public class TestRequest  public static void main(final String. args) throws Exception  String url = "http://httpbin.org/anything"; String response = Request .get(url) // use GET HTTP method .execute() // perform the call .returnContent() // handle and return response .asString(); // convert response to string System.out.println(response); > > 

We are not handling the response nor checking for errors. It is a simplified version of a real-use case. But we can see on the result that the request was successful, and our IP shows as the origin. We’ll solve that in a moment.

Proxy Request

There are many reasons to add proxies to an HTTP request, such as security or anonymity. In any case, Java libraries (usually) make adding proxies complicated. In our case, we can use viaProxy with the proxy URL as long as we don’t need authentication. More on that later. For now, we’ll use a proxy from a free list. Note that these free proxies might not work for you. They are short-time lived.

import org.apache.hc.client5.http.fluent.Request; public class TestRequest  public static void main(final String. args) throws Exception  String url = "http://httpbin.org/anything"; String proxy = "http://169.57.1.85:8123"; // Free proxy String response = Request.get(url) .viaProxy(proxy) // will set the passed proxy .execute().returnContent().asString(); System.out.println(response); > > 

Proxy with Authentication

Paid or private proxy providers — such as ZenRows — frequently use authentication in each call. Sometimes it is done via IP allowed lists, but it’s frequent to use other means like Proxy-Authorization headers. Calling the proxy without the proper auth method will result in an error: Exception in thread «main» org.apache.hc.client5.http.HttpResponseException: status code: 407, reason phrase: Proxy Authentication Required . Following the example, we will need two things: auth and passing the proxy as a Host. Proxy-Authorization contains the user and password base64 encoded. Then, we need to change how viaProxy gets the proxy since it does not allow URLs with user and password. For that, we will create a new HttpHost passing in the whole URL. It will internally handle the problem and omit the unneeded parts.

import java.net.URI; import java.util.Base64; import org.apache.hc.client5.http.fluent.Request; import org.apache.hc.core5.http.HttpHost; public class TestRequest  public static void main(final String. args) throws Exception  String url = "http://httpbin.org/anything"; URI proxyURI = new URI("http://YOUR_API_KEY:@proxy.zenrows.com:8001"); // Proxy URL as given by the provider String basicAuth = new String( Base64.getEncoder() // get the base64 encoder .encode( proxyURI.getUserInfo().getBytes() // get user and password from the proxy URL )); String response = Request.get(url) .addHeader("Proxy-Authorization", "Basic " + basicAuth) // add auth .viaProxy(HttpHost.create(proxyURI)) // will set the passed proxy as a host .execute().returnContent().asString(); System.out.println(response); > > 

Ignore SSL Certificates

When adding proxies to SSL (https) connections, libraries tend to raise a warning/error about the certificate. From a security perspective, that is awesome! We avoid being shown or redirected to sites we prefer to avoid. But what about forcing our connections through our own proxies? There is no security risk in those cases, so we want to ignore those warnings. That is, again, not an easy task in Java. The error goes something like this: Exception in thread «main» javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target . For this case, we will modify the target URL by switching it to https . And also, call a helper method that we’ll create next. Nothing else changes on the main function.

public class TestRequest  public static void main(final String. args) throws Exception  ignoreCertWarning(); // new method that will ignore certificate warnings String url = "https://httpbin.org/anything"; // switch to https // . > > 

Now to the complicated and verbose part. We need to create an SSL context and fake certificates. As you can see, the certificates manager and its methods do nothing. It will just bypass the inner working and thus avoid the problems. Lastly, initialize the context with the created fake certs and set it as default. And we are good to go!

import java.security.cert.X509Certificate; import javax.net.ssl.*; public class TestRequest  // . private static void ignoreCertWarning()  SSLContext ctx = null; TrustManager[] trustAllCerts = new X509TrustManager[]  new X509TrustManager()  public X509Certificate[] getAcceptedIssuers() return null;> public void checkClientTrusted(X509Certificate[] certs, String authType) <> public void checkServerTrusted(X509Certificate[] certs, String authType) <> > >; try  ctx = SSLContext.getInstance("SSL"); ctx.init(null, trustAllCerts, null); SSLContext.setDefault(ctx); > catch (Exception e) <> > > 

Conclusion

Accessing data (or scraping) in Java can get complicated and verbose. But with the right tools and libraries, we got to tame its verbosity — but for the certificate. We might get back to this topic in the future. The HttpComponents library offers attractive functionalities such as async and multi-threaded execution.

Источник

Saved searches

Use saved searches to filter your results more quickly

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session.

HTTP/HTTPS MITM Java Proxy with the possibility of intercepting and investigating/altering both requests and responses.

License

tkohegyi/mitmJavaProxy

This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?

Sign In Required

Please sign in to use Codespaces.

Launching GitHub Desktop

If nothing happens, download GitHub Desktop and try again.

Launching GitHub Desktop

If nothing happens, download GitHub Desktop and try again.

Launching Xcode

If nothing happens, download Xcode and try again.

Launching Visual Studio Code

Your codespace will open once ready.

There was a problem preparing your codespace, please try again.

Latest commit

Git stats

Files

Failed to load latest commit information.

README.md

This Men-In-The-Middle Java Proxy is a derivative work that was originated from BrowserMob Proxy, then tailored first for the old Wilma versions, then reviewed and reworked again for the next generation of Wilma which is a proxy/stub/service virtualization tool and relies on this MitmJavaProxy.

Right now everything seems to be working so can be used.

Also please be aware that the main aim of creating this MITM Java Proxy is to support the proxy need of Wilma. Therefore none of the original browsermob-proxy features should be expected as working, on the other side, other features which was not in the original browsermob-proxy become available. Also, this version is prepered for Java 11 (and above) versions, and supports TSL. In sort, what you can do with it:

• proxying the HTTP/HTTPS messages and
• by intercepting both requests and responses
• do whatever you want to do with the intercepted messages. It is possible to alter both the request before it hits the server (what is more — you can alter the target URL too) and the response before it arrives to the client.

This proxy is a Men-In-The-Middle type of proxy that is able to capture ANY http traffic, everything that is transferred between the client and the server, even if it is encrypted. Use it carefully and only when you know what you do, and what you do is allowed/legal, and on environment where you are allowed/authorized to use the Proxy.

If you’re using Java, the easiest way to get started is to embed the project directly. The jar is available in maven central repository.

Related gradle file should look like similar to the below:

repositories < mavenCentral() >dependencies

Latest announced release is available here.

Before building the proxy locally, create a gradle.properties file in root folder with the following content:

Then use this command to start the build:

To publish the library on local machine (in local maven repository):

./gradlew clean build publishToMavenLocal 

To publish the library for public use (into public maven repository), you need the necessary right there and use this command:

./gradlew clean build publish -PbuildNumber=x 

Please note that without specifying the build number, the build will be a SNAPSHOT build.

See detailed information of its usage at wiki pages.

HTTP Request Manipulation

Just add a Request Interceptor to the proxy server, and manipulate the request as you wish. See details here.

HTTP Response Manipulation

Just add a Response Interceptor to the proxy server, and you will get access to the responses.

The key to manipulate responses is the Response Volatility attribute. If a response is volatile, the proxy (or you) must work with the response a lot (call interceptor, extract, manipulate the response, compress, then release response towards the Client). This takes time. If a response is not volatile, then the proxy don’t need to do such things. This of course a much faster method, so in case you don’t need to manipulate the response, just leave responses as not volatile. Response volatility can be set in general via static method: ProxyServer.setResponseVolatility(boolean) Or can be set per request-response pair by using the Request Interceptors. See more details here.

While the proxy supports SSL, it requires that a Certificate Authority be installed in to the browser / at client from where you call a server via the proxy. This allows the client to trust all the SSL traffic coming from the proxy, which will be proxied using a classic man-in-the-middle technique.

IT IS CRITICAL THAT YOU DO NOT INSTALL THIS CERTIFICATE AUTHORITY ON A CLIENT/BROWSER THAT IS USED FOR ANYTHING OTHER THAN TESTING.

It is recommended to use -Djdk.tls.namedGroups=»secp256r1, secp384r1, ffdhe2048, ffdhe3072″ java arguments to address some issues still existing in some JDK implementations.

Please note that in V2.5.x the certificate used by this proxy is upgraded.

About

HTTP/HTTPS MITM Java Proxy with the possibility of intercepting and investigating/altering both requests and responses.

Источник