- Timeouts with Java 8 CompletableFuture: You’re probably doing it wrong
- Running with a timeout the synchronous way
- Asynchronous way of doing timeouts
- Using a ScheduledExecutorService with CompletableFuture API
- Our hypothetical application
- Possible Fix: Go back to Java Futures
- Timeouts and how to handline in Java
- Connection timeout
- Socket timeout
- Read timeout
- Write timeout
- Apache Camel
- Rest Template
- WebClient
- Summary
Timeouts with Java 8 CompletableFuture: You’re probably doing it wrong
This was meant to be a quick blog post on using CompletableFuture in Java 8 for timeouts. But researching further on the same led to a lot of interesting discoveries! For an introduction to CompletableFuture in Java8, I would recommend reading this blog post.
A lot of places where similar approaches for timeouts have been mentioned do not mention the pitfalls, so I hope I can provide some light on them.
The objective: We have many situations where we want to run some functions in our application with timeouts. Eg: a dependency being called over HTTP might need to be timed out, a long computation which might affect the user and it is better to avoid it.
Let’s explore the various options we have for running something against a timeout.
Running with a timeout the synchronous way
This is a very simple API provided by CompletableFuture.get()
// The synchronous method timeoutFuture = new CompletableFuture(); try < timeoutFuture.get(1000, TimeUnit.MILLIS); >catch (TimeoutException | InterruptedException | ExecutionException e)
A TimeoutException is thrown from the running thread if the timeout period passes which in this case would always happen since timeoutFuture isn’t completed ever.
The code here is really simple, but with one major disadvantage of being synchronous. So you can’t start doing anything apart from busy waiting for the execution to complete.
Asynchronous way of doing timeouts
We will explore a couple of ways. And will also discuss why you might want to prefer one over the other.
Using a ScheduledExecutorService with CompletableFuture API
timeoutFuture = new CompletableFuture(); // Run a scheduled task which runs after 100 milliseconds scheduler.schedule(timeoutFuture.completeExceptionally(new TimeoutException()), 100, TimeUnit.MILLIS)); finalFuture = CompletableFuture.anyOf(future, timeoutFuture);
Let’s talk a bit of how this works. The finalFuture represents the first future that gets completed, and returns the value of that scheduler is an instance of ScheduledExecutorService, and allows us to run futures at arbitrary times.
How many threads for ScheduledExecutorService?
The ScheduledExecutorService doesn’t do anything particularly time consuming. It just completes the future whenever the scheduler is supposed to run. Since we aren’t doing anything at all in the scheduler, we might as well use a single threaded executor.
-
- Note that the subsequent executions with .thenApply() may happen on the scheduler so it is best to not let anything run on the scheduler, by calling .supplyAsync() with some other executor explicitly on the result. Otherwise you should use a smaller thread pool for the ScheduledExecutorService.
- The dependency future gets cancelled in case of a timeout, but the underlying task keeps running, and there is no easy way to cancel the underlying task execution! What problem could it cause for your application? Let me explain with give an example of a hypothetical application.
Our hypothetical application
Our application runs 10 dependency calls. Each dependency call has a timeout of 100ms and the calls are made concurrently. We have 10 threads for the executor which runs these calls.
What would be the throughput of our application?
Ideally we should be able to repeat this the complete cycle in 100ms(the timeout we have considered).
class MyApplication < public static Integer getValue() < System.out.println("I am called"); // Simulating a long network call of 1 second in the worst case try < Thread.sleep(1000); >catch (InterruptedException e) < e.printStackTrace(); >return 10; > public static void main() < ExecutorService executor = new ThreadPoolExecutor(10, 10, 0L, TimeUnit.MILLISECONDS, // This is an unbounded Queue. This should never be used // in real life. That is the first step to failure. new LinkedBlockingQueue()); // We want to call the dummy service 10 times for (int i=0; i getValue(), executor); CompletableFuture futureTimeout = new CompletableFuture(); schedulerService.schedule(() -> futureTimeout.completeExceptionally(throw new TimeoutException(), 100, TimeUnit.MILLISECONDS); CompletableFuture result = CompletableFuture.anyOf(dependencyFuture, futureTimeout); allFutures[i] = result; > // Finally wait for all futures to join CompletableFuture.allOf(allFutures).join(); System.out.println("All futures completed"); System.out.println(executor.toString()); >
We get a very interesting output from our example(note that I have modified the Executors to add some additional information):
Running pool-1-thread-1 I am called Running pool-1-thread-2 I am called Running pool-1-thread-3 I am called Running pool-1-thread-4 I am called Running pool-1-thread-5 I am called Running pool-1-thread-6 I am called Running pool-1-thread-7 I am called Running pool-1-thread-8 I am called Running pool-1-thread-9 I am called Running pool-1-thread-10 I am called ### Good till here, each thread spawns the particular dependency ### All futures completed ### All futures have completed ### [email protected][Running, pool size = 10, active threads = 10, queued tasks = 0, completed tasks = 0] ### WHOOPS, our Executor says that we still are running 10 threads ### Finished running Finished running Finished running Finished running Finished running Finished running Finished running Finished running Finished running Finished running ### Our threads complete now(roughly at their sleep endings) ###
Let’s try to understand what happened:
- CompletableFuture doesn’t get tied to the execution happening, and unlike Future, there is no Cancel for CompletableFuture. Look at this blog post for more details.
- Our expected Throughput was 10 per second. What do we get with this restriction? In this case, we can run only once in 1 second, so our throughput is just 1. It can actually hurt your application in a serious way.
- If you model your dependencies so that they never fail on their own, they’ll consume an executor thread forever!
Possible Fix: Go back to Java Futures
The way out of here is to go back to using Java Futures instead of the CompletableFuture API. The reason is that Java Futures are tied to the execution tasks, and allow us to cancel a thread.
I am a believer in learning through examples, so let’s quickly pick up the same application we used above.class MyApplication < public static Integer getValue() < System.out.println("I am called"); // Simulating a long network call of 1 second in the worst case try < Thread.sleep(1000); >catch (InterruptedException e) < e.printStackTrace(); >return 10; > public static void main() < ExecutorService executor = new ThreadPoolExecutor(10, 10, 0L, TimeUnit.MILLISECONDS, // This is an unbounded Queue. This should never be used // in real life. That is the first step to failure. new LinkedBlockingQueue()); // We want to call the dummy service 10 times for (int i=0; i < // Instead of using CompletableFuture.supplyAsync, directly create a future from the executor Future future = executorService.submit(() ->getValue()); schedulerService.schedule(() -> future.cancel(true), 100, TimeUnit.MILLISECONDS); try < return future.get(); >catch (InterruptedException | ExecutionException | CancellationException e) < // pass >// You can choose to return a dummy value here return null; >); > // Finally wait for all futures to join CompletableFuture.allOf(allFutures).join(); System.out.println("All futures completed"); System.out.println(executor.toString()); >
Running pool-1-thread-2 I am called Running pool-1-thread-3 I am called Running pool-1-thread-1 I am called Finished running Finished running Finished running Running pool-1-thread-4 I am called Running pool-1-thread-5 I am called Running pool-1-thread-6 I am called Finished running Finished running Finished running Running pool-1-thread-7 I am called Running pool-1-thread-9 I am called Running pool-1-thread-8 I am called Finished running Finished running Finished running Running pool-1-thread-10 I am called Finished running ### Looks good ### All futures completed [email protected][Running, pool size = 10, active threads = 0, queued tasks = 0, completed tasks = 10] ### Aha! 0 queued and active threads after completion ###
What are we doing here?
Same as the previous case, except now we cancel the future that we provide to the executor. So now, the task doesn’t run on the executor anymore. Ideally, here we get the throughput of 10 that we expect.
BUT…
This approach too has caveats. Java Futures are blocking, so I had to wrap it inside CompletableFuture.supplyAsync() in order to make it asynchronous. This means that we need an executor to run this code too. In the above example, I haven’t specified any executor so it runs on ForkJoinPool, but you might need to use a custom executor for the same.Unfortunately this means that we are using some threads to run the execution and some threads to just wait for the execution to finish. How to find the number of threads you would need? The same number of threads that we use for running the executions. So effectively, we are using double the number of threads that we actually need.
The moral of the story is, asynchronous programming is hard! A lot of your assumptions may be incorrect. Definitely try to understand how to use the CompletableFuture model before using it!Timeouts and how to handline in Java
In this article we will try to cover why it’s important to define timeouts for out bound rest calls. Before configuring any timeout let’s understand below some common exceptions for http outbound calls,
Connection timeout
maximum time to wait for the other side to answer «yes, I’m here, let’s talk» when creating a new connection, (ConnectTimeout eventually calls socket.connect(address, timeout), If the connection is not established within the ConnectTimeout specified by you or the library you are using then, you get an error ‘connect timedout
Socket timeout
Is the timeout for waiting for data or, put differently, a maximum period inactivity between two consecutive data packets
Read timeout
Read timeout can happen when there is successful connection established between client and the server and there is an inactivity between data packets while waiting for the server response.
Write timeout
Similar to Read timeout, write timeout can happen when there is successful connection established between client and server, and there is inactivity between data packets while sending the request to the server. The important topic to remember here is that based on the choice of library we use for outbound calls it’s very important that we configure the properties to handle the above mentioned one’s and handle the exception gracefully.
Apache Camel
If we are using Apache Camel ‘http’ component to make the outbound calls then we can configure these properties in following ways, please note that if we don’t define this properties the default values is -1! means connection will never timeout and can have advert effect on the application performance overall. camel.property
http.urlProxy = http4://ThirdPartyServers?throwExceptionOnFailure=false&httpClient.socketTimeout=$&httpClient.connectTimeout=$
@Override public void configure() throws Exception configureTimeout(); > private void configureTimeout() HttpComponent httpComponent = getContext().getComponent("http4", HttpComponent.class); httpComponent.setConnectionTimeToLive(VALUE_IN_MILI);// for closing the idle connection - in milliseconds httpComponent.setSocketTimeout(VALUE_IN_MILI); //socket timeout - in milliseconds httpComponent.setConnectTimeout(VALUE_IN_MILI); // connection timeout - in milliseconds*/ >
Rest Template
final RequestConfig requestConfig = RequestConfig.custom() .setConnectionRequestTimeout(VALUE_IN_MILI) .setConnectTimeout(VALUE_IN_MILI) .setSocketTimeout(VALUE_IN_MILIs) .build(); final HttpClient httpClient = HttpClients.custom() .setConnectionTimeToLive(VALUE_IN_MILI, SECONDS) .setRetryHandler((IOException exception, int executionCount, HttpContext context) -> return executionCount 3; >) .setServiceUnavailableRetryStrategy(new DefaultServiceUnavailableRetryStrategy(3, 1)) .setDefaultRequestConfig(requestConfig) .build(); final RestTemplate restTemplate = new RestTemplate(new HttpComponentsClientHttpRequestFactory(httpClient));
WebClient
public WebClient getWebClient() HttpClient httpClient = HttpClient.create() .tcpConfiguration(client -> client.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, VALUE_IN_MILI) .doOnConnected(conn -> conn .addHandlerLast(new ReadTimeoutHandler(rest.timeout.millis)) .addHandlerLast(new WriteTimeoutHandler(rest.timeout.millis)))); ClientHttpConnector connector = new ReactorClientHttpConnector(httpClient.wiretap(true)); return WebClient.builder() .baseUrl("http://localhost:3000") .clientConnector(connector) .defaultHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE) .build(); >
Summary
All in all it’s very important to configure these values i.e. connection timeout, read timeout, socket timeout etc so as to terminate the connections after waiting for a specific amount of time rather keeping the connection open indefinitely which can bring issues to overall application performance and stability.