Question

Issue with S3Client on DigitalOcean - Unable to execute HTTP request: Timeout waiting for connection from pool

In our app, we have attachments that are supposed to be secured. These attachments are stored in DO Spaces and we are generating resigned URLs to allow secured access.

The logic of signing the URLs looks like this:

String getSignUrl(String objectKey, Long expirationMillis) {
        String signedUrl = null;
        try {
            AmazonS3 s3Client = awsClientProviderService.getS3Client()

            // Set the presigned URL to expire after one hour.
            Date expiration = new Date()
            Long expTimeMillis = expiration.getTime();
            expTimeMillis += expirationMillis;
            expiration.setTime(expTimeMillis);

            GeneratePresignedUrlRequest generatePresignedUrlRequest =
                    new GeneratePresignedUrlRequest(spaceName, objectKey)
                            .withMethod(HttpMethod.GET)
                            .withExpiration(expiration);
            URL url = s3Client.generatePresignedUrl(generatePresignedUrlRequest);

            signedUrl = url.toString()
        } catch (AmazonServiceException e) {
            log.error(e.message, e)
        } catch (SdkClientException e) {
            log.error(e.message, e)
        }

        signedUrl
    }

Where getS3Client is:

AmazonS3 getS3Client() {
        if (!s3Client) {
            AWSStaticCredentialsProvider credentials =
                    new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKey, secret))
            s3Client = AmazonS3ClientBuilder
                    .standard()
                    .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(endpoint, region))
                    .withCredentials(credentials)
                    .build()
        }

        s3Client
    }

All good until now and works great.

However, when we start signing the URLs, after 50 of these requests we are getting the following error:

Unable to execute HTTP request: Timeout waiting for connection from pool

Digging around a bit, I tried adding maxConnections to the client: .withClientConfiguration(new ClientConfiguration().withMaxConnections(20))

This caused to get the error after 20 such requests. By the way, even if we sign 5 URLs first, then wait for 10 min, try another batch. The error occurs after 20 requests collectively.

Tried, 100 connections. Same story after 100 requests.

This made me think that the connections are not getting released for some reason. And the only thing that I could find is people proposing to increase maxConnections to 1000. But, that would just defer the issue.

What is the solution here? How can we release connections of S3Client?


Submit an answer

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer