Question

Issue with S3Client on DigitalOcean - Unable to execute HTTP request: Timeout waiting for connection from pool

In our app, we have attachments that are supposed to be secured. These attachments are stored in DO Spaces and we are generating resigned URLs to allow secured access.

The logic of signing the URLs looks like this:

String getSignUrl(String objectKey, Long expirationMillis) {
        String signedUrl = null;
        try {
            AmazonS3 s3Client = awsClientProviderService.getS3Client()

            // Set the presigned URL to expire after one hour.
            Date expiration = new Date()
            Long expTimeMillis = expiration.getTime();
            expTimeMillis += expirationMillis;
            expiration.setTime(expTimeMillis);

            GeneratePresignedUrlRequest generatePresignedUrlRequest =
                    new GeneratePresignedUrlRequest(spaceName, objectKey)
                            .withMethod(HttpMethod.GET)
                            .withExpiration(expiration);
            URL url = s3Client.generatePresignedUrl(generatePresignedUrlRequest);

            signedUrl = url.toString()
        } catch (AmazonServiceException e) {
            log.error(e.message, e)
        } catch (SdkClientException e) {
            log.error(e.message, e)
        }

        signedUrl
    }

Where getS3Client is:

AmazonS3 getS3Client() {
        if (!s3Client) {
            AWSStaticCredentialsProvider credentials =
                    new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKey, secret))
            s3Client = AmazonS3ClientBuilder
                    .standard()
                    .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(endpoint, region))
                    .withCredentials(credentials)
                    .build()
        }

        s3Client
    }

All good until now and works great.

However, when we start signing the URLs, after 50 of these requests we are getting the following error:

Unable to execute HTTP request: Timeout waiting for connection from pool

Digging around a bit, I tried adding maxConnections to the client: .withClientConfiguration(new ClientConfiguration().withMaxConnections(20))

This caused to get the error after 20 such requests. By the way, even if we sign 5 URLs first, then wait for 10 min, try another batch. The error occurs after 20 requests collectively.

Tried, 100 connections. Same story after 100 requests.

This made me think that the connections are not getting released for some reason. And the only thing that I could find is people proposing to increase maxConnections to 1000. But, that would just defer the issue.

What is the solution here? How can we release connections of S3Client?


Submit an answer


This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer

Try DigitalOcean for free

Click below to sign up and get $200 of credit to try our products over 60 days!

Sign up

card icon
Get our biweekly newsletter

Sign up for Infrastructure as a Newsletter.

Sign up
card icon
Hollie's Hub for Good

Working on improving health and education, reducing inequality, and spurring economic growth? We’d like to help.

Learn more
card icon
Become a contributor

You get paid; we donate to tech nonprofits.

Learn more
Welcome to the developer cloud

DigitalOcean makes it simple to launch in the cloud and scale up as you grow – whether you’re running one virtual machine or ten thousand.

Learn more ->
DigitalOcean Cloud Control Panel
Get started for free

Enter your email to get $200 in credit for your first 60 days with DigitalOcean.

New accounts only. By submitting your email you agree to our Privacy Policy.