How to make presigned urls in python without AWS libs?

September 6, 2019 109 views
Object Storage Ubuntu 18.04

Dear support/colleagues.

I am trying to generate my own presigned url for DO spaces and has started from Amazons example here:

https://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html

but without luck and I am now asking ou for help.

This presigned url

https://fra1.digitaloceanspaces.com/cpi-germany/cpi-green-transparent-700x700.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=IB6Q77Q7CAUEX3PJ4DE6%2F20190906%2Ffra1%2Fs3%2Faws4_request&X-Amz-Date=20190906T072212Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=2a1a17581e0c0e59d3b188972c3848dc3867f7c5d8a44fa429c7d300acbcbea1

is generated by the web interface and works for me. I can download the png with curl.

I am then trying this:

import sys, os, base64, datetime, hashlib, hmac, urllib
import requests # pip install requests

# ************* REQUEST VALUES *************
method = 'GET'
service = 's3'
host = 'fra1.digitaloceanspaces.com'
region = 'fra1'
endpoint = 'https://fra1.digitaloceanspaces.com'

spacename = "cpi-germany"   #Or bucket
objectname = "cpi-green-transparent-700x700.png"

# Key derivation functions. See:
# http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html#signature-v4-examples-python
def sign(key, msg):
    return hmac.new(key, msg.encode('utf-8'), hashlib.sha256).digest()

def getSignatureKey(key, dateStamp, regionName, serviceName):
    kDate = sign(('AWS4' + key).encode('utf-8'), dateStamp)
    kRegion = sign(kDate, regionName)
    kService = sign(kRegion, serviceName)
    kSigning = sign(kService, 'aws4_request')
    return kSigning

# Read AWS access key from env. variables or configuration file. Best practice is NOT
# to embed credentials in code.
access_key = os.environ.get('AWS_ACCESS_KEY_ID')
secret_key = os.environ.get('AWS_SECRET_ACCESS_KEY')
if access_key is None or secret_key is None:
    print('No access key is available.')
    sys.exit()

# Create a date for headers and the credential string
t = datetime.datetime.utcnow()
amz_date = t.strftime('%Y%m%dT%H%M%SZ') # Format date as YYYYMMDD'T'HHMMSS'Z'
datestamp = t.strftime('%Y%m%d') # Date w/o time, used in credential scope


# ************* TASK 1: CREATE A CANONICAL REQUEST *************
# http://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html

# Because almost all information is being passed in the query string,
# the order of these steps is slightly different than examples that
# use an authorization header.

# Step 1: Define the verb (GET, POST, etc.)--already done.

# Step 2: Create canonical URI--the part of the URI from domain to query 
# string (use '/' if no path)
canonical_uri = '/'+spacename+'/'+objectname

# Step 3: Create the canonical headers and signed headers. Header names
# must be trimmed and lowercase, and sorted in code point order from
# low to high. Note trailing \n in canonical_headers.
# signed_headers is the list of headers that are being included
# as part of the signing process. For requests that use query strings,
# only "host" is included in the signed headers.
canonical_headers = 'host:' + host + '\n'
signed_headers = 'host'

# Match the algorithm to the hashing algorithm you use, either SHA-1 or
# SHA-256 (recommended)
algorithm = 'AWS4-HMAC-SHA256'
credential_scope = datestamp + '/' + region + '/' + service + '/' + 'aws4_request'

# Step 4: Create the canonical query string. In this example, request
# parameters are in the query string. Query string values must
# be URL-encoded (space=%20). The parameters must be sorted by name.
# use urllib.parse.quote_plus() if using Python 3
canonical_querystring = ''
canonical_querystring += 'X-Amz-Algorithm=AWS4-HMAC-SHA256'
canonical_querystring += '&X-Amz-Credential=' + urllib.quote_plus(access_key + '/' + credential_scope)
canonical_querystring += '&X-Amz-Date=' + amz_date
canonical_querystring += '&X-Amz-Expires=30'
canonical_querystring += '&X-Amz-SignedHeaders=' + signed_headers

# Step 5: Create payload hash. For GET requests, the payload is an
# empty string ("").
payload_hash = hashlib.sha256(('').encode('utf-8')).hexdigest()

# Step 6: Combine elements to create canonical request
canonical_request = method + '\n' + canonical_uri + '\n' + canonical_querystring + '\n' + canonical_headers + '\n' + signed_headers + '\n' + payload_hash


# ************* TASK 2: CREATE THE STRING TO SIGN*************
string_to_sign = algorithm + '\n' +  amz_date + '\n' +  credential_scope + '\n' +  hashlib.sha256(canonical_request.encode('utf-8')).hexdigest()

# ************* TASK 3: CALCULATE THE SIGNATURE *************
# Create the signing key
signing_key = getSignatureKey(secret_key, datestamp, region, service)

# Sign the string_to_sign using the signing_key
signature = hmac.new(signing_key, (string_to_sign).encode("utf-8"), hashlib.sha256).hexdigest()


# ************* TASK 4: ADD SIGNING INFORMATION TO THE REQUEST *************
# The auth information can be either in a query string
# value or in a header named Authorization. This code shows how to put
# everything into a query string.
canonical_querystring += '&X-Amz-Signature=' + signature


# ************* SEND THE REQUEST *************
# The 'host' header is added automatically by the Python 'request' lib. But it
# must exist as a header in the request.
request_url = endpoint + canonical_uri + "?" + canonical_querystring

print request_url;

print('\nBEGIN REQUEST++++++++++++++++++++++++++++++++++++')
print('Request URL = ' + request_url)
r = requests.get(request_url)

print('\nRESPONSE++++++++++++++++++++++++++++++++++++')
print('Response code: %d\n' % r.status_code)
print(r.text)

but get the following output

python main.py
https://fra1.digitaloceanspaces.com/cpi-germany/cpi-green-transparent-700x700.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=VPXIVAYZ5QOPAITB2VA4%2F20190906%2Ffra1%2Fs3%2Faws4_request&X-Amz-Date=20190906T072814Z&X-Amz-Expires=30&X-Amz-SignedHeaders=host&X-Amz-Signature=1e5c9c73150eb3095e694a2bbab6220402aa596d9ea62b72e3a89550cd20e686

BEGIN REQUEST++++++++++++++++++++++++++++++++++++
Request URL = https://fra1.digitaloceanspaces.com/cpi-germany/cpi-green-transparent-700x700.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=VPXIVAYZ5QOPAITB2VA4%2F20190906%2Ffra1%2Fs3%2Faws4_request&X-Amz-Date=20190906T072814Z&X-Amz-Expires=30&X-Amz-SignedHeaders=host&X-Amz-Signature=1e5c9c73150eb3095e694a2bbab6220402aa596d9ea62b72e3a89550cd20e686

RESPONSE++++++++++++++++++++++++++++++++++++
Response code: 403

<?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><RequestId>tx000000000000009440137-005d720a8e-c8079e-fra1a</RequestId><HostId>c8079e-fra1a-fra1</HostId></Error>

When I compare the url I generate with this python script with the one I got from the DO webinterface, they are almost identical.
The access key is different, obviously I assume, and the signature is different.

Now, I am basically doing whatever the Amazon example does, so I am a bit out of ideas.

Can anyone spot my obvious error or does anyone have a snippet that works?

Best regards,
Morten

P.S.: The python script is only used to get to understand the process, as I need to use Dart for production. As far as I can tell, there are no snippets on making presigned urls for dart, so I feel a bit on my own here. If anybody has a snippet for dart that I would be happy to see it.

1 Answer

Hi,

Spaces Engineering team member here.

Your code looks almost entirely correct, there’s only one bug, you need to include a specific payload hash for presigned GET requests.
payload_hash = 'UNSIGNED-PAYLOAD'

Rather than the AWS doc you pointed to, there’s another one that gives a testcase to validate your code as working:
https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html
(if you print out the canonical request, and the string to sign, then compare it against their example, you can validate it).

The other bit that stands out is that the access key you used VPXIVAYZ5QOPAITB2VA4 is not one associated with your account. Maybe it’s an old key that you invalidated already?

Is your Dart code only going to be for doing GET requests? If it’s going to be used for PUT requests, or more complex headers, there’s a number of other nuances to the AWS signature protocol that are not well-covered by the AWS documentation.

  • Robin
Have another answer? Share your knowledge.