How to make presigned urls in python without AWS libs?
I am trying to generate my own presigned url for DO spaces and has started from Amazons example here:
but without luck and I am now asking ou for help.
This presigned url
is generated by the web interface and works for me. I can download the png with curl.
I am then trying this:
import sys, os, base64, datetime, hashlib, hmac, urllib import requests # pip install requests # ************* REQUEST VALUES ************* method = 'GET' service = 's3' host = 'fra1.digitaloceanspaces.com' region = 'fra1' endpoint = 'https://fra1.digitaloceanspaces.com' spacename = "cpi-germany" #Or bucket objectname = "cpi-green-transparent-700x700.png" # Key derivation functions. See: # http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html#signature-v4-examples-python def sign(key, msg): return hmac.new(key, msg.encode('utf-8'), hashlib.sha256).digest() def getSignatureKey(key, dateStamp, regionName, serviceName): kDate = sign(('AWS4' + key).encode('utf-8'), dateStamp) kRegion = sign(kDate, regionName) kService = sign(kRegion, serviceName) kSigning = sign(kService, 'aws4_request') return kSigning # Read AWS access key from env. variables or configuration file. Best practice is NOT # to embed credentials in code. access_key = os.environ.get('AWS_ACCESS_KEY_ID') secret_key = os.environ.get('AWS_SECRET_ACCESS_KEY') if access_key is None or secret_key is None: print('No access key is available.') sys.exit() # Create a date for headers and the credential string t = datetime.datetime.utcnow() amz_date = t.strftime('%Y%m%dT%H%M%SZ') # Format date as YYYYMMDD'T'HHMMSS'Z' datestamp = t.strftime('%Y%m%d') # Date w/o time, used in credential scope # ************* TASK 1: CREATE A CANONICAL REQUEST ************* # http://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html # Because almost all information is being passed in the query string, # the order of these steps is slightly different than examples that # use an authorization header. # Step 1: Define the verb (GET, POST, etc.)--already done. # Step 2: Create canonical URI--the part of the URI from domain to query # string (use '/' if no path) canonical_uri = '/'+spacename+'/'+objectname # Step 3: Create the canonical headers and signed headers. Header names # must be trimmed and lowercase, and sorted in code point order from # low to high. Note trailing \n in canonical_headers. # signed_headers is the list of headers that are being included # as part of the signing process. For requests that use query strings, # only "host" is included in the signed headers. canonical_headers = 'host:' + host + '\n' signed_headers = 'host' # Match the algorithm to the hashing algorithm you use, either SHA-1 or # SHA-256 (recommended) algorithm = 'AWS4-HMAC-SHA256' credential_scope = datestamp + '/' + region + '/' + service + '/' + 'aws4_request' # Step 4: Create the canonical query string. In this example, request # parameters are in the query string. Query string values must # be URL-encoded (space=%20). The parameters must be sorted by name. # use urllib.parse.quote_plus() if using Python 3 canonical_querystring = '' canonical_querystring += 'X-Amz-Algorithm=AWS4-HMAC-SHA256' canonical_querystring += '&X-Amz-Credential=' + urllib.quote_plus(access_key + '/' + credential_scope) canonical_querystring += '&X-Amz-Date=' + amz_date canonical_querystring += '&X-Amz-Expires=30' canonical_querystring += '&X-Amz-SignedHeaders=' + signed_headers # Step 5: Create payload hash. For GET requests, the payload is an # empty string (""). payload_hash = hashlib.sha256(('').encode('utf-8')).hexdigest() # Step 6: Combine elements to create canonical request canonical_request = method + '\n' + canonical_uri + '\n' + canonical_querystring + '\n' + canonical_headers + '\n' + signed_headers + '\n' + payload_hash # ************* TASK 2: CREATE THE STRING TO SIGN************* string_to_sign = algorithm + '\n' + amz_date + '\n' + credential_scope + '\n' + hashlib.sha256(canonical_request.encode('utf-8')).hexdigest() # ************* TASK 3: CALCULATE THE SIGNATURE ************* # Create the signing key signing_key = getSignatureKey(secret_key, datestamp, region, service) # Sign the string_to_sign using the signing_key signature = hmac.new(signing_key, (string_to_sign).encode("utf-8"), hashlib.sha256).hexdigest() # ************* TASK 4: ADD SIGNING INFORMATION TO THE REQUEST ************* # The auth information can be either in a query string # value or in a header named Authorization. This code shows how to put # everything into a query string. canonical_querystring += '&X-Amz-Signature=' + signature # ************* SEND THE REQUEST ************* # The 'host' header is added automatically by the Python 'request' lib. But it # must exist as a header in the request. request_url = endpoint + canonical_uri + "?" + canonical_querystring print request_url; print('\nBEGIN REQUEST++++++++++++++++++++++++++++++++++++') print('Request URL = ' + request_url) r = requests.get(request_url) print('\nRESPONSE++++++++++++++++++++++++++++++++++++') print('Response code: %d\n' % r.status_code) print(r.text)
but get the following output
python main.py https://fra1.digitaloceanspaces.com/cpi-germany/cpi-green-transparent-700x700.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=VPXIVAYZ5QOPAITB2VA4%2F20190906%2Ffra1%2Fs3%2Faws4_request&X-Amz-Date=20190906T072814Z&X-Amz-Expires=30&X-Amz-SignedHeaders=host&X-Amz-Signature=1e5c9c73150eb3095e694a2bbab6220402aa596d9ea62b72e3a89550cd20e686 BEGIN REQUEST++++++++++++++++++++++++++++++++++++ Request URL = https://fra1.digitaloceanspaces.com/cpi-germany/cpi-green-transparent-700x700.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=VPXIVAYZ5QOPAITB2VA4%2F20190906%2Ffra1%2Fs3%2Faws4_request&X-Amz-Date=20190906T072814Z&X-Amz-Expires=30&X-Amz-SignedHeaders=host&X-Amz-Signature=1e5c9c73150eb3095e694a2bbab6220402aa596d9ea62b72e3a89550cd20e686 RESPONSE++++++++++++++++++++++++++++++++++++ Response code: 403 <?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><RequestId>tx000000000000009440137-005d720a8e-c8079e-fra1a</RequestId><HostId>c8079e-fra1a-fra1</HostId></Error>
When I compare the url I generate with this python script with the one I got from the DO webinterface, they are almost identical.
The access key is different, obviously I assume, and the signature is different.
Now, I am basically doing whatever the Amazon example does, so I am a bit out of ideas.
Can anyone spot my obvious error or does anyone have a snippet that works?
P.S.: The python script is only used to get to understand the process, as I need to use Dart for production. As far as I can tell, there are no snippets on making presigned urls for dart, so I feel a bit on my own here. If anybody has a snippet for dart that I would be happy to see it.
These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.