Question

In Functions, I handle a payload, how do I stop DigitalOcean storing their data or access the raw payload?

I have been debugging an error with a Function to handle webhooks.

I verify the expected payload of the webhooks with key/signatures.

DigitalOcean is writing their own information into the payload (e.g. __ow_headers, __ow_path, __ow_method).

Although this information is useful, as they are manipulating the payload any verification that I do will fail.

My preferred option is to just access the raw payload which was sent by the originator. Does anyone know how to do this in Python using DigitalOcean Functions?

My backup is to strip out the information I need, however if DigitalOcean or my originator change the payload then the code will break, for this reason it is not preferred.


Submit an answer
Answer a question...

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Matt Welke
DigitalOcean Employee
DigitalOcean Employee badge
January 21, 2023

If you want to have the raw request body sent by the client come through in your event parameter (the first parameter of your Python function), there are a few options.

One is to use raw as the value of your web setting in your project’s project.yml file.

For example, the Python starter code (when generated with doctl sls init) includes a project.yml file that looks like this:

packages:
  - name: sample
    shared: false
    functions:
      - name: hello
        binary: false
        main: ""
        runtime: python:default
        web: true
        websecure: false

If you change it to this:

packages:
  - name: sample
    shared: false
    functions:
      - name: hello
        binary: false
        main: ""
        runtime: python:default
        web: raw
        websecure: false

And then deploy, you’ll notice that the event parameter looks different:

{
  "__ow_body": "{\"a\":\"b\"}",
  "__ow_headers": {
    "accept": "*/*",
    "accept-encoding": "gzip",
    "cdn-loop": "cloudflare",
    "cf-connecting-ip": "<ip>",
    "cf-ipcountry": "CA",
    "cf-ray": "78ccf506cef2a216-YYZ",
    "cf-visitor": "{\"scheme\":\"https\"}",
    "content-type": "application/json",
    "host": "ccontroller",
    "user-agent": "curl/7.81.0",
    "x-forwarded-for": "<ip>",
    "x-forwarded-proto": "https",
    "x-request-id": "15cd6d3d8d4df319c018888983a13a84"
  },
  "__ow_isBase64Encoded": false,
  "__ow_method": "post",
  "__ow_path": "",
  "__ow_query": "",
  "http": {
    "body": "{\"a\":\"b\"}",
    "headers": {
      "accept": "*/*",
      "accept-encoding": "gzip",
      "cdn-loop": "cloudflare",
      "cf-connecting-ip": "<ip>",
      "cf-ipcountry": "CA",
      "cf-ray": "78ccf506cef2a216-YYZ",
      "cf-visitor": "{\"scheme\":\"https\"}",
      "content-type": "application/json",
      "host": "ccontroller",
      "user-agent": "curl/7.81.0",
      "x-forwarded-for": "<ip>",
      "x-forwarded-proto": "https",
      "x-request-id": "15cd6d3d8d4df319c018888983a13a84"
    },
    "isBase64Encoded": false,
    "method": "POST",
    "path": "",
    "queryString": ""
  }
}

Notice how there is a property called http.body. This is the raw HTTP body that the client sent. It sounds to me like this would be a good way to solve your use case if you want to use this escape hatch to disable the HTTP request parsing that DO does before calling your function code.

One thing you’ll notice too when using web: raw mode is that if your content type is one associated with binary data (like image/png or application/octet-stream), then the request body will be Base64-encoded:

> file 1by1.png 
1by1.png: PNG image data, 1 x 1, 8-bit/color RGB, non-interlaced
> curl -XPOST -H 'Content-Type: image/png' --data-binary "@1by1.png" $URL
{
  "__ow_body": "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAAAMSURBVBhXY/j//z8ABf4C/qc1gYQAAAAASUVORK5CYII=",
  "__ow_headers": {
    "accept": "*/*",
    "accept-encoding": "gzip",
    "cdn-loop": "cloudflare",
    "cf-connecting-ip": "<ip>",
    "cf-ipcountry": "CA",
    "cf-ray": "78cd03ee78043ff7-YYZ",
    "cf-visitor": "{\"scheme\":\"https\"}",
    "content-type": "image/png",
    "host": "ccontroller",
    "user-agent": "curl/7.81.0",
    "x-forwarded-for": "<ip>",
    "x-forwarded-proto": "https",
    "x-request-id": "32bfac3ebd304897d5e9c033d2b6a392"
  },
  "__ow_isBase64Encoded": true,
  "__ow_method": "post",
  "__ow_path": "",
  "__ow_query": "",
  "http": {
    "body": "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAAAMSURBVBhXY/j//z8ABf4C/qc1gYQAAAAASUVORK5CYII=",
    "headers": {
      "accept": "*/*",
      "accept-encoding": "gzip",
      "cdn-loop": "cloudflare",
      "cf-connecting-ip": "<ip>",
      "cf-ipcountry": "CA",
      "cf-ray": "78cd03ee78043ff7-YYZ",
      "cf-visitor": "{\"scheme\":\"https\"}",
      "content-type": "image/png",
      "host": "ccontroller",
      "user-agent": "curl/7.81.0",
      "x-forwarded-for": "<ip>",
      "x-forwarded-proto": "https",
      "x-request-id": "32bfac3ebd304897d5e9c033d2b6a392"
    },
    "isBase64Encoded": true,
    "method": "POST",
    "path": "",
    "queryString": ""
  }
}

So even when you’re sending binary data instead of text data, this escape hatch will work when you want access to the raw request body.

Note, however, that Functions currently doesn’t support streaming. The request body must be able to fit in the memory you provision for your function because the Functions runtime will attempt to deserialize it into a Python dict in memory when your function is invoked.

If you want to fall back to your backup plan to strip out the properties other than ones you want in your data model, I think this is a good option too. I know you stated that your code might break in the future if you do this, but you might be able to put together something stable depending on the details of your use case.

For example, if you have the following JSON Schema used to validate your event parameter:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "properties": {
    "a": {
      "type": "string"
    }
  },
  "required": ["a"],
  "additionalProperties": false
}

Then it is true that the event passed in would fail this validation. The event coming in might look like:

{
  "__ow_headers": {
    "accept": "*/*",
    "accept-encoding": "gzip",
    "cdn-loop": "cloudflare",
    "cf-connecting-ip": "<ip>",
    "cf-ipcountry": "CA",
    "cf-ray": "78cce2092ee6a20e-YYZ",
    "cf-visitor": "{\"scheme\":\"https\"}",
    "host": "ccontroller",
    "user-agent": "curl/7.81.0",
    "x-forwarded-for": "<ip>",
    "x-forwarded-proto": "https",
    "x-request-id": "c03756a0503ee64203016478690e1e1f"
  },
  "__ow_method": "get",
  "__ow_path": "",
  "a": "b",
  "http": {
    "headers": {
      "accept": "*/*",
      "accept-encoding": "gzip",
      "cdn-loop": "cloudflare",
      "cf-connecting-ip": "<ip>",
      "cf-ipcountry": "CA",
      "cf-ray": "78cce2092ee6a20e-YYZ",
      "cf-visitor": "{\"scheme\":\"https\"}",
      "host": "ccontroller",
      "user-agent": "curl/7.81.0",
      "x-forwarded-for": "<ip>",
      "x-forwarded-proto": "https",
      "x-request-id": "c03756a0503ee64203016478690e1e1f"
    },
    "method": "GET",
    "path": ""
  }
}

The properties __ow_headers, __ow_method, __ow_path, and http make it fail this validation.

But you could then transform the event to one that only has the properties you know you need for your validation:

for key in list(event.keys()):
  if key != "a":
    obj.pop(key)

Then, if you validated it, the extra properties provided by the Functions runtime wouldn’t cause validation to fail.

This example used JSON Schema for validation, but this concept should apply to any validation done by you or frameworks you use that use the event data.