By Harry
Hi,
I’m using App Platform forwarding the runtime logs to managed OpenSearch.
The forwarded messages are wrapped inside of DigitalOcean’s wrapped object. For example:
{
"@timestamp": <ts>,
"do_component": "<value>",
...
"message": "<runtime log>"
}
However, my runtime log is JSON object so it gets escaped when log is forwarded to OpenSearch like so:
{
"@timestamp": <ts>,
"do_component": "<value>",
...
"message": "{\"myfield\":\"myvalue\"}"
}
So OpenSearch isn’t able to correctly index the field. Instead it should be:
{
"@timestamp": <ts>,
"do_component": "<value>",
...
"message": {
"myfield": "myvalue"
}
}
What are the ways to achieve this?
I think it makes sense that DO’s log forwarder treats each line as a string, so then i need some sort of transformer between the OpenSearch ingestion but unsure if this is something OpenSearch can provide out of box.
Thanks Harry
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
Accepted Answer
Heya, @4b3bae0d29864a88a7f226aa364fe5
I think you’re right about the root cause here. The App Platform’s log forwarder ships each log line as a string and puts it into a message, so your JSON ends up double-encoded. There isn’t a setting in App Platform today that will “unescape” the message before forwarding.
A potential fix is to parse the message on the OpenSearch side with an ingest pipeline. Create a pipeline that runs a JSON processor against the message field, writes the parsed object into a new field (or overwrites the message), and optionally drops the original string.
Hope that this helps!
Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.
Full documentation for every DigitalOcean product.
The Wave has everything you need to know about building a business, from raising funding to marketing your product.
Stay up to date by signing up for DigitalOcean’s Infrastructure as a Newsletter.
New accounts only. By submitting your email you agree to our Privacy Policy
Scale up as you grow — whether you're running one virtual machine or ten thousand.
Sign up and get $200 in credit for your first 60 days with DigitalOcean.*
*This promotional offer applies to new accounts only.