Static Site w/ next js custom catchall page not working

I want to be able to setup a custom 404 catchall page for my static site which is built using next js.

I am able to set the catchall to index.html and it works as expected, when the user lands on a page that doesn’t exist they see the home page content. (e.g. domain.com/wrong-path-here)

However, when I change this to 404.html or any other page that isn’t the index.html then the user is shown a 404 NOT FOUND Page instead of the content in that page.

We’ve tried to add paths in the page name, but that is not valid syntax. e.g. (/404.html or ./404.html)