Question

this.form.submit() refreshes the page, clears and ignores selection from dropdown (Selenium, Python, Docker)

I’m developing a data scraper using Selenium and Python. This data scraper is supposed to be run on Digital Ocean Kubernetes pod as a Docker image.

Local scenario:

I build the Docker image.
I run the Docker image successfully.
Data scraper selects an option from dropdown.
this.form.submit() gets executed.
Page refreshes and selected option gets applied, just as expected.

Server scenario:

I build the Docker image.
I push it to the server and it starts.
Data scraper selects an option from dropdown.
this.form.submit() gets executed.
Page refreshes and the option gets cleared.

Background: Kubernetes droplet has 8gb RAM, 4CPUs, so the resources doesn’t seem to be the problem. It doesn’t seem to have anything to do with timings as well, because I have tried all the delays possible - implicit, explicit and Javascript injection to wait before form submission.

The unbeatable problem: Why does webdriver browser of the same Docker image execute this.form.submit() without any problems on my local machine, but can’t successfully execute this.form.submit() within a pod of Kubernetes?

Side note: I have noticed that whenever a cookies consent dialog appears (which happens rarely for unknown reasons to me) and the scraper accepts the cookies, this.form.submit() executes successfully and option gets selected after the page refresh.

I have been trying to solve this for almost a week now and without any success, I’m hoping to get an answer here. ANY answer is highly appreciated and I will be as responsive as possible to let you know about the outcomes.

Submit an answer

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Bobby Iliev

Site Moderator

• May 6, 2024

Hey!

Indeed, this sounds very tricky to troubleshoot. One thing that comes to my mind here is to add some extra logging to see where exactly in the code the problem might be arrising.

For example, add logging statements before and after form.submit() to ensure that the proper dropdown option is selected and visible before the form is submitted, eg:

import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

logger.info(f"Selected option before submission: {selected_option}")
logger.info("Submitting the form...")
form.submit()
logger.info("Form submitted.")

That way you will be able to see where the two environments are diverging.

Also, as you’ve observed that the cookies consent dialog appears occasionally, you can try explicitly accepting the consent dialog to make sure that it is not blocking the submission.

If you are still seeing the issue, feel free to share your Dockerfile and a code snippet that I can use to reproduce this on my end and try to further troubleshoot it!

Best,

Bobby