How to start a spark cluster on Kubernetes on Digital Ocean step-by-step. On the web I found the following video that explains how to set up Kubernetes cluster (including kubectl) on Digital Ocean: https://www.youtube.com/watch?v=_waZw9jiyhQ. Starting from there i.e. assuming that I have setup my kubectl on my local machine I am interested in the following questions:
- How to prepare a base docker spark image for the Kubernetes on Digital Ocean.
- How to tell Kubernetes which image to use for Spark?
- Has this image be deployed somehow to Digital Ocean? If yes then how?
- What is the best was to transfer a large csv file (~100GB) to Digital Ocean Cloud and enable my spark jobs to access and read that file.
- Can I parquet data format on Digital Ocean? If yes how one can store it?
- How to save the result of the spark jobs on the Digital Ocean Cloud? As the simplies option I would prefer the results to be written into a persistent data storage which is not deleted after the Kubernetes cluster has finished the jobs and ceased to exists. Thank you!
Become a contributor for community
Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.
DigitalOcean Documentation
Full documentation for every DigitalOcean product.
Resources for startups and AI-native businesses
The Wave has everything you need to know about building a business, from raising funding to marketing your product.
The developer cloud
Scale up as you grow — whether you're running one virtual machine or ten thousand.
Start building today
From GPU-powered inference and Kubernetes to managed databases and storage, get everything you need to build, scale, and deploy intelligent applications.
