Question
How to export data from DigitalOcean to Google Cloud Services (BigQuery)?
- Posted on December 12, 2022
- DigitalOcean Managed MySQL Database
Asked by Arseniy
Hi!
Is there a way to send data from DigitalOcean (DO) to Google Cloud Services (GCS) by standard means (whithout intermediate soft)?
- I followed step by step Datastream setup with DO as a source for BigQuery as a destination. All intermediate tests passed (including connection test) but when i actually start Datastream it fails due to some unspecified connection problem. No specific error code shown.
- I also tried to migrate DO to GCS by standard migration service of GCS and it also fails everytime (failed on full dump).
I see there’s a number of 3rd party integration software to perform such task which drives me into thinking DO might not be working with GCS by default. Are DigitalOcean and Google Sloud Service compatible at all?
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.
Hollie's Hub for Good
Working on improving health and education, reducing inequality, and spurring economic growth? We’d like to help.
Hello! When I follow the steps described here: https://cloud.google.com/datastream/docs/configure-your-source-postgresql-database#selfhostedpostgresql I come into ‘ALTER ROLE WITH REPLICATION’ , however I can only do this as a superuser, even doadmin cannot do this (even though doadmin has the replication rights). We do not want to use doadmin . Is there any way to do differently?
Thanks a lot
I created empty database and set up datastream and it worked fine (status “working”). The moment i added table into it datatstream failed with the following error “The log file position has been lost. This might be due to a long disconnect or error, a purge on the log, or a short log retention policy. This stream will need to be recreated.”
So i pretty much sure it’s about expire_logs_days set to 0 by default. I just can’t understand how to change it.
Hi!
I have a hypothesis the issue is due to expire_logs_days=0 in DO by default.
Is there any way to set it =7 days as it’s required on GCS side?
When trying “set global expire_logs_days=7;” on MySQL Workbench i get an error “Error Code: 1227. Access denied; you need (at least one of) the SUPER or SYSTEM_VARIABLES_ADMIN privilege(s) for this operation”
Ho do i get those permissions on it there a way to change expire_logs_days parameter at all in DO?