Question

How to export data from DigitalOcean to Google Cloud Services (BigQuery)?

Hi!

Is there a way to send data from DigitalOcean (DO) to Google Cloud Services (GCS) by standard means (whithout intermediate soft)?

  1. I followed step by step Datastream setup with DO as a source for BigQuery as a destination. All intermediate tests passed (including connection test) but when i actually start Datastream it fails due to some unspecified connection problem. No specific error code shown.
  2. I also tried to migrate DO to GCS by standard migration service of GCS and it also fails everytime (failed on full dump).

I see there’s a number of 3rd party integration software to perform such task which drives me into thinking DO might not be working with GCS by default. Are DigitalOcean and Google Sloud Service compatible at all?


Submit an answer


This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Hello! When I follow the steps described here: https://cloud.google.com/datastream/docs/configure-your-source-postgresql-database#selfhostedpostgresql I come into ‘ALTER ROLE WITH REPLICATION’ , however I can only do this as a superuser, even doadmin cannot do this (even though doadmin has the replication rights). We do not want to use doadmin . Is there any way to do differently?

Thanks a lot

I created empty database and set up datastream and it worked fine (status “working”). The moment i added table into it datatstream failed with the following error “The log file position has been lost. This might be due to a long disconnect or error, a purge on the log, or a short log retention policy. This stream will need to be recreated.”

So i pretty much sure it’s about expire_logs_days set to 0 by default. I just can’t understand how to change it.

Hi!

I have a hypothesis the issue is due to expire_logs_days=0 in DO by default.

Is there any way to set it =7 days as it’s required on GCS side?

When trying “set global expire_logs_days=7;” on MySQL Workbench i get an error “Error Code: 1227. Access denied; you need (at least one of) the SUPER or SYSTEM_VARIABLES_ADMIN privilege(s) for this operation”

Ho do i get those permissions on it there a way to change expire_logs_days parameter at all in DO?

Try DigitalOcean for free

Click below to sign up and get $200 of credit to try our products over 60 days!

Sign up

card icon
Get our biweekly newsletter

Sign up for Infrastructure as a Newsletter.

Sign up
card icon
Hollie's Hub for Good

Working on improving health and education, reducing inequality, and spurring economic growth? We’d like to help.

Learn more
card icon
Become a contributor

You get paid; we donate to tech nonprofits.

Learn more
Welcome to the developer cloud

DigitalOcean makes it simple to launch in the cloud and scale up as you grow – whether you’re running one virtual machine or ten thousand.

Learn more ->
DigitalOcean Cloud Control Panel