Below is an overview of the data solution I came up with. I would say data warehouse but this is not a data warehouse. So my solution relies on the data source of google sheets populated by Google form but if you have a better data source that you can access freely then great.
This file is imported on a schedule by the Keboola Engine. The Flow I have created loads the data source in. As I am just starting out with all these systems so it is doing a full load. The file is then stored in the bucket on Keboola and pushed into a MySQL database I have created (again for free in the cloud). I then have a view sitting over the table although that logic could easily be in the next step of the snowflake transformation. The Snowflake transformation takes place on a Snowflake database where you can run a series of SQL statements and processes to trans form your data. The output from this is then pushed back into the MySQL database. I have then connected Google Data Studio to the mySQL database to act as the visualisation software.
No comments:
Post a Comment