Solving the Issue of Record Limitations

[mcmasty]

@Katherine

Not sure if you have solved your data record challenges. I’ve implemented a Data warehouse + BI Tool solution for my existing client. I’ve open-sourced some of my internal code.

More details on this thread.

ETL, Data Pipelines, Data Warehousing

I have built a Python ETL tool that exports every record in every table in the knack database and saves it to data warehouse. The primary driver for my client is the steep cost of adding rows. In order to keep his database below the threshold on plan, we are purge no longer needed records. In this particular use case, not every record is needed in the “live app,” so we are purging rows and archiving in a data warehouse (Motherduck) and building simple BI reports (Preset.io) to access the histo…