The Snowflake web-based data loader is incredibly painful and clunky to use in my opinion, and so I have just been using Einblick to throw data onto there — whether a demo CSV someone wants to use, or automating movement from our Hubspot CRM to Snowflake, I’ve run this quick script dozens of times this year.
You can upload a CSV and drop it directly in as a dataframe. Or finish your whole notebook and then just plug the results dataframe in and write it to database.
!pip install sqlalchemy #https://docs.sqlalchemy.org/en/14/core/connections.html
!pip install snowflake.sqlalchemy #this example uses Snowflake
from sqlalchemy import create_engine
from snowflake.sqlalchemy import URL
#set your connection to Snowflake -- you can use any DB here with SQLAlchemy
engine = create_engine(URL(
account = 'xx',
user = 'xx',
password = 'xx',
database = 'xx',
schema = 'xx',
warehouse = 'xx',
role='xx'
))
con=engine.connect()
chunksize = 10000 ##how many rows to send at a time
tablename = 'xx' ##what table to write into
df.to_sql(name=tablename, con=engine.connect(), if_exists='append', index=False, index_label=None, chunksize=chunksize)
About
Einblick is an agile data science platform that provides data scientists with a collaborative workflow to swiftly explore data, build predictive models, and deploy data apps. Founded in 2020, Einblick was developed based on six years of research at MIT and Brown University. Einblick customers include Cisco, DARPA, Fuji, NetApp and USDA. Einblick is funded by Amplify Partners, Flybridge, Samsung Next, Dell Technologies Capital, and Intel Capital. For more information, please visit www.einblick.ai and follow us on LinkedIn and Twitter.