Pandas to SQL – importing CSV data files into PostgreSQL

My goal with this post is to cover what I have learned while inserting pandas DataFrame values into a PostgreSQL table using SQLAlchemy. Interested in learning about this yourself? Want to see a simple example? You are in the right place so keep reading and learn with me…

• Xubuntu Linux 18.04.2 LTS (Bionic Beaver)
• PostgreSQL 11.4
• Python 3.7.4
• pandas-0.25.0

Self-Promotion:

If you enjoy the content written here, by all means, share this blog and your favorite post(s) with others who may benefit from or like it as well. Since coffee is my favorite drink, you can even buy me one if you would like!

The final destination for all of my walking stats is this PostgreSQL table I have on my local learning/development environment:

Also as part of the schema, I have a ‘staging’ table (description provided below) where I import all records from a CSV file. As I mentioned in the opening paragraph, we’ll populate it with, SQLAlchemy and pandas.

Getting started, we create a connection to the database with SQLAlchemy’s create_engine object:

From the SQLAlchemy engine configuration page, we can see the basic structure and syntax is relatively straight-forward:

Using a common convention, I’ll get pandas imported:

This is the structure of the staging table:

The staging table is simply a mirror of the ‘stats’ table, with the exception that all columns are implemented as a TEXT data type.

CSV file with April’s walking stats in hand, let’s create a pandas DataFrame object from it with the read_csv() method (Check out this post I wrote on this method and other handy pandas functionality goodies):

Then view the first 5 rows of data using the head() function:

Table ‘stat_staging’ is empty at this time:

But, we are about to change all that and INSERT some data with pandas. But first, I need to import the String type from sqlalchemy:

Next, let’s compose – and execute – the INSERT operation in pandas:

You likely know why I needed the String() data type available. I am type casting the DataFrame column values to a compatible data type – String() – for table ‘stat_staging’, TEXT columns. The dtype{} dictionary parameter is what enables you to set the column data types.

Table ‘stat_staging’ has the DataFrame object records now:

How simple was that? Now all I need to do is type cast the column values – from table ‘stat_staging’ – to the appropriate data types during the INSERT when I move the rows of data over to table ‘stats’. I wrote all about it right here so you should definitely read this post if you are interested in how I accomplished it in PostgreSQL.

Prior to closing out this post, I want to call attention to perhaps, the most important parameter in the above call to to_sql(): if_exists.

The snippet below from the to_sql() documentation page, shows what values are acceptable, with their individual meanings:

If you don’t specify anything, the INSERT will fail. Which is good, in my opinion, that way you don’t overwrite records unintentionally. Both replace and append are self-explanatory.

If your thinking is like mine, this thought has crossed your mind: “Since I can type cast the column values with SQLAlchemy, can’t I just skip the whole ‘staging’ table gambit and load them directly into table ‘stats’?”

And that my friends, is another post for another day. Check back in when it drops and see what it is all about!

Like what you have read? See anything incorrect? Please comment below and thanks for reading!!!

A Call To Action!

Thank you for taking the time to read this post. I truly hope you discovered something interesting and enlightening. Please share your findings here, with someone else you know who would get the same value out of it as well.

Visit the Portfolio-Projects page to see blog post/technical writing I have completed for clients.

Have I mentioned how much I love a cup of coffee?!?!

To receive email notifications (Never Spam) from this blog (“Digital Owl’s Prose”) for the latest blog posts as they are published, please subscribe (of your own volition) by clicking the ‘Click To Subscribe!’ button in the sidebar on the homepage! (Feel free at any time to review the Digital Owl’s Prose Privacy Policy Page for any questions you may have about: email updates, opt-in, opt-out, contact forms, etc…)

Be sure and visit the “Best Of” page for a collection of my best blog posts.

Josh Otwell has a passion to study and grow as a SQL Developer and blogger. Other favorite activities find him with his nose buried in a good book, article, or the Linux command line. Among those, he shares a love of tabletop RPG games, reading fantasy novels, and spending time with his wife and two daughters.

Disclaimer: The examples presented in this post are hypothetical ideas of how to achieve similar types of results. They are not the utmost best solution(s). The majority, if not all, of the examples provided, is performed on a personal development/learning workstation-environment and should not be considered production quality or ready. Your particular goals and needs may vary. Use those practices that best benefit your needs and goals. Opinions are my own.