Database Sessions and User facing API

June 01, 2017

Newcoder Scraping (Part 4)
SQLAlchemy Session Basics

Context
I struggled with structuring database sessions in my python program.
Reading through a few articles and tutorials on SQLAlchemy best practices, and studying Kenneth Reitz’s fantastic requests library, I realize I want a clean interface to the package.
Kenneth does this by having an “api” module that wraps/uses the underlying classes/objects into neat little functions, so you can do request.get("https://www.google.com").

Modules

models.py
This contains all the database models created using sqlalchemy. It’s likely to change and evolve, so thought to best keep it separate from the database module.
I need to look into Alembic to handle migrations. As I understand it, I’ll need that for when the schema changes.

database.py
This contains all the helper methods to connect to connect to the database and manage a session scope when interacting with the database.
I didn’t really find great examples of how to structure database interactions in a non-web context.
SQLAlchemy docs state that the session and transaction should be kept separate from the business logic.

I’ve tried to do this by building methods here and plan on using it in a “run.py” module at the root of the directory.

stoneland.py
This module contains all the scraping code for stoneland.stoneprofits.com and puts it in an object called “Stoneland”.

utils.py
This contains random utility methods that I need in multiple modules. For example a module to lowercase all the keys in a dictionary.
I use that when loading the config files.

__init__.py

First, how to write __init__.py or any underscored word in markdown:

\_\_init\_\_.py

This will contain code to initialize logging and import useful modules, classes and methods.