Database Sync

The database sync framework allows tables between two databases with just a few lines of code. Currently supported database types are:

The DBSync class is not a connector, but rather a class that joins in database classes and moves data seamlessly between them.

Quick Start

Full Sync Of Tables

Copy all data from a source table to a destination table.

# Create source and destination database objects
source_rs = Redshift()
destination_rs = Postgres()

# Create db sync object and run sync.
db_sync = DBSync(source_rs, destination_rs) # Create DBSync Object
db_sync.table_sync_full('parsons.source_data', 'parsons.destination_data')

Incremental Sync of Tables

Copy just new data in the table. Utilize this method for tables with distinct primary keys.

# Create source and destination database objects
source_rs = Postgres()
destination_rs = Postgres()

# Create db sync object and run sync.
db_sync = DBSync(source_pg, destination_pg) # Create DBSync Object
db_sync.table_sync_incremental('parsons.source_data', 'parsons.destination_data', 'myid')

API

class parsons.DBSync(source_db, destination_db, chunk_size=100000)[source]

Sync tables between databases. Works with Postgres, Redshift, MySQL databases.

Args:
source_db: Database connection object

A database object.

destination_db: Database connection object

A database object.

chunk_size: int

The number of rows per transaction copy when syncing a table. The default value is 100,000 rows.

Returns:

A DBSync object.

table_sync_full(source_table, destination_table, if_exists='drop', **kwargs)[source]

Full sync of table from a source database to a destination database. This will wipe all data from the destination table.

Args:
source_table: str

Full table path (e.g. my_schema.my_table)

destination_table: str

Full table path (e.g. my_schema.my_table)

if_exists: str

If destination table exists either drop or truncate. Truncate is useful when there are dependent views associated with the table.

**kwargs: args

Optional copy arguments for destination database.

Returns:

None

table_sync_incremental(source_table, destination_table, primary_key, distinct_check=True, **kwargs)[source]

Incremental sync of table from a source database to a destination database using an incremental primary key.

Args:
source_table: str

Full table path (e.g. my_schema.my_table)

destination_table: str

Full table path (e.g. my_schema.my_table)

if_exists: str

If destination table exists either drop or truncate. Truncate is useful when there are dependent views associated with the table.

primary_key: str

The name of the primary key. This must be the same for the source and destination table.

distinct_check: bool

Check that the source table primary key is distinct prior to running the sync. If it is not, an error will be raised.

**kwargs: args

Optional copy arguments for destination database.

Returns:

None