Google
Google Cloud services utilize a credentials JSON file for authentication. If you are the administrator of your Google Cloud account, you can generate them in the Google Cloud Console APIs and Services.
BigQuery
Overview
Google BigQuery is a cloud data warehouse solution. Data is stored in tables, and users can query using SQL. BigQuery uses datasets as top level containers for tables, and datasets are themselves contained within Google Cloud projects.
API
- class parsons.GoogleBigQuery(app_creds=None, project=None, location=None)[source]
Class for querying BigQuery table and returning the data as Parsons tables.
This class requires application credentials in the form of a json. It can be passed in the following ways:
Set an environmental variable named
GOOGLE_APPLICATION_CREDENTIALS
with the local path to the credentials json.Example:
GOOGLE_APPLICATION_CREDENTALS='path/to/creds.json'
Pass in the path to the credentials using the
app_creds
argument.Pass in a json string using the
app_creds
argument.
- Args:
- project_id: str
The project which the client is acting on behalf of. If not passed then will use the default inferred environment.
- app_creds: str
A credentials json string or a path to a json file. Not required if
GOOGLE_APPLICATION_CREDENTIALS
env variable set.- location: str
Default geographic location for tables
- copy(table_obj, table_name, if_exists='fail', tmp_gcs_bucket=None, gcs_client=None, job_config=None, **load_kwargs)[source]
Copy a Parsons Table into Google BigQuery via Google Cloud Storage.
- Args:
- table_obj: obj
The Parsons Table to copy into BigQuery.
- table_name: str
The table name to load the data into.
- if_exists: str
If the table already exists, either
fail
,append
,drop
ortruncate
the table.- temp_gcs_bucket: str
The name of the Google Cloud Storage bucket to use to stage the data to load into BigQuery. Required if GCS_TEMP_BUCKET is not specified.
- gcs_client: object
The GoogleCloudStorage Connector to use for loading data into Google Cloud Storage.
- job_config: object
A LoadJobConfig object to provide to the underlying call to load_table_from_uri on the BigQuery client. The function will create its own if not provided.
- **load_kwargs: kwargs
Arguments to pass to the underlying load_table_from_uri call on the BigQuery client.
- delete_table(table_name)[source]
Delete a BigQuery table.
- Args:
- table_name: str
The name of the table to delete.
- query(sql, parameters=None)[source]
Run a BigQuery query and return the results as a Parsons table.
To include python variables in your query, it is recommended to pass them as parameters, following the BigQuery style where parameters are prefixed with @`s. Using the ``parameters` argument ensures that values are escaped properly, and avoids SQL injection attacks.
Parameter Examples
name = “Beatrice O’Brady” sql = ‘SELECT * FROM my_table WHERE name = %s’ rs.query(sql, parameters=[name])
name = “Beatrice O’Brady” sql = “SELECT * FROM my_table WHERE name = %(name)s” rs.query(sql, parameters={‘name’: name})
- Args:
- sql: str
A valid BigTable statement
- parameters: dict
A dictionary of query parameters for BigQuery.
- Returns:
- Parsons Table
See Parsons Table for output options.
- table_exists(table_name)[source]
Check whether or not the Google BigQuery table exists in the specified dataset.
- Args:
- table_name: str
The name of the BigQuery table to check for
- Returns:
- bool
True if the table exists in the specified dataset, false otherwise
- property client
Get the Google BigQuery client to use for making queries.
- Returns:
google.cloud.bigquery.client.Client
:inherited-members:
Cloud Storage
Overview
Google Cloud Storage is a cloud file storage system. It uses buckets in which to store arbitrary files referred to as blobs.
API
- class parsons.GoogleCloudStorage(app_creds=None, project=None)[source]
This class requires application credentials in the form of a json. It can be passed in the following ways:
Set an environmental variable named
GOOGLE_APPLICATION_CREDENTIALS
with the local path to the credentials json.Example:
GOOGLE_APPLICATION_CREDENTALS='path/to/creds.json'
Pass in the path to the credentials using the
app_creds
argument.Pass in a json string using the
app_creds
argument.
- Args:
- credentials: str
A credentials json string or a path to a json file. Not required if
GOOGLE_APPLICATION_CREDENTIALS
env variable set.- project: str
The project which the client is acting on behalf of. If not passed then will use the default inferred environment.
- Returns:
GoogleCloudStorage Class
- client
Access all methods of google.cloud package
- bucket_exists(bucket_name)[source]
Verify that a bucket exists
- Args:
- bucket_name: str
The name of the bucket
- Returns:
boolean
- get_bucket(bucket_name)[source]
Returns a bucket object
- Args:
- bucket_name: str
The name of bucket
- Returns:
GoogleCloud Storage bucket
- create_bucket(bucket_name)[source]
Create a bucket.
- Args:
- bucket_name: str
A globally unique name for the bucket.
- Returns:
None
- delete_bucket(bucket_name, delete_blobs=False)[source]
Delete a bucket. Will fail if not empty unless
delete_blobs
argument is set toTrue
.- Args:
- bucket_name: str
The name of the bucket
- delete_blobs: boolean
Delete blobs in the bucket, if it is not empty
- Returns:
None
- list_blobs(bucket_name, max_results=None, prefix=None)[source]
List all of the blobs in a bucket
- Args:
- bucket_name: str
The name of the bucket
- max_results: int
TBD
- prefix_filter: str
A prefix to filter files
- Returns:
A list of blob names
- blob_exists(bucket_name, blob_name)[source]
Verify that a blob exists in the specified bucket
- Args:
- bucket_name: str
The bucket name
- blob_name: str
The name of the blob
- Returns:
boolean
- get_blob(bucket_name, blob_name)[source]
Get a blob object
- `Args:
- bucket_name: str
A bucket name
- blob_name: str
A blob name
- Returns:
A Google Storage blob object
- put_blob(bucket_name, blob_name, local_path)[source]
Puts a blob (aka file) in a bucket
- Args:
- blob_name:
The name of blob to be stored in the bucket
- bucket_name:
The name of the bucket to store the blob
- local_path: str
The local path of the file to upload
- Returns:
None
- download_blob(bucket_name, blob_name, local_path=None)[source]
Gets a blob from a bucket
- Args:
- bucket_name: str
The name of the bucket
- blob_name: str
The name of the blob
- local_path: str
The local path where the file will be downloaded. If not specified, a temporary file will be created and returned, and that file will be removed automatically when the script is done running.
- Returns:
- str
The path of the downloaded file
- delete_blob(bucket_name, blob_name)[source]
Delete a blob
- Args:
- bucket_name: str
The bucket name
- blob_name: str
The blob name
- Returns:
None
- upload_table(table, bucket_name, blob_name, data_type='csv', default_acl=None)[source]
Load the data from a Parsons table into a blob.
- Args:
- table: obj
- bucket_name: str
The name of the bucket to upload the data into.
- blob_name: str
The name of the blob to upload the data into.
- data_type: str
The file format to use when writing the data. One of: csv or json
:inherited-members:
Civic
Overview
Google Civic is an API which provides helpful information about elections. In order to access Google Civic you must create a Google Developer Key in their API console. In order to use Google Civic, you must enable this specific end point.
The Google Civic API utilizes the Voting Information Project to collect key civic information such as personalized ballots and polling location information.
API
- class parsons.GoogleCivic(api_key=None)[source]
- Args:
- api_keystr
A valid Google api key. Not required if
GOOGLE_CIVIC_API_KEY
env variable set.
- Returns:
class
- get_elections()[source]
Get a collection of information about elections and voter information.
- Returns:
- Parsons Table
See Parsons Table for output options.
- get_polling_location(election_id, address)[source]
Get polling location information for a given address.
- Args:
- election_id: int
A valid election id. Election ids can be found by running the
get_elections()
method.- address: str
A valid US address in a single string.
- Returns:
- Parsons Table
See Parsons Table for output options.
- get_polling_locations(election_id, table, address_field='address')[source]
Get polling location information for a table of addresses.
- Args:
- election_id: int
A valid election id. Election ids can be found by running the
get_elections()
method.- address: str
A valid US address in a single string.
- address_field: str
The name of the column where the address is stored.
- Returns:
- Parsons Table
See Parsons Table for output options.
:inherited-members:
Google Sheets
Overview
The GoogleSheets class allows you to interact with a Google Drive spreadsheet.
In order to instantiate the class, you must pass Google Drive credentials as a dictionary, or store the credentials as a JSON string in the GOOGLE_DRIVE_CREDENTIALS
environment variable. Typically you’ll get the credentials from the Google Developer Console (look for the “Google Drive API”).
API
- class parsons.GoogleSheets(google_keyfile_dict=None)[source]
A connector for Google Sheets, handling data import and export.
- Args:
- google_keyfile_dict: dict
A dictionary of Google Drive API credentials, parsed from JSON provided by the Google Developer Console. Required if env variable
GOOGLE_DRIVE_CREDENTIALS
is not populated.
- list_worksheets(spreadsheet_id)[source]
Return a list of worksheets in the spreadsheet.
- Args:
- spreadsheet_id: str
The ID of the spreadsheet (Tip: Get this from the spreadsheet URL)
- Returns:
- list
A List of worksheets order by their index
- get_worksheet_index(spreadsheet_id, title)[source]
Get the first sheet in a Google spreadsheet with the given title. The title is case sensitive and the index begins with 0.
- Args:
- spreadsheet_id: str
The ID of the spreadsheet (Tip: Get this from the spreadsheet URL)
- title: str
The sheet title
- Returns:
- str
The sheet index
- get_worksheet(spreadsheet_id, worksheet=0)[source]
Create a
parsons table
from a sheet in a Google spreadsheet, given the sheet index.- Args:
- spreadsheet_id: str
The ID of the spreadsheet (Tip: Get this from the spreadsheet URL)
- worksheet: str or int
The index or the title of the worksheet. The index begins with 0.
- Returns:
- Parsons Table
See Parsons Table for output options.
Share a spreadsheet with a user, group of users, domain and/or the public.
- Args:
- spreadsheet_id: str
The ID of the spreadsheet (Tip: Get this from the spreadsheet URL)
- sharee: str
User or group e-mail address, domain name to share the spreadsheet with. To share publicly, set sharee value to
None
.- share_type: str
The sharee type. Allowed values are:
user
,group
,domain
,anyone
.- role: str
The primary role for this user. Allowed values are:
owner
,writer
,reader
.- notify: boolean
Whether to send an email to the target user/domain.
- email_message: str
The email to be sent if notify kwarg set to True.
- with_link: boolean
Whether a link is required for this permission.
- get_spreadsheet_permissions(spreadsheet_id)[source]
List the permissioned users and groups for a spreadsheet.
- Args:
- spreadsheet_id: str
The ID of the spreadsheet (Tip: Get this from the spreadsheet URL)
- Returns:
- Parsons Table
See Parsons Table for output options.
- create_spreadsheet(title, editor_email=None)[source]
Create a Google spreadsheet from a Parsons table. Optionally shares the new doc with the given email address.
- Args:
- title: str
The human-readable title of the new spreadsheet
- editor_email: str (optional)
Email address which should be given permissions on this spreadsheet
- Returns:
- str
The spreadsheet ID
- delete_spreadsheet(spreadsheet_id)[source]
Deletes a Google spreadsheet.
- Args:
- spreadsheet_id: str
The ID of the spreadsheet (Tip: Get this from the spreadsheet URL)
- add_sheet(spreadsheet_id, title=None, rows=100, cols=25)[source]
Adds a sheet to a Google spreadsheet.
- Args:
- spreadsheet_id: str
The ID of the spreadsheet (Tip: Get this from the spreadsheet URL)
- rows: int
Number of rows
- cols
Number of cols
- Returns:
- str
The sheet index
- append_to_sheet(spreadsheet_id, table, worksheet=0, user_entered_value=False, **kwargs)[source]
Append data from a Parsons table to a Google sheet. Note that the table’s columns are ignored, as we’ll be keeping whatever header row already exists in the Google sheet.
- Args:
- spreadsheet_id: str
The ID of the spreadsheet (Tip: Get this from the spreadsheet URL)
- table: obj
Parsons table
- worksheet: str or int
The index or the title of the worksheet. The index begins with 0.
- user_entered_value: bool (optional)
If True, will submit cell values as entered (required for entering formulas). Otherwise, values will be entered as strings or numbers only.
- overwrite_sheet(spreadsheet_id, table, worksheet=0, user_entered_value=False, **kwargs)[source]
Replace the data in a Google sheet with a Parsons table, using the table’s columns as the first row.
- Args:
- spreadsheet_id: str
The ID of the spreadsheet (Tip: Get this from the spreadsheet URL)
- table: obj
Parsons table
- worksheet: str or int
The index or the title of the worksheet. The index begins with 0.
- user_entered_value: bool (optional)
If True, will submit cell values as entered (required for entering formulas). Otherwise, values will be entered as strings or numbers only.