Utilities

Cloud Storage

The Parsons cloud storage utility was created to interact with APIs that require access to files to run an asynchronous process. While this utility currently only works with S3, the goal is add functionality for additional cloud storage services in the future.

The cloud storage utility is currently being utilitized primarily by the NGPVAN class methods such as upload_scores() and Bulk Import methods.

These methods have arguments specific their method, but all also contain the following cloud storage arguments:

  • url_type - The type of cloud storage to utilize. Currently only S3.

  • **url_kwargs - These are arguments specific to the cloud storage type in order to initialize.

S3

When you select the url_type S3, the Parsons table will be converted to a csv and compressed. The file will be posted to an S3 bucket. A presigned public url will be generated and returned. The url will be active by default for 60 minutes, however you may adjust that time.

Argument

Required

Description

bucket

Yes

The S3 bucket to post the file

aws_access_key

No

Required if AWS_ACCESS_KEY_ID env variable not set.

aws_secret_access_key

No

Required if AWS_SECRET_ACCESS_KEY env variable not set.

public_url_expires

No

Defaults is 60 minutes.