Pipeline Configuration Documentation

class utils.ConfigManager.ConfigManager(config_file_path: str = 'config.ini', input_directory: str = None, output_directory: str = None, data_directory: str = None, eco_file_structure: bool = False)

A helpful object to manage configuration

Attributes:

config_file_pathstr

The path to the config.ini file for the pipeline (e.g. “full/path/to/config.ini”). Defaults to “config.ini” This file should contain login information for MySQL database where data is to be loaded.

input_directorystr

The path to the input directory for the pipeline (e.g. “full/path/to/pipeline/input/”). Defaults to the input directory defined in the config.ini configuration file

output_directorystr

The path to the output directory for the pipeline (e.g. “full/path/to/pipeline/output/”). Defaults to the output directory defined in the config.ini configuration file

data_directorystr

The path to the data directory for the pipeline (e.g. “full/path/to/pipeline/data/”). Defaults to the data directory defined in the config.ini configuration file

eco_file_structureboolean

Set to True if this is a data pipeline running on Ecotope’s server for file path reconfiguration. Set False if not running at Ecotope. Defaults to False

Methods

connect_db()	Create a connection with the mySQL server.
get_db_name()	returns name of database that data will be uploaded to
get_db_table_info(table_headers)	Reads the config.ini file stored in the config_file_path file path.
get_event_log_path()	Returns path to the full path to the Event_Log.csv file.
get_site_name([config_key])	returns name of site
get_var_names_path()	Returns path to the full path to the Variable_Names.csv file.
get_weather_dir_path()	Returns path to the directory that holds NOAA weather data files.

get_fm_device_id
get_fm_token
get_table_name
get_thingsboard_token

connect_db() -> (<class 'mysql.connector.connection.MySQLConnection'>, <class 'mysql.connector.cursor.MySQLCursor'>)

Create a connection with the mySQL server.

Parameters:

None

Returns:

mysql.connector.MySQLConnection, mysql.connector.cursor.MySQLCursor:

A connection and cursor object. THe cursor can be used to execute mySQL queries and the connection object can be used to save those changes.

get_db_name()

returns name of database that data will be uploaded to

get_db_table_info(table_headers: list) → dict

Reads the config.ini file stored in the config_file_path file path.

Parameters:

table_headerslist

A list of table headers. These headers must correspond to the section headers in the config.ini file. Your list must contain the section header for each table you wish to write into. The first header must correspond to the login information of the database. The other are the tables which you wish to write to.

Returns:

dict:

A dictionary containing all relevant information is returned. This includes information used to create a connection with a mySQL server and information (table names and column names) used to load the data into tables.

get_event_log_path() → str

Returns path to the full path to the Event_Log.csv file. This file should be in the pipeline’s input directory “/” (i.e. “full/path/to/pipeline/input/Event_Log.csv”)

get_site_name(config_key: str = 'minute')

returns name of site

Parameters:

config_keystr

The key in the config.ini file that points to the minute table data for the site. The name of this table is also the site name.

get_var_names_path() → str

Returns path to the full path to the Variable_Names.csv file. This file should be in the pipeline’s input directory “/” (i.e. “full/path/to/pipeline/input/Variable_Names.csv”)

get_weather_dir_path() → str

Returns path to the directory that holds NOAA weather data files. This diectory should be in the pipeline’s data directory “/” (i.e. “full/path/to/pipeline/data/weather”)