Pipeline Configuration Documentation¶
- class utils.ConfigManager.ConfigManager(config_file_path: str = 'config.ini', input_directory: str = None, output_directory: str = None, data_directory: str = None, eco_file_structure: bool = False)¶
A helpful object to manage configuration
- Attributes:
- config_file_pathstr
The path to the config.ini file for the pipeline (e.g. “full/path/to/config.ini”). Defaults to “config.ini” This file should contain login information for MySQL database where data is to be loaded.
- input_directorystr
The path to the input directory for the pipeline (e.g. “full/path/to/pipeline/input/”). Defaults to the input directory defined in the config.ini configuration file
- output_directorystr
The path to the output directory for the pipeline (e.g. “full/path/to/pipeline/output/”). Defaults to the output directory defined in the config.ini configuration file
- data_directorystr
The path to the data directory for the pipeline (e.g. “full/path/to/pipeline/data/”). Defaults to the data directory defined in the config.ini configuration file
- eco_file_structureboolean
Set to True if this is a data pipeline running on Ecotope’s server for file path reconfiguration. Set False if not running at Ecotope. Defaults to False
Methods
Create a connection with the mySQL server.
returns name of database that data will be uploaded to
get_db_table_info
(table_headers)Reads the config.ini file stored in the config_file_path file path.
Returns path to the full path to the Event_Log.csv file.
get_site_name
([config_key])returns name of site
Returns path to the full path to the Variable_Names.csv file.
Returns path to the directory that holds NOAA weather data files.
get_fm_device_id
get_fm_token
get_table_name
get_thingsboard_token
- connect_db() -> (<class 'mysql.connector.connection.MySQLConnection'>, <class 'mysql.connector.cursor.MySQLCursor'>)¶
Create a connection with the mySQL server.
- Parameters:
- None
- Returns:
- mysql.connector.MySQLConnection, mysql.connector.cursor.MySQLCursor:
A connection and cursor object. THe cursor can be used to execute mySQL queries and the connection object can be used to save those changes.
- get_db_name()¶
returns name of database that data will be uploaded to
- get_db_table_info(table_headers: list) dict ¶
Reads the config.ini file stored in the config_file_path file path.
- Parameters:
- table_headerslist
A list of table headers. These headers must correspond to the section headers in the config.ini file. Your list must contain the section header for each table you wish to write into. The first header must correspond to the login information of the database. The other are the tables which you wish to write to.
- Returns:
- dict:
A dictionary containing all relevant information is returned. This includes information used to create a connection with a mySQL server and information (table names and column names) used to load the data into tables.
- get_event_log_path() str ¶
Returns path to the full path to the Event_Log.csv file. This file should be in the pipeline’s input directory “/” (i.e. “full/path/to/pipeline/input/Event_Log.csv”)
- get_site_name(config_key: str = 'minute')¶
returns name of site
- Parameters:
- config_keystr
The key in the config.ini file that points to the minute table data for the site. The name of this table is also the site name.
- get_var_names_path() str ¶
Returns path to the full path to the Variable_Names.csv file. This file should be in the pipeline’s input directory “/” (i.e. “full/path/to/pipeline/input/Variable_Names.csv”)
- get_weather_dir_path() str ¶
Returns path to the directory that holds NOAA weather data files. This diectory should be in the pipeline’s data directory “/” (i.e. “full/path/to/pipeline/data/weather”)