muteria.common.fs module

fs

This module implements some basic file system operation that are useful for loading and storing data

class muteria.common.fs.CheckpointState(store_filepath, backup_filepath)

Bases: object

AGG_TIME_KEY = 'AGGREGATED_TIME'

CHECKPOINT_DATA_KEY = 'CHECKPOINT_DATA'

The different states are: - Destroyed - starting - Executing - Finished (Completed)

DETAILED_TIME_KEY = 'DETAILED_TIME'

EXEC_COMPLETED = 'CHECK_POINTED_TASK_COMPLETED'

EXEC_STARTING = 'CHECK_POINTED_TASK_STARTING'

add_dep_checkpoint_state(dep_cp)

destroy_checkpoint()

get_dep_checkpoint_states()

get_detailed_execution_time()

get_execution_time()

is_destroyed()

is_finished()

load_checkpoint_or_start(ret_detailed_exectime_obj=False)

This function also show a fresh starting of the execution Return None as checkpoint data if start (was not yet started)

If ret_detailed_exectime_obj is enable, return detailed_exectime_obj as second returned value

Note: In case of continue from a checkpoint checkpoint,

the object already in sync with files here (see __init__)

restart_task()

set_finished(detailed_exectime_obj=None)

write_checkpoint(json_obj, detailed_exectime_obj=None)

class muteria.common.fs.FileDirStructureHandling(top_dir, top_dir_key, file_dir_dict)

Bases: object

Can be used for the organization of the output directory. Provides methods to access the files and directories Can get, create, remove, get_or_create files and dirs

clean_create_and_get_dir(dirname, rel_path=False)

dir_exists(dirname)

file_exists(filename)

get_dir_pathname(dirname, rel_path=False)

get_existing_dir_pathname(dirname, rel_path=False)

get_existing_file_pathname(filename, rel_path=False)

get_file_pathname(filename, rel_path=False)

get_or_create_and_get_dir(dirname, rel_path=False)

remove_dir_and_get(dirname, rel_path=False)

remove_file_and_get(filename, rel_path=False)

resolve(name)

class muteria.common.fs.TarGz

Bases: object

File preperties preserving archiving (using tar gz)

archive_ext = '.tar.gz'

classmethod compressDir(in_directory, out_archive_pathname=None, remove_in_directory=False)

classmethod compressFile(in_file, out_archive_pathname=None, remove_in_file=False)

classmethod decompressDir(in_archive_pathname, out_directory=None, remove_in_archive=False)

classmethod decompressFile(in_archive_pathname, out_directory=None, remove_in_archive=False)

classmethod extractFromArchive(in_archive_pathname, extract_pathname, out_location=None, is_folder=False)

classmethod get_archive_filename_of(file_dir)

get_element(elem)

is_archive_file()

Return True if name points to a tar archive that we are able to handle, else return False.

‘name’ should be a string, file, or file-like object.

open_read_flag = 'r:gz'

open_write_flag = 'w:gz'

classmethod opening_function(name=None, mode='r', fileobj=None, bufsize=10240, **kwargs)

Open a tar archive for reading, writing or appending. Return an appropriate TarFile class.

mode: ‘r’ or ‘r:*’ open for reading with transparent compression ‘r:’ open for reading exclusively uncompressed ‘r:gz’ open for reading with gzip compression ‘r:bz2’ open for reading with bzip2 compression ‘r:xz’ open for reading with lzma compression ‘a’ or ‘a:’ open for appending, creating the file if necessary ‘w’ or ‘w:’ open for writing without compression ‘w:gz’ open for writing with gzip compression ‘w:bz2’ open for writing with bzip2 compression ‘w:xz’ open for writing with lzma compression

‘x’ or ‘x:’ create a tarfile exclusively without compression, raise

an exception if the file is already created

‘x:gz’ create a gzip compressed tarfile, raise an exception

if the file is already created

‘x:bz2’ create a bzip2 compressed tarfile, raise an exception

if the file is already created

‘x:xz’ create an lzma compressed tarfile, raise an exception

if the file is already created

‘r|*’ open a stream of tar blocks with transparent compression ‘r|’ open an uncompressed stream of tar blocks for reading ‘r|gz’ open a gzip compressed stream of tar blocks ‘r|bz2’ open a bzip2 compressed stream of tar blocks ‘r|xz’ open an lzma compressed stream of tar blocks ‘w|’ open an uncompressed stream for writing ‘w|gz’ open a gzip compressed stream for writing ‘w|bz2’ open a bzip2 compressed stream for writing ‘w|xz’ open an lzma compressed stream for writing

classmethod removeFromArchive(in_archive_pathname, rem_pathname, is_folder=False)

class muteria.common.fs.Zip

Bases: TarGz

File properties non preserving but with option to addd to archive

classmethod addToArchive(archive_pathname, added_pathname, in_archive_name=None, is_folder=False)

Create archive if not existing

archive_ext = '.zip'

get_element(elem)

is_archive_file()

Quickly see if a file is a ZIP file by checking the magic number.

The filename argument may be a file or file-like object too.

open_read_flag = 'r'

open_write_flag = 'w'

opening_function

alias of ZipFile

muteria.common.fs.dumpCSV(dataframe, out_file_pathname, separator=' ')

Store a dataframe in CSV format into a file.

Parameters:

• dataframe – data to store in CSV format.

• out_file_pathname – Pathname of the CSV file to store the dataframe.

• separator – Separator used in the CSV file.

Returns:

None on success and error message on failure.

muteria.common.fs.dumpJSON(data_object, out_file_pathname, pretty=False)

Store a data object in Json format into a file.

Parameters:

• data_object – data to store in Json format.

• out_file_pathname – Pathname of the Json file to store the data.

• pretty – Enables visual friendly layout of json file (spaces).

Returns:

None on success and error message on failure.

muteria.common.fs.loadCSV(in_file_pathname, separator=' ')

Load a CSV from file and return it as a pandas dataframe.

Parameters:

• in_file_pathname – Pathname of the CSV file to load.

• separator – The separator used in the CSV file. dafault is space.

Returns:

loaded csv data as pandas dataframe.

muteria.common.fs.loadJSON(in_file_pathname)

Load a Json from file and return it as a python object.

Parameters:

in_file_pathname – Pathname of the Json file to load.

Returns:

loaded data as corresponding python object.