Skip to content

LudwigModel

LudwigModel class [source]

ludwig.api.LudwigModel(
  config,
  logging_level=40,
  backend=None,
  gpus=None,
  gpu_memory_limit=None,
  allow_parallel_threads=True,
  callbacks=None
)

Class that allows access to high level Ludwig functionalities.

Inputs

  • config (Union[str, dict]): in-memory representation of config or string path to a YAML config file.
  • logging_level (int): Log level that will be sent to stderr.
  • backend (Union[Backend, str]): Backend or string name of backend to use to execute preprocessing / training steps.
  • gpus (Union[str, int, List[int]], default: None): GPUs to use (it uses the same syntax of CUDA_VISIBLE_DEVICES)
  • gpu_memory_limit (float: default: None): maximum memory fraction [0, 1] allowed to allocate per GPU device.
  • allow_parallel_threads (bool, default: True): allow Torch to use multithreading parallelism to improve performance at the cost of determinism.

Example usage:

from ludwig.api import LudwigModel

Train a model:

config = {...}
ludwig_model = LudwigModel(config)
train_stats, _, _ = ludwig_model.train(dataset=file_path)

or

train_stats, _, _ = ludwig_model.train(dataset=dataframe)

If you have already trained a model you can load it and use it to predict

ludwig_model = LudwigModel.load(model_dir)

Predict:

predictions, _ = ludwig_model.predict(dataset=file_path)

or

predictions, _ = ludwig_model.predict(dataset=dataframe)

Evaluation:

eval_stats, _, _ = ludwig_model.evaluate(dataset=file_path)

or

eval_stats, _, _ = ludwig_model.evaluate(dataset=dataframe)

PublicAPI: This API is stable across Ludwig releases.

LudwigModel methods

collect_activations

collect_activations(
  layer_names,
  dataset,
  data_format=None,
  split='full',
  batch_size=128
)

Loads a pre-trained model model and input data to collect the values of the activations contained in the tensors.

Inputs

  • layer_names (list): list of strings for layer names in the model to collect activations.
  • dataset (Union[str, Dict[str, list], pandas.DataFrame]): source containing the data to make predictions.
  • data_format (str, default: None): format to interpret data sources. Will be inferred automatically if not specified. Valid formats are 'auto', 'csv', 'df', 'dict', 'excel', 'feather', 'fwf', 'hdf5' (cache file produced during previous training), 'html' (file containing a single HTML <table>), 'json', 'jsonl', 'parquet', 'pickle' (pickled Pandas DataFrame), 'sas', 'spss', 'stata', 'tsv'.
  • split (str, default= 'full'):: if the input dataset contains a split column, this parameter indicates which split of the data to use. Possible values are 'full', 'training', 'validation', 'test'.
  • batch_size (int, default: 128): size of batch to use when making predictions.

Return

  • return (list): list of collected tensors.

collect_weights

collect_weights(
  tensor_names=None
)

Load a pre-trained model and collect the tensors with a specific name.

Inputs

  • tensor_names (list, default: None): List of tensor names to collect weights

Return

  • return (list): List of tensors

create_model

create_model(
  config_obj,
  random_seed=42
)

Instantiates BaseModel object.

Inputs

  • config_obj (Union[Config, dict]): Ludwig config object
  • random_seed (int, default: ludwig default random seed): Random seed used for weights initialization, splits and any other random function.

Return

  • return (ludwig.models.BaseModel): Instance of the Ludwig model object.

evaluate

ludwig.evaluate(
  dataset=None,
  data_format=None,
  split='full',
  batch_size=None,
  skip_save_unprocessed_output=True,
  skip_save_predictions=True,
  skip_save_eval_stats=True,
  collect_predictions=False,
  collect_overall_stats=False,
  output_directory='results',
  return_type=<class 'pandas.core.frame.DataFrame'>
)

This function is used to predict the output variables given the input variables using the trained model and compute test statistics like performance measures, confusion matrices and the like.

Inputs

  • dataset (Union[str, dict, pandas.DataFrame]): source containing the entire dataset to be evaluated.
  • data_format (str, default: None): format to interpret data sources. Will be inferred automatically if not specified. Valid formats are 'auto', 'csv', 'df', 'dict', 'excel', 'feather', 'fwf', 'hdf5' (cache file produced during previous training), 'html' (file containing a single HTML <table>), 'json', 'jsonl', 'parquet', 'pickle' (pickled Pandas DataFrame), 'sas', 'spss', 'stata', 'tsv'.
  • split (str, default='full'):: if the input dataset contains a split column, this parameter indicates which split of the data to use. Possible values are 'full', 'training', 'validation', 'test'.
  • batch_size (int, default: None): size of batch to use when making predictions. Defaults to model config eval_batch_size
  • skip_save_unprocessed_output (bool, default: True): if this parameter is False, predictions and their probabilities are saved in both raw unprocessed numpy files containing tensors and as postprocessed CSV files (one for each output feature). If this parameter is True, only the CSV ones are saved and the numpy ones are skipped.
  • skip_save_predictions (bool, default: True): skips saving test predictions CSV files.
  • skip_save_eval_stats (bool, default: True): skips saving test statistics JSON file.
  • collect_predictions (bool, default: False): if True collects post-processed predictions during eval.
  • collect_overall_stats (bool, default: False): if True collects overall stats during eval.
  • output_directory (str, default: 'results'): the directory that will contain the training statistics, TensorBoard logs, the saved model and the training progress files.
  • return_type (Union[str, dict, pd.DataFrame], default: pandas.DataFrame): indicates the format to of the returned predictions.

Return

  • return (evaluation_statistics, predictions, output_directory): evaluation_statistics dictionary containing evaluation performance statistics, postprocess_predictions contains predicted values, output_directory is location where results are stored.

experiment

experiment(
  dataset=None,
  training_set=None,
  validation_set=None,
  test_set=None,
  training_set_metadata=None,
  data_format=None,
  experiment_name='experiment',
  model_name='run',
  model_resume_path=None,
  eval_split='test',
  skip_save_training_description=False,
  skip_save_training_statistics=False,
  skip_save_model=False,
  skip_save_progress=False,
  skip_save_log=False,
  skip_save_processed_input=False,
  skip_save_unprocessed_output=False,
  skip_save_predictions=False,
  skip_save_eval_stats=False,
  skip_collect_predictions=False,
  skip_collect_overall_stats=False,
  output_directory='results',
  random_seed=42
)

Trains a model on a dataset's training and validation splits and uses it to predict on the test split. It saves the trained model and the statistics of training and testing.

Inputs

  • dataset (Union[str, dict, pandas.DataFrame], default: None): source containing the entire dataset to be used in the experiment. If it has a split column, it will be used for splitting (0 for train, 1 for validation, 2 for test), otherwise the dataset will be randomly split.
  • training_set (Union[str, dict, pandas.DataFrame], default: None): source containing training data.
  • validation_set (Union[str, dict, pandas.DataFrame], default: None): source containing validation data.
  • test_set (Union[str, dict, pandas.DataFrame], default: None): source containing test data.
  • training_set_metadata (Union[str, dict], default: None): metadata JSON file or loaded metadata. Intermediate preprocessed structure containing the mappings of the input dataset created the first time an input file is used in the same directory with the same name and a '.meta.json' extension.
  • data_format (str, default: None): format to interpret data sources. Will be inferred automatically if not specified. Valid formats are 'auto', 'csv', 'df', 'dict', 'excel', 'feather', 'fwf', 'hdf5' (cache file produced during previous training), 'html' (file containing a single HTML <table>), 'json', 'jsonl', 'parquet', 'pickle' (pickled Pandas DataFrame), 'sas', 'spss', 'stata', 'tsv'.
  • experiment_name (str, default: 'experiment'): name for the experiment.
  • model_name (str, default: 'run'): name of the model that is being used.
  • model_resume_path (str, default: None): resumes training of the model from the path specified. The config is restored. In addition to config, training statistics and loss for epoch and the state of the optimizer are restored such that training can be effectively continued from a previously interrupted training process.
  • eval_split (str, default: test): split on which to perform evaluation. Valid values are training, validation and test.
  • skip_save_training_description (bool, default: False): disables saving the description JSON file.
  • skip_save_training_statistics (bool, default: False): disables saving training statistics JSON file.
  • skip_save_model (bool, default: False): disables saving model weights and hyperparameters each time the model improves. By default Ludwig saves model weights after each epoch the validation metric improves, but if the model is really big that can be time consuming. If you do not want to keep the weights and just find out what performance a model can get with a set of hyperparameters, use this parameter to skip it, but the model will not be loadable later on and the returned model will have the weights obtained at the end of training, instead of the weights of the epoch with the best validation performance.
  • skip_save_progress (bool, default: False): disables saving progress each epoch. By default Ludwig saves weights and stats after each epoch for enabling resuming of training, but if the model is really big that can be time consuming and will uses twice as much space, use this parameter to skip it, but training cannot be resumed later on.
  • skip_save_log (bool, default: False): disables saving TensorBoard logs. By default Ludwig saves logs for the TensorBoard, but if it is not needed turning it off can slightly increase the overall speed.
  • skip_save_processed_input (bool, default: False): if input dataset is provided it is preprocessed and cached by saving an HDF5 and JSON files to avoid running the preprocessing again. If this parameter is False, the HDF5 and JSON file are not saved.
  • skip_save_unprocessed_output (bool, default: False): by default predictions and their probabilities are saved in both raw unprocessed numpy files containing tensors and as postprocessed CSV files (one for each output feature). If this parameter is True, only the CSV ones are saved and the numpy ones are skipped.
  • skip_save_predictions (bool, default: False): skips saving test predictions CSV files
  • skip_save_eval_stats (bool, default: False): skips saving test statistics JSON file
  • skip_collect_predictions (bool, default: False): skips collecting post-processed predictions during eval.
  • skip_collect_overall_stats (bool, default: False): skips collecting overall stats during eval.
  • output_directory (str, default: 'results'): the directory that will contain the training statistics, TensorBoard logs, the saved model and the training progress files.
  • random_seed (int: default: 42): random seed used for weights initialization, splits and any other random function.

Return

  • return (Tuple[dict, dict, tuple, str)): (evaluation_statistics, training_statistics, preprocessed_data, output_directory) evaluation_statistics dictionary with evaluation performance statistics on the test_set, training_statistics is a nested dictionary of dataset -> feature_name -> metric_name -> List of metrics. Each metric corresponds to each training checkpoint. preprocessed_data tuple containing preprocessed (training_set, validation_set, test_set), output_directory filepath string to where results are stored.

forecast

forecast(
  dataset,
  data_format=None,
  horizon=1,
  output_directory=None,
  output_format='parquet'
)

free_gpu_memory

free_gpu_memory(
)

Manually moves the model to CPU to force GPU memory to be freed.

For more context: https://discuss.pytorch.org/t/how-can-we-release-gpu-memory-cache/14530/35

is_merge_and_unload_set

is_merge_and_unload_set(
)

Check whether the encapsulated model is of type LLM and is configured to merge_and_unload QLoRA weights.

Return

:return (bool): whether merge_and_unload should be done.

load

load(
  model_dir,
  logging_level=40,
  backend=None,
  gpus=None,
  gpu_memory_limit=None,
  allow_parallel_threads=True,
  callbacks=None
)

This function allows for loading pretrained models.

Inputs

  • model_dir (str): path to the directory containing the model. If the model was trained by the train or experiment command, the model is in results_dir/experiment_dir/model.
  • logging_level (int, default: 40): log level that will be sent to stderr.
  • backend (Union[Backend, str]): Backend or string name of backend to use to execute preprocessing / training steps.
  • gpus (Union[str, int, List[int]], default: None): GPUs to use (it uses the same syntax of CUDA_VISIBLE_DEVICES)
  • gpu_memory_limit (float: default: None): maximum memory fraction [0, 1] allowed to allocate per GPU device.
  • allow_parallel_threads (bool, default: True): allow Torch to use multithreading parallelism to improve performance at the cost of determinism.
  • callbacks (list, default: None): a list of ludwig.callbacks.Callback objects that provide hooks into the Ludwig pipeline.

Return

  • return (LudwigModel): a LudwigModel object

Example usage

ludwig_model = LudwigModel.load(model_dir)

load_weights

load_weights(
  model_dir
)

Loads weights from a pre-trained model.

Inputs

  • model_dir (str): filepath string to location of a pre-trained model

Return

  • return ( Non):None`

Example usage

ludwig_model.load_weights(model_dir)

predict

ludwig.predict(
  dataset=None,
  data_format=None,
  split='full',
  batch_size=128,
  generation_config=None,
  skip_save_unprocessed_output=True,
  skip_save_predictions=True,
  output_directory='results',
  return_type=<class 'pandas.core.frame.DataFrame'>,
  callbacks=None
)

Using a trained model, make predictions from the provided dataset.

Inputs

  • dataset (Union[str, dict, pandas.DataFrame]):: source containing the entire dataset to be evaluated.
  • data_format (str, default: None): format to interpret data sources. Will be inferred automatically if not specified. Valid formats are 'auto', 'csv', 'df', 'dict', 'excel', 'feather', 'fwf', 'hdf5' (cache file produced during previous training), 'html' (file containing a single HTML <table>), 'json', 'jsonl', 'parquet', 'pickle' (pickled Pandas DataFrame), 'sas', 'spss', 'stata', 'tsv'.
  • split (str, default= 'full'):: if the input dataset contains a split column, this parameter indicates which split of the data to use. Possible values are 'full', 'training', 'validation', 'test'.
  • batch_size (int, default: 128): size of batch to use when making predictions.
  • generation_config (Dict, default: None): config for the generation of the predictions. If None, the config that was used during model training is used. This is only used if the model type is LLM. Otherwise, this parameter is ignored. See Large Language Models under "Generation" for an example generation config.
  • skip_save_unprocessed_output (bool, default: True): if this parameter is False, predictions and their probabilities are saved in both raw unprocessed numpy files containing tensors and as postprocessed CSV files (one for each output feature). If this parameter is True, only the CSV ones are saved and the numpy ones are skipped.
  • skip_save_predictions (bool, default: True): skips saving test predictions CSV files.
  • output_directory (str, default: 'results'): the directory that will contain the training statistics, TensorBoard logs, the saved model and the training progress files.
  • return_type (Union[str, dict, pandas.DataFrame], default: pd.DataFrame): indicates the format of the returned predictions.
  • callbacks (Optional[List[Callback]], default: None): optional list of callbacks to use during this predict operation. Any callbacks already registered to the model will be preserved.

Return

:return (predictions, output_directory): (Tuple[Union[dict, pd.DataFrame], str]) predictions predictions from the provided dataset, output_directory filepath string to where data was stored.

preprocess

preprocess(
  dataset=None,
  training_set=None,
  validation_set=None,
  test_set=None,
  training_set_metadata=None,
  data_format=None,
  skip_save_processed_input=True,
  random_seed=42
)

This function is used to preprocess data.

Args:

  • dataset (Union[str, dict, pandas.DataFrame], default: None): source containing the entire dataset to be used in the experiment. If it has a split column, it will be used for splitting (0 for train, 1 for validation, 2 for test), otherwise the dataset will be randomly split.
  • training_set (Union[str, dict, pandas.DataFrame], default: None): source containing training data.
  • validation_set (Union[str, dict, pandas.DataFrame], default: None): source containing validation data.
  • test_set (Union[str, dict, pandas.DataFrame], default: None): source containing test data.
  • training_set_metadata (Union[str, dict], default: None): metadata JSON file or loaded metadata. Intermediate preprocessed structure containing the mappings of the input dataset created the first time an input file is used in the same directory with the same name and a '.meta.json' extension.
  • data_format (str, default: None): format to interpret data sources. Will be inferred automatically if not specified. Valid formats are 'auto', 'csv', 'df', 'dict', 'excel', 'feather', 'fwf', 'hdf5' (cache file produced during previous training), 'html' (file containing a single HTML <table>), 'json', 'jsonl', 'parquet', 'pickle' (pickled Pandas DataFrame), 'sas', 'spss', 'stata', 'tsv'.
  • skip_save_processed_input (bool, default: False): if input dataset is provided it is preprocessed and cached by saving an HDF5 and JSON files to avoid running the preprocessing again. If this parameter is False, the HDF5 and JSON file are not saved.
  • random_seed (int, default: 42): a random seed that will be used anywhere there is a call to a random number generator: data splitting, parameter initialization and training set shuffling

Returns:

  • __:return__: (PreprocessedDataset) data structure containing (proc_training_set, proc_validation_set, proc_test_set, training_set_metadata).

Raises:

  • RuntimeError: An error occured while preprocessing the data. Examples include training dataset being empty after preprocessing, lazy loading not being supported with RayBackend, etc.

save

save(
  save_path
)

This function allows to save models on disk.

Inputs

  • __ save_path__ (str): path to the directory where the model is going to be saved. Both a JSON file containing the model architecture hyperparameters and checkpoints files containing model weights will be saved.

Return

  • return (None): None

Example usage

ludwig_model.save(save_path)

save_config

save_config(
  save_path
)

Save config to specified location.

Inputs

  • save_path (str): filepath string to save config as a JSON file.

Return

  • return ( None):None`

save_torchscript

save_torchscript(
  save_path,
  model_only=False,
  device=None
)

Saves the Torchscript model to disk.

Inputs

  • save_path (str) (str):: The path to the directory where the model will be saved.
  • model_only (bool, optional) (bool, optional):: If True, only the ECD model will be converted to Torchscript. Else, the preprocessing and postprocessing steps will also be converted to Torchscript.
  • device (TorchDevice, optional) (TorchDevice, optional):: If None, the model will be converted to Torchscript on the same device to ensure maximum model parity.

Return

  • return ( None):None`

set_logging_level

set_logging_level(
  logging_level
)

Sets level for log messages.

Inputs

  • logging_level (int): Set/Update the logging level. Use logging constants like logging.DEBUG , logging.INFO and logging.ERROR.

Return

  • return ( None):None`

to_torchscript

to_torchscript(
  model_only=False,
  device=None
)

Converts the trained model to Torchscript.

Inputs

  • __ model_only (bool, optional)__ (bool, optional):: If True, only the ECD model will be converted to Torchscript. Else, preprocessing and postprocessing steps will also be converted to Torchscript.
  • device (TorchDevice, optional) (TorchDevice, optional):: If None, the model will be converted to Torchscript on the same device to ensure maximum model parity.

Returns

  • return ( A torch.jit.ScriptModule that can be used to predict on a dictionary of inputs): A torch.jit.ScriptModule that can be used to predict on a dictionary of inputs.

train

train(
  dataset=None,
  training_set=None,
  validation_set=None,
  test_set=None,
  training_set_metadata=None,
  data_format=None,
  experiment_name='api_experiment',
  model_name='run',
  model_resume_path=None,
  skip_save_training_description=False,
  skip_save_training_statistics=False,
  skip_save_model=False,
  skip_save_progress=False,
  skip_save_log=False,
  skip_save_processed_input=False,
  output_directory='results',
  random_seed=42
)

This function is used to perform a full training of the model on the specified dataset.

During training if the skip parameters are False the model and statistics will be saved in a directory [output_dir]/[experiment_name]_[model_name]_n where all variables are resolved to user specified ones and n is an increasing number starting from 0 used to differentiate among repeated runs.

Inputs

  • dataset (Union[str, dict, pandas.DataFrame], default: None): source containing the entire dataset to be used in the experiment. If it has a split column, it will be used for splitting (0 for train, 1 for validation, 2 for test), otherwise the dataset will be randomly split.
  • training_set (Union[str, dict, pandas.DataFrame], default: None): source containing training data.
  • validation_set (Union[str, dict, pandas.DataFrame], default: None): source containing validation data.
  • test_set (Union[str, dict, pandas.DataFrame], default: None): source containing test data.
  • training_set_metadata (Union[str, dict], default: None): metadata JSON file or loaded metadata. Intermediate preprocessed structure containing the mappings of the input dataset created the first time an input file is used in the same directory with the same name and a '.meta.json' extension.
  • data_format (str, default: None): format to interpret data sources. Will be inferred automatically if not specified. Valid formats are 'auto', 'csv', 'df', 'dict', 'excel', 'feather', 'fwf', 'hdf5' (cache file produced during previous training), 'html' (file containing a single HTML <table>), 'json', 'jsonl', 'parquet', 'pickle' (pickled Pandas DataFrame), 'sas', 'spss', 'stata', 'tsv'.
  • experiment_name (str, default: 'experiment'): name for the experiment.
  • model_name (str, default: 'run'): name of the model that is being used.
  • model_resume_path (str, default: None): resumes training of the model from the path specified. The config is restored. In addition to config, training statistics, loss for each epoch and the state of the optimizer are restored such that training can be effectively continued from a previously interrupted training process.
  • skip_save_training_description (bool, default: False): disables saving the description JSON file.
  • skip_save_training_statistics (bool, default: False): disables saving training statistics JSON file.
  • skip_save_model (bool, default: False): disables saving model weights and hyperparameters each time the model improves. By default Ludwig saves model weights after each epoch the validation metric improves, but if the model is really big that can be time consuming. If you do not want to keep the weights and just find out what performance a model can get with a set of hyperparameters, use this parameter to skip it, but the model will not be loadable later on and the returned model will have the weights obtained at the end of training, instead of the weights of the epoch with the best validation performance.
  • skip_save_progress (bool, default: False): disables saving progress each epoch. By default Ludwig saves weights and stats after each epoch for enabling resuming of training, but if the model is really big that can be time consuming and will uses twice as much space, use this parameter to skip it, but training cannot be resumed later on.
  • skip_save_log (bool, default: False): disables saving TensorBoard logs. By default Ludwig saves logs for the TensorBoard, but if it is not needed turning it off can slightly increase the overall speed.
  • skip_save_processed_input (bool, default: False): if input dataset is provided it is preprocessed and cached by saving an HDF5 and JSON files to avoid running the preprocessing again. If this parameter is False, the HDF5 and JSON file are not saved.
  • output_directory (str, default: 'results'): the directory that will contain the training statistics, TensorBoard logs, the saved model and the training progress files.
  • random_seed (int, default: 42): a random seed that will be used anywhere there is a call to a random number generator: data splitting, parameter initialization and training set shuffling
  • kwargs (dict, default: {}): a dictionary of optional parameters.

Return

  • return (Tuple[Dict, Union[Dict, pd.DataFrame], str]): tuple containing (training_statistics, preprocessed_data, output_directory). training_statistics is a nested dictionary of dataset -> feature_name -> metric_name -> List of metrics. Each metric corresponds to each training checkpoint. preprocessed_data is the tuple containing these three data sets (training_set, validation_set, test_set). output_directory filepath to where training results are stored.

train_online

train_online(
  dataset,
  training_set_metadata=None,
  data_format='auto',
  random_seed=42
)

Performs one epoch of training of the model on dataset.

Inputs

  • dataset (Union[str, dict, pandas.DataFrame], default: None): source containing the entire dataset to be used in the experiment. If it has a split column, it will be used for splitting (0 for train, 1 for validation, 2 for test), otherwise the dataset will be randomly split.
  • training_set_metadata (Union[str, dict], default: None): metadata JSON file or loaded metadata. Intermediate preprocessed structure containing the mappings of the input dataset created the first time an input file is used in the same directory with the same name and a '.meta.json' extension.
  • data_format (str, default: None): format to interpret data sources. Will be inferred automatically if not specified. Valid formats are 'auto', 'csv', 'df', 'dict', 'excel', 'feather', 'fwf', 'hdf5' (cache file produced during previous training), 'html' (file containing a single HTML <table>), 'json', 'jsonl', 'parquet', 'pickle' (pickled Pandas DataFrame), 'sas', 'spss', 'stata', 'tsv'.
  • random_seed (int, default: 42): a random seed that is going to be used anywhere there is a call to a random number generator: data splitting, parameter initialization and training set shuffling

Return

  • return (None): None

upload_to_hf_hub

ludwig.upload_to_hf_hub(
  repo_id,
  model_path,
  repo_type='model',
  private=False,
  commit_message='Upload trained [Ludwig](https://ludwig.ai/latest/) model weights',
  commit_description=None
)

Uploads trained model artifacts to the HuggingFace Hub.

Inputs

  • repo_id (str) (str):: A namespace (user or an organization) and a repo name separated by a /.
  • model_path (str) (str):: The path of the saved model. This is the top level directory where the models weights as well as other associated training artifacts are saved.
  • private (bool, optional, defaults to False) (bool, optional, defaults to False):: Whether the model repo should be private.
  • repo_type (str, optional) (str, optional):: Set to "dataset" or "space" if uploading to a dataset or space, None or "model" if uploading to a model. Default is None.
  • commit_message (str, optional) (str, optional):: The summary / title / first line of the generated commit. Defaults to: f"Upload {path_in_repo} with huggingface_hub"
  • commit_description (str optional) (str optional):: The description of the generated commit

Returns

  • return (bool): True for success, False for failure.

Module functions

kfold_cross_validate

ludwig.api.kfold_cross_validate(
  num_folds,
  config,
  dataset=None,
  data_format=None,
  skip_save_training_description=False,
  skip_save_training_statistics=False,
  skip_save_model=False,
  skip_save_progress=False,
  skip_save_log=False,
  skip_save_processed_input=False,
  skip_save_predictions=False,
  skip_save_eval_stats=False,
  skip_collect_predictions=False,
  skip_collect_overall_stats=False,
  output_directory='results',
  random_seed=42,
  gpus=None,
  gpu_memory_limit=None,
  allow_parallel_threads=True,
  backend=None,
  logging_level=20
)

Performs k-fold cross validation and returns result data structures.

Inputs

  • num_folds (int): number of folds to create for the cross-validation
  • config (Union[dict, str]): model specification required to build a model. Parameter may be a dictionary or string specifying the file path to a yaml configuration file. Refer to the User Guide for details.
  • dataset (Union[str, dict, pandas.DataFrame], default: None): source containing the entire dataset to be used for k_fold processing.
  • data_format (str, default: None): format to interpret data sources. Will be inferred automatically if not specified. Valid formats are 'auto', 'csv', 'df', 'dict', 'excel', 'feather', 'fwf', 'html' (file containing a single HTML <table>), 'json', 'jsonl', 'parquet', 'pickle' (pickled Pandas DataFrame), 'sas', 'spss', 'stata', 'tsv'. Currently hdf5 format is not supported for k_fold cross validation.
  • skip_save_training_description (bool, default: False): disables saving the description JSON file.
  • skip_save_training_statistics (bool, default: False): disables saving training statistics JSON file.
  • skip_save_model (bool, default: False): disables saving model weights and hyperparameters each time the model improves. By default Ludwig saves model weights after each epoch the validation metric improves, but if the model is really big that can be time consuming. If you do not want to keep the weights and just find out what performance a model can get with a set of hyperparameters, use this parameter to skip it, but the model will not be loadable later on and the returned model will have the weights obtained at the end of training, instead of the weights of the epoch with the best validation performance.
  • skip_save_progress (bool, default: False): disables saving progress each epoch. By default Ludwig saves weights and stats after each epoch for enabling resuming of training, but if the model is really big that can be time consuming and will uses twice as much space, use this parameter to skip it, but training cannot be resumed later on.
  • skip_save_log (bool, default: False): disables saving TensorBoard logs. By default Ludwig saves logs for the TensorBoard, but if it is not needed turning it off can slightly increase the overall speed.
  • skip_save_processed_input (bool, default: False): if input dataset is provided it is preprocessed and cached by saving an HDF5 and JSON files to avoid running the preprocessing again. If this parameter is False, the HDF5 and JSON file are not saved.
  • skip_save_predictions (bool, default: False): skips saving test predictions CSV files.
  • skip_save_eval_stats (bool, default: False): skips saving test statistics JSON file.
  • skip_collect_predictions (bool, default: False): skips collecting post-processed predictions during eval.
  • skip_collect_overall_stats (bool, default: False): skips collecting overall stats during eval.
  • output_directory (str, default: 'results'): the directory that will contain the training statistics, TensorBoard logs, the saved model and the training progress files.
  • random_seed (int, default: 42): Random seed used for weights initialization, splits and any other random function.
  • gpus (list, default: None): list of GPUs that are available for training.
  • gpu_memory_limit (float: default: None): maximum memory fraction [0, 1] allowed to allocate per GPU device.
  • allow_parallel_threads (bool, default: True): allow Torch to use multithreading parallelism to improve performance at the cost of determinism.
  • backend (Union[Backend, str]): Backend or string name of backend to use to execute preprocessing / training steps.
  • logging_level (int, default: INFO): log level to send to stderr.

Return

  • return (tuple(kfold_cv_statistics, kfold_split_indices), dict): a tuple of dictionaries kfold_cv_statistics: contains metrics from cv run. kfold_split_indices: indices to split training data into training fold and test fold.

PublicAPI: This API is stable across Ludwig releases.

hyperopt

ludwig.hyperopt.run.hyperopt(
  config,
  dataset=None,
  training_set=None,
  validation_set=None,
  test_set=None,
  training_set_metadata=None,
  data_format=None,
  experiment_name='hyperopt',
  model_name='run',
  resume=None,
  skip_save_training_description=False,
  skip_save_training_statistics=False,
  skip_save_model=False,
  skip_save_progress=False,
  skip_save_log=False,
  skip_save_processed_input=True,
  skip_save_unprocessed_output=False,
  skip_save_predictions=False,
  skip_save_eval_stats=False,
  skip_save_hyperopt_statistics=False,
  output_directory='results',
  gpus=None,
  gpu_memory_limit=None,
  allow_parallel_threads=True,
  callbacks=None,
  tune_callbacks=None,
  backend=None,
  random_seed=42,
  hyperopt_log_verbosity=3
)

This method performs an hyperparameter optimization.

Inputs

  • config (Union[str, dict]): config which defines the different parameters of the model, features, preprocessing and training. If str, filepath to yaml configuration file.
  • dataset (Union[str, dict, pandas.DataFrame], default: None): source containing the entire dataset to be used in the experiment. If it has a split column, it will be used for splitting (0 for train, 1 for validation, 2 for test), otherwise the dataset will be randomly split.
  • training_set (Union[str, dict, pandas.DataFrame], default: None): source containing training data.
  • validation_set (Union[str, dict, pandas.DataFrame], default: None): source containing validation data.
  • test_set (Union[str, dict, pandas.DataFrame], default: None): source containing test data.
  • training_set_metadata (Union[str, dict], default: None): metadata JSON file or loaded metadata. Intermediate preprocessed structure containing the mappings of the input dataset created the first time an input file is used in the same directory with the same name and a '.meta.json' extension.
  • data_format (str, default: None): format to interpret data sources. Will be inferred automatically if not specified. Valid formats are 'auto', 'csv', 'df', 'dict', 'excel', 'feather', 'fwf', 'hdf5' (cache file produced during previous training), 'html' (file containing a single HTML <table>), 'json', 'jsonl', 'parquet', 'pickle' (pickled Pandas DataFrame), 'sas', 'spss', 'stata', 'tsv'.
  • experiment_name (str, default: 'experiment'): name for the experiment.
  • model_name (str, default: 'run'): name of the model that is being used.
  • resume (bool): If true, continue hyperopt from the state of the previous run in the output directory with the same experiment name. If false, will create new trials, ignoring any previous state, even if they exist in the output_directory. By default, will attempt to resume if there is already an existing experiment with the same name, and will create new trials if not.
  • skip_save_training_description (bool, default: False): disables saving the description JSON file.
  • skip_save_training_statistics (bool, default: False): disables saving training statistics JSON file.
  • skip_save_model (bool, default: False): disables saving model weights and hyperparameters each time the model improves. By default Ludwig saves model weights after each epoch the validation metric improves, but if the model is really big that can be time consuming. If you do not want to keep the weights and just find out what performance a model can get with a set of hyperparameters, use this parameter to skip it, but the model will not be loadable later on and the returned model will have the weights obtained at the end of training, instead of the weights of the epoch with the best validation performance.
  • skip_save_progress (bool, default: False): disables saving progress each epoch. By default Ludwig saves weights and stats after each epoch for enabling resuming of training, but if the model is really big that can be time consuming and will uses twice as much space, use this parameter to skip it, but training cannot be resumed later on.
  • skip_save_log (bool, default: False): disables saving TensorBoard logs. By default Ludwig saves logs for the TensorBoard, but if it is not needed turning it off can slightly increase the overall speed.
  • skip_save_processed_input (bool, default: False): if input dataset is provided it is preprocessed and cached by saving an HDF5 and JSON files to avoid running the preprocessing again. If this parameter is False, the HDF5 and JSON file are not saved.
  • skip_save_unprocessed_output (bool, default: False): by default predictions and their probabilities are saved in both raw unprocessed numpy files containing tensors and as postprocessed CSV files (one for each output feature). If this parameter is True, only the CSV ones are saved and the numpy ones are skipped.
  • skip_save_predictions (bool, default: False): skips saving test predictions CSV files.
  • skip_save_eval_stats (bool, default: False): skips saving test statistics JSON file.
  • skip_save_hyperopt_statistics (bool, default: False): skips saving hyperopt stats file.
  • output_directory (str, default: 'results'): the directory that will contain the training statistics, TensorBoard logs, the saved model and the training progress files.
  • gpus (list, default: None): list of GPUs that are available for training.
  • gpu_memory_limit (float: default: None): maximum memory fraction [0, 1] allowed to allocate per GPU device.
  • allow_parallel_threads (bool, default: True): allow PyTorch to use multithreading parallelism to improve performance at the cost of determinism.
  • callbacks (list, default: None): a list of ludwig.callbacks.Callback objects that provide hooks into the Ludwig pipeline.
  • backend (Union[Backend, str]): Backend or string name of backend to use to execute preprocessing / training steps.
  • random_seed (int: default: 42): random seed used for weights initialization, splits and any other random function.
  • hyperopt_log_verbosity (int: default: 3): controls verbosity of ray tune log messages. Valid values: 0 = silent, 1 = only status updates, 2 = status and brief trial results, 3 = status and detailed trial results.

Return

  • return (List[dict]): List of results for each trial, ordered by descending performance on the target metric.