What is Ludwig?

Introduction

Ludwig is an open-source, declarative machine learning framework that makes it easy to define deep learning pipelines with a simple and flexible data-driven configuration system. Ludwig is suitable for a wide variety of AI tasks, and is hosted by the Linux Foundation AI & Data.

Ludwig enables you to apply state-of-the-art tabular, natural language processing, and computer vision models to your existing data and put them into production with just a few short commands.

CLIPythondata.csvconfig.yaml

ludwig train --config config.yaml --dataset data.csv
ludwig predict --model_path results/experiment_run/model --dataset test.csv

from ludwig.api import LudwigModel
import pandas as pd

# train a model
config = {
    "input_features": [
        {
            "name": "sepal_length_cm",
            "type": "number"
        },
        {
            "name": "sepal_width_cm",
            "type": "number"
        },
        {
            "name": "petal_length_cm",
            "type": "number"
        },
        {
            "name": "petal_width_cm",
            "type": "number"
        }
    ],
    "output_features": [
        {
            "name": "class",
            "type": "category"
        }
    ]
}
model = LudwigModel(config)
data = pd.read_csv("data.csv")
train_stats, _, model_dir = model.train(data)

# or load a model
model = LudwigModel.load(model_dir)

# obtain predictions
predictions = model.predict(data)

sepal_length_cm,sepal_width_cm,petal_length_cm,petal_width_cm
4.9,3.0,1.4,0.2
4.7,3.2,1.3,0.2
4.6,3.1,1.5,0.2
5.0,3.6,1.4,0.2
5.4,3.9,1.7,0.4
4.6,3.4,1.4,0.3
5.0,3.4,1.5,0.2
4.4,2.9,1.4,0.2
4.9,3.1,1.5,0.1

input_features:
- name: sepal_length_cm
  type: number
- name: sepal_width_cm
  type: number
- name: petal_length_cm
  type: number
- name: petal_width_cm
  type: number
output_features:
- name: class
  type: category

Ludwig makes this possible through its declarative approach to structuring machine learning pipelines. Instead of writing code for your model, training loop, preprocessing, postprocessing, evaluation and hyperparameter optimization, you only need to declare the schema of your data with a simple YAML configuration:

input_features:
  - name: title
    type: text
  - name: author
    type: category
  - name: description
    type: text
  - name: cover
    type: image

output_features:
  - name: genre
    type: set
  - name: price
    type: number

Starting from a simple config like the one above, any and all aspects of the model architecture, training loop, hyperparameter search, and backend infrastructure can be modified as additional fields in the declarative configuration to customize the pipeline to meet your requirements:

input_features:
  - name: title
    type: text
    encoder: rnn
    cell: lstm
    num_layers: 2
    state_size: 128
    preprocessing:
      tokenizer: space_punct
  - name: author
    type: category
    embedding_size: 128
    preprocessing:
      most_common: 10000
  - name: description
    type: text
    encoder: bert
  - name: cover
    type: image
    encoder: resnet
    num_layers: 18

output_features:
  - name: genre
    type: set
  - name: price
    type: number
    preprocessing:
      normalization: zscore

trainer:
  epochs: 50
  batch_size: 256
  optimizer:
    type: adam
    beat1: 0.9
  learning_rate: 0.001

backend:
  type: local
  cache_format: parquet

hyperopt:
  metric: f1
  sampler: random
  parameters:
    title.num_layers:
      lower: 1
      upper: 5
    training.learning_rate:
      values: [0.01, 0.003, 0.001]

Ludwig is a single framework that guides you through machine learning end-to-end; from experimenting with different training recipes, exploring state-of-the-art model architectures, to scaling up to large out-of-memory datasets and multi-node clusters, and finally serving the best model in production.

Why Declarative Machine Learning Systems

Ludwig’s declarative approach to machine learning provides the simplicity of an AutoML solution with the flexibility of writing your own PyTorch code. This is achieved by creating an extensible, declarative configuration with optional parameters for every aspect of the pipeline.

Multi-modal, multi-task learning out-of-the-box

Mix and match tabular data, text, images, and even audio into complex model configurations without writing code.

Fully customizable and extensible

Every part of the model and training process can be controlled through a simple configuration interface.

Minimal machine learning boilerplate

Engineering complexity of deep learning is handled out of the box, enabling research scientists to focus on building models at the highest level of abstraction.

Data preprocessing, hyperparameter optimization, device management, and distributed training for newly registered torch.nn.Module models come completely free.

Why Ludwig

Ludwig’s declarative programming model allows for key features such as:

Highly configurable data preprocessing, modeling, and metrics

Any and all aspects of the model architecture, training loop, hyperparameter search, and backend infrastructure can be modified as additional fields in the declarative configuration to customize the pipeline to meet your requirements.

For details on what can be configured, check out Ludwig Configuration docs.

Integration with any structured data source

If it can be read into a SQL table or Pandas DataFrame, Ludwig can train a model on it.

Hyperparameter optimization

Perform a variety of hyperparameter search algorithms locally or across many workers in parallel using Ray Tune.

Rich model exporting and tracking

Automatically track all trials and metrics with tools like Tensorboard, Comet ML, Weights & Biases, and MLflow.

Automatically scale training to multi-GPU, multi-node clusters

Go from training on your local machine to the cloud without code or config changes.

Easily build your benchmarks

Creating a state-of-the-art baseline and comparing it with a new model is a simple config change.

Easily apply new architectures to multiple problems and datasets

Apply new models across the extensive set of tasks and datasets that Ludwig supports. Ludwig includes a full benchmarking toolkit accessible to any user, for running experiments with multiple models across multiple datasets with just a simple configuration.

Low-code interface for state-of-the-art models, including pre-trained Huggingface Transformers

Ludwig also natively integrates with pre-trained models, such as the ones available in Huggingface Transformers. Users can choose from a vast collection of state-of-the-art pre-trained PyTorch models to use without needing to write any code at all. For example, training a BERT-based sentiment analysis model with Ludwig is as simple as:

ludwig train --dataset sst5 -–config_str “{input_features: [{name: sentence, type: text, encoder: bert}], output_features: [{name: label, type: category}]}”

Low-code interface for AutoML

Ludwig AutoML allows users to obtain trained models by providing just a dataset, the target column, and a time budget.

auto_train_results = ludwig.automl.auto_train(dataset=my_dataset_df, target=target_column_name, time_limit_s=7200)

Easy productionisation

Ludwig makes it easy to serve deep learning models, including on GPUs. Launch a REST API for your trained Ludwig model.

ludwig serve --model_path=/path/to/model

Ludwig supports exporting models to efficient Torschscript bundles.

ludwig export_torchscript -–model_path=/path/to/model