Training and Inference of LLMs with PyTorch Fully Sharded Data Parallel and Better Transformer
In this blog we show how to perform efficient and optimized distributed training and inference of large language models using PyTorch’s Fully Sharded Data Parallel and Better Transformer implementations, on the Spark platform. In this implementation, we combine Microsoft Fabric for data preparation and model inference, and Azure Databricks for model training, having all our data under Microsoft Fabric’s OneLake. The code for this blog is available at this GitHub repository, as a series of PySpark notebooks for Microsoft Fabric and Azure Databricks.
