Multi-Node LLM Training at Scale on DigitalOcean

This report provides a comprehensive analysis of the end-to-end efficacy and performance of pretraining and finetuning of large language models in a multinode setting. We use DO’s bare-metal infrastructure on 8 nodes.

MosaicML LLM Foundry provides a framework in which multinode pretraining and finetuning is supported, including not only model finetuning, but verification of model inference functionality and model accuracy via evaluation, representing a true end-to-end system.

• Finetuning tool:
  • MosaicML LLM Foundry
• Tasks:
  • Full pretraining from scractch
  • Finetuning of pretrained model
• Models:
  • Pretraining
    • MPT-125M, 350M, 760M, 1B, 3B, 7B, 13B, 30B, 70B
  • Finetuning
    • MPT-7B-Dolly-SFT
    • MPT-30B-Instruct
• Data:
  • Pretraining
    • C4
  • Finetuning
    • Dolly HH-RLHF
    • Instruct-v3
• Key metrics:
  • Token throughput (tok/s)
  • Model FLOPS uilization (MFU)
  • Runtime (wallclock)
• Scenario dimensions:
  • Pretraining
    • Model sizes: 125M, 350M, 760M, 1B
  • Finetuning
    • Number of nodes: 1, 2, 4, 8

We designed experiments to reflect real-world usage and minimize variance by running locally hosted, fully automated tests under controlled environments.

For full details of the hardware, software, pretraining & finetuning datasets, and network speed tests, see the appendices.

In this section, we examine the results of our full pretraining and finetuning experiments. We find that model pretraining and finetuning both run successfully on DO multinode bare-metal machines, and can be run by users.

Full pretraining

Our full pretraining runs are shown in Table 1. We do not graph these due to the small number of rows and variety of relevant results encapsulated in the table.

Table 1: Results of full pretraining runs for MosaicML models from 125M to 1B parameters on 8 nodes

Main observations for full pretraining:

• In all cases, inference using the trained model, and model accuracy via evaluation on unseen testing data are verified.
• Model conversion after training to Hugging Face format, which gives a lighter weight model that is more efficient for inference, is also verified.
• The larger models in the MPT series, MPT-3B, -7B, -13B, -30B, and -70B, would run in the exact same way, but would require increasingly more wallclock time.
• Projection of runtime indicates that the largest model, MPT-70B, could be fully pretrained on DO in approximately 2 months on 8 nodes, or approximately 1 week on 64 nodes.
• Model FLOPS utilization, a measure of GPU usage more effective than raw utilization provided by LLM Foundry, is good, showing that the increased runtimes are genuine larger compute required by the larger models, not inefficiency of the infrastructure.

Finetuning

Similar to pretraining, we tabulate results for finetuning in Table 2.

Table 2: Results of full finetuning runs for MosaicML MPT-7B and MPT-30B models for 1-8 nodes

Main observations for finetuning:

• The ideal speedup versus one node is N times for N nodes. We observe 2.67, 4.28, and 5.77x for 2, 4, and 8 nodes respectively. This indicates that we strongly gain from 2-4 nodes, a bit less so for 8. There is some overhead for model checkpoint saving after each epoch, but this is a normal operation.
• In all cases, inference using the trained model, and model accuracy via evaluation on unseen testing data are verified.
• Evaluation accuracy of the model on unseen data is higher than for the pretrained smaller models, as expected.

Datasets for pretraining and finetuning

Pretraining dataset

For large language models, full pretraining from scratch requires a large generic dataset on which to train the model to give it basic language capabilities. MosaicML’s LLM Foundry provides support for the C4 dataset, which we used throughout pretraining.

C4 is a standard text dataset for pretraining large language models, consisting of over 10 billion rows of data. It is a cleaned version of the Common Crawl web corpus. The data are downloaded and preprocessed as part of the end-to-end support of LLM Foundry for pretraining.

Finetuning dataset

Finetuning datasets are more specialized and smaller than pretraining datasets. We again used two examples supported by MosaicML, with some modifications for the 30B model.

MPT-7B-Dolly-SFT

For finetuning the 7B model, we used the Dolly HH-RHLF dataset.

From the dataset card, “This dataset is a combination of Databrick’s dolly-15k dataset and a filtered subset of Anthropic’s HH-RLHF. It also includes a test split, which was missing in the original dolly set. That test set is composed of 200 randomly selected samples from dolly + 4,929 of the test set samples from HH-RLHF which made it through the filtering process. The train set contains 59,310 samples; 15,014 - 200 = 14,814 from Dolly, and the remaining 44,496 from HH-RLHF.”

MPT-30B-Instruct

For finetuning the 30B model, we used the Instruct-v3 dataset, which consists of prompts and responses for instruction finetuning. The original MosaicML location contained a bug in which the dataset had incorrect columns, so we used a copy at kowndinya23/instruct-v3 where the relevant correction had been made. This was preferable to manually correcting because by default the end-to-end process points to remote dataset locations such as Hugging Face.

Network Speed: NCCL Tests

NCCL tests were run on the machines used for this project, verifying the efficacy of the network.

Our purpose was simple hardware verification, and more extensive testing has been carried out elsewhere by DO infrastructure teams. However, these results have been of interest as an available representation of our typical networking speeds by customers interested in multinode work, so we supply them here.

This command was used to generate the results:

mpirun \
-H hostfile \
-np 128 \
-N 8 \
--allow-run-as-root \
-x NCCL_IB_PCI_RELAXED_ORDERING=1 \
-x NCCL_IB_CUDA_SUPPORT=1 \
-x NCCL_IB_HCA^=mlx5_1,mlx5_2,mlx5_7,mlx5_8 \
-x NCCL_CROSS_NIC=0 -x NCCL_IB_GID_INDEX=1 \
$(pwd)/nccl-tests/build/all_reduce_perf -b 8 -e 8G -f 2 -g 1

The results are tabulated below for the case of 16 nodes.

Table: NCCL test results for machines used in this report, for 16 nodes. These verify that their network has appropriate bandwidth for our model pretraining and finetuning.

The same results are plotted in the figure, for bus bandwidth as a function of 1-8 nodes.

Figure: NCCL bus bandwidth test results for machines used in this report, for 1-8 nodes

Hardware and Software

DigitalOcean Bare-Metal machines

DO bare metal machines consisting of 8 H100x8 nodes and the nodes were linked by an RDMA over Converged Ethernet (RoCE) network with a common VPC, with a shared storage filesystem providing multiple terabytes of storage. Each node consisted of 8 H100 GPUs connected by NVLink, for a total of 64 GPUs. As bare-metal machines, the Ubuntu operating system was provided, and MosaicML was then run from a Docker container.

As a consequence of MosaicML’s usage of shared drives without requiring SLURM, Docker containers had to be duplicated on each node, but once done funetuning runs can be executed using a single command on each node.

The shared nature of the machines resulted in 8 of the 16 nodes being used for this project.

MosaicML LLM Foundry

This is an open source tool that allows pretraining and finetuning of models in a multinode setting. Particular positive attributes of this tool are:

• Emphasis on end-to-end usage, including data preparation, model training/finetuning, inference, and evaluation, like in real customer use cases.
• Command-line based interface that allows pretraining and finetuning to be instantiated in this setting without the requirement to write a lot of code specialized to a particular application.
• Shared disk multinode setup that avoided the need for setting up and usage of SLURM, unlike most multinode tools that introduce this complex additional overhead taken from the HPC world.
• Proof of efficient usage of GPU compute via the model FLOPS utilization (MFU) metric
• Existing benchmarks from MosaicML enabling calibration of expectations (e.g., of MFU), albeit on different machines so not 1:1 comparable with our numbers
• Support of real-world large language models in the MPT series
• Weights & Biases logging integration

• Anthropic’s HH-RLHF dataset
• Bare metal clusters
• C4 dataset
• Databricks’ dolly-15k dataset
• DigitalOcean AI/ML
• Dolly HH-RHLF dataset
• GPU usage
• Hugging Face
• Instruct-v3 dataset with correction
• LLM Foundry
• MosaicML MPT models
• RDMA over converged Ethernet (RoCE)