Multi-node inference

[hrbigelow]

Feature request

Hi,

I wondered if it's possible to do multi-node inference? In my case I would like to run DeepSeek-V3 671B inference on 9 or more H100s. I have two nodes, each with 8 H100s attached.

I looked through the codebase a bit but so far couldn't see any obvious entrypoint to doing this.

Motivation

Would like to run DeepSeek-V3 for an ML experiment

Your contribution

None so far!