Llm

A Compose file with a models: section runs an AI application out of the box on a laptop. Shipping the same stack to Kubernetes used to mean writing the model server’s Deployment, Service, ConfigMap, and PVC by hand, and remembering to point your application’s environment variables at the right place. Compose Bridge now does that for you, in two distinct topologies. This is the final post of the Compose Bridge Deep Dive series. Parts 1 and 2 covered the fundamentals and customization. This one focuses on a concrete, end-to-end scenario built on top of the new model-runner support in the default transformers. ...

LLMs are now first-class citizens in Compose. The models top-level key lets you declare which models your application needs and wire them into your services, all in the same Compose file. Basic usage Declare a model at the top level, reference it from a service: models: smollm: model: ai/smollm2 services: app: image: myapp models: - smollm When the stack starts, Compose ensures the model is available locally and connects the app service to it. The container receives endpoint information via environment variables. ...