AI Blueprints

Simplify the deployment of large language models (LLMs) and vision language models (VLMs) using an open source interface engine called virtual large language model (vLLM). Deploy a custom model or select from a variety of open models on Hugging Face.

Streamline infrastructure benchmarking for fine-tuning using the MLCommons methodology. It fine-tunes a quantized Llama-2-70B model with a standard data set.

OCI AI Blueprints enable efficient model tuning using low-rank adaptation (LoRA), a highly efficient method of LLM fine-tuning. Fine-tune a custom LLM or use most open LLMs from Hugging Face.

Prior to deploying production or research workloads, you can use a robust, precheck blueprint for thorough GPU health validation to proactively detect and address issues. Verify that your GPU infrastructure is primed for high-demand experiments across both single- and multi-node environments.

Adopt a comprehensive framework for serving LLMs on CPUs using the Ollama platform with a variety of supported models, such as Mistral, Gemma, and others.

With this blueprint, you can distribute inference serving across several computing nodes, each typically equipped with one or more GPUs. For example, deploy Llama 405B–sized LLMs across multiple H100 nodes with RDMA using vLLM and LeaderWorkerSet.

Serve LLMs with autoscaling using KEDA, which scales to multiple GPUs and nodes using application metrics, such as inference latency.

Deploy LLMs to a fraction of a GPU with NVIDIA’s multi-instance GPUs and serve them with a vLLM.