Private Large Language Model Service
The Private Large Language Model Service is a feature of the Private AI Services Container that hosts open-weight large language models in a secure, private environment. It enables database applications to offload Large Language Model (LLM) inference without sending data to public AI services, helping protect sensitive information and meet data residency requirements.
You can choose to use the default model (mistralai/Ministral-3-3B-Reasoning-2512) which ships with the large container, or you can download open-weight models from Hugging Face.
The service is a REST server that can run on either CPU or GPU, and it works with many open-weight LLMs.
Why Private Large Language Model Service?
Private LLM Inference
The Private AI Services Container runs directly on your own infrastructure, which includes your own servers, your own private cloud, or public cloud within your own tenancy. This allows you to use Large Language Models (LLMs) while keeping your data private. Because all processing happens in your environment, your data is never sent to a third-party service for inference or fine-tuning.
Model Choice
Choose the AI model that best fits your needs. You can use the default LLM (mistralai/Ministral-3-3B-Reasoning-2512) included with the large container or download and deploy one of your preferred open-weight models. The Private Large Language Model Service works with more than 40 open-weight LLMs, giving you the flexibility to select the model that works best for your applications.
GPU Optional
A GPU is not required to run open-weight Large Language Models, however, the Private Large Language Model Service supports both CPU and GPU deployments, allowing you to use your existing infrastructure without requiring additional hardware. If you do have GPUs available, you can definitely use them to improve inference performance.
Cost Effective
Run AI workloads on the infrastructure you already have. When using CPUs for inference, performance scales with the number of CPU cores, while the amount of system memory determines the size of the models you can run. This gives you the flexibility to optimize your deployment for either maximum performance or the best price-to-performance ratio.
Unlimited Tokens
Use as many tokens as you need without worrying about usage limits or ongoing fees. The Private Large Language Model Service is your own service – you're in control. By definition, there are no subscription costs and no token-based pricing. This means you can run unlimited inference on your own infrastructure at no additional charge.
Built-in Guardrails
Every model is protected by default with built-in guardrails to help defend against prompt injection and jailbreak attacks. The service uses the meta-llama/Llama-Prompt-Guard-2-86M guardrail for protection. Guardrails can be enabled or disabled on a per-model basis to meet your deployment requirements.
No External Dependencies
The Private AI Services Container is fully self-contained. It includes all required services, monitoring capabilities, and user-configurable rate limiting within the container itself. It does not rely on external services, third-party monitoring tools, or external rate-limiting solutions to operate.
Key Features of the Private Large Language Model Service
Air Gap Deployments
Can run in fully air-gapped environments with no external internet connectivity, enabling secure AI deployments while keeping models and data entirely within your infrastructure. Use the default open-weight LLM included with the Private AI Services Container, or import a model from Hugging Face using removable media, such as a USB drive.
vLLM Runtime
Supports the high-performance vLLM inference runtime on both CPU and GPU deployments. vLLM delivers excellent concurrency and low time-to-first-token, making it ideal for responsive AI applications. As one of the most widely adopted inference engines, vLLM is used by many leading cloud providers for their hosted LLM services.
llama.cpp Runtime
Supports the llama.cpp inference runtime for CPU-based deployments. Optimized for fast model loading and low-latency inference, llama.cpp is an excellent choice for single-user and lightweight AI workloads. It is the same inference engine used by popular tools such as Ollama and LM Studio.
Enterprise-Grade Security
Designed with security in mind, the Private Large Language Model Service protects all REST API communication with TLS 1.3 encryption, securely stores passwords in an encrypted PKCS#12 keystore, and authenticates every request using API keys. The service also supports SELinux in enforcing mode for enhanced operating system security. User data is not stored within the container, helping to protect sensitive information.
High Availability and Scalability
Supports high availability and scalability when deployed on Kubernetes or Red Hat OpenShift using the Oracle Database Operator for Kubernetes. By running multiple container replicas behind a load balancer, the service can continue processing requests even if an individual instance becomes unavailable, improving reliability and minimizing downtime. By using multiple replicas, the service can scale to increase throughput.
Rate Limiting
Control AI resource usage by configuring rate limits for each open-weight LLM. You can set a separate throughput limit for each model, helping to ensure fair resource allocation, maintain consistent performance, and prevent any single model from consuming excessive system resources.
REST APIs
The Private Large Language Model Service implements the OpenAI-compatible REST API. This allows many existing applications, SDKs, and AI tools that support the OpenAI API to work with little or no code changes, making it easy to integrate private LLM inference into your existing workflows.
Open-Weight LLMs for the vLLM Runtime
Open-weight LLMs which are supported by vLLM 0.21.0, which use Safetensors, and do not require remote code should work in this release.
The following list covers the models which are known to work with vLLM in the Private Large Language Model release 26.2.
Examples of Open-Weight LLMs for the vLLM Runtime
Open-Weight Models
Llama 3.1 and 3.3 are multi-lingual chat LLMs.
Scout and Maverick are multi-modal and multi-linugal mixture of expert LLMs.
These models use the Safe Tensor file format. Safe Tensor models can be quantized (compressed).
Quantiztion is a tradeoff bettween size and quality. Safe Tensor files tend to use a precision of 4 to 16 bits for quantization.
Open-Weight Models
Ministral-7B instruct is a chat LLM.
Ministral-3-3B-Reasoning is a multi-modal agentic LLM.
Ministral-3-3B-Instruct and Ministral-3-14B-Instruct are multi-modal chat LLMs.
Ministral-Small-4-119B-2603 is a hybrid model which can do chat, agentic reasoning and coding.
These models use the Safe Tensor file format. Safe Tensor models can be quantized (compressed).
Quantiztion is a tradeoff bettween size and quality. Safe Tensor files tend to use a precision of 4 to 16 bits for quantization.
Open-Weight Models
Gemma 4 is a family of multi-modal and multi-lingual LLMs which can do chat, agentic reasoning and summarization
These models use the Safe Tensor file format. Safe Tensor models can be quantized (compressed).
Quantiztion is a tradeoff bettween size and quality. Safe Tensor files tend to use a precision of 4 to 16 bits for quantization.
Open-Weight Models
Ornith-1.0 is a family of open-source LLMs specialized for agentic coding.
Ornith was post-trained on Gemma 4 and Qwen 3.5.
The model learns its own scaffold during reinforcement learning, jointly optimizing the harness and the solution.
Ornith is competitive with frontier models for code agents.
Open-Weight Models
Granite-4.x is a family of multi-lingual LLMs which can do chat and reasoning.
These models use the Safe Tensor file format. Safe Tensor models can be quantized (compressed).
Quantiztion is a tradeoff bettween size and quality. Safe Tensor files tend to use a precision of 4 to 16 bits for quantization.
Open-Weight LLMs for the llama.cpp Runtime
Open-weight LLMs which are supported by llama.cpp b9370 on CPUs should work in this release.
The following list covers the models which are known to work with llama.cpp in the Private Large Language Model release 26.2.
Examples of Open-Weight LLMs for the Llama.cpp Runtime
Open-Weight Models
Llama 3.1 and 3.3 are multi-lingual chat LLMs.
Scout and Maverick are multi-modal and multi-linugal mixture of expert LLMs.
These models use the GGUF file format. GGUF models can be quantized (compressed).
Quantization is a tradeoff between size and quality. GGUF files tend to use a precision of 1 to 16 bits for quantization.
Open-Weight Model
The Ministral 3 family of models have dense and sparse mixture of experts models of various sizes.
These models use the GGUF file format. GGUF models can be quantized (compressed).
Quantization is a tradeoff between size and quality. GGUF files tend to use a precision of 1 to 16 bits for quantization.
Open-Weight Models
Gemma 4 is a family of multi-modal and multi-lingual LLMs which can do chat, agentic reasoning and summarization.
These models use the GGUF file format. GGUF models can be quantized (compressed).
Quantization is a tradeoff between size and quality. GGUF files tend to use a precision of 1 to 16 bits for quantization.
Open-Weight Models
Ornith-1.0 is a family of open-source LLMs specialized for agentic coding.
Ornith was post-trained on Gemma 4 and Qwen 3.5.
The model learns its own scaffold during reinforcement learning, jointly optimizing the harness and the solution.
Ornith is competitive with frontier models for code agents.
Open-Weight Models
Nvidia-Nemotron-3 is a family of multi-lingual LLMs which can do chat and agentic reasoning.
These models use the GGUF file format. GGUF models can be quantized (compressed).
Quantization is a tradeoff between size and quality. GGUF files tend to use a precision of 1 to 16 bits for quantization.
Clients for the Private LLM Service
SQL and PLSQL
The Oracle AI Database 26ai PLSQL packages DBMS_VECTOR and DBMS_VECTOR_CHAIN can be used to chat to LLMs via the UTL_TO_GENERATE_TEXT procedure.
The OpenAI API for /v1/chat/completions is used as the REST protocol.
The URL and model determine which LLM is used.
Private Agent Factory
Oracle Private Agent Factory provides a low code / no code development environment for creating, testing and deploying Agentic solutions.
The LLM configured in the Model Management of the Private Agent Factory can be used as an Agent in the workflow.
The configured LLM can be from the LLM Service of the Private AI Services Container.
OpenAI Clients
The LLM Service of the Private AI Services Container supports the /v1/chat/completions endpoint.
This means that OpenAI API clients (eg Python, JavaScript, Java and .NET) can chat with the LLM Service.
The following attributes need to be defined:
- BASE_URL - the URL of the LLM Service
- API_KEY- The container's API Key
- Model Name - the model name from the container's config.json
LangChain
LangChain supports OpenAI-compatible backends.
You need to set the following parameters for the LLM configuration:
- model
- openai_api_key
- openai_api_base
Your API KEY enables free tokens and you can use any Llama.cpp or vLLM agent enabled model that you have configured in the Private AI Services Container.
OpenCode
OpenCode is an open source coding agent which can be used to generate code, or to debug code.
OpenCode can be used with these medium size models for code generation for simpler tasks:
OpenCode can be used with these large models for more complex code generation and debugging tasks:
Hermes Agent
The Hermes Agent enables autonomous agents to be built which do not require a human in the loop.
Choose the size/power of a model based on the complexity of the task, eg:
- mistralai/Ministral-3-3B-Reasoning-2512 for simpler tasks
- NousResearch/Hermes-4.3-36B for medium tasks
- deepreinforce-ai/Ornith-1.0-397B for complex tasks
Postman
Postman enables raw HTTP POST messages to be sent to REST Servers.
The LLM Service of Private AI Service Container is a REST server which implements the OpenAI API for the /v1/chat/completions endpoint.
You can use Postman to Chat with an LLM in the LLM Service by sending messages to the /v1/chat/completions endpoint.
Download, Install and Configure
Download and Install
The Private Large Language Model Service is included as a feature of the Private AI Services Container and is downloaded as part of the container.
Download and Install the Private AI Services Container.
Configure Large Language Model
The standard config.json file for the Private AI Services Container is used to configure LLMs:
- Name - A string which represents the model
- Path - Provider/model-name from Hugging Face
- Capabilities - The type of output for the model
- Runtime - The LLM runtime (vLLM or llama.cpp)
- StartupTimeout - How many seconds to wait for the runtime to start
- Runtime_arguments - Optional list of LLM runtime parameters
Getting Started with the Private LLM Service
Douglas Hood, Product Manager, OracleThis blog shows you how to use the Private Large Language Model Service after it has been installed.
Featured Private Large Language Model Blogs
- March 23, 2026Installing the Private AI Services Container
- August 14, 2026Using the Private Large Language Model Service with PLSQL with HTTP
- August 14, 2026Using the Private Large Language Model Service with PLSQL with SSL