Oracle Private AI Services Container FAQ

Currently two services are provided and more are planned:

• Vector Embedding Service
• Vector Index Service

For the Vector Embedding Service:

• You use REST APIs to communicate with the container
• The OpenAI API (REST API) is used to create vector embeddings and list models

For the Vector Index Service:

• The Oracle AI Database uses an HTTP/2 binary protocol to implement the CREATE VECTOR INDEX SQL statement
  

For the Vector Embedding Service: 

You must have at least 16 GB of free memory to use the container to create vector embeddings. If you want to use large embedding models, or to create multiple different types of vectors at the same time, then you will need more memory.

For the Vector Indexing Service:

You need at least 16 GB of freem memory and about twice th needed VRAM.

For the Vector Embedding Service:

Only one CPU core is needed to make the Private AI Services Container work. Using more than one CPU core can improve performance. Either a single request can be processed in parallel for lower latency, or multiple different requests can be processed in parallel for better throughput. Your desired embedding model, number of concurrent requests and desired latency will determine the number of CPU cores per container and how many containers that you need.  You need to benchmark your workload to determine the optimal configuration.

For the Vector Indexing Service:

You need a minimum of 4 CPU cores.  More than 16 CPU cores will tend to give diminishing performance gains.

For the Vector Embedding Service:

The container does not require a GPU to run.

For the Vector indexing Service:

A modern NVIDIA GPU is required with Linux x86-64 CPU.  The NVIDIA compute capability 7.5 or later is supported.

The 3rd party licenses for the Private AI Services Container can be found here.

Information about logging can be found here.

The standard port for HTTP is 8080.

The standard port for SSL/TLS is 8443.

You can configure alternative ports for HTTP and TLS.

The config.json config file is optional. The config.json file is needed for more advanced configurations. These environment variables can also be used with the config.json file.

The list of the vector embedding models that are included with the container can be found here.

The container can work with any Oracle ONNX Pipeline model. The ONNX Pipeline enables embedding models based on the Transformers or Sentence Transformer Libraries to be converted into the ONNX format.  The ONNX Pipeline models can be run using either the Oracle AI Database 26ai, or the Oracle Private AI Services Container.  More information about creating pretrained models in ONNX Pipeline format can be found here.

A list of the embedding models, including text, image, and reranker models, that are known to work in the container can be found here.

There is no best vector embedding model. The model that you decide on will depend on your use case and your data.

Consider the following factors when choosing your embedding model(s): Are you processing English text only? Are you processing text for non-English languages? Are you processing image data? Are you processing video data? Are you processing DNA data? Are you using text to search for images? Are you using images to search for text? Is the quality of the search the most important factor? Is the speed of the search the most important factor? Do you need a good blend of quality and performance? Do you have a corporate standard embedding model that needs to be used?

A list of compatible runtimes can be found here.

For the Vector Embedding Service:

The REST requests used to create vector embeddings are stateless, so you can use as many containers as you want. Multiple machines can be used.  Also multiple containers can be run on the same host as long as each container listens on a different TCP port. A load balancer can be used to route the REST requests to a container. The Oracle Database Operator for Kubernetes 2.1 and later enables deployments with multiple pods and a load balancer.

For the Vector Index Service:

You can use more than one container, but you must configure the containers and load balancers yourself.  More details are available here.

Using multiple containers with a load balancer makes high availability transparent to the REST clients. If a load balancer is not used, high availability is still possible, but it will not be transparent to the client as you will need to know the relevant TCP ports to route the REST requests to.

The container should not run on the same machine as the Oracle AI Database Server.   The container should run on a Linux machine that is close to the Oracle AI Database Server. The container is designed to accelerate resource intensive tasks such as creating vectors or vector index creation to machines other than the Oracle AI Database. This enables the Oracle AI Database to have low latency and high throughput without being burdened with these resource intensive AI infrastructure tasks.

For the Vector Embedding Service:

The container enables multiple concurrent users. The effective number of concurrent users will be determined by the number of CPU cores and the embedding model.

For the Vector Indexing Service:

Multiple concurrent CREATE VECTOR INDEX statements can be processed at the same time.  However, the GPU needs sufficient VRAM for this to work.  Due to the VRAM constraint, you will likely only create a single vector index at a time within one container.

The container uses the following security:

• TLS 1.3 for SSL
• PKCS12 keystores for encrypted passwords
• API Keys for authentication
• SELinux in enforcing mode is supported

Yes, a modern NVIDIA GPU is required.

Any Linux x86-64 supported GPU with NVIDIA compute capability 7.5 or later should work.

Yes, the RTX 30x0, 40x0 and 50x0 gaming GPU are supported.

Any Linux x86-64 supported GPU with NVIDIA compute capability 7.5 or later should work.

The NVIDIA 580.65.06+ or later drivers are needed.

It can work with CUDA 13.0, 13.1 or 13.2 on the Linux host machine.

You want to use the latest version.

1.18.0, 1.18.2 and 1.19.0 are known to work.

If you have multiple GPUs, you can use the --gpu parameter. 

This means that the NVIDIA drivers or Container Toolkit were not correctly installed.

Check the following:

• podman should give valid output
• nvidia-smi should give valid output
• nvidia-ctk --debug cdi list should give valid output

The first method check whether the podman container is running:

podman ps

The second method check which the container is responding to REST requests:

curl -i --cacert $SECRETS_DIR/cert.pem https://$HOST:8443/health

You need to add the following to the parameters section:

• OFFLOAD_CREDENTIAL_NAME
• OFFLOAD_URL

The following all need to be correctly configured:

• The Container's firewall needs to have port 8443 open
• The cloud network security needs to be able to use port 8443 on the container to send and receive data
• OFFLOAD_CREDENTIAL_NAME needs to be valid
• OFFLOAD_URL needs to be valid
• The database credential needs to have the correct API KEY value for the container
• The network ACE needs to be configured correctly
• A database wallet must exist
• The container's digital certificate needs to be a trusted certificate in the Oracle Wallet on the database
• The database wallet needs to use the correct path