A reference architecture for zero-egress LLMs

This is a reference design for an internal AI application whose runtime must not initiate connections outside a defined boundary. It is not a claim that a particular environment has been deployed or verified. Use it to identify components and questions before producing an environment-specific configuration.

The design assumes approved users can access the application, the serving model is available for deployment under its license, and the organization has an operating owner.

Separate preparation from operation

Create two distinct workflows. In preparation, acquire model files, serving software, dependencies, and any required artifacts from approved sources. Record versions and integrity information, review the relevant licenses, and prepare a transfer package.

In operation, load approved artifacts from storage inside the runtime boundary. The application should not need to download weights, fetch packages, or retrieve a tokenizer from the internet at startup.

NVIDIA’s air-gap deployment guide describes this two-phase pattern for NIM: prepare assets while connected, then run using staged local assets. That implementation is one option, not a requirement for this architecture.

Draw the full runtime

A text schematic is enough to expose the important paths:

Authorized internal user
          |
          v
+--- Defined runtime boundary ------------------+
| Identity / gateway -> application             |
|                           |                   |
|              +------------+-----------+       |
|              v                        v       |
|        Retrieval service       Model server   |
|              |                        |       |
|     Documents and indexes      Local weights  |
|                                               |
| Local logs, metrics, secrets, artifact store   |
+-----------------------------------------------+
       External outbound paths denied

The identity service belongs inside this diagram only if it actually operates within the boundary. Otherwise, show its external dependency and decide whether that violates the requirement. Apply the same rule to every component.

Define each connection

For every arrow, record the source identity, destination, protocol, data transferred, and business purpose. Network policy permits required internal paths and denies the rest. Access to the application also needs authentication and authorization; egress isolation does not replace ingress control.

Distinguish workload networking from the host and platform. A policy applied to model pods may not constrain image pulls by nodes, control-plane traffic, or administrator tooling. Those paths belong in the infrastructure review.

DNS deserves its own design. Internal name resolution should not become an indirect path for arbitrary external queries. Kubernetes documents that its default cluster DNS behavior can forward non-cluster names upstream. Inspect the resolver’s behavior, not just the application’s access to it.

Keep supporting AI services inside

For a document assistant, local inference alone is insufficient. OCR, embeddings, reranking, and any evaluator that receives document content need approved processing locations. Put their model assets through the same preparation workflow.

For agents, an external tool can transmit information even when reasoning is local. A tool broker should enforce a defined action and data policy. If approved external actions are required, label the system as having controlled egress and describe the boundary precisely.

Plan updates and recovery

The transfer process should produce a versioned release bundle and an inventory of what changed. Test a candidate bundle in a representative environment before promotion. Retain a compatible rollback bundle and document whether indexes need rebuilding.

Keep logs and backups in approved locations. Include the support workflow: if diagnostic bundles are exported, define approval and redaction procedures. An operational exception is still part of the architecture.

Specify acceptance evidence

The verification plan should show that the useful workload succeeds while prohibited connections fail. Exercise startup, restart, ingestion, inference, errors, and update paths. Record which identities and network layers were tested.

Deliver the diagram, connection inventory, deployed versions, policy configuration, test results, and known gaps together. The verification guide expands that acceptance process; the operations guide covers the continuing responsibilities.