Run a Private AI Chatbot or Team Chat Over a Site VPN

An internal chatbot, support assistant, search interface, or team chat server is a natural Site VPN use case. The browser interface can remain entirely private, reachable only from enrolled devices, while the service connects to local models, a hosted model API, internal documents, or collaboration databases. Removing the public login page reduces scanning and opportunistic abuse, and it makes experimental tools easier to restrict to a small group during evaluation.

Network privacy is only the outer boundary. A connected user can still submit malicious instructions, upload sensitive files, copy output, or trigger tools with excessive permissions. A hosted model provider may still receive prompts, and logs may retain conversations. The design must therefore separate network access, user identity, model trust, data classification, tool authorization, and retention. A VPN answers who can reach the interface; it does not decide what the assistant is allowed to know or do.

Choose the processing boundary first

Map every component that receives message content: the browser, reverse proxy, chat application, database, retrieval service, embedding model, language model, tool server, observability platform, and backup system. A “private chatbot” hosted behind a VPN may still send every prompt to an external API. That can be acceptable when contracts, regional processing, retention, and data classification permit it, but it should be an explicit decision rather than an assumption created by the private URL.

For highly sensitive use cases, run the model and retrieval stack on infrastructure you control and restrict egress. For lower-risk use cases, a hosted model may provide better quality and operations. Hybrid designs can classify prompts and route only approved content externally. Whatever the model location, display a clear notice about permitted data and give users a way to report unsafe output. Do not rely on a long policy document that users must remember while chatting.

Make the web interface private by default

Bind the chat service to loopback or a WireGuard address and place a reverse proxy on the private interface. Use private DNS and HTTPS. The gateway firewall should expose only the VPN port publicly; the chat application, vector database, model runtime, and administration ports should not listen on the public interface. If a separate model server sits on another host, route only the required address and port between service subnets rather than giving every user direct network access to it.

Keep development interfaces separate from production. Notebook servers, model dashboards, debug endpoints, and API documentation can expose secrets or powerful actions. Put them on a different private hostname and restrict their VPN routes or proxy policy to administrators. A single broad “AI network” makes it easy for an ordinary chat user to discover an internal management service. Segment by role and function even when all traffic is encrypted.

Use named users in addition to device peers

A unique WireGuard key identifies an enrolled device, but a shared laptop or compromised account can still misuse the application. Add named user authentication and role-based authorization. Separate ordinary chat, document ingestion, prompt-template editing, connector management, model configuration, and audit viewing. Require stronger reauthentication for changes that affect data sources or tool permissions. Record high-impact actions with a user identity and device context.

Avoid one shared “team” account. It prevents reliable offboarding and makes abuse investigation difficult. Use an identity system the organization already maintains when possible, but keep the application functional if the identity service is temporarily unavailable only according to a deliberate policy. Break-glass access should be rare, monitored, and stored outside the chat system. Removing a person should disable both their application identity and any dedicated VPN peers.

Treat prompt injection as an application threat

A VPN does not make user input trustworthy. Prompt injection can be entered directly by a user or indirectly through a document, webpage, email, image text, or data connector processed by the model. The OWASP LLM Prompt Injection Prevention Cheat Sheet describes the risk of instructions embedded in untrusted content and recommends layered controls. Design the system as though retrieved data may actively attempt to alter model behavior.

Separate system instructions from user and retrieved content in the application structure, but do not assume formatting alone creates a security boundary. Validate tool arguments, constrain output formats, use allowlists, and require confirmation for consequential actions. Test with adversarial documents and messages. A model that summarizes internal text may be low impact; a model that can send email, change DNS, query production databases, or execute Python has a much larger blast radius.

Give tools the minimum authority

Run each connector under a service identity with the smallest permissions needed. A documentation assistant may need read-only access to one collection, not the entire file platform. A support bot may draft a response but should not send it automatically. A Python execution tool should run in an isolated environment with resource limits, no default secrets, controlled network egress, and disposable storage. Never pass broad administrator credentials into a model context.

Use deterministic policy outside the model. The application should decide which tool is available to which user, validate parameters, and enforce approval steps. Do not ask the model to police itself with a sentence such as “never perform dangerous actions.” For payment, infrastructure, or account changes, show the user the exact proposed action and require a separate confirmation. Log the request, validated parameters, result, and approver without storing more sensitive content than necessary.

Protect documents, embeddings, and conversation history

Retrieval systems create additional copies of source content: parsed text, chunks, embeddings, metadata, caches, and indexes. Apply the source document’s access policy to every derivative. If two teams have different permissions, do not place their content in one index and rely only on the model to omit restricted answers. Filter retrieval by authenticated user and source ACL before content enters the prompt. Test for cross-tenant leakage with realistic queries.

Decide how long conversations are retained and whether users can delete them. Training, quality review, and debugging are different purposes and may need separate consent and access. Encrypt storage, restrict administrator access, and avoid placing API keys or credentials in chat history. Backups and exported transcripts need the same protection as the live database. If the system supports file upload, scan files, limit type and size, and remove them according to a documented schedule.

Control model and dependency changes

Model behavior can change after an API version update, a local model replacement, a system prompt edit, or a retrieval pipeline change. Treat those changes like application releases. Maintain a representative evaluation set covering safety, data boundaries, tool use, refusal behavior, latency, and output quality. Record the model identifier and configuration used for each release so unexpected behavior can be reproduced.

Patch the operating system, Python environment, web framework, model server, and dependencies. Pin packages where appropriate and scan them before deployment. A private network reduces exposure but does not protect against a compromised package or malicious model artifact. Verify model and container sources, limit build credentials, and separate development secrets from production. The Site VPN should not be the only control protecting a powerful Python service from supply-chain risk.

Monitor the service without turning logs into a data leak

Operational logs need request identifiers, latency, errors, model version, tool calls, and security events. Full prompts and outputs are often the most sensitive data in the system. Default to redaction or limited capture, and grant access only to people with a defined review purpose. If sample conversations are retained for quality, select and sanitize them through a controlled workflow rather than collecting every exchange indefinitely.

Alert on unusual connector use, repeated attempts to override system behavior, bulk document access, large exports, new administrator actions, and access from unexpected peers. Monitor the VPN and application together so a newly enrolled device generating high-volume retrieval is visible. Prepare an incident process that can revoke a peer, disable a user, rotate model and connector credentials, suspend tools, and preserve necessary evidence without keeping the risky service online.