Configuring the AI Engine — Azure AI Foundry vs Local Ollama

AI Insights 365 On-Prem Edition ships with a pluggable AI engine. Every AI feature in the platform — natural-language-to-SQL chat, AI auto-report generation, chart and anomaly explanation, transform suggestions, and reusable named agents — routes through a single provider abstraction. You pick the provider once during setup and can change it later without touching the rest of the platform.

This guide explains the two supported providers, how to configure each, how to switch between them, how to set and monitor token budgets, and how to verify the engine is working.

Because the engine is abstracted, switching providers does not require re-learning the product or rebuilding any content. Reports, agents, datasources, and dashboards are unaffected by the choice; only the model behind the AI features changes. That separation is deliberate — it lets you treat the model as an operational decision that can evolve as your hardware, compliance posture, or quality requirements change.

The two provider options

The AI engine supports exactly two backends. The choice is a deployment decision driven by your data-residency, latency, and operational requirements.

Option A — Azure AI Foundry (cloud)

The platform calls a model hosted in Azure AI Foundry. You provide an endpoint, an API key, and a model deployment name. Azure AI Foundry gives you access to large, high-quality models without provisioning or maintaining GPU hardware on-site.

The engine can be configured with two distinct deployments:

• A heavy model for demanding work such as generating a full auto-report or writing complex SQL.
• A fast model for interactive, latency-sensitive tasks such as chart explanations and short follow-up questions in chat.

An optional model router can sit in front of both deployments and direct each request to the most appropriate model based on cost and latency targets, so simple prompts do not consume heavy-model capacity.

Option B — Local Ollama (fully air-gapped)

The platform calls a model served by Ollama running on the customer's own hardware. There are no external network calls of any kind — the model weights, the inference, and your data never leave your environment. This is the correct choice for air-gapped networks and for organizations whose policies prohibit sending data to a cloud service.

Because inference runs locally, model quality and throughput are bounded by the hardware you provision. The trade-offs are predictable and are discussed below.

Azure AI Foundry vs Local Ollama

Neither option changes how AI features behave functionally. NL-to-SQL, auto-reports, and explanations work the same way regardless of provider; only the model serving the requests differs.

Choosing a provider during setup

The provider is selected in the setup wizard when you first stand up the platform. The wizard prompts for the provider and then for the provider-specific connection details. The selection is written to configuration and is fully editable later in System Settings, so an initial choice is never permanent.

If you are unsure: pick Ollama when any data-residency or air-gap requirement applies, and pick Azure AI Foundry when you want the highest model quality and have no restriction on outbound calls to Azure.

Configuring Azure AI Foundry

Provide the endpoint, API key, and model deployment from your Azure AI Foundry resource. The platform reads the following configuration keys:

Ai:Provider=AzureAi
AzureAi:Endpoint=https://your-resource.openai.azure.com
AzureAi:ApiKey=<your-api-key>
AzureAi:Model=<heavy-model-deployment-name>
AzureAi:FastModel=<fast-model-deployment-name>
AzureAi:UseModelRouter=true

Notes on each key:

• AzureAi:Endpoint — the base endpoint of your Foundry resource.
• AzureAi:ApiKey — the resource key. Store it as a secret; it is encrypted at rest by the platform along with other sensitive settings.
• AzureAi:Model — the deployment name of the heavy model used for report generation and complex SQL.
• AzureAi:FastModel — the deployment name of the fast model used for interactive tasks. If you do not configure a fast model, all requests fall back to the heavy model.
• AzureAi:UseModelRouter — set to true to route requests between the two deployments by cost and latency.

After saving, the platform validates connectivity to the endpoint before enabling AI features.

When to enable the model router

Enable the router when you have both a heavy and a fast deployment and want the platform to keep interactive latency low while reserving heavy-model capacity for report generation. If you run a single deployment, leave the router off.

Configuring local Ollama for air-gapped operation

The goal of an Ollama deployment is zero external dependency. The high-level steps are: provision hardware, install Ollama, pull the model you intend to use, and point the platform at the local Ollama endpoint.

1. Install Ollama and load a model

On the host that will serve inference, install Ollama and pull a model. In a true air-gapped environment, transfer the model artifact through your approved data-diode or media process rather than pulling over the internet, then load it locally.

# On a host with controlled connectivity, or via approved offline transfer:
ollama pull <model-name>

# Verify the model is present and serves locally
ollama list
ollama run <model-name> "Return the number 42."

Confirm Ollama is listening on its local endpoint (by default on the host's loopback or a private interface on port 11434) and that no outbound internet route is required for inference.

2. Point the platform at Ollama

Set the provider and connection keys:

Ai:Provider=Ollama
Ollama:Endpoint=http://localhost:11434
Ollama:Model=<model-name>

• Ai:Provider=Ollama switches the engine to the local backend.
• Ollama:Endpoint is the address of your Ollama server. Use a private host or interface; never route it through anything that egresses the network boundary.
• Ollama:Model is the name of the model you loaded in step 1.

Because Ollama serves a single model per endpoint, there is no separate fast-model or router configuration. If you need both a quality tier and a fast tier, run two Ollama endpoints and configure the platform's interactive features to use the lighter one.

Hardware and GPU guidance

Local inference performance is a function of model size and the hardware behind it. Plan capacity against your concurrency and latency expectations.

• GPU is strongly recommended for any interactive use. NL-to-SQL chat and chart explanations feel responsive only when the model can generate quickly; CPU-only inference is viable for small models and light, non-interactive workloads.
• VRAM is the gating resource. The model and its context must fit in GPU memory for best throughput. Larger models produce better SQL and richer reports but require proportionally more VRAM.
• Right-size the model to the hardware. A smaller model on a well-provisioned GPU often delivers a better end-user experience than a large model that spills out of VRAM.
• Plan for concurrency. Token budgets and multiple simultaneous users multiply throughput requirements. Size headroom for peak interactive load, not average.
• Keep the heavy work batchable. Auto-report generation is more tolerant of latency than chat, so a single capable GPU can serve both if interactive volume is modest.

Model size vs quality trade-off

Larger local models generally write more reliable SQL and produce cleaner narrative explanations, but they cost more VRAM and run slower. Start with a mid-sized model that fits comfortably in your GPU, validate output quality against representative datasources, and scale the model up only if quality is insufficient.

Switching providers in System Settings

The provider chosen at setup is not locked in. In System Settings, an administrator can change Ai:Provider and update the corresponding connection keys. A common path is to begin on Azure AI Foundry during a proof of concept, then move to local Ollama before going live in a regulated environment — or the reverse, to validate features quickly against a cloud model before standing up GPU hardware.

When you switch:

1. Open System Settings and select the target provider.
2. Enter the provider-specific keys (endpoint, key/model as applicable).
3. Save. The platform validates the new connection.
4. Run a verification prompt (below) to confirm the engine responds.

RBAC governs who may change these settings; restrict provider configuration to platform administrators. Treat a provider switch as a change-controlled operation: AI features briefly depend on a fresh, unvalidated connection until the platform confirms it, so schedule switches in a maintenance window and run the verification steps before declaring the change complete.

What carries over and what does not

A few practical points when moving between providers:

• Content is preserved. Saved reports, named agents, datasources, and dashboards do not change. They simply route their next AI request to the new model.
• Custom agent prompts are portable, but tune for the model. A system prompt written for a large cloud model may need light adjustment for a smaller local model. Re-test your named agents after a switch.
• Token accounting continues uninterrupted. The per-organization budget meter keeps tracking across the switch; you are changing the model, not resetting usage.
• Output quality may shift. A move from a large Azure model to a modest local model can change how SQL and explanations read. Validate against representative datasources rather than assuming parity.

Token budget configuration and monitoring

The platform enforces a per-organization monthly token budget. The budget is a soft gate: it tracks consumption and gates AI features when the organization reaches its monthly allotment, then resets monthly. Token usage is tracked per user and per organization, so you can see who is consuming capacity.

Use the budget to:

• Keep cloud spend predictable when running Azure AI Foundry.
• Prevent a single workspace from monopolizing local GPU throughput when running Ollama.
• Attribute usage during chargeback or capacity reviews.

Configure the monthly budget per organization in administration, then monitor consumption in the usage views. Because the gate is soft and resets monthly, hitting the budget pauses further AI calls for that organization until the reset rather than failing other platform features. Adjust the allotment if a team consistently runs out before month end.

Verifying the AI engine works

After configuring or switching providers, confirm the engine end to end:

1. Connection check — saving provider settings triggers a connectivity validation. A failure here points to a bad endpoint, key, or an Ollama server that is not reachable.
2. NL-to-SQL chat — open a chat over a known datasource and ask a simple question (for example, a row count or a top-N query). The model should write SQL and explain the result.
3. Chart explanation — open a chart and request an explanation; expect two to four insight bullets.
4. Auto-report — generate a report from a datasource with a short prompt and confirm a dashboard is produced.
5. Token tracking — confirm the requests appear in per-user and per-organization usage, proving the budget meter is recording.

If the connection check passes but features return errors, confirm the model name matches what is actually deployed (Azure) or loaded (Ollama), and that the model is large enough to produce valid SQL for your schemas.

Summary

The AI engine is provider-agnostic by design. Azure AI Foundry delivers top-tier model quality with a heavy/fast split and an optional router, at the cost of outbound calls to the cloud. Local Ollama delivers identical features with zero external network calls, bounded only by the GPU hardware you provision. Both are selected in the setup wizard, editable in System Settings, governed by RBAC, and metered by a per-organization monthly token budget. Choose based on your data-residency posture, validate output against real datasources, and size hardware to the model when you run air-gapped.