TOML Datasource Configuration & ETL Pipeline

Phase 2: Two Pillars, Fifteen Connectors

Phase 2 of AIInsights365.net is organized around two architectural pillars: a TOML-based configuration system that makes datasource, agent, and ETL job definitions version-controllable and human-readable, and an AI Insight ETL pipeline that uses DAX queries to extract curated semantic metrics from tabular models and load them into a structured insight store. Layered on top of both pillars is a dramatic expansion of the connector catalog to fifteen production-grade datasources.

What Is TOML?

TOML stands for Tom’s Obvious Minimal Language. It was designed by Tom Preston-Werner (co-founder of GitHub) as a configuration format that is semantically unambiguous, easy for humans to read and write, and easy for machines to parse. Unlike JSON, TOML supports inline comments (lines beginning with #) and avoids the error-prone trailing-comma rules that trip up junior engineers. Unlike YAML, TOML uses explicit typing (strings are always quoted, integers are never ambiguous, datetimes are RFC 3339 values) and does not rely on indentation for structure, eliminating the “two spaces vs four spaces” class of bugs. Unlike INI files, TOML supports nested sections and typed arrays, making it expressive enough for complex configuration without becoming a mini-programming-language.

In a CI/CD world where configuration changes are reviewed in pull requests, TOML’s line-by-line readability produces clean diffs that any engineer can approve without deep domain knowledge. Switching a datasource from one host to another produces exactly one changed line. Adding a new agent produces one new [[agent]] block. These are the kinds of review interactions that build confidence and reduce misconfiguration incidents.

TOML in AIInsights365.net Phase 2

Phase 2 introduces TOML configuration for three object types: datasources, agents, and ETL jobs. Each type has its own section syntax. Here are representative examples.

Datasource Block

# Snowflake analytics warehouse
[datasource.analytics_warehouse]
type     = "snowflake"
account  = "myorg.us-east-1"
database = "ANALYTICS"
schema   = "PUBLIC"
warehouse = "COMPUTE_WH"
role     = "AIINSIGHTS_ROLE"

[datasource.analytics_warehouse.auth]
method           = "key_pair"
username         = "svc_aiinsights"
private_key_path = "/run/secrets/snowflake_rsa.p8"

Agent Block

[[agent]]
name       = "Finance Analyst"
datasource = "analytics_warehouse"
model      = "gpt-4o"
token_budget = 50000

system_prompt = """
You are a financial analyst assistant for Acme Corp.
Revenue always means net revenue after discounts and returns.
Our fiscal year starts July 1. Never query staging_ prefixed tables.
"""

ETL Job Block

[etl.daily_kpis]
schedule   = "0 3 * * *"        # 03:00 UTC every day
source     = "analytics_warehouse"
engine     = "dax"
output     = "insight_store"

[etl.daily_kpis.measures]
include = [
  "Total Revenue",
  "Revenue YTD",
  "Revenue YoY %",
  "Gross Margin %",
  "Customer Count",
  "Average Order Value",
]

Why TOML for Datasource Config?

Configuration files for database connections are frequently the first thing a new team member must touch when onboarding. JSON forces them to hunt for missing commas. YAML silently converts ON to boolean true and port numbers that look like octal to decimals. TOML makes the right thing obvious: strings are quoted, numbers are numbers, and the structure is immediately clear from the section headers. When the same file is committed to a Git repository, code reviewers can see exactly what changed without needing to understand the full system. At AIInsights365.net we believe the operational burden of managing many datasources and agents should scale sub-linearly with team size, and TOML configuration is a key part of that bet.

Phase 2 Connector Lineup

Phase 2 expands the AIInsights365.net connector catalog from five to fifteen production-grade sources:

1. Snowflake
2. Google BigQuery
3. Databricks (Delta Lake & Unity Catalog)
4. Amazon Redshift
5. Oracle Database 19c+
6. SAP HANA
7. MongoDB Atlas
8. Elasticsearch / OpenSearch
9. ClickHouse
10. Salesforce (SOQL)
11. HubSpot (CRM API)
12. Google Analytics 4
13. Stripe (payments & billing)
14. Shopify (orders & products)
15. Generic REST / GraphQL endpoints

Each connector ships with a metadata importer that maps the source’s native schema concepts — Salesforce objects, GA4 dimensions and metrics, Stripe resources — into the AIInsights365.net unified schema model. Agents trained on these connectors can answer questions across heterogeneous sources in the same natural-language conversation, making cross-system analysis available to any analyst without SQL expertise.

AI Insight ETL Overview

The AI Insight ETL pipeline is Phase 2’s second major capability. It exists to solve a fundamental problem with raw SQL-based AI analytics: when an agent must reason from raw relational tables, it reconstructs business logic from scratch on every query — reinventing joins, recalculating year-to-date measures, and approximating fiscal calendar logic. This is slow, token-expensive, and error-prone.

The insight store is a read-optimized layer that holds pre-computed, semantically named business measures: “Total Revenue,” “Revenue YTD,” “Gross Margin %,” “Customer Retention Rate.” When an AI agent answers “What is our YTD revenue by region?” it retrieves a pre-calculated row from the insight store rather than generating a multi-join SQL query. The result is faster, more accurate, and consumes far fewer tokens.

The ETL pipeline that populates the insight store uses DAX queries against tabular models, because DAX already encodes the business logic (time intelligence, fiscal calendars, hierarchy navigation) that agents need. A single TOML-configured ETL job, scheduled via cron syntax and managed from the AIInsights365.net admin panel, keeps the insight store current. Phase 2 documentation will include a full walkthrough of the DAX ETL pipeline, the insight store schema, and best practices for measure naming.

What Comes After Phase 2?

The Phase 2 roadmap is ambitious but focused. After shipping TOML configuration, the fifteen-connector lineup, and the DAX Insight ETL pipeline, Phase 3 plans to introduce collaborative workspaces, real-time collaboration on reports (similar to Google Docs), and a public-facing API for embedding AIInsights365.net visualizations in third-party applications. Subscribe to the blog or follow us on LinkedIn to stay up to date as we ship.