Atlas · Ask, Don't Query
Case Study · AI · BI · System Architecture

Ask,
Don't Query.

Building a reliable natural-language-to-BI layer on Apache Superset: entity resolution, deterministic validation, and why LLMs need guardrails, not trust.

5pipeline stages
3LLM calls
2Python guardrails
0hallucinated charts shipped

Live pipeline trace

atlas · live trace
█

Each line is a real pipeline stage. Amber = LLM call. Green = deterministic Python. Blue = Superset execution.

01

The Problem

Most BI conversations focus on the wrong bottleneck. The complaint is usually "we need more dashboards." But dashboards are not the problem. The problem is exploration.

When a KPI moves, the question is never just what moved. It's why. Answering that requires following a chain of causes downward: from the top-line metric, through its component levers, through the dimensions that drive each lever, until you find the signal. That chain can go four or five branches deep. Most organisations' analytical workflows stop at branch two.

Not because the data isn't there. Because getting to it requires knowing where to look.

That knowledge lives in engineers' heads, not in the interface.

01

Exploration is stuck

KPI drill-downs go four branches deep. Most teams stop at two, not from lack of ability but from lack of orientation.

02

Wrong answers look right

Wrong grain, wrong filter, wrong metric. These errors are easy to make and hard to catch without deep schema familiarity.

03

Dashboards don't fix it

Every new dashboard was a project: a BRD, a UAT cycle, weeks of back-and-forth. You cannot iterate toward an answer when each question takes a sprint.

The result is a predictable pattern: surface-level analysis, gaps in the explanation, signals missed, not from lack of ability but from lack of orientation.

What was needed was a layer between the analyst and the data: one that understood business language, knew the full data estate, and could produce a correct answer without the analyst needing to know the schema. Not a better GUI. A system that could be asked a question and trusted to answer it. That is the gap Atlas was designed to close.

02

The Hypothesis

The hypothesis wasn't that AI could answer questions about data. That claim is too broad to be interesting and too easy to make. The hypothesis was more specific: that a system could be built that understood business language, resolved it to actual schema, and produced a correct chart reliably enough to be trusted by analysts who would use the output to explain business movement to leadership.

That is a much harder bar.

The system was designed around three modes, not one. If a user's question was specific enough to resolve to a known dataset and chart configuration, Atlas would plot it. If it wasn't, Atlas would not guess. It would surface the closest matching tables and fields, giving the user the vocabulary they needed to go further. And because Atlas maintains session context, a user looking at a chart could ask "explain this" and get an interpretation with no re-specification required.

The design choice to include a graceful fallback was not defensive programming. It was a statement about where the reliability boundary actually lies.

The Riskiest Assumption

When a user asks "What are daily sales for KFC in Dubai?", the system must correctly infer that "KFC" is a value in a Brand field, and "Dubai" is a value in a City field. Not the other way around, and not mapped to the wrong table. The LLM has no inherent schema awareness. Getting this wrong silently is the worst possible outcome: a chart that looks correct but answers the wrong question.

A Specific Definition of Reliable

Failure Mode What Happens Verdict
Wrong value, right field Filter returns no results or incorrect results within the correct concept. Visible. Recoverable. Acceptable
Right value, wrong field Query executes. Chart renders. The analyst doesn't know it's wrong. Unacceptable

That asymmetry drove the entire validation layer. The LLM proposes. Python decides whether the proposal is legal.

From MCP to a Custom Agent

The earliest thinking started with MCP, exposing Superset's capabilities through a standard protocol and letting an LLM agent drive. It was a reasonable starting point. But as the requirements sharpened, a custom agent architecture made more sense.

MCP works well for tool exposure. It is less suited to enforcing the kind of opinionated logic a business deployment requires: the entity resolution rules, the validation asymmetry, the fallback behaviour. These are not generic capabilities. They are decisions about how the system should behave when it is uncertain, and they needed to live in code that could be reasoned about, tested, and changed.

The second consideration was auditability. Every stage of Atlas's pipeline is deterministic and logged. The dispatcher's classification, the resolver's field mappings, the planner's chart decisions, the compiler's SQL: each produces a structured log entry. For an organisation where analysts are presenting outputs to leadership, that traceability is not a nice-to-have. It is a governance requirement.

03

The Architecture

The obvious question: why not pass the user's question directly to an LLM and ask it to generate a chart?

The answer is that a single LLM call cannot simultaneously do all of the following reliably: classify intent, resolve entity references to actual schema values, select the right dataset, choose an appropriate chart type, generate valid SQL, and validate the output against a live database. Asking one model to do all of it produces a system that fails in ways that are hard to predict, hard to diagnose, and hard to fix.

Atlas separates these concerns into five discrete stages. Each stage does exactly one thing. Each failure can be localised to a specific component and addressed without touching the rest of the pipeline.

Pipeline

Input

"monthly sale KFC-UAE by channel"

→

Dispatcher

intent classification

LLM
→

Resolver

entity → schema

Python
→

Planner

chart config

LLM
→

Compiler

SQL + form_data

LLM
→

Validator

schema check → execute

Python
LLM call
Deterministic Python
scroll →
query trace: step through the pipeline

Design Decision

01

Localisable failures

When something goes wrong, the log tells you exactly which stage failed and why. There is no black box.

02

Replaceable components

Model assignments are config values, not architectural decisions. Swapping models does not require rewriting logic.

03

Governed outputs

No chart reaches the user that has not passed through a deterministic validator. The LLM can generate whatever it likes. Python has the final say.

04

Design Decisions

Every architecture is a set of bets. These are the four that mattered most.

From the outset, LLMs in Atlas were given narrow, bounded roles: classify this intent, select this chart type, generate this SQL. They were never asked to reason about the schema, enforce validation rules, or decide what happens when something goes wrong. That reasoning lives in Python.

The driving principle was containment. A system where LLMs make structural decisions is a system where failures are diffuse: hard to locate, hard to reproduce, hard to fix without disturbing something else. A system where LLMs operate within deterministic scaffolding has localisable failures. When the resolver misidentifies an entity, you fix the resolver. The rest of the pipeline is untouched.

This also made the data catalogue a first-class architectural concern. The LLMs needed to be bounded by something real: actual field names, actual metric definitions, actual vocabulary. That meant building the catalogue as part of the system, not assuming it would exist.

Ruled out: LLM-driven schema reasoning Chosen: Deterministic Python scaffolding

The conventional counterargument is that something is better than nothing. In a general-purpose assistant, that is defensible. In a system where analysts are explaining KPI movement to leadership, it is wrong.

A wrong chart that looks right destroys trust in a way that is very hard to recover from. More importantly, it leaves the user with no path forward. They do not know why the answer is wrong, so they cannot change the question.

A failure that explains itself does the opposite. Users learn from a system that fails clearly. They learn what Atlas can and cannot resolve. They learn to phrase questions in ways the system understands. That learning curve is a feature. A system that hallucinates fluently denies users that feedback loop entirely.

Ruled out: Best-effort chart with warnings Chosen: Hard stop → fallback to semantic search

The dispatcher, a classification task with a binary output, runs on the cheapest, fastest model available. The planner, which must reason over schema context and make chart decisions, runs on a more capable one. The compiler sits in between.

Cost was the primary driver. But the deeper motivation was a question worth asking explicitly: how lean can this system be made? Every time a simpler model handles a task adequately, it reduces the capability floor required to run Atlas at all. The end goal (run Atlas against a local model, with no external API dependency) is not guaranteed, but the architecture does not foreclose it. That matters for organisations with data residency requirements or latency constraints.

Stage Task Model tier
DispatcherBinary classificationCheapest
CompilerSQL from specMid-tier
PlannerSchema reasoning + chart decisionMost capable

Two things make a data LLM reliable: metadata and context. Metadata is deterministic: field names, metric definitions, vocabulary values. Context is not, but it can be approximated and refined.

Atlas builds both through a profiling step: Superset's own database stores enriched metadata (descriptions, relationships, vocabulary maps) and Superset's own UI surfaces it. The user can see exactly what Atlas sees. The catalogue is not a hidden index. It is a visible, editable artefact that the user can inspect and correct.

This creates a feedback loop that most AI systems do not have. When Atlas fails to resolve a query, the user can look at the catalogue, understand why, and improve it. Better metadata directly improves Atlas's performance. The user is not just a consumer of the system. They are an active participant in refining it.

Ruled out: Hidden semantic index Chosen: Superset as the visible, editable catalogue

The LLM proposes. Python decides.

05

The Proof

Seven snippets from the actual codebase. Each one makes a design decision visible that prose alone cannot.

def resolve_context(message: str, state: SessionState) -> ResolvedContext:
    """Pure Python - no LLM calls."""
    fqn = extract_fqn(message)
    if fqn:
        return ResolvedContext(fqn=fqn, source=ContextSource.EXPLICIT)

    position = extract_position_reference(message)
    if position and state.last_search_results:
        if 1 <= position <= len(state.last_search_results):
            result = state.last_search_results[position - 1]
            return ResolvedContext(fqn=result.fqn, source=ContextSource.POSITION)
        raise ContextError(f"No result at position {position}")

    if has_pronoun_reference(message) and state.focus:
        return ResolvedContext(fqn=state.focus.fqn, source=ContextSource.PRONOUN)

    if references_last_chart(message) and state.last_chart_fqn:
        return ResolvedContext(fqn=state.last_chart_fqn, source=ContextSource.LAST_CHART)

    return ResolvedContext(fqn=None, source=ContextSource.NONE)
def validate_filters(filters, schema):
    vocab_map = schema.get("vocab_map", {})
    normalized_filters = {}
    errors = []

    for column, values in filters.items():
        if column not in vocab_map:
            normalized_filters[column] = values
            continue

        valid_lower_to_original_map = {v.lower(): v for v in vocab_map[column]}
        resolved_values = []
        for value in values:
            if value.lower() in valid_lower_to_original_map:
                resolved_values.append(valid_lower_to_original_map[value.lower()])
            else:
                errors.append(
                    f"Value '{value}' is not valid for column '{column}'. "
                    f"Valid options include: {', '.join(vocab_map[column][:5])}..."
                )
        normalized_filters[column] = resolved_values

    if errors:
        raise ValidationError(errors)
    return normalized_filters
def _slim_candidate(candidate):
    if candidate.get("type") != "dataset":
        return {"fqn": candidate["fqn"], "name": candidate["name"], "type": candidate["type"]}
    return {
        "fqn": candidate["fqn"],
        "name": candidate["name"],
        "type": candidate["type"],
        "table_name": candidate.get("table_name", ""),
        "main_dttm_col": candidate.get("main_dttm_col", ""),
        "columns": [
            {"column_name": c["column_name"], "description": c.get("description", "")}
            for c in candidate.get("columns", [])
        ],
        "metrics": [{"metric_name": m["metric_name"]} for m in candidate.get("metrics", [])],
    }
def _build_time_where_clause(time_requirements, main_dttm_col):
    if not time_requirements:
        return ""
    if time_requirements.type == "relative":
        value = time_requirements.value
        if value == "last_month":
            return f"{main_dttm_col} >= DATE_TRUNC('month', CURRENT_DATE - INTERVAL '1 month') AND {main_dttm_col} < DATE_TRUNC('month', CURRENT_DATE)"
        elif value == "yesterday":
            return f"{main_dttm_col} >= CURRENT_DATE - INTERVAL '1 day' AND {main_dttm_col} < CURRENT_DATE"
    elif time_requirements.type == "quarter":
        q, y = time_requirements.quarter, time_requirements.year
        start = f"{y}-{3*q-2:02d}-01"
        end = f"{y+1}-01-01" if q == 4 else f"{y}-{3*q+1:02d}-01"
        return f"{main_dttm_col} >= '{start}' AND {main_dttm_col} < '{end}'"
async def dispatch(message: str, history: List[Message]) -> str:
    instructions = Path(__file__).with_name("prompt.md").read_text()
    history_text = "\n".join(f"{m.role}: {m.content}" for m in history[-10:])
    model_name = os.getenv("SUPERSET_DISPATCHER_MODEL")
    llm = ChatOpenAI(model=model_name, temperature=0, response_format={"type": "json_object"})
    response = await llm.ainvoke([
        SystemMessage(content=f"{instructions}\n\nConversation history:\n{history_text}"),
        HumanMessage(content=message),
    ])
    return json.loads(response.content)["intent"]
def find_candidates(message: str, resolved_fqn: str | None) -> List[Dict]:
    if resolved_fqn:
        asset = load_asset_by_fqn(resolved_fqn)
        if asset:
            return [asset]
        raise AssetNotFoundError(resolved_fqn)
    return faiss_search(message, top_k=3)
@dataclass
class SessionState:
    conversation_id: str
    history: List[Message]
    focus: Optional[Focus] = None
    last_search_results: Optional[List[SearchResult]] = None  # For "the 2nd one"
    last_chart_fqn: Optional[str] = None                      # For "update that chart"

@dataclass
class PlannerResponse:
    sub_intent: str
    asset_fqn: str
    asset_type: str
    reasoning: str
    dimensions: Optional[List[str]] = None
    metrics: Optional[List[str]] = None
    filters: Optional[Dict[str, List[str]]] = None
    time_requirements: Optional[TimeRequirements] = None
    viz_hint: Optional[str] = None
    chart_title: Optional[str] = None
    limit: Optional[int] = None
06

What's Next

A system's maturity is measured not just by what it does, but by how clearly its builder understands what it does not yet do. Three honest assessments.

Prototype → Product

Metadata management is still a manual operation

The profiling step currently runs as a CLI command. An engineer triggers it. That is not a product; that is a prototype.

The longer-term requirement is more fundamental. Metadata management cannot remain a standalone feature that runs adjacent to Atlas. It needs to be baked into the system's operational loop. When a new dataset is added to Superset, Atlas should know. The catalogue is not a one-time artefact; it is a live dependency, and it needs to be treated as one.

Architectural Regret

I would build a plugin, not a browser extension

The current interface is a browser extension. It works. It is also a workaround. The extension authenticates to Superset using an admin account, so every query runs with elevated permissions. The user's RBAC configuration and data access restrictions are bypassed entirely.

If I were starting over, I would build Atlas as a native Superset plugin. That integration would inherit the user's session, permissions, and row-level security rules automatically. The authentication problem disappears. The surface area for security incidents shrinks to near zero.

Prerequisite, Not Roadmap

RBAC and row-level security must be resolved before enterprise deployment

Atlas, as currently deployed, runs as an admin. In a single-team context with a known set of analysts and a curated dataset, that is a manageable risk. In a 500-analyst organisation with data segmented by region, brand, or role, it is not a deployment. It is a liability.

The fix is architectural: move authentication to per-user session tokens, enforce Superset's native RBAC at the API call layer, and ensure that row-level security filters apply to every query Atlas generates. The pipeline design supports this change. Swapping admin credentials for user-scoped tokens is a contained modification. But it is a prerequisite, not a roadmap item.