How Manu Agrawal Is Building the Governance Layer for Enterprise AI

The gap between building an AI prototype and deploying an enterprise AI system that runs reliably in production has become one of the industry's most important problems. Pilots that work in controlled environments often stall when they meet real operational workflows, fragmented data, compliance requirements, cost constraints, and human oversight needs. Two years into the generative AI cycle, the question has shifted from 'what can these models do?' to 'how do organisations operate them safely?'

That shift is the territory Manu Agrawal works in. Agrawal is a Software Architect at Oracle Health, where she serves as Chief Architect, leading AI/ML initiatives for healthcare providers, national health systems, and state health programmes. Before Oracle Health, she worked at Amazon Web Services, contributing to large-scale AI/ML, cloud infrastructure, Amazon Bedrock, CloudFront, and multimodal AI systems. Her work sits at the intersection of governed AI, agentic systems, healthcare operations, and production-scale enterprise architecture.

Agrawal's view of enterprise AI is structured around three requirements she sees as inseparable: governance, orchestration, and human oversight. She does not treat them as policy language or post-deployment controls. She treats them as system-level design requirements that determine whether AI can move from a promising demo to a trusted production system.

Operationalisation Is the New Bottleneck

Governance, in Agrawal's framing, is what makes an AI system deployable in high-stakes environments. It covers data access, policy enforcement, source traceability, accountability, auditability, and the trust contract between the AI system and the organisation using it.

'A key differentiator in my work has been focusing not only on AI model capability but also on the infrastructure, orchestration, governance, and operational reliability required to safely deploy AI systems in mission-critical enterprise environments,' Agrawal said.

That distinction matters because many enterprise AI efforts still start with the model instead of the operating environment around the model. In regulated settings, a model output must be traceable, reviewable, and governed. If an organisation cannot understand what data was used, what evidence was retrieved, who reviewed the output, what was changed, and how the result was applied, the system is not ready for production.

For Agrawal, governance is not a blocker to AI adoption. It is what allows adoption to happen responsibly. The organisations that succeed with enterprise AI will be the ones that can prove not only that the model works, but that the workflow around it is safe, monitored, and accountable.

Healthcare Shows Why General-Purpose AI Is Not Enough

The recent emergence of healthcare-specific AI platforms has made the production gap more visible. General-purpose generative AI tools are powerful for broad knowledge work. They can summarise complex information, draft useful content, and answer questions across many domains. But they are not inherently grounded in healthcare-specific entities, clinical coding systems, payer rules, medical evidence, PHI workflows, privacy controls, or the operational realities of care delivery.

Healthcare workflows require more than fluent language generation. The goal is rarely just to 'summarise this note' or 'answer this question.' A useful healthcare AI system may need to recognise clinical entities, understand longitudinal patient context, map information to the right coding systems, distinguish between a patient-facing explanation and a clinician-facing recommendation, retrieve supporting evidence, identify gaps or inconsistencies, and support decisions without overstepping into unsupported clinical judgement.

That is why general-purpose models have often required substantial customisation in healthcare settings. Retrieval layers, terminology mapping, policy grounding, data governance, access controls, audit logs, validation workflows, human review, and safety guardrails all become part of the production system. The model is important, but the model alone is not the system.

Where Enterprise AI Projects Stall

The second requirement is orchestration. This is the architectural problem of getting multiple agents, retrieval systems, data sources, rules, APIs, and downstream workflows to behave coherently. A single AI assistant performing one task can look impressive in a demo. A network of AI systems coordinating across documents, images, structured data, policies, and operational systems is a different challenge. It is closer to a distributed-systems problem than a chatbot problem.

Agrawal has built at that layer. At AWS, she contributed to the architecture of Amazon Bedrock Data Automation and worked across large-scale AI/ML, cloud infrastructure, and multimodal AI systems. Those experiences shaped her approach to enterprise AI systems, where retrieval, multimodal processing, orchestration, and workflow execution have to operate together.

That experience is especially relevant in healthcare. Provider environments are complex. Different EHR implementations, coding systems, documentation practices, and operational workflows can change what the 'right' answer means. Production healthcare AI usually requires workflow-specific configuration, local terminology mapping, and integration with existing governance processes.

In many cases, the most reliable systems are not pure GenAI systems. They combine clinical algorithms, deterministic rules, retrieval, policy logic, and generative models. The generative model is powerful, but it should not be asked to carry the entire burden of reasoning, compliance, and workflow interpretation by itself.

Measurable Value Comes From Connecting Context to Workflow

Agrawal sees the clearest near-term value in operational, administrative, and analytical workflows where AI can reduce the amount of manual context assembly required from healthcare teams. Healthcare is full of translation problems: translating vendor data into the right longitudinal patient context, mapping concepts across coding systems, identifying the right population subset, preparing chart-review summaries, assembling prior authorisation or claims documentation, and helping teams navigate policies, guidelines, and internal knowledge.

Cohort building is one example. Creating the right patient cohort is not simply a database query. It often requires understanding clinical concepts, inclusion and exclusion criteria, longitudinal records, coding variation, medications, labs, procedures, notes, and timing. An out-of-the-box natural-language query layer can be useful for early exploration, but cohort building in production usually requires more than translating a question into a database query.

Healthcare-specific AI systems can help by translating user intent into candidate concepts, surfacing relevant evidence, identifying gaps in the cohort definition, and helping analysts or clinicians arrive at the right population faster. Similar value exists in clinical documentation support, EHR-based assistance, patient communication drafts, therapy-matching support, trial feasibility, life sciences workflows, and care management. The measurable value comes when AI connects fragmented data to the right clinical or operational context and fits naturally into the workflow.

Human Oversight Must Be Risk-Calibrated

The third requirement is human oversight. Agrawal is skeptical of enterprise AI architectures that treat oversight as something added after deployment. In healthcare and other regulated sectors, oversight has to be designed into the workflow from the beginning.

Her point is not that every AI task needs the same level of human review. The better approach is risk-calibrated oversight. Some steps, such as context gathering, summarisation, documentation support, and operational worklist generation, can be automated or semi-automated when the evidence is clear and the risk is low. But when a workflow reaches a point where the system is influencing clinical judgement, patient safety, eligibility, therapy selection, or resource allocation, it must be able to identify that decision point and route it to human review.

The critical capability is not simply keeping a human in the loop everywhere. It is knowing where human review is required, why it is required, and how that review is captured. That means designing systems with source traceability, feedback loops, review states, override mechanisms, escalation paths, risk signals, and monitoring.

This also changes how organisations should think about autonomy. Trustworthy AI systems are not necessarily the ones with the least autonomy. They are the ones that combine automation with appropriate control, reviewability, and accountability. The goal is not to slow every workflow down with manual checks. The goal is to identify where the risk level changes and route those decisions appropriately.

Healthcare AI Needs Its Own Measurement System

Agrawal argues that enterprise AI evaluation cannot stop at benchmark scores, demo quality, or model fluency. In healthcare, leaders need a measurement system that reflects the workflow being supported. A documentation assistant, a cohort-building tool, a prior authorisation workflow, and a clinical evidence synthesis tool should not be measured with the same scorecard.

General AI metrics such as accuracy, fairness, completeness, and robustness matter. But healthcare needs additional measures: clinical correctness, evidence grounding, source completeness, repeatability, bias across populations, privacy risk, escalation accuracy, human override rates, time saved, cost to serve, and downstream operational impact.

Repeatability is especially important. In clinical evidence workflows, the same patient context and evidence base should not lead to unpredictable reasoning paths without explanation. If an output is questioned later, the organisation should be able to reconstruct what information was available, what the AI produced, what evidence it used, who reviewed it, what changed, and what action was taken.

The same logic applies to guardrails. Generic safety filters can be useful, but they can also over-flag legitimate clinical content or miss workflow-specific risks if they are not calibrated for the healthcare context. A term that looks sensitive in a consumer setting may be clinically necessary in a patient note, behavioural health screening, medication review, reproductive health workflow, or safety assessment. In healthcare, guardrails themselves need validation, monitoring, and tuning against real workflows.

The Next Phase of Enterprise AI

Agrawal's framework is not limited to healthcare. Public-sector services, financial institutions, insurance organisations, and any enterprise operating under audit or regulatory pressure face similar constraints. Healthcare simply makes the problem harder to ignore.

The pattern she describes is becoming central to the next phase of enterprise AI. Governance, orchestration, measurement, and human oversight are no longer secondary considerations. They are the architecture that determines whether AI systems can be trusted in production.

The companies still treating AI as a model-procurement exercise are likely to struggle. The next advantage will come from organisations that know how to operate AI responsibly at scale. That means building systems where the model is only one part of a larger governed environment, supported by data controls, orchestration, monitoring, source traceability, healthcare-specific metrics, human review, and clear accountability.

For Agrawal, that is the real shift underway. Enterprise AI is moving from model capability to operational trust. The winners will not simply be the organisations with the most powerful models. They will be the ones that can prove their AI systems are safe, useful, auditable, and governed in the workflows where they matter most.