Security & Compliance

AI Workflows for Regulated Industries

Grove executes workflows as one-shot directed graphs: input in, DAG runs, output out. No hidden state, no implicit memory, no surprise side effects. Deploy within your own infrastructure, maintain a complete audit trail of every run, and keep regulated data under your control.

Workflows as Bounded Executions

Grove's execution model is predictable by design: each workflow run takes an explicit input, traverses a directed graph of nodes, and produces an explicit output. The graph is defined up front and cannot change mid-run. Every data dependency is visible in the definition. For non-conversational workloads — extraction, classification, enrichment, multi-step analysis — there is no implicit state between runs.

Air-Gap Compatible

Grove runs as a self-contained Kubernetes deployment within your VPC. The orchestration engine, database, and all workflow state remain inside your infrastructure perimeter.

Your LLM Keys, Your Contracts

API keys for LLM providers are passed per-request via HTTP headers. You use your own enterprise agreements with Anthropic, OpenAI, or Google. We are not a party to the LLM inference chain — your data flows directly from your infrastructure to your provider.

Self-Hosted Model Support

For workflows processing the most sensitive data, route specific nodes to self-hosted models running within your VPC. Different nodes in the same workflow can use different providers — cloud APIs for non-sensitive tasks, self-hosted models for everything else.

No Implicit State

A workflow run is self-contained. It receives its inputs, executes its nodes, and returns its outputs. Nothing carries over to the next run except the persisted audit trail. Conversational state is an optional, opt-in layer — most workflows do not use it at all.

Encryption & Secrets Management

Credentials and sensitive configuration are encrypted at rest using industry-standard cryptography. Secret values are never exposed through the API.

Algorithm AES-256-GCM — authenticated encryption with per-value nonces

Key Management Server-side encryption key loaded from environment; never persisted to storage

API Design Values write-only — the API never returns secret content, only confirms existence

Workflow Documentation

Grove supports plain-language workflow documentation for compliance review. Each workflow can carry human-authored metadata and per-node descriptions, and the platform can export the current workflow as a CCO-friendly markdown document.

Plain-Language Metadata

Workflows and agents can include a description, purpose statement, category, data categories, owning team, accountable owner, risk tier, and known limitations. These fields are designed for human reviewers rather than model execution.

Per-Node Documentation

Every node can carry its own description so compliance staff can understand what each step does without reading prompts or application code.

Markdown Compliance Export

A dedicated API endpoint renders the workflow, execution order, node details, referenced models, and accessible tools as a markdown document that can be reviewed directly or printed to PDF.

Metadata Change History

Metadata edits are audit-logged field by field. Compliance staff can see who changed a workflow description, what the previous value was, and what it said at any historical point.

Per-Node Audit Trail

Every workflow execution produces a complete, timestamped record of what ran, when, with what inputs and outputs, and how long it took. This audit trail is persisted to PostgreSQL and queryable via API.

Node Execution Records

Each node in a workflow — every LLM call, tool invocation, and data transformation — generates a durable execution record capturing its upstream inputs, output value, status, error details, and wall-clock duration. Full data provenance for every decision.

Real-Time Event Stream

Server-Sent Events (SSE) stream every workflow event as it happens: node started, node completed, tool calls requested, results received, run completed. Build monitoring and alerting on top of the native event stream.

Run Configuration Capture

The execution configuration for every run — which model, what tools, timeout settings, session bindings — is captured and persisted at run creation time. Full reproducibility of the execution context.

Tool Call Tracking

Every tool invocation within a workflow is recorded: what was requested, what inputs were provided, and when results were received. Both built-in and application-defined tool calls are captured in the audit trail.

Attributed Access-Change Log

Every change to who can do what — roles, grants, API and MCP keys, identity-provider role sync — is written by database triggers in the same transaction as the change, so no code path can skip it. Each entry names the acting principal or the automated source (identity-provider sync, cascade, system), the log is append-only, and it exports to CSV or NDJSON for auditors.

Immutable Records & Non-Destructive Lifecycle

Workflow definitions and execution history are preserved throughout their lifecycle. The audit trail is append-only at the operational level — records are not overwritten or removed during normal operation.

Workflow Immutability

Workflow definitions are create-and-retire only. There is no update operation. A workflow deployed on day 1 remains byte-identical throughout its operational lifetime, so every audit record unambiguously references the exact workflow that was executed. Retired workflows are soft-deleted and retained.

Non-Destructive Audit Trail

Run records and node execution history are not deleted during normal operation. Failed runs retain their complete execution trail — which nodes ran, what inputs they received, what outputs they produced, and where execution halted — so incidents can be fully reconstructed after the fact.

First-Class Multi-Tenant Isolation

Grove models tenants as a first-class entity in the database, not an opaque label. Every tenant-owned table — workflows, runs, sessions, agents, secrets, storage profiles, triggers, disposal log — carries a non-null tenant_id with a foreign key to tenants. Postgres Row-Level Security policies enforce the boundary as defense-in-depth. The repository layer filters every SELECT, UPDATE, and DELETE by tenant_id, and a cross-tenant fetch returns 404, not 403 — existence is not leaked across the tenant boundary.

The Tenant Entity

A real row in a tenants table with id, name, slug, status, and lifecycle states (active, suspended, deleted). A suspended tenant's API keys are rejected by the auth layer at authentication time, before any handler runs.

Per-Tenant Secrets & LLM Credentials

Secrets, provider records, and model-group definitions are keyed on (tenant_id, name) — one tenant's Anthropic key is invisible to another. The SecretStore trait takes a tenant_id on every operation; OAuth tokens, git credentials, and connector credentials live in the calling tenant's namespace.

Per-Tenant Quotas & Budgets

Operators set per-tenant limits for concurrent runs, runs per hour, and USD budget per period. The rate limiter, run scheduler, and broker check the limits before launching work. Caps are observable through tenant usage views.

Tenant Lifecycle API

An admin-gated lifecycle surface — POST /tenants provisions a tenant and mints its initial API key in one call; PATCH /tenants/:id toggles status; GET /tenants/:id/export bundles every tenant-owned row for portability; DELETE /tenants/:id drives the disposal pipeline.

Trusted-Caller Assertion

Keys come in two kinds. Tenant keys are pinned to one tenant and cannot escape it. Service keys may assert a tenant per request via signed headers — the trust model is "trust who, enforce what". A tenant key's assertion headers are ignored at the auth layer.

Per-Run Execution Context

Every workflow run constructs its own execution context — inputs, tool registry, scratch state — at run start, and discards it when the run completes. One run cannot read another's data, even within the same tenant.

Data Disposal & Right-to-Erasure

Grove provides provable data disposal with a forever-retained audit trail. When customer data needs to be removed — whether for retention policies, customer offboarding, or regulatory deletion requests — the disposal is genuine, recorded, and cryptographically attested.

Tombstones for Audit Records

Run history and per-node execution records are tombstoned: their content is erased while the row remains as a marker. The audit trail's structural integrity is preserved — foreign key references survive — without retaining any of the disposed data.

Hard Delete for Sensitive Data

Sessions, conversation messages, and session memory are hard-deleted on disposal. PII-bearing records are physically removed from storage, not flagged or hidden.

Disposal Audit Log

Every disposal event is recorded in a separate, forever-retained audit log: what was disposed, when, by whom, why, and a SHA-256 hash of the original record content. The hash proves the record existed without retaining its content.

Owner-Scoped Bulk Disposal

Records can be tagged with an opaque owner label at creation time. A single API call disposes every workflow, run, and session associated with a given label — ideal for customer offboarding and tenant-scoped erasure requests.

Disposal Endpoints Per-record (run, workflow, session) and bulk (by owner label) with dry-run preview

Cascade Semantics Disposing a workflow cascades to its runs and node executions; sessions are intentionally not cascaded

Audit Query API Disposal log is queryable by table, record ID, owner label, and date range

Durability & Crash Recovery

Grove persists execution state at every step. If the server crashes mid-workflow, failed runs can be resumed from the last completed checkpoint — no data loss, no re-execution of already-completed work.

Checkpoint Persistence Per-node outputs persisted to PostgreSQL as each node completes

Stale Run Detection On startup, orphaned in-progress runs are automatically detected and marked failed

Resume from Checkpoint Failed runs resume execution from the last completed node — already-finished work is not repeated

Atomic Claims Concurrent resume requests are safely handled — exactly one succeeds

Agent Runtime Safety

Grove's autonomous agent runtime is designed for execution inside regulated environments. A goal-driven agent is bounded at four layers — sandbox, allowlist, budget, and durability — each of which is independently auditable.

Sandboxed Workspace

Agents that write files or run shells get a per-run ephemeral workspace under a container sandbox (namespaces, cgroups, --network none, bind-mounted root). Real-path containment resolves symlinks on every file op. A file-writing run is refused when no usable sandbox is available — never run unsandboxed.

Allowlisted Tools

An agent definition declares exactly which tools it may use. The runtime validates the allowlist at definition time against the registry's known tool names and builds a fresh per-run tool registry on every run — nothing leaks between runs.

Budget Caps

Per-run token and USD budget caps stop the loop the moment they trip. Sub-agent delegation passes the parent's remaining budget down so the tree's worst-case spend is bounded, not unbounded. Cost is computed from the model_prices table.

Durable Turn History

Every turn writes to agent_run_turns incrementally. Each tool call is bracketed by a Started / Completed idempotency marker so a crash mid-tool does not re-run a side-effecting operation on resume — the LLM sees an "interrupted" error rather than risking duplicate work.

Sub-Agent Containment

Delegated children are capped on depth (default 5) and fan-out (default 10) per parent. The child-count check is a single atomic UPDATE … WHERE child_count < max — no race that lets siblings exceed the cap. Cancellation cascades to every descendant via a shared atomic flag.

Tenant-Scoped Git Credentials

Agent git_* tools fetch credentials through a tenant-bound SecretStoreCredentialProvider. Per-host tokens come from the calling tenant's secret namespace; they never appear in the agent's prompt or the request body.

Enterprise LLM Provider Management

Register and manage LLM providers through a secure API with support for multiple authentication methods. Route workflows through named model groups with automatic failover.

Five Provider Backends

Anthropic, OpenAI, Google Gemini, Vertex AI (Claude on GCP), and Azure OpenAI. Mix providers within a single workflow based on sensitivity, cost, or capability requirements.

Enterprise Auth

API keys (encrypted in the secrets store), OAuth2 client credentials, GCP managed identity, and Azure managed identity. No plaintext credentials in configuration.

Named Model Groups

Define model tiers — fast, standard, frontier — and the broker resolves them to concrete providers. Change what "fast" means across all workflows without modifying any workflow definition.

Automatic Failover

Model groups support ordered backends. If the primary provider is unavailable (rate limited, down, or timing out), the broker automatically falls through to the next backend.

AI Risk Management & Human Oversight

Grove's controls are designed to support the NIST AI Risk Management Framework (AI RMF 1.0): measuring whether an AI workflow still performs as intended, keeping people in the loop where the stakes require it, and leaving evidence behind for reviewers.

Evaluation Suites

Define a fixed set of test cases for a workflow or agent, each with explicit pass/fail checks, and run it on demand or on a schedule. Every eval run records per-case results, pass rate, latency, tokens, and cost — durable test-and-evaluation evidence you can export for review.

Model Change Regression Testing

Before switching a workflow to a new model, run its eval suite against the candidate model group and compare the results side by side: pass-rate and cost deltas, and exactly which cases regressed or were fixed. Model changes become a measured decision rather than a guess.

Human Approval Gates

Add an approval step anywhere in a workflow. The run pauses durably — holding no resources, surviving restarts, resumable from any replica — until an authorized reviewer approves or rejects with a reason. The decision, reviewer, and timestamp are recorded; approvals can be scoped to specific roles, expire to a safe default, and reject paths fail closed.

Approved Means Unchanged

A paused run is bound to the exact workflow definition it started with. If the workflow is edited while a decision is pending, the run fails closed instead of executing something the reviewer never saw, and a resumed run always acts with its original starter's permissions — never the approver's.

Seal: Human Sign-Off on Agent Actions

Operators write seal rules for what agents may do on their own — by connection, operation, read/write/destructive effect, agent, or parameter values (“payments over $100”). A governed action does not run until a named person, signed in through the browser, seals the exact parameters it will execute. This covers chat, autonomous, scheduled and triggered runs. API keys and agents cannot seal; separation of duties and recent re-authentication can be required per rule.

Sealed Means Exactly What Ran

Each action is hashed in a canonical form when it is proposed, when it is sealed, and again just before it executes; if anything differs, it fails closed and the attempt is recorded. Each decision carries a server-keyed integrity code and the seal audit trail is hash-chained, so an edit to the stored record is detectable. This proves what was sealed matches what ran, as Grove recorded it — it does not protect against someone holding Grove's own keys, and checks on tools that run in your application only apply where your code verifies them.

Outcome Monitoring

Grove tracks trustworthiness signals in production — run success, acceptance verdicts, data-quality check failures, rejected tool calls, structured-output failures, and LLM errors and latency by model group. They are exported to Prometheus with bounded, content-free labels and summarized per workflow and agent with a daily trend, so drift after a model or provider change is visible rather than anecdotal.

AI System Inventory

Every workflow and agent can carry an accountable owner, risk tier, intended oversight mode, known limitations, data categories, and a review cadence. A tenant-wide inventory lists every AI system with overdue reviews and missing governance flagged, and every change is captured in an append-only governance audit log.

Output Feedback & Disputes

Users, operators, MCP clients, and backend services can flag a workflow run, a specific step, an agent run, or a chat reply as incorrect, harmful, unsafe, or a privacy concern. Flags land in a triage queue with an append-only status history; submitters can reopen a resolved item to dispute the outcome, and feedback text is scrubbed when the underlying run is disposed.

PII Redaction Before Inference

An opt-in, tenant-wide policy can detect or redact emails, phone numbers, SSNs, payment cards, IBANs, IP addresses, and custom patterns before any prompt, message, or tool result reaches an LLM provider. Values are replaced with stable placeholders and restored in the model’s output and tool arguments, so workflows keep working while the provider never sees the raw value. Workflows and agents can tighten the tenant policy but never loosen it, and policy changes are audit-logged. Pattern-based detection is best-effort, not a DLP guarantee.

Regulatory Alignment

Grove's architecture supports compliance with data protection frameworks across regulated industries — financial services, healthcare, legal, and any environment where data custody and auditability are required.

Administrative, technical, and physical safeguards

Air-gap deployment, AES-256-GCM encryption (with optional HashiCorp Vault KV v2 backend), first-class tenant entity with Postgres row-level isolation, Kubernetes namespace controls

Service provider oversight

Client-owned API keys for LLM providers — your enterprise agreements, your data relationship with the inference provider. The orchestration vendor is not a party to the LLM inference chain.

Incident detection and response

Per-node audit trail, real-time SSE event stream, persistent run history in PostgreSQL — the technical substrate for detection, investigation, and documentation.

AI risk measurement (NIST AI RMF — Measure)

Evaluation suites with recorded per-case results, scheduled re-evaluation, and side-by-side regression comparison before model changes; per-run acceptance verdicts against declared outcome checks.

Human oversight of AI systems (NIST AI RMF — Map 3.5, Govern 3.2)

Durable approval gates with role-scoped approvers, recorded decisions, expiry to a safe default, a queryable approvals queue, and fail-closed resumption bound to the approved workflow definition; operator-defined seal rules requiring a signed-in person to seal an agent's exact action before it runs, with hash-bound execution and a hash-chained audit trail.

AI system monitoring (NIST AI RMF — Measure, Manage)

Production outcome metrics on Prometheus and a per-workflow/agent outcomes report with daily trends: success rate, acceptance pass rate, check failures, tool-call rejections, latency, tokens and cost.

Inventory and accountability for AI systems (NIST AI RMF — Govern 1.6, Map 1)

A per-tenant AI system inventory with owner, risk tier, oversight mode, limitations, data categories and review dates; overdue-review and missing-governance filters; trigger-written, append-only governance audit log; fields included in the compliance export.

Feedback and appeal of AI outcomes (NIST AI RMF — Measure 3.3, Govern 5)

Attributed per-run, per-step and per-reply feedback with categories, a filterable triage queue, append-only status history, submitter re-open as a dispute path, and scrubbing on disposal.

Privacy risk in AI inference (NIST AI RMF — Measure 2.10)

Opt-in detect or redact policy applied to every LLM provider call, with reversible per-run placeholders, tighten-only overrides, content-free finding counts per step, and audited policy changes.

Recordkeeping and retention

PostgreSQL persistence of workflow definitions, run history, node executions, session data, and execution configuration. Retention is enforced through per-record and bulk disposal endpoints, run on the schedule your regulatory requirements dictate.

Data custody and sovereignty

Entire platform deploys within the institution's cloud boundary. Customer information never routes through external SaaS.

Data disposal and right to erasure

Per-record and bulk disposal endpoints with cryptographically attested audit log. Tombstone-based disposal preserves audit integrity for run history while hard-deleting PII-bearing session data. Owner-scoped purges support customer offboarding workflows.

Grove provides the technical infrastructure for compliance. Regulatory compliance programs, written policies, and legal assessments are the responsibility of the deploying institution and should be developed with qualified compliance counsel.