Building a human-in-the-loop Slackbot agent

Mays released a reference implementation called maysco-demo-agent to demonstrate resilient architectures for human-in-the-loop artificial intelligence agents in production. According to mays.co's report, the application simulates a marketplace issuing seller payouts through Stripe Connect while screening legal names against the US Treasury OFAC sanctions list. The system delegates low-risk judgments to an autonomous model while escalating high-risk evaluations to human operators in Slack.

This technical analysis covers how the reference project combines AWS Lambda, Amazon SQS, PostgreSQL, and Anthropic Claude to survive real-world operational failures. It explains triage score thresholds, saga workflows, model error handling, and transactional outbox guarantees for software engineers building automated payment pipelines.

01Three tiers of sanctions screening

The screening engine scores seller legal names against federal sanctions entries to produce a similarity score between zero and one. Scores below 0.82 are cleared instantly by deterministic code without invoking the language model. For instance, Brindlecove Ceramics LLC scored 0.77 and was marked paid just 4.6 seconds after Stripe delivered the webhook.

Names scoring between 0.82 and 0.90 enter a gray zone evaluated by Claude. The agent clears payouts of $500 or less if it determines no match with high confidence, as seen when Copperline Marine Supply cleared in 9.6 seconds. Payouts exceeding $500 or showing naming ambiguity route to a human reviewer in Slack.

Scores from 0.90 to 0.95 require one human reviewer who inspects the model's reasoning on a Slack card without granting the agent clearing authority. Any score of 0.95 or higher mandates approvals from two distinct reviewers while bypassing the model entirely. Hardcoded programmatic rules enforce these boundaries before model results can be considered.

02Payout saga across queue and database

The architecture runs on AWS Lambda, Amazon SQS, and PostgreSQL, defined entirely in sst.config.ts. The application interfaces with Stripe, Slack, and the Claude API. A dedicated Slack Lambda verifies button clicks within Slack's three-second limit, while a worker Lambda handles queue processing, model interactions, and Stripe money transfers.

Each payout executes as a saga beginning when a buyer pays $125.00 through Stripe. An intake transaction records the webhook event, initializes the saga record, and writes an outbox message. The worker fetches seller details, scores the legal name, and posts an interactive card with Approve and Reject buttons to Slack when review is required.

If human reviewers fail to respond, the review expires and funds remain held safely. When approved, the worker rechecks seller details to ensure information remained unchanged during the review window. The worker then executes a $112.50 Stripe transfer to the seller and marks the saga paid in PostgreSQL.

03Typed failure handling for model output

Standard software bugs throw exceptions, allowing Amazon SQS to retry five times before moving messages to a dead-letter queue. Model calls can return HTTP 200 while producing truncated, refused, off-schema, or hallucinated responses. Because these outputs are model limitations rather than system bugs, sending them to a dead-letter queue would leave payouts hidden from human oversight.

To handle unparseable outputs, assess.ts inspects response metadata before downstream code reads the content. The code explicitly checks if the stopping reason equals max_tokens to flag token exhaustion. By using messages.create rather than the SDK helper messages.parse, the system avoids throwing uncaught exceptions on truncated text and returns structured failure reasons instead.

During testing, capping model output at 64 tokens on a Quillmark transaction triggered a max_tokens failure that delivered an actionable review card to Slack. The application only retries calls when facing an unavailable API. Any structural model failure routes directly to human reviewers, preventing repeated automated failures on identical prompts.

04Transactional outbox preventing dual write loss

Saving state to PostgreSQL while dispatching events to Amazon SQS risks data loss across independent systems. If Lambda terminates after updating the database but before queueing the message, the payout stalls invisibly without throwing errors. The application resolves this dual-write problem by implementing the transactional outbox pattern to link database mutations with message dispatch.

Every processing step writes outgoing queue messages to an outbox table within the same PostgreSQL transaction that records the status update. A background sweeper runs every minute, scanning for outbox entries older than 60 seconds that remain unsent. During failure testing on Brindlecove, the sweeper discovered and dispatched stranded queue messages 118 seconds later.

Because the outbox sweeper guarantees at-least-once delivery, downstream components must process messages idempotently. Inbound webhooks store event IDs in an inbox table, returning HTTP 200 with duplicate indicators when resent. Similarly, worker messages contain expected status markers, enabling handlers to safely ignore messages if the saga has already transitioned.

05Idempotency safeguards and live Stripe testing

Live third-party testing revealed operational challenges that local mock services could not reproduce. The transfer logic explicitly verifies whether a Stripe transfer already exists before initiating creation, guarding against cases where Stripe deletes idempotency keys after twenty-four hours. This invariant ensures sellers are never paid twice and funds never move without required approvals.

Interactions with the live Stripe API also exposed fee calculation bugs in the initial transfer implementation. The prototype initially attempted to transfer the entire gross buyer payment of $125.00 to the seller. Live testing proved that platform fees must be subtracted, requiring a net transfer of $112.50 to prevent transfer failures.

By combining programmatic boundaries with asynchronous human approval, the architecture maintains strict financial safety against model hallucinations and infrastructure outages. Payouts never become stuck invisibly, and the language model can never execute actions beyond hardcoded code permissions. The demo proves that robust autonomous agents rely on resilient distributed systems patterns rather than unchecked model trust.