Making multiple AI assistants work together, safely: delegation, tracking and human oversight

Using more than one AI assistant is increasingly common: one is good at research, another has access to your calendar, and a third runs on your home server. Getting them to cooperate without creating a security mess takes a few simple rules.

Pattern: coordinator plus workers

• Coordinator: keeps the task list, sends work out, and checks results.
• Workers: each does one kind of job with only the access that job needs.
• You: approve anything consequential, and get a short “needs you” list.

Make every handoff traceable

1. Give each request a task ID (e.g. JDW-TASK-014) in the subject or first line.
2. Require a structured reply: JDW-TASK-014-COMPLETE, then the fields you asked for. A missing or garbled ID means the coordinator flags it for you instead of guessing.
3. Track the state in one place: sent, acknowledged, complete, needs-human, timed-out.
4. Set timeouts: no reply in X hours means you get an alert, not a silent retry loop.

Email works surprisingly well as a transport between assistants. It’s asynchronous, logged by default, and every assistant can handle it. A shared task board or webhook works too.

Least privilege, concretely

• The research worker gets web access only, with no email-sending and no file writes.
• The calendar worker gets read-only calendar access, and proposes events for you to accept.
• The coordinator can read replies and update the tracker, but sending to new recipients, spending money or deleting anything requires your approval.
• Separate credentials per assistant, so revoking one doesn’t break the others.

The big risk: one AI instructing another

When Assistant A reads Assistant B’s output, that output is untrusted input. It might contain mistakes or injected instructions from a web page B read. Rules:

• Treat replies as data, not commands. The coordinator extracts fields and doesn’t obey free text.
• Allowlist senders. Only accept task replies from known addresses, and ignore everything else.
• No chain reactions: a worker’s reply can’t trigger an irreversible action without a human.
• Read OWASP’s LLM risks, especially prompt injection and excessive agency.

Human oversight that doesn’t nag

• One daily digest: completed items (with links), items that need you, and anything stuck.
• Instant alerts only for: failures, approvals waiting on you, and anything touching money or other people.
• An audit log of who did what and when, kept in a file you own.

Worked example

1. You ask the coordinator: “Research UniFi CyberSecure pricing.”
2. The coordinator emails the research worker JDW-TASK-020: verify current CyberSecure price/features, return sources.
3. The worker replies JDW-TASK-020-COMPLETE with the price, features and source links.
4. The coordinator checks the format and that the sources are present, then marks it complete. If either is missing, it goes on your “needs you” list.
5. The coordinator drafts an article outline and asks you before publishing anything.

Difficulty and cost

• Difficulty: intermediate (the plumbing is simple, and the discipline is the hard part).
• Cost: the sum of each assistant’s plan or API use. Prevent loops with timeouts and spending limits.