0–10 min: draw the stack
Map gateway, runtime, sandbox, tools, memory, scheduler, domain services, providers, databases and notification channels.
Success checkEvery component, credential flow and network boundary has an owner.
Self-hosted agent architecture guide · Updated 2026-07-05
Self-hosted Agent Stack Architecture Guide: Runtime, Memory, Sandboxes, Gateways, Domain Workflows, Secrets, and Recovery
A practical architecture guide for combining self-hosted agent runtimes, persistent memory, tool execution, communication gateways, domain-specific workflows, scheduling, provider credentials, observability, backup, and recovery.
A self-hosted agent stack is rarely one product. The useful architecture separates the agent runtime from persistent memory, privileged tool execution, messaging gateways, domain workflows, scheduled jobs, provider credentials, and the operations needed to recover from bad runs or compromised integrations.
RepoDaily's current self-hosted set illustrates different layers rather than interchangeable competitors. DeerFlow is a broad long-horizon agent harness with subagents, memory, sandboxes, skills, and a gateway. Hermes Agent emphasizes an always-available personal agent with cross-session learning, reusable skills, multiple communication channels, cron, and deployment backends. Cognee is a dedicated memory control plane. Vibe-Trading is a domain-heavy research agent with backtests, tools, swarm workflows, persistent state, and financial controls. Daily Stock Analysis is a scheduled intelligence pipeline with dashboards and notifications.
RepoDaily verdict
Start by choosing the operating model, not the most feature-rich repository. Use a broad runtime such as DeerFlow when the product needs multi-step research, code execution, subagents, sandboxes, and a programmable gateway. Evaluate Hermes Agent when persistent personal context, skill learning, channels, cron, and flexible deployment are central. Use Cognee as a separable memory layer when multiple agents need durable, queryable, governed context. Treat Vibe-Trading and Daily Stock Analysis as domain workflow patterns rather than general runtimes. Keep provider credentials, memory stores, sandboxes, channels, and scheduled jobs as independently revocable boundaries.
Quick matrix
| Layer | Primary job | RepoDaily example | Architecture question |
|---|---|---|---|
| Agent runtime | Plan, call tools, delegate, manage long tasks | DeerFlow / Hermes Agent | Does one runtime own the whole task lifecycle or only orchestration? |
| Subagents / swarm | Parallel or specialized task execution | DeerFlow / Vibe-Trading | How are task scopes, budgets, failures, and outputs isolated? |
| Sandbox / execution | Run code and tools away from sensitive host state | DeerFlow / Hermes backends | What filesystem, network, process, and secret access reaches the task? |
| Memory | Persist context and retrieve useful prior knowledge | Cognee / Hermes Agent / DeerFlow | Is memory per user, per agent, per project, or globally shared? |
| Gateway / channels | Expose agent sessions to UI, CLI, messaging, or API clients | DeerFlow gateway / Hermes gateway | How are clients authenticated and sessions separated? |
| Domain workflow | Encode research, finance, SEO, or other specialist process | Vibe-Trading / Daily Stock Analysis | Which steps are deterministic and which depend on LLM judgment? |
| Scheduler | Run recurring briefings, audits, or background tasks | Hermes cron / Daily Stock Analysis schedules | What happens after missed runs, retries, duplicates, and stale data? |
| Notifications | Deliver output to humans or downstream systems | Hermes channels / Daily Stock Analysis delivery | Can notification credentials trigger actions or only deliver reports? |
| Provider layer | LLM, search, market data, browser, embeddings | All stacks | Are keys scoped, rotated, budgeted, and isolated per environment? |
| Operations | Health, logs, backups, upgrades, rollback, incident response | All stacks | Can the team stop jobs, revoke keys, restore state, and explain what happened? |
Self-hosted agent stack readiness scorecard
Score the complete stack, not only the agent loop.
| Control | 0 points | 1 point | 2 points | Owner question |
|---|---|---|---|---|
| Runtime boundary | Runs with broad host access | Some isolation | Explicit task runtime, filesystem, network, process and secret scopes | What can one compromised task reach? |
| Memory isolation | One global store | Manual namespaces | Per-user/agent/project scopes with deletion, retention and backup policy | Can one tenant retrieve another tenant's context? |
| Tool policy | Generic shell/browser/filesystem | Some wrappers | Narrow tools, approvals, budgets and audit events | Which actions can create irreversible side effects? |
| Provider secrets | Shared .env everywhere | Separate environments | Scoped identities, vault/secret boundary, rotation and budget controls | Which service owns each credential? |
| Gateway security | Open local/LAN API | Authentication only | Auth, session isolation, origin policy, rate limits and kill switch | Who can start, resume, or steer a session? |
| Job reliability | Best-effort scripts | Basic retries | Idempotency, retry policy, duplicate prevention, dead-letter/recovery path | What happens when a scheduled job fails halfway? |
| Observability | Console logs | Central logs | Task IDs, tool events, provider usage, errors, redaction and retention | Can we reconstruct one bad run end to end? |
| Recovery | Reinstall is the plan | Manual backup | Tested config/data backup, key rotation, rollback and clean-room restore | Can the service recover without trusting compromised state? |
60-minute self-hosted agent stack architecture review
Use this before standardizing a runtime or granting persistent privileged access.
10–20 min: trace one task
Follow a representative request from client identity through planning, tool calls, memory reads/writes, artifacts, notification and logs.
Success checkThe team can explain where authority and data cross boundaries.
20–30 min: inspect secrets and tools
List every credential and privileged tool available to each service and execution context.
Success checkNo component has broad secrets or actions without a justified need.
30–40 min: attack isolation
Test cross-session memory access, sandbox escape assumptions, unauthorized gateway clients, wrong-scope tool calls and stale scheduled jobs.
Success checkThe stack fails closed before external side effects.
40–50 min: test operations
Stop a task, restart a service, retry a failed job, rotate one provider key, restore memory/config backup and inspect resulting logs.
Success checkCommon failures have deterministic operational paths.
50–60 min: choose the minimum stack
Remove unnecessary agents, gateways, memory systems, tools, providers and schedules from the first production design.
Success checkThe initial architecture is small enough for the team to own and recover.
Self-hosted agent architecture decision flow
- Start with the job: general research/coding agent, persistent personal agent, shared memory service, domain research workspace, or scheduled intelligence pipeline.
- Choose the runtime boundary. Decide whether the agent runs on the host, in containers, through SSH, in ephemeral sandboxes, or through serverless/remote execution backends.
- Separate privileged tools from conversational state. List shell, filesystem, browser, network, database, container, finance, and notification actions, then narrow scopes and approval requirements.
- Design memory as its own data system. Define source ingestion, namespaces, retention, deletion, backup, model/embedding dependencies, tenant separation, and how retrieval evidence is inspected.
- Design the gateway separately from the runtime. Authenticate clients, isolate sessions, validate browser origins where relevant, limit message size/rate, and provide an emergency stop path.
- Classify domain workflows into deterministic stages and LLM judgment stages. Preserve source data, intermediate artifacts, prompts/config, tool results, and final outputs so recurring reports can be audited.
- Assign provider identities by service and environment. Do not let every component inherit every LLM, search, browser, market-data, messaging, or database credential.
- Define lifecycle behavior: startup dependencies, health checks, scheduled runs, retry policy, duplicate prevention, stale task cleanup, upgrade order, migration tests, and graceful shutdown.
- Prove recovery before adding more autonomy: revoke a provider key, stop a runaway task, restore memory from backup, rebuild the runtime from clean state, and verify that logs are useful without exposing secrets.
Scenario table
| Scenario | Recommended starting architecture | Why | Main risk |
|---|---|---|---|
| Team research and coding agent | DeerFlow-style runtime + isolated sandbox + scoped memory + authenticated gateway | Broad runtime capabilities fit long-horizon research and code tasks | Sandbox, shell, file-write and provider-key exposure |
| Persistent personal assistant | Hermes-style always-on agent + personal memory + channels + cron + remote execution backend | Cross-session learning and messaging presence are core requirements | Channel identity, autonomous skill growth and long-lived credentials |
| Shared memory for multiple agents | Agent runtimes + separate Cognee-style memory control plane | Memory lifecycle and tenant boundaries can evolve independently from runtimes | Namespace leakage, retention mistakes and model/embedding dependency drift |
| Quant research workspace | Vibe-Trading-style domain runtime + market data + backtest + bounded connectors + audit ledger | Specialized tools and deterministic evaluation matter more than generic chat | Financial loss, data quality, lookahead leakage and connector permissions |
| Daily market intelligence briefing | Daily Stock Analysis-style scheduled ETL/research pipeline + dashboard + notifications | Recurring report generation is a pipeline problem more than a general agent problem | Data drift, duplicated jobs, stale reports and over-trusting LLM conclusions |
| Small internal automation | One narrow agent service + minimal memory + explicit tools + scheduler | Avoid unnecessary multi-agent and gateway complexity | Overbuilding before ownership and recovery are clear |
| Multi-tenant SaaS agent | Gateway + tenant-isolated runtime jobs + isolated memory namespaces + centralized policy/audit | User separation must be designed across every layer | Cross-tenant data/tool leakage and shared-secret blast radius |
| High-risk privileged automation | Human-approved broker service around agent planning + disposable execution + immutable logs | Separate planning from irreversible execution | Agent prompt/tool compromise causing host or external side effects |
Architecture risks to review
Monolithic agent container
Putting runtime, browser, shell, memory database, provider keys, messaging credentials, and host mounts into one container creates a large compromise radius.
Global memory
A shared knowledge store without explicit user, agent, project, and retention boundaries can leak sensitive context across tasks or tenants.
Tool capability inflation
General shell, filesystem, browser, arbitrary fetch, database, broker, and container controls are convenient but turn prompt compromise into operational compromise.
Gateway equals runtime
A public or LAN-facing gateway should not automatically inherit all runtime privileges. Treat session admission and privileged execution as separate decisions.
Autonomous skill accumulation
Self-created or imported skills can become persistent executable policy. Track provenance, review changes, constrain tools, and support disable/rollback.
Scheduler without idempotency
Recurring agent jobs can duplicate notifications, reports, trades, writes, or API costs when retries and overlap are not designed.
Provider-key sprawl
A single .env shared across runtime, memory, search, browser, messaging, and domain connectors makes one component compromise affect the whole stack.
No clean-room recovery
Restoring the same mutable state after compromise can reintroduce malicious skills, poisoned memory, unsafe config, or leaked credentials.
Reference architecture patterns
Pattern A: General agent workspace
Authenticated gateway → agent runtime → per-task sandbox → narrow tools → scoped memory. Keep provider keys and host mounts outside the sandbox by default.
Pattern B: Personal always-on agent
Messaging gateway → personal session identity → agent runtime → user-scoped memory → cron jobs → remote/ephemeral execution backend. Separate notification credentials from privileged tool credentials.
Pattern C: Domain intelligence pipeline
Scheduler/event → deterministic data collection → agent analysis → evidence/artifact store → review gate → dashboard/notification. Preserve raw inputs and intermediate outputs for audit.
Memory as a service
Expose remember/recall/forget/improve-style operations or another narrow memory contract so runtime replacement does not require migrating every agent integration at once.
Brokered privileged tools
Place shell, browser, container, finance, or production actions behind a narrow broker that validates scope, identity, budget, approvals, and audit metadata.
Clean-room recovery bundle
Back up reviewed configuration, memory exports, domain data, schemas and audit logs separately from ephemeral runtime state, caches and generated skills awaiting review.
FAQ
Short answers for teams designing self-hosted agent infrastructure.
Do I need a multi-agent runtime?
Usually not for the first deployment. Use subagents or swarms when tasks are genuinely parallel or specialized and you can observe budgets, failures and output quality independently.
Should memory live inside the agent runtime?
Small personal deployments can keep memory integrated. Teams with multiple agents, tenants, retention requirements or audit needs should evaluate a separable memory layer so lifecycle and isolation can be managed independently.
Is Docker enough sandboxing for agent tools?
Containerization helps but the real answer depends on mounts, capabilities, network access, runtime sockets, secrets and host integration. Review the actual execution boundary and use disposable or remote environments for higher-risk tasks.
What is the difference between an agent runtime and a domain workflow?
A runtime provides general planning, tools, memory and execution mechanics. A domain workflow encodes specialized data sources, deterministic stages, evaluation rules, reports and controls for a specific job such as finance research.
What should be backed up?
Back up reviewed configuration, memory or knowledge exports, schemas, domain datasets, approved skills and audit evidence. Treat caches, temporary sandboxes and unreviewed generated state as disposable unless the product explicitly depends on them.
What is the first production control to add?
A reliable stop path. The team should be able to halt scheduled jobs and active tasks, revoke provider credentials and disable privileged tools before increasing autonomy.
Related radar
Related RepoDaily briefs
DeerFlow 2.0: ByteDance's Open-Source SuperAgent Harness for Research, Code, and Creationbytedance/deer-flow · Self-hosted appHermes Agent: The Self-Improving AI Agent That Learns From Every ConversationNousResearch/hermes-agent · Self-hosted appCognee: Self-Hosted AI Agent Memory with Tenant Isolation and Audit Trailstopoteretes/cognee · Security toolVibe-Trading: A Natural-Language Trading Agent with 450+ Quant Alphas, Swarm Orchestration, and Broker ConnectorsHKUDS/Vibe-Trading · Infrastructure / RuntimeDaily Stock Analysis: An LLM-Powered Multi-Market Stock Intelligence SystemZhuLinsen/daily_stock_analysis · Self-hosted app
Sources
- DeerFlow official repository
- DeerFlow backend architecture
- Hermes Agent official repository
- Hermes Agent configuration docs
- Cognee official repository
- Cognee core concepts
- Cognee integrations
- Vibe-Trading official repository
- Daily Stock Analysis official English README
- RepoDaily Local Service Security Checklist
Feedback
Did this page help you make a decision?
Anonymous feedback helps RepoDaily improve what is actually useful.