Skip to content

Test Plan

How to verify pi-vault-mind works for agents, humans, and human-in-the-loop (HITL) workflows.

Three personas

Persona Role How they interact
Agent LLM / pi Discovers tools, calls them with correct parameters, reads promptGuidelines
Human User Types /wiki init, reviews output, runs commands
Human-in-the-loop Approver Confirm/reject via strict mode or /wiki approve batch review

Agent-only test (autopilot mode)

Goal: The agent captures facts without human interaction.

User: Record that users must authenticate via SSO.

Agent flow:

  1. describe_wiki → confirms schema fields
  2. append_wiki with mode="autopilot"
  3. Checks dedup → appends if unique
  4. Returns success confirmation

Verify:

query_wiki(collection="main", filters={tag: "auth"})

Returns the entry.

Human-only test (commands)

Goal: A human can set up and use the collection without the agent writing code.

# 1. Install
pi install npm:pi-vault-mind

# 2. Check health
/wiki validate

# 3. Scaffold
/wiki init

# 4. Check config created
cat pi-vault-mind.config.json

# 5. Index docs (if wiki collections exist)
/wiki reindex

# 6. Search
wiki_search(query="authentication strategy")

# 7. Check stats
wiki_stats

Expected: No tool calls needed; all interaction is via slash commands.

Human-in-the-loop test (strict mode)

Goal: Agent proposes entries; human approves each one.

User: Add a fact about SSO but I want to review it first.

Agent flow:

  1. append_wiki with mode="strict"
  2. ctx.ui.confirm fires → human sees a dialog
  3. Human clicks Approve → entry appended
  4. Or human clicks Reject → agent sees rejection, pivots

Verify dialog shows:

  • Full JSON of proposed entry
  • Collection name
  • "Approve entry for 'main'?"

Human-in-the-loop test (gated + approval batch)

Goal: Agent queues many facts; human reviews all at once.

User: Queue these 5 facts for later review.

Agent flow (5x):

  1. append_wiki with mode="gated" each time
  2. Facts go to collections/pending.jsonl

Human later types:

/wiki approve main

Verify:

  • Each pending entry shows in a confirm dialog
  • Approve → moved to collections/main.jsonl
  • Reject → stays in collections/pending.jsonl (or removed)
  • Final count message: "Approved 4 → 'main'. Rejected 1."

Context injection test

Goal: Typing a keyword triggers automatic context loading.

Setup: Ensure config has:

"injectors": [
  { "name": "draft-context", "regex": "draft\\s+(\\S+)", "collection": "main", "filterField": "tag" }
]

Add an entry:

append_wiki(collection="main", mode="autopilot", entry={id:"r1", domain:"auth", source:"s1", fact:"SSO required", tag:"login", artifact:""})

Test:

User: draft login

Expected: The before_agent_start event injects all entries where tag === "login" into the system prompt.

Verify by checking the LLM response references "SSO required".

Indexing test (wiki pipeline)

Goal: Documents are searchable after indexing.

# 1. Have markdown files in ./docs
ls docs/*.md

# 2. Add to wiki (via /wiki init or direct collection config)
# 3. Index (BM25 + embeddings)
/wiki reindex

# 4. Check status
wiki_status
# Expected: shows collection, file count, zero pending embeddings

# 5. Search
wiki_search(query="deployment strategy", limit=3)
# Expected: returns relevant markdown chunks

Export test

Goal: Data portability.

wiki_export(collection="main", format="csv")

Verify: Files are valid CSV/Markdown with schema headers.

Regression checklist

Run before each release:

Check How
Build pnpm build → zero errors
Check pnpm check → zero errors (biome + tsc)
Version cat package.json \| grep version matches planned release
Changelog CHANGELOG.md has entry for this version
Skills manifest skills/vault-mind/SKILL.md has valid frontmatter
No stale refs No wiki-ledger, qmd-init, /approve-ledger, or _ledger tool names in docs

Smoke test (one-liner)

pi -e npm:pi-vault-mind
/wiki init
/wiki validate

If this sequence completes without errors, the extension is healthy.