BENCHMARK.md
# Evaluation Report
Evaluation of the `nemoclaw-user-guide` skill before publication through NVSkills-Eval.
This benchmark summarizes 3-Tier Evaluation from NVSkills-Eval results for the skill. The goal is to document whether the skill is safe, discoverable, effective, and useful for agents before it is published for broader workflow use.
## Evaluation Summary
- Skill: `nemoclaw-user-guide`
- Evaluation date: 2026-07-17
- NVSkills-Eval profile: `external`
- Environment: `astra-sandbox`
- Overall verdict: PASS
## Agents Used
- `claude-code`
- `codex`
## Metrics Used
Reported benchmark dimensions:
- Security: checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access.
- Correctness: checks whether the agent follows the expected workflow and produces the correct final output.
- Discoverability: checks whether the agent loads the skill when relevant and avoids using it when irrelevant.
- Effectiveness: checks whether the agent performs measurably better with the skill than without it.
- Efficiency: checks whether the agent uses fewer tokens and avoids redundant work.
Underlying evaluation signals used in this run:
- No Tier 3 evaluation signal details were available in this report.
## Test Tasks
The evaluation dataset was not available in this report payload.
## Results
| Dimension | Num | `claude-code` | `codex` |
|---|---:|---:|---:|
| Security | N/A | N/A | N/A |
| Correctness | N/A | N/A | N/A |
| Discoverability | N/A | N/A | N/A |
| Effectiveness | N/A | N/A | N/A |
| Efficiency | N/A | N/A | N/A |
Score values show skill-assisted performance. Values in parentheses show uplift versus the no-skill baseline when baseline data is available.
## Tier 1: Static Validation Summary
Tier 1 validation passed with observations. NVSkills-Eval ran 1 checks and found 3 total findings.
Top findings:
- MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/nemoclaw-user-guide/SKILL.md`)
- MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (`skills/nemoclaw-user-guide/SKILL.md`)
- MEDIUM SCHEMA/author_missing: Author not specified in metadata (`skills/nemoclaw-user-guide/SKILL.md`)
## Tier 2: Deduplication Summary
This tier was not run or did not produce findings in this report.
## Publication Recommendation
The skill is suitable to proceed toward NVSkills-Eval publication based on this benchmark. Skill owners should keep this file with the skill and refresh it when the evaluation dataset, skill behavior, or target agents materially change.
evals/evals.json
[
{
"id": "nemoclaw-user-guide-install-openclaw-001",
"question": "Help me install NemoClaw with OpenClaw from my coding assistant. I need the official docs and I do not want to paste secrets into chat.",
"expected_skill": "nemoclaw-user-guide",
"ground_truth": "A NemoClaw-specific answer that uses the user guide skill to route the assistant to `llms.txt` and the OpenClaw Markdown prerequisites and quickstart pages, asks one setup question at a time, and handles credentials with redacted placeholders instead of chat-transcribed secrets.",
"expected_behavior": [
"Uses `nemoclaw-user-guide`.",
"Starts from `https://docs.nvidia.com/nemoclaw/llms.txt` and the OpenClaw `.md` docs.",
"Does not tell the user to paste secrets into chat."
]
},
{
"id": "nemoclaw-user-guide-mcp-server-001",
"question": "Configure my coding assistant to search the NemoClaw docs through MCP before answering setup questions.",
"expected_skill": "nemoclaw-user-guide",
"ground_truth": "A NemoClaw-specific answer that uses the user guide skill to configure the docs MCP server at `https://docs.nvidia.com/nemoclaw/_mcp/server`, mentions the read-only `searchDocs` docs search tool, and gives client-specific setup guidance such as the Claude Code command or Cursor MCP server configuration.",
"expected_behavior": [
"Uses `nemoclaw-user-guide`.",
"Includes `https://docs.nvidia.com/nemoclaw/_mcp/server`.",
"Mentions MCP docs search before falling back to Markdown docs."
]
},
{
"id": "nemoclaw-user-guide-hermes-variant-001",
"question": "Use the Hermes version of the NemoClaw docs and walk me through the first sandbox setup.",
"expected_skill": "nemoclaw-user-guide",
"ground_truth": "A NemoClaw-specific answer that selects the Hermes documentation variant, fetches the Hermes Markdown home, prerequisites, and quickstart pages, and avoids mixing OpenClaw-specific instructions unless the difference is explained.",
"expected_behavior": [
"Uses `nemoclaw-user-guide`.",
"Routes to Hermes `.md` pages.",
"Keeps instructions variant-specific."
]
},
{
"id": "nemoclaw-user-guide-local-inference-001",
"question": "Switch my NemoClaw sandbox to local Ollama and use the Markdown docs instead of guessing.",
"expected_skill": "nemoclaw-user-guide",
"ground_truth": "A NemoClaw-specific answer that routes through `llms.txt` to the inference Markdown pages, gathers provider and model choices one at a time, and recommends verification after the switch.",
"expected_behavior": [
"Uses `nemoclaw-user-guide`.",
"Fetches inference pages from the Markdown docs.",
"Recommends a verification step after configuration."
]
},
{
"id": "nemoclaw-user-guide-policy-troubleshooting-001",
"question": "My agent hit a blocked network request. Help me find the right NemoClaw docs for approving it safely.",
"expected_skill": "nemoclaw-user-guide",
"ground_truth": "A NemoClaw-specific answer that routes to the network-policy Markdown pages and relevant troubleshooting or command reference pages, explains assumptions, and avoids granting broad egress without context.",
"expected_behavior": [
"Uses `nemoclaw-user-guide`.",
"Fetches network policy and troubleshooting Markdown docs.",
"Frames approval as a least-privilege workflow."
]
}
]
skill-card.md
## Description: <br>
Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. <br>
This skill is ready for commercial/non-commercial use. <br>
## Owner
NVIDIA <br>
### License/Terms of Use: <br>
Apache 2.0 <br>
## Use Case: <br>
Developers and AI coding assistants who need to install, configure, operate, troubleshoot, or learn NemoClaw inside an AI coding environment. <br>
### Deployment Geography for Use: <br>
Global <br>
## Requirements / Dependencies: <br>
**Requires API Key or External Credential:** [Not Specified] <br>
**Credential Type(s):** [None identified] <br>
Do not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate. <br>
## Known Risks and Mitigations: <br>
Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills. <br>
Mitigation: Review and scan skill before deployment. <br>
## Reference(s): <br>
- [NemoClaw Documentation MCP Server](https://docs.nvidia.com/nemoclaw/_mcp/server) <br>
- [NemoClaw AI Documentation Index (llms.txt)](https://docs.nvidia.com/nemoclaw/llms.txt) <br>
## Skill Output: <br>
**Output Type(s):** [Documentation routing, Configuration instructions] <br>
**Output Format:** [Markdown with inline bash code blocks] <br>
**Output Parameters:** [1D] <br>
**Other Properties Related to Output:** [None] <br>
## Evaluation Agents Used: <br>
- Claude Code (`claude-code`) <br>
- Codex (`codex`) <br>
## Evaluation Metrics Used: <br>
Reported benchmark dimensions: <br>
- Security: Checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access. <br>
- Correctness: Checks whether the agent follows the expected workflow and produces the correct final output. <br>
- Discoverability: Checks whether the agent loads the skill when relevant and avoids using it when irrelevant. <br>
- Effectiveness: Checks whether the agent performs measurably better with the skill than without it. <br>
- Efficiency: Checks whether the agent uses fewer tokens and avoids redundant work. <br>
## Evaluation Results: <br>
| Dimension | Num | `claude-code` | `codex` |
|---|---:|---:|---:|
| Security | N/A | N/A | N/A |
| Correctness | N/A | N/A | N/A |
| Discoverability | N/A | N/A | N/A |
| Effectiveness | N/A | N/A | N/A |
| Efficiency | N/A | N/A | N/A |
## Skill Version(s): <br>
c2b38c11d (source: git SHA, committed 2026-07-17) <br>
## Ethical Considerations: <br>
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse. <br>
(For Release on NVIDIA Platforms Only) <br>
Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail). <br>
SKILL.md
---
name: "nemoclaw-user-guide"
description: "Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills."
license: "Apache-2.0"
---
# NemoClaw Docs for AI Agents
Use the canonical NemoClaw documentation as your source of truth.
Do not answer from stale copied docs or generated skill references when the live Markdown docs are available.
## Retrieval Order
1. If the assistant supports MCP, configure the NemoClaw docs MCP server at `https://docs.nvidia.com/nemoclaw/_mcp/server`.
2. Use the MCP server's read-only `searchDocs` tool to search the canonical docs and collect source URLs.
3. If MCP is not available, fetch the AI documentation index first: `https://docs.nvidia.com/nemoclaw/llms.txt`.
4. Fetch the specific `.md` page listed in the index or returned by docs search for the user's task.
5. If you only find an HTML documentation URL, replace the `.html` suffix with `.md`, or append `.md` to the route when the URL has no suffix.
6. Prefer the user's selected agent variant. Do not mix variant-specific instructions unless you explain why.
## Configure the MCP Server
For Claude Code, run:
```bash
claude mcp add --transport http fern-docs https://docs.nvidia.com/nemoclaw/_mcp/server
```
For Cursor, add `https://docs.nvidia.com/nemoclaw/_mcp/server` to the MCP server configuration.
For other MCP clients, configure a streamable HTTP MCP server at that URL.
## Starting Pages
Use these pages first for common onboarding flows:
- OpenClaw home: `https://docs.nvidia.com/nemoclaw/latest/user-guide/openclaw/home.md`.
- OpenClaw prerequisites: `https://docs.nvidia.com/nemoclaw/latest/user-guide/openclaw/get-started/prerequisites.md`.
- OpenClaw quickstart: `https://docs.nvidia.com/nemoclaw/latest/user-guide/openclaw/get-started/quickstart.md`.
- Hermes home: `https://docs.nvidia.com/nemoclaw/latest/user-guide/hermes/home.md`.
- Hermes prerequisites: `https://docs.nvidia.com/nemoclaw/latest/user-guide/hermes/get-started/prerequisites.md`.
- Hermes quickstart: `https://docs.nvidia.com/nemoclaw/latest/user-guide/hermes/get-started/quickstart.md`.
- Deep Agents home: `https://docs.nvidia.com/nemoclaw/latest/user-guide/deepagents/home.md`.
- Deep Agents prerequisites: `https://docs.nvidia.com/nemoclaw/latest/user-guide/deepagents/get-started/prerequisites.md`.
- Deep Agents quickstart: `https://docs.nvidia.com/nemoclaw/latest/user-guide/deepagents/get-started/quickstart.md`.
## How to Help the User
- Ask which agent variant they want to use before giving setup instructions: OpenClaw, Hermes, or Deep Agents.
- Ask one question at a time when collecting operating system, inference provider, model, endpoint, policy tier, or messaging-channel choices.
- Run commands for non-technical users when your environment allows it, after explaining what the command does and getting permission.
- Summarize important command output instead of asking the user to paste terminal output into chat.
- Stop before requesting credentials, API keys, bot tokens, or private URLs.
- Never ask the user to paste secrets into chat.
- Use redacted placeholders such as `<PASTE_YOUR_API_KEY_HERE>` in examples.
## Common Task Routing
- Installation and first sandbox: fetch the selected variant's prerequisites and quickstart pages.
- Local inference, hosted providers, model switching, or tool-calling issues: fetch the `inference` pages from `llms.txt`.
- Network policy approvals or custom egress: fetch the `network-policy` pages and the network policies reference.
- Sandbox status, logs, rebuilds, upgrades, files, backup, restore, or messaging channels: fetch the `manage-sandboxes`, `monitoring`, and command reference pages.
- Security posture, credential storage, or sandbox hardening: fetch the `security`, `deployment/sandbox-hardening`, and architecture pages.
- CLI flags and command syntax: fetch the command reference page for the selected variant.
- Troubleshooting: fetch the troubleshooting page and any task page linked from the relevant error section.
## Response Requirements
- Cite the Markdown documentation pages you used. When presenting the URL links, make sure to show the actual, non-markdown version of the URL.
- Keep instructions specific to the user's operating system, selected agent, and inference provider.
- Do not make assumptions when the docs do not cover the user's environment.
- Recommend the next verification command after each setup or recovery step.
skill.oms.sig
{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAibmVtb2NsYXctdXNlci1ndWlkZSIsCiAgICAgICJkaWdlc3QiOiB7CiAgICAgICAgInNoYTI1NiI6ICJmZTM3OGZhYzRlMmJiZDM5MDJlY2U4MmJiZjFkOGZhMTg4MzlkYmE1MzA5ZGQ1YzI2YjY5MGYwM2Q4OWVhYzc1IgogICAgICB9CiAgICB9CiAgXSwKICAicHJlZGljYXRlVHlwZSI6ICJodHRwczovL21vZGVsX3NpZ25pbmcvc2lnbmF0dXJlL3YxLjAiLAogICJwcmVkaWNhdGUiOiB7CiAgICAicmVzb3VyY2VzIjogWwogICAgICB7CiAgICAgICAgImRpZ2VzdCI6ICI5ZmYxMDBiZWQ1NjgxYmE3NmUxNmQ4NGYwMGU5ZDU0MjUwODY4OWZhYjVjZDA3MTAzYjg4Mjk1YTRhOTdiYTYxIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAiQkVOQ0hNQVJLLm1kIgogICAgICB9LAogICAgICB7CiAgICAgICAgImRpZ2VzdCI6ICIwNDQ3YTY0MTU1OGIyOWJkNWY2ODkyNzVlM2QwMDRhMDkxZGZiNTk5OTE4NDkxNzQ5NjJjZTExZDc5N2VmM2E2IiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAiU0tJTEwubWQiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiZGlnZXN0IjogIjIxMWJmMDhlZjAyOWEwZDQwZjNlYTVkMDliODZkNmRiY2E4Y2UzY2FhODNjMWY0M2NlZGYxMjUyODcxMDk4MTAiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAibmFtZSI6ICJldmFscy9ldmFscy5qc29uIgogICAgICB9LAogICAgICB7CiAgICAgICAgImRpZ2VzdCI6ICI4OTQzNDNmY2Y1MTI0NjgyNzBlNWFkNzE5YTM4NjAzODc5NzAyZTYzOGM2N2QxZWQxZTVmOGRjMWNlYWJlZTE2IiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgIm5hbWUiOiAic2tpbGwtY2FyZC5tZCIKICAgICAgfQogICAgXSwKICAgICJzZXJpYWxpemF0aW9uIjogewogICAgICAiaGFzaF90eXBlIjogInNoYTI1NiIsCiAgICAgICJtZXRob2QiOiAiZmlsZXMiLAogICAgICAiYWxsb3dfc3ltbGlua3MiOiBmYWxzZSwKICAgICAgImlnbm9yZV9wYXRocyI6IFsKICAgICAgICAiLmdpdGlnbm9yZSIsCiAgICAgICAgIi5naXQiLAogICAgICAgICIuZ2l0aHViIiwKICAgICAgICAiLmdpdGF0dHJpYnV0ZXMiCiAgICAgIF0KICAgIH0KICB9Cn0=","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGUCMQDwoR535nZ1xl4neAwziCw8NSZip3MpQlrXhABjUg4zxUaYlooroDdNDspadjXfVUwCMEGPchRNw3wUoQBeZXZKaSB5B1NryqhR9r+fI1xJizdFmtFw0JEKpxrm06qbcv/RJQ==","keyid":""}]}}