references/ax-profiles.md
# ax Profile Setup
Consult this when authentication fails (401, missing profile, missing API key). Do NOT run these checks proactively.
Use this when there is no profile, or a profile has incorrect settings (wrong API key, wrong region, etc.).
## 1. Inspect the current state
```bash
ax profiles show
```
Look at the output to understand what's configured:
- `API Key: (not set)` or missing → key needs to be created/updated
- No profile output or "No profiles found" → no profile exists yet
- Connected but getting `401 Unauthorized` → key is wrong or expired
- Connected but wrong endpoint/region → region needs to be updated
## 2. Fix a misconfigured profile
If a profile exists but one or more settings are wrong, patch only what's broken.
Profile region controls API and Flight routing for `ax` commands. Prefer `--region` over hard-coding SaaS hosts; examples include `us-central-1a`, `us-east-1b`, `eu-west-1a`, and `ca-central-1a`. It does not configure where a separately running app exports traces.
For EU and Canada, `--region eu-west-1a` and `--region ca-central-1a` automatically set `flight_host` to `flight.eu-west-1a.arize.com` or `flight.ca-central-1a.arize.com` and `flight_port` to `443`. Do not set those fields separately for standard SaaS profiles. Set them directly only for custom/private routing, using interactive profile setup or a complete `--from-file` profile; keep `flight_host` hostname-only (no `:443`) and set `flight_port` separately.
**Never pass a raw API key value as a flag.** Always reference it via the `ARIZE_API_KEY` environment variable. If the variable is not already set in the shell, instruct the user to set it first, then run the command:
```bash
# If ARIZE_API_KEY is already exported in the shell:
ax profiles update --api-key $ARIZE_API_KEY
# Fix the region (no secret involved — safe to run directly)
ax profiles update --region us-east-1b
# Fix both at once
ax profiles update --api-key $ARIZE_API_KEY --region us-east-1b
```
`update` only changes the fields you specify — all other settings are preserved. If no profile name is given, the active profile is updated.
## 3. Create a new profile
If no profile exists, or if the existing profile needs to point to a completely different setup (different org, different region):
**Always reference the key via `$ARIZE_API_KEY`, never inline a raw value.**
```bash
# Requires ARIZE_API_KEY to be exported in the shell first
ax profiles create --api-key $ARIZE_API_KEY
# Create with a region
ax profiles create --api-key $ARIZE_API_KEY --region us-east-1b
# Create a named profile
ax profiles create work --api-key $ARIZE_API_KEY --region us-east-1b
```
To use a named profile, switch to it first with `ax profiles use NAME` (see **Managing Multiple Profiles** below).
For browser-based OAuth instead of an API key, create the profile with `ax profiles create --auth-method oauth`, then sign in with `ax auth login`.
## 4. Getting the API key
**Never ask the user to paste their API key into the chat. Never log, echo, or display an API key value.**
If `ARIZE_API_KEY` is not already set, instruct the user to export it in their shell:
```bash
export ARIZE_API_KEY="..." # user pastes their key here in their own terminal
```
They can find their key at https://app.arize.com/admin > API Keys. Recommend they create a **scoped service key** (not a personal user key) — service keys are not tied to an individual account and are safer for programmatic use. Keys are space-scoped — make sure they copy the key for the correct space.
Once the user confirms the variable is set, proceed with `ax profiles create --api-key $ARIZE_API_KEY` or `ax profiles update --api-key $ARIZE_API_KEY` as described above.
## 5. Verify
After any create or update:
```bash
ax profiles show
# Or validate connectivity and configuration explicitly:
ax profiles validate
```
Confirm the API key and region are correct, then retry the original command.
## Managing Multiple Profiles
List all available profiles, with the active one marked:
```bash
ax profiles list
```
Switch to a different profile:
```bash
ax profiles use work
```
Delete a profile. You cannot delete the currently active profile; switch to another first if needed.
```bash
ax profiles delete staging --force
```
## Space
The CLI has no default space. Pass `-s`/`--space` (name or base64 ID) whenever a resource is given by **name**, or it fails with `project '...' not found. Provide 'space'`.
Resolve it once per session; first match wins:
1. The space the user named.
1. `ARIZE_SPACE_ID` (the tracing space; not a secret): `echo "$ARIZE_SPACE_ID"`, else `grep -E '^ARIZE_SPACE_ID=' .env` — read only that line.
1. `ax spaces list -o json`: use the only space, otherwise ask the user to pick a name.
Tell the user which space you used and why.
```bash
ax spans export my-project --space "$ARIZE_SPACE_ID"
```
## Save Credentials for Future Use
At the **end of the session**, if the user manually provided any credentials during this conversation **and** those values were NOT already loaded from a saved profile or environment variable, offer to save them.
**Skip this entirely if:**
- The API key was already loaded from an existing profile or `ARIZE_API_KEY` env var
- The user only used base64 project/space IDs and never needed to persist a space
**How to offer:** Use **AskQuestion**: *"Would you like to save your Arize credentials so you don't have to enter them next time?"* with options `"Yes, save them"` / `"No thanks"`.
**If the user says yes:**
1. **API key** — Run `ax profiles show` to check the current state. Then run `ax profiles create --api-key $ARIZE_API_KEY` or `ax profiles update --api-key $ARIZE_API_KEY` (the key must already be exported as an env var — never pass a raw key value).
references/REFERENCE.md
# Arize Admin — Enterprise Workflows & Troubleshooting
## Workflow 1: Onboard a New Team
```bash
# 1. Get the org ID
ax organizations get "Platform Team" -o json
# 2. Create a space for the team
ax spaces create --name "team-alpha" --organization-id ORG_ID
# 3. Create or reuse a custom role
ax roles create \
--name "Data Scientist" \
--permissions "PROJECT_READ,DATASET_CREATE,EXPERIMENT_CREATE" \
--description "Read traces, create datasets and run experiments"
# 4. Get stable role IDs (for SAML mappings or role bindings)
ax roles list --is-custom -o json
# 5. Invite team members (or look up existing users)
ax users create \
--full-name "Jane Doe" \
--email jane@example.com \
--role MEMBER \
--invite-mode EMAIL_LINK
# 6. Get the user's global ID
ax users get jane@example.com -o json
# 7. Add the user to the org
ax organizations add-user "Platform Team" --user-id USER_ID --role MEMBER
# 8. Add the user to the space
ax spaces add-user "team-alpha" --user-id USER_ID --role MEMBER
# 9. Create a service key for the team's CI/CD pipeline (scoped via --assignments)
ax api-keys create-service-key \
--name "team-alpha-service-key" \
--assignments '[{"org_id": "ORG_ID", "spaces": [{"space": "team-alpha"}]}]'
```
## Workflow 2: Configure SAML/SSO Role Mappings
SAML group-to-role mappings require stable role IDs. Retrieve them:
```bash
# List all custom roles with their IDs
ax roles list --is-custom -o json
# Get a specific role's ID and permissions
ax roles get "Data Scientist" -o json
```
Use the `id` field (base64 string like `Um9sZTo1...`) in your SAML `values.yaml` or IdP attribute mapping. These IDs are stable and do not change unless the role is deleted and recreated.
## Workflow 3: Restrict a Project to Specific Users
```bash
# 1. Find the project ID
ax projects get PROJECT_NAME --space "team-alpha" -o json
# 2. Restrict the project (excludes space-level roles)
ax resource-restrictions restrict --resource-id PROJECT_GLOBAL_ID
# 3. Find the user's global ID
ax users get jane@example.com -o json
# 4. Find the role ID to assign
ax roles list --is-custom -o json
# 5. Explicitly grant access to allowed users on that project
ax role-bindings create \
--user-id USER_GLOBAL_ID \
--role-id ROLE_GLOBAL_ID \
--resource-type PROJECT \
--resource-id PROJECT_GLOBAL_ID
```
## Workflow 4: Audit Access
```bash
# List all users and their status
ax users list -l 100 -o json
# List all custom roles with their permissions
ax roles list --is-custom -o json
# Inspect a specific role's permission set
ax roles get "Data Scientist" -o json
# List role bindings for a resource type (--resource-type is required)
ax role-bindings list --resource-type SPACE -l 100 -o json
ax role-bindings list --resource-type PROJECT -l 100 -o json
# List your own active API keys
ax api-keys list --status ACTIVE -o json
```
> **CLI audit limitations:** `ax api-keys list` only returns the authenticated user's own keys — it cannot enumerate other users' API keys. For a full access audit including other users' keys, use the Arize UI (Settings > API Keys).
## Workflow 5: Offboard a User
`ax users delete` cascades to all org memberships, space memberships, API keys, and role bindings in one operation.
**Always confirm with the user before running the delete.** Summarize the cascade impact, then ask for explicit confirmation.
```bash
# 1. Find the user's global ID
ax users get jane@example.com -o json
# 2. Confirm with the user: "Delete jane@example.com? This will cascade-remove
# all org/space memberships, API keys, and role bindings."
# 3. After user confirms, delete
ax users delete --id USER_ID --force
```
If you also need to deactivate the user in your IdP (to prevent SSO re-login), do that separately in Okta/Azure AD/etc. For real-time automated key invalidation on IdP deactivation, configure SCIM 2.0 provisioning.
## Workflow 6: Multi-Tenant Service Key Management
One service key per tenant space — scoped permissions, no org-admin required:
```bash
# Create a service key per tenant space (ORG_ID from `ax organizations list -o json`)
ax api-keys create-service-key --name "tenant-acme-key" --assignments '[{"org_id": "ORG_ID", "spaces": [{"space": "tenant-acme"}]}]'
ax api-keys create-service-key --name "tenant-beta-key" --assignments '[{"org_id": "ORG_ID", "spaces": [{"space": "tenant-beta"}]}]'
# Revoke a key (immediate invalidation)
ax api-keys list --key-type SERVICE -o json # find KEY_ID by name
ax api-keys revoke KEY_ID --force
# Rotate a key (same scope); --grace-period-seconds keeps the old key valid while clients switch over
ax api-keys refresh KEY_ID --grace-period-seconds 3600
```
Service keys can only write traces to their scoped space — they cannot access other spaces or perform admin operations.
---
## Troubleshooting
| Error | Cause | Fix |
|---|---|---|
| `403 Forbidden` | Profile lacks admin privileges | Authenticate with an admin API key (`ax profiles update --api-key $ARIZE_API_KEY`) |
| `401 Unauthorized` | Missing or invalid API key | See [ax-profiles.md](ax-profiles.md) |
| Role binding already exists | Idempotent — not an error | Safe to ignore; the existing binding is unchanged |
| User not found in space add-user | User not yet an org member | Run `ax organizations add-user` first, then `ax spaces add-user` |
| Role create fails with duplicate name | Name already in use | Use `ax roles list --is-custom` to find the existing role |
| Service key create fails | `--assignments` malformed or references a nonexistent org/space | Verify org ID with `ax organizations list -o json` and space with `ax spaces list`; `--assignments` is required for `create-service-key` |
| Key value not saved | Raw key was not captured at creation | Refresh the key: `ax api-keys refresh KEY_ID` |
| `No such command 'delete'` on `ax api-keys` | Keys are invalidated with `revoke` | `ax api-keys revoke KEY_ID --force` |
SKILL.md
---
name: arize-admin
description: "Manages Arize users, organizations, spaces, projects, roles, role bindings, resource restrictions, and API keys via the ax CLI. Use for enterprise admin workflows: inviting and offboarding users, onboarding new teams, creating custom roles for SAML/SSO mappings, assigning roles to users, restricting project-level access, and managing service keys for multi-tenant architectures. Covers ax users, ax organizations, ax spaces, ax projects, ax roles, ax role-bindings, and ax api-keys."
metadata:
author: arize
version: "1.0"
compatibility: Requires the ax CLI (≥ 0.39.0) and a configured Arize profile with org-admin privileges.
---
# Arize Admin Skill
Programmatic management of Arize users, organizations, spaces, roles, permissions, and API keys — the building blocks for enterprise access control.
> **Privilege requirement:** Most operations require **org-admin** or **account-admin** privileges. If commands return `403 Forbidden`, the authenticated profile lacks sufficient permissions.
> **Destructive-action rule:** Commands that delete, revoke, remove, or irreversibly modify resources (`delete`, `revoke`, `remove-user`, `unrestrict`) require **explicit user confirmation before execution**. When a user asks you to perform one of these operations:
> 1. Summarize exactly what will happen (e.g., "This will delete user jane@example.com and cascade-revoke their API keys and remove all their org/space memberships and role bindings.")
> 2. Ask the user to confirm (use `AskUserQuestion`).
> 3. Only after the user confirms, run the command with `--force` to skip the CLI's interactive prompt.
>
> Never run a `--force` destructive command without confirming with the user first.
## When to Use
- Invite users to the account, assign them to orgs and spaces
- Offboard a user and revoke all their access in one command
- Onboard a new team: create a space, create a custom role, assign users, generate a service key
- Create custom roles for SAML/SSO attribute mappings (need stable role IDs)
- Restrict a project so only explicitly bound users can access it
- Create scoped service keys for CI/CD pipelines or multi-tenant architectures
- Rotate or revoke API keys
- Create or delete projects within a space
## Upfront Questions
For multi-step workflows, **collect all required information before running any `ax` commands**. Use `AskUserQuestion` to avoid back-and-forth mid-workflow. Fetch live data first (e.g. org list) so you can present real options rather than asking the user to recall IDs.
### Onboarding a new team
1. Run `ax organizations get NAME -o json` when the org name is known; use `ax organizations list -o json` to present org names as options.
2. Use `AskUserQuestion` (single call, up to 4 questions) to gather:
- **Which org?** — present the org names from the list as options
- **Space name** — what to call the new team's space
- **Team members** — names and emails to invite (user can type via "Other"; ask if none yet)
- **Service key?** — whether to generate a service key for CI/CD pipelines
### Offboarding a user
Ask before running any commands:
- **Which user?** — email address (then look up with `ax users get EMAIL -o json`)
### Restricting a project
Ask before running any commands:
- **Which space and project?** — to look up the project global ID
- **Which users get explicit access?** — emails of users to bind to the restricted project
### Inviting users (standalone)
Ask before running any commands:
- **Name and email** — for each user to invite
- **Role** — `ADMIN`, `MEMBER`, or `ANNOTATOR` (present as options; account-level user creation has no `READ_ONLY` role)
- **Invite mode** — `EMAIL_LINK`, `TEMPORARY_PASSWORD`, or `NONE` (`--invite-mode` is required)
### Revoking or rotating an API key
Ask before running any commands:
- **Which key?** — run `ax api-keys list -o json` and present options by name and status; or ask for `KEY_ID`
- **Revoke or rotate?** — `revoke` invalidates immediately; `refresh` issues a new key with the same scope and invalidates the old one immediately unless `--grace-period-seconds` is passed
If the user says "delete" an API key, use `ax api-keys revoke` to invalidate it.
## Concepts
- **Organization** — a named grouping within an account (e.g. one per business unit). Spaces live inside organizations. Users are added to the account first, then to orgs, then to spaces.
- **Space** — a workspace that isolates traces, datasets, and projects. A user must be an org member before they can be added to a space within that org.
- **Role** — a named set of permissions. Predefined roles are system-managed. Custom roles are created by admins. The roles for org/space membership (`ADMIN`, `MEMBER`, `READ_ONLY`, `ANNOTATOR`) are separate from custom RBAC roles used with `ax role-bindings`.
- **Role binding** — fine-grained assignment of a custom role to a user on a specific resource (a space or a project).
- **Resource restriction** — marks a project so that only users with an explicit role binding on that project can access it. Roles bound at any higher hierarchy level (space, org, account) are excluded.
- **API key** — either a *user* key (authenticates as the creator, full user permissions) or a *service* key (scoped to a specific space, for automated pipelines).
## Prerequisites
Proceed directly — run the `ax` command you need. Do NOT check versions or profiles upfront.
If an `ax` command fails:
- `command not found` or version error → see [references/ax-setup.md](references/ax-setup.md)
- `401 Unauthorized` / missing API key → run `ax profiles show`; follow [references/ax-profiles.md](references/ax-profiles.md)
- `403 Forbidden` → the active profile lacks admin privileges; see [references/ax-profiles.md](references/ax-profiles.md) (never ask the user to paste an admin key into chat)
- **Security:** Never read `.env` files or search the filesystem for credentials. Exception: the non-secret `ARIZE_SPACE_ID` line (see [Space](references/ax-profiles.md#space)). Use `ax profiles` for Arize credentials. Never ask the user to paste secrets into chat. Never echo, log, or display raw API key values. For missing credentials, see [references/ax-profiles.md](references/ax-profiles.md).
> **OAuth login option:** Users can authenticate via browser-based OAuth PKCE instead of API keys by running `ax auth login` (then `ax auth logout` to revoke). The active profile must be an OAuth profile, created with `ax profiles create --auth-method oauth`. Inform users of this option if they ask about authentication alternatives — do **not** run `ax auth login` yourself, as it opens a browser interactively.
---
## Users
A user must exist in the account before they can be added to an org or space. **Account-level roles:** `ADMIN`, `MEMBER`, `ANNOTATOR`
```bash
ax users list # all users
ax users list --email "jane" # substring filter
ax users list --status ACTIVE # ACTIVE or INVITED
ax users list -l 100 -o json # paginate, get global IDs
ax users get USER_ID_OR_EMAIL # direct lookup; list is for browsing
ax users create \
--full-name "Jane Doe" \
--email jane@example.com \
--role MEMBER \
--invite-mode EMAIL_LINK # or: NONE | TEMPORARY_PASSWORD
ax users update USER_ID --full-name "Jane Smith"
ax users update USER_ID --is-developer # grant developer flag
ax users delete --id USER_ID --force # ⚠ confirm first — cascades: org/space memberships, API key revocation, role bindings
ax users delete --email jane@example.com --force # or resolve by email instead of ID
ax users resend-invitation USER_ID
ax users reset-password USER_ID
```
---
## Organizations
**Organization roles:** `ADMIN`, `MEMBER`, `READ_ONLY`, `ANNOTATOR`
```bash
ax organizations list
ax organizations list --name "platform"
ax organizations list -l 100 -o json
ax organizations get "Platform Team"
ax organizations create --name "Platform Team" --description "Core ML platform"
ax organizations update "Platform Team" --name "ML Platform" --description "Updated"
# Add user (must exist in account first)
ax organizations add-user "Platform Team" --user-id USER_ID --role MEMBER
# Remove user (also removes from all child spaces) — ⚠ confirm first
ax organizations remove-user "Platform Team" --user-id USER_ID --force
# Delete organization: ⚠ irreversible; deletes org and all resources inside (spaces, projects, experiments, datasets)
ax organizations delete "Platform Team" --force
```
---
## Spaces
**Space roles:** `ADMIN`, `MEMBER`, `READ_ONLY`, `ANNOTATOR`
```bash
ax spaces list
ax spaces list --organization-id ORG_ID
ax spaces get "my-workspace"
# --organization-id required; get ORG_ID from ax organizations list -o json
ax spaces create --name "team-alpha" --organization-id ORG_ID
ax spaces update "team-alpha" --name "team-alpha-v2"
ax spaces delete "team-alpha" --force # ⚠ confirm first — irreversible; deletes all resources
# User must be an org member before being added to a space
ax spaces add-user "team-alpha" --user-id USER_ID --role MEMBER
ax spaces remove-user "team-alpha" --user-id USER_ID --force # ⚠ confirm first
```
---
## Roles
Custom RBAC roles used with `ax role-bindings`. Separate from the simpler `ADMIN`/`MEMBER`/`READ_ONLY`/`ANNOTATOR` roles in org/space membership.
```bash
ax roles list # all roles
ax roles list --is-custom -o json # custom only — get stable IDs for SAML mappings
ax roles list --is-predefined
ax roles get "Data Scientist" # inspect permissions
# --permissions is comma-separated; fully replaces on update
ax roles create \
--name "Data Scientist" \
--permissions "PROJECT_READ,DATASET_CREATE,EXPERIMENT_CREATE" \
--description "Read traces, create datasets and experiments"
ax roles update "Data Scientist" --permissions "PROJECT_READ,DATASET_CREATE,EXPERIMENT_CREATE,EVALUATOR_CREATE"
ax roles delete "Data Scientist" --force # ⚠ confirm first — predefined roles cannot be deleted
```
**Finding available permissions:** Run `ax roles get <predefined-role> -o json` on a system role (e.g. `Member`, `Admin`) to see valid permission names.
---
## Role Bindings
Fine-grained assignment of a custom role to a user on a specific resource (space or project).
```bash
# Assign at space level
ax role-bindings create \
--user-id USER_GLOBAL_ID \
--role-id ROLE_GLOBAL_ID \
--resource-type SPACE \
--resource-id SPACE_GLOBAL_ID
# Assign at project level
ax role-bindings create \
--user-id USER_GLOBAL_ID \
--role-id ROLE_GLOBAL_ID \
--resource-type PROJECT \
--resource-id PROJECT_GLOBAL_ID
ax role-bindings get BINDING_ID
ax role-bindings update BINDING_ID --role-id NEW_ROLE_ID
ax role-bindings delete BINDING_ID --force # ⚠ confirm first
# List bindings for a resource type (--resource-type is required)
ax role-bindings list --resource-type SPACE
ax role-bindings list --resource-type PROJECT --user-id USER_GLOBAL_ID -o json
```
Idempotent — if a binding already exists for the user on that resource, exits without error.
---
## Resource Restrictions
Restricts a **project, dashboard, or prompt** so only users with an explicit role binding on that resource can access it. Space/org-level roles are excluded.
```bash
ax resource-restrictions list # all restrictions, paginated
ax resource-restrictions list --resource-type DASHBOARD # filter to one resource type (PROJECT, DASHBOARD, or PROMPT)
ax resource-restrictions list --limit 50 --cursor PAGINATION_CURSOR # fetch next page
ax resource-restrictions restrict --resource-id RESOURCE_GLOBAL_ID # idempotent
ax resource-restrictions unrestrict --resource-id RESOURCE_GLOBAL_ID --force # ⚠ confirm first
# Finding a project ID
ax projects get NAME --space "my-workspace" -o json
```
---
## API Keys
> **Scope:** `ax api-keys list` returns only keys owned by the **authenticated user**. For org-wide auditing, use the Arize UI (Settings > API Keys).
```bash
ax api-keys list
ax api-keys list --key-type SERVICE --status ACTIVE -o json
# User key — authenticates as creator, inherits their full permissions
ax api-keys create --name "CI pipeline" --expires-at "2027-01-01T00:00:00"
# Service key — bot user scoped to org(s)/space(s) via --assignments JSON
# (recommended for CI/CD pipelines; get ORG_ID from `ax organizations list -o json`)
ax api-keys create-service-key \
--name "team-alpha-traces" \
--assignments '[{"org_id": "ORG_ID", "spaces": [{"space": "team-alpha"}]}]' \
--expires-at "2027-01-01T00:00:00"
ax api-keys revoke KEY_ID --force # ⚠ confirm first — invalidates the key immediately
# Rotation — issues a new key with same scope; old key dies immediately
# unless --grace-period-seconds keeps it valid while clients switch over
ax api-keys refresh KEY_ID --grace-period-seconds 3600
ax api-keys refresh KEY_ID --expires-at "2028-01-01T00:00:00"
```
> **The raw key is displayed once.** Save it immediately in your secrets manager. It cannot be retrieved again.
Scope `create-service-key` entirely through `--assignments`; run `ax api-keys create-service-key --help` for the assignment JSON shape, role defaults, and `--account-role`. Pass `--env-file PATH` to write the new key to `ARIZE_API_KEY` in a dotenv or .envrc file, so the key lands in a file rather than the conversation.
---
## Projects
Projects live inside spaces and contain traces, datasets, and experiments.
```bash
ax projects list --space SPACE
ax projects list --space SPACE --name "playground" # substring filter
ax projects list --space SPACE -l 100 -o json # get base64 IDs
ax projects get NAME_OR_ID --space SPACE
ax projects create --name "my-project" --space SPACE
ax projects update NAME_OR_ID --name "new-project-name" --space SPACE
ax projects delete NAME_OR_ID --space SPACE --force # ⚠ confirm first — deletes all traces and datasets
```
> **Note:** Project IDs (base64 strings) are used by `ax spans export`, `ax traces export`, and `ax resource-restrictions`. If commands reject a project name, take the `id` field from `ax projects get NAME --space SPACE -o json`; `ax projects list` is for browsing.
---
## Enterprise Workflows & Troubleshooting
Step-by-step workflows (onboard a team, SAML/SSO mappings, project restriction, offboarding, multi-tenant keys) and a troubleshooting table are in [references/REFERENCE.md](references/REFERENCE.md).
---
## Related Skills
- **arize-instrumentation**: Set up tracing in an LLM app once a space is ready.
- **arize-trace**: Export and inspect traces within a managed space.
- **arize-dataset**: Create and manage datasets within a space.