Getting Started

Get from zero to your first evaluation in under 5 minutes.

Prerequisites

  • Node.js 20+ — download
  • promptfoo — npm i -g promptfoo
  • Eqho API key — get one from your Eqho admin

Install

npm i -g eqho-eval

Verify the installation:

eqho-eval --version

Tip: eqho-evals (with an "s") also works — it's an alias for the common typo.

From source (contributors)

git clone https://github.com/Eqho-Solutions-Engineering/promptfoo-evals.git
cd promptfoo-evals
npm run setup    # install, build, link globally

Authenticate

Connect to the Eqho platform and backend proxy:

eqho-eval auth --key <your-eqho-api-key>

This registers your machine with the backend and stores a JWT locally. The JWT is used for all subsequent API calls — you never need to manage individual provider API keys.

Your credentials are stored in ~/.eqho-eval/config.json.

Interactive quickstart

The fastest way to get started:

eqho-eval start

This walks you through:

  1. Authentication — verifies or prompts for your API key
  2. Campaign selection — shows your active Eqho campaigns
  3. Project generation — scaffolds a complete eval project
  4. First eval — offers to run your evaluation immediately

Manual setup

If you prefer step-by-step control:

# 1. Authenticate
eqho-eval auth --key <your-api-key>

# 2. Scaffold a project from a campaign
eqho-eval init --campaign <campaign-id> -o ./my-eval

# 3. Navigate to the project
cd my-eval

# 4. Run the evaluation
eqho-eval eval

# 5. View results in the browser
eqho-eval view

CI / non-interactive

For automated environments:

export EQHO_API_KEY=your-key
eqho-eval start --yes --campaign <id>

The --yes flag accepts all defaults without prompting.

Check your setup

Run the diagnostic command to verify everything is configured correctly:

eqho-eval doctor

Example output:

  ✓ Node.js v22.22.0 (>=20 required)
  ✓ eqho-eval v0.5.3
  ✓ Eqho API key configured (abcd1234...)
  ✓ Eqho API reachable (15+ campaigns)
  ✓ Backend proxy connected (https://evals.eqho-solutions.dev)
  ✓ promptfoo installed via local (v0.120.25)
  ✗ Project config — no config found
    → eqho-eval init --campaign <id>

  7/8 checks passed

The doctor command also runs provider connectivity checks if a promptfooconfig.yaml exists in the current directory.

Generated project structure

After running eqho-eval init or eqho-eval start, you get:

my-eval/
├── promptfooconfig.yaml       # main config — edit tests here
├── prompts/
│   └── <agent-slug>.json      # assembled system prompt
├── tools/
│   └── <agent-slug>.json      # OpenAI tool definitions from Eqho actions
├── eqho.config.json           # campaign/agent IDs for sync
├── .env                       # proxy token + base URL (auto-generated)
├── tests/                     # custom test case files
└── output/                    # eval results (after running)
    ├── eval-results.json
    └── eval-report.html

The .env file contains:

OPENAI_API_KEY=eyJ...           # JWT token (not a real OpenAI key)
OPENAI_BASE_URL=https://evals.eqho-solutions.dev/api/v1

This routes all LLM calls through the backend proxy automatically.

Next steps