Getting Started

Quick start

Install VIP, then point it at your server:

uv pip install posit-vip
uv run playwright install chromium

Using RStudio or Positron?

You can set up VIP as an RStudio project instead. Go to File → New Project → Version Control → Git and enter https://github.com/posit-dev/vip.git as the Repository URL. Then install dependencies from the Terminal tab:

uv sync
uv run playwright install chromium

See the Shiny app page for the full graphical workflow.

Run your first test against a Connect server:

vip verify --connect-url https://connect.example.com --interactive-auth

The --interactive-auth flag opens a Chromium window so you can log in. After authentication completes, VIP runs the test suite headlessly and cleans up the session automatically.

Check product health without running the full test suite:

vip status

Test multiple products at once:

vip verify \
  --connect-url https://connect.example.com \
  --workbench-url https://workbench.example.com \
  --package-manager-url https://packagemanager.example.com \
  --interactive-auth

Authentication

Interactive auth

Pass --interactive-auth to open a Chromium window where you can log in through any identity provider (Okta, SAML, OIDC, password). After login, VIP mints a temporary API key, saves the session, and runs all tests headlessly. No credentials need to be configured in advance.

vip verify --connect-url https://connect.example.com --interactive-auth

Headless auth (no display)

On a headless Linux server, use --headless-auth to automate the OIDC login in a headless browser. Set idp in your vip.toml to tell VIP which identity provider to automate. If MFA is enabled, VIP prompts for the verification code in the terminal.

vip verify --config vip.toml --headless-auth

Credential-based auth

If you have API keys or test user credentials, disable interactive auth and set environment variables instead:

export VIP_CONNECT_API_KEY="your-api-key"
export VIP_TEST_USERNAME="test-user"
export VIP_TEST_PASSWORD="test-password"
vip verify --connect-url https://connect.example.com --no-interactive-auth

Running tests

Prefer a graphical interface? VIP includes a Shiny app that lets you select categories, run tests, and view the report from your browser or the Workbench Viewer pane.

Target specific products or test categories:

# Run all tests
vip verify --connect-url https://connect.example.com

# Run tests for a specific category
vip verify --connect-url https://connect.example.com --categories connect
vip verify --workbench-url https://workbench.example.com --categories workbench
vip verify --package-manager-url https://pm.example.com --categories package-manager

# Combine markers
vip verify --connect-url https://connect.example.com --categories "performance and connect"

# Pass extra flags to pytest
vip verify --connect-url https://connect.example.com -- -x --tb=long -k 'login'

Test categories

Version gating

Tests can target specific product versions. Tests are skipped automatically if the deployment is older than required:

@pytest.mark.min_version(product="connect", version="2024.05.0")
def test_new_api_feature():
    ...

Troubleshooting

Add --verbose for detailed output, including full tracebacks, authentication progress, and IdP login steps. This is especially helpful when diagnosing headless auth or MFA issues.

vip verify --connect-url https://connect.example.com --headless-auth --verbose

Configuration file

For repeated testing or advanced configuration, use a vip.toml file instead of URL flags:

cp vip.toml.example vip.toml
# Edit vip.toml with your deployment details

Then run tests against it:

vip verify --config vip.toml --no-interactive-auth

If no --config is specified and no URL flags are given, VIP looks for vip.toml in the current directory or the path in the VIP_CONFIG environment variable.

Each product section (Connect, Workbench, Package Manager) can be enabled or disabled individually. See vip.toml.example for the full template.

Generating reports

After running tests, generate and render the report:

# Run tests and save results
vip verify --connect-url https://connect.example.com --report report/results.json

# Render the report (requires quarto CLI)
vip report

# Or run Quarto directly
cd report && quarto render

# Publish to Connect (optional)
quarto publish connect --server https://connect.example.com

vip report is a convenience wrapper around quarto render. It reads results from report/results.json by default, or a custom path via --results. The Quarto CLI must be installed.

Extending VIP

Add site-specific tests without modifying the VIP source tree. Create a directory with .feature and .py files following the same conventions:

vip verify --connect-url https://connect.example.com \
  --extensions /opt/vip-custom-tests

Or configure extension directories in vip.toml:

# vip.toml
[general]
extension_dirs = ["/opt/vip-custom-tests"]

See examples/custom_tests/ in the repository for a working example.