User Guide

Authentication

Pick the mode that matches where you run VIP and what credentials you have:

Interactive auth

Pass --interactive-auth to open a Chromium window where you can log in through any identity provider (Okta, SAML, OIDC, password). After login, VIP mints a temporary API key, saves the session, and runs all tests headlessly. No credentials need to be configured in advance. Because it opens a visible browser window, run it from a machine with a display, such as your workstation. On a host without one, VIP raises an error telling you to use --headless-auth instead.

vip verify --connect-url https://connect.example.com --interactive-auth

Connect behind an OIDC forward-auth gateway works with interactive auth. No extra configuration is needed.

Headless auth (no display)

If you need to run VIP directly on a headless Linux server (or any host without a display), use --headless-auth to automate the OIDC login in a headless browser. Set idp in your vip.toml to tell VIP which identity provider to automate. If MFA is enabled, VIP prompts for the verification code in the terminal.

vip verify --config vip.toml --headless-auth

For fully unattended runs against an MFA-protected service account, set VIP_TEST_TOTP_SECRET to the account's base32 TOTP seed. VIP generates the current code automatically when the IdP asks for it. The seed is equivalent to bypassing 2FA. Only use it with a dedicated test service account, never with a personal IdP account, and store it in the same secret store as VIP_TEST_PASSWORD.

API-key-only auth

If you only have an API key and no browser credentials, use --api-auth to run only the tests that can authenticate via API key. Tests that require a browser login (username/password or SSO) are automatically skipped:

export VIP_CONNECT_API_KEY="your-api-key"
vip verify --connect-url https://connect.example.com --api-auth

This is useful in CI pipelines or environments where interactive login is not possible and no test user credentials are available.

Credential-based auth

If you have API keys and test user credentials, disable interactive auth and set environment variables:

export VIP_CONNECT_API_KEY="your-api-key"
export VIP_TEST_USERNAME="test-user"
export VIP_TEST_PASSWORD="test-password"
vip verify --connect-url https://connect.example.com --no-interactive-auth

Environment variables

Running tests

Target specific products or test categories:

# Run all tests
vip verify --connect-url https://connect.example.com

# Run tests for a specific category
vip verify --connect-url https://connect.example.com --categories connect
vip verify --workbench-url https://workbench.example.com --categories workbench
vip verify --package-manager-url https://pm.example.com --categories package-manager

# Combine markers
vip verify --connect-url https://connect.example.com --categories "performance and connect"

# Filter by test name (shortcut for pytest -k)
vip verify --connect-url https://connect.example.com --filter login
vip verify --connect-url https://connect.example.com -f "login and not saml"

# Skip tests whose names match an expression
vip verify --connect-url https://connect.example.com --filter "not login"

# Pass extra flags to pytest
vip verify --connect-url https://connect.example.com -- -x --tb=long

Test categories

Troubleshooting

Add --verbose for detailed output, including full tracebacks, authentication progress, and IdP login steps. This is especially helpful when diagnosing headless auth or MFA issues.

vip verify --connect-url https://connect.example.com --headless-auth --verbose

Configuration file

For repeated testing or advanced configuration, use a vip.toml file instead of URL flags. The template vip.toml.example is not part of the installed package. Download the copy from the release that matches your installed version:

vip --version
curl -fsSL https://raw.githubusercontent.com/posit-dev/vip/vX.Y.Z/vip.toml.example -o vip.toml
# Edit vip.toml with your deployment details

Replace X.Y.Z with the version that vip --version prints.

Then run tests against it:

vip verify --config vip.toml --no-interactive-auth

If no --config is specified and no URL flags are given, VIP looks for vip.toml in the current directory or the path in the VIP_CONFIG environment variable.

Each product section (Connect, Workbench, Package Manager) can be enabled or disabled individually. The template lists every setting.

Running behind a proxy

On hosts that reach your Posit deployment through a forward proxy, VIP reads the standard proxy environment variables: HTTP_PROXY, HTTPS_PROXY, ALL_PROXY, and NO_PROXY (upper- or lowercase). They cover every outbound request VIP makes, including the browser login. No configuration is needed:

export HTTPS_PROXY=http://proxy.corp.example:8080
export NO_PROXY=localhost,.internal.example.com
vip verify --connect-url https://connect.example.com --headless-auth

To set the proxy explicitly instead, pass flags with your product URLs:

vip verify --connect-url https://connect.example.com \
  --proxy http://proxy.corp.example:8080 \
  --no-proxy localhost,.internal.example.com

The --proxy and --no-proxy flags only work when you pass product URLs directly. If the run loads a config file (with --config, or a ./vip.toml in the working directory), VIP warns that it ignored them. Set the proxy in the file instead:

[proxy]
url = "http://proxy.corp.example:8080"
no_proxy = ["localhost", ".internal.example.com"]

An explicit url (or --proxy) replaces the environment's NO_PROXY instead of merging with it, so repeat any bypass hosts under no_proxy. An explicit setting applies to the whole run, the same as the environment variables. vip status and vip cleanup also honor [proxy].

To send every request direct and ignore any proxy environment variables, set enabled = false under [proxy], or pass --no-proxy '' along with product-URL flags.

Gotchas

• A lone HTTP_PROXY is used for https too. If only HTTP_PROXY is set, VIP sends https traffic through it with a CONNECT tunnel. An explicit HTTPS_PROXY or ALL_PROXY always takes precedence, and is clearer if you want a specific https gateway.
• vip install honors environment variables only. It downloads the Playwright Chromium build and reads no config file. Export HTTPS_PROXY before you run it. A proxy set only under [proxy] in vip.toml does not reach it, and the install fails with an opaque network error.
• Raw TLS handshake checks cannot use a proxy. The SSL and HTTPS suites, and the listener probe that resolves URL schemes, open a direct TCP connection. They fail on a host with no direct route out, whatever you configure.
• Always give product URLs an explicit scheme when a proxy applies. For a URL without a scheme (such as --connect-url connect.example.com), VIP normally tries https:// first and falls back to http://. It decides this with a direct socket, which says nothing about the path through the proxy. So when a proxy applies to the host, VIP keeps https:// and reports the proxy as the likely problem, rather than sending credentials over plain http. If a product is served over http, write --connect-url http://connect.example.com, or list the host under no_proxy to reach it directly. This also applies when only HTTP_PROXY is set.

Generating reports

After running tests, generate and render the report:

# Run tests and save results
vip verify --connect-url https://connect.example.com --report report/results.json

# Render the report (requires quarto CLI)
vip report

# Publish to Connect (optional)
cd report && quarto publish connect --server https://connect.example.com

vip report reads results from report/results.json by default, or a custom path via --results. The Quarto CLI must be installed.

Extending VIP

Add site-specific tests without modifying the VIP source tree. The quickest start is vip scaffold, which writes a working extension you can edit:

vip scaffold --list
vip scaffold --template minimal --output ./my-tests

Two templates ship with VIP:

• minimal — a single-scenario health check against a product you already have configured. Start here.
• cross-product — R and Python runtime version checks plus package installability across Connect and Workbench, the pattern regulated environments tend to need. This is the default.

Every scaffolded directory also gets an AGENTS.md describing the extension contract: which markers drive auto-skip, how min_version gating works, and the fixtures, clients and shared Gherkin steps your tests can use. It is written for whoever authors the tests, whether that is you or a coding assistant you point at the directory.

Run the extension by pointing VIP at the directory:

vip verify --connect-url https://connect.example.com \
  --extensions ./my-tests

Or configure extension directories in vip.toml:

# vip.toml
[general]
extension_dirs = ["./my-tests"]

Extensions have full access to VIP's own fixtures, so your tests can reuse the authenticated clients rather than building their own. See the test architecture guide for the four-layer structure extensions follow.

Version gating

Tests can target specific product versions. Tests are skipped automatically if the deployment is older than required:

@pytest.mark.min_version(product="connect", version="2024.05.0")
def test_new_api_feature():
    ...

Installing a pinned version

uv tool install posit-vip resolves the newest versions each release allows. For a reproducible install pinned to a release's tested dependency set, pass that release's constraints file (attached to every GitHub release as constraints-<version>.txt):

uv tool install posit-vip \
  -c https://github.com/posit-dev/vip/releases/download/vX.Y.Z/constraints-X.Y.Z.txt

Replace X.Y.Z with the release you are installing (see the releases page).

Uninstalling

When you are done with VIP, run vip uninstall to preview what will be removed. It prints a dry-run plan without changing anything:

vip uninstall

Pass --yes to remove those artifacts. On Linux, VIP does not remove the system libraries it installed and instead it prints a sudo command for you to run. Complete the uninstall in this order:

vip uninstall --yes
# then run the sudo removal command that vip uninstall prints, if any
uv tool uninstall posit-vip