Getting Started

Quick start

VIP verifies Posit Team across standalone deployments.

Install VIP to start:

uv tool install posit-vip
vip install

vip install installs Playwright's Chromium browser and the system libraries it needs. These can later be removed safely, along with other test artifacts, with vip uninstall.

Run VIP from your own workstation or a CI runner, not the Connect, Workbench, or Package Manager servers themselves. VIP checks the deployment the way a real client would — over the same URL, proxy, and TLS termination your users go through — and --interactive-auth opens a visible browser window, which needs a display those servers typically don't have. If you must run VIP directly on a headless server, use --headless-auth or --api-auth instead (see Authentication below).

uv tool install posit-vip resolves the newest versions each release allows. For a reproducible install pinned to a release's tested dependency set, pass that release's constraints file (attached to every GitHub release as constraints-<version>.txt):

uv tool install posit-vip \
  -c https://github.com/posit-dev/vip/releases/download/vX.Y.Z/constraints-X.Y.Z.txt

Replace X.Y.Z with the release you are installing (see the releases page).

Run your first VIP test against Posit Public Package Manager:

vip verify --package-manager-url https://p3m.dev --filter 'not test_product_does_not_expose_sensitive_headers and not test_prometheus_metrics'

The above command should finish in under a minute and show a successful run with some tests skipped.

Now it is time to run against one of your own servers. If you have Connect set up, you can run against it like this:

vip verify --connect-url https://connect.example.com --interactive-auth

The --interactive-auth flag opens a Chromium window so you can log in. After authentication completes, VIP runs the test suite headlessly and cleans up the session automatically.

If you have an API key, you can skip browser auth entirely with --api-auth:

export VIP_CONNECT_API_KEY="your-api-key"
vip verify --connect-url https://connect.example.com --api-auth

There are more authentication options, including headless auth with OIDC, explained below.

At this point, you may see some tests failing and want to fix what isn't right and then try again. You can rerun just those tests with the --filter flag:

vip verify --connect-url https://connect.example.com --interactive-auth --filter test_deploy_shiny

As you get comfortable with how one server is running, you can move on to others using not just --connect-url but also --workbench-url and --package-manager-url. You can also see what other options are available by reading below or reading the command-line help:

vip --help

Authentication

Interactive auth

Pass --interactive-auth to open a Chromium window where you can log in through any identity provider (Okta, SAML, OIDC, password). After login, VIP mints a temporary API key, saves the session, and runs all tests headlessly. No credentials need to be configured in advance. Because it opens a visible browser window, run it from a machine with a display — your workstation, not a headless product server — or VIP raises an error telling you to use --headless-auth instead.

vip verify --connect-url https://connect.example.com --interactive-auth

VIP also supports Connect deployments that sit behind an OIDC/SSO forward-auth gateway (for example, an Okta proxy that intercepts /__api__/ requests and redirects unauthenticated traffic to the IdP). The browser session cookies captured during interactive auth are automatically forwarded alongside the API key on every httpx API call, so the gateway passes the request through instead of redirecting to the IdP login. Security checks that intentionally send unauthenticated requests accept a cross-host redirect (gateway intercept) as an equivalent "access denied" outcome.

Headless auth (no display)

If you need to run VIP directly on a headless Linux server (or any host without a display), use --headless-auth to automate the OIDC login in a headless browser instead of --interactive-auth. Set idp in your vip.toml to tell VIP which identity provider to automate. If MFA is enabled, VIP prompts for the verification code in the terminal.

For fully unattended runs against an MFA-protected service account, set VIP_TEST_TOTP_SECRET to the account's base32 TOTP seed. VIP generates the current code automatically when the IdP asks for it. The seed is equivalent to bypassing 2FA — only use it with a dedicated test service account, never with a personal IdP account, and store it in the same secret store as VIP_TEST_PASSWORD.

vip verify --config vip.toml --headless-auth

API-key-only auth

If you only have an API key and no browser credentials, use --api-auth to run only the tests that can authenticate via API key. Tests that require a browser login (username/password or SSO) are automatically skipped:

export VIP_CONNECT_API_KEY="your-api-key"
vip verify --connect-url https://connect.example.com --api-auth

This is useful in CI pipelines or environments where interactive login is not possible and no test user credentials are available.

Credential-based auth

If you have API keys and test user credentials, disable interactive auth and set environment variables:

export VIP_CONNECT_API_KEY="your-api-key"
export VIP_TEST_USERNAME="test-user"
export VIP_TEST_PASSWORD="test-password"
vip verify --connect-url https://connect.example.com --no-interactive-auth

Running tests

Target specific products or test categories:

# Run all tests
vip verify --connect-url https://connect.example.com

# Run tests for a specific category
vip verify --connect-url https://connect.example.com --categories connect
vip verify --workbench-url https://workbench.example.com --categories workbench
vip verify --package-manager-url https://pm.example.com --categories package-manager

# Combine markers
vip verify --connect-url https://connect.example.com --categories "performance and connect"

# Filter by test name (shortcut for pytest -k)
vip verify --connect-url https://connect.example.com --filter login
vip verify --connect-url https://connect.example.com -f "login and not saml"

# Skip tests whose names match an expression
vip verify --connect-url https://connect.example.com --filter "not login"

# Pass extra flags to pytest
vip verify --connect-url https://connect.example.com -- -x --tb=long

Test categories

Version gating

Tests can target specific product versions. Tests are skipped automatically if the deployment is older than required:

@pytest.mark.min_version(product="connect", version="2024.05.0")
def test_new_api_feature():
    ...

Troubleshooting

Add --verbose for detailed output, including full tracebacks, authentication progress, and IdP login steps. This is especially helpful when diagnosing headless auth or MFA issues.

vip verify --connect-url https://connect.example.com --headless-auth --verbose

Configuration file

For repeated testing or advanced configuration, use a vip.toml file instead of URL flags:

cp vip.toml.example vip.toml
# Edit vip.toml with your deployment details

Then run tests against it:

vip verify --config vip.toml --no-interactive-auth

If no --config is specified and no URL flags are given, VIP looks for vip.toml in the current directory or the path in the VIP_CONFIG environment variable.

Each product section (Connect, Workbench, Package Manager) can be enabled or disabled individually. See vip.toml.example for the full template.

Running behind a proxy

On hosts that reach your Posit deployment through a forward proxy, VIP routes its core outbound HTTP(S) through that proxy — the product API clients (Connect, Workbench, Package Manager), the authentication and URL-probe requests, and the Playwright browser login all take the same path. This matters because httpx disables environment-proxy resolution on a client that uses a custom transport (which VIP's API clients require), so without this handling the API calls would silently bypass the proxy while the browser and probe paths used it — and the Connect API-key flow would fail behind a proxy.

By default VIP reads the ambient environment — HTTP_PROXY / HTTPS_PROXY / ALL_PROXY / NO_PROXY (upper- or lowercase) — exactly as httpx does. The environment variables cover every outbound HTTP request VIP makes. No configuration is needed for the common case:

export HTTPS_PROXY=http://proxy.corp.example:8080
export NO_PROXY=localhost,.internal.example.com
vip verify --connect-url https://connect.example.com --headless-auth

If your site uses a single forward proxy as its only outbound tunnel and sets just HTTP_PROXY (no HTTPS_PROXY), VIP routes its https traffic through that proxy too (via the standard CONNECT tunnel), rather than sending it direct the way httpx would. An explicit HTTPS_PROXY or ALL_PROXY, if set, always takes precedence.

Lone HTTP_PROXY. When only HTTP_PROXY is set, VIP promotes it to carry https too (via CONNECT). On a vip verify run that promotion reaches every path, including the environment-honoring reachability and performance probes, because VIP reconciles the test subprocess's environment (see “Scope of the explicit config” below). Naming HTTPS_PROXY / ALL_PROXY explicitly is still clearer if you want a specific https gateway.

Setup. vip install downloads the Playwright Chromium build, and it reads no configuration file — so that download honors the proxy environment variables only. Export HTTPS_PROXY before running it; a proxy configured solely under [proxy] in vip.toml cannot reach it, and the install fails with an opaque network error.

To set the proxy explicitly instead of via the environment:

vip verify --connect-url https://connect.example.com \
  --proxy http://proxy.corp.example:8080 \
  --no-proxy localhost,.internal.example.com

The flags only apply on a run like the one above, where you pass product URLs directly — they are written into the configuration VIP generates for that run. On a run that loads a config file, whether via --config or a ./vip.toml found in the working directory, there is nothing for them to write into and VIP warns that it ignored them. Configure the proxy in the file instead:

[proxy]
url = "http://proxy.corp.example:8080"
no_proxy = ["localhost", ".internal.example.com"]

Note that an explicit url (or --proxy) replaces the environment's NO_PROXY rather than merging with it, so repeat any bypass hosts under no_proxy.

To force every request direct and ignore any proxy environment variables, disable proxying: enabled = false under [proxy], or vip verify --no-proxy '' alongside product-URL flags.

Scope of the explicit config. On a vip verify run VIP reconciles the test subprocess's environment with your [proxy] / --proxy setting, so it applies uniformly — the product API clients, authentication, URL probes, the browser login, and the environment-honoring paths in the suites (some reachability probes and the performance/load engine). An enabled = false (or --no-proxy '') likewise forces the whole run direct. The one exception is the raw-socket TLS handshake checks (the SSL and HTTPS suites, and the listener probe behind scheme resolution): they open a TCP connection directly to test the plaintext/handshake boundary, so they have no proxy to speak through and will fail on a host with no direct route out, whatever you configure. (vip status and vip cleanup run in-process and honor the [proxy] config directly.)

Always give product URLs an explicit scheme when a proxy is in effect. For a scheme-less URL (e.g. --connect-url connect.example.com) VIP probes https:// first and falls back to http:// if nothing answers — but it decides that using a direct socket, which says nothing about a path that goes through the proxy. So whenever a proxy applies to the host, VIP keeps https:// and reports the proxy as the likely problem rather than silently rewriting your URL to plaintext and sending credentials in the clear.

The practical consequence: a product genuinely served over plain http will not be found from a scheme-less URL on a host with a proxy configured. Write --connect-url http://connect.example.com (an explicit scheme is never probed or rewritten), or list the host under no_proxy / NO_PROXY if it should be reached directly. Note that this applies when only HTTP_PROXY is set too, since VIP routes https through it as described above.

Generating reports

After running tests, generate and render the report:

# Run tests and save results
vip verify --connect-url https://connect.example.com --report report/results.json

# Render the report (requires quarto CLI)
vip report

# Or run Quarto directly
cd report && quarto render

# Publish to Connect (optional)
quarto publish connect --server https://connect.example.com

vip report is a convenience wrapper around quarto render. It reads results from report/results.json by default, or a custom path via --results. The Quarto CLI must be installed.

Extending VIP

Add site-specific tests without modifying the VIP source tree. Create a directory with .feature and .py files following the same conventions:

vip verify --connect-url https://connect.example.com \
  --extensions /opt/vip-custom-tests

Or configure extension directories in vip.toml:

# vip.toml
[general]
extension_dirs = ["/opt/vip-custom-tests"]

See examples/custom_tests/ in the repository for a working example.

Uninstalling

When you are done with VIP, run vip uninstall to preview what will be removed. It prints a dry-run plan without changing anything:

vip uninstall

Pass --yes to remove those artifacts. On Linux, VIP does not remove the system libraries it installed and instead it prints a sudo command for you to run. Complete the uninstall in this order:

vip uninstall --yes
# then run the sudo removal command that vip uninstall prints, if any
uv tool uninstall posit-vip