Getting Started

Quick start

VIP verifies Posit Team across standalone deployments.

Install VIP to start:

uv tool install posit-vip
vip install

vip install installs Playwright's Chromium browser and the system libraries it needs. These can later be removed safely, along with other test artifacts, with vip uninstall.

Run your first VIP test against Posit Public Package Manager:

vip verify --package-manager-url https://p3m.dev --filter 'not test_product_does_not_expose_sensitive_headers and not test_prometheus_metrics'

The above command should finish in under a minute and show a successful run with some tests skipped.

Now it is time to run against one of your own servers. If you have Connect set up, you can run against it like this:

vip verify --connect-url https://connect.example.com --interactive-auth

The --interactive-auth flag opens a Chromium window so you can log in. After authentication completes, VIP runs the test suite headlessly and cleans up the session automatically.

If you have an API key, you can skip browser auth entirely with --api-auth:

export VIP_CONNECT_API_KEY="your-api-key"
vip verify --connect-url https://connect.example.com --api-auth

There are more authentication options, including headless auth with OIDC, explained below.

At this point, you may see some tests failing and want to fix what isn't right and then try again. You can rerun just those tests with the --filter flag:

vip verify --connect-url https://connect.example.com --interactive-auth --filter test_deploy_shiny

As you get comfortable with how one server is running, you can move on to others using not just --connect-url but also --workbench-url and --package-manager-url. You can also see what other options are available by reading below or reading the command-line help:

vip --help

Authentication

Interactive auth

Pass --interactive-auth to open a Chromium window where you can log in through any identity provider (Okta, SAML, OIDC, password). After login, VIP mints a temporary API key, saves the session, and runs all tests headlessly. No credentials need to be configured in advance.

vip verify --connect-url https://connect.example.com --interactive-auth

VIP also supports Connect deployments that sit behind an OIDC/SSO forward-auth gateway (for example, an Okta proxy that intercepts /__api__/ requests and redirects unauthenticated traffic to the IdP). The browser session cookies captured during interactive auth are automatically forwarded alongside the API key on every httpx API call, so the gateway passes the request through instead of redirecting to the IdP login. Security checks that intentionally send unauthenticated requests accept a cross-host redirect (gateway intercept) as an equivalent "access denied" outcome.

Headless auth (no display)

On a headless Linux server, use --headless-auth to automate the OIDC login in a headless browser. Set idp in your vip.toml to tell VIP which identity provider to automate. If MFA is enabled, VIP prompts for the verification code in the terminal.

For fully unattended runs against an MFA-protected service account, set VIP_TEST_TOTP_SECRET to the account's base32 TOTP seed. VIP generates the current code automatically when the IdP asks for it. The seed is equivalent to bypassing 2FA — only use it with a dedicated test service account, never with a personal IdP account, and store it in the same secret store as VIP_TEST_PASSWORD.

vip verify --config vip.toml --headless-auth

API-key-only auth

If you only have an API key and no browser credentials, use --api-auth to run only the tests that can authenticate via API key. Tests that require a browser login (username/password or SSO) are automatically skipped:

export VIP_CONNECT_API_KEY="your-api-key"
vip verify --connect-url https://connect.example.com --api-auth

This is useful in CI pipelines or environments where interactive login is not possible and no test user credentials are available.

Credential-based auth

If you have API keys and test user credentials, disable interactive auth and set environment variables:

export VIP_CONNECT_API_KEY="your-api-key"
export VIP_TEST_USERNAME="test-user"
export VIP_TEST_PASSWORD="test-password"
vip verify --connect-url https://connect.example.com --no-interactive-auth

Running tests

Target specific products or test categories:

# Run all tests
vip verify --connect-url https://connect.example.com

# Run tests for a specific category
vip verify --connect-url https://connect.example.com --categories connect
vip verify --workbench-url https://workbench.example.com --categories workbench
vip verify --package-manager-url https://pm.example.com --categories package-manager

# Combine markers
vip verify --connect-url https://connect.example.com --categories "performance and connect"

# Filter by test name (shortcut for pytest -k)
vip verify --connect-url https://connect.example.com --filter login
vip verify --connect-url https://connect.example.com -f "login and not saml"

# Skip tests whose names match an expression
vip verify --connect-url https://connect.example.com --filter "not login"

# Pass extra flags to pytest
vip verify --connect-url https://connect.example.com -- -x --tb=long

Test categories

Version gating

Tests can target specific product versions. Tests are skipped automatically if the deployment is older than required:

@pytest.mark.min_version(product="connect", version="2024.05.0")
def test_new_api_feature():
    ...

Troubleshooting

Add --verbose for detailed output, including full tracebacks, authentication progress, and IdP login steps. This is especially helpful when diagnosing headless auth or MFA issues.

vip verify --connect-url https://connect.example.com --headless-auth --verbose

Configuration file

For repeated testing or advanced configuration, use a vip.toml file instead of URL flags:

cp vip.toml.example vip.toml
# Edit vip.toml with your deployment details

Then run tests against it:

vip verify --config vip.toml --no-interactive-auth

If no --config is specified and no URL flags are given, VIP looks for vip.toml in the current directory or the path in the VIP_CONFIG environment variable.

Each product section (Connect, Workbench, Package Manager) can be enabled or disabled individually. See vip.toml.example for the full template.

Generating reports

After running tests, generate and render the report:

# Run tests and save results
vip verify --connect-url https://connect.example.com --report report/results.json

# Render the report (requires quarto CLI)
vip report

# Or run Quarto directly
cd report && quarto render

# Publish to Connect (optional)
quarto publish connect --server https://connect.example.com

vip report is a convenience wrapper around quarto render. It reads results from report/results.json by default, or a custom path via --results. The Quarto CLI must be installed.

Extending VIP

Add site-specific tests without modifying the VIP source tree. Create a directory with .feature and .py files following the same conventions:

vip verify --connect-url https://connect.example.com \
  --extensions /opt/vip-custom-tests

Or configure extension directories in vip.toml:

# vip.toml
[general]
extension_dirs = ["/opt/vip-custom-tests"]

See examples/custom_tests/ in the repository for a working example.

Uninstalling

When you are done with VIP, run vip uninstall to preview what will be removed. It prints a dry-run plan without changing anything:

vip uninstall

Pass --yes to remove those artifacts. On Linux, VIP does not remove the system libraries it installed and instead it prints a sudo command for you to run. Complete the uninstall in this order:

vip uninstall --yes
# then run the sudo removal command that vip uninstall prints, if any
uv tool uninstall posit-vip