Vip Tests
26 tests — 18 passed, 2 failed, 6 skipped
PASSUser can sign out of WorkbenchWorkbench
As a Posit Team administrator
Test procedure
- Given Workbench is accessible and I am logged in
- When I sign out of Workbench
- Then I am redirected to the Workbench login page
PASSWorkbench server is reachablePrerequisites
As a Posit Team administrator
Test procedure
- Given <product> is configured in vip.toml
- When I request the <product> health endpoint
- Then the server responds with a successful status code
PASSPackage Manager server is reachablePrerequisites
As a Posit Team administrator
Test procedure
- Given <product> is configured in vip.toml
- When I request the <product> health endpoint
- Then the server responds with a successful status code
PASSWorkbench server is reachable but not configuredPrerequisites
As a VIP developer
Test procedure
- Given Workbench is expected to be configured
- When I check the Workbench configuration
- Then Workbench should be reachable
PASSConnect version matches configurationPrerequisites
As a Posit Team administrator
Test procedure
- Given Connect is configured in vip.toml with a version expectation
- When I fetch the Connect server version
- Then the Connect version matches the configured value
SKIPWorkbench version matches configurationPrerequisites
As a Posit Team administrator
Test procedure
- Given Workbench is configured in vip.toml with a version expectation
- When I fetch the Workbench server version
- Then the Workbench version matches the configured value
PASSPackage Manager version matches configurationPrerequisites
As a Posit Team administrator
Test procedure
- Given Package Manager is configured in vip.toml with a version expectation
- When I fetch the Package Manager server version
- Then the Package Manager version matches the configured value
FAILUser can log in to Workbench via the web UIWorkbench
As a Posit Team administrator
Test procedure
- Given Workbench is accessible at the configured URL
- When a user navigates to the Workbench login page and enters valid credentials
- Then the Workbench homepage is displayed
- And the current user element is visible and non-empty in the header
test_workbench_login[chromium]: Login failed: Error: Temporary server error, please try again
Full error traceback
fixturefunc = <function navigate_and_login at 0x7f04b2228300>
request = <FixtureRequest for <Function test_workbench_login[chromium]>>
kwargs = {'page': <Page url='http://localhost:8787/auth-sign-in?appUri=%2F&error=2'>, 'workbench_url': 'http://localhost:8787', 'test_username': 'testuser', 'test_password': 'iq707mQegWCgJc4p'}
def call_fixture_func(
fixturefunc: _FixtureFunc[FixtureValue], request: FixtureRequest, kwargs
) -> FixtureValue:
if inspect.isgeneratorfunction(fixturefunc):
fixturefunc = cast(Callable[..., Generator[FixtureValue]], fixturefunc)
generator = fixturefunc(**kwargs)
try:
fixture_result = next(generator)
except StopIteration:
raise ValueError(f"{request.fixturename} did not yield a value") from None
finalizer = functools.partial(_teardown_yield_fixture, fixturefunc, generator)
request.addfinalizer(finalizer)
else:
fixturefunc = cast(Callable[..., FixtureValue], fixturefunc)
> fixture_result = fixturefunc(**kwargs)
^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.14/site-packages/_pytest/fixtures.py:1005:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/vip_tests/workbench/test_auth.py:82: in navigate_and_login
workbench_login(page, workbench_url, test_username, test_password)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
page = <Page url='http://localhost:8787/auth-sign-in?appUri=%2F&error=2'>
workbench_url = 'http://localhost:8787', username = 'testuser'
password = 'iq707mQegWCgJc4p', auth_provider = 'password'
interactive_auth = False, auth_mode = 'none', workbench_auth_error = None
max_retries = 3, retry_delay = 2.0
def workbench_login(
page: Page,
workbench_url: str,
username: str,
password: str,
auth_provider: str = "password",
interactive_auth: bool = False,
*,
auth_mode: str = "none",
workbench_auth_error: str | None = None,
max_retries: int = 3,
retry_delay: float = 2.0,
) -> None:
"""Navigate to Workbench homepage, logging in only if required.
This function:
- Navigates directly to Workbench's URL
- Handles OIDC/SSO via pre-loaded storage state (--interactive-auth / --headless-auth)
- Only fills login form for password auth
- Retries on transient server errors (e.g., too many logins)
Args:
page: Playwright page object
workbench_url: Base URL for Workbench (e.g., http://localhost:8787)
username: Login username
password: Login password
auth_provider: Auth type (e.g., "password", "oidc", "saml")
interactive_auth: True when an auth session is pre-loaded (either
--interactive-auth or --headless-auth)
auth_mode: Active auth mode ("interactive", "headless", or "none"),
used to name the responsible CLI flag in skip messages
workbench_auth_error: Reason the pre-test auth flow could not
establish a Workbench session, if known. Quoted in the skip
message so users see the real cause instead of a guess.
max_retries: Max login attempts on transient errors (default 3)
retry_delay: Seconds to wait between retries (default 2.0)
Raises:
pytest.skip: For non-password auth without a pre-loaded auth session,
or when the session's storage state doesn't cover Workbench
AssertionError: When password login fails after retries
"""
homepage_logo = page.locator(Homepage.POSIT_LOGO)
# For non-password auth without a pre-loaded auth session, skip immediately
if auth_provider != "password" and not interactive_auth:
pytest.skip(
f"Login form not available for auth provider {auth_provider!r}. "
"Pass --interactive-auth or --headless-auth to pre-load browser storage state."
)
page.goto(workbench_url)
page.wait_for_load_state("load")
# Fast path: already logged in (common with interactive_auth)?
if homepage_logo.is_visible():
return
# Check if we landed on a login/IdP page
if _on_login_page(page.url):
# The sign-in page renders client-side after ``load``; wait once for
# either the password form's username field or an OIDC "Sign in with ..."
# button to appear before deciding which flow applies. A short fixed wait
# on only the SSO button races the render and can misread a slow OIDC
# sign-in page (e.g. an ``?error=2`` bounce) as a password deployment --
# which then fails the retry loop with "Login failed after 3 attempts"
# instead of skipping. Waiting for either control settles that race
# without penalising real password deployments (the username field
# appears promptly there).
try:
page.locator(f"{LoginPage.USERNAME}, button:has-text('Sign in')").first.wait_for(
state="visible", timeout=TIMEOUT_PAGE_LOAD
)
except Exception:
pass
# An OIDC sign-in page shows a "Sign in with ..." button and no username
# field. The "sign in" role-name also matches a password form's submit
# button, so the *absence* of the username field is what distinguishes a
# true SSO-only page from a password form.
sso_button = page.get_by_role("button", name=re.compile(r"sign in", re.IGNORECASE)).first
sso_only = sso_button.is_visible() and not page.locator(LoginPage.USERNAME).is_visible()
if interactive_auth and sso_only:
# Storage state was pre-loaded by --interactive-auth / --headless-auth.
# Workbench's SSO sign-in page does not auto-redirect to the IdP; it
# renders a "Sign in with OpenID" button. Clicking it triggers a
# silent SSO round-trip using the saved IdP cookies, landing on the
# authenticated homepage with no credentials required.
sso_button.click()
try:
homepage_logo.wait_for(state="visible", timeout=TIMEOUT_PAGE_LOAD)
return # Silent SSO succeeded
except Exception:
# No usable IdP session (expired, or storage state was stripped
# for the password-login test) — silent SSO can't complete on an
# OIDC deployment, so skip gracefully.
pytest.skip(
_workbench_session_skip_message(
auth_mode=auth_mode,
workbench_auth_error=workbench_auth_error,
landed_url=page.url,
)
)
if auth_provider != "password":
pytest.skip(
_workbench_session_skip_message(
auth_mode=auth_mode,
workbench_auth_error=workbench_auth_error,
landed_url=page.url,
)
)
# Even when auth_provider is reported as "password", the deployment may
# actually present an SSO/OIDC sign-in page (a "Sign in with ..." button
# and no username field) — e.g. auth_provider defaulted to "password" on
# a config-less run against an OIDC deployment. The password login form
# is unavailable there, so skip rather than fail the retry loop.
if sso_only:
pytest.skip(
"Workbench is configured for SSO/OIDC (no password login form); "
"the password-login scenario does not apply. Pass --interactive-auth "
"or --headless-auth to exercise authenticated Workbench tests."
)
# Password auth - proceed with form login below
else:
# Not on homepage, not on login page - unexpected state
# Give it one more check in case page is still loading
try:
homepage_logo.wait_for(state="visible", timeout=TIMEOUT_QUICK)
return
except Exception:
pass
# Password authentication with retry logic
login_form = page.locator(LoginPage.USERNAME)
error_panel = page.locator(LoginPage.ERROR_PANEL)
for attempt in range(max_retries):
if attempt > 0:
time.sleep(retry_delay)
page.goto(workbench_url)
# Fast path check on retry
if homepage_logo.is_visible():
return
# Wait for login form to be ready
try:
login_form.wait_for(state="visible", timeout=TIMEOUT_QUICK)
except Exception:
continue
# Fill and submit
page.fill(LoginPage.USERNAME, username)
page.fill(LoginPage.PASSWORD, password)
stay_signed_in = page.locator(LoginPage.STAY_SIGNED_IN)
if stay_signed_in.is_visible() and not stay_signed_in.is_checked():
stay_signed_in.click()
page.click(LoginPage.BUTTON)
# Wait for either homepage (success) or error panel (failure)
homepage_or_error = homepage_logo.or_(error_panel)
try:
homepage_or_error.wait_for(state="visible", timeout=TIMEOUT_PAGE_LOAD)
except Exception:
if attempt == max_retries - 1:
raise AssertionError(f"Login failed after {max_retries} attempts: no response")
continue
# Check which one appeared
if homepage_logo.is_visible():
return # Success!
# Error appeared - extract message and maybe retry
if attempt == max_retries - 1:
error_text = page.locator(LoginPage.ERROR_TEXT).text_content()
> raise AssertionError(f"Login failed: {error_text or 'Unknown error'}")
E AssertionError: Login failed: Error: Temporary server error, please try again
src/vip_tests/workbench/conftest.py:487: AssertionErrorPASSConnect server is reachablePrerequisites
As a Posit Team administrator
Test procedure
- Given <product> is configured in vip.toml
- When I request the <product> health endpoint
- Then the server responds with a successful status code
PASSAdmin user exists and has admin privilegesConnect
As a Posit Team administrator
Test procedure
- Given Connect is accessible at the configured URL
- When I retrieve the current user profile
- Then the user has admin privileges
FAILUsers can be listedConnect
As a Posit Team administrator
Test procedure
- Given Connect is accessible at the configured URL
- When I list all users
- Then the user list is not empty
- And the test user exists in the user list
test_list_users: Test user 'testuser' (from 'testuser') not found in user list: ['__bootstrap_admin__']
assert 'testuser' in ['__bootstrap_admin__']
Full error traceback
fixturefunc = <function check_test_user_in_list at 0x7f04b21d62a0>
request = <FixtureRequest for <Function test_list_users>>
kwargs = {'user_list': [{'email': '', 'username': '__bootstrap_admin__', 'first_name': '', 'last_name': '', ...}], 'test_username': 'testuser'}
def call_fixture_func(
fixturefunc: _FixtureFunc[FixtureValue], request: FixtureRequest, kwargs
) -> FixtureValue:
if inspect.isgeneratorfunction(fixturefunc):
fixturefunc = cast(Callable[..., Generator[FixtureValue]], fixturefunc)
generator = fixturefunc(**kwargs)
try:
fixture_result = next(generator)
except StopIteration:
raise ValueError(f"{request.fixturename} did not yield a value") from None
finalizer = functools.partial(_teardown_yield_fixture, fixturefunc, generator)
request.addfinalizer(finalizer)
else:
fixturefunc = cast(Callable[..., FixtureValue], fixturefunc)
> fixture_result = fixturefunc(**kwargs)
^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.14/site-packages/_pytest/fixtures.py:1005:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
user_list = [{'email': '', 'username': '__bootstrap_admin__', 'first_name': '', 'last_name': '', ...}]
test_username = 'testuser'
@then("the test user exists in the user list")
def check_test_user_in_list(user_list, test_username):
if not test_username or not test_username.strip():
pytest.skip("No test user configured — skipping user lookup assertion")
test_username = test_username.strip()
expected = test_username.split("@", 1)[0]
usernames = [u.get("username") for u in user_list]
> assert expected in usernames, (
f"Test user {expected!r} (from {test_username!r}) not found in user list: {usernames}"
)
E AssertionError: Test user 'testuser' (from 'testuser') not found in user list: ['__bootstrap_admin__']
E assert 'testuser' in ['__bootstrap_admin__']
src/vip_tests/connect/test_users.py:58: AssertionErrorPASSGroups can be listedConnect
As a Posit Team administrator
Test procedure
- Given Connect is accessible at the configured URL
- When I list all groups
- Then the response is successful
SKIPExpected R versions are available on ConnectConnect
As a Posit Team administrator
Test procedure
- Given Connect is accessible at the configured URL
- And expected R versions are specified in vip.toml
- When I query Connect for available R versions
- Then all expected R versions are present
SKIPExpected Python versions are available on ConnectConnect
As a Posit Team administrator
Test procedure
- Given Connect is accessible at the configured URL
- And expected Python versions are specified in vip.toml
- When I query Connect for available Python versions
- Then all expected Python versions are present
PASSExpected Quarto versions are availableConnect
As a Posit Team administrator
Test procedure
- Given Connect is accessible at the configured URL
- When I query Connect for available Quarto versions
- Then at least one Quarto version is available
PASSUser can log in via the web UIConnect
As a Posit Team administrator
Test procedure
- Given Connect is accessible at the configured URL
- When a user navigates to the Connect login page
- And enters valid credentials
- Then the user is successfully authenticated
- And the Connect dashboard is displayed
PASSAPI key authentication worksConnect
As a Posit Team administrator
Test procedure
- Given Connect is accessible at the configured URL
- And a valid API key is configured
- When I request the current user via the API
- Then the API returns user information
PASSInvalid API key returns 401Security
As a Posit Team administrator
Test procedure
- Given Connect is configured in vip.toml
- When I make an API request to Connect with an invalid key
- Then the response status is 401
PASSNon-existent endpoint returns 404Security
As a Posit Team administrator
Test procedure
- Given Connect is configured in vip.toml
- When I request a non-existent endpoint on Connect
- Then the response status is 404
PASSCRAN mirror is accessiblePackage Manager
As a Posit Team administrator
Test procedure
- Given Package Manager is running
- When I query the CRAN repository for the "Matrix" package
- Then the package is found in the repository
SKIPPyPI mirror is accessiblePackage Manager
As a Posit Team administrator
Test procedure
- Given Package Manager is running
- When I query the PyPI repository for the "requests" package
- Then the package is found in the repository
SKIPBioconductor mirror is accessiblePackage Manager
As a Posit Team administrator
Test procedure
- Given Package Manager is running
- When I query the Bioconductor repository for the "BiocGenerics" package
- Then the package is found in the repository
SKIPOpenVSX mirror is accessiblePackage Manager
As a Posit Team administrator
Test procedure
- Given Package Manager is running
- When I query the OpenVSX repository for the "golang.Go" extension
- Then the package is found in the repository
PASSAt least one repository is configuredPackage Manager
As a Posit Team administrator
Test procedure
- Given Package Manager is running
- When I list all repositories
- Then at least one repository exists
PASSConnect system checks can be run and the report downloadedConnect
As a Posit Team administrator
Test procedure
- Given Connect is accessible at the configured URL
- And a valid API key is configured
- When I trigger a new system check run via the Connect API
- Then the system check report is returned
- And I can download the system check report artifact
PASSUnauthenticated API request returns 401Security
As a Posit Team administrator
Test procedure
- Given Connect is configured in vip.toml
- When I make an unauthenticated API request to Connect
- Then the response status is 401