Test Inventory

As a Posit Team administrator I want to verify that authentication credentials are available So that login-dependent tests can proceed

tests/prerequisites/test_auth_configured.feature

• Test credentials are provided

  1. Given VIP is configured with test user credentials
  2. Then the username is not empty
  3. And the password is not empty

As a Posit Team administrator I want to verify that all configured components are running So that I know the basic installation is healthy before running deeper tests

tests/prerequisites/test_components.feature

• <product> server is reachable

  1. Given <product> is configured in vip.toml
  2. When I request the <product> health endpoint
  3. Then the server responds with a successful status code

As a VIP developer I want to see what a failed test looks like in the report So that I can verify the report renders failures correctly

tests/prerequisites/test_expected_failure.feature

• Workbench server is reachable but not configured

  1. Given Workbench is expected to be configured
  2. When I check the Workbench configuration
  3. Then Workbench should be reachable

As a Posit Team administrator I want to verify that the running product versions match the configured expectations So that I know the correct software versions are deployed

tests/prerequisites/test_versions.feature

• Connect version matches configuration

  1. Given Connect is configured in vip.toml with a version expectation
  2. When I fetch the Connect server version
  3. Then the Connect version matches the configured value
• Workbench version matches configuration

  1. Given Workbench is configured in vip.toml with a version expectation
  2. When I fetch the Workbench server version
  3. Then the Workbench version matches the configured value
• Package Manager version matches configuration

  1. Given Package Manager is configured in vip.toml with a version expectation
  2. When I fetch the Package Manager server version
  3. Then the Package Manager version matches the configured value

As a Posit Team administrator I want to verify that users can log in to Connect So that I know authentication is properly configured

tests/connect/test_auth.feature

• User can log in via the web UI

  1. Given Connect is accessible at the configured URL
  2. When a user navigates to the Connect login page
  3. And enters valid credentials
  4. Then the user is successfully authenticated
  5. And the Connect dashboard is displayed
• API key authentication works

  1. Given Connect is accessible at the configured URL
  2. And a valid API key is configured
  3. When I request the current user via the API
  4. Then the API returns user information

As a Posit Team administrator I want to verify that content can be deployed and executed So that I know the Connect publishing pipeline is functional

tests/connect/test_content_deploy.feature

• Deploy and execute a Quarto document

  1. Given Connect is accessible at the configured URL
  2. When I create a VIP test content item named "vip-quarto-test"
  3. And I upload and deploy a minimal Quarto bundle
  4. And I wait for the deployment to complete
  5. Then the content is accessible via HTTP
  6. And the content renders expected output
  7. And I clean up the test content
• Deploy and execute a Plumber API

  1. Given Connect is accessible at the configured URL
  2. When I create a VIP test content item named "vip-plumber-test"
  3. And I upload and deploy a minimal Plumber bundle
  4. And I wait for the deployment to complete
  5. Then the content is accessible via HTTP
  6. And the content renders expected output
  7. And I clean up the test content
• Deploy and execute a Shiny application

  1. Given Connect is accessible at the configured URL
  2. When I create a VIP test content item named "vip-shiny-test"
  3. And I upload and deploy a minimal Shiny bundle
  4. And I wait for the deployment to complete
  5. Then the content is accessible via HTTP
  6. And the content renders expected output
  7. And I clean up the test content
• Deploy and execute a Dash application

  1. Given Connect is accessible at the configured URL
  2. When I create a VIP test content item named "vip-dash-test"
  3. And I upload and deploy a minimal Dash bundle
  4. And I wait for the deployment to complete
  5. Then the content is accessible via HTTP
  6. And the content renders expected output
  7. And I clean up the test content
• Deploy and execute an R Markdown document

  1. Given Connect is accessible at the configured URL
  2. When I create a VIP test content item named "vip-rmarkdown-test"
  3. And I upload and deploy a minimal R Markdown bundle
  4. And I wait for the deployment to complete
  5. Then the content is accessible via HTTP
  6. And the content renders expected output
  7. And I clean up the test content
• Deploy and execute a Jupyter Notebook

  1. Given Connect is accessible at the configured URL
  2. When I create a VIP test content item named "vip-jupyter-test"
  3. And I upload and deploy a minimal Jupyter Notebook bundle
  4. And I wait for the deployment to complete
  5. Then the content is accessible via HTTP
  6. And the content renders expected output
  7. And I clean up the test content
• Deploy and execute a FastAPI application

  1. Given Connect is accessible at the configured URL
  2. When I create a VIP test content item named "vip-fastapi-test"
  3. And I upload and deploy a minimal FastAPI bundle
  4. And I wait for the deployment to complete
  5. Then the content is accessible via HTTP
  6. And the content renders expected output
  7. And I clean up the test content

As a Posit Team administrator I want to verify that external data sources connect and function So that published content can access the databases and services it needs

tests/connect/test_data_sources.feature

• External data sources are reachable from Connect

  1. Given Connect is accessible at the configured URL
  2. And external data sources are configured in vip.toml
  3. When I test connectivity to each data source
  4. Then all data sources respond successfully

As a Posit Team administrator I want to verify that Connect can send emails So that scheduled report delivery and notifications work

tests/connect/test_email.feature

• Connect can send a test email

  1. Given Connect is accessible at the configured URL
  2. And email delivery is enabled
  3. When I send a test email via the Connect API
  4. Then the email task completes without error

As a Posit Team administrator I want to verify that Connect installs packages from the correct source So that package provenance is as expected (PPPM, PPM, CRAN, etc.)

tests/connect/test_packages.feature

• Connect is configured to use the expected package repository

  1. Given Connect is accessible at the configured URL
  2. When I query the Connect server settings for package repositories
  3. Then the configured R repository URL is present in the settings
• Package Manager URL is the default repository source

  1. Given Connect is accessible at the configured URL
  2. And Package Manager is configured in vip.toml
  3. When I deploy a content item that installs R packages
  4. Then the deployment logs show packages installed from Package Manager
  5. And I clean up the deployed content

As a Posit Team administrator I want to verify that the expected R and Python versions are available So that content can be deployed with the correct runtime

tests/connect/test_runtime_versions.feature

• Expected R versions are available on Connect

  1. Given Connect is accessible at the configured URL
  2. And expected R versions are specified in vip.toml
  3. When I query Connect for available R versions
  4. Then all expected R versions are present
• Expected Python versions are available on Connect

  1. Given Connect is accessible at the configured URL
  2. And expected Python versions are specified in vip.toml
  3. When I query Connect for available Python versions
  4. Then all expected Python versions are present
• Expected Quarto versions are available

  1. Given Connect is accessible at the configured URL
  2. When I query Connect for available Quarto versions
  3. Then at least one Quarto version is available

As a Posit Team administrator I want to run the Connect system diagnostics So that I can verify the Connect installation is healthy

tests/connect/test_system_checks.feature

• Connect system checks can be run and the report downloaded

  1. Given Connect is accessible at the configured URL
  2. And a valid API key is configured
  3. When I trigger a new system check run via the Connect API
  4. Then the system check report is returned
  5. And I can download the system check report artifact

As a Posit Team administrator I want to verify that user and group management APIs are functional So that I know access control is properly configured

tests/connect/test_users.feature

• Admin user exists and has admin privileges

  1. Given Connect is accessible at the configured URL
  2. When I retrieve the current user profile
  3. Then the user has admin privileges
• Users can be listed

  1. Given Connect is accessible at the configured URL
  2. When I list all users
  3. Then the user list is not empty
  4. And the test user exists in the user list
• Groups can be listed

  1. Given Connect is accessible at the configured URL
  2. When I list all groups
  3. Then the response is successful

As a Posit Team administrator I want to verify that products are integrated with each other So that the deployment functions as a cohesive platform

tests/cross_product/test_integration.feature

• Content deployed on Connect uses packages from Package Manager

  1. Given Connect is configured in vip.toml
  2. And Package Manager is configured in vip.toml
  3. When I deploy a content item that installs R packages on Connect
  4. Then the deployment logs mention the Package Manager URL as the package source
  5. And I clean up the integration test content

As a Posit Team administrator I want to verify that monitoring and logging are configured So that operational issues can be detected and diagnosed

tests/cross_product/test_monitoring.feature

• Monitoring is configured

  1. Given monitoring is enabled in vip.toml
  2. When I check product health endpoints
  3. Then all configured products respond to health checks

As a Posit Team administrator I want to verify that all configured product servers respond to health checks So that I know the services are running and reachable

tests/cross_product/test_resources.feature

• All configured products respond to health checks

  1. Given at least one product is configured
  2. When I check the health of each configured product
  3. Then all products respond with a healthy status

As a Posit Team administrator I want to verify that SSL certificates are valid and HTTPS is enforced So that all traffic is encrypted

tests/cross_product/test_ssl.feature

• SSL certificate is valid for <product>

  1. Given <product> is configured in vip.toml
  2. When I check the SSL certificate for <product>
  3. Then the certificate is valid and not expired
  4. And the certificate chain is complete
• HTTP redirects to HTTPS for <product>

  1. Given <product> is configured in vip.toml
  2. When I request the HTTP URL for <product>
  3. Then the response redirects to HTTPS
  4. And the HTTP port is not open
• TLS 1.2 or higher is enforced for <product>

  1. Given <product> is configured in vip.toml
  2. When I attempt a TLS connection to <product>
  3. Then TLS 1.0 and TLS 1.1 connections are rejected
  4. And TLS 1.2 or higher succeeds

As a Posit Team administrator I want to verify that private package repos resolve correctly So that internal packages can be installed

tests/package_manager/test_private_repos.feature

• Private repositories are reachable

  1. Given Package Manager is running
  2. And private repositories are configured
  3. When I query each private repository
  4. Then each repository responds successfully

As a Posit Team administrator I want to verify that CRAN and PyPI mirrors are working So that R and Python packages can be installed from the correct source

tests/package_manager/test_repos.feature

• CRAN mirror is accessible

  1. Given Package Manager is running
  2. When I query the CRAN repository for the "Matrix" package
  3. Then the package is found in the repository
• PyPI mirror is accessible

  1. Given Package Manager is running
  2. When I query the PyPI repository for the "requests" package
  3. Then the package is found in the repository
• Bioconductor mirror is accessible

  1. Given Package Manager is running
  2. When I query the Bioconductor repository for the "BiocGenerics" package
  3. Then the package is found in the repository
• At least one repository is configured

  1. Given Package Manager is running
  2. When I list all repositories
  3. Then at least one repository exists

As a Posit Team administrator I want to verify system stability under concurrent usage So that the deployment can handle multiple simultaneous users

tests/performance/test_concurrency.feature

• Multiple concurrent API requests to Connect succeed

  1. Given Connect is configured in vip.toml
  2. When I send concurrent health-check requests to Connect
  3. Then all requests succeed
  4. And the average response time is within the configured threshold
• Multiple concurrent requests to Package Manager succeed

  1. Given Package Manager is configured in vip.toml
  2. When I send concurrent status requests to Package Manager
  3. Then all requests succeed
  4. And the average response time is within the configured threshold
• Workbench handles concurrent health check requests

  1. Given Workbench is configured in vip.toml
  2. When I send concurrent health-check requests to Workbench
  3. Then all requests succeed
  4. And the average response time is within the configured threshold

As a Posit Team administrator I want to verify that login and page load times are acceptable So that users have a responsive experience

tests/performance/test_login_load_times.feature

• <product> login page loads within acceptable time

  1. Given <product> is configured in vip.toml
  2. When I measure the <product> login page load time
  3. Then the page loads within the configured timeout

As a Posit Team administrator I want to verify that package downloads from Package Manager are fast So that users and deployments are not blocked by slow installs

tests/performance/test_package_install_speed.feature

• CRAN package downloads within acceptable time

  1. Given Package Manager is running and has a CRAN repo
  2. When I download a small CRAN package
  3. Then the download completes within the configured timeout
• PyPI package downloads within acceptable time

  1. Given Package Manager is running and has a PyPI repo
  2. When I download a small PyPI package
  3. Then the download completes within the configured timeout

As a Posit Team administrator I want to verify that products respond quickly under moderate load So that users experience acceptable performance

tests/performance/test_resource_usage.feature

• Products respond within acceptable time under moderate load

  1. Given at least one product is configured
  2. When I generate moderate API traffic for 10 seconds
  3. Then the p95 response time is within the configured threshold
  4. And the error rate is below 10 percent
• Prometheus metrics endpoint is enabled

  1. Given at least one product is configured
  2. Then each product has a working Prometheus metrics endpoint

As a Posit Team administrator I want to verify that authentication aligns with organizational policy So that access controls meet compliance requirements

tests/security/test_auth_policy.feature

• Auth provider matches expected configuration

  1. Given the expected auth provider is specified in vip.toml
  2. When I check the auth configuration
  3. Then the configured provider matches expectations
• Unauthenticated API access is denied

  1. Given Connect is configured in vip.toml
  2. When I make an unauthenticated API request to Connect
  3. Then the request is rejected with 401 or 403

As a Posit Team administrator I want to verify that the API returns correct error responses So that authentication failures and bad requests are handled correctly

tests/security/test_error_handling.feature

• Unauthenticated API request returns 401

  1. Given Connect is configured in vip.toml
  2. When I make an unauthenticated API request to Connect
  3. Then the response status is 401
• Invalid API key returns 401

  1. Given Connect is configured in vip.toml
  2. When I make an API request to Connect with an invalid key
  3. Then the response status is 401
• Non-existent endpoint returns 404

  1. Given Connect is configured in vip.toml
  2. When I request a non-existent endpoint on Connect
  3. Then the response status is 404

As a Posit Team administrator I want to verify that HTTPS is enforced across all products So that traffic is encrypted in transit

tests/security/test_https.feature

• <product> enforces HTTPS

  1. Given <product> is configured with an HTTPS URL
  2. When I make an HTTP request to <product>
  3. Then the connection is refused or redirected to HTTPS
• <product> does not expose sensitive headers

  1. Given <product> is configured in vip.toml
  2. When I inspect response headers from <product>
  3. Then the server does not expose version information in headers

As a Posit Team administrator I want to verify that secrets and tokens are stored securely So that credentials are not exposed

tests/security/test_secrets.feature

• API keys are not stored in the VIP config file

  1. Given a VIP configuration file is in use
  2. When I inspect the configuration file contents
  3. Then no plaintext API keys or passwords are present in the file
• Connect API key is provided via environment variable

  1. Given Connect is configured with an API key
  2. Then the API key was loaded from the VIP_CONNECT_API_KEY environment variable

As a Posit Team administrator I want to verify that users can log in to Workbench So that I know authentication is properly configured

tests/workbench/test_auth.feature

• User can log in to Workbench via the web UI

  1. Given Workbench is accessible at the configured URL
  2. When a user navigates to the Workbench login page and enters valid credentials
  3. Then the Workbench homepage is displayed
  4. And the current user is shown in the header

As a Posit Team administrator I want to verify that external data sources are accessible from Workbench So that users can connect to databases and services from their sessions

tests/workbench/test_data_sources.feature

• External data sources are reachable from Workbench

  1. Given Workbench is accessible at the configured URL
  2. And external data sources are configured in vip.toml
  3. When I verify data source connectivity
  4. Then all data sources are reachable

As a Posit Team administrator I want to verify that all configured IDEs launch properly So that users can start working in their preferred environment

tests/workbench/test_ide_launch.feature

• RStudio IDE session can be launched

  1. Given the user is logged in to Workbench
  2. When the user starts a new RStudio session
  3. Then the session transitions to Active state
  4. And the RStudio IDE is displayed and functional
  5. And the RStudio IDE can execute R code
  6. And the session is cleaned up
• VS Code session can be launched

  1. Given the user is logged in to Workbench
  2. When the user starts a new VS Code session
  3. Then the session transitions to Active state
  4. And the VS Code IDE is displayed
  5. And the VS Code terminal is accessible
  6. And the session is cleaned up
• JupyterLab session can be launched

  1. Given the user is logged in to Workbench
  2. When the user starts a new JupyterLab session
  3. Then the session transitions to Active state
  4. And the JupyterLab IDE is displayed
  5. And JupyterLab can execute code in a notebook
  6. And the session is cleaned up
• Positron session can be launched

  1. Given the user is logged in to Workbench
  2. When the user starts a new Positron session
  3. Then the session transitions to Active state
  4. And the Positron IDE is displayed
  5. And the Positron console is accessible
  6. And the session is cleaned up

As a Posit Team administrator I want to verify that Workbench installs packages from the correct source So that package provenance matches organizational policy

tests/workbench/test_packages.feature

• R repos.conf points to the expected repository

  1. Given the user is logged in to Workbench
  2. When I check R repository configuration in an RStudio session
  3. Then the expected package repository URL is present

tests/workbench/test_sessions.feature

• Session can be suspended and resumed

  1. Given the user is logged in to Workbench
  2. When the user starts a new RStudio Pro session
  3. And the session reaches Active state
  4. And the user suspends the session
  5. Then the session reaches Suspended state
  6. When the user resumes the session
  7. Then the session reaches Active state again
  8. And the session is cleaned up