Validating External Resources with {valtools} • valtools

# remotes::install_github("phuse-org/valtools")
library(valtools)

This vignette steps through the process of validating external resources with {valtools}. “External resources” is defined for this vignette as any software or environment that does not follow the validation process defined in the Starting New Validation Package using {valtools} vignette. This means packages installed from other sources, collections of packages, external software, and environments.

This process may be invoked when an R package was already developed and only after the fact was validation decided to be of value, validating an externally generated package (for example a package installed from CRAN), or validating an environment or system.

Creating A New Validation Packet

Starting a new validation packet using {valtools} starts with vt_create_packet(). Here, the user passes the directory the project is to be performed in, and optionally the target of validation.

The “target of validation” is what the packet intends to validate - be it an external package, environment, software, etc.

If the target is not provided, {valtools} will request the user provide this information


# create a new validation packet skeleton
vt_create_packet(
  path = "/my/path/to/example_validation_packet", 
  target = "super.package"
  )

The validation packet folder is constructed using the usethis::create_project() function, and then all the necessary validation infrastructure required for {valtools}.

Importantly, there is now a validation folder, which is the working directory for the validation. This is where almost all the content for validation will be created.

Inside this folder, there is the validation.yml file, which will be referred as the validation config file going forward. This YML file informs {valtools} how to interact with the various pieces of validation that will be created, and information that needs to be shared across multiple sessions/users. The user does not need to interact with this file directly, the functions inside {valtools} will update this file as necessary.

Tutorial

Run the chunk of code below to create a validation packet in a temporary directory to follow along with the tutorial for the {whoami} package. The new packet project will be opened in a new session. Run all subsequent code in that new session.


valtools::vt_create_packet(
  path = file.path(tempdir(),"validation_packet_whoami"),
  target = "whoami"
  )

To examine the folder structure of the new package, run the following function:


fs::dir_tree(recurse = TRUE)

Add Requirements

Requirements document users needs of the target - what are the problems the target solves for the users - and must be documented before any test cases are written. Requirements are recorded within the validation/requirements folder by default. The collection of requirements may be called specifications.

Requirements are written as markdown (.md) documents with special roxygen headers. Each requirement must have the following roxygen comments in the header: title, editor, editDate, and riskAssessment. The last the roxygen comments are custom {valtools} supported roxygen tags to support validation. @editor is for tracking the last editor of the function, @editDate is for recording whenever a function is modified, and @riskAssessment is for tracking risks for each requirement.

To make adding validation content easy, {valtools} extended the usethis approach to package contents creation through a family of “vt_use_*” functions.

vt_use_req() creates a new requirement in the validation/requirements folder, with the main argument being the name of the requirement, and an optional argument username to record the name of the person writing the requirement.

If the username argument is not passed, {valtools} will automatically get the computer username of the user creating the requirement and attempt to put in their full name. If the user has not created any validation contents before, it will ask the user some questions (Name, Title, and Role) and record them in the validation config file for documentation in the validation report.


valtools::vt_use_req("Requirement_001")

Tutorial

Run the command above and in the newly opened requirements file, on line 5, Replace REQUIREMENTS with 1.1, and ASSESSMENT with 1, Low Risk, Small Impact to indicate requirement 1.1 has a risk assessment that determined it has a low risk and small impact when it is wrong.

Add a new line underneath the line above (at line 6) line that contains: #' 1.2: 5, Low risk, Medium Impact

Copy the following content:

+ 1.1 Collect user id for current session
+ 1.2 Collect full name of user for current session

Change Log

Similar to a news file, {valtools} suggests the use of a change log that is directly tied to validation for recording changes. The purpose of this is to capture update and information that is useful for developers from information that is important to capture in validation.

To create this change log file, {valtools} has the function vt_use_change_log(). It will create the change log file inside the working directory and open it up for editing.

The header information tracks the version of validation and the date of the release of validation. This is a markdown file, so normal markdown can be used to document the changes. However, critically here, only bullets marked with [validation] will be recorded in the validation report.


valtools::vt_use_change_log()

Tutorial

Run the command above to create a change log.

Testing

Testing is done to ensure that the target meets the requirements that were set out for the project. Testing is done in two major steps: the firsts consists of writing out a series of cases that would prove that the requirements have been met, the second is the application of these cases.

Test Cases

The addition and writing of test cases is handled by the vt_use_test_case() function. Similarly to vt_use_req(), a username can be passed, or it will look to determine which user is calling the function and input their information.

This function creates the test case file in the validation/test_cases folder of the package and opens it for editing.


valtools::vt_use_test_case("Test_case_001")

Tutorial

Run the code above and in the newly opened test case file, replace TESTCASE with 1.1, and REQUIREMENT with 1.1 to indicate test case 1.1 shows that requirement 1.1 is being met.

Add a new line underneath the line above (at line 6) line that contains: #' 1.2: 1.1, 1.2

This is to indicate test case 1.2 shows requirements 1.1 and 1.2 are being met.

Copy the following test case into file where test cases are to be documented:

+ 1.1 Test that the software can identify the username pf the user by
setting the environment variable `LOGNAME` to `jsmith` for the duration of
the test and  confirming that the output of whoami::username is `jsmith`.
+ 1.2 Test that the software can identify the full name of the user by 
setting the environment variable `FULLNAME` to "John Smith" for the
duration of the test and confirming  that the output of whoami::fullname()
is "John Smith"

Test Code

Test code is the implementation of the test cases as code. The goal is that the code is completely reproducible and able to be run without human interaction. Additionally, test code is written by a third party - someone that was not involved with writing the actual code or the test case. This helps ensure the integrity of the testing as well as providing valuable review of the documentation of the test cases and package code.

Similarly to vt_use_req() for requirements and vt_use_test_case for test cases, {valtools} provide a function for creating test code files and recording which user created the file.


valtools::vt_use_test_code("Test_code_001")

Tutorial

Add “Val A Dashun” to the validation config file:


valtools::vt_add_user_to_config(
  username = "user_b",
  name = "Val A Dashun",
  title = "Programmer II",
  role = "tester"
)

Now that this persons information is recorded, construct the test code file that they will use to record the test code through the code below.


valtools::vt_use_test_code("Test_code_001", username = "Val A Dashun")

In the newly opened test code file. Update TESTNUMBER to 1.1 in the new test code file and copy the code below into the body of the test:


withr::with_envvar( 
  new = list(LOGNAME = "jsmith"),
  {
  
  user <- whoami::username()
  
  expect_equal(
  user,
  "jsmith"
 )
 
})

add a new test with the following beneath the test. Replace “TODAYS DATE” with today’s date.


#' @editor Val A Dashun
#' @editDate TODAYS DATE
test_that("1.2",{

  withr::with_envvar( 
    new = list(FULLNAME = "John Smith"),
    {
    
    user_full_name <- whoami::fullname()
    
    expect_equal(
    user_full_name,
    "John Smith"
   )
  
  })
})

Authoring Validation Reports

{valtools} provides dynamic access via a Rmarkdown file to details necessary for generating a validation report at push of button. This validation report documents that the package meets stated goals and can be re-evaluated as necessary to generate the report in PDF or HTML format.

The function vt_use_report() creates a validation report rmarkdown file pre-populated with code to scrape all the pieces of information that were generated in the prior steps to create the final report when being knit.

vt_use_report() saves the validation report rmarkdown file in the working directory identified in the validation config file. Within packages this defaults to the base folder. This rmarkdown file will have a default name validation.Rmd if unspecified.

valtools::vt_use_report(template = "packet")

There are several sections included by default in the provided validation report rmarkdown:

Signatures: Capture signatures of everyone involved in the validation.
Release Details:
- Records the validation environment
- Presents the change log of the validation.
- Subsections to show the last editor for each piece of the validation; requirements, test cases and test code.
- Traceability table to show which requirements are being
- Risk Assessment: Combines all the risk assessments made into a single table
Validation: record each requirement, test case, and results of the test code

{valtools} also supports a concept called “dynamic referencing”, which will be explained in another vignette.

When editing the report, some key functions to know for extending the report included by {valtools} are:

vt_path() allows user to base path from the validation directory. Similar idea to the {here} package, but for validation.
vt_file() allows the user to point to specific files and render them as child documents within the report.
vt_scrape_* family of functions allows users to scrape various pieces of information from the validation infrastructure and returns a data.frame.
vt_kable_* family functions provides an opinionated formatting to the vt_scrape_* functions to help quickly construct the report.
vt_get_child_files() returns the list of files that are indicated in the validation.yml to be included in the validation report. This allows for batch creation of the dynamic content in the report.

Keep in mind, the report is an Rmarkdown, so there is no limit to editing and customization, and templates.

Tutorial

Run the code above to generate the report, and inspect the overall structure of the report. See what happens when contents are moved around.

Running a Validation Report

Now that there is a validation report as an Rmarkdown, validation is only a compiling of the report away. To validate the target, we execute the report non-interactively and save the results.

Tutorial

Run the validation report

Validation Mode: Running on Source

valtools::vt_validate_report()