Introduction

Below is a list of resources on using R in regulated clinical trial environments…Below some of the links is a comment or note that I thought was interesting…Feel free to send me any I may have missed.

5/9/2019 Update:

Much of this information is now in the appendix of this doc:

How do I select an R package for my clinical workflow?

Paper: https://www.phusewiki.org/docs/2019%20US%20Connect%20Final%20Papers/TT/how-do-i-pick-an-r-package-for-my-clinical-workflow-bowsher-lopp-20454.pdf

Slides: https://www.phusewiki.org/docs/2019%20US%20Connect%20Final%20Papers/Presentations/US-Connect-Baltimore-2019-Presentations%20combined/PR_TT11.pdf

Repo: https://github.com/philbowsher/phuse-2019-r-packages

Best Practices for Reproducible Package Management in R

Paper: https://www.phusewiki.org/docs/2019%20US%20Connect%20Final%20Papers/AB/ab09-20245.pdf

Updated information on environments can be found here:

http://environments.rstudio.com/

These pages are helpful:

https://cran.r-project.org/web/views/ClinicalTrials.html

https://cran.r-project.org/web/views/ReproducibleResearch.html

For Tables, check out this package:

https://github.com/rstudio/GT/

If you are interest in validation, keep updated with this group:

https://www.pharmar.org/

Also see the R in Pharma conference:

http://rinpharma.com/

The key document is here:

https://www.r-project.org/doc/R-FDA.pdf

The following presents a hirearchy to understand testing and validation for R and R packages.

  1. R Core + Base and Recommended Packages

The source for R includes 3 things: R source code, a set of base packages, and a set of recommended packages.

The source for R and the R base packages are maintained by a closed group of developers called the R core team. The recommended packages are maintained by the same group and other select contributors.

All 3 components follow the release and testing cycle outlined here: https://www.r-project.org/doc/R-FDA.pdf

Specifically, the /tests directory of the R source includes tests for R core, the base packages, and the recommended packages: https://svn.r-project.org/R/branches/R-3-4-branch/tests/

You can navigate up the URL directory to see the /test folder for different branches of R, eg https://svn.r-project.org/R/branches

  1. CRAN Checks on Community Packages

In addition to the base and recommended packages, R developers can make use of an extensive set of community-built packages hosted on CRAN. Any package accepted on CRAN must pass a series of automated tests: https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Checking-packages which enforce the CRAN submission policies: https://cran.r-project.org/web/packages/policies.html

The results of these tests are available on CRAN for each package: http://cran.us.r-project.org/web/checks/check_summary_by_package.html#summary_by_package

The same tests can be run locally against any package source tar file using a command like: R CMD check package_0.1.tar.gz

The actual test code is part of R itself, not part of the package being tested.

CRAN check are run for the package plus the latest release of R, patch releases, and the current development branch of R. The tests are run on a variety of operating systems including Linux, Windows, and Mac OS.

  1. Community Packages + Developer Contributed Unit Tests

In addition to the CRAN checks, many package authors have added their own tests which help ensure the package’s functions work as intended. Most often, these tests are built into the package in a specific way so that whenever the automatic CRAN checks are run the custom tests also run.

These additional tests are part of the package source.

More information: http://r-pkgs.had.co.nz/tests.html#test-cran

  1. The Environment

As mentioned, CRAN tests packages across a matrix of R versions and operating systems. However, ideally the tests are run again for the specific, targeted validated environment. This process would include:

  1. Building R from source in the validated environment. Running the tests as described in Part 1.
  2. For each community R package, download the package source into the validated environment and then run R CMD check. Once complete, install the package.

It is NOT recommended that steps 1 and 2 happen while an analysis is developed, as both steps would introduce a burden for R developers. It is sufficient to complete the process at the end of an analysis while preparing for a submission or for “freezing” the production environment.

Further info:

https://github.com/tidyverse/ggplot2/tree/master/tests

Other unit tests can be executed and are usually determined by the package requirements and in the risk assessment.

Some orgs develop a test framework in which each R package in the installation is associated with a test or “validation” package containing additional tests.

These package tests are usually written using the “testthat” framework for R package testing. Here is an example:

https://github.com/rOpenHealth/openfda/tree/master/tests/testthat

https://github.com/hadley/testthat

You can also find CRAN package check results for packages as explained above.

You can find published checks/tests by package, and the link to the tests results is found on the package’s main page in CRAN.

This is the link to the latest results of dplyr:

http://cran.us.r-project.org/web/checks/check_results_dplyr.html

The link above shows timing.

To know what tests are performed, you can find that here:

http://r-pkgs.had.co.nz/check.html

For packages, CRAN runs test on multiple OS versions, it links to the results of the checks for each, here is the one for Debian GCC:

https://www.r-project.org/nosvn/R.check/r-devel-linux-x86_64-debian-clang/dplyr-00check.html

The link is on the Status column. Those are the results of CRAN itself running the R CMD check in Hadley’s book - http://r-pkgs.had.co.nz/check.html.

Landscape of Architecture

Git/Github/Gitlab

https://peerj.com/preprints/3159.pdf

http://happygitwithr.com/

https://github.com/tidyverse/reprex

Travis CI…to run your tests whenever code is updated in a controlled environment, compute code coverage

https://juliasilge.com/blog/beginners-guide-to-travis/

testthat package

http://kbroman.org/pkg_primer/pages/tests.html

testthat: Get Started with Testing

http://rjournal.github.io/archive/2011-1/RJournal_2011-1.pdf

http://r-pkgs.had.co.nz/tests.html

Testing Shiny

https://rstudio.github.io/shinytest/articles/shinytest.html

https://rstudio.github.io/shinyloadtest/

Validation document via R Markdown - easily updated or even run nightly

http://rmarkdown.rstudio.com/gallery.html

http://rmarkdown.rstudio.com/lesson-1.html

https://bookdown.org/yihui/bookdown/

https://github.com/philbowsher/Job-Scheduling-R-Markdown-Reports-via-R

https://rviews.rstudio.com/2017/06/28/printing-from-flex-dashboard/

https://yihui.name/knitr/demo/vignette/

https://github.com/rstudio/rticles

http://rmarkdown.rstudio.com/developer_document_templates.html

Virtual machines and docker containers can be used to freeze a particular build of software

http://dirk.eddelbuettel.com/blog/2014/10/23/

https://github.com/philbowsher/Boston-Rocker

https://hub.docker.com/r/rocker/

“With a syntax that is simpler than other provisioning tools (e.g. Chef, Puppet, Ansible) or Continuous Integration (CI) platforms (e.g. Travis CI, Shippable CI); users need little more than a basic familiarity with shell scripts and a Linux distribution software environment (e.g. Debian-based apt-get) to get started writing Dockerfiles.”

https://arxiv.org/pdf/1410.0846.pdf

For R packages, packrat and similar tools can be used to freeze analysis.

https://rstudio.github.io/packrat/rstudio.html

https://beta.rstudioconnect.com/content/2715/Up-to-Bat-with-Packrat.html

https://cran.r-project.org/web/packages/miniCRAN/index.html

https://cran.r-project.org/web/packages/switchr/index.html

Package Info

https://github.com/r-lib/sessioninfo

sessioninfo::session_info()

sessionInfo()

devtools::session_info()

https://rpubs.com/jesuscastagnetto/caret-knn-cancer-prediction

Other tools

https://mran.microsoft.com/rro/#repos

https://cran.r-project.org/web/packages/checkpoint/index.html

https://www.rstudio.com/resources/videos/user-lightning-talks/

Make

https://wlandau.github.io/2016/06/14/workflow/

http://opr.princeton.edu/workshops/Downloads/2017Jan_ReproducibleReasearchToolsPratt.pdf

https://github.com/richfitz/remake

https://github.com/wlandau/remakeInPackage

Shiny

http://www.mbswonline.com/upload/presentation_5-23-2016-13-1-14.mbsw%20slides%20man%202016%20final.pdf

https://cran.r-project.org/web/packages/BEACH/BEACH.pdf

https://www.user2017.brussels/uploads/useR_2017_posterV2.pdf

http://onlinelibrary.wiley.com/doi/10.1002/psp4.12052/full

https://isop.shinyapps.io/DoxycyclineSimulation/

https://cran.r-project.org/web/packages/RxODE/vignettes/RxODE-intro.html

https://user2016.sched.com/event/7GxO/r-shiny-application-for-the-evaluation-of-surrogacy-in-clinical-trials

https://cran.r-project.org/web/packages/Surrogate/Surrogate.pdf

https://open.fda.gov/

https://openfda.shinyapps.io/LRTest/

https://www.pharmasug.org/proceedings/2017/AD/PharmaSUG-2017-AD14.html#/

https://statist.shinyapps.io/bioeq_en/

http://onlinelibrary.wiley.com/doi/10.1002/psp4.21/abstract

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4394611/

http://bmjopen.bmj.com/content/6/3/e010579

http://ehdviz.dudleylab.org/help.html#introduction

http://pubmedcentralcanada.ca/pmcc/articles/PMC5418887/

http://labrtorian.com/2016/08/15/a-shiny-app-for-passing-bablok-and-deming-regression/

https://bahar.shinyapps.io/method_compare/

http://www.lexjansen.com/phuse/2016/dv/DV02.pdf

http://www.phusewiki.org/docs/2017_CSS_US/PP34_Final.pdf

http://www.phusewiki.org/docs/2017_SDEs/NorthChicago/Presentations/Jimmy%20Wong.pdf

https://ww2.amstat.org/meetings/jsm/2017/onlineprogram/AbstractDetails.cfm?abstractid=323938

http://www.phusewiki.org/docs/2017_SDES_FosterCity_Presentations/Data%20Visualization%20Applications%20Using%20R:Shiny.pdf

http://www.phusewiki.org/docs/Conference%202014%20CS%20Presentations/CS05.pdf

http://www.phusewiki.org/docs/CSS2015Presentations/PP26FINAL.pdf

http://www.pharmasug.org/proceedings/2016/DG/PharmaSUG-2016-DG10.pdf

Other info

R and Revolution R Enterprise in regulatory environments

http://www.lexjansen.com/phuse/2014/sd/SD01.pdf

Validation is an essential aspect regarding the employed software in regulatory environments, in particular in the pharmaceutical industry.

The FDA summarizes validation as follows: “Establishing documented evidence which provides a high degree of assurance that a specific process will consistently produce a product meeting its predetermined specifications and quality attributes.”

How R is used by the FDA for regulatory compliance

http://blog.revolutionanalytics.com/2017/06/r-fda.html

http://www.baselr.org/presentations/2010/10/BaselR_-_R_Validation_in_Life_Sciences_-_Ian_Francis_-_20101013.pdf

http://www.phusewiki.org/docs/2010/2010%20PAPERS/RG03%20Paper.pdf

http://blog.revolutionanalytics.com/2009/02/using-r-in-the-pharmaceutical-industry.html

http://blog.revolutionanalytics.com/2009/01/analyzing-clinical-trial-data-with-r.html

http://user2007.org/program/presentations/harrell.pdf

http://user2007.org/program/presentations/harrell-report.pdf

https://www.amazon.com/Applied-Surrogate-Endpoint-Evaluation-Biostatistics/dp/1482249367

https://twitter.com/hspter/status/748264778468974592

http://www.kcrcro.com/uploads/attachments/news_pdf/EPC%20November%202016%20p18-21.pdf

https://berlinrug.github.io/BerlinRUG/20151029/unfulfilled-potential-of-r-in-clinical-research/Parsadanyan_R%20in%20Clinical%20Research_fixed.pdf

http://blog.revolutionanalytics.com/life-sciences/

http://blog.revolutionanalytics.com/2015/08/a-short-introduction-to-bioconductor.html

https://www.r-bloggers.com/lewis-rendell-what-i-learnt-at-mango-solutions/

https://dataclarityconsulting.wordpress.com/2016/03/29/validated-statistical-software/

https://cran.r-project.org/doc/contrib/Olszewski-R_for_biostatisticians.pdf

Tools Info

https://github.com/rstudio/shinytest

https://github.com/rstudio/shinyloadtest

https://rviews.rstudio.com/2017/06/21/analytics-administration-for-r/

Haven enables R to read and write various data formats used by other statistical packages by wrapping the fantastic ReadStat C library written by Evan Miller. Haven is part of the tidyverse. Currently it supports:

SAS: read_sas() reads .sas7bdat + .sas7bcat files and read_xpt() reads SAS transport files (version 5 and version 8). write_sas() writes .sas7bdat files.

https://www.r-statistics.com/tag/reproducible-research/

If you use R in a production environment, you have most likely experienced that some circumstances change in ways that will make your R scripts run into trouble. Many things can go wrong; package updates, external data sources, daylight savings time, etc. There is a general increasing focus on this within the R community and words like “reproducibility”, “portability” and “unit testing” are buzzing big time. Many really neat solutions are already helping a lot: RStudio’s Packrat project, Revolution Analytic’s “snapshot” feaure, and Hadley Wickham’s testthat package to name a few. Another interesting package under development is Edwin de Jonge’s “validate” package.

https://cran.r-project.org/web/packages/ensurer/

https://4dpiecharts.com/2015/07/03/the-state-of-assertions-in-r/

Reports & Tables

There are several packages that can be used to make very nice packages:

printr xtable stargazer tables pander

pander::pandoc.table()

http://rmarkdown.rstudio.com/rtf_document_format.html

https://github.com/ns-ctru/ctru/blob/b3b30bd128af9546fa2aa1750b2a7f100f6d6fcd/vignettes/ctru-r-functions.Rmd

https://github.com/dpastoor/nonmem/tree/master/ACCP1

https://github.com/DanniYuGithub/RTF2/

https://cran.r-project.org/web/packages/ReporteRs/index.html

https://cran.r-project.org/web/packages/rtf/rtf.pdf

https://cran.r-project.org/web/packages/knitr/index.html

https://www.latex-project.org/get/

https://support.rstudio.com/hc/en-us/articles/200552056-Using-Sweave-and-knitr

https://cran.r-project.org/web/packages/gridExtra/index.html

https://cran.r-project.org/web/packages/summarytools/index.html

https://cran.r-project.org/web/packages/stargazer/vignettes/stargazer.pdf

http://biostat.mc.vanderbilt.edu/wiki/Main/Greport

http://biostat.mc.vanderbilt.edu/wiki/Main/HmiscNew

http://biostat.mc.vanderbilt.edu/wiki/pub/Main/StatReport/summary.pdf

https://davidgohel.github.io/ReporteRs/articles/word.html

https://www.r-statistics.com/2013/01/stargazer-package-for-beautiful-latex-tables-from-r-statistical-models-output/

https://cran.r-project.org/web/packages/stargazer/stargazer.pdf

https://www.jakeruss.com/cheatsheets/stargazer/

https://www.princeton.edu/~otorres/NiceOutputR.pdf

http://www.r-clinical-research.com

https://cran.r-project.org/web/packages/ascii/index.html

https://cran.r-project.org/web/packages/txtplot/index.html

https://cran.r-project.org/web/packages/textreg/index.html

https://cran.r-project.org/web/packages/broom/vignettes/broom.html

http://r4ds.had.co.nz/many-models.html

https://github.com/tidyverse/broom

https://lsru.github.io/tv_course/lecture09_broom.html

https://github.com/rstudio/cheatsheets/raw/master/rmarkdown-2.0.pdf

https://cran.r-project.org/web/packages/xtable/vignettes/xtableGallery.pdf

https://cran.r-project.org/web/packages/xtable/index.html

http://rstudio.github.io/DT/

https://www.bioconductor.org/packages/release/bioc/vignettes/biobroom/inst/doc/biobroom_vignette.html

http://varianceexplained.org/r/broom-intro/

https://www.rforge.net/doc/packages/knitr/kable.html

https://haozhu233.github.io/kableExtra/awesome_table_in_pdf.pdf

https://cran.r-project.org/web/packages/kableExtra/vignettes/awesome_table_in_html.html

https://sachsmc.github.io/knit-git-markr-guide/knitr/knit.html