Posts on Tidyverse

hardhat 0.1.0

Mon, 16 Dec 2019 00:00:00 +0000

We’re excited to announce that the first version of hardhat is now on CRAN. hardhat is a developer-focused package with the goal of making it easier to create new modeling packages, while simultaneously promoting good R modeling package standards. To accomplish this, hardhat provides tooling around preprocessing, predicting, and validating user input, along with a way to set up the structure of a new modeling package with a single function call.

library(modeldata)
library(tibble)
library(hardhat)
library(recipes)

data("biomass")
biomass <- as_tibble(biomass)

Setup

One exciting feature included with hardhat is create_modeling_package(). Built on top of usethis::create_package(), this allows you to quickly set up a new modeling package with pre-generated infrastructure in place for an S3 generic to go with your user-facing modeling function. It also includes a predict() method and other best practices outlined further in Conventions for R Modeling Packages. If you’ve never created a modeling package before, this is a great place to start so you can focus more on the implementation rather than the details around package setup.

Preprocessing

When building a model, there are often preprocessing steps that you perform on the training set before fitting. Take this biomass dataset for example. The goal is to predict the HHV for each sample, an acronym for the Higher Heating Value, defined as the amount of heat released by an object during combustion. To do this, you might use the numeric columns containing the amounts of different atomic elements that make up each sample.

training <- biomass[biomass$dataset == "Training",]
testing <- biomass[biomass$dataset == "Testing",]

training
#> # A tibble: 456 x 8
#>    sample                 dataset  carbon hydrogen oxygen nitrogen sulfur   HHV
#>    <chr>                  <chr>     <dbl>    <dbl>  <dbl>    <dbl>  <dbl> <dbl>
#>  1 Akhrot Shell           Training   49.8     5.64   42.9     0.41   0     20.0
#>  2 Alabama Oak Wood Waste Training   49.5     5.7    41.3     0.2    0     19.2
#>  3 Alder                  Training   47.8     5.8    46.2     0.11   0.02  18.3
#>  4 Alfalfa                Training   45.1     4.97   35.6     3.3    0.16  18.2
#>  5 Alfalfa Seed Straw     Training   46.8     5.4    40.7     1      0.02  18.4
#>  6 Alfalfa Stalks         Training   45.4     5.75   40.2     2.04   0.1   18.5
#>  7 Alfalfa Stems          Training   47.2     5.99   38.2     2.68   0.2   18.7
#>  8 Alfalfa Straw          Training   45.7     5.7    39.7     1.7    0.2   18.3
#>  9 Almond                 Training   48.8     5.5    40.9     0.8    0     18.6
#> 10 Almond Hull            Training   47.1     5.9    40       1.2    0.1   18.9
#> # … with 446 more rows

Depending on the model you choose, you might need to center and scale your data before passing it along to the fitting function. There are two main ways you might do this: a formula, or a recipe. As a modeling package developer, ideally you’d support both in your user-facing modeling function, like so:

my_model <- function(x, ...) {
  UseMethod("my_model")
}

my_model.formula <- function(formula, data, ...) {
  
}

my_model.recipe <- function(x, data, ...) {
  
}

Unfortunately, each have their own nuances and tricks to be aware of, which you probably don’t want to spend too much time thinking about. Ideally, you’d be able to focus on your package’s implementation, and easily be able to support a number of different user input methods. This is where hardhat can help. hardhat::mold() is a preprocessing function that knows how to preprocess formulas, prep recipes, and deal with the more basic XY input (two data frames, one holding predictors and one holding outcomes). The best part is that the output from mold() is standardized across all 3 preprocessing methods, so you always know what data structures you’ll be getting back.

rec <- recipe(HHV ~ carbon, training) %>%
  step_normalize(carbon)

processed_formula <- mold(HHV ~ scale(carbon), training)
processed_recipe <- mold(rec, training)

names(processed_formula)
#> [1] "predictors" "outcomes"   "blueprint"  "extras"
names(processed_recipe)
#> [1] "predictors" "outcomes"   "blueprint"  "extras"

predictors is a data frame of the preprocessed predictors.
outcomes is a data frame of the preprocessed outcomes.
blueprint is the best part of hardhat. It records the preprocessing activities, so that it can replay them on top of new data that needs to be preprocessed at prediction time.
extras is a data frame of any “extra” columns in your data set that aren’t considered predictors or outcomes. These might be offsets in a formula, or extra roles from a recipe.

processed_recipe$predictors
#> # A tibble: 456 x 1
#>     carbon
#>      <dbl>
#>  1  0.140 
#>  2  0.110 
#>  3 -0.0513
#>  4 -0.313 
#>  5 -0.153 
#>  6 -0.284 
#>  7 -0.114 
#>  8 -0.255 
#>  9  0.0428
#> 10 -0.120 
#> # … with 446 more rows

processed_recipe$outcomes
#> # A tibble: 456 x 1
#>      HHV
#>    <dbl>
#>  1  20.0
#>  2  19.2
#>  3  18.3
#>  4  18.2
#>  5  18.4
#>  6  18.5
#>  7  18.7
#>  8  18.3
#>  9  18.6
#> 10  18.9
#> # … with 446 more rows

Generally you won’t call mold() interactively, but will, instead, call it from your top-level modeling function as the first step to standardize and validate a user’s input.

my_model <- function(x, ...) {
  UseMethod("my_model")
}

my_model.formula <- function(formula, data, ...) {
  processed <- hardhat::mold(formula, data)
  # ... pass on to implementation
}

my_model.recipe <- function(x, data, ...) {
  processed <- hardhat::mold(x, data)
  # ... pass on to implementation
}

Predicting

Once you’ve used the preprocessed data to fit your model, you’ll probably want to make predictions on a test set. To do this, you’ll need to reapply any preprocessing that you did on the training set to the test set as well. hardhat makes this easy with hardhat::forge(). forge() takes a data frame of new predictors, as well as a blueprint that was created in the call to mold(), and reapplies the correct preprocessing for you. Again, no matter what the original preprocessing method was, the output is consistent and predictable.

forged_formula <- forge(testing, processed_formula$blueprint)
forged_recipe <- forge(testing, processed_recipe$blueprint)

names(forged_formula)
#> [1] "predictors" "outcomes"   "extras"
names(forged_recipe)
#> [1] "predictors" "outcomes"   "extras"

forged_recipe$predictors
#> # A tibble: 80 x 1
#>     carbon
#>      <dbl>
#>  1 -0.193 
#>  2 -0.490 
#>  3 -0.543 
#>  4 -0.188 
#>  5  0.0390
#>  6 -0.390 
#>  7 -0.904 
#>  8 -0.601 
#>  9 -1.84  
#> 10 -1.97  
#> # … with 70 more rows

Like mold(), forge() is not intended for interactive use. Instead, you’ll call it from your predict() method.

predict.my_model <- function(object, new_data, ...) {
  processed <- hardhat::forge(new_data, object$blueprint)
  # ... pass on to predict() implementation
}

object here is a model fit of class "my_model" that should be the result of a user calling your high level my_model() function. To enable forge() to work as shown here, you’ll need to attach and return the blueprint that is created from mold() to this model object.

forge() has powerful data type validation built in. It checks for a number of things including:

Missing predictors
Predictors with the correct name, but wrong data type
Factor predictors with “novel levels”
Factor predictors with missing levels, which can be recovered automatically

Learning more

There are 3 key vignettes for hardhat.

There is also a video of Max Kuhn speaking about hardhat at the XI Jornadas de Usuarios de R conference.

odbc 1.2.0

Mon, 02 Dec 2019 00:00:00 +0000

odbc 1.2.0 is now on CRAN! The odbc package provides a DataBase Interface (DBI) to Open DataBase Connectivity (ODBC) drivers. ODBC drivers exist for nearly all widely used databases, including SQL Server, Oracle, MySQL, PostgreSQL, SQLite and others. RStudio also provides a set of Professional ODBC Drivers free of charge for customers who have RStudio Server Pro, RStudio Connect, or Shiny Server Pro. In addition, db.rstudio.com has extensive resources on connecting to and working with databases from R. The odbc package allows you to query these databases from within R. Get the latest version with:

install.packages("odbc")

Generally you will not need to load the odbc package with library(), instead you can load the DBI package and use odbc::odbc() to reference the driver when connecting. e.g.

library(DBI)
con <- dbConnect(odbc::odbc(), "MicrosoftSQLServer", UID = "SA", PWD = "Password12")

The biggest improvements in this release are related to querying within schemas, an API for immediate execution, and a new timezone_out parameter to control the displayed time of timezones returned by the query. See the change log for the full set of changes for this release.

Querying schemas

Use DBI::Id() reference a table within a schema. This will allow odbc to handle any quoting necessary for your particular database. e.g. you can write to a table named my.iris even if there is a schema named my.

library(DBI)

con <- dbConnect(odbc::odbc(), "MicrosoftSQLServer", UID = "SA", PWD = "Password12")

my_tbl <- Id(schema = "my", table = "my.iris")
dbWriteTable(con, my_tbl, iris)

tbl <- dbReadTable(con, my_tbl)
# The quoting for "my"."my.iris" is done automatically when 
# using DBI::Id()
tbl2 <- dbGetQuery(con, 'SELECT * FROM "my"."my.iris"')
all.equal(tbl, tbl2)
#> TRUE

This feature has actually existed for a number of odbc releases, but due to driver inconsistencies was not working properly on SQL Server, which is now fixed.

Immediate/direct execution

The odbc package uses Prepared Statements to compile the query once and reuse it, allowing large or repeated queries to be more efficient. However, prepared statements can actually perform worse in some cases, such as many different small queries that are all only executed once. Because of this the odbc package now also supports direct queries by specifying immediate = TRUE.

# This will use a prepared statement
dbGetQuery("SELECT * from iris")

# This will execute the statement directly
dbGetQuery("SELECT * from iris", immediate = TRUE)

Timezone display

The odbc package has historically imported date times with a UTC timezone. This ensures that the same code will produce the same output regardless of the local time. However this can confuse users, particularly if the server timezones are stored or displayed in a non-UTC timezone. Because of this, the odbc package now supports a timezone_out parameter, which allows users to set the timezone the times should be displayed in. Setting this to your local timezone, or the timezone of the database may reduce this confusion.

library(DBI)
# Create a new connection, specifying a timezone_out of UTC (this is the default)
con_utc <- dbConnect(odbc::odbc(), "MicrosoftSQLServer", UID="SA", PWD="Password12", timezone_out = "UTC")

# Create a table with the current timestamp as a value
dbExecute(con_utc, "SELECT CURRENT_TIMESTAMP AS x INTO now")
#> [1] 1

# Read that table, the time is displayed in UTC
res_utc <- dbReadTable(con_utc, "now")
res_utc
#>                     x
#> 1 2019-11-29 15:03:59

# Create another connection, this time with the timezone in United States eastern time
con_est <- dbConnect(odbc::odbc(), "MicrosoftSQLServer", UID="SA", PWD="Password12", timezone_out = "US/Eastern")

# Read the same table again, this time the time is displayed in EST
res_est <- dbReadTable(con_est, "now")
res_est
#>                     x
#> 1 2019-11-29 10:03:59

# These two times equal the same time point, the only difference is the display
res_utc == res_est
#> Warning in check_tzones(e1, e2): 'tzone' attributes are inconsistent
#>         x
#> [1,] TRUE

# You can convert res_utc to res_est by changing the `tzone` attribute
attr(res_utc$x, "tzone") <- "US/Eastern"
res_utc
#>                     x
#> 1 2019-11-29 10:03:59

There were a number of additional features and bug fixes in this version, see the change log for details.

Acknowledgements

Thanks to Xianying Tan, James Blair and Kirill Müller who all submitted multiple pull requests with improvements, and to all the 114 GitHub contributors who have opened issues or submitted code improvements to help make this release happen!

@ajholguin, @ammarelsh, @anchal02, @andreaspano, @andrewsali, @arestrom, @aryoda, @berkorbay, @blairj09, @blmayer, @cboettig, @cdumoulin-usgs, @CerebralMastication, @cfisher5, @chrishaug, @ChristianAlvaradoAP, @cnolanminich, @colearendt, @copernican, @crossxwill, @david-cortes, @davidchall, @DavisVaughan, @detule, @dhaycraft, @dirkschumacher, @dpprdan, @duncanrellis, @edgararuiz, @elbamos, @elben10, @etienne-s, @etiennebr, @felipegerard, @foundinblank, @Freekers, @ghost, @hadley, @halpo, @happyshows, @harrismcgehee, @hiltonmbr, @hoxo-m, @iamsaini87, @its-gazza, @JarkoDubbeldam, @jasperDD, @javierluraschi, @jeroenhabets, @jimhester, @jrisi256, @jsonbecker, @jtelleria, @jtelleriar, @juliasilge, @kbzsl, @kerry-ja, @kevinushey, @khotilov, @KimmoMW, @kjaanson, @kohleth, @krlmlr, @lee170, @leungi, @madlogos, @martindut, @mateusz1981, @matthijsvanderloos, @maverix13, @maxPunck, @mbfsdatascience, @md0u80c9, @meztez, @mhsilvav, @mingwandroid, @mlaviolet, @mmastand, @move[bot], @muranyia, @nwstephens, @nzbart, @patperu, @PatWilson, @pchiacchiari-coatue, @pgensler, @pythiantech, @quartin, @r2evans, @ralsouza, @renkun-ken, @revodavid, @ronblum, @rtgdk, @s-fleck, @satvenkat, @scmck17, @sebschub, @shapenaji, @shizidushu, @shrektan, @smingerson, @stlouiso, @timabe, @totalgit74, @TTudino, @UpsideDownRide, @versipellis, @vh-d, @vpanfilov, @warnes, @washcycle, @wibeasley, and @yutannihilation

tidyverse 1.3.0

Fri, 22 Nov 2019 00:00:00 +0000

We’re thrilled to announce that tidyverse 1.3.0 is now on CRAN. The tidyverse is a set of packages that work in harmony because they share common data representations and API design. The tidyverse package is a “meta” package designed to make it easy to install and load core packages from the tidyverse in a single command. This is great for teaching and interactive use, but for package development purposes we recommend that authors import only the specific packages that they use. For a complete list of changes, please see the release notes.

You can install the latest version of tidyverse with:

install.packages("tidyverse")

And, as always, attach the package by running:

library(tidyverse)

Citing the tidyverse

The most significant update in this version is a new vignette, “Welcome to the Tidyverse”, which is a mirror of the recently-released paper of the same name in the Journal of Open Source Software. The (long) list of authors includes all members of the tidyverse organisation and its component packages, and is now the canonical way to cite tidyverse packages.

When to cite the tidyverse

If you’re in a position where you can easily cite every package that materially contributed to your analysis, you should totally do it. But it is often hard because many packages don’t have accompanying papers, and many journals still haven’t got the memo that software is an important research artifact. Alternatively, you may be publishing in a venue with very tight page constraints and you simply don’t have the room to cite every package. For a more comprehensive discussion of best practices regarding citing software, see Software Citation Principles by Arfon Smith, Daniel Katz, Kyle Niemeyer, and The FORCE11 Software Citation Working Group.¹

We generally recommend citing the tidyverse paper instead of citing individual packages. This is less work, making it more likely that people will do it, and helps concentrate the citations across the whole tidyverse into a single place, which makes it easier to show the academic merit of our work. Of course, you’re still free to cite individual tidyverse packages if you feel like they have been particularly important for your analysis, but you shouldn’t feel obliged.

How to cite the tidyverse

The R citation() function makes it easy to cite R and R packages in publications. The latest version of the tidyverse package has the citation built in:

citation("tidyverse")
#> 
#>   Wickham et al., (2019). Welcome to the tidyverse. Journal of Open
#>   Source Software, 4(43), 1686, https://doi.org/10.21105/joss.01686
#> 
#> A BibTeX entry for LaTeX users is
#> 
#>   @Article{,
#>     title = {Welcome to the {tidyverse}},
#>     author = {Hadley Wickham and Mara Averick and Jennifer Bryan and Winston Chang and Lucy D'Agostino McGowan and Romain François and Garrett Grolemund and Alex Hayes and Lionel Henry and Jim Hester and Max Kuhn and Thomas Lin Pedersen and Evan Miller and Stephan Milton Bache and Kirill Müller and Jeroen Ooms and David Robinson and Dana Paige Seidel and Vitalie Spinu and Kohske Takahashi and Davis Vaughan and Claus Wilke and Kara Woo and Hiroaki Yutani},
#>     year = {2019},
#>     journal = {Journal of Open Source Software},
#>     volume = {4},
#>     number = {43},
#>     pages = {1686},
#>     doi = {10.21105/joss.01686},
#>   }

Since the tidyverse wouldn’t be possible without R, we strongly recommend that you also cite R:

citation()
#> 
#> To cite R in publications use:
#> 
#>   R Core Team (2019). R: A language and environment for statistical
#>   computing. R Foundation for Statistical Computing, Vienna, Austria.
#>   URL https://www.R-project.org/.
#> 
#> A BibTeX entry for LaTeX users is
#> 
#>   @Manual{,
#>     title = {R: A Language and Environment for Statistical Computing},
#>     author = {{R Core Team}},
#>     organization = {R Foundation for Statistical Computing},
#>     address = {Vienna, Austria},
#>     year = {2019},
#>     url = {https://www.R-project.org/},
#>   }
#> 
#> We have invested a lot of time and effort in creating R, please cite it
#> when using it for data analysis. See also 'citation("pkgname")' for
#> citing R packages.

Acknowledgments

A huge thank you to the paper’s reviewers, Laura DeCicco and Jeff Hanson, and journal editor, Karthik Ram, who improved the quality of the paper immensely.

We’re also grateful to the 89 people who contributed to this release: @andrewheiss, @arcole, @arfon, @Armos05, @arne1921KF, @batpigandme, @BenoitLondon, @bschneidr, @cawoodjm, @christa88, @coatless, @cryptic0, @cwickham, @d8aninja, @daltonhance, @dan-reznik, @danhalligan, @DataXujing, @DavoOZ, @dchiu911, @dhslone, @Fredo-XVII, @gaborcsardi, @genewch, @grayskripko, @gvwilson, @gwd999, @hadley, @hammer, @harrismcgehee, @ijlyttle, @jennybc, @jeroen, @jflynn264, @jimhester, @jnolis, @JoeFernando, @JoFAM, @jonocarroll, @josegonzalez, @jrosen48, @jtelleria, @jzadra, @karawoo, @karthik, @kent37, @kevinushey, @kevinykuo, @krlmlr, @ljmills, @malcolmbarrett, @maptracker, @martinamorris, @martinjhnhadley, @MartinMSPedersen, @mattsgithub, @maurolepore, @mfherman, @mgsosna, @mikeyEcology, @mloop, @moodymudskipper, @msberends, @njtierney, @osmelmillan, @pgensler, @PoGibas, @psychelzh, @rkalescky, @rmcd1024, @romainfrancois, @rsheppar, @sampath2510, @Selvamjn, @SinfantiHU, @steenharsted, @stefvanbuuren, @steveharoz, @sulgik, @talban14, @thanosgatos, @tmalsburg, @toouggy, @topepo, @tsufz, @willbowditch, @XiangyunHuang, @yatharth, and @yutannihilation.

Smith AM, Katz DS, Niemeyer KE, FORCE11 Software Citation Working Group. (2016) Software Citation Principles. PeerJ Computer Science 2:e86. doi: 10.7717/peerj-cs.86. ↩︎

scales 1.1.0

Mon, 18 Nov 2019 00:00:00 +0000

We’re thrilled to announce the release of scales 1.1.0. The scales package provides much of the infrastructure that underlies ggplot2’s scales, and using it allow you to customize the transformations, breaks, and labels used by ggplot2. Get the latest version with:

install.packages("scales")

The biggest improvements in this release are related to usability and documentation:

Axis breaks and labels have a new naming scheme: functions that generate breaks from limits are called breaks_; functions that generate labels from breaks are called labels_.
All the examples for breaks and labels have been overhauled to use new demo_continuous(), demo_discrete(), and demo_log10(). These make it much easier to see how you’d use scales functions with ggplot2, so when you’re browsing the documentation, you can immediately see how the functions work with ggplot2.

library(scales)

There are also a few new breaks and labels functions:

New breaks_width() allows you to specify the distance between breaks rather than the number of them.

demo_continuous(c(0, 100), breaks = breaks_width(10))
#> scale_x_continuous(breaks = breaks_width(10))

label_number() and label_percent() do a better job of picking the default accuracy, which means that you should generally get the correct number of decimal place by default.

demo_continuous(c(0, 0.1), labels = label_number())
#> scale_x_continuous(labels = label_number())

demo_continuous(c(0, 0.1), labels = label_percent())
#> scale_x_continuous(labels = label_percent())

New label_bytes() replaces number_bytes_format() with a more convenient interface. It takes a single unit argument which can either be an SI unit (e.g. “kB”), a binary unit (e.g. “kIB”), or an automatic unit (either “auto_si” or “auto_binary”).

# default is "auto_si"
demo_continuous(c(1, 1e6), label = label_bytes())
#> scale_x_continuous(label = label_bytes())

# supply a unit if you want all labels to use the same unit
demo_continuous(c(1, 1e6), label = label_bytes("kB"))
#> scale_x_continuous(label = label_bytes("kB"))

New label_number_auto() automatically picks between label_number() and label_scientific() based on the range of the input. It should produce compact labels over a very wide range of inputs. We are considering making this the default labeller for a future version of ggplot2.
```
demo_continuous(c(1, 1e5), labels = label_number_auto())
#> scale_x_continuous(labels = label_number_auto())
```
```
demo_continuous(c(1, 1e10), labels = label_number_auto())
#> scale_x_continuous(labels = label_number_auto())
```

New label_number_si() formats numeric vectors with SI units.

demo_continuous(c(1, 1e9), label = label_number_si())
#> scale_x_continuous(label = label_number_si())

demo_log10(c(1, 1e9), breaks = log_breaks(10), labels = label_number_si())
#> scale_x_log10(breaks = log_breaks(10), labels = label_number_si())

New label_date_short() creates labels for a date axis that only show the components of the date that have changed since the previous label.

five_months <- as.POSIXct(c("2019-11-01", "2020-03-01"))
demo_datetime(five_months)
#> scale_x_datetime()

demo_datetime(five_months, labels = label_date_short())
#> scale_x_datetime(labels = label_date_short())

See the change log for a full list of changes and bug fixes in this version.

Acknowledgements

A big thanks to all the GitHub contributors who helped make this release happen! @agila5, @Anirudhsekar96, @apsalverda, @b-lev, @bhogan-mitre, @billdenney, @bjedwards, @bkkkk, @blairj09, @clauswilke, @const-ae, @davidmasp, @dpseidel, @eliocamp, @GegznaV, @hadley, @HenrikBengtsson, @hvaret, @jcheng5, @kiernann, @kuriwaki, @mikmart, @njtierney, @paleolimbot, @sada1993, @schloerke, @sflippl, @slowkow, @sobrietie, @tdawry, @teramonagi, @thomasp85, @topepo, @tungmilan, @turgeonmaxime, @wibeasley, @woodwards, and @zamorarr.

dtplyr 1.0.0

Tue, 12 Nov 2019 00:00:00 +0000

I’m very excited to announce that dtplyr 1.0.0 is now on CRAN. dtplyr provides a data.table backend for dplyr, allowing you to write dplyr code that is automatically translated to the equivalent data.table code.

dtplyr 1.0.0 gives you the speed of data.table with the syntax of dplyr, unlocking the value of data.table to every user of dplyr. Of course, if you know data.table, you can still write it directly, just as we expect SQL experts to continue to write SQL rather than having dbplyr generate it for them. Understanding these foundational tools is particularly important if you want to eke out every last drop of performance.

This version of dtplyr is a complete rewrite which allows it to generate significantly more performant translations. dtplyr now works like dbplyr, where computation only happens when requested by as.data.table(), as.data.frame() or as_tibble() (this idea can also be seen in the table.express and rqdatatable packages). Unfortunately, this rewrite breaks pretty much all existing uses of dtplyr. But frankly, the prior version of dtplyr was not very good and few people used it, so a major overhaul should break little code in the wild.

In this article, I’ll introduce you to the basic usage of dtplyr, talk about some of the performance implications, and show off some of the translations that I’m most proud of.

Usage

To use dtplyr, you must at least load dtplyr and dplyr. You might also want to load data.table so you can access the other goodies that it provides:

library(data.table)
library(dtplyr)
library(dplyr, warn.conflicts = FALSE)

Then use lazy_dt() to create a “lazy” data table that tracks the operations performed on it:

mtcars2 <- lazy_dt(mtcars)

You can use dplyr verbs with this object as if it was a data frame. But there’s a big difference behind the scenes: instead of immediately performing the operation, dtplyr just records what you’re doing so when needed it can generate a single, efficient, data.table statement. You can preview the transformation (including the generated data.table code) by printing the result:

mtcars2 %>% 
  filter(wt < 5) %>% 
  mutate(l100k = 235.21 / mpg) %>% # liters / 100 km
  group_by(cyl) %>% 
  summarise(l100k = mean(l100k))
#> Source: local data table [?? x 2]
#> Call:   `_DT1`[wt < 5][, `:=`(l100k = 235.21/mpg)][, .(l100k = mean(l100k)), 
#>     keyby = .(cyl)]
#> 
#>     cyl l100k
#>   <dbl> <dbl>
#> 1     4  9.05
#> 2     6 12.0 
#> 3     8 14.9 
#> 
#> # Use as.data.table()/as.data.frame()/as_tibble() to access results

Generally, however, you should reserve this preview for exploration and debugging, and instead use as.data.table(), as.data.frame(), or as_tibble() to indicate that you’re done writing the transformation and want to access the results:

mtcars2 %>% 
  filter(wt < 5) %>% 
  mutate(l100k = 235.21 / mpg) %>% # liters / 100 km
  group_by(cyl) %>% 
  summarise(l100k = mean(l100k)) %>% 
  as_tibble()
#> # A tibble: 3 x 2
#>     cyl l100k
#>   <dbl> <dbl>
#> 1     4  9.05
#> 2     6 12.0 
#> 3     8 14.9

Performance

How fast is dtplyr? data.table is generally faster than dplyr, but dtplyr has to do some work to perform the translation, so it’s reasonable to ask if it’s worth it. Do the benefits of using data.table outweigh the cost of the automated translation? My experimentation suggests that it is: the cost of translation is low, and independent of the size of the data. In this section, I’ll explore the performance trade-off through three lenses: translation cost, copies, and interface mismatch.

Translation cost

Each dplyr verb must do some work to convert dplyr syntax to data.table syntax. We can use the bench package to time the cost of the four-step pipeline that I used above:

bench::mark(
  translate = mtcars2 %>% 
    filter(wt < 5) %>% 
    mutate(l100k = 235.21 / mpg) %>% # liters / 100 km
    group_by(cyl) %>% 
    summarise(l100k = mean(l100k))
)
#> # A tibble: 1 x 6
#>   expression      min   median `itr/sec` mem_alloc `gc/sec`
#>   <bch:expr> <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl>
#> 1 translate     787µs    969µs     1028.      280B     26.4

Because this pipeline does not use as.data.table() or print() it only generates the data.table code, it doesn’t run it, so we’re timing the translation cost. The translation cost scales with the complexity of the pipeline, not the size of the data, so these timings will apply regardless of the size of the underlying data.

My intial experiments suggest that the translation cost is typically a few milliseconds. Since the computational cost increases with the size of the data, the translation cost becomes a smaller proportion of the total as the data size grows, suggesting the dtplyr does not impose a significant overhead on top of data.table.

Take the following example, which uses the large nycflights13 dataset. This isn’t really big enough for data.table to really shine, but it’s about as big as you can get in an R package. Here I’m going to compute the average arrival delay by destination. It takes raw dplyr about 40ms to do the work. Again, the dtplyr translation is fast, around 1ms, and then computation using data.table only takes about 20ms, almost twice as fast as dplyr.

library(nycflights13)
flights_dt <- lazy_dt(flights)

delay_by_dest <- function(df) {
  df %>%
    filter(!is.na(arr_delay)) %>% 
    group_by(dest) %>% 
    summarise(n = n(), delay = mean(arr_delay))
}

bench::mark(
  flights %>% delay_by_dest(),
  flights_dt %>% delay_by_dest(),
  flights_dt %>% delay_by_dest() %>% as_tibble(),
  check = FALSE
)
#> # A tibble: 3 x 6
#>   expression                                         min  median `itr/sec`
#>   <bch:expr>                                     <bch:t> <bch:t>     <dbl>
#> 1 flights %>% delay_by_dest()                     35.7ms    36ms      27.8
#> 2 flights_dt %>% delay_by_dest()                 671.9µs 824.6µs    1230. 
#> 3 flights_dt %>% delay_by_dest() %>% as_tibble()  18.7ms  20.2ms      48.0
#> # … with 2 more variables: mem_alloc <bch:byt>, `gc/sec` <dbl>

Copies

There is one place where dtplyr does have to add overhead: when translations would generate data.table code that modifies the input in place, like mutate(). dtplyr matches dplyr semantics (which never modifies in place), so most expressions involving mutate() must make a copy:

mtcars2 %>% 
  mutate(x = x * 2) %>% 
  show_query()
#> copy(`_DT1`)[, `:=`(x = x * 2)]

However, dtplyr never generates more than one copy (no matter how many mutates you use), and it also recognises many situations where data.table creates an implicit copy:

mtcars2 %>% 
  mutate(y = x * 2) %>% 
  mutate(z = y * 2) %>% 
  show_query()
#> copy(`_DT1`)[, `:=`(y = x * 2)][, `:=`(z = y * 2)]

mtcars2 %>% 
  filter(x == 1) %>% 
  mutate(x = x * 2) %>% 
  show_query()
#> `_DT1`[x == 1][, `:=`(x = x * 2)]

However, if you have very datasets, creating a deep copy can be expensive. dtplyr allows you to opt out by setting immutable = FALSE. This ensures that dtplyr never makes a copy:

mtcars3 <- lazy_dt(as.data.table(mtcars), immutable = FALSE)

mtcars3 %>% 
  mutate(x2 = x * 2) %>% 
  show_query()
#> `_DT3`[, `:=`(x2 = x * 2)]

Interface mismatch

The hardest overhead to measure is the cost of interface mismatch, i.e. where data.table has features that dplyr doesn’t. For example, there’s no way to express cross- or rolling-joins with dplyr, so there’s no way to generate efficient data.table code for these cases. It’s hard to estimate this cost, but it’s something that we think about when considering what features to add to dplyr next.

Translation

If you’re familiar with data.table, you might be interested to learn more about how the translation works. Here I’ll show a few things I think are particularly interesting, using show_query().

df <- data.frame(a = 1:5, b = 1:5, c = 1:5, d = 1:5)
dt <- lazy_dt(df)

Most uses of the basic dplyr verbs generate calls to [.data.table:

dt %>% select(-c, -d) %>% show_query()
#> `_DT4`[, .(a, b)]
dt %>% summarise(x = mean(x)) %>% show_query()
#> `_DT4`[, .(x = mean(x))]
dt %>% mutate(x = a + b) %>% show_query()
#> copy(`_DT4`)[, `:=`(x = a + b)]
dt %>% filter(a == 1) %>% show_query()
#> `_DT4`[a == 1]
dt %>% arrange(a, desc(b)) %>% show_query()
#> `_DT4`[order(a, desc(b))]

As do simple left and right joins:

dt2 <- lazy_dt(data.frame(a = 1, y = 1, z = 1))
dt %>% left_join(dt2, by = "a") %>% show_query()
#> `_DT5`[`_DT4`, on = .(a), allow.cartesian = TRUE]
dt %>% right_join(dt2, by = "a") %>% show_query()
#> `_DT4`[`_DT5`, on = .(a), allow.cartesian = TRUE]

Where possible, dtplyr will collapse multiple calls to [:

dt %>% 
  filter(a == 1) %>% 
  select(-a) %>% 
  show_query()
#> `_DT4`[a == 1, .(b, c, d)]

dt %>% 
  left_join(dt2, by = "a") %>% 
  select(a, b, z) %>% 
  show_query()
#> `_DT5`[`_DT4`, .(a, b, z), on = .(a), allow.cartesian = TRUE]

But note that the order is important, as a select() followed by a filter() has to generate two statements:

dt %>% 
  select(a = b) %>% 
  filter(a == 1) %>% 
  show_query()
#> `_DT4`[, .(a = b)][a == 1]

When you mix basic dplyr verbs with group_by(), dtplyr adds the keyby argument:

dt %>% 
  group_by(a) %>% 
  summarise(b = mean(b)) %>% 
  show_query()
#> `_DT4`[, .(b = mean(b)), keyby = .(a)]

And when filtering, this automatically uses .SD:

dt %>% 
  group_by(a) %>% 
  filter(b < mean(b)) %>% 
  show_query()
#> `_DT4`[, .SD[b < mean(b)], keyby = .(a)]

You can learn more in vignette("translation").

There are a couple of limitations that I hope to address in the next version of dtplyr. Currently, you can’t translate the _if variants, and there is weak support for the group_ functions. If you discover other functions that don’t work as you expect, please file an issue!.

Acknowledgements

Big thanks to the data.table community, particularly Michael Chirico, for their help educating me on the best way to translate dplyr code into performant, idiomatic, data.table code.

I’d also like to thank everyone to helped make this release happen through their contributions on GitHub: @AlanFeder, @batpigandme, @benjaminleroy, @clayphan, @ColinFay, @daranzolin, @edgararuiz, @franknarf1, @hadley, @hlynurhallgrims, @hope-data-science, @ianmcook, @jl5000, @jonthegeek, @JoshuaSturm, @lenay12, @MichaelChirico, @nlbjan1, @quantitative-technologies, @richpauloo, @S-UP, @tmastny, @TobiRoby, @tomazweiss, @torema-ed, @Vidaringa, @vlahm, @vspinu, @xiaodaigh, and @yiqinfu.

roxygen2 7.0.0

Tue, 12 Nov 2019 00:00:00 +0000

We’re exceedingly happy to announce the release of roxygen2 7.0.0. roxygen2 allows you to write specially formatted R comments that generate R documentation files (man/*.Rd) and a NAMESPACE file. roxygen2 is used by over 8,000 CRAN packages.

Install the latest version of roxygen2 with:

install.packages("roxygen2")

This a huge release containing many minor improvements and bug fixes. This blog post focusses on seven major improvements:

roxygen2 is no longer (ironically!) the worst documented package. It has a fresh new website, https://roxygen2.r-lib.org, and the vignettes have all been updated.
We’ve made a number of tweaks to the rendering of .Rd. Most importantly you no longer needed to escape % in markdown, and functions with many arguments are displayed with one argument per line. We’ve also removed a couple of old features that are no longer supported.
You can create tables and headings from markdown.
You can share text and code between your README/vignettes and your documentation with the new @includeRmd tag.
You can now document R6 classes!
A package can declare how roxygen2 should load its code, making it easier to use roxygen2 in wider variety of workflows.
roxygen2 has a documented extension mechanism so that it’s easy to implement new tags and new roclets in other packages.

Improved documentation

roxygen2 finally (!!) has a pkgdown website!

I used this as an opportunity to look at all the vignettes and make sure they are comprehensive and readable. These are now the best place to go if you want more details on any roxygen2 tag:

Of course, documentation can always be improved, so if you find something hard to follow, please file an issue!

Changes to `.Rd` output

When you run roxygen2 7.0.0 for the first time, you’ll notice a number of changes to the rendered .Rd. The two most important are:

% (the Rd comment symbol) is now automatically escaped in markdown text. That means if you previously escaped it with \%, you’ll need to remove the backslash and take it back to %.

If you forget to do this, you’ll see confusing R CMD check notes like:
- unknown macro '\item'
- unexpected section header '\description'
- unexpected END_OF_INPUT
In the “Usage” section, you’ll notice the formatting of function calls has changed. Previously, function calls were wrapped to produce the smallest number of lines, e.g.:
```
parse_package(path = ".", env = env_package(path),
  registry = default_tags(), global_options = list())
```
Now long function calls are wrapped so that each argument gets its own line:
```
parse_package(
  path = ".",
  env = env_package(path),
  registry = default_tags(),
  global_options = list()
)
```
If you prefer the old behaviour you can put the following in your DESCRIPTION:
```
Roxygen: list(old_usage = TRUE)
```

You’ll also notice a number of small improvements:

Markdown code is converted to to either \code{} or \verb{}, depending on whether it not is valid R code. For example, `foofy()` will become \code{foofy()} but `1 +` will become \verb{1 + }. This better matches the intended usage of the \code{} and \verb{} macros, and should make it easier to include arbitrary “code” snippets in documentation without causing Rd failures.
@family automatically adds () when linking to functions, and prints each link on its own line (to improve diffs).

We’ve also removed a few features to simplify the code and/or clearly advertise that certain features are no longer supported:

@S3method has been removed. It was deprecated in roxygen2 4.0.0 released 2014-05-02, over 5 years ago.
Using the wrap option will now trigger a warning, as it hasn’t worked for quite some time. Suppress the warning by remove the option from your DESCRIPTION.
@docType package will no longer add -name to the file name. If you relied on this, either switch to the new workflow, or use @name to manaully override the default file name.

Markdown improvements

There are two major improvements to roxygen2’s markdown support: you can now use markdown headers and tables.

Markdown headings can be used in top-level tags like @description, @details, and @returns. Level 1 headings create a new top-level \section{} and Level 2 headings (and below) create nested \subsection{}s:

#' @details
#' First sentence
#'
#' ## Subheading
#' Second sentence
#'
#' # New section
#' Third sentence

is translated to

\details{
  First sentence

  \subsection{Subheading}{
    Second sentence
  }
}
\section{New section}{
  Third sentence
}

Markdown tables use the GFM table syntax and are converted to \tabular{} macros. For example,

#' | foo | bar |
#' | :-- | --: |
#' | baz | bim |

is translated to

\tabular{lr}{
   foo \tab bar \cr
   baz \tab bim \cr
}

Lastly, using unsupported markdown features (like blockquotes, inline HTML, and horizontal rules) will now produce an informative message.

`@includeRmd`

@includeRmd provides a new tool that allows you to share text and code amongst .Rmd and .Rd. For example, if you have vignettes/common.Rmd, you can include it:

In documentation, with:
```
#' @includeRmd vignettes/common.Rmd
```
In other vignettes with:
```
```{r child = "common.Rmd"}
```
```

In README.Rmd, with:

```{r child = "vignettes/common.Rmd"}
```

Learn more in vignette("rd")

R6 documentation

You can now document R6 classes! The basic usage is straighforward and works similarly to documenting functions. The main difference is that methods require explicit @description and @detail tags.

#' R6 Class representing a person
#'
#' A person has a name and a hair color.
Person <- R6::R6Class("Person",
  public = list(
    #' @field name First or full name of the person.
    name = NULL,

    #' @field hair Hair color of the person.
    hair = NULL,

    #' @description
    #' Create a new person object.
    #' @param name Name.
    #' @param hair Hair color.
    #' @return A new `Person` object.
    initialize = function(name = NA, hair = NA) {
      self$name <- name
      self$hair <- hair
      self$greet()
    },

    #' @description
    #' Change hair color.
    #' @param val New hair color.
    #' @examples
    #' P <- Person("Ann", "black")
    #' P$hair
    #' P$set_hair("red")
    #' P$hair
    set_hair = function(val) {
      self$hair <- val
    },

    #' @description
    #' Say hi.
    greet = function() {
      cat(paste0("Hello, my name is ", self$name, ".\n"))
    }
  )
)

Learn more in vignette("rd").

R6 documentation is a work in progress, so please let us know if you find anything missing or confusing. If you document a package with many R6 classes, you will get many warnings about missing documentation. If you want to suppress those warnings, you can turn off R6 documetation with the r6 option, i.e. put Roxygen: list(r6 = FALSE) in your DESCRIPTION.

Code loading

roxygen2 now provides three strategies for loading your code:

load_pkgload(), the default, uses pkgload. Compared to the previous release, this now automatically recompiles your package if needed.
load_source() attaches required packages and source()s all files in R/. This is a cruder simulation of package loading than pkgload (and e.g. is unreliable if you use S4 extensively), but it does not require that the package be compiled. Try it if the default strategy (used in roxygen2 6.1.0 and above) causes you grief.
load_installed() assumes you have installed the package. This is best used as part of a bigger automated workflow.

You can override the default either by calling (e.g.) roxygenise(load_code = "source") or by setting the load option in your DESCRIPTION: Roxygen: list(load = "source").

Extending roxygen2

The process for extending roxygen2 with new tags and new roclets has been completely overhauled, and is now documented in vignette("extending"). A big thanks goes to Mikkel Meyer Andersen for starting on the vignette and motivating me to make the extension process much more pleasant.

If you’re one of the few people who have written a roxygen2 extension, sorry for breaking your code! But I genuinely believe that the improvements to the documentation, object structure, and print methods make it worth it. If you have previously made a new roclet, see the major changes in the news. Since this interface is now documented, it will not change again without warning and a deprecation cycle.

Acknowledgements

A big thanks to all 69 contributors who helped make this giant release possible! @achubaty, @adam52, @alaagalal, @andrewmarx, @batpigandme, @billdenney, @Bisaloo, @coolbutuseless, @DominiqueMakowski, @drjxf, @EmilBode, @espinielli, @Freguglia, @gaborcsardi, @gdurif, @gustavdelius, @ha0ye, @hadley, @halldc, @HenrikBengtsson, @hongyuanjia, @huangwb8, @iembry, @IndrajeetPatil, @inmybrain, @jackwasey, @JazzyPierrot, @jennybc, @jeroen, @jhchou, @jimhester, @jonthegeek, @kenahoo, @kevinrue, @kevinushey, @krlmlr, @leej3, @leepface, @lorenzwalthert, @mamueller, @maurolepore, @michaelquinn32, @mikemc, @mikldk, @mjskay, @mnazarov, @msberends, @msenn, @nealrichardson, @NikKrieger, @nteetor, @paleolimbot, @pat-s, @peterdesmet, @phargarten2, @ramiromagno, @Robinlovelace, @sfirke, @Shians, @SoerenXD, @stefanoborini, @SteveBronder, @tbates, @thalesmello, @the-knife, @thomasp85, @trestletech, @vivekbhr, and @wenjie2wang.

haven 2.2.0

Fri, 08 Nov 2019 00:00:00 +0000

We’re delighted to announce that haven 2.2.0 is now on CRAN. haven enables R to read and write various data formats used by other statistical packages by wrapping the ReadStat C library written by Evan Miller.

You can install haven from CRAN with:

install.packages("haven")

This release features big improvements thanks to the hard work of Mikko Marttila: all read_*() functions gain three new arguments that allow you to read in only part of a large file. I’ll quickly show of these features by saving out the diamonds dataset to a Stata file:

library(haven)
write_dta(ggplot2::diamonds, "diamonds.dta")

n_max and skip allow you to read in just a portion of the rows:

read_dta("diamonds.dta", n_max = 5)
#> # A tibble: 5 x 10
#>   carat         cut     color   clarity depth table price     x     y     z
#>   <dbl>   <dbl+lbl> <dbl+lbl> <dbl+lbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 0.23  5 [Ideal]       2 [E]   2 [SI2]  61.5    55   326  3.95  3.98  2.43
#> 2 0.21  4 [Premium]     2 [E]   3 [SI1]  59.8    61   326  3.89  3.84  2.31
#> 3 0.23  2 [Good]        2 [E]   5 [VS1]  56.9    65   327  4.05  4.07  2.31
#> 4 0.290 4 [Premium]     6 [I]   4 [VS2]  62.4    58   334  4.2   4.23  2.63
#> 5 0.31  2 [Good]        7 [J]   2 [SI2]  63.3    58   335  4.34  4.35  2.75
read_dta("diamonds.dta", skip = 4, n_max = 5)
#> # A tibble: 5 x 10
#>   carat          cut     color  clarity depth table price     x     y     z
#>   <dbl>    <dbl+lbl> <dbl+lbl> <dbl+lb> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1  0.31 2 [Good]         7 [J] 2 [SI2]   63.3    58   335  4.34  4.35  2.75
#> 2  0.24 3 [Very Goo…     7 [J] 6 [VVS2]  62.8    57   336  3.94  3.96  2.48
#> 3  0.24 3 [Very Goo…     6 [I] 7 [VVS1]  62.3    57   336  3.95  3.98  2.47
#> 4  0.26 3 [Very Goo…     5 [H] 3 [SI1]   61.9    55   337  4.07  4.11  2.53
#> 5  0.22 1 [Fair]         2 [E] 4 [VS2]   65.1    61   337  3.87  3.78  2.49

And col_select() allows you to read in just some of the columns, using the same syntax that you use with dplyr::select():

read_dta("diamonds.dta", col_select = c(x:z))
#> # A tibble: 53,940 x 3
#>        x     y     z
#>    <dbl> <dbl> <dbl>
#>  1  3.95  3.98  2.43
#>  2  3.89  3.84  2.31
#>  3  4.05  4.07  2.31
#>  4  4.2   4.23  2.63
#>  5  4.34  4.35  2.75
#>  6  3.94  3.96  2.48
#>  7  3.95  3.98  2.47
#>  8  4.07  4.11  2.53
#>  9  3.87  3.78  2.49
#> 10  4     4.05  2.39
#> # … with 53,930 more rows

read_dta("diamonds.dta", col_select = starts_with("c"))
#> # A tibble: 53,940 x 4
#>    carat           cut     color   clarity
#>    <dbl>     <dbl+lbl> <dbl+lbl> <dbl+lbl>
#>  1 0.23  5 [Ideal]         2 [E]  2 [SI2] 
#>  2 0.21  4 [Premium]       2 [E]  3 [SI1] 
#>  3 0.23  2 [Good]          2 [E]  5 [VS1] 
#>  4 0.290 4 [Premium]       6 [I]  4 [VS2] 
#>  5 0.31  2 [Good]          7 [J]  2 [SI2] 
#>  6 0.24  3 [Very Good]     7 [J]  6 [VVS2]
#>  7 0.24  3 [Very Good]     6 [I]  7 [VVS1]
#>  8 0.26  3 [Very Good]     5 [H]  3 [SI1] 
#>  9 0.22  1 [Fair]          2 [E]  4 [VS2] 
#> 10 0.23  3 [Very Good]     5 [H]  5 [VS1] 
#> # … with 53,930 more rows

These features allow you to read in datasets that would otherwise not fit in memory, and should substantially improve performance when you only need a few rows or columns from a large file.

This release includes a number of other bug fixes and small improvements, see the changelog for a complete list.

Acknowledgements

A big thanks to everyone who helped out with this release: @aaronrudkin, @batpigandme, @ccccfys, @courtiol, @hadley, @Hadsga, @jeremy17-Endo, @KyleHaynes, @lguangyu, @mihagazvoda, @mikmart, @MokeEire, @npaszty, @pvanheus, @RaymondBalise, @sclewis23, @shubham1637, @sigbertklinke, @steffen-stell, and @ttrodrigz.

testthat 2.3.0

Wed, 06 Nov 2019 00:00:00 +0000

We’re pumped to announce that testthat 2.3.0 is now available on CRAN! testthat makes it easy to turn your existing informal tests into formal automated tests that you can rerun quickly and easily. testthat is the most popular unit-testing package for R, and is used by over 4,000 CRAN and Bioconductor packages. You can learn more about unit testing at https://r-pkgs.org/tests.html.

(We didn’t write a blog post about testthat 2.2.0 because it only introduced a single, experimental, new feature: verify_output(). It has now matured to the point we think you should also try it out, so it’s discussed below.)

Install the latest version of testthat with:

install.packages("testthat")

This release features two main improvements:

A general overhaul of condition and error handling.
A polished verify_output() that is ready for you to use to test your print methods and error messages.

For a complete list of changes, please see the release notes.

library(testthat)

Errors

The main improvements are mostly behind the scenes: we have overhauled the handling of errors and backtraces so that you should get more informative outputs when tests error unexpectedly or fail. It’s hard to authentically demonstrate this in an RMarkdown document, but if you have an error inside a test, like this:

f <- function() g()
g <- function() h()
h <- function() stop("This is an error!")

test_that("f() works as expected", {
  expect_equal(f(), 10)
})

You’ll now get an informative backtrace that should allow you to quickly locate the source of the error:

test-error.R:6: error: f() works as expected
This is an error!
Backtrace:
 1. testthat::expect_equal(f(), 10) tests/testthat/test-error.R:6:2
 4. testthat:::f()
 5. testthat:::g() tests/testthat/test-catch.R:1:5
 6. testthat:::h() tests/testthat/test-catch.R:2:5

The previous version only showed the error message, which wasn’t terribly useful!

`verify_output()`

verify_output() is designed to test output aimed at a human, like print methods and error messages. Here you want to test that the output is useful to a human, but there’s obviously no way to do that automatically. Instead, the best you can do is to check the results with your eyeballs every time the results change. verify_output() is designed to help you do this, making it a type of visual regression test.

You’ll need to use verify_output() in concert with git. Whenever the output changes, you’ll get a test failure, but to see the change, you’ll need to use git. If the change is correct, commit it with git. If it’s incorrect, fix your code and rerun the tests. Once fixed, the git diff will disappear.

verify_output() works a little like RMarkdown: you give it some R code and it will run it, interleaving the input and output. For example, imagine we were writing some tests to check that tibbles print correctly:

library(tibble)

test_that("tibbles print usefully", {
  verify_output(test_path("test-print-dataframe.txt"), {
    df1 <- tibble(x = 1:1e6)
    print(df1)
  })
})

That test yields a test-print-dataframe.txt containing this output:

> df1 <- tibble(x = 1:1e+06)
> print(df1)
# A tibble: 1,000,000 x 1
       x
   <int>
 1     1
 2     2
 3     3
 4     4
 5     5
 6     6
 7     7
 8     8
 9     9
10    10
# ... with 999,990 more rows

Unfortunately, there’s no way for verify_output() to capture comments, so you can instead use bare strings if you want comments to appear in the output. If you start the comment with # it will be formatted as a heading:

test_that("tibbles print usefully", {
  verify_output(test_path("test-print-dataframe-2.txt"), {
    "# long tibbles"
    df1 <- tibble(x = 1:1e6)
    print(df1)
    
    "# wide tibbles"
    "not yet written"
  })
})


long tibbles
============

> df1 <- tibble(x = 1:1e+06)
> print(df1)
# A tibble: 1,000,000 x 1
       x
   <int>
 1     1
 2     2
 3     3
 4     4
 5     5
 6     6
 7     7
 8     8
 9     9
10    10
# ... with 999,990 more rows


wide tibbles
============

> # not yet written

verify_output() is automatically skipped when run on CRAN. This avoids false positives because it’s very easy to accidentally depend on the code from other packages, and failure does not imply incorrect computation, just a change in presentation. In other words, verify_output() is meant to monitor the evolution of outputs produced by your package rather than checking for regressions. In this way, it is similar to the vdiffr extension to testthat which uses the same approach to monitor the evolution of plots.

Acknoweldgements

A big thanks to everyone who helped make this version happen! @aaronrudkin, @aneudecker, @atheriel, @batpigandme, @billdenney, @cimentadaj, @CorradoLanera, @cosi1, @DavisVaughan, @dtm2451, @EmielSchmeink, @flying-sheep, @gaborcsardi, @gdurif, @hadley, @hughjonesd, @ianmcook, @jameslamb, @jcheng5, @jcubic, @jennybc, @jeroen, @jimhester, @jozefhajnala, @jpritikin, @keesh0, @kenahoo, @KrishanBhasin, @krlmlr, @lionel-, @lorenzwalthert, @maxheld83, @mcol, @MichaelChirico, @mkurdej, @MLopez-Ibanez, @moodymudskipper, @mwmclean, @Nachtfeuer, @nalzok, @nbenn, @oxr463, @patr1ckm, @richierocks, @rillig, @sebastiangonsal, @shrektan, @StevenMaude, @tdhock, @theclue, @tjbell, @topepo, @torbjorn, @trevorld, @wlandau, and @zappingseb

Tidyverse dev day 2020

Tue, 05 Nov 2019 00:00:00 +0000

We’re thrilled to announce that we’ll be holding our next tidyverse developer day on the Friday following rstudio::conf in San Francisco. The event will take place from 9:00 AM – 5:00 PM at Microsoft Reactor, which is about a mile away from the conference venue (though you’re welcome to attend even if you’re not coming to the conferece).

📆 When: 31 January, 9 am – 5 pm.
🌉 Where: Microsoft Reactor, San Francisco.
👥 Who: Anyone who would like to get better at contributing to the tidyverse!
💻 What: A day of learning and coding. We’ll provide lunch, snacks, and coffee; you’ll bring your laptop and enthusiasm. The tidyverse team will be on hand to help out and answer your questions.

Tickets 🎫

Tickets are on sale now!

Tickets cost $10; we’re charging a small fee because space is limited. All fees will be donated to a non-profit.

Want to know more? Read about the last tidy dev day.

Code of conduct

All attendees will be expected to follow the Code of Conduct.

styler 1.2.0

Mon, 04 Nov 2019 00:00:00 +0000

We are pleased to announce that styler 1.2.0 is now available on CRAN. The initial version of styler was released in December 2017. Since then, five new versions were released, and styler was improved substantially, but many improvements were related to special cases or even edge cases. These are all contained in the change log. Here, we want to focus on a few changes since version 1.0.0 that we believe are the most relevant. You can install the latest stable version from CRAN with install.packages("styler"). Let’s get started:

library(styler)

Alignment detection

styler can finally detect aligned code and keep it aligned! For example, the following code won’t be modified by styler:

call(
  some_arg = 123,
  more     = "x"
)

We’ve tried to make this as intuitive as possible, but have a look at dedicated vignette when you are dealing with more complicated calls than the one above. Note that the detection currently only works inside function calls, so styler will not recognize the below as aligned.

x  <- 2
x2 <- f(x)

and turn it into

x <- 2
x2 <- f(x)

unless you also use strict = FALSE.^[E.g. styler::style_text(..., strict = FALSE), but note that this also has other effects on styling that you might not want.]

rlang’s {{

In rlang 0.4.0 a new interpolation operator was introduced: {{ (read curly curly). Because styler < 1.2.0 understood these as regular curly braces, you ended up with

call({
  {
    x
  }
})

when styling call({{x}}), which is nonsense. Now styler yields call({{ x }}).

Addins

The Style active file addin now remembers the cursor position and the details of styling can be specified as an R option:

options(
  styler.addins_style_transformer = "styler::tidyverse_style(scope = 'spaces')"
)

You can also set the value of this option interactively with the set style addin (not persistent over sessions). For details, see help("styler_addins", "styler") and help("tidyverse_style", "styler").

The customization of the styling does not affect the command-line API (styler::style_text() and friends). We are not sure how users could best customize styling, but you can track our progress on that in r-lib/styler#319.

You can also set the environment variable save_after_styling to TRUE, if you are tired of saving the file after styling it with the addin.

Braces in function calls

tryCatch() expressions often look like this:

tryCatch(
  {
    exp(x)
  },
  error = function(x) x
)

Prior to version 1.2.0, styler would return this odd formatting:

tryCatch({
  exp(x)
},
error = function(x) x
)

Now, the line is broken before the opening curly brace in function calls, except if there is only one brace expression and it’s the last in the function call. The typical use case is testthat::test_that(...), i.e. the following code won’t be modified:

test_that("some condition holds", {
  some_code()
})

Other changes

styler depends on tibble >= 1.4.2 and runs 2x as fast as initially.
styler can style roxygen code examples in the source code of packages.
styler can style .Rnw files.
The print method for the output of style_text() returns syntax-highlighted code by default, controllable via the option styler.colored_print.vertical.

Adaption of styler

We’d like to highlight that styler integrates with various other tools you might be using:

As a git pre-commit hook. Two standard calls from the R console, and you are all set. We are convinced that this is the preferred way of using styler to ensure all your files are consistently formatted. Check out the precommit package that also implements many other useful hooks.
usethis::use_tidy_style() styles your project according to the tidyverse style guide.
knitr::knitr() and friends recognize the R code chunk option tidy = "styler" for .Rnw and .Rmd files to pretty-print code.
reprex::reprex(..., style = TRUE) to prettify reprex code before printing. To permanently use style = TRUE without specifying it every time, you can add the following line to your .Rprofile (e.g. via usethis::edit_r_profile()): options(reprex.styler = TRUE).
There are plugins for Emacs and VIM.

Outlook

We have some cool new features in the pipeline such as caching for faster styling, and making styler ignore some lines, which you can try out by installing from the respective branches. Feedback welcome.

Acknowledgments

We are grateful to all of the people who contributed not just code, but also issues and comments over the last two years:

@aaronrudkin, @aedobbyn, @ArthurPERE, @Banana1530, @batpigandme, @Bio7, @ClaytonJY, @courtiol, @crew102, @cpsievert, @dchiu911, @devSJR, @dirkschumacher, @ellessenne, @Emiller88, @fny, @hadley, @Hasnep, @igordot, @IndrajeetPatil, @jackwasey, @jcrodriguez1989, @jennybc, @jjramsey, @jkgrain @jonmcalder, @joranE, @kalibera, @katrinleinweber, @kiranmaiganji, @krivit, @krlmlr, @llrs, @lorenzwalthert, @lwjohnst86, @martin-mfg, @maurolepore, @michaelquinn32, @mine-cetinkaya-rundel, @Moohan, @msberends, @NGaffney, @nxskok, @oliverbeagley, @pat-s, @ramnathv, @raynamharris, @reddy-ia, @riccardoporreca, @rillig, @rjake, @Robinlovelace, @RMHogervorst, @rorynolan, @russHyde, @samhinshaw, @skirmer, @thalesmello, @tobiasgerstenberg, @tonytonov, @tvatter, @vnijs, @wdearden, @wlandau, @wmayner, @yech1990 and @yutannihilation.

discrim 0.0.1

Thu, 17 Oct 2019 00:00:00 +0000

The new package discrim contains parsnip bindings for additional classification models, including:

Linear discriminant analysis (LDA, simple and L2 regularized)
Regularized discriminant analysis (RDA, via Friedman (1989))
Flexible discriminant analysis (FDA) using MARS features
Naive Bayes models

The package can also be used as a template for adding new models to tidymodels without having to directly involve parsnip.

As an example, the package contains a simulated data set with two factors and two classes:

library(tidyverse)
#> ── Attaching packages ────────────────────────────────────────────────────────────────────────────────────── tidyverse 1.2.1 ──
#> ✔ tibble  2.1.3     ✔ purrr   0.3.2
#> ✔ tidyr   1.0.0     ✔ dplyr   0.8.3
#> ✔ readr   1.3.1     ✔ stringr 1.4.0
#> ✔ tibble  2.1.3     ✔ forcats 0.4.0
#> ── Conflicts ───────────────────────────────────────────────────────────────────────────────────────── tidyverse_conflicts() ──
#> ✖ dplyr::filter() masks stats::filter()
#> ✖ dplyr::lag()    masks stats::lag()
#> ✖ dplyr::select() masks MASS::select()
library(rsample)
library(discrim)
#> Loading required package: parsnip
library(earth)

data("parabolic", package = "rsample")
#> Warning in data("parabolic", package = "rsample"): data set 'parabolic' not
#> found
parabolic
#> # A tibble: 500 x 3
#>        X1     X2 class 
#>     <dbl>  <dbl> <fct> 
#>  1  3.29   1.66  Class1
#>  2  1.47   0.414 Class2
#>  3  1.66   0.791 Class2
#>  4  1.60   0.276 Class2
#>  5  2.17   3.17  Class1
#>  6  1.94   3.83  Class1
#>  7 -0.588 -0.977 Class2
#>  8 -0.951  1.40  Class1
#>  9  0.275  0.370 Class2
#> 10 -1.13  -1.14  Class1
#> # … with 490 more rows

ggplot(parabolic, aes(x = X1, y = X2)) + 
  geom_point(aes(col = class), alpha = .3) + 
  coord_equal() + 
  theme(legend.position = "top")

How would a flexible discriminant model do here? We’ll split the data then fit the model:

set.seed(115)
data_split <- initial_split(parabolic, prop = 2/3)
data_tr <- training(data_split)
data_te <- testing(data_split)

fda_mod <- discrim_flexible() %>% set_engine("earth")

fda_fit <- 
  fda_mod %>% 
  fit(class ~ X1 + X2, data = data_tr)

fda_fit 
#> parsnip model object
#> 
#> Call:
#> mda::fda(formula = formula, data = data, method = earth::earth)
#> 
#> Dimension: 1 
#> 
#> Percent Between-Group Variance Explained:
#>  v1 
#> 100 
#> 
#> Training Misclassification Error: 0.105 ( N = 334 )

Since no model tuning parameters were specified, the MARS algorithm follows its own internal method for optimizing the number of features that are included in the model. The underlying MARS model is:

summary(fda_fit$fit$fit)
#> Call: earth(x=x, y=Theta, weights=weights)
#> 
#>                  coefficients
#> (Intercept)            -0.892
#> h(X1- -1.42535)         2.780
#> h(X1- -0.907457)       -1.972
#> h(X1- -0.450335)       -0.766
#> h(X2-0.351695)         -1.634
#> h(X2-1.44562)           1.461
#> 
#> Selected 6 of 13 terms, and 2 of 2 predictors
#> Termination condition: Reached nk 21
#> Importance: X2, X1
#> Number of terms at each degree of interaction: 1 5 (additive model)
#> GCV 0.333    RSS 104    GRSq 0.669    RSq 0.688

The classification boundary, overlaid on the test set, shows a series of segmented lines:

pred_grid <- 
  expand.grid(X1 = seq(-5, 5, length = 100), X2 = seq(-5, 5, length = 100))

pred_grid <- 
  bind_cols(
    pred_grid,
    predict(fda_fit, pred_grid, type = "prob") %>% 
      select(.pred_Class1) %>% 
      setNames("fda_pred")
  )

p <-
  ggplot(data_te, aes(x = X1, y = X2)) + 
  geom_point(aes(col = class), alpha = .3) + 
  coord_equal() + 
  theme(legend.position = "top")

p + 
  geom_contour(data = pred_grid, aes(z = fda_pred), breaks = .5, col = "black")

This boundary seems pretty reasonable.

These models also work with the new tune package. To demonstrate, a regularized discriminant analysis model^[Despite the name, this type of regularization is different from the more commonly used lasso ($L_1$) or ridge ($L_2$) regression methods. Here, the covariance matrix of the predictors is regularized in different ways as described here.] will be fit to the data and optimized using a simple grid search.

We’ll use the devel version of dials:

# devtools::install_github("tidymodels/tune")
# We use the devel version of several tidymodels packages:
library(tidymodels)
#> Registered S3 method overwritten by 'xts':
#>   method     from
#>   as.zoo.xts zoo
#> ── Attaching packages ───────────────────────────────────────────────────────────────────────────────────── tidymodels 0.0.3 ──
#> ✔ broom     0.5.2          ✔ recipes   0.1.7.9001
#> ✔ dials     0.0.3.9001     ✔ yardstick 0.0.4     
#> ✔ infer     0.5.0
#> ── Conflicts ──────────────────────────────────────────────────────────────────────────────────────── tidymodels_conflicts() ──
#> ✖ scales::discard() masks purrr::discard()
#> ✖ dplyr::filter()   masks stats::filter()
#> ✖ recipes::fixed()  masks stringr::fixed()
#> ✖ dplyr::lag()      masks stats::lag()
#> ✖ dials::margin()   masks ggplot2::margin()
#> ✖ dials::offset()   masks stats::offset()
#> ✖ dplyr::select()   masks MASS::select()
#> ✖ yardstick::spec() masks readr::spec()
#> ✖ recipes::step()   masks stats::step()
library(tune)

First, we mark the parameters for tuning:

rda_mod <- 
  discrim_regularized(frac_common_cov = tune(), frac_identity = tune()) %>% 
  set_engine("klaR")

In order to tune the model, we require a grid of candidate values along with a resampling specification. We’ll also setup a yardstick object to measure the area under the ROC curve for each candidate model:

set.seed(20014)
folds <- vfold_cv(data_tr, repeats = 5)

# Use a space-filling design with 30 candidate models
candidates <- 
  rda_mod %>% 
  param_set() %>% 
  grid_max_entropy(size = 30)

roc_values <- metric_set(roc_auc)

Now we can tune the model:

rda_res <-
  tune_grid(class ~ X1 + X2,
            model = rda_mod,
            rs = folds,
            grid = candidates,
            perf = roc_values)

The resampling estimates rank the models (starting with the best) as:

auc_values <- estimate(rda_res) %>% arrange(desc(mean)) 
auc_values %>% slice(1:5)
#> # A tibble: 5 x 7
#>   frac_common_cov frac_identity .metric .estimator  mean     n std_err
#>             <dbl>         <dbl> <chr>   <chr>      <dbl> <int>   <dbl>
#> 1          0.0223         0.171 roc_auc binary     0.949    50 0.00392
#> 2          0.107          0.362 roc_auc binary     0.943    50 0.00421
#> 3          0.255          0.410 roc_auc binary     0.941    50 0.00445
#> 4          0.434          0.685 roc_auc binary     0.939    50 0.00454
#> 5          0.357          0.584 roc_auc binary     0.939    50 0.00454

Let’s plot the resampling results:

ggplot(auc_values, aes(x = frac_common_cov, y = frac_identity, size = mean)) + 
  geom_point(alpha = .5) + 
  coord_equal()

There is a wide range of parameter combinations associated with good performance here. The poor results occur mostly when the model tries to enforce a mostly LDA covariance matrix (frac_common_cov > 0.9) along with frac_identity <= 0.6. The latter parameter tries to shrink the covariance matrix towards one where the parameters are considered to be nearly independent.

The parsnip model object can be updated with the best parameter combination (frac_common_cov = 0.022 and frac_identity = 0.171). These parameter values result in a model close to a pure QDA model. The merge() function can be used to insert these values into our original parsnip object:

final_param <- 
  auc_values %>% 
  slice(1) %>% 
  select(frac_common_cov, frac_identity)

rda_mod <- 
  rda_mod %>% 
  merge(final_param) %>% 
  pull(x) %>% 
  pluck(1)

rda_mod
#> Regularized Discriminant Model Specification (classification)
#> 
#> Main Arguments:
#>   frac_common_cov = 0.0222721435129642
#>   frac_identity = 0.171292591374367
#> 
#> Computational engine: klaR

rda_fit <- 
  rda_mod %>% 
  fit(class ~ X1 + X2, data = data_tr)

To show the class boundary:

pred_grid <- 
  bind_cols(
    pred_grid,
    predict(rda_fit, pred_grid, type = "prob") %>% 
      select(.pred_Class1) %>% 
      setNames("rda_pred")
  )

p + 
  geom_contour(data = pred_grid, aes(z = fda_pred), breaks = .5, col = "black", 
               alpha = .5, lty = 2) + 
  geom_contour(data = pred_grid, aes(z = rda_pred), breaks = .5, col = "black")

This is pretty close to the true simulated boundary, which is parabolic in nature.

The test sets results are:

probs_te <- 
  predict(rda_fit, data_te, type = "prob") %>% 
  bind_cols(data_te %>% select(class))
probs_te
#> # A tibble: 166 x 3
#>    .pred_Class1 .pred_Class2 class 
#>           <dbl>        <dbl> <fct> 
#>  1        0.851     0.149    Class1
#>  2        0.205     0.795    Class2
#>  3        1.000     0.000339 Class1
#>  4        0.326     0.674    Class2
#>  5        0.377     0.623    Class2
#>  6        0.202     0.798    Class2
#>  7        0.333     0.667    Class1
#>  8        0.472     0.528    Class1
#>  9        0.979     0.0210   Class1
#> 10        0.166     0.834    Class2
#> # … with 156 more rows

roc_auc(probs_te, class, .pred_Class1)
#> # A tibble: 1 x 3
#>   .metric .estimator .estimate
#>   <chr>   <chr>          <dbl>
#> 1 roc_auc binary         0.971

Pretty good!

dials 0.0.3

Tue, 01 Oct 2019 00:00:00 +0000

A new version of dials is on CRAN. The package has contains basic frameworks for managing tuning parameters for models. It is a significant update to the package. The major change is that parameter objects are now generated by functions (as opposed to the prototype objects in the previous version). For example, to make a dials object for the number of PCA components in a model:

# previously
pca_comps <- num_comp

# now
pca_comps <- num_comp()

For numeric parameters, the range of values can be set using the first argument:

library(tidymodels)

## ── Attaching packages ──────────────────────────────────────── tidymodels 0.0.2 ──

## ✔ broom     0.5.2       ✔ purrr     0.3.2  
## ✔ dials     0.0.3       ✔ recipes   0.1.7  
## ✔ dplyr     0.8.3       ✔ rsample   0.0.5  
## ✔ ggplot2   3.2.1       ✔ tibble    2.1.3  
## ✔ infer     0.4.0.1     ✔ yardstick 0.0.4  
## ✔ parsnip   0.0.3.1

## ── Conflicts ─────────────────────────────────────────── tidymodels_conflicts() ──
## ✖ purrr::discard()  masks scales::discard()
## ✖ dplyr::filter()   masks stats::filter()
## ✖ dplyr::lag()      masks stats::lag()
## ✖ ggplot2::margin() masks dials::margin()
## ✖ dials::offset()   masks stats::offset()
## ✖ recipes::step()   masks stats::step()

num_comp()

## # Components  (quantitative)
## Range: [1, ?]

num_comp(range = c(2, 10))

## # Components  (quantitative)
## Range: [2, 10]

Sets of tuning parameters can be created and managed:

boosting_set <- param_set(list(trees(), splits = tree_depth(), min_n()))
boosting_set

## Collection of 3 parameters for tuning
## 
##      id parameter type object class
##   trees          trees    nparam[+]
##  splits     tree_depth    nparam[+]
##   min_n          min_n    nparam[+]

# modifying the parameter range:
boosting_set %>% update(trees = trees(c(100, 1000)))

## Collection of 3 parameters for tuning
## 
##      id parameter type object class
##   trees          trees    nparam[+]
##  splits     tree_depth    nparam[+]
##   min_n          min_n    nparam[+]

Note that the tree depth parameter has a user-defined identification variable. This can come in handy when there are multiple tuning parameters of the same type. For example, suppose two variables (x1 and x2) were modeled using splines. The flexibility of each grouped be represented in a parameter set:

splines <- param_set(list(x1_df = deg_free(), x2_df = deg_free()))
splines

## Collection of 2 parameters for tuning
## 
##     id parameter type object class
##  x1_df       deg_free    nparam[+]
##  x2_df       deg_free    nparam[+]

This version of dials also contains two functions for creating space-filling designs, a technique from statistical experimental design theory. The two functions are grid_max_entropy() and grid_latin_hypercube().

svm_set <- param_set(list(rbf_sigma(), cost()))
set.seed(463)
me_grid <- grid_max_entropy(svm_set, size = 20) %>% mutate(type = "max entropy")
ls_grid <- grid_latin_hypercube(svm_set, size = 20) %>% mutate(type = "latin hypercube")
rn_grid <- grid_random(svm_set, size = 20) %>% mutate(type = "random")

bind_rows(me_grid, ls_grid, rn_grid) %>% 
  ggplot(aes(x = cost, y = rbf_sigma)) + 
  geom_point() + 
  facet_wrap( ~ type) +
  scale_x_log10() + 
  scale_y_log10()  + 
  coord_fixed(ratio = 1/4)

dials will be central to the upcoming framework for optimizing tuning parameters so there is much more to come regarding this package.

devtools 2.2.1

Thu, 26 Sep 2019 00:00:00 +0000

Introduction

devtools 2.2.1 is now on CRAN!

devtools makes package development easier by providing R functions that simplify and expedite common tasks. R Packages is a book based around this workflow, which you can read online for free (1st edition, in-progress 2nd edition).

Apart from the new features mentioned below this release also contains a number of smaller changes and bug fixes. As always a complete list of all the changes is available in the package Changelog.

New features

`create()`

The create() function has been re-added to the package. This functions was previously removed in favor of usethis::create_package() but it turns out the RStudio IDE uses devtools::create() as part of its create new project dialog, so removing the function would break this functionality in older versions of RStudio. create() is however simply an alias for usethis::create_package(), so you are free to use whichever you would prefer in your own use.

`dev_sitrep()`

A new function dev_sitrep() can be used to get a “situation report” about your development setup. This is important when working on your own packages, but also to help people get up and running quickly during developer events such as the Tidyverse Dev Days. It provides a series of checks that you have the latest versions of R, RStudio, RTools and package dependencies, along with instructions on how to update them if needed.

ellipsis

The ellipsis package provides a function check_dots_used() which ensures that all arguments specified in ... have been used. Devtools uses ... to pass arguments down to base R functions such as install.packages(). If the arguments are invalid, such as mis-typing a argument name they would often be silently ignored rather than throwing an error. Using ellipsis instead causes a full error to occur in these cases, catching many more bugs when they happen. check_dots_used() is now used for any devtools function taking ....

devtools 2.2.1 introduces a new option devtools.ellipsis_action to control the behavior of ellipsis in devtools. Because there are some cases, like when a given package is already installed, that devtools does not actually use any of the arguments in ....

devtools.ellipsis_action takes one of the following arguments

rlang::abort - to emit an error if arguments are unused
rlang::warn - to emit a warning if arguments are unused
rlang::inform - to emit a message if arguments are unused
rlang::signal - to emit a message if arguments are unused

Using rlang::signal will produce no output unless the custom condition is caught, so it is the best way to retain backwards compatibility with devtools behavior prior to 2.2.0.

Acknowledgements

We are of grateful to all of the 29 people who contributed not just code, but also issues and comments for this release: @amit0thesingularity, @bbimber, @Cervangirard, @dachosen1, @DavisVaughan, @deslaur, @djnavarro, @DSLituiev, @DzLGasoline, @f527, @hadley, @hrbrmstr, @IndrajeetPatil, @jameslamb, @jennybc, @jimhester, @jrowen, @k-doering-NOAA, @mezerji1365, @MichaelChirico, @neonira, @njtierney, @p-rocha, @programgirl, @realDongWang, @RoundNose, @rstub, @tbates, and @TomKellyGenetics

tidyr 1.0.0

Fri, 13 Sep 2019 00:00:00 +0000

I’m very excited to announce the release of tidyr 1.0.0! tidyr provides a set of tools for transforming data frames to and from tidy data, where each variable is a column and each observation is a row. Tidy data is a convention for matching the semantics and structure of your data that makes using the rest of the tidyverse (and many other R packages) much easier.

Install tidyr with:

install.packages("tidyr")

As you might guess from the version number, this is a major release, and the 1.0.0 moniker indicates that I’m finally happy with the overall interface of the package. This has been a long time coming: it’s five years since the first tidyr release, nine years since the first reshape2 release, and fourteen years since the first reshape release!

This blog post summarises the four major changes to the package:

New pivot_longer() and pivot_wider() provide improved tools for reshaping, superceding spread() and gather(). The new functions are substantially more powerful, thanks to ideas from the data.table and cdata packages, and I’m confident that you’ll find them easier to use and remember than their predecessors.
New unnest_auto(), unnest_longer(), unnest_wider(), and hoist() provide new tools for rectangling, converting deeply nested lists into tidy data frames.
nest() and unnest() have been changed to match an emerging principle for the design of ... interfaces. Four new functions (pack()/unpack(), and chop()/unchop()) reveal that nesting is the combination of two simpler steps.
New expand_grid(), a variant of base::expand.grid(). This is a useful function to know about, but also serves as a good reason to discuss the important role that vctrs plays behind the scenes. You shouldn’t ever have to learn about vctrs, but it brings improvements to consistency and performance.

As well as implementing the new features, I’ve spent considerable time on the documentation, including four major new vignettes:

vignette("pivot"), vignette("rectangle"), and vignette("nest") provide detailed documentation and case studies of pivotting, rectangling, and nesting respectively.
vignette("in-packages") provides best practices for using tidyr inside another package, and detailed advice on working with multiple versions of tidyr if an interface change has affected your package.

You can see a list of all the other minor bug fixes and improvements in the release notes. I strongly recommend reading the complete release notes if you’re a package developer.

library(tidyr)
library(dplyr)

Pivoting

New pivot_longer() and pivot_wider() provide modern alternatives to spread() and gather(). They have been carefully redesigned to be easier to learn and remember, and include many new features. spread() and gather() won’t go away, but they’ve been retired which means that they’re no longer under active development.

The best place to learn about pivot_longer() and pivot_wider() is vignette("pivot"), or by watching my presentation to the Vienna R users group. Here I’ll quickly show off a few of the coolest new features:

pivot_longer() can now separate column names into multiple variables in a single step. For example, take the who dataset which has column names that look like new_{diagnosis}_{gender}{age}:

names(who)
#>  [1] "country"      "iso2"         "iso3"         "year"        
#>  [5] "new_sp_m014"  "new_sp_m1524" "new_sp_m2534" "new_sp_m3544"
#>  [9] "new_sp_m4554" "new_sp_m5564" "new_sp_m65"   "new_sp_f014" 
#> [13] "new_sp_f1524" "new_sp_f2534" "new_sp_f3544" "new_sp_f4554"
#> [17] "new_sp_f5564" "new_sp_f65"   "new_sn_m014"  "new_sn_m1524"
#> [21] "new_sn_m2534" "new_sn_m3544" "new_sn_m4554" "new_sn_m5564"
#> [25] "new_sn_m65"   "new_sn_f014"  "new_sn_f1524" "new_sn_f2534"
#> [29] "new_sn_f3544" "new_sn_f4554" "new_sn_f5564" "new_sn_f65"  
#> [33] "new_ep_m014"  "new_ep_m1524" "new_ep_m2534" "new_ep_m3544"
#> [37] "new_ep_m4554" "new_ep_m5564" "new_ep_m65"   "new_ep_f014" 
#> [41] "new_ep_f1524" "new_ep_f2534" "new_ep_f3544" "new_ep_f4554"
#> [45] "new_ep_f5564" "new_ep_f65"   "newrel_m014"  "newrel_m1524"
#> [49] "newrel_m2534" "newrel_m3544" "newrel_m4554" "newrel_m5564"
#> [53] "newrel_m65"   "newrel_f014"  "newrel_f1524" "newrel_f2534"
#> [57] "newrel_f3544" "newrel_f4554" "newrel_f5564" "newrel_f65"

You can now tease apart the variable names in a single step (i.e. without separate())) by supplying a vector of variable names to names_to and a regular expression to names_pattern (simpler cases might only need names_sep):

who %>% pivot_longer(
  cols = new_sp_m014:newrel_f65,
  names_to = c("diagnosis", "gender", "age"), 
  names_pattern = "new_?(.*)_(.)(.*)",
  values_to = "count",
  values_drop_na = TRUE
)
#> # A tibble: 76,046 x 8
#>   country     iso2  iso3   year diagnosis gender age   count
#>   <chr>       <chr> <chr> <int> <chr>     <chr>  <chr> <int>
#> 1 Afghanistan AF    AFG    1997 sp        m      014       0
#> 2 Afghanistan AF    AFG    1997 sp        m      1524     10
#> 3 Afghanistan AF    AFG    1997 sp        m      2534      6
#> 4 Afghanistan AF    AFG    1997 sp        m      3544      3
#> 5 Afghanistan AF    AFG    1997 sp        m      4554      5
#> 6 Afghanistan AF    AFG    1997 sp        m      5564      2
#> # … with 7.604e+04 more rows

pivot_longer() can now work with rows that contain multiple observations (this feature was inspired by data.table’s dcast() method). For example, take the base anscombe dataset. Each row consists of four pairs of x and y measurements:

head(anscombe)
#>   x1 x2 x3 x4   y1   y2    y3   y4
#> 1 10 10 10  8 8.04 9.14  7.46 6.58
#> 2  8  8  8  8 6.95 8.14  6.77 5.76
#> 3 13 13 13  8 7.58 8.74 12.74 7.71
#> 4  9  9  9  8 8.81 8.77  7.11 8.84
#> 5 11 11 11  8 8.33 9.26  7.81 8.47
#> 6 14 14 14  8 9.96 8.10  8.84 7.04

We can now tidy this in a single step by using the special .value variable name:

anscombe %>% 
  pivot_longer(
    everything(),
    names_to = c(".value", "set"),
    names_pattern = "(.)(.)"
  ) %>% 
  as_tibble()
#> # A tibble: 44 x 3
#>   set       x     y
#>   <chr> <dbl> <dbl>
#> 1 1        10  8.04
#> 2 2        10  9.14
#> 3 3        10  7.46
#> 4 4         8  6.58
#> 5 1         8  6.95
#> 6 2         8  8.14
#> # … with 38 more rows

pivot_wider() can now do simple aggregations (reshape2::dcast() fans rejoice!). For example, take the base warpbreaks dataset (converted to a tibble to print more compactly):

warpbreaks <- warpbreaks %>% as_tibble() %>% select(wool, tension, breaks)
warpbreaks
#> # A tibble: 54 x 3
#>   wool  tension breaks
#>   <fct> <fct>    <dbl>
#> 1 A     L           26
#> 2 A     L           30
#> 3 A     L           54
#> 4 A     L           25
#> 5 A     L           70
#> 6 A     L           52
#> # … with 48 more rows

This is a designed experiment with nine replicates for every combination of wool (A and B) and tension (L, M, H). If we attempt to pivot the levels of wool into the columns, we get a warning and the output contains list-columns:

warpbreaks %>% pivot_wider(names_from = wool, values_from = breaks)
#> Warning: Values in `breaks` are not uniquely identified; output will contain list-cols.
#> * Use `values_fn = list(breaks = list)` to suppress this warning.
#> * Use `values_fn = list(breaks = length)` to identify where the duplicates arise
#> * Use `values_fn = list(breaks = summary_fun)` to summarise duplicates
#> # A tibble: 3 x 3
#>   tension           A           B
#>   <fct>   <list<dbl>> <list<dbl>>
#> 1 L               [9]         [9]
#> 2 M               [9]         [9]
#> 3 H               [9]         [9]

You can now summarise the duplicates with the values_fn argument:

warpbreaks %>% 
  pivot_wider(
    names_from = wool, 
    values_from = breaks,
    values_fn = list(breaks = mean)
  )
#> # A tibble: 3 x 3
#>   tension     A     B
#>   <fct>   <dbl> <dbl>
#> 1 L        44.6  28.2
#> 2 M        24    28.8
#> 3 H        24.6  18.8

Learn the full details and see many more examples in vignette("pivot").

Rectangling

Rectangling is the art and craft of taking a deeply nested list (often sourced from wild-caught JSON or XML) and taming it into a tidy data set of rows and columns. tidyr 1.0.0 provides four new functions to aid rectangling:

unnest_longer() takes each element of a list-column and makes a new row.
unnest_wider() takes each element of a list-column and makes a new column.
unnest_auto() uses some heuristics to guess whether you want unnest_longer() or unnest_wider().
hoist() is similar to unnest_wider() but only plucks out selected components, and can reach down multiple levels.

To see them in action, take this small sample from repurrrsive::got_chars. It contains data about three characters from the Game of Thrones:

characters <- list(
  list(
    name = "Theon Greyjoy",
    aliases = c("Prince of Fools", "Theon Turncloak", "Theon Kinslayer"),
    alive = TRUE
  ),
  list(
    name = "Tyrion Lannister",
    aliases = c("The Imp", "Halfman", "Giant of Lannister"),
    alive = TRUE
  ),
  list(
    name = "Arys Oakheart",
    alive = FALSE
  )
)

To work with the new tidyr rectangling tools, we first put the list into a data frame, creating a list-column:

got <- tibble(character = characters)
got
#> # A tibble: 3 x 1
#>   character       
#>   <list>          
#> 1 <named list [3]>
#> 2 <named list [3]>
#> 3 <named list [2]>

We can then use unnest_wider() to make each element of that list into a column:

got %>% 
  unnest_wider(character)
#> # A tibble: 3 x 3
#>   name             aliases   alive
#>   <chr>            <list>    <lgl>
#> 1 Theon Greyjoy    <chr [3]> TRUE 
#> 2 Tyrion Lannister <chr [3]> TRUE 
#> 3 Arys Oakheart    <???>     FALSE

Followed by unnest_longer() to turn each alias into its own row:

got %>% 
  unnest_wider(character) %>% 
  unnest_longer(aliases)
#> # A tibble: 7 x 3
#>   name             aliases            alive
#>   <chr>            <chr>              <lgl>
#> 1 Theon Greyjoy    Prince of Fools    TRUE 
#> 2 Theon Greyjoy    Theon Turncloak    TRUE 
#> 3 Theon Greyjoy    Theon Kinslayer    TRUE 
#> 4 Tyrion Lannister The Imp            TRUE 
#> 5 Tyrion Lannister Halfman            TRUE 
#> 6 Tyrion Lannister Giant of Lannister TRUE 
#> 7 Arys Oakheart    <NA>               FALSE

Even more conveniently, you can use unnest_auto() to guess which direction a list column should be unnested in. Here it yields the same results as above, and the messages tell you why:

got %>% 
  unnest_auto(character) %>% 
  unnest_auto(aliases)
#> Using `unnest_wider(character)`; elements have 2 names in common
#> Using `unnest_longer(aliases)`; no element has names

Alternatively, you can use hoist() to reach deeply into a data structure and put out just the pieces you need:

got %>% hoist(character, 
  name = "name",
  alias = list("aliases", 1),
  alive = "alive"
)
#> # A tibble: 3 x 4
#>   name             alias           alive character       
#>   <chr>            <chr>           <lgl> <list>          
#> 1 Theon Greyjoy    Prince of Fools TRUE  <named list [1]>
#> 2 Tyrion Lannister The Imp         TRUE  <named list [1]>
#> 3 Arys Oakheart    <NA>            FALSE <named list [0]>

This syntax provides a more approachable alternative to using purrr::map() inside dplyr::mutate(), as we’d previously recommended:

got %>% mutate(
  name = purrr::map_chr(character, "name"),
  alias = purrr::map_chr(character, list("aliases", 1), .default = NA),
  alive = purrr::map_lgl(character, "alive")
)

Learn more in vignette("rectangle").

Nesting

nest() and unnest() have been updated with new interfaces that are more closely aligned to evolving tidyverse conventions. The biggest change is to their operation with multiple columns:

# old
df %>% nest(x, y, z)
# new
df %>% nest(data = c(x, y, z))

# old
df %>% unnest(x, y, z)
# new
df %>% unnest(c(x, y, z))

I’ve done my best to ensure that common uses of nest() and unnest() will continue to work, generating an informative warning telling you precisely how you need to update your code. If this doesn’t work, you can use nest_legacy() or unnest_legacy() to access the previous interfaces; see vignette("in-packages") for more advice on managing this transition.

Behind the scenes, I discovered that nesting (and unnesting) can be decomposed into the combination of two simpler operations:

pack() and unpack() multiple columns into and out of a data frame column.
chop() and unchop() chop up rows into and out of list-columns. It’s a bit like an explicit form of dplyr::group_by().

This is primarily of internal interest, but it considerably simplifies the implementation of nest(), and you may occasionally find the underlying functions useful when working with exotic data structures.

`expand_grid()`

expand_grid() completes the existing family of expand(), nesting(), and crossing() with a low-level function that works with vectors:

expand_grid(
  x = 1:3,
  y = letters[1:3],
  z = LETTERS[1:3]
)
#> # A tibble: 27 x 3
#>       x y     z    
#>   <int> <chr> <chr>
#> 1     1 a     A    
#> 2     1 a     B    
#> 3     1 a     C    
#> 4     1 b     A    
#> 5     1 b     B    
#> 6     1 b     C    
#> # … with 21 more rows

Compared to the existing base function expand.grid(), expand_grid():

Varies the first element most slowly (not most quickly).
Never converts strings to factors and doesn’t add any additional attributes.
Returns a tibble, not a data frame.
Can expand data frames.

The last feature is quite powerful, as it allows you to generate partial grids:

students <- tribble(
  ~ school, ~ student,
  "A",      "John",
  "A",      "Mary",
  "A",      "Susan",
  "B",      "John"
)
expand_grid(students, semester = 1:2)
#> # A tibble: 8 x 3
#>   school student semester
#>   <chr>  <chr>      <int>
#> 1 A      John           1
#> 2 A      John           2
#> 3 A      Mary           1
#> 4 A      Mary           2
#> 5 A      Susan          1
#> 6 A      Susan          2
#> 7 B      John           1
#> 8 B      John           2

This is made possible by the vctrs package. vctrs is primarily of interest to package developers but I want to talk about it briefly here because I’ve been having a lot of fun working with it. It’s hard to concisely describe vctrs, but one aspect is carefully defining what a “vector” is, and providing a set of useful functions that work on all types of vctrs, without any special cases. One interesting finding is that thinking of a data frame as a vector of rows (not columns, as R usually does) is suprisingly useful, and something you can expect to see in more places in the tidyverse in the future.

Note that when data frame inputs are unnamed, they’re automatically unpacked into individual columns in the output. It’s also possible to create a column that is itself a data frame, a df-column, if you name it:

expand_grid(student = students, semester = 1:2)
#> # A tibble: 8 x 2
#>   student$school $student semester
#>   <chr>          <chr>       <int>
#> 1 A              John            1
#> 2 A              John            2
#> 3 A              Mary            1
#> 4 A              Mary            2
#> 5 A              Susan           1
#> 6 A              Susan           2
#> 7 B              John            1
#> 8 B              John            2

Df-columns aren’t particularly useful yet, but they provide powerful building blocks for future extensions. For example, we expect a future version of dplyr will support df-columns as a way for mutate() and summarise() to create multiple new columns from a single function call.

Thanks

A big thanks to all 95 people who help to make this release possible! @abiyug, @AdvikS, @ahcyip, @alexpghayes, @aneisse, @apreshill, @atusy, @ax42, @banfai, @billdenney, @brentthorne, @brunj7, @coolbutuseless, @courtiol, @cwickham, @DanielReedOcean, @DavisVaughan, @dewittpe, @donboyd5, @earowang, @eElor, @enricoferrero, @fresques, @garrettgman, @gederajeg, @georgevbsantiago, @giocomai, @gireeshkbogu, @gorkang, @ha0ye, @hadley, @hplieninger, @iago-pssjd, @IndrajeetPatil, @jackdolgin, @japhir, @jayhesselberth, @jennybc, @jeroenjanssens, @jestarr, @jgendrinal, @Jim89, @jl5000, @jmcastagnetto, @justasmundeikis, @Kaz272, @KenatRSF, @koncina, @krlmlr, @kuriwaki, @lazappi, @leeper, @lionel-, @markdly, @martinjhnhadley, @MatthieuStigler, @mattwarkentin, @mehrgoltiv, @meriops, @mikemc, @mikmart, @mine-cetinkaya-rundel, @mitchelloharawild, @mixolydianpink, @mkapplebee, @moodymudskipper, @mpaulacaldas, @Myfanwy, @ogorodriguez, @onesandzeroes, @Overlytic, @paleolimbot, @PMassicotte, @psychometrician, @Rekyt, @romainfrancois, @romatik, @rubenarslan, @SchollJ, @seabbs, @sethmund, @sfirke, @SimonCoulombe, @Stephonomon, @stufield, @tdjames1, @thierrygosselin, @tklebel, @tmastny, @trannhatanh89, @TrentLobdell, @wcmbishop, @wulixin, @yutannihilation, and @zeehio.

Multi Process Task Queue in 100 Lines of R Code

Mon, 09 Sep 2019 00:00:00 +0000

Introduction

This post is a demo of callr::r_session, a persistent R session you can use to run R code asynchronously. I set out to build a task queue, which runs tasks in subprocesses, concurrently, in a mere 100 lines of R code.

Here is a short teaser for how the queue will work. task_q$new() creates a new R6 object, which represents the queue. Its push() method adds a task, which is a function and its arguments, similar to callr::r(). The pop() method gets the results of the first task that has finished. pop() has a timeout argument, which lets you wait for a task to finish, if all pushed tasks are still running. It returns NULL if no task has finished before the timeout was over. The timeout can be 0 or Inf, meaning no wait at all, or wait indefinitely. The default timeout is 0.

q <- task_q$new()
q$push(function() { Sys.getpid() })
q$push(function() { Sys.sleep(.5); Sys.getpid() })
q$pop()
#> NULL

This pop() call returned NULL, as none of the tasks are done yet. Even though running Sys.getpid() is fast, the worker processes also need 200-500 ms time start up, when the queue is created. If you are willing to wait a bit, at least one task should be done in less than half a second (500 ms), but usually not the second one yet:

q$pop(500)$result
#> [1] 18383
q$pop()
#> NULL

The poll() method checks for finished tasks without removing their results from the queue. It also has a timeout parameter, which works the same way as pop()’s timeout. poll() returns the identifiers of all tasks that are done.

q$poll(Inf)
#> [1] ".2"
q$pop()$result
#> [1] 18385

If the queue is empty, i.e. no tasks are running and no tasks are waiting, then pop() always returns NULL, immediately, because there is nothing to wait for:

q$pop()
#> NULL

API design

The task queue will be an R6 class, with push(), pop(), poll() methods like above, and some other query methods:

task_q <- R6::R6Class(
  "task_q",
  public = list(
    initialize      = function(num_workers = 4L) { },
    get_num_waiting = function() { },
    get_num_running = function() { },
    get_num_done    = function() { },
    is_idle         = function() { },
    list_tasks      = function() { },

    push = function(fun, args = list(), id = NULL) { },
    poll = function(timeout = 0) { },
    pop = function(timeout = 0) { }
  )
)

initialize() has an argument to set the number of workers. The size of the worker pool remains fixed for the lifetime of the queue.

The get_num_*() methods return the number of waiting, running and completed tasks. get_num_done() includes tasks that haven’t been pop()-d yet. Once a task is pop()-d, it is removed completely from the queue.

is_idle() returns TRUE if the queue does not have any tasks (in any state). list_tasks() returns a data frame (tibble) with data about the tasks. This is especially useful for debugging.

push() adds a task to the queue. poll() returns the ids of all tasks that are done. pop() returns the result of the oldest task that is done.

Data structure

Before writing the methods, I’ll design the data structure that will store all information about the tasks and the worker processes. The standard data structure for a list of records is a data frame in R.

A logical choice would be to have two data frames, one for the tasks, and another one for the workers. Then we could assign unique identifiers to both tasks and processes and cross-reference them in the two data frames, to mark which task a worker is running, and which worker a task is running on.

I’ll go a step further here, and store both the tasks and the workers in the same data frame. This will simplify the implementation considerably. The queue will create a dummy idle task for each worker. Each worker will (pretend to) run its dummy task if there are no other, real tasks waiting in the queue. So the task list will always contain at least as many tasks as the number of workers in the queue.

An example for a task data frame:

q$list_tasks()
#> # A tibble: 9 x 7
#>   id      idle  state   fun    args       worker     result
#>   <chr>   <lgl> <chr>   <list> <list>     <list>     <list>
#> 1 .11     FALSE running <fn>   <list [1]> <r_sessin> <NULL>
#> 2 .12     FALSE running <fn>   <list [1]> <r_sessin> <NULL>
#> 3 .13     FALSE running <fn>   <list [1]> <r_sessin> <NULL>
#> 4 .14     FALSE running <fn>   <list [1]> <r_sessin> <NULL>
#> 5 .15     FALSE waiting <fn>   <list [1]> <NULL>     <NULL>
#> 6 .idle-1 TRUE  waiting <NULL> <NULL>     <NULL>     <NULL>
#> 7 .idle-2 TRUE  waiting <NULL> <NULL>     <NULL>     <NULL>
#> 8 .idle-3 TRUE  waiting <NULL> <NULL>     <NULL>     <NULL>
#> 9 .idle-4 TRUE  waiting <NULL> <NULL>     <NULL>     <NULL>

The columns are:

id: a character id, which can be user-supplied or auto-assigned (if the user did not supply it). This is useful to identify tasks.
idle: a logical flag, whether this is a dummy idle task or not.
state: current state of the task. More about this shortly.
fun: the function the task needs to run. This is a list column.
args: arguments to pass to the function. This is a list itself, so the column is a list column.
worker: the callr::r_session object, the R session that is running the task, or NULL if the task is not running.
result: another list column, the result of the run, if the task is already done, NULL otherwise.

The possible task states are: waiting, running, ready and done. The first two are not very surprising. The distinction between the last two is somewhat technical. A task is ready if the background R session has finished running it. The queue hasn’t read out its result yet, and the R session is still assigned to it. (I.e. the task’s worker column is not NULL.) A task is done if the queue has already read out the result of the function call, and has reassigned the R session to another task, so its worker column is NULL.

Tasks that are running and ready always have an R session assigned to them. Since all R sessions are always assigned to tasks (dummy idle tasks, if there is nothing else), this means the that the sum of the running and ready tasks always equals the number of workers.

The idle tasks are somewhat special, because they are never done. If an idle task is ready and its worker is reassigned, it will be waiting again. They are also almost never in the running state. When the queue assigns a worker to an idle task, the task will immediately go into the ready state, since the queue is immediately allowed to re-assign the worker, should a real task be waiting.

However, when a worker is starting up, its idle task is running, until their background R process has started up. After this initial running state the idle tasks are always either waiting or ready.

Implementation

I am ready to start the implementation now. I’ll focus on the individual methods here, and show the complete code of the R6 class at the end. Let’s start with the internal data. tasks contains the task data frame, initialize() will create it. next_id and get_next_id will provide us unique task ids. I prefix these with a dot, to increase the probability that they won’t interfere with user supplied task ids. So they’ll be ".1", ".2", etc.

  private = list(
    tasks = NULL,
    next_id = 1L,
    get_next_id = function() {
      id <- private$next_id
      private$next_id <- id + 1L
      paste0(".", id)
    }
  )

The initialize() method will just defer the work to a private method.

    initialize = function(num_workers = 4L) {
      private$start_workers(num_workers)
      invisible(self)
    }

The private start_workers() method that actually starts the workers and creates the tasks data frame:

    start_workers = function(num_workers) {
      private$tasks <- tibble::tibble(
        id = character(), idle = logical(),
        state = c("waiting", "running", "ready", "done")[0],
        fun = list(), args = list(), worker = list(), result = list())
      for (i in seq_len(num_workers)) {
        rs <- callr::r_session$new(wait = FALSE)
        private$tasks <- tibble::add_row(private$tasks,
          id = paste0(".idle-", i), idle = TRUE, state = "running",
          fun = list(NULL), args = list(NULL), worker = list(rs),
          result = list(NULL))
      }
    }

The starting values of the empty task data frame are mostly straightforward. If you are wondering about the indexing with zero here, it is a simple way to list all possible task states in the code, in one place, as a note for the code reader.

callr::r_session$new() starts a background R process. The wait = FALSE argument tells callr not to wait until the process is ready to run R code. This way the R processes start up in parallel, which is worth the trouble of making our dummy tasks a bit more complicated. The idle tasks are named .idle-*. After initialization, the workers are started, and the running idle tasks are added to the data frame.

The query methods are next:

    list_tasks = function() private$tasks,
    get_num_waiting = function()
      sum(!private$tasks$idle & private$tasks$state == "waiting"),
    get_num_running = function() sum(private$tasks$state == "running"),
    get_num_done = function() sum(private$tasks$state == "done"),
    is_idle = function() sum(!private$tasks$idle) == 0

list_tasks() will just return the task data frame, for simplicity. If not all data is needed, the get_num_*() functions are simpler. For the first two, we need to exclude the dummy idle tasks, because they can be in the waiting and running state as well. They cannot be in the done state.

We still need to write the push(), pop() and poll() public methods. As the reader might suspect, these are more involved. Let’s start with push().

    push = function(fun, args = list(), id = NULL) {
      if (is.null(id)) id <- private$get_next_id()
      if (id %in% private$tasks$id) stop("Duplicate task id")
      before <- which(private$tasks$idle)[1]
      private$tasks <- tibble::add_row(private$tasks, .before = before,
        id = id, idle = FALSE, state = "waiting", fun = list(fun),
        args = list(args), worker = list(NULL), result = list(NULL))
      private$schedule()
      invisible(id)
    }

The queue needs to run the tasks in the same order as they were added. The data frame will keep the correct order, with the additional tweak that the idle tasks are always at the end. Indeed, these should only run if there is no other task waiting. So push() adds the new task right before the idle tasks.

The schedule() private method is the core of the queue. It starts the tasks on the selected background R workers, and it also reads out the results after they are done. I.e. it performs the waiting to running and ready to done task state transitions. We will show it later.

push() returns the id of the newly added task, this can be helpful to follow the task and match it to the results of a pop() call.

pop() uses poll() to get a list of tasks that are done, and returns the result of the oldest one, which is always the first, thanks to the ordering of the task data frame.

    pop = function(timeout = 0) {
      if (is.na(done <- self$poll(timeout)[1])) return(NULL)
      row <- match(done, private$tasks$id)
      result <- private$tasks$result[[row]]
      private$tasks <- private$tasks[-row, ]
      c(result, list(task_id = done))
    }

If no task is done, then it returns NULL. The returned task is removed from the task data frame, and from the queue in general, for good. pop() adds the id of the task to the returned result as task_id, for easier matching of tasks to results.

poll() is the only method that checks on the running workers. This is important to remember, and unfortunately easy to forget. If the user does not call poll(), either directly or via pop(), the state of a running task cannot change, even if the background R session itself has finished. In other words, one cannot check the status of the tasks by listing the task data frame with list_tasks() periodically. This will never change if poll() is not called.

I start with an initial version of poll(), which will need changes later, but this is hopefully easier to understand first:

    poll = function(timeout = 0) {
      as_ms <- function(x) if (x == Inf) -1L else as.integer(x)
      topoll <- which(private$tasks$state == "running")
      conns <- lapply(
        private$tasks$worker[topoll],
        function(x) x$get_poll_connection())
      pr <- processx::poll(conns, as_ms(timeout))
      private$tasks$state[topoll][pr == "ready"] <- "ready"
      private$schedule()
      private$tasks$id[private$tasks$state == "done"]
    }

We only need to check on tasks that are running. poll() uses the processx::poll() function that can wait on several callr::r_sessions at once. More precisely, I extract the poll connections of the r_session objects and call processx::poll() on these. An r_session may have multiple pollable connections, one for its standard output stream, one for its standard error stream. These are not used by default in r_sessions, and I only want to check on the poll connection, which signals if the R session has finished with the computation (or encountered an error while working on it). processx::poll() returns a list of character vectors, one entry for each (running) task. This is "ready" if the session is ready with the task. (Or it is "silent" if it is not ready, or "timeout" if the time limit expired and no workers are ready.)

All tasks that returned "ready" are indeed set to the ready state. After this poll() calls schedule() to read out the results of the ready tasks and reassign their workers to waiting ones.

poll() returns the ids of all tasks that are done.

This version of poll() has a small issue when the R sessions are starting up: it might return without any results, before the specified timeout value is over. At startup the idle tasks are running, and they are polled by processx::poll(). If any of the R sessions start up before the timeout is over, processx::poll() returns with "ready" for them. But schedule() cannot mark these tasks as done, because they are idle tasks, they’ll be waiting, and with no task done, poll() will return an empty vector. This is problematic, because poll() promises to either wait until the specified timeout or return a task that is done. So we need to wrap the simplified poll() into a loop, and keep calling processx::poll() until either the timeout expires or a task is done. The final poll() looks like this:

    poll = function(timeout = 0) {
      limit <- Sys.time() + timeout
      as_ms <- function(x) if (x == Inf) -1L else as.integer(x)
      repeat{
        topoll <- which(private$tasks$state == "running")
        conns <- lapply(
          private$tasks$worker[topoll],
          function(x) x$get_poll_connection())
        pr <- processx::poll(conns, as_ms(timeout))
        private$tasks$state[topoll][pr == "ready"] <- "ready"
        private$schedule()
        ret <- private$tasks$id[private$tasks$state == "done"]
        if (is.finite(timeout)) timeout <- limit - Sys.time()
        if (length(ret) || timeout < 0) break;
      }
      ret
    }

Only the private schedule() method is missing now:

    schedule = function() {
      ready <- which(private$tasks$state == "ready")
      if (!length(ready)) return()
      rss <- private$tasks$worker[ready]

      private$tasks$result[ready] <- lapply(rss, function(x) x$read())
      private$tasks$worker[ready] <- replicate(length(ready), NULL)
      private$tasks$state[ready] <-
        ifelse(private$tasks$idle[ready], "waiting", "done")

      waiting <- which(private$tasks$state == "waiting")[1:length(ready)]
      private$tasks$worker[waiting] <- rss
      private$tasks$state[waiting] <-
        ifelse(private$tasks$idle[waiting], "ready", "running")
      lapply(waiting, function(i) {
        if (! private$tasks$idle[i]) {
          private$tasks$worker[[i]]$call(private$tasks$fun[[i]],
                                         private$tasks$args[[i]])
        }
      })
    }

schedule()’s job is to perform the ready to done and the waiting to running state transitions. The first involves reading out the results of the ready tasks and the second involves starting new computation on the workers.

For every ready task, schedule() perform three steps:

Reads out and stores its result. (It can do this for the idle tasks as well, for these r_session$read() will return NULL.)
Removes its worker, i.e. sets it to NULL.
Updates its state to done. (Or to waiting if it is an idle task.)

Then it deals with the waiting tasks, but not more than the number of ready tasks the queue had. For these waiting tasks schedule() performs three steps:

Assigns a just removed worker to it.
Sets state to running. (Or to ready for idle tasks.)
Calls fun(args) in the background session.

When selecting the waiting tasks to run, the ordering of the task table makes sure that the oldest task is selected first, and that idle tasks are only selected if there is nothing else to run. The idle tasks make sure that schedule() always has at least as many waiting tasks as ready.

It is possible that schedule() first sets an idle task to waiting and then selects it and (re-)assigns a worker to it. This is perfectly fine.

Try it out

As a simple example, we add a bunch of fake tasks to a queue, and then run a simple event loop to completion. (To run this code, first you need to run the complete code at the end of the post.)

q <- task_q$new()
for (i in 1:10) {
  q$push(function(i) { Sys.sleep(runif(1)); paste(i, "done") }, list(i = i))
}

This is how the queue looks after adding all these tasks:

q$list_tasks()

#> # A tibble: 14 x 7
#>    id      idle  state   fun    args             worker     result
#>    <chr>   <lgl> <chr>   <list> <list>           <list>     <list>
#>  1 .1      FALSE waiting <fn>   <named list [1]> <NULL>     <NULL>
#>  2 .2      FALSE waiting <fn>   <named list [1]> <NULL>     <NULL>
#>  3 .3      FALSE waiting <fn>   <named list [1]> <NULL>     <NULL>
#>  4 .4      FALSE waiting <fn>   <named list [1]> <NULL>     <NULL>
#>  5 .5      FALSE waiting <fn>   <named list [1]> <NULL>     <NULL>
#>  6 .6      FALSE waiting <fn>   <named list [1]> <NULL>     <NULL>
#>  7 .7      FALSE waiting <fn>   <named list [1]> <NULL>     <NULL>
#>  8 .8      FALSE waiting <fn>   <named list [1]> <NULL>     <NULL>
#>  9 .9      FALSE waiting <fn>   <named list [1]> <NULL>     <NULL>
#> 10 .10     FALSE waiting <fn>   <named list [1]> <NULL>     <NULL>
#> 11 .idle-1 TRUE  running <NULL> <NULL>           <r_sessin> <NULL>
#> 12 .idle-2 TRUE  running <NULL> <NULL>           <r_sessin> <NULL>
#> 13 .idle-3 TRUE  running <NULL> <NULL>           <r_sessin> <NULL>
#> 14 .idle-4 TRUE  running <NULL> <NULL>           <r_sessin> <NULL>

Probably no tasks are running just yet. The queue only has the chance to change its state when you push(), pop() or poll(). When pushing the tasks to the queue, the workers were still starting up (i.e. the idle tasks are running), so push() could not start any real tasks. Never mind, as soon as you try to pop() or poll(), they’ll start running:

q$poll(1000L)
#> [1] ".2"
q$list_tasks()

#> # A tibble: 14 x 7
#>    id      idle  state   fun    args             worker     result    
#>    <chr>   <lgl> <chr>   <list> <list>           <list>     <list>    
#>  1 .1      FALSE running <fn>   <named list [1]> <r_sessin> <NULL>    
#>  2 .2      FALSE done    <fn>   <named list [1]> <NULL>     <cllr_ss_>
#>  3 .3      FALSE running <fn>   <named list [1]> <r_sessin> <NULL>    
#>  4 .4      FALSE running <fn>   <named list [1]> <r_sessin> <NULL>    
#>  5 .5      FALSE running <fn>   <named list [1]> <r_sessin> <NULL>    
#>  6 .6      FALSE waiting <fn>   <named list [1]> <NULL>     <NULL>    
#>  7 .7      FALSE waiting <fn>   <named list [1]> <NULL>     <NULL>    
#>  8 .8      FALSE waiting <fn>   <named list [1]> <NULL>     <NULL>    
#>  9 .9      FALSE waiting <fn>   <named list [1]> <NULL>     <NULL>    
#> 10 .10     FALSE waiting <fn>   <named list [1]> <NULL>     <NULL>    
#> 11 .idle-1 TRUE  waiting <NULL> <NULL>           <NULL>     <cllr_ss_>
#> 12 .idle-2 TRUE  waiting <NULL> <NULL>           <NULL>     <cllr_ss_>
#> 13 .idle-3 TRUE  waiting <NULL> <NULL>           <NULL>     <cllr_ss_>
#> 14 .idle-4 TRUE  waiting <NULL> <NULL>           <NULL>     <cllr_ss_>

while (!q$is_idle()) {
  task_result <- q$pop(Inf)
  print(task_result$result)
}
#> [1] "1 done"
#> [1] "2 done"
#> [1] "3 done"
#> [1] "4 done"
#> [1] "6 done"
#> [1] "5 done"
#> [1] "7 done"
#> [1] "8 done"
#> [1] "9 done"
#> [1] "10 done"

pop() just returns whatever r_session$read() returns. Here is the last result from the loop:

task_result
#> $code
#> [1] 200
#> 
#> $message
#> [1] "done file47c57a1f62b5"
#> 
#> $result
#> [1] "10 done"
#> 
#> $stdout
#> [1] ""
#> 
#> $stderr
#> [1] ""
#> 
#> $error
#> NULL
#> 
#> $task_id
#> [1] ".10"

The important fields are:

result: the R object returned from the function. This is NULL on error.
stdout: the standard output of the background session, while running the function.
stderr: the standard error.
error: error object if the function failed. NULL otherwise.
task_id: the user supplied or auto-generated task id.

Let’s see a task that errors.

q$push(function() stop("This failed, sorry"))
res <- q$pop(Inf)
res$error
#> <callr_status_error: callr subprocess failed: This failed, sorry>
#>  in process 
#> -->
#> <callr_remote_error in (function () stop("This failed, sorry"))(): This failed, sorry>

The error has two parts, the first refers to the main process, and the second is the original error, thrown in the background process. To help with debugging, the error from the background process includes a stack trace:

res$error$parent$trace

#> 
#>  ERROR TRACE for simpleError
#> 
#>  12. (function ()  ...
#>  13. base:::stop("This failed, sorry")
#>     R/<text>:1:8
#>  14. base:::.handleSimpleError(function (e)  ...
#>  15. h(simpleError(msg, call))
#> 
#>  x This failed, sorry

For this simple function that just calls stop(), the trace is not very exciting, but it can be very helpful in general.

How about process cleanup?

Luckily we don’t have to do anything extra to clean up the R processes. callr::r_session objects kill their background R session in their finalizer, i.e. when they are garbage collected. As soon as the workers have no references, because e.g. the queue object itself has no references, the garbage collector will clean them up. An explicit kill() method would be still useful sometimes, but we leave that as an exercise to the reader.

Possible improvements

To use this task queue in real code, you would need to make it a bit more robust and flexible.

Most importantly, you would need to handle crashes and freezes in the worker tasks. callr::r_session does handle crashes properly, i.e. poll() returns immediately if the session crashes, and then read() returns an informative error result. But the task queue should also do something sensible in this case, e.g. return the error result, and restart the worker.
To handle freezing worker tasks, the queue could support task timeouts, and then kill the tasks that don’t finish before their timeout expires. This can be probably implemented using the r_session$interrupt() and r_session$kill() methods.
Make the queue interrupt-safe. All operations of the queue (e.g. poll(), pop(), etc.) are interruptible by the user, but they don’t always leave the task data frame and the background sessions in a consistent state. E.g. if schedule() is interrupted and you are unlucky, you might lose all worker processes. This is a very hard issue to solve, the relatively new suspendInterrupts() function probably helps a lot.
It would be great to be able to change the number of worker tasks of the queue dynamically, i.e. add and remove worker processes.
The whole queue could be implemented in a background process, so that the scheduler runs concurrently with the main R process. This is far from being trivial, especially if one wants to avoid copying data (the function arguments) twice for every task.

Complete code

It is also available on GitHub.

task_q <- R6::R6Class(
  "task_q",
  public = list(
    initialize = function(concurrency = 4L) {
      private$start_workers(concurrency)
      invisible(self)
    },
    list_tasks = function() private$tasks,
    get_num_waiting = function()
      sum(!private$tasks$idle & private$tasks$state == "waiting"),
    get_num_running = function()
      sum(!private$tasks$idle & private$tasks$state == "running"),
    get_num_done = function() sum(private$tasks$state == "done"),
    is_idle = function() sum(!private$tasks$idle) == 0,

    push = function(fun, args = list(), id = NULL) {
      if (is.null(id)) id <- private$get_next_id()
      if (id %in% private$tasks$id) stop("Duplicate task id")
      before <- which(private$tasks$idle)[1]
      private$tasks <- tibble::add_row(private$tasks, .before = before,
        id = id, idle = FALSE, state = "waiting", fun = list(fun),
        args = list(args), worker = list(NULL), result = list(NULL))
      private$schedule()
      invisible(id)
    },

    poll = function(timeout = 0) {
      limit <- Sys.time() + timeout
      as_ms <- function(x) if (x == Inf) -1L else as.integer(x)
      repeat{
        topoll <- which(private$tasks$state == "running")
        conns <- lapply(
          private$tasks$worker[topoll],
          function(x) x$get_poll_connection())
        pr <- processx::poll(conns, as_ms(timeout))
        private$tasks$state[topoll][pr == "ready"] <- "ready"
        private$schedule()
        ret <- private$tasks$id[private$tasks$state == "done"]
        if (is.finite(timeout)) timeout <- limit - Sys.time()
        if (length(ret) || timeout < 0) break;
      }
      ret
    },

    pop = function(timeout = 0) {
      if (is.na(done <- self$poll(timeout)[1])) return(NULL)
      row <- match(done, private$tasks$id)
      result <- private$tasks$result[[row]]
      private$tasks <- private$tasks[-row, ]
      c(result, list(task_id = done))
    }
  ),

  private = list(
    tasks = NULL,
    next_id = 1L,
    get_next_id = function() {
      id <- private$next_id
      private$next_id <- id + 1L
      paste0(".", id)
    },

    start_workers = function(concurrency) {
      private$tasks <- tibble::tibble(
        id = character(), idle = logical(),
        state = c("waiting", "running", "ready", "done")[NULL],
        fun = list(), args = list(), worker = list(), result = list())
      for (i in seq_len(concurrency)) {
        rs <- callr::r_session$new(wait = FALSE)
        private$tasks <- tibble::add_row(private$tasks,
          id = paste0(".idle-", i), idle = TRUE, state = "running",
          fun = list(NULL), args = list(NULL), worker = list(rs),
          result = list(NULL))
      }
    },

    schedule = function() {
      ready <- which(private$tasks$state == "ready")
      if (!length(ready)) return()
      rss <- private$tasks$worker[ready]

      private$tasks$result[ready] <- lapply(rss, function(x) x$read())
      private$tasks$worker[ready] <- replicate(length(ready), NULL)
      private$tasks$state[ready] <-
        ifelse(private$tasks$idle[ready], "waiting", "done")

      waiting <- which(private$tasks$state == "waiting")[1:length(ready)]
      private$tasks$worker[waiting] <- rss
      private$tasks$state[waiting] <-
        ifelse(private$tasks$idle[waiting], "ready", "running")
      lapply(waiting, function(i) {
        if (! private$tasks$idle[i]) {
          private$tasks$worker[[i]]$call(private$tasks$fun[[i]],
                                         private$tasks$args[[i]])
        }
      })
    }
  )
)

Tidy dev day take two: Toulouse

Fri, 06 Sep 2019 00:00:00 +0000

We had our second official tidyverse developer day just before useR! 2019 in Toulouse. It was a fun day, and we managed to merge around 40 pull requests on the day, and nearly 30 more afterwards. You can learn more about the experience in the words of attendees Laura Acion and Dragos Moldovan-Grünfeld.

Huge thanks to everyone who contributed to tidyverse developer day! We merged over 40 PRs and have almost 30 more to review over the next few days pic.twitter.com/uM9Tg1dMeX
— Hadley Wickham (@hadleywickham) July 8, 2019

The event was particularly successful thanks to a few changes we made after our first try:

🤝 Helpers The goal of TDD is to provide opportunities for everyone to contribute, regardless of their level of expertise. This time we made sure to recruit plenty of helpers (~20 for ~70 participants) and to clearly identify them with snazzy nametags. Having friendly faces circulating and ready to lend a hand made the event much warmer for newcomers (and helping others is pretty satisfying, too).
🏷 Post-its Searching GitHub issues for a problem you can help solve is hard enough at the best of times, but it’s darn near impossible when you’re competing with a room full of people. To make it easier we went back to the basics and paired every GitHub issue with a colour-coded sticky note. This made it much easier to find issues, and as a side-benefit made it possible to see collective progress over the day as post-its migrated from To Do to Under Review to Done.
🛎 The gong Getting your PR merged is a cause for celebration, and Hadley likes to travel with a gong. It was only natural to combine these two truths by ringing the gong every time a PR was merged!

We will be hosting our next tidyverse dev day on January 31st in San Francisco, following rstudio::conf(2020). We haven’t yet finalized a venue, but it’ll be convenient to the conference location.

pkgdown 1.4.0

Thu, 05 Sep 2019 00:00:00 +0000

We’re happy to announce that pkgdown 1.4.0 is now available on CRAN. pkgdown is designed to make it quick and easy to build a website for your package. Install it with:

install.packages("pkgdown")

Development and deployment modes

The main change in this version is that build_site(), build_reference() and build_home() gain a devel parameter that controls whether you’re in deployment or development mode.

devel = FALSE activates deployment mode, which installs the package into a temporary library, and runs examples/articles in a new process. It’s the default for build_site().

If you’re using build_site() as part of an existing deployment workflow, you may need to set install = FALSE so that the package doesn’t get installed twice.
devel = TRUE activates development mode, simulates package loading with pkgload::load_all() and does all work in the same process. This makes it most suitable for rapid iteration, and so is the default for build_reference().

There were two related changes to build_reference() and build_home(): build_reference() no longer runs devtools::document() and build_home() no longer rebuilds README.Rmd. This makes pkgdown’s responsibilities more clear: it only modifies files in docs/.

Other important changes

build_home() now strips quotes from Title and Description fields when generating page metadata. Additionally, you can now override the defaults via the title and description fields in the home section of _pkgdown.yml. Setting a custom title is good practice as it allows you to add “R” and “package” to the title, which is likely to be useful for SEO.
vignette("linking") describes how pkgdown’s automatic linking works, and how to ensure that cross-package links point to the right place.
For external packages, if we can’t find an existing pkgdown website to link to we now link to https://rdrr.io.
The rendering of \examples{}, \dots, \tabular{}, \subsection{}, and \preformatted{} has been improved.
The navbar is now automatically hidden with headroom.js, and the sidebar is implemented with pure CSS.

Acknowledgements

A big thank you goes to @jayhesselberth (the co-maintainer of pkgdown), and to to the 77 other people who contributed to the release: @abodnahk, @agila5, @AliciaSchep, @andrie, @aravind-j, @asardaes, @Bisaloo, @Blackglade, @burgerga, @cboettig, @cderv, @chfleming, @choisy, @CrumpLab, @davidski, @DavisVaughan, @ddiez, @docbiz, @dongzhuoer, @dpprdan, @echasnovski, @EmilHvitfeldt, @emilyriederer, @espinielli, @florisvdh, @flying-sheep, @Fredo-XVII, @grabear, @GregorDeCillia, @gustavdelius, @gvegayon, @hadley, @ijlyttle, @IndrajeetPatil, @jackwasey, @jangorecki, @jemus42, @jennybc, @jeroen, @JohnCoene, @katrinleinweber, @kevinushey, @kkmann, @krlmlr, @maelle, @maksymiuks, @manish-saraswat-olx, @maurolepore, @maxheld83, @maximilianmordig, @mcanouil, @MichaelChirico, @mitchelloharawild, @mjsteinbaugh, @msberends, @muschellij2, @NEONKID, @noamross, @nteetor, @paul-buerkner, @peterdesmet, @ramiromagno, @rorynolan, @royfrancis, @s-fleck, @scizmeli, @sdhutchins, @seanhardison1, @seymakalay, @sharlagelfand, @surmann, @talgalili, @TheOnlyArtz, @thierrygosselin, @thomasp85, @tobyhodges, and @yiluheihei

tidymodels updates

Thu, 05 Sep 2019 00:00:00 +0000

We’ve sent a few packages to CRAN recently. Here’s a recap of the changes (and some notes at the bottom):

recipes 0.1.6

Breaking Changes

Since 2018, a warning has been issued when the wrong argument was used in bake(recipe, newdata). The deprecation period is over and new_data is officially required.
Previously, if step_other() did not collapse any levels, it would still add an “other” level to the factor. This would lump new factor levels into “other” when data were baked (as step_novel() does). This no longer occurs, since it was inconsistent with ?step_other, which said that: “If no pooling is done the data are unmodified”.

New Operations:

step_normalize() centers and scales the data (if you are, like Max, too lazy to use two separate steps).
step_unknown() will convert missing data in categorical columns to “unknown” and update factor levels.

Other Changes:

If the threshold argument of step_other() is greater than one, it specifies the minimum sample size before the levels of the factor are collapsed into the “other” category. #289
step_knnimpute() can now pass two options to the underlying knn code, including the number of threads (#323).
Due to changes by CRAN, step_nnmf() only works on versions of R >= 3.6.0 due to dependency issues.
step_dummy() and step_other() are now tolerant to cases where that step’s selectors do not capture any columns. In this case, no modifications to the data are made. (#290, #348)
step_dummy() can now retain the original columns that are used to make the dummy variables by setting preserve = TRUE. (#328)
step_other()‘s print method only reports the variables with collapsed levels (as opposed to any column that was tested to see if it needed collapsing). (#338)
step_pca(), step_kpca(), step_ica(), step_nnmf(), step_pls(), and step_isomap() now accept zero components. In this case, the original data are returned. Please use this with great care.

embed 0.0.3

Two new steps were added:

step_umap() was added for both supervised and unsupervised encodings.
step_woe() creates weight of evidence encodings. Thanks to Athos Petri Damiani for this.

rsample 0.0.5

Added three functions to compute different bootstrap confidence intervals.
A new function (add_resample_id()) augments a data frame with columns for the resampling identifier.
Updated initial_split(), mc_cv(), vfold_cv(), bootstraps(), and group_vfold_cv() to use tidyselect on the stratification variable.
Updated initial_split(), mc_cv(), vfold_cv(), and bootstraps() with new breaks parameter that specifies the number of bins to stratify by for a numeric stratification variable.

parsnip 0.0.3.1

Unplanned release based on CRAN requirements for Solaris.

Breaking Changes

The method that parsnip uses to store the model information has changed. Any custom models from previous versions will need to use the new method for registering models. The methods are detailed in ?get_model_env and the package vignette for adding models.
The mode needs to be declared for models that can be used for more than one mode prior to fitting and/or translation.
For surv_reg(), the engine that uses the survival package is now called survival instead of survreg.
For glmnet models, the full regularization path is always fit regardless of the value given to penalty. Previously, the model was fit by passing penalty to glmnet's lambda argument, and the model could only make predictions at those specific values. (#195)

New Features

add_rowindex() can create a column called .row to a data frame.
If a computational engine is not explicitly set, a default will be used. Each default is documented on the corresponding model page. A warning is issued at fit time unless verbosity is zero.
nearest_neighbor() gained a multi_predict method. The multi_predict() documentation is a little better organized.
A suite of internal functions were added to help with upcoming model tuning features.
A parsnip object always saved the name(s) of the outcome variable(s) for proper naming of the predicted values.

corrr 0.4

New features

New function called dice() function, wraps focus(x,..., mirror = TRUE)
A new retract() function does the opposite of stretch()
A new argument was added to stretch() called remove.dups. It removes duplicates with out removing all NAs.

Improvements

correlate()'s interface for databases was improved. It now only calculates unique pairs, and simplifies the formula that ultimately runs in-database. We also re-added the vignette to the package, which is also available on the site as an article

tidypredict 0.4.3

New models

The new version is now able to parse the following models:

cubist(), from the Cubist package
ctree(), from the partykit package
XGBoost trained models, via the xgboost package

New features

Integration with broom's tidy() function. It works with Regression models only
Adds support for parsnip fitted models: lm, randomForest, ranger, and earth
Adds as_parsed_model() function. It adds the proper class components to the list. This allows any model exported in the correct spec to be read in by tidypredict. See the Save Models and Non-R models for more information

Improvements

Now supports classification models from ranger

Website

The package’s official website has been expanded greatly. Here are some highlights:

An article per each supported model, they are found under Model List
A how to guide to save and reload models, link here
How to integrate non-R models to tidypredict, link here

yardstick 0.0.4

New Metrics

Two new metrics have been added to yardstick:

iic() is a numeric metric for computing the index of ideality of correlation. It is a potential alternative to the traditional correlation coefficient, and has been used in QSAR models (#115).
average_precision() is a probability metric that can be used as an alternative to pr_auc(). It has the benefit of avoiding any issues of ambiguity in the edge case where recall == 0 and the current number of false positives is 0.

Improvements

pr_curve() (and by extension pr_auc()) has been greatly improved to better handle edge cases when duplicate class probability values are present. Additionally, the first precision value in the curve is now a 1, rather than an NA, which results in a more practical curve, and generates a more correct AUC value (#93).
Each metric function now has a direction attribute, which specifies the direction required for optimization, either minimization or maximization.
Documentation for class probability metrics has been improved with more informative examples (#100).
mn_log_loss() now uses the min/max rule before computing the log of the estimated probabilities to avoid problematic undefined log values (#103).

Upcoming Changes and Directions

We are currently working on two general use packages: workflows and tune. The former bundles together recipes, model object, and other items so that there can be single fit() and predict() methods. tune will have tools for… um… tuning models. We are hoping to make these public in the next month or so.

There will be some changes to accommodate model tuning. The dials package has been re-factored substantially (see the current GH master branch) and there were some small interfaces changes to recipes too (mostly backwards compatible and also on GH). We are pretty close to end of “Phase I” of our tidymodels work.

gmailr v1.0.0

Mon, 26 Aug 2019 00:00:00 +0000

Introduction

gmailr v1.0.0 (https://gmailr.r-lib.org) is now on CRAN!

gmailr wraps the Gmail REST API v3. It provides a variety of funcitons to query your mailbox and create and send new email messages. You can retrieve, create and send emails and drafts, manage email labels, trash and untrash emails, and delete messages, drafts and threads.

The tidyverse team uses gmailr most often to send emails to the maintainers of packages which depend on our packages, to inform them when a new package release is imminent or if any of our changes require action on their part.

Install gmailr with:

install.packages("gmailr")

The release of version 1.0.0 marks three events:

There are changes in the auth interface that are not backwards compatible.
The built-in application has been removed.
The functions have all been prefixed with gm_(), to avoid name conflicts with functions in other packages (including the base package).

There is also new functionality that make writing emails with non-ASCII characters more robust, and improved documentation. See the changelog for the full details on this release.

Auth from gargle

gmailr’s auth functionality now comes from the gargle package, which provides R infrastructure to work with Google APIs, in general. We’ve just blogged about gargle’s initial release, so check out that post for more details.

We’re adopting gargle for auth in several other packages, such as bigrquery (>= v1.2.0), googledrive (>= v1.0.0), and googlesheets4 (currently GitHub-only, successor of googlesheets). This makes new token flows available in these packages, such as Application Default Credentials, and makes auth less idiosyncratic.

Auth changes the typical user will notice

The functions used for authentication have changed. Use gm_auth_configure() to configure your application and then gm_auth() to actually authenticate your user with the application.

OAuth2 tokens are now cached at the user level, by default, instead of in .httr-oauth in the current project. We will ask if it’s OK to create a new folder to hold your OAuth tokens. We recommend that you delete any vestigial .httr-oauth files lying around your gmailr projects and re-authorize gmailr, i.e. get a new token, stored in the new way.

The new strategy makes it harder to accidentally push your tokens to the cloud, easier to use multiple Google identities, and easier to share tokens across projects and packages.

Overall, gmailr has gotten more careful about getting your permission to use a cached token. See the gargle vignette Non-interactive auth to learn how to prevent attempts to interact with you, especially the section “Arrange for an oauth token to be re-discovered”.

gmailr also removes the built-in OAuth “app”. This was nessesary to comply with stricter rules and enforcement by google for Gmail API applications. It was not possible to comply with these rules and continue to have a default application embedded in gmailr. See the setup section of the readme for how to create a new project and authenticate it with only a few steps. gmailr operates under the same Privacy Policy as other tidyverse API packages, the most relevant bit for users is ‘The packages only communicate with Google APIs. No user data is shared with the owners of the Tidyverse API Packages, RStudio, or any other servers.’

Preventing name conflicts

Historically functions in gmailr did not attempt to avoid conflicts, including common functions in the base package like body() and message(). It was assumed users would use namespaced calls, e.g. gmailr::body() if there were issues. However this inadvertantly causes confusing errors for users. All of the functions are now prefixed with gm_() to solve this conflict. This also increases function discoverability via auto-complete.

The un-exported function gmailr:::gm_convert_file() converts existing scripts to use the new functions. Use gmailr:::gm_convert_file(list.files(pattern = "[.]R$", recursive = TRUE)) to convert all of your R scripts in the current directory and below. (Be sure to inspect the changes manually!)

Shared workflows

The shared use of gargle allows us to create centralized articles for several workflows that can be tricky for users:

Thanks!

Thanks to the 58 people who contributed issues, code, and comments to this release!

@abhishek1608, @alansz, @alkashef, @ascheinwald, @batpigandme, @Btibert3, @choisy, @coatless, @csalgads, @ctbrown, @eluerken, @EricGoldsmith, @fiol92, @grepinsight, @hadley, @Hellengeremias, @hopeful-coder, @howard-gu, @jamespmcguire, @jennybc, @jim0wheel, @jimhester, @jlegewie, @jnolis, @josibake, @kazuya030, @kevin-dyrland, @krlmlr, @lawremi, @lmmx, @lwasser, @maaraaj, @marco-vene, @martijnvv, @matt-negrin, @mattbaggott, @msgoussi, @msrodrigues, @MST2803, @nicocriscuolo, @obarisk, @paddytobias, @prithajnath, @RockScience, @RozennGZ, @rpietro, @scottgrimes, @SimonCoulombe, @slomanl1, @songyurita, @sumxf, @TheKashe, @tingmar, @tmamiya, @Vestaxis, @vikram-rawat, @williamgunn, and @xhudik

gargle's debut on CRAN

Tue, 20 Aug 2019 00:00:00 +0000

We’re elated to (somewhat belatedly) announce the initial release of gargle on CRAN!

The gargle package (https://gargle.r-lib.org) is meant to take some of the pain out of working with Google APIs. It’s mostly aimed at the maintainers of R packages that call Google APIs. If we’re successful, users won’t really notice gargle, they’ll just feel like these packages got easier to work with and less idiosyncratic.

The timing of this post is motivated by the release of googledrive v1.0.0. As of this version, googledrive gets all of its auth functionality (and more) from gargle. Although we did not blog about it, bigrquery also recently made the switch, with the release of v1.2.0.

If you are interested in the user-facing implications of using gargle, check out the blog post announcing googledrive v1.0.0 and the changelog for bigrquery v1.2.0.

gargle

The gargle package first appeared on CRAN in early June 2019 and has gotten a few small updates as we learn more by wiring it into packages. gargle is already used by:

bigrquery (>= v1.2.0)
googledrive (>= v1.0.0)
gmailr (>= v1.0.0 coming soon to CRAN)
googleAuthR (dev version)
googlesheets4 GitHub only
gcalendr GitHub only

gargle’s current functionality falls into two domains, which a client package can adopt (or not) separately:

Auth: help users authenticate themselves with their Google identity and obtain a token that the wrapper package can use to make authorized requests.
Request preparation and response handling: check requests against the machine-readable Discovery Documents that describe Google APIs and process API responses, especially errors.

The long-term stretch goal is to do for R what the official Google API Client Libraries do for other languages, like Python and Java.

Auth via gargle

Under the hood, gargle’s main auth function is token_fetch(), which tries a series of different methods for obtaining a token. The intent is to make auth “just work” in a wide variety of contexts. This makes some flows newly available in the client packages, such as Application Default Credentials (official Google docs and a more readable 3rd party blog post).

The main change that users will notice is that gargle implements its own way of storing user OAuth tokens between sessions. The httr package, which gargle depends on, has historically used a local .httr-oauth file for this. In contrast, gargle encourages tokens to be stored outside the project, in a hidden directory at the user level, in a key-value store that incorporates the Google user’s email. This makes it harder to accidentally push your tokens to the cloud, easier to use multiple Google identities, and easier to share tokens across projects and packages. Users will notice they need to re-auth and may want to track down and delete vestigial .httr-oauth files.

Articles and docs

The release of gargle has provided an occasion to document several recurring workflows that can be tricky for useRs, which now work the same across multiple packages:

gargle also offers a mechanism for client packages to build the documentation for their auth function from templates stored in gargle. This means that packages using a common design will also use the same words to describe it.

Another set of articles is aimed at package maintainers interested in delegating auth or request & response handling to gargle:

Google is tightening the rules around various aspects of API access (see, e.g., Project Strobe and Elevating user trust in our API ecosystem). I predict the main thing that useRs will notice is that more and more R packages and apps will require the user to get involved in the nitty gritty details of auth. Users will need to create their own Google Cloud Platform projects, obtain their own API keys, and their own OAuth client IDs and secrets. The overall rationale for Google’s changes makes sense, but the way people use open source R packages is an awkward fit with their recommended auth solutions. It’s becoming more difficult for package developers to make auth feel like it “just works”.

The Google-wrapping packages maintained by the tidyverse / r-lib team are now governed by a shared Privacy Policy. This is linked from each package and also from the consent screen whenever auth is facilitated through our OAuth client.

Some gargle history

I find gargle’s origin story find very satisfying. I suspect that Hadley Wickham’s bigrquery (first on CRAN January 2015) has shaped the basic design for how most R packages handle Google auth, directly or indirectly. I can certainly say that Joanna Zhao and I consulted it when developing googlesheets (first on CRAN July 2015). I know that Mark Edmondson was influenced, in turn, by googlesheets, when he developed googleAuthR (first on CRAN August 2015), used in his suite of packages. By the time Lucy D’Agostino McGowan and I created googledrive (first on CRAN August 2017), I’d developed some strong opinions about how to modify httr’s default behaviour for work with Google APIs. I was also working with Hadley Wickham at RStudio by that point and Lucy and I benefited from his design advice. In May-ish 2017, leading up to rOpenSci’s 2017 Unconf, Craig Citro opened an issue to generate discussion about how to de-duplicate even more API pain, especially for Google APIs. He has a wealth of experience from his role in shaping the official Python client and he made the initial commits that laid the foundation for gargle::token_fetch().

Starting from a common origin (bigrquery), all these people and packages have explored different aspects of this problem space and have developed various solutions. gargle represents the distillation of a lot of hard-won experience and a promising space for future consolidation.

Thanks!

Thanks to all those who have helped get gargle to its first release:

@andrie, @batpigandme, @byapparov, @craigcitro, @dlebech, @hadley, @jennybc, @jonthegeek, @MarkEdmondson1234, @wlongabaugh, and @ZainRizvi

googledrive v1.0.0

Tue, 20 Aug 2019 00:00:00 +0000

Introduction

We’re jazzed to announce the release of googledrive v1.0.0 (https://googledrive.tidyverse.org).

googledrive wraps the Drive REST API v3. The most common file operations are implemented in high-level functions designed for ease of use. You can find, list, create, trash, delete, rename, move, copy, browse, download, share and publish Drive files, including those on Team Drives.

Install googledrive with:

install.packages("googledrive")

The release of version 1.0.0 marks two events:

The overall design of googledrive has survived ~2 years on CRAN, with very little need for change. The interface and feature set are fairly stable. googledrive facilitated around 7 million requests to the Drive API in the past month. But also …
There are changes in the auth interface that are not backwards compatible.

There is also new functionality that makes it less likely you’ll create multiple files with the same name, without actually meaning to.

Auth from gargle

googledrive’s auth functionality now comes from the gargle package, which provides R infrastructure to work with Google APIs, in general. We’ve just blogged about gargle’s initial release, so check out that post for more details.

We’re adopting gargle for auth in several other packages, such as bigrquery (>= v1.2.0), gmailr (>= v1.0.0 coming soon to CRAN), and googlesheets4 (currently GitHub-only, successor of googlesheets). This makes new token flows available in these packages, such as Application Default Credentials, and makes auth less idiosyncratic.

Auth changes the typical user will notice

If you’ve always let googledrive guide you through auth, here is the one change you will notice:

OAuth2 tokens are now cached at the user level, by default, instead of in .httr-oauth in the current project. We will ask if it’s OK to create a new folder to hold your OAuth tokens. We recommend that you delete any vestigial .httr-oauth files lying around your googledrive projects and re-authorize googledrive, i.e. get a new token, stored in the new way.

The new strategy makes it harder to accidentally push your tokens to the cloud, easier to use multiple Google identities, and easier to share tokens across projects and packages.

Overall, googledrive has gotten more careful about getting your permission to use a cached token. See the gargle vignette Non-interactive auth to learn how to prevent attempts to interact with you, especially the section “I just want my .Rmd to render”.

googledrive also uses a new OAuth “app”, owned by a verified Google Cloud Project entitled “Tidyverse API Packages”, which is the project name you will see on the OAuth consent screen. See our new Privacy Policy for details.

For more advanced users who call drive_auth() directly or who configure auth settings, such as their own OAuth app or API key, see the changelog for more details.

Preventing name clashes

Google Drive doesn’t impose a 1-to-1 relationship between files and filepaths, the way your local file system does. Therefore, when working via the Drive API (instead of in the browser), it’s fairly easy to create multiple Drive files with the same name or filepath, without actually meaning to. This is perfectly valid on Drive, which identifies file by ID, but can be confusing and undesirable for humans. Very few people actually want this:

googledrive v1.0.0 offers some new ways to prevent writing more than one file to the same filepath.

All functions that create a new item or rename/move an existing item have gained an overwrite argument:

drive_create() this function is new in v1.0.0
drive_cp()
drive_mkdir()
drive_mv()
drive_rename()
drive_upload()

The default of overwrite = NA corresponds to the existing behaviour, which does not consider pre-existing files at all. overwrite = TRUE requests to move a pre-existing file at the target filepath to the trash, prior to creating the new item. If 2 or more files are found, an error is thrown, because it’s not clear which one(s) to trash. overwrite = FALSE means the new item will only be created if there is no pre-existing file at that filepath. Existence checks based on filepath (or name) can be expensive. This is why the default is overwrite = NA, in addition to backwards compatibility.

drive_put() is a new convenience wrapper that figures out whether to call drive_upload() or drive_update().

Sometimes you have a file you will repeatedly send to Drive, i.e. the first time you run an analysis, you create the file and, when you re-run it, you update the file. Previously this was hard to express with googledrive.

drive_put() is useful here and refers to the HTTP verb PUT: create the thing if it doesn’t exist or, if it does, replace its contents. A good explanation of PUT is RESTful API Design — PUT vs PATCH.

In pseudo-code, here’s the basic idea of drive_put():

target_filepath <- <determined from arguments `path`, `name`, and `media`>
hits <- <get all Drive files at target_filepath>
if (no hits) {
 drive_upload(media, path, name, type, ..., verbose)
} else if (exactly 1 hit) {
 drive_update(hit, media, ..., verbose)
} else {
 ERROR
}

Shared workflows

The shared use of gargle allows us to create centralized articles for several workflows that can be tricky for useRs:

Thanks!

Thank you to the 41 people who contributed issues, code, and comments to this release:

@abeburnett, @admahood, @alexpghayes, @arendsee, @batpigandme, @benmarwick, @Chanajit, @cowlumbus, @ctlamb, @DavidGarciaEstaun, @dgplaco, @Diego-MX, @dsdaveh, @eeenilsson, @efh0888, @giocomai, @grabear, @hwsamuel, @ianmcook, @jarodmeng, @jennybc, @jimhester, @lohancock, @lotard, @LucyMcGowan, @lukaskawerau, @MariaMetriplica, @Martin-Jung, @medewitt, @njudd, @philmikejones, @prokulski, @RNA-Ninja, @romunov, @sanjmeh, @Serenthia, @shawzhifei, @stapial, @svenhalvorson, @tarunparmar, and @tsmith64

ragg 0.1.0

Mon, 15 Jul 2019 00:00:00 +0000

We’re stoked to announce the release of ragg 0.1.0 on CRAN. ragg provides a set of high quality and high performance raster devices, capable of producing png, tiff, or ppm files, or a matrix of raw color values directly within R.

ragg is part of our broader effort to improve graphics performance and quality in R at all levels of the stack, so that you’ll benefit no matter what plotting framework you choose to use. Other parts of this efforts have been:

Developing the devoid package to allow more precise benchmarking of plotting code.
Multiple improvements to rendering speed in grid in the latest R release (3.6.0).
Performance improvements in ggplot2 3.2.0 both broadly and for sf plotting specifically.
Performance improvements in gtable 0.3.0.

The devices

An R graphic device is an object that receives instructions from the graphic engine in R and translates that into some meaningful format for viewing. The graphic engine in R is the layer that sits between the graphic generating code in R and the devices, and is responsible for allowing the plethora of different output options from the same plot code. The output from a graphic device can either be a file, on screen, or in some other form. ragg provides a set of raster devices, that is, devices that rasterize the instructions and write it to some sort of raster output (e.g. a png file). This is opposed to vector devices such as pdf() and svg() that do not perform rasterization but write the instructions directly into a vector graphics format.

ragg provides three different file outputs (png, tiff, and ppm), though it is important to note that everything, except for the serialization of the data into the file format, is equivalent. A case could be made for also including jpeg output, but this format is generally not useful for graphics as it introduces noticeable artifacts with line graphics.

Apart from the three file-based devices, ragg also provides a device that gives direct access to the raster buffer from R. This means that you can plot directly into a matrix of color values which you can then further process in R, should you wish.

Features

There are 5 main areas where ragg sets itself apart from the graphic devices already available:

Performance: ragg is faster than cairo, the standard anti-aliased device in R. Tests show that it is about twice as fast to render a moderately complex ggplot. See the Performance vignette for more in-depth benchmarking.
Anti-aliasing: ragg is fully anti-aliased, as opposed to the cairo devices that only apply anti-aliasing to strokes and text (not fill). For a comparison, see the shape rendering section of the Quality vignette.
Text rendering: ragg provides high-quality rendering of rotated text, something that other raster devices struggle with. See the text section of the Quality vignette for examples.
Font access: ragg has direct access to all your system fonts, without you having to do anything. All installed fonts on your system should be ready to use.
System independence: The rendering code in ragg is system independent and should be identical whether it has been rendered on Windows, macOS, or Linux. The only difference is the available fonts on the different systems.

Example

The ragg devices are used just like any other device, by starting them, running your plotting code, and turning them off (code below add some complexity in order to embed the created file in the page):

library(ragg)
library(ggplot2)

file <- knitr::fig_path('.png')

agg_png(file, width = 700, height = 500, units = 'px')
ggplot(diamonds) + 
  geom_bar(aes(color, fill = color)) + 
  ggtitle("A fancy font") + 
  theme(text = element_text(family = "Daubmark", size = 50))
invisible(dev.off())

knitr::include_graphics(file)

ragg can also be used with ggsave() by passing the device in as an argument:

file <- knitr::fig_path('.png')
p <- ggplot(mpg, aes(displ, hwy)) + 
  geom_point()

ggsave(file, p, device = agg_png, res = 300)
#> Saving 7 x 4.33 in image

knitr::include_graphics(file)

Life cycle

ragg is currently to be considered experimental. That is not to indicate that any API changes are to be expected, or that using it is not a safe long-term strategy. But we are continuing to invest and improve upon the graphic stack in R, and we cannot say whether ragg will be part of our final solution, or if it is just an interesting experiment. The experience gained in the graphic stack from building ragg will definitely be put to good use though, and you can expect more improvements in the future.

Acknowledgements

Thanks to @jeroen for much assistance in getting ragg to compile on all systems!

callr 3.3.0

Mon, 08 Jul 2019 00:00:00 +0000

We have just updated the callr package to version 3.3.0 on CRAN. The biggest change in this release is better support for debugging the background process. See the full changelog here.

callr helps with running R code in a separate R process, synchronously or asynchronously. With synchronous execution the main R process waits until the separate R subprocess finishes, see callr::r(). Asynchronous execution uses processx processes, see callr::r_bg() and callr::r_process() for one-off and callr::r_session() for persistent background R processes.

callr error objects

Debugging code running in a background process is notoriously difficult. Most of the time you cannot use an interactive debugger, and often even print-debugging, i.e. inserting print() and cat() calls into the code that runs in the background, can be non-trivial.

The new 3.3.0 version of callr aims to help with this, by creating better error messages and error traces for errors originating from the background process. In particular, callr now always throws error objects that contain:

the exit status of the R process, if the process terminated,
the full error object thrown in the subprocess,
the call that generated the error,
the process id of the subprocess, and
the full stack trace in the subprocess.

Here is an example for a trivial error that shows how to extract this information if the error was caught in the main process:

err <- tryCatch(
  callr::r(function() library(Callr)),
  error = function(e) e)
err
#> <callr_status_error: callr subprocess failed: there is no package called ‘Callr’>
#>  in process 
#> -->
#> <callr_remote_error in library(Callr): there is no package called ‘Callr’>

The error objects has two parts. The first is the error object thrown in the main process, and the second is the error object from the the subprocess. We can extract more information from err:

err$status
#> [1] 0
err$parent
#> <callr_remote_error in library(Callr): there is no package called ‘Callr’>
err$parent$call
#> function() library(Callr)
err$parent$`_pid`
#> [1] 79124

err$status is the exit status of the subprocess. This is not present for persistent background processes, i.e. the ones created by r_session, because these do not exit on error, but continue running. err$parent is the error object, thrown in the subprocess. err$parent$call is the call that generated the error, and err$parent$`_pid` is the process id of the subprocess.

The stack trace of the error in subprocess can be printed via err$parent$trace. By default the trace omits the boilerplate frames added by callr, these are usually not very useful for the user. Nevertheless they are still included in err$parent$trace$calls.

err$parent$trace

#> 
#>  ERROR TRACE for packageNotFoundError
#> 
#>  12. (function ()  ...
#>  13. base:::library(Callr)
#>     R/<text>:2:12
#>  14. base:::stop(packageNotFoundError(package, lib.loc, sys.call()))
#>  15. (function (e)  ...
#> 
#>  x there is no package called ‘Callr’

The trace starts with the anonymous function that we passed to callr::r(), and it is annotated with package names and source references, if they are available.

The last error

Often, the error object is uncaught, i.e. we don’t tryCatch() the error in the main R process. Then the error message is printed, but the actual error object is lost, and you need to re-run the code in a tryCatch(), hoping that it would produce the same error.

For a better workflow, whenever a callr error is uncaught, callr assigns it to the .Last.error variable, that can be inspected. Of course, a subsequent callr error will overwrite .Last.error, it works very much like .Last.value, but for errors. Here is the same code as above but without the tryCatch():

callr::r(function() library(Callr))
#> Error: callr subprocess failed: there is no package called ‘Callr’

.Last.error
#> <callr_status_error: callr subprocess failed: there is no package called ‘Callr’>
#> -->
#> <callr_remote_error in library(Callr): there is no package called ‘Callr’>
.Last.error$parent$call
#> function() library(Callr)

The last error trace

If the error is uncaught, then callr adds a trace to the error object of the main process as well. The trace will have two parts in this case. callr also sets the .Last.error.trace variable for convenience, this is easier to type than .Last.error$trace.

.Last.error.trace

#> 
#>  ERROR TRACE for callr_status_error, callr_error, rlib_error
#> 
#>  Process 79108:
#>  30. callr::r(function() library(Callr))
#>  31. callr:::get_result(output = out, options)
#>     R/eval.R:149:3
#>  32. base:::throw(new_callr_error(output, msg), parent = err[[2]])
#>     R/result.R:73:5
#> 
#>  x callr subprocess failed: there is no package called ‘Callr’ 
#> 
#>  Process 79135:
#>  44. (function ()  ...
#>  45. base:::library(Callr)
#>     R/<text>:1:10
#>  46. base:::stop(packageNotFoundError(package, lib.loc, sys.call()))
#>  47. (function (e)  ...
#> 
#>  x there is no package called ‘Callr’

The top part of the trace contains the frames in the main process, and the bottom part contains the frames in the subprocess, starting with the anonymous function.

dplyr 0.8.3

Thu, 04 Jul 2019 00:00:00 +0000

Introduction

We’re pleased (and a little embarrassed) to announce the release of dplyr 0.8.3 on CRAN 😬 !

This is an emergency release, one week after 0.8.2 to fix a major performance regression as reported by the community on twitter.

Nostra culpa

This commit was pushed to dplyr on May 10, as a seemingly very harmless update to better support list columns being used in summarise().

In addition to the changes that were necessary to fix the problem reported in issue 4349, the commit removed two lines of code in a very central piece of dplyr infrastructure, namely its data mask layout.

As part of dplyr 0.8.0, internals of the data mask layout has dramatically changed and uses a data mask composed of two environments. The first environment contains a set of active bindings for each of the columns in the data frame to process, the first time a variable is used in an expression, presumably on the first group, the active binding is resolved to get the relevant slice of that column. This is a somewhat expensive operation, therefore subsequent groups pro actively materialise the slice of columns which are known to be needed, using the second environment of the data mask.

The two lines that were removed by mistake are central to this system, without them each group would invoke the costly active binding. Even worse, the list of indices of columns to be rematerialised, as maintained by a vector of integers, would grow each time, so on the second group the column slice would be materialized twice to be then forgotten, on the third group three times …

Classic embarrassing quadratic performance regression.

Thanks

Thanks to the community for quickly alerting us of the situation, the 🐌 had been in the code base for almost two months but we had not noticed because our continuous integration protects us from regressions in functionality, but not regression in performance.

We might investigate in that direction in the future.Thanks to all contributors for this release.

@ajkroeg, @bschneidr, @DavisVaughan, @dhicks, @gvfarns, @IndrajeetPatil, @krlmlr, @romainfrancois, and @shane-kercheval

dplyr 0.8.2

Sat, 29 Jun 2019 00:00:00 +0000

Introduction

We’re delighted to announce the release of dplyr 0.8.2 on CRAN 🍉 !

This is a minor maintenance release in the 0.8.* series, addressing a collection of issues since the 0.8.1 and 0.8.0 versions.

top_n() and top_frac()

top_n() has been around for a long time in dplyr, as a convenient wrapper around filter() and min_rank(), to select top (or bottom) entries in each group of a tibble.

In this release, top_n() is no longer limited to a constant number of entries per group, its n argument is now quoted to be evaluated later in the context of the group.

Here are the top half countries, i.e. n() / 2, in terms of life expectancy in 2007.

library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
gapminder::gapminder %>% 
  filter(year == 2007) %>% 
  group_by(continent) %>% 
  top_n(n() / 2, lifeExp)
#> # A tibble: 70 x 6
#> # Groups:   continent [5]
#>    country   continent  year lifeExp        pop gdpPercap
#>    <fct>     <fct>     <int>   <dbl>      <int>     <dbl>
#>  1 Algeria   Africa     2007    72.3   33333216     6223.
#>  2 Argentina Americas   2007    75.3   40301927    12779.
#>  3 Australia Oceania    2007    81.2   20434176    34435.
#>  4 Austria   Europe     2007    79.8    8199783    36126.
#>  5 Bahrain   Asia       2007    75.6     708573    29796.
#>  6 Belgium   Europe     2007    79.4   10392226    33693.
#>  7 Benin     Africa     2007    56.7    8078314     1441.
#>  8 Canada    Americas   2007    80.7   33390141    36319.
#>  9 Chile     Americas   2007    78.6   16284741    13172.
#> 10 China     Asia       2007    73.0 1318683096     4959.
#> # … with 60 more rows

top_frac() is new convenience shortcut for the top n percent, i.e.

gapminder::gapminder %>% 
  filter(year == 2007) %>% 
  group_by(continent) %>% 
  top_frac(0.5, lifeExp)
#> # A tibble: 70 x 6
#> # Groups:   continent [5]
#>    country   continent  year lifeExp        pop gdpPercap
#>    <fct>     <fct>     <int>   <dbl>      <int>     <dbl>
#>  1 Algeria   Africa     2007    72.3   33333216     6223.
#>  2 Argentina Americas   2007    75.3   40301927    12779.
#>  3 Australia Oceania    2007    81.2   20434176    34435.
#>  4 Austria   Europe     2007    79.8    8199783    36126.
#>  5 Bahrain   Asia       2007    75.6     708573    29796.
#>  6 Belgium   Europe     2007    79.4   10392226    33693.
#>  7 Benin     Africa     2007    56.7    8078314     1441.
#>  8 Canada    Americas   2007    80.7   33390141    36319.
#>  9 Chile     Americas   2007    78.6   16284741    13172.
#> 10 China     Asia       2007    73.0 1318683096     4959.
#> # … with 60 more rows

tbl_vars() and group_cols()

tbl_vars() now returns a dplyr_sel_vars object that keeps track of the grouping variables. This information powers group_cols(), which can now be used in every function that uses tidy selection of columns.

Functions in the tidyverse and beyond may start to use the tbl_vars()/group_cols() duo, starting from tidyr and this pull request

# pak::pkg_install("tidyverse/tidyr#668")

iris %>%
  group_by(Species) %>% 
  tidyr::gather("flower_att", "measurement", -group_cols())
#> # A tibble: 600 x 3
#> # Groups:   Species [3]
#>    Species flower_att   measurement
#>    <fct>   <chr>              <dbl>
#>  1 setosa  Sepal.Length         5.1
#>  2 setosa  Sepal.Length         4.9
#>  3 setosa  Sepal.Length         4.7
#>  4 setosa  Sepal.Length         4.6
#>  5 setosa  Sepal.Length         5  
#>  6 setosa  Sepal.Length         5.4
#>  7 setosa  Sepal.Length         4.6
#>  8 setosa  Sepal.Length         5  
#>  9 setosa  Sepal.Length         4.4
#> 10 setosa  Sepal.Length         4.9
#> # … with 590 more rows

group_split(), group_map(), group_modify()

group_split() always keeps a ptype attribute to track the prototype of the splits.

mtcars %>%
  group_by(cyl) %>%
  filter(hp < 0) %>% 
  group_split()
#> list()
#> attr(,"ptype")
#> # A tibble: 0 x 11
#> # … with 11 variables: mpg <dbl>, cyl <dbl>, disp <dbl>, hp <dbl>,
#> #   drat <dbl>, wt <dbl>, qsec <dbl>, vs <dbl>, am <dbl>, gear <dbl>,
#> #   carb <dbl>

group_map() and group_modify() benefit from this in the edge case where there are no groups.

mtcars %>%
  group_by(cyl) %>%
  filter(hp < 0) %>% 
  group_map(~.x)
#> list()
#> attr(,"ptype")
#> # A tibble: 0 x 10
#> # … with 10 variables: mpg <dbl>, disp <dbl>, hp <dbl>, drat <dbl>,
#> #   wt <dbl>, qsec <dbl>, vs <dbl>, am <dbl>, gear <dbl>, carb <dbl>

mtcars %>%
  group_by(cyl) %>%
  filter(hp < 0) %>% 
  group_modify(~.x)
#> # A tibble: 0 x 11
#> # Groups:   cyl [0]
#> # … with 11 variables: cyl <dbl>, mpg <dbl>, disp <dbl>, hp <dbl>,
#> #   drat <dbl>, wt <dbl>, qsec <dbl>, vs <dbl>, am <dbl>, gear <dbl>,
#> #   carb <dbl>

Thanks

Thanks to all contributors for this release.

@abirasathiy, @ajkroeg, @alejandroschuler, @anuj2054, @arider2, @arielfuentes, @artidata, @BenPVD, @bkmontgom, @brodieG, @cderv, @clanker, @clemenshug, @CSheehan1, @danielecook, @dannyparsons, @daskandalis, @davidbaniadam, @DavisVaughan, @deliciouslytyped, @earowang, @fkatharina, @hadley, @Hardervidertsie, @iago-pssjd, @IndrajeetPatil, @jackdolgin, @jangorecki, @jimhester, @jjesusfilho, @jonjhitchcock, @jxu, @krlmlr, @laresbernardo, @lionel-, @LukeGoodsell, @madmark81, @MarkusBerroth, @matheus-donato, @mattfidler, @MatthieuStigler, @md0u80c9, @michaelhogersosis, @MikeJohnPage, @MJL9588, @moodymudskipper, @mwillumz, @Nelson-Gon, @qdread, @randomgambit, @rcorty, @romainfrancois, @romatik, @spressi, @sstoeckl, @stephLH, @urskalbitzer, @vpanfilov, and @ZahraEconomist.

rlang 0.4.0

Thu, 27 Jun 2019 00:00:00 +0000

It is with great excitement that we announce the release of rlang 0.4.0 on CRAN. rlang is a toolkit for working with core R and Tidyverse features, and hosts the tidy evaluation framework. The full set of changes can be found in the changelog.

In this article, we introduce the most important of these, the new tidy evaluation operator {{. We will use a simple dplyr pipeline as a running example, let’s start by attaching the package:

library(dplyr)

The good and bad of tidy evaluation

Tidy eval powers packages like dplyr and tidyr. It makes it possible to manipulate data frame columns as if they were defined in the workspace:

gender
#> Error in eval(expr, envir, enclos): object 'gender' not found
mass
#> Error in eval(expr, envir, enclos): object 'mass' not found

starwars %>%
  group_by(gender) %>%
  summarise(mass_maximum = max(mass, na.rm = TRUE))
#> # A tibble: 5 x 2
#>   gender        mass_maximum
#>   <chr>                <dbl>
#> 1 <NA>                    75
#> 2 female                  75
#> 3 hermaphrodite         1358
#> 4 male                   159
#> 5 none                   140

We call this syntax data masking. This feature is unique to the R language and greatly streamlines the writing and reading of code in interactive scripts. Unfortunately, it also makes it more complex to reuse common patterns inside functions:

max_by <- function(data, var, by) {
  data %>%
    group_by(by) %>%
    summarise(maximum = max(var, na.rm = TRUE))
}

starwars %>% max_by(mass, by = gender)
#> Error: Column `by` is unknown

Technically, this is because data-masked code needs to be delayed and transported to the data context. Behind the scenes, dplyr verbs achieve this by capturing the blueprint of your code, and resuming its evaluation inside the data mask. The example above fails because group_by() is capturing the wrong piece of blueprint. To solve this, tidy evaluation provides enquo() to delay the interpretation of code and capture its blueprint, and the surgery operator !! for modifying blueprints. The combination of using enquo() and !! is called the quote-and-unquote pattern:

max_by <- function(data, var, by) {
  data %>%
    group_by(!!enquo(by)) %>%
    summarise(maximum = max(!!enquo(var), na.rm = TRUE))
}

starwars %>% max_by(mass, by = gender)
#> # A tibble: 5 x 2
#>   gender        maximum
#>   <chr>           <dbl>
#> 1 <NA>               75
#> 2 female             75
#> 3 hermaphrodite    1358
#> 4 male              159
#> 5 none              140

We have come to realise that this pattern is difficult to teach and to learn because it involves a new, unfamiliar syntax, and because it introduces two new programming concepts (quote and unquote) that are hard to understand intuitively. This complexity is not really justified because this pattern is overly flexible for basic programming needs.

A simpler interpolation pattern with `{{`

rlang 0.4.0 provides a new operator, {{ (read: curly curly), which abstracts quote-and-unquote into a single interpolation step. The curly-curly operator should be straightforward to use. When you create a function around a tidyverse pipeline, wrap the function arguments containing data frame variables with {{:

max_by <- function(data, var, by) {
  data %>%
    group_by({{ by }}) %>%
    summarise(maximum = max({{ var }}, na.rm = TRUE))
}

starwars %>% max_by(height)
#> # A tibble: 1 x 1
#>   maximum
#>     <int>
#> 1     264

starwars %>% max_by(height, by = gender)
#> # A tibble: 5 x 2
#>   gender        maximum
#>   <chr>           <int>
#> 1 <NA>              167
#> 2 female            213
#> 3 hermaphrodite     175
#> 4 male              264
#> 5 none              200

This syntax should be reminiscent of string interpolation in the glue package by Jim Hester:

var <- sample(c("woof", "meow", "mooh"), size = 1)
glue::glue("Did you just say {var}?")
#> Did you just say mooh?

Other simple tidy evaluation patterns

There are a few existing patterns that aren’t emphasised enough in the existing documentation. We are changing our teaching strategy to focus on these simpler patterns.

If you would like to pass multiple arguments to a data-masking verb, pass ... directly:

summarise_by <- function(data, ..., by) {
  data %>%
    group_by({{ by }}) %>%
    summarise(...)
}
  
starwars %>%
  summarise_by(
    average = mean(height, na.rm = TRUE),
    maximum = max(height, na.rm = TRUE),
    by = gender
  )
#> # A tibble: 5 x 3
#>   gender        average maximum
#>   <chr>           <dbl>   <int>
#> 1 <NA>             120      167
#> 2 female           165.     213
#> 3 hermaphrodite    175      175
#> 4 male             179.     264
#> 5 none             200      200

You only need quote-and-unquote (with the plural variants enquos() and !!!) when you need to modify the inputs or their names in some way.

If you have string inputs, use the .data pronoun:

max_by <- function(data, var, by) {
  data %>%
    group_by(.data[[by]]) %>%
    summarise(maximum = max(.data[[var]], na.rm = TRUE))
}
  
starwars %>% max_by("height", by = "gender")
#> # A tibble: 5 x 2
#>   gender        maximum
#>   <chr>           <int>
#> 1 <NA>              167
#> 2 female            213
#> 3 hermaphrodite     175
#> 4 male              264
#> 5 none              200

The . pronoun from magrittr is not appropriate here because it represents the whole data frame, whereas .data represents the subset for the current group.

To learn more about the different ways of programming around tidyverse pipelines, we recommend reading the new programming vignette in ggplot2, written by Dewey Dunnington who is currently interning at RStudio.

Thanks!

The following people have contributed to this release by posting issues and pull requests:

@001ben, @asardaes, @BillDunlap, @burchill, @cpsievert, @DavisVaughan, @egnha, @flying-sheep, @gaborcsardi, @gaelledoucet, @GaGaMan1101, @grayskripko, @hadley, @harrysouthworth, @holgerbrandl, @IndrajeetPatil, @jazzmoe, @jennybc, @jjesusfilho, @juangomezduaso, @krlmlr, @lionel-, @Marieag, @mmuurr, @moodymudskipper, @paulponcet, @riccardopinosio, @richierocks, @RolandASc, @romainfrancois, @s-fleck, @siddharthprabhu, @subratiter1, @wch, @wetlandscapes, @wlandau, @x1o, @XWeiZhou, @yenzichun, @yonicd, and @zachary-foster

ggplot2 3.2.0

Sun, 16 Jun 2019 00:00:00 +0000

We’re thrilled to announce the release of ggplot2 3.2.0 on CRAN. ggplot2 is a system for declaratively creating graphics, based on The Grammar of Graphics. You provide the data, tell ggplot2 how to map variables to aesthetics, what graphical primitives to use, and it takes care of the details.

The 3.2.0 release is a minor release which focuses on performance improvements, but also includes a few new features and a range of bug fixes. There are no breaking changes in this release, though a few changes may affect the visual appearance of plots in minor ways, noted below. Further, there are a few changes that developers of extension packages may need to take into account, but which won’t affect regular users. For a full overview of the changes, please see the release notes.

This release also includes a range of contributions made during our tidyverse developer day in Austin, many from first-time contributors. We hope these contributors have been inspired to continue taking part in the development of ggplot2.

Lastly, this release also sees the entrance of Hiroaki Yutani (yutannihilation on both Github and Twitter) into the core developer team. Hiroaki has been amazing in tackling large and small issues, and we are very lucky to have him.

Performance

A large part of this release relates to internal changes intended to speed up ggplot2. Most of these are general and will affect the rendering speed of all the plots you make, but there has also been a specific focus on geom_sf(). This is all part of a larger effort to make plotting in R more performant, and also includes changes in gtable, sf, and R itself. Make sure to use the latest version of all of these packages to get the full benefit of the improvements.

geom_sf

While most changes are general, geom_sf() has received special attention. Most of this comes down to how geom_sf() converted its input to grobs, which was done row by row. This meant that plotting 10,000 ST_POINT objects would create 10,000 point grobs instead of a single grob containing 10,000 points. The same was true for all other data types. With the new release, ggplot2 will try to pack all objects into a single grob, which is possible when all objects are of the same type (MULTI* types can be mixed with scalar types). Packing polygons into a single grob is only possible with R 3.6.0 and upwards, but all other types are backwards compatible with older versions of R. If the data contains a mix of types, ggplot2 will fall back to creating a grob for each row, but this is a much less frequent situation, and can usually be remedied by creating multiple sf layers instead. The other big performance bottleneck in plotting sf data was the normalization of the coordinates. As sf stores its data in nested lists, the standard vectorization in R doesn’t apply, which led to much worse performance compared to normalizing data stored in standard data frame format. The latest release of sf includes optimized functions for these operations implemented in C which ggplot2 now uses, so plotting performance has improved immensely.

New features

With this release, ggplot2 gains the ability to plot polygons with holes (only in R 3.6 and later). Rather than providing a new geom, the functionality is built into geom_polygon() through a new subgroup aesthetic. Much as the group aesthetic separates polygons in the data, the subgroup aesthetic separates parts of each polygon. The first occurring subgroup should always be the outer ring of the polygon and any subsequent subgroup will describe holes in this (or, potentially, polygons within the holes and so on). There will not be any checks performed on the position of the subgroups, so it is the responsibility of the user to make sure they are inside the outer ring.

library(ggplot2)
library(tibble)
radians <- seq(0, 2 * pi, length.out = 101)[-1]
circle <- tibble(
  x = c(cos(radians), cos(radians) * 0.5),
  y = c(sin(radians), sin(radians) * 0.5),
  subgroup = rep(c(1, 2), each = 100)
)
ggplot(circle) +
  geom_polygon(
    aes(x = x, y = y, subgroup = subgroup),
    fill = "firebrick",
    colour = "black"
  )

The other bigger new feature is the ability to modify the guide representation of a layer through a new key_glyph argument in all geoms. While the defaults are often fine, there are situations where a different look serves the visualisation:

ggplot(economics, aes(date, psavert, color = "savings rate")) +
  geom_line(key_glyph = "timeseries")

geom_rug has seen a range of improvements to give users more control over the appearance of the rug lines. The length of the rug lines can now be controlled and it is further possible to specify that they should be placed outside of the plotting region:

ggplot(mtcars, aes(x = wt, y = mpg)) +
  geom_point() +
  geom_rug(sides = "tr", length = unit(1, "mm"), outside = TRUE) +
  # Need to turn clipping off if rug is outside plot area
  coord_cartesian(clip = "off")

Aesthetics will now accept a function returning NULL, and treat it as setting the aesthetic to NULL. This can make it easier to program with ggplot2 e.g. by catching errors from non-existing variables:

df <- data.frame(x = 1:10, y = 1:10)
wrap <- function(...) tryCatch(..., error = function(e) NULL)

ggplot(df, aes(x, y, colour = wrap(no_such_column))) +
  geom_point()

Lastly, stat_function() gains the ability to accept purrr-style lambda functions. The use of the formula notation for creating lambda functions has become widespread, and it is only natural that ggplot2 accepts it as well:

df <- data.frame(x = 1:10, y = (1:10)^2)
ggplot(df, aes(x, y)) +
  geom_point() +
  stat_function(fun = ~ .x^2)

Minor fixes and improvements

coord_sf now behaves the same as the other coords in relation to how it draws grid lines. This means that the aesthetics of the grid matches that of the other coordinate systems, and that it can be turned off through the theming system. You may see slight visual changes when using geom_sf() as the default grid lines were slightly thicker in coord_sf() prior to this change.

library(sf)
#> Linking to GEOS 3.6.1, GDAL 2.1.3, PROJ 4.9.3

nc <- st_read(system.file("gpkg/nc.gpkg", package = "sf"), quiet = TRUE)

ggplot(nc) +
  geom_sf(data = nc, aes(fill = AREA)) +
  theme_void()

The automatic naming of scales has been refined and no longer contains back-ticks when the scale name is based on a complex aesthetic expression (e.g. aes(x = a + b)). Again, this may result in slight changes to the visual appearance of plots, but only on a very superficial level.

Acknowledgements

Thank you to the 171 people who who contributed issues, code and comments to this release: @abl0719, @agila5, @ahmohamed, @amysheep, @andhamel, @anthonytw, @Atomizer15, @atusy, @baderstine, @bakaburg1, @batpigandme, @bdschwartzkopf, @beckymaust, @behrman, @bfgray3, @billdenney, @Bisaloo, @bjreisman, @blueskypie, @brfitzpatrick, @brianwdavis, @brodieG, @caayala, @cderv, @chambox, @clairemcwhite, @clauswilke, @codetrainee, @ColinFay, @connorlewis, @coolbutuseless, @courtiol, @cschwarz-stat-sfu-ca, @cystein, @dan-booth, @daniel-wells, @DanielReedOcean, @danielsjf, @daranzolin, @dariyasydykova, @dempseynoel, @dipterix, @dirkschumacher, @dkahle, @dominicroye, @dongzhuoer, @dpseidel, @dvcv, @efehandanisman, @Eisit, @Eli-Berkow, @eliocamp, @ellessenne, @eoppe1022, @felipegerard, @fereshtehizadi, @flying-sheep, @fmmattioni, @foehnwind, @fostvedt, @gagnagaman, @gangstertiny, @GBuchanon, @gdkrmr, @gdmcdonald, @ghost, @gibran-ali, @hadley, @HaoLi111, @has2k1, @heavywatal, @Henrik-P, @HenrikBengtsson, @hlendway, @hoasxyz, @hwt, @hyiltiz, @idavydov, @Ilia-Kosenkov, @IndrajeetPatil, @ismayc, @JamesCuster, @jarauh, @JayVii, @jmarshallnz, @JohnMount, @jonoyuan, @jprice80, @jpritikin, @jrnold, @jsekamane, @jtelleria, @jugularvein, @karawoo, @Katiedaisey, @katossky, @kdarras, @kendonB, @kilterwind, @LDalby, @linzi-sg, @lionel-, @llendway, @llrs, @lpantano, @LuffyLuffy, @lwjohnst86, @lz1nwm, @m-macaskill, @malcolmbarrett, @martin-ueding, @matthewParksViome, @maxheld83, @MaximOtt, @mbertolacci, @mcguinlu, @MCM-Math, @miguelmorin, @mine-cetinkaya-rundel, @mitchelloharawild, @mluerig, @moodymudskipper, @njtierney, @noahmotion, @npjc, @oranwutang, @osorensen, @pachamaltese, @paleolimbot, @PaulLantos, @peterhurford, @phmarek, @powerbilayeredmap, @ppanko, @ptoche, @puhachov, @rajkstats, @richierocks, @sabahzero, @sahilseth, @SavasAli, @sctyner, @sebneus, @shauyin520, @sjackman, @skanskan, @slowkow, @smouksassi, @sn248, @sowla, @sschloss1, @StefanBRas, @steffilazerte, @tcastrosantos, @thomasp85, @topepo, @Torvaney, @touala, @tungmilan, @vanzanden, @vbuchhold, @W-L, @wongjingping, @wrightaprilm, @x1o, @yosukefk, @yudong862, @yutannihilation, @zeehio, @zlskidmore, and @zuccaval.

It depends - A dialog about dependencies

Wed, 29 May 2019 00:00:00 +0000

It depends - A dialog about dependencies

This post paraphrases my talk with the same name at rstudio::conf 2019.

Software dependencies can be a double-edged sword.

On one hand, they let you take advantage of others’ work, giving your software marvelous new features and reducing bugs.

On the other hand, they can change over time, causing your software to break unexpectedly and (potentially) increasing your maintenance burden. They can also take additional disk space and installation time.

These downsides have led some to suggest a ‘dependency zero’ mindset.

We feel that this is a major mistake for most projects, and will instead increase maintenance burden, lower functionality and introduce additional bugs.

Dependencies are not equal

One issue with striving for minimal dependencies is it treats all dependencies as equivalent. However, in reality this is far from the case. There are many axes upon which dependencies can differ, but some of the most important include:

The type of the dependency. Some dependencies come bundled with R itself (e.g. base, utils, stats) or are part of the ‘Recommended’ packages (e.g. Matrix, survival). These packages are very low cost to depend on, as they are (nearly) universally installed on all users systems, and change generally only with new R versions. In contrast dependencies can also come from CRAN or even from non-CRAN repositories such as Bioconductor, Omegahat or drat which require users to configure additional repositories before installation.
The number of upstream dependencies. For example, the rlang package has no upstream dependencies apart from R; while the CNVscope package has 125 upstream dependencies, on both CRAN and Bioconductor.
Already fulfilled dependencies. If your package depends on dplyr then taking an additional dependency on tibble comes at no additional cost, as dplyr itself already depends on tibble. Additionally some of the most popular packages (e.g. ggplot2) will already be installed on the majority of users’ machines. So adding a ggplot2 dependency is unlikely to incur additional installation costs in most cases.
The time taken to compile the package from source. Packages containing R code generally install in roughly the same amount of time, but those that contain C, C++ or Fortran code can take very different amounts of time depending on the complexity of the code. For example, the glue package takes ~5 seconds to compile on CRAN’s build machines, whereas the readr package takes ~500 seconds to install on the same machines.
The size of the built binary package. Users installing binary packages need to download them, so the size of the built binary is relevant, particularly for those with slow internet connections. This also varies a great deal across packages. The awspack binary is only 8.6 Kb in size, while the h2o package is 123 Mb, and the largest binary on Bioconductor is over 4 Gb!
The system requirements to build and use the package. Some packages require additional system dependencies in order to be used. For instance, the rgdal package requires the GDAL and PROJ.4 dependencies, each of which in turn have additional system dependencies. Another example is rJava which requires a Java SDK and also has additional steps needed to configure R for the proper Java installation, which has caused installation issues for many people.

I hope the above examples illustrate that not all package dependencies are equal, and we should not treat them as such by using a simple count of dependencies as our metric.

Prefer a holistic, balanced and quantitative approach

Instead of striving for a minimal number of dependencies, I would instead suggest a more holistic, balanced, and quantitative approach.

A holistic approach looks at the project as a whole and asks who is the primary audience? If the audience is other package authors, then a leaner package with fewer dependencies may be more appropriate. If instead it is primarily data scientists or statisticians they will likely already have many popular dependencies installed and a more feature-full package is preferred.

A balanced approach understands that adding (or removing) dependencies come with trade-offs. Adding a dependency gives you additional features, bug fixes, and real-world testing, at the costs of increased installation time, disk space and maintenance if the dependency has breaking changes. In some cases it makes sense to increase dependencies for a package, even if an implementation already exists. For instance base R has a number of different implementations of non-standard evaluation with varying semantics across its functions. The same used to be true of tidyverse packages as well, but now they all depend on the implementations in the tidyselect and rlang packages, which improves consistency of the feature substantially.

In contrast, removing a dependency lowers installation time, disk space and avoids potential breaking changes. However it also removes potential features, increases bugs and takes additional development time.

A quantitative approach lets you more accurately measure your dependencies, and see which are heavier or lighter along various axes. Measuring each of the dependencies gives you more power to reason about which dependencies would be most beneficial to remove (if any).

How you weigh these costs and benefits varies by the skills and experience of the maintainer and the goals of the project.

Beware illusory superiority

When making these decisions it is important to consider the psychological phenomenon of Illusory Superiority. Illusory superiority is the idea that people (regardless of background) overestimate (sometimes greatly) their own abilities in a given field. No one is truly immune to this phenomenon, including the author of this post! This phenomenon will cause you to overestimate your ability to re-implement features, and underestimate the introduction of new bugs.

However, even if you truly are a more skilled programmer than the maintainer of a dependency, we would still advise caution. The existing package has the significant benefits of time and users. Users who have undoubtedly uncovered numerous bugs using the package in the real world, which are now fixed.

itdepends, a package to help with dependencies

The itdepends package aids you in having a holistic, balanced and quantitative approach. It provides tools to assess usage, measure weights, visualize proportions and if needed, assist removal of dependencies.

Assess usage with `itdepends::dep_usage()`

dep_usage_proj() and dep_usage_pkg() return a tibble of every function call in your project or package. This information can then be used to do things like

show which packages you are using the most in a project

library(tidyverse)
itdepends::dep_usage_pkg("ggplot2") %>%
  count(pkg, sort = TRUE)
#> # A tibble: 23 x 2
#>    pkg           n
#>    <chr>     <int>
#>  1 base      10333
#>  2 ggplot2    1225
#>  3 grid        313
#>  4 scales      118
#>  5 gtable       78
#>  6 rlang        68
#>  7 plyr         53
#>  8 stats        44
#>  9 utils        24
#> 10 grDevices    20
#> # … with 13 more rows

show the most frequently used function from each package

itdepends::dep_usage_pkg("ggplot2") %>%
  group_by(pkg) %>%
  count(fun) %>%
  top_n(1) %>%
  arrange(desc(n)) %>%
  head()
#> Selecting by n
#> # A tibble: 6 x 3
#> # Groups:   pkg [6]
#>   pkg     fun                 n
#>   <chr>   <chr>           <int>
#> 1 base    <-               1627
#> 2 grid    unit              101
#> 3 ggplot2 waiver             99
#> 4 gtable  gtable_add_grob    30
#> 5 scales  alpha              17
#> 6 utils   head               12

Measure weights with `itdepends::dep_weight()`

dep_weight() weighs the supplied packages and returns a tibble of 24 different metrics that can be used to compare them. The most important metric varies by person, so it is useful to examine the full spectrum.

weights <- itdepends::dep_weight(c("dplyr", "data.table"))
weights
#> # A tibble: 2 x 25
#>   package num_user bin_self bin_user install_self install_user  funs
#>   <chr>      <int>    <int>    <dbl>        <dbl>        <dbl> <int>
#> 1 dplyr         21  1821891 23147844        272.         436.    260
#> 2 data.t…        0  6222622  6222622         29.5         29.5   108
#> # … with 18 more variables: downloads <dbl>, last_release <dttm>,
#> #   open_issues <int>, last_updated <dttm>, stars <int>, forks <int>,
#> #   first_release <dttm>, total_releases <dbl>, releases_last_52 <int>,
#> #   num_dev <int>, install_dev <dbl>, bin_dev <dbl>, src_size <int>,
#> #   user_deps <list>, dev_deps <list>, self_timings <list>,
#> #   user_timings <list>, dev_timings <list>

The measures break down into a few rough categories:

package and repository information (package, funs, downloads, open_issues, last_updated, forks, stars)
dependency information (num_users, num_dev, user_deps, dev_deps)
compiled binary sizes (bin_self, bin_user, bin_dev)
time to compile dependencies, and the raw times (install_self, install_user, install_dev, self_timings, user_timings, dev_timings)
CRAN release information (downloads, first_release, last_release, total_releases, releases_last_52)

The total number of dependencies, binary sizes, and installation times give you a good idea the magnitude of the dependency load.

Equally important, however, are the downloads, release information and open issues, which tell you how long the package has existed, how many people are using it and how many open issues exist. Packages that have existed longer, with more users and fewer issues should generally be more reliable.

Visualize proportions with `itdepends::dep_plot()`

There are two functions available to plot the relative weights of dependencies.

dep_plot_time() plots the median compilation time on CRAN’s servers.

itdepends::dep_plot_time("dplyr")

From the above plot we see that the installation time of dplyr for users is mostly due to compiling dplyr itself. However, for development dependencies, the readr, stringi and RSQLite packages also take significant time to install.

dep_plot_size() plots the pre-compiled binary size, which is often the more useful measure, as most users install pre-compiled binaries.

itdepends::dep_plot_size("dplyr")

From this we see the largest user dependencies are BH and Rcpp, and the largest development dependencies are stringi, and two data packages, Lahman and nycflights13.

Assist removal with `itdepends::dep_locate()`

If you do decide that a given dependency is too heavy or is too unstable to keep, what are the best practices for removing it?

First carefully review the operation of the function you are replacing. Many functions in CRAN packages appear superficially similar to base equivalents, but often handle edge cases (that you’ve likely never encountered) more consistently. Next and most importantly, write tests for the existing behavior. This is an essential step which will greatly reduce the introduction of new bugs. Only once you have written the tests should you start performing the replacement. If you are porting a function with existing tests a good practice is to port its tests as well.

Then, use the dep_locate() function to find all the instances when you use functions from a given dependency in your code base. If dep_locate() is run in RStudio the function will add source markers for all the locations, which can then be clicked individually to do the replacement.

itdepends::dep_locate("purrr", path = "~/p/tidyversedashboard")

Be balanced and use data

In almost any area of life striving towards absolutes rarely provides the best outcome. Package dependencies are no exception. Keep the expected users of your package in mind, and use the tools provided in itdepends to obtain data which can help you assess which dependencies to remove (if any).

Donald Knuth has great advice about the importance of measurement during performance optimization

Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.

Performance optimization is important, but we should not seek to optimize it until we have measured our program to see where the this critical 3% lies. The same is true of dependencies, lowering the dependencies is important for some packages, however we should not do so blindly without identifying if it is truly necessary and measuring to discover what the most critical dependencies are.

Finally it is important to remember Goodhart’s Law

When a measure becomes a target, it ceases to be a good measure.

Resource Cleanup in C and the R API

Wed, 22 May 2019 00:00:00 +0000

Introduction

We have just released the cleancall package to CRAN. cleancall is a new package that helps avoiding resource leaks in the C code of your R package. In this blog post, we show some example resource leaks from base R and R packages, and also show three different ways to fix them, one of which uses the cleancall package.

The problem

When writing C code in R packages, one non-trivial issue is to make sure that resources acquired by a C function are correctly released or wrapped before giving back control to R. The difficulties partially arise from the many ways a C function can terminate and transfer control back to R:

regular termination,
error,
warning or other condition caught with tryCatch(),
debugger exit,
keyboard interruption.

Resources that need to be released include:

memory areas,
file handles,
connections (sockets),
database connections,
Windows handles for threads, events, etc.,
others.

Regular termination

Most functions terminate successfully, most of the time. Releasing resources is easy in this case, we can simply do it at the end of the function, before returning.

Error

Errors may happen two ways. They can be thrown directly, or by the R API. A function that reads a file will throw an error it if fails to open the file for reading. These errors are easy to handle, we can release all resources before throwing the error.

Errors can also be thrown by the R API. The R C API functions do not return on error, instead they perform an error (a longjmp technically), that can be caught by R or (recently) C code. Releasing resources is trickier in this case, as our C function never gets back the control after the failed R API call.

Warning or other condition caught with `tryCatch()`

Usually we assume that throwing a warning (or other non-error condition) does not cause an early exit, but this is not always the case. Warnings can be caught by an exiting tryCatch().

Debugger exit

If the C function calls back to R to evaluate R expressions, then these might invoke the debugger, see ?debug or ?trace. The Q debugger command invokes the abort restart, to get back to the top-level prompt. In this case most functions on the (R and C) stack are removed at once.

Keyboard interruption

C code can call R_CheckUserInterrupt(), and it is indeed suggested to do so during long computation or I/O. If the user initiates an interrupt via the keyboard or a signal, then R_CheckUserInterrupt() never returns, and the control goes back to top level.

To illustrate the resource cleanup problem more, we’ll show some examples from base R and CRAN packages that contain potential resource leaks.

Example 1: `download.file(method = "internal")`

The C implementation of download.file() opens a file for saving the downloaded data to, and it does not clean or even close this file on a keyboard interrupt. The file is opened here: https://github.com/wch/r-source/blob/f3a67c36c5cf4c715dac374e33168cfc348106aa/src/modules/internet/internet.c#L553

    FILE *out;
    [...]
    out = R_fopen(R_ExpandFileName(file), mode);
    [...]

We can easily show the problem in practice as well. First, we create a handy function that interrupts the current process one second after calling it. This allows us to simulate an interrupt from the keyboard. It starts a background process that will send an interrupt (SIGINT on Unix, CTRL+C on Windows) to our R process.

interrupt <- function(expr, after = 1) {
  tryCatch({
    p <- callr::r_bg(function(pid, after) {
      Sys.sleep(after)
      ps::ps_interrupt(ps::ps_handle(pid))
    }, list(pid = Sys.getpid(), after = after))
    expr
    p$kill()
  }, interrupt = function(e) e)
}

ps::ps_open_files() lists all open files of a process,ps::ps_handle() returns a handle for the current R process. You should see the three file descriptors corresponding to standard input, output and error here, and another file is used by the blogdown engine rendering this page. If you run this code in your R session you’ll not see that file, but of course you may see others.

ps::ps_open_files(ps::ps_handle())
#> # A tibble: 4 x 2
#>      fd path
#>   <int> <chr>
#> 1     0 /dev/ttys021
#> 2     1 /dev/ttys021
#> 3     2 /dev/ttys021
#> 4     7 /Users/gaborcsardi/R/blogdown/scripts/render_page.R

Now we will start downloading a file, and while it is downloading, we’ll send an interrupt to our R process, causing an early exit.

interrupt(
  download.file("http://httpbin.org/delay/3", tempfile(), method = "internal")
)
#> <interrupt: >

The list of open files will now include the one opened by download.file():

ps::ps_open_files(ps::ps_handle())
#> # A tibble: 5 x 2
#>      fd path
#>   <int> <chr>
#> 1     0 /dev/ttys021
#> 2     1 /dev/ttys021
#> 3     2 /dev/ttys021
#> 4     7 /Users/gaborcsardi/R/blogdown/scripts/render_page.R
#> 5     9 /private/var/folders/59/0gkmw1yj2w7bf2dfc3jznv5w0000gn/T/Rtmp0Uww3…

Because download.file() fails to clean up this file handle, it is not released until the R session quits. On Windows this file is probably locked, and cannot be removed until R exits.

Example 2: `download.file(method = "libcurl")`

There is a very similar bug in the implementation of the download.file() libcurl method, which opens an output file here: https://github.com/wch/r-source/blob/def075db88ae87104c38437a57e9327b078bb804/src/modules/internet/libcurl.c#L566 but it does not close it on error or interrupt. This method can download many files in parallel, in which case multiple file handles may be lost:

ps::ps_open_files(ps::ps_handle())
#> # A tibble: 5 x 2
#>      fd path
#>   <int> <chr>
#> 1     0 /dev/ttys021
#> 2     1 /dev/ttys021
#> 3     2 /dev/ttys021
#> 4     7 /Users/gaborcsardi/R/blogdown/scripts/render_page.R
#> 5     9 /private/var/folders/59/0gkmw1yj2w7bf2dfc3jznv5w0000gn/T/Rtmp0Uww3…
interrupt(
  download.file(
    rep("https://httpbin.org/delay/3", 3), paste0(tempfile(), 1:3),
    method = "libcurl")
)
#> <interrupt: >
ps::ps_open_files(ps::ps_handle())
#> # A tibble: 8 x 2
#>      fd path
#>   <int> <chr>
#> 1     0 /dev/ttys021
#> 2     1 /dev/ttys021
#> 3     2 /dev/ttys021
#> 4     7 /Users/gaborcsardi/R/blogdown/scripts/render_page.R
#> 5     9 /private/var/folders/59/0gkmw1yj2w7bf2dfc3jznv5w0000gn/T/Rtmp0Uww3…
#> 6    15 /private/var/folders/59/0gkmw1yj2w7bf2dfc3jznv5w0000gn/T/Rtmp0Uww3…
#> 7    16 /private/var/folders/59/0gkmw1yj2w7bf2dfc3jznv5w0000gn/T/Rtmp0Uww3…
#> 8    17 /private/var/folders/59/0gkmw1yj2w7bf2dfc3jznv5w0000gn/T/Rtmp0Uww3…

Example 3: `edit()`

edit() invokes the text editor specified by the editor option, with the supplied object to be edited. edit() can also write the edited version to a file. It opens a file here to create its first version, before starting the editor: https://github.com/wch/r-source/blob/019f16d3dec4f97c1b4e4f0ec905148e36979e37/src/main/edit.c#L121-L128

121    if((fp=R_fopen(R_ExpandFileName(filename), "w")) == NULL)
122        errorcall(call, _("unable to open file"));
123    if (LENGTH(STRING_ELT(fn, 0)) == 0) EdFileUsed++;
124    PROTECT(src = deparse1(x, 0, FORSOURCING)); /* deparse for sourcing, not for display */
125    for (i = 0; i < LENGTH(src); i++)
126        fprintf(fp, "%s\n", translateChar(STRING_ELT(src, i)));
127    UNPROTECT(1); /* src */
128    fclose(fp);

If all goes well, then it will close it just 7 code lines later. However, it also calls the R API a number of times before closing the file, so if any of these API calls fail, the file is never closed. In particular, it calls deparse1() which is roughly equivalent to the R deparse() function, and deparse()` fails on long vectors:

ps::ps_open_files(ps::ps_handle())
#> # A tibble: 8 x 2
#>      fd path
#>   <int> <chr>
#> 1     0 /dev/ttys021
#> 2     1 /dev/ttys021
#> 3     2 /dev/ttys021
#> 4     7 /Users/gaborcsardi/R/blogdown/scripts/render_page.R
#> 5     9 /private/var/folders/59/0gkmw1yj2w7bf2dfc3jznv5w0000gn/T/Rtmp0Uww3…
#> 6    15 /private/var/folders/59/0gkmw1yj2w7bf2dfc3jznv5w0000gn/T/Rtmp0Uww3…
#> 7    16 /private/var/folders/59/0gkmw1yj2w7bf2dfc3jznv5w0000gn/T/Rtmp0Uww3…
#> 8    17 /private/var/folders/59/0gkmw1yj2w7bf2dfc3jznv5w0000gn/T/Rtmp0Uww3…
edit(1:10e10, file = tempfile())
#> Error in edit.default(1:1e+11, file = tempfile()): long vectors not supported yet: ../../../../R-3.5.3/src/include/Rinlinedfuns.h:519
ps::ps_open_files(ps::ps_handle())
#> # A tibble: 9 x 2
#>      fd path
#>   <int> <chr>
#> 1     0 /dev/ttys021
#> 2     1 /dev/ttys021
#> 3     2 /dev/ttys021
#> 4     7 /Users/gaborcsardi/R/blogdown/scripts/render_page.R
#> 5     8 /private/var/folders/59/0gkmw1yj2w7bf2dfc3jznv5w0000gn/T/Rtmp0Uww3…
#> 6     9 /private/var/folders/59/0gkmw1yj2w7bf2dfc3jznv5w0000gn/T/Rtmp0Uww3…
#> 7    15 /private/var/folders/59/0gkmw1yj2w7bf2dfc3jznv5w0000gn/T/Rtmp0Uww3…
#> 8    16 /private/var/folders/59/0gkmw1yj2w7bf2dfc3jznv5w0000gn/T/Rtmp0Uww3…
#> 9    17 /private/var/folders/59/0gkmw1yj2w7bf2dfc3jznv5w0000gn/T/Rtmp0Uww3…

Notice that we have one more extra open file here.

Example 4: the wait() method in processx:

processx::process is an R6 class for a subprocess. Its $wait() method waits for the subprocess to finish, with a timeout. To implement $wait() on Unix, processx opens a pair of pipe file descriptors. These are temporary and should be closed once the function exits. $wait() is interruptible, it calls R_CheckUserInterrupt() periodically. However, in the current, 3.3.0 version of processx it does not close the pipe file descriptors on an interrupt. Here is an illustration:

ps::ps_num_fds(ps::ps_handle())
#> [1] 20

p <- processx::process$new("sleep", "10")
interrupt(
  p$wait()
)
#> <interrupt: >
p$kill()
#> [1] TRUE
gc()
#>           used (Mb) gc trigger (Mb) limit (Mb) max used (Mb)
#> Ncells  626041 33.5    1203691 64.3         NA  1203691 64.3
#> Vcells 1188386  9.1    8388608 64.0      16384  2160124 16.5

ps::ps_num_fds(ps::ps_handle())
#> [1] 22

ps_num_fds() prints the number of open file descriptors of a process. The two extra file descriptors that are open after garbage collection are the two ends of the (supposedly) temporary pipe, just opened by $wait().

Fix 1: External pointer and finalizer

One generic solution to resource cleanup is to wrap all C resources into one or more R external pointer objects, and add finalizers to them. We show how this can fix the processx problem in the previous example.

This is how the processx_wait() C function looks like before the fix: https://github.com/r-lib/processx/blob/a8f09d147fead78347a87fcf4e0fbd1c07de1c21/src/unix/processx.c#L507-L589

First, we need to create a finalizer function, that will be called by the R garbage collector, after $wait() has finished, at the next garbage collection:

static void processx__wait_finalizer(SEXP xptr) {
  SEXP tag = R_ExternalPtrTag(xptr);
  if (INTEGER(tag)[0] >= 0) close(INTEGER(tag)[0]);
  if (INTEGER(tag)[1] >= 0) close(INTEGER(tag)[1]);
}

An external pointer can have a tag, which is an R object that is kept alive as long the pointer object itself is alive. In this case we can put the file descriptors in the tag, in an integer vector of length two. In more complicated cases the resources cannot easily be represented as R objects, so you would use the actual C pointer, with a custom C struct to store them.

Now we need to create the external pointer, before the pipes are opened:

  SEXP tag = PROTECT(allocVector(INTSXP, 2));
  INTEGER(tag)[0] = INTEGER(tag)[1] = -1;
  SEXP xptr = PROTECT(R_MakeExternalPtr(NULL, tag, R_NilValue));
  R_RegisterCFinalizerEx(xptr, processx__wait_finalizer, 0);

We initialize the file descriptors to -1, which is guard value, meaning that no cleanup is needed.

We can now open the pipes and save their file descriptors in the tag of the external pointer.

  if (pipe(handle->waitpipe)) {
    processx__unblock_sigchld();
    error("processx error: %s", strerror(errno));
  }
  INTEGER(tag)[0] = handle->waitpipe[0];
  INTEGER(tag)[1] = handle->waitpipe[1];

At the end of the function, we need to unprotect the external pointer, and the tag:

  UNPROTECT(2);
  return ScalarLogical(ret != 0);
}

One potential problem with the external pointer fix is that the resources will only be cleaned up at the next garbage collection, and sometimes this is too late. For example, if an open file is locked by the operating system, then we won’t be able to remove that file, or its directory, until the garbage collector runs and closes it. It is easier to program if resource cleanup is immediate, and luckily the other two fixes below are.

Fix 2: the `R_ExecWithCleanup()` function

R_ExecWithCleanup() is a function in the R API, that can add a cleanup function to a regular C function call. The cleanup function is always executed, even on early exit:

SEXP R_ExecWithCleanup(SEXP (*fun)(void *), void *data,
               void (*cleanfun)(void *), void *cleandata);

R_ExecWithCleanup() calls fun with data, and then calls cleanfun with cleandata. If fun calls the R API and exits early, then it will still call cleanfun with cleandata, before throwing the error up the stack.

To fix $wait() with R_ExecWithCleanup(), we’ll need to define a cleanup function first. The cleanup function receives the cleanup data as a void pointer, we’ll define a struct for this. For simplicity, this struct will also include the arguments to processx_wait(), so we can use the same struct for both fun and cleanupfun.

struct processx_wait_data {
  SEXP status;
  SEXP timeout;
  int fds[2];
};

void processx_wait_cleanup(void *data) {
  struct processx_wait_data *pdata = data;
  if (pdata->fds[0] >= 0) close(pdata->fds[0]);
  if (pdata->fds[1] >= 0) close(pdata->fds[1]);
}

The new processx_wait() function will call R_ExecWithCleanup(), to call the original processx_wait(), under a new name:

SEXP processx_wait_internal(void *data);
SEXP processx_wait(SEXP status, SEXP timeout) {
  struct processx_wait_data pdata = { status, timeout, { -1, -1 } };
  SEXP result = R_ExecWithCleanup(processx_wait_internal, &pdata,
                                  processx_wait_cleanup, &pdata);
  return result;
}

processx_wait_internal() is very much like processx_wait() used to be, but we need to extract the arguments from the struct at the beginning:

SEXP processx_wait_internal(void *data) {
  struct processx_wait_data *pdata = data;
  SEXP status = pdata->status;
  SEXP timeout = pdata->timeout;
  int *fds = pdata->fds;
  ...

Now the only other thing we need to do is saving the file descriptors in the data struct:

  if (pipe(handle->waitpipe)) {
    processx__unblock_sigchld();
    error("processx error: %s", strerror(errno));
  }
  fds[0] = handle->waitpipe[0];
  fds[1] = handle->waitpipe[1];

Fix 3: the cleancall package

R_ExecWithCleanup() is a good fix to the resource cleanup problem, but it can be verbose and error prone. It also requires that you replace your original function with a wrapper that packs the original function arguments into a struct and an internal function that only has a void* argument.

We created the cleancall package to make resource cleanup easier. This package automates wrapping your functions with R_ExecWithCleanup(). cleancall has been just published on CRAN. Here we show how to use it to fix the processx $wait() method.

To use it in your package, you need to specify cleancall as a dependency, both as LinkingTo and Imports:

...
Imports: cleancall
LinkingTo: cleancall
...

cleancall defines the call_with_cleanup() R function and the r_call_on_exit() and r_call_on_early_exit() C functions.

You need to replace .Call() with call_with_cleanup() in your R code:

cleancall::call_with_cleanup(c_processx_wait, private$status,
                               as.integer(timeout))

In your C code, include the cleancall.h header:

#include <cleancall.h>

Next, create a cleanup function for the resource type:

void processx__close_fd(void *ptr) {
  int *fd = ptr;
  if (*fd >= 0) close(*fd);
}

This cleanup function closes a file descriptor. Once your resource type has a cleanup function, you can call r_call_on_exit() or r_call_on_early_exit() every time you acquire a resource of that type. Use r_call_on_exit() if the resource must be released on normal termination as well, and r_call_on_early_exit() if it must be released on early termination only:

  if (pipe(handle->waitpipe)) {
    processx__unblock_sigchld();
    error("processx error: %s", strerror(errno));
  }
  r_call_on_exit(processx__close_fd, handle->waitpipe);
  r_call_on_exit(processx__close_fd, handle->waitpipe + 1);

Typically r_call_on_exit() is more convenient for temporary resources. r_call_on_early_exit() is more convenient if the C function returns a handle (e.g. external pointer), for which it needs to allocate resources gradually. If all resource allocations are successful, and the function returns normally, then no cleanup is needed. However if an intermediate step fails, you need to release the resources acquired before the failure.

In does not matter much in this simple example, but it in important in general that exit handlers are always executed in reverse order (last one in is the first one out, LIFO). This makes it easy to build a resource gradually. Exit handlers installed via r_call_on_exit() and r_call_on_early_exit() share the same stack.

We suggest that exit handlers are kept as simple and fast as possible. In particular, errors (and other early exits) triggered from exit handlers are not caught currently. If an exit handler exits early the others do not run. If this is an issue, you can wrap the exit handler in R_tryCatch() (available for R 3.4.0 and later). But in general the best error handlers do not call the R API at all.

What about C++?

The resource cleanup problem is also present if you interface C++ code. While cleancall can be used with C++, it works best with C code, since it follows C idioms.

Using external pointers and finalizers works for C++ as well, as does wrapping R API calls in R_ExecWithCleanup().

Alternatively, you can wrap your R API calls in R_tryCatch() (R 3.4.0 and later), or use R_UnwindProtect() (R 3.5.0 and later).

If your C++ code needs to support older R versions, that is more challenging, and one possibility is to call back to R and call tryCatch() there.

Summary

Resource cleanup in C code can be challenging, especially given that there is not very much documentation on this topic.

Hopefully this post and the cleancall package will make resource cleanup much simpler, and fewer R packages will suffer from resource leaks in the future.

dplyr 0.8.1

Fri, 17 May 2019 00:00:00 +0000

Introduction

We’re delighted to announce the release of dplyr 0.8.1 on CRAN 🎉 !

This is a minor release that address follow ups from the community after the release of the 0.8.0 version.

group_map() and group_modify()

Shortly after the release of 0.8.0, we were notified by several members of the community that group_map() was great, except it didn’t do what they had expected 😬.

Because the function was (and still is) marked as experimental, we allowed ourselves to rectify the situation:

The name group_map() is now used for iterating on groups of grouped tibbles, characterised by .x and .y as before, but making no assumptions about the return type of each operation and combining the results in a list. We can see this as iterating, in the purrr::map() sense on the groups.

library(dplyr, warn.conflicts = FALSE)

# a list of vectors
iris %>%
  group_by(Species) %>%
  group_map(~ quantile(.x$Petal.Length, probs = c(0.25, 0.5, 0.75)))
#> [[1]]
#>   25%   50%   75% 
#> 1.400 1.500 1.575 
#> 
#> [[2]]
#>  25%  50%  75% 
#> 4.00 4.35 4.60 
#> 
#> [[3]]
#>   25%   50%   75% 
#> 5.100 5.550 5.875

The behaviour we previously had was renamed group_modify() to loosely echo purrr::modify(). In particular, group_modify() always returns a grouped tibble, which combines the tibbles returned by evaluating each operation with a reconstructed grouping structure.

# to use group_modify() the lambda must return a data frame
iris %>%
  group_by(Species) %>%
  group_modify(~ {
     quantile(.x$Petal.Length, probs = c(0.25, 0.5, 0.75)) %>%
     tibble::enframe(name = "prob", value = "quantile")
  })
#> # A tibble: 9 x 3
#> # Groups:   Species [3]
#>   Species    prob  quantile
#>   <fct>      <chr>    <dbl>
#> 1 setosa     25%       1.4 
#> 2 setosa     50%       1.5 
#> 3 setosa     75%       1.58
#> 4 versicolor 25%       4   
#> 5 versicolor 50%       4.35
#> 6 versicolor 75%       4.6 
#> 7 virginica  25%       5.1 
#> 8 virginica  50%       5.55
#> 9 virginica  75%       5.88

Attention to details in column wise functions

As we are phasing funs() out and prefer use of purrr-style lambda functions in column wise verbs, we missed a few subtleties.

Specifically, lambdas can now refer to:

local variables (from the scope):

to_inch <- function(data, ...) {
  # the local variable `inch` can be used in the lambda
  inch <- 0.393701
  data %>% 
    mutate_at(vars(...), ~ . * inch)
}
iris %>% 
  as_tibble() %>% 
  to_inch(-Species)
#> # A tibble: 150 x 5
#>    Sepal.Length Sepal.Width Petal.Length Petal.Width Species
#>           <dbl>       <dbl>        <dbl>       <dbl> <fct>  
#>  1         2.01        1.38        0.551      0.0787 setosa 
#>  2         1.93        1.18        0.551      0.0787 setosa 
#>  3         1.85        1.26        0.512      0.0787 setosa 
#>  4         1.81        1.22        0.591      0.0787 setosa 
#>  5         1.97        1.42        0.551      0.0787 setosa 
#>  6         2.13        1.54        0.669      0.157  setosa 
#>  7         1.81        1.34        0.551      0.118  setosa 
#>  8         1.97        1.34        0.591      0.0787 setosa 
#>  9         1.73        1.14        0.551      0.0787 setosa 
#> 10         1.93        1.22        0.591      0.0394 setosa 
#> # … with 140 more rows

other columns of the data (from the data mask):

iris %>% 
  as_tibble() %>% 
  mutate_at(vars(starts_with("Sepal")), ~ . / Petal.Width)
#> # A tibble: 150 x 5
#>    Sepal.Length Sepal.Width Petal.Length Petal.Width Species
#>           <dbl>       <dbl>        <dbl>       <dbl> <fct>  
#>  1         25.5       17.5           1.4         0.2 setosa 
#>  2         24.5       15             1.4         0.2 setosa 
#>  3         23.5       16             1.3         0.2 setosa 
#>  4         23.0       15.5           1.5         0.2 setosa 
#>  5         25         18             1.4         0.2 setosa 
#>  6         13.5        9.75          1.7         0.4 setosa 
#>  7         15.3       11.3           1.4         0.3 setosa 
#>  8         25         17             1.5         0.2 setosa 
#>  9         22         14.5           1.4         0.2 setosa 
#> 10         49         31             1.5         0.1 setosa 
#> # … with 140 more rows

Thanks

Thanks to all contributors for this release.

@abalter, @ambevill, @amitusa17, @AntoineHffmnn, @anuj2054, @batpigandme, @behrman, @billdenney, @burchill, @cgrandin, @clemenshug, @codetrainee, @ColinFay, @dan-reznik, @davidsjoberg, @DesiQuintans, @dirkschumacher, @earowang, @echasnovski, @eipi10, @grabear, @grandtiger, @gregorp, @hadley, @hanyroze, @hidekoji, @huftis, @iago-pssjd, @javierluraschi, @jennybc, @jgellar, @jhrcook, @jimhester, @joel23888, @JohnMount, @johnmous, @jonathan-g, @jwbeck97, @jzadra, @karimn, @kendonB, @koncina, @kperkins, @kputschko, @krlmlr, @kyzphong, @lionel-, @llrs, @mariodejung, @MichaelAdolph, @michaelwhammer, @MilesMcBain, @mjherold, @moodymudskipper, @msberends, @mvkorpel, @nathancday, @nicokuz, @nolistic, @oscci, @paulponcet, @PhilippRuchser, @philstraforelli, @psychometrician, @Ranonymous, @rinebob, @romagnolid, @romainfrancois, @rvg02010, @slyrus, @snp, @sowla, @ThiAmm, @thothal, @wfmackey, @will458, @wkdavis, @yutannihilation, @ZahraEconomist, and @zooman.

forcats 0.4.0

Fri, 10 May 2019 00:00:00 +0000

We are pleased to announce that forcats 0.4.0 is now on CRAN. The forcats package provides a suite of useful tools that solve common problems with factors in R. This version benefited from the hard work of contributors new and old at our first tidyverse dev day. For a complete set of changes, please see the release notes.

To install the latest version, run:

install.packages("forcats")

As always, attach the package with:

library(forcats)

New functions

fct_cross() creates a new factor containing the combined levels from two or more input factors, similar to base::interaction().

fruit <- factor(c("apple", "kiwi", "apple", "apple"))
colour <- factor(c("green", "green", "red", "green"))
fct_cross(fruit, colour)
#> [1] apple:green kiwi:green  apple:red   apple:green
#> Levels: apple:green apple:red kiwi:green

fct_lump_min() preserves levels that appear at least min times (can also be used with the w weighted argument).

x <- factor(letters[rpois(50, 3)])
fct_lump_min(x, min = 10)
#>  [1] Other b     Other b     Other Other Other b     Other Other b    
#> [12] Other Other Other b     Other b     Other Other b     b     Other
#> [23] Other Other b     b     Other Other Other Other Other b     Other
#> [34] Other Other b     Other Other Other Other Other Other Other Other
#> [45] Other b     Other b    
#> Levels: b Other

fct_match() tests for the presence of levels in a factor, providing a safer alternative to %in% by throwing an error when there are unexpected levels.

table(fct_match(gss_cat$marital, c("Married", "Divorced")))
#> 
#> FALSE  TRUE 
#>  7983 13500
table(gss_cat$marital %in% c("Maried", "Davorced"))
#> 
#> FALSE 
#> 21483
table(fct_match(gss_cat$marital, c("Maried", "Davorced")))
#> Error: Levels not present in factor: "Maried", "Davorced"

Other improvements

fct_relevel() can now relevel factors using a function that is passed the current levels.

f <- factor(c("a", "b", "c", "d"), levels = c("b", "c", "d", "a"))
fct_relevel(f, sort)
#> [1] a b c d
#> Levels: a b c d
fct_relevel(f, rev)
#> [1] a b c d
#> Levels: a d c b

as_factor() now has a numeric method which orders factors in numeric order, unlike the other methods which default to order of appearance.

y <- c("1.1", "11", "2.2", "22")
as_factor(y)
#> [1] 1.1 11  2.2 22 
#> Levels: 1.1 11 2.2 22
z <- as.numeric(y)
as_factor(z)
#> [1] 1.1 11  2.2 22 
#> Levels: 1.1 2.2 11 22

fct_inseq() reorders labels numerically, when possible.

Thanks to Emily Robinson, forcats also has a new introductory vignette.

Acknowledgements

We’re grateful for the 35 people who contributed to this release: @ahaque-utd, @AmeliaMN, @ashiklom, @batpigandme, @billdenney, @brianwdavis, @corybrunson, @dalewsteele, @ewenharrison, @grayskripko, @gtm19, @hack-r, @hadley, @huftis, @isteves, @jimhester, @jonocarroll, @jrosen48, @jthomasmock, @kbodwin, @mdjeric, @orchid00, @richierocks, @robinsones, @rosedu1, @RoyalTS, @russHyde, @Ryo-N7, @s-fleck, @seaaan, @spedygiorgio, @tslumley, @xuhuizhang, @zhiiiyang, and @zx8754.

vroom 1.0.0

Tue, 07 May 2019 00:00:00 +0000

I’m excited to announce that vroom 1.0.0 is now available on CRAN!

vroom reads rectangular data, such as comma separated (csv), tab separated (tsv) or fixed width files (fwf) into R. It performs similar roles to functions like readr::read_csv(), data.table::fread() or read.csv(). But for many datasets vroom::vroom() can read them much, much faster (hence the name).

The main reason vroom can be faster is because character data is read from the file lazily; you only pay for the data you use. This lazy access is done automatically, so no changes to your R data-manipulation code are needed.

vroom also provides efficient, multi-threaded writing that is multiple times faster on most inputs than the readr::write_*() functions.

Install vroom with:

install.packages("vroom")

The best way to get acquainted with the package is the getting started vignette.

vroom vs readr

What does the release of vroom mean for readr? For now we plan to let the two packages evolve separately, but likely we will unite the packages in the future. One disadvantage to vroom’s lazy reading is certain data problems can’t be reported up front, so how best to unify them requires some thought.

Reading delimited files

Compared to readr, the first difference you may note is you use only one function to read the files, vroom(). This is because vroom() guesses the delimiter of the file automatically based on the first few lines (this feature is inspired by a similar feature in data.table::fread()). This works well most of the time, but may fail to guess properly in some cases. The delim argument can be used to specify the delimiter of the file explicitly.

library(vroom)

data <- vroom("flights.tsv")
#> Observations: 336,776
#> Variables: 19
#> chr  [ 4]: carrier, tailnum, origin, dest
#> dbl  [14]: year, month, day, dep_time, sched_dep_time, dep_delay, arr_time, sched_arr...
#> dttm [ 1]: time_hour
#> 
#> Call `spec()` for a copy-pastable column specification
#> Specify the column types with `col_types` to quiet this message

The summary message after reading also differs from readr. We hope this output gives a more informative indication as to whether the types of your columns are being guessed properly. However you can still retrieve and print the full column specification with spec().

spec(data)
#> cols(
#>   year = col_double(),
#>   month = col_double(),
#>   day = col_double(),
#>   dep_time = col_double(),
#>   sched_dep_time = col_double(),
#>   dep_delay = col_double(),
#>   arr_time = col_double(),
#>   sched_arr_time = col_double(),
#>   arr_delay = col_double(),
#>   carrier = col_character(),
#>   flight = col_double(),
#>   tailnum = col_character(),
#>   origin = col_character(),
#>   dest = col_character(),
#>   air_time = col_double(),
#>   distance = col_double(),
#>   hour = col_double(),
#>   minute = col_double(),
#>   time_hour = col_datetime(format = "")
#> )

The message will be disabled if you supply a column specification to col_types when reading.

s <- spec(data)

data <- vroom("flights.tsv", col_types = s)

Reading multiple files

One feature new to vroom is built-in support for reading sets of files with the same columns into one table. Just pass the filenames to be read directly to vroom(). Imagine we have a directory of files containing the flights data, where each file corresponds to a single airline.

Then, we can efficiently read all of the files into one tibble by passing a vector of the filenames directly to vroom().

files <- fs::dir_ls(glob = "flights_*tsv")
files
#> flights_9E.tsv flights_AA.tsv flights_AS.tsv flights_B6.tsv flights_DL.tsv 
#> flights_EV.tsv flights_F9.tsv flights_FL.tsv flights_HA.tsv flights_MQ.tsv 
#> flights_OO.tsv flights_UA.tsv flights_US.tsv flights_VX.tsv flights_WN.tsv 
#> flights_YV.tsv
data <- vroom(files)
#> Observations: 336,776
#> Variables: 19
#> chr  [ 4]: carrier, tailnum, origin, dest
#> dbl  [14]: year, month, day, dep_time, sched_dep_time, dep_delay, arr_time, sched_arr...
#> dttm [ 1]: time_hour
#> 
#> Call `spec()` for a copy-pastable column specification
#> Specify the column types with `col_types` to quiet this message

Reading and writing compressed files

Just like readr, vroom automatically reads and writes zip, gzip, bz2 and xz compressed files with the standard file extensions.

vroom_write(flights, "flights.tsv.gz")

# Check file sizes to show file is compressed
fs::file_size(c("flights.tsv", "flights.tsv.gz"))
#> 29.62M  7.87M

# Read the file back in
data <- vroom("flights.tsv.gz")
#> Observations: 336,776
#> Variables: 19
#> chr  [ 4]: carrier, tailnum, origin, dest
#> dbl  [14]: year, month, day, dep_time, sched_dep_time, dep_delay, arr_time, sched_arr...
#> dttm [ 1]: time_hour
#> 
#> Call `spec()` for a copy-pastable column specification
#> Specify the column types with `col_types` to quiet this message

Reading remote files

vroom can also read files from the internet as well by passing the URL of the file to vroom().

file <- "https://raw.githubusercontent.com/r-lib/vroom/master/inst/extdata/mtcars.csv"
data <- vroom(file)
#> Observations: 32
#> Variables: 12
#> chr [ 1]: model
#> dbl [11]: mpg, cyl, disp, hp, drat, wt, qsec, vs, am, gear, carb
#> 
#> Call `spec()` for a copy-pastable column specification
#> Specify the column types with `col_types` to quiet this message

It can even read gzipped files from the internet (although currently not the other compressed formats).

Reading and writing from pipe connections

vroom provides efficient input and output from pipe() connections.

This is useful for doing things like pre-filtering large inputs with command line tools like grep.

# Return only flights on United Airlines
data <- vroom(pipe("grep -w UA flights.tsv"), col_names = names(flights))
#> Observations: 58,665
#> Variables: 19
#> chr  [ 4]: carrier, tailnum, origin, dest
#> dbl  [14]: year, month, day, dep_time, sched_dep_time, dep_delay, arr_time, sched_arr...
#> dttm [ 1]: time_hour
#> 
#> Call `spec()` for a copy-pastable column specification
#> Specify the column types with `col_types` to quiet this message

Or using multi-threaded compression programs like pigz, which can greatly reduce the time to write compressed files.

bench::workout({
  vroom_write(flights, "flights.tsv.gz")
  vroom_write(flights, pipe("pigz > flights.tsv.gz"))
})
#> # A tibble: 2 x 3
#>   exprs                                                process     real
#>   <bch:expr>                                          <bch:tm> <bch:tm>
#> 1 vroom_write(flights, "flights.tsv.gz")                  3.5s    2.69s
#> 2 vroom_write(flights, pipe("pigz > flights.tsv.gz"))    1.54s 975.09ms

Column selection

vroom introduces a new argument, col_select, which makes selecting columns to keep (or omit) more straightforward.

col_select uses the same interface as dplyr::select(), so you can do flexible selection operations.

Select with the column names

data <- vroom("flights.tsv", col_select = c(year, flight, tailnum))
#> Observations: 336,776
#> Variables: 3
#> chr [1]: tailnum
#> dbl [2]: year, flight
#> 
#> Call `spec()` for a copy-pastable column specification
#> Specify the column types with `col_types` to quiet this message

Drop columns by name

data <- vroom("flights.tsv", col_select = c(-dep_time, -air_time:-time_hour))
#> Observations: 336,776
#> Variables: 13
#> chr [4]: carrier, tailnum, origin, dest
#> dbl [9]: year, month, day, sched_dep_time, dep_delay, arr_time, sched_arr_time, arr...
#> 
#> Call `spec()` for a copy-pastable column specification
#> Specify the column types with `col_types` to quiet this message

Use the selection helpers

data <- vroom("flights.tsv", col_select = ends_with("time"))
#> Observations: 336,776
#> Variables: 5
#> dbl [5]: dep_time, sched_dep_time, arr_time, sched_arr_time, air_time
#> 
#> Call `spec()` for a copy-pastable column specification
#> Specify the column types with `col_types` to quiet this message

Or rename columns

data <- vroom("flights.tsv", col_select = list(plane = tailnum, everything()))
#> Observations: 336,776
#> Variables: 19
#> chr  [ 4]: carrier, tailnum, origin, dest
#> dbl  [14]: year, month, day, dep_time, sched_dep_time, dep_delay, arr_time, sched_arr...
#> dttm [ 1]: time_hour
#> 
#> Call `spec()` for a copy-pastable column specification
#> Specify the column types with `col_types` to quiet this message
data
#> # A tibble: 336,776 x 19
#>    plane  year month   day dep_time sched_dep_time dep_delay arr_time
#>    <chr> <dbl> <dbl> <dbl>    <dbl>          <dbl>     <dbl>    <dbl>
#>  1 N142…  2013     1     1      517            515         2      830
#>  2 N242…  2013     1     1      533            529         4      850
#>  3 N619…  2013     1     1      542            540         2      923
#>  4 N804…  2013     1     1      544            545        -1     1004
#>  5 N668…  2013     1     1      554            600        -6      812
#>  6 N394…  2013     1     1      554            558        -4      740
#>  7 N516…  2013     1     1      555            600        -5      913
#>  8 N829…  2013     1     1      557            600        -3      709
#>  9 N593…  2013     1     1      557            600        -3      838
#> 10 N3AL…  2013     1     1      558            600        -2      753
#> # … with 336,766 more rows, and 11 more variables: sched_arr_time <dbl>,
#> #   arr_delay <dbl>, carrier <chr>, flight <dbl>, origin <chr>,
#> #   dest <chr>, air_time <dbl>, distance <dbl>, hour <dbl>, minute <dbl>,
#> #   time_hour <dttm>

Name repair

Often the names of columns in the original dataset are not ideal to work with. vroom() uses the same .name_repair argument as tibble, so you can use one of the default name repair strategies or provide a custom function. A great approach is to use the janitor make_clean_names() function as the input.

vroom("flights.tsv", .name_repair = janitor::make_clean_names)
#> # A tibble: 336,776 x 19
#>     year month   day dep_time sched_dep_time dep_delay arr_time
#>    <dbl> <dbl> <dbl>    <dbl>          <dbl>     <dbl>    <dbl>
#>  1  2013     1     1      517            515         2      830
#>  2  2013     1     1      533            529         4      850
#>  3  2013     1     1      542            540         2      923
#>  4  2013     1     1      544            545        -1     1004
#>  5  2013     1     1      554            600        -6      812
#>  6  2013     1     1      554            558        -4      740
#>  7  2013     1     1      555            600        -5      913
#>  8  2013     1     1      557            600        -3      709
#>  9  2013     1     1      557            600        -3      838
#> 10  2013     1     1      558            600        -2      753
#> # … with 336,766 more rows, and 12 more variables: sched_arr_time <dbl>,
#> #   arr_delay <dbl>, carrier <chr>, flight <dbl>, tailnum <chr>,
#> #   origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>, hour <dbl>,
#> #   minute <dbl>, time_hour <dttm>

vroom("flights.tsv", .name_repair = ~ janitor::make_clean_names(., case = "lower_camel"))
#> # A tibble: 336,776 x 19
#>     year month   day depTime schedDepTime depDelay arrTime schedArrTime
#>    <dbl> <dbl> <dbl>   <dbl>        <dbl>    <dbl>   <dbl>        <dbl>
#>  1  2013     1     1     517          515        2     830          819
#>  2  2013     1     1     533          529        4     850          830
#>  3  2013     1     1     542          540        2     923          850
#>  4  2013     1     1     544          545       -1    1004         1022
#>  5  2013     1     1     554          600       -6     812          837
#>  6  2013     1     1     554          558       -4     740          728
#>  7  2013     1     1     555          600       -5     913          854
#>  8  2013     1     1     557          600       -3     709          723
#>  9  2013     1     1     557          600       -3     838          846
#> 10  2013     1     1     558          600       -2     753          745
#> # … with 336,766 more rows, and 11 more variables: arrDelay <dbl>,
#> #   carrier <chr>, flight <dbl>, tailnum <chr>, origin <chr>, dest <chr>,
#> #   airTime <dbl>, distance <dbl>, hour <dbl>, minute <dbl>,
#> #   timeHour <dttm>

Column types

Like readr, vroom guesses the data types of columns as they are read. readr simply used the first n rows of data, vroom uses an improved heuristic of looking at data throughout the file, which should improve guessing accuracy. However if the guessing fails it can be necessary to change the type of one or more columns.

The available specifications are: (with single letter abbreviations in quotes)

col_logical() ‘l’, containing only T, F, TRUE, FALSE, 1 or 0.
col_integer() ‘i’, integer values.
col_double() ‘d’, floating point values.
col_number() [n], numbers containing the grouping_mark
col_date(format = "") [D]: with the locale’s date_format.
col_time(format = "") [t]: with the locale’s time_format.
col_datetime(format = "") [T]: ISO8601 date times.
col_factor(levels, ordered) ‘f’, a fixed set of values.
col_character() ‘c’, everything else.
col_skip() ‘_, -', don’t import this column.
col_guess() ‘?', parse using the “best” type based on the input.

You can tell vroom what columns to use with the col_types() argument in a number of ways.

If you only need to override a single column, the most concise way is to use a named vector.

# read the 'year' column as an integer
data <- vroom("flights.tsv", col_types = c(year = "i"))

# also skip reading the 'time_hour' column
data <- vroom("flights.tsv", col_types = c(year = "i", time_hour = "_"))

# also read the carrier as a factor
data <- vroom("flights.tsv", col_types = c(year = "i", time_hour = "_", carrier = "f"))

However, you can also use the col_*() functions in a list.

data <- vroom("flights.tsv",
  col_types = list(year = col_integer(), time_hour = col_skip(), carrier = col_factor())
)

This is most useful when a column type needs additional information, such as for categorical data when you know all of the levels of a factor.

data <- vroom("flights.tsv",
  col_types = list(dest = col_factor(levels = c("EWR", "JFK", "LGA")))
)

Speed

vroom is fast, but how fast? We benchmarked vroom using a real-world dataset of taxi-trip data, with 14.7 million rows, 11 columns. It contains a mix of numeric and text data, and has a total file size of 1.55 GB.

#> Observations: 14,776,615
#> Variables: 11
#> $ medallion       <chr> "89D227B655E5C82AECF13C3F540D4CF4", "0BD7C8F5B...
#> $ hack_license    <chr> "BA96DE419E711691B9445D6A6307C170", "9FD8F69F0...
#> $ vendor_id       <chr> "CMT", "CMT", "CMT", "CMT", "CMT", "CMT", "CMT...
#> $ pickup_datetime <chr> "2013-01-01 15:11:48", "2013-01-06 00:18:35", ...
#> $ payment_type    <chr> "CSH", "CSH", "CSH", "CSH", "CSH", "CSH", "CSH...
#> $ fare_amount     <dbl> 6.5, 6.0, 5.5, 5.0, 9.5, 9.5, 6.0, 34.0, 5.5, ...
#> $ surcharge       <dbl> 0.0, 0.5, 1.0, 0.5, 0.5, 0.0, 0.0, 0.0, 1.0, 0...
#> $ mta_tax         <dbl> 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0...
#> $ tip_amount      <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0...
#> $ tolls_amount    <dbl> 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 4.8, 0.0, 0...
#> $ total_amount    <dbl> 7.0, 7.0, 7.0, 6.0, 10.5, 10.0, 6.5, 39.3, 7.0...

We performed a series of simple manipulations with each approach.

Reading the data
print()
head()
tail()
Sampling 100 random rows
Filtering for “UNK” payment, this is 6434 rows (0.0435% of total).
Summarizing the mean fare amount per payment type.

package	read	print	head	tail	sample	filter	summarise	total
read.delim	1m 21.5s	6ms	1ms	1ms	1ms	315ms	764ms	1m 22.6s
readr	33.1s	90ms	1ms	1ms	2ms	202ms	825ms	34.2s
data.table	15.7s	13ms	1ms	1ms	1ms	129ms	394ms	16.3s
vroom	3.6s	86ms	1ms	1ms	2ms	1.4s	1.9s	7s

Some things to note in the results. The initial reading is much faster in vroom than any other method, and most of the manipulations, such as print(), head(), tail() and sample() are equally fast, so fast they can’t be seen in the plots. However because the character data is read lazily, operations such as filter and summarise, which need character values, require additional time. However, this cost will only occur once. After the values have been read, they will be stored in memory, and subsequent accesses will be equivalent to other packages.

For more details on how the benchmarks were performed and additional benchmarks with other types of data see the benchmark vignette.

Feedback welcome!

vroom is a new package and, like any newborn, may fall down a few times before learning to run. If you do run into a bug or think of a new feature that would work well in vroom please open an issue so we can discuss it!

Acknowledgements

Even though this is a new release, a number of people have been testing out pre-release versions on their datasets and opening issues, which has been a huge help in making the package more robust.

A big thanks to @alex-gable, @andrie, @dan-reznik, @Evgeniy-, @ginolhac, @ibarraespinosa, @KasperSkytte, @ldecicco-USGS, @LuisQ95, @matthieu-haudiquet, @md0u80c9, @mkiang, @R3myG, @randomgambit, @slowkow, @telaroz, @thierrygosselin, and @xiaodaigh!

Also this package would not be possible without the following significant contributions to the R ecosystem.

Gabe Becker, Luke Tierney and Tomas Kalibera for conceiving, implementing and maintaining the Altrep framework used extensively in vroom.
Romain François, whose Altrepisode package and related blog-posts were a great guide for creating new Altrep objects in C++.
Matt Dowle and the rest of the Rdatatable team, data.table::fread() is blazing fast and a great motivator to think about how to read delimited files fast!

dbplyr 1.4.0

Tue, 30 Apr 2019 00:00:00 +0000

We’re stoked to announce the release of dbplyr 1.4.0, the database backend for dplyr that translates R code into SQL. dbplyr allows you to use a remote database as if it was a collection of local data frames: you write ordinary dplyr code and dbplyr turns it into SQL for you.

You can install the released version from CRAN:

install.packages("dbplyr")

While there are many small improvements and bug fixes (see the full full details in the changelog), the biggest improvements in this release are related to SQL generation. In this blog post, I’ll show you how find out what dplyr is doing behind the scenes, discuss some major simplifications to the SQL that dbplyr generates, and then show off a few of the new function translations.

To get started, I’ll load dplyr and dbplyr:

library(dplyr)
library(dbplyr)

SQL simulation

This version of dbplyr substantially improves the tools that allow you to see what SQL dbplyr generates without needing to connect to a live database. You won’t generally need these tools for real data, but they’re very useful for blog posts (like this one!), for generating reprexes (as discussed in the new vignette("reprex")), and for dbplyr’s internal tests.

The basic idea is that you can create a “lazy” frame:

df1 <- lazy_frame(x = 1:3, y = 3:1)

I call this tibble lazy because dplyr operations don’t do any work; instead they just record the action so it can later be turned in to a SQL query. You can see this query by printing the lazy tibble:

df1
#> <SQL>
#> SELECT *
#> FROM `df`

This is most useful when you add on a few steps:

df1 %>% 
  group_by(x) %>% 
  summarise(z = mean(y))
#> <SQL>
#> Warning: Missing values are always removed in SQL.
#> Use `mean(x, na.rm = TRUE)` to silence this warning
#> This warning is displayed only once per session.
#> SELECT `x`, AVG(`y`) AS `z`
#> FROM `df`
#> GROUP BY `x`

(This example also highlights another small improvement to dbplyr: it’ll only warn you about missing values being automatically removed in the database once per session.)

Importantly, lazy_frame() has a con argument that allows you to specify which database should be used for the translation. This makes it easier to see the differences in SQL generation between databases:

df1 %>% 
  mutate(z = paste("item:", x)) %>% 
  head(5)
#> <SQL>
#> SELECT `x`, `y`, CONCAT_WS(' ', 'item:', `x`) AS `z`
#> FROM `df`
#> LIMIT 5

df2 <- lazy_frame(x = 1, y = 2, con = simulate_mssql())
df2 %>% 
  mutate(z = paste("item:", x)) %>% 
  head(5)
#> <SQL>
#> SELECT TOP(5) `x`, `y`, 'item:' + ' ' + `x` AS `z`
#> FROM `df`

When you don’t specify a connection, dplyr uses its standard translation, which tries to follow the ANSI SQL standard as closely as possible.

Simpler SQL

Two improvements have considerably reduced the number of subqueries that dbplyr needs:

Joins, semi joins, and set operations no longer add additional unneeded subqueries, and now generate the minimum set:

df1 <- lazy_frame(x = 1, y = 2, a = 2)
df2 <- lazy_frame(x = 1, y = 2, b = 2)
    
union(df1, df2)
#> <SQL>
#> (SELECT `x`, `y`, `a`, NULL AS `b`
#> FROM `df`)
#> UNION
#> (SELECT `x`, `y`, NULL AS `a`, `b`
#> FROM `df`)
    
left_join(df1, df2, by = c("x", "y"))
#> <SQL>
#> SELECT `LHS`.`x` AS `x`, `LHS`.`y` AS `y`, `LHS`.`a` AS `a`, `RHS`.`b` AS `b`
#> FROM `df` AS `LHS`
#> LEFT JOIN `df` AS `RHS`
#> ON (`LHS`.`x` = `RHS`.`x` AND `LHS`.`y` = `RHS`.`y`)
    
semi_join(df1, df2, by = c("x", "y"))
#> <SQL>
#> SELECT * FROM `df` AS `LHS`
#> WHERE EXISTS (
#>   SELECT 1 FROM `df` AS `RHS`
#>   WHERE (`LHS`.`x` = `RHS`.`x` AND `LHS`.`y` = `RHS`.`y`)
#> )

Many sequences of mutate(), select(), rename(), and transmute() steps are collapsed into a single query:

df <- lazy_frame(x = 1, y = 2)
df %>% 
  select(2:1) %>% 
  select(2:1) %>% 
  select(2:1) %>% 
  select(2:1) %>% 
  mutate(z = x + y) %>% 
  select(3:1)
#> <SQL>
#> SELECT `x` + `y` AS `z`, `y`, `x`
#> FROM `df`

Note that dbplyr will still generate multiple subqueries from a single mutate statement when needed. This resolves one of my biggest frustrations with SQL:

df %>% 
  mutate(
    a = x + 1, 
    b1 = a * 3, 
    b2 = a ^ 2,
    c = b1 / b2
  )
#> <SQL>
#> SELECT `x`, `y`, `a`, `b1`, `b2`, `b1` / `b2` AS `c`
#> FROM (SELECT `x`, `y`, `a`, `a` * 3.0 AS `b1`, POWER(`a`, 2.0) AS `b2`
#> FROM (SELECT `x`, `y`, `x` + 1.0 AS `a`
#> FROM `df`) `dbplyr_001`) `dbplyr_002`

I touch on this advantage of dbplyr over SQL in a new vignette, vignette("sql"), which also gives some advice about how to write literal SQL, when dbplyr’s built-in translations don’t work.

SQL translation

As well as improving the translation of high-level dplyr functions, we’ve also considerably added to the set of low-level vector functions that dbplyr can translate. Firstly, MySQL (>= 8.0), MariaDB (>= 10.2) and SQLite (>3.25) gain support for window functions. These allow you to use summary functions (like mean() or sum()) inside of mutate(), as well as unlocking useful function like min_rank(), first(), and lead()/lag().

Thanks to Cole Arendt, dbplyr now supports translations for a selection of useful functions from stringr (str_c(), str_sub(), str_length(), str_to_upper(), str_to_lower(), and str_to_title()), and lubridate ( today(), now(), year(), month() (numeric value only), day(), hour(), minute(), second()):

df <- lazy_frame(name = character(), birthday = character())

df %>% 
  transmute(
    name = str_to_lower(name), 
    month = month(birthday),
    mday = mday(birthday)
  )
#> <SQL>
#> SELECT LOWER(`name`) AS `name`, EXTRACT(month FROM `birthday`) AS `month`, EXTRACT(day FROM `birthday`) AS `mday`
#> FROM `df`

Thanks to David C Hall we have translations for bitwise operations (bitwNot(), bitwAnd(), bitwOr(), bitwXor(), bitwShiftL(), and bitwShiftR()):

df <- lazy_frame(x = integer(), y = integer())
df %>% 
  transmute(and = bitwAnd(x, y), or = bitwOr(x, y))
#> <SQL>
#> SELECT `x` & `y` AS `and`, `x` | `y` AS `or`
#> FROM `df`

Thanks to E. David Aja‘s research on tidyverse developer day we have improved translations for median() and quantile() for all ANSI compliant databases (SQL Server, Postgres, MariaDB), along with custom translations for Hive and Teradata.

lazy_frame(g = character(), x = numeric()) %>% 
  group_by(g) %>% 
  summarise(y = median(x))
#> <SQL>
#> SELECT `g`, PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY `x`) AS `y`
#> FROM `df`
#> GROUP BY `g`

Finally, I have improved translation of [, so that you can use expressions like sum(a[b == 0]) as you would in R:

tbl_lazy(mtcars) %>% 
  group_by(cyl) %>% 
  summarise(mpg_vs = sum(mpg[vs == 1], na.rm = TRUE))
#> <SQL>
#> SELECT `cyl`, SUM(CASE WHEN (`vs` = 1.0) THEN (`mpg`) END) AS `mpg_vs`
#> FROM `df`
#> GROUP BY `cyl`

You can also now use use x$y to access nested fields:

lazy_frame(x = list()) %>% 
  mutate(z = x$y$z)
#> <SQL>
#> SELECT `x`, `x`.`y`.`z` AS `z`
#> FROM `df`

Acknowledgements

As always, a big thank goes to the entire community who helped make this release of dbplyr a reality - I really appreciated all your bug reports, helpful comments, SQL suggestions, and pull requests!

@alex-gable, @blairj09, @carvalhomb, @cderv, @colearendt, @DanielStay, @davidchall, @dlindelof, @edgararuiz, @FrancoisGuillem, @FranGoitia, @hadley, @imanuelcostigan, @JakeRuss, @javierluraschi, @jcfisher, @jkylearmstrong, @JohnMount, @jrisi256, @jsekamane, @klmedeiros, @leungi, @Liubuntu, @lpatruno, @lymanmark, @mkearney, @mkirzon, @mpettis, @mtoto, @N1h1l1sT, @nwstephens, @QuLogic, @r2evans, @rlh1994, @shgoke, @tomauer, and @verajosemanuel

How parsnip works

Tue, 30 Apr 2019 00:00:00 +0000

parsnip was accepted to CRAN late last year. Our first blog post was a top-level introduction. Here, we will go into the design of some of the internals. If you end up looking at the package sources, there are going to be times that you might question our sanity. This post is intended to go over some design decisions and provide some context.

My first solution: caret

Having developed caret in early 2005, I was not prepared for what it would be like to maintain a package that wrapped more than 200 models. What caret does is to create code modules that define the different aspects of modeling, such as model fitting, prediction, and so on. These code modules are contained in lists, one per specific model. For example, here is one of the simplest lists for MASS::lda (slightly abbreviated):

list(
  label = "Linear Discriminant Analysis",
  library = "MASS",
  type = "Classification",
  fit = function(x, y, wts, param, lev, last, classProbs, ...)
    MASS::lda(x, y, ...)  ,
  predict = function(modelFit, newdata, submodels = NULL)
    predict(modelFit, newdata)$class,
  prob = function(modelFit, newdata, submodels = NULL)
    predict(modelFit, newdata)$posterior
  )

Some notes:

✔ The modules are fairly well-defined and mutually exclusive.
✔ The structure prevents information leakage. The fitting parts are isolated from the prediction parts and the prediction modules only return predictions in a standard format. caret compartmentalizes the modeling process and prevents people from “teaching to the test” unless they deliberately go out of their way to do so.
✔ These modules cleanly document issues with using the packages. We’d like each model function to have a predictable interface for fitting and prediction, but that’s not the reality for many packages. If nothing else, these modules document the idiosyncratic nature of some packages.
✔ The lda code above is a fairly clean case. Others are not as nice to look at, especially if they can predict multiple sub-models at once or have problematic interfaces.
✖ There is a lot of code duplication. For this reason, the directory containing the current set of 238 modules is about 1.3MB.
✖ The system in caret rigidly defines what parameters can be tuned. It is a bit of a walled garden in this respect.
✔ Code like MASS::lda would be exposed to R CMD check if it is contained in a package’s R directory. Because of this, there was a large number of package dependencies in early versions. caret was almost intolerable when it came time for CRAN to check the package. One way that I fixed this was to compile these code modules in an R list object and treat that as data in the package. In this way, the package R files do not contain much specific model code and, many formal dependencies are avoided.
✔ The ellipses (...) are heavily utilized here. This makes passing other arguments to the underlying fitting function trivial. This is probably still my favorite thing about the S language.
✖ In some cases, caret needs to grab an object that might be in the ellipses in order to modify the value. For example, the main tuning parameters for rpart models are in rpart.control. If a user passes in an argument by the name of control, it will need to be captured and the appropriate arguments (like cp, maxdepth, or minsplit) are modified without changing the other arguments. That’s not hard to do but it eliminates the benefits you get by using the ellipses. In the end, do.call("rpart", args) is used to fit the model. The downside of this is that the data objects are embedded in args and, as an unhappy side-effect, the data set gets embedded in rpart's call object. That’s really bad.

When I began at RStudio, I had already been thinking about a different and, hopefully more elegant, way to do this for tidymodels.

A focus on calls and quosures

When parsnip fits a model, it constructs a call object that will be evaluated to create the model fit object (rlang::call2 and rlang::call_modify are excellent). For example, if we were doing this “by-hand” for something simple like glm, an initial function using rlang could be:

library(rlang)
glm_fit <- function(formula, data, evaluate = FALSE, ...) {
  # capture the specific arguments and ellipses as quosures:
  args <- list(formula = enquo(formula), data = rlang::enquo(data))
  # capture any extra arguments
  args <- c(args, rlang::enquos(...))
  # make the call
  model_call <- rlang::call2("glm", .ns = "stats", !!!args)

  if (evaluate) {
    res <- rlang::eval_tidy(model_call)
  } else {
    res <- model_call
  }
  res
}

glm_fit()

#> stats::glm(formula = ~, data = ~)

glm_fit(family = stats::binomial)

#> stats::glm(formula = ~, data = ~, family = ~stats::binomial)

When these are printed, the tilde indicates which values are quosures. A quosure is a combination of an rlang expression and a reference to the environment in which it originated. Since it is partly an expression, the value has not yet been evaluated inside of glm_fit(). For this reason, when we pass family = stats::binomial, the object stats::binomial is not evaluated. If it were, the value of that object would be embedded into the call (type unclass(binomial()) at an R prompt to see what this looks like).

Here’s a better example:

glm_fit(mpg ~ ., data = mtcars, x = FALSE)

#> stats::glm(formula = ~(mpg ~ .), data = ~mtcars, x = ~FALSE)

The eval_tidy function can evaluate the quosure arguments when the call itself is evaluated. We get our model object as expected:

glm_fit(mpg ~ ., data = mtcars, x = FALSE, evaluate = TRUE)

#> 
#> Call:  stats::glm(formula = ~(mpg ~ .), data = ~mtcars, x = ~FALSE)
#> 
#> Coefficients:
#> (Intercept)          cyl         disp           hp         drat           wt  
#>     12.3034      -0.1114       0.0133      -0.0215       0.7871      -3.7153  
#>        qsec           vs           am         gear         carb  
#>      0.8210       0.3178       2.5202       0.6554      -0.1994  
#> 
#> Degrees of Freedom: 31 Total (i.e. Null);  21 Residual
#> Null Deviance:	    1130 
#> Residual Deviance: 147 	AIC: 164

For parsnip, there are utility functions that create the call and others to evaluate it. This means that the only model-related information needed to by the package to define the general glm fitting function would be:

glm_data <- list(func = "glm", package = "stats", required_args = c("formula", "data"))

All of the other arguments are assembled from the model specification object or extra arguments provided to set_engine() and fit(). This allows for a fairly compact representation of the information for the fitting module. Also, it does not expose the code in a way that requires a package dependency and doesn’t contaminate the model call with actual object values. The same strategy can be used to produce predictions and other quantities.

There are some other niceties too. If a package, like glmnet has a non-formula interface and requires the predictors to be in a matrix, that fitting function can just insert x = as.matrix(x) into the glmnet call instead of doing the matrix conversion prior to the model fit.

Unsurprisingly, most model are more complex than our glm example. There are some modeling packages, such as keras or xgboost, that don’t have a one-line call to fit the model (see the keras regression example here). In this case, a wrapper function is needed and this most likely results in an extra package dependency.

Also, some prediction methods give back results that require post-processing. For example, class probability predictions for a multiclass xgboost model come back as a vector. For example, if you were to predict four samples of iris data, you would get a 12 element vector back that requires you to reshape the results into the appropriate 4x3 data frame. parsnip handles these by having slots for pre-processing the data and/or post-processing the raw prediction results. More information can be found on the vignette for creating a model object. Keep in mind that, as we and other contributors work with the package more, these internals may slightly change in the first few versions.

Also, a lot of the package code revolves around getting the arguments right. There are some default arguments set by the package, such as family = binomial for logistic regression. These defaults can be overwritten but may also depend on the mode of the model (e.g. regression, classification, etc.). There are also some arguments that parsnp protects in case the user tries to modify them (e.g. data). Finally, the main arguments to parsnip model functions are standardized and need to eventually be converted back to their engine-specific names.

The down-side to using calls

Suppose that you are fitting a random forest regression model and want to tune over mtry, the number of randomly selected predictors to evaluate at each split of the tree. This is a function of the data since it depends on the number of predictors. A simplistic version of the code that iterates over mtry might look like:

library(tidymodels)
rf_model <- rand_forest(mode = "regression")

for (i in 1:4) {
  new_rf <- 
    rf_model %>% 
    update(mtry = i)
  rf_fit <- 
    new_rf %>%
    set_engine("ranger") %>%
    fit(Sepal.Width ~ ., data = iris)
  # now evaluate the model for performance
}

# what is the current value of `mtry`? 
new_rf

#> Random Forest Model Specification (regression)
#> 
#> Main Arguments:
#>   mtry = i

The specification depends on i and its value is not in new_rf, only a reference to a symbol i in the global environment. What if the value of i changes?

i <- 300
new_rf %>%
  set_engine("ranger") %>%
  fit(Sepal.Width ~ ., data = iris)

#> Error in ranger::ranger(formula = formula, data = data, mtry = ~i, num.threads = 1, : User interrupt or internal error.

# mtry should be between 1 and 4 for these data.

In this context, using quosures for parameter values is problematic because (by default) their values are not frozen when the specification is created. However, we can fix this using quasiquotiation. The value of i can be embedded into the call using !!:

for (i in 1:4) {
  new_rf <- 
    rf_model %>% 
    update(mtry = !!i)
}

# What is the current value of `mtry` now? 
new_rf

#> Random Forest Model Specification (regression)
#> 
#> Main Arguments:
#>   mtry = 4

One other downside is related to the data used in the call. In the random forest example above, the value of mtry is dependent on the number of predictors. What if you don’t know how many predictors you have until the model is fit? How could you then specify mtry?

This could happen for a few reasons. For example, if you are using a recipe that has a filter based on reducing correlation, the number of predictors may not be known until the recipe is prepped (and the number may vary inside of resampling). A more common example is related to dummy variables. If I use fit_xy() instead of fit() in the code above, the Species predictor in the iris data is expanded from one column into two dummy variable columns just prior to the model fit. This could affect the possible range of mtry values.

There are a few ways to get around this. The first (and worst) is to use the data in an expression. Since mtry isn’t evaluated until the model fit, you could try to use mtry = floor(sqrt(ncol(data))). That’s very brittle for a few different reasons. For one, you may use the parsnip interface fit(formula, data), but the underlying model may be model_fn(x, y) and the data object doesn’t exist when the model call is evaluated.

For this reason, we added data descriptors to parsnip. These are small functions that only work when the model is being fit and they capture relevant aspects of the data at the time of fit. For example, the function .obs() can be used in an argument value to reference the number of rows in the data. To illustrate, the argument min_n for a random forest model corresponds to how many data points are required to make further splits. For regression models, this defaults to 5 when using ranger. Suppose you want this to be one-tenth of the data but at least 8. If your data are being resampled, you might not know the training set size. You could use:

rf_model %>% 
    update(min_n = max(8, floor(.obs()/10))) %>%
    set_engine("ranger") %>%
    fit(Sepal.Width ~ ., data = iris)

#> parsnip model object
#> 
#> Ranger result
#> 
#> Call:
#>  ranger::ranger(formula = formula, data = data, min.node.size = ~max(8,      floor(.obs()/10)), num.threads = 1, verbose = FALSE, seed = sample.int(10^5,      1)) 
#> 
#> Type:                             Regression 
#> Number of trees:                  500 
#> Sample size:                      150 
#> Number of independent variables:  4 
#> Mtry:                             2 
#> Target node size:                 15 
#> Variable importance mode:         none 
#> Splitrule:                        variance 
#> OOB prediction error (MSE):       0.0827 
#> R squared (OOB):                  0.565

# The math checks out for the "Target node size" above:
max(8, floor(nrow(iris)/10))

#> [1] 15

# Does this work outside of the function call? 
max(8, floor(.obs()/10))

#> Descriptor context not set

# Nope!

The different descriptors should enable a wide variety of values for tuning parameters.

Why not just evaluate the arguments?

In many cases, the model specification arguments (e.g. mtry or min_n) and engine arguments are simple objects or scalar values. It makes sense to quote data, x or y but why not just evaluate the other arguments as usual?

There are a few reasons. First, there are some arguments whose evaluation should be deferred. For example, stan and ranger models have their own random seed arguments. To enable reproducibility, parsnip gives these function default values of seed = sample.int(10^5, 1). If this argument were unquoted, then the seed value would be fixed when the package was compiled. There are solutions for this simple example though.

As seen above for data descriptors, there is the need to wait for some argument values to be evaluated at the same time that the call is evaluated. Originally, parsnip immediately evaluated almost all of the arguments and our advice was to have users quote special arguments using rlang::expr(). The feedback on this aspect of parsnip was uniformity unfavorable since it would require many casual users to learn rlang and metaprogramming techniques. For this reason, we moved the metaprogramming parts within the function to accomplish the same goals but without the user being required to understand the technical minutiae. When a user uses an argument like min_n = max(8, floor(.obs()/10)), the use of quosures is hidden from view and it looks like they are using an ordinary function called .obs().

One final reason to leave arguments unevaluated in the model specification is related to future plans. As previously mentioned, caret rigidly defined which model parameters were available for performance tuning. The approach taken by parsnip is very much the opposite. We want to enable users to tune any aspect of the model that they see fit, including some of the engine specific parameters.

For example, when fitting a Bayesian regression model, a user might want to tune over how diffuse the prior distribution should be and so on. Rather than formally defining every possible tunable parameter, parsnip allows the user to have a placeholder for parameters in the model specification that declares, “I want to change this parameter, but I don’t know what the exact value should be.”

To do this, a special function called varying() is used. A model cannot be fit if it has any varying parameters but future packages will be able to detect any of these parameters and construct tuning grids accordingly. The parameter values can be changed to specific candidate values, and these are then tested to see which value is the most appropriate. The code to do this is not ready at this point and will be part of another package. However, we can demonstrate how this happens inside of parsnip:

lr_stan_spec <-
  logistic_reg() %>%
  set_engine(
    "stan",
    iter = 5000,
    prior_intercept = rstanarm::student_t(df = varying()),
    seed = 2347
  )

# Which, if any, arguments, should be tuned?
varying_args(lr_stan_spec)

#> # A tibble: 5 x 4
#>   name            varying id           type      
#>   <chr>           <lgl>   <chr>        <chr>     
#> 1 penalty         FALSE   logistic_reg model_spec
#> 2 mixture         FALSE   logistic_reg model_spec
#> 3 iter            FALSE   logistic_reg model_spec
#> 4 prior_intercept TRUE    logistic_reg model_spec
#> 5 seed            FALSE   logistic_reg model_spec

nnet_spec <-
  mlp(
    hidden_units = varying(),
    epochs = varying(),
    dropout = varying()
  ) %>%
  set_engine(
    "keras", 
    batch_size = varying(),
    callbacks = callback_early_stopping(monitor = 'loss', min_delta = varying())
  )

varying_args(nnet_spec)

#> # A tibble: 7 x 4
#>   name         varying id    type      
#>   <chr>        <lgl>   <chr> <chr>     
#> 1 hidden_units TRUE    mlp   model_spec
#> 2 penalty      FALSE   mlp   model_spec
#> 3 dropout      TRUE    mlp   model_spec
#> 4 epochs       TRUE    mlp   model_spec
#> 5 activation   FALSE   mlp   model_spec
#> 6 batch_size   TRUE    mlp   model_spec
#> 7 callbacks    TRUE    mlp   model_spec

In the keras example, the argument names in nnet_spec names match up with objects in the dials package, and it will be possible to automatically create tuning grids for all of the parameters.

By avoiding evaluation in the model specification, we are enabling some interesting upcoming features.

Also, this feature will also be used with recipes and other future packages. This will enable joint optimization of parameters associated with pre-processing, model fitting, and post-processing activities.

Tidyverse dev day at useR! 2019

Wed, 24 Apr 2019 00:00:00 +0000

Our first tidyverse developer day, held after rstudio::conf(2019), was a great success. We’re excited to announce that we’ll be holding another one – this time the day before the start of useR! 2019 in Toulouse, France (Monday, July 8th).

📆 When: 8 July, 9 am – 5 pm.
📍 Where: The same location as useR!, the congress centre “Pierre Baudis”.
👥 Who: Anyone who would like to get better at contributing to the tidyverse! Everyone is welcome regardless of whether you’ve never done a PR before, or if you’ve already made your 10th package.
💻 What: A day of learning and coding. We’ll provide lunch, snacks, and coffee; you’ll bring your laptop and enthusiasm. The tidyverse team will be on hand to help out and answer your questions.

Tickets 🎫

Tickets are on sale now!

Tickets cost €10; we’re charging a small fee because space is limited. All fees will be donated to a non-profit (TBD).

Code of conduct

All attendees will be expected to follow the Code of Conduct.

testthat 2.1.0

Tue, 23 Apr 2019 00:00:00 +0000

We’re happy to announce that testthat 2.1.0 is now available on CRAN! testthat makes it easy to turn your existing informal tests into formal automated tests that you can rerun quickly and easily. testthat is the most popular unit-testing package for R, and is used by over 4,000 CRAN and Bioconductor packages. You can learn more about unit testing at https://r-pkgs.org/tests.html.

Install the latest version of testthat with:

install.packages("testthat")

testthat 2.1.0 mostly contains a large number of minor improvements and bug fixes, as described in the release notes. This blog post discusses the bigger improvements:

context()s are now optional.
Two new expectations: expect_invisible() and expect_mapequal().

`context()` is now optional

The biggest change in this version is that context() is now optional, and in fact we recommend that you no longer use it. context() made sense with the original design of testthat, which supported a very flexible mapping between your R code and the corresponding tests.

Now, however, we have a stronger recommendation: organise the files in tests/testthat in the same way that you organise the files in R/, so that tests for code in R/my-file.R live in tests/testthat-my-file.R. (This begs the question of how you should organise your R/ directory, which we don’t have a good answer for yet, but at least you only need to struggle to organise one directory). With this convention, context() tends to end up duplicating the file name, causing needless hassle if you reorganise your code structure.

The convention that every file in R/ has a corresponding file in tests/testthat (and vice versa) is used by two other helpful functions:

usethis::use_test(), which, if you use RStudio, will automatically create and open a test file corresponding to the R file in the right location. (If you’ve written the test file first, you can instead use usethis::use_r())
devtools::test_coverage_file(): again, if you use RStudio, this will look at the active file, run just the tests for that file, and report the coverage results. This is a great way to rapidly iterate to ensure that you have tested all the branches of new code.

New expectations

library(testthat)

This version of testthat introduces two important new expectations:

expect_invisible() makes it easier to check if a function returns its results invisibly. This is useful if you are testing a function that is called primarily for its side-effects, which should (as a general rule) invisibly return its first argument.
```
greet <- function(name) {
  cat("Hello ", name, "!\n", sep = "")
  invisible(name)
}
    
x <- expect_invisible(greet("Hadley"))
#> Hello Hadley!
expect_equal(x, "Hadley")
```
For symmetry, expect_visible() is also available, but you would not generally test for it, as visible return values are the default. Only use it if you find a bug related to visibilty and want to programmatically verify that it’s fixed.

New expect_mapequal(x, y) checks that x and y have the same names, and the same value associated with each name (i.e. it compares the values of the vector standardising the order of the names).

exp <- list(a = 1, b = 2)
expect_mapequal(list(a = 1, b = 2), exp)
expect_mapequal(list(b = 2, a = 1), exp)
    
expect_mapequal(list(b = 2), exp)
#> Error: Names absent from `object`: "a",
expect_mapequal(list(a = 3, b = 2), exp)
#> Error: act$val[exp_nms] not equal to exp$val.
#> Component "a": Mean relative difference: 0.6666667
expect_mapequal(list(a = 1, b = 2, c = 3), exp)
#> Error: Names absent from `expected`: "c",

expect_mapequal() is related to expect_setequal(), which compares values, ignoring order and count:

expect_setequal(c("a", "b"), c("b", "a"))
expect_setequal(c("a", "b"), c("a", "a", "b"))

Acknowledgements

A big thanks to all 51 people who helped contribute to this release by reporting bugs, suggesting new features, or creating pull requests: @aabor, @AEBilgrau, @antaldaniel, @bflammers, @billdenney, @BillDunlap, @Bisaloo, @burchill, @chasemc, @colearendt, @comicfans, @dmenne, @euklid321, @flying-sheep, @gaborcsardi, @gabrielodom, @gvwilson, @hadley, @harvey131, @Hong-Revo, @HughParsonage, @jackhannah95, @jackwasey, @jarodmeng, @jennybc, @jsilve24, @kevinushey, @kforner, @lionel-, @maelle, @markvanderloo, @masaha03, @mb706, @mbjoseph, @merliseclyde, @mikejiang, @Mooskey, @olsgaard, @patr1ckm, @ramiromagno, @randy3k, @renkun-ken, @smbache, @stevecondylios, @topepo, @tramontini, @wch, @wsherwin, @Yuri-M-Dias, @yutannihilation, and @zappingseb.

usethis 1.5.0

Tue, 09 Apr 2019 00:00:00 +0000

We’re gratified to announce that usethis 1.5.0 is now available on CRAN. Learn more about usethis at https://usethis.r-lib.org. Detailed notes are always in the change log.

usethis is a package that facilitates interactive workflows for R project creation and development. It has broad coverage of package development tasks, but many of its functions are also applicable to non-package projects.

usethis is part of the package development toolkit whose main public face is the venerable devtools package. When devtools went through its conscious uncoupling, the functionality related to package/project setup landed in usethis.

Install the latest release of usethis like so:

install.packages("usethis")

devtools exposes all of usethis’s functions and most users should just use library(devtools) to make usethis available in the session.

library(devtools)

It is also, of course, possible to just attach usethis specifically.

library(usethis)

usethis::use_devtools() can help you put a code snippet in your .Rprofile to do this automatically at startup for interactive sessions.

Overview of recent developments

We last blogged about usethis in February 2018, at the release of v1.3.0, so there’s a lot to catch up on! We’ll cover cumulative developments in v1.4.0 and v1.5.0. The current release also marks the passing of the maintainership torch from Hadley Wickham to Jenny Bryan.

Main themes:

Improve support for the Git/GitHub aspects of project initiation and development.
Make it possible for others to extend usethis, i.e. to implement specialized workflows or organization-specific standards.
Implement what the tidyverse/r-lib team regards as best practices, often in functions that feature the word “tidy”. These are mostly for our use and are, honestly, occasionally aspirational, but others may find it interesting or useful.
Embrace the fs package internally for everything related to filepath management. We’re also gradually adding better tooling for programmatic management of specific files or for specific parts of a file, such as the badge section of README.Rmd.

Below we highlight specific changes or new features that are of special interest to users. Again, detailed notes are in the change log.

Options to set in `.Rprofile`

usethis consults certain options when it creates a new package and, if found, it favors this info over built-in defaults when populating the DESCRIPTION fields. devtools has long had this feature, but it’s a good time to open your .Rprofile (maybe via usethis::edit_r_profile()) and refresh your settings to look more like so:

options(
  usethis.full_name = "Jane Doe",
  usethis.description = list(
    `Authors@R` = 'person("Jane", "Doe", email = "jane@example.com", role = c("aut", "cre"), 
    comment = c(ORCID = "YOUR-ORCID-ID"))',
    License = "MIT + file LICENSE",
    Version = "0.0.0.9000"
  ),
  usethis.protocol  = "ssh"
)

Note the usethis.* name prefixes and that field-specific options like devtools.desc.author and devtools.desc.license have been replaced with a single named list, usethis.description. If you are firmly an “SSH person” or an “HTTPS person”, in terms of your preferred Git transport protocol, this is a great place to express that. It affects remote URLs created by usethis.

More setup tips can be found in the usethis setup article. For example, you will unlock a lot of usethis’s GitHub functionality by setting up a personal access token. usethis::browse_github_token() walks you through that process.

Git and GitHub

The git_sitrep() function gives a report on your current situation as a Git and GitHub user and specifics on the current repo’s remotes and branches. Here’s an example where Jane Doe is working in a local copy of OWNER’s package, maybe preparing to contribute a bug fix or new feature:

usethis::git_sitrep()
#> Git user
#> * Name: 'Jane Doe'
#> * Email: 'jane.doe@example.com'
#> * Vaccinated: TRUE
#> usethis + git2r
#> * Default usethis protocol: 'ssh'
#> * git2r supports SSH: TRUE
#> * Credentials: '<usethis + git2r default behaviour>'
#> GitHub
#> * Personal access token: '<found in env var>'
#> * User: 'janedoe'
#> * Name: 'Jane Doe'
#> Repo
#> * Path: '/Users/jane/tmp/REPO/.git'
#> * Local branch -> remote tracking branch: 'master' -> 'origin/master'
#> GitHub pull request readiness
#> * origin: janedoe/REPO, can push, forked from OWNER/REPO
#> * upstream: OWNER/REPO, read only

git_vaccinate() makes sure your global, i.e. user-level, gitignore file contains entries relevant to useRs. You are less likely to push confidential or unnecessary and semi-embarrassing information to GitHub if you gitignore files like .Rhistory and .Rdata.

Several other new functions are mostly for internal use but can also be used for troubleshooting and taking greater control of the Git side of usethis: git_remotes(), use_git_remote(), git_protocol(), use_git_protocol(), git_credentials(), use_git_credentials().

Two existing functions keep improving and deserve a mention:

create_from_github("OWNER/REPO) creates a local project from a GitHub repository. It’s pretty smart about when it should clone vs fork-and-clone (or you can specify this) and, when it forks, it sets up the origin and upstream remotes correctly.

usethis::create_from_github("OWNER/REPO", protocol = "ssh")
#> ✔ Creating '/Users/jane/rrr/REPO/'
#> ✔ Forking 'OWNER/REPO'
#> ✔ Cloning repo from 'git@github.com:janedoe/REPO.git' into '/Users/jane/rrr/REPO'
#> ✔ Setting active project to '/Users/jane/rrr/REPO'
#> ✔ Adding 'upstream' remote: 'git@github.com:OWNER/REPO.git'
#> ✔ Pulling changes from GitHub source repo 'upstream/master'

use_github() connects an existing local repo to GitHub post hoc. It is more robust and, when it fails, it does so early and with actionable messages about what you can fix.

A new experimental family of pr_*() functions makes GitHub pull requests (PR) easier, both for maintainers and contributors. We’re quite excited about these, as we handle an extraordinary number of pull requests across the tidyverse, r-lib, and tidymodels organizations. For maintainers, pr_fetch(<PR_NUMBER>) allows you to check out a PR locally, explore it, check/test it, or even fix or extend it. pr_push() pushes back into the actual PR, where it can now be merged or receive more work from the original author. For contributors, pr_init() and pr_push() help to initiate a PR. In all cases, pr_finish() does the necessary local clean up when the PR is no longer in play.

Extending usethis

We’ve made usethis easier for others to wrap and extend:

usethis is very chatty and this is now under the control of the option usethis.quiet. This makes it easier for a wrapper to muffle usethis’s messaging or to temporarily toggle it with functions like withr::with_options() or withr::local_options().
Project-related helpers proj_path(), with_project(), local_project(), and proj_activate() are exported.
File editing helpers edit_file(), write_over(), and write_union() are exported.
use_template() can use a template file stored in any package, not just usethis.

We’ve also exported the ui_*() functions that implement usethis’s messaging, conditions, and interactive menus. This is in response to repeated community requests. These functions benefit greatly from the glue and crayon packages to provide nifty features like string interpolation and formatting.

There are inline styles (ui_field(), ui_value(), ui_path(), ui_code()), which can be used within the block styles (ui_line(), ui_done(), ui_todo(), ui_oops(), ui_info()). Some example usage (flanking newlines added for readability):

ui_stop() and ui_warn() raise the associated condition, but with the same features for the message:

Finally, ui_yeah() and ui_nope() facilitate control flow based on user input:

These functions may ultimately move to a more appropriate home, so we can use them to create a more consistent UI across a broad set of tidyverse, r-lib, and tidymodels packages.

Acknowledgements

We give a tremendous thanks to the 108 useRs who contributed to the v1.4.0 and v1.5.0 releases, especially those who joined us at the inaugural Tidyverse Developer Day following rstudio::conf 2019:

@Aariq, @adam-gruer, @akgold, @alandipert, @alexpghayes, @andrie, @angela-li, @apreshill, @atheriel, @batpigandme, @beanumber, @behrman, @benmarwick, @bestdan, @bfgray3, @boshek, @cboettig, @cderv, @chris-billingham, @Chris-Engelhardt, @chris-prener, @coatless, @colearendt, @cwickham, @dirkschumacher, @dpprdan, @dragosmg, @duckmayr, @echasnovski, @EmilHvitfeldt, @erictleung, @etiennebr, @Fazendaaa, @friendly, @gaborcsardi, @gadenbuie, @GregorDeCillia, @GShotwell, @gvelasq, @hadley, @hafen, @HanjoStudy, @haozhu233, @heavywatal, @ijlyttle, @IndrajeetPatil, @jackwasey, @Jadamso, @jayhesselberth, @JBGruber, @jdblischak, @jennybc, @jimhester, @jjchern, @jmgirard, @jonocarroll, @jonthegeek, @jooyoungseo, @jsta, @jtr13, @karawoo, @kevinushey, @khailper, @kiwiroy, @krlmlr, @lbusett, @leonawicz, @lionel-, @llrs, @lorenzwalthert, @lwjohnst86, @maelle, @maislind, @markdly, @martinjhnhadley, @MatthieuStigler, @maurolepore, @maxheld83, @mdlincoln, @mine-cetinkaya-rundel, @MishaCivey, @mkearney, @move[bot], @muschellij2, @nbrgraphs, @nijibabulu, @njtierney, @noamross, @overmar, @pat-s, @petermeissner, @romainfrancois, @rorynolan, @ryapric, @sriharitn, @statquant, @statwonk, @strboul, @stufield, @tgerke, @thomasp85, @topepo, @trestletech, @VerenaHeld, @VincentGuyader, @vnijs, @webbedfeet, and @wlandau

R version support

Mon, 01 Apr 2019 00:00:00 +0000

Which versions of R do tidyverse packages support?

Though package-specific minimum versions of R are given in the Depends fields of individual DESCRIPTION files, our general policy is to support the current version, the devel version, and four previous versions of R.

With the prerelease of R version 3.6.0 (Planting of a Tree) already available, and its final release slated for 2019-04-26, you can expect tidyverse packages to drop support for R 3.1 in the coming weeks. This aligns R version support in the tidyverse with RMarkdown, which as of the latest release requires R 3.2.3 (due to an update to the highr package).

tibble 2.1.1

Tue, 19 Mar 2019 00:00:00 +0000

Version 2.1.1 of the tibble package is on CRAN now. Tibbles are a modern reimagining of the data frame, keeping what time has shown to be effective, and throwing out what is not, with nicer default output too! Grab the latest version with:

install.packages("tibble")

This release mostly focuses on the name repair introduced in tibble 2.0.1. We have specific regrets about one aspect of name repair and we think the pros of fixing it outweigh the cons: when a column name is completely absent, the numbered suffix we add becomes the entire name. Originally, we chose ..j (two dots and a number). However, that produces names that require special handling, because names of the form ..j have a special meaning: they are reserved words, you can assign them a value but not query them.

..2 <- 5
..2
#> Error in eval(expr, envir, enclos): ..2 used in an incorrect context, no ... to look in

The ability to query a value by name is very important for data frames in general and especially for the tidy evaluation framework:

df <- tibble(`..1` = "not ok", .name_repair = "minimal")
with(df, `..1`)
#> Error in eval(substitute(expr), data, enclos = parent.frame()): ..1 used in an incorrect context, no ... to look in
dplyr::select(df, `..1`)
#> Error in .f(.x[[i]], ...): ..1 used in an incorrect context, no ... to look in

Since name repair is often something that happens automatically, we think it’s best to suffix with ...j (three dots and a number) and leave people with names that are easier to work with.

as_tibble(list(1, 2, 3), .name_repair = "unique")
#> New names:
#> * `` -> ...1
#> * `` -> ...2
#> * `` -> ...3

#> # A tibble: 1 x 3
#>    ...1  ...2  ...3
#>   <dbl> <dbl> <dbl>
#> 1     1     2     3

Existing names of the form ..j (and also ...) are repaired too:

tibble(... = "a", ..1 = "b", .name_repair = "unique")
#> New names:
#> * `...` -> ...1
#> * `..1` -> ...2

#> # A tibble: 1 x 2
#>   ...1  ...2 
#>   <chr> <chr>
#> 1 a     b

By extension, this means that names of this form are rejected unless you specify a name repair strategy:

tibble(..1 = "a")
#> Error: Column 1 must not have names of the form ... or ..j.
#> Use .name_repair to specify repair.
tibble(..1 = "a", .name_repair = "minimal")

#> # A tibble: 1 x 1
#>   ..1  
#>   <chr>
#> 1 a

For consistency, three dots are used for all disambiguating suffixes, not only for empty names.

This might affect you if you use readxl or another package that uses the new name repair, and we’re sorry for the disruption. We’re confident that a bit of short-term pain now is better than the agony that would have come from the existent behavior. Also, name repair currently is in the “maturing” lifecycle, Read more about name repair in the tidyverse design principles.

stringr 1.4.0

Thu, 21 Feb 2019 00:00:00 +0000

We are happy to announce that stringr 1.4.0 is now on CRAN. stringr provides a cohesive set of functions designed to make working with strings as easy as possible. For a full list of changes, please see the release notes.

You can install the released version from CRAN:

install.packages("stringr")

library(stringr)

New functions

Thanks to the hard work of John Harmon at Tidyverse Developer Day, stringr has three new functions.

str_starts() and str_ends() detect the presence or absence of patterns at the beginning or end of strings.

fruit <- c("apple", "banana", "pear", "pineapple")
str_starts(fruit, "p")
#> [1] FALSE FALSE  TRUE  TRUE
str_starts(fruit, "p", negate = TRUE)
#> [1]  TRUE  TRUE FALSE FALSE
str_ends(fruit, "e")
#> [1]  TRUE FALSE FALSE  TRUE
str_ends(fruit, "e", negate = TRUE)
#> [1] FALSE  TRUE  TRUE FALSE

The new str_to_sentence() function capitalizes strings with sentence case, like so:

str_to_sentence("the quick brown dog")
#> [1] "The quick brown dog"

Support for `negate`

str_subset(), str_detect(), and str_which() now have the negate argument, which is used to find the elements that do not match a pattern (as seen above in the str_starts() and str_ends() examples).

Acknowledgements

Thank you to everyone who contributed to this release: @AmeliaMN, @batpigandme, @beckymaust, @BenjaminLouis, @blablablerg, @bschneidr, @bwiernik, @ctmann, @damianooldoni, @dan-reznik, @denrou, @diegogarcilazo, @DieselAnalytics, @elisakreiss, @giovannikraushaar, @hadley, @hammer, @jennybc, @jimhester, @jonocarroll, @jonthegeek, @jrnold, @juanrocha, @kmace, @krlmlr, @osorensen, @paleolimbot, @pdelboca, @pgrandinetti, @PirateGrunt, @samhinshaw, @sastoudt, @seanpor, @yj-danielyang, and @yutannihilation.

haven 2.1.0

Tue, 19 Feb 2019 00:00:00 +0000

We’re delighted to announce that haven 2.1.0 is now on CRAN. haven enables R to read and write various data formats used by other statistical packages by wrapping the ReadStat C library written by Evan Miller. For a full account of updates in this release, see the Changelog.

Improved labelling

Both labelled() and labelled_spss() now allow NULL labels. This makes both classes more flexible, allowing you to use them for their other attributes.labelled() also now tests that value labels are unique.

labelled objects now get pretty printing that shows the labels and NA values when inside of a tbl_df. You can turn this behaviour off by using option(haven.show_pillar_labels = FALSE).

tibble::tibble(s = haven::labelled(c(1, 10), labels = c("A" = 1, "B" = 10)))
#> # A tibble: 2 x 1
#>           s
#>   <dbl+lbl>
#> 1     1 [A]
#> 2    10 [B]

Minor improvements and fixes

This release is updated to the latest version of Evan Miller’s ReadStat, which includes the following changes:

read_por() can now read files from SPSS 25.
read_por() uses base-10 instead of base-30 for the exponent.
read_sas() can read zero-column files.
read_sav() now reads long strings, and has greater memory limit, allowing it to read more labels.
read_spss() reads long variable labels.
write_sav() no longer creates incorrect column names when >10k columns.
write_sav() no longer crashes when writing long label names.

Acknowledgements

Thank you to Evan Miller, as well as @armenic, @beckerbenj, @caayala, @gergness, @jeffeaton, @philstraforelli, @thays42, and @visseho for their contributions to this release.

bigrquery 1.1.0

Fri, 15 Feb 2019 00:00:00 +0000

We’re delighted to announce that bigrquery 1.1.0 is now on CRAN. bigrquery makes it easy to work with data stored in Google BigQuery, a hosted database for big data. This is a minor release, with some improved type support, and SQL translation. For a complete list of updates, please see the Changelog.

Improved type support

bq_table_download() and the DBI::dbConnect method now have a bigint argument, which governs how BigQuery integer columns are imported into R. As before, the default will be bigint = "integer". However, you can now set bigint = "integer64" to import BigQuery integer columns as bit64::integer64 columns in R. This allows for values outside the range of integer (-2147483647 to 2147483647). Thanks to Rasmus Bååth for implementing these changes.

bq_table_download() now treats NUMERIC columns in the same way as FLOAT columns.

bq_table_upload() now works with date-time (POSIXct) variables. Unparseable date times return NA.

SQL translation

as.character() now translates to SAFE_CAST(x AS STRING).
median() now translates to APPROX_QUANTILES(x, 2)[SAFE_ORDINAL(2)].

Minor fixes

bq_job() tracks location, so bigrquery now works painlessly with non-US/EU locations.

Acknowledgements

Thank you to everyone who contributed to this release: @414theodore, @ajh3000, @AlekseyBuzmakov, @batpigandme, @byapparov, @ChrisJohnsonUMG, @cpcgoogle, @dujm, @eddelbuettel, @edgararuiz, @everron, @guillaumed90, @hadley, @Iuiu1234, @izzetagoren, @j450h1, @jberninger, @jordanwebb10, @jrecasens, @KarimZaoui, @ldanai, @paulsendavidjay, @ras44, @rasmusab, @Saikri5hna, @selcukakbas, @SergeiMakarovWeb, @skydavis435, @tdsmith, @tinoater, @tortorellij1, @valentinumbach, and @zacdav.

dplyr 0.8.0

Fri, 15 Feb 2019 00:00:00 +0000

We’re tickled pink to announce the release of version 0.8.0 of dplyr, the grammar of data manipulation in the tidyverse. This is a major update that has kept us busy for almost a year. We take the coincidence of a Valentine’s day release as a sign of continuous ❤️ for dplyr's approach to tidy data manipulation.

Important changes are discussed in detail in the pre-release post, we are grateful to members of the community for their feedback in the last couple of months, this has been tremendously useful in making the release process smoother.

The bulk of the changes are internal, and part of an ongoing effort to make the codebase more robust and less surprising. This is an investment that will continue to pay off for years, and serve as a foundation for more innovations in the future.

For a comprehensive list of changes, please see the NEWS for the 0.8.0 release, the sections below discusses the main changes.

Group hug

Grouping has always been at the center of what dplyr is about, this release expands on the existing group_by() with a set of experimental functions with a variety of perspectives on the notion of grouping.

We believe they offer new unique possibilities, but we welcome community feedback and use cases before we put a 💍 on them. Let’s illustrate them with a subset from the well-known gapminder data.

oceania <- gapminder::gapminder %>% 
  filter(continent == "Oceania") %>% 
  mutate(yr1952 = year - 1952) %>% 
  select(-continent) %>% 
  group_by(country)
oceania
#> # A tibble: 24 x 6
#> # Groups:   country [2]
#>    country    year lifeExp      pop gdpPercap yr1952
#>    <fct>     <int>   <dbl>    <int>     <dbl>  <dbl>
#>  1 Australia  1952    69.1  8691212    10040.      0
#>  2 Australia  1957    70.3  9712569    10950.      5
#>  3 Australia  1962    70.9 10794968    12217.     10
#>  4 Australia  1967    71.1 11872264    14526.     15
#>  5 Australia  1972    71.9 13177000    16789.     20
#>  6 Australia  1977    73.5 14074100    18334.     25
#>  7 Australia  1982    74.7 15184200    19477.     30
#>  8 Australia  1987    76.3 16257249    21889.     35
#>  9 Australia  1992    77.6 17481977    23425.     40
#> 10 Australia  1997    78.8 18565243    26998.     45
#> # … with 14 more rows

group_nest() is similar to tidyr::nest(), but focuses on the variables to nest by instead of the nested columns.

oceania %>% 
  group_nest()
#> # A tibble: 2 x 2
#>   country     data             
#>   <fct>       <list>           
#> 1 Australia   <tibble [12 × 5]>
#> 2 New Zealand <tibble [12 × 5]>

group_split() is a tidy version of base::split(). In particular, it respects a group_by()-like grouping specification, and refuses to name its result.

oceania %>% 
  group_split()
#> [[1]]
#> # A tibble: 12 x 6
#>    country    year lifeExp      pop gdpPercap yr1952
#>    <fct>     <int>   <dbl>    <int>     <dbl>  <dbl>
#>  1 Australia  1952    69.1  8691212    10040.      0
#>  2 Australia  1957    70.3  9712569    10950.      5
#>  3 Australia  1962    70.9 10794968    12217.     10
#>  4 Australia  1967    71.1 11872264    14526.     15
#>  5 Australia  1972    71.9 13177000    16789.     20
#>  6 Australia  1977    73.5 14074100    18334.     25
#>  7 Australia  1982    74.7 15184200    19477.     30
#>  8 Australia  1987    76.3 16257249    21889.     35
#>  9 Australia  1992    77.6 17481977    23425.     40
#> 10 Australia  1997    78.8 18565243    26998.     45
#> 11 Australia  2002    80.4 19546792    30688.     50
#> 12 Australia  2007    81.2 20434176    34435.     55
#> 
#> [[2]]
#> # A tibble: 12 x 6
#>    country      year lifeExp     pop gdpPercap yr1952
#>    <fct>       <int>   <dbl>   <int>     <dbl>  <dbl>
#>  1 New Zealand  1952    69.4 1994794    10557.      0
#>  2 New Zealand  1957    70.3 2229407    12247.      5
#>  3 New Zealand  1962    71.2 2488550    13176.     10
#>  4 New Zealand  1967    71.5 2728150    14464.     15
#>  5 New Zealand  1972    71.9 2929100    16046.     20
#>  6 New Zealand  1977    72.2 3164900    16234.     25
#>  7 New Zealand  1982    73.8 3210650    17632.     30
#>  8 New Zealand  1987    74.3 3317166    19007.     35
#>  9 New Zealand  1992    76.3 3437674    18363.     40
#> 10 New Zealand  1997    77.6 3676187    21050.     45
#> 11 New Zealand  2002    79.1 3908037    23190.     50
#> 12 New Zealand  2007    80.2 4115771    25185.     55

group_map() and group_walk() offer a way to iterate on groups of a grouped data frame.

oceania %>% 
  mutate(yr1952 = year - 1952) %>% 
  group_map(~broom::tidy(stats::lm(lifeExp ~ yr1952, data = .x)))
#> # A tibble: 4 x 6
#> # Groups:   country [2]
#>   country     term        estimate std.error statistic  p.value
#>   <fct>       <chr>          <dbl>     <dbl>     <dbl>    <dbl>
#> 1 Australia   (Intercept)   68.4      0.337      203.  2.07e-19
#> 2 Australia   yr1952         0.228    0.0104      21.9 8.67e-10
#> 3 New Zealand (Intercept)   68.7      0.437      157.  2.66e-18
#> 4 New Zealand yr1952         0.193    0.0135      14.3 5.41e- 8

group_data(), group_rows(), and group_keys() expose the grouping information, that has been restructured in a tibble.

oceania %>% 
  group_data()
#> # A tibble: 2 x 2
#>   country     .rows     
#>   <fct>       <list>    
#> 1 Australia   <int [12]>
#> 2 New Zealand <int [12]>

oceania %>% 
  group_keys()
#> # A tibble: 2 x 1
#>   country    
#>   <fct>      
#> 1 Australia  
#> 2 New Zealand

oceania %>% 
  group_rows()
#> [[1]]
#>  [1]  1  2  3  4  5  6  7  8  9 10 11 12
#> 
#> [[2]]
#>  [1] 13 14 15 16 17 18 19 20 21 22 23 24

group_by() gains a .drop argument which you can set to FALSE to respect empty groups associated with factors (more on this below).

Give factors some love

The internal grouping algorithm has been redesigned to make it possible to better respect factor levels and empty groups. To limit the disruption, we have not made this the default behaviour. To keep empty groups, you have to set group_by()‘s .drop argument to FALSE.

This can make data manipulation more predictable and reliable, because when factors are involved, the groups are based on the levels of the factors, rather than which levels have data points.

Let’s illustrate this with our favourite flowers 💐, and a function, species_count(), that counts the number of each species after a filter(), and structures it as a tibble with one column per species.

species_count <- function(...) {
  iris %>% 
    filter(...) %>% 
    group_by(Species, .drop = FALSE) %>% 
    summarise(n = n()) %>% 
    tidyr::spread(Species, n)
}

Because we use .drop = FALSE we get one column per level of the factor, even when there’s no data associated with a level:

species_count(Petal.Length > 3)
#> # A tibble: 1 x 3
#>   setosa versicolor virginica
#>    <int>      <int>     <int>
#> 1      0         49        50
species_count(Petal.Length > 6.5)
#> # A tibble: 1 x 3
#>   setosa versicolor virginica
#>    <int>      <int>     <int>
#> 1      0          0         4
species_count(Petal.Length > 42)
#> # A tibble: 1 x 3
#>   setosa versicolor virginica
#>    <int>      <int>     <int>
#> 1      0          0         0

These 0 instead of missing columns make the experience easier when you want to combine multiple results:

limits <- seq(0, 8, by = .5)
limits %>% 
  purrr::map_dfr( ~species_count(Petal.Length > .x)) %>% 
  mutate(Sepal.Length = limits) %>% 
  select(Sepal.Length, everything())
#> # A tibble: 17 x 4
#>    Sepal.Length setosa versicolor virginica
#>           <dbl>  <int>      <int>     <int>
#>  1          0       50         50        50
#>  2          0.5     50         50        50
#>  3          1       49         50        50
#>  4          1.5     13         50        50
#>  5          2        0         50        50
#>  6          2.5      0         50        50
#>  7          3        0         49        50
#>  8          3.5      0         45        50
#>  9          4        0         34        50
#> 10          4.5      0         14        49
#> 11          5        0          1        41
#> 12          5.5      0          0        25
#> 13          6        0          0         9
#> 14          6.5      0          0         4
#> 15          7        0          0         0
#> 16          7.5      0          0         0
#> 17          8        0          0         0

Thanks

Thanks to all contributors for this release.

@abouf, @adisarid, @adrfantini, @aetiologicCanada, @afdta, @albertomv83, @alistaire47, @aloes2512, @andresimi, @antaldaniel, @AnthonyEbert, @ArtemSokolov, @AshesITR, @bakaburg1, @batpigandme, @bbachrach, @bbolker, @behrman, @BenjaminLouis, @bifouba, @billdenney, @bnicenboim, @BobMuenchen, @brooke-watson, @CarolineBarret, @cbailiss, @CerebralMastication, @cfhammill, @cfry-propeller, @choisy, @ChrisBeeley, @chrsigg, @clauswilke, @ClaytonJY, @colearendt, @ColinFay, @coolbutuseless, @Copepoda, @cpsievert, @dah33, @damianooldoni, @DanChaltiel, @danyal123, @DavisVaughan, @Demetrio92, @dewoller, @dfalbel, @DiogoFerrari, @dirkschumacher, @dmenne, @dmvianna, @dongzhuoer, @earowang, @echasnovski, @eddelbuettel, @EdwinTh, @eijoac, @elbersb, @Eli-Berkow, @EmilHvitfeldt, @epetrovski, @erblast, @etienne-s, @foundinblank, @FrancoisGuillem, @geotheory, @ggrothendieck, @GoldbergData, @gowerc, @grayskripko, @GrimTrigger88, @grizzthepro64, @hadley, @hafen, @heavywatal, @helix123, @henrikmidtiby, @hpeaker, @htc502, @hughjonesd, @ignacio82, @igoldin2u, @igordot, @ilarischeinin, @Ilia-Kosenkov, @IndrajeetPatil, @ipofanes, @jasonmhoule, @jayhesselberth, @jennybc, @jepusto, @jflynn264, @jialu512, @JiaxiangBU, @jimhester, @jkylearmstrongibx, @jnolis, @JohnMount, @jonkeane, @jonthegeek, @jschelbert, @jsekamane, @jtelleria, @kendonB, @kevinykuo, @krlmlr, @langbe, @ldecicco-USGS, @leungi, @libbieweimer, @lionel-, @liz-is, @lloven, @ltrgoddard, @luccastermans, @maicel1978, @Make42, @MalditoBarbudo, @markdly, @markvanderloo, @mattbk, @maxheld83, @melissakey, @mem48, @mgirlich, @mikmart, @MilesMcBain, @minhsphuc12, @mkoohafkan, @momeara, @moodymudskipper, @move[bot], @nealpsmith, @NightWinkle, @o1iv3r, @PascalKieslich, @petermeissner, @peterzsohar, @philstraforelli, @PMassicotte, @PPICARDO, @privefl, @prokulski, @quartin, @rabutler-usbr, @ramongallego, @randomgambit, @rappster, @rensa, @reshmamena, @richard987, @richierocks, @RickPack, @riship2009, @RobertMyles, @romainfrancois, @rontomer, @roumail, @rozsoma, @rundel, @rupesh2017, @s-fleck, @S-UP, @salmansyed0709, @schloerke, @seasmith, @sharlagelfand, @shizidushu, @simon-anasta, @skaltman, @skylarhopkins, @sowla, @statsccpr, @stenhaug, @streamline55, @stuartE9, @stufield, @suzanbaert, @sverchkov, @thackl, @the-knife, @ThiAmm, @thisisnic, @tinyheero, @tmelconian, @tobadia, @tonyelhabr, @torbjorn, @trueNico, @tungmilan, @TylerGrantSmith, @ukkonen, @vincentanutama, @vnijs, @wanfahmi, @waynelapierre, @wch, @wdenton, @wgrundlingh, @wmayner, @wolski, @yiqinfu, @yutannihilation, @Zanidean, @Zedseayou, @zslajchrt, @zx8754, and @zzygyx9119.

purrr 0.3.0

Wed, 06 Feb 2019 00:00:00 +0000

We’re excited to announce the release of purrr 0.3.0! purrr enhances R’s functional programming toolkit by providing a complete and consistent set of tools for working with functions and vectors.

# Install the latest version with:
install.packages("purrr")

# Start working with purrr:
library(purrr)

In this release, pluck() gets a few new variants:

pluck<-, assign_in() and modify_in() allow deep modifications of nested structures.
chuck() is a stricter variant of pluck() that consistently fails instead of returning NULL when the structure does not have the expected shape.

A new set of tools helps limiting the rate at which a function is called:

slowly() forces a function to sleep between each invokation.
insistently() automatically invokes a function again on error until it succeeds, and sleeps between invokations.
The rate_delay() and rate_backoff() helpers control the invokation rate of slowly() and insistently().

The reduce and map functions gain a few improvements:

map_if() accepts an optional function with the .else parameter. This function is applied on elements for which the predicate is FALSE.
map_at() now accepts vars() selections. This lets you use selection helpers like dplyr::starts_with() to determine the elements of a list which should be mapped.
reduce() now supports early termination of a computation. Just return a value wrapped in a done() to signal to reduce() that you’re done.

Besides these new functions and tools, purrr 0.3.0 is mostly a polishing release. We have improved the consistency of behaviour:

modify() is now a wrapper around [[<- instead of [<-. This makes it compatible with a larger variety of S3 vector classes.
Predicate functions (such that you would pass to map_if()) now must return a single TRUE or FALSE. Missing values and integers are no longer valid predicate outputs.

Finally, we improved the consistency of the interface:

The direction of iteration/application is now consistently specified with a .dir argument.
Many missing functions were added to fill the gaps: accumulate2(), imodify(), map_depth(), …
partial() has a much improved and more flexible interface.

Find a detailed account of the changes in the NEWS file.

New pluck variants

pluck() implements a generalised form of [[ that allow you to index deeply and flexibly into data structures. For instance, pluck(x, "foo", 2) is equivalent to x[["foo"]][[2]]. You can also supply a default value in case the element does not exist. For instance, pluck(x, "foo", 2, .default = NA) is equivalent to x[["foo"]][[2]], returning an NA if that element doesn’t exist. purrr 0.3.0 introduces variants of pluck() to make it easier to work with deep data structures.

Pluck assignment

This release introduces the new functions pluck<-, assign_in() and modify_in() as assignment variants of pluck(). To illustrate deep assignment, let’s create a nested data structure:

x <- list(foo = list(1, 2), bar = list(3, 4))
str(x)
#> List of 2
#>  $ foo:List of 2
#>   ..$ : num 1
#>   ..$ : num 2
#>  $ bar:List of 2
#>   ..$ : num 3
#>   ..$ : num 4

This sort of repeated structure is the kind of data where pluck() shines:

pluck(x, "foo", 2)
#> [1] 2

pluck(x, "bar", 1)
#> [1] 3

You can now use the same syntax to modify the data:

pluck(x, "foo", 2) <- 100
str(x)
#> List of 2
#>  $ foo:List of 2
#>   ..$ : num 1
#>   ..$ : num 100
#>  $ bar:List of 2
#>   ..$ : num 3
#>   ..$ : num 4

pluck<- also has a functional form that does not modify objects in your environment, but instead returns a modified copy:

out <- assign_in(x, list("foo", 2), 2000)

# The object is still the same as before
str(x)
#> List of 2
#>  $ foo:List of 2
#>   ..$ : num 1
#>   ..$ : num 100
#>  $ bar:List of 2
#>   ..$ : num 3
#>   ..$ : num 4

# The modified data is in `out`
str(out)
#> List of 2
#>  $ foo:List of 2
#>   ..$ : num 1
#>   ..$ : num 2000
#>  $ bar:List of 2
#>   ..$ : num 3
#>   ..$ : num 4

Finally, modify_in() is a variant of modify() that only changes the pluck location with the result of applying a function:

out <- modify_in(x, list("foo", 2), as.character)
str(out)
#> List of 2
#>  $ foo:List of 2
#>   ..$ : num 1
#>   ..$ : chr "100"
#>  $ bar:List of 2
#>   ..$ : num 3
#>   ..$ : num 4

Stricter pluck()

Thanks to Daniel Barnett (@daniel-barnett on Github), pluck() now has a stricter cousin chuck(). Whereas pluck() is very permissive regarding non-existing locations and returns NULL in these cases, and [[ inconsistently returns NULL, NA, or throws an error, chuck() fails consistently with informative messages (i.e., it “chucks” an error message):

pluck(list(1), "foo")
#> NULL

chuck(list(1), "foo")
#> Error: Index 1 is attempting to pluck from an unnamed vector using a string name

Rates

Thanks to Richie Cotton (@richierocks) and Ian Lyttle (@ijlyttle), purrr gains a function operator to make a function call itself repeatedly when an error occurs.

counter <- 0

f <- function(...) {
  if (counter < 2) {
    counter <<- counter + 1
    stop("tilt!")
  }
  "result"
}

f()
#> Error in f(): tilt!

If the function is wrapped with insistently(), it will try a few times before giving up:

# Reset counter
counter <- 0

f2 <- insistently(f)
f2()
#> [1] "result"

Another rate limiting function is slowly(). While insistently() loops by itself, slowly() is designed to be used in your own loops, for instance in a map iteration:

f <- function(...) print(Sys.time())

walk(1:3, f)
#> [1] "2019-02-06 16:22:28 CET"
#> [1] "2019-02-06 16:22:28 CET"
#> [1] "2019-02-06 16:22:28 CET"

walk(1:3, slowly(f))
#> [1] "2019-02-06 16:22:28 CET"
#> [1] "2019-02-06 16:22:29 CET"
#> [1] "2019-02-06 16:22:30 CET"

slowly() uses a constant rate by default while insistently() uses a backoff rate. The rate limiting can be configured with optional jitter via rate_backoff() and rate_delay(), which implement exponential backoff rate and constant rate respectively.

walk(1:3, slowly(f, rate_backoff(2, max_times = Inf)))
#> [1] "2019-02-06 16:22:30 CET"
#> [1] "2019-02-06 16:22:32 CET"
#> [1] "2019-02-06 16:22:34 CET"

Map and reduce improvements

`map_if()`… or else?

If you like using map_if(), perhaps you’ll find the new .else argument useful. .else is a function applied to elements for which the predicate is FALSE:

map_if(iris, is.numeric, mean, .else = nlevels)
#> $Sepal.Length
#> [1] 5.843333
#> 
#> $Sepal.Width
#> [1] 3.057333
#> 
#> $Petal.Length
#> [1] 3.758
#> 
#> $Petal.Width
#> [1] 1.199333
#> 
#> $Species
#> [1] 3

New `map_at()` features

Colin Fay (@ColinFay) has added support for tidyselect expressions to map_at() and other _at mappers. This brings the interface of these functions closer to scoped functions from the dplyr package, such as dplyr::mutate_at(). Note that vars() is currently not reexported from purrr, so you need to use dplyr::vars() or ggplot2::vars() for the time being.

suppressMessages(library("dplyr"))

x <- list(
  foo = 1:5,
  bar = 6:10,
  baz = 11:15
)

map_at(x, vars(starts_with("b")), mean)
#> $foo
#> [1] 1 2 3 4 5
#> 
#> $bar
#> [1] 8
#> 
#> $baz
#> [1] 13

map_at() now also supports negative selections:

map_at(x, -2, `*`, 1000)
#> $foo
#> [1] 1000 2000 3000 4000 5000
#> 
#> $bar
#> [1]  6  7  8  9 10
#> 
#> $baz
#> [1] 11000 12000 13000 14000 15000

Early termination of reduction

reduce() is an operation that combines the elements of a vector into a single value by calling a binary function repeatedly with the result so far and the next input of a vector. reduce() and its variant accumulate() now support early termination of the reduction. To halt the computation, just return the last value wrapped in a done() box:

# This computes the total sum of the input vector
reduce(1:100, ~ .x + .y)
#> [1] 5050

# This stops as soon as the sum is greater than 50
reduce(1:100, ~ if (.x > 50) done(.x) else .x + .y)
#> [1] 55

This feature takes inspiration from the Clojure language.

Consistency

In this polishing release, a lot of effort went towards consistency of behaviour and of the interface.

Behaviour

Better support for S3 vectors

We are working hard on improving support for S3 vectors in the tidyverse. As of this release, modify() is now a wrapper around [[<- instead of [<-. This should make it directly compatible with a larger set of vector classes. Thanks to the work of Mikko Marttila (@mikmart), pmap() and pwalk() also do a better job of preserving S3 classes. Finally, pluck() now properly calls the [[ methods of S3 objects.

In the next version of purrr, we plan to use the in-development vctrs package to provide more principled and predictable vector operations. This should help us preserve the class and properties of S3 vectors like factors, dates, or your custom classes.

Stricter predicate checking

purrr now checks the results of your predicate functions, which must now consistently return TRUE or FALSE. We no longer offer support for NA or for boolish numeric values (R normally interprets 0 as FALSE and all other values as TRUE). The purpose of this change is to detect errors earlier with a more relevant error message.

keep(c(1, NA, 3), ~ . %% 2 == 0)
#> Error: Predicate functions must return a single `TRUE` or `FALSE`, not a missing value

Interface

Direction of application

The direction of application is now specified the same way across purrr functions. reduce(), compose() and detect() now have a .dir parameter that can take the value "forward" or "backward". This terminology should be less ambiguous than “left” and “right”:

reduce(1:4, `-`, .dir = "backward")

compose(foo, bar, .dir = "forward")

detect(1:5, ~ . %% 2 == 0, .dir = "backward")

Note that the backward version of reduce() (called right-reduce in the literature) applies the reduced function in a slightly different way than reduce_right(). The new algorithm is more consistent with how this operation is usually defined in other languages.

Following the introduction of the .dir parameters, the _right variants such as reduce_right() have been soft-deprecated, as well as the .right parameter of detect() and detect_index().

partial()

partial() has been rewritten to be a simple wrapper around call_modify() and eval_tidy() from the rlang package. Consequently, the .env, .lazy and .first arguments are soft-deprecated and replaced by a flexible syntax.

To control the timing of evaluation, unquote the partialised arguments that should be evaluated only once when the function is created. The non-unquoted arguments are evaluated at each invokation of the function:

my_list <- partial(list, lazy = rnorm(3), eager = !!rnorm(3))

my_list()
#> $lazy
#> [1]  0.2945451  0.3897943 -1.2080762
#> 
#> $eager
#> [1] -0.1842525 -1.3713305 -0.5991677

my_list()
#> $lazy
#> [1] -0.3636760 -1.6266727 -0.2564784
#> 
#> $eager
#> [1] -0.1842525 -1.3713305 -0.5991677

You can also control the position of the future arguments by passing an empty ... = parameter. This syntax is powered by rlang::call_modify() and allows you to add or move dots in a quoted function call. In the case of partial(), the dots represent the future arguments. We use this syntax in the following snippet to position the future arguments right between two partialised arguments:

my_list <- partial(list, 1, ... = , 2)

my_list()
#> [[1]]
#> [1] 1
#> 
#> [[2]]
#> [1] 2

my_list("foo")
#> [[1]]
#> [1] 1
#> 
#> [[2]]
#> [1] "foo"
#> 
#> [[3]]
#> [1] 2

`exec()` replaces `invoke()`

We are retiring invoke() and invoke_map() in favour of exec(). Retirement means that we’ll keep these functions indefinitely in the package, but we won’t add features or recommend using them.

We are now favouring exec(), which uses the tidy dots syntax for passing lists of arguments:

# Before:
invoke(mean, list(na.rm = TRUE), x = 1:10)
#> [1] 5.5

# After
exec(mean, 1:10, !!!list(na.rm = TRUE))
#> [1] 5.5

Filling the missing parts

purrr 0.3.0 introduces accumulate2(), modify2() and imodify() variants.
By popular request, at_depth() is back as map_depth(). Unlike modify_depth() which preserves the class structure of the input tree, this variant only returns trees made of lists of lists (up to the given depth), coercing vectors if needed.

Thanks!

Thanks to all the contributors for this release!

＠ArtemSokolov, ＠batpigandme, ＠bbrewington, ＠billdenney, ＠cderv, ＠cfhammill, ＠ColinFay, ＠dan-reznik, ＠daniel-barnett, ＠danilinares, ＠drtjc, ＠egnha, ＠Eluvias, ＠flying-sheep, ＠gergness, ＠gvwilson, ＠hadley, ＠hammer, ＠ijlyttle, ＠ilarischeinin, ＠IndrajeetPatil, ＠ISPritchin, ＠jameslairdsmith, ＠jameslamb, ＠jrnold, ＠kcf-jackson, ＠leungi, ＠lionel-, ＠loladze, ＠maxheld83, ＠mikmart, ＠MilesMcBain, ＠moodymudskipper, ＠mrstlee, ＠namelessjon, ＠r-cheologist, ＠randomgambit, ＠rmflight, ＠roumail, ＠Ryo-N7, ＠serina-robinson, ＠skaltman, ＠suraggupta, ＠thays42, ＠tyluRp, ＠tzakharko, ＠VincentGuyader, ＠wlandau, ＠wmayner, ＠yanxianl, ＠yutannihilation, and ＠yysh12

tibble 2.0.1

Tue, 15 Jan 2019 00:00:00 +0000

I’m pleased to announce that version 2.0.1 of the tibble package is on CRAN now, just in time for rstudio::conf(). Tibbles are a modern reimagining of the data frame, keeping what time has shown to be effective, and throwing out what is not, with nicer default output too! Grab the latest version with:

install.packages("tibble")

This release required a bit of preparation, including a pre-release blog post that described the breaking changes, mostly in as_tibble(), new_tibble(), set_tidy_names(), tidy_names(), and names<-(), and a patch release that fixed problems found after the initial 2.0.0 release. In this blog post, I focus on a few user- and programmer-related changes, and give an outlook over future development:

view(), nameless enframe(), 2D columns
Lifecycle, robustness, name repair, row names, glimpse() for subclasses
vctrs, dependencies, decorations

For a complete overview please see the release notes.

Use the issue tracker to submit bugs or suggest ideas, your contributions are always welcome.

Changes that affect users

view

The experimental view() function forwards its input to utils::View() (only in interactive mode) and always returns its input invisibly, which is useful for pipe-based workflows. Currently it is unclear if this functionality should live in tibble or elsewhere.

# This is a no-op in non-interactive mode.
# In interactive mode, a viewer window/pane will open.
iris %>%
  view()

Nameless enframe

The enframe() function always has been a good way to convert a (named) vector to a two-column data frame. In this version, conversion to a one-column data frame is also supported by setting the name argument to NULL. This is now the recommended way to turn a vector to a one-column tibble, due to changes to the default implementation of as_tibble().

enframe(letters[1:3])

#> # A tibble: 3 x 2
#>    name value
#>   <int> <chr>
#> 1     1 a    
#> 2     2 b    
#> 3     3 c

enframe(letters[1:3], name = NULL)

#> # A tibble: 3 x 1
#>   value
#>   <chr>
#> 1 a    
#> 2 b    
#> 3 c

2D columns

tibble() now supports columns that are matrices or data frames. These have always been supported in data frames and are used in some modelling functions. We are looking forward to supporting these and other exciting use cases, see also the Matrix and data frame columns chapter of adv-r. The number of rows in these objects must be consistent with the length of the other columns. Internally, this feature required using NROW() instead of length() in a few spots, which conveniently returns the length for vectors and the number of rows for 2D objects. The required support in pillar has been added earlier last year.

tibble(
  a = 1:3,
  b = tibble(c = 4:6),
  d = tibble(e = 7:9, f = tibble(g = 10, h = 11)),
  i = diag(3)
)

#> # A tibble: 3 x 4
#>       a   b$c   d$e  $f$g   $$h i[,1]  [,2]  [,3]
#>   <int> <int> <int> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1     1     4     7    10    11     1     0     0
#> 2     2     5     8    10    11     0     1     0
#> 3     3     6     9    10    11     0     0     1

Changes that affect package developers

Lifecycle

All functions have been assigned a lifecycle. The tibble package has now reached the “stable” lifecycle, functions in a different lifecycle stage are marked as such in their documentation. One example is the add_row() function: it is unclear if it should ensure that all columns have length one by wrapping in a list if necessary, and a better implementation is perhaps possible once tibble uses the vctrs package, see below. Therefore this function is marked “questioning”. Learn more about lifecycle in the tidyverse at https://www.tidyverse.org/lifecycle/.

Robustness

The new .rows argument to tibble() and as_tibble() allows specifying the expected number of rows explicitly, even if it’s evident from the data. This supports writing even more defensive code. The nrow argument to the low-level new_tibble() constructor is now mandatory, on the other hand most expensive checks have been moved to the new validate_tibble() function. This means that constructions of tibbles is now faster by default if you know that the inputs are correct, but you can always double-check if needed. See also the S3 classes chapter in adv-r for motivation.

tibble(a = 1, b = 1:3, .rows = 3)

#> # A tibble: 3 x 2
#>       a     b
#>   <dbl> <int>
#> 1     1     1
#> 2     1     2
#> 3     1     3

tibble(a = 1, b = 2:3, .rows = 3)
#> Error: Tibble columns must have consistent lengths, only values of length one are recycled:
#> * Length 3: Requested with `.rows` argument
#> * Length 2: Column `b`
tibble(a = 1, .rows = 3)

#> # A tibble: 3 x 1
#>       a
#>   <dbl>
#> 1     1
#> 2     1
#> 3     1

as_tibble(iris[1:3, ], .rows = 3)

#> # A tibble: 3 x 5
#>   Sepal.Length Sepal.Width Petal.Length Petal.Width Species
#>          <dbl>       <dbl>        <dbl>       <dbl> <fct>  
#> 1          5.1         3.5          1.4         0.2 setosa 
#> 2          4.9         3            1.4         0.2 setosa 
#> 3          4.7         3.2          1.3         0.2 setosa

new_tibble(list(a = 1:3), nrow = 3)

#> # A tibble: 3 x 1
#>       a
#>   <int>
#> 1     1
#> 2     2
#> 3     3

bad <- new_tibble(list(a = 1:2), nrow = 3)
validate_tibble(bad)
#> Error: Tibble columns must have consistent lengths, only values of length one are recycled:
#> * Length 3: Requested with `nrow` argument
#> * Length 2: Column `a`

Name repair

Column name repair has more direct support, via the new .name_repair argument to tibble() and as_tibble(). It takes the following values:

"minimal": No name repair or checks, beyond basic existence.
"unique": Make sure names are unique and not empty.
"check_unique": (default value), no name repair, but check they are unique.
"universal": Make the names unique and syntactic.
a function: apply custom name repair (e.g., .name_repair = make.names or .name_repair = ~make.names(., unique = TRUE) for names in the style of base R).

## by default, duplicate names are not allowed
tibble(`1a` = 1, `1a` = 2)
#> Error: Column name `1a` must not be duplicated.
#> Use .name_repair to specify repair.

## you can authorize duplicate names
tibble(`1a` = 1, `1a` = 2, .name_repair = "minimal")

#> # A tibble: 1 x 2
#>    `1a`  `1a`
#>   <dbl> <dbl>
#> 1     1     2

## or request that the names be made unique
tibble(`1a` = 1, `1a` = 2, .name_repair = "unique")
#> New names:
#> * `1a` -> `1a..1`
#> * `1a` -> `1a..2`

#> # A tibble: 1 x 2
#>   `1a..1` `1a..2`
#>     <dbl>   <dbl>
#> 1       1       2

## or universal
tibble(`1a` = 1, `1a` = 2, .name_repair = "universal")
#> New names:
#> * `1a` -> ..1a..1
#> * `1a` -> ..1a..2

#> # A tibble: 1 x 2
#>   ..1a..1 ..1a..2
#>     <dbl>   <dbl>
#> 1       1       2

Row names

Row name handling is stricter. Row names were never supported in tibble() and new_tibble(), and are now stripped by default in as_tibble(). The rownames argument to as_tibble() supports:

NULL: remove row names (default),
NA: keep row names,
A string: the name of the new column that will contain the existing row names, which are no longer present in the result.

The old default can be restored by calling pkgconfig::set_config("tibble::rownames", NA), this also works for packages that import tibble.

rownames(as_tibble(mtcars))
#>  [1] "1"  "2"  "3"  "4"  "5"  "6"  "7"  "8"  "9"  "10" "11" "12" "13" "14"
#> [15] "15" "16" "17" "18" "19" "20" "21" "22" "23" "24" "25" "26" "27" "28"
#> [29] "29" "30" "31" "32"
as_tibble(mtcars, rownames = "make_model")

#> # A tibble: 32 x 12
#>    make_model   mpg   cyl  disp    hp  drat    wt  qsec    vs    am  gear
#>    <chr>      <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
#>  1 Mazda RX4   21       6  160    110  3.9   2.62  16.5     0     1     4
#>  2 Mazda RX4…  21       6  160    110  3.9   2.88  17.0     0     1     4
#>  3 Datsun 710  22.8     4  108     93  3.85  2.32  18.6     1     1     4
#>  4 Hornet 4 …  21.4     6  258    110  3.08  3.22  19.4     1     0     3
#>  5 Hornet Sp…  18.7     8  360    175  3.15  3.44  17.0     0     0     3
#>  6 Valiant     18.1     6  225    105  2.76  3.46  20.2     1     0     3
#>  7 Duster 360  14.3     8  360    245  3.21  3.57  15.8     0     0     3
#>  8 Merc 240D   24.4     4  147.    62  3.69  3.19  20       1     0     4
#>  9 Merc 230    22.8     4  141.    95  3.92  3.15  22.9     1     0     4
#> 10 Merc 280    19.2     6  168.   123  3.92  3.44  18.3     1     0     4
#> # … with 22 more rows, and 1 more variable: carb <dbl>

glimpse for subclasses

The glimpse() function shows information obtained from tbl_sum() in the header, e.g. grouping information for grouped_df from dplyr, or other information from packages that override the tbl_df class.

iris %>%
  group_by(Species) %>%
  glimpse()

#> Observations: 150
#> Variables: 5
#> Groups: Species [3]
#> $ Sepal.Length <dbl> 5.1, 4.9, 4.7, 4.6, 5.0, 5.4, 4.6, 5.0, 4.4, 4.9, 5…
#> $ Sepal.Width  <dbl> 3.5, 3.0, 3.2, 3.1, 3.6, 3.9, 3.4, 3.4, 2.9, 3.1, 3…
#> $ Petal.Length <dbl> 1.4, 1.4, 1.3, 1.5, 1.4, 1.7, 1.4, 1.5, 1.4, 1.5, 1…
#> $ Petal.Width  <dbl> 0.2, 0.2, 0.2, 0.2, 0.2, 0.4, 0.3, 0.2, 0.2, 0.1, 0…
#> $ Species      <fct> setosa, setosa, setosa, setosa, setosa, setosa, set…

Outlook

vctrs

The plan is to use vctrs in tibble 2.1.0. This package is a solid foundation for handling coercion, concatenation and recycling in vectors of arbitrary type. The support provided by vctrs will yield a better add_row() implementation, in return name repair which is currently defined in tibble should likely live in vctrs.

Dependencies

Currently, installing tibble can bring in almost dozen other packages:

tools::package_dependencies("tibble", recursive = TRUE, which = "Imports")
#> $tibble
#>  [1] "cli"        "crayon"     "fansi"      "methods"    "pillar"    
#>  [6] "pkgconfig"  "rlang"      "utils"      "assertthat" "grDevices" 
#> [11] "utf8"       "tools"

Some of them, namely fansi and utf8, contain code that requires compilation and are only required for optional features. The plan is to make these packages, and crayon, a suggested package to cli, and provide fallback implementations there. When finished, taking a strong dependency on tibble won’t add too many new dependencies (again): rlang, vctrs and cli will be used by most of the tidyverse anyway, pillar is the only truly new strong dependency. Packages that subclass tbl_df should import tibble to make sure that the subsetting operator [ always behaves the same. Constructing (subclasses of) tibbles should happen through new_tibble() only.

Decorations

Tibbles have a very opinionated way to print their data, not always in line with users’ expectations, and sometimes clearly wrong (e.g. for numerical data where the absolute mean is much larger than the standard deviation). It seems difficult to devise a formatting that suits all needs, especially for numbers: how do we tell if a number represents money, or perhaps is a misspecified categorical variable or a UID? Decorations are an idea that might help here. A decoration is applied only when printing a vector, which behaves identically to a bare vector otherwise. Decorations can be “learned” from the data (using heuristics), or specified directly after import or when creating column, and stored in attribues like "class". It will be important to make sure that these attributes survive subsetting and perhaps some arithmetic transformations, easiest to achieve with the help of vctrs.

Acknowledgments

Thanks to Brodie Gaslam (@brodieG) for his help with formatting this blog post and for spotting inaccurate wording.

We also received issues, pull requests, and comments from 108 people since tibble 1.4.2. Thanks to everyone:

@adam-gruer, @aegerton, @alaindanet, @alexpghayes, @alexwhan, @alistaire47, @anhqle, @batpigandme, @brendanf, @brodieG, @cfhammill, @christophsax, @cimentadaj, @czeildi, @DasHammett, @DavisVaughan, @earowang, @Eluvias, @Enchufa2, @esford3, @flying-sheep, @gavinsimpson, @GeorgeHayduke, @gregorp, @hadley, @IndrajeetPatil, @iron0012, @isteves, @jeffreyhanson, @jennybc, @jimhester, @JLYJabc, @joranE, @jtelleriar, @karldw, @kendonB, @kevinushey, @kovla, @lbusett, @lionel-, @lorenzwalthert, @lwiklendt, @mattfidler, @MatthieuStigler, @maxheld83, @michaelweylandt, @mingsu, @momeara, @PalaceChan, @pat-s, @plantarum, @prosoitos, @ptoche, @QuLogic, @ralonso-igenomix, @randomgambit, @riccardopinosio, @romainfrancois, @tomroh, @Woosah, @yonicd, and @yutannihilation.

rlang 0.3.1

Thu, 10 Jan 2019 00:00:00 +0000

The patch release 0.3.1 of rlang is now on CRAN! This release polishes the rlang backtraces introduced in 0.3.0. See the NEWS for the complete set of changes. The main feature is rlang::entrace(), that you can set as a global error handler to get rlang backtraces for all errors.

Numbered backtraces

Since rlang 0.3.0, errors thrown with abort() embed a backtrace that can be consulted by calling last_error(). We have improved the appearance of the backtraces by numbering its components. Let’s trigger an error within a complicated call stack:

f <- function() tryCatch(g(), warning = identity) # Try g()
g <- function() evalq(h())                        # Eval h()
h <- function() abort("Oh no!")                   # And fail!

f()

#> Error: Oh no!
#> Call `rlang::last_error()` to see a backtrace

The simplified backtrace shown on error is now numbered:

last_error()

#> <error>
#> message: Oh no!
#> class:   `rlang_error`
#> backtrace:
#>  1. global::f()
#>  6. global::g()
#>  9. global::h()
#> Call `rlang::last_trace()` to see the full backtrace

Notice how the numbering is not sequential? That’s because last_error() displays a simplified backtrace by default, that only includes the calls that are most likely to help you figure out a problem. Call the new function last_trace() to get the full picture:

last_trace()

#>     █
#>  1. └─global::f()
#>  2.   ├─base::tryCatch(g(), warning = identity)
#>  3.   │ └─base:::tryCatchList(expr, classes, parentenv, handlers)
#>  4.   │   └─base:::tryCatchOne(expr, names, parentenv, handlers[[1L]])
#>  5.   │     └─base:::doTryCatch(return(expr), name, parentenv, handler)
#>  6.   └─global::g()
#>  7.     ├─base::evalq(h())
#>  8.     │ └─base::evalq(h())
#>  9.     └─global::h()

Note that this full backtrace is the equivalent of base::traceback(), only structured as a tree, presented in reverse order, and with namespace prefixes. Here is the output of traceback() for comparison:

#> 9: h()
#> 8: evalq(h())
#> 7: evalq(h()) at #1
#> 6: g()
#> 5: doTryCatch(return(expr), name, parentenv, handler)
#> 4: tryCatchOne(expr, names, parentenv, handlers[[1L]])
#> 3: tryCatchList(expr, classes, parentenv, handlers)
#> 2: tryCatch(g(), warning = identity) at #1
#> 1: f()

Backtraces for base errors!

These backtraces are normally only recorded for errors thrown with abort(). This leaves out errors thrown with stop(), errors thrown from native C code, and warnings converted to errors. Starting from this release, insert this snippet in your RProfile to enable backtraces for all errors:

if (requireNamespace("rlang", quietly = TRUE)) {
  options(error = rlang::entrace)
}

With this in place, any unhandled error will record an rlang backtrace automatically:

h <- function() stop("stop!")
f()

#> Error: stop!
#> Call `rlang::last_error()` to see a backtrace

The backtrace can be consulted in the ordinary way:

last_error()

#> <error>
#> message: stop!
#> class:   `rlang_error`
#> backtrace:
#>  1. global::f()
#>  6. global::g()
#>  9. global::h()
#> Call `rlang::last_trace()` to see the full backtrace

last_trace()

#>     █
#>  1. └─global::f()
#>  2.   ├─base::tryCatch(g(), warning = identity)
#>  3.   │ └─base:::tryCatchList(expr, classes, parentenv, handlers)
#>  4.   │   └─base:::tryCatchOne(expr, names, parentenv, handlers[[1L]])
#>  5.   │     └─base:::doTryCatch(return(expr), name, parentenv, handler)
#>  6.   └─global::g()
#>  7.     ├─base::evalq(h())
#>  8.     │ └─base::evalq(h())
#>  9.     └─global::h()

Both the entracing of base errors and the simplification of backtraces are experimental, but they should already work well enough to be useful in your day-to-day work. We’d love to hear about your feedback!

dbplyr 1.3.0

Wed, 09 Jan 2019 00:00:00 +0000

We’re stoked to announce that dbplyr 1.3.0 is now available on CRAN. dbplyr is the database backend for dplyr, translating dplyr syntax into SQL. This is a minor release primarily for compatibility with dplyr 0.8.0. However, there have been some changes to the API, and minor improvements (the full details of which can be found in the changelog). A more substantive update can be expected after rstudio::conf.

API changes

Calls of the form dplyr::foo() are now evaluated in the database, rather than locally.
The vars argument to tbl_sql() has been formally deprecated.
src and tbl objects now include a class generated from the class of the underlying connection object. This makes it possible for dplyr backends to implement different behaviour at the dplyr level, when needed.

SQL translation

The new as.integer64(x) is translated to CAST(x AS BIGINT).
cummean() now translates to SQL AVG() as opposed to MEAN().
x %in% y is now translated to FALSE if y is empty.

Acknowledgements

Thank you to the 49 people who contributed to this release: @AkhilGNair, @andypohl, @batpigandme, @bogdanrau, @cboettig, @cderv, @chris-park, @colearendt, @copernican, @cseidman, @DavisVaughan, @dgrtwo, @dlindelof, @dpprdan, @edgararuiz, @foonwong, @gbisschoff, @hadley, @halldc, @happyshows, @iangow, @javierluraschi, @jimhester, @jinaliu, @JohnMount, @jonthegeek, @jrjohnson0821, @karldw, @kevinykuo, @kmace, @krlmlr, @lincis, @lionel-, @mfalcioni1, @mgirlich, @mkirzon, @mkuehn10, @mtmorgan, @N1h1l1sT, @Prometheus77, @ramnathv, @romainfrancois, @sandan, @sverchkov, @tdsmith, @tmastny, @TomWeishaar, @vitallish, and @yutannihilation.

vdiffr 0.3.0

Thu, 03 Jan 2019 00:00:00 +0000

We’re thrilled to announce that vdiffr 0.3.0 is now on CRAN! vdiffr is a testthat extension that makes it easy to add visual unit tests for R plots. Testing visualisations is hard because it is difficult to integrate in an automated workflow. vdiffr achieves this by generating SVG renditions of your plot, storing them in the tests/fig folder in your package, and comparing the SVGs when running the tests. If you’re developing a package for statistical graphics, you might be interested in adding vdiffr to your checking workflow.

This is the first publicly announced version of vdiffr because previous versions had issues with the comparison of SVGs across platforms, such as Travis or the CRAN machines. These problems should now be fixed. If you’re already a vdiffr user, please note that you will need to regenerate all your figures with the new SVG generation engine.

Create visual unit tests

expect_doppelganger() is the main function you’ll be using to create visual unit tests. It accepts three types of input:

ggplot2 objects, which are specially integrated in vdiffr (see below).
More generally, any object whose print() method draws the object on the graphics device.
Functions. Those will be called by vdiffr and should print the plot as side effect.

Here is an example of how to use it:

context("Distributions")

test_that("histograms draw correctly", {
  hist_ggplot <- ggplot(mtcars, aes(disp)) + geom_histogram()
  vdiffr::expect_doppelganger("ggplot2 histogram", hist_ggplot)

  hist_base <- function() hist(mtcars$disp)
  vdiffr::expect_doppelganger("Base graphics histogram", hist_base)
})

The first argument of expect_doppelganger() is the plot title. It should be a description of what is being tested exactly by the plot. The title is also standardised (all special characters are converted to -) and used as a filename to store the visual case. In addition, the current testthat context (here, “Distributions”) is used as directory for all figures generated in that context. The example above creates two visual tests whose whose saved SVG files live (once validated) in tests/figs/distributions/ggplot2-histogram.svg and tests/figs/distributions/base-graphics-histogram.svg.

vdiffr treats ggplot2 figures specially. First, it adds the figure title as ggtitle() (if the plot does not already have one). This makes the test figure self-explanatory when you open the SVG file. Secondly, if the plot does not have a ggplot2 theme, a minimalistic testing theme is automatically applied to it. This theme will not be tweaked in future ggplot2 versions to ensure some stability to the visual unit tests.

Manage visual tests with a Shiny app

New visual expectations must be validated first. To validate new figures, call vdiffr::manage_cases() or use the RStudio Addin installed by vdiffr. This opens a Shiny app that you will use to validate the figures of new visual expectations.

Once the figures are validated, run devtools::test() in the usual way to run the visual tests. If the figure generated during the test doesn’t match its recorded version, testthat fails (but not on CRAN, see next section).

When testthat reports that a visual comparison fails, open the Shiny app again. The app features several diffing widgets to help you compare the figures and detect why they don’t match. You can also revalidate the figure if the failure is the result of fixing a bug, adding a feature, or some other benign upstream change that caused a change in the appearance of your figure.

vdiffr in RStudio

Testing versus Monitoring

When a figure doesn’t match its saved version, it is only reported as a failure under these circumstances:

When the NOT_CRAN environment variable is set. Note that devtools sets this automatically when the tests are run interactively.
On Travis, Appveyor, or any environment where the Sys.getenv("CI") is set.

Otherwise, the failure is ignored. The motivation for this is that vdiffr is a monitoring tool and shouldn’t cause R CMD check failures on the CRAN machines.

This behaviour is motivated by the inherent fragility of visual comparisons. The exact way plots are rendered depends on a lot of upstream logic, such as the way margins are computed. vdiffr uses a special ggplot2 theme that should change very rarely, but there are just too many upstream factors that could cause breakages. In the end, visual testing is not an alternative to writing unit tests for the internal data transformations performed during the creation of your figure. It is more of a monitoring tool that allows you to quickly check how the appearance of your figures changes over time, and to manually assess whether changes reflect actual problems in your packages.

If you need to override the default vdiffr behaviour on CRAN (not recommended) or Travis (for example to run the tests in a particular builds but not others), set the VDIFFR_RUN_TESTS environment variable to “true” or “false”.

haven 2.0.0

Wed, 02 Jan 2019 00:00:00 +0000

We’re pleased to announce that haven 2.0.0 is now on CRAN. haven enables R to read and write various data formats used by other statistical packages by wrapping the ReadStat C library written by Evan Miller. As always, you can find a detailed account of all the changes in this release in the Changelog.

Breaking changes

The biggest change is that labelled() and labelled_spss() now produce objects with class “haven_labelled” and “haven_labelled_spss”. This resolves a problematic clash with the labelled class defined by Frank Harrell’s Hmisc package.

Given Hmisc has been around much longer than haven (Hmisc’s oldest source archive on CRAN is from 2003), it makes sense that haven should be the one to make the change. This will require some changes to packages that use haven, but shouldn’t affect user code.

x1 <- haven::labelled(c(1, 1, 2), c(Grunt = 1, Sgt = 2),
                      label = "Army status")

class(x1)
#> [1] "haven_labelled"

x2 <- haven::labelled_spss(1:10, c(Good = 1, Bad = 8), na_range = c(9, Inf),
                           label = "Quality rating")

class(x2)
#> [1] "haven_labelled_spss" "haven_labelled"

Minor improvements

labelled() and labelled_spss() now support adding the label attribute to the resulting object (as seen in the code above). The label is a short, human-readable description of the object, and is now also used when printing, and can be easily removed using the new zap_label() function.

x1
#> <Labelled double>: Army status
#> [1] 1 1 2
#> 
#> Labels:
#>  value label
#>      1 Grunt
#>      2   Sgt

zap_labels(x1)
#> [1] 1 1 2
#> attr(,"label")
#> [1] "Army status"

Acknowledgements

A special thanks to Karl Ove Hufthammer for implementing zap_label(), as well as to the 30 others who contributed to this release: @ajdamico, @alexeyknorre, @batpigandme, @beckerbenj, @caayala, @Crismoc, @dar143, @davidc92, @elbersb, @gergness, @hadley, @helge-baumann, @HughParsonage, @iadli100, @JanMarvin, @jeffeaton, @jimhester, @jmetzel, @larmarange, @lemzwerg, @Majid-Eismann, @nealrichardson, @oscardssmith, @philstraforelli, @reyntjesr, @rubenarslan, @sjkiss, @visseho, @zkamvar, and @zross.

Posts on Tidyverse

hardhat 0.1.0

Setup

Preprocessing

Predicting

Learning more

odbc 1.2.0

Querying schemas

Immediate/direct execution

Timezone display

Acknowledgements

tidyverse 1.3.0

Citing the tidyverse

When to cite the tidyverse

How to cite the tidyverse

Acknowledgments

scales 1.1.0

Acknowledgements

dtplyr 1.0.0

Usage

Performance

Translation cost

Copies

Interface mismatch

Translation

Acknowledgements

roxygen2 7.0.0

Improved documentation

Changes to .Rd output

Markdown improvements

@includeRmd

R6 documentation

Code loading

Extending roxygen2

Acknowledgements

haven 2.2.0

Acknowledgements

testthat 2.3.0

Errors

verify_output()

Acknoweldgements

Tidyverse dev day 2020

Tickets 🎫

Code of conduct

styler 1.2.0

Alignment detection

rlang’s {{

Addins

Braces in function calls

Other changes

Adaption of styler

Outlook

Acknowledgments

discrim 0.0.1

dials 0.0.3

devtools 2.2.1

Introduction

New features

create()

dev_sitrep()

ellipsis

Acknowledgements

tidyr 1.0.0

Pivoting

Rectangling

Nesting

expand_grid()

Thanks

Multi Process Task Queue in 100 Lines of R Code

Introduction

API design

Data structure

Implementation

Try it out

How about process cleanup?

Possible improvements

Complete code

Tidy dev day take two: Toulouse

pkgdown 1.4.0

Development and deployment modes

Changes to `.Rd` output

`@includeRmd`

`verify_output()`

`create()`

`dev_sitrep()`

`expand_grid()`

A simpler interpolation pattern with `{{`