How to interpret dplyr message summarise regrouping output by x override with groups argument

Navigating the world of data manipulation in R often leads us through the powerful dplyr package, a core component of the Tidyverse. Its functions like group_by() and summarise() are indispensable for aggregating data. However, if you’ve been working with these tools, you’ve likely encountered the message: summarise() regrouping output by ‘x’ (override with .groups argument). This isn’t just a friendly heads-up; it’s a crucial signal about how your data is being processed and, more importantly, how it will behave in subsequent operations. Understanding this message is key to ensuring the integrity and predictability of your data analysis workflows, preventing subtle bugs, and mastering efficient R programming practices. Let’s delve into what this message means and how the .groups argument empowers you to take full control.

Understanding the summarise() Regrouping Message

The summarise() function in dplyr is designed to reduce multiple rows to a single summary row for each group. When you use it after group_by() with multiple grouping variables, dplyr has a default behavior: it automatically “drops” the last grouping variable from the resulting tibble. This means that if you group your data by, say, continent and then country, and then apply summarise(), the output will still be grouped by continent, but not by country. The message summarise() regrouping output by ‘x’ (override with .groups argument) specifically tells you which grouping variable ('x' being a placeholder for the actual variable name) has been implicitly dropped, and reminds you that the output is still grouped by the remaining variables.

This default behavior, while often convenient, can lead to unexpected results if you chain further dplyr operations without being aware of the remaining grouping structure. Imagine you calculate the average population for each country, and then you want to calculate the average population per continent. If country was dropped but continent remained, a subsequent summarise() call without explicit ungrouping would perform calculations at the continent level, which might not be your intention if you expected an ungrouped tibble. According to the official dplyr documentation for summarise(), this behavior was introduced in dplyr 1.0.0 to make the grouping structure more transparent and controllable.

For instance, if you group by year and month, then summarise, the output will be grouped by year. The message would then say: summarise() regrouping output by ‘year’ (override with .groups argument). This implicit regrouping helps maintain a logical flow for many common aggregation tasks, allowing you to progressively aggregate data from finer to coarser granularities without manually calling ungroup() every time. However, explicit control with .groups is often preferred for robust and predictable code.

The .groups Argument: Taking Control of Your Data

The .groups argument offers explicit control over the grouping structure of the tibble returned by summarise(). This is incredibly powerful for maintaining data integrity and ensuring your analytical pipeline behaves exactly as intended. It accepts several values, each dictating a specific outcome:

• "drop_last": This is the default behavior. It drops the last grouping variable, and the output remains grouped by all but the last variable. This is what generates the warning message you see.
• "drop": This option completely ungroups the output. The resulting tibble will have no grouping attributes, behaving like a standard data frame for subsequent operations. This is often the safest choice if you want to ensure no hidden grouping affects later steps.
• "keep": This retains all original grouping variables. The output tibble will have the same grouping structure as the input. This can be useful if you’re performing multiple summarizations on the same grouping levels.
• "rowwise": This rare option treats each row as its own group. While less common for summarise(), it can be powerful in other contexts where row-wise operations are needed.

For example, if you want your summarized data to be completely ungrouped after calculation, explicitly setting .groups = "drop" is the cleanest way. This prevents any unintended grouping from influencing subsequent operations, making your code more predictable and easier to debug. Conversely, if you want to perform further summarizations on the same grouping levels, using .groups = "keep" ensures the grouping structure is preserved. This fine-grained control allows data analysts to build more resilient and transparent data pipelines, a cornerstone of effective data science. The introduction of the .groups argument was a significant enhancement in dplyr version 1.0.0, addressing a common source of confusion and error for users.

To effectively interpret the summarise() regrouping message and manage your data’s grouping state, use the .groups argument. Specifically, passing .groups = "drop" to your summarise() call will completely remove all grouping attributes from the resulting tibble, ensuring that subsequent operations treat the output as an ungrouped dataset, thus preventing unexpected aggregations based on prior grouping variables.

Practical Applications and Best Practices

Knowing how to use .groups effectively transforms your dplyr code from something that works to something that is robust and predictable. A common scenario involves calculating summary statistics at multiple levels of granularity. For instance, you might first want to calculate daily totals, then monthly averages. Without careful management of grouping, you might find your monthly average inadvertently applying a daily grouping Question & Answer :

I started getting a new message (see post title) when running group_by and summarise() after updating to dplyr development version 0.8.99.9003.

Here is an example to recreate the output:

library(tidyverse) library(hablar) df <- read_csv("year, week, rat_house_females, rat_house_males, mouse_wild_females, mouse_wild_males 2018,10,1,1,1,1 2018,10,1,1,1,1 2018,11,2,2,2,2 2018,11,2,2,2,2 2019,10,3,3,3,3 2019,10,3,3,3,3 2019,11,4,4,4,4 2019,11,4,4,4,4") %>% convert(chr(year,week)) %>% mutate(total_rodents = rowSums(select_if(., is.numeric))) %>% convert(num(year,week)) %>% group_by(year,week) %>% summarise(average = mean(total_rodents)) 

The output tibble is correct, but this message appears:

summarise() regrouping output by ‘year’ (override with .groups argument)

How should this be interpreted? Why does it report regrouping only by ‘year’ when I grouped by both year and week? Also, what does it mean to override and why would I want to do that?

I don’t think the message indicates a problem because it appears throughout the dplyr vignette: https://cran.r-project.org/web/packages/dplyr/vignettes/programming.html

I believe it is a new message because it has only appeared on very recent SO questions such as How to melt pairwise.wilcox.test output using dplyr? and R Aggregate over multiple columns (neither of which addresses the regrouping/override message).

Thank you!

It is just a friendly warning message about the resulting grouping structure; your output is correct. By default, if there is any grouping before the summarise, it drops one group variable i.e. the last one specified in the group_by. If there is only one grouping variable, there won’t be any grouping attribute after the summarise. If there are more than one, the grouping is reduced by 1. So in your example since the input to summarise had two variables, the attribute for grouping is reduced to one, i.e. the resulting data frame would have ‘year’ as the grouping attribute.

As a reproducible example:

library(dplyr) mtcars %>% group_by(am) %>% summarise(mpg = sum(mpg)) #`summarise()` ungrouping output (override with `.groups` argument) # A tibble: 2 x 2 # am mpg #* <dbl> <dbl> #1 0 326. #2 1 317. 

The message is that it is ungrouping i.e when there is a single group_by, it drops that grouping after the summarise

mtcars %>% group_by(am, vs) %>% summarise(mpg = sum(mpg)) #`summarise()` regrouping output by 'am' (override with `.groups` argument) # A tibble: 4 x 3 # Groups: am [2] # am vs mpg # <dbl> <dbl> <dbl> #1 0 0 181. #2 0 1 145. #3 1 0 118. #4 1 1 199. 

Here, it drops the last grouping and regroup with the ‘am’

If we check the ?summarise, there is .groups argument which by default is "drop_last" and the other options are "drop", "keep", "rowwise"

.groups - Grouping structure of the result.

“drop_last”: dropping the last level of grouping. This was the only supported option before version 1.0.0.

“drop”: All levels of grouping are dropped.

“keep”: Same grouping structure as .data.

“rowwise”: Each row is its own group.

When .groups is not specified, you either get “drop_last” when all the results are size 1, or “keep” if the size varies. In addition, a message informs you of that choice, unless the option “dplyr.summarise.inform” is set to FALSE.

i.e. if we change the .groups in summarise, we don’t get the message because the group attributes are removed

mtcars %>% group_by(am) %>% summarise(mpg = sum(mpg), .groups = 'drop') # A tibble: 2 x 2 # am mpg #* <dbl> <dbl> #1 0 326. #2 1 317. mtcars %>% group_by(am, vs) %>% summarise(mpg = sum(mpg), .groups = 'drop') # A tibble: 4 x 3 # am vs mpg #* <dbl> <dbl> <dbl> #1 0 0 181. #2 0 1 145. #3 1 0 118. #4 1 1 199. mtcars %>% group_by(am, vs) %>% summarise(mpg = sum(mpg), .groups = 'drop') %>% str #tibble [4 × 3] (S3: tbl_df/tbl/data.frame) # $ am : num [1:4] 0 0 1 1 # $ vs : num [1:4] 0 1 0 1 # $ mpg: num [1:4] 181 145 118 199 

Previously, this warning was not issued and it could lead to situations where the OP does a mutate or something else assuming there is no grouping and results in unexpected output. Now, the warning gives the user an indication that we should be careful that there is a grouping attribute

NOTE: The .groups right now is experimental in its lifecycle. So, the behaviour could be modified in the future releases

Depending upon whether we need any further transformation of the data based on the same grouping variable (or not needed), we could select the different options in .groups.