During the summer, many students lose skills gained during the school year. An obvious solution is encouraging kids to practice what they’ve learned, particularly when it comes to the key skill of reading. But some studies have found that simply getting kids to pick up a book may not actually help that much. Besides, as many parents can attest, forcing children to read when they aren’t interested can be a miserable experience for everyone involved.

Education researchers Thomas G. White and James S. Kim conducted two experiments, working with 58 teachers and 886 students completing grades 3, 4, and 5. To begin with, they used surveys to help match students with books they might enjoy. They also administered tests of reading comprehension, vocabulary, and oral reading fluency.

Next, they got the teachers to spend several lessons in June preparing for the summer. The lessons offered reading comprehension strategies, oral reading practice, and encouragement to read aloud to parents over the summer. The researchers also asked parents to listen and provide feedback as the kids read, or talked about what they had read.

Then, once a week for the eight weeks of summer vacation, White and Kim sent each student a book, matched to the child’s interests and reading ability. They also sent a letter to parents asking them to encourage the children to read and a postcard with questions for the students to answer and send back.

To generate useful data, the researchers created control groups of kids who didn’t receive the books. (They gave those kids books in September, after the end of the experiment, instead). In the second experiment, they also divided the children into subgroups, with some only receiving books, but not extra support from teachers and parents.

In the end, the students who received both the books and support didn’t do any better in oral reading than the control group, but their reading ability was significantly better. There was a particularly strong effect for low-income students, who tend to see a particularly large decline over the summer months. The difference in this group was equivalent to four extra months of learning—enough to completely offset summer loss found in other studies.

Mailing the books seemed to encourage kids to read. One parent attested that their daughter “does not read as often as I’d like her to. Your program has changed that. She enjoys receiving the books in the mail.”

And yet, one of the most interesting findings of the study was that simply getting kids to read wasn’t enough to support their skills. The group that only received books actually reported reading at a higher rate than those also getting extra support. But, when it came to the tests of reading ability, they didn’t do much better than the control group.

The takeaway for parents and teachers seems to be that kids need some help and support if we want them to get academic benefits from summer reading.