Research at City Connects

 

How do we know City Connects makes a difference?

We can see the impact in a child’s smile. And we know that parents, principals, and our community partners are very satisfied with City Connects.

But because we are part of a major research university, Boston College, we also take a hard look at the data; and external researchers continuously review our work.

The gold standard for this kind of evaluation is a randomized controlled trial or RCT. In these trials, one group of randomly chosen individuals receives an intervention that might be, for example, a cancer drug or an enrichment program. A second group is randomly assigned to a “control” group that does not receive the intervention.

The two groups are compared to see how well the intervention works.

It’s a powerful tool.

But it isn’t feasible for every type of school intervention.

When City Connects is invited into a school or a district, we help every child. That’s a core part of our model. And that means we don’t randomly assign children to a control group that receives no intervention.

“Dentists don’t want to randomly assign their patients to not flossing,” Eric Dearing explained recently. Dearing is a professor in the department of Counseling, Developmental, and Educational Psychology in Boston College’s Lynch School of Education. He’s also a faculty adviser to the City Connects Evaluation Team.

In addition, Dearing jokes, there are no RCTs for jumping out of a plane with a parachute.

What works instead?

Conducting a series of well-designed, quasi-experimental research studies on City Connects programs. Any one of these studies alone is not as powerful as an RCT, but used together they can create a multifaceted view of City Connects.

“Together they tell a fairly powerful story,” Dearing says of the studies.

When children in City Connects schools do well, the use of multiple studies enables researchers to understand City Connects’ impact.

We tell this story in detail in our 2016 progress report, “The Impact of City Connects: Student Outcomes.”

Among the overlapping questions that City Connects’ research asks:

• Is the evidence robust across sites? Are Boston’s results comparable to Springfield’s?
• Is the evidence robust across samples? Or as the progress report says, do the “effects of City Connects seen previously for Boston Public Schools students endure when new cohorts of Boston students are included in analyses”?
• Is the evidence robust across different research methods?

The short answer to these questions is, yes.

Taken together, the studies make a strong case that City Connects has a causal impact on students’ success.

As the progress report notes: “The robustness of findings on student outcomes across methods, samples, and sites strengthens the evidence that addressing out-of-school factors promotes students’ achievement and life chances. Careful attention to the unique skills, talents, and needs of each student makes a difference.”

And as Dearing says: “We keep our eye on the prize of whether this intervention is generating positive effects for children.”