A program aimed at reducing recidivism among young people jailed on Rikers Island is being shut down because of disappointing results. It’s being hailed as a success anyway because it showed that government, philanthropy, research organizations and direct-service agencies can collaborate in policy experiments.

But while looking narrowly at one policy makes for valid research, it may not give public officials the elbow in the ribs they need to make common-sense changes to how they approach criminal justice.

The Adolescent Behavioral Learning Experience or ABLE program, launched in 2012, was the country’s first “social impact bond” project. Social impact bonds use money from private funders to pay for public policy interventions. If the policy produces results, the government pays the funder back with a profit. If it falls short, the investor eats the cost. It’s a way of funding policy experiments without risking government money: Taxpayers only pay if the idea works.

In the case of ABLE, the investor was Goldman Sachs, which put up $7.2 million, $6 million of it covered by Bloomberg Philanthropies. MDRC oversaw implementation of the program, the Osborne Association actually implemented it and the Vera Institute measured the results. The program involved the use of moral reconation therapy (MRT), a form of cognitive behavioral therapy that, according to Vera, “focuses on improving social skills, personal responsibility, and decision making.”

About 87 percent of the 1,691 16- to 18-year-olds who passed through Rikers during 2013 had at least one MRT session and 44 percent of those received an amount of therapy that’s been associated with positive results elsewhere.

But the recidivism rate, which the Osborne Association says is about 50 percent for that age group, didn’t budge. So it was announced last week that the program is being discontinued.

In some ways, the project did exactly what it was supposed to do: It permitted an experiment at no cost to the public. And carrying out such an effort on Rikers, where security lockdowns are common and stays of uncertain and varying duration are the norm, can’t have been easy—especially during a period when, as we now know, conditions on the island were deteriorating.

The question is whether the experiment framed the policy issue properly. Is the main driver of youth recidivism a lack of social skills, a deficiency of personal responsibility or an epidemic of poor decision-making? Those are certainly factors in many youth arrests. But is someone picked up in a trespassing sweep in a friend’s building displaying a lack of responsibility? Is a youth who’s homeless really able to make decisions that keep him out of the criminal justice system? Those are precisely the doubts that underlay the whole critique of the mass-arrest strategy employed under Mayor Bloomberg and Commissioner Kelly, which was in place for much of the ABLE trial.

The ABLE program did compare the performance of its test group to historical data on 16- to 18-year olds in the system as well as to the experience of 19-year-old jail contemporaries to try to control for the influence of different policing policies. So the results are valid—moral reconation therapy didn’t make a difference.

The bigger issue is whether, given the goal of reducing recidivism, MRT was a better option than, say, reducing the number of questionable misdemeanor arrests or making access to bail easier (90 percent of the adolescents on Rikers are pretrial detainees). Those moves wouldn’t have permitted a policy test like the one ABLE offered and they would have required a deeper strategic shift by city government, but they may have made a bigger difference in the young lives in play.

“I think the test was promising in that everybody adhered to the terms and conditions. The bank did lose money. Nevertheless, it wants to try more of these. It looks promising for the future of trying these programs,” says David Butler, a senior policy adviser at MDRC. “That still leaves the call of what you do to try to improve the outcomes for this population. That remains an open question.”

To be sure, many individual young people doubtless benefited from ABLE. There are more social impact bond projects underway across the country. And there has been real progress in New York of late in reducing arrests, exploring ways to improve mental-health services in jail and reforming bail—experiments of a different sort that are long overdue. As Osborne noted in its statement reacting to the end of the social impact bond project: “ABLE will conclude on August 31, but the kids on Rikers and those who will be detained there in the future need and deserve opportunities and tools to create a positive and productive future.”

Follow City Limits’ other investigations. Get our free, weekly newsletter.