DoITT

More than one-third of the data sets planned for release on the city's portal in the last 17 months were not released at all or had their deadline pushed back.

It’s good Bill de Blasio the public advocate didn’t start a letter-grading system on how agencies comply with the Open Data Law because right now his Office of the Mayor would probably get a D.

Since the city’s landmark Open Data Law was passed in 2012, city agencies have released more than 1,400 data sets through the NYC OpenData portal. It’s impressive, but the vast majority of those sets (more than 1,100) were public by the time Bloomberg left office.

As de Blasio wraps up his second year in office, City Limits took a look under the hood of the city’s open data initiative, a program — and philosophy — the mayor championed as public advocate, during his campaign and to this day. What we found is little progress beyond what was built under Bloomberg.

A pioneering law with no teeth

To be fair, the city is largely considered a pioneer among municipalities when it comes to Open Data. The law itself is more than most cities have and the administrative support for it miraculously survived a change in leadership. Indeed, the new administration hasn’t ignored the issue. In July, de Blasio re-branded the city’s program as “Open Data for All,” and called it the “start of a new era.” Last month he even signed five new bills to strengthen the law. The biggest success, perhaps, and possibly a beacon of what’s to come is that 130 data sets are automated so data feeds from agencies to the portal automatically.

Despite all that, getting agencies to release new data sets, and to update that data as they are supposed to, is still a problem. Part of the reason for that is the law has no teeth. When the bill was passed, advocates fought hard for some sort of compliance mechanism so there would be a way to put pressure on delinquent agencies to release data. The advocates lost and that provision never made it into the bill. Without some sort of enforcement, the law is only as strong as the commitment of the administration.

In 2014, six months after de Blasio took office, the city’s Department of Information Technology and Telecommunications (DoITT) put out its Open Data Plan, which contained an update on how city agencies were complying with the 2012 law. This plan include planned release dates that, in large part, were chosen by the agencies themselves, as dates they felt were realistic to release certain data sets. Each scheduled data set also includes an “Update Frequency” so the public knows how often the data will be updated: daily, weekly, monthly, annually, etc.

City Limits checked the status of every report that was scheduled to be released since that Open Data Plan was issued. There were 177 datasets. Of those:

—58 were released into the city’s Open Data portal and have been updated as planned
—68 are missing entirely or had their release date pushed back
—51 were published but have not been updated as promised, include partial data or were published on agency websites but not in the portal

In fairness, a spokesman for DoITT pointed out that an additional nine datasets scheduled for release in 2016 and beyond were published early. But still, that means more than one-third of the data sets planned for release in the last 17 months were not released at all or had their deadline pushed back.

Moving deadlines, and missing them

De Blasio’s own office is not leading by example. The data sets the Office of the Mayor is responsible for uploading and updating are emblematic of problems across all agencies: dead links, a lack of updates and data contained in PDFs in “not machine readable format” on web sites but not within DoITT’s portal.

The July 2014 plan included 27 datasets that the Office of the Mayor was supposed to release. Of those, only one was released and updated as scheduled. Four of them — data on complying with Local Law 63 of 2011 (Procurement contracts), data on compliance with Local Law 1 of 2013 (M/WBE), information on upcoming construction projects and a list of planned contracts related to human services — all lead to dead links reading “This Page is Not Available.” A spokesperson for the Mayor’s Office said they are working to get that data into the portal but it is right now not in “machine readable format.”

Still odder is that City Hall’s reporting has masked how behind schedule it is. In the most recent report, released in July 2015, many of the datasets that the Office of the Mayor had not been updating were listed as “New” data sets and given new deadlines for late 2015 and 2016. According to a spokesperson at the mayor’s office, those existing data sets should not have been listed as “New” and called it a “clerical error.”

Even after the Mayor’s Office was given a new, later deadline, many of the deadlines were not hit. For example, the 2013 and 2014 data plans promised seven data sets on the city’s M/WBE program. One of those data sets is posted in the portal, but leads to a broken link. The other six were included in a single .zip file titled “Fiscal 2013 Appendices.” That data was scheduled to be updated annually, but it hasn’t been. Instead, in July, the city shifted its categories into a single, “New” data set. For that, the Office of the Mayor gave itself a later October 15, 2015 deadline. The data has still not been released into the portal. There is a quarterly report on the M/WBE program on the Small Business Services’ site, but the PDF contains no data. So this means most recent data on the M/WBE program in the city’s data portal is from when Bloomberg was mayor.

Other data sets that the Office of the Mayor did release haven’t been updated a single time since that first release. It’s currently five months behind monthly updates for “resiliency data,” four quarters behind for data on the city’s greenhouse gas emission reduction projects, and 16 different data sets didn’t get annual updates this year.

Say you wanted to know how the city doing in phasing out boilers using the highly polluting No. 4 and No. 6 heating oil? The Office of the Mayor was supposed to release Clean Heat program data on every boiler on August 1, 2014 and every month after. The data was released late, on October 21, 2014 and has not been updated once in the past 13 months.

Mayor’s Office spokeswoman Natalie Grybauskas wrote in an email that “The Mayor’s Office is actively working to post all data in the plan as well as to make new data available. We are also taking the time and care to format that data in ways that make it more usable and accessible. In the spirit of Open Data for All, our goal of making more of our City data available and accessible is an evolving process that places significant value on user feedback.”

*Seeing problems with how the law was being implemented, a group of Council members this year introduced a slew of bills to make it stronger, including legislation to create data dictionaries for every data set, standardize address and geospatial information to help with mapping, require an agency to post data in the portal if its already on its website and require DoITT to respond to requests made through the portal.

De Blasio signed five of those bills last month, but has strongly opposed a bill introduced by Councilman Jimmy Vacca that would have had the Department of Investigation audit agencies to ensure compliance with the law. That remains in negotiation at the Council.

In catch-up mode

John Kaehny, executive director of good government group Reinvent Albany, says the city needs to shake up its staffing, and let the public sue if agencies aren’t releasing data.

Kaehny — who said the fact that the city has an Open Data law and has set up 130 automated feeds is a reason to be optimistic about the program — says part of the reason the de Blasio administration continues to struggle is the city has no dedicated person, or team, responsible for overseeing the open data program. He says the Mayor’s Office of Data Analytics (MODA) is busy handling data analysis for the mayor’s office, while DoITT has no power to enforce other agencies to fork over data. The fact that de Blasio didn’t appoint a commissioner of MODA until October 2014 slowed things up, he adds.

“As of today, the de Blasio administration is still in catch-up mode. That’s how we perceive it,” Kaehny said in a recent interview. “Under the current staffing structure they are going to keep falling further behind.”

When it comes to enforcement, he says agencies would be more responsive to requirements if the public were allowed a private right of action, namely, the ability to sue if an agency isn’t releasing data it’s supposed to. The Bloomberg administration strongly opposed including anything like that in the original bill and new administration continues to oppose it.

Dominic Mauro, an attorney for Reinvent Albany, notes that a similar right to sue is key to the Freedom of Information Law (FOIL) in place in New York and other states.

“FOIL simply wouldn’t work without the private right of action,” Mauro said, mentioning the recent case in Chicago where multiple FOIL requests for video showing a Chicago Police Officer shooting Laquan McDonald were denied until a journalist sued and the video was released.

Despite the lack of recourse for citizens fed up with waiting for data, there are signs of progress. When de Blasio re-branded the initiative “Open Data for All” in July, MODA stepped up its meetings with “open data coordinators” — the go-to person for open data at every city agency. DoITT also added a “Data Lens” feature to its portal to make the data more accessible to anyone not tech savvy. And data related to policies near and dear to his heart, like universal pre-K and Vision Zero, are easily accessible.

The NYPD — which we feel obliged to point is years behind releasing annual data on “reasonable suspicion stops” into the OpenData portal — has started releasing CompStat crime statistics in Excel format on its website. And since one of the bills De Blasio signed into law requires any agency that posts data on its website also post it in the data portal, next year the NYPD should start releasing crime statistics into the city’s OpenData portal.

* Clarification: The initial version of this story reported: “The de Blasio administration was strongly opposed to recent legislation to firm up the Open Data law.” As the story detailed, the mayor was not opposed to (and ultimately signed) five pieces of legislation related to the law. He did, however, strongly oppose the bill that would have had the Department of Investigation audit agencies to ensure compliance with the law.