Updated: Stumbling Towards Our Statistical Future

--Updated Post--June 15, 2011

I wanted to repost my comments from June 2010 at a National Academies' conference on how to increase innovation in federal statistics as the final report from the National Academies of that event is now available. T

hey picked up on some of my comments on user innovation, as did some of the agencies in the room. I am pleased to report that some of the concepts I raised at the event have been implemented in different agencies — most concretely, the idea of putting notices of opportunities to comment closer to where users are getting their data (Business R&D and Innovation Survey from NSF and Census is latest). 

These are some small but good changes. I still worry that the concreteness of that recommendation came through but some of the spirit of it might not have. I'm going to be mulling over some of these concepts for an upcoming presentation I am contributing to, coincidentally at the National Academies, so will be thinking more on the subject, again. 

As for the final report, I think there are some good things in it, but I must admit I was very disappointed in the event. 

It was a discussion of innovation led almost entirely by the people who are already heavyweights in the system, which made range of the discussion very narrow. I didn't see much of an outsider perspective. The statistical system's real concern with innovation should be that they have so few interested parties in the room. 

At that event, there were a few of us not from the system as speakers but that was about it. The room was packed with agency representatives. Every major industry association and business group should have been concerned with that meeting, if it was actually relevant, because it would mean that the system was trying to find ways to be relevant. 

As I found the meeting, it was a discussion of how we can be better at things we probably should have been better at 10 years ago; not, how we can be an innovative statistical system that measures the pulse of the U.S. and global economies and sets the tone for what innovation means in statistical systems around the world. 

--Original Post--June 28, 2010

Tuesday, I will be a discussant at the National Academies’ Committee on National Statistics workshop in Washington, DC, looking at how to facilitate innovation in federal statistics. It’s a topic I am passionate about and unfortunately it’s an area of interest where we often feel like soloists rather than members of a choir. 

As we raise our calls for improvement, and the need to better measure business activity, we need more charitable foundations, academics, businesses, and local economic development officials at the table. If it weren’t for the tireless work of groups like the Association for Public Data Users (APDU), I fear that the meager levels of innovation which currently exist in the system would be next to nonexistent. 

Desire for Internal Innovation. In our experience dealing with entrepreneurship and innovation measurement, I’ve seen a lot of personnel within specific agencies that want to see more innovation in their products. 

Sadly, too often these personnel have no forums or opportunities to explore possible improvements and few incentives that would encourage innovation. More than any funds we’ve spent, our efforts to spur improvements in federal data on entrepreneurship and innovation have come mostly through convening of symposiums and workshops that allow agencies to present their plans for improvement, debate with academic and agency experts, and see what other exciting developments are occurring in academic and agency environments. The returns to this dialog seem only to increase over time and seem well-placed for topics-based statistical discussions.   

Recommendation: Create more opportunities for agency representatives to present and receive feedback on their plans for the future before plans are finalized.

Users Can Drive Innovation. When you provide basic information, like the statistical agencies often do, one of the saddest parts of the process is that much of the information goes on to uses which you never become aware. As a major producer of data, at Kauffman, I know how excited I get to see some of our data used in an analysis, or something new, but too often the public doesn’t circle back with this material. We’ve seen in our own experience, that if you create events, online platforms, and other vehicles to show your support of users, that it can result in a positive feedback loop and improved data quality. Through the Kauffman Firm Survey, we’ve undertaken an annual open call for suggested improvements in the data we collect which has over the years led to major improvements in the usability of the data for research and analysis. Had we not undertaken this process of trying to interact with the users of our data, we would have missed significant improvements in data collected on the finances of new firms, which are going to greatly improve our analysis of the current recession. 

Sadly, most agencies know few of the users of their data, and even where they know them they are often hesitant to interact and hear their needs, feedback, etc.

Recommendation: Create opportunities for the statistical agencies to create communities of high-end users for specific products in an open, online environment.

Interaction, Interaction, Interaction. I am not sure how many ways I can say it but I’ll say it once more – interaction. The thing which I find most disappointing with the current statistical system is that individuals within agencies often want more interaction with each other and other experts on their topical collections but for the large part this does not happen. Money to travel to conferences is very limited and the interagency groups and agencies overseeing some of this process only have opportunities for interaction at very high levels, not on more granular areas of interest and need. At Kauffman, we try to bring academics into the conversation but know that from trying to do this that it’s very tough. Key constituencies like state representatives, and sub-state representatives, do not have formal bodies where the agencies can receive feedback on possible data publications and so they do not hear what these groups actually want.

Recommendation: The Office of Management and Budget (OMB) should create a council of state data representatives to provide a sounding board for new state-level data development, to proactively develop a list of priorities in state-level data development, and to allocate additional funds agencies for priority projects.

Push Ourselves First. As someone who takes time to read most notices about data collection on the Federal Register, I applaud the few efforts that actually submit meaningful requests for feedback and are open to feedback submitted. In reality, these processes are largely a formality that is hidden away so that as few people as possible find them. Even submitted comments, following outlined procedures, can be dismissed as, “it is too late in the process.” Let’s stop this self-perpetuating process of mediocrity. If we are going to seek comment, seek comment. 

Recommendation: Look for feedback and post-prospective notices of data to be collected alongside the actual web pages where data is disseminated and start to create opt-in user groups.

I’ve offered a lot of criticism here but I recognize that we at Kauffman still have a lot to learn in this regard. One of the things are beta testing is this concept of a Data Wiki for the Kauffman Firm Survey. It’s not something which was easy for us to do and opens us up to user feedback, positive and negative, in a much greater way than we’ve done before. But this is also an area where we believe what we learn can help to inform the statistical agencies. 

In my opinion, until OMB and the agencies become more genuine in their efforts to draw data users into the conversation we have little room to complain about lack of funding. These are things within their power to change. We cannot afford to idly hope for improvements in national statistics. I hope others out there agree with me and will add your voice to the discussion.

comments powered by Disqus
e.j. reedy data maven

E. J. Reedy

As a director in Research and Policy, E.J. Reedy oversees the Ewing Marion Kauffman Foundation’s research initiatives related to education, human capital development, and data.

Since joining the Kauffman Foundation in 2003, Reedy has been significantly involved in the coordination of the Foundation’s entrepreneurship and innovation data-related initiatives, including the Kauffman Firm Survey, for which he served as a principal investigator, and the Foundation’s multi-year series of symposiums on data, as well as many web-related projects and initiatives. He is a globally recognized expert in entrepreneurship and innovation measurement and has consulted for a variety of agencies.