6/15/2011 6:30:45 AM By
E.J. Reedy
--Updated Post--June 15, 2011
I wanted to repost my comments from June 2010 at a National Academies' conference on how to increase innovation in federal statistics as the final report from the National Academies of that event is now available. They picked up on some of my comments on user innovation, as did some of the agencies in the room. I am pleased to report that some of the concepts I raised at the event have been implemented in different agencies - most concretely the idea of putting notices of opportunities to comment closer to where users are getting their data (Business R&D and Innovation Survey from NSF and Census is latest). These are some small but good changes. I still worry that the concreteness of that recommendation came through but some of the spirit of it might not have. I'm going to be mulling over some of these concepts for an upcoming presentation I am contributing to, coincidentally at the National Academies, so will be thinking more on the subject again.
As for the final report, I think there are some good things in it, but I must admit I was very disappointed in the event. It was a discussion of innovation led almost entirely by the people who are already heavyweights in the system and that made range of the discussion very narrow. I didn't see much of an outsider perspective. The statistical system's real concern with innovation should be that they have so few interested parties in the room. At that event there were a few of us not from the system as speakers but that was about it. The room was packed with agency representatives. Every major industry association and business group should have been concerned with that meeting if it was actually relevant because it'd mean that the system was trying to find ways to be relevant. As I found the meeting, it was a discussion of how we can be better at things we probably should have been better at 10 years ago; not how we can be an innovative statistical system that measures the pulse of the U.S. and global economies and sets the tone for what innovation means in statistical systems around the world.
--Original Post--June 28, 2010
Tuesday I will be a discussant at the National Academies’ Committee on National Statistics workshop in Washington, DC, looking at how to facilitate innovation in federal statistics. It’s a topic I am passionate about and unfortunately it’s an area of interest where we often feel like soloists rather than members of a choir. As we raise our calls for improvement and the need to better measure business activity we need more charitable foundations, academics, businesses, and local economic development officials at the table. If it weren’t for the tireless work of groups like the Association for Public Data Users (APDU), I fear that the meager levels of innovation which currently exist in the system would be next to nonexistent.
Desire for Internal Innovation. In our experience dealing with entrepreneurship and innovation measurement, I’ve seen a lot of personnel within specific agencies that want to see more innovation in their products. Sadly, too often these personnel have no forums or opportunities to explore possible improvements and few incentives that would encourage innovation. More than any funds we’ve spent, our efforts to spur improvements in federal data on entrepreneurship and innovation have come mostly through convening of symposiums and workshops that allow agencies to present their plans for improvement, debate with academic and agency experts, and see what other exciting developments are occurring in academic and agency environments. The returns to this dialog seem only to increase over time and seem well-placed for topics-based statistical discussions.
Recommendation: Create more opportunities for agency representatives to present and receive feedback on their plans for the future before plans are finalized.
Users Can Drive Innovation. When you provide basic information, like the statistical agencies often do, one of the saddest parts of the process is that much of the information goes on to uses which you never become aware. As a major producer of data, at Kauffman, I know how excited I get to see some of our data used in an analysis or something new but too often the public doesn’t circle back with this material. We’ve seen in our own experience that if you create events, online platforms, and other vehicles to show your support of users that it can result in a positive feedback loop and improved data quality. Through the Kauffman Firm Survey, we’ve undertaken an annual open call for suggested improvements in the data we collect which has over the years led to major improvements in the usability of the data for research and analysis. Had we not undertaken this process of trying to interact with the users of our data we would have missed significant improvements in data collected on the finances of new firms which are going to greatly improve our analysis of the current recession.
Sadly, most agencies know few of the users of their data, and even where they know them they are often hesitant to interact and hear their needs, feedback, etc.
Recommendation: Create opportunities for the statistical agencies to create communities of high-end users for specific products in an open, online environment.
Interaction, Interaction, Interaction. I am not sure how many ways I can say it but I’ll say it one more – interaction. The thing which I find most disappointing with the current statistical system is that individuals within agencies often want more interaction with each other and other experts on their topical collections but for the large part this does not happen. Money to travel to conferences is very limited and the interagency groups and agencies overseeing some of this process only have opportunities for interaction at very high levels, not on more granular areas of interest and need. At Kauffman we try to bring academics into the conversation but know that from trying to do this that it’s very tough. Key constituencies like state representatives and sub-state representatives don’t have formal bodies where the agencies can receive feedback on possible data publications and so they don’t hear what these groups actually want.
Recommendation: The Office of Management and Budget (OMB) should create a council of state data representatives to provide a sounding board for new state-level data development, to proactively develop a list of priorities in state-level data development, and to allocate additional funds agencies for priority projects.
Push Ourselves First. As someone who takes time to read most notices about data collection on the Federal Register, I applaud the few efforts that actually submit meaningful requests for feedback and are open to feedback submitted. In reality these processes are largely a formality that is hidden away so that as few people as possible find them. Even submitted comments following outlined procedures can be dismissed as “it’s too late in the process.” Let’s stop this self-perpetuating process of mediocrity. If we are going to seek comment, seek comment.
Recommendation: Look for feedback and post prospective notices of data to be collected alongside the actual web pages where data is disseminated and start to create opt-in user groups.
I’ve offered a lot of criticism here but I recognize that we at Kauffman still have a lot to learn in this regard. One of the things are beta testing is this concept of a Data Wiki for the Kauffman Firm Survey. It’s not something which was easy for us to do and opens us up to user feedback, positive and negative, in a much greater way that we’ve done before. But this is also an area where we believe what we learn can help to inform the statistical agencies.
In my opinion, until OMB and the agencies become more genuine in their efforts to draw data users into the conversation we have little room to complain about lack of funding. These are things within their power to change. We cannot afford to idly hope for improvements in national statistics. I hope others out there agree with me and will add your voice to the discussion.
7/1/2010 8:00:00 AM By
E.J. Reedy
Let me preface this post by saying I am not a macro guy. I get the national accounts, but only so far. But I understand the concept that we are misconceptualizing our national output by not recognizing that R&D investments are longer-term investments and should be capitalized.
The Bureau of Economic Analysis has a new data release looking at revised GDP estimates through 2007 if R&D investments were capitalized (which it has not been in the past). The biggest thing that stood out to me in their analysis was this: "Biotechnology and information, communication, and technology (ICT) industries accounted for four-fifths of the business sector’s R&D contribution to GDP growth between 1998 and 2007." While ICT is a usual suspect in driving growth during this period, I hadn't expected to see the significant growth in biotechnology. This makes me think back to some recent testimony from a colleague, Lesa Mitchell, which included a section on the seemingly sad returns to recent investments in life sciences research.
More detailed information about the 2010 R&D Satellite Account will be available in an article in the Survey of Current Business later this year. The article will describe the satellite account in more detail and will include statistics for R&D-intensive industries for 1987-2007, as well as statistics for the regional and international accounts.
6/30/2010 8:37:38 AM By
E.J. Reedy
This month, Kauffman's Ideas at Work features an article by me highlighting some of the key successes and learnings from our work to improve data over the last decade. Take a look.
6/29/2010 4:00:00 PM By
E.J. Reedy
Still lots of interest in small business financing information but to date no U.S. statistical agencies are stepping up to the plate to replace the Federal Reserve's cancellation of the Survey of Small Business Finance (Rebel Cole has an interesting piece out today using this data for the SBA). John Tozzi has an article reviewing small business lending data in the U.S. for Bloomberg Businessweek. And, lastly, the G-20 SME Finance Working Group on Data has preliminary outcome steps from a recent meeting of experts.
6/29/2010 8:00:00 AM By
E.J. Reedy
The National Science Foundation (NSF) has an exciting new survey in the works on microbusiness innovation (which was featured at our recent Kauffman Interagency Forum on Entrepreneurship and Innovation Data). They are seeking interested contributors to the development of this work.
The Science Resources Statistics (SRS) Division of the National Science Foundation is planning a survey of microbusinesses (fewer than five employees). The microbusiness survey will collect data on R&D, innovation and related activities (such as sales of significantly improved goods and services; operating agreements and licensing activities; technology transfer; patents and intellectual property; and sources of technical knowledge), and measures of entrepreneurial effectiveness.
As we move forward in designing the survey we will be conducting workshops to help (1) gain a better perspective on data user needs and priorities of needs among users and (2) understand how microbusiness data will be used. Potential users include, but are not limited to, government officials at the federal, state, and local levels; international users; businesses and trade associations; and academic researchers. In addition, there are likely to be other categories of users that have not been specifically identified, as this is a new area of study.
If you are interested in contributing to the microbusiness discussion please forward your name to Audrey Kindlon at akindlon@nsf.gov.
6/25/2010 10:33:15 AM By
E.J. Reedy
One of my favorite surveys done on entrepreneurship is the Eurobarometer out of the European Union (EU) through Gallup. It is the result of the European Union’s increased focus on entrepreneurship development following their 2000 meeting in Lisbon. The EU Commission is kind enough to include several non-European Union countries in the survey (this year had the U.S. and China, in particular). So, their roughly triennial survey brings important cross-country comparisons. The 2009 version of this survey was recently published.
The current year survey instrument (which is at the back of the report) is a big improvement over past instruments and tries to get into topics like how entrepreneurs are viewed in society, the experience of the population with entrepreneurial endeavors, and perceptions of risks, barriers, and opportunities.
The overview report is about 200 pages so there is too much to cover here but a couple of things that stood out to me in my first pass of the report:
- “Looking at those who were not self-employed, in most countries, the proportion of respondents who considered it feasible to become self-employed in the next five years was lower than the proportion of those who would like to be self-employed. In the Nordic countries, however, an opposite trend was seen – i.e. the preference to be self-employed was lower than the perceived feasibility of gaining such a status; for example, 49% of non-self-employed Swedish respondents said it would be feasible to become self-employed in the next five years, whereas just 28% had an actual preference for changing their status.” This is an interesting measure – plans vs. perceptions – and I’d imagine our colleagues in the Nordic countries in charge of entrepreneurship should be very pleased with this result.
- The rate of entrepreneurial activity for the U.S. that they measure has gone up from 15 percent in 2004 to 18 percent in 2007 and 21 percent in 2009. My colleague Mike Horrell pulled together some of the historical data from their reports to show that this change is driven by an increase in reported embryonic activity as well as those reporting work in established businesses (see table of all countries measured in Eurobarometer).
- This is one of the few surveys that can tell us anything about how efforts to promote entrepreneurship through education are going. What they find is “a comparison, between 2007 and 2009 results, concerning the extent to which respondents agreed that their school education prepared them to become entrepreneurs showed that, in 2007, a number of European countries scored better than the US in stimulating entrepreneurship (e.g. Norway 74%, Portugal 71% vs. US 63%); in 2009, however, the US outscored all European countries (US 73%, Cyprus 64%, Portugal 63%).”
6/24/2010 3:00:00 PM By
E.J. Reedy
Few would dispute the need to invest in research and development activities but the impact measurement of such investments in most cases can feel rather unsatisfying. The Chronicle of Higher Education in their June 10 issue highlighted one new project piloted as a result of the Federal economic stimulus program last year with leadership at the National Science Foundation and other funding agencies. Star Metrics is a program which creates a base of knowledge for future impact studies by allowing university HR systems to directly tag individuals being paid under federally-funded research projects. These underlying records are then aggregated automatically to create a database looking at the names of scientists (and emerging scientists) who do work on specific research projects. With this backbone, it should allow for much easier (and mostly automated) matching of citation databases (publication, patent, and other) or other output measures (such as companies formed, disclosures of inventions) in future years.
This is a direction which makes a lot of sense to me as it appears less burdensome on universities than many possible alternatives while potentially giving us much more detail on the people working on specific projects. Some of these people will be postdoc researchers, a nebulous area of the scientific human capital pipeline that are all too often missed in measurements of the scientific workforce. We are funding some work in this area through the STARS Database being developed by Lynne Zucker and Michael Darby but it is not yet available for researchers.
June 9, 2010
Tracking the Value (at Least in Jobs) of Federal Research Spending
By Paul Basken
Chronicle of Higher Education
Washington, DC
More than five years ago, the then-White House science adviser, John H. Marburger III, asked researchers a seemingly simple question: Given the billions of tax dollars they get each year, why don't they have good data on the value of what they produce?
That question may finally be getting answered.
Sparked by job-tracking requirements in the $787-billion economic-stimulus bill approved last year by Congress, the government's major science-financing agencies have been working with universities to devise a way to bring scientific precision to the explanation of how their expenditures help the national economy.
The organizers of the effort, known as Star Metrics, reached a milestone this month by declaring the success of what they consider a critical element—their ability to create a low-cost system for universities to easily compile and assemble their records of federal grants, including the faculty members and students employed by them, in a way that the information can be automatically fed into the overall nationwide database.
Their method, tested at seven universities, took the institutions only about 12 to 15 hours of staff time to set up, and virtually no time to run after that, said Julia I. Lane, a program director at the National Science Foundation.
Users agree. "It was literally a nonissue in terms of the administrative burden," said Susan Wyatt Sedwick, associate vice president for research and director of the Office of Sponsored Projects at the University of Texas at Austin, one of the participating institutions.
The idea, eventually, is to create a comprehensive system that relies largely on existing computerized databases—such as science-journal publications, patent registrations, venture-capital expenditures, and employment histories—to give policy makers a precise idea of what they are getting for their more than $31-billion of annual federal spending on university-based research.
"There is huge pressure" from the White House and Congress, Ms. Lane said, "to show the value of investments."
Benefits and Concerns for Institutions
Universities are largely cheering the project. For one thing, Ms. Sedwick said, the administrative burden alone on the university might be reduced, because current federal data requirements translate into more than two hours of work for each of the 160 or more grants that Austin received in the past year.
And, said Tobin L. Smith, associate vice president for federal relations at the Association of American Universities, institutions surely will be helped by the ability to tell taxpayers with greater precision the benefits of their expenditures. "It's all in all a pretty good thing," Mr. Smith said.
The organizers of Star Metrics are emphasizing to universities the voluntary and cooperative nature of the project, particularly its reliance on low-cost sources of existing data, Ms. Lane said.
The idea grew out of a 2005 speech by Mr. Marburger, then serving as director of the Office of Science and Technology Policy in the administration of President George W. Bush, to the American Association for the Advancement of Science. Mr. Marburger pointed out that systems for measuring the effectiveness of science spending were already nearly three decades old. A National Research Council report, he said, called them "often ill-suited for the purposes to which they have been employed."
The effort took a more concrete form after the economic-stimulus measure, containing more than $15-billion in federal money for science research, required recipients to report back to the government on the jobs being created by their projects.
The resulting Star Metrics effort doesn't come without some concern and uncertainties. For one thing, Ms. Lane said, the voluntary nature of the effort leaves unclear its ultimate comprehensiveness. And the final level of detail is also unclear at this stage, she said, with many questions remaining about how smoothly all the anticipated sources of data will be linked together.
Also, some colleges have objected in other contexts to the inclusion of personally identifiable data in nationwide research projects. In the case of Star Metrics, Ms. Lane said, all faculty members and students involved in federally sponsored research are expected to be assigned some form of identification number, to allow tracking of their outside activities such as journal publications, patents, and jobs.
And some had greeted Mr. Marburger's original idea with suspicion, tied to their unhappiness with the Bush administration's level of support for science spending. Mr. Marburger may largely be trying, with the new measures, "to make it impossible to assess the Bush record relative to past spending," Sally T. Hillsman, executive officer of the American Sociological Association, wrote at the time.
Advocates of research spending also recognize the possible political risks of letting the government science budget be judged primarily for its job-producing qualities, even though they largely see that trade-off as positive.
New Incentives for Students
At the Polytechnic Institute of New York University, which was not one of the seven institutions that tested the data-collection system, the initiative raises hope of creating badly needed new incentives for students, said the institution's president, Jerry M. Hultin.
Research students, including prospective new faculty members, have broad ambitions for applying their research expertise toward solving problems in the commercial marketplace, and they too often feel "boxed in" by current measures of academic success that are tied to traditional pathways, such as journal-publication citations, Mr. Hultin said.
"This is an unboxing of young faculty," he said, "and letting them head to some of the places they really want to go."
In fact, one startup enterprise that originated from the institute, he said, could be part of the solution. The company, ChubbyBrain, is compiling a directory of private companies that contains detailed information in such areas as their financing sources, mergers and acquisitions, and customers.
That's exactly the type of easily obtainable information that organizers of Star Metrics may hope to incorporate into their network, Mr. Hultin said. Ms. Lane agrees. "The notion here is to leverage existing data," she said.
The assembled information could eventually form the basis for reports that reviewers, at agencies such as the NSF and the National Institutes of Health, consider when they decide which grant applications to approve, Ms. Lane said.
But that doesn't mean the potential for job creation will be the only or even the primary measure for future grant applications, she said. Although the impetus from the economic-stimulus measure centered on job creation, Star Metrics is being designed to include the effects of federal science spending in three other broad areas, including the generation of basic scientific knowledge and improvements in long-term health and environmental conditions.
A classic example that the developers of Star Metrics keep in mind, Ms. Lane said, is that of Sergey M. Brin, the co-founder of Google Inc. whose research at Stanford University—helped by a graduate fellowship from the National Science Foundation—led to the development, in a rented garage, of the world's most popular Internet search engine.
"One grant," she said, "doesn't typically lead to one particular outcome."
6/11/2010 8:36:35 AM By
E.J. Reedy
6/11/2010 8:01:34 AM By
E.J. Reedy
Few researchers realize it but the Internal Revenue Service (IRS) has a program to encourage research that require access to confidential tax micro data. Proposals are due to the IRS Office of Procurement by June 30, 2010. The Statistics of Income (SOI) Division "requires high quality proposals for research projects that necessitate access to federal tax micro data and that are for economic or statistical purposes. The results of these projects must provide insights that advance the administration of the Federal tax system and/or SOI operations. Proposals must also further SOI’s human capital development efforts. For this statement of work, consideration will be given only to proposals involving research on statistical methodologies, international income and taxation, non-profit organizations, estate and/or gift taxation, individual income taxation related to unincorporated businesses (except partnerships), or individual taxpayers with retirement accounts." (I've added the bold emphasis here to highlight the area of most likely entrepreneurship interest.)
One advantage of the IRS research program, besides the fact that they actually help pay for the research, is that the IRS has widely distributed offices around the country where the research can occur. The offices are in about sixty locations, if I remember correctly, compared to the Census Bureau's ten locations. We saw one proposal on entrepreneurship get approved last year so do know that proposals do actually get through but this is a proposal process where researchers really have to write both for themselves (in terms of the research-type questions they want to address) but also have to put themselves in the IRS's shoes to make clear how the IRS will benefit from the microdata work.
6/10/2010 9:00:00 AM By
E.J. Reedy
The 2010 Kauffman Interagency Forum on Entrepreneurship and Innovation Data (on June 4) brought together more than 50 representatives from the federal statistical community, academia, and other interested parties to talk about upcoming changes to ways in which the federal statistical agencies will be collecting data. The level of interaction among the agencies at the Forum and real interest in overall improvements was heartening to see. We were privileged to be the conveners of this conversation but clearly the hard work of making progress on entrepreneurship and innovation measurement is underway at the agency level. The Forum was mostly meant for in-person conversations and interaction but the agencies presenting did consent for us to post their PowerPoints for those interested. Below are the presentations and a couple of highlights from my perspective of their work.
Bureau of Labor Statistics ( QCEW PPT and NLSY PPT)
- BLS plans to release a new series of tables on age and survival of the establishment (and firm, I believe) in August 2010 as a part of the Business Employment Dynamics series. This will add to the flood of new data becoming available on cohorts of firms and should prove interesting to compare with the Census Bureau's Business Dynamics Statistics.
- Birth and Death counts, without suppressions, by detailed industry and county are in the works. The concept of county-level business data without suppression is something which should excite a lot of local researchers and economic development agencies as most county-level data has to be aggregated to a level which is frustrating for looking at specific industries.
- The National Longitudinal Survey of Youth is collecting data on their 1979 cohort related to activities in business ownership and patenting. While not available for another two years it should be rich data for researchers.
Census Bureau ( PPT)
- Updates are underway to the Business Dynamics Statistics that will bring it up to the end of 2008 and likely produce more useful tables for following cohorts of firms over time at the state level.
- Synthetic microdata will be available through the Cornell RDC in the fall for the Longitudinal Business Database. This will be interesting to watch as synthetic microdata isn't actual data but rather a data set which has been built off of real data but with crucial changes to aspects of the data to ensure confidentiality of individual responses. I think the consensus on synthetic microdata is still out but its definitely one of the emerging solutions to providing microdata access to government data in an era of increasing confidentiality concerns with putting public-use microdata files on the Internet.
- I am really excited to see that the integration of characteristics of business owners into the business data is finally on the agenda in an ongoing manner. There is so much potential for new useful public products here like women in high-tech businesses or high-growth minority businesses. Many policy interventions are aimed at underrepresented minority groups in entrepreneurship and this has the potential to finally allow for tracking of changes in this arena annually, not just every five years, which is currently the most up-to-date data available matching business and ownership characteristics.
Federal Reserve Board ( PPT)
- The Federal Reserve didn't have huge updates for the room but did talk about their impromptu panel of households from the 2007 Survey of Consumer Finance who were contacted in 2009 again. Specific questions were included relating to the changes in business ownership among the panel which will help to inform some aspects of our understanding about the impact of the financial crisis on business-owning households.
National Science Foundation ( PPT)
- Anyone following NSF's innovation surveys over the last decade knows that this is a time of huge payoff for them. After much review, planning, and pain, the new Business R&D and Innovation Survey (BRDIS) is finally available to the public. And while the current releases are on 2008 collections, improvements made in 2009 look very positive are likely to provide even more meaningful information.
- Quite a bit of the discussion came around what to do on measuring technology commercialization in the university space. NSF indicated a new report should be coming out soon from an external group with recommendations for them on that topic. I'll be following up with that and adding more commentary then.
- The Microbusiness Innovation Survey is very early in development but still promises to be the most exciting possible change underway in the next few years in our understanding of innovation in the U.S. NSF is showing real leadership in taking on this survey and considering what topics are most appropriate to get innovation activities in very small businesses. More expert meetings and user groups are planned for this fall into 2010.
Patent and Trademark Office( PPT)
- Did you know that the Patent and Trademark Office now has a chief economist? Well, Stuart Graham, the first occupant of this seat, is really taking the lead in improving the data the PTO makes available from their work.
- The PTO has recently done a bulk upload of information to Google and has plans for many future transfers of patent and trademark data.
- The PTO is also actively looking at what other agencies would make sense to partner in matching PTO data into more traditional statistical agency products. The PTO isn't really a statistical office (at least yet) so I think they are smart to look at partnerships that could produce really meaningful new information.
|
|
Developing better data is part of Kauffman's long-term strategy for advancing better research and policy on entrepreneurship and innovation. Data Maven is place you can connect with new data developments, provide us feedback on possible new projects, and contribute to the community seeking to improve entrepreneurship and innovation measurement.
E.J. Reedy is a manager in Research and Policy at the Kauffman Foundation. Learn more ...
|
|