Indexes: Challenges and Opportunities for Policymakers and Researchers

aug indexesThe need for measures is evident across governments and in the field of public policy monitoring and evaluation. To make the case for particular policy strategies and to monitor their peers, decision makers increasingly rely on numbers, charts and rankings. In the field of entrepreneurship, we are seeing new efforts to quantify public policy performance and compare it with the goals of other governments as well as against other countries.

Recently we have seen an increase in the number of country indexes that rank relative performance toward a wide range of development goals. The annual releases generate media attention, expanding the audiences of those who use them. Calls for better benchmarking, measurement, mapping and comparisons in entrepreneurship are fueling this but also raising the bar for economists and researchers in improving their accuracy and methodology.

An index is an approach for assessing public policy that is derived from and points into some greater volume of data, information, or knowledge. The best indexes comprehensively map a specific policy area and facilitate the assessment and evaluation of multiple interrelated metrics within a single, comparable measure. With regard to entrepreneurship, one well-known example is the Kauffman Index, which since 2004 has brought together the latest data available on entrepreneurial trends within the United States at the state and city levels.

The Kauffman Foundation’s index has three components: 

  • Rate of new entrepreneurs (an early and broad measure of business ownership);
  • Opportunity share of new entrepreneurs (a proxy of the percent of new entrepreneurs starting businesses who see market opportunities); and 
  • Startup density (the number of startup firms divided by total population of a place – something I recently discussed in this blog). 

The Kauffman Index is carefully constructed, using robust, up-to-date data, and peer-review-validated methodologies. 

For the past 10 years, the Kauffman Index has been a trusted, early indicator of entrepreneurial activity in the U.S. used by entrepreneurs and policymakers from the federal to state and local levels. It offers reliable clarity and insight into entrepreneurial trends. For governments, think-tanks and entrepreneurship supporters, it provides a window for identifying policy successes and helps point out areas where targeted interventions would have the most impact on an ecosystem.

In recent years, there has been a noticeable increase in the publication of cross-country indexes, the majority of which were developed in the 2000s. The demand for these indexes is driven by similar multiple factors, such as the “urge to measure,” their growing operational relevance in decision-making, and, according to Andrew Sharpe, who conducted a review of their growing use, “a recognition of the integral role macro-indicators play as a tool for evaluating the impact of policy.”

An OECD handbook says that cross-country policy indexes are “indispensable tools for policy analysis and for raising awareness and discussion with the public in general, through their explicit rankings and benchmarking.” Indexes also make it easier for the general public to identify common trends across many separate measures. Indeed, indexes that benchmark and rank different countries are popular with the media, which helps to spotlight key issues, such as entrepreneurship policy, in the public domain. Further, highlighting a state’s or country’s relative performance has become a motivating factor, triggering political pressure which encourages additional policy making.

One comprehensive effort is the Global Entrepreneurship Index (GEI) which provides a set of 14 comparable values as a frame of reference for entrepreneurship development across more than 130 of the world’s countries. GEI is produced by the GEDI Institute and is comprised of a collection of data on the entrepreneurial attitudes, abilities and aspirations of the local population weighted against the prevailing social and economic “infrastructure” – including such aspects as broadband connectivity and the transport links to external markets. The process results in 14 “pillars” that are used to measure, assess, and compare the health of national ecosystems. For example, a recent analysis of Kenya by the GEDI Institute points to progress that may not otherwise have been evident. 

But, as the Economist pointed out last year, there are clearly significant challenges and limitations to quantifying public policy decisions and performances. According to Neil Carter’s How Organisations Measure Success, the current use of indexes to measure the effectiveness of government policy is a recent development having its origins in the 1980s when the British government, seeking to decentralize while responding to calls for increased accountability, searched for methods to assess and improve performance. To provide greater value for taxpayer money, the U.K. government developed sets of indicators as a systemic way for assessing and measuring its own performance.

Many other countries followed suit, and developed indicators of their own. When combined, they become composite indicators, which condense a wide range of information on different but related phenomena into a single measure. The first composite indicators published were developed by Moody’s in 1914 to evaluate United States bonds and became its famous Sovereign Credit Ratings. An index is created by pooling composite indicators to evaluate performance across different actors (such as entrepreneurs, mentors, and angel investors).

But this use of policy indexes to apply analysis to policy development by turning disparate datasets into useful, actionable information for government decision makers is where the debate over policy indexes rages. Indexes such as the Kauffman Index that are fairly constructed with composite indicators that are directly relevant to the task at hand can indeed translate and transform a wide scope of information into a form useful for policy making. However, as Martin Ravallion has pointed out, there is no agreed upon and standardized technique for the normalizing and weighting of indicators. As a result, the process remains in general subjective and not all indexes are of high quality and those that are poorly constructed can send misleading policy messages. 

While researchers and economists work on improving sources of data and more accurate and cost effective ways of collecting data alongside other issues around standardizing methodologies, policymakers should not shy away from indexes but utilize them carefully. Rather they should view the “big picture” results that policy indexes produce as a starting point for policy makers. GEDI for example has emphasized this message suggesting policymakers who use the GEI, further test and validate conclusions - and even offers specialists to help do this.

On balance, there is no doubt that policy indexes have become invaluable tools for comparing public policy across countries. By summarizing complex, multidimensional phenomena, capturing the dynamic nature of entrepreneurship, and assessing progress over time, the better entrepreneurship indexes enable their users to understand the relative strengths and weaknesses of a given location, be it an entrepreneurship ecosystem or a city or a country. Even with the imperfect science within them, by placing entrepreneurship performance and progress in the public eye, these indexes facilitate better understanding and the development of the kinds of smart policies that foster and support entrepreneurs going forward – even if like science itself, their most important role is in defining fresh unanswered questions. 

Photo Credit: Flickr


comments powered by Disqus