Strategies for Executing Federated Queries in SPARQL1.1

A common way for exposing RDF data on the Web is by means of SPARQL endpoints which allow end users and applications to query just the RDF data they want. However, servers hosting SPARQL endpoints often restrict access to the data by limiting the amount of results returned per query or the amount of queries per time that a client may issue. As this may affect query completeness when using SPARQL1.1's federated query extension, we analysed different strategies to implement federated queries with the goal to circumvent endpoint limits. We show that some seemingly intuitive methods for decomposing federated queries provide unsound results in the general case, and provide fixes or discuss under which restrictions these recipes are still applicable. Finally, we evaluate the proposed strategies for checking their feasibility in practice.

[1]  J. Schumpeter.  The Theory of Economic Development: An Inquiry into Profits, Capital, Credit, Interest, and the Business Cycle , 1934 .

[2]  Hallie Preskill,et al.  Utilization-focused evaluation: The new century text (3rd edition) , 1998 .

[3]  B. Gibson,et al.  From Transfer to Transformation: Rethinking the Relationship between Research and Policy , 2003 .

[4]  Igor Mayer,et al.  A Policy Sciences View on Policy Analysis , 2013 .

[5]  Luke Georghiou,et al.  Challenging Europe?s Research ?Rationales for the European Research Are , 2008 .

[6]  Geert Bouckaert,et al.  Die Dynamik von Verwaltungsreformen: Zusammenhänge und Kontexte von Reform und Wandel , 2004 .

[7]  Melvin M. Mark.  Toward better research on—and thinking about—evaluation influence, especially in multisite evaluations , 2011 .

[8]  Carol H. Weiss,et al.  Have We Learned Anything New About the Use of Evaluation? , 1998 .

[9]  W. Scott.  'Institutionalized Organizations: Formal Structure as Myth and Ceremony', American Journal of Sociology, 83, pp. 340-63. , 2016 .

[10]  Philippe Larédo,et al.  Research and Innovation Policies in the New Global Economy: An International Comparative Analysis , 2001 .

[11]  Carol Hirschon Weiss.  Rooting for Evaluation: A Cliff Notes Version of My Work , 2004 .

[12]  Eun-Kyung Chung,et al.  Program Evaluation: Alternative Approaches and Practical Guidelines, 3rd Edition (Jody L Fitzpatrick, James R Sanders, Blaine R Worthen) , 2008 .

[13]  C. Hood.  A PUBLIC MANAGEMENT FOR ALL SEASONS , 1991 .

[14]  Robert D. Putnam,et al.  Bowling alone: the collapse and revival of American community , 2000, CSCW '00.

[15]  Erik-Hans Klijn,et al.  Introduction: A Management Perspective on Policy Networks , 1997 .

[16]  Richard Rose,et al.  What is Lesson-Drawing? , 1991, Journal of Public Policy.

[17]  Richard Rose,et al.  Lesson-drawing in Public Policy: A Guide to Learning Across Time and Space , 1993 .

[18]  I. Sanderson.  Evaluation, Policy Learning and Evidence-Based Policy Making , 2002 .

[19]  D. North,et al.  Economic performance through time , 2012 .

[20]  Donald Kossmann,et al.  The state of the art in distributed query processing , 2000, CSUR.

[21]  K. Weick.  FROM SENSEMAKING IN ORGANIZATIONS , 2021, The New Economic Sociology.

[22]  Tom Christensen,et al.  The Whole‐of‐Government Approach to Public Sector Reform , 2007 .

[23]  Huey-Tsyh Chen,et al.  A Comprehensive Typology for Program Evaluation , 1996 .

[24]  Marcelo Arenas,et al.  Semantics and complexity of SPARQL , 2006, TODS.

[25]  Norman Lee,et al.  Evidence-based policy-making in Europe: an evaluation of European Commission integrated impact assessments , 2006 .

[26]  Koen Verhoest,et al.  Agencification Processes and Agency Governance: Organizational Innovation at a Global Scale? , 2013 .

[27]  K. Leithwood,et al.  Current Empirical Research on Evaluation Utilization , 1986 .

[28]  Katy Whitelegg.  Patchwork Policy Making – Linking Innovation and Transport Policy in Austria. Austrian Transport Case Study for the OECD NIS MONIT Network , 2004 .

[29]  Claire A. Dunlop,et al.  SYSTEMATIZING POLICY LEARNING: FROM MONOLITHS TO DIMENSIONS , 2011 .

[30]  S. Verba,et al.  The Civic Culture: Political Attitudes and Democracy in Five Nations , 1964 .

[31]  C. Weiss.  Using social research in public policy making , 1979 .

[32]  Peter Carswell,et al.  Understanding Evaluation Influence Within Public Sector Partnerships , 2012 .

[33]  Franco Malerba,et al.  Increase Learning, Break Knowledge Lock-ins and Foster Dynamic Complementarities: Evolutionary and System Perspectives on Technology Policy in Industrial Dynamics , 2009 .

[34]  Sven Steinmo,et al.  Structuring politics : historical institutionalism in comparative analysis , 1992 .

[35]  Johan P. Olsen,et al.  The New Institutionalism: Organizational Factors in Political Life , 1983, American Political Science Review.

[36]  Jenny Pickerill,et al.  Why Does Occupy Matter? , 2012 .

[37]  Tanja A. Börzel,et al.  European Governance: Negotiation and Competition in the Shadow of Hierarchy , 2010 .

[38]  C. Lindblom.  THE SCIENCE OF MUDDLING THROUGH , 1959 .

[39]  John M. Owen,et al.  Evaluation culture: A definition and analysis of its development within organisations , 2003 .

[40]  Bas Straathof,et al.  More R&D with tax incentives? A meta-analysis , 2015 .

[41]  Suzanne Mettler,et al.  Reconstituting the Submerged State: The Challenges of Social Policy Reform in the Obama Era , 2010, Perspectives on Politics.

[42]  Bjorn Johnson,et al.  A policy learning perspective on developing sustainable energy technologies. Argumenta Oeconomica No 2 (23) 2009, s. 9-33 , 2008 .

[43]  J. Bradley Cousins,et al.  Commentary: Minimizing Evaluation Misuse as Principled Practice. , 2004 .

[44]  Matthias Wingens,et al.  Toward a General Utilization Theory , 1990 .

[45]  Lyn M. Shulha,et al.  Evaluation Use: Theory, Research, and Practice since 1986 , 1997 .

[46]  Claudio M. Radaelli,et al.  The role of knowledge in the policy process , 1995 .

[47]  Paul DiMaggio.  Interest and Agency in Institutional Theory , 1988 .

[48]  Renate E. Meyer,et al.  Public Management Reform: An Identity Project , 2006 .

[49]  C. Boswell,et al.  The political functions of expert knowledge: knowledge and legitimation in European Union immigration policy , 2008 .

[50]  Peter Teirlinck,et al.  Chapter 2: The practice of impact assessment in the policy cycle in Europe , 2011 .

[51]  Christine Nadel,et al.  Case Study Research Design And Methods , 2016 .

[52]  P. Wells.  New Labour and Evidence Based Policy Making , 2004 .

[53]  Mathias Kluge.  Using Evaluations Does Evaluation Make A Difference , 2016 .

[54]  Kurt Mayer.  Running after the international trend: Keynesian power balances and the sustainable repulsion of the innovation paradigm in Austria , 2003 .

[55]  C. Weiss.  The many meanings of research utilization. , 1979 .

[56]  Cheryl-Anne N. Poth.  Promoting evaluation use within dyamic organizations: a case study examining evaluator behaviour , 2008 .

[57]  Keith Smith.  Innovation as a Systemic Phenomenon: Rethinking the Role of Policy , 2000 .

[58]  D. Easton,et al.  The Political System: An Inquiry into the State of Political Science. , 1953 .

[59]  John Young,et al.  Bridging Research and Policy: An Annotated Bibliography , 2002 .

[60]  Koen Verhoest,et al.  Government agencies : practices and lessons from 30 countries , 2012 .

[61]  Anthony B. Atkinson,et al.  Top incomes over the twentieth century: A contrast between continental european and english-speaking countries , 2007 .

[62]  Melvin M. Mark,et al.  The Mechanisms and Outcomes of Evaluation Influence , 2004 .

[63]  Ross Upshur,et al.  Evidence-based health policy: context and utilisation. , 2004, Social science & medicine.

[64]  Marisa Allen,et al.  STEEPING THE ORGANIZATION'S TEA: EXAMINING THE RELATIONSHIP BETWEEN EVALUATION USE, ORGANIZATIONAL CONTEXT, AND EVALUATOR CHARACTERISTICS , 2010 .

[65]  Luke Georghiou,et al.  Evaluation of national foresight activities: Assessing rationale, process and impact , 2006 .

[66]  P. Sabatier,et al.  Advocacy-Koalitionen, Policy-Wandel und Policy-Lernen: Eine Alternative zur Phasenheuristik , 1993 .

[67]  B. Peters.  Institutional Theory in Political Science: The New Institutionalism , 1999 .

[68]  N. C. Bandelow,et al.  Lerntheoretische Ansätze in der Policy-Forschung , 2003 .

[69]  Karl E. Weick,et al.  Puzzles in Organizational Learning: An Exercise in Disciplined Imagination , 2002 .

[70]  Lars Frederiksen,et al.  The Agora and the Role of Research Evaluation , 2003 .

[71]  Charles O. Jones.  An introduction to the study of public policy , 1977 .

[72]  Rupert Pichler.  Delegierte Koordination. Zur Koordinationsfunktion intermediärer Organisationen in der österreichischen Forschungsförderung , 2014 .

[73]  John W. Kingdon.  Agendas, alternatives, and public policies , 1984 .

[74]  John Clarke,et al.  The Managerial State: Power, Politics and Ideology in the Remaking of Social Welfare , 1999 .

[75]  Jurian Edelenbos,et al.  Evaluation in Multi-Actor Policy Processes , 2006 .

[76]  Tanja A. Börzel,et al.  Organizing Babylon ‐ On the Different Conceptions of Policy Networks , 1998 .

[77]  Hallie Preskill,et al.  Evaluation and Organizational Learning: Past, Present, and Future , 2001 .

[78]  K. Eisenhardt.  Building theories from case study research , 1989, STUDI ORGANIZZATIVI.

[79]  F. R. Eisele,et al.  Modern Social Politics in Britain and Sweden: From Relief to Income Maintenance. , 1975 .

[80]  Michael Gibbons.  THE EVALUATION OF GOVERNMENT POLICIES FOR INNOVATION , 1984 .

[81]  P. Hedström.  Dissecting the Social: On the Principles of Analytical Sociology , 2005 .

[82]  K. Arrow.  Economic Welfare and the Allocation of Resources for Invention , 1962 .

[83]  Abdullah Gok,et al.  INNO-Appraisal. Understanding Evaluation of Innovation Policy in Europe , 2010 .

[84]  Susana Borrás,et al.  Policy Learning and Organizational Capacities in Innovation Policies , 2011 .

[85]  Morten T Thulstrup,et al.  European Welfare Policy in the Shadow of Hierarchy , 2005 .

[86]  E. Crewe,et al.  Bridging Research and Policy: Context, Evidence and Links , 2002 .

[87]  M. L. Oliver,et al.  Evaluation of emergency response: Humanitarian aid agencies and evaluation influence , 2008 .

[88]  Holger Bähr.  The Politics of Means and Ends: Policy Instruments in the European Union , 2010 .

[89]  Susan Fleck,et al.  The Compensation-Productivity Gap: A Visual Essay , 2011 .

[90]  J. Eduard Alexy,et al.  Tradition und Wandel , 1984 .

[91]  Daniel A. Levinthal,et al.  The myopia of learning , 1993 .

[92]  Jürgen Janger,et al.  Boosting Austria's Innovation Performance , 2007 .

[93]  C. Edquist,et al.  Institutions and Organizations in Systems of Innovation , 2013 .

[94]  M. Bleaney,et al.  Economic growth and income inequality , 2002 .

[95]  Rosalie Ruegg,et al.  A toolkit for evaluating public R&D investment models, methods, and findings from ATP's first decade , 2003 .

[96]  Terttu Luukkonen,et al.  Research evaluation in Europe: state of the art , 2002 .

[97]  Andrew van Hulten,et al.  Remapping the Fiscal State After the Global Financial Crisis , 2012 .

[98]  Brian Head,et al.  Three lenses of evidence-based policy , 2008 .

[99]  Valérie Pattyn.  Why organizations (do not) evaluate: a search for necessary and sufficient conditions , 2012 .

[100]  Luke Georghiou,et al.  Evaluating the demand side: New challenges for evaluation , 2012 .

[101]  Katja Hose,et al.  FedX: Optimization Techniques for Federated Query Processing on Linked Data , 2011, SEMWEB.

[102]  Klaus von Beyme,et al.  Political Institutions—Old and New , 2014 .

[103]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[104]  Jonathan V. Beaverstock,et al.  Servicing the Super-Rich: New Financial Elites and the Rise of the Private Wealth Management Retail Ecology , 2013 .

[105]  Susana Borrás,et al.  The innovation policy of the EU , 2001 .

[106]  Michael Quinn Patton,et al.  Discovering Process Use , 1998 .

[107]  B. Peters,et al.  The Politics of Path Dependency: Political Conflict in Historical Institutionalism , 2005, The Journal of Politics.

[108]  Jorge Pérez,et al.  Static analysis and optimization of semantic web queries , 2012, PODS '12.

[109]  Erin Murphy-Graham,et al.  An Alternate Route to Policy Influence , 2005 .

[110]  John W. Meyer,et al.  The “Actors” of Modern Society: The Cultural Construction of Social Agency* , 2000 .

[111]  Tanja A. Börzel,et al.  Governance without a state: Can it work? , 2010 .

[112]  Patricia H. Thornton,et al.  The Institutional Logics Perspective: A New Approach to Culture, Structure and Process , 2012 .

[113]  H.A.M. van Lieshout,et al.  Different hands. Markets for intermediate skills in Germany, the. U.S. and the Netherlands , 2008 .

[114]  John Mayne,et al.  Instruments and procedures for assuring evaluation quality: a Swiss perspective , 2005 .

[115]  Udo Kuckartz.  Qualitative Inhaltsanalyse. Methoden, Praxis, Computerunterstützung , 2016 .

[116]  B. David.  The iron cage revisited : Institutional isomorphism and collec... , 2007 .

[117]  Richard Peet,et al.  INEQUALITY AND POVERTY: A MARXIST-GEOGRAPHIC THEORY∗ , 1975 .

[118]  Raymund Werle,et al.  An Institutional Approach to Technology , 1998 .

[119]  Emmanuel Saez,et al.  How progressive is the US federal tax system? A historical and international perspective , 2007 .

[120]  Koen Frenken,et al.  A Complexity-Theoretic Perspective on Innovation Policy , 2016 .

[121]  Stefan Kuhlmann,et al.  Changing Governance in European Research and Technology Policy. Possible Trajectories and the ?European Research Area , 2003 .

[122]  Melvin M. Mark,et al.  Beyond Use: Understanding Evaluation’s Influence on Attitudes and Actions , 2003 .

[123]  Bruno S. Frey,et al.  Evaluitis - Eine Neue Krankheit , 2008 .

[124]  Anton Pelinka,et al.  Österreichische Politik : Grundlagen, Strukturen, Trends , 2007 .

[125]  Ulf Leser,et al.  Querying Distributed RDF Data Sources with SPARQL , 2008, ESWC.

[126]  C. Weible,et al.  The Advocacy Coalition Framework , 2012, Theories of the Policy Process.

[127]  Roger Friedland,et al.  God, Love, and Other Good Reasons for Practice: Thinking Through Institutional Logics , 2013 .

[128]  C. Jensen,et al.  A Study on R&D Tax Incentives , 2014 .

[129]  Manfred M. Fischer,et al.  How do firm characteristics affect behavioural additionalities of public R&D subsidies? Evidence for the Austrian transport sector , 2013 .

[130]  H. Rohracher,et al.  Legitimizing research, technology and innovation policies for transformative change , 2012 .

[131]  Peter Beike,et al.  Power And The Structure Of Society , 2016 .

[132]  Jeremy Richardson,et al.  Policy Styles in Western Europe , 1982 .

[133]  Philip Selznick,et al.  TVA and the Grass Roots; a Study in the Sociology of Formal Organization , 2010 .

[134]  Jason Seawright,et al.  Democracy and the Policy Preferences of Wealthy Americans , 2013, Perspectives on Politics.

[135]  David L. Rigby,et al.  International trade and wage inequality in Canada , 2010 .

[136]  Johan P. Olsen,et al.  Rediscovering institutions: The organizational basis of politics , 1989 .

[137]  Stefan Kuhlmann,et al.  Scenarios of technology and innovation policies in Europe: Investigating future governance , 2003 .

[138]  Phokion G. Kolaitis,et al.  The containment problem for Real conjunctive queries with inequalities , 2006, PODS '06.

[139]  Ville Valovirta.  Evaluation Utilization as Argumentation , 2002 .

[140]  Maribel Acosta,et al.  A Heuristic-Based Approach for Planning Federated SPARQL Queries , 2012, COLD.

[141]  F. W. Scharpf.  Interaktionsformen: Akteurzentrierter Institutionalismus in der Politikforschung , 2000 .

[142]  C. Achilles,et al.  Evaluation: A Systematic Approach , 1980 .

[143]  John A. McLaughlin,et al.  Logic models: a tool for telling your programs performance story , 1999 .

[144]  Gerhard Streicher,et al.  Stärkefelder im Innovationssystem. Wissenschaftliche Profilbildung und wirtschaftliche Synergien , 2015 .

[145]  Tanja A. Börzel.  Governance with/out Government , 2010 .

[146]  Matthew Sparke,et al.  From Global Dispossession to Local Repossession: Towards a Worldly Cultural Geography of Occupy Activism , 2013 .

[147]  Laura R. Peck,et al.  An Evaluation Use Framework and Empirical Assessment , 2009 .

[148]  Murray Saunders,et al.  The use and usability of evaluation outputs: A social practice approach , 2012 .

[149]  Christina A. Christie.  Reported Influence of Evaluation Data on Decision Makers’ Actions , 2007 .

[150]  V. Gilsing,et al.  A system failure framework for innovation policy design , 2005 .

[151]  Iain Hay,et al.  ‘That Tiny, Stratospheric Apex That Owns Most of the World’– Exploring Geographies of the Super‐Rich , 2012 .

[152]  D. Evers,et al.  UvA-DARE ( Digital Academic Repository ) Building for consumption : an institutional analysis of peripheral shopping center development in northwest Europe , 2004 .

[153]  Kari Lukka,et al.  The problem of generalizability: anecdotes and evidence in accounting research , 1995 .

[154]  Andreas Schibany,et al.  Instrumente der Technologieförderung und ihr Mix , 2005 .

[155]  M. Shubik,et al.  A Behavioral Theory of the Firm. , 1964 .

[156]  Emma Clarence,et al.  Technocracy Reinvented: The New Evidence Based Policy Movement , 2002 .

[157]  Emmanuel Saez,et al.  Income inequality in the United States , 2003 .

[158]  M. Becker.  Evidence Argument And Persuasion In The Policy Process , 2016 .

[159]  Philippe Larédo.  Six major challenges facing public intervention in higher education, science, technology and innovation , 2003 .

[160]  I. Sanderson,et al.  Evaluation in Complex Policy Systems , 2000 .

[161]  Claudio M. Radaelli,et al.  Europeanization, Policy Learning, and New Modes of Governance , 2008, Regional Comparisons in Comparative Policy Analysis Studies.

[162]  Staffan Jacobsson,et al.  The diffusion of renewable energy technology: an analytical framework and key issues for research , 2000 .

[163]  Nathan Caplan,et al.  The Two-Communities Theory and Knowledge Utilization , 1979 .

[164]  Fritz W. Scharpf,et al.  Der Ansatz des akteurzentrierten Institutionalismus , 1995 .

[165]  Erich Staudt.  Forschungs- und Technologiepolitik , 1996 .

[166]  Sascha Ruhland,et al.  Systemevaluierung der österreichischen Forschungsförderung und -finanzierung. Report 3: Governance in der FTI-Politik im Wechselspiel zwischen Ministerien und Agenturen , 2009 .

[167]  Michael Huberman,et al.  Steps Toward an Integrated Model of Research Utilization , 1987 .

[168]  Jürgen Umbrich,et al.  SPARQL Web-Querying Infrastructure: Ready for Action? , 2013, SEMWEB.

[169]  D. North.  Institutions, Institutional Change and Economic Performance: Economic performance , 1990 .

[170]  Hayagreeva Rao,et al.  Caveat Emptor: The Construction of Nonprofit Consumer Watchdog Organizations1 , 1998, American Journal of Sociology.

[171]  Erik Arnold,et al.  Evaluating research and innovation policy: a systems world needs systems evaluations , 2004 .

[172]  Carol Hirschon Weiss,et al.  The Interface between Evaluation and Public Policy , 1999 .

[173]  Sandra Nutley,et al.  Delivering evidence-based policy and practice: lessons from the implementation of the UK Crime Reduction Programme , 2006 .

[174]  A. Langer,et al.  How do researchers influence decision-makers? Case studies of Mexican policies. , 1999, Health policy and planning.

[175]  Gonzalo Carracedo,et al.  Ready for action , 2016, Journal of optometry.

[176]  Nora Jacobson,et al.  Consulting as a strategy for knowledge transfer. , 2005, The Milbank quarterly.

[177]  Erin Mehalic Burr.  Evaluation Use and Influence among Project Directors of State GEAR UP Grants , 2009 .

[178]  Erwin Häckel,et al.  Politische Kybernetik : Modelle und Perspektiven , 1973 .

[179]  T. Lawrence,et al.  Institutional Work: Refocusing Institutional Studies of Organization , 2011 .

[180]  M. Rogers,et al.  Innovation, Intellectual Property, and Economic Growth , 2010 .

[181]  Arthur Benz,et al.  Der moderne Staat. Grundlagen der politologischen Analyse , 2008 .

[182]  Michael Pregernig,et al.  Zwischen Alibi und Aushandlung — Ein empirischer Blick auf die Interaktion zwischen Wissenschaft und Politik am Beispiel der österreichischen Umwelt- und Ressourcenpolitik , 2007 .

[183]  Marvin C. Alkin,et al.  Evaluation Use Revisited , 2003 .

[184]  Claudia Goldin,et al.  The Great Compression: The Wage Structure in the United States at Mid- Century , 1991 .

[185]  Peter Biegelbauer,et al.  Wie lernt die Politik - Lernen aus Erfahrung in Politik und Verwaltung , 2013 .

[186]  H. Simon,et al.  Bounded rationality in social science: Today and tomorrow , 2000 .

[187]  Colin J. Bennett,et al.  The lessons of learning: Reconciling theories of policy learning and policy change , 1992 .

[188]  Marvin C. Alkin,et al.  UNBUNDLING EVALUATION USE , 2002 .

[189]  Lora Aroyo,et al.  The Semantic Web - ISWC 2011 - 10th International Semantic Web Conference, Bonn, Germany, October 23-27, 2011, Proceedings, Part I , 2011, SEMWEB.

[190]  W. Powell,et al.  The iron cage revisited institutional isomorphism and collective rationality in organizational fields , 1983 .

[191]  G. Dutfield,et al.  Innovation without patents: harnessing the creative spirit in a diverse world , 2008 .

[192]  Steven Højlund.  Evaluation use in the organizational context – changing focus to improve theory , 2014 .

[193]  C. Pollitt,et al.  Public management reform : a comparative analysis - new public management, governance, and the Neo-Weberian State , 2011 .

[194]  Daniel L. Stufflebeam,et al.  Evaluation Models: New Directions for Evaluation , 2001 .

[195]  Valerie J. Caracelli.  Evaluation use at the threshold of the twenty‐first century , 2000 .

[196]  Marvin C. Alkin,et al.  Thoughts on Evaluation Utilization, Misutilization and Non-Utilization. , 1988 .

[197]  Christian Rammer,et al.  Schwerpunktsetzungen in der Forschungsund Technologiepolitik – eine Analyse der Paradigmenwechsel seit 1945 , 2006 .

[198]  R. Friedland.  Bringing Society Back In : Symbols, Practices, and Institutional Contradictions , 1991 .

[199]  Jean A. King,et al.  Evaluation Utilization: A Literature Review and Research Agenda. , 1981 .

[200]  Patrick Kenis,et al.  Taking stock of policy networks : Do they matter? , 2007 .

[201]  H. Simon,et al.  Models Of Man : Social And Rational , 1957 .

[202]  Óscar Corcho,et al.  Federating queries in SPARQL 1.1: Syntax, semantics and evaluation , 2013, J. Web Semant..

[203]  Óscar Corcho,et al.  Semantics and Optimization of the SPARQL 1.1 Federation Extension , 2011, ESWC.

[204]  Benedetto Lepori,et al.  Between policy and science: Research councils' responsiveness in Austria, Norway and Switzerland , 2007 .

[205]  Philip Shapira,et al.  Learning from Science and Technology Policy Evaluation: Experiences from the United States and Europe , 2003 .

[206]  Karl Aiginger,et al.  Evaluation of Government Funding in RTDI from a Systems Perspective in Austria. Synthesis Report , 2009 .

[207]  K. Thelen.  Historical Institutionalism in comparative politics , 1997 .

[208]  C. Edquist.  Systems of Innovation: Technologies, Institutions and Organizations , 1997 .

[209]  Michael Schmidt,et al.  Foundations of SPARQL query optimization , 2008, ICDT '10.

[210]  Donald A. Schön,et al.  Organizational Learning: A Theory Of Action Perspective , 1978 .

[211]  Renate E. Meyer,et al.  OF BUREAUCRATS AND PASSIONATE PUBLIC MANAGERS: INSTITUTIONAL LOGICS, EXECUTIVE IDENTITIES, AND PUBLIC SERVICE MOTIVATION , 2014 .

[212]  Lora Aroyo,et al.  The Semantic Web – ISWC 2013 , 2013, Lecture Notes in Computer Science.

[213]  R. Nelson.  The Simple Economics of Basic Scientific Research , 1959, Journal of Political Economy.

[214]  Steven Højlund,et al.  Evaluation use in evaluation systems – the case of the European Commission , 2014 .

[215]  Merrill Goozner.  The coming transformation. , 2016, Modern healthcare.

[216]  Luke Georghiou.  Research evaluation in European national science and technology systems , 1995 .

[217]  F. Scharpf.  Games Real Actors Play: Actor-centered Institutionalism In Policy Research , 1997 .

[218]  P. Hall.  Policy paradigms, social learning, and the state: the case of economic policymaking in Britain , 1993 .

[219]  H. Gray,et al.  Why Egalitarianism Might Be Good for Your Health , 2007, Psychological science.

[220]  Lora Aroyo,et al.  The Semantic Web: Research and Applications , 2009, Lecture Notes in Computer Science.

[221]  A. J. Veal,et al.  The spirit level: why more equal societies almost always do better , 2011 .

[222]  Jörg Balsiger,et al.  Logic of appropriateness , 2014 .

[223]  J. March.  Bounded rationality, ambiguity, and the engineering of choice , 1978 .

[224]  R. Greenwood,et al.  Theorizing Change: The Role of Professional Associations in the Transformation of Institutionalized Fields , 2002 .

[225]  W. Jenkins,et al.  Policy Analysis: A Political and Organisational Perspective , 1978 .

[226]  Jonathan A. Morell,et al.  Why Are There Unintended Consequences of Program Action, and What Are the Implications for Doing Evaluation? , 2005 .

[227]  H. Gassler,et al.  Internationalisation of R&D and embeddedness: the case of Austria , 2008 .

[228]  Henry Chesbrough,et al.  Open Innovation: The New Imperative for Creating and Profiting from Technology , 2003 .

[229]  P. Sabatier.  Knowledge, Policy-Oriented Learning, and Policy Change , 1987 .

[230]  Keith Smith,et al.  Policy learning and innovation theory: an interactive and co-evolving process , 2002 .

[231]  B. Ebersberger,et al.  Improving policy understanding by means of secondary analyses of policy evaluation , 2008 .

[232]  Ben R. Martin,et al.  Innovation studies : evolution and future challenges , 2013 .

[233]  Markku Lehtonen,et al.  Environmental policy evaluation in the service of sustainable development: influence of the OECD Environmental Performance Reviews from the perspective of institutional economics , 2005 .

[234]  Klaus Gottstein,et al.  Political Problems Facing Governments in Using Scientific Advice, and Legal and Other Problems Facing Scientists when Trying to Give Independent Advice to Governments , 2008 .

[235]  Sandra Nutley,et al.  Bridging the policy / research divide Reflections and Lessons from the UK , 2003 .

[236]  Katherine E. Ryan,et al.  Exploring evaluator role and identity , 2002 .