8. Assessment of the Plan
8.1. Introduction
In order to ensure the proper application of the Research and Innovation Plan, suitable systems must be established to follow up and assess the actions foreseen and the Plan as a whole. The aim of assessing the Plan lies in improving research and innovation policies in Catalonia with all systematic, independent (objective) and professionalised control based on indicators that allow us to determine whether the public resources for the promotion of research and innovation are used to comply with the objectives established in the Plan and in the specific programmes and competitive calls for applications.
The effort put into the production and exploitation of knowledge must be of use to the needs and interests of society and effectively and efficiently end up in it. Thus, by assessing the Plan, it will be possible to improve the science and technology policy, and re-steer or correct the aspects that can be improved or are inefficient in accordance with this country’s needs for research and innovation.
In short, a good assessment system must be available in order to carry out a transparent, rigorous research and innovation policy, which is effective in the satisfaction of the particular objectives of each programme and of those linked with the research and innovation system as a whole. In this way, in the Research and Innovation Plan 2005-2008, the systems and mechanisms for the assessment of the policy on research and innovation included in previous research and innovation plans will be reinforced. In this respect, assessment activities carried out by existing bodies will be promoted. An independent, professionalised Catalan Agency for Assessment, will be set up, which, in coordination with all other bodies, will complement the assessment of the science and technology policy.
Likewise, a good system of assessment will be guaranteed that will allow us to determine the effectiveness of the measures aimed at promoting the real and effective equality of women and men in the Plan’s areas of intervention.
8.2. Plan Assessment System
The assessment of research and innovation activities will be done at different levels at different times. The basic levels distinguished are the assessment of projects and of competitive calls for applications, of programmes and the overall assessment of the Plan.
a) The assessment of projects and competitive calls for applications
The assessment process will start at micro level with the assessment of projects. This includes the ex ante assessment of the proposals with the establishment of the criteria and indicators that allow the selection of projects that are better situated with regard to the aims of the competitive call for applications. This assessment determines the pertinence of the project and the quality of its aims, the work method and the results expected. Thus, assessment indicators will be established for the medium term, which will be measured throughout the execution of the project, with the aim of checking the compliance of the activities and the foreseen objectives and make any necessary amendments. There will also be an ex post facto assessment, once the project has finished, in order to determine its success and thus how to assess the results, effects and repercussions it has brought about.
The bodies directly related with the planning and management of each of the competitive calls for applications that will be included in the Plan’s programmes and subprogrammes will intervene in the definition of the control and assessment indicators. The assessment of projects requires combining two types of criteria and indicators. The first group of indicators has a quantitative nature and refers to the capacity and quality of the corresponding research and innovation of the researcher, research groups or enterprise; the second group serves to evaluate the relevance, innovative nature and quality of the project. For this evaluation, experts in the specific field of research and innovation offering guarantees of independence will be called upon.
For each competitive call for applications, the control and assessment indicators will be established in the medium term and ex post facto so as to determine what their operation has been and whether the aims for which they were drafted have been fulfilled. The bodies directly relating to the planning and management of each of the competitive calls for applications that will be included in the Plan’s programmes and subprogrammes will intervene in the definition of these indicators.
b) The assessment of programmes
For each programme of the Plan, the specific aims and a set of indicators of reference have been defined to analyse the evolution of the specific aspects of the research and innovation system which the programme affects. The indicators will provide information on the state of the situation as regards the themes that are the object of the programme and they also have an anticipatory function on possible necessary programmes and structures of new creation.
c) Global follow-up and assessment of the Plan
The global follow-up and assessment of the Plan will be carried out by means of a set of indicators, instruments and bodies. Thus, two periodical reports will be drafted, the first a follow-up of the Plan and the second, an analysis of the evolution of the research and innovation system. These reports will be complemented with the actions that the Government of Catalonia’s research assessment agencies consider necessary, with the intervention of an international assessment committee and with the final assessment of the effect of the Plan on the research and innovation system and on the economic growth and welfare of Catalonia.
Annual report on the Research and Innovation Plan
Each year, a follow-up report will be drawn up on the actions and programmes of the Plan in which the indicators defined for each programme of action are integrated. This report, co-ordinated by the CIRIT, will allow the evaluation of the Plan’s good execution and compliance with the established objectives.
Report on the research and innovation system
A report will also be prepared on the evolution of the indicators of the research and innovation system to oversee its situation and evolution and assess the consequences and effect of the Plan. This report will include at least two sets of indicators. The first set will consist of global, easily compared indicators and will be used to examine the current situation in Catalonia and the aims of the Plan annually and in which a comparison will be made with Spain, the European Union and the leading innovative regions. The second set is complementary in nature, and is made up of different statistical sources that are also related with actions of the Plan and it contains information on resources, results, the co-ordination of the system (the transfer of knowledge and technology) and participation in competitive calls for programmes of the Spanish State General Administration and of the European Commission. The follow-up and analysis of the research and innovation system require a great improvement to the available statistical information. For this reason, during the process of drafting the Plan, a joint working group was set up between IDESCAT - the Catalan Statistics Institute, the DURSI and CIDEM. The aim is to make headway in the quality and the availability of information concerning the research and innovation system, and to place the development of these statistics on equal terms with the central role played by the policy on promoting research and innovation.
International assessment committee
So as to guarantee independent, high quality assessment, a committee will be set up principally consisting of international experts. Its purpose will be to assess the functioning of the Plan prior to its second year in force and to propose, if necessary, any measures and actions to improve the policies for the promotion of research and innovation of the Government of Catalonia.
Final assessment
Once the Plan finishes, there will be an assessment to determine the results achieved as well as the effects and impacts produced. This assessment will include analysis of the quantitative and qualitative effects on the direct receivers of the actions, the study of the additional nature of their research and innovation activities, the effects on other agents derived from the possible existence of external economies and the global consequences on the economy as a whole. The final assessment is fundamental in determining the possible benefits of the policy and to learn from experience. To carry out this assessment, there is a need for abundant information and for the assessment, like the Plan, to be an integral part of the science and technology policy.