Monday, November 21, 2016

Academic evaluation and hiring in Italy: The curious incident of Giovanni Sambin in the ASN 2016

Disclaimer: This post might contain imprecisions about the ASN, since I have never worked at an Italian university myself and I have never applied for the Italian ASN. I welcome corrections from whoever reads this post and has experience with this Italian evaluation exercise. Let me state at the outset that what I write pertains to fields such as computer science and mathematics. I do not know what is done in the humanities. 

An Italian law dated 30 December 2010 specifies a procedure for academic hirings in Italy at the level of associate and full professor. According to that law, recruiting for those positions should be "based on scientific qualification criteria. A national commission evaluates and assesses the candidates scientific qualification." See this outdated web site, which should be compared with the one in Italian.  Only candidates that have obtained the so-called Abilitazione Scientifica Nazionale (ASN, National Scientific Qualification) can then apply for a professor position at an Italian university, if and when such positions are advertised.

One can wonder why Italy uses this two-step system for academic hirings, whose need is not felt in any of the countries where I have worked so far. I guess that the first step is meant to filter out potential candidates who do not meet minimum requirements for being a professor at any of the 63 public universities in Italy. 

According to the regulations a "national commission evaluates and assesses the candidates scientific qualification." In fact, there is one national commission for each of the many scientific areas considered in Italy. Many of these commissions have to examine hundreds of applications, and their members play the role of gatekeepers and paladins of quality in the Italian university system. I can only imagine how much work is needed to do a thoughtful job in one of those committee and how easy it is to make enemies regardless of how considerate one is in justifying one's opinions. The evaluation is partly based on bibliometric criteria, which are known beforehand, should simplify the work of the commissions and should give a look of objectivity to their decisions. However, as far as I know, the commissions can also base their decisions on a qualitative analysis of the applicants.

Given the crucial role played by the members of the evaluation committees, one would expect that their members are chosen by taking the candidates' scientific profile and experience carefully into account. As it turns out, however, the qualifications of candidates for the committee are evaluated using only the following three bibliometric criteria:
  1. Number of publications in the period 2006-2016 (threshold 9);
  2. Total number of citations  in the period 2001-2015 (threshold 80);
  3. H-index in the last 15 years (threshold 5). 
In order to be eligible, one has to meet the thresholds in at least two of the above criteria. This might even seem reasonable. Note, however, that only publications indexed in Web of Science or Scopus count. In particular, journal papers published in outlets that are not indexed by Web of Science/Scopus are not taken into consideration (regardless of their content and impact) and conference papers don't count at all. Books and monographs don't count either, regardless of how influential they might be. Web of Science/Scopus are also used for calculating citations and the h-index. Again, this provides a smaller coverage than the one offered by Google Scholar, say.

By way of example, recently Giovanni Sambin, one of the most famous, currently active Italian logicians and an expert academic  one would trust to lead a national evaluation committee for Mathematical Logic, was considered to be ineligible as an evaluator because he met only one of the above-mentioned criteria. His Google Scholar profile is here.

This kind of decisions makes me wonder whether there is an overemphasis on bibliometric evaluations in Italian academia. If experience over a long and distinguished academic career plays second fiddle to fairly arbitrary thresholds calculated using only Web of Science and Scopus, I wonder how reliable the decisions of the evaluation committees will be considered by Italian academics. Most importantly, having so many people spend a lot of time seeking the holy grail of the national qualification and small committees devote endless hours examining their qualifications looks like a huge waste of energy and resources. I cannot help but think that that energy and time would be best used for research, teaching and all the other tasks that make up our work.

Tuesday, November 15, 2016

Workshops at LICS 2017

The following six workshops will be co-located with LICS 2017 and will take place on Monday, 19 June 2017, on the premises of Reykjavik University:
  • WiL: Women in Logic. Proposers: Valeria de Paiva, Amy Felty, Anna Ingolfsdottir, Ursula Martin
  • LCC: Logic and Computational Complexity. Proposers: Norman Danner, Anuj Dawar, Isabel Oitavem, Heribert Vollmer
  • LMW: Logic Mentoring Workshop. Proposers: Anupam Das, Valeria Vignudelli, Fabio Zanasi
  • LA: Learning and Automata. Proposers: Borja Balle, Leonor Becerra-Bonache, Remi Eyraud
  • LOLA: Syntax and Semantics of Low-Level Languages. Proposer: Matija Pretnar, Noam Zeilberger
  • Metafinite model theory and definability and complexity of numeric graph parameters. Proposers: Andrew Goodall, Janos A. Makowsky, Elena V. Ravve.
More details on this workshops will be available from the conference web page in due course. I hope that you'll consider attending them and submitting excellent papers to the conference. 

Wednesday, November 09, 2016

What makes a research institution excellent?

Some time ago I stumbled across the video of the panel discussion "IST Austria: On the Way to the Top: What Makes a Research Institution Excellent?". (There is also a much shorter, 11-minute version of the video here.) I watched the discussion with great interest, and found it inspirational and thought-provoking.

The panelists were Patrick Aebischer (president of EPFL until the end of 2016), Jonathan Dorfan (president of the Okinawa Institute of Science and Technology), Peter Gruss (former president of the Max-Planck-Gesellschaft from 2002 till 2014), Helga Nowotny (former President of the European Research Council), Rolf-Dieter Heuer (Director General of CERN from 2009 to 2015), Haim Harari (President, from 1988 to 2001, of the Weizmann Institute of Science) and Olaf Kübler (former president of ETH Zurich). It doesn't get much better than this, in terms of experience about the subject matter and, if you are interested in the topic or even just in hearing experienced academics discuss it, I'd encourage you to have a glass of your favourite beverage, relax and have a look. It is remarkable how much agreement there was in isolating the key ingredients leading to research excellence.

Here is my quarter-baked summary of some of the contributions, with apologies for not covering the whole discussion, possibly biased reporting and for any error I might have made.

Patrick Aebischer stated that Europe lacks super-brands such as Berkeley, CalTech, CMU, Harvard, MIT and Stanford. One needs elite universities to attract talents. The US attracts the best graduate students, the best young researchers with their tenure-track system and also people in high-ranking management positions. He also mentioned that to foster excellence, it is useful to have some competition between public and private universities. He said that integration of research and education is key to achieve excellence, as are attracting and keeping the best faculty, and giving early independence to young individuals. 
In order to achieve excellence, funding must be significant. A flexible organizational structure is needed to be able to compete at the highest level.
In this era, one can rise fast, but one can also fall faster than before.
Peter Gruss started by asking a fundamental question: what makes creative research possible? In his words, it is amazing how easy the answer is and how difficult it is to achieve it: "Hire the most brilliant minds and give them everything they need to stay brilliant." That's it.

He referred to the work of the historian Rogers Hollingsworth who isolated the following ingredients for excellence in research institutions:
  1. Excellence in research and leadership. On this point, Gruss said that is critical that one hires top people because top people hire people who are better than themselves. To get them, one has to do head hunting plus advertising. One should strike a good balance between tenured and non-tenured people to maintain flexibility.
  2. Small research settings.
  3. Small group size, but large context.
  4. Multidisciplinary contacts. One has to install interfaces between different disciplines. (Examples: Have only one coffee room.)
  5. Independence as early as possible. Give young people stability for a certain period of time to allow them to unfold their creativity. Coaching and mentoring of young researchers must be provided.
  6. Core institutional and flexible funds. There should be a balance between high-trust and low-trust funding. When handing out high trust funding, an agency must trust the funded institution: Give them the money that you can afford and let them do what they want with it. Trust them to make the most of the received funding. 
He also mentioned that a study in the US pointed out that 75% of citations in patents are to papers funded by public money. Of this 75% of papers most of them belong to the top10% of the papers cited within the scientific community. Hence one should invest in research, in fact, in top research. Hire the best without compromise. Put a lot of weight on top scientists.

Rolf-Dieter Heuer mentioned the importance of "taking society with you." One has to promote science in society.

For research, one needs to continuously develop a vision, which will drive innovation and technology, partnership with industry and feed back to research. Every excellent institution must keep this virtuous circle.  One must think strategically and long term.

All staff needs to have intellectual challenges, including administrative staff. Excellence can be in individuals, but also in cooperation. Excellence must allow for failure, for some research that might fail. This is doing science at the edge. What one can guarantee is that the path will be fruitful.

Olaf Kuebler stated that the strategy to create a leading research institution is deceptively simple: "Search, appoint and retain world-leading scientists. All else will follow."

The reputation of a university is made by the people who leave the university students, graduate students, assistant professors etc.

He also stated that an excellent research institution must:
  1. Make significant contributions to themes of global importance.
  2. Identify and develop new themes of global importance. 
  3. Harmonize its portfolio with its funders.
Helga Nowotny said that being open towards the future is the key aspect of excellence. Invest in excellent young people, who are competent rebels and understand that scientific knowledge is always preliminary. One has to bear in mind that excellence is always a multi-dimensional concept.
One should provide the best possible working conditions. This involves
  • a space component: space that makes it almost obligatory to run into each other and discuss, as ideas emerge by talking to each other, and 
  • a time component: give time for the unexpected, for the unforeseen, for serendipity.

Jonathan Dorfan mentioned that one should establish a setting that is conducive for inter-disciplinary research, where researchers from different fields can cooperate and exchange ideas.

Haim Harari closed the meeting with an articulate and thought-provoking short address. He started by pointing out what he considers to be key ingredients for an excellent research institution. 

Funding must be versatile and come from many sources. Only if one is versatile one can have the right mix. Government funding leads inevitably to egalitarianism  and democracy. However, science is not democratic. Still there has to be a balance between the power of the president and the faculty.

A research institution should be as international as possible and as national as possible. It should give something back to the taxpayers: education and touching society. Technology transfer is the other thing one return to society.

The Weizmann Institute put all the different subjects in the same campus, which leads to inter-disciplinary research that cannot be done by any single subject alone. 

Harari also said that the excellence of a research institution should be evaluated according to  three different measures:
  1. its best ten people,
  2. the average quality of its professors and
  3. its worst professor.
The quality of the worst professor says what the threshold of the institution is for hiring and is a very important indicator of the standards of the research institute/university.

In Harari's opinion, the president of a research institute/university should regularly ask herself/himself: If I could fire some of my professors, how many would I fire?" If the number is a non-trivial fraction of the faculty, then the threshold of the institution is not high enough.

So, in your opinion, what makes a research institution excellent?

Friday, October 21, 2016

October issue of the Bulletin of the EATCS

The October 2016 issue of the EATCS Bulletin is now available online at, featuring the following interesting columns
If you prefer the whole issue, you can download a pdf with the printed version of the bulletin from

As usual, thanks to the support of the EATCS members, the EATCS Bulletin is published in open access form. Consider joining the association!

Friday, October 07, 2016

Ágnes Cseh receives one of the 2016 Klaus Tschira Awards for Achievements in Public Understanding of Science

It is fair to say that not many computer scientists try to present innovative research findings in a way that is accessible to an interested, but rather unspecialized, public. Even fewer succeed and the rewards for those who do are relatively minor. As a consequence, the number of essays and books about computer science that have a wide readership is substantially smaller than those about astronomy and physics, say. In my humble opinion, this is a pity, since many of intellectual achievements of computer science research deserve to be known by any intellectually curious layperson.

I was therefore happy to learn about the Klaus Tschira Award for Achievements in Public Understanding of Science. Since 2006, the Klaus Tschira Stiftung has looked for young scientists who can write a generally understandable article (8,000 to 9,000 words) in German about their research and the content of their PhD thesis. The prize is awarded in each of biology, chemistry, information technology, mathematics, neurosciences and physics as well as in closely related fields. The contributions are judged by a panel of experts on science and communication, which selects the winners based on scientific quality and on how well the scientific contribution is presented in a way that is amenable to public understanding. Yearly, up to six winners receive the award, which is endowed with prize money of 5,000 Euros. The prize-winning contributions are published in a supplementary issue of the popular science magazine bild der wissenschaft (German). Moreover, all competitors are off ered a participation in a two-day workshop for science communication.

The piece by Ágnes Cseh (a former postdoc of Magnús M. Halldórsson's at ICE-TCS, Reykjavík University) you can find here is the English translation of the German original that was selected as one of the prize-winning contributions for 2016. (It will appear in the October issue of the Bulletin of the EATCS.) It is based on Ágnes’ PhD thesis Complexity and algorithms in matching problems under preferences, which she defended in 2015 under the supervision of Martin Skutella at TU Berlin. I am sure that you will enjoy reading it as much as I did, regardless of whether you believe that algorithms can help us find stable marriages in real life.

Thursday, October 06, 2016

The calls for nominations for most of the EATCS-related awards are out

As usual, the EATCS has issued its calls for nominations for the 2017 edition of most of its awards with deadline 31 December 2016. You can find the calls on the EATCS web page, but I collect them below for ease of reference.
The calls for the Alonzo Church Prize and the Gödel Prize, which the EATCS  awards jointly with other associations, will follow.

Let's make the job of the award committees difficult by nominating some of the many colleagues who would richly deserve the awards for these accolades!

Friday, September 30, 2016

CS@Aalborg University: Research evaluation 2011-2015

Every five years, the Department of Computer Science at Aalborg University undergoes a research evaluation. The purpose of this exercise is to provide the department with qualified and independent opinions on its "actual research topics, results, and performance, but also on strategic issues like funding, internal organization and synergies, possible new directions, collaboration with industry, internationalization, positioning IT as a key enabler in society, etc." So the overall aim is to improve the quality and impact of the research carried out within the department.

The evaluation committee for the period 2011-2015 consisted of Peter Apers (University of Twente, the Netherlands), Jan Gulliksen (KTH Royal Institute of Technology, Stockholm, Sweden), Chris Hankin (Institute for Security Science and Technology and Imperial College, UK), Heikki Mannila (Aalto University, President of the Academy of Finland, Finland) and Torben Bach Pedersen (Aalborg University, Denmark), who was the internal member and chair of the committee.

The report resulting from the latest such evaluation has recently been released and can be found here. The editors of the report were Manfred Jaeger, Jesper Kjeldskov, Hua Lu and Brian Nielsen. As a former editor of such a report in days long gone, I know that their job required a considerable use of time and effort.

So, what did the evaluation committee have to say? Quoting from its evaluation of the department as a whole,
"The Computer Science Department has two world-class groups and excellent staff in all groups. The Danish IT benchmarking exercise of 2014 shows that the Department is the best department in Denmark for number of refereed publications and BFI points per full-time faculty member. The Department is also top in a number of other metrics. During the review it was also reported that Aalborg Computer Science graduates are highly prized by industry. The Department thus deserves to be ranked even higher in the QS World University Rankings by Subject or the Academic Ranking of World University (ARWU \Shanghai") Subject ranking. The current rankings are to a large degree caused by the poor coverage of computer science publications in the commercial bibliometric indices used in these rankings (WoS, Scopus). Here, Google Scholar provides a much better coverage. However, the Department clearly has the potential to rise considerably in these rankings but will require support from the Faculty and University to achieve this."
The two world-class groups mentioned in the above quotation are the Database and Programming Technologies and the Distributed and Embedded Systems units. (The latter is now called Distributed, Embedded and Intelligent Systems unit as it now also includes researcers from what used to be the Machine Intelligence group.) Those two groups are led by the Danish computer scientists with the highest h-index, and have a truly impressive publication and grant-winning record.

You can find the committee's evaluations for each of the research groups in the report. Here I'll limit myself to mentioning an excerpt of what the committee wrote about the Distributed and Embedded Systems unit, where I had the pleasure to work for ten years.

"The Distributed and Embedded Systems group is a world-class group. It is involved in a broad range of activities from semantic foundations through tool development for verification and validation to real-world applications. The group is making excellent contributions across the whole spectrum of activity; this is internationally recognized by prestigious awards such as:
  • The ERC Advanced Grant LASSO
  • The 2013 CAV Award for Uppaal - the first time that this award has been granted to a non-US team
  • The ranking of  "Uppaal in a Nutshell" as the 9th most influential paper in Software Engineering since 1972
  • Best paper awards, medals and other awards to Associate Professors
The h-index of Kim Guldstrand Larsen is outstanding and places him among the top echelon of researchers in this area; his h-index is higher than some Turing Award winners in cognate areas. It is also pleasing to note that some of the Associate Professors also have high h-indices for their career point. .....

The group has published well during the review period with 175 conference papers - 75% of which are in A and B venues - and 63 journals - 92% of which are in A and B venues. .....
The group has secured 37 new grants to a total value of DKK103.8M. ....

The major strength of the group is the people; not only the group leader but the strong group of more junior academic staff and an excellent group of support staff. The broad span from foundational work to applications is also unusual in such groups in other universities and is a considerable strength of DES. The profile and reach of the group is enhanced by its dissemination activities but also the engagement of senior staff in policy-related activities at national and European levels."

Of course there is still a lot of room for improvement, but this will require support from the university as a whole, high-profile new hires in the future and the development of the talent the department already boasts. However, the opinion of the evaluation committee clearly highlights the current strength of a CS department that, in my admittedly biased opinion, deserves to be better known worldwide.