Motivation

• The main question is: "Is your self-evaluation about your own publications is consistent with the bibliometric statistics of your publications?"

• The publications are mostly evaluated by outsiders. Referees and editor accept your articles to prestigious journals and other researchers cite your publications if they find it worthwhile. 
  
• These outside opinions form bibliometric statistics that are used to evaluate your research
• You are rarely asked for your opinion about your own research. In fact, you spend more time on your research than others so you may have more information about your research. 
• Social scientists are particularly disadvantaged in bibliometric statistics. Books, non-English publications and social science journals are not covered completely by the indices. Therefore, social scientists get less citations. Our research shows how important the social science publications that are not covered by the indices. 

Survey Questions

Only two questions are asked to Social Scientists. Question A asks about top 3 publications and Question B asks about full publication lists. 

Question A: Could you please name your top three publications (published in 2011-2020) according to your self-assessment, in descending order (i.e. first publication is your best)? It is sufficient if you can give a short title and publication year of your publications.

Question B: Could you please give me the source where I can find your rather complete list of publications (published in 2011-2020)?  Possible sources that come to my mind are your home page in your institution, your personal home page (e.g. google sites), and publication indices (e.g. google scholar). If the list is not complete in any of the sources that I can access, could you please attach the list in response to this e-mail? 

Sample

• We e-mailed 32,573 academics who work in the economics, political science, psychology, and sociology departments of the top 500 universities according to Shanghai Rankings.
• In the first round, we manually e-mailed one e-mail at a time. In the second round we sent 50 e-mails at once -> Time consuming procedure but we aimed to decrease the chances that the e-mails fall into spam folders. We got 1641 answers in first round & 690 answers in second round

• Overall slightly above 7% response rate – Does not change by the academic field or ranking of the university but particularly low in East Asia (2%) and high in South America (14%)
• The response rate was below our expectations but we got answers from respondents from diverse backgrounds. The 2331 answers came from 42 countries: 

Question B: Where to get full publication list?

• Question B asks about the full publication lists of the respondents. This is a framed question so the answers may be biased.
• We use the answers to Question B to construct the Google Scholar profiles if they are not publicly available. The answers also let us refrain from name confusions and to understand the sources that one can find the full publication list of social scientists. 
• The total answers below in the figure do not add up to 100% because some respondents gave more than one answer
• 1270 respondents (54%) gave the name of a publication index. Google Scholar is the most prevalent answer but framing of the question may have caused this prevalence. 
  
  

Question A: Time-Frame (Respondents did not like this!)

• Time-frame is clearly stated on the subject line and in texts of both Questions A & B as 2011-2020. 
• However, the respondents did not like this and many has ignored the time-frame while giving their answers. 509 (22%) respondents a publication after 2020 and 146 (6%) chose a publication before 2011 as their top publications. 
• The aim of the time-frame was to get answers in an era that most researchers are active.

Question A: Document Type

• The books that are chosen by respondents as one of their top 3 publications are largely not covered by Scopus.(See table below)
• For each social science field: the percentage of book publications is approximately the same as the percentage of publications that are not indexed in Scopus. (See table below)
• Most respondents who gave a top 3 publication that is not covered by Scopus has more than 3 Scopus indexed publications. In fact, half of these respondents have more than 10 Scopus indexed publications. Therefore, the lack of Scopus indexed publications is not the reason for the lack of top 3 publications that are indexed in Scopus. 
• There are book publications in Scopus profiles, however, most of the book publications that are selected by respondents are not indexed in Scopus. 
• It is more likely that the top choice is a book publication and a publication that is not indexed in Scopus than second and third choices. 

Question A: Document Language

• Around 4% of the publications in the respondents’ Scopus profiles are non-English. However, the non-English publications that are chosen by respondents as one of their top 3 publications are largely not covered by Scopus.
• Many respondents commented that the question asks for English publication, although there is nothing indicating as such. This may have lowered the percentage of non-English publications in Top 3. 

Factors affecting top 3 choices (linear regression with respondent fixed effect)

Top 3 in self-evaluations but not top 3 in citation performance: Near miss?

• Only 40 percent of the top 3 choices has top 3 citation performance in respondent's profiles.
• But is this a near miss?
• Even when we triple(!) the citation performance of top 3 choices, only 70 percent of the top 3 publications become top 3 in citation performance. 

The Journals that host top 3 publications

• Although a significant amount of publications are not journal publications, we can use top 3 choices to see which journals are more likely to publish top 3 publications.

• There are few publications in some journals so Revealed Journal Score (RJS) may not give reliable answers for them. Therefore, we rank 369 journals that include publications from at least 20 respondents within the time-frame.

• Suppose that 1/3 of all publications of respondent i is in journal j. Then, if she chooses 1 of her top 3 publications from journal j, then the journal has the expected success. In other words, the ratio_i,j would be equal to 1 if the journal j has the expected success. That is, ratio_i,j is equal to the ratio of publications from journal j in respondent i's top 3 publications over the ratio of journal j publications in the respondent i's profile. ratio_i,j is restricted to be below 10 in order to limit one respondent's contribution to the index.   

• RJS_j is the average of ratio_i,j of all respondents who have journal j publications in their profiles. 

Conclusion

• The social scientists have journal articles indexed in Scopus, and Scopus has books and non-English documents in its index. These facts may fool university administrators that they can use a journal-based index to evaluate social scientists.
• 17 % of the top 3 publications are not indexed in Scopus.
• Self-evaluations are positively correlated with citation statistics.

• However, only 40 percent of the publications that are top 3 in respondents’ self-evaluations are at the same time top 3 in citation performance. Even when the citation performance of top 3 publications in self-evaluations are tripled, 30 percent of them still cannot become top 3 in citation performance. 
• The most favored papers are rejected from journals for some reason that authors do not agree. Revealing top publications would give these undervalued publications their true value.
   
• The undervaluation of social sciences is a big problem. The undervaluation of the social science publication output may cause universities to downsize the social science departments. 
• The administrators who evaluate social scientists with the wrong criteria may alienate social scientists from their publication output.