I’d like to sum up the past two weeks of Data Science in Public posts to discuss what I did, and what we learned.  I’m interested in interdisciplinary science, because so much of science is done by interdisciplinary teams. Very few of the problems that we care about: fighting cancer, clean energy systems, ecological resilience and human well-being, have relevant discoveries that can be made by the idealized model of a lone specialist who is very knowledgeable at one type of thing. The National Science Foundation recognizes, and has been aiding in the transformation of science, in part with the IGERT/NRT grant series.  These awards are for about $3,000,000 over five years, with most of the funding going towards graduate student stipends to encourage innovative models in STEM graduate education on high-priority interdisciplinary topics. There have been over 400 of these awards made since 1997, for something like $1.2 billion in total.
​
I think it’s important to know if this program is working.  The 2018 NRT funding solicitation states: “The program is dedicated to effective training of STEM graduate students in high priority interdisciplinary research areas, through the use of a comprehensive traineeship model that is innovative, evidence-based, and aligned with changing workforce and research needs.”  In an ideal world, we’d be able to embed trained ethnographers in every one of these grants, to track the development of students into scientists.  That’s not feasible, and we can’t go back to the earliest grants without a time machine, but one of the nice things about scientists is that they publish what they discover, those publications are indexed by databases like the Thomson Reuters - ISI Web of Knowledge, and we can use the richly structured information in publications to say something about the grants that produced them.

In one sentence, can we use data science to show that the IGERT/NRT solicitation produced more interdisciplinary scholarship?

You can see the source of the data and how it was transformed at each step in the flowchart below. We go from all the IGERTs, to a sample of five, to a list of 202 scientists, a corpus of over 3400 publications, and then a dataframe of feature vectors amenable to data science techniques. An anonymized version of the final dataset (5-scientometrics101-cites.csv) along with some code is available on my GitHub.

I describe how to evaluate interdisciplinarity in DSiP 2, but I have a better metaphor I'd like to share.  Image all of scientific knowledge like the night sky, each star a different fact known to science.  New scientific knowledge is like adding a new star to the heavens. Scientists don’t just throw their ideas up in the air: they construct truth by tying each new claim into what is already known.  Scientific writing is a lot like finding constellations. Constellations aren’t “real” features of the universe, the individual stars may be separated by thousands of lightyears, but they are real features of human experience. The connections that we draw between facts are just as important to understanding as the facts themselves.   

Building on that analogy, we’d expect different kinds of scientists, say a chemist and a botanist, to make different kinds of “constellations” when they’re writing their papers.  And we’d expect a monodisciplinary paper to look different from an interdisciplinary paper.  This concept is operationalized in the Rao-Stirling Diversity Index (SDI), which measures how the citations of a paper are distributed across Web of Science subject categories, and varies between 0 for work entirely within one discipline, and approaches 1 as work becomes increasingly interdisciplinary.
​
The tedious part is going from a list of grants, to a roster of scientists, to a corpus of publications.  Then comes the fun part.  Using the Metaknowledge package by Reid McIlroy-Young, John McLevey, and Jillian Anderson, which is an absolutely fantastic tool for bibliographic work, a directory mapping journal abbreviations to Web of Science subject categories provided by Thompson-Reuters customer service, and a similarity matrix between all Web of Science subject categories by Ismael Rafols, Alan Porter, and Loet Leydesdorff, we’re able to process Web of Science records into a table which looks like this.

​The first thing to check is how the Sterling Diversity Index changes over time.  A boxplot shows that for each of the five groups, the Sterling Diversity Index increased.

A more detailed look at the SDI, with mean and interquartile range plotted, shows the patterns for each group year by year, with the start of each IGERT grant marked by a vertical line. The steep increase of Group B five years after the start of the grant, and the way that SDI stopped declining and actually increased when Group D got their grant are particularly interesting.

I ran t-tests on the change in each of the groups, and the results are clear. The increase is statistically significant, and averages 0.38 standard deviations. If we imagine that SDI is analogous to IQ, where each standard deviation is 15 points, the increase in scores is equivalent to a gain of 5.7 points. That’s pretty good, given how hard it is to make major structural changes in science.

t-test    : p-value  : increase in st.dev
Group A: 0.00833 : +0.33257
Group B: 0.00000 : +0.72993
Group C: 0.00879 : +0.15155
Group D: 0.00250 : +0.31565
Group E: 0.00030 : +0.38804

The IGERT/NRT program has meet its primary stated policy objective. Going forward, I think it'd be interesting to look at the program as a whole, as opposed to just a sample. And we need to look in further detail at the groups that met these goals exceptionally well in order to understand what they did, and how those practices can be translated to other research groups.
​

Everything above is descriptive statistics and data visualization, which are important, but something that I've done before. What's new to me is machine learning.  Machine learning differs from traditional statistics in that while in theory I could have calculated everything above by hand with a lot of tedium, machine learning is about computer programs that optimize the parameters of an algorithm according to a performance measure. There's not a firm line between machine learning and statistics, and indeed most machine learning curriculum start with linear regression, drawing best fit lines through points.

I had some trouble coming up with a good machine learning task for this dataset.  The major types of machine learning operations are regression, predicting a continuous value from a set of inputs; classification, which involves assigning an input to one or more categories; and clustering, finding patterns in data that does not have a prior structure. I toyed with coming up a regression model to predict which scientific papers would have the greatest impact, but discarded it. Important ideas are important because of their content, not because of their citations.

A flaw in my dataset presented the solution. For some reason, 309 of the papers in my sample aren’t associated with any research group.  All five of my IGERTs work in different areas, so maybe I can use patterns of citations to match my misfiled papers with the right research group.  I ran three common classification algorithms, Logistic Regression, a Support Vector Machine, and a Decision Tree, and found that I could associate 70% of papers in the test split with the correct group.  The Support Vector Machine had slightly higher accuracy, but performance was similar across the board.  Looking at the confusion matrices for each algorithm, we can see that most of the errors involve sorting papers in Groups B and E with Group C.  This makes sense, since Group C is the most prolific group, and all three groups work on nanotechnology.  There is overlap in papers about basic topics like the properties of carbon nanotubes.

Looking at the workings of the decision tree can be useful, because it mimics a common human way to sort things.  Pick an attribute where there is a clear difference between Category A and the rest of the data, split, and repeat until each little clump is all of one kind.  To read the decision tree, start at the top, check the condition, and if True go left, and False go right.  Geography shows up immediately as a way to distinguish Group D, medical sciences as a way to separate Group E, and material science tends to lead to Group C.

(click for big)

Knowing that, I can assign the missing papers to their most likely groups with 70% accuracy.  I wrote up a script to do that, and then go back to the articles check which university the authors are at. In the majority of cases, at least one of the authors is at the university that my classifiers assigned the paper to.

Thank you to my instructor, Nathan Grossman, who asked questions that lead a significantly more interesting project, my cohort in the Metis Live Online class, for listening to me explain versions of this work four times, and Sebastian Raschka, for his great book on Python Machine Learning.

Last time, we proved that there was an increase in the Sterling Diversity Index for papers published by a group after an IGERT was awarded. But that is basic statistics, not data science. It turns out that along with the 3139 papers that were published by the five IGERT groups in my sample, I have 307 papers which aren't affiliated with any group. I'm not sure how that happened, but it provides a useful finale to the project. Can I fairly assign each of the 307 error papers to one of my five IGERT groups?
​
Each of the five IGERTs works on its own research topic, so to do this, we need to find a data science method to figure out what the topics of the five groups are, and assign new cases to each group with some degree of certainty. This sounds a lot like good case for logistic regression to me. To do this, I've expanded the data with the citation categories for each paper, which I used to calculate the Stirling Diversity Index. It that looks like this, extended out through the 224 Web of Science subject categories.

We import LogisticRegression and some the usual data wrangling packages, then set up our X to be all the citation categories of papers not in the ERROR group, and y to be the group that they are actually in.  Actually going it is about a dozen lines of code to select the right parts of the data frame, scale the counts, split the data into train and test sets, and then run the logistic regression.

Our accuracy is 0.72, which is acceptably above chance, 0.2 in this case. I tried changing the hyperparameters, and using a support vector machine, and couldn't get higher accuracy.

Let's inspect the confusion matrix, to see where we're guessing wrong. It's clear that in terms of raw numbers, papers from Group C dominate, and we have trouble distinguishing papers in Group B and Group E from those in group C. This makes sense, since all I know from my prior investifation those three groups work in nanotechnology, among other things. It is plausible that there is research on something fundamental, like the properties of carbon nanotubes, which all three groups contribute to.

[ 55 7 16 8 3]
[ 7 99 58 1 12]
[ 4 16 359 4 21]
[ 6 4 20 92 6]
[ 3 8 54 0 79]

And finally, we grab the ERROR papers and run them through the linear regression to see that most of them belong to group C.  

A 8
B 0
C 280
D 13
E 6

This is as good a place as any to put the end of the hard work of the data science in public series.  I'll have one more post wrapping it all up.

From last time, we ran into some problems with the scientists data, which meant it's time to reappraise. Some discussion with my cohort and instructor suggested a few things. First, the scientists data is hard to interprate. Most of the data comes from Americans who are part of an IGERT grant, i.e. a Federal effort to create more interdisciplinary scholars by new funding sources. However about a quarter of the data are Germans who are members of interdisciplinary research centers that formed organically, and then were recognized by the German government. Second, in general scientific careers switch directions very slowly, and it's hard to do much more than categorize them. Third, the interesting science policy question can be summed up as "Did the IGERTs work?" Did these grants foster more interdisciplianary scholarship?
Since I really want to avoid scraping and cleaning more data, I decided to go back to the original data and evaluate them by papers. Each paper is described by several statistics.

• first_author (bool): is the first author a member of an IGERT grant
• group (categorical): Five IGERTs, A, B, C, D, E
• group_start (int): the year the group was awarded the grant
• number_authors (int) : how many authors on the paper
• sdi (float) : the Stirling Diversity Index for the paper
• times cited (int) : how many times the article has been cited
• journal (string) : the journal that published the paper
• pub_year (int) : date of publication
• velocity (float) : number of citations / years since publication. A measure of impact
• grant_product (categorical) : if the paper was published before or after the grant was awarded

​We import the data, and run our initial descriptive stats tests.

And finally, a little light reading for the week.  As always, data and code is available at https://github.com/mburnamfink/scientometrics_101_project​

This project is about distinguishing interdisciplinary scholars from monodisciplinary scholars, which means that we need a measure of how interdisciplinary scholars are. The National Academis of Science, in their 2015 report Enhancing the Effectiveness of Team Science define interdisciplinarity as follows:
“Interdisciplinary research integrates the data, tools, perspectives, and theories of two or more disciplines to advance understanding or solve problems. Transdisciplinary research aims to deeply integrate and also transcend disciplinary approaches to generate fundamentally new conceptual frameworks, theories, models, and applications.”

In an ideal universe, we'd have highly trained experts who are able to closely examine the papers written by the person or group under consideration, interview the people involved, and even ethnographically follow their work. Let's say that there were really fast, and could do this at a rate of one hour per person (absurdly quick, if you know anything about qualitative social science). My little sample of 40 people would take all week to analyze.

Let's use a data science approach. In particular, pay attention to the ideas of integration and diversity. From reading Latour's Science in Action, we see that scientific claims are constructed and made firm by tying a new claim to an existing web of established facts, which are recorded in the peer-reviewed scientific literature. Or in plain language, scientific papers have citations, and it's reasonable to expect that an interdisciplinary paper has a different pattern of citations than a monodisciplinary paper.

Over the past decade, this idea has been transformed into metric via the Rao-Stirling diversity index, or SDI, which varies between 0 for monodisciplinary and 1 for maximally interdisciplinary. The form is similar to the Gini index. In a 2007 paper, "A general framework for analyzing diversity in science, technology and society", Andy Stirling translated ideas from ecology to note that mathematically, diversity is how a set of objects is apportioned over a set of categories. In particular, diversity is represented by the sum of proportions between all categories i and j multiplied by the distance between i and j.

For scientific papers, we can use citations as our objects, and library categories as our categories. We then need to figure out the distance measure. Rafols, Porter, and Leydesdorff in 2010 ("Science Overlay Maps: A New Tool for Research Policy and Library Management") looked at the patterns of citations across every paper indexed in Web of Science in 2007 to see how often papers in Web of Science subject category A cited papers in Web of Science subject category B, and normalized that matrix to find the cosine similarity between all pairs of categories. 1 - the cosine similarity gives us a distance. Rafols et al point out that the large scale epistemic structure of science is fairly stable, so this matrix should still give us a good idea of the diversity of categories.

It gets a little more complicated, because while the Rao-Stirling diversity index makes sense for a single object in one category that cites many things in many categories, we're dealing with corpi of all the papers written by a scientist. Imagine that we have a person who's written an interdisciplinary paper mostly in chemistry (SDI=0.5). If they write another paper that with the same pattern of citations, their Rao-Stirling score shouldn't change. Yet if we have a physicist who written a monodisciplinary paper in biology (SDI=0.1) and she suddenly writes a paper that cites a bunch of physics paper (SDI = 0.1), her Rao-Stirling score should increase. If we simply average out the SDIs, that isn't the case.

Cassi, Mescheba, and Turckheim point out in 2014's "How to evaluate the degree of interdisciplinarity of an institution" that Rao-Stirling diversity is mathematically isomorphic to the equations used to calculate the rotational moment of inertia of a cluster of points. If we're calculating the Rao-Stirling diversity index for many objects, we can find the moment of intertia around the common "center of gravity" for all the citations. Cassi and Turckheim expanded their work in 2017 "Analysing Institutions Interdisciplinarity by Extensive Use of Rao-Stirling Diversity Index", and have some R code available.

My github has a notebook, 2-Measuring Interdisciplinarity with Rao-Stirling Diversity Indexes, which has the code that demonstrates how this all works.  Meanwhile one outcome is this visualization of 2018 Nobel Laureate Kip Thorne's SDI's per year,

We can see a few things in this plot.  First, Kip has a Nobel Prize, but  he is not really an interdisciplinary scholar.  Porter et al (2007) in "Measuring researcher interdisciplinarity" suggest an SDI of 0.45 is a reasonable threshold for interdisciplinarity, a value which has been backed up by my studies as well.  We can see a moderate trend towards more interdisciplinary publications after 1990. And the dense cluster of papers at the end are mostly about LIGO, which came online in 2002, was upgraded in 2010, and detected gravity waves for the first time in 2015.  LIGO is a classic Big Science project, with large teams organized around expensive equipment.  As a principal investigator on LIGO, Kip contributed to many papers, which outweighs even an impressive career as a theoretical physicist. 

The first step in any data science project is cleaning the data. Web of Science records are about as good as bibliographic data gets, but good is not yet perfect. In particular, WoS records a problem with ambiguous researcher names. We know that 'John Smith' and 'John L Smith' and 'John L. Smith' are probably all the same person, but computers see those as different strings.

In general, disambiguation is challenge in any data science project where entities are identified by non-unique records, like ordinary names, or errors are introduced into the data stream through a manual entry point. I looked at some Python libraries that handle disambiguation, but they didn't seem precisely suited for this project. I absolutely need to avoid improperly combining records for different scientists, the data is small enough that I can clean it by hand (with a little computer assistance), and the ambiguities have regular patterns, rather than being the result of a random process that swaps characters of similar.

The most common problems are:

• A scientist is inconsistent in use of a middle initial
• A scientist is referred to by a nickname
• A scientist's first name is misspelled in one instance

I'm reusing some older code, so this is not as elegant as I'd like. But the best code is code that runs.

And now there are files in my github account! https://github.com/mburnamfink/scientometrics_101_project

For the past few weeks I've been taking the Intro to Data Science live online course through Metis, as a prelude to starting their data science bootcamp in the fall. Part of the course involves doing a final project on a topic of your choice.

I have a strong background from previous work on my evaluating interdisciplinary groups through scientometrics and network analytics, and it seems reasonable to extend that work.  I've already rewritten the code from the previous post to make it more maintainable, but what is a reasonable next step?

I considered a few things.  Redoing the visualization step to use Bokeh instead of Kumu.io would give me finer control over the visualizations, and possibly extend past some technical limits I'm encountering on Kumu. These coauthorship networks are similar to scale-free networks created using a preferential attachment model, but constructed in a different way.  Specifically, each new paper in a corpus is a clique of size K, where K is drawn from a distribution that is the sum of a Poisson distribution and an exponential distribution, and each author A has some probability of being an existing member of the network P.  And finally, I'd like to develop some better analytics for describing these coauthorship networks, because the basic network measures like density and clustering coefficient don't seem to match any useful quality of these groups.

These are all worthy projects, but I'm not going to do any of them, because of a mismatch between what they demand, and the skills I have as a beginner data scientist.  Data science, at least the 101 level, has a specific ontology. The world is made out of observations, which are described by a feature vector.  The feature vector can be manipulated by your choice of algorithms to produce, in supervised learning, classifications and regressions which allow new observations to be predicted, or, in unsupervised learning, to find patterns in the data as a whole.  And as I look at the social network data, I can't see a good way to generate feature vectors.  There are techniques to evaluate social networks in data science, but I'm moving in a week, and I don't have time to pack, unpack, and learn a bunch of new math in two weeks.

So we go with Plan B.  I have complete bibliographic records for a few hundred interdisciplinary scholars, as identified by their participation in one of my interdisciplinary research groups.  I'm going to see if I can distinguish interdisciplinary scholars from (assumed) non-interdisciplinary scholars.  As a first step, I've selected a subsample of 20 random interdisciplinary scholars from my previous data, and paired them with scholars of the same rank (Assistant, Associate, or Full Professor) and department at the same university for a total of 40 professors.  Then I scraped all their publications from Web of Science.

Next up, cleaning the data!

And before I go, this final project has several goals.
1) Use machine learning to develop an understand of scientific publication patterns between interdisciplinary and monodisciplinary scholars.
2) Document my thought process via blog post using the DSiP tag.
3) Make the code publicly available at https://github.com/mburnamfink/scientometrics_101_project as the first step in building a data science portfolio.

Comments welcome, especially on step 3, since this is a new form of publishing for me.