The first step in any data science project is cleaning the data. Web of Science records are about as good as bibliographic data gets, but good is not yet perfect. In particular, WoS records a problem with ambiguous researcher names. We know that 'John Smith' and 'John L Smith' and 'John L. Smith' are probably all the same person, but computers see those as different strings.
In general, disambiguation is challenge in any data science project where entities are identified by non-unique records, like ordinary names, or errors are introduced into the data stream through a manual entry point. I looked at some Python libraries that handle disambiguation, but they didn't seem precisely suited for this project. I absolutely need to avoid improperly combining records for different scientists, the data is small enough that I can clean it by hand (with a little computer assistance), and the ambiguities have regular patterns, rather than being the result of a random process that swaps characters of similar.
The most common problems are:
And now there are files in my github account! https://github.com/mburnamfink/scientometrics_101_project
In general, disambiguation is challenge in any data science project where entities are identified by non-unique records, like ordinary names, or errors are introduced into the data stream through a manual entry point. I looked at some Python libraries that handle disambiguation, but they didn't seem precisely suited for this project. I absolutely need to avoid improperly combining records for different scientists, the data is small enough that I can clean it by hand (with a little computer assistance), and the ambiguities have regular patterns, rather than being the result of a random process that swaps characters of similar.
The most common problems are:
- A scientist is inconsistent in use of a middle initial
- A scientist is referred to by a nickname
- A scientist's first name is misspelled in one instance
And now there are files in my github account! https://github.com/mburnamfink/scientometrics_101_project
RSS Feed
