Manual text and data mining not appreciated by publisher

I will introduce myself first as I failed to do so in my first blog: I am an organizational researcher, mostly interested in inter-organizational relationships, networks, and organizational learning. As a hobby, I like to invest some time in meta-science as well. This time I will focus on the hobby.

I recently started working on a project where I, together with Chris Hartgerink, am checking how many statistical errors are made in articles from the organizational sciences. To this end, we intended to download empirical articles from a range of journals and publishers so that we could extract statistical test results . Given that Chris has some experience with Text- and Data Mining (TDM) on a large scale (and especially with publishers), I started investigating the terms and conditions of some publishers in my own field. To be clear: the downloading was for research purposes only and the articles would not be redistributed.

When reading the terms and conditions of the INFORMS website, a major publisher in management science, I saw that we could not “systematically download (robots or other automated processes)” articles. Therefore, I elected to manually download the articles from one of their journals. I downloaded a total of 429 articles in two days (first day I downloaded 96 articles in 21 minutes, second day I downloaded 333 in 5 hours).

To my surprise, my university was notified by INFORMS after the downloads of the first day that my IP-address was blocked from accessing all content of the publisher (this includes other journals that I did not download from). Our librarian team notified me a few days later of the notice and instructed me to stop downloading (which I did). They told me that downloading for TDM required an additional agreement with the publisher that would require different tariffs. Note that the original email from INFORMS stated “100 PDFs have been downloaded within a single session from this IP address, which is excessive and a violation of permissible activity.”

In conclusion, I thought I followed the publisher’s terms and conditions by manually downloading content from their website. Apparently, this was not the case. I do not understand why I can read the papers one-by-one myself, but am not allowed to save these for computer reading.