Over the last few weeks, I’ve wrestled with tracking down data (primarily qPCR data) from the litany of projects the Friedman Lab has had over the last decade or so. I’ve also noticed that it’s increasingly difficult for me to track down my own data from individual projects where data is not generated continuously over time, but in chunks. Tracking down 6 different qPCR runs that were conducted over the course of a year is tedious.
In order to save myself, and others who might need/want to review my data, a lot of time in the future, I decided to write a script that will allow me to compile all of my qPCR data into a single, massive CSV file. Accessing this CSV file via spreadsheet (Excel/Calc/Sheets) or database (SQL) means I’ll always be able to quickly search for all the related data I need, since it will reside in a single file!
The script is written in bash, called qpcr_aggregation.sh, and is currently hosted here: https://github.com/kubu4/Scripts/blob/master/bash/qpcr_aggregation.sh
Basically, the script does the following:
- Replace the spaces in the BioRad filenames with underscores (used to simplify parsing of the filename for use in downstream steps in the script)
- Replace the header row to accommodate two new fields: qPCR_filename and qPCR_date
- Add qPCR_filename and qPCR_date to each file.
- Concatenate all the files into a single “master” CSV file.
Now, the easy part – exporting the data from hundreds of qPCR files…