Tag Archives: script

Data Management – Script for Compiling qPCR Data

Over the last few weeks, I’ve wrestled with tracking down data (primarily qPCR data) from the litany of projects the Friedman Lab has had over the last decade or so. I’ve also noticed that it’s increasingly difficult for me to track down my own data from individual projects where data is not generated continuously over time, but in chunks. Tracking down 6 different qPCR runs that were conducted over the course of a year is tedious.

In order to save myself, and others who might need/want to review my data, a lot of time in the future, I decided to write a script that will allow me to compile all of my qPCR data into a single, massive CSV file. Accessing this CSV file via spreadsheet (Excel/Calc/Sheets) or database (SQL) means I’ll always be able to quickly search for all the related data I need, since it will reside in a single file!

The script is written in bash, called qpcr_aggregation.sh, and is currently hosted here: https://github.com/kubu4/Scripts/blob/master/bash/qpcr_aggregation.sh

Basically, the script does the following:

  • Replace the spaces in the BioRad filenames with underscores (used to simplify parsing of the filename for use in downstream steps in the script)
  • Replace the header row to accommodate two new fields: qPCR_filename and qPCR_date
  • Add qPCR_filename and qPCR_date to each file.
  • Concatenate all the files into a single “master” CSV file.

Now, the easy part – exporting the data from hundreds of qPCR files…


Automatic Notebook Backups – wget Script & Synology Task Scheduler


I’ve been tweaking a shell script (notebook_backups.sh) to use the shell program wget to retrieve fully functional HTML versions of our online notebooks for offline viewing. I had been planning on setting up a cron job to automatically run this script on our Synology server (Eagle) at a set day/time. However, I came across the Task Scheduler that’s built right into the Synology GUI! So, I set up the Task Scheduler to run the notebook_backups.sh script every Sunday. See screenshots below.




UPDATE 201507114

The Task Scheduler was not running the script. Additionally, the Task Scheduler would not run the script even when I manually instructed the Task Scheduler to run. Some internet searching revealed that the Task Scheduler requires you to indicate what type of task is being run (e.g. bash, shell, ash, php, etc.), even if your script contains the proper “shebang” or header that normally instructs the computer which program to use to run the script. See the image below for how the Task Scheduler is currently set up. The arrow indicates that addition of “sh” to the beginning of the Task Scheduler’s path to the script. This tells the Task Scheduler to use the Shell to run the script.