Tag Archives: SRA

Data Management – SRA Submission Oly GBS Batch Submission

An earlier attempt at submitting these files failed.

I re-uploaded the failed files (indicated in my previous notebook entry linked above) and tried again.

 

It failed again, despite having successfully uploaded just minutes before.

I re-uploaded that “missing” file and tried again.

This time, it succeeded (and no end-of-stream error for the 1SN_1A file!)!

Will post here with the SRA accession number once it goes live!

 

Share

Data Management – SRA Submission Oly GBS Batch Submission Fail

I had previously submitted the two non-demultiplexed genotype-by-sequencing (GBS) files provided by BGI to the NCBI short read archive (SRA).

Recently, Jay responded to an issue I had posted on the GitHub repo for the manuscript we’re writing about this data.

He noticed that the SRA no longer wants “raw data dumps” (i.e. the type of submission I made before). So, that means I had to prepare the demultiplexed files provided by BGI to actually submit to the SRA.

Last week, I uploaded all 192 of the files via FTP. It took over 10hrs.

(FTP tips: – Use ftp -i to initiate FTP. – Then use open ftp.address.IP to connect. – Then can use mput with regular expressions to upload multiple files)

Today, I created a batch BioSample submission:

 

 

 

Initiated the submission process (Ummm, this looks like it’s going to take awhile…):

 

 

 

Aaaaaand, it failed:

 

 

It seems like the FTP failed at some point, as there’s nothing about those seven files that would separate them from the remaining 185 files. Additional support for FTP failure is that the 1SN_1A_1.fq.gz error message makes it sound like only part of the file got transferred.

I’ll retrieve those files from our UW Google Drive (since their original home on Owl is still down) and get them trasnferred to the SRA.

Share

Data Management – SRA Submission of Ostrea lurida GBS FASTQ Files

Prepared a short read archive (SRA) submission for archiving our Olympia oyster genotype-by-sequencing (GBS) data in NCBI. This is in preparation for submission of the mansucript we’re putting together.

I followed my outline/guideline for navigating the SRA submission process, as it’s a bit of a pain in the neck. Glad my notes were actually useful!

The following two files are currently being uploaded via FTP; the process will take about 3hrs, as each file is ~18GB in size:

 

They are being submitted under the following accession numbers (note: a final accession number will be provided once this is publicly available; I will update this post when that happens):

Share

SRA Release – Transcriptomic Profiles of Adult Female & Male Gonads in Panopea generosa (Pacific geoduck)

The RNAseq data that I previously submitted to NCBI short read archive (SRA) has been released to the public today. Here are the various links for the project:

Study: SRP072283http://trace.ncbi.nlm.nih.gov/Traces/study/?acc=SRP072283

 

BioProject: PRJNA316216http://www.ncbi.nlm.nih.gov/bioproject/PRJNA316216

Study: SRP072283http://trace.ncbi.nlm.nih.gov/Traces/sra/?study=SRP072283

Female Pool Experiment: SRX1659865http://www.ncbi.nlm.nih.gov/sra/SRX1659865

Male Pool Experiment: SRX1659865http://www.ncbi.nlm.nih.gov/sra/SRX1659866

Share

SRA Submission – Genome sequencing of the Olympia oyster (Ostrea lurida)

Adding our Olympia oyster genome sequencing (sequencing done by BGI) to the NCBI Sequence Read Archive (SRS). The current status can be seen in the screen cap below. Release date is set for a year from now, but will likely bump it up. Need Steven to review the details of the submission (BioProject, Experiment descriptions, etc.) before I initiate the public release. Will update this post with the SRA number once we receive it.

Here’s the list of files uploaded to the SRA:

151114_I191_FCH3Y35BCXX_L1_wHAIPI023992-37_1.fq.gz
151114_I191_FCH3Y35BCXX_L1_wHAIPI023992-37_2.fq.gz
151114_I191_FCH3Y35BCXX_L2_wHAMPI023991-66_1.fq.gz
151114_I191_FCH3Y35BCXX_L2_wHAMPI023991-66_2.fq.gz
151118_I137_FCH3KNJBBXX_L5_wHAXPI023905-96_1.fq.gz
151118_I137_FCH3KNJBBXX_L5_wHAXPI023905-96_2.fq.gz
160103_I137_FCH3V5YBBXX_L3_WHOSTibkDCABDLAAPEI-62_1.fq.gz
160103_I137_FCH3V5YBBXX_L3_WHOSTibkDCABDLAAPEI-62_2.fq.gz
160103_I137_FCH3V5YBBXX_L3_WHOSTibkDCACDTAAPEI-75_1.fq.gz
160103_I137_FCH3V5YBBXX_L3_WHOSTibkDCACDTAAPEI-75_2.fq.gz
160103_I137_FCH3V5YBBXX_L4_WHOSTibkDCABDLAAPEI-62_1.fq.gz
160103_I137_FCH3V5YBBXX_L4_WHOSTibkDCABDLAAPEI-62_2.fq.gz
160103_I137_FCH3V5YBBXX_L4_WHOSTibkDCACDTAAPEI-75_1.fq.gz
160103_I137_FCH3V5YBBXX_L4_WHOSTibkDCACDTAAPEI-75_2.fq.gz
160103_I137_FCH3V5YBBXX_L5_WHOSTibkDCAADWAAPEI-74_1.fq.gz
160103_I137_FCH3V5YBBXX_L5_WHOSTibkDCAADWAAPEI-74_2.fq.gz
160103_I137_FCH3V5YBBXX_L6_WHOSTibkDCAADWAAPEI-74_1.fq.gz
160103_I137_FCH3V5YBBXX_L6_WHOSTibkDCAADWAAPEI-74_2.fq.gz

Mate pair sequencing files were uploaded together within a single “Run”.

Share

SRA Submission – Genome sequencing of the Pacific geoduck (Panopea generosa)

Adding our geoduck genome sequencing (sequencing done by BGI) to the NCBI Sequence Read Archive (SRS). The current status can be seen in the screen cap below. Release date is set for a year from now, but will likely bump it up. Need Steven to review the details of the submission (BioProject, Experiment descriptions, etc.) before I initiate the public release. Will update this post with the SRA number once we receive it.

Here’s the list of files uploaded to the SRA:

151114_I191_FCH3Y35BCXX_L1_wHAIPI023989-79_1.fq.gz
151114_I191_FCH3Y35BCXX_L1_wHAIPI023989-79_2.fq.gz
151122_I136_FCH3L2FBBXX_L7_wHAXPI023990-97_1.fq.gz
151122_I136_FCH3L2FBBXX_L7_wHAXPI023990-97_2.fq.gz
160103_I137_FCH3V5YBBXX_L3_WHPANwalDDAADWAAPEI-101_1.fq.gz
160103_I137_FCH3V5YBBXX_L3_WHPANwalDDAADWAAPEI-101_2.fq.gz
160103_I137_FCH3V5YBBXX_L4_WHPANwalDDAADWAAPEI-101_1.fq.gz
160103_I137_FCH3V5YBBXX_L4_WHPANwalDDAADWAAPEI-101_2.fq.gz
160103_I137_FCH3V5YBBXX_L5_WHPANwalDDABDLAAPEI-100_1.fq.gz
160103_I137_FCH3V5YBBXX_L5_WHPANwalDDABDLAAPEI-100_2.fq.gz
160103_I137_FCH3V5YBBXX_L5_WHPANwalDDACDTAAPEI-102_1.fq.gz
160103_I137_FCH3V5YBBXX_L5_WHPANwalDDACDTAAPEI-102_2.fq.gz
160103_I137_FCH3V5YBBXX_L6_WHPANwalDDABDLAAPEI-100_1.fq.gz
160103_I137_FCH3V5YBBXX_L6_WHPANwalDDABDLAAPEI-100_2.fq.gz
151114_I191_FCH3Y35BCXX_L2_wHAMPI023988-81_1.fq.gz
151114_I191_FCH3Y35BCXX_L2_wHAMPI023988-81_2.fq.gz
160103_I137_FCH3V5YBBXX_L6_WHPANwalDDACDTAAPEI-102_1.fq.gz
160103_I137_FCH3V5YBBXX_L6_WHPANwalDDACDTAAPEI-102_2.fq.gz

Mate pair sequencing files were uploaded together within a single “Run”.

Share

SRA Submission – Transcriptomic Profiles of Adult Female & Male Gonads in Panopea generosa (Pacific geoduck).

RNAseq experiment, which is part of a larger project that involves characterizing geoduck gonad development across multiple stages: histologically, proteomically, and transcriptomically. Initial sample collection performed by Grace Crandall.

The current status can be seen in the screen cap below. Current release date is set for a year from now, but will likely bump it up. Need Steven to review the details of the submission (BioProject, Experiment descriptions, etc.) before I initiate the public release. Will update this post with the SRA number once we receive it.

Here’s the list of files uploaded to the SRA:

Geo_Pool_F_GGCTAC_L006_R1_001.fastq.gz
Geo_Pool_F_GGCTAC_L006_R2_001.fastq.gz
Geo_Pool_M_CTTGTA_L006_R1_001.fastq.gz
Geo_Pool_M_CTTGTA_L006_R2_001.fastq.gz

Mate pair sequencing files were uploaded together within a single “Run”.

Share

SRA Submission – Individual Transcriptomic Profiles of C.gigas Before & After Heat Shock

RNA-seq experiment conducted by Claire in 2013.

She sampled mantle tissue from three adult oysters, allowed them to recover from the sampling (one week?) and then subjected those same oysters to a 1hr heat shock at 40C and collected mantle tissue from them again.

As this is our first Small Read Archive (SRA) submission in many years, I decided to submit these to the SRA due to the small number of samples (6) from the Illumina sequencing we had done to make sure it was manageable.

An overview of the basic SRA submission process is here.

The current status can be seen in the screen cap below. Current release date is set for a year from now, but will likely bump it up. Need Steven to review the details of the submission (BioProject, Experiment descriptions, etc.) before I initiate the public release. Will update this post with the SRA number once we receive it.

Here’s the list of files uploaded to the SRA:

2M-HS_CCGTCC_L001_R1_001.fastq.gz
2M_AGTCAA_L001_R1_001.fastq.gz
4M-HS_GTCCGC_L001_R1_001.fastq.gz
4M_AGTTCC_L001_R1_001.fastq.gz
6M-HS_GTGAAA_L001_R1_001.fastq.gz
6M_ATGTCA_L001_R1_001.fastq.gz

SRA Accession: SRP072251

Share

Data Management – SRA Submission Overview

We have an enormous backlog of high-throughput sequencing files (641 FASTQ files, to be exact) that we need/want to get added to the NCBI Sequence Read Archive (SRA).

This post provides a brief summary of what’s involved in the process (mostly via screen shots) and attempts to identify the various pitfalls/pains that I’ve already stumbled through trying to get a set of six FASTQ files submitted properly.

OVERALL – It’s horrible and tedious.

Important things to note:

  • Once any of the three required components for SRA submission have been created (SRA, BioProject, and BioSamples), they can no longer be edited/deleted by the user! Understandable if they’ve already been publicly released, but if they’re still in pre-public release status, I think the user should be able to make changes as they see fit. As it currently stands, the user has to email the help desk at SRA and/or BioProjects to make any changes.
  • Extremely difficult to figure out which information will show up (and where it will show up) in the final, formatted SRA record – no guide to this that I could find. Thus, if you screw it up, it’s a major, major hassle to try to change anything.
  • When creating a “Run” (within an “Experiment”, within your SRA submission), only include sequencing files that provide the same data (e.g. if you have multiple sequence files, each generated from different individuals/samples, then you need to create a separate “Experiment” and “Run” for each of those files – otherwise, all files uploaded to a “Run” are combined into a single SRA file that loses any distinguishing info from the separate sequencing files).
  • When creating a batch submission for BioSamples, there’s no way to set a Title attribute. This means all of your submissions (in my case) will have all have a title of “Invertebrate sample”. Considering that I will likely end up with dozens of BioSamples, that means there’s no easy way to distinguish them from each other without some extra clicking and poking around.

 

Here’s the best way to proceed:

  1. Create a BioProject. This will sit at the top of the hierarchy in the SRA submission and will be displayed as the STUDY associated with the SRA.

  2. Create BioSample(s). This will be the next level of the hierarchy in the SRA submission and will be displayed as SAMPLE. This only shows up in the SRA when you create a new “Experiment”

  3. Create SRA. This will end up encompassing any BioProject(s) and BioSample(s) that you need to include to describe the sequencing files you’re submitting to the SRA.

  4. Create an Experiment.

  5. Create a Run. This option is available once you’ve saved your experiment. This is where you provide your sequencing filename and associated MD5 checksum. This will also provide you with the login info to upload your sequencing files via FTP to NCBI servers. You can associate multiple sequencing files within a single run. This should be done if your sequencing files all provide data for the BioSample you selected. However, if you have sequencing files that are associated with different BioSamples, then you need to create an individual Experiment (and Run) for each BioSample!

 

Here are some links that might come in handy (although, none are that great)…

SRA Getting Started (helpful): http://trace.ncbi.nlm.nih.gov/Traces/sra/?cmd=show&f=sra_sub_expl&view=get_started

SRA Metadata Overview (this is helpful): http://www.ncbi.nlm.nih.gov/sra/docs/submitmeta/

SRA Submission Quick Start Guide (this is useful!): http://www.ncbi.nlm.nih.gov/sra/docs/submit/

FTP Upload Instructions: http://www.ncbi.nlm.nih.gov/sra/docs/submitfiles/

User UN-friendly SRA Guide: http://www.ncbi.nlm.nih.gov/books/NBK47528/?report=reader

 

And, here are the screen caps, roughly in chronological order of how the process presents itself. It’s too time consuming to caption any of these, so I’m putting them up for a reference. Also, all of the information seen in these screen caps has been deleted (because the entire submission was totally jacked up in multiple facets), so don’t look for any of the various submission IDs – they no longer exist. This is really just to visually show how many steps there are in order to get stuff submitted – it’s brutal.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Share