Hello! Many parts of this tutorial “Running Serratus”are confusing to me. Could you be more specific?

[NickShanyt]

Hello, seraatus is a great tool and I would like to run it on my AWS server. Maybe I don't know enough about it, I think this document is not clear enough and many parts are very confusing. For example, are there folders and files that I need to create manually?
If possible, could you update the documentation with full deployment details and command lines?
Thank you!

[rcedgar]

Serratus is a large and complex tool and the documentation is a work in progress. If you could explain more about what you're trying to do we'll be glad to help with a configuration that meets your goals.

[NickShanyt]

Serratus is a large and complex tool and the documentation is a work in progress. If you could explain more about what you're trying to do we'll be glad to help with a configuration that meets your goals.

Many thanks.
We have some SRA databases that we would like to search based on RdRP and using your pipeline, hoping to discover a new RNA virus.

[ababaian]

Hi Nick, sure I'm happy to help. The documentation could use a little bit of updating.

Let's start at the beginning of the Running Serratus documentation, please follow these instructions until you reach an instruction which is unclear or not working and we can fix and update the documentation as we go through it.

Edit: The caveat here is that as you go through the process, please let me know any steps which you take which are not in the documentation such that I can document them.

[ababaian]

hey @NickShanyt, do you have any feedback for improving the documentation?

[pingleig]

I went through the wiki and got it running (btw: I am software engineer from Amazon volunteer with UBC CIC, so I only know things are running but have no knowledge related to genome)

AWS account

• AWS module can use newer version, example commit
• Some organization use SSO with session token (e.g. AWS itself) and they don't need to create user and save access key id & secret. Configure ~/.aws/config or ~/.aws/credentials to assume IAM role with admin access works out of box with updated version of terraform AWS module
• Configure ~/.ssh/config in the wiki way means using same pem file for all EC2 instances while this global config is only for the generated ./create_tunnel.sh . I think it's better to config the tunnel script to use specific pem file directly, example in my fork

ssh -i ~/Downloads/pem/serratus.pem -Nf -L 3000:localhost:3000 -L 9090:localhost:9090 ec2-user@ec2-3-215-88-150.compute-1.amazonaws.com

Packer

• I can find the public stable AMI but I cannot use it because the AMI search filter use owner self, so terraform will not find the AMI unless I build a new one with different name under my AWS account.

serratus/terraform/scheduler/main.tf

Line 44 in 18486ad

owners = ["self"]

Without changing name I got error for name conflict (I am not sure why ec2 make AMI name global ...

packer build docker-ami.json
amazon-ebs: output will be in this color.

==> amazon-ebs: Prevalidating any provided VPC information
==> amazon-ebs: Prevalidating AMI Name: packer-amazon-linux-2-docker-004
==> amazon-ebs: Error: AMI Name: 'packer-amazon-linux-2-docker-004' is used by an existing AMI: ami-01aa4fbd5cad1f0c1
Build 'amazon-ebs' errored after 1 second 918 milliseconds: Error: AMI Name: 'packer-amazon-linux-2-docker-004' is used by an existing AMI: ami-01aa4fbd5cad1f0c1

Terraform

• There is unresolved merge conflict, I picked the newer diff and it worked ... commit in fork

serratus/terraform/main/main.tf

Lines 49 to 59 in 18486ad

	<<<<<<< HEAD
	//variable "metrics_ip" {
	// type = string
	//}
	=======
	variable "metrics_ip" {
	type = string
	}
	>>>>>>> d0fff74d593fc0da9cf5adc7da41fbd26498cefc
	// PROVIDER/AWS ##############################

• Better mention the CIDR is for configuring security group for later SSH tunnel and submitting job. If one change physical location, turn on/off VPN, they need to update the security group accordingly.
• btw: Some organization use prefix list to configure security group inbound rules, though I didn't update the tf in my fork and just edited in EC2 console directly ...
• dockerhub have throttle, switch to public/private ECR. If one just need to run experiment in their own cluster, private ECR is better, public ECR still throttle if you pull image hard enough... I see there is an open issue for ECR Migrating from Dockerhub to ECR #164

SSH Tunnel

• Already mentioned in AWS Account section, I feel it's better to configure locally instead of updating global ssh config
• No stop tunnel sh, so I just grep and kill

Load SRA

• I didn't find existing SraRunInfo.csv in the repo ... and I don't know what NCBI SRA is until I googled and found https://www.ncbi.nlm.nih.gov/sra/docs/sradownload/#download-metadata-associated-wit
  It would be easier for software engineers if there is some link for download csv.
• Generated uploadSRA.sh does not work on macOS because split is BSD flavour (no -d) and md5sum does not ship with mac. brew install coreutils and change split to gsplit and gmd5sum works, the script could detect the distro and use different binary names

Launch cluster nodes

• I found there are scripts like ./dl_set_capacity.sh 5 in notebook, but I didn't find the script (in repo nor generated by tf). I suppose they are just wrapper for jq and curl.
• I am not sure when does the split begin, the default config for "DL_SCALING_MAX" was 0 when I forked the repo in Feb, after I changed it to 1 and updated the state changed from new to splitting. Though in the end it failed at Blocks after Accessions is done. I suppose something is run with aligner, but the ec2 instance is already terminating when I decided to ssh into it and run docker logs.

[ababaian]

:D :D :D This is fantastic @pingleig!!

This is a great layout of how to tackle this, issue by issue. If you get a chance can you join the slack: https://join.slack.com/t/hackseq-rna/shared_invite/zt-ewlzh9qf-SiNkxvvTJflcutFN0h5jIQ .

I think if we can distill this down and really clean-up the command-line interface and documentation.