Adventures installing gkmersvm

gkmersvm is an svm kernel implementation from the Beer lab: http://www.beerlab.org/gkmsvm/

I ran into a couple of issues when trying to compile it, and I thought I'd describe them here.

When I first typed in make, I got: fatal error: 'tr1/unordered_map' file not found

This was fixed by upgrading my gcc using the instructions on: http://stackoverflow.com/questions/837992/update-gcc-on-osx

Then make resulted in: "'getopt' was not declared in this scope"

This was fixed by adding "#include <getopt.h>" to src/mainSVMclassify.cpp:

When I did this on our linux server, I also got a bunch of errors about "intptr_t" has not been declared. This was fixed by adding #include <stdint.h> to the following files:

src/CountKLmersGeneral.cpp:19

src/MLEstimKLmers.cpp

src/LKTree.cpp

src/LTree.cpp

src/LKTree.h 

After this, "make" produced the necessary binaries gkmsvm_classify, gkmsvm_kernel and gkmsvm_train. Curiously enough, there's no "make install", so I manually copied the binaries to /usr/local/bin.

Finally, when I tried to run the modshogun script, I got the error "Cannot create new instances of type 'Labels'". I think this is because I was using shogun 4.0.0, and the script was written for Shogun version 0.9.2 ~ 1.1.0. Changing every instance of "Labels" to "BinaryLabels" fixed the issue. Note, if you are getting a segfault with the modular interface, or are having trouble installing it, see my post about shogun.

Also...after that gcc upgrade, I ran into the following error with a line containing "dispatch_block_t", which once again requires modifying some source code: http://hamelot.co.uk/programming/osx-gcc-dispatch_block_t-has-not-been-declared-invalid-typedef/

Copying files via the terminal gets around "read only" error from USB in Ubuntu

This was very weird. I was recently trying to get files off an old laptop that had suffered a fall. The internet connection was not working. Bluetooth was not working. And it looked like an old USB I found, which I erased and reformatted multiple times, wasn't working either. When I tried pasting files into the drive via the GUI, I got the following error (yes I know how to take screenshots, but remember that whether or not I could get files off this computer was an open question):

And yet, copying into /media/[drive name] via the terminal worked, although it was slow:

[shakes head]

Course waiver info for Stanford courses

The C.S. PhD program has a course waiver process, and given that so many of the students come from similar institutions, there should really be a pre-approved list of courses to transfer over for every area, like there is for "Software Systems". But no, I had to fill out the course waiver form in its entirety, and didn't want to risk annoying the approver by leaving fields blank because I assumed they were familiar with the course. But the least I can do is dump the info here for the two courses I did apply for a waiver for (CS 173 and CS 161, using 6.047 and 6.046 from MIT respectively)

General Info

Stanford course to be waived

: CS 173

Advisor name

: Gill Bejerano

Non-Stanford Course Info

University Name

: Massachusetts Institute of Technology

Instructor

: Manolis Kellis

Year

: 2012

Course website

: http://stellar.mit.edu/S/course/6/fa12/6.047/index.html

Link to course assignments

: http://stellar.mit.edu/S/course/6/fa12/6.047/materials.html

Description:

Covers the algorithmic and machine learning foundations of computational biology, combining theory with practice. Principles of algorithm design, influential problems and techniques, and analysis of large-scale biological datasets. Topics include (a) genomes: sequence analysis, gene finding, RNA folding, genome alignment and assembly, database search; (b) networks: gene expression analysis, regulatory motifs, biological network analysis; (c) evolution: comparative genomics, phylogenetics, genome duplication, genome rearrangements, evolutionary theory. These are coupled with fundamental algorithmic techniques including: dynamic programming, hashing, Gibbs sampling, expectation maximization, hidden Markov models, stochastic context-free grammars, graph clustering, dimensionality reduction, Bayesian networks.

Syllabus:

Dynamic Programming, Global and local alignment
Database search, Rapid string matching, BLAST, BLOSUM
Multiple, Progressive, Phylogenetic, Whole-genome alignment
Whole-Genome Comparative genomics: Evolutionary signatures, Duplication
Genome Assembly: Consensus-alignment-overlap, Graph-based assembly
Hidden Markov Models Part 1: Evaluation / Parsing, Viterbi/Forward algorithm
Hidden Markov Models Part 2: PosteriorDecoding / Learning Baum Welch
Structural RNAs: Fold prediction, genome-wide annotation
Transcript structure analysis, Differential Expression, Significance Testing
Small and Large regulatory RNAs: lincRNA, miRNA, piRNA
Expression analysis: Clustering and Classification, K-means, Naïve-Bayes
Clustering: affinity propagation; Classification: Random Forests
Epigenomics: ChIP-Seq, Burrows-Wheeler alignment, Chromatin States
Regulatory Motifs: Discovery, Representation, PBMs, Gibbs Sampling, EM
Expression deconvolution functional data of mixed samples
Network Inference: Physical and functional networks, information integration
The ENCODE Project: Systematic experimentation and integrative genomics
Dimentionality reduction, PCA, Self-Organizing Maps, SVMs
Disease association mapping, GWAS, organismal phenotypes
Quantitative trait mapping, eQTLs, molecular trait variation
Linkage Disequilibrium, Haplotype phasing, variant imputat
Molecular Evolution, Tree Building, Phylogenetic inference
Gene/species trees, reconciliation, recombination graphs
Mutation rate estimation, coalescent, models of evolution
Missing Heritability, Complex Traits, Interpret GWAS, Rank-based enrichment
Recent human evolution: Human history, human selection
Personal Genomics, Disease Epigenomics: Systems approaches to disease
Three-dimentional chromatin interactions: 3C, 5C, HiC, ChIA-Pet
Pharmacogenomics: Network-based systems biology of drug response
Synthetic Biology: Reading and writing genomes and cellular circuits

+ Final project

Textbook list:

No textbook; Kellis was formulating extensive class notes with the help of the students.

Stanford Course Info

Description:

Introduction to computational biology through an informatic exploration of the human genome. Topics include: genome sequencing; functional landscape of the human genome (genes, gene regulation, repeats, RNA genes, epigenetics); genome evolution (comparative genomics, ultraconservation, co-option). Additional topics may include population genetics, personalized genomics, and ancient DNA. Course includes primers on molecular biology, the UCSC Genome Browser, and text processing languages. Guest lectures on current genomic research topics.

Syllabus:

Introductory biology
Protein Coding Genes
UCSC Genome Browser Tutorial
Introduction to Text Processing Tutorial
Non-protein Coding Genes
Transcriptional Activation I
Transcriptional Regulation II
Transcriptional Regulation III
Genome Evolution I: Repeats
Genome Evolution II
Chains & Nets, Conservation & Function
Sequencing, Human Variation, and Disease
Personal Genomics, GSEA/GREAT
Transcription factor binding sites - Functions and Complexes
Population Genetics & Evo-Devo
Ancestral genome-phenotype mapping

Textbook list:

No specific textbook

General Info

Stanford course to be waived

: CS 161

Advisor name

: Serge Plotkin

Non-Stanford Course Info

University Name

: Massachusetts Institute of Technology

Instructor

: Bruce Tidor, Dana Moshkovitz

Year

: 2012

Course website

: http://stellar.mit.edu/S/course/6/sp12/6.046/index.html

Link to course assignments

: http://stellar.mit.edu/S/course/6/sp12/6.046/materials.html (assignments not public)

Description:

Techniques for the design and analysis of efficient algorithms, emphasizing methods useful in practice. Topics include sorting; search trees, heaps, and hashing; divide-and-conquer; dynamic programming; greedy algorithms; amortized analysis; graph algorithms; and shortest paths. Advanced topics may include network flow; computational geometry; number-theoretic algorithms; polynomial and matrix calculations; caching; and parallel computing.

(This is the second algorithms class in the standard sequence)

Syllabus:

Median Finding
Scheduling
Minimum Spanning Trees
Fast Fourier Transform
All-Pairs Shortest Paths
Randomized algorithms, high probability bounds
Hashing
Amortized Analysis
Competitive Analysis
Network Flow
van Emde Boas Data Structure
Disjoint Sets Data Structures
P vs. NP
Approximation Algorithms
Compression
Sub-linear time algorithms
Clustering
Derandomization
Computational Geometry

Textbook list:

Introduction to Algorithms (CLRS)

Stanford Course Info

Description:

Worst and average case analysis. Recurrences and asymptotics. Efficient algorithms for sorting, searching, and selection. Data structures: binary search trees, heaps, hash tables. Algorithm design techniques: divide-and-conquer, dynamic programming, greedy algorithms, amortized analysis, randomization. Algorithms for fundamental graph problems: minimum-cost spanning tree, connected components, topological sort, and shortest paths.

Syllabus:

Algorithmic complexity and analysis (4 lectures)
Randomization, divide and conquer (2 lectures)
Heaps and counting sort (1 lecture)
Hashing (2 lectures)
Tree and graph definitions and properties (1 lecture)
Binary Search Trees (1 lecture)
Greedy Algorithms (2 lectures)
Dynamic programming (3 lectures)
Graph algorithms (4 lectures)

Textbook list:

CLRS

Execute a command for every line in a file in bash

Another very handy bash snippet:

cat nodesToConsider.txt | while read in; do cp ../'layer0_inputGuess_n'$in'.png' .; done

Diagonal axis labels in R

I'm dumping a relevant code snippet here because I know I will want to come back to it again:
(found here)

> bp = barplot(data$informativeness[1:len], axes=FALSE, axisnames=FALSE)
> text(bp, par("usr")[3], labels = data$feature[1:len], srt = 45, adj = c(1.1,1.1), xpd = TRUE, cex=.9)
> axis(2)

For those times you have to turn in source code on a problem set...

I just discovered http://www.planetb.ca/syntax-highlight-word and it makes things much nicer (it'll do syntax highlight in a way that can be copied into Word)

Stratified sampling code + av_scripts repo

Here's an issue I've encountered before, so I think it's worth documenting: when splitting into training/testing/validation sets, if your data is heterogenous it's important to keep the proportions of the various classes roughly equal or you can get frustrating results. A random split often doesn't achieve this.

I've written code to do this; it's in my av_scripts repo and it's called splitIntoTrainingTestAndValidationSets.py. You specify the columns representing the classes and the proportions of the train/test/validation files; it will produce three files in the same directory as your input file with the splits.

Setup instructions
These instructions are also present here but I am reproducing them for convenience:
$ git clone https://github.com/kundajelab/av_scripts.git
$ export UTIL_SCRIPTS_DIR=/path/to/av_scripts
$ export PATH=$PATH:$UTIL_SCRIPTS_DIR/exec
(see the linked page for more details)

Example
splitIntoTrainingTestAndValidationSets.py --inputFile allLabels_200K.tsv --categoryColumns 1 2 --splitProportions 0.85 0.075 0.075

produces split_train_allLabels_200K.tsv, split_valid_allLabels_200K.tsv and split_test_allLabels_200K.tsv, with the proportions of the values from columns 1 and 2 are roughly even in those files.

If you don't have a validation split, you can do this:
splitIntoTrainingTestAndValidationSets.py --inputFile allLabels_200K.tsv --categoryColumns 1 2 --splitProportions 0.85 0.15 --splitNames train test

To get split_train_allLabels_200K.tsv and split_test_allLabels_200K.tsv. You can also view help with splitIntoTrainingAndValidationSets.py --help

Differences compared to the scikit-learn implementation
Scikitlearn can only do the split by the explicit class variable; sometimes, the class is not what you're predicting, so it isn't part of either your training data or testing label, but it still affects performance. To take my example, my prediction task was enhancer/not-enhancer, but I still wanted an even representation of cell-types in my positive set. Another issue with the scikit learn implementation is that it requires reading the data into memory. If your data doesn't fit into memory, my code will do the split by reading chunks of size --batchSize and doing the split batch by batch. One perk of this is that if your data is sorted by some score, by setting --batchSize small enough you can get a roughly even representation of scores.