How to parse a caffe deploy.prototxt file or solver.prototxt file using python.

1) Install google's protobuf by downloading the source from https://developers.google.com/protocol-buffers/docs/downloads (I had protobuf-2.6.1.tar.gz), untarring (tar -zxvf [filename]), and following the instructions in the README

2) Get your caffe .proto file. It lives in [caffe folder]/src/caffe/proto/caffe.proto

3) Compile the caffe.proto file with protoc --python_out=. caffe.proto; it will produce caffe_pb2.py in the same directory.

4.1) For parsing a deploy.prototxt file, use:

import caffe_pb2 #you created this module with the protoc command

from google.protobuf.text_format import Merge

net = caffe_pb2.NetParameter()

Merge((open("deploy.prototxt",'r').read()), net)

4.2) For parsing a solver.prototxt file, use:

import caffe_pb2 #you created this module with the protoc command

from google.protobuf.text_format import Merge

solver = caffe_pb2.SolverParameter()

Merge((open("solver.prototxt",'r').read()), solver)

Remember, you can inspect the attributes of the object using dir(net) or dir(solver)

Yaml is really slow in vim

Apparently this is a known issue.

Here I'm documenting some specifics of the fix that weren't clear from the stackoverflow answer. Not totally sure if I did the "right" thing but it works.

1) Download https://raw.githubusercontent.com/stephpy/vim-yaml/master/after/syntax/yaml.vim to ~/.vim
2) Put "au BufNewFile,BufRead *.yaml,*.yml so ~/.vim/yaml.vim" at the top of ~/.vimrc

You should be good to go.

Recursive find and replace

This was way too hard to figure out. I gave up on using sed because I was getting "invalid command code ." and "illegal byte sequence" due to the -i option being interpreted differently on osx, and ended up corrupting my git repository..

find . -type f -print0 | xargs -0 perl -i -pe '$_ =~ s/progressUpdates/progressUpdate/g' 

Numpy str on integers extremely slow?

I found a weird performance difference today: create a numpy array of integers, and then convert it to an array of strings in a list comprehension. It's weirdly slow. Cast the numpy integers to python int's before calling str(), and you get nearly a 10x speedup. Observe:

$ python -m timeit 'import numpy as np; newX = [x for x in np.arange(4096)]; [str(x) for x in newX]'

100 loops, best of 3: 10.5 msec per loop

$ python -m timeit 'import numpy as np; newX = [x for x in np.arange(4096)]; [str(int(x)) for x in newX]'

1000 loops, best of 3: 1.23 msec per loop

Line by line python performance profiling (in particular, about where that @profile decorator comes from)

Here's the thing I didn't understand about rkern's line-by-line performance profiler: the kernprof executable contains information about the @profile decorator - no import is necessary. When you run your script without kernprof, you will get an error. But if you run kernprof -l [original python call], it works. Happy profiling!

Caffe incompatible with Boost 1.58.0

Boost 1.58.0 was released on April 17th, and it breaks compilation with Caffe on OSX (when the build reaches one of the "NVCC" steps, you get a string of errors starting with complaining about the absence of a semicolon, that look like this: /usr/local/include/boost/smart_ptr/detail/sp_counted_base_clang.hpp(27): error: expected a ";"

Googling landed me on this page which suggested an incompatibility with boost 1.58.0 (https://groups.google.com/forum/#!topic/caffe-users/fY2r6bO3_0w). However, the steps suggested in the solution (as of May 26th; it seems to have been updated since) didn't work for me. So this is what I did, and it's kindof sketchy but oh well...

[July 16th update: Ian Blenke put together a boost-1.57 brew and mentioned how to use it in the comments; it looks a lot easier and less sketchy to install, but if that does not work for you, read on...]

1) cd $(brew --prefix) (for me this cd's to /usr/local/).
2) cd Library/Formula
3) Make a backup of the files boost.rb and boost-python.rb
4) Replace the boost.rb file with the contents of https://raw.githubusercontent.com/Homebrew/homebrew/6fd6a9b6b2f56139a44dd689d30b7168ac13effb/Library/Formula/boost.rb
5) Replace the boost-python.rb file with the contents of https://raw.githubusercontent.com/Homebrew/homebrew/3141234b3473717e87f3958d4916fe0ada0baba9/Library/Formula/boost-python.rb

6) brew uninstall boost (this should uninstall the 1.58.0 version)
7) brew install boost (this should install 1.57.0 since you've replaced the rb files)

(Edit: You *may* have to do "brew link boost" after this - can't recall exactly. Also, Parang Saraf pointed out a "no rule to make target" error in the comments, which they solved with "make clean")

Also, I missed this guide the first time I went through the page, so I'm linking to it in case you did too: http://caffe.berkeleyvision.org/install_osx.html

Adventures installing shogun and shogun's python modular interface

First of all, whatever instructions they have on their main website seem to be out of date, because there is no "configure", so ./configure does not do anything.

Second, I think there used to be a file called "INSTALL", but there isn't anymore. And the README is utterly unhelpful. This is all I could find, and following it seemed to work. That github repo also seems to be the most up to date in terms of documentation.

If ccmake is installed, use it - it will give you insight into all the instructions. If not, well, you need to "cmake .. -DCmdLineStatic=ON -DPythonModular=ON" to get both the static command line interfaces and the python modular interface (and I even tossed in a -DPythonStatic=ON although I'm not sure it did anything...I was driving blind due to no ccmake, and for whatever reason none of this easily accessible online!). Also, my colleague pointed me to this quickstart, also on the GitHub.

I had to install swig for the python modular interface. IMPORTANT: with swig 3.0.2, the python modular interface gave me a segfault; but downgrading to 2.0.12 worked. I've done it in the past without root access, but that time I had ccmake and could easily tell shogun where to find swig - this time, I didn't have ccmake, so I couldn't figure out what to specify (seriously, this is awful documentation even by open source standards!). If you can't get root access, playing around with CMakeCache.txt in the build directory might be a start (note that the build directory has one CMakeChache.txt and the main directory also has one; you should modify the one in the build directory).

Finally, shogun installed completely error-free, but "import shogun" was not working. It was because shogun had installed to /usr/local/lib/python2.7/site-packages/, which wasn't even in my actual python's path, let alone anaconda's. To get it to work with anaconda, I copied the folder shogun and the files modshogup.py and _modshogun.so to /path/to/anaconda/lib/python2.7/site-packages.

To diagnose the issue with your particular installation of python, start the python shell, do "import sys", and then do "print sys.path" - if the place that shogun decided to put the python libraries (displayed on the screen when you do "make install", and also present in CMakeCache.txt in the build folder) is not in that list, well, that's the problem. By hook or by crook, get it on that list. One thing you can do is go back and recompile shogun, but with PYTHON_PACKAGES_PATH set to the right value - as I don't have ccmake, the only way I can think to set this is to open up CMakeCache.txt in your build folder and modify it there. I also hear something funky with the "PYTHONPATH" environment variable is supposed to work but for whatever reason, it never does for me. As a last resort, you can do "sys.path.insert(0, /path/to/shogun's/python/packages/folder/)"

I was then confronted with a whiny "ImportError: libshogun.so.17: cannot open shared object file: No such file or directory". I looked at the messages when shogun was installed and confirmed that libshogun.so.17 did in fact live in /usr/local/lib. Turns out I had to do "sudo ldconfig" to get that to behave. NO ONE MENTIONED THAT ANYWHERE.