Tuesday, April 7, 2015

Adventures installing shogun and shogun's python modular interface

First of all, whatever instructions they have on their main website seem to be out of date, because there is no "configure", so ./configure does not do anything.

Second, I think there used to be a file called "INSTALL", but there isn't anymore. And the README is utterly unhelpful. This is all I could find, and following it seemed to work. That github repo also seems to be the most up to date in terms of documentation.

If ccmake is installed, use it - it will give you insight into all the instructions. If not, well, you need to "cmake .. -DCmdLineStatic=ON -DPythonModular=ON" to get both the static command line interfaces and the python modular interface (and I even tossed in a -DPythonStatic=ON although I'm not sure it did anything...I was driving blind due to no ccmake, and for whatever reason none of this easily accessible online!). Also, my colleague pointed me to this quickstart, also on the GitHub.

I had to install swig for the python modular interface. IMPORTANT: with swig 3.0.2, the python modular interface gave me a segfault; but downgrading to 2.0.12 worked. I've done it in the past without root access, but that time I had ccmake and could easily tell shogun where to find swig - this time, I didn't have ccmake, so I couldn't figure out what to specify (seriously, this is awful documentation even by open source standards!). If you can't get root access, playing around with CMakeCache.txt in the build directory might be a start (note that the build directory has one CMakeChache.txt and the main directory also has one; you should modify the one in the build directory).

Finally, shogun installed completely error-free, but "import shogun" was not working. It was because shogun had installed to /usr/local/lib/python2.7/site-packages/, which wasn't even in my actual python's path, let alone anaconda's. To get it to work with anaconda, I copied the folder shogun and the files modshogup.py and _modshogun.so to /path/to/anaconda/lib/python2.7/site-packages.

To diagnose the issue with your particular installation of python, start the python shell, do "import sys", and then do "print sys.path" - if the place that shogun decided to put the python libraries (displayed on the screen when you do "make install", and also present in CMakeCache.txt in the build folder) is not in that list, well, that's the problem. By hook or by crook, get it on that list. One thing you can do is go back and recompile shogun, but with PYTHON_PACKAGES_PATH set to the right value - as I don't have ccmake, the only way I can think to set this is to open up CMakeCache.txt in your build folder and modify it there. I also hear something funky with the "PYTHONPATH" environment variable is supposed to work but for whatever reason, it never does for me. As a last resort, you can do "sys.path.insert(0, /path/to/shogun's/python/packages/folder/)"

I was then confronted with a whiny "ImportError: libshogun.so.17: cannot open shared object file: No such file or directory". I looked at the messages when shogun was installed and confirmed that libshogun.so.17 did in fact live in /usr/local/lib. Turns out I had to do "sudo ldconfig" to get that to behave. NO ONE MENTIONED THAT ANYWHERE.

2 comments:

  1. Hi Avanti,
    Shogun documentation needs some serious updating, I agree with you on that one.
    Anyways, I am trying to run shogun python modular interface. I couldn't even go past building Shogun 4.0 with ccmake. Running ccmake recursively from the build folder, whatever I do, I am always slapped with the error " -- NO supported backend(s) for linear algebra libraries found". And whatever I do, I couldn't get past this point. Any suggestions?

    PS: This is the content of my cmake options

    ARPACK_LIB ARPACK_LIB-NOTFOUND
    ATLAS_CBLAS /usr/lib/atlas-base/libcblas.so.3
    ATLAS_F77BLAS /usr/lib/atlas-base/libf77blas.so.3
    ATLAS_LAPACK /usr/lib/atlas-base/liblapack_atlas.so.3
    ATLAS_LIB /usr/lib/atlas-base/libatlas.so.3
    ATLAS_REFERENCE_LAPACK /usr/lib/liblapack.so.3
    BUILD_DASHBOARD_REPORTS OFF
    BUILD_EXAMPLES OFF
    BUNDLE_ARPREC OFF
    BUNDLE_COLPACK OFF
    BUNDLE_EIGEN ON
    BUNDLE_JSON OFF
    BUNDLE_NLOPT OFF
    CMAKE_BUILD_TYPE Release
    CMAKE_CXX_COMPILER /usr/bin/c++
    CMAKE_C_COMPILER /usr/bin/cc
    CMAKE_INSTALL_PREFIX /home/yoda/shogun-install
    CPLEX_ROOT_DIR
    CSharpModular OFF
    CmdLineStatic OFF
    CoreLib ON
    EIGEN_INCLUDE_DIRS /home/yoda/Desktop/anpr/shogun/eigen-3/cmake
    ENABLE_CCACHE ON
    ENABLE_COVERAGE OFF
    ENABLE_TESTING OFF
    EigenSolver ON
    GLPK_ROOT_DIR
    HAVE_LARGEFILE ON
    HDF5_DIR HDF5_DIR-NOTFOUND
    JavaModular OFF
    LinAlgBackend OFF
    LinSolverLib OFF
    LuaModular OFF
    ...

    ReplyDelete
  2. Hi Mr. Z,

    Based on this gist (https://gist.github.com/srgnuclear?direction=asc&sort=updated), I am guessing that none of the 3 possible linear algebra libraries (Eigen3/Viennacl/Lapack) were found. I looked at my own CMakeCache.txt and it looks like I installed with Lapack, but my Lapack-related config lines look identical to yours - perhaps double check that they are correct on your machine? I also note you have something specified for EIGEN_INCLUDE_DIRS, whereas I just seem to have "EIGEN_INCLUDE_DIR-NOTFOUND". Not sure what to make of that. Anyway, these are the contents of all my lapack-related lines in CMakeCache.txt, fwiw

    ATLAS_INCLUDES:PATH=/usr/include/atlas
    ATLAS_LAPACK:FILEPATH=/usr/lib/atlas-base/liblapack_atlas.so.3
    ATLAS_REFERENCE_LAPACK:FILEPATH=/usr/lib/liblapack.so.3

    ReplyDelete