Wednesday, November 6, 2013

Why I hate OpenMP

At +MOSEK we use OpenMP to parallelize our code. Well, in fact we used to use OpenMP exclusively. However, we have now moved to use pthreads or native Windows threads.

This seems like a very bad idea because the normal recommendation is to OpenMP, CiLK or Intel parallel building blocks or like. But I will now explain why I ignore this recommendation  and in fact think it is a bit naive.

At MOSEK we build a shared library (a DLL on Windows) that we sell to our customers. This library is called from the users program or programs like MATLAB, R etc. We have absolutely no control over the calling application. The calling application may be linked OpenMP or another threading library. Our users expects our application to work almost independently of what they do.

However, OpenMP is build on the implicit assumption you are an academic who have all the source code when building the application  and if an application is linked with OpenMP then the same version of OpenMP is used throughout the application. This definitely does not hold for us.

We have to ask ourselves what happen if the user build their application linking with MOSEK using gcc OpenMP and we build our library with Intel OpenMP. I think the answer is that the final applications crashes.

I recently reported crash in the Intel MKL BLAS library when called from MATLAB that seems to be caused by OpenMP issue. See the report. The answer from Intel is are you sure you use the same OpenMP as MATLAB. This is illustrates very well what I hate about OpenMP. It does not work unconditionally. I mean if you link with pthreads it then it works because pthreads is integrated into the operating system.

So my conclusion is. If you are a commercial library vendor then do not use OpenMP unless you want to get into trouble.

2 comments:

  1. Hypothetically problem should go away when OpenMP library is linked statically.
    Unfortunately this is not the case anymore (Intel's compilers had an option to link OpenMP statically in the past, but not now).

    As a test, a year ago I converted their libiomp5md.dll into static library and linked into my MEX.
    MATLAB was still crashing because of different OpenMP versions in MEX and in its core.

    I assume Intel's OpenMP allocates and uses some shared resources within one process space (locks?).
    So it is not possible to use different OpenMPs in one process, even with static linking :(.

    Moving to native threads as well...

    ReplyDelete
  2. As for the Linux (my comment above is related to Windows).

    I am using GCC and static inking to OpenMP does help. My toolbox works fine independently from what MATLAB is using.

    However - all libraries (like libstd++, libgcc, etc) should be linked statically as well.
    This is the main difficulty - standard GCC distributions do not support linking C++ libraries statically.
    GCC should be customarily re-built from the scratch to be able to do this.

    At the end, if everything is linked statically (and the only dynamic dependency is libc) - OpenMP works fine and (more importantly) binary is independent from any Linux distribution & version. Software can be distributed in binary form in Linux world.


    ReplyDelete