Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error: lazy symbol binding failed: Symbol not found: ___emutls_get_address #8217

Closed
arshtepe opened this issue Sep 8, 2019 · 35 comments
Closed
Assignees
Labels
DevOps Issues related to CI/CD and pipelines LIBND4J

Comments

@arshtepe
Copy link

arshtepe commented Sep 8, 2019

Issue Description

Please describe our issue, along with:
I have error when run my project in 1.0.0-beta5, it is working in 1.0.0-beta4

dyld: lazy symbol binding failed: Symbol not found: ___emutls_get_address
  Referenced from: ~/.javacpp/cache/nd4j-native-1.0.0-beta5-macosx-x86_64.jar/org/nd4j/nativeblas/macosx-x86_64/libnd4jcpu.dylib
  Expected in: /usr/lib/libSystem.B.dylib

Version Information

Please indicate relevant versions, including, if relevant:

  • Deeplearning4j: 1.0.0-beta5
  • Platform information (OS, etc): macOs Mojave 10.14.4
  • CUDA version, if used
  • NVIDIA driver version, if in use

Additional Information

Where applicable, please also provide:

  • Full log or exception stack trace (ideally in a Gist: gist.github.com):
dyld: lazy symbol binding failed: Symbol not found: ___emutls_get_address
  Referenced from: ~/.javacpp/cache/nd4j-native-1.0.0-beta5-macosx-x86_64.jar/org/nd4j/nativeblas/macosx-x86_64/libnd4jcpu.dylib
  Expected in: /usr/lib/libSystem.B.dylib
@saudet
Copy link
Contributor

saudet commented Sep 8, 2019

@sshepel Which versions of GCC and Xcode are we building with these days?

@saudet saudet added C++ DevOps Issues related to CI/CD and pipelines LIBND4J labels Sep 8, 2019
@sshepel
Copy link
Contributor

sshepel commented Sep 8, 2019

@saudet for release and generic builds we have:

-- The C compiler identification is GNU 8.3.0
-- The CXX compiler identification is GNU 8.3.0

@raver119
Copy link
Contributor

raver119 commented Sep 8, 2019

@arshtepe can you share the code you're running?

I've been running examples before release, they were running fine. Just ran examples right now - still running fine. Tried lenet example, and some basic lstm/dense examples. No issues as well.

@raver119
Copy link
Contributor

raver119 commented Sep 8, 2019

Screen Shot 2019-09-08 at 15 45 39

@arshtepe
Copy link
Author

arshtepe commented Sep 8, 2019

@saudet
Xcode 10.3
Build version 10G8

cmake version 3.15.2
gcc -dumpversion -> 4.2.1
@raver119 sorry I can't share the code :( but I can provide any details which you want

@saudet
Copy link
Contributor

saudet commented Sep 8, 2019

@sshepel And the version of Xcode?

@sshepel
Copy link
Contributor

sshepel commented Sep 8, 2019

@saudet Xcode version: 8.3.3

@saudet
Copy link
Contributor

saudet commented Sep 9, 2019

GCC 8.x was released beginning of 2018: https://www.gnu.org/software/gcc/releases.html
So let's try to use a version of Xcode that was released around the same time frame. That would be close to Xcode 9.2, which is the last to run on Mac OS X 10.12.6: https://en.wikipedia.org/wiki/Xcode

@sshepel
Copy link
Contributor

sshepel commented Sep 9, 2019

@saudet sure, will do that ASAP.

@sshepel
Copy link
Contributor

sshepel commented Sep 9, 2019

Ok, Xcode has been updated to version 9.2.0. Snapshots are on the way!
@arshtepe could you please try to use snapshots tomorrow to check if the issue has been resolved for you...

@arshtepe
Copy link
Author

arshtepe commented Sep 9, 2019

@sshepel okey, I will let you know, thanks :)

@zhaihao
Copy link

zhaihao commented Sep 10, 2019

I have the same problem.

code:

import org.nd4s.Implicits._
val arr = (1 to 9).toNDArray(3,3)
println(arr)

output:

19:58:02.671 [info ] o.n.linalg.factory.Nd4jBackend(196)      - Loaded [CpuBackend] backend
19:58:03.135 [info ] o.n.nativeblas.NativeOpsHolder(104)      - Number of threads used for OpenMP: 4
dyld: lazy symbol binding failed: Symbol not found: ___emutls_get_address
19:58:03.602 [info ] org.nd4j.nativeblas.Nd4jBlas(55)         - Number of threads used for OpenMP BLAS: 4
  Referenced from: /Users/zhaihao/.javacpp/cache/nd4j-native-1.0.0-beta5-macosx-x86_64.jar/org/nd4j/nativeblas/macosx-x86_64/libnd4jcpu.dylib
  Expected in: /usr/lib/libSystem.B.dylib

dyld: Symbol not found: ___emutls_get_address
  Referenced from: /Users/zhaihao/.javacpp/cache/nd4j-native-1.0.0-beta5-macosx-x86_64.jar/org/nd4j/nativeblas/macosx-x86_64/libnd4jcpu.dylib
  Expected in: /usr/lib/libSystem.B.dylib

system env:

Mac OSX 10.14.6

$ pkgutil --pkg-info=com.apple.pkg.CLTools_Executables
package-id: com.apple.pkg.CLTools_Executables
version: 10.3.0.0.1.1562985497
volume: /
location: /
install-time: 1563950915
groups: com.apple.FindSystemFiles.pkg-group

$ cmake --version
cmake version 3.14.5

CMake suite maintained and supported by Kitware (kitware.com/cmake).

$ gcc -v
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include/c++/4.2.1
Apple LLVM version 10.0.1 (clang-1001.0.46.4)
Target: x86_64-apple-darwin18.7.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

I hope this would help.

@raver119
Copy link
Contributor

at this moment I'm not sure why it's reproducible for you, but we suppose that's an issue related to thread_local storage specifier, so I've disabled it for macos now. once snapshots rebuilt please try again

@arshtepe
Copy link
Author

@raver119 I can't update to snapshot, it is loading infinitely
my pom

    <repositories>
        <repository>
            <id>snapshots-repo</id>
            <url>https://oss.sonatype.org/content/repositories/snapshots</url>
            <releases>
                <enabled>false</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
                <updatePolicy>daily</updatePolicy>  <!-- Optional, update daily -->
            </snapshots>
        </repository>
    </repositories>
    <dependencies>
        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-core</artifactId>
            <version>1.0.0-SNAPSHOT</version>
        </dependency>

this way doesn't work too

    <repositories>
        <repository>
            <id>snapshots-repo</id>
            <url>https://oss.sonatype.org/content/repositories/snapshots</url>
            <releases>
                <enabled>false</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
                <updatePolicy>daily</updatePolicy>  <!-- Optional, update daily -->
            </snapshots>
        </repository>
    </repositories>
    <dependencies>
        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-core</artifactId>
            <version>1.0.0-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>org.nd4j</groupId>
            <artifactId>nd4j-native-platform</artifactId>
            <version>1.0.0-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-modelimport</artifactId>
            <version>1.0.0-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-ui_2.12</artifactId>
            <version>1.0.0-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>org.nd4j</groupId>
            <artifactId>nd4j-api</artifactId>
            <version>1.0.0-SNAPSHOT</version>
        </dependency>

@zhaihao
Copy link

zhaihao commented Sep 11, 2019

@raver119
Copy link
Contributor

@raver119 I can't update to snapshot, it is loading infinitely

It looks like sonatype has some problems atm, we've noticed similar problems here as well

@arshtepe
Copy link
Author

arshtepe commented Sep 11, 2019

now I have new error ,
btw code https://gitlab.com/Eliza_H/cnn_network/blob/master/src/main/java/NetworkModel.java

Exception in thread "main" java.lang.ExceptionInInitializerError
	at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.<init>(NativeOpExecutioner.java:91)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at java.lang.Class.newInstance(Class.java:442)
	at org.nd4j.linalg.factory.Nd4j.initWithBackend(Nd4j.java:5246)
	at org.nd4j.linalg.factory.Nd4j.initContext(Nd4j.java:5153)
	at org.nd4j.linalg.factory.Nd4j.<clinit>(Nd4j.java:182)
	at org.deeplearning4j.nn.conf.NeuralNetConfiguration$Builder.seed(NeuralNetConfiguration.java:579)
	at NetworkCNNModel.execute(NetworkCNNModel.java:115)
	at NetworkCNNModel.main(NetworkCNNModel.java:276)
Caused by: java.lang.RuntimeException: ND4J is probably missing dependencies. For more information, please refer to: http://nd4j.org/getstarted.html
	at org.nd4j.nativeblas.NativeOpsHolder.<init>(NativeOpsHolder.java:106)
	at org.nd4j.nativeblas.NativeOpsHolder.<clinit>(NativeOpsHolder.java:34)
	... 12 more
Caused by: java.lang.UnsatisfiedLinkError: no jnind4jcpu in java.library.path
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
	at java.lang.Runtime.loadLibrary0(Runtime.java:870)
	at java.lang.System.loadLibrary(System.java:1122)
	at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1487)
	at org.bytedeco.javacpp.Loader.load(Loader.java:1190)
	at org.bytedeco.javacpp.Loader.load(Loader.java:1040)
	at org.nd4j.nativeblas.Nd4jCpu.<clinit>(Nd4jCpu.java:10)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at org.nd4j.nativeblas.NativeOpsHolder.<init>(NativeOpsHolder.java:84)
	... 13 more
Caused by: java.lang.UnsatisfiedLinkError: ..../.javacpp/cache/nd4j-native-1.0.0-20190911.011548-14906-macosx-x86_64.jar/org/nd4j/nativeblas/macosx-x86_64/libjnind4jcpu.dylib: dlopen(/Users/artemshtepenko/.javacpp/cache/nd4j-native-1.0.0-20190911.011548-14906-macosx-x86_64.jar/org/nd4j/nativeblas/macosx-x86_64/libjnind4jcpu.dylib, 1): Library not loaded: @rpath/libmkldnn.0.dylib
  Referenced from: ..../.javacpp/cache/nd4j-native-1.0.0-20190911.011548-14906-macosx-x86_64.jar/org/nd4j/nativeblas/macosx-x86_64/./libnd4jcpu.dylib
  Reason: image not found
	at java.lang.ClassLoader$NativeLibrary.load(Native Method)
	at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
	at java.lang.Runtime.load0(Runtime.java:809)
	at java.lang.System.load(System.java:1086)
	at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1461)
	... 19 more

@raver119
Copy link
Contributor

@saudet any ideas why it still tries to load libmkldnn?

  1. for x86_64 binary libnd4j doesn't use mkldnn atm.
  2. all libmkldnn references from java were removed.

@raver119
Copy link
Contributor

p.s. @AlexDBlack tried to run latest snapshot on windows and it runs fine for him.

@saudet
Copy link
Contributor

saudet commented Sep 12, 2019

The artifacts are probably just not up-to-date..

@streambuf
Copy link

streambuf commented Sep 15, 2019

i have same problem when try run org.deeplearning4j.examples.convolution.AnimalsClassification
commit 391ec1bcfa69b13804b78103a645f425c1dd5ac0

@saudet
Copy link
Contributor

saudet commented Sep 16, 2019

This problem is probably caused by the upgrade to GCC 8.3.
These patches needed to support rpath haven't been upgraded on the CI server:
https://github.com/bytedeco/javacpp-presets/blob/master/ci/install-travis.sh#L218
The workaround for now is to install GCC 8.3 with brew install gcc@8.

@streambuf
Copy link

gcc -v
gcc version 9.2.0 (Homebrew GCC 9.2.0)
not helped

@streambuf
Copy link

This problem is probably caused by the upgrade to GCC 8.3.
These patches needed support rpath haven't been upgraded on the CI server:
https://github.com/bytedeco/javacpp-presets/blob/master/ci/install-travis.sh#L218
The workaround for now is to install GCC 8.3 with brew install gcc@8.

It's help me. Thanks!

@timothyjward
Copy link

In case it is helpful I have created a simple test repo which shows this issue.

After running brew install gcc@8 I see both beta4 and beta5 working, but the latest snapshots still fail with the UnsatisfiedLinkError

@saudet
Copy link
Contributor

saudet commented Sep 25, 2019

Yeah, the snapshots are currently broken, unfortunately. It may take a couple of weeks to get things running well again.

@timothyjward
Copy link

Yeah, the snapshots are currently broken, unfortunately. It may take a couple of weeks to get things running well again.

That's understandable, multi-platform native code can be a real pain!

@rafmontano
Copy link

Hi All

I am new on this, so please forget me for anything that should I know or it should be given for granted.

Look, I have been using a model for quite a while and since i upgraded to Catalina, I have been issues with all the models including the test demo. I am using now 1.0.0-beta5.

I upgraded from Java 8 to Java 12, and avoid core dump problems... however now I am having the following error Message just compiling, any of the demo cases and my model.

Please any work around / suggestion is really appreciated.

__"/Library/Java/JavaVirtualMachines/jdk-12.0.2.jdk
Picked up _JAVA_OPTIONS: -Xmn40G -Xms50G -Xmx50G -XX:+UseG1GC
o.n.l.f.Nd4jBackend - Loaded [CpuBackend] backend
o.n.n.NativeOpsHolder - Number of threads used for OpenMP: 24
o.n.n.Nd4jBlas - Number of threads used for OpenMP BLAS: 24
dyld: lazy symbol binding failed: Symbol not found: ___emutls_get_address
Referenced from: /Users/Rafa/.javacpp/cache/nd4j-native-1.0.0-beta5-macosx-x86_64.jar/org/nd4j/nativeblas/macosx-x86_64/libnd4jcpu.dylib
Expected in: /usr/lib/libSystem.B.dylib

dyld: Symbol not found: ___emutls_get_address
Referenced from: /Users/Rafa/.javacpp/cache/nd4j-native-1.0.0-beta5-macosx-x86_64.jar/org/nd4j/nativeblas/macosx-x86_64/libnd4jcpu.dylib
Expected in: /usr/lib/libSystem.B.dylib

Process finished with exit code 134 (interrupted by signal 6: SIGABRT)
"__

@eraly
Copy link
Contributor

eraly commented Oct 22, 2019

@rafmontano Try after running brew install gcc@8. Thank you for your patience! Let us know if you are still running into trouble.

@rafmontano
Copy link

Hi Susan,

Thanks for come back to me. Running Brew Install gcc8 didn't work for me in MacOS Catalina and Java 8. Gave me this error:

/Library/Java/JavaVirtualMachines/jdk1.8.0_231.jdk/Contents/Home/bin/java "-javaagent:/Applications/IntelliJ IDEA CE.app/Contents/lib/idea_rt.jar=63118:/Applications/IntelliJ IDEA CE.app/Contents/bin" -Dfile.encoding=UTF-8 -classpath
Picked up _JAVA_OPTIONS: -Xmn8G -Xms8G -Xmx8G -XX:+UseG1GC

A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00007fff674d20b4, pid=38904, tid=0x0000000000000f03

JRE version: Java(TM) SE Runtime Environment (8.0_231-b11) (build 1.8.0_231-b11)

Java VM: Java HotSpot(TM) 64-Bit Server VM (25.231-b11 mixed mode bsd-amd64 compressed oops)

Problematic frame:

C [libc++abi.dylib+0x30b4] __cxxabiv1::__si_class_type_info::has_unambiguous_public_base(__cxxabiv1::__dynamic_cast_info*, void*, int) const+0x4

Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

An error report file with more information is saved as:

/Users/Rafa/Dropbox/DeepLearning4j/NewProjects/1.0.0-beta4/dl4j-examples-predict/hs_err_pid38904.log

If you would like to submit a bug report, please visit:

http://bugreport.java.com/bugreport/crash.jsp

The crash happened outside the Java Virtual Machine in native code.

See problematic frame for where to report the bug.

Process finished with exit code 134 (interrupted by signal 6: SIGABRT)


I have 2 machines, I managed to roll back 1 to MacOS Mojave (it was not that easy), and now i have the model operational in that one.

Please I have a questions, Which Version of Java do you recommend? I upgraded in the rush of the moment to java 12 (from 8), to see if that solve the problem and solve it that one, however that caused the lazy symbol issue on this post, which in MacOS Catalina I have not managed to pass.

@sabarishs
Copy link

@rafmontano Try after running brew install gcc@8. Thank you for your patience! Let us know if you are still running into trouble.

This helped solve my problem! Thanks

@rafmontano
Copy link

Thanks Sabarishs,I have managed to run my model. Moving to Java 11 and running brew install gcc@8, solved the problem.

@raver119 raver119 removed the C++ label Nov 10, 2019
@raver119
Copy link
Contributor

AFAIK, this issue is resolved now, and updated binaries are available via snapshots. If, for some reason, issue is still reproducible for you - feel free to reopen this issue, or file a new one.

@hinsbergen
Copy link

hinsbergen commented Nov 25, 2019

Hi, I'm running into the same problems. The command "brew install gcc@8" indeed solved it for my Mac, but not for Windows where I run another instance.

I have downloaded the latest version (beta5) from Maven Central; it is dated September 6. Where can I get new jars where this error has been fixed? Or do I have to wait for beta6? Thanks!

@AlexDBlack
Copy link
Contributor

@hinsbergen The fix is only available via snapshots at present: https://deeplearning4j.org/docs/latest/deeplearning4j-config-snapshots

It will be in the next release of course (1.0.0-beta6), which should be out within 1-2 weeks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
DevOps Issues related to CI/CD and pipelines LIBND4J
Projects
None yet
Development

No branches or pull requests