ImportError: Error importing numpy: you should not try to import numpy from its source directory

[Trentonom0r3]

I'm working from within an embedded environment.
When trying to run the following code;

        try {
            py::exec(R"(
        import sys
        import os
        import traceback
        from io import StringIO

        # Redirect stdout and stderr
        old_stdout = sys.stdout
        old_stderr = sys.stderr
        sys.stdout = StringIO()
        sys.stderr = StringIO()

        try:
            import numpy as np
            print(np.__version__)
        except Exception as e:
            traceback.print_exc()

        # Get the captured output
        stdout_output = sys.stdout.getvalue()
        stderr_output = sys.stderr.getvalue()

        # Restore stdout and stderr
        sys.stdout = old_stdout
        sys.stderr = old_stderr
        )");

            // Retrieve the captured output from the Python execution
            py::object main_module = py::module::import("__main__");
            py::object stdout_output = main_module.attr("stdout_output");
            py::object stderr_output = main_module.attr("stderr_output");

            // Print the captured output
            std::string stdout_str = stdout_output.cast<std::string>();
            std::string stderr_str = stderr_output.cast<std::string>();
            std::cout << "Captured stdout:\n" << stdout_output.cast<std::string>() << std::endl;
            std::cerr << "Captured stderr:\n" << stderr_output.cast<std::string>() << std::endl;

        }
        catch (const py::error_already_set& e) {
            std::cerr << "Error in Python execution: " << e.what() << std::endl;
        }

I receive the traceback:

Traceback (most recent call last):
  File "C:\Program Files\Python311\Lib\site-packages\numpy\_core\__init__.py", line 23, in <module>
    from . import multiarray
  File "C:\Program Files\Python311\Lib\site-packages\numpy\_core\multiarray.py", line 10, in <module>
    from . import overrides
  File "C:\Program Files\Python311\Lib\site-packages\numpy\_core\overrides.py", line 8, in <module>
    from numpy._core._multiarray_umath import (
ModuleNotFoundError: No module named 'numpy._core._multiarray_umath'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Python311\Lib\site-packages\numpy\__init__.py", line 128, in <module>
    from numpy.__config__ import show as show_config
  File "C:\Program Files\Python311\Lib\site-packages\numpy\__config__.py", line 4, in <module>
    from numpy._core._multiarray_umath import (
  File "C:\Program Files\Python311\Lib\site-packages\numpy\_core\__init__.py", line 49, in <module>
    raise ImportError(msg)
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.11 from "C:\Program Files\Adobe\Adobe After Effects 2023\Support Files\AfterFX.exe"
  * The NumPy version is: "2.0.1"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy._core._multiarray_umath'


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 15, in <module>
  File "C:\Program Files\Python311\Lib\site-packages\numpy\__init__.py", line 133, in <module>
    raise ImportError(msg) from e
ImportError: Error importing numpy: you should not try to import numpy from
        its source directory; please exit the numpy source tree, and relaunch
        your python interpreter from there.

I've tried installing, uninstalling, using different site-packages paths, etc, etc, but all to no avail.
Numpy works fine with imported in a shell or running a script, but in the embedded context, its a no go.

[mheyman]

I got the same thing. I've built with vcpkg and, apparently, the libpython3.11.so.1.0 that gets build as part of the python3 port (a pybind11 port dependency) doesn't have the built-in math module on linux (I've had no problems on Windows although a quick look at the python3 vcpkg port doesn't show some "do not build the built-in math module on linux" action so maybe the executable is just finding the correct DLL on windows).

To fix it (for me):

1. Build with RPATH=$ORIGIN to add the local directory to the executable's runtime search path like:

    set_property(TARGET prog APPEND PROPERTY BUILD_RPATH "$ORIGIN")
    set_property(TARGET prog APPEND PROPERTY INSTALL_RPATH "$ORIGIN")

2. /usr/lib/x86_64-linux-gnu/libpython3.11.so.1.0 (or where ever your "real" python lives) to sit next to your executable got past the issue.

I suspect somehow setting the runtime search path to ignore the vcpkg-built libpython3.11.so.1.0 would also work.

--edited to add--
And, I was right! CMake's SKIP_BUILD_RPATH will give no RUNPATH entry in the executable. But I still needed RUNPATH=$ORIGIN for another library so I had to add that manually via a target_link_option(). So, instead of setting BUILD_RPATH and INSTALL_RPATH, you can turn off the RUNPATH setting completely and then manually turn it on for the build:

    set_property(TARGET prog PROPERTY SKIP_BUILD_RPATH TRUE)
    target_link_options(prog PRIVATE -Wl,-rpath,\$ORIGIN)
    set_property(TARGET prog APPEND PROPERTY INSTALL_RPATH "$ORIGIN")

--edited to add--

Maybe best solution:

Doing the following (setting various values prior to find_package(Python...)) also bypasses trying to link with the vcpkg python

set(My_Python_Version "3.11")
set(Python_EXECUTABLE "/usr/bin/python${My_Python_Version}")
set(Python_INCLUDE_DIR "/usr/include/python${My_Python_Version}")
set(Python_LIBRARY "/usr/lib/x86_64-linux-gnu/libpython${My_Python_Version}.so")
execute_process(COMMAND ${Python_EXECUTABLE} -c "import numpy;print(numpy.get_include())" OUTPUT_VARIABLE Python_NumPy_INCLUDE_DIR OUTPUT_STRIP_TRAILING_WHITESPACE)
find_package(Python ${My_Python_Version} EXACT COMPONENTS Interpreter Development NumPy REQUIRED)

but this may be pretty specific to Ubuntu (the only thing I've run on).