lto-wrapper.exe: fatal error: CreateProcess: No such file or directory

Note that in order to recreate this issue, parallelised LTO should be employed using -flto=<n>

So, with 8.3.1-2019q3 things are even worse, even adding the `bin` directory of the toolchain doesn't help gcc/ld in finding `lto-wrapper.exe` causing the error. The problem applies only to scenarios where linking gcc is invoked with `-flto=n` or `-fuse-linker-plugin. The inability to use the former strips us from even more aggressive optimisations.

Does anyone knows method how to overcome this issue?

Michal,

For some reason I cannot reproduce the issue you reported. Here is what I tried:
-----------------
G:\test>PATH C:\WINDOWS\system32;C:\WINDOWS

G:\test>"c:\Program Files (x86)\GNU Tools ARM Embedded\8 2019-q3-update\bin\arm-none-eabi-gcc" a.c -mthumb -mcpu=cortex-m4 -Os -flto -fuse-linker-plugin -specs=rdimon.specs -specs=nano.specs -lc -lc -lrdimon -o a.axf

G:\test>

Can you please share a simple case so that we can reproduce?

Thank for getting back to me @jinyun-ye, but your example it too simple.

I enclosed dumb demo that demonstrates the problem, in general you need to link multiple object files or archives into a final binary for the bug to manifest itself.

The `build.bat` uses -flto=3 in the last step, which fails:
C:\...\AppData\Local\GCC_ARM\8.3.1-2019q3\bin\arm-none-eabi-g++.exe -flto=3 -mcpu=cortex-m7 -mthumb -ggdb -ffunction-sections -fdata-sections -std=gnu++14 -fno-rtti wolf.o hornet.o main.o syscalls.o -o application.elf
lto-wrapper.exe: fatal error: CreateProcess: No such file or directory
compilation terminated.
c:/users/sesa404214/appdata/local/gcc_arm/8.3.1-2019q3/bin/../lib/gcc/arm-none-eabi/8.3.1/../../../../arm-none-eabi/bin/ld.exe: error: lto-wrapper failed
collect2.exe: error: ld returned 1 exit status

This example doesn't manifest the case with -flto -fuse-linker-plugin, which I believe requires some archive to be linked (that's the case of the project I working on at the moment). I'll see if I can prepare an example for such case as well.

I can't find the rule where use of -fuse-linker-plugin causes the lto-wrapper.exe problem. Interestingly, I know build all with exceptions disabled...

Reproduced with -flto=3. Looks like a LTO bug on windows

At the same time please avoid -flto=n on Windows to workaround. -flto=n needs an installed make, which is easily to mess up on Windows.

-flto should just give you the same result with a bit more compile time.

Further analysis suggests that missing make.exe in PATH is the root cause of the reported error message. As make.exe is not part of GCC, but is needed in -flto=n, user must make sure that make.exe be accessible in PATH.

Ich bin bis 28.10. nicht im Haus und kann Ihre Nachricht daher leider nicht bearbeiten. In dringenden Fällen wenden Sie sich bitte an <email address hidden> bzw. für technische Fragen an <email address hidden>.

I am out of office until October 28th and won't be able to read your message. In urgent cases, please refer to <email address hidden> or for technical questions to <email address hidden>.

Mit freundlichen Grüßen / Best regards

Steffen Wolfer

--
Dipl.-Inform. Steffen Wolfer
Software Engineer Embedded Systems

WEISS ROBOTICS GmbH & Co. KG
Karl-Heinrich-Käferle-Str. 8
D-71640 Ludwigsburg, Germany

Phone: +49 7141 94702-22
Fax: +49 7141 94702-99
http://www.weiss-robotics.com

Sitz der Gesellschaft: Ludwigsburg
Registergericht Stuttgart, HRA725006

Pers. haftende Gesellschafterin:
Weiss Robotics Verwaltungs-GmbH, Sitz Ludwigsburg
Registergericht Stuttgart, HRB73310
Geschäftsführer: Dr. Karsten Weiß

Further analysis suggests that missing make.exe in PATH is the root
cause of the reported error message. As make.exe is not part of GCC, but
is needed in -flto=n, user must make sure that make.exe be accessible in
PATH.

** Changed in: gcc-arm-embedded
       Status: Confirmed => Invalid

--
You received this bug notification because you are subscribed to GNU Arm
Embedded Toolchain.
Matching subscriptions: Älles
https://bugs.launchpad.net/bugs/1689294

Title:
  lto-wrapper.exe: fatal error: CreateProcess: No such file or directory

Status in GNU Arm Embedded Toolchain:
  Invalid

Bug description:
  On Windows the GCC 6.3.1-2017q1 has a problem with `lto-wrapper.exe`
  if the compiler `bin` directory IS NOT in the PATH environment
  variable.

  However this is unacceptable on our build server were we build
  branches with different compiler versions. We didn't have similar
  problems with any form 5.x series, but on the branch for 6.3.1 we
  enabled LTO, that seem to work in general in manual builds.

  It doesn't matter if the bundle in located in the directory which path
  contains spaces or doesn't. It seems that while the long standing bug
  for MinGW has been fixed for all other sub-tools, it's still present
  for `lto-wrapper`.

  Is there any workaround know to make it work without adding the
  compiler directory into PATH?

To manage notifications about this bug go to:
https://bugs.launchpad.net/gcc-arm-embedded/+bug/1689294/+subscriptions

My project with -fno-exceptions and -flto (so, not parallel & no plugin) still fails with the same error, however I can't make minimal example failing.

I added locations of make into PATH enviromnent variable, but that doesn't help.

Is there any diagnosis I can run to figure out what's wrong?

Is it really "Invalid" as the error isn't informative regarding the problem? It's very hard to guess that's missing make problem (not an issue on Linux system, quite a trouble on Windows).

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=89183
Is this bug fix included in 8.3.1-2019q3?

Ich bin bis 28.10. nicht im Haus und kann Ihre Nachricht daher leider nicht bearbeiten. In dringenden Fällen wenden Sie sich bitte an <email address hidden> bzw. für technische Fragen an <email address hidden>.

I am out of office until October 28th and won't be able to read your message. In urgent cases, please refer to <email address hidden> or for technical questions to <email address hidden>.

Mit freundlichen Grüßen / Best regards

Steffen Wolfer

--
Dipl.-Inform. Steffen Wolfer
Software Engineer Embedded Systems

WEISS ROBOTICS GmbH & Co. KG
Karl-Heinrich-Käferle-Str. 8
D-71640 Ludwigsburg, Germany

Phone: +49 7141 94702-22
Fax: +49 7141 94702-99
http://www.weiss-robotics.com

Sitz der Gesellschaft: Ludwigsburg
Registergericht Stuttgart, HRA725006

Pers. haftende Gesellschafterin:
Weiss Robotics Verwaltungs-GmbH, Sitz Ludwigsburg
Registergericht Stuttgart, HRB73310
Geschäftsführer: Dr. Karsten Weiß

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=89183
Is this bug fix included in 8.3.1-2019q3?

** Changed in: gcc-arm-embedded
       Status: Invalid => New

--
You received this bug notification because you are subscribed to GNU Arm
Embedded Toolchain.
Matching subscriptions: Älles
https://bugs.launchpad.net/bugs/1689294

Title:
  lto-wrapper.exe: fatal error: CreateProcess: No such file or directory

Status in GNU Arm Embedded Toolchain:
  New

Bug description:
  On Windows the GCC 6.3.1-2017q1 has a problem with `lto-wrapper.exe`
  if the compiler `bin` directory IS NOT in the PATH environment
  variable.

  However this is unacceptable on our build server were we build
  branches with different compiler versions. We didn't have similar
  problems with any form 5.x series, but on the branch for 6.3.1 we
  enabled LTO, that seem to work in general in manual builds.

  It doesn't matter if the bundle in located in the directory which path
  contains spaces or doesn't. It seems that while the long standing bug
  for MinGW has been fixed for all other sub-tools, it's still present
  for `lto-wrapper`.

  Is there any workaround know to make it work without adding the
  compiler directory into PATH?

To manage notifications about this bug go to:
https://bugs.launchpad.net/gcc-arm-embedded/+bug/1689294/+subscriptions

Michal,

Now I cannot reproduce as your claimed error with flto and fno-exceptions. Taking your case with following build.bat, it run successfully on my Windows box:
------------
path c:\windows;c:\windows\system32
set GXX="c:\Program Files (x86)\GNU Tools ARM Embedded\8 2019-q3-update\bin\arm-none-eabi-g++.exe"
set CXXFLAGS=-ggdb -ffunction-sections -fdata-sections -std=gnu++14 -fno-rtti -fno-exceptions

%GXX% -flto -mcpu=cortex-m7 -mthumb %CXXFLAGS% -c wolf.cpp -o wolf.o
%GXX% -flto -mcpu=cortex-m7 -mthumb %CXXFLAGS% -c hornet.cpp -o hornet.o
%GXX% -flto -mcpu=cortex-m7 -mthumb %CXXFLAGS% -c syscalls.cpp -o syscalls.o
%GXX% -flto -mcpu=cortex-m7 -mthumb %CXXFLAGS% -c main.cpp -o main.o
%GXX% -flto -mcpu=cortex-m7 -mthumb %CXXFLAGS% wolf.o hornet.o main.o syscalls.o -o application.elf
------------
What did I miss?

Yes, that's going to be hard. The problem exists on my proprietary project I cannot publish from obvious reasons. It's big, it's several libraries + main code, I can't figure out which library triggers lto-wrapper.exe at the final stage. A variant with exceptions works on one build server, but fails on another with the same error.

Moreover, removing -flto from my main project, the one that links others (build with -flto) still calls for lto-wrapper.exe.

I tried to recreate problem in an example code, but so far no luck. The example build even with extra stuff in it. I'll be trying, because I need working solution with -flto for all.

I have not idea why adding locations of make.exe and the compiler into PATH doesn't solve the problem. I suspect there are some nested calls to gcc and/or something else. Is there any extra environment variable I should have set???

Can you answer this question:
- Is a bug fix for https://gcc.gnu.org/bugzilla/show_bug.cgi?id=89183 included in 8.3.1-2019q3?

Michal,

I am running out of intelligence on this issue now.

89183 has been fixed in 8.3.1-2019q3. The attached case passed with that version.

> I am running out of intelligence on this issue now.

Is there any debugging/logging I can enable to have better insight why my main project still calls for lto-wrapper?

> I am running out of intelligence on this issue now.

Is there any debugging/logging I can enable to have better insight why my main project still calls for lto-wrapper?

Some hints to the overall problem:
process_begin: CreateProcess(NULL, sh.exe -c "/bin/arm-none-eabi-g++.exe
process_begin: CreateProcess(NULL, echo ../../../../COMPONENTS/Common/Crc.cpp, ...) failed.

You know, my build environment doesn't have typical Unix shell tools in the PATH...

Things getting worse when I add Git's usr/bin into PATH:
/usr/bin/sh: /bin/arm-none-eabi-g++.exe: No such file or directory

OK, the last two lines of previous comment are my fault - I fixed it.

The enclosed demo returns to -flto=n, but with make.exe and echo.exe still accessible from PATH environment variable, and the result is:

The system cannot find the path specified.
make: [C:\Users\SESA40~1\AppData\Local\Temp\cc37Et9s.ltrans0.ltrans.o] Error 1 (ignored)

Adding following to the PATH:
%LOCALAPPDATA%\GCC_ARM\8.3.1-2019q3\bin;;%LOCALAPPDATA%\GCC_ARM\8.3.1-2019q3\arm-none-eabi\bin;%LOCALAPPDATA%\GCC_ARM\8.3.1-2019q3\lib\gcc\arm-none-eabi\8.3.1\;C:\Program Files\Git\usr\bin"
results in:
make: [C:\Users\SESA40~1\AppData\Local\Temp\cc0feLNJ.ltrans0.ltrans.o] Error 1 (ignored)

So, appears like different issue than the one I'm trying to beat.

I tried to debug to find why lto-wrapper fails on process creation, but with no luck - I don't have proper tools for the job on Windows.

try gcc options -save-temps -v, the former keeps all temp files, the later prints all tools componenents.

Nothing conclusive. The -v only display what GCC invokes, but verbosity is not passed down to ld and lto-wrapper.

Interesting observation: the example build with -save-temps doesn't produce the error:
make: [C:\Users\SESA40~1\AppData\Local\Temp\cc0feLNJ.ltrans0.ltrans.o] Error 1 (ignored)

while removal of that option from last line, the problem returns.

Any ideas has to pass any verbosity options down to lto-wrapper.exe?

I managed to rebuild the toolchain in Windows Subsystem for Linux on Ubuntu 16.04.

Then I added some debug logs in `lto-wrapper.c` before any `fork_execute()` and one `collect_execute()` and that's what I've got in my debug log file:

* DEBUG: fork_execute C:\Users\MExxxx\AppData\Local/GCC_ARM/8.3.1-2019q3/bin/arm-none-eabi-g++.exe
* DEBUG: fork_execute C:\Users\MExxxx\AppData\Local/GCC_ARM/8.3.1-2019q3/bin/arm-none-eabi-g++.exe

Mix of slashes and backslashes worries me, but none of the above seem to be the offender causing `CreateProcess()` to fail.