Parallel builds of crate dependencies leads to intermittent linker errors

[tcosprojects]

Summary

While cargo is compiling dependencies I'm seeing intermittent linker errors. It appears the issue is caused by how cargo uses job objects on windows detailed in this issue: rust-lang/rust#33145

Output

  Compiling log v0.3.6
 Documenting winapi v0.2.8
   Compiling rustc-serialize v0.3.19
   Compiling cfg-if v0.1.0
   Compiling utf8-ranges v0.1.3
   Compiling clang-sys v0.7.2
   Compiling strsim v0.5.1
   Compiling clang-sys v0.10.1
   Compiling winapi-build v0.1.1
   Compiling winapi v0.2.8
   Compiling backtrace-sys v0.1.4
   Compiling kernel32-sys v0.2.2
   Compiling dbghelp-sys v0.2.0
error: linking with `link.exe` failed: exit code: 1318
  |
  = note: "C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\BIN\\link.exe" "/NOLOGO" "/NXCOMPAT" "/LARGEADDRESSAWARE" "/SAFESEH" "/LIBPATH:<lots of paths to libs>
  = note: libpanic_unwind-40393716.rlib(panic_unwind-40393716.0.o) : fatal error LNK1318: Unexpected PDB error; RPC (23) '(0x000006BA)'


error: aborting due to previous error

error: linking with `link.exe` failed: exit code: 1318
  |
  = note: "C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\BIN\\link.exe" "/NOLOGO" "/NXCOMPAT" "/LARGEADDRESSAWARE" "/SAFESEH" "/LIBPATH:<lots of paths to libs>
  = note: LINK : fatal error LNK1318: Unexpected PDB error; RPC (23) '(0x000006BA)'


error: aborting due to previous error

Build failed, waiting for other jobs to finish...
Build failed, waiting for other jobs to finish...

System info

Windows 10 x64

rustc 1.12.0 (3191fbae9 2016-09-23)
binary: rustc
commit-hash: 3191fbae9da539442351f883bdabcad0d72efcb6
commit-date: 2016-09-23
host: i686-pc-windows-msvc
release: 1.12.0

rustup 0.6.3 (a0e2132 2016-08-28)

cargo 0.13.0-nightly (109cb7c 2016-08-19)

[alexcrichton]

Thanks for the report! You also mentioned that there are parallel builds of Cargo going in the background as well, which I think is a key ingredient here. In that sense, I think the sequence of steps here are:

• Cargo process A starts
  • This process spawns rustc
  • This process spawns link.exe
  • This process spawns mspdbsrv.exe
• Meanwhile, Cargo process B starts
  • This process spawns rustc, then link.exe
  • This link.exe realizes mspdbsrv.exe is already running, and starts communicating
• Cargo process A finishes (link.exe returns, rustc.exe returns)
• The job object for Cargo process A kills mspdbsrv.exe
• Cargo process B's link.exe doesn't like mspdbsrv.exe going away, so it bails

I think the solution here is to do the same thing as the Rust build system which is to tweak our usage of job objects:

• By default, do the same thing as today, tear down everything
• If Cargo finishes entirely, then we kill all sub-processes that aren't whitelisted
• We whitelist to persist mspdbsrv all the time

That way by default this scenario would work, and otherwise we'd still get the same benefits that if you interrupt things then everything is torn down.

[tcosprojects]

To clarify, I'm using cargo interactively and launching cargo build once. I was under the impression that cargo built dependencies in parallel internally and that was where the conflict was coming from.

[Nuxij]

I'm using the script from this guy's blog to isolate mspdbsrv. http://blog.peter-b.co.uk/2017/02/stop-mspdbsrv-from-breaking-ci-build.html

1. Run an mspdbsrv manually, overriding environment variable
2. Run link.exe (or whatever else, devenv et al)
3. Using python's 'with', mspdbsrv will automatically close, whether we succeed or fail.

[alexcrichton]

@joeasaurus to confirm, is that required to fix this today? In theory this has been fixed in Cargo and shouldn't be an issue for Rust projects.

[Nuxij]

@alexcrichton I don't think so, sorry for the necro post! Didn't look at the dates ;D