`BackgroundDownload.exe` preventing file deletion

[thalassemia]

Description

The Visual Studio BackgroundDownload.exe process is holding a temporary build file open, causing [WinError 32] The process cannot access the file because it is being used by another process to be raised when trying to delete this file in Python.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• macOS 13 Arm64
• Windows Server 2019
• Windows Server 2022

Image version and build link

Version: 20240108.2.0
Build link

Is it regression?

No: found a case of this on 20231217.2.0, https://github.com/scipy/scipy/actions/runs/7466155226/job/20316910737

Expected behavior

BackgroundDownload.exe should not have handles to temporary build files that I create.

Actual behavior

By calling SysInternals' Handle right before trying to delete my temporarily build directory, I confirmed that BackgroundDownload.exe was holding one of my temporary build files open, preventing me from deleting it. This does not happen on every build (see attempts 3 and 4 in the build link) but happens often enough that it should only take a few attempts to confirm (see all other attempts in the build link). See mesonbuild/meson-python#560 for more discussion/linked issues.

Repro steps

1. Fork scipy/scipy and set up a branch at commit c604fdfaebea593434e2b722a55ad1cb2145e673
2. On the forked repo, create a PR with the new branch to run the GitHub Actions workflows
3. Check the output of Windows tests / cp311 (build sdist + wheel), full, no pythran (pull_request) (might take a few tries to see the error)

[sergei-pyshnoi]

Hello @thalassemia . Thanks for your report. We will take a look.

[erik-bershel]

Hey @thalassemia!
After a little investigation, I can say the following:
The problem lies outside our area of responsibility. Interaction between components has always been a very complex and problematic area. In this case, as I see, you have at least three options. Wait for the new release of python3.11 / python3.12, use the workaround described by your colleague with a path change, use the workaround I presented below. You can also report a problem on the Visual Studio developer community portal, but I suspect that they will recommend the same workaround that I did (example).

We cannot disable this service on our end - this may reduce security and also not recommended by Visual Studio support. But you may do it on your own as one of the preparation steps:

jobs:
  service:
    runs-on: windows-2019
    steps:
    - name: Configure BackgroundDownload to be disabled
      run: New-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\VisualStudio\Setup" -Name "BackgroundDownloadDisabled" -Value 1 -PropertyType DWORD -Force

[thalassemia]

Awesome, thanks for looking into this for me!