Optimize determine_rpath_dirs #17815

Job	Run time
Arch / PyPy	21m 41s
Arch Linux	20m 48s
Ubuntu Rolling (gcc, g++)	21m 25s
CUDA (on Arch)	7m 9s
Ubuntu Rolling (--unity=on -Ddefault_library=static, --no-unittests, gcc, g++)	15m 48s
Fedora	20m 10s
Ubuntu Rolling (-Ddefault_library=both, --no-unittests, gcc, g++)	16m 14s
Gentoo	25m 37s
Ubuntu Rolling (clang, clang++)	25m 17s
OpenSUSE	18m 22s
Ubuntu Rolling (--unity=on, --no-unittests, clang, clang++)	18m 10s
Ubuntu Rolling (--cross ubuntu-armhf.json --cross linux-mingw-w64-64bit.json, 0, gcc, g++)	14m 31s
Ubuntu Bionic	15m 29s
Ubuntu Rolling (--cross ubuntu-armhf.json --cross linux-mingw-w64-64bit.json, --unity=on, gcc, g++)	14m 23s
	4h 15m 4s

Provide feedback