Rank 1: Thread
This issue have been plagueing our deployments for some time now but since we switch back and forth from TF to pytorch in between each build, we never made the connection until now that the reason some of our builds are stuck forever is because of pytorch.
More specifially, if we use this in our requirements.txt:
transformers[tf-cpu], the build complete as expected in about 6 minutes.
But the with same exact function but with "transformers[torch"]:
- the log says the build is finished after about the same time and there seems to be no particular error that we can see (first screenshot)
- however, the function never goes into active state and is stuck on "building" forever (second screenshot)
- in our deployment list, it's marked as "Active" even though its size is 0 and it can't be used since when you try to use it, it says "Build with the requested ID is building and not ready for execution"
Why might this happen? The log is quite unhelpful and we don't know how to troubleshoot this further.