Function build with pytorch stuck in "building" state indefinitely despite log saying its finished
This issue have been plagueing our deployments for some time now but since we switch back and forth from TF to pytorch in between each build, we never made the connection until now that the reason some of our builds are stuck forever is because of pytorch.
More specifially, if we use this in our requirements.txt: transformers[tf-cpu], the build complete as expected in about 6 minutes.
But the with same exact function but with "transformers[torch"]:
- the log says the build is finished after about the same time and there seems to be no particular error that we can see (first screenshot)
- however, the function never goes into active state and is stuck on "building" forever (second screenshot)
- in our deployment list, it's marked as "Active" even though its size is 0 and it can't be used since when you try to use it, it says "Build with the requested ID is building and not ready for execution"
Why might this happen? The log is quite unhelpful and we don't know how to troubleshoot this further.
By the way, I know there is this post here for example: https://discord.com/channels/564160730845151244/1278544034117189685 However, in our case, the reason for the build to be stuck is oddly specific, we have tried many times yesterday, and the build is consistently stuck like thiswhen using pytorch, and does almost consistently finish when using tf so we assume our issues is a different one from what we experienced a few weeks ago, or the issue in the post above.
