Transcription pipeline reliability improvements 🚨
The main goal for this is to reduce the number of stuck files.
In the past, files would appear to be stuck because (1) we had very long timeouts, and (2) when there was a crash the files would not fail until the timeout.
Now, when there is a crash that cannot be recovered by a retry we just mark it as failed. We timeout the file based on the file length and the file size. And much earlier!
We automatically retry transcribing the file after a timeout, and the time increases on every attempt just in case