Transcription capability isn't very good

%3CLINGO-SUB%20id%3D%22lingo-sub-87071%22%20slang%3D%22en-US%22%3ETranscription%20capability%20isn't%20very%20good%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-87071%22%20slang%3D%22en-US%22%3E%3CP%3EHi%20-%20The%20new%20features%20of%20Stream%20are%20cool%2C%20but%20so%20far%20we've%20found%20the%20transcription%2Fclosed-captioning%20services%20in%20Stream%20are%20not%20good%20(specifically%20poor%20accuracy)%20versus%20some%20other%20tools%20we%20use%20for%20this%20task.%3C%2FP%3E%3CP%3EIs%20there%20any%20work%20being%20done%20to%20improve%26nbsp%3Bin%20this%20area%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-166382%22%20slang%3D%22en-US%22%3ERe%3A%20Transcription%20capability%20isn't%20very%20good%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-166382%22%20slang%3D%22en-US%22%3E%3CP%3EHi%20Trevor%20-%3C%2FP%3E%0A%3CP%3EWe%20are%20absolutely%26nbsp%3Bplanning%20to%20improve%20the%20auto-generated%20captions%20as%20much%20as%20we%20can!%26nbsp%3B%20Sorry%20to%20hear%20that%20you%20are%26nbsp%3Bhaving%20to%20go%20through%20the%20trouble%20of%20editing%20your%20captions%20to%20provide%20value.%26nbsp%3B%20We%20hope%20to%20decrease%20our%20error%20rate%20considerably%20in%20the%20coming%20year%20as%20we%20upgrade%20the%20underlying%20speech%20infrastructure.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-166211%22%20slang%3D%22en-US%22%3ERe%3A%20Transcription%20capability%20isn't%20very%20good%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-166211%22%20slang%3D%22en-US%22%3E%3CP%3EHi%2C%20are%20they%20any%20plans%20to%20improve%20on%20the%20auto-generated%20transcription%20accuracy%3F%20We%20have%20tried%20it%20a%20couple%20of%20times%20and%20it%20isn't%20really%20anywhere%20close%20to%20what%20is%20actually%20being%20said%20(without%20any%20background%20music%20or%20noise).%20This%20means%20we%20would%20have%20to%20spend%20a%20lot%20of%20time%20editing%20the%20transcription%20for%20it%20to%20be%20of%20any%20benefit.%20This%20means%20we%20just%20don't%20use%20it.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-90970%22%20slang%3D%22en-US%22%3ERe%3A%20Transcription%20capability%20isn't%20very%20good%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-90970%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F2628%22%20target%3D%22_blank%22%3E%40Adrian%20Hyde%3C%2FA%3E%3A%3C%2FP%3E%0A%3CP%3EFortunately%2C%20I%26nbsp%3Bam%20the%20Azure%20Media%20Services%20contact%20for%20Speech-to-text%20%3A)%3C%2Fimg%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EI%20would%20hesitate%26nbsp%3Bmake%20any%20assumptions%20on%20quality%20without%20testing%20on%20identical%20content%2C%20as%20there%20are%20many%20subtle%20variables%20that%20can%20lead%20to%20low%20quality%20transcription.%20%26nbsp%3BStream%20should%20have%20transcript%20quality%20that%20is%20at-par%20with%20other%20Microsoft%20services%20utilizing%20speech-to-text.%20%26nbsp%3BNote%20that%20Stream%26nbsp%3Bshows%20the%20unedited%20automatic%20transcript.%20%26nbsp%3BWe%20are%20currently%20building%20a%20feature%20that%20will%20allow%20a%20user%20to%20edit%20the%20automatic%20transcript%20to%20fix%20any%20errors.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ERe%3A%203PlayMedia%2C%20this%20service%20utilizes%20human%26nbsp%3Bediting%20in%20addition%20to%20automatic%20transcript%20generation.%20%26nbsp%3BA%20more%20fair%20comparison%20would%20be%20to%20take%20the%20output%20of%20our%20automatic%20transcript%20generation%20and%20send%20it%20to%20a%20human%20transcript%20editor%20to%20correct%20the%20transcript%20prior%20to%20publishing.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-90163%22%20slang%3D%22en-US%22%3ERe%3A%20Transcription%20capability%20isn't%20very%20good%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-90163%22%20slang%3D%22en-US%22%3E%3CP%3EHey%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F63623%22%20target%3D%22_blank%22%3E%40Adarsh%20Solanki%3C%2FA%3E%20-%20For%20Skype%20Broadcast%20versus%20Stream%2C%20I%20can%20only%20provide%20anecdotal%20evidence%20that%20one%20works%20better%20than%20the%20other.%26nbsp%3B%20We%20have%20not%20done%20the%20same%20media%20through%20both%20and%20compared.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EThe%20one%20area%20however%20we%20do%20see%20a%20significant%20difference%20is%20between%20the%20transcoding%20between%20Stream%20and%20a%203rd-party%20we%20have%20typically%20used%20in%20the%20past%20(3PlayMedia)%20for%20this%20function.%26nbsp%3B%20We%20have%20run%20several%20videos%20through%20both%20and%20find%203PlayMedia%20much%20more%20accurate.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI'm%20open%20to%20suggestions%20on%20how%20we%20could%20improve%20this....should%20I%20take%20this%20up%20with%20the%20Azure%20Media%20folks%3F%26nbsp%3B%20Or%20are%20there%20some%20settings%20within%20Stream%20we%20can%20tweak%20to%20see%20if%20performance%20can%20be%20improved%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-89969%22%20slang%3D%22en-US%22%3ERe%3A%20Transcription%20capability%20isn't%20very%20good%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-89969%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F2628%22%20target%3D%22_blank%22%3E%40Adrian%20Hyde%3C%2FA%3E%2C%20I%20am%20sorry%20to%20hear%20that%20you%20are%20unsatisfied%20with%20the%20transcription%20quality%20%3A(%3C%2Fimg%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EYou%20are%20correct%2C%20the%20Skype%20Broadcast%20solution%20is%26nbsp%3Butilizing%20the%20same%20core%20technology%20--%26nbsp%3Bwould%20you%20be%20able%20to%20share%20examples%20of%20the%20discrepancy%20in%20output%3F%20%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EWe%20are%20aware%20of%20the%20issues%20that%20we%20have%20with%20background%20noise%2Fmusic%2C%20and%20unfortunately%20there%20is%20nothing%20we%20can%26nbsp%3Bdo%20in%20the%20short-term%20to%20fix%20this.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-88390%22%20slang%3D%22en-US%22%3ERe%3A%20Transcription%20capability%20isn't%20very%20good%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-88390%22%20slang%3D%22en-US%22%3EWe%20also%20use%20Skype%20Broadcast%2C%20and%20find%20the%20transcoding%20capability%20with%20it%20is%20fairly%20good.%20I%20thought%20that%20Stream%20and%20Broadcast%20was%20using%20the%20same%20technology%20under-the-covers%2C%20however%20given%20the%20results%20we%20see%20I%20guess%20that%20isn't%20the%20case.%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-87242%22%20slang%3D%22en-US%22%3ERe%3A%20Transcription%20capability%20isn't%20very%20good%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-87242%22%20slang%3D%22en-US%22%3EAgreed.%20Finding%20that%20videos%20that%20contain%20background%20music%20don't%20translate.%3C%2FLINGO-BODY%3E
Regular Contributor

Hi - The new features of Stream are cool, but so far we've found the transcription/closed-captioning services in Stream are not good (specifically poor accuracy) versus some other tools we use for this task.

Is there any work being done to improve in this area?

7 Replies
Agreed. Finding that videos that contain background music don't translate.
We also use Skype Broadcast, and find the transcoding capability with it is fairly good. I thought that Stream and Broadcast was using the same technology under-the-covers, however given the results we see I guess that isn't the case.

@Adrian Hyde, I am sorry to hear that you are unsatisfied with the transcription quality :( 

 

You are correct, the Skype Broadcast solution is utilizing the same core technology -- would you be able to share examples of the discrepancy in output?  

 

We are aware of the issues that we have with background noise/music, and unfortunately there is nothing we can do in the short-term to fix this.

Hey @Adarsh Solanki - For Skype Broadcast versus Stream, I can only provide anecdotal evidence that one works better than the other.  We have not done the same media through both and compared.

 

The one area however we do see a significant difference is between the transcoding between Stream and a 3rd-party we have typically used in the past (3PlayMedia) for this function.  We have run several videos through both and find 3PlayMedia much more accurate.

 

I'm open to suggestions on how we could improve this....should I take this up with the Azure Media folks?  Or are there some settings within Stream we can tweak to see if performance can be improved?

@Adrian Hyde:

Fortunately, I am the Azure Media Services contact for Speech-to-text :) 

 

I would hesitate make any assumptions on quality without testing on identical content, as there are many subtle variables that can lead to low quality transcription.  Stream should have transcript quality that is at-par with other Microsoft services utilizing speech-to-text.  Note that Stream shows the unedited automatic transcript.  We are currently building a feature that will allow a user to edit the automatic transcript to fix any errors.

 

Re: 3PlayMedia, this service utilizes human editing in addition to automatic transcript generation.  A more fair comparison would be to take the output of our automatic transcript generation and send it to a human transcript editor to correct the transcript prior to publishing.

Hi, are they any plans to improve on the auto-generated transcription accuracy? We have tried it a couple of times and it isn't really anywhere close to what is actually being said (without any background music or noise). This means we would have to spend a lot of time editing the transcription for it to be of any benefit. This means we just don't use it.

Hi Trevor -

We are absolutely planning to improve the auto-generated captions as much as we can!  Sorry to hear that you are having to go through the trouble of editing your captions to provide value.  We hope to decrease our error rate considerably in the coming year as we upgrade the underlying speech infrastructure.