Facing 429 Error Now A Days

[AcharyaTejaswi963]

Hi,

I am trying to integrate Hubot with MS Teams and done with that stuff by following some useful links that are available.I have built an application using hubot framework and was uploaded into a channel previously a few months ago and worked at that moment.So I have uploaded the same into some other channel recently and facing an error as mentioned below:
ERROR Error: POST to 'https://smba.trafficmanager.net/amer/v3/conversations/19%3A6e011df0be3a4eb7934018bcf460370f%40thread...' failed: [429] "

at Request._callback (/opt/hip_bot/node_modules/botbuilder/lib/bots/ChatConnector.js:675:46)

at Request.self.callback (/opt/hip_bot/node_modules/request/request.js:185:22)

at Request.emit (events.js:210:5)

at Request.<anonymous> (/opt/hip_bot/node_modules/request/request.js:1161:10)

at Request.emit (events.js:210:5)

at IncomingMessage.<anonymous> (/opt/hip_bot/node_modules/request/request.js:1083:12)

at Object.onceWrapper (events.js:299:28)

at IncomingMessage.emit (events.js:215:7)

at endReadableNT (_stream_readable.js:1184:12)

at processTicksAndRejections (internal/process/task_queues.js:80:21)

In the recent days when I am trying to run the same app it is throwing the above mentioned error and as per my guess form the last line of the error it is found to be an 429 Error (Too Many Requests) and could anyone please help me on sorting this issue.

Thanks,

Acharya

[#2]

@AcharyaTejaswi963 , Could you please send me the logs of request/messages sent before the 429 error occurs? 

[#3]

@Trinetra-MSFT ,

Thanks for your response.Please find the below logs:

[Thu Jan 16 2020 12:02:24 GMT+0000 (Coordinated Universal Time)] INFO @User indurthy, acharya tejaswi requested dialog incident creation
[Thu Jan 16 2020 12:02:24 GMT+0000 (Coordinated Universal Time)] INFO Channel/Message ID is: 19:6e011df0be3a4eb7934018bcf460370f@thread.skype;messageid=1579176143832
[Thu Jan 16 2020 12:02:24 GMT+0000 (Coordinated Universal Time)] INFO User room: 19:6e011df0be3a4eb7934018bcf460370f@thread.skypeName passed: indurthy, acharya tejaswi
[Thu Jan 16 2020 12:02:24 GMT+0000 (Coordinated Universal Time)] INFO hubot-botframework-adapter: send
[Thu Jan 16 2020 12:02:24 GMT+0000 (Coordinated Universal Time)] INFO hubot-botframework-adapter: reply
[Thu Jan 16 2020 12:02:24 GMT+0000 (Coordinated Universal Time)] INFO hubot-botframework-middleware: creating middleware...
[Thu Jan 16 2020 12:02:24 GMT+0000 (Coordinated Universal Time)] INFO hubot-msteams: Restricting tenants to [\"60ae5b28-c69f-4658-85bf-98ac6db337fa\"]
[Thu Jan 16 2020 12:02:24 GMT+0000 (Coordinated Universal Time)] INFO hubot-msteams: toSendable
[Thu Jan 16 2020 12:02:24 GMT+0000 (Coordinated Universal Time)] INFO hubot-botframework-adapter: send
[Thu Jan 16 2020 12:02:24 GMT+0000 (Coordinated Universal Time)] INFO hubot-botframework-adapter: reply
[Thu Jan 16 2020 12:02:24 GMT+0000 (Coordinated Universal Time)] INFO hubot-botframework-middleware: creating middleware...
[Thu Jan 16 2020 12:02:24 GMT+0000 (Coordinated Universal Time)] INFO hubot-msteams: Restricting tenants to [\"60ae5b28-c69f-4658-85bf-98ac6db337fa\"]
[Thu Jan 16 2020 12:02:24 GMT+0000 (Coordinated Universal Time)] INFO hubot-msteams: toSendable
[Thu Jan 16 2020 12:02:24 GMT+0000 (Coordinated Universal Time)] INFO hubot-botframework-adapter: send
[Thu Jan 16 2020 12:02:24 GMT+0000 (Coordinated Universal Time)] INFO hubot-botframework-adapter: reply
[Thu Jan 16 2020 12:02:24 GMT+0000 (Coordinated Universal Time)] INFO hubot-botframework-middleware: creating middleware...
[Thu Jan 16 2020 12:02:24 GMT+0000 (Coordinated Universal Time)] INFO hubot-msteams: Restricting tenants to [\"60ae5b28-c69f-4658-85bf-98ac6db337fa\"]
[Thu Jan 16 2020 12:02:24 GMT+0000 (Coordinated Universal Time)] INFO hubot-msteams: toSendable
[Thu Jan 16 2020 12:02:24 GMT+0000 (Coordinated Universal Time)] INFO hubot-botframework-adapter: send
[Thu Jan 16 2020 12:02:24 GMT+0000 (Coordinated Universal Time)] INFO hubot-botframework-adapter: reply
[Thu Jan 16 2020 12:02:24 GMT+0000 (Coordinated Universal Time)] INFO hubot-botframework-middleware: creating middleware...
[Thu Jan 16 2020 12:02:24 GMT+0000 (Coordinated Universal Time)] INFO hubot-msteams: Restricting tenants to [\"60ae5b28-c69f-4658-85bf-98ac6db337fa\"]
[Thu Jan 16 2020 12:02:24 GMT+0000 (Coordinated Universal Time)] INFO hubot-msteams: toSendable

After this I am getting the following error as I have mentioned in the previous message:
[Thu Jan 16 2020 12:02:24 GMT+0000 (Coordinated Universal Time)] ERROR Error: POST to 'https://smba.trafficmanager.net/amer/v3/conversations/19%3A6e011df0be3a4eb7934018bcf460370f%40thread...' failed: [429]
at Request._callback (/opt/hip_bot/node_modules/botbuilder/lib/bots/ChatConnector.js:675:46)
at Request.self.callback (/opt/hip_bot/node_modules/request/request.js:185:22)
at Request.emit (events.js:210:5)
at Request.<anonymous> (/opt/hip_bot/node_modules/request/request.js:1161:10)
at Request.emit (events.js:210:5)
at IncomingMessage.<anonymous> (/opt/hip_bot/node_modules/request/request.js:1083:12)
at IncomingMessage.emit (events.js:215:7)
at endReadableNT (_stream_readable.js:1184:12)
at processTicksAndRejections (internal/process/task_queues.js:80:21)
[Thu Jan 16 2020 12:05:05 GMT+0000 (Coordinated Universal Time)] INFO hubot-botframework-adapter: onBotEvents
[Thu Jan 16 2020 12:05:05 GMT+0000 (Coordinated Universal Time)] INFO hubot-botframework-adapter: Handling activity Channel: msteams; type: message

As per the my requirement the functionality what I intended to do is working fine within my local environment.But facing the error that I have mentioned in the previous message when trying to run within the ms teams.

I have shared screenshots of the sequence of question & answers that are working in my local environment.Please find the attached images and let me know if you need any further info from my end.

[#4]

@AcharyaTejaswi963 , Thanks for the logs but it is very hard to read, Could you please send us the logs in a file so we can investigate it further. 

[#5]

Hi @Trinetra-MSFT,

I have appended the required logs into a file as you requested and please find the attached document of it.let me know if you need any further info.

Thanks,

Acharya

[#6]

Hi Trinetra,

Can you please make the process of your investigations a bit fast as we need this issue to be fixed as soon as possible.

Thanks,
Acharya

[#7]

@AcharyaTejaswi963 , Thanks for sharing the information, We are looking into your issue. in the mean time could you please tell us how many messages were send to  bot which got reply before 429, by seeing the logs it seems there were 8 messages sent, Hubot rate limit should be causing the failure of message. We are investigating the cause of failure.   

[#8 · Accepted solution]

@AcharyaTejaswi963 , We found the bot is throttling the rate limit for teams channel which is 7 (max allowed operations) within 1 sec. Please take a look at per bot thread limit.

[#9]

@Trinetra-MSFT, Can we increase this rate-limit to 10 or above as per my requirement.

[#10]

@AcharyaTejaswi963, Why do you need to increase the rate limit? You can handle the rate limit at your end.