Forum Discussion
Solved
Running server-less in azure?
Hello, i have developed a continuous running script/app that integrates two systems, pulling data out of one and posting into another. It is currently written in PowerShell, it has no UI, it just ne...
Interesting scenario, thanks for sharing AndrewX. Does your IDP ever run out of information? As in does the return url ever return an empty result? Is there a concern that if you change to every minute, more than one function could post the same information to the elasticsearch?
Depending on where you see this solution going, you might want to look at pushing this information into the Event Hub. Take this as a basis. Your situation is simpler now but the nice thing is once you are in the cloud you have more flexibility as to how to handle the information.
My suggestion though is to start with what you have and see if you can get it to run in Azure Functions. You will need something to start your function: http call, file drop or timer for example. Then once it is running you could try to just run continuously. My concern is after an unpredictable amount of time the function will stop without a clear reason why so you will need some mechanism to start it again. That is why I like the timer idea.
This is an interesting one though, and I for one would appreciate if you did a post on how you solve this. Cheers - Jeff
Hello AndrewX,
As it is written in powershell, I would first take a look at Powershell Azure Functions. The first step is get your head around structure and gain some familiarity with the platform so I suggest running through Create your first PowerShell before diving in with yours.
In my opinion, continuous is not necessarily a good fit for functions though as I find they work best with a trigger of some sort (even a timer). In other words, get triggered, do some work and then complete. So maybe a rethink of your integration pattern might be a good idea. Continuous implies it has a constant steady stream of work which sounds like it might fit better with Data Factory.
The good news is you have something working now so you can use it while developing a cloud-based solution.
Hello chilberto
Thank you for this reply, this is very helpful. A little more context around the app, is that it is used to stream authentication logs from our IDP into Elasticsearch (equivalent of streaming the Azure Signin Logs from the Azure APIs to Elasticsearch). Each page of data it collects, has a "next url" link, to retrieve the next page of results. Because i am not using specific dates, any simply following the next url, the next url will ALWAYS exist and is infinite.
I could technically modify the script to pull a page of results and store the next url and stop. Then have a schedule trigger of "every minute" to pull the next stream of results starting from the "nexturl". This way it would act more like a start/stop on a recurring trigger. This might fit into the Azure Functions framework better.
Do you see any issue with that?
Interesting scenario, thanks for sharing AndrewX. Does your IDP ever run out of information? As in does the return url ever return an empty result? Is there a concern that if you change to every minute, more than one function could post the same information to the elasticsearch?
Depending on where you see this solution going, you might want to look at pushing this information into the Event Hub. Take this as a basis. Your situation is simpler now but the nice thing is once you are in the cloud you have more flexibility as to how to handle the information.
My suggestion though is to start with what you have and see if you can get it to run in Azure Functions. You will need something to start your function: http call, file drop or timer for example. Then once it is running you could try to just run continuously. My concern is after an unpredictable amount of time the function will stop without a clear reason why so you will need some mechanism to start it again. That is why I like the timer idea.
This is an interesting one though, and I for one would appreciate if you did a post on how you solve this. Cheers - Jeff
- Sometimes the IDP does run out of information, in which case, it simply returns 0 results and the next url doesn't change, so it simply makes the same request in the next iteration and will do over and over again until there is new data and a new next url. The org has enough activity and users, that i have only seen this happen occasionally.
Thanks for all your help.
