I'm making this thread with a couple a questions which I hope could lead to a discussion with people whose knowledge in Azure or cloud service for big data in this site. so, the background is I want to make a system for my client which would make it possible for them to collect data from their products in one big pot in the cloud side. now, this system would be implemented on Azure with this kind of capability as target :
- the data which has been collected will be separated per department (R&D, sales, HR, Production, and on). in principle a department would only have full authorization to read & write their own department's space and data but in some case, they could also have the authority to access specific data from another departments space after they got the permission from the department's admin
- the collected data access tiers would be changed with rules after the latest date of it was accessed, but also possible for the user to change it manually
- the data lake for this system would have enough security for highly classified data and personal information.
so after reading some book and information about azure, I'm thinking an architecture which looks like this:
in this architecture, I'm thinking 3 steps are available for the service:
- data input : data from another service or local server would be collected into azure. if the data size big and my client network capacity is not enough to handle it I'm thinking to use Data Box to upload the data. while if my client network has to capacity to upload all data that they have, I would use Data Factory. while for small size data, they could upload it directly from Storage Explorer. and of course, Database Migration to migrate their DB to azure.
- data storage : I'm using Azure Storage for non-relational data with a file like CSV and log mainly, and SQL database for their relational DB. with Azure storage I could also set rules to change my file access tiers automatically.
- data access : my client will access this system through their PC, with A web apps hosted on App Sevices as their UI. to access into this UI they will first do an authentication process with their ADID and from there, my system would get the user department and only showing each user access-allowed space and files. here is where the first target function will be achieved. user will also have some access request function on this UI to ask for permission to another dept space or files. and user with Admin privilege would be able to accept or decline this request. not only that, but admin would also have the privilege to change each files access tiers manually.
now, these are the questions that I want to discuss :
- is this architecture appropriate for big data service and utilization?if not, what kind of thing that I am missing?
- for most of the data access, I'm mainly thinking to build an app from scratch. but is there any service in Azure, or that could be integrated to Azure to do this kind of function?
- what kind of security that usually you set for this kind of service with highly confidential and personal information data?
- is it possible for azure to calculate each departments member access so we could know which department is highly using the services?
bests,