How do I architect an Azure application?
September 30, 2023 5:39 PM Subscribe
I'm used to AWS and really like the CDK/SAM/modular architecture/ability to generate API endpoints/wrapping it in localstack, etc. Most of all AWS is really well-documented. I'm also really confused about the Azure infrastructure. Do I use the SDK? The Azure Graph API? Basically I need to add users to Teams projects, create folders in Teams "drives," place files on them based on what's in other drives, create calendars, monitor files for changes, etc. I have a bunch of bash scripts that have gotten out of control and other people want to do this. I'd like there to be a Web API of some sort, a "service principal" that monitors and does tasks based on events like what would come out of EventBridge (file changed, monitor what changed, and respond to it). Also, store things in a db of some sort at some point.
I'm not tied to any language/framework for the backend but will probably go with Typescript or Python and for the frontend I care less about organizing as it will just authenticate and call the API (leaning towards WebComponents/Lit with a MaterialUI component library). Thoughts on the backend having some like AWS has? Bonus points if it is well documented?
Ignore this background that explains some idiosyncracies.
So long-winded story is that my IT department is incompetent to the point of being ridiculous. They do things like disable the Azure portal and I can still access what I need to do with Graph Explorer. To even run scripts (?!) I need to get permission from IT, but which ended up being a really great learning event I found out their incompetence helps me as they will get "remove Portal access" and just disable people from seeing it but not executing it. I tried to get admin access and get this properly approved but its near impossible because IT assists any script (Bash, Powershell, let alone apps) has to go through security review, legal review and architecture review. As a result anything automated ends up being done through Excel spreadsheets with VBA as that skirts the rules. It is a mess. So keep this in mind, I'm doing nothing I can't do manually and nothing I'm doing is secure info (HIPPA, PII, etc.).
Okay the real questions
I'm open to anything but I'd love some advice on this. Especially around folder structure and organizing the backend, and some other questions.
1. For this MSAL unless I create a service principal or a shared I need to use the Oauth2.0 "Interactive Login" for the Web App, register the application once through Microsoft Graph (not host it), have the user login which opens up a browser window And eventually gets a token back. Then my actual API will be another registered app with have both a service principal or certificate so it can run daemon-type tasks and also accept requests "on behalf of" so if a user tries to do something I can set permissions and log it, and the cert will be able to basically do anything. Is that right? For the "front-end" app I'll need to run it locally can I have to set the return URI to http://localhost:3030, can that be an environment variable or do I need a register an app per environment? I can't get a clear answer on this. I don't know why their documentation isn't more clear and relies on creating things through the UI.
2. I'm running under the assumption I won't be able to be serverless immediately as those are resources that'll get billed and noticed, the first "test users" will just need to probably run a docker compose or something similar. That said the CDK/SAM seemed to take care of way more than I realized. I'd have a folder structure that would be /src/web-api/lib/lambda/create-channel/(cdk.json, tests, actual lambda code, etc.) that would have it all the cdk/stack/test info on it. Then I'd have /src/web-api/lib/cross-services/create-new-project.ts, that would allow me to share resources between stacks. What's the non-CDK way of doing this especially if I can't use Azure functions and need to run something like docker?
I'm not trying to do like one function per lib with its own server environment just trying to organize this in a similar modular fashion. Like I kind of want to keep it clean and modular.
3. What about the permissions and all the generators for an API gateway that AWS kind of makes it easy to do? Anything I should be looking at? Or is Terraform or docker-compose pretty much it? Really, I'm trying to isolate things and have been spoiled.
4. Events are sent to the EventBridge and assuming I can't use that due to cost if I installed Kafka and the connector locally will that cost?
As you can see I'm trying to keep things in a serverless structure as much as possible, infrastructure as code, and things that are infra (API connector with MSAL authorization, etc.) as easy and light weight as possible.
5. I'm leaning towards TypeScript with Deno and using the Graph endpoint as the MSFT SDKs are kind of a mess, the GoLang SDK even says "this was done by an intern and wasn't done right." I'm assuming the graph client is somewhat reasonable with Apollo or something.
Please ignore the questions if they're way out there, but keep in mind I'd usually hack this together with clean code concepts in mind and as the app got bigger or the team got bigger would move to a more modular architecture. But once I get approval -- somehow, the team beyond my boss will look for any reason to move it to a turnkey solution which will be a crappy Excel spreadsheet or something out of Sharepoint and no one will use it and it'll be back to square one. Plus this is a fun side project to keep my skills up.
The actual programming of this is pretty basic. The hardest part being not everyone will use the app since it'll require docker-compose up to get going or maybe something easier if I can figure it out and host it on a separate platform for free. That and even in a perfect scenario if I had a "person on project spreadsheet" that people would use and a daemon would subscribe to as often some countries have very restrictive firewall and security permissions and even an approved app will be blocked (hence the overreliance on spreadsheets).
I know this is a bit of a horrible set of assumptions but in a way best work sometimes comes out of restrictions and it is a learning experience as I've been focused on getting work done and not infrastructure so I know there's ways *to* do it but hoping there's maybe a better way to do it?
Ignore this background that explains some idiosyncracies.
So long-winded story is that my IT department is incompetent to the point of being ridiculous. They do things like disable the Azure portal and I can still access what I need to do with Graph Explorer. To even run scripts (?!) I need to get permission from IT, but which ended up being a really great learning event I found out their incompetence helps me as they will get "remove Portal access" and just disable people from seeing it but not executing it. I tried to get admin access and get this properly approved but its near impossible because IT assists any script (Bash, Powershell, let alone apps) has to go through security review, legal review and architecture review. As a result anything automated ends up being done through Excel spreadsheets with VBA as that skirts the rules. It is a mess. So keep this in mind, I'm doing nothing I can't do manually and nothing I'm doing is secure info (HIPPA, PII, etc.).
Okay the real questions
I'm open to anything but I'd love some advice on this. Especially around folder structure and organizing the backend, and some other questions.
1. For this MSAL unless I create a service principal or a shared I need to use the Oauth2.0 "Interactive Login" for the Web App, register the application once through Microsoft Graph (not host it), have the user login which opens up a browser window And eventually gets a token back. Then my actual API will be another registered app with have both a service principal or certificate so it can run daemon-type tasks and also accept requests "on behalf of" so if a user tries to do something I can set permissions and log it, and the cert will be able to basically do anything. Is that right? For the "front-end" app I'll need to run it locally can I have to set the return URI to http://localhost:3030, can that be an environment variable or do I need a register an app per environment? I can't get a clear answer on this. I don't know why their documentation isn't more clear and relies on creating things through the UI.
2. I'm running under the assumption I won't be able to be serverless immediately as those are resources that'll get billed and noticed, the first "test users" will just need to probably run a docker compose or something similar. That said the CDK/SAM seemed to take care of way more than I realized. I'd have a folder structure that would be /src/web-api/lib/lambda/create-channel/(cdk.json, tests, actual lambda code, etc.) that would have it all the cdk/stack/test info on it. Then I'd have /src/web-api/lib/cross-services/create-new-project.ts, that would allow me to share resources between stacks. What's the non-CDK way of doing this especially if I can't use Azure functions and need to run something like docker?
I'm not trying to do like one function per lib with its own server environment just trying to organize this in a similar modular fashion. Like I kind of want to keep it clean and modular.
3. What about the permissions and all the generators for an API gateway that AWS kind of makes it easy to do? Anything I should be looking at? Or is Terraform or docker-compose pretty much it? Really, I'm trying to isolate things and have been spoiled.
4. Events are sent to the EventBridge and assuming I can't use that due to cost if I installed Kafka and the connector locally will that cost?
As you can see I'm trying to keep things in a serverless structure as much as possible, infrastructure as code, and things that are infra (API connector with MSAL authorization, etc.) as easy and light weight as possible.
5. I'm leaning towards TypeScript with Deno and using the Graph endpoint as the MSFT SDKs are kind of a mess, the GoLang SDK even says "this was done by an intern and wasn't done right." I'm assuming the graph client is somewhat reasonable with Apollo or something.
Please ignore the questions if they're way out there, but keep in mind I'd usually hack this together with clean code concepts in mind and as the app got bigger or the team got bigger would move to a more modular architecture. But once I get approval -- somehow, the team beyond my boss will look for any reason to move it to a turnkey solution which will be a crappy Excel spreadsheet or something out of Sharepoint and no one will use it and it'll be back to square one. Plus this is a fun side project to keep my skills up.
The actual programming of this is pretty basic. The hardest part being not everyone will use the app since it'll require docker-compose up to get going or maybe something easier if I can figure it out and host it on a separate platform for free. That and even in a perfect scenario if I had a "person on project spreadsheet" that people would use and a daemon would subscribe to as often some countries have very restrictive firewall and security permissions and even an approved app will be blocked (hence the overreliance on spreadsheets).
I know this is a bit of a horrible set of assumptions but in a way best work sometimes comes out of restrictions and it is a learning experience as I've been focused on getting work done and not infrastructure so I know there's ways *to* do it but hoping there's maybe a better way to do it?
1. Azure AD (now called Entra ID) App Registrations can have many, many call back addresses listed.
2. I don't know what CDK means in this context.
3. Terraform is well supported by Azure. I don't use it personally but people adjacent to me use it extensively but it strikes me as a thing that you actively need your IT staff to support your use of it.
4. There are many Azure event queue and service bus solutions but you will be working through getting your people to allow you to use it and pay for it.
5. MSAL is pretty great. I've been happy with it in python and the .Net version (via PowerShell).
6. Each of the SDKs are built independently so they vary is quality and how much TLC they have received. You can expect the python, .Net/C#, JavaScript ones to be most up to date. For end user things I bet the iOS ones also get attention. The good thing about the Graph is that Microsoft does a good job of describing how all the language neutral HTTPS calls work such you can use any language you want that can make web requests if you are willing to put in the work. For example you can do all of this in AWS talking to Azure/O365 if you wanted to.
I think your biggest struggle is fighting against your internal gatekeepers.
posted by mmascolino at 6:53 PM on September 30, 2023 [2 favorites]
2. I don't know what CDK means in this context.
3. Terraform is well supported by Azure. I don't use it personally but people adjacent to me use it extensively but it strikes me as a thing that you actively need your IT staff to support your use of it.
4. There are many Azure event queue and service bus solutions but you will be working through getting your people to allow you to use it and pay for it.
5. MSAL is pretty great. I've been happy with it in python and the .Net version (via PowerShell).
6. Each of the SDKs are built independently so they vary is quality and how much TLC they have received. You can expect the python, .Net/C#, JavaScript ones to be most up to date. For end user things I bet the iOS ones also get attention. The good thing about the Graph is that Microsoft does a good job of describing how all the language neutral HTTPS calls work such you can use any language you want that can make web requests if you are willing to put in the work. For example you can do all of this in AWS talking to Azure/O365 if you wanted to.
I think your biggest struggle is fighting against your internal gatekeepers.
posted by mmascolino at 6:53 PM on September 30, 2023 [2 favorites]
I am an internal gatekeeper of sorts but within a secure system and your request would give me a panic attack at the thought of security, maintenance and compliance. My very first request would be: is there an existing SaaS product you can buy to do this within our current ecosystem? Then it would be can you do this with power platform, microsoft's existing customisation rpa type tool? Then after that it would be okay, who is your system owner, your business sponsor, your budget for the associated work and your onboarding and offboarding plan etc, plus approvals. Azure access to run stuff should be strictly controlled, especially anything touching production data/users. But they almost certainly have a development or UAT system which you might be able to get access to.
Having said that I absolutely get your frustration working with a system where you have some surprising access - bash scripts! - but not enough to do major work.
Microsoft will give you a developer's account with access to tools to fool around in and experiment pretty easily. I would sandbox a prototype of this first on your own individual account and make a proposal.
posted by dorothyisunderwood at 11:01 PM on September 30, 2023 [1 favorite]
Having said that I absolutely get your frustration working with a system where you have some surprising access - bash scripts! - but not enough to do major work.
Microsoft will give you a developer's account with access to tools to fool around in and experiment pretty easily. I would sandbox a prototype of this first on your own individual account and make a proposal.
posted by dorothyisunderwood at 11:01 PM on September 30, 2023 [1 favorite]
Response by poster: @dorothy ironic you should mention that. I’ve worked in secure environments and they’re easier to work in. Part of my motivation is a local environment is a local environment but our security is so screwed up it’s hard to describe. Everyone defaults to Excel and the Simple’s documents they can assuming that’s the one thing everyone in our 100+ countries with conflicting policies have access to. The estimation worksheet with 20 tabs and a lot of complexity in a huge VBA macro should be out of the box software. Of course that would mean getting everyone who needs to use it or get approved so it defaults to what it is now. Because levels/grades are not consistent across business units, software purchased needs IT support, etc so Excel it is. I’d argue at this point things like that are assuredly a program.
It was written by someone who doesn’t use it or isn’t a programmer as it is very fragile. Remove a resource and the spreadsheet breaks and you need to start over again! So there’s tricks: you don’t remove resources you see their bill rate to zero, etc. how do you know it has been completed and approved? Hope you take good notes in meetings! Even finding the latest version is a mess because the naming convention changes, but in a consistent enough way I can guess and prompt the user to check if this is the correct version. If anything my scripts have added consistency and security. Doing this locally, custom and using the graph db to facilitate this seems the best way. Graph seems to be used internally on O365 products and that’s why I seem to have access to do what I do normally in those apps.
Dealing with people across different cultures, countries and business units. People are begging me for scripts I wrote on my own to just get things done but really that’ll create the same problem of a bunch of versioned scripts that are hard to use and same problem we have now only passed around. The reason this fell on me was my reputation for just getting things done correctly (IT has no idea how to help or do anything outside of pushing it off to telling me I created the support ticket wrong but not guiding me how to create it correctly, trust me I tried to do this right).
To give you an idea our designers don’t have access to Adobe products because not everyone has access and have to do all their “designing” in PowerPoint. When you restrict people they find out ways around it: buy Adobe Creative Suite personally and put it on their personal laptop.
Again this is largely an issue I’m taking on myself to learn maybe some gaps in my cloud knowledge. Ideally I’d be given a test data wherein can do this in with test data, thar just won’t happen. I’ve gotten down to “okay I can’t use lambda statements but the serverless model can be recreated locally or through docker correct? Like ignoring my limitations I’m sure some companies have to for whatever reason not use the cloud but still take advantage that things like Lambda functions or spinning up a pub/sub system should abstract me from the underlying infra concerns. So it is kind of fun in that way yeah I might need to mock out data to prove it works but mocking a teams interface itself is hard. You know? And in the end people won’t want to see a test passing that a usernwasnaddednto teams they want to see it “work.”
posted by geoff. at 10:32 AM on October 1, 2023 [1 favorite]
It was written by someone who doesn’t use it or isn’t a programmer as it is very fragile. Remove a resource and the spreadsheet breaks and you need to start over again! So there’s tricks: you don’t remove resources you see their bill rate to zero, etc. how do you know it has been completed and approved? Hope you take good notes in meetings! Even finding the latest version is a mess because the naming convention changes, but in a consistent enough way I can guess and prompt the user to check if this is the correct version. If anything my scripts have added consistency and security. Doing this locally, custom and using the graph db to facilitate this seems the best way. Graph seems to be used internally on O365 products and that’s why I seem to have access to do what I do normally in those apps.
Dealing with people across different cultures, countries and business units. People are begging me for scripts I wrote on my own to just get things done but really that’ll create the same problem of a bunch of versioned scripts that are hard to use and same problem we have now only passed around. The reason this fell on me was my reputation for just getting things done correctly (IT has no idea how to help or do anything outside of pushing it off to telling me I created the support ticket wrong but not guiding me how to create it correctly, trust me I tried to do this right).
To give you an idea our designers don’t have access to Adobe products because not everyone has access and have to do all their “designing” in PowerPoint. When you restrict people they find out ways around it: buy Adobe Creative Suite personally and put it on their personal laptop.
Again this is largely an issue I’m taking on myself to learn maybe some gaps in my cloud knowledge. Ideally I’d be given a test data wherein can do this in with test data, thar just won’t happen. I’ve gotten down to “okay I can’t use lambda statements but the serverless model can be recreated locally or through docker correct? Like ignoring my limitations I’m sure some companies have to for whatever reason not use the cloud but still take advantage that things like Lambda functions or spinning up a pub/sub system should abstract me from the underlying infra concerns. So it is kind of fun in that way yeah I might need to mock out data to prove it works but mocking a teams interface itself is hard. You know? And in the end people won’t want to see a test passing that a usernwasnaddednto teams they want to see it “work.”
posted by geoff. at 10:32 AM on October 1, 2023 [1 favorite]
It sounds like you have a business case that your superiors can take to IT leadership to make this effort the way it really should work.
posted by mmascolino at 3:26 PM on October 1, 2023
posted by mmascolino at 3:26 PM on October 1, 2023
You are not logged in, either login or create an account to post comments
posted by geoff. at 6:20 PM on September 30, 2023