How do I create full environments on a single Fedora Silverblue machine?
April 28, 2024 12:35 PM Subscribe
This is a bit beyond my expertise with networking and certificate signing. I want to create a template that replicates a prod environment. Keeping this a bit abstract, I want to have a monorepo or polyrepo for dev, a way to test dev, push it into a private git repo, use build actions on pre-commit, then on post-commit run the equivalent of a "GitHub Action" to push the resulting OCI image to a private image repository, then eventually into a K8s cluster. It obviously doesn't have to be accessible outside my machine, but I'd like to add a host entry for each "workspace" using certs and a daemon for the various services to talk to each other. More details inside.
For now my Internet is weak so I have to do my work on a high-end laptop. I'm running Fedora Silverblue but I need to run demos and keep this isolated. So it needs to be as realistic as possible with local hostnames and a daemon providing keys. This is a bit beyond my ops skills so there might be a better way to do it, but I'm okay with whatever toolset as long as I can scaffold this out quickly. Assume the audience is diverse: devs who want to see it run end to end, business users who just want to see their domain in the browser and a feature created in a ticketing system, automated build, showing up in the end environment, etc.
I'm used to obviously using GItHub, Jira, authentication systems for this so that's where I'm a bit confused.
Think of it this way:
1. Just for me to see and setup:
a. Script that lives outside this that scaffolds and be like a Makefile but I want basically: ``cue export inject.cue -t env=projecta.com`
b. It sets up certificates, ssh keys via a daemon, adds the host file entries, etc. I can have multiple of these running at time. Basically while I can explain away the ticketing isn't Jira (or could be if there's an OSS variant), GitHub, DockerHub, etc. I still want it to be like infra.git.project-a.com and look as seamless as possible with SSH, certs etc/
c. There can be multiple of these so on my laptop it could be *.project-b.com too.
3. For devs/PMs
a. A blank "workspace" that'd probably be a mono/poly repo, but essentially an empty directory with nothing but source files. So like someone might be working on a back-end integration piece that uses I don't know a datalake to sync between services, and another that may be a front end dev that goes to design system internal site with some sort of SSO that logs into obviously an internal SSO piece to see a component that needs to be updated, both do the same for the ticketing system and I guess if I wanted to get fancy spin up a VM for each OS with this directory mounted on each file system so someone doesn't go "It just works on Linux" ... they branch create an ephemeral environment they can work on ... tickets get updated, it runs through Earthly on pre-commit for linting, testing etc., then goes through whatever the local git server "GitHub Actions" equivalent is on post-commit, ticket is updated ... sees the OCI image being created, if it passes it gets integrated into K8s. Ticket closed (I guess after review if we're being precise), ephemeral environment shut down.
b. Again a key process of this is a working demo so not only do tickets and things get committed, but workflow things happen like a message through Slack or whatever OSS equivalent I can find.
4. For business stakeholders
a. They know it is a demo but I want to at least have a small feature I can build out to show the process working end-to-end and how much time it saves and everything looking real. Even build something out that fails and doesn't get into the prod K8s cluster.
Before my Internet went to the crapper due to street construction that will never end and me not wanting to lease an office I had a tunnel to cloud flare, basically a server stack rack used Ansible/Vault/etc. to spin up a VM/container with ProxMox with each of these servers and proper networking infrastructure. I'd send them a short recorded demo, let them signup and create an SSO like login based on their corporate domain for verification and they could play around and even reset the whole environment. I had "layered" scripts so if I knew the CTO freaked out over something it was kinda easy for me to just change out scripts. I'd also have the ability for them to check out an individual VDI.
I could do this over AWS/GCP/Azure and call it done but that'd freak someone out even though that goes against the immutable/deterministic/etc. buzzwords that go around. I try to make everything agnostic and using Make/Bash/etc. or fancy new tools like Cue/Timoni/Dagger I can when this goes from demo to reality change things with a few CLI changes.
In any case I'm between a bad DSL and a bad backup 5G hotspot. So setting up something like 'infra.git.project-a.realdomain.com' is hard. I'm also changing my workflow significantly so I need to rebuild all this and I've never had to host fake domains on laptop.
I feel as if I'm really close I just don't have the networking prowess or ability to create systemd files or whatever it takes for SSH keys to pass back and forth for multi-tenant setups essentially on one machine. I also got "lazy" doing this with basically an on-prem cloud and 10gbps both direction fiber connection.
Realistically the apps are just a collection of things to make devs lives more configurable and less about coding. Someone commits a file in Figma, a ticket gets created and a design website (in K8s cluster sure) creates a page with a link and screenshot, a ticket gets created, creating a pull request with an ephemeral environment. Messages get sent out to Teams/Slack too. Collaboration is improved, dev environment is converted into something K8s compatible. Usually it is a simple GoLang app/Typescript front end. A lot of the "hard stuff" is talking between systems I have no control over (Salesforce) to extract data import it in. I have that down to a science too. Features go out and are deployed to x% of users and as KPIs are met (not 500 errors, but say conversion KPIs that might be created by ML ha!). It get gets deployed to more and more users. A lot of it is monitoring as nearly all corporate customers use locally-hosted (or at least hosted on EC2 not OpenAI), models. Data scientists want to see which models work "best," by deploying a dozen variations, other KPIs are tracked if say UI is updated (which AI does too sometimes) so really its a simple system with complex moving parts and people have been so burned they need to see it look like it would work in reality.
So a lot of these strange requests come from well it looks like that in a demo.
Sorry to drop buzz words and another long explanation but I thought you'd see why I have strange requirements. Keep in mind these aren't the most tech oriented people I'm talking to and can barely embrace the cloud so if they see something that would seem obvious to you or I as being simple or wouldn't matter (we can just move to Azure if he's hosting the demo on AWS), get projects killed. Or even having the name of the company in the browser is a huge, huge deal.
There's just a lot here I'm not used to dealing with, and I found out the hard way in the Linux world configuration and automation can take over your life. Oddly the "ops" part is not my job but helping them improve migrating to a new "digital platform" (read: talk to executives that the old way of buying into the Oracle world for everything isn't feasible). I also think it'd be kinda cool if I could have pretty much an end-to-end production scenario on one machine.
Looking for any advice as to how to get this done, and I'm in the mindset that developers have a laptop with whatever (in this case VM + mounted volume), a production K8S cluster, a GitRepo, general Auth/Authentication handled between sites by SSH+Ca-Certs (beyond K8S specific requirements, like logging into an internal app in the cluster), and a segregate network mapped to DNS. I was thinking of separate folders with VMs/Containers for each of those but maybe a multi-cluster Kubernetes is best? Like infrastructure in one cluster, the actual production in another? I honestly don't know what works and what doesn't anymore. If you told me one cluster could hold the images repo/git/build/etc that runs the actual workload I wouldn't be surprised.
For now my Internet is weak so I have to do my work on a high-end laptop. I'm running Fedora Silverblue but I need to run demos and keep this isolated. So it needs to be as realistic as possible with local hostnames and a daemon providing keys. This is a bit beyond my ops skills so there might be a better way to do it, but I'm okay with whatever toolset as long as I can scaffold this out quickly. Assume the audience is diverse: devs who want to see it run end to end, business users who just want to see their domain in the browser and a feature created in a ticketing system, automated build, showing up in the end environment, etc.
I'm used to obviously using GItHub, Jira, authentication systems for this so that's where I'm a bit confused.
Think of it this way:
1. Just for me to see and setup:
a. Script that lives outside this that scaffolds and be like a Makefile but I want basically: ``cue export inject.cue -t env=projecta.com`
b. It sets up certificates, ssh keys via a daemon, adds the host file entries, etc. I can have multiple of these running at time. Basically while I can explain away the ticketing isn't Jira (or could be if there's an OSS variant), GitHub, DockerHub, etc. I still want it to be like infra.git.project-a.com and look as seamless as possible with SSH, certs etc/
c. There can be multiple of these so on my laptop it could be *.project-b.com too.
3. For devs/PMs
a. A blank "workspace" that'd probably be a mono/poly repo, but essentially an empty directory with nothing but source files. So like someone might be working on a back-end integration piece that uses I don't know a datalake to sync between services, and another that may be a front end dev that goes to design system internal site with some sort of SSO that logs into obviously an internal SSO piece to see a component that needs to be updated, both do the same for the ticketing system and I guess if I wanted to get fancy spin up a VM for each OS with this directory mounted on each file system so someone doesn't go "It just works on Linux" ... they branch create an ephemeral environment they can work on ... tickets get updated, it runs through Earthly on pre-commit for linting, testing etc., then goes through whatever the local git server "GitHub Actions" equivalent is on post-commit, ticket is updated ... sees the OCI image being created, if it passes it gets integrated into K8s. Ticket closed (I guess after review if we're being precise), ephemeral environment shut down.
b. Again a key process of this is a working demo so not only do tickets and things get committed, but workflow things happen like a message through Slack or whatever OSS equivalent I can find.
4. For business stakeholders
a. They know it is a demo but I want to at least have a small feature I can build out to show the process working end-to-end and how much time it saves and everything looking real. Even build something out that fails and doesn't get into the prod K8s cluster.
Before my Internet went to the crapper due to street construction that will never end and me not wanting to lease an office I had a tunnel to cloud flare, basically a server stack rack used Ansible/Vault/etc. to spin up a VM/container with ProxMox with each of these servers and proper networking infrastructure. I'd send them a short recorded demo, let them signup and create an SSO like login based on their corporate domain for verification and they could play around and even reset the whole environment. I had "layered" scripts so if I knew the CTO freaked out over something it was kinda easy for me to just change out scripts. I'd also have the ability for them to check out an individual VDI.
I could do this over AWS/GCP/Azure and call it done but that'd freak someone out even though that goes against the immutable/deterministic/etc. buzzwords that go around. I try to make everything agnostic and using Make/Bash/etc. or fancy new tools like Cue/Timoni/Dagger I can when this goes from demo to reality change things with a few CLI changes.
In any case I'm between a bad DSL and a bad backup 5G hotspot. So setting up something like 'infra.git.project-a.realdomain.com' is hard. I'm also changing my workflow significantly so I need to rebuild all this and I've never had to host fake domains on laptop.
I feel as if I'm really close I just don't have the networking prowess or ability to create systemd files or whatever it takes for SSH keys to pass back and forth for multi-tenant setups essentially on one machine. I also got "lazy" doing this with basically an on-prem cloud and 10gbps both direction fiber connection.
Realistically the apps are just a collection of things to make devs lives more configurable and less about coding. Someone commits a file in Figma, a ticket gets created and a design website (in K8s cluster sure) creates a page with a link and screenshot, a ticket gets created, creating a pull request with an ephemeral environment. Messages get sent out to Teams/Slack too. Collaboration is improved, dev environment is converted into something K8s compatible. Usually it is a simple GoLang app/Typescript front end. A lot of the "hard stuff" is talking between systems I have no control over (Salesforce) to extract data import it in. I have that down to a science too. Features go out and are deployed to x% of users and as KPIs are met (not 500 errors, but say conversion KPIs that might be created by ML ha!). It get gets deployed to more and more users. A lot of it is monitoring as nearly all corporate customers use locally-hosted (or at least hosted on EC2 not OpenAI), models. Data scientists want to see which models work "best," by deploying a dozen variations, other KPIs are tracked if say UI is updated (which AI does too sometimes) so really its a simple system with complex moving parts and people have been so burned they need to see it look like it would work in reality.
So a lot of these strange requests come from well it looks like that in a demo.
Sorry to drop buzz words and another long explanation but I thought you'd see why I have strange requirements. Keep in mind these aren't the most tech oriented people I'm talking to and can barely embrace the cloud so if they see something that would seem obvious to you or I as being simple or wouldn't matter (we can just move to Azure if he's hosting the demo on AWS), get projects killed. Or even having the name of the company in the browser is a huge, huge deal.
There's just a lot here I'm not used to dealing with, and I found out the hard way in the Linux world configuration and automation can take over your life. Oddly the "ops" part is not my job but helping them improve migrating to a new "digital platform" (read: talk to executives that the old way of buying into the Oracle world for everything isn't feasible). I also think it'd be kinda cool if I could have pretty much an end-to-end production scenario on one machine.
Looking for any advice as to how to get this done, and I'm in the mindset that developers have a laptop with whatever (in this case VM + mounted volume), a production K8S cluster, a GitRepo, general Auth/Authentication handled between sites by SSH+Ca-Certs (beyond K8S specific requirements, like logging into an internal app in the cluster), and a segregate network mapped to DNS. I was thinking of separate folders with VMs/Containers for each of those but maybe a multi-cluster Kubernetes is best? Like infrastructure in one cluster, the actual production in another? I honestly don't know what works and what doesn't anymore. If you told me one cluster could hold the images repo/git/build/etc that runs the actual workload I wouldn't be surprised.
How do you intend to use this environment? Ship it to the client and have them play around with it? Record a demonstration on your local machine and send that to them? Live demo?
posted by hoyland at 3:09 PM on April 28 [1 favorite]
posted by hoyland at 3:09 PM on April 28 [1 favorite]
Best answer: If you want to replicate all the cloud instances you had, maybe Qubes OS is the host/hypervisor you need -- all the apps install and run in "personality" virtual machines (VM's) so you have to bridge (and firewall) the VM's network adapter to real physical ethernet or wireless adapters. Then everything has tidy L2 routing and you can show local DNS resolution and certificates as if world-facing.
Systemd has the machinectl suite for achieving similar with either full VM's or containers.
If using VM's, create a bridge and set MAC addresses for each virtual ethernet device so that a tool like dnsmasq can assign IP addresses from a text database you maintain at /etc/ethers. dnsmasq can be configured to not respond to DHCP requests from specific network devices, say wired or wifi connections.
If using containers, you can move these virtual ethernet interfaces into network namespaces manually with the ip command to isolate them without needing to configure a bridge.
Whichever suite you're using, say machinectl, podman, k8s/k3s or docker, they're all talking to the same Linux Kernel functions as your cloud hosts: control groups, network namespaces, mounts, and seccomp filtering. Use the tools you're familiar with.
I feel as if I'm really close I just don't have the networking prowess or ability to create systemd files or whatever it takes for SSH keys to pass back and forth for multi-tenant setups essentially on one machine.
Keep the certs, cert private keys, ssh public/private pairs in a folder structure you can bind-mount into any container or VM on your single host.
posted by k3ninho at 3:23 AM on April 29 [1 favorite]
Systemd has the machinectl suite for achieving similar with either full VM's or containers.
If using VM's, create a bridge and set MAC addresses for each virtual ethernet device so that a tool like dnsmasq can assign IP addresses from a text database you maintain at /etc/ethers. dnsmasq can be configured to not respond to DHCP requests from specific network devices, say wired or wifi connections.
If using containers, you can move these virtual ethernet interfaces into network namespaces manually with the ip command to isolate them without needing to configure a bridge.
Whichever suite you're using, say machinectl, podman, k8s/k3s or docker, they're all talking to the same Linux Kernel functions as your cloud hosts: control groups, network namespaces, mounts, and seccomp filtering. Use the tools you're familiar with.
I feel as if I'm really close I just don't have the networking prowess or ability to create systemd files or whatever it takes for SSH keys to pass back and forth for multi-tenant setups essentially on one machine.
Keep the certs, cert private keys, ssh public/private pairs in a folder structure you can bind-mount into any container or VM on your single host.
posted by k3ninho at 3:23 AM on April 29 [1 favorite]
Response by poster: I got laid off due to lack of billable hours, so rambling desperate questions like this will stop. Stress was super high to be billable so I took on projects we needed to automate and lost due to taking a month to stand up an environment. Or worse a fake figma demo.
Anyway I got it working last night. Rather than go down the rabbit hole of running this on a Linux box in a production k8s and a few infra containers (or infra k8s). I got lazy ansible to create a per project VM added ca certs to the root while setting up a VM in that to do the magic. I ran cue timoni to get kubernetes client/demo specific (super easy). Then just ran some docker files for infra that I found laying around. Mounted the repo volume to windows (created profanity with nix then got rid of nix to not scare anyone… super easy. Stayed away from WSL to just avoid any weirdness.
I was trying to turn my laptop into a true multi tenant environment but got it down to a mock http api server for any outside service like Salesforce with mock data and end to end pre commit test, messages, linting, etc, submit to build create docker goes to Kubernetes.
But all for nothing I guess. Quick demos + NVidia Drivers + VLAN + multi cluster is hard on Linux.
posted by geoff. at 6:46 PM on April 29
Anyway I got it working last night. Rather than go down the rabbit hole of running this on a Linux box in a production k8s and a few infra containers (or infra k8s). I got lazy ansible to create a per project VM added ca certs to the root while setting up a VM in that to do the magic. I ran cue timoni to get kubernetes client/demo specific (super easy). Then just ran some docker files for infra that I found laying around. Mounted the repo volume to windows (created profanity with nix then got rid of nix to not scare anyone… super easy. Stayed away from WSL to just avoid any weirdness.
I was trying to turn my laptop into a true multi tenant environment but got it down to a mock http api server for any outside service like Salesforce with mock data and end to end pre commit test, messages, linting, etc, submit to build create docker goes to Kubernetes.
But all for nothing I guess. Quick demos + NVidia Drivers + VLAN + multi cluster is hard on Linux.
posted by geoff. at 6:46 PM on April 29
Response by poster: Host VM being Windows I should say so it looks familiar.
posted by geoff. at 6:47 PM on April 29
posted by geoff. at 6:47 PM on April 29
Response by poster: One more thing I was using a Fedora Silverblue variant that uses rpm-latter. NVidia Optimus or any drivers tend to create small issues since the idea is to have an immutable atomic desktop and not all apps are built for that. I think it has to do with the way a container loads a kernel. Development wise? Great. But you have to stick to the path of any system wide changes need to accommodate the format or it’ll break, and in strange ways. I actually built out a live windows OS and the Linux layer on it then created an ISO with a Debian flavor that supports Cuda, built all that into a single ISO. I didn’t test a lot and realizing I didn’t need consul or vault for key management simplified it a lot.
We (they) didn’t have a SaaS product but needed a complex environment to create workflows and aggregate data. Think asking a slack channel for info on revenue per client in natural language then asking it to create a workflow, etc. the dev teams have no idea about DevOps, automation, Kubernetes or AI. But they want to see it working realistically (add a feature, see it go live). Similarly, the upper management wants to see the mock data http api layers between services actually talk to each other and produce results. So the Windows desktop was because they both don’t want a web UI or gradio demo. These again are 90s enterprise Java devs.
I think k3ninho hit the nail on the head on how to do this. If I added say Debian base with tested Cuda/Optimis pass through, a Live Windows ISO and multi cluster with bridging id be golden. I could even use ZFSBootMenu to switch between demos or something. Build it with ansible/cue/timomi, then basically have a fairly hardened demo environment. Wouldn’t be that hard.
The biggest problem is that laptops have poor driver support for Linux in general. And handing off this to the client is close to impossible with the needed AI component. Plus I’d haven’t thought about the legal implications of passing a Windows VM. Probably would require further negotiation on hosting on a cloud platform if they want to test it live.
It does look slick and “production ready,” again hosting a multi tenant cloud infrastructure isn’t best on a laptop. But I kept being drawn into a corner and had to make do.
posted by geoff. at 8:31 PM on April 29
We (they) didn’t have a SaaS product but needed a complex environment to create workflows and aggregate data. Think asking a slack channel for info on revenue per client in natural language then asking it to create a workflow, etc. the dev teams have no idea about DevOps, automation, Kubernetes or AI. But they want to see it working realistically (add a feature, see it go live). Similarly, the upper management wants to see the mock data http api layers between services actually talk to each other and produce results. So the Windows desktop was because they both don’t want a web UI or gradio demo. These again are 90s enterprise Java devs.
I think k3ninho hit the nail on the head on how to do this. If I added say Debian base with tested Cuda/Optimis pass through, a Live Windows ISO and multi cluster with bridging id be golden. I could even use ZFSBootMenu to switch between demos or something. Build it with ansible/cue/timomi, then basically have a fairly hardened demo environment. Wouldn’t be that hard.
The biggest problem is that laptops have poor driver support for Linux in general. And handing off this to the client is close to impossible with the needed AI component. Plus I’d haven’t thought about the legal implications of passing a Windows VM. Probably would require further negotiation on hosting on a cloud platform if they want to test it live.
It does look slick and “production ready,” again hosting a multi tenant cloud infrastructure isn’t best on a laptop. But I kept being drawn into a corner and had to make do.
posted by geoff. at 8:31 PM on April 29
« Older Picking up Piano Again as an Adult | Blueprints for building a library (of construction... Newer »
You are not logged in, either login or create an account to post comments
If really wanted GitHub and not an OSS or free self-hosted alternatives (GitLab) it is easy to convert actions/workflows for my purposes with Earthly/Cue or many other tools. I'd be okay with giving them the entire entity or better yet hosting it somewhere if they have specific concerns. But that turns from demo to eating pre-sales costs or charging them.
In any case I want to run a script and generate all this independently with host names set at least on the internal machine I'm on obviously then customize it for their various needs, save it to a USB and pull it up again without thinking.
These are not high level engineers so as much polish I can get out of it is the key. You never know if they'd be okay with Kind vs K8s, IP addresses vs host names, etc.
posted by geoff. at 1:40 PM on April 28