2.4 KiB
Overview
So we have some machines(we call those "nodes") and each of those nodes runs (1) a nomad client; and (2) a consul client. Also, all these nodes are in a wireguard network.
nomad is responsible for actually deploying the servers on an available machine. actually the Nomad Server has state and knows what all clients are doing, while Nomad Clients are just doing what the server tells them to, like deploying a job.
A job is just a docker container, or the smallest deployable unit ig.
the same server-client-relationship exists with consul. Here, Consul's main purpose is to resolve hosts. A job registers itself (via a Consul Client) like "hey im a postgres server and i run on 10.0.2.1:5432" and then the Consul Master updates its catalogue: "oh so postgres is 10.0.2.1:5432" and then in our jobs, we can just use "postgres.service.consul" and it resolves to 10.0.2.1:5432 so thats cool.
Now wireguard is needed because while consul resolves hostnames, it doesn't help us connect to the resolved IPs. With a single node, that's fine but if we want to access a service on another node, we need wireguard(or something similar). Actually if we run all nodes in the same network and it'd be fine without.
Most Basic Implementation
- single node
- runs
dnsmasq -d,nomad agent -dev, andconsul agent -devas root - /etc/nomad.d/consul.hcl: consul { address = "127.0.0.1:8500" }
- /etc/dnsmasq.conf: server=/consul/127.0.0.1#8600
- all jobs define a consul service
- TODO: make dns/connect work
useful stuff
nomad job run <job>.hclconsul catalog services<- list registered services- http://localhost:4646/ui/jobs <- web UI accessible after
nomad agent -dev - https://developer.hashicorp.com/nomad/docs/configuration
- https://developer.hashicorp.com/nomad/docs/job-declare/service-discovery
Storage/SPOT
the papermc container already copies all files in /mnt/* to /data, so it's easy to add overwrites. in ./lobby.nomad.hcl I just use a prestart container that downloads the plugin to a temporary volume that is then used as /mnt. A lot of boilerplate and it's also not easy to add other files. Ideally, we'd just copy a whole directory over sftp. it would look like this: /.gitignore <- data /Makefile <- downloads plugins specified to /data/plugins /static <- static files that are copied by make to /data But ig that wouldn't play nice with CI/CD...
Also currently we need to duplicate a lot of this boilerplate btw.