61 lines
2.4 KiB
Markdown
61 lines
2.4 KiB
Markdown
# Overview
|
|
|
|
So we have some machines(we call those "nodes")
|
|
and each of those nodes runs (1) a nomad client; and (2) a consul client.
|
|
Also, all these nodes are in a wireguard network.
|
|
|
|
nomad is responsible for actually deploying the servers on an available machine.
|
|
actually the Nomad *Server* has state and knows what all clients are doing,
|
|
while Nomad *Clients* are just doing what the server tells them to, like deploying
|
|
a job.
|
|
|
|
A job is just a docker container, or the smallest deployable unit ig.
|
|
|
|
the same server-client-relationship exists with consul. Here, Consul's main
|
|
purpose is to resolve hosts. A job registers itself (via a Consul Client) like
|
|
"hey im a postgres server and i run on 10.0.2.1:5432" and then the Consul Master
|
|
updates its catalogue: "oh so postgres is 10.0.2.1:5432" and then in our jobs, we
|
|
can just use "postgres.service.consul" and it resolves to 10.0.2.1:5432 so thats cool.
|
|
|
|
Now wireguard is needed because while consul resolves hostnames, it doesn't help
|
|
us connect to the resolved IPs. With a single node, that's fine but if we want to
|
|
access a service on another node, we need wireguard(or something similar).
|
|
Actually if we run all nodes in the same network and it'd be fine without.
|
|
|
|
## Most Basic Implementation
|
|
|
|
- single node
|
|
- runs `dnsmasq -d`, `nomad agent -dev`, and `consul agent -dev` as root
|
|
- /etc/nomad.d/consul.hcl:
|
|
consul {
|
|
address = "127.0.0.1:8500"
|
|
}
|
|
- /etc/dnsmasq.conf:
|
|
server=/consul/127.0.0.1#8600
|
|
- all jobs define a consul service
|
|
- TODO: make dns/connect work
|
|
|
|
|
|
## useful stuff
|
|
|
|
- `nomad job run <job>.hcl`
|
|
- `consul catalog services` <- list registered services
|
|
- http://localhost:4646/ui/jobs <- web UI accessible after `nomad agent -dev`
|
|
- https://developer.hashicorp.com/nomad/docs/configuration
|
|
- https://developer.hashicorp.com/nomad/docs/job-declare/service-discovery
|
|
|
|
## Storage/SPOT
|
|
|
|
the papermc container already copies all files in /mnt/*
|
|
to /data, so it's easy to add overwrites.
|
|
in ./lobby.nomad.hcl I just use a prestart container that downloads the plugin
|
|
to a temporary volume that is then used as /mnt.
|
|
A lot of boilerplate and it's also not easy to add other files.
|
|
Ideally, we'd just copy a whole directory over sftp. it would look
|
|
like this:
|
|
/.gitignore <- data
|
|
/Makefile <- downloads plugins specified to /data/plugins
|
|
/static <- static files that are copied by make to /data
|
|
But ig that wouldn't play nice with CI/CD...
|
|
|
|
Also currently we need to duplicate a lot of this boilerplate btw.
|