tests: Begin integration tests

Cargo uses the sources in the `tests` directory to build and run integration tests. For each `tests/foo.rs` or `tests/foo/main.rs`, it creates an executable that runs the test functions therein. These executables are separate crates from the main package, and thus do not have access to its private members. Integration tests are expected to test only the public functionality of the package. Application crates do not have any public members; their public interface is the command line. Integration tests would typically run the command (e.g. using `std::process::Command`) and test its output. Since *dynk8s-provisioner* is not really a command-line tool, testing it this way would be difficult; each test would need to start the server, make requests to it, and then stop it. This would be slow and cumbersome. In order to avoid this tedium and be able to use Rocket's built-in test client, I have converted *dynk8s-provisioner* into a library crate that also includes an executable. The library makes the `rocket` function public, which allows the integration tests to import it and pass it to the Rocket test client. The point of integration tests, of course, is to validate the functionality of the application as a whole. This necessarily requires allowing it to communicate with the Kubernetes API. In the Jenkins CI environment, the application will need the appropriate credentials, and will need to use a separate Kubernetes namespace from the production deployment. The `setup.yaml` manifest in the `tests` directory defines the resources necessary to run integration tests, and the `genkubeconfig.sh` script can be used to create the appropriate kubeconfig file containing the credentials. The kubeconfig is exposed to the tests via the `KUBECONFIG` environment variable, which is populated from a Jenkins secret file credential. Note: The `data` directory moved from `test` to `tests` to avoid duplication and confusing names.
events: Delete bootstrap tokens on termination
2022-10-07 07:37:20 -05:00 · 2022-10-07 06:52:06 -05:00 · 2022-10-07 06:52:06 -05:00 · 2022-10-03 18:29:47 -05:00 · 2022-10-03 18:20:46 -05:00 · 2022-10-01 12:17:32 -05:00
34 changed files with 2077 additions and 30 deletions
--- a/Cargo.lock
+++ b/Cargo.lock
@ -37,6 +37,24 @@ dependencies = [
 "subtle",
 ]

+[[package]]
+name = "aho-corasick"
+version = "0.7.19"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b4f55bd91a0978cbfd91c457a164bab8b4001c833b7f323132c0a4e1922dd44e"
+dependencies = [
+ "memchr",
+]
+
+[[package]]
+name = "android_system_properties"
+version = "0.1.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "819e7219dbd41043ac279b19830f2efc897156490d7fd6ea916720117ee66311"
+dependencies = [
+ "libc",
+]
+
 [[package]]
 name = "ansi_term"
 version = "0.12.1"
@ -59,7 +77,7 @@ dependencies = [
 "num-traits",
 "rusticata-macros",
 "thiserror",
- "time",
+ "time 0.3.14",
 ]

 [[package]]
@ -206,6 +224,22 @@ version = "1.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd"

+[[package]]
+name = "chrono"
+version = "0.4.22"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "bfd4d1b31faaa3a89d7934dbded3111da0d2ef28e3ebccdb4f0179f5929d1ef1"
+dependencies = [
+ "iana-time-zone",
+ "js-sys",
+ "num-integer",
+ "num-traits",
+ "serde",
+ "time 0.1.44",
+ "wasm-bindgen",
+ "winapi",
+]
+
 [[package]]
 name = "cipher"
 version = "0.3.0"
@ -235,7 +269,7 @@ dependencies = [
 "rand",
 "sha2",
 "subtle",
- "time",
+ "time 0.3.14",
 "version_check",
 ]

@ -368,6 +402,27 @@ dependencies = [
 "subtle",
 ]

+[[package]]
+name = "dirs-next"
+version = "2.0.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b98cf8ebf19c3d1b223e151f99a4f9f0690dca41414773390fc824184ac833e1"
+dependencies = [
+ "cfg-if",
+ "dirs-sys-next",
+]
+
+[[package]]
+name = "dirs-sys-next"
+version = "0.1.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4ebda144c4fe02d1f7ea1a7d9641b6fc6b580adcfa024ae48797ecdeb6825b4d"
+dependencies = [
+ "libc",
+ "redox_users",
+ "winapi",
+]
+
 [[package]]
 name = "displaydoc"
 version = "0.2.3"
@ -384,12 +439,18 @@ name = "dynk8s-provisioner"
 version = "0.1.0"
 dependencies = [
 "base64",
+ "chrono",
+ "k8s-openapi",
+ "kube",
 "log",
+ "rand",
+ "regex",
 "reqwest",
 "rocket",
 "rsa",
 "serde",
 "serde_json",
+ "serde_yaml 0.9.13",
 "sha1",
 "x509-parser",
 ]
@ -471,6 +532,7 @@ checksum = "7f21eda599937fba36daeb58a22e8f5cee2d14c4a17b5b7739c7c8e5e3b8230c"
 dependencies = [
 "futures-channel",
 "futures-core",
+ "futures-executor",
 "futures-io",
 "futures-sink",
 "futures-task",
@ -493,12 +555,34 @@ version = "0.3.24"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "4e5aa3de05362c3fb88de6531e6296e85cde7739cccad4b9dfeeb7f6ebce56bf"

+[[package]]
+name = "futures-executor"
+version = "0.3.24"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9ff63c23854bee61b6e9cd331d523909f238fc7636290b96826e9cfa5faa00ab"
+dependencies = [
+ "futures-core",
+ "futures-task",
+ "futures-util",
+]
+
 [[package]]
 name = "futures-io"
 version = "0.3.24"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "bbf4d2a7a308fd4578637c0b17c7e1c7ba127b8f6ba00b29f717e9655d85eb68"

+[[package]]
+name = "futures-macro"
+version = "0.3.24"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "42cd15d1c7456c04dbdf7e88bcd69760d74f3a798d6444e16974b505b0e62f17"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn",
+]
+
 [[package]]
 name = "futures-sink"
 version = "0.3.24"
@ -520,6 +604,7 @@ dependencies = [
 "futures-channel",
 "futures-core",
 "futures-io",
+ "futures-macro",
 "futures-sink",
 "futures-task",
 "memchr",
@ -559,7 +644,7 @@ checksum = "4eb1a864a501629691edf6c15a593b7a51eebaa1e8468e9ddc623de7c9b58ec6"
 dependencies = [
 "cfg-if",
 "libc",
- "wasi",
+ "wasi 0.11.0+wasi-snapshot-preview1",
 ]

 [[package]]
@ -652,6 +737,12 @@ dependencies = [
 "pin-project-lite",
 ]

+[[package]]
+name = "http-range-header"
+version = "0.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0bfe8eed0a9285ef776bb792479ea3834e8b94e13d615c2f66d03dd50a435a29"
+
 [[package]]
 name = "httparse"
 version = "1.8.0"
@ -688,6 +779,36 @@ dependencies = [
 "want",
 ]

+[[package]]
+name = "hyper-openssl"
+version = "0.9.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d6ee5d7a8f718585d1c3c61dfde28ef5b0bb14734b4db13f5ada856cdc6c612b"
+dependencies = [
+ "http",
+ "hyper",
+ "linked_hash_set",
+ "once_cell",
+ "openssl",
+ "openssl-sys",
+ "parking_lot",
+ "tokio",
+ "tokio-openssl",
+ "tower-layer",
+]
+
+[[package]]
+name = "hyper-timeout"
+version = "0.4.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "bbb958482e8c7be4bc3cf272a766a2b0bf1a6755e7a6ae777f017a31d11b13b1"
+dependencies = [
+ "hyper",
+ "pin-project-lite",
+ "tokio",
+ "tokio-io-timeout",
+]
+
 [[package]]
 name = "hyper-tls"
 version = "0.5.0"
@ -701,6 +822,19 @@ dependencies = [
 "tokio-native-tls",
 ]

+[[package]]
+name = "iana-time-zone"
+version = "0.1.50"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "fd911b35d940d2bd0bea0f9100068e5b97b51a1cbe13d13382f132e0365257a0"
+dependencies = [
+ "android_system_properties",
+ "core-foundation-sys",
+ "js-sys",
+ "wasm-bindgen",
+ "winapi",
+]
+
 [[package]]
 name = "idna"
 version = "0.2.3"
@ -759,6 +893,96 @@ dependencies = [
 "wasm-bindgen",
 ]

+[[package]]
+name = "jsonpath_lib"
+version = "0.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "eaa63191d68230cccb81c5aa23abd53ed64d83337cacbb25a7b8c7979523774f"
+dependencies = [
+ "log",
+ "serde",
+ "serde_json",
+]
+
+[[package]]
+name = "k8s-openapi"
+version = "0.16.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6d9455388f4977de4d0934efa9f7d36296295537d774574113a20f6082de03da"
+dependencies = [
+ "base64",
+ "bytes",
+ "chrono",
+ "http",
+ "percent-encoding",
+ "serde",
+ "serde-value",
+ "serde_json",
+ "url",
+]
+
+[[package]]
+name = "kube"
+version = "0.75.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9bb19108692aeafebb108fd0a1c381c06ac4c03859652599420975165e939b8a"
+dependencies = [
+ "k8s-openapi",
+ "kube-client",
+ "kube-core",
+]
+
+[[package]]
+name = "kube-client"
+version = "0.75.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "97e1a80ecd1b1438a2fc004549e155d47250b9e01fbfcf4cfbe9c8b56a085593"
+dependencies = [
+ "base64",
+ "bytes",
+ "chrono",
+ "dirs-next",
+ "either",
+ "futures",
+ "http",
+ "http-body",
+ "hyper",
+ "hyper-openssl",
+ "hyper-timeout",
+ "jsonpath_lib",
+ "k8s-openapi",
+ "kube-core",
+ "openssl",
+ "pem",
+ "pin-project",
+ "secrecy",
+ "serde",
+ "serde_json",
+ "serde_yaml 0.8.26",
+ "thiserror",
+ "tokio",
+ "tokio-util",
+ "tower",
+ "tower-http",
+ "tracing",
+]
+
+[[package]]
+name = "kube-core"
+version = "0.75.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f4d780f2bb048eeef64a4c6b2582d26a0fe19e30b4d3cc9e081616e1779c5d47"
+dependencies = [
+ "chrono",
+ "form_urlencoded",
+ "http",
+ "k8s-openapi",
+ "once_cell",
+ "serde",
+ "serde_json",
+ "thiserror",
+]
+
 [[package]]
 name = "lazy_static"
 version = "1.4.0"
@ -780,6 +1004,21 @@ version = "0.2.5"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "292a948cd991e376cf75541fe5b97a1081d713c618b4f1b9500f8844e49eb565"

+[[package]]
+name = "linked-hash-map"
+version = "0.5.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0717cef1bc8b636c6e1c1bbdefc09e6322da8a9321966e8928ef80d20f7f770f"
+
+[[package]]
+name = "linked_hash_set"
+version = "0.1.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "47186c6da4d81ca383c7c47c1bfc80f4b95f4720514d860a5407aaf4233f9588"
+dependencies = [
+ "linked-hash-map",
+]
+
 [[package]]
 name = "lock_api"
 version = "0.4.8"
@ -855,7 +1094,7 @@ checksum = "57ee1c23c7c63b0c9250c339ffdc69255f110b298b901b9f6c82547b7b87caaf"
 dependencies = [
 "libc",
 "log",
- "wasi",
+ "wasi 0.11.0+wasi-snapshot-preview1",
 "windows-sys",
 ]

@ -1051,6 +1290,15 @@ dependencies = [
 "vcpkg",
 ]

+[[package]]
+name = "ordered-float"
+version = "2.10.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7940cf2ca942593318d07fcf2596cdca60a85c9e7fab408a5e21a4f9dcd40d87"
+dependencies = [
+ "num-traits",
+]
+
 [[package]]
 name = "parking_lot"
 version = "0.12.1"
@ -1097,6 +1345,15 @@ dependencies = [
 "syn",
 ]

+[[package]]
+name = "pem"
+version = "1.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "03c64931a1a212348ec4f3b4362585eca7159d0d09cbdf4a7f74f02173596fd4"
+dependencies = [
+ "base64",
+]
+
 [[package]]
 name = "pem-rfc7468"
 version = "0.3.1"
@ -1112,6 +1369,26 @@ version = "2.1.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "d4fd5641d01c8f18a23da7b6fe29298ff4b55afcccdf78973b24cf3175fee32e"

+[[package]]
+name = "pin-project"
+version = "1.0.12"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ad29a609b6bcd67fee905812e544992d216af9d755757c05ed2d0e15a74c6ecc"
+dependencies = [
+ "pin-project-internal",
+]
+
+[[package]]
+name = "pin-project-internal"
+version = "1.0.12"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "069bdb1e05adc7a8990dce9cc75370895fbe4e3d58b9b73bf1aee56359344a55"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn",
+]
+
 [[package]]
 name = "pin-project-lite"
 version = "0.2.9"
@ -1240,6 +1517,17 @@ dependencies = [
 "bitflags",
 ]

+[[package]]
+name = "redox_users"
+version = "0.4.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b033d837a7cf162d7993aded9304e30a83213c648b6e389db233191f891e5c2b"
+dependencies = [
+ "getrandom",
+ "redox_syscall",
+ "thiserror",
+]
+
 [[package]]
 name = "ref-cast"
 version = "1.0.9"
@ -1266,6 +1554,8 @@ version = "1.6.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "4c4eb3267174b8c6c2f654116623910a0fef09c4753f8dd83db29c48a0df988b"
 dependencies = [
+ "aho-corasick",
+ "memchr",
 "regex-syntax",
 ]

@ -1360,7 +1650,7 @@ dependencies = [
 "serde_json",
 "state",
 "tempfile",
- "time",
+ "time 0.3.14",
 "tokio",
 "tokio-stream",
 "tokio-util",
@ -1407,7 +1697,7 @@ dependencies = [
 "smallvec",
 "stable-pattern",
 "state",
- "time",
+ "time 0.3.14",
 "tokio",
 "uncased",
 ]
@ -1475,6 +1765,16 @@ version = "1.1.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "d29ab0c6d3fc0ee92fe66e2d99f700eab17a8d57d1c1d3b748380fb20baa78cd"

+[[package]]
+name = "secrecy"
+version = "0.8.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9bd1c54ea06cfd2f6b63219704de0b9b4f72dcc2b8fdef820be6cd799780e91e"
+dependencies = [
+ "serde",
+ "zeroize",
+]
+
 [[package]]
 name = "security-framework"
 version = "2.7.0"
@ -1507,6 +1807,16 @@ dependencies = [
 "serde_derive",
 ]

+[[package]]
+name = "serde-value"
+version = "0.7.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f3a1a3341211875ef120e117ea7fd5228530ae7e7036a779fdc9117be6b3282c"
+dependencies = [
+ "ordered-float",
+ "serde",
+]
+
 [[package]]
 name = "serde_derive"
 version = "1.0.144"
@ -1524,6 +1834,7 @@ version = "1.0.85"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "e55a28e3aaef9d5ce0506d0a14dbba8054ddc7e499ef522dd8b26859ec9d4a44"
 dependencies = [
+ "indexmap",
 "itoa",
 "ryu",
 "serde",
@ -1541,6 +1852,31 @@ dependencies = [
 "serde",
 ]

+[[package]]
+name = "serde_yaml"
+version = "0.8.26"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "578a7433b776b56a35785ed5ce9a7e777ac0598aac5a6dd1b4b18a307c7fc71b"
+dependencies = [
+ "indexmap",
+ "ryu",
+ "serde",
+ "yaml-rust",
+]
+
+[[package]]
+name = "serde_yaml"
+version = "0.9.13"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8613d593412a0deb7bbd8de9d908efff5a0cb9ccd8f62c641e7b2ed2f57291d1"
+dependencies = [
+ "indexmap",
+ "itoa",
+ "ryu",
+ "serde",
+ "unsafe-libyaml",
+]
+
 [[package]]
 name = "sha1"
 version = "0.10.2"
@ -1718,6 +2054,17 @@ dependencies = [
 "once_cell",
 ]

+[[package]]
+name = "time"
+version = "0.1.44"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6db9e6914ab8b1ae1c260a4ae7a49b6c5611b40328a735b21862567685e73255"
+dependencies = [
+ "libc",
+ "wasi 0.10.0+wasi-snapshot-preview1",
+ "winapi",
+]
+
 [[package]]
 name = "time"
 version = "0.3.14"
@ -1771,6 +2118,16 @@ dependencies = [
 "winapi",
 ]

+[[package]]
+name = "tokio-io-timeout"
+version = "1.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "30b74022ada614a1b4834de765f9bb43877f910cc8ce4be40e89042c9223a8bf"
+dependencies = [
+ "pin-project-lite",
+ "tokio",
+]
+
 [[package]]
 name = "tokio-macros"
 version = "1.8.0"
@ -1792,6 +2149,18 @@ dependencies = [
 "tokio",
 ]

+[[package]]
+name = "tokio-openssl"
+version = "0.6.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c08f9ffb7809f1b20c1b398d92acf4cc719874b3b2b2d9ea2f09b4a80350878a"
+dependencies = [
+ "futures-util",
+ "openssl",
+ "openssl-sys",
+ "tokio",
+]
+
 [[package]]
 name = "tokio-stream"
 version = "0.1.9"
@ -1826,6 +2195,49 @@ dependencies = [
 "serde",
 ]

+[[package]]
+name = "tower"
+version = "0.4.13"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b8fa9be0de6cf49e536ce1851f987bd21a43b771b09473c3549a6c853db37c1c"
+dependencies = [
+ "futures-core",
+ "futures-util",
+ "pin-project",
+ "pin-project-lite",
+ "tokio",
+ "tokio-util",
+ "tower-layer",
+ "tower-service",
+ "tracing",
+]
+
+[[package]]
+name = "tower-http"
+version = "0.3.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3c530c8675c1dbf98facee631536fa116b5fb6382d7dd6dc1b118d970eafe3ba"
+dependencies = [
+ "base64",
+ "bitflags",
+ "bytes",
+ "futures-core",
+ "futures-util",
+ "http",
+ "http-body",
+ "http-range-header",
+ "pin-project-lite",
+ "tower-layer",
+ "tower-service",
+ "tracing",
+]
+
+[[package]]
+name = "tower-layer"
+version = "0.3.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "343bc9466d3fe6b0f960ef45960509f84480bf4fd96f92901afe7ff3df9d3a62"
+
 [[package]]
 name = "tower-service"
 version = "0.3.2"
@ -1839,6 +2251,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "2fce9567bd60a67d08a16488756721ba392f24f29006402881e43b19aac64307"
 dependencies = [
 "cfg-if",
+ "log",
 "pin-project-lite",
 "tracing-attributes",
 "tracing-core",
@ -1962,6 +2375,12 @@ dependencies = [
 "subtle",
 ]

+[[package]]
+name = "unsafe-libyaml"
+version = "0.2.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c1e5fa573d8ac5f1a856f8d7be41d390ee973daf97c806b2c1a465e4e1406e68"
+
 [[package]]
 name = "url"
 version = "2.2.2"
@ -2002,6 +2421,12 @@ dependencies = [
 "try-lock",
 ]

+[[package]]
+name = "wasi"
+version = "0.10.0+wasi-snapshot-preview1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1a143597ca7c7793eff794def352d41792a93c481eb1042423ff7ff72ba2c31f"
+
 [[package]]
 name = "wasi"
 version = "0.11.0+wasi-snapshot-preview1"
@ -2216,7 +2641,16 @@ dependencies = [
 "oid-registry",
 "rusticata-macros",
 "thiserror",
- "time",
+ "time 0.3.14",
+]
+
+[[package]]
+name = "yaml-rust"
+version = "0.4.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "56c1936c4cc7a1c9ab21a1ebb602eb942ba868cbd44a99cb7cdc5892335e1c85"
+dependencies = [
+ "linked-hash-map",
 ]

 [[package]]
--- a/Cargo.toml
+++ b/Cargo.toml
@ -5,11 +5,19 @@ edition = "2021"

 [dependencies]
 base64 = "0.13.0"
+chrono = "0.4.22"
+k8s-openapi = { version = "0.16.0", features = ["v1_22"] }
+kube = "0.75.0"
 log = "0.4.17"
+rand = "0.8.5"
 reqwest = "0.11.11"
 rocket = { version = "0.5.0-rc.2", features = ["json"] }
 rsa = "0.6.1"
 serde = { version = "1.0", features = ["derive"] }
 serde_json = "1.0.85"
+serde_yaml = "0.9.13"
 sha1 = "0.10.2"
 x509-parser = "0.14.0"
+
+[dev-dependencies]
+regex = "1.6.0"
--- a/ci/Jenkinsfile
+++ b/ci/Jenkinsfile
@ -20,10 +20,15 @@ pipeline {
        stage('Test') {
            steps {
                container('build') {
+                    withCredentials([file(
+                        credentialsId: 'dynk8s-test-kubeconfig',
+                        variable: 'KUBECONFIG'
+                    )]) {
                        sh '. ci/test.sh'
                    }
                }
            }
+        }

        stage('Build') {
            steps {
--- a/src/events.rs
+++ b/src/events.rs
@ -0,0 +1,57 @@
+//! Event handlers
+//!
+//! Functions in this module are called to handle events from outside sources,
+//! such as Amazon EventBridge events delivered via Amazon Simple Notification
+//! Service.
+use log::{debug, error};
+
+use crate::k8s::{
+    assign_wireguard_config, create_bootstrap_token, delete_bootstrap_tokens,
+    unassign_wireguard_config,
+};
+use crate::model::events::*;
+
+/// Handle an EC2 instance state change event
+///
+/// This function manages the lifecycle of the Kubernetes Secret resources
+/// associated with ephemeral nodes running as EC2 instances.
+///
+/// When an instance starts:
+/// 1. A WireGuard config is assigned to the instance
+/// 2. A Kubernetes bootstrap token is generated, to be used by `kubeadm` to
+///    add the node to the cluster.
+///
+/// When an instance is terminated:
+/// 1. Any WireGuard configs assigned to the instance are unassigned
+/// 2. All bootstrap tokens for the instance are deleted
+pub async fn on_ec2_instance_state_change(evt: Ec2InstanceStateChange) {
+    debug!("EC2 instance {} is now {}", &evt.instance_id, &evt.state);
+    if evt.state == "running" {
+        if let Err(e) = assign_wireguard_config(&evt.instance_id).await {
+            error!(
+                "Failed to assign WireGuard config to instnce {}: {}",
+                &evt.instance_id, e
+            );
+            return;
+        }
+        if let Err(e) = create_bootstrap_token(&evt.instance_id).await {
+            error!(
+                "Failed to create bootstrap token for instance {}: {}",
+                &evt.instance_id, e
+            )
+        };
+    } else if evt.state == "terminated" {
+        if let Err(e) = unassign_wireguard_config(&evt.instance_id).await {
+            error!(
+                "Failed to unassign WireGuard config from instance: {}: {}",
+                &evt.instance_id, e
+            );
+        }
+        if let Err(e) = delete_bootstrap_tokens(&evt.instance_id).await {
+            error!(
+                "Failed to delete bootstrap tokens for instance {}: {}",
+                &evt.instance_id, e
+            );
+        }
+    }
+}
--- a/src/k8s.rs
+++ b/src/k8s.rs
@ -0,0 +1,539 @@
+//! Kubernetes Integration
+use std::collections::btree_map::BTreeMap;
+
+use chrono::offset::Utc;
+use chrono::{DateTime, Duration};
+use k8s_openapi::api::core::v1::{ConfigMap, Secret};
+use k8s_openapi::apimachinery::pkg::apis::meta::v1::ObjectMeta;
+use kube::core::params::{ListParams, Patch, PatchParams, PostParams};
+use kube::{Api, Client};
+use log::{debug, error, info, warn};
+use rand::seq::SliceRandom;
+
+use crate::model::k8s::*;
+
+/// The set of characters allowed to appear in bootstrap tokens
+const TOKEN_CHARS: [char; 36] = [
+    'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o',
+    'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', '0', '1', '2', '3',
+    '4', '5', '6', '7', '8', '9',
+];
+
+/// Kubernetes Bootstrap Token
+///
+/// Bootstrap tokens are typically used to add new nodes to the cluster using
+/// `kubeadm join`.  They are bearer tokens consisting of two parts: a token ID
+/// and a secret.  For additional information, see [Authenticating with
+/// Bootstrap Tokens][0].
+///
+/// The Dynk8s Provisioner allocates bootstrap tokens for ephemeral nodes.
+/// Each token is assigned to a specific EC2 instance; the instance ID is
+/// stored in the token Secret's metadata, using the
+/// `dynk8s.du5t1n.me/ec2-instance-id` label.
+///
+/// [0]: https://kubernetes.io/docs/reference/access-authn-authz/bootstrap-tokens/
+#[derive(Clone, Debug)]
+struct BootstrapToken {
+    /// The token ID (generated)
+    token_id: String,
+    /// The token secret (generated)
+    secret: String,
+    /// The date and time the token expires
+    expiration: DateTime<Utc>,
+    /// The EC2 instance ID to which the token is assigned
+    instance_id: Option<String>,
+}
+
+#[allow(dead_code)]
+impl BootstrapToken {
+    /// Generate a new token
+    ///
+    /// Initially, the token is *not* assigned to an EC2 instance.  Use
+    /// [`Self::instance_id`] to set the associated instance ID.
+    pub fn new() -> Self {
+        let mut rng = rand::thread_rng();
+        let mut token_id = String::with_capacity(6);
+        while token_id.len() < 6 {
+            token_id.push(TOKEN_CHARS.choose(&mut rng).unwrap().clone());
+        }
+        let mut secret = String::with_capacity(16);
+        while secret.len() < 16 {
+            secret.push(TOKEN_CHARS.choose(&mut rng).unwrap().clone());
+        }
+        let expiration = Utc::now() + Duration::hours(1);
+        Self {
+            token_id,
+            secret,
+            expiration,
+            instance_id: None,
+        }
+    }
+
+    /// Return the expiration date/time
+    pub fn expiration(&self) -> &DateTime<Utc> {
+        &self.expiration
+    }
+
+    /// Set the ID of the EC2 instance associated with the token
+    pub fn instance_id(mut self, instance_id: String) -> Self {
+        self.instance_id = Some(instance_id);
+        self
+    }
+
+    /// Return the secret part of the token
+    pub fn secret(&self) -> &str {
+        &self.secret
+    }
+
+    /// Set the expiration date/time from a time-to-live duration
+    pub fn set_ttl(mut self, ttl: Duration) -> Self {
+        self.expiration = Utc::now() + ttl;
+        self
+    }
+
+    /// Set the expiration date/time
+    pub fn set_expiration(mut self, expiration: DateTime<Utc>) -> Self {
+        self.expiration = expiration;
+        self
+    }
+
+    /// Return the ID part of the token
+    pub fn token_id(&self) -> &str {
+        &self.token_id
+    }
+
+    /// Return the token as a string
+    pub fn token(&self) -> String {
+        format!("{}.{}", self.token_id, self.secret)
+    }
+}
+
+impl Into<Secret> for BootstrapToken {
+    /// Create a [`Secret`] for the token
+    ///
+    /// Converting a [`BootstrapToken`] into a [`Secret`] populates the fields
+    /// necessary to store the token in Kubernetes.  The `Secret` can be passed
+    /// to e.g. [`kube::Api::create`].
+    fn into(self) -> Secret {
+        let mut data = BTreeMap::<String, String>::new();
+        data.insert("token-id".into(), self.token_id.clone());
+        data.insert("token-secret".into(), self.secret);
+        data.insert("expiration".into(), self.expiration.to_rfc3339());
+        data.insert("usage-bootstrap-authentication".into(), "true".into());
+        data.insert("usage-bootstrap-signing".into(), "true".into());
+        data.insert(
+            "auth-extra-groups".into(),
+            "system:bootstrappers:kubeadm:default-node-token".into(),
+        );
+        let mut labels = BTreeMap::<String, String>::new();
+        if let Some(instance_id) = self.instance_id {
+            labels.insert(
+                "dynk8s.du5t1n.me/ec2-instance-id".into(),
+                instance_id.into(),
+            );
+        }
+        Secret {
+            data: None,
+            immutable: None,
+            metadata: ObjectMeta {
+                annotations: None,
+                cluster_name: None,
+                creation_timestamp: None,
+                deletion_grace_period_seconds: None,
+                deletion_timestamp: None,
+                finalizers: None,
+                generate_name: None,
+                generation: None,
+                labels: Some(labels),
+                managed_fields: None,
+                name: Some(format!("bootstrap-token-{}", self.token_id)),
+                namespace: None,
+                owner_references: None,
+                resource_version: None,
+                self_link: None,
+                uid: None,
+            },
+            string_data: Some(data),
+            type_: Some("bootstrap.kubernetes.io/token".into()),
+        }
+    }
+}
+
+/// Assign an existing WireGuard configuration to the specified EC2 instance
+///
+/// This function finds the first unused WireGuard client configuration, stored
+/// as a Kubernetes Secret resource, and assigns it to the specified EC2
+/// instance.  Configs are assigned by setting the
+/// `dynk8s.du5t1n.me/ec2-instance-id` label in the Secret resource's metadata.
+///
+/// Secret resources for WireGuard configuration have a *type* of
+/// `dynk8s.du5t1n.me/wireguard-config`.  The Secret's `data` field must
+/// contain a `wireguard-config` property, which contains the WireGuard client
+/// configuration the node should use.  Configs must be created ahead of time
+/// and must refer to working keys already configured on the WireGuard server.
+pub async fn assign_wireguard_config(
+    instance_id: &str,
+) -> Result<(), kube::Error> {
+    let client = Client::try_default().await?;
+    let secrets: Api<Secret> = Api::default_namespaced(client);
+    debug!(
+        "Checking for WireGuard configs already assigned to instance {}",
+        instance_id
+    );
+    let lp = ListParams::default()
+        .fields("type=dynk8s.du5t1n.me/wireguard-config")
+        .labels(&format!("dynk8s.du5t1n.me/ec2-instance-id={}", instance_id));
+    let res = secrets.list(&lp).await?;
+    if !res.items.is_empty() {
+        info!(
+            "WireGuard config already assigned to instance {}",
+            instance_id
+        );
+        return Ok(());
+    }
+    debug!("Looking for available WireGuard configs");
+    let lp = ListParams::default()
+        .fields("type=dynk8s.du5t1n.me/wireguard-config")
+        .labels("dynk8s.du5t1n.me/ec2-instance-id=");
+    let res = secrets.list(&lp).await?;
+    if res.items.is_empty() {
+        error!(
+            "No WireGuard config available for instance {}",
+            &instance_id
+        );
+    } else {
+        if let Some(name) = &res.items[0].metadata.name {
+            let mut labels = BTreeMap::<String, String>::new();
+            labels.insert(
+                "dynk8s.du5t1n.me/ec2-instance-id".into(),
+                instance_id.into(),
+            );
+            let mut secret = Secret::default();
+            secret.metadata.labels = Some(labels);
+            let patch = Patch::Apply(&secret);
+            let pp = PatchParams::apply(env!("CARGO_PKG_NAME")).force();
+            secrets.patch(&name, &pp, &patch).await?;
+            info!(
+                "Assigned WireGuard config {} to instance {}",
+                name, &instance_id
+            );
+        }
+    }
+    Ok(())
+}
+
+/// Unassign all WireGuard configs from the specified EC2 instance
+///
+/// This function finds all WireGuard configs, stored as Kubernetes Secret
+/// resources, associated with the specified EC2 instance and unassigns them.
+/// Unassigned configs have the `dynk8s.du5t1n.me/ec2-instance-id` label set to
+/// the empty string.
+pub async fn unassign_wireguard_config(
+    instance_id: &str,
+) -> Result<(), kube::Error> {
+    let client = Client::try_default().await?;
+    let secrets: Api<Secret> = Api::default_namespaced(client);
+    let lp = ListParams::default()
+        .fields("type=dynk8s.du5t1n.me/wireguard-config")
+        .labels(&format!("dynk8s.du5t1n.me/ec2-instance-id={}", instance_id));
+    info!(
+        "Unassigning WireGuard configs from instance {}",
+        instance_id
+    );
+    for s in secrets.list(&lp).await? {
+        if let Some(name) = &s.metadata.name {
+            let mut labels = BTreeMap::<String, String>::new();
+            labels
+                .insert("dynk8s.du5t1n.me/ec2-instance-id".into(), "".into());
+            let mut secret = Secret::default();
+            secret.metadata.labels = Some(labels);
+            let patch = Patch::Apply(&secret);
+            let pp = PatchParams::apply(env!("CARGO_PKG_NAME")).force();
+            secrets.patch(&name, &pp, &patch).await?;
+            info!(
+                "Unassigned WireGuard config {} from instance {}",
+                name, &instance_id
+            );
+        }
+    }
+    Ok(())
+}
+
+/// Retrieve the WireGuard config assigned to the specified EC2 instance
+///
+/// This function finds the first WireGuard client configuration, stored as a
+/// Kubernetes Secret resource, associated with the specified EC2 instance.
+///
+/// If multiple WireGuard configs are assigned to an EC2 instance, only the
+/// first one returned by the Kubernetes list query is returned.
+pub async fn get_wireguard_config(
+    instance_id: &str,
+) -> Result<Option<String>, kube::Error> {
+    let client = Client::try_default().await?;
+    let secrets: Api<Secret> = Api::default_namespaced(client);
+    let lp = ListParams::default()
+        .fields("type=dynk8s.du5t1n.me/wireguard-config")
+        .labels(&format!("dynk8s.du5t1n.me/ec2-instance-id={}", instance_id));
+    for s in secrets.list(&lp).await? {
+        if let Some(data) = s.data {
+            match data.get("wireguard-config") {
+                Some(s) => match String::from_utf8(s.0.clone()) {
+                    Ok(s) => return Ok(Some(s)),
+                    Err(e) => {
+                        error!("Invalid WireGuard configuration: {}", e);
+                    }
+                },
+                None => {
+                    error!(concat!(
+                        "Invalid WireGuard configuration: ",
+                        "missing wireguard-config property"
+                    ));
+                }
+            };
+        }
+    }
+    Ok(None)
+}
+
+/// Generate and store a bootstrap token for the specified EC2 instance
+///
+/// This function generates a new bootstrap token and stores it as a Kubernetes
+/// Secret resource.  The token is assigned to the given EC2 instance, and will
+/// be provided to that instance when it is ready to join the cluster.
+pub async fn create_bootstrap_token<I: AsRef<str>>(
+    instance_id: I,
+) -> Result<(), kube::Error> {
+    let instance_id = instance_id.as_ref();
+    info!("Creating bootstrap token for instance {}", instance_id);
+    let token = BootstrapToken::new().instance_id(instance_id.into());
+    let client = Client::try_default().await?;
+    let secrets: Api<Secret> = Api::namespaced(client, "kube-system");
+    let pp: PostParams = Default::default();
+    let secret = secrets.create(&pp, &token.into()).await?;
+    info!("Successfully created secret {:?}", &secret.metadata.name);
+    Ok(())
+}
+
+/// Delete bootstrap tokens associated withthe specified EC2 instance
+pub async fn delete_bootstrap_tokens<I: AsRef<str>>(
+    instance_id: I,
+) -> Result<(), kube::Error> {
+    let instance_id = instance_id.as_ref();
+    info!("Deleting bootstrap tokens for instance {}", &instance_id);
+    let client = Client::try_default().await?;
+    let secrets: Api<Secret> = Api::namespaced(client, "kube-system");
+    let lp = ListParams::default()
+        .fields("type=bootstrap.kubernetes.io/token")
+        .labels(&format!("dynk8s.du5t1n.me/ec2-instance-id={}", &instance_id));
+    secrets.delete_collection(&Default::default(), &lp).await?;
+    Ok(())
+}
+
+/// Get the `kubeadm join` configuration for the specified EC2 instance
+///
+/// This function creates a kubeconfig file that can be passed to `kubeadm
+/// join` to add the specified EC2 instance to the Kubernetes cluster as a
+/// worker node.  The cluster configuration is read from the `cluster-info`
+/// ConfigMap in the *kube-public* namespace.  The bootstrap token assigned to
+/// the instance is included for client authentication.
+pub async fn get_kubeconfig<I: AsRef<str>>(
+    instance_id: I,
+) -> Result<Option<KubeConfig>, kube::Error> {
+    let instance_id = instance_id.as_ref();
+    let token = match get_bootstrap_token(&instance_id).await {
+        Ok(Some(t)) => t,
+        Ok(None) => {
+            warn!("No bootstrap token assigned to instance {}", &instance_id);
+            return Ok(None);
+        }
+        Err(e) => {
+            error!(
+                "Could not get bootstrap token for instance {}: {}",
+                &instance_id, e
+            );
+            return Ok(None);
+        }
+    };
+    match get_cluster_info().await? {
+        Some(config) => {
+            let cluster = Cluster {
+                name: "kubernetes".into(),
+                cluster: config.clusters[0].cluster.clone(),
+            };
+            let context = Context {
+                name: "kubeadm".into(),
+                context: ContextInfo {
+                    cluster: "kubernetes".into(),
+                    user: "kubeadm".into(),
+                },
+            };
+            let user = User {
+                name: "kubeadm".into(),
+                user: UserInfo { token: token },
+            };
+            let mut kubeconfig = KubeConfig::default();
+            kubeconfig.clusters = vec![cluster];
+            kubeconfig.contexts = Some(vec![context]);
+            kubeconfig.current_context = "kubeadm".into();
+            kubeconfig.users = Some(vec![user]);
+            Ok(Some(kubeconfig))
+        }
+        None => {
+            warn!("No kubeconfig loaded from cluster-info");
+            Ok(None)
+        }
+    }
+}
+
+/// Retrieve the bootstrap token assigned to an EC2 instance
+async fn get_bootstrap_token<I: AsRef<str>>(
+    instance_id: I,
+) -> Result<Option<String>, kube::Error> {
+    let instance_id = instance_id.as_ref();
+    let client = Client::try_default().await?;
+    let secrets: Api<Secret> = Api::namespaced(client, "kube-system");
+    let lp = ListParams::default()
+        .fields("type=bootstrap.kubernetes.io/token")
+        .labels(&format!(
+            "dynk8s.du5t1n.me/ec2-instance-id={}",
+            &instance_id
+        ));
+    for s in secrets.list(&lp).await? {
+        match token_string(&s) {
+            Ok(t) => return Ok(Some(t)),
+            Err(e) => {
+                error!("Invalid bootstrap token: {}", e);
+            }
+        }
+    }
+    Ok(None)
+}
+
+/// Get cluster information from the ConfigMap
+async fn get_cluster_info() -> Result<Option<KubeConfig>, kube::Error> {
+    let client = Client::try_default().await?;
+    let configmaps: Api<ConfigMap> = Api::namespaced(client, "kube-public");
+    let cluster_info = configmaps.get("cluster-info").await?;
+    if let Some(data) = cluster_info.data {
+        if let Some(config) = data.get("kubeconfig") {
+            match serde_yaml::from_str::<KubeConfig>(config) {
+                Ok(c) => return Ok(Some(c)),
+                Err(e) => {
+                    error!(
+                        "Could not load kubeconfig from cluster-info: {}",
+                        e
+                    );
+                }
+            };
+        } else {
+            error!("No kubeconfig property found in cluster-info ConfigMap");
+        }
+    } else {
+        error!("No data property found in cluster-info ConfigMap");
+    }
+    Ok(None)
+}
+
+/// Get the string representation of a bootstrap token from a Secret
+fn token_string(secret: &Secret) -> Result<String, String> {
+    let data = match &secret.data {
+        Some(d) => d,
+        None => return Err("Missing data property".into()),
+    };
+    let token_id = match data.get("token-id") {
+        Some(s) => match String::from_utf8(s.0.clone()) {
+            Ok(s) => s,
+            Err(e) => return Err(e.to_string()),
+        },
+        None => return Err("Missing token-id".into()),
+    };
+    let secret = match data.get("token-secret") {
+        Some(s) => match String::from_utf8(s.0.clone()) {
+            Ok(s) => s,
+            Err(e) => return Err(e.to_string()),
+        },
+        None => return Err("Missing token-secret".into()),
+    };
+    Ok(format!("{}.{}", token_id, secret))
+}
+
+#[cfg(test)]
+mod test {
+    use super::*;
+    use k8s_openapi::ByteString;
+    use regex::Regex;
+
+    #[test]
+    fn test_bootstrap_token_new() {
+        let token = BootstrapToken::new();
+        let id_re = Regex::new(r"^[a-z0-9]{6}$").unwrap();
+        let secret_re = Regex::new(r"^[a-z0-9]{16}$").unwrap();
+        let token_re = Regex::new(r"[a-z0-9]{6}\.[a-z0-9]{16}$").unwrap();
+        assert!(id_re.is_match(&token.token_id()));
+        assert!(secret_re.is_match(&token.secret()));
+        assert!(token_re.is_match(&token.token()));
+    }
+
+    #[test]
+    fn test_bootstrap_token_into_secret() {
+        let token = BootstrapToken::new();
+        let secret: Secret = token.clone().into();
+        let data = secret.string_data.unwrap();
+        assert_eq!(
+            &secret.metadata.name,
+            &Some(format!("bootstrap-token-{}", token.token_id))
+        );
+        assert_eq!(data.get("token-id").unwrap(), &token.token_id);
+        assert_eq!(data.get("token-secret").unwrap(), &token.secret);
+        assert_eq!(
+            data.get("expiration").unwrap(),
+            &token.expiration.to_rfc3339()
+        );
+        assert!(secret
+            .metadata
+            .labels
+            .unwrap()
+            .get("dynk8s.du5t1n.me/ec2-instance-id")
+            .is_none());
+    }
+
+    #[test]
+    fn test_bootstrap_token_into_secret_instance_id() {
+        let token =
+            BootstrapToken::new().instance_id("i-0a1b2c3d4e5f6f7f8".into());
+        let secret: Secret = token.clone().into();
+        let data = secret.string_data.unwrap();
+        assert_eq!(
+            &secret.metadata.name,
+            &Some(format!("bootstrap-token-{}", token.token_id))
+        );
+        assert_eq!(data.get("token-id").unwrap(), &token.token_id);
+        assert_eq!(data.get("token-secret").unwrap(), &token.secret);
+        assert_eq!(
+            data.get("expiration").unwrap(),
+            &token.expiration.to_rfc3339()
+        );
+        assert_eq!(
+            secret
+                .metadata
+                .labels
+                .unwrap()
+                .get("dynk8s.du5t1n.me/ec2-instance-id")
+                .unwrap(),
+            "i-0a1b2c3d4e5f6f7f8"
+        );
+    }
+
+    #[test]
+    fn test_token_string() {
+        let token = BootstrapToken::new();
+        let mut data = BTreeMap::new();
+        data.insert("token-id".into(), ByteString(token.token_id().into()));
+        data.insert("token-secret".into(), ByteString(token.secret().into()));
+        let secret = Secret {
+            data: Some(data),
+            ..Default::default()
+        };
+        assert_eq!(token.token(), token_string(&secret).unwrap());
+    }
+}
--- a/src/lib.rs
+++ b/src/lib.rs
@ -0,0 +1,25 @@
+mod error;
+mod events;
+mod k8s;
+mod model;
+mod routes;
+mod sns;
+
+#[doc(hidden)]
+pub fn rocket() -> rocket::Rocket<rocket::Build> {
+    rocket::build().mount(
+        "/",
+        rocket::routes![
+            routes::health::get_health,
+            routes::kubeadm::get_node_kubeconfig,
+            routes::kubeadm::post_node_kubeconfig,
+            routes::kubeadm::patch_node_kubeconfig,
+            routes::kubeadm::put_node_kubeconfig,
+            routes::wireguard::get_node_wireguard,
+            routes::sns::post_sns_notify,
+            routes::sns::get_sns_notify,
+            routes::sns::put_sns_notify,
+            routes::sns::patch_sns_notify,
+        ],
+    )
+}
--- a/src/main.rs
+++ b/src/main.rs
@ -1,19 +1,6 @@
-mod error;
-mod model;
-mod routes;
-mod sns;
+use dynk8s_provisioner::rocket;

-#[doc(hidden)]
-#[rocket::launch]
-fn rocket() -> _ {
-    rocket::build().mount(
-        "/",
-        rocket::routes![
-            routes::health::get_health,
-            routes::sns::post_sns_notify,
-            routes::sns::get_sns_notify,
-            routes::sns::put_sns_notify,
-            routes::sns::patch_sns_notify,
-        ],
-    )
+#[rocket::main]
+async fn main() {
+    let _ = rocket().launch().await;
 }
--- a/src/model/events.rs
+++ b/src/model/events.rs
@ -0,0 +1,46 @@
+//! Amazon EventBridge event types
+//!
+//! These data structures are sent by [Amazon EventBridge][0], encapsulated in
+//! SNS notification messages.
+//!
+//! [0]: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-what-is.html
+use serde::{Deserialize, Serialize};
+
+/// EC2 Instance State-change Notification
+///
+/// EventBridge event emitted when an EC2 instance changes state
+#[derive(Deserialize, Serialize)]
+#[serde(rename_all = "kebab-case")]
+pub struct Ec2InstanceStateChange {
+    pub instance_id: String,
+    pub state: String,
+}
+
+/// Enumeration of EventBridge detail objects
+///
+/// EventBridge events sent by AWS services include a `detail` property, the
+/// contents of which vary depending on the `detail-type` field.
+#[derive(Deserialize, Serialize)]
+#[serde(untagged)]
+pub enum EventDetail {
+    Ec2InstanceStateChange(Ec2InstanceStateChange),
+}
+
+/// EventBridge event
+///
+/// See also: [Amazon EventBridge events][0]
+///
+/// [0]: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html
+#[derive(Deserialize, Serialize)]
+#[serde(rename_all = "kebab-case")]
+pub struct Event {
+    pub version: String,
+    pub id: String,
+    pub detail_type: String,
+    pub source: String,
+    pub account: String,
+    pub time: String,
+    pub region: String,
+    pub resources: Vec<String>,
+    pub detail: EventDetail,
+}
--- a/src/model/k8s.rs
+++ b/src/model/k8s.rs
@ -0,0 +1,91 @@
+//! kubeadm configuration data types
+//!
+//! The Kubernetes API reference does not include a specification for the
+//! `Config` resource, and as such there is no model for it in [`k8s_openapi`].
+//! Since *dynk8s* needs to read and write objects of this type, to provide
+//! configuration for `kubeadm` on dynamic nodes, a subset of the required
+//! model is defined here.
+use serde::{Deserialize, Serialize};
+
+/// Cluster information
+#[derive(Clone, Debug, Deserialize, Serialize)]
+#[serde(rename_all = "kebab-case")]
+pub struct ClusterInfo {
+    /// X.509 certificate of the Kubernetes certificate authority
+    pub certificate_authority_data: String,
+    /// URL of the Kubernetes API server
+    pub server: String,
+}
+
+/// Cluster definition
+#[derive(Clone, Debug, Deserialize, Serialize)]
+pub struct Cluster {
+    /// Cluster information
+    pub cluster: ClusterInfo,
+    /// Cluster name
+    pub name: String,
+}
+
+/// kubeconfig context information
+#[derive(Clone, Debug, Deserialize, Serialize)]
+pub struct ContextInfo {
+    /// The cluster to use
+    pub cluster: String,
+    /// The user to use
+    pub user: String,
+}
+
+/// kubeconfig context definition
+#[derive(Clone, Debug, Deserialize, Serialize)]
+pub struct Context {
+    /// Context information
+    pub context: ContextInfo,
+    /// Context name
+    pub name: String,
+}
+
+/// User information
+#[derive(Clone, Debug, Deserialize, Serialize)]
+pub struct UserInfo {
+    /// Bootstrap token for authentication
+    pub token: String,
+}
+
+/// User definition
+#[derive(Clone, Debug, Deserialize, Serialize)]
+pub struct User {
+    /// User name
+    pub name: String,
+    /// User information
+    pub user: UserInfo,
+}
+
+/// kubeconfig
+#[derive(Clone, Debug, Deserialize, Serialize)]
+pub struct KubeConfig {
+    #[serde(rename = "apiVersion")]
+    pub api_version: String,
+    pub kind: String,
+    /// List of defined clusters
+    pub clusters: Vec<Cluster>,
+    /// List of defined contexts (user–cluster associations)
+    pub contexts: Option<Vec<Context>>,
+    #[serde(rename = "current-context")]
+    /// Current context
+    pub current_context: String,
+    /// List of defined users
+    pub users: Option<Vec<User>>,
+}
+
+impl Default for KubeConfig {
+    fn default() -> Self {
+        Self {
+            api_version: "v1".into(),
+            kind: "Config".into(),
+            clusters: vec![],
+            contexts: None,
+            current_context: "".into(),
+            users: None,
+        }
+    }
+}
--- a/src/model/mod.rs
+++ b/src/model/mod.rs
@ -1,2 +1,4 @@
 //! The dynk8s provisioner data model
+pub mod events;
+pub mod k8s;
 pub mod sns;
--- a/src/routes/kubeadm.rs
+++ b/src/routes/kubeadm.rs
@ -0,0 +1,72 @@
+use rocket::http::Status;
+
+use crate::k8s::get_kubeconfig;
+
+#[rocket::get("/kubeadm/kubeconfig/<instance_id>")]
+pub async fn get_node_kubeconfig(instance_id: String) -> Option<String> {
+    if let Ok(Some(kubeconfig)) = get_kubeconfig(&instance_id).await {
+        Some(serde_yaml::to_string(&kubeconfig).unwrap())
+    } else {
+        None
+    }
+}
+
+#[rocket::post("/kubeadm/kubeconfig/<_instance_id>")]
+pub async fn post_node_kubeconfig(_instance_id: String) -> Status {
+    Status::MethodNotAllowed
+}
+
+#[rocket::patch("/kubeadm/kubeconfig/<_instance_id>")]
+pub async fn patch_node_kubeconfig(_instance_id: String) -> Status {
+    Status::MethodNotAllowed
+}
+
+#[rocket::put("/kubeadm/kubeconfig/<_instance_id>")]
+pub async fn put_node_kubeconfig(_instance_id: String) -> Status {
+    Status::MethodNotAllowed
+}
+
+#[cfg(test)]
+mod test {
+    use super::*;
+    use crate::rocket;
+    use rocket::local::blocking::Client;
+    use rocket::uri;
+
+    #[test]
+    fn test_get_node_token_404() {
+        let client = Client::tracked(rocket()).unwrap();
+        let res = client
+            .get(uri!(get_node_kubeconfig(
+                instance_id = "i-0a1b2c3d4e5f6f7f8"
+            )))
+            .dispatch();
+        assert_eq!(res.status(), Status::NotFound);
+    }
+
+    #[test]
+    fn test_kubeconfig_msg_wrong_method() {
+        let client = Client::tracked(rocket()).unwrap();
+
+        let res = client
+            .post(uri!(get_node_kubeconfig(
+                instance_id = "i-0a1b2c3d4e5f6f7f8"
+            )))
+            .dispatch();
+        assert_eq!(res.status(), Status::MethodNotAllowed);
+
+        let res = client
+            .patch(uri!(get_node_kubeconfig(
+                instance_id = "i-0a1b2c3d4e5f6f7f8"
+            )))
+            .dispatch();
+        assert_eq!(res.status(), Status::MethodNotAllowed);
+
+        let res = client
+            .put(uri!(get_node_kubeconfig(
+                instance_id = "i-0a1b2c3d4e5f6f7f8"
+            )))
+            .dispatch();
+        assert_eq!(res.status(), Status::MethodNotAllowed);
+    }
+}
--- a/src/routes/mod.rs
+++ b/src/routes/mod.rs
@ -1,3 +1,5 @@
 //! Rocket route handlers
 pub mod health;
+pub mod kubeadm;
 pub mod sns;
+pub mod wireguard;
--- a/src/routes/sns.rs
+++ b/src/routes/sns.rs
@ -50,7 +50,7 @@ mod test {
    fn test_sub_conf_msg() {
        let client = Client::tracked(rocket()).unwrap();
        let data = std::fs::read_to_string(
-            "test/data/sns/subscriptionconfirmation.json",
+            "tests/data/sns/subscriptionconfirmation.json",
        )
        .unwrap();
        let res = client.post(uri!(post_sns_notify)).body(&data).dispatch();
@ -68,7 +68,7 @@ mod test {
    fn test_sub_conf_msg_bad() {
        let client = Client::tracked(rocket()).unwrap();
        let data = std::fs::read_to_string(
-            "test/data/sns/subscriptionconfirmation-bad.json",
+            "tests/data/sns/subscriptionconfirmation-bad.json",
        )
        .unwrap();
        let res = client.post(uri!(post_sns_notify)).body(&data).dispatch();
@ -88,7 +88,7 @@ mod test {
    fn test_sub_conf_msg_bad_cert_url() {
        let client = Client::tracked(rocket()).unwrap();
        let data = std::fs::read_to_string(
-            "test/data/sns/subscriptionconfirmation-bad-url.json",
+            "tests/data/sns/subscriptionconfirmation-bad-url.json",
        )
        .unwrap();
        let res = client.post(uri!(post_sns_notify)).body(&data).dispatch();
--- a/src/routes/wireguard.rs
+++ b/src/routes/wireguard.rs
@ -0,0 +1,30 @@
+use crate::k8s::get_wireguard_config;
+
+#[rocket::get("/wireguard/config/<instance_id>")]
+pub async fn get_node_wireguard(instance_id: String) -> Option<String> {
+    if let Ok(Some(token)) = get_wireguard_config(&instance_id).await {
+        Some(token)
+    } else {
+        None
+    }
+}
+
+#[cfg(test)]
+mod test {
+    use super::*;
+    use crate::rocket;
+    use rocket::http::Status;
+    use rocket::local::blocking::Client;
+    use rocket::uri;
+
+    #[test]
+    fn test_get_node_wireguard_404() {
+        let client = Client::tracked(rocket()).unwrap();
+        let res = client
+            .get(uri!(get_node_wireguard(
+                instance_id = "i-0a1b2c3d4e5f6f7f8"
+            )))
+            .dispatch();
+        assert_eq!(res.status(), Status::NotFound);
+    }
+}
--- a/src/sns/mod.rs
+++ b/src/sns/mod.rs
@ -8,6 +8,8 @@ use log::{debug, error, info};
 use reqwest::Url;
 use serde::Serialize;

+use crate::events;
+use crate::model::events::*;
 use crate::model::sns::*;
 use error::SnsError;
 use sig::SignatureVerifier;
@ -38,11 +40,23 @@ pub async fn handle_unsubscribe(

 /// Handle an notification message
 ///
-/// After verifying the message signature, the message contents are written to
-/// a file for later inspection.
+/// This function handles varions SNS notification messages based on their
+/// contents/sub-type.
 pub async fn handle_notify(msg: NotificationMessage) -> Result<(), SnsError> {
    verify(&msg, &msg.signing_cert_url).await?;
    save_message(&msg.topic_arn, &msg.timestamp, &msg.message_id, &msg);
+    let event: Event = match serde_json::from_str(&msg.message) {
+        Ok(evt) => evt,
+        Err(e) => {
+            error!("Failed to deserialize notification message: {}", e);
+            return Ok(());
+        }
+    };
+    match event.detail {
+        EventDetail::Ec2InstanceStateChange(d) => {
+            events::on_ec2_instance_state_change(d).await;
+        }
+    };
    Ok(())
 }

--- a/terraform/.gitattributes
+++ b/terraform/.gitattributes
@ -0,0 +1,2 @@
+.terraform.lock.hcl -diff
+terraform.tfstate -diff
--- a/terraform/.gitignore
+++ b/terraform/.gitignore
@ -0,0 +1,2 @@
+.terraform/
+terraform.tfstate.backup
--- a/terraform/.terraform.lock.hcl
+++ b/terraform/.terraform.lock.hcl
@ -0,0 +1,22 @@
+# This file is maintained automatically by "terraform init".
+# Manual edits may be lost in future updates.
+
+provider "registry.terraform.io/hashicorp/aws" {
+  version     = "4.30.0"
+  constraints = "~> 4.16"
+  hashes = [
+    "h1:BFfhRf8my/aa0+YOSJv0xfjLQkToF475TJTMhTZfYec=",
+    "zh:08213f3ba960621448754211f148730edb59194919ee476b0231b769a5355028",
+    "zh:29c90d6f8bdae0e1469417ade28fa79c74c2af49593c1e2f24f07bacbca9e2c9",
+    "zh:5c6e9fab64ad68de6cd4ec6cbb20b0f75ba1e51a8efaeda3fe65419f096a06cb",
+    "zh:9b12af85486a96aedd8d7984b0ff811a4b42e3d88dad1a3fb4c0b580d04fa425",
+    "zh:9bf42718580e8c5097227df34e1bfa0a10a23eac9f527d97c2819c163087b402",
+    "zh:9f87e42e0f3d145fb0ad4aaff7ddded5720a64f9303956b33bd274c6dd05c05b",
+    "zh:bf0519ed9615bc408b72a0aebe1cc075d4c2042325590ba13dd264cd264907ea",
+    "zh:c3ac9e1cbd0935614f5a3c9cdb4cf9c6a1045937fe38e61da7c5c0fb7a069870",
+    "zh:d0c184476ada38c50acc068214ed1252b4fcf80b6be900fc1aed32cbb49f8ff6",
+    "zh:d4987dc7b7a69ea58f2b3ff0ea4ffc1b61a97881dbb8583c9fcf9444b753a6c2",
+    "zh:e8037376c81aeb98d8286dc19fba7f8eb053444d4b9484ea6a922382cffc1a85",
+    "zh:ecdabb44b48addc8483bca7bd683614a347367ae950ca8b6a6880679f5c12abd",
+  ]
+}
--- a/terraform/README.md
+++ b/terraform/README.md
@ -0,0 +1,20 @@
+# Configuring AWS Using Terraform
+
+## Prerequisites
+
+The IAM user or role that runs Terraform must have the appropriate permissions.
+The `iam-policy.json` file defines a policy that will allow the Terraform to
+manage all of the necessary resources.  Before running Terraform, create an IAM
+policy and assign it to a user, group, or role.  Be sure to replace the AWS
+account ID in the various target resource names.
+
+To use an IAM role, set the `iam_role` Terraform variable when executing
+`terraform plan`/`terraform apply`.
+
+## Create Resources
+
+Terraform will create all resources automatically:
+
+```sh
+terraform apply
+```
--- a/terraform/eventbridge.tf
+++ b/terraform/eventbridge.tf
@ -0,0 +1,16 @@
+resource "aws_cloudwatch_event_rule" "instance_state" {
+  name = "instance-state-events"
+
+  event_pattern = <<EOF
+{
+  "source": ["aws.ec2"],
+  "detail-type": ["EC2 Instance State-change Notification"]
+}
+EOF
+}
+
+resource "aws_cloudwatch_event_target" "sns" {
+  rule      = aws_cloudwatch_event_rule.instance_state.name
+  target_id = "SendToSNS"
+  arn       = aws_sns_topic.ec2_events.arn
+}
--- a/terraform/iam-policy.json
+++ b/terraform/iam-policy.json
@ -0,0 +1,49 @@
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Sid": "VisualEditor0",
+            "Effect": "Allow",
+            "Action": [
+                "sns:ListTagsForResource",
+                "events:DescribeRule",
+                "sns:GetTopicAttributes",
+                "events:EnableRule",
+                "sns:DeleteTopic",
+                "events:PutRule",
+                "sns:CreateTopic",
+                "sns:SetTopicAttributes",
+                "events:DeleteRule",
+                "events:PutTargets",
+                "events:ListTagsForResource",
+                "sns:Subscribe",
+                "events:RemoveTargets",
+                "events:ListTargetsByRule",
+                "events:DisableRule"
+            ],
+            "Resource": [
+                "arn:aws:sns:*:566967686773:*",
+                "arn:aws:events:*:566967686773:rule/*"
+            ]
+        },
+        {
+            "Sid": "VisualEditor1",
+            "Effect": "Allow",
+            "Action": [
+                "events:DescribeEventBus",
+                "events:CreateEventBus",
+                "events:DeleteEventBus"
+            ],
+            "Resource": "arn:aws:events:*:566967686773:event-bus/*"
+        },
+        {
+            "Sid": "VisualEditor2",
+            "Effect": "Allow",
+            "Action": [
+                "sns:Unsubscribe",
+                "sns:GetSubscriptionAttributes"
+            ],
+            "Resource": "*"
+        }
+    ]
+}
--- a/terraform/main.tf
+++ b/terraform/main.tf
@ -0,0 +1,19 @@
+terraform {
+  required_providers {
+    aws = {
+      source  = "hashicorp/aws"
+      version = "~> 4.16"
+    }
+  }
+  required_version = ">= 1.2.0"
+}
+
+provider "aws" {
+  region = var.ec2_region
+
+  assume_role {
+    role_arn = var.iam_role
+  }
+}
+
+data "aws_caller_identity" "current" {}
--- a/terraform/sns.tf
+++ b/terraform/sns.tf
@ -0,0 +1,61 @@
+data "aws_iam_policy_document" "ec2_events_access_policy" {
+  statement {
+    sid = "__default_statement_ID"
+    effect = "Allow"
+    principals {
+      type        = "AWS"
+      identifiers = ["*"]
+    }
+    actions = [
+      "SNS:Subscribe",
+      "SNS:SetTopicAttributes",
+      "SNS:RemovePermission",
+      "SNS:Receive",
+      "SNS:Publish",
+      "SNS:ListSubscriptionsByTopic",
+      "SNS:GetTopicAttributes",
+      "SNS:DeleteTopic",
+      "SNS:AddPermission",
+    ]
+    resources = [
+      aws_sns_topic.ec2_events.arn,
+    ]
+    condition {
+      test     = "StringEquals"
+      variable = "AWS:SourceOwner"
+
+      values = [
+        data.aws_caller_identity.current.account_id
+      ]
+    }
+
+  }
+
+  statement {
+    sid = "AllowEventBridgePublish"
+    effect = "Allow"
+    principals {
+      type        = "Service"
+      identifiers = ["events.amazonaws.com"]
+    }
+    actions   = ["sns:Publish"]
+    resources = [aws_sns_topic.ec2_events.arn]
+  }
+}
+
+resource "aws_sns_topic" "ec2_events" {
+  name = "ec2-events"
+}
+
+resource "aws_sns_topic_policy" "ec2_events_policy" {
+  arn    = aws_sns_topic.ec2_events.arn
+  policy = data.aws_iam_policy_document.ec2_events_access_policy.json
+}
+
+resource "aws_sns_topic_subscription" "dynk8s_provisoner" {
+  topic_arn = aws_sns_topic.ec2_events.arn
+  protocol  = "https"
+  endpoint  = "https://dynk8s-provisioner.pyrocufflink.net/sns/notify"
+}
+
+
--- a/terraform/terraform.tfstate
+++ b/terraform/terraform.tfstate
@ -0,0 +1,275 @@
+{
+  "version": 4,
+  "terraform_version": "1.2.9",
+  "serial": 49,
+  "lineage": "a100be74-c98e-0769-2d6a-bf6a2c5f3ebf",
+  "outputs": {},
+  "resources": [
+    {
+      "mode": "data",
+      "type": "aws_caller_identity",
+      "name": "current",
+      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
+      "instances": [
+        {
+          "schema_version": 0,
+          "attributes": {
+            "account_id": "566967686773",
+            "arn": "arn:aws:sts::566967686773:assumed-role/dynk8s-terraform/aws-go-sdk-1664301518318294107",
+            "id": "566967686773",
+            "user_id": "AROAYIAPIKZ25DFDOYZHT:aws-go-sdk-1664301518318294107"
+          },
+          "sensitive_attributes": []
+        }
+      ]
+    },
+    {
+      "mode": "data",
+      "type": "aws_iam_policy_document",
+      "name": "ec2_events_access_policy",
+      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
+      "instances": [
+        {
+          "schema_version": 0,
+          "attributes": {
+            "id": "2883441170",
+            "json": "{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Sid\": \"__default_statement_ID\",\n      \"Effect\": \"Allow\",\n      \"Action\": [\n        \"SNS:Subscribe\",\n        \"SNS:SetTopicAttributes\",\n        \"SNS:RemovePermission\",\n        \"SNS:Receive\",\n        \"SNS:Publish\",\n        \"SNS:ListSubscriptionsByTopic\",\n        \"SNS:GetTopicAttributes\",\n        \"SNS:DeleteTopic\",\n        \"SNS:AddPermission\"\n      ],\n      \"Resource\": \"arn:aws:sns:us-east-2:566967686773:ec2-events\",\n      \"Principal\": {\n        \"AWS\": \"*\"\n      },\n      \"Condition\": {\n        \"StringEquals\": {\n          \"AWS:SourceOwner\": \"566967686773\"\n        }\n      }\n    },\n    {\n      \"Sid\": \"AllowEventBridgePublish\",\n      \"Effect\": \"Allow\",\n      \"Action\": \"sns:Publish\",\n      \"Resource\": \"arn:aws:sns:us-east-2:566967686773:ec2-events\",\n      \"Principal\": {\n        \"Service\": \"events.amazonaws.com\"\n      }\n    }\n  ]\n}",
+            "override_json": null,
+            "override_policy_documents": null,
+            "policy_id": null,
+            "source_json": null,
+            "source_policy_documents": null,
+            "statement": [
+              {
+                "actions": [
+                  "SNS:AddPermission",
+                  "SNS:DeleteTopic",
+                  "SNS:GetTopicAttributes",
+                  "SNS:ListSubscriptionsByTopic",
+                  "SNS:Publish",
+                  "SNS:Receive",
+                  "SNS:RemovePermission",
+                  "SNS:SetTopicAttributes",
+                  "SNS:Subscribe"
+                ],
+                "condition": [
+                  {
+                    "test": "StringEquals",
+                    "values": [
+                      "566967686773"
+                    ],
+                    "variable": "AWS:SourceOwner"
+                  }
+                ],
+                "effect": "Allow",
+                "not_actions": [],
+                "not_principals": [],
+                "not_resources": [],
+                "principals": [
+                  {
+                    "identifiers": [
+                      "*"
+                    ],
+                    "type": "AWS"
+                  }
+                ],
+                "resources": [
+                  "arn:aws:sns:us-east-2:566967686773:ec2-events"
+                ],
+                "sid": "__default_statement_ID"
+              },
+              {
+                "actions": [
+                  "sns:Publish"
+                ],
+                "condition": [],
+                "effect": "Allow",
+                "not_actions": [],
+                "not_principals": [],
+                "not_resources": [],
+                "principals": [
+                  {
+                    "identifiers": [
+                      "events.amazonaws.com"
+                    ],
+                    "type": "Service"
+                  }
+                ],
+                "resources": [
+                  "arn:aws:sns:us-east-2:566967686773:ec2-events"
+                ],
+                "sid": "AllowEventBridgePublish"
+              }
+            ],
+            "version": "2012-10-17"
+          },
+          "sensitive_attributes": []
+        }
+      ]
+    },
+    {
+      "mode": "managed",
+      "type": "aws_cloudwatch_event_rule",
+      "name": "instance_state",
+      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
+      "instances": [
+        {
+          "schema_version": 0,
+          "attributes": {
+            "arn": "arn:aws:events:us-east-2:566967686773:rule/instance-state-events",
+            "description": "",
+            "event_bus_name": "default",
+            "event_pattern": "{\"detail-type\":[\"EC2 Instance State-change Notification\"],\"source\":[\"aws.ec2\"]}",
+            "id": "instance-state-events",
+            "is_enabled": true,
+            "name": "instance-state-events",
+            "name_prefix": "",
+            "role_arn": "",
+            "schedule_expression": "",
+            "tags": {},
+            "tags_all": {}
+          },
+          "sensitive_attributes": [],
+          "private": "bnVsbA=="
+        }
+      ]
+    },
+    {
+      "mode": "managed",
+      "type": "aws_cloudwatch_event_target",
+      "name": "sns",
+      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
+      "instances": [
+        {
+          "schema_version": 1,
+          "attributes": {
+            "arn": "arn:aws:sns:us-east-2:566967686773:ec2-events",
+            "batch_target": [],
+            "dead_letter_config": [],
+            "ecs_target": [],
+            "event_bus_name": "default",
+            "http_target": [],
+            "id": "instance-state-events-SendToSNS",
+            "input": "",
+            "input_path": "",
+            "input_transformer": [],
+            "kinesis_target": [],
+            "redshift_target": [],
+            "retry_policy": [],
+            "role_arn": "",
+            "rule": "instance-state-events",
+            "run_command_targets": [],
+            "sqs_target": [],
+            "target_id": "SendToSNS"
+          },
+          "sensitive_attributes": [],
+          "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
+          "dependencies": [
+            "aws_cloudwatch_event_rule.instance_state",
+            "aws_sns_topic.ec2_events"
+          ]
+        }
+      ]
+    },
+    {
+      "mode": "managed",
+      "type": "aws_sns_topic",
+      "name": "ec2_events",
+      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
+      "instances": [
+        {
+          "schema_version": 0,
+          "attributes": {
+            "application_failure_feedback_role_arn": "",
+            "application_success_feedback_role_arn": "",
+            "application_success_feedback_sample_rate": 0,
+            "arn": "arn:aws:sns:us-east-2:566967686773:ec2-events",
+            "content_based_deduplication": false,
+            "delivery_policy": "",
+            "display_name": "",
+            "fifo_topic": false,
+            "firehose_failure_feedback_role_arn": "",
+            "firehose_success_feedback_role_arn": "",
+            "firehose_success_feedback_sample_rate": 0,
+            "http_failure_feedback_role_arn": "",
+            "http_success_feedback_role_arn": "",
+            "http_success_feedback_sample_rate": 0,
+            "id": "arn:aws:sns:us-east-2:566967686773:ec2-events",
+            "kms_master_key_id": "",
+            "lambda_failure_feedback_role_arn": "",
+            "lambda_success_feedback_role_arn": "",
+            "lambda_success_feedback_sample_rate": 0,
+            "name": "ec2-events",
+            "name_prefix": "",
+            "owner": "566967686773",
+            "policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"__default_statement_ID\",\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"*\"},\"Action\":[\"SNS:Subscribe\",\"SNS:SetTopicAttributes\",\"SNS:RemovePermission\",\"SNS:Receive\",\"SNS:Publish\",\"SNS:ListSubscriptionsByTopic\",\"SNS:GetTopicAttributes\",\"SNS:DeleteTopic\",\"SNS:AddPermission\"],\"Resource\":\"arn:aws:sns:us-east-2:566967686773:ec2-events\",\"Condition\":{\"StringEquals\":{\"AWS:SourceOwner\":\"566967686773\"}}},{\"Sid\":\"AllowEventBridgePublish\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"events.amazonaws.com\"},\"Action\":\"sns:Publish\",\"Resource\":\"arn:aws:sns:us-east-2:566967686773:ec2-events\"}]}",
+            "sqs_failure_feedback_role_arn": "",
+            "sqs_success_feedback_role_arn": "",
+            "sqs_success_feedback_sample_rate": 0,
+            "tags": {},
+            "tags_all": {}
+          },
+          "sensitive_attributes": [],
+          "private": "bnVsbA=="
+        }
+      ]
+    },
+    {
+      "mode": "managed",
+      "type": "aws_sns_topic_policy",
+      "name": "ec2_events_policy",
+      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
+      "instances": [
+        {
+          "schema_version": 0,
+          "attributes": {
+            "arn": "arn:aws:sns:us-east-2:566967686773:ec2-events",
+            "id": "arn:aws:sns:us-east-2:566967686773:ec2-events",
+            "owner": "566967686773",
+            "policy": "{\"Statement\":[{\"Action\":[\"SNS:Subscribe\",\"SNS:SetTopicAttributes\",\"SNS:RemovePermission\",\"SNS:Receive\",\"SNS:Publish\",\"SNS:ListSubscriptionsByTopic\",\"SNS:GetTopicAttributes\",\"SNS:DeleteTopic\",\"SNS:AddPermission\"],\"Condition\":{\"StringEquals\":{\"AWS:SourceOwner\":\"566967686773\"}},\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"*\"},\"Resource\":\"arn:aws:sns:us-east-2:566967686773:ec2-events\",\"Sid\":\"__default_statement_ID\"},{\"Action\":\"sns:Publish\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"events.amazonaws.com\"},\"Resource\":\"arn:aws:sns:us-east-2:566967686773:ec2-events\",\"Sid\":\"AllowEventBridgePublish\"}],\"Version\":\"2012-10-17\"}"
+          },
+          "sensitive_attributes": [],
+          "private": "bnVsbA==",
+          "dependencies": [
+            "aws_sns_topic.ec2_events",
+            "data.aws_caller_identity.current",
+            "data.aws_iam_policy_document.ec2_events_access_policy"
+          ]
+        }
+      ]
+    },
+    {
+      "mode": "managed",
+      "type": "aws_sns_topic_subscription",
+      "name": "dynk8s_provisoner",
+      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
+      "instances": [
+        {
+          "schema_version": 0,
+          "attributes": {
+            "arn": "arn:aws:sns:us-east-2:566967686773:ec2-events:3be7a77b-4b89-4925-bad3-dab7223d07f3",
+            "confirmation_timeout_in_minutes": 1,
+            "confirmation_was_authenticated": false,
+            "delivery_policy": "",
+            "endpoint": "https://dynk8s-provisioner.pyrocufflink.net/sns/notify",
+            "endpoint_auto_confirms": false,
+            "filter_policy": "",
+            "id": "arn:aws:sns:us-east-2:566967686773:ec2-events:3be7a77b-4b89-4925-bad3-dab7223d07f3",
+            "owner_id": "566967686773",
+            "pending_confirmation": false,
+            "protocol": "https",
+            "raw_message_delivery": false,
+            "redrive_policy": "",
+            "subscription_role_arn": "",
+            "topic_arn": "arn:aws:sns:us-east-2:566967686773:ec2-events"
+          },
+          "sensitive_attributes": [],
+          "private": "bnVsbA==",
+          "dependencies": [
+            "aws_sns_topic.ec2_events"
+          ]
+        }
+      ]
+    }
+  ]
+}
--- a/terraform/variables.tf
+++ b/terraform/variables.tf
@ -0,0 +1,11 @@
+variable "ec2_region" {
+  description = "AWS region where all resources will be created"
+  type        = string
+  default     = "us-east-2"
+}
+
+variable "iam_role" {
+  description = "(Optional) IAM role ARN to assume when managing resources"
+  type        = string
+  default     = null
+}
--- a/tests/data/sns/notification-running.json
+++ b/tests/data/sns/notification-running.json
@ -0,0 +1,11 @@
+{
+  "Type": "Notification",
+  "MessageId": "b87b7cca-d698-552b-bf91-2764a57ca6a8",
+  "TopicArn": "arn:aws:sns:us-east-2:566967686773:ec2-events",
+  "Message": "{\"version\":\"0\",\"id\":\"ae9cb41b-2d06-c688-8d20-82e90720cfa4\",\"detail-type\":\"EC2 Instance State-change Notification\",\"source\":\"aws.ec2\",\"account\":\"566967686773\",\"time\":\"2022-09-28T16:09:08Z\",\"region\":\"us-east-2\",\"resources\":[\"arn:aws:ec2:us-east-2:566967686773:instance/i-0e50d560c8bf9f0f8\"],\"detail\":{\"instance-id\":\"i-0e50d560c8bf9f0f8\",\"state\":\"running\"}}",
+  "Subject": null,
+  "Timestamp": "2022-09-28T16:09:08.795Z",
+  "SignatureVersion": "1",
+  "Signature": "LBaVKayZhbAzQa8k78oKNUgo62KqFU/GZJ8GCkNmMJJSeAyZNFOPY7Rcy7x3tmvXnds9ns/xWnzgrK2SdD/Q0Zniu9st2o1lDKfQnbUHU1Wv7g65jOuXTJlBu6teuziJ/bpsFTv4z9yw4fPm7gNYZ3xF5yjwXn0j0IHv92YgzEJtewV+MKRgMtp0vq9+TvRhpOiYZC8DCWZdqQoHm7B9VruPPPG9yHSZ2eF9H8cOQgjRA0IweanGpU+qej6Ts0IMeMTPPcPzhSynCdQJ8UfBSNPPfnlkkbq714XUjcms01UcMFuAQyMHrrr9CsTOtOJn4R/h9qccjZ7DhzXSkGVocw==",
+  "SigningCertURL": "https://sns.us-east-2.amazonaws.com/SimpleNotificationService-56e67fcb41f6fec09b0196692625d385.pem"
+}
--- a/tests/data/sns/notification-terminated.json
+++ b/tests/data/sns/notification-terminated.json
@ -0,0 +1,11 @@
+{
+  "Type": "Notification",
+  "MessageId": "7d38c409-0edc-5cd0-87cd-9fa77aff18bc",
+  "TopicArn": "arn:aws:sns:us-east-2:566967686773:ec2-events",
+  "Message": "{\"version\":\"0\",\"id\":\"49c93129-5cb6-c94f-54c9-3b0c4fc07b03\",\"detail-type\":\"EC2 Instance State-change Notification\",\"source\":\"aws.ec2\",\"account\":\"566967686773\",\"time\":\"2022-09-28T16:10:11Z\",\"region\":\"us-east-2\",\"resources\":[\"arn:aws:ec2:us-east-2:566967686773:instance/i-0e50d560c8bf9f0f8\"],\"detail\":{\"instance-id\":\"i-0e50d560c8bf9f0f8\",\"state\":\"terminated\"}}",
+  "Subject": null,
+  "Timestamp": "2022-09-28T16:10:11.889Z",
+  "SignatureVersion": "1",
+  "Signature": "YVeRPXDW4zpPhBow/Le73zN1RJ754a/o9IHBAG8wSDNC2lLushh0ztknfp4G1qNA5ZenrKgTKyta0avLECh2qiU9lhp+M5qeY9CRIW2SH9xAL25MIN7psxkhJgSaqblIBENUE8gjge0pzDuKepgY5zD9f68Uf/0voMWTVWjcI2IdtOoS3LMqx6XC9K8LYMPgPf+0wGdH7AUXumQJTNBwX5jjMql2uIccN/xQxmUmIRHubWvDX82m78PZL+2mZlbGvsRRpkSPBPT/xu1fBwWkLQ0ziOf8sX7FyyKMKn8x4ohLkQjwU/21nSY8ei4n749ggJRNB0CxfQFPnNW5/iA4Qw==",
+  "SigningCertURL": "https://sns.us-east-2.amazonaws.com/SimpleNotificationService-56e67fcb41f6fec09b0196692625d385.pem"
+}
--- a/tests/data/sns/subscriptionconfirmation-bad-url.json
+++ b/tests/data/sns/subscriptionconfirmation-bad-url.json
--- a/tests/data/sns/subscriptionconfirmation-bad.json
+++ b/tests/data/sns/subscriptionconfirmation-bad.json
--- a/tests/data/sns/subscriptionconfirmation.json
+++ b/tests/data/sns/subscriptionconfirmation.json
--- a/tests/genkubeconfig.sh
+++ b/tests/genkubeconfig.sh
@ -0,0 +1,38 @@
+#!/bin/sh
+
+server=$(
+    kubectl config view --minify --raw \
+        -o jsonpath='{.clusters[].cluster.server}'
+)
+token=$(
+    kubectl get secret \
+        -n dynk8s-test dynk8s-provisioner \
+        -o jsonpath='{.data.token}' \
+    | base64 -d
+)
+ca=$(
+    kubectl get secret \
+        -n dynk8s-test dynk8s-provisioner \
+        -o jsonpath='{.data.ca\.crt}'
+)
+
+cat <<EOF
+apiVersion: v1
+kind: Config
+clusters:
+- name: kubernetes
+  cluster:
+    certificate-authority-data: ${ca}
+    server: ${server}
+contexts:
+- name: dynk8s-test@kubernetes
+  context:
+    cluster: kubernetes
+    namespace: dynk8s-test
+    user: dynk8s-test
+current-context: dynk8s-test@kubernetes
+users:
+- name: dynk8s-test
+  user:
+    token: ${token}
+EOF
--- a/tests/integration/main.rs
+++ b/tests/integration/main.rs
@ -0,0 +1,65 @@
+//! *dynk8s-provisioner* Integration Tests
+//!
+//! Create the test resources:
+//!
+//! ```sh
+//! kubectl apply -f tests/setup.yaml
+//! ```
+//!
+//! Generate a kubeconfig for the test service account:
+//!
+//! ```sh
+//! sh tests/genkubeconfig.sh > kubeconfig
+//! ```
+//!
+//! Run the tests:
+//!
+//! ```sh
+//! KUBECONFIG=${PWD}/kubeconfig cargo test
+//! ```
+use std::collections::btree_map::BTreeMap;
+
+use k8s_openapi::api::core::v1::Secret;
+use kube::core::params::ListParams;
+
+mod sns;
+
+const WIREGUARD_CONFIG: &str = "\
+[Interface]
+Address = 10.11.12.13/14
+PrivateKey = UEdAkIaF80zhlOpgacOYL2UckrfCAWXfsDDSAAzNH3g=
+
+[Peer]
+PublicKey = zbeTpUFA014kvTezIEGBt4yi3BVocST9j1dBElp9liI=
+PreSharedKey = V6hAm01dxv2ib8AML2dSyX68hlPZm8En+IXfsknK3Zc=
+AllowedIPs = 0.0.0.0/0
+Endpoint = wireguard.example.org:24680
+";
+
+async fn setup() {
+    let client = kube::Client::try_default().await.unwrap();
+    let secrets: kube::Api<Secret> = kube::Api::default_namespaced(client);
+    let lp =
+        ListParams::default().fields("type=dynk8s.du5t1n.me/wireguard-config");
+    secrets
+        .delete_collection(&Default::default(), &lp)
+        .await
+        .unwrap();
+
+    let mut labels = BTreeMap::<String, String>::new();
+    labels.insert(
+        "app.kubernetes.io/part-of".into(),
+        "dynk8s-provisioner".into(),
+    );
+    labels.insert("dynk8s.du5t1n.me/ec2-instance-id".into(), "".into());
+    let mut data = BTreeMap::<String, String>::new();
+    data.insert("wireguard-config".into(), WIREGUARD_CONFIG.into());
+    let mut secret = Secret::default();
+    secret.type_ = Some("dynk8s.du5t1n.me/wireguard-config".into());
+    secret.immutable = Some(true);
+    secret.metadata.generate_name = Some("wireguard-config-".into());
+    secret.metadata.labels = Some(labels);
+    secret.string_data = Some(data);
+    secrets.create(&Default::default(), &secret).await.unwrap();
+}
+
--- a/tests/integration/sns.rs
+++ b/tests/integration/sns.rs
@ -0,0 +1,57 @@
+use rocket::http::Status;
+use rocket::local::asynchronous::Client;
+
+use dynk8s_provisioner::rocket;
+
+use crate::{setup, WIREGUARD_CONFIG};
+
+#[rocket::async_test]
+async fn test_sns_ec2_lifecycle() {
+    setup().await;
+    let client = Client::tracked(rocket()).await.unwrap();
+
+    // Simulate an instance state-change event indicating an instance has
+    // started. This should generate a bootstrap token and assign the WireGuard
+    // config to the instance.
+    let data =
+        std::fs::read_to_string("tests/data/sns/notification-running.json")
+            .unwrap();
+    let res = client.post("/sns/notify").body(&data).dispatch().await;
+    assert_eq!(res.status(), Status::NoContent);
+
+    // Ensure the bootstrap token was generated
+    let res = client
+        .get("/kubeadm/kubeconfig/i-0e50d560c8bf9f0f8")
+        .dispatch()
+        .await;
+    assert_eq!(res.status(), Status::Ok);
+    // Ensure the WireGuard config was assigned
+    let res = client
+        .get("/wireguard/config/i-0e50d560c8bf9f0f8")
+        .dispatch()
+        .await;
+    assert_eq!(res.status(), Status::Ok);
+    assert_eq!(res.into_string().await, Some(WIREGUARD_CONFIG.into()));
+
+    // Simulate an instance state-change event indicating the instance has
+    // terminated.  This should delete the bootstrap token and unassing the
+    // WireGuard config.
+    let data =
+        std::fs::read_to_string("tests/data/sns/notification-terminated.json")
+            .unwrap();
+    let res = client.post("/sns/notify").body(&data).dispatch().await;
+    assert_eq!(res.status(), Status::NoContent);
+
+    // Ensure the bootstrap token was deleted
+    let res = client
+        .get("/kubeadm/kubeconfig/i-0e50d560c8bf9f0f8")
+        .dispatch()
+        .await;
+    assert_eq!(res.status(), Status::NotFound);
+    // Ensure the WireGuard config was deleted
+    let res = client
+        .get("/wireguard/config/i-0e50d560c8bf9f0f8")
+        .dispatch()
+        .await;
+    assert_eq!(res.status(), Status::NotFound);
+}
--- a/tests/setup.yaml
+++ b/tests/setup.yaml
@ -0,0 +1,76 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: dynk8s-test
+  labels:
+    kubernetes.io/metadata.name: dynk8s
+
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: dynk8s-provisioner
+  namespace: dynk8s-test
+  labels:
+    app.kubernetes.io/name: dynk8s-provisioner
+    app.kubernetes.io/instance: default
+    app.kubernetes.io/component: http-api
+    app.kubernetes.io/part-of: dynk8s-provisioner
+automountServiceAccountToken: true
+
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: dynk8s-provisioner
+  namespace: dynk8s-test
+  annotations:
+    kubernetes.io/service-account.name: dynk8s-provisioner
+type: kubernetes.io/service-account-token
+
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: dynk8s-provisioner
+  namespace: dynk8s-test
+  labels:
+    app.kubernetes.io/name: dynk8s-provisioner
+    app.kubernetes.io/instance: default
+    app.kubernetes.io/component: http-api
+    app.kubernetes.io/part-of: dynk8s-provisioner
+rules:
+- apiGroups:
+  - ''
+  resources:
+  - secrets
+  verbs:
+  - '*'
+- apiGroups:
+  - ''
+  resources:
+  - configmaps
+  resourceNames:
+  - cluster-info
+  verbs:
+  - get
+
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: dynk8s-provisioner
+  namespace: dynk8s-test
+  labels:
+    app.kubernetes.io/name: dynk8s-provisioner
+    app.kubernetes.io/instance: default
+    app.kubernetes.io/part-of: dynk8s-provisioner
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: dynk8s-provisioner
+subjects:
+- kind: ServiceAccount
+  name: dynk8s-provisioner
+  namespace: dynk8s-test
Author	SHA1	Message	Date
Dustin	d85f314a8b	tests: Begin integration tests dustin/dynk8s-provisioner/pipeline/head There was a failure building this commit Details Cargo uses the sources in the `tests` directory to build and run integration tests. For each `tests/foo.rs` or `tests/foo/main.rs`, it creates an executable that runs the test functions therein. These executables are separate crates from the main package, and thus do not have access to its private members. Integration tests are expected to test only the public functionality of the package. Application crates do not have any public members; their public interface is the command line. Integration tests would typically run the command (e.g. using `std::process::Command`) and test its output. Since dynk8s-provisioner is not really a command-line tool, testing it this way would be difficult; each test would need to start the server, make requests to it, and then stop it. This would be slow and cumbersome. In order to avoid this tedium and be able to use Rocket's built-in test client, I have converted dynk8s-provisioner into a library crate that also includes an executable. The library makes the `rocket` function public, which allows the integration tests to import it and pass it to the Rocket test client. The point of integration tests, of course, is to validate the functionality of the application as a whole. This necessarily requires allowing it to communicate with the Kubernetes API. In the Jenkins CI environment, the application will need the appropriate credentials, and will need to use a separate Kubernetes namespace from the production deployment. The `setup.yaml` manifest in the `tests` directory defines the resources necessary to run integration tests, and the `genkubeconfig.sh` script can be used to create the appropriate kubeconfig file containing the credentials. The kubeconfig is exposed to the tests via the `KUBECONFIG` environment variable, which is populated from a Jenkins secret file credential. Note: The `data` directory moved from `test` to `tests` to avoid duplication and confusing names.	2022-10-07 07:37:20 -05:00
Dustin	3e3904cd4f	events: Delete bootstrap tokens on termination When an instance is terminated, any bootstrap tokens assigned to it are now deleted. Though these would expire anyway, deleting them ensures that they cannot be used again if they happened to be leaked while the instance was running. Further, it ensures that attempting to fetch the `kubeadm` configuration for the instance will return an HTTP 404 Not Found response once the instance has terminated.	2022-10-07 06:52:06 -05:00
Dustin	df39fe46eb	routes: Add kubeadm kubeconfig resource The GET /kubeadm/kubeconfig/<instance-id> operation returns a configuration document for `kubeadm` to add the node to the cluster as a worker. The document is derived from the kubeconfig stored in the `cluster-info` ConfigMap, which includes the external URL of the Kubernetes API server and the root CA certificate used in the cluster. The bootstrap token assigned to the specified instance is added to the document for `kubeadm` to use for authentication. The kubeconfig is stored in the ConfigMap as a string, so extracting data from it requires deserializing the YAML document first. In order to access the cluster information ConfigMap, the service account bound to the pod running the provisioner service must have the appropriate permissions.	2022-10-07 06:52:06 -05:00
Dustin	25524d5290	routes: Add WireGuard configuration resource The * GET /wireguard/config/<instance-id>* resource returns the WireGuard client configuration assigned to the specified instance ID. The resource contents are stored in the Kubernetes Secret, in a data field named `wireguard-config`. The contents of this field are returned directly as a string, without any transformation. Thus, the value must be a complete, valid WireGuard configuration document. Instances will fetch and save this configuration when they first launch, to configure their access to the VPN.	2022-10-03 18:29:47 -05:00
Dustin	3f17373624	Change WireGuard keys -> configs Setting up the WireGuard client requires several pieces of information, beyond the node private key and peer's public key. The peer endpoint address/port, peer public key, and node IP address are also required. As such, naming the resource a "key" is somewhat misleading.	2022-10-03 18:20:46 -05:00
Dustin	3916e0eac9	Assign WireGuard keys to EC2 instances In order to join the on-premises Kubernetes cluster, EC2 instances will need to first connect to the WireGuard VPN. The dynk8s provisioner will provide keys to instances to configure their WireGuard clients. WireGuard keys must be pre-configured on the server and stored in Kubernetes as dynk8s.du5t1n.me/wireguard-key Secret resources. They must also have a `dynk8s.du5t1n.me/ec2-instance-id` label. If this label is empty, the key is available to be assigned to an instance. When an EventBridge event is received indicating an instance is now running, a WireGuard key is assigned to that instance (by setting the `dynk8s.du5t1n.me/ec2-instance-id` label). Conversely, when an event is received indicating that the instance is terminated, any WireGuard keys assigned to that instance are freed.	2022-10-01 12:17:32 -05:00
Dustin	25d7be004c	Begin EC2 instance state event handler The lifecycle of ephemeral Kubernetes worker nodes is driven by events emitted by Amazon EventBridge and delivered via Amazon Simple Notification Service. These events trigger the dynk8s provisioner to take the appropriate action based on the state of an EC2 instance. In order to add a node to the cluster using `kubeadm`, a "bootstrap token" needs to be created. When manually adding a node, this would be done e.g. using `kubeadm token create`. Since bootstrap tokens are just a special type of Secret, they can be easily created programmatically as well. When a new EC2 instance enters the "running" state, the provisioner creates a new bootstrap token and associates it with the instance by storing the instance ID in a label in the Secret resource's metadata. The initial implementation of the event handler is rather naïve. It generates a token for every instance, though some instances may not be intended to be used as Kubernetes workers. Ideally, the provisioner would only allocate tokens for instances matching some configurable criteria, such as AWS tags. Further, a token is allocated every time the instance enters the running state, even if a token already exists or is not needed.	2022-10-01 10:34:03 -05:00
Dustin	8e1165eb95	terraform: Begin AWS configuration The `terraform` directory contains the resource descriptions for all AWS services that need to be configured in order for the dynamic K8s provisioner to work. Specifically, it defines the EventBridge rule and SNS topic/subscriptions that instruct AWS to send EC2 instance state change notifications to the dynk8s-provisioner's HTTP interface.	2022-09-27 12:58:51 -05:00