Compare commits

...

8 Commits

Author SHA1 Message Date
Dustin d85f314a8b tests: Begin integration tests
dustin/dynk8s-provisioner/pipeline/head There was a failure building this commit Details
Cargo uses the sources in the `tests` directory to build and run
integration tests.  For each `tests/foo.rs` or `tests/foo/main.rs`, it
creates an executable that runs the test functions therein.  These
executables are separate crates from the main package, and thus do not
have access to its private members.  Integration tests are expected to
test only the public functionality of the package.

Application crates do not have any public members; their public
interface is the command line.  Integration tests would typically run
the command (e.g. using `std::process::Command`) and test its output.

Since *dynk8s-provisioner* is not really a command-line tool, testing it
this way would be difficult; each test would need to start the server,
make requests to it, and then stop it.  This would be slow and
cumbersome.

In order to avoid this tedium and be able to use Rocket's built-in test
client, I have converted *dynk8s-provisioner* into a library crate that
also includes an executable.  The library makes the `rocket` function
public, which allows the integration tests to import it and pass it to
the Rocket test client.

The point of integration tests, of course, is to validate the
functionality of the application as a whole.  This necessarily requires
allowing it to communicate with the Kubernetes API.  In the Jenkins CI
environment, the application will need the appropriate credentials, and
will need to use a separate Kubernetes namespace from the production
deployment.  The `setup.yaml` manifest in the `tests` directory defines
the resources necessary to run integration tests, and the
`genkubeconfig.sh` script can be used to create the appropriate
kubeconfig file containing the credentials.  The kubeconfig is exposed
to the tests via the `KUBECONFIG` environment variable, which is
populated from a Jenkins secret file credential.

Note: The `data` directory moved from `test` to `tests` to avoid
duplication and confusing names.
2022-10-07 07:37:20 -05:00
Dustin 3e3904cd4f events: Delete bootstrap tokens on termination
When an instance is terminated, any bootstrap tokens assigned to it are
now deleted.  Though these would expire anyway, deleting them ensures
that they cannot be used again if they happened to be leaked while the
instance was running.  Further, it ensures that attempting to fetch the
`kubeadm` configuration for the instance will return an HTTP 404 Not
Found response once the instance has terminated.
2022-10-07 06:52:06 -05:00
Dustin df39fe46eb routes: Add kubeadm kubeconfig resource
The *GET /kubeadm/kubeconfig/<instance-id>* operation returns a
configuration document for `kubeadm` to add the node to the cluster as a
worker.  The document is derived from the kubeconfig stored in the
`cluster-info` ConfigMap, which includes the external URL of the
Kubernetes API server and the root CA certificate used in the cluster.
The bootstrap token assigned to the specified instance is added to the
document for `kubeadm` to use for authentication.  The kubeconfig is
stored in the ConfigMap as a string, so extracting data from it requires
deserializing the YAML document first.

In order to access the cluster information ConfigMap, the service
account bound to the pod running the provisioner service must have the
appropriate permissions.
2022-10-07 06:52:06 -05:00
Dustin 25524d5290 routes: Add WireGuard configuration resource
The * GET /wireguard/config/<instance-id>* resource returns the
WireGuard client configuration assigned to the specified instance ID.
The resource contents are stored in the Kubernetes Secret, in a data
field named `wireguard-config`.  The contents of this field are returned
directly as a string, without any transformation.  Thus, the value must
be a complete, valid WireGuard configuration document.  Instances will
fetch and save this configuration when they first launch, to configure
their access to the VPN.
2022-10-03 18:29:47 -05:00
Dustin 3f17373624 Change WireGuard keys -> configs
Setting up the WireGuard client requires several pieces of information,
beyond the node private key and peer's public key.  The peer endpoint
address/port, peer public key, and node IP address are also required.
As such, naming the resource a "key" is somewhat misleading.
2022-10-03 18:20:46 -05:00
Dustin 3916e0eac9 Assign WireGuard keys to EC2 instances
In order to join the on-premises Kubernetes cluster, EC2 instances will
need to first connect to the WireGuard VPN.  The *dynk8s* provisioner
will provide keys to instances to configure their WireGuard clients.

WireGuard keys must be pre-configured on the server and stored in
Kubernetes as *dynk8s.du5t1n.me/wireguard-key* Secret resources.  They
must also have a `dynk8s.du5t1n.me/ec2-instance-id` label.  If this
label is empty, the key is available to be assigned to an instance.

When an EventBridge event is received indicating an instance is now
running, a WireGuard key is assigned to that instance (by setting the
`dynk8s.du5t1n.me/ec2-instance-id` label).  Conversely, when an event is
received indicating that the instance is terminated, any WireGuard keys
assigned to that instance are freed.
2022-10-01 12:17:32 -05:00
Dustin 25d7be004c Begin EC2 instance state event handler
The lifecycle of ephemeral Kubernetes worker nodes is driven by events
emitted by Amazon EventBridge and delivered via Amazon Simple
Notification Service.  These events trigger the *dynk8s* provisioner to
take the appropriate action based on the state of an EC2 instance.

In order to add a node to the cluster using `kubeadm`, a "bootstrap
token" needs to be created.  When manually adding a node, this would be
done e.g. using `kubeadm token create`.  Since bootstrap tokens are just
a special type of Secret, they can be easily created programmatically as
well.  When a new EC2 instance enters the "running" state, the
provisioner creates a new bootstrap token and associates it with the
instance by storing the instance ID in a label in the Secret resource's
metadata.

The initial implementation of the event handler is rather naïve.  It
generates a token for every instance, though some instances may not be
intended to be used as Kubernetes workers.  Ideally, the provisioner
would only allocate tokens for instances matching some configurable
criteria, such as AWS tags.  Further, a token is allocated every time
the instance enters the running state, even if a token already exists or
is not needed.
2022-10-01 10:34:03 -05:00
Dustin 8e1165eb95 terraform: Begin AWS configuration
The `terraform` directory contains the resource descriptions for all AWS
services that need to be configured in order for the dynamic K8s
provisioner to work.  Specifically, it defines the EventBridge rule and
SNS topic/subscriptions that instruct AWS to send EC2 instance state
change notifications to the *dynk8s-provisioner*'s HTTP interface.
2022-09-27 12:58:51 -05:00
34 changed files with 2077 additions and 30 deletions

448
Cargo.lock generated
View File

@ -37,6 +37,24 @@ dependencies = [
"subtle", "subtle",
] ]
[[package]]
name = "aho-corasick"
version = "0.7.19"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b4f55bd91a0978cbfd91c457a164bab8b4001c833b7f323132c0a4e1922dd44e"
dependencies = [
"memchr",
]
[[package]]
name = "android_system_properties"
version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "819e7219dbd41043ac279b19830f2efc897156490d7fd6ea916720117ee66311"
dependencies = [
"libc",
]
[[package]] [[package]]
name = "ansi_term" name = "ansi_term"
version = "0.12.1" version = "0.12.1"
@ -59,7 +77,7 @@ dependencies = [
"num-traits", "num-traits",
"rusticata-macros", "rusticata-macros",
"thiserror", "thiserror",
"time", "time 0.3.14",
] ]
[[package]] [[package]]
@ -206,6 +224,22 @@ version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd" checksum = "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd"
[[package]]
name = "chrono"
version = "0.4.22"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bfd4d1b31faaa3a89d7934dbded3111da0d2ef28e3ebccdb4f0179f5929d1ef1"
dependencies = [
"iana-time-zone",
"js-sys",
"num-integer",
"num-traits",
"serde",
"time 0.1.44",
"wasm-bindgen",
"winapi",
]
[[package]] [[package]]
name = "cipher" name = "cipher"
version = "0.3.0" version = "0.3.0"
@ -235,7 +269,7 @@ dependencies = [
"rand", "rand",
"sha2", "sha2",
"subtle", "subtle",
"time", "time 0.3.14",
"version_check", "version_check",
] ]
@ -368,6 +402,27 @@ dependencies = [
"subtle", "subtle",
] ]
[[package]]
name = "dirs-next"
version = "2.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b98cf8ebf19c3d1b223e151f99a4f9f0690dca41414773390fc824184ac833e1"
dependencies = [
"cfg-if",
"dirs-sys-next",
]
[[package]]
name = "dirs-sys-next"
version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4ebda144c4fe02d1f7ea1a7d9641b6fc6b580adcfa024ae48797ecdeb6825b4d"
dependencies = [
"libc",
"redox_users",
"winapi",
]
[[package]] [[package]]
name = "displaydoc" name = "displaydoc"
version = "0.2.3" version = "0.2.3"
@ -384,12 +439,18 @@ name = "dynk8s-provisioner"
version = "0.1.0" version = "0.1.0"
dependencies = [ dependencies = [
"base64", "base64",
"chrono",
"k8s-openapi",
"kube",
"log", "log",
"rand",
"regex",
"reqwest", "reqwest",
"rocket", "rocket",
"rsa", "rsa",
"serde", "serde",
"serde_json", "serde_json",
"serde_yaml 0.9.13",
"sha1", "sha1",
"x509-parser", "x509-parser",
] ]
@ -471,6 +532,7 @@ checksum = "7f21eda599937fba36daeb58a22e8f5cee2d14c4a17b5b7739c7c8e5e3b8230c"
dependencies = [ dependencies = [
"futures-channel", "futures-channel",
"futures-core", "futures-core",
"futures-executor",
"futures-io", "futures-io",
"futures-sink", "futures-sink",
"futures-task", "futures-task",
@ -493,12 +555,34 @@ version = "0.3.24"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4e5aa3de05362c3fb88de6531e6296e85cde7739cccad4b9dfeeb7f6ebce56bf" checksum = "4e5aa3de05362c3fb88de6531e6296e85cde7739cccad4b9dfeeb7f6ebce56bf"
[[package]]
name = "futures-executor"
version = "0.3.24"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9ff63c23854bee61b6e9cd331d523909f238fc7636290b96826e9cfa5faa00ab"
dependencies = [
"futures-core",
"futures-task",
"futures-util",
]
[[package]] [[package]]
name = "futures-io" name = "futures-io"
version = "0.3.24" version = "0.3.24"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bbf4d2a7a308fd4578637c0b17c7e1c7ba127b8f6ba00b29f717e9655d85eb68" checksum = "bbf4d2a7a308fd4578637c0b17c7e1c7ba127b8f6ba00b29f717e9655d85eb68"
[[package]]
name = "futures-macro"
version = "0.3.24"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "42cd15d1c7456c04dbdf7e88bcd69760d74f3a798d6444e16974b505b0e62f17"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]] [[package]]
name = "futures-sink" name = "futures-sink"
version = "0.3.24" version = "0.3.24"
@ -520,6 +604,7 @@ dependencies = [
"futures-channel", "futures-channel",
"futures-core", "futures-core",
"futures-io", "futures-io",
"futures-macro",
"futures-sink", "futures-sink",
"futures-task", "futures-task",
"memchr", "memchr",
@ -559,7 +644,7 @@ checksum = "4eb1a864a501629691edf6c15a593b7a51eebaa1e8468e9ddc623de7c9b58ec6"
dependencies = [ dependencies = [
"cfg-if", "cfg-if",
"libc", "libc",
"wasi", "wasi 0.11.0+wasi-snapshot-preview1",
] ]
[[package]] [[package]]
@ -652,6 +737,12 @@ dependencies = [
"pin-project-lite", "pin-project-lite",
] ]
[[package]]
name = "http-range-header"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0bfe8eed0a9285ef776bb792479ea3834e8b94e13d615c2f66d03dd50a435a29"
[[package]] [[package]]
name = "httparse" name = "httparse"
version = "1.8.0" version = "1.8.0"
@ -688,6 +779,36 @@ dependencies = [
"want", "want",
] ]
[[package]]
name = "hyper-openssl"
version = "0.9.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d6ee5d7a8f718585d1c3c61dfde28ef5b0bb14734b4db13f5ada856cdc6c612b"
dependencies = [
"http",
"hyper",
"linked_hash_set",
"once_cell",
"openssl",
"openssl-sys",
"parking_lot",
"tokio",
"tokio-openssl",
"tower-layer",
]
[[package]]
name = "hyper-timeout"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bbb958482e8c7be4bc3cf272a766a2b0bf1a6755e7a6ae777f017a31d11b13b1"
dependencies = [
"hyper",
"pin-project-lite",
"tokio",
"tokio-io-timeout",
]
[[package]] [[package]]
name = "hyper-tls" name = "hyper-tls"
version = "0.5.0" version = "0.5.0"
@ -701,6 +822,19 @@ dependencies = [
"tokio-native-tls", "tokio-native-tls",
] ]
[[package]]
name = "iana-time-zone"
version = "0.1.50"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fd911b35d940d2bd0bea0f9100068e5b97b51a1cbe13d13382f132e0365257a0"
dependencies = [
"android_system_properties",
"core-foundation-sys",
"js-sys",
"wasm-bindgen",
"winapi",
]
[[package]] [[package]]
name = "idna" name = "idna"
version = "0.2.3" version = "0.2.3"
@ -759,6 +893,96 @@ dependencies = [
"wasm-bindgen", "wasm-bindgen",
] ]
[[package]]
name = "jsonpath_lib"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "eaa63191d68230cccb81c5aa23abd53ed64d83337cacbb25a7b8c7979523774f"
dependencies = [
"log",
"serde",
"serde_json",
]
[[package]]
name = "k8s-openapi"
version = "0.16.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6d9455388f4977de4d0934efa9f7d36296295537d774574113a20f6082de03da"
dependencies = [
"base64",
"bytes",
"chrono",
"http",
"percent-encoding",
"serde",
"serde-value",
"serde_json",
"url",
]
[[package]]
name = "kube"
version = "0.75.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9bb19108692aeafebb108fd0a1c381c06ac4c03859652599420975165e939b8a"
dependencies = [
"k8s-openapi",
"kube-client",
"kube-core",
]
[[package]]
name = "kube-client"
version = "0.75.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "97e1a80ecd1b1438a2fc004549e155d47250b9e01fbfcf4cfbe9c8b56a085593"
dependencies = [
"base64",
"bytes",
"chrono",
"dirs-next",
"either",
"futures",
"http",
"http-body",
"hyper",
"hyper-openssl",
"hyper-timeout",
"jsonpath_lib",
"k8s-openapi",
"kube-core",
"openssl",
"pem",
"pin-project",
"secrecy",
"serde",
"serde_json",
"serde_yaml 0.8.26",
"thiserror",
"tokio",
"tokio-util",
"tower",
"tower-http",
"tracing",
]
[[package]]
name = "kube-core"
version = "0.75.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f4d780f2bb048eeef64a4c6b2582d26a0fe19e30b4d3cc9e081616e1779c5d47"
dependencies = [
"chrono",
"form_urlencoded",
"http",
"k8s-openapi",
"once_cell",
"serde",
"serde_json",
"thiserror",
]
[[package]] [[package]]
name = "lazy_static" name = "lazy_static"
version = "1.4.0" version = "1.4.0"
@ -780,6 +1004,21 @@ version = "0.2.5"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "292a948cd991e376cf75541fe5b97a1081d713c618b4f1b9500f8844e49eb565" checksum = "292a948cd991e376cf75541fe5b97a1081d713c618b4f1b9500f8844e49eb565"
[[package]]
name = "linked-hash-map"
version = "0.5.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0717cef1bc8b636c6e1c1bbdefc09e6322da8a9321966e8928ef80d20f7f770f"
[[package]]
name = "linked_hash_set"
version = "0.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "47186c6da4d81ca383c7c47c1bfc80f4b95f4720514d860a5407aaf4233f9588"
dependencies = [
"linked-hash-map",
]
[[package]] [[package]]
name = "lock_api" name = "lock_api"
version = "0.4.8" version = "0.4.8"
@ -855,7 +1094,7 @@ checksum = "57ee1c23c7c63b0c9250c339ffdc69255f110b298b901b9f6c82547b7b87caaf"
dependencies = [ dependencies = [
"libc", "libc",
"log", "log",
"wasi", "wasi 0.11.0+wasi-snapshot-preview1",
"windows-sys", "windows-sys",
] ]
@ -1051,6 +1290,15 @@ dependencies = [
"vcpkg", "vcpkg",
] ]
[[package]]
name = "ordered-float"
version = "2.10.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7940cf2ca942593318d07fcf2596cdca60a85c9e7fab408a5e21a4f9dcd40d87"
dependencies = [
"num-traits",
]
[[package]] [[package]]
name = "parking_lot" name = "parking_lot"
version = "0.12.1" version = "0.12.1"
@ -1097,6 +1345,15 @@ dependencies = [
"syn", "syn",
] ]
[[package]]
name = "pem"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "03c64931a1a212348ec4f3b4362585eca7159d0d09cbdf4a7f74f02173596fd4"
dependencies = [
"base64",
]
[[package]] [[package]]
name = "pem-rfc7468" name = "pem-rfc7468"
version = "0.3.1" version = "0.3.1"
@ -1112,6 +1369,26 @@ version = "2.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d4fd5641d01c8f18a23da7b6fe29298ff4b55afcccdf78973b24cf3175fee32e" checksum = "d4fd5641d01c8f18a23da7b6fe29298ff4b55afcccdf78973b24cf3175fee32e"
[[package]]
name = "pin-project"
version = "1.0.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ad29a609b6bcd67fee905812e544992d216af9d755757c05ed2d0e15a74c6ecc"
dependencies = [
"pin-project-internal",
]
[[package]]
name = "pin-project-internal"
version = "1.0.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "069bdb1e05adc7a8990dce9cc75370895fbe4e3d58b9b73bf1aee56359344a55"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]] [[package]]
name = "pin-project-lite" name = "pin-project-lite"
version = "0.2.9" version = "0.2.9"
@ -1240,6 +1517,17 @@ dependencies = [
"bitflags", "bitflags",
] ]
[[package]]
name = "redox_users"
version = "0.4.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b033d837a7cf162d7993aded9304e30a83213c648b6e389db233191f891e5c2b"
dependencies = [
"getrandom",
"redox_syscall",
"thiserror",
]
[[package]] [[package]]
name = "ref-cast" name = "ref-cast"
version = "1.0.9" version = "1.0.9"
@ -1266,6 +1554,8 @@ version = "1.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4c4eb3267174b8c6c2f654116623910a0fef09c4753f8dd83db29c48a0df988b" checksum = "4c4eb3267174b8c6c2f654116623910a0fef09c4753f8dd83db29c48a0df988b"
dependencies = [ dependencies = [
"aho-corasick",
"memchr",
"regex-syntax", "regex-syntax",
] ]
@ -1360,7 +1650,7 @@ dependencies = [
"serde_json", "serde_json",
"state", "state",
"tempfile", "tempfile",
"time", "time 0.3.14",
"tokio", "tokio",
"tokio-stream", "tokio-stream",
"tokio-util", "tokio-util",
@ -1407,7 +1697,7 @@ dependencies = [
"smallvec", "smallvec",
"stable-pattern", "stable-pattern",
"state", "state",
"time", "time 0.3.14",
"tokio", "tokio",
"uncased", "uncased",
] ]
@ -1475,6 +1765,16 @@ version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d29ab0c6d3fc0ee92fe66e2d99f700eab17a8d57d1c1d3b748380fb20baa78cd" checksum = "d29ab0c6d3fc0ee92fe66e2d99f700eab17a8d57d1c1d3b748380fb20baa78cd"
[[package]]
name = "secrecy"
version = "0.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9bd1c54ea06cfd2f6b63219704de0b9b4f72dcc2b8fdef820be6cd799780e91e"
dependencies = [
"serde",
"zeroize",
]
[[package]] [[package]]
name = "security-framework" name = "security-framework"
version = "2.7.0" version = "2.7.0"
@ -1507,6 +1807,16 @@ dependencies = [
"serde_derive", "serde_derive",
] ]
[[package]]
name = "serde-value"
version = "0.7.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f3a1a3341211875ef120e117ea7fd5228530ae7e7036a779fdc9117be6b3282c"
dependencies = [
"ordered-float",
"serde",
]
[[package]] [[package]]
name = "serde_derive" name = "serde_derive"
version = "1.0.144" version = "1.0.144"
@ -1524,6 +1834,7 @@ version = "1.0.85"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e55a28e3aaef9d5ce0506d0a14dbba8054ddc7e499ef522dd8b26859ec9d4a44" checksum = "e55a28e3aaef9d5ce0506d0a14dbba8054ddc7e499ef522dd8b26859ec9d4a44"
dependencies = [ dependencies = [
"indexmap",
"itoa", "itoa",
"ryu", "ryu",
"serde", "serde",
@ -1541,6 +1852,31 @@ dependencies = [
"serde", "serde",
] ]
[[package]]
name = "serde_yaml"
version = "0.8.26"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "578a7433b776b56a35785ed5ce9a7e777ac0598aac5a6dd1b4b18a307c7fc71b"
dependencies = [
"indexmap",
"ryu",
"serde",
"yaml-rust",
]
[[package]]
name = "serde_yaml"
version = "0.9.13"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8613d593412a0deb7bbd8de9d908efff5a0cb9ccd8f62c641e7b2ed2f57291d1"
dependencies = [
"indexmap",
"itoa",
"ryu",
"serde",
"unsafe-libyaml",
]
[[package]] [[package]]
name = "sha1" name = "sha1"
version = "0.10.2" version = "0.10.2"
@ -1718,6 +2054,17 @@ dependencies = [
"once_cell", "once_cell",
] ]
[[package]]
name = "time"
version = "0.1.44"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6db9e6914ab8b1ae1c260a4ae7a49b6c5611b40328a735b21862567685e73255"
dependencies = [
"libc",
"wasi 0.10.0+wasi-snapshot-preview1",
"winapi",
]
[[package]] [[package]]
name = "time" name = "time"
version = "0.3.14" version = "0.3.14"
@ -1771,6 +2118,16 @@ dependencies = [
"winapi", "winapi",
] ]
[[package]]
name = "tokio-io-timeout"
version = "1.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "30b74022ada614a1b4834de765f9bb43877f910cc8ce4be40e89042c9223a8bf"
dependencies = [
"pin-project-lite",
"tokio",
]
[[package]] [[package]]
name = "tokio-macros" name = "tokio-macros"
version = "1.8.0" version = "1.8.0"
@ -1792,6 +2149,18 @@ dependencies = [
"tokio", "tokio",
] ]
[[package]]
name = "tokio-openssl"
version = "0.6.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c08f9ffb7809f1b20c1b398d92acf4cc719874b3b2b2d9ea2f09b4a80350878a"
dependencies = [
"futures-util",
"openssl",
"openssl-sys",
"tokio",
]
[[package]] [[package]]
name = "tokio-stream" name = "tokio-stream"
version = "0.1.9" version = "0.1.9"
@ -1826,6 +2195,49 @@ dependencies = [
"serde", "serde",
] ]
[[package]]
name = "tower"
version = "0.4.13"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b8fa9be0de6cf49e536ce1851f987bd21a43b771b09473c3549a6c853db37c1c"
dependencies = [
"futures-core",
"futures-util",
"pin-project",
"pin-project-lite",
"tokio",
"tokio-util",
"tower-layer",
"tower-service",
"tracing",
]
[[package]]
name = "tower-http"
version = "0.3.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3c530c8675c1dbf98facee631536fa116b5fb6382d7dd6dc1b118d970eafe3ba"
dependencies = [
"base64",
"bitflags",
"bytes",
"futures-core",
"futures-util",
"http",
"http-body",
"http-range-header",
"pin-project-lite",
"tower-layer",
"tower-service",
"tracing",
]
[[package]]
name = "tower-layer"
version = "0.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "343bc9466d3fe6b0f960ef45960509f84480bf4fd96f92901afe7ff3df9d3a62"
[[package]] [[package]]
name = "tower-service" name = "tower-service"
version = "0.3.2" version = "0.3.2"
@ -1839,6 +2251,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2fce9567bd60a67d08a16488756721ba392f24f29006402881e43b19aac64307" checksum = "2fce9567bd60a67d08a16488756721ba392f24f29006402881e43b19aac64307"
dependencies = [ dependencies = [
"cfg-if", "cfg-if",
"log",
"pin-project-lite", "pin-project-lite",
"tracing-attributes", "tracing-attributes",
"tracing-core", "tracing-core",
@ -1962,6 +2375,12 @@ dependencies = [
"subtle", "subtle",
] ]
[[package]]
name = "unsafe-libyaml"
version = "0.2.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c1e5fa573d8ac5f1a856f8d7be41d390ee973daf97c806b2c1a465e4e1406e68"
[[package]] [[package]]
name = "url" name = "url"
version = "2.2.2" version = "2.2.2"
@ -2002,6 +2421,12 @@ dependencies = [
"try-lock", "try-lock",
] ]
[[package]]
name = "wasi"
version = "0.10.0+wasi-snapshot-preview1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1a143597ca7c7793eff794def352d41792a93c481eb1042423ff7ff72ba2c31f"
[[package]] [[package]]
name = "wasi" name = "wasi"
version = "0.11.0+wasi-snapshot-preview1" version = "0.11.0+wasi-snapshot-preview1"
@ -2216,7 +2641,16 @@ dependencies = [
"oid-registry", "oid-registry",
"rusticata-macros", "rusticata-macros",
"thiserror", "thiserror",
"time", "time 0.3.14",
]
[[package]]
name = "yaml-rust"
version = "0.4.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "56c1936c4cc7a1c9ab21a1ebb602eb942ba868cbd44a99cb7cdc5892335e1c85"
dependencies = [
"linked-hash-map",
] ]
[[package]] [[package]]

View File

@ -5,11 +5,19 @@ edition = "2021"
[dependencies] [dependencies]
base64 = "0.13.0" base64 = "0.13.0"
chrono = "0.4.22"
k8s-openapi = { version = "0.16.0", features = ["v1_22"] }
kube = "0.75.0"
log = "0.4.17" log = "0.4.17"
rand = "0.8.5"
reqwest = "0.11.11" reqwest = "0.11.11"
rocket = { version = "0.5.0-rc.2", features = ["json"] } rocket = { version = "0.5.0-rc.2", features = ["json"] }
rsa = "0.6.1" rsa = "0.6.1"
serde = { version = "1.0", features = ["derive"] } serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0.85" serde_json = "1.0.85"
serde_yaml = "0.9.13"
sha1 = "0.10.2" sha1 = "0.10.2"
x509-parser = "0.14.0" x509-parser = "0.14.0"
[dev-dependencies]
regex = "1.6.0"

5
ci/Jenkinsfile vendored
View File

@ -20,10 +20,15 @@ pipeline {
stage('Test') { stage('Test') {
steps { steps {
container('build') { container('build') {
withCredentials([file(
credentialsId: 'dynk8s-test-kubeconfig',
variable: 'KUBECONFIG'
)]) {
sh '. ci/test.sh' sh '. ci/test.sh'
} }
} }
} }
}
stage('Build') { stage('Build') {
steps { steps {

57
src/events.rs Normal file
View File

@ -0,0 +1,57 @@
//! Event handlers
//!
//! Functions in this module are called to handle events from outside sources,
//! such as Amazon EventBridge events delivered via Amazon Simple Notification
//! Service.
use log::{debug, error};
use crate::k8s::{
assign_wireguard_config, create_bootstrap_token, delete_bootstrap_tokens,
unassign_wireguard_config,
};
use crate::model::events::*;
/// Handle an EC2 instance state change event
///
/// This function manages the lifecycle of the Kubernetes Secret resources
/// associated with ephemeral nodes running as EC2 instances.
///
/// When an instance starts:
/// 1. A WireGuard config is assigned to the instance
/// 2. A Kubernetes bootstrap token is generated, to be used by `kubeadm` to
/// add the node to the cluster.
///
/// When an instance is terminated:
/// 1. Any WireGuard configs assigned to the instance are unassigned
/// 2. All bootstrap tokens for the instance are deleted
pub async fn on_ec2_instance_state_change(evt: Ec2InstanceStateChange) {
debug!("EC2 instance {} is now {}", &evt.instance_id, &evt.state);
if evt.state == "running" {
if let Err(e) = assign_wireguard_config(&evt.instance_id).await {
error!(
"Failed to assign WireGuard config to instnce {}: {}",
&evt.instance_id, e
);
return;
}
if let Err(e) = create_bootstrap_token(&evt.instance_id).await {
error!(
"Failed to create bootstrap token for instance {}: {}",
&evt.instance_id, e
)
};
} else if evt.state == "terminated" {
if let Err(e) = unassign_wireguard_config(&evt.instance_id).await {
error!(
"Failed to unassign WireGuard config from instance: {}: {}",
&evt.instance_id, e
);
}
if let Err(e) = delete_bootstrap_tokens(&evt.instance_id).await {
error!(
"Failed to delete bootstrap tokens for instance {}: {}",
&evt.instance_id, e
);
}
}
}

539
src/k8s.rs Normal file
View File

@ -0,0 +1,539 @@
//! Kubernetes Integration
use std::collections::btree_map::BTreeMap;
use chrono::offset::Utc;
use chrono::{DateTime, Duration};
use k8s_openapi::api::core::v1::{ConfigMap, Secret};
use k8s_openapi::apimachinery::pkg::apis::meta::v1::ObjectMeta;
use kube::core::params::{ListParams, Patch, PatchParams, PostParams};
use kube::{Api, Client};
use log::{debug, error, info, warn};
use rand::seq::SliceRandom;
use crate::model::k8s::*;
/// The set of characters allowed to appear in bootstrap tokens
const TOKEN_CHARS: [char; 36] = [
'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o',
'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', '0', '1', '2', '3',
'4', '5', '6', '7', '8', '9',
];
/// Kubernetes Bootstrap Token
///
/// Bootstrap tokens are typically used to add new nodes to the cluster using
/// `kubeadm join`. They are bearer tokens consisting of two parts: a token ID
/// and a secret. For additional information, see [Authenticating with
/// Bootstrap Tokens][0].
///
/// The Dynk8s Provisioner allocates bootstrap tokens for ephemeral nodes.
/// Each token is assigned to a specific EC2 instance; the instance ID is
/// stored in the token Secret's metadata, using the
/// `dynk8s.du5t1n.me/ec2-instance-id` label.
///
/// [0]: https://kubernetes.io/docs/reference/access-authn-authz/bootstrap-tokens/
#[derive(Clone, Debug)]
struct BootstrapToken {
/// The token ID (generated)
token_id: String,
/// The token secret (generated)
secret: String,
/// The date and time the token expires
expiration: DateTime<Utc>,
/// The EC2 instance ID to which the token is assigned
instance_id: Option<String>,
}
#[allow(dead_code)]
impl BootstrapToken {
/// Generate a new token
///
/// Initially, the token is *not* assigned to an EC2 instance. Use
/// [`Self::instance_id`] to set the associated instance ID.
pub fn new() -> Self {
let mut rng = rand::thread_rng();
let mut token_id = String::with_capacity(6);
while token_id.len() < 6 {
token_id.push(TOKEN_CHARS.choose(&mut rng).unwrap().clone());
}
let mut secret = String::with_capacity(16);
while secret.len() < 16 {
secret.push(TOKEN_CHARS.choose(&mut rng).unwrap().clone());
}
let expiration = Utc::now() + Duration::hours(1);
Self {
token_id,
secret,
expiration,
instance_id: None,
}
}
/// Return the expiration date/time
pub fn expiration(&self) -> &DateTime<Utc> {
&self.expiration
}
/// Set the ID of the EC2 instance associated with the token
pub fn instance_id(mut self, instance_id: String) -> Self {
self.instance_id = Some(instance_id);
self
}
/// Return the secret part of the token
pub fn secret(&self) -> &str {
&self.secret
}
/// Set the expiration date/time from a time-to-live duration
pub fn set_ttl(mut self, ttl: Duration) -> Self {
self.expiration = Utc::now() + ttl;
self
}
/// Set the expiration date/time
pub fn set_expiration(mut self, expiration: DateTime<Utc>) -> Self {
self.expiration = expiration;
self
}
/// Return the ID part of the token
pub fn token_id(&self) -> &str {
&self.token_id
}
/// Return the token as a string
pub fn token(&self) -> String {
format!("{}.{}", self.token_id, self.secret)
}
}
impl Into<Secret> for BootstrapToken {
/// Create a [`Secret`] for the token
///
/// Converting a [`BootstrapToken`] into a [`Secret`] populates the fields
/// necessary to store the token in Kubernetes. The `Secret` can be passed
/// to e.g. [`kube::Api::create`].
fn into(self) -> Secret {
let mut data = BTreeMap::<String, String>::new();
data.insert("token-id".into(), self.token_id.clone());
data.insert("token-secret".into(), self.secret);
data.insert("expiration".into(), self.expiration.to_rfc3339());
data.insert("usage-bootstrap-authentication".into(), "true".into());
data.insert("usage-bootstrap-signing".into(), "true".into());
data.insert(
"auth-extra-groups".into(),
"system:bootstrappers:kubeadm:default-node-token".into(),
);
let mut labels = BTreeMap::<String, String>::new();
if let Some(instance_id) = self.instance_id {
labels.insert(
"dynk8s.du5t1n.me/ec2-instance-id".into(),
instance_id.into(),
);
}
Secret {
data: None,
immutable: None,
metadata: ObjectMeta {
annotations: None,
cluster_name: None,
creation_timestamp: None,
deletion_grace_period_seconds: None,
deletion_timestamp: None,
finalizers: None,
generate_name: None,
generation: None,
labels: Some(labels),
managed_fields: None,
name: Some(format!("bootstrap-token-{}", self.token_id)),
namespace: None,
owner_references: None,
resource_version: None,
self_link: None,
uid: None,
},
string_data: Some(data),
type_: Some("bootstrap.kubernetes.io/token".into()),
}
}
}
/// Assign an existing WireGuard configuration to the specified EC2 instance
///
/// This function finds the first unused WireGuard client configuration, stored
/// as a Kubernetes Secret resource, and assigns it to the specified EC2
/// instance. Configs are assigned by setting the
/// `dynk8s.du5t1n.me/ec2-instance-id` label in the Secret resource's metadata.
///
/// Secret resources for WireGuard configuration have a *type* of
/// `dynk8s.du5t1n.me/wireguard-config`. The Secret's `data` field must
/// contain a `wireguard-config` property, which contains the WireGuard client
/// configuration the node should use. Configs must be created ahead of time
/// and must refer to working keys already configured on the WireGuard server.
pub async fn assign_wireguard_config(
instance_id: &str,
) -> Result<(), kube::Error> {
let client = Client::try_default().await?;
let secrets: Api<Secret> = Api::default_namespaced(client);
debug!(
"Checking for WireGuard configs already assigned to instance {}",
instance_id
);
let lp = ListParams::default()
.fields("type=dynk8s.du5t1n.me/wireguard-config")
.labels(&format!("dynk8s.du5t1n.me/ec2-instance-id={}", instance_id));
let res = secrets.list(&lp).await?;
if !res.items.is_empty() {
info!(
"WireGuard config already assigned to instance {}",
instance_id
);
return Ok(());
}
debug!("Looking for available WireGuard configs");
let lp = ListParams::default()
.fields("type=dynk8s.du5t1n.me/wireguard-config")
.labels("dynk8s.du5t1n.me/ec2-instance-id=");
let res = secrets.list(&lp).await?;
if res.items.is_empty() {
error!(
"No WireGuard config available for instance {}",
&instance_id
);
} else {
if let Some(name) = &res.items[0].metadata.name {
let mut labels = BTreeMap::<String, String>::new();
labels.insert(
"dynk8s.du5t1n.me/ec2-instance-id".into(),
instance_id.into(),
);
let mut secret = Secret::default();
secret.metadata.labels = Some(labels);
let patch = Patch::Apply(&secret);
let pp = PatchParams::apply(env!("CARGO_PKG_NAME")).force();
secrets.patch(&name, &pp, &patch).await?;
info!(
"Assigned WireGuard config {} to instance {}",
name, &instance_id
);
}
}
Ok(())
}
/// Unassign all WireGuard configs from the specified EC2 instance
///
/// This function finds all WireGuard configs, stored as Kubernetes Secret
/// resources, associated with the specified EC2 instance and unassigns them.
/// Unassigned configs have the `dynk8s.du5t1n.me/ec2-instance-id` label set to
/// the empty string.
pub async fn unassign_wireguard_config(
instance_id: &str,
) -> Result<(), kube::Error> {
let client = Client::try_default().await?;
let secrets: Api<Secret> = Api::default_namespaced(client);
let lp = ListParams::default()
.fields("type=dynk8s.du5t1n.me/wireguard-config")
.labels(&format!("dynk8s.du5t1n.me/ec2-instance-id={}", instance_id));
info!(
"Unassigning WireGuard configs from instance {}",
instance_id
);
for s in secrets.list(&lp).await? {
if let Some(name) = &s.metadata.name {
let mut labels = BTreeMap::<String, String>::new();
labels
.insert("dynk8s.du5t1n.me/ec2-instance-id".into(), "".into());
let mut secret = Secret::default();
secret.metadata.labels = Some(labels);
let patch = Patch::Apply(&secret);
let pp = PatchParams::apply(env!("CARGO_PKG_NAME")).force();
secrets.patch(&name, &pp, &patch).await?;
info!(
"Unassigned WireGuard config {} from instance {}",
name, &instance_id
);
}
}
Ok(())
}
/// Retrieve the WireGuard config assigned to the specified EC2 instance
///
/// This function finds the first WireGuard client configuration, stored as a
/// Kubernetes Secret resource, associated with the specified EC2 instance.
///
/// If multiple WireGuard configs are assigned to an EC2 instance, only the
/// first one returned by the Kubernetes list query is returned.
pub async fn get_wireguard_config(
instance_id: &str,
) -> Result<Option<String>, kube::Error> {
let client = Client::try_default().await?;
let secrets: Api<Secret> = Api::default_namespaced(client);
let lp = ListParams::default()
.fields("type=dynk8s.du5t1n.me/wireguard-config")
.labels(&format!("dynk8s.du5t1n.me/ec2-instance-id={}", instance_id));
for s in secrets.list(&lp).await? {
if let Some(data) = s.data {
match data.get("wireguard-config") {
Some(s) => match String::from_utf8(s.0.clone()) {
Ok(s) => return Ok(Some(s)),
Err(e) => {
error!("Invalid WireGuard configuration: {}", e);
}
},
None => {
error!(concat!(
"Invalid WireGuard configuration: ",
"missing wireguard-config property"
));
}
};
}
}
Ok(None)
}
/// Generate and store a bootstrap token for the specified EC2 instance
///
/// This function generates a new bootstrap token and stores it as a Kubernetes
/// Secret resource. The token is assigned to the given EC2 instance, and will
/// be provided to that instance when it is ready to join the cluster.
pub async fn create_bootstrap_token<I: AsRef<str>>(
instance_id: I,
) -> Result<(), kube::Error> {
let instance_id = instance_id.as_ref();
info!("Creating bootstrap token for instance {}", instance_id);
let token = BootstrapToken::new().instance_id(instance_id.into());
let client = Client::try_default().await?;
let secrets: Api<Secret> = Api::namespaced(client, "kube-system");
let pp: PostParams = Default::default();
let secret = secrets.create(&pp, &token.into()).await?;
info!("Successfully created secret {:?}", &secret.metadata.name);
Ok(())
}
/// Delete bootstrap tokens associated withthe specified EC2 instance
pub async fn delete_bootstrap_tokens<I: AsRef<str>>(
instance_id: I,
) -> Result<(), kube::Error> {
let instance_id = instance_id.as_ref();
info!("Deleting bootstrap tokens for instance {}", &instance_id);
let client = Client::try_default().await?;
let secrets: Api<Secret> = Api::namespaced(client, "kube-system");
let lp = ListParams::default()
.fields("type=bootstrap.kubernetes.io/token")
.labels(&format!("dynk8s.du5t1n.me/ec2-instance-id={}", &instance_id));
secrets.delete_collection(&Default::default(), &lp).await?;
Ok(())
}
/// Get the `kubeadm join` configuration for the specified EC2 instance
///
/// This function creates a kubeconfig file that can be passed to `kubeadm
/// join` to add the specified EC2 instance to the Kubernetes cluster as a
/// worker node. The cluster configuration is read from the `cluster-info`
/// ConfigMap in the *kube-public* namespace. The bootstrap token assigned to
/// the instance is included for client authentication.
pub async fn get_kubeconfig<I: AsRef<str>>(
instance_id: I,
) -> Result<Option<KubeConfig>, kube::Error> {
let instance_id = instance_id.as_ref();
let token = match get_bootstrap_token(&instance_id).await {
Ok(Some(t)) => t,
Ok(None) => {
warn!("No bootstrap token assigned to instance {}", &instance_id);
return Ok(None);
}
Err(e) => {
error!(
"Could not get bootstrap token for instance {}: {}",
&instance_id, e
);
return Ok(None);
}
};
match get_cluster_info().await? {
Some(config) => {
let cluster = Cluster {
name: "kubernetes".into(),
cluster: config.clusters[0].cluster.clone(),
};
let context = Context {
name: "kubeadm".into(),
context: ContextInfo {
cluster: "kubernetes".into(),
user: "kubeadm".into(),
},
};
let user = User {
name: "kubeadm".into(),
user: UserInfo { token: token },
};
let mut kubeconfig = KubeConfig::default();
kubeconfig.clusters = vec![cluster];
kubeconfig.contexts = Some(vec![context]);
kubeconfig.current_context = "kubeadm".into();
kubeconfig.users = Some(vec![user]);
Ok(Some(kubeconfig))
}
None => {
warn!("No kubeconfig loaded from cluster-info");
Ok(None)
}
}
}
/// Retrieve the bootstrap token assigned to an EC2 instance
async fn get_bootstrap_token<I: AsRef<str>>(
instance_id: I,
) -> Result<Option<String>, kube::Error> {
let instance_id = instance_id.as_ref();
let client = Client::try_default().await?;
let secrets: Api<Secret> = Api::namespaced(client, "kube-system");
let lp = ListParams::default()
.fields("type=bootstrap.kubernetes.io/token")
.labels(&format!(
"dynk8s.du5t1n.me/ec2-instance-id={}",
&instance_id
));
for s in secrets.list(&lp).await? {
match token_string(&s) {
Ok(t) => return Ok(Some(t)),
Err(e) => {
error!("Invalid bootstrap token: {}", e);
}
}
}
Ok(None)
}
/// Get cluster information from the ConfigMap
async fn get_cluster_info() -> Result<Option<KubeConfig>, kube::Error> {
let client = Client::try_default().await?;
let configmaps: Api<ConfigMap> = Api::namespaced(client, "kube-public");
let cluster_info = configmaps.get("cluster-info").await?;
if let Some(data) = cluster_info.data {
if let Some(config) = data.get("kubeconfig") {
match serde_yaml::from_str::<KubeConfig>(config) {
Ok(c) => return Ok(Some(c)),
Err(e) => {
error!(
"Could not load kubeconfig from cluster-info: {}",
e
);
}
};
} else {
error!("No kubeconfig property found in cluster-info ConfigMap");
}
} else {
error!("No data property found in cluster-info ConfigMap");
}
Ok(None)
}
/// Get the string representation of a bootstrap token from a Secret
fn token_string(secret: &Secret) -> Result<String, String> {
let data = match &secret.data {
Some(d) => d,
None => return Err("Missing data property".into()),
};
let token_id = match data.get("token-id") {
Some(s) => match String::from_utf8(s.0.clone()) {
Ok(s) => s,
Err(e) => return Err(e.to_string()),
},
None => return Err("Missing token-id".into()),
};
let secret = match data.get("token-secret") {
Some(s) => match String::from_utf8(s.0.clone()) {
Ok(s) => s,
Err(e) => return Err(e.to_string()),
},
None => return Err("Missing token-secret".into()),
};
Ok(format!("{}.{}", token_id, secret))
}
#[cfg(test)]
mod test {
use super::*;
use k8s_openapi::ByteString;
use regex::Regex;
#[test]
fn test_bootstrap_token_new() {
let token = BootstrapToken::new();
let id_re = Regex::new(r"^[a-z0-9]{6}$").unwrap();
let secret_re = Regex::new(r"^[a-z0-9]{16}$").unwrap();
let token_re = Regex::new(r"[a-z0-9]{6}\.[a-z0-9]{16}$").unwrap();
assert!(id_re.is_match(&token.token_id()));
assert!(secret_re.is_match(&token.secret()));
assert!(token_re.is_match(&token.token()));
}
#[test]
fn test_bootstrap_token_into_secret() {
let token = BootstrapToken::new();
let secret: Secret = token.clone().into();
let data = secret.string_data.unwrap();
assert_eq!(
&secret.metadata.name,
&Some(format!("bootstrap-token-{}", token.token_id))
);
assert_eq!(data.get("token-id").unwrap(), &token.token_id);
assert_eq!(data.get("token-secret").unwrap(), &token.secret);
assert_eq!(
data.get("expiration").unwrap(),
&token.expiration.to_rfc3339()
);
assert!(secret
.metadata
.labels
.unwrap()
.get("dynk8s.du5t1n.me/ec2-instance-id")
.is_none());
}
#[test]
fn test_bootstrap_token_into_secret_instance_id() {
let token =
BootstrapToken::new().instance_id("i-0a1b2c3d4e5f6f7f8".into());
let secret: Secret = token.clone().into();
let data = secret.string_data.unwrap();
assert_eq!(
&secret.metadata.name,
&Some(format!("bootstrap-token-{}", token.token_id))
);
assert_eq!(data.get("token-id").unwrap(), &token.token_id);
assert_eq!(data.get("token-secret").unwrap(), &token.secret);
assert_eq!(
data.get("expiration").unwrap(),
&token.expiration.to_rfc3339()
);
assert_eq!(
secret
.metadata
.labels
.unwrap()
.get("dynk8s.du5t1n.me/ec2-instance-id")
.unwrap(),
"i-0a1b2c3d4e5f6f7f8"
);
}
#[test]
fn test_token_string() {
let token = BootstrapToken::new();
let mut data = BTreeMap::new();
data.insert("token-id".into(), ByteString(token.token_id().into()));
data.insert("token-secret".into(), ByteString(token.secret().into()));
let secret = Secret {
data: Some(data),
..Default::default()
};
assert_eq!(token.token(), token_string(&secret).unwrap());
}
}

25
src/lib.rs Normal file
View File

@ -0,0 +1,25 @@
mod error;
mod events;
mod k8s;
mod model;
mod routes;
mod sns;
#[doc(hidden)]
pub fn rocket() -> rocket::Rocket<rocket::Build> {
rocket::build().mount(
"/",
rocket::routes![
routes::health::get_health,
routes::kubeadm::get_node_kubeconfig,
routes::kubeadm::post_node_kubeconfig,
routes::kubeadm::patch_node_kubeconfig,
routes::kubeadm::put_node_kubeconfig,
routes::wireguard::get_node_wireguard,
routes::sns::post_sns_notify,
routes::sns::get_sns_notify,
routes::sns::put_sns_notify,
routes::sns::patch_sns_notify,
],
)
}

View File

@ -1,19 +1,6 @@
mod error; use dynk8s_provisioner::rocket;
mod model;
mod routes;
mod sns;
#[doc(hidden)] #[rocket::main]
#[rocket::launch] async fn main() {
fn rocket() -> _ { let _ = rocket().launch().await;
rocket::build().mount(
"/",
rocket::routes![
routes::health::get_health,
routes::sns::post_sns_notify,
routes::sns::get_sns_notify,
routes::sns::put_sns_notify,
routes::sns::patch_sns_notify,
],
)
} }

46
src/model/events.rs Normal file
View File

@ -0,0 +1,46 @@
//! Amazon EventBridge event types
//!
//! These data structures are sent by [Amazon EventBridge][0], encapsulated in
//! SNS notification messages.
//!
//! [0]: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-what-is.html
use serde::{Deserialize, Serialize};
/// EC2 Instance State-change Notification
///
/// EventBridge event emitted when an EC2 instance changes state
#[derive(Deserialize, Serialize)]
#[serde(rename_all = "kebab-case")]
pub struct Ec2InstanceStateChange {
pub instance_id: String,
pub state: String,
}
/// Enumeration of EventBridge detail objects
///
/// EventBridge events sent by AWS services include a `detail` property, the
/// contents of which vary depending on the `detail-type` field.
#[derive(Deserialize, Serialize)]
#[serde(untagged)]
pub enum EventDetail {
Ec2InstanceStateChange(Ec2InstanceStateChange),
}
/// EventBridge event
///
/// See also: [Amazon EventBridge events][0]
///
/// [0]: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html
#[derive(Deserialize, Serialize)]
#[serde(rename_all = "kebab-case")]
pub struct Event {
pub version: String,
pub id: String,
pub detail_type: String,
pub source: String,
pub account: String,
pub time: String,
pub region: String,
pub resources: Vec<String>,
pub detail: EventDetail,
}

91
src/model/k8s.rs Normal file
View File

@ -0,0 +1,91 @@
//! kubeadm configuration data types
//!
//! The Kubernetes API reference does not include a specification for the
//! `Config` resource, and as such there is no model for it in [`k8s_openapi`].
//! Since *dynk8s* needs to read and write objects of this type, to provide
//! configuration for `kubeadm` on dynamic nodes, a subset of the required
//! model is defined here.
use serde::{Deserialize, Serialize};
/// Cluster information
#[derive(Clone, Debug, Deserialize, Serialize)]
#[serde(rename_all = "kebab-case")]
pub struct ClusterInfo {
/// X.509 certificate of the Kubernetes certificate authority
pub certificate_authority_data: String,
/// URL of the Kubernetes API server
pub server: String,
}
/// Cluster definition
#[derive(Clone, Debug, Deserialize, Serialize)]
pub struct Cluster {
/// Cluster information
pub cluster: ClusterInfo,
/// Cluster name
pub name: String,
}
/// kubeconfig context information
#[derive(Clone, Debug, Deserialize, Serialize)]
pub struct ContextInfo {
/// The cluster to use
pub cluster: String,
/// The user to use
pub user: String,
}
/// kubeconfig context definition
#[derive(Clone, Debug, Deserialize, Serialize)]
pub struct Context {
/// Context information
pub context: ContextInfo,
/// Context name
pub name: String,
}
/// User information
#[derive(Clone, Debug, Deserialize, Serialize)]
pub struct UserInfo {
/// Bootstrap token for authentication
pub token: String,
}
/// User definition
#[derive(Clone, Debug, Deserialize, Serialize)]
pub struct User {
/// User name
pub name: String,
/// User information
pub user: UserInfo,
}
/// kubeconfig
#[derive(Clone, Debug, Deserialize, Serialize)]
pub struct KubeConfig {
#[serde(rename = "apiVersion")]
pub api_version: String,
pub kind: String,
/// List of defined clusters
pub clusters: Vec<Cluster>,
/// List of defined contexts (usercluster associations)
pub contexts: Option<Vec<Context>>,
#[serde(rename = "current-context")]
/// Current context
pub current_context: String,
/// List of defined users
pub users: Option<Vec<User>>,
}
impl Default for KubeConfig {
fn default() -> Self {
Self {
api_version: "v1".into(),
kind: "Config".into(),
clusters: vec![],
contexts: None,
current_context: "".into(),
users: None,
}
}
}

View File

@ -1,2 +1,4 @@
//! The dynk8s provisioner data model //! The dynk8s provisioner data model
pub mod events;
pub mod k8s;
pub mod sns; pub mod sns;

72
src/routes/kubeadm.rs Normal file
View File

@ -0,0 +1,72 @@
use rocket::http::Status;
use crate::k8s::get_kubeconfig;
#[rocket::get("/kubeadm/kubeconfig/<instance_id>")]
pub async fn get_node_kubeconfig(instance_id: String) -> Option<String> {
if let Ok(Some(kubeconfig)) = get_kubeconfig(&instance_id).await {
Some(serde_yaml::to_string(&kubeconfig).unwrap())
} else {
None
}
}
#[rocket::post("/kubeadm/kubeconfig/<_instance_id>")]
pub async fn post_node_kubeconfig(_instance_id: String) -> Status {
Status::MethodNotAllowed
}
#[rocket::patch("/kubeadm/kubeconfig/<_instance_id>")]
pub async fn patch_node_kubeconfig(_instance_id: String) -> Status {
Status::MethodNotAllowed
}
#[rocket::put("/kubeadm/kubeconfig/<_instance_id>")]
pub async fn put_node_kubeconfig(_instance_id: String) -> Status {
Status::MethodNotAllowed
}
#[cfg(test)]
mod test {
use super::*;
use crate::rocket;
use rocket::local::blocking::Client;
use rocket::uri;
#[test]
fn test_get_node_token_404() {
let client = Client::tracked(rocket()).unwrap();
let res = client
.get(uri!(get_node_kubeconfig(
instance_id = "i-0a1b2c3d4e5f6f7f8"
)))
.dispatch();
assert_eq!(res.status(), Status::NotFound);
}
#[test]
fn test_kubeconfig_msg_wrong_method() {
let client = Client::tracked(rocket()).unwrap();
let res = client
.post(uri!(get_node_kubeconfig(
instance_id = "i-0a1b2c3d4e5f6f7f8"
)))
.dispatch();
assert_eq!(res.status(), Status::MethodNotAllowed);
let res = client
.patch(uri!(get_node_kubeconfig(
instance_id = "i-0a1b2c3d4e5f6f7f8"
)))
.dispatch();
assert_eq!(res.status(), Status::MethodNotAllowed);
let res = client
.put(uri!(get_node_kubeconfig(
instance_id = "i-0a1b2c3d4e5f6f7f8"
)))
.dispatch();
assert_eq!(res.status(), Status::MethodNotAllowed);
}
}

View File

@ -1,3 +1,5 @@
//! Rocket route handlers //! Rocket route handlers
pub mod health; pub mod health;
pub mod kubeadm;
pub mod sns; pub mod sns;
pub mod wireguard;

View File

@ -50,7 +50,7 @@ mod test {
fn test_sub_conf_msg() { fn test_sub_conf_msg() {
let client = Client::tracked(rocket()).unwrap(); let client = Client::tracked(rocket()).unwrap();
let data = std::fs::read_to_string( let data = std::fs::read_to_string(
"test/data/sns/subscriptionconfirmation.json", "tests/data/sns/subscriptionconfirmation.json",
) )
.unwrap(); .unwrap();
let res = client.post(uri!(post_sns_notify)).body(&data).dispatch(); let res = client.post(uri!(post_sns_notify)).body(&data).dispatch();
@ -68,7 +68,7 @@ mod test {
fn test_sub_conf_msg_bad() { fn test_sub_conf_msg_bad() {
let client = Client::tracked(rocket()).unwrap(); let client = Client::tracked(rocket()).unwrap();
let data = std::fs::read_to_string( let data = std::fs::read_to_string(
"test/data/sns/subscriptionconfirmation-bad.json", "tests/data/sns/subscriptionconfirmation-bad.json",
) )
.unwrap(); .unwrap();
let res = client.post(uri!(post_sns_notify)).body(&data).dispatch(); let res = client.post(uri!(post_sns_notify)).body(&data).dispatch();
@ -88,7 +88,7 @@ mod test {
fn test_sub_conf_msg_bad_cert_url() { fn test_sub_conf_msg_bad_cert_url() {
let client = Client::tracked(rocket()).unwrap(); let client = Client::tracked(rocket()).unwrap();
let data = std::fs::read_to_string( let data = std::fs::read_to_string(
"test/data/sns/subscriptionconfirmation-bad-url.json", "tests/data/sns/subscriptionconfirmation-bad-url.json",
) )
.unwrap(); .unwrap();
let res = client.post(uri!(post_sns_notify)).body(&data).dispatch(); let res = client.post(uri!(post_sns_notify)).body(&data).dispatch();

30
src/routes/wireguard.rs Normal file
View File

@ -0,0 +1,30 @@
use crate::k8s::get_wireguard_config;
#[rocket::get("/wireguard/config/<instance_id>")]
pub async fn get_node_wireguard(instance_id: String) -> Option<String> {
if let Ok(Some(token)) = get_wireguard_config(&instance_id).await {
Some(token)
} else {
None
}
}
#[cfg(test)]
mod test {
use super::*;
use crate::rocket;
use rocket::http::Status;
use rocket::local::blocking::Client;
use rocket::uri;
#[test]
fn test_get_node_wireguard_404() {
let client = Client::tracked(rocket()).unwrap();
let res = client
.get(uri!(get_node_wireguard(
instance_id = "i-0a1b2c3d4e5f6f7f8"
)))
.dispatch();
assert_eq!(res.status(), Status::NotFound);
}
}

View File

@ -8,6 +8,8 @@ use log::{debug, error, info};
use reqwest::Url; use reqwest::Url;
use serde::Serialize; use serde::Serialize;
use crate::events;
use crate::model::events::*;
use crate::model::sns::*; use crate::model::sns::*;
use error::SnsError; use error::SnsError;
use sig::SignatureVerifier; use sig::SignatureVerifier;
@ -38,11 +40,23 @@ pub async fn handle_unsubscribe(
/// Handle an notification message /// Handle an notification message
/// ///
/// After verifying the message signature, the message contents are written to /// This function handles varions SNS notification messages based on their
/// a file for later inspection. /// contents/sub-type.
pub async fn handle_notify(msg: NotificationMessage) -> Result<(), SnsError> { pub async fn handle_notify(msg: NotificationMessage) -> Result<(), SnsError> {
verify(&msg, &msg.signing_cert_url).await?; verify(&msg, &msg.signing_cert_url).await?;
save_message(&msg.topic_arn, &msg.timestamp, &msg.message_id, &msg); save_message(&msg.topic_arn, &msg.timestamp, &msg.message_id, &msg);
let event: Event = match serde_json::from_str(&msg.message) {
Ok(evt) => evt,
Err(e) => {
error!("Failed to deserialize notification message: {}", e);
return Ok(());
}
};
match event.detail {
EventDetail::Ec2InstanceStateChange(d) => {
events::on_ec2_instance_state_change(d).await;
}
};
Ok(()) Ok(())
} }

2
terraform/.gitattributes vendored Normal file
View File

@ -0,0 +1,2 @@
.terraform.lock.hcl -diff
terraform.tfstate -diff

2
terraform/.gitignore vendored Normal file
View File

@ -0,0 +1,2 @@
.terraform/
terraform.tfstate.backup

View File

@ -0,0 +1,22 @@
# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.
provider "registry.terraform.io/hashicorp/aws" {
version = "4.30.0"
constraints = "~> 4.16"
hashes = [
"h1:BFfhRf8my/aa0+YOSJv0xfjLQkToF475TJTMhTZfYec=",
"zh:08213f3ba960621448754211f148730edb59194919ee476b0231b769a5355028",
"zh:29c90d6f8bdae0e1469417ade28fa79c74c2af49593c1e2f24f07bacbca9e2c9",
"zh:5c6e9fab64ad68de6cd4ec6cbb20b0f75ba1e51a8efaeda3fe65419f096a06cb",
"zh:9b12af85486a96aedd8d7984b0ff811a4b42e3d88dad1a3fb4c0b580d04fa425",
"zh:9bf42718580e8c5097227df34e1bfa0a10a23eac9f527d97c2819c163087b402",
"zh:9f87e42e0f3d145fb0ad4aaff7ddded5720a64f9303956b33bd274c6dd05c05b",
"zh:bf0519ed9615bc408b72a0aebe1cc075d4c2042325590ba13dd264cd264907ea",
"zh:c3ac9e1cbd0935614f5a3c9cdb4cf9c6a1045937fe38e61da7c5c0fb7a069870",
"zh:d0c184476ada38c50acc068214ed1252b4fcf80b6be900fc1aed32cbb49f8ff6",
"zh:d4987dc7b7a69ea58f2b3ff0ea4ffc1b61a97881dbb8583c9fcf9444b753a6c2",
"zh:e8037376c81aeb98d8286dc19fba7f8eb053444d4b9484ea6a922382cffc1a85",
"zh:ecdabb44b48addc8483bca7bd683614a347367ae950ca8b6a6880679f5c12abd",
]
}

20
terraform/README.md Normal file
View File

@ -0,0 +1,20 @@
# Configuring AWS Using Terraform
## Prerequisites
The IAM user or role that runs Terraform must have the appropriate permissions.
The `iam-policy.json` file defines a policy that will allow the Terraform to
manage all of the necessary resources. Before running Terraform, create an IAM
policy and assign it to a user, group, or role. Be sure to replace the AWS
account ID in the various target resource names.
To use an IAM role, set the `iam_role` Terraform variable when executing
`terraform plan`/`terraform apply`.
## Create Resources
Terraform will create all resources automatically:
```sh
terraform apply
```

16
terraform/eventbridge.tf Normal file
View File

@ -0,0 +1,16 @@
resource "aws_cloudwatch_event_rule" "instance_state" {
name = "instance-state-events"
event_pattern = <<EOF
{
"source": ["aws.ec2"],
"detail-type": ["EC2 Instance State-change Notification"]
}
EOF
}
resource "aws_cloudwatch_event_target" "sns" {
rule = aws_cloudwatch_event_rule.instance_state.name
target_id = "SendToSNS"
arn = aws_sns_topic.ec2_events.arn
}

49
terraform/iam-policy.json Normal file
View File

@ -0,0 +1,49 @@
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"sns:ListTagsForResource",
"events:DescribeRule",
"sns:GetTopicAttributes",
"events:EnableRule",
"sns:DeleteTopic",
"events:PutRule",
"sns:CreateTopic",
"sns:SetTopicAttributes",
"events:DeleteRule",
"events:PutTargets",
"events:ListTagsForResource",
"sns:Subscribe",
"events:RemoveTargets",
"events:ListTargetsByRule",
"events:DisableRule"
],
"Resource": [
"arn:aws:sns:*:566967686773:*",
"arn:aws:events:*:566967686773:rule/*"
]
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"events:DescribeEventBus",
"events:CreateEventBus",
"events:DeleteEventBus"
],
"Resource": "arn:aws:events:*:566967686773:event-bus/*"
},
{
"Sid": "VisualEditor2",
"Effect": "Allow",
"Action": [
"sns:Unsubscribe",
"sns:GetSubscriptionAttributes"
],
"Resource": "*"
}
]
}

19
terraform/main.tf Normal file
View File

@ -0,0 +1,19 @@
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 4.16"
}
}
required_version = ">= 1.2.0"
}
provider "aws" {
region = var.ec2_region
assume_role {
role_arn = var.iam_role
}
}
data "aws_caller_identity" "current" {}

61
terraform/sns.tf Normal file
View File

@ -0,0 +1,61 @@
data "aws_iam_policy_document" "ec2_events_access_policy" {
statement {
sid = "__default_statement_ID"
effect = "Allow"
principals {
type = "AWS"
identifiers = ["*"]
}
actions = [
"SNS:Subscribe",
"SNS:SetTopicAttributes",
"SNS:RemovePermission",
"SNS:Receive",
"SNS:Publish",
"SNS:ListSubscriptionsByTopic",
"SNS:GetTopicAttributes",
"SNS:DeleteTopic",
"SNS:AddPermission",
]
resources = [
aws_sns_topic.ec2_events.arn,
]
condition {
test = "StringEquals"
variable = "AWS:SourceOwner"
values = [
data.aws_caller_identity.current.account_id
]
}
}
statement {
sid = "AllowEventBridgePublish"
effect = "Allow"
principals {
type = "Service"
identifiers = ["events.amazonaws.com"]
}
actions = ["sns:Publish"]
resources = [aws_sns_topic.ec2_events.arn]
}
}
resource "aws_sns_topic" "ec2_events" {
name = "ec2-events"
}
resource "aws_sns_topic_policy" "ec2_events_policy" {
arn = aws_sns_topic.ec2_events.arn
policy = data.aws_iam_policy_document.ec2_events_access_policy.json
}
resource "aws_sns_topic_subscription" "dynk8s_provisoner" {
topic_arn = aws_sns_topic.ec2_events.arn
protocol = "https"
endpoint = "https://dynk8s-provisioner.pyrocufflink.net/sns/notify"
}

275
terraform/terraform.tfstate Normal file
View File

@ -0,0 +1,275 @@
{
"version": 4,
"terraform_version": "1.2.9",
"serial": 49,
"lineage": "a100be74-c98e-0769-2d6a-bf6a2c5f3ebf",
"outputs": {},
"resources": [
{
"mode": "data",
"type": "aws_caller_identity",
"name": "current",
"provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
"instances": [
{
"schema_version": 0,
"attributes": {
"account_id": "566967686773",
"arn": "arn:aws:sts::566967686773:assumed-role/dynk8s-terraform/aws-go-sdk-1664301518318294107",
"id": "566967686773",
"user_id": "AROAYIAPIKZ25DFDOYZHT:aws-go-sdk-1664301518318294107"
},
"sensitive_attributes": []
}
]
},
{
"mode": "data",
"type": "aws_iam_policy_document",
"name": "ec2_events_access_policy",
"provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
"instances": [
{
"schema_version": 0,
"attributes": {
"id": "2883441170",
"json": "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Sid\": \"__default_statement_ID\",\n \"Effect\": \"Allow\",\n \"Action\": [\n \"SNS:Subscribe\",\n \"SNS:SetTopicAttributes\",\n \"SNS:RemovePermission\",\n \"SNS:Receive\",\n \"SNS:Publish\",\n \"SNS:ListSubscriptionsByTopic\",\n \"SNS:GetTopicAttributes\",\n \"SNS:DeleteTopic\",\n \"SNS:AddPermission\"\n ],\n \"Resource\": \"arn:aws:sns:us-east-2:566967686773:ec2-events\",\n \"Principal\": {\n \"AWS\": \"*\"\n },\n \"Condition\": {\n \"StringEquals\": {\n \"AWS:SourceOwner\": \"566967686773\"\n }\n }\n },\n {\n \"Sid\": \"AllowEventBridgePublish\",\n \"Effect\": \"Allow\",\n \"Action\": \"sns:Publish\",\n \"Resource\": \"arn:aws:sns:us-east-2:566967686773:ec2-events\",\n \"Principal\": {\n \"Service\": \"events.amazonaws.com\"\n }\n }\n ]\n}",
"override_json": null,
"override_policy_documents": null,
"policy_id": null,
"source_json": null,
"source_policy_documents": null,
"statement": [
{
"actions": [
"SNS:AddPermission",
"SNS:DeleteTopic",
"SNS:GetTopicAttributes",
"SNS:ListSubscriptionsByTopic",
"SNS:Publish",
"SNS:Receive",
"SNS:RemovePermission",
"SNS:SetTopicAttributes",
"SNS:Subscribe"
],
"condition": [
{
"test": "StringEquals",
"values": [
"566967686773"
],
"variable": "AWS:SourceOwner"
}
],
"effect": "Allow",
"not_actions": [],
"not_principals": [],
"not_resources": [],
"principals": [
{
"identifiers": [
"*"
],
"type": "AWS"
}
],
"resources": [
"arn:aws:sns:us-east-2:566967686773:ec2-events"
],
"sid": "__default_statement_ID"
},
{
"actions": [
"sns:Publish"
],
"condition": [],
"effect": "Allow",
"not_actions": [],
"not_principals": [],
"not_resources": [],
"principals": [
{
"identifiers": [
"events.amazonaws.com"
],
"type": "Service"
}
],
"resources": [
"arn:aws:sns:us-east-2:566967686773:ec2-events"
],
"sid": "AllowEventBridgePublish"
}
],
"version": "2012-10-17"
},
"sensitive_attributes": []
}
]
},
{
"mode": "managed",
"type": "aws_cloudwatch_event_rule",
"name": "instance_state",
"provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
"instances": [
{
"schema_version": 0,
"attributes": {
"arn": "arn:aws:events:us-east-2:566967686773:rule/instance-state-events",
"description": "",
"event_bus_name": "default",
"event_pattern": "{\"detail-type\":[\"EC2 Instance State-change Notification\"],\"source\":[\"aws.ec2\"]}",
"id": "instance-state-events",
"is_enabled": true,
"name": "instance-state-events",
"name_prefix": "",
"role_arn": "",
"schedule_expression": "",
"tags": {},
"tags_all": {}
},
"sensitive_attributes": [],
"private": "bnVsbA=="
}
]
},
{
"mode": "managed",
"type": "aws_cloudwatch_event_target",
"name": "sns",
"provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
"instances": [
{
"schema_version": 1,
"attributes": {
"arn": "arn:aws:sns:us-east-2:566967686773:ec2-events",
"batch_target": [],
"dead_letter_config": [],
"ecs_target": [],
"event_bus_name": "default",
"http_target": [],
"id": "instance-state-events-SendToSNS",
"input": "",
"input_path": "",
"input_transformer": [],
"kinesis_target": [],
"redshift_target": [],
"retry_policy": [],
"role_arn": "",
"rule": "instance-state-events",
"run_command_targets": [],
"sqs_target": [],
"target_id": "SendToSNS"
},
"sensitive_attributes": [],
"private": "eyJzY2hlbWFfdmVyc2lvbiI6IjEifQ==",
"dependencies": [
"aws_cloudwatch_event_rule.instance_state",
"aws_sns_topic.ec2_events"
]
}
]
},
{
"mode": "managed",
"type": "aws_sns_topic",
"name": "ec2_events",
"provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
"instances": [
{
"schema_version": 0,
"attributes": {
"application_failure_feedback_role_arn": "",
"application_success_feedback_role_arn": "",
"application_success_feedback_sample_rate": 0,
"arn": "arn:aws:sns:us-east-2:566967686773:ec2-events",
"content_based_deduplication": false,
"delivery_policy": "",
"display_name": "",
"fifo_topic": false,
"firehose_failure_feedback_role_arn": "",
"firehose_success_feedback_role_arn": "",
"firehose_success_feedback_sample_rate": 0,
"http_failure_feedback_role_arn": "",
"http_success_feedback_role_arn": "",
"http_success_feedback_sample_rate": 0,
"id": "arn:aws:sns:us-east-2:566967686773:ec2-events",
"kms_master_key_id": "",
"lambda_failure_feedback_role_arn": "",
"lambda_success_feedback_role_arn": "",
"lambda_success_feedback_sample_rate": 0,
"name": "ec2-events",
"name_prefix": "",
"owner": "566967686773",
"policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"__default_statement_ID\",\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"*\"},\"Action\":[\"SNS:Subscribe\",\"SNS:SetTopicAttributes\",\"SNS:RemovePermission\",\"SNS:Receive\",\"SNS:Publish\",\"SNS:ListSubscriptionsByTopic\",\"SNS:GetTopicAttributes\",\"SNS:DeleteTopic\",\"SNS:AddPermission\"],\"Resource\":\"arn:aws:sns:us-east-2:566967686773:ec2-events\",\"Condition\":{\"StringEquals\":{\"AWS:SourceOwner\":\"566967686773\"}}},{\"Sid\":\"AllowEventBridgePublish\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"events.amazonaws.com\"},\"Action\":\"sns:Publish\",\"Resource\":\"arn:aws:sns:us-east-2:566967686773:ec2-events\"}]}",
"sqs_failure_feedback_role_arn": "",
"sqs_success_feedback_role_arn": "",
"sqs_success_feedback_sample_rate": 0,
"tags": {},
"tags_all": {}
},
"sensitive_attributes": [],
"private": "bnVsbA=="
}
]
},
{
"mode": "managed",
"type": "aws_sns_topic_policy",
"name": "ec2_events_policy",
"provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
"instances": [
{
"schema_version": 0,
"attributes": {
"arn": "arn:aws:sns:us-east-2:566967686773:ec2-events",
"id": "arn:aws:sns:us-east-2:566967686773:ec2-events",
"owner": "566967686773",
"policy": "{\"Statement\":[{\"Action\":[\"SNS:Subscribe\",\"SNS:SetTopicAttributes\",\"SNS:RemovePermission\",\"SNS:Receive\",\"SNS:Publish\",\"SNS:ListSubscriptionsByTopic\",\"SNS:GetTopicAttributes\",\"SNS:DeleteTopic\",\"SNS:AddPermission\"],\"Condition\":{\"StringEquals\":{\"AWS:SourceOwner\":\"566967686773\"}},\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"*\"},\"Resource\":\"arn:aws:sns:us-east-2:566967686773:ec2-events\",\"Sid\":\"__default_statement_ID\"},{\"Action\":\"sns:Publish\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"events.amazonaws.com\"},\"Resource\":\"arn:aws:sns:us-east-2:566967686773:ec2-events\",\"Sid\":\"AllowEventBridgePublish\"}],\"Version\":\"2012-10-17\"}"
},
"sensitive_attributes": [],
"private": "bnVsbA==",
"dependencies": [
"aws_sns_topic.ec2_events",
"data.aws_caller_identity.current",
"data.aws_iam_policy_document.ec2_events_access_policy"
]
}
]
},
{
"mode": "managed",
"type": "aws_sns_topic_subscription",
"name": "dynk8s_provisoner",
"provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
"instances": [
{
"schema_version": 0,
"attributes": {
"arn": "arn:aws:sns:us-east-2:566967686773:ec2-events:3be7a77b-4b89-4925-bad3-dab7223d07f3",
"confirmation_timeout_in_minutes": 1,
"confirmation_was_authenticated": false,
"delivery_policy": "",
"endpoint": "https://dynk8s-provisioner.pyrocufflink.net/sns/notify",
"endpoint_auto_confirms": false,
"filter_policy": "",
"id": "arn:aws:sns:us-east-2:566967686773:ec2-events:3be7a77b-4b89-4925-bad3-dab7223d07f3",
"owner_id": "566967686773",
"pending_confirmation": false,
"protocol": "https",
"raw_message_delivery": false,
"redrive_policy": "",
"subscription_role_arn": "",
"topic_arn": "arn:aws:sns:us-east-2:566967686773:ec2-events"
},
"sensitive_attributes": [],
"private": "bnVsbA==",
"dependencies": [
"aws_sns_topic.ec2_events"
]
}
]
}
]
}

11
terraform/variables.tf Normal file
View File

@ -0,0 +1,11 @@
variable "ec2_region" {
description = "AWS region where all resources will be created"
type = string
default = "us-east-2"
}
variable "iam_role" {
description = "(Optional) IAM role ARN to assume when managing resources"
type = string
default = null
}

View File

@ -0,0 +1,11 @@
{
"Type": "Notification",
"MessageId": "b87b7cca-d698-552b-bf91-2764a57ca6a8",
"TopicArn": "arn:aws:sns:us-east-2:566967686773:ec2-events",
"Message": "{\"version\":\"0\",\"id\":\"ae9cb41b-2d06-c688-8d20-82e90720cfa4\",\"detail-type\":\"EC2 Instance State-change Notification\",\"source\":\"aws.ec2\",\"account\":\"566967686773\",\"time\":\"2022-09-28T16:09:08Z\",\"region\":\"us-east-2\",\"resources\":[\"arn:aws:ec2:us-east-2:566967686773:instance/i-0e50d560c8bf9f0f8\"],\"detail\":{\"instance-id\":\"i-0e50d560c8bf9f0f8\",\"state\":\"running\"}}",
"Subject": null,
"Timestamp": "2022-09-28T16:09:08.795Z",
"SignatureVersion": "1",
"Signature": "LBaVKayZhbAzQa8k78oKNUgo62KqFU/GZJ8GCkNmMJJSeAyZNFOPY7Rcy7x3tmvXnds9ns/xWnzgrK2SdD/Q0Zniu9st2o1lDKfQnbUHU1Wv7g65jOuXTJlBu6teuziJ/bpsFTv4z9yw4fPm7gNYZ3xF5yjwXn0j0IHv92YgzEJtewV+MKRgMtp0vq9+TvRhpOiYZC8DCWZdqQoHm7B9VruPPPG9yHSZ2eF9H8cOQgjRA0IweanGpU+qej6Ts0IMeMTPPcPzhSynCdQJ8UfBSNPPfnlkkbq714XUjcms01UcMFuAQyMHrrr9CsTOtOJn4R/h9qccjZ7DhzXSkGVocw==",
"SigningCertURL": "https://sns.us-east-2.amazonaws.com/SimpleNotificationService-56e67fcb41f6fec09b0196692625d385.pem"
}

View File

@ -0,0 +1,11 @@
{
"Type": "Notification",
"MessageId": "7d38c409-0edc-5cd0-87cd-9fa77aff18bc",
"TopicArn": "arn:aws:sns:us-east-2:566967686773:ec2-events",
"Message": "{\"version\":\"0\",\"id\":\"49c93129-5cb6-c94f-54c9-3b0c4fc07b03\",\"detail-type\":\"EC2 Instance State-change Notification\",\"source\":\"aws.ec2\",\"account\":\"566967686773\",\"time\":\"2022-09-28T16:10:11Z\",\"region\":\"us-east-2\",\"resources\":[\"arn:aws:ec2:us-east-2:566967686773:instance/i-0e50d560c8bf9f0f8\"],\"detail\":{\"instance-id\":\"i-0e50d560c8bf9f0f8\",\"state\":\"terminated\"}}",
"Subject": null,
"Timestamp": "2022-09-28T16:10:11.889Z",
"SignatureVersion": "1",
"Signature": "YVeRPXDW4zpPhBow/Le73zN1RJ754a/o9IHBAG8wSDNC2lLushh0ztknfp4G1qNA5ZenrKgTKyta0avLECh2qiU9lhp+M5qeY9CRIW2SH9xAL25MIN7psxkhJgSaqblIBENUE8gjge0pzDuKepgY5zD9f68Uf/0voMWTVWjcI2IdtOoS3LMqx6XC9K8LYMPgPf+0wGdH7AUXumQJTNBwX5jjMql2uIccN/xQxmUmIRHubWvDX82m78PZL+2mZlbGvsRRpkSPBPT/xu1fBwWkLQ0ziOf8sX7FyyKMKn8x4ohLkQjwU/21nSY8ei4n749ggJRNB0CxfQFPnNW5/iA4Qw==",
"SigningCertURL": "https://sns.us-east-2.amazonaws.com/SimpleNotificationService-56e67fcb41f6fec09b0196692625d385.pem"
}

38
tests/genkubeconfig.sh Normal file
View File

@ -0,0 +1,38 @@
#!/bin/sh
server=$(
kubectl config view --minify --raw \
-o jsonpath='{.clusters[].cluster.server}'
)
token=$(
kubectl get secret \
-n dynk8s-test dynk8s-provisioner \
-o jsonpath='{.data.token}' \
| base64 -d
)
ca=$(
kubectl get secret \
-n dynk8s-test dynk8s-provisioner \
-o jsonpath='{.data.ca\.crt}'
)
cat <<EOF
apiVersion: v1
kind: Config
clusters:
- name: kubernetes
cluster:
certificate-authority-data: ${ca}
server: ${server}
contexts:
- name: dynk8s-test@kubernetes
context:
cluster: kubernetes
namespace: dynk8s-test
user: dynk8s-test
current-context: dynk8s-test@kubernetes
users:
- name: dynk8s-test
user:
token: ${token}
EOF

65
tests/integration/main.rs Normal file
View File

@ -0,0 +1,65 @@
//! *dynk8s-provisioner* Integration Tests
//!
//! Create the test resources:
//!
//! ```sh
//! kubectl apply -f tests/setup.yaml
//! ```
//!
//! Generate a kubeconfig for the test service account:
//!
//! ```sh
//! sh tests/genkubeconfig.sh > kubeconfig
//! ```
//!
//! Run the tests:
//!
//! ```sh
//! KUBECONFIG=${PWD}/kubeconfig cargo test
//! ```
use std::collections::btree_map::BTreeMap;
use k8s_openapi::api::core::v1::Secret;
use kube::core::params::ListParams;
mod sns;
const WIREGUARD_CONFIG: &str = "\
[Interface]
Address = 10.11.12.13/14
PrivateKey = UEdAkIaF80zhlOpgacOYL2UckrfCAWXfsDDSAAzNH3g=
[Peer]
PublicKey = zbeTpUFA014kvTezIEGBt4yi3BVocST9j1dBElp9liI=
PreSharedKey = V6hAm01dxv2ib8AML2dSyX68hlPZm8En+IXfsknK3Zc=
AllowedIPs = 0.0.0.0/0
Endpoint = wireguard.example.org:24680
";
async fn setup() {
let client = kube::Client::try_default().await.unwrap();
let secrets: kube::Api<Secret> = kube::Api::default_namespaced(client);
let lp =
ListParams::default().fields("type=dynk8s.du5t1n.me/wireguard-config");
secrets
.delete_collection(&Default::default(), &lp)
.await
.unwrap();
let mut labels = BTreeMap::<String, String>::new();
labels.insert(
"app.kubernetes.io/part-of".into(),
"dynk8s-provisioner".into(),
);
labels.insert("dynk8s.du5t1n.me/ec2-instance-id".into(), "".into());
let mut data = BTreeMap::<String, String>::new();
data.insert("wireguard-config".into(), WIREGUARD_CONFIG.into());
let mut secret = Secret::default();
secret.type_ = Some("dynk8s.du5t1n.me/wireguard-config".into());
secret.immutable = Some(true);
secret.metadata.generate_name = Some("wireguard-config-".into());
secret.metadata.labels = Some(labels);
secret.string_data = Some(data);
secrets.create(&Default::default(), &secret).await.unwrap();
}

57
tests/integration/sns.rs Normal file
View File

@ -0,0 +1,57 @@
use rocket::http::Status;
use rocket::local::asynchronous::Client;
use dynk8s_provisioner::rocket;
use crate::{setup, WIREGUARD_CONFIG};
#[rocket::async_test]
async fn test_sns_ec2_lifecycle() {
setup().await;
let client = Client::tracked(rocket()).await.unwrap();
// Simulate an instance state-change event indicating an instance has
// started. This should generate a bootstrap token and assign the WireGuard
// config to the instance.
let data =
std::fs::read_to_string("tests/data/sns/notification-running.json")
.unwrap();
let res = client.post("/sns/notify").body(&data).dispatch().await;
assert_eq!(res.status(), Status::NoContent);
// Ensure the bootstrap token was generated
let res = client
.get("/kubeadm/kubeconfig/i-0e50d560c8bf9f0f8")
.dispatch()
.await;
assert_eq!(res.status(), Status::Ok);
// Ensure the WireGuard config was assigned
let res = client
.get("/wireguard/config/i-0e50d560c8bf9f0f8")
.dispatch()
.await;
assert_eq!(res.status(), Status::Ok);
assert_eq!(res.into_string().await, Some(WIREGUARD_CONFIG.into()));
// Simulate an instance state-change event indicating the instance has
// terminated. This should delete the bootstrap token and unassing the
// WireGuard config.
let data =
std::fs::read_to_string("tests/data/sns/notification-terminated.json")
.unwrap();
let res = client.post("/sns/notify").body(&data).dispatch().await;
assert_eq!(res.status(), Status::NoContent);
// Ensure the bootstrap token was deleted
let res = client
.get("/kubeadm/kubeconfig/i-0e50d560c8bf9f0f8")
.dispatch()
.await;
assert_eq!(res.status(), Status::NotFound);
// Ensure the WireGuard config was deleted
let res = client
.get("/wireguard/config/i-0e50d560c8bf9f0f8")
.dispatch()
.await;
assert_eq!(res.status(), Status::NotFound);
}

76
tests/setup.yaml Normal file
View File

@ -0,0 +1,76 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: dynk8s-test
labels:
kubernetes.io/metadata.name: dynk8s
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: dynk8s-provisioner
namespace: dynk8s-test
labels:
app.kubernetes.io/name: dynk8s-provisioner
app.kubernetes.io/instance: default
app.kubernetes.io/component: http-api
app.kubernetes.io/part-of: dynk8s-provisioner
automountServiceAccountToken: true
---
apiVersion: v1
kind: Secret
metadata:
name: dynk8s-provisioner
namespace: dynk8s-test
annotations:
kubernetes.io/service-account.name: dynk8s-provisioner
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: dynk8s-provisioner
namespace: dynk8s-test
labels:
app.kubernetes.io/name: dynk8s-provisioner
app.kubernetes.io/instance: default
app.kubernetes.io/component: http-api
app.kubernetes.io/part-of: dynk8s-provisioner
rules:
- apiGroups:
- ''
resources:
- secrets
verbs:
- '*'
- apiGroups:
- ''
resources:
- configmaps
resourceNames:
- cluster-info
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: dynk8s-provisioner
namespace: dynk8s-test
labels:
app.kubernetes.io/name: dynk8s-provisioner
app.kubernetes.io/instance: default
app.kubernetes.io/part-of: dynk8s-provisioner
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: dynk8s-provisioner
subjects:
- kind: ServiceAccount
name: dynk8s-provisioner
namespace: dynk8s-test