dustin/xactfetch/pipeline/head This commit looks goodDetails
If the `SECRET_SOCKET_PATH` environment variable is not set, or refers
to a non-existent path, then we assume we need to manage the
`secretsocket` server ourselves.
Using the Playwrigt async API is the only way to wait for one of
multiple conditions. We will need this capability in order to detect
certain abnormal conditions, such as spurious 2FA auth or interstitial
ads.
`xactfetch` has three different ways of reading secret values:
* From environment variables
* By reading the contents of a file (specified by environment variables)
* By looking them up in the Bitwarden vault
This is very cumbersome to work with, especially when trying to
troubleshoot using the container image locally.
To make this easier, I've factored out all secret lookup functionality
into a separate process. This process listens on a UNIX socket and
implements a very simple secret lookup protocol. The client
(`xactfetch` itself in this case) sends a string key, identifying the
secret it wants to look up, terminated by a single line feed character.
The `secretsocket` server looks up the secret associated with that key,
using the method defined in a TOML configuration file. There are four
supported methods:
* Environment variables
* External programs
* File contents
* Static strings
The value returned by the corresponding method is then sent back to the
client via the socket connection, again as a string terminated with a
line feed.
Moving the secret handling into a separate process simplifies the
environment configuration needed in order to run `xactfetch`. Notably,
when running it in a container, only the `secretsocket` soket needs to
be mounted into the container. Since `rbw` is executed by the server
process now, rather than `xactfetch` directly, the vault does not need
to be present in the `xactfetch` container. Indeed, none of the secret
values need to be present in the container.
dustin/xactfetch/pipeline/head This commit looks goodDetails
When debugging a failure for one bank's website, I often want to run
the fetch for just that bank. To date, I've been commenting out the
other bank, but that is silly. Now, `xactfetch` can target a subset
of banks by specifying their name slug(s) as CLI arguments.
Chase changed the name of my credit card from *CREDIT CARD* to *Amazon
Visa*. Just in case they change it again or something, let's match only
on the card number.
By default, the transaction list for the Chase credit card shows
transactions that have posted since the last statement. This list can
sometimes be empty, particularly on the day the the statement is issued.
When this is the case, clicking the _Download Account Activity_ button
does not work; it simply displays a message stating "There's no account
activity showing to download." Since we are going to adjust the date
range on the download form anyway, it doesn't matter what's showing,
we just need the button to work. Thus, we now set the page show all
transactions and then click the button.
Playright needs to be updated frequently in order to update its Firefox
build. The Chase website has a very strict browser support policy, and
frequently drops support for old Firefox versions.
I've moved the bank website credentials to a shared collection in
Bitwarden and made them accessible to an account dedicated to
`xactfetch`. Using the `pinentry-stub` script, `rbw` can now
auto-unlock the vault, using the password in the file referred to by the
`PINENTRY_PASSWORD_FILE` environment variable. This means that
`xactfetch` can now run completely automatically, without any input from
me.
While debugging `xactfetch`, I do not need it to send me notifications
about failures, etc., since I am sitting at my computer. To suppress
them, I can now set the `DEBUG_NTFY` environment variable to `0`.
Sometimes transactions show up in the export with the previous day's
date. When this happens, these transactions may get skipped, since they
might have the same date as the most recent transaction in Firefly. To
help avoid skipping transactions, we need the start date to be the same
as the most recent transaction, rather than the next day. This can
cause duplicate imports, though, but fortunately, the Firefly Data
Importer handles this fairly well.
If the latest transaction was recent enough to skip importing
transactions, we don't even need to log in to the bank websites. Thus,
we should delay the login step until after we've checked this.
Since I ulimately want to run `xactfetch` in Kubernetes, running the
importer in a container as a child process doesn't make much sense.
While running `podman` in a Kubernetes container is possible, getting it
to work is non trivial. Rather than go through all that effort, I think
it makes more sense to just use HTTP to communicate with the importer I
already have running.
I had originally chosen not to use the web importer because of how I
have it configured to use Authelia for authentication. The importer
itself does not have any authentication beyond the "secret" parameter
(which is not secret at all, given that it is passed in the query string
and thus visible to anyone and stored in access logs), so I was hesitant
to add an access control rule to bypass authentication for the
`/autoupload` path. Fortunately, I discovered that Authelia will use
the value of the `Proxy-Authorization` header to authenticate the
request without redirecting to the login screen. With just a couple of
lines in the Ingress configuration, I got it to work using the regular
`Authorization` header as well:
```yaml
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/auth-snippet: |
proxy_set_header Proxy-Authorization $http_authorization;
proxy_set_header X-Forwarded-Method $request_method;
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header Authorization "";
```
Apparently, Chase has switched back to the CSV schema without the Card
column at the beginning. Just in case they decide to flip-flop on that
field forever, we better try to handle both cases.
Chase made some minor updates to their site recently which affected some
of the element locators. The propaganda in the right-hand column of the
landing page has changed, and the Downlod Account Activity form is still
really terrible, and now behaves even more strangely.
Commerce likes to occasionally inject ads and other propaganda after the
login page, before loading the account summary page. To handle this, we
may need to specifically navigate to the account summary page after
logging in.
The Commerce Bank website no longer allows navigating directly to
`Download.ashx`; doing so just returns a generic "we're sorry" error.
They appear to have added some CSRF protection or something that makes
this not work. As a result, we have to go fill out the form on the
*Download Transactions* modal dialog in order to get the download to
work correctly.
In order to set the message for a notification with an attachment, the
text must be specified in the `Message` request header. Unfortunately,
HTTP header values are limited to the Latin-1 character set, so Unicode
characters cannot be included. As of *ntfy* 2.4.0, however, the server
can decode base64-encoded headers using the RFC 2047 scheme.
To maintain compatibility with older *ntfy* servers, the `ntfy` function
will only encode message contents this way if the string cannoto be
encoded as ASCII.
When there are multiple accounts associated with a Chase online banking
user, the dashboard page layout changes. Detailed account history is no
longer shown, so the elements we were waiting for in the "Waiting for
page to load completely" step never appear. Since we're navigating
directly to the download account transactions page now, anyway, we do
not even need to wait for this button to appear.
Although it is undocumented, *ntfy* accepts a `Message` header along
with a file upload, which sets the message content of the notification
when a file is attached. Since HTTP headers cannot contain multiple
lines, the newline character has to be escaped. The *ntfy* server
performs unescaping automatically.
When there are no transactions in the default display, the *Download
account activity* button is disabled. To avoid failing in this case, we
now navigate directly to the download page. This requires explicitly
selecting the credit card account from the dropdown list, as it is not
pre-filled when the page is loaded directly.
The `ntfyerror` context manager replaces `screenshot_failure` for
handling online banking interaction failures. It has several
advantages, notably:
* takes a screenshot of the browser page *before* logging out
* cleaner suppression of exceptions, with success tracking
* sends an `ntfy` message, with the screenshot attached
Dustin