+++
title = 'AT&T Internet'
date = 2020-08-18T22:26:00-05:00
+++

AT&T Technician arrived at 08:40 on Monday 17 August 2020 to install *Internet
1000* service.  It took approximately 5 hours for him to complete the hookup,
because he had to run fiber line on the utility poles in the back yard,
presumably from the nearest fiber pedestal, trimming trees and shrubs in the
process, and had to return to the office to fetch different equipment.
Apparently, mine was the first house he installed with the capability for 10
gigabit/sec bandwidth, and he was unprepared.  He completed his work and left
around 13:45.

## First Tests

I performed the first tests of the AT&T Internet connection by connecting
_Toad_ directly to the provided "modem."  This device, an Arris BGW210-700, or
one of other similar devices, seems to be mandatory for all AT&T
Internet/U-Verse customers.  As far as I can tell, it is a typical Ethernet
router, with support for 802.1x/EAP wired port authentication.

With this setup, I got a pretty disappointing Speed Test score:
http://www.dslreports.com/speedtest/65265252.  Additionally, I am not sure the
"IP Passthrough" functionality of the device is going to be sufficient for my
needs.  Although it makes it appear to the downstream device as though it is
directly connected to the Internet, the intermediate router still seems to be
managing the traffic.  I have seen a few vague reports that certain services
"don't work" when the router is set up in this mode.

## Bypassing the Residential Gateway

With the poor network performance presumably caused by the AT&T-provided
_Residential Gateway_, and my suspicions that it would cause me trouble with
my self-hosted services (particularly the IPsec VPN), I decided to research a
way to bypass it.

### Initial Research: EAP Proxy

The first method I found for trying to bypass the AT&T-provided _Residential
Gateway_ (RG) was to use an "EAP Proxy" running on the USG:
https://medium.com/@mrtcve/at-t-gigabit-fiber-modem-bypass-using-unifi-usg-updated-c628f7f458cf.
The proxy would receive the EAPOL frames from the _Optical Network Terminator_
(ONT) and forward them to the RG, which is connected to the _LAN 2_ port on
the USG.  It would then forward the responses from the RG to the ONT, thus the
ONT would think the USG itself was performing the authentication.

I was initially turned off from this method, because it seemed like it
_required_ connecting the RG directly to the _LAN 2_ port, without a switch in
between.  Since I am already using the _LAN 2_ port for the auxiliary VLANs on
my network (test, guest, Home Assistant, management), I decided to revisit it
only if I could find no other option.

### Attempt 1: Device Swapping

The next bypass mechanism I found was this "true bridge mode" post on
DSLReports:
https://www.dslreports.com/forum/r29903721-AT-T-Residential-Gateway-Bypass-True-bridge-mode.
This method uses the RG temporarily to perform the 802.1x port
authentication/authorization, and then swap it out for another router.  The
described procedure uses port-based VLANs on a NetGear ProSafe GS108E.  By
changing the VLAN assignment of the ports where the AT&T RG and the desired
router are connected, the idea is to "trick" the _Optical Network Terminator_
(ONT), such that it does not notice the device on the other end has changed,
and thus does not require it to reauthenticate.

I attempted this method using my UniFi Switch 48, assigning 3 ports to a
special "AT&T" VLAN.  I planned to bring up the ONT and the RG together on the
same VLAN, then once the authentication was complete, kill the RG and bring up
the USG.  Unfortunately, the RG never managed to authenticate.  The
"Broadband" light blinked red most of the time, and never acquired an IP
address.  I suspect the issue was that the switch was filtering the 802.1x
traffic, since this is a layer 2 protocol that is intended to authenticate a
_device_ to a _switch_, rather than a _device_ to another _device_.

Some musings in the DSLReports thread suggested that it should also be
possible to use a "dumb switch" to work around this problem.  If the switch is
dumb enough (as is apparently the GS108E), it will forward EAPOL frames
exactly the same as it would any other frame.  I have an extremely cheap
5-port Tenda gigabit switch, so I tried using it instead of the UniFi switch.
In this scenario, the RG _did_ manage to authenticate to the ONT, and come
online completely.  Unfortunately, even with MAC address spoofing, I was not
able to "swap out" the RG.  When I connected _Toad_ to the dumb switch and
disconnected the RG, _Toad_ never received a DHCP response, and using a static
IP address did not work.  (In hindsight, I realized that this was probably
because I needed to use an 802.1Q tag to mark the outbound frames with VLAN ID
0.)

This method seemed really fragile to me, so even if I had gotten it to work, I
probably would not have been happy with it.  I considered using some trick
like this "2 dumb switch relay" method
(https://www.dslreports.com/forum/r32061385-) to automatically disconnect the
USG and reconnect the RG whenever some issue caused the port to become
deauthenticated.  If this had been my only option, I probably would have tried
harder, but instead, I decided to investigate some other options.

### Attempt 2: wpa_supplicant

During my initial research for a bypass method, I noted some posts regarding
using `wpa_supplicant` on a UniFi Dream Machine to perform the 802.1x
authentication directly.  Since I had had no luck with the first couple of
options, I decided to research this option further and see if there was an
equivalent option for the UniFi Security Gateway.  Sure enough, a quick Duck
Duck Go revealed a blog post specifically about that:
https://wells.ee/journal/2020-03-01-bypassing-att-fiber-modem-unifi-usg/
(Aside: the title of this person's blog, _It Kinda Works_ initially gave me
pause, as I interpreted that to refer to this specific technique.  Luckily, I
decided to read it anyway :).

I knew wpa_supplicant could be used to perform 802.1x authentication for wired
interfaces, the same as it does for WPA2-Enterprise networks on wireless
interfaces, so I read the post hoping for some insight into how to set that up
on the USG.  Sadly, it seemed like this may be difficult, since it involves
"obtaining" the X.509 certificates and private keys from the RG.  The author
of this post advocates using eBay to obtain them, but I was not willing to
pursue that avenue.  Instead, I decided to research a mechanism for extracting
them from the RG itself.

A quick DDG search for _bgw210 certificate_ led me to this blog post:
https://www.dupuis.xyz/bgw210-700-root-and-certs/.  The author does a
magnificent job describing the vulnerability in the RG that allows this, how
an exploit was developed, and how to use it to gain root access to the
operating system.  The process itself was **absolutely trivial**.  In only a
few minutes, I was able to exploit the vulnerability, gain root access to the
OS, and extract the `mfg.dat` file.

The only tool I could find to decode the`mfg.dat` file and extract the
certificate and private key from it was a closed-source program called
(naturally) `mfg_dat_decode`:
https://www.devicelocksmith.com/2018/12/eap-tls-credentials-decoder-for-nvg-and.html?m=1.
Fortunately, the author provided a native Linux executable, so although I
would rather have inspected the code to see what it does and how it works, I
could at least use it without any hassle.  Again, the process was so trivial,
I had access to everything I needed to set up my USG to perform the EAP-TLS
authentication itself, completely and permanently eliminating the AT&T RG from
my network.

I first tested this method using `wpa_supplicant` on _Toad_, and it worked
flawlessly.  Later, when I had some spare time, I went ahead and deployed it
on the USG.  With some minor variation (I put the certificates and private key
in `/config/auth`, the `wpa_supplicant` binary in `/config/scripts`, the
`wpa_supplicant.conf` file in `/config`, and wrote my own
`/config/scripts/post-config.d/wpa_supplicant.sh` because the one suggested
was written by someone who had extremely limited knowledge of shell
scripting), I was able to set it up as described in the _Bypassing the AT&T
Fiber modem…_ blog post.  The process was actually rather simple:

1. Install the `wpa_supplicant` binary
2. Copy the CA certificate, client certificate, and private key files
3. Configure  `wpa_supplicant` to use these to authenticate the port using EAP-TLS
4. Ensure `wpa_supplicant` starts automatically using a script in `/config/scripts/post-config.d`
5. Change the MAC address of the WAN port on the USG to that of the RG
6. Set the WAN port to use tagged VLAN ID 0
7. Reboot the USG

The only thing that bothers me about this method is using the precompiled
`wpa_supplicant` binary.  I definitely want to come up with a process for
building it myself, preferably automatically in Jenkins, so I can keep it
up-to-date.

Fortunately, the certificate I extracted from the RG does not expire until
2040!  Hopefully sometime in 20 years, something changes…

Without the RG in the path, the next speed test was *much* better:
http://www.dslreports.com/speedtest/65268235.

## IPv6 Support

During my initial test, _Toad_ was assigned a couple of IPv6 addresses
(presumably 1 via SLAAC and 1 by DHCPv6).  Once I eliminated the RG and
connected the USG directly, no IPv6 addresses were assigned either to the USG
or to its clients.  Based on various posts on the web, I believe this is a
"known issue" with AT&T.  Apparently, there is a 2-week period after changing
devices during which IPv6 address assignment is unavailable.  This seems to be
because the DHCPv6 lease is assigned with a 14-day lifetime, and only a single
lease is allowed per customer connection.  I will again try to get IPv6
working in a couple of weeks, then.

## Self-Hosting

So far, after switching to AT&T Internet, I have not noticed any issues with
my self-hosted services.  I have confirmed that all of my websites (including
Nextcloud, Bitwarden, Gitea, etc.) are all working, as is my Wireguard VPN.

Before I signed up with AT&T, I read the Terms of Service and the Acceptable
Use Policy.  Obtuse legalese notwithstanding, I did not see anything that
forbade self-hosted services, so I imagine there will not be any issues going
forward, especially with the crappy RG out of the way.
