dustin.web/content/cv/firemon.md

+++ title = 'FireMon' date = 2013-12-01 [extra] years = '2013–Present' +++

FireMon is a software development company based in Overland Park, KS. As the System Architect, I focus on building a scalable platform for delivering FireMon software to customers that is easy to use. FMOS, the FireMon Operating System, is a mechanism for delivering the FireMon SIP to customers and a collection of tools for deploying and managing the software in a wide array of environments, ranging from a single server to massive multi-node ecosystems.

FMOS Generation II

• Based on CentOS 7
• Full-disk encryption using LUKS
• Anaconda installer with custom addon for generating machine-specific LUKS master key passphrase
• Kickstart script for fully-automated installation
• Used Koji to build RPM packages for first- and third-party software
• Distribution included Ansible for configuration management
• systemd units for controlling FireMon application services
• Configuration policy for deployment of all FireMon software and third-party dependencies
  • Support for single-server and distributed deployments
  • Automatically compute JVM heap sizes for each process based on available resources
  • Configures Elasticsearch in single-node or clustered mode
  • Configures PostgreSQL with optional replication to standby servers
  • Configures Kernel NFS server and client to share filesystem data between machines
  • Configures FireMon application server processes, including connection and authentication information for PostgreSQL, Elasticsearch
  • Configures strongSwan IPsec/IKEv2 key management daemon for opportunistic encryption of Elasticsearch communication
  • Configures operating system login, password policy, including support for external authentication providers such as LDAP or Kerberos
  • Sets up collectd and Carbon (Graphite data storage engine) to track system performance metrics, optionally replicating metrics data to a FireMon-managed central storage for real-time review
  • Optionally configures rsyslog to send log messages to remote destinations over UDP, TCP, or TCP+TLS
  • Configures tmux to automatically launch at user login
• …

FMOS Generation III

• Based on CentOS 7, later CentOS 8 (Stream)
• Immutable SquashFS root filesystem image
• …

DevOps Team Lead

• Deployed and maintained hundreds of internal and cloud systems
  • HashiCorp Vault
  • Elasticsearch
  • Atlassian Bitbucket
  • Jenkins
• Used PXE for provisioning on-premises virtual machines
• Ansible configuration management

Internal Tools

FMOS Web Tools

• Internal application used by software developers and support agents
• Multi-tiered architecture with multiple nodes at each tier to avoid any single point of failure
  • Application Server Tier: Python 3.6/FastAPI
  • Storage Tier: GlusterFS
  • Index Tier: Elasticsearch
  • Cache Tier: Redis
  • Message Tier: RabbitMQ
  • Worker Tier: Python 3.6/Celery
  • Ingress: HAProxy
  • User Interface: Typescript/Vue+Vuetify

PR Bot

• Implements a web hook for Atlassian Bitbucket (stash)
• Reacts to new and updated Pull Requests
• Automatically checks Git commits and changed code to enforce style guide and other project-specific requirements
• Adds comments to Pull Requests indicating check results, marks PR as approved or needs work
• Written in Python, no external dependencies

QEMU VM Log Socket Proxy

• Component of FMOS End-to-End tests running on-premises using QEMU/libvirt
• Uses kernel inotify(7) events to detect virtual machine log channel socket files appearing on the VM host
• Automatically connects to sockets as they appear
• Receives all data from channel sockets and writes them to a file in the libvirt storage pool
• Written in Rust

FMOS ISO Writer

• Internal application used by development and QA teams to write FMOS installer images to USB disks attached to remote physical appliances
• Accessible via purpose-built, ultra-minimal Linux distribution (Kernel and Busybox only) delivered by network boot/PXE
• Written in Rust