Video Summary
Affected by Atlassian layoffs after eight years and sharing a reflective post-mortem.
Hired after a rigorous interview; tasked to build a self-service load balancer via an Open Service Broker (OSB).
Replaced enterprise load balancers with Envoy and built a FastAPI-based management server called Sovereign.
Infrastructure deployed with AWS CloudFormation; AMIs produced via Packer and SaltStack for consistent proxy nodes.
Platform centralized security (DDoS, rate limiting, auth) and gave dev teams self-service capabilities with sidecars and dynamic config (XDS).','Non-technical growth: diplomacy, conflict resolution, onboarding, and hands
He was asked to build a self‑service load balancer application—an Open Service Broker (OSB) style web API—to let internal developers provision and bind resources in a Kubernetes environment.
Envoy was selected because it’s a modern, cloud‑native proxy that supports dynamic reconfiguration via an API (XDS), offering more flexibility than traditional enterprise load balancers.
Proxies were deployed with AWS CloudFormation using prebuilt AMIs. AMIs were created with HashiCorp Packer and configured using SaltStack so each proxy instance had consistent software and hardening.
Sovereign is the FastAPI‑based Envoy management server the speaker open‑sourced; it renders templates using dynamic context (DB, S3, etc.) to serve configuration via the XDS control plane.
He emphasized growth in diplomacy, conflict avoidance and resolution, mentoring, onboarding, and the ability to teach and persuade across teams.
"I wanted to take some time out to reflect on the time that I spent working for Atlassian."
The speaker discusses being affected by the recent layoffs at Atlassian after eight years with the company and expresses a desire to reflect on his experiences and contributions during that time.
He emphasizes the hope that his reflections may help others who are in similar situations, possibly offering inspiration and guidance for overcoming challenges.
The talk will include both technical and non-technical aspects of his experience, with the video organized into chapters for easy navigation.
"I still remember the interview process, which was different nowadays."
The speaker shares his reminisces about the interview process from eight years ago, noting the distinct qualities of interviewers whom he now considers friends.
A coding quiz on HackerRank was part of the process, where he performed exceptionally well. Following this, the first technical interview involved reviewing a Cloudflare white paper and discussing microservices.
He faced a troubleshooting exercise in the second technical interview, which included simulating responses to real incidents and emerged despite a small error in his understanding of latency-based DNS.
The values interview became particularly memorable for him when he was prompted to envision what he would need to achieve in a year for the company to affirm their hiring decision.
"When you join Atlassian, you're drinking from the fire hose because there's so much information..."
Upon joining, he faced the challenge of absorbing a vast amount of information quickly, which he described as "drinking from the fire hose."
His first task was to develop an application for facilitating self-service load balancers, rooted in the confidence he expressed during his interview.
The app was designed to operate within a Kubernetes environment, aimed at simplifying the provisioning of internal resources for developers.
He began building the application with various technologies, initially using an Open Service Broker to create the necessary API infrastructure, later transitioning to frameworks like Flask and FastAPI.
"I chose to build this in Python using Flask."
The speaker outlines the technical architecture of the application he built, which involved handling provisioning tasks through asynchronous worker processes communicating via SQS.
He explains that clients would request resource provisioning, with workers managing tasks like creating DNS records and handling API calls while ensuring that the process remained efficient and straightforward.
The speaker also discusses challenges he encountered as he unraveled project requirements and highlights the necessity for adaptability in response to evolving needs.
"We wanted to replace the enterprise load balancers with a self-service solution for developers."
The team decided to replace costly enterprise load balancers at Atlassian with an open-source, cloud-native proxy solution known as Envoy.
Envoy proxy is compared to Nginx but is touted as being more modern and flexible, allowing for dynamic reconfiguration via an API.
This transition aimed to enable developers to set up their own load balancing configurations without needing to communicate with the infrastructure team.
"I open-sourced this software and called it Sovereign."
The Envoy management server, referred to as Sovereign, was developed using a Fast API application to dynamically manage proxy configurations.
Sovereign polls configuration templates and context dynamically, generating API endpoints for the Envoy proxies to request.
Templates in Sovereign define resource types like clusters, routes, and listeners, which are used to produce configurations based on the current context.
"The management server polls various data sources to generate new configurations."
The context required for rendering configurations comes from a combination of a database and other data sources, including a potentially changing S3 bucket.
This dynamic management server enables the configuration to adapt to changing data, influencing how Envoy proxies handle traffic based on real-time context.
"These proxies are provisioned by a CloudFormation template."
To manage the deployment of numerous proxies, the team utilized AWS CloudFormation for infrastructure as code.
The CloudFormation template specifies resources like VPCs, subnets, and auto-scaling groups necessary for deploying the proxies across different regions.
Key components such as AMIs (Amazon Machine Images) are referenced in the template, not provisioned directly, indicating a structured approach to managing cloud resources.
"We need to produce a standard image for these proxies."
The process of creating AMIs was facilitated using HashiCorp Packer alongside Salt Stack, which is a tool for configuration management similar to Puppet or Ansible.
The plan involved provisioning an EC2 instance in a development environment, applying the Salt Stack configuration, and then creating an AMI from this instance for consistent proxy deployment.
This AMI included essential configurations such as installing and configuring Envoy, logging agents, and ensuring security hardening among other deployment requirements.
"Parameters at runtime would handle secrets and keys."
The CloudFormation template utilizes parameters that allow for the dynamic provision of secrets and keys at runtime while deploying proxies.
Once provisioned with the necessary configurations, the proxies become operational, ready to accept traffic and serve the services defined by the developers.
"This was the foundation of our team, essentially the product that we were going forward with."
The initial stage involved provisioning tasks where requests were made and prepared for deployment. These tasks included writing to a database and communicating the readiness status to the management server.
The management server acted as a broker to gather current state data, integrate it into templates, and create resources from those templates, effectively setting up long-lived infrastructure with CloudFormation.
CloudFormation utilized an Amazon Machine Image (AMI) for provisioning, laying a solid groundwork for the centralized load balancing managed by the team.
"We got products like Jira, Confluence, Bitbucket, Status Page, and many others behind this edge infrastructure."
The subsequent milestone was the integration of larger products with a newly developed platform component, facilitating centralized load balancing.
Migration of Atlassian's microservices was relatively straightforward, as the platform could enforce this transition by mandating usage of the new infrastructure.
Previously basic load balancing was deprecated, and services were required to explicitly use the centralized load balancing infrastructure, enhancing security and control over service exposure.
"There's a lot of extensions that can be applied to a listener or a cluster."
The Envoy-based infrastructure allowed for complex routing and handling of requests, facilitating diverse configurations including traffic management and response actions.
Development focused on ensuring that all parameters fed into this system were validated, helping to produce valid resources effectively.
The introduction of various extensions, such as network filters and HTTP connection managers, enriched the platform's capabilities, handling more intricate networking tasks dynamically.
"The platform allowed us to centralize management of resources and implementation of critical features."
One of the core advantages of the centralized infrastructure was addressing various security concerns, including DDoS protection, rate limiting, and authentication, all before requests reached back-end services.
This approach significantly reduced the development burden on multiple teams, streamlining processes and preventing potential security vulnerabilities.
The implementation of features such as access logging was achieved through dynamic configurations, reinforcing the effectiveness of the proxy in managing these tasks efficiently.
"We have a programmable proxy with sidecars that have their own separate logic from the proxy and can receive dynamic configuration."
The speaker discusses the implementation of sidecar containers within their architecture, which were created both by themselves and by other teams.
The authentication sidecar was specifically crafted by the speaker using Rust, emphasizing the use of their preferred programming language.
They elaborate on roles taken by other teams, including authorization and rate limiting through their respective sidecars, which contributed to a more efficient and modular system overall.
The speaker mentions a provisioning flow for automatic setup and configuration on an AMI, highlighting the importance of streamlined processes in software development.
"I have grown tremendously in my diplomacy skills, conflict avoidance, and conflict resolution."
The speaker reflects on personal growth in non-technical skills throughout their eight-year tenure at Atlassian, particularly in diplomacy, conflict avoidance, and resolution.
They acknowledge the importance of being able to persuade, propose ideas, teach, educate, and mentor others effectively within the workplace.
A mention of tedious compliance work indicates a shift away from creative problem-solving, suggesting a broader concern about the monotony of fulfilling non-technical requirements.
"When you build something, there’s a lot of onboarding initially, but over time, as people come and go, that onboarding process needs to happen again."
The speaker shares their observations regarding the maintenance of software and systems, stating that identifying weaknesses early on can help mitigate future problems.
They emphasize the necessity of comprehensive documentation and training for new team members to ensure they can effectively contribute and troubleshoot.
The speaker identifies ongoing challenges in maintaining and onboarding as team dynamics shift, stressing how new hires bring in varied perspectives and ideas, which can lead to changes in the existing codebase.
"I was exposed to different types of managers and colleagues over time, and everyone has different personalities and styles of working."
The speaker discusses the complexity of interpersonal relationships at work, noting that differing personalities can lead to conflicts, even among respected colleagues.
They highlight the significance of self-awareness and understanding others’ psychological profiles to manage potential conflicts proactively.
The mention of stress caused by interpersonal conflicts illustrates the impact these dynamics can have on performance and the importance of developing resilience.
"Mentoring is distinct from simply helping others understand; it's a challenging balance between providing guidance and allowing independence."
The speaker reflects on their experience mentoring an intern, noting the intern's success and high rating but also expressing their own struggles in balancing support with independence.
They articulate the challenge of not wanting to overwhelm the mentee with direct answers while still ensuring they don’t become frustrated.
Despite their difficulty with mentoring, the speaker is open about their own limitations, acknowledging that other colleagues helped the intern in areas where the speaker felt less capable.
"I was always available to help, and I could boil down hard topics into something that was understandable."
The speaker discusses the distinction between general mentoring and the specific type of mentoring they are uncertain about. They emphasize their role in training colleagues, which became a significant part of their responsibilities during the last half of their employment.
They recall how they actively participated in problem-solving sessions with colleagues, indicating that this collaborative work shaped their professional contributions.
Feedback received from colleagues highlighted the speaker's availability and ability to simplify complex topics, demonstrating their effectiveness as a resource and educator in the workplace.
"If people are interested, I could actually go through and build some of these things from scratch on stream."
The speaker contemplates making a follow-up video to cover additional topics left unaddressed in the current discussion.
They express a willingness to create content that showcases their skills, potentially teaching others through live streams or directly uploaded videos.
However, the speaker notes that the decision to pursue these ideas will depend on viewer demand and their personal workload, as they have many items on their to-do list.
