I replaced a manual, production-critical Excel process (3h/week) with Databricks Medallion ETL and automated reporting.

2. 🎯 Project Overview

3. 🧩 Why the Project Was Difficult: Before State

Chapter summary: The customer account team spent hours producing and sending each set of customer reports manually. The work was spread across regional customer inputs, spreadsheets, internal data sources, dashboards, AWS tools, QuickSight, Power Query, Google Drive, and email. This made the process time-consuming, error-prone, and difficult to understand end to end.

3.1 Manual, Time-Consuming, and Error-Prone Customer Reporting Workflow

Why the input arrived in separate files: Customers are not managed as one central entity. They have several independent regions, and each region provides its own ground-truth Excel file through its information-management workflow.

The customer account team had to complete the following work for each report cycle:

1. Collect regional files: Receive separate ground-truth Excel files from the customer regions.
2. Merge inputs: Combine the different regional files into one report input.
3. Reconcile history: Compare the new batch with the previous batch, which was typically sent about a month earlier.
4. Clean data: Clean the messy source data before it could be used.
5. Join internal data: Manually join the customer-reported ground-truth data with internal data that existed in AWS, Athena, and QuickSight.
6. Apply internal enrichment: Download data from a Databricks dashboard as an Excel file, join it with the other Excel data, and upload the result to QuickSight.
7. Apply QuickSight joins: Use QuickSight to apply further joins, then download the result again.
8. Make manual corrections: Fix parts of the output by hand after the QuickSight export.
9. Send the report: Email the final report to the customer.

3.2 Fragmented Data and Low Transparency

Source inventory — what systems did the manual workflow depend on?

The information required to produce the report was distributed across roughly twenty sources. The workflow passed through incoming customer email, a local Power Query process, a Google Drive folder containing the working files, a Databricks dashboard, and Databricks Delta tables owned by roughly three different teams. It then passed through a poorly understood legacy layer in AWS, Athena, and QuickSight, further local Excel work, and finally outgoing email distribution.

Why direct access to the sources was limited

Access to these systems was also fragmented. I did not have an Excel license or a QuickSight license, even though I was responsible for migrating the QuickSight-based process. Licenses were restricted to control costs, and it had been expected that the migration could be completed without direct access. In practice, this made the discovery substantially more difficult: I had to infer parts of the QuickSight setup through the AWS CLI or ask colleagues questions instead of inspecting it directly. Not every team member had access to every resource involved in the workflow.

Why the end-to-end workflow was not documented

No single stakeholder understood this process end to end. Each person or team performed one part and could explain that part, but generally did not know what the other stakeholders did before or after it. The complete workflow was not documented anywhere; it existed as fragmented tribal knowledge distributed across the people involved.

How I reconstructed the process from partial information

The source-and-tool inventory above was therefore not an input to the project. I created it by identifying the stakeholders, collecting each person's partial account, tracing how the outputs of one step became the inputs of another, and establishing which sources were required to produce the final customer outcome.

3.3 Fragmented Ownership and Conflicting Requirements

Stakeholder ownership — who controlled the requirements and inputs?

Ownership, knowledge, and requirements were as fragmented as the data. Six senior stakeholders across several teams each controlled a different part of the desired outcome, source data, existing system, or technical direction:

Why requirements could not be gathered from one owner

No stakeholder owned the complete process or had authority over every component. I had to reconcile the customer account team's desired customer outcome with incomplete and sometimes conflicting information from the owners of the underlying data and systems, while also satisfying the technical requirements set by my team lead and the principal data engineer. The external customer was the beneficiary of this production delivery, rather than one of the six internal stakeholders whose requirements I coordinated.

4. ✅ What the Project Delivered: After State

Chapter summary: The project automated recurring report production after the remaining manual file-intake step, expanded customer reporting coverage, and delivered a reusable Databricks data product. The outcomes and the concrete software deliverables are distinguished below.

4.1 Business Outcomes

The automation delivered three business outcomes:

• Multi-hour weekly time savings for the customer account team: Only one manual ingestion step remains: a colleague from the customer account team receives the new Excel files by email, uploads them to Google Drive, and adds them as sources in Power Query. From that point onward, data processing, report generation, recipient routing, scheduling, and email delivery run automatically without further human work. This eliminates multiple hours of recurring manual work each week and frees the customer account team to focus on revenue-generating activities.

• Increased customer satisfaction through expanded report coverage: The customer account team previously produced these reports manually for one customer. The automated process now serves three customers, so two additional customers receive reporting that was not available to them before.

• Reusable and consolidated internal data product: The data required for this workflow is now consolidated in Databricks Delta tables instead of being spread across AWS, QuickSight, dashboards, and disconnected files. As part of that consolidation, I also corrected gaps in datasets owned by other teams. Account and data-science teams now have one clear, accessible source that they use for further analysis and additional value-driving insights.

4.2 Software Deliverables

The delivered software covers the complete workflow after the remaining manual file-ingestion handoff:

Manual Excel-file handoff
    -> Spark Medallion ETL and data product
    -> Consolidated gold Delta tables
    -> Excel report generation
    -> Recipient and regional routing
    -> Time-zone-aware scheduling
    -> Email delivery

Software components delivered:

• Spark Medallion ETL bundle: Ingests the required source data into Databricks, reconciles and cleans it, enriches it with the required internal data, preserves history, and supports both full and incremental processing. Its reusable batch-ETL framework provides schema contracts, validation, composable processing steps, metadata, and shared execution infrastructure.
• Consolidated gold-table data product: Provides the canonical, documented Delta tables consumed by both the reporting automation and other internal analysis.
• Excel report generator: Reads the gold tables and produces the formatted customer reports.
• Recipient and regional routing: Uses the configured users and groups to select the correct recipients and report content for each customer region.
• Scheduled Databricks jobs: Run the required report flows at the correct times for the different customer schedules and time zones.
• Automated email delivery: Sends the generated reports to the configured customer mailing lists without further manual processing.

5. 🧭 How I Decomposed the Problem

Chapter summary: I split the work into two independent implementation tracks: reconstructing a reusable data product and automating the customer reports that consume it.

Why I chose two implementation tracks: I designed this problem-solving approach and decomposed the project into the following steps myself; the breakdown was not provided by my manager. After developing the approach, I presented it to the principal data engineer, who approved it. I separated the reusable data product from the report-generation and delivery automation through a clear interface between them. This allowed independent agents to implement different parts concurrently without creating conflicting changes, shortening the implementation compared with purely sequential work.

Data-product track

1. Define done: Create a canonical target Delta table from the reference report and a source Delta table from the starting Excel files.
2. Reproduce the target as a black box: Use AI agents iteratively to reproduce the canonical output column by column, while discovering missing information and correcting wrong assumptions.
3. Make the generated implementation manageable: Refactor the codebase until its review, testing, extension, and production operation were manageable.
4. Make processing computationally cheap: Add slowly changing dimensions and incremental processing so the pipeline does not need to perform full backfills across millions of rows.

Report-automation track

1. Define the report target: Use the reference Excel report as the desired output and the canonical gold Delta table as the input.
2. Reproduce the Excel report as a black box: Use AI agents iteratively to generate the report, compare it with the reference, and resolve discrepancies.
3. Feed data-quality issues back into the pipeline: When report discrepancies revealed underlying data problems rather than formatting bugs, correct the data-product pipeline.
4. Recreate recipient and schedule configuration: Extract recipients, regions, and schedules from the customer account team's Google Calendar configuration.
5. Automate delivery: Create the required users and groups, configure scheduled Databricks jobs, and send the correct reports by email.

6. 🏗️ Build a Correct, Maintainable, and Efficient Data Product

Chapter summary: I first made the expected output testable, then reconstructed it with AI agents, refactored the resulting code into a production Spark system, and added incremental processing.

Implementation steps: Sections 6.1–6.4 describe these four stages of the data-product work.

6.1 Define Done with Source and Target Fixtures

Step summary: The first step was to make the definition of done executable by creating concrete source and target fixtures.

Turn high ambiguity into a testable definition of done

The existing process contained too much uncertainty, ambiguity, and unknown information for me to design the pipeline bottom up in advance. A bottom-up design would have required knowing the final goal and being able to predict the sequence of source tables, intermediate tables, joins, and transformations needed to reach it.

In this project, much of the system existed as tribal knowledge distributed across people in different teams who performed different parts of the manual process. I could not begin by confidently decomposing that process into a predefined architecture. The only certain thing I had was what the final report should look like.

I therefore started from the known end result and used it to reverse-engineer both the required transformations and the architecture of the pipeline. Defining the source and target fixtures turned an ambiguous outcome into something concrete and testable before I understood the full path between them.

1. Create the canonical target: I extracted the canonical data from the reference Excel reports I had received and stored it in a Delta table. This table represented the exact output that the pipeline should reproduce.
2. Create the source fixture: I ingested the starting Excel files into a separate Delta table.

Together, these two Delta tables gave me a known source dataset and a canonical target dataset against which I could compare the pipeline output.

6.2 Reproduce the Canonical Target Iteratively with AI Agents

Step summary: With the source and target fixtures in place, I used AI agents iteratively to reproduce the canonical result column by column.

Engineering challenges and responses: The following sections explain the decisions, obstacles, and verification methods involved in reproducing the target.

Verify correctness from observable outputs before stabilising the code

During this phase, I skimmed the generated code, but I deliberately did not try to prove the correctness of every implementation detail by reviewing the code itself. I knew the implementation would change repeatedly while I was still discovering what the process needed to do, so a detailed code review at this stage would have meant repeatedly reviewing code that would soon be replaced.

Instead, I evaluated the implementation through its output. The source fixture was the input, the canonical fixture was the expected output, and matching the two defined success. I asked the agent to run the pipeline, compare the generated Delta table with the canonical target, and identify the discrepancies. I also inspected parts of the resulting tables myself. The next iteration focused on explaining and resolving those concrete discrepancies rather than judging the implementation from the code alone.

Deliver verified progress one gold-table column at a time

I did not begin by defining all the bronze and silver tables, because I did not yet know which tables or data sources the pipeline would require. I did know which columns the final gold table needed. I therefore created a checklist of the target columns and built the pipeline one gold-table column at a time:

1. Select one required column from the canonical target.
2. Trace that column backwards through all required cleaning, transformations, joins, and source data.
3. Let the agent implement the path from the raw input to that one gold-table column.
4. Run the pipeline and compare every produced value in that column with the canonical target.
5. Investigate and resolve the remaining discrepancies.
6. Commit the work only after that column matched end to end, then continue with the next target column.

This kept the bronze and silver layers inside the initial black box. At this point, their structure was allowed to evolve; the stable contract was that each completed gold-table column had to match the corresponding canonical column exactly.

Make every column's origin and use traceable

The same logical column could appear in several tables or Excel files and could be used as a key for joining those sources. Referring to it only through repeated string literals would have made it difficult to find every use in the IDE or understand how it moved through the system. I therefore developed a column-centric model that made the relationships between canonical columns navigable.

The model distinguished the different roles columns played in the pipeline:

• Pass-through and metadata columns: Values propagated from the raw input through intermediate tables to the gold table.
• Join-key columns: The same logical column appearing in different sources and connecting those sources.
• Enrichment columns: Values brought into the result by joining another table or file.
• Fact columns: Source facts preserved in the result.
• Conclusion columns: Values derived by calculating a result from other columns.

This provided a graph-like overview of where a canonical column appeared, how it was produced, and where it was used. It also made the implementation easier to navigate through go-to-definition and reference searches than code based on repeated hard-coded column names.

Turn dozens of open questions into a controlled work queue

At some points I had dozens of simultaneous questions: why a value had a different sign, why a row was missing or duplicated, which source was responsible for a number, whether a filter was missing, or whether I needed clarification from another person. Keeping these questions in a notes application or Markdown file produced walls of text that were difficult to maintain and use.

I therefore developed a programmatic discrepancy-management framework. Its tests reported both the number and the types of discrepancies that remained, including:

• missing or additional columns;
• missing or additional rows;
• incorrect values;
• unexpected null values.

The framework also distinguished discrepancies that were known or unknown and resolved or unresolved. This turned the remaining uncertainty into an inspectable work queue and made it possible to see the current state of the reproduction at any time.

Keep delivery moving despite limited stakeholder availability

Much of the information required to resolve these discrepancies existed only as tribal knowledge in the customer account team. The relevant stakeholders were based in the United Kingdom while I worked from the German office, so we did not have regular in-person contact. They were also frequently travelling and had limited availability.

This made the volume and format of the questions difficult to handle asynchronously. A single discrepancy could require a nuanced paragraph of context, and at some points I had roughly fifty open questions. Sending many long questions over Slack did not reliably produce timely answers, while the next meeting might not be available until the following week. I still needed to make progress between those meetings despite not yet having access to all the knowledge required to explain the data.

The black-box comparison and programmatic discrepancy framework made that possible. I could preserve each unresolved question precisely, continue with discrepancies that did not depend on a stakeholder answer, and return to the blocked items when the relevant person became available.

Reproduce historical results despite continuously changing data

The reference reports represented a point in the past, while the underlying data continued to change. Running the same transformations against current source data could therefore produce a different result even when the code was correct.

To reproduce the reference output exactly, I had to identify and load the correct historical version of each relevant source at the date represented by the report. This meant coordinating different historical versions and dates across the source systems. When I later evaluated the production pipeline with current data, the discrepancy framework could distinguish expected differences caused by the newer data from unknown discrepancies that indicated a data-quality issue.

Reduce each Databricks feedback cycle from fifteen minutes to one

A normal Databricks pipeline run could take roughly fifteen minutes because of compute start-up and execution time. That was too slow for a workflow that required many source-to-target comparisons. I therefore kept most Python logic in importable modules rather than notebooks and built a local entry-point script using Databricks Connect. The local runner executed the same logic against Databricks without waiting for a new cluster start-up, reducing an iteration from roughly fifteen minutes to roughly one minute.

Recover undocumented business logic through measurable discrepancies

Resolving the discrepancies was also how I discovered the undocumented process:

• I repeatedly found that information I needed had not been provided earlier.
• New information sometimes showed that the current implementation was moving in the wrong direction.
• A newly discovered requirement could require throwing away part of the existing work and rebuilding it differently.
• I used the canonical output to identify which sources were needed and how data from Google Drive, AWS, Athena, and different Databricks Delta tables had been combined.
• The rejected QuickSight discovery story would have represented two or three story points, but I still performed that discovery work inside this reproduction phase.

By the end of this step, the pipeline reproduced the canonical target. The code was still a generated mixture of SQL, Pandas, and some Spark and was not yet ready for production; making that implementation understandable, testable, and maintainable was the separate refactoring step that followed.

6.3 Refactor the Generated Code into a Production System

Step summary: The reference output was correct, but the generated implementation still needed a coherent Spark architecture, explicit data contracts, and maintainable execution and validation paths.

Productionisation challenges and responses: The following sections describe the risks and design decisions behind that refactoring.

Expose the production risk hidden behind correct output (generated SQL, Pandas, and Spark prototype)

The black-box reproduction step generated several thousand lines of code. It reproduced the reference result, but the implementation was chaotic, contained many edge cases, and was not yet something I could responsibly review, test, extend, or operate as production code.

The generated implementation was a mixture of SQL, Pandas, and some Spark. I had initially used SQL and Pandas because those were the tools I knew, but production-critical applications at the company are required to use Spark.

Regain control over thousands of lines of unfamiliar code (learning Spark and repeated refactoring)

I therefore learned Spark, discarded most of the initial SQL-based pipeline, and rebuilt the implementation as a coherent Spark system. The existing SDK is focused on streaming and was too complex for this non-streaming use case, so I also had to design a separate architecture for the batch ETL pipeline without an internal reference implementation to copy.

I remained accountable for the correctness of the system. To review and understand the generated implementation, I had to reduce its complexity, remove repetition and hard-coded assumptions, and create a structure in which I could reason about every new change. Productionising the generated reference implementation required sustained, high-effort review and repeated refactoring.

To handle the complexity, I repeatedly broke the problem into smaller parts and refactored the implementation until every component had a clear place and purpose. By the end, the codebase was easy to navigate, review, and extend. I understood every component and could reason about what each part of the code did, which gave me high confidence in its correctness and in deploying it to production.

Turn the refactoring into a reusable greenfield batch-ETL methodology (self-designed Spark architecture)

The result of this journey was not only a cleaned-up implementation for one pipeline. I designed a reusable methodology for building and operating this type of batch ETL from scratch. The following components are parts of that methodology.

Make data contracts explicit and transformations reviewable (schema-driven ETL steps)

I developed an ETL framework in which each processing step consists of three explicit components:

1. Input contracts: A list of schemas for the input tables.
2. Output contracts: A list of schemas for the output tables.
3. Transformation: A function that receives a DataFrame and returns a DataFrame.

This separates table structure from transformation logic. Column definitions and validation rules are declared once in the input and output schemas rather than repeated or hard-coded inside business logic. A developer can navigate directly to the input or output schema, while the transformation function remains focused on cleaning, joining, and transforming the data.

Isolate changes and avoid unnecessary compute (composable and selective execution)

I broke large transformations into partial transformations that each take a DataFrame and return a DataFrame. These partial transformations can be composed into a complete step between an input and an output schema.

Multiple pipeline steps can then be composed into larger executable units. This makes it possible to run a relevant subset without rerunning the entire pipeline when its upstream data has not changed. The design isolates processing steps, avoids unnecessary runs, and reduces wasted compute.

Validate table contracts and document data (validation and metadata)

The framework uses four table-level validation controls and metadata conventions:

• Schema validation: The framework uses the declared schemas to validate pipeline inputs.
• Known values: Enum definitions identify allowed values and make it possible to detect rows containing unknown or corrupt values so they can be handled or filtered.
• Primary keys: Tables define primary keys and enforce their uniqueness.
• Documentation: Every table and column has descriptions and metadata; tables also carry metadata tags.

Centralize pipeline execution (shared entry point)

Single entry-point abstraction: Instead of maintaining a separate entry-point file for every pipeline step—which could result in roughly twenty entry-point files—I abstracted entry-point behaviour behind one file and a set of classes. This reduced the number of files and removed duplicated orchestration code.

Together, these components turned the generated black-box implementation into a production system whose complexity I could manage and established a methodology I can reuse in later data-engineering work.

6.4 Add Incremental Processing with Slowly Changing Dimensions

Step summary: After making the codebase manageable, I reduced its computational cost. I introduced slowly changing dimensions so the pipeline could process changes incrementally instead of repeatedly performing full backfills across millions of rows.

7. 📬 Automate Customer Reporting End to End

Chapter summary: Against the gold-table interface, I reconstructed the reference Excel report, traced report discrepancies back to source data where necessary, restored routing and schedule configuration, and automated delivery.

Implementation steps: Sections 7.1–7.5 describe the five stages of report automation.

7.1 Define the Report Target

Once the boundary was clear—the ETL pipeline produces the reusable gold-table data product—the report generator could be implemented independently against that interface. This allowed one agent to work on report generation while other agents continued working on the data pipeline, without the implementations conflicting with each other.

The canonical gold Delta table was the input to this track, and the reference Excel report supplied by the customer account team defined its required output.

7.2 Reproduce the Reference Excel Report

I treated report generation as a second black-box reproduction problem:

1. Use the canonical gold Delta table as the input.
2. Ask an AI agent to write code that turns the Delta-table data into an Excel report.
3. Generate the Excel file.
4. Compare it with the reference Excel report supplied by the customer account team.
5. Identify discrepancies and iterate until the generated report matched the desired result.

Some discrepancies were bugs in the report-generation code, such as an oversized logo, incorrect formatting, or a column whose values had disappeared. Other discrepancies exposed questions that required stakeholder clarification.

At first, I deliberately did not review the report-generation code line by line. I focused on whether the generated Excel report matched the agreed target. Once the black-box loop produced the correct report, the implementation could be reviewed and productionised.

The visual comparison loop was very time-consuming because the AI coding tool I used during the project was not approved to read the customer-data Excel files. The agent could generate a report but could not inspect its own output or compare it directly with the reference. I therefore had to inspect every generated file myself, describe each visual or data discrepancy, and reprompt the agent manually.

After the deadline, I switched to an AI-tool subscription approved for the required level of data privacy. With that tool, the agent can directly inspect the generated and reference files, identify discrepancies, and fix bugs, making the same workflow substantially easier.

7.3 Feed Data-Quality Issues Back into the Pipeline

Some discrepancies in the generated report revealed data problems that the earlier data-product comparison had not caught. When that happened, I traced the discrepancy back into the data-product pipeline, corrected the underlying data processing, and regenerated the report. This prevented report formatting work from hiding data-quality issues in the canonical data product.

7.4 Recreate Recipient and Schedule Configuration

I then reconstructed the delivery configuration for recipients across customers and regions.

1. Extract recipients and schedules: I used Google Apps Script to fetch email recipients, report schedules, regional routing, and related configuration from the customer account team's Google Calendar.
2. Create users and groups: I recreated the required Cognito users for the recipient email addresses and created the corresponding AWS user groups.

Protect customer data during agent-assisted development

Customer files and recipient email addresses could not be exposed to AI tools that were not approved for that data-protection level. This affected both the Excel comparison loop and the user-routing work.

To work safely with agents, I created an agentic safety setup with two layers:

• Skills and instructions: Defined how agents were allowed to work with the repository and sensitive data.
• Deterministic hooks: Blocked agents from accidentally reading protected files containing customer data or email addresses.

These controls allowed me to use agent-assisted development without relying only on the agent to remember privacy rules, while keeping customer data and recipient information outside tools that were not approved to process them.

7.5 Automate Delivery

1. Configure scheduled execution: In the Databricks report-generator bundle, I created jobs running at different times to represent the required schedules.
2. Route and send reports: The system selects the correct report and recipients for each customer region and sends the reports automatically by email.

This part was necessary to complete the end-to-end automation, although the data-product work was the more technically difficult part of the project.

8. 🧠 Capabilities I Can Apply to Future Projects

Chapter summary: The project expanded both the systems I can build and the engineering and coordination decisions I can own in future work.

8.1 Build Production Spark and Batch-ETL Systems

For future ETL and analytics pipelines, I can draw on these technical capabilities:

• Spark: Learned Spark while rebuilding the production pipeline, expanding from prior Pandas and SQL experience to a skill set that can contribute to other Spark pipelines.
• Batch-ETL architecture: Designed an architecture for a non-streaming Spark ETL pipeline where the existing streaming-focused SDK was not suitable.
• Data discovery: Traced and understood unfamiliar customer-reported ground-truth and product predictive-indicator data, including what was undocumented or initially misunderstood.
• Databricks migration: Migrated pipeline and reporting work from AWS and QuickSight to Databricks.
• SQL and metrics: Adapted existing queries and domain-specific metrics to fit the new pipeline and reporting work.

8.2 Coordinate Stakeholders and Maintain Traceable Requirements

I coordinated the work directly with the six internal stakeholders rather than relying on my manager to manage the communication. The project crossed several teams: the customer account team was the internal customer for the automation, while the required data, existing systems, and technical decisions were owned elsewhere.

The requirements came from four distinct areas of responsibility:

• Internal customer needs: The customer account team needed the existing manual reporting process automated.
• Data ownership: The data scientist owned parts of the AWS and Databricks data required for the work.
• System ownership: Another team owned the Lambda and a separate Delta table that also processed the Excel-file inputs.
• Technical direction: My team lead and the principal data engineer set technical requirements, including the Spark requirement for the production pipeline.

I had to coordinate these requirements simultaneously while continuing to design and build the pipeline.

Stakeholder-management practices: The following sections explain how I preserved decisions, kept questions actionable, and routed communication to the relevant people.

Preserve requirements and their evidence for people and agents

I used an agent-assisted form of spec-driven development similar to the approach promoted for Kiro. I maintained Markdown files that preserved each requirement together with its evidence. The evidence could be a Slack message, a meeting transcript, or a dated note I wrote immediately after an unrecorded meeting. Requirements received stable identifiers such as R1, R2, and R3.

This gave the agents a durable source of truth instead of relying on the current chat context or my memory. I could repeatedly ask an agent to review the implementation against every numbered requirement and report which requirements were implemented, incomplete, contradicted, or still unclear.

Separate measurable discrepancies from stakeholder questions

I tracked output discrepancies programmatically through automated tests. At each committed stage, the tests made explicit which differences remained between the generated result and the canonical target.

Questions that required human knowledge were tracked separately in a Markdown file maintained with agent assistance. Each question had an explicit status, so unresolved items survived the long gaps between stakeholder meetings rather than being lost in Slack history or personal notes. Before a meeting, I could batch the relevant questions; during and after the meeting, I could record the answers and have the agent update their status and the affected requirements.

Route each message only to stakeholders who could act on it

I organised different communication loops for the project rather than sending every update to everyone:

• Inside-team feedback loop: My manager and the colleague reviewing my code received the implementation and review discussions relevant to the team.
• Primary stakeholder loop: The customer account team stakeholders received the questions and decisions that affected the customer-facing outcome.
• Direct messages: Individual questions went directly to the person who could answer them when the wider group did not need the interruption.

This reduced noise for colleagues who could not act on a message while still keeping the necessary stakeholders informed.

Make complex questions easy to triage before asking for attention

Long collections of detailed questions were unlikely to receive a response when posted directly into a Slack channel. I therefore kept the main Slack message concise enough for a stakeholder to decide whether the issue required their attention. I placed the supporting context and detail in the thread, then used a meeting when the issue required a deeper discussion or several related decisions.

Together, these practices turned fragmented stakeholder knowledge into traceable requirements, controlled open questions, and reviewable decisions. They also gave me a reusable method for leading agent-assisted delivery when stakeholders are busy, distributed across teams, and not continuously available.

8.3 Produce Maintainable Architecture Without an Internal Reference

The architectural practices I can reuse from this work are:

• Large-codebase design: Used abstractions, classes, layering, and refactoring to manage a new multi-thousand-line application built alone in a technology that was new to me.
• Extensibility: Built a codebase intended to make future iterations faster and safer.

My lead reviewed the ETL architecture and compared it with other more critical production applications. He described my code as resembling a backend Java service, with clear abstraction layers, classes, and organised code. He also noted that some higher-stakes production applications were simpler, less refactored, and contained more duplication. This review provided external evidence that I had produced an unusually maintainable architecture despite building the system alone, in a new technology, and without an internal reference implementation.

This capability has also carried forward into more critical production work:

Transferable expertise for higher-criticality data systems: Designing this first-of-its-kind Medallion architecture from scratch gave me skills and experience that now improve my contribution to another, even more production-critical Medallion architecture. It also gave the team the confidence to trust me with implementing a new status field in one of the company's highest-criticality data products. The additional status-field work is effectively the second project of this kind, so I can apply the lessons from this project and contribute design ideas that are also new and valuable to the wider team.

8.4 Evaluate Top-Down Validation and Incremental Delivery Trade-Offs

Capability demonstrated: Evaluate implementation strategies against correctness, reviewability, and delivery speed rather than treating any one approach as universally superior.

Review feedback: what delivery trade-off went wrong?

From my lead's perspective, two fair criticisms were visible in the delivery pattern: I went several days without merging an intermediate result, and the eventual pull request was large and appeared to have a higher level of code quality than the task required. He had expected a different trade-off: faster delivery with a simpler implementation.

Root cause: why the large implementation could not be merged incrementally

Those observations were correct, but they described the consequences rather than the decision that caused them. The root cause was the earlier approach in sections 6.1 and 6.2: define the canonical output, treat the implementation as a black box, and let agents work top down until the complete table matched. I proposed this approach, and the principal data engineer agreed that it was worth trying. What neither of us could know in advance was how large and tightly coupled the generated implementation would become.

Once several thousand lines of generated code produced the correct output, I could not safely extract and merge one small, cohesive unit. The components were too entangled. Refactoring and introducing abstraction layers were prerequisites for separating the implementation into units that I could understand, test, review, and merge. This made the recovery work look like over-engineering and made the delivery pattern look inconsistent with continuous delivery, even though I understood and wanted to follow both principles. At that stage, the earlier architectural bet had already removed the option of delivering small increments quickly.

Alternative risk: why bottom-up delivery alone was not a guaranteed fix

A purely bottom-up alternative would also have carried substantial risk. Because the required sources, transformations, and intermediate architecture were unknown, I could have built clean, mergeable components that moved toward the wrong result. Without continuous comparison against the canonical target, discovering that mistake later could have required major rework or many additional stakeholder meetings.

Future approach: alternate output validation with small delivered slices

My current learning is therefore not that the top-down approach was simply wrong or that the hypothetical bottom-up approach would certainly have worked better. I need to test a hybrid approach in future projects: keep the canonical output as the top-down correctness check, but implement only a small vertical slice at a time; refactor it into an understandable unit; test and merge that unit; then extend the next slice. This would alternate output validation, productionisation, and delivery instead of postponing all refactoring until the complete output had been reproduced. Trying both approaches in comparable work will give me stronger evidence about when each is appropriate.