Data pipelines play an important role in enabling the flow of information in real estate applications for everything from property listings and market analytics to automated home value estimates with tools like Xome’s ValueYourHome®.
Xome® collects massive amounts of data from various sources to equip home buyers, sellers, and real estate investors with actionable insights. However, this raw data can often be messy, unstructured, and stored in different formats across systems.
To streamline the processing of this data, our DevOps team at Xome automates the entire data pipeline through a managed continuous integration and continuous delivery (CI/CD) process using Azure DevOps. This includes both automated database changes and Azure Data Factory resource deployments.
Optimizing database deployments with Azure DevOps
Updates to source data structures, such as property listings, typically require changes to both the database and the pipelines that process them. Automating and integrating these changes into the same DevOps pipeline deploys them in the correct sequence, maintaining compatibility and preventing failures due to outdated schema references or missing datasets.
This is one reason we adopted Azure DevOps as the CI/CD platform during our cloud migration to manage end-to-end automation and orchestration of deployments. This way, we can automate database deployments and make them auditable when managing schema updates across the full environment lifecycle from Development to Quality Assurance (QA), User Acceptance Testing (UAT), and Production.
A few ways we implemented this include:
-
- Version-controlled SQL scripts: All scripts are checked into a Git repository, enabling traceability and collaborative development.
-
- Automated execution: SQL scripts are run using SqlPackage.exe or SQLCMD for consistency across environments.
-
- Support for DacPacs (Data-tier Application Package): We implemented support for incremental DacPac deployments, pre-deployment validations, and post-deployment scripts to handle more complex update scenarios.
To reduce the risk of unintended schema alterations and data loss, we also put in place approval gates that require a formal review and sign-off from database administrators before changes can proceed to higher environments. As a result, we’ve been able to reduce manual errors and maintain a complete version history of all schema changes.
Automating Azure Data Factory deployments with Azure Resource Manager Templates
While the database manages the structure of the target data storage, Azure Data Factory orchestrates the movement and transformation of that data, creating a smooth pipeline from ingestion to loading.
To streamline and standardize the deployment of Azure Data Factory pipelines and resources, we adopted Infrastructure-as-Code (IaC) practices using Azure Resource Manager (ARM) templates.
Azure Data Factory components, including pipelines, datasets, linked services, and triggers, are defined in parameterized ARM templates. Instead of hard coding environment-specific details, like storage account names or URLs, parameters are injected during deployment using:
-
- Reusable templates: The same template can be deployed to all environments with minimal adjustments.
-
- Pipeline-integrated deployments: Templates are deployed using Azure DevOps pipelines and guarantee repeatable deployments across environments.
Azure Data Factory deployments are a part of our overall CI/CD pipeline that aligns with application releases. This ensures that data transformation workflows stay synchronized with app logic and reduces the likelihood of data mismatches or broken dependencies caused by schema changes or outdated pipeline configurations.
By automating the configuration and deployment of Azure Data Factory resources, we’ve been able to deliver data solutions quicker with improved visibility into infrastructure changes. Schema updates and ETL (extract, transform, load) logic can be released together, avoiding mismatches between data structure and pipeline behavior.
Accelerating real estate innovation with automated data workflows
Since leveraging Azure DevOps pipelines and Azure Data Factory for automated deployment and data orchestration, we’ve created a reliable data pipeline that helps us to deliver our digital real estate solutions quicker and with more consistency.
Having deployment of data processing fully integrated with our CI/CD platform allows us to streamline our deployment process across environments and provide our users with the most up-to-date property details.
To learn more about how Xome is leading advancements in real estate technology and innovation, browse our other articles on the Xome Tech Hub.
