In Spring 2025, I completed my Bachelor’s in Computer Science at SUNY Polytechnic Institute. For my capstone project, I designed and built a data pipeline and analytics dashboard focused on the Central New York real estate market.
This project reflects my approach to building systems that collect, process, and present data in a meaningful way—particularly when working with external APIs and real-world data constraints.
My capstone project involved designing and building a data pipeline to collect publicly available real estate data and make it accessible through an interactive analytics dashboard. The goal was to provide focused insight into the Central New York real estate market.
Project Overview
Part 1: ETL Pipeline for Data Collection
Part 1: ETL Pipeline for Data Collection
Developing the data pipeline posed several challenges, including limited availability of free real estate data sources. Leveraging public APIs provided by New York State, I implemented a pipeline capable of:
- Extracting property assessment data for Central New York counties (Cayuga, Cortland, Madison, Onondaga, and Oswego) via REST APIs using SodaPy
- Transforming raw data into a normalized schema using Pydantic models, ensuring consistency and validation
- Enriching data with additional values required for frontend use
- Loading processed data into a relational SQLite database
The pipeline can be run locally or via a GitHub Actions workflow:
https://github.com/michellemark/etl-pipeline/actions
- Persisting data to AWS S3 using Boto3, including version tracking to reduce unnecessary transfer costs
The system was designed with extensibility in mind, allowing for additional data sources (such as ATTOM APIs) to be integrated in future iterations.
Part 2: Real Estate Analytics Dashboard
The second part of the project focuses on making the collected data usable and accessible.
Using Streamlit, I built an interactive web application that:
- Visualizes key insights from the dataset
- Allows users to explore data through filterable tables, charts, and graphs
The dashboard is publicly hosted on Streamlit Cloud using free-tier infrastructure.
View The Repository
Explore the Live Dashboard (If asleep, please wake it up).
Skills and Technologies Utilized
This project demonstrates experience with:
- ETL pipeline design and implementation
- Working with REST APIs and external data sources
- Data validation and transformation
- Relational database design
- Cloud storage using AWS S3
- Data visualization using Python and Streamlit
Technologies used include: Python, SodaPy, Pydantic, Pandas, Boto3, and Streamlit.
How This Relates to My Work
While this was an academic project, it reflects the same types of challenges I’ve worked on professionally—particularly around working with APIs, handling data inconsistencies, and building reliable data workflows.