Distributed Systems

ttd-databricks — PySpark SDK for Databricks → TTD Data APIs
  • Open-source PySpark SDK that lets clients push segment data from Databricks to The Trade Desk’s Data APIs
  • Incremental Delta Lake batch pipelines with checkpointing, per-row error tracking and distributed HTTP connection reuse across Spark workers
ttd-data — Python SDK for The Trade Desk Data APIs
  • Open-source PyPI SDK for The Trade Desk’s Data APIs, adopted by clients driving 5,000+ requests/second of production traffic
  • Standardized authentication, retries and rate-limit handling across client integrations
Surfstore — A Fault-Tolerant Dropbox Clone
  • Built a fault-tolerant distributed file store scaling to 128 nodes with concurrent shared-file access and full CRUD support
  • Used RAFT-replicated MetadataStore and BlockStore services to maintain availability with up to 50% node failures
Networked Sort — Distributed Sorting Program
  • Developed a distributed sort over raw socket programming spanning 16 server nodes
  • Built an even data-partitioning algorithm that sorted 2GB+ datasets in under 15 seconds