The New York Times sends almost 4 billion emails a year to its customers. So it is no small thing when the paper overhauls its email platform.
In a story today, the Times reports that it moved its infrastructure to the cloud to the cloud earlier this year. The piece is noteworthy on two fronts. First, it is a detailed case study of an enterprise email system. And it is a remarkable display of transparency. Here is a summary.
Most of the Times’ emails are newsletters for audiences that vary in size from tens of thousands to millions. But a small portion are transactional emails such as paid subscription confirmations.
Before this, the paper was saddled with an unscalable system full of legacy code. So it decided to move to the cloud and build a platform that could grow with demand. To this end, a serverless architecture built on top of Google Cloud Platform.
The Times writes that “the core sending infrastructure relied on layers of archaic and often undocumented scripts to select audiences and compile emails. Countless frustrating hours of debugging and coaxing were needed to address the near-weekly failures.”
In technical terms, the Times relies on “a GAE API with Datastore to manage each individual user data object. Every time a user object is modified, a record of the updates to the data (e.g. a new subscription to a given newsletter product) is published to a Pub/Sub from which it is then inserted into BigQuery, GCP’s data warehouse that runs SQL-like queries.”
The paper continues: “GAE’s quick scaling means that many instances of the email compiler can spin up very quickly to handle the potentially millions of compile tasks coming from the Pub/Sub, and this architecture makes it easy to add additional API calls as data sources for email content (such as the personalization API or an A/B testing framework) without re-tuning the scaling.” Transactional email dispatches also go through this pathway.
The result? The Times writes that the news system enables it to:
The system also provides these benefits:
The Times concludes: “We now have confidence that we can rely on a platform that’s scalable, resilient, and sustainable. Rather than simply trying to keep things running, we’re now able to focus on building out features and tools that will help editors create more engaging newsletters to build even deeper relationships with our readers.