News UK (part of global media corporation News Corp) is one of the leading multichannel media businesses in the UK and Ireland. It creates content that entertains, informs, and inspires. Its brands include The Times, The Sunday Times, The Sun, talkSPORT, Virgin Radio, Times Radio, and talkRADIO.

Google Cloud results

• Unifies first-party customer data with product service interaction data, giving teams an integrated view of reader behavior
• Enables sales teams to improve accurate insights for advertising partners, creating a more targeted browsing experience for readers
• Drives innovation and customer personalization by empowering the data platforms team to support more self-serviceable business intelligence queries

The newspaper industry has faced big challenges over the last 20 years. The way people consume news media has changed irrevocably. Readers are now more likely to swipe through their daily dose of news on their phones than buy a printed newspaper. And they want news that is tailored to them.

At News UK, a media group best known for its big print titles such as The Sun and The Times, its experienced editors carefully curate relevant, interesting content on its digital news platforms and applications for its readers. Digital readers have overtaken print ones across News UK’s brands: The Times and The Sunday Times have more than 380,000 digital-only subscribers. Finding ways to engage these readers is central to the company’s success.

That’s not without challenges though, explains Karthik Ramani, Head of Data Platforms at News UK. “In the digital space, there’s even more competition for the attention of our audience and advertisers than in print. To stay on top, we have to foster a loyal relationship with our readers, providing them with great content, while maximizing ad revenue from targeted advertising.”

To balance both these goals, News UK’s business teams must make increasingly data-driven decisions. "We’re committed to providing our digital readers with the best possible content and services, especially for our subscription-based services," explains Karthik. "Accessible data insights are crucial to enable the newsroom to curate more relevant content for our audiences, while increasing our ad revenue and supporting our sales and marketing teams."

"We want to create a sustainable, cost-effective cloud strategy that will provide flexibility to support business decisions over the next five years, so we're looking at solutions that can grow with us. That’s what we’ve found in Google Cloud and its BigQuery data warehouse."
—Karthik Ramani, Head of Data Platforms, News UK

Reducing maintenance, improving access across teams

Karthik's team is responsible for data warehousing and data engineering and managing data processing and the cloud infrastructure behind it. But when Karthik joined News UK in June 2018, connecting the dots between News UK's first-party customer datasets and the data generated from customer interactions on its websites and apps was a challenge. With two data warehouses on two separate cloud platforms, the marketing, analytics, data science, subscription, and customer services teams couldn't easily gain insights due to information silos.

Because behavioral data was stored separately from data relating to demographics and subscriptions, end users had to manually combine these datasets in marketing and analytics tools to deliver insights to the business teams. With duplicate workflows required for every single use case, drawing insights from data was a time-consuming process.

"Having two parallel data warehouses made things harder for business teams that needed a holistic view of our data assets,” says Karthik. “It wasn't cost-effective, as the only way to fully integrate them was to transfer data between the warehouses or within external tools such as tableau. That required extra engineering work and transfer costs.”

What Karthik and his team needed was a single source for both batch and real-time data, to provide deeper insights into News UK’s customer behavior with lower maintenance efforts. When the rollout of GDPR required an audit of News UK’s data compliance strategies, the team used it as an opportunity to also rethink the company’s overall approach to data.

"We wanted a scalable solution for data storage and access, and based on our previous experience, BigQuery was the ideal choice. The serverless aspect is a huge win, enabling us to focus on adding value to the business, not worrying about our data warehouse’s backend scalability."
—Karthik Ramani, Head of Data Platforms, News UK

Selecting BigQuery for a solution that’s growth-ready

News UK had already been using BigQuery for its data science use cases for five years, so the digital datasets for customer interactions were already on Google Cloud. With firsthand knowledge of the scalability and performance of Google Cloud, the team decided to expand its footprint, streamlining its data warehousing with a single, flexible solution.

"We want to create a sustainable, cost-effective cloud strategy that will provide flexibility to support business decisions over the next five years, so we're looking at solutions that can grow with us. That’s what we’ve found in Google Cloud and its BigQuery data warehouse," says Karthik.

Karthik was confident that BigQuery could handle the growing number and complexity of stream, batch, and API-based datasets that News UK ingested, in addition to easily accommodating future growth. "We wanted a scalable solution for data storage and access, and based on our previous experience, BigQuery was the ideal choice,” says Karthik. “The serverless aspect is a huge win, enabling us to focus on adding value to the business, not worrying about our data warehouse’s backend scalability."

Karthik’s goal was to empower his data analysts looking to analyze data for real-time insights and easily run ML models and achieve AI initiatives, without the need for additional features or tools. His data analysts also wanted to be able to focus on running cutting-edge analytics instead of worrying about operational overheads. News UK’s previous data warehouse was unequipped to produce these analytics, and it faced performance challenges that inhibited agile decision-making. With BigQuery, Karthik recognized the opportunity to unlock faster business value while running on fully managed infrastructure and lowering costs.

To make sure its customer data was completely secure, News UK conducted an evaluation before transferring it to BigQuery. "During the process, we found that the security we needed in terms of access controls was built in, and that was reassuring," says Karthik. "We didn't need to invest in building additional internal security layers. We could control access at table and column level, and use BigQuery Data Access Audit logs."

“Everyone is more confident in providing business teams with new capabilities. It used to take a couple of months of planning to scale up and add new nodes to our Redshift data warehouse. Now with BigQuery, we can do it within an hour.”
—Karthik Ramani, Head of Data Platforms, News UK

Working with Google Cloud Consulting for a smooth migration

Karthik's team approached its Redshift to BigQuery migration in two phases. The first phase scaled up its Google Cloud data-processing infrastructure over six months to add an additional 200 pipelines. "We wanted business intelligence developers, who are not advanced programmers, to be able to create pipelines without having to write python scripts, so the tooling phase was really important for us," he explains. The second phase involved migrating 20 TB of data into BigQuery before the new data warehouse was tested out by the business teams and signed off.

Throughout the migration, Google Cloud Consulting guided the team through planning and optimizing for the best possible solution. "We first undertook a series of week-long workshops to share our requirements and ideas, which helped us to understand the options for solving our challenges," says Karthik. "The Google Cloud consultants showed us capabilities that we weren’t previously aware of, and working with product-specific teams helped us to decide which products were right for this stage in our journey. This saved us a lot of time that we would have spent looking for the answers ourselves.”

The whole project took about 15 months, end-to-end. News UK now uses Cloud Composer as its workflow orchestration service, alongside Cloud Storage and BigQuery for its batch processing, with Dataflow and Pub/Sub for its real-time data pipeline. It uses Compute Engine virtual machines for data cleansing processes, and App Engine for a data portal UI called NewsFlow, which helps its data engineering team to deploy code to the CI/CD systems.

Empowering data engineers to better support the business

Since making the move to BigQuery, Karthik's data platform team feels more empowered. "Everyone is more confident in providing business teams with new capabilities," says Karthik. "It used to take a couple of months of planning to scale up and add new nodes to our Redshift data warehouse. Now with BigQuery, we can do it within an hour.”

For the business teams, it's much quicker to get the data they need. "It only takes three weeks to ingest a new dataset, where it used to take three months," says Karthik. "And the performance of our data transformation pipelines is 40 times better."

It's also easier to access that data once it's ready, as News UK no longer has to rely on an outsourced administration team to enable access. "Instead of taking a week, it only takes an hour to enable access," says Karthik. "It's removed a number of bottlenecks, so the whole life cycle moves faster, and we can also handle three times as many data demands with the same team." And, because it has removed duplicate capabilities, it's able to make financial savings as well as adding new capabilities.

With greater access to larger and more complex datasets, News UK can provide its advertisers with much more accurate information about its readers. The next step? To translate that benefit for its readers through more personalized experiences. "Making use of our new capabilities with new data science and modeling products, we can improve our reader experience," says Karthik. "We also want to improve our data governance, and are looking at Data Catalog and Cloud Data Loss Prevention to help us do that. We’ve worked with Google Cloud Premier Tier Partner SoftServe to explore capabilities of the above services, which are crucial in terms of meeting our compliance requirements for handling customers’ data.”

With a reading experience that’s carefully curated for its readers, News UK is giving the newspaper experience a digital makeover for the digital age. "There’s fierce competition for the attention of our readers, but by leveraging Google Cloud, we can stand out with content they love," says Karthik. "We're excited to use our Google Cloud solutions to provide an even more personalized experience on our platforms in the future."