The Weather Company is moving to a NoSQL-powered platform to collect some 20 terabytes of weather data per day. What is the biggest challenge?
When this winter’s ice storms, artic deep freezes, and nasty nor’easters hit, it provided a great time to hunker down and get some inside projects done. Weather Company CIO Bryson Koehler was engaged on an important one: consolidating 13 datacenters all the way down to four, relying extensively on public cloud providers, and moving to a NoSQL-powered big-data platform.
When we last spoke to Koehler his company was preparing to head its Weather Underground business onto the brand new big-data platform, which runs the Riak database on Amazon public cloud computing resources, with backup resources at the Google Compute Cloud. Next up, plans called for the flagship Weather Channel to transport to that very same platform within a question of weeks. Koehler’s team has learned some key lessons along the manner, particularly concerning the challenge of predicting costs when using external cloud services.
Despite horrible winter weather that kept everybody on the Weather Company greater than somewhat busy, its SUN (Storage Utility Network) project is on course, in step with Koehler. SUN captures some 2.25 billion (with a “b”) weather data points 15 times every hour, up from 2.2 million (with an “m”) data points four times per hour at the company’s legacy on-premises platform. All that new data — some 20 terabytes per day — supports more accurate weather prediction around the world.
[Need to learn more on Koehler’s big-data plans? Hear him on the InformationWeek Conference, March 31-April 1.]
The Weather Company expects SUN to aid it consolidate 13 datacenters all the way down to four. The four remaining datacenters may be focused mainly at the company’s broadcast infrastructure, which can not be moved to the cloud. The datacenter count is already all the way down to eight, and it’ll visit seven by April. There were challenges with the hot platform, Koehler admits, with predictable costs being on the top of the list.
“We need to be sure we engineer [the system] so we understand the precise cost per transaction,” Koehler explains. By year’s end the corporate expects to address greater than 15 billion transactions per day at the platform, “so every 100th of a penny starts so as to add up.” Those transactions are mostly web- and mobile-app service calls against the company’s hundreds of APIs.
The cost levers include choices as to which sort of public cloud service the corporate uses for a selected computing task, how much data it caches, and, within the vein of huge-data analysis, how frequently it refreshes data and triggers new forecasts. SUN now offers a rich trove of information whatever where on this planet forecasts are needed, however the Weather Company must decide how often to update the info and faucet computing power to generate new forecasts. In a stable, highly predictable climate period in a city like Phoenix, to illustrate, the demands are quite different than they’re when, say, a chilly snap is hitting the Midwest or a nor’easter is bearing down on tens of million of folks from Washington, D.C., to Boston.
The Weather Company’s story is on the cross hairs of giant data and cloud computing, but Koehler — who admits he likes being edgy with technology and “turning it as much as 11” — says the challenges are usually in regards to the fundamentals of networking and ensuring that failovers occur seamlessly.
“We’re still at the learning curve on the right way to best tune the system, how we monitor, and the way we respond when things get it wrong.”
Engage with Oracle president Mark Hurd, NFL CIO Michelle McKenna-Doyle, General Motors CIO Randy Mott, Box founder Aaron Levie, UPMC CIO Dan Drawbaugh, GE Power CIO Jim Fowler, and other leaders of the Digital Business movement on the InformationWeek Conference and Elite 100 Awards Ceremony, to be held along with Interop in Las Vegas, March 31 to April 1, 2014. See the complete agenda here.
Doug Henschen is Executive Editor of InformationWeek, where he covers the intersection of enterprise applications with information management, business intelligence, big data and analytics. He previously served as editor in chief of Intelligent Enterprise, editor in chief of … View Full Bio
More Insights