Big Data Progress Requires Better Development Tools

The big data community needs better app development tools, no more Hadoop whizzes, says Continuuity’s CEO.

16 Top Big Data Analytics Platforms

16 Top Big Data Analytics Platforms

(Click image for larger view and slideshow.)

“Big data for the loads” is a well-liked rallying cry among those that see the complexity of Hadoop and related tools as an impediment to an information-driven nirvana. a contemporary Wikibon survey of giant data vendors and repair providers shows that companies’ technical staff often lack the educational to cope the notoriously complex Hadoop effectively.

But in preference to training more Hadoop experts, enterprises should specialize in convalescing tools to aid their developers build big data apps, says Jonathan Gray, CEO and cofounder of Continuuity, a two-year-old startup with an app development platform designed specifically for Hadoop. The Palo Alto, Calif., company has 30 employees, two-thirds of whom are engineers.

“We’re a Hadoop-focused company. What makes us different from everyone else is that we’re a developer-focused company,” Gray told InformationWeek in a phone interview. “We’ve built middleware, or an application server, on top of Hadoop.”

Before cofounding Continuuity with Nitin Motgi (formerly of Yahoo) in late 2011, Gray was a software engineer at Facebook, where he was an early adopter of Hadoop who steered the company’s HBase engineering efforts. The pair’s mission was to make it easier for developers to construct and manage Hadoop apps.

[What does it take to land a Hadoop gig? Read Hadoop Jobs: 6 Recruiter Tips.]

As Gray sees it, loads of businesses today — most notably, telecom, financial services, retail, and insurance firms — are swimming in data, but they are not sure the way to benefit from it. These firms are Hadoop’s early adopters and are those which could benefit most from big data apps. “That’s where we slot in. Enabling some bank with 10,000 developers — but only three Hadoop developers — to get those 10,000 people to construct on Hadoop.”

This legion of developers doesn’t need vast expertise in Hadoop, nor do they want significant retraining to become full-fledged data scientists. Rather, it needs a strong toolkit for developing big data apps — a market that Continuuity hopes to take advantage of. “We separate the information scientist from the developer, and people are generally two different people. There’s been a bit of an excessive amount of emphasis at the science component of this equation and never enough at the developer side of it.”

Emerging hardware categories consisting of wearables, connected cars, and at last smart homes show great promise for data-driven apps. Telcos are in an especially strong position to learn from the big volumes of user data they amass, Gray said. “They’ve got location data, demographic data, web browsing history. They’ve tons and plenty of data. And the interesting thing is, some of these companies are sitting on all this knowledge and essentially saying, ‘How do I turn this pile of bits right into a pile of greenbacks?'”

The big data space is crowded, obviously, with various venture capital pouring into Hadoop companies. “I’d say we’re differentiated in a single particular way: Our emphasis at the developer and on what we call big data applications,” Gray said. “In the event you have a look at what a few of the traditional vendors are offering, and in case you also examine what the Cloudera and Hortonworks of the realm are offering, they’re selling a knowledge warehouse. They’re selling a BI tool for analytics.”

BI tools are valuable, after all , but there’s less focus at the present time on services and products built on top of knowledge. “It’s really all about applications, and lots of the market isn’t inquisitive about that. That presents a challenge because we’re prior to the market slightly.”

Still, the focal point on big data apps appears promising. “We’re looking past the info scientist on the next person in line, and that is the developer,” Gray said. “And there is a huge problem there. There are thousands of developers obtainable, but they do not have the ideal skills, and the infrastructure [of Hadoop] is just too hard for them.”

Engage with Oracle president Mark Hurd, NFL CIO Michelle McKenna-Doyle, General Motors CIO Randy Mott, Box founder Aaron Levie, UPMC CIO Dan Drawbaugh, GE Power CIO Jim Fowler, and other leaders of the Digital Business movement on the InformationWeek Conference and Elite 100 Awards Ceremony, to be held along side Interop in Las Vegas, March 31 to April 1, 2014. See the total agenda here.

Jeff Bertolucci is a technology journalist in La who writes mostly for Kiplinger’s Personal Finance, The Saturday Evening Post, and InformationWeek. View Full Bio

More Insights