HPC End Users Confirm Mixed IO is the Hardest Problem to Solve Locally, and that Security and Data Sharing Complexity are the toughest problems for Multisite Collaboration
Addressing the mixed I/O demands on the storage infrastructure of local High Performance Computing (HPC) environments continues to be the biggest challenge for HPC storage architects as they look to solve their Big Data and storage needs in 2015 according to the results of a DataDirect Networks (DDN) High Performance Computing (HPC) Trends survey, released today.
Conducted by TechValidate, a leading customer insights firm, the survey polled a cross-section of 112 HPC end users representing hundreds of petabytes of storage investment, which included individuals responsible for HPC allocations comprising high availability storage and file systems across a variety of industries. Of those surveyed, 36 percent manage or use at least one petabyte of storage, and over a quarter manage or use 6 PBs or more.
According to the survey results, more than two-thirds (67%) of those polled agree that performance is the number one storage and Big Data challenge with nearly half the respondents citing mixed I/O performance as the biggest bottleneck. In comparison to last year's survey results, this represents a 13 percent increase indicating that performance related issues have surged in the face of increasing IO demand.
In addressing this challenge, three quarters of respondents believe that burst buffers are the most likely technology to push storage to the next level, with response rates significantly higher (79%) for sites with over 6PB of storage. This suggests that planning for exascale, accelerating time to discovery and extracting results from massive data sets is exhausting the capabilities of today's architectures, leading more than 25 percent of respondents to seek faster and more efficient alternatives for provisioning I/O and accelerating applications.
In terms of remote sites, while multi-site HPC collaboration may be on the rise, security and data sharing complexity remain the biggest impediments according to 75 percent of those surveyed, with the issue becoming even more serious (78%) among respondents using or managing more than 6PBs of storage.
User Insights Reveal that Storage Performance Continues to be Critical for HPC and Analytics Workflows
High Performance Computing (HPC) is a unique environment that places special demands on the storage infrastructure, creating a slew of storage and Big Data challenges for HPC users. Looking to 2015 more than half of the survey respondents cite mixed I/O performance the primary issue (51%), with managing rapid data growth and managing bandwidth performance tying for second place at almost 20% each.
The storage I/O challenge extends to a variety of workflows including analytics with nearly three quarters (74%) of those polled indicating their analytics systems are hitting a wall as they approach Big Data scale, suggesting that performance bottlenecks of MapReduce is no longer the main challenge for these platforms.
With storage performance a critical requirement especially at today's petascale levels, of those respondents with an opinion, over 80 percent agree that site-wide file systems like those at Los Alamos National Labs, The University of Florida's Interdisciplinary Center for Biotechnology Research (ICBR) and Indiana University continue to be a significant infrastructure trend in HPC environments, allowing HPC architects to either consolidate multiple computers on the same system and/or the flexibility to upgrade storage and servers as needed.
Finally, despite recent economic recovery and some increase in grant funding, HPC organizations are still under increasing funding scrutiny, where, according to nearly three quarters of the survey respondents, it continues to be difficult for HPC sites to obtain funding for compute infrastructure without demonstrating a clear return on investment (ROI).
Supporting Quotes
Molly Rector, Chief Marketing Officer, DDN: "The results of DDN's annual HPC Trends Survey reiterate what HPC end users continue to tell us – storage I/O is a strategic issue for HPC sites and becoming more critical every year. Users rely on DDN to deliver continued technology innovation which enables research and business organizations to accelerate business insights and in turn bottom line results, year-over-year. DDN is focused on conquering the I/O performance, content distribution and collaboration demands of the HPC industry faster, smarter, more efficiently and most cost effectively."
