NetApp IT Perspective: Robust File Upload Solution using NetApp StorageGRID
Murthy Patnaik
As an IT architect who works on big data and cloud platform solutions, one common challenge is uploading files, especially files larger than 1TB. The problem becomes exasperated when there is no browser plug-in for a reliable and secure upload.
We solved this problem inside NetApp IT by implementing a highly scalable, reliable, and secure upload solution using StorageGRID®, a software-defined, object-based storage solution from NetApp. Because StorageGRID leverages S3, we were able to secure and upload over https with pre-assigned URLS for added security. There were no client-side plug-ins as with the UDP transport layer communication protocol. As a TCP-based solution, the file uploads were fast and reliable. Our solution supports up to 2 TB with multi-part upload features including pause, resume, and cancel features. It was fast and easy to implement with well known technology stacks like S3 SDK, Angular, and Node.
Multipart uploading
The best way to quickly upload large objects reliably is by splitting files into smaller 100MG segments using multipart uploading. Here’s some sample code to show how it was done.
Also, I recommend you store the credentials locally in ~/.aws/credentials or in a properties file—and never in code.
| [sg-us-east-1] aws_access_key_id = PPIF2PCJK7KXXXXXXXXX aws_secret_access_key = ZM8L9BIRkK5UTjmLm6NXxxMXXXxxXxxxxXXXxXxx |
How it works
As shown in the process diagram below, a user selects his nearest geographic location from a dropdown menu. The location choices are North America (NA), Europe, Middle East & Africa (EMEA), or Asia Pacific (APAC). For NA and EMEA, the file will be stored in the location’s corresponding StorageGRID. For APAC, the file will be stored in AWS, then later synced between AWS US-East region and StorageGRID. Once the file is in StorageGRID, it will be routed accordingly for core processing or data processing.
The NetApp-on-NetApp blogs feature advice from subject matter experts from NetApp IT who share their real experiences using NetApp’s industry-leading data management solutions to support business goals. Visit www.NetAppIT.com to learn more.
Murthy Patnaik
Murthy Patnaik is an IT Architect at NetApp and his team build diagnostic tools for NetApp products. His focus is on technology evaluations and building working architectures with Big Data and Cloud platforms. Murthy has 17 years of IT experience.
