Business of Film: Flying Files and Safe Storage
Media files are large, they’re valuable, and they’re getting bigger as 4K and newer technologies enter the picture. Where will they be stored and how will they get there?
By T. Hunter Byrd
Film production workflows are all about moving content or assets between production teams, and what you need varies by where it is in the workflow. The resolution of raw footage is not the same as what you use for work in process, and that isn’t the same as the final resolutions you deliver and archive.
Because this content moves from user to user in various locations, you want to optimize the resolution based on capabilities of the transport connection available. “You can use WAN accelerators, FTP, or dedicated lines between facilities,” explains Alex Grossman, VP, Media and Entertainment at Quantum Corp. San Jose, Calif., “but you may need to transcode to a lower resolution codec along the way. Smarter transfer technologies will transcode to a lower bit rate codec that’s not playable; they encrypt the files on the fly during transfer and then do the reverse at the destination. For even greater speed and security, parallel transfer technologies break the content into pieces and send them separately.”
Regardless of how you transfer, you want to avoid making unnecessary copies to limit the number of files that need to be stored and managed. But you don’t want to lose your high-resolution content, and, with today’s tight deadlines, you need to maintain operational efficiency. For that you need solutions that offer intelligent, policy-based storage and asset management.
“Quantum’s StorNext supports every major popular file transfer technology on the market today,” says Grossman. “To make file transfers seamless, we offer an API that integrates with applications that send or receive the content. This goes along with StorNext’s vast ecosystem of integrated workflow applications, from MAM (Media Asset Management) to production automation tools.”
To help manage where the content is stored, StorNext Storage Manager automatically migrates files to different storage media—disk, object storage, tape—according to policies you set up. “That way, the files you need the quickest access to stay on the highest-performance storage while those that haven’t been used recently get moved to less expensive media,” Grossman says. “We’re simplifying things even more with StorNext in the Cloud, which allows you to take your existing workflow into the cloud, including your applications for editing, production automation, transcoding and more, so that team members can collaborate easily and your content is stored securely and efficiently.”
How to fly files
Filmmakers have become accustomed to file-based workflows in post-production and, with digital capture now the norm, production processes rapidly are following suit. Moving large media files around within a facility is fairly straightforward, but things get challenging when it’s necessary to move content over long distances. While the tried-and-true method of shipping physical media hasn’t gone away, electronic transfer is increasingly practical and offers a number of benefits.
Margaret Craig, CEO of Signiant, Burlington, Mass., says, “The never-ending push to shorten production timelines is a key reason to consider electronic file transfer: it is generally the fastest way to get the content from A to B. And with the right tools, it can be far more cost-effective, reliable, and secure than physical media. Networks around the world keep getting better and faster, with great connectivity between major media centers and often surprisingly good bandwidth in remote locations. Against this backdrop, it is critical that filmmakers become familiar with software tools for fast, secure movement of their valuable content over public and private networks.”
Old technology, such as FTP servers, can be used to move media files, but a better answer can be found in specialized file transfer software that is tuned for moving large, high-value digital assets. These solutions move the files much faster than FTP and provide a high level of security, along with central control and workflow automation.
Craig comments, “In assessing file transfer solutions, filmmakers would be well-served by asking themselves a few questions such as, ‘Is the solution easy to use and administer?’ If not, adoption will be difficult and crews will default to old methods. ‘Is it secure?’ File transfer software must have full enterprise-class security features for studio work, and should provide full control and visibility of the content at all times. And perhaps most important, ‘Is it scalable?’ As the workload ebbs and flows, it’s important to be able to quickly ramp up or down.”
A Signiant offering of particular interest to film producers is Media Shuttle, which provides a fast, secure, simple means of sending a large file anywhere in the world. It is the only file transfer solution that is delivered as a hybrid SaaS offering. Users log on to a web interface provided by Signiant. The content itself is never turned over to Signiant or a third party—it remains under the customer’s control, either on-premises or in the customer’s cloud storage tenancy.
Clouds from both sides
“Cloud” is the buzzword of the 21st century—so far. But most filmmakers tend to think like Joni Mitchell sang, “I really don’t know clouds at all….” Cloud computing refers to both the applications delivered as services over the Internet and the hardware and systems software in the datacenters that provide those services. The services themselves have long been referred to as SaaS—Software as a Service. The datacenter hardware and software is what is called a Cloud.
The cloud is an outgrowth of several computing concepts that started more than 60 years ago with the “big iron” of IBM and its timeshare computing model. Distributed computing and the Internet changed the way we looked at data sharing and each added a new feature set to the concept. Today, the Internet allows computers—and computing devices such as digital cinema cameras—to connect across thousands of miles, virtually seamlessly and instantly.
The cloud is for storage, but there is more to it than just a warehouse in the sky, figuratively speaking. Patrick Macdonald-King, president of Prime Focus Technologies–North America, Los Angeles, puts the cloud in focus. “Why should a film producer study large file transfer and storage? Emerging, multiple file formats, increasing shooting ratio and the exponential risk of misplacing—not losing, mind you, misplacing—files call for extreme, organized, automated, monitored care of your content; security during transfer, acute awareness of content metadata after it is transferred. Note my use of ‘misplacing.’ I mean the consciously placing of assets in a deep structure (without metadata) mixed with the inability to find required shots despite a 100 searches and visual scanning.”
He adds, “Then there is the risk of losing footage; not being able to retrieve it in time; unreliable infrastructure; and many other issues that compound the need for a robust storage infrastructure, not just a portable LTO or bunk-head room of storage, for assurance of zero content loss; precision service levels of retrieving the content in agreed timelines; and always process controlled, which in fact is becoming a norm mandated by insurance requirements.”
Macdonald-King also notes, “The media and entertainment industry’s storage needs are at an interesting cusp—you have the likes of Amazon storage options, but they cannot really be used for operational purposes as they do not support frame-accurate transfers. Such storage is not meta-data aware, not-content aware. On the other hand, Prime Focus has designed for media assets thus meeting all these needs. Our focus has been on the downstream benefits to film producers rather than focusing on the benefits of using transfer/storage infrastructure per se.”
Continuing, Macdonald-King says, “The Prime Focus solution provides the connecting tissue between the studios and networks for seamless, creative collaboration; and all this on one platform, with one metadata bus and one end-to-end digital supply chain. You can come on-board at any stage to engage with your vendors and customers on the same platform.”
For media developers and producers, the cloud offers several advantages over local storage for the large files being generated in 4K and greater high-definition video. But every week we hear of hackers gaining access to databases—Sony’s being one that hits close to home for most filmmakers—leading to concerns of security and reliability. Perhaps the cloud shouldn’t be the first choice. Various file sharing systems are available that use local networked storage—servers owned or leased on premise or in co-located server farms.
Not too long ago, it was the norm for broadcasters to physically ship hard drives of the media assets they created. This method was conceived to address legacy business agreements that had been drafted before the advent of specialized network technologies to transport large media files, and the networks’ inability to handle such files due to insufficient bandwidth. An intern would take hard drives loaded with raw footage from remote filming locations or productions to the closest courier, where they would then be sent for review or processing to those with an invested stake in the project’s progress and direction.
Shane Guthrie, senior manager for global solutions architecture, Equinix, Redwood City, Calif., picks up the lesson. “This physical process created a logistics and expense nightmare, slowing down workflows by days, if not weeks. This one-way flow of information to solitary endpoints also made it very difficult—if not impossible—to respond in real-time to changes or requests by producers or financiers of the project. Once a project was finished, films and television spots would again be sent by physical courier to content delivery networks, which would then distribute the completed work in the same laborious, expensive manner.”
Figuring out a way for gigantic, copyrighted files to be transferred worldwide, more securely and faster than a courier service, while allowing for constructive dialogue between sender and receiver, was crucial. “For example, if a shoot was done in Tunisia, England and Guatemala, studios and broadcasters would sign long-term contracts with service providers in those nations to build out additional infrastructure to handle the bandwidth necessary for a particular production,” recalls Guthrie. “This process often took months, but it would ensure that large amounts of raw footage could be sent back to headquarters and, from there, to the various teams involved in production.
“To complete the transfer, production companies also would have to beef up the networks of the intended endpoints to cope with receiving a high volume of large assets, as well as sending edited or annotated versions back into the field. When production ended, studios and broadcasters often would be stuck with a long-term contract for lots of fiber and throughput that they might not ever use again. While building physical infrastructure for a singular purpose is nothing new in the movie business, having to continually pay for these high-speed, data-intensive connections after wrap was a new financial burden that few could handle.”
Thankfully, this initial fix brought to bear another trend—that of carrier networks fortifying their own infrastructure to handle the explosive amounts of data being created and transferred daily worldwide. As Guthrie says, “These network providers were co-locating in third-party, multi-tenant data centers to exchange traffic and connect directly to their customers, ensuring quality of service and security were not compromised by over-reliance on the public Internet. The huge influx of capital spent by wide-area network providers and Ethernet carriers to upgrade capacity and reach of their networks is a boon to the media and entertainment business.”
Jeff Herzog, product manager at EditShare, Boston, Mass., runs the numbers: “It seems as though we have come full circle. Back in the early days of HD editing, uncompressed HD was all the rage. At around 180 Mbps, you needed the fastest storage available at the time, and even then it seemed like capture and playback of one stream was almost on the edge of the capabilities of the equipment. Then along came extremely high-quality compressed codecs like Apple ProRes and Avid DNxHD, and uncompressed HD diminished in popularity.
“Now, 4K acquisition and editing is everywhere, data rates have jumped back up, and producers, editors and other content professionals must equip themselves to push around much bigger files once again,” Herzog continues. “When someone says they are working in 4K, it could be one of dozens of different scenarios, and the only way to effectively support the full range of 4K workflows is by investing in a scalable infrastructure that can be configured to meet any of these demands.”
EditShare’s Shared Storage is a scalable platform that can easily be configured to meet the varied demands of high bitrate workflows such as 4K. According to Herzog, “Some compressed 4K workflows can be enabled with a 10-Gigabit network infrastructure linking your workstations to your EditShare server, especially if you have just a couple editors working at this resolution and each of them is playing only one stream. For example, the data rate of 4K UltraHD at 30p in ProRes 4444 is 165 Mbps, easily supported with your EditShare server and workstation both connected via 10-Gigabit, including 10G-BaseT, which uses regular Cat 6 cable.”
However, if your workflow calls for multiple editors sharing compressed or uncompressed 4K footage or each working with multiple 4K streams, you likely will need more bandwidth between your storage server and your switch than a single 10-Gigabit connection can accommodate. “EditShare supports 4K editing over 40-Gigabit Ethernet, changing the game for large-file transfer, storage, capture, and playback,” claims Herzog. “For not much more than the price of a 10-Gigabit infrastructure, you get a single pipe that is four times the size, allowing you to much more easily support playback of uncompressed or RAW 4K files, where data rates can be up to 48MB/frame or a whopping 1,150 Mbps at 24 fps. This bigger pipe also allows you to quickly push these files around your facility, whether it is from SSDs of camera originals or to an archive.”
As 4K, 8K and beyond becomes a daily reality, large files will be the norm. Advanced compression algorithms will help, but the numerous editions of a shoot still will require flexible and scalable storage and fast file transfer options.