This FilmmakerIQ Lesson is proudly sponsored by RØDE Microphones. Premium microphones

and audio accessories for studio, live and location recording.

Hi, John Hess from FilmmakerIQ.com - today we’ll get into the basics of digital data

storage from how we count it, how we name it, how we store it, and finally how we make

files from it.

Digital data at it’s heart is very simple: it’s either on or off - one or zero.

The binary number system that is central to modern computing can actually be traced back

to Gottfried Leibniz with his 'On the Art of Combination’ published in 1666. Leibniz

was interested in creating a pure mathematical language guided by perfect logic. Towards

his later life, the binary system came to adopt quasi-religious mysticism with one being

God and zero being the void.

It didn’t work out so good for Leibniz. but when people started training machines

to add - that’s when binary really grew some legs. By being only one of two options,

binary offered a level of precision that analog signals could never match.

The smallest bit of information in binary is... the bit. The bit can only express one

of two states - either zero or one. When we add a second bit and we now have four possible

states - zero zero, zero one, one zero, one one. If we add another bit we now have 8 possible

states. Each new bit doubles the number of possible states - to 16, 32, 64, 128 and 256.

And now we arrive at our first marker - the Byte which is 8 bits. Coined by IBM engineer

Werner Buchholz in July 1956 a byte historically has been designated any number of bits necessary

to express a character. This meant it was hardware dependent number with no definitive

standards. Early computers used four bit or six bit. When ASCII, a standard for encoding

English text and numbers came out in 1963, it was a seven bit system.

So how did we get 8 bits to the byte? During ASCII’s development, IBM introduced an eight-bit

Extended Binary Coded Decimal Interchange Code which was quite popular. Into the 70s

eight-bit microprocessors such as the Intel 8008, the direct predecessor of the 8080,

and the 8086, the precursor the x86 line of processors, popularized the 8 bit standard.

In those early days, computer RAM memory was labeled by the exact number of bytes they

contained - like a 4096, 8192, or 16384 - it was generally a power of two because that’s

what played nicely with processor’s architecture.

But that numbering system wasn’t going to work as memory got bigger and bigger.

In 1960 the International System of Units, abbreviated SI, formalized the metric system

and naming practices for units. Although there is no SI unit for memory, computer manufacturers

began to borrow prefixes like Kilo, Mega, and Giga to describe computer memory and hard

drive space. But there was just one problem:

Some people used the letter K or kilobyte to refer to 1024 bytes - that’s 2 to the

10th power. But SI unit prefix of kilo is for 1000 units, not 1024. Computer memory

manufacturers who were tied into the 8 bit architecture of the CPU just started using

capital K for Kilo as a shorthand for 1024 bytes. They would later use M for megabytes

as 1024 to the second power and G for gigabytes as 1024 to the third power.

But hard drive manufacturers took a different tack. There was rule saying the size of a

hard drive that said it had to be a power of two. In fact the very first commercial

hard drive, the IBM 350, first shipped in June 1956 and had 50 physical disk "platters"

containing a whopping 3.75 MB.

So to avoid the extra numbers that binary introduced, hard drives manufactures used

the SI unit prefix in standard decimal. One K wasn’t 1024 bytes but exactly 1000. This

dual definition between 1024 or 1000 became common practice in the 70s which is why even

today when you plug in your brand new 300 gigabyte hard drive your computer will only

show it as 279.4 GB. It’s not that they’re cheating you, it’s that operating systems

like Windows count in multiples of 1024 while the manufacturers count by the 1000s.

In 1998, the IEC, a governing board which sets standards on electronics, created a new

prefixes to try to clear up the confusion. Basically you take the first two letters of

the SI unit and adds bi resulting sizes like kibibyte, mebibyte, gibibyte and tebibyte.

A kilobyte would remain its SI decimal version but a kibibyte adhere to the binary version.

So we go from kilobytes at 1000 bytes to megabytes at 1 million bytes to gigabytes at one billion

and terabytes at 1 trillion bytes. What’s after that?

Well to help get a sense of size, let’s compare these data sizes to an regular single

layer DVD which holds about 4.7 gigabytes.

If we wanted to burn one terabyte onto DVDs, we would need 213 DVDs. If we stack them up

it would be a just shy of 1 foot high. For comparison - if we were shooting uncompressed

4K raw, this would only give us 34 minutes of shooting time.

Let’s use a very conservative 10:1 shooting ratio - that is for every finished minute

of film, there’s 10 minutes of raw footage - a two hour narrative would eat through almost

35 terabytes. In that case our stack of DVDs would be almost 3 stories high. A most realistic

30:1 shooting ratio would get that stack up to 8 stories.

You can see why only major productions shoot in uncompressed RAW, but even then it’s

a lot of data to wrangle. Not impossible as it isn’t stored on DVDs but there’s a

lot of data.

What’s after terabyte?

The Petabyte - one thousand terabytes.

If we took every single feature film listed on IMDB and compress each one to fit exactly

on a single 4.7 gig DVD we would have a little over 1.5 Petabytes of information and a stack

of DVDs taller than the Petronas Towers of Kuala Lumpur.

But social media is pumping out even more data. One full year of tweets weighs in at

4 petabytes. On an average day in 2008, Google processes 20 Petabytes of information. And

according to a stock report in 2013, Facebook stores over 100 petabytes of status updates,

photos and video. At almost 84,000 feet, Our stack of DVDs is now about the cruising altitude

of the SR-71 Blackbird - at the edge of the stratosphere.

The next step up is the Exabyte - 1000 petabytes. To create one Exabyte our stack climbs up

to 159 miles, the same altitude that chimpanzee Ham reached in an early spaceflight test on

board Mercury-Redstone 2 in January of 1961.

A thousand Exabytes is a Zettabyte that’s one sextillion bytes. The size of the entire

web has been put as 4 Zettabytes as of 2013 - to put that all on DVD would require a stack

that’s three times the distance from the Earth to the moon.

Mark Liberman calculated that if we digitized every single word spoken as 16kHz 16-bit audio,

we would need 42 zettabytes - taking us now a stack of 6 million miles - the view of earth

would look something like this taken from the Juno Spacecraft en route to Jupiter.

And lastly we reach our biggest named data size so far - the Yottabyte. one septillion

bytes: One million million million kilobytes. If we were to store that on DVDs our stack

would reach 158 million miles into space - which is more than the average distance from here

to Mars.

Even if we could get a quantity discount of say 10 cents per DVD which is a real bargain,

this tower to Mars would cost over $21 trillion dollars and weigh about a third of the moon’s

mass.

In our little thought experiment we used DVDs to store the data. DVDs, along with CDs and

Blurays are optical mediums. That is they bounce light, in the form of a laser, off

the surface of the disc. If there is pit in the surface, the reader sees it as a 0 or

off. A land and the reader sees it as a 1.

Optical media has been a great way of distributing media from music to movies but it’s for

the most part a write once deal. For storing data that we can work with like movie assets,

we’ll need something more malleable.

Traditionally this has come in the form of the spinning hard drive disk - one or more

platters coated with a magnetic material which can be written and read with a magnetic head.

This magnetic head looks at the the polarity of the material on the disk. If the polarity

of a section remains constant, that bit is read as a zero. If there is a switch in polarity,

a small voltage is created within the magnetic head as it sweeps over the surface. This spike

in voltage, whether it is positive or negative, is read as a one.

The newer type of memory storage is the Solid State Drive based on Flash memory. First introduced

in 1984 by Toshiba - flash memory is very similar to design of a MOSFET transistor with

the addition of what’s called a floating gate. You can think of a transistors as a

switch - when we apply a positive voltage to the gate the electrical field opens up

a channel which allows current to flow between the source and drain. A flash memory cell

adds this floating gate between the control gate and the semiconductor. This floating

gate is isolated with non conductive oxide which means once we put a negative charge

on it, it should hold it indefinitely.

If this floating gate has no negative charge, the transistor will switch on when a certain

positive voltage is applied to the control gate. If the gate has a negative charge - it

will cancel out some of the charge from the control gate which means we have to run a

higher voltage through the control gate in order to open the switch.

So now in order to read this memory cell, we run a voltage that’s somewhere in between.

If the switch opens, we have no electrons stored in that floating gate and therefor

we have a one. If the switch is closed, that means there are electrons in the floating

gate canceling out the current from our gate - and we have a zero.

That’s how you read a flash memory cell. But how do we write to it? There’s two ways

- the first is through quantum tunneling. By applying a large voltage to the gate we

can actually get electrons to quantum tunnel through the oxide into the floating gate.

The other way is something called Hot Electron Injection which again uses high voltage to

get the kenetic energy of the electrons to power through the oxide substrate.

I’ll spare you and myself the details but both these techniques require higher voltages

and oxide layer separating the floating gate eventually becomes damaged with all the electrons

traveling through it. For this reason flash memory can only be written so many times before

it fails.

But the beauty of flash memory is it can be made incredibly small. Remember that crazy

tower to Mars of DVDs? If we instead created a Yottabyte using 200 GB microSDXC cards (the

most compact data storage medium available as of this video) we would only need a pile

of disks about one third the size of the Great Pyramid of Giza.

When cameras started going digital, the solid state drives were the perfect replacement

as we transitioned out of tape. There were many different formats from the Panasonic

P2 to Sony’s Memory stick and SxS system. Some cameras even use computer SSD hard drives

but I want to focus on two particular types of Flash memory you’ll see in professional

and consumer cameras.

The first is the CF card or Compact Flash card. First manufactured by SanDisk in 1994,

the CF card is still widely used in photography and video equipment. It’s a very robust

card although you do have to be a little careful when pushing it into a reader or camera as

the contact pins do bend easily. CF cards read speeds are either written as Megabytes

per second or as a value followed by an x. The “X” is a base of 150 Kilobytes per

second write speed. So a 200X means it’s capable of being reading at 30 megabytes per second.

A major factor in the read and write speed is the communication protocol either: PIO

(Programmed Input/Output) mode and UDMA (Ultra Direct Memory Access) mode. PIO is for industrial

use. For video and photography you want UDMA. There are several modes - so far 0 to 7. With

the UDMA 7 supporting up to 167 MB/sec. But that’s only supporting, the actual speed

may be a lot lower than that.

Very recently a variation of the CF card - the CFast card came on the market. These cards

use a Serial ATA bus rather than a Parallel ATA Bus that the regular CF card uses. This

enables speeds up to 600 MB/sec and are being used in some high data stream cameras especially

those shooting in RAW..

The other popular format I want to talk about is the Secure Digital Card or SD card. These

cards come in four families: Standard-Capacity (SDSC), the High-Capacity (SDHC), the eXtended-Capacity

(SDXC), and the Secure Digital In Out which is really more of an interface. SD cards are

available in three size forms: Standard, Mini and Micro.

Unlike CF cards which can be a little confusing about their actually transfer speeds, SD cards

have class ratings that guarantee the minimum read and write speeds. For most HD video applications

you’ll need at least a class 10 or a UHS Speed Class 1 card which guarantees 10 MB/sec.

For 4K look for the little U symbol UHS Speed Class 3 which guarantees 30 MB/sec

Of course, these speed recommendations are only suggestions for compressed formats - your

camera may have specific requirements so it’s worth consulting manual.

There’s just one final topic I want to briefly cover in this overview of storage. We’ve

been talking about storing all those ones and zeros but when you have a long string

of them - how do you tell where the data starts and ends? How do we delineate one file from

another?

That’s where file systems come into play. There are many different kinds of file systems

and they can differ in structure and logic, properties of speed, flexibility, security,

size and more. There are file systems for optical disks, RAM, tape disks, you name it.

We could go pretty deep into that rabbit hole but for this discussion let’s only focus

on the disk file systems you’ll likely run into and a little bit about them.

If you’re using an Apple product - your hard drive will be utilizing the Apple Proprietary

HFS+ file system. Windows systems will be utilizing Microsoft proprietary NTFS. Unfortunately

these two file systems don’t play nicely with each other. On Windows you can just plug

in a Mac Drive and expect it to be readable. Luckily there are software options that essentially

translate the systems and allow one filesystem to read and write to the other.

Flash media will most often be formated in a File Allocation Table (FAT) system such

as FAT32. A relatively old file system, the maximum possible size for a file on a FAT32

file system is 4 GB minus 1 byte. Cameras that record onto cards formatted with FAT32

will split up large files either at 2 gigs or 4 gigs in order to stay under the 4 gig limit.

exFAT, which debuted in 2006 from Microsoft does away with this file limitation and is

the default file system for SDXC cards larger than 32 gigabytes. Windows and Mac Systems

can both read and write to flash memory that is formated in FAT32 or exFAT.

We are producing data at an astounding rate. How that data is stored is a challenge for

today’s computer scientists and engineers. If anything they’ve come up with some amazing

technologies. Data is the lifeblood of the digital filmmaker, it’s key that we protect

it, back it up, and when we’re finished, archive it which is quickly becoming it’s

own challenge all by itself. But first you have to get out there and make something great.

I’m John Hess and I’ll see you at FilmmakerIQ.com