Dec 052014

Today’s blog post is a summary of several useful formulas for dimensioning video storage systems. Before you start reading it, it is very important to understand the connection between the Kilo, Mega, Giga and Tera prefix, the difference between Bit and Byte and difference between an indication of size (like MB) and an indication of speed (like MBps respectively MB/s). I created all the equation using the great Online LaTeX Equation Editor.

This post uses the following terminology:

  • A capital B stands for Byte
    Example: MB means Megabyte
  • A lower case b stands for Bit
    Example: Mb means Megabit
  • To indicate a transmission speed the “per second” postfix is appended
    Example: MBps means Megabytes per second

Now let’s start…

How much Space of my Volume should I use?

You should never use all the available space as this forces the filesystem to split new files into many small junks. These small junks lead to increased seek times and thereby slower transfer rates.

Tipp: Use quotas to limit the user visible size.

How much Storage do I need?

As described above you have to add a little extra space to avoid high seek times.

CodeCogsEqn (7)

How much Hours of Video can I Store on my Volume?

This is often used to find out if there is already enough storage capacity to handle a new project or if you have to add more.

CodeCogsEqn (8)

What Performance do I need?

The first formula calculates the Theoretical Minimum Performance Requirement in MBps. To be honest, it’s a pretty useless calculation as you need a lot more raw storage performance than this formula indicates. For examples it doesn’t take disk seek times or CPU limitations into account.

CodeCogsEqn (4)

For a more realistic estimate use the following one. However this is still just an estimate to get a rough overview. It is very hard to size a video storage correctly and I haven’t found the correct formula yet.

CodeCogsEqn (9)

Can I go with 1Gb Ethernet or do I need 10Gb Ethernet?

Many people think that they have to use 10Gb Ethernet if they want to edit videos using AFP, SMB or NFS. However in reality they often don’t need that much performance and can better spend the money on more storage capacity or better networking infrastructure.

CodeCogsEqn (6)

It’s fine to go with 1Gb Ethernet if the Theoretical Minimum Connection Speed is 850Mbps or less. Go with 10Gb Ethernet as long as you need less than 3000 Mbps otherwise please talk to the system manufacturer.

What Network Protocol should I use?

That’s an easy one: Always use SMB on Linux, OS X and Windows!

(Except if you use OS X in combination with FCPX libraries stored on your NAS than go with NFS)

Mar 012012

We always use QLogic Fibre Channel Switches for our SAN installation. These switches have some options to increase throughput and decrease spikes during large file transfers, namely I/O Stream Guard and Device Scan.

It’s quite important to set this options correct for a good overall performance of the SAN regardless of the used software (XSan, metaSAN, HyperFS, …).

Here’s what you have to use:

Initiators (your clients)

I/O Stream Guard: Enabled
Device Scan: Disabled

Target (your RAIDs)

I/O Stream Guard: Disable
Device Scan: Enabled

Apple also posted a KB article a few years ago about this topic.

Feb 272012

It was quite hard for me to find a good resource on how to calculate the IP header checksum.

The RFC 791 only says:

The checksum algorithm is:
The checksum field is the 16 bit one’s complement of the one’s complement sum of all 16 bit words in the header. For purposes of computing the checksum, the value of the checksum field is zero.

I used the implementation of Ken Christensen as a reference and was able to get a valid checksum. You can find my algorithm over at Google Code.

Feb 262012

Recently we switched over to VMware ESXi 5.0 for all our new Windows Server installations.

For my first installation I have chosen an HP ProLiant ML350 G6.
After looking up the specs and checking with VMware’s Compatibility Guide everything looked quite straight forward. After installing the machine at the client they started to complain about very very low system performance.

After doing some tests I found the problem at the I/O performance of the RAID 1. Googling around showed up a not widely known problem regarded a not installed RAID controller battery backup (BBWC). After getting the “missing” part and finishing the first loading cycle (which took about 2 hours) everything went back to normal!

Always get a BBWC for your P410i controllers when you want to use VMware ESXi