Welcome back my fellow 10Gb fanatics.

So you made it all the way to part 3 in our 10Gb Home Area Network Series.

As promised, we are going to overcome the obstacles and expense of networking more than

two systems using a custom built 10Gb switching solution.

If you missed part 1 or part 2 of our 10Gb Home Area Network Series, make sure you go

back and watch those videos first.

Part 3 builds off concepts explained in the first and second videos.

Follow the embedded links now to watch those videos first.

Otherwise, it's time to build a bad-ass 10Gb switch.

Here is a brief overview of what we are going to cover in this video.

Why build a 10Gb switch?

Number one, a custom built switch allows you to meet a specific requirement.

If you are only connecting three systems together, then why pay for an expensive 8 port vendor

switch?

Number two, you can achieve an excellent level of customization.

This solution employs an advanced feature set that you can leverage, if you choose to.

Lastly, for some it just comes down to cash flow.

A custom solution allows you to control your costs better.

Now, the part everyone has been waiting for.

How in the world do you build a custom 10Gb switch?

Well for starters, you have to build a bridge.

According to Wikipedia, a network switch is officially known as a MAC Bridge.

A switch is a multiport network bridge that uses hardware addresses to process and forward

data at the data link layer (layer 2) of the OSI model.

So to summarize, a switch is a bridge is a switch.

So, the plan is to combine multiple Network Interface Cards (NICs) into a reasonable platform,

load an optimized network operating system, and finally, tie multiple ports together using

interface bridging.

So those of you that guessed bridging, you were correct!

Gold star for you!

The end result is a customizable 10Gb switching, and routing, platform that allows for media

conversion.

In other words, you can throw together a dual port 10Gb SFP+ card, a quad port 1Gb copper

card, and a fiber card.

Bridge the ports, and you have just built a media converter.

Now, are you starting to see the possibilities?

There are some pretty interesting things you can do with this type of flexibility.

I'll cover a few scenarios in an upcoming section.

First things first!

Let's talk hardware.

You can do this with a PC or server.

Just like with anything else, your mileage will vary depending on what you use.

See the video description for my setup.

You will need a system with at least two PCIe x8 slots for a 10Gb - 4 Port switch.

Your switch will be limited by the number of PCIe slots that you have, and cards you

use.

Don't forget about your embedded network ports too.

When it comes to the hardware, focus on a motherboard with plenty of bus bandwidth,

maximum PCIe slots (x8 for 10Gb dual cards), and a decent multi-core processor to reduce

CPU loads and increase parallel bandwidth.

With the introduction of technologies like DMA, I/OAT, DCA, and NAPI, PC hardware is

achieving line rate network speeds formally only seen in datacenters on enterprise switches.

Software Defined Networking (SDN) is a prime example of this concept already in play.

For home users, you don't need a heavy-handed solution.

On the other end of the spectrum, aiming for maximum performance, go with a Xeon (E3 or

better) and I/OAT-enabled motherboard when planning for port density.

Now for the most critical component in this entire setup, the network operating system.

And the winner is, VyOS!

What is VyOS?

Why VyOS?

Okay, I'm getting there.

VyOS was originally developed by Brocade Communications and known as Vyatta.

In 2013 they stopped development on Vyatta Core.

A group of enthusiast developers forked the source code and created what is now VyOS.

"VyOS is a Linux-based routing solution built on the Debian Linux distribution, and currently

runs on x86 and x86-64 platforms."

VyOS is an incredibly light-weight, heavily customized, and optimized distro with the

sole purpose of providing Layer 2 and Layer 3 network functionality on a physical or virtual

platform.

Why did I choose VyOS over pfSense (as suggested by viewers)?

The answer is performance, among other things.

To be clear, I have only tested the aforementioned distros.

I'm sure there are other solutions available though.

I discuss further details about my pfSense testing in the summary.

I can personally vouch for VyOS as a routing/switching solution in a production environment.

I know others use it as an edge router in datacenters, and it is widely used in cloud

implementations like Amazon Web Services (AWS).

Awesome!

Now let's dig in with some digital grease, and get these gears moving.

Follow along as I narrate the BIOS Configuration, VyOS Install, Basic Setup, Interface Reorder,

System Setup, Bridge Creation, Bandwidth Testing, and Useful Utilities Demo.

First up, we are going to start by reviewing some important settings in the BIOS.

Check your CPU section and enable: Direct Cache Access (DCA).

Next, under Chipset/Northbridge check for IOAT and VT-d settings.

Enable IOAT for sure and try "No Snoop" and "Relaxed Ordering" if present.

In addition I enabled VT-d and Interrupt Remapping.

Make your way to the PCIe Configuration and look for Above 4G Decoding, Maximum Payload,

and Maximum Read Request.

Maximum Payload and Maximum Read Request are very important and you will see why later

in the video.

Just know that they should be set to the max value.

Save your BIOS settings, and have a bootable copy of VyOS ready.

Boot your system to the VyOS CD.

It's a live CD and allows you to run from boot or install an image to disk or USB.

The default username/password is vyos/vyos.

Run the "install image" command to kick off the install.

You can get context sensitive help at any point in the command line by typing question

mark.

Most of the install process is accepting the defaults.

The install is self-explanatory as you can see.

I installed to a disk drive, but you can install to USB as you only need about 2GB of space total.

Set your new VyOS user password and continue accepting defaults.

When the install completes, make sure to reboot your system without the CD.

Otherwise, any changes you make at this point will be lost.

When you boot into your newly installed image, you are in "Operational Mode".

Take a look at your network interfaces with the "show interfaces" command.

Interface name on the left, and link status on the right.

Capital "D" signifies nothing is connected to our switch at this point.

Plug a copper Cat5 cable into one of your ports and run "show interfaces" again.

Now you can see a lower-case "u" in the right column, signifying the port is up.

A better method is to "blink" the interface you plan to configure.

Run "show interface ethernet eth0 identify" to visually blink the interface you plan to configure.

At this point you should notice the command line is structural in nature.

And remember to use the question mark to better understand your options.

We've been working in Operational Mode.

Now we are going to enter what is known as Configuration Mode.

This allows you to modify your system configuration.

We are going to set the IP address of interface eth1 where we connected our cable.

Notice the slash 24 at the end of the IP address.

This is our subnet mask specified in CIDR notation.

Slash 24 is equivalent to 255.255.255.0.

Now set a description on the interface.

Remember to always label for easier troubleshooting.

Next allow a terminal program like PuTTy to connect to your new interface.

Run the command "compare" to see a list of queued commands ready for execution.

They do not take effect until you issue a "commit" command.

Following the commit, you need to save, so your changes are persistent across reboots.

Did you notice the plus sign next to the queued commands?

This means they are a new addition to your configuration, just as a minus sign would

mean they are being removed from configuration.

Exit Configure Mode and show interfaces to see if changes have taken effect.

In this optional section, I'm going to show how you can reorder your ports however you like.

VyOS may not place your ports in an order that makes sense.

So if you are particular like me, I'll show you how to customize it exactly how you want it.

I'm going to use the Linux ifconfig command to pull, and sort by the MAC address.

Essentially, we are going to reassign the MAC address to the interfaces in a different

order.

Fun fact: The first 3 octets of a MAC address identify the vendor of the hardware.

So try macvendors.com to lookup a NIC vendor.

Awesome troubleshooting technique by the way.

Once you sort by your MAC addresses, copy it out to an editor for reference.

In my list, the bottom 4 MACs belong to 4 embedded Intel NICs.

You can tell by the matching first 5 octets, and last octet order.

So I will make these interfaces eth0 through eth3.

The remaining interfaces, all 10Gb, will be eth4 through eth11 for a total of 12 ports

on my home brew switch.

Before you make this kind of change, it does not hurt to backup your main configuration

file at /config/config.boot.

Enter Configure Mode and use the "hw-id" command to reassign the physical port to the logical

ethernet interface in your preferred order.

Notice I'm using abbreviated versions of commands to minimize typing.

You only have to type enough of the command to make it distinct from other commands.

Here, I'm taking the lower 4 MAC IDs, and assigning them to eth0 through eth3.

Then I assign the top 8 MAC IDs to eth4 through eth11.

Since It's sorted by MAC ID, it will have the effect of correcting the ethernet interface order.

See, how the compare command shows a greater than symbol?

This means you are replacing a configuration item instead of adding or subtracting from

the configuration.

Finally, "commit", "save", "exit", and "reboot" for the re-mapping to take effect.

After rebooting, you may need to relocate your IP address so you can PuTTy back into

your switch.

Here, I show how to remove the configuration.

If you just tried to set the IP again it would add a secondary IP address on the interface.

On the system console, I will remove the IP from eth1 and add it back to eth0 where my

link status is "up".

Now I can PuTTy back in and everything looks good once again.

Connect to your switch with PuTTy, and enter Configure Mode.

If you type "show interfaces" in config mode it will show the current, or running, state

of your configuration.

This is completely different than displaying the contents of your config.boot file.

You can also show other subsections of your running config.

For example "show system", shows the system related aspects of your configuration.

Let's go through some basic configuration settings to get things running smoothly, starting

with your hostname.

Next, set your gateway address so you can communicate with the outside world.

Then finally DNS, and time zone.

Compare your changes, verify, commit and save.

Ping sun.com to test your connectivity to the outside world, and check that time is

syncing correctly with the command "ntpq -p".

You can also use the command "show ntp" from Operational Mode.

This completes the section on System Setup.

Time to build our bridge.

Use a console, or serial connection for this portion, as we need to reconfigure our management

IP again.

Login, show your interfaces, enter Configure Mode, and remove the IP address from eth0.

Notice this time around my delete command stops at "address".

VyOS allows you to remove configuration at various levels.

Now compare, verify, and commit.

Here is where we create our pseudo interface "br0", and assign it an IP address.

This interface represents our bridge, or collection of interfaces forwarding packets to each other

at layer 2.

Assign a meaningful description to your bridge interface.

Looking at our compare statement, notice how default options are assigned to the new bridge.

Commit and Save.

While in Configure Mode you can "show interfaces" by using the keyword "run".

You can now see the newly created interface br0 with assigned IP address, subnet mask

and description.

Let's identify a port where we can connect our 1Gb uplink.

Again, blink eth2 to visually identify.

Okay, plugged in and showing link up on eth2.

Since we have our bridge interface in place, we will assign ethernet ports 2 through 11

to the bridge.

This give us 10 bridged ports.

You could assign all your interfaces to the bridge, but it's not required and you can

change it later anyway.

Remember eth0 through eth3 represent my 1Gb copper ports, while eth4 through eth11 represent

my 10Gb Direct Attach Copper ports.

In the video I mistakenly started with eth3 when I should have started with eth2.

Next add a description to each interface.

For your 10Gb interfaces you want to set your MTU to 9000 as I discussed in part two of the series.

When finished assigning interfaces, setting descriptions, and adjusting MTU don't forget

to compare, review, commit, and save.

After I make some quick corrections to my descriptions

show interfaces, and marvel at your shinny new 10Gb switch that you lovingly crafted

by hand.

Isn't she a beauty?!?

I run the command "sudo ifconfig

-s" to verify my 10Gb interfaces are set to use jumbo frames.

At this point save your configuration.

Your 10Gb switch build is complete!

Now, lets see what this baby can do!

Let's start by checking our bandwidth to the bridge itself.

Open a few terminal windows to the new switch on Workstation #1.

The top left window is running a session of iPerf server and binding to the switch IP.

The lower left window is running the "top" command, so we can monitor CPU utilization,

context switching, and the iPerf process.

The right window is the jPerf client.

Workstation #1 is connected directly to the 10Gb bridge via Direct Attach Copper.

So at this point we are just going from Workstation #1 to the switch.

For this test we are getting around 9.5Gbs, which is excellent!

Everything looks great!

For the second bandwidth test example, Workstation #1 is connected to the VyOS switch, and Workstation

#2 is connected to the VyOS switch.

The top left window is running a utility (on switch) called "bmon" (bandwidth monitor)

to visualize our receive and transmit traffic, and top is running in the bottom left window

on Workstation #2.

Workstation #2 is using a slower dual core processor to demonstrate how it changes the

bandwidth picture.

Notice iPerf is pegged at 100% CPU with one stream of traffic from Workstation #1.

So lets try pushing two streams of traffic and see what happens.

You can see two streams of traffic allow us to hit our 10Gb mark.

So the VyOS switch is not our problem, it's the architecture of Workstation #2 that can't

handle a single stream of 10Gb traffic.

Another good test!

For the third test, consider what happens when heavy traffic has to traverse your system

bus.

In other words, inbound traffic on card 1 is forwarded to card3 outbound.

In my testing I discovered great speeds when I forwarded traffic across the same card (inbound

card 1 to outbound card 1).

This made sense as the traffic was local to the Network Interface Card.

But when I passed traffic across cards, my results were less than desirable and a bit

shaky to boot.

Digging in a bit further, I realized it was important to adjust BIOS settings to ensure

my PCIe bus was using maximum possible payload values.

This made all the difference in the world.

For this test I was using a much more capable Workstation #2 so I could avoid any processor

bottlenecks.

Upon making the PCIe adjustment in my BIOS (shown in the BIOS Settings Segment) things

were screaming fast once again.

You can see the chart wavers slightly, but in my testing without recording video, it

was a very consistent 10Gb per second.

Here is what the CPU looks like on Workstation #1.

Bear in mind, it's consuming proc cycles while I record video too.

And here is the 10Gb ethernet adapter in Workstation #1.

Perfect!

These were the results I was looking for.

As promised I threw in some power goodies for being so patient while I put the final

touches on the last video in the 3-part series of 10Gb Home Network.

Since these last 3 short sections are self-explanatory, kick back, and enjoy the music.

Here are some amazingly powerful commands I demonstrate.

And make sure you catch the summary, following the utility demo, as there are some really

important tips I include.

There are some other pretty amazing things you can do with this build.

We built a 10Gb switch, but you could build a 10Gb router, a 10Gb firewall, a 10Gb media

converter, or a 10Gb VPN.

Keep in mind it will only be as good as the hardware you use.

Make sure your network adapters receive good airflow from your case fans as they tend to

get hot.

Especially if you are stacking multiple cards next to each other.

On the topic of why I selected VyOS over pfSense, or some other distro, I selected VyOS because

it's a purpose-built tool with a focus on switching, routing, NAT, and firewall.

On the other hand pfSense is focused on boundary protection acting as a firewall.

You can bridge interfaces, but packets are still inspected and require a firewall rule,

unless you completely disable the firewall functionality on your pfSense system.

Even at that point, performance was still erratic during my testing.

I encourage you to test for yourself if pfSense is your preferred platform.

My intent is not to start a flame war or distro showdown.

My point is, that you always want to use the right tool for the right job, and testing

is how you get there.

Alright guys, I had an amazing time assembling this three part series, and interacting with everyone

throughout the process.

The feedback I have been receiving is overwhelmingly positive, and helped me structure this series

in a way that benefits you even more than I originally intended.

I can't thank you enough for all the thumbs-up and shares.

Keep the comments flowing, and subscribe if you enjoyed this series.

This is iTechStorm wishing you happy bandwidth trails.