10 Gigabit Networking Between Home Server and Computer

2020-01-13T00:00:00-07:00

In my other life, I film a lot of videos on fitness.

After a recent lecture a few friends and I did for charity, I found out… I’m out of storage space!

I built a server for my home to not only store the videos I film, but also as a media server to store and watch some of the bluray and DVD content I’ve purchased over the years.

Problem is I have about 5TB of my own storage. And I have a few computers lying around that can help with the job of transcoding my ripped bluray and DVD media.

But if I do that on multiple devices, I’ll still have to transfer them back to the server.

So I decided to set up 10 gigabit ethernet between my main computer and my server.

Bits vs Bytes Networking often describes speed in bits. In our daily computing, we usually use bytes. One byte is comprised of eight bits.

Ex 1) 1 gigabit networking means 125 megabytes per second Ex 2) 10 gigabit networking means 1.25 gigabytes per second

Inventory - what you’ll need to get this going

I will outline what I bought exactly at the end of this section, but here’s an overview so you can pick and choose what you need.

At least two computers
Two networking interfaces
A 10 gigabit switch
The appropriate cables (one for each computer)

Network card

My server is in a huge tower (Fractal Design Define XL R2) with a huge motherboard (Supermicro X9DRH-7F). There is a variant of this motherboard with onboard 10 gigabit ethernet, but the cost was double at the time of purchase, so I opted for this cheaper one.

To get 10 gigabit networking, I will need a 10 gigabit network adapter. I can use one of my six PCIe 3.0 x8 slots, with a max transfer rate of about 8GBps, is plenty fast enough to take a 10 gigabit network adapter card. There could even be two or four 10GbE interfaces on that card and it would still work, since 10 Gb = 1.25 GB and 40 Gb = 5 GB.

Now there are two major types of connectors used for 10 gigabit ethernet: the standard-looking RJ45, and the intimidating SFP+.

Connector type

There are four major methods for home use

Direct Attach Copper (DAC) cable – cheap but only works for short distances
LC connector for multi-mode fiber optic cable – better for longer distances across the home
LC connector for single-mode fiber optics – good for REALLY long distances, but doesn’t seem to make sense for home use
RJ45 – only really makes sense if you already have at least Cat 6a ethernet cables

I found this video from Craft Computing on YouTube very helpful here. More on this when we talk about the 10 gigabit switch in a few sections.

My server is sitting right underneath my work desk, so I didn’t need long cables. I was going to pick an RJ45 connector until I discovered…

Thunderbolt 3 to 10 Gigabit Adapter

It would have been easy to just buy two of those network cards above and call it day, but my main computer is actually a Lenovo X1 Extreme Gen 1 laptop. I can’t just plug in a traditional PCIe network card.

This laptop is very capable (read: expensive) and comes with Thunderbolt 3, which is EVEN FASTER than 10 GbE, running 40 Gbps through the proper cables. But I need an adapter to get the 10 GbE into the TB 3 port on my laptop.

QNAP makes just such an adapter – one for SFP+ connections, and one for RJ45 connections. The SFP+ adapter was cheaper, so I chose that.

10 Gigabit Switch

I had intended to just plug the two computers together and call it a day, but I decided a much more elegant solution would be to plug each machine into a switch and let the switch facilitate the connections.

The aforementioned YouTube video from Craft Computing detailed a super affordable product: the Mikrotik CRS305-1G-4S+IN. It comes with a power supply, is nearly silent when running, and has one gigabit port to hook up to the main router (if desired) and four 10 gigabit ports to use as a switch.

Summary Purchase List

This is assuming your have a laptop with Thunderbolt 3 and a tower server with open PCIe lanes.

Mikrotik CRS305-1G-4S+IN - 10 gigabit switch
Mellanox MNPA19-XTR - 10 gigabit SFP+ network card for server (2-port and 4-port variants)
QNAP QNA-T310G1S - 10 gigabit to Thunderbolt 3 network adapter for laptop
(2x) DAC cable

Getting It Working - NOT quite plug and play

One thing I need to make perfectly clear: I am not a networking guru. I just like to tinker around and learn new stuff. So when that Craft Computing video said this solution was “plug and play”, it left me a little confused when I plugged everything in and it wasn’t working.

I just needed to realize how my computers were connecting.

Samba Shares

When I first got my server running, I set up a Samba SMB share to my main storage folder so that I had a new place to move my old files.

That worked pretty quickly. I was happy with that.

So I plugged in all my 10 gigabit hardware and tried transferring files…

I was able to transfer files, but only at a dismal 40-50 MBps. That was not even saturating the write speed on my HDD, which I expected to be at MINIMUM 80 MBps.

The problem was that after setting up my 10 gigabit connections, I kept using that plain ol’ 1 gigabit connection I had already been using. Notice what is highlighted on the left side of the following two screenshots.

Both of those network locations appear the same. When I click one of them in the sidebar on my Windows 10 laptop, it shows the same root folders. Navigating through shows the same folders as well.

The difference is that when connecting, I typed in a different IP address.

Setting Up Windows Network Share

First, we need to find addresses. The Mikrotik CRS305-1G-4S+IN comes configured with an IP of 192.168.88.1. I decided to move my Mellanox 10 GbE server card and my QNAP QNA-T310G1S 10 GbE laptop adapter to the 192.168.88.xxx subnet.

Configure IP on Windows Adapter

Right click your networking icon in your Windows task tray and select “Open Network & Internet settings”.

Select “Change Adapter Options”.

Right-click the 10 gigabit network adapter and select “Properties”.

Make sure “Internet Protocol Version 4 (TCP/IPv4)” is enabled and double click it.

Configure it manually to use the same sub-network as your switch. If you bought the Mikrotik one and didn’t change it, the subnet is 192.168.88.xxx. You just have to fill in the last number to give this network adapter it’s own location.

Since I’m not connecting to the internet, I just left the default gateway and DNS server addresses blank because I don’t need them. My Internet traffic is routed through my Gigabit ethernet or my WiFI adapter.

Accept the changes and move to your server.

Configure IP on Ubuntu

Go to network settings.

Click the settings gear icon for your 10 gigabit network card.

In IPv4, add an IP address on the same sub-network you selected earlier. If you went with the default for the Mikrotik switch, then the first three numbers have to be 192.168.88.

Set up a Samba Share on Linux

Now we need to have a folder that we want to share. In my case, I have four 10 TB HDDs that I have mirrored and pooled together. So two serve as a RAID 1 redundant mirror for the other two, which are pooled together to be displayed on my file browser as one folder. It’s pretty neat and ensures I’ll always have my stuff even if a drive fails.

I have mounted this pooled virtual drive in my /mnt folder. The easiest and most visual way to share this folder on my local network is to navigate to it in my Ubuntu Desktop file browser. The default file browser is “Nautilus”.

Right-click the folder and select “Local Network Share”.

Configure your sharing settings. I kept the name of the folder and allowed more access to it so that people on my local network dont need my help getting logged into the folder.

Then hit “Create Share”.

If you run into an error that says Nautilus needs to add some permissions to your folder in order to share it, try to add them automatically (if that is indeed what you want to do).

If you’re hit with another error that says Could not change the permissions of folder, then we need superuser privileges to get things straightened out.

To confirm this is actually the problem, right-click on your desired share folder and select “Properties” then “Permissions”. Here’s a screenshot from another folder of mine that I want to share.

This is showing us that the folder is owned by the user “root”. Since I am logged in as “lance”, I cannot change root’s folder. We’ll need to go to the terminal and tell it “I am root”.

Hit Ctrl + Alt + T to open up the Linux Terminal.

We need superuser privileges to change the permissions on this folder. And we need to know the path of the folder. Mine is /mnt/Intel660P. First we’ll change the owner of the directory and all of its contents.

$ sudo chown --recursive username:username /path/to/folder

sudo gives us superuser privileges (telling the terminal “I am root”)

chown means “change the owner”

--recursive means “do this on all the sub-directories and contained files”

username:username represents the user and the group that owns this folder. Each user has a group that is the same as their username, and that’s what I want to set my folder to.

/path/to/folder designates the file or folder we want to change

Putting it all together, here is what I wrote to change the owner of my folder

$ sudo chown --recursive lance:lance /mnt/Intel660P

And we can check to see if it worked by refreshing our Nautilus file viewing and checking the Properties > Permissions again.

Nice! Now, I need others to be able to access the folder, so we’ll need to change the permissions of this folder and the enclosed files.

Easiest way on Ubuntu desktop is to change the “access” for “group” and for “others”.

Then click on “Change Permissions for Enclosed Files…” and change to the settings you want.

Check out the chmod command if you want to do it in the terminal.

You can get a lot fancier with permissions if you set up samba shares. Click here for the basics of setting up samba shares and click here to get a more advanced samba configuration.

Create New Network Share on Windows

NOW! This step was my big problem. I already had a folder on my Linux server marked as a share. And I already had a link to that share set up on my Windows computer. This gave my Windows computer access to the storage on my Linux server.

The problem was that this link was made BEFORE installing the 10 gigabit network cards. So it was using plain ol’ gigabit (which is honestly fine for my home use, but doesn’t seem as cool so I “needed” to upgrade).

Let’s create a new network share on Windows that now uses this new connection.

On your Windows computer, hit WIN + E to open up Windows Explorer.

Right-click “This PC” in the left sidebar and select “Add a Network Location”. Note: you can also select “Map network drive…” if you want it to be stored as a letter.

This will open the Add Network Location Wizard.

Hit Next to continue.

Select “Choose a custom network location” and select Next.

Now is time to tell Windows where to look. You need your server’s 10 gigabit IPv4 address for this step. This was the last IP address we set just a few steps ago. We will write…

\\IPA.DDR.ESS.XXX\SHARED_FOLDER

Mine looks like this…

Success!

Check out those transfer speeds! This is from a local Samsung NVMe drive, through a Direct Attach Copper cord, into a 10 gigabit switch, out another DAC cord, and writing onto one of two spinning hard disks in my server.

As with most technology-based projects, this was TREMENDOUSLY FRUSTRATING and then TREMENDOUSLY REWARDING. Hope this helps one of you out there.

What are you using your 10 gigabit network for? Anybody doing any photo or video editing through it? Leave a comment below if you want an excuse for a humble brag.

Fixing Bad Blocks Finder burnin “ERROR - unable to open device: /dev/sda [Device or resource is busy]”

2019-12-20T00:00:00-07:00

My situation

I had four equivalent drives.

I had 2 x RAID 1 mirrors created with mdadm (ref)

I had both of these RAID 1 mirrors pooled together using mergerfs (ref)

The problem

I tried to run a burnin with bad blocks finder.

sudo bbf -c  burnin /dev/sda

I received an error.

ERROR - unable to open device: /dev/sda [Device or resource is busy]

Unmounting the drives

I went in and unmounted my mergerfs pool and both of my mdadm arrays from their respective spots in my /mnt folder.

$ sudo umount /mnt/storage
$ sudo umount /mnt/mirror1
$ sudo umount /mnt/mirror2

I tried the burnin again here and was still receiving the ERROR.

Deactivating the mdadm array

I then stopped the mdadm arrays, zeroed their superblocks, commented out persistent references to the array in /etc/fstab file, commented out the array definition in my /etc/mdadm/mdadm.conf file, and updated initramfs so that the early boot process doesn’t try to find these arrays. (ref)

Happily ever after

Now my first 10TB drive is doing a burnin… only 28 more hours to go ?

Hopefully this helps someone!

Special thanks to trapexit for not only his help figuring out the issue, but for developing these tools in the first place. Check out his website or his Github.

Here’s a link to the original issue a opened on Github.

Start Coding Now - linux