After waiting for a few months, I finally upgraded my main storage server from TureNAS core to TureNAS Scale during my server upgrade. Now I am asking myself, do I really need this upgrade?
I have had a storage server upgrade last few weeks, and just did some IO benchmark in my hypervisor.
I am using VMware ESXi as my hypervisor on a dedicated machine. it hooks on my Truenas Scale storage server with a 10G iSCSI connection and standard 1500 MTU (no jumble frame.)
Tests are performed with fio, and Google provides a simple guideline for VM disk IO benchmark https://cloud.google.com/compute/docs/disks/benchmarking-pd-performance
Test write throughput by performing sequential writes with multiple parallel streams (8+), using an I/O block size of 1 MB and an I/O depth of at least 64:
sudo fio --name=write_throughput --directory=$TEST_DIR --numjobs=8 \--size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio \--direct=1 --verify=0 --bs=1M --iodepth=64 --rw=write \--group_reporting=1
# sudo fio --name=write_throughput --directory=$TEST_DIR --numjobs=8 \--size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio \--direct=1 --verify=0 --bs=1M --iodepth=64 --rw=write \--group_reporting=1
write_throughput: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, 
1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.19
Starting 8 processes
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
Jobs: 7 (f=7): [W(4),_(1),W(3)][52.5%][w=64.9MiB/s][w=64 IOPS][eta 00m:58s]
write_throughput: (groupid=0, jobs=8): err= 0: pid=2502: Thu Jun 16 17:30:06 2022
write: IOPS=224, BW=229MiB/s (240MB/s)(13.7GiB/61466msec); 0 zone resets
slat (usec): min=47, max=1918.2k, avg=22289.12, stdev=138620.30
clat (msec): min=19, max=49855, avg=1843.07, stdev=5712.01
lat (msec): min=176, max=50052, avg=1865.65, stdev=5811.18
clat percentiles (msec):
| 1.00th=[ 180], 5.00th=[ 184], 10.00th=[ 300], 20.00th=[ 735],
| 30.00th=[ 776], 40.00th=[ 793], 50.00th=[ 827], 60.00th=[ 860],
| 70.00th=[ 919], 80.00th=[ 1301], 90.00th=[ 1569], 95.00th=[ 2072],
| 99.00th=[17113], 99.50th=[17113], 99.90th=[17113], 99.95th=[17113],
| 99.99th=[17113]
bw ( KiB/s): min=28617, max=1160657, per=100.00%, avg=252403.65, stdev=41662.93, samples=472
iops : min= 21, max= 1129, avg=240.85, stdev=40.74, samples=472
lat (msec) : 20=0.01%, 100=0.01%, 250=9.83%, 500=0.87%, 750=13.68%
lat (msec) : 1000=53.79%, 2000=18.23%, >=2000=5.37%
cpu : usr=0.20%, sys=0.38%, ctx=13534, majf=0, minf=467
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.1%, 16=0.6%, 32=1.2%, >=64=98.1%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,13827,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=229MiB/s (240MB/s), 229MiB/s-229MiB/s (240MB/s-240MB/s), io=13.7GiB (14.8GB), run=61466-61466msec
Disk stats (read/write):
dm-0: ios=0/16134, merge=0/0, ticks=0/12396824, in_queue=12396824, util=99.90%, aggrios=0/16134, aggrmerge=0/1, aggrticks=0/12424846, aggrin_queue=12424845, aggrutil=99.87%
sda: ios=0/16134, merge=0/1, ticks=0/12424846, in_queue=12424845, util=99.87%
[root@localhost Data]#
Test write IOPS by performing random writes, using an I/O block size of 4 KB and an I/O depth of at least 64:
sudo fio --name=write_iops --directory=$TEST_DIR --size=10G \--time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 \--verify=0 --bs=4K --iodepth=64 --rw=randwrite --group_reporting=1
[root@io-test ~]# clear
[root@io-test ~]# sudo fio --name=write_iops --directory=$TEST_DIR --size=10G \
> --time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 \
> --verify=0 --bs=4K --iodepth=64 --rw=randwrite --group_reporting=1
write_iops: (g=0): rw=randwrite, bs=(R) 4096B-4096B, 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.19
Starting 1 process
fio: io_u error on file /Data/fiotest/write_iops.0.0: No space left on device: write offset=303255552, buflen=4096
fio: pid=2596, err=28/file:io_u.c:1803, func=io_u error, error=No space left on device
write_iops: (groupid=0, jobs=1): err=28 (file:io_u.c:1803, func=io_u error, error=No space left on device): pid=2596: Thu Jun 16 17:32:46 2022
write: IOPS=60.0k, BW=238MiB/s (250MB/s)(5709MiB/23972msec); 0 zone resets
slat (usec): min=3, max=36791, avg= 8.45, stdev=39.28
clat (usec): min=184, max=177493, avg=1040.16, stdev=2205.25
lat (usec): min=198, max=177499, avg=1048.82, stdev=2206.88
clat percentiles (usec):
| 1.00th=[ 635], 5.00th=[ 717], 10.00th=[ 750], 20.00th=[ 799],
| 30.00th=[ 832], 40.00th=[ 857], 50.00th=[ 889], 60.00th=[ 922],
| 70.00th=[ 963], 80.00th=[ 1020], 90.00th=[ 1139], 95.00th=[ 1647],
| 99.00th=[ 3294], 99.50th=[ 4686], 99.90th=[ 9765], 99.95th=[ 15008],
| 99.99th=[160433]
bw ( KiB/s): min=82464, max=294088, per=100.00%, avg=245998.02, stdev=67413.87, samples=47
iops : min=20616, max=73522, avg=61499.38, stdev=16853.44, samples=47
lat (usec) : 250=0.01%, 500=0.01%, 750=9.57%, 1000=67.11%
lat (msec) : 2=18.91%, 4=3.75%, 10=0.55%, 20=0.05%, 50=0.02%
lat (msec) : 100=0.01%, 250=0.02%
cpu : usr=14.15%, sys=45.55%, ctx=156240, majf=0, minf=68
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1461581,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=238MiB/s (250MB/s), 238MiB/s-238MiB/s (250MB/s-250MB/s), io=5709MiB (5987MB), run=23972-23972msec
Disk stats (read/write):
dm-0: ios=0/1592185, merge=0/0, ticks=0/1429699, in_queue=1429699, util=99.56%, aggrios=0/1591823, aggrmerge=0/362, aggrticks=0/1422162, aggrin_queue=1422162, aggrutil=99.56%
sda: ios=0/1591823, merge=0/362, ticks=0/1422162, in_queue=1422162, util=99.56%
[root@io-test ~]#
Test read throughput by performing sequential reads with multiple parallel streams (8+), using an I/O block size of 1 MB and an I/O depth of at least 64:
sudo fio --name=read_throughput --directory=$TEST_DIR --numjobs=8 \
--size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio \
--direct=1 --verify=0 --bs=1M --iodepth=64 --rw=read \
--group_reporting=1
[root@io-test ~]# sudo fio --name=read_throughput --directory=$TEST_DIR --numjobs=8 \
> --size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio \
> --direct=1 --verify=0 --bs=1M --iodepth=64 --rw=read \
> --group_reporting=1
read_throughput: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
...
fio-3.19
Starting 8 processes
Jobs: 2 (f=2): [_(1),R(2),_(5)][100.0%][r=14.0GiB/s][r=15.3k IOPS][eta 00m:00s]
read_throughput: (groupid=0, jobs=8): err= 0: pid=1881: Thu Jun 16 17:48:29 2022
read: IOPS=21.4k, BW=20.9GiB/s (22.4GB/s)(1259GiB/60292msec)
slat (usec): min=13, max=76074, avg=197.95, stdev=422.19
clat (nsec): min=1404, max=693868k, avg=23722148.51, stdev=59989033.56
lat (usec): min=86, max=693925, avg=23920.38, stdev=60025.44
clat percentiles (msec):
| 1.00th=[ 12], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 12],
| 30.00th=[ 12], 40.00th=[ 13], 50.00th=[ 13], 60.00th=[ 13],
| 70.00th=[ 13], 80.00th=[ 13], 90.00th=[ 13], 95.00th=[ 14],
| 99.00th=[ 342], 99.50th=[ 347], 99.90th=[ 384], 99.95th=[ 414],
| 99.99th=[ 456]
bw ( MiB/s): min=18584, max=26532, per=100.00%, avg=21483.09, stdev=176.33, samples=954
iops : min=18579, max=26529, avg=21478.41, stdev=176.35, samples=954
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 100=0.01%, 250=0.01%
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.07%, 20=96.31%, 50=0.02%
lat (msec) : 250=0.01%, 500=3.57%, 750=0.01%
cpu : usr=0.82%, sys=49.54%, ctx=41872, majf=0, minf=466
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=1288207,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=20.9GiB/s (22.4GB/s), 20.9GiB/s-20.9GiB/s (22.4GB/s-22.4GB/s), io=1259GiB (1351GB), run=60292-60292msec
Disk stats (read/write):
dm-0: ios=47642/87, merge=0/0, ticks=15825386/36345, in_queue=15861731, util=100.00%, aggrios=47652/75, aggrmerge=1189/12, aggrticks=15817689/23298, aggrin_queue=15840987, aggrutil=100.00%
sda: ios=47652/75, merge=1189/12, ticks=15817689/23298, in_queue=15840987, util=100.00%
[root@io-test ~]#
Test read IOPS by performing random reads, using an I/O block size of 4 KB and an I/O depth of at least 64:
sudo fio --name=read_iops --directory=$TEST_DIR --size=10G \
--time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 \
--verify=0 --bs=4K --iodepth=64 --rw=randread --group_reporting=1
[root@io-test ~]# sudo fio --name=read_iops --directory=$TEST_DIR --size=10G \
> --time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 \
> --verify=0 --bs=4K --iodepth=64 --rw=randread --group_reporting=1
read_iops: (g=0): rw=randread, bs=(R) 4096B-4096B, 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.19
Starting 1 process
read_iops: Laying out IO file (1 file / 10240MiB)
fio: ENOSPC on laying out file, stopping
Jobs: 1 (f=1): [r(1)][100.0%][r=1061MiB/s][r=272k IOPS][eta 00m:00s]
read_iops: (groupid=0, jobs=1): err= 0: pid=1913: Thu Jun 16 17:50:33 2022
read: IOPS=271k, BW=1057MiB/s (1108MB/s)(61.9GiB/60001msec)
slat (nsec): min=1569, max=697200, avg=2087.86, stdev=1413.17
clat (usec): min=2, max=11769, avg=233.84, stdev=26.11
lat (usec): min=5, max=11886, avg=236.04, stdev=26.43
clat percentiles (usec):
| 1.00th=[ 182], 5.00th=[ 204], 10.00th=[ 217], 20.00th=[ 225],
| 30.00th=[ 225], 40.00th=[ 227], 50.00th=[ 229], 60.00th=[ 233],
| 70.00th=[ 237], 80.00th=[ 243], 90.00th=[ 260], 95.00th=[ 273],
| 99.00th=[ 330], 99.50th=[ 359], 99.90th=[ 441], 99.95th=[ 562],
| 99.99th=[ 816]
bw ( MiB/s): min= 946, max= 1085, per=100.00%, avg=1058.05, stdev=22.12, samples=119
iops : min=242382, max=277826, avg=270859.52, stdev=5662.33, samples=119
lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec) : 250=85.50%, 500=14.43%, 750=0.05%, 1000=0.02%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=48.33%, sys=51.25%, ctx=281, majf=0, minf=58
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=16234920,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=1057MiB/s (1108MB/s), 1057MiB/s-1057MiB/s (1108MB/s-1108MB/s), io=61.9GiB (66.5GB), run=60001-60001msec
Disk stats (read/write):
dm-0: ios=2831/16, merge=0/0, ticks=1340/15, in_queue=1355, util=6.47%, aggrios=2833/10, aggrmerge=0/6, aggrticks=1338/9, aggrin_queue=1346, aggrutil=6.45%
sda: ios=2833/10, merge=0/6, ticks=1338/9, in_queue=1346, util=6.45%
[root@io-test ~]#
Yesterday, I was check my home network regularly and see a message of possible to upgrade my Ubiquity Cloud Key to version 6. But unfortunately lost all configuration after the upgrade.
I bought a Mikrotik CCR1009-8G-1S-1S+ several years ago and use as my main router at home. It was relatively quiet with lower load when put on a shelf, but after I move to the new apartment and put it into a network cabinet, the noise was much higher especially in hot summer. So I decided to replace the factory fan with something quilter.
Today, a hard working day, stay in server room from 2:00 PM yesterday to 6:00AM today. Lots of things has been finished, including most part of physical networking setup. Now it look like a proper setup for corporation environment.
It seems there is something wrong with my Plex Media Server today, I can not see any new item after the library scan, although logs show plex do have access to those files in the console.
In this tutorial, we are going to creating a CentOS 7 base image for our Atlassian applications.
In our cases, the base image is will be created on an ESXi based virtual machine. If you are using other hypervisor or running on a bare-metal you can also follow this tutorial.
If you are using a cloud provider such as AWS, Google Cloud, DigitalOcean, the base image we are creating is very similar to their image.
Today I get another Supermicro SC846 4U Chassis for my Homelab.
Two months ago, one my friend wanted a dual-CPU motherboard for an old workstation setup and asking for my opinion. I replied if you need a second-hand server board it’s better to buy a barebone with a chassis to avoid repaired parts. So we bought a chassis with a Supermicro X9DRI-LN4F+ motherboard delivery to my friend’s place, and he keeps the motherboard, and I take the chassis. The system arrived in one week, but I was not going to take my chassis due to Coronavirus.
Finally today I clean up some space on my rack and pick up this 20kg metal box from my friend’s storage.
This chassis is a Supermicro SC846 4U chassis OEM for Inspur – Chinese server manufacture, as the I bought this server in China. Besides there are several Chinese stickers on the outside of the box and Supermicro logo is replaced with Inspur logo on right server handle, there are no different to any Supermicro chassis. There are Supermicro marker and part number everywhere inside.
Chassis
The first look is the chassis is in excellent conditions. The top of HDD cage, chassis door and chassis handle was protected by original plastic film, all surface is shine without rust or dust. Beside of reasonable marks, there is one visible dent on top of HDD cage.
Fans
The recent model of CSE-846 may come with three Nidec 7000 rpm fan (FAN-0127L4) with 72.5 CFM and 53.5 dBA in the middle and two Sanyo Denki 6700 rpm fan (FAN-0125L4) with 59.6 CFM and 47 dBA at the exhaust, but my configuration have five Sanyo San Ace 80 6700 rpm fans. Those fans are very loud at any configurable speed and not suitable in any home office environment, so I will replace them with much quieter Noctua fans.
Backplane
My chassis comes with a SAS846EL1 backplane, which is a very old SAS 1 backplane with only 12Gbit/s (4 lines x 3 Gbit/s) uplink to the controller, which will be probably replaced by direct plane or SAS 2 plane. I have summarised all backplane for SC846 chassis in this post.
Drive Tray
There are 24 Gen 5.5 hot-swap 3.5 hard drive trays (MCP-220-00075-0B) come with this chassis. As always Supermicro drive tray is ugly, I don’t like the wave design, and the small tray handle makes it difficult to pull drive out. The Gen 6.5 tray (MCP-220-00094-0B) has a much better look and handle design, but cost a fortune (RRP US$ 887.76) to replace all of my 72 trays and additional time to screw all the hard drives.
On the other hand, compared to most HP and DELL tray, most of Supermicro drive tray has much large hole, allow more airflow to go through the hard drive. Also, all of them includes plastic dummy hard drive, which allows airflow to go through the hard drive and preventing airflow bypass from the empty drive bay.
Lucky the chassis includes an internal drive tray (MCP-220-84603-0N), it is expensive (cost US$ 29.99 brand new), and very hard to find in China.
Power Supply
Chassis support redundant power supply, but only one 900W power supply (PWS-902-1R) come with the package, it’s load and not 80 Plus certified. All of my chassis come with this type of PSU, but I replace with much quieter 920W PSU with SQ (PWS-920P-SQ).
On Rack
Installed with 4U Rail Kit (MCP-290-00057-0N) on the rack. The Supermicro rail looks thin, but very well made, easy to install (no tool or stupid cage nuts required). It’s a ball-bearing rail, I can easily pull a full load server all the way out.
Finally, looks great with other servers. I left 1U between server for better cooling, and avoid a collision when I pull out the server.
Title Photo by Magnus Engø on Unsplash
Supermicro SC846 is a great chassis for homelab users, which offering 24 3.5inch hard drive bay with varieties of backplanes. Most users buy second-hand chassis and want to change backplane because it doesn’t fit their needs. I list all backplanes for SC846 until April 2020 and also my recommendations for the different use cases.
Title Photo by Magnus Engø on Unsplash
Group your equipment together, make sure you can access most of your equipment at the same location at the same time. This will reduce lots of effort when tracing an unknown device issue – you don’t have to run between different places.
You can have your equipment group in two or three locations, but you should not put them everywhere at home, except some special network device such as WiFi access point, or communication device, for example, VoIP phone.
The system should be located on a fixed location at home. Depends on the country you located, house structure you stay and types of equipment you have, the location of your home lab can be different.
Choose good locations is one of the most challenge part of your home lab setup. there are many variables have to consider
First of all, there must be a power source close to your homelab device. it is not a good idea of running long extension power cable without a certified electrician. Using extension power cable with high power equipment can be a fire hazard with, and not secured cable can cause accidents.
Provide dedicated circuit and power lets for your equipment if possible. Do not share a circuit with other high power electrical appliances such as air-conditioning, electrical oven or electrical heater. You don’t want your server lost power because of an overcurrent or short circuit in a different room far from your homelab.
Make sure you have enough existing network cable access to your homelab location. You should also have a backup line for your core network in case of a cable problem.
Evaluate your cable for future upgrade, for example, it will be very difficult to run a 10G network on a CAT5e cable. Consider the difference between a copper-based network or a fibre-based network, and which type of network your device supports.
Many enterprise equipments are designed for running inside a server room or datacenter, in most of the cases server noise is not a concern. Several new server models from HP and DELL is much quieter than before, but running noise may sill too loud for most of the people, especially at night.
Homelab equipment consumes electric and produces heat. Now new server cloud operates under higher environment temperature such as 28 Celsius, but some parts such as hard-drive may lose service life.
Heat can also cause server fan running at very high speed, and cause extra noise. We are human, we can not work in an extremely hot environment.
You should keep enough workspace for your homelab and make sure all equipment is easy to access. This should including enough spaces when not maintain your server and when you are moving your equipment.
Most servers are flat and long, they may difficult to go through narrow corner or doors. A full hight server cabinet can easily over 2 meter high, which makes them not fit in some basement or attic.
Servers are generally made by steal, which means they are heavy, very heavy. An empty 2U DELL or HP rack server can be easily over 20+ kg without any hard drive installed. Fully loaded 4U storage boxes can be heavy as 50+ kg, and you need at least 2 person to move those boxes, or using lift equipment. so make sure you still have space for your friends and carry equipment for your server.
A fully loaded 42U server cabinet can be heavy as 2000 kg, with footage less than 2 square meters. make sure your floor support a heavy load, and not make holes on it.
There are some commonplace for your homelab. they have different pros and cons.
Pros: This could be the best place for your home lab.
Cons: This could be the most expensive plan.
Pros:
Cons:
Pros:
Cons:
Pros:
Cons:
Pros:
Cons:
Pros:
Cons:
Pros:
Cons:
You can have your equipment in different place of your home, for example, I have my network gear in a small network cabinet just above the front door and my main servers located in a larger cabinet in my study room.
Be in a fixed location not means you can not move them. You can have your server sets on a cabinet with wheels, or have a cluster of Raspberry Pi on a box you can move from one place to another. but your homelab equipment should not like your laptop or iPad that you can take where you goes to.
Use server cabinet or rack can significantly make your homelab clean and make your device easily accessible because most of the industry equipment is design for rack mount.
But use or not use a server cabinet can depend on type and quantity of the equipment you have. For user with several heavy server and equipment, use standard server cabinet cloud be a better decision. The smaller size network rack is also a good choice for network device only user.
Need Not to Know 