iXsystems 5680 Detailed Hardware and System Build Instruction Manual

iXsystems 5680 Detailed Hardware and System Build

Product Information

Brand: iXsystems

Product Name: TrueNAS

Version: 1.24 (2023-05-03)

Author: WI_Hedgehog

Detailed Hardware and System Build Notes
The TrueNAS user manual provides detailed hardware and system build notes to assist users in setting up their systems properly.

New User Advice / Help
TrueNAS may seem simple in videos, but it is recommended to read the documentation and follow the guidelines to ensure a smooth setup process. It is advised not to deviate from the recommended procedures.

Documentation and Resources
To save time and money, users can find helpful information in the Documentation and Resources tabs on the product website. The TrueNAS manuals are particularly useful. There is also a powerful search tool available for quick access to relevant information.

Precautions

New users should be aware of potential issues that can arise when using TrueNAS. Some precautions include:

• Avoiding excessive customization that may cause compatibility issues.
• Understanding the differences between operating systems (e.g., TrueNAS, Ubuntu Server, Windows) and their respective interfaces.
• Considering the power and noise factors when choosing hardware components.

Fundamentals

The following fundamental topics are important for TrueNAS users:

• Basic understanding of TrueNAS, including terminology and abbreviations.
• The hidden cost of using ZFS for a home NAS.
• Explanation of ZFS concepts such as VDev, zpool, ZIL, and L2ARC.
• ZFS pool layout and capacity calculation.
• Importance of avoiding RAID rebuild failure.
• Reasons why USB drives are not recommended for NAS usage.
• Considerations for ECC memory usage.
• Differences between Ryzen and Xenon ECC support.
• Importance of using a Host Bus Adapter (HBA) instead of RAID controllers.
• Understanding Serial Attached SCSI (SAS) and Serial AT Attachment (SATA) Port Multipliers.

Hardware Range of Builds

The TrueNAS system supports various hardware configurations.
Users can choose from the following build options:

• Under 30W Systems: Economical systems with low power consumption.
• Power Efficient Home Server: DIY full-size server with power consumption around 23W.
• Super Powerful Micro Server: Supermicro Superserver E300-9D-8CN8TP, priced at approximately $2,200 USD as of April 2023.

Required Hardware
The Hardware Guide provides essential information on selecting the right hardware components for a TrueNAS system. It is recommended to read the guide before making any purchasing decisions.

Storage
The LSI HBA Card Insight provides insights into storage options for TrueNAS systems, including cost-effective solutions.

Product Usage Instructions

Step 1: Familiarize Yourself with the Documentation
Before starting the setup process, thoroughly read the TrueNAS user manual and other available documentation to gain a comprehensive understanding of the product.

Step 2: Follow Hardware Build Guidelines
Refer to the Hardware Guide to select compatible hardware components for your TrueNAS system. Ensure that you have all the necessary components before proceeding.

Step 3: Install TrueNAS
Follow the installation instructions provided in the user manual to install TrueNAS on your chosen hardware platform. Make sure to follow the recommended operating system installation procedures.

Step 4: Configure TrueNAS
Once TrueNAS is installed, access the system interface and follow the configuration wizard to set up your storage and network settings. It is recommended to refer to the user manual for detailed instructions on configuring various aspects of TrueNAS.

Step 5: Add Storage Devices
Connect the storage devices (hard drives, SSDs, etc.) to your TrueNAS system according to the hardware guidelines. Use the TrueNAS interface to add and configure the storage devices for optimal performance and data protection.

Step 6: Set Up Data Sharing
Configure data sharing protocols such as SMB (Windows File Sharing), NFS (Network File System), or AFP (Apple File Protocol) to allow access to your TrueNAS storage from other devices on your
network. Refer to the user manual for detailed instructions on setting up data sharing.

Step 7: Implement Backup and Security Measures
Ensure that you have appropriate backup and security measures in place to protect your data. Consider setting up regular backups, implementing user access controls, and enabling encryption if necessary.

Specifications

• Brand: iXsystems
• Product Name: TrueNAS
• Version: 1.24 (2023-05-03)
• Author: WI_Hedgehog

Frequently Asked Questions (FAQ)

Q: Is TrueNAS suitable for streaming video?
A: TrueNAS can handle video streaming, but for low-power and quiet streaming solutions, consider using a Raspberry Pi with Kodi or other alternatives.

Q: Can I use TrueNAS for industrial storage?
A: Yes, TrueNAS is recommended for users who require an industrial storage solution on server-grade equipment.

Q: Are there any precautions for new TrueNAS users?
A: New users should avoid excessive customization, be familiar with the operating system’s interface, and consider power and noise factors when selecting hardware components.

Q: Where can I find more information about hardware and system builds?
A: The Hardware Guide provides detailed information on hardware selection and build options. Additionally, the user manual and documentation offer insights into various hardware configurations.

version 1.2
4 2023 05 03
including
Detailed Hardware and System Build Notes (plus new user advice / help)
Author: WI_Hedgehog

NOT AN OFFICIAL iX systems document.

CREDIT: This list was originally based on @Chris Moore ‘s Links to useful threads and @Davvo ‘s list (in his DISCLAIMER: What you do is up to you , opinions and comments herein non withstanding. Commen ts are based on enterprise level hardware reliability statistics and critical data retention, which to different people means different things and holds different valu es. You could conceivably run True NAS in a VM without ECC memory on generic consumer hard ware for years without problems, buuut that’s not really “best practice.”
Sometimes though, “you gotta do what you gotta do and throw caution to the wind.” I’ll just mention you might want to keep on t op of your backups.

TrueNAS is not as simple as videos make it look, though it is fairly easy to get a system running if you read the documentation and don’t color outside the lines.

APPLICATION

Briefly:

• Use TrueNAS, on bare metal server equipment, with as few modifications as possible, if your data is really valuable.
• Use something other than TrueNAS that is more suited to your needs if you want a custom solution.

TrueNAS has a solid reputation for being a data serving powerhouse for a reason: It’s designed to be in a Datacenter for the purpose of attaching a crap-ton of storage to the network. Using it for something other than what it is designed for can be a recipe for disaster. Using it for home storage is…well, you should really have a need first….and I think that’s why people are trying to shoehorn it into all these “also” situations, like “also gaming” and “also a Virtual Machine server” and “also works on hardware designed to run Windows 3.11.”

Seriously, if you don’t have an actual need, [used] server-grade hardware, and way more time to invest than you’re expecting, load up Ubuntu Server and work on a great data backup strategy–you’ll be way farther ahead, have easy-to- understand guides, and your electricity bill will be far less. If you have to run Proxmox: set up file shares, Proxmox can do that. If you really have to have a lab in your home because like me you live in an arctic climate where it’s dark 23.75 hours of the day and the high is -12°F and you need the heat so your coffee cup isn’t simply holding a solid block of ice, set up TrueNAS on one box (with ECC memory and a real NIC of course), and Proxmox on another box (with ECC memory and a real NIC of course), and a workstation or two (with ECC memory and a real NIC–of course!), and a crap router so nothing works the way it should.
(I have a good router, I’m kidding you.)

If you want to save yourself lots of money and loads of time, the top of the page has Documentation and Resources tabs which contain extremely helpful information like the TrueNAS manuals. There’s also a Search tool at the upper- right that works extremely well. If you understand the system and requirements before jumping in you’ll save money, time, and frustration.
As general advice, if you run TrueNAS as a NAS and don’t add things to it it tends to run fine and without issue, though I suggest using an Uninterruptible Power Supply and NUT.

Precautions: (dreaming big is fine, but let’s nail down a few things that go badly for new-to-TrueNAS users)

• Adding Virtual Machines, Jails, Containers, TrueCharts, scripts, and especially running TrueNAS in a virtual environment (Proxmox) is where people get themselves into trouble and start losing data. It appears most initial problems occur due to network bridging not being configured properly, with drive access perhaps being the second biggest issue. (keep reading)

• If you saw a video with TrueNAS 1) in a VM, 2) being used as a gaming server, or 3) Plex/Jellyfin server: You’re probably in for more work than you expect. First build TrueNAS as a NAS on server-grade hardware, and when that’s tested and stable incrementally expand it, or better yet use a different platform completely as TrueNAS doesn’t work spectacularly for any of those purposes (though it does work well as NAS, not surprisingly). (keep reading)

• TrueNAS CORE (BSD) is faster and more stable than SCALE (Linux), so consider running CORE. There’s almost no interaction with the operating system anyway.

• DO NOT use gaming hardware for a NAS; TrueNAS is a Datacenter-grade platform and will not run well on gaming hardware just like games don’t run well on server hardware.

• If you’re trying to repurpose an old desktop system use Ubuntu Server (possibly with ZFS though that’s often not the best choice for older systems, use LVM instead), not TrueNAS. Videos showing TrueNAS working for a day of filming before tearing down the system for another build is entertaining, but this isn’t good for long-term stability (or even short-term stability). TrueNAS uses hardware in ways desktops were not designed for and they’ll eventually break and your data will be lost forever. Use Ubuntu, –or Gentoo if you want to wander into the deep end of the pool. Repurposing a desktop can get real expensive real quick and you’re better off buying used server equipment–you’ll save money and time.
  (Note that Ubuntu Server and TrueNAS have a text-only terminal attached, unlike Windows or Ubuntu which are graphical and use a mouse.)

• Is TrueNAS “a lot cheaper and as easy to use as a Synology NAS?” Well, cheaper…that depends on Total Cost of Ownership, including the time you invest in learning any system. It’s hard to beat Synology if you want an easy-to-use home system that you just plug in and click around a bit, though Synology gets expensive when storing a lot of data.

• TrueNAS as a VM Under Proxmox or VMWare: Don’t do this, Proxmox can operate as a SMB Server for Windows or NSF Server for Linux. VMWare sharing methods: (link1, link2)

• Minecraft: This isn’t a good idea, Minecraft is best run on an old desktop where it will run far faster and smoother–and without the cost of server hardware TrueNAS requres. The ZFS filesystem TrueNAS uses can be used there but you don’t want to do it as it will wreck performance.

• Steam Games and Deduplication: If you have the money, it’s doable. Otherwise…

• Plex/Jellyfin: This isn’t a great idea for many reasons, though it can be done. Transcoding is somewhat pointless as even cheap phones can display 4K data streams, so a Raspberry Pi [alternative] with Kodi is enough to stream video and will use far less power and be much quieter. Both work great under Proxmox (which is designed to run containers), however be prepared for a lot of hardware and configuration headaches you’ll most likely face if trying to do this with TrueNAS.

If the above suggestions don’t meet your needs because you have valuable data and truckloads of it, and you’re convinced you need an industrial storage solution on server-grade equipment to handle it (without breaking the bank), then TrueNAS is probably what you want.
(If you’re reading this on a phone, turn it sideways and read in Landscape Mode, longer lines will make more sense)

Fundamentals

Basic Understanding

• Search Before Posting (get immediate answers!)
• About YouTube Videos and TrueNAS
• Terminology and Abbreviations
• The ‘Hidden’ Cost of Using ZFS for Your Home NAS
• Slideshow explaining ZFS, VDev, zpool, ZIL and L2ARC for noobs
• ZFS Introduction
• ZFS Pool Layout
• ZFS Capacity Calculator
• RAID-Z1 is Dangerous (well, -Z0 is worse)
• Avoiding RAID rebuild failure
• Universal Serial Bus of Doom (why USB drives aren’t used in a NAS)
• ECC Memory: Do I have to??? (explained in a build thread)
• Ryzen “may-be ECC” vs Xenon 4-bit ECC (normal Intel ECC is 1-bit)
• RAID Controllers won’t work; Use a Host Bus Adapter (HBA) (realllly important, even though…)
• Understanding Serial Attached SCSI (SAS)
• Serial AT Attachment (SATA) Port Multipliers [of Trouble]
• System Manuals (each version’s manual is under Documentation at the top of the page)

Hardware

Range of Builds:

• Under 30W Systems (The most economical systems / Die sparsamsten Systeme)
• Building a Power Efficient Home Server (23W D-I-Y full-size server, YouTube – Wolfgang’s Channel)
• General Overview (UNRAID, though applies to TrueNAS as well)
• Super Powerful Micro Server (Supermicro Superserver E300-9D-8CN8TP about $2,200 USD 04/2023) Required Hardware:
• Hardware Guide (READ THIS BEFORE BUYING ANYTHING!!!)

Storage:

• LSI HBA Card Insight (and ‘how to frugal’)
• Fake LSI Cards (eBay is full of them)
• Westerm Digital Red / SMR Drive Incompatibility with ZFS (ZFS eats them for lunch)
• Non-Running Drive when Shucking Western Digital Hard Drives from USB Enclosures
• Hard Disk Drive Price/Performance Analysis
• 5,400 RPM Drive Speed (a good thing)
• Serial Attatched SCSI (SAS) Informational Videos
• Fibre Channel (playin’ with the big boys) QLogic QLE2564, QLogic QLE2694-SR.

Power:

• Power Supply Unit (PSU) Selection and Sizing (also important)
• Detailed Look Inside Power Supplies
• How Power Supplies Work
• Current Ripple and Life Calculation (triple your ripple and say goodbye to your power supply) (explained, filter considerations)
• Measuring Ripple
• [Official] CPU Heatsink Recommendations

Network:

• Realtek Network Adapters ARE Bad (why to avoid cheap Ethernet)
• 10GBASE-T Over Cat6 vs Cat6a vs Cat7 Cabling

CPU:

• Intel Atom vs CORE
• Intel processor comparison/specifications
• Note: The Intel Quark SoC, lower-power chips than Atom, were discontinued in 2019.

Low Power Mainboard:

• Low power Mainboard & CPU
• ‘On the frugal’: Mainboard: Supermicro X9SCM-F, CPU: Intel Xeon E3-1230
• More Power: Supermicro A2SDi-2C-HLN4F / A2SDi-TP8F, X10SDV-TLN4F Insight:
• How to Buy Used Hardware (eBay hard drives)
• Zebras All the Way Down (why simplicity is important)

Example Systems

• ESXi and Plex on a Node 304
• Pheran’s 80TB build (with photos)
• beard-strokers

Building

• Choosing a TrueNAS Version
• Building, Burn-In, and Reliability Testing
• Hard Drive Reliability Testing
• Hard Drive Troubleshooting Guide
• SFF-8644 External HD MiniSAS Connectors: Not necessarily difficult to plug them in wrong.
• Solnet Array Test Script

Configuration

• – Leave the Boot Drive Alone (do not partition it for data storage)
  – Setting the Time Correctly (NPT drift)
  – Structuring Datasets
  – Useful Commands
  – Useful Settings
  – Supermicro’s Fans Fix via IPMI Sensor Thresholds
  – Removing SMB1, NetBIOS, and WINS
  – rsync fails when copying Windows datasets

Maintenance

• Backups
• Drive Life Expectancy (about 50,000 hours for Enterprise drives)
• Hard Disk Drive Study (Backblaze)
• Solid State Drive (Study (Backblaze)) (Article)
• Western Digital Red
• Replacing [Failed] Drives
• System Update Procedure – !!! READ THIS FIRST !!!
• Managing Pools
• How do files ONLY IN A SNAPSHOT get corrupted?
• How to diagnose strange crashes
• How Thermal Paste Works and How to Apply It (Intel)

DO NOT add the following “stuff” to your system until you actually need to as it will unnecessarily complicate things and possibly break other stuff:

Ease of Use

• S.M.A.R.T. Reporting Script (multi_report.sh for CORE and SCALE)
• TrueNAS-Report*
• HDD Spindown Timer Script
• Pool Rebalancing Script
• FreeNAS Scripts
• ZFS transfers with netcat on trusted networks bc the basic calculator is in CORE and the script will work there out of the box; SCALE however does not include bc in the base system (yet…this may change) but can be added if you want to use the script there.

Advanced

Storage
LSI/Avago/Broadcom Host Bus Adapters:

• LSIutil (fixing your LSI card)
• Crossflashing LSI 9211/9300/9311 HBA (and variants)
• Erasing the LSI 9207-8i BIOS (not the firmware) for Faster Boot

Data Deduplication:

• How to Kill Performance (and raid your wallet) With Deduplication (why you shouldn’t use de-dup)
• Deduplication Usage Example
• Use Case
• iXsystems manual page

Bottleneck Reduction:

• What are IOPS ???? (un-bogging drive-bound systems)
• Simple SLOG analogy
• ZIL & SLOG (additional info) (sync write backup & using a separate backup device)
• SLOG Benchmarking (additional info)
• Fusion Pools (ZFS allocation classes, ZFS special vdevs, and metadata vdevs)
• Intent: Offload high IOPS traffic onto SSD or Optane
• Necessary Redundancy
• Some confusing aspects explained
• Fragmentation
• Hot spare
• Nope: Boot, SLOG and Fusion Pool cannot be on the same mirror (how to break everything)
• Hierarchical Storage Management (also known as Tiered storage)
• Oracle HSM is done via the Sun StorageTek SAM-FS software
• Open Source: Online Hierarchical Storage Manager (under development, whitepaper)

Boot Resiliency:

• Highly Available Boot Pool (3 SSDs for redundancy)

Large-Scale RAID:

• dRAID (CLI currently in SCALE, GUI in upcoming SCALE Cobia)
• Videos explaining dRAID

iSCSI / Block Storage

• How to Configure Block Storage (not file shares)
• Why iSCSI often requires more resources for the same result
• Why Use Mirrors for Block Storage

Network

SAMBA Performance:

• Hardware
• Tuning
• Faster (SMB / CIFS) Share Performance
• In-Depth Tuning
• SMB Multichannel

Bandwidth:

• 10 Gbps Ethernet Primer (a diet for your wallet)
• Intel X710, XL710, XXV710, and X722 cards
• 40 Gbps Mellanox InfiniBand (about, notes, also note the switches are different than Ethernet and a bit spendy)
• InfiniBand vs. Ethernet

Switch, Firewall:

• Ultimate CHEAP 2.5 & 10GbE Switch Guide (generally under $200 in 1st quarter 2023)
• CHEAP powerhouse 2.5GbE Mini PC Router/Firewall ($300 in 1st quarter 2023)
• MikroTik CRS518 (16) 25GbE & (2) 100GbE port Low Power Switch ($600 in 4th quarter 2022) Special-Use Cases (Fast Networking by Breaking Convention):
• Understanding LACP (Link Aggregation)
• Remote Direct Memory Access (RDMA)
• Overview (Slideshow) (Outline)
• How RDMA Works and Bypassing RDMA Security (USENIX Security 2021 video)

UPS

• Network UPS Tools (NUT support) – Energenie UPS

Dangerous Territory

• Nextcloud-iocage-FreeNAS (maybe TrueNAS)
• Plex (I think…I didn’t try it, your mileage may vary)
• Virtualize TrueNAS (iXsystems)
• A Guide To [not] Completely Losing Your Data

Applications
TrueNAS is an excellent NAS appliance, and there are sound reasons to stretch that further, though doing so is not without inherent risk. While the linked threads contain some “bold exaggerations” they also contain valuable information on what to expect and how to keep your system out of trouble while capitolizing on the resources.

• TrueCharts: Things may not go as expected (so plan ahead)
• Community App Repository (similar situation to TrueCharts, plan ahead)

Versions

v1.24 has the following updates:

Fundamentals

• Hardware
  • Low Power Mainboard
  • ‘On the frugal’
  • Maintenance
  • Drive Life Expectency
    • HDD Study
    • SSD Study & Article
    • WD Red
  • Changed:
  • Phone-friendly formatting.
  • PDF download now available.
  • Clickable hyperlinks (hopefully).

v1.20

• Add bookmark
• Yesterday at 11:25 AM

Added:

• more TrueNAS CORE links
• more Raspberry Pi alternatives

Fundamentals Hardware

• low-power system links
• CPU (section)
• Low Power Mainboard (section)

Building

• Choosing a TrueNAS Version

Maintenance

• Backups
• How Thermal Paste Works and How to Apply It
• Advanced

Storage

• LSI/Avago/Broadcom
• Data Deduplication

Bottle neck Reduction

• Fusion Pools

Large Scale RAID

• dRAID videos

Network

• SAMBA Performance (section)

Bandwidth 10 Gbps

• Intel X710, XL710, XXV710, and X722 cards
• Special Use Cases (section)
• Remote Direct Memory Access (RDMA) (section)
• Applications (section)

Changed:

• faster loading images
• minor organization, formatting, link, and text changes

v1.10

• Add bookmark
• Apr 26, 202 3

Added:

• more Proxmox information
• Steam Games
• Fibre Channel
• more Power Supply information
• Hardware insights (Zebras All The Way Down)
• SFF 8644 cabling notes
• NPT (time)
• Useful Settings
• more Ease of Use tools
• more De duplication information
• dRAID
• tiered storage information (we don’t have it yet)
• more Block Storage information
• Switches, Routers, Firewall
• UPS Tools
• more Somewhat Dangerous “stuff”

Minor link and text changes.

v1.0 Major Overhaul

• Add bookmark
• Apr 24, 202 3

@Davvo’s signature was rolled into Hedgehog’s signature and expanded upon, when that grew too large this resource was created. These updates were made:

Added:

• Content: Resources now cover most major topics.
• Milk toast disclaimer

Changes:

• Topics are grouped into sections with similar topics located together.
• Formatting corrections and updates for easier/faster viewing.

NOTE: This is not a guide, although it is grouped into a logical format that the index to a guide might follow.

References

• How To Create A Proxmox Ubuntu NFS Server - First2Host.co.uk - NVMe VPS, NVMe Web Hosting. NVMe ISP
• InfiniBand vs. Ethernet: What Are They? | FS Community
• Cat6 vs Cat6a vs Cat7? Which 10G Ethernet cable wins? | FS Community
• SFF-8644 - CS Electronics
• Using the lsiutil Application
• Oracle Hierarchical Storage Manager and StorageTek QFS Software Customer Documentation Library
• Share Local Folders and Drives
• Faster Samba (SMB / CIFS) Share Performance - Hilltop Software
• Data deduplication - Wikipedia
• Hierarchical storage management - Wikipedia
• Host adapter - Wikipedia
• InfiniBand - Wikipedia
• Intel Atom - Wikipedia
• Intel Quark - Wikipedia
• LSI Corporation - Wikipedia
• [Official] CPU Heatsink Recommendations - Hardware - serverbuilds.net Forums
• SMB Performance Tuning - General - Unraid
• GitHub - markusressel/zfs-inplace-rebalancing: Simple bash script to rebalance pool data between all mirrors when adding vdevs to a pool.
• 15 Single Board Computers: Alternatives to Raspberry Pi
• iXsystems Jira
• Open Source Home Theater Software | Kodi
• The 'hidden' cost of using ZFS for your home NAS
• bc(1)
• Network UPS Tools - Welcome
• Supermicro SYS-E300-9D-8CN8TP IOT/Embedded System
• ubuntu.com/server
• Unraid | Running an Energy Efficient Unraid Server
• Adding a Samba share to Proxmox as Storage – Virtualize Everything
• Performance Tuning - SambaWiki
• ZFS Capacity Calculator - WintelGuy.com
• Supermicro Superserver E300-9D-8CN8TP Intel Xeon D-2146NT Networking PC w/ 2X SFP+, 2X 10GbE LAN, 4X GbE LAN, IPMI : Electronics
• Hard Drive Life Expectancy
• The SSD Edition: 2022 Drive Stats Mid-year Review
• What Is a Data Center? - Cisco
• The Latest - Whether RAID 5 is still safe in 2019 - Digistor Australia

Read User Manual Online (PDF format)

Download This Manual (PDF format)

Download this manual  >>