Shamoozal.com - Smorgas Board

Ok, so I have run out of disk space again. I need to go over what I am storing from my HTPC, I am sure there are some shows/movies I can dump, but this has gotten to the point that I really need to get a real solution. So I am thinking of building a real storage server. As I am planning on upgrading my current gaming PC/storage server, I will have a lot of space hardware which is still usable, just not good enough for my gaming needs anymore. Well, it turns out that I really did spec my existing system pretty well, since the motherboard is fully certified for Solaris, BSD, FreeBSD, Open Solaris (and derivative versions), Linux, and of course Windows.

So I have been doing some research and will just need to get a few parts (other than the hard drives themselves). So I am looking at getting this case:

Norco RPC-4224 case


3x IBM M1015 SAS/SATA RAID Controllers (supports 8 SAS/SATA disks each)

ZOTAC ZT-50608-10L GeForce GT 520 (PCIe 1x slot video card)

Hard drives... At least 1 SSD for use as L2ARC (read cache for frequently accessed data from the disk pool) and then at least 6 3TB drives (will get more later as I need more space)

Then combined with my existing stuff:
EVGA 780i SLI motherboard (6x SATA ports)
8GB DDR2 RAM
Intel Q9450 Quad Core CPU
2x 500GB drives for OS mirror'ed pair

I would probably install either OpenSolaris, OpenIndiana, or Solaris 11 on the system to use ZFS. The reason for the 6 drives is that the controllers and backplanes are designed around that number (i.e. 3 controllers each with 2 8087 connectors connect to the 6 8087 connectors on the hot-swap drive backplanes). I will also disable the ZFS ZIL log, since I will be on a UPS, I won't worry too much about data not getting to the drives in the event of a power outage (as ZFS caches reads and writes using RAM).

So assuming I use RAIDZ (like I plan), that will give me just under 14TB of usable storage with the 6 disks, or 56TB when fully populated. I think that should handle my needs....

_________________

Dude, how in the WORLD are you running out of disc space? You filled up 14TB worth of drives? I can't even comprehend it.

I have two 1TB drives and an old 500GB one. The two 1TB drives aren't even close to being full (one of them is only a back up). At one point I even stole a ton of movies that were on a server here and it still wasn't close to being full.

Are you recording all of TV in HD or something? It's the only solution I can come up with.

I really need to upgrade.
I only have a 120GB hard drive at home. Its got about 20GB free. 80GB is music. I don't keep movies or games on there. I delete them after I watch them or finish playing them. And still use Windows XP. I need to go to 7 especially since I finally got it in work about 6 months ago.

There's just so much shit to backup and programs to reinstall. I'm too lazy.
The last thing I want to do is work on a computer when I'm not at work.

_________________
WWW.MOVIEWISEGUYS.COM

Yeah, I hate the whole process of putting everything back onto a PC. I had to go through that when I upgraded to Win 7, which is also when I added my new drive.

But yeah dude, I have no idea how 14TB is not more than enough space to deal with. Did you download all of Steam?

I got an exclusive sneak peek at Fallen Kell's bedroom.

_________________
WWW.MOVIEWISEGUYS.COM

Yeah right now I have 3TB on my server and 8 or 9 TB on my HTPC (well, I HAD 8 or 9 TB, I had another disk crap out AGAIN!!!! And I have been so fed up with those disks that I havn't even bothered to figure out which one died and what I even lost).

My TV shows record at about 1GB per half hour. And I know I have a lot of stuff on there that I can probably cut. But I also have all my DVD's ripped, and several Blu-rays ripped. Some of the DVD's have been converted to mkv files with H.264 compression (which get them to about 1 to 1.3 GBs for the movie), but the ones that have not are 4-8GBs, and the Blu-rays are around 20-40GBs each.

So, its a LOT of space for the movies and then add in a few TV series (especially like the Simpsons which are almost up to 500 episodes, of which I have almost all of them, so probably 400GB right there) and it adds up quick...

_________________

I have 500GB. I don't for see the need to get another hard drive any time soon. If you're running out of disk space and you have 14TBs, you should probably look to readjust your life style.

What in the world could you possible be hoarding? My god. We should probably call up Hoarders right now and send them to your house because you my friend, are a digital hoarder. It's called DELETE/FORMAT. Use it. You don't need to be hoarding all those shows, movies, software, porn, and whatever. Just go a head and delete it all. it's okay. The world will not end if you delete that stuff. Learn to let go. Please. If you love yourself, you'll do it. We believe in you.

_________________
"If you always put limit on everything you do, physical or anything else. It will spread into your work and into your life. There are no limits. There are only plateaus, and you must not stay there, you must go beyond them. "

HAHAHAAHHA dude you beat me to the digital hoarder thing, I was seriously thinking that myself.

Yeah man, you don't need to have all those movies and shows backed up. Heck I'm about to get rid of my entire CD collection (which isn't huge) and for a moment I thought "I should rip all of this" and then I realized I just didn't give a shit. If I want to listen to any of that stuff again I'll either find it on Spotify or download it some other time.

My games are the only things I like hoarding, but that's like the only thing I collect, and even still, when I put that shelf together my first thought was "some of this has to go."

_________________

Holy shit, I can't even begin to comprehend needing that much space.

Yeah that's kind of crazy although after this year in residence I kind of get it. I've probably downloaded about 800GB-1TB since moving in. I came with a 1.5TB external hard drive and I was expecting to maybe fill half of it by the end of the year and at this rate it will likely be pretty close to full. While I get the logic of deleting stuff when I am at home I have a download cap and re-downloading even one of these things to watch it again chews into a decent amount of that cap.

_________________

I then pose this question to you.

How often are you rewatching anything? Be honest.

You think I'm rewatching Transformers 3? Fat chance. DELETED.

_________________
"If you always put limit on everything you do, physical or anything else. It will spread into your work and into your life. There are no limits. There are only plateaus, and you must not stay there, you must go beyond them. "

A collector never stops collectin'.

Of the movies on my external drive I would say between 20-30% are movies I have seen before and decided I wanted to watch again. I do delete some movies that I consider real stinkers.

_________________

I have quite a bit of space on my two computers about 4 Terabytes, but most of it is open. I have a 2tb in my HTPC which is half full, and 2 1tbs in my computer I'm on now. One is half full the other is empty.

My media drive on this computer is a disaster, I always want to sit down and go through it and delete alot of the shit on here but I always ignore it and more shit piles on.

I just burned through 5tb and because of this, I'm about to finally build a backend for my HTPC (read: basically a media server). And i'm thinking of doing something similar.

About to pick up one of these and one of these to attch one of these.

I Thought about going the 4U server route with a Supermicro, but i just don't have a place to put it. With the HP mini server and the 8 bay external, i can just throw it in a closet.

I was thinking of going RAID5 for the data and raid0 for the OS with Windows Home Server, but idk for sure yet.

What's the deal with ZFS? Tell me about it.

_________________
"Cook Free, or Die" -- Anthony Bourdain

ZFS is a filesystem developed by Sun Microsystems and released under one of Sun's open source licenses. It was ported over to BSD, OpenBSD, FreeBSD, FreeNAS, and a couple Linux distributions (the linux ports only ran in user space, not kernel space as that was not compatible with the GPL license, thus performance under linux suffers).

ZFS is pretty cool, but as stated, you really need to run either Solaris, OpenSolaris, or one of the BSD's to really utilize it and as such, need to be familiar with Unix administration.

Basically, ZFS allows filesystems up to 16 Exabytes (16 million terabytes) in size, containing up to 281 trillion files. It is a software disk management system as well, allowing you to create a pool of disks which you can configure for performance, redundancy, etc., and create multiple filesystems on the same disk pool (in other words, allows you to present the group of disks as one or multiple disks). ZFS uses system RAM to cache reads and write operations which gives a dramatic performance boost when dealing with files smaller than the amount of RAM on the machine (i.e. your write speed to ZFS will be in the GB's per second assuming you can get data that fast to the system up until you hit the size of your RAM, and depending on your system tunings (i.e. no ZIL log, or separate ZIL log on fast SSD drive(s))). You can also use SSD drives to expand the read/write cache above and beyond the size of your RAM. You can create RAID 0, RAID 1, RAID 5, RAID 6, straight disks, concatenations, hot spares, and pretty much any combination there-in. At work I have setup a ZFS filesystem that uses 4 external disk arrays of 28 disks each such that there are multiple RAID 6 devices striped together in such a way that you can lose one of the 4 disk arrays and the data would still be available (losing an entire array is something entirely possible, maybe the system board dies, or both power supplies fail, etc). The flexibility that ZFS gives you in the creation of the disk pool also allows you to expand the number of disks in a pool and increase you space over time.

ZFS also is one of the only filesystems which creates and saves a checksum of the data so it can tell you if your data has been corrupted (and exactly what files). This is something that most people do not even think about, but in larger and larger systems, the chance that there is a bit-flip on a disk gets more and more likely simply due to the sheer amount of bits that are being stored. There are lots of things which can cause this, from electro-magnetic fields, radiation, even physical shock, and in other filesystems, you would never know it happened other than things might start crashing, or failing, or just don't look right. It supports creating snapshots, so say you want to make a change to your system, well, you can create a snapshot before you make the change, and if you don't like the change, you can simply roll back to the snapshot, or even just copy the individual files from the snapshot to essentially go back in time to that previous version of the file. You can set it up that snapshots happen on a schedule (i.e. daily, weekly, monthly, keep X daily, Y weekly, Z monthly and remove the oldest, etc). Sharing out to windows machines is extremely easy with just a couple commands that need to be run.


Now for the setup I intend to use, I would install disks vertically in the case I linked, which would spread the drives across all 6 SAS backplanes, and the 6 ports on the 3 SAS controller cards. This would maximize the performance of the drives as the load would be spread evenly across everything. I intend to create up to 4 groups of 6 disk raidz (raidz is effectively raid 5) which are then striped together (raid 0), so essentially a raid5+0 setup. I would be able to lose 1 disk out of each vertical columns of disks without losing data in that setup.

_________________

Are we going to have to revoke your posting privileges?

_________________
"If you always put limit on everything you do, physical or anything else. It will spread into your work and into your life. There are no limits. There are only plateaus, and you must not stay there, you must go beyond them. "

_________________
WWW.MOVIEWISEGUYS.COM

16 million terabytes....16 MILLIONTERABYES?!?!

_________________

hahaha.... well he asked me to tell him about ZFS and what it was

_________________

The invention of electronic computers in the 1940s, along with the development of mathematical information theory, led to a realization that brains can potentially be understood as information processing systems. This concept formed the basis of the field of cybernetics, and eventually gave rise to the field now known as computational neuroscience. The earliest attempts at cybernetics were somewhat crude in that they treated the brain as essentially a digital computer in disguise, as for example in John von Neumann's 1958 book, The Computer and the Brain. Over the years, though, accumulating information about the electrical responses of brain cells recorded from behaving animals has steadily moved theoretical concepts in the direction of increasing realism.

The essence of the information processing approach is to try to understand brain function in terms of information flow and implementation of algorithms. One of the most influential early contributions was a 1959 paper titled What the frog's eye tells the frog's brain: the paper examined the visual responses of neurons in the retina and optic tectum of frogs, and came to the conclusion that some neurons in the tectum of the frog are wired to combine elementary responses in a way that makes them function as "bug perceivers". A few years later David Hubel and Torsten Wiesel discovered cells in the primary visual cortex of monkeys that become active when sharp edges move across specific points in the field of view—a discovery that eventually brought them a Nobel Prize. Followup studies in higher-order visual areas found cells that detect binocular disparity, color, movement, and aspects of shape, with areas located at increasing distances from the primary visual cortex showing increasingly complex responses. Other investigations of brain areas unrelated to vision have revealed cells with a wide variety of response correlates, some related to memory, some to abstract types of cognition such as space.

Theorists have worked to understand these response patterns by constructing mathematical models of neurons and neural networks, which can be simulated using computers. Some useful models are abstract, focusing on the conceptual structure of neural algorithms rather than the details of how they are implemented in the brain; other models attempt to incorporate data about the biophysical properties of real neurons. No model on any level is yet considered to be a fully valid description of brain function, though. The essential difficulty is that sophisticated computation by neural networks requires distributed processing in which hundreds or thousands of neurons work cooperatively—current methods of brain activity recording are only capable of isolating action potentials from a few dozen neurons at a time.

_________________
"If you always put limit on everything you do, physical or anything else. It will spread into your work and into your life. There are no limits. There are only plateaus, and you must not stay there, you must go beyond them. "

It can also support a single file being that large too, unlike window's NTFS which the largest file only just recently moved to 16TB (it use to be 2.2 GB).

_________________

Yeah, but I didn't just copy/paste from Wikipedia

http://en.wikipedia.org/wiki/Brain#Info ... processing

_________________

That wasn't the point of that post.

_________________
"If you always put limit on everything you do, physical or anything else. It will spread into your work and into your life. There are no limits. There are only plateaus, and you must not stay there, you must go beyond them. "