Whino needs a new Storage Solution

[Rhino]

Hey again guys.

Been around 5 months since my last upgrade here: https://www.realitymod.com/forum/f329-h ... grade.html

And since then, the SSD I got, the Corsair 120GB Force GT SSD has slowed down quite a bit since when I first got it, due to me using up every kb of it a few times over and its now not much faster than a normal HDD

So I think sometime soon I'm going to need to format it to get its speed back but I'm also thinking I should change the Drive setup to try and avoid this being a problem in the future.

The main problem I've had is that since my SSD has been my main drive with Windows and my other main programs on it its been used up really, really fast. Even before I started working off of it It was mostly used up with only ~50gbs of free space.

First I think its best I explain my situation and what I want to get out of an a drive so hopefully you guys can understand where I'm coming from. The main thing I want is really fast write times as I'm working with very large .max scenes, currently 150mbs large although will most likely go above 200mbs in the near future and every time I save, which is every 10mins or so I also save a backup "copy" of that stage of development so I can easily revert back etc (saved my *** many times over) so basically, each 10mbs I use up another 150ms to 200mbs of HDD space, which I save at least 50 copies each day, ie, around 10gbs of backup data a day... At the moment every few days I'm moving all my backups from my SSD to my 1.5TB HDD. The problem with this thou is that the data area on the SSD has already been used and even with nothing else on the SSD, I only have 12 days tops of working with the SSD with full speed as after that, I'm reusing old previously used space...
Read times are not a big issue for me although having the editor, max and PS with quick loadup times is nice, its not very critical to my work where write times are since to save a large max scene (twice) takes quite a long time, especially on a normal HDD.

So ye, ideally I'm looking for some kind of solution where write speeds will be constantly high, or at least high for a long period of time with ideally high reliability and lowish price tag. Would rather not have to fork out 300+ squid on a new SSD but if that's the only option I'll consider it.


So ye would really like to hear your ideas on this

Cheers!


EDIT: Also just had a thought. Dose anyone know of a program that can detect if a file is modified, then if true, it automatically makes a copy to another Drive, not overwriting any other backup copies that are already there and lables the backup by number or date/time?

If I could find a program like that, then I'm thinking that working off a SSD wont be so much of a problem, since I will only be saving to the same part of the SSD I was, with saving times being much faster since I'm not saving two files at the same time, and copying to the HDD wont affect my work as that will be happing automatically in the bg while I'm working?

[LITOralis.nMd]

What price range?
DO you have a home server or extra PC and a gigabit router?

I'll write up a better reply later.
Windows backup, disaster recovery and drive image software
Data backup and disaster recovery software for workstations
Backup Software - File Recovery | Norton Ghost
http://www.areca-backup.org/documentation.php

Norton Ghost and Acronis lead the way and are the two I've used at work over the years., ShadowProtect is a new kid on the block, IDK much about it.
Areca is free but has a learning curve. Areca might work for you.

info to read:
Incremental backup vs differential backup: difference and benefits of each one

[MaSSive]

Take a look at this. If you have Z68 chipset and juice to get new SSD we can work on that.

If not lets move to this one.

I'm not using much backup software but I stumbled upon this one and it sounds like a good one or exact what you need, and is free as a plus. It has big bother which will do even more, but costs money.

You must understand that features that you require will cost extra money in some point and is not quite easy to setup. I'm thinking of cache drives, raid arrays which is very close to ones used in servers.

[Rhino]

I'm not looking for backup software per se, more of something that will create revisions automatically every time the main core file is modified, into its own folder, a bit like this:



Afghan_Buildings.max being the main, most upto date file with all the others being steps saved as I've gone on, which I've done with max'es "Save Copy of File" and it automatically numbers them, based on the other files with the same name/numbering setup.

And ye I do have a extra PC on the network, although I've just used up the HDD on it I was backing up on so will need to go a new one for it

Backing up onto the net/cloud isn't an option, too slow internet and no need.

[Psyrus]

IMO raid5 would work reasonably well in this situation (block striping with distributed parity) as you're looking for redundancy plus large transfer rates on contiguous files. The access times won't be any/much better than on a single 7200rpm drive, however your transfer rates will basically be N(T-O) [number of drives * (transfer rate of a single drive - raid overheads)]... plus with raid 5 you can have a single drive failure without dataloss.

Throwing more SSD space at the problem is silly as your capacity requirements far outstrip any kind of meaningfully cost effective SSD solution. What I would personally do (especially since I always have tons of extra storage space since TB drives are cheap as chips these days):

1. Create a script (in whatever language you're proficient in) to handle moving files from SSD save-space to raid5 storage area.
2. Continue saving current files to the SSD
3. Script does its 5 minute check, sees new files & copies to new drive, deleting SSD version once copy is confirmed - Tooltip display to confirm move has taken place
4. Your naming scheme can be implemented very easily within the script
5. Daily manual check to verify script hasn't screwed things up, until you're 100% comfortable with the script.

[Rhino]

IMO raid5 would work reasonably well in this situation (block striping with distributed parity) as you're looking for redundancy plus large transfer rates on contiguous files. The access times won't be any/much better than on a single 7200rpm drive, however your transfer rates will basically be N(T-O) [number of drives * (transfer rate of a single drive - raid overheads)]... plus with raid 5 you can have a single drive failure without dataloss.

Ye, was considering a raid setup but wasn't sure which one. Buying 3 new HDDs thou would be quite expensive

Throwing more SSD space at the problem is silly as your capacity requirements far outstrip any kind of meaningfully cost effective SSD solution. What I would personally do (especially since I always have tons of extra storage space since TB drives are cheap as chips these days):

1. Create a script (in whatever language you're proficient in) to handle moving files from SSD save-space to raid5 storage area.
2. Continue saving current files to the SSD
3. Script does its 5 minute check, sees new files & copies to new drive, deleting SSD version once copy is confirmed - Tooltip display to confirm move has taken place
4. Your naming scheme can be implemented very easily within the script
5. Daily manual check to verify script hasn't screwed things up, until you're 100% comfortable with the script.

Yes this is what I was thinking too, although hoping I can find an existing program to do it first if there is one instead of pleading someone else to write me a script to do it as I have no hope in doing that hehe. But ye, what I'm basically looking for and thinking too, other than probably not using a raid setup for it to copy onto since the copying process wouldn't interfere with my working, how ever fast/slow

[Psyrus]

Ye, was considering a raid setup but wasn't sure which one. Buying 3 new HDDs thou would be quite expensive

Well the exact same principle can be applied to a single drive (existing or new), it just wouldn't have the advantages of a higher throughput and fault tolerance like the raid 3/5 would.

Yes this is what I was thinking too, although hoping I can find an existing program to do it first if there is one instead of pleading someone else to write me a script to do it as I have no hope in doing that hehe. But ye, what I'm basically looking for and thinking too, other than probably not using a raid setup for it to copy onto since the copying process wouldn't interfere with my working, how ever fast/slow

I realize you're rather busy, but a great place to start is with AutoIt for windows scripting if you ever do want to get into it. The APIs and guidance are excellent and it's very, very easy to learn and work with. My brother with no formal training was able to pick it up and do some pretty impressive things with it after just a few months of tinkering. It's a rather basic language and you can't really expect to be able to "program" huge projects after becoming proficient at it, but for its domain of purpose (windows scripting/automation) it is a fantastic tool to have under one's belt.

[MaSSive]

I dont think he needs RAID5. If you read this, and compare to what Rhinos requirements are you will see that he needs complete opposite of that.

Speed
There is a decrease in write speed due to calculations that have to be made before data is written to the drives. If you want to increase read/write speed and have the benefits of RAID 5, you need to implement RAID 10. Disk Usage
The loss of disk space is basically 100 divided by the number of disk drives. With 3 drives, there is a 33% loss of disk space. With 5 drives, there is a 20% loss of disk space. Requirements
RAID 5 is more expensive to implement. You will need additional drives and a RAID controller. The cost of implementing RAID 5 could be in the range of $1000 to $5000 depending on the total number of drives, type of drives and controller required.

^ That should be enough to sum things up.

Better solution for improving IO operations is cache drive ( SSD ) or RAID 10, and again if you look in those two I think single SSD for caching proves to be the cheapest and better solution.

[Psyrus]

MaSSive;1723636']I dont think he needs RAID5. If you read this, and compare to what Rhinos requirements are you will see that he needs complete opposite of that.



^ That should be enough to sum things up.

Better solution for improving IO operations is cache drive ( SSD ) or RAID 10, and again if you look in those two I think single SSD for caching proves to be the cheapest and better solution.

If you'd read my solution properly, you wouldn't be saying he needs the complete opposite of that at all. The reason I selected 5 over 10 in that case is exactly because 5 has been more common on the consumer level boards' controllers that I've seen, but I did some reading and 10 is actually available on one of the extra controllers on Rhino's mobo. Furthermore if reasoning of 'loss of storage space' is being used as a downside for 5, 10 loses even more space comparatively - however I would definitely agree that when the choice is cost-neutral between the two technologies that 10 should be chosen.

Could you detail the exact setup of the proposed SSD cache-drive system? I'm interested in your implementation.

[MaSSive]

Using any of those arrays on integrated controllers or in soft level raid is not recommended. Getting additional dedicated raid controller and additional hard drives to achieve what he needs is quite pricey and I dont think he should go with it at all.

Only thing needed here is better read/write operation which can be achieved with cache drives. Imo these are much better solutions than using any kind of raid array for improving performance, considering the cost, power efficiency, noise, and space it occupies. Complexity of setup and maintenance is one more reason why not.

I cant be absolutely sure that this is the real way to go, but by reading numerous articles Im concluding it should be.

Remember Ready Boost? It never worked right for me, but now with new cache drives I think this is going to be changed. Read this article and please say what you think. My opinion is that this is a bulletproof solution in this case, considering all of the above reasons I mentioned.

Drive that I would recommend for this is Accelerator SSD from Corsair.

[Rhino]

The cache drive is interesting but its basically a Hybrid from my understanding, with basically learning what your common used files on your HDD are and putting them on the SSD so it can access them faster.

Thing with that thou from my understanding, is that new files your saving etc, will just have normal HDD speeds, and mainly things like booting windows and commonly used programs will be faster, but not much more and I'm guessing it will also have the same problems as an SSD as slowing down over time when it gets more used up, although might not happen so much?

Not sure its the solution to my problems.

[MaSSive]

Well then keep the system on SSD and build raid 0 ( cheapest of all and has the best performance ), and save your data on it. Only disadvantage of this raid setup is fault tolerance. If it fails you could lose data and backup is a must, but you already have backup in plans.

What is it? RAID 0 uses a method of writing to the disks called striping. Let's assume you have a server with three drives of 500MB, 1 GB and 2 GB. Normally a server would treat each of these drives individually. By incorporating striping, the system would see all of the drives as only one drive for a total of 1.5GB. Why only 1.5GB? Because the maximum used on each drive can't exceed the size of the smallest drive. So it is best to have drives of the same size when using striping.
When the system writes data to the disk, the RAID 0 striping kicks in and automatically distributes the data across all three drives. Part of a file (chunks of data) will be written to the first drive, the next part to the second drive, the next part to the third drive and then it starts all over again until the entire contents of the file have been written.SpeedWhat this does is increase the speed of the reading/writing process. If you have two drives on your server, it increases the speed by about 25%. If you have three drives, it increases the speed about 33%. When you consider that the main task a server is performing is reading and writing data, any increase in speed is highly welcome.Disk UsageBesides increasing speed, the other benefit is that the drives can be of different sizes.
Because RAID 0 only writes the data once, it does not achieve data redundancy. If one of the drives fails, the entire system has to be restored because all files are split or striped across all drives. Because there is no data redundancy, there is no loss of disk space.RequirementsAdding a RAID controller and more drives.

I said keep the system on SSD because I dont recommend placing it on raid 0 array for reasons quoted above.

Here is a quite good review of SSD vs raid10 ( raid 10 combines features of raid 0 and 1 - speed and safety )



Here is quick overview of raid 5 vs raid 10. Keep in mind that any of those two ( raid5 or raid10 ) will cost a lot more money, to be properly setup. 3-5 fast hard drives and a dedicated hardware raid controller. Plus a power consumption, needed space, heat and noise.

[Rhino]

I was thinking more of normal HDDs mixed with my SSD, with SSD having the stuff I need to run fast off and main place for saving the main file, then revisions made automatically by some program or script onto another HDD and also running most of my other programs and keeping most of my other files on the HDDs too?

RAID is either too unstable or doesn't really offer what I need from what I can see, apart from possibly RAID10 which looks to expensive for me?

[MaSSive]

What I would do in this case, considering all said above and looking first at price and complexity of setup, is to use SSD for system only and use another separate drive as PS scratch disk and data storage for 3Dmax files.

Id go with with super fast Seagate Cheetah 15k.6 for that or OCZ Vertex 3 SSD. Again if you compare those two you will see that this best SSD will outperform even fastest HDD on market, and the price is on side of SSD again. Durability is on HDD side though, but I guess you cant have both.

Keep in mind that you will need additional SAS controller to run Cheetah and to achieve full speed of this SSD you will need additional sata III controller. It will work on sata II board ( its backward compatible ) but with reduced speeds.

[LITOralis.nMd]

ARECA IS SPECIFICALLY MADE FOR THIS....

So is this http://www.nongnu.org/rdiff-backup/

What is it?

rdiff-backup backs up one directory to another, possibly over a network. The target directory ends up a copy of the source directory, but extra reverse diffs are stored in a special subdirectory of that target directory, so you can still recover files lost some time ago. The idea is to combine the best features of a mirror and an incremental backup. rdiff-backup also preserves subdirectories, hard links, dev files, permissions, uid/gid ownership, modification times, extended attributes, acls, and resource forks. Also, rdiff-backup can operate in a bandwidth efficient manner over a pipe, like rsync. Thus you can use rdiff-backup and ssh to securely back a hard drive up to a remote location, and only the differences will be transmitted. Finally, rdiff-backup is easy to use and settings have sensical defaults.

Rhino;1723615']I'm not looking for backup software per se, more of something that will create revisions automatically every time the main core file is modified, into its own folder, a bit like this:



Afghan_Buildings.max being the main, most upto date file with all the others being steps saved as I've gone on, which I've done with max'es "Save Copy of File" and it automatically numbers them, based on the other files with the same name/numbering setup.

And ye I do have a extra PC on the network, although I've just used up the HDD on it I was backing up on so will need to go a new one for it

Backing up onto the net/cloud isn't an option, too slow internet and no need.

[Rhino]

Ye, not sure if buying a new SSD is worth it since my current one was good and should be good again after a format.

What are the disadvantages of not installing your OS on an SSD other than boot times? I'm wondering if its really worth installing your OS on a SSD and if the space could be put to better use to other things?

ARECA IS SPECIFICALLY MADE FOR THIS....

Cool cheers, I didn't see it in is features list but I'll look into it more!

[MaSSive]

What are the disadvantages of not installing your OS on an SSD other than boot times? I'm wondering if its really worth installing your OS on a SSD and if the space could be put to better use to other things?

I can't empathize enough the importance of having the OS in your ssd.

It's just a quantum leap on system performance.

You can tweak it further for better performance though. Use this guide or this tool for it.

If you're thinking of placing system on HDD and using SSD for data Im not sure if that is the real way to go with it. You will have quite slower system and in time SSD will suffer again due to large read/write operations.

If you use SSD now as system drive and saving your work in same time on it, I think you should only get one more additional drive and use it as storage drive, and keep your system on current SSD.

As for ARECA that is quite pricey and used commonly in server systems. Not quite optimal if you look at price again for this setup. I assume you mean on this Areca

[Rhino]

LITOralis.nMd;1723685']ARECA IS SPECIFICALLY MADE FOR THIS....

So is this http://www.nongnu.org/rdiff-backup/

I've looked into Areca and this rdiff-backup some more and I still can't see the main feature I need and I just want to make 100% sure that there is no confusion here for what I need and what these programs offer.

I see that they both have "Keeps increments" features but I don't see any feature in them that will 100% automatically detect a modification in the main file instantly and then will automatically back it up with an increment onto another drive/folder. From what I can see, they both require to be told to manually backup or at best, will backup based on a set time?

The detecting a modification in the file and then backing up if modified is a key thing I need here otherwise its pretty useless to me tbh so can you please assure me that at least one of these programs can do this and possibly point me into the right direction to setting it up?

Cheers!

MaSSive;1723700']I can't empathize enough the importance of having the OS in your ssd.

It's just a quantum leap on system performance.

You can tweak it further for better performance though. Use this guide or this tool for it.

If you're thinking of placing system on HDD and using SSD for data Im not sure if that is the real way to go with it. You will have quite slower system and in time SSD will suffer again due to large read/write operations.

If you use SSD now as system drive and saving your work in same time on it, I think you should only get one more additional drive and use it as storage drive, and keep your system on current SSD.

rgr cheers.

[LITOralis.nMd]

well then, how about these commercial out of the box solutions:

Paragon Backup & Recovery Home - Backup and restore partitions, hard disks or files
Scheduler and Script Generator Chain any set of operations into one job to automate the routine with built-in scheduler. Script generator offers advanced scripting options like remote and conditional execution, subroutines, repeatable iterations, disk/partition properties analysis, error management, and much more.

Windows Backup - Ridiculously Simple Backup Software - Rebit Inc.
Continuous Data Protection

Rebit 5 monitors changes to your files and backs them up within moments. File versions are maintained so you can go back in time to any previous version.

[LITOralis.nMd]

regarding the SSD, use it as the OS drive and scratch drive and build a HDD array for storage.
Some of these softwares I suggested would allow you to create two levels of backup archives, so you could say, keep your 5 most recent backups on the SSD, when you make a 6th backup, the oldest backup would be moved to the hddin a asynchronous file transfer. This would give you the speed you are looking for and the IF(version change)THEN(backup) feature.
It would cause alot of read/write wear on the SSD, but buy the time you burn out the SSD we'll all be buying Sata6GB SSDs for US$.50/GB.