Building a private file sync cloud

I have been looking for the perfect file sync tool for a while now and I wanted to share my experiences. They will hopefully be usefull for someone…

What about Dropbox

Dropbox… For many people, it is one of the best. But for me, there is one thing which is a showstopper : why the hell do you need to reorganize your data to sync with Dropbox ? Why can’t you just say “ok, I would like to sync c:\myshareddata and c:\myprivatedata… and… c:\mymedia\myphotos”.

No… you are REQUIRED to put everything in whatever unique directory or share your all C: drive !

So I found Wuala…

Nice little java GUI, where you can select any existing directory. No need to move your files around. You can sync multiple computers. Wuala provide both sync and backup features, file versionning… In addition, It is base in Swisszerland so if you are allergic to store your private data in the US, it is a nice solution.

A bit expensive though and if you get an issue… don’t ask for help ! I did get a memory leak on Windows 8. And after I sent an email to the support, they did send me immediately a small patch… which did not solve the problem. Without looking further (they did not ask for any log files, diag informations whatsoever) they just said “Ok, it does not work, we’ll reimbourse you”…

Ok… that could mean one thing from my point of view : they really don’t give a sh… about their customer (and can afford reimboursing them) because the service will not last long…

And no client for my NAS…

I’ll eventually gave up, get my money back, and start looking for a new solution.

A quite original solution : “lend” you free disk space

ON my Synology NAS, I also found another Cloud Sync tool : Symform. It is a a real “Cloud” storage in that it is storing data on many customer PCs : you actually “lending” your own disk space for storing other customer data and get cloud storage space in return. You actually give twice as much as you get but I find this a very good idea.

I tried it on my PC for a while and it worked quite well. I then decided to install it on my NAS but, I don’t know why, upload time became soooooo looooonnnnng I just gave up. I think I will try this again when I get FTTH connection.

What about not using a public cloud service

I got a nice Synlogy NAS (a DS411) and I wondered : “Why can’t I make my own sync cloud ?”.

The first I tried was Synology own Cloud solution : Sync Station. But it is not build for high volumes. When I started the sync, 10 Gb of data became… 20 Gb because of the way the tool is backuping old version… even on first sync ! Silly… But I must admit I tried a while ago and they may have made some progress.

And then came Bittorrent Sync…

I discovered BT Sync a while ago as they were still at version 1.3. A very nice sync tools making used of Bittorrent protocol and allowing sync of multiple hosts in a quite simple way.

I did sync all my photos and music between my home PC and my NAS and it worked quite well. I also sync’ed my personnal file between my home PC, my NAS and my laptop. And finally, I convinced on of my coworkers to install it so we can easily share files.

Worked for a while… then, you don’t know while, it just get stuck at some point. You remove a partially sync file (you need to enable debugging to identify the file) and then, it restart.

I had a few corrupted file also. I was able to get a backup from the BT Sync trash… on another machine. BT Sync team, if you read this : why, when I delete a file locally, it does not got to local trash but only to remote trash ?

1.4 version, which is quite recent, brought a much nicer UI. You can easily share files…

2014-09-13 00_06_46-BitTorrent Sync

And I was hoping a bit more reliabilty. Alas… Still get sync hanging for no apparent reason, but in addition, BT Sync tends to lock the files which are open in MS Office… which then you can’t save !

And right now, while I am writing this post, my NAS and home PC are both saying “OK, everything is now in sync” but my laptop is still pretending there is more than 1 Gb of files to sync…

How can you trust this…

Will the next solution be the good one

And while I was looking for some tips and trick about BT Sync and find solutions to my problems, I found a new sync tool, thank to this post : SyncThings. It seems that since Korben published this, they have made some progress and it looks like quite promising.

2014-09-13 00_12_53-Syncthing _ PCJS

Another important point : it is Open Source.

So I’ll try that and hopefully, it will cover my needs !

How I solved “E” status on a RAID0 volume for a Synology DS411

I recently had a problem with my Synology DS411 NAS and spend quite some time to solve it as information about this is quite sparse.

In the middle on the night (sigh), my DS411 started to beep because of an hard drive failure. It woke me up… I switched it off, thinking that I would disable ALL beeps and alarms in the morning so it won’t happen again !

In the morning, I restarted it. And although I was using a RAID0 array with 2 disks, I did not loose any data as the RAID0 array was still available but mounted as Read Only. I did a backup of data I needed… just in case the disk would really fail…

Looking at disks logs in DSM, it showed 11 write errors. Running a S.M.A.R.T. test from DSM did not report any issue. I then stopped the NAS, extracted the faulty disk, connected it to a PC and ran vendor’s test utility (full test) and also a full test using a very nice tool called Spinrite. No error whatsoever.

It seems that one of the two disks encountered a transient read / write error and that Synology software, for the sake of the data, decided to set an alarm and protect the volume, mounting it as Read Only. Which is, I believe, a very good thing.

This is where it becomes tricky : having the disk thouroughly checked, how to reset the alarm and remount the volume in Read Write mode ? Nothing found on Synology web sites, forums or on the web…

I finally opened a case at Synology who has been quite helpless… They made me reinstall DSM because I was using non official package source although it was obvioulsy an hardware problem. But they finally, at least, put me on the track of solving the problem.

The solution I found and describe here worked for me. But it may not for you. Unless you know what you are doing, have proper backup or are ready to loose all data on the disk, don’t apply that. I can’t be held responsible for the loss of data.

I connected to the NAS through SSH and ran the following :

Digging a bit more, I discovered that the “E” status in [UE] is very specific to Synology and hence not documented in standard documentation… but neither in Synology documentation ! As you guessed, it means Error.

So now, how to reset it ? And that is where Synology have been helpless… they did not immediately pinpoint the problem and gave me the solution although it is Synology specific and they rather told me to reinstall DSM and that they were not providing Linux support…

I did something… scary… But I did backup important files before. So, again, if this worked for me and not for you, don’t blame me !

I stopped the RAID volume and recreated it (using guidance from here.)  Obviously, the values in the command line need to be adapted to your context.

Just answer yes when it asked for confirmation because disks are already part on an array.

And TA DA !

The –create option “just” writes the configuration of the array, leaving all data intact. After, you can reboot or mount the volume manualy.

And all the data is there !

Hope it will help !