Loading Profile...
Wuala Traffic for 1 GB ?
Hello!
Does anybody know how much traffic is generated when you upload 1 GB ?
Including maintaining and spreading the fragments ?
As many people would like to use Wuala for backup and have a slow upload rate due to asynchronous connections (> 1 MBit down, but most 1 MBit up) it may take a while until everything is uploaded and maintained.
Example:
Many people use Outlook with PST-files where a small change results in the backup of the complete PST-file. Also a few pictures, little music, etc. etc. - let's say a weekly change of 2-3 GB.
If a user synchronizes his data with Wuala every Sunday, is one week enough to upload and maintain all the data on Wuala before the new changes need to be uploaded the next Sunday ?
Does anybody know how much traffic is generated when you upload 1 GB ?
Including maintaining and spreading the fragments ?
As many people would like to use Wuala for backup and have a slow upload rate due to asynchronous connections (> 1 MBit down, but most 1 MBit up) it may take a while until everything is uploaded and maintained.
Example:
Many people use Outlook with PST-files where a small change results in the backup of the complete PST-file. Also a few pictures, little music, etc. etc. - let's say a weekly change of 2-3 GB.
If a user synchronizes his data with Wuala every Sunday, is one week enough to upload and maintain all the data on Wuala before the new changes need to be uploaded the next Sunday ?
Follow this discussion to get notifications on your dashboard.
Get Satisfaction loves Zappos because they care about customer service.
-
Inappropriate?This usage of changing files like those PST files is not an intended use of Wuala, since it takes a few days (maybe weeks?) to spread your files in the P2P cloud.
It would be better if you could find a way to archive for example all mails from 1 week into 1 "container" and upload this static file instead of uploading a growing file with high reundancy each week.
1 Week however should be enough to ensure not only that the files are on the server but that they are also in the P2P cloud - maybe not the famous "99.999%" reliable, but surely sufficient to not generate too much traffic on Caleido's end.
With a network of 25% online storage nodes, you'll need 5.1something copies uploaded to ensure 99,999% uptime of your files - additional to the 1 initial upload to the server. -
Inappropriate?I know, but I cannot change Microsoft :-)
--
ok, so 1 GB equals ~5-6 GB Traffic
that could work in one week
so all that's left is a working synchronisation with Wuala
I’m hopeful
-
Inappropriate?When spread the Wuala server the files?
I think that was planned.1 Comment
Add a comment
-
The next thing is the Open Source Expo, I think they are polishing the code for that, and then are again full time back to issues in Wuala. -
Inappropriate?I read somewhere that every 100 MB of data gets turned into 150 MB so your 1 GB will be turned into 1.5 GB and then uploaded.
I know that the google tech talk said that due to the erasure coding that your file only needed to be replicated 5.17 times to ensure reliability equivalent to replicating the data 24 times (99.9% reliability) assuming 25% availability for each node and also assuming that each file is split into 100 fragments. I guess the Wuala team settled for less reliability if it's only replicated 1.5 times or maybe each file is split into much more than 100 fragments to offset that.
Does anyone know how many fragments each file is split into or is it seperated into X KB chunks (X possibly being 512 or 256).
Would someone from the Wuala team please confirm this and also shed some light in regards to what reliability you are aiming for.3 Comments Add a comment
-
afaik it's a fixed amount of 100 fragments/file, wich is also a problem with small files + huge files (> ~5 GB) - I hope "they" redesign this sometime soon.
-
Thanks Bugreport. This sounds like it's very inefficient for small files like pictures because you would end up with 100 fragments each being only 5KB in size. The meta data would account for quite a large percantage of the storage for such files.
By redesigning this portion, Wuala should be able to increase the redundancy without actually increasing the amount of storage that is used. -
Small files are not fragmented to avoid the overhead of small fragments.
-
Inappropriate?A file is fragmente into about 100 pieces. After that, erasure codes are used to generate about 500 fragments out of this (the exact values can vary.
The first 100 fragments are uploaded to our fragment servers (as a backup and to bootstrap the file) and all fragments are stored in the P2P network.2 Comments Add a comment
-
Thanks Roger. You mentioned above that small files aren't fragmented, how large does a file need to be for it to be fragmented? (eg greater than 500 KB?).
What is the reliability of small files if they are not fragmented? (eg. 5 times redundancy?)
Have you considered making all the fragments a fixed size? (such as 512 or 256 KB etc) So for any file that doesn't generate 100 initiall fragments, they would just be replicated a little more in order to reach the required reliability. -
Reliability should be the same I guess... My guess is that Fragments have a min. size and below that there are just less fragments generated to prevent massive overhead data.
Fixed size fragments would be great, especially for huge (and I mean huge!) files like 100GB backup archives, Blu-Ray Disk ISOs and similar things.
On the other hand this would mean a rewrite of a vital part of Wuala, maybe something that HAS to be done some day but they don't know when they have the time or what's going to break... -
Inappropriate?Hey Bugreport, with less fragments, reliability is actually reduced.
Using the same example as the one used in the google tech talk (100 fragments, replicated about 5 times, node reliability at 25%), with 100 fragments replicated about 5 times produces 99.9% reliability but if you have a small file that's split up into only 2 fragments and still replicated 5 times then the reliability is only 75.6%.
So small files that aren't split up into enough fragments would need to be replicated more in order to achieve the same reliability.1 Comment Add a comment
-
...or Wuala is smart enough to put fragments of files with less fragments on high availabe storage nodes, like mine that has 90%+ uptime already.
I don't know, 25% is just an assumption - would be interesting to know what the current Storagenodes onlinetime looks like (I guess it's still growing since I get quite few fragments compared to the start of the beta atm - so I guess quite a few people are starting to trade storage) -
Inappropriate?Can someone from the Wuala team answer my original question concerning resulting traffic from uploading 1 GB ?1 Comment Add a comment
-
"A file is fragmente into about 100 pieces. After that, erasure codes are used to generate about 500 fragments out of this (the exact values can vary.
The first 100 fragments are uploaded to our fragment servers (as a backup and to bootstrap the file) and all fragments are stored in the P2P network." --> ~5 times the 1 GB = ~5 GB! -
Inappropriate?I've been thinking...
This means that the P2P network must have about have 5 times the space I have in Wuala just to support my files, right?
I wonder if that much space is available in the P2P network... Sure that most of the people don't use all their space in wuala and that some files are stored multiple times. But still... -
Inappropriate?The wuala network is still pretty empty. I'll get worried when i reach 100% shared space (currently 6.3% with very slow incline). There have been requests to see some figures about total network capacity with no avail.2 Comments Add a comment
-
I'm also @13 GB and very slowly increasing (2-4 new *.db files/day)
-
45GB on my server. 93% uptime.
-
Inappropriate?so to backup 50 GB of data it would result in 250 GB (upload + maintainance) of traffic :-(
At a maximum upload speed of 1 MBit (= 128 KB/s) - and assuming that you don't use your internet connection any other way!! - this would take almost 3,5 weeks (!!!) of 24/7 uploading to make sure your data is safe !!!
And also assuming you do not change any data in this time period!!
OH NO.....
Now I'm really frustrated....
I’m really concerned about the future of Wuala
3 Comments Add a comment
-
No, your data is safe after the initial 50 GB already! It's faster to download after the few weeks however, since you have not only the server but also other sources for the parts + more parts to choose from.
-
I know but this would assume that all the fragments are available and none of the nodes goes offline...
-
No, the 1st upload goes to the server, which is online 100% already.
-
Inappropriate?The inefficiency lies in the fact that files are only split into 100 fragments.
If files were split into 300 fragments then this would result in a 10% space savings while still achieving the same reliability. So in the case of backing up 50 GB, you would save 25 GB of bandwidth because you would only upload 225 GB instead of 250 GB.
Fixed fragment sizes are starting to look even better now because large files would be replicated less but still be as reliable because they are fragmented more. This would save bandwidth for us and save space for Wuala.5 Comments Add a comment
-
Why the "same availability"? There are 500 _different_ fragments computed, as far as I understand...
-
I thought there are 100 fragments that will uploaded up to 5 times.. . Employees - help! :D
-
Well, that would mean erasure codes are useless... since you'd need all of the 100 fragments, not 100 of the 500 out there!
-
Wuala splits a file into 100 fragments which are then used to generate 500 fragments (not replicated 5 times but it does use 5 times as much space).
With my example of 300, each file would be split up into 300 fragments which would be used to generate 1350 fragments (This would take up 4.5 times the amount of data instead of 5 times. So a savings of 10% while achieving the same reliability) -
To clarify: there are about 500 different fragments and you need any 100 of them. And again, 500 and 100 are NOT the real values. This differs from file to file.
-
Inappropriate?I completely agree!
And it would be great to set in options how many times to upload a single file because in case of emergency (ex. file deleted and need to restore) I don't care about speed but the file has to be 100% available whenever I need it !!1 Comment Add a comment
-
Assuming the average user in Wuala is online 25% of the time, your files are available 99.9% of the time (or unavailable for about 9 hours in a year).
To achieve greater reliability, the file doesn't need to be replicated more but instead split into more fragments. As an example, if each file is split into 200 fragments (instead of 100 like it's currently done) and then used to generate about 1000 fragments to upload (this is still a 5 times factor) then the reliability would increase by 10 times! So by splitting into 200 fragments then each file is guaranteed to be available 99.99% (or unavailable for about 53 minutes in a year) -
Inappropriate?wuala garanties the aviability of files after the "normal" upload to their servers is finished. The uploads into the p2p-cloud is to increase the speed. Your files are stored complety on wuala servers (and backupservers) in the first run.
For myself I would like to see a function that the wuala-client "fall back" to a dht-system in case the wuala-server goes down for any reason to garanty the aviability also in that situation. This is not true at the moment, so the aviability will be less then 100%. Maybe 98-99% - therefore, its the aviability every netservice can provide...
Employees pls. correct me when I am wrong.
2 Comments Add a comment
-
I hope that the servers are only required when uploading files. I hope that my files will still be accessible when the Wuala servers are down.
Any insight from the Wuala team would be greatly appreciated -
Our servers are required for things like identification, search, downloading small files, meta data and so on. But the fragment server are not needed as long as there are enough fragments in the P2P cloud (this is not the case in the bootstrap phase when you just finished uploading the first 100 fragments to the servers).
-
Inappropriate?If wuala's servers are down i don't think you can browse wuala. Also, there has been speculation that someday (When wuala is very stable) your data would not be put on wualas servers at all.2 Comments Add a comment
-
I don't think this would happen, since it's really cheap to buy storage but traffic is quite expensive - Wuala is more of a traffic-saving mechanism thatn a reliability/storage thingie.
-
true
-
Inappropriate?Ok, so the "100" fragments are turned through erasure coding into "500" fragments. But if you have a 1MB sized file to start with, how much data do you in fact have to upload to "the cloud" to have all the 100+400 fragments out there?1 Comment Add a comment
-
That is answered above. You would have to upload 5MB
EMPLOYEE
EMPLOYEE
EMPLOYEE
EMPLOYEE
EMPLOYEE