0Box-Data Availability questions about losing and recovering data

@saswata
Saswata

  1. You said “5. images and videos when split up cannot be recovered in parts.”

A) Can you explain in simple language:
When would such a situation occur - what must the user do to end up in your described situation where a media file would split up and become non recoverable? And how to avoid such non recoverable situation?

B) Why is this situation only occuring for media files and not other file types?

  1. Can you describe one or two scenarios, where a user would not be able to download her previously uploaded file and lose access to it (things linked to the 0Chain protocol, disregard human error factors)?

  2. And what happens when a user pays for 100 days worth of storage, uploads a file, then forgets to extend her subscripton and comes back at the 150th day. How can the user access her previously uploaded file in that situation, any chance or is all lost forever?

  3. What is the reason, you are demanding regular renewals of storage contracts and not offering a second option for users to pay for a very long period at once (for example like Arweave)? Can you provide such alternative pay-once-for-long-term storage?

Thank you,

  1. I think Saswata was talking generally. I don’t think it excludes other file types, All files are split up (sharded) across multiple blobbers.
    A specific characteristic of media files is that even if you were able to access part of a file, it is typically useless without the file header anyway, but that might just have been a passing comment. If a file is encrypted by the user, then its basically unhackable, or lets say as secure as the wallet itself.

  2. Other than a catastrophic failure, the protocol is that a file will be available to download for the duration defined in the allocation. This depends on the requisite number of blobbers successfully being able to server their shards of the file. This ratio is defined in the EC settings, the higher parity, the more failed blobbers can be tolerated and files still served successfully. Blobbers are constantly challenged and incentivized to retain data faithfully. Also, If funds are depleted in the read pool, the file would not be able to be downloaded unless the read pool was topped up.

  3. 0Box App uses a fiat subscription and therefore the user doesn’t have to manually renew allocations etc. The 0stor layer is more for sysadmins and app developers to interface so they need their own systems in place to renew allocations before expiry.

  4. The max duration of an allocation will be increased with time as the network matures, so durations of a year or more may be feasible. But the longer this lasts, the more likely a user would be to forget about it anyway. Much longer than a year may not be feasible depending on the typical rotation of blobbers. There do not appear to be many use cases for perpetual storage, but if a demand was identified, the protocol is flexible enough to allow for different groups of blobbers. e.g. from slow access, (cold) storage using magnetic tape to ultra-responsive ssd/nvme-based storage.

@sculptex, helpful but you are vague on some answers, I am still not understanding the issue in which scenarios files become non recoverable. This is the entire meaning of 0chain to keep files safe and make them available. Please be very detailed in explaining when files may become non recoverable.

1. I cannot find an explanation for question 1. A) in your reply:

What must the user do to end up in a situation where her uploaded file becomes non recoverable? Please describe the steps that have to occur by the user or the protocol to render a file non recoverable.

Please provide more, two or three examples for scenarios where files can become non recoverable (ignore human errors) after they have been successfully uploaded.

2. You say “therefore the user doesn’t have to manually renew allocations”

How does it work then, how is a user making sure her file is being kept stored, what are the steps from the user to be taken? Just to stake and hold ZCN, that’s it?

A) WIll you provide some kind of a slider which will show the payment amounts required depending from storage duration? A simple transparent calculator within the app?

3. you say " failed blobbers can be tolerated and files still served successfully. Blobbers are constantly challenged and incentivized to retain data faithfully"

A) What happens if blobbers go offline/shutoff their services? How is the protocol handling this situation? Are you automatically replacing randomly such non functional blobbers?

B) When are blobbers being replaced automatically, in which cases?

4. "If funds are depleted in the read pool, the file would not be able to be downloaded "

A) what is the read pool?

B) Can you describe the scenarios where the read pool could be depleted, and by whom exactly? Is this a automatic or manual process?

5. “use cases for perpetual storage, but if a demand was identified,”

Most immediat use case, actually a huge enterprise/governmental one: By law, many corporations and governments are required to keep and archive files of customers for longer than 10 years (banks for example). Just one use case.

Or Data Lakes at large corporations are also very long term.´For data analytics, machine learning purposes.
Or Social Media and entire Websites, another longer term use case.

  1. EC (erasure coding) has 2 parameters, Data and Parity. The higher the Data, the more distributed the data. The higher the Parity, the more redundancy (but the less efficient it becomes).

Using EC10/15 (or EC10+5) as an example, 10 Data Shards with 5 Parity Shards, 15 Shards Total.
Any 10 Shards from the 15 can reconstruct the data, or to put it another way, any 5 shards can fail and the data can be reconstructed from the remainder.

Each storage contract with multiple blobbers can have customized EC settings on a protocol level. (On the 0box app, this may be fixed). These contracts are called allocations. They can be extended in duration at any time before expiry.

So the reliability depends on enough blobbers (the Data figure in the EC ratio) being able to serve the file. So any more than the Parity figure of failed blobbers will result in being unable to read data.

  1. The 0box App uses fiat payment, the zcn locking, allocation renewal etc. is all handled in the background, you just keep your fiat payment regular.

From the 0Stor angle, where you obtain zcn and stake/lock using CLI tools or interfacing API, there is no GUI, hence no slider, but protocol and tools allow you to query prices etc. Someone could build a GUI slider to interface with these.

  1. In case if data loss, say by a particular blobber, a repair function can reconstruct data from existing, still working blobbers. (The failing blobber compensates for this).

In case of total blobber failure, blobber replacement isn’t currently facilitated, but this may be something that could be looked at in the future. Normally, the user would take a fresh contract with proven more reliable blobbers instead.

Such failed blobbers would be losing their stake so it would likely be financially better for them to fix their server to pay for repair (from the other blobbers) than to quit altogether.

  1. A read pool sits alongside allocations and pay for reads. It is a contract with the blobbers to play for x amount of GB of reads at a certain price.
    So if price is 0.001 zcn/GB and you lock 1 token to the read pool, it will have been depleted when 100GB has been read. A client using 0stor can check the balance at any time. When using 0box, a generous read pool will be created along with allocation

An option also exists for to have the reader to pay for reads.

  1. 0chain is not intended to offer direct solutions to all situations. This niche could easily be achieved by a provider layer using 0chain underneath.

I call it niche, because a huge corporation, bank or government wanting to employ this technology at scale would be better to license the technology and use it on their own network (0stor private).

That way, they wouldn’t be dealing with tokens or external blobbers.

@saswata @sculptex

“Using EC10/15 (or EC10+5) as an example, 10 Data Shards with 5 Parity Shards, 15 Shards Total.
Any 10 Shards from the 15 can reconstruct the data, or to put it another way, any 5 shards can fail and the data can be reconstructed from the remainder.”

“any more than the Parity figure of failed blobbers will result in being unable to read data.”

"In case of total blobber failure, blobber replacement isn’t currently facilitated, "

These are very, very troubling security flaws in my view.
Please discuss with your team, in my view this is a single failure point, you should enable a smart algorithm that replaces automatically failed blobbers or storage nodes.

Who needs your static blobber system? Where is the benefit? We are in a blockchain and machine learning era, you should be able to implement smart algorithms that deal with such worse case scenarios in a automatic and swift fashion.

It is very dangerous. You are servicing enterprises and users with SECURITY in mind.
But you are allowing for the worst case to occur - BEING DEPENDENT on blobber’s “good behavior” and if only a handful (depending on your config EC) fail, you lose your precious data.

You should get rid of such dependencies, corporate users will not like that at all. At the end it will be like AWS.

**Where is the difference in AWS failing five of their servers or you, failing five of your EC configured nodes?**

**

Please reconsider and implement algorithmic, automatic blobber replacements in case blobbers totally fail or become unreliable.

**

You will have hundreds of blobbers - link them algorithmically,randomly, just like Internet Traffic is routed in a smart way to avoid congeston and achieve best fastest routes, you should do smart routing among blobbers to replace for failed ones.

Thanks

You do not seem to understand the technology. Your question “where is the difference in AWS failing five of their servers” shows a lack of understanding, AWS and other cloud providers typically triplicate the data, so that is effectively 3 server failures, (a tolerance of only 2).

As I stated, the EC ratio is completely user-configurable. EC10/15 was just an example. For sensitive data, you increase the Parity ratio to increase the number of failures that can be tolerated.

You asked for (extreme) examples of where the system would be unable to read user data, then proceeded to criticize from an angle without a clear understanding of what you were talking about.

For example, you mis-categorized these as security flaws. What you described are data-integrity flaws. Then you said corporate users would not like it, that it would be like AWS, yet many corporates trust AWS implicitly. I already stated that huge corporations would be more likely to license the technology in any case.

You simply don’t seem to grasp that the blobbers are incentivized to serve data and respond to challenges and risk losing their stake for persistent poor performance. If blobbers are suitably distributed, the likelihood of multiple blobber failure is actually much lower than with cloud because the blobbers are independent of each other.

If you are trying to explain poor blobbers as an attack vector to sabotage the network then say that, it is a different matter. Careful Blobber selection can be used to avoid ‘farms’ or multiple blobbers at the same location, but it would be an expensive attack to take because each different allocation will essentially have a random selection of blobbers. (Currently blacklisting of untrusted blobbers is not supported but whitelisting is, so you would just need to whitelist all trusted blobbers).

The miner challenge protocol constantly and frequently challenges the blobbers. The blobbers do not get paid if they do not pass the challenges. The challenge results can be monitored in real-time on chain so potentially under-performing blobbers can be highlighted. For large/medium corporations using the public 0stor they can monitor and decide what they want to do with this information, but as I described, they reliability of a well distributed blobber set is already an order of magnitude higher than any existing cloud provider.

I suggest reading the whitepapers to familiarize yourself with how all this has been carefully thought out and balanced between the staking, challenges and rewards.

“If you are trying to explain poor blobbers as an attack vector to sabotage the network then say that, it is a different matter.”

No, this is just one example. But I am not interested in the question of WHY blobbers might get offline or corrupted.

I am interested in what happens AFTER such an event.

And what I like to propose is that 0Chain protocol itself handles such situations AUTOMATICALLY, in that it REPLACES failed blobbers immediately.

Right now replacements has to be done manually by the user of your service. This is not acceptable in times of automation and machine learning.

Your network should handle such situations where blobbers fail automatically and replace them by predefined conditions.

The user of your service could for example define manually the conditions they like to have for blobbers. for example that they like to have blobbers only from jurisdictions in the USA or EU.

But as soon as conditions are clear, the network should handle replacements automatically, whenever there arises a need to replace failed blobbers (for any reason).

Do you understand my point? @saswata @sculptex

I understand your point but it is moot for reasons already explained.

@sculptex

is it? Right now you are implementing your service in such way that the enterprise clients have to manually re-adjust blobbers whenever there are failures.

As long as you do not automatise , it is not moot but pending issue in my opinion.

Or explain again why it is moot

It’s moot because the client has the control over EC ratio and duration of allocation contract.
They can renew (extend) before expiry if they are happy
If there are several failing blobbers, they will choose different blobbers on a new contract instead.

I totally understand how it works now. I am saying in my opinion this is not satisfactory for user friendliness and not satisfactory for data-security reasons.

Let me state again what should be the case to have a suprerior technology in the market:

this process you described above “If there are several failing blobbers, they will choose different blobbers on a new contract instead.”

This, exactly this, should be fully automatised.

My opinion is, as long as you keep this process manual, your tech will be inferior and not enough to satisfy data-security.

Ask your customers: Are you happy to have lost important company documents or client data because you did not pay attention to the quality of your EC configuration?

Or do you like 0Chain protocol to automatically solve EC ratio conflicts for you?

Answer is clear. @saswata

The user can delegate the renewal and monitoring of data to 3rd parties if they choose, that is up to them, but they will be giving the keys to that 3rd party.

There is no entity at present that would facilitate what you suggest. Miners constantly challenge blobbers with for (pseudo) random chunks of data without user intervention, this is a trustless process and a user can monitor these blobbers challenges without any intervention, (e.g. having to wastefully actually download chunks of data themselves). This is deliberately a very ‘light’ process on the miners, yet statistically very reliable for overall blobber ‘health’.

Basically, it is a different (and ingenious) approach to this entire situation and no matter how much you insist that your requested feature is implemented, it is moot in the face of this alternative approach to ensuring data integrity.