How to Put Data on the Sia Network
It has recently come to my attention that we don’t have much documentation about uploading files to the Sia network. It is our goal that the user experience becomes so intuitive that documentation is not necessary. Unfortunately, the user experience is not very intuitive yet, and so I’ve written this post to help fill the gap between what you can figure out intuitively from our current UI, and what you need to know to make the most out of Sia.
Step One: Synchronizing the Blockchain
Sia is foremost about security, and about not needing to trust anyone. This is more powerful than mere decentralization — instead of just having your data in multiple places, you have your data set up in a way that there is no single point of failure, including the humans in charge of protecting your data. It’s about having the freedom to know that you are in control of what happens to your data and nobody else.
This requires downloading and verifying a blockchain. For Sia, this means downloading about 6 GB, and it takes a few hours on an SSD, and a few days on an HDD.
As soon as you start Sia, it’s going to connect to the network and start downloading the blockchain. You will be able to see the progress as it downloads, and you will not be able to do much until the blockchain download completes.
Step Two: Get Siacoin
The Sia network runs on siacoins. You need them to upload and to download, and it’s how both hosts and miners are paid. Sia needs to use a native asset to automatically execute contracts — if we used something like the US Dollar, there would need to be a custodian who would have to approve every contract and every payment. That custodian would be a source of centralization. Cryptocurrency is a requirement for decentralized payments.
The easiest way to get a small amount of siacoin is through Shapeshift. You can convert any cryptocurrency to siacoin with minimal hassle. You can also trade for Sia on Poloniex, Yunbi, Bittrex, and a handful of other cryptocurrency exchanges. If you don’t have any cryptocurrency, you can start by getting Bitcoin.
Step Three: Creating Contracts
This is probably the most confusing part of Sia, and it’s something we’re actively working on improving. The biggest issue here is that blockchains aren’t scalable. If we had to put a transaction on the blockchain every time someone uploaded files, not only would uploading be very slow, it would also mean that only a few thousand people could ever use the network. We solve this problem using incremental contracts. They are a little complicated, but ultimately not so hard to understand.
Instead of creating a new blockchain transaction every time we pay a host, we create an incremental contract with the host that locks up our money. That locked up money has 2 payouts: one for you, and one for the host. Initially, it says that you get all of the money back and the host doesn’t get any money at all. But as we upload and download, we update the contract to say that the host gets a little money, and then more and more as we keep using the host. The scalability comes from the fact that we only need to broadcast the version of the contract to the blockchain. Over the course of a month, we can pay the host thousands or even millions of times, and in the end only a single blockchain transaction is created.
When you hit ‘Create Allowance’, your renter goes and forms 50 incremental contracts with hosts, though generally we just call them ‘Contracts’. You put some amount of money into these contracts which you then spend as you upload and download. The process of creating contracts takes about an hour, and cannot be interrupted. There is also no indication of progress in v1.2.2, however we will be adding a progress indicator in v1.3.0.
The UI has a nice indicator for your balance, which should say something like ‘220 SC Spent / 500 SC Allocated’. What this means is that the total value of the contracts that you formed is about 500 SC, and that you’ve got about 280 SC left to spend on uploads and downloads. The full 500 has been subtracted from your wallet, but you can get up to the 280 SC back if you don’t spend all of the money in the contracts.
You may notice that immediately upon creating your allowance, it says that some of the money has already been spent. This is because you have to pay transaction fees, and contract fees. Those 50 contracts are each a transaction which goes onto the blockchain, and that means you need to pay fees. The host is also going to have to put storage proofs onto the blockchain, and you pay the host up-front for these transaction fees as well.
The contract system on Sia operates similar to a utility bill. You pay for 3 months in advance, and then whatever you didn’t use is returned to you at the end of the three months.
If you use the UI, the number of contracts you form is fixed, and the hosts you choose is selected automatically based on a bunch of factors, including price, age, and uptime. The UI is not intended to be for power users, so if you want more customization, you’ll have to use either the command line or the API. The command line can be operated from the ‘Terminal’ tab in the UI. Run the command renter -h to see a list of options. The API documentation can be found here: https://github.com/NebulousLabs/Sia/blob/master/doc/API.md
Step Four: Uploading Files
You can upload a file using the ‘Upload Files’ button. You can upload just a single file, or a whole folder. Currently, the atomic size for Sia is 40MB. We are working on improving this, but currently it means that if you upload a bunch of tiny files, they will upload as though they were a full 40MB each. We therefore recommend zipping up small files before uploading them to Sia.
Files are uploaded at 3x redundancy to 30 hosts. What this means is that 30 hosts end up with fragments of your data, and out of those hosts any 10 are enough to recover the original data. The magic that makes this possible is an algorithm called Reed-Solomon codes, and it was actually invented in 1960. The result is that as long as your hosts are 90% reliable each, your data has 99.999999999999999% overall reliability. It’s a very powerful way to protect data, even when the individual hosts themselves are on home connections and constantly unplugging, rebooting, or losing power. It also means that up to 20 of your hosts can fail, and your files are still able to be recovered. As the network stabilizes and the software improves, we will be switching to 20-of-50, which has the benefit of only 2.5x total redundancy, while still being about equivalently reliable (more hosts means you can get away with less redundancy).
The UI displays the redundancy of each file. As the file uploads, you will see this redundancy increasing. Red means ‘file cannot be recovered’, yellow means ‘file has low redundancy’, and green means ‘file is safe’.
Step Five: Maintaining Your Files
Contracts by default only last for 3 months. When the contract expires, the host deletes all the data, because they are no longer being paid to store it. To prevent data loss, your client will automatically renew your contracts with the hosts. To do this though, the client needs to be running, and it needs the wallet to be unlocked (because it has to pay for the time extension on the data). Your client will renew your contracts when there are six weeks remaining in the contracts. The contracts will extend the time back out to 3 months, and you will have to pay for the additional 6 weeks of data storage that the host is committing to.
Over time, some of your hosts will inevitably disappear. As this happens, the hosts will be replaced and your files will be automatically repaired. The repair operation is moderately expensive, so it only triggers once you’ve lost half of your redundant hosts (in a 10-of-30 scheme, it’ll trigger when only 20 of your hosts are left). The repair operation cannot happen unless your client is running, because the redundancy is encrypted. If hosts were able to repair your files without you, they would also be able to cheat and pretend that your files are redundant when they actually are not.
This means that you should let your client run overnight at least once a month, so that it can perform all of the housekeeping on your files. If you have a lot of files and a slow upload speed, you may want to let it run for a few days, because re-uploading files that have lost redundancy can take a while. You can tell the progress of the repair by looking at the redundancy for each file and folder. If it is green, it means the file is good to go, and no more work is necessary. If it is yellow, it means you should let Sia run until the file has been repaired.
I hope this helps everyone who was confused about how to use Sia, and what it’s doing behind the scenes to protect your data. Eventually we hope that this information is a lot more intuitive, and that the user experience gives you confidence and understanding without needing to reference any documentation. In the meantime, we’re happy to help in any way we can.
