How To Build A Decentralized Cloud Storage Solution Like Storj.io

The cloud storage market is expected to grow to $92.49 billion by 2022. The market is currently dominated by technology giants Amazon, Google, Microsoft and IBM. Currently, cloud computing is a completely centralized environment where the service provider has complete control over the data stored in their cloud.

Centralized cloud storage service providers charge a significant fee for their services. While these fees might not mean much to larger companies, smaller companies can feel the pinch. Amazon, for example, charges $23 a month for the storage service they provide.

While they may not handle the data in an unethical manner, they still have the ability to use data analytics to trend the data.

In some cases, cloud storage providers can also use data stored in their servers to directed  tailored ads to the data owner, something which many companies find intrusive. Worst of all, cloud platforms currently operate a server model that has a single point of failure making data breaches far more likely.

Decentralized cloud storage is a service that has already began to cemented itself in the day-to-day process of businesses and companies. Service providers like Storj and Siacoin have already built a credible presence in the market by offering a level of security that the server-client cloud providers cant match.

Undoubtedly, more service providers will enter the market as it grows. Startups such as Filecoin and MaidSafe are already using innovation to try to claw out their own share of the market.

In this guide, I will look at how to develop a decentralized cloud storage solution as well as analyze some of the existing service providers who are set to become big players in this marketplace.

All these providers use blockchain technology to implement even more secure cloud storage solutions.

Contents

What should a decentralized cloud storage service accomplish?
How does a blockchain-based distributed storage service work?
How to create a decentralized cloud storage network using the Storj platform:
How to build a blockchain-based distributed storage solution from scratch:
Examples of decentralized cloud storage networks:

What should a decentralized cloud storage service accomplish?

cloud

What should a decentralized cloud storage service be?

It should meet the following criteria:

Download Our Project Specification Template

  • A decentralized storage network will ensure that no one entity controls the entire service. If one service provider controls the entire network and provides sub-optimal service, the customers have no option other than receiving a below-par service. A competitive marketplace for cloud storage services will require multiple providers competing to store customers’ data, thus ensuring quality.
  • Customers should be able to delete the data from their computer systems and be assured that once they get their data back from the service provider, it’s still the same.
  • Only the customers should have access to their data and no one else. The service provider mustn’t be able to view the content.
  • The service should have inbuilt redundancy, so that customer is assured that their data is safe, and hasn’t been destroyed.
  • The service should have a good performance level so that the customers are assured of being able to access their data as and when they need it.
  • The service should be cost-effective for customers. For example, the $23 per month fee  charged by Amazon isn’t optimal in today’s era when storage technology has become very cheap.
  • The entire system should have incentives for the storage providers to participate and for the customers to use it.

The central idea to a decentralized cloud storage network is that people with excess computer storage will rent their storage space for a fee. Users looking to store their data on the cloud will naturally be the ones to pay a fee for this service.

While providers may not necessarily be able to promise rock bottom prices, this type of cloud service will ensure better safety and security of the data.

A transparent mechanism will ensure that the user pays these fees while users will reward the best service providers through continued use of their services. Sub-optimal service providers, on the other hand, will be held accountable for their poor quality of service and quickly put out of business.

How does a blockchain-based distributed storage service work?

cloud-aims A blockchain-based distributed storage service works in the following manner:

  • The basic premise is to share a file across a peer-to-peer (P2P) network and track the encryption, storage, retrieval and contract administration using a blockchain.
  • The customers encrypt their file and control the encryption key. The customer is called the ‘renter’.
  • The file is then divided into many pieces, and duplicates of each piece are created to ensure redundancy.
  • The shards, i.e. smaller pieces of the file are then stored in multiple computers in the P2P network. The owners of the individual computers on the network are called the ‘farmers’.
  • Each farmer gets only the shards which are already encrypted, and hence they can’t access the content of the shard. Even if hackers attack a node they will only get the encrypted shards, which they can’t read, meaning they can’t possibly reconstruct the entire file from it.
  • When the renters request their file back, they will need to use a blockchain-based hash table along with the encryption key, so that the entire file gets reconstructed from the individual shards. The file will be rebuilt after the farmers send back the individual shards they hold.
  • The renter then uses the encryption key to decrypt the file.
  • The smart contracts in the blockchain govern the terms and conditions, the renters pay for the file they had stored while the farmers get paid for their storage volume in a  cryptocurrency such as Bitcoin.
  • There is an audit process to ensure that the farmers are holding the shard they had agreed to hold. When farmers successfully deliver on their obligation they will accumulate a good reputation as well as cryptocurrencies. If farmers default on the terms and conditions then they aren’t paid. Repeated instances of default may invite disqualification.

Note: There is one differentiation in the current centralized cloud storage space that decentralized blockchain-powered cloud storage makes redundant. In the current landscape ‘public cloud’ and ‘private cloud’ both use the technology framework of cloud computing.

However, public cloud is a multi-tenant environment where the cloud storage service provider stores massive amounts of data from many individuals and businesses, whereas private cloud is designed for a single organization.

In a decentralized cloud storage network like Storj, this differentiation isn’t relevant. This is because the service provider or any node on the network is unable to access any of the content on network. Everything is encrypted, and as only shards of data are stored by farmers and not the entire file, there is no way they can access the data.

The blockchain start-up Storj Labs has already built a blockchain-powered decentralized cloud storage network and so has assumed a leadership position in this exciting new market.

They have also built a platform comprising of open-source tools, and developers can use it to build their own decentralized cloud storage network. While their number of active daily users remains small, this is one of he most promising decentralized cloud storage networks around.

How to create a decentralized cloud storage network using the Storj platform:

space

Let’s examine how companies can use the Storj platform to create a decentralized cloud storage network. After which I will analyze the Storj network as an example of a leading decentralized blockchain file storage plaform.

Fortunately, developers trying to build a decentralized cloud storage network can use the Storj toolkit to save massive amounts of time as well as making the whole process much easier.

There are three main tools, as follows:

StorjShare: This is the reference farming client of Storj. Developers can use this in their proposed blockchain file storage system so that the farmers are able to set up their machines for the network.

The non-technical farmers can use a ‘graphical user interface’ (GUI) to set various parameters such as the amount of storage space they can share, the location of the storage, and crypto payment address etc.

Each farmer can allow the system to collect their telemetry data so that the system can identify available farming nodes for a new file storage request using this data.

The StorjShare tool also handles the contract negotiations, audit responses, and other network communication. For more technical farmers, there is a ‘command line interface’ (CLI) available, and the GUI is just a wrapper around the CLI.

There is excellent documentation available to set up the GUI as well as the CLI.

Bridge: This is the tool that developers can use in their proposed decentralized cloud storage system to facilitate access by renters.

Read How We Helped a Marketing Company to Build a Back-Office Custom Ads Dashboard

In the production system, this tool will allow the renters to negotiate contracts. This also addresses auditing, payments, availability and other aspects of renters’ interaction with the network.

The developers use this to let the renters store data in their blockchain storage. It exposes the above-mentioned services and storage resources via ‘application programming interfaces’ (APIs).

There is also a client component of the Bridge. It encrypts the files at their points of entry, takes care of the sharding, and sends the shards to multiple farmers.

The centralized component of the Bridge handles the entire gamut of the P2P network communications. It’s important to note that the bridge doesn’t store files, it only deals with the cryptographic hash references of the files and their shards.

The instructions to configure the Bridge is available on the GitHub.

The Storj API: Developers will finally need to provide the core service of decentralized file storage in their system, and for this they can use the Storj API. When the renters upload their file in the new system, they will interact with the system using the bridge, which will involve the API.

The API will find the farmers who have earlier set up their farming nodes using the StorjShare, negotiate contracts, manage audits and payments.

In short, the services that the renters access using the Bridge and the ones the farmers access using the StorjShare are all provided by the Storj API and operate in the background.

The API documentation can be accessed here.

There is also good documentation on the overall took kit in the Medium channel of the Storj project.

How to build a blockchain-based distributed storage solution from scratch:

Creating a decentralized cloud storage network from the ground-up is a very complex and time-consuming project. Developers need a high level of programming skills since they aren’t using any existing platform like Storj and its’ APIs. If your team doesn’t have this expertise then you should seek outside help.

The development team needs the following broad skillset:

  • UI/UX – for designing and developing the UI for the renters and farmers;
  • Networking – to build the P2P network;
  • Encryption – to enable the system to encrypt the data;
  • Database management – to implement sharding of files and patching the files back from the shards;
  • Blockchain distributed apps (DApps) and smart contracts – to build the decentralized framework that will oversee the decentralized storage solution including enforcing contracts;

The developers need to accomplish the following while hand-coding:

  1. Develop the UIs for the renters and farmers including CLI.
  2. Build the P2P network, so that the renters can upload their file while the farmers can download and store on their computers.
  3. Develop the module to encrypt the file before the renter uploads it in the network, with only the renter being able to decrypt the file after getting it back using the private key.
  4. Implement the redundancy in the system: A P2P network may have unreliable farmers, and sufficient redundancy is needed to ensure that the file is stored in multiple locations. There are two distinct concepts that can be used for ensuring redundancy in a P2P network, and these are called ‘simple replication’ and ‘erasure coding’. In this guide, I recommend the simple replication method where multiple copies will be replicated in different farming nodes. The reason is that the erasure coding method is highly complex and adds additional overheads to the file upload and download stages.
  5. Implement the sharding of the renters’ file: Developers need to evaluate various sharding options, for e.g. sharding by keeping the size of a shard constant, and sharding by keeping the number of shards constant. They need to balance the trade-offs. The development team will also need to build a module to keep track of the shards.
  6. Implement an audit process to prove that the file is retrievable from the shards.
  7. Build the blockchain DApp that will run the smart contracts so that the renters can enter into a contract with the farmers, the quality of the farmers’ service can be audited, and the payment (made in cryptocurrencies) can be made upon fulfillment of contractual terms and conditions.

This will be a complex project, and we recommend the following learning resources the developers:

  1. A tutorial on how to build a P2P network;
  2. This Hackernoon article on using encryption in an app;
  3. This guide to implementing simple replication in a P2P network;
  4. This guide to understanding and implement sharding;
  5. The Ethereum development tutorial;
  6. This blockgeeks online course for Solidity, i.e. the proprietary language of Ethereum for coding smart contracts;
  7. The website ‘dappsforbeginners’ for the developers to get guidance on how to create DApps.

 

Examples of decentralized cloud storage networks:

The decentralized cloud storage space is most certainly a lucrative one. The potential to disrupt the existing cloud storage behemoths is real and enormous. Blockchain start-ups are already racing to gain market share.

Here are a few examples of such start-ups:

Storj: In addition to providing the platform I described above, Storj has built an impressive decentralized cloud storage network, making it the market leader.

It follows the technical and operational model I described above and provides an elegant user experience. Signing up is easy for the renters as well as farmers. Renters can pay in the USD while farmers are paid in the STORJ token. This token is built using the ERC-20 standard because Storj network is itself a DApp on the Ethereum blockchain. There is no crypto mining for STORJ token.

Although Storj controls the access to the network, it’s far more decentralized than the current centralized service providers, because Storj can’t access the data the renters upload.

Sia Network: Sia is a competitor to Storj, and ranks second in this market. It works like Storj, i.e. uses encryption, sharding, redundancy, and audit. However, it has its’ own blockchain network, which supports smart contracts.

Sia doesn’t accept fiat currencies so renters must pay for the services consumed by Siacoin (SC). The farmers are paid in SC, too. Siacoin requires mining.

Filecoin: This company hasn’t yet launched their services. However, their model is likely to follow the Storj and Sia networks. Unlike fixed pricing offered by Storj and Sia, Filecoin will offer a marketplace with bids and asks. The team behind Filecoin is the same one that built the famous InterPlanetary File System (IPFS) protocol.

MaidSafe: They are still working on their platform, which is currently in the Alpha stage. In addition to a decentralized cloud storage network, they also plan to provide computing resources on the cloud. People can put their excess network bandwidth for rent on this network. The project promises complete anonymity for the users, and transactions will take place using the MaidSafe (MAID) coin.

Download Our Project Specification Template

Aran Davies

Aran Davies

Blockchain Expert | Developer | Writer | Photographer
Aran Davies