Skynet Summer 2021 Update
The Sia team spent the first 6 years of its existence primarily working on internal infrastructure. With the advent of Skynet, that has started to change. The underlying technology is now strong enough to carry the aspirations of millions of users on its back, and as a team we get to transition our focus from strengthening the technological core of Sia to building developer infrastructure that allows others to leverage the technology we’ve built.
A Brief Summary of Progress
In February 2020, we launched Skynet, which at a fundamental level looked like a finished version of IPFS+Filecoin. Anyone could now use the Sia platform to share static files and host static websites, including heavy content like video. Unlike competing alternatives, Skynet did not depend on a robust caching layer to achieve high performance — anyone with a fresh node could access any content with strong performance.
In June 2020 we closed a $3 million fundraising round led by Paradigm, giving the team enough capital to see the Skynet idea all the way through. In a world where our direct competitor was rumored to be sitting on almost a billion dollars, Paradigm chose to place their bets on our team.
In October 2020 we figured out how to put mutable data on Skynet, solving one of the biggest challenges of the decentralized web. At a fundamental level, mutable data makes Skynet equivalent to the centralized web. Content creators can have profiles and page links, and once the link is shared, users can continue to follow updates and new content using the same link. It became possible to use Skynet to build decentralized alternatives to sites like Reddit, Twitter, and Google Docs.
In April 2021 we released a new identity system for the decentralized web called MySky, with a data sharing pattern called “DACs”. With MySky and DACs, we are able to create a collaborative paradigm for the decentralized web that surpasses the capabilities of the centralized web.
Through the whole journey we’ve been working on performance and scalability. In the past year, the total load a single Skynet server can handle has increased by over 100x. We’ve also jumped from manually managing every node on the network to using automated tooling to configure and update dozens of servers effortlessly, all using an entirely open source stack that anyone can run themselves.
The Immediate Future: Performance and Reliability
The bulk of our raw engineering effort today is pointed at reliability. We have four major API endpoints that we are focused on: upload, download, registry read, and registry write. These four endpoints form the backbone for storage and data operations on Skynet, and a strong developer and user experience depends on these endpoints being high speed and high reliability.
We measure reliability and performance using a metric called ‘p999’, which is like a median, but it’s focused at the 99.9% mark instead of the 50% mark. If your endpoint has a p999 of 200ms, it means that 99.9% of all requests to that endpoint return in under 200ms.
Importantly, when we publish statistics about the performance of our endpoints, we are ignoring any requests that were able to benefit from a caching layer. The statistics we present represent the worst case usage patterns on our network, and demonstrate the results you can expect when accessing rare or un-cached data. This strategy is a departure from our competitors, who either don’t publish performance data at all, or depend entirely on their caching layer to achieve competitive numbers.
The main dashboard that our engineering team uses to monitor the real time performance of Skynet can be found at https://siasky-dash.hns.siasky.net/. While it defies conventional ideas of beauty, it excels at providing dense information that is easily digested by our engineers. We will be upgrading the dashboard over time, but generally only as our engineers need changes in order to be more effective.
As of writing, the dashboard contains 5 columns that measure the p99/p999 of our endpoints, and 2 out of the 5 columns have strong performance results: the registry read time and the registry write time. These are the two endpoints for dealing with dynamic data on Skynet, which is most important for things like real time games, live chat, and live streaming.
Today, all of our energy is being focused on the download endpoint, which is important to web pages and applications in addition to being important for media and content distribution. We’ve made a significant number of improvements that haven’t been deployed to production yet, and overall we project that one to two more months of improvements will be necessary. Once downloads are looking superb, our attention will turn to uploads.
We’re putting so much energy into the raw performance of our infrastructure because reliability is fundamental to both a successful user experience and also a successful developer experience. Decentralization is supposed to build infrastructure that cannot fail, and we are building Skynet to follow through on that vision.
The Immediate Future: Integration
We’ve started to be more proactive about integrating Skynet with the rest of the crypto ecosystem. One of the first places this is starting is giving Skynet full feature parity with IPFS.
Today, hosting an app on IPFS and Skynet sometimes requires some extra configuration steps. For example, some react apps work on IPFS in slash-routing mode, but need to use hash-routing mode to work on Skynet. There are some other differences in how Skynet emulates a traditional server as well. We’ve identified the steps we need to make so that Skynet can perfectly match IPFS, allowing any IPFS application to be hosted on Skynet without any changes. We’re expecting to finish and deploy these changes in the next one to two months.
Another popular feature on IPFS that we will be adding to Skynet this summer is DNSLink, which allows someone to host a website at their own domain, but load the page from IPFS via a gateway. We’ve been successful in getting our development portal to replicate this feature, and it’s something we’ll be rolling out to users later this month.
Beyond that, we’ve started working on other integrations within the broader crypto ecosystem. Full announcements are still pending, but we’ve been working to add Skynet support to projects like ENS and Metamask, and we’ve been working on products that target opportunities in the Defi and NFT space.
The Immediate Future: Portal Versioning and Infrastructure
We have had an ongoing effort to fully automate the tasks of setting up and maintaining portals. Our portal infrastructure at Siasky today has about 20 servers total, which is enough that even smaller tasks are tedious if they aren’t automated.
We’ve been adding automation in place using Ansible. A big part of the automation process has been better version control, so that every portal across our network is running the same software. We now have production branches that we sync before rollout, and soon we’ll be tagging versions as well.
This benefits not only ourselves but third party portal operators as well. Now that everything is cleanly divided into production branches, third party operators know what commits they should be running to provide a stable and feature-complete experience to their users. The process isn’t fully completed yet, we should be using version tags more than we do, but we’re making progress on all fronts.
We’ve also been working on improving the process for spinning up a new portal from scratch. Our long term goal is to minimize both the effort required and technical skills required to start and maintain your own portal. We’ve made huge strides from 6 months ago, and we’re continuing to focus on this goal.
Once we are happy with the full process for operating, maintaining, and upgrading a portal, we will do a big push within the community to get more portals online, and to get more Skynet users running on third party portals rather than siasky.net.
The Near Future: Skynet-js and MySky Changes
Skynet was a big jump for our team because we had little to no experience designing things for the web. But we’ve since realized how much better the user experience is when you can do everything from a web browser. It just works, it works everywhere (your laptop, your phone, your desktop, your friend’s Nintendo switch), and all of your data follows you around.
We’re now committing fully to making Skynet a web-first experience. The first part of this is a heavy focus on building tools that allow developers to build fully decentralized apps that are web native. A newer portion of this is an increased focus on allowing existing centralized applications to incorporate some of the advantages of Skynet even if the application as a whole isn’t decentralization first. Examples of this would be a new CDN library we are working on, and a new SSO product we are working on.
We’re also going to be making some changes to the core architecture of the skynet-js and MySky libraries. Over the past year, we’ve continually needed to break compatibility as we’ve rolled out new features and upgrades, placing a burden on our long term developers that like making use of our new features.
The new architecture we’re working on is explicitly designed to be easily upgradeable without breaking compatibility with previous versions. We want developers to be able to make use of new features on multiple-year-old applications without having to change any of their legacy code. We’re still working through the exact details, but we believe we can release such an architecture in the next few months.
Finally, our internal research has produced enormous advances in the capabilities and decentralization of MySky itself. We will be going into greater detail in another blog post, but we have figured out a way to make MySky simultaneously significantly more decentralized, and also substantially easier to integrate into the traditional web browser.
The Near Future: Documentation and Tutorials
Internally, we’re spinning up a much more structured approach to feature releases, especially with regards to documentation, tutorials, and support tooling. Historically, as we released features for Sia most of the advantage was limited to the core team. As a result, our process for documenting and supporting new features is primarily internally facing.
We’ve now transitioned to primarily building features for external users and developers. Our documentation and support processes are still catching up, but we’ve realized the gap that exists and have committed to closing that gap.
We’re planning on splitting the formal documentation into two pieces. The first piece is a slate document that is highly technical and provides the full details for how every API endpoint and SDK function works. The target audience is developers who already know what they need to build, and want an accurate reference that describes exactly how each endpoint works.
The second part of our documentation will be a gitbook that is aimed at being more narrative driven. Where the slate document is aimed at answering the question “how do I do X?”, the gitbook is aimed at answering the question “what all is possible on Skynet?”. The gitbook is meant to be a launch point that helps developers find interesting things to build and also links to more educational resources to teach them more about decentralization as a whole.
The other major change we are making is with regards to supporting documents and tooling. When we put out a new feature, we will also release blog posts detailing how to use that feature, and where necessary we will release support skapps that make it easy to play around with the feature and accomplish common tasks directly in the web browser.
The Near Future: Creator Monetization
We have been continually iterating on our ideas to monetize creators, builders, and content on Skynet. Our previous discussions about recursive content monetization now feel almost archaic.
We aren’t ready yet to release the full details of what we have been working on. But in iterating, we have been asking ourselves a few questions:
- How do we make the user experience as simple and seamless as possible?
- How do we deliver value to the user, such that by paying the user gets things in return that they could never get via piracy?
- How can we mirror the core process of creating content, properly acknowledging all contributors that were important to that content?
- How do we make spending money exciting and fulfilling?
We like the answers that we have come up with, and we will be releasing our monetization product when the time is right.
The Horizon
Especially when you get deep in the weeds of the next thing that needs to happen, it can be easy to lose sight of the long term mission. It is just as important to keep your eyes to the horizon as it is to watch the footing in front of you.
Skynet is a crusade against the cloud. YouTube, Facebook, Twitch, Discord, Snapchat — each represent a data silo where users put in the hard work of building platforms, communities, and relationships, but give the control over to a faceless corporation that is designed to take profit at all costs, regardless of the well being of their users or humanity as a whole.
The Internet is the most valuable resource that mankind has ever created. It’s a global information layer that accelerates education, accelerates product ideas, accelerates specialization and professionalism, and overall increases the rate at which humanity is able to converge and evolve. But it’s a resource that has been crippled and corrupted by massive for-profit interests that care more about ensuring users never leave their walled garden than they do about actually benefiting mankind.
Skynet is here to fix the Internet. The Internet is broken because the incentives are broken. The most profitable, competitive thing for a business to do today is to lock valuable data away and use that data to wage war against users and against other corporations.
To fix the Internet, we need to fix the incentives. We need to make it so that companies using open data can run circles around their silo hugging counterparts. We need to make it so that sharing data openly with the world is more profitable than locking it away. We need to change the Internet so users would find it absurd that data they put into one application would be unavailable on other applications.
Our team is committed to Skynet because we know it’s possible. A decentralized, open access cloud is fundamentally more powerful than a closed, walled garden cloud. We are building a future where any startup or business can make use of the treasure trove of data that forms the Internet, and have certainty that their access to that data will remain. We are building a future where anyone who contributes to this amazing living archive can be compensated financially if their data is useful to someone else.
Today, we are very early. We don’t know exactly what the open Internet looks like. In many cases, we aren’t trying to guess. Instead, we’re following our noses. When we look at what features to build, we ask ourselves what unlocks the greatest amount of fundamental power for the ecosystem. We aren’t looking 1 year into the future but 5, and we’ve been doing this for the entire 7 years that we’ve been building. And it’s starting to pay off. The things we can do today surpass our greatest dreams from 5 years ago.
As the growth continues, the mission will continue to evolve. Today, our horizon is the cloud, and we are building a competitive ecosystem and economy that will be able to overtake that cloud. We are 1 million users strong already, and we are growing.
