Introducing hostd
Since its launch in 2014, the Sia network has been at the forefront of the decentralized storage space, offering private and cost-effective storage. Throughout its life, the primary software, siad, has been the cornerstone for both uploading files and providing storage. While significant advancements were made to the renter module, the host module remained relatively stagnant, limiting the network’s potential. The architecture of siad's host posed challenges for detecting issues and increasing performance. But now, with the beta release of hostd, storage providers can take advantage of a comprehensive set of enhancements and features that are fully compatible with the existing network.
With enhanced stability, faster upload and download speeds, improved resource management, and a user-friendly interface, hostd empowers storage providers to optimize their nodes and maximize their efficiency. By addressing the limitations of its predecessor, hostd enables providers to deliver a superior experience for consumers and drive the growth of the Sia network.
Performance & Stability
With hostd, the Sia network takes a leap forward in performance and stability. This new software has been designed to address and fix many of the performance-related issues and system lockups that were associated with the previous host software, siad. Major optimizations to the host’s hot path provides significantly increased performance and reduced system resource usage. Hosts should immediately notice much smoother operation and fewer interruptions in service.
The improvements in performance are evident when we compare average RPC times, expressed in milliseconds. Renters and hosts interact through Sia’s renter-host protocol. The append sector RPC is used to upload and store one 4MiB sector on the host, whereas the read sector RPC sends a sector from the host to the renter. Each RPC is now significantly faster, contributing to the performance boost.
As illustrated in the Performance Comparison graph, we observed significant improvements in both egress and ingress speeds. Specifically, egress speed (the speed at which the host sends data to renters) nearly doubled, and ingress speed (the speed at which the host receives and stores data) increased more than 16 times. These optimizations lead to a substantial acceleration in data processing, which in turn yields a more efficient and responsive network.
Data Integrity
hostd introduces new data integrity tools, which empowers users to verify the completeness and accuracy of their stored data. These tools allow hosts to identify signs of data corruption early. A proactive approach enables timely resolution of issues, preventing them from escalating or affecting a wider range of data.
These tools not only increase the resilience of individual hosts, but also enhances the robustness and reliability of the whole Sia network. As we continue to refine and expand these tools, hosts can look forward to even more effective ways to manage and protect their data.
Overhauled UI
In the past, siad was accessed via a command-line interface, siac. For a more user-friendly experience, desktop users could utilize Sia-UI and Sia Central’s Host Manager. However, there were no options available for server-based users to manage their hosts. With hostd, this concern has been addressed by including an embedded web interface. This web-based dashboard allows both local and remote management of hosts, making it easier for users to interact with and manage their storage nodes from anywhere.
Metrics
When accessing the new UI, storage providers are immediately presented with various metric graphs, which enables users to track the performance of their host over time. These visualizations offer an easy-to-understand overview of how the host is performing.
The metric graphs present various crucial data points, such as bandwidth usage, data storage, pricing, among others. Providers can observe trends and patterns to identify areas where their host excels and where there may be room for improvement.
Volume Management
The volume management capabilities in hostd have been redesigned to provide greater control and convenience for storage providers. These features help manage how data is stored and transferred within the host system.
First, hostd introduces a feature to mark volumes as read-only. This means storage providers can designate certain volumes where new data cannot be written, allowing for more control over where data is being migrated within the software.
Second, hostd addresses issues with data migration that were present in siad. Specifically, hostd has resolved the issue of lockups that occurred when moving data from one volume to another. Now, storage providers can add, remove, or resize volumes without the system becoming unresponsive.
Wallet
hostd uses a new hot wallet using standard BIP-39 12-word seeds. By migrating all of our new apps to BIP-39, users can enjoy the convenience and interoperability provided by this widely accepted standard for mnemonic phrases.
In addition, we’ve introduced a comprehensive transaction list, a feature highly requested by hosts. Unlike in siad, where host contract payouts were not visible, hostd ensures these payouts are included in the transaction list. This feature improves the transparency of the wallet, enabling hosts to accurately track and audit their earnings over time.
The introduction of hostd is a major step forward for the Sia network, marking the beginning of a new era in performance and usability. By addressing the limitations of its predecessor, hostd not only optimizes the experience for storage providers — it also significantly contributes to the robustness of the network. With an ongoing commitment to further optimizing hostd and renterd, the Sia Foundation continues to push the boundaries of decentralized storage. As we continue to add and improve features, we look forward to seeing how our community of storage providers will leverage these advancements to deliver superior services and drive the future of decentralized storage.
You can find the latest software at https://sia.tech/software/hostd
We conducted performance tests for this article on both siad and hostd to compare their capabilities. These tests were carried out on the same server running Ubuntu 20.04. In this setup, both softwares used the most common setup for Sia hosts: an NVMe drive to store metadata and a SATA HDD for data storage.
The benchmarking process consisted of ten tests for each software, uploading and downloading 1GiB of data in each test. For those interested in replicating these tests, the benchmarking utility is available on our GitHub: https://github.com/SiaFoundation/host-bench.
