Growing Pains
For the past month we’ve been working closely with Poloniex and Bittrex to fix wallet issues that were preventing deposits and withdrawals. It appears things are now starting to function as expected. Bittrex withdrawals have been working for several days, and Poloniex is clearing out its backlog of failed withdrawals. Most new Poloniex withdrawals are successful, and many users are receiving withdrawals from weeks ago.
Thank you all so much for your patience and support as we worked with the exchanges to fix the issues. In this post, we will explain what went wrong, how we responded, and what we are doing to make sure these issues don’t reoccur anytime soon.
What Went Wrong
Overall, the wallet issues were caused by a major spike in transaction volume on the Sia blockchain.
We plan for growth, but we were not expecting so much growth, so fast, without warning. Siacoin entered May with a market cap of ~$30M, and quickly increased to over $400M by early June. As users transferred their coins out of exchanges, and became hosts and renters on the Sia network, the number of transactions significantly grew.
This huge spike in transaction volume meant that our transaction pool (aka mempool), at 2 MB in size, occasionally filled up at peak times. If the pool is full, new transactions get thrown away. Since miners pick transactions from the pool to form blocks, any transaction that is not included in the pool will not be added to the Sia blockchain.
This is our fault. We made the decision two years ago to cap the transaction pool at 2 MB, due to significant performance issues above that size, and postponed the engineering work that would allow for a larger pool.
So, as users requested withdrawals from Poloniex and Bittrex, the exchanges broadcasted the transactions to the network — but didn’t check to make sure they actually were added to the blockchain. The transactions were marked as complete as soon as they were broadcasted. This caused a lot of confusion, since users with completed transactions were not seeing their coins appearing in their wallets.
How We Responded
After receiving user reports of Poloniex wallet issues, we emailed them on May 22 to ask if we could help. We provided a quick fix on May 27 that would allow exchanges to re-broadcast transactions that were not added to the transaction pool. Poloniex re-enabled the wallet on May 30, but had to disable it overnight due to similar problems occurring (the transaction pool quickly filled up again, and they were not rebroadcasting transactions automatically).
We continued to communicate with Poloniex, and provided them with the IP addresses of a major mining pool on May 31 (so that they could broadcast transactions directly to the pool), and better logging on June 1. We then implemented a send-to-many endpoint on June 5, so that exchanges could batch up to 250 withdrawals in a single transaction.
Poloniex implemented these fixes and went back online on June 8. They are working through the backlog, and we expect them to finish that today. This means things should be pretty much back to normal.
We worked with Bittrex simultaneously to implement the fixes and they responded really fast — getting back up and running in just a few days.
In total, we merged 14 fixes into the Sia code in a span of two weeks. These include both the short-term fixes mentioned above and long-term upgrades to allow the Sia network to better scale as transaction volume continues to increase.
Fixes and Upgrades
- Add support for getting and sending raw transactions
Allows exchanges to grab a raw transaction associated with an ID and re-broadcast it to the network. - Add logging to the wallet
Improve logging so that exchanges can more easily know if a transaction confirmed. - Try reverted transactions in tpool Update()
Every reverted transaction will now be retried to be added in the pool. Long-term upgrade for transaction pool to make reorgs less disruptive. - Add logging for failed transaction sends
Further improved logging. - Prune unconfirmed transactions older than 12 blocks from the tpool
Allows transaction pool to naturally clear of glut, preventing obtuse network fragmentation. - Add API support for send-to-many
Allows exchanges to process 25x the withdrawals per hour, using 1/3 the blockchain space per withdrawal. - Add market based transaction accepting and fee estimation
Add a fee market to allow high priority transactions to go through. - Tpool block estimation
Help drive intelligent fee estimation. - Optimize Block.MerkleRoot by replacing tree.PushObject calls
Performance optimization to safely allow for a bigger transaction pool. - Add /tpool/fee endpoint
Allow visibility into fee market prices. - Add support for user-set txn fees
Allow users to choose their own fees. - Optimize Transaction.ID() by adding direct calls to MarshalSia
Optimization to allow safe larger transaction pool. - New Block Fee Estimation Algo
More optimal fee estimation. - Switch to diff-based updates for tpool
Scalability upgrade with stress tests to allow safe larger transaction pool.
Next Steps
Most of the above pull requests focus on long-term upgrades to allow the Sia network to better scale.
We’ll be increasing the size of the transaction pool from 2mb to 5mb, with the ability to automatically grow further if there are large quantities of high-fee transactions. We will also be creating a fee market, similar to Bitcoin, so that miners can prioritize transactions based on fees. With the upgrades implemented, the Sia network should be able to safely scale by an order of magnitude.
These scalability upgrades are a main priority, and will be included in the v1.3.0 Capricorn release later this month, along with other important features like remote file repair — which will allow Sia serve as a viable cloud backup solution. In order to experience these scalability upgrades, a large portion of Sia users will need to upgrade to v1.3.0 upon its release.
Once again, thank you all so much for your patience and support in this time of extreme growth!
