UPDATED 4AM CT 4/5/18
This issue has been resolved.
We’re currently experiencing an unexpected issue with our database server which has caused the site to remain offline since approximately 7 AM CT this morning. Our entire Dev Team is working as hard as they can to resolve this issue. __The should be no data lost due to this issue __(and we have existing back-up’s), but the process to check the data integrity and bring the database back online is taking longer than we had hoped.
In the meantime, we are making the site available in read-only mode. This means that if you already had a Roll20 account with a game created, you are able to get into your game and play with anyone you had already invited. Since the system which allows us to power our games is separate from the system which handles things like account and game creation, your changes to the game itself (such as moving tokens, modifying character sheets, etc) will be properly saved just like normal.
However, while in this mode you cannot:
Create new accounts
Create new games or invite people into games
Make new purchases or modify your subscription
Make forum posts
Your API scripts will function, but API state will not be saved and will revert back after your API sandbox restarts
Upload new art (maps or tokens), music, etc.
So if you already had a game planned and everything ready to go, you should be able to get in and play. We’ll keep working to restore the site to its full functionality, which should happen later today.
Additionally, thread on the forum are not loading.
We apologize for the inconvenience this issue has caused, and we’ll be doing a full postmortem of the cause and what we’ll be doing to keep it from happening again once we get things fully operational.
Update (7 PM CT): Our Dev Team is still working on the issue. Although we will be doing a full postmortem once things are fully operational, we did want to let you know why this is taking so long.
This morning our database experienced an error which caused it to go into a read-only mode.
We followed the recommended steps listed by the database software provider for remedying the situation, but it’s become apparent that due to the size of our database, this approach is going to take much longer than we estimated.
About 4 hours ago we started executing our plan to do a full database restore after taking a snapshot of the database from this morning when it went into read-only mode. Again, due to the size of our database this is not a fast process. Although we are still letting the original process run in the hope that it will finish sooner, we estimate that the database restore will finish around 6-8 hours from now. We will leave the site in read-only mode until that time.
Again, we apologize for the interruption to your gaming schedules that this has caused, and we will give more information on the details of what caused this issue, as well as what we will do to keep it from ever happening again, in our postmortem. In the mean time, we appreciate your patience while our Dev Team burns the midnight oil tonight to get things back up and running ASAP.
Update (4 AM CT): Our database restoration process has now completed and the site is once again fully operational. As previously stated, absolutely no data was lost during this process. We will post more details about what caused this issue and what steps we will be taking in the future to prevent a similar issue from occurring in a post-mortem on the blog soon. Thanks for your patience and again we apologize for the inconvenience this caused to your games! Happy rolling.