# A HUGE thanks to Mark Sarson - For Restoring ClarionHub **URL:** \ **Category:** Site Announcements\ **Tags:** discourse\ **Created:** [February 6, 2025, 3:42pm UTC](https://clarionhub.com/t/a-huge-thanks-to-mark-sarson-for-restoring-clarionhub/7893 "2025-02-06T15:42:58Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![MarkGoldberg](https://clarionhub.com/user_avatar/clarionhub.com/markgoldberg/32/1511_2.png) [@MarkGoldberg](https://clarionhub.com/u/MarkGoldberg)\ **Post date:** [February 6, 2025, 3:42pm UTC](https://clarionhub.com/t/a-huge-thanks-to-mark-sarson-for-restoring-clarionhub/7893/1 "2025-02-06T15:42:58Z")
@Mark_Sarson spent a lot of time and put in a lot of effort to bring back ClarionHub, Restoring the data in the process. THANK YOU THANK YOU THANK YOU My understanding is an automatic update came through, which had a requirement that was not supported by the version of the OS that ClarionHub is (was?) using. The update left ClarionHub well and truly broken. ---
**Author:** ![StuAndrews](https://clarionhub.com/letter_avatar_proxy/v4/letter/s/76d3ee/32.png) [@StuAndrews](https://clarionhub.com/u/StuAndrews)\ **Post date:** [February 6, 2025, 11:20pm UTC](https://clarionhub.com/t/a-huge-thanks-to-mark-sarson-for-restoring-clarionhub/7893/2 "2025-02-06T23:20:52Z")
Agreed. Many thanks @Mark_Sarson - massive effort. ---
**Author:** ![pbouma](https://clarionhub.com/letter_avatar_proxy/v4/letter/p/d6d6ee/32.png) [@pbouma](https://clarionhub.com/u/pbouma)\ **Post date:** [February 7, 2025, 10:49am UTC](https://clarionhub.com/t/a-huge-thanks-to-mark-sarson-for-restoring-clarionhub/7893/3 "2025-02-07T10:49:04Z")
Yes, Thank you Mark\_Sarson ---
**Author:** ![Mark\_Sarson](https://clarionhub.com/user_avatar/clarionhub.com/mark_sarson/32/6132_2.png) [@Mark\_Sarson](https://clarionhub.com/u/Mark_Sarson)\ **Post date:** [February 7, 2025, 11:12am UTC](https://clarionhub.com/t/a-huge-thanks-to-mark-sarson-for-restoring-clarionhub/7893/4 "2025-02-07T11:12:12Z")
Thanks, Mark. Yes, the forum outage was caused by a Discourse update. The admin panel indicated that the update could be applied directly from within the hub, so I went ahead—but that turned out to be a mistake. Somewhere in the process (either due to the update itself or my early attempts to fix things), the Docker container was removed from the droplet, effectively taking the forum offline. At first, this looked pretty bad. However, thanks to **Brahn’s original setup** , where the database files were stored **outside** of the Discourse container, all the data remained intact. That was a huge relief because it meant that, in theory, everything could be restored once I figured out how to rebuild the system properly. On top of that, I had peace of mind knowing that **daily backups to AWS S3** were in place, configured within Discourse itself. So even if the worst had happened, I could have restored the forum from a recent backup. ### **What Went Wrong?** The biggest issue turned out to be PostgreSQL, but **not in the way I originally thought**. Discourse installs **PostgreSQL 15 inside the Docker container** as part of its package build process, meaning the host OS **shouldn’t** have been a problem. However, the droplet was still running **Ubuntu 16** , which is **long past end-of-life** and lacks modern system libraries like newer versions of **glibc**. The Discourse package inside the container likely relied on system features that Ubuntu 16 simply didn’t support. The only way forward was a **full system upgrade** , which meant: 1. **Upgrading from Ubuntu 16 → 18 → 20 → 22** (since major versions can’t be skipped). 2. **Ensuring Docker and all dependencies still worked properly** after each upgrade. 3. **Rebuilding the Discourse container from scratch** , making sure it functioned in the new environment. ### **Why Was This Such a Mess?** Honestly, Discourse **shouldn’t have allowed** the update in the first place. A well-designed update system should: - **Check the host OS version** and refuse to proceed if it’s unsupported. - **Warn in advance** if the update requires newer system libraries. - **Provide clear error messages** explaining what needs to be done first (e.g., upgrading Ubuntu). Instead, the update process proceeded blindly, only to fail because the host OS wasn’t up to scratch. It’s frustrating that something so easily avoidable ended up causing such a major disruption. ### **How I Fixed It** Once I upgraded the droplet to **Ubuntu 22** , I was able to: - **Rebuild the Discourse container** with the updated dependencies. - **Reattach the database** to ensure all content was intact. - **Verify that everything was working as expected** before bringing the forum back online. Finally— **boom, we were live again!** ### **Lessons Learned & Reassurance** This was a tough lesson in making sure **system compatibility is checked before hitting “update.”** If you’re running Discourse, **learn from my pain** —make sure your OS is up to date **before** attempting an upgrade! That said, thanks to the **daily AWS S3 backups** , I always had a fallback if something went wrong. Even though I didn’t need to restore from a backup this time, knowing they were there gave me confidence throughout the recovery process. Moving forward, I’ll be more cautious about updates and will review the process to prevent similar issues in the future. I’ve also set up a cron job to take snapshots of the droplet every weekend as an additional safeguard. Lesson learned! ---
**Author:** ![Koen](https://clarionhub.com/letter_avatar_proxy/v4/letter/k/a9a28c/32.png) [@Koen](https://clarionhub.com/u/Koen)\ **Post date:** [February 7, 2025, 12:58pm UTC](https://clarionhub.com/t/a-huge-thanks-to-mark-sarson-for-restoring-clarionhub/7893/5 "2025-02-07T12:58:46Z")
I am not here that often, but thank for restoring this source of information. ---
**Author:** ![jkuijt](https://clarionhub.com/user_avatar/clarionhub.com/jkuijt/32/6168_2.png) [@jkuijt](https://clarionhub.com/u/jkuijt)\ **Post date:** [February 7, 2025, 1:05pm UTC](https://clarionhub.com/t/a-huge-thanks-to-mark-sarson-for-restoring-clarionhub/7893/6 "2025-02-07T13:05:24Z")
Thank you Mark! Best regards Jeffrey ---
**Author:** ![seanh](https://clarionhub.com/letter_avatar_proxy/v4/letter/s/4af34b/32.png) [@seanh](https://clarionhub.com/u/seanh)\ **Post date:** [February 8, 2025, 4:53am UTC](https://clarionhub.com/t/a-huge-thanks-to-mark-sarson-for-restoring-clarionhub/7893/10 "2025-02-08T04:53:47Z")
Thanks for the info Mark, it’s nice to know what went on rather than a black box of stuff, much appreciated: Both the info and your effort! ---
**Author:** ![PaulAttryde](https://clarionhub.com/user_avatar/clarionhub.com/paulattryde/32/3519_2.png) [@PaulAttryde](https://clarionhub.com/u/PaulAttryde)\ **Post date:** [February 8, 2025, 8:32pm UTC](https://clarionhub.com/t/a-huge-thanks-to-mark-sarson-for-restoring-clarionhub/7893/11 "2025-02-08T20:32:53Z")
Mark, Thanks for all the effort you put in to get it working again, and for the write-up of what happened.