Search maintenance

Apr. 22nd, 2026 09:19 am
mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
[staff profile] mark posting in [site community profile] dw_maintenance

Happy Wednesday!

I'm taking search offline sometime today to upgrade the server to a new instance type. It should be down for a day or so -- sorry for the inconvenience. If you're curious, the existing search machine is over 10 years old and was starting to accumulate a decade of cruft...!

Also, apparently these older machines cost more than twice what the newer ones cost, on top of being slower. Trying to save a bit of maintenance and cost, and hopefully a Wednesday is okay!

Edited: The other cool thing is that this also means that the search index will be effectively realtime afterwards... no more waiting a few minutes for the indexer to catch new content.

The case of the missing notifications

Apr. 11th, 2026 11:58 pm
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise posting in [site community profile] dw_maintenance

I keep forgetting to post about this: we've been troubleshooting the "missing notifications" problem for the past few days. (Well, I say "we", really I mean Mark and Robby; I'm just the amanuensis.) It's been one of those annoying loops of "find a logical explanation for what could be causing the problem, fix that thing, observe that the problem gets better for some people but doesn't go away completely, go back to step one and start again", sigh.

Mark is hauling out the heavy debugging ordinance to try to find the root cause. Once he's done building all the extra logging tools he needs, he'll comment to this entry. After he does, if you find a comment that should have gone to your inbox and sent an email notification but didn't, leave him a link to the comment that should have sent the notification, as long as the comment itself was made after Mark says he's collecting them. (I'd wait and post this after he gets the debug code in but I need to go to sleep and he's not sure how long it will take!)

We're sorry about the hassle! Irregular/sporadic issues like this are really hard to troubleshoot because it's impossible to know if they're fixed or if they're just not happening while you're looking. With luck, this will give us enough information to figure out the root cause for real this time.

Profile

lisajeannn: (Default)
lisajeannn

April 2026

S M T W T F S
   1234
567891011
12131415161718
19202122232425
2627282930  

Style Credit

Expand Cut Tags

No cut tags
Page generated Apr. 22nd, 2026 10:54 pm
Powered by Dreamwidth Studios