#MastoAdmins

Thank you so much fellow #Mastonauts and #Mastoadmins
The response to the plight of handmade.social has been overwhelming.
It looks like we may not need to shut down after all.
None of you really know me by my personal account but I am ASD and the caregiver for my 7yo ASD-nonverbal son so I am approaching a bit of burnout (but for a happy reason).
I will be taking a few days to process everything and then will be getting the ball rolling on keeping the instance up and running.

#MastoAdmins I'm shutting down https://handmade.social on September 15th in case your server is a good place for the member accounts to migrate. I just don't have the funds to keep it going beyond that.
Please also do not render unsolicited advice about cheaper DIY hosting. I'm the primary caregiver to a special needs child and do not have the tech chops or time. That is why masto.host is so awesome.
If you'd like to take over the instance, I'm open to transferring.
#MastoAdmin

:bc: Attention Beige Party-goers! :bc:

I think we might have zeroed in on an actual solution to our server problems. Yesterday I noticed that when the server was struggling it was because it was constantly retrying the same batch of jobs that kept failing because they were trying to deliver to instances that were offline. The system was overwhelmed but it was spending most of its time waiting for timeouts on jobs that had already previously failed instead of processing new jobs that could actually be completed successfully.

I tooted about this and @victor pointed out that he has to periodically check the failed jobs and stop delivery to instances that are offline. Mastodon does this automatically After 7 days of failed attempts, but that's 7 days of generating jobs that will ultimately fail and be retried several times, clogging up your Sidekiq queues for no reason. After going through and checking which instances had the most failing jobs and stopping delivery to the ones that were offline, I was able to reduce the retry count from over 8,000 to about 200. More importantly, after I stopped delivery to offline instances, my failed job count fell to nearly zero. Without all these failed jobs clogging up the works all of the sidekiq queues have remained mostly clear, and items that get added to the queue get processed within a matter of seconds.

I don't think I'm ready to say we're completely out of the woods yet, but the results have been really encouraging over the last 24 hours. For any new #mastoadmins out there, this is something I'd recommend getting in the habit of checking on from the very beginning. I haven't seen this mentioned in any guides on how to scale Mastodon, but it will definitely become an issue as the number of Sidekiq jobs processed by your instance increases. If your retry counts get up into the thousands, you might want to check which jobs are failing because of connection errors and stop delivery to those instances that are down. If the instances come back online and attempts to make deliveries, the Mastodon application will automatically resume sending deliveries to those instances, so you don't have to worry about keeping track of sites that you have stopped deliveries to.

Thanks everyone for your patience while I try to work through this. And thanks to everyone who has offered support, ideas, and general good vibes. I truly appreciate it.

Beige-bless :bb:

I've noticed that 8 out of the last 12 accounts that have signed up on our instance have done nothing since they signed up. No profile image, no banner, no bio, absolutely nothing other than sign up.

Do y'all experience the same types of numbers when dealing with new accounts? What do y'all do about it, if anything? Do you just let it happen and if they become active then so be it, and if they don't, so be it as well?

#MastoAdmin #MastoAdmins #Admins #Questions

Official Mastodon development is exploring adding paid sign-ups, pay per use storage to the #Mastodon software.

Roadmap Items in progress include:
MAS-15 Groups
MAS-86 Markdown formatting for posts
MAS-42 Add option to exclude list members from home feed

===========
Roadmap Item MAS-24:
Paid sign-ups, pay per use storage.

#MastoAdmin #MastoAdmins

🔗 Public Roadmap: This is a glimpse into what we're working on and what we're planning to work on.
https://joinmastodon.org/roadmap

we must be due a Mastodon code update soon? It's been almost two months since the last one. It looks like there have been several good fixes and update to code, as yet unpublished.

#MastoAdmin #MastoAdmins

Would love if admins had the choice to disable downloading and storing remote media from other servers and/or fine grain control if profiles are downloaded, only images, only video, etc.

Maybe put a placeholder image indicating remote media is available?

Thoughts?

#Mastodon #Calckey #Pixelfed #MastoAdmin #MastoAdmins #AskFedi

Ok, question for #mastoadmins or anyone else out there with knowledge on the subject:

Adding pgbouncer on top of the postgres database seems to have at least reduced the problem with the sidekiq queues backing up. However, the push queue is still getting backed up to a few thousand jobs occasionally. This doesn't seem like it should be that big of a problem because I've added more sidekiq processes for that queue and they should be able to get through a few thousand jobs with minimal latency. The real problem, however, is that if the push queue starts showing as little as 1 minute of latency, all remote media previews break (they just show the fuzzy gradient with the "Not Available" caption), and all media uploads break. The only way to fix it, even after the latency drops back to zero is to do a hard reboot of the server.

It feels like I'm running up against some kind of timeout setting that's set too low. I could see a latency of 10 or even 5 minutes being a problem, but expecting that latency will never hit 1 minute seems unreasonable. Any ideas on what I might be doing wrong here? It seems to be tied to the database because if I reboot the web server without also rebooting the db server, the images stay broken.

Any #MastoAdmins know why my federated timeline would NOT be streaming, but my home timeline would?

Refresh and federated is up to date, but not live...

I'd also love to hear tips and strategies admins use to fund their servers. Do you rely on donations? Do you charge for access? Sell merch?

#fediverse #MastoAdmins #FediverseAdmins #server #instance #admins

Question for instance admins. Is your server financially sustainable?

I'd say this applies to people running their personal instances.

#fediverse #MastoAdmins #FediverseAdmins #server #instance #admins

Aaaaand here we go!

First #spammers hit all the open-reg sites. As you read this, #mastoadmins are already restricting or closing registrations again. This will resolve with lots of effort, and most will #defederate any site without effective #captcha. It weakens the network, but we're still mostly OK.

The next steps are *BAD*.

After the network is #captcha and admins (or their automation) are handling complaints, the next escalation will arrive. A little work turns one of the cool, minimalist servers into an automatic spammer. Suddenly every new #federated server is a spam hose and we can only #federate if the #mastoadmin gives up showers and only sleeps every other evening.

A week later, when the obvious holes are patched, the spammers reappear, acting like real nodes and relaying messages around a while. THEN it will start spamming like mad. The only prevention is to close registrations AND #defederate and bam, we're back to the email problem..

#spam #spamwave

Lazy question for #mastoadmins do servers typically have API rate limits?

Getting ready to start on a bot that (consensually) goes deep into a user's history and collects followers and interactions. The idea is to build a graph of servers you interact with to recommend an instance to migrate to based on your personal network.

This will be a lot of data and I want to be respectful of the servers I'm hitting and not get IP banned

@stux

Visual aid for #MastoAdmins to block the doge #spam coming from the most recent mastodon.social attack quickly via moderation search:

Admin -> Moderations -> accounts -> remote -> "mastodon.social"

#FediTips #Mastodon #MastoAdmin

🔗 #MastodonSocial status: Current Status as of this toot post: Under Attack Website and API spam attack https://status.mastodon.social/

What’s this? A spam DM? 😱

I wonder where this one originated from…ahhh Mastodon.social. I am not surprised. Didn’t we just have this happen a couple weeks ago?

/me Thinks the “flagship” #Mastodon server might be too big to moderate and Mega Servers are bad for the Fedi.

I can possibly see #MastoAdmins defederating from that server and Mastodon.Online soon. I’m considering it.

Hey #mastoadmins! For a single user instance, what do you recommend for RAM allocation?

If you run an instance, or know someone who does, now is a good time to

1. Go to https://relay.fedi.buzz/
2. Under tag, type 'eurovision'
3. Paste the resulting URL into Preferences -> Administration -> Relays
4. Repeat for all the hashtags that there are:

#Mastovision #Eurovision #Eurovision2023 #ESC #ESC2023 #ESC23 #Eurovision23 etc. etc.

Thank you, lovely #MastoAdmins #FediAdmin #MastoAdmin

Is there an easy way to tell what software is running on a fediverse server?

I know Mastodon has the /api/v2/instance point.

#fediverse #mastodon #calckey #misskey #MastoAdmins #developers

#MastoAdmins, what do you set the "Content cache retention period" at? I've never been sure about that.

@bluefox vllt findest du antworten mit dem Hashtag #mastoadmins
Das dürfte die Reichweite deiner Frage erhöhen. Hoffentlich.

Well, I was able to find some of the people from mastodon.social I unfollowed by accidentally blocking the domain.

I opened a feature request to make blocking domains a bit harder. https://github.com/mastodon/mastodon/issues/24853

#mastodon #block #DomainBlock #BlockList #MastoAdmins #spam #FeatureRequest #github #OpenSource

Have you ever made a donation to a fediverse server admin?

#fediverse #mastodon #MastoAdmins #poll #survey #SocialMedia #donation

Looks like a trolling server has been spun up/unleashed on the #Fedi again, think activitypub-troll.cf like at the end of 2022.

@selea's post with more info: https://social.linux.pizza/@selea/110303325604994805

#MastoAdmins #MastoAdmin

@fediversenews

I wonder how much extra processing power is going to be needed with full text search being added to #Mastodon. My understanding was that search beyond usernames and hashtags was avoided since it can be costly in terms of CPU power.

Thousands of users searching means more database transactions having to be processed meaning costs to run servers will possibly go up.

#MastoAdmins #MastoAdmin

#Sidekiq must be doing something. My #Selfhosted Mastodon is loading with two toot posts and the dreaded "load more" label that doesn't load more.

It does this like twice a day for a few minutes.

#MastoAdmin #Mastoadmins

Well fellow #mastoadmins it looks like a new build of Mastodon will be released shortly.

#mastodon

If you are someone who is wondering about migration between #Mastodon servers (not just your followers).

@tokyo_0 has been posting about their work on a tool to move your content (posts & media) from one server to another. Note that it seems that this is still in testing and may release a test version in a week or so.

If you are curious about this, make sure to give them a follow to stay up to date on the progress.

#MastoAdmins #MastoAdmin

@tchambers @mmasnick @fediversenews

:bc: Attention Beige Party-goers! :bc:

Emails are back up after being offline for a week since the last Mastodon upgrade. This probably didn't affect any current users, unless you were trying to do a password reset, but the confirmation emails should now be going out again for new sign ups. I'm gonna dig into my database backups to see if I can find the few new sign ups that fell through the cracks in the last week.

In case any #MastoAdmins or anyone else is interested, here's what the problem was:

After the upgrade to version 4.1.2 the Sidekiq mailer service was encountering an error in message.rb in the ruby mail gem. The error message was:

ArgumentError: :arguments expected to be an Array of individual string args

After doing some googling I found this article: https://www.gamecreatures.com/blog/2022/12/19/incompatibility-rails-5-2-and-mail-gem-argumenterror-arguments-expected-to-be-an-array-of-individual-string-args/

Apparently this is a known issue in version 2.8.0 of the mail gem. I went back into the snapshot I took before the upgrade, and indeed the Mastodon upgrade resulted in the mail gem being upgraded from 2.7.1 to 2.8.0. Using the workaround suggested in the article I added the following code to the end (just before the final "end" line) of the production.rb file at mastodon/live/config/environments:
config.action_mailer.sendmail_settings = {
location: "/usr/sbin/sendmail", arguments: ["-i"]
}

I tried recompiling the Mastodon application but it looks like that was not necessary, so I just restarted the mastodon-web and mastodon-sidekiq services and the emails started working again.

I should add that I'm using YunoHost, so I'm not sure if this was an issue with the YunoHost version of Mastodon 4.1.2 or the Mastodon application itself, or, more probably, the result of some clever stupidity on my part.

Anyways, I figured I would put it out there in case anyone was else was having the same problem. And thanks to @rick for writing the article that led me to the solution!

Beige-bless :bb:

Hey #MastoAdmins #FediAdmins, what are some of the ways folks can help support your instance?

Outside of donations and maybe helping with moderation.

@NickGates Hopefully a challenge our #MastoAdmins can help us with.

#MastoAdmin #MastoAdmins

Not sure if others are seeing this or not, but I've been followed by these four accounts in the last 45 minutes or so.

Same account name, just different instances that the accounts are being created on.

I've reported them to the instance, as well as blocked the accounts.

Just wanted to give a heads up to others.

@cosmality #MastoAdmins can mass search and block #thegx #spam accounts all at once to all of their server users easily by going to Preferences> Moderations> Accounts, and doing a search.

They pop up so often, I have the direct link to search bookmarked on my browser's favorite bar.

https://oldfriends.live/@paul/110244845691645073

I assume "roles" are still broken in Mastodon 4.1.2?

I have assigned a role to a user on my instance and no icon or anything shows up next to the users name in the admin panel or on the instance itself as far as I can see.

Anyone know if this is still broken or if it's just me?

#MastoAdmin #MastoAdmins #Mastodon #Roles #Moderation #Administration #Admin

For people who run their own fediverse server, what was the biggest reason for this?

(Leave a comment if it's another reason!)

#fediverse #admin #admins #MastoAdmins #poll #survey

Hey #mastoadmin and bot builders. I've been playing with ChatGPT and other LLM and built a bot in node.js to be the assistant for my instance.

I'm having fun with it so I figured I'd put the code up on my Github here: https://github.com/pviglucci/bot

It's a bit rough and assumes knowledge of Mastodon applications, the API, openai, and node, so use it in the spirit of tinkering.

It's configured to be a service available to all users on an instance, but could easily be reconfigured to be a personal bot for an individual.

This is just for fun and not meant to prompt a flamewar on the ethical, political, economic, etc arguments of AI.

I just wanted a simple interface to ChatGPT in the app I spend a lot of time in during the day.

#mastoadmins #mastoadmintips

What on earth is this popup that just appeared when I was browsing my #Mastodon instance?

"Dow you want to allow https://mstdn.thms.uk to use up to 3.5 GB of storage on your Mac?"

Appears every time I refresh under #Safari #macOS

#mastoadmin #mastodev #mastoadmins

I'd love to hear from instance admins, but everyone's opinion is welcome.

Some time ago I made a site (1) that lets you browse tags across a large portion (2) of the fediverse.

The way the site works is thanks to Mastodon having RSS feeds for hashtags (3). I am only able to retrieve posts that are public and included in this public feed.

On a technical level, this is how RSS works. But this might not be what a typical person would expect.

#mastodon #MastoAdmins #rss

My other (Glitch) account shows my local fixed IP address for each logged-in device at Preferences -> Account -> Account settings -> Sessions. My Vivaldi account shows my IP for sessions last year, but shows random CloudFlare IPs for every login since Jan 2023. Whatever can that mean?

Both sometimes recognize my iPad as "Safari on macOS" - maybe that's when I "request desktop site"? But I don't think I've ever chosen that for Masto pages.

#MastoAdmins #Ipad #Sessions #Settings #CloudFlare

I've been having an issue with a user getting hit with the rate limit error message frequently and I'm wondering if any other #MastoAdmins have seen this before.

I took a look at the mastodon-web logs and verified that it's real traffic coming from this user's IP address. When they're hitting the rate limit there is a flood of GET requests averaging about one every 2 1/2 seconds, alternating between the notifications and timeline end points. This doesn't seem like a normal usage pattern. The logs show two requests in the space of a second, then another two a few seconds later. I think this must be some sort of automated process.

The user primarily uses Mastodon through a browser: Chrome on a laptop and Samsung browser on mobile. This issue seems to be unique to this user. The logs show a few other users occasionally hitting the rate limit, but the vast majority of the rate limit log entries are coming from this one user.

Any ideas would be greatly appreciated. Thanks!

I've just released v4.2.0 for #FediFetcher

FediFetcher now has multi-user support, so admins can now run FediFetcher for multiple users on the same instance.

I've been asked for this a few times over the last few days, and I hope some of you running instances for families, or admins with multiple accounts themselves will find this helpful.

For more details, check out the blog:
https://blog.thms.uk/2023/04/muli-user-support-for-fedifetcher?utm_source=mastodon

#MastoAdmin #mastoAdmins #fediAdmin #SingleUserInstance #mastodon

RE: #FediFetcher It is so nice not having blank, empty profiles of new followers and follows, @michael

Thanks for the work you've done on this, Michael.

#mastoadmin #fediadmin #singleuserinstance #mastoadmins #Mastodon

https://github.com/nanos/FediFetcher

#MastoAdmins Can someone explain to me why all of the #PendingReview #hashtags aren’t all listed in the pending section of the Review Status sort options? Why must I go through pages of tending hashtags to find them all?

#MastoAdmin #MastodonAdmin #MastodonAdmins

@willoremus Get ready #FediAdmins and #MastoAdmins.

Hi #mastoadmins,

This instance is running a small but fine implementation of an activity pub server called #epicyon. This has been written by @bob. I am quite happy with its simplicity.
However I have replaced the old pleroma code with this without changing URIs. So all of the old links will fails for other servers.

Lately I am running into troubles with chaos.social. Users from there that have been following users here lose their follow status. It seems that this instance receives undo activities for follows -> unfollow requests.
The reason for this is unknown. The admins of chaos.social have confirmed that they didn't do anything to cause this. Yet there it is.

What they could confirm is that they have seen troubles delivering messages to our instance, which I cannot confirm so far as I need to enable more logging on my TLS termination side (#OpenBSD's relayd).

So this issue is still under investigation and I hope to find the cause of this. Until then connections to chaos.social users will be a bit flaky it seems. Subscribing to them is fine, maintaining subscriptions _from_ them is not.

Is anyone out there having similar issues?

Thanks for listening.

mastodon_get_replies is now FediFetcher!

FediFetcher is a simple Python script that can help you pull missing responses from other #Fediverse instances into your own #Mastodon instance. It can also backfill profiles of new followers and followings.

It can be run as cron job, container, or even a GitHub action, meaning you don't need any infrastructure at all.

For more information check out

https://blog.thms.uk/fedifetcher?utm_source=mastodon

#mastoadmin #fediadmin #singleuserinstance #mastoadmins

Un #FollowFriday de #MastodonAdmins:

@acambronero
@cgseife
@FrankSonntag
@hugo
@jerry
@lizcovart
@markigra
@mikarv
@nicdex
@paulgp
@profcarroll
@redegelde
@rober
@ruud
@tchambers
@wood

#FF #Follow #Admins #MastoAdmin #MastoAdmins #FediAdmin

I'm totally at a loss here about this account.

It's on an instance mostr.pub and there doesn't seem to be much information on said instance.

Why does this user have a username like this and why is it sending a link to a Telegram channel and why would it be sending it to me?

It also looks like it's mass following users.

#MastoAdmin #MastoAdmins #Fediverse #Mastodon

Spam, spam and more spam.

Today's another ordinary day of bots and hashtag pollution on Mastodon, as the admins of the big instances have apparently given up on keeping the place clean.

I'm just one step away from temporarily blocking all content from mstdn.social and mstdn.ca until their admins get their shit back together. How are other #mastoadmins currently tackling these spam attacks?

In the meantime, if you're on my instance and see more of this crap, please report it to me - I'm doing my best to block the hashtags and suspend content from these profiles, but something will inevitably fall between the cracks.

This instance has been down for ~1 hour earlier - sorry!

After the 3rd disk usage alert in 2 weeks, and ~70GB of system cache even when adopting a one-day retention policy for remote media content, I've decided that it's time to move the static storage of social.platypush.tech to an S3-compatible bucket, rather than bleeding more money for a Linode TB-sized dedicated server.

After doing some research I've found Scaleway and it seems to fit my needs. What do other #mastoadmins use?

@thomas_virtubox If anyone else has any other ideas of what I should do please let me know.

I am a new admin of an instance so I don't know what's best and what's not, so suggestions are open.

But this is ridiculous that I wake up to notifications like this.

#MastoAdmin #MastoAdmins #FediBlock

I just woke up for the day, got to the computer and logged into Mastodon as per the usual.

As soon as I sat down and scrolled through my notifications I have realized that I have now been followed by 29 other accounts from Mastodon.Top that @thomas_virtubox is the admin of and those accounts all have malicious links in their profiles.

Yesterday I sent an email to that admin and I have reported several accounts that followed me from that account. (cont...)

#MastoAdmin #MastoAdmins #Fediblock

And the third account from Mastodon.top with malicious links in their bio has just followed me.

@thomas_virtubox I'm tagging you in on this since you're the admin of the instance.

The account in question is:

@predinnnanin

I have reported and blocked this account as well as the other two.

#MastoAdmin #MastoAdmins #Mastodon #FediBlock

Not sure what's going on here with Mastodon.top and the accounts being created, but this is the second one in just over an hour that has followed me that has a dangerous link in its bio.

I have reported @poloraticw now as well.

Might be time to contact the admin of that instance and let them know what's going on.

#MastoAdmin #MastoAdmins #Mastodon #FediBlock

10M users is an exciting moment. Congratulations to the @Mastodon team, and all of the wonderful #MastoAdmins out there running the boxes of the fediverse!

From: @mastodonusercount
https://mastodon.social/@mastodonusercount/110051957865629817

If you run your own Mastodon instance, this GitHub action (can also be run locally with cron) looks really promising. https://fedi.aeracode.org/@andrew/110029009466041431

#mastodon #MastoAdmins #selfhosted #GitHubActions

Sharing a few thoughts on hosting a single-person Mastodon instance. I'd love to hear from others with relevant experience!

https://stefanbohacek.com/blog/lessons-learned-from-running-a-single-person-mastodon-instance/

#mastodon #MastoAdmins #MastodonAdmins #selfhosted #fediverse

Are you running your own Mastodon instance? How's your experience been? Any tips you'd share with folks wanting to give this a try? (Responses might be included in a blog post I'm working on.)

#mastodon #MastoAdmins #MastodonAdmins #selfhosted #poll

For #MastoAdmins out there:

Go to /sidekiq/queues on your personal instance and check to see how your Sidekiq process is handling the traffic.

Ideally, it looks a lot like this screenshot:

In particular, the Size and Latency columns at 0.

If these are routinely large in size (or if the Latency column says something crazy like 4 hours) you need to tune Sidekiq or upgrade your server, or both.

Can anyone provide me with links to any site that they feel one should list their Mastodon instance on so that others can easily find it if they are looking for a new instance?

I know servers are listed on https://joinmastodon.org but I'm not finding a way to add my instance there so others can discover it.

Surely there are other places to add your instance for discovery as well.

#NewInstance #AllThingsTech #MastoAdmin #MastoAdmins

THREAD...

I feel at this time, I'm ready to open up https://AllThingsTech.social for anyone who is looking for a new instance to join.

Few things to note here for everyone.

1) All accounts must be approved by me personally before you are approved. If I know you/follow you/you follow me, and you want to be a member, please let me know. Sign-up requires you to enter a reason why you're requesting to join.

#NewInstance #Mastodon #MastoAdmin #MastoAdmins

I'm sure a lot of you have seen me mention AllThingsTech.social over the past week or so and have been asking yourself, what is AllThingsTech.social exactly?

You can find that answer here via the first blog post:

https://blog.allthingstech.social/what-is-allthingstech-social/

#MastodonInstance #Mastodon #MastoAdmin #MastoAdmins

For anyone who followed my @CliffWade@allthingstech.social account, I have some bad news. It looks like I'm going to have to redo things from scratch due to some issues.

I'll be working on that over the next several days to get it back up and running again.

Sorry for any inconveniences, but I'd rather have these issues now before there are other users there that get affected by this.

#NewMastodonInstance #MastoAdmins #MastoAdmin