The Italian instance Friendica "Poliverso" apologizes to the whole fediverse! (long post)
Last night, between 23.00 on 23 June and 1.00 AM on 24, the Poliverse instance "exploded" a thousand messages around the fediverse!
** We have to apologize to a lot of people: **
• we apologize to anyone who was ** woken up by our notifications **;
• we apologize to those who have cisted their ** federated timeline invaded by messages ** coming from Poliverso;
• We apologize to the ** administrators and instance moderators who know us ** who probably wondered who hacked our server;
• We apologize to the ** administrators who didn't know us ** and who noticed us in the worst way
• We apologize most of all ** to some of our users ** who, through our fault, have ** UNKnowingly ** behaved as a spammer ...
Unfortunately, we could not foresee this collapse and we did everything possible to limit the damage: unfortunately we are sure that our instance or some of our users have ended up in some blacklist.
** If so, we ask you to read this message, in which we explained what happened. **
PS: if any Friendica expert should realize that some considerations are wrong or should be better defined, please let us have your own considerations in the comments!
What happened to the Poliverso server?
The update
The new Friendica release, the 2022.06 called “Giant Rhubarb” is a seemingly useless release, because nothing changes for users! Yet it has greatly improved compatibility with other software in the fediverse: there has in fact been immense work by the developers and there have been many changes; in such cases it is possible that disservices arise and, although some instances have already installed an exploratory version, it is clear that some problems can only emerge during the production phase.
Precisely for this reason we decided to proceed with the installation only several days after the release, to understand what the problems would be for users and notify them in time; and yesterday 23 June at 22.00, the Poliverso server was updated.
The problem
At first it didn't seem like there were any problems and, in fact, after half an hour we stopped monitoring. But at 11.00 something happened ...
Admin 1
22.56 (the developers of friendica) have changed something: some users get a lot of notifications about old things .. then everything is fixed
22.56 (I'm seeing a lot of queue in processes)
Admin 2
23.00 So far I am not having this type of problem. Actually, I am not having any problems
23.01 To be more precise, I still can't see anything different than before ... 😅
23.04 Right now, after a few tens of minutes of pause, he's posting stuff on Twitter
23.07 Holy shit! my accounts are flooding the universe with messages
Admin 1
23.14 in what sense? : D
Admin 2
23.15 in what sense? The system is shitting out all the messages posted by the RSS Feed. It is a disaster. I'm emptying the ocean with a spoon 🤣
23.15 This time we are banned from all the galaxies of the fediverse
Admin 1
23.16 I noticed a lot of RSS stuff, but I said to myself "boh"
Admin 2
23.16 They are the queues of those messages
In short, yesterday there was a real disaster for which the Poliverse instance will pay the consequences for a long time as in reputational terms. However, the management of the problem was absolutely timely both from the point of view of resolution (at 1:00 AM almost all the messages had been deleted) and from the point of view of communication.
https://poliverso.org/display/0477a01e-4062-b4e8-2a73-375308279474
The reasons for the problem
However, the problem was not systemic or due to problems of administration of the instance; the problem was due to a series of factors that multiplied its impact:
- Friendica upgrade bugs exist: the "system" upgrade all together, some integration problems with other activitypub instances; but all this causes a "re-reading" of the notifications from the RSS FEED; in this way, users who also repost some posts from the feed get** a new repost (😱!)** of the feed.
- this system (republishing from feed) is not very common among Friendica users; so the problem was not noticed too much on the forums
- this system, however, is very powerful and we Poliverse administrators have often sponsored it with users: therefore some of our users use it; and some "service" and information users that we manage directly on a personal level or as administrators also use it: [@ macfranc @ poliverso.org [(https://poliverso.org/profile/macfranc), [@ informapirata @ poliverso .org [(https://poliverso.org/profile/informapirata), [@ news @ poliverso.org [(https://poliverso.org/profile/notizie), [@ cybersecurity @ poliverso.org [(https: //poliverso.org/profile/cybersecurity), [@ privacypost @ poliverso.org [(https://poliverso.org/profile/privacypost), [@ piratepost @ poliverso.org [(https://poliverso.org/ profile / piratepost), [@ instruction @ poliverso.org] (https://poliverso.org/profile/ Formazione)
- inside Friendica there are no valid administration or moderation tools (an ancestral and well-known Friendica problem: https://github.com/friendica/friendica/issues/8724), which would allow us to delete messages in bulk or to immediately stop the republishing service from feed or to temporarily suspend the "unwittingly spammy" users. So we either manually deleted and deactivated everything or had users manually delete their messages - this was an incredible waste of time!
The reputational damage
All of this prevented us from predicting the problem and then amplified it. However, we can assure you that the management of the application was professional and within a couple of hours from the alert, we have erased all traces of the problem!
The fact remains that some users who followed the accounts in question probably wondered "what's going on?"; others have probably defollowed, silenced or banned the accounts of innocent and unaware users ...
The same thing will have happened for administrators: some of them, to protect their users from spam, will have silenced or banned users; or they will have silenced or banned the instance!
Fortunately, those in the Central European Time Zone were probably asleep at the time and were not online on Friendica, Mastodon, Misskey, Pleroma or Hubzilla
Final thoughts
Of course during those hours we felt [a furious rage] (https://poliverso.org/display/0477a01e-1062-b4db-9d28-f53436523782) towards the Friendica developers who caused us such a problem...
https://poliverso.org/display/0477a01e-1062-b4db-9d28-f53436523782
...but we clearly don't get mad at them. Moreover, it is thanks to them that Friendica exists; it is thanks to them that there is what we believe is the software of the fediverse that allows the most complete social experience today.
Next time, however, we recommend that you report the problem well and above all, express at least a little regret to the Friendica instance administrators and other administrators who had to manage what seemed to all intents and purposes a massive spam activity!
But above all, we send a message to all instance administrators and users who are reading us: if you have defollowed, silenced or banned some Poliverse users, think again! They had no fault, other than to use the RSS feed reposting system: give them a second chance!
Feature Request: implement Moderation system · Issue #8724 · friendica/friendica
I think it would be good to have a moderation system similar to Mastodon where you can report problematic content, an admin will be notified and decide what to do with that reported post: warn the ...GitHub
like this
Tio and BlueTardis like this.
reshared this
Liwott reshared this.
Tio
in reply to Poliverso • •Poliverso
in reply to Tio • • •Tio
in reply to Poliverso • •Poliverso
in reply to Tio • • •Tio
in reply to Poliverso • •Poliverso
in reply to Tio • • •In fact, the additional problem is that if a mastodon, or pleroma or misskey user reports a Friendica content as if it were spam, the administrator would not even notice it, because there is no moderation interface in Friendica ...
And this can expose Friendica instances to ban.
But that's another problem 😅
PS: However, I take this opportunity to tell you that I really like your instance and I really like how you manage it but also all the fantastic posts you wrote on friendica
Tio likes this.
Tio
in reply to Poliverso • •Ephera
in reply to Tio • • •Liwott likes this.
Tio
in reply to Ephera • •Signor Amministratore likes this.
Ephera
in reply to Tio • • •A user on https://mastodon.technology can migrate to https://fosstodon.org and will generally find the same people + whoever may have been silenced by more aggressive moderation.
Same thing the other way around. Most users don't care about seeing everything, and may just find it more convenient when trolls and assholes are already filtered out for them, so may migrate from Fosstodon to mastodon.technology.
No, this is not a perfect system on an individual user basis, since most users won't really know what they're not seeing.
But I do think, in the grand scheme of things, people will bundle up around instances that offer a good balance between interesting content and trolls/assholes.
Liwott likes this.
Tio
in reply to Ephera • •The solution is to, for one, make all instance-level blocking of users or other instances transparent and notify all users when any future such decision is taken. And second, try not to panic. It is fine. Words, no swords, we are dealing with. Users have a lot of control over that they see. Teach them to exercise that. This is a far better approach.
Signor Amministratore likes this.
Signor Amministratore reshared this.
Signor Amministratore
in reply to Tio • • •The measures of the administrators of the instance and of the moderators of each community are made evident. This is a very useful thing to understand how an instance is administered and understand if the moderators practice abuse or support a balanced moderation aimed at maintaining a plural and peaceful environment.
https://lemmy.ml/modlog
Tio likes this.
Tio
Unknown parent • •Signor Amministratore likes this.
Tio
Unknown parent • •Darren likes this.
Liwott
Unknown parent • • •Alternatively, there could be a public modlog with a RSS feed (granted, not all fedi software allow to follow RSS feed :) )
Informapirata
in reply to Liwott • • •https://lemmy.ml/modlog
https:// feddit.it/modlog
Liwott likes this.
Informapirata
Unknown parent • • •These instances are forced to silence and sometimes block users who post content of this type without the content warning tag.
This can be considered excessive, but it is the only way to keep an environment free of disturbing elements such as overly fiery political speech, explicit content, foul language.
All this concerns above all the generalist instances.
As for the others, it becomes more rare to practice BAN and silencing.
Unfortunately, however, there are two types of drastic measures: those towards users of your own instance and those towards users of other instances.
In my opinion, the latter should undergo a periodic review, a sort of amnesty that occasionally cleans up the sentences imposed or at least reconsiders some users who at the beginning of their life in the fed... show more
These instances are forced to silence and sometimes block users who post content of this type without the content warning tag.
This can be considered excessive, but it is the only way to keep an environment free of disturbing elements such as overly fiery political speech, explicit content, foul language.
All this concerns above all the generalist instances.
As for the others, it becomes more rare to practice BAN and silencing.
Unfortunately, however, there are two types of drastic measures: those towards users of your own instance and those towards users of other instances.
In my opinion, the latter should undergo a periodic review, a sort of amnesty that occasionally cleans up the sentences imposed or at least reconsiders some users who at the beginning of their life in the fediverse behaved badly because they were used to the toxic environment of Facebook or of Twitter.
In fact, the problem with these users is that they risk being invisible for life from some instances.
Liwott likes this.
Liwott
in reply to Informapirata • • •Yes I probably had it in mind when saying that (I mean I did not use the word "modlog" for free :) ). All it misses to fit @Tio 's ideal is a RSS-like feature (that could be an actual RSS feed)