We already run CI since 2014, first via Travis CI, then when it became
paid we switched to self-hosted BuildBot in 2019. Later that year
GitHub Actions came also in existence, but we already switched over to
BuildBot by then so didn't use it.
We will still use BuildBot on self-hosted to test various Ubuntu and
Debian distro versions, FreeBSD and Windows. Also, in the BuildBot we
have our own pre-build environment where we run Services tests (with
both anope and atheme), we run TLS there (again on all those distros
with various OpenSSL versions). And we also test both clang and gcc.
So what is new? Well, now we will also run a "quick test" via GitHub
Actions, like most projects out there on GitHub. Not the services test,
not the TLS tests, but simply latest Ubuntu and then clang+gcc.
The main benefit of this is that it will also show up on Pull Requests
and makes it "public" as our BuildBot page is restricted.
Previous was way too confusing where user was actually a client and
channels was actually a membership struct. And then you got like
user->user and channels->channel. No, let's make this conform to
the same style that we use elsewhere. Who the hell wrote this !??
Oh, it seems I did :D
1) Similar to sendto_local_common_channels() go through local_members
instead of all channel members
2) We have the membership info, so use user_can_see_member_fast()
This mainly affects MODE #channel +vhoaq and such. And as with all these
optimizations it mostly affects channels with more than 10 people (eg
hundreds or thousands).
Also did add_member_mode() and del_member_mode() but those are not
used by our own code because we always use fast versions anyway.
Oh yeah and the +D invisibility shit via set_user_invisible():
that one i didn't benchmark but should be better as well for
large channels.
Only downside is that mtags would be NULL, but we don't use it in
cmd_uid() so that's okay. This saves us from generating mtags in
do_cmd() when we don't need it. And also a command handler lookup
and all that. Saves around 8% of unrealircd CPU for 100k connects.
synched. Both need to be checked, because:
* The "far" server may be fully synched to "near" (and thus tagged as synced)
but the "near" server may be introducing the "far" server, when
we are connecting to "near"
* The "near" server may be fully synched but the "far" server is connecting
in and may thus not be synched yet
In reality, things are even more complex, since one would have to verify
the whole chain of links. But.. yeah.
Long-story short: this fixes things like "User xyz joined #xxxxx" logging
where this showed up while the server was linking in. It is not supposed to
log that, similar to how we not log all 1000 users as newly connecting when
a 1000-user-server links in. In fact, it didn't already log that for
directly-connected-servers, but for far servers it did previously.
And... that again gave performance issues if you were connecting like a
100k-user far server.. since you suddenly had 100k * numchannels join events
being logged (which surprisingly still only took 6 seconds for 100k entries,
but still, it is wrong to do so and can be avoided).
This didn't show up in initial profiling, but now that other areas
are faster, this one starts to show up with 15% for 100k-clone remote
server traffic. Easy change :D
This also makes them proper list items, again to make certain fast operations
possible. Main thing is that removing an entry does not require us to walk
all of those lists. Not all code has been modified yet to benefit this,
actually only very little, the most performance-impacting ones.
This fixes SQUIT of a server with 100k users in a single channel taking
40 seconds of 100% CPU. It now takes only 1 second.
Reported by craftxbox in https://bugs.unrealircd.org/view.php?id=6484
(Can't make member & membership one entry atm, that would be too much change in U6)
The first one iterates through all channel members (can be hundreds or thousands)
and the latter goes through the channels a user is in (typically <15).
This so we can use fast(er) techniques here and there.
New functions are:
channel_has_invisible_users(client)
set_user_invisible(client, channel, 1|0)
Existing functions:
invisible_user_in_channel(client, channel)
user_can_see_member(user, target, channel)
user_can_see_member_fast()
This is work in progress, although the tests seem to pass atm.
When the channel is +H we broadcast to all servers, so we can simplify
that case and don't need to iterate the channel->members.
The same is true if set::broadcast-channel-messages is set to 'always',
though that is not known to be used much.
This too should be a significant performance improvement for multi-
server networks. Especially since this isn't just num_channel_members
of 1 channel, but about common channels, so could easily be like all
channel members of 10 channels combined.
This function is used for NICK, QUIT, and notification for CAP-
enabled clients for setname, account and away changes.
This makes things a lot faster on multi-server networks, especially for
big channels where most of the clients in the channel are remote users.
This should be non-module-API-breaking, as all code uses the
add_user_to_channel() and remove_user_from_channel() functions.
Still need to spread this to other code, more optimizations possible.
That is, during my tests with 1000 TLS clients doing a couple of commands,
including one big one (WHO #channel on a 1000 user channel).
I also tested an SSL_writev() implementation (which would gather up to 16k)
but it gives very comparable speed and caries more risk of doing so in a
stable series. I think we can live with the 4 kilobyte extra per local
client in the year 2025 (and later).
The whox one saves a lookup for each channel member (so eg 500 for a
channel with 500 members). The extended-monitor saves it on delivering
watch/monitor notifications, so depends on the # of subscriptions.
And that's each time such a command is called. We now only lookup on
MOD_LOAD.
things by making the keys with the most lookups first, e.g. "reputation",
"geoip", "certfp". This order is based on actual lookup counts during a
quick test with 250 clones doing some typical IRC traffic.
Key: Lookups: Position before: After split: After split+order:
"reputation" 20362 37 14 1
"geoip" 10555 44 15 2
"certfp" 9264 23 8 3
"webirc" 7407 27 10 4
"websocket" 7110 55 19 5
We could also consider going for a hash table, but this may be "good enough" for now.
up moddata_client_get() etc -> findmoddata_byname().
Apparently we have 52 moddata registrations (that is without 3rd party modules)
so otherwise it is a loooong linked list.
This was done in lr_pre_command() and lr_post_command().
Nowadays we have BIGLINES stuff from servers that cause MAXLINELENGTH
to be 16k, so the LabeledResponseContext ended up being 16k+.
Although we normally have the policy to zero out complete structs
in UnrealIRCd instead of only individual members (for safety,
easy to overlook security bugs), in this case we will do zeroing
of struct members explicitly. Added some warnings about this too
in the source code. Zeroing 16k twice for each command is a bit
too much waste.
simply by re-using the context.
The slowdown happened due to commit a541b8f4ad
in June 2021 when converting to OpenSSL 3+ code. Now it is basically
back to the pre-openssl-v3 speeds.
and find_user_mode(). That's one array of 256 elements, instead of
iterating a linked list where - if you are unfortunate - one may
need like 26 iterations.
In sendto_channel() we did the check for user mode +T before the
sendflags & SKIP_CTCP, that makes no sense and caused useless CPU.
We now do it the other way around, and also only lookup the user
mode just once (if needed).
The umode_letter_to_handler[] code may crash, it is not well tested
yet, only had two runs so far. Seems to work ok even with REHASH tho,
but have not tested delayed module unloading for example.
This was previously a "long", which could cause issues on 32 bit archs.
We ship with 28 CAPs now, and that's without 3rd party modules, so...
This is similar to the client->flags bumping in 2023
(a3ed1eabd9).
This fixes something like TLINE ~country:us not automatically converting
to ~country:US, since previously conv_param() was not called. But it also
means other code is used in the same way as GLINE (other type of rejections),
for example invalid server ext ban will print a better error with syntax
info (e.g. TLINE ~certfp:xx).
That ~country issue was reported by adamus1red in https://bugs.unrealircd.org/view.php?id=6581
Bram Matthys