This wasn't done before, because optimizing stuff can always introduce
nice new issues. But is kinda necessary now since the previous way was
very inefficient. This now builds all the necessary buffers for multiline
clients and for non-multiline clients. And then iterates through both
types of clients, sending what they need. Instead of doing it the other
way around.
I had the dillema to either expose the linecache API and have everything
in multiline.c. Or, i do not expose linecache, and we do everything in
send.c. The downside of the latter is that if there is mistake then we
can't simply reload (or unload) the module to solve it. So, I have chosen
to expose the linecache API (sure, less clean) since that leaves us with
options if we screw up, plus it means everything related to multiline
sending is nicely in multiline.c, which is i guess just as good as an
argument as well ;)
Add a little fake lag based on history result: 400ms for 50 lines
under normal conditions where 50 lines = 50 lines. But this can go
up to 5000ms for worst-case amplification attacks where requesting
50 lines actually returns 50*15=750 lines when each line is a multiline
with max-lines, which gets you close to 350k+. This would only happen
if someone on the channel is doing evil stuff (with presumably consent
of the ops).
Also guard against hiting max sendq. If we are too close, then we
reject the CHATHISTORY request rather than quiting with "Max SendQ
exceeded". This protects against an attack where someone would be
tricked into joining a channel with amplified history (as explained
in previous paragraph), their client would do an automatic CHATHISTORY
request and then the victim would exceed max sendq and thus be killed.
And yes, this and maaaaany other multiline + history interactions
and many "buts" and security/flood concerns are why this implemtnation
took (and still takes) a lot of hours to get right :D.
For +H we now temporarily allow overshooting. This only matters for low limits.
Multiline batches are atomic so we have to choose to keep them as a whole
or remove the complete batch. So if +H 5:1h and the last message was a 15-line
multiline event, what do we do? We allow temporary overshooting to store the
15 lines. As said, the alternative would be to store 0 lines which would be
worse in terms of functionality, and the small overshoot is defensible.
For higher limits (where the +H line limit is bigger than multiline max-lines),
we always stay under the +H limit. Eg if all history in a channel consists
of 15 line multiline events and we have +H 100 then we will store 90, not 105.
It's only for +H linelimit < max-lines that this matters, because there the
zero-lines consequence sucks too much ;)
This way you can limit the number of pastes going on in a channel, as
this is from everyone in that channel (like 'm') not individual (like 't').
If it is exceeded then we will simply reject the BATCH, similar to
how action d(rop) works for some other subtypes. You won't see the paste
on the channel, only the sending user receives an error (MULTILINE_PASTE_LIMIT).
Small note: a multiline BATCH of just 2 lines is not considered a paste.
We consider a multiline of 3+ lines as a paste. I think that is reasonable,
since a two-line-multiline is not that much of a paste ;).
In the default anti-flood profile (+F normal) we also set 2p per 15s,
so this means channels are by default limited to 2 pastes per 15s max.
Of course, you can override this with +f [4p]:15 or whatever you like.
In terms of +F profiles, the defaults are (maximum x pastes per 15 seconds):
very-strict: 1p
strict: 1p
normal: 2p
relaxed: 2p
very-relaxed: 3p
and 7 for unknown-users (with max-bytes 5250 and 1500 respectively). This
allows pasting a short snippet of code, config file, text from a site, etc.
With multiline you have the guarantee that:
1) You will see the entire text with no delay between lines
2) You won't see another persons chat half-way through such a paste
3) For multiline supporting clients it is now clear that all the text
belongs to each other, which can make selecting/copying it easier.
This basically means short snippets/pastes like that can be completely on
IRC again. No need for a pastebin for it. Though, you may still need such
a service if you are pasting more lines.
Regarding the implementation in UnrealIRCd:
* Clients without multiline get individual fallback lines (concat lines
merged, blank lines skipped, as per spec). And we know that clients like
weechat - which does support multiline - also shows all lines and not
only a few plus snippet style "[.."]. That is another reason for only
allowing 15 lines by default and not something much more. Otherwise all
those clients would get a big wall of text, which just sucks.
* Spamfilter (also) runs on the full text of all lines together, so
splitting a phrase across lines does not evade spamfilter.
* Fakelag: a client can send the BATCH start+PRIVMSG (or NOTICE)+BATCH end
at full speed. We impose no fake lag there. Also, the multiline default
max-lines and max-bytes are lower than the example class::recvq of 8000,
so should be perfectly safe. If the entire BATCH is accepted then we
will impose fake-lag afterwards, with a cap of 15 seconds maximum.
If the BATCH is rejected, we impose half the fakelag plus 2sec.
* If the time between BATCH start and BATCH end is more than 15 seconds
then the BATCH is rejected (set::multiline::batch-timeout).
* The BATCH is atomic (either you see it all, or you see none of it):
* When the client sends it to server, it is buffered first.
* Only after the batch close the server indicates if it is accepted
or rejected. This has various reasons, two of them are: 1) The client
is going to send everything in one go anyway and not wait for a
response between each PRIVMSG, and 2) we can't do many checks in the
buffering stage and skip those after, that would cause a TOCTOU
problem (eg. a banned user still being able to speak).
* If any line gets rejected due to spamfilter or other case
(eg +c, +b ~text with block, etc etc), the entire batch is rejected
* Locally we deliver all or nothing (as said)
* S2S we buffer the batch as well, so if a server splits after having
received 10 lines out of 15, then clients will not see anything.
* We send max-lines and max-bytes, this is the hard upper limit.
* A multiline can still be limited more tight if:
* +f with 't' or 'm' restricts to fewer lines,
eg +f [5t]:15, which means max 5 lines per 15 seconds,
means the max accepted multiline is 5 for that channel.
* +F works the same, except that default +F normal does not
have a 't' at the moment and 'm' is very high (50) so
practically not limited by default.
* There will be a future +f flood subtype for some more control
TODO: we will send CAP NEW on unknown-users <-> known-users to
indicate the new max-lines value if you transition security groups
TODO: chat history does not yet include multiline batches.
This is mainly due to licensing. The libmaxminddb library uses the
Apache license, which meant if we would compile it in by default it
would effectively transform our "GPLv2 or later" to "GPLv3 or later".
Our implementation is ISC licensed, so we can include and enable it
by default and keep things at "GPLv2 or later". This is also why we
used geoip_classic in the first place as default and compiled in,
and not the mmdb variant.
The mmdb.c is based on the specification, using the Go implementation
as a reference during development (ISC licensed), initially implemented
with the help of Claude Opus 4.6. After that substantial changes were
made to make it match UnrealIRCd's style and to make things less error
prone: C style changes, allocation and zero termination of strings in
the library, auto-NULL in variadic functions so the caller cannot
forget NULL there (similar to our unreal_log/do_unreal_log), using
enums as the return type instead of int (similar to curl), adding
doxygen docs, etc.
This also means the old mmdb library dependency has been dropped,
including from configure/autoconf.
At the moment we still use the geoip classic library by default,
including those DB files. The idea is we will switch over sometime
later after this current new MMDB stuff has received more testing.
This also makes us more flexible, since .mmdb files have become the
de-facto standard for pretty much all geoip vendors.
This module is rarely used but analysis showed that there was an
OOB write in the country name, and two small off-by-ones in code
and continent.
Again, this only matters if the CSV file you are importing is bad
or malicious. And we use stack protection in UnrealIRCd so this
should then "only" cause a crash.
For the extbans that we ship, no problem, as this isn't used in
any of our extbans, but for third party it may matter, or for us
in the future.
Just something we came across while looking into the issue from
previous commit.
This affects servers without NEXTBANS, such as anope 2.0.x series
(anope 2.1.x is not affected as it supports NEXTBANS).
Non-NEXTBANS servers only support letter extbans so we are supposed
to convert ~security-group:known-users to ~G:known-users when sending
to such a server, in unreal_server_compat. And we did this well for
the MODE command for +beI. In SJOIN we did this correctly for +b/+e
but not for +I due to a silly code mistake.
This bug is present since 6.0.0 but wasn't noticed until now.
To be a real problem you need something like:
1. Anope 2.0.x series (or other services without NEXTBANS)
2. A channel with +I extbans
3. KEEPMODES set on that channel
Then what happens is when services boot:
1. UnrealIRCd will sync with anope 2.0.x and incorrectly send
named bans, which will confuse anope. But nothing strange
happens yet at this point.
2. Then on next server sync (eg anope restart or unreal restart)
anope will try to restore these but they end up with weird
entries like +I *!*@~security-group:known-users
(note the *!*@ prefix)
And it should be noted that this would also happen in a situation
with UnrealIRCd 5 + UnrealIRCd 6 servers, but UnrealIRCd 5 is
End Of Life anyway.
Reported by BlackBishop and Sadie two days ago. Thanks!
NOTE: Linked servers are considered trusted in UnrealIRCd.
This is not exploitable beyond a crash, due to -fstack-protector-all,
a hardening compiler flag we added many years ago. Even without
that flag it would be rather difficult, and i didn't manage to,
but this should never happen anyway since this flag is only
missing in gcc/clang versions that are more than 15 years old.
This issue was introduced by the move to CMD_BIGLINES in
6c5de62c18 in 6.2.2 release.
In particular, this disables default +F for #__SYNC__ channels.
The test suite has a "+F off" but when on 3 servers, each 75
clones are connecting, the MODE is too late and the join limit
is already reached sometimes. Causing tests to fail.
because it has no internet access, like when fetching the repository
(modules.list file) of 3rd party modules.
Previously I had..
url_start_async(request);
synchronous_http_request_in_progress = 1;
.. which worked fine for the "cannot connect case", like port blocked
or timeout connecting. But if DNS fails then the step of setting
synchronous_http_request_in_progress = -1 (so failed) already happens
during the url_start_async(request); call, and then the line after it
sets 'synchronous_http_request_in_progress = 1;' so we miss that it
failed and wait in the I/O loop forever.
Simply swapping the two lines of code fixes this.
The other change is that when running the ModuleManager in "make" we should
ignore the exit code. I probably broke that while refactoring and adding
non-zero exit codes in de modulemanager past few months for this release.
in NameList, Tag, Watch and HistoryLogLine.
This does mean the allocation routines need a +1 everywhere, but
I think I got all of them. I also don't see them being used directly
in such a way in 3rd party modules (which is logical, as they
should use the API and not allocate such structs directly).
Also, SpamExcept has been removed as it was not used anywhere.
the particular extended ban module if you don't want it.
For example, if you include the default modules.default.conf and, say,
you don't want ~quiet extbans then you add this in your unrealircd.conf:
blacklist-module "extbans/quiet";
This would cause a bit of a mess, that usually would be resolved a few
seconds later, but still a mess. I had this on irc*.unrealircd.org
myself when rerouting a server from a backup-hub to primary-hub
a few months ago.
This is not an issue now in all code paths, but if someone accidentally uses
SupportXYZ() without checking IsServer() then it would be an issue.
In the past we used client->local->proto for client flags as well, but this
has been split off to client->local->caps a while ago.
I guess we should rename client->local->proto to something more server-ish
in a later major release to indicate this as well.
and callback data in non-DEBUGMODE. Also because exposing pointers like
this can defeat ASLR. These STATS are oper-only though, but hey, defense in
depth... and the pointer values don't make sense to non-devs anyway,
so why show them in the first place.
We use mtag_add_issued_by() to prepare it but then pass NULL
in do_cmd() so it was basically useless.
Also compile fix for previous (forgot to git ammend)
Previously it showed this warning and said "Allowing user .. in unchecked"
when the user got shunend by CBL. Usually harmless but.. had a report
where it possibly was not (though that was an older UnrealIRCd version).
In any case, confusing, solved now!
Reject it with an ERR_INVALIDMODEPARAM, just like we do for +k.
I think the higher number transforming is fine, but this <=0 transformation
is odd as it almost never is what the user actually intended.
In S2S traffic we still transform, as rejecting there is more problematic,
(causing a desync) and transforming it there is not a major issue, anyway.
Reported by ProgVal in https://bugs.unrealircd.org/view.php?id=6602
since the message/notice would not make it through either.
This also means someone can no longer iterate through users to see who
is +D/+R by sending a "silent" TAGMSG. (Silent in the sense that the
end-user usually would not have noticed)
Suggested in https://bugs.unrealircd.org/view.php?id=6579 by zw32h (I think)
This also means HOOKTYPE_CAN_SEND_TO_USER now allows you to NOT to
set errmsg, to silently drop a message. Previously we would crash
deliberately on such a situation to enforce that all modules would
set a proper errmsg.
Bram Matthys