Basically if a $variable is empty, and there is a space before it in the
template string then we delete that space.
May seem (or is) a bit over the top but this way the template stays clean,
and it may be used/useful in other places as well.
This is a behavior change, but I think we can live with it. One can opt-
out via BUILDVARSTRING_KEEP_SPACE_FOR_EMPTY_VAR.
TKL_EXPIRE and SPAMFILTER_MATCH messages.
This uses the newly added functions log_data_optional_string() and
log_data_optional_name_value(). The first shows the optional string
like "abc" and the second expands to "[name: value]". What's also new
is that both of these will swallow a preceding space if there is no value.
This so you can just use "Something. $optional_string" and it will
expand to "Something." if $optional_string is empty. This makes things
less hacky and more human readable :)
Unlike non-config-based TKLs - which go through tkldb - they are still not
preserved through restarts. But at least they are not lost due to REHASH.
This is done via a save+restore, a bit complicated, but we have little
choice (other than not doing this at all).
This also moves remove_config_tkls() from conf.c to tkl.c
of the last hit, eg in `STATS gline` for GLINEs. These counts happen on
each individual server and are not network-wide. This allows IRCOps to see
which entries never get any hits and can potentially be removed.
* Important exception: config-based spamfilters/bans lose their counters
on `REHASH` and restart atm.
* For non-config TKLs, the hit count and last hit timestamp are preserved
across reboots (via tkldb).
* Again, see *Developers and protocol* for the exact STATS field.
The spamfilter hits already existed but all the rest is new.
Suggested by BlackBishop in https://bugs.unrealircd.org/view.php?id=6304
(in particular, time of the last hit)
By default - assuming you don't set set::reject-message things by yourself -
the *LINE id is appended at the end of the rejection that is shown to the
user, like: [ID: G7K2MP9WQX3].
Also new is spamfilter to *LINE mapping, so you can see which *LINE was
set by which SPAMFILTER. For this STATS gline and friends were enhanced.
In fact, multiple fields were added there, including some that are 0
(zero) placeholders at the moment. These will be set in a future commit.
Some things were combined here so we only have to break STATS and tkldb
database format once (unless i made a mistake, then the follow up commit
will correct that i guess :D).
This was requested by Hero in https://bugs.unrealircd.org/view.php?id=4397
in 2015. Again by musk in https://bugs.unrealircd.org/view.php?id=4397
in 2022. And on IRC by Chris and others.
As you can see it was not SUPER easy and a lot of thought went into this
(and in terms of S2S traffic it is part of something bigger too)
And the defines are more clear now (if .max_size is not set by caller.
DOWNLOAD_MAX_SIZE_MEMORY_BACKED: 1M
DOWNLOAD_MAX_SIZE_FILE_BACKED: 50M
The file-backed is mostly a defense-in-depth measure, so we don't
store infinite amounts of data in a download. Even though, in practice,
these - at least at the moment in unrealircd itself - all come from
trusted paths like remote includes.
In url_unreal.c we do the counting ourselves. In url_curl.c we use the
option CURLOPT_MAXFILESIZE_LARGE but this does not ensure it in all
cases so we still do our own counting as well in that file as well.
And ship with these by default (no need to copy this set block):
set {
log-throttle {
CONNTHROTTLE_IPV6_LIMIT 100:60;
MAXPERIP_LIMIT 100:60;
};
};
You can do the same for other events, or even override existing ones,
and use the special value "unlimited" to turn default set ratelimits off:
set {
log-throttle {
CONNTHROTTLE_IPV6_LIMIT 50:60;
MAXPERIP_LIMIT unlimited;
};
};
Suggested in 2020 at https://bugs.unrealircd.org/view.php?id=5523
(and keeping it simple)
When a client is rejected by maxperip (not new) or connthrottle
ipv6-unknown-users-limit (that one is new), a notice to +s +x will be sent.
maxperip ipv4 example:
*** Client testuser4 with IP 1.2.3.4 rejected: maxperip limit exceeded (4 global, max 3)
maxperip ipv6 with /64 example:
*** Client testuser4 with IP 2001:dbe:0:0:0:0:0:4 rejected: maxperip limit exceeded for 2001:dbe::/64 (4 local, max 3)
connthrottle example where /56 limit is exceeded:
*** Client testuser5 with IP 2001:db8:cafe:abcd:0:0:0:5 rejected:
connthrottle ipv6-unknown-users-limit (cidr-56, max 4) exceeded for
2001:db8:cafe::/56 (5 unknown / 0 excepted / 0 known)
Oh and this commit also fixes a typo in existing CONNTHROTTLE events,
which previously were CONNTHROTLE (a missing T).
Changed "Too many connections from your IP" to have "[maxperip]" at the end.
Also create new setting and swap it with existing-one-during-development.
Long story short, we now have 3 different messages for these limits:
set::reject-message::too-many-connections
"Too many connections from your IP [maxperip]"
set::reject-message::too-many-connections-ipv6-range
"Too many connections from your IPv6 range ($prefix_addr/$prefix_len) [maxperip]"
set::reject-message::too-many-new-connections-ipv6-range
"Too many new connections from this IPv6 range ($prefix_addr/$prefix_len) [connthrottle]"
So we explicitly mention whether it is maxperip or connthrottle limiting the
user, that should provide enough clue to the IRCOp if the user pastes the
message to them.
connect-flood and maxperip module. This so they actually take
set::default-ipv6-clone-mask into account.
This also changes the maxperip module to a more simple method of
just freeing all entries and rebuilding the hash table on load.
That's necessary since now set::default-ipv6-clone-mask can change.
The OOB write did not happen on file-backed downloads, such as remote
includes. It only happened for memory-backed requests, which are only
these 4 in standard UnrealIRCd: centralblocklist, central spam report,
other spamreport blocks (eg to dronebl) and the log block with
destination webhook. All those 4 cases are very likely to be trusted
web servers, given the nature of the data you are sending to them.
The fix was to extend the size fields everywhere to 64 bits. It was
applied to both URL backends: url_unreal.c and url_curl.c.
The new API feature is a 'max_size' in OutgoingWebRequest, which
defaults to 1MB. This is only used for memory-backed responses,
so not for real file downloads. This fixes not only the reported
bug but also the case where a rogue webserver was unbounded in
terms of what response it could send back, potentially filling
up gigabytes of server memory.
Reported by Link420.
There is no hard cap on batch reference length, so we had to make one up.
It is now a clear #define MAXBATCHREFLEN 48, which should be plenty.
No sane client is going to use like a 64 byte batch reference :D
So we did use 48, but we also accidentally used BATCHLEN at another
place. BATCHLEN is 22 and refers to how many bytes we generate, so
that is not appropritate.
Thanks to Valware for spotting this.
This wasn't done before, because optimizing stuff can always introduce
nice new issues. But is kinda necessary now since the previous way was
very inefficient. This now builds all the necessary buffers for multiline
clients and for non-multiline clients. And then iterates through both
types of clients, sending what they need. Instead of doing it the other
way around.
I had the dillema to either expose the linecache API and have everything
in multiline.c. Or, i do not expose linecache, and we do everything in
send.c. The downside of the latter is that if there is mistake then we
can't simply reload (or unload) the module to solve it. So, I have chosen
to expose the linecache API (sure, less clean) since that leaves us with
options if we screw up, plus it means everything related to multiline
sending is nicely in multiline.c, which is i guess just as good as an
argument as well ;)
Add a little fake lag based on history result: 400ms for 50 lines
under normal conditions where 50 lines = 50 lines. But this can go
up to 5000ms for worst-case amplification attacks where requesting
50 lines actually returns 50*15=750 lines when each line is a multiline
with max-lines, which gets you close to 350k+. This would only happen
if someone on the channel is doing evil stuff (with presumably consent
of the ops).
Also guard against hiting max sendq. If we are too close, then we
reject the CHATHISTORY request rather than quiting with "Max SendQ
exceeded". This protects against an attack where someone would be
tricked into joining a channel with amplified history (as explained
in previous paragraph), their client would do an automatic CHATHISTORY
request and then the victim would exceed max sendq and thus be killed.
And yes, this and maaaaany other multiline + history interactions
and many "buts" and security/flood concerns are why this implemtnation
took (and still takes) a lot of hours to get right :D.
For +H we now temporarily allow overshooting. This only matters for low limits.
Multiline batches are atomic so we have to choose to keep them as a whole
or remove the complete batch. So if +H 5:1h and the last message was a 15-line
multiline event, what do we do? We allow temporary overshooting to store the
15 lines. As said, the alternative would be to store 0 lines which would be
worse in terms of functionality, and the small overshoot is defensible.
For higher limits (where the +H line limit is bigger than multiline max-lines),
we always stay under the +H limit. Eg if all history in a channel consists
of 15 line multiline events and we have +H 100 then we will store 90, not 105.
It's only for +H linelimit < max-lines that this matters, because there the
zero-lines consequence sucks too much ;)
This way you can limit the number of pastes going on in a channel, as
this is from everyone in that channel (like 'm') not individual (like 't').
If it is exceeded then we will simply reject the BATCH, similar to
how action d(rop) works for some other subtypes. You won't see the paste
on the channel, only the sending user receives an error (MULTILINE_PASTE_LIMIT).
Small note: a multiline BATCH of just 2 lines is not considered a paste.
We consider a multiline of 3+ lines as a paste. I think that is reasonable,
since a two-line-multiline is not that much of a paste ;).
In the default anti-flood profile (+F normal) we also set 2p per 15s,
so this means channels are by default limited to 2 pastes per 15s max.
Of course, you can override this with +f [4p]:15 or whatever you like.
In terms of +F profiles, the defaults are (maximum x pastes per 15 seconds):
very-strict: 1p
strict: 1p
normal: 2p
relaxed: 2p
very-relaxed: 3p
and 7 for unknown-users (with max-bytes 5250 and 1500 respectively). This
allows pasting a short snippet of code, config file, text from a site, etc.
With multiline you have the guarantee that:
1) You will see the entire text with no delay between lines
2) You won't see another persons chat half-way through such a paste
3) For multiline supporting clients it is now clear that all the text
belongs to each other, which can make selecting/copying it easier.
This basically means short snippets/pastes like that can be completely on
IRC again. No need for a pastebin for it. Though, you may still need such
a service if you are pasting more lines.
Regarding the implementation in UnrealIRCd:
* Clients without multiline get individual fallback lines (concat lines
merged, blank lines skipped, as per spec). And we know that clients like
weechat - which does support multiline - also shows all lines and not
only a few plus snippet style "[.."]. That is another reason for only
allowing 15 lines by default and not something much more. Otherwise all
those clients would get a big wall of text, which just sucks.
* Spamfilter (also) runs on the full text of all lines together, so
splitting a phrase across lines does not evade spamfilter.
* Fakelag: a client can send the BATCH start+PRIVMSG (or NOTICE)+BATCH end
at full speed. We impose no fake lag there. Also, the multiline default
max-lines and max-bytes are lower than the example class::recvq of 8000,
so should be perfectly safe. If the entire BATCH is accepted then we
will impose fake-lag afterwards, with a cap of 15 seconds maximum.
If the BATCH is rejected, we impose half the fakelag plus 2sec.
* If the time between BATCH start and BATCH end is more than 15 seconds
then the BATCH is rejected (set::multiline::batch-timeout).
* The BATCH is atomic (either you see it all, or you see none of it):
* When the client sends it to server, it is buffered first.
* Only after the batch close the server indicates if it is accepted
or rejected. This has various reasons, two of them are: 1) The client
is going to send everything in one go anyway and not wait for a
response between each PRIVMSG, and 2) we can't do many checks in the
buffering stage and skip those after, that would cause a TOCTOU
problem (eg. a banned user still being able to speak).
* If any line gets rejected due to spamfilter or other case
(eg +c, +b ~text with block, etc etc), the entire batch is rejected
* Locally we deliver all or nothing (as said)
* S2S we buffer the batch as well, so if a server splits after having
received 10 lines out of 15, then clients will not see anything.
* We send max-lines and max-bytes, this is the hard upper limit.
* A multiline can still be limited more tight if:
* +f with 't' or 'm' restricts to fewer lines,
eg +f [5t]:15, which means max 5 lines per 15 seconds,
means the max accepted multiline is 5 for that channel.
* +F works the same, except that default +F normal does not
have a 't' at the moment and 'm' is very high (50) so
practically not limited by default.
* There will be a future +f flood subtype for some more control
TODO: we will send CAP NEW on unknown-users <-> known-users to
indicate the new max-lines value if you transition security groups
TODO: chat history does not yet include multiline batches.
As usual, this is mostly for configuration templates that you use for
multiple servers, that sort of things, eg.
@if !environment("ADMIN")
@error "Environment variable ADMIN is not set"
@endif
This also adds a change in conf.c so @define, @error and
@warning are skipped in @if blocks that evaluate to false
(that's obviously what everyone wants :D). So that fixes a
previous bug with @define in @if.
to check environment variables.
This also means functions can now return values, so some changes
under the hood. This also moves the <=, >=, <, > ops code.
loadmodule + set config items
This checks the file on-disk, which is slightly different than
@if module-loaded("third/coolmod") which checks if it is loaded.
geoip_classic and geoip_mmdb in modules.default.conf with Conditional
Config, a dynamic loadmodule line, and auto-updates.
Somewhere in a later version, probably 6.2.5, we will default to mmdb
for all cases.
This is mainly due to licensing. The libmaxminddb library uses the
Apache license, which meant if we would compile it in by default it
would effectively transform our "GPLv2 or later" to "GPLv3 or later".
Our implementation is ISC licensed, so we can include and enable it
by default and keep things at "GPLv2 or later". This is also why we
used geoip_classic in the first place as default and compiled in,
and not the mmdb variant.
The mmdb.c is based on the specification, using the Go implementation
as a reference during development (ISC licensed), initially implemented
with the help of Claude Opus 4.6. After that substantial changes were
made to make it match UnrealIRCd's style and to make things less error
prone: C style changes, allocation and zero termination of strings in
the library, auto-NULL in variadic functions so the caller cannot
forget NULL there (similar to our unreal_log/do_unreal_log), using
enums as the return type instead of int (similar to curl), adding
doxygen docs, etc.
This also means the old mmdb library dependency has been dropped,
including from configure/autoconf.
At the moment we still use the geoip classic library by default,
including those DB files. The idea is we will switch over sometime
later after this current new MMDB stuff has received more testing.
This also makes us more flexible, since .mmdb files have become the
de-facto standard for pretty much all geoip vendors.
even if it costs an extra round-trip due to HRR (Hello Retry Request).
This is IRC after all, where connections live minutes, hours, days,
so that extra round trip is worth it if it means better security.
The TL;DR is: we try harder to use X25519MLKEM768.
The longer story is as follows:
In TLSv1.3, the client will indicate which groups it supports (eg
a list of 4 items) and which ones it speculates to be used (very
often just 2 items). Some TLS clients may not include X25519MLKEM768
in this initial speculation, but only f.e. X25519 and prime256v1
even though X25519MLKEM768 is communicated via their "supported" list.
Without this patch, we would then settle with one of those 2.
With this patch, we will send a Hello Retry Request, allowing to
use X25519MLKEM768.
This is rare, though, most TLS client implementations that have
X25519MLKEM768 will bet on it to be used (the 2 they bet on is
often X25519MLKEM768 & X25519). That's many browsers like Chrome,
OpenSSL, Go, etc.
GnuTLS usually will do this as well, but under some configurations
it may bet on 2 classic crypto to be used. For that specific (type
of) situation, this patch will help to use X25519MLKEM768.
This can be tested with OpenSSL to simulate such an implementation:
openssl s_client -connect 127.0.0.1:6697 -groups X25519MLKEM768:*X25519
Before this patch, it would result in X25519 (because that is the
speculated group, with the asterisk). After this patch it will
cause X25519MLKEM768 to be used.
The tuple syntax is in 3.5.0+ and our UNREALIRCD_DEFAULT_TLS_GROUPS_PRIMARY
with X25519MLKEM768 also requires 3.5.0+ so this is an easy change.
Oh and, this commit comment is rather long for a 1 byte change :D
For the extbans that we ship, no problem, as this isn't used in
any of our extbans, but for third party it may matter, or for us
in the future.
Just something we came across while looking into the issue from
previous commit.
Bram Matthys