Suggested by westid in https://bugs.unrealircd.org/view.php?id=6477
* New [crule function](https://www.unrealircd.org/docs/Crule) that return
the number of times a flood was blocked for that user. For example,
`server_flood_count('away')` returns the number of time away-flood
was exceeded. Aslo available: `nick`, `join`, `invite`, `knock`,
`vhost` and `conversations`. Plus, there is `all` for a total of all.
* This can be used in a security-group::rule or spamfilter::rule.
Eg: `spamfilter { rule "server_flood_count('nick')>4"; action gline; }`
This also - internally - adds a mechanism to run spamfilter rule-only-
filters after the command handler, whenever a tag value or other thing
changed. That's part of this commit.
Basically if a $variable is empty, and there is a space before it in the
template string then we delete that space.
May seem (or is) a bit over the top but this way the template stays clean,
and it may be used/useful in other places as well.
This is a behavior change, but I think we can live with it. One can opt-
out via BUILDVARSTRING_KEEP_SPACE_FOR_EMPTY_VAR.
TKL_EXPIRE and SPAMFILTER_MATCH messages.
This uses the newly added functions log_data_optional_string() and
log_data_optional_name_value(). The first shows the optional string
like "abc" and the second expands to "[name: value]". What's also new
is that both of these will swallow a preceding space if there is no value.
This so you can just use "Something. $optional_string" and it will
expand to "Something." if $optional_string is empty. This makes things
less hacky and more human readable :)
of the last hit, eg in `STATS gline` for GLINEs. These counts happen on
each individual server and are not network-wide. This allows IRCOps to see
which entries never get any hits and can potentially be removed.
* Important exception: config-based spamfilters/bans lose their counters
on `REHASH` and restart atm.
* For non-config TKLs, the hit count and last hit timestamp are preserved
across reboots (via tkldb).
* Again, see *Developers and protocol* for the exact STATS field.
The spamfilter hits already existed but all the rest is new.
Suggested by BlackBishop in https://bugs.unrealircd.org/view.php?id=6304
(in particular, time of the last hit)
By default - assuming you don't set set::reject-message things by yourself -
the *LINE id is appended at the end of the rejection that is shown to the
user, like: [ID: G7K2MP9WQX3].
Also new is spamfilter to *LINE mapping, so you can see which *LINE was
set by which SPAMFILTER. For this STATS gline and friends were enhanced.
In fact, multiple fields were added there, including some that are 0
(zero) placeholders at the moment. These will be set in a future commit.
Some things were combined here so we only have to break STATS and tkldb
database format once (unless i made a mistake, then the follow up commit
will correct that i guess :D).
This was requested by Hero in https://bugs.unrealircd.org/view.php?id=4397
in 2015. Again by musk in https://bugs.unrealircd.org/view.php?id=4397
in 2022. And on IRC by Chris and others.
As you can see it was not SUPER easy and a lot of thought went into this
(and in terms of S2S traffic it is part of something bigger too)
And the defines are more clear now (if .max_size is not set by caller.
DOWNLOAD_MAX_SIZE_MEMORY_BACKED: 1M
DOWNLOAD_MAX_SIZE_FILE_BACKED: 50M
The file-backed is mostly a defense-in-depth measure, so we don't
store infinite amounts of data in a download. Even though, in practice,
these - at least at the moment in unrealircd itself - all come from
trusted paths like remote includes.
In url_unreal.c we do the counting ourselves. In url_curl.c we use the
option CURLOPT_MAXFILESIZE_LARGE but this does not ensure it in all
cases so we still do our own counting as well in that file as well.
When a client is rejected by maxperip (not new) or connthrottle
ipv6-unknown-users-limit (that one is new), a notice to +s +x will be sent.
maxperip ipv4 example:
*** Client testuser4 with IP 1.2.3.4 rejected: maxperip limit exceeded (4 global, max 3)
maxperip ipv6 with /64 example:
*** Client testuser4 with IP 2001:dbe:0:0:0:0:0:4 rejected: maxperip limit exceeded for 2001:dbe::/64 (4 local, max 3)
connthrottle example where /56 limit is exceeded:
*** Client testuser5 with IP 2001:db8:cafe:abcd:0:0:0:5 rejected:
connthrottle ipv6-unknown-users-limit (cidr-56, max 4) exceeded for
2001:db8:cafe::/56 (5 unknown / 0 excepted / 0 known)
Oh and this commit also fixes a typo in existing CONNTHROTTLE events,
which previously were CONNTHROTLE (a missing T).
The OOB write did not happen on file-backed downloads, such as remote
includes. It only happened for memory-backed requests, which are only
these 4 in standard UnrealIRCd: centralblocklist, central spam report,
other spamreport blocks (eg to dronebl) and the log block with
destination webhook. All those 4 cases are very likely to be trusted
web servers, given the nature of the data you are sending to them.
The fix was to extend the size fields everywhere to 64 bits. It was
applied to both URL backends: url_unreal.c and url_curl.c.
The new API feature is a 'max_size' in OutgoingWebRequest, which
defaults to 1MB. This is only used for memory-backed responses,
so not for real file downloads. This fixes not only the reported
bug but also the case where a rogue webserver was unbounded in
terms of what response it could send back, potentially filling
up gigabytes of server memory.
Reported by Link420.
There is no hard cap on batch reference length, so we had to make one up.
It is now a clear #define MAXBATCHREFLEN 48, which should be plenty.
No sane client is going to use like a 64 byte batch reference :D
So we did use 48, but we also accidentally used BATCHLEN at another
place. BATCHLEN is 22 and refers to how many bytes we generate, so
that is not appropritate.
Thanks to Valware for spotting this.
This wasn't done before, because optimizing stuff can always introduce
nice new issues. But is kinda necessary now since the previous way was
very inefficient. This now builds all the necessary buffers for multiline
clients and for non-multiline clients. And then iterates through both
types of clients, sending what they need. Instead of doing it the other
way around.
I had the dillema to either expose the linecache API and have everything
in multiline.c. Or, i do not expose linecache, and we do everything in
send.c. The downside of the latter is that if there is mistake then we
can't simply reload (or unload) the module to solve it. So, I have chosen
to expose the linecache API (sure, less clean) since that leaves us with
options if we screw up, plus it means everything related to multiline
sending is nicely in multiline.c, which is i guess just as good as an
argument as well ;)
and 7 for unknown-users (with max-bytes 5250 and 1500 respectively). This
allows pasting a short snippet of code, config file, text from a site, etc.
With multiline you have the guarantee that:
1) You will see the entire text with no delay between lines
2) You won't see another persons chat half-way through such a paste
3) For multiline supporting clients it is now clear that all the text
belongs to each other, which can make selecting/copying it easier.
This basically means short snippets/pastes like that can be completely on
IRC again. No need for a pastebin for it. Though, you may still need such
a service if you are pasting more lines.
Regarding the implementation in UnrealIRCd:
* Clients without multiline get individual fallback lines (concat lines
merged, blank lines skipped, as per spec). And we know that clients like
weechat - which does support multiline - also shows all lines and not
only a few plus snippet style "[.."]. That is another reason for only
allowing 15 lines by default and not something much more. Otherwise all
those clients would get a big wall of text, which just sucks.
* Spamfilter (also) runs on the full text of all lines together, so
splitting a phrase across lines does not evade spamfilter.
* Fakelag: a client can send the BATCH start+PRIVMSG (or NOTICE)+BATCH end
at full speed. We impose no fake lag there. Also, the multiline default
max-lines and max-bytes are lower than the example class::recvq of 8000,
so should be perfectly safe. If the entire BATCH is accepted then we
will impose fake-lag afterwards, with a cap of 15 seconds maximum.
If the BATCH is rejected, we impose half the fakelag plus 2sec.
* If the time between BATCH start and BATCH end is more than 15 seconds
then the BATCH is rejected (set::multiline::batch-timeout).
* The BATCH is atomic (either you see it all, or you see none of it):
* When the client sends it to server, it is buffered first.
* Only after the batch close the server indicates if it is accepted
or rejected. This has various reasons, two of them are: 1) The client
is going to send everything in one go anyway and not wait for a
response between each PRIVMSG, and 2) we can't do many checks in the
buffering stage and skip those after, that would cause a TOCTOU
problem (eg. a banned user still being able to speak).
* If any line gets rejected due to spamfilter or other case
(eg +c, +b ~text with block, etc etc), the entire batch is rejected
* Locally we deliver all or nothing (as said)
* S2S we buffer the batch as well, so if a server splits after having
received 10 lines out of 15, then clients will not see anything.
* We send max-lines and max-bytes, this is the hard upper limit.
* A multiline can still be limited more tight if:
* +f with 't' or 'm' restricts to fewer lines,
eg +f [5t]:15, which means max 5 lines per 15 seconds,
means the max accepted multiline is 5 for that channel.
* +F works the same, except that default +F normal does not
have a 't' at the moment and 'm' is very high (50) so
practically not limited by default.
* There will be a future +f flood subtype for some more control
TODO: we will send CAP NEW on unknown-users <-> known-users to
indicate the new max-lines value if you transition security groups
TODO: chat history does not yet include multiline batches.
As usual, this is mostly for configuration templates that you use for
multiple servers, that sort of things, eg.
@if !environment("ADMIN")
@error "Environment variable ADMIN is not set"
@endif
This also adds a change in conf.c so @define, @error and
@warning are skipped in @if blocks that evaluate to false
(that's obviously what everyone wants :D). So that fixes a
previous bug with @define in @if.
to check environment variables.
This also means functions can now return values, so some changes
under the hood. This also moves the <=, >=, <, > ops code.
loadmodule + set config items
This checks the file on-disk, which is slightly different than
@if module-loaded("third/coolmod") which checks if it is loaded.
in NameList, Tag, Watch and HistoryLogLine.
This does mean the allocation routines need a +1 everywhere, but
I think I got all of them. I also don't see them being used directly
in such a way in 3rd party modules (which is logical, as they
should use the API and not allocate such structs directly).
Also, SpamExcept has been removed as it was not used anywhere.
user is in known-users or in unknown-users. Not used anywhere yet.
Every 2 minutes we rescore all users. Or more specifically: every
5 seconds we rescore 1/24th of all users. That's the slow update path.
On certain events that cause a likely/possible transition, we update
the cache immediately. At the moment that is on IP change and account
login/logout. More will be added later.
* Add some missing fields, such as destination, but mostly in the
exclude- area where a bunch were missing (some of those are a bit
far fetched, but hey, they exist, so should be shown if in use).
* Re-order fields to more closely match the struct (still not 100%)
* Extended fields, such as "account" and "country", now show up
directly under the security group, just like the other fields,
such as "reputation_score". This is also how they show up in the
config file, so hide the the fact that internally in the struct it
is stored differently.
* Add a comment in SecurityGroup struct in include/struct.h to make
it clear you have to add/update stuff at 7 places if you are adding
something new.
New RPC methods:
- security_group.list: List all security groups
- security_group.get: Get details of a specific security group
- connthrottle.status: Get full connection throttle status, counters, and config
- connthrottle.set: Enable/disable connection throttling
- connthrottle.reset: Reset connection throttling counts
This also adds json_expand_mask_list(), json_expand_name_list(), and
json_expand_nvplist() to src/json.c for reuse by RPC modules.
synched. Both need to be checked, because:
* The "far" server may be fully synched to "near" (and thus tagged as synced)
but the "near" server may be introducing the "far" server, when
we are connecting to "near"
* The "near" server may be fully synched but the "far" server is connecting
in and may thus not be synched yet
In reality, things are even more complex, since one would have to verify
the whole chain of links. But.. yeah.
Long-story short: this fixes things like "User xyz joined #xxxxx" logging
where this showed up while the server was linking in. It is not supposed to
log that, similar to how we not log all 1000 users as newly connecting when
a 1000-user-server links in. In fact, it didn't already log that for
directly-connected-servers, but for far servers it did previously.
And... that again gave performance issues if you were connecting like a
100k-user far server.. since you suddenly had 100k * numchannels join events
being logged (which surprisingly still only took 6 seconds for 100k entries,
but still, it is wrong to do so and can be avoided).
This also makes them proper list items, again to make certain fast operations
possible. Main thing is that removing an entry does not require us to walk
all of those lists. Not all code has been modified yet to benefit this,
actually only very little, the most performance-impacting ones.
This fixes SQUIT of a server with 100k users in a single channel taking
40 seconds of 100% CPU. It now takes only 1 second.
Reported by craftxbox in https://bugs.unrealircd.org/view.php?id=6484
(Can't make member & membership one entry atm, that would be too much change in U6)
This so we can use fast(er) techniques here and there.
New functions are:
channel_has_invisible_users(client)
set_user_invisible(client, channel, 1|0)
Existing functions:
invisible_user_in_channel(client, channel)
user_can_see_member(user, target, channel)
user_can_see_member_fast()
This is work in progress, although the tests seem to pass atm.
This makes things a lot faster on multi-server networks, especially for
big channels where most of the clients in the channel are remote users.
This should be non-module-API-breaking, as all code uses the
add_user_to_channel() and remove_user_from_channel() functions.
Still need to spread this to other code, more optimizations possible.
This was previously a "long", which could cause issues on 32 bit archs.
We ship with 28 CAPs now, and that's without 3rd party modules, so...
This is similar to the client->flags bumping in 2023
(a3ed1eabd9).
Something like:
#ifdef TLS1_3_VERSION
w->minimum_tls_version = TLS1_3_VERSION;
#endif
url_start_async(w);
Require TLSv1.3 for central-blocklist and spamreport calls, unless your
OpenSSL does not support it, which should be rare.
At some point in the future I will make this endpoint TLSv1.3+ only.
We don't set it in UnrealIRCd at the moment, so this is just to override
the OpenSSL defaults at the moment. It is good to have this exposed, in
case some vulnerability is discovered or you need some flexibility in
tweaking this.
allow {
mask *;
password "secret";
password "letmein";
}
This is always an "OR" type of match, any match means you pass.
I was actually doing this for the dual-cert stuff from previous commit,
where this can come in handy:
link irc1.example.org {
...
password "AHMYBevUxXKU/S3pdBSjXP4zi4VOetYQQVJXoNYiBR0=" { spkifp; };
password "jNw8P4QMg9tqjEJ4/lFikXBNHdIGSeN2B4/T322VjIo=" { spkifp; };
...
}
In the past a dual cert/key setup could have been useful for RSA + ECDSA
but nowadays all clients support ECDSA so that makes little sense.
The reason it is added now is so you can use ECDSA + ML-DSA or some
other [regular crypto] + [post quantum crypto] combination.
Actually, you could even use more than two.
To use this in the config file, simply use the certificate and key
directive multiple times. Just be sure to load the certificates and keys
in the same order. We will print a helpful error if you fail to do so.
Note that for Post Quantum Cryptography the most important step today
was/is to protect against the "Harvest now, decrypt later" scenario
https://en.wikipedia.org/wiki/Harvest_now,_decrypt_later which is a
"passive attack". That's why in UnrealIRCd 6.2.0 we enabled
X25519MLKEM768 if it is available (OpenSSL 3.5.0 and later).
While, this commit, and this talk about dual ECDSA and ML-DSA, is about
when a quantum computer exists and actively does a man in the middle
attack. That's not a realistic scenario in 2025 and according to experts
also not in the next few years. We just make the UnrealIRCd code-
base ready to have this feature for when it is needed / will be used,
and to get this tested properly.
For testing the dual ECDSA and ML-DSA setup I used the following
command to create the 2nd cert/key (self-signed):
openssl req -x509 -nodes -newkey mldsa65 \
-keyout ~/unrealircd/conf/tls/server.key.mdsa65.pem \
-out ~/unrealircd/conf/tls/server.cert.mdsa65.pem \
-days 3650
And then:
listen {
ip *;
port 6697;
options { tls; }
tls-options {
certificate "ssl/server.cert.pem";
key "ssl/server.key.pem";
certificate "ssl/server.cert.mdsa65.pem";
key "ssl/server.key.mdsa65.pem";
}
}
When running openssl s_client -connect 127.0.0.1:6697 it shows ML-DSA is used:
...
Peer signature type: mldsa65
Negotiated TLS1.3 group: X25519MLKEM768
...
And with openssl s_client -connect 127.0.0.1:6697 -sigalgs "RSA+SHA256:RSA+SHA384:ECDSA+SHA256:ECDSA+SHA384"
it shows ECDSA is used:
..
Peer signature type: ecdsa_secp384r1_sha384
Negotiated TLS1.3 group: X25519MLKEM768
..
This is just for testing purposes (self signed cert). As of right
now (Sep 2025), you can not get a trusted certificate with ML-DSA,
as the CA/Browser Forum only allows issueing RSA and ECDSA keys.
Also, all the trusted Certificate Authorities use RSA or ECDSA.
And, again, all this is not ML-DSA specific, it should work for
other dual/multi combinations, and.. who knows they even go for
something hybrid.
A downside of dual certs is that this makes the whole spkifp thing more
complicated because if you use 2 certs/keys you now have 2 possible
fingerprints (spkifp) that could match in e.g. server linking.
While coding this, I also changed the 'STATS P' output to use the txt
numeric instead of notice, and be more verbose in its output for TLS
listeners: printing the certificate(s) and key(s).
This function was added a short while ago, and well it seems to be
able to be possible in a module. Since the 'isupport' module is mandatory
and this is ISUPPORT related, it is the right place.
Can't move isupport_snapshot() because modules might not be loaded yet
or things are currently unloading, i think. Not important anyway.
Also, make things work if there are more changes than would fit
on one isupport line. Although I didn't really test this..
Ended up splitting things in 3 helper functions to avoid some
goto and/or duplicate code and stuff. The alternative was, surprisingly,
even more ugly.
Maybe a bit odd since only <10 things use this category but it makes it
stand out as a separate thing much better. As for a level (not that it
matters) it is between 'info' and 'warn'.
users by server port (eg 6667, 6697, 8000, etc).
This also adds security-group::exclude-server-port for consistency.
And in crules the function server_port() returns the server port number,
so you can use rule 'server_port()>6690' for example.
Note that for remote clients this will only work after previous
commit (b2d0ec1af3) is loaded on all
servers, otherwise all remote clients are seen as having a server_port
of zero (0). Though you probably usually only care about this on local
users anyway.
This will return the number of characters that are in the unicode block
with that name.
spamfilter {
rule "unicode_count('Emoticons')>2";
target { private; channel; private-notice; channel-notice; }
action block;
reason "Too much emotion";
}
In this commit we also make it so we pass the ClientContext (including
clictx->textanalysis) in crule_context.
In CommandAdd() the flag CMD_TEXTANALYSIS now means that the last
parameter of the command will run through the text analysis system.
This flag is set in PRIVMSG NOTICE PART QUIT AWAY SETNAME TOPIC
Make match_spamfilter use the clictx->textanalysis->deconfused rather than
calculating its own. The latter will probably disappear altogether.
Unrelated but also fixed: properly set e->unicode_blocks.
switches like antimixedutf8 did, and counts the number of characters
used per unicode block. Potentially more can be added later, this is
flexible and modules can add stuff (..well not yet.. the struct is
missing some members..).
Use it from antimixedutf8 so that it now uses the new code, which is
similar to what I made and then reverted in July 2023:
https://github.com/unrealircd/unrealircd/commit/3e2f668f10fccedfd035526d7b20d7ca6819a8ae
..except that it now calculated in src/modules/utf8functions.c.
But yeah, this needs more testing and possibly (default) score
adjustments to deal with false positives !! And a warning in release notes :D
Put the text analysis in ClientContext member textanalysis,
so typically accessed through clictx->textanalysis.
Note that this struct can (and often is) NULL, for example if it is
a remote client, if it is not a PRIVMSG/NOTICE (will improve later)
or if the utf8functions module is not loaded (to keep things optional).
BREAKING CHANGE is that ClientContext is now passed in the
HOOKTYPE_CAN_SEND_TO_CHANNEL and HOOKTYPE_CAN_SEND_TO_USER hooks.
So HOOKTYPE_CAN_SEND_TO_USER prototype changed from:
int hooktype_can_send_to_user(Client *client, Client *target, const char **text, const char **errmsg, SendType sendtype);
To:
int hooktype_can_send_to_user(Client *client, Client *target, const char **text, const char **errmsg, SendType sendtype, ClientContext *clictx);
And HOOKTYPE_CAN_SEND_TO_CHANNEL prototype changes from:
int hooktype_can_send_to_channel(Client *client, Channel *channel, Membership *member, const char **text, const char **errmsg, SendType sendtype);
To:
int hooktype_can_send_to_channel(Client *client, Channel *channel, Membership *member, const char **text, const char **errmsg, SendType sendtype, ClientContext *clictx);
A side-affect of this change for antimixedutf8 purposes is that,
while the analysis is only done once per line, the 'actions' are
performed for each target, so the action will run 4 times for
"PRIVMSG a,b,c,d :text" although that may not be important in
practice. Just mentioning.
It now passes 'clictx' which at the moment only has clictx->cmd which
points to the command handler. So only useful in very few cases where
you have like a generic command handler and thus have no idea for which
command you are being called. In the future, with this new ClientContext
struct, we can simply add new fields to the struct without breaking
things in the core and in (third party) modules.
If you use the magic functions in your modules CMD_FUNC(cmd_mycmd),
OVERRIDE_FUNC(myoverride), CALL_NEXT_COMMAND_OVERRIDE() and such then
you shouldn't have any compile errors as these will use the correct
prototypes and variable names automatically. In a few cases you can't
use these, in which case you will need to update your modules.
Bram Matthys