There is no hard cap on batch reference length, so we had to make one up.
It is now a clear #define MAXBATCHREFLEN 48, which should be plenty.
No sane client is going to use like a 64 byte batch reference :D
So we did use 48, but we also accidentally used BATCHLEN at another
place. BATCHLEN is 22 and refers to how many bytes we generate, so
that is not appropritate.
Thanks to Valware for spotting this.
This way you can limit the number of pastes going on in a channel, as
this is from everyone in that channel (like 'm') not individual (like 't').
If it is exceeded then we will simply reject the BATCH, similar to
how action d(rop) works for some other subtypes. You won't see the paste
on the channel, only the sending user receives an error (MULTILINE_PASTE_LIMIT).
Small note: a multiline BATCH of just 2 lines is not considered a paste.
We consider a multiline of 3+ lines as a paste. I think that is reasonable,
since a two-line-multiline is not that much of a paste ;).
In the default anti-flood profile (+F normal) we also set 2p per 15s,
so this means channels are by default limited to 2 pastes per 15s max.
Of course, you can override this with +f [4p]:15 or whatever you like.
In terms of +F profiles, the defaults are (maximum x pastes per 15 seconds):
very-strict: 1p
strict: 1p
normal: 2p
relaxed: 2p
very-relaxed: 3p
and 7 for unknown-users (with max-bytes 5250 and 1500 respectively). This
allows pasting a short snippet of code, config file, text from a site, etc.
With multiline you have the guarantee that:
1) You will see the entire text with no delay between lines
2) You won't see another persons chat half-way through such a paste
3) For multiline supporting clients it is now clear that all the text
belongs to each other, which can make selecting/copying it easier.
This basically means short snippets/pastes like that can be completely on
IRC again. No need for a pastebin for it. Though, you may still need such
a service if you are pasting more lines.
Regarding the implementation in UnrealIRCd:
* Clients without multiline get individual fallback lines (concat lines
merged, blank lines skipped, as per spec). And we know that clients like
weechat - which does support multiline - also shows all lines and not
only a few plus snippet style "[.."]. That is another reason for only
allowing 15 lines by default and not something much more. Otherwise all
those clients would get a big wall of text, which just sucks.
* Spamfilter (also) runs on the full text of all lines together, so
splitting a phrase across lines does not evade spamfilter.
* Fakelag: a client can send the BATCH start+PRIVMSG (or NOTICE)+BATCH end
at full speed. We impose no fake lag there. Also, the multiline default
max-lines and max-bytes are lower than the example class::recvq of 8000,
so should be perfectly safe. If the entire BATCH is accepted then we
will impose fake-lag afterwards, with a cap of 15 seconds maximum.
If the BATCH is rejected, we impose half the fakelag plus 2sec.
* If the time between BATCH start and BATCH end is more than 15 seconds
then the BATCH is rejected (set::multiline::batch-timeout).
* The BATCH is atomic (either you see it all, or you see none of it):
* When the client sends it to server, it is buffered first.
* Only after the batch close the server indicates if it is accepted
or rejected. This has various reasons, two of them are: 1) The client
is going to send everything in one go anyway and not wait for a
response between each PRIVMSG, and 2) we can't do many checks in the
buffering stage and skip those after, that would cause a TOCTOU
problem (eg. a banned user still being able to speak).
* If any line gets rejected due to spamfilter or other case
(eg +c, +b ~text with block, etc etc), the entire batch is rejected
* Locally we deliver all or nothing (as said)
* S2S we buffer the batch as well, so if a server splits after having
received 10 lines out of 15, then clients will not see anything.
* We send max-lines and max-bytes, this is the hard upper limit.
* A multiline can still be limited more tight if:
* +f with 't' or 'm' restricts to fewer lines,
eg +f [5t]:15, which means max 5 lines per 15 seconds,
means the max accepted multiline is 5 for that channel.
* +F works the same, except that default +F normal does not
have a 't' at the moment and 'm' is very high (50) so
practically not limited by default.
* There will be a future +f flood subtype for some more control
TODO: we will send CAP NEW on unknown-users <-> known-users to
indicate the new max-lines value if you transition security groups
TODO: chat history does not yet include multiline batches.
because it has no internet access, like when fetching the repository
(modules.list file) of 3rd party modules.
Previously I had..
url_start_async(request);
synchronous_http_request_in_progress = 1;
.. which worked fine for the "cannot connect case", like port blocked
or timeout connecting. But if DNS fails then the step of setting
synchronous_http_request_in_progress = -1 (so failed) already happens
during the url_start_async(request); call, and then the line after it
sets 'synchronous_http_request_in_progress = 1;' so we miss that it
failed and wait in the I/O loop forever.
Simply swapping the two lines of code fixes this.
The other change is that when running the ModuleManager in "make" we should
ignore the exit code. I probably broke that while refactoring and adding
non-zero exit codes in de modulemanager past few months for this release.
simply by re-using the context.
The slowdown happened due to commit a541b8f4ad
in June 2021 when converting to OpenSSL 3+ code. Now it is basically
back to the pre-openssl-v3 speeds.
Something like:
#ifdef TLS1_3_VERSION
w->minimum_tls_version = TLS1_3_VERSION;
#endif
url_start_async(w);
Require TLSv1.3 for central-blocklist and spamreport calls, unless your
OpenSSL does not support it, which should be rare.
At some point in the future I will make this endpoint TLSv1.3+ only.
(there will be an error if you use `cloak_md5`, but everything
will work fine if you use `cloak_sha256`).
We phased out MD5 usage years ago, so it is only contained to
the old cloaking module. In fact that was the only reason we
started to provide the SHA256 cloaking module, simply so it
isn't using old MD5.
Of course, for module coders this means they should not call
DoMD5() or md5hash(), but that would be rare. Currently zero
modules in unrealircd contrib do this and it makes no sense
to start using it nowadays anyway.
Example output:
*** SPAMINFO ***
This will show the original text and the deconfused text which can be used in a spamfilter block with input-conversion deconfused;
Original spam text: ẔŽŽẐ𝞕ȤℤΖℨℨ𝒁𝓩ẒŹƵᏃŻẒŽℨŹ𝒵𝛧Ż𝝛𝛧ℨℤ𝜡Ƶ𝞕𝘡ŹẐ𝑍ẔẐẐΖ𝜡Ẕ𝜡Ẕ𝞕ꓜ𝚭ᏃẐẔ𝙕
Deconfused spam text: ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ
AntiMixedUTF8 points: 64
Number of Unicode characters in total: 50
Number of different Unicode blocks used: 8
Unicode Block breakdown (name: bytes [capped at 255]):
- Latin Extended-A: 8
- Latin Extended-B: 3
- Greek and Coptic: 2
- Cherokee: 2
- Latin Extended Additional: 12
- Letterlike Symbols: 6
- Lisu: 1
- Mathematical Alphanumeric Symbols: 16
switches like antimixedutf8 did, and counts the number of characters
used per unicode block. Potentially more can be added later, this is
flexible and modules can add stuff (..well not yet.. the struct is
missing some members..).
Use it from antimixedutf8 so that it now uses the new code, which is
similar to what I made and then reverted in July 2023:
https://github.com/unrealircd/unrealircd/commit/3e2f668f10fccedfd035526d7b20d7ca6819a8ae
..except that it now calculated in src/modules/utf8functions.c.
But yeah, this needs more testing and possibly (default) score
adjustments to deal with false positives !! And a warning in release notes :D
Put the text analysis in ClientContext member textanalysis,
so typically accessed through clictx->textanalysis.
Note that this struct can (and often is) NULL, for example if it is
a remote client, if it is not a PRIVMSG/NOTICE (will improve later)
or if the utf8functions module is not loaded (to keep things optional).
BREAKING CHANGE is that ClientContext is now passed in the
HOOKTYPE_CAN_SEND_TO_CHANNEL and HOOKTYPE_CAN_SEND_TO_USER hooks.
So HOOKTYPE_CAN_SEND_TO_USER prototype changed from:
int hooktype_can_send_to_user(Client *client, Client *target, const char **text, const char **errmsg, SendType sendtype);
To:
int hooktype_can_send_to_user(Client *client, Client *target, const char **text, const char **errmsg, SendType sendtype, ClientContext *clictx);
And HOOKTYPE_CAN_SEND_TO_CHANNEL prototype changes from:
int hooktype_can_send_to_channel(Client *client, Channel *channel, Membership *member, const char **text, const char **errmsg, SendType sendtype);
To:
int hooktype_can_send_to_channel(Client *client, Channel *channel, Membership *member, const char **text, const char **errmsg, SendType sendtype, ClientContext *clictx);
A side-affect of this change for antimixedutf8 purposes is that,
while the analysis is only done once per line, the 'actions' are
performed for each target, so the action will run 4 times for
"PRIVMSG a,b,c,d :text" although that may not be important in
practice. Just mentioning.
I started work on this back then but didn't finalize it. Now I
have to figure out what was left to be done :D. Other than the
obvious case of seeing some debugging code that prints out for
every converted character. Not yet visible / usable by end-users!
Eg: vhost "$operlogin@$operclass.example.net";
Also add potentially_valid_vhost() function which can be used in
config code to ignore invalid $vars. Then at runtime you use the
real valid_vhost() function after variable expansion by
unreal_expand_string().
This so if there is ever an issue, we can hot-patch it. This affects
exit_client(), exit_client_fmt(), exit_client_ex(), banned_client(),
and various (internal) help functions.
This also means you cannot call these functions during TEST/INIT (eg
during REHASH) since the 'quit' module which provides these modules
may not be loaded yet. I don't think that's a situation/problem but
this needs some more testing.
This so you can get the same spamreport data to your own custom system.
It works similar to set::central-blocklist::url but then on a
spamreport { } basis which is better, since then you can still
submit to UnrealIRCd central spamreport too.
So you can have two blocks:
spamreport unrealircd { type central-spamreport; }
spamreport custom { type central-spamreport; url 'https://www.example.org/xyz'; }
And then a /SPAMREPORT or 'report;' action will report it to BOTH.
Requested by Chris
Note that this is still a dumb interface and not a real proper
authentication framework.
This adds HOOKTYPE_SASL_AUTHENTICATE and HOOKTYPE_SASL_MECHS and
also provides 3 functions: sasl_succeeded(), sasl_failed() and
a helper function decode_authenticate_plain() for AUTHENTICATE PLAIN.
set::central-blocklist::spamreport and ::spamreport-enabled are now GONE.
We now require a normal spamreport block, just like for other spamreport
functionality. So, if you want to enable this feature, use:
spamreport unrealircd { type central-spamreport; }
See https://www.unrealircd.org/docs/Central_spamreport for all info.
You can use CBL with central spamreport or central spamreport without CBL.
All explained at that URL.
... in case you want to do fine-tuning.
Defaults to DOWNLOAD_CONNECT_TIMEOUT (15 seconds) and
DOWNLOAD_TRANSFER_TIMEOUT (20 seconds).
For example, the module manager uses a shorter timeout of 7 and 20.
(that was already the case, but now it uses the generic api so
it needed an option to set it to those values)
No longer url_start_async(a,b,c,d,e,f,g,...) but usings structs so
simply url_start_async(tehstruct);
makes it easy to add fields later without forcing all modules to
change the prototype.
Work in progress....
This will hide the *LINE reason to other users if the *LINE contains the
IP of the user. This to protect the privacy of the user for cases such
as a KLINE due to a blacklist with a DroneBL URL.
Other possible settings are `no` (never hide, the previous default) and
`yes` to always hide the *LINE reason. In all cases the user affected by
the server ban can still see the reason and IRCOps too.
https://bugs.unrealircd.org/view.php?id=6362
This fixes the issue where +e/+I ~operclass:name gets cut off if the
name contains any digits.
Reported by BlackBishop in https://bugs.unrealircd.org/view.php?id=6353
Also, we previously allowed any characters in the operclass, which is not
a great idea.
action { set REPUTATION--; } and similar.
Also enhancement to reputation S2S traffic, to support decreasing:
*
+ * Since UnrealIRCd 6.0.2+ there is now also asterisk-score-asterisk:
+ * :server REPUTATION 1.2.3.4 *2*
+ * The leading asterisk means no reply will be sent back, ever, and the
+ * trailing asterisk will mean it is a "FORCED SET", which means that
+ * servers should set the reputation to that value, even if it is lower.
+ * This way reputation can be reduced and the reducation can be synced
+ * across servers, which was not possible before 6.0.2.
+ *
So if you are actually decreasing reputation, you need all servers on
6.0.2 or higher for it to work properly, otherwise the other servers
don't decrease it, and next connect the highest wins again, etc.
Often you have default values for the config, and then a subsequent config
parsing run would overwrite the return value (= memory leak), merging/appending
would make no sense either, so it would force a free in all code before
calling us, well... let's just deal with it ourselves instead then ;)
The spamfilter::action stop ill prevent processing other spamfilters.
This would normally be a bit unusual, and potentially dangerous when you
do exclude things this way, but can be useful in some circumstances.
Stopping only affects the same type of spamfilters (general or central
spamfilters), so they don't interfere.
The tkldb write DB bug had to do with that it was processing
central spamfilters, which should be skipped just like config
based spamfilters were already skipped.
to limit actions to limit-ban-action as the highest, and limit
ban times to limit-ban-time the highest, see
https://www.unrealircd.org/docs/Central_Spamfilter
This also changes highest_spamfilter_action() to highest_ban_action().
This uses the RPC2 API. Tested with staging.
Note that there are likely some bugs here or there, like memory leaks,
but the functionality is there.
Also still need to implement various stuff, including spamreport::rate-limit
a function called start_dns_and_ident_lookup(). This can then
be easily called from other places as well, like the code k4be
did in src/modules/websocket.c to handle proxies.
Side-effect is that ident lookups would now be done, if we are
configured to do so, for forwarded webirc stuff (not that I
think many people use that feature at the moment...).
It is now possible to override some set settings per-security group by
having a set block with a name, like `set unknown-users { }`
* You could use this to set more limitations for unknown-users:
```
set unknown-users {
max-channels-per-user 5;
static-quit "Quit";
static-part yes;
}
```
* Or to set higher values (higher than the normal set block)
for trusted users:
```
security-group trusted-bots {
account { BotOne; BotTwo; }
}
set trusted-bots {
max-channels-per-user 25;
}
```
* Currently the following settings can be used in a set xxx { } block:
set::auto-join, set::modes-on-connect, set::restrict-usermodes,
set::max-channels-per-user, set::static-quit, set::static-part.
Bram Matthys