I still investigated, what is the problem.
Server Settings (default):

Quote:# ENet peer liveness. The ENet default of 5000 ms is shorter than a level
# streaming hitch on a slower client, which drops a healthy session during a
# scripted teleport and forces a full reconnect. Lower these only if you would
# rather reclaim a crashed client's slot faster than tolerate loading stalls.
peerTimeoutMinimumMs=20000
peerTimeoutMaximumMs=40000
peerTimeoutLimit=32

I don't think these settings are working correctly.
The server drops my character and reconnects it after 3-4 seconds after the first few teleports and it doesn't wait 20s.
(or maybe I don't understand the description)

But after a couple teleports, it looks fine even if I restart the game (not the server).
So, it is possible, the server doesn't load this value well at first.