Dedicated Server randomly stops allowing players to join

[Stachelbeere1248]

Steam or GOG

Steam

Single Player/Multiplayer

Multi

Operating System

Linux Other

Terraria Version

1.4.4.9

Controls Used

Keyboard/Mouse

A similar thread (not inside bug-reports) can be found here.
Basically I'm hosting a dedicated server on my vps. It works fine and everything, except it often suddenly prevents any players from joining (even if there are slots open).
The players will get a screen saying "Connecting to IP". The weird part is, this can happen even while other players are playing on the server.
As said in the thread above, restarting the server executable solves this issue. However, this can be quite annoying for both me as host and the players who dont want their game interrupted.
I downloaded the server files from https://terraria.org/api/download/pc-dedicated-server/terraria-server-1449.zip.
I'm aware that other factors related to the vps might be part of the cause, but even if this was the case, it would be nice if this bug was fixed and the server could reopen to let player join again automatically, without having to restart.

 

[DontBanMeIWannaReportABug]

Hey, can confirm the same issue on my Linux VPS too. The official Linux binary has the problem as well as the .exe binary ran through mono supplied through my distro's package manager.

Here are some things I noticed:
- Does not happen with a single player. I can reconnect without an issue. Some time after my friend joins me and I try to reconnect, I fail.
- The CPU goes insane. Normally, when the server is idle, there's around 5% usage. After one player joins, it's around 45%. When two players join, it's around 80%. When the bug occurs after multiple players are online, it goes insane around 150% usage, completely hogging one of the CPU cores.
- The server is rejecting new connections. I've put the server behind a proxy to see what's happening. When the bug occurs, established connections are left alone. Online players can keep playing. New connections get "Connection refused" error.
- Ignoring the rejected connections and high CPU usage, the server seems fine. Day night cycle works, online players can keep playing, console commands work.
- Temporary fix happens to be just typing exit into the console and restarting. Server works again until more players join again and somehow trigger the bug.
- It does not happen always? I've managed to play with a friend for like 6 hours on a fairly normal usage and could reconnect several times. It's not exactly like 2+ players online = bug.

Last thing I would like to mention is that this is not the only issue causing the insane CPU usage. The console is bugged too. When I start the server manually, usage's fine. When I let my service manager start it, the CPU goes insane from the start. Does it perhaps need an actual terminal screen? A little workaround to this bug is to make the service manager start the server inside a detached GNU screen session which emulates a terminal with screen -D -m -S terrariaserver mono TerrariaServer.exe -config /path/to/config. Have a nice day.

EDIT: Please don't let this issue die out, going to the console to restart the server few times a day is not a pleasant experience. Is there perhaps anything we can do in order to help you find out what's going wrong? If there are some C# debugging tools to see what thread/function/whatever is going crazy, please let us know, I will be more than happy to help.

 

[Stachelbeere1248]

It would be helpful if these forums had some kind of system to show whether bug reports were submitted properly..... from what it looks like to the people making reports, staff just randomly check the latest bug reports and if one got flooded away... bad luck.

 

[DontBanMeIWannaReportABug]

Hi, yeah I completely agree. I also quite understand the developers' perspectives - you can either try to fix a bug you have no idea how to and only a handful of people reported it, or you can just respond to that other post that NPC movement is intentional and call it a day . We can however try to keep this bug report active on the top and hope for the best!

Apart from just ranting a waiting, today I tried some basic debugging even though I had exactly zero knowledge about C#/.NET or debugging in general. We've tried to make the bug happen again with a friend and here is how it went:

I started the server manually with /usr/bin/mono --debug --server --gc=sgen -O=all /path/to/TerrariaServer.exe -config /path/to/config (from what I've seen, the --debug flag was rather useless. I've also tried --trace, but that resulted in a heavy output spam and crashed afterwards). After the world loaded, I (player 1) connected. The server was running fine up to a point when my friend (player 2) tried to join. Sometimes, the server was doing fine even after the second player joined, however in most of the cases, it went rogue instead. When we confirmed multiple times that the bug occurs on a second connection attempt and starts to reject new incoming connections, I tried to dig, fight, explore and build further .

Looking at the CPU usage, a thread that hogged a whole CPU core had PID 8646 assigned to it, so I fired up gdb (GNU Debugger) and listed all threads. There was one with some "LWP" value set to the PID number and it was named "TCP Listen Thread", so I suppose it's rather the network stack than the game logic what's causing the issue.

Spoiler: Full thread list

Id Target Id Frame
* 1 Thread 0x7d371bd860c0 (LWP 8573) "Main Thread" 0x00007d371be1468e in __futex_abstimed_wait_common64 (private=0, futex_word=0x5a53f82c8768, expected=0, op=137, abstime=0x7ffdeb639fa0, cancel=true) at futex-internal.c:57
2 Thread 0x7d36994f46c0 (LWP 8675) "Thread Pool Wor" 0x00007d371be1468e in __futex_abstimed_wait_common64 (private=<optimized out>, futex_word=0x5a53f82b3f48, expected=0, op=393, abstime=0x7d36994f3d10, cancel=true) at futex-internal.c:57
3 Thread 0x7d36996f56c0 (LWP 8674) "Thread Pool Wor" 0x00007d371be1468e in __futex_abstimed_wait_common64 (private=<optimized out>, futex_word=0x5a53f82b3f48, expected=0, op=393, abstime=0x7d36996f4d10, cancel=true) at futex-internal.c:57
4 Thread 0x7d36998f66c0 (LWP 8673) "Thread Pool Wor" 0x00007d371be1468e in __futex_abstimed_wait_common64 (private=<optimized out>, futex_word=0x5a53f82b3f48, expected=0, op=393, abstime=0x7d36998f5d10, cancel=true) at futex-internal.c:57
5 Thread 0x7d3699af76c0 (LWP 8672) "Thread Pool Wor" 0x00007d371be1468e in __futex_abstimed_wait_common64 (private=<optimized out>, futex_word=0x5a53f82b3f48, expected=0, op=393, abstime=0x7d3699af6d10, cancel=true) at futex-internal.c:57
6 Thread 0x7d3699dff6c0 (LWP 8668) "Thread Pool Wor" 0x00007d371be1468e in __futex_abstimed_wait_common64 (private=<optimized out>, futex_word=0x5a53f82b3f48, expected=0, op=393, abstime=0x7d3699dfed10, cancel=true) at futex-internal.c:57
7 Thread 0x7d369a1fd6c0 (LWP 8667) "Thread Pool Wor" 0x00007d371be1468e in __futex_abstimed_wait_common64 (private=<optimized out>, futex_word=0x5a53f82b3f48, expected=0, op=393, abstime=0x7d369a1fcd10, cancel=true) at futex-internal.c:57
8 Thread 0x7d369a3fe6c0 (LWP 8652) "Thread Pool Wor" 0x00007d371be1468e in __futex_abstimed_wait_common64 (private=<optimized out>, futex_word=0x5a53f82b3f48, expected=0, op=393, abstime=0x7d369a3fdd10, cancel=true) at futex-internal.c:57
9 Thread 0x7d369a5ff6c0 (LWP 8651) "Thread Pool Wor" 0x00007d371be1468e in __futex_abstimed_wait_common64 (private=<optimized out>, futex_word=0x5a53f82b3f48, expected=0, op=393, abstime=0x7d369a5fed10, cancel=true) at futex-internal.c:57
10 Thread 0x7d36c1d0b6c0 (LWP 8650) "Thread Pool Wor" 0x00007d371be1468e in __futex_abstimed_wait_common64 (private=<optimized out>, futex_word=0x5a53f82b3f48, expected=0, op=393, abstime=0x7d36c1d0ad10, cancel=true) at futex-internal.c:57
11 Thread 0x7d369b3ef6c0 (LWP 8646) "TCP Listen Thre" __memset_avx2_unaligned_erms_rtm () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:326
12 Thread 0x7d369b6f76c0 (LWP 8602) "Thread Pool Wor" 0x00007d371be1468e in __futex_abstimed_wait_common64 (private=<optimized out>, futex_word=0x5a53f82b3f48, expected=0, op=393, abstime=0x7d369b6f6d10, cancel=true) at futex-internal.c:57
13 Thread 0x7d36c15076c0 (LWP 8601) "Thread Pool Wor" 0x00007d371be1468e in __futex_abstimed_wait_common64 (private=<optimized out>, futex_word=0x5a53f82b3f48, expected=0, op=393, abstime=0x7d36c1506d10, cancel=true) at futex-internal.c:57
14 Thread 0x7d369bdfe6c0 (LWP 8600) "Thread Pool Wor" 0x00007d371be1468e in __futex_abstimed_wait_common64 (private=<optimized out>, futex_word=0x5a53f82b3f48, expected=0, op=393, abstime=0x7d369bdfdd10, cancel=true) at futex-internal.c:57
15 Thread 0x7d369bfff6c0 (LWP 8599) "Thread Pool Wor" 0x00007d371be1468e in __futex_abstimed_wait_common64 (private=<optimized out>, futex_word=0x5a53f82b3f48, expected=0, op=393, abstime=0x7d369bffed10, cancel=true) at futex-internal.c:57
16 Thread 0x7d36c03fd6c0 (LWP 8598) "Thread Pool Wor" 0x00007d371be1468e in __futex_abstimed_wait_common64 (private=<optimized out>, futex_word=0x5a53f82b3f48, expected=0, op=393, abstime=0x7d36c03fcd10, cancel=true) at futex-internal.c:57
17 Thread 0x7d36c05fe6c0 (LWP 8597) "Thread Pool Wor" 0x00007d371be1468e in __futex_abstimed_wait_common64 (private=<optimized out>, futex_word=0x5a53f82b3f48, expected=0, op=393, abstime=0x7d36c05fdd10, cancel=true) at futex-internal.c:57
18 Thread 0x7d36c07ff6c0 (LWP 8596) "Thread Pool Wor" 0x00007d371be1468e in __futex_abstimed_wait_common64 (private=<optimized out>, futex_word=0x5a53f82b3f48, expected=0, op=393, abstime=0x7d36c07fed10, cancel=true) at futex-internal.c:57
19 Thread 0x7d36c0eda6c0 (LWP 8595) "Thread Pool Wor" 0x00007d371be1468e in __futex_abstimed_wait_common64 (private=<optimized out>, futex_word=0x5a53f82b3f48, expected=0, op=393, abstime=0x7d36c0ed9d10, cancel=true) at futex-internal.c:57
20 Thread 0x7d36c17086c0 (LWP 8593) "Thread Pool I/O" 0x00007d371be8b67f in __GI___poll (fds=0x7d36b800cdb0, nfds=3, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
21 Thread 0x7d36c19096c0 (LWP 8590) "Server Loop Thr" 0x00007d371be1468e in __futex_abstimed_wait_common64 (private=0, futex_word=0x5a53f82c8768, expected=0, op=137, abstime=0x7d36c1908730, cancel=true) at futex-internal.c:57
22 Thread 0x7d36c1b0a6c0 (LWP 8589) "Server Loop Thr" 0x00007d371be1468e in __futex_abstimed_wait_common64 (private=0, futex_word=0x5a53f82c8768, expected=0, op=137, abstime=0x7d36c1b097e0, cancel=true) at futex-internal.c:57
23 Thread 0x7d36c274b6c0 (LWP 8587) "Server Input Th" 0x00007d371be8bc9c in __GI___libc_read (fd=0, buf=0x7d37157909b0, nbytes=1024) at ../sysdeps/unix/sysv/linux/read.c:26
24 Thread 0x7d36f53fe6c0 (LWP 8578) "Thread Pool Wor" 0x00007d371be1468e in __futex_abstimed_wait_common64 (private=<optimized out>, futex_word=0x5a53f82b3f48, expected=0, op=393, abstime=0x7d36f53fdd10, cancel=true) at futex-internal.c:57
25 Thread 0x7d36f55ff6c0 (LWP 8577) "Thread Pool Wor" 0x00007d371be1468e in __futex_abstimed_wait_common64 (private=<optimized out>, futex_word=0x5a53f82b3f48, expected=0, op=393, abstime=0x7d36f55fed10, cancel=true) at futex-internal.c:57
26 Thread 0x7d36fe1976c0 (LWP 8576) "Main Thread" 0x00007d371be1468e in __futex_abstimed_wait_common64 (private=0, futex_word=0x5a53f82c8768, expected=0, op=137, abstime=0x7d36fe196c70, cancel=true) at futex-internal.c:57
27 Thread 0x7d3717bff6c0 (LWP 8575) "Finalizer" 0x00007d371be1468e in __futex_abstimed_wait_common64 (private=<optimized out>, futex_word=0x5a53f82b5400, expected=0, op=393, abstime=0x0, cancel=true) at futex-internal.c:57
28 Thread 0x7d371afff6c0 (LWP 8574) "SGen worker" 0x00007d371be1468e in __futex_abstimed_wait_common64 (private=0, futex_word=0x5a53f82b5208, expected=0, op=393, abstime=0x0, cancel=true) at futex-internal.c:57

I tried to get each thread's backtrace, however there's pretty much zero information about the problematic thread - from what I understand, it's because I don't have a debug build of the server to play around with. I'll attach it anyways though.

We really hope this issue won't get lost in this chaotic mess burdened under other bug reports as we would like to go on and play with tModLoader sometime in the future as well. I'm willing to join the Terraria Discord and debug further with the devs if that would be helpful in any way . Have a perfect day.