"Did not find mosh server startup message" when connecting to a busybox ash shell (embedded device)

[uvok]

When I try to run mosh (client), trying to connect to my vaccum robot (I compiled the server) I get the message

$ mosh --server=/tmp/mosh-server [email protected]
Connection to roborock.lan closed.
/usr/bin/mosh: Did not find mosh server startup message. (Have you installed mosh on your server?)

also mentioned in #1165, #1169.

Running the debug command, I get

$ ssh -S none -o ProxyCommand='mosh --fake-proxy -- %h %p' -n -tt [email protected] '/tmp/mosh-server new'
MOSH IP 192.168.24.2
Connection to roborock.lan closed.

If I connect directly via ssh, the output is

$ ssh [email protected]


BusyBox v1.24.1 (2019-11-15 15:22:50 CST) built-in shell (ash)

[root@rockrobo ~]# 

Does the "Busybox..." output confuse the mosh client? Unfortunately, there's no way to turn that off. Or is something else wrong?

When I run mosh-server directly on the device, I get

[root@rockrobo tmp]# /tmp/mosh-server new


MOSH CONNECT 60001 fETMGM1odQki7BU1n1ViSA

mosh-server (mosh 1.4.0) [build mosh 1.4.0]
Copyright 2012 Keith Winstein <[email protected]>
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

[mosh-server detached, pid = 14214]
[root@rockrobo tmp]# 

• mosh-client: 1.4.0, installed from debian repos
• mosh-server: 1.4.0 built from source

edit:
When running the command directly instead of starting an interactive shell:

$ ssh -S none  [email protected] '/tmp/mosh-server new'
MOSH CONNECT 60001 JLKgzz6Q50AeuBa2SUKKDA

mosh-server (mosh 1.4.0) [build mosh 1.4.0]
Copyright 2012 Keith Winstein <[email protected]>
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

[mosh-server detached, pid = 15197]

edit:

Running client and server separately as per https://mosh.org/#faq "How do I run the mosh client and server separately?" works perfectly fine btw.

[Riteo]

Heads up, I have the same issue and I fixed it my passing --no-ssh-pty to the wrapper script :D

While messing around with the debug command, it looks like the -n flag is confusing ash. I suppose that it doesn't like having /dev/null passed to stdin, at least when it is interacting with a pseudo-terminal.

I wonder why we create a pty if it's all just for messaging anyways, mh.

Update: looks like SSH behaves wildly differently without the PTY, including not printing the motd. That must be it.

Update 2: Yup. It indeed works identically AFAICT by not insisting on setting up the pty, which tells login(1) that it's not supposed to be interactive and so to not print the MOTD. I got initially confused by the fact that there was no alternate-screen support but it looks like it's just a current limitation of mosh.

So, in other words, --no-ssh-pty seems to make no difference whatsoever and, from my uninformed view, looks like it should be the default anyways. I really do wonder what's the advantage of allocating a PTY by default.

[Riteo]

Update: I have a feeling that the reason that --no-ssh-pty is not the default is mostly historic. I just found out about a commit from 2014 hashed as 70d48fb :

commit 70d48fbcc038d4f746bab029e8449812263f2cf9
Author: John Hood <[email protected]>
Date:   Sat May 17 01:34:14 2014 -0400

    Add --no-ssh-pty option.
    
    This eliminates issues with typeahead being echoed by the remote pty
    and corrupting remote output from mosh-server to the mosh script, but
    cannot be made default because older mosh-servers require a pty.

It looks like in the past a pty was needed for the server to work properly, up until 74e1a30, which comes from 2013, so the commit message above makes sense, as one year is quite little for compatibility stuff.

If my research is right, I suppose that it would finally be time to switch the default around, as I'm quite sure that nowadays the majority of servers run 1.2.4 onwards, which got released 11 years ago :P