LCP terminated by peer
I run pacman -Syu
about once a week on my Arch Linux
machines, to stay up-to-date,
avoid security problems and generally stay at the bleeding edge
of software revisions.
Today, I ran pacman -Syu
on my Dell R530 server
and it updated the PPP package.
Trouble ensued.
I happened to notice that the ppp
package got updated today,
from v2.5.0-3 to v2.5.1-1.
My server’s been running ppp
v2.5.0-3 since 2024-01-06,
and there was a problem from that update, too.
I seem to recall meddling with /etc/ppp/peers/centurylink
,
but I didn’t write down what I did that time.
Nevertheless, I was alert for problems after I did the ritual reboot after updating.
It takes about 2 minutes from typing sudo systemctl reboot
to logging in
to a freshly rebooted R530 server.
During those 2 minutes, lots of stuff happens,
including setting up a Point to Point Protocol
link to my ISP, currently CenturyLink.
That’s where the ppp
and rp-pppoe
packages come in.
After finishing rebooting, no network connection to CenturyLink existed.
I looked in the log to see what was them matter,
with journalctl -f -u mypppd.service
Oct 20 17:41:22 monarch systemd[1]: Started PPP link to Centurylink.
Oct 20 17:41:22 monarch pppd[882]: Plugin /usr/lib/rp-pppoe/rp-pppoe.so loaded.
Oct 20 17:41:22 monarch pppd[882]: Plugin /usr/lib/rp-pppoe/rp-pppoe.so loaded.
Oct 20 17:41:22 monarch pppd[882]: RP-PPPoE plugin version 4.0 compiled against pppd 2.5.1
Oct 20 17:41:22 monarch pppd[882]: RP-PPPoE plugin version 4.0 compiled against pppd 2.5.1
Oct 20 17:41:22 monarch pppd[882]: pppd 2.5.1 started by root, uid 0
Oct 20 17:41:37 monarch pppd[882]: PPP session is 4164 (0x1044)
Oct 20 17:41:37 monarch pppd[882]: Connected to 78:fe:3d:a3:27:c2 via interface eno3.201
Oct 20 17:41:37 monarch pppd[882]: Using interface ppp0
Oct 20 17:41:37 monarch pppd[882]: Connect: ppp0 <--> eno3.201
Oct 20 17:41:37 monarch pppd[882]: peer from calling number 78:FE:3D:A3:27:C2 authorized
Oct 20 17:41:37 monarch pppd[882]: LCP terminated by peer
Oct 20 17:41:40 monarch pppd[882]: Connection terminated.
Oct 20 17:41:40 monarch pppd[882]: Sent PADT
Oct 20 17:41:40 monarch pppd[882]: Modem hangup
Oct 20 17:42:10 monarch pppd[882]: PPP session is 4286 (0x10be)
Oct 20 17:42:10 monarch pppd[882]: Connected to 78:fe:3d:a3:27:c2 via interface eno3.201
Oct 20 17:42:10 monarch pppd[882]: Using interface ppp0
Oct 20 17:42:10 monarch pppd[882]: Connect: ppp0 <--> eno3.201
Oct 20 17:42:10 monarch pppd[882]: peer from calling number 78:FE:3D:A3:27:C2 authorized
Oct 20 17:42:10 monarch pppd[882]: LCP terminated by peer
Oct 20 17:42:13 monarch pppd[882]: Connection terminated.
Oct 20 17:42:14 monarch pppd[882]: Sent PADT
Oct 20 17:42:14 monarch pppd[882]: Modem hangup
Oct 20 17:42:44 monarch pppd[882]: PPP session is 4286 (0x10be)
...
pppd
, the actual PPP process, was looping like that.
The relevant error message appeared to be LCP terminated by peer
Googling for that phrase led me nowhere. There’s several references to be found on the compendium of human knowledge, the World Wide Web, but they didn’t seem to apply.
- Enabling “MPPE” in some Windowsy point-n-click dialog box
I put
mppe-stateful
in/etc/ppp/peers/centurylink
No joy. - Don’t know what this one means.
- ???
I did glean that putting debug
in /etc/ppp/peers/centurylink
might show more in the systemd journal, and it does.
Unfortunately, what extra logging it shows isn’t helpful
I checked in /etc/ppp/
, the configuration directory for pppd
.
I noticed that the PPP (or PPPoE) login credentials files were
only kind of present:
-rw------- 1 root root 78 Sep 19 14:24 chap-secrets.example
-rw------- 1 root root 117 Jan 4 2021 chap-secrets.pacsave
-rw------- 1 root root 77 Sep 19 14:24 pap-secrets.example
-rw------- 1 root root 116 Jan 4 2021 pap-secrets.pacsave
The Arch Linux package manager, pacman
,
will save copies of files if it installs new versions.
Those are the chap-secrets.pacsave
and pap-secrets.pacsave
files.
I didn’t recognize this immediately,
and did some paniced searching for copies I might have saved.
I finally remembered that .pacsave
is the suffix pacman
puts
on config files it might overwrite when installing a new package version.
The solution:
sudo cp /etc/ppp/pap-secrets.pacsave /etc/ppp/pap-secrets
sudo cp /etc/ppp/chap-secrets.pacsave /etc/ppp/chap-secrets
sudo systemctl restart mypppd.service
The mppe-stateful entry in /etc/ppp/peers/centurylink
did not matter.
I took it out, restarted pppd
, and everything still works.
The failure of pppd
to set up a connection to CenturyLink
also deeply confused dnsmasq
, which I’m using for DNS aggregation.
journalctl -u dnsmasq
shows a whole mess of problems like:
Oct 20 17:42:46 monarch dnsmasq[927]: query[A] ssl.gstatic.com.glamour from 10.0.20.35
Oct 20 17:42:46 monarch dnsmasq[927]: config error is REFUSED (EDE: network error)
dnsmasq
also tried to send A and AAAA queries to the configured DNS servers
that it should have answered out of its own cache and local DNS config.
Oct 20 19:49:40 monarch dnsmasq[927]: query[A] monarch.glamour from 10.0.20.35
Oct 20 19:49:40 monarch dnsmasq[927]: forwarded monarch.glamour to 9.9.9.9
Oct 20 19:49:40 monarch dnsmasq[927]: forwarded monarch.glamour to 8.8.8.8
Oct 20 19:49:40 monarch dnsmasq[927]: query[AAAA] monarch.glamour from 10.0.20.35
Oct 20 19:49:40 monarch dnsmasq[927]: forwarded monarch.glamour to 8.8.8.8
Oct 20 19:49:40 monarch dnsmasq[927]: reply monarch.glamour is NXDOMAIN
Oct 20 19:49:40 monarch dnsmasq[927]: reply monarch.glamour is NXDOMAIN
monarch.glamour is 10.0.0.1 on my LAN.
dnsmasq
knows this because that name → IP address matchup is
in /etc/hosts
on the Dell R530.
I think this is due to no PPP connection, no route, to any big DNS server
like 8.8.8.8 or 9.9.9.9.
The PPP-over-ethernet plugin, rp-pppoe,
got updated 2024-09-20 without me even noticing.
Between ppp
and rp-pppoe
, this connection to CenturyLink seems to
be on shakey metaphysical ground.