It is a familiar incident-room sentence: “The service is up, but nobody can connect.” Someone spots port 8080 in a process list, someone else opens a firewall rule, and twenty minutes disappear. The missing question is usually not whether a port is “open.” It is where the socket is bound, in which network namespace it exists, and from whose point of view it can be reached.

Start at the process, then walk outward one boundary at a time. That habit is slower than guessing for about thirty seconds—and much faster for the rest of the outage.

The first snapshot: ask the kernel about sockets

Terminalbash
sudo ss -lntup
Netid State  Recv-Q Send-Q Local Address:Port  Peer Address:Port Process
tcp   LISTEN 0      4096   127.0.0.1:8080      0.0.0.0:*     users:(("example",pid=1234,fd=7))
udp   UNCONN 0      0      0.0.0.0:5353        0.0.0.0:*     users:(("example",pid=5678,fd=9))

Read the flags before reading the example output

  • -l selects listening sockets, -n keeps addresses and ports numeric, -t selects TCP, -u selects UDP, and -p requests owning-process information.

  • sudo is not required to see every socket, but Linux may hide another user’s process name, PID, or file descriptor without sufficient privilege. A blank process column does not mean the socket has no owner.

  • The two rows are illustrative, not output captured from this development host. Your addresses, queues, processes, and ports will differ.

  • ss comes from iproute2 and is the modern first choice. netstat remains useful on old systems, but it is no longer the primary path here.

The bind address tells you who gets an invitation

  • Loopback — `127.0.0.1` or `[::1]`: intended for the same host and namespace. Opening a perimeter firewall does not make a loopback-only service remotely reachable.

  • IPv4 wildcard — `0.0.0.0`: bound on all eligible local IPv4 addresses. This widens the local bind; it does not bypass any firewall.

  • IPv6 wildcard — `[::]`: bound on the IPv6 unspecified address. Whether that socket also accepts IPv4-mapped connections depends on application and system IPV6_V6ONLY behavior, so test both families instead of assuming.

  • Specific address — such as `192.0.2.10`: limited to that local address. A service can work through one interface and fail through another by design.

TCP listens; UDP waits without a handshake

TCP has a LISTEN state because the kernel waits for connection handshakes. UDP is connectionless, so there is no equivalent handshake state. ss commonly shows an unconnected UDP endpoint as UNCONN; that does not mean the application is broken or that a remote UDP probe must receive a reply.

Terminalbash
sudo ss -ltnp 'sport = :8080'
sudo ss -lunp 'sport = :5353'
<matching TCP listeners, if any>
<matching UDP sockets, if any>

A narrow query is easier to trust during an outage

  • sport means source, or local, port in this ss filter. Keep the expression quoted so the shell passes it intact.

  • The first command limits output to listening TCP sockets; the second selects locally bound UDP sockets.

  • No output means no matching socket was visible in this namespace at that moment. It does not establish what is running inside another container or namespace.

Follow the PID, but do not stop at the PID

Terminalbash
sudo lsof -nP -iTCP:8080 -sTCP:LISTEN
sudo systemctl list-sockets --all --no-pager
<processes with a TCP listener on port 8080>
<systemd socket units, listening addresses, and services they activate>

Two ownership models can explain one port

  • lsof treats sockets as open files. -nP suppresses host and service-name conversion, while -iTCP:8080 -sTCP:LISTEN selects only TCP listeners on that port.

  • systemctl list-sockets reveals socket activation: systemd may own the listening file descriptor and launch or wake the service only when traffic arrives. Looking only for the expected daemon PID can therefore mislead you.

  • The systemd listing is designed for humans; upstream documentation warns that addresses may contain spaces, so do not parse its columns as a stable machine interface.

  • If a process is unexpected, inspect its unit, executable, package provenance, configuration, and logs before stopping it. A port number alone is not proof of compromise.

Prove the shortest path before testing the whole network

Terminalbash
nc -vz -w 3 127.0.0.1 8080
nc -vz -w 3 <server-lan-address> 8080
Connection to 127.0.0.1 8080 port [tcp/*] succeeded!
<success, refusal, timeout, or routing error from the LAN-address path>

Each connection attempt answers one carefully bounded question

  • -z asks netcat to test connection setup without an application payload, -v reports the result, and -w 3 bounds the wait. Option behavior varies among netcat implementations, so check local nc -h when portability matters.

  • Loopback success plus LAN-address failure points toward the bind address, host policy, or local routing—not proof of an upstream internet problem.

  • A refusal normally means the destination was reached but nothing accepted that TCP connection at that address and port. A timeout can mean filtering, packet loss, or an unreachable return path; it does not identify the cause by itself.

  • Replace the placeholder only with an address you administer. These displayed results are illustrative and were not captured from a production service.

Now walk outward through the actual boundaries

  1. Confirm the socket and bind address with ss in the environment where the service process runs.

  2. Test loopback, then the host’s specific interface address. Do not jump straight to an internet scanner.

  3. Inspect the host firewall using the tool that owns policy on this system—such as nftables, firewalld, or UFW—without stacking contradictory managers.

  4. For containers, confirm published ports and inspect both the container and host network namespaces. A listener inside a container is not automatically published on the host.

  5. For virtual machines or Kubernetes, account for hypervisor NAT, Services, ingress, NetworkPolicy, and the node or pod boundary relevant to the client.

  6. For cloud or internet paths, examine security groups, network ACLs, load balancers, router forwarding, NAT, and upstream filtering as separate layers.

  7. Test from an authorized client at the exact boundary users occupy. Record the source, destination address family, port, protocol, and time.

Namespaces explain the “but I can see it” argument

Socket tables belong to network namespaces. Running ss on the host does not necessarily show a process isolated in a container namespace, and running it inside the container does not prove a host port was published. Likewise, a rootless container may traverse user-space networking that changes how ownership and reachability appear.

Terminalbash
sudo nsenter -t <service-pid> -n ss -lntup
<sockets visible in the service process network namespace>

Enter only a namespace you are authorized to inspect

  • -t selects the target process and -n requests its network namespace; the trailing command executes ss inside that view.

  • Replace the placeholder with a confirmed service PID. The command is read-only, but entering another process namespace requires privilege and can expose operational metadata.

  • A PID can exit or be reused between observation and inspection. Reconfirm the process identity immediately before relying on the result.

Remote scanning is a permission boundary

When you own or are explicitly authorized to assess the target, a narrowly scoped remote test shows what that client can reach. Nmap distinguishes open, closed, and filtered observations, but those labels describe the scan path and technique—not an eternal property of the server.

Terminalbash
nmap -sT -Pn -p 8080 <authorized-server-address>
PORT     STATE     SERVICE
8080/tcp <open|closed|filtered> <service-label>

Risk level: caution. Review the command before running it.

Keep the remote test narrow, visible, and agreed

  • -sT uses the operating system’s TCP connect() path, -Pn skips host discovery, and -p 8080 limits the request to one known port.

  • Only scan infrastructure you own or have explicit permission to test. Coordinate production testing because connection attempts may trigger logs, alerts, rate limits, or fragile services.

  • The service label is a port-name association, not proof of which application is running. Version detection is intentionally omitted from this minimal reachability test.

  • UDP diagnosis requires protocol-aware probes and often cannot distinguish an open service from a filtered or silent one based on no reply alone.

Leave the incident with evidence, not folklore

  • Socket evidence: protocol, local address, port, namespace, process or socket unit, and observation time.

  • Path evidence: client source, destination address and family, each policy boundary, and the result at application level.

  • Change evidence: exact firewall, publish, bind, or cloud-policy change; approver; verification; and rollback.

  • Negative evidence: whether the result was refusal, timeout, DNS failure, TLS failure, or wrong application response. “Port down” throws away the useful part.

Continue from the boundary you found

Primary references