Since last year, I’ve been working on a project called EpubPress.
EpubPress and I generally get along great.
It serves my requests. It restores from errors. It does what’s expected.
But all that being said, I’d noticed a persistent issue with EpubPress - sometimes it would just stop talking to me 😭.
EpubPress! Give me the homepage!
*crickets* … *timeout*
EpubPress! Let me connect to you via SSH!
*crickets* … *timeout*
EpubPress! Respond to my pings!
*crickets* … *timeout*
What was going on? It had just created a beautiful book for me an hour ago! Why the cold shoulder?
Whenever this happened, I would go through the same steps:
- Is the server overloaded? Nope. CPU usage was stable.
- Did the app need a restart? Nope. App is running fine.
- Had Nginx failed? Nope. Nginx was working normal.
- Did the logs for the instance have clues? Nope. Nothing out of the usual.
By this point I’d be freaking out. My server was losing ‘nines’ of uptime by the minute!
My last resort was to restart the machine - that put everything back to normal. Phew! 😌
…But only for so long.
By the next day, the issue would be happening again 😫.
Enough is enough
Communication is important to me - so I was bothered that my cherished server wasn’t using its words.
After multiple occurrences of this issue, I decided to get to the bottom of it.
Talking to friends
When someone is giving you the silent treatment - sometimes you can find out why via a friend.
In this metaphor, my friend was DownForEveryoneOrJustMe.
Harold: Hey DownForEveryoneOrJustMe, EpubPress is ignoring me. Is it talking to you?
DownForEveryoneOrJustMe: EpubPress is talking to me. Problem is only for you!
Huh? That can’t be true…
I tried talking to EpubPress via another server.
Harold: Hey HaroldTreen.com, is EpubPress talking to you these days?
HaroldTreen.com: It sure is! EpubPress is responding to all my texts.
What?! Betrayal! 🔥😡🔥
I now knew that…
- EpubPress was not responding to anything I sent it.
- It wasn’t because EpubPress was busy. I checked and it was sitting idle at home.
- It wasn’t because EpubPress was unwell. All the internal components were operational.
- It wasn’t because EpubPress was anti-social. It was talking to everyone else.
I was stumped.
Reaching out to others
At this point, I decided I needed more guidance. What else could cause my server to ignore me at random?
Better debugging ™
My initial debugging strategies were pretty rudimentary.
- Make a request using Chrome.
- Make a request using
grepsome log files.
Talking to others introduced to me to an awesome set of tools that helped get to the root of the issue.
netstat (network statistics) is a command line tool for displaying information on network traffic. I used it to see all the ports my server was listening to.
netstat -ap, I was also able to see what processes were listening to each port. There was a normal number of connections and my app was listening to the right ports…
In the output you can see
pm2 connected and listening away:
Active UNIX domain sockets (servers and established) Proto RefCnt Flags Type State I-Node PID/Program name unix 3 [ ] STREAM CONNECTED XXXXXX 31485/nginx -g daem unix 3 [ ] STREAM CONNECTED XXXXXX 31485/nginx -g daem unix 2 [ ACC ] STREAM LISTENING XXXXX 1486/.pm2) unix 2 [ ACC ] STREAM LISTENING XXXXX 1486/.pm2)
Maybe my packets aren’t making it to the server?
mtr (my traceroute) is a program that combines traceroute and ping. It continually pings the provided host and shows a live view of the path taken to deliver the ping.
mtr <host>, I was able to see how packets were travelling to EpubPress. I noticed packets were getting blocked somewhere down the line (all the
It’s interesting to note that some intermediate servers were also returning
???s. This could be because they are configured to forward traffic but ignore pings. That’s fine, but my server is not one to ignore a friendly ping.
Could it be a server in the middle ignoring me?
tcpdump is a program that lets you inspect all the packets being sent and received by a machine. It was like peeking into EpubPress’s brain and seeing every word received and how it was responding.
tcpdump host <my-ip> I discovered that my packets were arriving, but no packets were being returned. Not even a modest
SYN-ACKs was being returned 😥.
This is what
tcpdump produced on the server while my laptop was sending pings:
06:08:21.069515 IP pool-XXX-XX-XXX-XX.nycmny.fios.verizon.net > epubpress-backend.c.epubpress-XXXX.internal: ICMP echo request, id 10677, seq 33022, length 44 06:08:21.167963 IP pool-XXX-XX-XXX-XX.nycmny.fios.verizon.net > epubpress-backend.c.epubpress-XXXX.internal: ICMP echo request, id 10677, seq 33023, length 44 ...etc
This is what working output looks like:
06:13:49.763522 IP pool-XXX-XX-XXX-XX.nycmny.fios.verizon.net > epubpress-backend.c.epubpress-XXXX.internal: ICMP echo request, id 10770, seq 33161, length 44 06:13:49.763556 IP epubpress-backend.c.epubpress-XXXX.internal > pool-XXX-XX-XXX-XX.nycmny.fios.verizon.net: ICMP echo reply, id 10770, seq 33161, length 44
ping uses a protocol called
ICMP (Internet Control Message Protocol). No matter how many
ICMP echo requests I sent, EpubPress would not reply.
Could it be a firewall issue?
netcat is a program for reading/writing directly to a network connection. I think of it like
cat, but for printing whatever a given port receives.
nc <host> <port> on both machines, I was able to test if a program separate from my application could communicate back to my laptop. As expected, this did not work - but I tried it anyway because it was fun using all these new tools.
On working computers, I was able to type in the terminal and see the characters appear on the other end! Wow!
Teehee, I’m chatting with myself!
*5 minutes later*
Alright, enough tomfoolery! What is our firewall doing…
iptables is a tool for modifying tables of firewall rules. It can be used to define how individual packets should be treated.
iptables -L I was able to list all the rules in the tables. Sure enough, there it was:
chain sshguard (1 references) target prot opt source destination DROP all -- <my-ip> anywhere
sshguard chain had a rule for blocking all my requests! 😱
Who is this sshguard character?!
sshguard is a tool for protecting against brute force attacks. It aggregates and inspects system logs to detect suspicious activity. It then blocks the suspicious traffic.
In other words, EpubPress had this friend
sshguard who I didn’t know existed.
sshguard had noticed me constantly talking to EpubPress and become worried whether this was a good use of EpubPress’s time.
sshguard decided to brainwash EpubPress into ignoring me so it could focus on creating books.
Issue Solved 🎉
I had a talk with
sshguard, settled our differences and had it add me to its whitelist.
sshguard -w <my-ip>
Now whenever I to talk to EpubPress,
sshguard doesn’t try to interfere.
The moral of the story?
- Smart tools exist for protecting against brute force attacks!
- Sometimes strange problems arise and it makes no sense what’s going on - but if you dig deep enough you’ll likely find a fascinating explanation!
- Reach out to others when you’re facing an issue - they might have a wider array of tools to help debug the problem.
EpubPress and I resolved our differences and we are looking forward to making more beautiful ebooks together.