Being involved in different kind of technologies nowadays is inevitable, especially when you work in Data Center’s, and when I say it, I mean Big Data Centers.
Last week I was trying to figure out a problem that occurred to me in an environment inside a Data Center. Of course the first thing that I tried was to isolate the problem as much as possible in order to find out the issue and solve it. Well.. let me show you how it looks like at the beginning.
As shown in the diagram, we have a Citrix Farm with a subnet of 192.168.50.0/23. This environment is secured by the Firewall ‘FWN10’ which is a Checkpoint firewall. I used Citrix Secure Gateway as a platform to access all the devices inside the Data Center. I assume you already know how Citrix works, if not than have a look at this link.
So, once on Citrix, I tried using Putty (already installed by Citrix Admins) to SSH to the FWN25 (destination) Firewall, and BANGGG I get a ‘Connection Error’. Hmm, well probably some interruption during my first try I thought, let’s try again.. aannd.. ‘Connection Error’, well this time it doesn’t makes sense at all I thought. Having a look at the routing table and policies on all the firewalls FWN10 – FWN15 – FWN25 doesn’t show any problem at all, the source and destination subnets are permitted on all firewalls and even the SSH ports are allowed.
Trying different things to troubleshoot the issue and nothing heads me to the right solution. Well… seem’s like everything is OK (routing table, policies etc), but how come I can’t SSH to the destination.
In a moment, I was thinking why don’t I try a TCPdump since Juniper SRX (dst firewall) supports this tool. Immediatelly plugged the console cable to the SRX and turned on TCPdump by using this command:
‘ tcpdump -i eth0 src 192.168.50.25 and dst 10.25.30.10 and dst port 22 ’
After typing this I waited for the results, and already knew that I should get some hits on the firewall since the source is specified correctly, the destination IP and port number is also correct, but what I got from this was NOTHING at all, looks Strange I thought because I should see the source (192.168.50.25) trying to connect to port 22 on destination firewall (FWN25) but TCPdump doesn’t give me the prove that this is happening.
Somehow I felt like either I’m wrong or the device is not responding the right way because I don’t see the expected result. Anyway, now I was thinking to try a sniff by using as source everything and destination 10.25.30.10 port 22, in this way I should see every device that is trying to connect to SSH port 22, let’s go and do it:
‘ tcpdump -i eth0 dst 10.25.30.10 and dst port 22 ’
Lucky me I thought, Now I see some source IP trying to SSH in port 22 on our firewall, but something looks Strange here, I see a source that I didn’t expect to see, Woow!!! What happened, I thought??
Source IP ‘146.210.210.35’ is trying to SSH, but the firewall blocked it, of course “Policies”. But why do I get a different source and not the one from Citrix Farm. Shortly I found out the environment of this NEW source and let’s have a look at the new diagram below:
After a Call with the Citrix guys I found out that all the applications available (Putty, WinScp, Browsers etc..) through the Citrix Secure Gateway (Citrix Farm) are actually distributed, which means they are installed on the Server Farm Environment (subnet 146.210.210.0/27) and the Citrix gateway (subnet 192.168.50.0/23) by itself is located in the Citrix Farm Environment.
Well, to be honest I never thought that Applications and Citrix are split in this way on different Environments, but finally looks like I solved the mystery, and yes I DID IT. Now it makes sense why we get a source of 146.210.210.35, because this is the server farm where Putty is installed, and it’s as simple as that.
Finally the configuration on FWN25 was changed by accepting connections on port 22 from the new source (146.210.210.35) and Bingo, now we are able to SSH.
At first this looked as simple issue, but what I learned from this one is that not always you can rely on first impression, because sometimes during troubleshooting it happens that everything from the config’s is OK but it doesn’t work.
My suggestion: “Always use kind of sniffing stuff (tcpdump, wireshark, etc), in this way you can see everything what happens on your network and you’ll never be wrong, also it avoids a lot of time consuming “.
Categories: General Networking
CCIE #49534 
Thanks for sharing it buddy 🙂