In most cases, you will probably use SYN scan, however there are circumstances where other types of scanning may be needed. Unfortunately, modern firewalls detect different port scanning techniques. And there comes well known tool called nmap, which supports few of them which will help you to bypass firewalls. When full scan is chosen, there will be 65 536 ports checked.

TCP SYN 

As mentioned, this is the most common (and default for nmap) of TCP port scans. Firstly, it needs root privileges to run it, because raw sockets are used to send packet with SYN flag. If SYN/ACK is received, the port is open. If RST/ACK – closed. In case of no response, we can set how many times scanner should resend request (--max-retries in nmap). If still does not receives a response – it will be marked as filtered.

Hence SYN scan never establishes a TCP connection, it is relatively stealthy.

TCP ACK

This type of scan is used to check if firewall is tracking the state of existing TCP connections. A TCP ACK scan sends a packet with the ACK flag set. When request is received by the server, it will probably assume that the second party is attempting to continue three way handshake, so it will send back a packet with RST flag.

If firewall do not track opened connections, it will allow to send back RST response, hence we can assume that port is unfiltered.

If it tracks TCP connections – there will be no matches in internal TCP sessions – it will drop the packet or send response via ICMP with type 3 (code may vary depending on OS).

Lastly, when firewall is configured in a proper way, we will do not receive any response, because all of the ACK requests will be dropped.

TCP Connect

A TCP connect scan establishes TCP connection using connect() function and then tears it down. It is used as default when user has no root privileges or IPv6 network is being scanned. Disadvantage of this type of scan is probing may be logged in server and therefore easily detected. Hence it needs more requests to complete, it is also more time-consuming process. Furthermore, it can lead to overload and sometimes produce DoS situations.

NULL  |  FIN  |  Xmas

NULL scan has no flags set.
FIN scan has FIN flag.
Xmas scan has FIN, PSH and URG flags – packet lighted like a christmas tree.

All of those are almost the same – the only difference is in flags set. If in response we will receive RST packet – port is considered closed. No response means open|filtered. If there is ICMP message – filtered.

Main advantage of above three scans are low detectability – just a little bigger than in SYN scan. The biggest disadvantage – not every system is compatible with RFC793, so responses may be independent from actual port state.

Maimon

Very similar to above three – it uses FIN/ACK packet. In all cases RST flag should be received, but OS based on`BSD will respond only when a port is closed.

TCP window

This is slightly modified version of the ACK scan. It uses TCP Window field to determine whether port is open. This field is a part of TCP's flow control, it helps to manage data transmission volumes. Some OSes positive TCP Window size will indicate that port is open (even though a RST packet is being sent). Conversely – when a TCP Window size is 0 – means that port is closed.

This type of scan is not very common and it’s quite unreliable – it needs deep understanding what is going on to draw some conclusions.