Mirroring a website from WGET

Wget is a very useful tool for system admins. You can download files and move files between servers through wget. Also you can mirror static websites for offline browsing.

It’s a handy tool for programmers to find html respond header details.

Here are some useful commands.
Mirror a website
wget -mk -w 20 https://jwellex.com/

Find broken links
wget -o wget.log -r -l 10 –spider https://jwellex.com/
grep -B 2 ‘404’ wget.log | grep “http” | cut -d ” ” -f 4 | sort -u

Get respond header
wget -S http://www.bbc.co.uk
wget -S –spider http://www.bbc.co.uk