Getting website Offline though CLI

Mirroring a site was never that easy. No software required just a single line command all is done.Let me explain how i did that.
1. Found a site had good AWK manual.
2. Went through the AWK manual and located two good options to the command that worked just well, though i had few wrong attempts earlier.
- m : Mirrors the web site
[man] Turn on options suitable for mirroring. This option turns on recursion and time-stamping, sets infinite recursion depth and keeps FTP directory listings. It is currently equivalent to -r -N -l inf --no-remove-listing.[/man]

-k : Makes the local copy of site browsable, by making all of the links relative to the local location.
[man]After the download is complete, convert the links in the document to make them suitable for local viewing. This affects not only the visible hyperlinks, but any part of the document that links to external content, such as embedded images, links to style sheets, hyperlinks to non-HTML content, etc.[/man]

-w : Introduces a delay of x secs between each hit to the server and hence, prevents our IP identified as a crawler.
[man]Wait the specified number of seconds between the retrievals. Use of this option is recommended, as it lightens the server load by making the requests less frequent. Instead of in seconds, the time can be specified in minutes using the "m" suffix, in hours using "h" suffix, or in days using "d" suffix.[/man]

So finally the command for mirroring the whole website looks like this.
#wget -mk -w 5 http://www.server.com

The '5' mentioned here along with -w means the number of seconds to wait.

Comments

Post a Comment

Popular posts from this blog

Address book configuration [LDAP] on Evolution

Select new address book creation using LDAP from the Contacts tab. General Tab - Name : My LDAP [Any Name] Server : Port : 389 Encryption : No Encryption Login method : Using DN Login : Details tab - DC=xxxx,DC=xxxx,DC=xxxx [This can be searched by evolution search button over there] Search scope : Sub Search Filter :(objectclass=*) Edit > Prefs > Auto-completion - Select check box for : Always show addressses of the autocompleted contact Enable the check box for LDAP server [with the name defined at the time of creation as 'My LDAP' under general tab].
Read more

Startup error with Thunderbird 3.0

Updated to Thunderbird 3.0 released a month back from its beta version with lightning installed but was greeted with a 2 error messages. On starting the client. Error dialog: An error was encountered preparing the calendar located at moz-profile-calendar:// for use. It will not be available. The issue was locked under bugzilla at redhat, to be a schema issue with the database used for the calender which was to be updated. So finally followed the steps and the issue was solved, and the calender was back in action. 1. Closed Thunderbird. 2. Switched to the profile directory of thunderbird. # /home/sawrub/.thunderbird/*********.default 3. Located the directory holding the calender data. ll calendar-data/ 4. There was the local.sqlite file which actually is a SQLite database. 5. Enter the sqlite console, by executing the command an dwas greeted with the corresponding prompt as. [sawrub@mybox calendar-data]$ sqlite3 local.sqlite SQLite version 3.6.12 Enter ".help"...
Read more

Today was the ....

Here is the latest a fully off topic thing....My farewell mail sent to the people i worked/ interacted at XXXXXX India [Mail] From : Saurabh Sharma Reply-to : sawrub@yahoo.co.in To : Bcc : Subject : Today was the .... Date : Fri, 04 Dec 2009 15:30:38 +0530 Hi All, By god's grace, your well wishes and efforts from my side, finally the day has come when I can subject a mail like this, and be in a situation to bid you all a special bye. And this is how I'll like to orate out the whole thing. - Today was the last morning when I had to get up early at 5:30 AM get ready and then run past the stray dogs barking at me for disturbing them at that hour, and also fearing a call from office that the cab has left from the pick-up point, though i had been a victim of the same a couple of times. - Today was the last morning when I was supposed to log-in to my Linux box and do some maths for getting the sum up at 8+ Hrs for the activities done yesterday and then stamp the same...
Read more