Home Forums Classifieds Blogs Today's Posts Search Social Groups



  
SIGN-UP
Notices

Glock Talk
Welcome To The Glock Talk Forums.

 
  
Reply
 
Thread Tools Display Modes
Old 12-04-2004, 20:28   #1
Team Greenbaum
Senior Member
 
Team Greenbaum's Avatar
 
Join Date: Jul 1999
Posts: 742
Web Site Download Tool

Does anyone know a good web site download tool? One that downloads media and archive files? I've tried a few, most recently "7 Download Service". It worked great on html and image files but I never could get it to download .mpg, .mp3 or .zip files...
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 signatures.
Team Greenbaum is offline   Reply With Quote
Old 12-04-2004, 20:57   #2
chevrofreak
Senior Member
 
chevrofreak's Avatar
 
Join Date: Dec 2001
Location: Billings, Montana
Posts: 9,364


I like Net Transport from Xi
__________________
smoke free since 4:58AM July 29th 2002!

To view links or images in signatures your post count must be 10 or greater. You currently have 0 signatures.
<-- currently 0 auction(s) running

To view links or images in signatures your post count must be 10 or greater. You currently have 0 signatures.
The best IWB holster money can buy!
chevrofreak is offline   Reply With Quote
Old 12-05-2004, 07:15   #3
Team Greenbaum
Senior Member
 
Team Greenbaum's Avatar
 
Join Date: Jul 1999
Posts: 742
Is there a way to make it download the whole web site? I could only get it to download 1 file at a time. I'm looking for something that will crawl a site and download the whole thing.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 signatures.
Team Greenbaum is offline   Reply With Quote
Old 12-05-2004, 11:01   #4
Sinister Angel
I'd Hit It!
 
Join Date: Oct 2004
Location: Traverse City, Michigan
Posts: 1,433
Send a message via AIM to Sinister Angel Send a message via Yahoo to Sinister Angel
Do they have a port of wget for windows?
Sinister Angel is offline   Reply With Quote
Old 12-05-2004, 12:00   #5
Team Greenbaum
Senior Member
 
Team Greenbaum's Avatar
 
Join Date: Jul 1999
Posts: 742
Yep! WGET for Windows
I'll check it out.
Thanks!
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 signatures.
Team Greenbaum is offline   Reply With Quote
Old 12-05-2004, 12:35   #6
Sinister Angel
I'd Hit It!
 
Join Date: Oct 2004
Location: Traverse City, Michigan
Posts: 1,433
Send a message via AIM to Sinister Angel Send a message via Yahoo to Sinister Angel
Quote:
Originally posted by Scottbert
Yep! WGET for Windows
I'll check it out.
Thanks!
Glad to help! I know it works wonders on my linux box.
Sinister Angel is offline   Reply With Quote
Old 12-06-2004, 00:35   #7
lomfs24
Senior Member
 
lomfs24's Avatar
 
Join Date: Apr 2003
Location: Kansas
Posts: 4,813
Send a message via AIM to lomfs24 Send a message via Yahoo to lomfs24
Quote:
Originally posted by Sinister Angel
Glad to help! I know it works wonders on my linux box.
How do you make wget pull an entire website? I have worked with it a little but mostly as a single file transport tool.
__________________
The simple believeth every word: but the prudent man looketh well to his going. ~Proverbs 14:15
lomfs24 is offline   Reply With Quote
Old 12-06-2004, 09:02   #8
grantglock
/dev/null
 
grantglock's Avatar
 
Join Date: Feb 2004
Location: Iowa
Posts: 932
Send a message via Yahoo to grantglock
Quote:
Originally posted by lomfs24
How do you make wget pull an entire website? I have worked with it a little but mostly as a single file transport tool.

wget -r http://glocktalk.com
grantglock is offline   Reply With Quote
Old 12-06-2004, 10:20   #9
lomfs24
Senior Member
 
lomfs24's Avatar
 
Join Date: Apr 2003
Location: Kansas
Posts: 4,813
Send a message via AIM to lomfs24 Send a message via Yahoo to lomfs24
Quote:
Originally posted by grantglock
wget -r http://glocktalk.com
kewl. Thanks.
__________________
The simple believeth every word: but the prudent man looketh well to his going. ~Proverbs 14:15
lomfs24 is offline   Reply With Quote
Old 12-14-2004, 08:55   #10
Team Greenbaum
Senior Member
 
Team Greenbaum's Avatar
 
Join Date: Jul 1999
Posts: 742
Wget rocks! Almost.

It worked great on the first site I tried it on. It downloaded everything, including media files and changed all URLs to local links. However, on the second site I tried, it immediately gets a 302 redirect to a completely different site. It's as if the web server is recognizing that I'm using wget instead of a browser and responding with the 302 redirect. Any ideas on what I can do to fix this?

Here are the options I'm using:
wget --output-file="wget.log" --recursive --level=inf --timestamping --convert-links --wait=1 --random-wait https://user:password@www.website.com
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 signatures.
Team Greenbaum is offline   Reply With Quote
Old 12-14-2004, 09:42   #11
HerrGlock
CLM Number 2
Scouts Out
 
HerrGlock's Avatar
 
Join Date: Dec 2000
Posts: 64,487


Quote:
Originally posted by Scottbert
It's as if the web server is recognizing that I'm using wget instead of a browser and responding with the 302 redirect. Any ideas on what I can do to fix this?
The web site is seeing that you have a getter instead of a browser, you are right.

There are things you can do, but most of the people who really don't want you to slurp their site already know them and have something to counter that too.

There is, however, a plugin for firefox/mozilla that would work as it's a browser doing the slurping.

Just something to think about.
DanH
__________________
Sent from my rotary phone
"The way I see it as soon as a baby is born, he should be issued a banjo!"- Linus Van Pelt
UNIX - Not just for Vestal Virgins any more
HerrGlock is offline   Reply With Quote
Old 12-14-2004, 10:17   #12
Sinister Angel
I'd Hit It!
 
Join Date: Oct 2004
Location: Traverse City, Michigan
Posts: 1,433
Send a message via AIM to Sinister Angel Send a message via Yahoo to Sinister Angel
Actually, you can have WGET send a forged AGENT header or maybe its a refer header as well.

--referer=url
Include `Referer: url' header in HTTP request. Useful for retrieving documents with server-side processing that assume they are always being retrieved by interactive web browsers and only come out properly when Referer is set to one of the pages that point to them.


and


-U agent-string
--user-agent=agent-string
Identify as agent-string to the HTTP server.

The HTTP protocol allows the clients to identify themselves using a "User-Agent" header field. This enables distinguishing the WWW software, usually for statistical purposes or for tracing of protocol violations. Wget normally identifies as Wget/version, version being the current version number of Wget.

However, some sites have been known to impose the policy of tailoring the output according to the "User-Agent"-supplied information. While conceptually this is not such a bad idea, it has been abused by servers denying information to clients other than "Mozilla" or Microsoft "Internet Explorer". This option allows you to change the "User-Agent" line issued by Wget. Use of this option is discouraged, unless you really know what you are doing.

Hope this helps!
Sinister Angel is offline   Reply With Quote

 
  
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump




All times are GMT -6. The time now is 12:41.




Homepage
FAQ
Forums
Calendar
Advertise
Gallery
GT Wiki
GT Blogs
Social Groups
Classifieds


Users Currently Online: 1,288
361 Members
927 Guests

Most users ever online: 2,672
Aug 11, 2014 at 2:31