--001636c5b06f43cb970495b79575
Content-Type: text/plain; charset=UTF-8

On Tue, Nov 23, 2010 at 2:19 PM, Luis G. <l17339 / gmail.com> wrote:

> Hey guys... Thanks for your replies.
>
> I thought the program I built was not so heavy for the website I'm
> trying to get info from.
> The thing is, I'm accessing the website to get information, but I just
> access there to specific pages in that domain, so I'm not really
> crawling everything.
> I build the url based in some info I have in my DB and after I have the
> url, I access to that url directly and collect the information in that
> specific page. And more, the page I'm accessing have just some <p> html
> tags, so not so much info to look into. And I'm just accessing the
> web-pages I didn't access before (just the new ones).
>
> So, of course I understand that they need to protect webserver, but I
> think my program is not really a threat :D
>
> I'm gonna run the script in steps and in different days like I thought
> before and like you told me.
>
>
One more thought, what are you using for user-agent? Some sites block empty
or known to be automated user-agents.

Regards,
Ammar

--001636c5b06f43cb970495b79575--