logfile reporting for search engine bots

[Chris Hope]

Is there any reporting software that parses weblogs and gives you stats
on when they visited, which pages they requested etc? I use AWStats for
my sites' statistics and it only shows the number of requests in a
month and when the last one was.

If there's nothing available I'll probably write something like this
myself in the New Year and make it available as open source. If I do,
what sort of features would people want in such software? I would most
likely write it in Perl - I personally prefer PHP for programming but
have found Perl to be much faster at processing logfiles than PHP.

--
Chris Hope - The Electric Toolbox - http://www.electrictoolbox.com/

[Tony]

Chris Hope wrote:

> Is there any reporting software that parses weblogs and gives you stats
> on when they visited, which pages they requested etc? I use AWStats for
> my sites' statistics and it only shows the number of requests in a
> month and when the last one was.
>
> If there's nothing available I'll probably write something like this
> myself in the New Year and make it available as open source. If I do,
> what sort of features would people want in such software? I would most
> likely write it in Perl - I personally prefer PHP for programming but
> have found Perl to be much faster at processing logfiles than PHP.


Chris,

Take a look at http://www.weblogexpert.com/

It might give you a few ideas. There's also a free LITE version.
http://www.weblogexpert.com/lite.htm (shows a comparison chart)


--
Tony

[Chris Hope]

Tony wrote:

> Chris Hope wrote:
>
>> Is there any reporting software that parses weblogs and gives you
>> stats on when they visited, which pages they requested etc? I use
>> AWStats for my sites' statistics and it only shows the number of
>> requests in a month and when the last one was.
>>
>> If there's nothing available I'll probably write something like this
>> myself in the New Year and make it available as open source. If I do,
>> what sort of features would people want in such software? I would
>> most likely write it in Perl - I personally prefer PHP for
>> programming but have found Perl to be much faster at processing
>> logfiles than PHP.

> Take a look at http://www.weblogexpert.com/
>
> It might give you a few ideas. There's also a free LITE version.
> http://www.weblogexpert.com/lite.htm (shows a comparison chart)

That looks like a nice stats package. What it doesn't cover is the
search engine bot stuff which is what I'm after. Re-reading my post now
I had that in the subject line but didn't mention it in the body...

--
Chris Hope - The Electric Toolbox - http://www.electrictoolbox.com/

[Tony]

Chris Hope wrote:


> That looks like a nice stats package. What it doesn't cover is the
> search engine bot stuff which is what I'm after. Re-reading my post now
> I had that in the subject line but didn't mention it in the body...


Example here:
http://www.weblogexpert.com/sample/Referrers.htm#Search_Engines



Not sure how up to date their bot listing is though.


If so, this may be of use in designing your own (June 2004):
http://dev.pnconcept.com/showbotlist.php


--
Tony

[Chris Hope]

Tony wrote:

> Chris Hope wrote:
>
>> That looks like a nice stats package. What it doesn't cover is the
>> search engine bot stuff which is what I'm after. Re-reading my post
>> now I had that in the subject line but didn't mention it in the
>> body...
>
> Example here:
> http://www.weblogexpert.com/sample/Referrers.htm#Search_Engines
>
> Not sure how up to date their bot listing is though.

That shows who has come to the site via means of a search engine and
what keywords were searched on etc. What I want is a report which shows
me what search engine spiders are spidering my site, which pages they
have requested and when.

> If so, this may be of use in designing your own (June 2004):
> http://dev.pnconcept.com/showbotlist.php

That botlist is excellent. Thank you.

--
Chris Hope - The Electric Toolbox - http://www.electrictoolbox.com/

[Tony]

Chris Hope wrote:


> That shows who has come to the site via means of a search engine and
> what keywords were searched on etc. What I want is a report which shows
> me what search engine spiders are spidering my site, which pages they
> have requested and when.


Maybe adapt this (PHP,) to INSERT into a db each time the bot visits
http://www.nukecops.com/postt1630.html

by 'including' the file on each page of your site.


Then build the report from the db as required.


>>If so, this may be of use in designing your own (June 2004):
>>http://dev.pnconcept.com/showbotlist.php
>
>
> That botlist is excellent. Thank you.


Glad to help.

--
Tony

[Big Bill]

On Tue, 21 Dec 2004 10:46:17 +1300, Chris Hope
<blackhole@electrictoolbox.com> wrote:

>Tony wrote:
>
>> Chris Hope wrote:
>>
>>> That looks like a nice stats package. What it doesn't cover is the
>>> search engine bot stuff which is what I'm after. Re-reading my post
>>> now I had that in the subject line but didn't mention it in the
>>> body...
>>
>> Example here:
>> http://www.weblogexpert.com/sample/Referrers.htm#Search_Engines
>>
>> Not sure how up to date their bot listing is though.
>
>That shows who has come to the site via means of a search engine and
>what keywords were searched on etc. What I want is a report which shows
>me what search engine spiders are spidering my site, which pages they
>have requested and when.
>
>> If so, this may be of use in designing your own (June 2004):
>> http://dev.pnconcept.com/showbotlist.php
>
>That botlist is excellent. Thank you.

You want Robot Manager then. It isn't free.

BB
--
www.kruse.co.uk SEO@kruse.demon.co.uk
home of SEO that's shiny!
--

[Chris Hope]

Big Bill wrote:

> On Tue, 21 Dec 2004 10:46:17 +1300, Chris Hope
> <blackhole@electrictoolbox.com> wrote:
>
>>Tony wrote:
>>
>>> Chris Hope wrote:
>>>
>>>> That looks like a nice stats package. What it doesn't cover is the
>>>> search engine bot stuff which is what I'm after. Re-reading my post
>>>> now I had that in the subject line but didn't mention it in the
>>>> body...
>>>
>>> Example here:
>>> http://www.weblogexpert.com/sample/Referrers.htm#Search_Engines
>>>
>>> Not sure how up to date their bot listing is though.
>>
>>That shows who has come to the site via means of a search engine and
>>what keywords were searched on etc. What I want is a report which
>>shows me what search engine spiders are spidering my site, which pages
>>they have requested and when.
>>
>>> If so, this may be of use in designing your own (June 2004):
>>> http://dev.pnconcept.com/showbotlist.php
>>
>>That botlist is excellent. Thank you.
>
> You want Robot Manager then. It isn't free.

Would that be this one?
http://www.websitemanagementtools.com/robot-manager/

I don't mind paying for software if it's good value for money and will
save me time coding something myself. I might give the trial version of
that a go and see what it's like but I would prefer something that will
work on my server and can be run on a daily basis and show the reports
in a web browser.

--
Chris Hope - The Electric Toolbox - http://www.electrictoolbox.com/

[John Bokma]

Chris Hope wrote:

>>> what sort of features would people want in such software? I would
>>> most likely write it in Perl - I personally prefer PHP for
>>> programming but have found Perl to be much faster at processing
>>> logfiles than PHP.

Also Perl is way better developed and has less quirks compared to PHP.

> That looks like a nice stats package. What it doesn't cover is the
> search engine bot stuff which is what I'm after. Re-reading my post now
> I had that in the subject line but didn't mention it in the body...

If you decide to do it in Perl (sound), check out CPAN.

Also google for "perl log analyzer" etc. you might find a project you can
join.

I am doing my own perl hacking, but it's certainly finished to be usable as
open source (I focus more at getting keywords used in queries anyway).

--
John -> http://johnbokma.com/ Firefox: http://johnbokma.com/firefox/
Perl SEO tools: http://johnbokma.com/perl/
Experienced (web) developer: http://castleamber.com/
Get SEO help: http://johnbokma.com/websitedesign/seo-expert-help.html

[John Bokma]

Chris Hope wrote:

> I don't mind paying for software if it's good value for money and will
> save me time coding something myself. I might give the trial version of
> that a go and see what it's like but I would prefer something that will
> work on my server and can be run on a daily basis and show the reports
> in a web browser.

If you want to pay for custom software (ie written to your specific needs)
drop me a line (mail at johnbokma.com ). You can specify how you want your
reports etc.

--
John -> http://johnbokma.com/ Firefox: http://johnbokma.com/firefox/
Perl SEO tools: http://johnbokma.com/perl/
Experienced (web) developer: http://castleamber.com/
Get SEO help: http://johnbokma.com/websitedesign/seo-expert-help.html

[Chris Hope]

John Bokma wrote:

> Chris Hope wrote:
>
>> I don't mind paying for software if it's good value for money and
>> will save me time coding something myself. I might give the trial
>> version of that a go and see what it's like but I would prefer
>> something that will work on my server and can be run on a daily basis
>> and show the reports in a web browser.
>
> If you want to pay for custom software (ie written to your specific
> needs) drop me a line (mail at johnbokma.com ). You can specify how
> you want your reports etc.

Heh, I'm quite capable of writing log processing software myself and
have done it several times before. I was meaning if it's an off the
shelf type app then I might consider paying for it as it would mean I
save time myself and could be charging other people for it. But I would
rather spend the time myself if it was a custom app.

Thanks for the offer though ;)

--
Chris Hope - The Electric Toolbox - http://www.electrictoolbox.com/

[Chris Hope]

John Bokma wrote:

> Chris Hope wrote:
>
>>>> what sort of features would people want in such software? I would
>>>> most likely write it in Perl - I personally prefer PHP for
>>>> programming but have found Perl to be much faster at processing
>>>> logfiles than PHP.
>
> Also Perl is way better developed and has less quirks compared to PHP.

Possibly. I still prefer PHP over Perl. But then it's been a few years
since I did any serious Perl development.

>> That looks like a nice stats package. What it doesn't cover is the
>> search engine bot stuff which is what I'm after. Re-reading my post
>> now I had that in the subject line but didn't mention it in the
>> body...
>
> If you decide to do it in Perl (sound), check out CPAN.
>
> Also google for "perl log analyzer" etc. you might find a project you
> can join.
>
> I am doing my own perl hacking, but it's certainly finished to be
> usable as open source (I focus more at getting keywords used in
> queries anyway).

Thanks. I must remember not to forget CPAN - there's bound to be
something in there along the lines of what I want which will save a lot
of time.

--
Chris Hope - The Electric Toolbox - http://www.electrictoolbox.com/