27 comments have been added.
Add your comments.
1.
David Sifry said...
I think that this is because of the software being used, most probably. All of the aggregators/indexers that have been mentioned take advantage of the XML-RPC calls to www.weblogs.com and some others to notify the spiders that a blog has been updated. No XML-RPC ping? No indexing. The purportedly biggest blogging tool, blogger, does not have a list of recently updated weblogs, which means that again, no indexing. Lobby the makers of your blog software to include XML-RPC pings in their default settings, and you will see more of your neighborhood blogs getting indexed, and therefore, ranking on the indexes.
Dave
2.
Nilesh said...
Dave, we needed just that kind of insight into why indian blogs seem so hidden. You mentioned that XML-RPC pings to weblogs.com might be one of the reasons and I too agree with you. Many guys don't bother to ping updates to weblogs.com. Still, I think, ranking up blogs should be made more subjective that just checking the number of links coming to the blogs. Heck, Technorati shows me having 63 incoming links and I am surprised to find out that all of them are from my blog itself! Although I know many people are linking to me. No indication of my `real' index. (Take this just as an example. Don't read too much into it :-) As of now, I can't think of anything that can be added to the existing system to improvise on the rankings. Maybe we can give a hard thought to it.
3.
Kiruba Shankar said...
Nilesh, if you are looking for a reason to work on an index for Indian Blogs, you'll get all the encouragement. If nothing else, it would be a good opportunity for you to exercise your programming skills.
I would agree with David that Ev must have the pinging feature turned on. Wonder why he didnt already.
Thankfully, a bunch of us use w.Bloggar that does the pinging.
4.
anand said...
Nilesh, you have my "go ahead" as well :)
We indians are supposed to be among the best in IT .. unfortunately not many of us participate in such open source / interesting work .. This could be a welcome change ...
On a bigger note, This could be the basis for the indian blogger's portal ...
I am willing to help in any such effort to the best I can :)
5. Reference from
masala
XML RPC server ?
Why an XML RPC server ?
Indian Blogdex ?
Read more in
Three posts and one question. »hey let do one thing at a time.. let us develop the Indi Blog list at Rightho and then maybe mould it to incorp the Indi ecosystem! say wat
7.
Arnab Nandi said...
Anita's IndianBlogger List is already quite exhaustive, The Pepys project has an Indian category (most other blog directories do too).
The RightHo Directory seems good, but we'd need loads of features to get it going. I think Madman had once proposed we get Dave Winer to issue us india.weblogs.com, and put our directory there, and we can simply filter the weblogs.com feed by checking the link from the directory.
There's loads of people really interested in this, Vikas Kamat's written loads about it, and so have many others.
I guess it'll be best for one person to take up the responsibility to plan and factorize this, and assign modules to people - almost all of us are PHP coders!!
8.
Nilesh said...
Arnab is right! We have a lot already done. We just need to collate all that work. Anita, Ram and Vikas have already put in a lot of effort. I volunteer for doing the integration work. Anybody else?
So what do we do? Say,
- a directory
- an index
- a recently updated system
- a rating system
These are standard stuff. Some thing more. ... Can you think of anything?
9.
Nilesh said...
Do we need to have a domain? I did some research and Cameron of Blogdex sometime back posted on his blog about metadex.com being available. Wot say?
10.
Nilesh said...
Coincidentally, Vikas Kamat has blogged about such a directory 2 days ago. I read about it only via Srijith via Anand. :-) So guys, what are we waiting for? I think we should strike when the iron is hot!
11.
Arnab Nandi said...
So guys, what are we waiting for?
Waiting for these awful, stupid exams of mine to finish :)
Actually, you guys go ahead... I'll join the party in a few weeks.
I mean, we really have a thing going... MadMan and Suman can do the usability, Nilesh can make a sexy design... Satyen can devise some crafty meme-tracking algorithms... we can use the Kamat's image library as a design resource, and Kiruba can interview all of us in the end!!
...And then Nidhi can make all of us famous in the print Media... lol.
12.
(\/)ystic said...
>I read about it only via Srijith via Anand. :-)
Just to clarify, when I said in my post that I would like an XML RPC server designed just for Indian bloggers, I was not asking for a Blogdex like system, but more like a weblogs.com system. I wanted it to become a directory listing which is dynamic enough to know when some Indian blogger updates his blog.
But I sure am for a Ecosystem sytled implementation for us bloggers.
Me being a Perl coder, I can help out if Perl expertise is required, but only after Feb.
13.
Nilesh said...
Mystic, did you read my comment on your post at trinetre? That tells the difference....
14.
anand said...
I can see that I did my bit in connecting the two conversations .. happy :-)
While on the matter of choosing a name, wouldnt it be better to choose an indian name rather than something like metadex ? I am willing to pay for the domain name ...
A crawler will be better suited for our task since adding ping support to the tools is going to be a non-trivial task ...
Some sort of indian blogger's portal needs to be made due to the increasing number of indian bloggers .. It is time to get together :-)
15.
Nilesh said...
Anand said:
wouldnt it be better to choose an indian name rather than something like metadex
Something like indiadex or indiaweblogs or indiablogs or indiablogger? The point is we need identity!
A crawler will be better suited for our task since adding ping support to the tools is going to be a non-trivial task ...
I agree with that absolutely. The post says that.
It is time to get together :-)
Let's rock!
16.
anand said...
Another idea that could be pursued is an "RSS" portal ... services like http://logicerror.com/blogifyYourPage can be used to add RSS support to people whose tools dont support it .. Then all the RSS feeds could be combined and displayed like the page for the flash blog news, http://www.thedevilneversleeps.com/flog/ ...
What do you say ?
17.
(\/)ystic said...
Nilesh said: Mystic, did you read my comment on your post at trinetre? That tells the difference....
Yup.. read it and understood.
Anand said: A crawler will be better suited for our task since adding ping support to the tools is going to be a non-trivial task ...
I am not too sure about that. I do't know what the system is that you guys have in place, but a crawl based update will not scale, for that matter anything that is server initiated will not scale. It has to be client initiated. Maybe RPC or email based?
18.
anand said...
how about this ?
http://www.hitormiss.org/projects/dailycrawl/
DAILY CRAWL is a set of CGI scripts that allow you to track updates to a list of weblogs that you read though the update list at weblogs.com.
DAILY CRAWL requires you to have:
An account on a webserver that allows you to run custom CGI scripts.
Perl, including the LWP module.
The ability to add a cron job to your webserver.
19.
anand said...
Mystic : Blogdex, DayPop, PopDex are all crawlers .. heck, google is a crawler ;-)
It depends on what is your aim .. weblogs couldnt scale because they were crawling every hour or so to find fresh updates ... on the other hand, if we do it once in a day then we could be better off ...
Again, we are talking far less websites here when compared with what weblogs.com has to deal with ..
The number of things you could do with a XML RPC server idea is limited ... On the other hand, crawling will let you give a whole set of features as seen on daypop or blogdex or even google ;-)
20.
anand said...
The XML RPC server idea has been implemented by kamat ... Read more at the URL below.
http://www.kamat.com/vikas/blog.php?date=12/17/2002
21.
Amit 'Netahoy' Agarwal said...
ok why not put up forum [pref on some personal srv] and start collecting ideas and fatures list.. lest hear from each and every1 around.....
Let the RighthO project come up live and start building on that with necessary features and ideas.
may we also have -
blog owners login for admin
ecosys as sugested above
.........brainstrom!!!
22.
Amit 'Netahoy' Agarwal said...
and ya.... lets do colobrative efforts and avoid duplication..... Kamat and Ram have enuf together to start the first leg.
Blog db courtsey Anita's Indi Blog Ring and Indi Blog List and Kamat's too
Join Up Guys!
P.S. Kamat how abt MAKDEE as the crawler name ;)
23.
Arnab Nandi said...
Makdee sounds nice. You can use the Hourglass Forums for discussions. I've set up an "Indibloggers Board" for this.
It's running PHPBB, and I think it should suit the needs. Pass the URL around if you're ok with it.
24.
MadMan said...
Somebody mention my name?
(If you want me to take notice, email me at webguru at vsnl.net)
I'd be happy to help out with design, interface, and usability.
Actually, all we need is a bookmarklet that will automatically ping weblogs.com
Let me see if I can whip it up.
25.
steve said...
The Waypath Project identifies the weblogs it crawls through ping services, like weblogs.com. If there's an underrepresentation of Indian weblogs at Waypath, the ping issue you're discussing is probably why. Waypath's configuration allows us to submit lists of weblogs that can be crawled periodically, without relying on weblogs.com. If you all would like to compile and email me (all at once or from time to time) a list of weblog URLs you'd like added, I can make sure they get into the Waypath Project.
That said, Waypath will still present results with similar topics, regardless of the weblog heritage. But, at least, your weblogs will be part of the results.
I'd be interested to know if any of you are encoding any information into your sites in a way that can be used to filter results. We're looking at blogchalking, for example, as a way of filtering along regional lines.
26.
anand said...
Steve, What encoding are you talking about here ? I would like to see some examples of this ...
As for the under-representation of indian blogs, it is only a matter of time .. The numbers are certainly on our side ... just need more people to start blogging :-)
27.
Mangal Pande said...
Guys, you might be interested in visiting our blog at Industelegraph.com.
The Indus Telegraph is a unique forum for Indo-philes everywhere. We feature articles and viewpoints relating to Southern Asia and the Indian Subcontinent from many diverse sources.
Our mission is to create a self-sustaining entity for the expression of culture, opinion, news, analysis, and ideas about and from Southern Asia.
The Indus Telegraph works on the principle of collaborative media, which means members not only contribute content but they also vote on what should make it to the front page. This site is a unique community powered by you, so please contribute and take part!
Your Comments
* Please do not put off-topic comments. We reserve right to delete them at our discretion. You can post anonymously. If you are unable to see your posted comment immediately, it may have been queued for moderation. So do not submit it again. HTML formatting is allowed (only a, b, i, br, p, strong, em, ul, li & blockquote are allowed). Do not put paragraph tags. They are automatically inserted.
27 comments have been added. Add your comments.
1. David Sifry said...
I think that this is because of the software being used, most probably. All of the aggregators/indexers that have been mentioned take advantage of the XML-RPC calls to www.weblogs.com and some others to notify the spiders that a blog has been updated. No XML-RPC ping? No indexing. The purportedly biggest blogging tool, blogger, does not have a list of recently updated weblogs, which means that again, no indexing. Lobby the makers of your blog software to include XML-RPC pings in their default settings, and you will see more of your neighborhood blogs getting indexed, and therefore, ranking on the indexes.
Dave
2. Nilesh said...
Dave, we needed just that kind of insight into why indian blogs seem so hidden. You mentioned that XML-RPC pings to weblogs.com might be one of the reasons and I too agree with you. Many guys don't bother to ping updates to weblogs.com. Still, I think, ranking up blogs should be made more subjective that just checking the number of links coming to the blogs. Heck, Technorati shows me having 63 incoming links and I am surprised to find out that all of them are from my blog itself! Although I know many people are linking to me. No indication of my `real' index. (Take this just as an example. Don't read too much into it :-) As of now, I can't think of anything that can be added to the existing system to improvise on the rankings. Maybe we can give a hard thought to it.
3. Kiruba Shankar said...
Nilesh, if you are looking for a reason to work on an index for Indian Blogs, you'll get all the encouragement. If nothing else, it would be a good opportunity for you to exercise your programming skills.
I would agree with David that Ev must have the pinging feature turned on. Wonder why he didnt already.
Thankfully, a bunch of us use w.Bloggar that does the pinging.
4. anand said...
Nilesh, you have my "go ahead" as well :)
We indians are supposed to be among the best in IT .. unfortunately not many of us participate in such open source / interesting work .. This could be a welcome change ...
On a bigger note, This could be the basis for the indian blogger's portal ...
I am willing to help in any such effort to the best I can :)
5. Reference from masala
Read more in Three posts and one question. »
6. Amit 'Netahoy' Agarwal said...
hey let do one thing at a time.. let us develop the Indi Blog list at Rightho and then maybe mould it to incorp the Indi ecosystem! say wat
7. Arnab Nandi said...
Anita's IndianBlogger List is already quite exhaustive, The Pepys project has an Indian category (most other blog directories do too).
The RightHo Directory seems good, but we'd need loads of features to get it going. I think Madman had once proposed we get Dave Winer to issue us india.weblogs.com, and put our directory there, and we can simply filter the weblogs.com feed by checking the link from the directory.
There's loads of people really interested in this, Vikas Kamat's written loads about it, and so have many others.
I guess it'll be best for one person to take up the responsibility to plan and factorize this, and assign modules to people - almost all of us are PHP coders!!
8. Nilesh said...
Arnab is right! We have a lot already done. We just need to collate all that work. Anita, Ram and Vikas have already put in a lot of effort. I volunteer for doing the integration work. Anybody else?
So what do we do? Say,
These are standard stuff. Some thing more. ... Can you think of anything?
9. Nilesh said...
Do we need to have a domain? I did some research and Cameron of Blogdex sometime back posted on his blog about metadex.com being available. Wot say?
10. Nilesh said...
Coincidentally, Vikas Kamat has blogged about such a directory 2 days ago. I read about it only via Srijith via Anand. :-) So guys, what are we waiting for? I think we should strike when the iron is hot!
11. Arnab Nandi said...
So guys, what are we waiting for?
Waiting for these awful, stupid exams of mine to finish :)
Actually, you guys go ahead... I'll join the party in a few weeks.
I mean, we really have a thing going... MadMan and Suman can do the usability, Nilesh can make a sexy design... Satyen can devise some crafty meme-tracking algorithms... we can use the Kamat's image library as a design resource, and Kiruba can interview all of us in the end!!
...And then Nidhi can make all of us famous in the print Media... lol.
12. (\/)ystic said...
>I read about it only via Srijith via Anand. :-)
Just to clarify, when I said in my post that I would like an XML RPC server designed just for Indian bloggers, I was not asking for a Blogdex like system, but more like a weblogs.com system. I wanted it to become a directory listing which is dynamic enough to know when some Indian blogger updates his blog.
But I sure am for a Ecosystem sytled implementation for us bloggers.
Me being a Perl coder, I can help out if Perl expertise is required, but only after Feb.13. Nilesh said...
Mystic, did you read my comment on your post at trinetre? That tells the difference....
14. anand said...
I can see that I did my bit in connecting the two conversations .. happy :-)
While on the matter of choosing a name, wouldnt it be better to choose an indian name rather than something like metadex ? I am willing to pay for the domain name ...
A crawler will be better suited for our task since adding ping support to the tools is going to be a non-trivial task ...
Some sort of indian blogger's portal needs to be made due to the increasing number of indian bloggers .. It is time to get together :-)
15. Nilesh said...
Anand said:
wouldnt it be better to choose an indian name rather than something like metadex
Something like indiadex or indiaweblogs or indiablogs or indiablogger? The point is we need identity!
A crawler will be better suited for our task since adding ping support to the tools is going to be a non-trivial task ...
I agree with that absolutely. The post says that.
It is time to get together :-)
Let's rock!
16. anand said...
Another idea that could be pursued is an "RSS" portal ... services like http://logicerror.com/blogifyYourPage can be used to add RSS support to people whose tools dont support it .. Then all the RSS feeds could be combined and displayed like the page for the flash blog news, http://www.thedevilneversleeps.com/flog/ ...
What do you say ?
17. (\/)ystic said...
Nilesh said: Mystic, did you read my comment on your post at trinetre? That tells the difference....
Yup.. read it and understood.
Anand said: A crawler will be better suited for our task since adding ping support to the tools is going to be a non-trivial task ...
I am not too sure about that. I do't know what the system is that you guys have in place, but a crawl based update will not scale, for that matter anything that is server initiated will not scale. It has to be client initiated. Maybe RPC or email based?
18. anand said...
how about this ?
http://www.hitormiss.org/projects/dailycrawl/
DAILY CRAWL is a set of CGI scripts that allow you to track updates to a list of weblogs that you read though the update list at weblogs.com.
DAILY CRAWL requires you to have:
An account on a webserver that allows you to run custom CGI scripts.
Perl, including the LWP module.
The ability to add a cron job to your webserver.
19. anand said...
Mystic : Blogdex, DayPop, PopDex are all crawlers .. heck, google is a crawler ;-)
It depends on what is your aim .. weblogs couldnt scale because they were crawling every hour or so to find fresh updates ... on the other hand, if we do it once in a day then we could be better off ...
Again, we are talking far less websites here when compared with what weblogs.com has to deal with ..
The number of things you could do with a XML RPC server idea is limited ... On the other hand, crawling will let you give a whole set of features as seen on daypop or blogdex or even google ;-)
20. anand said...
The XML RPC server idea has been implemented by kamat ... Read more at the URL below.
http://www.kamat.com/vikas/blog.php?date=12/17/2002
21. Amit 'Netahoy' Agarwal said...
ok why not put up forum [pref on some personal srv] and start collecting ideas and fatures list.. lest hear from each and every1 around.....
Let the RighthO project come up live and start building on that with necessary features and ideas.
may we also have -
.........brainstrom!!!
22. Amit 'Netahoy' Agarwal said...
and ya.... lets do colobrative efforts and avoid duplication..... Kamat and Ram have enuf together to start the first leg.
Blog db courtsey Anita's Indi Blog Ring and Indi Blog List and Kamat's too
Join Up Guys! P.S. Kamat how abt MAKDEE as the crawler name ;)
23. Arnab Nandi said...
Makdee sounds nice. You can use the Hourglass Forums for discussions. I've set up an "Indibloggers Board" for this.
It's running PHPBB, and I think it should suit the needs. Pass the URL around if you're ok with it.
24. MadMan said...
Somebody mention my name?
(If you want me to take notice, email me at webguru at vsnl.net)
I'd be happy to help out with design, interface, and usability.
Actually, all we need is a bookmarklet that will automatically ping weblogs.com
Let me see if I can whip it up.
25. steve said...
The Waypath Project identifies the weblogs it crawls through ping services, like weblogs.com. If there's an underrepresentation of Indian weblogs at Waypath, the ping issue you're discussing is probably why. Waypath's configuration allows us to submit lists of weblogs that can be crawled periodically, without relying on weblogs.com. If you all would like to compile and email me (all at once or from time to time) a list of weblog URLs you'd like added, I can make sure they get into the Waypath Project.
That said, Waypath will still present results with similar topics, regardless of the weblog heritage. But, at least, your weblogs will be part of the results.
I'd be interested to know if any of you are encoding any information into your sites in a way that can be used to filter results. We're looking at blogchalking, for example, as a way of filtering along regional lines.
26. anand said...
Steve, What encoding are you talking about here ? I would like to see some examples of this ...
As for the under-representation of indian blogs, it is only a matter of time .. The numbers are certainly on our side ... just need more people to start blogging :-)
27. Mangal Pande said...
Guys, you might be interested in visiting our blog at Industelegraph.com.
The Indus Telegraph is a unique forum for Indo-philes everywhere. We feature articles and viewpoints relating to Southern Asia and the Indian Subcontinent from many diverse sources.
Our mission is to create a self-sustaining entity for the expression of culture, opinion, news, analysis, and ideas about and from Southern Asia.
The Indus Telegraph works on the principle of collaborative media, which means members not only contribute content but they also vote on what should make it to the front page. This site is a unique community powered by you, so please contribute and take part!
Your Comments
* Please do not put off-topic comments. We reserve right to delete them at our discretion. You can post anonymously. If you are unable to see your posted comment immediately, it may have been queued for moderation. So do not submit it again. HTML formatting is allowed (only a, b, i, br, p, strong, em, ul, li & blockquote are allowed). Do not put paragraph tags. They are automatically inserted.