it's been crawled. I used google webmaster tools and everything is fine now.
Thanks, Vivek On Sun, Mar 1, 2009 at 8:55 AM, ropers <[email protected]> wrote: > 2009/2/18 Vivek Ayer <[email protected]>: >> Hi guys, >> >> Just wondering why google hasn't indexed my website yet. I have a >> mediawiki running on OpenBSD 4.3 behind a router that's running >> OpenBSD 4.3 which has the public IP. Do I need to create a robots.txt >> file in /var/www and /var/www/mediawiki to let robots know it's ok to >> crawl this space. I thought robots.txt is just there to >> restrict/limiting crawling. >> >> Could PF in any way prevent crawling? I'm running two CARP'd web >> servers behind two CARP'd routers just so that you know. I also have >> Redirect 301 in the .htaccess of my apache root to go straight to the >> mediawiki folder. I'm not sure whether robots pay attention to the >> Redirect 301. >> >> Thanks, >> Vivek > > This may be too obvious, and I don't know the details, but: > > Well, what is your web site, and is anyone else linking to it? If > googlebot doesn't stumble upon a link to your web site, while crawling > pages it already knows about, it may never find your site. > > regards, > --ropers
