Ask MetaFilter questions tagged with webprogramming

Can I build a Craigslist Crawler?

wetpaint — Thu, 27 Aug 2009 15:01:00 -0800

Can I safely and legally build a Craigslist crawler? If so, help me figure out how! See inside for details... I want to have a crawler built to scan Craigslist in all U.S. cities. I'm launching a free online marketplace for the re-sale of niche products, and want to be able to contact Craigslist sellers to inform them about posting ads on my site. Is it legal? And if so, does anybody know what Craigslist has put in place to block this type of thing? Does CL have a daily e-mail limit? Do they change their parameters frequently so that crawlers have to be modified?

Hopefully, this doesn't raise anybody's evil-spammer-red-flags. I hope to never send an e-mail to anyone who wouldn't be interested! I'm just trying to reach a very targeted audience of sellers, and offering them a (free) alternative to CL. If there's something un-kosher about that, you can also let me know why. Nicely please :)

Thanks MeFites!

Web development big picture for a non-web programmer?

silentbicycle — Mon, 08 Jun 2009 05:48:00 -0800

Can you give me any pointers on learning web development? Unlike other similar questions in the archives (1, 2), I have a lot of programming experience. I haven't really followed web programming practices closely, however, because without keeping closely involved, it has been hard to sort out useful stuff from buzzwords and fads. (Non-web programming has enough of those!) I strongly prefer having a big-picture understanding of systems when working with them, so the various "here's stuff to copy-and-paste to make this happen" sites don't do me much good. I'm not concerned with that level, rather, I want to pick up the underlying skill set, so I can teach myself the surface details as necessary. (FWIW, I recently learned standard practices for HTML, RSS, and have gotten a sense of how to (and not to) use CSS. The stuff about semantic markup makes a lot of sense to me. I suppose Javascript and jQuery come next?)

A good example of a page I find opaque is the wikipedia page on REST. It sounds like it's about managing state in a stateless protocol, and about abstracting away storage so that proxies, replacement servers, etc. can be used transparently, but the terminology is a bit alien, and I wonder if I'm just missing something.

For programming, I'm quite comfortable with Python, Lua, C, OCaml, and Scheme. Of those, Lua is especially noteworthy, both because it would be my first choice (Python second) and because it's semantically quite close to Javascript. I'm also familiar with RDBMSs (SQLite and postgres) and the relational model, and have about a decade's experience with Linux, BSD, and the associated toolchain.

For other languages people tend to suggest: I don't know PHP, and everything I've read about its design makes me wary. I have tried Ruby a bit, but don't care for it, and Lua + Python already cover that niche for me.

I'm not clear if learning a web framework would be a good idea for me or not. It would probably help with picking up the vocabulary I'm missing, but at the same time, the "handle the details for me" aspect would interfere with learning what's actually happening. I'm not worried about ORMs, etc., as I'm fine with setting up abstractions for the database myself. I'm also not concerned with scaling - I know a lot about algorithms, caching, etc. already, but I also know about premature optimization, so I won't worry about that unless something I write needs to scale. Abstracting away browser incompatibilities would be quite welcome, though. (Frameworks also seem to be where most of the buzz is -- advice like "check out _, it's teh new hotness" really doesn't tell me anything...)

What should I learn in order to do web development?

giggleknickers — Wed, 15 Oct 2008 03:09:46 -0800

What should I learn in order to do web development? I would like to create some feature-rich websites with some Web 2.0 aspects to them. I'm trying to figure out what I need to learn.

At present, I know XHTML, CSS, and javascript. There is so much else out there - php, perl, xml, ajax, ruby, sql, c#, java applets, and on and on and on.

This has been asked before, but not recently, and I know how fast the web changes.

I want some initials after my name, please!

fvox13 — Mon, 08 Sep 2008 13:38:18 -0800

What are the must-have certifications for a web developer working in higher education? I just started a new job as a web developer at a liberal arts college in the upstate new york area, and my boss has indicated that there will be both funding and administrative support for me to obtain certifications in my field (and, of course, the extra knowledge I will gain from preparing will be a great asset to me and the college!)

We have a mixed environment around here, with servers running Windows 2003, AIX, FreeBSD, Debian Linux, and OSX. We're probably going to be having fewer and fewer Windows 2003 servers as time goes by, and as we move to a virtualization setup, we'll have a lot more. We're writing in HTML and CSS, of course, but there's a fair amount of ColdFusion, PHP, Perl, and Python going on as well, and since we're using Django for some things, we're going to be doing a lot more with Python as time goes by.

So far, I'm looking at:

CISSP

Certified Ethical Hacker

Project Management Professional

W3C HTML

CompTIA Linux+ and Security+

Any bad experiences with any of these? Any good experiences?
Anything I'm missing from my list?

Need a site like AskMetaFilter. Any Suggestions?

ISeemToBeAVerb — Tue, 15 Apr 2008 23:33:59 -0800

Does anyone know of a template that is similar to AskMetaFilter? I'm trying to set up a communications site for my company, and I think that this format would suit it perfectly. I'm not much of a programmer, but I really like the social format of this site and think that my company could benefit from a similar structure for our interior communications. Any suggestions?

Adding tagging to my php CMS

yeahyeahyeahwhoo — Wed, 22 Mar 2006 13:51:49 -0800

Whats the best way to setup and implement a tagging/folksonomy system into a php content management system from scratch? I use a custom php based content management system at my work and i'm wondering what what the best way to set up a tagging/folksonomy type system like the ones you see here and delisous, etc. i'm building it into my system by hand, but i'd really like some advice on where to start, such as table structure, useful functions, etc.

I appreciate any help, this has been a difficult thing to google for.

How can other people map their domain names to my subdomains?

subverted — Sat, 20 Nov 2004 13:19:33 -0800

How do you do that thing whereby a person can go to your website and map their domain to a subdirectory of your own website? Do you need to have root on your server to do this or are there easier (read: cheaper) ways to do it?

How do I provide separate content for non-Flash web browsers?

signal — Thu, 22 Jan 2004 14:19:24 -0800

Web Dev: Is there any way to define alternate content (i.a.: an html menu) for non-flash enabled browsers?