Is this HTML test too hard for applicants?

November 17, 2004 2:19 PM

HTML Gurus and Browser Mechanics: Is this test too hard? Too much to expect an applicant to complete? [more inside]
posted by weston to computers & internet (32 comments total)

The first person we've had take it complained: the exercises would take too long, 3-4 hours. We decided to compromise: for several of the exercises, applicants are welcome to submit links to sites in their portfolio where they've solved similar problems. There's still concern that it's too much.

I will admit that it could be seen as a hard hoop to jump through -- I'd estimate the whole test would in fact take me about 4 hours -- but with portfolio substitution, I think I could do it in an hour and a half. And really, that amount of time doesn't seem unreasonable to ask a potential full-time employee to invest in applying -- and heck, I've spent four hours sprucing up a resume for jobs I've been interested in. I'm less sure of asking that much from potential freelancers -- and yet, I've also been willing to spend 4-6 hours acquiring a potential client when I've been doing freelance work, as long as it was clear that they weren't asking me to do actual work for free.

So what's your verdict? Too much? Just right? Will we scare off the true professionals who don't want to waste their time on our silly test? Or am I on the right track thinking that true pros would be willing and able to complete things quickly?
posted by weston at 2:19 PM on November 17, 2004

I read the more inside after the test. I thought it was for a class and thought it sufficient, though very uneven. Up to question 7 just about any half decent designer would know. After that, it's a much steeper bit of info you're asking for, especially 9- 10.

Were I applying for a job and given this test I would more than likely just leave. Not because I can't do it but anyone who expects this of an applicant probably has unrealistic working environment and undervalues my time. Perhaps that's too harsh an assessment but that's what I'd think.

In addition, it matters not what one will do to get a freelance client. If the person wanted to do that shit they wouldn't be applying for a full time job.

And comparing time spent on a resume which, with minimal change, can be used over and over again for different jobs to a job-specific test for an application? That ain't even apples to oranges. One of 'em ain't even a fruit. I don't even know if it's food.
posted by dobbs at 2:34 PM on November 17, 2004

I think that would take even an excellent applicant a few hours to complete satisfactorily. I would look elsewhere for employment.
posted by mfbridges at 2:40 PM on November 17, 2004

Questions 1-5 are reasonable. Question 6 is either a matter of taste or of parroting what you were told in class.

Question 7, if one is allowed to "describe", could one just insert "well, this is how they do it at the Noodle Incident"?

For the rest of it, how perfect does the code need to be? Valid XHTML 1.0 Strict with absolutely no cosmetic bugs in any browser ever? The amount of time one would put in would depend on what would be required to get 10 out of 10 for that problem. You should specify what's good enough in these cases. I could probably get 90% there in 1-2 hours, but it's that last 10% that'll kill ya.

(on preview) I agree with dobbs and mfb that this is a pretty demanding test for a job applicant, especially for an HTML monkey. OTOH, there are enough con artists selling others' work as their own that it's worth knowing that applicants can actually produce working code under controlled conditions. I've heard of people getting hired for programming jobs who can't program.
posted by adamrice at 2:45 PM on November 17, 2004

Maybe the next thing I should ask:

If you find it too demanding, what would you cut to make it reasonable?
posted by weston at 2:48 PM on November 17, 2004

i don't undestand the context - is this pre-screening for permanent jobs, or for particular projects, or what?

i might do something like this if i really wanted to work for you. it's not particularly hard, but it would use up time.

since i'm not a web designer, but think i could complete the test, maybe it needs to be made harder somehow?

hmmm. also, i think it's better as a basis for face to face discussion. you could use it as part of a discussion and get a good handle on the candidate's abilities in fifteen minutes, or half an hour, i think (especially if they had also submitted a portfolio).

do you really have so many good applicants you need to screen before you see them? or are you hoping this will identify good people even though everyone looks pretty bad? or was this designed to answer some other question - if so, what?
posted by andrew cooke at 2:50 PM on November 17, 2004

Instead of the later "write code to show..." questions, why not just have a request like "Please include a portfolio of your work"?
posted by gimonca at 2:53 PM on November 17, 2004

What type of person are you testing for? Just a production person that works off established mockups and specs? Someone who makes *any* sort of design decisions?

If it were me, I'd be a bit dismayed at the emphasis on code for code's sake, rather than looking how I would work within a team environment.

My biggest problem with finding good HTML people is getting someone who knows how to create a prototype for programming based on a set of specs, some wireframes, mockups of related pages, and rudimentary knowledge of how database calls, loops, and other core web programming concepts work.

I can train someone about when to use SPAN and DIV easy. Training him how to follow specs and make a component-based design (which in itself could have variable content) with markup readable by the programmers or another designer... not so easy.

Your question #9 is an example of how you can go deeper to see what sort of production-ready markup your prospects can create. I see that mockup and immediately begin to think "category name, category slogan, long description (WYSIWYG editor for content management?), thumbnail, caption (optional?), etc."

You might want to boil it down to 1 sample "project" that follows a bit how your own development process works.
posted by Sangre Azul at 2:54 PM on November 17, 2004

adamrice: for questions 9-10, I really don't want to see perfect validated code, I want to see what kind of html geometry an applicant breaks a given page design into. Maybe I could do better at explaining that.

andrew cooke: the problem is that we've been getting freelancers who know some basic HTML syntax but really aren't expereinced and have produced some fragile code that didn't wear well when content changed. Also, when we asked them to make changes, they said they didn't know how.
posted by weston at 2:55 PM on November 17, 2004

Is this test taken on-site (at the interview location), or is it done beforehand? If the test is done prior to an interview, I'm disturbed by question 10: 1. Tables can be easily created in a WYSIWYG editor - how is this an accurate assessment of their skills? It looks as if you're wanting someone skilled in CSS. Why no CSS positioning? 2. As a user of Macromedia Fireworks, I'm unable to properly open the .eps file used for testing.

Additionally, the misspelling of the domain name in the graphic says "unprofessional" to me in a way that would not attract professionals.

I agree that some sort of test is needed to screen out the less-talented, but this might be a tad on the extreme side. For my interview at my current employment, I showed examples of sites and had to take an oral CSS test.
posted by LeiaS at 2:58 PM on November 17, 2004

As a freelancer, I wouldn't apply for any job that asked me to build specific work samples -- it's reasonable to ask, say, "please show an example from your portfolio that demonstrates liquid layout", but asking them to build a specific layout for you isn't appropriate.

The first six questions are perfectly reasonable.

The remainder, I might see as the sort of thing you'd ask for a verbal description of "how would you accomplish this?" during an interview, but not as prescreening exercises -- anybody who has the skills you're looking for probably isn't going to be willing to do work for free (which is what asking for these work samples is asking for.)
posted by ook at 3:04 PM on November 17, 2004

Yes, I'd def. want something verbal beyond those initial questions - from a position of either side of the hiring table.

When you ask for portfolio submissions, do you request them to be in any sort of case study format? I too often just get collections of links/screengrabs. No real context at all.
posted by Sangre Azul at 3:08 PM on November 17, 2004

I'd only do it if you paid me.
posted by spilon at 3:15 PM on November 17, 2004

What if you only asked questions 1-6, and then later ask one of the other questions when you interviewed an applicant on-site...?
posted by rajbot at 3:43 PM on November 17, 2004

I wouldn't do it. I won't spend hours working for free to just get an interview. The interview process is also an opportunity for them to see if they want to work for you. If I had to do several hours of work just to get in the door, forget it.

Why don't you do a standard interview, identify your top two or three candidates and then ask them to do a single small page that tests the skills you require? It might still turn off a few people, but at least you won't scare off everyone at the get go.
posted by letitrain at 3:55 PM on November 17, 2004

I think it's a great test, but that's because I would not want to hire anyone that could not pass it easily.
posted by ericost at 4:52 PM on November 17, 2004

It's not too hard, but it is too time consuming and therefore creates a poor impression of the potential employer. An interview is as much about the applicant interviewing the employer, or at least should be.

Were I applying for a job and given this test I would more than likely just leave. Not because I can't do it but anyone who expects this of an applicant probably has unrealistic working environment and undervalues my time.

Bingo.

If you require applicants of this caliber, then you should be prepared to do some of the legwork, too. See how many of these questions can be answered in advance, by studying the submitted portfolio, then only ask the applicant about knowledge or skills they haven't already demonstrated.
posted by normy at 5:02 PM on November 17, 2004

Two things:

(1) I've rewritten it, based on feedback given here. I'm curious to know if this looks better.

(2) I think I should make it clear: the initial purpose of this test was designed to screen freelancers we will likely never meet. They could live in West Virginia, Arkansas, or Poland. It was later picked up by others here as having potential for in-house employment screening. But its main purpose is to screen potential freelancers, and I didn't make that clear enough in my question I think.
posted by weston at 5:56 PM on November 17, 2004

I can't speak to the test itself, since I'm not in that business. However, I would be completely willing to take a four-hour test for a job if and only if a test of this nature was given under the conditions of "We like you but we're not totally sure about your technical competence. If you perform satisfactorily, we will extend to you the following offer..."

As a newly-minted college grad, I once had to take a whole-day aptitude test for a potential employer. They told me that they had never seen a higher score, and offered me $24,000 per year.

I was pissed.
posted by trharlan at 6:00 PM on November 17, 2004

The way you'd approach Qs 7, 8 and 9 could be described in a few minutes, but would take a lot of time to implement.

The questions are also very specific. The last time I had a job interview with such specific examples, it was rigged so that an internal person would get the job. I would assume the same in this case.
posted by krisjohn at 6:13 PM on November 17, 2004

"...anyone who expects this of an applicant probably has unrealistic working environment and undervalues my time."

We have established, right here, why I would walk out of the room without a further word if handed this.
posted by majick at 7:12 PM on November 17, 2004

ah, ok, so this is all you get to go on. no interview.
i have never been involved in hiring someone like that (sounds terrifying!) - sorry (in my uninformed opinion, a test like this sounds pretty reasonable for the circumstances and the improved version is better).
incidentally, i do play bass (marginally), but is that really meant to be there?

why I would walk out of the room
what room? read this - it's for people applying remotely (if i understand correctly).
posted by andrew cooke at 7:14 PM on November 17, 2004

Oh, whoah, whoah, whoah. I thought this was for permanent hires. If you're using this to screen freelancers, and it's their first and only contact with you, then no wonder you're winding up with underqualified candidates.

Seriously, this'd be fine as part of a second round interview for a permanent gig, or maaaaaybe for a long-term contracting job -- but I can't imagine ever being desperate enough for work that I'd be willing to spend several hours answering quiz questions just for the privelege of having someone look at my portfolio. You're limiting yourself to students and dilettantes by doing this.

In other words, what trharlan, normy, letitrain, sangre azul, majick, and dobbs said.

It's not like there's a shortage of talented web monkeys out there, and it's usually really easy to tell whether someone knows what they're doing by looking at their portfolio. If you're worried they're faking it or borrowing other people's code, a five-minute phone interview should be enough to clear it up.
posted by ook at 7:23 PM on November 17, 2004

If you're using this to screen freelancers, and it's their first and only contact with you, then no wonder you're winding up with underqualified candidates.

Actually, this test has only been given to one person outside of the company. All the rest of our freelancers up to this point have come in screened only on portfolio or word-of-mouth recommendation at this point. The fact that we're having trouble with some of them is why the quiz is being developed.

In the revised version, I've changed the emphasis back to portfolio, but tried to highlight a preferrence for seeing sites that demonstrate solutions to specific problems. Does this seem like a better approach?
posted by weston at 7:41 PM on November 17, 2004

The test wouldn't take me that long to complete, and I can sympathize with your situation having read a lot of resumes and interviewed a number of HTMLers in the past year (there may not be an actual shortage of good coders, but they sure are hard to find).

Having said that, I think the test is a bit of a waste of time as you can answer these questions for yourself by looking through someone's portfolio.

And now having said that, if you're digging through a hundred resumes, something like this is very helpful. A couple quick thoughts:

Question 5: I would additionally ask a similar question, along the lines of: "Please describe two or more common CSS rendering bugs, the situations under which they appear and how to overcome them."

Question 6: By "CSS Positioning" do you mean "CSS-Based Techniques" or "Absolute Positioning"? Your phrase (in my not-humble experience) is ambiguous.

Question 7: Instead of phrasing this as a yes or no question, I would ask them to do it.
posted by o2b at 9:10 PM on November 17, 2004

Anyone who can do this test easily is way over qualified.

Anyone who would want to work for you given this test lacks a spine. You aren't going to find good html folk with this test, but rather people who will put up with your BS.

weston - I interview and hire html folk. This test is useless. Html is only slightly more complicated than programming a VCR. And most good html monkeys use copy and paste for 90% of their work. Someone who writes from scratch is not productive.

The trick is hiring people who you want to work with. People who churn out code quickly. People who will do things right and on time rather than nit-pik.

View source will serve you better than this test. And it won't make applicants dispise you.
posted by y6y6y6 at 9:49 PM on November 17, 2004

I thought the test was very, very easy. That said, I also thought the test screamed WE LIKE CSS! WE JUST LOVE IT! DID WE MENTION THAT WE ONLY CODE IN CSS? YOU KNOW WHAT'S A FUNNY NAME FOR A CAT? CSS. THAT WOULD BE FUNNY! CSS. CSS. CSS.

You could save yourself a heap of time (and heaps of your interviewee's time) by just asking, "Have you ever coded something in a strict doctype? Can I see it?" Nice and easy.

Also, and I guess this is just interjecting personal opinion, but CSS-only places irritate the hell out of me. The world still has many non-standard browsers in it. Hell, there are idiosyncracies in the way CSS is handled by different browsers, requiring ugly, ugly hacks to look the same. But you know what ALL browsers handle well? Tables. Yes, old-fashioned as they are, every browser handles tables wonderfully. Why bloat up your code writing work-arounds when you just want to get the job done and have it work? Are you trying to save the world, one website at a time?
posted by Civil_Disobedient at 11:07 PM on November 17, 2004

C_D: I have just not-literally spewed my drink on the screen. :)

The CSS is religion bugs me too... but I do want people who know it well, both its strengths and weaknesses... and then who, except when asked very nicely by someone who knows what they're talking about, use tables for layout.

Also, I want a pony.

Even a chocolate one.

Man, I had no idea how fraught with trouble this whole test thing was going to be.
posted by weston at 12:30 AM on November 18, 2004

Well, this is turning into an odd little pile-on. (C_D, the questions do make a point of asking for both CSS and non-CSS approaches to some problems, and asks about when not to use it... And y6y6y6, your 3rd para is just - wow - so many kinds of wrong.) Anyway.

While I still find the whole quiz prescreening idea rather offputting, the revised version is much more reasonable. I think any of what's left would be fine questions in an interview setting; finetuning the text isn't so much the issue -- it's just the idea of doing it up front that feels wrong.
posted by ook at 12:36 AM on November 18, 2004

Maybe the thing to do, then, is break it out. The portfolio invitation comes first. For the portfolios we like, we follow up with the questions, as a matter of course to make sure people didn't just stuff their portfolio with stuff they found out there on the web somewhere. This similates the process of giving the questions in an interview as closely as we can with remote applicants...
posted by weston at 2:02 AM on November 18, 2004

I don't find the screening process as off-putting as I find it irrelevant. Once anyone gets away from the Dreamweaver/Frontpage milieu and into Notepad/UltraEdit (/whatever) arena, you know that there are about a hundred ways to skin a webpage. What's necessarily "correct" depends on a lot of criteria -- do you want it fast loading, cross-browser happy, wowee-zowee, cool and slick, hard-sized, fluid...?

And for most of the scripting or coding I've done in my life, I almost always keep a reference guide nearby. For PHP, for Javascript, for Java, for C++... there are just so many functions and parameters to memorize. The point isn't whether you've memorized them, but whether you know to look for them in the first place. A test doesn't really prove a whole lot. For example, questions 2 (b) asks for a 5 px. border. Now, what if I remember that there's a CSS tag for margin, but forget if it's "text-margin" or "spacing" just plain "margin"? I know there's a tag, I know what it does, and I can get the information in about 2 seconds. Does that count?

Weston's suggestion about a portfolio revue is far more useful. Go through a couple of their previous sites, ask them how they went about working with the client, what the requirements of it were, if there were any challenges along the way, if they would have done anything differently now... these are the kinds of open-ended questions will reveal a lot more than, "Well, you answered C and it was B. Sorry."
posted by Civil_Disobedient at 2:48 AM on November 18, 2004

The revised version is much more reasonable. Well done.
posted by mfbridges at 8:43 AM on November 18, 2004