Modern Software Development Best Practices
July 20, 2016 10:53 AM Subscribe
Can you recommend practical, concrete resources discussing modern software development best practices, particularly for web development?
A substantial minority of my job involves software development. So far my CS education (BS & MS) and the relatively simple nature of most of our projects has enabled me to do what I would consider an adequate, workmanlike job, but I think I can do better.
Many of the projects I work on are some mixture of small, ad hoc, and internal, but a few are larger, external-facing, and are expected to last for some years. Those are what I'm primarily concerned about here. Some details that may be helpful:
Languages: PHP/Laravel for web development and Python for everything else, mostly text analysis and machine learning. Development is done on OS X using MAMP, PHPStorm, & PyCharm. I am open to other languages and to using Python for web development if there are compelling arguments for it.
Deployment: We deploy to CentOS virtual machines in a very plain way (no Docker/Vagrant/Heroku/virtualenv/etc). More than one project is hosted on each VM, which seems unnecessarily risky without containerization. We have separate testing and production VMs at least.
Testing: Completely ad hoc. I really want to start using unit, integration, and functional testing in my web applications but feel a bit adrift getting started.
Concurrency: For web applications in which users modify data on the server, my concurrency model generally consists of "play the odds and hope users don't step on each other's toes." This is obviously unworkable.
Databases: We use MySQL. None of our databases are especially complex. We do use limited access, per-application accounts. Are there compelling reasons to switch away from MySQL if it's working for us?
Fine-grained changelogs and data versioning: For many applications it would be very useful to be able to do things like "show and optionally rollback all changes made by user X to this object between dates Y and Z." I suspect there's no magic way to do that, but are there good general strategies?
Version Control: Almost all of our projects are single-developer, but that may change. I use git, but since I'm the only developer on my projects, my usage is pretty simplistic, and my mental model for it is mostly limited to a one-dimensional timeline.
Security: To give you a sense of where we are: one of my predecessors didn't use https and stored passwords in plaintext. Fixing that was a big step up, but we could do a lot more. I'm aware of OWASP, but specific practical recommendations are appreciated.
A substantial minority of my job involves software development. So far my CS education (BS & MS) and the relatively simple nature of most of our projects has enabled me to do what I would consider an adequate, workmanlike job, but I think I can do better.
Many of the projects I work on are some mixture of small, ad hoc, and internal, but a few are larger, external-facing, and are expected to last for some years. Those are what I'm primarily concerned about here. Some details that may be helpful:
Languages: PHP/Laravel for web development and Python for everything else, mostly text analysis and machine learning. Development is done on OS X using MAMP, PHPStorm, & PyCharm. I am open to other languages and to using Python for web development if there are compelling arguments for it.
Deployment: We deploy to CentOS virtual machines in a very plain way (no Docker/Vagrant/Heroku/virtualenv/etc). More than one project is hosted on each VM, which seems unnecessarily risky without containerization. We have separate testing and production VMs at least.
Testing: Completely ad hoc. I really want to start using unit, integration, and functional testing in my web applications but feel a bit adrift getting started.
Concurrency: For web applications in which users modify data on the server, my concurrency model generally consists of "play the odds and hope users don't step on each other's toes." This is obviously unworkable.
Databases: We use MySQL. None of our databases are especially complex. We do use limited access, per-application accounts. Are there compelling reasons to switch away from MySQL if it's working for us?
Fine-grained changelogs and data versioning: For many applications it would be very useful to be able to do things like "show and optionally rollback all changes made by user X to this object between dates Y and Z." I suspect there's no magic way to do that, but are there good general strategies?
Version Control: Almost all of our projects are single-developer, but that may change. I use git, but since I'm the only developer on my projects, my usage is pretty simplistic, and my mental model for it is mostly limited to a one-dimensional timeline.
Security: To give you a sense of where we are: one of my predecessors didn't use https and stored passwords in plaintext. Fixing that was a big step up, but we could do a lot more. I'm aware of OWASP, but specific practical recommendations are appreciated.
The biggest thing that it sounds like you can benefit from right now is moving towards better test coverage. Having better automated testing will bail you out and cover your ass when you have more developers working on your projects. Tests help them contribute to the code and catch earlier where their change breaks something. Some of the other solutions for whether you need to revisit your concurrency model or whether you need to change your git workflow will depend on how you grow.
Things to consider for best practices w/ automated testing (note that a lot of these pieces are written for organizations with big teams and complex projects so it may feel a bit over-engineered, but a lot of the basic principles still hold)
The Testing Pyramid - an argument for keeping the bulk of your test coverage at a large number of unit tests that are the foundation for integration/API tests that are then supplemented by automated UI tests, that are then backed up with manual testing. also useful reference for avoiding inverting your testing pyramid.
Depth of Test - an argument for focusing your unit tests on testing the smallest reasonable collection of components.
Clean Code - is also just a good book for development principles in general.
Though, one argument for moving your web development from PHP to Python is that you can adopt development frameworks, like Django, that incorporate OWASP Top 10 security features just as a matter of course.
posted by bl1nk at 11:19 AM on July 20, 2016 [7 favorites]
Things to consider for best practices w/ automated testing (note that a lot of these pieces are written for organizations with big teams and complex projects so it may feel a bit over-engineered, but a lot of the basic principles still hold)
The Testing Pyramid - an argument for keeping the bulk of your test coverage at a large number of unit tests that are the foundation for integration/API tests that are then supplemented by automated UI tests, that are then backed up with manual testing. also useful reference for avoiding inverting your testing pyramid.
Depth of Test - an argument for focusing your unit tests on testing the smallest reasonable collection of components.
Clean Code - is also just a good book for development principles in general.
Though, one argument for moving your web development from PHP to Python is that you can adopt development frameworks, like Django, that incorporate OWASP Top 10 security features just as a matter of course.
posted by bl1nk at 11:19 AM on July 20, 2016 [7 favorites]
Best answer: I don't think there isn't an end-all-be-all set of best practices in this field. Partly because the landscape is changing so fast. Partly because if there are BMPs, they're for big, big companies with DevOps teams and the whole nine yards.
For tinkering around with unit testing, you can start a little sample (non-work-related) Laravel project on Github and set up Travis CI to run the tests and coveralls to run coverage testing. That kind of got to understand unit testing, and that work-flow was more exciting for me because I can see instant feedback. I haven't used PHP in a decade, but I'm sure there is a workflow that can work for that.
And then if you like it, and you get buy-in from your company, there are many CI tools which you can research, for instance drone.io, or jenkins, which handle more of a build->test->deploy to Docker setup.
For how to do testing, I liked the Google tech talks. They have a few good talks on unit testing by Miško Hevery on unit testing and dependency injection. And of course read the Laravel manual.
Lastly - I'm just reading the tea leaves, so forgive me if I'm wrong - It sounds like you're working more or less in isolation on this stuff. You might connect with a local professionals group. For instance, there's a very interesting Javascript group locally and nowadays that covers a very diverse group of people. Software development is a job best not to do in isolation.
posted by Llamadogdad at 8:17 AM on July 21, 2016 [1 favorite]
For tinkering around with unit testing, you can start a little sample (non-work-related) Laravel project on Github and set up Travis CI to run the tests and coveralls to run coverage testing. That kind of got to understand unit testing, and that work-flow was more exciting for me because I can see instant feedback. I haven't used PHP in a decade, but I'm sure there is a workflow that can work for that.
And then if you like it, and you get buy-in from your company, there are many CI tools which you can research, for instance drone.io, or jenkins, which handle more of a build->test->deploy to Docker setup.
For how to do testing, I liked the Google tech talks. They have a few good talks on unit testing by Miško Hevery on unit testing and dependency injection. And of course read the Laravel manual.
Lastly - I'm just reading the tea leaves, so forgive me if I'm wrong - It sounds like you're working more or less in isolation on this stuff. You might connect with a local professionals group. For instance, there's a very interesting Javascript group locally and nowadays that covers a very diverse group of people. Software development is a job best not to do in isolation.
posted by Llamadogdad at 8:17 AM on July 21, 2016 [1 favorite]
This thread is closed to new comments.
Proper use of transactions / row locking will get you a lot of the way there for the critical stuff. Other than that,
basic optimistic concurrency control is not hard to implement.
Databases: We use MySQL. None of our databases are especially complex. We do use limited access, per-application accounts. Are there compelling reasons to switch away from MySQL if it's working for us?
The one problem I've had is enterprise people _still_ turning their nose up at the fact that their expensive backup gizmo / bozo administrator /etc can't play well with it out of the box. Other than that, be aware of the MySQL/MariaDB dichotomy - they're supposed to remain drop-in compatible in the long term, but you might want to pick one and stick with it.
posted by Dr Dracator at 11:14 AM on July 20, 2016 [2 favorites]