How can I make JavaScript more enjoyable? What tooling do I need?
October 19, 2020 12:01 AM Subscribe
Who should I follow on GitHub that's producing some quality JavaScript applications? Doesn't really matter what they are doing, but the code that makes you wish that you wrote it. I think at one point, one of the founders of Optimizely was committing great code. I'm more partial to backend practices. Second, I'm having to architect or guide an application that will have developers of varying skillsets commit and it'll be in Javascript. What's some advice you have? Examples inside:
I'm going to ask this very broadly and I'm not attempting to over-architect this but my Javascript is rusty. Previous applications I've used for large scale architecture was a cpp/Java/Kafka one which was easy once you got through the hell of setting up a large distributed system, because you just consumed/produced messages. Another was a C# application where I had a compiled assembly and developers had access to that and an XML config file. If they wanted access to the say "BeforeRequestMade" function inside the application they'd go into the config, inside the config they'd find the BeforeHTTPRequestMade and then enter the details of their own assembly class name and method name they'd wanted called. This method signature had to match it so that when I started up the application I'd reflect on all the loaded assemblies, find it, then execute it in the write order.
Here's my somewhat contrived example:
CEO comes to me with a URL and wants me to extract words that start with "A" ... it doesn't really matter just assume there's some logic that needs to be implemented to extract the data. He hands me this:
The point being I'm now relying on data previously entered to generate and enter the URLs. I have to extract first names from the URLs, and I also have to extract last names. I have two team members that are joining to help me so I can continue working on the first names method, another member who works on finding the last names, and another who works on generating URLs from the database. Maybe even all of us writing code for tracing, etc.
How would you break that up? The changing requirements is something that'll come up and I think the example is simple but complex enough to be real. I can assume I don't want to cram "find first name" and "find last name" into one method and that I'll need some sort of way to queue up a bunch of here's a page run filters. And also an orchestrator to run URLs, some sort of caching so I'm not hitting the live site 500,000 times, etc.
How would you break that out? Is that enough info? Maybe it is just to me that Javascript's strength is that it is so malleable and non-opinionated this can go in a lot of different directions. I'd like to break it up so that it is flexible but also so that no one has to think too hard if they need to do something on a new name being submitted they don't have to manipulate a monolithic code base, and just hook into an event that's like, "Before first name submitted" check the database, do the logic and be done.
Not to add more to this, but the C# example worked great in that I effectively had a distributed "Version 1" of the main application. A team completely separate could create their own code base, get it running against Version 1 to make sure it worked, not have to fork and add code to the main application or anyone else's, send out the module as a zip file and have someone drop the zip file in, add something to a config file and it was done. It kept a single architect from lording over the project, it allowed multiple teams from different companies with varying skillsets working on the same project to get things done. I'm pretty sure a similar thing is done with dropping JAR files into Idea's line of products. I guess my point is that while a jQuery plugin author will have no control over what the main project does, this is a private project so there can definitely be some uptake involved by the main project. It isn't a hard isolation like an open source project might experience. If that helps what others might have seen on projects that went well in the JS world.
I'm going to ask this very broadly and I'm not attempting to over-architect this but my Javascript is rusty. Previous applications I've used for large scale architecture was a cpp/Java/Kafka one which was easy once you got through the hell of setting up a large distributed system, because you just consumed/produced messages. Another was a C# application where I had a compiled assembly and developers had access to that and an XML config file. If they wanted access to the say "BeforeRequestMade" function inside the application they'd go into the config, inside the config they'd find the BeforeHTTPRequestMade and then enter the details of their own assembly class name and method name they'd wanted called. This method signature had to match it so that when I started up the application I'd reflect on all the loaded assemblies, find it, then execute it in the write order.
Here's my somewhat contrived example:
CEO comes to me with a URL and wants me to extract words that start with "A" ... it doesn't really matter just assume there's some logic that needs to be implemented to extract the data. He hands me this:
+-------------------------------------------------+ | URLs | +-------------------------------------------------+ | https://www.gutenberg.org/files/1342/1342-0.txt | | https://www.gutenberg.org/files/11/11-0.txt | +-------------------------------------------------+I then write this:
import axios from 'axios';
class NameExtractor {
static getNamesAndStoreByUrl(url) {
axios.get(url).then(resp => this.StoreNames(this.extractLatestPostId(resp.data)));
};
static NamesFinder(response) {
// Name logic
return names
}
static StoreNames(names) {
// Database logic
}
}
export default NameExtractor;
NameExtractor.getNamesAndStoreByUrl('https://www.gutenberg.org/files/1342/1342-0.tx') which isn't great as I have to do that twice for each URL but pretty simple job. Plus encapsulating this.StoreNames(this.extractLatestPostId(resp.data)). I then get handed another list, and additional information that they want to match another subset of logic, say last names:
+-------------------------------------------------+ | URLs | +-------------------------------------------------+ | https://www.gutenberg.org/files/Abby/sdssksksk.txt | | https://www.gutenberg.org/files/Bobby/sdssksksk.txt | +-------------------------------------------------+Those URLs are different but appear to be have the names masked in there, as if the names from the first assignment were used to generate those. To be sure I'd probably want to give a report to the CEO with the URLs he gave me AND a report from the names in the database. So if the CEO gave me a list of URLs that didn't have "Charles" in any of them but that's in the database might as well hit the URLs with Charles, do the name extraction and give a report on that too. If the CEO missed it great, if he didn't no harm. He also wants last names too.
The point being I'm now relying on data previously entered to generate and enter the URLs. I have to extract first names from the URLs, and I also have to extract last names. I have two team members that are joining to help me so I can continue working on the first names method, another member who works on finding the last names, and another who works on generating URLs from the database. Maybe even all of us writing code for tracing, etc.
How would you break that up? The changing requirements is something that'll come up and I think the example is simple but complex enough to be real. I can assume I don't want to cram "find first name" and "find last name" into one method and that I'll need some sort of way to queue up a bunch of here's a page run filters. And also an orchestrator to run URLs, some sort of caching so I'm not hitting the live site 500,000 times, etc.
How would you break that out? Is that enough info? Maybe it is just to me that Javascript's strength is that it is so malleable and non-opinionated this can go in a lot of different directions. I'd like to break it up so that it is flexible but also so that no one has to think too hard if they need to do something on a new name being submitted they don't have to manipulate a monolithic code base, and just hook into an event that's like, "Before first name submitted" check the database, do the logic and be done.
Not to add more to this, but the C# example worked great in that I effectively had a distributed "Version 1" of the main application. A team completely separate could create their own code base, get it running against Version 1 to make sure it worked, not have to fork and add code to the main application or anyone else's, send out the module as a zip file and have someone drop the zip file in, add something to a config file and it was done. It kept a single architect from lording over the project, it allowed multiple teams from different companies with varying skillsets working on the same project to get things done. I'm pretty sure a similar thing is done with dropping JAR files into Idea's line of products. I guess my point is that while a jQuery plugin author will have no control over what the main project does, this is a private project so there can definitely be some uptake involved by the main project. It isn't a hard isolation like an open source project might experience. If that helps what others might have seen on projects that went well in the JS world.
Response by poster: Hmm well I don't think React is what I was looking for. I took a step back and just reading Node's about section gave me a lot of information:
This is in contrast to today's more common concurrency model, in which OS threads are employed
I think that's where my hangup was, not understanding callbacks and realizing that node isn't just a normal runtime but one that listens for network connections before doing anything. It helps to read the manual sometimes. Node is basically just a simple web server. I wonder how deno compares.
In any event, this makes it a bit more complex. Let's say I wanted to accept plugins that did things in a certain order? That mean unlike my C# example I couldn't guarantee order execution by just loading things in order right? I'd have to do something like from the node.js docs:
posted by geoff. at 3:13 AM on October 19, 2020
This is in contrast to today's more common concurrency model, in which OS threads are employed
I think that's where my hangup was, not understanding callbacks and realizing that node isn't just a normal runtime but one that listens for network connections before doing anything. It helps to read the manual sometimes. Node is basically just a simple web server. I wonder how deno compares.
In any event, this makes it a bit more complex. Let's say I wanted to accept plugins that did things in a certain order? That mean unlike my C# example I couldn't guarantee order execution by just loading things in order right? I'd have to do something like from the node.js docs:
fs.rename('/tmp/hello', '/tmp/world', (err) => {
if (err) throw err;
fs.stat('/tmp/world', (err, stats) => {
if (err) throw err;
console.log(`stats: ${JSON.stringify(stats)}`);
});
});
That's a bit awkward?posted by geoff. at 3:13 AM on October 19, 2020
I am not particularly familiar with Javascript, never mind node or backend functional programming in Javascript, but that's what the Try monad is for. A quick google suggests Javascript uses Either for that purpose (but what library it comes from is left as an exercise to the reader).
posted by hoyland at 4:48 AM on October 19, 2020
posted by hoyland at 4:48 AM on October 19, 2020
That's kind of a large sprawling question, but I like VS Code and TypeScript. You can do types and class interfaces as if you were in C# or Java. The async/await keywords will make everything much nicer.
And if this is a server-side app, there's a ton of middleware you can use, but maybe start by mocking out some objects and using SQLite or something, and writing some unit tests.
posted by RobotVoodooPower at 5:17 AM on October 19, 2020 [1 favorite]
And if this is a server-side app, there's a ton of middleware you can use, but maybe start by mocking out some objects and using SQLite or something, and writing some unit tests.
posted by RobotVoodooPower at 5:17 AM on October 19, 2020 [1 favorite]
Seconding TypeScript, and maybe some of these *generator functions?
posted by johngoren at 6:42 AM on October 19, 2020
posted by johngoren at 6:42 AM on October 19, 2020
My other thought in re-reading your code, with all those OOP-like static methods, is:
I too went into JS from a background in more OOP-ish languages, and found myself having better luck getting away from JS classes and more into functional styles of writing code. I know some people may find FP a bit of a fad. But last time I was in JS-land (admittedly this is a few years ago and maybe classes have improved), I found that JS's notion of classes was sort of dorky and hacky and awkward, and that anything with constructors or whatever tended to hearken back to the early bad days of 90s "it's like a script version of Java!" JS.
posted by johngoren at 7:26 AM on October 19, 2020 [1 favorite]
I too went into JS from a background in more OOP-ish languages, and found myself having better luck getting away from JS classes and more into functional styles of writing code. I know some people may find FP a bit of a fad. But last time I was in JS-land (admittedly this is a few years ago and maybe classes have improved), I found that JS's notion of classes was sort of dorky and hacky and awkward, and that anything with constructors or whatever tended to hearken back to the early bad days of 90s "it's like a script version of Java!" JS.
posted by johngoren at 7:26 AM on October 19, 2020 [1 favorite]
Thirding Typescript. The learning curve is pretty shallow, the documentation is pretty good, and I find having type info makes collaboration a lot easier. The tooling is also pretty good.
You might also want to look into rxjs for stream/message/data processing, although that's more of a commitment (steeper learning curve, lots of semantics) than Typescript and may not be appropriate for your purpose. However the composability of rxjs operators does a lot to aid reuse & refactoring and gives you a toolbox to manage queuing / throttling / retries / debounce / de-duplication etc.
posted by dmh at 10:19 AM on October 19, 2020
You might also want to look into rxjs for stream/message/data processing, although that's more of a commitment (steeper learning curve, lots of semantics) than Typescript and may not be appropriate for your purpose. However the composability of rxjs operators does a lot to aid reuse & refactoring and gives you a toolbox to manage queuing / throttling / retries / debounce / de-duplication etc.
posted by dmh at 10:19 AM on October 19, 2020
>That mean unlike my C# example I couldn't guarantee order execution by just loading things in order right?
Node owes this massively-parallel race to the V8 js engine it runs on. To win a "faster browser" war, it churns through work and gets back to polling for more data to arrive over the network.
How about:
'async' keyword for stuff you know will work concurrently that yields (keyword 'yield') at the coroutine* where it can be interrupted and resumed.
or promises for controlling the queue of things that have to happen in sequence and escalating when they fail to do what you expect.
*: it's not entirely a coroutine, like much of js it's a part-implemented version inspired by other programming ecosystems.
posted by k3ninho at 7:30 AM on October 20, 2020
Node owes this massively-parallel race to the V8 js engine it runs on. To win a "faster browser" war, it churns through work and gets back to polling for more data to arrive over the network.
How about:
'async' keyword for stuff you know will work concurrently that yields (keyword 'yield') at the coroutine* where it can be interrupted and resumed.
or promises for controlling the queue of things that have to happen in sequence and escalating when they fail to do what you expect.
*: it's not entirely a coroutine, like much of js it's a part-implemented version inspired by other programming ecosystems.
posted by k3ninho at 7:30 AM on October 20, 2020
Best answer: The idea is to run this using Node.js or something, not inside a browser, correct? Your instinct about React is absolutely correct in this case, it's definitely the wrong tool for this job (and arguably based on your requirements, might be the wrong tool even if you were running it in a browser). I'm going to Nth Typescript and VSCode; you may want to use npm (which comes with node.js) or yarn (a popular alternative package manager) if you're used to package management systems in other ecosystems and want to pull in some external libraries.
My opinion is that JS really shines when you treat it more like a functional programming language than a classical OOP language. You can do classes and such in JS, and the support is pretty good in modern JS, but it's something that got bolted onto the language later, so classes end up being syntactic sugar for the older object prototype style of writing OOP in JS. Usually this isn't a problem but the abstraction can leak. Plus after writing JS for several years I've found that I don't really miss classes for code organization much. Typically I organize by "module"; related functionality is grouped together in a module (which is either a file or a set of files that import from each other), and then export the functions that I want to represent the public API surface for that module. Using Typescript I can enforce strict typing which helps keep the API formalized.
posted by Aleyn at 10:25 AM on October 20, 2020 [1 favorite]
My opinion is that JS really shines when you treat it more like a functional programming language than a classical OOP language. You can do classes and such in JS, and the support is pretty good in modern JS, but it's something that got bolted onto the language later, so classes end up being syntactic sugar for the older object prototype style of writing OOP in JS. Usually this isn't a problem but the abstraction can leak. Plus after writing JS for several years I've found that I don't really miss classes for code organization much. Typically I organize by "module"; related functionality is grouped together in a module (which is either a file or a set of files that import from each other), and then export the functions that I want to represent the public API surface for that module. Using Typescript I can enforce strict typing which helps keep the API formalized.
posted by Aleyn at 10:25 AM on October 20, 2020 [1 favorite]
Best answer: not understanding callbacks
It took me forever to figure out callbacks.
I'm approaching your question as a devout JavaScript/React developer who tried - and failed - Java I courses three times and could never wrap my head around how complicated it is.
Build tools can do things in a certain order. Webpack can process all kinds of files and you can order the processes as you wish.
In terms of waiting for things to happen, the newer versions of JavaScript us async functions with an await keyword. I see Node as a library with a ton of functions that are already written that you can drop into your JS to perform tasks - format phone numbers, make API calls, provide build tools, etc.
In your example above you could just chain your functions with then().
Here are a few examples.
posted by bendy at 8:57 PM on October 20, 2020 [1 favorite]
It took me forever to figure out callbacks.
I'm approaching your question as a devout JavaScript/React developer who tried - and failed - Java I courses three times and could never wrap my head around how complicated it is.
Build tools can do things in a certain order. Webpack can process all kinds of files and you can order the processes as you wish.
In terms of waiting for things to happen, the newer versions of JavaScript us async functions with an await keyword. I see Node as a library with a ton of functions that are already written that you can drop into your JS to perform tasks - format phone numbers, make API calls, provide build tools, etc.
In your example above you could just chain your functions with then().
Here are a few examples.
posted by bendy at 8:57 PM on October 20, 2020 [1 favorite]
Response by poster: Hey! What helped me was learning Deno. It did nothing magical but it is so new it doesn't have a bunch of libraries which ironically helped me not get really confused by what I call hipster coder vocabulary that tends to obfuscate what someone is trying to actually say.
Typically I organize by "module"; related functionality is grouped together in a module (which is either a file or a set of files that import from each other), and then export the functions that I want to represent the public API surface for that module. Using Typescript I can enforce strict typing which helps keep the API formalized.
Exactly what I was looking for! Yes it seems that organizing code into a module of related functionality is the way to go. So to give you an example more concretely I was trying to simply get NFL data to build out an app. Due to legal restrictions getting the data is incredibly hard and the NFL doesn't even publish an API. They even forbid you from crawling their site! So usually I would say I have something called "retrievers" that extracts the data, something like transformers that are DTOs and then loaders which puts it in a database so in the end you have a nice API. All I wanted to do was have this on GitHub and not vagrantly violate NFL copyrights. Except getting historical data vs live data vs something as simple as a list of teams is really, really hard when you can't even seed the database with teams. How I obtain teams might be crawling Wikipedia, but getting live data is not conceptually the same. Or rather abstraction would be sort of forced. Modules ended up what I ended up doing.
In your example above you could just chain your functions with then().
Chaining helped and using await/async did give me the ease of synchronous syntax without getting into callback hell. Since I needed to make things happen in a certain order and I ended up using the express module paradigm.
In the end the whole project ended up being a thin TypeScript layer over a fork of Deno in Rust. All the heavy lifting I needed done I wrote in Rust for performance and exposed it to TypeScript. This was more of a pedagogical project for me to get my head around writing in TypeScript. This was probably overkill but what the final application more or less looked like:
- A bunch of E2E modules. Take a string formatted one way, manipulate the string, output that string to a predefined JSON schema.
- Since things like Wikipedia are incredibly hard to parse. Luckily I just used some tools already available in Rust. Note I didn't know Rust before starting this.
- NFL data is really fragile, multiple times while scraping sites (not NFL.com!) that licensed NFL data and had no stupid blanket restrictions of dubious legality anyway, the NFL API was down. I noticed tended to occur with Covid updates to schedule which ironically participated the need for this project as my carefully constructed spreadsheets did not cope well with games moving around all the time -- and when games did move around everyone involved in using the spreadsheet came to a non-programmer consensus of what should be done about the bets involving the games. Sometimes we'd move the game and not change bets. Sometimes it was decided to move the games and change the bets as it wasn't fair. If I were an actual sportsbook this would probably be more absolute but the easier way of deciding what was fair or not from a human perspective was actually more complex to account for in software. In any case NFL's own strict restrictions on how its data may be consumed and used introduced fragility.
- Live scores are extremely tricky. Important non-scoring event are also important. These were not in the requirements or specifically JavaScript related, but I can't hit a site multiple times a second for a score. I also can't really track this by score alone. So a big interception in a tight game would be interesting to notify everyone of. I'm currently trying to use a text classifier using Twitter's feeds to try and figure this out. There's been some attempts to do this with soccer, but everything is CPP/Python. Luckily with the micro-architecture or module based approach I can plugin this relatively easily.
In short it ended up being a huge boon on how to architect a large enterprisey application in a modern JavaScript/Typescript environment.
posted by geoff. at 2:01 PM on October 23, 2020 [1 favorite]
Typically I organize by "module"; related functionality is grouped together in a module (which is either a file or a set of files that import from each other), and then export the functions that I want to represent the public API surface for that module. Using Typescript I can enforce strict typing which helps keep the API formalized.
Exactly what I was looking for! Yes it seems that organizing code into a module of related functionality is the way to go. So to give you an example more concretely I was trying to simply get NFL data to build out an app. Due to legal restrictions getting the data is incredibly hard and the NFL doesn't even publish an API. They even forbid you from crawling their site! So usually I would say I have something called "retrievers" that extracts the data, something like transformers that are DTOs and then loaders which puts it in a database so in the end you have a nice API. All I wanted to do was have this on GitHub and not vagrantly violate NFL copyrights. Except getting historical data vs live data vs something as simple as a list of teams is really, really hard when you can't even seed the database with teams. How I obtain teams might be crawling Wikipedia, but getting live data is not conceptually the same. Or rather abstraction would be sort of forced. Modules ended up what I ended up doing.
In your example above you could just chain your functions with then().
Chaining helped and using await/async did give me the ease of synchronous syntax without getting into callback hell. Since I needed to make things happen in a certain order and I ended up using the express module paradigm.
In the end the whole project ended up being a thin TypeScript layer over a fork of Deno in Rust. All the heavy lifting I needed done I wrote in Rust for performance and exposed it to TypeScript. This was more of a pedagogical project for me to get my head around writing in TypeScript. This was probably overkill but what the final application more or less looked like:
- A bunch of E2E modules. Take a string formatted one way, manipulate the string, output that string to a predefined JSON schema.
- Since things like Wikipedia are incredibly hard to parse. Luckily I just used some tools already available in Rust. Note I didn't know Rust before starting this.
- NFL data is really fragile, multiple times while scraping sites (not NFL.com!) that licensed NFL data and had no stupid blanket restrictions of dubious legality anyway, the NFL API was down. I noticed tended to occur with Covid updates to schedule which ironically participated the need for this project as my carefully constructed spreadsheets did not cope well with games moving around all the time -- and when games did move around everyone involved in using the spreadsheet came to a non-programmer consensus of what should be done about the bets involving the games. Sometimes we'd move the game and not change bets. Sometimes it was decided to move the games and change the bets as it wasn't fair. If I were an actual sportsbook this would probably be more absolute but the easier way of deciding what was fair or not from a human perspective was actually more complex to account for in software. In any case NFL's own strict restrictions on how its data may be consumed and used introduced fragility.
- Live scores are extremely tricky. Important non-scoring event are also important. These were not in the requirements or specifically JavaScript related, but I can't hit a site multiple times a second for a score. I also can't really track this by score alone. So a big interception in a tight game would be interesting to notify everyone of. I'm currently trying to use a text classifier using Twitter's feeds to try and figure this out. There's been some attempts to do this with soccer, but everything is CPP/Python. Luckily with the micro-architecture or module based approach I can plugin this relatively easily.
In short it ended up being a huge boon on how to architect a large enterprisey application in a modern JavaScript/Typescript environment.
posted by geoff. at 2:01 PM on October 23, 2020 [1 favorite]
Response by poster: I should add I'll open source this and post it on Metatalk if I get anywhere. I'm thinking of writing a dashboard that shows the loaded modules and their status. Again, this is all overkill but if you're wanting to learn the language, the syntax how it compiles -- really not a bad way to learn. I always find learning to write software for other developers the most challenging and rewarding part. I could've probably done this with a few shell scripts but now I have a useless program anyone can uselessly contribute into.
posted by geoff. at 2:55 PM on October 23, 2020 [1 favorite]
posted by geoff. at 2:55 PM on October 23, 2020 [1 favorite]
« Older Christmas cheer in a time of COVID | Recommend me a Canadian based climate change... Newer »
This thread is closed to new comments.
Redux can give you stricter control over state - you should investigate them both.
I created the same app in react and redux. here are the differences
The newest version of React implements hooks which can be just as useful as Redux.
React state management: React Hooks vs Redux
posted by bendy at 1:04 AM on October 19, 2020