One general way I like to think about this is that most software you use has passed through some filter – it needed to be complete enough for people to use it, people needed to find it somehow (eg through marketing), etc. If you have some fixed amount of resources to spend on making that software, there is a point where investing more of them in reducing bugs harms one’s chances of passing the filter more than it helps. In particularly competitive markets you are likely to find that the most popular software is relatively buggy (because it won by spending more on marketing or features) and you are often more likely to be using that software (for eg interoperability reasons) too.
Conversely, the occasional success Open Source tooling has is in large part due it not competing, therefore not being forced by competitive pressure to spend ~all resources on marketing, and ~nil on development. I'm not sure where computing would be today if all software was marketing-driven, but I guess nowhere near as far as it is now.
> How do I know whether my tests are passing because they're properly testing correct code or because they're failing to test incorrect code?
One mechanism to verify that is by running a mutation testing [0] tool. They are available for many languages; mutmut [1] is a great example for Python.
If there's one thing that engineer engineers have considered standard practice for ever and ever, but software engineers seem to still not entirely grok, it's destructive testing.
I see this a lot with performance measurement, for example. A team will run small-scale benchmarks, and then try to estimate how a system will scale by linearly extrapolating those results. I don't think I've ever seen it work out well in practice. Nothing scales linearly forever, and there's no reliable way to know when and how it will break down unless you actually push it to the point of breaking down.
> It's not enough for a program to work, it has to work for the right reasons. Code working for the wrong reasons is code that's going to break when you least expect it.
This reminds me of the recent discussion of gettiers[1]. That article focused on Gettier bugs, but this passage discusses what you might call Gettier features.
Something that's gotten me before is Python's willingness to interpret a comma as a tuple. So instead of:
my_event.set()
I wrote:
my_event,set()
Which was syntactically correct, equivalent to:
_ = (my_event, set())
The auto formatter does insert a space though, which helps. Maybe it could be made to transform it as I did above, that would make it screamingly obvious.
You know what, I do use a small font size in my editor. I like to see a lot of code at once. And if memory serves I spotted this in the browser, where I do the opposite.
I'll have to look into hyper legible monospace fonts. Or maybe I'll just use Atkinson and deal with the variable spacing.
I’m increasingly of the opinion that TDD is only as good as your system is testable.
This means that the time of writing your first test is too late. It’s part of the core business logic architecture – the whiteboard stage.
If you can make it testable, TDD isn’t just good practice – it’s what you want to do because it’s so natural. Similar to how unit tests are already natural when you write hermetic code (like say a string formatter).
If, OTOH, your business logic is inseparable from prod databases, files, networking, current time & time zone, etc, then TDD and tests in general are both cumbersome to write and simultaneously delivers much less value (as in finding errors) per test-case. Controversially, I think that for a spaghetti code application tests are quite useless and are largely ritualistic.
The only way I know how to design such testable systems (or subsystems) is through the “functional core - imperative shell” pattern. Not necessarily religious adherence to “no side effects”, but isolation is a must.
> If, OTOH, your business logic is inseparable from prod databases, files, networking, current time & time zone, etc, then TDD and tests in general are both cumbersome to write and simultaneously delivers much less value (as in finding errors) per test-case. Controversially, I think that for a spaghetti code application tests are quite useless and are largely ritualistic.
I don't disagree with this and I have found it to be quite true -- though IMO it still has to be said that you can mock / isolate a lot of stuff, system time included. I am guessing you already accounted for that when you said that tests can become cumbersome to write and I agree. But we should still try because there are projects where you can't ever get a truly isolated system to test f.ex. I recently finished a contract where I had to write an server for dispatching SMS jobs to the right per-tenant & per-data-center instances of the actual connected-to-the-telco-network SMS servers; the dev environment was practically useless because the servers there did not emit half the events my application needed to function properly so I had to record the responses from the production servers and use them as mocks in my dev env tests.
Did the test succeed? Sure they did but ultimately gave me almost no confidence. :/
But yeah, anyway, I agree with your premise, I just think that we should still go the extra mile to reduce entropy and chaos as much as we can. Because nobody likes being woken up to fight a fire in production.
There are few things that terrify me more nowadays at this point in my career than spending a lot of time writing something and setting it up, only to turn it on for the first time and it works without any issues.
Oh god this is such a nightmare. It takes much longer to build something that works not the first try, because then I have to force-simulate a mistake to make sure things were actually correct in the first place.
Test Driven Development had a fix for this, which I used to do back in day when I was evangelical about the one true way the write software. You wrote a test that failed, and added or wrote code only to make that test pass. Never add any code except to make a failing test pass.
It didn't guarantee 100% correct software, of course, but it prevented you from gaslighting yourself for being too awesome.
Tests are like the burning sun in your eyes after you wake up for a night of drinking.
I prefer separating writing some code down, making it functionally work on screen and writing tests. I usually cover cases in step 2 but when you add sometime new later it is nice to have step 3.
One general way I like to think about this is that most software you use has passed through some filter – it needed to be complete enough for people to use it, people needed to find it somehow (eg through marketing), etc. If you have some fixed amount of resources to spend on making that software, there is a point where investing more of them in reducing bugs harms one’s chances of passing the filter more than it helps. In particularly competitive markets you are likely to find that the most popular software is relatively buggy (because it won by spending more on marketing or features) and you are often more likely to be using that software (for eg interoperability reasons) too.
Conversely, the occasional success Open Source tooling has is in large part due it not competing, therefore not being forced by competitive pressure to spend ~all resources on marketing, and ~nil on development. I'm not sure where computing would be today if all software was marketing-driven, but I guess nowhere near as far as it is now.
> How do I know whether my tests are passing because they're properly testing correct code or because they're failing to test incorrect code?
One mechanism to verify that is by running a mutation testing [0] tool. They are available for many languages; mutmut [1] is a great example for Python.
[0] https://en.wikipedia.org/wiki/Mutation_testing
[1] https://mutmut.readthedocs.io
That’s basically the approach mentioned in the article’s paragraph starting with “A broader technique I follow is make it work, make it break.”
If there's one thing that engineer engineers have considered standard practice for ever and ever, but software engineers seem to still not entirely grok, it's destructive testing.
I see this a lot with performance measurement, for example. A team will run small-scale benchmarks, and then try to estimate how a system will scale by linearly extrapolating those results. I don't think I've ever seen it work out well in practice. Nothing scales linearly forever, and there's no reliable way to know when and how it will break down unless you actually push it to the point of breaking down.
In my regressions, i make sure i include an "always fail" test, to make sure the test infrastructure is capable of correctly flagging it.
This opens up a philosophical can of worms. Does the test pass when it fails? Is it marked green or red?
could you give a concrete example of what you mean by this?
[delayed]
> It's not enough for a program to work, it has to work for the right reasons. Code working for the wrong reasons is code that's going to break when you least expect it.
This reminds me of the recent discussion of gettiers[1]. That article focused on Gettier bugs, but this passage discusses what you might call Gettier features.
Something that's gotten me before is Python's willingness to interpret a comma as a tuple. So instead of:
I wrote: Which was syntactically correct, equivalent to: The auto formatter does insert a space though, which helps. Maybe it could be made to transform it as I did above, that would make it screamingly obvious.[1a] https://jsomers.net/blog/gettiers
[1b] https://news.ycombinator.com/item?id=41840390
Your font and/or eyesight might need attention!
You know what, I do use a small font size in my editor. I like to see a lot of code at once. And if memory serves I spotted this in the browser, where I do the opposite.
I'll have to look into hyper legible monospace fonts. Or maybe I'll just use Atkinson and deal with the variable spacing.
Tell me you're a 20-something engineer without telling me you're a 20-something engineer.
I had a customer complain once about how great Akamai WAF was, because it never had false positives. (My company's WAF solution had many)
Is that actually desirable? This article articulates my exact gut feeling.
> This is why test-driven development gurus tell people to write a failing test first.
To be precise, it’s one of the big reasons, but it’s far from the only reason to write the test first.
I’m increasingly of the opinion that TDD is only as good as your system is testable.
This means that the time of writing your first test is too late. It’s part of the core business logic architecture – the whiteboard stage.
If you can make it testable, TDD isn’t just good practice – it’s what you want to do because it’s so natural. Similar to how unit tests are already natural when you write hermetic code (like say a string formatter).
If, OTOH, your business logic is inseparable from prod databases, files, networking, current time & time zone, etc, then TDD and tests in general are both cumbersome to write and simultaneously delivers much less value (as in finding errors) per test-case. Controversially, I think that for a spaghetti code application tests are quite useless and are largely ritualistic.
The only way I know how to design such testable systems (or subsystems) is through the “functional core - imperative shell” pattern. Not necessarily religious adherence to “no side effects”, but isolation is a must.
This is my problem, I don't worry about tests until I'm already putting the marinara sauce on my main functions.
> If, OTOH, your business logic is inseparable from prod databases, files, networking, current time & time zone, etc, then TDD and tests in general are both cumbersome to write and simultaneously delivers much less value (as in finding errors) per test-case. Controversially, I think that for a spaghetti code application tests are quite useless and are largely ritualistic.
I don't disagree with this and I have found it to be quite true -- though IMO it still has to be said that you can mock / isolate a lot of stuff, system time included. I am guessing you already accounted for that when you said that tests can become cumbersome to write and I agree. But we should still try because there are projects where you can't ever get a truly isolated system to test f.ex. I recently finished a contract where I had to write an server for dispatching SMS jobs to the right per-tenant & per-data-center instances of the actual connected-to-the-telco-network SMS servers; the dev environment was practically useless because the servers there did not emit half the events my application needed to function properly so I had to record the responses from the production servers and use them as mocks in my dev env tests.
Did the test succeed? Sure they did but ultimately gave me almost no confidence. :/
But yeah, anyway, I agree with your premise, I just think that we should still go the extra mile to reduce entropy and chaos as much as we can. Because nobody likes being woken up to fight a fire in production.
Website not quite loading.. HN hug of death?
Buttondown is a great non-success. Ergo they are a good company.
I like buttondown because I can directly contact the developer when I have problems. Some downsides to small companies, lots of upsides too.
There are few things that terrify me more nowadays at this point in my career than spending a lot of time writing something and setting it up, only to turn it on for the first time and it works without any issues.
"If it ain't broke, open it up and see what makes it so bloody special." -- The BOFH
Yes and the first thing I might ask is "how can I break this?" If I can't easily break it with a small change, I've probably missed something.
Oh god this is such a nightmare. It takes much longer to build something that works not the first try, because then I have to force-simulate a mistake to make sure things were actually correct in the first place.
Test Driven Development had a fix for this, which I used to do back in day when I was evangelical about the one true way the write software. You wrote a test that failed, and added or wrote code only to make that test pass. Never add any code except to make a failing test pass.
It didn't guarantee 100% correct software, of course, but it prevented you from gaslighting yourself for being too awesome.
Tests are like the burning sun in your eyes after you wake up for a night of drinking.
I prefer separating writing some code down, making it functionally work on screen and writing tests. I usually cover cases in step 2 but when you add sometime new later it is nice to have step 3.
Site is buggy
Works fine for me?
Nvidea ?