Tested to destruction
Now I'm not thinking about a yet to be rated movie, or a song by Onslaught, but a technical system.
@loadstorm tweeted a question about the different kinds of testing to me.
I've had to deal with (i.e. clean up) the creations of application programmers, or novice games programmers playing with the internet (never again!!) and trying to build a system.
I've found there are very different ways of approach technical creation, a different understanding of what is acceptable, and a few fairs of being confident you've tested it. I could wax lyrical about quality, though Zen and the Art of Motorcycle Maintenance: An Inquiry into Values (P.S.)
, does that nicely.
A little while ago I was load testing a system I've build to observe it's behaviour, over time and to the point of non response (0 performance and over all system fail) i.e. destruction. The non use state is nice and easy to measure, do nothing and see what the spare system resources & capacities are, then run the simplest set of use case tests while measuring.
What is it doing, how fast and how does that effect it. I'm working with a usual LAMP stack, so the "system boundary" as I see it, is everything between the back of the clients browser (as far as I can detect at the TCP/IP level what is going over the wire) to the disk (SSD in this case) where the data sits.
Though given latency and other routing, I'll take responsibility as soon as the request hits httpd (but am aware of the network system interface).
We all know it's a state based machine, so you poke it and it (sometimes) changes and (usually) responses !
The typical performance test I've observed is client response, how fast did the state response (from moment of request) get back to the client.
Having been asked what is the difference between load, stress and performance testing, I don't want to define each one and try and separate them, as they are not in my view, they are exhibited behaviours or attributes of a system at a given time in a given state.
Load being the increase in requests of the system to perform an action which can be client requests or dB back up etc. Try running a full client load test ......... then do a mysqldump on a large MyISAM that locks the tables and see what happens. Have to remember it's the system as a whole.
Stress Is a measurement or another word for load in my view, though lots of it and in increasing measures i.e. "stressing" the system to the point here performance drops below what is acceptable. Load is any point of measurement, stress is typically when the cracks appear.
Performance being quite literally the metric of the measurement from the test, how fast did it do X task, how much data did it process in Y time.
As a correlation of all 3 and as an outcome of testing you can stress a system with load to see it's performance (typically) degrade to a point where it is deemed overloaded (response time is over X and process Y data per second etc) all the way to failing.
Though given that I try to take a systems view, it's how everything works together, most importantly how the components interact, or fail to in a certain situation.
Having a 48G web server with quad hex cores pointed at a single HD machine with a 5400rpm ATA drive might look good when you're running your PHP with no dB access, though hit it with 5000 requests that use table locks and see what happens.
I don't see the system stopping under testing as a "failure", though a gaining of knowledge as to it's behaviour and typically what interconnection of a sub system that I've not paid attention to, or if I have, what the system gets "rated to" ....... hopefully being the target requirement of the project.
Much like tires on my motorbike, rated to 170 mph.
Not knowing that number is the failure, not that the system itself fails at a given point.
@loadstorm tweeted a question about the different kinds of testing to me.
I've had to deal with (i.e. clean up) the creations of application programmers, or novice games programmers playing with the internet (never again!!) and trying to build a system.
I've found there are very different ways of approach technical creation, a different understanding of what is acceptable, and a few fairs of being confident you've tested it. I could wax lyrical about quality, though Zen and the Art of Motorcycle Maintenance: An Inquiry into Values (P.S.)
, does that nicely.A little while ago I was load testing a system I've build to observe it's behaviour, over time and to the point of non response (0 performance and over all system fail) i.e. destruction. The non use state is nice and easy to measure, do nothing and see what the spare system resources & capacities are, then run the simplest set of use case tests while measuring.
What is it doing, how fast and how does that effect it. I'm working with a usual LAMP stack, so the "system boundary" as I see it, is everything between the back of the clients browser (as far as I can detect at the TCP/IP level what is going over the wire) to the disk (SSD in this case) where the data sits.
Though given latency and other routing, I'll take responsibility as soon as the request hits httpd (but am aware of the network system interface).
We all know it's a state based machine, so you poke it and it (sometimes) changes and (usually) responses !
The typical performance test I've observed is client response, how fast did the state response (from moment of request) get back to the client.
Having been asked what is the difference between load, stress and performance testing, I don't want to define each one and try and separate them, as they are not in my view, they are exhibited behaviours or attributes of a system at a given time in a given state.
Load being the increase in requests of the system to perform an action which can be client requests or dB back up etc. Try running a full client load test ......... then do a mysqldump on a large MyISAM that locks the tables and see what happens. Have to remember it's the system as a whole.
Stress Is a measurement or another word for load in my view, though lots of it and in increasing measures i.e. "stressing" the system to the point here performance drops below what is acceptable. Load is any point of measurement, stress is typically when the cracks appear.
Performance being quite literally the metric of the measurement from the test, how fast did it do X task, how much data did it process in Y time.
As a correlation of all 3 and as an outcome of testing you can stress a system with load to see it's performance (typically) degrade to a point where it is deemed overloaded (response time is over X and process Y data per second etc) all the way to failing.
Though given that I try to take a systems view, it's how everything works together, most importantly how the components interact, or fail to in a certain situation.
Having a 48G web server with quad hex cores pointed at a single HD machine with a 5400rpm ATA drive might look good when you're running your PHP with no dB access, though hit it with 5000 requests that use table locks and see what happens.
I don't see the system stopping under testing as a "failure", though a gaining of knowledge as to it's behaviour and typically what interconnection of a sub system that I've not paid attention to, or if I have, what the system gets "rated to" ....... hopefully being the target requirement of the project.
Much like tires on my motorbike, rated to 170 mph.
Not knowing that number is the failure, not that the system itself fails at a given point.
Hi
ReplyDeleteWhat are the pass criteria for a performance test?
Like you I think this are pretty fluid. I doubt a CEO of a new on-line store is going to delay launch because the response time is 0.1 seconds more than their specificed performance requirements.
However, it is good to have goals.
Hi Andrew,
ReplyDeleteI tend to think about things as "good enough most of the time" opposed to black and white pass fail etc. Much like my golf game.
I try to set as pass criteria with the wording of "It must do X 95% of the time".
Dealing with CEOs can always be fun, though here we don't have launch dates, we have "ready assessments" and as soon as it's passed then it's up to business processes etc to deal with going to market etc.
We've found that projects get done faster without them, much like the observations of teachers who don't tell students the marks, typically they go up.