You know one of the things I hate about benchmarking is realizing that after the many, many tests you do it hits you that you forgot your new environment doesn’t have the same optimizations or configurations the one in production does.
Benchmarking/stress testing is exhausting. I don’t think I envy anyone that really has to benchmark web applications in new environments and compare them with the current ones. When the numbers are equal that’s where you realize there’s this uneasiness building inside you slowly eating you away that either there’s a bottleneck in your tests or the configuration that was done is incorrect.
But there’s actually a third one that I absolutely hated. The bottleneck is in the networking but not exactly the way you envision.
I present to you. My dumb realization:
I have my servers in Germany. I started benchmarking from my United States servers. Now, soak that in for a few seconds and realize that sending requests across nations on complex networking systems for it to hit my Germany servers took more time than I thought. It doesn’t help that for me to get to my German servers a ping come at 200ms where I live.
So I had to change my mindset a bit. If I’m really going to stress test my servers to see how many requests I can serve concurrent then I have to think what would be the easiest way. The answer for that is quite simple… I just had to create another server on my German host provider so I could actually do proper stress tests/benchmarking.
But there’s another miss on this. I didn’t have any of my production configurations in the server! Even though I was getting good results from static files up to 4000 requests per second and around 1000 with dynamic requests (wordpress site) I missed PHP optimizations that would have pushed it a bit more.
Thus… honestly yesterday I spent way too much time benchmarking and stress testing this setup. I didn’t want more of it. I’m super happy with what I have right now and I hope this is a good setup for the future to come.