Edge Cache

Do you lead or follow? Frequently I observe application level caching designed by, "what other's are doing", without thinking about all available solutions. If your application utilizes things like redis or memcahce and you think that endpoints serving up the cached data are fast, think again. While the de-facto, "monkey see, monkey do" patterns of an application cache layer is a good starting point, you are not a monkey and it's time to branch out.

Out of scope:

In this discussion invalidating cache data is out of scope, as most of the time this is application specific. For example you could have a layer that detects DB changes and then deletes any associated cache keys based on a common pattern, or if data changes are not at the upmost importance it could be as simply as setting a cache key expiration when inserting into the cache.

De-facto Example

Edge/Gw Example

Why?

Why take the additional step of transitioning a local application cache to a gateway layer? Primarily, if you're leveraging OpenResty with LuaJIT, the performance benefits are substantial compared to common application frameworks like Java, Python, Node.js, Ruby, and others. Numerous benchmarks indicate that LuaJIT-optimized code exhibits speed close to C and C++ in many instances.

Performance: The numbers speak for themselves:

Openresty

ab -n 10000 -c 512 http://127.0.0.1:8080/cacheable

Server Hostname:        127.0.0.1
Server Port:            8080

Document Path:          /cacheable
Document Length:        14 bytes

Concurrency Level:      512
Time taken for tests:   0.428 seconds
Complete requests:      10000
Failed requests:        0
Total transferred:      1690000 bytes
HTML transferred:       140000 bytes
Requests per second:    23361.48 [#/sec] (mean)
Time per request:       21.916 [ms] (mean)
Time per request:       0.043 [ms] (mean, across all concurrent requests)
Transfer rate:          3855.56 [Kbytes/sec] received

Connection Times (ms)
min  mean[+/-sd] median   max
Connect:        0   10   3.3      9      27
Processing:     3   12   3.7     11      29
Waiting:        0    8   3.4      7      24
Total:         12   21   3.6     20      34

Percentage of the requests served within a certain time (ms)
50%     20
66%     21
75%     23
80%     25
90%     27
95%     28
98%     32
99%     33
100%     34 (longest request)

Ruby(v3.2.3)
bundle exec puma -w 8 -t 8:32 -p 8081

ab -n 10000 -c 512 http://127.0.0.1:8081/cacheable


Server Hostname:        127.0.0.1
Server Port:            8081

Document Path:          /cacheable
Document Length:        13 bytes

Concurrency Level:      512
Time taken for tests:   0.789 seconds
Complete requests:      10000
Failed requests:        0
Total transferred:      1860000 bytes
HTML transferred:       130000 bytes
Requests per second:    12681.10 [#/sec] (mean)
Time per request:       40.375 [ms] (mean)
Time per request:       0.079 [ms] (mean, across all concurrent requests)
Transfer rate:          2303.40 [Kbytes/sec] received

Connection Times (ms)
min  mean[+/-sd] median   max
Connect:        0   14   3.9     13      30
Processing:     6   26  10.1     23      90
Waiting:        1   21   9.6     18      86
Total:         18   39  11.4     36     110

Percentage of the requests served within a certain time (ms)
50%     36
66%     40
75%     43
80%     45
90%     51
95%     61
98%     79
99%     88
100%    110 (longest request)

node.js(v20.11.0):
NODE_ENV=production node raw_nodejs_server.js

ab -n 10000 -c 512 http://127.0.0.1:8082/cacheable

Server Hostname:        127.0.0.1
Server Port:            8082

Document Path:          /cacheable
Document Length:        13 bytes

Concurrency Level:      512
Time taken for tests:   0.837 seconds
Complete requests:      10000
Failed requests:        0
Total transferred:      880000 bytes
HTML transferred:       130000 bytes
Requests per second:    11946.48 [#/sec] (mean)
Time per request:       42.858 [ms] (mean)
Time per request:       0.084 [ms] (mean, across all concurrent requests)
Transfer rate:          1026.65 [Kbytes/sec] received

Connection Times (ms)
min  mean[+/-sd] median   max
Connect:        0    1   4.4      0      21
Processing:    20   40   6.1     40      54
Waiting:        1   40   6.9     40      54
Total:         20   41   5.6     41      54

Percentage of the requests served within a certain time (ms)
50%     41
66%     41
75%     45
80%     45
90%     50
95%     51
98%     53
99%     53
100%     54 (longest request)

Closing Thoughts

In my endeavor to illustrate the benefits of shifting an application cache to an edge layer for enhanced speed through a simplified API caching strategy example, the most unexpected finding from the benchmark tests was Ruby's remarkable progress. The last comparison I made between openresty, Ruby, and Node.js was several years back, where Node.js distinctly outperformed Ruby. At that time, I believe Ruby was around versions 2.2 to 2.3, although I'm not certain of the exact version. For enthusiasts of Ruby facing suggestions to transition to Node.js, I recommend conducting your own benchmarks with Ruby 3.2.x. The advancements in Ruby's JIT are noteworthy. In numerous iterations of apache bench, not once did Node.js surpass Ruby.

All testing was performed by the code at ccyphers/api_cache_comparison

• cache.lua
• bench/raw_nodejs_server.js
• bench/sinatra_app.rb