Link Search Menu Expand Document
Serverless Literature Guide

Benchmarks and Data Sets

Serverless computing stands to benefit from broadly accepted benchmarks. A number of these have been proposed, though a leader has not yet emerged. Contenders include FunctionBench [1], FaaSdom [2], and Serverlessbench [3]. DeathStarBench [4] is targeted at microservices as well as FaaS applications. Work by Martins et al. [5] also proposes a benchmark and uses it to compare cloud providers. Scheuner and Leitner [6] provide a literature review of various FaaS performance evaluations.

The need for new benchmarks is especially evident because serverless computing emphasizes autoscaling. The quality of this autoscaling is often referred to as “elasticity,” a metaphor that suggests it might be described by a simple number or perhaps a relationship between two variables, as is the case in physics or engineering. So far no such metric has emerged, though work by Kuhlenkamp et al. [7] moves in this direction.

  • [1]Jeongchul Kim and Kyungyong Lee. 2019. FunctionBench: A Suite of Workloads for Serverless Cloud Function Service. In 2019 IEEE 12th International Conference on Cloud Computing (CLOUD), IEEE, 502–504.
  • [2]Pascal Maissen, Pascal Felber, Peter Kropf, and Valerio Schiavoni. 2020. FaaSdom: A Benchmark Suite for Serverless Computing. In Proceedings of the 14th ACM International Conference on Distributed and Event-based Systems, 73–84.
  • [3]Tianyi Yu, Qingyuan Liu, Dong Du, Yubin Xia, Binyu Zang, Ziqian Lu, Pingchao Yang, Chenggang Qin, and Haibo Chen. 2020. Characterizing Serverless Platforms With Serverlessbench. In Proceedings of the 11th ACM Symposium on Cloud Computing, 30–44.
  • [4]Yu Gan, Yanqi Zhang, Dailun Cheng, Ankitha Shetty, Priyal Rathi, Nayan Katarki, Ariana Bruno, Justin Hu, Brian Ritchken, Brendon Jackson, and others. 2019. An Open-Source Benchmark Suite for Microservices and Their Hardware-Software Implications for Cloud & Edge Systems. In Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, 3–18.
  • [5]Horácio Martins, Filipe Araujo, and Paulo Rupino da Cunha. 2020. Benchmarking Serverless Computing Platforms. Journal of Grid Computing 18, 4 (2020), 691–709.
  • [6]Joel Scheuner and Philipp Leitner. 2020. Function-as-a-Service Performance Evaluation: A Multivocal Literature Review. Journal of Systems and Software 170, (2020), 110708.
  • [7]Jörn Kuhlenkamp, Sebastian Werner, Maria C. Borges, Dominik Ernst, and Daniel Wenzel. 2020. Benchmarking Elasticity of FaaS Platforms as a Foundation for Objective-Driven Design of Serverless Applications. In Proceedings of the 35th Annual ACM Symposium on Applied Computing, 1576–1585.