Please use this identifier to cite or link to this item: https://hdl.handle.net/20.500.11851/8612
Title: A Performance Study Depending on Execution Times of Various Frameworks in Machine Learning Inference
Authors: Sever M.
Ogut S.
Keywords: inference
machine learning
ONNX Runtime
optimization
TensorRT
Average power
File formats
Inference
Machine-learning
ONNX runtime
Optimisations
Performance study
Power calculation
Runtimes
Tensorrt
Machine learning
Issue Date: 2021
Publisher: Institute of Electrical and Electronics Engineers Inc.
Source: Sever, M., & Öğüt, S. (2021, November). A Performance Study Depending on Execution Times of Various Frameworks in Machine Learning Inference. In 2021 15th Turkish National Software Engineering Symposium (UYMS) (pp. 1-5). IEEE.
Abstract: This work is intended to compare the latency of various frameworks in machine learning inference through an average power calculation model. This model is created in terms of a 2-layer neural network with PyTorch, in Python. Then, it is converted to a traced Torch Script module and also to ONNX file format. Afterwards, the C++ front-end is used for the inference process. The traced model is run with Libtorch on CPU and GPU, the ONNX file is run with ONNX Runtime on both CPU and GPU and it is also run with TensorRT on GPU. The inference execution times for 100 trials are averaged for all cases and it is realized that TensorRT with ONNX file format significantly outperforms its counterparts as expected. Hence, this work highlights the performance of TensorRT in machine learning inference and sheds light into the future by proposing several extensions. © 2021 IEEE.
Description: 15th Turkish National Software Engineering Symposium, UYMS 2021 -- 17 November 2021 through 19 November 2021 -- -- 176220
URI: https://doi.org/10.1109/UYMS54260.2021.9659677
https://hdl.handle.net/20.500.11851/8612
ISBN: 9781665410700
Appears in Collections:Bilgisayar Mühendisliği Bölümü / Department of Computer Engineering
Scopus İndeksli Yayınlar Koleksiyonu / Scopus Indexed Publications Collection
WoS İndeksli Yayınlar Koleksiyonu / WoS Indexed Publications Collection

Show full item record

CORE Recommender

Page view(s)

26
checked on Dec 26, 2022

Google ScholarTM

Check

Altmetric


Items in GCRIS Repository are protected by copyright, with all rights reserved, unless otherwise indicated.