A Quantitative Analysis Framework for Recurrent Neural Network

Xiaoning Du, Xiaofei Xie, Yi Li, Lei Ma, Yang Liu, and Jianjun Zhao

In Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2019

Abstract: Recurrent neural network (RNN) has achieved great success in processing sequential inputs for applications such as automatic speech recognition, natural language processing and machine translation. However, quality and reliability issues of RNNs make them vulnerable to adversarial attacks and hinder their deployment in real-world applications. In this paper, we propose a quantitative analysis framework—DeepStellar—to pave the way for effective quality and security analysis of software systems powered by RNNs. DeepStellar is generic to handle various RNN architectures, including LSTM and GRU, scalable to work on industrial-grade RNN models, and extensible to develop customized analyzers and tools. We demonstrated that, with DeepStellar, users are able to design efficient test generation tools, and develop effective adversarial sample detectors. We tested the developed applications on three real RNN models, including speech recognition and image classification. DeepStellar outperforms existing approaches three hundred times in generating defect-triggering tests and achieves 97% accuracy in detecting adversarial attacks. A video demonstration which shows the main features of DeepStellar is available at: https://sites.google.com/view/deepstellar/tool-demo.

Cite:

@inproceedings{Du2019AQA,
  author = {Du, Xiaoning and Xie, Xiaofei and Li, Yi and Ma, Lei and Liu, Yang and Zhao, Jianjun},
  booktitle = {Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering (ASE)},
  month = nov,
  pages = {1062--1065},
  title = {A Quantitative Analysis Framework for Recurrent Neural Network},
  year = {2019}
}