Haiyu Mao

selected publications

1. ASPLOS

   PAPI: Exploiting Dynamic Parallelism in Large Language Model Decoding with a Processing-In-Memory-Enabled Computing System

   Yintao He, Haiyu Mao, Christina Giannoula, Mohammad Sadrosadati, Juan Gomez-Luna, Huawei Li, Xiaowei Li, Ying Wang, and Onur Mutlu

   2025

   Abs Bib PDF

   Large language models (LLMs) are widely used for natural language understanding and text generation. An LLM model relies on a time-consuming step called LLM decoding to generate output tokens. Several prior works focus on improving the performance of LLM decoding using parallelism techniques, such as batching and speculative decoding. State-of-the-art LLM decoding has both compute-bound and memory-bound kernels. Some prior works statically identify and map these different kernels to a heterogeneous architecture consisting of both processing-in-memory (PIM) units and computation-centric accelerators. We observe that characteristics of LLM decoding kernels (e.g., whether or not a kernel is memory-bound) can change dynamically due to parameter changes to meet user and/or system demands, making (1) static kernel mapping to PIM units and computation-centric accelerators suboptimal, and (2) one-size-fits-all approach of designing PIM units inefficient due to a large degree of heterogeneity even in memory-bound kernels. In this paper, we aim to accelerate LLM decoding while considering the dynamically changing characteristics of the kernels involved. We propose PAPI (PArallel Decoding with PIM), a PIM-enabled heterogeneous architecture that exploits dynamic scheduling of compute-bound or memory-bound kernels to suitable hardware units. PAPI has two key mechanisms: (1) online kernel characterization to dynamically schedule kernels to the most suitable hardware units at runtime and (2) a PIM-enabled heterogeneous computing system that harmoniously orchestrates both computation-centric processing units and hybrid PIM units with different computing capabilities. Our experimental results on three broadly-used LLMs show that PAPI achieves 1.8× and 11.1× speedups over a state-of-the-art heterogeneous LLM accelerator and a state-of-the-art PIM-only LLM accelerator, respectively.

   @article{papi,
     title = {PAPI: Exploiting Dynamic Parallelism in Large Language Model Decoding with a Processing-In-Memory-Enabled Computing System},
     author = {He, Yintao and Mao, Haiyu and Giannoula, Christina and Sadrosadati, Mohammad and Gomez-Luna, Juan and Li, Huawei and Li, Xiaowei and Wang, Ying and Mutlu, Onur},
     booktitle = {The 30th International Conference on Architectural Support for Programming Languages and Operating Systems},
     year = {2025}
   }

2. MICRO

   GenPIP: In-Memory Acceleration of Genome Analysis via Tight Integration of Basecalling and Read Mapping

   Haiyu Mao, Mohammed Alser, Mohammad Sadrosadati, Can Firtina, Akanksha Baranwal, Damla Senol Cali, Aditya Manglik, Nour Almadhoun Alserr, and Onur Mutlu

   2022

   Abs Bib PDF

   Nanopore sequencing is a widely-used high-throughput genome sequencing technology that can sequence long fragments of a genome into raw electrical signals at low cost. Nanopore sequencing requires two computationally-costly processing steps for accurate downstream genome analysis. The first step, basecalling, translates the raw electrical signals into nucleotide bases (i.e., A, C, G, T). The second step, read mapping, finds the correct location of a read in a reference genome. In existing genome analysis pipelines, basecalling and read mapping are executed separately. We observe in this work that such separate execution of the two most time-consuming steps inherently leads to (1) significant data movement and (2) redundant computations on the data, slowing down the genome analysis pipeline. This paper proposes GenPIP, an in-memory genome analysis accelerator that tightly integrates basecalling and read mapping. GenPIP improves the performance of the genome analysis pipeline with two key mechanisms: (1) in-memory fine-grained collaborative execution of the major genome analysis steps in parallel; (2) a new technique for early-rejection of low-quality and unmapped reads to timely stop the execution of genome analysis for such reads, reducing inefficient computation. Our experiments show that, for the execution of the genome analysis pipeline, GenPIP provides 41.6× (8.4×) speedup and 32.8× (20.8×) energy savings with negligible accuracy loss compared to the state-of-the-art software genome analysis tools executed on a state-of-the-art CPU (GPU). Compared to a design that combines state-of-the-art in-memory basecalling and read mapping accelerators, GenPIP provides 1.39× speedup and 1.37× energy savings.

   @article{mao2022genpip,
     title = {GenPIP: In-Memory Acceleration of Genome Analysis via Tight Integration of Basecalling and Read Mapping},
     author = {Mao, Haiyu and Alser, Mohammed and Sadrosadati, Mohammad and Firtina, Can and Baranwal, Akanksha and Cali, Damla Senol and Manglik, Aditya and Alserr, Nour Almadhoun and Mutlu, Onur},
     booktitle = {2022 55th Annual IEEE/ACM International Symposium on Microarchitecture},
     year = {2022}
   }

3. TC

   LrGAN: A Compact and Energy Efficient PIM-Based Architecture for GAN Training

   Haiyu Mao, Jiwu Shu, Mingcong Song, and Tao Li

   IEEE Transactions on Computers 2020

   Abs Bib PDF

   As a powerful unsupervised learning method, Generative Adversarial Network (GAN) plays an essential role in many domains. However, training a GAN imposes four more challenges: (1) intensive communication caused by complex train phases of GAN, (2) much more ineffectual computations caused by peculiar convolutions, (3) more frequent off-chip memory accesses for exchanging intermediate data between the generator and the discriminator and (4) high energy consumption of unnecessary fine-grained MLC programming. In this paper, we propose LrGAN, a PIM-based GAN accelerator, to address the challenges of training GAN. We first propose a zero-free data reshaping scheme for ReRAM-based PIM, which removes the zero-related computations. We then propose a 3D-connected PIM, which can reconfigure connections inside PIM dynamically according to dataflows of propagation and updating. After that, we propose an approximate weight update algorithm to avoid unnecessary fine-grain MLC programming. Finally, we propose LrGAN based on these three techniques, providing different levels of accelerating GAN for programmers. Experiments show that LrGAN achieves 47.2×, 21.42×, and 7.46× speedup over FPGA-based GAN accelerator, GPU platform, and ReRAM-based neural network accelerator respectively. Besides, LrGAN achieves 13.65×, 10.75×, and 1.34× energy saving on average over GPU platform, PRIME, and FPGA-based GAN accelerator, respectively.

   @article{mao2020lrgan,
     title = {LrGAN: A Compact and Energy Efficient PIM-Based Architecture for GAN Training},
     author = {Mao, Haiyu and Shu, Jiwu and Song, Mingcong and Li, Tao},
     journal = {IEEE Transactions on Computers},
     year = {2020},
     publisher = {IEEE}
   }

4. MICRO

   Lergan: A zero-free, low data movement and pim-based gan architecture

   Haiyu Mao, Mingcong Song, Tao Li, Yuting Dai, and Jiwu Shu

   In 2018 51st Annual IEEE/ACM International Symposium on Microarchitecture 2018

   Abs Bib PDF Lightning talk

   As a powerful unsupervised learning method, Generative Adversarial Network (GAN) plays an important role in many domains such as video prediction and autonomous driving. It is one of the ten breakthrough technologies in 2018 reported in MIT Technology Review. However, training a GAN imposes three more challenges: (1) intensive communication caused by complex train phases of GAN, (2) much more ineffectual computations caused by special convolutions, and (3) more frequent off-chip memory accesses for exchanging inter-mediate data between the generator and the discriminator. In this paper, we propose LerGAN, a PIM-based GAN accelerator to address the challenges of training GAN. We first propose a zero-free data reshaping scheme for ReRAM-based PIM, which removes the zero-related computations. We then propose a 3D-connected PIM, which can reconfigure connections inside PIM dynamically according to dataflows of propagation and updating. Our proposed techniques reduce data movement to a great extent, avoiding I/O to become a bottleneck of training GANs. Finally, we propose LerGAN based on these two techniques, providing different levels of accelerating GAN for programmers. Experiments shows that LerGAN achieves 47.2×, 21.42× and 7.46× speedup over FPGA-based GAN accelerator, GPU platform, and ReRAM-based neural network accelerator respectively. Moreover, LerGAN achieves 9.75×, 7.68× energy saving on average over GPU platform, ReRAM-based neural network accelerator respectively, and has 1.04× energy consuming over FPGA-based GAN accelerator.

   @inproceedings{mao2018lergan,
     title = {Lergan: A zero-free, low data movement and pim-based gan architecture},
     author = {Mao, Haiyu and Song, Mingcong and Li, Tao and Dai, Yuting and Shu, Jiwu},
     booktitle = {2018 51st Annual IEEE/ACM International Symposium on Microarchitecture},
     lightningtalk = {https://www.youtube.com/watch?v=dmsGaoJKbAU},
     year = {2018},
     organization = {IEEE}
   }

5. DATE

   Protect non-volatile memory from wear-out attack based on timing difference of row buffer hit/miss

   Haiyu Mao, Xian Zhang, Guangyu Sun, and Jiwu Shu

   In Design, Automation & Test in Europe Conference & Exhibition 2017

   Abs Bib PDF

   Non-volatile Memories (NVMs), such as PCM and ReRAM, have been widely proposed for future main memory design because of their low standby power, high storage density, fast access speed. However, these NVMs suffer from the write endurance problem. In order to prevent a malicious program from wearing out NVMs deliberately, researchers have proposed various wear-leveling methods, which remap logical addresses to physical addresses randomly and dynamically. However, we discover that side channel leakage based on NVM row buffer hit information can reveal details of address remappings. Consequently, it can be leveraged to side-step the wear-leveling. Our simulation shows that the proposed attack method in this paper can wear out a NVM within 137 seconds, even with the protection of state-of-the-art wear-leveling schemes. To counteract this attack, we further introduce an effective countermeasure named Intra-Row Swap (IRS) to hide the wear-leveling details. The basic idea is to enable an additional intra-row block swap when a new logical address is remapped to the memory row. Experiments demonstrate that IRS can secure NVMs with negligible timing/energy overhead, compared with previous works.

   @inproceedings{mao2017protect,
     title = {Protect non-volatile memory from wear-out attack based on timing difference of row buffer hit/miss},
     author = {Mao, Haiyu and Zhang, Xian and Sun, Guangyu and Shu, Jiwu},
     booktitle = {Design, Automation \& Test in Europe Conference \& Exhibition},
     year = {2017},
     organization = {IEEE}
   }

6. NVMSA

   Exploring data placement in racetrack memory based scratchpad memory

   Haiyu Mao, Chao Zhang, Guangyu Sun, and Jiwu Shu

   In IEEE Non-Volatile Memory System and Applications Symposium 2015

   Abs Bib PDF

   Scratchpad Memory (SPM) has been widely adopted in various computing systems to improve performance of data access. Recently, non-volatile memory technologies (NVMs) have been employed for SPM design to improve its capacity and reduce its energy consumption. In this paper, we explore data allocation in SPM based on racetrack memory (RM), which is an emerging NVM with ultra-high storage density and fast access speed. Since a shift operation is needed to access data in RM, data allocation has an impact on performance of RM based SPM. Several allocation methods have been discussed and compared in this work. Especially, we addressed how to leverage genetic algorithm to achieve near-optimal data allocation.

   @inproceedings{mao2015exploring,
     title = {Exploring data placement in racetrack memory based scratchpad memory},
     author = {Mao, Haiyu and Zhang, Chao and Sun, Guangyu and Shu, Jiwu},
     booktitle = {IEEE Non-Volatile Memory System and Applications Symposium},
     year = {2015},
     organization = {IEEE}
   }