Efficient Batching Strategy
method
ID: efficientbatchingstrategy
Related Papers
A Streaming Approach For Efficient Batched Beam Search
AbstractWe propose an efficient batching strategy for variable-length decoding on GPU architectures. During decoding, when candidates terminate or are pruned according to heuristics, our streaming approach periodically “refills” the batch …...