python 3.12.13, torch 2.14.1, transformers 5.17.0, arm64, device cpu, 4 CPU threads, float32, greedy, max_new_tokens=128, 5 interleaved repeats
target Qwen/Qwen2.5-1.5B-Instruct, draft Qwen/Qwen2.5-0.5B-Instruct

[explain] prompt 41 tokens: 'Explain what a hash table is in about five sentences.'
  method                 tokens  tok/s speedup target calls tok/call  same output
  1.5B alone                108   10.9   1.00x          108     1.00  yes
  1.5B + 0.5B draft         108    9.0   0.82x           39     2.77  yes
  1.5B + prompt lookup      108    8.3   0.76x           96     1.12  yes
  output (1.5B alone): 'A hash table is a data structure that uses a hash function to map keys to array indices, allowing for efficient insertion, deletion, and search operations. It is designed to provide average-case constant time complexity for these operations, making it suitable for applications requiring fast data re'

[code] prompt 51 tokens: 'Write a Python function is_palindrome(s) that ignores case and non-let'
  method                 tokens  tok/s speedup target calls tok/call  same output
  1.5B alone                128   10.9   1.00x          128     1.00  yes
  1.5B + 0.5B draft         128    8.5   0.78x           13     9.85  yes
  1.5B + prompt lookup      128    6.6   0.60x          113     1.13  yes
  output (1.5B alone): '```python\ndef is_palindrome(s):\n    """\n    Check if the given string is a palindrome, ignoring case and non-letter characters.\n    \n    A palindrome is a word, phrase, number, or other sequences of characters that reads the same forward and backward (ignoring spaces, punctuation, and capitalization'

[rewrite] prompt 150 tokens: 'Rewrite this news item as a bulleted list of facts. Keep the numbers e'
  method                 tokens  tok/s speedup target calls tok/call  same output
  1.5B alone                100    9.3   1.00x          100     1.00  yes
  1.5B + 0.5B draft         100    4.4   0.48x           15     6.67  yes
  1.5B + prompt lookup      100   13.6   1.47x           39     2.56  yes
  output (1.5B alone): '- City council voted to extend central library opening hours\n  - From March, library will open at 8 a.m. instead of 9 a.m. on weekdays\n  - Library will close at 9 p.m. instead of 8 p.m. on weekdays\n  - Change costs 240,000 dollars a year\n  - 2 new staff members hired\n  - Weekend hours remain at 10 a'

[story] prompt 47 tokens: 'Write the opening of a short story about a lighthouse keeper who finds'
  method                 tokens  tok/s speedup target calls tok/call  same output
  1.5B alone                128    6.6   1.00x          128     1.00  yes
  1.5B + 0.5B draft         128    3.6   0.54x           56     2.29  yes
  1.5B + prompt lookup      128    5.9   0.89x          126     1.02  yes
  output (1.5B alone): "The old lighthouse stood tall on the cliff's edge, its beacon casting a warm glow over the rugged coastline. The keeper, a man named Thomas, had spent his life guarding this place, his eyes accustomed to the harsh light of the sea. One stormy night, as the waves crashed against the rocks below, Thom"

[draft model alone, for reference]
  explain    91 tokens,   34.7 tok/s
  code      128 tokens,   35.0 tok/s
  rewrite    94 tokens,   34.9 tok/s
  story     128 tokens,   35.6 tok/s

[cost of one 1.5B forward pass that checks k tokens after a cached prompt]
  k= 1:  132.7 ms  (1.00x the cost of k=1)
  k= 4:  162.9 ms  (1.23x the cost of k=1)
  k=10:  153.3 ms  (1.16x the cost of k=1)

assistant settings used: num_assistant_tokens=None, schedule=None, confidence_threshold=None