GigaToken: ~1000x faster Language model tokenization

244 points

by syrusakbary

3 hours ago

47 comments