This paper introduces a new pre-training method for bidirectional transformers that improves performance on a variety of language understanding problems.