Training for Specialization

What do LLMs that are designed to be fine-tuned later look like? For ex, if you are Thinking Machines, where the main product is a fine-tuning platform, do you train the base model any differently besides trying to cover a lot of different domains?

• How do you make sure the model weight are amenable to new behavior via further training rather than fighting it?

• Can you preserve general solution strategies, and meta heuristics in the model weights; while making it easy to swap out more factual knowledge? Does this split even make sense?

• Do you have to sacrifice performance on typical frontier model evals, for faster/better specialization?

• What are good proxies for how fine-tunable a model artifact is? In-context learnability proportional to gains feasible with finetuning?