About
Deep learning has scaled remarkably well, but the model sizes needed for state-of-the-art results are growing exponentially and outpacing hardware improvements. Can we use parameters more efficiently?
Almost all of the literature assumes a fixed architecture: choose a model, then train it. A rapidly growing body of work breaks that assumption and changes the trainable structure during optimization. Examples include:
- Growing neurons and layers
- Stacking and progressive scaling of transformers
- Dynamic sparse training and prune-and-regrow schemes that rewire the network’s topology as it learns
- Modular expansion for non-stationary tasks in continual and reinforcement learning
In each case, capacity is decided during training rather than fixed in advance, so parameters are used only where and when they are useful.
The open problem
These methods share one unresolved problem: once the trainable structure changes, how should optimization continue? A growth step changes the set of trainable parameters, and with it the optimizer state, learning-rate schedule and normalization statistics. Handling these badly can destroy trainability even when the new architecture is well motivated. Open questions include:
- When and where should a model change its structure within a single training run?
- How should parameters, optimizer state and normalization statistics be transferred across a growth step?
- How can training be stabilized immediately afterwards?
- How can we measure the quality of a growth transition?
Why this workshop
This line of work is split across the growth, sparse-training, continual-learning, reinforcement-learning and optimization communities. Each uses its own vocabulary, baselines and evaluation practices, and there is little direct collaboration between them. This one-day workshop brings these groups into one room to build a shared understanding of the problem before the field fragments further into method-specific benchmarks.
It is also a deliberately frugal research direction, in line with the goals of the MANOLO project: progress comes from algorithmic insight rather than compute budget, which makes the area accessible to smaller labs priced out of large-scale deep learning.
The day features five invited talks, a poster session over lunch, and a closing panel discussion. See the programme and the call for posters.
Important dates
| Poster abstract deadline | 27 November 2026 |
|---|---|
| Notification of acceptance | 30 November 2026 |
| Registration closes | 2 December 2026 |
| Workshop | 7 December 2026 |
Invited speakers
- Manon VerbockhavenENS Lyon Expressivity bottlenecks and optimal growth of neural networks.
- Pierre WolinskiUniversité Paris Dauphine–PSL Neural network theory, initialization, pruning, regularization and second-order methods.
- Kaitlin MaileGoogle Retrieval-centric deep learning
- Rupert MitchellTU Darmstadt Growing the width and depth of neural networks during training as capacity becomes necessary.
1 more speaker, to be announced.
Organizers
- Alex DaveyInria
- Stéphane RivaudUniversité Paris-Saclay
- Sylvain ChevallierUniversité Paris-Saclay
- Guillaume CharpiatInria