The Model Is the Byproduct

| 0.71 | 1.80 | 100+ |

Key Findings:

• Hardware-Driven Optima: Agents on H100s converge on aggressive learning rates and large batch sizes, while constrained hardware (e.g., Mac Mini M4, consumer GPUs) forces architectural simplification, efficient normalization, and targeted initialization strategies.
• Transferable Discoveries: ~20 additive optimizations (attention scaling, regularization tuning, initialization corrections) discovered on depth-12 models transferred successfully to larger architectures, reducing benchmark time-to-GPT-2 by 11%.
• Vocabulary-Agnostic Evaluation: Validation bits per byte enables fair comparison across tokenizer changes, embedding dimension shifts, and architectural modifications, eliminating metric fragmentation.
• Distributed Convergence: 35 heterogeneous agents independently rediscovered established ML techniques (RMSNorm, tied embeddings, specific initialization patterns) within 17 hours, proving that constraint diversity accelerates solution convergence.

Core Solution

The autoresearch framework abstracts model development into a closed-loop optimization system: define an objective, let an agent iterate on the system, measure against a vocabulary-agnostic metric, keep or discard, repeat. The implementation is deliberately minimal to maximize agent autonomy and hardware portability.

Architecture & Implementation:

• Core Files: train.py (training loop), prepare.py (data preprocessing), program.md (agent instruction set)
• Experiment Cadence: Fixed 5-minute windows per run. This duration is not arbitrary; it forces hardware-aware optimization by aligning experiment length with target device throughput, preventing overfitting and enabling high-frequency iteration.
• Agent Modification Scope: The agent autonomously adjusts neural network architecture (layer depth, attention mechanisms, normalization choices), optimizer configurations, learning rates, batch sizes, and embedding dimensions.
• Evaluation Metric: Validation bits per byte. This metric measures predictive efficiency independent of vocabulary size, ensuring that tokenizer swaps, embedding dimension changes, and architectural shifts remain directly comparable.
• Hardware Diversity as Optimization Variable: The system treats hardware constraints as first-class citizens. Agents adapt search strategies based on available FLOPs, memory bandwidth, and thermal limits. Results from constrained devices often yield more efficient architectures than brute-force scaling.
• Abstracted Pattern: The framework decouples the optimization loop from neural networks. Any system with a measurable outcome, an iterable configuration space, and a bounded evaluation window can adopt this pattern.

Pitfall Guide

1. Ignoring Hardware-Aware Constraints: Treating all compute environments identically leads to suboptimal configurations. H100s tolerate aggressive batch sizes and learning rates, while consumer GPUs or CPUs require architectural simplification and memory-efficient initialization. Always bound experiments to target hardware throughput.
2. Using Vocabulary-Dependent Metrics: Metrics like perplexity or accuracy tied to fixed tokenizers break cross-architecture comparisons. Use vocabulary-agnostic metrics (e.g., validation bits per byte) to ensure fair evaluation across embedding dimension and tokenizer changes.
3. Unbounded Experiment Cycles: Allowing agents to run indefinitely wastes compute and causes optimization drift. Enforce strict time budgets (e.g., 5-minute windows) to force rapid iteration, prevent overfitting to transient loss spikes, and maintain high experiment throughput.
4. Vague or Overly Restrictive program.md: The agent operates strictly within the constraints of the prompt. Overly broad instructions trigger random search; overly restrictive ones prevent discovery. Provide clear objective functions, allowed modification ranges, evaluation criteria, and failure thresholds.
5. Assuming Linear Transferability: Optimizations discovered on small models (e.g., depth-12) do not always scale proportionally. Validate discovered architectural tweaks (attention scaling, normalization, initialization patterns) on larger targets before merging into production codebases.
6. Misaligning Data Relevance with Model Scale: Scaling model capacity without aligning to domain-specific data yields diminishing returns. Prioritize high-signal, task-relevant datasets over internet-scale scrapes for narrow applications.
7. Neglecting Failure Mode Logging: Discarding failed experiments without structured logging loses valuable negative space information. Track crash patterns, divergence triggers, and invalid hyperparameter combinations to guide future agent search boundaries.

Deliverables

• 📘 Autonomous Experimentation Blueprint: Step-by-step architecture for deploying autoresearch-style loops, including hardware profiling, metric selection, agent prompt engineering, and transfer validation pipelines.
• ✅ Experiment Readiness Checklist: Pre-flight validation for data pipeline integrity, metric compatibility, hardware constraint mapping, agent instruction clarity, and failure logging configuration.
• ⚙️ Configuration Templates:
  • program.md Template: Structured prompt format defining objectives, modification boundaries, evaluation metrics, and iteration rules.
  • Hardware-Aware Hyperparameter Bounds Matrix: Pre-configured search spaces for H100, consumer GPUs, and CPU/M4 environments to prevent invalid configurations and accelerate convergence.