Milestone Zero - “M0”

Table of contents

1. Milestone Zero - “M0”
2. Goals of Milestone Zero
3. Consortium Training Proofs of Concept (PoCs)
   1. BharatGen and Monash University
   2. Consortium Training Using the Flower Federated Framework
      1. Model, Data, and Training Configuration
      2. Deployment and Model Exchange
      3. Operational Resilience and Results
4. Cultural Alignment
   1. Proof of Concept for Alignment Based on Inglehart-Welzel Cultural Map
      1. Project Tapestry’s Novel Contribution Process
   2. Cultural-CPT Validation Harness
5. Other Contributions
6. Data Governance and Management Requirements
7. Software Development Policies and Practices
8. Acknowledgements
   1. What’s Next?

Goals of Milestone Zero

Milestone “zero” (M0) was Project Tapestry’s first technical milestone. It was completed September 1, 2026. We used “zero””, rather than “one”, because M0 was about building the consortium, while also pursuing initial goals. We started work in several key areas:

We worked in several key areas:

• Demonstrate the feasibility of consortium training, as defined in Training Approaches: Centralized, Federated, and Consortium (where it is also compared to federated learning). M0 included two PoCs (proofs of concept), each of which used two, geographically-distributed “sovereign nodes” (training clusters) collaborating to fine-tune a model.
• Explore techniques for cultural alignment.
• Start defining the requirements for our data governance and management strategy.
• Establish our software development policies and practices.

This page provides details for these work streams. Some of the M0 project teams will publish more detailed reports separately. We will update this page when more information becomes available for those reports. See also the M0 release notes.

Consortium Training Proofs of Concept (PoCs)

Two separate PoCs explored consortium training techniques.

BharatGen and Monash University

The first PoC for consortium training, “epic” #189, was conducted by a joint team from BharatGen, in India, and Monash University, in Australia.

The contributors to this project include the following people:

• From BharatGen (and affiliated institutions): Maneesh Kumar Singh (BharatGen), Bapi Chatterjee (BharatGen and Indraprastha Institute of Information Technology Delhi - IIITD), Anant Jain (IIITD), Gauranshi Gupta (IIITD), Mounendra Desarkar (IIT Hyderabad), Ganesh Ramakrishnan (IIT Bombay), Piyush Sawarkar (BharatGen), Aamod Thakur (BharatGen), Samarth Pradhan (BharatGen, IIT Bombay), Shubhankar Atre (BharatGen, IIT Bombay), Samrit Kumar Maity (CDAC, India), and Somshekhar M (BharatGen).
• From Monash University: Lizhen Qu, Trang Vu, Minghan Wang, William Chien, and Reza Haffari.

The primary objective of this PoC was to work through practical details of coordinated, distributed training between autonomous data centers (sovereign nodes), with the secondary goal of beginning the exploration of instruction fine tuning (IFT) to perform cultural alignment in this geo-localized consortium training setting, using a reasonably sized LLM. The team completed the primary objective and made progress on the secondary objective, too.

After meeting the interoperability objectives, the team did a preliminary tuning experiment using the OLMo 2 7B model¹. Each sovereign node, one in India and one in Australia, tuned locally with separate, culturally-specific datasets (disjoint partitions of locally-relevant data extracted from a common dataset, CultureInstruct²). They did periodic merges of compressed LoRA weight deltas (no tuning data was exchanged). As expected, they confirmed that the local updates improved the local model’s performance on the corresponding cultural behaviors.

To evaluate the effects of cultural alignment and the effects of merging model updates from the two sites, the Jensen-Shannon Distance (JSD) was measured with the GlobalOpinionQA³ (GOQA) evaluation data set. The JSD compares the model’s option distribution with the available human response distribution; lower values are better.

For seven rounds of LoRA tuning, the base model was loaded plus each round’s LoRA adapter (without merging adapters into full model copies). The evaluation retained 1,106 questions with at least one Australia, New Zealand, or Indian human distribution.

Figure 1 shows some of the experimental results:

Figure 1. GlobalOpinionQA trajectory for the base model and retained Round 1–7 adapters.

Round 3 is the best checkpoint on the equal two-region metric and on the Australia/New Zealand component. Round 1 was narrowly best on the India component. The four checkpoints with confirmed distinct peer merges remained materially better than the base model, but none improved on Round 3. The two-region metric rose slightly from Round 3 to Round 7 while training loss continued to fall (not shown). In this short run, fitting the training data more closely therefore did not translate into monotonic improvement on GOQA.

However, these preliminary results should not be interpreted as evidence that federation harmed performance. The early rounds already contained substantial Australian local adaptation. Also, many system properties and hyperparameters impact performance and all of them need to be studied in subsequent work.

The team also concluded the following:

1. Robustness against connection instability: Local training successfully continued even after peer disconnects, establishing an important robustness requirement for geo-distributed federated learning, that progress is tolerant to network disruptions.
2. Making progress with asynchronous training: The training stack is natively asynchronous. The last obtained delta is used for further synchronization and the nodes do not wait or block for deltas to be received from the peers beyond a preset maximum delay. The successful training convergence demonstrated the efficacy of the training framework.
3. Impacts of system heterogeneity: Although asynchrony helps, system heterogeneity plays an important role in determining the overall efficiency of federated training and requires extra care when merging weights with respect to their staleness. The GPU and networking environment on the Indian side had higher capacities than on the Australian side⁴. Hence, the training over the available tokens on the Indian side completed in a much shorter time compared to the Australian side. This limited the number of updates exchanged between the peers, so that the Indian side didn’t block. Hence, the Indian side only merged in a couple of model deltas. This indicates an area of future work: can we derive theoretical upper bounds on system heterogeneity so that before actual training we can estimate the likelihood of non-exchange of learned representations and implications for overall training progress? Despite this, we still found the robustness of the training framework prevented local training from diverging too much, even under the experiment’s heterogeneity.
4. The ratio of inner vs. outer loops: As was expected, more frequent outer merges between environments improved the quality of training. Specifically, in two different experimental runs, the total number of passes over the available dataset was kept identical, but for the second run the number of weight synchronization rounds, i.e., outer loop merges, was halved while the number of inner loop steps between merges was doubled. This is the run shown in Figure 1 above. Note that the best value for the Australia-New Zealand number is about 0.40 in Figure 1. This is approximately 33% worse than the best value of approximately 0.29 that was observed in the first run with more frequent outer merges.

More details about their conclusions are in this report (PDF).

Consortium Training Using the Flower Federated Framework

This proof of concept, epic #184, tested continued pre-training under the consortium learning approach and applied it to OLMo 3 7B across two independently operated AWS GPU sites using the Flower framework. Sites in Sydney and Virginia trained on disjoint local partitions of the Dolma 3 mixture, exchanged model parameters only, and combined their work through a coordination node in Ohio. The objective was to demonstrate that geographically separated organizations could contribute to a shared training run without moving their underlying data.

Collaborators on this project include Elaine Chan (independent), Joe Olson (The AI Alliance and IBM), Nic Lane (Flower Labs), Patrick Foley (Flower Labs), and Lorenzo Sani (Flower Labs).

Model, Data, and Training Configuration

OLMo 3 7B is a decoder-only Transformer with 32 layers, a hidden dimension of 4,096, an intermediate dimension of 11,008, 32 attention heads, 32 key/value heads, and a vocabulary of 100,278 tokens. Although the model supports contexts of up to 65,536 tokens, the experiment used an 8,192-token sequence length.

Local training at each site ran with TorchTitan in BF16, using AdamW with an initial learning rate of 5 × 10⁻⁵, 100 warm-up steps followed by linear decay, FSDP across all eight GPUs, and selective activation checkpointing. Each optimizer step processed 400,000 tokens: a 10,000-token microbatch per GPU with five gradient-accumulation steps. Each site completed 7,500 local steps per round, or 3 billion tokens, before aggregation. Two rounds produced 6 billion tokens of training per site and 12 billion tokens across the consortium.

The already-tokenized Dolma 3 data remained at the training sites. Documents were assigned deterministically to disjoint partitions while approximately preserving the published mixture, terminated with an end-of-sequence token, and packed into 8,192-token sequences. After partitioning, no data was sent over the network during model training.

Deployment and Model Exchange

Both the Sydney and Virginia training sites used AWS p5.48xlarge instances with 8 × NVIDIA H100 80 GB GPUs, 192 virtual CPUs, 2 TiB of host memory, and NVSwitch connectivity. The Ohio coordination node had 256 GB of memory to receive and combine the models. Measured bandwidth between Ohio and Sydney was approximately 0.53 Gbit/s upstream and 0.54 Gbit/s downstream, with 186 ms round-trip latency. Virginia reached approximately 7.9–8.9 Gbit/s upstream and 9.7 Gbit/s downstream, with 12.5 ms latency.

The communication design was adapted to these relatively slow link conditions. An initial test transferred the model layer by layer, reducing peak memory use but requiring many separate exchanges. Distributing the model to both sites took roughly 48 minutes and achieved about 164 Mbit/s across the two transfers; the repeated coordination was particularly costly on the higher-latency Sydney route. For the final run, each site instead sent its complete 14.6 GB BF16 model to Ohio in one continuous transfer. Flower aggregation averaged the two contributions equally and returned the merged weights for the next round. This reduced latency-sensitive coordination and made round boundaries faster and more predictable. The longer local training interval (7,500 steps between exchanges) also amortized the high communication cost.

Operational Resilience and Results

A 7,500-step local phase took approximately 19.3 hours, exceeding Flower’s default 12-hour message lifetime. Both sites nevertheless completed training and produced valid distributed checkpoints. The team extended the message lifetime to seven days and resumed directly from the saved state, avoiding another full local training phase. Checkpoints written every 1,000 steps provided clear recovery points and allowed the coordination layer to be reconfigured without losing expensive training progress.

Both sites sustained approximately 44,400 tokens per second, or 5,550 tokens per second per GPU, and their training curves tracked closely. The final aggregate checkpoint was persisted and independently verified with a cross-entropy loss of 2.28053 and perplexity of 9.78186.The training process and progress of loss and cross-entropy during training is illustrated in Figure 2. This result shows training progressing as normal despite the extreme physical distances between participating sites.

Figure 2. Federated training cross-entropy loss and perplexity.

Overall, this trial demonstrated that Flower could coordinate full-parameter continued pre-training across widely separated GPU clusters while keeping training data local. It also showed that communication strategy, aggregation cadence, and checkpointing can be tuned to the infrastructure: the deployment accommodated a fifteen-fold latency difference between sites, recovered cleanly from a configuration change, and completed a reproducible two-round training run. If allowed to continue, this globally distributed topology would have reproduced the OLMo 3 7B training run after continuing pre-training from a midway starting position.

Cultural Alignment

The consortium training PoC #189 discussed above did instruction fine tuning and evaluation using data for cultural alignment, although this objective wasn’t its primary focus. Two other experiments focused on cultural alignment were performed by separate teams during M0.

Proof of Concept for Alignment Based on Inglehart-Welzel Cultural Map

This feasibility study on cultural alignment shift, Issue #22, is part of TAP-003: Cultural Alignment as the Primary Differentiator. The team used the LoRA fine-tuning with the goal of demonstrating simultaneous (a) socio-cultural alignment shift and (b) no performance loss in general capabilities (e.g., as measured by benchmarks like MMLU (see below).

Christopher Nguyen (Aitomatic) is the principal investigator, with the bulk of the work performed by William Nguyen (Aitomatic), with the assistance of Joe Olson (IBM and The AI Alliance) and Anthony Annunziata (IBM and The AI Alliance).

We summarize the results here. A research paper with more details about this work will be available soon. The code for this investigation can be found in the repository location contrib/nguyennm1024-sociocultural-alignment.

The team chose the Llama-3.2-3B-Instruct model because it is familiar and it is simple to post-train, due to its permissive license and the fact it is a dense model (not MoE - mixture of experts - which is a harder architecture to tune), etc. The longer-term model choice (issue #25) will be based in part on which options provide the lowest-resistance path towards the strategic objectives of (a) high/leading performance, while (b) affording sovereignty (national, socio-cultural, industrial). Ultimately, Project Tapestry plans to train foundation models from scratch.

For this work, a capability-rehearsal corpus was used to limit catastrophic forgetting, with the culturally-aligned and rehearsal members fused via weight-space averaging (50/50). Cultural position was measured via the Inglehart-Welzel projection method⁵ and capability was measured using MMLU⁶.

Figure 3 shows the preliminary tuning results showing a 26% improvement:

Figure 3. Tuning moves the model on the IW Cultural Map.

Figure 4 shows the final results, in a different representation. The end of the tuning experiment showed a 45% improvement:

Figure 4. Final tuning results with movement of the model on the IW Cultural Map. Table 1. Tuning results for the IW Cultural Map.

The non-significance finding is a direct quote from the Preliminary results section of the README. One model, one culture, staging-quality code — but a positive directional result on the axis TAP-003 identified as the differentiator: a measurable cultural shift with no significant capability drop.

Project Tapestry’s Novel Contribution Process

By the way, this work was provided using our novel contribution process, which allows interested parties to contribute ideas to Tapestry in a staged way that allows them to be more carefully considered by the larger collaboration and in some cases, adopted into the “main” Tapestry code base. Contributions live in a special contrib directory tree in the Tapestry repository, such as this contrib/nguyennm1024-sociocultural-alignment project.

Cultural-CPT Validation Harness

A second proof of concept contribution for cultural alignment was Cultural-CPT Validation Harness, contributed by Jesse Neumann (@jneums). It builds on an earlier contribution of his, Consortium experiment metrics, which adds a deterministic measurement layer around an early consortium-training proof of concept.

This project also pursues the Inglehart-Welzel projection method that the contrib/nguyennm1024-sociocultural-alignment project pursued, but from different perspectives. The latter was an alignment recipe that used LoRA SFT (supervised fine tuning) on synthesized data, evaluated against the Tao et al. projection. The former is a validation harness, a pre-registered, control-structured, noise-banded test of whether a measured shift is genuine, deep, and capability-safe.

This project framed its hypothesis as follows:

H1. Continued pretraining on culturally grounded data produces a shift in the model’s expressed values, measured on the Inglehart-Welzel / World Values Survey (WVS) framework, that is:

• (a) real — larger than seed/paraphrase noise;
• (b) attributable to cultural content — larger than the shift from language-matched, value-neutral data in the same language;
• (c) representational, not surface mimicry — visible in open-ended behavior, not only in survey-answering mode;
• (d) capability- and safety-preserving — does not destroy general capability or erode base-model safety.

The results of this preliminary work are discussed in FINDINGS.md. In summary, using an Arabic value-laden corpus was more effective at shifting the cultural metric than an Arabic corpus that is more value-neutral. However, for H1c, the effect observed was more superficial alignment, where better survey-answering results were seen, but they were not sufficiently deep enough to change behavior significantly. More investigation of efficacy is required to ensure cultural alignment goals are truly met.

Other Contributions

In addition to the two contributions related to cultural alignment, six more contributions explored the following topics:

1. Conflict-Aware Fusion: Training a Shared Base Model to Recognize Broken Premises) - Techniques for ensuring that models avoid making deductions with inconsistent logical premises. (Contributor: Qiming Bao (@14H034160212))
2. Logically-Grounded DPO: Answer-Grounded Preference Optimization for Explanation Generation) - Using direct preference optimization to better ensure that explanations for correct answers are accurate. (Contributor: Qiming Bao (@14H034160212))
3. Consortium Experiment Metrics) - (Mentioned above) Adds metric to the consortium training demonstration code. (Contributor: Jesse Neumann (@jneums))
4. Flower WAN Weight-Transfer Spike) - Measures the overhead of model weight exchanges between sovereign nodes running the Flower Labs stack in a semi-realistic experimental setting. (Contributor: Jesse Neumann (@jneums))
5. Tapestry Formal Specs (Quint)) - Demonstrates the use of the Quint formal specification language for defining and enforcing logical behavior specifications. (Contributor: Mykyta Luzan (@luzanikita))
6. Sovereign Evaluation Evidence Layer) - Proposes a small evidence layer for Tapestry’s evaluation and certification work. (Contributor: Mykyta Luzan (@welttowelt))

Data Governance and Management Requirements

We started defining the requirements for our data governance and management strategy (“V0.1”) and we organized work groups for these areas.

• Data governance requirements
• Data management requirements

Software Development Policies and Practices

Finally, we established our software development policies and practices following standard best practices for GitHub repositories (14 issues). Of note is our novel contribution process, which was described above.

Acknowledgements

We wish to thank all the contributors to Tapestry for M0. Besides the collaborators listed above, many people contributed code, issues, etc. to the Tapestry repository: @dean-wampler, @ctn, @jneums, @Rohithmatham12, @kb-bhatta, @luzanikita, @AnthonyJAnnunziata, @jolson-allianceai, @ThibautMelen, @Milian0402, @EC, @NovusEdge, @14H034160212, @d3v07, @adampingel, @nguyennm1024, @mzkarami, @Amertos, @andrewmusselman, @ArjunSrivastava1, @bapichatterjee, @billbrietstout, @dbckz, @elainechan, @JulienAu, @kb-bhatta, @niclane7, @psfoley, @Phaethon1, and let us not forget the ever-present @dependabot[bot] and @copilot-swe-agent[bot] .

Special thanks to Dean Wampler (IBM / AI Alliance) for coordinating contributions, releases, and leading working groups, Kaushik Bhatta (B3 Alliance / AI Alliance) for recruiting partners, sovereign nodes, and compute resources, and Agata Ferretti (IBM / AI Alliance) for work with EMEA partners. Anthony Annunziata (IBM / AI Alliance) provides overall Project Tapestry and AI Alliance leadership, Christopher Nguyen (Aitomatic) is the lead architect for Project Tapestry, and Yann Lecun (AMI Labs, Turing Award winner) provided the inspiration for Project Tapestry.

What’s Next?

Milestone One (M1) is our next objective, covering our work from September through November, 2026. Our M1 dashboard shows the work planned and our progress. Like M0, the major themes will be expanding our capabilities in these areas:

• Consortium Training
• Cultural Alignment
• Data Governance and Management

We welcome your help! See the project README for individual contributor guidance and how your organization can join Project Tapestry.

---

---