Project Name
High-Performance Cross-Instance Asset and Filestore Synchronisation in Odoo Community
Our client operates a highly active enterprise business environment running Odoo 13 as their primary ERP and PLM platform, with a large and complex filestore accumulation including design documents, product engineering attachments, and multi-relational PLM records linked to specific product variants. The organisation required a clean migration from their legacy Odoo 13 host to a modern Odoo 18 Cloud deployment, preserving not just the binary file content but the full relational structure linking documents back to their associated records across modules.
The standard migration tooling could not handle the combination of filestore volume, schema field identifier changes between versions, and PLM relational complexity without producing data loss or requiring manual reconciliation at scale.
The migration required a controlled approach to overcome these architectural, schema, data transfer, and continuity challenges at scale.
- Decoupled Asset Architecture – Metadata and Binary Content in Separate Layers: Odoo stores transactional metadata in the ir.attachment table while binary files reside separately in a SHA-1-based filestore. Standard database extracts transfer metadata but leave physical files behind, making complete asset migration impossible without addressing both layers independently.
- Schema Field Identifier Changes Across Versions: Odoo 13 to Odoo 18 introduced schema changes that altered custom relational field identifiers. Legacy pointers such as x_oz_tsk_208 had to be dynamically mapped to target fields such as x_oz_tsk_683 to ensure migrated records referenced valid v18 fields.
- Network Protocol and API Limits Preventing Standard Transfer: Transferring large binary volumes through XML-RPC or JSON-RPC created memory overhead, Nginx 504 timeouts, and worker instability. The API-based approach was therefore unsuitable for a production-scale filestore migration.
- PLM Document Relational Mapping Complexity: PLM documents and engineering attachments were linked to specific Product variants through complex relationships that had to be reconstructed in the target environment. Without controlled relational mapping, files could migrate while losing their operational associations.
- Resumability and Idempotency Under Network Disruption: Large binary migrations cannot rely on uninterrupted network connections. Without processing checkpoints, disruptions could cause duplication, force restarts, or leave the migration in an uncertain partial state requiring manual verification.
Ksolves, an AI-first Odoo development company, implemented a decoupled data pipeline with four sequenced operations, including idempotency control, metadata replication, SFTP binary streaming, and relational reconstruction, enabling fast, reliable, and resumable migration.
- Idempotency Control via older_attachment_id Tracker: A tracking field was added to the v18 ir.attachment model to store each source record's ID. This allows the pipeline to identify processed records, safely resume after disruptions, and prevent duplication.
- Direct Metadata Replication via psycopg2 Batch Processing: The pipeline connects directly to the source database using psycopg2, processes records in batches of 100, dynamically maps legacy field identifiers to v18 equivalents, and creates metadata through fast SQL inserts.
- SFTP Binary Streaming via Paramiko: After metadata creation, the pipeline resolves each filestore path using its checksum, connects to the legacy filesystem through authenticated SFTP using Paramiko, streams the binary content, and writes it directly to the target attachment, bypassing RPC limitations.
- Relational Reconstruction via JSON Control Mapping: A JSON control mapping defines cross-module relationships between PLM documents and Product variants. The pipeline resolves migrated attachment references and restores these relationships using Odoo's relation field syntax, preserving the documents' operational context.
Technology Stack
| Category | Technology |
|---|---|
| Platform | Odoo 13 → Odoo 18 |
| Database | psycopg2 |
| File Transfer | Paramiko (SFTP / SSH) |
| Encoding | Python Base64 |
| Relational Mapping | JSON Control Mapping Dictionary |
| Architecture | Idempotency Tracker (older_attachment_id) |
From a migration architecture constrained by filestore volume, schema changes, and API limitations to a direct binary pipeline that delivered data integrity with zero application disruption.
- 100% Data Integrity Across Thousands of Engineering Records: psycopg2 validation and SFTP verification successfully re-linked thousands of engineering records, preserving binary assets and their relationships with Product variants and PLM records without data corruption.
- Zero Application Timeouts – API Layer Bypassed: Direct psycopg2 connections and Paramiko SFTP streaming bypassed XML-RPC and JSON-RPC, eliminating Nginx 504 timeouts, memory overhead, and application worker instability.
- Predictable Resource Usage Through Binary Streaming: SFTP-based binary streaming avoided large JSON serialization and RAM spikes, keeping resource consumption stable regardless of file size or migration batch volume.
- Resilient Network Handling – Safe Resumption Without Duplication: The older_attachment_id tracker maintained migration state across network disruptions, enabling the pipeline to resume from the interruption point without duplicating processed records.
- PLM Relational Structure Fully Reconstructed: JSON-based control mapping restored document-to-Product variant relationships and design document references in Odoo 18, ensuring migrated assets retained their operational context.
Ksolves transformed a complex Odoo 13-to-Odoo 18 filestore migration into a reliable, direct, and resumable pipeline. By combining direct database replication, SFTP-based binary streaming, dynamic field mapping, idempotency controls, and PLM relationship reconstruction, the migration achieved 100% data integrity across thousands of engineering records with zero missing assets, zero application timeouts, and fully preserved relational context.
Is Your Odoo Migration Plan Still Relying on Standard API Transfer for Large Binary Filestores?
