I recently worked on a pipeline to pre-process the UN Comtrade dataset before it is incorporated into the GLORIA MRIO. GLORIA (Global Resource Input-Output Assessment) is a multi-regional input-output (MRIO) database that was built by the University of Sydney using the IELab infrastructure for the UN International Resource Panel (UN IRP). GLORIA has 164 regions, 120 sectors per region, forms a continuous time series for 1990-2020 and is one of the largest and most up-to-date published MRIO databases (at the time of writing).
While the Comtrade1 is an excellent source of bilateral trade data, there are issues with this dataset that must be resolved before the data can be incorporated into a MRIO framework. These issues include:
- Comtrade contains 2 perspectives on the same trade, representing both the importer and exporter’s reports. However the values of these data points do not always agree, and in some cases one is zero and the other is not.
- UN SNA MA trade data provides an additional trade dataset, and is taken to be the point-of-truth. SNA MA details total imports and exports by country, however these totals do not agree with the combined totals from the Comtrade and Services trade databases. UN SN MA sets total import and exports by country but not bilateral trading pairs.
- Trade in certain commodities contains gaps (zero entries) in the timeseries, where such a gap is unlikely to occur in reality.
The raw UN Comtrade and Services trade data are unpacked into a sparse tensor H, with elements hivrst, and where r and s are the origin and destination countries respectively. t indexes the year dimension. Commodities and services categories are indexed by i, using the custom classification HSCPC (containing > 6,357 distinct categories). HSCPC is the union of the HS (commodities) and CPC (services) classification systems. The index v denotes valuation, where imports (v = 1) are valued as CIF (purchaser’s prices) and exports (v = 2) are valued as FOB (producer or basic prices).
A reconciliation algorithm is then implemented to resolve the errors and inconsistencies in the Comtrade database. A RAS procedure involving a series of scaling operations is used; the sparse tensor data structure makes this scaling relatively easy to implement. Smoothing is also implemented to fill zero entries.
1 The term ‘Comtrade’ is used lazily here to refer to both the UN’s Comtrade and Services trade databases.