Section 4 Datasets

4.1 Before downloading datasets

If you are going to download data, you have to read the Code of Conduct first.

4.2 API

The advantage of the API over https download is that you can filter what to obtain and also access some additional tables.

To obtain exactly the same data as with compressed files, please refer to 4.2.5.

If you use R you’ll need jsonlite and dplyr packages.

library(jsonlite)

These packages are also useful:

library(dplyr)
library(stringr)

4.2.1 Available tables

as_tibble(fromJSON("https://api.tradestatistics.io/tables"))
## # A tibble: 16 x 3
##    table      description                        source                    
##    <chr>      <chr>                              <chr>                     
##  1 countries  Countries metadata                 UN Comtrade               
##  2 products   Product metadata                   UN Comtrade               
##  3 reporters  Reporting countries                UN Comtrade               
##  4 communiti… Product communities                Center for International …
##  5 product_s… Product short names                The Observatory of Econom…
##  6 country_r… Ranking of countries               Open Trade Statistics     
##  7 product_r… Ranking of products                Open Trade Statistics     
##  8 yrpc       Bilateral trade at product level … Open Trade Statistics     
##  9 yrp        Reporter trade at aggregated leve… Open Trade Statistics     
## 10 yrp_short  Reporter trade at aggregated leve… Open Trade Statistics     
## 11 yrc        Reporter trade at aggregated leve… Open Trade Statistics     
## 12 yrc_expor… Reporter trade at aggregated leve… Open Trade Statistics     
## 13 yrc_impor… Reporter trade at aggregated leve… Open Trade Statistics     
## 14 yr         Reporter trade at aggregated leve… Open Trade Statistics     
## 15 yr_short   Reporter trade at aggregated leve… Open Trade Statistics     
## 16 yc         Product trade at aggregated level… Open Trade Statistics

4.2.2 Metadata

## Countries (no filter)
rda_countries <- "countries.rda"

if (!file.exists(rda_countries)) {
  countries <- as_tibble(fromJSON(
    "https://api.tradestatistics.io/countries"
  ))
  
  save(countries, file = rda_countries, compress = "xz")
  
  countries
} else {
  load(rda_countries)
  
  countries
}
## # A tibble: 249 x 6
##    country_iso country_name_en… country_fullnam… continent_id continent
##    <chr>       <chr>            <chr>                   <int> <chr>    
##  1 afg         Afghanistan      Afghanistan                 1 Asia     
##  2 alb         Albania          Albania                     2 Europe   
##  3 dza         Algeria          Algeria                     3 Africa   
##  4 asm         American Samoa   American Samoa              4 Oceania  
##  5 and         Andorra          Andorra                     2 Europe   
##  6 ago         Angola           Angola                      3 Africa   
##  7 aia         Anguilla         Anguilla                    5 Americas 
##  8 atg         Antigua and Bar… Antigua and Bar…            5 Americas 
##  9 arg         Argentina        Argentina                   5 Americas 
## 10 arm         Armenia          Armenia                     1 Asia     
## # … with 239 more rows, and 1 more variable: eu28_member <int>
## Products (no filter)
rda_products <- "products.rda"

if (!file.exists(rda_products)) {
  products <- as_tibble(fromJSON(
    "https://api.tradestatistics.io/products"
  ))
  
  save(products, file = rda_products, compress = "xz")
  
  products
} else {
  load(rda_products)
  
  products
}
## # A tibble: 1,320 x 4
##    product_code product_fullname_english          group_code group_name    
##    <chr>        <chr>                             <chr>      <chr>         
##  1 0101         Horses, asses, mules and hinnies… 01         Animals; live 
##  2 0102         Bovine animals; live              01         Animals; live 
##  3 0103         Swine; live                       01         Animals; live 
##  4 0104         Sheep and goats; live             01         Animals; live 
##  5 0105         Poultry; live, fowls of the spec… 01         Animals; live 
##  6 0106         Animals, n.e.c. in chapter 01; l… 01         Animals; live 
##  7 0201         Meat of bovine animals; fresh or… 02         Meat and edib…
##  8 0202         Meat of bovine animals; frozen    02         Meat and edib…
##  9 0203         Meat of swine; fresh, chilled or… 02         Meat and edib…
## 10 0204         Meat of sheep or goats; fresh, c… 02         Meat and edib…
## # … with 1,310 more rows

Please notice that these tables include some aliases.

countries includes some meta-codes, c-xx where xx must the first two letters of a continent and all, this is:

Alias Meaning
c-af Alias for all valid ISO codes in Africa
c-am Alias for all valid ISO codes in the Americas
c-as Alias for all valid ISO codes in Asia
c-eu Alias for all valid ISO codes in Europe
c-oc Alias for all valid ISO codes in Oceania
all Alias for all valid ISO codes in the World

products also includes some meta-codes, xx for the first two digits of a code and those digits are the product group and all, this is:

Alias Meaning
01 Alias for all codes in the group Animals; live
02 Alias for all codes in the group Meat and edible meat offal
03 Alias for all codes in the group Fish and crustaceans, molluscs and other aquatic invertebrates
04 Alias for all codes in the group Dairy produce; birds’ eggs; natural honey; edible products of animal origin, not elsewhere specified or included
05 Alias for all codes in the group Animal originated products; not elsewhere specified or included
06 Alias for all codes in the group Trees and other plants, live; bulbs, roots and the like; cut flowers and ornamental foliage
07 Alias for all codes in the group Vegetables and certain roots and tubers; edible
08 Alias for all codes in the group Fruit and nuts, edible; peel of citrus fruit or melons
09 Alias for all codes in the group Coffee, tea, mate and spices
10 Alias for all codes in the group Cereals
11 Alias for all codes in the group Products of the milling industry; malt, starches, inulin, wheat gluten
12 Alias for all codes in the group Oil seeds and oleaginous fruits; miscellaneous grains, seeds and fruit, industrial or medicinal plants; straw and fodder
13 Alias for all codes in the group Lac; gums, resins and other vegetable saps and extracts
14 Alias for all codes in the group Vegetable plaiting materials; vegetable products not elsewhere specified or included
15 Alias for all codes in the group Animal or vegetable fats and oils and their cleavage products; prepared animal fats; animal or vegetable waxes
16 Alias for all codes in the group Meat, fish or crustaceans, molluscs or other aquatic invertebrates; preparations thereof
17 Alias for all codes in the group Sugars and sugar confectionery
18 Alias for all codes in the group Cocoa and cocoa preparations
19 Alias for all codes in the group Preparations of cereals, flour, starch or milk; pastrycooks’ products
20 Alias for all codes in the group Preparations of vegetables, fruit, nuts or other parts of plants
21 Alias for all codes in the group Miscellaneous edible preparations
22 Alias for all codes in the group Beverages, spirits and vinegar
23 Alias for all codes in the group Food industries, residues and wastes thereof; prepared animal fodder
24 Alias for all codes in the group Tobacco and manufactured tobacco substitutes
25 Alias for all codes in the group Salt; sulphur; earths, stone; plastering materials, lime and cement
26 Alias for all codes in the group Ores, slag and ash
27 Alias for all codes in the group Mineral fuels, mineral oils and products of their distillation; bituminous substances; mineral waxes
28 Alias for all codes in the group Inorganic chemicals; organic and inorganic compounds of precious metals; of rare earth metals, of radio-active elements and of isotopes
29 Alias for all codes in the group Organic chemicals
30 Alias for all codes in the group Pharmaceutical products
31 Alias for all codes in the group Fertilizers
32 Alias for all codes in the group Tanning or dyeing extracts; tannins and their derivatives; dyes, pigments and other colouring matter; paints, varnishes; putty, other mastics; inks
33 Alias for all codes in the group Essential oils and resinoids; perfumery, cosmetic or toilet preparations
34 Alias for all codes in the group Soap, organic surface-active agents; washing, lubricating, polishing or scouring preparations; artificial or prepared waxes, candles and similar articles, modelling pastes, dental waxes and dental preparations with a basis of plaster
35 Alias for all codes in the group Albuminoidal substances; modified starches; glues; enzymes
36 Alias for all codes in the group Explosives; pyrotechnic products; matches; pyrophoric alloys; certain combustible preparations
37 Alias for all codes in the group Photographic or cinematographic goods
38 Alias for all codes in the group Chemical products n.e.c.
39 Alias for all codes in the group Plastics and articles thereof
40 Alias for all codes in the group Rubber and articles thereof
41 Alias for all codes in the group Raw hides and skins (other than furskins) and leather
42 Alias for all codes in the group Articles of leather; saddlery and harness; travel goods, handbags and similar containers; articles of animal gut (other than silk-worm gut)
43 Alias for all codes in the group Furskins and artificial fur; manufactures thereof
44 Alias for all codes in the group Wood and articles of wood; wood charcoal
45 Alias for all codes in the group Cork and articles of cork
46 Alias for all codes in the group Manufactures of straw, esparto or other plaiting materials; basketware and wickerwork
47 Alias for all codes in the group Pulp of wood or other fibrous cellulosic material; recovered (waste and scrap) paper or paperboard
48 Alias for all codes in the group Paper and paperboard; articles of paper pulp, of paper or paperboard
49 Alias for all codes in the group Printed books, newspapers, pictures and other products of the printing industry; manuscripts, typescripts and plans
50 Alias for all codes in the group Silk
51 Alias for all codes in the group Wool, fine or coarse animal hair; horsehair yarn and woven fabric
52 Alias for all codes in the group Cotton
53 Alias for all codes in the group Vegetable textile fibres; paper yarn and woven fabrics of paper yarn
54 Alias for all codes in the group Man-made filaments; strip and the like of man-made textile materials
55 Alias for all codes in the group Man-made staple fibres
56 Alias for all codes in the group Wadding, felt and nonwovens, special yarns; twine, cordage, ropes and cables and articles thereof
57 Alias for all codes in the group Carpets and other textile floor coverings
58 Alias for all codes in the group Fabrics; special woven fabrics, tufted textile fabrics, lace, tapestries, trimmings, embroidery
59 Alias for all codes in the group Textile fabrics; impregnated, coated, covered or laminated; textile articles of a kind suitable for industrial use
60 Alias for all codes in the group Fabrics; knitted or crocheted
61 Alias for all codes in the group Apparel and clothing accessories; knitted or crocheted
62 Alias for all codes in the group Apparel and clothing accessories; not knitted or crocheted
63 Alias for all codes in the group Textiles, made up articles; sets; worn clothing and worn textile articles; rags
64 Alias for all codes in the group Footwear; gaiters and the like; parts of such articles
65 Alias for all codes in the group Headgear and parts thereof
66 Alias for all codes in the group Umbrellas, sun umbrellas, walking-sticks, seat sticks, whips, riding crops; and parts thereof
67 Alias for all codes in the group Feathers and down, prepared; and articles made of feather or of down; artificial flowers; articles of human hair
68 Alias for all codes in the group Stone, plaster, cement, asbestos, mica or similar materials; articles thereof
69 Alias for all codes in the group Ceramic products
70 Alias for all codes in the group Glass and glassware
71 Alias for all codes in the group Natural, cultured pearls; precious, semi-precious stones; precious metals, metals clad with precious metal, and articles thereof; imitation jewellery; coin
72 Alias for all codes in the group Iron and steel
73 Alias for all codes in the group Iron or steel articles
74 Alias for all codes in the group Copper and articles thereof
75 Alias for all codes in the group Nickel and articles thereof
76 Alias for all codes in the group Aluminium and articles thereof
78 Alias for all codes in the group Lead and articles thereof
79 Alias for all codes in the group Zinc and articles thereof
80 Alias for all codes in the group Tin; articles thereof
81 Alias for all codes in the group Metals; n.e.c., cermets and articles thereof
82 Alias for all codes in the group Tools, implements, cutlery, spoons and forks, of base metal; parts thereof, of base metal
83 Alias for all codes in the group Metal; miscellaneous products of base metal
84 Alias for all codes in the group Nuclear reactors, boilers, machinery and mechanical appliances; parts thereof
85 Alias for all codes in the group Electrical machinery and equipment and parts thereof; sound recorders and reproducers; television image and sound recorders and reproducers, parts and accessories of such articles
86 Alias for all codes in the group Railway, tramway locomotives, rolling-stock and parts thereof; railway or tramway track fixtures and fittings and parts thereof; mechanical (including electro-mechanical) traffic signalling equipment of all kinds
87 Alias for all codes in the group Vehicles; other than railway or tramway rolling stock, and parts and accessories thereof
88 Alias for all codes in the group Aircraft, spacecraft and parts thereof
89 Alias for all codes in the group Ships, boats and floating structures
90 Alias for all codes in the group Optical, photographic, cinematographic, measuring, checking, medical or surgical instruments and apparatus; parts and accessories
91 Alias for all codes in the group Clocks and watches and parts thereof
92 Alias for all codes in the group Musical instruments; parts and accessories of such articles
93 Alias for all codes in the group Arms and ammunition; parts and accessories thereof
94 Alias for all codes in the group Furniture; bedding, mattresses, mattress supports, cushions and similar stuffed furnishings; lamps and lighting fittings, n.e.c.; illuminated signs, illuminated name-plates and the like; prefabricated buildings
95 Alias for all codes in the group Toys, games and sports requisites; parts and accessories thereof
96 Alias for all codes in the group Miscellaneous manufactured articles
97 Alias for all codes in the group Works of art; collectors’ pieces and antiques
99 Alias for all codes in the group Commodities not specified according to kind
all Alias for all codes

4.2.3 API parameters

The tables provided withing our API contain at least one of these fields:

  • Year (y)
  • Reporter ISO (r)
  • Partner ISO (p)
  • Product Code (c)

The most detailed table is yrpc that contains all bilateral flows at product level.

With respect to y you can pass any integer contained in \([1962,2018]\).

Both r and p accept any valid ISO code or alias contained in the countries table. For example, both chl (valid ISO code) and c-am (continent Americas, an alias) are valid API filtering parameters.

c takes any valid product code or alias from the products. For example, both 0101 (valid HS product code) and 01 (valid HS group code) are valid API filtering parameters.

By default the API takes c = "all" by default.

You can always skip c, but y, r and p are requiered to return data.

4.2.4 Available reporters

The only applicable filter is by year.

# Available reporters (filter by year)
as_tibble(fromJSON(
  "https://api.tradestatistics.io/reporters?y=2015"
))
## # A tibble: 225 x 1
##    reporter_iso
##    <chr>       
##  1 nru         
##  2 npl         
##  3 nor         
##  4 nld         
##  5 niu         
##  6 nic         
##  7 nga         
##  8 nfk         
##  9 ner         
## 10 ncl         
## # … with 215 more rows

4.2.5 YRPC (Year, Reporter, Partner and Product Code)

The applicable filters here are year, reporter, partner, product code and (optionally) product code length.

# Year - Reporter - Partner - Product Code (filter by year, reporter and partner)

## file to store the query results
rda_1962 <- "api_data_1962.rda"

## filter by product code length (parameter `l`)
if (!file.exists(rda_1962)) {
  yrpc_1 <- as_tibble(fromJSON(
    "https://api.tradestatistics.io/yrpc?y=1962&r=usa&p=all"
  ))
  
  save(yrpc_1, file = rda_1962, compress = "xz")
  
  yrpc_1
} else {
  load(rda_1962)
  
  yrpc_1
}
## # A tibble: 35,346 x 6
##     year reporter_iso partner_iso product_code import_value_usd
##    <int> <chr>        <chr>       <chr>                   <int>
##  1  1962 usa          zaf         7502                     2758
##  2  1962 usa          zaf         7213                    76363
##  3  1962 usa          zaf         7206                   143415
##  4  1962 usa          zaf         7201                   189295
##  5  1962 usa          zaf         7118                     6687
##  6  1962 usa          zaf         7102                 22770831
##  7  1962 usa          zaf         5105                   268857
##  8  1962 usa          zaf         5102                    48423
##  9  1962 usa          zaf         5101                 26275600
## 10  1962 usa          zaf         4702                  2224566
## # … with 35,336 more rows, and 1 more variable: export_value_usd <int>
## filter by product group (parameter `c`)
yrpc_2 <- as_tibble(fromJSON(
  "https://api.tradestatistics.io/yrpc?y=2015&r=chl&p=arg&c=01"
))

yrpc_2
## # A tibble: 2 x 6
##    year reporter_iso partner_iso product_code export_value_usd
##   <int> <chr>        <chr>       <chr>                   <int>
## 1  2015 chl          arg         0106                    53213
## 2  2015 chl          arg         0101                   123800
## # … with 1 more variable: import_value_usd <int>

Some columns requiere an explanation:

  • product_code: HS07 product codes (e.g. according to the table within this package, 0101 stands for “Horses, etc.”)
  • product_code_length: How many digits does product_code contain, this can be useful to filter by depth when using HS codes (HS 6 digits is a more detailed version of HS 4 digits, and therefore you don’t have to sum both or you’ll be counting exports/imports twice)
  • group_code: International categorization of group products defined after product ID
  • group_name: English name corresponding to group_id
  • export_value_usd: Exports measured in nominal United States Dollars (USD)
  • import_value_usd: Imports measured in nominal United States Dollars (USD)
  • export_value_usd_percentage_change_1_year: Nominal increase/decrease in exports measured as percentage with respect to last year
  • export_value_usd_percentage_change_5_years: Nominal increase/decrease in exports measured as percentage with respect to five years ago
  • export_value_usd_change_1_year: Nominal increase/decrease in exports measured in USD with respect to last year
  • export_value_usd_change_5_years: Nominal increase/decrease in exports measured in USD with respect to five years ago

4.2.6 YRC (Year, Reporter and Product Code)

The only applicable filter is by year, reporter, product code and (optionally) product code length.

# Year - Reporter - Product Code (filter by year and reporter)

## filter by reporter ISO (parameter `r`)
yrc_1 <- as_tibble(fromJSON(
  "https://api.tradestatistics.io/yrc?y=2015&r=chl"
))

yrc_1
## # A tibble: 1,214 x 7
##     year reporter_iso product_code export_value_usd import_value_usd
##    <int> <chr>        <chr>                   <dbl>            <dbl>
##  1  2015 chl          0101                  9957995          6990584
##  2  2015 chl          0102                 88938806              200
##  3  2015 chl          0103                   956016           748456
##  4  2015 chl          0104                        0            83122
##  5  2015 chl          0105                        0          4456918
##  6  2015 chl          0106                  5289994          1311775
##  7  2015 chl          0201                  7812729        736351672
##  8  2015 chl          0202                 25146879         72809378
##  9  2015 chl          0203                402815589         85212541
## 10  2015 chl          0204                 32743703           242876
## # … with 1,204 more rows, and 2 more variables: export_rca <dbl>,
## #   import_rca <dbl>
## filter by reporter alias (also parameter `r`)
yrc_2 <- as_tibble(fromJSON(
  "https://api.tradestatistics.io/yrc?y=2015&r=c-am"
))

yrc_2
## # A tibble: 47,760 x 7
##     year reporter_iso product_code export_value_usd import_value_usd
##    <int> <chr>        <chr>                   <dbl>            <dbl>
##  1  2015 vgb          0101                        0             3600
##  2  2015 ven          0101                    70213          1476722
##  3  2015 vct          0101                        0               86
##  4  2015 usa          0101                465665894        686115555
##  5  2015 ury          0101                  3770776          1502781
##  6  2015 tto          0101                    15141           628945
##  7  2015 tca          0101                        0             3250
##  8  2015 sur          0101                        0            51022
##  9  2015 slv          0101                    79722           179874
## 10  2015 pry          0101                    11397           615655
## # … with 47,750 more rows, and 2 more variables: import_rca <dbl>,
## #   export_rca <dbl>

Here the export_rca* and import_rca* fields contain the Revealed Comparative Advantage (RCA) of an exported product with respect to all the products with the same number of digits. The definition of RCA is detailed on Open Trade Statistics Documentation.

4.2.7 YRP (Year, Reporter and Partner)

The only applicable filter is by year, reporter and partner.

# Year - Reporter - Partner (filter by year, reporter and partner)
yrp <- as_tibble(fromJSON(
  "https://api.tradestatistics.io/yrp?y=2015&r=chl&p=arg"
))

4.2.8 YC (Year and Product Code)

The only applicable filter is by year, product and (optionally) product code length.

# Year - Product Code (filter by year)
yc <- as_tibble(fromJSON(
  "https://api.tradestatistics.io/yc?y=2015&c=0101"
))

Let’s explore the first rows of yr:

yc
## # A tibble: 1 x 14
##    year product_code export_value_usd import_value_usd pci_fitness_met…
##   <int> <chr>                   <dbl>            <dbl>            <dbl>
## 1  2015 0101               3480714171       3480714171            0.243
## # … with 9 more variables: pci_rank_fitness_method <int>,
## #   pci_reflections_method <dbl>, pci_rank_reflections_method <int>,
## #   pci_eigenvalues_method <dbl>, pci_rank_eigenvalues_method <int>,
## #   top_exporter_iso <chr>, top_exporter_trade_value_usd <int>,
## #   top_importer_iso <chr>, top_importer_trade_value_usd <int>

Here some fields deserve an explanation:

  • pci_4_digits_product_code: Product Complexity Index (PCI) which is detailed on Open Trade Statistics Documentation. This index is built by using just four digits product codes.
  • pci_6_digits_product_code: Similar to the previous field but built by using just six digits product codes.
  • pci_rank_4_digits_product_code: The rank of a product given its PCI (e.g. the highest PCI obtains the #1)
  • pci_rank_4_digits_product_code_delta_1_year: How many places a country increased or decreased with respect to last year

4.2.8.1 YR (Year and Reporter)

The only applicable filter is by year and reporter.

## Year - Reporter (filter by year and reporter)
yr <- as_tibble(fromJSON(
  "https://api.tradestatistics.io/yr?y=2015&r=chl"
))

Let’s explore the first rows of yr:

yr
## # A tibble: 1 x 14
##    year reporter_iso export_value_usd import_value_usd eci_fitness_met…
##   <int> <chr>                   <dbl>            <dbl>            <dbl>
## 1  2015 chl               69650286456      73742549298            0.471
## # … with 9 more variables: eci_rank_fitness_method <int>,
## #   eci_reflections_method <dbl>, eci_rank_reflections_method <int>,
## #   eci_eigenvalues_method <dbl>, eci_rank_eigenvalues_method <int>,
## #   top_export_product_code <chr>, top_export_trade_value_usd <dbl>,
## #   top_import_product_code <chr>, top_import_trade_value_usd <dbl>

Some fields here require more detail:

  • eci_4_digits_product_code: Economic Complexity Index (ECI) which is detailed on Open Trade Statistics Documentation. This index is built by using just four digits product codes.
  • eci_rank_4_digits_product_code: The rank of a country given its ECI (e.g. the highest ECI obtains the #1)
  • eci_rank_4_digits_product_code_delta_1_year: How many places a country increased or decreased with respect to last year

4.2.9 Country rankings

The only applicable filter is by year.

# Country rankings (filter by year)
country_rankings <- as_tibble(fromJSON(
  "https://api.tradestatistics.io/country_rankings?y=2015"
))

4.2.10 Product rankings

The only applicable filter is by year.

# Product rankings (filter by year)
product_rankings <- as_tibble(fromJSON(
  "https://api.tradestatistics.io/product_rankings?y=2015"
))

4.3 R Package

To ease API using, we provide an R Package. This package is a part of ROpenSci and its documentation is available on a separate pkgdown site.

4.4 Dashboard (beta)

To ease API using, we provide a Shiny Dashboard that is still under improvements.

4.5 Compressed data

4.5.1 How to use

There is a special consideration you should have with our datasets, and is that you should always read the trade values as a numeric column and the product codes as a character column.

Different R packages, and statistical software in general, have includeded functions to autodetect column types. In our experience, that can read product codes as integers and that would ignore leading zeroes in product codes. The same applies to trade values that can be detected as integers after the program reads the first \(n\) rows, and that would lead to read large values incorrectly due to integer class maximum value of 2,147,483,647.

As an example, let’s read 1962 data and explore what the United Stated exported:

# packages

library(data.table)
library(dplyr)
library(stringr)
library(janitor)

# custom functions

messageline <- function() {
  message(rep("-", 60))
}

fread2 <- function(file, select = NULL, character = NULL, numeric = NULL) {
  messageline()
  message("function fread2")
  message("file: ", file)
  
  if(str_sub(file, start = -2) == "gz") {
    d <- fread(
      cmd = paste("zcat", file),
      select = select,
      colClasses = list(
        character = character,
        numeric = numeric
      )
    ) %>%
      as_tibble() %>%
      clean_names()
  } else {
    d <- fread(
      input = file,
      select = select,
      colClasses = list(
        character = character,
        numeric = numeric
      )
    ) %>%
      as_tibble() %>%
      clean_names()
  }
  
  return(d)
}

# download data

url_1962 <- "https://data.tradestatistics.io/06-tables/hs-rev2007/1-yrpc/yrpc-1962.csv.gz"
gz_1962 <- "yrpc-1962.csv.gz"

if (!file.exists(gz_1962)) {
  try(download.file(url_1962, gz_1962))
}

# read data

# same filename as in the YRPC example from the API section (it's the same data)
rda_1962 <- "api_data_1962.rda"

if (!file.exists(rda_1962)) {
  yrpc_1 <- fread2(
        gz_1962,
        character = "product_code",
        numeric = c(
          "export_value_usd",
          "import_value_usd"
        )
      ) %>% 
    filter(reporter_iso == "usa")
} else {
  load(rda_1962)
}

yrpc_1
## # A tibble: 35,346 x 6
##     year reporter_iso partner_iso product_code import_value_usd
##    <int> <chr>        <chr>       <chr>                   <int>
##  1  1962 usa          zaf         7502                     2758
##  2  1962 usa          zaf         7213                    76363
##  3  1962 usa          zaf         7206                   143415
##  4  1962 usa          zaf         7201                   189295
##  5  1962 usa          zaf         7118                     6687
##  6  1962 usa          zaf         7102                 22770831
##  7  1962 usa          zaf         5105                   268857
##  8  1962 usa          zaf         5102                    48423
##  9  1962 usa          zaf         5101                 26275600
## 10  1962 usa          zaf         4702                  2224566
## # … with 35,336 more rows, and 1 more variable: export_value_usd <int>

4.5.2 Available datasets

Please check the md5sums to verify data integrity after downloading.

4.5.2.1 Metadata

4.5.2.2 Yearly datasets

Year Last updated File size (MB) MD5 sum
1962 2019-09-18 5.30 17d95791e8841c9b4f45fcacbfff447e
1963 2019-09-18 6.30 af4b84e896c819de451965f6856cf746
1964 2019-09-18 6.81 d4c935b30653e992724c78be05a83bf8
1965 2019-09-18 7.50 64581a7ae6855595b1e0036ca81a2ecd
1966 2019-09-18 7.87 ffc83d81b3e32483409d6cf1bf0c7cb6
1967 2019-09-18 8.16 66b0c94c4ff0feda38396e3209c13cd9
1968 2019-09-18 8.45 2cd60c91860f216a5e266791e1b9e4bd
1969 2019-09-18 8.82 93e8f86ab5265e52cddcdb02a9c1ff8d
1970 2019-09-18 9.65 5769076f44ce8ff33a2fe06efdef29b2
1971 2019-09-18 9.93 147347a714336aa0d9c205515bf02761
1972 2019-09-18 10.41 8edca4fab2fada65bc5d043bf13a66aa
1973 2019-09-18 11.11 ea390ba70d244b2f4ffaacee9810f162
1974 2019-09-18 11.94 347cffe33efd6e0dd33bf1d88d3594d7
1975 2019-09-18 12.11 0177b8148da8f8632e5f35a11198dcea
1976 2019-09-18 12.01 3f2ba67046fbf0817cfbcf06cd6bd1bc
1977 2019-09-18 12.63 55d8fdb9a825a3fe5121e58b0b7792c0
1978 2019-09-18 13.39 c104ada85813c8f0c431e5c117168641
1979 2019-09-18 14.02 b7bf023148a6123fdc9a5dad689fb3a5
1980 2019-09-18 14.63 64c3fd6c8aff07772cd6a48ab53ac54a
1981 2019-09-18 14.79 37e1e3b7daadc121138b0a5bb52913e0
1982 2019-09-18 14.61 93d05ab34fcbcaa298a6fa82b22beb6c
1983 2019-09-18 14.74 5ff9584270c4626783916afc64287e0d
1984 2019-09-18 14.57 77540f64a50f18c74b4c96d795b29c9e
1985 2019-09-18 15.08 bf68dc6b46b360d998659cbbdd6b4ee6
1986 2019-09-18 15.68 e8611596cee97f9931e6100c61e009b4
1987 2019-09-18 16.11 c86d0c2b28025c0874e3a60445e02331
1988 2019-09-18 16.82 a53a0f2f7a46f5cf6655a0fbdb9534fd
1989 2019-09-18 17.53 6444a8aa8681a85fc02387342385fb61
1990 2019-09-18 18.22 ba329d6ad8a20fab824e4053ebe766c4
1991 2019-09-18 18.28 e1553975076c5748f237edde8edae955
1992 2019-09-18 18.39 1dbdb0a6eff0760ac8d1c4e03326e7ad
1993 2019-09-18 19.50 71f6edead95bb15573478e76e38fd856
1994 2019-09-18 21.45 4930b7c4a62d47b088bbe2a37a5c7deb
1995 2019-09-18 22.93 4083ef7577883450b6d5979ae2de4bc8
1996 2019-09-18 22.06 a7faa3266a521520f2fae7e2799e54ce
1997 2019-09-18 23.95 e320a8a8f1558337ef7fb10a4e060a7e
1998 2019-09-18 24.80 f940c8dacaf6e2c80015e62b25202619
1999 2019-09-18 26.06 468ffa5658660c123e23b26d5fb9b98d
2000 2019-09-18 30.39 7f331160d221937ff80f82b68d337ef4
2001 2019-09-18 31.58 02bef28bf048054f26bb33b59125c969
2002 2019-09-18 29.80 f6ff6c9ed3e5e711067cce22d4b8b2c5
2003 2019-09-18 31.74 dee9f5956cd9f28226acfd69b170167a
2004 2019-09-18 33.99 26945ca56004b4e34e8c97d76849f226
2005 2019-09-18 35.79 05dcbdc47cacbaba6681e346cef4a88d
2006 2019-09-18 37.43 f5548ef80697fd10b830fbf6b2d4280f
2007 2019-09-18 35.32 025bdb14aa2811eee6c96070dbaca3a6
2008 2019-09-18 37.98 aa3045d722a7a88e2ed8374b07472c34
2009 2019-09-18 38.95 a13e167b11dffa40965e3087ad21027a
2010 2019-09-18 40.35 8f735656b30580a0a92208bbc6efc489
2011 2019-09-18 41.34 25f77ee85d98102bd8565da856289360
2012 2019-09-18 42.18 e726d696e98cc9dd43f2ec4e5eb1c881
2013 2019-09-18 43.30 e8c0ec2c06808e8c63026feefd704b61
2014 2019-09-18 43.61 d24fcca4d250ad6bed5cc95743896e33
2015 2019-09-18 44.01 37828dddba43212cc82edd660b9fb2c5
2016 2019-09-18 44.09 d9fa180aa3e2b5ee769fbc5ac47efec6
2017 2019-09-18 44.33 93009120f11f7fc3ae75faad716a5106
2018 2019-09-18 41.45 412bcdb29e146c8cb404116045a7b511

4.5.3 Note for Windows users

If you use Windows the scripts will only use a single core because we used a parallelization that depends on fork system call that is only supported on Unix systems. You can always run the scripts on Windows and the only difference will be that it will use less RAM and processor, and it will be slower to compute.

Also, before running the scripts on Windows verify that you installed GNU Utilities beforehand. One easy option is to install Chocolatey first and then install the GNU Utilities by running choco install unxutils on Cmd or Power Shell as administrator.

4.5.4 Note for spreadsheets users

If you plan to use the datasets with a software such as Microsoft Excel, Libre Office Calc or similar, please use 7-zip to decompress the datasets. Also, be sure that your software reads the product_code column as character or it will trim codes starting with zero.