10 Validating PX-files via API

10.1 The `pxweb` package

The pxweb package has functions to work with PX-web APIs. It works with any PX-web database that has the API feature enabled. Below we collect a table from the Ghanaian StatsBank.

library(pxweb)
library(tidyverse)

from_api <- pxweb_get(url = "https://statsbank.statsghana.gov.gh:443/api/v1/en/PHC 2021 StatsBank/Education and Literacy/attended_table.px",
                      query = list(
                        Highest_level_of_education = c("*"),
                        Geographic_Area = c("Ghana"),
                        Locality = c("*"),
                        Sex = c("*"),
                        Age = c("All ages", "15-17 years", "18 years & older"))) %>% 
  as.data.frame() %>% 
  as_tibble() # as_tibble for nicer presentation

head(from_api) %>% 
  kableExtra::kable()

Highest_level_of_education	Geographic_Area	Locality	Sex	Age	Population (15 years and older)
Total	Ghana	All Locality Types	Both sexes	All ages	11289655
Total	Ghana	All Locality Types	Both sexes	15-17 years	187272
Total	Ghana	All Locality Types	Both sexes	18 years & older	11102383
Total	Ghana	All Locality Types	Male	All ages	5834212
Total	Ghana	All Locality Types	Male	15-17 years	88760
Total	Ghana	All Locality Types	Male	18 years & older	5745452

If we want to update this table with data for a new year, we can use data from the API to validate that our data for the new year looks as expected. Now we create some new random data from the dataset for a new year.

generate random data

set.seed(42); new_year <- from_api %>% 
  mutate(`Population (15 years and older)_new_year` = sample(`Population (15 years and older)`),
         .keep = "unused")

Now we can join the data together and check for developments.

pct_change <- from_api %>% 
  left_join(new_year,
            by = join_by(Highest_level_of_education, 
                         Geographic_Area, Locality, 
                         Sex, Age)) %>% 
  mutate(pct_change = (`Population (15 years and older)_new_year` - `Population (15 years and older)`)/
           `Population (15 years and older)`*100)

Now we can use this new column pct_change to see if we have some unexpectedly high changes (above 10%), which might indicate problems in our code generating the data.

if(max(abs(pct_change$pct_change), na.rm = TRUE) > 10){
    warning("There are changes above 10% for the new year. This may indicate problems with the data")
  print(pct_change %>% 
          filter(abs(pct_change) > 10)) %>% 
    kableExtra::kable()
} else {
  cat("No changes above 10%")
}

Warning: There are changes above 10% for the new year. This may indicate
problems with the data

# A tibble: 445 × 8
   Highest_level_of_education Geographic_Area Locality           Sex       Age  
   <chr>                      <chr>           <chr>              <chr>     <chr>
 1 Total                      Ghana           All Locality Types Both sex… All …
 2 Total                      Ghana           All Locality Types Both sex… 15-1…
 3 Total                      Ghana           All Locality Types Both sex… 18 y…
 4 Total                      Ghana           All Locality Types Male      All …
 5 Total                      Ghana           All Locality Types Male      15-1…
 6 Total                      Ghana           All Locality Types Male      18 y…
 7 Total                      Ghana           All Locality Types Female    All …
 8 Total                      Ghana           All Locality Types Female    15-1…
 9 Total                      Ghana           All Locality Types Female    18 y…
10 Total                      Ghana           Rural              Both sex… All …
# ℹ 435 more rows
# ℹ 3 more variables: `Population (15 years and older)` <dbl>,
#   `Population (15 years and older)_new_year` <dbl>, pct_change <dbl>

Highest_level_of_education	Geographic_Area	Locality	Sex	Age	Population (15 years and older)	Population (15 years and older)_new_year	pct_change
Total	Ghana	All Locality Types	Both sexes	All ages	11289655	569	-9.999496e+01
Total	Ghana	All Locality Types	Both sexes	15-17 years	187272	153601	-1.797973e+01
Total	Ghana	All Locality Types	Both sexes	18 years & older	11102383	173400	-9.843817e+01
Total	Ghana	All Locality Types	Male	All ages	5834212	195	-9.999666e+01
Total	Ghana	All Locality Types	Male	15-17 years	88760	49698	-4.400856e+01
Total	Ghana	All Locality Types	Male	18 years & older	5745452	0	-1.000000e+02
Total	Ghana	All Locality Types	Female	All ages	5455443	19340	-9.964549e+01
Total	Ghana	All Locality Types	Female	15-17 years	98512	4527	-9.540462e+01
Total	Ghana	All Locality Types	Female	18 years & older	5356931	43518	-9.918763e+01
Total	Ghana	Rural	Both sexes	All ages	3876215	184216	-9.524753e+01
Total	Ghana	Rural	Both sexes	15-17 years	100607	3707120	3.584754e+03
Total	Ghana	Rural	Both sexes	18 years & older	3775608	815026	-7.841338e+01
Total	Ghana	Rural	Male	All ages	2087334	0	-1.000000e+02
Total	Ghana	Rural	Male	15-17 years	49002	39338	-1.972164e+01
Total	Ghana	Rural	Male	18 years & older	2038332	2487408	2.203154e+01
Total	Ghana	Rural	Female	All ages	1788881	88854	-9.503298e+01
Total	Ghana	Rural	Female	15-17 years	51605	86665	6.793915e+01
Total	Ghana	Rural	Female	18 years & older	1737276	0	-1.000000e+02
Total	Ghana	Urban	Both sexes	All ages	7413440	117606	-9.841361e+01
Total	Ghana	Urban	Both sexes	15-17 years	86665	16277	-8.121849e+01
Total	Ghana	Urban	Both sexes	18 years & older	7326775	47526	-9.935134e+01
Total	Ghana	Urban	Male	All ages	3746878	96574	-9.742255e+01
Total	Ghana	Urban	Male	15-17 years	39758	0	-1.000000e+02
Total	Ghana	Urban	Male	18 years & older	3707120	43388	-9.882960e+01
Total	Ghana	Urban	Female	All ages	3666562	2045	-9.994423e+01
Total	Ghana	Urban	Female	15-17 years	46907	3618744	7.614721e+03
Total	Ghana	Urban	Female	18 years & older	3619655	88760	-9.754783e+01
Nursery	Ghana	All Locality Types	Both sexes	All ages	3391	140	-9.587142e+01
Nursery	Ghana	All Locality Types	Both sexes	15-17 years	143	2045	1.330070e+03
Nursery	Ghana	All Locality Types	Both sexes	18 years & older	3248	425714	1.300696e+04
Nursery	Ghana	All Locality Types	Male	All ages	1161	78357	6.649096e+03
Nursery	Ghana	All Locality Types	Male	15-17 years	68	8271	1.206324e+04
Nursery	Ghana	All Locality Types	Male	18 years & older	1093	1	-9.990851e+01
Nursery	Ghana	All Locality Types	Female	All ages	2230	298162	1.327049e+04
Nursery	Ghana	All Locality Types	Female	15-17 years	75	0	-1.000000e+02
Nursery	Ghana	All Locality Types	Female	18 years & older	2155	0	-1.000000e+02
Nursery	Ghana	Rural	Both sexes	All ages	1530	18	-9.882353e+01
Nursery	Ghana	Rural	Both sexes	15-17 years	91	133700	1.468231e+05
Nursery	Ghana	Rural	Both sexes	18 years & older	1439	19290	1.240514e+03
Nursery	Ghana	Rural	Male	All ages	592	13075	2.108615e+03
Nursery	Ghana	Rural	Male	15-17 years	43	1162448	2.703267e+06
Nursery	Ghana	Rural	Male	18 years & older	549	83163	1.504809e+04
Nursery	Ghana	Rural	Female	All ages	938	99756	1.053497e+04
Nursery	Ghana	Rural	Female	15-17 years	48	6164	1.274167e+04
Nursery	Ghana	Rural	Female	18 years & older	890	0	-1.000000e+02
Nursery	Ghana	Urban	Both sexes	All ages	1861	3756	1.018270e+02
Nursery	Ghana	Urban	Both sexes	15-17 years	52	201	2.865385e+02
Nursery	Ghana	Urban	Both sexes	18 years & older	1809	5834212	3.224103e+05
Nursery	Ghana	Urban	Male	All ages	569	49729	8.639719e+03
Nursery	Ghana	Urban	Male	15-17 years	25	77155	3.085200e+05
Nursery	Ghana	Urban	Male	18 years & older	544	141	-7.408088e+01
Nursery	Ghana	Urban	Female	All ages	1292	12	-9.907121e+01
Nursery	Ghana	Urban	Female	15-17 years	27	1670667	6.187556e+06
Nursery	Ghana	Urban	Female	18 years & older	1265	54328	4.194704e+03
Kindergarten	Ghana	All Locality Types	Both sexes	All ages	8552	16487	9.278531e+01
Kindergarten	Ghana	All Locality Types	Both sexes	15-17 years	653	972799	1.488738e+05
Kindergarten	Ghana	All Locality Types	Both sexes	18 years & older	7899	117606	1.388872e+03
Kindergarten	Ghana	All Locality Types	Male	All ages	3305	11102383	3.358269e+05
Kindergarten	Ghana	All Locality Types	Male	15-17 years	329	0	-1.000000e+02
Kindergarten	Ghana	All Locality Types	Male	18 years & older	2976	11027	2.705309e+02
Kindergarten	Ghana	All Locality Types	Female	All ages	5247	78357	1.393368e+03
Kindergarten	Ghana	All Locality Types	Female	15-17 years	324	983163	3.033454e+05
Kindergarten	Ghana	All Locality Types	Female	18 years & older	4923	1162448	2.351259e+04
Kindergarten	Ghana	Rural	Both sexes	All ages	4735	592	-8.749736e+01
Kindergarten	Ghana	Rural	Both sexes	15-17 years	458	13734	2.898690e+03
Kindergarten	Ghana	Rural	Both sexes	18 years & older	4277	1093	-7.444470e+01
Kindergarten	Ghana	Rural	Male	All ages	1915	267222	1.385415e+04
Kindergarten	Ghana	Rural	Male	15-17 years	230	0	-1.000000e+02
Kindergarten	Ghana	Rural	Male	18 years & older	1685	2459	4.593472e+01
Kindergarten	Ghana	Rural	Female	All ages	2820	50	-9.822695e+01
Kindergarten	Ghana	Rural	Female	15-17 years	228	360619	1.580662e+05
Kindergarten	Ghana	Rural	Female	18 years & older	2592	1390	-4.637346e+01
Kindergarten	Ghana	Urban	Both sexes	All ages	3817	26156	5.852502e+02
Kindergarten	Ghana	Urban	Both sexes	15-17 years	195	75	-6.153846e+01
Kindergarten	Ghana	Urban	Both sexes	18 years & older	3622	96	-9.734953e+01
Kindergarten	Ghana	Urban	Male	All ages	1390	1788881	1.285965e+05
Kindergarten	Ghana	Urban	Male	15-17 years	99	77155	7.783434e+04
Kindergarten	Ghana	Urban	Male	18 years & older	1291	138112	1.059806e+04
Kindergarten	Ghana	Urban	Female	All ages	2427	7	-9.971158e+01
Kindergarten	Ghana	Urban	Female	15-17 years	96	815102	8.489646e+05
Kindergarten	Ghana	Urban	Female	18 years & older	2331	1491460	6.388370e+04
Primary	Ghana	All Locality Types	Both sexes	All ages	1504172	0	-1.000000e+02
Primary	Ghana	All Locality Types	Both sexes	15-17 years	80993	1504172	1.757163e+03
Primary	Ghana	All Locality Types	Both sexes	18 years & older	1423179	16277	-9.885629e+01
Primary	Ghana	All Locality Types	Male	All ages	628895	0	-1.000000e+02
Primary	Ghana	All Locality Types	Male	15-17 years	41655	3043	-9.269475e+01
Primary	Ghana	All Locality Types	Male	18 years & older	587240	13075	-9.777348e+01
Primary	Ghana	All Locality Types	Female	All ages	875277	7	-9.999920e+01
Primary	Ghana	All Locality Types	Female	15-17 years	39338	0	-1.000000e+02
Primary	Ghana	All Locality Types	Female	18 years & older	835939	7899	-9.905507e+01
Primary	Ghana	Rural	Both sexes	All ages	794395	709777	-1.065188e+01
Primary	Ghana	Rural	Both sexes	15-17 years	50031	16831	-6.635886e+01
Primary	Ghana	Rural	Male	All ages	361673	5	-9.999862e+01
Primary	Ghana	Rural	Male	15-17 years	27029	1846980	6.733327e+03
Primary	Ghana	Rural	Male	18 years & older	334644	1265	-9.962199e+01
Primary	Ghana	Rural	Female	All ages	432722	84082	-8.056905e+01
Primary	Ghana	Rural	Female	15-17 years	23002	29147	2.671507e+01
Primary	Ghana	Rural	Female	18 years & older	409720	338736	-1.732500e+01
Primary	Ghana	Urban	Both sexes	All ages	709777	73614	-8.962857e+01
Primary	Ghana	Urban	Both sexes	15-17 years	30962	2976	-9.038822e+01
Primary	Ghana	Urban	Both sexes	18 years & older	678815	43382	-9.360916e+01
Primary	Ghana	Urban	Male	All ages	267222	84087	-6.853291e+01
Primary	Ghana	Urban	Male	15-17 years	14626	426219	2.814119e+03
Primary	Ghana	Urban	Male	18 years & older	252596	3	-9.999881e+01
Primary	Ghana	Urban	Female	All ages	442555	729319	6.479737e+01
Primary	Ghana	Urban	Female	15-17 years	16336	1708944	1.036121e+04
Primary	Ghana	Urban	Female	18 years & older	426219	2592	-9.939186e+01
JSS/JHS	Ghana	All Locality Types	Both sexes	All ages	3618744	0	-1.000000e+02
JSS/JHS	Ghana	All Locality Types	Both sexes	15-17 years	88854	11289655	1.260585e+04
JSS/JHS	Ghana	All Locality Types	Both sexes	18 years & older	3529890	641563	-8.182484e+01
JSS/JHS	Ghana	All Locality Types	Male	All ages	1708944	60	-9.999649e+01
JSS/JHS	Ghana	All Locality Types	Male	15-17 years	38277	0	-1.000000e+02
JSS/JHS	Ghana	All Locality Types	Male	18 years & older	1670667	414	-9.997522e+01
JSS/JHS	Ghana	All Locality Types	Female	All ages	1909800	549	-9.997125e+01
JSS/JHS	Ghana	All Locality Types	Female	15-17 years	50577	694442	1.273039e+03
JSS/JHS	Ghana	All Locality Types	Female	18 years & older	1859223	150187	-9.192206e+01
JSS/JHS	Ghana	Rural	Both sexes	All ages	1491460	5	-9.999966e+01
JSS/JHS	Ghana	Rural	Both sexes	15-17 years	45336	0	-1.000000e+02
JSS/JHS	Ghana	Rural	Both sexes	18 years & older	1446124	3666562	1.535441e+02
JSS/JHS	Ghana	Rural	Male	All ages	736145	404	-9.994512e+01
JSS/JHS	Ghana	Rural	Male	15-17 years	19340	68	-9.964840e+01
JSS/JHS	Ghana	Rural	Male	18 years & older	716805	0	-1.000000e+02
JSS/JHS	Ghana	Rural	Female	All ages	755315	108098	-8.568836e+01
JSS/JHS	Ghana	Rural	Female	15-17 years	25996	49002	8.849823e+01
JSS/JHS	Ghana	Rural	Female	18 years & older	729319	3529890	3.839981e+02
JSS/JHS	Ghana	Urban	Both sexes	All ages	2127284	414	-9.998054e+01
JSS/JHS	Ghana	Urban	Both sexes	15-17 years	43518	425759	8.783515e+02
JSS/JHS	Ghana	Urban	Both sexes	18 years & older	2083766	37906	-9.818089e+01
JSS/JHS	Ghana	Urban	Male	All ages	972799	55526	-9.429214e+01
JSS/JHS	Ghana	Urban	Male	15-17 years	18937	68	-9.964091e+01
JSS/JHS	Ghana	Urban	Male	18 years & older	953862	2	-9.999979e+01
JSS/JHS	Ghana	Urban	Female	All ages	1154485	91	-9.999212e+01
JSS/JHS	Ghana	Urban	Female	15-17 years	24581	305230	1.141731e+03
JSS/JHS	Ghana	Urban	Female	18 years & older	1129904	145128	-8.715572e+01
Middle	Ghana	All Locality Types	Both sexes	All ages	1162448	83147	-9.284725e+01
Middle	Ghana	All Locality Types	Both sexes	15-17 years	0	27029	Inf
Middle	Ghana	All Locality Types	Both sexes	18 years & older	1162448	5745452	3.942545e+02
Middle	Ghana	All Locality Types	Male	All ages	641563	36438	-9.432043e+01
Middle	Ghana	All Locality Types	Male	15-17 years	0	252281	Inf
Middle	Ghana	All Locality Types	Male	18 years & older	641563	2230	-9.965241e+01
Middle	Ghana	All Locality Types	Female	All ages	520885	5618	-9.892145e+01
Middle	Ghana	All Locality Types	Female	18 years & older	520885	28172	-9.459151e+01
Middle	Ghana	Rural	Both sexes	All ages	454344	10691	-9.764694e+01
Middle	Ghana	Rural	Both sexes	15-17 years	0	43	Inf
Middle	Ghana	Rural	Both sexes	18 years & older	454344	265804	-4.149719e+01
Middle	Ghana	Rural	Male	All ages	280944	4277	-9.847763e+01
Middle	Ghana	Rural	Male	18 years & older	280944	23002	-9.181260e+01
Middle	Ghana	Rural	Female	All ages	173400	36505	-7.894752e+01
Middle	Ghana	Rural	Female	15-17 years	0	16336	Inf
Middle	Ghana	Rural	Female	18 years & older	173400	8	-9.999539e+01
Middle	Ghana	Urban	Both sexes	All ages	708104	347485	-5.092741e+01
Middle	Ghana	Urban	Both sexes	15-17 years	0	755315	Inf
Middle	Ghana	Urban	Both sexes	18 years & older	708104	203	-9.997133e+01
Middle	Ghana	Urban	Male	All ages	360619	5	-9.999861e+01
Middle	Ghana	Urban	Male	15-17 years	0	5	Inf
Middle	Ghana	Urban	Male	18 years & older	360619	5763	-9.840191e+01
Middle	Ghana	Urban	Female	All ages	347485	6	-9.999827e+01
Middle	Ghana	Urban	Female	15-17 years	0	150187	Inf
Middle	Ghana	Urban	Female	18 years & older	347485	1530	-9.955969e+01
SSS/SHS	Ghana	All Locality Types	Both sexes	All ages	2503309	133700	-9.465907e+01
SSS/SHS	Ghana	All Locality Types	Both sexes	15-17 years	15901	8271	-4.798440e+01
SSS/SHS	Ghana	All Locality Types	Both sexes	18 years & older	2487408	0	-1.000000e+02
SSS/SHS	Ghana	All Locality Types	Male	All ages	1385416	1291	-9.990681e+01
SSS/SHS	Ghana	All Locality Types	Male	15-17 years	8041	44788	4.569954e+02
SSS/SHS	Ghana	All Locality Types	Male	18 years & older	1377375	708104	-4.859033e+01
SSS/SHS	Ghana	All Locality Types	Female	All ages	1117893	432722	-6.129129e+01
SSS/SHS	Ghana	All Locality Types	Female	15-17 years	7860	30014	2.818575e+02
SSS/SHS	Ghana	All Locality Types	Female	18 years & older	1110033	11381	-9.897472e+01
SSS/SHS	Ghana	Rural	Both sexes	All ages	656329	2127284	2.241185e+02
SSS/SHS	Ghana	Rural	Both sexes	15-17 years	4520	509846	1.117978e+04
SSS/SHS	Ghana	Rural	Both sexes	18 years & older	651809	938	-9.985609e+01
SSS/SHS	Ghana	Rural	Male	All ages	396490	257597	-3.503064e+01
SSS/SHS	Ghana	Rural	Male	15-17 years	2278	3619655	1.587962e+05
SSS/SHS	Ghana	Rural	Male	18 years & older	394212	3248	-9.917608e+01
SSS/SHS	Ghana	Rural	Female	15-17 years	2242	3756	6.752899e+01
SSS/SHS	Ghana	Rural	Female	18 years & older	257597	3622	-9.859393e+01
SSS/SHS	Ghana	Urban	Both sexes	All ages	1846980	106460	-9.423600e+01
SSS/SHS	Ghana	Urban	Both sexes	15-17 years	11381	324	-9.715315e+01
SSS/SHS	Ghana	Urban	Both sexes	18 years & older	1835599	21629	-9.882169e+01
SSS/SHS	Ghana	Urban	Male	All ages	988926	6	-9.999939e+01
SSS/SHS	Ghana	Urban	Male	15-17 years	5763	4923	-1.457574e+01
SSS/SHS	Ghana	Urban	Male	18 years & older	983163	0	-1.000000e+02
SSS/SHS	Ghana	Urban	Female	All ages	858054	5247	-9.938850e+01
SSS/SHS	Ghana	Urban	Female	15-17 years	5618	50577	8.002670e+02
SSS/SHS	Ghana	Urban	Female	18 years & older	852436	0	-1.000000e+02
Secondary	Ghana	All Locality Types	Both sexes	All ages	445319	3391	-9.923852e+01
Secondary	Ghana	All Locality Types	Both sexes	15-17 years	404	187272	4.625446e+04
Secondary	Ghana	All Locality Types	Both sexes	18 years & older	444915	360619	-1.894654e+01
Secondary	Ghana	All Locality Types	Male	All ages	265804	0	-1.000000e+02
Secondary	Ghana	All Locality Types	Male	15-17 years	203	0	-1.000000e+02
Secondary	Ghana	All Locality Types	Male	18 years & older	265601	305256	1.493029e+01
Secondary	Ghana	All Locality Types	Female	All ages	179515	0	-1.000000e+02
Secondary	Ghana	All Locality Types	Female	15-17 years	201	23	-8.855721e+01
Secondary	Ghana	All Locality Types	Female	18 years & older	179314	3775608	2.005585e+03
Secondary	Ghana	Rural	Both sexes	All ages	106583	14173	-8.670238e+01
Secondary	Ghana	Rural	Both sexes	15-17 years	123	16831	1.358374e+04
Secondary	Ghana	Rural	Both sexes	18 years & older	106460	544	-9.948901e+01
Secondary	Ghana	Rural	Male	All ages	72196	2104	-9.708571e+01
Secondary	Ghana	Rural	Male	15-17 years	63	52	-1.746032e+01
Secondary	Ghana	Rural	Male	18 years & older	72133	653	-9.909473e+01
Secondary	Ghana	Rural	Female	All ages	34387	179314	4.214587e+02
Secondary	Ghana	Rural	Female	15-17 years	60	37183	6.187167e+04
Secondary	Ghana	Rural	Female	18 years & older	34327	520885	1.417421e+03
Secondary	Ghana	Urban	Both sexes	All ages	338736	3876215	1.044317e+03
Secondary	Ghana	Urban	Both sexes	15-17 years	281	106583	3.782989e+04
Secondary	Ghana	Urban	Both sexes	18 years & older	338455	125721	-6.285444e+01
Secondary	Ghana	Urban	Male	All ages	193608	23	-9.998812e+01
Secondary	Ghana	Urban	Male	15-17 years	140	4527	3.133571e+03
Secondary	Ghana	Urban	Male	18 years & older	193468	90191	-5.338195e+01
Secondary	Ghana	Urban	Female	All ages	145128	0	-1.000000e+02
Secondary	Ghana	Urban	Female	15-17 years	141	0	-1.000000e+02
Secondary	Ghana	Urban	Female	18 years & older	144987	88	-9.993930e+01
Voc/technical/commercial	Ghana	All Locality Types	Both sexes	All ages	265826	28167	-8.940397e+01
Voc/technical/commercial	Ghana	All Locality Types	Both sexes	15-17 years	187	34327	1.825668e+04
Voc/technical/commercial	Ghana	All Locality Types	Both sexes	18 years & older	265639	744364	1.802164e+02
Voc/technical/commercial	Ghana	All Locality Types	Male	All ages	138112	5466	-9.604234e+01
Voc/technical/commercial	Ghana	All Locality Types	Male	15-17 years	96	1423179	1.482378e+06
Voc/technical/commercial	Ghana	All Locality Types	Male	18 years & older	138016	265826	9.260521e+01
Voc/technical/commercial	Ghana	All Locality Types	Female	All ages	127714	5466	-9.572012e+01
Voc/technical/commercial	Ghana	All Locality Types	Female	15-17 years	91	26	-7.142857e+01
Voc/technical/commercial	Ghana	All Locality Types	Female	18 years & older	127623	144987	1.360570e+01
Voc/technical/commercial	Ghana	Rural	Both sexes	All ages	49729	127623	1.566370e+02
Voc/technical/commercial	Ghana	Rural	Both sexes	15-17 years	31	252250	8.136097e+05
Voc/technical/commercial	Ghana	Rural	Both sexes	18 years & older	49698	678815	1.265880e+03
Voc/technical/commercial	Ghana	Rural	Male	All ages	30014	1110033	3.598384e+03
Voc/technical/commercial	Ghana	Rural	Male	15-17 years	8	280944	3.511700e+06
Voc/technical/commercial	Ghana	Rural	Male	18 years & older	30006	708104	2.259875e+03
Voc/technical/commercial	Ghana	Rural	Female	All ages	19715	179744	8.117119e+02
Voc/technical/commercial	Ghana	Rural	Female	15-17 years	23	21629	9.393913e+04
Voc/technical/commercial	Ghana	Rural	Female	18 years & older	19692	8	-9.995937e+01
Voc/technical/commercial	Ghana	Urban	Both sexes	All ages	216097	8552	-9.604252e+01
Voc/technical/commercial	Ghana	Urban	Both sexes	15-17 years	156	31	-8.012821e+01
Voc/technical/commercial	Ghana	Urban	Both sexes	18 years & older	215941	3305	-9.846949e+01
Voc/technical/commercial	Ghana	Urban	Male	All ages	108098	4520	-9.581861e+01
Voc/technical/commercial	Ghana	Urban	Male	15-17 years	88	90191	1.023898e+05
Voc/technical/commercial	Ghana	Urban	Male	18 years & older	108010	29149	-7.301268e+01
Voc/technical/commercial	Ghana	Urban	Female	All ages	107999	43509	-5.971352e+01
Voc/technical/commercial	Ghana	Urban	Female	15-17 years	68	628895	9.247456e+05
Post middle/secondary Certificate	Ghana	All Locality Types	Both sexes	All ages	73626	694510	8.432945e+02
Post middle/secondary Certificate	Ghana	All Locality Types	Both sexes	15-17 years	12	19715	1.641917e+05
Post middle/secondary Certificate	Ghana	All Locality Types	Both sexes	18 years & older	73614	43509	-4.089575e+01
Post middle/secondary Certificate	Ghana	All Locality Types	Male	All ages	37183	520885	1.300869e+03
Post middle/secondary Certificate	Ghana	All Locality Types	Male	15-17 years	7	126551	1.807771e+06
Post middle/secondary Certificate	Ghana	All Locality Types	Male	18 years & older	37176	641563	1.625745e+03
Post middle/secondary Certificate	Ghana	All Locality Types	Female	All ages	36443	0	-1.000000e+02
Post middle/secondary Certificate	Ghana	All Locality Types	Female	15-17 years	5	1737276	3.474542e+07
Post middle/secondary Certificate	Ghana	All Locality Types	Female	18 years & older	36438	19692	-4.595752e+01
Post middle/secondary Certificate	Ghana	Rural	Both sexes	All ages	19298	7860	-5.927039e+01
Post middle/secondary Certificate	Ghana	Rural	Both sexes	15-17 years	0	2155	Inf
Post middle/secondary Certificate	Ghana	Rural	Both sexes	18 years & older	19298	280944	1.355819e+03
Post middle/secondary Certificate	Ghana	Rural	Male	All ages	11027	852436	7.630443e+03
Post middle/secondary Certificate	Ghana	Rural	Male	15-17 years	0	228	Inf
Post middle/secondary Certificate	Ghana	Rural	Male	18 years & older	11027	38277	2.471207e+02
Post middle/secondary Certificate	Ghana	Rural	Female	All ages	8271	298144	3.504691e+03
Post middle/secondary Certificate	Ghana	Rural	Female	15-17 years	0	361673	Inf
Post middle/secondary Certificate	Ghana	Rural	Female	18 years & older	8271	1154485	1.385823e+04
Post middle/secondary Certificate	Ghana	Urban	Both sexes	All ages	54328	41655	-2.332683e+01
Post middle/secondary Certificate	Ghana	Urban	Both sexes	15-17 years	12	34387	2.864583e+05
Post middle/secondary Certificate	Ghana	Urban	Both sexes	18 years & older	54316	0	-1.000000e+02
Post middle/secondary Certificate	Ghana	Urban	Male	All ages	26156	1117893	4.173945e+03
Post middle/secondary Certificate	Ghana	Urban	Male	15-17 years	7	193608	2.765729e+06
Post middle/secondary Certificate	Ghana	Urban	Male	18 years & older	26149	91	-9.965199e+01
Post middle/secondary Certificate	Ghana	Urban	Female	All ages	28172	442555	1.470904e+03
Post middle/secondary Certificate	Ghana	Urban	Female	15-17 years	5	7	4.000000e+01
Post middle/secondary Certificate	Ghana	Urban	Female	18 years & older	28167	173400	5.156140e+02
Post middle/secondary Diploma	Ghana	All Locality Types	Both sexes	All ages	252281	835939	2.313523e+02
Post middle/secondary Diploma	Ghana	All Locality Types	Both sexes	15-17 years	31	1909800	6.160545e+06
Post middle/secondary Diploma	Ghana	All Locality Types	Male	All ages	126551	100607	-2.050083e+01
Post middle/secondary Diploma	Ghana	All Locality Types	Male	15-17 years	22	454344	2.065100e+06
Post middle/secondary Diploma	Ghana	All Locality Types	Male	18 years & older	126529	4170	-9.670431e+01
Post middle/secondary Diploma	Ghana	All Locality Types	Female	All ages	125730	265639	1.112773e+02
Post middle/secondary Diploma	Ghana	All Locality Types	Female	15-17 years	9	0	-1.000000e+02
Post middle/secondary Diploma	Ghana	All Locality Types	Female	18 years & older	125721	329	-9.973831e+01
Post middle/secondary Diploma	Ghana	Rural	Both sexes	15-17 years	8	8268	1.032500e+05
Post middle/secondary Diploma	Ghana	Rural	Both sexes	18 years & older	72529	2104	-9.709909e+01
Post middle/secondary Diploma	Ghana	Rural	Male	All ages	43388	0	-1.000000e+02
Post middle/secondary Diploma	Ghana	Rural	Male	15-17 years	6	12	1.000000e+02
Post middle/secondary Diploma	Ghana	Rural	Male	18 years & older	43382	858054	1.877903e+03
Post middle/secondary Diploma	Ghana	Rural	Female	All ages	29149	875277	2.902769e+03
Post middle/secondary Diploma	Ghana	Rural	Female	15-17 years	2	179515	8.975650e+06
Post middle/secondary Diploma	Ghana	Rural	Female	18 years & older	29147	1129904	3.776570e+03
Post middle/secondary Diploma	Ghana	Urban	Both sexes	15-17 years	23	76	2.304348e+02
Post middle/secondary Diploma	Ghana	Urban	Both sexes	18 years & older	179721	54316	-6.977760e+01
Post middle/secondary Diploma	Ghana	Urban	Male	All ages	83163	0	-1.000000e+02
Post middle/secondary Diploma	Ghana	Urban	Male	15-17 years	16	736145	4.600806e+06
Post middle/secondary Diploma	Ghana	Urban	Male	18 years & older	83147	0	-1.000000e+02
Post middle/secondary Diploma	Ghana	Urban	Female	All ages	96581	0	-1.000000e+02
Post middle/secondary Diploma	Ghana	Urban	Female	15-17 years	7	2503309	3.576146e+07
Post middle/secondary Diploma	Ghana	Urban	Female	18 years & older	96574	120584	2.486176e+01
Tertiary/HND	Ghana	All Locality Types	Both sexes	All ages	298162	36443	-8.777745e+01
Tertiary/HND	Ghana	All Locality Types	Both sexes	15-17 years	18	1861	1.023889e+04
Tertiary/HND	Ghana	All Locality Types	Both sexes	18 years & older	298144	1161	-9.961059e+01
Tertiary/HND	Ghana	All Locality Types	Male	All ages	184228	0	-1.000000e+02
Tertiary/HND	Ghana	All Locality Types	Male	15-17 years	12	62080	5.172333e+05
Tertiary/HND	Ghana	All Locality Types	Male	18 years & older	184216	1446124	6.850154e+02
Tertiary/HND	Ghana	All Locality Types	Female	All ages	113934	27	-9.997630e+01
Tertiary/HND	Ghana	All Locality Types	Female	15-17 years	6	0	-1.000000e+02
Tertiary/HND	Ghana	All Locality Types	Female	18 years & older	113928	0	-1.000000e+02
Tertiary/HND	Ghana	Rural	Both sexes	All ages	44788	716805	1.500440e+03
Tertiary/HND	Ghana	Rural	Both sexes	15-17 years	1	50031	5.003000e+06
Tertiary/HND	Ghana	Rural	Both sexes	18 years & older	44787	4017	-9.103088e+01
Tertiary/HND	Ghana	Rural	Male	All ages	30615	107999	2.527650e+02
Tertiary/HND	Ghana	Rural	Male	15-17 years	0	988926	Inf
Tertiary/HND	Ghana	Rural	Male	18 years & older	30615	890	-9.709293e+01
Tertiary/HND	Ghana	Rural	Female	All ages	14173	334644	2.261137e+03
Tertiary/HND	Ghana	Rural	Female	15-17 years	1	15120	1.511900e+06
Tertiary/HND	Ghana	Rural	Female	18 years & older	14172	656329	4.531167e+03
Tertiary/HND	Ghana	Urban	Both sexes	15-17 years	17	0	-1.000000e+02
Tertiary/HND	Ghana	Urban	Male	All ages	153613	10691	-9.304030e+01
Tertiary/HND	Ghana	Urban	Male	15-17 years	12	2242	1.858333e+04
Tertiary/HND	Ghana	Urban	Male	18 years & older	153601	939	-9.938868e+01
Tertiary/HND	Ghana	Urban	Female	All ages	99761	125730	2.603121e+01
Tertiary/HND	Ghana	Urban	Female	15-17 years	5	107931	2.158520e+06
Tertiary/HND	Ghana	Urban	Female	18 years & older	99756	18937	-8.101668e+01
Tertiary - Bachelor’s Degree	Ghana	All Locality Types	Both sexes	All ages	815102	120592	-8.520529e+01
Tertiary - Bachelor’s Degree	Ghana	All Locality Types	Both sexes	15-17 years	76	5356931	7.048493e+06
Tertiary - Bachelor’s Degree	Ghana	All Locality Types	Both sexes	18 years & older	815026	113934	-8.602081e+01
Tertiary - Bachelor’s Degree	Ghana	All Locality Types	Male	All ages	509846	253357	-5.030715e+01
Tertiary - Bachelor’s Degree	Ghana	All Locality Types	Male	15-17 years	50	509796	1.019492e+06
Tertiary - Bachelor’s Degree	Ghana	All Locality Types	Male	18 years & older	509796	19298	-9.621456e+01
Tertiary - Bachelor’s Degree	Ghana	All Locality Types	Female	All ages	305256	3817	-9.874957e+01
Tertiary - Bachelor’s Degree	Ghana	All Locality Types	Female	15-17 years	26	7413440	2.851313e+07
Tertiary - Bachelor’s Degree	Ghana	All Locality Types	Female	18 years & older	305230	2427	-9.920486e+01
Tertiary - Bachelor’s Degree	Ghana	Rural	Both sexes	All ages	120592	0	-1.000000e+02
Tertiary - Bachelor’s Degree	Ghana	Rural	Both sexes	15-17 years	8	45336	5.666000e+05
Tertiary - Bachelor’s Degree	Ghana	Rural	Both sexes	18 years & older	120584	17	-9.998590e+01
Tertiary - Bachelor’s Degree	Ghana	Rural	Male	All ages	84087	3746878	4.355954e+03
Tertiary - Bachelor’s Degree	Ghana	Rural	Male	15-17 years	5	37176	7.434200e+05
Tertiary - Bachelor’s Degree	Ghana	Rural	Male	18 years & older	84082	0	-1.000000e+02
Tertiary - Bachelor’s Degree	Ghana	Rural	Female	All ages	36505	68	-9.981372e+01
Tertiary - Bachelor’s Degree	Ghana	Rural	Female	15-17 years	3	30615	1.020400e+06
Tertiary - Bachelor’s Degree	Ghana	Rural	Female	18 years & older	36502	587240	1.508789e+03
Tertiary - Bachelor’s Degree	Ghana	Urban	Both sexes	All ages	694510	51605	-9.256958e+01
Tertiary - Bachelor’s Degree	Ghana	Urban	Both sexes	15-17 years	68	26149	3.835441e+04
Tertiary - Bachelor’s Degree	Ghana	Urban	Both sexes	18 years & older	694442	80993	-8.833697e+01
Tertiary - Bachelor’s Degree	Ghana	Urban	Male	All ages	425759	155512	-6.347417e+01
Tertiary - Bachelor’s Degree	Ghana	Urban	Male	15-17 years	45	1859223	4.131507e+06
Tertiary - Bachelor’s Degree	Ghana	Urban	Male	18 years & older	425714	123	-9.997111e+01
Tertiary - Bachelor’s Degree	Ghana	Urban	Female	All ages	268751	6164	-9.770643e+01
Tertiary - Bachelor’s Degree	Ghana	Urban	Female	15-17 years	23	47526	2.065348e+05
Tertiary - Bachelor’s Degree	Ghana	Urban	Female	18 years & older	268728	12	-9.999553e+01
Tertiary - Post graduate Certificate/Diploma	Ghana	All Locality Types	Both sexes	All ages	155512	25	-9.998392e+01
Tertiary - Post graduate Certificate/Diploma	Ghana	All Locality Types	Both sexes	15-17 years	0	265601	Inf
Tertiary - Post graduate Certificate/Diploma	Ghana	All Locality Types	Both sexes	18 years & older	155512	454344	1.921601e+02
Tertiary - Post graduate Certificate/Diploma	Ghana	All Locality Types	Male	All ages	77155	1809	-9.765537e+01
Tertiary - Post graduate Certificate/Diploma	Ghana	All Locality Types	Male	15-17 years	0	153613	Inf
Tertiary - Post graduate Certificate/Diploma	Ghana	All Locality Types	Male	18 years & older	77155	127714	6.552913e+01
Tertiary - Post graduate Certificate/Diploma	Ghana	All Locality Types	Female	All ages	78357	230	-9.970647e+01
Tertiary - Post graduate Certificate/Diploma	Ghana	All Locality Types	Female	15-17 years	0	1385416	Inf
Tertiary - Post graduate Certificate/Diploma	Ghana	Rural	Both sexes	All ages	37906	30006	-2.084103e+01
Tertiary - Post graduate Certificate/Diploma	Ghana	Rural	Both sexes	15-17 years	0	445319	Inf
Tertiary - Post graduate Certificate/Diploma	Ghana	Rural	Both sexes	18 years & older	37906	19298	-4.908985e+01
Tertiary - Post graduate Certificate/Diploma	Ghana	Rural	Male	All ages	21629	14172	-3.447686e+01
Tertiary - Post graduate Certificate/Diploma	Ghana	Rural	Male	15-17 years	0	2820	Inf
Tertiary - Post graduate Certificate/Diploma	Ghana	Rural	Male	18 years & older	21629	30962	4.315040e+01
Tertiary - Post graduate Certificate/Diploma	Ghana	Rural	Female	All ages	16277	0	-1.000000e+02
Tertiary - Post graduate Certificate/Diploma	Ghana	Rural	Female	18 years & older	16277	8268	-4.920440e+01
Tertiary - Post graduate Certificate/Diploma	Ghana	Urban	Both sexes	All ages	117606	55526	-5.278642e+01
Tertiary - Post graduate Certificate/Diploma	Ghana	Urban	Both sexes	15-17 years	0	24581	Inf
Tertiary - Post graduate Certificate/Diploma	Ghana	Urban	Both sexes	18 years & older	117606	11027	-9.062378e+01
Tertiary - Post graduate Certificate/Diploma	Ghana	Urban	Male	All ages	55526	72537	3.063610e+01
Tertiary - Post graduate Certificate/Diploma	Ghana	Urban	Male	15-17 years	0	99761	Inf
Tertiary - Post graduate Certificate/Diploma	Ghana	Urban	Male	18 years & older	55526	347485	5.258059e+02
Tertiary - Post graduate Certificate/Diploma	Ghana	Urban	Female	All ages	62080	1685	-9.728576e+01
Tertiary - Post graduate Certificate/Diploma	Ghana	Urban	Female	15-17 years	0	16	Inf
Tertiary - Post graduate Certificate/Diploma	Ghana	Urban	Female	18 years & older	62080	4735	-9.237274e+01
Tertiary - Master’s Degree	Ghana	All Locality Types	Both sexes	All ages	150187	2278	-9.848322e+01
Tertiary - Master’s Degree	Ghana	All Locality Types	Both sexes	15-17 years	0	458	Inf
Tertiary - Master’s Degree	Ghana	All Locality Types	Both sexes	18 years & older	150187	13734	-9.085540e+01
Tertiary - Master’s Degree	Ghana	All Locality Types	Male	All ages	102661	1377375	1.241673e+03
Tertiary - Master’s Degree	Ghana	All Locality Types	Male	15-17 years	0	15120	Inf
Tertiary - Master’s Degree	Ghana	All Locality Types	Male	18 years & older	102661	16487	-8.394035e+01
Tertiary - Master’s Degree	Ghana	All Locality Types	Female	All ages	47526	216097	3.546922e+02
Tertiary - Master’s Degree	Ghana	All Locality Types	Female	15-17 years	0	23	Inf
Tertiary - Master’s Degree	Ghana	All Locality Types	Female	18 years & older	47526	4170	-9.122586e+01
Tertiary - Master’s Degree	Ghana	Rural	Both sexes	All ages	16487	1915	-8.838479e+01
Tertiary - Master’s Degree	Ghana	Rural	Both sexes	15-17 years	0	19290	Inf
Tertiary - Master’s Degree	Ghana	Rural	Both sexes	18 years & older	16487	338455	1.952860e+03
Tertiary - Master’s Degree	Ghana	Rural	Male	All ages	12470	0	-1.000000e+02
Tertiary - Master’s Degree	Ghana	Rural	Male	15-17 years	0	63	Inf
Tertiary - Master’s Degree	Ghana	Rural	Male	18 years & older	12470	3043	-7.559743e+01
Tertiary - Master’s Degree	Ghana	Rural	Female	All ages	4017	9	-9.977595e+01
Tertiary - Master’s Degree	Ghana	Rural	Female	15-17 years	0	8041	Inf
Tertiary - Master’s Degree	Ghana	Rural	Female	18 years & older	4017	14626	2.641026e+02
Tertiary - Master’s Degree	Ghana	Urban	Both sexes	All ages	133700	39758	-7.026328e+01
Tertiary - Master’s Degree	Ghana	Urban	Both sexes	15-17 years	0	394212	Inf
Tertiary - Master’s Degree	Ghana	Urban	Both sexes	18 years & older	133700	108010	-1.921466e+01
Tertiary - Master’s Degree	Ghana	Urban	Male	All ages	90191	409720	3.542804e+02
Tertiary - Master’s Degree	Ghana	Urban	Male	15-17 years	0	96	Inf
Tertiary - Master’s Degree	Ghana	Urban	Male	18 years & older	90191	45	-9.995011e+01
Tertiary - Master’s Degree	Ghana	Urban	Female	All ages	43509	15901	-6.345354e+01
Tertiary - Master’s Degree	Ghana	Urban	Female	15-17 years	0	138016	Inf
Tertiary - Master’s Degree	Ghana	Urban	Female	18 years & older	43509	651809	1.398102e+03
Tertiary - PhD	Ghana	All Locality Types	Both sexes	All ages	19290	126529	5.559305e+02
Tertiary - PhD	Ghana	All Locality Types	Both sexes	15-17 years	0	187	Inf
Tertiary - PhD	Ghana	All Locality Types	Both sexes	18 years & older	19290	281	-9.854329e+01
Tertiary - PhD	Ghana	All Locality Types	Male	All ages	15120	2459	-8.373677e+01
Tertiary - PhD	Ghana	All Locality Types	Male	15-17 years	0	25996	Inf
Tertiary - PhD	Ghana	All Locality Types	Male	18 years & older	15120	4017	-7.343254e+01
Tertiary - PhD	Ghana	All Locality Types	Female	All ages	4170	0	-1.000000e+02
Tertiary - PhD	Ghana	All Locality Types	Female	18 years & older	4170	179721	4.209856e+03
Tertiary - PhD	Ghana	Rural	Both sexes	All ages	2459	143	-9.418463e+01
Tertiary - PhD	Ghana	Rural	Both sexes	15-17 years	0	2083766	Inf
Tertiary - PhD	Ghana	Rural	Both sexes	18 years & older	2459	0	-1.000000e+02
Tertiary - PhD	Ghana	Rural	Male	All ages	2045	12470	5.097800e+02
Tertiary - PhD	Ghana	Rural	Male	15-17 years	0	99	Inf
Tertiary - PhD	Ghana	Rural	Male	18 years & older	2045	22	-9.892421e+01
Tertiary - PhD	Ghana	Rural	Female	All ages	414	46907	1.123019e+04
Tertiary - PhD	Ghana	Rural	Female	15-17 years	0	1835599	Inf
Tertiary - PhD	Ghana	Rural	Female	18 years & older	414	1	-9.975845e+01
Tertiary - PhD	Ghana	Urban	Both sexes	All ages	16831	37906	1.252154e+02
Tertiary - PhD	Ghana	Urban	Both sexes	15-17 years	0	102661	Inf
Tertiary - PhD	Ghana	Urban	Both sexes	18 years & older	16831	31	-9.981582e+01
Tertiary - PhD	Ghana	Urban	Male	All ages	13075	102661	6.851702e+02
Tertiary - PhD	Ghana	Urban	Male	15-17 years	0	953862	Inf
Tertiary - PhD	Ghana	Urban	Male	18 years & older	13075	44787	2.425392e+02
Tertiary - PhD	Ghana	Urban	Female	All ages	3756	0	-1.000000e+02
Tertiary - PhD	Ghana	Urban	Female	15-17 years	0	2038332	Inf
Tertiary - PhD	Ghana	Urban	Female	18 years & older	3756	72529	1.831017e+03
Other (specify)	Ghana	All Locality Types	Both sexes	All ages	13734	215941	1.472310e+03
Other (specify)	Ghana	All Locality Types	Both sexes	15-17 years	0	30615	Inf
Other (specify)	Ghana	All Locality Types	Both sexes	18 years & older	13734	184228	1.241401e+03
Other (specify)	Ghana	All Locality Types	Male	All ages	8268	2331	-7.180697e+01
Other (specify)	Ghana	All Locality Types	Male	15-17 years	0	2087334	Inf
Other (specify)	Ghana	All Locality Types	Male	18 years & older	8268	96581	1.068130e+03
Other (specify)	Ghana	All Locality Types	Female	All ages	5466	444915	8.039682e+03
Other (specify)	Ghana	All Locality Types	Female	18 years & older	5466	1439	-7.367362e+01
Other (specify)	Ghana	Rural	Both sexes	All ages	3043	0	-1.000000e+02
Other (specify)	Ghana	Rural	Both sexes	15-17 years	0	939	Inf
Other (specify)	Ghana	Rural	Both sexes	18 years & older	3043	12470	3.097930e+02
Other (specify)	Ghana	Rural	Male	All ages	2104	7326775	3.481308e+05
Other (specify)	Ghana	Rural	Male	15-17 years	0	98512	Inf
Other (specify)	Ghana	Rural	Male	18 years & older	2104	48	-9.771863e+01
Other (specify)	Ghana	Rural	Female	All ages	939	156	-8.338658e+01
Other (specify)	Ghana	Rural	Female	15-17 years	0	1292	Inf
Other (specify)	Ghana	Rural	Female	18 years & older	939	8	-9.914803e+01
Other (specify)	Ghana	Urban	Both sexes	All ages	10691	12	-9.988776e+01
Other (specify)	Ghana	Urban	Both sexes	15-17 years	0	62080	Inf
Other (specify)	Ghana	Urban	Both sexes	18 years & older	10691	5455443	5.092837e+04
Other (specify)	Ghana	Urban	Male	All ages	6164	396490	6.332349e+03
Other (specify)	Ghana	Urban	Male	15-17 years	0	36502	Inf
Other (specify)	Ghana	Urban	Male	18 years & older	6164	0	-1.000000e+02
Other (specify)	Ghana	Urban	Female	All ages	4527	155512	3.335211e+03
Other (specify)	Ghana	Urban	Female	15-17 years	0	73626	Inf
Other (specify)	Ghana	Urban	Female	18 years & older	4527	253374	5.496952e+03

We see that we have some extremely high changes. For example, the value for the total across the variables has gone from 11,289,655 to 569. This clearly indicates a mistake. In this case, the mistake stems from our random data generation. We can instead do it in a more controlled process.

Controlled data generation

set.seed(42); new_year <- from_api %>% 
  mutate(`Population (15 years and older)_new_year` = 
           `Population (15 years and older)` * runif(n(), 0.9, 1.1),
         .keep = "unused")

pct_change <- from_api %>% 
  left_join(new_year,
            by = join_by(Highest_level_of_education, 
                         Geographic_Area, Locality, 
                         Sex, Age)) %>% 
  mutate(pct_change = (`Population (15 years and older)_new_year` - `Population (15 years and older)`)/
           `Population (15 years and older)`*100)

if(max(abs(pct_change$pct_change), na.rm = TRUE) > 10){
  print(pct_change %>% 
          filter(abs(pct_change) > 10))
  stop("There are changes above 10% for the new year. This may indicate problems with the data")
} else {
  cat("No changes above 10%")
}

No changes above 10%

Now we have no year-to-year changes above 10% (of course, because our data generation process set a limit of 10%). This limit of 10% is not set in stone and should be adjusted according to the content of the data. For example, some countries have seen inflation rates above 10%, so in that case the relevant number might be different.

Here we checked using data from the PX-web API, also to showcase the API. If the PX-file is available on disk, it may be easier to just load that and then do some validation checks.

The R package validate is specifically focused on tools and functions for validating data. For more information on data validation, see The Data Validation Cookbook, which introduces the R package validate and data validation concepts and implementations in R in general.

10.2 Check cross-sums

Usually in a Statbank/PX-web database, we have multiple tables covering one topic, e.g., multiple tables to cover the theme of population statistics. Often these tables will share some variables (sex and age would be prime examples), and therefore we expect values to be the same across tables.

We have our data collected from the Ghanaian StatsBank about education statistics for persons 15 years and older who attended school in the past.

We have another table about Population (3 years and older) by School Attendance Status, District, Region, Type of Locality, Age and Sex, and here we would expect that it shares some values with our table from earlier if we filter school attendance status to “attended in past” and age to over 15 years. Now we collect this data using the pxweb package.

check_table <- pxweb_get(url = "https://statsbank.statsghana.gov.gh:443/api/v1/en/PHC 2021 StatsBank/Education and Literacy/sch_attend_stat_table.px",
                      query = list(
                        Schoolattendancestatus = c("Attended in the past"),
                        Geographic_Area = c("Ghana"),
                        Locality = c("*"),
                        Sex = c("*"),
                        Age = c("15-19", "20-24", "25-29", "30-34",
                                "35-39", "40-44", "45-49", "50-54",
                                "55-59", "60-64", "65-69", "70-74",
                                "75-79", "80-84", "85-89", "90-94",
                                "95-99", "100+"))) %>% 
  as.data.frame() %>% 
  mutate(Locality = if_else(Locality == "All locality types", 
                            "All Locality Types",
                            Locality)) %>% 
  as_tibble() # as_tibble for nicer presentation

head(check_table) %>% 
  kableExtra::kable()

Schoolattendancestatus	Geographic_Area	Locality	Sex	Age	Population (3 years and older)
Attended in the past	Ghana	All Locality Types	Both sexes	15-19	542151
Attended in the past	Ghana	All Locality Types	Both sexes	20-24	1661537
Attended in the past	Ghana	All Locality Types	Both sexes	25-29	1923249
Attended in the past	Ghana	All Locality Types	Both sexes	30-34	1725033
Attended in the past	Ghana	All Locality Types	Both sexes	35-39	1448599
Attended in the past	Ghana	All Locality Types	Both sexes	40-44	1059990

This table we want to check uses another age definition, so we just need to sum up all the age groups to have persons over the age of 15.

check_table_above_15 <- check_table %>% 
  summarise(`Population (3 years and older)` = sum(`Population (3 years and older)`),
            .by = c(Schoolattendancestatus, Geographic_Area, Locality, Sex))

Now we have the values for persons aged 15 and over, which would correspond to the value for all ages in our initial table. Let’s select the corresponding variables between the tables and join them together.

cross_check <- from_api %>%
  filter(Age == "All ages" & Highest_level_of_education == "Total") %>% 
  select(Geographic_Area, Locality, Sex, `Population (15 years and older)`) %>% 
  ## Joining data together
  left_join(check_table_above_15 %>% 
              select(Geographic_Area, Locality, Sex, `Population (3 years and older)`),
            by = join_by(Geographic_Area, Locality, Sex)) %>% 
  ## Calculating difference
  mutate(diff = `Population (15 years and older)` - `Population (3 years and older)`)

Now we have joined the two tables together and calculated a difference variable that hopefully should be 0.

unique(cross_check$diff)

[1] 0

Which it is. The values shared across the two tables are the same, which is good as it indicates a consistent methodology in the two tables. This could also be a way to check a newly generated PX-table if you know that you have a PX-table in your database that shares some of the values.

10.1 The pxweb package

10.2 Check cross-sums

10.1 The `pxweb` package