Agricultural Data Preparation

Module 2: Economic Impact

Overview

Economic impact assessment requires detailed information about maize farmers: their land holdings, production volumes, input costs, and market participation. This information enables calculation of potential economic gains from biofortified maize adoption and identification of farmers most likely to benefit.

ENCOVI 2023 provides a comprehensive agricultural module, but the data require substantial preparation before analysis. Raw survey responses contain heterogeneous measurement units, missing values, and logical inconsistencies that must be systematically addressed.

This document describes the preparation of agricultural data through three stages:

  1. Unit corrections — Standardize land area to manzanas and production to quintales, resolving mixed units in survey responses
  2. Data integration — Combine information from multiple survey modules into a unified household-level dataset
  3. Quality treatment — Apply systematic cleaning including allocation normalization, structural NA handling, outlier detection, and imputation

Data Sources

ENCOVI 2023 includes a comprehensive agricultural module that captures farm-level information across multiple survey sections. We integrate data from:

  • Agricultural unit registry — Farm identification and poverty status
  • Crop production module — Cultivation practices, harvest quantities, input costs, and sales
  • Household demographics — Composition, location, and socioeconomic characteristics

The production data is filtered to white maize only (cod_cultivo = 142), the staple crop targeted by biofortification efforts in Guatemala.

Unit Corrections

ENCOVI respondents report land area and production quantities in heterogeneous units, and the recorded unit field is not always consistent with the magnitude of the reported value. Corrections are therefore applied on a plausibility basis rather than uniformly: each value is first assessed as if it were already expressed in the standard unit (manzanas for land, quintales for production), and a conversion is applied only when the reported magnitude is implausible under that assumption. Values that are already plausible are kept unchanged.

This conditional logic is reflected in the correction outcomes below: the large majority of land records (around 90%) require no correction, while the remaining records are reassigned to their most likely original unit based on the magnitude of the reported value. The “likely unit” terminology denotes this inference: the most plausible unit given the value, not a unit recorded as such in the source data.

Land Area Standardization

Values are standardized to manzanas (approximately 0.7 hectares) using range-based detection:

  • < 0.05 — Likely caballerías (×45)
  • 40–60 — Likely acres (×0.7)
  • 60–100 — Likely cuerdas (×0.25)
  • 100–350 — Likely cuadras (×0.025)
  • ≥ 350 — Likely square meters (÷7,000)
  • Zero/NA — Assigned minimum viable plot (0.05 mz)
Table 1: Land area unit corrections applied to white maize records
Land Area Unit Corrections. White maize records (cod_cultivo = 142)
Land Area Unit Corrections
White maize records (cod_cultivo = 142)
Correction Applied Records Percentage
No correction (manzanas) 1,130 92.3%
Zero/NA → 0.05 mz 22 1.8%
Caballerías (×45) 22 1.8%
Acres (×0.7) 15 1.2%
Cuerdas (×0.25) 4 0.3%
Cuadras (×0.025) 17 1.4%
m² (÷7000) 14 1.1%
Source: ENCOVI 2023, agricultural production module

Yield Validation

After unit corrections, yield distribution is validated against expected ranges for Guatemalan white maize production.

Table 2: White maize yield distribution after unit corrections
White Maize Yield Distribution. After unit corrections (qq/mz)
White Maize Yield Distribution
After unit corrections (qq/mz)
Metric Value
Sample Size
Total records 1,224
Records with valid yield 1,215
Distribution
Minimum (qq/mz) 0.0
Percentile 5 (qq/mz) 3.5
Percentile 25 (qq/mz) 12.0
Median (qq/mz) 25.0
Percentile 75 (qq/mz) 48.0
Percentile 95 (qq/mz) 100.0
Maximum (qq/mz) 138.9
Extreme Values
Records with yield < 3 qq/mz 17
Records with yield > 150 qq/mz 0
Expected range for Guatemala: 20-40 qq/mz for non-hybrid varieties (locally known as criollo or OPV), up to 80-100 qq/mz for commercial hybrids
NoteYield Validation

The median yield of approximately 25 qq/mz sits in the range reported for traditional Guatemalan non-hybrid maize varieties, which is the reference point against which the unit conversion is read.

Data Integration

Variables are derived from multiple ENCOVI modules and integrated at the household level. The integration includes land metrics, input expenditures, revenue and self-consumption, land characteristics, and farmer demographics.

Table 3: Data integration summary
Data Integration Summary. White maize farmers from ENCOVI 2023
Data Integration Summary
White maize farmers from ENCOVI 2023
Source Variables Description
B1: Land & Production 8 Land area, production, yield, harvest frequency
B2: Input Expenditures 25 All input expenditures (13 categories) and per-mz rates
B3: Revenue & Consumption 14 Revenue, self-consumption quantities and values, price
B4: Land Characteristics 6 Department, region, land tenure, poverty status
B5: Farmer Demographics 11 Farmer demographics and household composition
B6: Derived Metrics 3 Economic gain metrics (with/without costs)
Total Integrated 65 Complete household-level agricultural dataset
All dataframes joined by no_hogar (household ID)

Data Quality Treatment

This stage addresses data quality issues through systematic processes: allocation normalization, structural NA handling, impossible value detection, and imputation.

Production Allocation Consistency

Survey respondents often report destinations (sales, household consumption, animal feed/seed) that do not sum to total production. All destination quantities are scaled proportionally to match total_production.

The scaling preserves the relative distribution reported by each farmer. Every destination quantity (sales, household consumption, animal feed and seed) is multiplied by the ratio between total reported production and the sum of the reported destinations, so the adjusted destinations sum to total production while the share of each destination stays at its reported value.

\[ \text{qty\_adj}_i = \frac{\text{qty\_raw}_i}{\sum \text{qty\_raw}} \times \text{total\_production} \]

Table 4: Production allocation consistency diagnostic
Production Allocation Consistency Diagnostic. Distribution of farmers by allocation status
Production Allocation Consistency Diagnostic
Distribution of farmers by allocation status
Status Records % Records
Balanced 538 44.9%
Deficit (destinations < production) 357 29.8%
Excess (destinations > production) 265 22.1%
Positive prod, zero destinations 22 1.8%
Zero production 16 1.3%
Zero prod, positive destinations 1 0.1%
Excess: Reported destinations exceed production. Deficit: Destinations below production. Status compares the sum of destinations (sales, household consumption, animal feed and seed) against total production.

Structural NA Handling

Variables with NA representing “no activity” are converted to explicit zeros. Price variables remain NA for non-sellers since price is undefined without a transaction.

Table 5: Structural NAs converted to zero
Structural NAs Converted to Zero. NAs representing ‘no activity’ replaced with explicit zeros
Structural NAs Converted to Zero
NAs representing ‘no activity’ replaced with explicit zeros
Variable
NA Count
Converted Action
NAs Before NAs After
yield_wtcorn_annual 14 0 14 Converted to 0
exp_seeds_per_mz_annual 14 0 14 Converted to 0
exp_non_seed_per_mz_annual 14 0 14 Converted to 0
exp_total_per_mz_annual 14 0 14 Converted to 0
price_per_qq 853 853 0 Remains NA (price/rate)
gain_per_qq_with_cost 848 848 0 Remains NA (price/rate)
gain_per_qq_no_cost 848 848 0 Remains NA (price/rate)
farmer_score 43 43 0 Remains NA (price/rate)
Annual rates convert to 0 when no land; price variables remain NA when undefined

Impossible Value Detection

Domain-based rules identify logically impossible values, converting them to NA for subsequent imputation:

  • Harvest frequency > 3/year — Exceeds Guatemala’s agricultural calendar
  • Price outside 50–500 Q/qq — Outside typical Guatemalan range (150–250 Q/qq)
  • Seed expenditure > 5,000 Q/mz — Exceeds premium hybrid costs (~1,500 Q/mz max)
  • Household consumption > 15 qq/person/year — Physiologically implausible (>680 kg)
  • Animal/seed reserves > 500 qq — Exceeds plausible storage (>22 tons)
  • Yield > 160 qq/mz — Exceeds biological limits for white maize (>7.3 tons/mz)

Additionally, farmer_score — a composite indicator of likelihood of using non-hybrid varieties, computed from seed expenditure and yield patterns (see farmer segmentation for the full scoring methodology) — has existing NAs from the scoring process that will also be imputed.

Table 6: Impossible values flagged for imputation
Impossible Values Flagged for Imputation. Domain-based outlier detection
Impossible Values Flagged for Imputation
Domain-based outlier detection
Detection Criterion Variable Set to NA Cases
avg_harvests_per_plot > 3 n_harvests_total 10
price_per_qq > 500 qty_sold_qq 5
price_per_qq < 50 (when > 0) qty_sold_qq 19
qty_sold_qq = 0 & revenue > 0 qty_sold_qq 4
exp_seeds_per_mz > 5,000 exp_seeds 5
qty_household_qq / hh_size > 15 qty_household_qq 248
qty_animal_seed_qq > 500 qty_animal_seed_qq 1
farmer_score (existing NA) farmer_score 43
These base variables will be imputed using Random Forest in the next step

Random Forest Imputation

Missing values in base variables are imputed using a Random Forest-based multivariate imputation procedure. The method is non-parametric, handles mixed continuous and categorical data, and captures complex non-linear relationships between variables. Implementation uses the impute_rf_parallel() utility function, which runs a multi-threaded ranger engine and reports out-of-bag (OOB) error estimates separately for continuous variables (OOBE) and categorical variables (PFC).

Variables imputed: n_harvests_total, qty_sold_qq, exp_seeds, qty_household_qq, qty_animal_seed_qq, total_production, and farmer_score.

After imputation, structural values are restored for non-sellers (is_seller = 0): qty_sold_qq is set to zero (non-sellers have no sales by definition), and price and gain variables remain NA (undefined without a transaction).

Table 7: Random Forest imputation results
Random Forest Imputation Results. Multivariate imputation via impute_rf_parallel() (ranger engine)
Random Forest Imputation Results
Multivariate imputation via impute_rf_parallel() (ranger engine)
Variable NAs Before NAs After Values Imputed Status
n_harvests_total 10 0 10 Complete
qty_sold_qq 877 0 877 Complete
exp_seeds 5 0 5 Complete
qty_household_qq 248 0 248 Complete
qty_animal_seed_qq 1 0 1 Complete
farmer_score 43 43 0 Partial
Total — 1,184 — 1,141 —
Imputation parameters: maxiter = 10, ntree = 100, seed = 789
TipImputation Quality

Out-of-bag (OOB) error metrics — OOBE for continuous variables and PFC for categorical variables — provide internal quality estimates calculated from observations randomly excluded during tree construction within the Random Forest algorithm. This avoids the need for a separate test set. OOBE values below 0.50 are considered acceptable; PFC values below 0.20 indicate acceptable categorical accuracy.

Quality Treatment Summary

Table 8: Complete data quality treatment pipeline
Data Quality Treatment Summary. Systematic cleaning of agricultural dataset
Data Quality Treatment Summary
Systematic cleaning of agricultural dataset
Step Action Description Records Affected
Allocation Normalization
C1 Normalize allocations Redistribute destinations to sum to total_production All 1,199 households
Structural Values
C2 Structural NAs → 0 Annual rate variables when land_wtcorn_annual = 0 14 households
C2 Structural NAs → NA Price/gain variables for non-sellers (undefined, not zero) 844 households
Error Detection
C3 Impossible values → NA Domain-based detection of data entry errors ~90 values flagged
Imputation
C4 Random Forest imputation Impute base variables using random forest 1,167 values imputed
C4 Restore non-seller values qty_sold = 0, price/gain = NA for non-sellers 844 households
Recalculation & Correction
C5 Recalculate derived variables Regenerate all derived metrics from clean base All 1,199 households
C5 Normalize allocation quantities Ensure qty_sold + qty_household + qty_animal_seed = total_production All 1,199 households
C5 Hot-deck price correction Replace extreme prices (< 50 or > 500 Q/qq) with sampled valid values 87 households
Final dataset: white maize farmers ready for MAGA weight calibration and downstream segmentation

Summary

Key Findings

  1. Unit heterogeneity: A proportion of land area and production records required unit conversion, highlighting the importance of systematic standardization

  2. Yield distribution: Median yield aligns with expected values for traditional Guatemalan maize varieties, validating the conversion approach

  3. Quality treatment: The cleaning pipeline detected approximately 335 impossible values across 6 base variables and imputed 1,141 values via Random Forest, while preserving structural NAs for undefined metrics (price and gain for non-sellers)

Application

This cleaned dataset provides the foundation for subsequent analyses in this module:

  • MAGA weight calibration — Calibrate survey weights to official agricultural statistics
  • Farmer segmentation — Identify farmer typologies based on land holdings, input use, and market orientation
  • Economic impact calculation — Estimate potential gains from biofortified maize adoption
  • Adoption modeling — Predict which farmers are most likely to adopt biofortified varieties
Back to top